2024-09-19T07:05:19.428 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-19T07:05:19.451 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-19T07:05:19.542 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786 branch: squid description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7911786' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: cephfs: fs: - name: a - name: b session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on pool no app: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - Replacing daemon mds - SLOW_OPS - slow request sha1: 2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: cephfs_name: a client.1: cephfs_name: b install: ceph: flavor: default sha1: 2f599a46ad10e37c929e5c43da2f6907ddcd1961 kclient: client.0: cephfs_name: a client.1: cephfs_name: b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 6fad5dd73c93545e52f57f681e140a9c3529c11e owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 1747 sha1: 2f599a46ad10e37c929e5c43da2f6907ddcd1961 sleep_before_teardown: 0 subset: 176/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 6fad5dd73c93545e52f57f681e140a9c3529c11e syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGrcnfKIxfjcdLikolghc+tQMAndbmmY8LCPtFFmM5TBSbg66m9BPt2D5UU1KWULm7cukELeAjFxF0jIpaJmJPA= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF79BURdpun5WHyfcAsEejNcmvyG5AN+Fe9314mey3XSnMTMxXkONeNQ/K5IwZ7lvs5BsX67aumEzD9ia9eS7d8= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_21:24:17 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-19T07:05:19.542 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa; will attempt to use it 2024-09-19T07:05:19.543 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa/tasks 2024-09-19T07:05:19.543 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-19T07:05:19.545 INFO:teuthology.task.internal:Checking packages... 2024-09-19T07:05:19.567 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '2f599a46ad10e37c929e5c43da2f6907ddcd1961' 2024-09-19T07:05:19.568 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-19T07:05:19.568 INFO:teuthology.packaging:ref: None 2024-09-19T07:05:19.568 INFO:teuthology.packaging:tag: None 2024-09-19T07:05:19.568 INFO:teuthology.packaging:branch: squid 2024-09-19T07:05:19.568 INFO:teuthology.packaging:sha1: 2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:05:19.568 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-19T07:05:19.782 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-363-g6fad5dd7-1jammy 2024-09-19T07:05:19.783 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-19T07:05:19.790 INFO:teuthology.task.internal:no buildpackages task found 2024-09-19T07:05:19.790 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-19T07:05:19.851 INFO:teuthology.task.internal:Saving configuration 2024-09-19T07:05:19.875 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-19T07:05:19.883 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-19T07:05:19.907 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-19 06:59:44.642941', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHGDOpyoa34Dqf74Epi0A92YeNQigPspiTdEGrkhtlhK'} 2024-09-19T07:05:19.932 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-19 06:59:44.644641', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICyK5qebpgmWeHTE3Dm2zpyN6Vl+Kmou4xBzu7GspMzl'} 2024-09-19T07:05:19.932 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-19T07:05:19.939 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-19T07:05:19.940 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-09-19T07:05:19.940 INFO:teuthology.run_tasks:Running task console_log... 2024-09-19T07:05:20.004 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff9d511b0a0>, signals=[15]) 2024-09-19T07:05:20.004 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-19T07:05:20.011 INFO:teuthology.task.internal:Opening connections... 2024-09-19T07:05:20.011 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-09-19T07:05:20.013 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:05:20.097 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-09-19T07:05:20.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:05:20.174 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-19T07:05:20.181 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-09-19T07:05:20.187 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-09-19T07:05:20.187 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-09-19T07:05:20.232 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-19T07:05:20.232 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="Ubuntu" 2024-09-19T07:05:20.232 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="22.04" 2024-09-19T07:05:20.232 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-19T07:05:20.232 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_CODENAME=jammy 2024-09-19T07:05:20.233 INFO:teuthology.orchestra.run.smithi083.stdout:ID=ubuntu 2024-09-19T07:05:20.233 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE=debian 2024-09-19T07:05:20.233 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-19T07:05:20.233 INFO:teuthology.orchestra.run.smithi083.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-19T07:05:20.233 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-19T07:05:20.233 INFO:teuthology.orchestra.run.smithi083.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-19T07:05:20.233 INFO:teuthology.orchestra.run.smithi083.stdout:UBUNTU_CODENAME=jammy 2024-09-19T07:05:20.233 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-09-19T07:05:20.254 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2024-09-19T07:05:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2024-09-19T07:05:20.259 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-09-19T07:05:20.305 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-19T07:05:20.305 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="Ubuntu" 2024-09-19T07:05:20.305 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="22.04" 2024-09-19T07:05:20.305 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-19T07:05:20.305 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_CODENAME=jammy 2024-09-19T07:05:20.305 INFO:teuthology.orchestra.run.smithi184.stdout:ID=ubuntu 2024-09-19T07:05:20.306 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE=debian 2024-09-19T07:05:20.306 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-19T07:05:20.306 INFO:teuthology.orchestra.run.smithi184.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-19T07:05:20.306 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-19T07:05:20.306 INFO:teuthology.orchestra.run.smithi184.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-19T07:05:20.306 INFO:teuthology.orchestra.run.smithi184.stdout:UBUNTU_CODENAME=jammy 2024-09-19T07:05:20.306 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-09-19T07:05:20.329 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-19T07:05:20.339 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-19T07:05:20.347 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-19T07:05:20.347 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-09-19T07:05:20.350 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2024-09-19T07:05:20.353 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-19T07:05:20.360 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-19T07:05:20.360 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-09-19T07:05:20.398 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2024-09-19T07:05:20.423 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-19T07:05:20.480 INFO:teuthology.run_tasks:Running task kernel... 2024-09-19T07:05:20.495 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-19T07:05:20.496 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-19T07:05:20.496 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-19T07:05:20.497 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-19T07:05:20.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:05:20.503 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-09-19T07:05:20.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:05:20.505 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-09-19T07:05:20.550 INFO:teuthology.orchestra.run.smithi083.stdout:5.15.0-56-generic 2024-09-19T07:05:20.551 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.15.0-56-generic 2024-09-19T07:05:20.551 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get clean 2024-09-19T07:05:20.556 INFO:teuthology.orchestra.run.smithi184.stdout:5.15.0-56-generic 2024-09-19T07:05:20.557 INFO:teuthology.task.kernel:Running kernel on smithi184: 5.15.0-56-generic 2024-09-19T07:05:20.557 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get clean 2024-09-19T07:05:20.688 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get update 2024-09-19T07:05:20.703 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get update 2024-09-19T07:05:20.970 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-19T07:05:20.970 INFO:teuthology.orchestra.run.smithi083.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-19T07:05:20.972 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-19T07:05:20.972 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-19T07:05:21.026 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-19T07:05:21.035 INFO:teuthology.orchestra.run.smithi083.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-19T07:05:21.382 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-19T07:05:21.391 INFO:teuthology.orchestra.run.smithi083.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-19T07:05:21.398 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-19T07:05:21.412 INFO:teuthology.orchestra.run.smithi083.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-19T07:05:21.521 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [701 kB] 2024-09-19T07:05:21.529 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [701 kB] 2024-09-19T07:05:21.691 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-19T07:05:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,058 kB] 2024-09-19T07:05:21.736 INFO:teuthology.orchestra.run.smithi083.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,058 kB] 2024-09-19T07:05:21.835 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-19T07:05:21.851 INFO:teuthology.orchestra.run.smithi083.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-19T07:05:21.859 INFO:teuthology.orchestra.run.smithi083.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-19T07:05:21.860 INFO:teuthology.orchestra.run.smithi083.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-19T07:05:21.864 INFO:teuthology.orchestra.run.smithi184.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-19T07:05:21.877 INFO:teuthology.orchestra.run.smithi184.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [355 kB] 2024-09-19T07:05:21.877 INFO:teuthology.orchestra.run.smithi184.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-19T07:05:21.877 INFO:teuthology.orchestra.run.smithi184.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-19T07:05:21.885 INFO:teuthology.orchestra.run.smithi083.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [355 kB] 2024-09-19T07:05:21.890 INFO:teuthology.orchestra.run.smithi184.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-19T07:05:21.891 INFO:teuthology.orchestra.run.smithi184.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,495 kB] 2024-09-19T07:05:21.899 INFO:teuthology.orchestra.run.smithi083.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-19T07:05:21.900 INFO:teuthology.orchestra.run.smithi083.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-19T07:05:21.901 INFO:teuthology.orchestra.run.smithi083.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,495 kB] 2024-09-19T07:05:21.956 INFO:teuthology.orchestra.run.smithi083.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-19T07:05:21.961 INFO:teuthology.orchestra.run.smithi184.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-19T07:05:21.963 INFO:teuthology.orchestra.run.smithi184.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-19T07:05:21.970 INFO:teuthology.orchestra.run.smithi083.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-19T07:05:21.995 INFO:teuthology.orchestra.run.smithi184.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [429 kB] 2024-09-19T07:05:21.995 INFO:teuthology.orchestra.run.smithi184.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-19T07:05:22.002 INFO:teuthology.orchestra.run.smithi184.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-19T07:05:22.003 INFO:teuthology.orchestra.run.smithi184.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-19T07:05:22.003 INFO:teuthology.orchestra.run.smithi184.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [628 kB] 2024-09-19T07:05:22.004 INFO:teuthology.orchestra.run.smithi184.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-19T07:05:22.005 INFO:teuthology.orchestra.run.smithi083.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [429 kB] 2024-09-19T07:05:22.005 INFO:teuthology.orchestra.run.smithi083.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-19T07:05:22.007 INFO:teuthology.orchestra.run.smithi083.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [903 kB] 2024-09-19T07:05:22.013 INFO:teuthology.orchestra.run.smithi083.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-19T07:05:22.013 INFO:teuthology.orchestra.run.smithi083.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-19T07:05:22.014 INFO:teuthology.orchestra.run.smithi184.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [903 kB] 2024-09-19T07:05:22.017 INFO:teuthology.orchestra.run.smithi184.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-19T07:05:22.024 INFO:teuthology.orchestra.run.smithi083.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [628 kB] 2024-09-19T07:05:22.028 INFO:teuthology.orchestra.run.smithi184.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-19T07:05:22.031 INFO:teuthology.orchestra.run.smithi184.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-19T07:05:22.031 INFO:teuthology.orchestra.run.smithi184.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-19T07:05:22.031 INFO:teuthology.orchestra.run.smithi184.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-19T07:05:22.033 INFO:teuthology.orchestra.run.smithi083.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-19T07:05:22.035 INFO:teuthology.orchestra.run.smithi184.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-19T07:05:22.036 INFO:teuthology.orchestra.run.smithi083.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-19T07:05:22.039 INFO:teuthology.orchestra.run.smithi083.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-19T07:05:22.040 INFO:teuthology.orchestra.run.smithi184.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-19T07:05:22.040 INFO:teuthology.orchestra.run.smithi083.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-19T07:05:22.041 INFO:teuthology.orchestra.run.smithi083.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-19T07:05:22.041 INFO:teuthology.orchestra.run.smithi184.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-19T07:05:22.041 INFO:teuthology.orchestra.run.smithi184.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-19T07:05:22.044 INFO:teuthology.orchestra.run.smithi083.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-19T07:05:22.048 INFO:teuthology.orchestra.run.smithi083.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-19T07:05:22.049 INFO:teuthology.orchestra.run.smithi083.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-19T07:05:22.049 INFO:teuthology.orchestra.run.smithi083.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-19T07:05:22.050 INFO:teuthology.orchestra.run.smithi083.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-19T07:05:22.096 INFO:teuthology.orchestra.run.smithi083.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-19T07:05:22.097 INFO:teuthology.orchestra.run.smithi083.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-19T07:05:22.109 INFO:teuthology.orchestra.run.smithi184.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-19T07:05:22.110 INFO:teuthology.orchestra.run.smithi184.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-19T07:05:22.120 INFO:teuthology.orchestra.run.smithi184.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-19T07:05:22.121 INFO:teuthology.orchestra.run.smithi184.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-19T07:05:22.121 INFO:teuthology.orchestra.run.smithi184.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-19T07:05:22.122 INFO:teuthology.orchestra.run.smithi184.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-19T07:05:22.124 INFO:teuthology.orchestra.run.smithi184.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-19T07:05:22.125 INFO:teuthology.orchestra.run.smithi184.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-19T07:05:22.125 INFO:teuthology.orchestra.run.smithi184.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-19T07:05:22.126 INFO:teuthology.orchestra.run.smithi184.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-19T07:05:22.130 INFO:teuthology.orchestra.run.smithi083.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-19T07:05:22.131 INFO:teuthology.orchestra.run.smithi083.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-19T07:05:22.132 INFO:teuthology.orchestra.run.smithi083.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-19T07:05:22.133 INFO:teuthology.orchestra.run.smithi083.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-19T07:05:22.134 INFO:teuthology.orchestra.run.smithi083.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-19T07:05:22.134 INFO:teuthology.orchestra.run.smithi083.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-19T07:05:22.135 INFO:teuthology.orchestra.run.smithi083.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-19T07:05:22.205 INFO:teuthology.orchestra.run.smithi184.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-19T07:05:22.205 INFO:teuthology.orchestra.run.smithi184.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-19T07:05:22.212 INFO:teuthology.orchestra.run.smithi083.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-19T07:05:22.212 INFO:teuthology.orchestra.run.smithi083.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-19T07:05:28.152 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 16.2 MB in 3s (5,911 kB/s) 2024-09-19T07:05:28.224 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 16.2 MB in 3s (5,879 kB/s) 2024-09-19T07:05:29.582 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-19T07:05:29.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-19T07:05:29.661 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-19T07:05:29.671 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-19T07:05:29.688 DEBUG:teuthology.orchestra.run.smithi083:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-19T07:05:29.753 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-19T07:05:29.870 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2024-09-19T07:05:29.870 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2024-09-19T07:05:29.947 INFO:teuthology.orchestra.run.smithi083.stdout:Building dependency tree... 2024-09-19T07:05:29.947 INFO:teuthology.orchestra.run.smithi083.stdout:Reading state information... 2024-09-19T07:05:30.049 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages were automatically installed and are no longer required: 2024-09-19T07:05:30.049 INFO:teuthology.orchestra.run.smithi184.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-19T07:05:30.049 INFO:teuthology.orchestra.run.smithi184.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-19T07:05:30.050 INFO:teuthology.orchestra.run.smithi184.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-19T07:05:30.050 INFO:teuthology.orchestra.run.smithi184.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-19T07:05:30.050 INFO:teuthology.orchestra.run.smithi184.stdout: python2.7-minimal 2024-09-19T07:05:30.050 INFO:teuthology.orchestra.run.smithi184.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-19T07:05:30.051 INFO:teuthology.orchestra.run.smithi184.stdout:The following additional packages will be installed: 2024-09-19T07:05:30.051 INFO:teuthology.orchestra.run.smithi184.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-19T07:05:30.051 INFO:teuthology.orchestra.run.smithi184.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-19T07:05:30.052 INFO:teuthology.orchestra.run.smithi184.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-19T07:05:30.053 INFO:teuthology.orchestra.run.smithi184.stdout:Suggested packages: 2024-09-19T07:05:30.053 INFO:teuthology.orchestra.run.smithi184.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-19T07:05:30.053 INFO:teuthology.orchestra.run.smithi184.stdout:Recommended packages: 2024-09-19T07:05:30.053 INFO:teuthology.orchestra.run.smithi184.stdout: thermald 2024-09-19T07:05:30.099 INFO:teuthology.orchestra.run.smithi184.stdout:The following NEW packages will be installed: 2024-09-19T07:05:30.099 INFO:teuthology.orchestra.run.smithi184.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-19T07:05:30.099 INFO:teuthology.orchestra.run.smithi184.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-19T07:05:30.100 INFO:teuthology.orchestra.run.smithi184.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-19T07:05:30.100 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages will be upgraded: 2024-09-19T07:05:30.101 INFO:teuthology.orchestra.run.smithi184.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-19T07:05:30.116 INFO:teuthology.orchestra.run.smithi083.stdout:The following packages were automatically installed and are no longer required: 2024-09-19T07:05:30.116 INFO:teuthology.orchestra.run.smithi083.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-19T07:05:30.116 INFO:teuthology.orchestra.run.smithi083.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-19T07:05:30.117 INFO:teuthology.orchestra.run.smithi083.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-19T07:05:30.117 INFO:teuthology.orchestra.run.smithi083.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-19T07:05:30.117 INFO:teuthology.orchestra.run.smithi083.stdout: python2.7-minimal 2024-09-19T07:05:30.117 INFO:teuthology.orchestra.run.smithi083.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-19T07:05:30.118 INFO:teuthology.orchestra.run.smithi083.stdout:The following additional packages will be installed: 2024-09-19T07:05:30.118 INFO:teuthology.orchestra.run.smithi083.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-19T07:05:30.118 INFO:teuthology.orchestra.run.smithi083.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-19T07:05:30.119 INFO:teuthology.orchestra.run.smithi083.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-19T07:05:30.120 INFO:teuthology.orchestra.run.smithi083.stdout:Suggested packages: 2024-09-19T07:05:30.120 INFO:teuthology.orchestra.run.smithi083.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-19T07:05:30.120 INFO:teuthology.orchestra.run.smithi083.stdout:Recommended packages: 2024-09-19T07:05:30.120 INFO:teuthology.orchestra.run.smithi083.stdout: thermald 2024-09-19T07:05:30.170 INFO:teuthology.orchestra.run.smithi083.stdout:The following NEW packages will be installed: 2024-09-19T07:05:30.171 INFO:teuthology.orchestra.run.smithi083.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-19T07:05:30.171 INFO:teuthology.orchestra.run.smithi083.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-19T07:05:30.171 INFO:teuthology.orchestra.run.smithi083.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-19T07:05:30.172 INFO:teuthology.orchestra.run.smithi083.stdout:The following packages will be upgraded: 2024-09-19T07:05:30.173 INFO:teuthology.orchestra.run.smithi083.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-19T07:05:30.299 INFO:teuthology.orchestra.run.smithi184.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-19T07:05:30.299 INFO:teuthology.orchestra.run.smithi184.stdout:Need to get 113 MB of archives. 2024-09-19T07:05:30.299 INFO:teuthology.orchestra.run.smithi184.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-19T07:05:30.299 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-19T07:05:30.354 INFO:teuthology.orchestra.run.smithi083.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-19T07:05:30.354 INFO:teuthology.orchestra.run.smithi083.stdout:Need to get 113 MB of archives. 2024-09-19T07:05:30.354 INFO:teuthology.orchestra.run.smithi083.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-19T07:05:30.354 INFO:teuthology.orchestra.run.smithi083.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-19T07:05:31.755 INFO:teuthology.orchestra.run.smithi083.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-19T07:05:31.785 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-19T07:05:32.136 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-19T07:05:32.163 INFO:teuthology.orchestra.run.smithi083.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-19T07:05:34.304 INFO:teuthology.orchestra.run.smithi083.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-19T07:05:34.304 INFO:teuthology.orchestra.run.smithi083.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-19T07:05:34.305 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-19T07:05:34.336 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-19T07:05:34.337 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-19T07:05:34.337 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-19T07:05:34.648 INFO:teuthology.orchestra.run.smithi083.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-19T07:05:34.677 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-19T07:05:34.728 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-19T07:05:34.756 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-19T07:05:35.121 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 113 MB in 5s (24.9 MB/s) 2024-09-19T07:05:35.179 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 113 MB in 5s (24.4 MB/s) 2024-09-19T07:05:35.304 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-19T07:05:35.339 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-19T07:05:37.347 INFO:teuthology.orchestra.run.smithi083.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-19T07:05:37.352 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-19T07:05:37.438 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-19T07:05:37.442 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-19T07:05:37.460 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:05:37.712 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:05:39.789 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-19T07:05:39.810 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-19T07:05:39.878 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-19T07:05:39.899 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-19T07:05:39.903 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:05:39.999 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:05:40.350 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-19T07:05:40.371 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-19T07:05:40.408 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:05:40.481 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-19T07:05:40.501 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-19T07:05:40.550 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:05:47.044 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-19T07:05:47.153 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-19T07:05:47.311 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-19T07:05:47.411 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-19T07:05:47.478 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-19T07:05:47.598 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-19T07:05:47.693 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-19T07:05:47.814 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-19T07:05:47.910 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-19T07:05:47.932 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-19T07:05:47.968 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-19T07:05:48.126 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-19T07:05:48.148 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-19T07:05:48.192 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-19T07:05:53.243 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-19T07:05:53.279 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-19T07:05:53.317 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:05:54.179 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-19T07:05:54.216 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-19T07:05:54.281 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:05:55.918 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-19T07:05:56.044 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-19T07:05:57.295 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-19T07:05:57.424 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:05:57.550 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-19T07:05:57.654 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:05:58.045 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-19T07:05:58.177 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-19T07:05:58.551 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-19T07:05:58.664 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:05:58.765 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-19T07:05:58.894 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:05:58.975 INFO:teuthology.orchestra.run.smithi184.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-19T07:05:58.975 INFO:teuthology.orchestra.run.smithi184.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-19T07:05:58.975 INFO:teuthology.orchestra.run.smithi184.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-19T07:05:58.975 INFO:teuthology.orchestra.run.smithi184.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-19T07:05:59.130 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:06:00.229 INFO:teuthology.orchestra.run.smithi083.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-19T07:06:00.229 INFO:teuthology.orchestra.run.smithi083.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-19T07:06:00.229 INFO:teuthology.orchestra.run.smithi083.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-19T07:06:00.229 INFO:teuthology.orchestra.run.smithi083.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-19T07:06:00.396 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:06:00.556 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-19T07:06:00.674 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-19T07:06:00.783 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:06:02.148 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-19T07:06:02.273 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-19T07:06:02.383 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:06:02.794 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:06:03.481 INFO:teuthology.orchestra.run.smithi184.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-19T07:06:03.481 INFO:teuthology.orchestra.run.smithi184.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-19T07:06:04.512 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-19T07:06:04.572 INFO:teuthology.orchestra.run.smithi083.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-19T07:06:04.572 INFO:teuthology.orchestra.run.smithi083.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-19T07:06:17.468 INFO:teuthology.orchestra.run.smithi184.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-19T07:06:17.468 INFO:teuthology.orchestra.run.smithi184.stdout:Sourcing file `/etc/default/grub' 2024-09-19T07:06:17.474 INFO:teuthology.orchestra.run.smithi184.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-19T07:06:17.488 INFO:teuthology.orchestra.run.smithi184.stdout:Generating grub configuration file ... 2024-09-19T07:06:17.756 INFO:teuthology.orchestra.run.smithi184.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-19T07:06:17.776 INFO:teuthology.orchestra.run.smithi184.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-19T07:06:18.062 INFO:teuthology.orchestra.run.smithi184.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-19T07:06:18.067 INFO:teuthology.orchestra.run.smithi184.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-19T07:06:18.129 INFO:teuthology.orchestra.run.smithi184.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-19T07:06:18.134 INFO:teuthology.orchestra.run.smithi184.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-19T07:06:18.299 INFO:teuthology.orchestra.run.smithi184.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-19T07:06:18.300 INFO:teuthology.orchestra.run.smithi184.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-19T07:06:18.300 INFO:teuthology.orchestra.run.smithi184.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-19T07:06:18.338 INFO:teuthology.orchestra.run.smithi184.stdout:done 2024-09-19T07:06:18.748 INFO:teuthology.orchestra.run.smithi083.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-19T07:06:18.748 INFO:teuthology.orchestra.run.smithi083.stdout:Sourcing file `/etc/default/grub' 2024-09-19T07:06:18.780 INFO:teuthology.orchestra.run.smithi083.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-19T07:06:18.802 INFO:teuthology.orchestra.run.smithi083.stdout:Generating grub configuration file ... 2024-09-19T07:06:18.852 DEBUG:teuthology.orchestra.run.smithi184:> dpkg -s linux-image-generic 2024-09-19T07:06:18.875 INFO:teuthology.orchestra.run.smithi184.stdout:Package: linux-image-generic 2024-09-19T07:06:18.876 INFO:teuthology.orchestra.run.smithi184.stdout:Status: install ok installed 2024-09-19T07:06:18.876 INFO:teuthology.orchestra.run.smithi184.stdout:Priority: optional 2024-09-19T07:06:18.876 INFO:teuthology.orchestra.run.smithi184.stdout:Section: kernel 2024-09-19T07:06:18.876 INFO:teuthology.orchestra.run.smithi184.stdout:Installed-Size: 21 2024-09-19T07:06:18.876 INFO:teuthology.orchestra.run.smithi184.stdout:Maintainer: Ubuntu Kernel Team 2024-09-19T07:06:18.876 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: amd64 2024-09-19T07:06:18.876 INFO:teuthology.orchestra.run.smithi184.stdout:Source: linux-meta 2024-09-19T07:06:18.876 INFO:teuthology.orchestra.run.smithi184.stdout:Version: 5.15.0.122.122 2024-09-19T07:06:18.876 INFO:teuthology.orchestra.run.smithi184.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-19T07:06:18.876 INFO:teuthology.orchestra.run.smithi184.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-19T07:06:18.877 INFO:teuthology.orchestra.run.smithi184.stdout:Recommends: thermald 2024-09-19T07:06:18.877 INFO:teuthology.orchestra.run.smithi184.stdout:Description: Generic Linux kernel image 2024-09-19T07:06:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: This package will always depend on the latest generic kernel image 2024-09-19T07:06:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: available. 2024-09-19T07:06:18.877 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-19T07:06:18.877 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-19T07:06:18.877 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-19T07:06:18.878 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-19T07:06:18.878 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2024-09-19T07:06:18.878 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get clean 2024-09-19T07:06:19.040 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get update 2024-09-19T07:06:19.069 INFO:teuthology.orchestra.run.smithi083.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-19T07:06:19.093 INFO:teuthology.orchestra.run.smithi083.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-19T07:06:19.171 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-19T07:06:19.272 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-19T07:06:19.354 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-19T07:06:19.411 INFO:teuthology.orchestra.run.smithi083.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-19T07:06:19.417 INFO:teuthology.orchestra.run.smithi083.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-19T07:06:19.434 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-19T07:06:19.487 INFO:teuthology.orchestra.run.smithi083.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-19T07:06:19.493 INFO:teuthology.orchestra.run.smithi083.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-19T07:06:19.647 INFO:teuthology.orchestra.run.smithi083.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-19T07:06:19.647 INFO:teuthology.orchestra.run.smithi083.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-19T07:06:19.647 INFO:teuthology.orchestra.run.smithi083.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-19T07:06:19.694 INFO:teuthology.orchestra.run.smithi083.stdout:done 2024-09-19T07:06:20.265 DEBUG:teuthology.orchestra.run.smithi083:> dpkg -s linux-image-generic 2024-09-19T07:06:20.293 INFO:teuthology.orchestra.run.smithi083.stdout:Package: linux-image-generic 2024-09-19T07:06:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Status: install ok installed 2024-09-19T07:06:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Priority: optional 2024-09-19T07:06:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Section: kernel 2024-09-19T07:06:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Installed-Size: 21 2024-09-19T07:06:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Maintainer: Ubuntu Kernel Team 2024-09-19T07:06:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: amd64 2024-09-19T07:06:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Source: linux-meta 2024-09-19T07:06:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Version: 5.15.0.122.122 2024-09-19T07:06:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-19T07:06:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-19T07:06:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Recommends: thermald 2024-09-19T07:06:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Description: Generic Linux kernel image 2024-09-19T07:06:20.295 INFO:teuthology.orchestra.run.smithi083.stdout: This package will always depend on the latest generic kernel image 2024-09-19T07:06:20.295 INFO:teuthology.orchestra.run.smithi083.stdout: available. 2024-09-19T07:06:20.295 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-19T07:06:20.295 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-19T07:06:20.295 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-19T07:06:20.295 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-19T07:06:20.295 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2024-09-19T07:06:20.295 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get clean 2024-09-19T07:06:20.457 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get update 2024-09-19T07:06:20.581 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-19T07:06:20.585 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-19T07:06:20.603 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-19T07:06:20.659 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-19T07:06:20.677 DEBUG:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-19T07:06:20.698 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-19T07:06:20.742 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-19T07:06:20.961 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2024-09-19T07:06:20.961 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2024-09-19T07:06:21.155 INFO:teuthology.orchestra.run.smithi184.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-19T07:06:21.155 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages were automatically installed and are no longer required: 2024-09-19T07:06:21.156 INFO:teuthology.orchestra.run.smithi184.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-19T07:06:21.156 INFO:teuthology.orchestra.run.smithi184.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-19T07:06:21.156 INFO:teuthology.orchestra.run.smithi184.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-19T07:06:21.156 INFO:teuthology.orchestra.run.smithi184.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-19T07:06:21.156 INFO:teuthology.orchestra.run.smithi184.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-19T07:06:21.156 INFO:teuthology.orchestra.run.smithi184.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-19T07:06:21.199 INFO:teuthology.orchestra.run.smithi184.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-19T07:06:21.201 DEBUG:teuthology.orchestra.run.smithi184:> dpkg -s linux-image-generic 2024-09-19T07:06:21.216 INFO:teuthology.orchestra.run.smithi184.stdout:Package: linux-image-generic 2024-09-19T07:06:21.216 INFO:teuthology.orchestra.run.smithi184.stdout:Status: install ok installed 2024-09-19T07:06:21.216 INFO:teuthology.orchestra.run.smithi184.stdout:Priority: optional 2024-09-19T07:06:21.216 INFO:teuthology.orchestra.run.smithi184.stdout:Section: kernel 2024-09-19T07:06:21.216 INFO:teuthology.orchestra.run.smithi184.stdout:Installed-Size: 21 2024-09-19T07:06:21.216 INFO:teuthology.orchestra.run.smithi184.stdout:Maintainer: Ubuntu Kernel Team 2024-09-19T07:06:21.217 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: amd64 2024-09-19T07:06:21.217 INFO:teuthology.orchestra.run.smithi184.stdout:Source: linux-meta 2024-09-19T07:06:21.217 INFO:teuthology.orchestra.run.smithi184.stdout:Version: 5.15.0.122.122 2024-09-19T07:06:21.217 INFO:teuthology.orchestra.run.smithi184.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-19T07:06:21.217 INFO:teuthology.orchestra.run.smithi184.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-19T07:06:21.217 INFO:teuthology.orchestra.run.smithi184.stdout:Recommends: thermald 2024-09-19T07:06:21.217 INFO:teuthology.orchestra.run.smithi184.stdout:Description: Generic Linux kernel image 2024-09-19T07:06:21.217 INFO:teuthology.orchestra.run.smithi184.stdout: This package will always depend on the latest generic kernel image 2024-09-19T07:06:21.217 INFO:teuthology.orchestra.run.smithi184.stdout: available. 2024-09-19T07:06:21.217 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-09-19T07:06:21.261 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.MQyDKEJ1Z6 2024-09-19T07:06:21.261 DEBUG:teuthology.orchestra.run.smithi184:> sudo cp /boot/grub/grub.cfg /tmp/tmp.MQyDKEJ1Z6 2024-09-19T07:06:21.316 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.MQyDKEJ1Z6 2024-09-19T07:06:21.439 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.MQyDKEJ1Z6 is 10KB 2024-09-19T07:06:21.452 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.MQyDKEJ1Z6 2024-09-19T07:06:21.458 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-19T07:06:21.514 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:06:21.514 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-19T07:06:21.514 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-19T07:06:21.581 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-19T07:06:21.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo update-grub 2024-09-19T07:06:22.028 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-19T07:06:22.046 DEBUG:teuthology.orchestra.run.smithi083:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-19T07:06:22.099 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-19T07:06:22.286 INFO:teuthology.orchestra.run.smithi083.stdout:Building dependency tree... 2024-09-19T07:06:22.286 INFO:teuthology.orchestra.run.smithi083.stdout:Reading state information... 2024-09-19T07:06:22.458 INFO:teuthology.orchestra.run.smithi083.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-19T07:06:22.458 INFO:teuthology.orchestra.run.smithi083.stdout:The following packages were automatically installed and are no longer required: 2024-09-19T07:06:22.458 INFO:teuthology.orchestra.run.smithi083.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-19T07:06:22.458 INFO:teuthology.orchestra.run.smithi083.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-19T07:06:22.458 INFO:teuthology.orchestra.run.smithi083.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-19T07:06:22.459 INFO:teuthology.orchestra.run.smithi083.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-19T07:06:22.459 INFO:teuthology.orchestra.run.smithi083.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-19T07:06:22.459 INFO:teuthology.orchestra.run.smithi083.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-19T07:06:22.499 INFO:teuthology.orchestra.run.smithi083.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-19T07:06:22.500 DEBUG:teuthology.orchestra.run.smithi083:> dpkg -s linux-image-generic 2024-09-19T07:06:22.562 INFO:teuthology.orchestra.run.smithi083.stdout:Package: linux-image-generic 2024-09-19T07:06:22.563 INFO:teuthology.orchestra.run.smithi083.stdout:Status: install ok installed 2024-09-19T07:06:22.563 INFO:teuthology.orchestra.run.smithi083.stdout:Priority: optional 2024-09-19T07:06:22.563 INFO:teuthology.orchestra.run.smithi083.stdout:Section: kernel 2024-09-19T07:06:22.563 INFO:teuthology.orchestra.run.smithi083.stdout:Installed-Size: 21 2024-09-19T07:06:22.563 INFO:teuthology.orchestra.run.smithi083.stdout:Maintainer: Ubuntu Kernel Team 2024-09-19T07:06:22.563 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: amd64 2024-09-19T07:06:22.563 INFO:teuthology.orchestra.run.smithi083.stdout:Source: linux-meta 2024-09-19T07:06:22.563 INFO:teuthology.orchestra.run.smithi083.stdout:Version: 5.15.0.122.122 2024-09-19T07:06:22.563 INFO:teuthology.orchestra.run.smithi083.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-19T07:06:22.563 INFO:teuthology.orchestra.run.smithi083.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-19T07:06:22.563 INFO:teuthology.orchestra.run.smithi083.stdout:Recommends: thermald 2024-09-19T07:06:22.564 INFO:teuthology.orchestra.run.smithi083.stdout:Description: Generic Linux kernel image 2024-09-19T07:06:22.564 INFO:teuthology.orchestra.run.smithi083.stdout: This package will always depend on the latest generic kernel image 2024-09-19T07:06:22.564 INFO:teuthology.orchestra.run.smithi083.stdout: available. 2024-09-19T07:06:22.564 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-09-19T07:06:22.607 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.krHFelS2ad 2024-09-19T07:06:22.608 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub/grub.cfg /tmp/tmp.krHFelS2ad 2024-09-19T07:06:22.663 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.krHFelS2ad 2024-09-19T07:06:22.792 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.krHFelS2ad is 10KB 2024-09-19T07:06:22.805 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.krHFelS2ad 2024-09-19T07:06:22.811 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-19T07:06:22.811 INFO:teuthology.orchestra.run.smithi184.stderr:Sourcing file `/etc/default/grub' 2024-09-19T07:06:22.811 INFO:teuthology.orchestra.run.smithi184.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-19T07:06:22.813 INFO:teuthology.orchestra.run.smithi184.stderr:Generating grub configuration file ... 2024-09-19T07:06:22.863 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:06:22.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-19T07:06:22.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-19T07:06:22.930 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-19T07:06:22.930 DEBUG:teuthology.orchestra.run.smithi083:> sudo update-grub 2024-09-19T07:06:22.977 INFO:teuthology.orchestra.run.smithi184.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-19T07:06:22.983 INFO:teuthology.orchestra.run.smithi184.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-19T07:06:23.214 INFO:teuthology.orchestra.run.smithi184.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-19T07:06:23.219 INFO:teuthology.orchestra.run.smithi184.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-19T07:06:23.256 INFO:teuthology.orchestra.run.smithi184.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-19T07:06:23.261 INFO:teuthology.orchestra.run.smithi184.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-19T07:06:23.358 INFO:teuthology.orchestra.run.smithi184.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-19T07:06:23.358 INFO:teuthology.orchestra.run.smithi184.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-19T07:06:23.358 INFO:teuthology.orchestra.run.smithi184.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-19T07:06:23.374 INFO:teuthology.orchestra.run.smithi184.stderr:done 2024-09-19T07:06:23.383 DEBUG:teuthology.orchestra.run.smithi184:> sudo shutdown -r now 2024-09-19T07:06:24.288 INFO:teuthology.orchestra.run.smithi083.stderr:Sourcing file `/etc/default/grub' 2024-09-19T07:06:24.289 INFO:teuthology.orchestra.run.smithi083.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-19T07:06:24.291 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2024-09-19T07:06:24.474 INFO:teuthology.orchestra.run.smithi083.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-19T07:06:24.480 INFO:teuthology.orchestra.run.smithi083.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-19T07:06:24.770 INFO:teuthology.orchestra.run.smithi083.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-19T07:06:24.775 INFO:teuthology.orchestra.run.smithi083.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-19T07:06:24.819 INFO:teuthology.orchestra.run.smithi083.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-19T07:06:24.825 INFO:teuthology.orchestra.run.smithi083.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-19T07:06:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-19T07:06:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-19T07:06:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-19T07:06:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:done 2024-09-19T07:06:25.008 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2024-09-19T07:06:53.391 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-19T07:06:53.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-19T07:06:53.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:06:55.011 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-19T07:06:55.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-19T07:06:55.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:07:11.809 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-09-19T07:07:13.341 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-09-19T07:07:20.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-19T07:07:20.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:07:22.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-19T07:07:22.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:07:23.869 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-09-19T07:07:25.405 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-09-19T07:07:35.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-19T07:07:35.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:07:37.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-19T07:07:37.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:07:54.305 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-09-19T07:07:55.841 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-09-19T07:08:09.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-19T07:08:09.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:08:09.575 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-09-19T07:08:10.144 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-19T07:08:10.144 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-122-generic"... 2024-09-19T07:08:10.144 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-09-19T07:08:10.191 INFO:teuthology.orchestra.run.smithi184.stdout:5.15.0-122-generic 2024-09-19T07:08:10.191 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-19T07:08:10.191 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-19T07:08:10.191 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-19T07:08:10.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-19T07:08:10.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:08:11.192 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-19T07:08:11.193 DEBUG:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-19T07:08:11.257 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2024-09-19T07:08:11.290 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-09-19T07:08:11.300 DEBUG:teuthology.parallel:result is None 2024-09-19T07:08:12.233 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-19T07:08:12.233 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-122-generic"... 2024-09-19T07:08:12.234 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-09-19T07:08:12.279 INFO:teuthology.orchestra.run.smithi083.stdout:5.15.0-122-generic 2024-09-19T07:08:12.279 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-19T07:08:12.279 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-19T07:08:12.280 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-19T07:08:13.281 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-19T07:08:13.281 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-19T07:08:13.342 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-09-19T07:08:13.365 DEBUG:teuthology.parallel:result is None 2024-09-19T07:08:13.365 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-19T07:08:13.374 INFO:teuthology.task.internal:Creating test directory... 2024-09-19T07:08:13.374 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-19T07:08:13.377 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-19T07:08:13.383 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-19T07:08:13.394 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-19T07:08:13.408 INFO:teuthology.task.internal:Creating archive directory... 2024-09-19T07:08:13.408 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-19T07:08:13.424 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-19T07:08:13.470 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-19T07:08:13.482 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-19T07:08:13.483 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-19T07:08:13.485 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-19T07:08:13.510 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T07:08:13.518 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T07:08:13.544 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T07:08:13.552 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T07:08:13.554 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-19T07:08:13.571 INFO:teuthology.task.internal:Configuring sudo... 2024-09-19T07:08:13.571 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-19T07:08:13.574 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-19T07:08:13.621 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-19T07:08:13.643 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-19T07:08:13.643 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-19T07:08:13.648 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-19T07:08:13.671 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T07:08:13.703 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T07:08:13.754 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:08:13.754 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-19T07:08:13.823 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T07:08:13.829 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T07:08:13.875 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:08:13.875 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-19T07:08:13.936 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-09-19T07:08:13.939 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2024-09-19T07:08:14.009 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-19T07:08:14.068 INFO:teuthology.task.internal:Starting timer... 2024-09-19T07:08:14.069 INFO:teuthology.run_tasks:Running task pcp... 2024-09-19T07:08:14.088 INFO:teuthology.run_tasks:Running task selinux... 2024-09-19T07:08:14.097 DEBUG:teuthology.task.selinux:Excluding smithi083: OS 'ubuntu' does not support SELinux 2024-09-19T07:08:14.097 DEBUG:teuthology.task.selinux:Excluding smithi184: OS 'ubuntu' does not support SELinux 2024-09-19T07:08:14.097 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-19T07:08:14.097 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-19T07:08:14.097 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-19T07:08:14.097 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-19T07:08:14.112 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-19T07:08:14.123 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-19T07:08:14.125 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-19T07:15:45.737 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2024-09-19T07:15:45.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-19T07:15:45.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:15:45.822 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-09-19T07:15:45.897 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-19T07:15:45.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-19T07:15:45.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T07:15:45.978 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-09-19T07:15:46.057 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-19T07:15:46.057 INFO:teuthology.run_tasks:Running task clock... 2024-09-19T07:15:46.068 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-19T07:15:46.069 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-19T07:15:46.069 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T07:15:46.071 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-19T07:15:46.071 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T07:15:46.101 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-19T07:15:46.101 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: Command line: ntpd -gq 2024-09-19T07:15:46.101 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: ---------------------------------------------------- 2024-09-19T07:15:46.101 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: ntp-4 is maintained by Network Time Foundation, 2024-09-19T07:15:46.101 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-19T07:15:46.101 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: corporation. Support and training for ntp-4 are 2024-09-19T07:15:46.101 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: available at https://www.nwtime.org/support 2024-09-19T07:15:46.101 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: ---------------------------------------------------- 2024-09-19T07:15:46.101 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: proto: precision = 0.058 usec (-24) 2024-09-19T07:15:46.102 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: basedate set to 2022-02-04 2024-09-19T07:15:46.102 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: gps base set to 2022-02-06 (week 2196) 2024-09-19T07:15:46.102 INFO:teuthology.orchestra.run.smithi083.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-19T07:15:46.103 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-19T07:15:46.103 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: restrict ::: KOD does nothing without LIMITED. 2024-09-19T07:15:46.103 INFO:teuthology.orchestra.run.smithi083.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-19T07:15:46.104 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: Listen and drop on 0 v6wildcard [::]:123 2024-09-19T07:15:46.104 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-19T07:15:46.104 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: Listen normally on 2 lo 127.0.0.1:123 2024-09-19T07:15:46.104 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: Listen normally on 3 enp3s0f1 172.21.15.83:123 2024-09-19T07:15:46.104 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: Listen normally on 4 lo [::1]:123 2024-09-19T07:15:46.104 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce91%5]:123 2024-09-19T07:15:46.105 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:46 ntpd[18490]: Listening on routing socket on fd #22 for interface updates 2024-09-19T07:15:46.122 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-19T07:15:46.123 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: Command line: ntpd -gq 2024-09-19T07:15:46.123 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: ---------------------------------------------------- 2024-09-19T07:15:46.123 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: ntp-4 is maintained by Network Time Foundation, 2024-09-19T07:15:46.123 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-19T07:15:46.123 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: corporation. Support and training for ntp-4 are 2024-09-19T07:15:46.123 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: available at https://www.nwtime.org/support 2024-09-19T07:15:46.123 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: ---------------------------------------------------- 2024-09-19T07:15:46.124 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: proto: precision = 0.045 usec (-24) 2024-09-19T07:15:46.124 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: basedate set to 2022-02-04 2024-09-19T07:15:46.124 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: gps base set to 2022-02-06 (week 2196) 2024-09-19T07:15:46.124 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-19T07:15:46.124 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: restrict ::: KOD does nothing without LIMITED. 2024-09-19T07:15:46.124 INFO:teuthology.orchestra.run.smithi184.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-19T07:15:46.124 INFO:teuthology.orchestra.run.smithi184.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-19T07:15:46.125 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: Listen and drop on 0 v6wildcard [::]:123 2024-09-19T07:15:46.125 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-19T07:15:46.125 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: Listen normally on 2 lo 127.0.0.1:123 2024-09-19T07:15:46.126 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: Listen normally on 3 enp3s0f1 172.21.15.184:123 2024-09-19T07:15:46.126 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: Listen normally on 4 lo [::1]:123 2024-09-19T07:15:46.126 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7b8f%5]:123 2024-09-19T07:15:46.126 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:46 ntpd[18492]: Listening on routing socket on fd #22 for interface updates 2024-09-19T07:15:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:47 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:47 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:47 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:47 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:47 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:47 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:47 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:47 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:48 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:48 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:48 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:48 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:48 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:48 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:48 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:48 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:48 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:48 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:48 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:48 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:48.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:48 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:48.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:48 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:48.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:48 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:48.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:48 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:48.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:48 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:48.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:48 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:48.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:48 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:48.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:48 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:48.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:48 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:48.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:48 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:48.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:48 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:48.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:48 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:49.101 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:49 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:49 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:49 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:49 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:49 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:49 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:49.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:49 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:49.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:49 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:49.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:49 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:49.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:49 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:49.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:49 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:49.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:49 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:50.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:50 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:50.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:50 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:50.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:50 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:50.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:50 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:50.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:50 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:50.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:50 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:50.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:50 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:50.103 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:50 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:50.103 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:50 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:50.103 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:50 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:50.103 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:50 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:50.103 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:50 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:50.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:50.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:50.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:50.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:50.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:50.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:50.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:50.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:50.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:50.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:50.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:50.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:50.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:50.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:50 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:51.101 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:51 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:51.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:51 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:51.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:51 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:51.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:51 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:51.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:51 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:51.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:51 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:51.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:51 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:51.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:51 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:51.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:51 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:51.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:51 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:51.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:51 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:51.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:51 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:52.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:52 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:52.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:52 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:52.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:52 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:52.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:52 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:52.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:52 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:52.102 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:52 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:52.103 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:52 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:52.103 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:52 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:52.103 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:52 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:52.103 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:52 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:52.103 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:52 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:52.103 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:52 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:52.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:52 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:52.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:52 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:52.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:52 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:52.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:52 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:52.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:52 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:52.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:52 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:52.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:52 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:52.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:52 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:52.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:52 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:52.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:52 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:52.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:52 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:52.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:52 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:53 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:53.104 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:53 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:53.104 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:53 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:53.104 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:53 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:53.104 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:53 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:53.104 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:53 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:53.105 INFO:teuthology.orchestra.run.smithi083.stdout:19 Sep 07:15:53 ntpd[18490]: ntpd: time slew -0.001457 s 2024-09-19T07:15:53.105 INFO:teuthology.orchestra.run.smithi083.stdout:ntpd: time slew -0.001457s 2024-09-19T07:15:53.105 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:53 ntpd[18490]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-19T07:15:53.105 INFO:teuthology.orchestra.run.smithi083.stderr:19 Sep 07:15:53 ntpd[18490]: can't open /var/log/ntpstats/loopstats.20240919: Permission denied 2024-09-19T07:15:53.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:53 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:53.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:53 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:53.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:53 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:53.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:53 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:53.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:53 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:53.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:53 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:53.153 INFO:teuthology.orchestra.run.smithi083.stdout: remote refid st t when poll reach delay offset jitter 2024-09-19T07:15:53.153 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================== 2024-09-19T07:15:53.153 INFO:teuthology.orchestra.run.smithi083.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-19T07:15:53.153 INFO:teuthology.orchestra.run.smithi083.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-19T07:15:53.153 INFO:teuthology.orchestra.run.smithi083.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-19T07:15:53.153 INFO:teuthology.orchestra.run.smithi083.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-19T07:15:54.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:54 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:54.123 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:54 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:54.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:54 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:54.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:54 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:54.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:54 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-19T07:15:54.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:54 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240919: Permission denied 2024-09-19T07:15:54.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:54 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-19T07:15:54.124 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:54 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240919: Permission denied 2024-09-19T07:15:54.124 INFO:teuthology.orchestra.run.smithi184.stdout:19 Sep 07:15:54 ntpd[18492]: ntpd: time slew +0.000938 s 2024-09-19T07:15:54.124 INFO:teuthology.orchestra.run.smithi184.stdout:ntpd: time slew +0.000938s 2024-09-19T07:15:54.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:54 ntpd[18492]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-19T07:15:54.125 INFO:teuthology.orchestra.run.smithi184.stderr:19 Sep 07:15:54 ntpd[18492]: can't open /var/log/ntpstats/loopstats.20240919: Permission denied 2024-09-19T07:15:54.172 INFO:teuthology.orchestra.run.smithi184.stdout: remote refid st t when poll reach delay offset jitter 2024-09-19T07:15:54.172 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================== 2024-09-19T07:15:54.172 INFO:teuthology.orchestra.run.smithi184.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-19T07:15:54.172 INFO:teuthology.orchestra.run.smithi184.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-19T07:15:54.172 INFO:teuthology.orchestra.run.smithi184.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-19T07:15:54.172 INFO:teuthology.orchestra.run.smithi184.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-19T07:15:54.173 INFO:teuthology.run_tasks:Running task install... 2024-09-19T07:15:54.182 DEBUG:teuthology.task.install:project ceph 2024-09-19T07:15:54.182 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961'}} 2024-09-19T07:15:54.182 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961'} 2024-09-19T07:15:54.183 INFO:teuthology.task.install:Using flavor: default 2024-09-19T07:15:54.188 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-19T07:15:54.188 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-19T07:15:54.189 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-key list | grep Ceph 2024-09-19T07:15:54.189 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-key list | grep Ceph 2024-09-19T07:15:54.254 INFO:teuthology.orchestra.run.smithi083.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-19T07:15:54.273 INFO:teuthology.orchestra.run.smithi184.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-19T07:15:54.282 INFO:teuthology.orchestra.run.smithi083.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-19T07:15:54.282 INFO:teuthology.orchestra.run.smithi083.stdout:uid [ unknown] Ceph.com (release key) 2024-09-19T07:15:54.283 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-09-19T07:15:54.284 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-09-19T07:15:54.284 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:15:54.300 INFO:teuthology.orchestra.run.smithi184.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-19T07:15:54.300 INFO:teuthology.orchestra.run.smithi184.stdout:uid [ unknown] Ceph.com (release key) 2024-09-19T07:15:54.300 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-09-19T07:15:54.300 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-09-19T07:15:54.300 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:15:54.436 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default/ 2024-09-19T07:15:54.436 INFO:teuthology.task.install.deb:Package version is 19.1.1-361-g2f599a46-1jammy 2024-09-19T07:15:54.445 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default/ 2024-09-19T07:15:54.445 INFO:teuthology.task.install.deb:Package version is 19.1.1-361-g2f599a46-1jammy 2024-09-19T07:15:54.545 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:15:54.545 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-19T07:15:54.555 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:15:54.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-19T07:15:54.559 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get update 2024-09-19T07:15:54.569 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get update 2024-09-19T07:15:54.729 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-19T07:15:54.729 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-19T07:15:54.738 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-19T07:15:54.747 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-19T07:15:54.764 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-19T07:15:54.780 INFO:teuthology.orchestra.run.smithi083.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy InRelease 2024-09-19T07:15:54.790 INFO:teuthology.orchestra.run.smithi184.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy InRelease 2024-09-19T07:15:54.803 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-19T07:15:54.811 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-19T07:15:54.825 INFO:teuthology.orchestra.run.smithi083.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-19T07:15:54.832 INFO:teuthology.orchestra.run.smithi184.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-19T07:15:54.846 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-19T07:15:54.853 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-19T07:15:54.860 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-19T07:15:54.872 INFO:teuthology.orchestra.run.smithi083.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,419 B] 2024-09-19T07:15:54.881 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,419 B] 2024-09-19T07:15:54.941 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-19T07:15:55.023 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-19T07:15:55.079 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 29.0 kB in 0s (84.7 kB/s) 2024-09-19T07:15:55.169 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 29.0 kB in 0s (69.9 kB/s) 2024-09-19T07:15:56.079 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-19T07:15:56.095 DEBUG:teuthology.orchestra.run.smithi083:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-361-g2f599a46-1jammy cephadm=19.1.1-361-g2f599a46-1jammy ceph-mds=19.1.1-361-g2f599a46-1jammy ceph-mgr=19.1.1-361-g2f599a46-1jammy ceph-common=19.1.1-361-g2f599a46-1jammy ceph-fuse=19.1.1-361-g2f599a46-1jammy ceph-test=19.1.1-361-g2f599a46-1jammy ceph-volume=19.1.1-361-g2f599a46-1jammy radosgw=19.1.1-361-g2f599a46-1jammy python3-rados=19.1.1-361-g2f599a46-1jammy python3-rgw=19.1.1-361-g2f599a46-1jammy python3-cephfs=19.1.1-361-g2f599a46-1jammy python3-rbd=19.1.1-361-g2f599a46-1jammy libcephfs2=19.1.1-361-g2f599a46-1jammy libcephfs-dev=19.1.1-361-g2f599a46-1jammy librados2=19.1.1-361-g2f599a46-1jammy librbd1=19.1.1-361-g2f599a46-1jammy rbd-fuse=19.1.1-361-g2f599a46-1jammy python3-cephfs=19.1.1-361-g2f599a46-1jammy cephfs-shell=19.1.1-361-g2f599a46-1jammy cephfs-top=19.1.1-361-g2f599a46-1jammy cephfs-mirror=19.1.1-361-g2f599a46-1jammy 2024-09-19T07:15:56.157 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-19T07:15:56.225 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-19T07:15:56.243 DEBUG:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-361-g2f599a46-1jammy cephadm=19.1.1-361-g2f599a46-1jammy ceph-mds=19.1.1-361-g2f599a46-1jammy ceph-mgr=19.1.1-361-g2f599a46-1jammy ceph-common=19.1.1-361-g2f599a46-1jammy ceph-fuse=19.1.1-361-g2f599a46-1jammy ceph-test=19.1.1-361-g2f599a46-1jammy ceph-volume=19.1.1-361-g2f599a46-1jammy radosgw=19.1.1-361-g2f599a46-1jammy python3-rados=19.1.1-361-g2f599a46-1jammy python3-rgw=19.1.1-361-g2f599a46-1jammy python3-cephfs=19.1.1-361-g2f599a46-1jammy python3-rbd=19.1.1-361-g2f599a46-1jammy libcephfs2=19.1.1-361-g2f599a46-1jammy libcephfs-dev=19.1.1-361-g2f599a46-1jammy librados2=19.1.1-361-g2f599a46-1jammy librbd1=19.1.1-361-g2f599a46-1jammy rbd-fuse=19.1.1-361-g2f599a46-1jammy python3-cephfs=19.1.1-361-g2f599a46-1jammy cephfs-shell=19.1.1-361-g2f599a46-1jammy cephfs-top=19.1.1-361-g2f599a46-1jammy cephfs-mirror=19.1.1-361-g2f599a46-1jammy 2024-09-19T07:15:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-19T07:15:56.369 INFO:teuthology.orchestra.run.smithi083.stdout:Building dependency tree... 2024-09-19T07:15:56.370 INFO:teuthology.orchestra.run.smithi083.stdout:Reading state information... 2024-09-19T07:15:56.484 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2024-09-19T07:15:56.484 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2024-09-19T07:15:56.579 INFO:teuthology.orchestra.run.smithi083.stdout:The following packages were automatically installed and are no longer required: 2024-09-19T07:15:56.579 INFO:teuthology.orchestra.run.smithi083.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-19T07:15:56.579 INFO:teuthology.orchestra.run.smithi083.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-19T07:15:56.579 INFO:teuthology.orchestra.run.smithi083.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-19T07:15:56.579 INFO:teuthology.orchestra.run.smithi083.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-19T07:15:56.580 INFO:teuthology.orchestra.run.smithi083.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-19T07:15:56.580 INFO:teuthology.orchestra.run.smithi083.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-19T07:15:56.580 INFO:teuthology.orchestra.run.smithi083.stdout:The following additional packages will be installed: 2024-09-19T07:15:56.580 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-19T07:15:56.580 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-19T07:15:56.581 INFO:teuthology.orchestra.run.smithi083.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-19T07:15:56.582 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-19T07:15:56.583 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-09-19T07:15:56.583 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-09-19T07:15:56.583 INFO:teuthology.orchestra.run.smithi083.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-19T07:15:56.583 INFO:teuthology.orchestra.run.smithi083.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-09-19T07:15:56.583 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-09-19T07:15:56.583 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-09-19T07:15:56.583 INFO:teuthology.orchestra.run.smithi083.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-09-19T07:15:56.583 INFO:teuthology.orchestra.run.smithi083.stdout:Suggested packages: 2024-09-19T07:15:56.583 INFO:teuthology.orchestra.run.smithi083.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-19T07:15:56.584 INFO:teuthology.orchestra.run.smithi083.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-19T07:15:56.584 INFO:teuthology.orchestra.run.smithi083.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-19T07:15:56.584 INFO:teuthology.orchestra.run.smithi083.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-19T07:15:56.584 INFO:teuthology.orchestra.run.smithi083.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-19T07:15:56.584 INFO:teuthology.orchestra.run.smithi083.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-19T07:15:56.584 INFO:teuthology.orchestra.run.smithi083.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-19T07:15:56.584 INFO:teuthology.orchestra.run.smithi083.stdout:Recommended packages: 2024-09-19T07:15:56.584 INFO:teuthology.orchestra.run.smithi083.stdout: btrfs-tools 2024-09-19T07:15:56.671 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages were automatically installed and are no longer required: 2024-09-19T07:15:56.671 INFO:teuthology.orchestra.run.smithi184.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-19T07:15:56.671 INFO:teuthology.orchestra.run.smithi184.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-19T07:15:56.671 INFO:teuthology.orchestra.run.smithi184.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-19T07:15:56.672 INFO:teuthology.orchestra.run.smithi184.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-19T07:15:56.672 INFO:teuthology.orchestra.run.smithi184.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-19T07:15:56.673 INFO:teuthology.orchestra.run.smithi184.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-19T07:15:56.673 INFO:teuthology.orchestra.run.smithi184.stdout:The following additional packages will be installed: 2024-09-19T07:15:56.673 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-19T07:15:56.673 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-19T07:15:56.673 INFO:teuthology.orchestra.run.smithi184.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-19T07:15:56.673 INFO:teuthology.orchestra.run.smithi184.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-19T07:15:56.674 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-19T07:15:56.674 INFO:teuthology.orchestra.run.smithi184.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-19T07:15:56.674 INFO:teuthology.orchestra.run.smithi184.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-19T07:15:56.674 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-19T07:15:56.674 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-19T07:15:56.674 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-09-19T07:15:56.674 INFO:teuthology.orchestra.run.smithi184.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-09-19T07:15:56.674 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-19T07:15:56.674 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-19T07:15:56.675 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-19T07:15:56.675 INFO:teuthology.orchestra.run.smithi184.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-19T07:15:56.675 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-19T07:15:56.675 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-19T07:15:56.675 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-09-19T07:15:56.675 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-09-19T07:15:56.675 INFO:teuthology.orchestra.run.smithi184.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-19T07:15:56.675 INFO:teuthology.orchestra.run.smithi184.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-09-19T07:15:56.675 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-09-19T07:15:56.675 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-09-19T07:15:56.675 INFO:teuthology.orchestra.run.smithi184.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-09-19T07:15:56.676 INFO:teuthology.orchestra.run.smithi184.stdout:Suggested packages: 2024-09-19T07:15:56.676 INFO:teuthology.orchestra.run.smithi184.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-19T07:15:56.676 INFO:teuthology.orchestra.run.smithi184.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-19T07:15:56.676 INFO:teuthology.orchestra.run.smithi184.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-19T07:15:56.676 INFO:teuthology.orchestra.run.smithi184.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-19T07:15:56.676 INFO:teuthology.orchestra.run.smithi184.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-19T07:15:56.676 INFO:teuthology.orchestra.run.smithi184.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-19T07:15:56.676 INFO:teuthology.orchestra.run.smithi184.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-19T07:15:56.676 INFO:teuthology.orchestra.run.smithi184.stdout:Recommended packages: 2024-09-19T07:15:56.676 INFO:teuthology.orchestra.run.smithi184.stdout: btrfs-tools 2024-09-19T07:15:56.694 INFO:teuthology.orchestra.run.smithi083.stdout:The following NEW packages will be installed: 2024-09-19T07:15:56.694 INFO:teuthology.orchestra.run.smithi083.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-19T07:15:56.694 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-19T07:15:56.694 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-19T07:15:56.694 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-09-19T07:15:56.694 INFO:teuthology.orchestra.run.smithi083.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-19T07:15:56.694 INFO:teuthology.orchestra.run.smithi083.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-09-19T07:15:56.694 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-09-19T07:15:56.694 INFO:teuthology.orchestra.run.smithi083.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-09-19T07:15:56.694 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-19T07:15:56.695 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-19T07:15:56.696 INFO:teuthology.orchestra.run.smithi083.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-19T07:15:56.696 INFO:teuthology.orchestra.run.smithi083.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-19T07:15:56.696 INFO:teuthology.orchestra.run.smithi083.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-19T07:15:56.696 INFO:teuthology.orchestra.run.smithi083.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-19T07:15:56.696 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-19T07:15:56.696 INFO:teuthology.orchestra.run.smithi083.stdout: socat unzip xmlstarlet zip 2024-09-19T07:15:56.783 INFO:teuthology.orchestra.run.smithi184.stdout:The following NEW packages will be installed: 2024-09-19T07:15:56.783 INFO:teuthology.orchestra.run.smithi184.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-19T07:15:56.783 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-19T07:15:56.783 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-19T07:15:56.783 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-09-19T07:15:56.785 INFO:teuthology.orchestra.run.smithi184.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-19T07:15:56.785 INFO:teuthology.orchestra.run.smithi184.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-09-19T07:15:56.785 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-09-19T07:15:56.785 INFO:teuthology.orchestra.run.smithi184.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-09-19T07:15:56.785 INFO:teuthology.orchestra.run.smithi184.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-09-19T07:15:56.785 INFO:teuthology.orchestra.run.smithi184.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-19T07:15:56.785 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-19T07:15:56.785 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-09-19T07:15:56.785 INFO:teuthology.orchestra.run.smithi184.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-09-19T07:15:56.785 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-19T07:15:56.785 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-19T07:15:56.786 INFO:teuthology.orchestra.run.smithi184.stdout: socat unzip xmlstarlet zip 2024-09-19T07:15:56.839 INFO:teuthology.orchestra.run.smithi083.stdout:0 upgraded, 124 newly installed, 0 to remove and 331 not upgraded. 2024-09-19T07:15:56.840 INFO:teuthology.orchestra.run.smithi083.stdout:Need to get 181 MB of archives. 2024-09-19T07:15:56.840 INFO:teuthology.orchestra.run.smithi083.stdout:After this operation, 831 MB of additional disk space will be used. 2024-09-19T07:15:56.840 INFO:teuthology.orchestra.run.smithi083.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-361-g2f599a46-1jammy [3,590 kB] 2024-09-19T07:15:56.846 INFO:teuthology.orchestra.run.smithi184.stdout:0 upgraded, 124 newly installed, 0 to remove and 331 not upgraded. 2024-09-19T07:15:56.846 INFO:teuthology.orchestra.run.smithi184.stdout:Need to get 181 MB of archives. 2024-09-19T07:15:56.846 INFO:teuthology.orchestra.run.smithi184.stdout:After this operation, 831 MB of additional disk space will be used. 2024-09-19T07:15:56.846 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-09-19T07:15:56.863 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-19T07:15:56.873 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-09-19T07:15:56.882 INFO:teuthology.orchestra.run.smithi083.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-09-19T07:15:56.914 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-19T07:15:56.927 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-361-g2f599a46-1jammy [3,590 kB] 2024-09-19T07:15:56.935 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-19T07:15:56.938 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-19T07:15:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-19T07:15:56.964 INFO:teuthology.orchestra.run.smithi083.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-19T07:15:56.997 INFO:teuthology.orchestra.run.smithi184.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-19T07:15:57.001 INFO:teuthology.orchestra.run.smithi184.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-19T07:15:57.010 INFO:teuthology.orchestra.run.smithi083.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-09-19T07:15:57.012 INFO:teuthology.orchestra.run.smithi184.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-19T07:15:57.016 INFO:teuthology.orchestra.run.smithi184.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-19T07:15:57.017 INFO:teuthology.orchestra.run.smithi184.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-19T07:15:57.018 INFO:teuthology.orchestra.run.smithi083.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-361-g2f599a46-1jammy [3,241 kB] 2024-09-19T07:15:57.025 INFO:teuthology.orchestra.run.smithi184.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-19T07:15:57.027 INFO:teuthology.orchestra.run.smithi184.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-19T07:15:57.029 INFO:teuthology.orchestra.run.smithi184.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-19T07:15:57.030 INFO:teuthology.orchestra.run.smithi184.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-19T07:15:57.030 INFO:teuthology.orchestra.run.smithi184.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-19T07:15:57.034 INFO:teuthology.orchestra.run.smithi184.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-19T07:15:57.036 INFO:teuthology.orchestra.run.smithi184.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-19T07:15:57.050 INFO:teuthology.orchestra.run.smithi184.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-19T07:15:57.050 INFO:teuthology.orchestra.run.smithi184.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-19T07:15:57.051 INFO:teuthology.orchestra.run.smithi184.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-19T07:15:57.052 INFO:teuthology.orchestra.run.smithi184.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-19T07:15:57.052 INFO:teuthology.orchestra.run.smithi184.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-19T07:15:57.053 INFO:teuthology.orchestra.run.smithi184.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-19T07:15:57.053 INFO:teuthology.orchestra.run.smithi184.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-19T07:15:57.053 INFO:teuthology.orchestra.run.smithi184.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-19T07:15:57.069 INFO:teuthology.orchestra.run.smithi184.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-19T07:15:57.069 INFO:teuthology.orchestra.run.smithi184.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-19T07:15:57.070 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-361-g2f599a46-1jammy [719 kB] 2024-09-19T07:15:57.081 INFO:teuthology.orchestra.run.smithi083.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-361-g2f599a46-1jammy [357 kB] 2024-09-19T07:15:57.086 INFO:teuthology.orchestra.run.smithi184.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-19T07:15:57.086 INFO:teuthology.orchestra.run.smithi184.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-19T07:15:57.087 INFO:teuthology.orchestra.run.smithi184.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-19T07:15:57.087 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-361-g2f599a46-1jammy [32.8 kB] 2024-09-19T07:15:57.087 INFO:teuthology.orchestra.run.smithi184.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-19T07:15:57.088 INFO:teuthology.orchestra.run.smithi083.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-361-g2f599a46-1jammy [180 kB] 2024-09-19T07:15:57.088 INFO:teuthology.orchestra.run.smithi184.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-19T07:15:57.088 INFO:teuthology.orchestra.run.smithi184.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-19T07:15:57.089 INFO:teuthology.orchestra.run.smithi184.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-19T07:15:57.090 INFO:teuthology.orchestra.run.smithi184.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-19T07:15:57.090 INFO:teuthology.orchestra.run.smithi083.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-361-g2f599a46-1jammy [67.8 kB] 2024-09-19T07:15:57.092 INFO:teuthology.orchestra.run.smithi083.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-361-g2f599a46-1jammy [333 kB] 2024-09-19T07:15:57.099 INFO:teuthology.orchestra.run.smithi083.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-361-g2f599a46-1jammy [6,875 kB] 2024-09-19T07:15:57.103 INFO:teuthology.orchestra.run.smithi184.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-19T07:15:57.103 INFO:teuthology.orchestra.run.smithi184.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-19T07:15:57.120 INFO:teuthology.orchestra.run.smithi184.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-19T07:15:57.121 INFO:teuthology.orchestra.run.smithi184.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-19T07:15:57.121 INFO:teuthology.orchestra.run.smithi184.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-19T07:15:57.121 INFO:teuthology.orchestra.run.smithi184.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-19T07:15:57.123 INFO:teuthology.orchestra.run.smithi184.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-19T07:15:57.124 INFO:teuthology.orchestra.run.smithi184.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-19T07:15:57.125 INFO:teuthology.orchestra.run.smithi184.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-19T07:15:57.126 INFO:teuthology.orchestra.run.smithi184.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-19T07:15:57.137 INFO:teuthology.orchestra.run.smithi184.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-19T07:15:57.139 INFO:teuthology.orchestra.run.smithi184.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-19T07:15:57.154 INFO:teuthology.orchestra.run.smithi184.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-19T07:15:57.155 INFO:teuthology.orchestra.run.smithi184.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-19T07:15:57.157 INFO:teuthology.orchestra.run.smithi184.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-19T07:15:57.206 INFO:teuthology.orchestra.run.smithi083.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-19T07:15:57.207 INFO:teuthology.orchestra.run.smithi184.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-19T07:15:57.208 INFO:teuthology.orchestra.run.smithi184.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-19T07:15:57.209 INFO:teuthology.orchestra.run.smithi184.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-19T07:15:57.209 INFO:teuthology.orchestra.run.smithi184.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-19T07:15:57.210 INFO:teuthology.orchestra.run.smithi184.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-19T07:15:57.238 INFO:teuthology.orchestra.run.smithi083.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-361-g2f599a46-1jammy [112 kB] 2024-09-19T07:15:57.240 INFO:teuthology.orchestra.run.smithi083.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-361-g2f599a46-1jammy [472 kB] 2024-09-19T07:15:57.240 INFO:teuthology.orchestra.run.smithi184.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-19T07:15:57.242 INFO:teuthology.orchestra.run.smithi184.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-19T07:15:57.242 INFO:teuthology.orchestra.run.smithi184.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-19T07:15:57.246 INFO:teuthology.orchestra.run.smithi083.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-361-g2f599a46-1jammy [25.9 MB] 2024-09-19T07:15:57.258 INFO:teuthology.orchestra.run.smithi184.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-19T07:15:57.258 INFO:teuthology.orchestra.run.smithi184.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-19T07:15:57.258 INFO:teuthology.orchestra.run.smithi184.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-19T07:15:57.258 INFO:teuthology.orchestra.run.smithi184.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-19T07:15:57.259 INFO:teuthology.orchestra.run.smithi184.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-19T07:15:57.259 INFO:teuthology.orchestra.run.smithi184.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-19T07:15:57.260 INFO:teuthology.orchestra.run.smithi184.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-19T07:15:57.277 INFO:teuthology.orchestra.run.smithi184.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-19T07:15:57.280 INFO:teuthology.orchestra.run.smithi184.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-19T07:15:57.282 INFO:teuthology.orchestra.run.smithi184.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-19T07:15:57.283 INFO:teuthology.orchestra.run.smithi184.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-19T07:15:57.283 INFO:teuthology.orchestra.run.smithi184.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-19T07:15:57.286 INFO:teuthology.orchestra.run.smithi184.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-19T07:15:57.289 INFO:teuthology.orchestra.run.smithi184.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-09-19T07:15:57.289 INFO:teuthology.orchestra.run.smithi184.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-19T07:15:57.294 INFO:teuthology.orchestra.run.smithi184.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-19T07:15:57.295 INFO:teuthology.orchestra.run.smithi184.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-19T07:15:57.298 INFO:teuthology.orchestra.run.smithi083.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-19T07:15:57.310 INFO:teuthology.orchestra.run.smithi083.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-19T07:15:57.311 INFO:teuthology.orchestra.run.smithi184.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-19T07:15:57.312 INFO:teuthology.orchestra.run.smithi184.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-19T07:15:57.314 INFO:teuthology.orchestra.run.smithi184.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-19T07:15:57.315 INFO:teuthology.orchestra.run.smithi184.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-19T07:15:57.315 INFO:teuthology.orchestra.run.smithi184.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-19T07:15:57.315 INFO:teuthology.orchestra.run.smithi184.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-19T07:15:57.317 INFO:teuthology.orchestra.run.smithi184.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-19T07:15:57.317 INFO:teuthology.orchestra.run.smithi184.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-19T07:15:57.328 INFO:teuthology.orchestra.run.smithi184.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-19T07:15:57.329 INFO:teuthology.orchestra.run.smithi184.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-19T07:15:57.345 INFO:teuthology.orchestra.run.smithi184.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-19T07:15:57.347 INFO:teuthology.orchestra.run.smithi184.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-19T07:15:57.348 INFO:teuthology.orchestra.run.smithi184.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-19T07:15:57.360 INFO:teuthology.orchestra.run.smithi083.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-19T07:15:57.534 INFO:teuthology.orchestra.run.smithi083.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-19T07:15:57.543 INFO:teuthology.orchestra.run.smithi083.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-19T07:15:57.570 INFO:teuthology.orchestra.run.smithi083.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-19T07:15:57.570 INFO:teuthology.orchestra.run.smithi184.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-361-g2f599a46-1jammy [3,241 kB] 2024-09-19T07:15:57.577 INFO:teuthology.orchestra.run.smithi083.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-19T07:15:57.579 INFO:teuthology.orchestra.run.smithi083.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-19T07:15:57.598 INFO:teuthology.orchestra.run.smithi083.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-19T07:15:57.602 INFO:teuthology.orchestra.run.smithi083.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-19T07:15:57.607 INFO:teuthology.orchestra.run.smithi083.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-19T07:15:57.609 INFO:teuthology.orchestra.run.smithi083.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-19T07:15:57.617 INFO:teuthology.orchestra.run.smithi083.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-19T07:15:57.692 INFO:teuthology.orchestra.run.smithi083.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-19T07:15:57.696 INFO:teuthology.orchestra.run.smithi083.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-19T07:15:57.700 INFO:teuthology.orchestra.run.smithi083.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-19T07:15:57.701 INFO:teuthology.orchestra.run.smithi083.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-19T07:15:57.702 INFO:teuthology.orchestra.run.smithi083.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-19T07:15:57.704 INFO:teuthology.orchestra.run.smithi083.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-19T07:15:57.705 INFO:teuthology.orchestra.run.smithi083.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-19T07:15:57.705 INFO:teuthology.orchestra.run.smithi083.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-19T07:15:57.706 INFO:teuthology.orchestra.run.smithi083.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-19T07:15:57.780 INFO:teuthology.orchestra.run.smithi083.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-19T07:15:57.839 INFO:teuthology.orchestra.run.smithi083.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-19T07:15:57.839 INFO:teuthology.orchestra.run.smithi083.stdout:Get:41 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-361-g2f599a46-1jammy [5,145 kB] 2024-09-19T07:15:57.839 INFO:teuthology.orchestra.run.smithi083.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-19T07:15:57.843 INFO:teuthology.orchestra.run.smithi184.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-361-g2f599a46-1jammy [719 kB] 2024-09-19T07:15:57.844 INFO:teuthology.orchestra.run.smithi083.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-19T07:15:57.845 INFO:teuthology.orchestra.run.smithi083.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-19T07:15:57.846 INFO:teuthology.orchestra.run.smithi083.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-19T07:15:57.847 INFO:teuthology.orchestra.run.smithi083.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-19T07:15:57.849 INFO:teuthology.orchestra.run.smithi083.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-19T07:15:57.849 INFO:teuthology.orchestra.run.smithi083.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-19T07:15:57.861 INFO:teuthology.orchestra.run.smithi083.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-19T07:15:57.873 INFO:teuthology.orchestra.run.smithi184.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-361-g2f599a46-1jammy [357 kB] 2024-09-19T07:15:57.882 INFO:teuthology.orchestra.run.smithi184.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-361-g2f599a46-1jammy [32.8 kB] 2024-09-19T07:15:57.884 INFO:teuthology.orchestra.run.smithi184.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-361-g2f599a46-1jammy [180 kB] 2024-09-19T07:15:57.898 INFO:teuthology.orchestra.run.smithi184.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-361-g2f599a46-1jammy [67.8 kB] 2024-09-19T07:15:57.908 INFO:teuthology.orchestra.run.smithi083.stdout:Get:50 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-361-g2f599a46-1jammy [245 kB] 2024-09-19T07:15:57.909 INFO:teuthology.orchestra.run.smithi184.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-361-g2f599a46-1jammy [333 kB] 2024-09-19T07:15:57.911 INFO:teuthology.orchestra.run.smithi083.stdout:Get:51 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-361-g2f599a46-1jammy [124 kB] 2024-09-19T07:15:57.912 INFO:teuthology.orchestra.run.smithi083.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-361-g2f599a46-1jammy [1,555 kB] 2024-09-19T07:15:57.923 INFO:teuthology.orchestra.run.smithi184.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-361-g2f599a46-1jammy [6,875 kB] 2024-09-19T07:15:57.932 INFO:teuthology.orchestra.run.smithi083.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-19T07:15:57.932 INFO:teuthology.orchestra.run.smithi083.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-361-g2f599a46-1jammy [6,142 kB] 2024-09-19T07:15:57.933 INFO:teuthology.orchestra.run.smithi083.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-19T07:15:57.934 INFO:teuthology.orchestra.run.smithi083.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-19T07:15:57.946 INFO:teuthology.orchestra.run.smithi083.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-19T07:15:57.947 INFO:teuthology.orchestra.run.smithi083.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-19T07:15:57.948 INFO:teuthology.orchestra.run.smithi083.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-19T07:15:57.949 INFO:teuthology.orchestra.run.smithi083.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-19T07:15:57.952 INFO:teuthology.orchestra.run.smithi083.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-19T07:15:57.956 INFO:teuthology.orchestra.run.smithi083.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-19T07:15:58.024 INFO:teuthology.orchestra.run.smithi083.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-19T07:15:58.027 INFO:teuthology.orchestra.run.smithi083.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-19T07:15:58.029 INFO:teuthology.orchestra.run.smithi083.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-19T07:15:58.075 INFO:teuthology.orchestra.run.smithi083.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-361-g2f599a46-1jammy [22.8 MB] 2024-09-19T07:15:58.106 INFO:teuthology.orchestra.run.smithi083.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-19T07:15:58.107 INFO:teuthology.orchestra.run.smithi083.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-19T07:15:58.109 INFO:teuthology.orchestra.run.smithi083.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-19T07:15:58.114 INFO:teuthology.orchestra.run.smithi083.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-19T07:15:58.280 INFO:teuthology.orchestra.run.smithi083.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-19T07:15:58.283 INFO:teuthology.orchestra.run.smithi083.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-19T07:15:58.286 INFO:teuthology.orchestra.run.smithi083.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-19T07:15:58.287 INFO:teuthology.orchestra.run.smithi083.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-19T07:15:58.290 INFO:teuthology.orchestra.run.smithi083.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-19T07:15:58.326 INFO:teuthology.orchestra.run.smithi083.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-19T07:15:58.353 INFO:teuthology.orchestra.run.smithi083.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-19T07:15:58.353 INFO:teuthology.orchestra.run.smithi083.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-19T07:15:58.386 INFO:teuthology.orchestra.run.smithi083.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-19T07:15:58.386 INFO:teuthology.orchestra.run.smithi083.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-19T07:15:58.449 INFO:teuthology.orchestra.run.smithi083.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-19T07:15:58.451 INFO:teuthology.orchestra.run.smithi083.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-19T07:15:58.451 INFO:teuthology.orchestra.run.smithi083.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-19T07:15:58.454 INFO:teuthology.orchestra.run.smithi083.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-19T07:15:58.454 INFO:teuthology.orchestra.run.smithi083.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-19T07:15:58.455 INFO:teuthology.orchestra.run.smithi083.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-19T07:15:58.465 INFO:teuthology.orchestra.run.smithi083.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-19T07:15:58.525 INFO:teuthology.orchestra.run.smithi184.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-361-g2f599a46-1jammy [112 kB] 2024-09-19T07:15:58.530 INFO:teuthology.orchestra.run.smithi083.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-19T07:15:58.533 INFO:teuthology.orchestra.run.smithi083.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-19T07:15:58.535 INFO:teuthology.orchestra.run.smithi083.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-19T07:15:58.538 INFO:teuthology.orchestra.run.smithi184.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-361-g2f599a46-1jammy [472 kB] 2024-09-19T07:15:58.567 INFO:teuthology.orchestra.run.smithi184.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-361-g2f599a46-1jammy [25.9 MB] 2024-09-19T07:15:58.611 INFO:teuthology.orchestra.run.smithi083.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-19T07:15:58.686 INFO:teuthology.orchestra.run.smithi083.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-09-19T07:15:58.687 INFO:teuthology.orchestra.run.smithi083.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-19T07:15:58.687 INFO:teuthology.orchestra.run.smithi083.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-361-g2f599a46-1jammy [14.1 kB] 2024-09-19T07:15:58.687 INFO:teuthology.orchestra.run.smithi083.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-361-g2f599a46-1jammy [877 kB] 2024-09-19T07:15:58.689 INFO:teuthology.orchestra.run.smithi083.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-19T07:15:58.690 INFO:teuthology.orchestra.run.smithi083.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-19T07:15:58.694 INFO:teuthology.orchestra.run.smithi083.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-361-g2f599a46-1jammy [2,462 kB] 2024-09-19T07:15:58.695 INFO:teuthology.orchestra.run.smithi083.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-19T07:15:58.736 INFO:teuthology.orchestra.run.smithi083.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-361-g2f599a46-1jammy [794 kB] 2024-09-19T07:15:58.736 INFO:teuthology.orchestra.run.smithi083.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-19T07:15:58.739 INFO:teuthology.orchestra.run.smithi083.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-19T07:15:58.739 INFO:teuthology.orchestra.run.smithi083.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-19T07:15:58.740 INFO:teuthology.orchestra.run.smithi083.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-19T07:15:58.740 INFO:teuthology.orchestra.run.smithi083.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-361-g2f599a46-1jammy [152 kB] 2024-09-19T07:15:58.741 INFO:teuthology.orchestra.run.smithi083.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-361-g2f599a46-1jammy [2,347 kB] 2024-09-19T07:15:58.775 INFO:teuthology.orchestra.run.smithi083.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-19T07:15:58.779 INFO:teuthology.orchestra.run.smithi083.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-19T07:15:58.781 INFO:teuthology.orchestra.run.smithi083.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-19T07:15:58.793 INFO:teuthology.orchestra.run.smithi083.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-361-g2f599a46-1jammy [8,625 kB] 2024-09-19T07:15:58.801 INFO:teuthology.orchestra.run.smithi083.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-19T07:15:58.856 INFO:teuthology.orchestra.run.smithi083.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-19T07:15:58.856 INFO:teuthology.orchestra.run.smithi083.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-19T07:15:58.863 INFO:teuthology.orchestra.run.smithi083.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-19T07:15:58.866 INFO:teuthology.orchestra.run.smithi083.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-19T07:15:59.084 INFO:teuthology.orchestra.run.smithi083.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-361-g2f599a46-1jammy [14.2 kB] 2024-09-19T07:15:59.085 INFO:teuthology.orchestra.run.smithi083.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-361-g2f599a46-1jammy [51.4 MB] 2024-09-19T07:16:00.022 INFO:teuthology.orchestra.run.smithi184.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-361-g2f599a46-1jammy [5,145 kB] 2024-09-19T07:16:00.221 INFO:teuthology.orchestra.run.smithi184.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-361-g2f599a46-1jammy [245 kB] 2024-09-19T07:16:00.244 INFO:teuthology.orchestra.run.smithi184.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-361-g2f599a46-1jammy [124 kB] 2024-09-19T07:16:00.247 INFO:teuthology.orchestra.run.smithi184.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-361-g2f599a46-1jammy [1,555 kB] 2024-09-19T07:16:00.336 INFO:teuthology.orchestra.run.smithi184.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-361-g2f599a46-1jammy [6,142 kB] 2024-09-19T07:16:00.681 INFO:teuthology.orchestra.run.smithi184.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-361-g2f599a46-1jammy [22.8 MB] 2024-09-19T07:16:00.796 INFO:teuthology.orchestra.run.smithi083.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-361-g2f599a46-1jammy [130 kB] 2024-09-19T07:16:00.799 INFO:teuthology.orchestra.run.smithi083.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.1.1-361-g2f599a46-1jammy [236 kB] 2024-09-19T07:16:00.803 INFO:teuthology.orchestra.run.smithi083.stdout:Get:120 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.1.1-361-g2f599a46-1jammy [34.0 kB] 2024-09-19T07:16:00.804 INFO:teuthology.orchestra.run.smithi083.stdout:Get:121 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.1.1-361-g2f599a46-1jammy [23.5 kB] 2024-09-19T07:16:00.805 INFO:teuthology.orchestra.run.smithi083.stdout:Get:122 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-361-g2f599a46-1jammy [39.0 kB] 2024-09-19T07:16:00.806 INFO:teuthology.orchestra.run.smithi083.stdout:Get:123 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-361-g2f599a46-1jammy [13.6 MB] 2024-09-19T07:16:01.086 INFO:teuthology.orchestra.run.smithi083.stdout:Get:124 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-361-g2f599a46-1jammy [91.5 kB] 2024-09-19T07:16:01.418 INFO:teuthology.orchestra.run.smithi184.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-361-g2f599a46-1jammy [14.1 kB] 2024-09-19T07:16:01.418 INFO:teuthology.orchestra.run.smithi184.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-361-g2f599a46-1jammy [877 kB] 2024-09-19T07:16:01.433 INFO:teuthology.orchestra.run.smithi184.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-361-g2f599a46-1jammy [2,462 kB] 2024-09-19T07:16:01.471 INFO:teuthology.orchestra.run.smithi184.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-361-g2f599a46-1jammy [794 kB] 2024-09-19T07:16:01.480 INFO:teuthology.orchestra.run.smithi184.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-361-g2f599a46-1jammy [152 kB] 2024-09-19T07:16:01.482 INFO:teuthology.orchestra.run.smithi184.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-361-g2f599a46-1jammy [2,347 kB] 2024-09-19T07:16:01.509 INFO:teuthology.orchestra.run.smithi184.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-361-g2f599a46-1jammy [8,625 kB] 2024-09-19T07:16:01.679 INFO:teuthology.orchestra.run.smithi184.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-361-g2f599a46-1jammy [14.2 kB] 2024-09-19T07:16:01.680 INFO:teuthology.orchestra.run.smithi184.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-361-g2f599a46-1jammy [51.4 MB] 2024-09-19T07:16:01.715 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 181 MB in 4s (41.4 MB/s) 2024-09-19T07:16:01.871 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pyperclip. 2024-09-19T07:16:01.928 INFO:teuthology.orchestra.run.smithi083.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-19T07:16:01.932 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-09-19T07:16:01.969 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-09-19T07:16:02.272 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-19T07:16:02.298 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-19T07:16:02.339 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-19T07:16:02.650 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cmd2. 2024-09-19T07:16:02.676 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-09-19T07:16:02.717 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-09-19T07:16:02.876 INFO:teuthology.orchestra.run.smithi184.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-361-g2f599a46-1jammy [130 kB] 2024-09-19T07:16:02.878 INFO:teuthology.orchestra.run.smithi184.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.1.1-361-g2f599a46-1jammy [236 kB] 2024-09-19T07:16:02.883 INFO:teuthology.orchestra.run.smithi184.stdout:Get:120 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.1.1-361-g2f599a46-1jammy [34.0 kB] 2024-09-19T07:16:02.884 INFO:teuthology.orchestra.run.smithi184.stdout:Get:121 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.1.1-361-g2f599a46-1jammy [23.5 kB] 2024-09-19T07:16:02.885 INFO:teuthology.orchestra.run.smithi184.stdout:Get:122 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-361-g2f599a46-1jammy [39.0 kB] 2024-09-19T07:16:02.886 INFO:teuthology.orchestra.run.smithi184.stdout:Get:123 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-361-g2f599a46-1jammy [13.6 MB] 2024-09-19T07:16:03.045 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-19T07:16:03.071 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-19T07:16:03.093 INFO:teuthology.orchestra.run.smithi184.stdout:Get:124 https://4.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-361-g2f599a46-1jammy [91.5 kB] 2024-09-19T07:16:03.137 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-19T07:16:03.482 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-19T07:16:03.509 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-19T07:16:03.549 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-19T07:16:03.722 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 181 MB in 6s (28.8 MB/s) 2024-09-19T07:16:03.868 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyperclip. 2024-09-19T07:16:03.885 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-19T07:16:03.911 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-19T07:16:03.952 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-19T07:16:03.968 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-19T07:16:03.973 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-09-19T07:16:04.016 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-09-19T07:16:04.347 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-19T07:16:04.374 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-19T07:16:04.505 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-19T07:16:04.536 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-19T07:16:04.564 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-19T07:16:04.603 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-19T07:16:04.914 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cmd2. 2024-09-19T07:16:04.917 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-19T07:16:04.940 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-09-19T07:16:04.943 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-19T07:16:04.981 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-09-19T07:16:04.983 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-19T07:16:05.309 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-19T07:16:05.320 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-19T07:16:05.335 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-19T07:16:05.346 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-19T07:16:05.389 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-19T07:16:05.417 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-19T07:16:05.757 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-19T07:16:05.762 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-19T07:16:05.783 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-19T07:16:05.789 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-19T07:16:05.823 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-19T07:16:05.829 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-19T07:16:06.124 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-19T07:16:06.143 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package librados2. 2024-09-19T07:16:06.150 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-19T07:16:06.170 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../010-librados2_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:06.190 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-19T07:16:06.210 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librados2 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:06.638 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package librbd1. 2024-09-19T07:16:06.652 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-19T07:16:06.664 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../011-librbd1_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:06.679 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-19T07:16:06.696 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librbd1 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:06.727 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-19T07:16:07.097 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-19T07:16:07.113 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-19T07:16:07.132 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libcephfs2. 2024-09-19T07:16:07.147 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-19T07:16:07.159 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../012-libcephfs2_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:07.199 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libcephfs2 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:07.475 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-19T07:16:07.493 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-rados. 2024-09-19T07:16:07.502 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-19T07:16:07.519 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../013-python3-rados_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:07.550 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-19T07:16:07.560 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-rados (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:07.863 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-19T07:16:07.889 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:07.895 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-19T07:16:07.921 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-19T07:16:07.929 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-ceph-argparse (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:07.962 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-19T07:16:08.157 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cephfs. 2024-09-19T07:16:08.173 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../015-python3-cephfs_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:08.215 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cephfs (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:08.257 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librados2. 2024-09-19T07:16:08.283 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../010-librados2_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:08.323 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librados2 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:08.535 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-19T07:16:08.562 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../016-python3-ceph-common_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:08.601 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-ceph-common (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:08.759 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librbd1. 2024-09-19T07:16:08.786 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../011-librbd1_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:08.826 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librbd1 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:08.938 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-prettytable. 2024-09-19T07:16:08.964 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-09-19T07:16:09.004 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-19T07:16:09.266 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-rbd. 2024-09-19T07:16:09.270 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libcephfs2. 2024-09-19T07:16:09.292 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../018-python3-rbd_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:09.297 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../012-libcephfs2_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:09.332 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-rbd (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:09.629 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libcephfs2 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:09.680 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-19T07:16:09.703 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-19T07:16:09.743 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-19T07:16:09.915 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rados. 2024-09-19T07:16:09.941 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../013-python3-rados_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:09.981 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rados (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:10.013 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-19T07:16:10.028 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-19T07:16:10.071 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-19T07:16:10.309 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-19T07:16:10.336 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:10.376 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-ceph-argparse (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:10.382 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-19T07:16:10.398 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-19T07:16:10.441 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-19T07:16:10.628 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cephfs. 2024-09-19T07:16:10.644 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../015-python3-cephfs_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:10.687 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cephfs (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:10.778 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package lua5.1. 2024-09-19T07:16:10.804 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-19T07:16:10.844 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-19T07:16:10.981 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-19T07:16:10.996 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../016-python3-ceph-common_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:11.031 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-ceph-common (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:11.197 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package lua-any. 2024-09-19T07:16:11.213 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-09-19T07:16:11.247 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-19T07:16:11.326 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-prettytable. 2024-09-19T07:16:11.352 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-09-19T07:16:11.392 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-19T07:16:11.500 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package zip. 2024-09-19T07:16:11.527 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-09-19T07:16:11.567 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking zip (3.0-12build2) ... 2024-09-19T07:16:11.637 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rbd. 2024-09-19T07:16:11.652 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../018-python3-rbd_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:11.695 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rbd (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:11.953 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package unzip. 2024-09-19T07:16:11.979 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-19T07:16:12.028 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-19T07:16:12.040 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-19T07:16:12.066 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-19T07:16:12.106 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-19T07:16:12.384 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-19T07:16:12.410 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-19T07:16:12.451 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-19T07:16:12.499 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package luarocks. 2024-09-19T07:16:12.525 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-19T07:16:12.566 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-19T07:16:12.729 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-19T07:16:12.755 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-19T07:16:12.787 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-19T07:16:12.982 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package librgw2. 2024-09-19T07:16:13.009 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../027-librgw2_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:13.058 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librgw2 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:13.107 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package lua5.1. 2024-09-19T07:16:13.133 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-19T07:16:13.173 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-19T07:16:13.485 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package lua-any. 2024-09-19T07:16:13.500 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-09-19T07:16:13.543 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-19T07:16:13.618 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-rgw. 2024-09-19T07:16:13.644 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../028-python3-rgw_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:13.693 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-rgw (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:13.779 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package zip. 2024-09-19T07:16:13.794 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-09-19T07:16:13.829 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking zip (3.0-12build2) ... 2024-09-19T07:16:14.091 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package unzip. 2024-09-19T07:16:14.104 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-19T07:16:14.117 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-19T07:16:14.130 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-19T07:16:14.182 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-19T07:16:14.405 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-19T07:16:14.603 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package luarocks. 2024-09-19T07:16:14.630 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-19T07:16:14.670 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-19T07:16:14.749 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libradosstriper1. 2024-09-19T07:16:14.775 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../030-libradosstriper1_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:14.824 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libradosstriper1 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:15.053 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librgw2. 2024-09-19T07:16:15.076 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../027-librgw2_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:15.119 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librgw2 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:15.245 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-common. 2024-09-19T07:16:15.271 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../031-ceph-common_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:15.311 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-common (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:15.655 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rgw. 2024-09-19T07:16:15.681 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../028-python3-rgw_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:15.721 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rgw (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:16.008 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-19T07:16:16.035 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-19T07:16:16.066 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-19T07:16:16.386 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libradosstriper1. 2024-09-19T07:16:16.413 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../030-libradosstriper1_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:16.453 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libradosstriper1 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:16.700 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-base. 2024-09-19T07:16:16.727 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../032-ceph-base_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:16.776 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-base (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:16.881 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-common. 2024-09-19T07:16:16.907 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../031-ceph-common_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:16.947 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-common (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:17.338 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-19T07:16:17.364 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-19T07:16:17.404 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-19T07:16:17.674 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cheroot. 2024-09-19T07:16:17.701 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-19T07:16:17.741 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-19T07:16:18.069 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-19T07:16:18.095 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-19T07:16:18.135 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-19T07:16:18.261 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-base. 2024-09-19T07:16:18.287 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../032-ceph-base_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:18.337 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-base (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:18.438 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-19T07:16:18.465 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-19T07:16:18.505 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-19T07:16:18.775 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-19T07:16:18.790 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-19T07:16:18.824 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-19T07:16:18.899 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-19T07:16:18.926 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-19T07:16:18.965 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-19T07:16:19.127 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-tempora. 2024-09-19T07:16:19.154 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-09-19T07:16:19.347 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-19T07:16:19.348 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cheroot. 2024-09-19T07:16:19.348 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-19T07:16:19.348 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-19T07:16:19.522 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-portend. 2024-09-19T07:16:19.549 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-09-19T07:16:19.590 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-19T07:16:19.638 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-19T07:16:19.665 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-19T07:16:19.705 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-19T07:16:19.901 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-19T07:16:19.926 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-19T07:16:19.967 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-19T07:16:20.216 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-19T07:16:20.236 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-19T07:16:20.243 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-19T07:16:20.262 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-19T07:16:20.282 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-19T07:16:20.294 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-19T07:16:20.585 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-19T07:16:20.598 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-natsort. 2024-09-19T07:16:20.611 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-19T07:16:20.624 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-09-19T07:16:20.652 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-19T07:16:20.664 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-19T07:16:20.959 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-logutils. 2024-09-19T07:16:20.971 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-tempora. 2024-09-19T07:16:20.986 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-09-19T07:16:20.997 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-09-19T07:16:21.026 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-19T07:16:21.038 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-19T07:16:21.282 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-portend. 2024-09-19T07:16:21.298 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-09-19T07:16:21.329 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-19T07:16:21.332 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-19T07:16:21.356 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-19T07:16:21.396 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-19T07:16:21.637 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-19T07:16:21.657 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-mako. 2024-09-19T07:16:21.664 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-19T07:16:21.683 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-19T07:16:21.704 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-19T07:16:21.715 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-19T07:16:21.998 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-19T07:16:22.025 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-19T07:16:22.035 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-19T07:16:22.056 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-19T07:16:22.062 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-19T07:16:22.102 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-19T07:16:22.393 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-natsort. 2024-09-19T07:16:22.405 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-19T07:16:22.420 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-09-19T07:16:22.431 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-19T07:16:22.460 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-19T07:16:22.471 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-19T07:16:22.766 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-webob. 2024-09-19T07:16:22.780 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-logutils. 2024-09-19T07:16:22.793 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-19T07:16:22.806 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-09-19T07:16:22.833 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-19T07:16:22.847 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-19T07:16:23.124 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-19T07:16:23.151 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-19T07:16:23.169 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-waitress. 2024-09-19T07:16:23.191 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-19T07:16:23.196 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-19T07:16:23.231 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-19T07:16:23.494 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-mako. 2024-09-19T07:16:23.505 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-tempita. 2024-09-19T07:16:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-19T07:16:23.532 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-19T07:16:23.560 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-19T07:16:23.572 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-19T07:16:23.825 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-paste. 2024-09-19T07:16:23.852 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-19T07:16:23.864 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-19T07:16:23.891 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-19T07:16:23.891 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-19T07:16:23.922 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-19T07:16:24.212 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-19T07:16:24.216 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-19T07:16:24.239 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-19T07:16:24.243 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-19T07:16:24.278 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-19T07:16:24.283 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-19T07:16:24.531 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-19T07:16:24.558 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-19T07:16:24.586 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-webob. 2024-09-19T07:16:24.597 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-19T07:16:24.613 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-19T07:16:24.652 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-19T07:16:24.910 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-webtest. 2024-09-19T07:16:24.936 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-09-19T07:16:24.976 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-19T07:16:25.039 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-waitress. 2024-09-19T07:16:25.066 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-19T07:16:25.109 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-19T07:16:25.296 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pecan. 2024-09-19T07:16:25.322 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-19T07:16:25.362 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-19T07:16:25.425 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-tempita. 2024-09-19T07:16:25.441 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-19T07:16:25.484 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-19T07:16:25.726 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-certifi. 2024-09-19T07:16:25.753 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-09-19T07:16:25.786 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-paste. 2024-09-19T07:16:25.792 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-19T07:16:25.810 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-19T07:16:25.853 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-19T07:16:26.087 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-idna. 2024-09-19T07:16:26.114 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../057-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-19T07:16:26.145 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-19T07:16:26.182 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-19T07:16:26.208 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-19T07:16:26.248 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-19T07:16:26.456 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-urllib3. 2024-09-19T07:16:26.483 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-19T07:16:26.523 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-19T07:16:26.551 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-19T07:16:26.578 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-19T07:16:26.617 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-19T07:16:26.877 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-requests. 2024-09-19T07:16:26.903 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-19T07:16:26.904 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-webtest. 2024-09-19T07:16:26.931 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-09-19T07:16:26.943 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-19T07:16:26.962 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-19T07:16:27.232 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pecan. 2024-09-19T07:16:27.237 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-19T07:16:27.258 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-19T07:16:27.264 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-19T07:16:27.298 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-19T07:16:27.304 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-19T07:16:27.599 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-19T07:16:27.626 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:27.665 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-modules-core (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:27.670 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-certifi. 2024-09-19T07:16:27.697 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-09-19T07:16:27.736 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-19T07:16:28.012 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-19T07:16:28.039 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:28.048 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-idna. 2024-09-19T07:16:28.070 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:28.075 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../057-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-19T07:16:28.114 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-19T07:16:28.390 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-19T07:16:28.400 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-urllib3. 2024-09-19T07:16:28.416 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-19T07:16:28.417 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-19T07:16:28.450 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-19T07:16:28.457 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-19T07:16:28.751 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-openssl. 2024-09-19T07:16:28.754 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-requests. 2024-09-19T07:16:28.778 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-09-19T07:16:28.780 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-19T07:16:28.818 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-19T07:16:28.828 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-19T07:16:29.098 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-19T07:16:29.113 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-19T07:16:29.156 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-19T07:16:29.426 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-19T07:16:29.449 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:29.484 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-modules-core (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:29.873 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-19T07:16:29.900 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:29.939 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:30.280 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr. 2024-09-19T07:16:30.308 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../065-ceph-mgr_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:30.346 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:30.725 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mon. 2024-09-19T07:16:30.751 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../066-ceph-mon_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:30.792 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mon (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:31.185 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-19T07:16:31.211 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-19T07:16:31.243 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-19T07:16:31.354 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-osd. 2024-09-19T07:16:31.380 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../067-ceph-osd_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:31.420 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-osd (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:31.571 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-openssl. 2024-09-19T07:16:31.598 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-09-19T07:16:31.637 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-19T07:16:31.957 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr. 2024-09-19T07:16:31.984 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../065-ceph-mgr_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:32.024 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:32.394 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mon. 2024-09-19T07:16:32.420 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../066-ceph-mon_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:32.460 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mon (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:32.473 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph. 2024-09-19T07:16:32.488 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../068-ceph_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:32.531 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:32.826 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-fuse. 2024-09-19T07:16:32.852 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../069-ceph-fuse_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:32.892 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-fuse (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:33.047 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-osd. 2024-09-19T07:16:33.074 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../067-ceph-osd_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:33.113 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-osd (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:33.237 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mds. 2024-09-19T07:16:33.264 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../070-ceph-mds_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:33.304 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mds (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:33.707 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package cephadm. 2024-09-19T07:16:33.733 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../071-cephadm_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:33.773 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking cephadm (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:34.110 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-19T07:16:34.136 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-19T07:16:34.176 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-19T07:16:34.200 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph. 2024-09-19T07:16:34.227 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../068-ceph_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:34.266 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:34.495 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-19T07:16:34.511 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-19T07:16:34.554 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-19T07:16:34.586 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-fuse. 2024-09-19T07:16:34.612 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../069-ceph-fuse_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:34.652 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-fuse (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:34.981 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mds. 2024-09-19T07:16:35.007 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../070-ceph-mds_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:35.047 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mds (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:35.442 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package cephadm. 2024-09-19T07:16:35.468 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../071-cephadm_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:35.508 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking cephadm (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:35.603 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-babel. 2024-09-19T07:16:35.630 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-19T07:16:35.669 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-19T07:16:35.844 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-19T07:16:35.871 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-19T07:16:35.911 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-19T07:16:35.989 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-jinja2. 2024-09-19T07:16:36.016 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-19T07:16:36.055 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-19T07:16:36.256 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-19T07:16:36.271 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-19T07:16:36.314 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-19T07:16:36.425 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-19T07:16:36.453 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:36.492 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-cephadm (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:36.837 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-19T07:16:36.865 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-09-19T07:16:36.904 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-19T07:16:37.174 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-routes. 2024-09-19T07:16:37.200 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-19T07:16:37.231 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-19T07:16:37.269 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-babel. 2024-09-19T07:16:37.296 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-19T07:16:37.335 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-19T07:16:37.576 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-19T07:16:37.603 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:37.643 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-dashboard (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:37.663 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jinja2. 2024-09-19T07:16:37.691 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-19T07:16:37.730 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-19T07:16:38.075 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-19T07:16:38.101 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:38.141 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-cephadm (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:38.470 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-19T07:16:38.496 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-09-19T07:16:38.536 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-19T07:16:38.847 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-routes. 2024-09-19T07:16:38.874 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-19T07:16:38.914 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-19T07:16:38.973 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-19T07:16:38.989 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-19T07:16:39.040 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-19T07:16:39.250 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-19T07:16:39.277 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:39.317 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-dashboard (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:39.611 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-joblib. 2024-09-19T07:16:39.638 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-19T07:16:39.685 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-19T07:16:40.134 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-19T07:16:40.144 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-19T07:16:40.475 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-19T07:16:40.506 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-19T07:16:40.522 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-19T07:16:40.564 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-19T07:16:40.753 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-sklearn. 2024-09-19T07:16:40.780 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-19T07:16:40.819 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-19T07:16:41.275 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-joblib. 2024-09-19T07:16:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-19T07:16:41.342 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-19T07:16:41.496 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-19T07:16:41.523 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:41.562 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:41.679 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-19T07:16:41.706 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-19T07:16:41.822 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-19T07:16:42.125 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-sklearn. 2024-09-19T07:16:42.153 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-19T07:16:42.192 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-19T07:16:42.532 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cachetools. 2024-09-19T07:16:42.559 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-09-19T07:16:42.598 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-19T07:16:42.876 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-19T07:16:42.901 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-19T07:16:42.903 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-19T07:16:42.929 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:42.942 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-19T07:16:42.967 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:43.213 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-19T07:16:43.229 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-19T07:16:43.264 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-19T07:16:43.507 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-rsa. 2024-09-19T07:16:43.523 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-09-19T07:16:43.557 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-19T07:16:43.870 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-google-auth. 2024-09-19T07:16:43.897 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-09-19T07:16:43.936 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-19T07:16:43.946 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cachetools. 2024-09-19T07:16:43.973 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-09-19T07:16:44.012 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-19T07:16:44.290 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-19T07:16:44.298 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-19T07:16:44.318 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-19T07:16:44.325 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-19T07:16:44.357 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-19T07:16:44.373 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-19T07:16:44.685 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-19T07:16:44.693 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-websocket. 2024-09-19T07:16:44.712 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-19T07:16:44.720 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-09-19T07:16:44.751 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-19T07:16:44.759 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-19T07:16:45.022 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rsa. 2024-09-19T07:16:45.048 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-09-19T07:16:45.080 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-19T07:16:45.096 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-19T07:16:45.124 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-19T07:16:45.163 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-19T07:16:45.392 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-google-auth. 2024-09-19T07:16:45.420 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-09-19T07:16:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-19T07:16:45.787 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-19T07:16:45.815 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-19T07:16:45.853 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-19T07:16:46.052 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-19T07:16:46.080 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:46.118 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:46.156 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-websocket. 2024-09-19T07:16:46.184 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-09-19T07:16:46.223 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-19T07:16:46.488 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-19T07:16:46.516 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-19T07:16:46.535 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-19T07:16:46.562 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-19T07:16:46.563 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-19T07:16:46.601 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-19T07:16:46.925 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-19T07:16:46.952 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-19T07:16:46.991 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-19T07:16:47.252 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package jq. 2024-09-19T07:16:47.268 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-19T07:16:47.299 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-19T07:16:47.311 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-19T07:16:47.315 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:47.357 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:47.572 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package socat. 2024-09-19T07:16:47.599 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-19T07:16:47.638 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-19T07:16:47.701 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-19T07:16:47.729 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-19T07:16:47.769 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-19T07:16:47.950 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package xmlstarlet. 2024-09-19T07:16:47.978 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-19T07:16:48.025 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-19T07:16:48.071 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-19T07:16:48.099 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-19T07:16:48.137 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-19T07:16:48.374 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package jq. 2024-09-19T07:16:48.401 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-19T07:16:48.413 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-test. 2024-09-19T07:16:48.429 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../099-ceph-test_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:48.440 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-19T07:16:48.471 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-test (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:48.701 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package socat. 2024-09-19T07:16:48.729 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-19T07:16:48.768 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-19T07:16:49.046 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package xmlstarlet. 2024-09-19T07:16:49.062 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-19T07:16:49.112 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-19T07:16:49.476 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-test. 2024-09-19T07:16:49.503 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../099-ceph-test_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:49.542 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-test (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:50.809 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-volume. 2024-09-19T07:16:50.837 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../100-ceph-volume_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:50.875 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-volume (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:51.255 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package cephfs-mirror. 2024-09-19T07:16:51.283 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../101-cephfs-mirror_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:51.321 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking cephfs-mirror (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:51.633 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-colorama. 2024-09-19T07:16:51.661 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-09-19T07:16:51.699 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-09-19T07:16:51.960 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package cephfs-shell. 2024-09-19T07:16:51.976 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../103-cephfs-shell_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:52.010 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking cephfs-shell (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:52.088 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-volume. 2024-09-19T07:16:52.116 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../100-ceph-volume_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:52.154 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-volume (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:52.263 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package cephfs-top. 2024-09-19T07:16:52.291 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../104-cephfs-top_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:52.330 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking cephfs-top (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:52.492 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package cephfs-mirror. 2024-09-19T07:16:52.520 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../101-cephfs-mirror_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:52.550 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking cephfs-mirror (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:52.582 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-19T07:16:52.598 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../105-libcephfs-dev_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:52.640 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libcephfs-dev (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:52.820 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-colorama. 2024-09-19T07:16:52.847 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-09-19T07:16:52.886 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-09-19T07:16:52.960 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-19T07:16:52.988 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-19T07:16:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-19T07:16:53.156 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package cephfs-shell. 2024-09-19T07:16:53.184 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../103-cephfs-shell_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:53.222 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking cephfs-shell (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:53.331 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-19T07:16:53.347 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-09-19T07:16:53.381 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-19T07:16:53.483 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package cephfs-top. 2024-09-19T07:16:53.511 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../104-cephfs-top_19.1.1-361-g2f599a46-1jammy_all.deb ... 2024-09-19T07:16:53.550 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking cephfs-top (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:53.743 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package nvme-cli. 2024-09-19T07:16:53.768 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-19T07:16:53.771 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-19T07:16:53.785 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../105-libcephfs-dev_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:53.810 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-19T07:16:53.827 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libcephfs-dev (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:54.130 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-19T07:16:54.158 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-19T07:16:54.196 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-19T07:16:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package pkg-config. 2024-09-19T07:16:54.260 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-19T07:16:54.298 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-19T07:16:54.526 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-19T07:16:54.554 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-09-19T07:16:54.560 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-19T07:16:54.587 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-19T07:16:54.592 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-19T07:16:54.643 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-19T07:16:54.980 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package nvme-cli. 2024-09-19T07:16:55.008 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-19T07:16:55.046 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-19T07:16:55.071 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-19T07:16:55.099 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-19T07:16:55.138 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-19T07:16:55.432 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pastescript. 2024-09-19T07:16:55.435 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package pkg-config. 2024-09-19T07:16:55.457 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-09-19T07:16:55.463 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-19T07:16:55.490 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-19T07:16:55.501 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-19T07:16:55.769 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pluggy. 2024-09-19T07:16:55.796 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-19T07:16:55.830 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-19T07:16:55.835 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-19T07:16:55.857 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-19T07:16:55.896 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-19T07:16:56.349 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-19T07:16:56.355 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-psutil. 2024-09-19T07:16:56.377 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-19T07:16:56.382 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-19T07:16:56.415 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-19T07:16:56.421 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-19T07:16:56.718 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pastescript. 2024-09-19T07:16:56.749 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-09-19T07:16:56.775 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-py. 2024-09-19T07:16:56.785 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-19T07:16:56.802 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-09-19T07:16:56.849 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-19T07:16:57.114 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pluggy. 2024-09-19T07:16:57.141 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-19T07:16:57.169 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pygments. 2024-09-19T07:16:57.180 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-19T07:16:57.185 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-19T07:16:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-19T07:16:57.483 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-psutil. 2024-09-19T07:16:57.510 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-19T07:16:57.541 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-19T07:16:57.691 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-19T07:16:57.718 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-19T07:16:57.757 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-19T07:16:57.861 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-py. 2024-09-19T07:16:57.888 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-09-19T07:16:57.927 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-19T07:16:58.026 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-toml. 2024-09-19T07:16:58.042 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-09-19T07:16:58.076 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-19T07:16:58.258 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pygments. 2024-09-19T07:16:58.285 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-19T07:16:58.324 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-19T07:16:58.355 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pytest. 2024-09-19T07:16:58.382 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-19T07:16:58.421 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-19T07:16:58.742 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-simplejson. 2024-09-19T07:16:58.769 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-19T07:16:58.795 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-19T07:16:58.800 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-19T07:16:58.823 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-19T07:16:58.861 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-19T07:16:59.061 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-19T07:16:59.077 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-19T07:16:59.110 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-19T07:16:59.172 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-toml. 2024-09-19T07:16:59.200 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-09-19T07:16:59.239 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-19T07:16:59.517 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pytest. 2024-09-19T07:16:59.533 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-19T07:16:59.567 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-19T07:16:59.815 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package radosgw. 2024-09-19T07:16:59.843 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../122-radosgw_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:16:59.881 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking radosgw (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:16:59.937 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-simplejson. 2024-09-19T07:16:59.965 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-19T07:17:00.003 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-19T07:17:00.265 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-19T07:17:00.292 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-19T07:17:00.331 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-19T07:17:00.626 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package rbd-fuse. 2024-09-19T07:17:00.643 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../123-rbd-fuse_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:17:00.676 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking rbd-fuse (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:00.975 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-19T07:17:01.102 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package radosgw. 2024-09-19T07:17:01.114 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../122-radosgw_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:17:01.143 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking radosgw (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:01.223 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-19T07:17:01.374 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-19T07:17:01.683 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-19T07:17:01.955 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package rbd-fuse. 2024-09-19T07:17:01.971 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../123-rbd-fuse_19.1.1-361-g2f599a46-1jammy_amd64.deb ... 2024-09-19T07:17:01.991 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-19T07:17:02.013 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking rbd-fuse (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:02.292 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-19T07:17:02.388 INFO:teuthology.orchestra.run.smithi083.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-19T07:17:02.460 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up cephadm (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:02.535 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-19T07:17:02.661 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-19T07:17:02.918 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-19T07:17:03.018 INFO:teuthology.orchestra.run.smithi083.stdout:Adding system user cephadm....done 2024-09-19T07:17:03.174 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-19T07:17:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-19T07:17:03.442 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-19T07:17:03.673 INFO:teuthology.orchestra.run.smithi184.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-19T07:17:03.694 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-19T07:17:03.738 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up cephadm (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:03.828 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-19T07:17:04.079 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-19T07:17:04.280 INFO:teuthology.orchestra.run.smithi184.stdout:Adding system user cephadm....done 2024-09-19T07:17:04.339 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-19T07:17:04.436 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-19T07:17:04.465 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-19T07:17:04.713 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-19T07:17:04.767 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-19T07:17:04.972 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-19T07:17:05.107 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-19T07:17:05.144 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-19T07:17:05.358 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-19T07:17:05.395 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-09-19T07:17:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-19T07:17:05.655 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-19T07:17:05.743 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-19T07:17:05.940 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-19T07:17:06.053 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-19T07:17:06.233 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-19T07:17:06.414 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-19T07:17:06.501 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-19T07:17:06.743 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-19T07:17:06.799 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-09-19T07:17:07.003 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-ceph-argparse (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:07.263 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-19T07:17:07.291 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-19T07:17:07.389 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-19T07:17:07.515 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-09-19T07:17:07.551 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-19T07:17:07.783 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-19T07:17:07.813 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-19T07:17:07.909 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-19T07:17:08.070 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-19T07:17:08.177 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-19T07:17:08.330 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-19T07:17:08.557 INFO:teuthology.orchestra.run.smithi083.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-19T07:17:08.581 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-ceph-argparse (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:08.592 INFO:teuthology.orchestra.run.smithi083.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-19T07:17:08.655 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-19T07:17:08.781 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-19T07:17:08.840 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-19T07:17:08.966 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-19T07:17:09.067 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-09-19T07:17:09.141 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-19T07:17:09.258 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-19T07:17:09.310 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-19T07:17:09.436 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-19T07:17:09.526 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-19T07:17:09.687 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-19T07:17:09.811 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-19T07:17:10.076 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-19T07:17:10.110 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-19T07:17:10.112 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up zip (3.0-12build2) ... 2024-09-19T07:17:10.190 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-19T07:17:10.238 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-19T07:17:10.316 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-19T07:17:10.668 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-19T07:17:10.802 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-19T07:17:10.949 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-19T07:17:11.061 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-19T07:17:11.209 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-19T07:17:11.338 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-19T07:17:11.351 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-19T07:17:11.502 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-19T07:17:11.647 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up zip (3.0-12build2) ... 2024-09-19T07:17:11.765 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-19T07:17:11.820 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-19T07:17:12.213 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-19T07:17:12.425 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-19T07:17:12.472 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-19T07:17:12.668 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-19T07:17:12.769 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-19T07:17:12.857 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-19T07:17:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-19T07:17:13.163 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-19T07:17:13.167 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-09-19T07:17:13.460 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-19T07:17:13.547 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-19T07:17:13.807 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-19T07:17:13.828 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-19T07:17:14.088 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-19T07:17:14.175 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-19T07:17:14.214 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-ceph-common (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:14.443 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-09-19T07:17:14.515 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-19T07:17:14.728 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-19T07:17:14.858 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-19T07:17:15.062 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-19T07:17:15.321 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-19T07:17:15.447 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-ceph-common (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:15.757 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-19T07:17:16.083 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-19T07:17:16.106 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-19T07:17:16.358 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-19T07:17:16.501 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-19T07:17:16.777 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-19T07:17:17.298 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-19T07:17:17.303 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-19T07:17:17.437 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-19T07:17:17.574 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-19T07:17:17.709 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-19T07:17:17.747 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-19T07:17:17.960 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-19T07:17:17.998 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-19T07:17:18.086 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-19T07:17:18.212 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-19T07:17:18.241 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-19T07:17:18.342 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-19T07:17:18.497 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-19T07:17:18.627 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-19T07:17:18.756 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-19T07:17:18.886 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-19T07:17:19.007 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-19T07:17:19.133 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-19T07:17:19.155 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-19T07:17:19.406 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-19T07:17:19.426 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-19T07:17:19.665 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-19T07:17:19.678 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-19T07:17:19.925 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-19T07:17:19.936 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-19T07:17:20.042 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-19T07:17:20.196 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-19T07:17:20.310 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-19T07:17:20.456 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-19T07:17:20.478 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-19T07:17:20.765 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-19T07:17:20.773 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-19T07:17:20.891 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-19T07:17:21.074 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-19T07:17:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-19T07:17:21.285 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-19T07:17:21.317 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-19T07:17:21.418 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-19T07:17:21.535 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-19T07:17:21.570 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-19T07:17:21.624 INFO:teuthology.orchestra.run.smithi083.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-19T07:17:21.862 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-19T07:17:21.871 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-19T07:17:21.988 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-19T07:17:22.122 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-19T07:17:22.256 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-19T07:17:22.381 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-19T07:17:22.382 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-19T07:17:22.471 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-19T07:17:22.641 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-19T07:17:22.767 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-19T07:17:22.775 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-19T07:17:22.868 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-19T07:17:23.043 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-19T07:17:23.169 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-19T07:17:23.245 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-19T07:17:23.487 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-19T07:17:23.521 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-19T07:17:23.647 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-19T07:17:23.796 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-19T07:17:23.923 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-19T07:17:24.066 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-19T07:17:24.072 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-19T07:17:24.292 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-19T07:17:24.384 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-19T07:17:24.535 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-19T07:17:24.668 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-19T07:17:24.795 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-19T07:17:24.936 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-19T07:17:25.071 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-19T07:17:25.147 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-19T07:17:25.432 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-19T07:17:25.457 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-19T07:17:25.684 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-19T07:17:25.750 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-19T07:17:25.926 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-19T07:17:26.102 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-19T07:17:26.228 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librados2 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:26.320 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-19T07:17:26.354 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libsqlite3-mod-ceph (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:26.480 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-19T07:17:26.605 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-19T07:17:26.808 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libcephfs2 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:26.918 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libradosstriper1 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:26.941 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-19T07:17:27.027 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-19T07:17:27.042 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librados2 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:27.143 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libsqlite3-mod-ceph (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:27.269 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-19T07:17:27.312 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librbd1 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:27.438 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-modules-core (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:27.564 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-fuse (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libcephfs2 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:27.748 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libradosstriper1 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:27.809 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-19T07:17:27.809 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-19T07:17:27.891 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-19T07:17:28.159 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librbd1 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:28.260 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-modules-core (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:28.332 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libcephfs-dev (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:28.369 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-fuse (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:28.450 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-rados (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:28.559 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librgw2 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:28.586 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-19T07:17:28.586 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-19T07:17:28.660 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up cephfs-top (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:28.786 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-19T07:17:29.054 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libcephfs-dev (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:29.180 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rados (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:29.306 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librgw2 (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:29.432 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up cephfs-top (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:29.558 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-19T07:17:29.990 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-rbd (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:30.116 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up rbd-fuse (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:30.250 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-rgw (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:30.401 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cephfs (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:30.544 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up cephfs-shell (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:30.670 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-common (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:30.745 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rbd (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:30.871 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up rbd-fuse (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:30.997 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rgw (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:31.124 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cephfs (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:31.205 INFO:teuthology.orchestra.run.smithi083.stdout:Adding group ceph....done 2024-09-19T07:17:31.233 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up cephfs-shell (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:31.334 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-common (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:31.532 INFO:teuthology.orchestra.run.smithi083.stdout:Adding system user ceph....done 2024-09-19T07:17:31.700 INFO:teuthology.orchestra.run.smithi083.stdout:Setting system user ceph properties....done 2024-09-19T07:17:31.713 INFO:teuthology.orchestra.run.smithi083.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-19T07:17:31.826 INFO:teuthology.orchestra.run.smithi184.stdout:Adding group ceph....done 2024-09-19T07:17:31.852 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-19T07:17:32.136 INFO:teuthology.orchestra.run.smithi184.stdout:Adding system user ceph....done 2024-09-19T07:17:32.195 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-19T07:17:32.297 INFO:teuthology.orchestra.run.smithi184.stdout:Setting system user ceph properties....done 2024-09-19T07:17:32.311 INFO:teuthology.orchestra.run.smithi184.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-19T07:17:32.449 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-19T07:17:32.664 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up cephfs-mirror (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:32.778 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-19T07:17:32.898 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-19T07:17:32.898 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-19T07:17:33.218 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up cephfs-mirror (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:33.398 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-test (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:33.422 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-19T07:17:33.422 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-19T07:17:33.524 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up radosgw (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:33.870 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-test (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:33.976 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-19T07:17:33.976 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-19T07:17:33.995 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up radosgw (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:34.426 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-base (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:34.485 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-19T07:17:34.486 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-19T07:17:34.795 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-19T07:17:34.956 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-base (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:35.236 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mds (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:35.303 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-19T07:17:35.466 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-19T07:17:35.466 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-19T07:17:35.766 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mds (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:35.938 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:35.985 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-19T07:17:35.986 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-19T07:17:36.201 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-19T07:17:36.202 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-19T07:17:36.426 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:36.637 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-19T07:17:36.638 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-19T07:17:36.865 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-osd (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:37.086 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-osd (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:37.165 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-19T07:17:37.165 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-19T07:17:37.357 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-19T07:17:37.357 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-19T07:17:37.676 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-k8sevents (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:37.821 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-k8sevents (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:37.851 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:37.974 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:38.100 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mon (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:38.148 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mon (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:38.297 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-19T07:17:38.298 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-19T07:17:38.361 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-19T07:17:38.361 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-19T07:17:38.728 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-cephadm (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:38.824 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-cephadm (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:38.869 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:38.975 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:38.995 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-dashboard (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:39.093 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-dashboard (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:39.146 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-volume (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:39.219 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-volume (19.1.1-361-g2f599a46-1jammy) ... 2024-09-19T07:17:39.499 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-19T07:17:39.555 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-19T07:17:39.599 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-19T07:17:39.664 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-19T07:17:39.727 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-19T07:17:39.816 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-19T07:17:40.859 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-19T07:17:40.886 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-19T07:17:41.531 INFO:teuthology.orchestra.run.smithi184.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-19T07:17:41.535 DEBUG:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-09-19T07:17:41.536 INFO:teuthology.orchestra.run.smithi083.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-19T07:17:41.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-09-19T07:17:41.593 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-19T07:17:41.632 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-19T07:17:41.787 INFO:teuthology.orchestra.run.smithi083.stdout:Building dependency tree... 2024-09-19T07:17:41.788 INFO:teuthology.orchestra.run.smithi083.stdout:Reading state information... 2024-09-19T07:17:41.832 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2024-09-19T07:17:41.833 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2024-09-19T07:17:41.970 INFO:teuthology.orchestra.run.smithi083.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-09-19T07:17:41.970 INFO:teuthology.orchestra.run.smithi083.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-09-19T07:17:41.970 INFO:teuthology.orchestra.run.smithi083.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-09-19T07:17:41.970 INFO:teuthology.orchestra.run.smithi083.stdout:The following packages were automatically installed and are no longer required: 2024-09-19T07:17:41.970 INFO:teuthology.orchestra.run.smithi083.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-19T07:17:41.970 INFO:teuthology.orchestra.run.smithi083.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-19T07:17:41.971 INFO:teuthology.orchestra.run.smithi083.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-19T07:17:41.971 INFO:teuthology.orchestra.run.smithi083.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-19T07:17:41.971 INFO:teuthology.orchestra.run.smithi083.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-19T07:17:41.972 INFO:teuthology.orchestra.run.smithi083.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-19T07:17:41.972 INFO:teuthology.orchestra.run.smithi083.stdout:The following additional packages will be installed: 2024-09-19T07:17:41.972 INFO:teuthology.orchestra.run.smithi083.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-09-19T07:17:41.972 INFO:teuthology.orchestra.run.smithi083.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-09-19T07:17:41.973 INFO:teuthology.orchestra.run.smithi083.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-09-19T07:17:41.973 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-09-19T07:17:41.973 INFO:teuthology.orchestra.run.smithi083.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-19T07:17:41.973 INFO:teuthology.orchestra.run.smithi083.stdout:Suggested packages: 2024-09-19T07:17:41.973 INFO:teuthology.orchestra.run.smithi083.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-09-19T07:17:41.974 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-09-19T07:17:41.974 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-09-19T07:17:42.015 INFO:teuthology.orchestra.run.smithi083.stdout:The following NEW packages will be installed: 2024-09-19T07:17:42.016 INFO:teuthology.orchestra.run.smithi083.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-09-19T07:17:42.016 INFO:teuthology.orchestra.run.smithi083.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-09-19T07:17:42.017 INFO:teuthology.orchestra.run.smithi083.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-09-19T07:17:42.017 INFO:teuthology.orchestra.run.smithi083.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-09-19T07:17:42.017 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-09-19T07:17:42.017 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-09-19T07:17:42.017 INFO:teuthology.orchestra.run.smithi083.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-19T07:17:42.018 INFO:teuthology.orchestra.run.smithi184.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-09-19T07:17:42.018 INFO:teuthology.orchestra.run.smithi184.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-09-19T07:17:42.018 INFO:teuthology.orchestra.run.smithi184.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-09-19T07:17:42.018 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages were automatically installed and are no longer required: 2024-09-19T07:17:42.018 INFO:teuthology.orchestra.run.smithi184.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-19T07:17:42.018 INFO:teuthology.orchestra.run.smithi184.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-19T07:17:42.018 INFO:teuthology.orchestra.run.smithi184.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-19T07:17:42.018 INFO:teuthology.orchestra.run.smithi184.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-19T07:17:42.019 INFO:teuthology.orchestra.run.smithi083.stdout:The following packages will be upgraded: 2024-09-19T07:17:42.019 INFO:teuthology.orchestra.run.smithi083.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-09-19T07:17:42.020 INFO:teuthology.orchestra.run.smithi184.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-19T07:17:42.020 INFO:teuthology.orchestra.run.smithi184.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-19T07:17:42.020 INFO:teuthology.orchestra.run.smithi184.stdout:The following additional packages will be installed: 2024-09-19T07:17:42.020 INFO:teuthology.orchestra.run.smithi184.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-09-19T07:17:42.020 INFO:teuthology.orchestra.run.smithi184.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-09-19T07:17:42.020 INFO:teuthology.orchestra.run.smithi184.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-09-19T07:17:42.021 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-09-19T07:17:42.021 INFO:teuthology.orchestra.run.smithi184.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-19T07:17:42.021 INFO:teuthology.orchestra.run.smithi184.stdout:Suggested packages: 2024-09-19T07:17:42.021 INFO:teuthology.orchestra.run.smithi184.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-09-19T07:17:42.022 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-09-19T07:17:42.022 INFO:teuthology.orchestra.run.smithi184.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-09-19T07:17:42.063 INFO:teuthology.orchestra.run.smithi184.stdout:The following NEW packages will be installed: 2024-09-19T07:17:42.063 INFO:teuthology.orchestra.run.smithi184.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-09-19T07:17:42.064 INFO:teuthology.orchestra.run.smithi184.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-09-19T07:17:42.064 INFO:teuthology.orchestra.run.smithi184.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-09-19T07:17:42.064 INFO:teuthology.orchestra.run.smithi184.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-09-19T07:17:42.064 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-09-19T07:17:42.065 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-09-19T07:17:42.065 INFO:teuthology.orchestra.run.smithi184.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-19T07:17:42.066 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages will be upgraded: 2024-09-19T07:17:42.066 INFO:teuthology.orchestra.run.smithi184.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-09-19T07:17:42.118 INFO:teuthology.orchestra.run.smithi184.stdout:4 upgraded, 39 newly installed, 0 to remove and 327 not upgraded. 2024-09-19T07:17:42.118 INFO:teuthology.orchestra.run.smithi184.stdout:Need to get 55.8 MB of archives. 2024-09-19T07:17:42.118 INFO:teuthology.orchestra.run.smithi184.stdout:After this operation, 203 MB of additional disk space will be used. 2024-09-19T07:17:42.118 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-09-19T07:17:42.199 INFO:teuthology.orchestra.run.smithi083.stdout:4 upgraded, 39 newly installed, 0 to remove and 327 not upgraded. 2024-09-19T07:17:42.199 INFO:teuthology.orchestra.run.smithi083.stdout:Need to get 55.8 MB of archives. 2024-09-19T07:17:42.199 INFO:teuthology.orchestra.run.smithi083.stdout:After this operation, 203 MB of additional disk space will be used. 2024-09-19T07:17:42.200 INFO:teuthology.orchestra.run.smithi083.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-09-19T07:17:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-09-19T07:17:42.259 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.18 [1,905 kB] 2024-09-19T07:17:42.298 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-09-19T07:17:42.299 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-09-19T07:17:42.300 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-09-19T07:17:42.310 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-09-19T07:17:42.311 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-09-19T07:17:42.313 INFO:teuthology.orchestra.run.smithi184.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-09-19T07:17:42.315 INFO:teuthology.orchestra.run.smithi184.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-09-19T07:17:42.315 INFO:teuthology.orchestra.run.smithi184.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.3 [86.9 kB] 2024-09-19T07:17:42.316 INFO:teuthology.orchestra.run.smithi184.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-09-19T07:17:42.317 INFO:teuthology.orchestra.run.smithi184.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-09-19T07:17:42.334 INFO:teuthology.orchestra.run.smithi184.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-09-19T07:17:42.334 INFO:teuthology.orchestra.run.smithi184.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-09-19T07:17:42.335 INFO:teuthology.orchestra.run.smithi184.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-09-19T07:17:42.654 INFO:teuthology.orchestra.run.smithi184.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-09-19T07:17:42.656 INFO:teuthology.orchestra.run.smithi184.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-09-19T07:17:42.656 INFO:teuthology.orchestra.run.smithi184.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-09-19T07:17:42.658 INFO:teuthology.orchestra.run.smithi184.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libndp0 amd64 1.8-0ubuntu3.1 [12.4 kB] 2024-09-19T07:17:42.658 INFO:teuthology.orchestra.run.smithi184.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-09-19T07:17:42.663 INFO:teuthology.orchestra.run.smithi184.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.13-0ubuntu0.22.04.1 [149 kB] 2024-09-19T07:17:42.664 INFO:teuthology.orchestra.run.smithi184.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-09-19T07:17:42.672 INFO:teuthology.orchestra.run.smithi184.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-09-19T07:17:42.672 INFO:teuthology.orchestra.run.smithi184.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.18 [2,375 kB] 2024-09-19T07:17:42.693 INFO:teuthology.orchestra.run.smithi184.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-09-19T07:17:42.693 INFO:teuthology.orchestra.run.smithi184.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-09-19T07:17:42.703 INFO:teuthology.orchestra.run.smithi184.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-09-19T07:17:42.722 INFO:teuthology.orchestra.run.smithi184.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-09-19T07:17:42.725 INFO:teuthology.orchestra.run.smithi184.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-09-19T07:17:42.725 INFO:teuthology.orchestra.run.smithi184.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-09-19T07:17:42.725 INFO:teuthology.orchestra.run.smithi184.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-09-19T07:17:42.726 INFO:teuthology.orchestra.run.smithi184.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.13-0ubuntu0.22.04.1 [1,225 kB] 2024-09-19T07:17:42.737 INFO:teuthology.orchestra.run.smithi184.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-09-19T07:17:42.737 INFO:teuthology.orchestra.run.smithi184.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.13-0ubuntu0.22.04.1 [16.2 MB] 2024-09-19T07:17:42.821 INFO:teuthology.orchestra.run.smithi083.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-09-19T07:17:42.837 INFO:teuthology.orchestra.run.smithi083.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.18 [1,905 kB] 2024-09-19T07:17:42.947 INFO:teuthology.orchestra.run.smithi184.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-09-19T07:17:42.947 INFO:teuthology.orchestra.run.smithi184.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-09-19T07:17:42.947 INFO:teuthology.orchestra.run.smithi184.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-09-19T07:17:42.948 INFO:teuthology.orchestra.run.smithi184.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-09-19T07:17:42.948 INFO:teuthology.orchestra.run.smithi184.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-09-19T07:17:42.948 INFO:teuthology.orchestra.run.smithi184.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2.1 [1,482 kB] 2024-09-19T07:17:42.959 INFO:teuthology.orchestra.run.smithi184.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-09-19T07:17:42.960 INFO:teuthology.orchestra.run.smithi184.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-09-19T07:17:42.961 INFO:teuthology.orchestra.run.smithi083.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-09-19T07:17:42.963 INFO:teuthology.orchestra.run.smithi083.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-09-19T07:17:42.964 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-09-19T07:17:42.992 INFO:teuthology.orchestra.run.smithi083.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-09-19T07:17:42.994 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-09-19T07:17:42.997 INFO:teuthology.orchestra.run.smithi083.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-09-19T07:17:43.000 INFO:teuthology.orchestra.run.smithi083.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-09-19T07:17:43.001 INFO:teuthology.orchestra.run.smithi083.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.3 [86.9 kB] 2024-09-19T07:17:43.003 INFO:teuthology.orchestra.run.smithi083.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-09-19T07:17:43.066 INFO:teuthology.orchestra.run.smithi083.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-09-19T07:17:43.066 INFO:teuthology.orchestra.run.smithi083.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-09-19T07:17:43.067 INFO:teuthology.orchestra.run.smithi083.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-09-19T07:17:43.147 INFO:teuthology.orchestra.run.smithi083.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-09-19T07:17:43.370 INFO:teuthology.orchestra.run.smithi184.stdout:Preconfiguring packages ... 2024-09-19T07:17:43.521 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 55.8 MB in 1s (63.0 MB/s) 2024-09-19T07:17:43.840 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173987 files and directories currently installed.) 2024-09-19T07:17:43.846 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-19T07:17:43.929 INFO:teuthology.orchestra.run.smithi083.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-09-19T07:17:43.933 INFO:teuthology.orchestra.run.smithi083.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-09-19T07:17:43.933 INFO:teuthology.orchestra.run.smithi083.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-09-19T07:17:43.939 INFO:teuthology.orchestra.run.smithi083.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libndp0 amd64 1.8-0ubuntu3.1 [12.4 kB] 2024-09-19T07:17:43.940 INFO:teuthology.orchestra.run.smithi083.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-09-19T07:17:43.950 INFO:teuthology.orchestra.run.smithi083.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.13-0ubuntu0.22.04.1 [149 kB] 2024-09-19T07:17:43.953 INFO:teuthology.orchestra.run.smithi083.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-09-19T07:17:43.969 INFO:teuthology.orchestra.run.smithi083.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-09-19T07:17:43.970 INFO:teuthology.orchestra.run.smithi083.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.18 [2,375 kB] 2024-09-19T07:17:43.993 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-19T07:17:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-09-19T07:17:44.055 INFO:teuthology.orchestra.run.smithi083.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-09-19T07:17:44.075 INFO:teuthology.orchestra.run.smithi083.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-09-19T07:17:44.146 INFO:teuthology.orchestra.run.smithi083.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-09-19T07:17:44.152 INFO:teuthology.orchestra.run.smithi083.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-09-19T07:17:44.153 INFO:teuthology.orchestra.run.smithi083.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-09-19T07:17:44.154 INFO:teuthology.orchestra.run.smithi083.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-09-19T07:17:44.155 INFO:teuthology.orchestra.run.smithi083.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.13-0ubuntu0.22.04.1 [1,225 kB] 2024-09-19T07:17:44.191 INFO:teuthology.orchestra.run.smithi083.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-09-19T07:17:44.194 INFO:teuthology.orchestra.run.smithi083.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.13-0ubuntu0.22.04.1 [16.2 MB] 2024-09-19T07:17:44.631 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-09-19T07:17:44.724 INFO:teuthology.orchestra.run.smithi083.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-09-19T07:17:44.724 INFO:teuthology.orchestra.run.smithi083.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-09-19T07:17:44.724 INFO:teuthology.orchestra.run.smithi083.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-09-19T07:17:44.758 INFO:teuthology.orchestra.run.smithi083.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-09-19T07:17:44.758 INFO:teuthology.orchestra.run.smithi083.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-09-19T07:17:44.759 INFO:teuthology.orchestra.run.smithi083.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2.1 [1,482 kB] 2024-09-19T07:17:44.789 INFO:teuthology.orchestra.run.smithi083.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-09-19T07:17:44.790 INFO:teuthology.orchestra.run.smithi083.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-09-19T07:17:45.209 INFO:teuthology.orchestra.run.smithi083.stdout:Preconfiguring packages ... 2024-09-19T07:17:45.375 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 55.8 MB in 3s (20.2 MB/s) 2024-09-19T07:17:45.542 INFO:teuthology.orchestra.run.smithi184.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-09-19T07:17:45.723 INFO:teuthology.orchestra.run.smithi083.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173987 files and directories currently installed.) 2024-09-19T07:17:45.729 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-19T07:17:45.807 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package flex. 2024-09-19T07:17:45.884 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-19T07:17:45.909 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173987 files and directories currently installed.) 2024-09-19T07:17:45.915 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-09-19T07:17:45.955 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking flex (2.6.4-8build2) ... 2024-09-19T07:17:46.479 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.18_amd64.deb ... 2024-09-19T07:17:46.514 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-09-19T07:17:46.627 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.18) over (3.0.2-0ubuntu1.7) ... 2024-09-19T07:17:47.032 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-19T07:17:47.404 INFO:teuthology.orchestra.run.smithi083.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-09-19T07:17:47.576 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174060 files and directories currently installed.) 2024-09-19T07:17:47.581 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-19T07:17:47.664 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package flex. 2024-09-19T07:17:47.704 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-19T07:17:47.765 INFO:teuthology.orchestra.run.smithi083.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173987 files and directories currently installed.) 2024-09-19T07:17:47.769 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-09-19T07:17:47.821 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking flex (2.6.4-8build2) ... 2024-09-19T07:17:48.113 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-19T07:17:48.321 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-19T07:17:48.337 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.18_amd64.deb ... 2024-09-19T07:17:48.510 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.18) over (3.0.2-0ubuntu1.7) ... 2024-09-19T07:17:48.657 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-09-19T07:17:48.963 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-19T07:17:49.030 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package bison. 2024-09-19T07:17:49.135 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174060 files and directories currently installed.) 2024-09-19T07:17:49.141 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-09-19T07:17:49.198 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-09-19T07:17:49.501 INFO:teuthology.orchestra.run.smithi083.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174060 files and directories currently installed.) 2024-09-19T07:17:49.506 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-19T07:17:49.600 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-09-19T07:17:49.628 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-09-19T07:17:49.634 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-19T07:17:49.666 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-09-19T07:17:49.936 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package dump. 2024-09-19T07:17:49.952 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-09-19T07:17:49.994 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking dump (0.4b47-3) ... 2024-09-19T07:17:50.052 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-19T07:17:50.183 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-19T07:17:50.273 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package indent. 2024-09-19T07:17:50.289 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-09-19T07:17:50.331 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-09-19T07:17:50.518 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-09-19T07:17:50.568 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-09-19T07:17:50.584 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-09-19T07:17:50.618 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-09-19T07:17:50.900 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package bison. 2024-09-19T07:17:50.963 INFO:teuthology.orchestra.run.smithi083.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174060 files and directories currently installed.) 2024-09-19T07:17:50.968 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-09-19T07:17:50.971 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-09-19T07:17:50.999 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.3_amd64.deb ... 2024-09-19T07:17:51.002 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-09-19T07:17:51.038 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.3) ... 2024-09-19T07:17:51.299 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-09-19T07:17:51.327 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-09-19T07:17:51.366 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-09-19T07:17:51.420 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-09-19T07:17:51.448 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-09-19T07:17:51.487 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-09-19T07:17:51.670 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libfl2:amd64. 2024-09-19T07:17:51.697 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-09-19T07:17:51.736 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-09-19T07:17:51.740 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package dump. 2024-09-19T07:17:51.768 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-09-19T07:17:51.798 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking dump (0.4b47-3) ... 2024-09-19T07:17:51.955 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-09-19T07:17:51.983 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-09-19T07:17:52.022 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-09-19T07:17:52.085 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package indent. 2024-09-19T07:17:52.101 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-09-19T07:17:52.143 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-09-19T07:17:52.275 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libjson-perl. 2024-09-19T07:17:52.291 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-09-19T07:17:52.333 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-09-19T07:17:52.496 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-09-19T07:17:52.512 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-09-19T07:17:52.554 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-09-19T07:17:52.646 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libllvm14:amd64. 2024-09-19T07:17:52.761 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-09-19T07:17:52.761 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-09-19T07:17:52.867 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-09-19T07:17:52.891 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.3_amd64.deb ... 2024-09-19T07:17:52.925 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.3) ... 2024-09-19T07:17:53.186 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-09-19T07:17:53.214 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-09-19T07:17:53.253 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-09-19T07:17:53.581 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libfl2:amd64. 2024-09-19T07:17:53.609 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-09-19T07:17:53.648 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-09-19T07:17:53.875 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-09-19T07:17:53.892 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-09-19T07:17:53.926 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-09-19T07:17:54.153 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libjson-perl. 2024-09-19T07:17:54.181 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-09-19T07:17:54.220 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-09-19T07:17:54.540 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libllvm14:amd64. 2024-09-19T07:17:54.567 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-09-19T07:17:54.606 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-09-19T07:17:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-09-19T07:17:54.951 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-09-19T07:17:54.990 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-09-19T07:17:55.259 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libmbim-proxy. 2024-09-19T07:17:55.276 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-09-19T07:17:55.318 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-09-19T07:17:55.637 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-09-19T07:17:55.665 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-09-19T07:17:55.704 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-09-19T07:17:56.040 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libndp0:amd64. 2024-09-19T07:17:56.068 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3.1_amd64.deb ... 2024-09-19T07:17:56.107 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3.1) ... 2024-09-19T07:17:56.343 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-09-19T07:17:56.370 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-09-19T07:17:56.409 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-09-19T07:17:56.427 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libnm0:amd64. 2024-09-19T07:17:56.455 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-09-19T07:17:56.493 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-09-19T07:17:56.629 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libmbim-proxy. 2024-09-19T07:17:56.656 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-09-19T07:17:56.695 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-09-19T07:17:56.805 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libpq5:amd64. 2024-09-19T07:17:56.833 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../16-libpq5_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-19T07:17:56.871 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libpq5:amd64 (14.13-0ubuntu0.22.04.1) ... 2024-09-19T07:17:57.031 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-09-19T07:17:57.059 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-09-19T07:17:57.098 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-09-19T07:17:57.208 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-09-19T07:17:57.235 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-09-19T07:17:57.274 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-09-19T07:17:57.443 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libndp0:amd64. 2024-09-19T07:17:57.470 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3.1_amd64.deb ... 2024-09-19T07:17:57.509 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3.1) ... 2024-09-19T07:17:57.588 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libqmi-proxy. 2024-09-19T07:17:57.616 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-09-19T07:17:57.771 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-09-19T07:17:57.795 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libnm0:amd64. 2024-09-19T07:17:57.823 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-09-19T07:17:57.853 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-09-19T07:17:58.138 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-09-19T07:17:58.154 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.18_amd64.deb ... 2024-09-19T07:17:58.188 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-19T07:17:58.207 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libpq5:amd64. 2024-09-19T07:17:58.234 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../16-libpq5_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-19T07:17:58.273 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libpq5:amd64 (14.13-0ubuntu0.22.04.1) ... 2024-09-19T07:17:58.637 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-09-19T07:17:58.662 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-09-19T07:17:58.700 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-09-19T07:17:58.701 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-09-19T07:17:58.728 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-09-19T07:17:58.767 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-09-19T07:17:58.979 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libqmi-proxy. 2024-09-19T07:17:59.008 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-09-19T07:17:59.037 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-09-19T07:17:59.087 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package modemmanager. 2024-09-19T07:17:59.115 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-09-19T07:17:59.153 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-09-19T07:17:59.290 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-09-19T07:17:59.317 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.18_amd64.deb ... 2024-09-19T07:17:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-19T07:17:59.777 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package network-manager. 2024-09-19T07:17:59.805 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-09-19T07:17:59.844 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-09-19T07:17:59.919 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-09-19T07:17:59.946 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-09-19T07:17:59.985 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-09-19T07:18:00.263 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package modemmanager. 2024-09-19T07:18:00.291 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-09-19T07:18:00.321 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-09-19T07:18:00.519 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ppp. 2024-09-19T07:18:00.547 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-09-19T07:18:00.817 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-09-19T07:18:00.962 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package network-manager. 2024-09-19T07:18:00.989 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-09-19T07:18:01.037 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-09-19T07:18:01.149 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package pptp-linux. 2024-09-19T07:18:01.166 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-09-19T07:18:01.207 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-09-19T07:18:01.427 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package network-manager-pptp. 2024-09-19T07:18:01.443 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-09-19T07:18:01.485 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-09-19T07:18:01.669 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ppp. 2024-09-19T07:18:01.697 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-09-19T07:18:01.789 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql-client-common. 2024-09-19T07:18:01.816 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-09-19T07:18:01.855 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql-client-common (238) ... 2024-09-19T07:18:01.957 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-09-19T07:18:02.159 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql-client-14. 2024-09-19T07:18:02.187 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../27-postgresql-client-14_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-19T07:18:02.225 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql-client-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-19T07:18:02.274 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package pptp-linux. 2024-09-19T07:18:02.301 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-09-19T07:18:02.340 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-09-19T07:18:02.593 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package network-manager-pptp. 2024-09-19T07:18:02.609 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-09-19T07:18:02.652 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-09-19T07:18:02.755 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql-common. 2024-09-19T07:18:02.783 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-09-19T07:18:02.841 INFO:teuthology.orchestra.run.smithi184.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-09-19T07:18:02.870 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql-common (238) ... 2024-09-19T07:18:02.921 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package postgresql-client-common. 2024-09-19T07:18:02.949 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-09-19T07:18:02.979 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking postgresql-client-common (238) ... 2024-09-19T07:18:03.275 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package postgresql-client-14. 2024-09-19T07:18:03.303 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../27-postgresql-client-14_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-19T07:18:03.313 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql-14. 2024-09-19T07:18:03.341 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../29-postgresql-14_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-19T07:18:03.342 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking postgresql-client-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-19T07:18:03.383 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-19T07:18:03.988 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package postgresql-common. 2024-09-19T07:18:04.016 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-09-19T07:18:04.074 INFO:teuthology.orchestra.run.smithi083.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-09-19T07:18:04.102 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking postgresql-common (238) ... 2024-09-19T07:18:04.529 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package postgresql-14. 2024-09-19T07:18:04.557 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../29-postgresql-14_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-19T07:18:04.599 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking postgresql-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-19T07:18:04.799 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql. 2024-09-19T07:18:04.816 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-09-19T07:18:04.857 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql (14+238) ... 2024-09-19T07:18:05.211 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql-client. 2024-09-19T07:18:05.228 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-09-19T07:18:05.261 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql-client (14+238) ... 2024-09-19T07:18:05.455 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql-contrib. 2024-09-19T07:18:05.471 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-09-19T07:18:05.505 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql-contrib (14+238) ... 2024-09-19T07:18:05.807 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package tcl8.6. 2024-09-19T07:18:05.835 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-09-19T07:18:05.873 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-09-19T07:18:05.984 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package postgresql. 2024-09-19T07:18:06.013 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-09-19T07:18:06.050 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking postgresql (14+238) ... 2024-09-19T07:18:06.168 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package tcl. 2024-09-19T07:18:06.197 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-09-19T07:18:06.239 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-09-19T07:18:06.269 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package postgresql-client. 2024-09-19T07:18:06.285 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-09-19T07:18:06.319 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking postgresql-client (14+238) ... 2024-09-19T07:18:06.546 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package wpasupplicant. 2024-09-19T07:18:06.555 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package postgresql-contrib. 2024-09-19T07:18:06.571 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-09-19T07:18:06.574 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2.1_amd64.deb ... 2024-09-19T07:18:06.612 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2.1) ... 2024-09-19T07:18:06.613 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking postgresql-contrib (14+238) ... 2024-09-19T07:18:06.941 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package tcl8.6. 2024-09-19T07:18:06.967 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-09-19T07:18:06.969 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-09-19T07:18:06.991 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-09-19T07:18:07.029 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-09-19T07:18:07.033 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-09-19T07:18:07.318 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package tcl. 2024-09-19T07:18:07.346 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-09-19T07:18:07.399 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-09-19T07:18:07.412 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package usb-modeswitch. 2024-09-19T07:18:07.440 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-09-19T07:18:07.487 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-09-19T07:18:07.680 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package wpasupplicant. 2024-09-19T07:18:07.708 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2.1_amd64.deb ... 2024-09-19T07:18:07.746 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2.1) ... 2024-09-19T07:18:07.800 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql-client-common (238) ... 2024-09-19T07:18:08.010 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up flex (2.6.4-8build2) ... 2024-09-19T07:18:08.134 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-09-19T07:18:08.135 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-09-19T07:18:08.150 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-09-19T07:18:08.192 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-09-19T07:18:08.262 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libpq5:amd64 (14.13-0ubuntu0.22.04.1) ... 2024-09-19T07:18:08.388 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-09-19T07:18:08.489 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-09-19T07:18:08.579 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package usb-modeswitch. 2024-09-19T07:18:08.590 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql-client-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-19T07:18:08.607 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-09-19T07:18:08.654 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-09-19T07:18:08.950 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up postgresql-client-common (238) ... 2024-09-19T07:18:09.057 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-09-19T07:18:09.137 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2.1) ... 2024-09-19T07:18:09.160 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up flex (2.6.4-8build2) ... 2024-09-19T07:18:09.294 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-09-19T07:18:09.429 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libpq5:amd64 (14.13-0ubuntu0.22.04.1) ... 2024-09-19T07:18:09.535 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-19T07:18:09.535 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-19T07:18:09.538 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-09-19T07:18:09.647 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-09-19T07:18:09.773 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up postgresql-client-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-19T07:18:10.024 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-09-19T07:18:10.158 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-09-19T07:18:10.257 INFO:teuthology.orchestra.run.smithi083.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-09-19T07:18:10.284 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-19T07:18:10.338 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2.1) ... 2024-09-19T07:18:10.410 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-09-19T07:18:10.537 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-09-19T07:18:10.662 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-09-19T07:18:10.721 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-19T07:18:10.721 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-19T07:18:10.780 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-09-19T07:18:10.906 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-09-19T07:18:11.216 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-09-19T07:18:11.341 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-09-19T07:18:11.484 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-19T07:18:11.553 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-09-19T07:18:11.619 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-09-19T07:18:11.679 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up dump (0.4b47-3) ... 2024-09-19T07:18:11.745 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-09-19T07:18:11.772 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-09-19T07:18:11.830 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-09-19T07:18:11.862 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-09-19T07:18:11.932 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.3) ... 2024-09-19T07:18:11.972 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-09-19T07:18:12.041 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up tcl (8.6.11+1build2) ... 2024-09-19T07:18:12.081 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-09-19T07:18:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libjson-perl (4.04000-1) ... 2024-09-19T07:18:12.293 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-09-19T07:18:12.383 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-09-19T07:18:12.453 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-09-19T07:18:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-09-19T07:18:12.728 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-09-19T07:18:12.755 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3.1) ... 2024-09-19T07:18:12.846 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up dump (0.4b47-3) ... 2024-09-19T07:18:12.856 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-09-19T07:18:12.957 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-09-19T07:18:13.064 INFO:teuthology.orchestra.run.smithi083.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-09-19T07:18:13.117 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql-client (14+238) ... 2024-09-19T07:18:13.125 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-09-19T07:18:13.240 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.3) ... 2024-09-19T07:18:13.243 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-09-19T07:18:13.366 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up tcl (8.6.11+1build2) ... 2024-09-19T07:18:13.492 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libjson-perl (4.04000-1) ... 2024-09-19T07:18:13.618 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-09-19T07:18:13.709 INFO:teuthology.orchestra.run.smithi083.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-09-19T07:18:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:13.719 INFO:teuthology.orchestra.run.smithi184.stdout:The following network interfaces were found in /etc/network/interfaces 2024-09-19T07:18:13.719 INFO:teuthology.orchestra.run.smithi184.stdout:which means they are currently configured by ifupdown: 2024-09-19T07:18:13.719 INFO:teuthology.orchestra.run.smithi184.stdout:- enp3s0f1 2024-09-19T07:18:13.719 INFO:teuthology.orchestra.run.smithi184.stdout:If you want to manage those interfaces with NetworkManager instead 2024-09-19T07:18:13.719 INFO:teuthology.orchestra.run.smithi184.stdout:remove their configuration from /etc/network/interfaces. 2024-09-19T07:18:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:13.786 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-09-19T07:18:13.831 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-09-19T07:18:13.913 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-09-19T07:18:14.064 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3.1) ... 2024-09-19T07:18:14.165 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-09-19T07:18:14.201 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-09-19T07:18:14.316 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-09-19T07:18:14.484 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up postgresql-client (14+238) ... 2024-09-19T07:18:14.517 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-09-19T07:18:14.610 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-09-19T07:18:15.012 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-09-19T07:18:15.073 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:15.074 INFO:teuthology.orchestra.run.smithi083.stdout:The following network interfaces were found in /etc/network/interfaces 2024-09-19T07:18:15.074 INFO:teuthology.orchestra.run.smithi083.stdout:which means they are currently configured by ifupdown: 2024-09-19T07:18:15.074 INFO:teuthology.orchestra.run.smithi083.stdout:- enp3s0f1 2024-09-19T07:18:15.074 INFO:teuthology.orchestra.run.smithi083.stdout:If you want to manage those interfaces with NetworkManager instead 2024-09-19T07:18:15.074 INFO:teuthology.orchestra.run.smithi083.stdout:remove their configuration from /etc/network/interfaces. 2024-09-19T07:18:15.074 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:15.164 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-09-19T07:18:15.190 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-09-19T07:18:15.492 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-09-19T07:18:15.648 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-09-19T07:18:15.749 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql-common (238) ... 2024-09-19T07:18:15.799 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-09-19T07:18:16.254 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-09-19T07:18:16.406 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-09-19T07:18:16.600 INFO:teuthology.orchestra.run.smithi184.stdout:Adding user postgres to group ssl-cert 2024-09-19T07:18:16.794 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-09-19T07:18:16.813 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:16.813 INFO:teuthology.orchestra.run.smithi184.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-09-19T07:18:17.014 INFO:teuthology.orchestra.run.smithi184.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-09-19T07:18:17.014 INFO:teuthology.orchestra.run.smithi184.stdout:Removing obsolete dictionary files: 2024-09-19T07:18:17.449 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up postgresql-common (238) ... 2024-09-19T07:18:17.630 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-09-19T07:18:18.010 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-19T07:18:18.333 INFO:teuthology.orchestra.run.smithi083.stdout:Adding user postgres to group ssl-cert 2024-09-19T07:18:18.348 INFO:teuthology.orchestra.run.smithi184.stdout:Creating new PostgreSQL cluster 14/main ... 2024-09-19T07:18:18.365 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-09-19T07:18:18.411 INFO:teuthology.orchestra.run.smithi184.stdout:The files belonging to this database system will be owned by user "postgres". 2024-09-19T07:18:18.412 INFO:teuthology.orchestra.run.smithi184.stdout:This user must also own the server process. 2024-09-19T07:18:18.412 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:18.412 INFO:teuthology.orchestra.run.smithi184.stdout:The database cluster will be initialized with locale "en_US". 2024-09-19T07:18:18.412 INFO:teuthology.orchestra.run.smithi184.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-09-19T07:18:18.412 INFO:teuthology.orchestra.run.smithi184.stdout:The default text search configuration will be set to "english". 2024-09-19T07:18:18.412 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:18.412 INFO:teuthology.orchestra.run.smithi184.stdout:Data page checksums are disabled. 2024-09-19T07:18:18.412 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:18.412 INFO:teuthology.orchestra.run.smithi184.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-09-19T07:18:18.412 INFO:teuthology.orchestra.run.smithi184.stdout:creating subdirectories ... ok 2024-09-19T07:18:18.412 INFO:teuthology.orchestra.run.smithi184.stdout:selecting dynamic shared memory implementation ... posix 2024-09-19T07:18:18.426 INFO:teuthology.orchestra.run.smithi184.stdout:selecting default max_connections ... 100 2024-09-19T07:18:18.446 INFO:teuthology.orchestra.run.smithi184.stdout:selecting default shared_buffers ... 128MB 2024-09-19T07:18:18.453 INFO:teuthology.orchestra.run.smithi184.stdout:selecting default time zone ... Etc/UTC 2024-09-19T07:18:18.454 INFO:teuthology.orchestra.run.smithi184.stdout:creating configuration files ... ok 2024-09-19T07:18:18.534 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:18.534 INFO:teuthology.orchestra.run.smithi083.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-09-19T07:18:18.569 INFO:teuthology.orchestra.run.smithi184.stdout:running bootstrap script ... ok 2024-09-19T07:18:18.699 INFO:teuthology.orchestra.run.smithi083.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-09-19T07:18:18.699 INFO:teuthology.orchestra.run.smithi083.stdout:Removing obsolete dictionary files: 2024-09-19T07:18:19.150 INFO:teuthology.orchestra.run.smithi184.stdout:performing post-bootstrap initialization ... ok 2024-09-19T07:18:19.233 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-09-19T07:18:19.601 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up postgresql-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-19T07:18:19.804 INFO:teuthology.orchestra.run.smithi184.stdout:syncing data to disk ... ok 2024-09-19T07:18:19.915 INFO:teuthology.orchestra.run.smithi083.stdout:Creating new PostgreSQL cluster 14/main ... 2024-09-19T07:18:19.925 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-09-19T07:18:19.965 INFO:teuthology.orchestra.run.smithi083.stdout:The files belonging to this database system will be owned by user "postgres". 2024-09-19T07:18:19.965 INFO:teuthology.orchestra.run.smithi083.stdout:This user must also own the server process. 2024-09-19T07:18:19.965 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:19.965 INFO:teuthology.orchestra.run.smithi083.stdout:The database cluster will be initialized with locale "en_US". 2024-09-19T07:18:19.965 INFO:teuthology.orchestra.run.smithi083.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-09-19T07:18:19.965 INFO:teuthology.orchestra.run.smithi083.stdout:The default text search configuration will be set to "english". 2024-09-19T07:18:19.966 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:19.966 INFO:teuthology.orchestra.run.smithi083.stdout:Data page checksums are disabled. 2024-09-19T07:18:19.966 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:19.966 INFO:teuthology.orchestra.run.smithi083.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-09-19T07:18:19.966 INFO:teuthology.orchestra.run.smithi083.stdout:creating subdirectories ... ok 2024-09-19T07:18:19.966 INFO:teuthology.orchestra.run.smithi083.stdout:selecting dynamic shared memory implementation ... posix 2024-09-19T07:18:19.977 INFO:teuthology.orchestra.run.smithi083.stdout:selecting default max_connections ... 100 2024-09-19T07:18:19.995 INFO:teuthology.orchestra.run.smithi083.stdout:selecting default shared_buffers ... 128MB 2024-09-19T07:18:20.004 INFO:teuthology.orchestra.run.smithi083.stdout:selecting default time zone ... Etc/UTC 2024-09-19T07:18:20.005 INFO:teuthology.orchestra.run.smithi083.stdout:creating configuration files ... ok 2024-09-19T07:18:20.116 INFO:teuthology.orchestra.run.smithi083.stdout:running bootstrap script ... ok 2024-09-19T07:18:20.175 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-09-19T07:18:20.757 INFO:teuthology.orchestra.run.smithi083.stdout:performing post-bootstrap initialization ... ok 2024-09-19T07:18:21.494 INFO:teuthology.orchestra.run.smithi083.stdout:syncing data to disk ... ok 2024-09-19T07:18:21.879 INFO:teuthology.orchestra.run.smithi083.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-09-19T07:18:22.899 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-09-19T07:18:23.001 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql-contrib (14+238) ... 2024-09-19T07:18:23.102 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql (14+238) ... 2024-09-19T07:18:23.228 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-09-19T07:18:23.354 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-09-19T07:18:23.606 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-09-19T07:18:23.607 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-09-19T07:18:24.215 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-19T07:18:24.307 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-09-19T07:18:24.576 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-19T07:18:24.624 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-09-19T07:18:24.703 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-19T07:18:24.742 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up postgresql-contrib (14+238) ... 2024-09-19T07:18:24.843 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up postgresql (14+238) ... 2024-09-19T07:18:25.003 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-09-19T07:18:25.129 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-09-19T07:18:25.383 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-09-19T07:18:25.383 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-09-19T07:18:25.948 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-19T07:18:26.049 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-09-19T07:18:26.359 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-19T07:18:26.486 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-19T07:18:26.724 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-09-19T07:18:26.824 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-19T07:18:27.316 INFO:teuthology.orchestra.run.smithi184.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-19T07:18:27.322 DEBUG:teuthology.parallel:result is None 2024-09-19T07:18:27.864 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-09-19T07:18:27.964 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-19T07:18:28.461 INFO:teuthology.orchestra.run.smithi083.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-19T07:18:28.466 DEBUG:teuthology.parallel:result is None 2024-09-19T07:18:28.467 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:18:28.609 DEBUG:teuthology.orchestra.run.smithi083:> dpkg-query -W -f '${Version}' ceph 2024-09-19T07:18:28.628 INFO:teuthology.orchestra.run.smithi083.stdout:19.1.1-361-g2f599a46-1jammy 2024-09-19T07:18:28.629 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-361-g2f599a46-1jammy 2024-09-19T07:18:28.629 INFO:teuthology.task.install:The correct ceph version 19.1.1-361-g2f599a46-1jammy is installed. 2024-09-19T07:18:28.631 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T07:18:28.772 DEBUG:teuthology.orchestra.run.smithi184:> dpkg-query -W -f '${Version}' ceph 2024-09-19T07:18:28.792 INFO:teuthology.orchestra.run.smithi184.stdout:19.1.1-361-g2f599a46-1jammy 2024-09-19T07:18:28.792 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-361-g2f599a46-1jammy 2024-09-19T07:18:28.792 INFO:teuthology.task.install:The correct ceph version 19.1.1-361-g2f599a46-1jammy is installed. 2024-09-19T07:18:28.794 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-19T07:18:28.794 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:18:28.794 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-19T07:18:28.808 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:18:28.808 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-19T07:18:28.849 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-19T07:18:28.849 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:18:28.849 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-09-19T07:18:28.863 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-19T07:18:28.917 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:18:28.918 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2024-09-19T07:18:28.931 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-19T07:18:28.987 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-19T07:18:28.987 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:18:28.988 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-19T07:18:29.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-19T07:18:29.053 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:18:29.053 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-19T07:18:29.065 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-19T07:18:29.119 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-19T07:18:29.119 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:18:29.119 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-09-19T07:18:29.131 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-19T07:18:29.184 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:18:29.185 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/stdin-killer 2024-09-19T07:18:29.201 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-19T07:18:29.256 INFO:teuthology.run_tasks:Running task ceph... 2024-09-19T07:18:29.337 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-19T07:18:29.338 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /var/log/ceph 2024-09-19T07:18:29.340 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /var/log/ceph 2024-09-19T07:18:29.353 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-19T07:18:29.353 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-19T07:18:29.395 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-19T07:18:29.407 INFO:tasks.ceph:Creating extra log directories... 2024-09-19T07:18:29.407 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-19T07:18:29.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-19T07:18:29.466 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-19T07:18:29.467 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-19T07:18:29.467 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-19T07:18:29.467 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:18:29.467 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-19T07:18:29.506 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-19T07:18:29.581 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:18:29.581 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-19T07:18:29.587 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-19T07:18:29.665 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-19T07:18:29.665 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-19T07:18:29.665 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786', 'branch': 'squid', 'description': 'fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7911786', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'fs': [{'name': 'a'}, {'name': 'b'}], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'ceph-fuse': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961'}}, 'kclient': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '6fad5dd73c93545e52f57f681e140a9c3529c11e'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'seed': 1747, 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961', 'sleep_before_teardown': 0, 'subset': '176/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '6fad5dd73c93545e52f57f681e140a9c3529c11e', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi083.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGrcnfKIxfjcdLikolghc+tQMAndbmmY8LCPtFFmM5TBSbg66m9BPt2D5UU1KWULm7cukELeAjFxF0jIpaJmJPA=', 'smithi184.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF79BURdpun5WHyfcAsEejNcmvyG5AN+Fe9314mey3XSnMTMxXkONeNQ/K5IwZ7lvs5BsX67aumEzD9ia9eS7d8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'fs': [{'name': 'a'}, {'name': 'b'}], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-18_21:24:17', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-19T07:18:29.666 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-19T07:18:29.668 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-19T07:18:29.709 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/run/ceph 2024-09-19T07:18:29.715 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/run/ceph 2024-09-19T07:18:29.763 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:18:29.764 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-09-19T07:18:29.774 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-19T07:18:29.774 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-09-19T07:18:29.823 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-19T07:18:29.823 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:18:29.823 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 874 Links: 1 2024-09-19T07:18:29.823 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:18:29.823 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-19 07:12:13.460592876 +0000 2024-09-19T07:18:29.823 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-19 07:12:13.216595055 +0000 2024-09-19T07:18:29.823 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-19 07:12:13.216595055 +0000 2024-09-19T07:18:29.823 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-09-19T07:18:29.824 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-19T07:18:29.876 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-19T07:18:29.876 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-19T07:18:29.876 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000338516 s, 1.5 MB/s 2024-09-19T07:18:29.877 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-19T07:18:29.927 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-09-19T07:18:29.974 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-19T07:18:29.974 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:18:29.974 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-09-19T07:18:29.974 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:18:29.974 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-19 07:12:13.912588839 +0000 2024-09-19T07:18:29.974 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-19 07:12:13.684590875 +0000 2024-09-19T07:18:29.975 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-19 07:12:13.684590875 +0000 2024-09-19T07:18:29.975 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-09-19T07:18:29.975 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-19T07:18:30.029 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-19T07:18:30.029 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-19T07:18:30.030 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000430801 s, 1.2 MB/s 2024-09-19T07:18:30.030 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-19T07:18:30.082 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-09-19T07:18:30.130 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-19T07:18:30.130 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:18:30.130 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 915 Links: 1 2024-09-19T07:18:30.130 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:18:30.131 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-19 07:12:14.384584623 +0000 2024-09-19T07:18:30.131 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-19 07:12:14.140586802 +0000 2024-09-19T07:18:30.131 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-19 07:12:14.140586802 +0000 2024-09-19T07:18:30.131 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-09-19T07:18:30.131 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-19T07:18:30.184 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-19T07:18:30.185 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-19T07:18:30.185 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000449227 s, 1.1 MB/s 2024-09-19T07:18:30.186 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-19T07:18:30.236 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-09-19T07:18:30.282 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-19T07:18:30.283 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:18:30.283 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 931 Links: 1 2024-09-19T07:18:30.283 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:18:30.283 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-19 07:12:14.852580443 +0000 2024-09-19T07:18:30.283 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-19 07:12:14.624582478 +0000 2024-09-19T07:18:30.283 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-19 07:12:14.624582478 +0000 2024-09-19T07:18:30.283 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-09-19T07:18:30.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-19T07:18:30.337 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-19T07:18:30.337 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-19T07:18:30.337 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000390444 s, 1.3 MB/s 2024-09-19T07:18:30.338 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-19T07:18:30.387 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-19T07:18:30.388 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:18:30.388 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2024-09-19T07:18:30.393 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-19T07:18:30.394 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 2024-09-19T07:18:30.441 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-19T07:18:30.441 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:18:30.441 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 873 Links: 1 2024-09-19T07:18:30.441 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:18:30.441 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-19 07:12:16.427635606 +0000 2024-09-19T07:18:30.441 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-19 07:12:16.171637879 +0000 2024-09-19T07:18:30.441 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-19 07:12:16.171637879 +0000 2024-09-19T07:18:30.441 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2024-09-19T07:18:30.441 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-19T07:18:30.495 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-19T07:18:30.495 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-19T07:18:30.495 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000382391 s, 1.3 MB/s 2024-09-19T07:18:30.497 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-19T07:18:30.548 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 2024-09-19T07:18:30.597 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-19T07:18:30.597 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:18:30.597 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-09-19T07:18:30.597 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:18:30.597 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-19 07:12:16.971630776 +0000 2024-09-19T07:18:30.597 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-19 07:12:16.675633405 +0000 2024-09-19T07:18:30.597 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-19 07:12:16.675633405 +0000 2024-09-19T07:18:30.597 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2024-09-19T07:18:30.597 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-19T07:18:30.651 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-19T07:18:30.651 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-19T07:18:30.651 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000423537 s, 1.2 MB/s 2024-09-19T07:18:30.652 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-19T07:18:30.702 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 2024-09-19T07:18:30.748 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-19T07:18:30.749 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:18:30.749 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 913 Links: 1 2024-09-19T07:18:30.749 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:18:30.749 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-19 07:12:17.475626303 +0000 2024-09-19T07:18:30.749 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-19 07:12:17.235628433 +0000 2024-09-19T07:18:30.749 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-19 07:12:17.235628433 +0000 2024-09-19T07:18:30.749 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2024-09-19T07:18:30.749 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-19T07:18:30.802 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-19T07:18:30.803 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-19T07:18:30.803 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000378606 s, 1.4 MB/s 2024-09-19T07:18:30.804 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-19T07:18:30.853 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 2024-09-19T07:18:30.900 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-19T07:18:30.900 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T07:18:30.900 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-09-19T07:18:30.900 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:18:30.900 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-19 07:12:17.983621793 +0000 2024-09-19T07:18:30.900 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-19 07:12:17.707624243 +0000 2024-09-19T07:18:30.900 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-19 07:12:17.707624243 +0000 2024-09-19T07:18:30.900 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2024-09-19T07:18:30.901 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-19T07:18:30.956 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-19T07:18:30.956 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-19T07:18:30.956 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000282549 s, 1.8 MB/s 2024-09-19T07:18:30.957 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-19T07:18:31.005 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-19T07:18:31.006 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-19T07:18:31.006 INFO:tasks.ceph:Generating config... 2024-09-19T07:18:31.007 INFO:tasks.ceph:[client] client mount timeout = 600 2024-09-19T07:18:31.007 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-09-19T07:18:31.007 INFO:tasks.ceph:[client] debug client = 20 2024-09-19T07:18:31.007 INFO:tasks.ceph:[client] debug ms = 1 2024-09-19T07:18:31.007 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-09-19T07:18:31.007 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-09-19T07:18:31.007 INFO:tasks.ceph:[mds] debug mds = 20 2024-09-19T07:18:31.007 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mds] debug ms = 1 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mds] mds debug frag = True 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-09-19T07:18:31.008 INFO:tasks.ceph:[mgr] debug client = 20 2024-09-19T07:18:31.009 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-19T07:18:31.009 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-19T07:18:31.009 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-19T07:18:31.009 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-19T07:18:31.009 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-19T07:18:31.009 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-09-19T07:18:31.009 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-09-19T07:18:31.009 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-19T07:18:31.009 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-19T07:18:31.009 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-19T07:18:31.009 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-19T07:18:31.009 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-19T07:18:31.009 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-19T07:18:31.009 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-19T07:18:31.010 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-19T07:18:31.010 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-19T07:18:31.010 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-19T07:18:31.010 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-19T07:18:31.010 INFO:tasks.ceph:[osd] filestore flush min = 0 2024-09-19T07:18:31.010 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-19T07:18:31.010 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-19T07:18:31.010 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-19T07:18:31.010 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-19T07:18:31.010 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-19T07:18:31.010 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-19T07:18:31.010 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-19T07:18:31.011 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-09-19T07:18:31.011 INFO:tasks.ceph:Setting up mon.a... 2024-09-19T07:18:31.011 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-19T07:18:31.047 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.keyring 2024-09-19T07:18:31.065 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-19T07:18:31.144 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-19T07:18:31.196 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.83'), ('mon.b', '172.21.15.184'), ('mon.c', '[v2:172.21.15.184:3301,v1:172.21.15.184:6790]')] 2024-09-19T07:18:31.197 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.83,172.21.15.184,[v2:172.21.15.184:3301,v1:172.21.15.184:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-09-19T07:18:31.198 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:18:31.198 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-19T07:18:31.282 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.83 --add b 172.21.15.184 --addv c '[v2:172.21.15.184:3301,v1:172.21.15.184:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:18:31.367 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:18:31.367 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: generated fsid 86ed7344-2eac-4937-889b-a144de44eb64 2024-09-19T07:18:31.368 INFO:teuthology.orchestra.run.smithi083.stdout:setting min_mon_release = quincy 2024-09-19T07:18:31.368 INFO:teuthology.orchestra.run.smithi083.stdout:epoch 0 2024-09-19T07:18:31.368 INFO:teuthology.orchestra.run.smithi083.stdout:fsid 86ed7344-2eac-4937-889b-a144de44eb64 2024-09-19T07:18:31.368 INFO:teuthology.orchestra.run.smithi083.stdout:last_changed 2024-09-19T07:18:31.369625+0000 2024-09-19T07:18:31.368 INFO:teuthology.orchestra.run.smithi083.stdout:created 2024-09-19T07:18:31.369625+0000 2024-09-19T07:18:31.368 INFO:teuthology.orchestra.run.smithi083.stdout:min_mon_release 17 (quincy) 2024-09-19T07:18:31.368 INFO:teuthology.orchestra.run.smithi083.stdout:election_strategy: 1 2024-09-19T07:18:31.368 INFO:teuthology.orchestra.run.smithi083.stdout:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-09-19T07:18:31.368 INFO:teuthology.orchestra.run.smithi083.stdout:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2024-09-19T07:18:31.368 INFO:teuthology.orchestra.run.smithi083.stdout:2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-19T07:18:31.368 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-19T07:18:31.370 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-19T07:18:31.418 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 86ed7344-2eac-4937-889b-a144de44eb64... 2024-09-19T07:18:31.419 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-19T07:18:31.463 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-19T07:18:31.491 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-09-19T07:18:31.491 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-09-19T07:18:31.492 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.493 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-19T07:18:31.493 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-09-19T07:18:31.493 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.493 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-09-19T07:18:31.493 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-19T07:18:31.493 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.493 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-09-19T07:18:31.493 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.493 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-09-19T07:18:31.493 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-09-19T07:18:31.493 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-09-19T07:18:31.493 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.494 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = "172.21.15.83,172.21.15.184,[v2:172.21.15.184:3301,v1:172.21.15.184:6790]" 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 86ed7344-2eac-4937-889b-a144de44eb64 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.495 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug trim objects = true 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-09-19T07:18:31.496 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore allocator = bitmap 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1/20 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1/20 2024-09-19T07:18:31.497 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 4/10 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: filestore flush min = 0 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: osd blocked scrub grace period = 3600 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: osd max markdown count = 1000 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-09-19T07:18:31.498 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-09-19T07:18:31.499 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-09-19T07:18:31.499 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-09-19T07:18:31.499 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-09-19T07:18:31.499 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-09-19T07:18:31.499 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-09-19T07:18:31.499 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-09-19T07:18:31.499 INFO:teuthology.orchestra.run.smithi083.stdout: debug client = 20 2024-09-19T07:18:31.499 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.499 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-09-19T07:18:31.499 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-09-19T07:18:31.499 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.500 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout: mon down mkfs grace = 300 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout: mon op complaint time = 120 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on pool no app = False 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-09-19T07:18:31.501 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout: client mount timeout = 600 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout: client_shutdown_timeout = 120 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout: debug client = 20 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.b] 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.c] 2024-09-19T07:18:31.502 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.a] 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.c] 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.b] 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.d] 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout:[mds] 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds = 20 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds balancer = 20 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout: mds bal fragment size max = 10000 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout: mds bal merge size = 5 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout: mds bal split bits = 3 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout: mds bal split size = 100 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug frag = True 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug scatterstat = True 2024-09-19T07:18:31.503 INFO:teuthology.orchestra.run.smithi083.stdout: mds op complaint time = 180 2024-09-19T07:18:31.504 INFO:teuthology.orchestra.run.smithi083.stdout: mds verify scatter = True 2024-09-19T07:18:31.504 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-09-19T07:18:31.504 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-09-19T07:18:31.504 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-09-19T07:18:31.504 INFO:teuthology.orchestra.run.smithi184.stdout:[global] 2024-09-19T07:18:31.504 INFO:teuthology.orchestra.run.smithi184.stdout: chdir = "" 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: auth supported = cephx 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: filestore xattr use omap = true 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: mon clock drift allowed = 1.000 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: osd crush chooseleaf type = 0 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: auth debug = true 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on old message = true 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on bug = true 2024-09-19T07:18:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: mon pg warn max object skew = 0 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: # disable pg_autoscaler by default for new pools 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default size = 2 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow primary affinity = true 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow pg remap = true 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on legacy crush tunables = false 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on crush straw calc version zero = false 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on no sortbitwise = false 2024-09-19T07:18:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on osd down out interval zero = false 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on too few osds = false 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: mon_allow_pool_size_one = true 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: osd default data pool replay window = 5 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: mon allow pool delete = true 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: mon cluster log file level = debug 2024-09-19T07:18:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: debug asserts on shutdown = true 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: mon health detail to clog = false 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: mon host = "172.21.15.83,172.21.15.184,[v2:172.21.15.184:3301,v1:172.21.15.184:6790]" 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: fsid = 86ed7344-2eac-4937-889b-a144de44eb64 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout:[osd] 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: osd journal size = 100 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub load threshold = 5.0 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub max interval = 600 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: osd mclock profile = high_recovery_ops 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: osd recover clone overlap = true 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: osd recovery max chunk = 1048576 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug shutdown = true 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug op order = true 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify stray on activate = true 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug trim objects = true 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: osd open classes on start = true 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug pg log writeout = true 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: osd deep scrub update digest min age = 30 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: osd map max advance = 10 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: journal zero on create = true 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: filestore ondisk finisher threads = 3 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: filestore apply finisher threads = 3 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: bdev debug aio = true 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug misdirected ops = true 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: bdev async discard = True 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: bdev enable discard = True 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore allocator = bitmap 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore block size = 96636764160 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore fsck on mount = True 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: debug bluefs = 1/20 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: debug bluestore = 1/20 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: debug osd = 20 2024-09-19T07:18:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: debug rocksdb = 4/10 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: filestore flush min = 0 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd backfillfull_ratio = 0.85 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd full ratio = 0.9 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd nearfull ratio = 0.8 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: osd blocked scrub grace period = 3600 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: osd failsafe full ratio = 0.95 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: osd max markdown count = 1000 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: osd objectstore = bluestore 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: osd op complaint time = 180 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr] 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2024-09-19T07:18:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: debug mgr = 20 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: mgr/telemetry/nag = false 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: debug client = 20 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout:[mon] 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: debug paxos = 20 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: mon data avail warn = 5 2024-09-19T07:18:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: mon mgr mkfs grace = 240 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd reporter subtree level = osd 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd prime pg temp = true 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: auth mon ticket ttl = 660 # 11m 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: auth service ticket ttl = 240 # 4m 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: # don't complain about insecure global_id in the test suite 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: # 1m isn't quite enough 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: mon_down_mkfs_grace = 2m 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_filestore_osds = false 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: mon down mkfs grace = 300 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: mon op complaint time = 120 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on pool no app = False 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout:[client] 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: rgw cache enabled = true 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable ops log = true 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable usage log = true 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: client mount timeout = 600 2024-09-19T07:18:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: client_shutdown_timeout = 120 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout: debug client = 20 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout: rados mon op timeout = 900 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout: rados osd op timeout = 900 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.a] 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.b] 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.c] 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.a] 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.c] 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.b] 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.d] 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout:[mds] 2024-09-19T07:18:31.515 INFO:teuthology.orchestra.run.smithi184.stdout: debug mds = 20 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: debug mds balancer = 20 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: mds bal fragment size max = 10000 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: mds bal merge size = 5 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: mds bal split bits = 3 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: mds bal split size = 100 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: mds debug frag = True 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: mds debug scatterstat = True 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: mds op complaint time = 180 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: mds verify scatter = True 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: osd op complaint time = 180 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: rados mon op timeout = 900 2024-09-19T07:18:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: rados osd op timeout = 900 2024-09-19T07:18:31.518 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-19T07:18:31.518 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-19T07:18:31.616 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-19T07:18:31.616 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:18:31.616 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-19T07:18:31.622 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:18:31.622 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-19T07:18:31.669 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi083.front.sepia.ceph.com 2024-09-19T07:18:31.669 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:18:31.670 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-19T07:18:31.670 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-19T07:18:31.732 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:18:31.732 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:18:31.778 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi184.front.sepia.ceph.com 2024-09-19T07:18:31.778 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:18:31.778 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-19T07:18:31.778 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-19T07:18:31.801 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:18:31.801 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:18:31.853 INFO:tasks.ceph:Setting up mon nodes... 2024-09-19T07:18:31.853 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-19T07:18:31.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-19T07:18:31.896 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-19T07:18:31.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-19T07:18:31.940 INFO:teuthology.orchestra.run.smithi184.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-19T07:18:31.965 INFO:tasks.ceph:Setting up mds nodes... 2024-09-19T07:18:31.965 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-19T07:18:32.004 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-19T07:18:32.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-19T07:18:32.062 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-19T07:18:32.143 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-19T07:18:32.148 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-19T07:18:32.202 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-19T07:18:32.246 INFO:teuthology.orchestra.run.smithi184.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-19T07:18:32.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-19T07:18:32.304 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-19T07:18:32.390 INFO:teuthology.orchestra.run.smithi184.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-19T07:18:32.392 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-19T07:18:32.447 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-19T07:18:32.447 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-19T07:18:32.484 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-19T07:18:32.495 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-19T07:18:32.530 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-19T07:18:32.539 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-19T07:18:32.539 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-19T07:18:32.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-19T07:18:32.551 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-19T07:18:32.552 INFO:tasks.ceph:role: osd.0 2024-09-19T07:18:32.552 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-19T07:18:32.552 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-19T07:18:32.613 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-19T07:18:32.613 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-19T07:18:32.613 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-19T07:18:32.613 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-19T07:18:32.613 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-19T07:18:32.613 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-09-19T07:18:32.613 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-19T07:18:32.613 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-19T07:18:32.614 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-19T07:18:32.614 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-19T07:18:33.106 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-09-19T07:18:33.108 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-09-19T07:18:33.108 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-19T07:18:33.270 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-19T07:18:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:18:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: /sbin/restorecon: command not found 2024-09-19T07:18:33.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-19T07:18:33.337 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-19T07:18:33.338 INFO:tasks.ceph:role: osd.1 2024-09-19T07:18:33.338 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-19T07:18:33.338 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-19T07:18:33.401 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-19T07:18:33.401 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-19T07:18:33.401 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-19T07:18:33.401 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-19T07:18:33.401 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-19T07:18:33.401 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-09-19T07:18:33.402 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-19T07:18:33.402 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-19T07:18:33.402 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-19T07:18:33.402 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-19T07:18:33.846 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-09-19T07:18:33.848 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-09-19T07:18:33.848 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-19T07:18:33.933 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-19T07:18:33.988 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: /sbin/restorecon: command not found 2024-09-19T07:18:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:18:33.989 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-19T07:18:34.042 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-19T07:18:34.042 INFO:tasks.ceph:role: osd.2 2024-09-19T07:18:34.042 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-19T07:18:34.042 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-19T07:18:34.107 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-19T07:18:34.107 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-19T07:18:34.107 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-19T07:18:34.107 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-19T07:18:34.107 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-19T07:18:34.107 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-09-19T07:18:34.107 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-19T07:18:34.108 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-19T07:18:34.108 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-19T07:18:34.108 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-19T07:18:34.559 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-09-19T07:18:34.561 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-09-19T07:18:34.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-19T07:18:34.688 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-19T07:18:34.699 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: /sbin/restorecon: command not found 2024-09-19T07:18:34.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:18:34.700 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-19T07:18:34.753 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-19T07:18:34.754 INFO:tasks.ceph:role: osd.3 2024-09-19T07:18:34.754 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-19T07:18:34.754 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-19T07:18:34.818 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-19T07:18:34.819 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-19T07:18:34.819 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-19T07:18:34.819 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-19T07:18:34.819 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-19T07:18:34.819 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-09-19T07:18:34.819 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-19T07:18:34.819 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-19T07:18:34.819 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-19T07:18:34.819 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-19T07:18:35.313 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-09-19T07:18:35.315 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-09-19T07:18:35.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-19T07:18:35.450 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-19T07:18:35.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:18:35.461 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: /sbin/restorecon: command not found 2024-09-19T07:18:35.462 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:18:35.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:35.576+0000 7fe27fbbd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-19T07:18:35.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:35.580+0000 7fe27fbbd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-19T07:18:35.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:35.580+0000 7fe27fbbd7c0 -1 bdev(0x55d17b71d800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-19T07:18:35.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:35.580+0000 7fe27fbbd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-19T07:18:38.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-19T07:18:38.764 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:18:38.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:38.840+0000 7f7223c567c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-19T07:18:38.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:38.840+0000 7f7223c567c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-19T07:18:38.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:38.840+0000 7f7223c567c0 -1 bdev(0x559573fc3800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-19T07:18:38.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:38.840+0000 7f7223c567c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-19T07:18:41.989 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-19T07:18:42.046 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:18:42.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:42.124+0000 7f2e681f67c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-19T07:18:42.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:42.124+0000 7f2e681f67c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-19T07:18:42.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:42.124+0000 7f2e681f67c0 -1 bdev(0x561186cdd800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-19T07:18:42.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:42.124+0000 7f2e681f67c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-19T07:18:45.753 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-19T07:18:45.805 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:18:45.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:45.884+0000 7fa9103ee7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-19T07:18:45.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:45.884+0000 7fa9103ee7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-19T07:18:45.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:45.884+0000 7fa9103ee7c0 -1 bdev(0x55b3a2911800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-19T07:18:45.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:18:45.884+0000 7fa9103ee7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-19T07:18:49.045 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-19T07:18:49.102 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-19T07:18:49.117 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-19T07:18:49.117 INFO:tasks.ceph:role: osd.4 2024-09-19T07:18:49.117 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-19T07:18:49.117 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-19T07:18:49.180 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-19T07:18:49.181 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-19T07:18:49.181 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-19T07:18:49.181 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-19T07:18:49.181 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-19T07:18:49.181 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2024-09-19T07:18:49.181 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-19T07:18:49.181 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-19T07:18:49.181 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-19T07:18:49.181 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-19T07:18:49.632 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2024-09-19T07:18:49.633 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2024-09-19T07:18:49.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-19T07:18:49.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-19T07:18:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:18:49.738 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2024-09-19T07:18:49.738 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-19T07:18:49.794 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-19T07:18:49.795 INFO:tasks.ceph:role: osd.5 2024-09-19T07:18:49.795 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-19T07:18:49.795 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-19T07:18:49.855 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-19T07:18:49.856 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-19T07:18:49.856 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-19T07:18:49.856 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-19T07:18:49.856 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-19T07:18:49.856 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2024-09-19T07:18:49.856 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-19T07:18:49.856 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-19T07:18:49.856 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-19T07:18:49.856 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-19T07:18:50.340 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2024-09-19T07:18:50.342 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2024-09-19T07:18:50.342 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-19T07:18:50.496 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-19T07:18:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:18:50.508 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2024-09-19T07:18:50.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-19T07:18:50.564 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-19T07:18:50.564 INFO:tasks.ceph:role: osd.6 2024-09-19T07:18:50.564 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-19T07:18:50.564 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-19T07:18:50.627 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-19T07:18:50.627 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-19T07:18:50.628 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-19T07:18:50.628 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-19T07:18:50.628 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-19T07:18:50.628 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2024-09-19T07:18:50.628 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-19T07:18:50.628 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-19T07:18:50.628 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-19T07:18:50.628 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-19T07:18:51.112 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2024-09-19T07:18:51.114 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2024-09-19T07:18:51.114 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-19T07:18:51.276 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-19T07:18:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:18:51.288 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2024-09-19T07:18:51.289 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-19T07:18:51.344 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-19T07:18:51.345 INFO:tasks.ceph:role: osd.7 2024-09-19T07:18:51.345 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-19T07:18:51.345 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-19T07:18:51.407 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-19T07:18:51.407 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-19T07:18:51.407 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-19T07:18:51.407 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-19T07:18:51.407 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-19T07:18:51.407 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2024-09-19T07:18:51.407 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-19T07:18:51.408 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-19T07:18:51.408 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-19T07:18:51.408 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-19T07:18:51.833 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2024-09-19T07:18:51.835 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2024-09-19T07:18:51.835 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-19T07:18:51.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-19T07:18:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:18:51.954 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2024-09-19T07:18:51.954 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:18:52.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:18:52.078+0000 7f0442afa7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-19T07:18:52.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:18:52.082+0000 7f0442afa7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-19T07:18:52.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:18:52.082+0000 7f0442afa7c0 -1 bdev(0x56404c8f3800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-19T07:18:52.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:18:52.082+0000 7f0442afa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-19T07:18:55.485 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-19T07:18:55.498 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:18:55.577 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:18:55.578+0000 7f71b1ddd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-19T07:18:55.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:18:55.578+0000 7f71b1ddd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-19T07:18:55.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:18:55.578+0000 7f71b1ddd7c0 -1 bdev(0x5635802ad800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-19T07:18:55.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:18:55.578+0000 7f71b1ddd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-19T07:18:58.731 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-19T07:18:58.793 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:18:58.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:18:58.874+0000 7f72e1b877c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-19T07:18:58.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:18:58.874+0000 7f72e1b877c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-19T07:18:58.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:18:58.874+0000 7f72e1b877c0 -1 bdev(0x55e09ef3d800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-19T07:18:58.875 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:18:58.874+0000 7f72e1b877c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-19T07:18:59.668 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:18:59.670 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:19:02.031 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-19T07:19:02.043 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:19:02.124 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:02.126+0000 7f623a0d97c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-19T07:19:02.125 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:02.126+0000 7f623a0d97c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-19T07:19:02.125 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:02.126+0000 7f623a0d97c0 -1 bdev(0x55d5a34a3800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-19T07:19:02.125 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:02.126+0000 7f623a0d97c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-19T07:19:05.255 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-19T07:19:05.313 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-19T07:19:05.313 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:19:05.313 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-19T07:19:05.329 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:19:05.329 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-19T07:19:05.381 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:19:05.381 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-19T07:19:05.436 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:19:05.437 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-19T07:19:05.492 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:19:05.492 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-19T07:19:05.545 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:19:05.545 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-19T07:19:05.601 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:19:05.601 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-19T07:19:05.658 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:19:05.658 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-19T07:19:05.671 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:19:05.671 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-19T07:19:05.727 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:19:05.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-19T07:19:05.780 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:19:05.780 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-19T07:19:05.839 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:19:05.840 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-19T07:19:05.894 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:19:05.895 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-19T07:19:05.946 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:19:05.946 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-19T07:19:05.998 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:19:05.998 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-19T07:19:06.004 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:19:06.004 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-19T07:19:06.044 INFO:tasks.ceph:Adding keys to all mons... 2024-09-19T07:19:06.044 DEBUG:teuthology.orchestra.run.smithi083:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-19T07:19:06.046 DEBUG:teuthology.orchestra.run.smithi184:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-19T07:19:06.091 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.y] 2024-09-19T07:19:06.091 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBH0OtmpG6KNRAAWZN7bc7PcuIq8AOEe/MafA== 2024-09-19T07:19:06.091 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.a] 2024-09-19T07:19:06.092 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBI0OtmWuVnABAAS+rZVUB7qbTjgoi3ko/4yA== 2024-09-19T07:19:06.092 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.c] 2024-09-19T07:19:06.092 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBI0OtmcmuyCBAAQt2tBSd+LQby20nnHlP+Bw== 2024-09-19T07:19:06.092 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2024-09-19T07:19:06.092 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBL0Otm/37IIhAAn9QwsB/B4N/4zo+sPnT/yQ== 2024-09-19T07:19:06.093 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2024-09-19T07:19:06.093 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBO0OtmqAdsMhAAI0PxDgcuCUeNrPFevTvcDA== 2024-09-19T07:19:06.093 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2024-09-19T07:19:06.093 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBS0Otmi3LGBxAAWGe38/nZtHy5tlLQ2TvzLQ== 2024-09-19T07:19:06.093 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.3] 2024-09-19T07:19:06.094 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBV0OtmNEcQNRAAMfJRxbI8raEnHln5NddBWw== 2024-09-19T07:19:06.094 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.x] 2024-09-19T07:19:06.094 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBH0Otm/ENKOBAA55VlyutNudlL834otFmI/w== 2024-09-19T07:19:06.094 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.b] 2024-09-19T07:19:06.094 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBI0OtmEg/nDhAAGQL+bwEzok2LEaOVJpr2fg== 2024-09-19T07:19:06.094 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.d] 2024-09-19T07:19:06.094 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBI0OtmUjd5FxAAd6qcldNV0jz0EBKV/MIq9g== 2024-09-19T07:19:06.094 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.4] 2024-09-19T07:19:06.095 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBc0Otmb+zTBBAAY3ThV7Yq7smS4zuduYGnCA== 2024-09-19T07:19:06.095 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.5] 2024-09-19T07:19:06.095 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBf0OtmjdivIhAAl3+/cGzTkIzDNU+d4bx3qg== 2024-09-19T07:19:06.095 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.6] 2024-09-19T07:19:06.095 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBi0OtmBBhdNBAAoubQ4YQPEpwFOf3qf545MA== 2024-09-19T07:19:06.096 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.7] 2024-09-19T07:19:06.096 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBm0OtmC5axBxAAJTCcG2DiOY+3o3FgyUWOow== 2024-09-19T07:19:06.096 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-09-19T07:19:06.096 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBI0OtmhVkBHRAAsnMDi8I1prCyhZl/ZDyN5Q== 2024-09-19T07:19:06.096 INFO:teuthology.orchestra.run.smithi083.stdout:[client.1] 2024-09-19T07:19:06.096 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBI0OtmTOfWHxAAujQbCY2cil5BF4b66js69w== 2024-09-19T07:19:06.098 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.y] 2024-09-19T07:19:06.098 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBH0OtmpG6KNRAAWZN7bc7PcuIq8AOEe/MafA== 2024-09-19T07:19:06.098 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.a] 2024-09-19T07:19:06.098 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBI0OtmWuVnABAAS+rZVUB7qbTjgoi3ko/4yA== 2024-09-19T07:19:06.098 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.c] 2024-09-19T07:19:06.098 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBI0OtmcmuyCBAAQt2tBSd+LQby20nnHlP+Bw== 2024-09-19T07:19:06.098 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBL0Otm/37IIhAAn9QwsB/B4N/4zo+sPnT/yQ== 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBO0OtmqAdsMhAAI0PxDgcuCUeNrPFevTvcDA== 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBS0Otmi3LGBxAAWGe38/nZtHy5tlLQ2TvzLQ== 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.3] 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBV0OtmNEcQNRAAMfJRxbI8raEnHln5NddBWw== 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.x] 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBH0Otm/ENKOBAA55VlyutNudlL834otFmI/w== 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.b] 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBI0OtmEg/nDhAAGQL+bwEzok2LEaOVJpr2fg== 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.d] 2024-09-19T07:19:06.099 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBI0OtmUjd5FxAAd6qcldNV0jz0EBKV/MIq9g== 2024-09-19T07:19:06.100 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.4] 2024-09-19T07:19:06.100 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBc0Otmb+zTBBAAY3ThV7Yq7smS4zuduYGnCA== 2024-09-19T07:19:06.100 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.5] 2024-09-19T07:19:06.100 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBf0OtmjdivIhAAl3+/cGzTkIzDNU+d4bx3qg== 2024-09-19T07:19:06.100 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.6] 2024-09-19T07:19:06.100 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBi0OtmBBhdNBAAoubQ4YQPEpwFOf3qf545MA== 2024-09-19T07:19:06.100 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.7] 2024-09-19T07:19:06.100 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBm0OtmC5axBxAAJTCcG2DiOY+3o3FgyUWOow== 2024-09-19T07:19:06.100 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2024-09-19T07:19:06.100 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBI0OtmhVkBHRAAsnMDi8I1prCyhZl/ZDyN5Q== 2024-09-19T07:19:06.100 INFO:teuthology.orchestra.run.smithi184.stdout:[client.1] 2024-09-19T07:19:06.100 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBI0OtmTOfWHxAAujQbCY2cil5BF4b66js69w== 2024-09-19T07:19:06.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-19T07:19:06.143 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-19T07:19:06.188 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-19T07:19:06.191 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-19T07:19:06.236 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-19T07:19:06.283 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-19T07:19:06.325 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:06.367 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:06.415 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:06.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:06.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:06.527 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:06.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:06.593 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:06.680 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-19T07:19:06.682 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-19T07:19:06.725 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-19T07:19:06.771 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-19T07:19:06.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-19T07:19:06.821 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-19T07:19:06.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:06.907 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:06.955 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:06.986 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:07.040 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:07.075 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:07.127 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:07.158 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-19T07:19:07.216 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-19T07:19:07.242 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-19T07:19:07.287 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-19T07:19:07.289 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-19T07:19:07.374 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-19T07:19:07.374 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-19T07:19:07.386 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-19T07:19:07.483 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-19T07:19:07.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-19T07:19:07.508 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-19T07:19:07.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-19T07:19:07.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-19T07:19:07.666 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-19T07:19:07.757 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-19T07:19:07.768 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:19:07.770 DEBUG:teuthology.orchestra.run.smithi184:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-19T07:19:07.815 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-19T07:19:07.816 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-19T07:19:07.816 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-19T07:19:07.818 INFO:tasks.ceph.mon.a:Started 2024-09-19T07:19:07.819 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-19T07:19:07.819 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-19T07:19:07.861 INFO:tasks.ceph.mon.b:Started 2024-09-19T07:19:07.861 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-19T07:19:07.861 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-19T07:19:07.863 INFO:tasks.ceph.mon.c:Started 2024-09-19T07:19:07.863 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-19T07:19:07.864 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-19T07:19:07.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-19T07:19:07.866 INFO:tasks.ceph.mgr.y:Started 2024-09-19T07:19:07.866 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-19T07:19:07.867 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-19T07:19:07.869 INFO:tasks.ceph.mgr.x:Started 2024-09-19T07:19:07.869 INFO:tasks.ceph:Setting crush tunables to default 2024-09-19T07:19:07.869 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd crush tunables default 2024-09-19T07:19:07.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.980+0000 7f0cbacc3640 1 Processor -- start 2024-09-19T07:19:07.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cbacc3640 1 -- start start 2024-09-19T07:19:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cbacc3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 0x7f0cb4159240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cbacc3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 0x7f0cb41605d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cbacc3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cb406d820 con 0x7f0cb4159780 2024-09-19T07:19:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cbacc3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f0cb4159be0 con 0x7f0cb40729b0 2024-09-19T07:19:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cbacc3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f0cb4159ea0 con 0x7f0cb414fb70 2024-09-19T07:19:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cb9260640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 0x7f0cb41605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cb9260640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 0x7f0cb41605d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46550/0 (socket says 172.21.15.83:46550) 2024-09-19T07:19:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cb9260640 1 -- 172.21.15.83:0/1224749690 learned_addr learned my addr 172.21.15.83:0/1224749690 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cb9260640 1 -- 172.21.15.83:0/1224749690 >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 msgr2=0x7f0cb41605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-19T07:19:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cb9260640 1 -- 172.21.15.83:0/1224749690 >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 msgr2=0x7f0cb41605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-19T07:19:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cb9260640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 0x7f0cb41605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T07:19:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cb9260640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 0x7f0cb41605d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T07:19:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cb8a5f640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 0x7f0cb4159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cb8a5f640 1 -- 172.21.15.83:0/1224749690 >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 msgr2=0x7f0cb4159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-19T07:19:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cb8a5f640 1 -- 172.21.15.83:0/1224749690 >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 msgr2=0x7f0cb4159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-19T07:19:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cb8a5f640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 0x7f0cb4159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T07:19:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:07.984+0000 7f0cb8a5f640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 0x7f0cb4159240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T07:19:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.188+0000 7f0cb9260640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 0x7f0cb41605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.188+0000 7f0cb9260640 1 -- 172.21.15.83:0/1224749690 >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 msgr2=0x7f0cb41605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-19T07:19:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.188+0000 7f0cb9260640 1 -- 172.21.15.83:0/1224749690 >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 msgr2=0x7f0cb41605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-19T07:19:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.188+0000 7f0cb9260640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 0x7f0cb41605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T07:19:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.188+0000 7f0cb9260640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 0x7f0cb41605d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T07:19:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.188+0000 7f0cb8a5f640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 0x7f0cb4159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.188+0000 7f0cb8a5f640 1 -- 172.21.15.83:0/1224749690 >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 msgr2=0x7f0cb4159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-19T07:19:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.188+0000 7f0cb8a5f640 1 -- 172.21.15.83:0/1224749690 >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 msgr2=0x7f0cb4159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-19T07:19:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.188+0000 7f0cb8a5f640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 0x7f0cb4159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T07:19:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.188+0000 7f0cb8a5f640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 0x7f0cb4159240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T07:19:08.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.220+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346168425 0 0) 0x7f0cb406d820 con 0x7f0cb4159780 2024-09-19T07:19:08.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.220+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ca4003530 con 0x7f0cb4159780 2024-09-19T07:19:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2458048223 0 0) 0x7f0ca4003530 con 0x7f0cb4159780 2024-09-19T07:19:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 msgr2=0x7f0cb4159240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:19:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.184:3300/0 conn(0x7f0cb414fb70 0x7f0cb4159240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 msgr2=0x7f0cb41605d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:19:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 --2- 172.21.15.83:0/1224749690 >> v2:172.21.15.83:3300/0 conn(0x7f0cb40729b0 0x7f0cb41605d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb406c730 con 0x7f0cb4159780 2024-09-19T07:19:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f0ca0001680 con 0x7f0cb4159780 2024-09-19T07:19:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 >> v1:172.21.15.184:6789/0 conn(0x7f0cb4159780 legacy=0x7f0cb40722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 --2- 172.21.15.83:0/1224749690 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ca4003a10 0x7f0ca4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 --2- 172.21.15.83:0/1224749690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ca4004810 0x7f0ca4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 --2- 172.21.15.83:0/1224749690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 0x7f0ca400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb4159be0 con 0x7f0ca40051d0 2024-09-19T07:19:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb4159ea0 con 0x7f0ca4004810 2024-09-19T07:19:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca400c120 con 0x7f0ca4003a10 2024-09-19T07:19:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb9260640 1 --2- 172.21.15.83:0/1224749690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 0x7f0ca400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb8a5f640 1 --2- 172.21.15.83:0/1224749690 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ca4003a10 0x7f0ca4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb3fff640 1 --2- 172.21.15.83:0/1224749690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ca4004810 0x7f0ca4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb9260640 1 -- 172.21.15.83:0/1224749690 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ca4003a10 msgr2=0x7f0ca4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb9260640 1 --2- 172.21.15.83:0/1224749690 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ca4003a10 0x7f0ca4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb9260640 1 -- 172.21.15.83:0/1224749690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ca4004810 msgr2=0x7f0ca4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb9260640 1 --2- 172.21.15.83:0/1224749690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ca4004810 0x7f0ca4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb9260640 1 -- 172.21.15.83:0/1224749690 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca4003530 con 0x7f0ca40051d0 2024-09-19T07:19:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.228+0000 7f0cb3fff640 1 --2- 172.21.15.83:0/1224749690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ca4004810 0x7f0ca4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:19:08.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.236+0000 7f0cb9260640 1 -- 172.21.15.83:0/1224749690 reap_dead start 2024-09-19T07:19:08.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.236+0000 7f0cb9260640 1 --2- 172.21.15.83:0/1224749690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 0x7f0ca400b9d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009cb0 tx=0x7f0cac01cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1446d0e1995c1c22 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:08.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.236+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac0ab070 con 0x7f0ca40051d0 2024-09-19T07:19:08.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.236+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cac0a9070 con 0x7f0ca40051d0 2024-09-19T07:19:08.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.236+0000 7f0cb37fe640 1 -- 172.21.15.83:0/1224749690 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac0098b0 con 0x7f0ca40051d0 2024-09-19T07:19:08.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.236+0000 7f0cbacc3640 1 -- 172.21.15.83:0/1224749690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 msgr2=0x7f0ca400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:08.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.236+0000 7f0cbacc3640 1 --2- 172.21.15.83:0/1224749690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 0x7f0ca400b9d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009cb0 tx=0x7f0cac01cd50 comp rx=0 tx=0).stop 2024-09-19T07:19:08.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.236+0000 7f0cbacc3640 1 -- 172.21.15.83:0/1224749690 shutdown_connections 2024-09-19T07:19:08.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.236+0000 7f0cbacc3640 1 --2- 172.21.15.83:0/1224749690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 0x7f0ca400b9d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.236+0000 7f0cbacc3640 1 -- 172.21.15.83:0/1224749690 >> 172.21.15.83:0/1224749690 conn(0x7f0cb40cb6e0 msgr2=0x7f0cb40cbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:08.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.236+0000 7f0cbacc3640 1 -- 172.21.15.83:0/1224749690 shutdown_connections 2024-09-19T07:19:08.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.236+0000 7f0cbacc3640 1 -- 172.21.15.83:0/1224749690 wait complete. 2024-09-19T07:19:08.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cbacc3640 1 Processor -- start 2024-09-19T07:19:08.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cbacc3640 1 -- start start 2024-09-19T07:19:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cbacc3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 0x7f0cb41eb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cbacc3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0cb41574d0 0x7f0cb414cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cbacc3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0cb414d2b0 0x7f0cb41ee4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cbacc3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb40c6060 con 0x7f0ca40051d0 2024-09-19T07:19:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cbacc3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb40c7870 con 0x7f0cb414d2b0 2024-09-19T07:19:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cbacc3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0cb40c7d90 con 0x7f0cb41574d0 2024-09-19T07:19:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb8a5f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 0x7f0cb41eb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb8a5f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 0x7f0cb41eb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46610/0 (socket says 172.21.15.83:46610) 2024-09-19T07:19:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb8a5f640 1 -- 172.21.15.83:0/3456388727 learned_addr learned my addr 172.21.15.83:0/3456388727 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb9260640 1 --2- 172.21.15.83:0/3456388727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0cb414d2b0 0x7f0cb41ee4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb3fff640 1 --2- 172.21.15.83:0/3456388727 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0cb41574d0 0x7f0cb414cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb8a5f640 1 -- 172.21.15.83:0/3456388727 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0cb41574d0 msgr2=0x7f0cb414cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb8a5f640 1 --2- 172.21.15.83:0/3456388727 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0cb41574d0 0x7f0cb414cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb8a5f640 1 -- 172.21.15.83:0/3456388727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0cb414d2b0 msgr2=0x7f0cb41ee4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb8a5f640 1 --2- 172.21.15.83:0/3456388727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0cb414d2b0 0x7f0cb41ee4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb8a5f640 1 -- 172.21.15.83:0/3456388727 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb414d760 con 0x7f0ca40051d0 2024-09-19T07:19:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb8a5f640 1 --2- 172.21.15.83:0/3456388727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 0x7f0cb41eb990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8013720 tx=0x7f0ca8013be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:08.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb17fa640 1 -- 172.21.15.83:0/3456388727 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca801a030 con 0x7f0ca40051d0 2024-09-19T07:19:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb17fa640 1 -- 172.21.15.83:0/3456388727 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ca8011a50 con 0x7f0ca40051d0 2024-09-19T07:19:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cbacc3640 1 -- 172.21.15.83:0/3456388727 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb40c5c60 con 0x7f0ca40051d0 2024-09-19T07:19:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb17fa640 1 -- 172.21.15.83:0/3456388727 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca800f040 con 0x7f0ca40051d0 2024-09-19T07:19:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cbacc3640 1 -- 172.21.15.83:0/3456388727 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb4150480 con 0x7f0ca40051d0 2024-09-19T07:19:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb17fa640 1 -- 172.21.15.83:0/3456388727 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0ca801d030 con 0x7f0ca40051d0 2024-09-19T07:19:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cb17fa640 1 -- 172.21.15.83:0/3456388727 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0ca80167a0 con 0x7f0ca40051d0 2024-09-19T07:19:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.240+0000 7f0cbacc3640 1 -- 172.21.15.83:0/3456388727 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cb406d820 con 0x7f0ca40051d0 2024-09-19T07:19:08.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.252+0000 7f0cb17fa640 1 -- 172.21.15.83:0/3456388727 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f0ca8024030 con 0x7f0ca40051d0 2024-09-19T07:19:08.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.320+0000 7f0cbacc3640 1 -- 172.21.15.83:0/3456388727 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f0cb4159110 con 0x7f0ca40051d0 2024-09-19T07:19:08.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.324+0000 7f0cb17fa640 1 -- 172.21.15.83:0/3456388727 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f0ca8011e60 con 0x7f0ca40051d0 2024-09-19T07:19:08.325 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2024-09-19T07:19:08.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.324+0000 7f0cbacc3640 1 -- 172.21.15.83:0/3456388727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 msgr2=0x7f0cb41eb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.324+0000 7f0cbacc3640 1 --2- 172.21.15.83:0/3456388727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 0x7f0cb41eb990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8013720 tx=0x7f0ca8013be0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.324+0000 7f0cbacc3640 1 -- 172.21.15.83:0/3456388727 shutdown_connections 2024-09-19T07:19:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.324+0000 7f0cbacc3640 1 --2- 172.21.15.83:0/3456388727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0cb414d2b0 0x7f0cb41ee4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.324+0000 7f0cbacc3640 1 --2- 172.21.15.83:0/3456388727 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0cb41574d0 0x7f0cb414cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.324+0000 7f0cbacc3640 1 --2- 172.21.15.83:0/3456388727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ca40051d0 0x7f0cb41eb990 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.324+0000 7f0cbacc3640 1 -- 172.21.15.83:0/3456388727 >> 172.21.15.83:0/3456388727 conn(0x7f0cb40cb6e0 msgr2=0x7f0cb40722a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.324+0000 7f0cbacc3640 1 -- 172.21.15.83:0/3456388727 shutdown_connections 2024-09-19T07:19:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:08.324+0000 7f0cbacc3640 1 -- 172.21.15.83:0/3456388727 wait complete. 2024-09-19T07:19:08.345 INFO:tasks.ceph:check_enable_crimson: False 2024-09-19T07:19:08.345 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-19T07:19:08.346 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:19:08.346 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-19T07:19:08.357 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:19:08.358 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-19T07:19:08.415 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:19:08.415 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-19T07:19:08.430 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:08.428+0000 7fb53e71f180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T07:19:08.440 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:19:08.441 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-19T07:19:08.501 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:19:08.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-19T07:19:08.505 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:08.506+0000 7fd37ab39180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T07:19:08.516 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:19:08.516 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-19T07:19:08.571 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:19:08.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-19T07:19:08.627 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:19:08.627 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-19T07:19:08.683 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new d1394d19-a843-4204-a708-4c443d4e24ea 0 2024-09-19T07:19:08.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.854+0000 7fae571fb640 1 Processor -- start 2024-09-19T07:19:08.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.854+0000 7fae571fb640 1 -- start start 2024-09-19T07:19:08.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.854+0000 7fae571fb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae500c3d40 0x7fae500ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:08.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.854+0000 7fae571fb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fae500685e0 0x7fae50156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:08.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.854+0000 7fae571fb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae50074220 con 0x7fae500ab620 2024-09-19T07:19:08.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.854+0000 7fae571fb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fae500ad510 con 0x7fae500685e0 2024-09-19T07:19:08.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.854+0000 7fae571fb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fae500ad030 con 0x7fae500c3d40 2024-09-19T07:19:08.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.854+0000 7fae4ffff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fae500685e0 0x7fae50156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.854+0000 7fae54f97640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae500c3d40 0x7fae500ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.854+0000 7fae4ffff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fae500685e0 0x7fae50156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:37006/0 (socket says 172.21.15.184:37006) 2024-09-19T07:19:08.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.854+0000 7fae4ffff640 1 -- 172.21.15.184:0/4226507733 learned_addr learned my addr 172.21.15.184:0/4226507733 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:08.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae4ffff640 1 -- 172.21.15.184:0/4226507733 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae500c3d40 msgr2=0x7fae500ab0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:08.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae4ffff640 1 --2- 172.21.15.184:0/4226507733 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae500c3d40 0x7fae500ab0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae4ffff640 1 -- 172.21.15.184:0/4226507733 >> v1:172.21.15.83:6789/0 conn(0x7fae500ab620 legacy=0x7fae501569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:08.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae4ffff640 1 -- 172.21.15.184:0/4226507733 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae500c8830 con 0x7fae500685e0 2024-09-19T07:19:08.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae4ffff640 1 --2- 172.21.15.184:0/4226507733 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae500685e0 0x7fae50156490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fae4400b920 tx=0x7fae4401c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85f5ac83704d7d77 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:08.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae4f7fe640 1 -- 172.21.15.184:0/4226507733 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae440b2070 con 0x7fae500685e0 2024-09-19T07:19:08.857 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae4f7fe640 1 -- 172.21.15.184:0/4226507733 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae440a57b0 con 0x7fae500685e0 2024-09-19T07:19:08.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae4f7fe640 1 -- 172.21.15.184:0/4226507733 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae440a5b80 con 0x7fae500685e0 2024-09-19T07:19:08.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae571fb640 1 -- 172.21.15.184:0/4226507733 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae500685e0 msgr2=0x7fae50156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:08.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae571fb640 1 --2- 172.21.15.184:0/4226507733 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae500685e0 0x7fae50156490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fae4400b920 tx=0x7fae4401c8e0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae571fb640 1 -- 172.21.15.184:0/4226507733 shutdown_connections 2024-09-19T07:19:08.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae571fb640 1 --2- 172.21.15.184:0/4226507733 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae500685e0 0x7fae50156490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae571fb640 1 --2- 172.21.15.184:0/4226507733 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae500c3d40 0x7fae500ab0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae571fb640 1 -- 172.21.15.184:0/4226507733 >> 172.21.15.184:0/4226507733 conn(0x7fae500c7b70 msgr2=0x7fae500c7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:08.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae571fb640 1 -- 172.21.15.184:0/4226507733 shutdown_connections 2024-09-19T07:19:08.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.858+0000 7fae571fb640 1 -- 172.21.15.184:0/4226507733 wait complete. 2024-09-19T07:19:08.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae571fb640 1 Processor -- start 2024-09-19T07:19:08.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae571fb640 1 -- start start 2024-09-19T07:19:08.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae571fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae500685e0 0x7fae501ea9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:08.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae571fb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae500ab620 0x7fae500a8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:08.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae571fb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae500c3d40 0x7fae501ee840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:08.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae571fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae500c42a0 con 0x7fae500685e0 2024-09-19T07:19:08.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae571fb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fae500c4120 con 0x7fae500c3d40 2024-09-19T07:19:08.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae571fb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fae500c6280 con 0x7fae500ab620 2024-09-19T07:19:08.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae55798640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae500c3d40 0x7fae501ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae4ffff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae500ab620 0x7fae500a8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae55798640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae500c3d40 0x7fae501ee840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:37020/0 (socket says 172.21.15.184:37020) 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae55798640 1 -- 172.21.15.184:0/2333130394 learned_addr learned my addr 172.21.15.184:0/2333130394 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae54f97640 1 --2- 172.21.15.184:0/2333130394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae500685e0 0x7fae501ea9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae55798640 1 -- 172.21.15.184:0/2333130394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae500ab620 msgr2=0x7fae500a8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae55798640 1 --2- 172.21.15.184:0/2333130394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae500ab620 0x7fae500a8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae55798640 1 -- 172.21.15.184:0/2333130394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae500685e0 msgr2=0x7fae501ea9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae55798640 1 --2- 172.21.15.184:0/2333130394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae500685e0 0x7fae501ea9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae55798640 1 -- 172.21.15.184:0/2333130394 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae501eaee0 con 0x7fae500c3d40 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae54f97640 1 --2- 172.21.15.184:0/2333130394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae500685e0 0x7fae501ea9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae4ffff640 1 --2- 172.21.15.184:0/2333130394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae500ab620 0x7fae500a8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae55798640 1 --2- 172.21.15.184:0/2333130394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae500c3d40 0x7fae501ee840 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fae4000c430 tx=0x7fae4000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae4d7fa640 1 -- 172.21.15.184:0/2333130394 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae40005220 con 0x7fae500c3d40 2024-09-19T07:19:08.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae4d7fa640 1 -- 172.21.15.184:0/2333130394 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae400053c0 con 0x7fae500c3d40 2024-09-19T07:19:08.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae571fb640 1 -- 172.21.15.184:0/2333130394 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae5006ebd0 con 0x7fae500c3d40 2024-09-19T07:19:08.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae4d7fa640 1 -- 172.21.15.184:0/2333130394 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae40007270 con 0x7fae500c3d40 2024-09-19T07:19:08.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae571fb640 1 -- 172.21.15.184:0/2333130394 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae501ee040 con 0x7fae500c3d40 2024-09-19T07:19:08.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae4d7fa640 1 -- 172.21.15.184:0/2333130394 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fae40019020 con 0x7fae500c3d40 2024-09-19T07:19:08.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae4d7fa640 1 -- 172.21.15.184:0/2333130394 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fae40003b20 con 0x7fae500c3d40 2024-09-19T07:19:08.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.862+0000 7fae571fb640 1 -- 172.21.15.184:0/2333130394 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae500c6690 con 0x7fae500c3d40 2024-09-19T07:19:08.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.874+0000 7fae4d7fa640 1 -- 172.21.15.184:0/2333130394 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fae40020070 con 0x7fae500c3d40 2024-09-19T07:19:08.897 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:08.892+0000 7fb53e71f180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T07:19:08.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.946+0000 7fae571fb640 1 -- 172.21.15.184:0/2333130394 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d1394d19-a843-4204-a708-4c443d4e24ea", "id": 0} v 0) -- 0x7fae5006fd90 con 0x7fae500c3d40 2024-09-19T07:19:08.949 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.950+0000 7fae4d7fa640 1 -- 172.21.15.184:0/2333130394 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d1394d19-a843-4204-a708-4c443d4e24ea", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fae40006830 con 0x7fae500c3d40 2024-09-19T07:19:08.950 INFO:teuthology.orchestra.run.smithi184.stdout:0 2024-09-19T07:19:08.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.950+0000 7fae571fb640 1 -- 172.21.15.184:0/2333130394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae500c3d40 msgr2=0x7fae501ee840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:08.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.950+0000 7fae571fb640 1 --2- 172.21.15.184:0/2333130394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae500c3d40 0x7fae501ee840 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fae4000c430 tx=0x7fae4000c8f0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.950+0000 7fae571fb640 1 -- 172.21.15.184:0/2333130394 shutdown_connections 2024-09-19T07:19:08.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.950+0000 7fae571fb640 1 --2- 172.21.15.184:0/2333130394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae500c3d40 0x7fae501ee840 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.950+0000 7fae571fb640 1 --2- 172.21.15.184:0/2333130394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae500ab620 0x7fae500a8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.950+0000 7fae571fb640 1 --2- 172.21.15.184:0/2333130394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae500685e0 0x7fae501ea9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:08.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.950+0000 7fae571fb640 1 -- 172.21.15.184:0/2333130394 >> 172.21.15.184:0/2333130394 conn(0x7fae500c7b70 msgr2=0x7fae50154220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:08.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.950+0000 7fae571fb640 1 -- 172.21.15.184:0/2333130394 shutdown_connections 2024-09-19T07:19:08.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:08.950+0000 7fae571fb640 1 -- 172.21.15.184:0/2333130394 wait complete. 2024-09-19T07:19:08.967 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new ece67475-bb01-4b2d-8e42-078cb90df26d 1 2024-09-19T07:19:08.981 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:08.982+0000 7fd37ab39180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T07:19:09.079 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:09.076+0000 7fb53e71f180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T07:19:09.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec5e1ae640 1 Processor -- start 2024-09-19T07:19:09.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec5e1ae640 1 -- start start 2024-09-19T07:19:09.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec5e1ae640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 0x7fec581500b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec5e1ae640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fec5806bd10 0x7fec5806c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec5e1ae640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec58074080 con 0x7fec580acf80 2024-09-19T07:19:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec5e1ae640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fec580ada70 con 0x7fec5806bd10 2024-09-19T07:19:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec5e1ae640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fec58153310 con 0x7fec5814fc90 2024-09-19T07:19:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec56ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 0x7fec581500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec57fff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fec5806bd10 0x7fec5806c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec56ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 0x7fec581500b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:53450/0 (socket says 172.21.15.184:53450) 2024-09-19T07:19:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec57fff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fec5806bd10 0x7fec5806c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:37030/0 (socket says 172.21.15.184:37030) 2024-09-19T07:19:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec56ffd640 1 -- 172.21.15.184:0/3636480194 learned_addr learned my addr 172.21.15.184:0/3636480194 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec57fff640 1 -- 172.21.15.184:0/3636480194 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 msgr2=0x7fec581500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec57fff640 1 --2- 172.21.15.184:0/3636480194 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 0x7fec581500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec57fff640 1 -- 172.21.15.184:0/3636480194 >> v1:172.21.15.83:6789/0 conn(0x7fec580acf80 legacy=0x7fec580ad360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.090+0000 7fec57fff640 1 -- 172.21.15.184:0/3636480194 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec58070220 con 0x7fec5806bd10 2024-09-19T07:19:09.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec57fff640 1 --2- 172.21.15.184:0/3636480194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec5806bd10 0x7fec5806c1a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fec50009d90 tx=0x7fec5001c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d54b307ba18ea179 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:09.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec567fc640 1 -- 172.21.15.184:0/3636480194 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec500b2070 con 0x7fec5806bd10 2024-09-19T07:19:09.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec567fc640 1 -- 172.21.15.184:0/3636480194 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec500a5710 con 0x7fec5806bd10 2024-09-19T07:19:09.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec567fc640 1 -- 172.21.15.184:0/3636480194 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec500a5ae0 con 0x7fec5806bd10 2024-09-19T07:19:09.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec5e1ae640 1 -- 172.21.15.184:0/3636480194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec5806bd10 msgr2=0x7fec5806c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec5e1ae640 1 --2- 172.21.15.184:0/3636480194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec5806bd10 0x7fec5806c1a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fec50009d90 tx=0x7fec5001c850 comp rx=0 tx=0).stop 2024-09-19T07:19:09.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec5e1ae640 1 -- 172.21.15.184:0/3636480194 shutdown_connections 2024-09-19T07:19:09.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec5e1ae640 1 --2- 172.21.15.184:0/3636480194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec5806bd10 0x7fec5806c1a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec5e1ae640 1 --2- 172.21.15.184:0/3636480194 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 0x7fec581500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec5e1ae640 1 -- 172.21.15.184:0/3636480194 >> 172.21.15.184:0/3636480194 conn(0x7fec580c84e0 msgr2=0x7fec580c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:09.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec5e1ae640 1 -- 172.21.15.184:0/3636480194 shutdown_connections 2024-09-19T07:19:09.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec5e1ae640 1 -- 172.21.15.184:0/3636480194 wait complete. 2024-09-19T07:19:09.095 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec5e1ae640 1 Processor -- start 2024-09-19T07:19:09.095 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.094+0000 7fec5e1ae640 1 -- start start 2024-09-19T07:19:09.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec5e1ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec5806bd10 0x7fec581ecae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec5e1ae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec580acf80 0x7fec581560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec5e1ae640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 0x7fec580c1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec5e1ae640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fec580aa060 con 0x7fec5806bd10 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec5e1ae640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fec580ab720 con 0x7fec580acf80 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec5e1ae640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fec5815fa30 con 0x7fec5814fc90 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec57fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 0x7fec580c1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec56ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec580acf80 0x7fec581560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec57fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 0x7fec580c1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:53456/0 (socket says 172.21.15.184:53456) 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec57fff640 1 -- 172.21.15.184:0/1285020639 learned_addr learned my addr 172.21.15.184:0/1285020639 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec577fe640 1 --2- 172.21.15.184:0/1285020639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec5806bd10 0x7fec581ecae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec57fff640 1 -- 172.21.15.184:0/1285020639 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec580acf80 msgr2=0x7fec581560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec57fff640 1 --2- 172.21.15.184:0/1285020639 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec580acf80 0x7fec581560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec57fff640 1 -- 172.21.15.184:0/1285020639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec5806bd10 msgr2=0x7fec581ecae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec57fff640 1 --2- 172.21.15.184:0/1285020639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec5806bd10 0x7fec581ecae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec57fff640 1 -- 172.21.15.184:0/1285020639 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec581ec300 con 0x7fec5814fc90 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec56ffd640 1 --2- 172.21.15.184:0/1285020639 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec580acf80 0x7fec581560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec577fe640 1 --2- 172.21.15.184:0/1285020639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec5806bd10 0x7fec581ecae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec57fff640 1 --2- 172.21.15.184:0/1285020639 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 0x7fec580c1290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec500b0040 tx=0x7fec500195f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec3ffff640 1 -- 172.21.15.184:0/1285020639 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec500b2050 con 0x7fec5814fc90 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec3ffff640 1 -- 172.21.15.184:0/1285020639 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec500be070 con 0x7fec5814fc90 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec5e1ae640 1 -- 172.21.15.184:0/1285020639 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec581f0090 con 0x7fec5814fc90 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec3ffff640 1 -- 172.21.15.184:0/1285020639 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec500ad890 con 0x7fec5814fc90 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec5e1ae640 1 -- 172.21.15.184:0/1285020639 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fec581ec5c0 con 0x7fec5814fc90 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec3ffff640 1 -- 172.21.15.184:0/1285020639 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fec500ada30 con 0x7fec5814fc90 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec3ffff640 1 -- 172.21.15.184:0/1285020639 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fec500b9450 con 0x7fec5814fc90 2024-09-19T07:19:09.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.098+0000 7fec5e1ae640 1 -- 172.21.15.184:0/1285020639 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec58076060 con 0x7fec5814fc90 2024-09-19T07:19:09.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.102+0000 7fec3ffff640 1 -- 172.21.15.184:0/1285020639 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fec50016560 con 0x7fec5814fc90 2024-09-19T07:19:09.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.170+0000 7fec5e1ae640 1 -- 172.21.15.184:0/1285020639 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ece67475-bb01-4b2d-8e42-078cb90df26d", "id": 1} v 0) -- 0x7fec581ec110 con 0x7fec5814fc90 2024-09-19T07:19:09.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.178+0000 7fec3ffff640 1 -- 172.21.15.184:0/1285020639 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ece67475-bb01-4b2d-8e42-078cb90df26d", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fec500c3460 con 0x7fec5814fc90 2024-09-19T07:19:09.178 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-19T07:19:09.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.178+0000 7fec5e1ae640 1 -- 172.21.15.184:0/1285020639 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 msgr2=0x7fec580c1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.178+0000 7fec5e1ae640 1 --2- 172.21.15.184:0/1285020639 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 0x7fec580c1290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec500b0040 tx=0x7fec500195f0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.178+0000 7fec5e1ae640 1 -- 172.21.15.184:0/1285020639 shutdown_connections 2024-09-19T07:19:09.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.178+0000 7fec5e1ae640 1 --2- 172.21.15.184:0/1285020639 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec5814fc90 0x7fec580c1290 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.178+0000 7fec5e1ae640 1 --2- 172.21.15.184:0/1285020639 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec580acf80 0x7fec581560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.178+0000 7fec5e1ae640 1 --2- 172.21.15.184:0/1285020639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec5806bd10 0x7fec581ecae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.178+0000 7fec5e1ae640 1 -- 172.21.15.184:0/1285020639 >> 172.21.15.184:0/1285020639 conn(0x7fec580c84e0 msgr2=0x7fec5814cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:09.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.178+0000 7fec5e1ae640 1 -- 172.21.15.184:0/1285020639 shutdown_connections 2024-09-19T07:19:09.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.178+0000 7fec5e1ae640 1 -- 172.21.15.184:0/1285020639 wait complete. 2024-09-19T07:19:09.196 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 392df97a-5766-45b5-89c5-fe5940e93e02 2 2024-09-19T07:19:09.201 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:09.202+0000 7fd37ab39180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T07:19:09.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.322+0000 7ffb7ec52640 1 Processor -- start 2024-09-19T07:19:09.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.322+0000 7ffb7ec52640 1 -- start start 2024-09-19T07:19:09.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7ec52640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780c3d40 0x7ffb780ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7ec52640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb78074220 con 0x7ffb780c3500 2024-09-19T07:19:09.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7ec52640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb781572b0 con 0x7ffb780685e0 2024-09-19T07:19:09.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7ec52640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ffb780ad510 con 0x7ffb780c3d40 2024-09-19T07:19:09.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7c9ee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780c3d40 0x7ffb780ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7c9ee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780c3d40 0x7ffb780ab0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:53462/0 (socket says 172.21.15.184:53462) 2024-09-19T07:19:09.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb77fff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7ffb780685e0 0x7ffb78156490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:55582/0 (socket says 172.21.15.184:55582) 2024-09-19T07:19:09.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7c9ee640 1 -- 172.21.15.184:0/578284163 learned_addr learned my addr 172.21.15.184:0/578284163 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:09.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7c9ee640 1 -- 172.21.15.184:0/578284163 >> v1:172.21.15.184:6789/0 conn(0x7ffb780685e0 legacy=0x7ffb78156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7c9ee640 1 -- 172.21.15.184:0/578284163 >> v1:172.21.15.83:6789/0 conn(0x7ffb780c3500 legacy=0x7ffb78156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7c9ee640 1 -- 172.21.15.184:0/578284163 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb780ad030 con 0x7ffb780c3d40 2024-09-19T07:19:09.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb777fe640 1 -- 172.21.15.184:0/578284163 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4077885240 0 0) 0x7ffb781572b0 con 0x7ffb780685e0 2024-09-19T07:19:09.326 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7c9ee640 1 --2- 172.21.15.184:0/578284163 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780c3d40 0x7ffb780ab0e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000d180 tx=0x7ffb7001eb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46e936e8d684280a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:09.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb777fe640 1 -- 172.21.15.184:0/578284163 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7000b2d0 con 0x7ffb780c3d40 2024-09-19T07:19:09.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb777fe640 1 -- 172.21.15.184:0/578284163 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb7000b470 con 0x7ffb780c3d40 2024-09-19T07:19:09.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb777fe640 1 -- 172.21.15.184:0/578284163 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb700a7a10 con 0x7ffb780c3d40 2024-09-19T07:19:09.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7ec52640 1 -- 172.21.15.184:0/578284163 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780c3d40 msgr2=0x7ffb780ab0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.326+0000 7ffb7ec52640 1 --2- 172.21.15.184:0/578284163 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780c3d40 0x7ffb780ab0e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000d180 tx=0x7ffb7001eb30 comp rx=0 tx=0).stop 2024-09-19T07:19:09.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 -- 172.21.15.184:0/578284163 shutdown_connections 2024-09-19T07:19:09.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 --2- 172.21.15.184:0/578284163 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780c3d40 0x7ffb780ab0e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 -- 172.21.15.184:0/578284163 >> 172.21.15.184:0/578284163 conn(0x7ffb780c7b70 msgr2=0x7ffb780c7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:09.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 -- 172.21.15.184:0/578284163 shutdown_connections 2024-09-19T07:19:09.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 -- 172.21.15.184:0/578284163 wait complete. 2024-09-19T07:19:09.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 Processor -- start 2024-09-19T07:19:09.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 -- start start 2024-09-19T07:19:09.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780685e0 0x7ffb7806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb780c3500 0x7ffb781ee100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb780c3d40 0x7ffb781ee640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffb780c42a0 con 0x7ffb780c3500 2024-09-19T07:19:09.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffb780c4120 con 0x7ffb780c3d40 2024-09-19T07:19:09.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ffb780c6280 con 0x7ffb780685e0 2024-09-19T07:19:09.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7c9ee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780685e0 0x7ffb7806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7c9ee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780685e0 0x7ffb7806ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:53476/0 (socket says 172.21.15.184:53476) 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7c9ee640 1 -- 172.21.15.184:0/1557839618 learned_addr learned my addr 172.21.15.184:0/1557839618 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7d1ef640 1 --2- 172.21.15.184:0/1557839618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb780c3d40 0x7ffb781ee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb77fff640 1 --2- 172.21.15.184:0/1557839618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb780c3500 0x7ffb781ee100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7c9ee640 1 -- 172.21.15.184:0/1557839618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb780c3d40 msgr2=0x7ffb781ee640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7c9ee640 1 --2- 172.21.15.184:0/1557839618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb780c3d40 0x7ffb781ee640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7c9ee640 1 -- 172.21.15.184:0/1557839618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb780c3500 msgr2=0x7ffb781ee100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7c9ee640 1 --2- 172.21.15.184:0/1557839618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb780c3500 0x7ffb781ee100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7c9ee640 1 -- 172.21.15.184:0/1557839618 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb780729f0 con 0x7ffb780685e0 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb77fff640 1 --2- 172.21.15.184:0/1557839618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb780c3500 0x7ffb781ee100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7d1ef640 1 --2- 172.21.15.184:0/1557839618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb780c3d40 0x7ffb781ee640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7c9ee640 1 --2- 172.21.15.184:0/1557839618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780685e0 0x7ffb7806ed70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffb700b1040 tx=0x7ffb7001b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:09.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb757fa640 1 -- 172.21.15.184:0/1557839618 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb700be020 con 0x7ffb780685e0 2024-09-19T07:19:09.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb757fa640 1 -- 172.21.15.184:0/1557839618 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb70009800 con 0x7ffb780685e0 2024-09-19T07:19:09.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb757fa640 1 -- 172.21.15.184:0/1557839618 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb70009bd0 con 0x7ffb780685e0 2024-09-19T07:19:09.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 -- 172.21.15.184:0/1557839618 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb7814a2d0 con 0x7ffb780685e0 2024-09-19T07:19:09.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 -- 172.21.15.184:0/1557839618 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb78154870 con 0x7ffb780685e0 2024-09-19T07:19:09.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb757fa640 1 -- 172.21.15.184:0/1557839618 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffb700c1020 con 0x7ffb780685e0 2024-09-19T07:19:09.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb757fa640 1 -- 172.21.15.184:0/1557839618 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7ffb7000b330 con 0x7ffb780685e0 2024-09-19T07:19:09.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.330+0000 7ffb7ec52640 1 -- 172.21.15.184:0/1557839618 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb780c6610 con 0x7ffb780685e0 2024-09-19T07:19:09.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.334+0000 7ffb757fa640 1 -- 172.21.15.184:0/1557839618 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7ffb700c8030 con 0x7ffb780685e0 2024-09-19T07:19:09.402 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.402+0000 7ffb7ec52640 1 -- 172.21.15.184:0/1557839618 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd new", "uuid": "392df97a-5766-45b5-89c5-fe5940e93e02", "id": 2} v 0) -- 0x7ffb78151ed0 con 0x7ffb780685e0 2024-09-19T07:19:09.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.410+0000 7ffb757fa640 1 -- 172.21.15.184:0/1557839618 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "392df97a-5766-45b5-89c5-fe5940e93e02", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7ffb70018840 con 0x7ffb780685e0 2024-09-19T07:19:09.409 INFO:teuthology.orchestra.run.smithi184.stdout:2 2024-09-19T07:19:09.410 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.410+0000 7ffb7ec52640 1 -- 172.21.15.184:0/1557839618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780685e0 msgr2=0x7ffb7806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.410 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.410+0000 7ffb7ec52640 1 --2- 172.21.15.184:0/1557839618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780685e0 0x7ffb7806ed70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffb700b1040 tx=0x7ffb7001b8d0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.410 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.410+0000 7ffb7ec52640 1 -- 172.21.15.184:0/1557839618 shutdown_connections 2024-09-19T07:19:09.410 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.410+0000 7ffb7ec52640 1 --2- 172.21.15.184:0/1557839618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb780c3d40 0x7ffb781ee640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.411 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.410+0000 7ffb7ec52640 1 --2- 172.21.15.184:0/1557839618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb780c3500 0x7ffb781ee100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.411 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.410+0000 7ffb7ec52640 1 --2- 172.21.15.184:0/1557839618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb780685e0 0x7ffb7806ed70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.411 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.410+0000 7ffb7ec52640 1 -- 172.21.15.184:0/1557839618 >> 172.21.15.184:0/1557839618 conn(0x7ffb780c7b70 msgr2=0x7ffb7815da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:09.411 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.410+0000 7ffb7ec52640 1 -- 172.21.15.184:0/1557839618 shutdown_connections 2024-09-19T07:19:09.411 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.410+0000 7ffb7ec52640 1 -- 172.21.15.184:0/1557839618 wait complete. 2024-09-19T07:19:09.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 3934cb42-b02b-41c9-ba7d-0c1d23652e5a 3 2024-09-19T07:19:09.542 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.542+0000 7f7e7391e640 1 Processor -- start 2024-09-19T07:19:09.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.542+0000 7f7e7391e640 1 -- start start 2024-09-19T07:19:09.544 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c0acf80 0x7f7e6c0ad360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.544 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7e6c14fc90 0x7f7e6c1500b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.544 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e6c074080 con 0x7f7e6c06bd10 2024-09-19T07:19:09.544 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7e6c153310 con 0x7f7e6c14fc90 2024-09-19T07:19:09.544 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6c070220 con 0x7f7e6c0acf80 2024-09-19T07:19:09.544 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e716ba640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c0acf80 0x7f7e6c0ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.544 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e716ba640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c0acf80 0x7f7e6c0ad360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:53484/0 (socket says 172.21.15.184:53484) 2024-09-19T07:19:09.545 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e716ba640 1 -- 172.21.15.184:0/983164807 learned_addr learned my addr 172.21.15.184:0/983164807 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:09.545 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e716ba640 1 -- 172.21.15.184:0/983164807 >> v2:172.21.15.83:3300/0 conn(0x7f7e6c14fc90 msgr2=0x7f7e6c1500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:09.545 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e716ba640 1 --2- 172.21.15.184:0/983164807 >> v2:172.21.15.83:3300/0 conn(0x7f7e6c14fc90 0x7f7e6c1500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.545 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e716ba640 1 -- 172.21.15.184:0/983164807 >> v1:172.21.15.83:6789/0 conn(0x7f7e6c06bd10 legacy=0x7f7e6c06c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:09.545 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e716ba640 1 -- 172.21.15.184:0/983164807 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c153560 con 0x7f7e6c0acf80 2024-09-19T07:19:09.546 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e716ba640 1 --2- 172.21.15.184:0/983164807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c0acf80 0x7f7e6c0ad360 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009870 tx=0x7f7e6401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe4da7bb3f4ac27 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:09.546 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e63fff640 1 -- 172.21.15.184:0/983164807 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e640b0070 con 0x7f7e6c0acf80 2024-09-19T07:19:09.546 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e63fff640 1 -- 172.21.15.184:0/983164807 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e640ab600 con 0x7f7e6c0acf80 2024-09-19T07:19:09.546 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e63fff640 1 -- 172.21.15.184:0/983164807 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e640ab9d0 con 0x7f7e6c0acf80 2024-09-19T07:19:09.547 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 -- 172.21.15.184:0/983164807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c0acf80 msgr2=0x7f7e6c0ad360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.547 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 --2- 172.21.15.184:0/983164807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c0acf80 0x7f7e6c0ad360 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009870 tx=0x7f7e6401c400 comp rx=0 tx=0).stop 2024-09-19T07:19:09.547 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 -- 172.21.15.184:0/983164807 shutdown_connections 2024-09-19T07:19:09.547 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 --2- 172.21.15.184:0/983164807 >> v2:172.21.15.83:3300/0 conn(0x7f7e6c14fc90 0x7f7e6c1500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.547 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 --2- 172.21.15.184:0/983164807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c0acf80 0x7f7e6c0ad360 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.547 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 -- 172.21.15.184:0/983164807 >> 172.21.15.184:0/983164807 conn(0x7f7e6c0c84e0 msgr2=0x7f7e6c0c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:09.548 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 -- 172.21.15.184:0/983164807 shutdown_connections 2024-09-19T07:19:09.548 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.546+0000 7f7e7391e640 1 -- 172.21.15.184:0/983164807 wait complete. 2024-09-19T07:19:09.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e7391e640 1 Processor -- start 2024-09-19T07:19:09.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e7391e640 1 -- start start 2024-09-19T07:19:09.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e7391e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c06bd10 0x7f7e6c0bdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e7391e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e6c0acf80 0x7f7e6c14e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e7391e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e6c14fc90 0x7f7e6c154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e7391e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c0a9fd0 con 0x7f7e6c14fc90 2024-09-19T07:19:09.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e7391e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c0a9e50 con 0x7f7e6c0acf80 2024-09-19T07:19:09.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e7391e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6c15fa30 con 0x7f7e6c06bd10 2024-09-19T07:19:09.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e716ba640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c06bd10 0x7f7e6c0bdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e716ba640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c06bd10 0x7f7e6c0bdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:53498/0 (socket says 172.21.15.184:53498) 2024-09-19T07:19:09.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e716ba640 1 -- 172.21.15.184:0/542974103 learned_addr learned my addr 172.21.15.184:0/542974103 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:09.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e70eb9640 1 --2- 172.21.15.184:0/542974103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e6c0acf80 0x7f7e6c14e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e716ba640 1 -- 172.21.15.184:0/542974103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e6c0acf80 msgr2=0x7f7e6c14e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e716ba640 1 --2- 172.21.15.184:0/542974103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e6c0acf80 0x7f7e6c14e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e716ba640 1 -- 172.21.15.184:0/542974103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e6c14fc90 msgr2=0x7f7e6c154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e71ebb640 1 --2- 172.21.15.184:0/542974103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e6c14fc90 0x7f7e6c154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e716ba640 1 --2- 172.21.15.184:0/542974103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e6c14fc90 0x7f7e6c154f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e716ba640 1 -- 172.21.15.184:0/542974103 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c156af0 con 0x7f7e6c06bd10 2024-09-19T07:19:09.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e71ebb640 1 --2- 172.21.15.184:0/542974103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e6c14fc90 0x7f7e6c154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:09.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e716ba640 1 --2- 172.21.15.184:0/542974103 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c06bd10 0x7f7e6c0bdd90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7e640ae040 tx=0x7f7e6401ac40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:09.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e61ffb640 1 -- 172.21.15.184:0/542974103 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e640ba020 con 0x7f7e6c06bd10 2024-09-19T07:19:09.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e61ffb640 1 -- 172.21.15.184:0/542974103 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e640b6640 con 0x7f7e6c06bd10 2024-09-19T07:19:09.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e7391e640 1 -- 172.21.15.184:0/542974103 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e6c1560f0 con 0x7f7e6c06bd10 2024-09-19T07:19:09.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e61ffb640 1 -- 172.21.15.184:0/542974103 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e640b0040 con 0x7f7e6c06bd10 2024-09-19T07:19:09.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e7391e640 1 -- 172.21.15.184:0/542974103 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e6c156780 con 0x7f7e6c06bd10 2024-09-19T07:19:09.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e61ffb640 1 -- 172.21.15.184:0/542974103 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7e640bd020 con 0x7f7e6c06bd10 2024-09-19T07:19:09.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e61ffb640 1 -- 172.21.15.184:0/542974103 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f7e640b43f0 con 0x7f7e6c06bd10 2024-09-19T07:19:09.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.550+0000 7f7e7391e640 1 -- 172.21.15.184:0/542974103 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e6c0ad360 con 0x7f7e6c06bd10 2024-09-19T07:19:09.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.554+0000 7f7e61ffb640 1 -- 172.21.15.184:0/542974103 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f7e640c4030 con 0x7f7e6c06bd10 2024-09-19T07:19:09.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.622+0000 7f7e7391e640 1 -- 172.21.15.184:0/542974103 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3934cb42-b02b-41c9-ba7d-0c1d23652e5a", "id": 3} v 0) -- 0x7f7e6c06fbd0 con 0x7f7e6c06bd10 2024-09-19T07:19:09.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.630+0000 7f7e61ffb640 1 -- 172.21.15.184:0/542974103 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3934cb42-b02b-41c9-ba7d-0c1d23652e5a", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f7e64016110 con 0x7f7e6c06bd10 2024-09-19T07:19:09.629 INFO:teuthology.orchestra.run.smithi184.stdout:3 2024-09-19T07:19:09.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.630+0000 7f7e7391e640 1 -- 172.21.15.184:0/542974103 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c06bd10 msgr2=0x7f7e6c0bdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.630+0000 7f7e7391e640 1 --2- 172.21.15.184:0/542974103 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c06bd10 0x7f7e6c0bdd90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7e640ae040 tx=0x7f7e6401ac40 comp rx=0 tx=0).stop 2024-09-19T07:19:09.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.630+0000 7f7e7391e640 1 -- 172.21.15.184:0/542974103 shutdown_connections 2024-09-19T07:19:09.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.630+0000 7f7e7391e640 1 --2- 172.21.15.184:0/542974103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e6c14fc90 0x7f7e6c154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.630+0000 7f7e7391e640 1 --2- 172.21.15.184:0/542974103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e6c0acf80 0x7f7e6c14e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.630+0000 7f7e7391e640 1 --2- 172.21.15.184:0/542974103 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6c06bd10 0x7f7e6c0bdd90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.630+0000 7f7e7391e640 1 -- 172.21.15.184:0/542974103 >> 172.21.15.184:0/542974103 conn(0x7f7e6c0c84e0 msgr2=0x7f7e6c14ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:09.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.630+0000 7f7e7391e640 1 -- 172.21.15.184:0/542974103 shutdown_connections 2024-09-19T07:19:09.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.630+0000 7f7e7391e640 1 -- 172.21.15.184:0/542974103 wait complete. 2024-09-19T07:19:09.633 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:09.632+0000 7fb53e71f180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T07:19:09.646 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 154ece4b-f44e-4b13-8370-c689d47887bf 4 2024-09-19T07:19:09.686 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:09.684+0000 7fb53e71f180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T07:19:09.738 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:09.736+0000 7fb53e71f180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T07:19:09.766 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.766+0000 7f70923c7640 1 Processor -- start 2024-09-19T07:19:09.766 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.766+0000 7f70923c7640 1 -- start start 2024-09-19T07:19:09.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.766+0000 7f70923c7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f708c0685e0 0x7f708c0c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.766+0000 7f70923c7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f708c074220 con 0x7f708c0c2a30 2024-09-19T07:19:09.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.766+0000 7f70923c7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f708c0ad510 con 0x7f708c0c3d40 2024-09-19T07:19:09.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.766+0000 7f70923c7640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f708c0ad030 con 0x7f708c0685e0 2024-09-19T07:19:09.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.766+0000 7f708b7fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f708c0685e0 0x7f708c0c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.766+0000 7f708bfff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f708c0c3d40 0x7f708c0ab0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:55598/0 (socket says 172.21.15.184:55598) 2024-09-19T07:19:09.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.766+0000 7f708bfff640 1 -- 172.21.15.184:0/3121822718 learned_addr learned my addr 172.21.15.184:0/3121822718 (peer_addr_for_me v1:172.21.15.184:0/0) 2024-09-19T07:19:09.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f708b7fe640 1 -- 172.21.15.184:0/3121822718 >> v1:172.21.15.184:6789/0 conn(0x7f708c0c3d40 legacy=0x7f708c0ab0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f708b7fe640 1 -- 172.21.15.184:0/3121822718 >> v1:172.21.15.83:6789/0 conn(0x7f708c0c2a30 legacy=0x7f708c0c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f708affd640 1 -- 172.21.15.184:0/3121822718 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006177474 0 0) 0x7f708c0ad510 con 0x7f708c0c3d40 2024-09-19T07:19:09.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f708b7fe640 1 -- 172.21.15.184:0/3121822718 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708c0c8830 con 0x7f708c0685e0 2024-09-19T07:19:09.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f708b7fe640 1 --2- 172.21.15.184:0/3121822718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f708c0685e0 0x7f708c0c24f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f708c0ad210 tx=0x7f708001e590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d063c363c9d60d27 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:09.770 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f708affd640 1 -- 172.21.15.184:0/3121822718 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70800b2070 con 0x7f708c0685e0 2024-09-19T07:19:09.770 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f708affd640 1 -- 172.21.15.184:0/3121822718 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70800ad600 con 0x7f708c0685e0 2024-09-19T07:19:09.770 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f708affd640 1 -- 172.21.15.184:0/3121822718 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70800ad9d0 con 0x7f708c0685e0 2024-09-19T07:19:09.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f70923c7640 1 -- 172.21.15.184:0/3121822718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f708c0685e0 msgr2=0x7f708c0c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f70923c7640 1 --2- 172.21.15.184:0/3121822718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f708c0685e0 0x7f708c0c24f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f708c0ad210 tx=0x7f708001e590 comp rx=0 tx=0).stop 2024-09-19T07:19:09.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f70923c7640 1 -- 172.21.15.184:0/3121822718 shutdown_connections 2024-09-19T07:19:09.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f70923c7640 1 --2- 172.21.15.184:0/3121822718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f708c0685e0 0x7f708c0c24f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f70923c7640 1 -- 172.21.15.184:0/3121822718 >> 172.21.15.184:0/3121822718 conn(0x7f708c0c7b70 msgr2=0x7f708c0c7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:09.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f70923c7640 1 -- 172.21.15.184:0/3121822718 shutdown_connections 2024-09-19T07:19:09.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.770+0000 7f70923c7640 1 -- 172.21.15.184:0/3121822718 wait complete. 2024-09-19T07:19:09.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f70923c7640 1 Processor -- start 2024-09-19T07:19:09.772 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f70923c7640 1 -- start start 2024-09-19T07:19:09.774 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f70923c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f708c0685e0 0x7f708c0a8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.774 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f70923c7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f708c0c2a30 0x7f708c06dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.774 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f70923c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f708c0c3d40 0x7f708c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.774 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f70923c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f708c0c42a0 con 0x7f708c0685e0 2024-09-19T07:19:09.774 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f70923c7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f708c0c4120 con 0x7f708c0c3d40 2024-09-19T07:19:09.774 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f70923c7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f708c0c6280 con 0x7f708c0c2a30 2024-09-19T07:19:09.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f708b7fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f708c0c2a30 0x7f708c06dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f7090964640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f708c0c3d40 0x7f708c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f708b7fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f708c0c2a30 0x7f708c06dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:53508/0 (socket says 172.21.15.184:53508) 2024-09-19T07:19:09.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f708b7fe640 1 -- 172.21.15.184:0/3490351947 learned_addr learned my addr 172.21.15.184:0/3490351947 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:09.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f708bfff640 1 --2- 172.21.15.184:0/3490351947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f708c0685e0 0x7f708c0a8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f708b7fe640 1 -- 172.21.15.184:0/3490351947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f708c0c3d40 msgr2=0x7f708c06e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f708b7fe640 1 --2- 172.21.15.184:0/3490351947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f708c0c3d40 0x7f708c06e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f708b7fe640 1 -- 172.21.15.184:0/3490351947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f708c0685e0 msgr2=0x7f708c0a8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f708b7fe640 1 --2- 172.21.15.184:0/3490351947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f708c0685e0 0x7f708c0a8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f708b7fe640 1 -- 172.21.15.184:0/3490351947 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708c14a0f0 con 0x7f708c0c2a30 2024-09-19T07:19:09.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f7090964640 1 --2- 172.21.15.184:0/3490351947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f708c0c3d40 0x7f708c06e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:09.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f708b7fe640 1 --2- 172.21.15.184:0/3490351947 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f708c0c2a30 0x7f708c06dfc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f70800b0040 tx=0x7f708001cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:09.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f7088ff9640 1 -- 172.21.15.184:0/3490351947 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70800bd070 con 0x7f708c0c2a30 2024-09-19T07:19:09.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f7088ff9640 1 -- 172.21.15.184:0/3490351947 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70800b8430 con 0x7f708c0c2a30 2024-09-19T07:19:09.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f70923c7640 1 -- 172.21.15.184:0/3490351947 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f708c072ea0 con 0x7f708c0c2a30 2024-09-19T07:19:09.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f7088ff9640 1 -- 172.21.15.184:0/3490351947 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70800b2040 con 0x7f708c0c2a30 2024-09-19T07:19:09.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f70923c7640 1 -- 172.21.15.184:0/3490351947 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f708c0721b0 con 0x7f708c0c2a30 2024-09-19T07:19:09.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f7088ff9640 1 -- 172.21.15.184:0/3490351947 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f70800c0070 con 0x7f708c0c2a30 2024-09-19T07:19:09.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f7088ff9640 1 -- 172.21.15.184:0/3490351947 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f70800b78e0 con 0x7f708c0c2a30 2024-09-19T07:19:09.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.774+0000 7f70923c7640 1 -- 172.21.15.184:0/3490351947 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f708c0c6610 con 0x7f708c0c2a30 2024-09-19T07:19:09.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.778+0000 7f7088ff9640 1 -- 172.21.15.184:0/3490351947 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f70800c7070 con 0x7f708c0c2a30 2024-09-19T07:19:09.785 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:09.786+0000 7fd37ab39180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T07:19:09.814 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:09.812+0000 7fb53e71f180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T07:19:09.839 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:09.838+0000 7fd37ab39180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T07:19:09.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.846+0000 7f70923c7640 1 -- 172.21.15.184:0/3490351947 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd new", "uuid": "154ece4b-f44e-4b13-8370-c689d47887bf", "id": 4} v 0) -- 0x7f708c06fd90 con 0x7f708c0c2a30 2024-09-19T07:19:09.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.854+0000 7f7088ff9640 1 -- 172.21.15.184:0/3490351947 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "154ece4b-f44e-4b13-8370-c689d47887bf", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f70800182a0 con 0x7f708c0c2a30 2024-09-19T07:19:09.852 INFO:teuthology.orchestra.run.smithi184.stdout:4 2024-09-19T07:19:09.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.854+0000 7f70923c7640 1 -- 172.21.15.184:0/3490351947 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f708c0c2a30 msgr2=0x7f708c06dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.854+0000 7f70923c7640 1 --2- 172.21.15.184:0/3490351947 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f708c0c2a30 0x7f708c06dfc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f70800b0040 tx=0x7f708001cdd0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.854+0000 7f70923c7640 1 -- 172.21.15.184:0/3490351947 shutdown_connections 2024-09-19T07:19:09.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.854+0000 7f70923c7640 1 --2- 172.21.15.184:0/3490351947 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f708c0c3d40 0x7f708c06e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.854+0000 7f70923c7640 1 --2- 172.21.15.184:0/3490351947 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f708c0c2a30 0x7f708c06dfc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.854+0000 7f70923c7640 1 --2- 172.21.15.184:0/3490351947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f708c0685e0 0x7f708c0a8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.854+0000 7f70923c7640 1 -- 172.21.15.184:0/3490351947 >> 172.21.15.184:0/3490351947 conn(0x7f708c0c7b70 msgr2=0x7f708c0c4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:09.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.854+0000 7f70923c7640 1 -- 172.21.15.184:0/3490351947 shutdown_connections 2024-09-19T07:19:09.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.854+0000 7f70923c7640 1 -- 172.21.15.184:0/3490351947 wait complete. 2024-09-19T07:19:09.866 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:09.864+0000 7fb53e71f180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T07:19:09.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new efd87884-f2b4-4edc-b9ba-eb6b9f636f8b 5 2024-09-19T07:19:09.895 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:09.894+0000 7fd37ab39180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T07:19:09.943 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:09.940+0000 7fb53e71f180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T07:19:09.974 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:09.974+0000 7fd37ab39180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T07:19:09.987 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.986+0000 7f8485e37640 1 Processor -- start 2024-09-19T07:19:09.987 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.986+0000 7f8485e37640 1 -- start start 2024-09-19T07:19:09.989 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f8485e37640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84800acf80 0x7f84800ad360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.989 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f8485e37640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f848014fc90 0x7f84801500e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.989 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f8485e37640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8480074080 con 0x7f848006bd10 2024-09-19T07:19:09.990 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f8485e37640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f8480153310 con 0x7f84800acf80 2024-09-19T07:19:09.990 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f8485e37640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f8480070220 con 0x7f848014fc90 2024-09-19T07:19:09.990 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f8476ffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f848014fc90 0x7f84801500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.990 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f8476ffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f848014fc90 0x7f84801500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:37076/0 (socket says 172.21.15.184:37076) 2024-09-19T07:19:09.990 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f8476ffd640 1 -- 172.21.15.184:0/1207016214 learned_addr learned my addr 172.21.15.184:0/1207016214 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:09.990 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f847f7fe640 1 --2- 172.21.15.184:0/1207016214 >> v2:172.21.15.83:3300/0 conn(0x7f84800acf80 0x7f84800ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.990 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f8476ffd640 1 -- 172.21.15.184:0/1207016214 >> v2:172.21.15.83:3300/0 conn(0x7f84800acf80 msgr2=0x7f84800ad360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:09.990 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f8476ffd640 1 --2- 172.21.15.184:0/1207016214 >> v2:172.21.15.83:3300/0 conn(0x7f84800acf80 0x7f84800ad360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.990 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f8476ffd640 1 -- 172.21.15.184:0/1207016214 >> v1:172.21.15.83:6789/0 conn(0x7f848006bd10 legacy=0x7f848006c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:09.990 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.990+0000 7f8476ffd640 1 -- 172.21.15.184:0/1207016214 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8480153560 con 0x7f848014fc90 2024-09-19T07:19:09.991 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8476ffd640 1 --2- 172.21.15.184:0/1207016214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f848014fc90 0x7f84801500e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8480070020 tx=0x7f846801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3145d352f95344d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:09.992 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f847effd640 1 -- 172.21.15.184:0/1207016214 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84680b0070 con 0x7f848014fc90 2024-09-19T07:19:09.992 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f847effd640 1 -- 172.21.15.184:0/1207016214 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84680a4e50 con 0x7f848014fc90 2024-09-19T07:19:09.992 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f847effd640 1 -- 172.21.15.184:0/1207016214 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84680ab820 con 0x7f848014fc90 2024-09-19T07:19:09.993 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 -- 172.21.15.184:0/1207016214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f848014fc90 msgr2=0x7f84801500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.993 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 --2- 172.21.15.184:0/1207016214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f848014fc90 0x7f84801500e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8480070020 tx=0x7f846801c3c0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.993 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 -- 172.21.15.184:0/1207016214 shutdown_connections 2024-09-19T07:19:09.993 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 --2- 172.21.15.184:0/1207016214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f848014fc90 0x7f84801500e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.993 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 --2- 172.21.15.184:0/1207016214 >> v2:172.21.15.83:3300/0 conn(0x7f84800acf80 0x7f84800ad360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.993 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 -- 172.21.15.184:0/1207016214 >> 172.21.15.184:0/1207016214 conn(0x7f84800c84e0 msgr2=0x7f84800c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:09.993 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 -- 172.21.15.184:0/1207016214 shutdown_connections 2024-09-19T07:19:09.993 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 -- 172.21.15.184:0/1207016214 wait complete. 2024-09-19T07:19:09.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 Processor -- start 2024-09-19T07:19:09.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 -- start start 2024-09-19T07:19:09.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f848006bd10 0x7f84801eccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84800acf80 0x7f84800bd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f848014fc90 0x7f8480156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:09.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f84800a9fd0 con 0x7f848006bd10 2024-09-19T07:19:09.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f84800a9e50 con 0x7f848014fc90 2024-09-19T07:19:09.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f848015fa30 con 0x7f84800acf80 2024-09-19T07:19:09.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f847ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f848014fc90 0x7f8480156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8476ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84800acf80 0x7f84800bd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f847ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f848014fc90 0x7f8480156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:37078/0 (socket says 172.21.15.184:37078) 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8476ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84800acf80 0x7f84800bd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:53524/0 (socket says 172.21.15.184:53524) 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f847ffff640 1 -- 172.21.15.184:0/3737624949 learned_addr learned my addr 172.21.15.184:0/3737624949 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f847f7fe640 1 --2- 172.21.15.184:0/3737624949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f848006bd10 0x7f84801eccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8476ffd640 1 -- 172.21.15.184:0/3737624949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f848014fc90 msgr2=0x7f8480156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8476ffd640 1 --2- 172.21.15.184:0/3737624949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f848014fc90 0x7f8480156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8476ffd640 1 -- 172.21.15.184:0/3737624949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f848006bd10 msgr2=0x7f84801eccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8476ffd640 1 --2- 172.21.15.184:0/3737624949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f848006bd10 0x7f84801eccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8476ffd640 1 -- 172.21.15.184:0/3737624949 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84800bf910 con 0x7f84800acf80 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f847ffff640 1 --2- 172.21.15.184:0/3737624949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f848014fc90 0x7f8480156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8476ffd640 1 --2- 172.21.15.184:0/3737624949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84800acf80 0x7f84800bd6e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f84680ab000 tx=0x7f8468019160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f847cff9640 1 -- 172.21.15.184:0/3737624949 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84680b0040 con 0x7f84800acf80 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f847cff9640 1 -- 172.21.15.184:0/3737624949 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84680b4640 con 0x7f84800acf80 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 -- 172.21.15.184:0/3737624949 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84800be170 con 0x7f84800acf80 2024-09-19T07:19:09.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f847cff9640 1 -- 172.21.15.184:0/3737624949 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84680b4a30 con 0x7f84800acf80 2024-09-19T07:19:09.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.994+0000 7f8485e37640 1 -- 172.21.15.184:0/3737624949 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84800be730 con 0x7f84800acf80 2024-09-19T07:19:09.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.998+0000 7f847cff9640 1 -- 172.21.15.184:0/3737624949 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f84680b4bd0 con 0x7f84800acf80 2024-09-19T07:19:09.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.998+0000 7f847cff9640 1 -- 172.21.15.184:0/3737624949 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f84680bdbd0 con 0x7f84800acf80 2024-09-19T07:19:09.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:09.998+0000 7f8485e37640 1 -- 172.21.15.184:0/3737624949 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84800ad360 con 0x7f84800acf80 2024-09-19T07:19:09.998 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:09.992+0000 7fb53e71f180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T07:19:10.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.002+0000 7f847cff9640 1 -- 172.21.15.184:0/3737624949 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f84680160d0 con 0x7f84800acf80 2024-09-19T07:19:10.031 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:10.030+0000 7fd37ab39180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T07:19:10.069 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.070+0000 7f8485e37640 1 -- 172.21.15.184:0/3737624949 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd new", "uuid": "efd87884-f2b4-4edc-b9ba-eb6b9f636f8b", "id": 5} v 0) -- 0x7f84800c3140 con 0x7f84800acf80 2024-09-19T07:19:10.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.078+0000 7f847cff9640 1 -- 172.21.15.184:0/3737624949 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "efd87884-f2b4-4edc-b9ba-eb6b9f636f8b", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f84680a4df0 con 0x7f84800acf80 2024-09-19T07:19:10.076 INFO:teuthology.orchestra.run.smithi184.stdout:5 2024-09-19T07:19:10.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.078+0000 7f8485e37640 1 -- 172.21.15.184:0/3737624949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84800acf80 msgr2=0x7f84800bd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.078+0000 7f8485e37640 1 --2- 172.21.15.184:0/3737624949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84800acf80 0x7f84800bd6e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f84680ab000 tx=0x7f8468019160 comp rx=0 tx=0).stop 2024-09-19T07:19:10.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.078+0000 7f8485e37640 1 -- 172.21.15.184:0/3737624949 shutdown_connections 2024-09-19T07:19:10.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.078+0000 7f8485e37640 1 --2- 172.21.15.184:0/3737624949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f848014fc90 0x7f8480156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.078+0000 7f8485e37640 1 --2- 172.21.15.184:0/3737624949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84800acf80 0x7f84800bd6e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.078+0000 7f8485e37640 1 --2- 172.21.15.184:0/3737624949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f848006bd10 0x7f84801eccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.078+0000 7f8485e37640 1 -- 172.21.15.184:0/3737624949 >> 172.21.15.184:0/3737624949 conn(0x7f84800c84e0 msgr2=0x7f84800c4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:10.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.078+0000 7f8485e37640 1 -- 172.21.15.184:0/3737624949 shutdown_connections 2024-09-19T07:19:10.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.078+0000 7f8485e37640 1 -- 172.21.15.184:0/3737624949 wait complete. 2024-09-19T07:19:10.092 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 1422f4b2-c4aa-46f9-89d1-38e8a0032b54 6 2024-09-19T07:19:10.110 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:10.110+0000 7fd37ab39180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T07:19:10.145 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:10.144+0000 7fb53e71f180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T07:19:10.165 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:10.166+0000 7fd37ab39180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T07:19:10.206 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:10.204+0000 7fb53e71f180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T07:19:10.227 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.226+0000 7f0c68e35640 1 Processor -- start 2024-09-19T07:19:10.227 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.226+0000 7f0c68e35640 1 -- start start 2024-09-19T07:19:10.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c68e35640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f0c6406d200 0x7f0c64155c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c68e35640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406ed10 0x7f0c641561c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c68e35640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6406f540 con 0x7f0c64153350 2024-09-19T07:19:10.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c68e35640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f0c6406f260 con 0x7f0c6406d200 2024-09-19T07:19:10.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c68e35640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0c6406f850 con 0x7f0c6406ed10 2024-09-19T07:19:10.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c62d9d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406ed10 0x7f0c641561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c5bfff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f0c6406d200 0x7f0c64155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c62d9d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406ed10 0x7f0c641561c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:53528/0 (socket says 172.21.15.184:53528) 2024-09-19T07:19:10.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c62d9d640 1 -- 172.21.15.184:0/1027291187 learned_addr learned my addr 172.21.15.184:0/1027291187 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:10.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c5bfff640 1 -- 172.21.15.184:0/1027291187 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406ed10 msgr2=0x7f0c641561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:10.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c5bfff640 1 --2- 172.21.15.184:0/1027291187 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406ed10 0x7f0c641561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c5bfff640 1 -- 172.21.15.184:0/1027291187 >> v1:172.21.15.83:6789/0 conn(0x7f0c64153350 legacy=0x7f0c64074f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c5bfff640 1 -- 172.21.15.184:0/1027291187 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6406fb30 con 0x7f0c6406d200 2024-09-19T07:19:10.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c62d9d640 1 --2- 172.21.15.184:0/1027291187 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406ed10 0x7f0c641561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:10.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.230+0000 7f0c5bfff640 1 --2- 172.21.15.184:0/1027291187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c6406d200 0x7f0c64155c80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0097c0 tx=0x7f0c4c01c350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cdf511f7f11e70e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:10.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c61d9b640 1 -- 172.21.15.184:0/1027291187 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c0b0070 con 0x7f0c6406d200 2024-09-19T07:19:10.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c61d9b640 1 -- 172.21.15.184:0/1027291187 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c4c004230 con 0x7f0c6406d200 2024-09-19T07:19:10.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c61d9b640 1 -- 172.21.15.184:0/1027291187 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c003620 con 0x7f0c6406d200 2024-09-19T07:19:10.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 -- 172.21.15.184:0/1027291187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c6406d200 msgr2=0x7f0c64155c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 --2- 172.21.15.184:0/1027291187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c6406d200 0x7f0c64155c80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0097c0 tx=0x7f0c4c01c350 comp rx=0 tx=0).stop 2024-09-19T07:19:10.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 -- 172.21.15.184:0/1027291187 shutdown_connections 2024-09-19T07:19:10.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 --2- 172.21.15.184:0/1027291187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c6406d200 0x7f0c64155c80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 --2- 172.21.15.184:0/1027291187 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406ed10 0x7f0c641561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 -- 172.21.15.184:0/1027291187 >> 172.21.15.184:0/1027291187 conn(0x7f0c640c84e0 msgr2=0x7f0c640c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:10.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 -- 172.21.15.184:0/1027291187 shutdown_connections 2024-09-19T07:19:10.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 -- 172.21.15.184:0/1027291187 wait complete. 2024-09-19T07:19:10.236 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 Processor -- start 2024-09-19T07:19:10.236 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 -- start start 2024-09-19T07:19:10.236 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406d200 0x7f0c64154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.236 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6406ed10 0x7f0c641f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.237 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c64153350 0x7f0c641eecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.237 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0c640756a0 con 0x7f0c6406ed10 2024-09-19T07:19:10.237 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0c64070610 con 0x7f0c64153350 2024-09-19T07:19:10.237 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0c64075820 con 0x7f0c6406d200 2024-09-19T07:19:10.237 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c6259c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406d200 0x7f0c64154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.237 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c62d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c64153350 0x7f0c641eecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.237 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c6259c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406d200 0x7f0c64154950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:53532/0 (socket says 172.21.15.184:53532) 2024-09-19T07:19:10.237 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c6259c640 1 -- 172.21.15.184:0/1638054616 learned_addr learned my addr 172.21.15.184:0/1638054616 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:10.237 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c5bfff640 1 --2- 172.21.15.184:0/1638054616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6406ed10 0x7f0c641f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.238 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c6259c640 1 -- 172.21.15.184:0/1638054616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c64153350 msgr2=0x7f0c641eecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.238 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c6259c640 1 --2- 172.21.15.184:0/1638054616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c64153350 0x7f0c641eecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.238 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c6259c640 1 -- 172.21.15.184:0/1638054616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6406ed10 msgr2=0x7f0c641f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.238 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c6259c640 1 --2- 172.21.15.184:0/1638054616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6406ed10 0x7f0c641f19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.238 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c6259c640 1 -- 172.21.15.184:0/1638054616 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c640ac380 con 0x7f0c6406d200 2024-09-19T07:19:10.238 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c62d9d640 1 --2- 172.21.15.184:0/1638054616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c64153350 0x7f0c641eecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:10.238 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c6259c640 1 --2- 172.21.15.184:0/1638054616 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406d200 0x7f0c64154950 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400c420 tx=0x7f0c5400c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:10.238 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c5affd640 1 -- 172.21.15.184:0/1638054616 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c540052d0 con 0x7f0c6406d200 2024-09-19T07:19:10.238 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c5affd640 1 -- 172.21.15.184:0/1638054616 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c54005470 con 0x7f0c6406d200 2024-09-19T07:19:10.238 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 -- 172.21.15.184:0/1638054616 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c641f2200 con 0x7f0c6406d200 2024-09-19T07:19:10.239 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c68e35640 1 -- 172.21.15.184:0/1638054616 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c641f27c0 con 0x7f0c6406d200 2024-09-19T07:19:10.239 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c5affd640 1 -- 172.21.15.184:0/1638054616 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c54007260 con 0x7f0c6406d200 2024-09-19T07:19:10.239 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c5affd640 1 -- 172.21.15.184:0/1638054616 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0c54007480 con 0x7f0c6406d200 2024-09-19T07:19:10.239 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.234+0000 7f0c5affd640 1 -- 172.21.15.184:0/1638054616 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f0c54003b50 con 0x7f0c6406d200 2024-09-19T07:19:10.239 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.238+0000 7f0c68e35640 1 -- 172.21.15.184:0/1638054616 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c480052a0 con 0x7f0c6406d200 2024-09-19T07:19:10.239 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.242+0000 7f0c5affd640 1 -- 172.21.15.184:0/1638054616 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f0c54003d50 con 0x7f0c6406d200 2024-09-19T07:19:10.309 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.310+0000 7f0c68e35640 1 -- 172.21.15.184:0/1638054616 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1422f4b2-c4aa-46f9-89d1-38e8a0032b54", "id": 6} v 0) -- 0x7f0c48005590 con 0x7f0c6406d200 2024-09-19T07:19:10.317 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.318+0000 7f0c5affd640 1 -- 172.21.15.184:0/1638054616 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1422f4b2-c4aa-46f9-89d1-38e8a0032b54", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f0c54003600 con 0x7f0c6406d200 2024-09-19T07:19:10.317 INFO:teuthology.orchestra.run.smithi184.stdout:6 2024-09-19T07:19:10.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.318+0000 7f0c68e35640 1 -- 172.21.15.184:0/1638054616 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406d200 msgr2=0x7f0c64154950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.318+0000 7f0c68e35640 1 --2- 172.21.15.184:0/1638054616 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406d200 0x7f0c64154950 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400c420 tx=0x7f0c5400c8e0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.318+0000 7f0c68e35640 1 -- 172.21.15.184:0/1638054616 shutdown_connections 2024-09-19T07:19:10.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.318+0000 7f0c68e35640 1 --2- 172.21.15.184:0/1638054616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c64153350 0x7f0c641eecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.318+0000 7f0c68e35640 1 --2- 172.21.15.184:0/1638054616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6406ed10 0x7f0c641f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.318+0000 7f0c68e35640 1 --2- 172.21.15.184:0/1638054616 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c6406d200 0x7f0c64154950 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.318+0000 7f0c68e35640 1 -- 172.21.15.184:0/1638054616 >> 172.21.15.184:0/1638054616 conn(0x7f0c640c84e0 msgr2=0x7f0c64074f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:10.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.318+0000 7f0c68e35640 1 -- 172.21.15.184:0/1638054616 shutdown_connections 2024-09-19T07:19:10.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.318+0000 7f0c68e35640 1 -- 172.21.15.184:0/1638054616 wait complete. 2024-09-19T07:19:10.324 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:10.326+0000 7fd37ab39180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T07:19:10.334 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 962fe708-81cb-4e58-be27-f0a8d0ba0df4 7 2024-09-19T07:19:10.389 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:10.390+0000 7fd37ab39180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T07:19:10.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0bb29da640 1 Processor -- start 2024-09-19T07:19:10.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0bb29da640 1 -- start start 2024-09-19T07:19:10.445 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0bb29da640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f0bac0c1120 0x7f0bac0c7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.445 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0bb29da640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0bac068290 0x7f0bac1596d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.445 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0bb29da640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bac0a6cb0 con 0x7f0bac0c1de0 2024-09-19T07:19:10.445 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0bb29da640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f0bac0c3180 con 0x7f0bac068290 2024-09-19T07:19:10.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0bb29da640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f0bac0a7250 con 0x7f0bac0c1120 2024-09-19T07:19:10.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0babfff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f0bac0c1120 0x7f0bac0c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0babfff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f0bac0c1120 0x7f0bac0c7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:37110/0 (socket says 172.21.15.184:37110) 2024-09-19T07:19:10.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0bab7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f0bac0c1de0 0x7f0bac0c1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.184:55612/0 (socket says 172.21.15.184:55612) 2024-09-19T07:19:10.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0babfff640 1 -- 172.21.15.184:0/852934514 learned_addr learned my addr 172.21.15.184:0/852934514 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:10.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0bb0f77640 1 --2- 172.21.15.184:0/852934514 >> v2:172.21.15.83:3300/0 conn(0x7f0bac068290 0x7f0bac1596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0babfff640 1 -- 172.21.15.184:0/852934514 >> v2:172.21.15.83:3300/0 conn(0x7f0bac068290 msgr2=0x7f0bac1596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:10.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0babfff640 1 --2- 172.21.15.184:0/852934514 >> v2:172.21.15.83:3300/0 conn(0x7f0bac068290 0x7f0bac1596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0babfff640 1 -- 172.21.15.184:0/852934514 >> v1:172.21.15.184:6789/0 conn(0x7f0bac0c1de0 legacy=0x7f0bac0c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0babfff640 1 -- 172.21.15.184:0/852934514 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bac074060 con 0x7f0bac0c1120 2024-09-19T07:19:10.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.446+0000 7f0baaffd640 1 -- 172.21.15.184:0/852934514 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1610871091 0 0) 0x7f0bac0a6cb0 con 0x7f0bac0c1de0 2024-09-19T07:19:10.447 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0babfff640 1 --2- 172.21.15.184:0/852934514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bac0c1120 0x7f0bac0c7590 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0696c0 tx=0x7f0b9c01c140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80479c00b989a8c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:10.447 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0baaffd640 1 -- 172.21.15.184:0/852934514 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c0b1070 con 0x7f0bac0c1120 2024-09-19T07:19:10.447 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0baaffd640 1 -- 172.21.15.184:0/852934514 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b9c0ac440 con 0x7f0bac0c1120 2024-09-19T07:19:10.447 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0baaffd640 1 -- 172.21.15.184:0/852934514 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c0ac810 con 0x7f0bac0c1120 2024-09-19T07:19:10.448 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 -- 172.21.15.184:0/852934514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bac0c1120 msgr2=0x7f0bac0c7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.449 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 --2- 172.21.15.184:0/852934514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bac0c1120 0x7f0bac0c7590 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0696c0 tx=0x7f0b9c01c140 comp rx=0 tx=0).stop 2024-09-19T07:19:10.449 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 -- 172.21.15.184:0/852934514 shutdown_connections 2024-09-19T07:19:10.449 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 --2- 172.21.15.184:0/852934514 >> v2:172.21.15.83:3300/0 conn(0x7f0bac068290 0x7f0bac1596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.449 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 --2- 172.21.15.184:0/852934514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bac0c1120 0x7f0bac0c7590 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.449 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 -- 172.21.15.184:0/852934514 >> 172.21.15.184:0/852934514 conn(0x7f0bac0c7b70 msgr2=0x7f0bac0c7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:10.449 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 -- 172.21.15.184:0/852934514 shutdown_connections 2024-09-19T07:19:10.449 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 -- 172.21.15.184:0/852934514 wait complete. 2024-09-19T07:19:10.449 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 Processor -- start 2024-09-19T07:19:10.450 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 -- start start 2024-09-19T07:19:10.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bac068290 0x7f0bac0ac660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0bac0c1120 0x7f0bac1f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bac0c1de0 0x7f0bac1f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac1633c0 con 0x7f0bac0c1de0 2024-09-19T07:19:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac068950 con 0x7f0bac068290 2024-09-19T07:19:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0bac163540 con 0x7f0bac0c1120 2024-09-19T07:19:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0babfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bac068290 0x7f0bac0ac660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0babfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bac068290 0x7f0bac0ac660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:37126/0 (socket says 172.21.15.184:37126) 2024-09-19T07:19:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0babfff640 1 -- 172.21.15.184:0/3712144851 learned_addr learned my addr 172.21.15.184:0/3712144851 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-19T07:19:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bab7fe640 1 --2- 172.21.15.184:0/3712144851 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0bac0c1120 0x7f0bac1f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb0f77640 1 --2- 172.21.15.184:0/3712144851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bac0c1de0 0x7f0bac1f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0babfff640 1 -- 172.21.15.184:0/3712144851 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0bac0c1120 msgr2=0x7f0bac1f1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0babfff640 1 --2- 172.21.15.184:0/3712144851 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0bac0c1120 0x7f0bac1f1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0babfff640 1 -- 172.21.15.184:0/3712144851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bac0c1de0 msgr2=0x7f0bac1f2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0babfff640 1 --2- 172.21.15.184:0/3712144851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bac0c1de0 0x7f0bac1f2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0babfff640 1 -- 172.21.15.184:0/3712144851 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bac0a89e0 con 0x7f0bac068290 2024-09-19T07:19:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb0f77640 1 --2- 172.21.15.184:0/3712144851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bac0c1de0 0x7f0bac1f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0babfff640 1 --2- 172.21.15.184:0/3712144851 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bac068290 0x7f0bac0ac660 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c0af040 tx=0x7f0b9c01a980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0ba8ff9640 1 -- 172.21.15.184:0/3712144851 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c0b1040 con 0x7f0bac068290 2024-09-19T07:19:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0ba8ff9640 1 -- 172.21.15.184:0/3712144851 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b9c004230 con 0x7f0bac068290 2024-09-19T07:19:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0ba8ff9640 1 -- 172.21.15.184:0/3712144851 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c003620 con 0x7f0bac068290 2024-09-19T07:19:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.450+0000 7f0bb29da640 1 -- 172.21.15.184:0/3712144851 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bac0a8cc0 con 0x7f0bac068290 2024-09-19T07:19:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.454+0000 7f0bb29da640 1 -- 172.21.15.184:0/3712144851 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bac0acba0 con 0x7f0bac068290 2024-09-19T07:19:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.454+0000 7f0ba8ff9640 1 -- 172.21.15.184:0/3712144851 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0b9c003b50 con 0x7f0bac068290 2024-09-19T07:19:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.454+0000 7f0ba8ff9640 1 -- 172.21.15.184:0/3712144851 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f0b9c0b7ba0 con 0x7f0bac068290 2024-09-19T07:19:10.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.454+0000 7f0bb29da640 1 -- 172.21.15.184:0/3712144851 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bac1596d0 con 0x7f0bac068290 2024-09-19T07:19:10.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.454+0000 7f0ba8ff9640 1 -- 172.21.15.184:0/3712144851 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f0b9c016110 con 0x7f0bac068290 2024-09-19T07:19:10.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.522+0000 7f0bb29da640 1 -- 172.21.15.184:0/3712144851 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "962fe708-81cb-4e58-be27-f0a8d0ba0df4", "id": 7} v 0) -- 0x7f0bac0c77a0 con 0x7f0bac068290 2024-09-19T07:19:10.532 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.534+0000 7f0ba8ff9640 1 -- 172.21.15.184:0/3712144851 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "962fe708-81cb-4e58-be27-f0a8d0ba0df4", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f0b9c003e40 con 0x7f0bac068290 2024-09-19T07:19:10.532 INFO:teuthology.orchestra.run.smithi184.stdout:7 2024-09-19T07:19:10.533 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.534+0000 7f0bb29da640 1 -- 172.21.15.184:0/3712144851 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bac068290 msgr2=0x7f0bac0ac660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.534+0000 7f0bb29da640 1 --2- 172.21.15.184:0/3712144851 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bac068290 0x7f0bac0ac660 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c0af040 tx=0x7f0b9c01a980 comp rx=0 tx=0).stop 2024-09-19T07:19:10.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.534+0000 7f0bb29da640 1 -- 172.21.15.184:0/3712144851 shutdown_connections 2024-09-19T07:19:10.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.534+0000 7f0bb29da640 1 --2- 172.21.15.184:0/3712144851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bac0c1de0 0x7f0bac1f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.534+0000 7f0bb29da640 1 --2- 172.21.15.184:0/3712144851 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0bac0c1120 0x7f0bac1f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.534+0000 7f0bb29da640 1 --2- 172.21.15.184:0/3712144851 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bac068290 0x7f0bac0ac660 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.534+0000 7f0bb29da640 1 -- 172.21.15.184:0/3712144851 >> 172.21.15.184:0/3712144851 conn(0x7f0bac0c7b70 msgr2=0x7f0bac0c1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:10.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.534+0000 7f0bb29da640 1 -- 172.21.15.184:0/3712144851 shutdown_connections 2024-09-19T07:19:10.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:10.534+0000 7f0bb29da640 1 -- 172.21.15.184:0/3712144851 wait complete. 2024-09-19T07:19:10.549 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-19T07:19:10.550 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-19T07:19:10.552 INFO:tasks.ceph.osd.0:Started 2024-09-19T07:19:10.552 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-19T07:19:10.553 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-19T07:19:10.554 INFO:tasks.ceph.osd.1:Started 2024-09-19T07:19:10.554 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-19T07:19:10.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-19T07:19:10.557 INFO:tasks.ceph.osd.2:Started 2024-09-19T07:19:10.557 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-19T07:19:10.557 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-19T07:19:10.559 INFO:tasks.ceph.osd.3:Started 2024-09-19T07:19:10.559 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-19T07:19:10.559 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-19T07:19:10.561 INFO:tasks.ceph.osd.4:Started 2024-09-19T07:19:10.561 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-19T07:19:10.561 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-19T07:19:10.563 INFO:tasks.ceph.osd.5:Started 2024-09-19T07:19:10.563 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-19T07:19:10.563 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-19T07:19:10.565 INFO:tasks.ceph.osd.6:Started 2024-09-19T07:19:10.565 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-19T07:19:10.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-19T07:19:10.567 INFO:tasks.ceph.osd.7:Started 2024-09-19T07:19:10.568 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:10.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f89df640 1 Processor -- start 2024-09-19T07:19:10.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f89df640 1 -- start start 2024-09-19T07:19:10.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f89df640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc5f414dbe0 0x7fc5f40c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f89df640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5f40c3ab0 0x7fc5f4157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f89df640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc5f40c44e0 0x7fc5f4157550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f89df640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc5f4154700 con 0x7fc5f414dbe0 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f89df640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc5f4073d90 con 0x7fc5f40c44e0 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f89df640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc5f4153b30 con 0x7fc5f40c3ab0 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc5f414dbe0 0x7fc5f40c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc5f414dbe0 0x7fc5f40c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54844/0 (socket says 172.21.15.83:54844) 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f259c640 1 -- 172.21.15.83:0/1784270299 learned_addr learned my addr 172.21.15.83:0/1784270299 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f2d9d640 1 --2- 172.21.15.83:0/1784270299 >> v2:172.21.15.184:3300/0 conn(0x7fc5f40c44e0 0x7fc5f4157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f1d9b640 1 --2- 172.21.15.83:0/1784270299 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5f40c3ab0 0x7fc5f4157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f259c640 1 -- 172.21.15.83:0/1784270299 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5f40c3ab0 msgr2=0x7fc5f4157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f259c640 1 --2- 172.21.15.83:0/1784270299 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5f40c3ab0 0x7fc5f4157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f259c640 1 -- 172.21.15.83:0/1784270299 >> v2:172.21.15.184:3300/0 conn(0x7fc5f40c44e0 msgr2=0x7fc5f4157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f259c640 1 --2- 172.21.15.83:0/1784270299 >> v2:172.21.15.184:3300/0 conn(0x7fc5f40c44e0 0x7fc5f4157550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f259c640 1 -- 172.21.15.83:0/1784270299 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f4153dd0 con 0x7fc5f414dbe0 2024-09-19T07:19:10.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.688+0000 7fc5f1d9b640 1 --2- 172.21.15.83:0/1784270299 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5f40c3ab0 0x7fc5f4157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f259c640 1 --2- 172.21.15.83:0/1784270299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5f414dbe0 0x7fc5f40c3fa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4073c90 tx=0x7fc5dc01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b30ffb8ef1b3ce server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f159a640 1 -- 172.21.15.83:0/1784270299 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc0b2070 con 0x7fc5f414dbe0 2024-09-19T07:19:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f159a640 1 -- 172.21.15.83:0/1784270299 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc0a9070 con 0x7fc5f414dbe0 2024-09-19T07:19:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f159a640 1 -- 172.21.15.83:0/1784270299 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc0ad880 con 0x7fc5f414dbe0 2024-09-19T07:19:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 -- 172.21.15.83:0/1784270299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5f414dbe0 msgr2=0x7fc5f40c3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 --2- 172.21.15.83:0/1784270299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5f414dbe0 0x7fc5f40c3fa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4073c90 tx=0x7fc5dc01c490 comp rx=0 tx=0).stop 2024-09-19T07:19:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 -- 172.21.15.83:0/1784270299 shutdown_connections 2024-09-19T07:19:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 --2- 172.21.15.83:0/1784270299 >> v2:172.21.15.184:3300/0 conn(0x7fc5f40c44e0 0x7fc5f4157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 --2- 172.21.15.83:0/1784270299 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5f40c3ab0 0x7fc5f4157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 --2- 172.21.15.83:0/1784270299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5f414dbe0 0x7fc5f40c3fa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 -- 172.21.15.83:0/1784270299 >> 172.21.15.83:0/1784270299 conn(0x7fc5f40c95a0 msgr2=0x7fc5f40c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:10.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 -- 172.21.15.83:0/1784270299 shutdown_connections 2024-09-19T07:19:10.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 -- 172.21.15.83:0/1784270299 wait complete. 2024-09-19T07:19:10.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 Processor -- start 2024-09-19T07:19:10.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 -- start start 2024-09-19T07:19:10.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5f40c44e0 0x7fc5f40adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5f414dbe0 0x7fc5f40ae400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5f40ac440 0x7fc5f4145990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:10.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f406e820 con 0x7fc5f414dbe0 2024-09-19T07:19:10.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f406e6a0 con 0x7fc5f40ac440 2024-09-19T07:19:10.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f89df640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc5f406e9a0 con 0x7fc5f40c44e0 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5f414dbe0 0x7fc5f40ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5f414dbe0 0x7fc5f40ae400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54848/0 (socket says 172.21.15.83:54848) 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f1d9b640 1 -- 172.21.15.83:0/1294653577 learned_addr learned my addr 172.21.15.83:0/1294653577 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f259c640 1 --2- 172.21.15.83:0/1294653577 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5f40c44e0 0x7fc5f40adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.692+0000 7fc5f2d9d640 1 --2- 172.21.15.83:0/1294653577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5f40ac440 0x7fc5f4145990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5f1d9b640 1 -- 172.21.15.83:0/1294653577 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5f40c44e0 msgr2=0x7fc5f40adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5f1d9b640 1 --2- 172.21.15.83:0/1294653577 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5f40c44e0 0x7fc5f40adb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5f1d9b640 1 -- 172.21.15.83:0/1294653577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5f40ac440 msgr2=0x7fc5f4145990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5f1d9b640 1 --2- 172.21.15.83:0/1294653577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5f40ac440 0x7fc5f4145990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5f1d9b640 1 -- 172.21.15.83:0/1294653577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f41f0ba0 con 0x7fc5f414dbe0 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5f1d9b640 1 --2- 172.21.15.83:0/1294653577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5f414dbe0 0x7fc5f40ae400 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00c7f0 tx=0x7fc5ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5eaffd640 1 -- 172.21.15.83:0/1294653577 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec015070 con 0x7fc5f414dbe0 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5eaffd640 1 -- 172.21.15.83:0/1294653577 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ec007a30 con 0x7fc5f414dbe0 2024-09-19T07:19:10.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5f89df640 1 -- 172.21.15.83:0/1294653577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f41f0e80 con 0x7fc5f414dbe0 2024-09-19T07:19:10.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5eaffd640 1 -- 172.21.15.83:0/1294653577 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec007e00 con 0x7fc5f414dbe0 2024-09-19T07:19:10.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5f89df640 1 -- 172.21.15.83:0/1294653577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5f40ad320 con 0x7fc5f414dbe0 2024-09-19T07:19:10.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5eaffd640 1 -- 172.21.15.83:0/1294653577 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc5ec018070 con 0x7fc5f414dbe0 2024-09-19T07:19:10.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5eaffd640 1 -- 172.21.15.83:0/1294653577 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fc5ec011b60 con 0x7fc5f414dbe0 2024-09-19T07:19:10.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5f89df640 1 -- 172.21.15.83:0/1294653577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5f40c3fa0 con 0x7fc5f414dbe0 2024-09-19T07:19:10.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.696+0000 7fc5eaffd640 1 -- 172.21.15.83:0/1294653577 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fc5ec011490 con 0x7fc5f414dbe0 2024-09-19T07:19:10.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.752+0000 7fc5f89df640 1 -- 172.21.15.83:0/1294653577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc5f4073f80 con 0x7fc5f414dbe0 2024-09-19T07:19:10.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.752+0000 7fc5eaffd640 1 -- 172.21.15.83:0/1294653577 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fc5ec011670 con 0x7fc5f414dbe0 2024-09-19T07:19:10.755 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:10.756 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":10,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:10.527409+0000","last_up_change":"0.000000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:10.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.752+0000 7fc5f89df640 1 -- 172.21.15.83:0/1294653577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5f414dbe0 msgr2=0x7fc5f40ae400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:10.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.752+0000 7fc5f89df640 1 --2- 172.21.15.83:0/1294653577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5f414dbe0 0x7fc5f40ae400 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00c7f0 tx=0x7fc5ec00ccb0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.752+0000 7fc5f89df640 1 -- 172.21.15.83:0/1294653577 shutdown_connections 2024-09-19T07:19:10.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.752+0000 7fc5f89df640 1 --2- 172.21.15.83:0/1294653577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5f40ac440 0x7fc5f4145990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.752+0000 7fc5f89df640 1 --2- 172.21.15.83:0/1294653577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5f414dbe0 0x7fc5f40ae400 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.752+0000 7fc5f89df640 1 --2- 172.21.15.83:0/1294653577 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5f40c44e0 0x7fc5f40adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:10.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.752+0000 7fc5f89df640 1 -- 172.21.15.83:0/1294653577 >> 172.21.15.83:0/1294653577 conn(0x7fc5f40c95a0 msgr2=0x7fc5f40aabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:10.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.752+0000 7fc5f89df640 1 -- 172.21.15.83:0/1294653577 shutdown_connections 2024-09-19T07:19:10.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:10.752+0000 7fc5f89df640 1 -- 172.21.15.83:0/1294653577 wait complete. 2024-09-19T07:19:10.771 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-19T07:19:10.771 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-19T07:19:10.861 INFO:tasks.ceph.mgr.y.smithi083.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-19T07:19:10.861 INFO:tasks.ceph.mgr.y.smithi083.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-19T07:19:10.861 INFO:tasks.ceph.mgr.y.smithi083.stderr: from numpy import show_config as show_numpy_config 2024-09-19T07:19:10.904 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:10.900+0000 7fb53e71f180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T07:19:11.046 INFO:tasks.ceph.mgr.x.smithi184.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-19T07:19:11.046 INFO:tasks.ceph.mgr.x.smithi184.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-19T07:19:11.046 INFO:tasks.ceph.mgr.x.smithi184.stderr: from numpy import show_config as show_numpy_config 2024-09-19T07:19:11.068 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:11.064+0000 7fb53e71f180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:19:11.092 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:11.094+0000 7fd37ab39180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T07:19:11.235 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:11.232+0000 7fb53e71f180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T07:19:11.247 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:11.250+0000 7fd37ab39180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:19:11.306 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:11.304+0000 7fb53e71f180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T07:19:11.427 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:11.426+0000 7fd37ab39180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T07:19:11.454 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:11.452+0000 7fb53e71f180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:19:11.497 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:11.498+0000 7fd37ab39180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T07:19:11.577 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:11.641 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:11.642+0000 7fd37ab39180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T07:19:11.687 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.684+0000 7f82c6eb9640 1 Processor -- start 2024-09-19T07:19:11.688 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.684+0000 7f82c6eb9640 1 -- start start 2024-09-19T07:19:11.689 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.684+0000 7f82c6eb9640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f82c00ab530 0x7f82c0153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:11.689 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.684+0000 7f82c6eb9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82c00c8210 con 0x7f82c00c5130 2024-09-19T07:19:11.689 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.684+0000 7f82c6eb9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82c00ace70 con 0x7f82c0150070 2024-09-19T07:19:11.689 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.684+0000 7f82c6eb9640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f82c00aa050 con 0x7f82c00ab530 2024-09-19T07:19:11.689 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.684+0000 7f82c4c55640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f82c00c5130 0x7f82c014f960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59290/0 (socket says 172.21.15.83:59290) 2024-09-19T07:19:11.689 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.684+0000 7f82c4c55640 1 -- 172.21.15.83:0/120609223 learned_addr learned my addr 172.21.15.83:0/120609223 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:11.689 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.684+0000 7f82c5456640 1 --2- 172.21.15.83:0/120609223 >> v2:172.21.15.184:3300/0 conn(0x7f82c00ab530 0x7f82c0153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:11.690 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82bf7fe640 1 -- 172.21.15.83:0/120609223 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3196413826 0 0) 0x7f82c00c8210 con 0x7f82c00c5130 2024-09-19T07:19:11.690 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82bf7fe640 1 -- 172.21.15.83:0/120609223 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82a8003530 con 0x7f82c00c5130 2024-09-19T07:19:11.690 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82bf7fe640 1 -- 172.21.15.83:0/120609223 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2753552964 0 0) 0x7f82a8003530 con 0x7f82c00c5130 2024-09-19T07:19:11.690 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82c5456640 1 -- 172.21.15.83:0/120609223 >> v1:172.21.15.184:6789/0 conn(0x7f82c0150070 legacy=0x7f82c0152940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:11.690 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82c5456640 1 -- 172.21.15.83:0/120609223 >> v1:172.21.15.83:6789/0 conn(0x7f82c00c5130 legacy=0x7f82c014f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:11.690 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82c5456640 1 -- 172.21.15.83:0/120609223 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c00ad150 con 0x7f82c00ab530 2024-09-19T07:19:11.691 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82c5456640 1 --2- 172.21.15.83:0/120609223 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82c00ab530 0x7f82c0153050 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009d90 tx=0x7f82b801d2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58af2483c2517bb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:11.691 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82bf7fe640 1 -- 172.21.15.83:0/120609223 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b80b2070 con 0x7f82c00ab530 2024-09-19T07:19:11.692 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82bf7fe640 1 -- 172.21.15.83:0/120609223 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82b80a55e0 con 0x7f82c00ab530 2024-09-19T07:19:11.692 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82bf7fe640 1 -- 172.21.15.83:0/120609223 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b4001680 con 0x7f82c00ab530 2024-09-19T07:19:11.692 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82c6eb9640 1 -- 172.21.15.83:0/120609223 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82c00ab530 msgr2=0x7f82c0153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:11.693 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82c6eb9640 1 --2- 172.21.15.83:0/120609223 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82c00ab530 0x7f82c0153050 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009d90 tx=0x7f82b801d2f0 comp rx=0 tx=0).stop 2024-09-19T07:19:11.693 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82c6eb9640 1 -- 172.21.15.83:0/120609223 shutdown_connections 2024-09-19T07:19:11.693 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82c6eb9640 1 --2- 172.21.15.83:0/120609223 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82c00ab530 0x7f82c0153050 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:11.693 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82c6eb9640 1 -- 172.21.15.83:0/120609223 >> 172.21.15.83:0/120609223 conn(0x7f82c006f4d0 msgr2=0x7f82c00c8480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:11.693 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82c6eb9640 1 -- 172.21.15.83:0/120609223 shutdown_connections 2024-09-19T07:19:11.693 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.688+0000 7f82c6eb9640 1 -- 172.21.15.83:0/120609223 wait complete. 2024-09-19T07:19:11.693 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c6eb9640 1 Processor -- start 2024-09-19T07:19:11.694 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c6eb9640 1 -- start start 2024-09-19T07:19:11.696 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c6eb9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82c00ab530 0x7f82c0153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:11.697 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c6eb9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82c00c5130 0x7f82c014c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:11.697 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c6eb9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82c0150070 0x7f82c00be9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:11.697 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c6eb9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f82c00c5690 con 0x7f82c0150070 2024-09-19T07:19:11.697 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c6eb9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f82c00c5510 con 0x7f82c00c5130 2024-09-19T07:19:11.697 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c6eb9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f82c006bbc0 con 0x7f82c00ab530 2024-09-19T07:19:11.697 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c5456640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82c0150070 0x7f82c00be9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:11.697 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c5456640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82c0150070 0x7f82c00be9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54864/0 (socket says 172.21.15.83:54864) 2024-09-19T07:19:11.697 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c5456640 1 -- 172.21.15.83:0/630065204 learned_addr learned my addr 172.21.15.83:0/630065204 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:11.697 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82bffff640 1 --2- 172.21.15.83:0/630065204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82c00c5130 0x7f82c014c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:11.697 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c4c55640 1 --2- 172.21.15.83:0/630065204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82c00ab530 0x7f82c0153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:11.697 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c5456640 1 -- 172.21.15.83:0/630065204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82c00ab530 msgr2=0x7f82c0153d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:11.697 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c5456640 1 --2- 172.21.15.83:0/630065204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82c00ab530 0x7f82c0153d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:11.698 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c5456640 1 -- 172.21.15.83:0/630065204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82c00c5130 msgr2=0x7f82c014c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:11.698 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c5456640 1 --2- 172.21.15.83:0/630065204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82c00c5130 0x7f82c014c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:11.698 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c5456640 1 -- 172.21.15.83:0/630065204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c0156920 con 0x7f82c0150070 2024-09-19T07:19:11.698 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c5456640 1 --2- 172.21.15.83:0/630065204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82c0150070 0x7f82c00be9b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f82b80b0040 tx=0x7f82b8019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:11.698 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82bd7fa640 1 -- 172.21.15.83:0/630065204 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b80bc030 con 0x7f82c0150070 2024-09-19T07:19:11.698 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82bd7fa640 1 -- 172.21.15.83:0/630065204 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82b80039a0 con 0x7f82c0150070 2024-09-19T07:19:11.698 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c6eb9640 1 -- 172.21.15.83:0/630065204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82c0156bc0 con 0x7f82c0150070 2024-09-19T07:19:11.698 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82bd7fa640 1 -- 172.21.15.83:0/630065204 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b80b2040 con 0x7f82c0150070 2024-09-19T07:19:11.698 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c6eb9640 1 -- 172.21.15.83:0/630065204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82c014cc00 con 0x7f82c0150070 2024-09-19T07:19:11.698 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82bd7fa640 1 -- 172.21.15.83:0/630065204 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f82b80bf070 con 0x7f82c0150070 2024-09-19T07:19:11.698 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82bd7fa640 1 -- 172.21.15.83:0/630065204 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f82b80adc10 con 0x7f82c0150070 2024-09-19T07:19:11.698 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.692+0000 7f82c6eb9640 1 -- 172.21.15.83:0/630065204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82c014f960 con 0x7f82c0150070 2024-09-19T07:19:11.699 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.696+0000 7f82bd7fa640 1 -- 172.21.15.83:0/630065204 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f82b8016480 con 0x7f82c0150070 2024-09-19T07:19:11.701 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-09-19T07:19:11.700+0000 7fc9947e77c0 -1 Falling back to public interface 2024-09-19T07:19:11.754 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.752+0000 7f82c6eb9640 1 -- 172.21.15.83:0/630065204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82c0156d90 con 0x7f82c0150070 2024-09-19T07:19:11.754 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.752+0000 7f82bd7fa640 1 -- 172.21.15.83:0/630065204 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f82b80ad440 con 0x7f82c0150070 2024-09-19T07:19:11.755 INFO:teuthology.misc.health.smithi083.stdout: 2024-09-19T07:19:11.755 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:10.527409+0000","last_up_change":"0.000000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.752+0000 7f82c6eb9640 1 -- 172.21.15.83:0/630065204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82c0150070 msgr2=0x7f82c00be9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.752+0000 7f82c6eb9640 1 --2- 172.21.15.83:0/630065204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82c0150070 0x7f82c00be9b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f82b80b0040 tx=0x7f82b8019630 comp rx=0 tx=0).stop 2024-09-19T07:19:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.752+0000 7f82c6eb9640 1 -- 172.21.15.83:0/630065204 shutdown_connections 2024-09-19T07:19:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.752+0000 7f82c6eb9640 1 --2- 172.21.15.83:0/630065204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82c0150070 0x7f82c00be9b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.752+0000 7f82c6eb9640 1 --2- 172.21.15.83:0/630065204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82c00c5130 0x7f82c014c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.752+0000 7f82c6eb9640 1 --2- 172.21.15.83:0/630065204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82c00ab530 0x7f82c0153d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.752+0000 7f82c6eb9640 1 -- 172.21.15.83:0/630065204 >> 172.21.15.83:0/630065204 conn(0x7f82c006f4d0 msgr2=0x7f82c014da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.752+0000 7f82c6eb9640 1 -- 172.21.15.83:0/630065204 shutdown_connections 2024-09-19T07:19:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:11.752+0000 7f82c6eb9640 1 -- 172.21.15.83:0/630065204 wait complete. 2024-09-19T07:19:11.769 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-19T07:19:11.993 INFO:tasks.ceph.osd.6.smithi184.stderr:2024-09-19T07:19:11.994+0000 7f48121507c0 -1 Falling back to public interface 2024-09-19T07:19:12.022 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-09-19T07:19:12.020+0000 7f6f102c47c0 -1 Falling back to public interface 2024-09-19T07:19:12.091 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:12.088+0000 7fb53e71f180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T07:19:12.166 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:12.164+0000 7fb53e71f180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T07:19:12.195 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-09-19T07:19:12.192+0000 7f459d5757c0 -1 Falling back to public interface 2024-09-19T07:19:12.221 INFO:tasks.ceph.osd.4.smithi184.stderr:2024-09-19T07:19:12.222+0000 7f1925b337c0 -1 Falling back to public interface 2024-09-19T07:19:12.247 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:12.246+0000 7fd37ab39180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T07:19:12.322 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:12.322+0000 7fd37ab39180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T07:19:12.351 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:12.348+0000 7fb53e71f180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T07:19:12.424 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:12.420+0000 7fb53e71f180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T07:19:12.459 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-09-19T07:19:12.456+0000 7ff5f25bc7c0 -1 Falling back to public interface 2024-09-19T07:19:12.504 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:12.506+0000 7fd37ab39180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T07:19:12.577 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:12.578+0000 7fd37ab39180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T07:19:12.700 INFO:tasks.ceph.osd.7.smithi184.stderr:2024-09-19T07:19:12.702+0000 7f736035a7c0 -1 Falling back to public interface 2024-09-19T07:19:12.701 INFO:tasks.ceph.osd.5.smithi184.stderr:2024-09-19T07:19:12.702+0000 7ff8d34467c0 -1 Falling back to public interface 2024-09-19T07:19:12.716 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:12.712+0000 7fb53e71f180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T07:19:12.778 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:12.776+0000 7fb53e71f180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T07:19:12.876 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:12.878+0000 7fd37ab39180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T07:19:12.938 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-19T07:19:12.938+0000 7fd37ab39180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T07:19:13.821 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:13.816+0000 7fb4e5358640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-19T07:19:13.821 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-09-19T07:19:13.816+0000 7fb4e5358640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-19T07:19:16.669 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-09-19T07:19:16.664+0000 7ff5f25bc7c0 -1 osd.3 0 log_to_monitors true 2024-09-19T07:19:16.699 INFO:tasks.ceph.osd.7.smithi184.stderr:2024-09-19T07:19:16.698+0000 7f736035a7c0 -1 osd.7 0 log_to_monitors true 2024-09-19T07:19:16.710 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-09-19T07:19:16.708+0000 7f459d5757c0 -1 osd.2 0 log_to_monitors true 2024-09-19T07:19:16.714 INFO:tasks.ceph.osd.6.smithi184.stderr:2024-09-19T07:19:16.714+0000 7f48121507c0 -1 osd.6 0 log_to_monitors true 2024-09-19T07:19:16.738 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-09-19T07:19:16.736+0000 7fc9947e77c0 -1 osd.1 0 log_to_monitors true 2024-09-19T07:19:16.803 INFO:tasks.ceph.osd.4.smithi184.stderr:2024-09-19T07:19:16.802+0000 7f1925b337c0 -1 osd.4 0 log_to_monitors true 2024-09-19T07:19:16.948 INFO:tasks.ceph.osd.5.smithi184.stderr:2024-09-19T07:19:16.950+0000 7ff8d34467c0 -1 osd.5 0 log_to_monitors true 2024-09-19T07:19:17.026 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-09-19T07:19:17.024+0000 7f6f102c47c0 -1 osd.0 0 log_to_monitors true 2024-09-19T07:19:18.577 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:18.705 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.700+0000 7f0fbeca7640 1 Processor -- start 2024-09-19T07:19:18.705 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.700+0000 7f0fbeca7640 1 -- start start 2024-09-19T07:19:18.706 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fbeca7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0fb80c63b0 0x7f0fb8153030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:18.706 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fbeca7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0fb8159d90 0x7f0fb81568f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:18.706 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fbeca7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fb80cbc90 con 0x7f0fb80bee60 2024-09-19T07:19:18.706 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fbeca7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f0fb815a350 con 0x7f0fb80c63b0 2024-09-19T07:19:18.706 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fbeca7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0fb80c96d0 con 0x7f0fb8159d90 2024-09-19T07:19:18.706 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fbca43640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0fb80c63b0 0x7f0fb8153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:18.707 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fb7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0fb80bee60 0x7f0fb81561e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59300/0 (socket says 172.21.15.83:59300) 2024-09-19T07:19:18.707 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fb7fff640 1 -- 172.21.15.83:0/4134449395 learned_addr learned my addr 172.21.15.83:0/4134449395 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:18.707 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fbd244640 1 --2- 172.21.15.83:0/4134449395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0fb8159d90 0x7f0fb81568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:18.707 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fb77fe640 1 -- 172.21.15.83:0/4134449395 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4189409289 0 0) 0x7f0fb80cbc90 con 0x7f0fb80bee60 2024-09-19T07:19:18.707 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fb77fe640 1 -- 172.21.15.83:0/4134449395 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fa0003530 con 0x7f0fb80bee60 2024-09-19T07:19:18.707 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fbca43640 1 -- 172.21.15.83:0/4134449395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0fb8159d90 msgr2=0x7f0fb81568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:18.708 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fbca43640 1 --2- 172.21.15.83:0/4134449395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0fb8159d90 0x7f0fb81568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:18.708 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fb77fe640 1 -- 172.21.15.83:0/4134449395 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2542193599 0 0) 0x7f0fa0003530 con 0x7f0fb80bee60 2024-09-19T07:19:18.708 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fbca43640 1 -- 172.21.15.83:0/4134449395 >> v1:172.21.15.83:6789/0 conn(0x7f0fb80bee60 legacy=0x7f0fb81561e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:18.708 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.704+0000 7f0fbca43640 1 -- 172.21.15.83:0/4134449395 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb80cbf30 con 0x7f0fb80c63b0 2024-09-19T07:19:18.710 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fbca43640 1 --2- 172.21.15.83:0/4134449395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fb80c63b0 0x7f0fb8153030 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0fac0097c0 tx=0x7f0fac01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2e51676d107cc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:18.710 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fb77fe640 1 -- 172.21.15.83:0/4134449395 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac0b0070 con 0x7f0fb80c63b0 2024-09-19T07:19:18.710 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fb77fe640 1 -- 172.21.15.83:0/4134449395 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fac0ab400 con 0x7f0fb80c63b0 2024-09-19T07:19:18.710 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fb77fe640 1 -- 172.21.15.83:0/4134449395 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa80016a0 con 0x7f0fb80c63b0 2024-09-19T07:19:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fbeca7640 1 -- 172.21.15.83:0/4134449395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fb80c63b0 msgr2=0x7f0fb8153030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fbeca7640 1 --2- 172.21.15.83:0/4134449395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fb80c63b0 0x7f0fb8153030 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0fac0097c0 tx=0x7f0fac01c330 comp rx=0 tx=0).stop 2024-09-19T07:19:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fbeca7640 1 -- 172.21.15.83:0/4134449395 shutdown_connections 2024-09-19T07:19:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fbeca7640 1 --2- 172.21.15.83:0/4134449395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0fb8159d90 0x7f0fb81568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fbeca7640 1 --2- 172.21.15.83:0/4134449395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fb80c63b0 0x7f0fb8153030 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:18.712 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fbeca7640 1 -- 172.21.15.83:0/4134449395 >> 172.21.15.83:0/4134449395 conn(0x7f0fb80734c0 msgr2=0x7f0fb80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:18.712 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fbeca7640 1 -- 172.21.15.83:0/4134449395 shutdown_connections 2024-09-19T07:19:18.712 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fbeca7640 1 -- 172.21.15.83:0/4134449395 wait complete. 2024-09-19T07:19:18.713 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fbeca7640 1 Processor -- start 2024-09-19T07:19:18.713 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.708+0000 7f0fbeca7640 1 -- start start 2024-09-19T07:19:18.715 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbeca7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fb80bee60 0x7f0fb81506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:18.715 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbeca7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0fb80c63b0 0x7f0fb8155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:18.715 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbeca7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0fb8159d90 0x7f0fb814e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:18.715 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbeca7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb80c6910 con 0x7f0fb80bee60 2024-09-19T07:19:18.715 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbeca7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb80c6790 con 0x7f0fb80c63b0 2024-09-19T07:19:18.716 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbeca7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0fb8162f30 con 0x7f0fb8159d90 2024-09-19T07:19:18.716 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbca43640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fb80bee60 0x7f0fb81506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:18.716 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbca43640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fb80bee60 0x7f0fb81506f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54964/0 (socket says 172.21.15.83:54964) 2024-09-19T07:19:18.716 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbca43640 1 -- 172.21.15.83:0/3294507911 learned_addr learned my addr 172.21.15.83:0/3294507911 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:18.716 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fb7fff640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0fb80c63b0 0x7f0fb8155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:18.716 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbd244640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0fb8159d90 0x7f0fb814e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:18.716 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbca43640 1 -- 172.21.15.83:0/3294507911 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0fb8159d90 msgr2=0x7f0fb814e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:18.716 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbca43640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0fb8159d90 0x7f0fb814e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:18.716 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbca43640 1 -- 172.21.15.83:0/3294507911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0fb80c63b0 msgr2=0x7f0fb8155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:18.717 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbca43640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0fb80c63b0 0x7f0fb8155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:18.717 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbca43640 1 -- 172.21.15.83:0/3294507911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb81ee210 con 0x7f0fb80bee60 2024-09-19T07:19:18.717 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbca43640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fb80bee60 0x7f0fb81506f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0fac0097c0 tx=0x7f0fac0190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:18.717 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fb57fa640 1 -- 172.21.15.83:0/3294507911 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac0b0040 con 0x7f0fb80bee60 2024-09-19T07:19:18.717 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fb57fa640 1 -- 172.21.15.83:0/3294507911 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fac0b9070 con 0x7f0fb80bee60 2024-09-19T07:19:18.717 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbeca7640 1 -- 172.21.15.83:0/3294507911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb81559e0 con 0x7f0fb80bee60 2024-09-19T07:19:18.717 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fb57fa640 1 -- 172.21.15.83:0/3294507911 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac0b4890 con 0x7f0fb80bee60 2024-09-19T07:19:18.717 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbeca7640 1 -- 172.21.15.83:0/3294507911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb81ee7b0 con 0x7f0fb80bee60 2024-09-19T07:19:18.718 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fbeca7640 1 -- 172.21.15.83:0/3294507911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fb8153030 con 0x7f0fb80bee60 2024-09-19T07:19:18.718 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.712+0000 7f0fb57fa640 1 -- 172.21.15.83:0/3294507911 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 4) ==== 87389+0+0 (secure 0 0 0) 0x7f0fac0b4a30 con 0x7f0fb80bee60 2024-09-19T07:19:18.718 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.716+0000 7f0fb57fa640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0fa4067830 0x7f0fa4069cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:18.718 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.716+0000 7f0fb57fa640 1 -- 172.21.15.83:0/3294507911 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2199+0+0 (secure 0 0 0) 0x7f0fac11ecc0 con 0x7f0fb80bee60 2024-09-19T07:19:18.718 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.716+0000 7f0fb7fff640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0fa4067830 0x7f0fa4069cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:18.718 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.716+0000 7f0fb7fff640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0fa4067830 0x7f0fa4069cf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8001ec0 tx=0x7f0fa8001d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:18.722 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.720+0000 7f0fb57fa640 1 -- 172.21.15.83:0/3294507911 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f0fac015fb0 con 0x7f0fb80bee60 2024-09-19T07:19:18.920 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.916+0000 7f0fbeca7640 1 -- 172.21.15.83:0/3294507911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fb814cc40 con 0x7f0fb80bee60 2024-09-19T07:19:18.951 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.948+0000 7f0fb57fa640 1 -- 172.21.15.83:0/3294507911 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+7058 (secure 0 0 0) 0x7f0fac0f0720 con 0x7f0fb80bee60 2024-09-19T07:19:18.951 INFO:teuthology.misc.health.smithi083.stdout: 2024-09-19T07:19:18.952 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":13,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:18.825338+0000","last_up_change":"0.000000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:18.955 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.952+0000 7f0fbeca7640 1 -- 172.21.15.83:0/3294507911 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0fa4067830 msgr2=0x7f0fa4069cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:18.955 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.952+0000 7f0fbeca7640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0fa4067830 0x7f0fa4069cf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8001ec0 tx=0x7f0fa8001d10 comp rx=0 tx=0).stop 2024-09-19T07:19:18.955 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.952+0000 7f0fbeca7640 1 -- 172.21.15.83:0/3294507911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fb80bee60 msgr2=0x7f0fb81506f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:18.955 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.952+0000 7f0fbeca7640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fb80bee60 0x7f0fb81506f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0fac0097c0 tx=0x7f0fac0190d0 comp rx=0 tx=0).stop 2024-09-19T07:19:18.955 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.952+0000 7f0fbeca7640 1 -- 172.21.15.83:0/3294507911 shutdown_connections 2024-09-19T07:19:18.955 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.952+0000 7f0fbeca7640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0fa4067830 0x7f0fa4069cf0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:18.955 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.952+0000 7f0fbeca7640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0fb8159d90 0x7f0fb814e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:18.955 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.952+0000 7f0fbeca7640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0fb80c63b0 0x7f0fb8155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:18.955 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.952+0000 7f0fbeca7640 1 --2- 172.21.15.83:0/3294507911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0fb80bee60 0x7f0fb81506f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:18.955 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.952+0000 7f0fbeca7640 1 -- 172.21.15.83:0/3294507911 >> 172.21.15.83:0/3294507911 conn(0x7f0fb80734c0 msgr2=0x7f0fb80c9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:18.955 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.952+0000 7f0fbeca7640 1 -- 172.21.15.83:0/3294507911 shutdown_connections 2024-09-19T07:19:18.956 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:18.952+0000 7f0fbeca7640 1 -- 172.21.15.83:0/3294507911 wait complete. 2024-09-19T07:19:18.976 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-19T07:19:19.516 INFO:tasks.ceph.osd.6.smithi184.stderr:2024-09-19T07:19:19.518+0000 7f480d8d6640 -1 osd.6 0 waiting for initial osdmap 2024-09-19T07:19:19.544 INFO:tasks.ceph.osd.6.smithi184.stderr:2024-09-19T07:19:19.546+0000 7f4805f1d640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:19:19.611 INFO:tasks.ceph.osd.4.smithi184.stderr:2024-09-19T07:19:19.610+0000 7f19212b9640 -1 osd.4 0 waiting for initial osdmap 2024-09-19T07:19:19.637 INFO:tasks.ceph.osd.4.smithi184.stderr:2024-09-19T07:19:19.638+0000 7f1919900640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:19:19.696 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-09-19T07:19:19.692+0000 7f4598cfb640 -1 osd.2 0 waiting for initial osdmap 2024-09-19T07:19:19.711 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-09-19T07:19:19.708+0000 7fc98ff6d640 -1 osd.1 0 waiting for initial osdmap 2024-09-19T07:19:19.712 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-09-19T07:19:19.708+0000 7ff5edd42640 -1 osd.3 0 waiting for initial osdmap 2024-09-19T07:19:19.719 INFO:tasks.ceph.osd.7.smithi184.stderr:2024-09-19T07:19:19.718+0000 7f735bae0640 -1 osd.7 0 waiting for initial osdmap 2024-09-19T07:19:19.722 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-09-19T07:19:19.720+0000 7f4591342640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:19:19.742 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-09-19T07:19:19.740+0000 7ff5e6389640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:19:19.745 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-09-19T07:19:19.744+0000 7fc9885b4640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:19:19.748 INFO:tasks.ceph.osd.7.smithi184.stderr:2024-09-19T07:19:19.750+0000 7f7354127640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:19:20.059 INFO:tasks.ceph.osd.5.smithi184.stderr:2024-09-19T07:19:20.058+0000 7ff8cebcc640 -1 osd.5 0 waiting for initial osdmap 2024-09-19T07:19:20.065 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-09-19T07:19:20.060+0000 7f6f0ca7f640 -1 osd.0 0 waiting for initial osdmap 2024-09-19T07:19:20.081 INFO:tasks.ceph.osd.5.smithi184.stderr:2024-09-19T07:19:20.082+0000 7ff8c7213640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:19:20.094 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-09-19T07:19:20.092+0000 7f6f04091640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T07:19:25.784 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:25.908 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.903+0000 7faf33f61640 1 Processor -- start 2024-09-19T07:19:25.908 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.903+0000 7faf33f61640 1 -- start start 2024-09-19T07:19:25.910 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.903+0000 7faf33f61640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf2c153470 0x7faf2c0737a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:25.910 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.903+0000 7faf33f61640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf2c068120 0x7faf2c153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:25.910 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.903+0000 7faf33f61640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7faf2c073ce0 0x7faf2c154310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:25.910 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.903+0000 7faf33f61640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7faf2c0ab820 con 0x7faf2c068120 2024-09-19T07:19:25.910 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.903+0000 7faf33f61640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7faf2c0aaca0 con 0x7faf2c073ce0 2024-09-19T07:19:25.910 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.903+0000 7faf33f61640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faf2c0aaf80 con 0x7faf2c153470 2024-09-19T07:19:25.910 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf314fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf2c068120 0x7faf2c153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:25.910 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf314fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf2c068120 0x7faf2c153dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33860/0 (socket says 172.21.15.83:33860) 2024-09-19T07:19:25.910 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf314fc640 1 -- 172.21.15.83:0/146409193 learned_addr learned my addr 172.21.15.83:0/146409193 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:25.911 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf31cfd640 1 --2- 172.21.15.83:0/146409193 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf2c153470 0x7faf2c0737a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:25.911 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf314fc640 1 -- 172.21.15.83:0/146409193 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf2c153470 msgr2=0x7faf2c0737a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:25.911 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf314fc640 1 --2- 172.21.15.83:0/146409193 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf2c153470 0x7faf2c0737a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:25.911 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf314fc640 1 -- 172.21.15.83:0/146409193 >> v2:172.21.15.184:3300/0 conn(0x7faf2c073ce0 msgr2=0x7faf2c154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:25.911 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf324fe640 1 --2- 172.21.15.83:0/146409193 >> v2:172.21.15.184:3300/0 conn(0x7faf2c073ce0 0x7faf2c154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:25.911 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf314fc640 1 --2- 172.21.15.83:0/146409193 >> v2:172.21.15.184:3300/0 conn(0x7faf2c073ce0 0x7faf2c154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:25.911 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf314fc640 1 -- 172.21.15.83:0/146409193 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2c0ab260 con 0x7faf2c068120 2024-09-19T07:19:25.911 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf31cfd640 1 --2- 172.21.15.83:0/146409193 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf2c153470 0x7faf2c0737a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:25.911 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf324fe640 1 --2- 172.21.15.83:0/146409193 >> v2:172.21.15.184:3300/0 conn(0x7faf2c073ce0 0x7faf2c154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:25.912 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf314fc640 1 --2- 172.21.15.83:0/146409193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2c068120 0x7faf2c153dd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faf2c071260 tx=0x7faf1c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7d1344fd07b612 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:25.912 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf30cfb640 1 -- 172.21.15.83:0/146409193 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c0b5070 con 0x7faf2c068120 2024-09-19T07:19:25.912 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf30cfb640 1 -- 172.21.15.83:0/146409193 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf1c0a89d0 con 0x7faf2c068120 2024-09-19T07:19:25.912 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf30cfb640 1 -- 172.21.15.83:0/146409193 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c0a8da0 con 0x7faf2c068120 2024-09-19T07:19:25.913 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf33f61640 1 -- 172.21.15.83:0/146409193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2c068120 msgr2=0x7faf2c153dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:25.913 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf33f61640 1 --2- 172.21.15.83:0/146409193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2c068120 0x7faf2c153dd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faf2c071260 tx=0x7faf1c01ed10 comp rx=0 tx=0).stop 2024-09-19T07:19:25.913 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf33f61640 1 -- 172.21.15.83:0/146409193 shutdown_connections 2024-09-19T07:19:25.913 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf33f61640 1 --2- 172.21.15.83:0/146409193 >> v2:172.21.15.184:3300/0 conn(0x7faf2c073ce0 0x7faf2c154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:25.913 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf33f61640 1 --2- 172.21.15.83:0/146409193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2c068120 0x7faf2c153dd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:25.913 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf33f61640 1 --2- 172.21.15.83:0/146409193 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf2c153470 0x7faf2c0737a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:25.914 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf33f61640 1 -- 172.21.15.83:0/146409193 >> 172.21.15.83:0/146409193 conn(0x7faf2c0a7870 msgr2=0x7faf2c0a7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:25.914 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf33f61640 1 -- 172.21.15.83:0/146409193 shutdown_connections 2024-09-19T07:19:25.914 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.907+0000 7faf33f61640 1 -- 172.21.15.83:0/146409193 wait complete. 2024-09-19T07:19:25.916 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf33f61640 1 Processor -- start 2024-09-19T07:19:25.916 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf33f61640 1 -- start start 2024-09-19T07:19:25.916 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf33f61640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf2c068120 0x7faf2c1ebd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:25.916 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf33f61640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf2c073ce0 0x7faf2c0c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:25.916 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf33f61640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2c153470 0x7faf2c14ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:25.916 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf33f61640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c154850 con 0x7faf2c153470 2024-09-19T07:19:25.916 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf33f61640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c153850 con 0x7faf2c068120 2024-09-19T07:19:25.916 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf33f61640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faf2c0adb70 con 0x7faf2c073ce0 2024-09-19T07:19:25.916 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf324fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2c153470 0x7faf2c14ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:25.916 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf324fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2c153470 0x7faf2c14ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33870/0 (socket says 172.21.15.83:33870) 2024-09-19T07:19:25.917 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf324fe640 1 -- 172.21.15.83:0/1344777061 learned_addr learned my addr 172.21.15.83:0/1344777061 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:25.917 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf314fc640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf2c073ce0 0x7faf2c0c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:25.917 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf31cfd640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf2c068120 0x7faf2c1ebd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:25.917 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf324fe640 1 -- 172.21.15.83:0/1344777061 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf2c073ce0 msgr2=0x7faf2c0c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:25.917 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf324fe640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf2c073ce0 0x7faf2c0c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:25.917 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf324fe640 1 -- 172.21.15.83:0/1344777061 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf2c068120 msgr2=0x7faf2c1ebd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:25.917 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf324fe640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf2c068120 0x7faf2c1ebd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:25.917 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf324fe640 1 -- 172.21.15.83:0/1344777061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2c155d60 con 0x7faf2c153470 2024-09-19T07:19:25.917 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf31cfd640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf2c068120 0x7faf2c1ebd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:25.917 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf324fe640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2c153470 0x7faf2c14ff20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faf2400c7f0 tx=0x7faf2400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:25.917 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf227fc640 1 -- 172.21.15.83:0/1344777061 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24015070 con 0x7faf2c153470 2024-09-19T07:19:25.917 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf227fc640 1 -- 172.21.15.83:0/1344777061 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf24007820 con 0x7faf2c153470 2024-09-19T07:19:25.918 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf33f61640 1 -- 172.21.15.83:0/1344777061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2c156040 con 0x7faf2c153470 2024-09-19T07:19:25.918 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf227fc640 1 -- 172.21.15.83:0/1344777061 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24007c10 con 0x7faf2c153470 2024-09-19T07:19:25.918 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf33f61640 1 -- 172.21.15.83:0/1344777061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf2c0bce50 con 0x7faf2c153470 2024-09-19T07:19:25.919 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.911+0000 7faf33f61640 1 -- 172.21.15.83:0/1344777061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf2c0c4040 con 0x7faf2c153470 2024-09-19T07:19:25.919 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.915+0000 7faf227fc640 1 -- 172.21.15.83:0/1344777061 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf24018070 con 0x7faf2c153470 2024-09-19T07:19:25.919 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.915+0000 7faf227fc640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf100678c0 0x7faf10069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:25.919 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.915+0000 7faf227fc640 1 -- 172.21.15.83:0/1344777061 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7faf24085de0 con 0x7faf2c153470 2024-09-19T07:19:25.919 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.915+0000 7faf31cfd640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf100678c0 0x7faf10069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:25.919 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.915+0000 7faf31cfd640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf100678c0 0x7faf10069d80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faf28007f40 tx=0x7faf28007490 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:25.922 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:25.919+0000 7faf227fc640 1 -- 172.21.15.83:0/1344777061 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7faf24056ea0 con 0x7faf2c153470 2024-09-19T07:19:26.085 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.079+0000 7faf33f61640 1 -- 172.21.15.83:0/1344777061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf2c067f00 con 0x7faf2c153470 2024-09-19T07:19:26.086 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf227fc640 1 -- 172.21.15.83:0/1344777061 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12896 (secure 0 0 0) 0x7faf240567a0 con 0x7faf2c153470 2024-09-19T07:19:26.086 INFO:teuthology.misc.health.smithi083.stdout: 2024-09-19T07:19:26.086 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":17,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:23.537135+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:26.088 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf33f61640 1 -- 172.21.15.83:0/1344777061 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf100678c0 msgr2=0x7faf10069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:26.088 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf33f61640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf100678c0 0x7faf10069d80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faf28007f40 tx=0x7faf28007490 comp rx=0 tx=0).stop 2024-09-19T07:19:26.089 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf33f61640 1 -- 172.21.15.83:0/1344777061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2c153470 msgr2=0x7faf2c14ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:26.089 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf33f61640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2c153470 0x7faf2c14ff20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faf2400c7f0 tx=0x7faf2400ccb0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.089 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf33f61640 1 -- 172.21.15.83:0/1344777061 shutdown_connections 2024-09-19T07:19:26.089 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf33f61640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf100678c0 0x7faf10069d80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.089 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf33f61640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2c153470 0x7faf2c14ff20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.089 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf33f61640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf2c073ce0 0x7faf2c0c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.089 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf33f61640 1 --2- 172.21.15.83:0/1344777061 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf2c068120 0x7faf2c1ebd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.089 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf33f61640 1 -- 172.21.15.83:0/1344777061 >> 172.21.15.83:0/1344777061 conn(0x7faf2c0a7870 msgr2=0x7faf2c152490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:26.089 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf33f61640 1 -- 172.21.15.83:0/1344777061 shutdown_connections 2024-09-19T07:19:26.089 INFO:teuthology.misc.health.smithi083.stderr:2024-09-19T07:19:26.083+0000 7faf33f61640 1 -- 172.21.15.83:0/1344777061 wait complete. 2024-09-19T07:19:26.104 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-19T07:19:26.104 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-19T07:19:26.104 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:19:26.104 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:19:26.107 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:19:26.107 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:19:26.107 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:19:26.109 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:19:26.109 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-19T07:19:26.110 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-19T07:19:26.112 INFO:tasks.ceph.mds.b:Started 2024-09-19T07:19:26.113 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-19T07:19:26.113 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-19T07:19:26.115 INFO:tasks.ceph.mds.d:Started 2024-09-19T07:19:26.115 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-19T07:19:26.116 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:19:26.180 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:19:26.181 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:19:26.210 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-19T07:19:26.211 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-19T07:19:26.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc82da640 1 Processor -- start 2024-09-19T07:19:26.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc82da640 1 -- start start 2024-09-19T07:19:26.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc82da640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7facc014a500 0x7facc00c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:26.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc82da640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7facc0154540 0x7facc0157490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:26.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc82da640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7facc00c4550 0x7facc01579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:26.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc82da640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7facc0151750 con 0x7facc014a500 2024-09-19T07:19:26.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc82da640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7facc0157f10 con 0x7facc0154540 2024-09-19T07:19:26.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc82da640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7facc00c7e90 con 0x7facc00c4550 2024-09-19T07:19:26.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc6076640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7facc014a500 0x7facc00c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:26.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc6076640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7facc014a500 0x7facc00c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33926/0 (socket says 172.21.15.83:33926) 2024-09-19T07:19:26.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc6076640 1 -- 172.21.15.83:0/4084277552 learned_addr learned my addr 172.21.15.83:0/4084277552 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:26.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc5875640 1 --2- 172.21.15.83:0/4084277552 >> v2:172.21.15.184:3300/0 conn(0x7facc0154540 0x7facc0157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:26.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc6877640 1 --2- 172.21.15.83:0/4084277552 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7facc00c4550 0x7facc01579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:26.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc6076640 1 -- 172.21.15.83:0/4084277552 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7facc00c4550 msgr2=0x7facc01579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:26.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc6076640 1 --2- 172.21.15.83:0/4084277552 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7facc00c4550 0x7facc01579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc6076640 1 -- 172.21.15.83:0/4084277552 >> v2:172.21.15.184:3300/0 conn(0x7facc0154540 msgr2=0x7facc0157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:26.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc6076640 1 --2- 172.21.15.83:0/4084277552 >> v2:172.21.15.184:3300/0 conn(0x7facc0154540 0x7facc0157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc6076640 1 -- 172.21.15.83:0/4084277552 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc0073110 con 0x7facc014a500 2024-09-19T07:19:26.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.227+0000 7facc6076640 1 --2- 172.21.15.83:0/4084277552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facc014a500 0x7facc00c4010 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7facc00c94f0 tx=0x7facb801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3c0501bded2fc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:26.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc5074640 1 -- 172.21.15.83:0/4084277552 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb80a5840 con 0x7facc014a500 2024-09-19T07:19:26.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc5074640 1 -- 172.21.15.83:0/4084277552 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb80a59e0 con 0x7facc014a500 2024-09-19T07:19:26.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc5074640 1 -- 172.21.15.83:0/4084277552 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb80a5db0 con 0x7facc014a500 2024-09-19T07:19:26.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 -- 172.21.15.83:0/4084277552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facc014a500 msgr2=0x7facc00c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:26.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 --2- 172.21.15.83:0/4084277552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facc014a500 0x7facc00c4010 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7facc00c94f0 tx=0x7facb801c9c0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 -- 172.21.15.83:0/4084277552 shutdown_connections 2024-09-19T07:19:26.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 --2- 172.21.15.83:0/4084277552 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7facc00c4550 0x7facc01579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 --2- 172.21.15.83:0/4084277552 >> v2:172.21.15.184:3300/0 conn(0x7facc0154540 0x7facc0157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 --2- 172.21.15.83:0/4084277552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facc014a500 0x7facc00c4010 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 -- 172.21.15.83:0/4084277552 >> 172.21.15.83:0/4084277552 conn(0x7facc00c9610 msgr2=0x7facc00c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:26.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 -- 172.21.15.83:0/4084277552 shutdown_connections 2024-09-19T07:19:26.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 -- 172.21.15.83:0/4084277552 wait complete. 2024-09-19T07:19:26.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 Processor -- start 2024-09-19T07:19:26.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 -- start start 2024-09-19T07:19:26.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facc00c4550 0x7facc0144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:26.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7facc014a500 0x7facc014ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:26.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facc0154540 0x7facc014b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:26.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7facc00c76d0 con 0x7facc0154540 2024-09-19T07:19:26.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7facc00741f0 con 0x7facc00c4550 2024-09-19T07:19:26.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7facc00c7850 con 0x7facc014a500 2024-09-19T07:19:26.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc6877640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facc0154540 0x7facc014b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:26.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc6877640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facc0154540 0x7facc014b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33934/0 (socket says 172.21.15.83:33934) 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc6877640 1 -- 172.21.15.83:0/598765142 learned_addr learned my addr 172.21.15.83:0/598765142 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc5875640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7facc014a500 0x7facc014ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc6076640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facc00c4550 0x7facc0144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc6877640 1 -- 172.21.15.83:0/598765142 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7facc014a500 msgr2=0x7facc014ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc6877640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7facc014a500 0x7facc014ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc6877640 1 -- 172.21.15.83:0/598765142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facc00c4550 msgr2=0x7facc0144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc6877640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facc00c4550 0x7facc0144eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc6877640 1 -- 172.21.15.83:0/598765142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc006d4a0 con 0x7facc0154540 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc6877640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facc0154540 0x7facc014b060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7facbc00d600 tx=0x7facbc00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facb6ffd640 1 -- 172.21.15.83:0/598765142 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc018020 con 0x7facc0154540 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facb6ffd640 1 -- 172.21.15.83:0/598765142 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facbc014620 con 0x7facc0154540 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 -- 172.21.15.83:0/598765142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc014b5a0 con 0x7facc0154540 2024-09-19T07:19:26.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facb6ffd640 1 -- 172.21.15.83:0/598765142 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc0149f0 con 0x7facc0154540 2024-09-19T07:19:26.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.231+0000 7facc82da640 1 -- 172.21.15.83:0/598765142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc006d740 con 0x7facc0154540 2024-09-19T07:19:26.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.235+0000 7facc82da640 1 -- 172.21.15.83:0/598765142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facc00c4010 con 0x7facc0154540 2024-09-19T07:19:26.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.235+0000 7facb6ffd640 1 -- 172.21.15.83:0/598765142 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7facbc01b020 con 0x7facc0154540 2024-09-19T07:19:26.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.235+0000 7facb6ffd640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faca40678f0 0x7faca4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:26.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.235+0000 7facb6ffd640 1 -- 172.21.15.83:0/598765142 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7facbc085c60 con 0x7facc0154540 2024-09-19T07:19:26.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.235+0000 7facc6076640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faca40678f0 0x7faca4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:26.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.235+0000 7facc6076640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faca40678f0 0x7faca4069db0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7facb80ae000 tx=0x7facb80a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:26.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.239+0000 7facb6ffd640 1 -- 172.21.15.83:0/598765142 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7facbc022020 con 0x7facc0154540 2024-09-19T07:19:26.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 -- 172.21.15.83:0/598765142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facc0073fd0 con 0x7facc0154540 2024-09-19T07:19:26.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facb6ffd640 1 -- 172.21.15.83:0/598765142 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1434 (secure 0 0 0) 0x7facbc0564a0 con 0x7facc0154540 2024-09-19T07:19:26.431 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:26.431 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":2,"btime":"2024-09-19T07:19:26:204326+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-19T07:19:26.431 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 2 2024-09-19T07:19:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 -- 172.21.15.83:0/598765142 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faca40678f0 msgr2=0x7faca4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faca40678f0 0x7faca4069db0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7facb80ae000 tx=0x7facb80a7070 comp rx=0 tx=0).stop 2024-09-19T07:19:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 -- 172.21.15.83:0/598765142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facc0154540 msgr2=0x7facc014b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facc0154540 0x7facc014b060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7facbc00d600 tx=0x7facbc00dac0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 -- 172.21.15.83:0/598765142 shutdown_connections 2024-09-19T07:19:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faca40678f0 0x7faca4069db0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facc0154540 0x7facc014b060 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7facc014a500 0x7facc014ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 --2- 172.21.15.83:0/598765142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7facc00c4550 0x7facc0144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 -- 172.21.15.83:0/598765142 >> 172.21.15.83:0/598765142 conn(0x7facc00c9610 msgr2=0x7facc00c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 -- 172.21.15.83:0/598765142 shutdown_connections 2024-09-19T07:19:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:26.427+0000 7facc82da640 1 -- 172.21.15.83:0/598765142 wait complete. 2024-09-19T07:19:28.450 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:19:28.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.579+0000 7fa9bd300640 1 Processor -- start 2024-09-19T07:19:28.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.579+0000 7fa9bd300640 1 -- start start 2024-09-19T07:19:28.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9bd300640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fa9b80c42e0 0x7fa9b806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9bd300640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81566b0 0x7fa9b80abb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9bd300640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9b81592c0 con 0x7fa9b8159a80 2024-09-19T07:19:28.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9bd300640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fa9b80ac070 con 0x7fa9b80c42e0 2024-09-19T07:19:28.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9bd300640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b81595f0 con 0x7fa9b81566b0 2024-09-19T07:19:28.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa9b8159a80 0x7fa9b8153060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49182/0 (socket says 172.21.15.83:49182) 2024-09-19T07:19:28.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b77fe640 1 -- 172.21.15.83:0/3737106060 learned_addr learned my addr 172.21.15.83:0/3737106060 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:28.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b67fc640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81566b0 0x7fa9b80abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b6ffd640 1 --2- 172.21.15.83:0/3737106060 >> v2:172.21.15.184:3300/0 conn(0x7fa9b80c42e0 0x7fa9b806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3178132383 0 0) 0x7fa9b81592c0 con 0x7fa9b8159a80 2024-09-19T07:19:28.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9a0003530 con 0x7fa9b8159a80 2024-09-19T07:19:28.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1733639502 0 0) 0x7fa9a0003530 con 0x7fa9b8159a80 2024-09-19T07:19:28.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81566b0 msgr2=0x7fa9b80abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:28.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81566b0 0x7fa9b80abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 >> v2:172.21.15.184:3300/0 conn(0x7fa9b80c42e0 msgr2=0x7fa9b806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:28.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 --2- 172.21.15.83:0/3737106060 >> v2:172.21.15.184:3300/0 conn(0x7fa9b80c42e0 0x7fa9b806e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b806e900 con 0x7fa9b8159a80 2024-09-19T07:19:28.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fa9b0001a00 con 0x7fa9b8159a80 2024-09-19T07:19:28.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 >> v1:172.21.15.83:6789/0 conn(0x7fa9b8159a80 legacy=0x7fa9b8153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:28.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9a00039c0 0x7fa9a0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9a0004810 0x7fa9a0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81566b0 0x7fa9a00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b81595f0 con 0x7fa9a0004810 2024-09-19T07:19:28.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a000b9d0 con 0x7fa9a00039c0 2024-09-19T07:19:28.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa9a000bb50 con 0x7fa9b81566b0 2024-09-19T07:19:28.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b67fc640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9a0004810 0x7fa9a0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b6ffd640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9a00039c0 0x7fa9a0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b77fe640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81566b0 0x7fa9a00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b67fc640 1 -- 172.21.15.83:0/3737106060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81566b0 msgr2=0x7fa9a00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:28.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b67fc640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81566b0 0x7fa9a00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b67fc640 1 -- 172.21.15.83:0/3737106060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9a00039c0 msgr2=0x7fa9a0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:28.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b67fc640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9a00039c0 0x7fa9a0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b67fc640 1 -- 172.21.15.83:0/3737106060 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a0003530 con 0x7fa9a0004810 2024-09-19T07:19:28.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.583+0000 7fa9b77fe640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81566b0 0x7fa9a00051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:28.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b67fc640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9a0004810 0x7fa9a0004c90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009ab0 tx=0x7fa9ac01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1fbfb24045d5be0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:28.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac0b2070 con 0x7fa9a0004810 2024-09-19T07:19:28.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9ac0a5560 con 0x7fa9a0004810 2024-09-19T07:19:28.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b5ffb640 1 -- 172.21.15.83:0/3737106060 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac0096b0 con 0x7fa9a0004810 2024-09-19T07:19:28.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 -- 172.21.15.83:0/3737106060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9a0004810 msgr2=0x7fa9a0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:28.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 --2- 172.21.15.83:0/3737106060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9a0004810 0x7fa9a0004c90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009ab0 tx=0x7fa9ac01ca50 comp rx=0 tx=0).stop 2024-09-19T07:19:28.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b77fe640 1 -- 172.21.15.83:0/3737106060 reap_dead start 2024-09-19T07:19:28.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 -- 172.21.15.83:0/3737106060 shutdown_connections 2024-09-19T07:19:28.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 -- 172.21.15.83:0/3737106060 >> 172.21.15.83:0/3737106060 conn(0x7fa9b806bc80 msgr2=0x7fa9b806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:28.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 -- 172.21.15.83:0/3737106060 shutdown_connections 2024-09-19T07:19:28.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 -- 172.21.15.83:0/3737106060 wait complete. 2024-09-19T07:19:28.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 Processor -- start 2024-09-19T07:19:28.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 -- start start 2024-09-19T07:19:28.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9b814a230 0x7fa9b81eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81eefa0 0x7fa9b8153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b81ef420 0x7fa9b8153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b8156af0 con 0x7fa9b81ef420 2024-09-19T07:19:28.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b806eec0 con 0x7fa9b814a230 2024-09-19T07:19:28.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9bd300640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b8153770 con 0x7fa9b81eefa0 2024-09-19T07:19:28.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b81ef420 0x7fa9b8153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b81ef420 0x7fa9b8153fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33956/0 (socket says 172.21.15.83:33956) 2024-09-19T07:19:28.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b77fe640 1 -- 172.21.15.83:0/545619595 learned_addr learned my addr 172.21.15.83:0/545619595 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:28.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b6ffd640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9b814a230 0x7fa9b81eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b67fc640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81eefa0 0x7fa9b8153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b77fe640 1 -- 172.21.15.83:0/545619595 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81eefa0 msgr2=0x7fa9b8153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:28.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b77fe640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81eefa0 0x7fa9b8153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b77fe640 1 -- 172.21.15.83:0/545619595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9b814a230 msgr2=0x7fa9b81eea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:28.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b77fe640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9b814a230 0x7fa9b81eea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b77fe640 1 -- 172.21.15.83:0/545619595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b814f550 con 0x7fa9b81ef420 2024-09-19T07:19:28.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.587+0000 7fa9b77fe640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b81ef420 0x7fa9b8153fe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa9b80aa140 tx=0x7fa9b0005a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:28.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.591+0000 7fa99f7fe640 1 -- 172.21.15.83:0/545619595 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b001a070 con 0x7fa9b81ef420 2024-09-19T07:19:28.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.591+0000 7fa99f7fe640 1 -- 172.21.15.83:0/545619595 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b0011040 con 0x7fa9b81ef420 2024-09-19T07:19:28.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.591+0000 7fa9bd300640 1 -- 172.21.15.83:0/545619595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b80a9f70 con 0x7fa9b81ef420 2024-09-19T07:19:28.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.591+0000 7fa99f7fe640 1 -- 172.21.15.83:0/545619595 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b000ed50 con 0x7fa9b81ef420 2024-09-19T07:19:28.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.591+0000 7fa9bd300640 1 -- 172.21.15.83:0/545619595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b8158a60 con 0x7fa9b81ef420 2024-09-19T07:19:28.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.591+0000 7fa9bd300640 1 -- 172.21.15.83:0/545619595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b8152b30 con 0x7fa9b81ef420 2024-09-19T07:19:28.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.591+0000 7fa99f7fe640 1 -- 172.21.15.83:0/545619595 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa9b001d070 con 0x7fa9b81ef420 2024-09-19T07:19:28.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.591+0000 7fa99f7fe640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9a40678f0 0x7fa9a4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.591+0000 7fa9b6ffd640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9a40678f0 0x7fa9a4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.591+0000 7fa99f7fe640 1 -- 172.21.15.83:0/545619595 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa9b0088410 con 0x7fa9b81ef420 2024-09-19T07:19:28.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.591+0000 7fa9b6ffd640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9a40678f0 0x7fa9a4069db0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa9b81efb20 tx=0x7fa9a8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:28.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.595+0000 7fa99f7fe640 1 -- 172.21.15.83:0/545619595 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa9b8152b30 con 0x7fa9b81ef420 2024-09-19T07:19:28.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.783+0000 7fa9bd300640 1 -- 172.21.15.83:0/545619595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9b8159f10 con 0x7fa9b81ef420 2024-09-19T07:19:28.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.783+0000 7fa99f7fe640 1 -- 172.21.15.83:0/545619595 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3909 (secure 0 0 0) 0x7fa9b0058e80 con 0x7fa9b81ef420 2024-09-19T07:19:28.785 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:28.785 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":3,"btime":"2024-09-19T07:19:27:205700+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-19T07:19:28.786 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 3 2024-09-19T07:19:28.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.787+0000 7fa9bd300640 1 -- 172.21.15.83:0/545619595 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9a40678f0 msgr2=0x7fa9a4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:28.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.787+0000 7fa9bd300640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9a40678f0 0x7fa9a4069db0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa9b81efb20 tx=0x7fa9a8008040 comp rx=0 tx=0).stop 2024-09-19T07:19:28.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.787+0000 7fa9bd300640 1 -- 172.21.15.83:0/545619595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b81ef420 msgr2=0x7fa9b8153fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:28.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.787+0000 7fa9bd300640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b81ef420 0x7fa9b8153fe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa9b80aa140 tx=0x7fa9b0005a60 comp rx=0 tx=0).stop 2024-09-19T07:19:28.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.787+0000 7fa9bd300640 1 -- 172.21.15.83:0/545619595 shutdown_connections 2024-09-19T07:19:28.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.787+0000 7fa9bd300640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9a40678f0 0x7fa9a4069db0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.787+0000 7fa9bd300640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b81ef420 0x7fa9b8153fe0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.787+0000 7fa9bd300640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b81eefa0 0x7fa9b8153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.787+0000 7fa9bd300640 1 --2- 172.21.15.83:0/545619595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9b814a230 0x7fa9b81eea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.787+0000 7fa9bd300640 1 -- 172.21.15.83:0/545619595 >> 172.21.15.83:0/545619595 conn(0x7fa9b806bc80 msgr2=0x7fa9b814dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:28.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.787+0000 7fa9bd300640 1 -- 172.21.15.83:0/545619595 shutdown_connections 2024-09-19T07:19:28.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.787+0000 7fa9bd300640 1 -- 172.21.15.83:0/545619595 wait complete. 2024-09-19T07:19:28.802 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-19T07:19:28.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee0a0f9640 1 Processor -- start 2024-09-19T07:19:28.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee0a0f9640 1 -- start start 2024-09-19T07:19:28.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee0a0f9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fee041578e0 0x7fee040aaf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee0a0f9640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fee040c2df0 0x7fee040c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee0a0f9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee040c28a0 con 0x7fee040c2350 2024-09-19T07:19:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee0a0f9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fee040c4950 con 0x7fee041578e0 2024-09-19T07:19:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee0a0f9640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fee040c3540 con 0x7fee040c2df0 2024-09-19T07:19:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee037fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fee041578e0 0x7fee040aaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee037fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fee041578e0 0x7fee040aaf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33966/0 (socket says 172.21.15.83:33966) 2024-09-19T07:19:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee037fe640 1 -- 172.21.15.83:0/2733970088 learned_addr learned my addr 172.21.15.83:0/2733970088 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee02ffd640 1 --2- 172.21.15.83:0/2733970088 >> v2:172.21.15.184:3300/0 conn(0x7fee040c2df0 0x7fee040c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee037fe640 1 -- 172.21.15.83:0/2733970088 >> v2:172.21.15.184:3300/0 conn(0x7fee040c2df0 msgr2=0x7fee040c3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee037fe640 1 --2- 172.21.15.83:0/2733970088 >> v2:172.21.15.184:3300/0 conn(0x7fee040c2df0 0x7fee040c3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee037fe640 1 -- 172.21.15.83:0/2733970088 >> v1:172.21.15.184:6789/0 conn(0x7fee040c2350 legacy=0x7fee0415d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.919+0000 7fee037fe640 1 -- 172.21.15.83:0/2733970088 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee040c2b60 con 0x7fee041578e0 2024-09-19T07:19:28.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee037fe640 1 --2- 172.21.15.83:0/2733970088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee041578e0 0x7fee040aaf90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee0406efe0 tx=0x7fedf401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea613708ff474627 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:28.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee027fc640 1 -- 172.21.15.83:0/2733970088 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf40a55e0 con 0x7fee041578e0 2024-09-19T07:19:28.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee027fc640 1 -- 172.21.15.83:0/2733970088 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedf40a5780 con 0x7fee041578e0 2024-09-19T07:19:28.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee027fc640 1 -- 172.21.15.83:0/2733970088 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf40a5b50 con 0x7fee041578e0 2024-09-19T07:19:28.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 -- 172.21.15.83:0/2733970088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee041578e0 msgr2=0x7fee040aaf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:28.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 --2- 172.21.15.83:0/2733970088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee041578e0 0x7fee040aaf90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee0406efe0 tx=0x7fedf401c920 comp rx=0 tx=0).stop 2024-09-19T07:19:28.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 -- 172.21.15.83:0/2733970088 shutdown_connections 2024-09-19T07:19:28.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 --2- 172.21.15.83:0/2733970088 >> v2:172.21.15.184:3300/0 conn(0x7fee040c2df0 0x7fee040c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 --2- 172.21.15.83:0/2733970088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee041578e0 0x7fee040aaf90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 -- 172.21.15.83:0/2733970088 >> 172.21.15.83:0/2733970088 conn(0x7fee0406bc80 msgr2=0x7fee0406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:28.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 -- 172.21.15.83:0/2733970088 shutdown_connections 2024-09-19T07:19:28.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 -- 172.21.15.83:0/2733970088 wait complete. 2024-09-19T07:19:28.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 Processor -- start 2024-09-19T07:19:28.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 -- start start 2024-09-19T07:19:28.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee040c2350 0x7fee041f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee040c2df0 0x7fee041f6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee041578e0 0x7fee04151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee040ab650 con 0x7fee040c2df0 2024-09-19T07:19:28.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fee040ab4d0 con 0x7fee041578e0 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fee04157cc0 con 0x7fee040c2350 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee02ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee040c2df0 0x7fee041f6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee02ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee040c2df0 0x7fee041f6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33976/0 (socket says 172.21.15.83:33976) 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee02ffd640 1 -- 172.21.15.83:0/413180276 learned_addr learned my addr 172.21.15.83:0/413180276 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee03fff640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee041578e0 0x7fee04151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee02ffd640 1 -- 172.21.15.83:0/413180276 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee040c2350 msgr2=0x7fee041f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee02ffd640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee040c2350 0x7fee041f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee02ffd640 1 -- 172.21.15.83:0/413180276 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee041578e0 msgr2=0x7fee04151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee02ffd640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee041578e0 0x7fee04151840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee02ffd640 1 -- 172.21.15.83:0/413180276 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee04159f30 con 0x7fee040c2df0 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee02ffd640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee040c2df0 0x7fee041f6aa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fedec00c7f0 tx=0x7fedec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fedfbfff640 1 -- 172.21.15.83:0/413180276 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec015070 con 0x7fee040c2df0 2024-09-19T07:19:28.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fedfbfff640 1 -- 172.21.15.83:0/413180276 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedec007820 con 0x7fee040c2df0 2024-09-19T07:19:28.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 -- 172.21.15.83:0/413180276 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee040bc7b0 con 0x7fee040c2df0 2024-09-19T07:19:28.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fedfbfff640 1 -- 172.21.15.83:0/413180276 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec007c10 con 0x7fee040c2df0 2024-09-19T07:19:28.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.923+0000 7fee0a0f9640 1 -- 172.21.15.83:0/413180276 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee04071870 con 0x7fee040c2df0 2024-09-19T07:19:28.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.927+0000 7fee0a0f9640 1 -- 172.21.15.83:0/413180276 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee04072390 con 0x7fee040c2df0 2024-09-19T07:19:28.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.927+0000 7fedfbfff640 1 -- 172.21.15.83:0/413180276 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fedec018070 con 0x7fee040c2df0 2024-09-19T07:19:28.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.927+0000 7fedfbfff640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fede40678c0 0x7fede4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:28.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.927+0000 7fee037fe640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fede40678c0 0x7fede4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:28.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.927+0000 7fedfbfff640 1 -- 172.21.15.83:0/413180276 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fedec085990 con 0x7fee040c2df0 2024-09-19T07:19:28.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.927+0000 7fee037fe640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fede40678c0 0x7fede4069d80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fedf40ae000 tx=0x7fedf4004ac0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:28.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:28.931+0000 7fedfbfff640 1 -- 172.21.15.83:0/413180276 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fee04072390 con 0x7fee040c2df0 2024-09-19T07:19:29.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 -- 172.21.15.83:0/413180276 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fee04073e70 con 0x7fee040c2df0 2024-09-19T07:19:29.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fedfbfff640 1 -- 172.21.15.83:0/413180276 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7fedec0568b0 con 0x7fee040c2df0 2024-09-19T07:19:29.094 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:29.094 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2024-09-19T07:19:29.095 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-19T07:19:29.095 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 1, 2024-09-19T07:19:29.095 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": ".mgr" 2024-09-19T07:19:29.095 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-19T07:19:29.095 INFO:teuthology.orchestra.run.smithi083.stdout:] 2024-09-19T07:19:29.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 -- 172.21.15.83:0/413180276 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fede40678c0 msgr2=0x7fede4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:29.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fede40678c0 0x7fede4069d80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fedf40ae000 tx=0x7fedf4004ac0 comp rx=0 tx=0).stop 2024-09-19T07:19:29.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 -- 172.21.15.83:0/413180276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee040c2df0 msgr2=0x7fee041f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:29.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee040c2df0 0x7fee041f6aa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fedec00c7f0 tx=0x7fedec00ccb0 comp rx=0 tx=0).stop 2024-09-19T07:19:29.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 -- 172.21.15.83:0/413180276 shutdown_connections 2024-09-19T07:19:29.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fede40678c0 0x7fede4069d80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:29.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee041578e0 0x7fee04151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:29.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee040c2df0 0x7fee041f6aa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:29.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 --2- 172.21.15.83:0/413180276 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee040c2350 0x7fee041f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:29.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 -- 172.21.15.83:0/413180276 >> 172.21.15.83:0/413180276 conn(0x7fee0406bc80 msgr2=0x7fee04165030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:29.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 -- 172.21.15.83:0/413180276 shutdown_connections 2024-09-19T07:19:29.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.095+0000 7fee0a0f9640 1 -- 172.21.15.83:0/413180276 wait complete. 2024-09-19T07:19:29.113 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'a' 2024-09-19T07:19:29.113 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_metadata --pg_num_min 64 2024-09-19T07:19:29.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.243+0000 7f774a7a6640 1 Processor -- start 2024-09-19T07:19:29.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.243+0000 7f774a7a6640 1 -- start start 2024-09-19T07:19:29.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.243+0000 7f774a7a6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7744151330 0x7f7744070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:29.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.243+0000 7f774a7a6640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f77440a9d30 0x7f774406e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:29.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.243+0000 7f774a7a6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f774406ea10 con 0x7f77440713e0 2024-09-19T07:19:29.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.243+0000 7f774a7a6640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f7744154910 con 0x7f77440a9d30 2024-09-19T07:19:29.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.243+0000 7f774a7a6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7744153d00 con 0x7f7744151330 2024-09-19T07:19:29.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.243+0000 7f7748d43640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f77440a9d30 0x7f774406e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:29.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.243+0000 7f7743fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7744151330 0x7f7744070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:29.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.243+0000 7f7748d43640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f77440a9d30 0x7f774406e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:60728/0 (socket says 172.21.15.83:60728) 2024-09-19T07:19:29.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.243+0000 7f7748d43640 1 -- 172.21.15.83:0/1967620856 learned_addr learned my addr 172.21.15.83:0/1967620856 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:29.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.243+0000 7f77437fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f77440713e0 0x7f774406ddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:41378/0 (socket says 172.21.15.83:41378) 2024-09-19T07:19:29.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f7748d43640 1 -- 172.21.15.83:0/1967620856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7744151330 msgr2=0x7f7744070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:29.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f7748d43640 1 --2- 172.21.15.83:0/1967620856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7744151330 0x7f7744070ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:29.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f7748d43640 1 -- 172.21.15.83:0/1967620856 >> v1:172.21.15.184:6789/0 conn(0x7f77440713e0 legacy=0x7f774406ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:29.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f7748d43640 1 -- 172.21.15.83:0/1967620856 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7744153fe0 con 0x7f77440a9d30 2024-09-19T07:19:29.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f7748d43640 1 --2- 172.21.15.83:0/1967620856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77440a9d30 0x7f774406e4d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f773c009e50 tx=0x7f773c01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dee75d0dd382852e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:29.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f7742ffd640 1 -- 172.21.15.83:0/1967620856 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c0a57c0 con 0x7f77440a9d30 2024-09-19T07:19:29.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f7742ffd640 1 -- 172.21.15.83:0/1967620856 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f773c0a5960 con 0x7f77440a9d30 2024-09-19T07:19:29.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f7742ffd640 1 -- 172.21.15.83:0/1967620856 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c0a5d30 con 0x7f77440a9d30 2024-09-19T07:19:29.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f774a7a6640 1 -- 172.21.15.83:0/1967620856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77440a9d30 msgr2=0x7f774406e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:29.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f774a7a6640 1 --2- 172.21.15.83:0/1967620856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77440a9d30 0x7f774406e4d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f773c009e50 tx=0x7f773c01c8e0 comp rx=0 tx=0).stop 2024-09-19T07:19:29.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f774a7a6640 1 -- 172.21.15.83:0/1967620856 shutdown_connections 2024-09-19T07:19:29.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f774a7a6640 1 --2- 172.21.15.83:0/1967620856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77440a9d30 0x7f774406e4d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:29.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f774a7a6640 1 --2- 172.21.15.83:0/1967620856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7744151330 0x7f7744070ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:29.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f774a7a6640 1 -- 172.21.15.83:0/1967620856 >> 172.21.15.83:0/1967620856 conn(0x7f774406bc80 msgr2=0x7f774406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:29.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f774a7a6640 1 -- 172.21.15.83:0/1967620856 shutdown_connections 2024-09-19T07:19:29.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.247+0000 7f774a7a6640 1 -- 172.21.15.83:0/1967620856 wait complete. 2024-09-19T07:19:29.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f774a7a6640 1 Processor -- start 2024-09-19T07:19:29.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f774a7a6640 1 -- start start 2024-09-19T07:19:29.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f774a7a6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77440713e0 0x7f7744152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:29.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f774a7a6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77440a9d30 0x7f77441f0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:29.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f774a7a6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7744151330 0x7f7744144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:29.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f774a7a6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7744151890 con 0x7f7744151330 2024-09-19T07:19:29.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f774a7a6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7744151710 con 0x7f77440a9d30 2024-09-19T07:19:29.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f774a7a6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7744160cb0 con 0x7f77440713e0 2024-09-19T07:19:29.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7748d43640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7744151330 0x7f7744144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:29.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7748d43640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7744151330 0x7f7744144e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33978/0 (socket says 172.21.15.83:33978) 2024-09-19T07:19:29.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7748d43640 1 -- 172.21.15.83:0/2838791169 learned_addr learned my addr 172.21.15.83:0/2838791169 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:29.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f77437fe640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77440a9d30 0x7f77441f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7743fff640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77440713e0 0x7f7744152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7748d43640 1 -- 172.21.15.83:0/2838791169 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77440713e0 msgr2=0x7f7744152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7748d43640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77440713e0 0x7f7744152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7748d43640 1 -- 172.21.15.83:0/2838791169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77440a9d30 msgr2=0x7f77441f0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7748d43640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77440a9d30 0x7f77441f0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7748d43640 1 -- 172.21.15.83:0/2838791169 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77441453a0 con 0x7f7744151330 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7743fff640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77440713e0 0x7f7744152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7748d43640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7744151330 0x7f7744144e60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f773c01a510 tx=0x7f773c019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7740ff9640 1 -- 172.21.15.83:0/2838791169 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c0bc020 con 0x7f7744151330 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7740ff9640 1 -- 172.21.15.83:0/2838791169 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f773c0b8670 con 0x7f7744151330 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f774a7a6640 1 -- 172.21.15.83:0/2838791169 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7744144b80 con 0x7f7744151330 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f7740ff9640 1 -- 172.21.15.83:0/2838791169 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c0b8a60 con 0x7f7744151330 2024-09-19T07:19:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f774a7a6640 1 -- 172.21.15.83:0/2838791169 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77441eff50 con 0x7f7744151330 2024-09-19T07:19:29.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.251+0000 7f774a7a6640 1 -- 172.21.15.83:0/2838791169 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7744070ea0 con 0x7f7744151330 2024-09-19T07:19:29.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.255+0000 7f7740ff9640 1 -- 172.21.15.83:0/2838791169 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f773c0bf020 con 0x7f7744151330 2024-09-19T07:19:29.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.255+0000 7f7740ff9640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77300678f0 0x7f7730069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:29.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.255+0000 7f7740ff9640 1 -- 172.21.15.83:0/2838791169 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f773c1209a0 con 0x7f7744151330 2024-09-19T07:19:29.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.255+0000 7f7743fff640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77300678f0 0x7f7730069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:29.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.255+0000 7f7743fff640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77300678f0 0x7f7730069db0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7734009a40 tx=0x7f7734008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:29.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.259+0000 7f7740ff9640 1 -- 172.21.15.83:0/2838791169 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f773c0c6030 con 0x7f7744151330 2024-09-19T07:19:29.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:29.419+0000 7f774a7a6640 1 -- 172.21.15.83:0/2838791169 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64} v 0) -- 0x7f774415e0b0 con 0x7f7744151330 2024-09-19T07:19:29.735 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:19:29.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:19:30.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.223+0000 7f7740ff9640 1 -- 172.21.15.83:0/2838791169 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64}]=0 pool 'a_metadata' created v18) ==== 128+0+0 (secure 0 0 0) 0x7f773c016620 con 0x7f7744151330 2024-09-19T07:19:30.225 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'a_metadata' created 2024-09-19T07:19:30.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.227+0000 7f774a7a6640 1 -- 172.21.15.83:0/2838791169 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77300678f0 msgr2=0x7f7730069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.227+0000 7f774a7a6640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77300678f0 0x7f7730069db0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7734009a40 tx=0x7f7734008040 comp rx=0 tx=0).stop 2024-09-19T07:19:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.227+0000 7f774a7a6640 1 -- 172.21.15.83:0/2838791169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7744151330 msgr2=0x7f7744144e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.227+0000 7f774a7a6640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7744151330 0x7f7744144e60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f773c01a510 tx=0x7f773c019680 comp rx=0 tx=0).stop 2024-09-19T07:19:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.227+0000 7f774a7a6640 1 -- 172.21.15.83:0/2838791169 shutdown_connections 2024-09-19T07:19:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.227+0000 7f774a7a6640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77300678f0 0x7f7730069db0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.227+0000 7f774a7a6640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7744151330 0x7f7744144e60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.227+0000 7f774a7a6640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77440a9d30 0x7f77441f0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.227+0000 7f774a7a6640 1 --2- 172.21.15.83:0/2838791169 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77440713e0 0x7f7744152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.227+0000 7f774a7a6640 1 -- 172.21.15.83:0/2838791169 >> 172.21.15.83:0/2838791169 conn(0x7f774406bc80 msgr2=0x7f774406e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:30.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.231+0000 7f774a7a6640 1 -- 172.21.15.83:0/2838791169 shutdown_connections 2024-09-19T07:19:30.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.231+0000 7f774a7a6640 1 -- 172.21.15.83:0/2838791169 wait complete. 2024-09-19T07:19:30.254 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-19T07:19:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.371+0000 7fb4ca8c2640 1 Processor -- start 2024-09-19T07:19:30.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.371+0000 7fb4ca8c2640 1 -- start start 2024-09-19T07:19:30.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4ca8c2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb4c40c32d0 0x7fb4c40c27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:30.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4ca8c2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4c4153e30 0x7fb4c414c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:30.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4ca8c2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4c41538b0 con 0x7fb4c4153450 2024-09-19T07:19:30.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4ca8c2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb4c40c1d70 con 0x7fb4c4153e30 2024-09-19T07:19:30.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4ca8c2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fb4c40c7a80 con 0x7fb4c40c32d0 2024-09-19T07:19:30.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4c8e5f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4c4153e30 0x7fb4c414c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:30.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4c8e5f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4c4153e30 0x7fb4c414c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55398/0 (socket says 172.21.15.83:55398) 2024-09-19T07:19:30.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4c8e5f640 1 -- 172.21.15.83:0/2807278579 learned_addr learned my addr 172.21.15.83:0/2807278579 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:30.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4c3fff640 1 --2- 172.21.15.83:0/2807278579 >> v2:172.21.15.184:3300/0 conn(0x7fb4c40c32d0 0x7fb4c40c27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:30.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4c8e5f640 1 -- 172.21.15.83:0/2807278579 >> v2:172.21.15.184:3300/0 conn(0x7fb4c40c32d0 msgr2=0x7fb4c40c27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:30.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4c8e5f640 1 --2- 172.21.15.83:0/2807278579 >> v2:172.21.15.184:3300/0 conn(0x7fb4c40c32d0 0x7fb4c40c27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:30.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4c8e5f640 1 -- 172.21.15.83:0/2807278579 >> v1:172.21.15.184:6789/0 conn(0x7fb4c4153450 legacy=0x7fb4c40c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:30.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4c8e5f640 1 -- 172.21.15.83:0/2807278579 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c40c7d60 con 0x7fb4c4153e30 2024-09-19T07:19:30.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4c8e5f640 1 --2- 172.21.15.83:0/2807278579 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c4153e30 0x7fb4c414c510 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40c4f10 tx=0x7fb4bc01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5418d40d04ae29 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:30.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4c2ffd640 1 -- 172.21.15.83:0/2807278579 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc0b0070 con 0x7fb4c4153e30 2024-09-19T07:19:30.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4c2ffd640 1 -- 172.21.15.83:0/2807278579 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4bc0ab600 con 0x7fb4c4153e30 2024-09-19T07:19:30.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4c2ffd640 1 -- 172.21.15.83:0/2807278579 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc0ab9d0 con 0x7fb4c4153e30 2024-09-19T07:19:30.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/2807278579 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c4153e30 msgr2=0x7fb4c414c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:30.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.375+0000 7fb4ca8c2640 1 --2- 172.21.15.83:0/2807278579 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c4153e30 0x7fb4c414c510 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40c4f10 tx=0x7fb4bc01c3c0 comp rx=0 tx=0).stop 2024-09-19T07:19:30.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/2807278579 shutdown_connections 2024-09-19T07:19:30.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 --2- 172.21.15.83:0/2807278579 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c4153e30 0x7fb4c414c510 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:30.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 --2- 172.21.15.83:0/2807278579 >> v2:172.21.15.184:3300/0 conn(0x7fb4c40c32d0 0x7fb4c40c27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:30.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/2807278579 >> 172.21.15.83:0/2807278579 conn(0x7fb4c406bc80 msgr2=0x7fb4c406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:30.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/2807278579 shutdown_connections 2024-09-19T07:19:30.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/2807278579 wait complete. 2024-09-19T07:19:30.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 Processor -- start 2024-09-19T07:19:30.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 -- start start 2024-09-19T07:19:30.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c40c32d0 0x7fb4c4072130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:30.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c4153450 0x7fb4c414ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:30.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c4153e30 0x7fb4c414b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c415fe70 con 0x7fb4c4153e30 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c415fcf0 con 0x7fb4c4153450 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb4c415fff0 con 0x7fb4c40c32d0 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4c8e5f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c4153e30 0x7fb4c414b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4c8e5f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c4153e30 0x7fb4c414b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55400/0 (socket says 172.21.15.83:55400) 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4c8e5f640 1 -- 172.21.15.83:0/3528053882 learned_addr learned my addr 172.21.15.83:0/3528053882 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4c8e5f640 1 -- 172.21.15.83:0/3528053882 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c40c32d0 msgr2=0x7fb4c4072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4c8e5f640 1 --2- 172.21.15.83:0/3528053882 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c40c32d0 0x7fb4c4072130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4c8e5f640 1 -- 172.21.15.83:0/3528053882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c4153450 msgr2=0x7fb4c414ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4c8e5f640 1 --2- 172.21.15.83:0/3528053882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c4153450 0x7fb4c414ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4c8e5f640 1 -- 172.21.15.83:0/3528053882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c40bc390 con 0x7fb4c4153e30 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4c8e5f640 1 --2- 172.21.15.83:0/3528053882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c4153e30 0x7fb4c414b220 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc0ae040 tx=0x7fb4bc019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4c0ff9640 1 -- 172.21.15.83:0/3528053882 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc0b0040 con 0x7fb4c4153e30 2024-09-19T07:19:30.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4c0ff9640 1 -- 172.21.15.83:0/3528053882 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4bc0b9070 con 0x7fb4c4153e30 2024-09-19T07:19:30.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/3528053882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c414ba80 con 0x7fb4c4153e30 2024-09-19T07:19:30.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4c0ff9640 1 -- 172.21.15.83:0/3528053882 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc0b4670 con 0x7fb4c4153e30 2024-09-19T07:19:30.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/3528053882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c40c10e0 con 0x7fb4c4153e30 2024-09-19T07:19:30.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.379+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/3528053882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4c40c27c0 con 0x7fb4c4153e30 2024-09-19T07:19:30.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.383+0000 7fb4c0ff9640 1 -- 172.21.15.83:0/3528053882 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb4bc0b4810 con 0x7fb4c4153e30 2024-09-19T07:19:30.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.383+0000 7fb4c0ff9640 1 --2- 172.21.15.83:0/3528053882 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4b00678c0 0x7fb4b0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:30.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.383+0000 7fb4c0ff9640 1 -- 172.21.15.83:0/3528053882 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5222+0+0 (secure 0 0 0) 0x7fb4bc11fef0 con 0x7fb4c4153e30 2024-09-19T07:19:30.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.383+0000 7fb4c3fff640 1 --2- 172.21.15.83:0/3528053882 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4b00678c0 0x7fb4b0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:30.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.383+0000 7fb4c3fff640 1 --2- 172.21.15.83:0/3528053882 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4b00678c0 0x7fb4b0069d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40070b0 tx=0x7fb4b4007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:30.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.387+0000 7fb4c0ff9640 1 -- 172.21.15.83:0/3528053882 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb4bc0160d0 con 0x7fb4c4153e30 2024-09-19T07:19:30.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:30.547+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/3528053882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb4c4152240 con 0x7fb4c4153e30 2024-09-19T07:19:31.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.235+0000 7fb4c0ff9640 1 -- 172.21.15.83:0/3528053882 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'a_data' created v19) ==== 160+0+0 (secure 0 0 0) 0x7fb4bc0f0c80 con 0x7fb4c4153e30 2024-09-19T07:19:31.236 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'a_data' created 2024-09-19T07:19:31.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.239+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/3528053882 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4b00678c0 msgr2=0x7fb4b0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.239+0000 7fb4ca8c2640 1 --2- 172.21.15.83:0/3528053882 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4b00678c0 0x7fb4b0069d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40070b0 tx=0x7fb4b4007040 comp rx=0 tx=0).stop 2024-09-19T07:19:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.239+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/3528053882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c4153e30 msgr2=0x7fb4c414b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.239+0000 7fb4ca8c2640 1 --2- 172.21.15.83:0/3528053882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c4153e30 0x7fb4c414b220 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc0ae040 tx=0x7fb4bc019160 comp rx=0 tx=0).stop 2024-09-19T07:19:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.239+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/3528053882 shutdown_connections 2024-09-19T07:19:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.239+0000 7fb4ca8c2640 1 --2- 172.21.15.83:0/3528053882 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4b00678c0 0x7fb4b0069d80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.239+0000 7fb4ca8c2640 1 --2- 172.21.15.83:0/3528053882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c4153e30 0x7fb4c414b220 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.239+0000 7fb4ca8c2640 1 --2- 172.21.15.83:0/3528053882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c4153450 0x7fb4c414ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.239+0000 7fb4ca8c2640 1 --2- 172.21.15.83:0/3528053882 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c40c32d0 0x7fb4c4072130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.239+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/3528053882 >> 172.21.15.83:0/3528053882 conn(0x7fb4c406bc80 msgr2=0x7fb4c4155590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.239+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/3528053882 shutdown_connections 2024-09-19T07:19:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.239+0000 7fb4ca8c2640 1 -- 172.21.15.83:0/3528053882 wait complete. 2024-09-19T07:19:31.270 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new a a_metadata a_data 2024-09-19T07:19:31.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.395+0000 7f4ebde90640 1 Processor -- start 2024-09-19T07:19:31.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.395+0000 7f4ebde90640 1 -- start start 2024-09-19T07:19:31.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4ebde90640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4eb80c3040 0x7f4eb8154120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:31.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4ebde90640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4eb80c2330 con 0x7f4eb80c3590 2024-09-19T07:19:31.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4ebde90640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4eb80c1d70 con 0x7f4eb8151130 2024-09-19T07:19:31.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4ebde90640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4eb80715d0 con 0x7f4eb80c3040 2024-09-19T07:19:31.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb6ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4eb80c3590 0x7f4eb8153a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41728/0 (socket says 172.21.15.83:41728) 2024-09-19T07:19:31.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb6ffd640 1 -- 172.21.15.83:0/2843432839 learned_addr learned my addr 172.21.15.83:0/2843432839 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:31.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb7fff640 1 --2- 172.21.15.83:0/2843432839 >> v2:172.21.15.184:3300/0 conn(0x7f4eb80c3040 0x7f4eb8154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:31.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3575722151 0 0) 0x7f4eb80c2330 con 0x7f4eb80c3590 2024-09-19T07:19:31.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ea0003530 con 0x7f4eb80c3590 2024-09-19T07:19:31.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 27243734 0 0) 0x7f4ea0003530 con 0x7f4eb80c3590 2024-09-19T07:19:31.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 >> v2:172.21.15.184:3300/0 conn(0x7f4eb80c3040 msgr2=0x7f4eb8154120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:31.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 --2- 172.21.15.83:0/2843432839 >> v2:172.21.15.184:3300/0 conn(0x7f4eb80c3040 0x7f4eb8154120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:31.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 >> v1:172.21.15.184:6789/0 conn(0x7f4eb8151130 legacy=0x7f4eb80c25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eb80c2050 con 0x7f4eb80c3590 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f4eac001680 con 0x7f4eb80c3590 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 >> v1:172.21.15.83:6789/0 conn(0x7f4eb80c3590 legacy=0x7f4eb8153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 --2- 172.21.15.83:0/2843432839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ea00039c0 0x7f4ea0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 --2- 172.21.15.83:0/2843432839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0004810 0x7f4ea0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 --2- 172.21.15.83:0/2843432839 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ea00051d0 0x7f4ea000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb80715d0 con 0x7f4ea00039c0 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea000c170 con 0x7f4ea0004810 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4ea000c370 con 0x7f4ea00051d0 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb77fe640 1 --2- 172.21.15.83:0/2843432839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ea00039c0 0x7f4ea0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb6ffd640 1 --2- 172.21.15.83:0/2843432839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0004810 0x7f4ea0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb77fe640 1 -- 172.21.15.83:0/2843432839 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ea00051d0 msgr2=0x7f4ea000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:31.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb77fe640 1 --2- 172.21.15.83:0/2843432839 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ea00051d0 0x7f4ea000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:31.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb77fe640 1 -- 172.21.15.83:0/2843432839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0004810 msgr2=0x7f4ea0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:31.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb77fe640 1 --2- 172.21.15.83:0/2843432839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0004810 0x7f4ea0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:31.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb77fe640 1 -- 172.21.15.83:0/2843432839 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea0003530 con 0x7f4ea00039c0 2024-09-19T07:19:31.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.399+0000 7f4eb7fff640 1 -- 172.21.15.83:0/2843432839 reap_dead start 2024-09-19T07:19:31.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb77fe640 1 --2- 172.21.15.83:0/2843432839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ea00039c0 0x7f4ea0003de0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8001780 tx=0x7f4ea801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e16375390bc5acc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:31.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea80013a0 con 0x7f4ea00039c0 2024-09-19T07:19:31.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ea80050d0 con 0x7f4ea00039c0 2024-09-19T07:19:31.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb67fc640 1 -- 172.21.15.83:0/2843432839 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8003600 con 0x7f4ea00039c0 2024-09-19T07:19:31.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 -- 172.21.15.83:0/2843432839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ea00039c0 msgr2=0x7f4ea0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:31.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 --2- 172.21.15.83:0/2843432839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ea00039c0 0x7f4ea0003de0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8001780 tx=0x7f4ea801c420 comp rx=0 tx=0).stop 2024-09-19T07:19:31.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 -- 172.21.15.83:0/2843432839 shutdown_connections 2024-09-19T07:19:31.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 --2- 172.21.15.83:0/2843432839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ea00039c0 0x7f4ea0003de0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:31.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 -- 172.21.15.83:0/2843432839 >> 172.21.15.83:0/2843432839 conn(0x7f4eb806bc80 msgr2=0x7f4eb806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:31.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 -- 172.21.15.83:0/2843432839 shutdown_connections 2024-09-19T07:19:31.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 -- 172.21.15.83:0/2843432839 wait complete. 2024-09-19T07:19:31.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 Processor -- start 2024-09-19T07:19:31.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 -- start start 2024-09-19T07:19:31.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea00039c0 0x7f4eb814dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:31.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4eb80c1250 0x7f4eb8149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:31.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb81570c0 0x7f4eb8157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:31.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb8151690 con 0x7f4eb81570c0 2024-09-19T07:19:31.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb8151510 con 0x7f4ea00039c0 2024-09-19T07:19:31.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4eb8153a10 con 0x7f4eb80c1250 2024-09-19T07:19:31.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb81570c0 0x7f4eb8157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:31.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb81570c0 0x7f4eb8157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55422/0 (socket says 172.21.15.83:55422) 2024-09-19T07:19:31.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb7fff640 1 -- 172.21.15.83:0/157623085 learned_addr learned my addr 172.21.15.83:0/157623085 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:31.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb6ffd640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4eb80c1250 0x7f4eb8149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:31.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb77fe640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea00039c0 0x7f4eb814dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:31.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb7fff640 1 -- 172.21.15.83:0/157623085 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4eb80c1250 msgr2=0x7f4eb8149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:31.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb7fff640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4eb80c1250 0x7f4eb8149fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:31.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb7fff640 1 -- 172.21.15.83:0/157623085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea00039c0 msgr2=0x7f4eb814dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:31.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb7fff640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea00039c0 0x7f4eb814dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:31.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb7fff640 1 -- 172.21.15.83:0/157623085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eb81ee530 con 0x7f4eb81570c0 2024-09-19T07:19:31.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4eb7fff640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb81570c0 0x7f4eb8157570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4eb806f240 tx=0x7f4eb000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:31.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4e9ffff640 1 -- 172.21.15.83:0/157623085 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0018020 con 0x7f4eb81570c0 2024-09-19T07:19:31.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4e9ffff640 1 -- 172.21.15.83:0/157623085 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eb000bcb0 con 0x7f4eb81570c0 2024-09-19T07:19:31.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 -- 172.21.15.83:0/157623085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eb80ac430 con 0x7f4eb81570c0 2024-09-19T07:19:31.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4e9ffff640 1 -- 172.21.15.83:0/157623085 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb000be60 con 0x7f4eb81570c0 2024-09-19T07:19:31.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.403+0000 7f4ebde90640 1 -- 172.21.15.83:0/157623085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eb80aa8a0 con 0x7f4eb81570c0 2024-09-19T07:19:31.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.407+0000 7f4ebde90640 1 -- 172.21.15.83:0/157623085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eb80aaaa0 con 0x7f4eb81570c0 2024-09-19T07:19:31.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.407+0000 7f4e9ffff640 1 -- 172.21.15.83:0/157623085 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4eb001b020 con 0x7f4eb81570c0 2024-09-19T07:19:31.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.407+0000 7f4e9ffff640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ea40678c0 0x7f4ea4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:31.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.407+0000 7f4e9ffff640 1 -- 172.21.15.83:0/157623085 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5599+0+0 (secure 0 0 0) 0x7f4eb0086220 con 0x7f4eb81570c0 2024-09-19T07:19:31.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.407+0000 7f4eb77fe640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ea40678c0 0x7f4ea4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:31.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.407+0000 7f4eb77fe640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ea40678c0 0x7f4ea4069d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8001750 tx=0x7f4ea80011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:31.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.411+0000 7f4e9ffff640 1 -- 172.21.15.83:0/157623085 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4eb0022020 con 0x7f4eb81570c0 2024-09-19T07:19:31.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:31.615+0000 7f4ebde90640 1 -- 172.21.15.83:0/157623085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"} v 0) -- 0x7f4eb8153da0 con 0x7f4eb81570c0 2024-09-19T07:19:32.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.279+0000 7f4e9ffff640 1 -- 172.21.15.83:0/157623085 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 157+0+0 (secure 0 0 0) 0x7f4eb0056e40 con 0x7f4eb81570c0 2024-09-19T07:19:32.281 INFO:teuthology.orchestra.run.smithi083.stderr:new fs with metadata pool 2 and data pool 3 2024-09-19T07:19:32.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.287+0000 7f4ebde90640 1 -- 172.21.15.83:0/157623085 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ea40678c0 msgr2=0x7f4ea4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.287+0000 7f4ebde90640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ea40678c0 0x7f4ea4069d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8001750 tx=0x7f4ea80011a0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.287+0000 7f4ebde90640 1 -- 172.21.15.83:0/157623085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb81570c0 msgr2=0x7f4eb8157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.287+0000 7f4ebde90640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb81570c0 0x7f4eb8157570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4eb806f240 tx=0x7f4eb000b480 comp rx=0 tx=0).stop 2024-09-19T07:19:32.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.287+0000 7f4ebde90640 1 -- 172.21.15.83:0/157623085 shutdown_connections 2024-09-19T07:19:32.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.287+0000 7f4ebde90640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ea40678c0 0x7f4ea4069d80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.287+0000 7f4ebde90640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb81570c0 0x7f4eb8157570 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.287+0000 7f4ebde90640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4eb80c1250 0x7f4eb8149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.287+0000 7f4ebde90640 1 --2- 172.21.15.83:0/157623085 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea00039c0 0x7f4eb814dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.287+0000 7f4ebde90640 1 -- 172.21.15.83:0/157623085 >> 172.21.15.83:0/157623085 conn(0x7f4eb806bc80 msgr2=0x7f4eb806d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:32.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.287+0000 7f4ebde90640 1 -- 172.21.15.83:0/157623085 shutdown_connections 2024-09-19T07:19:32.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.287+0000 7f4ebde90640 1 -- 172.21.15.83:0/157623085 wait complete. 2024-09-19T07:19:32.316 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:32.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.427+0000 7fa9949a2640 1 Processor -- start 2024-09-19T07:19:32.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.427+0000 7fa9949a2640 1 -- start start 2024-09-19T07:19:32.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa9949a2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9900c6120 0x7fa9900c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa9949a2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa990156200 con 0x7fa9901592f0 2024-09-19T07:19:32.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa9949a2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9901564e0 con 0x7fa990155d70 2024-09-19T07:19:32.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa9949a2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa9900c9930 con 0x7fa9900c6120 2024-09-19T07:19:32.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa98e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9900c6120 0x7fa9900c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:32.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa98e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9900c6120 0x7fa9900c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55434/0 (socket says 172.21.15.83:55434) 2024-09-19T07:19:32.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa98ed9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa9901592f0 0x7fa990157ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41744/0 (socket says 172.21.15.83:41744) 2024-09-19T07:19:32.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa98e59c640 1 -- 172.21.15.83:0/2182283574 learned_addr learned my addr 172.21.15.83:0/2182283574 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:32.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa98e59c640 1 -- 172.21.15.83:0/2182283574 >> v1:172.21.15.184:6789/0 conn(0x7fa990155d70 legacy=0x7fa9901575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:32.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa98e59c640 1 -- 172.21.15.83:0/2182283574 >> v1:172.21.15.83:6789/0 conn(0x7fa9901592f0 legacy=0x7fa990157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa98d59a640 1 -- 172.21.15.83:0/2182283574 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2293467262 0 0) 0x7fa990156200 con 0x7fa9901592f0 2024-09-19T07:19:32.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa98e59c640 1 -- 172.21.15.83:0/2182283574 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9901567c0 con 0x7fa9900c6120 2024-09-19T07:19:32.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa98e59c640 1 --2- 172.21.15.83:0/2182283574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900c6120 0x7fa9900c6500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa984009870 tx=0x7fa98401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9052d7ac1c51bb4d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:32.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa98d59a640 1 -- 172.21.15.83:0/2182283574 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9840b0070 con 0x7fa9900c6120 2024-09-19T07:19:32.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa98d59a640 1 -- 172.21.15.83:0/2182283574 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9840ab600 con 0x7fa9900c6120 2024-09-19T07:19:32.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.431+0000 7fa98d59a640 1 -- 172.21.15.83:0/2182283574 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9840ab9d0 con 0x7fa9900c6120 2024-09-19T07:19:32.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 -- 172.21.15.83:0/2182283574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900c6120 msgr2=0x7fa9900c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 --2- 172.21.15.83:0/2182283574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900c6120 0x7fa9900c6500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa984009870 tx=0x7fa98401c490 comp rx=0 tx=0).stop 2024-09-19T07:19:32.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 -- 172.21.15.83:0/2182283574 shutdown_connections 2024-09-19T07:19:32.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 --2- 172.21.15.83:0/2182283574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900c6120 0x7fa9900c6500 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 -- 172.21.15.83:0/2182283574 >> 172.21.15.83:0/2182283574 conn(0x7fa99006bc80 msgr2=0x7fa99006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:32.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 -- 172.21.15.83:0/2182283574 shutdown_connections 2024-09-19T07:19:32.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 -- 172.21.15.83:0/2182283574 wait complete. 2024-09-19T07:19:32.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 Processor -- start 2024-09-19T07:19:32.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 -- start start 2024-09-19T07:19:32.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9900c6120 0x7fa990146ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9901592f0 0x7fa9900caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9900c4960 0x7fa9901eed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa990164030 con 0x7fa9901592f0 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa990163eb0 con 0x7fa9900c6120 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa9901641b0 con 0x7fa9900c4960 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa98dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9901592f0 0x7fa9900caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa98dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9901592f0 0x7fa9900caf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55448/0 (socket says 172.21.15.83:55448) 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa98dd9b640 1 -- 172.21.15.83:0/3331568570 learned_addr learned my addr 172.21.15.83:0/3331568570 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa98e59c640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9900c6120 0x7fa990146ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa98dd9b640 1 -- 172.21.15.83:0/3331568570 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9900c4960 msgr2=0x7fa9901eed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa98dd9b640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9900c4960 0x7fa9901eed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa98dd9b640 1 -- 172.21.15.83:0/3331568570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9900c6120 msgr2=0x7fa990146ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa98dd9b640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9900c6120 0x7fa990146ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa98dd9b640 1 -- 172.21.15.83:0/3331568570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99014a9b0 con 0x7fa9901592f0 2024-09-19T07:19:32.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa98dd9b640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9901592f0 0x7fa9900caf50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa98000c450 tx=0x7fa98000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:32.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa97effd640 1 -- 172.21.15.83:0/3331568570 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980012830 con 0x7fa9901592f0 2024-09-19T07:19:32.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa97effd640 1 -- 172.21.15.83:0/3331568570 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa980017070 con 0x7fa9901592f0 2024-09-19T07:19:32.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 -- 172.21.15.83:0/3331568570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9900c2550 con 0x7fa9901592f0 2024-09-19T07:19:32.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa97effd640 1 -- 172.21.15.83:0/3331568570 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980012c30 con 0x7fa9901592f0 2024-09-19T07:19:32.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.435+0000 7fa9949a2640 1 -- 172.21.15.83:0/3331568570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa990150930 con 0x7fa9901592f0 2024-09-19T07:19:32.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.439+0000 7fa9949a2640 1 -- 172.21.15.83:0/3331568570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa990150b30 con 0x7fa9901592f0 2024-09-19T07:19:32.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.439+0000 7fa97effd640 1 -- 172.21.15.83:0/3331568570 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa98001c460 con 0x7fa9901592f0 2024-09-19T07:19:32.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.439+0000 7fa97effd640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9700678f0 0x7fa970069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.439+0000 7fa97effd640 1 -- 172.21.15.83:0/3331568570 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5689+0+0 (secure 0 0 0) 0x7fa980085910 con 0x7fa9901592f0 2024-09-19T07:19:32.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.439+0000 7fa98e59c640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9700678f0 0x7fa970069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:32.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.439+0000 7fa98e59c640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9700678f0 0x7fa970069db0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa9840ad000 tx=0x7fa984008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:32.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.443+0000 7fa97effd640 1 -- 172.21.15.83:0/3331568570 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa990150b30 con 0x7fa9901592f0 2024-09-19T07:19:32.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.607+0000 7fa9949a2640 1 -- 172.21.15.83:0/3331568570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa990156200 con 0x7fa9901592f0 2024-09-19T07:19:32.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.607+0000 7fa97effd640 1 -- 172.21.15.83:0/3331568570 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16658 (secure 0 0 0) 0x7fa9800564f0 con 0x7fa9901592f0 2024-09-19T07:19:32.609 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:32.610 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:31.616727+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:32.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.611+0000 7fa9949a2640 1 -- 172.21.15.83:0/3331568570 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9700678f0 msgr2=0x7fa970069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.611+0000 7fa9949a2640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9700678f0 0x7fa970069db0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa9840ad000 tx=0x7fa984008040 comp rx=0 tx=0).stop 2024-09-19T07:19:32.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.611+0000 7fa9949a2640 1 -- 172.21.15.83:0/3331568570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9901592f0 msgr2=0x7fa9900caf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.611+0000 7fa9949a2640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9901592f0 0x7fa9900caf50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa98000c450 tx=0x7fa98000c910 comp rx=0 tx=0).stop 2024-09-19T07:19:32.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.611+0000 7fa9949a2640 1 -- 172.21.15.83:0/3331568570 shutdown_connections 2024-09-19T07:19:32.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.611+0000 7fa9949a2640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9700678f0 0x7fa970069db0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.611+0000 7fa9949a2640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9900c4960 0x7fa9901eed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.611+0000 7fa9949a2640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9901592f0 0x7fa9900caf50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.611+0000 7fa9949a2640 1 --2- 172.21.15.83:0/3331568570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9900c6120 0x7fa990146ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.611+0000 7fa9949a2640 1 -- 172.21.15.83:0/3331568570 >> 172.21.15.83:0/3331568570 conn(0x7fa99006bc80 msgr2=0x7fa990152a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:32.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.611+0000 7fa9949a2640 1 -- 172.21.15.83:0/3331568570 shutdown_connections 2024-09-19T07:19:32.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.611+0000 7fa9949a2640 1 -- 172.21.15.83:0/3331568570 wait complete. 2024-09-19T07:19:32.629 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:32.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceff486640 1 Processor -- start 2024-09-19T07:19:32.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceff486640 1 -- start start 2024-09-19T07:19:32.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceff486640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8071100 0x7fcef81545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceff486640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef80a9e60 con 0x7fcef80c28e0 2024-09-19T07:19:32.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceff486640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef80aa140 con 0x7fcef806dea0 2024-09-19T07:19:32.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceff486640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcef8071b50 con 0x7fcef8071100 2024-09-19T07:19:32.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fcefca21640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcef80c28e0 0x7fcef8150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41758/0 (socket says 172.21.15.83:41758) 2024-09-19T07:19:32.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fcefca21640 1 -- 172.21.15.83:0/487537783 learned_addr learned my addr 172.21.15.83:0/487537783 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:32.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fcefd222640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8071100 0x7fcef81545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:32.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1997563676 0 0) 0x7fcef80a9e60 con 0x7fcef80c28e0 2024-09-19T07:19:32.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcee4003530 con 0x7fcef80c28e0 2024-09-19T07:19:32.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 369390264 0 0) 0x7fcee4003530 con 0x7fcef80c28e0 2024-09-19T07:19:32.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8071100 msgr2=0x7fcef81545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:32.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8071100 0x7fcef81545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 >> v1:172.21.15.184:6789/0 conn(0x7fcef806dea0 legacy=0x7fcef806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef80c2240 con 0x7fcef80c28e0 2024-09-19T07:19:32.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fcee8001680 con 0x7fcef80c28e0 2024-09-19T07:19:32.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 >> v1:172.21.15.83:6789/0 conn(0x7fcef80c28e0 legacy=0x7fcef8150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcee40039f0 0x7fcee4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8071100 0x7fcee4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcee4004d50 0x7fcee40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcef8071b50 con 0x7fcee4004d50 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcee400bb00 con 0x7fcee40039f0 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcee400bd00 con 0x7fcef8071100 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fcefda23640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcee4004d50 0x7fcee40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fcefd222640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcee40039f0 0x7fcee4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fcefca21640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8071100 0x7fcee4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fcefda23640 1 -- 172.21.15.83:0/487537783 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8071100 msgr2=0x7fcee4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fcefda23640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8071100 0x7fcee4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fcefda23640 1 -- 172.21.15.83:0/487537783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcee40039f0 msgr2=0x7fcee4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fcefda23640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcee40039f0 0x7fcee4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fcefda23640 1 -- 172.21.15.83:0/487537783 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee4003530 con 0x7fcee4004d50 2024-09-19T07:19:32.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.751+0000 7fcefca21640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8071100 0x7fcee4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:32.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fcefda23640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcee4004d50 0x7fcee40051d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcef0001e70 tx=0x7fcef001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8a47a9a0a88d66 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:32.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0001a90 con 0x7fcee4004d50 2024-09-19T07:19:32.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef0003b80 con 0x7fcee4004d50 2024-09-19T07:19:32.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceeffff640 1 -- 172.21.15.83:0/487537783 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0005080 con 0x7fcee4004d50 2024-09-19T07:19:32.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 -- 172.21.15.83:0/487537783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcee4004d50 msgr2=0x7fcee40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 --2- 172.21.15.83:0/487537783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcee4004d50 0x7fcee40051d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcef0001e70 tx=0x7fcef001c780 comp rx=0 tx=0).stop 2024-09-19T07:19:32.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fcefda23640 1 -- 172.21.15.83:0/487537783 reap_dead start 2024-09-19T07:19:32.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 -- 172.21.15.83:0/487537783 shutdown_connections 2024-09-19T07:19:32.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 -- 172.21.15.83:0/487537783 >> 172.21.15.83:0/487537783 conn(0x7fcef806bc80 msgr2=0x7fcef806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:32.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 -- 172.21.15.83:0/487537783 shutdown_connections 2024-09-19T07:19:32.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 -- 172.21.15.83:0/487537783 wait complete. 2024-09-19T07:19:32.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 Processor -- start 2024-09-19T07:19:32.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 -- start start 2024-09-19T07:19:32.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcef814a4f0 0x7fcef8146de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8147320 0x7fcef80c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef80c7410 0x7fcef81eca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcef806ea60 con 0x7fcef80c7410 2024-09-19T07:19:32.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcef80c1ff0 con 0x7fcef814a4f0 2024-09-19T07:19:32.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fceff486640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcef8151650 con 0x7fcef8147320 2024-09-19T07:19:32.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fcefda23640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef80c7410 0x7fcef81eca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fcefda23640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef80c7410 0x7fcef81eca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55468/0 (socket says 172.21.15.83:55468) 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fcefda23640 1 -- 172.21.15.83:0/696262129 learned_addr learned my addr 172.21.15.83:0/696262129 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fcefd222640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcef814a4f0 0x7fcef8146de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.755+0000 7fcefca21640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8147320 0x7fcef80c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fcefda23640 1 -- 172.21.15.83:0/696262129 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8147320 msgr2=0x7fcef80c6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fcefda23640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8147320 0x7fcef80c6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fcefda23640 1 -- 172.21.15.83:0/696262129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcef814a4f0 msgr2=0x7fcef8146de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fcefda23640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcef814a4f0 0x7fcef8146de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fcefda23640 1 -- 172.21.15.83:0/696262129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef814fee0 con 0x7fcef80c7410 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fcefd222640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcef814a4f0 0x7fcef8146de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fcefda23640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef80c7410 0x7fcef81eca40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcef0098190 tx=0x7fcee4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fceedffb640 1 -- 172.21.15.83:0/696262129 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef00b7070 con 0x7fcef80c7410 2024-09-19T07:19:32.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fceedffb640 1 -- 172.21.15.83:0/696262129 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef00012e0 con 0x7fcef80c7410 2024-09-19T07:19:32.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fceff486640 1 -- 172.21.15.83:0/696262129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef8149af0 con 0x7fcef80c7410 2024-09-19T07:19:32.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fceedffb640 1 -- 172.21.15.83:0/696262129 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0001a90 con 0x7fcef80c7410 2024-09-19T07:19:32.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fceff486640 1 -- 172.21.15.83:0/696262129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef80c66d0 con 0x7fcef80c7410 2024-09-19T07:19:32.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fceff486640 1 -- 172.21.15.83:0/696262129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef8147f60 con 0x7fcef80c7410 2024-09-19T07:19:32.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fceedffb640 1 -- 172.21.15.83:0/696262129 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcef00ba070 con 0x7fcef80c7410 2024-09-19T07:19:32.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fceedffb640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcedc0678c0 0x7fcedc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:32.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fceedffb640 1 -- 172.21.15.83:0/696262129 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5689+0+0 (secure 0 0 0) 0x7fcef011e920 con 0x7fcef80c7410 2024-09-19T07:19:32.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fcefd222640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcedc0678c0 0x7fcedc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:32.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.759+0000 7fcefd222640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcedc0678c0 0x7fcedc069d80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009be0 tx=0x7fcef4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:32.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.763+0000 7fceedffb640 1 -- 172.21.15.83:0/696262129 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcef814fee0 con 0x7fcef80c7410 2024-09-19T07:19:32.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.927+0000 7fceff486640 1 -- 172.21.15.83:0/696262129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcef80714e0 con 0x7fcef80c7410 2024-09-19T07:19:32.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceedffb640 1 -- 172.21.15.83:0/696262129 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16658 (secure 0 0 0) 0x7fcef00ac020 con 0x7fcef80c7410 2024-09-19T07:19:32.930 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:32.931 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:31.616727+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceff486640 1 -- 172.21.15.83:0/696262129 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcedc0678c0 msgr2=0x7fcedc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceff486640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcedc0678c0 0x7fcedc069d80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009be0 tx=0x7fcef4008040 comp rx=0 tx=0).stop 2024-09-19T07:19:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceff486640 1 -- 172.21.15.83:0/696262129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef80c7410 msgr2=0x7fcef81eca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceff486640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef80c7410 0x7fcef81eca40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcef0098190 tx=0x7fcee4005840 comp rx=0 tx=0).stop 2024-09-19T07:19:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceff486640 1 -- 172.21.15.83:0/696262129 shutdown_connections 2024-09-19T07:19:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceff486640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcedc0678c0 0x7fcedc069d80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceff486640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef80c7410 0x7fcef81eca40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceff486640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef8147320 0x7fcef80c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceff486640 1 --2- 172.21.15.83:0/696262129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcef814a4f0 0x7fcef8146de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceff486640 1 -- 172.21.15.83:0/696262129 >> 172.21.15.83:0/696262129 conn(0x7fcef806bc80 msgr2=0x7fcef8151d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceff486640 1 -- 172.21.15.83:0/696262129 shutdown_connections 2024-09-19T07:19:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:32.931+0000 7fceff486640 1 -- 172.21.15.83:0/696262129 wait complete. 2024-09-19T07:19:32.949 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a standby_count_wanted 0 2024-09-19T07:19:33.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf95630640 1 Processor -- start 2024-09-19T07:19:33.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf95630640 1 -- start start 2024-09-19T07:19:33.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf95630640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7faf90073720 0x7faf900c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:33.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf95630640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf9014fa90 0x7faf900c1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:33.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf95630640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf90073ee0 con 0x7faf9014f620 2024-09-19T07:19:33.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf95630640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7faf90073c90 con 0x7faf9014fa90 2024-09-19T07:19:33.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf95630640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7faf900c3700 con 0x7faf90073720 2024-09-19T07:19:33.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf8f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf9014fa90 0x7faf900c1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:33.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf8f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf9014fa90 0x7faf900c1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55480/0 (socket says 172.21.15.83:55480) 2024-09-19T07:19:33.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf8f7fe640 1 -- 172.21.15.83:0/2834237154 learned_addr learned my addr 172.21.15.83:0/2834237154 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:33.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf8e7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7faf9014f620 0x7faf9006dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41772/0 (socket says 172.21.15.83:41772) 2024-09-19T07:19:33.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf8effd640 1 --2- 172.21.15.83:0/2834237154 >> v2:172.21.15.184:3300/0 conn(0x7faf90073720 0x7faf900c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:33.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf8f7fe640 1 -- 172.21.15.83:0/2834237154 >> v2:172.21.15.184:3300/0 conn(0x7faf90073720 msgr2=0x7faf900c73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:33.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf8f7fe640 1 --2- 172.21.15.83:0/2834237154 >> v2:172.21.15.184:3300/0 conn(0x7faf90073720 0x7faf900c73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:33.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf8f7fe640 1 -- 172.21.15.83:0/2834237154 >> v1:172.21.15.83:6789/0 conn(0x7faf9014f620 legacy=0x7faf9006dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:33.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf8f7fe640 1 -- 172.21.15.83:0/2834237154 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf900c39e0 con 0x7faf9014fa90 2024-09-19T07:19:33.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf8effd640 1 --2- 172.21.15.83:0/2834237154 >> v2:172.21.15.184:3300/0 conn(0x7faf90073720 0x7faf900c73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:33.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf8f7fe640 1 --2- 172.21.15.83:0/2834237154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf9014fa90 0x7faf900c1f00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faf9006f140 tx=0x7faf8801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab29b5cdd21972a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:33.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.075+0000 7faf8dffb640 1 -- 172.21.15.83:0/2834237154 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf880a5590 con 0x7faf9014fa90 2024-09-19T07:19:33.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf8dffb640 1 -- 172.21.15.83:0/2834237154 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf880a5730 con 0x7faf9014fa90 2024-09-19T07:19:33.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf8dffb640 1 -- 172.21.15.83:0/2834237154 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf880a5b00 con 0x7faf9014fa90 2024-09-19T07:19:33.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 -- 172.21.15.83:0/2834237154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf9014fa90 msgr2=0x7faf900c1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:33.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 --2- 172.21.15.83:0/2834237154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf9014fa90 0x7faf900c1f00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faf9006f140 tx=0x7faf8801c8d0 comp rx=0 tx=0).stop 2024-09-19T07:19:33.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 -- 172.21.15.83:0/2834237154 shutdown_connections 2024-09-19T07:19:33.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 --2- 172.21.15.83:0/2834237154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf9014fa90 0x7faf900c1f00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:33.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 --2- 172.21.15.83:0/2834237154 >> v2:172.21.15.184:3300/0 conn(0x7faf90073720 0x7faf900c73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:33.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 -- 172.21.15.83:0/2834237154 >> 172.21.15.83:0/2834237154 conn(0x7faf9006bc80 msgr2=0x7faf9006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:33.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 -- 172.21.15.83:0/2834237154 shutdown_connections 2024-09-19T07:19:33.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 -- 172.21.15.83:0/2834237154 wait complete. 2024-09-19T07:19:33.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 Processor -- start 2024-09-19T07:19:33.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 -- start start 2024-09-19T07:19:33.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90073720 0x7faf901547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:33.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf9014f620 0x7faf90149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:33.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf9014fa90 0x7faf90149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:33.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf9006e4d0 con 0x7faf9014fa90 2024-09-19T07:19:33.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faf9006e350 con 0x7faf90073720 2024-09-19T07:19:33.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faf9015fbb0 con 0x7faf9014f620 2024-09-19T07:19:33.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf8f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf9014fa90 0x7faf90149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:33.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf8f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf9014fa90 0x7faf90149a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55482/0 (socket says 172.21.15.83:55482) 2024-09-19T07:19:33.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf8f7fe640 1 -- 172.21.15.83:0/1365942962 learned_addr learned my addr 172.21.15.83:0/1365942962 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:33.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf8effd640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90073720 0x7faf901547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:33.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf8f7fe640 1 -- 172.21.15.83:0/1365942962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf9014f620 msgr2=0x7faf90149550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:33.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf8f7fe640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf9014f620 0x7faf90149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:33.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf8f7fe640 1 -- 172.21.15.83:0/1365942962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90073720 msgr2=0x7faf901547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:33.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf8f7fe640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90073720 0x7faf901547e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:33.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf8f7fe640 1 -- 172.21.15.83:0/1365942962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9015d950 con 0x7faf9014fa90 2024-09-19T07:19:33.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf8f7fe640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf9014fa90 0x7faf90149a90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faf880af040 tx=0x7faf88019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:33.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf777fe640 1 -- 172.21.15.83:0/1365942962 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf880bc020 con 0x7faf9014fa90 2024-09-19T07:19:33.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf777fe640 1 -- 172.21.15.83:0/1365942962 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf880b8670 con 0x7faf9014fa90 2024-09-19T07:19:33.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf777fe640 1 -- 172.21.15.83:0/1365942962 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf88009510 con 0x7faf9014fa90 2024-09-19T07:19:33.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 -- 172.21.15.83:0/1365942962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf90156cb0 con 0x7faf9014fa90 2024-09-19T07:19:33.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.079+0000 7faf95630640 1 -- 172.21.15.83:0/1365942962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf900c0b70 con 0x7faf9014fa90 2024-09-19T07:19:33.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.083+0000 7faf95630640 1 -- 172.21.15.83:0/1365942962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf900c0980 con 0x7faf9014fa90 2024-09-19T07:19:33.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.083+0000 7faf777fe640 1 -- 172.21.15.83:0/1365942962 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf880bf020 con 0x7faf9014fa90 2024-09-19T07:19:33.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.083+0000 7faf777fe640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf7c0678c0 0x7faf7c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:33.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.083+0000 7faf777fe640 1 -- 172.21.15.83:0/1365942962 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5689+0+0 (secure 0 0 0) 0x7faf88120b30 con 0x7faf9014fa90 2024-09-19T07:19:33.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.083+0000 7faf8effd640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf7c0678c0 0x7faf7c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:33.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.083+0000 7faf8effd640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf7c0678c0 0x7faf7c069d80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faf80006fd0 tx=0x7faf80008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:33.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.087+0000 7faf777fe640 1 -- 172.21.15.83:0/1365942962 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7faf880c6030 con 0x7faf9014fa90 2024-09-19T07:19:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:33.283+0000 7faf95630640 1 -- 172.21.15.83:0/1365942962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7faf900c1bf0 con 0x7faf9014fa90 2024-09-19T07:19:34.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.283+0000 7faf777fe640 1 -- 172.21.15.83:0/1365942962 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"}]=0 v6) ==== 113+0+0 (secure 0 0 0) 0x7faf880165e0 con 0x7faf9014fa90 2024-09-19T07:19:34.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.287+0000 7faf95630640 1 -- 172.21.15.83:0/1365942962 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf7c0678c0 msgr2=0x7faf7c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:34.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.287+0000 7faf95630640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf7c0678c0 0x7faf7c069d80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faf80006fd0 tx=0x7faf80008040 comp rx=0 tx=0).stop 2024-09-19T07:19:34.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.287+0000 7faf95630640 1 -- 172.21.15.83:0/1365942962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf9014fa90 msgr2=0x7faf90149a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:34.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.287+0000 7faf95630640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf9014fa90 0x7faf90149a90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faf880af040 tx=0x7faf88019670 comp rx=0 tx=0).stop 2024-09-19T07:19:34.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.287+0000 7faf95630640 1 -- 172.21.15.83:0/1365942962 shutdown_connections 2024-09-19T07:19:34.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.287+0000 7faf95630640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf7c0678c0 0x7faf7c069d80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:34.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.287+0000 7faf95630640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf9014fa90 0x7faf90149a90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:34.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.287+0000 7faf95630640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf9014f620 0x7faf90149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:34.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.287+0000 7faf95630640 1 --2- 172.21.15.83:0/1365942962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90073720 0x7faf901547e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:34.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.287+0000 7faf95630640 1 -- 172.21.15.83:0/1365942962 >> 172.21.15.83:0/1365942962 conn(0x7faf9006bc80 msgr2=0x7faf9006dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:34.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.287+0000 7faf95630640 1 -- 172.21.15.83:0/1365942962 shutdown_connections 2024-09-19T07:19:34.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.287+0000 7faf95630640 1 -- 172.21.15.83:0/1365942962 wait complete. 2024-09-19T07:19:34.310 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-09-19T07:19:34.311 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false 2024-09-19T07:19:34.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.427+0000 7f5f0f882640 1 Processor -- start 2024-09-19T07:19:34.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.427+0000 7f5f0f882640 1 -- start start 2024-09-19T07:19:34.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0f882640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f0806dc40 0x7f5f0806e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:34.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0f882640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5f08073720 0x7f5f0814f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:34.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0f882640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0814fb60 0x7f5f080c1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:34.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0f882640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5f08073c90 con 0x7f5f0806dc40 2024-09-19T07:19:34.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0f882640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5f08073ee0 con 0x7f5f08073720 2024-09-19T07:19:34.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0f882640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5f080c3700 con 0x7f5f0814fb60 2024-09-19T07:19:34.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0d61e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f0806dc40 0x7f5f0806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:34.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0d61e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f0806dc40 0x7f5f0806e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55494/0 (socket says 172.21.15.83:55494) 2024-09-19T07:19:34.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0d61e640 1 -- 172.21.15.83:0/265075846 learned_addr learned my addr 172.21.15.83:0/265075846 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:34.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0d61e640 1 -- 172.21.15.83:0/265075846 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0814fb60 msgr2=0x7f5f080c1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:34.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0d61e640 1 --2- 172.21.15.83:0/265075846 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0814fb60 0x7f5f080c1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:34.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0d61e640 1 -- 172.21.15.83:0/265075846 >> v2:172.21.15.184:3300/0 conn(0x7f5f08073720 msgr2=0x7f5f0814f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:34.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0ce1d640 1 --2- 172.21.15.83:0/265075846 >> v2:172.21.15.184:3300/0 conn(0x7f5f08073720 0x7f5f0814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:34.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0d61e640 1 --2- 172.21.15.83:0/265075846 >> v2:172.21.15.184:3300/0 conn(0x7f5f08073720 0x7f5f0814f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:34.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0d61e640 1 -- 172.21.15.83:0/265075846 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f080c39e0 con 0x7f5f0806dc40 2024-09-19T07:19:34.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.431+0000 7f5f0d61e640 1 --2- 172.21.15.83:0/265075846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0806dc40 0x7f5f0806e020 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5f081521f0 tx=0x7f5f0401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed91a3be32e754be server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:34.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5efffff640 1 -- 172.21.15.83:0/265075846 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f040b0070 con 0x7f5f0806dc40 2024-09-19T07:19:34.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5efffff640 1 -- 172.21.15.83:0/265075846 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f040a4e50 con 0x7f5f0806dc40 2024-09-19T07:19:34.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5efffff640 1 -- 172.21.15.83:0/265075846 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f040ab820 con 0x7f5f0806dc40 2024-09-19T07:19:34.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 -- 172.21.15.83:0/265075846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0806dc40 msgr2=0x7f5f0806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:34.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 --2- 172.21.15.83:0/265075846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0806dc40 0x7f5f0806e020 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5f081521f0 tx=0x7f5f0401c400 comp rx=0 tx=0).stop 2024-09-19T07:19:34.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 -- 172.21.15.83:0/265075846 shutdown_connections 2024-09-19T07:19:34.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 --2- 172.21.15.83:0/265075846 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0814fb60 0x7f5f080c1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:34.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 --2- 172.21.15.83:0/265075846 >> v2:172.21.15.184:3300/0 conn(0x7f5f08073720 0x7f5f0814f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:34.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 --2- 172.21.15.83:0/265075846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0806dc40 0x7f5f0806e020 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:34.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 -- 172.21.15.83:0/265075846 >> 172.21.15.83:0/265075846 conn(0x7f5f0806bc80 msgr2=0x7f5f0806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:34.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 -- 172.21.15.83:0/265075846 shutdown_connections 2024-09-19T07:19:34.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 -- 172.21.15.83:0/265075846 wait complete. 2024-09-19T07:19:34.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 Processor -- start 2024-09-19T07:19:34.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 -- start start 2024-09-19T07:19:34.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f0806dc40 0x7f5f08147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f08073720 0x7f5f081f00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0814fb60 0x7f5f08150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5f08155af0 con 0x7f5f08073720 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5f08155970 con 0x7f5f0806dc40 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5f08155c70 con 0x7f5f0814fb60 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0ce1d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f08073720 0x7f5f081f00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0ce1d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f08073720 0x7f5f081f00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55504/0 (socket says 172.21.15.83:55504) 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0ce1d640 1 -- 172.21.15.83:0/541465503 learned_addr learned my addr 172.21.15.83:0/541465503 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0de1f640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0814fb60 0x7f5f08150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0d61e640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f0806dc40 0x7f5f08147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0ce1d640 1 -- 172.21.15.83:0/541465503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0814fb60 msgr2=0x7f5f08150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0ce1d640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0814fb60 0x7f5f08150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0ce1d640 1 -- 172.21.15.83:0/541465503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f0806dc40 msgr2=0x7f5f08147650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:34.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0ce1d640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f0806dc40 0x7f5f08147650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:34.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0ce1d640 1 -- 172.21.15.83:0/541465503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f08153ec0 con 0x7f5f08073720 2024-09-19T07:19:34.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0d61e640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f0806dc40 0x7f5f08147650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:34.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0ce1d640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f08073720 0x7f5f081f00a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800c7f0 tx=0x7f5ef800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:34.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5efdffb640 1 -- 172.21.15.83:0/541465503 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8015070 con 0x7f5f08073720 2024-09-19T07:19:34.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5efdffb640 1 -- 172.21.15.83:0/541465503 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ef8004240 con 0x7f5f08073720 2024-09-19T07:19:34.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5efdffb640 1 -- 172.21.15.83:0/541465503 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8003620 con 0x7f5f08073720 2024-09-19T07:19:34.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 -- 172.21.15.83:0/541465503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f081f0b30 con 0x7f5f08073720 2024-09-19T07:19:34.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.435+0000 7f5f0f882640 1 -- 172.21.15.83:0/541465503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f0815c630 con 0x7f5f08073720 2024-09-19T07:19:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.439+0000 7f5f0f882640 1 -- 172.21.15.83:0/541465503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f080c1bf0 con 0x7f5f08073720 2024-09-19T07:19:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.439+0000 7f5efdffb640 1 -- 172.21.15.83:0/541465503 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5ef80039d0 con 0x7f5f08073720 2024-09-19T07:19:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.439+0000 7f5efdffb640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5eec0678c0 0x7f5eec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.439+0000 7f5efdffb640 1 -- 172.21.15.83:0/541465503 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7f5ef8085cf0 con 0x7f5f08073720 2024-09-19T07:19:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.439+0000 7f5f0d61e640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5eec0678c0 0x7f5eec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.439+0000 7f5f0d61e640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5eec0678c0 0x7f5eec069d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f040ad000 tx=0x7f5f04008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:34.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.443+0000 7f5efdffb640 1 -- 172.21.15.83:0/541465503 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5ef8056830 con 0x7f5f08073720 2024-09-19T07:19:34.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:34.639+0000 7f5f0f882640 1 -- 172.21.15.83:0/541465503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f5f0814f620 con 0x7f5f08073720 2024-09-19T07:19:35.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.283+0000 7f5efdffb640 1 -- 172.21.15.83:0/541465503 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 117+0+0 (secure 0 0 0) 0x7f5ef8056130 con 0x7f5f08073720 2024-09-19T07:19:35.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.287+0000 7f5f0f882640 1 -- 172.21.15.83:0/541465503 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5eec0678c0 msgr2=0x7f5eec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:35.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.287+0000 7f5f0f882640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5eec0678c0 0x7f5eec069d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f040ad000 tx=0x7f5f04008040 comp rx=0 tx=0).stop 2024-09-19T07:19:35.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.287+0000 7f5f0f882640 1 -- 172.21.15.83:0/541465503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f08073720 msgr2=0x7f5f081f00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:35.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.287+0000 7f5f0f882640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f08073720 0x7f5f081f00a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800c7f0 tx=0x7f5ef800ccb0 comp rx=0 tx=0).stop 2024-09-19T07:19:35.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.287+0000 7f5f0f882640 1 -- 172.21.15.83:0/541465503 shutdown_connections 2024-09-19T07:19:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.287+0000 7f5f0f882640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5eec0678c0 0x7f5eec069d80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.287+0000 7f5f0f882640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0814fb60 0x7f5f08150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.287+0000 7f5f0f882640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f08073720 0x7f5f081f00a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.287+0000 7f5f0f882640 1 --2- 172.21.15.83:0/541465503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f0806dc40 0x7f5f08147650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.287+0000 7f5f0f882640 1 -- 172.21.15.83:0/541465503 >> 172.21.15.83:0/541465503 conn(0x7f5f0806bc80 msgr2=0x7f5f080716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.287+0000 7f5f0f882640 1 -- 172.21.15.83:0/541465503 shutdown_connections 2024-09-19T07:19:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.287+0000 7f5f0f882640 1 -- 172.21.15.83:0/541465503 wait complete. 2024-09-19T07:19:35.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a session_timeout 300 2024-09-19T07:19:35.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41ff500640 1 Processor -- start 2024-09-19T07:19:35.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41ff500640 1 -- start start 2024-09-19T07:19:35.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41ff500640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41f8150350 0x7f41f80c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:35.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41ff500640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f41f81564d0 0x7f41f8156960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:35.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41ff500640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41f80abff0 con 0x7f41f80aada0 2024-09-19T07:19:35.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41ff500640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f41f8150730 con 0x7f41f81564d0 2024-09-19T07:19:35.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41ff500640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f41f80a6d00 con 0x7f41f8150350 2024-09-19T07:19:35.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fca9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f41f80aada0 0x7f41f80c6850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41774/0 (socket says 172.21.15.83:41774) 2024-09-19T07:19:35.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fca9b640 1 -- 172.21.15.83:0/17723955 learned_addr learned my addr 172.21.15.83:0/17723955 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:35.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fda9d640 1 --2- 172.21.15.83:0/17723955 >> v2:172.21.15.184:3300/0 conn(0x7f41f81564d0 0x7f41f8156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:35.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fd29c640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41f8150350 0x7f41f80c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:35.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1615598412 0 0) 0x7f41f80abff0 con 0x7f41f80aada0 2024-09-19T07:19:35.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41e4003530 con 0x7f41f80aada0 2024-09-19T07:19:35.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 143223666 0 0) 0x7f41e4003530 con 0x7f41f80aada0 2024-09-19T07:19:35.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41f8150350 msgr2=0x7f41f80c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:35.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41f8150350 0x7f41f80c3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:35.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 >> v2:172.21.15.184:3300/0 conn(0x7f41f81564d0 msgr2=0x7f41f8156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:35.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 --2- 172.21.15.83:0/17723955 >> v2:172.21.15.184:3300/0 conn(0x7f41f81564d0 0x7f41f8156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:35.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f80a6f50 con 0x7f41f80aada0 2024-09-19T07:19:35.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f41e8001680 con 0x7f41f80aada0 2024-09-19T07:19:35.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 >> v1:172.21.15.83:6789/0 conn(0x7f41f80aada0 legacy=0x7f41f80c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:35.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41e40039c0 0x7f41e4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:35.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41f8150350 0x7f41e4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:35.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41e4004d50 0x7f41e40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f41f8150730 con 0x7f41e40039c0 2024-09-19T07:19:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41f80a6d00 con 0x7f41e4004d50 2024-09-19T07:19:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f41e400b9d0 con 0x7f41f8150350 2024-09-19T07:19:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fd29c640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41e40039c0 0x7f41e4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fd29c640 1 -- 172.21.15.83:0/17723955 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41f8150350 msgr2=0x7f41e4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fda9d640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41e4004d50 0x7f41e40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fca9b640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41f8150350 0x7f41e4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fd29c640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41f8150350 0x7f41e4004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fd29c640 1 -- 172.21.15.83:0/17723955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41e4004d50 msgr2=0x7f41e40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fd29c640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41e4004d50 0x7f41e40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fd29c640 1 -- 172.21.15.83:0/17723955 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e4003530 con 0x7f41e40039c0 2024-09-19T07:19:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.435+0000 7f41fda9d640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41e4004d50 0x7f41e40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:35.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.439+0000 7f41fd29c640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41e40039c0 0x7f41e4003de0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009ae0 tx=0x7f41f401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc136008e7f4029 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:35.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.439+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f40096b0 con 0x7f41e40039c0 2024-09-19T07:19:35.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.439+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f40a9450 con 0x7f41e40039c0 2024-09-19T07:19:35.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.439+0000 7f41effff640 1 -- 172.21.15.83:0/17723955 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f40a9840 con 0x7f41e40039c0 2024-09-19T07:19:35.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.439+0000 7f41ff500640 1 -- 172.21.15.83:0/17723955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41e40039c0 msgr2=0x7f41e4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:35.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.439+0000 7f41ff500640 1 --2- 172.21.15.83:0/17723955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41e40039c0 0x7f41e4003de0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009ae0 tx=0x7f41f401ca70 comp rx=0 tx=0).stop 2024-09-19T07:19:35.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.439+0000 7f41fda9d640 1 -- 172.21.15.83:0/17723955 reap_dead start 2024-09-19T07:19:35.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.439+0000 7f41ff500640 1 -- 172.21.15.83:0/17723955 shutdown_connections 2024-09-19T07:19:35.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.439+0000 7f41ff500640 1 -- 172.21.15.83:0/17723955 >> 172.21.15.83:0/17723955 conn(0x7f41f80c7840 msgr2=0x7f41f80c7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:35.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.439+0000 7f41ff500640 1 -- 172.21.15.83:0/17723955 shutdown_connections 2024-09-19T07:19:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.439+0000 7f41ff500640 1 -- 172.21.15.83:0/17723955 wait complete. 2024-09-19T07:19:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41ff500640 1 Processor -- start 2024-09-19T07:19:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41ff500640 1 -- start start 2024-09-19T07:19:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41ff500640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41e40039c0 0x7f41f80a8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41ff500640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41f81e0430 0x7f41f81df560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41ff500640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41f81ed750 0x7f41f814c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41ff500640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f41f80c6f60 con 0x7f41e40039c0 2024-09-19T07:19:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41ff500640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41f8156ea0 con 0x7f41f81e0430 2024-09-19T07:19:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41ff500640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f41f80c70e0 con 0x7f41f81ed750 2024-09-19T07:19:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41fd29c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41e40039c0 0x7f41f80a8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41fd29c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41e40039c0 0x7f41f80a8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55532/0 (socket says 172.21.15.83:55532) 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41fd29c640 1 -- 172.21.15.83:0/321845006 learned_addr learned my addr 172.21.15.83:0/321845006 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41fca9b640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41f81e0430 0x7f41f81df560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41fd29c640 1 -- 172.21.15.83:0/321845006 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41f81ed750 msgr2=0x7f41f814c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41fd29c640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41f81ed750 0x7f41f814c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41fd29c640 1 -- 172.21.15.83:0/321845006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41f81e0430 msgr2=0x7f41f81df560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41fd29c640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41f81e0430 0x7f41f81df560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41fd29c640 1 -- 172.21.15.83:0/321845006 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f81519c0 con 0x7f41e40039c0 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41fca9b640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41f81e0430 0x7f41f81df560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41fd29c640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41e40039c0 0x7f41f80a8170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f41f40ab040 tx=0x7f41f4019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41edffb640 1 -- 172.21.15.83:0/321845006 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f400f000 con 0x7f41e40039c0 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41edffb640 1 -- 172.21.15.83:0/321845006 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f40a9c20 con 0x7f41e40039c0 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41ff500640 1 -- 172.21.15.83:0/321845006 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f81dfaa0 con 0x7f41e40039c0 2024-09-19T07:19:35.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41edffb640 1 -- 172.21.15.83:0/321845006 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4004020 con 0x7f41e40039c0 2024-09-19T07:19:35.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41ff500640 1 -- 172.21.15.83:0/321845006 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f81dea80 con 0x7f41e40039c0 2024-09-19T07:19:35.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41ff500640 1 -- 172.21.15.83:0/321845006 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41f8152a40 con 0x7f41e40039c0 2024-09-19T07:19:35.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41edffb640 1 -- 172.21.15.83:0/321845006 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f41f40c2020 con 0x7f41e40039c0 2024-09-19T07:19:35.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41edffb640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f41dc0678f0 0x7f41dc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:35.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41edffb640 1 -- 172.21.15.83:0/321845006 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7f41f41203b0 con 0x7f41e40039c0 2024-09-19T07:19:35.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.443+0000 7f41fca9b640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f41dc0678f0 0x7f41dc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:35.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.447+0000 7f41fca9b640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f41dc0678f0 0x7f41dc069db0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41e8001210 tx=0x7f41e8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:35.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.451+0000 7f41edffb640 1 -- 172.21.15.83:0/321845006 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f41f81519c0 con 0x7f41e40039c0 2024-09-19T07:19:35.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:35.643+0000 7f41ff500640 1 -- 172.21.15.83:0/321845006 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"} v 0) -- 0x7f41f80c3ae0 con 0x7f41e40039c0 2024-09-19T07:19:36.210 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:19:36.211+0000 7f9e36606640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:19:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.299+0000 7f41edffb640 1 -- 172.21.15.83:0/321845006 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"}]=0 v8) ==== 110+0+0 (secure 0 0 0) 0x7f41f40ad020 con 0x7f41e40039c0 2024-09-19T07:19:36.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.303+0000 7f41ff500640 1 -- 172.21.15.83:0/321845006 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f41dc0678f0 msgr2=0x7f41dc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.303+0000 7f41ff500640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f41dc0678f0 0x7f41dc069db0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41e8001210 tx=0x7f41e8009040 comp rx=0 tx=0).stop 2024-09-19T07:19:36.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.303+0000 7f41ff500640 1 -- 172.21.15.83:0/321845006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41e40039c0 msgr2=0x7f41f80a8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.303+0000 7f41ff500640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41e40039c0 0x7f41f80a8170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f41f40ab040 tx=0x7f41f4019810 comp rx=0 tx=0).stop 2024-09-19T07:19:36.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.303+0000 7f41ff500640 1 -- 172.21.15.83:0/321845006 shutdown_connections 2024-09-19T07:19:36.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.303+0000 7f41ff500640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f41dc0678f0 0x7f41dc069db0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.303+0000 7f41ff500640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41f81ed750 0x7f41f814c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.303+0000 7f41ff500640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41f81e0430 0x7f41f81df560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.303+0000 7f41ff500640 1 --2- 172.21.15.83:0/321845006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41e40039c0 0x7f41f80a8170 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.303+0000 7f41ff500640 1 -- 172.21.15.83:0/321845006 >> 172.21.15.83:0/321845006 conn(0x7f41f80c7840 msgr2=0x7f41f80c6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:36.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.303+0000 7f41ff500640 1 -- 172.21.15.83:0/321845006 shutdown_connections 2024-09-19T07:19:36.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.303+0000 7f41ff500640 1 -- 172.21.15.83:0/321845006 wait complete. 2024-09-19T07:19:36.322 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'a' 2024-09-19T07:19:36.322 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:19:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.439+0000 7f6996d53640 1 Processor -- start 2024-09-19T07:19:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6996d53640 1 -- start start 2024-09-19T07:19:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6996d53640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f69900c6120 0x7f69900c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6996d53640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6990155d70 0x7f69901575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6996d53640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6990156200 con 0x7f69901592f0 2024-09-19T07:19:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6996d53640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f69900c9930 con 0x7f69900c6120 2024-09-19T07:19:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6996d53640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f69901564e0 con 0x7f6990155d70 2024-09-19T07:19:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6994aef640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f69900c6120 0x7f69900c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6994aef640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f69900c6120 0x7f69900c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55536/0 (socket says 172.21.15.83:55536) 2024-09-19T07:19:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f69952f0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f69901592f0 0x7f6990157b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41778/0 (socket says 172.21.15.83:41778) 2024-09-19T07:19:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6994aef640 1 -- 172.21.15.83:0/3281945255 learned_addr learned my addr 172.21.15.83:0/3281945255 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6987fff640 1 --2- 172.21.15.83:0/3281945255 >> v2:172.21.15.184:3300/0 conn(0x7f6990155d70 0x7f69901575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6994aef640 1 -- 172.21.15.83:0/3281945255 >> v2:172.21.15.184:3300/0 conn(0x7f6990155d70 msgr2=0x7f69901575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6994aef640 1 --2- 172.21.15.83:0/3281945255 >> v2:172.21.15.184:3300/0 conn(0x7f6990155d70 0x7f69901575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f69877fe640 1 -- 172.21.15.83:0/3281945255 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3961403060 0 0) 0x7f6990156200 con 0x7f69901592f0 2024-09-19T07:19:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6994aef640 1 -- 172.21.15.83:0/3281945255 >> v1:172.21.15.83:6789/0 conn(0x7f69901592f0 legacy=0x7f6990157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6994aef640 1 -- 172.21.15.83:0/3281945255 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69901567c0 con 0x7f69900c6120 2024-09-19T07:19:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.443+0000 7f6987fff640 1 --2- 172.21.15.83:0/3281945255 >> v2:172.21.15.184:3300/0 conn(0x7f6990155d70 0x7f69901575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6994aef640 1 --2- 172.21.15.83:0/3281945255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69900c6120 0x7f69900c6500 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f699006cca0 tx=0x7f698c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c4a10921417ed9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f69877fe640 1 -- 172.21.15.83:0/3281945255 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c0b0070 con 0x7f69900c6120 2024-09-19T07:19:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f69877fe640 1 -- 172.21.15.83:0/3281945255 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f698c0ab600 con 0x7f69900c6120 2024-09-19T07:19:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f69877fe640 1 -- 172.21.15.83:0/3281945255 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c0ab9d0 con 0x7f69900c6120 2024-09-19T07:19:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 -- 172.21.15.83:0/3281945255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69900c6120 msgr2=0x7f69900c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 --2- 172.21.15.83:0/3281945255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69900c6120 0x7f69900c6500 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f699006cca0 tx=0x7f698c01c520 comp rx=0 tx=0).stop 2024-09-19T07:19:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 -- 172.21.15.83:0/3281945255 shutdown_connections 2024-09-19T07:19:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 --2- 172.21.15.83:0/3281945255 >> v2:172.21.15.184:3300/0 conn(0x7f6990155d70 0x7f69901575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 --2- 172.21.15.83:0/3281945255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69900c6120 0x7f69900c6500 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 -- 172.21.15.83:0/3281945255 >> 172.21.15.83:0/3281945255 conn(0x7f699006bc80 msgr2=0x7f699006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 -- 172.21.15.83:0/3281945255 shutdown_connections 2024-09-19T07:19:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 -- 172.21.15.83:0/3281945255 wait complete. 2024-09-19T07:19:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 Processor -- start 2024-09-19T07:19:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 -- start start 2024-09-19T07:19:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69900c6120 0x7f69901ee0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69901592f0 0x7f69901481d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f69901f1d40 0x7f69901597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6990153860 con 0x7f69900c6120 2024-09-19T07:19:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6990162ea0 con 0x7f69901592f0 2024-09-19T07:19:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.447+0000 7f6996d53640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f699014c390 con 0x7f69901f1d40 2024-09-19T07:19:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6994aef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69900c6120 0x7f69901ee0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6994aef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69900c6120 0x7f69901ee0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55538/0 (socket says 172.21.15.83:55538) 2024-09-19T07:19:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6994aef640 1 -- 172.21.15.83:0/3214043549 learned_addr learned my addr 172.21.15.83:0/3214043549 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6987fff640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69901592f0 0x7f69901481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f69952f0640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f69901f1d40 0x7f69901597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6994aef640 1 -- 172.21.15.83:0/3214043549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f69901f1d40 msgr2=0x7f69901597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6994aef640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f69901f1d40 0x7f69901597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6994aef640 1 -- 172.21.15.83:0/3214043549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69901592f0 msgr2=0x7f69901481d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6994aef640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69901592f0 0x7f69901481d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6994aef640 1 -- 172.21.15.83:0/3214043549 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699014d1c0 con 0x7f69900c6120 2024-09-19T07:19:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6994aef640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69900c6120 0x7f69901ee0a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f698c01ca00 tx=0x7f698c01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f69857fa640 1 -- 172.21.15.83:0/3214043549 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c0ba020 con 0x7f69900c6120 2024-09-19T07:19:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f69857fa640 1 -- 172.21.15.83:0/3214043549 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f698c0b6620 con 0x7f69900c6120 2024-09-19T07:19:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6996d53640 1 -- 172.21.15.83:0/3214043549 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f699014d4a0 con 0x7f69900c6120 2024-09-19T07:19:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f69857fa640 1 -- 172.21.15.83:0/3214043549 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c0b0040 con 0x7f69900c6120 2024-09-19T07:19:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6996d53640 1 -- 172.21.15.83:0/3214043549 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69900c24d0 con 0x7f69900c6120 2024-09-19T07:19:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6996d53640 1 -- 172.21.15.83:0/3214043549 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f699014d670 con 0x7f69900c6120 2024-09-19T07:19:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f69857fa640 1 -- 172.21.15.83:0/3214043549 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f698c0bd020 con 0x7f69900c6120 2024-09-19T07:19:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f69857fa640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f697c0678c0 0x7f697c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f69857fa640 1 -- 172.21.15.83:0/3214043549 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7f698c11fff0 con 0x7f69900c6120 2024-09-19T07:19:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6987fff640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f697c0678c0 0x7f697c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.451+0000 7f6987fff640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f697c0678c0 0x7f697c069d80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f69901488e0 tx=0x7f6980008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:36.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.455+0000 7f69857fa640 1 -- 172.21.15.83:0/3214043549 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f699014d670 con 0x7f69900c6120 2024-09-19T07:19:36.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.651+0000 7f6996d53640 1 -- 172.21.15.83:0/3214043549 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69901f1610 con 0x7f69900c6120 2024-09-19T07:19:36.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.651+0000 7f69857fa640 1 -- 172.21.15.83:0/3214043549 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5216 (secure 0 0 0) 0x7f698c0c4030 con 0x7f69900c6120 2024-09-19T07:19:36.653 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:36.653 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":8,"btime":"2024-09-19T07:19:36:294053+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-09-19T07:19:36.654 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 8 2024-09-19T07:19:36.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.655+0000 7f6996d53640 1 -- 172.21.15.83:0/3214043549 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f697c0678c0 msgr2=0x7f697c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.655+0000 7f6996d53640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f697c0678c0 0x7f697c069d80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f69901488e0 tx=0x7f6980008040 comp rx=0 tx=0).stop 2024-09-19T07:19:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.655+0000 7f6996d53640 1 -- 172.21.15.83:0/3214043549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69900c6120 msgr2=0x7f69901ee0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.655+0000 7f6996d53640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69900c6120 0x7f69901ee0a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f698c01ca00 tx=0x7f698c01ad60 comp rx=0 tx=0).stop 2024-09-19T07:19:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.655+0000 7f6996d53640 1 -- 172.21.15.83:0/3214043549 shutdown_connections 2024-09-19T07:19:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.655+0000 7f6996d53640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f697c0678c0 0x7f697c069d80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.655+0000 7f6996d53640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f69901f1d40 0x7f69901597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.655+0000 7f6996d53640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69901592f0 0x7f69901481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.655+0000 7f6996d53640 1 --2- 172.21.15.83:0/3214043549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69900c6120 0x7f69901ee0a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.655+0000 7f6996d53640 1 -- 172.21.15.83:0/3214043549 >> 172.21.15.83:0/3214043549 conn(0x7f699006bc80 msgr2=0x7f69901528a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.655+0000 7f6996d53640 1 -- 172.21.15.83:0/3214043549 shutdown_connections 2024-09-19T07:19:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.655+0000 7f6996d53640 1 -- 172.21.15.83:0/3214043549 wait complete. 2024-09-19T07:19:36.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:36.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.791+0000 7f4eca10b640 1 Processor -- start 2024-09-19T07:19:36.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.791+0000 7f4eca10b640 1 -- start start 2024-09-19T07:19:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4eca10b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4ec40c28e0 0x7f4ec41545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4eca10b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec40a9e60 con 0x7f4ec406dea0 2024-09-19T07:19:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4eca10b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec40aa140 con 0x7f4ec4071100 2024-09-19T07:19:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4eca10b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f4ec4071b50 con 0x7f4ec40c28e0 2024-09-19T07:19:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4ebaffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4ec40c28e0 0x7f4ec41545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4ec3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4ec406dea0 0x7f4ec406e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41784/0 (socket says 172.21.15.83:41784) 2024-09-19T07:19:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4ec3fff640 1 -- 172.21.15.83:0/1368760640 learned_addr learned my addr 172.21.15.83:0/1368760640 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4ec2ffd640 1 -- 172.21.15.83:0/1368760640 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4110863270 0 0) 0x7f4ec40a9e60 con 0x7f4ec406dea0 2024-09-19T07:19:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4ec2ffd640 1 -- 172.21.15.83:0/1368760640 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4eb0003530 con 0x7f4ec406dea0 2024-09-19T07:19:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4ebaffd640 1 -- 172.21.15.83:0/1368760640 >> v1:172.21.15.184:6789/0 conn(0x7f4ec4071100 legacy=0x7f4ec4150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4ebaffd640 1 -- 172.21.15.83:0/1368760640 >> v1:172.21.15.83:6789/0 conn(0x7f4ec406dea0 legacy=0x7f4ec406e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4ebaffd640 1 -- 172.21.15.83:0/1368760640 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec40c2240 con 0x7f4ec40c28e0 2024-09-19T07:19:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4ebaffd640 1 --2- 172.21.15.83:0/1368760640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ec40c28e0 0x7f4ec41545e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4eac0097c0 tx=0x7f4eac01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c8b42d0a38de18 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4ec2ffd640 1 -- 172.21.15.83:0/1368760640 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac0b0070 con 0x7f4ec40c28e0 2024-09-19T07:19:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4ec2ffd640 1 -- 172.21.15.83:0/1368760640 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eac0ab600 con 0x7f4ec40c28e0 2024-09-19T07:19:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4ec2ffd640 1 -- 172.21.15.83:0/1368760640 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac0ab9d0 con 0x7f4ec40c28e0 2024-09-19T07:19:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4eca10b640 1 -- 172.21.15.83:0/1368760640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ec40c28e0 msgr2=0x7f4ec41545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4eca10b640 1 --2- 172.21.15.83:0/1368760640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ec40c28e0 0x7f4ec41545e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4eac0097c0 tx=0x7f4eac01c410 comp rx=0 tx=0).stop 2024-09-19T07:19:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4eca10b640 1 -- 172.21.15.83:0/1368760640 shutdown_connections 2024-09-19T07:19:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4eca10b640 1 --2- 172.21.15.83:0/1368760640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ec40c28e0 0x7f4ec41545e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4eca10b640 1 -- 172.21.15.83:0/1368760640 >> 172.21.15.83:0/1368760640 conn(0x7f4ec406bc80 msgr2=0x7f4ec406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:36.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.795+0000 7f4eca10b640 1 -- 172.21.15.83:0/1368760640 shutdown_connections 2024-09-19T07:19:36.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4eca10b640 1 -- 172.21.15.83:0/1368760640 wait complete. 2024-09-19T07:19:36.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4eca10b640 1 Processor -- start 2024-09-19T07:19:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4eca10b640 1 -- start start 2024-09-19T07:19:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4eca10b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ec406dea0 0x7f4ec40c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4eca10b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ec4071100 0x7f4ec4144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4eca10b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ec40c28e0 0x7f4ec41453a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4eca10b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec4071660 con 0x7f4ec40c28e0 2024-09-19T07:19:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4eca10b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec40714e0 con 0x7f4ec4071100 2024-09-19T07:19:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4eca10b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4ec4161c70 con 0x7f4ec406dea0 2024-09-19T07:19:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4ec3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ec40c28e0 0x7f4ec41453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4ec3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ec40c28e0 0x7f4ec41453a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55548/0 (socket says 172.21.15.83:55548) 2024-09-19T07:19:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4ec3fff640 1 -- 172.21.15.83:0/179145967 learned_addr learned my addr 172.21.15.83:0/179145967 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4ec3fff640 1 -- 172.21.15.83:0/179145967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ec406dea0 msgr2=0x7f4ec40c0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4ec3fff640 1 --2- 172.21.15.83:0/179145967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ec406dea0 0x7f4ec40c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4ec3fff640 1 -- 172.21.15.83:0/179145967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ec4071100 msgr2=0x7f4ec4144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4ec3fff640 1 --2- 172.21.15.83:0/179145967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ec4071100 0x7f4ec4144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4ec3fff640 1 -- 172.21.15.83:0/179145967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec404ffc0 con 0x7f4ec40c28e0 2024-09-19T07:19:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4ec3fff640 1 --2- 172.21.15.83:0/179145967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ec40c28e0 0x7f4ec41453a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00caa0 tx=0x7f4ebc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4ec0ff9640 1 -- 172.21.15.83:0/179145967 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc017070 con 0x7f4ec40c28e0 2024-09-19T07:19:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4ec0ff9640 1 -- 172.21.15.83:0/179145967 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ebc00ac00 con 0x7f4ec40c28e0 2024-09-19T07:19:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4eca10b640 1 -- 172.21.15.83:0/179145967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec4153150 con 0x7f4ec40c28e0 2024-09-19T07:19:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4ec0ff9640 1 -- 172.21.15.83:0/179145967 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc012400 con 0x7f4ec40c28e0 2024-09-19T07:19:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4eca10b640 1 -- 172.21.15.83:0/179145967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec4153710 con 0x7f4ec40c28e0 2024-09-19T07:19:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.799+0000 7f4eca10b640 1 -- 172.21.15.83:0/179145967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ec4151400 con 0x7f4ec40c28e0 2024-09-19T07:19:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.803+0000 7f4ec0ff9640 1 -- 172.21.15.83:0/179145967 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4ebc01a070 con 0x7f4ec40c28e0 2024-09-19T07:19:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.803+0000 7f4ec0ff9640 1 --2- 172.21.15.83:0/179145967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ea40678c0 0x7f4ea4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.803+0000 7f4ec0ff9640 1 -- 172.21.15.83:0/179145967 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7f4ebc0859a0 con 0x7f4ec40c28e0 2024-09-19T07:19:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.803+0000 7f4ec37fe640 1 --2- 172.21.15.83:0/179145967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ea40678c0 0x7f4ea4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:36.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.803+0000 7f4ec37fe640 1 --2- 172.21.15.83:0/179145967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ea40678c0 0x7f4ea4069d80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4000f70 tx=0x7f4eb40079e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:36.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.807+0000 7f4ec0ff9640 1 -- 172.21.15.83:0/179145967 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4ebc056580 con 0x7f4ec40c28e0 2024-09-19T07:19:36.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.967+0000 7f4eca10b640 1 -- 172.21.15.83:0/179145967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ec40c4f30 con 0x7f4ec40c28e0 2024-09-19T07:19:36.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.967+0000 7f4ec0ff9640 1 -- 172.21.15.83:0/179145967 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7f4ebc055e80 con 0x7f4ec40c28e0 2024-09-19T07:19:36.969 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:36.969 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":21,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:33.240224+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:36.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.971+0000 7f4eca10b640 1 -- 172.21.15.83:0/179145967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ea40678c0 msgr2=0x7f4ea4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.971+0000 7f4eca10b640 1 --2- 172.21.15.83:0/179145967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ea40678c0 0x7f4ea4069d80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4000f70 tx=0x7f4eb40079e0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.971+0000 7f4eca10b640 1 -- 172.21.15.83:0/179145967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ec40c28e0 msgr2=0x7f4ec41453a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.971+0000 7f4eca10b640 1 --2- 172.21.15.83:0/179145967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ec40c28e0 0x7f4ec41453a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00caa0 tx=0x7f4ebc00cf60 comp rx=0 tx=0).stop 2024-09-19T07:19:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.971+0000 7f4eca10b640 1 -- 172.21.15.83:0/179145967 shutdown_connections 2024-09-19T07:19:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.971+0000 7f4eca10b640 1 --2- 172.21.15.83:0/179145967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ea40678c0 0x7f4ea4069d80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.971+0000 7f4eca10b640 1 --2- 172.21.15.83:0/179145967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ec40c28e0 0x7f4ec41453a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.971+0000 7f4eca10b640 1 --2- 172.21.15.83:0/179145967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ec4071100 0x7f4ec4144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.971+0000 7f4eca10b640 1 --2- 172.21.15.83:0/179145967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ec406dea0 0x7f4ec40c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.971+0000 7f4eca10b640 1 -- 172.21.15.83:0/179145967 >> 172.21.15.83:0/179145967 conn(0x7f4ec406bc80 msgr2=0x7f4ec406e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.971+0000 7f4eca10b640 1 -- 172.21.15.83:0/179145967 shutdown_connections 2024-09-19T07:19:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:36.971+0000 7f4eca10b640 1 -- 172.21.15.83:0/179145967 wait complete. 2024-09-19T07:19:36.987 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-19T07:19:36.988 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:19:37.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d36a6640 1 Processor -- start 2024-09-19T07:19:37.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d36a6640 1 -- start start 2024-09-19T07:19:37.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d36a6640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb2cc155d70 0x7fb2cc0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d36a6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb2cc1592f0 0x7fb2cc157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d36a6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2cc156200 con 0x7fb2cc0c6120 2024-09-19T07:19:37.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d36a6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb2cc0c9930 con 0x7fb2cc1592f0 2024-09-19T07:19:37.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d36a6640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fb2cc1564e0 con 0x7fb2cc155d70 2024-09-19T07:19:37.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d1c43640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb2cc1592f0 0x7fb2cc157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d1c43640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb2cc1592f0 0x7fb2cc157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55554/0 (socket says 172.21.15.83:55554) 2024-09-19T07:19:37.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d1c43640 1 -- 172.21.15.83:0/3085991655 learned_addr learned my addr 172.21.15.83:0/3085991655 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:37.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d0c41640 1 --2- 172.21.15.83:0/3085991655 >> v2:172.21.15.184:3300/0 conn(0x7fb2cc155d70 0x7fb2cc0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d1c43640 1 -- 172.21.15.83:0/3085991655 >> v2:172.21.15.184:3300/0 conn(0x7fb2cc155d70 msgr2=0x7fb2cc0c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:37.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d1c43640 1 --2- 172.21.15.83:0/3085991655 >> v2:172.21.15.184:3300/0 conn(0x7fb2cc155d70 0x7fb2cc0c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d1c43640 1 -- 172.21.15.83:0/3085991655 >> v1:172.21.15.184:6789/0 conn(0x7fb2cc0c6120 legacy=0x7fb2cc1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:37.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d1c43640 1 -- 172.21.15.83:0/3085991655 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2cc1567c0 con 0x7fb2cc1592f0 2024-09-19T07:19:37.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2d1c43640 1 --2- 172.21.15.83:0/3085991655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc1592f0 0x7fb2cc157ce0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc06cca0 tx=0x7fb2c401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b077ee6661c6892 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:37.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.107+0000 7fb2c3fff640 1 -- 172.21.15.83:0/3085991655 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c40a5670 con 0x7fb2cc1592f0 2024-09-19T07:19:37.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2c3fff640 1 -- 172.21.15.83:0/3085991655 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2c40a5810 con 0x7fb2cc1592f0 2024-09-19T07:19:37.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2c3fff640 1 -- 172.21.15.83:0/3085991655 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c40a5be0 con 0x7fb2cc1592f0 2024-09-19T07:19:37.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 -- 172.21.15.83:0/3085991655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc1592f0 msgr2=0x7fb2cc157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 --2- 172.21.15.83:0/3085991655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc1592f0 0x7fb2cc157ce0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc06cca0 tx=0x7fb2c401c9b0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 -- 172.21.15.83:0/3085991655 shutdown_connections 2024-09-19T07:19:37.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 --2- 172.21.15.83:0/3085991655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc1592f0 0x7fb2cc157ce0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 --2- 172.21.15.83:0/3085991655 >> v2:172.21.15.184:3300/0 conn(0x7fb2cc155d70 0x7fb2cc0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 -- 172.21.15.83:0/3085991655 >> 172.21.15.83:0/3085991655 conn(0x7fb2cc06bc80 msgr2=0x7fb2cc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 -- 172.21.15.83:0/3085991655 shutdown_connections 2024-09-19T07:19:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 -- 172.21.15.83:0/3085991655 wait complete. 2024-09-19T07:19:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 Processor -- start 2024-09-19T07:19:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 -- start start 2024-09-19T07:19:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc0c6120 0x7fb2cc14b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb2cc1592f0 0x7fb2cc1ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb2cc0b1700 0x7fb2cc1eec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc153860 con 0x7fb2cc0c6120 2024-09-19T07:19:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc162ea0 con 0x7fb2cc0b1700 2024-09-19T07:19:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb2cc14bd00 con 0x7fb2cc1592f0 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d1442640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc0c6120 0x7fb2cc14b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d1442640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc0c6120 0x7fb2cc14b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55558/0 (socket says 172.21.15.83:55558) 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d1442640 1 -- 172.21.15.83:0/1599955925 learned_addr learned my addr 172.21.15.83:0/1599955925 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d0c41640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb2cc1592f0 0x7fb2cc1ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d1c43640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb2cc0b1700 0x7fb2cc1eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d1442640 1 -- 172.21.15.83:0/1599955925 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb2cc1592f0 msgr2=0x7fb2cc1ee6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d1442640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb2cc1592f0 0x7fb2cc1ee6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d1442640 1 -- 172.21.15.83:0/1599955925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb2cc0b1700 msgr2=0x7fb2cc1eec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d1442640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb2cc0b1700 0x7fb2cc1eec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d1442640 1 -- 172.21.15.83:0/1599955925 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2cc0adce0 con 0x7fb2cc0c6120 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d0c41640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb2cc1592f0 0x7fb2cc1ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d1442640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc0c6120 0x7fb2cc14b2d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800c480 tx=0x7fb2c800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2c1ffb640 1 -- 172.21.15.83:0/1599955925 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8012830 con 0x7fb2cc0c6120 2024-09-19T07:19:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2c1ffb640 1 -- 172.21.15.83:0/1599955925 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2c8017070 con 0x7fb2cc0c6120 2024-09-19T07:19:37.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 -- 172.21.15.83:0/1599955925 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2cc0c0850 con 0x7fb2cc0c6120 2024-09-19T07:19:37.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2c1ffb640 1 -- 172.21.15.83:0/1599955925 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8012c30 con 0x7fb2cc0c6120 2024-09-19T07:19:37.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.111+0000 7fb2d36a6640 1 -- 172.21.15.83:0/1599955925 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2cc14bf00 con 0x7fb2cc0c6120 2024-09-19T07:19:37.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.115+0000 7fb2d36a6640 1 -- 172.21.15.83:0/1599955925 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2cc0c0640 con 0x7fb2cc0c6120 2024-09-19T07:19:37.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.115+0000 7fb2c1ffb640 1 -- 172.21.15.83:0/1599955925 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb2c801c460 con 0x7fb2cc0c6120 2024-09-19T07:19:37.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.115+0000 7fb2c1ffb640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2b00678f0 0x7fb2b0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.115+0000 7fb2c1ffb640 1 -- 172.21.15.83:0/1599955925 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7fb2c8085a30 con 0x7fb2cc0c6120 2024-09-19T07:19:37.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.115+0000 7fb2d0c41640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2b00678f0 0x7fb2b0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.115+0000 7fb2d0c41640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2b00678f0 0x7fb2b0069db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0c9010 tx=0x7fb2bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:37.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.119+0000 7fb2c1ffb640 1 -- 172.21.15.83:0/1599955925 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb2c8056610 con 0x7fb2cc0c6120 2024-09-19T07:19:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.267+0000 7fb2d36a6640 1 -- 172.21.15.83:0/1599955925 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb2cc0c6500 con 0x7fb2b00678f0 2024-09-19T07:19:37.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.283+0000 7fb2c1ffb640 1 -- 172.21.15.83:0/1599955925 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251952 (secure 0 0 0) 0x7fb2cc0c6500 con 0x7fb2b00678f0 2024-09-19T07:19:37.288 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:37.289 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:19:37.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.287+0000 7fb2d36a6640 1 -- 172.21.15.83:0/1599955925 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2b00678f0 msgr2=0x7fb2b0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.287+0000 7fb2d36a6640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2b00678f0 0x7fb2b0069db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0c9010 tx=0x7fb2bc008040 comp rx=0 tx=0).stop 2024-09-19T07:19:37.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.287+0000 7fb2d36a6640 1 -- 172.21.15.83:0/1599955925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc0c6120 msgr2=0x7fb2cc14b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.287+0000 7fb2d36a6640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc0c6120 0x7fb2cc14b2d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800c480 tx=0x7fb2c800c940 comp rx=0 tx=0).stop 2024-09-19T07:19:37.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.287+0000 7fb2d36a6640 1 -- 172.21.15.83:0/1599955925 shutdown_connections 2024-09-19T07:19:37.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.287+0000 7fb2d36a6640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2b00678f0 0x7fb2b0069db0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.287+0000 7fb2d36a6640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb2cc0b1700 0x7fb2cc1eec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.287+0000 7fb2d36a6640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb2cc1592f0 0x7fb2cc1ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.287+0000 7fb2d36a6640 1 --2- 172.21.15.83:0/1599955925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc0c6120 0x7fb2cc14b2d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.287+0000 7fb2d36a6640 1 -- 172.21.15.83:0/1599955925 >> 172.21.15.83:0/1599955925 conn(0x7fb2cc06bc80 msgr2=0x7fb2cc159700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:37.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.287+0000 7fb2d36a6640 1 -- 172.21.15.83:0/1599955925 shutdown_connections 2024-09-19T07:19:37.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.287+0000 7fb2d36a6640 1 -- 172.21.15.83:0/1599955925 wait complete. 2024-09-19T07:19:37.304 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-19T07:19:36.812237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219172,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755548,"statfs":{"total":773094113280,"available":772869681152,"internally_reserved":0,"allocated":2957312,"data_stored":1778220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001596"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613309+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:19:33.613309+0000","last_peered":"2024-09-19T07:19:33.613309+0000","last_clean":"2024-09-19T07:19:33.613309+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:19:33.613309+0000","last_undegraded":"2024-09-19T07:19:33.613309+0000","last_fullsized":"2024-09-19T07:19:33.613309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246945+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:19:33.246945+0000","last_peered":"2024-09-19T07:19:33.246945+0000","last_clean":"2024-09-19T07:19:33.246945+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:19:33.246945+0000","last_undegraded":"2024-09-19T07:19:33.246945+0000","last_fullsized":"2024-09-19T07:19:33.246945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664687+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:19:33.664687+0000","last_peered":"2024-09-19T07:19:33.664687+0000","last_clean":"2024-09-19T07:19:33.664687+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:19:33.664687+0000","last_undegraded":"2024-09-19T07:19:33.664687+0000","last_fullsized":"2024-09-19T07:19:33.664687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:25.667168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.250169+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:19:33.250169+0000","last_peered":"2024-09-19T07:19:33.250169+0000","last_clean":"2024-09-19T07:19:33.250169+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:19:33.250169+0000","last_undegraded":"2024-09-19T07:19:33.250169+0000","last_fullsized":"2024-09-19T07:19:33.250169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612616+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:19:33.612616+0000","last_peered":"2024-09-19T07:19:33.612616+0000","last_clean":"2024-09-19T07:19:33.612616+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:19:33.612616+0000","last_undegraded":"2024-09-19T07:19:33.612616+0000","last_fullsized":"2024-09-19T07:19:33.612616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249303+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:19:33.249303+0000","last_peered":"2024-09-19T07:19:33.249303+0000","last_clean":"2024-09-19T07:19:33.249303+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:19:33.249303+0000","last_undegraded":"2024-09-19T07:19:33.249303+0000","last_fullsized":"2024-09-19T07:19:33.249303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612363+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:19:33.612363+0000","last_peered":"2024-09-19T07:19:33.612363+0000","last_clean":"2024-09-19T07:19:33.612363+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:19:33.612363+0000","last_undegraded":"2024-09-19T07:19:33.612363+0000","last_fullsized":"2024-09-19T07:19:33.612363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663377+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:19:33.663377+0000","last_peered":"2024-09-19T07:19:33.663377+0000","last_clean":"2024-09-19T07:19:33.663377+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:19:33.663377+0000","last_undegraded":"2024-09-19T07:19:33.663377+0000","last_fullsized":"2024-09-19T07:19:33.663377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613374+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:19:33.613374+0000","last_peered":"2024-09-19T07:19:33.613374+0000","last_clean":"2024-09-19T07:19:33.613374+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:19:33.613374+0000","last_undegraded":"2024-09-19T07:19:33.613374+0000","last_fullsized":"2024-09-19T07:19:33.613374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.295259+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:19:33.295259+0000","last_peered":"2024-09-19T07:19:33.295259+0000","last_clean":"2024-09-19T07:19:33.295259+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:19:33.295259+0000","last_undegraded":"2024-09-19T07:19:33.295259+0000","last_fullsized":"2024-09-19T07:19:33.295259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294981+0000","last_change":"2024-09-19T07:19:32.299956+0000","last_active":"2024-09-19T07:19:33.294981+0000","last_peered":"2024-09-19T07:19:33.294981+0000","last_clean":"2024-09-19T07:19:33.294981+0000","last_became_active":"2024-09-19T07:19:32.299168+0000","last_became_peered":"2024-09-19T07:19:32.299168+0000","last_unstale":"2024-09-19T07:19:33.294981+0000","last_undegraded":"2024-09-19T07:19:33.294981+0000","last_fullsized":"2024-09-19T07:19:33.294981+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.251583+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:19:33.251583+0000","last_peered":"2024-09-19T07:19:33.251583+0000","last_clean":"2024-09-19T07:19:33.251583+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:19:33.251583+0000","last_undegraded":"2024-09-19T07:19:33.251583+0000","last_fullsized":"2024-09-19T07:19:33.251583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664051+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:19:33.664051+0000","last_peered":"2024-09-19T07:19:33.664051+0000","last_clean":"2024-09-19T07:19:33.664051+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:19:33.664051+0000","last_undegraded":"2024-09-19T07:19:33.664051+0000","last_fullsized":"2024-09-19T07:19:33.664051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.665003+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:19:33.665003+0000","last_peered":"2024-09-19T07:19:33.665003+0000","last_clean":"2024-09-19T07:19:33.665003+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:19:33.665003+0000","last_undegraded":"2024-09-19T07:19:33.665003+0000","last_fullsized":"2024-09-19T07:19:33.665003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:43.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246656+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:19:33.246656+0000","last_peered":"2024-09-19T07:19:33.246656+0000","last_clean":"2024-09-19T07:19:33.246656+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:19:33.246656+0000","last_undegraded":"2024-09-19T07:19:33.246656+0000","last_fullsized":"2024-09-19T07:19:33.246656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249042+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:19:33.249042+0000","last_peered":"2024-09-19T07:19:33.249042+0000","last_clean":"2024-09-19T07:19:33.249042+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:19:33.249042+0000","last_undegraded":"2024-09-19T07:19:33.249042+0000","last_fullsized":"2024-09-19T07:19:33.249042+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.251207+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:19:33.251207+0000","last_peered":"2024-09-19T07:19:33.251207+0000","last_clean":"2024-09-19T07:19:33.251207+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:19:33.251207+0000","last_undegraded":"2024-09-19T07:19:33.251207+0000","last_fullsized":"2024-09-19T07:19:33.251207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.662152+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:19:33.662152+0000","last_peered":"2024-09-19T07:19:33.662152+0000","last_clean":"2024-09-19T07:19:33.662152+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:19:33.662152+0000","last_undegraded":"2024-09-19T07:19:33.662152+0000","last_fullsized":"2024-09-19T07:19:33.662152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:51.710997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613530+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:19:33.613530+0000","last_peered":"2024-09-19T07:19:33.613530+0000","last_clean":"2024-09-19T07:19:33.613530+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:19:33.613530+0000","last_undegraded":"2024-09-19T07:19:33.613530+0000","last_fullsized":"2024-09-19T07:19:33.613530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294874+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:19:33.294874+0000","last_peered":"2024-09-19T07:19:33.294874+0000","last_clean":"2024-09-19T07:19:33.294874+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:19:33.294874+0000","last_undegraded":"2024-09-19T07:19:33.294874+0000","last_fullsized":"2024-09-19T07:19:33.294874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.667108+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:19:33.667108+0000","last_peered":"2024-09-19T07:19:33.667108+0000","last_clean":"2024-09-19T07:19:33.667108+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:19:33.667108+0000","last_undegraded":"2024-09-19T07:19:33.667108+0000","last_fullsized":"2024-09-19T07:19:33.667108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:19.033192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249356+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:19:33.249356+0000","last_peered":"2024-09-19T07:19:33.249356+0000","last_clean":"2024-09-19T07:19:33.249356+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:19:33.249356+0000","last_undegraded":"2024-09-19T07:19:33.249356+0000","last_fullsized":"2024-09-19T07:19:33.249356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663459+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:19:33.663459+0000","last_peered":"2024-09-19T07:19:33.663459+0000","last_clean":"2024-09-19T07:19:33.663459+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:19:33.663459+0000","last_undegraded":"2024-09-19T07:19:33.663459+0000","last_fullsized":"2024-09-19T07:19:33.663459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249422+0000","last_change":"2024-09-19T07:19:32.273635+0000","last_active":"2024-09-19T07:19:33.249422+0000","last_peered":"2024-09-19T07:19:33.249422+0000","last_clean":"2024-09-19T07:19:33.249422+0000","last_became_active":"2024-09-19T07:19:32.271748+0000","last_became_peered":"2024-09-19T07:19:32.271748+0000","last_unstale":"2024-09-19T07:19:33.249422+0000","last_undegraded":"2024-09-19T07:19:33.249422+0000","last_fullsized":"2024-09-19T07:19:33.249422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:49:41.724626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663999+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:19:33.663999+0000","last_peered":"2024-09-19T07:19:33.663999+0000","last_clean":"2024-09-19T07:19:33.663999+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:19:33.663999+0000","last_undegraded":"2024-09-19T07:19:33.663999+0000","last_fullsized":"2024-09-19T07:19:33.663999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.295317+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:19:33.295317+0000","last_peered":"2024-09-19T07:19:33.295317+0000","last_clean":"2024-09-19T07:19:33.295317+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:19:33.295317+0000","last_undegraded":"2024-09-19T07:19:33.295317+0000","last_fullsized":"2024-09-19T07:19:33.295317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246831+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:19:33.246831+0000","last_peered":"2024-09-19T07:19:33.246831+0000","last_clean":"2024-09-19T07:19:33.246831+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:19:33.246831+0000","last_undegraded":"2024-09-19T07:19:33.246831+0000","last_fullsized":"2024-09-19T07:19:33.246831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.665291+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:19:33.665291+0000","last_peered":"2024-09-19T07:19:33.665291+0000","last_clean":"2024-09-19T07:19:33.665291+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:19:33.665291+0000","last_undegraded":"2024-09-19T07:19:33.665291+0000","last_fullsized":"2024-09-19T07:19:33.665291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.667231+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:19:33.667231+0000","last_peered":"2024-09-19T07:19:33.667231+0000","last_clean":"2024-09-19T07:19:33.667231+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:19:33.667231+0000","last_undegraded":"2024-09-19T07:19:33.667231+0000","last_fullsized":"2024-09-19T07:19:33.667231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249851+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:19:33.249851+0000","last_peered":"2024-09-19T07:19:33.249851+0000","last_clean":"2024-09-19T07:19:33.249851+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:19:33.249851+0000","last_undegraded":"2024-09-19T07:19:33.249851+0000","last_fullsized":"2024-09-19T07:19:33.249851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.661983+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:19:33.661983+0000","last_peered":"2024-09-19T07:19:33.661983+0000","last_clean":"2024-09-19T07:19:33.661983+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:19:33.661983+0000","last_undegraded":"2024-09-19T07:19:33.661983+0000","last_fullsized":"2024-09-19T07:19:33.661983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294542+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:19:33.294542+0000","last_peered":"2024-09-19T07:19:33.294542+0000","last_clean":"2024-09-19T07:19:33.294542+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:19:33.294542+0000","last_undegraded":"2024-09-19T07:19:33.294542+0000","last_fullsized":"2024-09-19T07:19:33.294542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.248752+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:19:33.248752+0000","last_peered":"2024-09-19T07:19:33.248752+0000","last_clean":"2024-09-19T07:19:33.248752+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:19:33.248752+0000","last_undegraded":"2024-09-19T07:19:33.248752+0000","last_fullsized":"2024-09-19T07:19:33.248752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664198+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:19:33.664198+0000","last_peered":"2024-09-19T07:19:33.664198+0000","last_clean":"2024-09-19T07:19:33.664198+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:19:33.664198+0000","last_undegraded":"2024-09-19T07:19:33.664198+0000","last_fullsized":"2024-09-19T07:19:33.664198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.667468+0000","last_change":"2024-09-19T07:19:32.289374+0000","last_active":"2024-09-19T07:19:33.667468+0000","last_peered":"2024-09-19T07:19:33.667468+0000","last_clean":"2024-09-19T07:19:33.667468+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:19:33.667468+0000","last_undegraded":"2024-09-19T07:19:33.667468+0000","last_fullsized":"2024-09-19T07:19:33.667468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294247+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:19:33.294247+0000","last_peered":"2024-09-19T07:19:33.294247+0000","last_clean":"2024-09-19T07:19:33.294247+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:19:33.294247+0000","last_undegraded":"2024-09-19T07:19:33.294247+0000","last_fullsized":"2024-09-19T07:19:33.294247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663597+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:19:33.663597+0000","last_peered":"2024-09-19T07:19:33.663597+0000","last_clean":"2024-09-19T07:19:33.663597+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:19:33.663597+0000","last_undegraded":"2024-09-19T07:19:33.663597+0000","last_fullsized":"2024-09-19T07:19:33.663597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663460+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:19:33.663460+0000","last_peered":"2024-09-19T07:19:33.663460+0000","last_clean":"2024-09-19T07:19:33.663460+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:19:33.663460+0000","last_undegraded":"2024-09-19T07:19:33.663460+0000","last_fullsized":"2024-09-19T07:19:33.663460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249549+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:19:33.249549+0000","last_peered":"2024-09-19T07:19:33.249549+0000","last_clean":"2024-09-19T07:19:33.249549+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:19:33.249549+0000","last_undegraded":"2024-09-19T07:19:33.249549+0000","last_fullsized":"2024-09-19T07:19:33.249549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246642+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:19:33.246642+0000","last_peered":"2024-09-19T07:19:33.246642+0000","last_clean":"2024-09-19T07:19:33.246642+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:19:33.246642+0000","last_undegraded":"2024-09-19T07:19:33.246642+0000","last_fullsized":"2024-09-19T07:19:33.246642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294674+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:19:33.294674+0000","last_peered":"2024-09-19T07:19:33.294674+0000","last_clean":"2024-09-19T07:19:33.294674+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:19:33.294674+0000","last_undegraded":"2024-09-19T07:19:33.294674+0000","last_fullsized":"2024-09-19T07:19:33.294674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.250618+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:19:33.250618+0000","last_peered":"2024-09-19T07:19:33.250618+0000","last_clean":"2024-09-19T07:19:33.250618+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:19:33.250618+0000","last_undegraded":"2024-09-19T07:19:33.250618+0000","last_fullsized":"2024-09-19T07:19:33.250618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.250735+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:19:33.250735+0000","last_peered":"2024-09-19T07:19:33.250735+0000","last_clean":"2024-09-19T07:19:33.250735+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:19:33.250735+0000","last_undegraded":"2024-09-19T07:19:33.250735+0000","last_fullsized":"2024-09-19T07:19:33.250735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664631+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:19:33.664631+0000","last_peered":"2024-09-19T07:19:33.664631+0000","last_clean":"2024-09-19T07:19:33.664631+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:19:33.664631+0000","last_undegraded":"2024-09-19T07:19:33.664631+0000","last_fullsized":"2024-09-19T07:19:33.664631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:48.308137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246511+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:19:33.246511+0000","last_peered":"2024-09-19T07:19:33.246511+0000","last_clean":"2024-09-19T07:19:33.246511+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:19:33.246511+0000","last_undegraded":"2024-09-19T07:19:33.246511+0000","last_fullsized":"2024-09-19T07:19:33.246511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.251942+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:19:33.251942+0000","last_peered":"2024-09-19T07:19:33.251942+0000","last_clean":"2024-09-19T07:19:33.251942+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:19:33.251942+0000","last_undegraded":"2024-09-19T07:19:33.251942+0000","last_fullsized":"2024-09-19T07:19:33.251942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294839+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:19:33.294839+0000","last_peered":"2024-09-19T07:19:33.294839+0000","last_clean":"2024-09-19T07:19:33.294839+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:19:33.294839+0000","last_undegraded":"2024-09-19T07:19:33.294839+0000","last_fullsized":"2024-09-19T07:19:33.294839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612637+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:19:33.612637+0000","last_peered":"2024-09-19T07:19:33.612637+0000","last_clean":"2024-09-19T07:19:33.612637+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:19:33.612637+0000","last_undegraded":"2024-09-19T07:19:33.612637+0000","last_fullsized":"2024-09-19T07:19:33.612637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.667357+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:19:33.667357+0000","last_peered":"2024-09-19T07:19:33.667357+0000","last_clean":"2024-09-19T07:19:33.667357+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:19:33.667357+0000","last_undegraded":"2024-09-19T07:19:33.667357+0000","last_fullsized":"2024-09-19T07:19:33.667357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:23.367365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.665141+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:19:33.665141+0000","last_peered":"2024-09-19T07:19:33.665141+0000","last_clean":"2024-09-19T07:19:33.665141+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:19:33.665141+0000","last_undegraded":"2024-09-19T07:19:33.665141+0000","last_fullsized":"2024-09-19T07:19:33.665141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:57.131912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249571+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:19:33.249571+0000","last_peered":"2024-09-19T07:19:33.249571+0000","last_clean":"2024-09-19T07:19:33.249571+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:19:33.249571+0000","last_undegraded":"2024-09-19T07:19:33.249571+0000","last_fullsized":"2024-09-19T07:19:33.249571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613088+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:19:33.613088+0000","last_peered":"2024-09-19T07:19:33.613088+0000","last_clean":"2024-09-19T07:19:33.613088+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:19:33.613088+0000","last_undegraded":"2024-09-19T07:19:33.613088+0000","last_fullsized":"2024-09-19T07:19:33.613088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294329+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:19:33.294329+0000","last_peered":"2024-09-19T07:19:33.294329+0000","last_clean":"2024-09-19T07:19:33.294329+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:19:33.294329+0000","last_undegraded":"2024-09-19T07:19:33.294329+0000","last_fullsized":"2024-09-19T07:19:33.294329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294663+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:19:33.294663+0000","last_peered":"2024-09-19T07:19:33.294663+0000","last_clean":"2024-09-19T07:19:33.294663+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:19:33.294663+0000","last_undegraded":"2024-09-19T07:19:33.294663+0000","last_fullsized":"2024-09-19T07:19:33.294663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.661513+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:19:33.661513+0000","last_peered":"2024-09-19T07:19:33.661513+0000","last_clean":"2024-09-19T07:19:33.661513+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:19:33.661513+0000","last_undegraded":"2024-09-19T07:19:33.661513+0000","last_fullsized":"2024-09-19T07:19:33.661513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612502+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:19:33.612502+0000","last_peered":"2024-09-19T07:19:33.612502+0000","last_clean":"2024-09-19T07:19:33.612502+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:19:33.612502+0000","last_undegraded":"2024-09-19T07:19:33.612502+0000","last_fullsized":"2024-09-19T07:19:33.612502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.251931+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:19:33.251931+0000","last_peered":"2024-09-19T07:19:33.251931+0000","last_clean":"2024-09-19T07:19:33.251931+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:19:33.251931+0000","last_undegraded":"2024-09-19T07:19:33.251931+0000","last_fullsized":"2024-09-19T07:19:33.251931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613244+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:19:33.613244+0000","last_peered":"2024-09-19T07:19:33.613244+0000","last_clean":"2024-09-19T07:19:33.613244+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:19:33.613244+0000","last_undegraded":"2024-09-19T07:19:33.613244+0000","last_fullsized":"2024-09-19T07:19:33.613244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.662216+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:19:33.662216+0000","last_peered":"2024-09-19T07:19:33.662216+0000","last_clean":"2024-09-19T07:19:33.662216+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:19:33.662216+0000","last_undegraded":"2024-09-19T07:19:33.662216+0000","last_fullsized":"2024-09-19T07:19:33.662216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613560+0000","last_change":"2024-09-19T07:19:32.292026+0000","last_active":"2024-09-19T07:19:33.613560+0000","last_peered":"2024-09-19T07:19:33.613560+0000","last_clean":"2024-09-19T07:19:33.613560+0000","last_became_active":"2024-09-19T07:19:32.291037+0000","last_became_peered":"2024-09-19T07:19:32.291037+0000","last_unstale":"2024-09-19T07:19:33.613560+0000","last_undegraded":"2024-09-19T07:19:33.613560+0000","last_fullsized":"2024-09-19T07:19:33.613560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:22.190763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246634+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:19:33.246634+0000","last_peered":"2024-09-19T07:19:33.246634+0000","last_clean":"2024-09-19T07:19:33.246634+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:19:33.246634+0000","last_undegraded":"2024-09-19T07:19:33.246634+0000","last_fullsized":"2024-09-19T07:19:33.246634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.251515+0000","last_change":"2024-09-19T07:19:32.276600+0000","last_active":"2024-09-19T07:19:33.251515+0000","last_peered":"2024-09-19T07:19:33.251515+0000","last_clean":"2024-09-19T07:19:33.251515+0000","last_became_active":"2024-09-19T07:19:32.275766+0000","last_became_peered":"2024-09-19T07:19:32.275766+0000","last_unstale":"2024-09-19T07:19:33.251515+0000","last_undegraded":"2024-09-19T07:19:33.251515+0000","last_fullsized":"2024-09-19T07:19:33.251515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:48.506399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.661835+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:19:33.661835+0000","last_peered":"2024-09-19T07:19:33.661835+0000","last_clean":"2024-09-19T07:19:33.661835+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:19:33.661835+0000","last_undegraded":"2024-09-19T07:19:33.661835+0000","last_fullsized":"2024-09-19T07:19:33.661835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613366+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:19:33.613366+0000","last_peered":"2024-09-19T07:19:33.613366+0000","last_clean":"2024-09-19T07:19:33.613366+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:19:33.613366+0000","last_undegraded":"2024-09-19T07:19:33.613366+0000","last_fullsized":"2024-09-19T07:19:33.613366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612290+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:19:33.612290+0000","last_peered":"2024-09-19T07:19:33.612290+0000","last_clean":"2024-09-19T07:19:33.612290+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:19:33.612290+0000","last_undegraded":"2024-09-19T07:19:33.612290+0000","last_fullsized":"2024-09-19T07:19:33.612290+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249670+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:19:33.249670+0000","last_peered":"2024-09-19T07:19:33.249670+0000","last_clean":"2024-09-19T07:19:33.249670+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:19:33.249670+0000","last_undegraded":"2024-09-19T07:19:33.249670+0000","last_fullsized":"2024-09-19T07:19:33.249670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.665399+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:19:33.665399+0000","last_peered":"2024-09-19T07:19:33.665399+0000","last_clean":"2024-09-19T07:19:33.665399+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:19:33.665399+0000","last_undegraded":"2024-09-19T07:19:33.665399+0000","last_fullsized":"2024-09-19T07:19:33.665399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.611996+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:19:33.611996+0000","last_peered":"2024-09-19T07:19:33.611996+0000","last_clean":"2024-09-19T07:19:33.611996+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:19:33.611996+0000","last_undegraded":"2024-09-19T07:19:33.611996+0000","last_fullsized":"2024-09-19T07:19:33.611996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.665475+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:19:33.665475+0000","last_peered":"2024-09-19T07:19:33.665475+0000","last_clean":"2024-09-19T07:19:33.665475+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:19:33.665475+0000","last_undegraded":"2024-09-19T07:19:33.665475+0000","last_fullsized":"2024-09-19T07:19:33.665475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:20.824916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.661830+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:19:33.661830+0000","last_peered":"2024-09-19T07:19:33.661830+0000","last_clean":"2024-09-19T07:19:33.661830+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:19:33.661830+0000","last_undegraded":"2024-09-19T07:19:33.661830+0000","last_fullsized":"2024-09-19T07:19:33.661830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.247106+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:19:33.247106+0000","last_peered":"2024-09-19T07:19:33.247106+0000","last_clean":"2024-09-19T07:19:33.247106+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:19:33.247106+0000","last_undegraded":"2024-09-19T07:19:33.247106+0000","last_fullsized":"2024-09-19T07:19:33.247106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249868+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:19:33.249868+0000","last_peered":"2024-09-19T07:19:33.249868+0000","last_clean":"2024-09-19T07:19:33.249868+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:19:33.249868+0000","last_undegraded":"2024-09-19T07:19:33.249868+0000","last_fullsized":"2024-09-19T07:19:33.249868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.250543+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:19:33.250543+0000","last_peered":"2024-09-19T07:19:33.250543+0000","last_clean":"2024-09-19T07:19:33.250543+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:19:33.250543+0000","last_undegraded":"2024-09-19T07:19:33.250543+0000","last_fullsized":"2024-09-19T07:19:33.250543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246829+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:19:33.246829+0000","last_peered":"2024-09-19T07:19:33.246829+0000","last_clean":"2024-09-19T07:19:33.246829+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:19:33.246829+0000","last_undegraded":"2024-09-19T07:19:33.246829+0000","last_fullsized":"2024-09-19T07:19:33.246829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.251329+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:19:33.251329+0000","last_peered":"2024-09-19T07:19:33.251329+0000","last_clean":"2024-09-19T07:19:33.251329+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:19:33.251329+0000","last_undegraded":"2024-09-19T07:19:33.251329+0000","last_fullsized":"2024-09-19T07:19:33.251329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613452+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:19:33.613452+0000","last_peered":"2024-09-19T07:19:33.613452+0000","last_clean":"2024-09-19T07:19:33.613452+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:19:33.613452+0000","last_undegraded":"2024-09-19T07:19:33.613452+0000","last_fullsized":"2024-09-19T07:19:33.613452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246699+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:19:33.246699+0000","last_peered":"2024-09-19T07:19:33.246699+0000","last_clean":"2024-09-19T07:19:33.246699+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:19:33.246699+0000","last_undegraded":"2024-09-19T07:19:33.246699+0000","last_fullsized":"2024-09-19T07:19:33.246699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.247108+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:19:33.247108+0000","last_peered":"2024-09-19T07:19:33.247108+0000","last_clean":"2024-09-19T07:19:33.247108+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:19:33.247108+0000","last_undegraded":"2024-09-19T07:19:33.247108+0000","last_fullsized":"2024-09-19T07:19:33.247108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:50.700962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664417+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:19:33.664417+0000","last_peered":"2024-09-19T07:19:33.664417+0000","last_clean":"2024-09-19T07:19:33.664417+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:19:33.664417+0000","last_undegraded":"2024-09-19T07:19:33.664417+0000","last_fullsized":"2024-09-19T07:19:33.664417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294946+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:19:33.294946+0000","last_peered":"2024-09-19T07:19:33.294946+0000","last_clean":"2024-09-19T07:19:33.294946+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:19:33.294946+0000","last_undegraded":"2024-09-19T07:19:33.294946+0000","last_fullsized":"2024-09-19T07:19:33.294946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.665154+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:19:33.665154+0000","last_peered":"2024-09-19T07:19:33.665154+0000","last_clean":"2024-09-19T07:19:33.665154+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:19:33.665154+0000","last_undegraded":"2024-09-19T07:19:33.665154+0000","last_fullsized":"2024-09-19T07:19:33.665154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:23.791880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.611979+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:19:33.611979+0000","last_peered":"2024-09-19T07:19:33.611979+0000","last_clean":"2024-09-19T07:19:33.611979+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:19:33.611979+0000","last_undegraded":"2024-09-19T07:19:33.611979+0000","last_fullsized":"2024-09-19T07:19:33.611979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.248962+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:19:33.248962+0000","last_peered":"2024-09-19T07:19:33.248962+0000","last_clean":"2024-09-19T07:19:33.248962+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:19:33.248962+0000","last_undegraded":"2024-09-19T07:19:33.248962+0000","last_fullsized":"2024-09-19T07:19:33.248962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246815+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:19:33.246815+0000","last_peered":"2024-09-19T07:19:33.246815+0000","last_clean":"2024-09-19T07:19:33.246815+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:19:33.246815+0000","last_undegraded":"2024-09-19T07:19:33.246815+0000","last_fullsized":"2024-09-19T07:19:33.246815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663805+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:19:33.663805+0000","last_peered":"2024-09-19T07:19:33.663805+0000","last_clean":"2024-09-19T07:19:33.663805+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:19:33.663805+0000","last_undegraded":"2024-09-19T07:19:33.663805+0000","last_fullsized":"2024-09-19T07:19:33.663805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.247503+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:19:33.247503+0000","last_peered":"2024-09-19T07:19:33.247503+0000","last_clean":"2024-09-19T07:19:33.247503+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:19:33.247503+0000","last_undegraded":"2024-09-19T07:19:33.247503+0000","last_fullsized":"2024-09-19T07:19:33.247503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664303+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:19:33.664303+0000","last_peered":"2024-09-19T07:19:33.664303+0000","last_clean":"2024-09-19T07:19:33.664303+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:19:33.664303+0000","last_undegraded":"2024-09-19T07:19:33.664303+0000","last_fullsized":"2024-09-19T07:19:33.664303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612858+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:19:33.612858+0000","last_peered":"2024-09-19T07:19:33.612858+0000","last_clean":"2024-09-19T07:19:33.612858+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:19:33.612858+0000","last_undegraded":"2024-09-19T07:19:33.612858+0000","last_fullsized":"2024-09-19T07:19:33.612858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.295119+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:19:33.295119+0000","last_peered":"2024-09-19T07:19:33.295119+0000","last_clean":"2024-09-19T07:19:33.295119+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:19:33.295119+0000","last_undegraded":"2024-09-19T07:19:33.295119+0000","last_fullsized":"2024-09-19T07:19:33.295119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.250914+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:19:33.250914+0000","last_peered":"2024-09-19T07:19:33.250914+0000","last_clean":"2024-09-19T07:19:33.250914+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:19:33.250914+0000","last_undegraded":"2024-09-19T07:19:33.250914+0000","last_fullsized":"2024-09-19T07:19:33.250914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612853+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:19:33.612853+0000","last_peered":"2024-09-19T07:19:33.612853+0000","last_clean":"2024-09-19T07:19:33.612853+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:19:33.612853+0000","last_undegraded":"2024-09-19T07:19:33.612853+0000","last_fullsized":"2024-09-19T07:19:33.612853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.250556+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:19:33.250556+0000","last_peered":"2024-09-19T07:19:33.250556+0000","last_clean":"2024-09-19T07:19:33.250556+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:19:33.250556+0000","last_undegraded":"2024-09-19T07:19:33.250556+0000","last_fullsized":"2024-09-19T07:19:33.250556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613176+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:19:33.613176+0000","last_peered":"2024-09-19T07:19:33.613176+0000","last_clean":"2024-09-19T07:19:33.613176+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:19:33.613176+0000","last_undegraded":"2024-09-19T07:19:33.613176+0000","last_fullsized":"2024-09-19T07:19:33.613176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664304+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:19:33.664304+0000","last_peered":"2024-09-19T07:19:33.664304+0000","last_clean":"2024-09-19T07:19:33.664304+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:19:33.664304+0000","last_undegraded":"2024-09-19T07:19:33.664304+0000","last_fullsized":"2024-09-19T07:19:33.664304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:15.036396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664287+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:19:33.664287+0000","last_peered":"2024-09-19T07:19:33.664287+0000","last_clean":"2024-09-19T07:19:33.664287+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:19:33.664287+0000","last_undegraded":"2024-09-19T07:19:33.664287+0000","last_fullsized":"2024-09-19T07:19:33.664287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663935+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:19:33.663935+0000","last_peered":"2024-09-19T07:19:33.663935+0000","last_clean":"2024-09-19T07:19:33.663935+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:19:33.663935+0000","last_undegraded":"2024-09-19T07:19:33.663935+0000","last_fullsized":"2024-09-19T07:19:33.663935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664953+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:19:33.664953+0000","last_peered":"2024-09-19T07:19:33.664953+0000","last_clean":"2024-09-19T07:19:33.664953+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:19:33.664953+0000","last_undegraded":"2024-09-19T07:19:33.664953+0000","last_fullsized":"2024-09-19T07:19:33.664953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:57.799442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":811008,"data_stored":665033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":74173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":74209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":74083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":74243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":819200,"data_stored":665089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:19:37.309 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:19:37.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.451+0000 7f60e9763640 1 Processor -- start 2024-09-19T07:19:37.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.451+0000 7f60e9763640 1 -- start start 2024-09-19T07:19:37.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e9763640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f60e4071100 0x7f60e41545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e9763640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e406dea0 0x7f60e406e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e9763640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60e40a9e60 con 0x7f60e40c28e0 2024-09-19T07:19:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e9763640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f60e4071b50 con 0x7f60e4071100 2024-09-19T07:19:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e9763640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f60e40c1ff0 con 0x7f60e406dea0 2024-09-19T07:19:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e2ffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f60e4071100 0x7f60e41545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e2ffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f60e4071100 0x7f60e41545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:56440/0 (socket says 172.21.15.83:56440) 2024-09-19T07:19:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e37fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e406dea0 0x7f60e406e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e27fc640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f60e40c28e0 0x7f60e4150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:57218/0 (socket says 172.21.15.83:57218) 2024-09-19T07:19:37.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e2ffd640 1 -- 172.21.15.83:0/818571560 learned_addr learned my addr 172.21.15.83:0/818571560 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:37.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e2ffd640 1 -- 172.21.15.83:0/818571560 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e406dea0 msgr2=0x7f60e406e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:37.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e2ffd640 1 --2- 172.21.15.83:0/818571560 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e406dea0 0x7f60e406e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e2ffd640 1 -- 172.21.15.83:0/818571560 >> v1:172.21.15.184:6789/0 conn(0x7f60e40c28e0 legacy=0x7f60e4150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e2ffd640 1 -- 172.21.15.83:0/818571560 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e40aa140 con 0x7f60e4071100 2024-09-19T07:19:37.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.455+0000 7f60e2ffd640 1 --2- 172.21.15.83:0/818571560 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e4071100 0x7f60e41545e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009870 tx=0x7f60d801c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2bb1875679cfd95 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:37.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e1ffb640 1 -- 172.21.15.83:0/818571560 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d80b1070 con 0x7f60e4071100 2024-09-19T07:19:37.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e1ffb640 1 -- 172.21.15.83:0/818571560 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d80ac620 con 0x7f60e4071100 2024-09-19T07:19:37.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e1ffb640 1 -- 172.21.15.83:0/818571560 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d80ac9f0 con 0x7f60e4071100 2024-09-19T07:19:37.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 -- 172.21.15.83:0/818571560 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e4071100 msgr2=0x7f60e41545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 --2- 172.21.15.83:0/818571560 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e4071100 0x7f60e41545e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009870 tx=0x7f60d801c4d0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 -- 172.21.15.83:0/818571560 shutdown_connections 2024-09-19T07:19:37.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 --2- 172.21.15.83:0/818571560 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e406dea0 0x7f60e406e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 --2- 172.21.15.83:0/818571560 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e4071100 0x7f60e41545e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 -- 172.21.15.83:0/818571560 >> 172.21.15.83:0/818571560 conn(0x7f60e406bc80 msgr2=0x7f60e406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:37.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 -- 172.21.15.83:0/818571560 shutdown_connections 2024-09-19T07:19:37.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 -- 172.21.15.83:0/818571560 wait complete. 2024-09-19T07:19:37.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 Processor -- start 2024-09-19T07:19:37.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 -- start start 2024-09-19T07:19:37.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e406dea0 0x7f60e4147420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e4071100 0x7f60e4145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e40c28e0 0x7f60e4142300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f60e4071660 con 0x7f60e406dea0 2024-09-19T07:19:37.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f60e40714e0 con 0x7f60e40c28e0 2024-09-19T07:19:37.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f60e4160c60 con 0x7f60e4071100 2024-09-19T07:19:37.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e406dea0 0x7f60e4147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e406dea0 0x7f60e4147420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55562/0 (socket says 172.21.15.83:55562) 2024-09-19T07:19:37.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e2ffd640 1 -- 172.21.15.83:0/1765833561 learned_addr learned my addr 172.21.15.83:0/1765833561 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:37.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e37fe640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e40c28e0 0x7f60e4142300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e27fc640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e4071100 0x7f60e4145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e2ffd640 1 -- 172.21.15.83:0/1765833561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e4071100 msgr2=0x7f60e4145660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e2ffd640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e4071100 0x7f60e4145660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e2ffd640 1 -- 172.21.15.83:0/1765833561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e40c28e0 msgr2=0x7f60e4142300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e2ffd640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e40c28e0 0x7f60e4142300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e2ffd640 1 -- 172.21.15.83:0/1765833561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e40c7090 con 0x7f60e406dea0 2024-09-19T07:19:37.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e27fc640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e4071100 0x7f60e4145660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:37.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e2ffd640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e406dea0 0x7f60e4147420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f60d80af040 tx=0x7f60d801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:37.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60cb7fe640 1 -- 172.21.15.83:0/1765833561 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d80b1040 con 0x7f60e406dea0 2024-09-19T07:19:37.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60cb7fe640 1 -- 172.21.15.83:0/1765833561 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d80ac420 con 0x7f60e406dea0 2024-09-19T07:19:37.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 -- 172.21.15.83:0/1765833561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e41ec330 con 0x7f60e406dea0 2024-09-19T07:19:37.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60cb7fe640 1 -- 172.21.15.83:0/1765833561 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d80b7660 con 0x7f60e406dea0 2024-09-19T07:19:37.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.459+0000 7f60e9763640 1 -- 172.21.15.83:0/1765833561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e41ec8f0 con 0x7f60e406dea0 2024-09-19T07:19:37.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.463+0000 7f60e9763640 1 -- 172.21.15.83:0/1765833561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e406e330 con 0x7f60e406dea0 2024-09-19T07:19:37.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.463+0000 7f60cb7fe640 1 -- 172.21.15.83:0/1765833561 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f60d80bf030 con 0x7f60e406dea0 2024-09-19T07:19:37.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.463+0000 7f60cb7fe640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60d00678c0 0x7f60d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.463+0000 7f60e27fc640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60d00678c0 0x7f60d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.463+0000 7f60cb7fe640 1 -- 172.21.15.83:0/1765833561 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7f60d811f710 con 0x7f60e406dea0 2024-09-19T07:19:37.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.463+0000 7f60e27fc640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60d00678c0 0x7f60d0069d80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f60d4007fd0 tx=0x7f60d4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:37.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.467+0000 7f60cb7fe640 1 -- 172.21.15.83:0/1765833561 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f60d80c6060 con 0x7f60e406dea0 2024-09-19T07:19:37.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.615+0000 7f60e9763640 1 -- 172.21.15.83:0/1765833561 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f60e40c4f30 con 0x7f60d00678c0 2024-09-19T07:19:37.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.627+0000 7f60cb7fe640 1 -- 172.21.15.83:0/1765833561 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251952 (secure 0 0 0) 0x7f60e40c4f30 con 0x7f60d00678c0 2024-09-19T07:19:37.631 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:37.632 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:19:37.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.631+0000 7f60e9763640 1 -- 172.21.15.83:0/1765833561 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60d00678c0 msgr2=0x7f60d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.631+0000 7f60e9763640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60d00678c0 0x7f60d0069d80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f60d4007fd0 tx=0x7f60d4009040 comp rx=0 tx=0).stop 2024-09-19T07:19:37.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.631+0000 7f60e9763640 1 -- 172.21.15.83:0/1765833561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e406dea0 msgr2=0x7f60e4147420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.631+0000 7f60e9763640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e406dea0 0x7f60e4147420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f60d80af040 tx=0x7f60d801ad10 comp rx=0 tx=0).stop 2024-09-19T07:19:37.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.631+0000 7f60e9763640 1 -- 172.21.15.83:0/1765833561 shutdown_connections 2024-09-19T07:19:37.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.631+0000 7f60e9763640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60d00678c0 0x7f60d0069d80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.631+0000 7f60e9763640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e40c28e0 0x7f60e4142300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.631+0000 7f60e9763640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e4071100 0x7f60e4145660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.631+0000 7f60e9763640 1 --2- 172.21.15.83:0/1765833561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e406dea0 0x7f60e4147420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.631+0000 7f60e9763640 1 -- 172.21.15.83:0/1765833561 >> 172.21.15.83:0/1765833561 conn(0x7f60e406bc80 msgr2=0x7f60e4150f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:37.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.631+0000 7f60e9763640 1 -- 172.21.15.83:0/1765833561 shutdown_connections 2024-09-19T07:19:37.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.631+0000 7f60e9763640 1 -- 172.21.15.83:0/1765833561 wait complete. 2024-09-19T07:19:37.648 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-19T07:19:36.812237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219172,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755548,"statfs":{"total":773094113280,"available":772869681152,"internally_reserved":0,"allocated":2957312,"data_stored":1778220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001596"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613309+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:19:33.613309+0000","last_peered":"2024-09-19T07:19:33.613309+0000","last_clean":"2024-09-19T07:19:33.613309+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:19:33.613309+0000","last_undegraded":"2024-09-19T07:19:33.613309+0000","last_fullsized":"2024-09-19T07:19:33.613309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246945+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:19:33.246945+0000","last_peered":"2024-09-19T07:19:33.246945+0000","last_clean":"2024-09-19T07:19:33.246945+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:19:33.246945+0000","last_undegraded":"2024-09-19T07:19:33.246945+0000","last_fullsized":"2024-09-19T07:19:33.246945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664687+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:19:33.664687+0000","last_peered":"2024-09-19T07:19:33.664687+0000","last_clean":"2024-09-19T07:19:33.664687+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:19:33.664687+0000","last_undegraded":"2024-09-19T07:19:33.664687+0000","last_fullsized":"2024-09-19T07:19:33.664687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:25.667168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.250169+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:19:33.250169+0000","last_peered":"2024-09-19T07:19:33.250169+0000","last_clean":"2024-09-19T07:19:33.250169+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:19:33.250169+0000","last_undegraded":"2024-09-19T07:19:33.250169+0000","last_fullsized":"2024-09-19T07:19:33.250169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612616+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:19:33.612616+0000","last_peered":"2024-09-19T07:19:33.612616+0000","last_clean":"2024-09-19T07:19:33.612616+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:19:33.612616+0000","last_undegraded":"2024-09-19T07:19:33.612616+0000","last_fullsized":"2024-09-19T07:19:33.612616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249303+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:19:33.249303+0000","last_peered":"2024-09-19T07:19:33.249303+0000","last_clean":"2024-09-19T07:19:33.249303+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:19:33.249303+0000","last_undegraded":"2024-09-19T07:19:33.249303+0000","last_fullsized":"2024-09-19T07:19:33.249303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612363+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:19:33.612363+0000","last_peered":"2024-09-19T07:19:33.612363+0000","last_clean":"2024-09-19T07:19:33.612363+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:19:33.612363+0000","last_undegraded":"2024-09-19T07:19:33.612363+0000","last_fullsized":"2024-09-19T07:19:33.612363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663377+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:19:33.663377+0000","last_peered":"2024-09-19T07:19:33.663377+0000","last_clean":"2024-09-19T07:19:33.663377+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:19:33.663377+0000","last_undegraded":"2024-09-19T07:19:33.663377+0000","last_fullsized":"2024-09-19T07:19:33.663377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613374+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:19:33.613374+0000","last_peered":"2024-09-19T07:19:33.613374+0000","last_clean":"2024-09-19T07:19:33.613374+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:19:33.613374+0000","last_undegraded":"2024-09-19T07:19:33.613374+0000","last_fullsized":"2024-09-19T07:19:33.613374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.295259+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:19:33.295259+0000","last_peered":"2024-09-19T07:19:33.295259+0000","last_clean":"2024-09-19T07:19:33.295259+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:19:33.295259+0000","last_undegraded":"2024-09-19T07:19:33.295259+0000","last_fullsized":"2024-09-19T07:19:33.295259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294981+0000","last_change":"2024-09-19T07:19:32.299956+0000","last_active":"2024-09-19T07:19:33.294981+0000","last_peered":"2024-09-19T07:19:33.294981+0000","last_clean":"2024-09-19T07:19:33.294981+0000","last_became_active":"2024-09-19T07:19:32.299168+0000","last_became_peered":"2024-09-19T07:19:32.299168+0000","last_unstale":"2024-09-19T07:19:33.294981+0000","last_undegraded":"2024-09-19T07:19:33.294981+0000","last_fullsized":"2024-09-19T07:19:33.294981+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.251583+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:19:33.251583+0000","last_peered":"2024-09-19T07:19:33.251583+0000","last_clean":"2024-09-19T07:19:33.251583+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:19:33.251583+0000","last_undegraded":"2024-09-19T07:19:33.251583+0000","last_fullsized":"2024-09-19T07:19:33.251583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664051+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:19:33.664051+0000","last_peered":"2024-09-19T07:19:33.664051+0000","last_clean":"2024-09-19T07:19:33.664051+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:19:33.664051+0000","last_undegraded":"2024-09-19T07:19:33.664051+0000","last_fullsized":"2024-09-19T07:19:33.664051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.665003+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:19:33.665003+0000","last_peered":"2024-09-19T07:19:33.665003+0000","last_clean":"2024-09-19T07:19:33.665003+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:19:33.665003+0000","last_undegraded":"2024-09-19T07:19:33.665003+0000","last_fullsized":"2024-09-19T07:19:33.665003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:43.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246656+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:19:33.246656+0000","last_peered":"2024-09-19T07:19:33.246656+0000","last_clean":"2024-09-19T07:19:33.246656+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:19:33.246656+0000","last_undegraded":"2024-09-19T07:19:33.246656+0000","last_fullsized":"2024-09-19T07:19:33.246656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249042+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:19:33.249042+0000","last_peered":"2024-09-19T07:19:33.249042+0000","last_clean":"2024-09-19T07:19:33.249042+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:19:33.249042+0000","last_undegraded":"2024-09-19T07:19:33.249042+0000","last_fullsized":"2024-09-19T07:19:33.249042+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.251207+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:19:33.251207+0000","last_peered":"2024-09-19T07:19:33.251207+0000","last_clean":"2024-09-19T07:19:33.251207+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:19:33.251207+0000","last_undegraded":"2024-09-19T07:19:33.251207+0000","last_fullsized":"2024-09-19T07:19:33.251207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.662152+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:19:33.662152+0000","last_peered":"2024-09-19T07:19:33.662152+0000","last_clean":"2024-09-19T07:19:33.662152+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:19:33.662152+0000","last_undegraded":"2024-09-19T07:19:33.662152+0000","last_fullsized":"2024-09-19T07:19:33.662152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:51.710997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613530+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:19:33.613530+0000","last_peered":"2024-09-19T07:19:33.613530+0000","last_clean":"2024-09-19T07:19:33.613530+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:19:33.613530+0000","last_undegraded":"2024-09-19T07:19:33.613530+0000","last_fullsized":"2024-09-19T07:19:33.613530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294874+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:19:33.294874+0000","last_peered":"2024-09-19T07:19:33.294874+0000","last_clean":"2024-09-19T07:19:33.294874+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:19:33.294874+0000","last_undegraded":"2024-09-19T07:19:33.294874+0000","last_fullsized":"2024-09-19T07:19:33.294874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.667108+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:19:33.667108+0000","last_peered":"2024-09-19T07:19:33.667108+0000","last_clean":"2024-09-19T07:19:33.667108+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:19:33.667108+0000","last_undegraded":"2024-09-19T07:19:33.667108+0000","last_fullsized":"2024-09-19T07:19:33.667108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:19.033192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249356+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:19:33.249356+0000","last_peered":"2024-09-19T07:19:33.249356+0000","last_clean":"2024-09-19T07:19:33.249356+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:19:33.249356+0000","last_undegraded":"2024-09-19T07:19:33.249356+0000","last_fullsized":"2024-09-19T07:19:33.249356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663459+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:19:33.663459+0000","last_peered":"2024-09-19T07:19:33.663459+0000","last_clean":"2024-09-19T07:19:33.663459+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:19:33.663459+0000","last_undegraded":"2024-09-19T07:19:33.663459+0000","last_fullsized":"2024-09-19T07:19:33.663459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249422+0000","last_change":"2024-09-19T07:19:32.273635+0000","last_active":"2024-09-19T07:19:33.249422+0000","last_peered":"2024-09-19T07:19:33.249422+0000","last_clean":"2024-09-19T07:19:33.249422+0000","last_became_active":"2024-09-19T07:19:32.271748+0000","last_became_peered":"2024-09-19T07:19:32.271748+0000","last_unstale":"2024-09-19T07:19:33.249422+0000","last_undegraded":"2024-09-19T07:19:33.249422+0000","last_fullsized":"2024-09-19T07:19:33.249422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:49:41.724626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663999+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:19:33.663999+0000","last_peered":"2024-09-19T07:19:33.663999+0000","last_clean":"2024-09-19T07:19:33.663999+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:19:33.663999+0000","last_undegraded":"2024-09-19T07:19:33.663999+0000","last_fullsized":"2024-09-19T07:19:33.663999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.295317+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:19:33.295317+0000","last_peered":"2024-09-19T07:19:33.295317+0000","last_clean":"2024-09-19T07:19:33.295317+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:19:33.295317+0000","last_undegraded":"2024-09-19T07:19:33.295317+0000","last_fullsized":"2024-09-19T07:19:33.295317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246831+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:19:33.246831+0000","last_peered":"2024-09-19T07:19:33.246831+0000","last_clean":"2024-09-19T07:19:33.246831+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:19:33.246831+0000","last_undegraded":"2024-09-19T07:19:33.246831+0000","last_fullsized":"2024-09-19T07:19:33.246831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.665291+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:19:33.665291+0000","last_peered":"2024-09-19T07:19:33.665291+0000","last_clean":"2024-09-19T07:19:33.665291+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:19:33.665291+0000","last_undegraded":"2024-09-19T07:19:33.665291+0000","last_fullsized":"2024-09-19T07:19:33.665291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.667231+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:19:33.667231+0000","last_peered":"2024-09-19T07:19:33.667231+0000","last_clean":"2024-09-19T07:19:33.667231+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:19:33.667231+0000","last_undegraded":"2024-09-19T07:19:33.667231+0000","last_fullsized":"2024-09-19T07:19:33.667231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249851+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:19:33.249851+0000","last_peered":"2024-09-19T07:19:33.249851+0000","last_clean":"2024-09-19T07:19:33.249851+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:19:33.249851+0000","last_undegraded":"2024-09-19T07:19:33.249851+0000","last_fullsized":"2024-09-19T07:19:33.249851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.661983+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:19:33.661983+0000","last_peered":"2024-09-19T07:19:33.661983+0000","last_clean":"2024-09-19T07:19:33.661983+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:19:33.661983+0000","last_undegraded":"2024-09-19T07:19:33.661983+0000","last_fullsized":"2024-09-19T07:19:33.661983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294542+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:19:33.294542+0000","last_peered":"2024-09-19T07:19:33.294542+0000","last_clean":"2024-09-19T07:19:33.294542+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:19:33.294542+0000","last_undegraded":"2024-09-19T07:19:33.294542+0000","last_fullsized":"2024-09-19T07:19:33.294542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.248752+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:19:33.248752+0000","last_peered":"2024-09-19T07:19:33.248752+0000","last_clean":"2024-09-19T07:19:33.248752+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:19:33.248752+0000","last_undegraded":"2024-09-19T07:19:33.248752+0000","last_fullsized":"2024-09-19T07:19:33.248752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664198+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:19:33.664198+0000","last_peered":"2024-09-19T07:19:33.664198+0000","last_clean":"2024-09-19T07:19:33.664198+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:19:33.664198+0000","last_undegraded":"2024-09-19T07:19:33.664198+0000","last_fullsized":"2024-09-19T07:19:33.664198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.667468+0000","last_change":"2024-09-19T07:19:32.289374+0000","last_active":"2024-09-19T07:19:33.667468+0000","last_peered":"2024-09-19T07:19:33.667468+0000","last_clean":"2024-09-19T07:19:33.667468+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:19:33.667468+0000","last_undegraded":"2024-09-19T07:19:33.667468+0000","last_fullsized":"2024-09-19T07:19:33.667468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294247+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:19:33.294247+0000","last_peered":"2024-09-19T07:19:33.294247+0000","last_clean":"2024-09-19T07:19:33.294247+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:19:33.294247+0000","last_undegraded":"2024-09-19T07:19:33.294247+0000","last_fullsized":"2024-09-19T07:19:33.294247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663597+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:19:33.663597+0000","last_peered":"2024-09-19T07:19:33.663597+0000","last_clean":"2024-09-19T07:19:33.663597+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:19:33.663597+0000","last_undegraded":"2024-09-19T07:19:33.663597+0000","last_fullsized":"2024-09-19T07:19:33.663597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663460+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:19:33.663460+0000","last_peered":"2024-09-19T07:19:33.663460+0000","last_clean":"2024-09-19T07:19:33.663460+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:19:33.663460+0000","last_undegraded":"2024-09-19T07:19:33.663460+0000","last_fullsized":"2024-09-19T07:19:33.663460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249549+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:19:33.249549+0000","last_peered":"2024-09-19T07:19:33.249549+0000","last_clean":"2024-09-19T07:19:33.249549+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:19:33.249549+0000","last_undegraded":"2024-09-19T07:19:33.249549+0000","last_fullsized":"2024-09-19T07:19:33.249549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246642+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:19:33.246642+0000","last_peered":"2024-09-19T07:19:33.246642+0000","last_clean":"2024-09-19T07:19:33.246642+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:19:33.246642+0000","last_undegraded":"2024-09-19T07:19:33.246642+0000","last_fullsized":"2024-09-19T07:19:33.246642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294674+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:19:33.294674+0000","last_peered":"2024-09-19T07:19:33.294674+0000","last_clean":"2024-09-19T07:19:33.294674+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:19:33.294674+0000","last_undegraded":"2024-09-19T07:19:33.294674+0000","last_fullsized":"2024-09-19T07:19:33.294674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.250618+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:19:33.250618+0000","last_peered":"2024-09-19T07:19:33.250618+0000","last_clean":"2024-09-19T07:19:33.250618+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:19:33.250618+0000","last_undegraded":"2024-09-19T07:19:33.250618+0000","last_fullsized":"2024-09-19T07:19:33.250618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.250735+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:19:33.250735+0000","last_peered":"2024-09-19T07:19:33.250735+0000","last_clean":"2024-09-19T07:19:33.250735+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:19:33.250735+0000","last_undegraded":"2024-09-19T07:19:33.250735+0000","last_fullsized":"2024-09-19T07:19:33.250735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664631+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:19:33.664631+0000","last_peered":"2024-09-19T07:19:33.664631+0000","last_clean":"2024-09-19T07:19:33.664631+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:19:33.664631+0000","last_undegraded":"2024-09-19T07:19:33.664631+0000","last_fullsized":"2024-09-19T07:19:33.664631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:48.308137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246511+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:19:33.246511+0000","last_peered":"2024-09-19T07:19:33.246511+0000","last_clean":"2024-09-19T07:19:33.246511+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:19:33.246511+0000","last_undegraded":"2024-09-19T07:19:33.246511+0000","last_fullsized":"2024-09-19T07:19:33.246511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.251942+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:19:33.251942+0000","last_peered":"2024-09-19T07:19:33.251942+0000","last_clean":"2024-09-19T07:19:33.251942+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:19:33.251942+0000","last_undegraded":"2024-09-19T07:19:33.251942+0000","last_fullsized":"2024-09-19T07:19:33.251942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294839+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:19:33.294839+0000","last_peered":"2024-09-19T07:19:33.294839+0000","last_clean":"2024-09-19T07:19:33.294839+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:19:33.294839+0000","last_undegraded":"2024-09-19T07:19:33.294839+0000","last_fullsized":"2024-09-19T07:19:33.294839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612637+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:19:33.612637+0000","last_peered":"2024-09-19T07:19:33.612637+0000","last_clean":"2024-09-19T07:19:33.612637+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:19:33.612637+0000","last_undegraded":"2024-09-19T07:19:33.612637+0000","last_fullsized":"2024-09-19T07:19:33.612637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.667357+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:19:33.667357+0000","last_peered":"2024-09-19T07:19:33.667357+0000","last_clean":"2024-09-19T07:19:33.667357+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:19:33.667357+0000","last_undegraded":"2024-09-19T07:19:33.667357+0000","last_fullsized":"2024-09-19T07:19:33.667357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:23.367365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.665141+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:19:33.665141+0000","last_peered":"2024-09-19T07:19:33.665141+0000","last_clean":"2024-09-19T07:19:33.665141+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:19:33.665141+0000","last_undegraded":"2024-09-19T07:19:33.665141+0000","last_fullsized":"2024-09-19T07:19:33.665141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:57.131912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249571+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:19:33.249571+0000","last_peered":"2024-09-19T07:19:33.249571+0000","last_clean":"2024-09-19T07:19:33.249571+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:19:33.249571+0000","last_undegraded":"2024-09-19T07:19:33.249571+0000","last_fullsized":"2024-09-19T07:19:33.249571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613088+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:19:33.613088+0000","last_peered":"2024-09-19T07:19:33.613088+0000","last_clean":"2024-09-19T07:19:33.613088+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:19:33.613088+0000","last_undegraded":"2024-09-19T07:19:33.613088+0000","last_fullsized":"2024-09-19T07:19:33.613088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294329+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:19:33.294329+0000","last_peered":"2024-09-19T07:19:33.294329+0000","last_clean":"2024-09-19T07:19:33.294329+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:19:33.294329+0000","last_undegraded":"2024-09-19T07:19:33.294329+0000","last_fullsized":"2024-09-19T07:19:33.294329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294663+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:19:33.294663+0000","last_peered":"2024-09-19T07:19:33.294663+0000","last_clean":"2024-09-19T07:19:33.294663+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:19:33.294663+0000","last_undegraded":"2024-09-19T07:19:33.294663+0000","last_fullsized":"2024-09-19T07:19:33.294663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.661513+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:19:33.661513+0000","last_peered":"2024-09-19T07:19:33.661513+0000","last_clean":"2024-09-19T07:19:33.661513+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:19:33.661513+0000","last_undegraded":"2024-09-19T07:19:33.661513+0000","last_fullsized":"2024-09-19T07:19:33.661513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612502+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:19:33.612502+0000","last_peered":"2024-09-19T07:19:33.612502+0000","last_clean":"2024-09-19T07:19:33.612502+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:19:33.612502+0000","last_undegraded":"2024-09-19T07:19:33.612502+0000","last_fullsized":"2024-09-19T07:19:33.612502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.251931+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:19:33.251931+0000","last_peered":"2024-09-19T07:19:33.251931+0000","last_clean":"2024-09-19T07:19:33.251931+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:19:33.251931+0000","last_undegraded":"2024-09-19T07:19:33.251931+0000","last_fullsized":"2024-09-19T07:19:33.251931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613244+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:19:33.613244+0000","last_peered":"2024-09-19T07:19:33.613244+0000","last_clean":"2024-09-19T07:19:33.613244+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:19:33.613244+0000","last_undegraded":"2024-09-19T07:19:33.613244+0000","last_fullsized":"2024-09-19T07:19:33.613244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.662216+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:19:33.662216+0000","last_peered":"2024-09-19T07:19:33.662216+0000","last_clean":"2024-09-19T07:19:33.662216+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:19:33.662216+0000","last_undegraded":"2024-09-19T07:19:33.662216+0000","last_fullsized":"2024-09-19T07:19:33.662216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613560+0000","last_change":"2024-09-19T07:19:32.292026+0000","last_active":"2024-09-19T07:19:33.613560+0000","last_peered":"2024-09-19T07:19:33.613560+0000","last_clean":"2024-09-19T07:19:33.613560+0000","last_became_active":"2024-09-19T07:19:32.291037+0000","last_became_peered":"2024-09-19T07:19:32.291037+0000","last_unstale":"2024-09-19T07:19:33.613560+0000","last_undegraded":"2024-09-19T07:19:33.613560+0000","last_fullsized":"2024-09-19T07:19:33.613560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:22.190763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246634+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:19:33.246634+0000","last_peered":"2024-09-19T07:19:33.246634+0000","last_clean":"2024-09-19T07:19:33.246634+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:19:33.246634+0000","last_undegraded":"2024-09-19T07:19:33.246634+0000","last_fullsized":"2024-09-19T07:19:33.246634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.251515+0000","last_change":"2024-09-19T07:19:32.276600+0000","last_active":"2024-09-19T07:19:33.251515+0000","last_peered":"2024-09-19T07:19:33.251515+0000","last_clean":"2024-09-19T07:19:33.251515+0000","last_became_active":"2024-09-19T07:19:32.275766+0000","last_became_peered":"2024-09-19T07:19:32.275766+0000","last_unstale":"2024-09-19T07:19:33.251515+0000","last_undegraded":"2024-09-19T07:19:33.251515+0000","last_fullsized":"2024-09-19T07:19:33.251515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:48.506399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.661835+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:19:33.661835+0000","last_peered":"2024-09-19T07:19:33.661835+0000","last_clean":"2024-09-19T07:19:33.661835+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:19:33.661835+0000","last_undegraded":"2024-09-19T07:19:33.661835+0000","last_fullsized":"2024-09-19T07:19:33.661835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613366+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:19:33.613366+0000","last_peered":"2024-09-19T07:19:33.613366+0000","last_clean":"2024-09-19T07:19:33.613366+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:19:33.613366+0000","last_undegraded":"2024-09-19T07:19:33.613366+0000","last_fullsized":"2024-09-19T07:19:33.613366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612290+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:19:33.612290+0000","last_peered":"2024-09-19T07:19:33.612290+0000","last_clean":"2024-09-19T07:19:33.612290+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:19:33.612290+0000","last_undegraded":"2024-09-19T07:19:33.612290+0000","last_fullsized":"2024-09-19T07:19:33.612290+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249670+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:19:33.249670+0000","last_peered":"2024-09-19T07:19:33.249670+0000","last_clean":"2024-09-19T07:19:33.249670+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:19:33.249670+0000","last_undegraded":"2024-09-19T07:19:33.249670+0000","last_fullsized":"2024-09-19T07:19:33.249670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.665399+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:19:33.665399+0000","last_peered":"2024-09-19T07:19:33.665399+0000","last_clean":"2024-09-19T07:19:33.665399+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:19:33.665399+0000","last_undegraded":"2024-09-19T07:19:33.665399+0000","last_fullsized":"2024-09-19T07:19:33.665399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.611996+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:19:33.611996+0000","last_peered":"2024-09-19T07:19:33.611996+0000","last_clean":"2024-09-19T07:19:33.611996+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:19:33.611996+0000","last_undegraded":"2024-09-19T07:19:33.611996+0000","last_fullsized":"2024-09-19T07:19:33.611996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.665475+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:19:33.665475+0000","last_peered":"2024-09-19T07:19:33.665475+0000","last_clean":"2024-09-19T07:19:33.665475+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:19:33.665475+0000","last_undegraded":"2024-09-19T07:19:33.665475+0000","last_fullsized":"2024-09-19T07:19:33.665475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:20.824916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.661830+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:19:33.661830+0000","last_peered":"2024-09-19T07:19:33.661830+0000","last_clean":"2024-09-19T07:19:33.661830+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:19:33.661830+0000","last_undegraded":"2024-09-19T07:19:33.661830+0000","last_fullsized":"2024-09-19T07:19:33.661830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.247106+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:19:33.247106+0000","last_peered":"2024-09-19T07:19:33.247106+0000","last_clean":"2024-09-19T07:19:33.247106+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:19:33.247106+0000","last_undegraded":"2024-09-19T07:19:33.247106+0000","last_fullsized":"2024-09-19T07:19:33.247106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.249868+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:19:33.249868+0000","last_peered":"2024-09-19T07:19:33.249868+0000","last_clean":"2024-09-19T07:19:33.249868+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:19:33.249868+0000","last_undegraded":"2024-09-19T07:19:33.249868+0000","last_fullsized":"2024-09-19T07:19:33.249868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.250543+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:19:33.250543+0000","last_peered":"2024-09-19T07:19:33.250543+0000","last_clean":"2024-09-19T07:19:33.250543+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:19:33.250543+0000","last_undegraded":"2024-09-19T07:19:33.250543+0000","last_fullsized":"2024-09-19T07:19:33.250543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246829+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:19:33.246829+0000","last_peered":"2024-09-19T07:19:33.246829+0000","last_clean":"2024-09-19T07:19:33.246829+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:19:33.246829+0000","last_undegraded":"2024-09-19T07:19:33.246829+0000","last_fullsized":"2024-09-19T07:19:33.246829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.251329+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:19:33.251329+0000","last_peered":"2024-09-19T07:19:33.251329+0000","last_clean":"2024-09-19T07:19:33.251329+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:19:33.251329+0000","last_undegraded":"2024-09-19T07:19:33.251329+0000","last_fullsized":"2024-09-19T07:19:33.251329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613452+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:19:33.613452+0000","last_peered":"2024-09-19T07:19:33.613452+0000","last_clean":"2024-09-19T07:19:33.613452+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:19:33.613452+0000","last_undegraded":"2024-09-19T07:19:33.613452+0000","last_fullsized":"2024-09-19T07:19:33.613452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246699+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:19:33.246699+0000","last_peered":"2024-09-19T07:19:33.246699+0000","last_clean":"2024-09-19T07:19:33.246699+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:19:33.246699+0000","last_undegraded":"2024-09-19T07:19:33.246699+0000","last_fullsized":"2024-09-19T07:19:33.246699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.247108+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:19:33.247108+0000","last_peered":"2024-09-19T07:19:33.247108+0000","last_clean":"2024-09-19T07:19:33.247108+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:19:33.247108+0000","last_undegraded":"2024-09-19T07:19:33.247108+0000","last_fullsized":"2024-09-19T07:19:33.247108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:50.700962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664417+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:19:33.664417+0000","last_peered":"2024-09-19T07:19:33.664417+0000","last_clean":"2024-09-19T07:19:33.664417+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:19:33.664417+0000","last_undegraded":"2024-09-19T07:19:33.664417+0000","last_fullsized":"2024-09-19T07:19:33.664417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.294946+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:19:33.294946+0000","last_peered":"2024-09-19T07:19:33.294946+0000","last_clean":"2024-09-19T07:19:33.294946+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:19:33.294946+0000","last_undegraded":"2024-09-19T07:19:33.294946+0000","last_fullsized":"2024-09-19T07:19:33.294946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.665154+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:19:33.665154+0000","last_peered":"2024-09-19T07:19:33.665154+0000","last_clean":"2024-09-19T07:19:33.665154+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:19:33.665154+0000","last_undegraded":"2024-09-19T07:19:33.665154+0000","last_fullsized":"2024-09-19T07:19:33.665154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:23.791880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.611979+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:19:33.611979+0000","last_peered":"2024-09-19T07:19:33.611979+0000","last_clean":"2024-09-19T07:19:33.611979+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:19:33.611979+0000","last_undegraded":"2024-09-19T07:19:33.611979+0000","last_fullsized":"2024-09-19T07:19:33.611979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.248962+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:19:33.248962+0000","last_peered":"2024-09-19T07:19:33.248962+0000","last_clean":"2024-09-19T07:19:33.248962+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:19:33.248962+0000","last_undegraded":"2024-09-19T07:19:33.248962+0000","last_fullsized":"2024-09-19T07:19:33.248962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.246815+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:19:33.246815+0000","last_peered":"2024-09-19T07:19:33.246815+0000","last_clean":"2024-09-19T07:19:33.246815+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:19:33.246815+0000","last_undegraded":"2024-09-19T07:19:33.246815+0000","last_fullsized":"2024-09-19T07:19:33.246815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663805+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:19:33.663805+0000","last_peered":"2024-09-19T07:19:33.663805+0000","last_clean":"2024-09-19T07:19:33.663805+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:19:33.663805+0000","last_undegraded":"2024-09-19T07:19:33.663805+0000","last_fullsized":"2024-09-19T07:19:33.663805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.247503+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:19:33.247503+0000","last_peered":"2024-09-19T07:19:33.247503+0000","last_clean":"2024-09-19T07:19:33.247503+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:19:33.247503+0000","last_undegraded":"2024-09-19T07:19:33.247503+0000","last_fullsized":"2024-09-19T07:19:33.247503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664303+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:19:33.664303+0000","last_peered":"2024-09-19T07:19:33.664303+0000","last_clean":"2024-09-19T07:19:33.664303+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:19:33.664303+0000","last_undegraded":"2024-09-19T07:19:33.664303+0000","last_fullsized":"2024-09-19T07:19:33.664303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612858+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:19:33.612858+0000","last_peered":"2024-09-19T07:19:33.612858+0000","last_clean":"2024-09-19T07:19:33.612858+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:19:33.612858+0000","last_undegraded":"2024-09-19T07:19:33.612858+0000","last_fullsized":"2024-09-19T07:19:33.612858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.295119+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:19:33.295119+0000","last_peered":"2024-09-19T07:19:33.295119+0000","last_clean":"2024-09-19T07:19:33.295119+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:19:33.295119+0000","last_undegraded":"2024-09-19T07:19:33.295119+0000","last_fullsized":"2024-09-19T07:19:33.295119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.250914+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:19:33.250914+0000","last_peered":"2024-09-19T07:19:33.250914+0000","last_clean":"2024-09-19T07:19:33.250914+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:19:33.250914+0000","last_undegraded":"2024-09-19T07:19:33.250914+0000","last_fullsized":"2024-09-19T07:19:33.250914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.612853+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:19:33.612853+0000","last_peered":"2024-09-19T07:19:33.612853+0000","last_clean":"2024-09-19T07:19:33.612853+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:19:33.612853+0000","last_undegraded":"2024-09-19T07:19:33.612853+0000","last_fullsized":"2024-09-19T07:19:33.612853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.250556+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:19:33.250556+0000","last_peered":"2024-09-19T07:19:33.250556+0000","last_clean":"2024-09-19T07:19:33.250556+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:19:33.250556+0000","last_undegraded":"2024-09-19T07:19:33.250556+0000","last_fullsized":"2024-09-19T07:19:33.250556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.613176+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:19:33.613176+0000","last_peered":"2024-09-19T07:19:33.613176+0000","last_clean":"2024-09-19T07:19:33.613176+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:19:33.613176+0000","last_undegraded":"2024-09-19T07:19:33.613176+0000","last_fullsized":"2024-09-19T07:19:33.613176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664304+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:19:33.664304+0000","last_peered":"2024-09-19T07:19:33.664304+0000","last_clean":"2024-09-19T07:19:33.664304+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:19:33.664304+0000","last_undegraded":"2024-09-19T07:19:33.664304+0000","last_fullsized":"2024-09-19T07:19:33.664304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:15.036396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664287+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:19:33.664287+0000","last_peered":"2024-09-19T07:19:33.664287+0000","last_clean":"2024-09-19T07:19:33.664287+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:19:33.664287+0000","last_undegraded":"2024-09-19T07:19:33.664287+0000","last_fullsized":"2024-09-19T07:19:33.664287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.663935+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:19:33.663935+0000","last_peered":"2024-09-19T07:19:33.663935+0000","last_clean":"2024-09-19T07:19:33.663935+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:19:33.663935+0000","last_undegraded":"2024-09-19T07:19:33.663935+0000","last_fullsized":"2024-09-19T07:19:33.663935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-19T07:19:33.664953+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:19:33.664953+0000","last_peered":"2024-09-19T07:19:33.664953+0000","last_clean":"2024-09-19T07:19:33.664953+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:19:33.664953+0000","last_undegraded":"2024-09-19T07:19:33.664953+0000","last_fullsized":"2024-09-19T07:19:33.664953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:57.799442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":811008,"data_stored":665033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":74173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":74209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":74083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":74243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":819200,"data_stored":665089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:19:37.655 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-19T07:19:37.655 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-19T07:19:37.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.799+0000 7fae5d13d640 1 Processor -- start 2024-09-19T07:19:37.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.799+0000 7fae5d13d640 1 -- start start 2024-09-19T07:19:37.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5d13d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae58150290 0x7fae580c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5d13d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae58150670 con 0x7fae58153eb0 2024-09-19T07:19:37.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5d13d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae580c2880 con 0x7fae58152b20 2024-09-19T07:19:37.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5d13d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fae580716c0 con 0x7fae58150290 2024-09-19T07:19:37.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5759e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fae58153eb0 0x7fae58154360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41794/0 (socket says 172.21.15.83:41794) 2024-09-19T07:19:37.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5759e640 1 -- 172.21.15.83:0/1297142899 learned_addr learned my addr 172.21.15.83:0/1297142899 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:37.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3980044996 0 0) 0x7fae58150670 con 0x7fae58153eb0 2024-09-19T07:19:37.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae40005740 con 0x7fae58153eb0 2024-09-19T07:19:37.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae56d9d640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae58150290 0x7fae580c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1600476411 0 0) 0x7fae40005740 con 0x7fae58153eb0 2024-09-19T07:19:37.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae58150290 msgr2=0x7fae580c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:37.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae58150290 0x7fae580c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 >> v1:172.21.15.184:6789/0 conn(0x7fae58152b20 legacy=0x7fae580c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae580c2b60 con 0x7fae58153eb0 2024-09-19T07:19:37.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fae50001a00 con 0x7fae58153eb0 2024-09-19T07:19:37.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 >> v1:172.21.15.83:6789/0 conn(0x7fae58153eb0 legacy=0x7fae58154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae58150290 0x7fae40005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae40006230 0x7fae40006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae40006f60 0x7fae400073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae4000dd70 con 0x7fae40006f60 2024-09-19T07:19:37.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fae4000df20 con 0x7fae40006230 2024-09-19T07:19:37.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fae4000e120 con 0x7fae58150290 2024-09-19T07:19:37.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5759e640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae40006f60 0x7fae400073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae56d9d640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae58150290 0x7fae40005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5659c640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae40006230 0x7fae40006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5759e640 1 -- 172.21.15.83:0/1297142899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae58150290 msgr2=0x7fae40005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:37.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5759e640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae58150290 0x7fae40005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5759e640 1 -- 172.21.15.83:0/1297142899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae40006230 msgr2=0x7fae40006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:37.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5759e640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae40006230 0x7fae40006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5759e640 1 -- 172.21.15.83:0/1297142899 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae40005740 con 0x7fae40006f60 2024-09-19T07:19:37.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5659c640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae40006230 0x7fae40006a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:37.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae5759e640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae40006f60 0x7fae400073e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fae5000a850 tx=0x7fae5001d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3c952867da3c45 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:37.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5000a4a0 con 0x7fae40006f60 2024-09-19T07:19:37.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae500ad650 con 0x7fae40006f60 2024-09-19T07:19:37.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.803+0000 7fae55d9b640 1 -- 172.21.15.83:0/1297142899 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae500ada80 con 0x7fae40006f60 2024-09-19T07:19:37.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 -- 172.21.15.83:0/1297142899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae40006f60 msgr2=0x7fae400073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 --2- 172.21.15.83:0/1297142899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae40006f60 0x7fae400073e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fae5000a850 tx=0x7fae5001d3a0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5759e640 1 -- 172.21.15.83:0/1297142899 reap_dead start 2024-09-19T07:19:37.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 -- 172.21.15.83:0/1297142899 shutdown_connections 2024-09-19T07:19:37.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 -- 172.21.15.83:0/1297142899 >> 172.21.15.83:0/1297142899 conn(0x7fae5806bc80 msgr2=0x7fae5806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:37.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 -- 172.21.15.83:0/1297142899 shutdown_connections 2024-09-19T07:19:37.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 -- 172.21.15.83:0/1297142899 wait complete. 2024-09-19T07:19:37.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 Processor -- start 2024-09-19T07:19:37.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 -- start start 2024-09-19T07:19:37.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae40006f60 0x7fae580c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae580c8200 0x7fae58149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5814f740 0x7fae58149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae580c3490 con 0x7fae5814f740 2024-09-19T07:19:37.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fae580c3120 con 0x7fae40006f60 2024-09-19T07:19:37.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fae580c4790 con 0x7fae580c8200 2024-09-19T07:19:37.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5814f740 0x7fae58149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5814f740 0x7fae58149d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55584/0 (socket says 172.21.15.83:55584) 2024-09-19T07:19:37.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5759e640 1 -- 172.21.15.83:0/3957076183 learned_addr learned my addr 172.21.15.83:0/3957076183 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:37.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae56d9d640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae40006f60 0x7fae580c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5659c640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae580c8200 0x7fae58149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5759e640 1 -- 172.21.15.83:0/3957076183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae580c8200 msgr2=0x7fae58149850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5759e640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae580c8200 0x7fae58149850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5759e640 1 -- 172.21.15.83:0/3957076183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae40006f60 msgr2=0x7fae580c0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5759e640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae40006f60 0x7fae580c0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5759e640 1 -- 172.21.15.83:0/3957076183 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae580c5f20 con 0x7fae5814f740 2024-09-19T07:19:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5759e640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5814f740 0x7fae58149d90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fae5001dc40 tx=0x7fae40007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae3f7fe640 1 -- 172.21.15.83:0/3957076183 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae500b2040 con 0x7fae5814f740 2024-09-19T07:19:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae3f7fe640 1 -- 172.21.15.83:0/3957076183 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae5000a480 con 0x7fae5814f740 2024-09-19T07:19:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 -- 172.21.15.83:0/3957076183 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae580c5520 con 0x7fae5814f740 2024-09-19T07:19:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae3f7fe640 1 -- 172.21.15.83:0/3957076183 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae500011d0 con 0x7fae5814f740 2024-09-19T07:19:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.807+0000 7fae5d13d640 1 -- 172.21.15.83:0/3957076183 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae580c11c0 con 0x7fae5814f740 2024-09-19T07:19:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.811+0000 7fae5d13d640 1 -- 172.21.15.83:0/3957076183 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae5806c2c0 con 0x7fae5814f740 2024-09-19T07:19:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.811+0000 7fae3f7fe640 1 -- 172.21.15.83:0/3957076183 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fae500be020 con 0x7fae5814f740 2024-09-19T07:19:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.811+0000 7fae3f7fe640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fae440678c0 0x7fae44069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.811+0000 7fae3f7fe640 1 -- 172.21.15.83:0/3957076183 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7fae5011fd20 con 0x7fae5814f740 2024-09-19T07:19:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.811+0000 7fae56d9d640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fae440678c0 0x7fae44069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.811+0000 7fae56d9d640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fae440678c0 0x7fae44069d80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fae4800b9b0 tx=0x7fae48008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.815+0000 7fae3f7fe640 1 -- 172.21.15.83:0/3957076183 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fae500c5020 con 0x7fae5814f740 2024-09-19T07:19:37.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.975+0000 7fae5d13d640 1 -- 172.21.15.83:0/3957076183 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fae5814d1f0 con 0x7fae5814f740 2024-09-19T07:19:37.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.975+0000 7fae3f7fe640 1 -- 172.21.15.83:0/3957076183 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v21) ==== 84+0+197 (secure 0 0 0) 0x7fae500170b0 con 0x7fae5814f740 2024-09-19T07:19:37.977 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:37.978 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2024-09-19T07:19:37.978 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-19T07:19:37.978 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 1, 2024-09-19T07:19:37.978 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": ".mgr" 2024-09-19T07:19:37.978 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-19T07:19:37.978 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-19T07:19:37.978 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 2, 2024-09-19T07:19:37.978 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": "a_metadata" 2024-09-19T07:19:37.979 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-19T07:19:37.979 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-19T07:19:37.979 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 3, 2024-09-19T07:19:37.979 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": "a_data" 2024-09-19T07:19:37.979 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-19T07:19:37.979 INFO:teuthology.orchestra.run.smithi083.stdout:] 2024-09-19T07:19:37.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.979+0000 7fae5d13d640 1 -- 172.21.15.83:0/3957076183 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fae440678c0 msgr2=0x7fae44069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.979+0000 7fae5d13d640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fae440678c0 0x7fae44069d80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fae4800b9b0 tx=0x7fae48008040 comp rx=0 tx=0).stop 2024-09-19T07:19:37.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.979+0000 7fae5d13d640 1 -- 172.21.15.83:0/3957076183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5814f740 msgr2=0x7fae58149d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:37.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.979+0000 7fae5d13d640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5814f740 0x7fae58149d90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fae5001dc40 tx=0x7fae40007a50 comp rx=0 tx=0).stop 2024-09-19T07:19:37.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.979+0000 7fae5d13d640 1 -- 172.21.15.83:0/3957076183 shutdown_connections 2024-09-19T07:19:37.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.979+0000 7fae5d13d640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fae440678c0 0x7fae44069d80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.979+0000 7fae5d13d640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5814f740 0x7fae58149d90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.979+0000 7fae5d13d640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae580c8200 0x7fae58149850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.979+0000 7fae5d13d640 1 --2- 172.21.15.83:0/3957076183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae40006f60 0x7fae580c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:37.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.979+0000 7fae5d13d640 1 -- 172.21.15.83:0/3957076183 >> 172.21.15.83:0/3957076183 conn(0x7fae5806bc80 msgr2=0x7fae58073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:37.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.979+0000 7fae5d13d640 1 -- 172.21.15.83:0/3957076183 shutdown_connections 2024-09-19T07:19:37.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:37.979+0000 7fae5d13d640 1 -- 172.21.15.83:0/3957076183 wait complete. 2024-09-19T07:19:37.996 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'b' 2024-09-19T07:19:37.997 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_metadata --pg_num_min 64 2024-09-19T07:19:38.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f22059a5640 1 Processor -- start 2024-09-19T07:19:38.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f22059a5640 1 -- start start 2024-09-19T07:19:38.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f22059a5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22001533d0 0x7f22000aa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:38.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f22059a5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f22000c2ee0 0x7f22000c3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:38.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f22059a5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2200153970 con 0x7f22000c3e80 2024-09-19T07:19:38.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f22059a5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f22000aadb0 con 0x7f22000c2ee0 2024-09-19T07:19:38.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f22059a5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f22000c2120 con 0x7f22001533d0 2024-09-19T07:19:38.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f21ff7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f22000c2ee0 0x7f22000c3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f21ff7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f22000c2ee0 0x7f22000c3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55592/0 (socket says 172.21.15.83:55592) 2024-09-19T07:19:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f21ff7fe640 1 -- 172.21.15.83:0/1943953113 learned_addr learned my addr 172.21.15.83:0/1943953113 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f21fe7fc640 1 --2- 172.21.15.83:0/1943953113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22001533d0 0x7f22000aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f21ff7fe640 1 -- 172.21.15.83:0/1943953113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22001533d0 msgr2=0x7f22000aa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f21ff7fe640 1 --2- 172.21.15.83:0/1943953113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22001533d0 0x7f22000aa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f21ff7fe640 1 -- 172.21.15.83:0/1943953113 >> v1:172.21.15.184:6789/0 conn(0x7f22000c3e80 legacy=0x7f22000c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.115+0000 7f21ff7fe640 1 -- 172.21.15.83:0/1943953113 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22000c4970 con 0x7f22000c2ee0 2024-09-19T07:19:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.119+0000 7f21ff7fe640 1 --2- 172.21.15.83:0/1943953113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22000c2ee0 0x7f22000c3370 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f21f0009870 tx=0x7f21f001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308285454fedcba6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.119+0000 7f21fdffb640 1 -- 172.21.15.83:0/1943953113 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f00b0070 con 0x7f22000c2ee0 2024-09-19T07:19:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.119+0000 7f21fdffb640 1 -- 172.21.15.83:0/1943953113 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f00ab600 con 0x7f22000c2ee0 2024-09-19T07:19:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.119+0000 7f21fdffb640 1 -- 172.21.15.83:0/1943953113 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f00ab9d0 con 0x7f22000c2ee0 2024-09-19T07:19:38.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.119+0000 7f22059a5640 1 -- 172.21.15.83:0/1943953113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22000c2ee0 msgr2=0x7f22000c3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:38.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.119+0000 7f22059a5640 1 --2- 172.21.15.83:0/1943953113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22000c2ee0 0x7f22000c3370 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f21f0009870 tx=0x7f21f001c490 comp rx=0 tx=0).stop 2024-09-19T07:19:38.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.119+0000 7f22059a5640 1 -- 172.21.15.83:0/1943953113 shutdown_connections 2024-09-19T07:19:38.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.119+0000 7f22059a5640 1 --2- 172.21.15.83:0/1943953113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22000c2ee0 0x7f22000c3370 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.119+0000 7f22059a5640 1 --2- 172.21.15.83:0/1943953113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22001533d0 0x7f22000aa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.119+0000 7f22059a5640 1 -- 172.21.15.83:0/1943953113 >> 172.21.15.83:0/1943953113 conn(0x7f220006bc80 msgr2=0x7f220006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:38.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.119+0000 7f22059a5640 1 -- 172.21.15.83:0/1943953113 shutdown_connections 2024-09-19T07:19:38.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.119+0000 7f22059a5640 1 -- 172.21.15.83:0/1943953113 wait complete. 2024-09-19T07:19:38.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f22059a5640 1 Processor -- start 2024-09-19T07:19:38.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f22059a5640 1 -- start start 2024-09-19T07:19:38.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f22059a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22000c2ee0 0x7f2200152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:38.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f22059a5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22000c3e80 0x7f2200150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:38.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f22059a5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22001533d0 0x7f2200147b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:38.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f22059a5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f220006e020 con 0x7f22000c2ee0 2024-09-19T07:19:38.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f22059a5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f220006dea0 con 0x7f22000c3e80 2024-09-19T07:19:38.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f22059a5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f220006e1a0 con 0x7f22001533d0 2024-09-19T07:19:38.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21feffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22000c2ee0 0x7f2200152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:38.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21feffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22000c2ee0 0x7f2200152620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55608/0 (socket says 172.21.15.83:55608) 2024-09-19T07:19:38.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21feffd640 1 -- 172.21.15.83:0/3703012607 learned_addr learned my addr 172.21.15.83:0/3703012607 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:38.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21fe7fc640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22000c3e80 0x7f2200150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:38.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21ff7fe640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22001533d0 0x7f2200147b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:38.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21feffd640 1 -- 172.21.15.83:0/3703012607 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22001533d0 msgr2=0x7f2200147b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:38.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21feffd640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22001533d0 0x7f2200147b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21feffd640 1 -- 172.21.15.83:0/3703012607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22000c3e80 msgr2=0x7f2200150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:38.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21feffd640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22000c3e80 0x7f2200150f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21feffd640 1 -- 172.21.15.83:0/3703012607 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2200147380 con 0x7f22000c2ee0 2024-09-19T07:19:38.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21fe7fc640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22000c3e80 0x7f2200150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:38.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21feffd640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22000c2ee0 0x7f2200152620 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f21f800c7f0 tx=0x7f21f800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:38.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21e7fff640 1 -- 172.21.15.83:0/3703012607 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8016020 con 0x7f22000c2ee0 2024-09-19T07:19:38.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21e7fff640 1 -- 172.21.15.83:0/3703012607 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f800aa50 con 0x7f22000c2ee0 2024-09-19T07:19:38.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21e7fff640 1 -- 172.21.15.83:0/3703012607 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f800ae40 con 0x7f22000c2ee0 2024-09-19T07:19:38.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f22059a5640 1 -- 172.21.15.83:0/3703012607 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220014d430 con 0x7f22000c2ee0 2024-09-19T07:19:38.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f22059a5640 1 -- 172.21.15.83:0/3703012607 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2200150920 con 0x7f22000c2ee0 2024-09-19T07:19:38.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f22059a5640 1 -- 172.21.15.83:0/3703012607 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22000c3370 con 0x7f22000c2ee0 2024-09-19T07:19:38.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21e7fff640 1 -- 172.21.15.83:0/3703012607 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f21f8019020 con 0x7f22000c2ee0 2024-09-19T07:19:38.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21e7fff640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21ec0678c0 0x7f21ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:38.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21e7fff640 1 -- 172.21.15.83:0/3703012607 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7f21f8085c80 con 0x7f22000c2ee0 2024-09-19T07:19:38.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.123+0000 7f21fe7fc640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21ec0678c0 0x7f21ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:38.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.127+0000 7f21fe7fc640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21ec0678c0 0x7f21ec069d80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2200148fd0 tx=0x7f21f4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:38.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.131+0000 7f21e7fff640 1 -- 172.21.15.83:0/3703012607 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f21f8020030 con 0x7f22000c2ee0 2024-09-19T07:19:38.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.287+0000 7f22059a5640 1 -- 172.21.15.83:0/3703012607 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64} v 0) -- 0x7f220006c9b0 con 0x7f22000c2ee0 2024-09-19T07:19:38.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.375+0000 7f21e7fff640 1 -- 172.21.15.83:0/3703012607 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64}]=0 pool 'b_metadata' created v22) ==== 128+0+0 (secure 0 0 0) 0x7f21f8056860 con 0x7f22000c2ee0 2024-09-19T07:19:38.377 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'b_metadata' created 2024-09-19T07:19:38.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.379+0000 7f22059a5640 1 -- 172.21.15.83:0/3703012607 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21ec0678c0 msgr2=0x7f21ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:38.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.379+0000 7f22059a5640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21ec0678c0 0x7f21ec069d80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2200148fd0 tx=0x7f21f4008040 comp rx=0 tx=0).stop 2024-09-19T07:19:38.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.379+0000 7f22059a5640 1 -- 172.21.15.83:0/3703012607 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22000c2ee0 msgr2=0x7f2200152620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:38.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.379+0000 7f22059a5640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22000c2ee0 0x7f2200152620 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f21f800c7f0 tx=0x7f21f800ccb0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.379+0000 7f22059a5640 1 -- 172.21.15.83:0/3703012607 shutdown_connections 2024-09-19T07:19:38.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.379+0000 7f22059a5640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21ec0678c0 0x7f21ec069d80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.379+0000 7f22059a5640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22001533d0 0x7f2200147b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.379+0000 7f22059a5640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22000c3e80 0x7f2200150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.379+0000 7f22059a5640 1 --2- 172.21.15.83:0/3703012607 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22000c2ee0 0x7f2200152620 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.379+0000 7f22059a5640 1 -- 172.21.15.83:0/3703012607 >> 172.21.15.83:0/3703012607 conn(0x7f220006bc80 msgr2=0x7f220015c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:38.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.379+0000 7f22059a5640 1 -- 172.21.15.83:0/3703012607 shutdown_connections 2024-09-19T07:19:38.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.379+0000 7f22059a5640 1 -- 172.21.15.83:0/3703012607 wait complete. 2024-09-19T07:19:38.400 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-19T07:19:38.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc361f21640 1 Processor -- start 2024-09-19T07:19:38.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc361f21640 1 -- start start 2024-09-19T07:19:38.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc361f21640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc35c0c55a0 0x7fc35c150d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:38.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc361f21640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc35c1512d0 0x7fc35c0ae360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:38.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc361f21640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc35c153d60 con 0x7fc35c0c8f30 2024-09-19T07:19:38.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc361f21640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc35c154970 con 0x7fc35c0c55a0 2024-09-19T07:19:38.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc361f21640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc35c154040 con 0x7fc35c1512d0 2024-09-19T07:19:38.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc35b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc35c0c55a0 0x7fc35c150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:38.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc35b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc35c0c55a0 0x7fc35c150d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55622/0 (socket says 172.21.15.83:55622) 2024-09-19T07:19:38.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc35b7fe640 1 -- 172.21.15.83:0/4103993664 learned_addr learned my addr 172.21.15.83:0/4103993664 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:38.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc35b7fe640 1 -- 172.21.15.83:0/4103993664 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc35c1512d0 msgr2=0x7fc35c0ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:38.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc35b7fe640 1 --2- 172.21.15.83:0/4103993664 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc35c1512d0 0x7fc35c0ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc35b7fe640 1 -- 172.21.15.83:0/4103993664 >> v1:172.21.15.184:6789/0 conn(0x7fc35c0c8f30 legacy=0x7fc35c0adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:38.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.523+0000 7fc35b7fe640 1 -- 172.21.15.83:0/4103993664 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35c154320 con 0x7fc35c0c55a0 2024-09-19T07:19:38.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35b7fe640 1 --2- 172.21.15.83:0/4103993664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc35c0c55a0 0x7fc35c150d90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0c7090 tx=0x7fc34c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca59e1667c89869 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:38.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35a7fc640 1 -- 172.21.15.83:0/4103993664 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c0b0070 con 0x7fc35c0c55a0 2024-09-19T07:19:38.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35a7fc640 1 -- 172.21.15.83:0/4103993664 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc34c0ab600 con 0x7fc35c0c55a0 2024-09-19T07:19:38.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35a7fc640 1 -- 172.21.15.83:0/4103993664 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c0ab9d0 con 0x7fc35c0c55a0 2024-09-19T07:19:38.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 -- 172.21.15.83:0/4103993664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc35c0c55a0 msgr2=0x7fc35c150d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:38.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 --2- 172.21.15.83:0/4103993664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc35c0c55a0 0x7fc35c150d90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0c7090 tx=0x7fc34c01c3e0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 -- 172.21.15.83:0/4103993664 shutdown_connections 2024-09-19T07:19:38.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 --2- 172.21.15.83:0/4103993664 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc35c1512d0 0x7fc35c0ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 --2- 172.21.15.83:0/4103993664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc35c0c55a0 0x7fc35c150d90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 -- 172.21.15.83:0/4103993664 >> 172.21.15.83:0/4103993664 conn(0x7fc35c0c6070 msgr2=0x7fc35c0c6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:38.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 -- 172.21.15.83:0/4103993664 shutdown_connections 2024-09-19T07:19:38.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 -- 172.21.15.83:0/4103993664 wait complete. 2024-09-19T07:19:38.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 Processor -- start 2024-09-19T07:19:38.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 -- start start 2024-09-19T07:19:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc35c0c55a0 0x7fc35c1edd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc35c0c8f30 0x7fc35c142910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc35c1512d0 0x7fc35c145630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c0ac6d0 con 0x7fc35c0c55a0 2024-09-19T07:19:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c0740d0 con 0x7fc35c0c8f30 2024-09-19T07:19:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc35c0c82a0 con 0x7fc35c1512d0 2024-09-19T07:19:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc35c0c55a0 0x7fc35c1edd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc35c0c55a0 0x7fc35c1edd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55634/0 (socket says 172.21.15.83:55634) 2024-09-19T07:19:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35b7fe640 1 -- 172.21.15.83:0/1612059398 learned_addr learned my addr 172.21.15.83:0/1612059398 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35b7fe640 1 -- 172.21.15.83:0/1612059398 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc35c1512d0 msgr2=0x7fc35c145630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35b7fe640 1 --2- 172.21.15.83:0/1612059398 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc35c1512d0 0x7fc35c145630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35b7fe640 1 -- 172.21.15.83:0/1612059398 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc35c0c8f30 msgr2=0x7fc35c142910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35b7fe640 1 --2- 172.21.15.83:0/1612059398 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc35c0c8f30 0x7fc35c142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35b7fe640 1 -- 172.21.15.83:0/1612059398 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35c151100 con 0x7fc35c0c55a0 2024-09-19T07:19:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc35b7fe640 1 --2- 172.21.15.83:0/1612059398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc35c0c55a0 0x7fc35c1edd30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0ae040 tx=0x7fc34c01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc343fff640 1 -- 172.21.15.83:0/1612059398 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c0ba020 con 0x7fc35c0c55a0 2024-09-19T07:19:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc343fff640 1 -- 172.21.15.83:0/1612059398 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc34c0b6620 con 0x7fc35c0c55a0 2024-09-19T07:19:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 -- 172.21.15.83:0/1612059398 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35c147350 con 0x7fc35c0c55a0 2024-09-19T07:19:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc343fff640 1 -- 172.21.15.83:0/1612059398 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c0b0040 con 0x7fc35c0c55a0 2024-09-19T07:19:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.527+0000 7fc361f21640 1 -- 172.21.15.83:0/1612059398 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc35c147e20 con 0x7fc35c0c55a0 2024-09-19T07:19:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.531+0000 7fc361f21640 1 -- 172.21.15.83:0/1612059398 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc35c0c5980 con 0x7fc35c0c55a0 2024-09-19T07:19:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.531+0000 7fc343fff640 1 -- 172.21.15.83:0/1612059398 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc34c0bd020 con 0x7fc35c0c55a0 2024-09-19T07:19:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.531+0000 7fc343fff640 1 --2- 172.21.15.83:0/1612059398 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc3480678c0 0x7fc348069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.531+0000 7fc343fff640 1 -- 172.21.15.83:0/1612059398 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6054+0+0 (secure 0 0 0) 0x7fc34c120430 con 0x7fc35c0c55a0 2024-09-19T07:19:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.531+0000 7fc35affd640 1 --2- 172.21.15.83:0/1612059398 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc3480678c0 0x7fc348069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.531+0000 7fc35affd640 1 --2- 172.21.15.83:0/1612059398 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc3480678c0 0x7fc348069d80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc350007fd0 tx=0x7fc3500079a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:38.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.535+0000 7fc343fff640 1 -- 172.21.15.83:0/1612059398 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc34c0c4030 con 0x7fc35c0c55a0 2024-09-19T07:19:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:38.695+0000 7fc361f21640 1 -- 172.21.15.83:0/1612059398 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc35c142e50 con 0x7fc35c0c55a0 2024-09-19T07:19:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.379+0000 7fc343fff640 1 -- 172.21.15.83:0/1612059398 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'b_data' created v23) ==== 160+0+0 (secure 0 0 0) 0x7fc34c0160f0 con 0x7fc35c0c55a0 2024-09-19T07:19:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'b_data' created 2024-09-19T07:19:39.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.383+0000 7fc361f21640 1 -- 172.21.15.83:0/1612059398 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc3480678c0 msgr2=0x7fc348069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:39.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.383+0000 7fc361f21640 1 --2- 172.21.15.83:0/1612059398 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc3480678c0 0x7fc348069d80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc350007fd0 tx=0x7fc3500079a0 comp rx=0 tx=0).stop 2024-09-19T07:19:39.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.383+0000 7fc361f21640 1 -- 172.21.15.83:0/1612059398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc35c0c55a0 msgr2=0x7fc35c1edd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:39.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.383+0000 7fc361f21640 1 --2- 172.21.15.83:0/1612059398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc35c0c55a0 0x7fc35c1edd30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0ae040 tx=0x7fc34c01ac20 comp rx=0 tx=0).stop 2024-09-19T07:19:39.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.383+0000 7fc361f21640 1 -- 172.21.15.83:0/1612059398 shutdown_connections 2024-09-19T07:19:39.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.383+0000 7fc361f21640 1 --2- 172.21.15.83:0/1612059398 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc3480678c0 0x7fc348069d80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:39.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.383+0000 7fc361f21640 1 --2- 172.21.15.83:0/1612059398 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc35c1512d0 0x7fc35c145630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:39.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.383+0000 7fc361f21640 1 --2- 172.21.15.83:0/1612059398 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc35c0c8f30 0x7fc35c142910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:39.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.383+0000 7fc361f21640 1 --2- 172.21.15.83:0/1612059398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc35c0c55a0 0x7fc35c1edd30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:39.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.383+0000 7fc361f21640 1 -- 172.21.15.83:0/1612059398 >> 172.21.15.83:0/1612059398 conn(0x7fc35c0c6070 msgr2=0x7fc35c14bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:39.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.383+0000 7fc361f21640 1 -- 172.21.15.83:0/1612059398 shutdown_connections 2024-09-19T07:19:39.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.383+0000 7fc361f21640 1 -- 172.21.15.83:0/1612059398 wait complete. 2024-09-19T07:19:39.415 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new b b_metadata b_data 2024-09-19T07:19:39.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.535+0000 7f6437777640 1 Processor -- start 2024-09-19T07:19:39.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.535+0000 7f6437777640 1 -- start start 2024-09-19T07:19:39.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6437777640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64300c55a0 0x7f6430150d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:39.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6437777640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6430153d60 con 0x7f64300c8f30 2024-09-19T07:19:39.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6437777640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6430154040 con 0x7f64301512d0 2024-09-19T07:19:39.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6437777640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6430154970 con 0x7f64300c55a0 2024-09-19T07:19:39.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6434d12640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f64300c8f30 0x7f64300adc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41810/0 (socket says 172.21.15.83:41810) 2024-09-19T07:19:39.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6434d12640 1 -- 172.21.15.83:0/1193934605 learned_addr learned my addr 172.21.15.83:0/1193934605 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:39.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6435513640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64300c55a0 0x7f6430150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:39.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3367863498 0 0) 0x7f6430153d60 con 0x7f64300c8f30 2024-09-19T07:19:39.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f641c003530 con 0x7f64300c8f30 2024-09-19T07:19:39.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1283743104 0 0) 0x7f641c003530 con 0x7f64300c8f30 2024-09-19T07:19:39.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64300c55a0 msgr2=0x7f6430150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:39.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64300c55a0 0x7f6430150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:39.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 >> v1:172.21.15.184:6789/0 conn(0x7f64301512d0 legacy=0x7f64300ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:39.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6430154320 con 0x7f64300c8f30 2024-09-19T07:19:39.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f6420001680 con 0x7f64300c8f30 2024-09-19T07:19:39.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 >> v1:172.21.15.83:6789/0 conn(0x7f64300c8f30 legacy=0x7f64300adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:39.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f641c0039f0 0x7f641c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:39.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f641c004810 0x7f641c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:39.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64300c55a0 0x7f641c0052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:39.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6430154970 con 0x7f641c004810 2024-09-19T07:19:39.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f641c00bb90 con 0x7f641c0039f0 2024-09-19T07:19:39.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f641c00bd90 con 0x7f64300c55a0 2024-09-19T07:19:39.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6434d12640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f641c004810 0x7f641c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:39.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6435513640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f641c0039f0 0x7f641c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:39.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6435d14640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64300c55a0 0x7f641c0052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:39.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6434d12640 1 -- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64300c55a0 msgr2=0x7f641c0052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:39.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6434d12640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64300c55a0 0x7f641c0052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:39.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6434d12640 1 -- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f641c0039f0 msgr2=0x7f641c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:39.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6434d12640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f641c0039f0 0x7f641c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:39.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6434d12640 1 -- 172.21.15.83:0/1193934605 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641c003530 con 0x7f641c004810 2024-09-19T07:19:39.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6435513640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f641c0039f0 0x7f641c003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:39.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6435d14640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64300c55a0 0x7f641c0052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:39.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.539+0000 7f6434d12640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f641c004810 0x7f641c004c90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f642000c210 tx=0x7f642001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de29b6f390b6d73 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:39.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6420007d20 con 0x7f641c004810 2024-09-19T07:19:39.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64200a9070 con 0x7f641c004810 2024-09-19T07:19:39.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6427fff640 1 -- 172.21.15.83:0/1193934605 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6420005d40 con 0x7f641c004810 2024-09-19T07:19:39.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 -- 172.21.15.83:0/1193934605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f641c004810 msgr2=0x7f641c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:39.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 --2- 172.21.15.83:0/1193934605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f641c004810 0x7f641c004c90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f642000c210 tx=0x7f642001cc60 comp rx=0 tx=0).stop 2024-09-19T07:19:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6435d14640 1 -- 172.21.15.83:0/1193934605 reap_dead start 2024-09-19T07:19:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 -- 172.21.15.83:0/1193934605 shutdown_connections 2024-09-19T07:19:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 -- 172.21.15.83:0/1193934605 >> 172.21.15.83:0/1193934605 conn(0x7f64300c6070 msgr2=0x7f64300c6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 -- 172.21.15.83:0/1193934605 shutdown_connections 2024-09-19T07:19:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 -- 172.21.15.83:0/1193934605 wait complete. 2024-09-19T07:19:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 Processor -- start 2024-09-19T07:19:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 -- start start 2024-09-19T07:19:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f641c004810 0x7f64300a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64300a9060 0x7f64300c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6430143800 0x7f6430143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6430074250 con 0x7f64300a9060 2024-09-19T07:19:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f64300740d0 con 0x7f641c004810 2024-09-19T07:19:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f64300c5980 con 0x7f6430143800 2024-09-19T07:19:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6434d12640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64300a9060 0x7f64300c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6434d12640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64300a9060 0x7f64300c2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55650/0 (socket says 172.21.15.83:55650) 2024-09-19T07:19:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6434d12640 1 -- 172.21.15.83:0/354615963 learned_addr learned my addr 172.21.15.83:0/354615963 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6434d12640 1 -- 172.21.15.83:0/354615963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6430143800 msgr2=0x7f6430143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6434d12640 1 --2- 172.21.15.83:0/354615963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6430143800 0x7f6430143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6434d12640 1 -- 172.21.15.83:0/354615963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f641c004810 msgr2=0x7f64300a94f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:19:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6434d12640 1 --2- 172.21.15.83:0/354615963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f641c004810 0x7f64300a94f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6434d12640 1 -- 172.21.15.83:0/354615963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64301529c0 con 0x7f64300a9060 2024-09-19T07:19:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6434d12640 1 --2- 172.21.15.83:0/354615963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64300a9060 0x7f64300c2c00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f642001ca90 tx=0x7f641c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6425ffb640 1 -- 172.21.15.83:0/354615963 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64200a5850 con 0x7f64300a9060 2024-09-19T07:19:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6425ffb640 1 -- 172.21.15.83:0/354615963 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64200a59f0 con 0x7f64300a9060 2024-09-19T07:19:39.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 -- 172.21.15.83:0/354615963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6430152c60 con 0x7f64300a9060 2024-09-19T07:19:39.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6425ffb640 1 -- 172.21.15.83:0/354615963 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64200a5da0 con 0x7f64300a9060 2024-09-19T07:19:39.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.543+0000 7f6437777640 1 -- 172.21.15.83:0/354615963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6430151d90 con 0x7f64300a9060 2024-09-19T07:19:39.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.547+0000 7f6437777640 1 -- 172.21.15.83:0/354615963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64300adc30 con 0x7f64300a9060 2024-09-19T07:19:39.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.547+0000 7f6425ffb640 1 -- 172.21.15.83:0/354615963 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f64200047f0 con 0x7f64300a9060 2024-09-19T07:19:39.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.547+0000 7f6425ffb640 1 --2- 172.21.15.83:0/354615963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f64140678c0 0x7f6414069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:39.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.547+0000 7f6425ffb640 1 -- 172.21.15.83:0/354615963 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6431+0+0 (secure 0 0 0) 0x7f642011f4c0 con 0x7f64300a9060 2024-09-19T07:19:39.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.547+0000 7f6435513640 1 --2- 172.21.15.83:0/354615963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f64140678c0 0x7f6414069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:39.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.547+0000 7f6435513640 1 --2- 172.21.15.83:0/354615963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f64140678c0 0x7f6414069d80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f642c009a00 tx=0x7f642c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:39.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.551+0000 7f6425ffb640 1 -- 172.21.15.83:0/354615963 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6420016970 con 0x7f64300a9060 2024-09-19T07:19:39.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:39.751+0000 7f6437777640 1 -- 172.21.15.83:0/354615963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"} v 0) -- 0x7f64300c3750 con 0x7f64300a9060 2024-09-19T07:19:40.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.415+0000 7f6425ffb640 1 -- 172.21.15.83:0/354615963 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"}]=0 new fs with metadata pool 4 and data pool 5 v9) ==== 157+0+0 (secure 0 0 0) 0x7f64200efdc0 con 0x7f64300a9060 2024-09-19T07:19:40.416 INFO:teuthology.orchestra.run.smithi083.stderr:new fs with metadata pool 4 and data pool 5 2024-09-19T07:19:40.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.419+0000 7f6437777640 1 -- 172.21.15.83:0/354615963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f64140678c0 msgr2=0x7f6414069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.419+0000 7f6437777640 1 --2- 172.21.15.83:0/354615963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f64140678c0 0x7f6414069d80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f642c009a00 tx=0x7f642c008040 comp rx=0 tx=0).stop 2024-09-19T07:19:40.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.419+0000 7f6437777640 1 -- 172.21.15.83:0/354615963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64300a9060 msgr2=0x7f64300c2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.419+0000 7f6437777640 1 --2- 172.21.15.83:0/354615963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64300a9060 0x7f64300c2c00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f642001ca90 tx=0x7f641c005840 comp rx=0 tx=0).stop 2024-09-19T07:19:40.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.419+0000 7f6437777640 1 -- 172.21.15.83:0/354615963 shutdown_connections 2024-09-19T07:19:40.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.419+0000 7f6437777640 1 --2- 172.21.15.83:0/354615963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f64140678c0 0x7f6414069d80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.419+0000 7f6437777640 1 --2- 172.21.15.83:0/354615963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6430143800 0x7f6430143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.419+0000 7f6437777640 1 --2- 172.21.15.83:0/354615963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64300a9060 0x7f64300c2c00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.419+0000 7f6437777640 1 --2- 172.21.15.83:0/354615963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f641c004810 0x7f64300a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.419+0000 7f6437777640 1 -- 172.21.15.83:0/354615963 >> 172.21.15.83:0/354615963 conn(0x7f64300c6070 msgr2=0x7f64300ade40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:40.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.419+0000 7f6437777640 1 -- 172.21.15.83:0/354615963 shutdown_connections 2024-09-19T07:19:40.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.419+0000 7f6437777640 1 -- 172.21.15.83:0/354615963 wait complete. 2024-09-19T07:19:40.444 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:40.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff61ed05640 1 Processor -- start 2024-09-19T07:19:40.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff61ed05640 1 -- start start 2024-09-19T07:19:40.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff61ed05640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff618152b20 0x7ff6180c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff61ed05640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff618150670 con 0x7ff6180c45c0 2024-09-19T07:19:40.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff61ed05640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6180c2880 con 0x7ff618150290 2024-09-19T07:19:40.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff61ed05640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff6180716c0 con 0x7ff618152b20 2024-09-19T07:19:40.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff61d2a2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff6180c45c0 0x7ff618153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51424/0 (socket says 172.21.15.83:51424) 2024-09-19T07:19:40.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff61d2a2640 1 -- 172.21.15.83:0/3697203962 learned_addr learned my addr 172.21.15.83:0/3697203962 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:40.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff617fff640 1 --2- 172.21.15.83:0/3697203962 >> v2:172.21.15.184:3300/0 conn(0x7ff618152b20 0x7ff6180c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1693205191 0 0) 0x7ff618150670 con 0x7ff6180c45c0 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff604003530 con 0x7ff6180c45c0 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1608027893 0 0) 0x7ff604003530 con 0x7ff6180c45c0 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 >> v2:172.21.15.184:3300/0 conn(0x7ff618152b20 msgr2=0x7ff6180c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 --2- 172.21.15.83:0/3697203962 >> v2:172.21.15.184:3300/0 conn(0x7ff618152b20 0x7ff6180c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 >> v1:172.21.15.184:6789/0 conn(0x7ff618150290 legacy=0x7ff6180c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6180c2b60 con 0x7ff6180c45c0 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7ff610001a00 con 0x7ff6180c45c0 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 >> v1:172.21.15.83:6789/0 conn(0x7ff6180c45c0 legacy=0x7ff618153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 --2- 172.21.15.83:0/3697203962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6040039c0 0x7ff604003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 --2- 172.21.15.83:0/3697203962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff604004810 0x7ff604004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 --2- 172.21.15.83:0/3697203962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6040051d0 0x7ff60400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff60400c040 con 0x7ff604004810 2024-09-19T07:19:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff60400c1f0 con 0x7ff6040039c0 2024-09-19T07:19:40.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff60400c3f0 con 0x7ff6040051d0 2024-09-19T07:19:40.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff617fff640 1 --2- 172.21.15.83:0/3697203962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff604004810 0x7ff604004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff61d2a2640 1 --2- 172.21.15.83:0/3697203962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6040051d0 0x7ff60400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff61caa1640 1 --2- 172.21.15.83:0/3697203962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6040039c0 0x7ff604003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff617fff640 1 -- 172.21.15.83:0/3697203962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6040051d0 msgr2=0x7ff60400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:40.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff617fff640 1 --2- 172.21.15.83:0/3697203962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6040051d0 0x7ff60400b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff617fff640 1 -- 172.21.15.83:0/3697203962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6040039c0 msgr2=0x7ff604003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:40.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff617fff640 1 --2- 172.21.15.83:0/3697203962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6040039c0 0x7ff604003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff617fff640 1 -- 172.21.15.83:0/3697203962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff604003530 con 0x7ff604004810 2024-09-19T07:19:40.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.563+0000 7ff61d2a2640 1 -- 172.21.15.83:0/3697203962 reap_dead start 2024-09-19T07:19:40.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff617fff640 1 --2- 172.21.15.83:0/3697203962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff604004810 0x7ff604004c90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff608009ae0 tx=0x7ff60801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77923109df5c0dc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:40.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6080b2070 con 0x7ff604004810 2024-09-19T07:19:40.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6080a5680 con 0x7ff604004810 2024-09-19T07:19:40.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff6177fe640 1 -- 172.21.15.83:0/3697203962 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6080096b0 con 0x7ff604004810 2024-09-19T07:19:40.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 -- 172.21.15.83:0/3697203962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff604004810 msgr2=0x7ff604004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 --2- 172.21.15.83:0/3697203962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff604004810 0x7ff604004c90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff608009ae0 tx=0x7ff60801ca90 comp rx=0 tx=0).stop 2024-09-19T07:19:40.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 -- 172.21.15.83:0/3697203962 shutdown_connections 2024-09-19T07:19:40.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 --2- 172.21.15.83:0/3697203962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff604004810 0x7ff604004c90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 -- 172.21.15.83:0/3697203962 >> 172.21.15.83:0/3697203962 conn(0x7ff61806bc80 msgr2=0x7ff61806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:40.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 -- 172.21.15.83:0/3697203962 shutdown_connections 2024-09-19T07:19:40.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 -- 172.21.15.83:0/3697203962 wait complete. 2024-09-19T07:19:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 Processor -- start 2024-09-19T07:19:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 -- start start 2024-09-19T07:19:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff604004810 0x7ff6180c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6180c5e40 0x7ff618145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff618146210 0x7ff6181417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff6180c3490 con 0x7ff618146210 2024-09-19T07:19:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff6180c3120 con 0x7ff604004810 2024-09-19T07:19:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61ed05640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff6180c4080 con 0x7ff6180c5e40 2024-09-19T07:19:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61d2a2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff618146210 0x7ff6181417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61d2a2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff618146210 0x7ff6181417d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40316/0 (socket says 172.21.15.83:40316) 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61d2a2640 1 -- 172.21.15.83:0/1419373773 learned_addr learned my addr 172.21.15.83:0/1419373773 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61d2a2640 1 -- 172.21.15.83:0/1419373773 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6180c5e40 msgr2=0x7ff618145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff617fff640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6180c5e40 0x7ff618145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61caa1640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff604004810 0x7ff6180c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61d2a2640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6180c5e40 0x7ff618145cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61d2a2640 1 -- 172.21.15.83:0/1419373773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff604004810 msgr2=0x7ff6180c6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61d2a2640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff604004810 0x7ff6180c6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61d2a2640 1 -- 172.21.15.83:0/1419373773 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff618154c50 con 0x7ff618146210 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.567+0000 7ff61d2a2640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff618146210 0x7ff6181417d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff6180c86b0 tx=0x7ff610010c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.571+0000 7ff6157fa640 1 -- 172.21.15.83:0/1419373773 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61001a070 con 0x7ff618146210 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.571+0000 7ff6157fa640 1 -- 172.21.15.83:0/1419373773 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff610011040 con 0x7ff618146210 2024-09-19T07:19:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.571+0000 7ff61ed05640 1 -- 172.21.15.83:0/1419373773 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff618151300 con 0x7ff618146210 2024-09-19T07:19:40.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.571+0000 7ff6157fa640 1 -- 172.21.15.83:0/1419373773 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61000ea70 con 0x7ff618146210 2024-09-19T07:19:40.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.571+0000 7ff61ed05640 1 -- 172.21.15.83:0/1419373773 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6180c5640 con 0x7ff618146210 2024-09-19T07:19:40.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.571+0000 7ff61ed05640 1 -- 172.21.15.83:0/1419373773 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6180c1ff0 con 0x7ff618146210 2024-09-19T07:19:40.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.571+0000 7ff6157fa640 1 -- 172.21.15.83:0/1419373773 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff61001d070 con 0x7ff618146210 2024-09-19T07:19:40.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.571+0000 7ff6157fa640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5f80678f0 0x7ff5f8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.571+0000 7ff6157fa640 1 -- 172.21.15.83:0/1419373773 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6521+0+0 (secure 0 0 0) 0x7ff6100886e0 con 0x7ff618146210 2024-09-19T07:19:40.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.571+0000 7ff61caa1640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5f80678f0 0x7ff5f8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.571+0000 7ff61caa1640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5f80678f0 0x7ff5f8069db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff600000c10 tx=0x7ff600009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:40.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.575+0000 7ff6157fa640 1 -- 172.21.15.83:0/1419373773 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff618151300 con 0x7ff618146210 2024-09-19T07:19:40.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.739+0000 7ff61ed05640 1 -- 172.21.15.83:0/1419373773 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff6180c3e90 con 0x7ff618146210 2024-09-19T07:19:40.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.739+0000 7ff6157fa640 1 -- 172.21.15.83:0/1419373773 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+20464 (secure 0 0 0) 0x7ff610058f80 con 0x7ff618146210 2024-09-19T07:19:40.741 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:40.741 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":24,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:39.753103+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:40.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.743+0000 7ff61ed05640 1 -- 172.21.15.83:0/1419373773 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5f80678f0 msgr2=0x7ff5f8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.743+0000 7ff61ed05640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5f80678f0 0x7ff5f8069db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff600000c10 tx=0x7ff600009040 comp rx=0 tx=0).stop 2024-09-19T07:19:40.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.743+0000 7ff61ed05640 1 -- 172.21.15.83:0/1419373773 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff618146210 msgr2=0x7ff6181417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.743+0000 7ff61ed05640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff618146210 0x7ff6181417d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff6180c86b0 tx=0x7ff610010c30 comp rx=0 tx=0).stop 2024-09-19T07:19:40.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.743+0000 7ff61ed05640 1 -- 172.21.15.83:0/1419373773 shutdown_connections 2024-09-19T07:19:40.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.743+0000 7ff61ed05640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5f80678f0 0x7ff5f8069db0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.743+0000 7ff61ed05640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff618146210 0x7ff6181417d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.743+0000 7ff61ed05640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6180c5e40 0x7ff618145cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.743+0000 7ff61ed05640 1 --2- 172.21.15.83:0/1419373773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff604004810 0x7ff6180c6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.743+0000 7ff61ed05640 1 -- 172.21.15.83:0/1419373773 >> 172.21.15.83:0/1419373773 conn(0x7ff61806bc80 msgr2=0x7ff61806dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:40.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.743+0000 7ff61ed05640 1 -- 172.21.15.83:0/1419373773 shutdown_connections 2024-09-19T07:19:40.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.743+0000 7ff61ed05640 1 -- 172.21.15.83:0/1419373773 wait complete. 2024-09-19T07:19:40.760 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.879+0000 7feec9bf6640 1 Processor -- start 2024-09-19T07:19:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec9bf6640 1 -- start start 2024-09-19T07:19:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec9bf6640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7feec40c42e0 0x7feec406e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec9bf6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feec41566b0 0x7feec40abb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec9bf6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feec41592c0 con 0x7feec4159a80 2024-09-19T07:19:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec9bf6640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7feec40ac070 con 0x7feec40c42e0 2024-09-19T07:19:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec9bf6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feec41595f0 con 0x7feec41566b0 2024-09-19T07:19:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7feec4159a80 0x7feec4153060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51434/0 (socket says 172.21.15.83:51434) 2024-09-19T07:19:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec3fff640 1 -- 172.21.15.83:0/3393537086 learned_addr learned my addr 172.21.15.83:0/3393537086 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:40.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec2ffd640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feec41566b0 0x7feec40abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec37fe640 1 --2- 172.21.15.83:0/3393537086 >> v2:172.21.15.184:3300/0 conn(0x7feec40c42e0 0x7feec406e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 60890353 0 0) 0x7feec41592c0 con 0x7feec4159a80 2024-09-19T07:19:40.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feeb0003530 con 0x7feec4159a80 2024-09-19T07:19:40.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2734038451 0 0) 0x7feeb0003530 con 0x7feec4159a80 2024-09-19T07:19:40.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feec41566b0 msgr2=0x7feec40abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:40.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feec41566b0 0x7feec40abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 >> v2:172.21.15.184:3300/0 conn(0x7feec40c42e0 msgr2=0x7feec406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 --2- 172.21.15.83:0/3393537086 >> v2:172.21.15.184:3300/0 conn(0x7feec40c42e0 0x7feec406e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec406e900 con 0x7feec4159a80 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7feeb4001680 con 0x7feec4159a80 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 >> v1:172.21.15.83:6789/0 conn(0x7feec4159a80 legacy=0x7feec4153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feeb00039c0 0x7feeb0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feec41566b0 0x7feeb0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feeb0004d50 0x7feeb00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feec41595f0 con 0x7feeb0004d50 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feeb000b9d0 con 0x7feeb00039c0 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feeb000bb50 con 0x7feec41566b0 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec3fff640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feeb0004d50 0x7feeb00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec37fe640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feeb00039c0 0x7feeb0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec2ffd640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feec41566b0 0x7feeb0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec3fff640 1 -- 172.21.15.83:0/3393537086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feec41566b0 msgr2=0x7feeb0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:40.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec3fff640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feec41566b0 0x7feeb0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec3fff640 1 -- 172.21.15.83:0/3393537086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feeb00039c0 msgr2=0x7feeb0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:40.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec3fff640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feeb00039c0 0x7feeb0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec3fff640 1 -- 172.21.15.83:0/3393537086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb0003530 con 0x7feeb0004d50 2024-09-19T07:19:40.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.883+0000 7feec37fe640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feeb00039c0 0x7feeb0003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:19:40.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec3fff640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feeb0004d50 0x7feeb00051d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feeb400c270 tx=0x7feeb401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82c5041e9e3c32f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:40.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb4007d40 con 0x7feeb0004d50 2024-09-19T07:19:40.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeb40a9020 con 0x7feeb0004d50 2024-09-19T07:19:40.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec27fc640 1 -- 172.21.15.83:0/3393537086 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb40a5bc0 con 0x7feeb0004d50 2024-09-19T07:19:40.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 -- 172.21.15.83:0/3393537086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feeb0004d50 msgr2=0x7feeb00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 --2- 172.21.15.83:0/3393537086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feeb0004d50 0x7feeb00051d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feeb400c270 tx=0x7feeb401cca0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec3fff640 1 -- 172.21.15.83:0/3393537086 reap_dead start 2024-09-19T07:19:40.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 -- 172.21.15.83:0/3393537086 shutdown_connections 2024-09-19T07:19:40.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 -- 172.21.15.83:0/3393537086 >> 172.21.15.83:0/3393537086 conn(0x7feec406bc80 msgr2=0x7feec406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:40.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 -- 172.21.15.83:0/3393537086 shutdown_connections 2024-09-19T07:19:40.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 -- 172.21.15.83:0/3393537086 wait complete. 2024-09-19T07:19:40.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 Processor -- start 2024-09-19T07:19:40.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 -- start start 2024-09-19T07:19:40.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feeb0004d50 0x7feec4152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feec40c3220 0x7feec41e8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feec40c3ae0 0x7feec41e8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feec4156af0 con 0x7feec40c3220 2024-09-19T07:19:40.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feec406eec0 con 0x7feec40c3ae0 2024-09-19T07:19:40.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec9bf6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feec4153770 con 0x7feeb0004d50 2024-09-19T07:19:40.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feec40c3220 0x7feec41e8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feec40c3220 0x7feec41e8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40328/0 (socket says 172.21.15.83:40328) 2024-09-19T07:19:40.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec2ffd640 1 -- 172.21.15.83:0/2010400400 learned_addr learned my addr 172.21.15.83:0/2010400400 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:40.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec37fe640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feeb0004d50 0x7feec4152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec3fff640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feec40c3ae0 0x7feec41e8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec2ffd640 1 -- 172.21.15.83:0/2010400400 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feeb0004d50 msgr2=0x7feec4152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec2ffd640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feeb0004d50 0x7feec4152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec2ffd640 1 -- 172.21.15.83:0/2010400400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feec40c3ae0 msgr2=0x7feec41e8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:40.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec2ffd640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feec40c3ae0 0x7feec41e8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:40.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec2ffd640 1 -- 172.21.15.83:0/2010400400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec41ee990 con 0x7feec40c3220 2024-09-19T07:19:40.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec3fff640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feec40c3ae0 0x7feec41e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:40.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.887+0000 7feec2ffd640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feec40c3220 0x7feec41e8650 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feeac011590 tx=0x7feeac011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:40.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.891+0000 7feebbfff640 1 -- 172.21.15.83:0/2010400400 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac004220 con 0x7feec40c3220 2024-09-19T07:19:40.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.891+0000 7feebbfff640 1 -- 172.21.15.83:0/2010400400 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeac0043c0 con 0x7feec40c3220 2024-09-19T07:19:40.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.891+0000 7feec9bf6640 1 -- 172.21.15.83:0/2010400400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec4073310 con 0x7feec40c3220 2024-09-19T07:19:40.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.891+0000 7feebbfff640 1 -- 172.21.15.83:0/2010400400 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac015870 con 0x7feec40c3220 2024-09-19T07:19:40.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.891+0000 7feec9bf6640 1 -- 172.21.15.83:0/2010400400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec40738d0 con 0x7feec40c3220 2024-09-19T07:19:40.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.891+0000 7feec9bf6640 1 -- 172.21.15.83:0/2010400400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feec4159f10 con 0x7feec40c3220 2024-09-19T07:19:40.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.891+0000 7feebbfff640 1 -- 172.21.15.83:0/2010400400 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feeac015a10 con 0x7feec40c3220 2024-09-19T07:19:40.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.891+0000 7feebbfff640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feea40678c0 0x7feea4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:40.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.891+0000 7feebbfff640 1 -- 172.21.15.83:0/2010400400 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6521+0+0 (secure 0 0 0) 0x7feeac086230 con 0x7feec40c3220 2024-09-19T07:19:40.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.891+0000 7feec37fe640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feea40678c0 0x7feea4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:40.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.891+0000 7feec37fe640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feea40678c0 0x7feea4069d80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feebc007440 tx=0x7feebc00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:40.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:40.895+0000 7feebbfff640 1 -- 172.21.15.83:0/2010400400 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7feeac019030 con 0x7feec40c3220 2024-09-19T07:19:41.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.059+0000 7feec9bf6640 1 -- 172.21.15.83:0/2010400400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feec406e050 con 0x7feec40c3220 2024-09-19T07:19:41.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.059+0000 7feebbfff640 1 -- 172.21.15.83:0/2010400400 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+20464 (secure 0 0 0) 0x7feeac07de20 con 0x7feec40c3220 2024-09-19T07:19:41.061 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:41.061 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":24,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:39.753103+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:41.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.063+0000 7feec9bf6640 1 -- 172.21.15.83:0/2010400400 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feea40678c0 msgr2=0x7feea4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:41.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.063+0000 7feec9bf6640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feea40678c0 0x7feea4069d80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feebc007440 tx=0x7feebc00b040 comp rx=0 tx=0).stop 2024-09-19T07:19:41.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.063+0000 7feec9bf6640 1 -- 172.21.15.83:0/2010400400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feec40c3220 msgr2=0x7feec41e8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:41.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.063+0000 7feec9bf6640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feec40c3220 0x7feec41e8650 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feeac011590 tx=0x7feeac011a50 comp rx=0 tx=0).stop 2024-09-19T07:19:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.063+0000 7feec9bf6640 1 -- 172.21.15.83:0/2010400400 shutdown_connections 2024-09-19T07:19:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.063+0000 7feec9bf6640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feea40678c0 0x7feea4069d80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.063+0000 7feec9bf6640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feec40c3ae0 0x7feec41e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.063+0000 7feec9bf6640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feec40c3220 0x7feec41e8650 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.063+0000 7feec9bf6640 1 --2- 172.21.15.83:0/2010400400 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feeb0004d50 0x7feec4152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.063+0000 7feec9bf6640 1 -- 172.21.15.83:0/2010400400 >> 172.21.15.83:0/2010400400 conn(0x7feec406bc80 msgr2=0x7feec40aba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.063+0000 7feec9bf6640 1 -- 172.21.15.83:0/2010400400 shutdown_connections 2024-09-19T07:19:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.063+0000 7feec9bf6640 1 -- 172.21.15.83:0/2010400400 wait complete. 2024-09-19T07:19:41.080 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b standby_count_wanted 0 2024-09-19T07:19:41.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba7767a640 1 Processor -- start 2024-09-19T07:19:41.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba7767a640 1 -- start start 2024-09-19T07:19:41.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba7767a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba7014fd70 0x7fba700caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:41.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba7767a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fba700c9580 0x7fba7006bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba7767a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba70073360 con 0x7fba700c6000 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba7767a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fba700cb3f0 con 0x7fba700c9580 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba7767a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fba70155e30 con 0x7fba7014fd70 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba75c17640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fba700c6000 0x7fba70159230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51446/0 (socket says 172.21.15.83:51446) 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba75c17640 1 -- 172.21.15.83:0/2663527907 learned_addr learned my addr 172.21.15.83:0/2663527907 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba75416640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba7014fd70 0x7fba700caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba74c15640 1 --2- 172.21.15.83:0/2663527907 >> v2:172.21.15.184:3300/0 conn(0x7fba700c9580 0x7fba7006bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 113949960 0 0) 0x7fba70073360 con 0x7fba700c6000 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba5c003530 con 0x7fba700c6000 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3126635360 0 0) 0x7fba5c003530 con 0x7fba700c6000 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba7014fd70 msgr2=0x7fba700caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba7014fd70 0x7fba700caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:41.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 >> v2:172.21.15.184:3300/0 conn(0x7fba700c9580 msgr2=0x7fba7006bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 --2- 172.21.15.83:0/2663527907 >> v2:172.21.15.184:3300/0 conn(0x7fba700c9580 0x7fba7006bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba700ca3d0 con 0x7fba700c6000 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fba68001a00 con 0x7fba700c6000 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 >> v1:172.21.15.83:6789/0 conn(0x7fba700c6000 legacy=0x7fba70159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba7014fd70 0x7fba5c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba5c003f90 0x7fba5c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba5c004d50 0x7fba5c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fba700cb3f0 con 0x7fba5c004d50 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fba70155e30 con 0x7fba5c003f90 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fba5c00ba50 con 0x7fba7014fd70 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba75c17640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba5c004d50 0x7fba5c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba75416640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba7014fd70 0x7fba5c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba74c15640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba5c003f90 0x7fba5c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:41.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba75c17640 1 -- 172.21.15.83:0/2663527907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba7014fd70 msgr2=0x7fba5c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:41.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba75c17640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba7014fd70 0x7fba5c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:41.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba75c17640 1 -- 172.21.15.83:0/2663527907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba5c003f90 msgr2=0x7fba5c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:41.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba75c17640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba5c003f90 0x7fba5c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:41.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba75c17640 1 -- 172.21.15.83:0/2663527907 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5c003530 con 0x7fba5c004d50 2024-09-19T07:19:41.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.207+0000 7fba74c15640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba5c003f90 0x7fba5c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:19:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba75c17640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba5c004d50 0x7fba5c0051d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fba680013e0 tx=0x7fba6801d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad4d143089c2051 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6800a410 con 0x7fba5c004d50 2024-09-19T07:19:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba680ad630 con 0x7fba5c004d50 2024-09-19T07:19:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba67fff640 1 -- 172.21.15.83:0/2663527907 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba680ada20 con 0x7fba5c004d50 2024-09-19T07:19:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 -- 172.21.15.83:0/2663527907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba5c004d50 msgr2=0x7fba5c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 --2- 172.21.15.83:0/2663527907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba5c004d50 0x7fba5c0051d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fba680013e0 tx=0x7fba6801d2e0 comp rx=0 tx=0).stop 2024-09-19T07:19:41.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba75c17640 1 -- 172.21.15.83:0/2663527907 reap_dead start 2024-09-19T07:19:41.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 -- 172.21.15.83:0/2663527907 shutdown_connections 2024-09-19T07:19:41.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 -- 172.21.15.83:0/2663527907 >> 172.21.15.83:0/2663527907 conn(0x7fba700c8100 msgr2=0x7fba700c8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:41.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 -- 172.21.15.83:0/2663527907 shutdown_connections 2024-09-19T07:19:41.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 -- 172.21.15.83:0/2663527907 wait complete. 2024-09-19T07:19:41.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 Processor -- start 2024-09-19T07:19:41.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 -- start start 2024-09-19T07:19:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba70158b40 0x7fba7014b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba70157400 0x7fba70157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba70157dc0 0x7fba701eec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fba7006c1c0 con 0x7fba70157dc0 2024-09-19T07:19:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fba700ca990 con 0x7fba70158b40 2024-09-19T07:19:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba7767a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fba70153840 con 0x7fba70157400 2024-09-19T07:19:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba75c17640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba70157dc0 0x7fba701eec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:41.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba75c17640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba70157dc0 0x7fba701eec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40352/0 (socket says 172.21.15.83:40352) 2024-09-19T07:19:41.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba75c17640 1 -- 172.21.15.83:0/2217360519 learned_addr learned my addr 172.21.15.83:0/2217360519 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:41.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba75416640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba70158b40 0x7fba7014b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:41.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.211+0000 7fba74c15640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba70157400 0x7fba70157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:41.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba75c17640 1 -- 172.21.15.83:0/2217360519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba70157400 msgr2=0x7fba70157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:41.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba75c17640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba70157400 0x7fba70157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:41.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba75c17640 1 -- 172.21.15.83:0/2217360519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba70158b40 msgr2=0x7fba7014b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:41.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba75c17640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba70158b40 0x7fba7014b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:41.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba75c17640 1 -- 172.21.15.83:0/2217360519 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7014b880 con 0x7fba70157dc0 2024-09-19T07:19:41.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba75416640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba70158b40 0x7fba7014b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:19:41.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba75c17640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba70157dc0 0x7fba701eec30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fba6801cd50 tx=0x7fba5c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:41.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba65ffb640 1 -- 172.21.15.83:0/2217360519 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba680ba070 con 0x7fba70157dc0 2024-09-19T07:19:41.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba65ffb640 1 -- 172.21.15.83:0/2217360519 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba6800a3f0 con 0x7fba70157dc0 2024-09-19T07:19:41.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba7767a640 1 -- 172.21.15.83:0/2217360519 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba701ec270 con 0x7fba70157dc0 2024-09-19T07:19:41.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba65ffb640 1 -- 172.21.15.83:0/2217360519 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68001f80 con 0x7fba70157dc0 2024-09-19T07:19:41.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba7767a640 1 -- 172.21.15.83:0/2217360519 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba700b0690 con 0x7fba70157dc0 2024-09-19T07:19:41.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba7767a640 1 -- 172.21.15.83:0/2217360519 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba70155110 con 0x7fba70157dc0 2024-09-19T07:19:41.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba65ffb640 1 -- 172.21.15.83:0/2217360519 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fba680bd070 con 0x7fba70157dc0 2024-09-19T07:19:41.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba65ffb640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba540678f0 0x7fba54069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:41.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba75416640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba540678f0 0x7fba54069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:41.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba65ffb640 1 -- 172.21.15.83:0/2217360519 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6521+0+0 (secure 0 0 0) 0x7fba6811fc10 con 0x7fba70157dc0 2024-09-19T07:19:41.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.215+0000 7fba75416640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba540678f0 0x7fba54069db0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fba700c5980 tx=0x7fba6c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:41.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.219+0000 7fba65ffb640 1 -- 172.21.15.83:0/2217360519 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fba7014b880 con 0x7fba70157dc0 2024-09-19T07:19:41.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:41.423+0000 7fba7767a640 1 -- 172.21.15.83:0/2217360519 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fba70068290 con 0x7fba70157dc0 2024-09-19T07:19:42.218 INFO:tasks.ceph.mds.b.smithi184.stderr:2024-09-19T07:19:42.219+0000 7f8e197c5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:19:42.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.427+0000 7fba65ffb640 1 -- 172.21.15.83:0/2217360519 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"}]=0 v12) ==== 113+0+0 (secure 0 0 0) 0x7fba681230b0 con 0x7fba70157dc0 2024-09-19T07:19:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.431+0000 7fba7767a640 1 -- 172.21.15.83:0/2217360519 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba540678f0 msgr2=0x7fba54069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.431+0000 7fba7767a640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba540678f0 0x7fba54069db0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fba700c5980 tx=0x7fba6c008040 comp rx=0 tx=0).stop 2024-09-19T07:19:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.431+0000 7fba7767a640 1 -- 172.21.15.83:0/2217360519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba70157dc0 msgr2=0x7fba701eec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.431+0000 7fba7767a640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba70157dc0 0x7fba701eec30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fba6801cd50 tx=0x7fba5c005840 comp rx=0 tx=0).stop 2024-09-19T07:19:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.431+0000 7fba7767a640 1 -- 172.21.15.83:0/2217360519 shutdown_connections 2024-09-19T07:19:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.431+0000 7fba7767a640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba540678f0 0x7fba54069db0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.431+0000 7fba7767a640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba70157dc0 0x7fba701eec30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.431+0000 7fba7767a640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba70157400 0x7fba70157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.431+0000 7fba7767a640 1 --2- 172.21.15.83:0/2217360519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba70158b40 0x7fba7014b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.431+0000 7fba7767a640 1 -- 172.21.15.83:0/2217360519 >> 172.21.15.83:0/2217360519 conn(0x7fba700c8100 msgr2=0x7fba70155630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.431+0000 7fba7767a640 1 -- 172.21.15.83:0/2217360519 shutdown_connections 2024-09-19T07:19:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.431+0000 7fba7767a640 1 -- 172.21.15.83:0/2217360519 wait complete. 2024-09-19T07:19:42.451 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-09-19T07:19:42.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b allow_standby_replay false 2024-09-19T07:19:42.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b87f02640 1 Processor -- start 2024-09-19T07:19:42.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b87f02640 1 -- start start 2024-09-19T07:19:42.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b87f02640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5b800c7520 0x7f5b800ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:42.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b87f02640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5b8006e1e0 0x7f5b8006e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:42.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b87f02640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b80156510 con 0x7f5b800c09d0 2024-09-19T07:19:42.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b87f02640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5b801567f0 con 0x7f5b8006e1e0 2024-09-19T07:19:42.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b87f02640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5b80156ad0 con 0x7f5b800c7520 2024-09-19T07:19:42.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b8649f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5b8006e1e0 0x7f5b8006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:42.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b8649f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5b8006e1e0 0x7f5b8006e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40364/0 (socket says 172.21.15.83:40364) 2024-09-19T07:19:42.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b8649f640 1 -- 172.21.15.83:0/1552358975 learned_addr learned my addr 172.21.15.83:0/1552358975 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:42.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b8649f640 1 -- 172.21.15.83:0/1552358975 >> v2:172.21.15.184:3300/0 conn(0x7f5b800c7520 msgr2=0x7f5b800ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b85c9e640 1 --2- 172.21.15.83:0/1552358975 >> v2:172.21.15.184:3300/0 conn(0x7f5b800c7520 0x7f5b800ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b8649f640 1 --2- 172.21.15.83:0/1552358975 >> v2:172.21.15.184:3300/0 conn(0x7f5b800c7520 0x7f5b800ab8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b8649f640 1 -- 172.21.15.83:0/1552358975 >> v1:172.21.15.83:6789/0 conn(0x7f5b800c09d0 legacy=0x7f5b80155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b8649f640 1 -- 172.21.15.83:0/1552358975 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8006d9e0 con 0x7f5b8006e1e0 2024-09-19T07:19:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b84c9c640 1 -- 172.21.15.83:0/1552358975 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3343514341 0 0) 0x7f5b80156510 con 0x7f5b800c09d0 2024-09-19T07:19:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.579+0000 7f5b85c9e640 1 --2- 172.21.15.83:0/1552358975 >> v2:172.21.15.184:3300/0 conn(0x7f5b800c7520 0x7f5b800ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:42.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b8649f640 1 --2- 172.21.15.83:0/1552358975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b8006e1e0 0x7f5b8006e650 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5b8006cba0 tx=0x7f5b7801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd56909a07087083 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:42.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b84c9c640 1 -- 172.21.15.83:0/1552358975 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b780ab430 con 0x7f5b8006e1e0 2024-09-19T07:19:42.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b84c9c640 1 -- 172.21.15.83:0/1552358975 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b780b0070 con 0x7f5b8006e1e0 2024-09-19T07:19:42.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b84c9c640 1 -- 172.21.15.83:0/1552358975 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b780ab7f0 con 0x7f5b8006e1e0 2024-09-19T07:19:42.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 -- 172.21.15.83:0/1552358975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b8006e1e0 msgr2=0x7f5b8006e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:42.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 --2- 172.21.15.83:0/1552358975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b8006e1e0 0x7f5b8006e650 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5b8006cba0 tx=0x7f5b7801c520 comp rx=0 tx=0).stop 2024-09-19T07:19:42.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 -- 172.21.15.83:0/1552358975 shutdown_connections 2024-09-19T07:19:42.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 --2- 172.21.15.83:0/1552358975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b8006e1e0 0x7f5b8006e650 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:42.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 --2- 172.21.15.83:0/1552358975 >> v2:172.21.15.184:3300/0 conn(0x7f5b800c7520 0x7f5b800ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:42.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 -- 172.21.15.83:0/1552358975 >> 172.21.15.83:0/1552358975 conn(0x7f5b8006bc80 msgr2=0x7f5b8006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:42.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 -- 172.21.15.83:0/1552358975 shutdown_connections 2024-09-19T07:19:42.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 -- 172.21.15.83:0/1552358975 wait complete. 2024-09-19T07:19:42.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 Processor -- start 2024-09-19T07:19:42.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 -- start start 2024-09-19T07:19:42.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b8006e1e0 0x7f5b801f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b800c09d0 0x7f5b800c4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b800c7520 0x7f5b801ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5b801625e0 con 0x7f5b8006e1e0 2024-09-19T07:19:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5b801538a0 con 0x7f5b800c09d0 2024-09-19T07:19:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b87f02640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5b80162760 con 0x7f5b800c7520 2024-09-19T07:19:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.583+0000 7f5b85c9e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b8006e1e0 0x7f5b801f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b85c9e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b8006e1e0 0x7f5b801f2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40378/0 (socket says 172.21.15.83:40378) 2024-09-19T07:19:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b85c9e640 1 -- 172.21.15.83:0/4056064079 learned_addr learned my addr 172.21.15.83:0/4056064079 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b8549d640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b800c09d0 0x7f5b800c4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b85c9e640 1 -- 172.21.15.83:0/4056064079 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b800c7520 msgr2=0x7f5b801ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b85c9e640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b800c7520 0x7f5b801ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b85c9e640 1 -- 172.21.15.83:0/4056064079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b800c09d0 msgr2=0x7f5b800c4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b85c9e640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b800c09d0 0x7f5b800c4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b85c9e640 1 -- 172.21.15.83:0/4056064079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b80157a70 con 0x7f5b8006e1e0 2024-09-19T07:19:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b8549d640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b800c09d0 0x7f5b800c4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b85c9e640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b8006e1e0 0x7f5b801f2100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5b7000ce00 tx=0x7f5b7000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b7e7fc640 1 -- 172.21.15.83:0/4056064079 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7000f890 con 0x7f5b8006e1e0 2024-09-19T07:19:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b7e7fc640 1 -- 172.21.15.83:0/4056064079 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b7000fa30 con 0x7f5b8006e1e0 2024-09-19T07:19:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b7e7fc640 1 -- 172.21.15.83:0/4056064079 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7000fe40 con 0x7f5b8006e1e0 2024-09-19T07:19:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b87f02640 1 -- 172.21.15.83:0/4056064079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b80157d50 con 0x7f5b8006e1e0 2024-09-19T07:19:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b87f02640 1 -- 172.21.15.83:0/4056064079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b801f1900 con 0x7f5b8006e1e0 2024-09-19T07:19:42.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b87f02640 1 -- 172.21.15.83:0/4056064079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b8006e650 con 0x7f5b8006e1e0 2024-09-19T07:19:42.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b7e7fc640 1 -- 172.21.15.83:0/4056064079 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5b70018400 con 0x7f5b8006e1e0 2024-09-19T07:19:42.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b7e7fc640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b600678c0 0x7f5b60069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:42.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b7e7fc640 1 -- 172.21.15.83:0/4056064079 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f5b70086db0 con 0x7f5b8006e1e0 2024-09-19T07:19:42.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b8549d640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b600678c0 0x7f5b60069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:42.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.587+0000 7f5b8549d640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b600678c0 0x7f5b60069d80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5b8015ff40 tx=0x7f5b68009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.591+0000 7f5b7e7fc640 1 -- 172.21.15.83:0/4056064079 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5b70014030 con 0x7f5b8006e1e0 2024-09-19T07:19:42.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:42.791+0000 7f5b87f02640 1 -- 172.21.15.83:0/4056064079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f5b8014f050 con 0x7f5b8006e1e0 2024-09-19T07:19:43.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.435+0000 7f5b7e7fc640 1 -- 172.21.15.83:0/4056064079 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"}]=0 v13) ==== 117+0+0 (secure 0 0 0) 0x7f5b7000ae20 con 0x7f5b8006e1e0 2024-09-19T07:19:43.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.439+0000 7f5b87f02640 1 -- 172.21.15.83:0/4056064079 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b600678c0 msgr2=0x7f5b60069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:43.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.439+0000 7f5b87f02640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b600678c0 0x7f5b60069d80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5b8015ff40 tx=0x7f5b68009040 comp rx=0 tx=0).stop 2024-09-19T07:19:43.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.439+0000 7f5b87f02640 1 -- 172.21.15.83:0/4056064079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b8006e1e0 msgr2=0x7f5b801f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:43.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.439+0000 7f5b87f02640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b8006e1e0 0x7f5b801f2100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5b7000ce00 tx=0x7f5b7000a650 comp rx=0 tx=0).stop 2024-09-19T07:19:43.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.439+0000 7f5b87f02640 1 -- 172.21.15.83:0/4056064079 shutdown_connections 2024-09-19T07:19:43.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.439+0000 7f5b87f02640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b600678c0 0x7f5b60069d80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:43.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.439+0000 7f5b87f02640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b800c7520 0x7f5b801ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:43.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.439+0000 7f5b87f02640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b800c09d0 0x7f5b800c4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:43.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.439+0000 7f5b87f02640 1 --2- 172.21.15.83:0/4056064079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b8006e1e0 0x7f5b801f2100 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:43.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.439+0000 7f5b87f02640 1 -- 172.21.15.83:0/4056064079 >> 172.21.15.83:0/4056064079 conn(0x7f5b8006bc80 msgr2=0x7f5b8006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:43.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.439+0000 7f5b87f02640 1 -- 172.21.15.83:0/4056064079 shutdown_connections 2024-09-19T07:19:43.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.439+0000 7f5b87f02640 1 -- 172.21.15.83:0/4056064079 wait complete. 2024-09-19T07:19:43.460 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b session_timeout 300 2024-09-19T07:19:43.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.591+0000 7fcd17313640 1 Processor -- start 2024-09-19T07:19:43.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.591+0000 7fcd17313640 1 -- start start 2024-09-19T07:19:43.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.591+0000 7fcd17313640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcd10151130 0x7fcd100c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:43.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.591+0000 7fcd17313640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd100c3590 0x7fcd10154120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:43.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.591+0000 7fcd17313640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd100c2330 con 0x7fcd100c2af0 2024-09-19T07:19:43.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.591+0000 7fcd17313640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fcd100715d0 con 0x7fcd10151130 2024-09-19T07:19:43.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.591+0000 7fcd17313640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcd100c1d70 con 0x7fcd100c3590 2024-09-19T07:19:43.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.591+0000 7fcd150af640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcd10151130 0x7fcd100c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:43.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.591+0000 7fcd158b0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd100c3590 0x7fcd10154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:43.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.591+0000 7fcd150af640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcd10151130 0x7fcd100c25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:60950/0 (socket says 172.21.15.83:60950) 2024-09-19T07:19:43.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.591+0000 7fcd150af640 1 -- 172.21.15.83:0/3748186900 learned_addr learned my addr 172.21.15.83:0/3748186900 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:43.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.591+0000 7fcd148ae640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fcd100c2af0 0x7fcd10153a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:42416/0 (socket says 172.21.15.83:42416) 2024-09-19T07:19:43.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd158b0640 1 -- 172.21.15.83:0/3748186900 >> v2:172.21.15.184:3300/0 conn(0x7fcd10151130 msgr2=0x7fcd100c25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:43.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd158b0640 1 --2- 172.21.15.83:0/3748186900 >> v2:172.21.15.184:3300/0 conn(0x7fcd10151130 0x7fcd100c25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:43.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd158b0640 1 -- 172.21.15.83:0/3748186900 >> v1:172.21.15.184:6789/0 conn(0x7fcd100c2af0 legacy=0x7fcd10153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:43.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd158b0640 1 -- 172.21.15.83:0/3748186900 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd100c2050 con 0x7fcd100c3590 2024-09-19T07:19:43.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd150af640 1 --2- 172.21.15.83:0/3748186900 >> v2:172.21.15.184:3300/0 conn(0x7fcd10151130 0x7fcd100c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:43.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd158b0640 1 --2- 172.21.15.83:0/3748186900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd100c3590 0x7fcd10154120 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcd080097c0 tx=0x7fcd0801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83bc57c3f55cfd54 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:43.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd03fff640 1 -- 172.21.15.83:0/3748186900 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd080b0070 con 0x7fcd100c3590 2024-09-19T07:19:43.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd03fff640 1 -- 172.21.15.83:0/3748186900 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd080ab600 con 0x7fcd100c3590 2024-09-19T07:19:43.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd03fff640 1 -- 172.21.15.83:0/3748186900 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd080ab9d0 con 0x7fcd100c3590 2024-09-19T07:19:43.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd17313640 1 -- 172.21.15.83:0/3748186900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd100c3590 msgr2=0x7fcd10154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:43.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd17313640 1 --2- 172.21.15.83:0/3748186900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd100c3590 0x7fcd10154120 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcd080097c0 tx=0x7fcd0801c410 comp rx=0 tx=0).stop 2024-09-19T07:19:43.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd17313640 1 -- 172.21.15.83:0/3748186900 shutdown_connections 2024-09-19T07:19:43.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd17313640 1 --2- 172.21.15.83:0/3748186900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd100c3590 0x7fcd10154120 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:43.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd17313640 1 --2- 172.21.15.83:0/3748186900 >> v2:172.21.15.184:3300/0 conn(0x7fcd10151130 0x7fcd100c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:43.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd17313640 1 -- 172.21.15.83:0/3748186900 >> 172.21.15.83:0/3748186900 conn(0x7fcd1006bc80 msgr2=0x7fcd1006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:43.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd17313640 1 -- 172.21.15.83:0/3748186900 shutdown_connections 2024-09-19T07:19:43.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.595+0000 7fcd17313640 1 -- 172.21.15.83:0/3748186900 wait complete. 2024-09-19T07:19:43.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd17313640 1 Processor -- start 2024-09-19T07:19:43.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd17313640 1 -- start start 2024-09-19T07:19:43.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd17313640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd100c2af0 0x7fcd1014d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:43.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd17313640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd100c3590 0x7fcd100bcb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:43.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd17313640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd10151130 0x7fcd100bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:43.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd17313640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcd10151690 con 0x7fcd100c3590 2024-09-19T07:19:43.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd17313640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcd10151510 con 0x7fcd10151130 2024-09-19T07:19:43.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd17313640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcd10160b90 con 0x7fcd100c2af0 2024-09-19T07:19:43.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd148ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd100c3590 0x7fcd100bcb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:43.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd148ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd100c3590 0x7fcd100bcb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40386/0 (socket says 172.21.15.83:40386) 2024-09-19T07:19:43.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd148ae640 1 -- 172.21.15.83:0/4101788785 learned_addr learned my addr 172.21.15.83:0/4101788785 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:43.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd150af640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd100c2af0 0x7fcd1014d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:43.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd148ae640 1 -- 172.21.15.83:0/4101788785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd100c2af0 msgr2=0x7fcd1014d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:43.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd158b0640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd10151130 0x7fcd100bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:43.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd148ae640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd100c2af0 0x7fcd1014d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:43.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd148ae640 1 -- 172.21.15.83:0/4101788785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd10151130 msgr2=0x7fcd100bd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:43.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd148ae640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd10151130 0x7fcd100bd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:43.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd148ae640 1 -- 172.21.15.83:0/4101788785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1014d070 con 0x7fcd100c3590 2024-09-19T07:19:43.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd150af640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd100c2af0 0x7fcd1014d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:43.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd158b0640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd10151130 0x7fcd100bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:43.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd148ae640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd100c3590 0x7fcd100bcb10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00c870 tx=0x7fcd0c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:43.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd01ffb640 1 -- 172.21.15.83:0/4101788785 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c017070 con 0x7fcd100c3590 2024-09-19T07:19:43.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd01ffb640 1 -- 172.21.15.83:0/4101788785 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0c00a640 con 0x7fcd100c3590 2024-09-19T07:19:43.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd17313640 1 -- 172.21.15.83:0/4101788785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd100c0e30 con 0x7fcd100c3590 2024-09-19T07:19:43.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd01ffb640 1 -- 172.21.15.83:0/4101788785 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c00a9f0 con 0x7fcd100c3590 2024-09-19T07:19:43.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd17313640 1 -- 172.21.15.83:0/4101788785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd10157940 con 0x7fcd100c3590 2024-09-19T07:19:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd17313640 1 -- 172.21.15.83:0/4101788785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd10153640 con 0x7fcd100c3590 2024-09-19T07:19:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.599+0000 7fcd01ffb640 1 -- 172.21.15.83:0/4101788785 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcd0c01a070 con 0x7fcd100c3590 2024-09-19T07:19:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.603+0000 7fcd01ffb640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fccf40678c0 0x7fccf4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.603+0000 7fcd01ffb640 1 -- 172.21.15.83:0/4101788785 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fcd0c0854d0 con 0x7fcd100c3590 2024-09-19T07:19:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.603+0000 7fcd150af640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fccf40678c0 0x7fccf4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.603+0000 7fcd150af640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fccf40678c0 0x7fccf4069d80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcd04009a30 tx=0x7fcd04008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:43.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.607+0000 7fcd01ffb640 1 -- 172.21.15.83:0/4101788785 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcd0c055720 con 0x7fcd100c3590 2024-09-19T07:19:43.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:43.795+0000 7fcd17313640 1 -- 172.21.15.83:0/4101788785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"} v 0) -- 0x7fcd1014e1d0 con 0x7fcd100c3590 2024-09-19T07:19:44.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.443+0000 7fcd01ffb640 1 -- 172.21.15.83:0/4101788785 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"}]=0 v14) ==== 110+0+0 (secure 0 0 0) 0x7fcd0c059a20 con 0x7fcd100c3590 2024-09-19T07:19:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.447+0000 7fcd17313640 1 -- 172.21.15.83:0/4101788785 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fccf40678c0 msgr2=0x7fccf4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.447+0000 7fcd17313640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fccf40678c0 0x7fccf4069d80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcd04009a30 tx=0x7fcd04008040 comp rx=0 tx=0).stop 2024-09-19T07:19:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.447+0000 7fcd17313640 1 -- 172.21.15.83:0/4101788785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd100c3590 msgr2=0x7fcd100bcb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.447+0000 7fcd17313640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd100c3590 0x7fcd100bcb10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00c870 tx=0x7fcd0c00cd30 comp rx=0 tx=0).stop 2024-09-19T07:19:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.447+0000 7fcd17313640 1 -- 172.21.15.83:0/4101788785 shutdown_connections 2024-09-19T07:19:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.447+0000 7fcd17313640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fccf40678c0 0x7fccf4069d80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.447+0000 7fcd17313640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd10151130 0x7fcd100bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.447+0000 7fcd17313640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd100c3590 0x7fcd100bcb10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.447+0000 7fcd17313640 1 --2- 172.21.15.83:0/4101788785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd100c2af0 0x7fcd1014d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.447+0000 7fcd17313640 1 -- 172.21.15.83:0/4101788785 >> 172.21.15.83:0/4101788785 conn(0x7fcd1006bc80 msgr2=0x7fcd1006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:44.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.447+0000 7fcd17313640 1 -- 172.21.15.83:0/4101788785 shutdown_connections 2024-09-19T07:19:44.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.447+0000 7fcd17313640 1 -- 172.21.15.83:0/4101788785 wait complete. 2024-09-19T07:19:44.470 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'b' 2024-09-19T07:19:44.470 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:19:44.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f82688640 1 Processor -- start 2024-09-19T07:19:44.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f82688640 1 -- start start 2024-09-19T07:19:44.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f82688640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f7c071100 0x7f7f7c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f82688640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7f7c0c28e0 0x7f7f7c150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f82688640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f7c0a9e60 con 0x7f7f7c151480 2024-09-19T07:19:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f82688640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7f7c071b50 con 0x7f7f7c071100 2024-09-19T07:19:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f82688640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f7f7c0c1ff0 con 0x7f7f7c0c28e0 2024-09-19T07:19:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f7bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f7c071100 0x7f7f7c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f7bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f7c071100 0x7f7f7c1545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40388/0 (socket says 172.21.15.83:40388) 2024-09-19T07:19:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f7bfff640 1 -- 172.21.15.83:0/223266891 learned_addr learned my addr 172.21.15.83:0/223266891 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f7b7fe640 1 --2- 172.21.15.83:0/223266891 >> v2:172.21.15.184:3300/0 conn(0x7f7f7c0c28e0 0x7f7f7c150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f7bfff640 1 -- 172.21.15.83:0/223266891 >> v2:172.21.15.184:3300/0 conn(0x7f7f7c0c28e0 msgr2=0x7f7f7c150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f7bfff640 1 --2- 172.21.15.83:0/223266891 >> v2:172.21.15.184:3300/0 conn(0x7f7f7c0c28e0 0x7f7f7c150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f7bfff640 1 -- 172.21.15.83:0/223266891 >> v1:172.21.15.83:6789/0 conn(0x7f7f7c151480 legacy=0x7f7f7c06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f7affd640 1 -- 172.21.15.83:0/223266891 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1413503863 0 0) 0x7f7f7c0a9e60 con 0x7f7f7c151480 2024-09-19T07:19:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.595+0000 7f7f7bfff640 1 -- 172.21.15.83:0/223266891 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c0aa140 con 0x7f7f7c071100 2024-09-19T07:19:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7bfff640 1 --2- 172.21.15.83:0/223266891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7c071100 0x7f7f7c1545e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0c2890 tx=0x7f7f7001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9cbdb95064e8a6f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7affd640 1 -- 172.21.15.83:0/223266891 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f700b2070 con 0x7f7f7c071100 2024-09-19T07:19:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7affd640 1 -- 172.21.15.83:0/223266891 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f700ad680 con 0x7f7f7c071100 2024-09-19T07:19:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7affd640 1 -- 172.21.15.83:0/223266891 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f700ada50 con 0x7f7f7c071100 2024-09-19T07:19:44.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 -- 172.21.15.83:0/223266891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7c071100 msgr2=0x7f7f7c1545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:44.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 --2- 172.21.15.83:0/223266891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7c071100 0x7f7f7c1545e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0c2890 tx=0x7f7f7001c490 comp rx=0 tx=0).stop 2024-09-19T07:19:44.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 -- 172.21.15.83:0/223266891 shutdown_connections 2024-09-19T07:19:44.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 --2- 172.21.15.83:0/223266891 >> v2:172.21.15.184:3300/0 conn(0x7f7f7c0c28e0 0x7f7f7c150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 --2- 172.21.15.83:0/223266891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7c071100 0x7f7f7c1545e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 -- 172.21.15.83:0/223266891 >> 172.21.15.83:0/223266891 conn(0x7f7f7c06bc80 msgr2=0x7f7f7c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:44.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 -- 172.21.15.83:0/223266891 shutdown_connections 2024-09-19T07:19:44.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 -- 172.21.15.83:0/223266891 wait complete. 2024-09-19T07:19:44.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 Processor -- start 2024-09-19T07:19:44.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 -- start start 2024-09-19T07:19:44.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7c071100 0x7f7f7c14eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f7c0c28e0 0x7f7f7c150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f7c151480 0x7f7f7c1505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c071660 con 0x7f7f7c071100 2024-09-19T07:19:44.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c0714e0 con 0x7f7f7c0c28e0 2024-09-19T07:19:44.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7f7c160c60 con 0x7f7f7c151480 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7c071100 0x7f7f7c14eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7c071100 0x7f7f7c14eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40404/0 (socket says 172.21.15.83:40404) 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7bfff640 1 -- 172.21.15.83:0/2859399724 learned_addr learned my addr 172.21.15.83:0/2859399724 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f80c25640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f7c151480 0x7f7f7c1505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7b7fe640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f7c0c28e0 0x7f7f7c150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7bfff640 1 -- 172.21.15.83:0/2859399724 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f7c151480 msgr2=0x7f7f7c1505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7bfff640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f7c151480 0x7f7f7c1505d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7bfff640 1 -- 172.21.15.83:0/2859399724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f7c0c28e0 msgr2=0x7f7f7c150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7bfff640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f7c0c28e0 0x7f7f7c150090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7bfff640 1 -- 172.21.15.83:0/2859399724 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c151b70 con 0x7f7f7c071100 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7b7fe640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f7c0c28e0 0x7f7f7c150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f80c25640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f7c151480 0x7f7f7c1505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f7bfff640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7c071100 0x7f7f7c14eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7f7001a0c0 tx=0x7f7f7001acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:44.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f78ff9640 1 -- 172.21.15.83:0/2859399724 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f700bc020 con 0x7f7f7c071100 2024-09-19T07:19:44.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f78ff9640 1 -- 172.21.15.83:0/2859399724 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f700adc30 con 0x7f7f7c071100 2024-09-19T07:19:44.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 -- 172.21.15.83:0/2859399724 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f7c152170 con 0x7f7f7c071100 2024-09-19T07:19:44.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f78ff9640 1 -- 172.21.15.83:0/2859399724 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f700b2040 con 0x7f7f7c071100 2024-09-19T07:19:44.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.599+0000 7f7f82688640 1 -- 172.21.15.83:0/2859399724 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f7c146dc0 con 0x7f7f7c071100 2024-09-19T07:19:44.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.603+0000 7f7f82688640 1 -- 172.21.15.83:0/2859399724 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f7c1545e0 con 0x7f7f7c071100 2024-09-19T07:19:44.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.603+0000 7f7f78ff9640 1 -- 172.21.15.83:0/2859399724 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7f700bf020 con 0x7f7f7c071100 2024-09-19T07:19:44.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.603+0000 7f7f78ff9640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f680678c0 0x7f7f68069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.603+0000 7f7f78ff9640 1 -- 172.21.15.83:0/2859399724 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f7f70120110 con 0x7f7f7c071100 2024-09-19T07:19:44.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.603+0000 7f7f7b7fe640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f680678c0 0x7f7f68069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.603+0000 7f7f7b7fe640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f680678c0 0x7f7f68069d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0ac8f0 tx=0x7f7f6c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:44.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.607+0000 7f7f78ff9640 1 -- 172.21.15.83:0/2859399724 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7f700c6030 con 0x7f7f7c071100 2024-09-19T07:19:44.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 -- 172.21.15.83:0/2859399724 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f7c0c4f30 con 0x7f7f7c071100 2024-09-19T07:19:44.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f78ff9640 1 -- 172.21.15.83:0/2859399724 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f7f70016110 con 0x7f7f7c071100 2024-09-19T07:19:44.791 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:44.791 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:19:44.791 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:19:44.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 -- 172.21.15.83:0/2859399724 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f680678c0 msgr2=0x7f7f68069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:44.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f680678c0 0x7f7f68069d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0ac8f0 tx=0x7f7f6c00b040 comp rx=0 tx=0).stop 2024-09-19T07:19:44.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 -- 172.21.15.83:0/2859399724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7c071100 msgr2=0x7f7f7c14eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:44.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7c071100 0x7f7f7c14eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7f7001a0c0 tx=0x7f7f7001acd0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 -- 172.21.15.83:0/2859399724 shutdown_connections 2024-09-19T07:19:44.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f680678c0 0x7f7f68069d80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f7c151480 0x7f7f7c1505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f7c0c28e0 0x7f7f7c150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 --2- 172.21.15.83:0/2859399724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7c071100 0x7f7f7c14eb30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 -- 172.21.15.83:0/2859399724 >> 172.21.15.83:0/2859399724 conn(0x7f7f7c06bc80 msgr2=0x7f7f7c073850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:44.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 -- 172.21.15.83:0/2859399724 shutdown_connections 2024-09-19T07:19:44.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.791+0000 7f7f82688640 1 -- 172.21.15.83:0/2859399724 wait complete. 2024-09-19T07:19:44.808 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.915+0000 7f3e882ba640 1 Processor -- start 2024-09-19T07:19:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.915+0000 7f3e882ba640 1 -- start start 2024-09-19T07:19:44.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e882ba640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3e80152b20 0x7f3e800c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e882ba640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e80150670 con 0x7f3e800c45c0 2024-09-19T07:19:44.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e882ba640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e800c2880 con 0x7f3e80150290 2024-09-19T07:19:44.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e882ba640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f3e800716c0 con 0x7f3e80152b20 2024-09-19T07:19:44.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e86857640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3e800c45c0 0x7f3e80153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51472/0 (socket says 172.21.15.83:51472) 2024-09-19T07:19:44.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e86857640 1 -- 172.21.15.83:0/3671242233 learned_addr learned my addr 172.21.15.83:0/3671242233 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:44.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85855640 1 --2- 172.21.15.83:0/3671242233 >> v2:172.21.15.184:3300/0 conn(0x7f3e80152b20 0x7f3e800c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4209097054 0 0) 0x7f3e80150670 con 0x7f3e800c45c0 2024-09-19T07:19:44.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e6c003530 con 0x7f3e800c45c0 2024-09-19T07:19:44.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3906127704 0 0) 0x7f3e6c003530 con 0x7f3e800c45c0 2024-09-19T07:19:44.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 >> v2:172.21.15.184:3300/0 conn(0x7f3e80152b20 msgr2=0x7f3e800c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:44.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 --2- 172.21.15.83:0/3671242233 >> v2:172.21.15.184:3300/0 conn(0x7f3e80152b20 0x7f3e800c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 >> v1:172.21.15.184:6789/0 conn(0x7f3e80150290 legacy=0x7f3e800c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:44.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e800c2b60 con 0x7f3e800c45c0 2024-09-19T07:19:44.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f3e78001a00 con 0x7f3e800c45c0 2024-09-19T07:19:44.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 >> v1:172.21.15.83:6789/0 conn(0x7f3e800c45c0 legacy=0x7f3e80153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:44.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 --2- 172.21.15.83:0/3671242233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 0x7f3e6c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 --2- 172.21.15.83:0/3671242233 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e6c004810 0x7f3e6c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 --2- 172.21.15.83:0/3671242233 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e6c0051d0 0x7f3e6c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3e6c00c040 con 0x7f3e6c0039c0 2024-09-19T07:19:44.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3e6c00c1f0 con 0x7f3e6c0051d0 2024-09-19T07:19:44.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3e6c00c3f0 con 0x7f3e6c004810 2024-09-19T07:19:44.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e86056640 1 --2- 172.21.15.83:0/3671242233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 0x7f3e6c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e86857640 1 --2- 172.21.15.83:0/3671242233 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e6c0051d0 0x7f3e6c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e85855640 1 --2- 172.21.15.83:0/3671242233 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e6c004810 0x7f3e6c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e86056640 1 -- 172.21.15.83:0/3671242233 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e6c004810 msgr2=0x7f3e6c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:44.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e86056640 1 --2- 172.21.15.83:0/3671242233 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e6c004810 0x7f3e6c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e86056640 1 -- 172.21.15.83:0/3671242233 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e6c0051d0 msgr2=0x7f3e6c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:44.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e86056640 1 --2- 172.21.15.83:0/3671242233 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e6c0051d0 0x7f3e6c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e86056640 1 -- 172.21.15.83:0/3671242233 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e6c003530 con 0x7f3e6c0039c0 2024-09-19T07:19:44.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e86857640 1 -- 172.21.15.83:0/3671242233 reap_dead start 2024-09-19T07:19:44.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.919+0000 7f3e86056640 1 --2- 172.21.15.83:0/3671242233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 0x7f3e6c003de0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3e70001780 tx=0x7f3e7001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df278493ba47842c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:44.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e700013a0 con 0x7f3e6c0039c0 2024-09-19T07:19:44.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e700052f0 con 0x7f3e6c0039c0 2024-09-19T07:19:44.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e85054640 1 -- 172.21.15.83:0/3671242233 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e70003840 con 0x7f3e6c0039c0 2024-09-19T07:19:44.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 -- 172.21.15.83:0/3671242233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 msgr2=0x7f3e6c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:44.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 --2- 172.21.15.83:0/3671242233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 0x7f3e6c003de0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3e70001780 tx=0x7f3e7001c400 comp rx=0 tx=0).stop 2024-09-19T07:19:44.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 -- 172.21.15.83:0/3671242233 shutdown_connections 2024-09-19T07:19:44.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 --2- 172.21.15.83:0/3671242233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 0x7f3e6c003de0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 -- 172.21.15.83:0/3671242233 >> 172.21.15.83:0/3671242233 conn(0x7f3e8006bc80 msgr2=0x7f3e8006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 -- 172.21.15.83:0/3671242233 shutdown_connections 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 -- 172.21.15.83:0/3671242233 wait complete. 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 Processor -- start 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 -- start start 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 0x7f3e801debb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e801eb9f0 0x7f3e801e1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e80149350 0x7f3e801497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3e800c3490 con 0x7f3e6c0039c0 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3e800c2610 con 0x7f3e80149350 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3e801546c0 con 0x7f3e801eb9f0 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e86056640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 0x7f3e801debb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e86056640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 0x7f3e801debb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40416/0 (socket says 172.21.15.83:40416) 2024-09-19T07:19:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e86056640 1 -- 172.21.15.83:0/2646329274 learned_addr learned my addr 172.21.15.83:0/2646329274 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:44.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e86056640 1 -- 172.21.15.83:0/2646329274 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e801eb9f0 msgr2=0x7f3e801e1620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:19:44.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e86857640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e80149350 0x7f3e801497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e86056640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e801eb9f0 0x7f3e801e1620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e86056640 1 -- 172.21.15.83:0/2646329274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e80149350 msgr2=0x7f3e801497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:44.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e86056640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e80149350 0x7f3e801497d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:44.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e86056640 1 -- 172.21.15.83:0/2646329274 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e801df0f0 con 0x7f3e6c0039c0 2024-09-19T07:19:44.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e86056640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 0x7f3e801debb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3e7001c230 tx=0x7f3e6c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:44.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e76ffd640 1 -- 172.21.15.83:0/2646329274 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e70007a40 con 0x7f3e6c0039c0 2024-09-19T07:19:44.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e76ffd640 1 -- 172.21.15.83:0/2646329274 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e700ba070 con 0x7f3e6c0039c0 2024-09-19T07:19:44.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e76ffd640 1 -- 172.21.15.83:0/2646329274 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e700054a0 con 0x7f3e6c0039c0 2024-09-19T07:19:44.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 -- 172.21.15.83:0/2646329274 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e801df920 con 0x7f3e6c0039c0 2024-09-19T07:19:44.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.923+0000 7f3e882ba640 1 -- 172.21.15.83:0/2646329274 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e801de3b0 con 0x7f3e6c0039c0 2024-09-19T07:19:44.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.927+0000 7f3e882ba640 1 -- 172.21.15.83:0/2646329274 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e80155fb0 con 0x7f3e6c0039c0 2024-09-19T07:19:44.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.927+0000 7f3e76ffd640 1 -- 172.21.15.83:0/2646329274 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3e70007030 con 0x7f3e6c0039c0 2024-09-19T07:19:44.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.927+0000 7f3e76ffd640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e640678c0 0x7f3e64069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:44.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.927+0000 7f3e76ffd640 1 -- 172.21.15.83:0/2646329274 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f3e7011fff0 con 0x7f3e6c0039c0 2024-09-19T07:19:44.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.927+0000 7f3e85855640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e640678c0 0x7f3e64069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:44.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.927+0000 7f3e85855640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e640678c0 0x7f3e64069d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e8014abb0 tx=0x7f3e7c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:44.931+0000 7f3e76ffd640 1 -- 172.21.15.83:0/2646329274 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3e70016110 con 0x7f3e6c0039c0 2024-09-19T07:19:45.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.079+0000 7f3e882ba640 1 -- 172.21.15.83:0/2646329274 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e80153b70 con 0x7f3e6c0039c0 2024-09-19T07:19:45.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.083+0000 7f3e76ffd640 1 -- 172.21.15.83:0/2646329274 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+20451 (secure 0 0 0) 0x7f3e700f0800 con 0x7f3e6c0039c0 2024-09-19T07:19:45.084 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:45.084 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:41.334020+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:45.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.083+0000 7f3e882ba640 1 -- 172.21.15.83:0/2646329274 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e640678c0 msgr2=0x7f3e64069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.083+0000 7f3e882ba640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e640678c0 0x7f3e64069d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e8014abb0 tx=0x7f3e7c008040 comp rx=0 tx=0).stop 2024-09-19T07:19:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.083+0000 7f3e882ba640 1 -- 172.21.15.83:0/2646329274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 msgr2=0x7f3e801debb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.083+0000 7f3e882ba640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 0x7f3e801debb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3e7001c230 tx=0x7f3e6c005840 comp rx=0 tx=0).stop 2024-09-19T07:19:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.087+0000 7f3e882ba640 1 -- 172.21.15.83:0/2646329274 shutdown_connections 2024-09-19T07:19:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.087+0000 7f3e882ba640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e640678c0 0x7f3e64069d80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.087+0000 7f3e882ba640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e80149350 0x7f3e801497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.087+0000 7f3e882ba640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e801eb9f0 0x7f3e801e1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.087+0000 7f3e882ba640 1 --2- 172.21.15.83:0/2646329274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e6c0039c0 0x7f3e801debb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.087+0000 7f3e882ba640 1 -- 172.21.15.83:0/2646329274 >> 172.21.15.83:0/2646329274 conn(0x7f3e8006bc80 msgr2=0x7f3e8006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.087+0000 7f3e882ba640 1 -- 172.21.15.83:0/2646329274 shutdown_connections 2024-09-19T07:19:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.087+0000 7f3e882ba640 1 -- 172.21.15.83:0/2646329274 wait complete. 2024-09-19T07:19:45.101 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-19T07:19:45.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:19:45.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab35457640 1 Processor -- start 2024-09-19T07:19:45.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab35457640 1 -- start start 2024-09-19T07:19:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab35457640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab300c6120 0x7fab300c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab35457640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab30155d70 0x7fab301575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab35457640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab30156200 con 0x7fab301592f0 2024-09-19T07:19:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab35457640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fab300c9930 con 0x7fab300c6120 2024-09-19T07:19:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab35457640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fab301564e0 con 0x7fab30155d70 2024-09-19T07:19:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab2effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab300c6120 0x7fab300c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab2f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fab301592f0 0x7fab30157b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51474/0 (socket says 172.21.15.83:51474) 2024-09-19T07:19:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab2f7fe640 1 -- 172.21.15.83:0/3907885180 learned_addr learned my addr 172.21.15.83:0/3907885180 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab2effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab300c6120 0x7fab300c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40432/0 (socket says 172.21.15.83:40432) 2024-09-19T07:19:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab2effd640 1 -- 172.21.15.83:0/3907885180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab30155d70 msgr2=0x7fab301575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab2e7fc640 1 --2- 172.21.15.83:0/3907885180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab30155d70 0x7fab301575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab2effd640 1 --2- 172.21.15.83:0/3907885180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab30155d70 0x7fab301575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab2effd640 1 -- 172.21.15.83:0/3907885180 >> v1:172.21.15.83:6789/0 conn(0x7fab301592f0 legacy=0x7fab30157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab2dffb640 1 -- 172.21.15.83:0/3907885180 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2181334136 0 0) 0x7fab30156200 con 0x7fab301592f0 2024-09-19T07:19:45.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab2effd640 1 -- 172.21.15.83:0/3907885180 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab301567c0 con 0x7fab300c6120 2024-09-19T07:19:45.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.219+0000 7fab2e7fc640 1 --2- 172.21.15.83:0/3907885180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab30155d70 0x7fab301575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:45.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2effd640 1 --2- 172.21.15.83:0/3907885180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab300c6120 0x7fab300c6500 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fab200097c0 tx=0x7fab2001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c95f73140d990d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:45.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2dffb640 1 -- 172.21.15.83:0/3907885180 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab200b0070 con 0x7fab300c6120 2024-09-19T07:19:45.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2dffb640 1 -- 172.21.15.83:0/3907885180 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab200ab600 con 0x7fab300c6120 2024-09-19T07:19:45.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2dffb640 1 -- 172.21.15.83:0/3907885180 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab200ab9d0 con 0x7fab300c6120 2024-09-19T07:19:45.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 -- 172.21.15.83:0/3907885180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab300c6120 msgr2=0x7fab300c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 --2- 172.21.15.83:0/3907885180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab300c6120 0x7fab300c6500 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fab200097c0 tx=0x7fab2001c4a0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 -- 172.21.15.83:0/3907885180 shutdown_connections 2024-09-19T07:19:45.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 --2- 172.21.15.83:0/3907885180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab30155d70 0x7fab301575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 --2- 172.21.15.83:0/3907885180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab300c6120 0x7fab300c6500 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 -- 172.21.15.83:0/3907885180 >> 172.21.15.83:0/3907885180 conn(0x7fab3006bc80 msgr2=0x7fab3006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:45.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 -- 172.21.15.83:0/3907885180 shutdown_connections 2024-09-19T07:19:45.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 -- 172.21.15.83:0/3907885180 wait complete. 2024-09-19T07:19:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 Processor -- start 2024-09-19T07:19:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 -- start start 2024-09-19T07:19:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab300c6120 0x7fab300cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab301592f0 0x7fab3014a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab300cba90 0x7fab3014b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fab30153860 con 0x7fab301592f0 2024-09-19T07:19:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab30162ea0 con 0x7fab300cba90 2024-09-19T07:19:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab35457640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fab3014a6c0 con 0x7fab300c6120 2024-09-19T07:19:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab301592f0 0x7fab3014a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab301592f0 0x7fab3014a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40436/0 (socket says 172.21.15.83:40436) 2024-09-19T07:19:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2e7fc640 1 -- 172.21.15.83:0/2675460038 learned_addr learned my addr 172.21.15.83:0/2675460038 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2effd640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab300c6120 0x7fab300cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2f7fe640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab300cba90 0x7fab3014b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2e7fc640 1 -- 172.21.15.83:0/2675460038 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab300c6120 msgr2=0x7fab300cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2e7fc640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab300c6120 0x7fab300cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2e7fc640 1 -- 172.21.15.83:0/2675460038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab300cba90 msgr2=0x7fab3014b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2e7fc640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab300cba90 0x7fab3014b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.223+0000 7fab2e7fc640 1 -- 172.21.15.83:0/2675460038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab301ef320 con 0x7fab301592f0 2024-09-19T07:19:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.227+0000 7fab2e7fc640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab301592f0 0x7fab3014a180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fab2400c7c0 tx=0x7fab2400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.227+0000 7fab177fe640 1 -- 172.21.15.83:0/2675460038 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab240079d0 con 0x7fab301592f0 2024-09-19T07:19:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.227+0000 7fab177fe640 1 -- 172.21.15.83:0/2675460038 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab24007b70 con 0x7fab301592f0 2024-09-19T07:19:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.227+0000 7fab35457640 1 -- 172.21.15.83:0/2675460038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab300ca030 con 0x7fab301592f0 2024-09-19T07:19:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.227+0000 7fab177fe640 1 -- 172.21.15.83:0/2675460038 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab240113f0 con 0x7fab301592f0 2024-09-19T07:19:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.227+0000 7fab35457640 1 -- 172.21.15.83:0/2675460038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab300b1700 con 0x7fab301592f0 2024-09-19T07:19:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.227+0000 7fab35457640 1 -- 172.21.15.83:0/2675460038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab300c96a0 con 0x7fab301592f0 2024-09-19T07:19:45.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.227+0000 7fab177fe640 1 -- 172.21.15.83:0/2675460038 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fab24011590 con 0x7fab301592f0 2024-09-19T07:19:45.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.227+0000 7fab177fe640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab1c0678c0 0x7fab1c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.227+0000 7fab177fe640 1 -- 172.21.15.83:0/2675460038 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fab240860f0 con 0x7fab301592f0 2024-09-19T07:19:45.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.227+0000 7fab2effd640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab1c0678c0 0x7fab1c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.227+0000 7fab2effd640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab1c0678c0 0x7fab1c069d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fab200ad000 tx=0x7fab200a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:45.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.231+0000 7fab177fe640 1 -- 172.21.15.83:0/2675460038 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fab24015020 con 0x7fab301592f0 2024-09-19T07:19:45.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.387+0000 7fab35457640 1 -- 172.21.15.83:0/2675460038 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fab301ef0d0 con 0x7fab1c0678c0 2024-09-19T07:19:45.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.423+0000 7fab177fe640 1 -- 172.21.15.83:0/2675460038 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490348 (secure 0 0 0) 0x7fab301ef0d0 con 0x7fab1c0678c0 2024-09-19T07:19:45.427 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:45.430 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:19:45.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.427+0000 7fab35457640 1 -- 172.21.15.83:0/2675460038 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab1c0678c0 msgr2=0x7fab1c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.427+0000 7fab35457640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab1c0678c0 0x7fab1c069d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fab200ad000 tx=0x7fab200a6000 comp rx=0 tx=0).stop 2024-09-19T07:19:45.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.427+0000 7fab35457640 1 -- 172.21.15.83:0/2675460038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab301592f0 msgr2=0x7fab3014a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.427+0000 7fab35457640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab301592f0 0x7fab3014a180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fab2400c7c0 tx=0x7fab2400cc80 comp rx=0 tx=0).stop 2024-09-19T07:19:45.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.427+0000 7fab35457640 1 -- 172.21.15.83:0/2675460038 shutdown_connections 2024-09-19T07:19:45.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.427+0000 7fab35457640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab1c0678c0 0x7fab1c069d80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.427+0000 7fab35457640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab300cba90 0x7fab3014b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.427+0000 7fab35457640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab301592f0 0x7fab3014a180 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.427+0000 7fab35457640 1 --2- 172.21.15.83:0/2675460038 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab300c6120 0x7fab300cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.427+0000 7fab35457640 1 -- 172.21.15.83:0/2675460038 >> 172.21.15.83:0/2675460038 conn(0x7fab3006bc80 msgr2=0x7fab301575d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:45.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.427+0000 7fab35457640 1 -- 172.21.15.83:0/2675460038 shutdown_connections 2024-09-19T07:19:45.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.427+0000 7fab35457640 1 -- 172.21.15.83:0/2675460038 wait complete. 2024-09-19T07:19:45.447 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-19T07:19:44.814241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":248,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":304,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223556,"kb_used_data":3156,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751164,"statfs":{"total":773094113280,"available":772865191936,"internally_reserved":0,"allocated":3231744,"data_stored":1920632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001481"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.382852+0000","last_change":"2024-09-19T07:19:40.358457+0000","last_active":"2024-09-19T07:19:40.382852+0000","last_peered":"2024-09-19T07:19:40.382852+0000","last_clean":"2024-09-19T07:19:40.382852+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:19:40.382852+0000","last_undegraded":"2024-09-19T07:19:40.382852+0000","last_fullsized":"2024-09-19T07:19:40.382852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:28.893439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370097+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:19:40.370097+0000","last_peered":"2024-09-19T07:19:40.370097+0000","last_clean":"2024-09-19T07:19:40.370097+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:19:40.370097+0000","last_undegraded":"2024-09-19T07:19:40.370097+0000","last_fullsized":"2024-09-19T07:19:40.370097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:57.799442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.343030+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:19:40.343030+0000","last_peered":"2024-09-19T07:19:40.343030+0000","last_clean":"2024-09-19T07:19:40.343030+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:19:40.343030+0000","last_undegraded":"2024-09-19T07:19:40.343030+0000","last_fullsized":"2024-09-19T07:19:40.343030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.339006+0000","last_change":"2024-09-19T07:19:39.401621+0000","last_active":"2024-09-19T07:19:40.339006+0000","last_peered":"2024-09-19T07:19:40.339006+0000","last_clean":"2024-09-19T07:19:40.339006+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:19:40.339006+0000","last_undegraded":"2024-09-19T07:19:40.339006+0000","last_fullsized":"2024-09-19T07:19:40.339006+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:32.095381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.380910+0000","last_change":"2024-09-19T07:19:40.366073+0000","last_active":"2024-09-19T07:19:40.380910+0000","last_peered":"2024-09-19T07:19:40.380910+0000","last_clean":"2024-09-19T07:19:40.380910+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:19:40.380910+0000","last_undegraded":"2024-09-19T07:19:40.380910+0000","last_fullsized":"2024-09-19T07:19:40.380910+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.343282+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:19:40.343282+0000","last_peered":"2024-09-19T07:19:40.343282+0000","last_clean":"2024-09-19T07:19:40.343282+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:19:40.343282+0000","last_undegraded":"2024-09-19T07:19:40.343282+0000","last_fullsized":"2024-09-19T07:19:40.343282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.361481+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:19:40.361481+0000","last_peered":"2024-09-19T07:19:40.361481+0000","last_clean":"2024-09-19T07:19:40.361481+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:19:40.361481+0000","last_undegraded":"2024-09-19T07:19:40.361481+0000","last_fullsized":"2024-09-19T07:19:40.361481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:15.036396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.349065+0000","last_change":"2024-09-19T07:19:39.403810+0000","last_active":"2024-09-19T07:19:40.349065+0000","last_peered":"2024-09-19T07:19:40.349065+0000","last_clean":"2024-09-19T07:19:40.349065+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:19:40.349065+0000","last_undegraded":"2024-09-19T07:19:40.349065+0000","last_fullsized":"2024-09-19T07:19:40.349065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:55.714964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392960+0000","last_change":"2024-09-19T07:19:40.390455+0000","last_active":"2024-09-19T07:19:40.392960+0000","last_peered":"2024-09-19T07:19:40.392960+0000","last_clean":"2024-09-19T07:19:40.392960+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:19:40.392960+0000","last_undegraded":"2024-09-19T07:19:40.392960+0000","last_fullsized":"2024-09-19T07:19:40.392960+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.368208+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:19:40.368208+0000","last_peered":"2024-09-19T07:19:40.368208+0000","last_clean":"2024-09-19T07:19:40.368208+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:19:40.368208+0000","last_undegraded":"2024-09-19T07:19:40.368208+0000","last_fullsized":"2024-09-19T07:19:40.368208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.348068+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:19:40.348068+0000","last_peered":"2024-09-19T07:19:40.348068+0000","last_clean":"2024-09-19T07:19:40.348068+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:19:40.348068+0000","last_undegraded":"2024-09-19T07:19:40.348068+0000","last_fullsized":"2024-09-19T07:19:40.348068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"24'1","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.433114+0000","last_change":"2024-09-19T07:19:39.408309+0000","last_active":"2024-09-19T07:19:40.433114+0000","last_peered":"2024-09-19T07:19:40.433114+0000","last_clean":"2024-09-19T07:19:40.433114+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:19:40.433114+0000","last_undegraded":"2024-09-19T07:19:40.433114+0000","last_fullsized":"2024-09-19T07:19:40.433114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:19.396261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.380454+0000","last_change":"2024-09-19T07:19:40.372543+0000","last_active":"2024-09-19T07:19:40.380454+0000","last_peered":"2024-09-19T07:19:40.380454+0000","last_clean":"2024-09-19T07:19:40.380454+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:19:40.380454+0000","last_undegraded":"2024-09-19T07:19:40.380454+0000","last_fullsized":"2024-09-19T07:19:40.380454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:23.474199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.360024+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:19:40.360024+0000","last_peered":"2024-09-19T07:19:40.360024+0000","last_clean":"2024-09-19T07:19:40.360024+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:19:40.360024+0000","last_undegraded":"2024-09-19T07:19:40.360024+0000","last_fullsized":"2024-09-19T07:19:40.360024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.341915+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:19:40.341915+0000","last_peered":"2024-09-19T07:19:40.341915+0000","last_clean":"2024-09-19T07:19:40.341915+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:19:40.341915+0000","last_undegraded":"2024-09-19T07:19:40.341915+0000","last_fullsized":"2024-09-19T07:19:40.341915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"24'1","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.433066+0000","last_change":"2024-09-19T07:19:39.407045+0000","last_active":"2024-09-19T07:19:40.433066+0000","last_peered":"2024-09-19T07:19:40.433066+0000","last_clean":"2024-09-19T07:19:40.433066+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:19:40.433066+0000","last_undegraded":"2024-09-19T07:19:40.433066+0000","last_fullsized":"2024-09-19T07:19:40.433066+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:24.846974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.397498+0000","last_change":"2024-09-19T07:19:40.396520+0000","last_active":"2024-09-19T07:19:40.397498+0000","last_peered":"2024-09-19T07:19:40.397498+0000","last_clean":"2024-09-19T07:19:40.397498+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:19:40.397498+0000","last_undegraded":"2024-09-19T07:19:40.397498+0000","last_fullsized":"2024-09-19T07:19:40.397498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.345192+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:19:40.345192+0000","last_peered":"2024-09-19T07:19:40.345192+0000","last_clean":"2024-09-19T07:19:40.345192+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:19:40.345192+0000","last_undegraded":"2024-09-19T07:19:40.345192+0000","last_fullsized":"2024-09-19T07:19:40.345192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.359616+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:19:40.359616+0000","last_peered":"2024-09-19T07:19:40.359616+0000","last_clean":"2024-09-19T07:19:40.359616+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:19:40.359616+0000","last_undegraded":"2024-09-19T07:19:40.359616+0000","last_fullsized":"2024-09-19T07:19:40.359616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"24'1","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.430561+0000","last_change":"2024-09-19T07:19:39.410500+0000","last_active":"2024-09-19T07:19:40.430561+0000","last_peered":"2024-09-19T07:19:40.430561+0000","last_clean":"2024-09-19T07:19:40.430561+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:19:40.430561+0000","last_undegraded":"2024-09-19T07:19:40.430561+0000","last_fullsized":"2024-09-19T07:19:40.430561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:04.182606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393523+0000","last_change":"2024-09-19T07:19:40.381259+0000","last_active":"2024-09-19T07:19:40.393523+0000","last_peered":"2024-09-19T07:19:40.393523+0000","last_clean":"2024-09-19T07:19:40.393523+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:19:40.393523+0000","last_undegraded":"2024-09-19T07:19:40.393523+0000","last_fullsized":"2024-09-19T07:19:40.393523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:59.407542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340656+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:19:40.340656+0000","last_peered":"2024-09-19T07:19:40.340656+0000","last_clean":"2024-09-19T07:19:40.340656+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:19:40.340656+0000","last_undegraded":"2024-09-19T07:19:40.340656+0000","last_fullsized":"2024-09-19T07:19:40.340656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.357584+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:19:40.357584+0000","last_peered":"2024-09-19T07:19:40.357584+0000","last_clean":"2024-09-19T07:19:40.357584+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:19:40.357584+0000","last_undegraded":"2024-09-19T07:19:40.357584+0000","last_fullsized":"2024-09-19T07:19:40.357584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.361778+0000","last_change":"2024-09-19T07:19:39.412123+0000","last_active":"2024-09-19T07:19:40.361778+0000","last_peered":"2024-09-19T07:19:40.361778+0000","last_clean":"2024-09-19T07:19:40.361778+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:19:40.361778+0000","last_undegraded":"2024-09-19T07:19:40.361778+0000","last_fullsized":"2024-09-19T07:19:40.361778+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.388968+0000","last_change":"2024-09-19T07:19:40.383214+0000","last_active":"2024-09-19T07:19:40.388968+0000","last_peered":"2024-09-19T07:19:40.388968+0000","last_clean":"2024-09-19T07:19:40.388968+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:19:40.388968+0000","last_undegraded":"2024-09-19T07:19:40.388968+0000","last_fullsized":"2024-09-19T07:19:40.388968+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:20.550115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340034+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:19:40.340034+0000","last_peered":"2024-09-19T07:19:40.340034+0000","last_clean":"2024-09-19T07:19:40.340034+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:19:40.340034+0000","last_undegraded":"2024-09-19T07:19:40.340034+0000","last_fullsized":"2024-09-19T07:19:40.340034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.348741+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:19:40.348741+0000","last_peered":"2024-09-19T07:19:40.348741+0000","last_clean":"2024-09-19T07:19:40.348741+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:19:40.348741+0000","last_undegraded":"2024-09-19T07:19:40.348741+0000","last_fullsized":"2024-09-19T07:19:40.348741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"24'1","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.433309+0000","last_change":"2024-09-19T07:19:39.408228+0000","last_active":"2024-09-19T07:19:40.433309+0000","last_peered":"2024-09-19T07:19:40.433309+0000","last_clean":"2024-09-19T07:19:40.433309+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:19:40.433309+0000","last_undegraded":"2024-09-19T07:19:40.433309+0000","last_fullsized":"2024-09-19T07:19:40.433309+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.396717+0000","last_change":"2024-09-19T07:19:40.395566+0000","last_active":"2024-09-19T07:19:40.396717+0000","last_peered":"2024-09-19T07:19:40.396717+0000","last_clean":"2024-09-19T07:19:40.396717+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:19:40.396717+0000","last_undegraded":"2024-09-19T07:19:40.396717+0000","last_fullsized":"2024-09-19T07:19:40.396717+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:04:27.088530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.359402+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:19:40.359402+0000","last_peered":"2024-09-19T07:19:40.359402+0000","last_clean":"2024-09-19T07:19:40.359402+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:19:40.359402+0000","last_undegraded":"2024-09-19T07:19:40.359402+0000","last_fullsized":"2024-09-19T07:19:40.359402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.354653+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:19:40.354653+0000","last_peered":"2024-09-19T07:19:40.354653+0000","last_clean":"2024-09-19T07:19:40.354653+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:19:40.354653+0000","last_undegraded":"2024-09-19T07:19:40.354653+0000","last_fullsized":"2024-09-19T07:19:40.354653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"24'2","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.429961+0000","last_change":"2024-09-19T07:19:39.424661+0000","last_active":"2024-09-19T07:19:40.429961+0000","last_peered":"2024-09-19T07:19:40.429961+0000","last_clean":"2024-09-19T07:19:40.429961+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:19:40.429961+0000","last_undegraded":"2024-09-19T07:19:40.429961+0000","last_fullsized":"2024-09-19T07:19:40.429961+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393009+0000","last_change":"2024-09-19T07:19:40.388972+0000","last_active":"2024-09-19T07:19:40.393009+0000","last_peered":"2024-09-19T07:19:40.393009+0000","last_clean":"2024-09-19T07:19:40.393009+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:19:40.393009+0000","last_undegraded":"2024-09-19T07:19:40.393009+0000","last_fullsized":"2024-09-19T07:19:40.393009+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:30.585686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370157+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:19:40.370157+0000","last_peered":"2024-09-19T07:19:40.370157+0000","last_clean":"2024-09-19T07:19:40.370157+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:19:40.370157+0000","last_undegraded":"2024-09-19T07:19:40.370157+0000","last_fullsized":"2024-09-19T07:19:40.370157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:23.791880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.346035+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:19:40.346035+0000","last_peered":"2024-09-19T07:19:40.346035+0000","last_clean":"2024-09-19T07:19:40.346035+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:19:40.346035+0000","last_undegraded":"2024-09-19T07:19:40.346035+0000","last_fullsized":"2024-09-19T07:19:40.346035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.358499+0000","last_change":"2024-09-19T07:19:39.412192+0000","last_active":"2024-09-19T07:19:40.358499+0000","last_peered":"2024-09-19T07:19:40.358499+0000","last_clean":"2024-09-19T07:19:40.358499+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:19:40.358499+0000","last_undegraded":"2024-09-19T07:19:40.358499+0000","last_fullsized":"2024-09-19T07:19:40.358499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.395575+0000","last_change":"2024-09-19T07:19:40.390217+0000","last_active":"2024-09-19T07:19:40.395575+0000","last_peered":"2024-09-19T07:19:40.395575+0000","last_clean":"2024-09-19T07:19:40.395575+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:19:40.395575+0000","last_undegraded":"2024-09-19T07:19:40.395575+0000","last_fullsized":"2024-09-19T07:19:40.395575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342531+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:19:40.342531+0000","last_peered":"2024-09-19T07:19:40.342531+0000","last_clean":"2024-09-19T07:19:40.342531+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:19:40.342531+0000","last_undegraded":"2024-09-19T07:19:40.342531+0000","last_fullsized":"2024-09-19T07:19:40.342531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356461+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:19:40.356461+0000","last_peered":"2024-09-19T07:19:40.356461+0000","last_clean":"2024-09-19T07:19:40.356461+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:19:40.356461+0000","last_undegraded":"2024-09-19T07:19:40.356461+0000","last_fullsized":"2024-09-19T07:19:40.356461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:50.700962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.338651+0000","last_change":"2024-09-19T07:19:39.434150+0000","last_active":"2024-09-19T07:19:40.338651+0000","last_peered":"2024-09-19T07:19:40.338651+0000","last_clean":"2024-09-19T07:19:40.338651+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:19:40.338651+0000","last_undegraded":"2024-09-19T07:19:40.338651+0000","last_fullsized":"2024-09-19T07:19:40.338651+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:13.654208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.386665+0000","last_change":"2024-09-19T07:19:40.380140+0000","last_active":"2024-09-19T07:19:40.386665+0000","last_peered":"2024-09-19T07:19:40.386665+0000","last_clean":"2024-09-19T07:19:40.386665+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:19:40.386665+0000","last_undegraded":"2024-09-19T07:19:40.386665+0000","last_fullsized":"2024-09-19T07:19:40.386665+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:46.389885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.350823+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:19:40.350823+0000","last_peered":"2024-09-19T07:19:40.350823+0000","last_clean":"2024-09-19T07:19:40.350823+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:19:40.350823+0000","last_undegraded":"2024-09-19T07:19:40.350823+0000","last_fullsized":"2024-09-19T07:19:40.350823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.368309+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:19:40.368309+0000","last_peered":"2024-09-19T07:19:40.368309+0000","last_clean":"2024-09-19T07:19:40.368309+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:19:40.368309+0000","last_undegraded":"2024-09-19T07:19:40.368309+0000","last_fullsized":"2024-09-19T07:19:40.368309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.369389+0000","last_change":"2024-09-19T07:19:39.411605+0000","last_active":"2024-09-19T07:19:40.369389+0000","last_peered":"2024-09-19T07:19:40.369389+0000","last_clean":"2024-09-19T07:19:40.369389+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:19:40.369389+0000","last_undegraded":"2024-09-19T07:19:40.369389+0000","last_fullsized":"2024-09-19T07:19:40.369389+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:55.654483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.379445+0000","last_change":"2024-09-19T07:19:40.372083+0000","last_active":"2024-09-19T07:19:40.379445+0000","last_peered":"2024-09-19T07:19:40.379445+0000","last_clean":"2024-09-19T07:19:40.379445+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:19:40.379445+0000","last_undegraded":"2024-09-19T07:19:40.379445+0000","last_fullsized":"2024-09-19T07:19:40.379445+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:23:15.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342175+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:19:40.342175+0000","last_peered":"2024-09-19T07:19:40.342175+0000","last_clean":"2024-09-19T07:19:40.342175+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:19:40.342175+0000","last_undegraded":"2024-09-19T07:19:40.342175+0000","last_fullsized":"2024-09-19T07:19:40.342175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.351077+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:19:40.351077+0000","last_peered":"2024-09-19T07:19:40.351077+0000","last_clean":"2024-09-19T07:19:40.351077+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:19:40.351077+0000","last_undegraded":"2024-09-19T07:19:40.351077+0000","last_fullsized":"2024-09-19T07:19:40.351077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356990+0000","last_change":"2024-09-19T07:19:39.420340+0000","last_active":"2024-09-19T07:19:40.356990+0000","last_peered":"2024-09-19T07:19:40.356990+0000","last_clean":"2024-09-19T07:19:40.356990+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:19:40.356990+0000","last_undegraded":"2024-09-19T07:19:40.356990+0000","last_fullsized":"2024-09-19T07:19:40.356990+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:29.338848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.390612+0000","last_change":"2024-09-19T07:19:40.383082+0000","last_active":"2024-09-19T07:19:40.390612+0000","last_peered":"2024-09-19T07:19:40.390612+0000","last_clean":"2024-09-19T07:19:40.390612+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:19:40.390612+0000","last_undegraded":"2024-09-19T07:19:40.390612+0000","last_fullsized":"2024-09-19T07:19:40.390612+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:42.470941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356094+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:19:40.356094+0000","last_peered":"2024-09-19T07:19:40.356094+0000","last_clean":"2024-09-19T07:19:40.356094+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:19:40.356094+0000","last_undegraded":"2024-09-19T07:19:40.356094+0000","last_fullsized":"2024-09-19T07:19:40.356094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.341157+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:19:40.341157+0000","last_peered":"2024-09-19T07:19:40.341157+0000","last_clean":"2024-09-19T07:19:40.341157+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:19:40.341157+0000","last_undegraded":"2024-09-19T07:19:40.341157+0000","last_fullsized":"2024-09-19T07:19:40.341157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.350681+0000","last_change":"2024-09-19T07:19:39.407135+0000","last_active":"2024-09-19T07:19:40.350681+0000","last_peered":"2024-09-19T07:19:40.350681+0000","last_clean":"2024-09-19T07:19:40.350681+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:19:40.350681+0000","last_undegraded":"2024-09-19T07:19:40.350681+0000","last_fullsized":"2024-09-19T07:19:40.350681+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:27.890595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.391138+0000","last_change":"2024-09-19T07:19:40.368866+0000","last_active":"2024-09-19T07:19:40.391138+0000","last_peered":"2024-09-19T07:19:40.391138+0000","last_clean":"2024-09-19T07:19:40.391138+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:19:40.391138+0000","last_undegraded":"2024-09-19T07:19:40.391138+0000","last_fullsized":"2024-09-19T07:19:40.391138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:47.944394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356301+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:19:40.356301+0000","last_peered":"2024-09-19T07:19:40.356301+0000","last_clean":"2024-09-19T07:19:40.356301+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:19:40.356301+0000","last_undegraded":"2024-09-19T07:19:40.356301+0000","last_fullsized":"2024-09-19T07:19:40.356301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.357998+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:19:40.357998+0000","last_peered":"2024-09-19T07:19:40.357998+0000","last_clean":"2024-09-19T07:19:40.357998+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:19:40.357998+0000","last_undegraded":"2024-09-19T07:19:40.357998+0000","last_fullsized":"2024-09-19T07:19:40.357998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"24'2","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.430689+0000","last_change":"2024-09-19T07:19:39.420497+0000","last_active":"2024-09-19T07:19:40.430689+0000","last_peered":"2024-09-19T07:19:40.430689+0000","last_clean":"2024-09-19T07:19:40.430689+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:19:40.430689+0000","last_undegraded":"2024-09-19T07:19:40.430689+0000","last_fullsized":"2024-09-19T07:19:40.430689+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:19.547270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393237+0000","last_change":"2024-09-19T07:19:40.387573+0000","last_active":"2024-09-19T07:19:40.393237+0000","last_peered":"2024-09-19T07:19:40.393237+0000","last_clean":"2024-09-19T07:19:40.393237+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:19:40.393237+0000","last_undegraded":"2024-09-19T07:19:40.393237+0000","last_fullsized":"2024-09-19T07:19:40.393237+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:53:37.421178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.357442+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:19:40.357442+0000","last_peered":"2024-09-19T07:19:40.357442+0000","last_clean":"2024-09-19T07:19:40.357442+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:19:40.357442+0000","last_undegraded":"2024-09-19T07:19:40.357442+0000","last_fullsized":"2024-09-19T07:19:40.357442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:20.824916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.354831+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:19:40.354831+0000","last_peered":"2024-09-19T07:19:40.354831+0000","last_clean":"2024-09-19T07:19:40.354831+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:19:40.354831+0000","last_undegraded":"2024-09-19T07:19:40.354831+0000","last_fullsized":"2024-09-19T07:19:40.354831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370650+0000","last_change":"2024-09-19T07:19:39.410752+0000","last_active":"2024-09-19T07:19:40.370650+0000","last_peered":"2024-09-19T07:19:40.370650+0000","last_clean":"2024-09-19T07:19:40.370650+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:40.370650+0000","last_undegraded":"2024-09-19T07:19:40.370650+0000","last_fullsized":"2024-09-19T07:19:40.370650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:47:09.376497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.394761+0000","last_change":"2024-09-19T07:19:40.392973+0000","last_active":"2024-09-19T07:19:40.394761+0000","last_peered":"2024-09-19T07:19:40.394761+0000","last_clean":"2024-09-19T07:19:40.394761+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:19:40.394761+0000","last_undegraded":"2024-09-19T07:19:40.394761+0000","last_fullsized":"2024-09-19T07:19:40.394761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340207+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:19:40.340207+0000","last_peered":"2024-09-19T07:19:40.340207+0000","last_clean":"2024-09-19T07:19:40.340207+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:19:40.340207+0000","last_undegraded":"2024-09-19T07:19:40.340207+0000","last_fullsized":"2024-09-19T07:19:40.340207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.344699+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:19:40.344699+0000","last_peered":"2024-09-19T07:19:40.344699+0000","last_clean":"2024-09-19T07:19:40.344699+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:19:40.344699+0000","last_undegraded":"2024-09-19T07:19:40.344699+0000","last_fullsized":"2024-09-19T07:19:40.344699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":176,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.368046+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:19:40.368046+0000","last_peered":"2024-09-19T07:19:40.368046+0000","last_clean":"2024-09-19T07:19:40.368046+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:19:40.368046+0000","last_undegraded":"2024-09-19T07:19:40.368046+0000","last_fullsized":"2024-09-19T07:19:40.368046+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342960+0000","last_change":"2024-09-19T07:19:39.429085+0000","last_active":"2024-09-19T07:19:40.342960+0000","last_peered":"2024-09-19T07:19:40.342960+0000","last_clean":"2024-09-19T07:19:40.342960+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:19:40.342960+0000","last_undegraded":"2024-09-19T07:19:40.342960+0000","last_fullsized":"2024-09-19T07:19:40.342960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393628+0000","last_change":"2024-09-19T07:19:40.382352+0000","last_active":"2024-09-19T07:19:40.393628+0000","last_peered":"2024-09-19T07:19:40.393628+0000","last_clean":"2024-09-19T07:19:40.393628+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:19:40.393628+0000","last_undegraded":"2024-09-19T07:19:40.393628+0000","last_fullsized":"2024-09-19T07:19:40.393628+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:56.356646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.364326+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:19:40.364326+0000","last_peered":"2024-09-19T07:19:40.364326+0000","last_clean":"2024-09-19T07:19:40.364326+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:19:40.364326+0000","last_undegraded":"2024-09-19T07:19:40.364326+0000","last_fullsized":"2024-09-19T07:19:40.364326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.360771+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:19:40.360771+0000","last_peered":"2024-09-19T07:19:40.360771+0000","last_clean":"2024-09-19T07:19:40.360771+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:19:40.360771+0000","last_undegraded":"2024-09-19T07:19:40.360771+0000","last_fullsized":"2024-09-19T07:19:40.360771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.344832+0000","last_change":"2024-09-19T07:19:39.427644+0000","last_active":"2024-09-19T07:19:40.344832+0000","last_peered":"2024-09-19T07:19:40.344832+0000","last_clean":"2024-09-19T07:19:40.344832+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:19:40.344832+0000","last_undegraded":"2024-09-19T07:19:40.344832+0000","last_fullsized":"2024-09-19T07:19:40.344832+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393689+0000","last_change":"2024-09-19T07:19:40.390380+0000","last_active":"2024-09-19T07:19:40.393689+0000","last_peered":"2024-09-19T07:19:40.393689+0000","last_clean":"2024-09-19T07:19:40.393689+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:19:40.393689+0000","last_undegraded":"2024-09-19T07:19:40.393689+0000","last_fullsized":"2024-09-19T07:19:40.393689+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:56.717809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.348289+0000","last_change":"2024-09-19T07:19:32.276600+0000","last_active":"2024-09-19T07:19:40.348289+0000","last_peered":"2024-09-19T07:19:40.348289+0000","last_clean":"2024-09-19T07:19:40.348289+0000","last_became_active":"2024-09-19T07:19:32.275766+0000","last_became_peered":"2024-09-19T07:19:32.275766+0000","last_unstale":"2024-09-19T07:19:40.348289+0000","last_undegraded":"2024-09-19T07:19:40.348289+0000","last_fullsized":"2024-09-19T07:19:40.348289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:48.506399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.354100+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:19:40.354100+0000","last_peered":"2024-09-19T07:19:40.354100+0000","last_clean":"2024-09-19T07:19:40.354100+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:19:40.354100+0000","last_undegraded":"2024-09-19T07:19:40.354100+0000","last_fullsized":"2024-09-19T07:19:40.354100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"24'2","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.431386+0000","last_change":"2024-09-19T07:19:39.410638+0000","last_active":"2024-09-19T07:19:40.431386+0000","last_peered":"2024-09-19T07:19:40.431386+0000","last_clean":"2024-09-19T07:19:40.431386+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:19:40.431386+0000","last_undegraded":"2024-09-19T07:19:40.431386+0000","last_fullsized":"2024-09-19T07:19:40.431386+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:32.073136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.380210+0000","last_change":"2024-09-19T07:19:40.378481+0000","last_active":"2024-09-19T07:19:40.380210+0000","last_peered":"2024-09-19T07:19:40.380210+0000","last_clean":"2024-09-19T07:19:40.380210+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:19:40.380210+0000","last_undegraded":"2024-09-19T07:19:40.380210+0000","last_fullsized":"2024-09-19T07:19:40.380210+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.371210+0000","last_change":"2024-09-19T07:19:32.292026+0000","last_active":"2024-09-19T07:19:40.371210+0000","last_peered":"2024-09-19T07:19:40.371210+0000","last_clean":"2024-09-19T07:19:40.371210+0000","last_became_active":"2024-09-19T07:19:32.291037+0000","last_became_peered":"2024-09-19T07:19:32.291037+0000","last_unstale":"2024-09-19T07:19:40.371210+0000","last_undegraded":"2024-09-19T07:19:40.371210+0000","last_fullsized":"2024-09-19T07:19:40.371210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:22.190763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.359197+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:19:40.359197+0000","last_peered":"2024-09-19T07:19:40.359197+0000","last_clean":"2024-09-19T07:19:40.359197+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:19:40.359197+0000","last_undegraded":"2024-09-19T07:19:40.359197+0000","last_fullsized":"2024-09-19T07:19:40.359197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.346303+0000","last_change":"2024-09-19T07:19:39.437656+0000","last_active":"2024-09-19T07:19:40.346303+0000","last_peered":"2024-09-19T07:19:40.346303+0000","last_clean":"2024-09-19T07:19:40.346303+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:19:40.346303+0000","last_undegraded":"2024-09-19T07:19:40.346303+0000","last_fullsized":"2024-09-19T07:19:40.346303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:41.468096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393503+0000","last_change":"2024-09-19T07:19:40.379289+0000","last_active":"2024-09-19T07:19:40.393503+0000","last_peered":"2024-09-19T07:19:40.393503+0000","last_clean":"2024-09-19T07:19:40.393503+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:19:40.393503+0000","last_undegraded":"2024-09-19T07:19:40.393503+0000","last_fullsized":"2024-09-19T07:19:40.393503+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:09.095760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.369220+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:19:40.369220+0000","last_peered":"2024-09-19T07:19:40.369220+0000","last_clean":"2024-09-19T07:19:40.369220+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:19:40.369220+0000","last_undegraded":"2024-09-19T07:19:40.369220+0000","last_fullsized":"2024-09-19T07:19:40.369220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.360228+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:19:40.360228+0000","last_peered":"2024-09-19T07:19:40.360228+0000","last_clean":"2024-09-19T07:19:40.360228+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:19:40.360228+0000","last_undegraded":"2024-09-19T07:19:40.360228+0000","last_fullsized":"2024-09-19T07:19:40.360228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.358088+0000","last_change":"2024-09-19T07:19:39.413589+0000","last_active":"2024-09-19T07:19:40.358088+0000","last_peered":"2024-09-19T07:19:40.358088+0000","last_clean":"2024-09-19T07:19:40.358088+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:19:40.358088+0000","last_undegraded":"2024-09-19T07:19:40.358088+0000","last_fullsized":"2024-09-19T07:19:40.358088+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392651+0000","last_change":"2024-09-19T07:19:40.387159+0000","last_active":"2024-09-19T07:19:40.392651+0000","last_peered":"2024-09-19T07:19:40.392651+0000","last_clean":"2024-09-19T07:19:40.392651+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:19:40.392651+0000","last_undegraded":"2024-09-19T07:19:40.392651+0000","last_fullsized":"2024-09-19T07:19:40.392651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.363062+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:19:40.363062+0000","last_peered":"2024-09-19T07:19:40.363062+0000","last_clean":"2024-09-19T07:19:40.363062+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:19:40.363062+0000","last_undegraded":"2024-09-19T07:19:40.363062+0000","last_fullsized":"2024-09-19T07:19:40.363062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.358828+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:19:40.358828+0000","last_peered":"2024-09-19T07:19:40.358828+0000","last_clean":"2024-09-19T07:19:40.358828+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:19:40.358828+0000","last_undegraded":"2024-09-19T07:19:40.358828+0000","last_fullsized":"2024-09-19T07:19:40.358828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370015+0000","last_change":"2024-09-19T07:19:39.406110+0000","last_active":"2024-09-19T07:19:40.370015+0000","last_peered":"2024-09-19T07:19:40.370015+0000","last_clean":"2024-09-19T07:19:40.370015+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:19:40.370015+0000","last_undegraded":"2024-09-19T07:19:40.370015+0000","last_fullsized":"2024-09-19T07:19:40.370015+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:10:05.224225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.374916+0000","last_change":"2024-09-19T07:19:40.365764+0000","last_active":"2024-09-19T07:19:40.374916+0000","last_peered":"2024-09-19T07:19:40.374916+0000","last_clean":"2024-09-19T07:19:40.374916+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:19:40.374916+0000","last_undegraded":"2024-09-19T07:19:40.374916+0000","last_fullsized":"2024-09-19T07:19:40.374916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:54.436493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342540+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:19:40.342540+0000","last_peered":"2024-09-19T07:19:40.342540+0000","last_clean":"2024-09-19T07:19:40.342540+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:19:40.342540+0000","last_undegraded":"2024-09-19T07:19:40.342540+0000","last_fullsized":"2024-09-19T07:19:40.342540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.349026+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:19:40.349026+0000","last_peered":"2024-09-19T07:19:40.349026+0000","last_clean":"2024-09-19T07:19:40.349026+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:19:40.349026+0000","last_undegraded":"2024-09-19T07:19:40.349026+0000","last_fullsized":"2024-09-19T07:19:40.349026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342359+0000","last_change":"2024-09-19T07:19:39.429768+0000","last_active":"2024-09-19T07:19:40.342359+0000","last_peered":"2024-09-19T07:19:40.342359+0000","last_clean":"2024-09-19T07:19:40.342359+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:19:40.342359+0000","last_undegraded":"2024-09-19T07:19:40.342359+0000","last_fullsized":"2024-09-19T07:19:40.342359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:47.503631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.380706+0000","last_change":"2024-09-19T07:19:40.373542+0000","last_active":"2024-09-19T07:19:40.380706+0000","last_peered":"2024-09-19T07:19:40.380706+0000","last_clean":"2024-09-19T07:19:40.380706+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:19:40.380706+0000","last_undegraded":"2024-09-19T07:19:40.380706+0000","last_fullsized":"2024-09-19T07:19:40.380706+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:32:34.524302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.359370+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:19:40.359370+0000","last_peered":"2024-09-19T07:19:40.359370+0000","last_clean":"2024-09-19T07:19:40.359370+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:19:40.359370+0000","last_undegraded":"2024-09-19T07:19:40.359370+0000","last_fullsized":"2024-09-19T07:19:40.359370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342449+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:19:40.342449+0000","last_peered":"2024-09-19T07:19:40.342449+0000","last_clean":"2024-09-19T07:19:40.342449+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:19:40.342449+0000","last_undegraded":"2024-09-19T07:19:40.342449+0000","last_fullsized":"2024-09-19T07:19:40.342449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"24'6","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.437011+0000","last_change":"2024-09-19T07:19:39.411074+0000","last_active":"2024-09-19T07:19:40.437011+0000","last_peered":"2024-09-19T07:19:40.437011+0000","last_clean":"2024-09-19T07:19:40.437011+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:19:40.437011+0000","last_undegraded":"2024-09-19T07:19:40.437011+0000","last_fullsized":"2024-09-19T07:19:40.437011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:53.992787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.372727+0000","last_change":"2024-09-19T07:19:40.364814+0000","last_active":"2024-09-19T07:19:40.372727+0000","last_peered":"2024-09-19T07:19:40.372727+0000","last_clean":"2024-09-19T07:19:40.372727+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:19:40.372727+0000","last_undegraded":"2024-09-19T07:19:40.372727+0000","last_fullsized":"2024-09-19T07:19:40.372727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.360602+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:19:40.360602+0000","last_peered":"2024-09-19T07:19:40.360602+0000","last_clean":"2024-09-19T07:19:40.360602+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:19:40.360602+0000","last_undegraded":"2024-09-19T07:19:40.360602+0000","last_fullsized":"2024-09-19T07:19:40.360602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:57.131912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370026+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:19:40.370026+0000","last_peered":"2024-09-19T07:19:40.370026+0000","last_clean":"2024-09-19T07:19:40.370026+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:19:40.370026+0000","last_undegraded":"2024-09-19T07:19:40.370026+0000","last_fullsized":"2024-09-19T07:19:40.370026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:23.367365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"24'3","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.429175+0000","last_change":"2024-09-19T07:19:39.431616+0000","last_active":"2024-09-19T07:19:40.429175+0000","last_peered":"2024-09-19T07:19:40.429175+0000","last_clean":"2024-09-19T07:19:40.429175+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:19:40.429175+0000","last_undegraded":"2024-09-19T07:19:40.429175+0000","last_fullsized":"2024-09-19T07:19:40.429175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:18.149581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.390972+0000","last_change":"2024-09-19T07:19:40.381443+0000","last_active":"2024-09-19T07:19:40.390972+0000","last_peered":"2024-09-19T07:19:40.390972+0000","last_clean":"2024-09-19T07:19:40.390972+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:19:40.390972+0000","last_undegraded":"2024-09-19T07:19:40.390972+0000","last_fullsized":"2024-09-19T07:19:40.390972+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:56:40.928844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.364027+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:19:40.364027+0000","last_peered":"2024-09-19T07:19:40.364027+0000","last_clean":"2024-09-19T07:19:40.364027+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:19:40.364027+0000","last_undegraded":"2024-09-19T07:19:40.364027+0000","last_fullsized":"2024-09-19T07:19:40.364027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.343214+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:19:40.343214+0000","last_peered":"2024-09-19T07:19:40.343214+0000","last_clean":"2024-09-19T07:19:40.343214+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:19:40.343214+0000","last_undegraded":"2024-09-19T07:19:40.343214+0000","last_fullsized":"2024-09-19T07:19:40.343214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356615+0000","last_change":"2024-09-19T07:19:39.423972+0000","last_active":"2024-09-19T07:19:40.356615+0000","last_peered":"2024-09-19T07:19:40.356615+0000","last_clean":"2024-09-19T07:19:40.356615+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:19:40.356615+0000","last_undegraded":"2024-09-19T07:19:40.356615+0000","last_fullsized":"2024-09-19T07:19:40.356615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:45.387040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.387950+0000","last_change":"2024-09-19T07:19:40.382092+0000","last_active":"2024-09-19T07:19:40.387950+0000","last_peered":"2024-09-19T07:19:40.387950+0000","last_clean":"2024-09-19T07:19:40.387950+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:19:40.387950+0000","last_undegraded":"2024-09-19T07:19:40.387950+0000","last_fullsized":"2024-09-19T07:19:40.387950+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370617+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:19:40.370617+0000","last_peered":"2024-09-19T07:19:40.370617+0000","last_clean":"2024-09-19T07:19:40.370617+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:19:40.370617+0000","last_undegraded":"2024-09-19T07:19:40.370617+0000","last_fullsized":"2024-09-19T07:19:40.370617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.353582+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:19:40.353582+0000","last_peered":"2024-09-19T07:19:40.353582+0000","last_clean":"2024-09-19T07:19:40.353582+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:19:40.353582+0000","last_undegraded":"2024-09-19T07:19:40.353582+0000","last_fullsized":"2024-09-19T07:19:40.353582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"24'2","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.429916+0000","last_change":"2024-09-19T07:19:39.424879+0000","last_active":"2024-09-19T07:19:40.429916+0000","last_peered":"2024-09-19T07:19:40.429916+0000","last_clean":"2024-09-19T07:19:40.429916+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:19:40.429916+0000","last_undegraded":"2024-09-19T07:19:40.429916+0000","last_fullsized":"2024-09-19T07:19:40.429916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:30.021543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.386869+0000","last_change":"2024-09-19T07:19:40.384136+0000","last_active":"2024-09-19T07:19:40.386869+0000","last_peered":"2024-09-19T07:19:40.386869+0000","last_clean":"2024-09-19T07:19:40.386869+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:19:40.386869+0000","last_undegraded":"2024-09-19T07:19:40.386869+0000","last_fullsized":"2024-09-19T07:19:40.386869+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:20.550115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.369794+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:19:40.369794+0000","last_peered":"2024-09-19T07:19:40.369794+0000","last_clean":"2024-09-19T07:19:40.369794+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:19:40.369794+0000","last_undegraded":"2024-09-19T07:19:40.369794+0000","last_fullsized":"2024-09-19T07:19:40.369794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:48.308137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.344608+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:19:40.344608+0000","last_peered":"2024-09-19T07:19:40.344608+0000","last_clean":"2024-09-19T07:19:40.344608+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:19:40.344608+0000","last_undegraded":"2024-09-19T07:19:40.344608+0000","last_fullsized":"2024-09-19T07:19:40.344608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"24'8","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.433804+0000","last_change":"2024-09-19T07:19:39.410445+0000","last_active":"2024-09-19T07:19:40.433804+0000","last_peered":"2024-09-19T07:19:40.433804+0000","last_clean":"2024-09-19T07:19:40.433804+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:19:40.433804+0000","last_undegraded":"2024-09-19T07:19:40.433804+0000","last_fullsized":"2024-09-19T07:19:40.433804+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:27.020327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392927+0000","last_change":"2024-09-19T07:19:40.389499+0000","last_active":"2024-09-19T07:19:40.392927+0000","last_peered":"2024-09-19T07:19:40.392927+0000","last_clean":"2024-09-19T07:19:40.392927+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:19:40.392927+0000","last_undegraded":"2024-09-19T07:19:40.392927+0000","last_fullsized":"2024-09-19T07:19:40.392927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.344552+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:19:40.344552+0000","last_peered":"2024-09-19T07:19:40.344552+0000","last_clean":"2024-09-19T07:19:40.344552+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:19:40.344552+0000","last_undegraded":"2024-09-19T07:19:40.344552+0000","last_fullsized":"2024-09-19T07:19:40.344552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.345793+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:19:40.345793+0000","last_peered":"2024-09-19T07:19:40.345793+0000","last_clean":"2024-09-19T07:19:40.345793+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:19:40.345793+0000","last_undegraded":"2024-09-19T07:19:40.345793+0000","last_fullsized":"2024-09-19T07:19:40.345793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.338817+0000","last_change":"2024-09-19T07:19:39.416943+0000","last_active":"2024-09-19T07:19:40.338817+0000","last_peered":"2024-09-19T07:19:40.338817+0000","last_clean":"2024-09-19T07:19:40.338817+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:19:40.338817+0000","last_undegraded":"2024-09-19T07:19:40.338817+0000","last_fullsized":"2024-09-19T07:19:40.338817+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:35.555826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.396408+0000","last_change":"2024-09-19T07:19:40.394993+0000","last_active":"2024-09-19T07:19:40.396408+0000","last_peered":"2024-09-19T07:19:40.396408+0000","last_clean":"2024-09-19T07:19:40.396408+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:19:40.396408+0000","last_undegraded":"2024-09-19T07:19:40.396408+0000","last_fullsized":"2024-09-19T07:19:40.396408+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:30.341693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"24'1","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.431200+0000","last_change":"2024-09-19T07:19:39.418885+0000","last_active":"2024-09-19T07:19:40.431200+0000","last_peered":"2024-09-19T07:19:40.431200+0000","last_clean":"2024-09-19T07:19:40.431200+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:19:40.431200+0000","last_undegraded":"2024-09-19T07:19:40.431200+0000","last_fullsized":"2024-09-19T07:19:40.431200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:35:45.790411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.348259+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:19:40.348259+0000","last_peered":"2024-09-19T07:19:40.348259+0000","last_clean":"2024-09-19T07:19:40.348259+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:19:40.348259+0000","last_undegraded":"2024-09-19T07:19:40.348259+0000","last_fullsized":"2024-09-19T07:19:40.348259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.346923+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:19:40.346923+0000","last_peered":"2024-09-19T07:19:40.346923+0000","last_clean":"2024-09-19T07:19:40.346923+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:19:40.346923+0000","last_undegraded":"2024-09-19T07:19:40.346923+0000","last_fullsized":"2024-09-19T07:19:40.346923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.379767+0000","last_change":"2024-09-19T07:19:40.374048+0000","last_active":"2024-09-19T07:19:40.379767+0000","last_peered":"2024-09-19T07:19:40.379767+0000","last_clean":"2024-09-19T07:19:40.379767+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:19:40.379767+0000","last_undegraded":"2024-09-19T07:19:40.379767+0000","last_fullsized":"2024-09-19T07:19:40.379767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:35:18.576120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"24'4","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.430201+0000","last_change":"2024-09-19T07:19:39.432426+0000","last_active":"2024-09-19T07:19:40.430201+0000","last_peered":"2024-09-19T07:19:40.430201+0000","last_clean":"2024-09-19T07:19:40.430201+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:19:40.430201+0000","last_undegraded":"2024-09-19T07:19:40.430201+0000","last_fullsized":"2024-09-19T07:19:40.430201+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.339727+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:19:40.339727+0000","last_peered":"2024-09-19T07:19:40.339727+0000","last_clean":"2024-09-19T07:19:40.339727+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:19:40.339727+0000","last_undegraded":"2024-09-19T07:19:40.339727+0000","last_fullsized":"2024-09-19T07:19:40.339727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.339280+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:19:40.339280+0000","last_peered":"2024-09-19T07:19:40.339280+0000","last_clean":"2024-09-19T07:19:40.339280+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:19:40.339280+0000","last_undegraded":"2024-09-19T07:19:40.339280+0000","last_fullsized":"2024-09-19T07:19:40.339280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.387760+0000","last_change":"2024-09-19T07:19:40.384934+0000","last_active":"2024-09-19T07:19:40.387760+0000","last_peered":"2024-09-19T07:19:40.387760+0000","last_clean":"2024-09-19T07:19:40.387760+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:19:40.387760+0000","last_undegraded":"2024-09-19T07:19:40.387760+0000","last_fullsized":"2024-09-19T07:19:40.387760+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.345456+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:19:40.345456+0000","last_peered":"2024-09-19T07:19:40.345456+0000","last_clean":"2024-09-19T07:19:40.345456+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:19:40.345456+0000","last_undegraded":"2024-09-19T07:19:40.345456+0000","last_fullsized":"2024-09-19T07:19:40.345456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.369661+0000","last_change":"2024-09-19T07:19:32.289374+0000","last_active":"2024-09-19T07:19:40.369661+0000","last_peered":"2024-09-19T07:19:40.369661+0000","last_clean":"2024-09-19T07:19:40.369661+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:19:40.369661+0000","last_undegraded":"2024-09-19T07:19:40.369661+0000","last_fullsized":"2024-09-19T07:19:40.369661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"24'2","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.431155+0000","last_change":"2024-09-19T07:19:39.410582+0000","last_active":"2024-09-19T07:19:40.431155+0000","last_peered":"2024-09-19T07:19:40.431155+0000","last_clean":"2024-09-19T07:19:40.431155+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:19:40.431155+0000","last_undegraded":"2024-09-19T07:19:40.431155+0000","last_fullsized":"2024-09-19T07:19:40.431155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:41:40.592907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.395801+0000","last_change":"2024-09-19T07:19:40.393950+0000","last_active":"2024-09-19T07:19:40.395801+0000","last_peered":"2024-09-19T07:19:40.395801+0000","last_clean":"2024-09-19T07:19:40.395801+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:19:40.395801+0000","last_undegraded":"2024-09-19T07:19:40.395801+0000","last_fullsized":"2024-09-19T07:19:40.395801+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340914+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:19:40.340914+0000","last_peered":"2024-09-19T07:19:40.340914+0000","last_clean":"2024-09-19T07:19:40.340914+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:19:40.340914+0000","last_undegraded":"2024-09-19T07:19:40.340914+0000","last_fullsized":"2024-09-19T07:19:40.340914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.355065+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:19:40.355065+0000","last_peered":"2024-09-19T07:19:40.355065+0000","last_clean":"2024-09-19T07:19:40.355065+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:19:40.355065+0000","last_undegraded":"2024-09-19T07:19:40.355065+0000","last_fullsized":"2024-09-19T07:19:40.355065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"24'2","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.433062+0000","last_change":"2024-09-19T07:19:39.410662+0000","last_active":"2024-09-19T07:19:40.433062+0000","last_peered":"2024-09-19T07:19:40.433062+0000","last_clean":"2024-09-19T07:19:40.433062+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:40.433062+0000","last_undegraded":"2024-09-19T07:19:40.433062+0000","last_fullsized":"2024-09-19T07:19:40.433062+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:19.196074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392075+0000","last_change":"2024-09-19T07:19:40.385581+0000","last_active":"2024-09-19T07:19:40.392075+0000","last_peered":"2024-09-19T07:19:40.392075+0000","last_clean":"2024-09-19T07:19:40.392075+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:19:40.392075+0000","last_undegraded":"2024-09-19T07:19:40.392075+0000","last_fullsized":"2024-09-19T07:19:40.392075+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342259+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:19:40.342259+0000","last_peered":"2024-09-19T07:19:40.342259+0000","last_clean":"2024-09-19T07:19:40.342259+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:19:40.342259+0000","last_undegraded":"2024-09-19T07:19:40.342259+0000","last_fullsized":"2024-09-19T07:19:40.342259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.380963+0000","last_change":"2024-09-19T07:19:40.377608+0000","last_active":"2024-09-19T07:19:40.380963+0000","last_peered":"2024-09-19T07:19:40.380963+0000","last_clean":"2024-09-19T07:19:40.380963+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:19:40.380963+0000","last_undegraded":"2024-09-19T07:19:40.380963+0000","last_fullsized":"2024-09-19T07:19:40.380963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356711+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:19:40.356711+0000","last_peered":"2024-09-19T07:19:40.356711+0000","last_clean":"2024-09-19T07:19:40.356711+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:19:40.356711+0000","last_undegraded":"2024-09-19T07:19:40.356711+0000","last_fullsized":"2024-09-19T07:19:40.356711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393077+0000","last_change":"2024-09-19T07:19:40.381262+0000","last_active":"2024-09-19T07:19:40.393077+0000","last_peered":"2024-09-19T07:19:40.393077+0000","last_clean":"2024-09-19T07:19:40.393077+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:19:40.393077+0000","last_undegraded":"2024-09-19T07:19:40.393077+0000","last_fullsized":"2024-09-19T07:19:40.393077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:35:35.760631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340773+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:19:40.340773+0000","last_peered":"2024-09-19T07:19:40.340773+0000","last_clean":"2024-09-19T07:19:40.340773+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:19:40.340773+0000","last_undegraded":"2024-09-19T07:19:40.340773+0000","last_fullsized":"2024-09-19T07:19:40.340773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.390804+0000","last_change":"2024-09-19T07:19:40.386432+0000","last_active":"2024-09-19T07:19:40.390804+0000","last_peered":"2024-09-19T07:19:40.390804+0000","last_clean":"2024-09-19T07:19:40.390804+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:19:40.390804+0000","last_undegraded":"2024-09-19T07:19:40.390804+0000","last_fullsized":"2024-09-19T07:19:40.390804+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.369564+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:19:40.369564+0000","last_peered":"2024-09-19T07:19:40.369564+0000","last_clean":"2024-09-19T07:19:40.369564+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:19:40.369564+0000","last_undegraded":"2024-09-19T07:19:40.369564+0000","last_fullsized":"2024-09-19T07:19:40.369564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.379966+0000","last_change":"2024-09-19T07:19:40.378630+0000","last_active":"2024-09-19T07:19:40.379966+0000","last_peered":"2024-09-19T07:19:40.379966+0000","last_clean":"2024-09-19T07:19:40.379966+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:19:40.379966+0000","last_undegraded":"2024-09-19T07:19:40.379966+0000","last_fullsized":"2024-09-19T07:19:40.379966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342627+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:19:40.342627+0000","last_peered":"2024-09-19T07:19:40.342627+0000","last_clean":"2024-09-19T07:19:40.342627+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:19:40.342627+0000","last_undegraded":"2024-09-19T07:19:40.342627+0000","last_fullsized":"2024-09-19T07:19:40.342627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.384631+0000","last_change":"2024-09-19T07:19:40.381365+0000","last_active":"2024-09-19T07:19:40.384631+0000","last_peered":"2024-09-19T07:19:40.384631+0000","last_clean":"2024-09-19T07:19:40.384631+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:19:40.384631+0000","last_undegraded":"2024-09-19T07:19:40.384631+0000","last_fullsized":"2024-09-19T07:19:40.384631+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:46.389885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.353296+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:19:40.353296+0000","last_peered":"2024-09-19T07:19:40.353296+0000","last_clean":"2024-09-19T07:19:40.353296+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:19:40.353296+0000","last_undegraded":"2024-09-19T07:19:40.353296+0000","last_fullsized":"2024-09-19T07:19:40.353296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392393+0000","last_change":"2024-09-19T07:19:40.387116+0000","last_active":"2024-09-19T07:19:40.392393+0000","last_peered":"2024-09-19T07:19:40.392393+0000","last_clean":"2024-09-19T07:19:40.392393+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:19:40.392393+0000","last_undegraded":"2024-09-19T07:19:40.392393+0000","last_fullsized":"2024-09-19T07:19:40.392393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.344093+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:19:40.344093+0000","last_peered":"2024-09-19T07:19:40.344093+0000","last_clean":"2024-09-19T07:19:40.344093+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:19:40.344093+0000","last_undegraded":"2024-09-19T07:19:40.344093+0000","last_fullsized":"2024-09-19T07:19:40.344093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.365871+0000","last_change":"2024-09-19T07:19:40.357629+0000","last_active":"2024-09-19T07:19:40.365871+0000","last_peered":"2024-09-19T07:19:40.365871+0000","last_clean":"2024-09-19T07:19:40.365871+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:19:40.365871+0000","last_undegraded":"2024-09-19T07:19:40.365871+0000","last_fullsized":"2024-09-19T07:19:40.365871+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:28.893439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342355+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:19:40.342355+0000","last_peered":"2024-09-19T07:19:40.342355+0000","last_clean":"2024-09-19T07:19:40.342355+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:19:40.342355+0000","last_undegraded":"2024-09-19T07:19:40.342355+0000","last_fullsized":"2024-09-19T07:19:40.342355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393476+0000","last_change":"2024-09-19T07:19:40.388597+0000","last_active":"2024-09-19T07:19:40.393476+0000","last_peered":"2024-09-19T07:19:40.393476+0000","last_clean":"2024-09-19T07:19:40.393476+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:19:40.393476+0000","last_undegraded":"2024-09-19T07:19:40.393476+0000","last_fullsized":"2024-09-19T07:19:40.393476+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.346684+0000","last_change":"2024-09-19T07:19:32.273635+0000","last_active":"2024-09-19T07:19:40.346684+0000","last_peered":"2024-09-19T07:19:40.346684+0000","last_clean":"2024-09-19T07:19:40.346684+0000","last_became_active":"2024-09-19T07:19:32.271748+0000","last_became_peered":"2024-09-19T07:19:32.271748+0000","last_unstale":"2024-09-19T07:19:40.346684+0000","last_undegraded":"2024-09-19T07:19:40.346684+0000","last_fullsized":"2024-09-19T07:19:40.346684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:49:41.724626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.386734+0000","last_change":"2024-09-19T07:19:40.383779+0000","last_active":"2024-09-19T07:19:40.386734+0000","last_peered":"2024-09-19T07:19:40.386734+0000","last_clean":"2024-09-19T07:19:40.386734+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:19:40.386734+0000","last_undegraded":"2024-09-19T07:19:40.386734+0000","last_fullsized":"2024-09-19T07:19:40.386734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.338969+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:19:40.338969+0000","last_peered":"2024-09-19T07:19:40.338969+0000","last_clean":"2024-09-19T07:19:40.338969+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:19:40.338969+0000","last_undegraded":"2024-09-19T07:19:40.338969+0000","last_fullsized":"2024-09-19T07:19:40.338969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393368+0000","last_change":"2024-09-19T07:19:40.376342+0000","last_active":"2024-09-19T07:19:40.393368+0000","last_peered":"2024-09-19T07:19:40.393368+0000","last_clean":"2024-09-19T07:19:40.393368+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:19:40.393368+0000","last_undegraded":"2024-09-19T07:19:40.393368+0000","last_fullsized":"2024-09-19T07:19:40.393368+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:37.532894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340727+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:19:40.340727+0000","last_peered":"2024-09-19T07:19:40.340727+0000","last_clean":"2024-09-19T07:19:40.340727+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:19:40.340727+0000","last_undegraded":"2024-09-19T07:19:40.340727+0000","last_fullsized":"2024-09-19T07:19:40.340727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.368358+0000","last_change":"2024-09-19T07:19:40.361504+0000","last_active":"2024-09-19T07:19:40.368358+0000","last_peered":"2024-09-19T07:19:40.368358+0000","last_clean":"2024-09-19T07:19:40.368358+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:19:40.368358+0000","last_undegraded":"2024-09-19T07:19:40.368358+0000","last_fullsized":"2024-09-19T07:19:40.368358+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.369423+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:19:40.369423+0000","last_peered":"2024-09-19T07:19:40.369423+0000","last_clean":"2024-09-19T07:19:40.369423+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:19:40.369423+0000","last_undegraded":"2024-09-19T07:19:40.369423+0000","last_fullsized":"2024-09-19T07:19:40.369423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:19.033192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.394565+0000","last_change":"2024-09-19T07:19:40.390205+0000","last_active":"2024-09-19T07:19:40.394565+0000","last_peered":"2024-09-19T07:19:40.394565+0000","last_clean":"2024-09-19T07:19:40.394565+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:19:40.394565+0000","last_undegraded":"2024-09-19T07:19:40.394565+0000","last_fullsized":"2024-09-19T07:19:40.394565+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342913+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:19:40.342913+0000","last_peered":"2024-09-19T07:19:40.342913+0000","last_clean":"2024-09-19T07:19:40.342913+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:19:40.342913+0000","last_undegraded":"2024-09-19T07:19:40.342913+0000","last_fullsized":"2024-09-19T07:19:40.342913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.394504+0000","last_change":"2024-09-19T07:19:40.390559+0000","last_active":"2024-09-19T07:19:40.394504+0000","last_peered":"2024-09-19T07:19:40.394504+0000","last_clean":"2024-09-19T07:19:40.394504+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:19:40.394504+0000","last_undegraded":"2024-09-19T07:19:40.394504+0000","last_fullsized":"2024-09-19T07:19:40.394504+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.368038+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:19:40.368038+0000","last_peered":"2024-09-19T07:19:40.368038+0000","last_clean":"2024-09-19T07:19:40.368038+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:19:40.368038+0000","last_undegraded":"2024-09-19T07:19:40.368038+0000","last_fullsized":"2024-09-19T07:19:40.368038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.364863+0000","last_change":"2024-09-19T07:19:40.358433+0000","last_active":"2024-09-19T07:19:40.364863+0000","last_peered":"2024-09-19T07:19:40.364863+0000","last_clean":"2024-09-19T07:19:40.364863+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:19:40.364863+0000","last_undegraded":"2024-09-19T07:19:40.364863+0000","last_fullsized":"2024-09-19T07:19:40.364863+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:34:05.875388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.358717+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:19:40.358717+0000","last_peered":"2024-09-19T07:19:40.358717+0000","last_clean":"2024-09-19T07:19:40.358717+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:19:40.358717+0000","last_undegraded":"2024-09-19T07:19:40.358717+0000","last_fullsized":"2024-09-19T07:19:40.358717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:51.710997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.379817+0000","last_change":"2024-09-19T07:19:40.374374+0000","last_active":"2024-09-19T07:19:40.379817+0000","last_peered":"2024-09-19T07:19:40.379817+0000","last_clean":"2024-09-19T07:19:40.379817+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:19:40.379817+0000","last_undegraded":"2024-09-19T07:19:40.379817+0000","last_fullsized":"2024-09-19T07:19:40.379817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:19.152425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342559+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:19:40.342559+0000","last_peered":"2024-09-19T07:19:40.342559+0000","last_clean":"2024-09-19T07:19:40.342559+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:19:40.342559+0000","last_undegraded":"2024-09-19T07:19:40.342559+0000","last_fullsized":"2024-09-19T07:19:40.342559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.390806+0000","last_change":"2024-09-19T07:19:40.387341+0000","last_active":"2024-09-19T07:19:40.390806+0000","last_peered":"2024-09-19T07:19:40.390806+0000","last_clean":"2024-09-19T07:19:40.390806+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:19:40.390806+0000","last_undegraded":"2024-09-19T07:19:40.390806+0000","last_fullsized":"2024-09-19T07:19:40.390806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.346439+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:19:40.346439+0000","last_peered":"2024-09-19T07:19:40.346439+0000","last_clean":"2024-09-19T07:19:40.346439+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:19:40.346439+0000","last_undegraded":"2024-09-19T07:19:40.346439+0000","last_fullsized":"2024-09-19T07:19:40.346439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.390692+0000","last_change":"2024-09-19T07:19:40.388484+0000","last_active":"2024-09-19T07:19:40.390692+0000","last_peered":"2024-09-19T07:19:40.390692+0000","last_clean":"2024-09-19T07:19:40.390692+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:19:40.390692+0000","last_undegraded":"2024-09-19T07:19:40.390692+0000","last_fullsized":"2024-09-19T07:19:40.390692+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:30.341693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.348499+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:19:40.348499+0000","last_peered":"2024-09-19T07:19:40.348499+0000","last_clean":"2024-09-19T07:19:40.348499+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:19:40.348499+0000","last_undegraded":"2024-09-19T07:19:40.348499+0000","last_fullsized":"2024-09-19T07:19:40.348499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.375037+0000","last_change":"2024-09-19T07:19:40.367578+0000","last_active":"2024-09-19T07:19:40.375037+0000","last_peered":"2024-09-19T07:19:40.375037+0000","last_clean":"2024-09-19T07:19:40.375037+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:19:40.375037+0000","last_undegraded":"2024-09-19T07:19:40.375037+0000","last_fullsized":"2024-09-19T07:19:40.375037+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:40.084827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.360136+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:19:40.360136+0000","last_peered":"2024-09-19T07:19:40.360136+0000","last_clean":"2024-09-19T07:19:40.360136+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:19:40.360136+0000","last_undegraded":"2024-09-19T07:19:40.360136+0000","last_fullsized":"2024-09-19T07:19:40.360136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:43.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.387682+0000","last_change":"2024-09-19T07:19:40.385641+0000","last_active":"2024-09-19T07:19:40.387682+0000","last_peered":"2024-09-19T07:19:40.387682+0000","last_clean":"2024-09-19T07:19:40.387682+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:19:40.387682+0000","last_undegraded":"2024-09-19T07:19:40.387682+0000","last_fullsized":"2024-09-19T07:19:40.387682+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340367+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:19:40.340367+0000","last_peered":"2024-09-19T07:19:40.340367+0000","last_clean":"2024-09-19T07:19:40.340367+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:19:40.340367+0000","last_undegraded":"2024-09-19T07:19:40.340367+0000","last_fullsized":"2024-09-19T07:19:40.340367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.375372+0000","last_change":"2024-09-19T07:19:40.367101+0000","last_active":"2024-09-19T07:19:40.375372+0000","last_peered":"2024-09-19T07:19:40.375372+0000","last_clean":"2024-09-19T07:19:40.375372+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:19:40.375372+0000","last_undegraded":"2024-09-19T07:19:40.375372+0000","last_fullsized":"2024-09-19T07:19:40.375372+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.362347+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:19:40.362347+0000","last_peered":"2024-09-19T07:19:40.362347+0000","last_clean":"2024-09-19T07:19:40.362347+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:19:40.362347+0000","last_undegraded":"2024-09-19T07:19:40.362347+0000","last_fullsized":"2024-09-19T07:19:40.362347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393667+0000","last_change":"2024-09-19T07:19:40.384212+0000","last_active":"2024-09-19T07:19:40.393667+0000","last_peered":"2024-09-19T07:19:40.393667+0000","last_clean":"2024-09-19T07:19:40.393667+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:19:40.393667+0000","last_undegraded":"2024-09-19T07:19:40.393667+0000","last_fullsized":"2024-09-19T07:19:40.393667+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:36.886449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.343312+0000","last_change":"2024-09-19T07:19:32.299956+0000","last_active":"2024-09-19T07:19:40.343312+0000","last_peered":"2024-09-19T07:19:40.343312+0000","last_clean":"2024-09-19T07:19:40.343312+0000","last_became_active":"2024-09-19T07:19:32.299168+0000","last_became_peered":"2024-09-19T07:19:32.299168+0000","last_unstale":"2024-09-19T07:19:40.343312+0000","last_undegraded":"2024-09-19T07:19:40.343312+0000","last_fullsized":"2024-09-19T07:19:40.343312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392829+0000","last_change":"2024-09-19T07:19:40.387401+0000","last_active":"2024-09-19T07:19:40.392829+0000","last_peered":"2024-09-19T07:19:40.392829+0000","last_clean":"2024-09-19T07:19:40.392829+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:19:40.392829+0000","last_undegraded":"2024-09-19T07:19:40.392829+0000","last_fullsized":"2024-09-19T07:19:40.392829+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.343788+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:19:40.343788+0000","last_peered":"2024-09-19T07:19:40.343788+0000","last_clean":"2024-09-19T07:19:40.343788+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:19:40.343788+0000","last_undegraded":"2024-09-19T07:19:40.343788+0000","last_fullsized":"2024-09-19T07:19:40.343788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.365845+0000","last_change":"2024-09-19T07:19:40.361430+0000","last_active":"2024-09-19T07:19:40.365845+0000","last_peered":"2024-09-19T07:19:40.365845+0000","last_clean":"2024-09-19T07:19:40.365845+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:19:40.365845+0000","last_undegraded":"2024-09-19T07:19:40.365845+0000","last_fullsized":"2024-09-19T07:19:40.365845+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:39.665475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370978+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:19:40.370978+0000","last_peered":"2024-09-19T07:19:40.370978+0000","last_clean":"2024-09-19T07:19:40.370978+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:19:40.370978+0000","last_undegraded":"2024-09-19T07:19:40.370978+0000","last_fullsized":"2024-09-19T07:19:40.370978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392029+0000","last_change":"2024-09-19T07:19:40.378172+0000","last_active":"2024-09-19T07:19:40.392029+0000","last_peered":"2024-09-19T07:19:40.392029+0000","last_clean":"2024-09-19T07:19:40.392029+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:19:40.392029+0000","last_undegraded":"2024-09-19T07:19:40.392029+0000","last_fullsized":"2024-09-19T07:19:40.392029+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.339480+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:19:40.339480+0000","last_peered":"2024-09-19T07:19:40.339480+0000","last_clean":"2024-09-19T07:19:40.339480+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:19:40.339480+0000","last_undegraded":"2024-09-19T07:19:40.339480+0000","last_fullsized":"2024-09-19T07:19:40.339480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.378910+0000","last_change":"2024-09-19T07:19:40.373909+0000","last_active":"2024-09-19T07:19:40.378910+0000","last_peered":"2024-09-19T07:19:40.378910+0000","last_clean":"2024-09-19T07:19:40.378910+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:19:40.378910+0000","last_undegraded":"2024-09-19T07:19:40.378910+0000","last_fullsized":"2024-09-19T07:19:40.378910+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:22.017682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.362811+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:19:40.362811+0000","last_peered":"2024-09-19T07:19:40.362811+0000","last_clean":"2024-09-19T07:19:40.362811+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:19:40.362811+0000","last_undegraded":"2024-09-19T07:19:40.362811+0000","last_fullsized":"2024-09-19T07:19:40.362811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.397128+0000","last_change":"2024-09-19T07:19:40.396146+0000","last_active":"2024-09-19T07:19:40.397128+0000","last_peered":"2024-09-19T07:19:40.397128+0000","last_clean":"2024-09-19T07:19:40.397128+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:19:40.397128+0000","last_undegraded":"2024-09-19T07:19:40.397128+0000","last_fullsized":"2024-09-19T07:19:40.397128+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340526+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:19:40.340526+0000","last_peered":"2024-09-19T07:19:40.340526+0000","last_clean":"2024-09-19T07:19:40.340526+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:19:40.340526+0000","last_undegraded":"2024-09-19T07:19:40.340526+0000","last_fullsized":"2024-09-19T07:19:40.340526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.380650+0000","last_change":"2024-09-19T07:19:40.373552+0000","last_active":"2024-09-19T07:19:40.380650+0000","last_peered":"2024-09-19T07:19:40.380650+0000","last_clean":"2024-09-19T07:19:40.380650+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:19:40.380650+0000","last_undegraded":"2024-09-19T07:19:40.380650+0000","last_fullsized":"2024-09-19T07:19:40.380650+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:35:11.516938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.359124+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:19:40.359124+0000","last_peered":"2024-09-19T07:19:40.359124+0000","last_clean":"2024-09-19T07:19:40.359124+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:19:40.359124+0000","last_undegraded":"2024-09-19T07:19:40.359124+0000","last_fullsized":"2024-09-19T07:19:40.359124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.360300+0000","last_change":"2024-09-19T07:19:40.356989+0000","last_active":"2024-09-19T07:19:40.360300+0000","last_peered":"2024-09-19T07:19:40.360300+0000","last_clean":"2024-09-19T07:19:40.360300+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:19:40.360300+0000","last_undegraded":"2024-09-19T07:19:40.360300+0000","last_fullsized":"2024-09-19T07:19:40.360300+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.347810+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:19:40.347810+0000","last_peered":"2024-09-19T07:19:40.347810+0000","last_clean":"2024-09-19T07:19:40.347810+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:19:40.347810+0000","last_undegraded":"2024-09-19T07:19:40.347810+0000","last_fullsized":"2024-09-19T07:19:40.347810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393286+0000","last_change":"2024-09-19T07:19:40.376487+0000","last_active":"2024-09-19T07:19:40.393286+0000","last_peered":"2024-09-19T07:19:40.393286+0000","last_clean":"2024-09-19T07:19:40.393286+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:19:40.393286+0000","last_undegraded":"2024-09-19T07:19:40.393286+0000","last_fullsized":"2024-09-19T07:19:40.393286+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:45.812485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.368708+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:19:40.368708+0000","last_peered":"2024-09-19T07:19:40.368708+0000","last_clean":"2024-09-19T07:19:40.368708+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:19:40.368708+0000","last_undegraded":"2024-09-19T07:19:40.368708+0000","last_fullsized":"2024-09-19T07:19:40.368708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:25.667168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.386792+0000","last_change":"2024-09-19T07:19:40.377604+0000","last_active":"2024-09-19T07:19:40.386792+0000","last_peered":"2024-09-19T07:19:40.386792+0000","last_clean":"2024-09-19T07:19:40.386792+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:19:40.386792+0000","last_undegraded":"2024-09-19T07:19:40.386792+0000","last_fullsized":"2024-09-19T07:19:40.386792+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356201+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:19:40.356201+0000","last_peered":"2024-09-19T07:19:40.356201+0000","last_clean":"2024-09-19T07:19:40.356201+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:19:40.356201+0000","last_undegraded":"2024-09-19T07:19:40.356201+0000","last_fullsized":"2024-09-19T07:19:40.356201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.379393+0000","last_change":"2024-09-19T07:19:40.377580+0000","last_active":"2024-09-19T07:19:40.379393+0000","last_peered":"2024-09-19T07:19:40.379393+0000","last_clean":"2024-09-19T07:19:40.379393+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:19:40.379393+0000","last_undegraded":"2024-09-19T07:19:40.379393+0000","last_fullsized":"2024-09-19T07:19:40.379393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370728+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:19:40.370728+0000","last_peered":"2024-09-19T07:19:40.370728+0000","last_clean":"2024-09-19T07:19:40.370728+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:19:40.370728+0000","last_undegraded":"2024-09-19T07:19:40.370728+0000","last_fullsized":"2024-09-19T07:19:40.370728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":847872,"data_stored":686333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":97611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":87896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":87932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":89418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":95453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":97177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":843776,"data_stored":678812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:19:45.457 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:19:45.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80db06b640 1 Processor -- start 2024-09-19T07:19:45.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80db06b640 1 -- start start 2024-09-19T07:19:45.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80db06b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f80d41545e0 0x7f80d406e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80db06b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d4154a70 con 0x7f80d40c28e0 2024-09-19T07:19:45.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80db06b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d40a9e60 con 0x7f80d4071100 2024-09-19T07:19:45.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80db06b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f80d4071b50 con 0x7f80d41545e0 2024-09-19T07:19:45.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80d9608640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f80d41545e0 0x7f80d406e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80d3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f80d40c28e0 0x7f80d406dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51490/0 (socket says 172.21.15.83:51490) 2024-09-19T07:19:45.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80d9608640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f80d41545e0 0x7f80d406e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40448/0 (socket says 172.21.15.83:40448) 2024-09-19T07:19:45.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80d3fff640 1 -- 172.21.15.83:0/3413647225 learned_addr learned my addr 172.21.15.83:0/3413647225 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:45.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80d9608640 1 -- 172.21.15.83:0/3413647225 >> v1:172.21.15.184:6789/0 conn(0x7f80d4071100 legacy=0x7f80d4150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:45.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80d9608640 1 -- 172.21.15.83:0/3413647225 >> v1:172.21.15.83:6789/0 conn(0x7f80d40c28e0 legacy=0x7f80d406dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80d9608640 1 -- 172.21.15.83:0/3413647225 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d40aa140 con 0x7f80d41545e0 2024-09-19T07:19:45.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.611+0000 7f80d37fe640 1 -- 172.21.15.83:0/3413647225 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2761305896 0 0) 0x7f80d4154a70 con 0x7f80d40c28e0 2024-09-19T07:19:45.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80d9608640 1 --2- 172.21.15.83:0/3413647225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80d41545e0 0x7f80d406e5b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f80cc009d90 tx=0x7f80cc01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c186ddc4f47aff4c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:45.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80d37fe640 1 -- 172.21.15.83:0/3413647225 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc0b2070 con 0x7f80d41545e0 2024-09-19T07:19:45.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80d37fe640 1 -- 172.21.15.83:0/3413647225 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80cc0a5850 con 0x7f80d41545e0 2024-09-19T07:19:45.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80d37fe640 1 -- 172.21.15.83:0/3413647225 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc0a5c20 con 0x7f80d41545e0 2024-09-19T07:19:45.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 -- 172.21.15.83:0/3413647225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80d41545e0 msgr2=0x7f80d406e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 --2- 172.21.15.83:0/3413647225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80d41545e0 0x7f80d406e5b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f80cc009d90 tx=0x7f80cc01c8a0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 -- 172.21.15.83:0/3413647225 shutdown_connections 2024-09-19T07:19:45.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 --2- 172.21.15.83:0/3413647225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80d41545e0 0x7f80d406e5b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 -- 172.21.15.83:0/3413647225 >> 172.21.15.83:0/3413647225 conn(0x7f80d406bc80 msgr2=0x7f80d406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:45.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 -- 172.21.15.83:0/3413647225 shutdown_connections 2024-09-19T07:19:45.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 -- 172.21.15.83:0/3413647225 wait complete. 2024-09-19T07:19:45.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 Processor -- start 2024-09-19T07:19:45.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 -- start start 2024-09-19T07:19:45.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80d4071100 0x7f80d41f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80d40c28e0 0x7f80d4151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80d41545e0 0x7f80d4152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f80d41517d0 con 0x7f80d41545e0 2024-09-19T07:19:45.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f80d4151650 con 0x7f80d40c28e0 2024-09-19T07:19:45.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80db06b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f80d40714e0 con 0x7f80d4071100 2024-09-19T07:19:45.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80d9608640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80d41545e0 0x7f80d4152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80d9608640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80d41545e0 0x7f80d4152410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40458/0 (socket says 172.21.15.83:40458) 2024-09-19T07:19:45.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80d9608640 1 -- 172.21.15.83:0/1563006418 learned_addr learned my addr 172.21.15.83:0/1563006418 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:45.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80d9608640 1 -- 172.21.15.83:0/1563006418 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80d4071100 msgr2=0x7f80d41f20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80d9608640 1 --2- 172.21.15.83:0/1563006418 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80d4071100 0x7f80d41f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80d9608640 1 -- 172.21.15.83:0/1563006418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80d40c28e0 msgr2=0x7f80d4151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80d9608640 1 --2- 172.21.15.83:0/1563006418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80d40c28e0 0x7f80d4151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.615+0000 7f80d9608640 1 -- 172.21.15.83:0/1563006418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d414ea70 con 0x7f80d41545e0 2024-09-19T07:19:45.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.619+0000 7f80d9608640 1 --2- 172.21.15.83:0/1563006418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80d41545e0 0x7f80d4152410 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0b0040 tx=0x7f80cc019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:45.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.619+0000 7f80d17fa640 1 -- 172.21.15.83:0/1563006418 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc0bd020 con 0x7f80d41545e0 2024-09-19T07:19:45.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.619+0000 7f80d17fa640 1 -- 172.21.15.83:0/1563006418 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80cc0ad6c0 con 0x7f80d41545e0 2024-09-19T07:19:45.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.619+0000 7f80db06b640 1 -- 172.21.15.83:0/1563006418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d414ed10 con 0x7f80d41545e0 2024-09-19T07:19:45.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.619+0000 7f80d17fa640 1 -- 172.21.15.83:0/1563006418 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc0b2040 con 0x7f80d41545e0 2024-09-19T07:19:45.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.619+0000 7f80db06b640 1 -- 172.21.15.83:0/1563006418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d414f2d0 con 0x7f80d41545e0 2024-09-19T07:19:45.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.619+0000 7f80db06b640 1 -- 172.21.15.83:0/1563006418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80d4150f40 con 0x7f80d41545e0 2024-09-19T07:19:45.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.619+0000 7f80d17fa640 1 -- 172.21.15.83:0/1563006418 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f80cc0c0020 con 0x7f80d41545e0 2024-09-19T07:19:45.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.619+0000 7f80d17fa640 1 --2- 172.21.15.83:0/1563006418 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80b80678c0 0x7f80b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.619+0000 7f80d17fa640 1 -- 172.21.15.83:0/1563006418 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f80cc121080 con 0x7f80d41545e0 2024-09-19T07:19:45.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.619+0000 7f80d8e07640 1 --2- 172.21.15.83:0/1563006418 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80b80678c0 0x7f80b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.619+0000 7f80d8e07640 1 --2- 172.21.15.83:0/1563006418 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80b80678c0 0x7f80b8069d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009670 tx=0x7f80c8006bd0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:45.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.623+0000 7f80d17fa640 1 -- 172.21.15.83:0/1563006418 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f80cc0c7030 con 0x7f80d41545e0 2024-09-19T07:19:45.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.771+0000 7f80db06b640 1 -- 172.21.15.83:0/1563006418 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f80d40c4f30 con 0x7f80b80678c0 2024-09-19T07:19:45.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.783+0000 7f80d17fa640 1 -- 172.21.15.83:0/1563006418 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490348 (secure 0 0 0) 0x7f80d40c4f30 con 0x7f80b80678c0 2024-09-19T07:19:45.784 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:45.790 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:19:45.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.787+0000 7f80db06b640 1 -- 172.21.15.83:0/1563006418 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80b80678c0 msgr2=0x7f80b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.787+0000 7f80db06b640 1 --2- 172.21.15.83:0/1563006418 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80b80678c0 0x7f80b8069d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009670 tx=0x7f80c8006bd0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.787+0000 7f80db06b640 1 -- 172.21.15.83:0/1563006418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80d41545e0 msgr2=0x7f80d4152410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.787+0000 7f80db06b640 1 --2- 172.21.15.83:0/1563006418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80d41545e0 0x7f80d4152410 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0b0040 tx=0x7f80cc019640 comp rx=0 tx=0).stop 2024-09-19T07:19:45.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.787+0000 7f80db06b640 1 -- 172.21.15.83:0/1563006418 shutdown_connections 2024-09-19T07:19:45.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.787+0000 7f80db06b640 1 --2- 172.21.15.83:0/1563006418 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80b80678c0 0x7f80b8069d80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.787+0000 7f80db06b640 1 --2- 172.21.15.83:0/1563006418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80d41545e0 0x7f80d4152410 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.787+0000 7f80db06b640 1 --2- 172.21.15.83:0/1563006418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80d40c28e0 0x7f80d4151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.787+0000 7f80db06b640 1 --2- 172.21.15.83:0/1563006418 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80d4071100 0x7f80d41f20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.787+0000 7f80db06b640 1 -- 172.21.15.83:0/1563006418 >> 172.21.15.83:0/1563006418 conn(0x7f80d406bc80 msgr2=0x7f80d406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:45.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.787+0000 7f80db06b640 1 -- 172.21.15.83:0/1563006418 shutdown_connections 2024-09-19T07:19:45.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.787+0000 7f80db06b640 1 -- 172.21.15.83:0/1563006418 wait complete. 2024-09-19T07:19:45.804 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-19T07:19:44.814241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":248,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":304,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223556,"kb_used_data":3156,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751164,"statfs":{"total":773094113280,"available":772865191936,"internally_reserved":0,"allocated":3231744,"data_stored":1920632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001481"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.382852+0000","last_change":"2024-09-19T07:19:40.358457+0000","last_active":"2024-09-19T07:19:40.382852+0000","last_peered":"2024-09-19T07:19:40.382852+0000","last_clean":"2024-09-19T07:19:40.382852+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:19:40.382852+0000","last_undegraded":"2024-09-19T07:19:40.382852+0000","last_fullsized":"2024-09-19T07:19:40.382852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:28.893439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370097+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:19:40.370097+0000","last_peered":"2024-09-19T07:19:40.370097+0000","last_clean":"2024-09-19T07:19:40.370097+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:19:40.370097+0000","last_undegraded":"2024-09-19T07:19:40.370097+0000","last_fullsized":"2024-09-19T07:19:40.370097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:57.799442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.343030+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:19:40.343030+0000","last_peered":"2024-09-19T07:19:40.343030+0000","last_clean":"2024-09-19T07:19:40.343030+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:19:40.343030+0000","last_undegraded":"2024-09-19T07:19:40.343030+0000","last_fullsized":"2024-09-19T07:19:40.343030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.339006+0000","last_change":"2024-09-19T07:19:39.401621+0000","last_active":"2024-09-19T07:19:40.339006+0000","last_peered":"2024-09-19T07:19:40.339006+0000","last_clean":"2024-09-19T07:19:40.339006+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:19:40.339006+0000","last_undegraded":"2024-09-19T07:19:40.339006+0000","last_fullsized":"2024-09-19T07:19:40.339006+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:32.095381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.380910+0000","last_change":"2024-09-19T07:19:40.366073+0000","last_active":"2024-09-19T07:19:40.380910+0000","last_peered":"2024-09-19T07:19:40.380910+0000","last_clean":"2024-09-19T07:19:40.380910+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:19:40.380910+0000","last_undegraded":"2024-09-19T07:19:40.380910+0000","last_fullsized":"2024-09-19T07:19:40.380910+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.343282+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:19:40.343282+0000","last_peered":"2024-09-19T07:19:40.343282+0000","last_clean":"2024-09-19T07:19:40.343282+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:19:40.343282+0000","last_undegraded":"2024-09-19T07:19:40.343282+0000","last_fullsized":"2024-09-19T07:19:40.343282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.361481+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:19:40.361481+0000","last_peered":"2024-09-19T07:19:40.361481+0000","last_clean":"2024-09-19T07:19:40.361481+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:19:40.361481+0000","last_undegraded":"2024-09-19T07:19:40.361481+0000","last_fullsized":"2024-09-19T07:19:40.361481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:15.036396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.349065+0000","last_change":"2024-09-19T07:19:39.403810+0000","last_active":"2024-09-19T07:19:40.349065+0000","last_peered":"2024-09-19T07:19:40.349065+0000","last_clean":"2024-09-19T07:19:40.349065+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:19:40.349065+0000","last_undegraded":"2024-09-19T07:19:40.349065+0000","last_fullsized":"2024-09-19T07:19:40.349065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:55.714964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392960+0000","last_change":"2024-09-19T07:19:40.390455+0000","last_active":"2024-09-19T07:19:40.392960+0000","last_peered":"2024-09-19T07:19:40.392960+0000","last_clean":"2024-09-19T07:19:40.392960+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:19:40.392960+0000","last_undegraded":"2024-09-19T07:19:40.392960+0000","last_fullsized":"2024-09-19T07:19:40.392960+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.368208+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:19:40.368208+0000","last_peered":"2024-09-19T07:19:40.368208+0000","last_clean":"2024-09-19T07:19:40.368208+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:19:40.368208+0000","last_undegraded":"2024-09-19T07:19:40.368208+0000","last_fullsized":"2024-09-19T07:19:40.368208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.348068+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:19:40.348068+0000","last_peered":"2024-09-19T07:19:40.348068+0000","last_clean":"2024-09-19T07:19:40.348068+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:19:40.348068+0000","last_undegraded":"2024-09-19T07:19:40.348068+0000","last_fullsized":"2024-09-19T07:19:40.348068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"24'1","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.433114+0000","last_change":"2024-09-19T07:19:39.408309+0000","last_active":"2024-09-19T07:19:40.433114+0000","last_peered":"2024-09-19T07:19:40.433114+0000","last_clean":"2024-09-19T07:19:40.433114+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:19:40.433114+0000","last_undegraded":"2024-09-19T07:19:40.433114+0000","last_fullsized":"2024-09-19T07:19:40.433114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:19.396261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.380454+0000","last_change":"2024-09-19T07:19:40.372543+0000","last_active":"2024-09-19T07:19:40.380454+0000","last_peered":"2024-09-19T07:19:40.380454+0000","last_clean":"2024-09-19T07:19:40.380454+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:19:40.380454+0000","last_undegraded":"2024-09-19T07:19:40.380454+0000","last_fullsized":"2024-09-19T07:19:40.380454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:23.474199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.360024+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:19:40.360024+0000","last_peered":"2024-09-19T07:19:40.360024+0000","last_clean":"2024-09-19T07:19:40.360024+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:19:40.360024+0000","last_undegraded":"2024-09-19T07:19:40.360024+0000","last_fullsized":"2024-09-19T07:19:40.360024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.341915+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:19:40.341915+0000","last_peered":"2024-09-19T07:19:40.341915+0000","last_clean":"2024-09-19T07:19:40.341915+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:19:40.341915+0000","last_undegraded":"2024-09-19T07:19:40.341915+0000","last_fullsized":"2024-09-19T07:19:40.341915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"24'1","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.433066+0000","last_change":"2024-09-19T07:19:39.407045+0000","last_active":"2024-09-19T07:19:40.433066+0000","last_peered":"2024-09-19T07:19:40.433066+0000","last_clean":"2024-09-19T07:19:40.433066+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:19:40.433066+0000","last_undegraded":"2024-09-19T07:19:40.433066+0000","last_fullsized":"2024-09-19T07:19:40.433066+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:24.846974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.397498+0000","last_change":"2024-09-19T07:19:40.396520+0000","last_active":"2024-09-19T07:19:40.397498+0000","last_peered":"2024-09-19T07:19:40.397498+0000","last_clean":"2024-09-19T07:19:40.397498+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:19:40.397498+0000","last_undegraded":"2024-09-19T07:19:40.397498+0000","last_fullsized":"2024-09-19T07:19:40.397498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.345192+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:19:40.345192+0000","last_peered":"2024-09-19T07:19:40.345192+0000","last_clean":"2024-09-19T07:19:40.345192+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:19:40.345192+0000","last_undegraded":"2024-09-19T07:19:40.345192+0000","last_fullsized":"2024-09-19T07:19:40.345192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.359616+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:19:40.359616+0000","last_peered":"2024-09-19T07:19:40.359616+0000","last_clean":"2024-09-19T07:19:40.359616+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:19:40.359616+0000","last_undegraded":"2024-09-19T07:19:40.359616+0000","last_fullsized":"2024-09-19T07:19:40.359616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"24'1","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.430561+0000","last_change":"2024-09-19T07:19:39.410500+0000","last_active":"2024-09-19T07:19:40.430561+0000","last_peered":"2024-09-19T07:19:40.430561+0000","last_clean":"2024-09-19T07:19:40.430561+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:19:40.430561+0000","last_undegraded":"2024-09-19T07:19:40.430561+0000","last_fullsized":"2024-09-19T07:19:40.430561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:04.182606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393523+0000","last_change":"2024-09-19T07:19:40.381259+0000","last_active":"2024-09-19T07:19:40.393523+0000","last_peered":"2024-09-19T07:19:40.393523+0000","last_clean":"2024-09-19T07:19:40.393523+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:19:40.393523+0000","last_undegraded":"2024-09-19T07:19:40.393523+0000","last_fullsized":"2024-09-19T07:19:40.393523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:59.407542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340656+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:19:40.340656+0000","last_peered":"2024-09-19T07:19:40.340656+0000","last_clean":"2024-09-19T07:19:40.340656+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:19:40.340656+0000","last_undegraded":"2024-09-19T07:19:40.340656+0000","last_fullsized":"2024-09-19T07:19:40.340656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.357584+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:19:40.357584+0000","last_peered":"2024-09-19T07:19:40.357584+0000","last_clean":"2024-09-19T07:19:40.357584+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:19:40.357584+0000","last_undegraded":"2024-09-19T07:19:40.357584+0000","last_fullsized":"2024-09-19T07:19:40.357584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.361778+0000","last_change":"2024-09-19T07:19:39.412123+0000","last_active":"2024-09-19T07:19:40.361778+0000","last_peered":"2024-09-19T07:19:40.361778+0000","last_clean":"2024-09-19T07:19:40.361778+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:19:40.361778+0000","last_undegraded":"2024-09-19T07:19:40.361778+0000","last_fullsized":"2024-09-19T07:19:40.361778+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.388968+0000","last_change":"2024-09-19T07:19:40.383214+0000","last_active":"2024-09-19T07:19:40.388968+0000","last_peered":"2024-09-19T07:19:40.388968+0000","last_clean":"2024-09-19T07:19:40.388968+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:19:40.388968+0000","last_undegraded":"2024-09-19T07:19:40.388968+0000","last_fullsized":"2024-09-19T07:19:40.388968+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:20.550115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340034+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:19:40.340034+0000","last_peered":"2024-09-19T07:19:40.340034+0000","last_clean":"2024-09-19T07:19:40.340034+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:19:40.340034+0000","last_undegraded":"2024-09-19T07:19:40.340034+0000","last_fullsized":"2024-09-19T07:19:40.340034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.348741+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:19:40.348741+0000","last_peered":"2024-09-19T07:19:40.348741+0000","last_clean":"2024-09-19T07:19:40.348741+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:19:40.348741+0000","last_undegraded":"2024-09-19T07:19:40.348741+0000","last_fullsized":"2024-09-19T07:19:40.348741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"24'1","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.433309+0000","last_change":"2024-09-19T07:19:39.408228+0000","last_active":"2024-09-19T07:19:40.433309+0000","last_peered":"2024-09-19T07:19:40.433309+0000","last_clean":"2024-09-19T07:19:40.433309+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:19:40.433309+0000","last_undegraded":"2024-09-19T07:19:40.433309+0000","last_fullsized":"2024-09-19T07:19:40.433309+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.396717+0000","last_change":"2024-09-19T07:19:40.395566+0000","last_active":"2024-09-19T07:19:40.396717+0000","last_peered":"2024-09-19T07:19:40.396717+0000","last_clean":"2024-09-19T07:19:40.396717+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:19:40.396717+0000","last_undegraded":"2024-09-19T07:19:40.396717+0000","last_fullsized":"2024-09-19T07:19:40.396717+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:04:27.088530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.359402+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:19:40.359402+0000","last_peered":"2024-09-19T07:19:40.359402+0000","last_clean":"2024-09-19T07:19:40.359402+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:19:40.359402+0000","last_undegraded":"2024-09-19T07:19:40.359402+0000","last_fullsized":"2024-09-19T07:19:40.359402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.354653+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:19:40.354653+0000","last_peered":"2024-09-19T07:19:40.354653+0000","last_clean":"2024-09-19T07:19:40.354653+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:19:40.354653+0000","last_undegraded":"2024-09-19T07:19:40.354653+0000","last_fullsized":"2024-09-19T07:19:40.354653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"24'2","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.429961+0000","last_change":"2024-09-19T07:19:39.424661+0000","last_active":"2024-09-19T07:19:40.429961+0000","last_peered":"2024-09-19T07:19:40.429961+0000","last_clean":"2024-09-19T07:19:40.429961+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:19:40.429961+0000","last_undegraded":"2024-09-19T07:19:40.429961+0000","last_fullsized":"2024-09-19T07:19:40.429961+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393009+0000","last_change":"2024-09-19T07:19:40.388972+0000","last_active":"2024-09-19T07:19:40.393009+0000","last_peered":"2024-09-19T07:19:40.393009+0000","last_clean":"2024-09-19T07:19:40.393009+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:19:40.393009+0000","last_undegraded":"2024-09-19T07:19:40.393009+0000","last_fullsized":"2024-09-19T07:19:40.393009+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:30.585686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370157+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:19:40.370157+0000","last_peered":"2024-09-19T07:19:40.370157+0000","last_clean":"2024-09-19T07:19:40.370157+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:19:40.370157+0000","last_undegraded":"2024-09-19T07:19:40.370157+0000","last_fullsized":"2024-09-19T07:19:40.370157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:23.791880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.346035+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:19:40.346035+0000","last_peered":"2024-09-19T07:19:40.346035+0000","last_clean":"2024-09-19T07:19:40.346035+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:19:40.346035+0000","last_undegraded":"2024-09-19T07:19:40.346035+0000","last_fullsized":"2024-09-19T07:19:40.346035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.358499+0000","last_change":"2024-09-19T07:19:39.412192+0000","last_active":"2024-09-19T07:19:40.358499+0000","last_peered":"2024-09-19T07:19:40.358499+0000","last_clean":"2024-09-19T07:19:40.358499+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:19:40.358499+0000","last_undegraded":"2024-09-19T07:19:40.358499+0000","last_fullsized":"2024-09-19T07:19:40.358499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.395575+0000","last_change":"2024-09-19T07:19:40.390217+0000","last_active":"2024-09-19T07:19:40.395575+0000","last_peered":"2024-09-19T07:19:40.395575+0000","last_clean":"2024-09-19T07:19:40.395575+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:19:40.395575+0000","last_undegraded":"2024-09-19T07:19:40.395575+0000","last_fullsized":"2024-09-19T07:19:40.395575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342531+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:19:40.342531+0000","last_peered":"2024-09-19T07:19:40.342531+0000","last_clean":"2024-09-19T07:19:40.342531+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:19:40.342531+0000","last_undegraded":"2024-09-19T07:19:40.342531+0000","last_fullsized":"2024-09-19T07:19:40.342531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356461+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:19:40.356461+0000","last_peered":"2024-09-19T07:19:40.356461+0000","last_clean":"2024-09-19T07:19:40.356461+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:19:40.356461+0000","last_undegraded":"2024-09-19T07:19:40.356461+0000","last_fullsized":"2024-09-19T07:19:40.356461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:50.700962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.338651+0000","last_change":"2024-09-19T07:19:39.434150+0000","last_active":"2024-09-19T07:19:40.338651+0000","last_peered":"2024-09-19T07:19:40.338651+0000","last_clean":"2024-09-19T07:19:40.338651+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:19:40.338651+0000","last_undegraded":"2024-09-19T07:19:40.338651+0000","last_fullsized":"2024-09-19T07:19:40.338651+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:13.654208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.386665+0000","last_change":"2024-09-19T07:19:40.380140+0000","last_active":"2024-09-19T07:19:40.386665+0000","last_peered":"2024-09-19T07:19:40.386665+0000","last_clean":"2024-09-19T07:19:40.386665+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:19:40.386665+0000","last_undegraded":"2024-09-19T07:19:40.386665+0000","last_fullsized":"2024-09-19T07:19:40.386665+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:46.389885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.350823+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:19:40.350823+0000","last_peered":"2024-09-19T07:19:40.350823+0000","last_clean":"2024-09-19T07:19:40.350823+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:19:40.350823+0000","last_undegraded":"2024-09-19T07:19:40.350823+0000","last_fullsized":"2024-09-19T07:19:40.350823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.368309+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:19:40.368309+0000","last_peered":"2024-09-19T07:19:40.368309+0000","last_clean":"2024-09-19T07:19:40.368309+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:19:40.368309+0000","last_undegraded":"2024-09-19T07:19:40.368309+0000","last_fullsized":"2024-09-19T07:19:40.368309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.369389+0000","last_change":"2024-09-19T07:19:39.411605+0000","last_active":"2024-09-19T07:19:40.369389+0000","last_peered":"2024-09-19T07:19:40.369389+0000","last_clean":"2024-09-19T07:19:40.369389+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:19:40.369389+0000","last_undegraded":"2024-09-19T07:19:40.369389+0000","last_fullsized":"2024-09-19T07:19:40.369389+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:55.654483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.379445+0000","last_change":"2024-09-19T07:19:40.372083+0000","last_active":"2024-09-19T07:19:40.379445+0000","last_peered":"2024-09-19T07:19:40.379445+0000","last_clean":"2024-09-19T07:19:40.379445+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:19:40.379445+0000","last_undegraded":"2024-09-19T07:19:40.379445+0000","last_fullsized":"2024-09-19T07:19:40.379445+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:23:15.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342175+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:19:40.342175+0000","last_peered":"2024-09-19T07:19:40.342175+0000","last_clean":"2024-09-19T07:19:40.342175+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:19:40.342175+0000","last_undegraded":"2024-09-19T07:19:40.342175+0000","last_fullsized":"2024-09-19T07:19:40.342175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.351077+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:19:40.351077+0000","last_peered":"2024-09-19T07:19:40.351077+0000","last_clean":"2024-09-19T07:19:40.351077+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:19:40.351077+0000","last_undegraded":"2024-09-19T07:19:40.351077+0000","last_fullsized":"2024-09-19T07:19:40.351077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356990+0000","last_change":"2024-09-19T07:19:39.420340+0000","last_active":"2024-09-19T07:19:40.356990+0000","last_peered":"2024-09-19T07:19:40.356990+0000","last_clean":"2024-09-19T07:19:40.356990+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:19:40.356990+0000","last_undegraded":"2024-09-19T07:19:40.356990+0000","last_fullsized":"2024-09-19T07:19:40.356990+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:29.338848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.390612+0000","last_change":"2024-09-19T07:19:40.383082+0000","last_active":"2024-09-19T07:19:40.390612+0000","last_peered":"2024-09-19T07:19:40.390612+0000","last_clean":"2024-09-19T07:19:40.390612+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:19:40.390612+0000","last_undegraded":"2024-09-19T07:19:40.390612+0000","last_fullsized":"2024-09-19T07:19:40.390612+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:42.470941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356094+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:19:40.356094+0000","last_peered":"2024-09-19T07:19:40.356094+0000","last_clean":"2024-09-19T07:19:40.356094+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:19:40.356094+0000","last_undegraded":"2024-09-19T07:19:40.356094+0000","last_fullsized":"2024-09-19T07:19:40.356094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.341157+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:19:40.341157+0000","last_peered":"2024-09-19T07:19:40.341157+0000","last_clean":"2024-09-19T07:19:40.341157+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:19:40.341157+0000","last_undegraded":"2024-09-19T07:19:40.341157+0000","last_fullsized":"2024-09-19T07:19:40.341157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.350681+0000","last_change":"2024-09-19T07:19:39.407135+0000","last_active":"2024-09-19T07:19:40.350681+0000","last_peered":"2024-09-19T07:19:40.350681+0000","last_clean":"2024-09-19T07:19:40.350681+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:19:40.350681+0000","last_undegraded":"2024-09-19T07:19:40.350681+0000","last_fullsized":"2024-09-19T07:19:40.350681+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:27.890595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.391138+0000","last_change":"2024-09-19T07:19:40.368866+0000","last_active":"2024-09-19T07:19:40.391138+0000","last_peered":"2024-09-19T07:19:40.391138+0000","last_clean":"2024-09-19T07:19:40.391138+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:19:40.391138+0000","last_undegraded":"2024-09-19T07:19:40.391138+0000","last_fullsized":"2024-09-19T07:19:40.391138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:47.944394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356301+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:19:40.356301+0000","last_peered":"2024-09-19T07:19:40.356301+0000","last_clean":"2024-09-19T07:19:40.356301+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:19:40.356301+0000","last_undegraded":"2024-09-19T07:19:40.356301+0000","last_fullsized":"2024-09-19T07:19:40.356301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.357998+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:19:40.357998+0000","last_peered":"2024-09-19T07:19:40.357998+0000","last_clean":"2024-09-19T07:19:40.357998+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:19:40.357998+0000","last_undegraded":"2024-09-19T07:19:40.357998+0000","last_fullsized":"2024-09-19T07:19:40.357998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"24'2","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.430689+0000","last_change":"2024-09-19T07:19:39.420497+0000","last_active":"2024-09-19T07:19:40.430689+0000","last_peered":"2024-09-19T07:19:40.430689+0000","last_clean":"2024-09-19T07:19:40.430689+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:19:40.430689+0000","last_undegraded":"2024-09-19T07:19:40.430689+0000","last_fullsized":"2024-09-19T07:19:40.430689+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:19.547270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393237+0000","last_change":"2024-09-19T07:19:40.387573+0000","last_active":"2024-09-19T07:19:40.393237+0000","last_peered":"2024-09-19T07:19:40.393237+0000","last_clean":"2024-09-19T07:19:40.393237+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:19:40.393237+0000","last_undegraded":"2024-09-19T07:19:40.393237+0000","last_fullsized":"2024-09-19T07:19:40.393237+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:53:37.421178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.357442+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:19:40.357442+0000","last_peered":"2024-09-19T07:19:40.357442+0000","last_clean":"2024-09-19T07:19:40.357442+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:19:40.357442+0000","last_undegraded":"2024-09-19T07:19:40.357442+0000","last_fullsized":"2024-09-19T07:19:40.357442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:20.824916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.354831+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:19:40.354831+0000","last_peered":"2024-09-19T07:19:40.354831+0000","last_clean":"2024-09-19T07:19:40.354831+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:19:40.354831+0000","last_undegraded":"2024-09-19T07:19:40.354831+0000","last_fullsized":"2024-09-19T07:19:40.354831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370650+0000","last_change":"2024-09-19T07:19:39.410752+0000","last_active":"2024-09-19T07:19:40.370650+0000","last_peered":"2024-09-19T07:19:40.370650+0000","last_clean":"2024-09-19T07:19:40.370650+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:40.370650+0000","last_undegraded":"2024-09-19T07:19:40.370650+0000","last_fullsized":"2024-09-19T07:19:40.370650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:47:09.376497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.394761+0000","last_change":"2024-09-19T07:19:40.392973+0000","last_active":"2024-09-19T07:19:40.394761+0000","last_peered":"2024-09-19T07:19:40.394761+0000","last_clean":"2024-09-19T07:19:40.394761+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:19:40.394761+0000","last_undegraded":"2024-09-19T07:19:40.394761+0000","last_fullsized":"2024-09-19T07:19:40.394761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340207+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:19:40.340207+0000","last_peered":"2024-09-19T07:19:40.340207+0000","last_clean":"2024-09-19T07:19:40.340207+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:19:40.340207+0000","last_undegraded":"2024-09-19T07:19:40.340207+0000","last_fullsized":"2024-09-19T07:19:40.340207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.344699+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:19:40.344699+0000","last_peered":"2024-09-19T07:19:40.344699+0000","last_clean":"2024-09-19T07:19:40.344699+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:19:40.344699+0000","last_undegraded":"2024-09-19T07:19:40.344699+0000","last_fullsized":"2024-09-19T07:19:40.344699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":176,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.368046+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:19:40.368046+0000","last_peered":"2024-09-19T07:19:40.368046+0000","last_clean":"2024-09-19T07:19:40.368046+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:19:40.368046+0000","last_undegraded":"2024-09-19T07:19:40.368046+0000","last_fullsized":"2024-09-19T07:19:40.368046+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342960+0000","last_change":"2024-09-19T07:19:39.429085+0000","last_active":"2024-09-19T07:19:40.342960+0000","last_peered":"2024-09-19T07:19:40.342960+0000","last_clean":"2024-09-19T07:19:40.342960+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:19:40.342960+0000","last_undegraded":"2024-09-19T07:19:40.342960+0000","last_fullsized":"2024-09-19T07:19:40.342960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393628+0000","last_change":"2024-09-19T07:19:40.382352+0000","last_active":"2024-09-19T07:19:40.393628+0000","last_peered":"2024-09-19T07:19:40.393628+0000","last_clean":"2024-09-19T07:19:40.393628+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:19:40.393628+0000","last_undegraded":"2024-09-19T07:19:40.393628+0000","last_fullsized":"2024-09-19T07:19:40.393628+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:56.356646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.364326+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:19:40.364326+0000","last_peered":"2024-09-19T07:19:40.364326+0000","last_clean":"2024-09-19T07:19:40.364326+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:19:40.364326+0000","last_undegraded":"2024-09-19T07:19:40.364326+0000","last_fullsized":"2024-09-19T07:19:40.364326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.360771+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:19:40.360771+0000","last_peered":"2024-09-19T07:19:40.360771+0000","last_clean":"2024-09-19T07:19:40.360771+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:19:40.360771+0000","last_undegraded":"2024-09-19T07:19:40.360771+0000","last_fullsized":"2024-09-19T07:19:40.360771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.344832+0000","last_change":"2024-09-19T07:19:39.427644+0000","last_active":"2024-09-19T07:19:40.344832+0000","last_peered":"2024-09-19T07:19:40.344832+0000","last_clean":"2024-09-19T07:19:40.344832+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:19:40.344832+0000","last_undegraded":"2024-09-19T07:19:40.344832+0000","last_fullsized":"2024-09-19T07:19:40.344832+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393689+0000","last_change":"2024-09-19T07:19:40.390380+0000","last_active":"2024-09-19T07:19:40.393689+0000","last_peered":"2024-09-19T07:19:40.393689+0000","last_clean":"2024-09-19T07:19:40.393689+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:19:40.393689+0000","last_undegraded":"2024-09-19T07:19:40.393689+0000","last_fullsized":"2024-09-19T07:19:40.393689+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:56.717809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.348289+0000","last_change":"2024-09-19T07:19:32.276600+0000","last_active":"2024-09-19T07:19:40.348289+0000","last_peered":"2024-09-19T07:19:40.348289+0000","last_clean":"2024-09-19T07:19:40.348289+0000","last_became_active":"2024-09-19T07:19:32.275766+0000","last_became_peered":"2024-09-19T07:19:32.275766+0000","last_unstale":"2024-09-19T07:19:40.348289+0000","last_undegraded":"2024-09-19T07:19:40.348289+0000","last_fullsized":"2024-09-19T07:19:40.348289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:48.506399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.354100+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:19:40.354100+0000","last_peered":"2024-09-19T07:19:40.354100+0000","last_clean":"2024-09-19T07:19:40.354100+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:19:40.354100+0000","last_undegraded":"2024-09-19T07:19:40.354100+0000","last_fullsized":"2024-09-19T07:19:40.354100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"24'2","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.431386+0000","last_change":"2024-09-19T07:19:39.410638+0000","last_active":"2024-09-19T07:19:40.431386+0000","last_peered":"2024-09-19T07:19:40.431386+0000","last_clean":"2024-09-19T07:19:40.431386+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:19:40.431386+0000","last_undegraded":"2024-09-19T07:19:40.431386+0000","last_fullsized":"2024-09-19T07:19:40.431386+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:32.073136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.380210+0000","last_change":"2024-09-19T07:19:40.378481+0000","last_active":"2024-09-19T07:19:40.380210+0000","last_peered":"2024-09-19T07:19:40.380210+0000","last_clean":"2024-09-19T07:19:40.380210+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:19:40.380210+0000","last_undegraded":"2024-09-19T07:19:40.380210+0000","last_fullsized":"2024-09-19T07:19:40.380210+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.371210+0000","last_change":"2024-09-19T07:19:32.292026+0000","last_active":"2024-09-19T07:19:40.371210+0000","last_peered":"2024-09-19T07:19:40.371210+0000","last_clean":"2024-09-19T07:19:40.371210+0000","last_became_active":"2024-09-19T07:19:32.291037+0000","last_became_peered":"2024-09-19T07:19:32.291037+0000","last_unstale":"2024-09-19T07:19:40.371210+0000","last_undegraded":"2024-09-19T07:19:40.371210+0000","last_fullsized":"2024-09-19T07:19:40.371210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:22.190763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.359197+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:19:40.359197+0000","last_peered":"2024-09-19T07:19:40.359197+0000","last_clean":"2024-09-19T07:19:40.359197+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:19:40.359197+0000","last_undegraded":"2024-09-19T07:19:40.359197+0000","last_fullsized":"2024-09-19T07:19:40.359197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.346303+0000","last_change":"2024-09-19T07:19:39.437656+0000","last_active":"2024-09-19T07:19:40.346303+0000","last_peered":"2024-09-19T07:19:40.346303+0000","last_clean":"2024-09-19T07:19:40.346303+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:19:40.346303+0000","last_undegraded":"2024-09-19T07:19:40.346303+0000","last_fullsized":"2024-09-19T07:19:40.346303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:41.468096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393503+0000","last_change":"2024-09-19T07:19:40.379289+0000","last_active":"2024-09-19T07:19:40.393503+0000","last_peered":"2024-09-19T07:19:40.393503+0000","last_clean":"2024-09-19T07:19:40.393503+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:19:40.393503+0000","last_undegraded":"2024-09-19T07:19:40.393503+0000","last_fullsized":"2024-09-19T07:19:40.393503+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:09.095760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.369220+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:19:40.369220+0000","last_peered":"2024-09-19T07:19:40.369220+0000","last_clean":"2024-09-19T07:19:40.369220+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:19:40.369220+0000","last_undegraded":"2024-09-19T07:19:40.369220+0000","last_fullsized":"2024-09-19T07:19:40.369220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.360228+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:19:40.360228+0000","last_peered":"2024-09-19T07:19:40.360228+0000","last_clean":"2024-09-19T07:19:40.360228+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:19:40.360228+0000","last_undegraded":"2024-09-19T07:19:40.360228+0000","last_fullsized":"2024-09-19T07:19:40.360228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.358088+0000","last_change":"2024-09-19T07:19:39.413589+0000","last_active":"2024-09-19T07:19:40.358088+0000","last_peered":"2024-09-19T07:19:40.358088+0000","last_clean":"2024-09-19T07:19:40.358088+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:19:40.358088+0000","last_undegraded":"2024-09-19T07:19:40.358088+0000","last_fullsized":"2024-09-19T07:19:40.358088+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392651+0000","last_change":"2024-09-19T07:19:40.387159+0000","last_active":"2024-09-19T07:19:40.392651+0000","last_peered":"2024-09-19T07:19:40.392651+0000","last_clean":"2024-09-19T07:19:40.392651+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:19:40.392651+0000","last_undegraded":"2024-09-19T07:19:40.392651+0000","last_fullsized":"2024-09-19T07:19:40.392651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.363062+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:19:40.363062+0000","last_peered":"2024-09-19T07:19:40.363062+0000","last_clean":"2024-09-19T07:19:40.363062+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:19:40.363062+0000","last_undegraded":"2024-09-19T07:19:40.363062+0000","last_fullsized":"2024-09-19T07:19:40.363062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.358828+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:19:40.358828+0000","last_peered":"2024-09-19T07:19:40.358828+0000","last_clean":"2024-09-19T07:19:40.358828+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:19:40.358828+0000","last_undegraded":"2024-09-19T07:19:40.358828+0000","last_fullsized":"2024-09-19T07:19:40.358828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370015+0000","last_change":"2024-09-19T07:19:39.406110+0000","last_active":"2024-09-19T07:19:40.370015+0000","last_peered":"2024-09-19T07:19:40.370015+0000","last_clean":"2024-09-19T07:19:40.370015+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:19:40.370015+0000","last_undegraded":"2024-09-19T07:19:40.370015+0000","last_fullsized":"2024-09-19T07:19:40.370015+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:10:05.224225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.374916+0000","last_change":"2024-09-19T07:19:40.365764+0000","last_active":"2024-09-19T07:19:40.374916+0000","last_peered":"2024-09-19T07:19:40.374916+0000","last_clean":"2024-09-19T07:19:40.374916+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:19:40.374916+0000","last_undegraded":"2024-09-19T07:19:40.374916+0000","last_fullsized":"2024-09-19T07:19:40.374916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:54.436493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342540+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:19:40.342540+0000","last_peered":"2024-09-19T07:19:40.342540+0000","last_clean":"2024-09-19T07:19:40.342540+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:19:40.342540+0000","last_undegraded":"2024-09-19T07:19:40.342540+0000","last_fullsized":"2024-09-19T07:19:40.342540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.349026+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:19:40.349026+0000","last_peered":"2024-09-19T07:19:40.349026+0000","last_clean":"2024-09-19T07:19:40.349026+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:19:40.349026+0000","last_undegraded":"2024-09-19T07:19:40.349026+0000","last_fullsized":"2024-09-19T07:19:40.349026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342359+0000","last_change":"2024-09-19T07:19:39.429768+0000","last_active":"2024-09-19T07:19:40.342359+0000","last_peered":"2024-09-19T07:19:40.342359+0000","last_clean":"2024-09-19T07:19:40.342359+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:19:40.342359+0000","last_undegraded":"2024-09-19T07:19:40.342359+0000","last_fullsized":"2024-09-19T07:19:40.342359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:47.503631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.380706+0000","last_change":"2024-09-19T07:19:40.373542+0000","last_active":"2024-09-19T07:19:40.380706+0000","last_peered":"2024-09-19T07:19:40.380706+0000","last_clean":"2024-09-19T07:19:40.380706+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:19:40.380706+0000","last_undegraded":"2024-09-19T07:19:40.380706+0000","last_fullsized":"2024-09-19T07:19:40.380706+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:32:34.524302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.359370+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:19:40.359370+0000","last_peered":"2024-09-19T07:19:40.359370+0000","last_clean":"2024-09-19T07:19:40.359370+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:19:40.359370+0000","last_undegraded":"2024-09-19T07:19:40.359370+0000","last_fullsized":"2024-09-19T07:19:40.359370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342449+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:19:40.342449+0000","last_peered":"2024-09-19T07:19:40.342449+0000","last_clean":"2024-09-19T07:19:40.342449+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:19:40.342449+0000","last_undegraded":"2024-09-19T07:19:40.342449+0000","last_fullsized":"2024-09-19T07:19:40.342449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"24'6","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.437011+0000","last_change":"2024-09-19T07:19:39.411074+0000","last_active":"2024-09-19T07:19:40.437011+0000","last_peered":"2024-09-19T07:19:40.437011+0000","last_clean":"2024-09-19T07:19:40.437011+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:19:40.437011+0000","last_undegraded":"2024-09-19T07:19:40.437011+0000","last_fullsized":"2024-09-19T07:19:40.437011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:53.992787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.372727+0000","last_change":"2024-09-19T07:19:40.364814+0000","last_active":"2024-09-19T07:19:40.372727+0000","last_peered":"2024-09-19T07:19:40.372727+0000","last_clean":"2024-09-19T07:19:40.372727+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:19:40.372727+0000","last_undegraded":"2024-09-19T07:19:40.372727+0000","last_fullsized":"2024-09-19T07:19:40.372727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.360602+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:19:40.360602+0000","last_peered":"2024-09-19T07:19:40.360602+0000","last_clean":"2024-09-19T07:19:40.360602+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:19:40.360602+0000","last_undegraded":"2024-09-19T07:19:40.360602+0000","last_fullsized":"2024-09-19T07:19:40.360602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:57.131912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370026+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:19:40.370026+0000","last_peered":"2024-09-19T07:19:40.370026+0000","last_clean":"2024-09-19T07:19:40.370026+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:19:40.370026+0000","last_undegraded":"2024-09-19T07:19:40.370026+0000","last_fullsized":"2024-09-19T07:19:40.370026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:23.367365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"24'3","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.429175+0000","last_change":"2024-09-19T07:19:39.431616+0000","last_active":"2024-09-19T07:19:40.429175+0000","last_peered":"2024-09-19T07:19:40.429175+0000","last_clean":"2024-09-19T07:19:40.429175+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:19:40.429175+0000","last_undegraded":"2024-09-19T07:19:40.429175+0000","last_fullsized":"2024-09-19T07:19:40.429175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:18.149581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.390972+0000","last_change":"2024-09-19T07:19:40.381443+0000","last_active":"2024-09-19T07:19:40.390972+0000","last_peered":"2024-09-19T07:19:40.390972+0000","last_clean":"2024-09-19T07:19:40.390972+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:19:40.390972+0000","last_undegraded":"2024-09-19T07:19:40.390972+0000","last_fullsized":"2024-09-19T07:19:40.390972+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:56:40.928844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.364027+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:19:40.364027+0000","last_peered":"2024-09-19T07:19:40.364027+0000","last_clean":"2024-09-19T07:19:40.364027+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:19:40.364027+0000","last_undegraded":"2024-09-19T07:19:40.364027+0000","last_fullsized":"2024-09-19T07:19:40.364027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.343214+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:19:40.343214+0000","last_peered":"2024-09-19T07:19:40.343214+0000","last_clean":"2024-09-19T07:19:40.343214+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:19:40.343214+0000","last_undegraded":"2024-09-19T07:19:40.343214+0000","last_fullsized":"2024-09-19T07:19:40.343214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356615+0000","last_change":"2024-09-19T07:19:39.423972+0000","last_active":"2024-09-19T07:19:40.356615+0000","last_peered":"2024-09-19T07:19:40.356615+0000","last_clean":"2024-09-19T07:19:40.356615+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:19:40.356615+0000","last_undegraded":"2024-09-19T07:19:40.356615+0000","last_fullsized":"2024-09-19T07:19:40.356615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:45.387040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.387950+0000","last_change":"2024-09-19T07:19:40.382092+0000","last_active":"2024-09-19T07:19:40.387950+0000","last_peered":"2024-09-19T07:19:40.387950+0000","last_clean":"2024-09-19T07:19:40.387950+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:19:40.387950+0000","last_undegraded":"2024-09-19T07:19:40.387950+0000","last_fullsized":"2024-09-19T07:19:40.387950+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370617+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:19:40.370617+0000","last_peered":"2024-09-19T07:19:40.370617+0000","last_clean":"2024-09-19T07:19:40.370617+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:19:40.370617+0000","last_undegraded":"2024-09-19T07:19:40.370617+0000","last_fullsized":"2024-09-19T07:19:40.370617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.353582+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:19:40.353582+0000","last_peered":"2024-09-19T07:19:40.353582+0000","last_clean":"2024-09-19T07:19:40.353582+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:19:40.353582+0000","last_undegraded":"2024-09-19T07:19:40.353582+0000","last_fullsized":"2024-09-19T07:19:40.353582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"24'2","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.429916+0000","last_change":"2024-09-19T07:19:39.424879+0000","last_active":"2024-09-19T07:19:40.429916+0000","last_peered":"2024-09-19T07:19:40.429916+0000","last_clean":"2024-09-19T07:19:40.429916+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:19:40.429916+0000","last_undegraded":"2024-09-19T07:19:40.429916+0000","last_fullsized":"2024-09-19T07:19:40.429916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:30.021543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.386869+0000","last_change":"2024-09-19T07:19:40.384136+0000","last_active":"2024-09-19T07:19:40.386869+0000","last_peered":"2024-09-19T07:19:40.386869+0000","last_clean":"2024-09-19T07:19:40.386869+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:19:40.386869+0000","last_undegraded":"2024-09-19T07:19:40.386869+0000","last_fullsized":"2024-09-19T07:19:40.386869+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:20.550115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.369794+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:19:40.369794+0000","last_peered":"2024-09-19T07:19:40.369794+0000","last_clean":"2024-09-19T07:19:40.369794+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:19:40.369794+0000","last_undegraded":"2024-09-19T07:19:40.369794+0000","last_fullsized":"2024-09-19T07:19:40.369794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:48.308137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.344608+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:19:40.344608+0000","last_peered":"2024-09-19T07:19:40.344608+0000","last_clean":"2024-09-19T07:19:40.344608+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:19:40.344608+0000","last_undegraded":"2024-09-19T07:19:40.344608+0000","last_fullsized":"2024-09-19T07:19:40.344608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"24'8","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.433804+0000","last_change":"2024-09-19T07:19:39.410445+0000","last_active":"2024-09-19T07:19:40.433804+0000","last_peered":"2024-09-19T07:19:40.433804+0000","last_clean":"2024-09-19T07:19:40.433804+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:19:40.433804+0000","last_undegraded":"2024-09-19T07:19:40.433804+0000","last_fullsized":"2024-09-19T07:19:40.433804+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:27.020327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392927+0000","last_change":"2024-09-19T07:19:40.389499+0000","last_active":"2024-09-19T07:19:40.392927+0000","last_peered":"2024-09-19T07:19:40.392927+0000","last_clean":"2024-09-19T07:19:40.392927+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:19:40.392927+0000","last_undegraded":"2024-09-19T07:19:40.392927+0000","last_fullsized":"2024-09-19T07:19:40.392927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.344552+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:19:40.344552+0000","last_peered":"2024-09-19T07:19:40.344552+0000","last_clean":"2024-09-19T07:19:40.344552+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:19:40.344552+0000","last_undegraded":"2024-09-19T07:19:40.344552+0000","last_fullsized":"2024-09-19T07:19:40.344552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.345793+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:19:40.345793+0000","last_peered":"2024-09-19T07:19:40.345793+0000","last_clean":"2024-09-19T07:19:40.345793+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:19:40.345793+0000","last_undegraded":"2024-09-19T07:19:40.345793+0000","last_fullsized":"2024-09-19T07:19:40.345793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.338817+0000","last_change":"2024-09-19T07:19:39.416943+0000","last_active":"2024-09-19T07:19:40.338817+0000","last_peered":"2024-09-19T07:19:40.338817+0000","last_clean":"2024-09-19T07:19:40.338817+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:19:40.338817+0000","last_undegraded":"2024-09-19T07:19:40.338817+0000","last_fullsized":"2024-09-19T07:19:40.338817+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:35.555826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.396408+0000","last_change":"2024-09-19T07:19:40.394993+0000","last_active":"2024-09-19T07:19:40.396408+0000","last_peered":"2024-09-19T07:19:40.396408+0000","last_clean":"2024-09-19T07:19:40.396408+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:19:40.396408+0000","last_undegraded":"2024-09-19T07:19:40.396408+0000","last_fullsized":"2024-09-19T07:19:40.396408+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:30.341693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"24'1","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.431200+0000","last_change":"2024-09-19T07:19:39.418885+0000","last_active":"2024-09-19T07:19:40.431200+0000","last_peered":"2024-09-19T07:19:40.431200+0000","last_clean":"2024-09-19T07:19:40.431200+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:19:40.431200+0000","last_undegraded":"2024-09-19T07:19:40.431200+0000","last_fullsized":"2024-09-19T07:19:40.431200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:35:45.790411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.348259+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:19:40.348259+0000","last_peered":"2024-09-19T07:19:40.348259+0000","last_clean":"2024-09-19T07:19:40.348259+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:19:40.348259+0000","last_undegraded":"2024-09-19T07:19:40.348259+0000","last_fullsized":"2024-09-19T07:19:40.348259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.346923+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:19:40.346923+0000","last_peered":"2024-09-19T07:19:40.346923+0000","last_clean":"2024-09-19T07:19:40.346923+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:19:40.346923+0000","last_undegraded":"2024-09-19T07:19:40.346923+0000","last_fullsized":"2024-09-19T07:19:40.346923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.379767+0000","last_change":"2024-09-19T07:19:40.374048+0000","last_active":"2024-09-19T07:19:40.379767+0000","last_peered":"2024-09-19T07:19:40.379767+0000","last_clean":"2024-09-19T07:19:40.379767+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:19:40.379767+0000","last_undegraded":"2024-09-19T07:19:40.379767+0000","last_fullsized":"2024-09-19T07:19:40.379767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:35:18.576120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"24'4","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.430201+0000","last_change":"2024-09-19T07:19:39.432426+0000","last_active":"2024-09-19T07:19:40.430201+0000","last_peered":"2024-09-19T07:19:40.430201+0000","last_clean":"2024-09-19T07:19:40.430201+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:19:40.430201+0000","last_undegraded":"2024-09-19T07:19:40.430201+0000","last_fullsized":"2024-09-19T07:19:40.430201+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.339727+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:19:40.339727+0000","last_peered":"2024-09-19T07:19:40.339727+0000","last_clean":"2024-09-19T07:19:40.339727+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:19:40.339727+0000","last_undegraded":"2024-09-19T07:19:40.339727+0000","last_fullsized":"2024-09-19T07:19:40.339727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.339280+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:19:40.339280+0000","last_peered":"2024-09-19T07:19:40.339280+0000","last_clean":"2024-09-19T07:19:40.339280+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:19:40.339280+0000","last_undegraded":"2024-09-19T07:19:40.339280+0000","last_fullsized":"2024-09-19T07:19:40.339280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.387760+0000","last_change":"2024-09-19T07:19:40.384934+0000","last_active":"2024-09-19T07:19:40.387760+0000","last_peered":"2024-09-19T07:19:40.387760+0000","last_clean":"2024-09-19T07:19:40.387760+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:19:40.387760+0000","last_undegraded":"2024-09-19T07:19:40.387760+0000","last_fullsized":"2024-09-19T07:19:40.387760+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.345456+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:19:40.345456+0000","last_peered":"2024-09-19T07:19:40.345456+0000","last_clean":"2024-09-19T07:19:40.345456+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:19:40.345456+0000","last_undegraded":"2024-09-19T07:19:40.345456+0000","last_fullsized":"2024-09-19T07:19:40.345456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.369661+0000","last_change":"2024-09-19T07:19:32.289374+0000","last_active":"2024-09-19T07:19:40.369661+0000","last_peered":"2024-09-19T07:19:40.369661+0000","last_clean":"2024-09-19T07:19:40.369661+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:19:40.369661+0000","last_undegraded":"2024-09-19T07:19:40.369661+0000","last_fullsized":"2024-09-19T07:19:40.369661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"24'2","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.431155+0000","last_change":"2024-09-19T07:19:39.410582+0000","last_active":"2024-09-19T07:19:40.431155+0000","last_peered":"2024-09-19T07:19:40.431155+0000","last_clean":"2024-09-19T07:19:40.431155+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:19:40.431155+0000","last_undegraded":"2024-09-19T07:19:40.431155+0000","last_fullsized":"2024-09-19T07:19:40.431155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:41:40.592907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.395801+0000","last_change":"2024-09-19T07:19:40.393950+0000","last_active":"2024-09-19T07:19:40.395801+0000","last_peered":"2024-09-19T07:19:40.395801+0000","last_clean":"2024-09-19T07:19:40.395801+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:19:40.395801+0000","last_undegraded":"2024-09-19T07:19:40.395801+0000","last_fullsized":"2024-09-19T07:19:40.395801+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340914+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:19:40.340914+0000","last_peered":"2024-09-19T07:19:40.340914+0000","last_clean":"2024-09-19T07:19:40.340914+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:19:40.340914+0000","last_undegraded":"2024-09-19T07:19:40.340914+0000","last_fullsized":"2024-09-19T07:19:40.340914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.355065+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:19:40.355065+0000","last_peered":"2024-09-19T07:19:40.355065+0000","last_clean":"2024-09-19T07:19:40.355065+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:19:40.355065+0000","last_undegraded":"2024-09-19T07:19:40.355065+0000","last_fullsized":"2024-09-19T07:19:40.355065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"24'2","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.433062+0000","last_change":"2024-09-19T07:19:39.410662+0000","last_active":"2024-09-19T07:19:40.433062+0000","last_peered":"2024-09-19T07:19:40.433062+0000","last_clean":"2024-09-19T07:19:40.433062+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:40.433062+0000","last_undegraded":"2024-09-19T07:19:40.433062+0000","last_fullsized":"2024-09-19T07:19:40.433062+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:19.196074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392075+0000","last_change":"2024-09-19T07:19:40.385581+0000","last_active":"2024-09-19T07:19:40.392075+0000","last_peered":"2024-09-19T07:19:40.392075+0000","last_clean":"2024-09-19T07:19:40.392075+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:19:40.392075+0000","last_undegraded":"2024-09-19T07:19:40.392075+0000","last_fullsized":"2024-09-19T07:19:40.392075+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342259+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:19:40.342259+0000","last_peered":"2024-09-19T07:19:40.342259+0000","last_clean":"2024-09-19T07:19:40.342259+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:19:40.342259+0000","last_undegraded":"2024-09-19T07:19:40.342259+0000","last_fullsized":"2024-09-19T07:19:40.342259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.380963+0000","last_change":"2024-09-19T07:19:40.377608+0000","last_active":"2024-09-19T07:19:40.380963+0000","last_peered":"2024-09-19T07:19:40.380963+0000","last_clean":"2024-09-19T07:19:40.380963+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:19:40.380963+0000","last_undegraded":"2024-09-19T07:19:40.380963+0000","last_fullsized":"2024-09-19T07:19:40.380963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356711+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:19:40.356711+0000","last_peered":"2024-09-19T07:19:40.356711+0000","last_clean":"2024-09-19T07:19:40.356711+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:19:40.356711+0000","last_undegraded":"2024-09-19T07:19:40.356711+0000","last_fullsized":"2024-09-19T07:19:40.356711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393077+0000","last_change":"2024-09-19T07:19:40.381262+0000","last_active":"2024-09-19T07:19:40.393077+0000","last_peered":"2024-09-19T07:19:40.393077+0000","last_clean":"2024-09-19T07:19:40.393077+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:19:40.393077+0000","last_undegraded":"2024-09-19T07:19:40.393077+0000","last_fullsized":"2024-09-19T07:19:40.393077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:35:35.760631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340773+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:19:40.340773+0000","last_peered":"2024-09-19T07:19:40.340773+0000","last_clean":"2024-09-19T07:19:40.340773+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:19:40.340773+0000","last_undegraded":"2024-09-19T07:19:40.340773+0000","last_fullsized":"2024-09-19T07:19:40.340773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.390804+0000","last_change":"2024-09-19T07:19:40.386432+0000","last_active":"2024-09-19T07:19:40.390804+0000","last_peered":"2024-09-19T07:19:40.390804+0000","last_clean":"2024-09-19T07:19:40.390804+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:19:40.390804+0000","last_undegraded":"2024-09-19T07:19:40.390804+0000","last_fullsized":"2024-09-19T07:19:40.390804+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.369564+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:19:40.369564+0000","last_peered":"2024-09-19T07:19:40.369564+0000","last_clean":"2024-09-19T07:19:40.369564+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:19:40.369564+0000","last_undegraded":"2024-09-19T07:19:40.369564+0000","last_fullsized":"2024-09-19T07:19:40.369564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.379966+0000","last_change":"2024-09-19T07:19:40.378630+0000","last_active":"2024-09-19T07:19:40.379966+0000","last_peered":"2024-09-19T07:19:40.379966+0000","last_clean":"2024-09-19T07:19:40.379966+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:19:40.379966+0000","last_undegraded":"2024-09-19T07:19:40.379966+0000","last_fullsized":"2024-09-19T07:19:40.379966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342627+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:19:40.342627+0000","last_peered":"2024-09-19T07:19:40.342627+0000","last_clean":"2024-09-19T07:19:40.342627+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:19:40.342627+0000","last_undegraded":"2024-09-19T07:19:40.342627+0000","last_fullsized":"2024-09-19T07:19:40.342627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.384631+0000","last_change":"2024-09-19T07:19:40.381365+0000","last_active":"2024-09-19T07:19:40.384631+0000","last_peered":"2024-09-19T07:19:40.384631+0000","last_clean":"2024-09-19T07:19:40.384631+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:19:40.384631+0000","last_undegraded":"2024-09-19T07:19:40.384631+0000","last_fullsized":"2024-09-19T07:19:40.384631+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:46.389885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.353296+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:19:40.353296+0000","last_peered":"2024-09-19T07:19:40.353296+0000","last_clean":"2024-09-19T07:19:40.353296+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:19:40.353296+0000","last_undegraded":"2024-09-19T07:19:40.353296+0000","last_fullsized":"2024-09-19T07:19:40.353296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392393+0000","last_change":"2024-09-19T07:19:40.387116+0000","last_active":"2024-09-19T07:19:40.392393+0000","last_peered":"2024-09-19T07:19:40.392393+0000","last_clean":"2024-09-19T07:19:40.392393+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:19:40.392393+0000","last_undegraded":"2024-09-19T07:19:40.392393+0000","last_fullsized":"2024-09-19T07:19:40.392393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.344093+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:19:40.344093+0000","last_peered":"2024-09-19T07:19:40.344093+0000","last_clean":"2024-09-19T07:19:40.344093+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:19:40.344093+0000","last_undegraded":"2024-09-19T07:19:40.344093+0000","last_fullsized":"2024-09-19T07:19:40.344093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.365871+0000","last_change":"2024-09-19T07:19:40.357629+0000","last_active":"2024-09-19T07:19:40.365871+0000","last_peered":"2024-09-19T07:19:40.365871+0000","last_clean":"2024-09-19T07:19:40.365871+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:19:40.365871+0000","last_undegraded":"2024-09-19T07:19:40.365871+0000","last_fullsized":"2024-09-19T07:19:40.365871+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:28.893439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342355+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:19:40.342355+0000","last_peered":"2024-09-19T07:19:40.342355+0000","last_clean":"2024-09-19T07:19:40.342355+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:19:40.342355+0000","last_undegraded":"2024-09-19T07:19:40.342355+0000","last_fullsized":"2024-09-19T07:19:40.342355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393476+0000","last_change":"2024-09-19T07:19:40.388597+0000","last_active":"2024-09-19T07:19:40.393476+0000","last_peered":"2024-09-19T07:19:40.393476+0000","last_clean":"2024-09-19T07:19:40.393476+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:19:40.393476+0000","last_undegraded":"2024-09-19T07:19:40.393476+0000","last_fullsized":"2024-09-19T07:19:40.393476+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.346684+0000","last_change":"2024-09-19T07:19:32.273635+0000","last_active":"2024-09-19T07:19:40.346684+0000","last_peered":"2024-09-19T07:19:40.346684+0000","last_clean":"2024-09-19T07:19:40.346684+0000","last_became_active":"2024-09-19T07:19:32.271748+0000","last_became_peered":"2024-09-19T07:19:32.271748+0000","last_unstale":"2024-09-19T07:19:40.346684+0000","last_undegraded":"2024-09-19T07:19:40.346684+0000","last_fullsized":"2024-09-19T07:19:40.346684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:49:41.724626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.386734+0000","last_change":"2024-09-19T07:19:40.383779+0000","last_active":"2024-09-19T07:19:40.386734+0000","last_peered":"2024-09-19T07:19:40.386734+0000","last_clean":"2024-09-19T07:19:40.386734+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:19:40.386734+0000","last_undegraded":"2024-09-19T07:19:40.386734+0000","last_fullsized":"2024-09-19T07:19:40.386734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.338969+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:19:40.338969+0000","last_peered":"2024-09-19T07:19:40.338969+0000","last_clean":"2024-09-19T07:19:40.338969+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:19:40.338969+0000","last_undegraded":"2024-09-19T07:19:40.338969+0000","last_fullsized":"2024-09-19T07:19:40.338969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393368+0000","last_change":"2024-09-19T07:19:40.376342+0000","last_active":"2024-09-19T07:19:40.393368+0000","last_peered":"2024-09-19T07:19:40.393368+0000","last_clean":"2024-09-19T07:19:40.393368+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:19:40.393368+0000","last_undegraded":"2024-09-19T07:19:40.393368+0000","last_fullsized":"2024-09-19T07:19:40.393368+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:37.532894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340727+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:19:40.340727+0000","last_peered":"2024-09-19T07:19:40.340727+0000","last_clean":"2024-09-19T07:19:40.340727+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:19:40.340727+0000","last_undegraded":"2024-09-19T07:19:40.340727+0000","last_fullsized":"2024-09-19T07:19:40.340727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.368358+0000","last_change":"2024-09-19T07:19:40.361504+0000","last_active":"2024-09-19T07:19:40.368358+0000","last_peered":"2024-09-19T07:19:40.368358+0000","last_clean":"2024-09-19T07:19:40.368358+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:19:40.368358+0000","last_undegraded":"2024-09-19T07:19:40.368358+0000","last_fullsized":"2024-09-19T07:19:40.368358+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.369423+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:19:40.369423+0000","last_peered":"2024-09-19T07:19:40.369423+0000","last_clean":"2024-09-19T07:19:40.369423+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:19:40.369423+0000","last_undegraded":"2024-09-19T07:19:40.369423+0000","last_fullsized":"2024-09-19T07:19:40.369423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:19.033192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.394565+0000","last_change":"2024-09-19T07:19:40.390205+0000","last_active":"2024-09-19T07:19:40.394565+0000","last_peered":"2024-09-19T07:19:40.394565+0000","last_clean":"2024-09-19T07:19:40.394565+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:19:40.394565+0000","last_undegraded":"2024-09-19T07:19:40.394565+0000","last_fullsized":"2024-09-19T07:19:40.394565+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342913+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:19:40.342913+0000","last_peered":"2024-09-19T07:19:40.342913+0000","last_clean":"2024-09-19T07:19:40.342913+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:19:40.342913+0000","last_undegraded":"2024-09-19T07:19:40.342913+0000","last_fullsized":"2024-09-19T07:19:40.342913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.394504+0000","last_change":"2024-09-19T07:19:40.390559+0000","last_active":"2024-09-19T07:19:40.394504+0000","last_peered":"2024-09-19T07:19:40.394504+0000","last_clean":"2024-09-19T07:19:40.394504+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:19:40.394504+0000","last_undegraded":"2024-09-19T07:19:40.394504+0000","last_fullsized":"2024-09-19T07:19:40.394504+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.368038+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:19:40.368038+0000","last_peered":"2024-09-19T07:19:40.368038+0000","last_clean":"2024-09-19T07:19:40.368038+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:19:40.368038+0000","last_undegraded":"2024-09-19T07:19:40.368038+0000","last_fullsized":"2024-09-19T07:19:40.368038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.364863+0000","last_change":"2024-09-19T07:19:40.358433+0000","last_active":"2024-09-19T07:19:40.364863+0000","last_peered":"2024-09-19T07:19:40.364863+0000","last_clean":"2024-09-19T07:19:40.364863+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:19:40.364863+0000","last_undegraded":"2024-09-19T07:19:40.364863+0000","last_fullsized":"2024-09-19T07:19:40.364863+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:34:05.875388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.358717+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:19:40.358717+0000","last_peered":"2024-09-19T07:19:40.358717+0000","last_clean":"2024-09-19T07:19:40.358717+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:19:40.358717+0000","last_undegraded":"2024-09-19T07:19:40.358717+0000","last_fullsized":"2024-09-19T07:19:40.358717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:51.710997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.379817+0000","last_change":"2024-09-19T07:19:40.374374+0000","last_active":"2024-09-19T07:19:40.379817+0000","last_peered":"2024-09-19T07:19:40.379817+0000","last_clean":"2024-09-19T07:19:40.379817+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:19:40.379817+0000","last_undegraded":"2024-09-19T07:19:40.379817+0000","last_fullsized":"2024-09-19T07:19:40.379817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:19.152425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.342559+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:19:40.342559+0000","last_peered":"2024-09-19T07:19:40.342559+0000","last_clean":"2024-09-19T07:19:40.342559+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:19:40.342559+0000","last_undegraded":"2024-09-19T07:19:40.342559+0000","last_fullsized":"2024-09-19T07:19:40.342559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.390806+0000","last_change":"2024-09-19T07:19:40.387341+0000","last_active":"2024-09-19T07:19:40.390806+0000","last_peered":"2024-09-19T07:19:40.390806+0000","last_clean":"2024-09-19T07:19:40.390806+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:19:40.390806+0000","last_undegraded":"2024-09-19T07:19:40.390806+0000","last_fullsized":"2024-09-19T07:19:40.390806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.346439+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:19:40.346439+0000","last_peered":"2024-09-19T07:19:40.346439+0000","last_clean":"2024-09-19T07:19:40.346439+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:19:40.346439+0000","last_undegraded":"2024-09-19T07:19:40.346439+0000","last_fullsized":"2024-09-19T07:19:40.346439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.390692+0000","last_change":"2024-09-19T07:19:40.388484+0000","last_active":"2024-09-19T07:19:40.390692+0000","last_peered":"2024-09-19T07:19:40.390692+0000","last_clean":"2024-09-19T07:19:40.390692+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:19:40.390692+0000","last_undegraded":"2024-09-19T07:19:40.390692+0000","last_fullsized":"2024-09-19T07:19:40.390692+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:30.341693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.348499+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:19:40.348499+0000","last_peered":"2024-09-19T07:19:40.348499+0000","last_clean":"2024-09-19T07:19:40.348499+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:19:40.348499+0000","last_undegraded":"2024-09-19T07:19:40.348499+0000","last_fullsized":"2024-09-19T07:19:40.348499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.375037+0000","last_change":"2024-09-19T07:19:40.367578+0000","last_active":"2024-09-19T07:19:40.375037+0000","last_peered":"2024-09-19T07:19:40.375037+0000","last_clean":"2024-09-19T07:19:40.375037+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:19:40.375037+0000","last_undegraded":"2024-09-19T07:19:40.375037+0000","last_fullsized":"2024-09-19T07:19:40.375037+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:40.084827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.360136+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:19:40.360136+0000","last_peered":"2024-09-19T07:19:40.360136+0000","last_clean":"2024-09-19T07:19:40.360136+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:19:40.360136+0000","last_undegraded":"2024-09-19T07:19:40.360136+0000","last_fullsized":"2024-09-19T07:19:40.360136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:43.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.387682+0000","last_change":"2024-09-19T07:19:40.385641+0000","last_active":"2024-09-19T07:19:40.387682+0000","last_peered":"2024-09-19T07:19:40.387682+0000","last_clean":"2024-09-19T07:19:40.387682+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:19:40.387682+0000","last_undegraded":"2024-09-19T07:19:40.387682+0000","last_fullsized":"2024-09-19T07:19:40.387682+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340367+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:19:40.340367+0000","last_peered":"2024-09-19T07:19:40.340367+0000","last_clean":"2024-09-19T07:19:40.340367+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:19:40.340367+0000","last_undegraded":"2024-09-19T07:19:40.340367+0000","last_fullsized":"2024-09-19T07:19:40.340367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.375372+0000","last_change":"2024-09-19T07:19:40.367101+0000","last_active":"2024-09-19T07:19:40.375372+0000","last_peered":"2024-09-19T07:19:40.375372+0000","last_clean":"2024-09-19T07:19:40.375372+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:19:40.375372+0000","last_undegraded":"2024-09-19T07:19:40.375372+0000","last_fullsized":"2024-09-19T07:19:40.375372+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.362347+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:19:40.362347+0000","last_peered":"2024-09-19T07:19:40.362347+0000","last_clean":"2024-09-19T07:19:40.362347+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:19:40.362347+0000","last_undegraded":"2024-09-19T07:19:40.362347+0000","last_fullsized":"2024-09-19T07:19:40.362347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393667+0000","last_change":"2024-09-19T07:19:40.384212+0000","last_active":"2024-09-19T07:19:40.393667+0000","last_peered":"2024-09-19T07:19:40.393667+0000","last_clean":"2024-09-19T07:19:40.393667+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:19:40.393667+0000","last_undegraded":"2024-09-19T07:19:40.393667+0000","last_fullsized":"2024-09-19T07:19:40.393667+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:36.886449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.343312+0000","last_change":"2024-09-19T07:19:32.299956+0000","last_active":"2024-09-19T07:19:40.343312+0000","last_peered":"2024-09-19T07:19:40.343312+0000","last_clean":"2024-09-19T07:19:40.343312+0000","last_became_active":"2024-09-19T07:19:32.299168+0000","last_became_peered":"2024-09-19T07:19:32.299168+0000","last_unstale":"2024-09-19T07:19:40.343312+0000","last_undegraded":"2024-09-19T07:19:40.343312+0000","last_fullsized":"2024-09-19T07:19:40.343312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392829+0000","last_change":"2024-09-19T07:19:40.387401+0000","last_active":"2024-09-19T07:19:40.392829+0000","last_peered":"2024-09-19T07:19:40.392829+0000","last_clean":"2024-09-19T07:19:40.392829+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:19:40.392829+0000","last_undegraded":"2024-09-19T07:19:40.392829+0000","last_fullsized":"2024-09-19T07:19:40.392829+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.343788+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:19:40.343788+0000","last_peered":"2024-09-19T07:19:40.343788+0000","last_clean":"2024-09-19T07:19:40.343788+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:19:40.343788+0000","last_undegraded":"2024-09-19T07:19:40.343788+0000","last_fullsized":"2024-09-19T07:19:40.343788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.365845+0000","last_change":"2024-09-19T07:19:40.361430+0000","last_active":"2024-09-19T07:19:40.365845+0000","last_peered":"2024-09-19T07:19:40.365845+0000","last_clean":"2024-09-19T07:19:40.365845+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:19:40.365845+0000","last_undegraded":"2024-09-19T07:19:40.365845+0000","last_fullsized":"2024-09-19T07:19:40.365845+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:39.665475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370978+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:19:40.370978+0000","last_peered":"2024-09-19T07:19:40.370978+0000","last_clean":"2024-09-19T07:19:40.370978+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:19:40.370978+0000","last_undegraded":"2024-09-19T07:19:40.370978+0000","last_fullsized":"2024-09-19T07:19:40.370978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.392029+0000","last_change":"2024-09-19T07:19:40.378172+0000","last_active":"2024-09-19T07:19:40.392029+0000","last_peered":"2024-09-19T07:19:40.392029+0000","last_clean":"2024-09-19T07:19:40.392029+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:19:40.392029+0000","last_undegraded":"2024-09-19T07:19:40.392029+0000","last_fullsized":"2024-09-19T07:19:40.392029+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.339480+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:19:40.339480+0000","last_peered":"2024-09-19T07:19:40.339480+0000","last_clean":"2024-09-19T07:19:40.339480+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:19:40.339480+0000","last_undegraded":"2024-09-19T07:19:40.339480+0000","last_fullsized":"2024-09-19T07:19:40.339480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.378910+0000","last_change":"2024-09-19T07:19:40.373909+0000","last_active":"2024-09-19T07:19:40.378910+0000","last_peered":"2024-09-19T07:19:40.378910+0000","last_clean":"2024-09-19T07:19:40.378910+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:19:40.378910+0000","last_undegraded":"2024-09-19T07:19:40.378910+0000","last_fullsized":"2024-09-19T07:19:40.378910+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:22.017682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.362811+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:19:40.362811+0000","last_peered":"2024-09-19T07:19:40.362811+0000","last_clean":"2024-09-19T07:19:40.362811+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:19:40.362811+0000","last_undegraded":"2024-09-19T07:19:40.362811+0000","last_fullsized":"2024-09-19T07:19:40.362811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.397128+0000","last_change":"2024-09-19T07:19:40.396146+0000","last_active":"2024-09-19T07:19:40.397128+0000","last_peered":"2024-09-19T07:19:40.397128+0000","last_clean":"2024-09-19T07:19:40.397128+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:19:40.397128+0000","last_undegraded":"2024-09-19T07:19:40.397128+0000","last_fullsized":"2024-09-19T07:19:40.397128+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.340526+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:19:40.340526+0000","last_peered":"2024-09-19T07:19:40.340526+0000","last_clean":"2024-09-19T07:19:40.340526+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:19:40.340526+0000","last_undegraded":"2024-09-19T07:19:40.340526+0000","last_fullsized":"2024-09-19T07:19:40.340526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.380650+0000","last_change":"2024-09-19T07:19:40.373552+0000","last_active":"2024-09-19T07:19:40.380650+0000","last_peered":"2024-09-19T07:19:40.380650+0000","last_clean":"2024-09-19T07:19:40.380650+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:19:40.380650+0000","last_undegraded":"2024-09-19T07:19:40.380650+0000","last_fullsized":"2024-09-19T07:19:40.380650+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:35:11.516938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.359124+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:19:40.359124+0000","last_peered":"2024-09-19T07:19:40.359124+0000","last_clean":"2024-09-19T07:19:40.359124+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:19:40.359124+0000","last_undegraded":"2024-09-19T07:19:40.359124+0000","last_fullsized":"2024-09-19T07:19:40.359124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.360300+0000","last_change":"2024-09-19T07:19:40.356989+0000","last_active":"2024-09-19T07:19:40.360300+0000","last_peered":"2024-09-19T07:19:40.360300+0000","last_clean":"2024-09-19T07:19:40.360300+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:19:40.360300+0000","last_undegraded":"2024-09-19T07:19:40.360300+0000","last_fullsized":"2024-09-19T07:19:40.360300+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.347810+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:19:40.347810+0000","last_peered":"2024-09-19T07:19:40.347810+0000","last_clean":"2024-09-19T07:19:40.347810+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:19:40.347810+0000","last_undegraded":"2024-09-19T07:19:40.347810+0000","last_fullsized":"2024-09-19T07:19:40.347810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.393286+0000","last_change":"2024-09-19T07:19:40.376487+0000","last_active":"2024-09-19T07:19:40.393286+0000","last_peered":"2024-09-19T07:19:40.393286+0000","last_clean":"2024-09-19T07:19:40.393286+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:19:40.393286+0000","last_undegraded":"2024-09-19T07:19:40.393286+0000","last_fullsized":"2024-09-19T07:19:40.393286+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:45.812485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.368708+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:19:40.368708+0000","last_peered":"2024-09-19T07:19:40.368708+0000","last_clean":"2024-09-19T07:19:40.368708+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:19:40.368708+0000","last_undegraded":"2024-09-19T07:19:40.368708+0000","last_fullsized":"2024-09-19T07:19:40.368708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:25.667168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.386792+0000","last_change":"2024-09-19T07:19:40.377604+0000","last_active":"2024-09-19T07:19:40.386792+0000","last_peered":"2024-09-19T07:19:40.386792+0000","last_clean":"2024-09-19T07:19:40.386792+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:19:40.386792+0000","last_undegraded":"2024-09-19T07:19:40.386792+0000","last_fullsized":"2024-09-19T07:19:40.386792+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.356201+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:19:40.356201+0000","last_peered":"2024-09-19T07:19:40.356201+0000","last_clean":"2024-09-19T07:19:40.356201+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:19:40.356201+0000","last_undegraded":"2024-09-19T07:19:40.356201+0000","last_fullsized":"2024-09-19T07:19:40.356201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.379393+0000","last_change":"2024-09-19T07:19:40.377580+0000","last_active":"2024-09-19T07:19:40.379393+0000","last_peered":"2024-09-19T07:19:40.379393+0000","last_clean":"2024-09-19T07:19:40.379393+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:19:40.379393+0000","last_undegraded":"2024-09-19T07:19:40.379393+0000","last_fullsized":"2024-09-19T07:19:40.379393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-19T07:19:40.370728+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:19:40.370728+0000","last_peered":"2024-09-19T07:19:40.370728+0000","last_clean":"2024-09-19T07:19:40.370728+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:19:40.370728+0000","last_undegraded":"2024-09-19T07:19:40.370728+0000","last_fullsized":"2024-09-19T07:19:40.370728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":847872,"data_stored":686333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":97611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":87896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":87932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":89418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":95453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":97177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":843776,"data_stored":678812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:19:45.811 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-19T07:19:45.811 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-19T07:19:45.811 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-19T07:19:45.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-19T07:19:45.812 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-19T07:19:45.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.955+0000 7fba54327640 1 Processor -- start 2024-09-19T07:19:45.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba54327640 1 -- start start 2024-09-19T07:19:45.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba54327640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fba4c14ca60 0x7fba4c159490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba54327640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fba4c0cbb00 0x7fba4c0cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba54327640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba4c1599d0 con 0x7fba4c0bef00 2024-09-19T07:19:45.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba54327640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fba4c0ca240 con 0x7fba4c0cbb00 2024-09-19T07:19:45.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba54327640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fba4c0c9c80 con 0x7fba4c14ca60 2024-09-19T07:19:45.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba528c4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fba4c0cbb00 0x7fba4c0cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba528c4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fba4c0cbb00 0x7fba4c0cbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40472/0 (socket says 172.21.15.83:40472) 2024-09-19T07:19:45.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba528c4640 1 -- 172.21.15.83:0/847837910 learned_addr learned my addr 172.21.15.83:0/847837910 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:45.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba520c3640 1 --2- 172.21.15.83:0/847837910 >> v2:172.21.15.184:3300/0 conn(0x7fba4c14ca60 0x7fba4c159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba528c4640 1 -- 172.21.15.83:0/847837910 >> v2:172.21.15.184:3300/0 conn(0x7fba4c14ca60 msgr2=0x7fba4c159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:45.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba528c4640 1 --2- 172.21.15.83:0/847837910 >> v2:172.21.15.184:3300/0 conn(0x7fba4c14ca60 0x7fba4c159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba528c4640 1 -- 172.21.15.83:0/847837910 >> v1:172.21.15.184:6789/0 conn(0x7fba4c0bef00 legacy=0x7fba4c156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:45.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.959+0000 7fba528c4640 1 -- 172.21.15.83:0/847837910 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4c0c9f60 con 0x7fba4c0cbb00 2024-09-19T07:19:45.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba528c4640 1 --2- 172.21.15.83:0/847837910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba4c0cbb00 0x7fba4c0cbf20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fba4c073250 tx=0x7fba4401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e17bf4cb9e4605 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:45.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba510c1640 1 -- 172.21.15.83:0/847837910 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba440a5670 con 0x7fba4c0cbb00 2024-09-19T07:19:45.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba510c1640 1 -- 172.21.15.83:0/847837910 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba440a5810 con 0x7fba4c0cbb00 2024-09-19T07:19:45.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba510c1640 1 -- 172.21.15.83:0/847837910 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba440a5be0 con 0x7fba4c0cbb00 2024-09-19T07:19:45.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 -- 172.21.15.83:0/847837910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba4c0cbb00 msgr2=0x7fba4c0cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 --2- 172.21.15.83:0/847837910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba4c0cbb00 0x7fba4c0cbf20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fba4c073250 tx=0x7fba4401c9b0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 -- 172.21.15.83:0/847837910 shutdown_connections 2024-09-19T07:19:45.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 --2- 172.21.15.83:0/847837910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba4c0cbb00 0x7fba4c0cbf20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 --2- 172.21.15.83:0/847837910 >> v2:172.21.15.184:3300/0 conn(0x7fba4c14ca60 0x7fba4c159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 -- 172.21.15.83:0/847837910 >> 172.21.15.83:0/847837910 conn(0x7fba4c072470 msgr2=0x7fba4c072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:45.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 -- 172.21.15.83:0/847837910 shutdown_connections 2024-09-19T07:19:45.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 -- 172.21.15.83:0/847837910 wait complete. 2024-09-19T07:19:45.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 Processor -- start 2024-09-19T07:19:45.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 -- start start 2024-09-19T07:19:45.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba4c0bef00 0x7fba4c14f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba4c0cbb00 0x7fba4c151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba4c14ca60 0x7fba4c152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c0c98c0 con 0x7fba4c14ca60 2024-09-19T07:19:45.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c153a50 con 0x7fba4c0bef00 2024-09-19T07:19:45.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fba4c0c9a40 con 0x7fba4c0cbb00 2024-09-19T07:19:45.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba528c4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba4c14ca60 0x7fba4c152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba528c4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba4c14ca60 0x7fba4c152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40482/0 (socket says 172.21.15.83:40482) 2024-09-19T07:19:45.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba528c4640 1 -- 172.21.15.83:0/2908263420 learned_addr learned my addr 172.21.15.83:0/2908263420 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:45.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba528c4640 1 -- 172.21.15.83:0/2908263420 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba4c0cbb00 msgr2=0x7fba4c151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba528c4640 1 --2- 172.21.15.83:0/2908263420 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba4c0cbb00 0x7fba4c151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba528c4640 1 -- 172.21.15.83:0/2908263420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba4c0bef00 msgr2=0x7fba4c14f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:45.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba528c4640 1 --2- 172.21.15.83:0/2908263420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba4c0bef00 0x7fba4c14f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:45.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba528c4640 1 -- 172.21.15.83:0/2908263420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4c0b1450 con 0x7fba4c14ca60 2024-09-19T07:19:45.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba528c4640 1 --2- 172.21.15.83:0/2908263420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba4c14ca60 0x7fba4c152450 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fba440af040 tx=0x7fba44019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:45.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba42ffd640 1 -- 172.21.15.83:0/2908263420 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba44009510 con 0x7fba4c14ca60 2024-09-19T07:19:45.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba42ffd640 1 -- 172.21.15.83:0/2908263420 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba440be070 con 0x7fba4c14ca60 2024-09-19T07:19:45.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 -- 172.21.15.83:0/2908263420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4c152ed0 con 0x7fba4c14ca60 2024-09-19T07:19:45.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba42ffd640 1 -- 172.21.15.83:0/2908263420 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba440b88b0 con 0x7fba4c14ca60 2024-09-19T07:19:45.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.963+0000 7fba54327640 1 -- 172.21.15.83:0/2908263420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4c153450 con 0x7fba4c14ca60 2024-09-19T07:19:45.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.967+0000 7fba54327640 1 -- 172.21.15.83:0/2908263420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba4c0cbf20 con 0x7fba4c14ca60 2024-09-19T07:19:45.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.967+0000 7fba42ffd640 1 -- 172.21.15.83:0/2908263420 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fba440b8a50 con 0x7fba4c14ca60 2024-09-19T07:19:45.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.967+0000 7fba42ffd640 1 --2- 172.21.15.83:0/2908263420 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba300678f0 0x7fba30069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:45.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.967+0000 7fba42ffd640 1 -- 172.21.15.83:0/2908263420 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fba44121400 con 0x7fba4c14ca60 2024-09-19T07:19:45.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.967+0000 7fba520c3640 1 --2- 172.21.15.83:0/2908263420 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba300678f0 0x7fba30069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:45.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.967+0000 7fba520c3640 1 --2- 172.21.15.83:0/2908263420 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba300678f0 0x7fba30069db0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00b850 tx=0x7fba3c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:45.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:45.971+0000 7fba42ffd640 1 -- 172.21.15.83:0/2908263420 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fba440c7020 con 0x7fba4c14ca60 2024-09-19T07:19:46.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.171+0000 7fba54327640 1 -- 172.21.15.83:0/2908263420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fba4c06ca90 con 0x7fba4c14ca60 2024-09-19T07:19:46.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.179+0000 7fba42ffd640 1 -- 172.21.15.83:0/2908263420 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+166887 (secure 0 0 0) 0x7fba440165a0 con 0x7fba4c14ca60 2024-09-19T07:19:46.182 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:46.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.183+0000 7fba54327640 1 -- 172.21.15.83:0/2908263420 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba300678f0 msgr2=0x7fba30069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.183+0000 7fba54327640 1 --2- 172.21.15.83:0/2908263420 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba300678f0 0x7fba30069db0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00b850 tx=0x7fba3c008040 comp rx=0 tx=0).stop 2024-09-19T07:19:46.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.183+0000 7fba54327640 1 -- 172.21.15.83:0/2908263420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba4c14ca60 msgr2=0x7fba4c152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.183+0000 7fba54327640 1 --2- 172.21.15.83:0/2908263420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba4c14ca60 0x7fba4c152450 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fba440af040 tx=0x7fba44019750 comp rx=0 tx=0).stop 2024-09-19T07:19:46.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.183+0000 7fba54327640 1 -- 172.21.15.83:0/2908263420 shutdown_connections 2024-09-19T07:19:46.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.183+0000 7fba54327640 1 --2- 172.21.15.83:0/2908263420 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba300678f0 0x7fba30069db0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.183+0000 7fba54327640 1 --2- 172.21.15.83:0/2908263420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba4c14ca60 0x7fba4c152450 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.183+0000 7fba54327640 1 --2- 172.21.15.83:0/2908263420 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba4c0cbb00 0x7fba4c151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.183+0000 7fba54327640 1 --2- 172.21.15.83:0/2908263420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba4c0bef00 0x7fba4c14f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.183+0000 7fba54327640 1 -- 172.21.15.83:0/2908263420 >> 172.21.15.83:0/2908263420 conn(0x7fba4c072470 msgr2=0x7fba4c1541f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:46.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.183+0000 7fba54327640 1 -- 172.21.15.83:0/2908263420 shutdown_connections 2024-09-19T07:19:46.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.183+0000 7fba54327640 1 -- 172.21.15.83:0/2908263420 wait complete. 2024-09-19T07:19:46.200 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":1237718513},{"type":"v1","addr":"172.21.15.83:6833","nonce":1237718513}]},"active_addr":"172.21.15.83:6833/1237718513","active_change":"2024-09-19T07:19:12.801007+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4104,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3470065708}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3571747417}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":1957196458}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":1052228512}]}]} 2024-09-19T07:19:46.203 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-19T07:19:46.203 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-19T07:19:46.203 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:46.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.311+0000 7f107a750640 1 Processor -- start 2024-09-19T07:19:46.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.311+0000 7f107a750640 1 -- start start 2024-09-19T07:19:46.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f107a750640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1074152b20 0x7f10740c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f107a750640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f10740c45c0 0x7f1074153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f107a750640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1074150670 con 0x7f1074150290 2024-09-19T07:19:46.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f107a750640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f10740716c0 con 0x7f10740c45c0 2024-09-19T07:19:46.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f107a750640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f10740c3920 con 0x7f1074152b20 2024-09-19T07:19:46.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f1078ced640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f10740c45c0 0x7f1074153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f1078ced640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f10740c45c0 0x7f1074153eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40486/0 (socket says 172.21.15.83:40486) 2024-09-19T07:19:46.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f1073fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1074150290 0x7f10740c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51498/0 (socket says 172.21.15.83:51498) 2024-09-19T07:19:46.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f1078ced640 1 -- 172.21.15.83:0/1281609985 learned_addr learned my addr 172.21.15.83:0/1281609985 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:46.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f10737fe640 1 --2- 172.21.15.83:0/1281609985 >> v2:172.21.15.184:3300/0 conn(0x7f1074152b20 0x7f10740c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f1078ced640 1 -- 172.21.15.83:0/1281609985 >> v2:172.21.15.184:3300/0 conn(0x7f1074152b20 msgr2=0x7f10740c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:46.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f1078ced640 1 --2- 172.21.15.83:0/1281609985 >> v2:172.21.15.184:3300/0 conn(0x7f1074152b20 0x7f10740c4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f1078ced640 1 -- 172.21.15.83:0/1281609985 >> v1:172.21.15.83:6789/0 conn(0x7f1074150290 legacy=0x7f10740c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f1078ced640 1 -- 172.21.15.83:0/1281609985 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10740c2880 con 0x7f10740c45c0 2024-09-19T07:19:46.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f1078ced640 1 --2- 172.21.15.83:0/1281609985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10740c45c0 0x7f1074153eb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1074070820 tx=0x7f106401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee38cdd56dc386b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:46.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f1072ffd640 1 -- 172.21.15.83:0/1281609985 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10640a56a0 con 0x7f10740c45c0 2024-09-19T07:19:46.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f1072ffd640 1 -- 172.21.15.83:0/1281609985 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10640a5840 con 0x7f10740c45c0 2024-09-19T07:19:46.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.315+0000 7f1072ffd640 1 -- 172.21.15.83:0/1281609985 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10640a5c10 con 0x7f10740c45c0 2024-09-19T07:19:46.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 -- 172.21.15.83:0/1281609985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10740c45c0 msgr2=0x7f1074153eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 --2- 172.21.15.83:0/1281609985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10740c45c0 0x7f1074153eb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1074070820 tx=0x7f106401c9f0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 -- 172.21.15.83:0/1281609985 shutdown_connections 2024-09-19T07:19:46.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 --2- 172.21.15.83:0/1281609985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10740c45c0 0x7f1074153eb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 --2- 172.21.15.83:0/1281609985 >> v2:172.21.15.184:3300/0 conn(0x7f1074152b20 0x7f10740c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 -- 172.21.15.83:0/1281609985 >> 172.21.15.83:0/1281609985 conn(0x7f107406bc80 msgr2=0x7f107406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:46.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 -- 172.21.15.83:0/1281609985 shutdown_connections 2024-09-19T07:19:46.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 -- 172.21.15.83:0/1281609985 wait complete. 2024-09-19T07:19:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 Processor -- start 2024-09-19T07:19:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 -- start start 2024-09-19T07:19:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10740c45c0 0x7f10740abd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1074150290 0x7f10740c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1074152b20 0x7f10741ed9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1074154980 con 0x7f1074152b20 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1074154800 con 0x7f1074150290 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f107406e6b0 con 0x7f10740c45c0 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1078ced640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1074152b20 0x7f10741ed9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1078ced640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1074152b20 0x7f10741ed9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40498/0 (socket says 172.21.15.83:40498) 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1078ced640 1 -- 172.21.15.83:0/2861818657 learned_addr learned my addr 172.21.15.83:0/2861818657 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f10737fe640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1074150290 0x7f10740c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1073fff640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10740c45c0 0x7f10740abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1078ced640 1 -- 172.21.15.83:0/2861818657 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10740c45c0 msgr2=0x7f10740abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1078ced640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10740c45c0 0x7f10740abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1078ced640 1 -- 172.21.15.83:0/2861818657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1074150290 msgr2=0x7f10740c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1078ced640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1074150290 0x7f10740c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1078ced640 1 -- 172.21.15.83:0/2861818657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10740aa090 con 0x7f1074152b20 2024-09-19T07:19:46.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1078ced640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1074152b20 0x7f10741ed9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f106401a620 tx=0x7f1064019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:46.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1070ff9640 1 -- 172.21.15.83:0/2861818657 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10640b8670 con 0x7f1074152b20 2024-09-19T07:19:46.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1070ff9640 1 -- 172.21.15.83:0/2861818657 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10640be070 con 0x7f1074152b20 2024-09-19T07:19:46.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 -- 172.21.15.83:0/2861818657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10740c14c0 con 0x7f1074152b20 2024-09-19T07:19:46.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f1070ff9640 1 -- 172.21.15.83:0/2861818657 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10640b8a50 con 0x7f1074152b20 2024-09-19T07:19:46.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 -- 172.21.15.83:0/2861818657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10740ab220 con 0x7f1074152b20 2024-09-19T07:19:46.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.319+0000 7f107a750640 1 -- 172.21.15.83:0/2861818657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10740c3e90 con 0x7f1074152b20 2024-09-19T07:19:46.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.323+0000 7f1070ff9640 1 -- 172.21.15.83:0/2861818657 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f10640b8bf0 con 0x7f1074152b20 2024-09-19T07:19:46.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.323+0000 7f1070ff9640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10540678c0 0x7f1054069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.323+0000 7f1073fff640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10540678c0 0x7f1054069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.323+0000 7f1070ff9640 1 -- 172.21.15.83:0/2861818657 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f1064121cc0 con 0x7f1074152b20 2024-09-19T07:19:46.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.323+0000 7f1073fff640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10540678c0 0x7f1054069d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f106c000f70 tx=0x7f106c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:46.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.327+0000 7f1070ff9640 1 -- 172.21.15.83:0/2861818657 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f10640165e0 con 0x7f1074152b20 2024-09-19T07:19:46.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.483+0000 7f107a750640 1 -- 172.21.15.83:0/2861818657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f107406eaa0 con 0x7f1074152b20 2024-09-19T07:19:46.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f1070ff9640 1 -- 172.21.15.83:0/2861818657 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+20451 (secure 0 0 0) 0x7f10640c23c0 con 0x7f1074152b20 2024-09-19T07:19:46.486 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:46.487 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:41.334020+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:46.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f107a750640 1 -- 172.21.15.83:0/2861818657 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10540678c0 msgr2=0x7f1054069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f107a750640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10540678c0 0x7f1054069d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f106c000f70 tx=0x7f106c009040 comp rx=0 tx=0).stop 2024-09-19T07:19:46.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f107a750640 1 -- 172.21.15.83:0/2861818657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1074152b20 msgr2=0x7f10741ed9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f107a750640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1074152b20 0x7f10741ed9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f106401a620 tx=0x7f1064019790 comp rx=0 tx=0).stop 2024-09-19T07:19:46.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f107a750640 1 -- 172.21.15.83:0/2861818657 shutdown_connections 2024-09-19T07:19:46.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f107a750640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10540678c0 0x7f1054069d80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f107a750640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1074152b20 0x7f10741ed9c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f107a750640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1074150290 0x7f10740c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f107a750640 1 --2- 172.21.15.83:0/2861818657 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10740c45c0 0x7f10740abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f107a750640 1 -- 172.21.15.83:0/2861818657 >> 172.21.15.83:0/2861818657 conn(0x7f107406bc80 msgr2=0x7f1074152f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:46.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f107a750640 1 -- 172.21.15.83:0/2861818657 shutdown_connections 2024-09-19T07:19:46.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.487+0000 7f107a750640 1 -- 172.21.15.83:0/2861818657 wait complete. 2024-09-19T07:19:46.504 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-19T07:19:46.504 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:46.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.611+0000 7fbc9ee0d640 1 Processor -- start 2024-09-19T07:19:46.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.611+0000 7fbc9ee0d640 1 -- start start 2024-09-19T07:19:46.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9ee0d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbc981545e0 0x7fbc9806e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9ee0d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc98154a70 con 0x7fbc98071100 2024-09-19T07:19:46.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9ee0d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc980a9e60 con 0x7fbc980c28e0 2024-09-19T07:19:46.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9ee0d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fbc98071b50 con 0x7fbc981545e0 2024-09-19T07:19:46.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9d3aa640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbc981545e0 0x7fbc9806e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9cba9640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbc98071100 0x7fbc98150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51506/0 (socket says 172.21.15.83:51506) 2024-09-19T07:19:46.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9d3aa640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbc981545e0 0x7fbc9806e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40510/0 (socket says 172.21.15.83:40510) 2024-09-19T07:19:46.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9cba9640 1 -- 172.21.15.83:0/3171684903 learned_addr learned my addr 172.21.15.83:0/3171684903 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:46.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9d3aa640 1 -- 172.21.15.83:0/3171684903 >> v1:172.21.15.184:6789/0 conn(0x7fbc980c28e0 legacy=0x7fbc9806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:46.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9d3aa640 1 -- 172.21.15.83:0/3171684903 >> v1:172.21.15.83:6789/0 conn(0x7fbc98071100 legacy=0x7fbc98150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9d3aa640 1 -- 172.21.15.83:0/3171684903 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc980aa140 con 0x7fbc981545e0 2024-09-19T07:19:46.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9d3aa640 1 --2- 172.21.15.83:0/3171684903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc981545e0 0x7fbc9806e5b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbc880097c0 tx=0x7fbc8801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743ac174a9fc48a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:46.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc977fe640 1 -- 172.21.15.83:0/3171684903 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc880b0070 con 0x7fbc981545e0 2024-09-19T07:19:46.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc977fe640 1 -- 172.21.15.83:0/3171684903 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc880ab600 con 0x7fbc981545e0 2024-09-19T07:19:46.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc977fe640 1 -- 172.21.15.83:0/3171684903 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc880ab9d0 con 0x7fbc981545e0 2024-09-19T07:19:46.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/3171684903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc981545e0 msgr2=0x7fbc9806e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9ee0d640 1 --2- 172.21.15.83:0/3171684903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc981545e0 0x7fbc9806e5b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbc880097c0 tx=0x7fbc8801c3d0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/3171684903 shutdown_connections 2024-09-19T07:19:46.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9ee0d640 1 --2- 172.21.15.83:0/3171684903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc981545e0 0x7fbc9806e5b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.615+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/3171684903 >> 172.21.15.83:0/3171684903 conn(0x7fbc9806bc80 msgr2=0x7fbc9806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:46.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/3171684903 shutdown_connections 2024-09-19T07:19:46.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/3171684903 wait complete. 2024-09-19T07:19:46.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 Processor -- start 2024-09-19T07:19:46.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 -- start start 2024-09-19T07:19:46.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc98071100 0x7fbc98146c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc980c28e0 0x7fbc98153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc981545e0 0x7fbc981417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbc981517d0 con 0x7fbc98071100 2024-09-19T07:19:46.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc98151650 con 0x7fbc980c28e0 2024-09-19T07:19:46.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbc980714e0 con 0x7fbc981545e0 2024-09-19T07:19:46.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9cba9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc98071100 0x7fbc98146c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9cba9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc98071100 0x7fbc98146c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40512/0 (socket says 172.21.15.83:40512) 2024-09-19T07:19:46.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9cba9640 1 -- 172.21.15.83:0/1499712131 learned_addr learned my addr 172.21.15.83:0/1499712131 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:46.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc97fff640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc980c28e0 0x7fbc98153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9cba9640 1 -- 172.21.15.83:0/1499712131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc981545e0 msgr2=0x7fbc981417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9cba9640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc981545e0 0x7fbc981417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9cba9640 1 -- 172.21.15.83:0/1499712131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc980c28e0 msgr2=0x7fbc98153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9cba9640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc980c28e0 0x7fbc98153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9cba9640 1 -- 172.21.15.83:0/1499712131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc980ab770 con 0x7fbc98071100 2024-09-19T07:19:46.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc97fff640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc980c28e0 0x7fbc98153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:46.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9cba9640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc98071100 0x7fbc98146c30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000ca70 tx=0x7fbc9000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:46.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc957fa640 1 -- 172.21.15.83:0/1499712131 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc90017070 con 0x7fbc98071100 2024-09-19T07:19:46.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc957fa640 1 -- 172.21.15.83:0/1499712131 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc9000abd0 con 0x7fbc98071100 2024-09-19T07:19:46.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc957fa640 1 -- 172.21.15.83:0/1499712131 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc90012400 con 0x7fbc98071100 2024-09-19T07:19:46.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/1499712131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc980aba50 con 0x7fbc98071100 2024-09-19T07:19:46.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/1499712131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc98152180 con 0x7fbc98071100 2024-09-19T07:19:46.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.619+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/1499712131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc98152650 con 0x7fbc98071100 2024-09-19T07:19:46.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.623+0000 7fbc957fa640 1 -- 172.21.15.83:0/1499712131 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbc9001a070 con 0x7fbc98071100 2024-09-19T07:19:46.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.623+0000 7fbc957fa640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc780678c0 0x7fbc78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.623+0000 7fbc97fff640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc780678c0 0x7fbc78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.623+0000 7fbc957fa640 1 -- 172.21.15.83:0/1499712131 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fbc90085cd0 con 0x7fbc98071100 2024-09-19T07:19:46.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.623+0000 7fbc97fff640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc780678c0 0x7fbc78069d80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbc980c1fc0 tx=0x7fbc80008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:46.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.627+0000 7fbc957fa640 1 -- 172.21.15.83:0/1499712131 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbc90056570 con 0x7fbc98071100 2024-09-19T07:19:46.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.775+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/1499712131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc98150f40 con 0x7fbc98071100 2024-09-19T07:19:46.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc957fa640 1 -- 172.21.15.83:0/1499712131 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+20451 (secure 0 0 0) 0x7fbc90055e70 con 0x7fbc98071100 2024-09-19T07:19:46.778 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:46.779 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:41.334020+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:46.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/1499712131 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc780678c0 msgr2=0x7fbc78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc9ee0d640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc780678c0 0x7fbc78069d80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbc980c1fc0 tx=0x7fbc80008040 comp rx=0 tx=0).stop 2024-09-19T07:19:46.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/1499712131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc98071100 msgr2=0x7fbc98146c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc9ee0d640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc98071100 0x7fbc98146c30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000ca70 tx=0x7fbc9000cf30 comp rx=0 tx=0).stop 2024-09-19T07:19:46.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/1499712131 shutdown_connections 2024-09-19T07:19:46.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc9ee0d640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc780678c0 0x7fbc78069d80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc9ee0d640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc981545e0 0x7fbc981417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc9ee0d640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc980c28e0 0x7fbc98153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc9ee0d640 1 --2- 172.21.15.83:0/1499712131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc98071100 0x7fbc98146c30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/1499712131 >> 172.21.15.83:0/1499712131 conn(0x7fbc9806bc80 msgr2=0x7fbc9806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:46.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/1499712131 shutdown_connections 2024-09-19T07:19:46.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.779+0000 7fbc9ee0d640 1 -- 172.21.15.83:0/1499712131 wait complete. 2024-09-19T07:19:46.796 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-19T07:19:46.797 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-19T07:19:46.797 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-19T07:19:46.798 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-19T07:19:46.798 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-19T07:19:46.799 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-19T07:19:46.799 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-19T07:19:46.800 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-19T07:19:46.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.971+0000 7f528cbab640 1 Processor -- start 2024-09-19T07:19:46.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.971+0000 7f528cbab640 1 -- start start 2024-09-19T07:19:46.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f528cbab640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5288155b00 0x7f528806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f528cbab640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52880c3440 con 0x7f5288148570 2024-09-19T07:19:46.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f528cbab640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52880c3720 con 0x7f52880c2ea0 2024-09-19T07:19:46.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f528cbab640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5288155670 con 0x7f5288155b00 2024-09-19T07:19:46.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f528659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5288148570 0x7f52880aa9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51520/0 (socket says 172.21.15.83:51520) 2024-09-19T07:19:46.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f528659c640 1 -- 172.21.15.83:0/3153045874 learned_addr learned my addr 172.21.15.83:0/3153045874 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:46.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f991b49a640 1 Processor -- start 2024-09-19T07:19:46.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f5286d9d640 1 --2- 172.21.15.83:0/3153045874 >> v2:172.21.15.184:3300/0 conn(0x7f5288155b00 0x7f528806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f991b49a640 1 -- start start 2024-09-19T07:19:46.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f5286d9d640 1 -- 172.21.15.83:0/3153045874 >> v1:172.21.15.184:6789/0 conn(0x7f52880c2ea0 legacy=0x7f52881525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f5286d9d640 1 -- 172.21.15.83:0/3153045874 >> v1:172.21.15.83:6789/0 conn(0x7f5288148570 legacy=0x7f52880aa9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f5286d9d640 1 -- 172.21.15.83:0/3153045874 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5288155300 con 0x7f5288155b00 2024-09-19T07:19:46.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f991b49a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99140c7e90 0x7f99140aa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f991b49a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9914148570 0x7f9914155b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f991b49a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99140aaf20 con 0x7f99140c2ea0 2024-09-19T07:19:46.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f991b49a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f9914155670 con 0x7f9914148570 2024-09-19T07:19:46.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f991b49a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f99140c3440 con 0x7f99140c7e90 2024-09-19T07:19:46.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f9919a37640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9914148570 0x7f9914155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f9919236640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99140c7e90 0x7f99140aa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f9919236640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99140c7e90 0x7f99140aa9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:36726/0 (socket says 172.21.15.83:36726) 2024-09-19T07:19:46.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f9919236640 1 -- 172.21.15.83:0/3449263328 learned_addr learned my addr 172.21.15.83:0/3449263328 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:46.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f9919236640 1 -- 172.21.15.83:0/3449263328 >> v2:172.21.15.184:3300/0 conn(0x7f9914148570 msgr2=0x7f9914155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:46.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f9919236640 1 --2- 172.21.15.83:0/3449263328 >> v2:172.21.15.184:3300/0 conn(0x7f9914148570 0x7f9914155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f9919236640 1 -- 172.21.15.83:0/3449263328 >> v1:172.21.15.184:6789/0 conn(0x7f99140c2ea0 legacy=0x7f99141525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:46.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.975+0000 7f9919236640 1 -- 172.21.15.83:0/3449263328 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99140c3720 con 0x7f99140c7e90 2024-09-19T07:19:46.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbefaab7640 1 Processor -- start 2024-09-19T07:19:46.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbefaab7640 1 -- start start 2024-09-19T07:19:46.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f0aca542640 1 Processor -- start 2024-09-19T07:19:46.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f0aca542640 1 -- start start 2024-09-19T07:19:46.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f320ab640 1 Processor -- start 2024-09-19T07:19:46.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f320ab640 1 -- start start 2024-09-19T07:19:46.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f5286d9d640 1 --2- 172.21.15.83:0/3153045874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5288155b00 0x7f528806da50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f528000aea0 tx=0x7f528001eaf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bec4998abf69540d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:46.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fdfa6e46640 1 Processor -- start 2024-09-19T07:19:46.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbefaab7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbef40c8d40 0x7fbef41532b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f528559a640 1 -- 172.21.15.83:0/3153045874 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52800a78d0 con 0x7f5288155b00 2024-09-19T07:19:46.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbefaab7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbef41537f0 0x7fbef40ca140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f0aca542640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ac4159e60 0x7f0ac40c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbefaab7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef4155f90 con 0x7fbef40c6370 2024-09-19T07:19:46.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbef3fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbef40c8d40 0x7fbef41532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbef3fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbef40c8d40 0x7fbef41532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:36736/0 (socket says 172.21.15.83:36736) 2024-09-19T07:19:46.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbef3fff640 1 -- 172.21.15.83:0/846928742 learned_addr learned my addr 172.21.15.83:0/846928742 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:46.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbef9054640 1 --2- 172.21.15.83:0/846928742 >> v2:172.21.15.184:3300/0 conn(0x7fbef41537f0 0x7fbef40ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbef8853640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fbef40c6370 0x7fbef4159e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:42490/0 (socket says 172.21.15.83:42490) 2024-09-19T07:19:46.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f0ac3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0ac41535d0 0x7f0ac40c7e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51524/0 (socket says 172.21.15.83:51524) 2024-09-19T07:19:46.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f320ab640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f2c150dd0 0x7f4f2c0c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbefaab7640 1 -- 172.21.15.83:0/846928742 --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbef4156b30 con 0x7fbef41537f0 2024-09-19T07:19:46.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f0ac3fff640 1 -- 172.21.15.83:0/1419213941 learned_addr learned my addr 172.21.15.83:0/1419213941 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:46.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbefaab7640 1 -- 172.21.15.83:0/846928742 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbef4156200 con 0x7fbef40c8d40 2024-09-19T07:19:46.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f320ab640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f2c070de0 con 0x7f4f2c0c1d70 2024-09-19T07:19:46.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f0aca542640 1 --2- 172.21.15.83:0/1419213941 >> v2:172.21.15.184:3300/0 conn(0x7f0ac40c44b0 0x7f0ac40c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f2b7fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f2c150dd0 0x7f4f2c0c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f2bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4f2c0c1d70 0x7f4f2c0c2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51534/0 (socket says 172.21.15.83:51534) 2024-09-19T07:19:46.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f2bfff640 1 -- 172.21.15.83:0/2843690095 learned_addr learned my addr 172.21.15.83:0/2843690095 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:46.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f320ab640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f2c0710c0 con 0x7f4f2c071550 2024-09-19T07:19:46.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fdfa6e46640 1 -- start start 2024-09-19T07:19:46.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f320ab640 1 -- 172.21.15.83:0/2843690095 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4f2c0c3040 con 0x7f4f2c150dd0 2024-09-19T07:19:46.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f0abbfff640 1 --2- 172.21.15.83:0/1419213941 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ac4159e60 0x7f0ac40c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f528559a640 1 -- 172.21.15.83:0/3153045874 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52800a7a70 con 0x7f5288155b00 2024-09-19T07:19:46.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f528559a640 1 -- 172.21.15.83:0/3153045874 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52800a7e40 con 0x7f5288155b00 2024-09-19T07:19:46.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f528cbab640 1 -- 172.21.15.83:0/3153045874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5288155b00 msgr2=0x7f528806da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f528cbab640 1 --2- 172.21.15.83:0/3153045874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5288155b00 0x7f528806da50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f528000aea0 tx=0x7f528001eaf0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f528cbab640 1 -- 172.21.15.83:0/3153045874 shutdown_connections 2024-09-19T07:19:46.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f528cbab640 1 --2- 172.21.15.83:0/3153045874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5288155b00 0x7f528806da50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f528cbab640 1 -- 172.21.15.83:0/3153045874 >> 172.21.15.83:0/3153045874 conn(0x7f52880c7370 msgr2=0x7f52880c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:46.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fdfa6e46640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fdf9800c400 0x7fdf98046f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fdfa6e46640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf9804acd0 0x7fdf980de7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fdfa6e46640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf980e2540 con 0x7fdf9800c020 2024-09-19T07:19:46.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fdfa6e46640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fdf9803f560 con 0x7fdf9800c400 2024-09-19T07:19:46.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fdfa6e46640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdf980e16c0 con 0x7fdf9804acd0 2024-09-19T07:19:46.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fdfa5e44640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdf9800c020 0x7fdf980e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51536/0 (socket says 172.21.15.83:51536) 2024-09-19T07:19:46.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fdfa5e44640 1 -- 172.21.15.83:0/397396416 learned_addr learned my addr 172.21.15.83:0/397396416 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:46.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbef37fe640 1 -- 172.21.15.83:0/846928742 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1458292407 0 0) 0x7fbef4155f90 con 0x7fbef40c6370 2024-09-19T07:19:46.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbef37fe640 1 -- 172.21.15.83:0/846928742 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbee4003530 con 0x7fbef40c6370 2024-09-19T07:19:46.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f9919236640 1 --2- 172.21.15.83:0/3449263328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99140c7e90 0x7f99140aa9e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9910009870 tx=0x7f991001c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=819afab3c518a2df server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:46.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fdfa6645640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf9804acd0 0x7fdf980de7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1703481727 0 0) 0x7fdf980e2540 con 0x7fdf9800c020 2024-09-19T07:19:46.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf8c003530 con 0x7fdf9800c020 2024-09-19T07:19:46.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f0aca542640 1 -- 172.21.15.83:0/1419213941 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ac40c39f0 con 0x7f0ac41535d0 2024-09-19T07:19:46.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f0aca542640 1 -- 172.21.15.83:0/1419213941 --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f0ac40c3cd0 con 0x7f0ac40c44b0 2024-09-19T07:19:46.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f0aca542640 1 -- 172.21.15.83:0/1419213941 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0ac4159380 con 0x7f0ac4159e60 2024-09-19T07:19:46.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f528cbab640 1 -- 172.21.15.83:0/3153045874 shutdown_connections 2024-09-19T07:19:46.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f2a7fc640 1 -- 172.21.15.83:0/2843690095 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1784814153 0 0) 0x7f4f2c070de0 con 0x7f4f2c0c1d70 2024-09-19T07:19:46.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f2a7fc640 1 -- 172.21.15.83:0/2843690095 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f14003530 con 0x7f4f2c0c1d70 2024-09-19T07:19:46.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f2a7fc640 1 -- 172.21.15.83:0/2843690095 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1993448100 0 0) 0x7f4f2c0710c0 con 0x7f4f2c071550 2024-09-19T07:19:46.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f2a7fc640 1 -- 172.21.15.83:0/2843690095 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f2c070de0 con 0x7f4f2c071550 2024-09-19T07:19:46.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f0ac8adf640 1 --2- 172.21.15.83:0/1419213941 >> v2:172.21.15.184:3300/0 conn(0x7f0ac40c44b0 0x7f0ac40c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbef3fff640 1 -- 172.21.15.83:0/846928742 >> v2:172.21.15.184:3300/0 conn(0x7fbef41537f0 msgr2=0x7fbef40ca140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:46.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbef3fff640 1 --2- 172.21.15.83:0/846928742 >> v2:172.21.15.184:3300/0 conn(0x7fbef41537f0 0x7fbef40ca140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbef3fff640 1 -- 172.21.15.83:0/846928742 >> v1:172.21.15.184:6789/0 conn(0x7fbef40c6370 legacy=0x7fbef4159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7fbef3fff640 1 -- 172.21.15.83:0/846928742 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef41564e0 con 0x7fbef40c8d40 2024-09-19T07:19:46.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f0abbfff640 1 -- 172.21.15.83:0/1419213941 >> v2:172.21.15.184:3300/0 conn(0x7f0ac40c44b0 msgr2=0x7f0ac40c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:46.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f0abbfff640 1 --2- 172.21.15.83:0/1419213941 >> v2:172.21.15.184:3300/0 conn(0x7f0ac40c44b0 0x7f0ac40c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f0abbfff640 1 -- 172.21.15.83:0/1419213941 >> v1:172.21.15.83:6789/0 conn(0x7f0ac41535d0 legacy=0x7f0ac40c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f990bfff640 1 -- 172.21.15.83:0/3449263328 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99100b0070 con 0x7f99140c7e90 2024-09-19T07:19:46.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f990bfff640 1 -- 172.21.15.83:0/3449263328 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99100ab400 con 0x7f99140c7e90 2024-09-19T07:19:46.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f990bfff640 1 -- 172.21.15.83:0/3449263328 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99100ab7d0 con 0x7f99140c7e90 2024-09-19T07:19:46.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6cf59e640 1 Processor -- start 2024-09-19T07:19:46.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6cf59e640 1 -- start start 2024-09-19T07:19:46.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f991b49a640 1 -- 172.21.15.83:0/3449263328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99140c7e90 msgr2=0x7f99140aa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f991b49a640 1 --2- 172.21.15.83:0/3449263328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99140c7e90 0x7f99140aa9e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9910009870 tx=0x7f991001c460 comp rx=0 tx=0).stop 2024-09-19T07:19:46.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa5643640 1 --2- 172.21.15.83:0/397396416 >> v2:172.21.15.184:3300/0 conn(0x7fdf9800c400 0x7fdf98046f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:46.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f528cbab640 1 -- 172.21.15.83:0/3153045874 wait complete. 2024-09-19T07:19:46.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f528cbab640 1 Processor -- start 2024-09-19T07:19:46.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f2b7fe640 1 -- 172.21.15.83:0/2843690095 >> v1:172.21.15.184:6789/0 conn(0x7f4f2c071550 legacy=0x7f4f2c153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f2b7fe640 1 -- 172.21.15.83:0/2843690095 >> v1:172.21.15.83:6789/0 conn(0x7f4f2c0c1d70 legacy=0x7f4f2c0c2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.979+0000 7f4f2b7fe640 1 -- 172.21.15.83:0/2843690095 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2c0c3320 con 0x7f4f2c150dd0 2024-09-19T07:19:46.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3979149354 0 0) 0x7fdf8c003530 con 0x7fdf9800c020 2024-09-19T07:19:46.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf9804acd0 msgr2=0x7fdf980de7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:46.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf9804acd0 0x7fdf980de7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 >> v2:172.21.15.184:3300/0 conn(0x7fdf9800c400 msgr2=0x7fdf98046f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:46.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 --2- 172.21.15.83:0/397396416 >> v2:172.21.15.184:3300/0 conn(0x7fdf9800c400 0x7fdf98046f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:46.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9803f7b0 con 0x7fdf9800c020 2024-09-19T07:19:46.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fdf9c0016a0 con 0x7fdf9800c020 2024-09-19T07:19:46.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa5643640 1 --2- 172.21.15.83:0/397396416 >> v2:172.21.15.184:3300/0 conn(0x7fdf9800c400 0x7fdf98046f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:46.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 >> v1:172.21.15.83:6789/0 conn(0x7fdf9800c020 legacy=0x7fdf980e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:46.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf8c0039c0 0x7fdf8c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf8c004810 0x7fdf8c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf8c0053d0 0x7fdf8c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:46.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9803f560 con 0x7fdf8c0039c0 2024-09-19T07:19:46.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdf980e16c0 con 0x7fdf8c004810 2024-09-19T07:19:46.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdf8c00c270 con 0x7fdf8c0053d0 2024-09-19T07:19:46.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa6645640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf9804acd0 0x7fdf980de7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:47.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f528cbab640 1 -- start start 2024-09-19T07:19:47.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa5e44640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf8c0039c0 0x7fdf8c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f991b49a640 1 -- 172.21.15.83:0/3449263328 shutdown_connections 2024-09-19T07:19:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f991b49a640 1 --2- 172.21.15.83:0/3449263328 >> v2:172.21.15.184:3300/0 conn(0x7f9914148570 0x7f9914155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f991b49a640 1 --2- 172.21.15.83:0/3449263328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99140c7e90 0x7f99140aa9e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f991b49a640 1 -- 172.21.15.83:0/3449263328 >> 172.21.15.83:0/3449263328 conn(0x7f99140c7370 msgr2=0x7f99140c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa6645640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf8c0053d0 0x7fdf8c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6cf59e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6d01541d0 0x7ff6d00c3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6cf59e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff6d00ad270 0x7ff6d0150e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6cf59e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d00ac7d0 con 0x7ff6d0071360 2024-09-19T07:19:47.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6cf59e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff6d006ece0 con 0x7ff6d00ad270 2024-09-19T07:19:47.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6cf59e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d00c3870 con 0x7ff6d01541d0 2024-09-19T07:19:47.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6ced9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff6d00ad270 0x7ff6d0150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6ced9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff6d00ad270 0x7ff6d0150e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:32908/0 (socket says 172.21.15.83:32908) 2024-09-19T07:19:47.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6ced9d640 1 -- 172.21.15.83:0/2760186195 learned_addr learned my addr 172.21.15.83:0/2760186195 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f0abbfff640 1 -- 172.21.15.83:0/1419213941 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac4159660 con 0x7f0ac4159e60 2024-09-19T07:19:47.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa5e44640 1 -- 172.21.15.83:0/397396416 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf8c0053d0 msgr2=0x7fdf8c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa5e44640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf8c0053d0 0x7fdf8c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa5e44640 1 -- 172.21.15.83:0/397396416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf8c004810 msgr2=0x7fdf8c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa5643640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf8c004810 0x7fdf8c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fbef3fff640 1 --2- 172.21.15.83:0/846928742 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbef40c8d40 0x7fbef41532b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009870 tx=0x7fbee001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=981fe7f44ced91c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7faf8cfd2640 1 Processor -- start 2024-09-19T07:19:47.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f991b49a640 1 -- 172.21.15.83:0/3449263328 shutdown_connections 2024-09-19T07:19:47.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f991b49a640 1 -- 172.21.15.83:0/3449263328 wait complete. 2024-09-19T07:19:47.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f528cbab640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52880c2ea0 0x7f5288146f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f528cbab640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5288148570 0x7f52880ab200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f528cbab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5288155b00 0x7f52880ab740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f528cbab640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f52881600b0 con 0x7f5288155b00 2024-09-19T07:19:47.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f528cbab640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f528815ff30 con 0x7f52880c2ea0 2024-09-19T07:19:47.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f528cbab640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5288160230 con 0x7f5288148570 2024-09-19T07:19:47.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa5e44640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf8c004810 0x7fdf8c004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7fdfa5e44640 1 -- 172.21.15.83:0/397396416 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9803f7b0 con 0x7fdf8c0039c0 2024-09-19T07:19:47.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7f4f2b7fe640 1 --2- 172.21.15.83:0/2843690095 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f2c150dd0 0x7f4f2c0c3840 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4f200097f0 tx=0x7f4f2001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cf887204cfd72bc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7faf8cfd2640 1 -- start start 2024-09-19T07:19:47.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7faf8cfd2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7faf800d35a0 0x7faf800d9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7faf8cfd2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf80046660 0x7faf8000acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7faf8cfd2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf800d3980 con 0x7faf8000b210 2024-09-19T07:19:47.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7faf8cfd2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7faf80031290 con 0x7faf80046660 2024-09-19T07:19:47.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7faf8cfd2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7faf800da1c0 con 0x7faf800d35a0 2024-09-19T07:19:47.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7faf85d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf80046660 0x7faf8000acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f528659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52880c2ea0 0x7f5288146f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f528659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52880c2ea0 0x7f5288146f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:32912/0 (socket says 172.21.15.83:32912) 2024-09-19T07:19:47.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f528659c640 1 -- 172.21.15.83:0/922999186 learned_addr learned my addr 172.21.15.83:0/922999186 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f5285d9b640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5288148570 0x7f52880ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f5286d9d640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5288155b00 0x7f52880ab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7faf85d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf80046660 0x7faf8000acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40532/0 (socket says 172.21.15.83:40532) 2024-09-19T07:19:47.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7faf85d9b640 1 -- 172.21.15.83:0/2557569107 learned_addr learned my addr 172.21.15.83:0/2557569107 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6ced9d640 1 -- 172.21.15.83:0/2760186195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6d01541d0 msgr2=0x7ff6d00c3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6ced9d640 1 --2- 172.21.15.83:0/2760186195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6d01541d0 0x7ff6d00c3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6ced9d640 1 -- 172.21.15.83:0/2760186195 >> v1:172.21.15.184:6789/0 conn(0x7ff6d0071360 legacy=0x7ff6d0153ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:19:47.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7faf8659c640 1 --2- 172.21.15.83:0/2557569107 >> v2:172.21.15.184:3300/0 conn(0x7faf800d35a0 0x7faf800d9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7faf85d9b640 1 -- 172.21.15.83:0/2557569107 >> v2:172.21.15.184:3300/0 conn(0x7faf800d35a0 msgr2=0x7faf800d9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7faf85d9b640 1 --2- 172.21.15.83:0/2557569107 >> v2:172.21.15.184:3300/0 conn(0x7faf800d35a0 0x7faf800d9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7faf85d9b640 1 -- 172.21.15.83:0/2557569107 >> v1:172.21.15.184:6789/0 conn(0x7faf8000b210 legacy=0x7faf80005a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.983+0000 7ff6ced9d640 1 -- 172.21.15.83:0/2760186195 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d00aca70 con 0x7ff6d00ad270 2024-09-19T07:19:47.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7faf85d9b640 1 -- 172.21.15.83:0/2557569107 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf80045cf0 con 0x7faf80046660 2024-09-19T07:19:47.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fbef37fe640 1 -- 172.21.15.83:0/846928742 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee00b1070 con 0x7fbef40c8d40 2024-09-19T07:19:47.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fbef37fe640 1 -- 172.21.15.83:0/846928742 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbee00a8050 con 0x7fbef40c8d40 2024-09-19T07:19:47.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fbef37fe640 1 -- 172.21.15.83:0/846928742 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee00ac860 con 0x7fbef40c8d40 2024-09-19T07:19:47.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f991b49a640 1 Processor -- start 2024-09-19T07:19:47.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f0abbfff640 1 --2- 172.21.15.83:0/1419213941 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ac4159e60 0x7f0ac40c3f70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0097c0 tx=0x7f0aac01c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ff41b7e66a272f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f5286d9d640 1 -- 172.21.15.83:0/922999186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5288148570 msgr2=0x7f52880ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f5286d9d640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5288148570 0x7f52880ab200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f5286d9d640 1 -- 172.21.15.83:0/922999186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52880c2ea0 msgr2=0x7f5288146f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f5286d9d640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52880c2ea0 0x7f5288146f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f5286d9d640 1 -- 172.21.15.83:0/922999186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52881543a0 con 0x7f5288155b00 2024-09-19T07:19:47.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f5286d9d640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5288155b00 0x7f52880ab740 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f52880ad6e0 tx=0x7f528001b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f4f2a7fc640 1 -- 172.21.15.83:0/2843690095 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f200b0070 con 0x7f4f2c150dd0 2024-09-19T07:19:47.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f4f2a7fc640 1 -- 172.21.15.83:0/2843690095 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f200ab600 con 0x7f4f2c150dd0 2024-09-19T07:19:47.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f4f2a7fc640 1 -- 172.21.15.83:0/2843690095 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f200ab9d0 con 0x7f4f2c150dd0 2024-09-19T07:19:47.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f0ac37fe640 1 -- 172.21.15.83:0/1419213941 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac0b0070 con 0x7f0ac4159e60 2024-09-19T07:19:47.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f0ac37fe640 1 -- 172.21.15.83:0/1419213941 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0aac0ab600 con 0x7f0ac4159e60 2024-09-19T07:19:47.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f0ac37fe640 1 -- 172.21.15.83:0/1419213941 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac0ab9d0 con 0x7f0ac4159e60 2024-09-19T07:19:47.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fbefaab7640 1 -- 172.21.15.83:0/846928742 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbef40c8d40 msgr2=0x7fbef41532b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fbefaab7640 1 --2- 172.21.15.83:0/846928742 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbef40c8d40 0x7fbef41532b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009870 tx=0x7fbee001c4d0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fbefaab7640 1 -- 172.21.15.83:0/846928742 shutdown_connections 2024-09-19T07:19:47.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fbefaab7640 1 --2- 172.21.15.83:0/846928742 >> v2:172.21.15.184:3300/0 conn(0x7fbef41537f0 0x7fbef40ca140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fbefaab7640 1 --2- 172.21.15.83:0/846928742 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbef40c8d40 0x7fbef41532b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fbefaab7640 1 -- 172.21.15.83:0/846928742 >> 172.21.15.83:0/846928742 conn(0x7fbef406bc80 msgr2=0x7fbef406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fbefaab7640 1 -- 172.21.15.83:0/846928742 shutdown_connections 2024-09-19T07:19:47.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f5276ffd640 1 -- 172.21.15.83:0/922999186 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528000a560 con 0x7f5288155b00 2024-09-19T07:19:47.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f5276ffd640 1 -- 172.21.15.83:0/922999186 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f528000a0d0 con 0x7f5288155b00 2024-09-19T07:19:47.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f5276ffd640 1 -- 172.21.15.83:0/922999186 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52800b9610 con 0x7f5288155b00 2024-09-19T07:19:47.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f4f320ab640 1 -- 172.21.15.83:0/2843690095 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f2c150dd0 msgr2=0x7f4f2c0c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f4f320ab640 1 --2- 172.21.15.83:0/2843690095 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f2c150dd0 0x7f4f2c0c3840 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4f200097f0 tx=0x7f4f2001c450 comp rx=0 tx=0).stop 2024-09-19T07:19:47.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f991b49a640 1 -- start start 2024-09-19T07:19:47.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f991b49a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99140c2ea0 0x7f99140c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f991b49a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99140c7e90 0x7f9914147620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f991b49a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9914148570 0x7f9914147b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f991b49a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9914156640 con 0x7f99140c2ea0 2024-09-19T07:19:47.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f991b49a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f99141564c0 con 0x7f9914148570 2024-09-19T07:19:47.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f991b49a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f99141567c0 con 0x7f99140c7e90 2024-09-19T07:19:47.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7ff6ced9d640 1 --2- 172.21.15.83:0/2760186195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6d00ad270 0x7ff6d0150e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff6d00c2ec0 tx=0x7ff6c8025030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8450b3600b204d04 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7ff6cd59a640 1 -- 172.21.15.83:0/2760186195 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c80ba070 con 0x7ff6d00ad270 2024-09-19T07:19:47.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fdfa5e44640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf8c0039c0 0x7fdf8c003de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdf98043140 tx=0x7fdf9c01ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c11baf40fcd41c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c007de0 con 0x7fdf8c0039c0 2024-09-19T07:19:47.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf9c0a9020 con 0x7fdf8c0039c0 2024-09-19T07:19:47.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fdfa4e42640 1 -- 172.21.15.83:0/397396416 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c005ae0 con 0x7fdf8c0039c0 2024-09-19T07:19:47.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f0aca542640 1 -- 172.21.15.83:0/1419213941 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ac4159e60 msgr2=0x7f0ac40c3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f0aca542640 1 --2- 172.21.15.83:0/1419213941 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ac4159e60 0x7f0ac40c3f70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0097c0 tx=0x7f0aac01c430 comp rx=0 tx=0).stop 2024-09-19T07:19:47.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f0aca542640 1 -- 172.21.15.83:0/1419213941 shutdown_connections 2024-09-19T07:19:47.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f9919a37640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9914148570 0x7f9914147b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f9919a37640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9914148570 0x7f9914147b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:32930/0 (socket says 172.21.15.83:32930) 2024-09-19T07:19:47.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f9919a37640 1 -- 172.21.15.83:0/4246206606 learned_addr learned my addr 172.21.15.83:0/4246206606 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f0aca542640 1 --2- 172.21.15.83:0/1419213941 >> v2:172.21.15.184:3300/0 conn(0x7f0ac40c44b0 0x7f0ac40c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f0aca542640 1 --2- 172.21.15.83:0/1419213941 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ac4159e60 0x7f0ac40c3f70 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0097c0 tx=0x7f0aac01c430 comp rx=0 tx=0).stop 2024-09-19T07:19:47.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f0aca542640 1 -- 172.21.15.83:0/1419213941 >> 172.21.15.83:0/1419213941 conn(0x7f0ac40c6070 msgr2=0x7f0ac40c6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f528cbab640 1 -- 172.21.15.83:0/922999186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5288153d50 con 0x7f5288155b00 2024-09-19T07:19:47.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f528cbab640 1 -- 172.21.15.83:0/922999186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5288154660 con 0x7f5288155b00 2024-09-19T07:19:47.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f4f320ab640 1 -- 172.21.15.83:0/2843690095 shutdown_connections 2024-09-19T07:19:47.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f4f320ab640 1 --2- 172.21.15.83:0/2843690095 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f2c150dd0 0x7f4f2c0c3840 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f4f320ab640 1 -- 172.21.15.83:0/2843690095 >> 172.21.15.83:0/2843690095 conn(0x7f4f2c0c7370 msgr2=0x7f4f2c0c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fdfa6e46640 1 -- 172.21.15.83:0/397396416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf8c0039c0 msgr2=0x7fdf8c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fdfa6e46640 1 --2- 172.21.15.83:0/397396416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf8c0039c0 0x7fdf8c003de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdf98043140 tx=0x7fdf9c01ccc0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fdfa6645640 1 -- 172.21.15.83:0/397396416 reap_dead start 2024-09-19T07:19:47.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fdfa6e46640 1 -- 172.21.15.83:0/397396416 shutdown_connections 2024-09-19T07:19:47.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fdfa6e46640 1 -- 172.21.15.83:0/397396416 >> 172.21.15.83:0/397396416 conn(0x7fdf98048e40 msgr2=0x7fdf98049250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f0aca542640 1 -- 172.21.15.83:0/1419213941 shutdown_connections 2024-09-19T07:19:47.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fbefaab7640 1 -- 172.21.15.83:0/846928742 wait complete. 2024-09-19T07:19:47.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fbefaab7640 1 Processor -- start 2024-09-19T07:19:47.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7ff6cd59a640 1 -- 172.21.15.83:0/2760186195 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c80adbb0 con 0x7ff6d00ad270 2024-09-19T07:19:47.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7ff6cd59a640 1 -- 172.21.15.83:0/2760186195 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c80b5440 con 0x7ff6d00ad270 2024-09-19T07:19:47.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7ff6cf59e640 1 -- 172.21.15.83:0/2760186195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6d00ad270 msgr2=0x7ff6d0150e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7ff6cf59e640 1 --2- 172.21.15.83:0/2760186195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6d00ad270 0x7ff6d0150e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff6d00c2ec0 tx=0x7ff6c8025030 comp rx=0 tx=0).stop 2024-09-19T07:19:47.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f0aca542640 1 -- 172.21.15.83:0/1419213941 wait complete. 2024-09-19T07:19:47.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f4f320ab640 1 -- 172.21.15.83:0/2843690095 shutdown_connections 2024-09-19T07:19:47.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fdfa6e46640 1 -- 172.21.15.83:0/397396416 shutdown_connections 2024-09-19T07:19:47.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f9919236640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99140c2ea0 0x7f99140c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7ff6cf59e640 1 -- 172.21.15.83:0/2760186195 shutdown_connections 2024-09-19T07:19:47.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7ff6cf59e640 1 --2- 172.21.15.83:0/2760186195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6d00ad270 0x7ff6d0150e30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7ff6cf59e640 1 --2- 172.21.15.83:0/2760186195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6d01541d0 0x7ff6d00c3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7ff6cf59e640 1 -- 172.21.15.83:0/2760186195 >> 172.21.15.83:0/2760186195 conn(0x7ff6d006dbc0 msgr2=0x7ff6d006dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f9919a37640 1 -- 172.21.15.83:0/4246206606 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99140c7e90 msgr2=0x7f9914147620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f9919a37640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99140c7e90 0x7f9914147620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f9919a37640 1 -- 172.21.15.83:0/4246206606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99140c2ea0 msgr2=0x7f99140c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f9919a37640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99140c2ea0 0x7f99140c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f9919a37640 1 -- 172.21.15.83:0/4246206606 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99140721b0 con 0x7f9914148570 2024-09-19T07:19:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f5274ff9640 1 -- 172.21.15.83:0/922999186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5260001040 con 0x7f5288155b00 2024-09-19T07:19:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7fdfa6e46640 1 -- 172.21.15.83:0/397396416 wait complete. 2024-09-19T07:19:47.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f9919a37640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9914148570 0x7f9914147b60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f990c00efc0 tx=0x7f990c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7faf85d9b640 1 --2- 172.21.15.83:0/2557569107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf80046660 0x7faf8000acd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faf80007ee0 tx=0x7faf7001d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498b39bc00cb3661 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbefaab7640 1 -- start start 2024-09-19T07:19:47.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7faf8559a640 1 -- 172.21.15.83:0/2557569107 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf700ac640 con 0x7faf80046660 2024-09-19T07:19:47.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7faf8559a640 1 -- 172.21.15.83:0/2557569107 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf700b1070 con 0x7faf80046660 2024-09-19T07:19:47.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7faf8559a640 1 -- 172.21.15.83:0/2557569107 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf700aca00 con 0x7faf80046660 2024-09-19T07:19:47.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f9909ffb640 1 -- 172.21.15.83:0/4246206606 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c018020 con 0x7f9914148570 2024-09-19T07:19:47.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f991b49a640 1 -- 172.21.15.83:0/4246206606 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9914071ed0 con 0x7f9914148570 2024-09-19T07:19:47.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f991b49a640 1 -- 172.21.15.83:0/4246206606 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f991414bd90 con 0x7f9914148570 2024-09-19T07:19:47.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f9909ffb640 1 -- 172.21.15.83:0/4246206606 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f990c014480 con 0x7f9914148570 2024-09-19T07:19:47.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f9909ffb640 1 -- 172.21.15.83:0/4246206606 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c014870 con 0x7f9914148570 2024-09-19T07:19:47.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbefaab7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbef40c6370 0x7fbef40c10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbefaab7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbef40c8d40 0x7fbef40c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbefaab7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbef41537f0 0x7fbef414b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbefaab7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbef4162ba0 con 0x7fbef40c6370 2024-09-19T07:19:47.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbefaab7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbef4162a20 con 0x7fbef41537f0 2024-09-19T07:19:47.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbefaab7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbef4162d20 con 0x7fbef40c8d40 2024-09-19T07:19:47.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0aca542640 1 Processor -- start 2024-09-19T07:19:47.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbef8853640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbef40c6370 0x7fbef40c10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbef8853640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbef40c6370 0x7fbef40c10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40552/0 (socket says 172.21.15.83:40552) 2024-09-19T07:19:47.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbef8853640 1 -- 172.21.15.83:0/3620457545 learned_addr learned my addr 172.21.15.83:0/3620457545 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.987+0000 7f4f320ab640 1 -- 172.21.15.83:0/2843690095 wait complete. 2024-09-19T07:19:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa6e46640 1 Processor -- start 2024-09-19T07:19:47.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f4f320ab640 1 Processor -- start 2024-09-19T07:19:47.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa6e46640 1 -- start start 2024-09-19T07:19:47.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbef9054640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbef41537f0 0x7fbef414b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbef3fff640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbef40c8d40 0x7fbef40c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7ff6cf59e640 1 -- 172.21.15.83:0/2760186195 shutdown_connections 2024-09-19T07:19:47.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0aca542640 1 -- start start 2024-09-19T07:19:47.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7faf8cfd2640 1 -- 172.21.15.83:0/2557569107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf80046660 msgr2=0x7faf8000acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7faf8cfd2640 1 --2- 172.21.15.83:0/2557569107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf80046660 0x7faf8000acd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faf80007ee0 tx=0x7faf7001d560 comp rx=0 tx=0).stop 2024-09-19T07:19:47.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0aca542640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ac40c44b0 0x7f0ac40aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0aca542640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ac41535d0 0x7f0ac40af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0aca542640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ac41557a0 0x7f0ac4155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0aca542640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac4162f10 con 0x7f0ac41557a0 2024-09-19T07:19:47.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0aca542640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac4162d90 con 0x7f0ac41535d0 2024-09-19T07:19:47.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0aca542640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0ac4163090 con 0x7f0ac40c44b0 2024-09-19T07:19:47.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbef8853640 1 -- 172.21.15.83:0/3620457545 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbef40c8d40 msgr2=0x7fbef40c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbef8853640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbef40c8d40 0x7fbef40c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbef8853640 1 -- 172.21.15.83:0/3620457545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbef41537f0 msgr2=0x7fbef414b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbef8853640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbef41537f0 0x7fbef414b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fbef8853640 1 -- 172.21.15.83:0/3620457545 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef41552d0 con 0x7fbef40c6370 2024-09-19T07:19:47.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0abbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ac41535d0 0x7f0ac40af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0abbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ac41535d0 0x7f0ac40af090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:32946/0 (socket says 172.21.15.83:32946) 2024-09-19T07:19:47.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0abbfff640 1 -- 172.21.15.83:0/1378206115 learned_addr learned my addr 172.21.15.83:0/1378206115 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa6e46640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf98042320 0x7fdf980447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa6e46640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf9802f540 0x7fdf9817a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa6e46640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf98032000 0x7fdf980db650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa6e46640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf980e8c00 con 0x7fdf98042320 2024-09-19T07:19:47.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa6e46640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdf98006db0 con 0x7fdf98032000 2024-09-19T07:19:47.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa6e46640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdf980e8d80 con 0x7fdf9802f540 2024-09-19T07:19:47.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0ac8adf640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ac41557a0 0x7f0ac4155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0ac3fff640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ac40c44b0 0x7f0ac40aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f4f320ab640 1 -- start start 2024-09-19T07:19:47.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f4f320ab640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f2c071550 0x7f4f2c1f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f4f320ab640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f2c0c1d70 0x7f4f2c14d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f4f320ab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f2c150dd0 0x7f4f2c14d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f4f320ab640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2c071ab0 con 0x7f4f2c150dd0 2024-09-19T07:19:47.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f4f320ab640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2c071930 con 0x7f4f2c0c1d70 2024-09-19T07:19:47.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f4f320ab640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4f2c161a00 con 0x7f4f2c071550 2024-09-19T07:19:47.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f991b49a640 1 -- 172.21.15.83:0/4246206606 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f98f4001040 con 0x7f9914148570 2024-09-19T07:19:47.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0abbfff640 1 -- 172.21.15.83:0/1378206115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ac40c44b0 msgr2=0x7f0ac40aeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0abbfff640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ac40c44b0 0x7f0ac40aeb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0abbfff640 1 -- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ac41557a0 msgr2=0x7f0ac4155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0abbfff640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ac41557a0 0x7f0ac4155bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0abbfff640 1 -- 172.21.15.83:0/1378206115 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac40ad1b0 con 0x7f0ac41535d0 2024-09-19T07:19:47.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0ac8adf640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ac41557a0 0x7f0ac4155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:47.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa5643640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf9802f540 0x7fdf9817a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa5643640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf9802f540 0x7fdf9817a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:36830/0 (socket says 172.21.15.83:36830) 2024-09-19T07:19:47.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa5643640 1 -- 172.21.15.83:0/3830773847 learned_addr learned my addr 172.21.15.83:0/3830773847 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa6645640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf98032000 0x7fdf980db650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f4f2affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f2c0c1d70 0x7f4f2c14d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f4f2b7fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f2c071550 0x7f4f2c1f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7fdfa5e44640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf98042320 0x7fdf980447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f5276ffd640 1 -- 172.21.15.83:0/922999186 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f52800c0060 con 0x7f5288155b00 2024-09-19T07:19:47.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f5276ffd640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52680678c0 0x7f5268069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f5276ffd640 1 -- 172.21.15.83:0/922999186 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f528011fb80 con 0x7f5288155b00 2024-09-19T07:19:47.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f5276ffd640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] conn(0x7f52680720d0 0x7f52680744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f5276ffd640 1 -- 172.21.15.83:0/922999186 --> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f52800abec0 con 0x7f52680720d0 2024-09-19T07:19:47.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f0abbfff640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ac41535d0 0x7f0ac40af090 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0aac019040 tx=0x7f0aac0191d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f4f2affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f2c0c1d70 0x7f4f2c14d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:32968/0 (socket says 172.21.15.83:32968) 2024-09-19T07:19:47.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7f4f2affd640 1 -- 172.21.15.83:0/2944623864 learned_addr learned my addr 172.21.15.83:0/2944623864 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7faf8cfd2640 1 -- 172.21.15.83:0/2557569107 shutdown_connections 2024-09-19T07:19:47.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7faf8cfd2640 1 --2- 172.21.15.83:0/2557569107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf80046660 0x7faf8000acd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7faf8cfd2640 1 --2- 172.21.15.83:0/2557569107 >> v2:172.21.15.184:3300/0 conn(0x7faf800d35a0 0x7faf800d9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7faf8cfd2640 1 -- 172.21.15.83:0/2557569107 >> 172.21.15.83:0/2557569107 conn(0x7faf80047d50 msgr2=0x7faf80048160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fdfa6645640 1 -- 172.21.15.83:0/3830773847 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf9802f540 msgr2=0x7fdf9817a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fdfa6645640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf9802f540 0x7fdf9817a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fdfa6645640 1 -- 172.21.15.83:0/3830773847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf98042320 msgr2=0x7fdf980447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fdfa6645640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf98042320 0x7fdf980447b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fdfa6645640 1 -- 172.21.15.83:0/3830773847 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf98044cf0 con 0x7fdf98032000 2024-09-19T07:19:47.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f528659c640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52680678c0 0x7f5268069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f5285d9b640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] conn(0x7f52680720d0 0x7f52680744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f4f2bfff640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f2c150dd0 0x7f4f2c14d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f9909ffb640 1 -- 172.21.15.83:0/4246206606 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f990c01b020 con 0x7f9914148570 2024-09-19T07:19:47.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7ff6cf59e640 1 -- 172.21.15.83:0/2760186195 wait complete. 2024-09-19T07:19:47.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.991+0000 7ff6cf59e640 1 Processor -- start 2024-09-19T07:19:47.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fdfa6645640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf98032000 0x7fdf980db650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdfa006a530 tx=0x7fdfa0069900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f0ac17fa640 1 -- 172.21.15.83:0/1378206115 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac0b8020 con 0x7f0ac41535d0 2024-09-19T07:19:47.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f0aca542640 1 -- 172.21.15.83:0/1378206115 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac414d200 con 0x7f0ac41535d0 2024-09-19T07:19:47.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f0aca542640 1 -- 172.21.15.83:0/1378206115 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac4153bf0 con 0x7f0ac41535d0 2024-09-19T07:19:47.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f5285d9b640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] conn(0x7f52680720d0 0x7f52680744f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f9909ffb640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98f80678f0 0x7f98f8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f0ac17fa640 1 -- 172.21.15.83:0/1378206115 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0aac0b4410 con 0x7f0ac41535d0 2024-09-19T07:19:47.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f0ac17fa640 1 -- 172.21.15.83:0/1378206115 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac0b0040 con 0x7f0ac41535d0 2024-09-19T07:19:47.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f9909ffb640 1 -- 172.21.15.83:0/4246206606 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f990c0869b0 con 0x7f9914148570 2024-09-19T07:19:47.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fdf967fc640 1 -- 172.21.15.83:0/3830773847 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0067c80 con 0x7fdf98032000 2024-09-19T07:19:47.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f9919236640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98f80678f0 0x7f98f8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f9909ffb640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] conn(0x7f98f8072100 0x7f98f8074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f9909ffb640 1 -- 172.21.15.83:0/4246206606 --> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98f8074bd0 con 0x7f98f8072100 2024-09-19T07:19:47.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fdf967fc640 1 -- 172.21.15.83:0/3830773847 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa0067e20 con 0x7fdf98032000 2024-09-19T07:19:47.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f9909ffb640 1 -- 172.21.15.83:0/4246206606 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f990c052d50 con 0x7f9914148570 2024-09-19T07:19:47.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fbef8853640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbef40c6370 0x7fbef40c10a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbee800c450 tx=0x7fbee800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f9918a35640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] conn(0x7f98f8072100 0x7f98f8074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fdf967fc640 1 -- 172.21.15.83:0/3830773847 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0063030 con 0x7fdf98032000 2024-09-19T07:19:47.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fdfa6e46640 1 -- 172.21.15.83:0/3830773847 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf98179e70 con 0x7fdf98032000 2024-09-19T07:19:47.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f4f2bfff640 1 -- 172.21.15.83:0/2944623864 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f2c071550 msgr2=0x7f4f2c1f2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f4f2bfff640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f2c071550 0x7f4f2c1f2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f4f2bfff640 1 -- 172.21.15.83:0/2944623864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f2c0c1d70 msgr2=0x7f4f2c14d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f4f2bfff640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f2c0c1d70 0x7f4f2c14d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f4f2bfff640 1 -- 172.21.15.83:0/2944623864 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2c14a740 con 0x7f4f2c150dd0 2024-09-19T07:19:47.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fdfa6e46640 1 -- 172.21.15.83:0/3830773847 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9802fa90 con 0x7fdf98032000 2024-09-19T07:19:47.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fbef17fa640 1 -- 172.21.15.83:0/3620457545 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee80074e0 con 0x7fbef40c6370 2024-09-19T07:19:47.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fbef17fa640 1 -- 172.21.15.83:0/3620457545 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbee80047f0 con 0x7fbef40c6370 2024-09-19T07:19:47.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fbef17fa640 1 -- 172.21.15.83:0/3620457545 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee80076a0 con 0x7fbef40c6370 2024-09-19T07:19:47.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f5276ffd640 1 -- 172.21.15.83:0/922999186 <== osd.2 v2:172.21.15.83:6816/229839114 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f52800abec0 con 0x7f52680720d0 2024-09-19T07:19:47.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f528659c640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52680678c0 0x7f5268069d80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f527c006120 tx=0x7f527c005ee0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f4f2bfff640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f2c150dd0 0x7f4f2c14d770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4f2400ca80 tx=0x7f4f2400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f0aba7fc640 1 -- 172.21.15.83:0/1378206115 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0ac40adf60 con 0x7f0ac41535d0 2024-09-19T07:19:47.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fbefaab7640 1 -- 172.21.15.83:0/3620457545 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbef414eec0 con 0x7fbef40c6370 2024-09-19T07:19:47.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fbefaab7640 1 -- 172.21.15.83:0/3620457545 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbef414d860 con 0x7fbef40c6370 2024-09-19T07:19:47.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fdfa6e46640 1 -- 172.21.15.83:0/3830773847 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdf98046f90 con 0x7fdf98032000 2024-09-19T07:19:47.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f9919236640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98f80678f0 0x7f98f8069db0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f99100ad000 tx=0x7f99100a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f4f13fff640 1 -- 172.21.15.83:0/2944623864 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24017070 con 0x7f4f2c150dd0 2024-09-19T07:19:47.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f4f13fff640 1 -- 172.21.15.83:0/2944623864 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f2400ac30 con 0x7f4f2c150dd0 2024-09-19T07:19:47.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7ff6cf59e640 1 -- start start 2024-09-19T07:19:47.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7ff6cf59e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6d0071360 0x7ff6d00bff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7ff6cf59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6d00ad270 0x7ff6d00bd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7ff6cf59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6d00be150 0x7ff6d00be5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7ff6cf59e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d0151780 con 0x7ff6d00be150 2024-09-19T07:19:47.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7ff6cf59e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d0151600 con 0x7ff6d00ad270 2024-09-19T07:19:47.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7ff6cf59e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d015eb70 con 0x7ff6d0071360 2024-09-19T07:19:47.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7fbefaab7640 1 -- 172.21.15.83:0/3620457545 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbef41532b0 con 0x7fbef40c6370 2024-09-19T07:19:47.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f5276ffd640 1 -- 172.21.15.83:0/922999186 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f52800ebfd0 con 0x7f5288155b00 2024-09-19T07:19:47.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7ff6cdd9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6d00ad270 0x7ff6d00bd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7ff6cdd9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6d00ad270 0x7ff6d00bd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:32984/0 (socket says 172.21.15.83:32984) 2024-09-19T07:19:47.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7ff6cdd9b640 1 -- 172.21.15.83:0/4170477821 learned_addr learned my addr 172.21.15.83:0/4170477821 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7ff6ce59c640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6d0071360 0x7ff6d00bff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f9918a35640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] conn(0x7f98f8072100 0x7f98f8074520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f0ac17fa640 1 -- 172.21.15.83:0/1378206115 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0aac0bb020 con 0x7f0ac41535d0 2024-09-19T07:19:47.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f0ac17fa640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0aa40678c0 0x7f0aa4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f0ac17fa640 1 -- 172.21.15.83:0/1378206115 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f0aac11fdf0 con 0x7f0ac41535d0 2024-09-19T07:19:47.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f0ac17fa640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] conn(0x7f0aa40720d0 0x7f0aa40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f0ac17fa640 1 -- 172.21.15.83:0/1378206115 --> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0aac0b2ea0 con 0x7f0aa40720d0 2024-09-19T07:19:47.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f0ac17fa640 1 -- 172.21.15.83:0/1378206115 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0aac120280 con 0x7f0ac41535d0 2024-09-19T07:19:47.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f4f13fff640 1 -- 172.21.15.83:0/2944623864 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24012600 con 0x7f4f2c150dd0 2024-09-19T07:19:47.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f4f320ab640 1 -- 172.21.15.83:0/2944623864 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f2c14dcb0 con 0x7f4f2c150dd0 2024-09-19T07:19:47.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7f4f320ab640 1 -- 172.21.15.83:0/2944623864 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f2c0aba10 con 0x7f4f2c150dd0 2024-09-19T07:19:47.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.995+0000 7ff6ced9d640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6d00be150 0x7ff6d00be5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7ff6ce59c640 1 -- 172.21.15.83:0/4170477821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6d00ad270 msgr2=0x7ff6d00bd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7ff6ce59c640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6d00ad270 0x7ff6d00bd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7ff6ce59c640 1 -- 172.21.15.83:0/4170477821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6d00be150 msgr2=0x7ff6d00be5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7ff6ce59c640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6d00be150 0x7ff6d00be5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7ff6ce59c640 1 -- 172.21.15.83:0/4170477821 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d00bc390 con 0x7ff6d0071360 2024-09-19T07:19:47.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7ff6ce59c640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6d0071360 0x7ff6d00bff40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff6c000b3f0 tx=0x7ff6c000b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7fbef17fa640 1 -- 172.21.15.83:0/3620457545 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbee801a020 con 0x7fbef40c6370 2024-09-19T07:19:47.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7faf8cfd2640 1 -- 172.21.15.83:0/2557569107 shutdown_connections 2024-09-19T07:19:47.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f4f320ab640 1 -- 172.21.15.83:0/2944623864 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4f2c0c3840 con 0x7f4f2c150dd0 2024-09-19T07:19:47.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7fbef17fa640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbed8067940 0x7fbed8069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7fbef17fa640 1 -- 172.21.15.83:0/3620457545 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fbee8085520 con 0x7fbef40c6370 2024-09-19T07:19:47.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7fbef17fa640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] conn(0x7fbed8072150 0x7fbed8074570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7fbef17fa640 1 -- 172.21.15.83:0/3620457545 --> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbee8009e50 con 0x7fbed8072150 2024-09-19T07:19:47.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7ff6beffd640 1 -- 172.21.15.83:0/4170477821 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0015020 con 0x7ff6d0071360 2024-09-19T07:19:47.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7ff6beffd640 1 -- 172.21.15.83:0/4170477821 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c0011850 con 0x7ff6d0071360 2024-09-19T07:19:47.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7ff6beffd640 1 -- 172.21.15.83:0/4170477821 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0011c40 con 0x7ff6d0071360 2024-09-19T07:19:47.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f0ac8adf640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] conn(0x7f0aa40720d0 0x7f0aa40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f0ac3fff640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0aa40678c0 0x7f0aa4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f0ac8adf640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] conn(0x7f0aa40720d0 0x7f0aa40744f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f0ac3fff640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0aa40678c0 0x7f0aa4069d80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4005890 tx=0x7f0ab4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7ff6cf59e640 1 -- 172.21.15.83:0/4170477821 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d00bc670 con 0x7ff6d0071360 2024-09-19T07:19:47.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7ff6cf59e640 1 -- 172.21.15.83:0/4170477821 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d00bcc30 con 0x7ff6d0071360 2024-09-19T07:19:47.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7ff6cf59e640 1 -- 172.21.15.83:0/4170477821 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff6d0150e30 con 0x7ff6d0071360 2024-09-19T07:19:47.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7fbef17fa640 1 -- 172.21.15.83:0/3620457545 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbee808c050 con 0x7fbef40c6370 2024-09-19T07:19:47.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f4f13fff640 1 -- 172.21.15.83:0/2944623864 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4f2401a070 con 0x7f4f2c150dd0 2024-09-19T07:19:47.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f9909ffb640 1 -- 172.21.15.83:0/4246206606 <== osd.5 v2:172.21.15.184:6818/767241772 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f98f8074bd0 con 0x7f98f8072100 2024-09-19T07:19:47.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f0ac17fa640 1 -- 172.21.15.83:0/1378206115 <== osd.0 v2:172.21.15.83:6808/1414865007 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0aac0b2ea0 con 0x7f0aa40720d0 2024-09-19T07:19:47.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7faf8cfd2640 1 -- 172.21.15.83:0/2557569107 wait complete. 2024-09-19T07:19:47.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7faf8cfd2640 1 Processor -- start 2024-09-19T07:19:47.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7fbef9054640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] conn(0x7fbed8072150 0x7fbed8074570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f4f13fff640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f18067990 0x7f4f18069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f4f13fff640 1 -- 172.21.15.83:0/2944623864 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f4f24085b80 con 0x7f4f2c150dd0 2024-09-19T07:19:47.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f4f2b7fe640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f18067990 0x7f4f18069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f4f13fff640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] conn(0x7f4f18072230 0x7f4f18074650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7fbef9054640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] conn(0x7fbed8072150 0x7fbed8074570 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7f4f2affd640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] conn(0x7f4f18072230 0x7f4f18074650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7fbef17fa640 1 -- 172.21.15.83:0/3620457545 <== osd.1 v2:172.21.15.83:6800/2685389983 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbee8009e50 con 0x7fbed8072150 2024-09-19T07:19:47.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7fbef3fff640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbed8067940 0x7fbed8069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7fbef3fff640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbed8067940 0x7fbed8069e00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbee00ae000 tx=0x7fbee0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7f4f2b7fe640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f18067990 0x7f4f18069e50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f200ad000 tx=0x7f4f200a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7f4f2affd640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] conn(0x7f4f18072230 0x7f4f18074650 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7f4f13fff640 1 -- 172.21.15.83:0/2944623864 --> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f2400ac30 con 0x7f4f18072230 2024-09-19T07:19:47.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7f4f13fff640 1 -- 172.21.15.83:0/2944623864 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4f2408a2a0 con 0x7f4f2c150dd0 2024-09-19T07:19:47.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf8cfd2640 1 -- start start 2024-09-19T07:19:47.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf8cfd2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf8000b210 0x7faf80180410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf8cfd2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf80046660 0x7faf801838d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf8cfd2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf800d35a0 0x7faf801759a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf8cfd2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf80006560 con 0x7faf80046660 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf8cfd2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faf8004ac60 con 0x7faf800d35a0 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf8cfd2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faf800e2e10 con 0x7faf8000b210 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf85d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf80046660 0x7faf801838d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf85d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf80046660 0x7faf801838d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40618/0 (socket says 172.21.15.83:40618) 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf85d9b640 1 -- 172.21.15.83:0/3478038251 learned_addr learned my addr 172.21.15.83:0/3478038251 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf8659c640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf8000b210 0x7faf80180410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf86d9d640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf800d35a0 0x7faf801759a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf85d9b640 1 -- 172.21.15.83:0/3478038251 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf8000b210 msgr2=0x7faf80180410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf85d9b640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf8000b210 0x7faf80180410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf85d9b640 1 -- 172.21.15.83:0/3478038251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf800d35a0 msgr2=0x7faf801759a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf85d9b640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf800d35a0 0x7faf801759a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf85d9b640 1 -- 172.21.15.83:0/3478038251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf8017f950 con 0x7faf80046660 2024-09-19T07:19:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf85d9b640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf80046660 0x7faf801838d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faf700ae040 tx=0x7faf7001a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7ff6beffd640 1 -- 172.21.15.83:0/4170477821 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff6c0018020 con 0x7ff6d0071360 2024-09-19T07:19:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7ff6beffd640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6b0067990 0x7ff6b0069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7ff6beffd640 1 -- 172.21.15.83:0/4170477821 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7ff6c00863a0 con 0x7ff6d0071360 2024-09-19T07:19:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7ff6beffd640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] conn(0x7ff6b0072220 0x7ff6b0074640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7ff6beffd640 1 -- 172.21.15.83:0/4170477821 --> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6c001ee70 con 0x7ff6b0072220 2024-09-19T07:19:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7ff6beffd640 1 -- 172.21.15.83:0/4170477821 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff6c0086790 con 0x7ff6d0071360 2024-09-19T07:19:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7ff6cdd9b640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6b0067990 0x7ff6b0069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7ff6ced9d640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] conn(0x7ff6b0072220 0x7ff6b0074640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7fdf967fc640 1 -- 172.21.15.83:0/3830773847 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdfa00631f0 con 0x7fdf98032000 2024-09-19T07:19:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:46.999+0000 7fdf967fc640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf840678f0 0x7fdf84069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7fdf967fc640 1 -- 172.21.15.83:0/3830773847 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fdfa0101f50 con 0x7fdf98032000 2024-09-19T07:19:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7fdf967fc640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] conn(0x7fdf84072190 0x7fdf840745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7fdf967fc640 1 -- 172.21.15.83:0/3830773847 --> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdfa00a2e50 con 0x7fdf84072190 2024-09-19T07:19:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7fdf967fc640 1 -- 172.21.15.83:0/3830773847 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdfa0102340 con 0x7fdf98032000 2024-09-19T07:19:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7fdfa5643640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] conn(0x7fdf84072190 0x7fdf840745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7fdfa5e44640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf840678f0 0x7fdf84069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7fdfa5e44640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf840678f0 0x7fdf84069db0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0aa000 tx=0x7fdf9c001a10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7fdfa5643640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] conn(0x7fdf84072190 0x7fdf840745b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf7effd640 1 -- 172.21.15.83:0/3478038251 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf700ba020 con 0x7faf80046660 2024-09-19T07:19:47.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf7effd640 1 -- 172.21.15.83:0/3478038251 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf700b5420 con 0x7faf80046660 2024-09-19T07:19:47.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf7effd640 1 -- 172.21.15.83:0/3478038251 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf700b57f0 con 0x7faf80046660 2024-09-19T07:19:47.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7f4f13fff640 1 -- 172.21.15.83:0/2944623864 <== osd.3 v2:172.21.15.83:6824/2952931730 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f2400ac30 con 0x7f4f18072230 2024-09-19T07:19:47.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7faf8cfd2640 1 -- 172.21.15.83:0/3478038251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf800cf800 con 0x7faf80046660 2024-09-19T07:19:47.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7faf8cfd2640 1 -- 172.21.15.83:0/3478038251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf8017fc10 con 0x7faf80046660 2024-09-19T07:19:47.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7faf8cfd2640 1 -- 172.21.15.83:0/3478038251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faf8000acd0 con 0x7faf80046660 2024-09-19T07:19:47.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7faf7effd640 1 -- 172.21.15.83:0/3478038251 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf700bd020 con 0x7faf80046660 2024-09-19T07:19:47.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.003+0000 7ff6cdd9b640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6b0067990 0x7ff6b0069e50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff6d014d910 tx=0x7ff6c40090e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7ff6ced9d640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] conn(0x7ff6b0072220 0x7ff6b0074640 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7faf7effd640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf68067910 0x7faf68069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7faf7effd640 1 -- 172.21.15.83:0/3478038251 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7faf7011fe60 con 0x7faf80046660 2024-09-19T07:19:47.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7faf7effd640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] conn(0x7faf68072120 0x7faf68074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7faf7effd640 1 -- 172.21.15.83:0/3478038251 --> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf700ade90 con 0x7faf68072120 2024-09-19T07:19:47.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7faf7effd640 1 -- 172.21.15.83:0/3478038251 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faf700ec260 con 0x7faf80046660 2024-09-19T07:19:47.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7fdf967fc640 1 -- 172.21.15.83:0/3830773847 <== osd.7 v2:172.21.15.184:6816/2447598614 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdfa00a2e50 con 0x7fdf84072190 2024-09-19T07:19:47.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7faf8659c640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf68067910 0x7faf68069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7ff6beffd640 1 -- 172.21.15.83:0/4170477821 <== osd.6 v2:172.21.15.184:6800/1304502305 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff6c001ee70 con 0x7ff6b0072220 2024-09-19T07:19:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.007+0000 7faf86d9d640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] conn(0x7faf68072120 0x7faf68074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.011+0000 7faf8659c640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf68067910 0x7faf68069dd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faf780079c0 tx=0x7faf78008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.011+0000 7faf86d9d640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] conn(0x7faf68072120 0x7faf68074540 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.011+0000 7faf7effd640 1 -- 172.21.15.83:0/3478038251 <== osd.4 v2:172.21.15.184:6808/3839042106 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf700ade90 con 0x7faf68072120 2024-09-19T07:19:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f5274ff9640 1 -- 172.21.15.83:0/922999186 --> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5260002de0 con 0x7f52680720d0 2024-09-19T07:19:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f5276ffd640 1 -- 172.21.15.83:0/922999186 <== osd.2 v2:172.21.15.83:6816/229839114 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5260002de0 con 0x7f52680720d0 2024-09-19T07:19:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f0aba7fc640 1 -- 172.21.15.83:0/1378206115 --> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0ac40c3f70 con 0x7f0aa40720d0 2024-09-19T07:19:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f991b49a640 1 -- 172.21.15.83:0/4246206606 --> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f98f4002de0 con 0x7f98f8072100 2024-09-19T07:19:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f0ac17fa640 1 -- 172.21.15.83:0/1378206115 <== osd.0 v2:172.21.15.83:6808/1414865007 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0ac40c3f70 con 0x7f0aa40720d0 2024-09-19T07:19:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f528cbab640 1 -- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] conn(0x7f52680720d0 msgr2=0x7f52680744f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f528cbab640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] conn(0x7f52680720d0 0x7f52680744f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f528cbab640 1 -- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52680678c0 msgr2=0x7f5268069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f528cbab640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52680678c0 0x7f5268069d80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f527c006120 tx=0x7f527c005ee0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f528cbab640 1 -- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5288155b00 msgr2=0x7f52880ab740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f528cbab640 1 --2- 172.21.15.83:0/922999186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5288155b00 0x7f52880ab740 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f52880ad6e0 tx=0x7f528001b8b0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f5286d9d640 1 -- 172.21.15.83:0/922999186 reap_dead start 2024-09-19T07:19:47.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f528cbab640 1 -- 172.21.15.83:0/922999186 shutdown_connections 2024-09-19T07:19:47.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f528cbab640 1 -- 172.21.15.83:0/922999186 >> 172.21.15.83:0/922999186 conn(0x7f52880c7370 msgr2=0x7f52880c63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f9909ffb640 1 -- 172.21.15.83:0/4246206606 <== osd.5 v2:172.21.15.184:6818/767241772 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f98f4002de0 con 0x7f98f8072100 2024-09-19T07:19:47.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f98ff7fe640 1 -- 172.21.15.83:0/4246206606 >> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] conn(0x7f98f8072100 msgr2=0x7f98f8074520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f98ff7fe640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] conn(0x7f98f8072100 0x7f98f8074520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.027+0000 7f0aca542640 1 -- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] conn(0x7f0aa40720d0 msgr2=0x7f0aa40744f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f0aca542640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] conn(0x7f0aa40720d0 0x7f0aa40744f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f98ff7fe640 1 -- 172.21.15.83:0/4246206606 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98f80678f0 msgr2=0x7f98f8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f98ff7fe640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98f80678f0 0x7f98f8069db0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f99100ad000 tx=0x7f99100a6000 comp rx=0 tx=0).stop 2024-09-19T07:19:47.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f0aca542640 1 -- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0aa40678c0 msgr2=0x7f0aa4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f0aca542640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0aa40678c0 0x7f0aa4069d80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4005890 tx=0x7f0ab4009040 comp rx=0 tx=0).stop 2024-09-19T07:19:47.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f0aca542640 1 -- 172.21.15.83:0/1378206115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ac41535d0 msgr2=0x7f0ac40af090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f0aca542640 1 --2- 172.21.15.83:0/1378206115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ac41535d0 0x7f0ac40af090 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0aac019040 tx=0x7f0aac0191d0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f98ff7fe640 1 -- 172.21.15.83:0/4246206606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9914148570 msgr2=0x7f9914147b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f98ff7fe640 1 --2- 172.21.15.83:0/4246206606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9914148570 0x7f9914147b60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f990c00efc0 tx=0x7f990c00c490 comp rx=0 tx=0).stop 2024-09-19T07:19:47.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f9919a37640 1 -- 172.21.15.83:0/4246206606 reap_dead start 2024-09-19T07:19:47.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f0ac8adf640 1 -- 172.21.15.83:0/1378206115 reap_dead start 2024-09-19T07:19:47.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f528cbab640 1 -- 172.21.15.83:0/922999186 shutdown_connections 2024-09-19T07:19:47.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f98ff7fe640 1 -- 172.21.15.83:0/4246206606 shutdown_connections 2024-09-19T07:19:47.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f98ff7fe640 1 -- 172.21.15.83:0/4246206606 >> 172.21.15.83:0/4246206606 conn(0x7f99140c7370 msgr2=0x7f99140c3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f0aca542640 1 -- 172.21.15.83:0/1378206115 shutdown_connections 2024-09-19T07:19:47.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f0aca542640 1 -- 172.21.15.83:0/1378206115 >> 172.21.15.83:0/1378206115 conn(0x7f0ac40c6070 msgr2=0x7f0ac4157a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f98ff7fe640 1 -- 172.21.15.83:0/4246206606 shutdown_connections 2024-09-19T07:19:47.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f0aca542640 1 -- 172.21.15.83:0/1378206115 shutdown_connections 2024-09-19T07:19:47.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f98ff7fe640 1 -- 172.21.15.83:0/4246206606 wait complete. 2024-09-19T07:19:47.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f0aca542640 1 -- 172.21.15.83:0/1378206115 wait complete. 2024-09-19T07:19:47.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.031+0000 7f528cbab640 1 -- 172.21.15.83:0/922999186 wait complete. 2024-09-19T07:19:47.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.035+0000 7f4f320ab640 1 -- 172.21.15.83:0/2944623864 --> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4f2c0c1a50 con 0x7f4f18072230 2024-09-19T07:19:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.035+0000 7fdfa6e46640 1 -- 172.21.15.83:0/3830773847 --> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdf9804a310 con 0x7fdf84072190 2024-09-19T07:19:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.035+0000 7fdf967fc640 1 -- 172.21.15.83:0/3830773847 <== osd.7 v2:172.21.15.184:6816/2447598614 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdf9804a310 con 0x7fdf84072190 2024-09-19T07:19:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.035+0000 7fdfa6e46640 1 -- 172.21.15.83:0/3830773847 >> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] conn(0x7fdf84072190 msgr2=0x7fdf840745b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.035+0000 7fdfa6e46640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] conn(0x7fdf84072190 0x7fdf840745b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.035+0000 7fdfa6e46640 1 -- 172.21.15.83:0/3830773847 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf840678f0 msgr2=0x7fdf84069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.035+0000 7fdfa6e46640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf840678f0 0x7fdf84069db0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0aa000 tx=0x7fdf9c001a10 comp rx=0 tx=0).stop 2024-09-19T07:19:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.035+0000 7fdfa6e46640 1 -- 172.21.15.83:0/3830773847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf98032000 msgr2=0x7fdf980db650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.035+0000 7fdfa6e46640 1 --2- 172.21.15.83:0/3830773847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf98032000 0x7fdf980db650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdfa006a530 tx=0x7fdfa0069900 comp rx=0 tx=0).stop 2024-09-19T07:19:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.035+0000 7fdfa6645640 1 -- 172.21.15.83:0/3830773847 reap_dead start 2024-09-19T07:19:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7fdfa6e46640 1 -- 172.21.15.83:0/3830773847 shutdown_connections 2024-09-19T07:19:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7fdfa6e46640 1 -- 172.21.15.83:0/3830773847 >> 172.21.15.83:0/3830773847 conn(0x7fdf98048e40 msgr2=0x7fdf98004ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7f4f13fff640 1 -- 172.21.15.83:0/2944623864 <== osd.3 v2:172.21.15.83:6824/2952931730 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4f2c0c1a50 con 0x7f4f18072230 2024-09-19T07:19:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7fdfa6e46640 1 -- 172.21.15.83:0/3830773847 shutdown_connections 2024-09-19T07:19:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7fdfa6e46640 1 -- 172.21.15.83:0/3830773847 wait complete. 2024-09-19T07:19:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7f4f11ffb640 1 -- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] conn(0x7f4f18072230 msgr2=0x7f4f18074650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7f4f11ffb640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] conn(0x7f4f18072230 0x7f4f18074650 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7f4f11ffb640 1 -- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f18067990 msgr2=0x7f4f18069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7f4f11ffb640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f18067990 0x7f4f18069e50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f200ad000 tx=0x7f4f200a6000 comp rx=0 tx=0).stop 2024-09-19T07:19:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7f4f11ffb640 1 -- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f2c150dd0 msgr2=0x7f4f2c14d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7f4f11ffb640 1 --2- 172.21.15.83:0/2944623864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f2c150dd0 0x7f4f2c14d770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4f2400ca80 tx=0x7f4f2400cf40 comp rx=0 tx=0).stop 2024-09-19T07:19:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7f4f2bfff640 1 -- 172.21.15.83:0/2944623864 reap_dead start 2024-09-19T07:19:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7f4f11ffb640 1 -- 172.21.15.83:0/2944623864 shutdown_connections 2024-09-19T07:19:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7f4f11ffb640 1 -- 172.21.15.83:0/2944623864 >> 172.21.15.83:0/2944623864 conn(0x7f4f2c0c7370 msgr2=0x7f4f2c15e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7ff6cf59e640 1 -- 172.21.15.83:0/4170477821 --> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff6d006e140 con 0x7ff6b0072220 2024-09-19T07:19:47.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7f4f11ffb640 1 -- 172.21.15.83:0/2944623864 shutdown_connections 2024-09-19T07:19:47.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7ff6beffd640 1 -- 172.21.15.83:0/4170477821 <== osd.6 v2:172.21.15.184:6800/1304502305 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff6d006e140 con 0x7ff6b0072220 2024-09-19T07:19:47.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7f4f11ffb640 1 -- 172.21.15.83:0/2944623864 wait complete. 2024-09-19T07:19:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7fbefaab7640 1 -- 172.21.15.83:0/3620457545 --> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbef4153550 con 0x7fbed8072150 2024-09-19T07:19:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7ff6cf59e640 1 -- 172.21.15.83:0/4170477821 >> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] conn(0x7ff6b0072220 msgr2=0x7ff6b0074640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7ff6cf59e640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] conn(0x7ff6b0072220 0x7ff6b0074640 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7fbef17fa640 1 -- 172.21.15.83:0/3620457545 <== osd.1 v2:172.21.15.83:6800/2685389983 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbef4153550 con 0x7fbed8072150 2024-09-19T07:19:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7ff6cf59e640 1 -- 172.21.15.83:0/4170477821 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6b0067990 msgr2=0x7ff6b0069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7ff6cf59e640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6b0067990 0x7ff6b0069e50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff6d014d910 tx=0x7ff6c40090e0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7ff6cf59e640 1 -- 172.21.15.83:0/4170477821 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6d0071360 msgr2=0x7ff6d00bff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7ff6cf59e640 1 --2- 172.21.15.83:0/4170477821 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6d0071360 0x7ff6d00bff40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff6c000b3f0 tx=0x7ff6c000b8b0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7ff6ced9d640 1 -- 172.21.15.83:0/4170477821 reap_dead start 2024-09-19T07:19:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7ff6cf59e640 1 -- 172.21.15.83:0/4170477821 shutdown_connections 2024-09-19T07:19:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7ff6cf59e640 1 -- 172.21.15.83:0/4170477821 >> 172.21.15.83:0/4170477821 conn(0x7ff6d006dbc0 msgr2=0x7ff6d0153940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7faf8cfd2640 1 -- 172.21.15.83:0/3478038251 --> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faf8004a320 con 0x7faf68072120 2024-09-19T07:19:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.039+0000 7faf7effd640 1 -- 172.21.15.83:0/3478038251 <== osd.4 v2:172.21.15.184:6808/3839042106 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faf8004a320 con 0x7faf68072120 2024-09-19T07:19:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.043+0000 7ff6cf59e640 1 -- 172.21.15.83:0/4170477821 shutdown_connections 2024-09-19T07:19:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.043+0000 7ff6cf59e640 1 -- 172.21.15.83:0/4170477821 wait complete. 2024-09-19T07:19:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.047+0000 7faf8cfd2640 1 -- 172.21.15.83:0/3478038251 >> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] conn(0x7faf68072120 msgr2=0x7faf68074540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.047+0000 7faf8cfd2640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] conn(0x7faf68072120 0x7faf68074540 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.047+0000 7faf8cfd2640 1 -- 172.21.15.83:0/3478038251 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf68067910 msgr2=0x7faf68069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.047+0000 7faf8cfd2640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf68067910 0x7faf68069dd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faf780079c0 tx=0x7faf78008040 comp rx=0 tx=0).stop 2024-09-19T07:19:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.047+0000 7faf8cfd2640 1 -- 172.21.15.83:0/3478038251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf80046660 msgr2=0x7faf801838d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.047+0000 7faf8cfd2640 1 --2- 172.21.15.83:0/3478038251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf80046660 0x7faf801838d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faf700ae040 tx=0x7faf7001a300 comp rx=0 tx=0).stop 2024-09-19T07:19:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.047+0000 7faf86d9d640 1 -- 172.21.15.83:0/3478038251 reap_dead start 2024-09-19T07:19:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.047+0000 7faf8cfd2640 1 -- 172.21.15.83:0/3478038251 shutdown_connections 2024-09-19T07:19:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.047+0000 7faf8cfd2640 1 -- 172.21.15.83:0/3478038251 >> 172.21.15.83:0/3478038251 conn(0x7faf80047d50 msgr2=0x7faf800d1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.047+0000 7faf8cfd2640 1 -- 172.21.15.83:0/3478038251 shutdown_connections 2024-09-19T07:19:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.047+0000 7faf8cfd2640 1 -- 172.21.15.83:0/3478038251 wait complete. 2024-09-19T07:19:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.059+0000 7fbefaab7640 1 -- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] conn(0x7fbed8072150 msgr2=0x7fbed8074570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.059+0000 7fbefaab7640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] conn(0x7fbed8072150 0x7fbed8074570 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.059+0000 7fbefaab7640 1 -- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbed8067940 msgr2=0x7fbed8069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.059+0000 7fbefaab7640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbed8067940 0x7fbed8069e00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbee00ae000 tx=0x7fbee0008040 comp rx=0 tx=0).stop 2024-09-19T07:19:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.059+0000 7fbefaab7640 1 -- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbef40c6370 msgr2=0x7fbef40c10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.059+0000 7fbefaab7640 1 --2- 172.21.15.83:0/3620457545 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbef40c6370 0x7fbef40c10a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbee800c450 tx=0x7fbee800c910 comp rx=0 tx=0).stop 2024-09-19T07:19:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.059+0000 7fbef9054640 1 -- 172.21.15.83:0/3620457545 reap_dead start 2024-09-19T07:19:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.059+0000 7fbefaab7640 1 -- 172.21.15.83:0/3620457545 shutdown_connections 2024-09-19T07:19:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.059+0000 7fbefaab7640 1 -- 172.21.15.83:0/3620457545 >> 172.21.15.83:0/3620457545 conn(0x7fbef406bc80 msgr2=0x7fbef4155530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.059+0000 7fbefaab7640 1 -- 172.21.15.83:0/3620457545 shutdown_connections 2024-09-19T07:19:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.059+0000 7fbefaab7640 1 -- 172.21.15.83:0/3620457545 wait complete. 2024-09-19T07:19:47.105 INFO:teuthology.orchestra.run.smithi083.stdout:60129542152 2024-09-19T07:19:47.105 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-19T07:19:47.106 INFO:teuthology.orchestra.run.smithi083.stdout:60129542152 2024-09-19T07:19:47.106 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-19T07:19:47.107 INFO:teuthology.orchestra.run.smithi083.stdout:60129542154 2024-09-19T07:19:47.107 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-19T07:19:47.108 INFO:teuthology.orchestra.run.smithi083.stdout:60129542152 2024-09-19T07:19:47.108 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-19T07:19:47.109 INFO:teuthology.orchestra.run.smithi083.stdout:60129542153 2024-09-19T07:19:47.109 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-19T07:19:47.110 INFO:teuthology.orchestra.run.smithi083.stdout:60129542154 2024-09-19T07:19:47.111 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-19T07:19:47.111 INFO:teuthology.orchestra.run.smithi083.stdout:60129542154 2024-09-19T07:19:47.112 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-19T07:19:47.112 INFO:teuthology.orchestra.run.smithi083.stdout:60129542153 2024-09-19T07:19:47.113 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-19T07:19:47.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.327+0000 7fb93c71f640 1 Processor -- start 2024-09-19T07:19:47.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.327+0000 7fdebde1a640 1 Processor -- start 2024-09-19T07:19:47.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.327+0000 7fdebde1a640 1 -- start start 2024-09-19T07:19:47.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.327+0000 7fb93c71f640 1 -- start start 2024-09-19T07:19:47.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.327+0000 7fb93c71f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb92c0d3570 0x7fb92c0d9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.327+0000 7fb93c71f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb92c0d3950 con 0x7fb92c046630 2024-09-19T07:19:47.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.327+0000 7fb93c71f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb92c045cc0 con 0x7fb92c048840 2024-09-19T07:19:47.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.327+0000 7fb93c71f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fb92c031260 con 0x7fb92c0d3570 2024-09-19T07:19:47.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.327+0000 7fb93acbc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb92c046630 0x7fb92c0e05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51550/0 (socket says 172.21.15.83:51550) 2024-09-19T07:19:47.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.327+0000 7fb93acbc640 1 -- 172.21.15.83:0/2819572962 learned_addr learned my addr 172.21.15.83:0/2819572962 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:47.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.327+0000 7fdebde1a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdeb806f330 0x7fdeb80c6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdebde1a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb80c71f0 0x7fdeb814ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.327+0000 7fb93a4bb640 1 --2- 172.21.15.83:0/2819572962 >> v2:172.21.15.184:3300/0 conn(0x7fb92c0d3570 0x7fb92c0d9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb77fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdeb806f330 0x7fdeb80c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb77fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdeb806f330 0x7fdeb80c6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40620/0 (socket says 172.21.15.83:40620) 2024-09-19T07:19:47.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdeb814cfd0 0x7fdeb8152c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51560/0 (socket says 172.21.15.83:51560) 2024-09-19T07:19:47.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feee1f7f640 1 Processor -- start 2024-09-19T07:19:47.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fb9394b9640 1 -- 172.21.15.83:0/2819572962 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3303462739 0 0) 0x7fb92c0d3950 con 0x7fb92c046630 2024-09-19T07:19:47.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fb9394b9640 1 -- 172.21.15.83:0/2819572962 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb920003530 con 0x7fb92c046630 2024-09-19T07:19:47.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81ecba3640 1 Processor -- start 2024-09-19T07:19:47.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb77fe640 1 -- 172.21.15.83:0/2735278191 learned_addr learned my addr 172.21.15.83:0/2735278191 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdebde1a640 1 -- 172.21.15.83:0/2735278191 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeb80abb50 con 0x7fdeb814cfd0 2024-09-19T07:19:47.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdebde1a640 1 -- 172.21.15.83:0/2735278191 --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fdeb80ac480 con 0x7fdeb806f330 2024-09-19T07:19:47.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1315792411 0 0) 0x7fdeb80abb50 con 0x7fdeb814cfd0 2024-09-19T07:19:47.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdebde1a640 1 -- 172.21.15.83:0/2735278191 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb80abe30 con 0x7fdeb80c71f0 2024-09-19T07:19:47.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea0005740 con 0x7fdeb814cfd0 2024-09-19T07:19:47.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb6ffd640 1 --2- 172.21.15.83:0/2735278191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb80c71f0 0x7fdeb814ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1722044835 0 0) 0x7fdea0005740 con 0x7fdeb814cfd0 2024-09-19T07:19:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb80c71f0 msgr2=0x7fdeb814ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 --2- 172.21.15.83:0/2735278191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb80c71f0 0x7fdeb814ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 >> v2:172.21.15.83:3300/0 conn(0x7fdeb806f330 msgr2=0x7fdeb80c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 --2- 172.21.15.83:0/2735278191 >> v2:172.21.15.83:3300/0 conn(0x7fdeb806f330 0x7fdeb80c6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb80ac110 con 0x7fdeb814cfd0 2024-09-19T07:19:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fdeb0001a00 con 0x7fdeb814cfd0 2024-09-19T07:19:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 >> v1:172.21.15.83:6789/0 conn(0x7fdeb814cfd0 legacy=0x7fdeb8152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 --2- 172.21.15.83:0/2735278191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdea0005c40 0x7fdea0006080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 --2- 172.21.15.83:0/2735278191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdea00065c0 0x7fdea0006a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 --2- 172.21.15.83:0/2735278191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdea00071b0 0x7fdea000bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb80ac480 con 0x7fdea00071b0 2024-09-19T07:19:47.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb80abe30 con 0x7fdea0005c40 2024-09-19T07:19:47.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdea000c140 con 0x7fdea00065c0 2024-09-19T07:19:47.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8747fff640 1 Processor -- start 2024-09-19T07:19:47.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8747fff640 1 -- start start 2024-09-19T07:19:47.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feee1f7f640 1 -- start start 2024-09-19T07:19:47.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feee1f7f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed400b880 0x7feed4049480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81ecba3640 1 -- start start 2024-09-19T07:19:47.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feee1f7f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7feed404ac30 0x7feed40499c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feee1f7f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feed40ddfa0 con 0x7feed4006c80 2024-09-19T07:19:47.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feee1f7f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7feed40dee70 con 0x7feed404ac30 2024-09-19T07:19:47.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feee1f7f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feed40de260 con 0x7feed400b880 2024-09-19T07:19:47.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb7fff640 1 --2- 172.21.15.83:0/2735278191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdea00071b0 0x7fdea000bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb6ffd640 1 --2- 172.21.15.83:0/2735278191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdea00065c0 0x7fdea0006a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81ecba3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f81e806cef0 0x7f81e80aa990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81ecba3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e80aaed0 con 0x7f81e8155ae0 2024-09-19T07:19:47.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81ecba3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e80c3260 con 0x7f81e806e6e0 2024-09-19T07:19:47.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81ecba3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f81e80c3540 con 0x7f81e806cef0 2024-09-19T07:19:47.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feedaffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed400b880 0x7feed4049480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feedb7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7feed4006c80 0x7feed40d45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:42516/0 (socket says 172.21.15.83:42516) 2024-09-19T07:19:47.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feedb7fe640 1 -- 172.21.15.83:0/4288077714 learned_addr learned my addr 172.21.15.83:0/4288077714 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:47.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f81e8155ae0 0x7f81e814b660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51570/0 (socket says 172.21.15.83:51570) 2024-09-19T07:19:47.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e7fff640 1 -- 172.21.15.83:0/586690716 learned_addr learned my addr 172.21.15.83:0/586690716 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:47.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feedbfff640 1 --2- 172.21.15.83:0/4288077714 >> v2:172.21.15.184:3300/0 conn(0x7feed404ac30 0x7feed40499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3732228267 0 0) 0x7f81e80aaed0 con 0x7f81e8155ae0 2024-09-19T07:19:47.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81d0003530 con 0x7f81e8155ae0 2024-09-19T07:19:47.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feeda7fc640 1 -- 172.21.15.83:0/4288077714 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4029355759 0 0) 0x7feed40ddfa0 con 0x7feed4006c80 2024-09-19T07:19:47.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb7fff640 1 -- 172.21.15.83:0/2735278191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdea00065c0 msgr2=0x7fdea0006a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb7fff640 1 --2- 172.21.15.83:0/2735278191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdea00065c0 0x7fdea0006a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb7fff640 1 -- 172.21.15.83:0/2735278191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdea0005c40 msgr2=0x7fdea0006080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb7fff640 1 --2- 172.21.15.83:0/2735278191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdea0005c40 0x7fdea0006080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fdeb7fff640 1 -- 172.21.15.83:0/2735278191 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb80ac110 con 0x7fdea00071b0 2024-09-19T07:19:47.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2893482963 0 0) 0x7f81d0003530 con 0x7f81e8155ae0 2024-09-19T07:19:47.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 >> v2:172.21.15.184:3300/0 conn(0x7f81e806cef0 msgr2=0x7f81e80aa990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:19:47.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 --2- 172.21.15.83:0/586690716 >> v2:172.21.15.184:3300/0 conn(0x7f81e806cef0 0x7f81e80aa990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 >> v1:172.21.15.184:6789/0 conn(0x7f81e806e6e0 legacy=0x7f81e81525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e81552e0 con 0x7f81e8155ae0 2024-09-19T07:19:47.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f81d8001680 con 0x7f81e8155ae0 2024-09-19T07:19:47.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fb93a4bb640 1 -- 172.21.15.83:0/2819572962 >> v1:172.21.15.184:6789/0 conn(0x7fb92c048840 legacy=0x7fb92c005a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fb93a4bb640 1 -- 172.21.15.83:0/2819572962 >> v1:172.21.15.83:6789/0 conn(0x7fb92c046630 legacy=0x7fb92c0e05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fb93a4bb640 1 -- 172.21.15.83:0/2819572962 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c045fa0 con 0x7fb92c0d3570 2024-09-19T07:19:47.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8747fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87400d42a0 0x7f87400d68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8747fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8740047620 0x7f8740009a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8747fff640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8740045cc0 con 0x7f8740046d20 2024-09-19T07:19:47.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8747fff640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f8740046890 con 0x7f87400d42a0 2024-09-19T07:19:47.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8747fff640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f874002ea40 con 0x7f8740047620 2024-09-19T07:19:47.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8746ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87400d42a0 0x7f87400d68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8746ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87400d42a0 0x7f87400d68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40646/0 (socket says 172.21.15.83:40646) 2024-09-19T07:19:47.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8746ffd640 1 -- 172.21.15.83:0/1039864000 learned_addr learned my addr 172.21.15.83:0/1039864000 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f87467fc640 1 --2- 172.21.15.83:0/1039864000 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8740047620 0x7f8740009a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8746ffd640 1 -- 172.21.15.83:0/1039864000 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8740047620 msgr2=0x7f8740009a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8746ffd640 1 --2- 172.21.15.83:0/1039864000 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8740047620 0x7f8740009a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8746ffd640 1 -- 172.21.15.83:0/1039864000 >> v1:172.21.15.184:6789/0 conn(0x7f8740046d20 legacy=0x7f8740006a20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:19:47.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f8746ffd640 1 -- 172.21.15.83:0/1039864000 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874004acf0 con 0x7f87400d42a0 2024-09-19T07:19:47.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feeda7fc640 1 -- 172.21.15.83:0/4288077714 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feecc003530 con 0x7feed4006c80 2024-09-19T07:19:47.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feedaffd640 1 -- 172.21.15.83:0/4288077714 >> v2:172.21.15.184:3300/0 conn(0x7feed404ac30 msgr2=0x7feed40499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feedaffd640 1 --2- 172.21.15.83:0/4288077714 >> v2:172.21.15.184:3300/0 conn(0x7feed404ac30 0x7feed40499c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feedaffd640 1 -- 172.21.15.83:0/4288077714 >> v1:172.21.15.184:6789/0 conn(0x7feed4006c80 legacy=0x7feed40d45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7feedaffd640 1 -- 172.21.15.83:0/4288077714 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed40de540 con 0x7feed400b880 2024-09-19T07:19:47.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 >> v1:172.21.15.83:6789/0 conn(0x7f81e8155ae0 legacy=0x7f81e814b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 --2- 172.21.15.83:0/586690716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81d00039c0 0x7f81d0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 --2- 172.21.15.83:0/586690716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81d0004810 0x7f81d0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 --2- 172.21.15.83:0/586690716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81d00053d0 0x7f81d000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f81e80c3540 con 0x7f81d00039c0 2024-09-19T07:19:47.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f81d000c290 con 0x7f81d0004810 2024-09-19T07:19:47.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f81d000c490 con 0x7f81d00053d0 2024-09-19T07:19:47.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e6ffd640 1 --2- 172.21.15.83:0/586690716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81d0004810 0x7f81d0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f81e7fff640 1 --2- 172.21.15.83:0/586690716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81d00053d0 0x7f81d000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7fb93a4bb640 1 --2- 172.21.15.83:0/2819572962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c0d3570 0x7fb92c0d9c50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb9240097c0 tx=0x7fb92401c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2cc6f7e6c290cb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139882e640 1 Processor -- start 2024-09-19T07:19:47.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f81e77fe640 1 --2- 172.21.15.83:0/586690716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81d00039c0 0x7f81d0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f81e6ffd640 1 -- 172.21.15.83:0/586690716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81d00053d0 msgr2=0x7f81d000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f81e6ffd640 1 --2- 172.21.15.83:0/586690716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81d00053d0 0x7f81d000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f81e6ffd640 1 -- 172.21.15.83:0/586690716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81d00039c0 msgr2=0x7f81d0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fb9394b9640 1 -- 172.21.15.83:0/2819572962 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9240b0070 con 0x7fb92c0d3570 2024-09-19T07:19:47.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139882e640 1 -- start start 2024-09-19T07:19:47.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139882e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f138c00b2c0 0x7f138c00b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139882e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f138c0de2d0 0x7f138c0de710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139882e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f138c0dec50 0x7f138c046f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139882e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f138c02b1e0 con 0x7f138c00b2c0 2024-09-19T07:19:47.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139882e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f138c03ed00 con 0x7f138c0dec50 2024-09-19T07:19:47.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139882e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f138c04a640 con 0x7f138c0de2d0 2024-09-19T07:19:47.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f81e6ffd640 1 --2- 172.21.15.83:0/586690716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81d00039c0 0x7f81d0003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f81e6ffd640 1 -- 172.21.15.83:0/586690716 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e81552e0 con 0x7f81d0004810 2024-09-19T07:19:47.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f81e7fff640 1 -- 172.21.15.83:0/586690716 reap_dead start 2024-09-19T07:19:47.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fb9394b9640 1 -- 172.21.15.83:0/2819572962 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9240ab600 con 0x7fb92c0d3570 2024-09-19T07:19:47.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fb9394b9640 1 -- 172.21.15.83:0/2819572962 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9240ab9f0 con 0x7fb92c0d3570 2024-09-19T07:19:47.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fdeb7fff640 1 --2- 172.21.15.83:0/2735278191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdea00071b0 0x7fdea000bbd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000c520 tx=0x7fdeb001f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45f27b9f4d5b674e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fb93c71f640 1 -- 172.21.15.83:0/2819572962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c0d3570 msgr2=0x7fb92c0d9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fb93c71f640 1 --2- 172.21.15.83:0/2819572962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c0d3570 0x7fb92c0d9c50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb9240097c0 tx=0x7fb92401c430 comp rx=0 tx=0).stop 2024-09-19T07:19:47.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139902f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f138c0dec50 0x7f138c046f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139902f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f138c0dec50 0x7f138c046f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:33048/0 (socket says 172.21.15.83:33048) 2024-09-19T07:19:47.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139902f640 1 -- 172.21.15.83:0/1570461512 learned_addr learned my addr 172.21.15.83:0/1570461512 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f1392d9d640 1 --2- 172.21.15.83:0/1570461512 >> v2:172.21.15.83:3300/0 conn(0x7f138c00b2c0 0x7f138c00b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139259c640 1 --2- 172.21.15.83:0/1570461512 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f138c0de2d0 0x7f138c0de710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb000c120 con 0x7fdea00071b0 2024-09-19T07:19:47.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.331+0000 7f9506c0e640 1 Processor -- start 2024-09-19T07:19:47.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9506c0e640 1 -- start start 2024-09-19T07:19:47.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9506c0e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94f80db810 0x7f94f800aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9506c0e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f94f80dec50 0x7f94f80e8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9506c0e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f803e270 con 0x7f94f800b1e0 2024-09-19T07:19:47.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9506c0e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f94f8030780 con 0x7f94f80db810 2024-09-19T07:19:47.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9506c0e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f94f8030a60 con 0x7f94f80dec50 2024-09-19T07:19:47.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f7f53fff640 1 Processor -- start 2024-09-19T07:19:47.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f7f53fff640 1 -- start start 2024-09-19T07:19:47.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7feedaffd640 1 --2- 172.21.15.83:0/4288077714 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed400b880 0x7feed4049480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feed00097c0 tx=0x7feed001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ab42996695c660c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139902f640 1 -- 172.21.15.83:0/1570461512 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f138c0de2d0 msgr2=0x7f138c0de710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139902f640 1 --2- 172.21.15.83:0/1570461512 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f138c0de2d0 0x7f138c0de710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139902f640 1 -- 172.21.15.83:0/1570461512 >> v2:172.21.15.83:3300/0 conn(0x7f138c00b2c0 msgr2=0x7f138c00b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139902f640 1 --2- 172.21.15.83:0/1570461512 >> v2:172.21.15.83:3300/0 conn(0x7f138c00b2c0 0x7f138c00b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9505c0c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94f80db810 0x7f94f800aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9505c0c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94f80db810 0x7f94f800aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40680/0 (socket says 172.21.15.83:40680) 2024-09-19T07:19:47.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9505c0c640 1 -- 172.21.15.83:0/2320791756 learned_addr learned my addr 172.21.15.83:0/2320791756 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9505c0c640 1 -- 172.21.15.83:0/2320791756 >> v2:172.21.15.184:3300/0 conn(0x7f94f80dec50 msgr2=0x7f94f80e8970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:19:47.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7feeda7fc640 1 -- 172.21.15.83:0/4288077714 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed00b2070 con 0x7feed400b880 2024-09-19T07:19:47.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7feeda7fc640 1 -- 172.21.15.83:0/4288077714 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed00ad640 con 0x7feed400b880 2024-09-19T07:19:47.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7feeda7fc640 1 -- 172.21.15.83:0/4288077714 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed00ad9f0 con 0x7feed400b880 2024-09-19T07:19:47.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f8746ffd640 1 --2- 172.21.15.83:0/1039864000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87400d42a0 0x7f87400d68e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8748050290 tx=0x7f8748098bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826b9d80c4b699f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9505c0c640 1 --2- 172.21.15.83:0/2320791756 >> v2:172.21.15.184:3300/0 conn(0x7f94f80dec50 0x7f94f80e8970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9505c0c640 1 -- 172.21.15.83:0/2320791756 >> v1:172.21.15.184:6789/0 conn(0x7f94f800b1e0 legacy=0x7f94f80de540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:19:47.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f9505c0c640 1 -- 172.21.15.83:0/2320791756 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f8030d40 con 0x7f94f80db810 2024-09-19T07:19:47.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fb93c71f640 1 -- 172.21.15.83:0/2819572962 shutdown_connections 2024-09-19T07:19:47.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fb93c71f640 1 --2- 172.21.15.83:0/2819572962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c0d3570 0x7fb92c0d9c50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fb93c71f640 1 -- 172.21.15.83:0/2819572962 >> 172.21.15.83:0/2819572962 conn(0x7fb92c047d20 msgr2=0x7fb92c048130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb0005da0 con 0x7fdea00071b0 2024-09-19T07:19:47.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fdeb67fc640 1 -- 172.21.15.83:0/2735278191 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0007290 con 0x7fdea00071b0 2024-09-19T07:19:47.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fdebde1a640 1 -- 172.21.15.83:0/2735278191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdea00071b0 msgr2=0x7fdea000bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fdebde1a640 1 --2- 172.21.15.83:0/2735278191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdea00071b0 0x7fdea000bbd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000c520 tx=0x7fdeb001f6a0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f8745ffb640 1 -- 172.21.15.83:0/1039864000 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8748121a50 con 0x7f87400d42a0 2024-09-19T07:19:47.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f8745ffb640 1 -- 172.21.15.83:0/1039864000 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8748121c30 con 0x7f87400d42a0 2024-09-19T07:19:47.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f8745ffb640 1 -- 172.21.15.83:0/1039864000 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8748129640 con 0x7f87400d42a0 2024-09-19T07:19:47.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fdeb7fff640 1 -- 172.21.15.83:0/2735278191 reap_dead start 2024-09-19T07:19:47.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fdebde1a640 1 -- 172.21.15.83:0/2735278191 shutdown_connections 2024-09-19T07:19:47.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fdebde1a640 1 -- 172.21.15.83:0/2735278191 >> 172.21.15.83:0/2735278191 conn(0x7fdeb80c53f0 msgr2=0x7fdeb80c5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fdebde1a640 1 -- 172.21.15.83:0/2735278191 shutdown_connections 2024-09-19T07:19:47.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7feee1f7f640 1 -- 172.21.15.83:0/4288077714 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed400b880 msgr2=0x7feed4049480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7feee1f7f640 1 --2- 172.21.15.83:0/4288077714 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed400b880 0x7feed4049480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feed00097c0 tx=0x7feed001c470 comp rx=0 tx=0).stop 2024-09-19T07:19:47.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7feee1f7f640 1 -- 172.21.15.83:0/4288077714 shutdown_connections 2024-09-19T07:19:47.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7feee1f7f640 1 --2- 172.21.15.83:0/4288077714 >> v2:172.21.15.184:3300/0 conn(0x7feed404ac30 0x7feed40499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7feee1f7f640 1 --2- 172.21.15.83:0/4288077714 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed400b880 0x7feed4049480 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7f139902f640 1 -- 172.21.15.83:0/1570461512 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138c04a920 con 0x7f138c0dec50 2024-09-19T07:19:47.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7feee1f7f640 1 -- 172.21.15.83:0/4288077714 >> 172.21.15.83:0/4288077714 conn(0x7feed4047d20 msgr2=0x7feed4048130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feee1f7f640 1 -- 172.21.15.83:0/4288077714 shutdown_connections 2024-09-19T07:19:47.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f8747fff640 1 -- 172.21.15.83:0/1039864000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87400d42a0 msgr2=0x7f87400d68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f8747fff640 1 --2- 172.21.15.83:0/1039864000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87400d42a0 0x7f87400d68e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8748050290 tx=0x7f8748098bb0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f8747fff640 1 -- 172.21.15.83:0/1039864000 shutdown_connections 2024-09-19T07:19:47.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f8747fff640 1 --2- 172.21.15.83:0/1039864000 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8740047620 0x7f8740009a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f8747fff640 1 --2- 172.21.15.83:0/1039864000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87400d42a0 0x7f87400d68e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f8747fff640 1 -- 172.21.15.83:0/1039864000 >> 172.21.15.83:0/1039864000 conn(0x7f8740047d20 msgr2=0x7f8740048130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f81e6ffd640 1 --2- 172.21.15.83:0/586690716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81d0004810 0x7f81d0004c90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f81e0001e70 tx=0x7f81e001c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea17734eaacc4272 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f7f53fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f5414feb0 0x7f7f5406bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f7f52ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f5414feb0 0x7f7f5406bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f7f52ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f5414feb0 0x7f7f5406bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40694/0 (socket says 172.21.15.83:40694) 2024-09-19T07:19:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f7f52ffd640 1 -- 172.21.15.83:0/2324023727 learned_addr learned my addr 172.21.15.83:0/2324023727 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e00b0070 con 0x7f81d0004810 2024-09-19T07:19:47.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81e00a7070 con 0x7f81d0004810 2024-09-19T07:19:47.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f81e67fc640 1 -- 172.21.15.83:0/586690716 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e0001a90 con 0x7f81d0004810 2024-09-19T07:19:47.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f7f53fff640 1 -- 172.21.15.83:0/2324023727 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f540ad250 con 0x7f7f5406e820 2024-09-19T07:19:47.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.335+0000 7fdebde1a640 1 -- 172.21.15.83:0/2735278191 wait complete. 2024-09-19T07:19:47.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdebde1a640 1 Processor -- start 2024-09-19T07:19:47.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f7f53fff640 1 -- 172.21.15.83:0/2324023727 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f5406c8d0 con 0x7f7f540ab3c0 2024-09-19T07:19:47.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f7f53fff640 1 -- 172.21.15.83:0/2324023727 --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7f54153750 con 0x7f7f5414feb0 2024-09-19T07:19:47.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdebde1a640 1 -- start start 2024-09-19T07:19:47.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f8747fff640 1 -- 172.21.15.83:0/1039864000 shutdown_connections 2024-09-19T07:19:47.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feee1f7f640 1 -- 172.21.15.83:0/4288077714 wait complete. 2024-09-19T07:19:47.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feee1f7f640 1 Processor -- start 2024-09-19T07:19:47.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f7f52ffd640 1 -- 172.21.15.83:0/2324023727 >> v1:172.21.15.184:6789/0 conn(0x7f7f540ab3c0 legacy=0x7f7f5406c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f7f52ffd640 1 -- 172.21.15.83:0/2324023727 >> v1:172.21.15.83:6789/0 conn(0x7f7f5406e820 legacy=0x7f7f5406ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:47.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f7f52ffd640 1 -- 172.21.15.83:0/2324023727 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f54153190 con 0x7f7f5414feb0 2024-09-19T07:19:47.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fb93c71f640 1 -- 172.21.15.83:0/2819572962 shutdown_connections 2024-09-19T07:19:47.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feee1f7f640 1 -- start start 2024-09-19T07:19:47.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f81ecba3640 1 -- 172.21.15.83:0/586690716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81d0004810 msgr2=0x7f81d0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f81ecba3640 1 --2- 172.21.15.83:0/586690716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81d0004810 0x7f81d0004c90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f81e0001e70 tx=0x7f81e001c760 comp rx=0 tx=0).stop 2024-09-19T07:19:47.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f9505c0c640 1 --2- 172.21.15.83:0/2320791756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f80db810 0x7f94f800aca0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f94f800d470 tx=0x7f94f001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e69685d3b07a0a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f139902f640 1 --2- 172.21.15.83:0/1570461512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f138c0dec50 0x7f138c046f70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f138c0097a0 tx=0x7f138401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca096086331dd876 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f9504c0a640 1 -- 172.21.15.83:0/2320791756 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f00b0070 con 0x7f94f80db810 2024-09-19T07:19:47.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdebde1a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb80c3e10 0x7fdeb814d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdebde1a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb81e71c0 0x7fdeb81e7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdebde1a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb81ea730 0x7fdeb81eabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdebde1a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb806f890 con 0x7fdeb80c3e10 2024-09-19T07:19:47.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdebde1a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb806f710 con 0x7fdeb81ea730 2024-09-19T07:19:47.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdebde1a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb814ca90 con 0x7fdeb81e71c0 2024-09-19T07:19:47.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f9504c0a640 1 -- 172.21.15.83:0/2320791756 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94f00ab600 con 0x7f94f80db810 2024-09-19T07:19:47.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f9504c0a640 1 -- 172.21.15.83:0/2320791756 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f00aba10 con 0x7f94f80db810 2024-09-19T07:19:47.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f8747fff640 1 -- 172.21.15.83:0/1039864000 wait complete. 2024-09-19T07:19:47.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f1391d9b640 1 -- 172.21.15.83:0/1570461512 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13840b0070 con 0x7f138c0dec50 2024-09-19T07:19:47.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f1391d9b640 1 -- 172.21.15.83:0/1570461512 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13840ab600 con 0x7f138c0dec50 2024-09-19T07:19:47.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f1391d9b640 1 -- 172.21.15.83:0/1570461512 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13840ab9d0 con 0x7f138c0dec50 2024-09-19T07:19:47.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdeb77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb80c3e10 0x7fdeb814d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdeb77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb80c3e10 0x7fdeb814d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40706/0 (socket says 172.21.15.83:40706) 2024-09-19T07:19:47.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdeb77fe640 1 -- 172.21.15.83:0/2244449791 learned_addr learned my addr 172.21.15.83:0/2244449791 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feee1f7f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed4006c80 0x7feed4041800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feee1f7f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed400b880 0x7feed4044100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feee1f7f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed404ac30 0x7feed40d15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feee1f7f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feed40d4e40 con 0x7feed404ac30 2024-09-19T07:19:47.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feee1f7f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feed40d4cc0 con 0x7feed400b880 2024-09-19T07:19:47.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feee1f7f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feed4007060 con 0x7feed4006c80 2024-09-19T07:19:47.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdeb6ffd640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb81e71c0 0x7fdeb81e7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fb93c71f640 1 -- 172.21.15.83:0/2819572962 wait complete. 2024-09-19T07:19:47.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fb93c71f640 1 Processor -- start 2024-09-19T07:19:47.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdeb77fe640 1 -- 172.21.15.83:0/2244449791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb81e71c0 msgr2=0x7fdeb81e7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdeb77fe640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb81e71c0 0x7fdeb81e7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdeb77fe640 1 -- 172.21.15.83:0/2244449791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb81ea730 msgr2=0x7fdeb81eabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdeb7fff640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb81ea730 0x7fdeb81eabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdeb77fe640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb81ea730 0x7fdeb81eabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fdeb77fe640 1 -- 172.21.15.83:0/2244449791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb81571f0 con 0x7fdeb80c3e10 2024-09-19T07:19:47.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f8747fff640 1 Processor -- start 2024-09-19T07:19:47.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f9506c0e640 1 -- 172.21.15.83:0/2320791756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f80db810 msgr2=0x7f94f800aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f9506c0e640 1 --2- 172.21.15.83:0/2320791756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f80db810 0x7f94f800aca0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f94f800d470 tx=0x7f94f001c3e0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f9506c0e640 1 -- 172.21.15.83:0/2320791756 shutdown_connections 2024-09-19T07:19:47.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f9506c0e640 1 --2- 172.21.15.83:0/2320791756 >> v2:172.21.15.184:3300/0 conn(0x7f94f80dec50 0x7f94f80e8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f9506c0e640 1 --2- 172.21.15.83:0/2320791756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f80db810 0x7f94f800aca0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f9506c0e640 1 -- 172.21.15.83:0/2320791756 >> 172.21.15.83:0/2320791756 conn(0x7f94f8047d20 msgr2=0x7f94f8048130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f9506c0e640 1 -- 172.21.15.83:0/2320791756 shutdown_connections 2024-09-19T07:19:47.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feedbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed404ac30 0x7feed40d15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feedbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed404ac30 0x7feed40d15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40712/0 (socket says 172.21.15.83:40712) 2024-09-19T07:19:47.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feedbfff640 1 -- 172.21.15.83:0/2899600521 learned_addr learned my addr 172.21.15.83:0/2899600521 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feedb7fe640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed4006c80 0x7feed4041800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7feedaffd640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed400b880 0x7feed4044100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f81ecba3640 1 -- 172.21.15.83:0/586690716 shutdown_connections 2024-09-19T07:19:47.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f81ecba3640 1 --2- 172.21.15.83:0/586690716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81d0004810 0x7f81d0004c90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f81ecba3640 1 -- 172.21.15.83:0/586690716 >> 172.21.15.83:0/586690716 conn(0x7f81e806dbc0 msgr2=0x7f81e806dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f81ecba3640 1 -- 172.21.15.83:0/586690716 shutdown_connections 2024-09-19T07:19:47.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f139882e640 1 -- 172.21.15.83:0/1570461512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f138c0dec50 msgr2=0x7f138c046f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f139882e640 1 --2- 172.21.15.83:0/1570461512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f138c0dec50 0x7f138c046f70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f138c0097a0 tx=0x7f138401c540 comp rx=0 tx=0).stop 2024-09-19T07:19:47.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f139882e640 1 -- 172.21.15.83:0/1570461512 shutdown_connections 2024-09-19T07:19:47.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f139882e640 1 --2- 172.21.15.83:0/1570461512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f138c0dec50 0x7f138c046f70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f139882e640 1 --2- 172.21.15.83:0/1570461512 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f138c0de2d0 0x7f138c0de710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f139882e640 1 --2- 172.21.15.83:0/1570461512 >> v2:172.21.15.83:3300/0 conn(0x7f138c00b2c0 0x7f138c00b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f139882e640 1 -- 172.21.15.83:0/1570461512 >> 172.21.15.83:0/1570461512 conn(0x7f138c048340 msgr2=0x7f138c048750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f139882e640 1 -- 172.21.15.83:0/1570461512 shutdown_connections 2024-09-19T07:19:47.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fdeb77fe640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb80c3e10 0x7fdeb814d920 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00e870 tx=0x7fdeac00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7feedbfff640 1 -- 172.21.15.83:0/2899600521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed4006c80 msgr2=0x7feed4041800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7feedbfff640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed4006c80 0x7feed4041800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7feedbfff640 1 -- 172.21.15.83:0/2899600521 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed400b880 msgr2=0x7feed4044100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7feedbfff640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed400b880 0x7feed4044100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7feedbfff640 1 -- 172.21.15.83:0/2899600521 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed40d37b0 con 0x7feed404ac30 2024-09-19T07:19:47.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7fb93c71f640 1 -- start start 2024-09-19T07:19:47.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93c71f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb92c046630 0x7fb92c041bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93c71f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c048840 0x7fb92c0e66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93c71f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c0d3570 0x7fb92c0d0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93c71f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c0e3cf0 con 0x7fb92c048840 2024-09-19T07:19:47.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93c71f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c006530 con 0x7fb92c0d3570 2024-09-19T07:19:47.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93c71f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb92c0e3e70 con 0x7fb92c046630 2024-09-19T07:19:47.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93a4bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb92c046630 0x7fb92c041bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93a4bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb92c046630 0x7fb92c041bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:36940/0 (socket says 172.21.15.83:36940) 2024-09-19T07:19:47.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93a4bb640 1 -- 172.21.15.83:0/247340572 learned_addr learned my addr 172.21.15.83:0/247340572 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb939cba640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c048840 0x7fb92c0e66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93acbc640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c0d3570 0x7fb92c0d0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fde9ffff640 1 -- 172.21.15.83:0/2244449791 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac017070 con 0x7fdeb80c3e10 2024-09-19T07:19:47.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fde9ffff640 1 -- 172.21.15.83:0/2244449791 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeac00c8d0 con 0x7fdeb80c3e10 2024-09-19T07:19:47.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fde9ffff640 1 -- 172.21.15.83:0/2244449791 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac00f040 con 0x7fdeb80c3e10 2024-09-19T07:19:47.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fdebde1a640 1 -- 172.21.15.83:0/2244449791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb814d640 con 0x7fdeb80c3e10 2024-09-19T07:19:47.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fdebde1a640 1 -- 172.21.15.83:0/2244449791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb81506a0 con 0x7fdeb80c3e10 2024-09-19T07:19:47.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f8747fff640 1 -- start start 2024-09-19T07:19:47.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8747fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8740046d20 0x7f87400e3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8747fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8740047620 0x7f87400c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8747fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87400d42a0 0x7f874002bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8747fff640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f87400072b0 con 0x7f8740046d20 2024-09-19T07:19:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8747fff640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8740007130 con 0x7f87400d42a0 2024-09-19T07:19:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8747fff640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f87400e1a70 con 0x7f8740047620 2024-09-19T07:19:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8746ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8740046d20 0x7f87400e3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8746ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8740046d20 0x7f87400e3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40718/0 (socket says 172.21.15.83:40718) 2024-09-19T07:19:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8746ffd640 1 -- 172.21.15.83:0/532158026 learned_addr learned my addr 172.21.15.83:0/532158026 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8746ffd640 1 -- 172.21.15.83:0/532158026 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8740047620 msgr2=0x7f87400c7580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:19:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8746ffd640 1 --2- 172.21.15.83:0/532158026 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8740047620 0x7f87400c7580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8746ffd640 1 -- 172.21.15.83:0/532158026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87400d42a0 msgr2=0x7f874002bf80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:19:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8746ffd640 1 --2- 172.21.15.83:0/532158026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87400d42a0 0x7f874002bf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8746ffd640 1 -- 172.21.15.83:0/532158026 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87400ca410 con 0x7f8740046d20 2024-09-19T07:19:47.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8746ffd640 1 --2- 172.21.15.83:0/532158026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8740046d20 0x7f87400e3990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f874802dd60 tx=0x7f8748063aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f9506c0e640 1 -- 172.21.15.83:0/2320791756 wait complete. 2024-09-19T07:19:47.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f9506c0e640 1 Processor -- start 2024-09-19T07:19:47.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7feedbfff640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed404ac30 0x7feed40d15a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7feedc0694d0 tx=0x7feedc069d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.339+0000 7f81ecba3640 1 -- 172.21.15.83:0/586690716 wait complete. 2024-09-19T07:19:47.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81ecba3640 1 Processor -- start 2024-09-19T07:19:47.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f872ffff640 1 -- 172.21.15.83:0/532158026 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8748129d60 con 0x7f8740046d20 2024-09-19T07:19:47.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8747fff640 1 -- 172.21.15.83:0/532158026 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f874002af70 con 0x7f8740046d20 2024-09-19T07:19:47.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f8747fff640 1 -- 172.21.15.83:0/532158026 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f874002ab90 con 0x7f8740046d20 2024-09-19T07:19:47.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fdebde1a640 1 -- 172.21.15.83:0/2244449791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeb80c6570 con 0x7fdeb80c3e10 2024-09-19T07:19:47.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93a4bb640 1 -- 172.21.15.83:0/247340572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c0d3570 msgr2=0x7fb92c0d0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93a4bb640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c0d3570 0x7fb92c0d0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93a4bb640 1 -- 172.21.15.83:0/247340572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c048840 msgr2=0x7fb92c0e66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93a4bb640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c048840 0x7fb92c0e66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93a4bb640 1 -- 172.21.15.83:0/247340572 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c044400 con 0x7fb92c046630 2024-09-19T07:19:47.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f9506c0e640 1 -- start start 2024-09-19T07:19:47.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f9506c0e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94f800b1e0 0x7f94f80407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f9506c0e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f80db810 0x7f94f80d4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f9506c0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f80dec50 0x7f94f80d4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f9506c0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f94f8045730 con 0x7f94f80dec50 2024-09-19T07:19:47.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f9506c0e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f94f80455b0 con 0x7f94f80db810 2024-09-19T07:19:47.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f9506c0e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f94f803e990 con 0x7f94f800b1e0 2024-09-19T07:19:47.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f139882e640 1 -- 172.21.15.83:0/1570461512 wait complete. 2024-09-19T07:19:47.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f872ffff640 1 -- 172.21.15.83:0/532158026 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8748131460 con 0x7f8740046d20 2024-09-19T07:19:47.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f872ffff640 1 -- 172.21.15.83:0/532158026 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8748131810 con 0x7f8740046d20 2024-09-19T07:19:47.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81ecba3640 1 -- start start 2024-09-19T07:19:47.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81ecba3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81d0004810 0x7f81e81f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81ecba3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81e814f510 0x7f81e80c6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81ecba3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81e81f3610 0x7f81e8145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81ecba3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f81e81529e0 con 0x7f81d0004810 2024-09-19T07:19:47.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81ecba3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8155ec0 con 0x7f81e814f510 2024-09-19T07:19:47.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81ecba3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f81e81f2e30 con 0x7f81e81f3610 2024-09-19T07:19:47.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81e77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81d0004810 0x7f81e81f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81e77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81d0004810 0x7f81e81f42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40744/0 (socket says 172.21.15.83:40744) 2024-09-19T07:19:47.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81e77fe640 1 -- 172.21.15.83:0/1485128482 learned_addr learned my addr 172.21.15.83:0/1485128482 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93a4bb640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb92c046630 0x7fb92c041bf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9240ae040 tx=0x7fb9240191d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7feec7fff640 1 -- 172.21.15.83:0/2899600521 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feedc067c80 con 0x7feed404ac30 2024-09-19T07:19:47.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7feec7fff640 1 -- 172.21.15.83:0/2899600521 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feedc067e20 con 0x7feed404ac30 2024-09-19T07:19:47.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7feec7fff640 1 -- 172.21.15.83:0/2899600521 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feedc071690 con 0x7feed404ac30 2024-09-19T07:19:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81e6ffd640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81e814f510 0x7f81e80c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f81e7fff640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81e81f3610 0x7f81e8145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f139882e640 1 Processor -- start 2024-09-19T07:19:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f950540b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f80db810 0x7f94f80d4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f950640d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f80dec50 0x7f94f80d4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f950640d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f80dec50 0x7f94f80d4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40730/0 (socket says 172.21.15.83:40730) 2024-09-19T07:19:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f9505c0c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94f800b1e0 0x7f94f80407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f950640d640 1 -- 172.21.15.83:0/490267073 learned_addr learned my addr 172.21.15.83:0/490267073 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb92affd640 1 -- 172.21.15.83:0/247340572 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9240b9020 con 0x7fb92c046630 2024-09-19T07:19:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb92affd640 1 -- 172.21.15.83:0/247340572 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9240ab9b0 con 0x7fb92c046630 2024-09-19T07:19:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb92affd640 1 -- 172.21.15.83:0/247340572 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9240b0040 con 0x7fb92c046630 2024-09-19T07:19:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f872ffff640 1 -- 172.21.15.83:0/532158026 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f87481393e0 con 0x7f8740046d20 2024-09-19T07:19:47.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fde9ffff640 1 -- 172.21.15.83:0/2244449791 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdeac01a070 con 0x7fdeb80c3e10 2024-09-19T07:19:47.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7fb93c71f640 1 -- 172.21.15.83:0/247340572 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92c17f120 con 0x7fb92c046630 2024-09-19T07:19:47.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7fb93c71f640 1 -- 172.21.15.83:0/247340572 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb92c17f6e0 con 0x7fb92c046630 2024-09-19T07:19:47.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f81e6ffd640 1 -- 172.21.15.83:0/1485128482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81e81f3610 msgr2=0x7f81e8145500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f81e6ffd640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81e81f3610 0x7f81e8145500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f81e6ffd640 1 -- 172.21.15.83:0/1485128482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81d0004810 msgr2=0x7f81e81f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f81e6ffd640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81d0004810 0x7f81e81f42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f81e6ffd640 1 -- 172.21.15.83:0/1485128482 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e81e4f50 con 0x7f81e814f510 2024-09-19T07:19:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7feee1f7f640 1 -- 172.21.15.83:0/2899600521 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feed40e1740 con 0x7feed404ac30 2024-09-19T07:19:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7feee1f7f640 1 -- 172.21.15.83:0/2899600521 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feed40e1d00 con 0x7feed404ac30 2024-09-19T07:19:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7fde9ffff640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea40679c0 0x7fdea4069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f950640d640 1 -- 172.21.15.83:0/490267073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94f800b1e0 msgr2=0x7f94f80407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f950640d640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94f800b1e0 0x7f94f80407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f950640d640 1 -- 172.21.15.83:0/490267073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f80db810 msgr2=0x7f94f80d4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f950640d640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f80db810 0x7f94f80d4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f950640d640 1 -- 172.21.15.83:0/490267073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f8183830 con 0x7f94f80dec50 2024-09-19T07:19:47.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7feee1f7f640 1 -- 172.21.15.83:0/2899600521 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feed40d45b0 con 0x7feed404ac30 2024-09-19T07:19:47.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f81e6ffd640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81e814f510 0x7f81e80c6440 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f81d000c460 tx=0x7f81d0005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7fde9ffff640 1 -- 172.21.15.83:0/2244449791 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fdeac085a60 con 0x7fdeb80c3e10 2024-09-19T07:19:47.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7fdeb6ffd640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea40679c0 0x7fdea4069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f950540b640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f80db810 0x7f94f80d4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.343+0000 7f139882e640 1 -- start start 2024-09-19T07:19:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f139882e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138c00b2c0 0x7f138c0e2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f139882e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f138c0de2d0 0x7f138c0e1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f139882e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f138c0dec50 0x7f138c183140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f139882e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f138c0078a0 con 0x7f138c00b2c0 2024-09-19T07:19:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f139882e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f138c007720 con 0x7f138c0dec50 2024-09-19T07:19:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f139882e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f138c0497c0 con 0x7f138c0de2d0 2024-09-19T07:19:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f1392d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138c00b2c0 0x7f138c0e2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f1392d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138c00b2c0 0x7f138c0e2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40750/0 (socket says 172.21.15.83:40750) 2024-09-19T07:19:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f1392d9d640 1 -- 172.21.15.83:0/506864356 learned_addr learned my addr 172.21.15.83:0/506864356 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f139902f640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f138c0dec50 0x7f138c183140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f81cffff640 1 -- 172.21.15.83:0/1485128482 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e00b9030 con 0x7f81e814f510 2024-09-19T07:19:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f81cffff640 1 -- 172.21.15.83:0/1485128482 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81e0003a40 con 0x7f81e814f510 2024-09-19T07:19:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f81cffff640 1 -- 172.21.15.83:0/1485128482 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e00b0040 con 0x7f81e814f510 2024-09-19T07:19:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f81ecba3640 1 -- 172.21.15.83:0/1485128482 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e81e4c70 con 0x7f81e814f510 2024-09-19T07:19:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f81ecba3640 1 -- 172.21.15.83:0/1485128482 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e81f7130 con 0x7f81e814f510 2024-09-19T07:19:47.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f950640d640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f80dec50 0x7f94f80d4db0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9500000b70 tx=0x7f95000714b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f1392d9d640 1 -- 172.21.15.83:0/506864356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f138c0de2d0 msgr2=0x7f138c0e1810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:19:47.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f1392d9d640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f138c0de2d0 0x7f138c0e1810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f1392d9d640 1 -- 172.21.15.83:0/506864356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f138c0dec50 msgr2=0x7f138c183140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f1392d9d640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f138c0dec50 0x7f138c183140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f1392d9d640 1 -- 172.21.15.83:0/506864356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138c0d7b50 con 0x7f138c00b2c0 2024-09-19T07:19:47.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7fdeb6ffd640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea40679c0 0x7fdea4069e80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80c4870 tx=0x7fdea8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f1392d9d640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138c00b2c0 0x7f138c0e2140 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1394051900 tx=0x7f1394069900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7fb93c71f640 1 -- 172.21.15.83:0/247340572 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb91c0052a0 con 0x7fb92c046630 2024-09-19T07:19:47.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f7f52ffd640 1 --2- 172.21.15.83:0/2324023727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f5414feb0 0x7f7f5406bc80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7f5406d740 tx=0x7f7f4401c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410ece8be1fa6027 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f8747fff640 1 -- 172.21.15.83:0/532158026 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87280052a0 con 0x7f8740046d20 2024-09-19T07:19:47.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f81ecba3640 1 -- 172.21.15.83:0/1485128482 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e814be70 con 0x7f81e814f510 2024-09-19T07:19:47.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7fb92affd640 1 -- 172.21.15.83:0/247340572 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb9240bc020 con 0x7fb92c046630 2024-09-19T07:19:47.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f872ffff640 1 --2- 172.21.15.83:0/532158026 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8734067990 0x7f8734069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f872ffff640 1 -- 172.21.15.83:0/532158026 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f874819bcc0 con 0x7f8740046d20 2024-09-19T07:19:47.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f87467fc640 1 --2- 172.21.15.83:0/532158026 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8734067990 0x7f8734069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7fb92affd640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb918067990 0x7fb918069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7fb92affd640 1 -- 172.21.15.83:0/247340572 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fb92411fc70 con 0x7fb92c046630 2024-09-19T07:19:47.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.347+0000 7f87467fc640 1 --2- 172.21.15.83:0/532158026 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8734067990 0x7f8734069e50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87400c8030 tx=0x7f873c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7fb939cba640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb918067990 0x7fb918069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f81cffff640 1 -- 172.21.15.83:0/1485128482 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f81e00050d0 con 0x7f81e814f510 2024-09-19T07:19:47.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7fb939cba640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb918067990 0x7fb918069e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0d9460 tx=0x7fb930009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f81cffff640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f81d4067990 0x7f81d4069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f81cffff640 1 -- 172.21.15.83:0/1485128482 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f81e0120830 con 0x7f81e814f510 2024-09-19T07:19:47.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f81e77fe640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f81d4067990 0x7f81d4069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f81e77fe640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f81d4067990 0x7f81d4069e50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00a840 tx=0x7f81dc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f7f51ffb640 1 -- 172.21.15.83:0/2324023727 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f440b0070 con 0x7f7f5414feb0 2024-09-19T07:19:47.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f137b7fe640 1 -- 172.21.15.83:0/506864356 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394067c80 con 0x7f138c00b2c0 2024-09-19T07:19:47.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f94f67fc640 1 -- 172.21.15.83:0/490267073 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95000698a0 con 0x7f94f80dec50 2024-09-19T07:19:47.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7fde9ffff640 1 -- 172.21.15.83:0/2244449791 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdeac00ee60 con 0x7fdeb80c3e10 2024-09-19T07:19:47.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f139882e640 1 -- 172.21.15.83:0/506864356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f138c0d8ca0 con 0x7f138c00b2c0 2024-09-19T07:19:47.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f139882e640 1 -- 172.21.15.83:0/506864356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f138c183680 con 0x7f138c00b2c0 2024-09-19T07:19:47.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f9506c0e640 1 -- 172.21.15.83:0/490267073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94f8183b10 con 0x7f94f80dec50 2024-09-19T07:19:47.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f9506c0e640 1 -- 172.21.15.83:0/490267073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94f803f740 con 0x7f94f80dec50 2024-09-19T07:19:47.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f9506c0e640 1 -- 172.21.15.83:0/490267073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94f8031080 con 0x7f94f80dec50 2024-09-19T07:19:47.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7f139882e640 1 -- 172.21.15.83:0/506864356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f138c0de710 con 0x7f138c00b2c0 2024-09-19T07:19:47.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.351+0000 7feec7fff640 1 -- 172.21.15.83:0/2899600521 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feedc071830 con 0x7feed404ac30 2024-09-19T07:19:47.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.355+0000 7feec7fff640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feecc067de0 0x7feecc06a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.355+0000 7feedb7fe640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feecc067de0 0x7feecc06a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.355+0000 7feedb7fe640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feecc067de0 0x7feecc06a2a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feec8007fd0 tx=0x7feec8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.355+0000 7f94f67fc640 1 -- 172.21.15.83:0/490267073 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9500071b30 con 0x7f94f80dec50 2024-09-19T07:19:47.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.355+0000 7f94f67fc640 1 -- 172.21.15.83:0/490267073 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500071d60 con 0x7f94f80dec50 2024-09-19T07:19:47.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.355+0000 7f872ffff640 1 -- 172.21.15.83:0/532158026 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f87480929f0 con 0x7f8740046d20 2024-09-19T07:19:47.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.355+0000 7f137b7fe640 1 -- 172.21.15.83:0/506864356 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1394067e20 con 0x7f138c00b2c0 2024-09-19T07:19:47.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.355+0000 7f137b7fe640 1 -- 172.21.15.83:0/506864356 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394063230 con 0x7f138c00b2c0 2024-09-19T07:19:47.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.355+0000 7feec7fff640 1 -- 172.21.15.83:0/2899600521 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7feedc0d02c0 con 0x7feed404ac30 2024-09-19T07:19:47.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.355+0000 7f7f51ffb640 1 -- 172.21.15.83:0/2324023727 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f440ab600 con 0x7f7f5414feb0 2024-09-19T07:19:47.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.355+0000 7f7f51ffb640 1 -- 172.21.15.83:0/2324023727 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f440ab9d0 con 0x7f7f5414feb0 2024-09-19T07:19:47.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f7f53fff640 1 -- 172.21.15.83:0/2324023727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f5414feb0 msgr2=0x7f7f5406bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f7f53fff640 1 --2- 172.21.15.83:0/2324023727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f5414feb0 0x7f7f5406bc80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7f5406d740 tx=0x7f7f4401c310 comp rx=0 tx=0).stop 2024-09-19T07:19:47.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f7f53fff640 1 -- 172.21.15.83:0/2324023727 shutdown_connections 2024-09-19T07:19:47.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f7f53fff640 1 --2- 172.21.15.83:0/2324023727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f5414feb0 0x7f7f5406bc80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f7f53fff640 1 -- 172.21.15.83:0/2324023727 >> 172.21.15.83:0/2324023727 conn(0x7f7f540aaaa0 msgr2=0x7f7f540c82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f94f67fc640 1 -- 172.21.15.83:0/490267073 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9500093060 con 0x7f94f80dec50 2024-09-19T07:19:47.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f94f67fc640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94e40678c0 0x7f94e4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f137b7fe640 1 -- 172.21.15.83:0/506864356 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f13940633d0 con 0x7f138c00b2c0 2024-09-19T07:19:47.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f7f53fff640 1 -- 172.21.15.83:0/2324023727 shutdown_connections 2024-09-19T07:19:47.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f7f53fff640 1 -- 172.21.15.83:0/2324023727 wait complete. 2024-09-19T07:19:47.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f9505c0c640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94e40678c0 0x7f94e4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f9505c0c640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94e40678c0 0x7f94e4069d80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f94f00ad000 tx=0x7f94f00a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f137b7fe640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f137c067a20 0x7f137c069ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f139259c640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f137c067a20 0x7f137c069ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f7f53fff640 1 Processor -- start 2024-09-19T07:19:47.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f139259c640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f137c067a20 0x7f137c069ee0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f138c045990 tx=0x7f1388009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.359+0000 7f7f53fff640 1 -- start start 2024-09-19T07:19:47.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f53fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f5406e820 0x7f7f54156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f53fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f540ab3c0 0x7f7f54151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f53fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f5414feb0 0x7f7f54153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f53fff640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7f540ad4f0 con 0x7f7f5414feb0 2024-09-19T07:19:47.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f53fff640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7f54150290 con 0x7f7f5406e820 2024-09-19T07:19:47.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f53fff640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7f5406f850 con 0x7f7f540ab3c0 2024-09-19T07:19:47.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f537fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f5414feb0 0x7f7f54153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f537fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f5414feb0 0x7f7f54153b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40760/0 (socket says 172.21.15.83:40760) 2024-09-19T07:19:47.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f537fe640 1 -- 172.21.15.83:0/1636212836 learned_addr learned my addr 172.21.15.83:0/1636212836 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:47.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f52ffd640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f5406e820 0x7f7f54156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f527fc640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f540ab3c0 0x7f7f54151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f537fe640 1 -- 172.21.15.83:0/1636212836 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f540ab3c0 msgr2=0x7f7f54151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f537fe640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f540ab3c0 0x7f7f54151830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f537fe640 1 -- 172.21.15.83:0/1636212836 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f5406e820 msgr2=0x7f7f54156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f537fe640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f5406e820 0x7f7f54156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f537fe640 1 -- 172.21.15.83:0/1636212836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5415de10 con 0x7f7f5414feb0 2024-09-19T07:19:47.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7fb92affd640 1 -- 172.21.15.83:0/247340572 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb9240c3030 con 0x7fb92c046630 2024-09-19T07:19:47.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f7f537fe640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f5414feb0 0x7f7f54153b00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800b410 tx=0x7f7f4800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.363+0000 7f81cffff640 1 -- 172.21.15.83:0/1485128482 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f81e00c3020 con 0x7f81e814f510 2024-09-19T07:19:47.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.367+0000 7f94f67fc640 1 -- 172.21.15.83:0/490267073 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f9500096de0 con 0x7f94f80dec50 2024-09-19T07:19:47.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.367+0000 7f7f3b7fe640 1 -- 172.21.15.83:0/1636212836 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48015020 con 0x7f7f5414feb0 2024-09-19T07:19:47.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.367+0000 7feec7fff640 1 -- 172.21.15.83:0/2899600521 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7feedc0d2220 con 0x7feed404ac30 2024-09-19T07:19:47.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.367+0000 7f7f53fff640 1 -- 172.21.15.83:0/1636212836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f54155d60 con 0x7f7f5414feb0 2024-09-19T07:19:47.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.367+0000 7f7f53fff640 1 -- 172.21.15.83:0/1636212836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f541523e0 con 0x7f7f5414feb0 2024-09-19T07:19:47.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.367+0000 7f137b7fe640 1 -- 172.21.15.83:0/506864356 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f13940d0380 con 0x7f138c00b2c0 2024-09-19T07:19:47.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.367+0000 7f7f53fff640 1 -- 172.21.15.83:0/1636212836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f5406ec70 con 0x7f7f5414feb0 2024-09-19T07:19:47.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.375+0000 7f94f67fc640 1 -- 172.21.15.83:0/490267073 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f95000d14e0 con 0x7f94f80dec50 2024-09-19T07:19:47.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.387+0000 7f137b7fe640 1 -- 172.21.15.83:0/506864356 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f139409a030 con 0x7f138c00b2c0 2024-09-19T07:19:47.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.387+0000 7f7f3b7fe640 1 -- 172.21.15.83:0/1636212836 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f48011830 con 0x7f7f5414feb0 2024-09-19T07:19:47.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.387+0000 7f7f3b7fe640 1 -- 172.21.15.83:0/1636212836 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48011c20 con 0x7f7f5414feb0 2024-09-19T07:19:47.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.387+0000 7f7f3b7fe640 1 -- 172.21.15.83:0/1636212836 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7f48018020 con 0x7f7f5414feb0 2024-09-19T07:19:47.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.387+0000 7f7f3b7fe640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f400678f0 0x7f7f40069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:47.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.387+0000 7f7f3b7fe640 1 -- 172.21.15.83:0/1636212836 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f7f48086480 con 0x7f7f5414feb0 2024-09-19T07:19:47.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.395+0000 7f7f52ffd640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f400678f0 0x7f7f40069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:47.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.395+0000 7f7f52ffd640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f400678f0 0x7f7f40069db0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7f440ad000 tx=0x7f7f440a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:47.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.395+0000 7f7f3b7fe640 1 -- 172.21.15.83:0/1636212836 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7f4801f020 con 0x7f7f5414feb0 2024-09-19T07:19:47.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.659+0000 7fdebde1a640 1 -- 172.21.15.83:0/2244449791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fdeb80ad500 con 0x7fdeb80c3e10 2024-09-19T07:19:47.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.663+0000 7fde9ffff640 1 -- 172.21.15.83:0/2244449791 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdeac055c00 con 0x7fdeb80c3e10 2024-09-19T07:19:47.665 INFO:teuthology.orchestra.run.smithi083.stdout:60129542151 2024-09-19T07:19:47.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.663+0000 7fde9dffb640 1 -- 172.21.15.83:0/2244449791 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea40679c0 msgr2=0x7fdea4069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.663+0000 7fde9dffb640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea40679c0 0x7fdea4069e80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80c4870 tx=0x7fdea8008040 comp rx=0 tx=0).stop 2024-09-19T07:19:47.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.663+0000 7fde9dffb640 1 -- 172.21.15.83:0/2244449791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb80c3e10 msgr2=0x7fdeb814d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.663+0000 7fde9dffb640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb80c3e10 0x7fdeb814d920 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00e870 tx=0x7fdeac00ed30 comp rx=0 tx=0).stop 2024-09-19T07:19:47.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.663+0000 7fde9dffb640 1 -- 172.21.15.83:0/2244449791 shutdown_connections 2024-09-19T07:19:47.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.663+0000 7fde9dffb640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea40679c0 0x7fdea4069e80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.663+0000 7fde9dffb640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb81ea730 0x7fdeb81eabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.663+0000 7fde9dffb640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb81e71c0 0x7fdeb81e7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.663+0000 7fde9dffb640 1 --2- 172.21.15.83:0/2244449791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb80c3e10 0x7fdeb814d920 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.667+0000 7fde9dffb640 1 -- 172.21.15.83:0/2244449791 >> 172.21.15.83:0/2244449791 conn(0x7fdeb80c53f0 msgr2=0x7fdeb806bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.667+0000 7fde9dffb640 1 -- 172.21.15.83:0/2244449791 shutdown_connections 2024-09-19T07:19:47.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.667+0000 7fde9dffb640 1 -- 172.21.15.83:0/2244449791 wait complete. 2024-09-19T07:19:47.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.679+0000 7f8747fff640 1 -- 172.21.15.83:0/532158026 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f8728005590 con 0x7f8740046d20 2024-09-19T07:19:47.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.683+0000 7f872ffff640 1 -- 172.21.15.83:0/532158026 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f874816e020 con 0x7f8740046d20 2024-09-19T07:19:47.686 INFO:teuthology.orchestra.run.smithi083.stdout:60129542152 2024-09-19T07:19:47.687 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.3 2024-09-19T07:19:47.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7feee1f7f640 1 -- 172.21.15.83:0/2899600521 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7feed40e1f40 con 0x7feed404ac30 2024-09-19T07:19:47.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7feec7fff640 1 -- 172.21.15.83:0/2899600521 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7feedc0d6520 con 0x7feed404ac30 2024-09-19T07:19:47.688 INFO:teuthology.orchestra.run.smithi083.stdout:60129542153 2024-09-19T07:19:47.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f8747fff640 1 -- 172.21.15.83:0/532158026 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8734067990 msgr2=0x7f8734069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f8747fff640 1 --2- 172.21.15.83:0/532158026 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8734067990 0x7f8734069e50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87400c8030 tx=0x7f873c008040 comp rx=0 tx=0).stop 2024-09-19T07:19:47.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f8747fff640 1 -- 172.21.15.83:0/532158026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8740046d20 msgr2=0x7f87400e3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f8747fff640 1 --2- 172.21.15.83:0/532158026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8740046d20 0x7f87400e3990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f874802dd60 tx=0x7f8748063aa0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f8747fff640 1 -- 172.21.15.83:0/532158026 shutdown_connections 2024-09-19T07:19:47.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f8747fff640 1 --2- 172.21.15.83:0/532158026 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8734067990 0x7f8734069e50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f8747fff640 1 --2- 172.21.15.83:0/532158026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87400d42a0 0x7f874002bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f8747fff640 1 --2- 172.21.15.83:0/532158026 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8740047620 0x7f87400c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f8747fff640 1 --2- 172.21.15.83:0/532158026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8740046d20 0x7f87400e3990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f8747fff640 1 -- 172.21.15.83:0/532158026 >> 172.21.15.83:0/532158026 conn(0x7f8740047d20 msgr2=0x7f874000a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f8747fff640 1 -- 172.21.15.83:0/532158026 shutdown_connections 2024-09-19T07:19:47.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f8747fff640 1 -- 172.21.15.83:0/532158026 wait complete. 2024-09-19T07:19:47.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.687+0000 7f81ecba3640 1 -- 172.21.15.83:0/1485128482 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f81e81e5220 con 0x7f81e814f510 2024-09-19T07:19:47.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7fb93c71f640 1 -- 172.21.15.83:0/247340572 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fb91c005590 con 0x7fb92c046630 2024-09-19T07:19:47.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7f81cffff640 1 -- 172.21.15.83:0/1485128482 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f81e0016470 con 0x7f81e814f510 2024-09-19T07:19:47.692 INFO:teuthology.orchestra.run.smithi083.stdout:60129542152 2024-09-19T07:19:47.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7feee1f7f640 1 -- 172.21.15.83:0/2899600521 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feecc067de0 msgr2=0x7feecc06a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7feee1f7f640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feecc067de0 0x7feecc06a2a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feec8007fd0 tx=0x7feec8009040 comp rx=0 tx=0).stop 2024-09-19T07:19:47.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7feee1f7f640 1 -- 172.21.15.83:0/2899600521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed404ac30 msgr2=0x7feed40d15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7feee1f7f640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed404ac30 0x7feed40d15a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7feedc0694d0 tx=0x7feedc069d00 comp rx=0 tx=0).stop 2024-09-19T07:19:47.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7feee1f7f640 1 -- 172.21.15.83:0/2899600521 shutdown_connections 2024-09-19T07:19:47.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7feee1f7f640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feecc067de0 0x7feecc06a2a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7feee1f7f640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed404ac30 0x7feed40d15a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7feee1f7f640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed400b880 0x7feed4044100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7feee1f7f640 1 --2- 172.21.15.83:0/2899600521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed4006c80 0x7feed4041800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7feee1f7f640 1 -- 172.21.15.83:0/2899600521 >> 172.21.15.83:0/2899600521 conn(0x7feed4047d20 msgr2=0x7feed40e80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7feee1f7f640 1 -- 172.21.15.83:0/2899600521 shutdown_connections 2024-09-19T07:19:47.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.691+0000 7feee1f7f640 1 -- 172.21.15.83:0/2899600521 wait complete. 2024-09-19T07:19:47.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb92affd640 1 -- 172.21.15.83:0/247340572 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb9240160b0 con 0x7fb92c046630 2024-09-19T07:19:47.694 INFO:teuthology.orchestra.run.smithi083.stdout:60129542151 2024-09-19T07:19:47.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7f81ecba3640 1 -- 172.21.15.83:0/1485128482 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f81d4067990 msgr2=0x7f81d4069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7f81ecba3640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f81d4067990 0x7f81d4069e50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00a840 tx=0x7f81dc008040 comp rx=0 tx=0).stop 2024-09-19T07:19:47.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7f81ecba3640 1 -- 172.21.15.83:0/1485128482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81e814f510 msgr2=0x7f81e80c6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7f81ecba3640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81e814f510 0x7f81e80c6440 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f81d000c460 tx=0x7f81d0005840 comp rx=0 tx=0).stop 2024-09-19T07:19:47.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7f81ecba3640 1 -- 172.21.15.83:0/1485128482 shutdown_connections 2024-09-19T07:19:47.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7f81ecba3640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f81d4067990 0x7f81d4069e50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7f81ecba3640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81e81f3610 0x7f81e8145500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7f81ecba3640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81e814f510 0x7f81e80c6440 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7f81ecba3640 1 --2- 172.21.15.83:0/1485128482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81d0004810 0x7f81e81f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7f81ecba3640 1 -- 172.21.15.83:0/1485128482 >> 172.21.15.83:0/1485128482 conn(0x7f81e806dbc0 msgr2=0x7f81e814f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7f81ecba3640 1 -- 172.21.15.83:0/1485128482 shutdown_connections 2024-09-19T07:19:47.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7f81ecba3640 1 -- 172.21.15.83:0/1485128482 wait complete. 2024-09-19T07:19:47.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb93c71f640 1 -- 172.21.15.83:0/247340572 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb918067990 msgr2=0x7fb918069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb93c71f640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb918067990 0x7fb918069e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0d9460 tx=0x7fb930009040 comp rx=0 tx=0).stop 2024-09-19T07:19:47.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb93c71f640 1 -- 172.21.15.83:0/247340572 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb92c046630 msgr2=0x7fb92c041bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb93c71f640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb92c046630 0x7fb92c041bf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9240ae040 tx=0x7fb9240191d0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb93c71f640 1 -- 172.21.15.83:0/247340572 shutdown_connections 2024-09-19T07:19:47.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb93c71f640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb918067990 0x7fb918069e50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb93c71f640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c0d3570 0x7fb92c0d0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb93c71f640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c048840 0x7fb92c0e66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb93c71f640 1 --2- 172.21.15.83:0/247340572 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb92c046630 0x7fb92c041bf0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb93c71f640 1 -- 172.21.15.83:0/247340572 >> 172.21.15.83:0/247340572 conn(0x7fb92c047d20 msgr2=0x7fb92c176150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb93c71f640 1 -- 172.21.15.83:0/247340572 shutdown_connections 2024-09-19T07:19:47.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.695+0000 7fb93c71f640 1 -- 172.21.15.83:0/247340572 wait complete. 2024-09-19T07:19:47.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.703+0000 7f9506c0e640 1 -- 172.21.15.83:0/490267073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f94f8007000 con 0x7f94f80dec50 2024-09-19T07:19:47.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.703+0000 7f94f67fc640 1 -- 172.21.15.83:0/490267073 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f95000d57e0 con 0x7f94f80dec50 2024-09-19T07:19:47.703 INFO:teuthology.orchestra.run.smithi083.stdout:60129542153 2024-09-19T07:19:47.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.703+0000 7f9506c0e640 1 -- 172.21.15.83:0/490267073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94e40678c0 msgr2=0x7f94e4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.703+0000 7f9506c0e640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94e40678c0 0x7f94e4069d80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f94f00ad000 tx=0x7f94f00a6000 comp rx=0 tx=0).stop 2024-09-19T07:19:47.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.707+0000 7f9506c0e640 1 -- 172.21.15.83:0/490267073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f80dec50 msgr2=0x7f94f80d4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.707+0000 7f9506c0e640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f80dec50 0x7f94f80d4db0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9500000b70 tx=0x7f95000714b0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.707+0000 7f9506c0e640 1 -- 172.21.15.83:0/490267073 shutdown_connections 2024-09-19T07:19:47.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.707+0000 7f9506c0e640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94e40678c0 0x7f94e4069d80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.707+0000 7f9506c0e640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f80dec50 0x7f94f80d4db0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.707+0000 7f9506c0e640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f80db810 0x7f94f80d4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.707+0000 7f9506c0e640 1 --2- 172.21.15.83:0/490267073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94f800b1e0 0x7f94f80407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.707+0000 7f9506c0e640 1 -- 172.21.15.83:0/490267073 >> 172.21.15.83:0/490267073 conn(0x7f94f8047d20 msgr2=0x7f94f8049a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.707+0000 7f9506c0e640 1 -- 172.21.15.83:0/490267073 shutdown_connections 2024-09-19T07:19:47.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.707+0000 7f9506c0e640 1 -- 172.21.15.83:0/490267073 wait complete. 2024-09-19T07:19:47.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.707+0000 7f139882e640 1 -- 172.21.15.83:0/506864356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f138c182a80 con 0x7f138c00b2c0 2024-09-19T07:19:47.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.711+0000 7f137b7fe640 1 -- 172.21.15.83:0/506864356 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f13940d29e0 con 0x7f138c00b2c0 2024-09-19T07:19:47.710 INFO:teuthology.orchestra.run.smithi083.stdout:60129542153 2024-09-19T07:19:47.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.715+0000 7f139882e640 1 -- 172.21.15.83:0/506864356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f137c067a20 msgr2=0x7f137c069ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.715+0000 7f139882e640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f137c067a20 0x7f137c069ee0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f138c045990 tx=0x7f1388009190 comp rx=0 tx=0).stop 2024-09-19T07:19:47.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.715+0000 7f139882e640 1 -- 172.21.15.83:0/506864356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138c00b2c0 msgr2=0x7f138c0e2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.715+0000 7f139882e640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138c00b2c0 0x7f138c0e2140 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1394051900 tx=0x7f1394069900 comp rx=0 tx=0).stop 2024-09-19T07:19:47.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.715+0000 7f139882e640 1 -- 172.21.15.83:0/506864356 shutdown_connections 2024-09-19T07:19:47.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.715+0000 7f139882e640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f137c067a20 0x7f137c069ee0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.715+0000 7f139882e640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f138c0dec50 0x7f138c183140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.715+0000 7f139882e640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f138c0de2d0 0x7f138c0e1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.715+0000 7f139882e640 1 --2- 172.21.15.83:0/506864356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138c00b2c0 0x7f138c0e2140 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.715+0000 7f139882e640 1 -- 172.21.15.83:0/506864356 >> 172.21.15.83:0/506864356 conn(0x7f138c048340 msgr2=0x7f138c005950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.715+0000 7f139882e640 1 -- 172.21.15.83:0/506864356 shutdown_connections 2024-09-19T07:19:47.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.715+0000 7f139882e640 1 -- 172.21.15.83:0/506864356 wait complete. 2024-09-19T07:19:47.721 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.7 2024-09-19T07:19:47.724 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.6 2024-09-19T07:19:47.729 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.1 2024-09-19T07:19:47.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 -- 172.21.15.83:0/1636212836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f7f5414e320 con 0x7f7f5414feb0 2024-09-19T07:19:47.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f3b7fe640 1 -- 172.21.15.83:0/1636212836 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7f48056c70 con 0x7f7f5414feb0 2024-09-19T07:19:47.731 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.5 2024-09-19T07:19:47.731 INFO:teuthology.orchestra.run.smithi083.stdout:60129542151 2024-09-19T07:19:47.732 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.4 2024-09-19T07:19:47.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 -- 172.21.15.83:0/1636212836 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f400678f0 msgr2=0x7f7f40069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f400678f0 0x7f7f40069db0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7f440ad000 tx=0x7f7f440a6000 comp rx=0 tx=0).stop 2024-09-19T07:19:47.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 -- 172.21.15.83:0/1636212836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f5414feb0 msgr2=0x7f7f54153b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:47.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f5414feb0 0x7f7f54153b00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800b410 tx=0x7f7f4800b8d0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 -- 172.21.15.83:0/1636212836 shutdown_connections 2024-09-19T07:19:47.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f400678f0 0x7f7f40069db0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f5414feb0 0x7f7f54153b00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f540ab3c0 0x7f7f54151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 --2- 172.21.15.83:0/1636212836 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f5406e820 0x7f7f54156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:47.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 -- 172.21.15.83:0/1636212836 >> 172.21.15.83:0/1636212836 conn(0x7f7f540aaaa0 msgr2=0x7f7f540c2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:47.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 -- 172.21.15.83:0/1636212836 shutdown_connections 2024-09-19T07:19:47.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:47.731+0000 7f7f53fff640 1 -- 172.21.15.83:0/1636212836 wait complete. 2024-09-19T07:19:47.740 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.2 2024-09-19T07:19:47.748 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.0 2024-09-19T07:19:48.688 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-19T07:19:48.722 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-19T07:19:48.725 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-19T07:19:48.731 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-19T07:19:48.732 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-19T07:19:48.733 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-19T07:19:48.741 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-19T07:19:48.748 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-19T07:19:48.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.863+0000 7f93c9485640 1 Processor -- start 2024-09-19T07:19:48.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.863+0000 7f93c9485640 1 -- start start 2024-09-19T07:19:48.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.867+0000 7f93c9485640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c4148570 0x7f93c40aa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.867+0000 7f93c9485640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c40aaf20 con 0x7f93c40c2ea0 2024-09-19T07:19:48.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.867+0000 7f93c9485640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c40c3440 con 0x7f93c4155b00 2024-09-19T07:19:48.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.867+0000 7f93c9485640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f93c4155670 con 0x7f93c4148570 2024-09-19T07:19:48.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.867+0000 7f93c27fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f93c40c2ea0 0x7f93c41525f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51584/0 (socket says 172.21.15.83:51584) 2024-09-19T07:19:48.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.867+0000 7f93c27fc640 1 -- 172.21.15.83:0/3951335301 learned_addr learned my addr 172.21.15.83:0/3951335301 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:48.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.867+0000 7f93c2ffd640 1 --2- 172.21.15.83:0/3951335301 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c4148570 0x7f93c40aa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.867+0000 7f93c2ffd640 1 -- 172.21.15.83:0/3951335301 >> v1:172.21.15.184:6789/0 conn(0x7f93c4155b00 legacy=0x7f93c406da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.867+0000 7f93c2ffd640 1 -- 172.21.15.83:0/3951335301 >> v1:172.21.15.83:6789/0 conn(0x7f93c40c2ea0 legacy=0x7f93c41525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.867+0000 7f93c2ffd640 1 -- 172.21.15.83:0/3951335301 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c40c3720 con 0x7f93c4148570 2024-09-19T07:19:48.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.871+0000 7f93c2ffd640 1 --2- 172.21.15.83:0/3951335301 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c4148570 0x7f93c40aa9e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f93b400a990 tx=0x7f93b401d680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dd35bfc272be145 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.871+0000 7f93c1ffb640 1 -- 172.21.15.83:0/3951335301 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b40ad070 con 0x7f93c4148570 2024-09-19T07:19:48.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.871+0000 7f93c1ffb640 1 -- 172.21.15.83:0/3951335301 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93b40a7610 con 0x7f93c4148570 2024-09-19T07:19:48.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.871+0000 7f93c1ffb640 1 -- 172.21.15.83:0/3951335301 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b400b040 con 0x7f93c4148570 2024-09-19T07:19:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.871+0000 7f93c9485640 1 -- 172.21.15.83:0/3951335301 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c4148570 msgr2=0x7f93c40aa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.871+0000 7f93c9485640 1 --2- 172.21.15.83:0/3951335301 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c4148570 0x7f93c40aa9e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f93b400a990 tx=0x7f93b401d680 comp rx=0 tx=0).stop 2024-09-19T07:19:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.871+0000 7f93c9485640 1 -- 172.21.15.83:0/3951335301 shutdown_connections 2024-09-19T07:19:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.871+0000 7f93c9485640 1 --2- 172.21.15.83:0/3951335301 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c4148570 0x7f93c40aa9e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.871+0000 7f93c9485640 1 -- 172.21.15.83:0/3951335301 >> 172.21.15.83:0/3951335301 conn(0x7f93c40c7370 msgr2=0x7f93c40c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.871+0000 7f93c9485640 1 -- 172.21.15.83:0/3951335301 shutdown_connections 2024-09-19T07:19:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.871+0000 7f93c9485640 1 -- 172.21.15.83:0/3951335301 wait complete. 2024-09-19T07:19:48.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c9485640 1 Processor -- start 2024-09-19T07:19:48.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c9485640 1 -- start start 2024-09-19T07:19:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c9485640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c40c2ea0 0x7f93c414ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c9485640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c4155b00 0x7f93c4141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c9485640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93c41f07b0 0x7f93c40ab0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c9485640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f93c4148ad0 con 0x7f93c40c2ea0 2024-09-19T07:19:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c9485640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f93c4148950 con 0x7f93c41f07b0 2024-09-19T07:19:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c9485640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f93c415fd20 con 0x7f93c4155b00 2024-09-19T07:19:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c40c2ea0 0x7f93c414ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c40c2ea0 0x7f93c414ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40766/0 (socket says 172.21.15.83:40766) 2024-09-19T07:19:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c2ffd640 1 -- 172.21.15.83:0/2556133283 learned_addr learned my addr 172.21.15.83:0/2556133283 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c37fe640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93c41f07b0 0x7f93c40ab0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c27fc640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c4155b00 0x7f93c4141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c2ffd640 1 -- 172.21.15.83:0/2556133283 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c4155b00 msgr2=0x7f93c4141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c2ffd640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c4155b00 0x7f93c4141070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c2ffd640 1 -- 172.21.15.83:0/2556133283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93c41f07b0 msgr2=0x7f93c40ab0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c2ffd640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93c41f07b0 0x7f93c40ab0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c2ffd640 1 -- 172.21.15.83:0/2556133283 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c4141d70 con 0x7f93c40c2ea0 2024-09-19T07:19:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.875+0000 7f93c37fe640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93c41f07b0 0x7f93c40ab0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:19:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.879+0000 7f93c2ffd640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c40c2ea0 0x7f93c414ee70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f93b401b2b0 tx=0x7f93b401a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.879+0000 7f93ab7fe640 1 -- 172.21.15.83:0/2556133283 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b40ad040 con 0x7f93c40c2ea0 2024-09-19T07:19:48.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.879+0000 7f93ab7fe640 1 -- 172.21.15.83:0/2556133283 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93b40b2860 con 0x7f93c40c2ea0 2024-09-19T07:19:48.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.879+0000 7f93c9485640 1 -- 172.21.15.83:0/2556133283 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c41efcd0 con 0x7f93c40c2ea0 2024-09-19T07:19:48.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.879+0000 7f93c9485640 1 -- 172.21.15.83:0/2556133283 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c4154c80 con 0x7f93c40c2ea0 2024-09-19T07:19:48.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.883+0000 7f93ab7fe640 1 -- 172.21.15.83:0/2556133283 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b40b2c30 con 0x7f93c40c2ea0 2024-09-19T07:19:48.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.883+0000 7f93ab7fe640 1 -- 172.21.15.83:0/2556133283 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f93b40b0070 con 0x7f93c40c2ea0 2024-09-19T07:19:48.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.883+0000 7f93ab7fe640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f93b00679c0 0x7f93b0069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.883+0000 7f93ab7fe640 1 -- 172.21.15.83:0/2556133283 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f93b41202f0 con 0x7f93c40c2ea0 2024-09-19T07:19:48.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.883+0000 7f93c27fc640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f93b00679c0 0x7f93b0069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.883+0000 7f93c9485640 1 -- 172.21.15.83:0/2556133283 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93a40052a0 con 0x7f93c40c2ea0 2024-09-19T07:19:48.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.891+0000 7f93ab7fe640 1 -- 172.21.15.83:0/2556133283 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f93b40173c0 con 0x7f93c40c2ea0 2024-09-19T07:19:48.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.895+0000 7f93c27fc640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f93b00679c0 0x7f93b0069e80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f93c40abc70 tx=0x7f93bc001820 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f36247640 1 Processor -- start 2024-09-19T07:19:48.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f36247640 1 -- start start 2024-09-19T07:19:48.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f36247640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f300aa630 0x7f7f300c2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f36247640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f300c2ff0 0x7f7f30157580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f36247640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f3015ac50 con 0x7f7f300c4150 2024-09-19T07:19:48.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f36247640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7f3015af30 con 0x7f7f300c2ff0 2024-09-19T07:19:48.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f36247640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7f300c22b0 con 0x7f7f300aa630 2024-09-19T07:19:48.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f2ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f300c2ff0 0x7f7f30157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f2ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f300c2ff0 0x7f7f30157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40772/0 (socket says 172.21.15.83:40772) 2024-09-19T07:19:48.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f2ffff640 1 -- 172.21.15.83:0/3183530910 learned_addr learned my addr 172.21.15.83:0/3183530910 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:48.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f2effd640 1 --2- 172.21.15.83:0/3183530910 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f300aa630 0x7f7f300c2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f2e7fc640 1 -- 172.21.15.83:0/3183530910 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4022587565 0 0) 0x7f7f3015ac50 con 0x7f7f300c4150 2024-09-19T07:19:48.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f2ffff640 1 -- 172.21.15.83:0/3183530910 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f300aa630 msgr2=0x7f7f300c2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:48.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f2ffff640 1 --2- 172.21.15.83:0/3183530910 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f300aa630 0x7f7f300c2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f2ffff640 1 -- 172.21.15.83:0/3183530910 >> v1:172.21.15.83:6789/0 conn(0x7f7f300c4150 legacy=0x7f7f3015a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.899+0000 7f7f2ffff640 1 -- 172.21.15.83:0/3183530910 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3015a910 con 0x7f7f300c2ff0 2024-09-19T07:19:48.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.903+0000 7f7f2ffff640 1 --2- 172.21.15.83:0/3183530910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f300c2ff0 0x7f7f30157580 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009e50 tx=0x7f7f2801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af8ad0c4c702ae7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.903+0000 7f7f2e7fc640 1 -- 172.21.15.83:0/3183530910 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f280a55d0 con 0x7f7f300c2ff0 2024-09-19T07:19:48.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.903+0000 7f7f2e7fc640 1 -- 172.21.15.83:0/3183530910 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f280a5770 con 0x7f7f300c2ff0 2024-09-19T07:19:48.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.907+0000 7f7f2e7fc640 1 -- 172.21.15.83:0/3183530910 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f280a5b40 con 0x7f7f300c2ff0 2024-09-19T07:19:48.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.907+0000 7f7f36247640 1 -- 172.21.15.83:0/3183530910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f300c2ff0 msgr2=0x7f7f30157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.907+0000 7f7f36247640 1 --2- 172.21.15.83:0/3183530910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f300c2ff0 0x7f7f30157580 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009e50 tx=0x7f7f2801c930 comp rx=0 tx=0).stop 2024-09-19T07:19:48.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.907+0000 7f7f36247640 1 -- 172.21.15.83:0/3183530910 shutdown_connections 2024-09-19T07:19:48.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.907+0000 7f7f36247640 1 --2- 172.21.15.83:0/3183530910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f300c2ff0 0x7f7f30157580 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.907+0000 7f7f36247640 1 --2- 172.21.15.83:0/3183530910 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f300aa630 0x7f7f300c2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.907+0000 7f7f36247640 1 -- 172.21.15.83:0/3183530910 >> 172.21.15.83:0/3183530910 conn(0x7f7f3006bc80 msgr2=0x7f7f3006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f36247640 1 -- 172.21.15.83:0/3183530910 shutdown_connections 2024-09-19T07:19:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f36247640 1 -- 172.21.15.83:0/3183530910 wait complete. 2024-09-19T07:19:48.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f36247640 1 Processor -- start 2024-09-19T07:19:48.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f36247640 1 -- start start 2024-09-19T07:19:48.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7fabb4983640 1 Processor -- start 2024-09-19T07:19:48.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7fabb4983640 1 -- start start 2024-09-19T07:19:48.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f36247640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f300aa630 0x7f7f30151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f36247640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f300c2ff0 0x7f7f301f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f36247640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f300c4150 0x7f7f30150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f2f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f300aa630 0x7f7f30151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f2f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f300aa630 0x7f7f30151880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40782/0 (socket says 172.21.15.83:40782) 2024-09-19T07:19:48.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f2f7fe640 1 -- 172.21.15.83:0/1410335803 learned_addr learned my addr 172.21.15.83:0/1410335803 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:48.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7f30157ed0 con 0x7f7f300aa630 2024-09-19T07:19:48.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3015daa0 con 0x7f7f300c4150 2024-09-19T07:19:48.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7f300c4740 con 0x7f7f300c2ff0 2024-09-19T07:19:48.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7fabb4983640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fabb00aa3b0 0x7fabb0157290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7fabb4983640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb006d4d0 con 0x7fabb015aa90 2024-09-19T07:19:48.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7fabb4983640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb006bd20 con 0x7fabb00c2980 2024-09-19T07:19:48.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7fabb4983640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fabb00aa7f0 con 0x7fabb00aa3b0 2024-09-19T07:19:48.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7fabae59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fabb015aa90 0x7fabb006cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51608/0 (socket says 172.21.15.83:51608) 2024-09-19T07:19:48.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7fabae59c640 1 -- 172.21.15.83:0/1457614250 learned_addr learned my addr 172.21.15.83:0/1457614250 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:48.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f2effd640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f300c2ff0 0x7f7f301f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7f7f2ffff640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f300c4150 0x7f7f30150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.911+0000 7fabaf59e640 1 --2- 172.21.15.83:0/1457614250 >> v2:172.21.15.184:3300/0 conn(0x7fabb00aa3b0 0x7fabb0157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6b3a1640 1 Processor -- start 2024-09-19T07:19:48.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f7f2f7fe640 1 -- 172.21.15.83:0/1410335803 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f300c2ff0 msgr2=0x7f7f301f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f7f2f7fe640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f300c2ff0 0x7f7f301f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f7f2f7fe640 1 -- 172.21.15.83:0/1410335803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f300c4150 msgr2=0x7f7f30150390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f7f2f7fe640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f300c4150 0x7f7f30150390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f7f2f7fe640 1 -- 172.21.15.83:0/1410335803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3014ab30 con 0x7f7f300aa630 2024-09-19T07:19:48.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7fabadd9b640 1 -- 172.21.15.83:0/1457614250 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2602487623 0 0) 0x7fabb006d4d0 con 0x7fabb015aa90 2024-09-19T07:19:48.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7fabadd9b640 1 -- 172.21.15.83:0/1457614250 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab98003530 con 0x7fabb015aa90 2024-09-19T07:19:48.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7fabaf59e640 1 -- 172.21.15.83:0/1457614250 >> v1:172.21.15.184:6789/0 conn(0x7fabb00c2980 legacy=0x7fabb00c33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7fabaf59e640 1 -- 172.21.15.83:0/1457614250 >> v1:172.21.15.83:6789/0 conn(0x7fabb015aa90 legacy=0x7fabb006cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7fabaf59e640 1 -- 172.21.15.83:0/1457614250 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb006c000 con 0x7fabb00aa3b0 2024-09-19T07:19:48.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f7f2f7fe640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f300aa630 0x7f7f30151880 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400c500 tx=0x7f7f2400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f7f17fff640 1 -- 172.21.15.83:0/1410335803 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24017070 con 0x7f7f300aa630 2024-09-19T07:19:48.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6b3a1640 1 -- start start 2024-09-19T07:19:48.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6b3a1640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3a640c3e90 0x7f3a6415a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6b3a1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a640c25e0 0x7f3a640c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6b3a1640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a64073c60 con 0x7f3a640716c0 2024-09-19T07:19:48.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6b3a1640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f3a6415ac70 con 0x7f3a640c3e90 2024-09-19T07:19:48.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6b3a1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3a64157ce0 con 0x7f3a640c25e0 2024-09-19T07:19:48.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f7f17fff640 1 -- 172.21.15.83:0/1410335803 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f24003600 con 0x7f7f300aa630 2024-09-19T07:19:48.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f7f17fff640 1 -- 172.21.15.83:0/1410335803 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f240039f0 con 0x7f7f300aa630 2024-09-19T07:19:48.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f3014ae10 con 0x7f7f300aa630 2024-09-19T07:19:48.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f3014b3d0 con 0x7f7f300aa630 2024-09-19T07:19:48.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6913d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3a640c3e90 0x7f3a6415a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6913d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3a640c3e90 0x7f3a6415a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:33174/0 (socket says 172.21.15.83:33174) 2024-09-19T07:19:48.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6913d640 1 -- 172.21.15.83:0/2625827367 learned_addr learned my addr 172.21.15.83:0/2625827367 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:48.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6993e640 1 --2- 172.21.15.83:0/2625827367 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a640c25e0 0x7f3a640c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f7f17fff640 1 -- 172.21.15.83:0/1410335803 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7f2401a070 con 0x7f7f300aa630 2024-09-19T07:19:48.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6913d640 1 -- 172.21.15.83:0/2625827367 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a640c25e0 msgr2=0x7f3a640c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:48.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6913d640 1 --2- 172.21.15.83:0/2625827367 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a640c25e0 0x7f3a640c2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6913d640 1 -- 172.21.15.83:0/2625827367 >> v1:172.21.15.184:6789/0 conn(0x7f3a640716c0 legacy=0x7f3a6415a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:48.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.915+0000 7f3a6913d640 1 -- 172.21.15.83:0/2625827367 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a64072650 con 0x7f3a640c3e90 2024-09-19T07:19:48.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3e81ccc640 1 Processor -- start 2024-09-19T07:19:48.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7fabaf59e640 1 --2- 172.21.15.83:0/1457614250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabb00aa3b0 0x7fabb0157290 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faba00097c0 tx=0x7faba001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=664f0cfa067f5f74 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f7f17fff640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f1c0678f0 0x7f7f1c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f7f17fff640 1 -- 172.21.15.83:0/1410335803 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f7f24085320 con 0x7f7f300aa630 2024-09-19T07:19:48.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f7f2effd640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f1c0678f0 0x7f7f1c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3e81ccc640 1 -- start start 2024-09-19T07:19:48.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7fabadd9b640 1 -- 172.21.15.83:0/1457614250 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba00b0070 con 0x7fabb00aa3b0 2024-09-19T07:19:48.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7fabadd9b640 1 -- 172.21.15.83:0/1457614250 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba00ab600 con 0x7fabb00aa3b0 2024-09-19T07:19:48.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7fabadd9b640 1 -- 172.21.15.83:0/1457614250 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba00ab9b0 con 0x7fabb00aa3b0 2024-09-19T07:19:48.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3a6913d640 1 --2- 172.21.15.83:0/2625827367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a640c3e90 0x7f3a6415a020 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c009e50 tx=0x7f3a5c01c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=258d38cc1ee378c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f100052a0 con 0x7f7f300aa630 2024-09-19T07:19:48.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3a5bfff640 1 -- 172.21.15.83:0/2625827367 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c0a5720 con 0x7f3a640c3e90 2024-09-19T07:19:48.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3a5bfff640 1 -- 172.21.15.83:0/2625827367 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a5c0a58c0 con 0x7f3a640c3e90 2024-09-19T07:19:48.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3a5bfff640 1 -- 172.21.15.83:0/2625827367 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c0a5c90 con 0x7f3a640c3e90 2024-09-19T07:19:48.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7fabb4983640 1 -- 172.21.15.83:0/1457614250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabb00aa3b0 msgr2=0x7fabb0157290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7fabb4983640 1 --2- 172.21.15.83:0/1457614250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabb00aa3b0 0x7fabb0157290 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faba00097c0 tx=0x7faba001c450 comp rx=0 tx=0).stop 2024-09-19T07:19:48.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2625827367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a640c3e90 msgr2=0x7f3a6415a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3a6b3a1640 1 --2- 172.21.15.83:0/2625827367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a640c3e90 0x7f3a6415a020 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c009e50 tx=0x7f3a5c01c8c0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f7f2effd640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f1c0678f0 0x7f7f1c069db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7f301f06c0 tx=0x7f7f20009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7fabb4983640 1 -- 172.21.15.83:0/1457614250 shutdown_connections 2024-09-19T07:19:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7fabb4983640 1 --2- 172.21.15.83:0/1457614250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabb00aa3b0 0x7fabb0157290 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7fabb4983640 1 -- 172.21.15.83:0/1457614250 >> 172.21.15.83:0/1457614250 conn(0x7fabb006dba0 msgr2=0x7fabb006dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7fabb4983640 1 -- 172.21.15.83:0/1457614250 shutdown_connections 2024-09-19T07:19:48.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2625827367 shutdown_connections 2024-09-19T07:19:48.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3a6b3a1640 1 --2- 172.21.15.83:0/2625827367 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a640c25e0 0x7f3a640c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3a6b3a1640 1 --2- 172.21.15.83:0/2625827367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a640c3e90 0x7f3a6415a020 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2625827367 >> 172.21.15.83:0/2625827367 conn(0x7f3a640a9870 msgr2=0x7f3a640a9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:48.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2625827367 shutdown_connections 2024-09-19T07:19:48.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2625827367 wait complete. 2024-09-19T07:19:48.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f6290d12640 1 Processor -- start 2024-09-19T07:19:48.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7fabb4983640 1 -- 172.21.15.83:0/1457614250 wait complete. 2024-09-19T07:19:48.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabb4983640 1 Processor -- start 2024-09-19T07:19:48.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6b3a1640 1 Processor -- start 2024-09-19T07:19:48.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f6290d12640 1 -- start start 2024-09-19T07:19:48.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabb4983640 1 -- start start 2024-09-19T07:19:48.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabb4983640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabb00aa3b0 0x7fabb00bcc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabb4983640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabb00c2980 0x7fabb00bd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabb4983640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabb015aa90 0x7fabb0155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabb4983640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fabb00c2ee0 con 0x7fabb00c2980 2024-09-19T07:19:48.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabb4983640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fabb00c2d60 con 0x7fabb015aa90 2024-09-19T07:19:48.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabb4983640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fabb0168440 con 0x7fabb00aa3b0 2024-09-19T07:19:48.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f6290d12640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f628c06e6e0 0x7f628c15d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabaed9d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabb00aa3b0 0x7fabb00bcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f628b7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f628c06cef0 0x7f628c0aa990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:42586/0 (socket says 172.21.15.83:42586) 2024-09-19T07:19:48.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f628b7fe640 1 -- 172.21.15.83:0/821120199 learned_addr learned my addr 172.21.15.83:0/821120199 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:48.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabaed9d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabb00aa3b0 0x7fabb00bcc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:37056/0 (socket says 172.21.15.83:37056) 2024-09-19T07:19:48.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabaed9d640 1 -- 172.21.15.83:0/3068426004 learned_addr learned my addr 172.21.15.83:0/3068426004 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:48.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabaf59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabb015aa90 0x7fabb0155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f6290d12640 1 -- 172.21.15.83:0/821120199 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f628c0c3260 con 0x7f628c1525d0 2024-09-19T07:19:48.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f6290d12640 1 -- 172.21.15.83:0/821120199 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f628c0c3540 con 0x7f628c06cef0 2024-09-19T07:19:48.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6b3a1640 1 -- start start 2024-09-19T07:19:48.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6b3a1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a640716c0 0x7f3a6415e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6b3a1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a640c25e0 0x7f3a64163020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f6290d12640 1 -- 172.21.15.83:0/821120199 --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f628c1552e0 con 0x7f628c06e6e0 2024-09-19T07:19:48.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6b3a1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a640c3e90 0x7f3a641553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6b3a1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3a640c4270 con 0x7f3a640c25e0 2024-09-19T07:19:48.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6b3a1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3a64072130 con 0x7f3a640716c0 2024-09-19T07:19:48.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6b3a1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3a640c43f0 con 0x7f3a640c3e90 2024-09-19T07:19:48.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6913d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a640716c0 0x7f3a6415e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6913d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a640716c0 0x7f3a6415e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:33188/0 (socket says 172.21.15.83:33188) 2024-09-19T07:19:48.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6913d640 1 -- 172.21.15.83:0/2345258849 learned_addr learned my addr 172.21.15.83:0/2345258849 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:48.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6993e640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a640c3e90 0x7f3a641553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f628bfff640 1 --2- 172.21.15.83:0/821120199 >> v2:172.21.15.83:3300/0 conn(0x7f628c06e6e0 0x7f628c15d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3e81ccc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e7c0c71f0 0x7f3e7c0c6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3e81ccc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3e7c06f330 0x7f3e7c14ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3e81ccc640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e7c159410 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3e81ccc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3e7c14d3b0 con 0x7f3e7c0c71f0 2024-09-19T07:19:48.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3e81ccc640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f3e7c0ac480 con 0x7f3e7c06f330 2024-09-19T07:19:48.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3e7bfff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f3e7c14cfd0 0x7f3e7c152c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:42578/0 (socket says 172.21.15.83:42578) 2024-09-19T07:19:48.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3e7affd640 1 --2- 172.21.15.83:0/3374246055 >> v2:172.21.15.184:3300/0 conn(0x7f3e7c06f330 0x7f3e7c14ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.919+0000 7f3e7bfff640 1 -- 172.21.15.83:0/3374246055 learned_addr learned my addr 172.21.15.83:0/3374246055 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:48.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3e7a7fc640 1 -- 172.21.15.83:0/3374246055 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 630721108 0 0) 0x7f3e7c159410 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3e7a7fc640 1 -- 172.21.15.83:0/3374246055 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e64003530 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3e7affd640 1 -- 172.21.15.83:0/3374246055 >> v2:172.21.15.83:3300/0 conn(0x7f3e7c0c71f0 msgr2=0x7f3e7c0c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:48.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3e7affd640 1 --2- 172.21.15.83:0/3374246055 >> v2:172.21.15.83:3300/0 conn(0x7f3e7c0c71f0 0x7f3e7c0c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3e7affd640 1 -- 172.21.15.83:0/3374246055 >> v1:172.21.15.184:6789/0 conn(0x7f3e7c14cfd0 legacy=0x7f3e7c152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3e7affd640 1 -- 172.21.15.83:0/3374246055 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7c0abb50 con 0x7f3e7c06f330 2024-09-19T07:19:48.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabaf59e640 1 -- 172.21.15.83:0/3068426004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabb00aa3b0 msgr2=0x7fabb00bcc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabaf59e640 1 --2- 172.21.15.83:0/3068426004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabb00aa3b0 0x7fabb00bcc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabaf59e640 1 -- 172.21.15.83:0/3068426004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabb00c2980 msgr2=0x7fabb00bd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabaf59e640 1 --2- 172.21.15.83:0/3068426004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabb00c2980 0x7fabb00bd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabaf59e640 1 -- 172.21.15.83:0/3068426004 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb00bd170 con 0x7fabb015aa90 2024-09-19T07:19:48.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6893c640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a640c25e0 0x7f3a64163020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3e7affd640 1 --2- 172.21.15.83:0/3374246055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e7c06f330 0x7f3e7c14ca90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0ac600 tx=0x7f3e7001c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5997ee4f24def3fb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7fabaf59e640 1 --2- 172.21.15.83:0/3068426004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabb015aa90 0x7fabb0155530 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabb00c8160 tx=0x7faba00191f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6893c640 1 -- 172.21.15.83:0/2345258849 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a640c3e90 msgr2=0x7f3a641553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6893c640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a640c3e90 0x7f3a641553e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6893c640 1 -- 172.21.15.83:0/2345258849 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a640716c0 msgr2=0x7f3a6415e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6893c640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a640716c0 0x7f3a6415e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6893c640 1 -- 172.21.15.83:0/2345258849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a6415da40 con 0x7f3a640c25e0 2024-09-19T07:19:48.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3e7a7fc640 1 -- 172.21.15.83:0/3374246055 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e700b0070 con 0x7f3e7c06f330 2024-09-19T07:19:48.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e7a7fc640 1 -- 172.21.15.83:0/3374246055 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e700ab600 con 0x7f3e7c06f330 2024-09-19T07:19:48.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e7a7fc640 1 -- 172.21.15.83:0/3374246055 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e700ab9f0 con 0x7f3e7c06f330 2024-09-19T07:19:48.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6913d640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a640716c0 0x7f3a6415e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:48.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f3a6993e640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a640c3e90 0x7f3a641553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:48.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3a6893c640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a640c25e0 0x7f3a64163020 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400c410 tx=0x7f3a5400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 -- 172.21.15.83:0/3374246055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e7c06f330 msgr2=0x7f3e7c14ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 --2- 172.21.15.83:0/3374246055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e7c06f330 0x7f3e7c14ca90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0ac600 tx=0x7f3e7001c3f0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 -- 172.21.15.83:0/3374246055 shutdown_connections 2024-09-19T07:19:48.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 --2- 172.21.15.83:0/3374246055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e7c06f330 0x7f3e7c14ca90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 --2- 172.21.15.83:0/3374246055 >> v2:172.21.15.83:3300/0 conn(0x7f3e7c0c71f0 0x7f3e7c0c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 -- 172.21.15.83:0/3374246055 >> 172.21.15.83:0/3374246055 conn(0x7f3e7c0c53f0 msgr2=0x7f3e7c0c5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:48.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 -- 172.21.15.83:0/3374246055 shutdown_connections 2024-09-19T07:19:48.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 -- 172.21.15.83:0/3374246055 wait complete. 2024-09-19T07:19:48.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 Processor -- start 2024-09-19T07:19:48.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 -- start start 2024-09-19T07:19:48.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f628bfff640 1 -- 172.21.15.83:0/821120199 >> v1:172.21.15.184:6789/0 conn(0x7f628c06cef0 legacy=0x7f628c0aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f628bfff640 1 -- 172.21.15.83:0/821120199 >> v1:172.21.15.83:6789/0 conn(0x7f628c1525d0 legacy=0x7f628c155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.923+0000 7f628bfff640 1 -- 172.21.15.83:0/821120199 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628c1555c0 con 0x7f628c06e6e0 2024-09-19T07:19:48.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f628bfff640 1 --2- 172.21.15.83:0/821120199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c06e6e0 0x7f628c15d4e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6284009d90 tx=0x7f6284020bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c22b7fb46637fa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e7c06f330 0x7f3e7c0731f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e7c0c71f0 0x7f3e7c148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7c14cfd0 0x7f3e7c1eafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c06f8e0 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c06f760 con 0x7f3e7c06f330 2024-09-19T07:19:48.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e81ccc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7c153770 con 0x7f3e7c0c71f0 2024-09-19T07:19:48.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e7bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7c14cfd0 0x7f3e7c1eafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e7bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7c14cfd0 0x7f3e7c1eafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40822/0 (socket says 172.21.15.83:40822) 2024-09-19T07:19:48.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e7bfff640 1 -- 172.21.15.83:0/4241538553 learned_addr learned my addr 172.21.15.83:0/4241538553 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:48.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e7b7fe640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e7c06f330 0x7f3e7c0731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.927+0000 7f3e7affd640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e7c0c71f0 0x7f3e7c148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7fab977fe640 1 -- 172.21.15.83:0/3068426004 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba00ba070 con 0x7fabb015aa90 2024-09-19T07:19:48.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7fab977fe640 1 -- 172.21.15.83:0/3068426004 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba00ab9b0 con 0x7fabb015aa90 2024-09-19T07:19:48.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7fab977fe640 1 -- 172.21.15.83:0/3068426004 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba00b0040 con 0x7fabb015aa90 2024-09-19T07:19:48.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7fabb4983640 1 -- 172.21.15.83:0/3068426004 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb0073010 con 0x7fabb015aa90 2024-09-19T07:19:48.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7fabb4983640 1 -- 172.21.15.83:0/3068426004 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb0070c00 con 0x7fabb015aa90 2024-09-19T07:19:48.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e7bfff640 1 -- 172.21.15.83:0/4241538553 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e7c0c71f0 msgr2=0x7f3e7c148780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e7bfff640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e7c0c71f0 0x7f3e7c148780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e7bfff640 1 -- 172.21.15.83:0/4241538553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e7c06f330 msgr2=0x7f3e7c0731f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e7bfff640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e7c06f330 0x7f3e7c0731f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e7bfff640 1 -- 172.21.15.83:0/4241538553 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7c071f00 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e7b7fe640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e7c06f330 0x7f3e7c0731f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:19:48.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e7bfff640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7c14cfd0 0x7f3e7c1eafa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400cb70 tx=0x7f3e7400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f7f17fff640 1 -- 172.21.15.83:0/1410335803 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7f24055590 con 0x7f7f300aa630 2024-09-19T07:19:48.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e63fff640 1 -- 172.21.15.83:0/4241538553 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74017070 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e81ccc640 1 -- 172.21.15.83:0/4241538553 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e7c0737d0 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e81ccc640 1 -- 172.21.15.83:0/4241538553 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e7c0c4410 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e63fff640 1 -- 172.21.15.83:0/4241538553 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e7400ad40 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e63fff640 1 -- 172.21.15.83:0/4241538553 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74021060 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7fab977fe640 1 -- 172.21.15.83:0/3068426004 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faba00bd070 con 0x7fabb015aa90 2024-09-19T07:19:48.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e81ccc640 1 -- 172.21.15.83:0/4241538553 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e5c0052a0 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.931+0000 7f3e63fff640 1 -- 172.21.15.83:0/4241538553 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3e7401a070 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.935+0000 7f3e63fff640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e680678c0 0x7f3e68069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.935+0000 7fab977fe640 1 --2- 172.21.15.83:0/3068426004 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab9c067910 0x7fab9c069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.935+0000 7fabb4983640 1 -- 172.21.15.83:0/3068426004 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabb0070e00 con 0x7fabb015aa90 2024-09-19T07:19:48.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.935+0000 7fab977fe640 1 -- 172.21.15.83:0/3068426004 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7faba011ef40 con 0x7fabb015aa90 2024-09-19T07:19:48.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.935+0000 7f3e63fff640 1 -- 172.21.15.83:0/4241538553 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f3e74085d50 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.935+0000 7f3e7b7fe640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e680678c0 0x7f3e68069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.935+0000 7fabaed9d640 1 --2- 172.21.15.83:0/3068426004 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab9c067910 0x7fab9c069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.935+0000 7fabaed9d640 1 --2- 172.21.15.83:0/3068426004 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab9c067910 0x7fab9c069dd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faba4007fd0 tx=0x7faba4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.935+0000 7f3e7b7fe640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e680678c0 0x7f3e68069d80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c006fd0 tx=0x7f3e6c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f3a59ffb640 1 -- 172.21.15.83:0/2345258849 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a54017070 con 0x7f3a640c25e0 2024-09-19T07:19:48.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2345258849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a6415ee20 con 0x7f3a640c25e0 2024-09-19T07:19:48.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2345258849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a6415cf60 con 0x7f3a640c25e0 2024-09-19T07:19:48.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f3a59ffb640 1 -- 172.21.15.83:0/2345258849 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a540069e0 con 0x7f3a640c25e0 2024-09-19T07:19:48.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f628a7fc640 1 -- 172.21.15.83:0/821120199 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62840b6070 con 0x7f628c06e6e0 2024-09-19T07:19:48.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f628a7fc640 1 -- 172.21.15.83:0/821120199 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62840a9a00 con 0x7f628c06e6e0 2024-09-19T07:19:48.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f628a7fc640 1 -- 172.21.15.83:0/821120199 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62840a9dd0 con 0x7f628c06e6e0 2024-09-19T07:19:48.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f3a59ffb640 1 -- 172.21.15.83:0/2345258849 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a54006e10 con 0x7f3a640c25e0 2024-09-19T07:19:48.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 -- 172.21.15.83:0/821120199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c06e6e0 msgr2=0x7f628c15d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 --2- 172.21.15.83:0/821120199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c06e6e0 0x7f628c15d4e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6284009d90 tx=0x7f6284020bb0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 -- 172.21.15.83:0/821120199 shutdown_connections 2024-09-19T07:19:48.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 --2- 172.21.15.83:0/821120199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c06e6e0 0x7f628c15d4e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f3e63fff640 1 -- 172.21.15.83:0/4241538553 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3e74055e00 con 0x7f3e7c14cfd0 2024-09-19T07:19:48.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 -- 172.21.15.83:0/821120199 >> 172.21.15.83:0/821120199 conn(0x7f628c06dbc0 msgr2=0x7f628c06dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:48.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 -- 172.21.15.83:0/821120199 shutdown_connections 2024-09-19T07:19:48.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 -- 172.21.15.83:0/821120199 wait complete. 2024-09-19T07:19:48.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 Processor -- start 2024-09-19T07:19:48.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 -- start start 2024-09-19T07:19:48.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f628c06cef0 0x7f628c14dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c06e6e0 0x7f628c071090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f628c1525d0 0x7f628c0715d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f628c06d450 con 0x7f628c06e6e0 2024-09-19T07:19:48.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f628c06d2d0 con 0x7f628c06cef0 2024-09-19T07:19:48.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.939+0000 7f6290d12640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f628c15fa30 con 0x7f628c1525d0 2024-09-19T07:19:48.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f628b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f628c06cef0 0x7f628c14dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f628bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f628c1525d0 0x7f628c0715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f628b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f628c06cef0 0x7f628c14dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:33206/0 (socket says 172.21.15.83:33206) 2024-09-19T07:19:48.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f628b7fe640 1 -- 172.21.15.83:0/1558636981 learned_addr learned my addr 172.21.15.83:0/1558636981 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:48.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f3a59ffb640 1 -- 172.21.15.83:0/2345258849 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3a5401a070 con 0x7f3a640c25e0 2024-09-19T07:19:48.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f628b7fe640 1 -- 172.21.15.83:0/1558636981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f628c1525d0 msgr2=0x7f628c0715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f628b7fe640 1 --2- 172.21.15.83:0/1558636981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f628c1525d0 0x7f628c0715d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f628b7fe640 1 -- 172.21.15.83:0/1558636981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c06e6e0 msgr2=0x7f628c071090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f628b7fe640 1 --2- 172.21.15.83:0/1558636981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c06e6e0 0x7f628c071090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f628b7fe640 1 -- 172.21.15.83:0/1558636981 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628c071b10 con 0x7f628c06cef0 2024-09-19T07:19:48.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f3a59ffb640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3a48067940 0x7f3a48069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f628b7fe640 1 --2- 172.21.15.83:0/1558636981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f628c06cef0 0x7f628c14dd40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f627c00c780 tx=0x7f627c00cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f3a6913d640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3a48067940 0x7f3a48069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f3a59ffb640 1 -- 172.21.15.83:0/2345258849 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f3a54053470 con 0x7f3a640c25e0 2024-09-19T07:19:48.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f3a6913d640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3a48067940 0x7f3a48069e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c009e50 tx=0x7f3a5c0a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.943+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2345258849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a440052a0 con 0x7f3a640c25e0 2024-09-19T07:19:48.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.947+0000 7f6273fff640 1 -- 172.21.15.83:0/1558636981 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c018070 con 0x7f628c06cef0 2024-09-19T07:19:48.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.947+0000 7f6273fff640 1 -- 172.21.15.83:0/1558636981 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f627c0072d0 con 0x7f628c06cef0 2024-09-19T07:19:48.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.947+0000 7f6290d12640 1 -- 172.21.15.83:0/1558636981 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f628c14d810 con 0x7f628c06cef0 2024-09-19T07:19:48.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.947+0000 7fab977fe640 1 -- 172.21.15.83:0/3068426004 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7faba00160d0 con 0x7fabb015aa90 2024-09-19T07:19:48.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.947+0000 7f6273fff640 1 -- 172.21.15.83:0/1558636981 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c0047f0 con 0x7f628c06cef0 2024-09-19T07:19:48.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.947+0000 7f6290d12640 1 -- 172.21.15.83:0/1558636981 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f628c14ca50 con 0x7f628c06cef0 2024-09-19T07:19:48.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.951+0000 7f6273fff640 1 -- 172.21.15.83:0/1558636981 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f627c01b070 con 0x7f628c06cef0 2024-09-19T07:19:48.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.951+0000 7f6273fff640 1 --2- 172.21.15.83:0/1558636981 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62780678f0 0x7f6278069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.951+0000 7f6273fff640 1 -- 172.21.15.83:0/1558636981 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f627c085390 con 0x7f628c06cef0 2024-09-19T07:19:48.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.951+0000 7f628affd640 1 --2- 172.21.15.83:0/1558636981 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62780678f0 0x7f6278069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.951+0000 7f628affd640 1 --2- 172.21.15.83:0/1558636981 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62780678f0 0x7f6278069db0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6280007fd0 tx=0x7f62800079a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.955+0000 7f3a59ffb640 1 -- 172.21.15.83:0/2345258849 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3a540553d0 con 0x7f3a640c25e0 2024-09-19T07:19:48.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.955+0000 7f6290d12640 1 -- 172.21.15.83:0/1558636981 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f628c0aa990 con 0x7f628c06cef0 2024-09-19T07:19:48.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.967+0000 7f6273fff640 1 -- 172.21.15.83:0/1558636981 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f627c055540 con 0x7f628c06cef0 2024-09-19T07:19:48.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.971+0000 7fa5f73d2640 1 Processor -- start 2024-09-19T07:19:48.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.971+0000 7fa5f73d2640 1 -- start start 2024-09-19T07:19:48.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.971+0000 7fa5f73d2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5e800acd0 0x7fa5e800b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.971+0000 7fa5f73d2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa5e8005b40 0x7fa5e8005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.971+0000 7fa5f73d2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5e80f7a80 con 0x7fa5e80f5040 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.971+0000 7fa5f73d2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa5e80f7d60 con 0x7fa5e8005b40 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.971+0000 7fa5f73d2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa5e80f8040 con 0x7fa5e800acd0 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.971+0000 7fa5f596f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa5e8005b40 0x7fa5e8005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.971+0000 7fa5f596f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa5e8005b40 0x7fa5e8005fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40846/0 (socket says 172.21.15.83:40846) 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fa5f596f640 1 -- 172.21.15.83:0/3520132618 learned_addr learned my addr 172.21.15.83:0/3520132618 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fa5f496d640 1 --2- 172.21.15.83:0/3520132618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5e800acd0 0x7fa5e800b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fa5f596f640 1 -- 172.21.15.83:0/3520132618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5e800acd0 msgr2=0x7fa5e800b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fa5f596f640 1 --2- 172.21.15.83:0/3520132618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5e800acd0 0x7fa5e800b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fa5f596f640 1 -- 172.21.15.83:0/3520132618 >> v1:172.21.15.184:6789/0 conn(0x7fa5e80f5040 legacy=0x7fa5e80491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fa5f596f640 1 -- 172.21.15.83:0/3520132618 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e80f8320 con 0x7fa5e8005b40 2024-09-19T07:19:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fa5f596f640 1 --2- 172.21.15.83:0/3520132618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e8005b40 0x7fa5e8005fd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa5f007bc70 tx=0x7fa5f0092cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872b63e446aae219 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fca71071640 1 Processor -- start 2024-09-19T07:19:48.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fca71071640 1 -- start start 2024-09-19T07:19:48.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fca71071640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fca6400aca0 0x7fca6400b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fca71071640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca640f7a50 con 0x7fca64005b10 2024-09-19T07:19:48.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fca71071640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca640f7d30 con 0x7fca640f5010 2024-09-19T07:19:48.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fca71071640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fca640f8010 con 0x7fca6400aca0 2024-09-19T07:19:48.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fca70870640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fca64005b10 0x7fca64005f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51624/0 (socket says 172.21.15.83:51624) 2024-09-19T07:19:48.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fca70870640 1 -- 172.21.15.83:0/2100205918 learned_addr learned my addr 172.21.15.83:0/2100205918 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:48.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.975+0000 7fca6b7fe640 1 --2- 172.21.15.83:0/2100205918 >> v2:172.21.15.83:3300/0 conn(0x7fca6400aca0 0x7fca6400b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fca6b7fe640 1 -- 172.21.15.83:0/2100205918 >> v1:172.21.15.184:6789/0 conn(0x7fca640f5010 legacy=0x7fca640491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fca6b7fe640 1 -- 172.21.15.83:0/2100205918 >> v1:172.21.15.83:6789/0 conn(0x7fca64005b10 legacy=0x7fca64005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fca6b7fe640 1 -- 172.21.15.83:0/2100205918 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca640f82f0 con 0x7fca6400aca0 2024-09-19T07:19:48.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5e7fff640 1 -- 172.21.15.83:0/3520132618 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f006eca0 con 0x7fa5e8005b40 2024-09-19T07:19:48.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5e7fff640 1 -- 172.21.15.83:0/3520132618 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f006ee40 con 0x7fa5e8005b40 2024-09-19T07:19:48.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5e7fff640 1 -- 172.21.15.83:0/3520132618 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f011ca30 con 0x7fa5e8005b40 2024-09-19T07:19:48.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fca6affd640 1 -- 172.21.15.83:0/2100205918 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3951169701 0 0) 0x7fca640f7a50 con 0x7fca64005b10 2024-09-19T07:19:48.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3520132618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e8005b40 msgr2=0x7fa5e8005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5f73d2640 1 --2- 172.21.15.83:0/3520132618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e8005b40 0x7fa5e8005fd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa5f007bc70 tx=0x7fa5f0092cc0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3520132618 shutdown_connections 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5f73d2640 1 --2- 172.21.15.83:0/3520132618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e8005b40 0x7fa5e8005fd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5f73d2640 1 --2- 172.21.15.83:0/3520132618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5e800acd0 0x7fa5e800b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3520132618 >> 172.21.15.83:0/3520132618 conn(0x7fa5e8047d50 msgr2=0x7fa5e8048160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3520132618 shutdown_connections 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3520132618 wait complete. 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5f73d2640 1 Processor -- start 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.979+0000 7fa5f73d2640 1 -- start start 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f73d2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e8005b40 0x7fa5e802f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f73d2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5e800acd0 0x7fa5e80f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f73d2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5e80f5040 0x7fa5e818d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f73d2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e802aac0 con 0x7fa5e8005b40 2024-09-19T07:19:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f73d2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e8006510 con 0x7fa5e80f5040 2024-09-19T07:19:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f73d2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa5e802f7e0 con 0x7fa5e800acd0 2024-09-19T07:19:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f516e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e8005b40 0x7fa5e802f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f516e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e8005b40 0x7fa5e802f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40866/0 (socket says 172.21.15.83:40866) 2024-09-19T07:19:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fca6b7fe640 1 --2- 172.21.15.83:0/2100205918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6400aca0 0x7fca6400b0d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fca60006d20 tx=0x7fca600206a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79eef79d87caf158 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fca6affd640 1 -- 172.21.15.83:0/2100205918 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca600a9590 con 0x7fca6400aca0 2024-09-19T07:19:48.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f516e640 1 -- 172.21.15.83:0/3610944962 learned_addr learned my addr 172.21.15.83:0/3610944962 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:48.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f516e640 1 -- 172.21.15.83:0/3610944962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5e800acd0 msgr2=0x7fa5e80f69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f516e640 1 --2- 172.21.15.83:0/3610944962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5e800acd0 0x7fa5e80f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f516e640 1 -- 172.21.15.83:0/3610944962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5e80f5040 msgr2=0x7fa5e818d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f516e640 1 --2- 172.21.15.83:0/3610944962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5e80f5040 0x7fa5e818d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f516e640 1 -- 172.21.15.83:0/3610944962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e8030850 con 0x7fa5e8005b40 2024-09-19T07:19:48.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f516e640 1 --2- 172.21.15.83:0/3610944962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e8005b40 0x7fa5e802f2a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000c3c0 tx=0x7fa5e000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fca6affd640 1 -- 172.21.15.83:0/2100205918 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca600a9730 con 0x7fca6400aca0 2024-09-19T07:19:48.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fca6affd640 1 -- 172.21.15.83:0/2100205918 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca600a9b40 con 0x7fca6400aca0 2024-09-19T07:19:48.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fca71071640 1 -- 172.21.15.83:0/2100205918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6400aca0 msgr2=0x7fca6400b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:48.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5e5ffb640 1 -- 172.21.15.83:0/3610944962 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0017070 con 0x7fa5e8005b40 2024-09-19T07:19:48.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fca71071640 1 --2- 172.21.15.83:0/2100205918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6400aca0 0x7fca6400b0d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fca60006d20 tx=0x7fca600206a0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fca71071640 1 -- 172.21.15.83:0/2100205918 shutdown_connections 2024-09-19T07:19:48.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fca71071640 1 --2- 172.21.15.83:0/2100205918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6400aca0 0x7fca6400b0d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fca71071640 1 -- 172.21.15.83:0/2100205918 >> 172.21.15.83:0/2100205918 conn(0x7fca64047d20 msgr2=0x7fca64048130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:48.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fca71071640 1 -- 172.21.15.83:0/2100205918 shutdown_connections 2024-09-19T07:19:48.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fca71071640 1 -- 172.21.15.83:0/2100205918 wait complete. 2024-09-19T07:19:48.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5e5ffb640 1 -- 172.21.15.83:0/3610944962 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5e00052d0 con 0x7fa5e8005b40 2024-09-19T07:19:48.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3610944962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5e80e75b0 con 0x7fa5e8005b40 2024-09-19T07:19:48.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3610944962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5e80f2e40 con 0x7fa5e8005b40 2024-09-19T07:19:48.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.983+0000 7fa5e5ffb640 1 -- 172.21.15.83:0/3610944962 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0003840 con 0x7fa5e8005b40 2024-09-19T07:19:48.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca71071640 1 Processor -- start 2024-09-19T07:19:48.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca71071640 1 -- start start 2024-09-19T07:19:48.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca71071640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca64005b10 0x7fca64044b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca71071640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6400aca0 0x7fca640efd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca71071640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca640f5010 0x7fca640f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca71071640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fca6402aa90 con 0x7fca6400aca0 2024-09-19T07:19:48.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca71071640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fca64006640 con 0x7fca640f5010 2024-09-19T07:19:48.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca71071640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fca640ef8f0 con 0x7fca64005b10 2024-09-19T07:19:48.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca70870640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca640f5010 0x7fca640f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca70870640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca640f5010 0x7fca640f1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:33226/0 (socket says 172.21.15.83:33226) 2024-09-19T07:19:48.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca70870640 1 -- 172.21.15.83:0/26254506 learned_addr learned my addr 172.21.15.83:0/26254506 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:48.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca70870640 1 -- 172.21.15.83:0/26254506 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca64005b10 msgr2=0x7fca64044b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:19:48.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fa5e5ffb640 1 -- 172.21.15.83:0/3610944962 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa5e001a070 con 0x7fa5e8005b40 2024-09-19T07:19:48.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca70870640 1 --2- 172.21.15.83:0/26254506 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca64005b10 0x7fca64044b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca70870640 1 -- 172.21.15.83:0/26254506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6400aca0 msgr2=0x7fca640efd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:19:48.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca70870640 1 --2- 172.21.15.83:0/26254506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6400aca0 0x7fca640efd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:48.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca70870640 1 -- 172.21.15.83:0/26254506 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca640f03f0 con 0x7fca640f5010 2024-09-19T07:19:48.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca70870640 1 --2- 172.21.15.83:0/26254506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca640f5010 0x7fca640f1d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0512d0 tx=0x7fca6c069900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fa5e5ffb640 1 --2- 172.21.15.83:0/3610944962 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa5dc067a50 0x7fa5dc069f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca68ff9640 1 -- 172.21.15.83:0/26254506 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c080070 con 0x7fca640f5010 2024-09-19T07:19:48.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fca68ff9640 1 -- 172.21.15.83:0/26254506 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca6c071630 con 0x7fca640f5010 2024-09-19T07:19:48.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fa5e5ffb640 1 -- 172.21.15.83:0/3610944962 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fa5e00851a0 con 0x7fa5e8005b40 2024-09-19T07:19:48.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fa5f496d640 1 --2- 172.21.15.83:0/3610944962 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa5dc067a50 0x7fa5dc069f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.987+0000 7fa5f496d640 1 --2- 172.21.15.83:0/3610944962 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa5dc067a50 0x7fa5dc069f10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa5e818e420 tx=0x7fa5ec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.991+0000 7fca68ff9640 1 -- 172.21.15.83:0/26254506 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0717e0 con 0x7fca640f5010 2024-09-19T07:19:48.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.991+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3610944962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d40052a0 con 0x7fa5e8005b40 2024-09-19T07:19:48.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.991+0000 7fca71071640 1 -- 172.21.15.83:0/26254506 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca640f06d0 con 0x7fca640f5010 2024-09-19T07:19:48.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.991+0000 7fca71071640 1 -- 172.21.15.83:0/26254506 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca640ef580 con 0x7fca640f5010 2024-09-19T07:19:48.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.991+0000 7fca68ff9640 1 -- 172.21.15.83:0/26254506 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fca6c083070 con 0x7fca640f5010 2024-09-19T07:19:48.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.991+0000 7fca68ff9640 1 --2- 172.21.15.83:0/26254506 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fca5c0679c0 0x7fca5c069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:48.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.991+0000 7fca6bfff640 1 --2- 172.21.15.83:0/26254506 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fca5c0679c0 0x7fca5c069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:48.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.991+0000 7fca68ff9640 1 -- 172.21.15.83:0/26254506 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fca6c0f0e80 con 0x7fca640f5010 2024-09-19T07:19:48.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.995+0000 7fca6bfff640 1 --2- 172.21.15.83:0/26254506 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fca5c0679c0 0x7fca5c069e80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fca640eee00 tx=0x7fca58001bb0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:48.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.999+0000 7fca71071640 1 -- 172.21.15.83:0/26254506 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca6400b0d0 con 0x7fca640f5010 2024-09-19T07:19:48.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:48.999+0000 7fa5e5ffb640 1 -- 172.21.15.83:0/3610944962 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa5e0005480 con 0x7fa5e8005b40 2024-09-19T07:19:49.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.011+0000 7fca68ff9640 1 -- 172.21.15.83:0/26254506 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fca6c0c1680 con 0x7fca640f5010 2024-09-19T07:19:49.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.183+0000 7f93c9485640 1 -- 172.21.15.83:0/2556133283 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f93a4005590 con 0x7f93c40c2ea0 2024-09-19T07:19:49.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.183+0000 7f93ab7fe640 1 -- 172.21.15.83:0/2556133283 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f93b40f0ae0 con 0x7f93c40c2ea0 2024-09-19T07:19:49.185 INFO:teuthology.orchestra.run.smithi083.stdout:60129542151 2024-09-19T07:19:49.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.191+0000 7f93a97fa640 1 -- 172.21.15.83:0/2556133283 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f93b00679c0 msgr2=0x7f93b0069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.191+0000 7f93a97fa640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f93b00679c0 0x7f93b0069e80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f93c40abc70 tx=0x7f93bc001820 comp rx=0 tx=0).stop 2024-09-19T07:19:49.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.191+0000 7f93a97fa640 1 -- 172.21.15.83:0/2556133283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c40c2ea0 msgr2=0x7f93c414ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.191+0000 7f93a97fa640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c40c2ea0 0x7f93c414ee70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f93b401b2b0 tx=0x7f93b401a420 comp rx=0 tx=0).stop 2024-09-19T07:19:49.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.191+0000 7f93a97fa640 1 -- 172.21.15.83:0/2556133283 shutdown_connections 2024-09-19T07:19:49.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.191+0000 7f93a97fa640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f93b00679c0 0x7f93b0069e80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.191+0000 7f93a97fa640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93c41f07b0 0x7f93c40ab0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.191+0000 7f93a97fa640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c4155b00 0x7f93c4141070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.191+0000 7f93a97fa640 1 --2- 172.21.15.83:0/2556133283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c40c2ea0 0x7f93c414ee70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.191+0000 7f93a97fa640 1 -- 172.21.15.83:0/2556133283 >> 172.21.15.83:0/2556133283 conn(0x7f93c40c7370 msgr2=0x7f93c406da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:49.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.191+0000 7f93a97fa640 1 -- 172.21.15.83:0/2556133283 shutdown_connections 2024-09-19T07:19:49.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.191+0000 7f93a97fa640 1 -- 172.21.15.83:0/2556133283 wait complete. 2024-09-19T07:19:49.212 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.3 2024-09-19T07:19:49.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.255+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f7f10005590 con 0x7f7f300aa630 2024-09-19T07:19:49.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.255+0000 7f7f17fff640 1 -- 172.21.15.83:0/1410335803 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7f24059890 con 0x7f7f300aa630 2024-09-19T07:19:49.256 INFO:teuthology.orchestra.run.smithi083.stdout:60129542153 2024-09-19T07:19:49.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.255+0000 7f3e81ccc640 1 -- 172.21.15.83:0/4241538553 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f3e5c005590 con 0x7f3e7c14cfd0 2024-09-19T07:19:49.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.255+0000 7f3e63fff640 1 -- 172.21.15.83:0/4241538553 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3e7405a100 con 0x7f3e7c14cfd0 2024-09-19T07:19:49.257 INFO:teuthology.orchestra.run.smithi083.stdout:60129542152 2024-09-19T07:19:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f1c0678f0 msgr2=0x7f7f1c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f7f36247640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f1c0678f0 0x7f7f1c069db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7f301f06c0 tx=0x7f7f20009190 comp rx=0 tx=0).stop 2024-09-19T07:19:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f300aa630 msgr2=0x7f7f30151880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f7f36247640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f300aa630 0x7f7f30151880 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400c500 tx=0x7f7f2400c9c0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 shutdown_connections 2024-09-19T07:19:49.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f7f36247640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f1c0678f0 0x7f7f1c069db0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f7f36247640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f300c4150 0x7f7f30150390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f7f36247640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f300c2ff0 0x7f7f301f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f7f36247640 1 --2- 172.21.15.83:0/1410335803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f300aa630 0x7f7f30151880 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 >> 172.21.15.83:0/1410335803 conn(0x7f7f3006bc80 msgr2=0x7f7f30159760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:49.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 shutdown_connections 2024-09-19T07:19:49.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f7f36247640 1 -- 172.21.15.83:0/1410335803 wait complete. 2024-09-19T07:19:49.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f3e81ccc640 1 -- 172.21.15.83:0/4241538553 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e680678c0 msgr2=0x7f3e68069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f3e81ccc640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e680678c0 0x7f3e68069d80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c006fd0 tx=0x7f3e6c008040 comp rx=0 tx=0).stop 2024-09-19T07:19:49.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f3e81ccc640 1 -- 172.21.15.83:0/4241538553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7c14cfd0 msgr2=0x7f3e7c1eafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f3e81ccc640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7c14cfd0 0x7f3e7c1eafa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400cb70 tx=0x7f3e7400a430 comp rx=0 tx=0).stop 2024-09-19T07:19:49.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f3e81ccc640 1 -- 172.21.15.83:0/4241538553 shutdown_connections 2024-09-19T07:19:49.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f3e81ccc640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e680678c0 0x7f3e68069d80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f3e81ccc640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7c14cfd0 0x7f3e7c1eafa0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f3e81ccc640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e7c0c71f0 0x7f3e7c148780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f3e81ccc640 1 --2- 172.21.15.83:0/4241538553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e7c06f330 0x7f3e7c0731f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f3e81ccc640 1 -- 172.21.15.83:0/4241538553 >> 172.21.15.83:0/4241538553 conn(0x7f3e7c0c53f0 msgr2=0x7f3e7c1509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:49.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f3e81ccc640 1 -- 172.21.15.83:0/4241538553 shutdown_connections 2024-09-19T07:19:49.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.259+0000 7f3e81ccc640 1 -- 172.21.15.83:0/4241538553 wait complete. 2024-09-19T07:19:49.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.267+0000 7fabb4983640 1 -- 172.21.15.83:0/3068426004 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fabb0073530 con 0x7fabb015aa90 2024-09-19T07:19:49.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.267+0000 7fab977fe640 1 -- 172.21.15.83:0/3068426004 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faba00ef160 con 0x7fabb015aa90 2024-09-19T07:19:49.267 INFO:teuthology.orchestra.run.smithi083.stdout:60129542153 2024-09-19T07:19:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.267+0000 7fabb4983640 1 -- 172.21.15.83:0/3068426004 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab9c067910 msgr2=0x7fab9c069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.267+0000 7fabb4983640 1 --2- 172.21.15.83:0/3068426004 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab9c067910 0x7fab9c069dd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faba4007fd0 tx=0x7faba4009040 comp rx=0 tx=0).stop 2024-09-19T07:19:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.267+0000 7fabb4983640 1 -- 172.21.15.83:0/3068426004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabb015aa90 msgr2=0x7fabb0155530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.267+0000 7fabb4983640 1 --2- 172.21.15.83:0/3068426004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabb015aa90 0x7fabb0155530 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabb00c8160 tx=0x7faba00191f0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.271+0000 7fabb4983640 1 -- 172.21.15.83:0/3068426004 shutdown_connections 2024-09-19T07:19:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.271+0000 7fabb4983640 1 --2- 172.21.15.83:0/3068426004 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab9c067910 0x7fab9c069dd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.271+0000 7fabb4983640 1 --2- 172.21.15.83:0/3068426004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabb015aa90 0x7fabb0155530 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.271+0000 7fabb4983640 1 --2- 172.21.15.83:0/3068426004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabb00c2980 0x7fabb00bd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.271+0000 7fabb4983640 1 --2- 172.21.15.83:0/3068426004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabb00aa3b0 0x7fabb00bcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.271+0000 7fabb4983640 1 -- 172.21.15.83:0/3068426004 >> 172.21.15.83:0/3068426004 conn(0x7fabb006dba0 msgr2=0x7fabb00c3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.271+0000 7fabb4983640 1 -- 172.21.15.83:0/3068426004 shutdown_connections 2024-09-19T07:19:49.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.271+0000 7fabb4983640 1 -- 172.21.15.83:0/3068426004 wait complete. 2024-09-19T07:19:49.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.283+0000 7fca71071640 1 -- 172.21.15.83:0/26254506 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fca64007000 con 0x7fca640f5010 2024-09-19T07:19:49.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.283+0000 7fca68ff9640 1 -- 172.21.15.83:0/26254506 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fca6c0c0f80 con 0x7fca640f5010 2024-09-19T07:19:49.285 INFO:teuthology.orchestra.run.smithi083.stdout:60129542153 2024-09-19T07:19:49.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.4 2024-09-19T07:19:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7fca71071640 1 -- 172.21.15.83:0/26254506 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fca5c0679c0 msgr2=0x7fca5c069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7fca71071640 1 --2- 172.21.15.83:0/26254506 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fca5c0679c0 0x7fca5c069e80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fca640eee00 tx=0x7fca58001bb0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7fca71071640 1 -- 172.21.15.83:0/26254506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca640f5010 msgr2=0x7fca640f1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7fca71071640 1 --2- 172.21.15.83:0/26254506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca640f5010 0x7fca640f1d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0512d0 tx=0x7fca6c069900 comp rx=0 tx=0).stop 2024-09-19T07:19:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7fca71071640 1 -- 172.21.15.83:0/26254506 shutdown_connections 2024-09-19T07:19:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7fca71071640 1 --2- 172.21.15.83:0/26254506 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fca5c0679c0 0x7fca5c069e80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7fca71071640 1 --2- 172.21.15.83:0/26254506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca640f5010 0x7fca640f1d90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7fca71071640 1 --2- 172.21.15.83:0/26254506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6400aca0 0x7fca640efd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7fca71071640 1 --2- 172.21.15.83:0/26254506 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca64005b10 0x7fca64044b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7fca71071640 1 -- 172.21.15.83:0/26254506 >> 172.21.15.83:0/26254506 conn(0x7fca64047d20 msgr2=0x7fca64009c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:49.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7fca71071640 1 -- 172.21.15.83:0/26254506 shutdown_connections 2024-09-19T07:19:49.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7fca71071640 1 -- 172.21.15.83:0/26254506 wait complete. 2024-09-19T07:19:49.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.287+0000 7f6290d12640 1 -- 172.21.15.83:0/1558636981 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f628c151520 con 0x7f628c06cef0 2024-09-19T07:19:49.290 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.6 2024-09-19T07:19:49.291 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.7 2024-09-19T07:19:49.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6273fff640 1 -- 172.21.15.83:0/1558636981 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f627c059840 con 0x7f628c06cef0 2024-09-19T07:19:49.291 INFO:teuthology.orchestra.run.smithi083.stdout:60129542151 2024-09-19T07:19:49.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6290d12640 1 -- 172.21.15.83:0/1558636981 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62780678f0 msgr2=0x7f6278069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6290d12640 1 --2- 172.21.15.83:0/1558636981 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62780678f0 0x7f6278069db0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6280007fd0 tx=0x7f62800079a0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6290d12640 1 -- 172.21.15.83:0/1558636981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f628c06cef0 msgr2=0x7f628c14dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6290d12640 1 --2- 172.21.15.83:0/1558636981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f628c06cef0 0x7f628c14dd40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f627c00c780 tx=0x7f627c00cc40 comp rx=0 tx=0).stop 2024-09-19T07:19:49.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6290d12640 1 -- 172.21.15.83:0/1558636981 shutdown_connections 2024-09-19T07:19:49.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6290d12640 1 --2- 172.21.15.83:0/1558636981 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62780678f0 0x7f6278069db0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6290d12640 1 --2- 172.21.15.83:0/1558636981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f628c1525d0 0x7f628c0715d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6290d12640 1 --2- 172.21.15.83:0/1558636981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f628c06e6e0 0x7f628c071090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6290d12640 1 --2- 172.21.15.83:0/1558636981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f628c06cef0 0x7f628c14dd40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6290d12640 1 -- 172.21.15.83:0/1558636981 >> 172.21.15.83:0/1558636981 conn(0x7f628c06dbc0 msgr2=0x7f628c06dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6290d12640 1 -- 172.21.15.83:0/1558636981 shutdown_connections 2024-09-19T07:19:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2345258849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f3a44005860 con 0x7f3a640c25e0 2024-09-19T07:19:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f6290d12640 1 -- 172.21.15.83:0/1558636981 wait complete. 2024-09-19T07:19:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.291+0000 7f3a59ffb640 1 -- 172.21.15.83:0/2345258849 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3a540596d0 con 0x7f3a640c25e0 2024-09-19T07:19:49.294 INFO:teuthology.orchestra.run.smithi083.stdout:60129542152 2024-09-19T07:19:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.295+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2345258849 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3a48067940 msgr2=0x7f3a48069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.295+0000 7f3a6b3a1640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3a48067940 0x7f3a48069e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c009e50 tx=0x7f3a5c0a7070 comp rx=0 tx=0).stop 2024-09-19T07:19:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.295+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2345258849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a640c25e0 msgr2=0x7f3a64163020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.295+0000 7f3a6b3a1640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a640c25e0 0x7f3a64163020 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400c410 tx=0x7f3a5400c8d0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.295+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2345258849 shutdown_connections 2024-09-19T07:19:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.295+0000 7f3a6b3a1640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3a48067940 0x7f3a48069e00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.295+0000 7f3a6b3a1640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a640c3e90 0x7f3a641553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.295+0000 7f3a6b3a1640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a640c25e0 0x7f3a64163020 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.295+0000 7f3a6b3a1640 1 --2- 172.21.15.83:0/2345258849 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a640716c0 0x7f3a6415e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.295+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2345258849 >> 172.21.15.83:0/2345258849 conn(0x7f3a640a9870 msgr2=0x7f3a6406e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:49.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.295+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2345258849 shutdown_connections 2024-09-19T07:19:49.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.295+0000 7f3a6b3a1640 1 -- 172.21.15.83:0/2345258849 wait complete. 2024-09-19T07:19:49.307 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.2 2024-09-19T07:19:49.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.5 2024-09-19T07:19:49.321 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.1 2024-09-19T07:19:49.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.319+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3610944962 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa5d4005860 con 0x7fa5e8005b40 2024-09-19T07:19:49.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.319+0000 7fa5e5ffb640 1 -- 172.21.15.83:0/3610944962 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa5e0055b20 con 0x7fa5e8005b40 2024-09-19T07:19:49.322 INFO:teuthology.orchestra.run.smithi083.stdout:60129542151 2024-09-19T07:19:49.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.323+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3610944962 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa5dc067a50 msgr2=0x7fa5dc069f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.323+0000 7fa5f73d2640 1 --2- 172.21.15.83:0/3610944962 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa5dc067a50 0x7fa5dc069f10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa5e818e420 tx=0x7fa5ec008040 comp rx=0 tx=0).stop 2024-09-19T07:19:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.323+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3610944962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e8005b40 msgr2=0x7fa5e802f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.323+0000 7fa5f73d2640 1 --2- 172.21.15.83:0/3610944962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e8005b40 0x7fa5e802f2a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000c3c0 tx=0x7fa5e000c880 comp rx=0 tx=0).stop 2024-09-19T07:19:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.323+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3610944962 shutdown_connections 2024-09-19T07:19:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.323+0000 7fa5f73d2640 1 --2- 172.21.15.83:0/3610944962 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa5dc067a50 0x7fa5dc069f10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.323+0000 7fa5f73d2640 1 --2- 172.21.15.83:0/3610944962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5e80f5040 0x7fa5e818d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.323+0000 7fa5f73d2640 1 --2- 172.21.15.83:0/3610944962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5e800acd0 0x7fa5e80f69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.323+0000 7fa5f73d2640 1 --2- 172.21.15.83:0/3610944962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e8005b40 0x7fa5e802f2a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.323+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3610944962 >> 172.21.15.83:0/3610944962 conn(0x7fa5e8047d50 msgr2=0x7fa5e8009cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.323+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3610944962 shutdown_connections 2024-09-19T07:19:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:49.323+0000 7fa5f73d2640 1 -- 172.21.15.83:0/3610944962 wait complete. 2024-09-19T07:19:49.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.0 2024-09-19T07:19:50.213 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-19T07:19:50.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-19T07:19:50.291 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-19T07:19:50.291 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-19T07:19:50.308 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-19T07:19:50.313 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-19T07:19:50.322 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-19T07:19:50.338 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-19T07:19:50.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.339+0000 7f9e63b17640 1 Processor -- start 2024-09-19T07:19:50.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.339+0000 7f9e63b17640 1 -- start start 2024-09-19T07:19:50.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e63b17640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e5c154460 0x7f9e5c15ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e63b17640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e5c15e460 0x7f9e5c0ab020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e63b17640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e5c0c2d50 con 0x7f9e5c0c2850 2024-09-19T07:19:50.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e63b17640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f9e5c15d980 con 0x7f9e5c15e460 2024-09-19T07:19:50.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e63b17640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9e5c15dc60 con 0x7f9e5c154460 2024-09-19T07:19:50.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e618b3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e5c154460 0x7f9e5c15ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e610b2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e5c15e460 0x7f9e5c0ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e618b3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e5c154460 0x7f9e5c15ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:56102/0 (socket says 172.21.15.83:56102) 2024-09-19T07:19:50.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e618b3640 1 -- 172.21.15.83:0/3565781776 learned_addr learned my addr 172.21.15.83:0/3565781776 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:50.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e618b3640 1 -- 172.21.15.83:0/3565781776 >> v2:172.21.15.83:3300/0 conn(0x7f9e5c15e460 msgr2=0x7f9e5c0ab020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:50.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e618b3640 1 --2- 172.21.15.83:0/3565781776 >> v2:172.21.15.83:3300/0 conn(0x7f9e5c15e460 0x7f9e5c0ab020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e618b3640 1 -- 172.21.15.83:0/3565781776 >> v1:172.21.15.184:6789/0 conn(0x7f9e5c0c2850 legacy=0x7f9e5c0c3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e618b3640 1 -- 172.21.15.83:0/3565781776 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5c15df40 con 0x7f9e5c154460 2024-09-19T07:19:50.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.343+0000 7f9e610b2640 1 --2- 172.21.15.83:0/3565781776 >> v2:172.21.15.83:3300/0 conn(0x7f9e5c15e460 0x7f9e5c0ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:19:50.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.347+0000 7f9e618b3640 1 --2- 172.21.15.83:0/3565781776 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e5c154460 0x7f9e5c15ad80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009e50 tx=0x7f9e5401ca70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a4861062292bbfe server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.347+0000 7f9e608b1640 1 -- 172.21.15.83:0/3565781776 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e540a5810 con 0x7f9e5c154460 2024-09-19T07:19:50.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.347+0000 7f9e608b1640 1 -- 172.21.15.83:0/3565781776 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e540a59b0 con 0x7f9e5c154460 2024-09-19T07:19:50.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.347+0000 7f9e608b1640 1 -- 172.21.15.83:0/3565781776 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e540a5d80 con 0x7f9e5c154460 2024-09-19T07:19:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.347+0000 7f9e63b17640 1 -- 172.21.15.83:0/3565781776 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e5c154460 msgr2=0x7f9e5c15ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.347+0000 7f9e63b17640 1 --2- 172.21.15.83:0/3565781776 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e5c154460 0x7f9e5c15ad80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009e50 tx=0x7f9e5401ca70 comp rx=0 tx=0).stop 2024-09-19T07:19:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.347+0000 7f9e63b17640 1 -- 172.21.15.83:0/3565781776 shutdown_connections 2024-09-19T07:19:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.347+0000 7f9e63b17640 1 --2- 172.21.15.83:0/3565781776 >> v2:172.21.15.83:3300/0 conn(0x7f9e5c15e460 0x7f9e5c0ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.347+0000 7f9e63b17640 1 --2- 172.21.15.83:0/3565781776 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e5c154460 0x7f9e5c15ad80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.347+0000 7f9e63b17640 1 -- 172.21.15.83:0/3565781776 >> 172.21.15.83:0/3565781776 conn(0x7f9e5c075590 msgr2=0x7f9e5c06bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.347+0000 7f9e63b17640 1 -- 172.21.15.83:0/3565781776 shutdown_connections 2024-09-19T07:19:50.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.347+0000 7f9e63b17640 1 -- 172.21.15.83:0/3565781776 wait complete. 2024-09-19T07:19:50.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e63b17640 1 Processor -- start 2024-09-19T07:19:50.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e63b17640 1 -- start start 2024-09-19T07:19:50.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e63b17640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e5c0c2850 0x7f9e5c1482f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e63b17640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e5c154460 0x7f9e5c155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e63b17640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e5c15e460 0x7f9e5c0c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e63b17640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c165470 con 0x7f9e5c15e460 2024-09-19T07:19:50.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e63b17640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c1652f0 con 0x7f9e5c154460 2024-09-19T07:19:50.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e63b17640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9e5c1655f0 con 0x7f9e5c0c2850 2024-09-19T07:19:50.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e620b4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e5c15e460 0x7f9e5c0c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e620b4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e5c15e460 0x7f9e5c0c3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54056/0 (socket says 172.21.15.83:54056) 2024-09-19T07:19:50.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e620b4640 1 -- 172.21.15.83:0/3001275838 learned_addr learned my addr 172.21.15.83:0/3001275838 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:50.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e618b3640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e5c0c2850 0x7f9e5c1482f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e620b4640 1 -- 172.21.15.83:0/3001275838 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e5c0c2850 msgr2=0x7f9e5c1482f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e620b4640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e5c0c2850 0x7f9e5c1482f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e620b4640 1 -- 172.21.15.83:0/3001275838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e5c154460 msgr2=0x7f9e5c155260 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:19:50.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e620b4640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e5c154460 0x7f9e5c155260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.351+0000 7f9e620b4640 1 -- 172.21.15.83:0/3001275838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5c156b30 con 0x7f9e5c15e460 2024-09-19T07:19:50.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.355+0000 7f9e620b4640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e5c15e460 0x7f9e5c0c3a20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800c390 tx=0x7f9e5800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.355+0000 7f9e527fc640 1 -- 172.21.15.83:0/3001275838 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e580052d0 con 0x7f9e5c15e460 2024-09-19T07:19:50.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.355+0000 7f9e63b17640 1 -- 172.21.15.83:0/3001275838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e5c14f990 con 0x7f9e5c15e460 2024-09-19T07:19:50.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.355+0000 7f9e63b17640 1 -- 172.21.15.83:0/3001275838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e5c1575a0 con 0x7f9e5c15e460 2024-09-19T07:19:50.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.355+0000 7f9e527fc640 1 -- 172.21.15.83:0/3001275838 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e58005470 con 0x7f9e5c15e460 2024-09-19T07:19:50.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.355+0000 7f9e527fc640 1 -- 172.21.15.83:0/3001275838 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58007220 con 0x7f9e5c15e460 2024-09-19T07:19:50.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.355+0000 7f9e527fc640 1 -- 172.21.15.83:0/3001275838 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9e580073c0 con 0x7f9e5c15e460 2024-09-19T07:19:50.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.355+0000 7f9e527fc640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e400678c0 0x7f9e40069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.355+0000 7f9e527fc640 1 -- 172.21.15.83:0/3001275838 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f9e580853f0 con 0x7f9e5c15e460 2024-09-19T07:19:50.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.359+0000 7f9e618b3640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e400678c0 0x7f9e40069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.359+0000 7f9e618b3640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e400678c0 0x7f9e40069d80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9e540ae000 tx=0x7f9e540a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.367+0000 7f9e63b17640 1 -- 172.21.15.83:0/3001275838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e440052a0 con 0x7f9e5c15e460 2024-09-19T07:19:50.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.375+0000 7f9e527fc640 1 -- 172.21.15.83:0/3001275838 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9e58055520 con 0x7f9e5c15e460 2024-09-19T07:19:50.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.459+0000 7fb30f31f640 1 Processor -- start 2024-09-19T07:19:50.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.459+0000 7fb30f31f640 1 -- start start 2024-09-19T07:19:50.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.459+0000 7fb30f31f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb308148570 0x7fb3080aa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.459+0000 7fb30f31f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3080aaf20 con 0x7fb308155b00 2024-09-19T07:19:50.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.459+0000 7fb30f31f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3080c3440 con 0x7fb3080c2ea0 2024-09-19T07:19:50.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.459+0000 7fb30f31f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb308155670 con 0x7fb308148570 2024-09-19T07:19:50.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.459+0000 7fb30d8bc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb308155b00 0x7fb30806da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35480/0 (socket says 172.21.15.83:35480) 2024-09-19T07:19:50.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.459+0000 7fb30d8bc640 1 -- 172.21.15.83:0/4173858037 learned_addr learned my addr 172.21.15.83:0/4173858037 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:50.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.459+0000 7fb30d0bb640 1 --2- 172.21.15.83:0/4173858037 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb308148570 0x7fb3080aa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.459+0000 7fb30d0bb640 1 -- 172.21.15.83:0/4173858037 >> v1:172.21.15.184:6789/0 conn(0x7fb3080c2ea0 legacy=0x7fb3081525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.459+0000 7fb30d0bb640 1 -- 172.21.15.83:0/4173858037 >> v1:172.21.15.83:6789/0 conn(0x7fb308155b00 legacy=0x7fb30806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.459+0000 7fb30d0bb640 1 -- 172.21.15.83:0/4173858037 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3080c3720 con 0x7fb308148570 2024-09-19T07:19:50.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.463+0000 7fb30d0bb640 1 --2- 172.21.15.83:0/4173858037 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb308148570 0x7fb3080aa9e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb30000c310 tx=0x7fb30001d880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad98c11658f91343 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.467+0000 7fb2fffff640 1 -- 172.21.15.83:0/4173858037 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3000b1070 con 0x7fb308148570 2024-09-19T07:19:50.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.467+0000 7fb2fffff640 1 -- 172.21.15.83:0/4173858037 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb30000a2d0 con 0x7fb308148570 2024-09-19T07:19:50.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.467+0000 7fb2fffff640 1 -- 172.21.15.83:0/4173858037 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30000a6a0 con 0x7fb308148570 2024-09-19T07:19:50.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.467+0000 7fb30f31f640 1 -- 172.21.15.83:0/4173858037 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb308148570 msgr2=0x7fb3080aa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.467+0000 7fb30f31f640 1 --2- 172.21.15.83:0/4173858037 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb308148570 0x7fb3080aa9e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb30000c310 tx=0x7fb30001d880 comp rx=0 tx=0).stop 2024-09-19T07:19:50.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.467+0000 7fb30f31f640 1 -- 172.21.15.83:0/4173858037 shutdown_connections 2024-09-19T07:19:50.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.467+0000 7fb30f31f640 1 --2- 172.21.15.83:0/4173858037 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb308148570 0x7fb3080aa9e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.467+0000 7fb30f31f640 1 -- 172.21.15.83:0/4173858037 >> 172.21.15.83:0/4173858037 conn(0x7fb3080c7370 msgr2=0x7fb3080c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30f31f640 1 -- 172.21.15.83:0/4173858037 shutdown_connections 2024-09-19T07:19:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30f31f640 1 -- 172.21.15.83:0/4173858037 wait complete. 2024-09-19T07:19:50.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30f31f640 1 Processor -- start 2024-09-19T07:19:50.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30f31f640 1 -- start start 2024-09-19T07:19:50.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30f31f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3080c2ea0 0x7fb3081f0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30f31f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb308148570 0x7fb308149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30f31f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb308155b00 0x7fb3080c0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30f31f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb3081560f0 con 0x7fb308148570 2024-09-19T07:19:50.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30f31f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb308155f70 con 0x7fb308155b00 2024-09-19T07:19:50.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30f31f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb30815d910 con 0x7fb3080c2ea0 2024-09-19T07:19:50.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30c8ba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb308148570 0x7fb308149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30c8ba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb308148570 0x7fb308149fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54064/0 (socket says 172.21.15.83:54064) 2024-09-19T07:19:50.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30c8ba640 1 -- 172.21.15.83:0/3911196149 learned_addr learned my addr 172.21.15.83:0/3911196149 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:50.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30d0bb640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3080c2ea0 0x7fb3081f0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.479+0000 7fb30d8bc640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb308155b00 0x7fb3080c0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.483+0000 7fb30c8ba640 1 -- 172.21.15.83:0/3911196149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3080c2ea0 msgr2=0x7fb3081f0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.483+0000 7fb30c8ba640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3080c2ea0 0x7fb3081f0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.483+0000 7fb30c8ba640 1 -- 172.21.15.83:0/3911196149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb308155b00 msgr2=0x7fb3080c0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.483+0000 7fb30c8ba640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb308155b00 0x7fb3080c0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.483+0000 7fb30c8ba640 1 -- 172.21.15.83:0/3911196149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3081ec9c0 con 0x7fb308148570 2024-09-19T07:19:50.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.483+0000 7fb30c8ba640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb308148570 0x7fb308149fa0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb2f800c580 tx=0x7fb2f800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.491+0000 7fb2fdffb640 1 -- 172.21.15.83:0/3911196149 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8017070 con 0x7fb308148570 2024-09-19T07:19:50.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.491+0000 7fb30f31f640 1 -- 172.21.15.83:0/3911196149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3080729d0 con 0x7fb308148570 2024-09-19T07:19:50.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.491+0000 7fb30f31f640 1 -- 172.21.15.83:0/3911196149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3080c0f40 con 0x7fb308148570 2024-09-19T07:19:50.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.495+0000 7fb2fdffb640 1 -- 172.21.15.83:0/3911196149 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2f8003600 con 0x7fb308148570 2024-09-19T07:19:50.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.495+0000 7fb2fdffb640 1 -- 172.21.15.83:0/3911196149 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f80054a0 con 0x7fb308148570 2024-09-19T07:19:50.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.495+0000 7fb2fdffb640 1 -- 172.21.15.83:0/3911196149 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb2f801a070 con 0x7fb308148570 2024-09-19T07:19:50.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.495+0000 7fb2fdffb640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2ec0679c0 0x7fb2ec069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.495+0000 7fb2fdffb640 1 -- 172.21.15.83:0/3911196149 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fb2f8085360 con 0x7fb308148570 2024-09-19T07:19:50.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.495+0000 7fb30d0bb640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2ec0679c0 0x7fb2ec069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.495+0000 7fb30d0bb640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2ec0679c0 0x7fb2ec069e80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb3000ae000 tx=0x7fb3000a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.503+0000 7fb30f31f640 1 -- 172.21.15.83:0/3911196149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e80052a0 con 0x7fb308148570 2024-09-19T07:19:50.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.503+0000 7efe7f19b640 1 Processor -- start 2024-09-19T07:19:50.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.503+0000 7efe7f19b640 1 -- start start 2024-09-19T07:19:50.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe7f19b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7efe780c2ea0 0x7efe781525f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe7f19b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe78155b00 0x7efe7806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe7f19b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe78152b30 con 0x7efe78148570 2024-09-19T07:19:50.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe7f19b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7efe78155670 con 0x7efe780c2ea0 2024-09-19T07:19:50.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe7f19b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efe780c3440 con 0x7efe78155b00 2024-09-19T07:19:50.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe7d738640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe78155b00 0x7efe7806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe7cf37640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7efe78148570 0x7efe780aa9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:51072/0 (socket says 172.21.15.83:51072) 2024-09-19T07:19:50.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe7cf37640 1 -- 172.21.15.83:0/3561802814 learned_addr learned my addr 172.21.15.83:0/3561802814 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:50.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe77fff640 1 --2- 172.21.15.83:0/3561802814 >> v2:172.21.15.184:3300/0 conn(0x7efe780c2ea0 0x7efe781525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe777fe640 1 -- 172.21.15.83:0/3561802814 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2680796473 0 0) 0x7efe700014a0 con 0x7efe78148570 2024-09-19T07:19:50.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe777fe640 1 -- 172.21.15.83:0/3561802814 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe60005740 con 0x7efe78148570 2024-09-19T07:19:50.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe7d738640 1 -- 172.21.15.83:0/3561802814 >> v2:172.21.15.184:3300/0 conn(0x7efe780c2ea0 msgr2=0x7efe781525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:50.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe7d738640 1 --2- 172.21.15.83:0/3561802814 >> v2:172.21.15.184:3300/0 conn(0x7efe780c2ea0 0x7efe781525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe7d738640 1 -- 172.21.15.83:0/3561802814 >> v1:172.21.15.184:6789/0 conn(0x7efe78148570 legacy=0x7efe780aa9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7efe7d738640 1 -- 172.21.15.83:0/3561802814 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe780c3720 con 0x7efe78155b00 2024-09-19T07:19:50.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7eff40c42640 1 Processor -- start 2024-09-19T07:19:50.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.507+0000 7eff40c42640 1 -- start start 2024-09-19T07:19:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7eff40c42640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3c08c4a0 0x7eff3c16e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7eff40c42640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3c0c1aa0 con 0x7eff3c08c0c0 2024-09-19T07:19:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7eff40c42640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3c0abad0 con 0x7eff3c06ea20 2024-09-19T07:19:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7eff40c42640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7eff3c0ac370 con 0x7eff3c08c4a0 2024-09-19T07:19:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7eff3b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7eff3c08c0c0 0x7eff3c162ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35484/0 (socket says 172.21.15.83:35484) 2024-09-19T07:19:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7eff3b7fe640 1 -- 172.21.15.83:0/3581629124 learned_addr learned my addr 172.21.15.83:0/3581629124 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7eff3bfff640 1 --2- 172.21.15.83:0/3581629124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3c08c4a0 0x7eff3c16e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7eff3a7fc640 1 -- 172.21.15.83:0/3581629124 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3872081508 0 0) 0x7eff3c0abad0 con 0x7eff3c06ea20 2024-09-19T07:19:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7eff3a7fc640 1 -- 172.21.15.83:0/3581629124 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff24003530 con 0x7eff3c06ea20 2024-09-19T07:19:50.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7efe7d738640 1 --2- 172.21.15.83:0/3561802814 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe78155b00 0x7efe7806da50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe6800c1e0 tx=0x7efe680206e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cb201e822704edc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.515+0000 7fb2fdffb640 1 -- 172.21.15.83:0/3911196149 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb2f8055590 con 0x7fb308148570 2024-09-19T07:19:50.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7efe777fe640 1 -- 172.21.15.83:0/3561802814 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe68008be0 con 0x7efe78155b00 2024-09-19T07:19:50.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7efe777fe640 1 -- 172.21.15.83:0/3561802814 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe68008d80 con 0x7efe78155b00 2024-09-19T07:19:50.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7efe777fe640 1 -- 172.21.15.83:0/3561802814 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6800a2f0 con 0x7efe78155b00 2024-09-19T07:19:50.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.515+0000 7efe7f19b640 1 -- 172.21.15.83:0/3561802814 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe78155b00 msgr2=0x7efe7806da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.515+0000 7efe7f19b640 1 --2- 172.21.15.83:0/3561802814 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe78155b00 0x7efe7806da50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe6800c1e0 tx=0x7efe680206e0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.515+0000 7efe7f19b640 1 -- 172.21.15.83:0/3561802814 shutdown_connections 2024-09-19T07:19:50.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.515+0000 7efe7f19b640 1 --2- 172.21.15.83:0/3561802814 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe78155b00 0x7efe7806da50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.515+0000 7efe7f19b640 1 --2- 172.21.15.83:0/3561802814 >> v2:172.21.15.184:3300/0 conn(0x7efe780c2ea0 0x7efe781525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.515+0000 7efe7f19b640 1 -- 172.21.15.83:0/3561802814 >> 172.21.15.83:0/3561802814 conn(0x7efe780c7370 msgr2=0x7efe780c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.515+0000 7efe7f19b640 1 -- 172.21.15.83:0/3561802814 shutdown_connections 2024-09-19T07:19:50.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7eff3bfff640 1 -- 172.21.15.83:0/3581629124 >> v1:172.21.15.184:6789/0 conn(0x7eff3c06ea20 legacy=0x7eff3c06cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7eff3bfff640 1 -- 172.21.15.83:0/3581629124 >> v1:172.21.15.83:6789/0 conn(0x7eff3c08c0c0 legacy=0x7eff3c162ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.511+0000 7eff3bfff640 1 -- 172.21.15.83:0/3581629124 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3c0abdb0 con 0x7eff3c08c4a0 2024-09-19T07:19:50.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.515+0000 7efe7f19b640 1 -- 172.21.15.83:0/3561802814 wait complete. 2024-09-19T07:19:50.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.515+0000 7eff3bfff640 1 --2- 172.21.15.83:0/3581629124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3c08c4a0 0x7eff3c16e070 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff34007d10 tx=0x7eff3401eab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6504356ada8ea025 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.515+0000 7efe7f19b640 1 Processor -- start 2024-09-19T07:19:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.515+0000 7efe7f19b640 1 -- start start 2024-09-19T07:19:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7f19b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe780c2ea0 0x7efe781f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7f19b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe78148570 0x7efe7814e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7f19b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe78155b00 0x7efe78072450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7f19b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efe7815ee90 con 0x7efe78148570 2024-09-19T07:19:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7f19b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efe7815ed10 con 0x7efe78155b00 2024-09-19T07:19:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7f19b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efe7815f010 con 0x7efe780c2ea0 2024-09-19T07:19:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7eff3a7fc640 1 -- 172.21.15.83:0/3581629124 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff340b4070 con 0x7eff3c08c4a0 2024-09-19T07:19:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7eff3a7fc640 1 -- 172.21.15.83:0/3581629124 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff340a76b0 con 0x7eff3c08c4a0 2024-09-19T07:19:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7eff3a7fc640 1 -- 172.21.15.83:0/3581629124 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff300015b0 con 0x7eff3c08c4a0 2024-09-19T07:19:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe77fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe78148570 0x7efe7814e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe77fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe78148570 0x7efe7814e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54068/0 (socket says 172.21.15.83:54068) 2024-09-19T07:19:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe77fff640 1 -- 172.21.15.83:0/3341457644 learned_addr learned my addr 172.21.15.83:0/3341457644 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7cf37640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe780c2ea0 0x7efe781f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7d738640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe78155b00 0x7efe78072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7cf37640 1 -- 172.21.15.83:0/3341457644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe78155b00 msgr2=0x7efe78072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7cf37640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe78155b00 0x7efe78072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7cf37640 1 -- 172.21.15.83:0/3341457644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe78148570 msgr2=0x7efe7814e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7cf37640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe78148570 0x7efe7814e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7cf37640 1 -- 172.21.15.83:0/3341457644 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe7814b690 con 0x7efe780c2ea0 2024-09-19T07:19:50.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe77fff640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe78148570 0x7efe7814e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:50.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7cf37640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe780c2ea0 0x7efe781f6ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efe7000ca70 tx=0x7efe7000cf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7eff40c42640 1 -- 172.21.15.83:0/3581629124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3c08c4a0 msgr2=0x7eff3c16e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7eff40c42640 1 --2- 172.21.15.83:0/3581629124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3c08c4a0 0x7eff3c16e070 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff34007d10 tx=0x7eff3401eab0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7eff40c42640 1 -- 172.21.15.83:0/3581629124 shutdown_connections 2024-09-19T07:19:50.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7eff40c42640 1 --2- 172.21.15.83:0/3581629124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3c08c4a0 0x7eff3c16e070 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7eff40c42640 1 -- 172.21.15.83:0/3581629124 >> 172.21.15.83:0/3581629124 conn(0x7eff3c06dbc0 msgr2=0x7eff3c06dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe757fa640 1 -- 172.21.15.83:0/3341457644 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe70017070 con 0x7efe780c2ea0 2024-09-19T07:19:50.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe757fa640 1 -- 172.21.15.83:0/3341457644 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe7000abd0 con 0x7efe780c2ea0 2024-09-19T07:19:50.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7f19b640 1 -- 172.21.15.83:0/3341457644 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe7814b970 con 0x7efe780c2ea0 2024-09-19T07:19:50.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe757fa640 1 -- 172.21.15.83:0/3341457644 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe70012400 con 0x7efe780c2ea0 2024-09-19T07:19:50.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.519+0000 7efe7f19b640 1 -- 172.21.15.83:0/3341457644 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efe780c28b0 con 0x7efe780c2ea0 2024-09-19T07:19:50.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.523+0000 7eff40c42640 1 -- 172.21.15.83:0/3581629124 shutdown_connections 2024-09-19T07:19:50.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.523+0000 7eff40c42640 1 -- 172.21.15.83:0/3581629124 wait complete. 2024-09-19T07:19:50.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.523+0000 7efe757fa640 1 -- 172.21.15.83:0/3341457644 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efe7001a070 con 0x7efe780c2ea0 2024-09-19T07:19:50.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.523+0000 7efe757fa640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efe60067e30 0x7efe6006a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.523+0000 7efe757fa640 1 -- 172.21.15.83:0/3341457644 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7efe70085c30 con 0x7efe780c2ea0 2024-09-19T07:19:50.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.523+0000 7efe77fff640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efe60067e30 0x7efe6006a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.523+0000 7eff40c42640 1 Processor -- start 2024-09-19T07:19:50.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.523+0000 7eff40c42640 1 -- start start 2024-09-19T07:19:50.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff40c42640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff3c06ea20 0x7eff3c08ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff40c42640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff3c08c0c0 0x7eff3c084960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff40c42640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3c08c4a0 0x7eff3c088a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff40c42640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c170dd0 con 0x7eff3c06ea20 2024-09-19T07:19:50.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff40c42640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c162a60 con 0x7eff3c08c0c0 2024-09-19T07:19:50.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff40c42640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7eff3c170f50 con 0x7eff3c08c4a0 2024-09-19T07:19:50.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff3affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff3c08c0c0 0x7eff3c084960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff3affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff3c08c0c0 0x7eff3c084960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:46972/0 (socket says 172.21.15.83:46972) 2024-09-19T07:19:50.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff3affd640 1 -- 172.21.15.83:0/3580030045 learned_addr learned my addr 172.21.15.83:0/3580030045 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:50.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff3b7fe640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff3c06ea20 0x7eff3c08ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff3bfff640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3c08c4a0 0x7eff3c088a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.523+0000 7efe77fff640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efe60067e30 0x7efe6006a2f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efe780c1450 tx=0x7efe6c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7efe7f19b640 1 -- 172.21.15.83:0/3341457644 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe781525f0 con 0x7efe780c2ea0 2024-09-19T07:19:50.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff3affd640 1 -- 172.21.15.83:0/3580030045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3c08c4a0 msgr2=0x7eff3c088a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff3affd640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3c08c4a0 0x7eff3c088a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff3affd640 1 -- 172.21.15.83:0/3580030045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff3c06ea20 msgr2=0x7eff3c08ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff3affd640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff3c06ea20 0x7eff3c08ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff3affd640 1 -- 172.21.15.83:0/3580030045 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3c089750 con 0x7eff3c08c0c0 2024-09-19T07:19:50.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7eff3affd640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff3c08c0c0 0x7eff3c084960 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7eff3000c780 tx=0x7eff3000cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7fc49d4ab640 1 Processor -- start 2024-09-19T07:19:50.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7fc49d4ab640 1 -- start start 2024-09-19T07:19:50.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7fc49d4ab640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc498165c90 0x7fc4980c3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7fc49d4ab640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4980ab2f0 con 0x7fc4980ab7e0 2024-09-19T07:19:50.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7fc49d4ab640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4980aa770 con 0x7fc49808ca90 2024-09-19T07:19:50.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7fc49d4ab640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc4980ac0d0 con 0x7fc498165c90 2024-09-19T07:19:50.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7fc497fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc4980ab7e0 0x7fc49808c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35500/0 (socket says 172.21.15.83:35500) 2024-09-19T07:19:50.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7fc497fff640 1 -- 172.21.15.83:0/2416992482 learned_addr learned my addr 172.21.15.83:0/2416992482 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:50.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.527+0000 7fc49ccaa640 1 --2- 172.21.15.83:0/2416992482 >> v2:172.21.15.184:3300/0 conn(0x7fc498165c90 0x7fc4980c3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.531+0000 7fc49ccaa640 1 -- 172.21.15.83:0/2416992482 >> v1:172.21.15.184:6789/0 conn(0x7fc49808ca90 legacy=0x7fc4980c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.531+0000 7fc49ccaa640 1 -- 172.21.15.83:0/2416992482 >> v1:172.21.15.83:6789/0 conn(0x7fc4980ab7e0 legacy=0x7fc49808c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.531+0000 7fc49ccaa640 1 -- 172.21.15.83:0/2416992482 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4980aaa50 con 0x7fc498165c90 2024-09-19T07:19:50.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.531+0000 7eff23fff640 1 -- 172.21.15.83:0/3580030045 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30017070 con 0x7eff3c08c0c0 2024-09-19T07:19:50.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.531+0000 7eff23fff640 1 -- 172.21.15.83:0/3580030045 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff30005290 con 0x7eff3c08c0c0 2024-09-19T07:19:50.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.531+0000 7eff23fff640 1 -- 172.21.15.83:0/3580030045 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30003840 con 0x7eff3c08c0c0 2024-09-19T07:19:50.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.531+0000 7eff40c42640 1 -- 172.21.15.83:0/3580030045 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff3c166640 con 0x7eff3c08c0c0 2024-09-19T07:19:50.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.531+0000 7eff40c42640 1 -- 172.21.15.83:0/3580030045 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff3c16e0a0 con 0x7eff3c08c0c0 2024-09-19T07:19:50.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.531+0000 7f5d1b4f2640 1 Processor -- start 2024-09-19T07:19:50.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.531+0000 7f5d1b4f2640 1 -- start start 2024-09-19T07:19:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7eff40c42640 1 -- 172.21.15.83:0/3580030045 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff3c06cc90 con 0x7eff3c08c0c0 2024-09-19T07:19:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7fc49ccaa640 1 --2- 172.21.15.83:0/2416992482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc498165c90 0x7fc4980c3150 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc488010bd0 tx=0x7fc4880204b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ab427f7e888d953 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d1b4f2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5d0c0db020 0x7f5d0c0d7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d1b4f2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5d0c04a380 0x7f5d0c04a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d1b4f2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d0c0db5e0 con 0x7f5d0c0d51b0 2024-09-19T07:19:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d1b4f2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5d0c049ef0 con 0x7f5d0c0db020 2024-09-19T07:19:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d1b4f2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5d0c0db8c0 con 0x7f5d0c04a380 2024-09-19T07:19:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d19cef640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5d0c0db020 0x7f5d0c0d7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d19cef640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5d0c0db020 0x7f5d0c0d7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54076/0 (socket says 172.21.15.83:54076) 2024-09-19T07:19:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d19cef640 1 -- 172.21.15.83:0/1031295180 learned_addr learned my addr 172.21.15.83:0/1031295180 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:50.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7fc496ffd640 1 -- 172.21.15.83:0/2416992482 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4880ae070 con 0x7fc498165c90 2024-09-19T07:19:50.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7fc496ffd640 1 -- 172.21.15.83:0/2416992482 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4880a8d90 con 0x7fc498165c90 2024-09-19T07:19:50.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7fc496ffd640 1 -- 172.21.15.83:0/2416992482 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4880b2620 con 0x7fc498165c90 2024-09-19T07:19:50.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d19cef640 1 -- 172.21.15.83:0/1031295180 >> v2:172.21.15.184:3300/0 conn(0x7f5d0c04a380 msgr2=0x7f5d0c04a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:50.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d1acf1640 1 --2- 172.21.15.83:0/1031295180 >> v2:172.21.15.184:3300/0 conn(0x7f5d0c04a380 0x7f5d0c04a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d19cef640 1 --2- 172.21.15.83:0/1031295180 >> v2:172.21.15.184:3300/0 conn(0x7f5d0c04a380 0x7f5d0c04a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d19cef640 1 -- 172.21.15.83:0/1031295180 >> v1:172.21.15.184:6789/0 conn(0x7f5d0c0d51b0 legacy=0x7f5d0c047dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7f5d19cef640 1 -- 172.21.15.83:0/1031295180 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0c0dbba0 con 0x7f5d0c0db020 2024-09-19T07:19:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7fc49d4ab640 1 -- 172.21.15.83:0/2416992482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc498165c90 msgr2=0x7fc4980c3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7fc49d4ab640 1 --2- 172.21.15.83:0/2416992482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc498165c90 0x7fc4980c3150 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc488010bd0 tx=0x7fc4880204b0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7fc49d4ab640 1 -- 172.21.15.83:0/2416992482 shutdown_connections 2024-09-19T07:19:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7fc49d4ab640 1 --2- 172.21.15.83:0/2416992482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc498165c90 0x7fc4980c3150 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7fc49d4ab640 1 -- 172.21.15.83:0/2416992482 >> 172.21.15.83:0/2416992482 conn(0x7fc49806dba0 msgr2=0x7fc49806dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7eff23fff640 1 -- 172.21.15.83:0/3580030045 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7eff3001a070 con 0x7eff3c08c0c0 2024-09-19T07:19:50.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7eff23fff640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff28067910 0x7eff28069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7eff23fff640 1 -- 172.21.15.83:0/3580030045 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7eff30085980 con 0x7eff3c08c0c0 2024-09-19T07:19:50.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7fc49d4ab640 1 -- 172.21.15.83:0/2416992482 shutdown_connections 2024-09-19T07:19:50.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7eff3b7fe640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff28067910 0x7eff28069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7eff3b7fe640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff28067910 0x7eff28069dd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7eff2c005890 tx=0x7eff2c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.535+0000 7fc49d4ab640 1 -- 172.21.15.83:0/2416992482 wait complete. 2024-09-19T07:19:50.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc49d4ab640 1 Processor -- start 2024-09-19T07:19:50.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7f5d19cef640 1 --2- 172.21.15.83:0/1031295180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d0c0db020 0x7f5d0c0d7ab0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5d14050290 tx=0x7f5d14098d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1f63dbe32c97ac7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7f5d194ee640 1 -- 172.21.15.83:0/1031295180 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d14121960 con 0x7f5d0c0db020 2024-09-19T07:19:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7f5d194ee640 1 -- 172.21.15.83:0/1031295180 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d14121b40 con 0x7f5d0c0db020 2024-09-19T07:19:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7f5d194ee640 1 -- 172.21.15.83:0/1031295180 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d14129440 con 0x7f5d0c0db020 2024-09-19T07:19:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc49d4ab640 1 -- start start 2024-09-19T07:19:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc49d4ab640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc49808ca90 0x7fc498165070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc49d4ab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4980ab7e0 0x7fc49815f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc49d4ab640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc498165c90 0x7fc49815f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc49d4ab640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc4980aa530 con 0x7fc4980ab7e0 2024-09-19T07:19:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc49d4ab640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc4980aa3b0 con 0x7fc49808ca90 2024-09-19T07:19:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc49d4ab640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc4980abbc0 con 0x7fc498165c90 2024-09-19T07:19:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc4977fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4980ab7e0 0x7fc49815f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc4977fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4980ab7e0 0x7fc49815f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54078/0 (socket says 172.21.15.83:54078) 2024-09-19T07:19:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc4977fe640 1 -- 172.21.15.83:0/3373591837 learned_addr learned my addr 172.21.15.83:0/3373591837 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/1031295180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d0c0db020 msgr2=0x7f5d0c0d7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7f5d1b4f2640 1 --2- 172.21.15.83:0/1031295180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d0c0db020 0x7f5d0c0d7ab0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5d14050290 tx=0x7f5d14098d20 comp rx=0 tx=0).stop 2024-09-19T07:19:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/1031295180 shutdown_connections 2024-09-19T07:19:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7f5d1b4f2640 1 --2- 172.21.15.83:0/1031295180 >> v2:172.21.15.184:3300/0 conn(0x7f5d0c04a380 0x7f5d0c04a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7f5d1b4f2640 1 --2- 172.21.15.83:0/1031295180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d0c0db020 0x7f5d0c0d7ab0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/1031295180 >> 172.21.15.83:0/1031295180 conn(0x7f5d0c02f040 msgr2=0x7f5d0c02f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc4977fe640 1 -- 172.21.15.83:0/3373591837 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc498165c90 msgr2=0x7fc49815f8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:19:50.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc4977fe640 1 --2- 172.21.15.83:0/3373591837 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc498165c90 0x7fc49815f8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc4977fe640 1 -- 172.21.15.83:0/3373591837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc49808ca90 msgr2=0x7fc498165070 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:19:50.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc4977fe640 1 --2- 172.21.15.83:0/3373591837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc49808ca90 0x7fc498165070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc4977fe640 1 -- 172.21.15.83:0/3373591837 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49808b6e0 con 0x7fc4980ab7e0 2024-09-19T07:19:50.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/1031295180 shutdown_connections 2024-09-19T07:19:50.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/1031295180 wait complete. 2024-09-19T07:19:50.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.539+0000 7fc4977fe640 1 --2- 172.21.15.83:0/3373591837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4980ab7e0 0x7fc49815f390 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00c440 tx=0x7fc48c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7fc494ff9640 1 -- 172.21.15.83:0/3373591837 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48c017070 con 0x7fc4980ab7e0 2024-09-19T07:19:50.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7fc494ff9640 1 -- 172.21.15.83:0/3373591837 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc48c0052f0 con 0x7fc4980ab7e0 2024-09-19T07:19:50.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7fc494ff9640 1 -- 172.21.15.83:0/3373591837 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48c003840 con 0x7fc4980ab7e0 2024-09-19T07:19:50.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7fc49d4ab640 1 -- 172.21.15.83:0/3373591837 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4980bc970 con 0x7fc4980ab7e0 2024-09-19T07:19:50.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7fc49d4ab640 1 -- 172.21.15.83:0/3373591837 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4981655b0 con 0x7fc4980ab7e0 2024-09-19T07:19:50.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1b4f2640 1 Processor -- start 2024-09-19T07:19:50.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7efe757fa640 1 -- 172.21.15.83:0/3341457644 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7efe78152b30 con 0x7efe780c2ea0 2024-09-19T07:19:50.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7fc49d4ab640 1 -- 172.21.15.83:0/3373591837 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc49808c380 con 0x7fc4980ab7e0 2024-09-19T07:19:50.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1b4f2640 1 -- start start 2024-09-19T07:19:50.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1b4f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d0c04a380 0x7f5d0c02de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1b4f2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d0c0d51b0 0x7f5d0c17a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1b4f2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d0c0db020 0x7f5d0c17aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1b4f2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0c0484e0 con 0x7f5d0c04a380 2024-09-19T07:19:50.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1b4f2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0c04b860 con 0x7f5d0c0db020 2024-09-19T07:19:50.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1b4f2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5d0c048660 con 0x7f5d0c0d51b0 2024-09-19T07:19:50.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1a4f0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d0c04a380 0x7f5d0c02de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1a4f0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d0c04a380 0x7f5d0c02de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54090/0 (socket says 172.21.15.83:54090) 2024-09-19T07:19:50.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1a4f0640 1 -- 172.21.15.83:0/2589933070 learned_addr learned my addr 172.21.15.83:0/2589933070 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:50.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1acf1640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d0c0db020 0x7f5d0c17aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d19cef640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d0c0d51b0 0x7f5d0c17a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1acf1640 1 -- 172.21.15.83:0/2589933070 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d0c0d51b0 msgr2=0x7f5d0c17a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1acf1640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d0c0d51b0 0x7f5d0c17a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1acf1640 1 -- 172.21.15.83:0/2589933070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d0c04a380 msgr2=0x7f5d0c02de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1acf1640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d0c04a380 0x7f5d0c02de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7f5d1acf1640 1 -- 172.21.15.83:0/2589933070 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0c02ed60 con 0x7f5d0c0db020 2024-09-19T07:19:50.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7fc494ff9640 1 -- 172.21.15.83:0/3373591837 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc48c01a070 con 0x7fc4980ab7e0 2024-09-19T07:19:50.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7fc494ff9640 1 --2- 172.21.15.83:0/3373591837 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc4840678c0 0x7fc484069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.543+0000 7fc494ff9640 1 -- 172.21.15.83:0/3373591837 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fc48c0852f0 con 0x7fc4980ab7e0 2024-09-19T07:19:50.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7fc497fff640 1 --2- 172.21.15.83:0/3373591837 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc4840678c0 0x7fc484069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f5d1acf1640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d0c0db020 0x7f5d0c17aac0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400c540 tx=0x7f5d0400ca00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7fc497fff640 1 --2- 172.21.15.83:0/3373591837 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc4840678c0 0x7fc484069d80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc49000aaa0 tx=0x7fc490009120 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f4fae478640 1 Processor -- start 2024-09-19T07:19:50.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f4fae478640 1 -- start start 2024-09-19T07:19:50.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f5d0affd640 1 -- 172.21.15.83:0/2589933070 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d04015070 con 0x7f5d0c0db020 2024-09-19T07:19:50.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f5d0affd640 1 -- 172.21.15.83:0/2589933070 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d04004240 con 0x7f5d0c0db020 2024-09-19T07:19:50.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f5d0affd640 1 -- 172.21.15.83:0/2589933070 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d04003620 con 0x7f5d0c0db020 2024-09-19T07:19:50.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/2589933070 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d0c0cd8d0 con 0x7f5d0c0db020 2024-09-19T07:19:50.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/2589933070 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d0c0429b0 con 0x7f5d0c0db020 2024-09-19T07:19:50.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7eff23fff640 1 -- 172.21.15.83:0/3580030045 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7eff30021020 con 0x7eff3c08c0c0 2024-09-19T07:19:50.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f4fae478640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4fa00d7480 0x7f4fa0046d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f4fae478640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fa00d7860 con 0x7f4fa0049950 2024-09-19T07:19:50.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f4fae478640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fa0008db0 con 0x7f4fa00da960 2024-09-19T07:19:50.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f4fae478640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4fa0048700 con 0x7f4fa00d7480 2024-09-19T07:19:50.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f4faca15640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4fa0049950 0x7f4fa00e4580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35510/0 (socket says 172.21.15.83:35510) 2024-09-19T07:19:50.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f4faca15640 1 -- 172.21.15.83:0/4293933368 learned_addr learned my addr 172.21.15.83:0/4293933368 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:50.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.547+0000 7f4fa7fff640 1 --2- 172.21.15.83:0/4293933368 >> v2:172.21.15.184:3300/0 conn(0x7f4fa00d7480 0x7f4fa0046d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.551+0000 7f4fa7fff640 1 -- 172.21.15.83:0/4293933368 >> v1:172.21.15.184:6789/0 conn(0x7f4fa00da960 legacy=0x7f4fa0007e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.551+0000 7f4fa7fff640 1 -- 172.21.15.83:0/4293933368 >> v1:172.21.15.83:6789/0 conn(0x7f4fa0049950 legacy=0x7f4fa00e4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.551+0000 7f4fa7fff640 1 -- 172.21.15.83:0/4293933368 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa00489e0 con 0x7f4fa00d7480 2024-09-19T07:19:50.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.551+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/2589933070 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d0c0d5590 con 0x7f5d0c0db020 2024-09-19T07:19:50.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.551+0000 7f5d0affd640 1 -- 172.21.15.83:0/2589933070 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5d040037c0 con 0x7f5d0c0db020 2024-09-19T07:19:50.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.551+0000 7f5d0affd640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5cf8067990 0x7f5cf8069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.551+0000 7f5d0affd640 1 -- 172.21.15.83:0/2589933070 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f5d04085c10 con 0x7f5d0c0db020 2024-09-19T07:19:50.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.551+0000 7f5d1a4f0640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5cf8067990 0x7f5cf8069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.555+0000 7f5d1a4f0640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5cf8067990 0x7f5cf8069e50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5d10007fd0 tx=0x7f5d10009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.555+0000 7fc494ff9640 1 -- 172.21.15.83:0/3373591837 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc48c055af0 con 0x7fc4980ab7e0 2024-09-19T07:19:50.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.559+0000 7f4fa7fff640 1 --2- 172.21.15.83:0/4293933368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fa00d7480 0x7f4fa0046d60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4f940097c0 tx=0x7f4f9401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=477859973ea37a8e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.563+0000 7f5d0affd640 1 -- 172.21.15.83:0/2589933070 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5d04055e40 con 0x7f5d0c0db020 2024-09-19T07:19:50.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.563+0000 7f4fa6ffd640 1 -- 172.21.15.83:0/4293933368 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f940aa5f0 con 0x7f4fa00d7480 2024-09-19T07:19:50.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.563+0000 7f4fa6ffd640 1 -- 172.21.15.83:0/4293933368 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f940af070 con 0x7f4fa00d7480 2024-09-19T07:19:50.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.563+0000 7f4fa6ffd640 1 -- 172.21.15.83:0/4293933368 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f940aa9b0 con 0x7f4fa00d7480 2024-09-19T07:19:50.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 -- 172.21.15.83:0/4293933368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fa00d7480 msgr2=0x7f4fa0046d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 --2- 172.21.15.83:0/4293933368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fa00d7480 0x7f4fa0046d60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4f940097c0 tx=0x7f4f9401c470 comp rx=0 tx=0).stop 2024-09-19T07:19:50.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 -- 172.21.15.83:0/4293933368 shutdown_connections 2024-09-19T07:19:50.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 --2- 172.21.15.83:0/4293933368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fa00d7480 0x7f4fa0046d60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 -- 172.21.15.83:0/4293933368 >> 172.21.15.83:0/4293933368 conn(0x7f4fa0048e60 msgr2=0x7f4fa0049240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 -- 172.21.15.83:0/4293933368 shutdown_connections 2024-09-19T07:19:50.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 -- 172.21.15.83:0/4293933368 wait complete. 2024-09-19T07:19:50.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 Processor -- start 2024-09-19T07:19:50.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 -- start start 2024-09-19T07:19:50.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fa0049950 0x7f4fa0042340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fa00d7480 0x7f4fa00daff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fa00da960 0x7f4fa00db530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa0047630 con 0x7f4fa00da960 2024-09-19T07:19:50.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa00474b0 con 0x7f4fa0049950 2024-09-19T07:19:50.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.567+0000 7f4fae478640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4fa00e84c0 con 0x7f4fa00d7480 2024-09-19T07:19:50.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.571+0000 7f4faca15640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fa00da960 0x7f4fa00db530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.571+0000 7f4fa7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fa0049950 0x7f4fa0042340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.571+0000 7f4faca15640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fa00da960 0x7f4fa00db530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54100/0 (socket says 172.21.15.83:54100) 2024-09-19T07:19:50.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.571+0000 7f4faca15640 1 -- 172.21.15.83:0/1599506406 learned_addr learned my addr 172.21.15.83:0/1599506406 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:50.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.571+0000 7f4fa77fe640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fa00d7480 0x7f4fa00daff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.571+0000 7f4faca15640 1 -- 172.21.15.83:0/1599506406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fa00d7480 msgr2=0x7f4fa00daff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.571+0000 7f4faca15640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fa00d7480 0x7f4fa00daff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.571+0000 7f4faca15640 1 -- 172.21.15.83:0/1599506406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fa0049950 msgr2=0x7f4fa0042340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.571+0000 7f4faca15640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fa0049950 0x7f4fa0042340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.571+0000 7f4faca15640 1 -- 172.21.15.83:0/1599506406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa00d19e0 con 0x7f4fa00da960 2024-09-19T07:19:50.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.571+0000 7f4faca15640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fa00da960 0x7f4fa00db530 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4fa807bdd0 tx=0x7f4fa8069900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.575+0000 7f4fa4ff9640 1 -- 172.21.15.83:0/1599506406 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8091070 con 0x7f4fa00da960 2024-09-19T07:19:50.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.575+0000 7f4fa4ff9640 1 -- 172.21.15.83:0/1599506406 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa8071630 con 0x7f4fa00da960 2024-09-19T07:19:50.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.575+0000 7f4fa4ff9640 1 -- 172.21.15.83:0/1599506406 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8071a00 con 0x7f4fa00da960 2024-09-19T07:19:50.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.575+0000 7f4fae478640 1 -- 172.21.15.83:0/1599506406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa00ccaf0 con 0x7f4fa00da960 2024-09-19T07:19:50.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.575+0000 7f4fae478640 1 -- 172.21.15.83:0/1599506406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fa00d0ab0 con 0x7f4fa00da960 2024-09-19T07:19:50.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.575+0000 7f4fae478640 1 -- 172.21.15.83:0/1599506406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f880052a0 con 0x7f4fa00da960 2024-09-19T07:19:50.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.579+0000 7f4fa4ff9640 1 -- 172.21.15.83:0/1599506406 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4fa8094070 con 0x7f4fa00da960 2024-09-19T07:19:50.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.579+0000 7f4fa4ff9640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f98067ad0 0x7f4f98069f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.579+0000 7f4fa4ff9640 1 -- 172.21.15.83:0/1599506406 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f4fa8101f30 con 0x7f4fa00da960 2024-09-19T07:19:50.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.579+0000 7f4fa7fff640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f98067ad0 0x7f4f98069f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.579+0000 7f4fa7fff640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f98067ad0 0x7f4f98069f90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4f940ae000 tx=0x7f4f940a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.587+0000 7f4fa4ff9640 1 -- 172.21.15.83:0/1599506406 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4fa80d2160 con 0x7f4fa00da960 2024-09-19T07:19:50.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.603+0000 7f7fe8ef5640 1 Processor -- start 2024-09-19T07:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.603+0000 7f7fe8ef5640 1 -- start start 2024-09-19T07:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.603+0000 7f7fe8ef5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7fe40c2ea0 0x7f7fe41525f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.603+0000 7f7fe8ef5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe4155b00 0x7f7fe406da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.603+0000 7f7fe8ef5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fe4152b30 con 0x7f7fe4148570 2024-09-19T07:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.603+0000 7f7fe8ef5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7fe4155670 con 0x7f7fe40c2ea0 2024-09-19T07:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.603+0000 7f7fe8ef5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe40c3440 con 0x7f7fe4155b00 2024-09-19T07:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.607+0000 7f7fe1d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7fe40c2ea0 0x7f7fe41525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.607+0000 7f7fe1d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7fe40c2ea0 0x7f7fe41525f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54106/0 (socket says 172.21.15.83:54106) 2024-09-19T07:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.607+0000 7f7fe1d9b640 1 -- 172.21.15.83:0/3814277183 learned_addr learned my addr 172.21.15.83:0/3814277183 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.607+0000 7f7fe2d9d640 1 --2- 172.21.15.83:0/3814277183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe4155b00 0x7f7fe406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.607+0000 7f7fe1d9b640 1 -- 172.21.15.83:0/3814277183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe4155b00 msgr2=0x7f7fe406da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.607+0000 7f7fe1d9b640 1 --2- 172.21.15.83:0/3814277183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe4155b00 0x7f7fe406da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.607+0000 7f7fe1d9b640 1 -- 172.21.15.83:0/3814277183 >> v1:172.21.15.184:6789/0 conn(0x7f7fe4148570 legacy=0x7f7fe40aa9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.607+0000 7f7fe1d9b640 1 -- 172.21.15.83:0/3814277183 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe40c3720 con 0x7f7fe40c2ea0 2024-09-19T07:19:50.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.611+0000 7f7fe1d9b640 1 --2- 172.21.15.83:0/3814277183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe40c2ea0 0x7f7fe41525f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009a50 tx=0x7f7fdc01cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1476b840aa9a9ac9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.615+0000 7f7fe159a640 1 -- 172.21.15.83:0/3814277183 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0a5a70 con 0x7f7fe40c2ea0 2024-09-19T07:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.615+0000 7f7fe159a640 1 -- 172.21.15.83:0/3814277183 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fdc0a5c10 con 0x7f7fe40c2ea0 2024-09-19T07:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.615+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/3814277183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe40c2ea0 msgr2=0x7f7fe41525f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.615+0000 7f7fe8ef5640 1 --2- 172.21.15.83:0/3814277183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe40c2ea0 0x7f7fe41525f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009a50 tx=0x7f7fdc01cc90 comp rx=0 tx=0).stop 2024-09-19T07:19:50.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.615+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/3814277183 shutdown_connections 2024-09-19T07:19:50.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.615+0000 7f7fe8ef5640 1 --2- 172.21.15.83:0/3814277183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe4155b00 0x7f7fe406da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.615+0000 7f7fe8ef5640 1 --2- 172.21.15.83:0/3814277183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe40c2ea0 0x7f7fe41525f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.615+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/3814277183 >> 172.21.15.83:0/3814277183 conn(0x7f7fe40c7370 msgr2=0x7f7fe40c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.615+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/3814277183 shutdown_connections 2024-09-19T07:19:50.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.615+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/3814277183 wait complete. 2024-09-19T07:19:50.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.615+0000 7f7fe8ef5640 1 Processor -- start 2024-09-19T07:19:50.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.615+0000 7f7fe8ef5640 1 -- start start 2024-09-19T07:19:50.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe8ef5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe4148570 0x7f7fe414c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe8ef5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe4155b00 0x7f7fe4151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe8ef5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe40a9cd0 0x7f7fe40736a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe8ef5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe415ee90 con 0x7f7fe4155b00 2024-09-19T07:19:50.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe8ef5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe415ed10 con 0x7f7fe40a9cd0 2024-09-19T07:19:50.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe8ef5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe415f010 con 0x7f7fe4148570 2024-09-19T07:19:50.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe4155b00 0x7f7fe4151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe4155b00 0x7f7fe4151010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54114/0 (socket says 172.21.15.83:54114) 2024-09-19T07:19:50.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe1d9b640 1 -- 172.21.15.83:0/1100889157 learned_addr learned my addr 172.21.15.83:0/1100889157 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:50.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe259c640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe4148570 0x7f7fe414c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe2d9d640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe40a9cd0 0x7f7fe40736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe1d9b640 1 -- 172.21.15.83:0/1100889157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe4148570 msgr2=0x7f7fe414c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe1d9b640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe4148570 0x7f7fe414c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe1d9b640 1 -- 172.21.15.83:0/1100889157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe40a9cd0 msgr2=0x7f7fe40736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe1d9b640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe40a9cd0 0x7f7fe40736a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe1d9b640 1 -- 172.21.15.83:0/1100889157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe414c770 con 0x7f7fe4155b00 2024-09-19T07:19:50.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.619+0000 7f7fe1d9b640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe4155b00 0x7f7fe4151010 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0198a0 tx=0x7f7fdc019a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.623+0000 7f7fd2ffd640 1 -- 172.21.15.83:0/1100889157 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0bc070 con 0x7f7fe4155b00 2024-09-19T07:19:50.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.623+0000 7f7fd2ffd640 1 -- 172.21.15.83:0/1100889157 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fdc0a5a70 con 0x7f7fe4155b00 2024-09-19T07:19:50.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.627+0000 7f7fd2ffd640 1 -- 172.21.15.83:0/1100889157 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0b8680 con 0x7f7fe4155b00 2024-09-19T07:19:50.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.627+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/1100889157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe40a9a60 con 0x7f7fe4155b00 2024-09-19T07:19:50.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.627+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/1100889157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe40a9780 con 0x7f7fe4155b00 2024-09-19T07:19:50.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.627+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/1100889157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe40c2ea0 con 0x7f7fe4155b00 2024-09-19T07:19:50.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.631+0000 7f7fd2ffd640 1 -- 172.21.15.83:0/1100889157 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7fdc0bf070 con 0x7f7fe4155b00 2024-09-19T07:19:50.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.631+0000 7f7fd2ffd640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7fc4067990 0x7f7fc4069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:50.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.631+0000 7f7fd2ffd640 1 -- 172.21.15.83:0/1100889157 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f7fdc121050 con 0x7f7fe4155b00 2024-09-19T07:19:50.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.631+0000 7f7fe259c640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7fc4067990 0x7f7fc4069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:50.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.631+0000 7f7fe259c640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7fc4067990 0x7f7fc4069e50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4007fd0 tx=0x7f7fd4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:50.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.639+0000 7f7fd2ffd640 1 -- 172.21.15.83:0/1100889157 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7fdc016940 con 0x7f7fe4155b00 2024-09-19T07:19:50.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.711+0000 7f9e63b17640 1 -- 172.21.15.83:0/3001275838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f9e44005590 con 0x7f9e5c15e460 2024-09-19T07:19:50.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.711+0000 7f9e527fc640 1 -- 172.21.15.83:0/3001275838 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9e58059820 con 0x7f9e5c15e460 2024-09-19T07:19:50.715 INFO:teuthology.orchestra.run.smithi083.stdout:60129542152 2024-09-19T07:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.715+0000 7f9e3ffff640 1 -- 172.21.15.83:0/3001275838 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e400678c0 msgr2=0x7f9e40069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.715+0000 7f9e3ffff640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e400678c0 0x7f9e40069d80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9e540ae000 tx=0x7f9e540a7070 comp rx=0 tx=0).stop 2024-09-19T07:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.715+0000 7f9e3ffff640 1 -- 172.21.15.83:0/3001275838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e5c15e460 msgr2=0x7f9e5c0c3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.715+0000 7f9e3ffff640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e5c15e460 0x7f9e5c0c3a20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800c390 tx=0x7f9e5800c850 comp rx=0 tx=0).stop 2024-09-19T07:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.715+0000 7f9e3ffff640 1 -- 172.21.15.83:0/3001275838 shutdown_connections 2024-09-19T07:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.715+0000 7f9e3ffff640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e400678c0 0x7f9e40069d80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.715+0000 7f9e3ffff640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e5c15e460 0x7f9e5c0c3a20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.715+0000 7f9e3ffff640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e5c154460 0x7f9e5c155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.715+0000 7f9e3ffff640 1 --2- 172.21.15.83:0/3001275838 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e5c0c2850 0x7f9e5c1482f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.715+0000 7f9e3ffff640 1 -- 172.21.15.83:0/3001275838 >> 172.21.15.83:0/3001275838 conn(0x7f9e5c075590 msgr2=0x7f9e5c151c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.715+0000 7f9e3ffff640 1 -- 172.21.15.83:0/3001275838 shutdown_connections 2024-09-19T07:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.715+0000 7f9e3ffff640 1 -- 172.21.15.83:0/3001275838 wait complete. 2024-09-19T07:19:50.739 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.3 2024-09-19T07:19:50.739 DEBUG:teuthology.parallel:result is None 2024-09-19T07:19:50.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 -- 172.21.15.83:0/3580030045 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7eff3c0c1880 con 0x7eff3c08c0c0 2024-09-19T07:19:50.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff23fff640 1 -- 172.21.15.83:0/3580030045 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7eff30005440 con 0x7eff3c08c0c0 2024-09-19T07:19:50.823 INFO:teuthology.orchestra.run.smithi083.stdout:60129542154 2024-09-19T07:19:50.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 -- 172.21.15.83:0/3580030045 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff28067910 msgr2=0x7eff28069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff28067910 0x7eff28069dd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7eff2c005890 tx=0x7eff2c009040 comp rx=0 tx=0).stop 2024-09-19T07:19:50.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 -- 172.21.15.83:0/3580030045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff3c08c0c0 msgr2=0x7eff3c084960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff3c08c0c0 0x7eff3c084960 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7eff3000c780 tx=0x7eff3000cb80 comp rx=0 tx=0).stop 2024-09-19T07:19:50.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 -- 172.21.15.83:0/3580030045 shutdown_connections 2024-09-19T07:19:50.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff28067910 0x7eff28069dd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3c08c4a0 0x7eff3c088a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff3c08c0c0 0x7eff3c084960 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 --2- 172.21.15.83:0/3580030045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff3c06ea20 0x7eff3c08ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 -- 172.21.15.83:0/3580030045 >> 172.21.15.83:0/3580030045 conn(0x7eff3c06dbc0 msgr2=0x7eff3c0c1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 -- 172.21.15.83:0/3580030045 shutdown_connections 2024-09-19T07:19:50.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.823+0000 7eff40c42640 1 -- 172.21.15.83:0/3580030045 wait complete. 2024-09-19T07:19:50.841 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.6 2024-09-19T07:19:50.841 DEBUG:teuthology.parallel:result is None 2024-09-19T07:19:50.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 -- 172.21.15.83:0/3911196149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb2e8005590 con 0x7fb308148570 2024-09-19T07:19:50.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb2fdffb640 1 -- 172.21.15.83:0/3911196149 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb2f8059890 con 0x7fb308148570 2024-09-19T07:19:50.843 INFO:teuthology.orchestra.run.smithi083.stdout:60129542154 2024-09-19T07:19:50.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 -- 172.21.15.83:0/3911196149 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2ec0679c0 msgr2=0x7fb2ec069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2ec0679c0 0x7fb2ec069e80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb3000ae000 tx=0x7fb3000a7000 comp rx=0 tx=0).stop 2024-09-19T07:19:50.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 -- 172.21.15.83:0/3911196149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb308148570 msgr2=0x7fb308149fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb308148570 0x7fb308149fa0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb2f800c580 tx=0x7fb2f800c980 comp rx=0 tx=0).stop 2024-09-19T07:19:50.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 -- 172.21.15.83:0/3911196149 shutdown_connections 2024-09-19T07:19:50.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2ec0679c0 0x7fb2ec069e80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb308155b00 0x7fb3080c0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb308148570 0x7fb308149fa0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 --2- 172.21.15.83:0/3911196149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3080c2ea0 0x7fb3081f0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 -- 172.21.15.83:0/3911196149 >> 172.21.15.83:0/3911196149 conn(0x7fb3080c7370 msgr2=0x7fb3080c7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 -- 172.21.15.83:0/3911196149 shutdown_connections 2024-09-19T07:19:50.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.843+0000 7fb30f31f640 1 -- 172.21.15.83:0/3911196149 wait complete. 2024-09-19T07:19:50.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.851+0000 7efe7f19b640 1 -- 172.21.15.83:0/3341457644 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7efe780c5150 con 0x7efe780c2ea0 2024-09-19T07:19:50.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.851+0000 7efe757fa640 1 -- 172.21.15.83:0/3341457644 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efe70055dd0 con 0x7efe780c2ea0 2024-09-19T07:19:50.853 INFO:teuthology.orchestra.run.smithi083.stdout:60129542152 2024-09-19T07:19:50.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.855+0000 7efe7f19b640 1 -- 172.21.15.83:0/3341457644 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efe60067e30 msgr2=0x7efe6006a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.855+0000 7efe7f19b640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efe60067e30 0x7efe6006a2f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efe780c1450 tx=0x7efe6c008040 comp rx=0 tx=0).stop 2024-09-19T07:19:50.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.855+0000 7efe7f19b640 1 -- 172.21.15.83:0/3341457644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe780c2ea0 msgr2=0x7efe781f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.855+0000 7efe7f19b640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe780c2ea0 0x7efe781f6ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efe7000ca70 tx=0x7efe7000cf30 comp rx=0 tx=0).stop 2024-09-19T07:19:50.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.855+0000 7efe7f19b640 1 -- 172.21.15.83:0/3341457644 shutdown_connections 2024-09-19T07:19:50.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.855+0000 7efe7f19b640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efe60067e30 0x7efe6006a2f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.855+0000 7efe7f19b640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe78155b00 0x7efe78072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.855+0000 7efe7f19b640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe78148570 0x7efe7814e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.855+0000 7efe7f19b640 1 --2- 172.21.15.83:0/3341457644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe780c2ea0 0x7efe781f6ee0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.855+0000 7efe7f19b640 1 -- 172.21.15.83:0/3341457644 >> 172.21.15.83:0/3341457644 conn(0x7efe780c7370 msgr2=0x7efe781547c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.855+0000 7efe7f19b640 1 -- 172.21.15.83:0/3341457644 shutdown_connections 2024-09-19T07:19:50.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.855+0000 7efe7f19b640 1 -- 172.21.15.83:0/3341457644 wait complete. 2024-09-19T07:19:50.862 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.4 2024-09-19T07:19:50.862 DEBUG:teuthology.parallel:result is None 2024-09-19T07:19:50.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.867+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/2589933070 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f5d0c00c2c0 con 0x7f5d0c0db020 2024-09-19T07:19:50.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.867+0000 7f5d0affd640 1 -- 172.21.15.83:0/2589933070 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5d0405a140 con 0x7f5d0c0db020 2024-09-19T07:19:50.868 INFO:teuthology.orchestra.run.smithi083.stdout:60129542153 2024-09-19T07:19:50.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.871+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/2589933070 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5cf8067990 msgr2=0x7f5cf8069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.871+0000 7f5d1b4f2640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5cf8067990 0x7f5cf8069e50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5d10007fd0 tx=0x7f5d10009040 comp rx=0 tx=0).stop 2024-09-19T07:19:50.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.871+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/2589933070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d0c0db020 msgr2=0x7f5d0c17aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.871+0000 7f5d1b4f2640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d0c0db020 0x7f5d0c17aac0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400c540 tx=0x7f5d0400ca00 comp rx=0 tx=0).stop 2024-09-19T07:19:50.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.871+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/2589933070 shutdown_connections 2024-09-19T07:19:50.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.871+0000 7f5d1b4f2640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5cf8067990 0x7f5cf8069e50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.871+0000 7f5d1b4f2640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d0c0db020 0x7f5d0c17aac0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.871+0000 7f5d1b4f2640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d0c0d51b0 0x7f5d0c17a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.871+0000 7f5d1b4f2640 1 --2- 172.21.15.83:0/2589933070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d0c04a380 0x7f5d0c02de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.871+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/2589933070 >> 172.21.15.83:0/2589933070 conn(0x7f5d0c02f040 msgr2=0x7f5d0c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.871+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/2589933070 shutdown_connections 2024-09-19T07:19:50.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.871+0000 7f5d1b4f2640 1 -- 172.21.15.83:0/2589933070 wait complete. 2024-09-19T07:19:50.872 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.5 2024-09-19T07:19:50.872 DEBUG:teuthology.parallel:result is None 2024-09-19T07:19:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.875+0000 7fc49d4ab640 1 -- 172.21.15.83:0/3373591837 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc498162670 con 0x7fc4980ab7e0 2024-09-19T07:19:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.875+0000 7fc494ff9640 1 -- 172.21.15.83:0/3373591837 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc48c0553f0 con 0x7fc4980ab7e0 2024-09-19T07:19:50.874 INFO:teuthology.orchestra.run.smithi083.stdout:60129542153 2024-09-19T07:19:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.875+0000 7fc49d4ab640 1 -- 172.21.15.83:0/3373591837 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc4840678c0 msgr2=0x7fc484069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.879+0000 7fc49d4ab640 1 --2- 172.21.15.83:0/3373591837 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc4840678c0 0x7fc484069d80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc49000aaa0 tx=0x7fc490009120 comp rx=0 tx=0).stop 2024-09-19T07:19:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.879+0000 7fc49d4ab640 1 -- 172.21.15.83:0/3373591837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4980ab7e0 msgr2=0x7fc49815f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.879+0000 7fc49d4ab640 1 --2- 172.21.15.83:0/3373591837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4980ab7e0 0x7fc49815f390 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00c440 tx=0x7fc48c00c900 comp rx=0 tx=0).stop 2024-09-19T07:19:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.879+0000 7fc49d4ab640 1 -- 172.21.15.83:0/3373591837 shutdown_connections 2024-09-19T07:19:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.879+0000 7fc49d4ab640 1 --2- 172.21.15.83:0/3373591837 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc4840678c0 0x7fc484069d80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.879+0000 7fc49d4ab640 1 --2- 172.21.15.83:0/3373591837 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc498165c90 0x7fc49815f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.879+0000 7fc49d4ab640 1 --2- 172.21.15.83:0/3373591837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4980ab7e0 0x7fc49815f390 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.879+0000 7fc49d4ab640 1 --2- 172.21.15.83:0/3373591837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc49808ca90 0x7fc498165070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.879+0000 7fc49d4ab640 1 -- 172.21.15.83:0/3373591837 >> 172.21.15.83:0/3373591837 conn(0x7fc49806dba0 msgr2=0x7fc4980c6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.879+0000 7fc49d4ab640 1 -- 172.21.15.83:0/3373591837 shutdown_connections 2024-09-19T07:19:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.879+0000 7fc49d4ab640 1 -- 172.21.15.83:0/3373591837 wait complete. 2024-09-19T07:19:50.893 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.7 2024-09-19T07:19:50.893 DEBUG:teuthology.parallel:result is None 2024-09-19T07:19:50.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.1 2024-09-19T07:19:50.901 DEBUG:teuthology.parallel:result is None 2024-09-19T07:19:50.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 -- 172.21.15.83:0/1599506406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f4f88005860 con 0x7f4fa00da960 2024-09-19T07:19:50.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fa4ff9640 1 -- 172.21.15.83:0/1599506406 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4fa80d6460 con 0x7f4fa00da960 2024-09-19T07:19:50.906 INFO:teuthology.orchestra.run.smithi083.stdout:60129542154 2024-09-19T07:19:50.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 -- 172.21.15.83:0/1599506406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f98067ad0 msgr2=0x7f4f98069f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f98067ad0 0x7f4f98069f90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4f940ae000 tx=0x7f4f940a7000 comp rx=0 tx=0).stop 2024-09-19T07:19:50.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 -- 172.21.15.83:0/1599506406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fa00da960 msgr2=0x7f4fa00db530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fa00da960 0x7f4fa00db530 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4fa807bdd0 tx=0x7f4fa8069900 comp rx=0 tx=0).stop 2024-09-19T07:19:50.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 -- 172.21.15.83:0/1599506406 shutdown_connections 2024-09-19T07:19:50.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f98067ad0 0x7f4f98069f90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fa00da960 0x7f4fa00db530 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fa00d7480 0x7f4fa00daff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 --2- 172.21.15.83:0/1599506406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fa0049950 0x7f4fa0042340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 -- 172.21.15.83:0/1599506406 >> 172.21.15.83:0/1599506406 conn(0x7f4fa0048e60 msgr2=0x7f4fa00d97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 -- 172.21.15.83:0/1599506406 shutdown_connections 2024-09-19T07:19:50.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.907+0000 7f4fae478640 1 -- 172.21.15.83:0/1599506406 wait complete. 2024-09-19T07:19:50.923 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.2 2024-09-19T07:19:50.923 DEBUG:teuthology.parallel:result is None 2024-09-19T07:19:50.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.923+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/1100889157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f7fe40c7b60 con 0x7f7fe4155b00 2024-09-19T07:19:50.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.923+0000 7f7fd2ffd640 1 -- 172.21.15.83:0/1100889157 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7fdc0f11f0 con 0x7f7fe4155b00 2024-09-19T07:19:50.925 INFO:teuthology.orchestra.run.smithi083.stdout:60129542152 2024-09-19T07:19:50.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.927+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/1100889157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7fc4067990 msgr2=0x7f7fc4069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.927+0000 7f7fe8ef5640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7fc4067990 0x7f7fc4069e50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4007fd0 tx=0x7f7fd4009040 comp rx=0 tx=0).stop 2024-09-19T07:19:50.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.927+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/1100889157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe4155b00 msgr2=0x7f7fe4151010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:50.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.927+0000 7f7fe8ef5640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe4155b00 0x7f7fe4151010 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0198a0 tx=0x7f7fdc019a30 comp rx=0 tx=0).stop 2024-09-19T07:19:50.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.927+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/1100889157 shutdown_connections 2024-09-19T07:19:50.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.927+0000 7f7fe8ef5640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7fc4067990 0x7f7fc4069e50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.927+0000 7f7fe8ef5640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe40a9cd0 0x7f7fe40736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.927+0000 7f7fe8ef5640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe4155b00 0x7f7fe4151010 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.927+0000 7f7fe8ef5640 1 --2- 172.21.15.83:0/1100889157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe4148570 0x7f7fe414c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:50.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.927+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/1100889157 >> 172.21.15.83:0/1100889157 conn(0x7f7fe40c7370 msgr2=0x7f7fe40c61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:50.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.927+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/1100889157 shutdown_connections 2024-09-19T07:19:50.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:50.927+0000 7f7fe8ef5640 1 -- 172.21.15.83:0/1100889157 wait complete. 2024-09-19T07:19:50.941 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.0 2024-09-19T07:19:50.941 DEBUG:teuthology.parallel:result is None 2024-09-19T07:19:50.941 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-19T07:19:50.942 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:19:51.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.087+0000 7ff230a6a640 1 Processor -- start 2024-09-19T07:19:51.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.087+0000 7ff230a6a640 1 -- start start 2024-09-19T07:19:51.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.087+0000 7ff230a6a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff22c152b20 0x7ff22c0c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.087+0000 7ff230a6a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff22c150670 con 0x7ff22c0c45c0 2024-09-19T07:19:51.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.087+0000 7ff230a6a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff22c0c2880 con 0x7ff22c150290 2024-09-19T07:19:51.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.087+0000 7ff230a6a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff22c0716c0 con 0x7ff22c152b20 2024-09-19T07:19:51.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.087+0000 7ff22ad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff22c0c45c0 0x7ff22c153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35516/0 (socket says 172.21.15.83:35516) 2024-09-19T07:19:51.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.087+0000 7ff22ad9d640 1 -- 172.21.15.83:0/1944506107 learned_addr learned my addr 172.21.15.83:0/1944506107 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:51.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.087+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3729600267 0 0) 0x7ff22c150670 con 0x7ff22c0c45c0 2024-09-19T07:19:51.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.087+0000 7ff229d9b640 1 --2- 172.21.15.83:0/1944506107 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff22c152b20 0x7ff22c0c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.087+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff214003530 con 0x7ff22c0c45c0 2024-09-19T07:19:51.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2102140887 0 0) 0x7ff214003530 con 0x7ff22c0c45c0 2024-09-19T07:19:51.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff22c152b20 msgr2=0x7ff22c0c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:51.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 --2- 172.21.15.83:0/1944506107 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff22c152b20 0x7ff22c0c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 >> v1:172.21.15.184:6789/0 conn(0x7ff22c150290 legacy=0x7ff22c0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22c0c2b60 con 0x7ff22c0c45c0 2024-09-19T07:19:51.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7ff224001a00 con 0x7ff22c0c45c0 2024-09-19T07:19:51.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 >> v1:172.21.15.83:6789/0 conn(0x7ff22c0c45c0 legacy=0x7ff22c153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 --2- 172.21.15.83:0/1944506107 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2140039f0 0x7ff214003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 --2- 172.21.15.83:0/1944506107 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff22c152b20 0x7ff214004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 --2- 172.21.15.83:0/1944506107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff214004d50 0x7ff2140051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff21400ba80 con 0x7ff214004d50 2024-09-19T07:19:51.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff21400bc00 con 0x7ff2140039f0 2024-09-19T07:19:51.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff21400be00 con 0x7ff22c152b20 2024-09-19T07:19:51.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22ad9d640 1 --2- 172.21.15.83:0/1944506107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff214004d50 0x7ff2140051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22a59c640 1 --2- 172.21.15.83:0/1944506107 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2140039f0 0x7ff214003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22ad9d640 1 -- 172.21.15.83:0/1944506107 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff22c152b20 msgr2=0x7ff214004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:51.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22ad9d640 1 --2- 172.21.15.83:0/1944506107 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff22c152b20 0x7ff214004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22ad9d640 1 -- 172.21.15.83:0/1944506107 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2140039f0 msgr2=0x7ff214003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:51.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22ad9d640 1 --2- 172.21.15.83:0/1944506107 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2140039f0 0x7ff214003e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22ad9d640 1 -- 172.21.15.83:0/1944506107 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff214003530 con 0x7ff214004d50 2024-09-19T07:19:51.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22ad9d640 1 --2- 172.21.15.83:0/1944506107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff214004d50 0x7ff2140051d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff224002240 tx=0x7ff22401d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582ce21174a9ae9a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:51.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22400a4a0 con 0x7ff214004d50 2024-09-19T07:19:51.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2240ad650 con 0x7ff214004d50 2024-09-19T07:19:51.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22959a640 1 -- 172.21.15.83:0/1944506107 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2240ada40 con 0x7ff214004d50 2024-09-19T07:19:51.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff230a6a640 1 -- 172.21.15.83:0/1944506107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff214004d50 msgr2=0x7ff2140051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff230a6a640 1 --2- 172.21.15.83:0/1944506107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff214004d50 0x7ff2140051d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff224002240 tx=0x7ff22401d360 comp rx=0 tx=0).stop 2024-09-19T07:19:51.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff22ad9d640 1 -- 172.21.15.83:0/1944506107 reap_dead start 2024-09-19T07:19:51.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff230a6a640 1 -- 172.21.15.83:0/1944506107 shutdown_connections 2024-09-19T07:19:51.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff230a6a640 1 -- 172.21.15.83:0/1944506107 >> 172.21.15.83:0/1944506107 conn(0x7ff22c06bc80 msgr2=0x7ff22c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:51.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff230a6a640 1 -- 172.21.15.83:0/1944506107 shutdown_connections 2024-09-19T07:19:51.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.091+0000 7ff230a6a640 1 -- 172.21.15.83:0/1944506107 wait complete. 2024-09-19T07:19:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff230a6a640 1 Processor -- start 2024-09-19T07:19:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff230a6a640 1 -- start start 2024-09-19T07:19:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff230a6a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff214004d50 0x7ff22c0c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff230a6a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff22c073520 0x7ff22c1e8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff230a6a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff22c1e8b60 0x7ff22c1eb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff230a6a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c0c3490 con 0x7ff22c1e8b60 2024-09-19T07:19:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff230a6a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c0c3120 con 0x7ff214004d50 2024-09-19T07:19:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff230a6a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff22c0c4080 con 0x7ff22c073520 2024-09-19T07:19:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff22ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff22c1e8b60 0x7ff22c1eb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff22ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff22c1e8b60 0x7ff22c1eb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54132/0 (socket says 172.21.15.83:54132) 2024-09-19T07:19:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff22ad9d640 1 -- 172.21.15.83:0/2053666161 learned_addr learned my addr 172.21.15.83:0/2053666161 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff22a59c640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff214004d50 0x7ff22c0c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff22ad9d640 1 -- 172.21.15.83:0/2053666161 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff22c073520 msgr2=0x7ff22c1e8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff22ad9d640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff22c073520 0x7ff22c1e8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff22ad9d640 1 -- 172.21.15.83:0/2053666161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff214004d50 msgr2=0x7ff22c0c0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff22ad9d640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff214004d50 0x7ff22c0c0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff22ad9d640 1 -- 172.21.15.83:0/2053666161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22c1debb0 con 0x7ff22c1e8b60 2024-09-19T07:19:51.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff22a59c640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff214004d50 0x7ff22c0c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:51.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff22ad9d640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff22c1e8b60 0x7ff22c1eb280 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff22401dc00 tx=0x7ff214005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:51.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff21affd640 1 -- 172.21.15.83:0/2053666161 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2240b2040 con 0x7ff22c1e8b60 2024-09-19T07:19:51.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff21affd640 1 -- 172.21.15.83:0/2053666161 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2240bc070 con 0x7ff22c1e8b60 2024-09-19T07:19:51.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff230a6a640 1 -- 172.21.15.83:0/2053666161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff22c1de8d0 con 0x7ff22c1e8b60 2024-09-19T07:19:51.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff21affd640 1 -- 172.21.15.83:0/2053666161 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22400a480 con 0x7ff22c1e8b60 2024-09-19T07:19:51.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff230a6a640 1 -- 172.21.15.83:0/2053666161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff22c14a130 con 0x7ff22c1e8b60 2024-09-19T07:19:51.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff230a6a640 1 -- 172.21.15.83:0/2053666161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff22c0c1ff0 con 0x7ff22c1e8b60 2024-09-19T07:19:51.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff21affd640 1 -- 172.21.15.83:0/2053666161 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff2240017c0 con 0x7ff22c1e8b60 2024-09-19T07:19:51.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff21affd640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff20c0678f0 0x7ff20c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff21affd640 1 -- 172.21.15.83:0/2053666161 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7ff224120770 con 0x7ff22c1e8b60 2024-09-19T07:19:51.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.095+0000 7ff22a59c640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff20c0678f0 0x7ff20c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.099+0000 7ff22a59c640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff20c0678f0 0x7ff20c069db0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff21c001680 tx=0x7ff21c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:51.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.099+0000 7ff21affd640 1 -- 172.21.15.83:0/2053666161 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff224017070 con 0x7ff22c1e8b60 2024-09-19T07:19:51.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.247+0000 7ff230a6a640 1 -- 172.21.15.83:0/2053666161 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff22c0c3e90 con 0x7ff20c0678f0 2024-09-19T07:19:51.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.271+0000 7ff21affd640 1 -- 172.21.15.83:0/2053666161 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490308 (secure 0 0 0) 0x7ff22c0c3e90 con 0x7ff20c0678f0 2024-09-19T07:19:51.275 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:51.279 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:19:51.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.275+0000 7ff230a6a640 1 -- 172.21.15.83:0/2053666161 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff20c0678f0 msgr2=0x7ff20c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.275+0000 7ff230a6a640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff20c0678f0 0x7ff20c069db0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff21c001680 tx=0x7ff21c009040 comp rx=0 tx=0).stop 2024-09-19T07:19:51.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.275+0000 7ff230a6a640 1 -- 172.21.15.83:0/2053666161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff22c1e8b60 msgr2=0x7ff22c1eb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.275+0000 7ff230a6a640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff22c1e8b60 0x7ff22c1eb280 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff22401dc00 tx=0x7ff214005840 comp rx=0 tx=0).stop 2024-09-19T07:19:51.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.275+0000 7ff230a6a640 1 -- 172.21.15.83:0/2053666161 shutdown_connections 2024-09-19T07:19:51.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.275+0000 7ff230a6a640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff20c0678f0 0x7ff20c069db0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.275+0000 7ff230a6a640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff22c1e8b60 0x7ff22c1eb280 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.275+0000 7ff230a6a640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff22c073520 0x7ff22c1e8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.275+0000 7ff230a6a640 1 --2- 172.21.15.83:0/2053666161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff214004d50 0x7ff22c0c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.275+0000 7ff230a6a640 1 -- 172.21.15.83:0/2053666161 >> 172.21.15.83:0/2053666161 conn(0x7ff22c06bc80 msgr2=0x7ff22c14dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:51.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.275+0000 7ff230a6a640 1 -- 172.21.15.83:0/2053666161 shutdown_connections 2024-09-19T07:19:51.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.275+0000 7ff230a6a640 1 -- 172.21.15.83:0/2053666161 wait complete. 2024-09-19T07:19:51.292 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-19T07:19:50.816657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223748,"kb_used_data":3336,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750972,"statfs":{"total":773094113280,"available":772864995328,"internally_reserved":0,"allocated":3416064,"data_stored":2010296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003296"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364888+0000","last_change":"2024-09-19T07:19:40.358457+0000","last_active":"2024-09-19T07:19:42.364888+0000","last_peered":"2024-09-19T07:19:42.364888+0000","last_clean":"2024-09-19T07:19:42.364888+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:19:42.364888+0000","last_undegraded":"2024-09-19T07:19:42.364888+0000","last_fullsized":"2024-09-19T07:19:42.364888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370195+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:19:42.370195+0000","last_peered":"2024-09-19T07:19:42.370195+0000","last_clean":"2024-09-19T07:19:42.370195+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:19:42.370195+0000","last_undegraded":"2024-09-19T07:19:42.370195+0000","last_fullsized":"2024-09-19T07:19:42.370195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:57.799442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372168+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:19:42.372168+0000","last_peered":"2024-09-19T07:19:42.372168+0000","last_clean":"2024-09-19T07:19:42.372168+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:19:42.372168+0000","last_undegraded":"2024-09-19T07:19:42.372168+0000","last_fullsized":"2024-09-19T07:19:42.372168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368511+0000","last_change":"2024-09-19T07:19:39.401621+0000","last_active":"2024-09-19T07:19:42.368511+0000","last_peered":"2024-09-19T07:19:42.368511+0000","last_clean":"2024-09-19T07:19:42.368511+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:19:42.368511+0000","last_undegraded":"2024-09-19T07:19:42.368511+0000","last_fullsized":"2024-09-19T07:19:42.368511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:32.095381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366428+0000","last_change":"2024-09-19T07:19:40.366073+0000","last_active":"2024-09-19T07:19:42.366428+0000","last_peered":"2024-09-19T07:19:42.366428+0000","last_clean":"2024-09-19T07:19:42.366428+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:19:42.366428+0000","last_undegraded":"2024-09-19T07:19:42.366428+0000","last_fullsized":"2024-09-19T07:19:42.366428+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372588+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:19:42.372588+0000","last_peered":"2024-09-19T07:19:42.372588+0000","last_clean":"2024-09-19T07:19:42.372588+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:19:42.372588+0000","last_undegraded":"2024-09-19T07:19:42.372588+0000","last_fullsized":"2024-09-19T07:19:42.372588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369500+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:19:42.369500+0000","last_peered":"2024-09-19T07:19:42.369500+0000","last_clean":"2024-09-19T07:19:42.369500+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:19:42.369500+0000","last_undegraded":"2024-09-19T07:19:42.369500+0000","last_fullsized":"2024-09-19T07:19:42.369500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:15.036396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364782+0000","last_change":"2024-09-19T07:19:39.403810+0000","last_active":"2024-09-19T07:19:42.364782+0000","last_peered":"2024-09-19T07:19:42.364782+0000","last_clean":"2024-09-19T07:19:42.364782+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:19:42.364782+0000","last_undegraded":"2024-09-19T07:19:42.364782+0000","last_fullsized":"2024-09-19T07:19:42.364782+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:55.714964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365790+0000","last_change":"2024-09-19T07:19:40.390455+0000","last_active":"2024-09-19T07:19:42.365790+0000","last_peered":"2024-09-19T07:19:42.365790+0000","last_clean":"2024-09-19T07:19:42.365790+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:19:42.365790+0000","last_undegraded":"2024-09-19T07:19:42.365790+0000","last_fullsized":"2024-09-19T07:19:42.365790+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367307+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:19:42.367307+0000","last_peered":"2024-09-19T07:19:42.367307+0000","last_clean":"2024-09-19T07:19:42.367307+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:19:42.367307+0000","last_undegraded":"2024-09-19T07:19:42.367307+0000","last_fullsized":"2024-09-19T07:19:42.367307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353956+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:19:41.353956+0000","last_peered":"2024-09-19T07:19:41.353956+0000","last_clean":"2024-09-19T07:19:41.353956+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:19:41.353956+0000","last_undegraded":"2024-09-19T07:19:41.353956+0000","last_fullsized":"2024-09-19T07:19:41.353956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366091+0000","last_change":"2024-09-19T07:19:39.408309+0000","last_active":"2024-09-19T07:19:42.366091+0000","last_peered":"2024-09-19T07:19:42.366091+0000","last_clean":"2024-09-19T07:19:42.366091+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:19:42.366091+0000","last_undegraded":"2024-09-19T07:19:42.366091+0000","last_fullsized":"2024-09-19T07:19:42.366091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:19.396261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353849+0000","last_change":"2024-09-19T07:19:40.372543+0000","last_active":"2024-09-19T07:19:41.353849+0000","last_peered":"2024-09-19T07:19:41.353849+0000","last_clean":"2024-09-19T07:19:41.353849+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:19:41.353849+0000","last_undegraded":"2024-09-19T07:19:41.353849+0000","last_fullsized":"2024-09-19T07:19:41.353849+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:53:15.682896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367705+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:19:42.367705+0000","last_peered":"2024-09-19T07:19:42.367705+0000","last_clean":"2024-09-19T07:19:42.367705+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:19:42.367705+0000","last_undegraded":"2024-09-19T07:19:42.367705+0000","last_fullsized":"2024-09-19T07:19:42.367705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355253+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:19:41.355253+0000","last_peered":"2024-09-19T07:19:41.355253+0000","last_clean":"2024-09-19T07:19:41.355253+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:19:41.355253+0000","last_undegraded":"2024-09-19T07:19:41.355253+0000","last_fullsized":"2024-09-19T07:19:41.355253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366176+0000","last_change":"2024-09-19T07:19:39.407045+0000","last_active":"2024-09-19T07:19:42.366176+0000","last_peered":"2024-09-19T07:19:42.366176+0000","last_clean":"2024-09-19T07:19:42.366176+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:19:42.366176+0000","last_undegraded":"2024-09-19T07:19:42.366176+0000","last_fullsized":"2024-09-19T07:19:42.366176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:24.846974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367577+0000","last_change":"2024-09-19T07:19:40.396520+0000","last_active":"2024-09-19T07:19:42.367577+0000","last_peered":"2024-09-19T07:19:42.367577+0000","last_clean":"2024-09-19T07:19:42.367577+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:19:42.367577+0000","last_undegraded":"2024-09-19T07:19:42.367577+0000","last_fullsized":"2024-09-19T07:19:42.367577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353504+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:19:41.353504+0000","last_peered":"2024-09-19T07:19:41.353504+0000","last_clean":"2024-09-19T07:19:41.353504+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:19:41.353504+0000","last_undegraded":"2024-09-19T07:19:41.353504+0000","last_fullsized":"2024-09-19T07:19:41.353504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367046+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:19:42.367046+0000","last_peered":"2024-09-19T07:19:42.367046+0000","last_clean":"2024-09-19T07:19:42.367046+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:19:42.367046+0000","last_undegraded":"2024-09-19T07:19:42.367046+0000","last_fullsized":"2024-09-19T07:19:42.367046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368034+0000","last_change":"2024-09-19T07:19:39.410500+0000","last_active":"2024-09-19T07:19:42.368034+0000","last_peered":"2024-09-19T07:19:42.368034+0000","last_clean":"2024-09-19T07:19:42.368034+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:19:42.368034+0000","last_undegraded":"2024-09-19T07:19:42.368034+0000","last_fullsized":"2024-09-19T07:19:42.368034+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:04.182606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349177+0000","last_change":"2024-09-19T07:19:40.381259+0000","last_active":"2024-09-19T07:19:41.349177+0000","last_peered":"2024-09-19T07:19:41.349177+0000","last_clean":"2024-09-19T07:19:41.349177+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:19:41.349177+0000","last_undegraded":"2024-09-19T07:19:41.349177+0000","last_fullsized":"2024-09-19T07:19:41.349177+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:22.017682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370825+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:19:42.370825+0000","last_peered":"2024-09-19T07:19:42.370825+0000","last_clean":"2024-09-19T07:19:42.370825+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:19:42.370825+0000","last_undegraded":"2024-09-19T07:19:42.370825+0000","last_fullsized":"2024-09-19T07:19:42.370825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367918+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:19:42.367918+0000","last_peered":"2024-09-19T07:19:42.367918+0000","last_clean":"2024-09-19T07:19:42.367918+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:19:42.367918+0000","last_undegraded":"2024-09-19T07:19:42.367918+0000","last_fullsized":"2024-09-19T07:19:42.367918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367682+0000","last_change":"2024-09-19T07:19:39.412123+0000","last_active":"2024-09-19T07:19:42.367682+0000","last_peered":"2024-09-19T07:19:42.367682+0000","last_clean":"2024-09-19T07:19:42.367682+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:19:42.367682+0000","last_undegraded":"2024-09-19T07:19:42.367682+0000","last_fullsized":"2024-09-19T07:19:42.367682+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367943+0000","last_change":"2024-09-19T07:19:40.383214+0000","last_active":"2024-09-19T07:19:42.367943+0000","last_peered":"2024-09-19T07:19:42.367943+0000","last_clean":"2024-09-19T07:19:42.367943+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:19:42.367943+0000","last_undegraded":"2024-09-19T07:19:42.367943+0000","last_fullsized":"2024-09-19T07:19:42.367943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367066+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:19:42.367066+0000","last_peered":"2024-09-19T07:19:42.367066+0000","last_clean":"2024-09-19T07:19:42.367066+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:19:42.367066+0000","last_undegraded":"2024-09-19T07:19:42.367066+0000","last_fullsized":"2024-09-19T07:19:42.367066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367094+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:19:42.367094+0000","last_peered":"2024-09-19T07:19:42.367094+0000","last_clean":"2024-09-19T07:19:42.367094+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:19:42.367094+0000","last_undegraded":"2024-09-19T07:19:42.367094+0000","last_fullsized":"2024-09-19T07:19:42.367094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369704+0000","last_change":"2024-09-19T07:19:39.408228+0000","last_active":"2024-09-19T07:19:42.369704+0000","last_peered":"2024-09-19T07:19:42.369704+0000","last_clean":"2024-09-19T07:19:42.369704+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:19:42.369704+0000","last_undegraded":"2024-09-19T07:19:42.369704+0000","last_fullsized":"2024-09-19T07:19:42.369704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355858+0000","last_change":"2024-09-19T07:19:40.395566+0000","last_active":"2024-09-19T07:19:41.355858+0000","last_peered":"2024-09-19T07:19:41.355858+0000","last_clean":"2024-09-19T07:19:41.355858+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:19:41.355858+0000","last_undegraded":"2024-09-19T07:19:41.355858+0000","last_fullsized":"2024-09-19T07:19:41.355858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:54:02.766021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367642+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:19:42.367642+0000","last_peered":"2024-09-19T07:19:42.367642+0000","last_clean":"2024-09-19T07:19:42.367642+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:19:42.367642+0000","last_undegraded":"2024-09-19T07:19:42.367642+0000","last_fullsized":"2024-09-19T07:19:42.367642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366078+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:19:42.366078+0000","last_peered":"2024-09-19T07:19:42.366078+0000","last_clean":"2024-09-19T07:19:42.366078+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:19:42.366078+0000","last_undegraded":"2024-09-19T07:19:42.366078+0000","last_fullsized":"2024-09-19T07:19:42.366078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368203+0000","last_change":"2024-09-19T07:19:39.424661+0000","last_active":"2024-09-19T07:19:42.368203+0000","last_peered":"2024-09-19T07:19:42.368203+0000","last_clean":"2024-09-19T07:19:42.368203+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:19:42.368203+0000","last_undegraded":"2024-09-19T07:19:42.368203+0000","last_fullsized":"2024-09-19T07:19:42.368203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365431+0000","last_change":"2024-09-19T07:19:40.388972+0000","last_active":"2024-09-19T07:19:42.365431+0000","last_peered":"2024-09-19T07:19:42.365431+0000","last_clean":"2024-09-19T07:19:42.365431+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:19:42.365431+0000","last_undegraded":"2024-09-19T07:19:42.365431+0000","last_fullsized":"2024-09-19T07:19:42.365431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371005+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:19:42.371005+0000","last_peered":"2024-09-19T07:19:42.371005+0000","last_clean":"2024-09-19T07:19:42.371005+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:19:42.371005+0000","last_undegraded":"2024-09-19T07:19:42.371005+0000","last_fullsized":"2024-09-19T07:19:42.371005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:23.791880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.357019+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:19:41.357019+0000","last_peered":"2024-09-19T07:19:41.357019+0000","last_clean":"2024-09-19T07:19:41.357019+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:19:41.357019+0000","last_undegraded":"2024-09-19T07:19:41.357019+0000","last_fullsized":"2024-09-19T07:19:41.357019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366707+0000","last_change":"2024-09-19T07:19:39.412192+0000","last_active":"2024-09-19T07:19:42.366707+0000","last_peered":"2024-09-19T07:19:42.366707+0000","last_clean":"2024-09-19T07:19:42.366707+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:19:42.366707+0000","last_undegraded":"2024-09-19T07:19:42.366707+0000","last_fullsized":"2024-09-19T07:19:42.366707+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365056+0000","last_change":"2024-09-19T07:19:40.390217+0000","last_active":"2024-09-19T07:19:42.365056+0000","last_peered":"2024-09-19T07:19:42.365056+0000","last_clean":"2024-09-19T07:19:42.365056+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:19:42.365056+0000","last_undegraded":"2024-09-19T07:19:42.365056+0000","last_fullsized":"2024-09-19T07:19:42.365056+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371269+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:19:42.371269+0000","last_peered":"2024-09-19T07:19:42.371269+0000","last_clean":"2024-09-19T07:19:42.371269+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:19:42.371269+0000","last_undegraded":"2024-09-19T07:19:42.371269+0000","last_fullsized":"2024-09-19T07:19:42.371269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368999+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:19:42.368999+0000","last_peered":"2024-09-19T07:19:42.368999+0000","last_clean":"2024-09-19T07:19:42.368999+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:19:42.368999+0000","last_undegraded":"2024-09-19T07:19:42.368999+0000","last_fullsized":"2024-09-19T07:19:42.368999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:50.700962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365712+0000","last_change":"2024-09-19T07:19:39.434150+0000","last_active":"2024-09-19T07:19:42.365712+0000","last_peered":"2024-09-19T07:19:42.365712+0000","last_clean":"2024-09-19T07:19:42.365712+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:19:42.365712+0000","last_undegraded":"2024-09-19T07:19:42.365712+0000","last_fullsized":"2024-09-19T07:19:42.365712+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:13.654208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352846+0000","last_change":"2024-09-19T07:19:40.380140+0000","last_active":"2024-09-19T07:19:41.352846+0000","last_peered":"2024-09-19T07:19:41.352846+0000","last_clean":"2024-09-19T07:19:41.352846+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:19:41.352846+0000","last_undegraded":"2024-09-19T07:19:41.352846+0000","last_fullsized":"2024-09-19T07:19:41.352846+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368156+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:19:42.368156+0000","last_peered":"2024-09-19T07:19:42.368156+0000","last_clean":"2024-09-19T07:19:42.368156+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:19:42.368156+0000","last_undegraded":"2024-09-19T07:19:42.368156+0000","last_fullsized":"2024-09-19T07:19:42.368156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367593+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:19:42.367593+0000","last_peered":"2024-09-19T07:19:42.367593+0000","last_clean":"2024-09-19T07:19:42.367593+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:19:42.367593+0000","last_undegraded":"2024-09-19T07:19:42.367593+0000","last_fullsized":"2024-09-19T07:19:42.367593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366996+0000","last_change":"2024-09-19T07:19:39.411605+0000","last_active":"2024-09-19T07:19:42.366996+0000","last_peered":"2024-09-19T07:19:42.366996+0000","last_clean":"2024-09-19T07:19:42.366996+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:19:42.366996+0000","last_undegraded":"2024-09-19T07:19:42.366996+0000","last_fullsized":"2024-09-19T07:19:42.366996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:55.654483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367982+0000","last_change":"2024-09-19T07:19:40.372083+0000","last_active":"2024-09-19T07:19:42.367982+0000","last_peered":"2024-09-19T07:19:42.367982+0000","last_clean":"2024-09-19T07:19:42.367982+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:19:42.367982+0000","last_undegraded":"2024-09-19T07:19:42.367982+0000","last_fullsized":"2024-09-19T07:19:42.367982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:39.665475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355342+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:19:41.355342+0000","last_peered":"2024-09-19T07:19:41.355342+0000","last_clean":"2024-09-19T07:19:41.355342+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:19:41.355342+0000","last_undegraded":"2024-09-19T07:19:41.355342+0000","last_fullsized":"2024-09-19T07:19:41.355342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368442+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:19:42.368442+0000","last_peered":"2024-09-19T07:19:42.368442+0000","last_clean":"2024-09-19T07:19:42.368442+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:19:42.368442+0000","last_undegraded":"2024-09-19T07:19:42.368442+0000","last_fullsized":"2024-09-19T07:19:42.368442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367339+0000","last_change":"2024-09-19T07:19:39.420340+0000","last_active":"2024-09-19T07:19:42.367339+0000","last_peered":"2024-09-19T07:19:42.367339+0000","last_clean":"2024-09-19T07:19:42.367339+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:19:42.367339+0000","last_undegraded":"2024-09-19T07:19:42.367339+0000","last_fullsized":"2024-09-19T07:19:42.367339+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:29.338848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369245+0000","last_change":"2024-09-19T07:19:40.383082+0000","last_active":"2024-09-19T07:19:42.369245+0000","last_peered":"2024-09-19T07:19:42.369245+0000","last_clean":"2024-09-19T07:19:42.369245+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:19:42.369245+0000","last_undegraded":"2024-09-19T07:19:42.369245+0000","last_fullsized":"2024-09-19T07:19:42.369245+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:54.436493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367222+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:19:42.367222+0000","last_peered":"2024-09-19T07:19:42.367222+0000","last_clean":"2024-09-19T07:19:42.367222+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:19:42.367222+0000","last_undegraded":"2024-09-19T07:19:42.367222+0000","last_fullsized":"2024-09-19T07:19:42.367222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354505+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:19:41.354505+0000","last_peered":"2024-09-19T07:19:41.354505+0000","last_clean":"2024-09-19T07:19:41.354505+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:19:41.354505+0000","last_undegraded":"2024-09-19T07:19:41.354505+0000","last_fullsized":"2024-09-19T07:19:41.354505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368035+0000","last_change":"2024-09-19T07:19:39.407135+0000","last_active":"2024-09-19T07:19:42.368035+0000","last_peered":"2024-09-19T07:19:42.368035+0000","last_clean":"2024-09-19T07:19:42.368035+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:19:42.368035+0000","last_undegraded":"2024-09-19T07:19:42.368035+0000","last_fullsized":"2024-09-19T07:19:42.368035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:27.890595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349118+0000","last_change":"2024-09-19T07:19:40.368866+0000","last_active":"2024-09-19T07:19:41.349118+0000","last_peered":"2024-09-19T07:19:41.349118+0000","last_clean":"2024-09-19T07:19:41.349118+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:19:41.349118+0000","last_undegraded":"2024-09-19T07:19:41.349118+0000","last_fullsized":"2024-09-19T07:19:41.349118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:28.893439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366065+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:19:42.366065+0000","last_peered":"2024-09-19T07:19:42.366065+0000","last_clean":"2024-09-19T07:19:42.366065+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:19:42.366065+0000","last_undegraded":"2024-09-19T07:19:42.366065+0000","last_fullsized":"2024-09-19T07:19:42.366065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348283+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:19:41.348283+0000","last_peered":"2024-09-19T07:19:41.348283+0000","last_clean":"2024-09-19T07:19:41.348283+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:19:41.348283+0000","last_undegraded":"2024-09-19T07:19:41.348283+0000","last_fullsized":"2024-09-19T07:19:41.348283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369806+0000","last_change":"2024-09-19T07:19:39.420497+0000","last_active":"2024-09-19T07:19:42.369806+0000","last_peered":"2024-09-19T07:19:42.369806+0000","last_clean":"2024-09-19T07:19:42.369806+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:19:42.369806+0000","last_undegraded":"2024-09-19T07:19:42.369806+0000","last_fullsized":"2024-09-19T07:19:42.369806+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:19.547270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356452+0000","last_change":"2024-09-19T07:19:40.387573+0000","last_active":"2024-09-19T07:19:41.356452+0000","last_peered":"2024-09-19T07:19:41.356452+0000","last_clean":"2024-09-19T07:19:41.356452+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:19:41.356452+0000","last_undegraded":"2024-09-19T07:19:41.356452+0000","last_fullsized":"2024-09-19T07:19:41.356452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:20:43.234793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365631+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:19:42.365631+0000","last_peered":"2024-09-19T07:19:42.365631+0000","last_clean":"2024-09-19T07:19:42.365631+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:19:42.365631+0000","last_undegraded":"2024-09-19T07:19:42.365631+0000","last_fullsized":"2024-09-19T07:19:42.365631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:20.824916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366356+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:19:42.366356+0000","last_peered":"2024-09-19T07:19:42.366356+0000","last_clean":"2024-09-19T07:19:42.366356+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:19:42.366356+0000","last_undegraded":"2024-09-19T07:19:42.366356+0000","last_fullsized":"2024-09-19T07:19:42.366356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371671+0000","last_change":"2024-09-19T07:19:39.410752+0000","last_active":"2024-09-19T07:19:42.371671+0000","last_peered":"2024-09-19T07:19:42.371671+0000","last_clean":"2024-09-19T07:19:42.371671+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:42.371671+0000","last_undegraded":"2024-09-19T07:19:42.371671+0000","last_fullsized":"2024-09-19T07:19:42.371671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:47:09.376497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367120+0000","last_change":"2024-09-19T07:19:40.392973+0000","last_active":"2024-09-19T07:19:42.367120+0000","last_peered":"2024-09-19T07:19:42.367120+0000","last_clean":"2024-09-19T07:19:42.367120+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:19:42.367120+0000","last_undegraded":"2024-09-19T07:19:42.367120+0000","last_fullsized":"2024-09-19T07:19:42.367120+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368064+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:19:42.368064+0000","last_peered":"2024-09-19T07:19:42.368064+0000","last_clean":"2024-09-19T07:19:42.368064+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:19:42.368064+0000","last_undegraded":"2024-09-19T07:19:42.368064+0000","last_fullsized":"2024-09-19T07:19:42.368064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354067+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:19:41.354067+0000","last_peered":"2024-09-19T07:19:41.354067+0000","last_clean":"2024-09-19T07:19:41.354067+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:19:41.354067+0000","last_undegraded":"2024-09-19T07:19:41.354067+0000","last_fullsized":"2024-09-19T07:19:41.354067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367007+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:19:42.367007+0000","last_peered":"2024-09-19T07:19:42.367007+0000","last_clean":"2024-09-19T07:19:42.367007+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:19:42.367007+0000","last_undegraded":"2024-09-19T07:19:42.367007+0000","last_fullsized":"2024-09-19T07:19:42.367007+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354541+0000","last_change":"2024-09-19T07:19:39.429085+0000","last_active":"2024-09-19T07:19:41.354541+0000","last_peered":"2024-09-19T07:19:41.354541+0000","last_clean":"2024-09-19T07:19:41.354541+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:19:41.354541+0000","last_undegraded":"2024-09-19T07:19:41.354541+0000","last_fullsized":"2024-09-19T07:19:41.354541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348394+0000","last_change":"2024-09-19T07:19:40.382352+0000","last_active":"2024-09-19T07:19:41.348394+0000","last_peered":"2024-09-19T07:19:41.348394+0000","last_clean":"2024-09-19T07:19:41.348394+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:19:41.348394+0000","last_undegraded":"2024-09-19T07:19:41.348394+0000","last_fullsized":"2024-09-19T07:19:41.348394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:31.024388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367107+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:19:42.367107+0000","last_peered":"2024-09-19T07:19:42.367107+0000","last_clean":"2024-09-19T07:19:42.367107+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:19:42.367107+0000","last_undegraded":"2024-09-19T07:19:42.367107+0000","last_fullsized":"2024-09-19T07:19:42.367107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348973+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:19:41.348973+0000","last_peered":"2024-09-19T07:19:41.348973+0000","last_clean":"2024-09-19T07:19:41.348973+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:19:41.348973+0000","last_undegraded":"2024-09-19T07:19:41.348973+0000","last_fullsized":"2024-09-19T07:19:41.348973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353111+0000","last_change":"2024-09-19T07:19:39.427644+0000","last_active":"2024-09-19T07:19:41.353111+0000","last_peered":"2024-09-19T07:19:41.353111+0000","last_clean":"2024-09-19T07:19:41.353111+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:19:41.353111+0000","last_undegraded":"2024-09-19T07:19:41.353111+0000","last_fullsized":"2024-09-19T07:19:41.353111+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348470+0000","last_change":"2024-09-19T07:19:40.390380+0000","last_active":"2024-09-19T07:19:41.348470+0000","last_peered":"2024-09-19T07:19:41.348470+0000","last_clean":"2024-09-19T07:19:41.348470+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:19:41.348470+0000","last_undegraded":"2024-09-19T07:19:41.348470+0000","last_fullsized":"2024-09-19T07:19:41.348470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:26.385490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354180+0000","last_change":"2024-09-19T07:19:32.276600+0000","last_active":"2024-09-19T07:19:41.354180+0000","last_peered":"2024-09-19T07:19:41.354180+0000","last_clean":"2024-09-19T07:19:41.354180+0000","last_became_active":"2024-09-19T07:19:32.275766+0000","last_became_peered":"2024-09-19T07:19:32.275766+0000","last_unstale":"2024-09-19T07:19:41.354180+0000","last_undegraded":"2024-09-19T07:19:41.354180+0000","last_fullsized":"2024-09-19T07:19:41.354180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:48.506399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370499+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:19:42.370499+0000","last_peered":"2024-09-19T07:19:42.370499+0000","last_clean":"2024-09-19T07:19:42.370499+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:19:42.370499+0000","last_undegraded":"2024-09-19T07:19:42.370499+0000","last_fullsized":"2024-09-19T07:19:42.370499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366597+0000","last_change":"2024-09-19T07:19:39.410638+0000","last_active":"2024-09-19T07:19:42.366597+0000","last_peered":"2024-09-19T07:19:42.366597+0000","last_clean":"2024-09-19T07:19:42.366597+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:19:42.366597+0000","last_undegraded":"2024-09-19T07:19:42.366597+0000","last_fullsized":"2024-09-19T07:19:42.366597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:32.073136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365442+0000","last_change":"2024-09-19T07:19:40.378481+0000","last_active":"2024-09-19T07:19:42.365442+0000","last_peered":"2024-09-19T07:19:42.365442+0000","last_clean":"2024-09-19T07:19:42.365442+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:19:42.365442+0000","last_undegraded":"2024-09-19T07:19:42.365442+0000","last_fullsized":"2024-09-19T07:19:42.365442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:30.585686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369508+0000","last_change":"2024-09-19T07:19:32.292026+0000","last_active":"2024-09-19T07:19:42.369508+0000","last_peered":"2024-09-19T07:19:42.369508+0000","last_clean":"2024-09-19T07:19:42.369508+0000","last_became_active":"2024-09-19T07:19:32.291037+0000","last_became_peered":"2024-09-19T07:19:32.291037+0000","last_unstale":"2024-09-19T07:19:42.369508+0000","last_undegraded":"2024-09-19T07:19:42.369508+0000","last_fullsized":"2024-09-19T07:19:42.369508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:22.190763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349468+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:19:41.349468+0000","last_peered":"2024-09-19T07:19:41.349468+0000","last_clean":"2024-09-19T07:19:41.349468+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:19:41.349468+0000","last_undegraded":"2024-09-19T07:19:41.349468+0000","last_fullsized":"2024-09-19T07:19:41.349468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353302+0000","last_change":"2024-09-19T07:19:39.437656+0000","last_active":"2024-09-19T07:19:41.353302+0000","last_peered":"2024-09-19T07:19:41.353302+0000","last_clean":"2024-09-19T07:19:41.353302+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:19:41.353302+0000","last_undegraded":"2024-09-19T07:19:41.353302+0000","last_fullsized":"2024-09-19T07:19:41.353302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:41.468096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.347816+0000","last_change":"2024-09-19T07:19:40.379289+0000","last_active":"2024-09-19T07:19:41.347816+0000","last_peered":"2024-09-19T07:19:41.347816+0000","last_clean":"2024-09-19T07:19:41.347816+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:19:41.347816+0000","last_undegraded":"2024-09-19T07:19:41.347816+0000","last_fullsized":"2024-09-19T07:19:41.347816+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:20.399106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368075+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:19:42.368075+0000","last_peered":"2024-09-19T07:19:42.368075+0000","last_clean":"2024-09-19T07:19:42.368075+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:19:42.368075+0000","last_undegraded":"2024-09-19T07:19:42.368075+0000","last_fullsized":"2024-09-19T07:19:42.368075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368710+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:19:42.368710+0000","last_peered":"2024-09-19T07:19:42.368710+0000","last_clean":"2024-09-19T07:19:42.368710+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:19:42.368710+0000","last_undegraded":"2024-09-19T07:19:42.368710+0000","last_fullsized":"2024-09-19T07:19:42.368710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366459+0000","last_change":"2024-09-19T07:19:39.413589+0000","last_active":"2024-09-19T07:19:42.366459+0000","last_peered":"2024-09-19T07:19:42.366459+0000","last_clean":"2024-09-19T07:19:42.366459+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:19:42.366459+0000","last_undegraded":"2024-09-19T07:19:42.366459+0000","last_fullsized":"2024-09-19T07:19:42.366459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370515+0000","last_change":"2024-09-19T07:19:40.387159+0000","last_active":"2024-09-19T07:19:42.370515+0000","last_peered":"2024-09-19T07:19:42.370515+0000","last_clean":"2024-09-19T07:19:42.370515+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:19:42.370515+0000","last_undegraded":"2024-09-19T07:19:42.370515+0000","last_fullsized":"2024-09-19T07:19:42.370515+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366844+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:19:42.366844+0000","last_peered":"2024-09-19T07:19:42.366844+0000","last_clean":"2024-09-19T07:19:42.366844+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:19:42.366844+0000","last_undegraded":"2024-09-19T07:19:42.366844+0000","last_fullsized":"2024-09-19T07:19:42.366844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.347952+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:19:41.347952+0000","last_peered":"2024-09-19T07:19:41.347952+0000","last_clean":"2024-09-19T07:19:41.347952+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:19:41.347952+0000","last_undegraded":"2024-09-19T07:19:41.347952+0000","last_fullsized":"2024-09-19T07:19:41.347952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371002+0000","last_change":"2024-09-19T07:19:39.406110+0000","last_active":"2024-09-19T07:19:42.371002+0000","last_peered":"2024-09-19T07:19:42.371002+0000","last_clean":"2024-09-19T07:19:42.371002+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:19:42.371002+0000","last_undegraded":"2024-09-19T07:19:42.371002+0000","last_fullsized":"2024-09-19T07:19:42.371002+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:10:05.224225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355606+0000","last_change":"2024-09-19T07:19:40.365764+0000","last_active":"2024-09-19T07:19:41.355606+0000","last_peered":"2024-09-19T07:19:41.355606+0000","last_clean":"2024-09-19T07:19:41.355606+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:19:41.355606+0000","last_undegraded":"2024-09-19T07:19:41.355606+0000","last_fullsized":"2024-09-19T07:19:41.355606+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:04:03.242292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352283+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:19:41.352283+0000","last_peered":"2024-09-19T07:19:41.352283+0000","last_clean":"2024-09-19T07:19:41.352283+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:19:41.352283+0000","last_undegraded":"2024-09-19T07:19:41.352283+0000","last_fullsized":"2024-09-19T07:19:41.352283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353746+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:19:41.353746+0000","last_peered":"2024-09-19T07:19:41.353746+0000","last_clean":"2024-09-19T07:19:41.353746+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:19:41.353746+0000","last_undegraded":"2024-09-19T07:19:41.353746+0000","last_fullsized":"2024-09-19T07:19:41.353746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370756+0000","last_change":"2024-09-19T07:19:39.429768+0000","last_active":"2024-09-19T07:19:42.370756+0000","last_peered":"2024-09-19T07:19:42.370756+0000","last_clean":"2024-09-19T07:19:42.370756+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:19:42.370756+0000","last_undegraded":"2024-09-19T07:19:42.370756+0000","last_fullsized":"2024-09-19T07:19:42.370756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:47.503631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352914+0000","last_change":"2024-09-19T07:19:40.373542+0000","last_active":"2024-09-19T07:19:41.352914+0000","last_peered":"2024-09-19T07:19:41.352914+0000","last_clean":"2024-09-19T07:19:41.352914+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:19:41.352914+0000","last_undegraded":"2024-09-19T07:19:41.352914+0000","last_fullsized":"2024-09-19T07:19:41.352914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:14.326662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366794+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:19:42.366794+0000","last_peered":"2024-09-19T07:19:42.366794+0000","last_clean":"2024-09-19T07:19:42.366794+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:19:42.366794+0000","last_undegraded":"2024-09-19T07:19:42.366794+0000","last_fullsized":"2024-09-19T07:19:42.366794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354258+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:19:41.354258+0000","last_peered":"2024-09-19T07:19:41.354258+0000","last_clean":"2024-09-19T07:19:41.354258+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:19:41.354258+0000","last_undegraded":"2024-09-19T07:19:41.354258+0000","last_fullsized":"2024-09-19T07:19:41.354258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"24'6","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370954+0000","last_change":"2024-09-19T07:19:39.411074+0000","last_active":"2024-09-19T07:19:42.370954+0000","last_peered":"2024-09-19T07:19:42.370954+0000","last_clean":"2024-09-19T07:19:42.370954+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:19:42.370954+0000","last_undegraded":"2024-09-19T07:19:42.370954+0000","last_fullsized":"2024-09-19T07:19:42.370954+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:53.992787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370720+0000","last_change":"2024-09-19T07:19:40.364814+0000","last_active":"2024-09-19T07:19:42.370720+0000","last_peered":"2024-09-19T07:19:42.370720+0000","last_clean":"2024-09-19T07:19:42.370720+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:19:42.370720+0000","last_undegraded":"2024-09-19T07:19:42.370720+0000","last_fullsized":"2024-09-19T07:19:42.370720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366287+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:19:42.366287+0000","last_peered":"2024-09-19T07:19:42.366287+0000","last_clean":"2024-09-19T07:19:42.366287+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:19:42.366287+0000","last_undegraded":"2024-09-19T07:19:42.366287+0000","last_fullsized":"2024-09-19T07:19:42.366287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:57.131912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368746+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:19:42.368746+0000","last_peered":"2024-09-19T07:19:42.368746+0000","last_clean":"2024-09-19T07:19:42.368746+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:19:42.368746+0000","last_undegraded":"2024-09-19T07:19:42.368746+0000","last_fullsized":"2024-09-19T07:19:42.368746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:23.367365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"24'3","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355049+0000","last_change":"2024-09-19T07:19:39.431616+0000","last_active":"2024-09-19T07:19:41.355049+0000","last_peered":"2024-09-19T07:19:41.355049+0000","last_clean":"2024-09-19T07:19:41.355049+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:19:41.355049+0000","last_undegraded":"2024-09-19T07:19:41.355049+0000","last_fullsized":"2024-09-19T07:19:41.355049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:18.149581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370545+0000","last_change":"2024-09-19T07:19:40.381443+0000","last_active":"2024-09-19T07:19:42.370545+0000","last_peered":"2024-09-19T07:19:42.370545+0000","last_clean":"2024-09-19T07:19:42.370545+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:19:42.370545+0000","last_undegraded":"2024-09-19T07:19:42.370545+0000","last_fullsized":"2024-09-19T07:19:42.370545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366843+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:19:42.366843+0000","last_peered":"2024-09-19T07:19:42.366843+0000","last_clean":"2024-09-19T07:19:42.366843+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:19:42.366843+0000","last_undegraded":"2024-09-19T07:19:42.366843+0000","last_fullsized":"2024-09-19T07:19:42.366843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354497+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:19:41.354497+0000","last_peered":"2024-09-19T07:19:41.354497+0000","last_clean":"2024-09-19T07:19:41.354497+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:19:41.354497+0000","last_undegraded":"2024-09-19T07:19:41.354497+0000","last_fullsized":"2024-09-19T07:19:41.354497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367083+0000","last_change":"2024-09-19T07:19:39.423972+0000","last_active":"2024-09-19T07:19:42.367083+0000","last_peered":"2024-09-19T07:19:42.367083+0000","last_clean":"2024-09-19T07:19:42.367083+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:19:42.367083+0000","last_undegraded":"2024-09-19T07:19:42.367083+0000","last_fullsized":"2024-09-19T07:19:42.367083+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:45.387040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367684+0000","last_change":"2024-09-19T07:19:40.382092+0000","last_active":"2024-09-19T07:19:42.367684+0000","last_peered":"2024-09-19T07:19:42.367684+0000","last_clean":"2024-09-19T07:19:42.367684+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:19:42.367684+0000","last_undegraded":"2024-09-19T07:19:42.367684+0000","last_fullsized":"2024-09-19T07:19:42.367684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368384+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:19:42.368384+0000","last_peered":"2024-09-19T07:19:42.368384+0000","last_clean":"2024-09-19T07:19:42.368384+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:19:42.368384+0000","last_undegraded":"2024-09-19T07:19:42.368384+0000","last_fullsized":"2024-09-19T07:19:42.368384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370238+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:19:42.370238+0000","last_peered":"2024-09-19T07:19:42.370238+0000","last_clean":"2024-09-19T07:19:42.370238+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:19:42.370238+0000","last_undegraded":"2024-09-19T07:19:42.370238+0000","last_fullsized":"2024-09-19T07:19:42.370238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366357+0000","last_change":"2024-09-19T07:19:39.424879+0000","last_active":"2024-09-19T07:19:42.366357+0000","last_peered":"2024-09-19T07:19:42.366357+0000","last_clean":"2024-09-19T07:19:42.366357+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:19:42.366357+0000","last_undegraded":"2024-09-19T07:19:42.366357+0000","last_fullsized":"2024-09-19T07:19:42.366357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:30.021543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353343+0000","last_change":"2024-09-19T07:19:40.384136+0000","last_active":"2024-09-19T07:19:41.353343+0000","last_peered":"2024-09-19T07:19:41.353343+0000","last_clean":"2024-09-19T07:19:41.353343+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:19:41.353343+0000","last_undegraded":"2024-09-19T07:19:41.353343+0000","last_fullsized":"2024-09-19T07:19:41.353343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369727+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:19:42.369727+0000","last_peered":"2024-09-19T07:19:42.369727+0000","last_clean":"2024-09-19T07:19:42.369727+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:19:42.369727+0000","last_undegraded":"2024-09-19T07:19:42.369727+0000","last_fullsized":"2024-09-19T07:19:42.369727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:48.308137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355901+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:19:41.355901+0000","last_peered":"2024-09-19T07:19:41.355901+0000","last_clean":"2024-09-19T07:19:41.355901+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:19:41.355901+0000","last_undegraded":"2024-09-19T07:19:41.355901+0000","last_fullsized":"2024-09-19T07:19:41.355901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"24'8","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365988+0000","last_change":"2024-09-19T07:19:39.410445+0000","last_active":"2024-09-19T07:19:42.365988+0000","last_peered":"2024-09-19T07:19:42.365988+0000","last_clean":"2024-09-19T07:19:42.365988+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:19:42.365988+0000","last_undegraded":"2024-09-19T07:19:42.365988+0000","last_fullsized":"2024-09-19T07:19:42.365988+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:27.020327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365513+0000","last_change":"2024-09-19T07:19:40.389499+0000","last_active":"2024-09-19T07:19:42.365513+0000","last_peered":"2024-09-19T07:19:42.365513+0000","last_clean":"2024-09-19T07:19:42.365513+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:19:42.365513+0000","last_undegraded":"2024-09-19T07:19:42.365513+0000","last_fullsized":"2024-09-19T07:19:42.365513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.520872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355599+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:19:41.355599+0000","last_peered":"2024-09-19T07:19:41.355599+0000","last_clean":"2024-09-19T07:19:41.355599+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:19:41.355599+0000","last_undegraded":"2024-09-19T07:19:41.355599+0000","last_fullsized":"2024-09-19T07:19:41.355599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.357039+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:19:41.357039+0000","last_peered":"2024-09-19T07:19:41.357039+0000","last_clean":"2024-09-19T07:19:41.357039+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:19:41.357039+0000","last_undegraded":"2024-09-19T07:19:41.357039+0000","last_fullsized":"2024-09-19T07:19:41.357039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368271+0000","last_change":"2024-09-19T07:19:39.416943+0000","last_active":"2024-09-19T07:19:42.368271+0000","last_peered":"2024-09-19T07:19:42.368271+0000","last_clean":"2024-09-19T07:19:42.368271+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:19:42.368271+0000","last_undegraded":"2024-09-19T07:19:42.368271+0000","last_fullsized":"2024-09-19T07:19:42.368271+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:35.555826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356615+0000","last_change":"2024-09-19T07:19:40.394993+0000","last_active":"2024-09-19T07:19:41.356615+0000","last_peered":"2024-09-19T07:19:41.356615+0000","last_clean":"2024-09-19T07:19:41.356615+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:19:41.356615+0000","last_undegraded":"2024-09-19T07:19:41.356615+0000","last_fullsized":"2024-09-19T07:19:41.356615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370672+0000","last_change":"2024-09-19T07:19:39.418885+0000","last_active":"2024-09-19T07:19:42.370672+0000","last_peered":"2024-09-19T07:19:42.370672+0000","last_clean":"2024-09-19T07:19:42.370672+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:19:42.370672+0000","last_undegraded":"2024-09-19T07:19:42.370672+0000","last_fullsized":"2024-09-19T07:19:42.370672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:35:45.790411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366817+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:19:42.366817+0000","last_peered":"2024-09-19T07:19:42.366817+0000","last_clean":"2024-09-19T07:19:42.366817+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:19:42.366817+0000","last_undegraded":"2024-09-19T07:19:42.366817+0000","last_fullsized":"2024-09-19T07:19:42.366817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353835+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:19:41.353835+0000","last_peered":"2024-09-19T07:19:41.353835+0000","last_clean":"2024-09-19T07:19:41.353835+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:19:41.353835+0000","last_undegraded":"2024-09-19T07:19:41.353835+0000","last_fullsized":"2024-09-19T07:19:41.353835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370413+0000","last_change":"2024-09-19T07:19:40.374048+0000","last_active":"2024-09-19T07:19:42.370413+0000","last_peered":"2024-09-19T07:19:42.370413+0000","last_clean":"2024-09-19T07:19:42.370413+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:19:42.370413+0000","last_undegraded":"2024-09-19T07:19:42.370413+0000","last_fullsized":"2024-09-19T07:19:42.370413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:40.084827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"24'4","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356842+0000","last_change":"2024-09-19T07:19:39.432426+0000","last_active":"2024-09-19T07:19:41.356842+0000","last_peered":"2024-09-19T07:19:41.356842+0000","last_clean":"2024-09-19T07:19:41.356842+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:19:41.356842+0000","last_undegraded":"2024-09-19T07:19:41.356842+0000","last_fullsized":"2024-09-19T07:19:41.356842+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366831+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:19:42.366831+0000","last_peered":"2024-09-19T07:19:42.366831+0000","last_clean":"2024-09-19T07:19:42.366831+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:19:42.366831+0000","last_undegraded":"2024-09-19T07:19:42.366831+0000","last_fullsized":"2024-09-19T07:19:42.366831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368900+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:19:42.368900+0000","last_peered":"2024-09-19T07:19:42.368900+0000","last_clean":"2024-09-19T07:19:42.368900+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:19:42.368900+0000","last_undegraded":"2024-09-19T07:19:42.368900+0000","last_fullsized":"2024-09-19T07:19:42.368900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369632+0000","last_change":"2024-09-19T07:19:40.384934+0000","last_active":"2024-09-19T07:19:42.369632+0000","last_peered":"2024-09-19T07:19:42.369632+0000","last_clean":"2024-09-19T07:19:42.369632+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:19:42.369632+0000","last_undegraded":"2024-09-19T07:19:42.369632+0000","last_fullsized":"2024-09-19T07:19:42.369632+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353660+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:19:41.353660+0000","last_peered":"2024-09-19T07:19:41.353660+0000","last_clean":"2024-09-19T07:19:41.353660+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:19:41.353660+0000","last_undegraded":"2024-09-19T07:19:41.353660+0000","last_fullsized":"2024-09-19T07:19:41.353660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368442+0000","last_change":"2024-09-19T07:19:32.289374+0000","last_active":"2024-09-19T07:19:42.368442+0000","last_peered":"2024-09-19T07:19:42.368442+0000","last_clean":"2024-09-19T07:19:42.368442+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:19:42.368442+0000","last_undegraded":"2024-09-19T07:19:42.368442+0000","last_fullsized":"2024-09-19T07:19:42.368442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348715+0000","last_change":"2024-09-19T07:19:39.410582+0000","last_active":"2024-09-19T07:19:41.348715+0000","last_peered":"2024-09-19T07:19:41.348715+0000","last_clean":"2024-09-19T07:19:41.348715+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:19:41.348715+0000","last_undegraded":"2024-09-19T07:19:41.348715+0000","last_fullsized":"2024-09-19T07:19:41.348715+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:41:40.592907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367452+0000","last_change":"2024-09-19T07:19:40.393950+0000","last_active":"2024-09-19T07:19:42.367452+0000","last_peered":"2024-09-19T07:19:42.367452+0000","last_clean":"2024-09-19T07:19:42.367452+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:19:42.367452+0000","last_undegraded":"2024-09-19T07:19:42.367452+0000","last_fullsized":"2024-09-19T07:19:42.367452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371158+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:19:42.371158+0000","last_peered":"2024-09-19T07:19:42.371158+0000","last_clean":"2024-09-19T07:19:42.371158+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:19:42.371158+0000","last_undegraded":"2024-09-19T07:19:42.371158+0000","last_fullsized":"2024-09-19T07:19:42.371158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366780+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:19:42.366780+0000","last_peered":"2024-09-19T07:19:42.366780+0000","last_clean":"2024-09-19T07:19:42.366780+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:19:42.366780+0000","last_undegraded":"2024-09-19T07:19:42.366780+0000","last_fullsized":"2024-09-19T07:19:42.366780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371594+0000","last_change":"2024-09-19T07:19:39.410662+0000","last_active":"2024-09-19T07:19:42.371594+0000","last_peered":"2024-09-19T07:19:42.371594+0000","last_clean":"2024-09-19T07:19:42.371594+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:42.371594+0000","last_undegraded":"2024-09-19T07:19:42.371594+0000","last_fullsized":"2024-09-19T07:19:42.371594+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:19.196074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372306+0000","last_change":"2024-09-19T07:19:40.385581+0000","last_active":"2024-09-19T07:19:42.372306+0000","last_peered":"2024-09-19T07:19:42.372306+0000","last_clean":"2024-09-19T07:19:42.372306+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:19:42.372306+0000","last_undegraded":"2024-09-19T07:19:42.372306+0000","last_fullsized":"2024-09-19T07:19:42.372306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:14:13.749729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353359+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:19:41.353359+0000","last_peered":"2024-09-19T07:19:41.353359+0000","last_clean":"2024-09-19T07:19:41.353359+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:19:41.353359+0000","last_undegraded":"2024-09-19T07:19:41.353359+0000","last_fullsized":"2024-09-19T07:19:41.353359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369574+0000","last_change":"2024-09-19T07:19:40.377608+0000","last_active":"2024-09-19T07:19:42.369574+0000","last_peered":"2024-09-19T07:19:42.369574+0000","last_clean":"2024-09-19T07:19:42.369574+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:19:42.369574+0000","last_undegraded":"2024-09-19T07:19:42.369574+0000","last_fullsized":"2024-09-19T07:19:42.369574+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348053+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:19:41.348053+0000","last_peered":"2024-09-19T07:19:41.348053+0000","last_clean":"2024-09-19T07:19:41.348053+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:19:41.348053+0000","last_undegraded":"2024-09-19T07:19:41.348053+0000","last_fullsized":"2024-09-19T07:19:41.348053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353277+0000","last_change":"2024-09-19T07:19:40.381262+0000","last_active":"2024-09-19T07:19:41.353277+0000","last_peered":"2024-09-19T07:19:41.353277+0000","last_clean":"2024-09-19T07:19:41.353277+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:19:41.353277+0000","last_undegraded":"2024-09-19T07:19:41.353277+0000","last_fullsized":"2024-09-19T07:19:41.353277+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:34.425472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354018+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:19:41.354018+0000","last_peered":"2024-09-19T07:19:41.354018+0000","last_clean":"2024-09-19T07:19:41.354018+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:19:41.354018+0000","last_undegraded":"2024-09-19T07:19:41.354018+0000","last_fullsized":"2024-09-19T07:19:41.354018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369264+0000","last_change":"2024-09-19T07:19:40.386432+0000","last_active":"2024-09-19T07:19:42.369264+0000","last_peered":"2024-09-19T07:19:42.369264+0000","last_clean":"2024-09-19T07:19:42.369264+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:19:42.369264+0000","last_undegraded":"2024-09-19T07:19:42.369264+0000","last_fullsized":"2024-09-19T07:19:42.369264+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368214+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:19:42.368214+0000","last_peered":"2024-09-19T07:19:42.368214+0000","last_clean":"2024-09-19T07:19:42.368214+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:19:42.368214+0000","last_undegraded":"2024-09-19T07:19:42.368214+0000","last_fullsized":"2024-09-19T07:19:42.368214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365212+0000","last_change":"2024-09-19T07:19:40.378630+0000","last_active":"2024-09-19T07:19:42.365212+0000","last_peered":"2024-09-19T07:19:42.365212+0000","last_clean":"2024-09-19T07:19:42.365212+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:19:42.365212+0000","last_undegraded":"2024-09-19T07:19:42.365212+0000","last_fullsized":"2024-09-19T07:19:42.365212+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371622+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:19:42.371622+0000","last_peered":"2024-09-19T07:19:42.371622+0000","last_clean":"2024-09-19T07:19:42.371622+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:19:42.371622+0000","last_undegraded":"2024-09-19T07:19:42.371622+0000","last_fullsized":"2024-09-19T07:19:42.371622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365853+0000","last_change":"2024-09-19T07:19:40.381365+0000","last_active":"2024-09-19T07:19:42.365853+0000","last_peered":"2024-09-19T07:19:42.365853+0000","last_clean":"2024-09-19T07:19:42.365853+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:19:42.365853+0000","last_undegraded":"2024-09-19T07:19:42.365853+0000","last_fullsized":"2024-09-19T07:19:42.365853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370004+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:19:42.370004+0000","last_peered":"2024-09-19T07:19:42.370004+0000","last_clean":"2024-09-19T07:19:42.370004+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:19:42.370004+0000","last_undegraded":"2024-09-19T07:19:42.370004+0000","last_fullsized":"2024-09-19T07:19:42.370004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365148+0000","last_change":"2024-09-19T07:19:40.387116+0000","last_active":"2024-09-19T07:19:42.365148+0000","last_peered":"2024-09-19T07:19:42.365148+0000","last_clean":"2024-09-19T07:19:42.365148+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:19:42.365148+0000","last_undegraded":"2024-09-19T07:19:42.365148+0000","last_fullsized":"2024-09-19T07:19:42.365148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356605+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:19:41.356605+0000","last_peered":"2024-09-19T07:19:41.356605+0000","last_clean":"2024-09-19T07:19:41.356605+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:19:41.356605+0000","last_undegraded":"2024-09-19T07:19:41.356605+0000","last_fullsized":"2024-09-19T07:19:41.356605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352940+0000","last_change":"2024-09-19T07:19:40.357629+0000","last_active":"2024-09-19T07:19:41.352940+0000","last_peered":"2024-09-19T07:19:41.352940+0000","last_clean":"2024-09-19T07:19:41.352940+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:19:41.352940+0000","last_undegraded":"2024-09-19T07:19:41.352940+0000","last_fullsized":"2024-09-19T07:19:41.352940+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371018+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:19:42.371018+0000","last_peered":"2024-09-19T07:19:42.371018+0000","last_clean":"2024-09-19T07:19:42.371018+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:19:42.371018+0000","last_undegraded":"2024-09-19T07:19:42.371018+0000","last_fullsized":"2024-09-19T07:19:42.371018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371827+0000","last_change":"2024-09-19T07:19:40.388597+0000","last_active":"2024-09-19T07:19:42.371827+0000","last_peered":"2024-09-19T07:19:42.371827+0000","last_clean":"2024-09-19T07:19:42.371827+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:19:42.371827+0000","last_undegraded":"2024-09-19T07:19:42.371827+0000","last_fullsized":"2024-09-19T07:19:42.371827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:34:18.402016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356021+0000","last_change":"2024-09-19T07:19:32.273635+0000","last_active":"2024-09-19T07:19:41.356021+0000","last_peered":"2024-09-19T07:19:41.356021+0000","last_clean":"2024-09-19T07:19:41.356021+0000","last_became_active":"2024-09-19T07:19:32.271748+0000","last_became_peered":"2024-09-19T07:19:32.271748+0000","last_unstale":"2024-09-19T07:19:41.356021+0000","last_undegraded":"2024-09-19T07:19:41.356021+0000","last_fullsized":"2024-09-19T07:19:41.356021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:49:41.724626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369047+0000","last_change":"2024-09-19T07:19:40.383779+0000","last_active":"2024-09-19T07:19:42.369047+0000","last_peered":"2024-09-19T07:19:42.369047+0000","last_clean":"2024-09-19T07:19:42.369047+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:19:42.369047+0000","last_undegraded":"2024-09-19T07:19:42.369047+0000","last_fullsized":"2024-09-19T07:19:42.369047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368684+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:19:42.368684+0000","last_peered":"2024-09-19T07:19:42.368684+0000","last_clean":"2024-09-19T07:19:42.368684+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:19:42.368684+0000","last_undegraded":"2024-09-19T07:19:42.368684+0000","last_fullsized":"2024-09-19T07:19:42.368684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348737+0000","last_change":"2024-09-19T07:19:40.376342+0000","last_active":"2024-09-19T07:19:41.348737+0000","last_peered":"2024-09-19T07:19:41.348737+0000","last_clean":"2024-09-19T07:19:41.348737+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:19:41.348737+0000","last_undegraded":"2024-09-19T07:19:41.348737+0000","last_fullsized":"2024-09-19T07:19:41.348737+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:22.997744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353771+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:19:41.353771+0000","last_peered":"2024-09-19T07:19:41.353771+0000","last_clean":"2024-09-19T07:19:41.353771+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:19:41.353771+0000","last_undegraded":"2024-09-19T07:19:41.353771+0000","last_fullsized":"2024-09-19T07:19:41.353771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371199+0000","last_change":"2024-09-19T07:19:40.361504+0000","last_active":"2024-09-19T07:19:42.371199+0000","last_peered":"2024-09-19T07:19:42.371199+0000","last_clean":"2024-09-19T07:19:42.371199+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:19:42.371199+0000","last_undegraded":"2024-09-19T07:19:42.371199+0000","last_fullsized":"2024-09-19T07:19:42.371199+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:30:31.110669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367582+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:19:42.367582+0000","last_peered":"2024-09-19T07:19:42.367582+0000","last_clean":"2024-09-19T07:19:42.367582+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:19:42.367582+0000","last_undegraded":"2024-09-19T07:19:42.367582+0000","last_fullsized":"2024-09-19T07:19:42.367582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:19.033192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365909+0000","last_change":"2024-09-19T07:19:40.390205+0000","last_active":"2024-09-19T07:19:42.365909+0000","last_peered":"2024-09-19T07:19:42.365909+0000","last_clean":"2024-09-19T07:19:42.365909+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:19:42.365909+0000","last_undegraded":"2024-09-19T07:19:42.365909+0000","last_fullsized":"2024-09-19T07:19:42.365909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354180+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:19:41.354180+0000","last_peered":"2024-09-19T07:19:41.354180+0000","last_clean":"2024-09-19T07:19:41.354180+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:19:41.354180+0000","last_undegraded":"2024-09-19T07:19:41.354180+0000","last_fullsized":"2024-09-19T07:19:41.354180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365447+0000","last_change":"2024-09-19T07:19:40.390559+0000","last_active":"2024-09-19T07:19:42.365447+0000","last_peered":"2024-09-19T07:19:42.365447+0000","last_clean":"2024-09-19T07:19:42.365447+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:19:42.365447+0000","last_undegraded":"2024-09-19T07:19:42.365447+0000","last_fullsized":"2024-09-19T07:19:42.365447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367349+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:19:42.367349+0000","last_peered":"2024-09-19T07:19:42.367349+0000","last_clean":"2024-09-19T07:19:42.367349+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:19:42.367349+0000","last_undegraded":"2024-09-19T07:19:42.367349+0000","last_fullsized":"2024-09-19T07:19:42.367349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352820+0000","last_change":"2024-09-19T07:19:40.358433+0000","last_active":"2024-09-19T07:19:41.352820+0000","last_peered":"2024-09-19T07:19:41.352820+0000","last_clean":"2024-09-19T07:19:41.352820+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:19:41.352820+0000","last_undegraded":"2024-09-19T07:19:41.352820+0000","last_fullsized":"2024-09-19T07:19:41.352820+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:10:25.226677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349337+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:19:41.349337+0000","last_peered":"2024-09-19T07:19:41.349337+0000","last_clean":"2024-09-19T07:19:41.349337+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:19:41.349337+0000","last_undegraded":"2024-09-19T07:19:41.349337+0000","last_fullsized":"2024-09-19T07:19:41.349337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:51.710997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365454+0000","last_change":"2024-09-19T07:19:40.374374+0000","last_active":"2024-09-19T07:19:42.365454+0000","last_peered":"2024-09-19T07:19:42.365454+0000","last_clean":"2024-09-19T07:19:42.365454+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:19:42.365454+0000","last_undegraded":"2024-09-19T07:19:42.365454+0000","last_fullsized":"2024-09-19T07:19:42.365454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:34:05.875388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356952+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:19:41.356952+0000","last_peered":"2024-09-19T07:19:41.356952+0000","last_clean":"2024-09-19T07:19:41.356952+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:19:41.356952+0000","last_undegraded":"2024-09-19T07:19:41.356952+0000","last_fullsized":"2024-09-19T07:19:41.356952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368071+0000","last_change":"2024-09-19T07:19:40.387341+0000","last_active":"2024-09-19T07:19:42.368071+0000","last_peered":"2024-09-19T07:19:42.368071+0000","last_clean":"2024-09-19T07:19:42.368071+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:19:42.368071+0000","last_undegraded":"2024-09-19T07:19:42.368071+0000","last_fullsized":"2024-09-19T07:19:42.368071+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:56:40.928844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353563+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:19:41.353563+0000","last_peered":"2024-09-19T07:19:41.353563+0000","last_clean":"2024-09-19T07:19:41.353563+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:19:41.353563+0000","last_undegraded":"2024-09-19T07:19:41.353563+0000","last_fullsized":"2024-09-19T07:19:41.353563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368148+0000","last_change":"2024-09-19T07:19:40.388484+0000","last_active":"2024-09-19T07:19:42.368148+0000","last_peered":"2024-09-19T07:19:42.368148+0000","last_clean":"2024-09-19T07:19:42.368148+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:19:42.368148+0000","last_undegraded":"2024-09-19T07:19:42.368148+0000","last_fullsized":"2024-09-19T07:19:42.368148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366974+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:19:42.366974+0000","last_peered":"2024-09-19T07:19:42.366974+0000","last_clean":"2024-09-19T07:19:42.366974+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:19:42.366974+0000","last_undegraded":"2024-09-19T07:19:42.366974+0000","last_fullsized":"2024-09-19T07:19:42.366974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352836+0000","last_change":"2024-09-19T07:19:40.367578+0000","last_active":"2024-09-19T07:19:41.352836+0000","last_peered":"2024-09-19T07:19:41.352836+0000","last_clean":"2024-09-19T07:19:41.352836+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:19:41.352836+0000","last_undegraded":"2024-09-19T07:19:41.352836+0000","last_fullsized":"2024-09-19T07:19:41.352836+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:00:25.357854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366048+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:19:42.366048+0000","last_peered":"2024-09-19T07:19:42.366048+0000","last_clean":"2024-09-19T07:19:42.366048+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:19:42.366048+0000","last_undegraded":"2024-09-19T07:19:42.366048+0000","last_fullsized":"2024-09-19T07:19:42.366048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:43.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366598+0000","last_change":"2024-09-19T07:19:40.385641+0000","last_active":"2024-09-19T07:19:42.366598+0000","last_peered":"2024-09-19T07:19:42.366598+0000","last_clean":"2024-09-19T07:19:42.366598+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:19:42.366598+0000","last_undegraded":"2024-09-19T07:19:42.366598+0000","last_fullsized":"2024-09-19T07:19:42.366598+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370694+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:19:42.370694+0000","last_peered":"2024-09-19T07:19:42.370694+0000","last_clean":"2024-09-19T07:19:42.370694+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:19:42.370694+0000","last_undegraded":"2024-09-19T07:19:42.370694+0000","last_fullsized":"2024-09-19T07:19:42.370694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369737+0000","last_change":"2024-09-19T07:19:40.367101+0000","last_active":"2024-09-19T07:19:42.369737+0000","last_peered":"2024-09-19T07:19:42.369737+0000","last_clean":"2024-09-19T07:19:42.369737+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:19:42.369737+0000","last_undegraded":"2024-09-19T07:19:42.369737+0000","last_fullsized":"2024-09-19T07:19:42.369737+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368379+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:19:42.368379+0000","last_peered":"2024-09-19T07:19:42.368379+0000","last_clean":"2024-09-19T07:19:42.368379+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:19:42.368379+0000","last_undegraded":"2024-09-19T07:19:42.368379+0000","last_fullsized":"2024-09-19T07:19:42.368379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348413+0000","last_change":"2024-09-19T07:19:40.384212+0000","last_active":"2024-09-19T07:19:41.348413+0000","last_peered":"2024-09-19T07:19:41.348413+0000","last_clean":"2024-09-19T07:19:41.348413+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:19:41.348413+0000","last_undegraded":"2024-09-19T07:19:41.348413+0000","last_fullsized":"2024-09-19T07:19:41.348413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:57.724404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355987+0000","last_change":"2024-09-19T07:19:32.299956+0000","last_active":"2024-09-19T07:19:41.355987+0000","last_peered":"2024-09-19T07:19:41.355987+0000","last_clean":"2024-09-19T07:19:41.355987+0000","last_became_active":"2024-09-19T07:19:32.299168+0000","last_became_peered":"2024-09-19T07:19:32.299168+0000","last_unstale":"2024-09-19T07:19:41.355987+0000","last_undegraded":"2024-09-19T07:19:41.355987+0000","last_fullsized":"2024-09-19T07:19:41.355987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366761+0000","last_change":"2024-09-19T07:19:40.387401+0000","last_active":"2024-09-19T07:19:42.366761+0000","last_peered":"2024-09-19T07:19:42.366761+0000","last_clean":"2024-09-19T07:19:42.366761+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:19:42.366761+0000","last_undegraded":"2024-09-19T07:19:42.366761+0000","last_fullsized":"2024-09-19T07:19:42.366761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356423+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:19:41.356423+0000","last_peered":"2024-09-19T07:19:41.356423+0000","last_clean":"2024-09-19T07:19:41.356423+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:19:41.356423+0000","last_undegraded":"2024-09-19T07:19:41.356423+0000","last_fullsized":"2024-09-19T07:19:41.356423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352920+0000","last_change":"2024-09-19T07:19:40.361430+0000","last_active":"2024-09-19T07:19:41.352920+0000","last_peered":"2024-09-19T07:19:41.352920+0000","last_clean":"2024-09-19T07:19:41.352920+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:19:41.352920+0000","last_undegraded":"2024-09-19T07:19:41.352920+0000","last_fullsized":"2024-09-19T07:19:41.352920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:23:05.185943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369329+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:19:42.369329+0000","last_peered":"2024-09-19T07:19:42.369329+0000","last_clean":"2024-09-19T07:19:42.369329+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:19:42.369329+0000","last_undegraded":"2024-09-19T07:19:42.369329+0000","last_fullsized":"2024-09-19T07:19:42.369329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371825+0000","last_change":"2024-09-19T07:19:40.378172+0000","last_active":"2024-09-19T07:19:42.371825+0000","last_peered":"2024-09-19T07:19:42.371825+0000","last_clean":"2024-09-19T07:19:42.371825+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:19:42.371825+0000","last_undegraded":"2024-09-19T07:19:42.371825+0000","last_fullsized":"2024-09-19T07:19:42.371825+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:52.321790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366422+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:19:42.366422+0000","last_peered":"2024-09-19T07:19:42.366422+0000","last_clean":"2024-09-19T07:19:42.366422+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:19:42.366422+0000","last_undegraded":"2024-09-19T07:19:42.366422+0000","last_fullsized":"2024-09-19T07:19:42.366422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356589+0000","last_change":"2024-09-19T07:19:40.373909+0000","last_active":"2024-09-19T07:19:41.356589+0000","last_peered":"2024-09-19T07:19:41.356589+0000","last_clean":"2024-09-19T07:19:41.356589+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:19:41.356589+0000","last_undegraded":"2024-09-19T07:19:41.356589+0000","last_fullsized":"2024-09-19T07:19:41.356589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366411+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:19:42.366411+0000","last_peered":"2024-09-19T07:19:42.366411+0000","last_clean":"2024-09-19T07:19:42.366411+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:19:42.366411+0000","last_undegraded":"2024-09-19T07:19:42.366411+0000","last_fullsized":"2024-09-19T07:19:42.366411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367511+0000","last_change":"2024-09-19T07:19:40.396146+0000","last_active":"2024-09-19T07:19:42.367511+0000","last_peered":"2024-09-19T07:19:42.367511+0000","last_clean":"2024-09-19T07:19:42.367511+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:19:42.367511+0000","last_undegraded":"2024-09-19T07:19:42.367511+0000","last_fullsized":"2024-09-19T07:19:42.367511+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353532+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:19:41.353532+0000","last_peered":"2024-09-19T07:19:41.353532+0000","last_clean":"2024-09-19T07:19:41.353532+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:19:41.353532+0000","last_undegraded":"2024-09-19T07:19:41.353532+0000","last_fullsized":"2024-09-19T07:19:41.353532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355009+0000","last_change":"2024-09-19T07:19:40.373552+0000","last_active":"2024-09-19T07:19:41.355009+0000","last_peered":"2024-09-19T07:19:41.355009+0000","last_clean":"2024-09-19T07:19:41.355009+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:19:41.355009+0000","last_undegraded":"2024-09-19T07:19:41.355009+0000","last_fullsized":"2024-09-19T07:19:41.355009+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:23:33.629831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366537+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:19:42.366537+0000","last_peered":"2024-09-19T07:19:42.366537+0000","last_clean":"2024-09-19T07:19:42.366537+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:19:42.366537+0000","last_undegraded":"2024-09-19T07:19:42.366537+0000","last_fullsized":"2024-09-19T07:19:42.366537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367996+0000","last_change":"2024-09-19T07:19:40.356989+0000","last_active":"2024-09-19T07:19:42.367996+0000","last_peered":"2024-09-19T07:19:42.367996+0000","last_clean":"2024-09-19T07:19:42.367996+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:19:42.367996+0000","last_undegraded":"2024-09-19T07:19:42.367996+0000","last_fullsized":"2024-09-19T07:19:42.367996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353682+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:19:41.353682+0000","last_peered":"2024-09-19T07:19:41.353682+0000","last_clean":"2024-09-19T07:19:41.353682+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:19:41.353682+0000","last_undegraded":"2024-09-19T07:19:41.353682+0000","last_fullsized":"2024-09-19T07:19:41.353682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348160+0000","last_change":"2024-09-19T07:19:40.376487+0000","last_active":"2024-09-19T07:19:41.348160+0000","last_peered":"2024-09-19T07:19:41.348160+0000","last_clean":"2024-09-19T07:19:41.348160+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:19:41.348160+0000","last_undegraded":"2024-09-19T07:19:41.348160+0000","last_fullsized":"2024-09-19T07:19:41.348160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:05.185451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369628+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:19:42.369628+0000","last_peered":"2024-09-19T07:19:42.369628+0000","last_clean":"2024-09-19T07:19:42.369628+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:19:42.369628+0000","last_undegraded":"2024-09-19T07:19:42.369628+0000","last_fullsized":"2024-09-19T07:19:42.369628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:25.667168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364909+0000","last_change":"2024-09-19T07:19:40.377604+0000","last_active":"2024-09-19T07:19:42.364909+0000","last_peered":"2024-09-19T07:19:42.364909+0000","last_clean":"2024-09-19T07:19:42.364909+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:19:42.364909+0000","last_undegraded":"2024-09-19T07:19:42.364909+0000","last_fullsized":"2024-09-19T07:19:42.364909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368557+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:19:42.368557+0000","last_peered":"2024-09-19T07:19:42.368557+0000","last_clean":"2024-09-19T07:19:42.368557+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:19:42.368557+0000","last_undegraded":"2024-09-19T07:19:42.368557+0000","last_fullsized":"2024-09-19T07:19:42.368557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365847+0000","last_change":"2024-09-19T07:19:40.377580+0000","last_active":"2024-09-19T07:19:42.365847+0000","last_peered":"2024-09-19T07:19:42.365847+0000","last_clean":"2024-09-19T07:19:42.365847+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:19:42.365847+0000","last_undegraded":"2024-09-19T07:19:42.365847+0000","last_fullsized":"2024-09-19T07:19:42.365847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369285+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:19:42.369285+0000","last_peered":"2024-09-19T07:19:42.369285+0000","last_clean":"2024-09-19T07:19:42.369285+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:19:42.369285+0000","last_undegraded":"2024-09-19T07:19:42.369285+0000","last_fullsized":"2024-09-19T07:19:42.369285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":864256,"data_stored":693530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":282624,"data_stored":104718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":103252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":282624,"data_stored":102720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":290816,"data_stored":105736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":102560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542153,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":286720,"data_stored":104284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":868352,"data_stored":693496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:19:51.299 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:19:51.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.447+0000 7f115bf5b640 1 Processor -- start 2024-09-19T07:19:51.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.447+0000 7f115bf5b640 1 -- start start 2024-09-19T07:19:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1154150290 0x7f11540c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1154152b20 0x7f11540c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1154150670 con 0x7f11540c45c0 2024-09-19T07:19:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f11540716c0 con 0x7f1154152b20 2024-09-19T07:19:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f11540c3920 con 0x7f1154150290 2024-09-19T07:19:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f11594f6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1154152b20 0x7f11540c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f11594f6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1154152b20 0x7f11540c4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54148/0 (socket says 172.21.15.83:54148) 2024-09-19T07:19:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f11594f6640 1 -- 172.21.15.83:0/3499522083 learned_addr learned my addr 172.21.15.83:0/3499522083 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115a4f8640 1 --1- 172.21.15.83:0/3499522083 >> v1:172.21.15.83:6789/0 conn(0x7f11540c45c0 0x7f1154153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35520/0 (socket says 172.21.15.83:35520) 2024-09-19T07:19:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f1159cf7640 1 --2- 172.21.15.83:0/3499522083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1154150290 0x7f11540c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f11594f6640 1 -- 172.21.15.83:0/3499522083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1154150290 msgr2=0x7f11540c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:51.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f11594f6640 1 --2- 172.21.15.83:0/3499522083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1154150290 0x7f11540c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f11594f6640 1 -- 172.21.15.83:0/3499522083 >> v1:172.21.15.83:6789/0 conn(0x7f11540c45c0 legacy=0x7f1154153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f11594f6640 1 -- 172.21.15.83:0/3499522083 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11540c2880 con 0x7f1154152b20 2024-09-19T07:19:51.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f1158cf5640 1 -- 172.21.15.83:0/3499522083 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3233026289 0 0) 0x7f1154150670 con 0x7f11540c45c0 2024-09-19T07:19:51.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f11594f6640 1 --2- 172.21.15.83:0/3499522083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1154152b20 0x7f11540c4080 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f114800bb40 tx=0x7f114801ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af9b20b324ace27 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f1158cf5640 1 -- 172.21.15.83:0/3499522083 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11480b4070 con 0x7f1154152b20 2024-09-19T07:19:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f1158cf5640 1 -- 172.21.15.83:0/3499522083 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11480a7af0 con 0x7f1154152b20 2024-09-19T07:19:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f1158cf5640 1 -- 172.21.15.83:0/3499522083 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11480af440 con 0x7f1154152b20 2024-09-19T07:19:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 -- 172.21.15.83:0/3499522083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1154152b20 msgr2=0x7f11540c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 --2- 172.21.15.83:0/3499522083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1154152b20 0x7f11540c4080 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f114800bb40 tx=0x7f114801ec10 comp rx=0 tx=0).stop 2024-09-19T07:19:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 -- 172.21.15.83:0/3499522083 shutdown_connections 2024-09-19T07:19:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 --2- 172.21.15.83:0/3499522083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1154152b20 0x7f11540c4080 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 --2- 172.21.15.83:0/3499522083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1154150290 0x7f11540c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 -- 172.21.15.83:0/3499522083 >> 172.21.15.83:0/3499522083 conn(0x7f115406bc80 msgr2=0x7f115406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 -- 172.21.15.83:0/3499522083 shutdown_connections 2024-09-19T07:19:51.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.451+0000 7f115bf5b640 1 -- 172.21.15.83:0/3499522083 wait complete. 2024-09-19T07:19:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f115bf5b640 1 Processor -- start 2024-09-19T07:19:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f115bf5b640 1 -- start start 2024-09-19T07:19:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f115bf5b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11540c45c0 0x7f1154144c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f115bf5b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1154150290 0x7f1154149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f115bf5b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1154152b20 0x7f11540c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f115bf5b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f11540c2530 con 0x7f11540c45c0 2024-09-19T07:19:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f115bf5b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f11540c3120 con 0x7f1154150290 2024-09-19T07:19:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f115bf5b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f115406e6b0 con 0x7f1154152b20 2024-09-19T07:19:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f1159cf7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11540c45c0 0x7f1154144c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f1159cf7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11540c45c0 0x7f1154144c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54162/0 (socket says 172.21.15.83:54162) 2024-09-19T07:19:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f1159cf7640 1 -- 172.21.15.83:0/2118675042 learned_addr learned my addr 172.21.15.83:0/2118675042 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f1159cf7640 1 -- 172.21.15.83:0/2118675042 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1154152b20 msgr2=0x7f11540c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f1159cf7640 1 --2- 172.21.15.83:0/2118675042 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1154152b20 0x7f11540c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f1159cf7640 1 -- 172.21.15.83:0/2118675042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1154150290 msgr2=0x7f1154149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f1159cf7640 1 --2- 172.21.15.83:0/2118675042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1154150290 0x7f1154149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f1159cf7640 1 -- 172.21.15.83:0/2118675042 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1154145140 con 0x7f11540c45c0 2024-09-19T07:19:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f1159cf7640 1 --2- 172.21.15.83:0/2118675042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11540c45c0 0x7f1154144c00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f114c00ce40 tx=0x7f114c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f11527fc640 1 -- 172.21.15.83:0/2118675042 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c003ba0 con 0x7f11540c45c0 2024-09-19T07:19:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f11527fc640 1 -- 172.21.15.83:0/2118675042 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f114c003d40 con 0x7f11540c45c0 2024-09-19T07:19:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f11527fc640 1 -- 172.21.15.83:0/2118675042 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c004240 con 0x7f11540c45c0 2024-09-19T07:19:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f115bf5b640 1 -- 172.21.15.83:0/2118675042 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f115414f4a0 con 0x7f11540c45c0 2024-09-19T07:19:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f115bf5b640 1 -- 172.21.15.83:0/2118675042 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f115414f740 con 0x7f11540c45c0 2024-09-19T07:19:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f115bf5b640 1 -- 172.21.15.83:0/2118675042 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f115406ea70 con 0x7f11540c45c0 2024-09-19T07:19:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f11527fc640 1 -- 172.21.15.83:0/2118675042 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f114c0043e0 con 0x7f11540c45c0 2024-09-19T07:19:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f11527fc640 1 --2- 172.21.15.83:0/2118675042 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f11400678c0 0x7f1140069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f11594f6640 1 --2- 172.21.15.83:0/2118675042 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f11400678c0 0x7f1140069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.455+0000 7f11527fc640 1 -- 172.21.15.83:0/2118675042 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f114c086a60 con 0x7f11540c45c0 2024-09-19T07:19:51.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.459+0000 7f11594f6640 1 --2- 172.21.15.83:0/2118675042 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f11400678c0 0x7f1140069d80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f11480b1000 tx=0x7f1148004740 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:51.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.463+0000 7f11527fc640 1 -- 172.21.15.83:0/2118675042 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f114c057300 con 0x7f11540c45c0 2024-09-19T07:19:51.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.607+0000 7f115bf5b640 1 -- 172.21.15.83:0/2118675042 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1154074060 con 0x7f11400678c0 2024-09-19T07:19:51.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.623+0000 7f11527fc640 1 -- 172.21.15.83:0/2118675042 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490308 (secure 0 0 0) 0x7f1154074060 con 0x7f11400678c0 2024-09-19T07:19:51.628 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:51.633 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:19:51.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.631+0000 7f115bf5b640 1 -- 172.21.15.83:0/2118675042 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f11400678c0 msgr2=0x7f1140069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.631+0000 7f115bf5b640 1 --2- 172.21.15.83:0/2118675042 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f11400678c0 0x7f1140069d80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f11480b1000 tx=0x7f1148004740 comp rx=0 tx=0).stop 2024-09-19T07:19:51.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.631+0000 7f115bf5b640 1 -- 172.21.15.83:0/2118675042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11540c45c0 msgr2=0x7f1154144c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.631+0000 7f115bf5b640 1 --2- 172.21.15.83:0/2118675042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11540c45c0 0x7f1154144c00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f114c00ce40 tx=0x7f114c00a660 comp rx=0 tx=0).stop 2024-09-19T07:19:51.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.631+0000 7f115bf5b640 1 -- 172.21.15.83:0/2118675042 shutdown_connections 2024-09-19T07:19:51.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.631+0000 7f115bf5b640 1 --2- 172.21.15.83:0/2118675042 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f11400678c0 0x7f1140069d80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.631+0000 7f115bf5b640 1 --2- 172.21.15.83:0/2118675042 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1154152b20 0x7f11540c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.631+0000 7f115bf5b640 1 --2- 172.21.15.83:0/2118675042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1154150290 0x7f1154149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.631+0000 7f115bf5b640 1 --2- 172.21.15.83:0/2118675042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11540c45c0 0x7f1154144c00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.631+0000 7f115bf5b640 1 -- 172.21.15.83:0/2118675042 >> 172.21.15.83:0/2118675042 conn(0x7f115406bc80 msgr2=0x7f115406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:51.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.631+0000 7f115bf5b640 1 -- 172.21.15.83:0/2118675042 shutdown_connections 2024-09-19T07:19:51.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.631+0000 7f115bf5b640 1 -- 172.21.15.83:0/2118675042 wait complete. 2024-09-19T07:19:51.646 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-19T07:19:50.816657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223748,"kb_used_data":3336,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750972,"statfs":{"total":773094113280,"available":772864995328,"internally_reserved":0,"allocated":3416064,"data_stored":2010296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003296"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364888+0000","last_change":"2024-09-19T07:19:40.358457+0000","last_active":"2024-09-19T07:19:42.364888+0000","last_peered":"2024-09-19T07:19:42.364888+0000","last_clean":"2024-09-19T07:19:42.364888+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:19:42.364888+0000","last_undegraded":"2024-09-19T07:19:42.364888+0000","last_fullsized":"2024-09-19T07:19:42.364888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370195+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:19:42.370195+0000","last_peered":"2024-09-19T07:19:42.370195+0000","last_clean":"2024-09-19T07:19:42.370195+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:19:42.370195+0000","last_undegraded":"2024-09-19T07:19:42.370195+0000","last_fullsized":"2024-09-19T07:19:42.370195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:57.799442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372168+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:19:42.372168+0000","last_peered":"2024-09-19T07:19:42.372168+0000","last_clean":"2024-09-19T07:19:42.372168+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:19:42.372168+0000","last_undegraded":"2024-09-19T07:19:42.372168+0000","last_fullsized":"2024-09-19T07:19:42.372168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368511+0000","last_change":"2024-09-19T07:19:39.401621+0000","last_active":"2024-09-19T07:19:42.368511+0000","last_peered":"2024-09-19T07:19:42.368511+0000","last_clean":"2024-09-19T07:19:42.368511+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:19:42.368511+0000","last_undegraded":"2024-09-19T07:19:42.368511+0000","last_fullsized":"2024-09-19T07:19:42.368511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:32.095381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366428+0000","last_change":"2024-09-19T07:19:40.366073+0000","last_active":"2024-09-19T07:19:42.366428+0000","last_peered":"2024-09-19T07:19:42.366428+0000","last_clean":"2024-09-19T07:19:42.366428+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:19:42.366428+0000","last_undegraded":"2024-09-19T07:19:42.366428+0000","last_fullsized":"2024-09-19T07:19:42.366428+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372588+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:19:42.372588+0000","last_peered":"2024-09-19T07:19:42.372588+0000","last_clean":"2024-09-19T07:19:42.372588+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:19:42.372588+0000","last_undegraded":"2024-09-19T07:19:42.372588+0000","last_fullsized":"2024-09-19T07:19:42.372588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369500+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:19:42.369500+0000","last_peered":"2024-09-19T07:19:42.369500+0000","last_clean":"2024-09-19T07:19:42.369500+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:19:42.369500+0000","last_undegraded":"2024-09-19T07:19:42.369500+0000","last_fullsized":"2024-09-19T07:19:42.369500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:15.036396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364782+0000","last_change":"2024-09-19T07:19:39.403810+0000","last_active":"2024-09-19T07:19:42.364782+0000","last_peered":"2024-09-19T07:19:42.364782+0000","last_clean":"2024-09-19T07:19:42.364782+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:19:42.364782+0000","last_undegraded":"2024-09-19T07:19:42.364782+0000","last_fullsized":"2024-09-19T07:19:42.364782+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:55.714964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365790+0000","last_change":"2024-09-19T07:19:40.390455+0000","last_active":"2024-09-19T07:19:42.365790+0000","last_peered":"2024-09-19T07:19:42.365790+0000","last_clean":"2024-09-19T07:19:42.365790+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:19:42.365790+0000","last_undegraded":"2024-09-19T07:19:42.365790+0000","last_fullsized":"2024-09-19T07:19:42.365790+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367307+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:19:42.367307+0000","last_peered":"2024-09-19T07:19:42.367307+0000","last_clean":"2024-09-19T07:19:42.367307+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:19:42.367307+0000","last_undegraded":"2024-09-19T07:19:42.367307+0000","last_fullsized":"2024-09-19T07:19:42.367307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353956+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:19:41.353956+0000","last_peered":"2024-09-19T07:19:41.353956+0000","last_clean":"2024-09-19T07:19:41.353956+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:19:41.353956+0000","last_undegraded":"2024-09-19T07:19:41.353956+0000","last_fullsized":"2024-09-19T07:19:41.353956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366091+0000","last_change":"2024-09-19T07:19:39.408309+0000","last_active":"2024-09-19T07:19:42.366091+0000","last_peered":"2024-09-19T07:19:42.366091+0000","last_clean":"2024-09-19T07:19:42.366091+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:19:42.366091+0000","last_undegraded":"2024-09-19T07:19:42.366091+0000","last_fullsized":"2024-09-19T07:19:42.366091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:19.396261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353849+0000","last_change":"2024-09-19T07:19:40.372543+0000","last_active":"2024-09-19T07:19:41.353849+0000","last_peered":"2024-09-19T07:19:41.353849+0000","last_clean":"2024-09-19T07:19:41.353849+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:19:41.353849+0000","last_undegraded":"2024-09-19T07:19:41.353849+0000","last_fullsized":"2024-09-19T07:19:41.353849+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:53:15.682896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367705+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:19:42.367705+0000","last_peered":"2024-09-19T07:19:42.367705+0000","last_clean":"2024-09-19T07:19:42.367705+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:19:42.367705+0000","last_undegraded":"2024-09-19T07:19:42.367705+0000","last_fullsized":"2024-09-19T07:19:42.367705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355253+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:19:41.355253+0000","last_peered":"2024-09-19T07:19:41.355253+0000","last_clean":"2024-09-19T07:19:41.355253+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:19:41.355253+0000","last_undegraded":"2024-09-19T07:19:41.355253+0000","last_fullsized":"2024-09-19T07:19:41.355253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366176+0000","last_change":"2024-09-19T07:19:39.407045+0000","last_active":"2024-09-19T07:19:42.366176+0000","last_peered":"2024-09-19T07:19:42.366176+0000","last_clean":"2024-09-19T07:19:42.366176+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:19:42.366176+0000","last_undegraded":"2024-09-19T07:19:42.366176+0000","last_fullsized":"2024-09-19T07:19:42.366176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:24.846974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367577+0000","last_change":"2024-09-19T07:19:40.396520+0000","last_active":"2024-09-19T07:19:42.367577+0000","last_peered":"2024-09-19T07:19:42.367577+0000","last_clean":"2024-09-19T07:19:42.367577+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:19:42.367577+0000","last_undegraded":"2024-09-19T07:19:42.367577+0000","last_fullsized":"2024-09-19T07:19:42.367577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353504+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:19:41.353504+0000","last_peered":"2024-09-19T07:19:41.353504+0000","last_clean":"2024-09-19T07:19:41.353504+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:19:41.353504+0000","last_undegraded":"2024-09-19T07:19:41.353504+0000","last_fullsized":"2024-09-19T07:19:41.353504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367046+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:19:42.367046+0000","last_peered":"2024-09-19T07:19:42.367046+0000","last_clean":"2024-09-19T07:19:42.367046+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:19:42.367046+0000","last_undegraded":"2024-09-19T07:19:42.367046+0000","last_fullsized":"2024-09-19T07:19:42.367046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368034+0000","last_change":"2024-09-19T07:19:39.410500+0000","last_active":"2024-09-19T07:19:42.368034+0000","last_peered":"2024-09-19T07:19:42.368034+0000","last_clean":"2024-09-19T07:19:42.368034+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:19:42.368034+0000","last_undegraded":"2024-09-19T07:19:42.368034+0000","last_fullsized":"2024-09-19T07:19:42.368034+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:04.182606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349177+0000","last_change":"2024-09-19T07:19:40.381259+0000","last_active":"2024-09-19T07:19:41.349177+0000","last_peered":"2024-09-19T07:19:41.349177+0000","last_clean":"2024-09-19T07:19:41.349177+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:19:41.349177+0000","last_undegraded":"2024-09-19T07:19:41.349177+0000","last_fullsized":"2024-09-19T07:19:41.349177+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:22.017682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370825+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:19:42.370825+0000","last_peered":"2024-09-19T07:19:42.370825+0000","last_clean":"2024-09-19T07:19:42.370825+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:19:42.370825+0000","last_undegraded":"2024-09-19T07:19:42.370825+0000","last_fullsized":"2024-09-19T07:19:42.370825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367918+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:19:42.367918+0000","last_peered":"2024-09-19T07:19:42.367918+0000","last_clean":"2024-09-19T07:19:42.367918+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:19:42.367918+0000","last_undegraded":"2024-09-19T07:19:42.367918+0000","last_fullsized":"2024-09-19T07:19:42.367918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367682+0000","last_change":"2024-09-19T07:19:39.412123+0000","last_active":"2024-09-19T07:19:42.367682+0000","last_peered":"2024-09-19T07:19:42.367682+0000","last_clean":"2024-09-19T07:19:42.367682+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:19:42.367682+0000","last_undegraded":"2024-09-19T07:19:42.367682+0000","last_fullsized":"2024-09-19T07:19:42.367682+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367943+0000","last_change":"2024-09-19T07:19:40.383214+0000","last_active":"2024-09-19T07:19:42.367943+0000","last_peered":"2024-09-19T07:19:42.367943+0000","last_clean":"2024-09-19T07:19:42.367943+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:19:42.367943+0000","last_undegraded":"2024-09-19T07:19:42.367943+0000","last_fullsized":"2024-09-19T07:19:42.367943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367066+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:19:42.367066+0000","last_peered":"2024-09-19T07:19:42.367066+0000","last_clean":"2024-09-19T07:19:42.367066+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:19:42.367066+0000","last_undegraded":"2024-09-19T07:19:42.367066+0000","last_fullsized":"2024-09-19T07:19:42.367066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367094+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:19:42.367094+0000","last_peered":"2024-09-19T07:19:42.367094+0000","last_clean":"2024-09-19T07:19:42.367094+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:19:42.367094+0000","last_undegraded":"2024-09-19T07:19:42.367094+0000","last_fullsized":"2024-09-19T07:19:42.367094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369704+0000","last_change":"2024-09-19T07:19:39.408228+0000","last_active":"2024-09-19T07:19:42.369704+0000","last_peered":"2024-09-19T07:19:42.369704+0000","last_clean":"2024-09-19T07:19:42.369704+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:19:42.369704+0000","last_undegraded":"2024-09-19T07:19:42.369704+0000","last_fullsized":"2024-09-19T07:19:42.369704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355858+0000","last_change":"2024-09-19T07:19:40.395566+0000","last_active":"2024-09-19T07:19:41.355858+0000","last_peered":"2024-09-19T07:19:41.355858+0000","last_clean":"2024-09-19T07:19:41.355858+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:19:41.355858+0000","last_undegraded":"2024-09-19T07:19:41.355858+0000","last_fullsized":"2024-09-19T07:19:41.355858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:54:02.766021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367642+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:19:42.367642+0000","last_peered":"2024-09-19T07:19:42.367642+0000","last_clean":"2024-09-19T07:19:42.367642+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:19:42.367642+0000","last_undegraded":"2024-09-19T07:19:42.367642+0000","last_fullsized":"2024-09-19T07:19:42.367642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366078+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:19:42.366078+0000","last_peered":"2024-09-19T07:19:42.366078+0000","last_clean":"2024-09-19T07:19:42.366078+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:19:42.366078+0000","last_undegraded":"2024-09-19T07:19:42.366078+0000","last_fullsized":"2024-09-19T07:19:42.366078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368203+0000","last_change":"2024-09-19T07:19:39.424661+0000","last_active":"2024-09-19T07:19:42.368203+0000","last_peered":"2024-09-19T07:19:42.368203+0000","last_clean":"2024-09-19T07:19:42.368203+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:19:42.368203+0000","last_undegraded":"2024-09-19T07:19:42.368203+0000","last_fullsized":"2024-09-19T07:19:42.368203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365431+0000","last_change":"2024-09-19T07:19:40.388972+0000","last_active":"2024-09-19T07:19:42.365431+0000","last_peered":"2024-09-19T07:19:42.365431+0000","last_clean":"2024-09-19T07:19:42.365431+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:19:42.365431+0000","last_undegraded":"2024-09-19T07:19:42.365431+0000","last_fullsized":"2024-09-19T07:19:42.365431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371005+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:19:42.371005+0000","last_peered":"2024-09-19T07:19:42.371005+0000","last_clean":"2024-09-19T07:19:42.371005+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:19:42.371005+0000","last_undegraded":"2024-09-19T07:19:42.371005+0000","last_fullsized":"2024-09-19T07:19:42.371005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:23.791880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.357019+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:19:41.357019+0000","last_peered":"2024-09-19T07:19:41.357019+0000","last_clean":"2024-09-19T07:19:41.357019+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:19:41.357019+0000","last_undegraded":"2024-09-19T07:19:41.357019+0000","last_fullsized":"2024-09-19T07:19:41.357019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366707+0000","last_change":"2024-09-19T07:19:39.412192+0000","last_active":"2024-09-19T07:19:42.366707+0000","last_peered":"2024-09-19T07:19:42.366707+0000","last_clean":"2024-09-19T07:19:42.366707+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:19:42.366707+0000","last_undegraded":"2024-09-19T07:19:42.366707+0000","last_fullsized":"2024-09-19T07:19:42.366707+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365056+0000","last_change":"2024-09-19T07:19:40.390217+0000","last_active":"2024-09-19T07:19:42.365056+0000","last_peered":"2024-09-19T07:19:42.365056+0000","last_clean":"2024-09-19T07:19:42.365056+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:19:42.365056+0000","last_undegraded":"2024-09-19T07:19:42.365056+0000","last_fullsized":"2024-09-19T07:19:42.365056+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371269+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:19:42.371269+0000","last_peered":"2024-09-19T07:19:42.371269+0000","last_clean":"2024-09-19T07:19:42.371269+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:19:42.371269+0000","last_undegraded":"2024-09-19T07:19:42.371269+0000","last_fullsized":"2024-09-19T07:19:42.371269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368999+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:19:42.368999+0000","last_peered":"2024-09-19T07:19:42.368999+0000","last_clean":"2024-09-19T07:19:42.368999+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:19:42.368999+0000","last_undegraded":"2024-09-19T07:19:42.368999+0000","last_fullsized":"2024-09-19T07:19:42.368999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:50.700962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365712+0000","last_change":"2024-09-19T07:19:39.434150+0000","last_active":"2024-09-19T07:19:42.365712+0000","last_peered":"2024-09-19T07:19:42.365712+0000","last_clean":"2024-09-19T07:19:42.365712+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:19:42.365712+0000","last_undegraded":"2024-09-19T07:19:42.365712+0000","last_fullsized":"2024-09-19T07:19:42.365712+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:13.654208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352846+0000","last_change":"2024-09-19T07:19:40.380140+0000","last_active":"2024-09-19T07:19:41.352846+0000","last_peered":"2024-09-19T07:19:41.352846+0000","last_clean":"2024-09-19T07:19:41.352846+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:19:41.352846+0000","last_undegraded":"2024-09-19T07:19:41.352846+0000","last_fullsized":"2024-09-19T07:19:41.352846+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368156+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:19:42.368156+0000","last_peered":"2024-09-19T07:19:42.368156+0000","last_clean":"2024-09-19T07:19:42.368156+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:19:42.368156+0000","last_undegraded":"2024-09-19T07:19:42.368156+0000","last_fullsized":"2024-09-19T07:19:42.368156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367593+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:19:42.367593+0000","last_peered":"2024-09-19T07:19:42.367593+0000","last_clean":"2024-09-19T07:19:42.367593+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:19:42.367593+0000","last_undegraded":"2024-09-19T07:19:42.367593+0000","last_fullsized":"2024-09-19T07:19:42.367593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366996+0000","last_change":"2024-09-19T07:19:39.411605+0000","last_active":"2024-09-19T07:19:42.366996+0000","last_peered":"2024-09-19T07:19:42.366996+0000","last_clean":"2024-09-19T07:19:42.366996+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:19:42.366996+0000","last_undegraded":"2024-09-19T07:19:42.366996+0000","last_fullsized":"2024-09-19T07:19:42.366996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:55.654483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367982+0000","last_change":"2024-09-19T07:19:40.372083+0000","last_active":"2024-09-19T07:19:42.367982+0000","last_peered":"2024-09-19T07:19:42.367982+0000","last_clean":"2024-09-19T07:19:42.367982+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:19:42.367982+0000","last_undegraded":"2024-09-19T07:19:42.367982+0000","last_fullsized":"2024-09-19T07:19:42.367982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:39.665475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355342+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:19:41.355342+0000","last_peered":"2024-09-19T07:19:41.355342+0000","last_clean":"2024-09-19T07:19:41.355342+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:19:41.355342+0000","last_undegraded":"2024-09-19T07:19:41.355342+0000","last_fullsized":"2024-09-19T07:19:41.355342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368442+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:19:42.368442+0000","last_peered":"2024-09-19T07:19:42.368442+0000","last_clean":"2024-09-19T07:19:42.368442+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:19:42.368442+0000","last_undegraded":"2024-09-19T07:19:42.368442+0000","last_fullsized":"2024-09-19T07:19:42.368442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367339+0000","last_change":"2024-09-19T07:19:39.420340+0000","last_active":"2024-09-19T07:19:42.367339+0000","last_peered":"2024-09-19T07:19:42.367339+0000","last_clean":"2024-09-19T07:19:42.367339+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:19:42.367339+0000","last_undegraded":"2024-09-19T07:19:42.367339+0000","last_fullsized":"2024-09-19T07:19:42.367339+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:29.338848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369245+0000","last_change":"2024-09-19T07:19:40.383082+0000","last_active":"2024-09-19T07:19:42.369245+0000","last_peered":"2024-09-19T07:19:42.369245+0000","last_clean":"2024-09-19T07:19:42.369245+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:19:42.369245+0000","last_undegraded":"2024-09-19T07:19:42.369245+0000","last_fullsized":"2024-09-19T07:19:42.369245+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:54.436493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367222+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:19:42.367222+0000","last_peered":"2024-09-19T07:19:42.367222+0000","last_clean":"2024-09-19T07:19:42.367222+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:19:42.367222+0000","last_undegraded":"2024-09-19T07:19:42.367222+0000","last_fullsized":"2024-09-19T07:19:42.367222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354505+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:19:41.354505+0000","last_peered":"2024-09-19T07:19:41.354505+0000","last_clean":"2024-09-19T07:19:41.354505+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:19:41.354505+0000","last_undegraded":"2024-09-19T07:19:41.354505+0000","last_fullsized":"2024-09-19T07:19:41.354505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368035+0000","last_change":"2024-09-19T07:19:39.407135+0000","last_active":"2024-09-19T07:19:42.368035+0000","last_peered":"2024-09-19T07:19:42.368035+0000","last_clean":"2024-09-19T07:19:42.368035+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:19:42.368035+0000","last_undegraded":"2024-09-19T07:19:42.368035+0000","last_fullsized":"2024-09-19T07:19:42.368035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:27.890595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349118+0000","last_change":"2024-09-19T07:19:40.368866+0000","last_active":"2024-09-19T07:19:41.349118+0000","last_peered":"2024-09-19T07:19:41.349118+0000","last_clean":"2024-09-19T07:19:41.349118+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:19:41.349118+0000","last_undegraded":"2024-09-19T07:19:41.349118+0000","last_fullsized":"2024-09-19T07:19:41.349118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:28.893439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366065+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:19:42.366065+0000","last_peered":"2024-09-19T07:19:42.366065+0000","last_clean":"2024-09-19T07:19:42.366065+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:19:42.366065+0000","last_undegraded":"2024-09-19T07:19:42.366065+0000","last_fullsized":"2024-09-19T07:19:42.366065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348283+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:19:41.348283+0000","last_peered":"2024-09-19T07:19:41.348283+0000","last_clean":"2024-09-19T07:19:41.348283+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:19:41.348283+0000","last_undegraded":"2024-09-19T07:19:41.348283+0000","last_fullsized":"2024-09-19T07:19:41.348283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369806+0000","last_change":"2024-09-19T07:19:39.420497+0000","last_active":"2024-09-19T07:19:42.369806+0000","last_peered":"2024-09-19T07:19:42.369806+0000","last_clean":"2024-09-19T07:19:42.369806+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:19:42.369806+0000","last_undegraded":"2024-09-19T07:19:42.369806+0000","last_fullsized":"2024-09-19T07:19:42.369806+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:19.547270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356452+0000","last_change":"2024-09-19T07:19:40.387573+0000","last_active":"2024-09-19T07:19:41.356452+0000","last_peered":"2024-09-19T07:19:41.356452+0000","last_clean":"2024-09-19T07:19:41.356452+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:19:41.356452+0000","last_undegraded":"2024-09-19T07:19:41.356452+0000","last_fullsized":"2024-09-19T07:19:41.356452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:20:43.234793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365631+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:19:42.365631+0000","last_peered":"2024-09-19T07:19:42.365631+0000","last_clean":"2024-09-19T07:19:42.365631+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:19:42.365631+0000","last_undegraded":"2024-09-19T07:19:42.365631+0000","last_fullsized":"2024-09-19T07:19:42.365631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:20.824916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366356+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:19:42.366356+0000","last_peered":"2024-09-19T07:19:42.366356+0000","last_clean":"2024-09-19T07:19:42.366356+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:19:42.366356+0000","last_undegraded":"2024-09-19T07:19:42.366356+0000","last_fullsized":"2024-09-19T07:19:42.366356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371671+0000","last_change":"2024-09-19T07:19:39.410752+0000","last_active":"2024-09-19T07:19:42.371671+0000","last_peered":"2024-09-19T07:19:42.371671+0000","last_clean":"2024-09-19T07:19:42.371671+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:42.371671+0000","last_undegraded":"2024-09-19T07:19:42.371671+0000","last_fullsized":"2024-09-19T07:19:42.371671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:47:09.376497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367120+0000","last_change":"2024-09-19T07:19:40.392973+0000","last_active":"2024-09-19T07:19:42.367120+0000","last_peered":"2024-09-19T07:19:42.367120+0000","last_clean":"2024-09-19T07:19:42.367120+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:19:42.367120+0000","last_undegraded":"2024-09-19T07:19:42.367120+0000","last_fullsized":"2024-09-19T07:19:42.367120+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368064+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:19:42.368064+0000","last_peered":"2024-09-19T07:19:42.368064+0000","last_clean":"2024-09-19T07:19:42.368064+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:19:42.368064+0000","last_undegraded":"2024-09-19T07:19:42.368064+0000","last_fullsized":"2024-09-19T07:19:42.368064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354067+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:19:41.354067+0000","last_peered":"2024-09-19T07:19:41.354067+0000","last_clean":"2024-09-19T07:19:41.354067+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:19:41.354067+0000","last_undegraded":"2024-09-19T07:19:41.354067+0000","last_fullsized":"2024-09-19T07:19:41.354067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367007+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:19:42.367007+0000","last_peered":"2024-09-19T07:19:42.367007+0000","last_clean":"2024-09-19T07:19:42.367007+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:19:42.367007+0000","last_undegraded":"2024-09-19T07:19:42.367007+0000","last_fullsized":"2024-09-19T07:19:42.367007+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354541+0000","last_change":"2024-09-19T07:19:39.429085+0000","last_active":"2024-09-19T07:19:41.354541+0000","last_peered":"2024-09-19T07:19:41.354541+0000","last_clean":"2024-09-19T07:19:41.354541+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:19:41.354541+0000","last_undegraded":"2024-09-19T07:19:41.354541+0000","last_fullsized":"2024-09-19T07:19:41.354541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348394+0000","last_change":"2024-09-19T07:19:40.382352+0000","last_active":"2024-09-19T07:19:41.348394+0000","last_peered":"2024-09-19T07:19:41.348394+0000","last_clean":"2024-09-19T07:19:41.348394+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:19:41.348394+0000","last_undegraded":"2024-09-19T07:19:41.348394+0000","last_fullsized":"2024-09-19T07:19:41.348394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:31.024388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367107+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:19:42.367107+0000","last_peered":"2024-09-19T07:19:42.367107+0000","last_clean":"2024-09-19T07:19:42.367107+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:19:42.367107+0000","last_undegraded":"2024-09-19T07:19:42.367107+0000","last_fullsized":"2024-09-19T07:19:42.367107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348973+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:19:41.348973+0000","last_peered":"2024-09-19T07:19:41.348973+0000","last_clean":"2024-09-19T07:19:41.348973+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:19:41.348973+0000","last_undegraded":"2024-09-19T07:19:41.348973+0000","last_fullsized":"2024-09-19T07:19:41.348973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353111+0000","last_change":"2024-09-19T07:19:39.427644+0000","last_active":"2024-09-19T07:19:41.353111+0000","last_peered":"2024-09-19T07:19:41.353111+0000","last_clean":"2024-09-19T07:19:41.353111+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:19:41.353111+0000","last_undegraded":"2024-09-19T07:19:41.353111+0000","last_fullsized":"2024-09-19T07:19:41.353111+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348470+0000","last_change":"2024-09-19T07:19:40.390380+0000","last_active":"2024-09-19T07:19:41.348470+0000","last_peered":"2024-09-19T07:19:41.348470+0000","last_clean":"2024-09-19T07:19:41.348470+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:19:41.348470+0000","last_undegraded":"2024-09-19T07:19:41.348470+0000","last_fullsized":"2024-09-19T07:19:41.348470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:26.385490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354180+0000","last_change":"2024-09-19T07:19:32.276600+0000","last_active":"2024-09-19T07:19:41.354180+0000","last_peered":"2024-09-19T07:19:41.354180+0000","last_clean":"2024-09-19T07:19:41.354180+0000","last_became_active":"2024-09-19T07:19:32.275766+0000","last_became_peered":"2024-09-19T07:19:32.275766+0000","last_unstale":"2024-09-19T07:19:41.354180+0000","last_undegraded":"2024-09-19T07:19:41.354180+0000","last_fullsized":"2024-09-19T07:19:41.354180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:48.506399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370499+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:19:42.370499+0000","last_peered":"2024-09-19T07:19:42.370499+0000","last_clean":"2024-09-19T07:19:42.370499+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:19:42.370499+0000","last_undegraded":"2024-09-19T07:19:42.370499+0000","last_fullsized":"2024-09-19T07:19:42.370499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366597+0000","last_change":"2024-09-19T07:19:39.410638+0000","last_active":"2024-09-19T07:19:42.366597+0000","last_peered":"2024-09-19T07:19:42.366597+0000","last_clean":"2024-09-19T07:19:42.366597+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:19:42.366597+0000","last_undegraded":"2024-09-19T07:19:42.366597+0000","last_fullsized":"2024-09-19T07:19:42.366597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:32.073136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365442+0000","last_change":"2024-09-19T07:19:40.378481+0000","last_active":"2024-09-19T07:19:42.365442+0000","last_peered":"2024-09-19T07:19:42.365442+0000","last_clean":"2024-09-19T07:19:42.365442+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:19:42.365442+0000","last_undegraded":"2024-09-19T07:19:42.365442+0000","last_fullsized":"2024-09-19T07:19:42.365442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:30.585686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369508+0000","last_change":"2024-09-19T07:19:32.292026+0000","last_active":"2024-09-19T07:19:42.369508+0000","last_peered":"2024-09-19T07:19:42.369508+0000","last_clean":"2024-09-19T07:19:42.369508+0000","last_became_active":"2024-09-19T07:19:32.291037+0000","last_became_peered":"2024-09-19T07:19:32.291037+0000","last_unstale":"2024-09-19T07:19:42.369508+0000","last_undegraded":"2024-09-19T07:19:42.369508+0000","last_fullsized":"2024-09-19T07:19:42.369508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:22.190763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349468+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:19:41.349468+0000","last_peered":"2024-09-19T07:19:41.349468+0000","last_clean":"2024-09-19T07:19:41.349468+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:19:41.349468+0000","last_undegraded":"2024-09-19T07:19:41.349468+0000","last_fullsized":"2024-09-19T07:19:41.349468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353302+0000","last_change":"2024-09-19T07:19:39.437656+0000","last_active":"2024-09-19T07:19:41.353302+0000","last_peered":"2024-09-19T07:19:41.353302+0000","last_clean":"2024-09-19T07:19:41.353302+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:19:41.353302+0000","last_undegraded":"2024-09-19T07:19:41.353302+0000","last_fullsized":"2024-09-19T07:19:41.353302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:41.468096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.347816+0000","last_change":"2024-09-19T07:19:40.379289+0000","last_active":"2024-09-19T07:19:41.347816+0000","last_peered":"2024-09-19T07:19:41.347816+0000","last_clean":"2024-09-19T07:19:41.347816+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:19:41.347816+0000","last_undegraded":"2024-09-19T07:19:41.347816+0000","last_fullsized":"2024-09-19T07:19:41.347816+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:20.399106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368075+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:19:42.368075+0000","last_peered":"2024-09-19T07:19:42.368075+0000","last_clean":"2024-09-19T07:19:42.368075+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:19:42.368075+0000","last_undegraded":"2024-09-19T07:19:42.368075+0000","last_fullsized":"2024-09-19T07:19:42.368075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368710+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:19:42.368710+0000","last_peered":"2024-09-19T07:19:42.368710+0000","last_clean":"2024-09-19T07:19:42.368710+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:19:42.368710+0000","last_undegraded":"2024-09-19T07:19:42.368710+0000","last_fullsized":"2024-09-19T07:19:42.368710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366459+0000","last_change":"2024-09-19T07:19:39.413589+0000","last_active":"2024-09-19T07:19:42.366459+0000","last_peered":"2024-09-19T07:19:42.366459+0000","last_clean":"2024-09-19T07:19:42.366459+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:19:42.366459+0000","last_undegraded":"2024-09-19T07:19:42.366459+0000","last_fullsized":"2024-09-19T07:19:42.366459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370515+0000","last_change":"2024-09-19T07:19:40.387159+0000","last_active":"2024-09-19T07:19:42.370515+0000","last_peered":"2024-09-19T07:19:42.370515+0000","last_clean":"2024-09-19T07:19:42.370515+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:19:42.370515+0000","last_undegraded":"2024-09-19T07:19:42.370515+0000","last_fullsized":"2024-09-19T07:19:42.370515+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366844+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:19:42.366844+0000","last_peered":"2024-09-19T07:19:42.366844+0000","last_clean":"2024-09-19T07:19:42.366844+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:19:42.366844+0000","last_undegraded":"2024-09-19T07:19:42.366844+0000","last_fullsized":"2024-09-19T07:19:42.366844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.347952+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:19:41.347952+0000","last_peered":"2024-09-19T07:19:41.347952+0000","last_clean":"2024-09-19T07:19:41.347952+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:19:41.347952+0000","last_undegraded":"2024-09-19T07:19:41.347952+0000","last_fullsized":"2024-09-19T07:19:41.347952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371002+0000","last_change":"2024-09-19T07:19:39.406110+0000","last_active":"2024-09-19T07:19:42.371002+0000","last_peered":"2024-09-19T07:19:42.371002+0000","last_clean":"2024-09-19T07:19:42.371002+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:19:42.371002+0000","last_undegraded":"2024-09-19T07:19:42.371002+0000","last_fullsized":"2024-09-19T07:19:42.371002+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:10:05.224225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355606+0000","last_change":"2024-09-19T07:19:40.365764+0000","last_active":"2024-09-19T07:19:41.355606+0000","last_peered":"2024-09-19T07:19:41.355606+0000","last_clean":"2024-09-19T07:19:41.355606+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:19:41.355606+0000","last_undegraded":"2024-09-19T07:19:41.355606+0000","last_fullsized":"2024-09-19T07:19:41.355606+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:04:03.242292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352283+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:19:41.352283+0000","last_peered":"2024-09-19T07:19:41.352283+0000","last_clean":"2024-09-19T07:19:41.352283+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:19:41.352283+0000","last_undegraded":"2024-09-19T07:19:41.352283+0000","last_fullsized":"2024-09-19T07:19:41.352283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353746+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:19:41.353746+0000","last_peered":"2024-09-19T07:19:41.353746+0000","last_clean":"2024-09-19T07:19:41.353746+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:19:41.353746+0000","last_undegraded":"2024-09-19T07:19:41.353746+0000","last_fullsized":"2024-09-19T07:19:41.353746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370756+0000","last_change":"2024-09-19T07:19:39.429768+0000","last_active":"2024-09-19T07:19:42.370756+0000","last_peered":"2024-09-19T07:19:42.370756+0000","last_clean":"2024-09-19T07:19:42.370756+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:19:42.370756+0000","last_undegraded":"2024-09-19T07:19:42.370756+0000","last_fullsized":"2024-09-19T07:19:42.370756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:47.503631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352914+0000","last_change":"2024-09-19T07:19:40.373542+0000","last_active":"2024-09-19T07:19:41.352914+0000","last_peered":"2024-09-19T07:19:41.352914+0000","last_clean":"2024-09-19T07:19:41.352914+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:19:41.352914+0000","last_undegraded":"2024-09-19T07:19:41.352914+0000","last_fullsized":"2024-09-19T07:19:41.352914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:14.326662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366794+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:19:42.366794+0000","last_peered":"2024-09-19T07:19:42.366794+0000","last_clean":"2024-09-19T07:19:42.366794+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:19:42.366794+0000","last_undegraded":"2024-09-19T07:19:42.366794+0000","last_fullsized":"2024-09-19T07:19:42.366794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354258+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:19:41.354258+0000","last_peered":"2024-09-19T07:19:41.354258+0000","last_clean":"2024-09-19T07:19:41.354258+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:19:41.354258+0000","last_undegraded":"2024-09-19T07:19:41.354258+0000","last_fullsized":"2024-09-19T07:19:41.354258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"24'6","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370954+0000","last_change":"2024-09-19T07:19:39.411074+0000","last_active":"2024-09-19T07:19:42.370954+0000","last_peered":"2024-09-19T07:19:42.370954+0000","last_clean":"2024-09-19T07:19:42.370954+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:19:42.370954+0000","last_undegraded":"2024-09-19T07:19:42.370954+0000","last_fullsized":"2024-09-19T07:19:42.370954+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:53.992787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370720+0000","last_change":"2024-09-19T07:19:40.364814+0000","last_active":"2024-09-19T07:19:42.370720+0000","last_peered":"2024-09-19T07:19:42.370720+0000","last_clean":"2024-09-19T07:19:42.370720+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:19:42.370720+0000","last_undegraded":"2024-09-19T07:19:42.370720+0000","last_fullsized":"2024-09-19T07:19:42.370720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366287+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:19:42.366287+0000","last_peered":"2024-09-19T07:19:42.366287+0000","last_clean":"2024-09-19T07:19:42.366287+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:19:42.366287+0000","last_undegraded":"2024-09-19T07:19:42.366287+0000","last_fullsized":"2024-09-19T07:19:42.366287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:57.131912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368746+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:19:42.368746+0000","last_peered":"2024-09-19T07:19:42.368746+0000","last_clean":"2024-09-19T07:19:42.368746+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:19:42.368746+0000","last_undegraded":"2024-09-19T07:19:42.368746+0000","last_fullsized":"2024-09-19T07:19:42.368746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:23.367365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"24'3","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355049+0000","last_change":"2024-09-19T07:19:39.431616+0000","last_active":"2024-09-19T07:19:41.355049+0000","last_peered":"2024-09-19T07:19:41.355049+0000","last_clean":"2024-09-19T07:19:41.355049+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:19:41.355049+0000","last_undegraded":"2024-09-19T07:19:41.355049+0000","last_fullsized":"2024-09-19T07:19:41.355049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:18.149581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370545+0000","last_change":"2024-09-19T07:19:40.381443+0000","last_active":"2024-09-19T07:19:42.370545+0000","last_peered":"2024-09-19T07:19:42.370545+0000","last_clean":"2024-09-19T07:19:42.370545+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:19:42.370545+0000","last_undegraded":"2024-09-19T07:19:42.370545+0000","last_fullsized":"2024-09-19T07:19:42.370545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366843+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:19:42.366843+0000","last_peered":"2024-09-19T07:19:42.366843+0000","last_clean":"2024-09-19T07:19:42.366843+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:19:42.366843+0000","last_undegraded":"2024-09-19T07:19:42.366843+0000","last_fullsized":"2024-09-19T07:19:42.366843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354497+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:19:41.354497+0000","last_peered":"2024-09-19T07:19:41.354497+0000","last_clean":"2024-09-19T07:19:41.354497+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:19:41.354497+0000","last_undegraded":"2024-09-19T07:19:41.354497+0000","last_fullsized":"2024-09-19T07:19:41.354497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367083+0000","last_change":"2024-09-19T07:19:39.423972+0000","last_active":"2024-09-19T07:19:42.367083+0000","last_peered":"2024-09-19T07:19:42.367083+0000","last_clean":"2024-09-19T07:19:42.367083+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:19:42.367083+0000","last_undegraded":"2024-09-19T07:19:42.367083+0000","last_fullsized":"2024-09-19T07:19:42.367083+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:45.387040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367684+0000","last_change":"2024-09-19T07:19:40.382092+0000","last_active":"2024-09-19T07:19:42.367684+0000","last_peered":"2024-09-19T07:19:42.367684+0000","last_clean":"2024-09-19T07:19:42.367684+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:19:42.367684+0000","last_undegraded":"2024-09-19T07:19:42.367684+0000","last_fullsized":"2024-09-19T07:19:42.367684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368384+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:19:42.368384+0000","last_peered":"2024-09-19T07:19:42.368384+0000","last_clean":"2024-09-19T07:19:42.368384+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:19:42.368384+0000","last_undegraded":"2024-09-19T07:19:42.368384+0000","last_fullsized":"2024-09-19T07:19:42.368384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370238+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:19:42.370238+0000","last_peered":"2024-09-19T07:19:42.370238+0000","last_clean":"2024-09-19T07:19:42.370238+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:19:42.370238+0000","last_undegraded":"2024-09-19T07:19:42.370238+0000","last_fullsized":"2024-09-19T07:19:42.370238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366357+0000","last_change":"2024-09-19T07:19:39.424879+0000","last_active":"2024-09-19T07:19:42.366357+0000","last_peered":"2024-09-19T07:19:42.366357+0000","last_clean":"2024-09-19T07:19:42.366357+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:19:42.366357+0000","last_undegraded":"2024-09-19T07:19:42.366357+0000","last_fullsized":"2024-09-19T07:19:42.366357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:30.021543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353343+0000","last_change":"2024-09-19T07:19:40.384136+0000","last_active":"2024-09-19T07:19:41.353343+0000","last_peered":"2024-09-19T07:19:41.353343+0000","last_clean":"2024-09-19T07:19:41.353343+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:19:41.353343+0000","last_undegraded":"2024-09-19T07:19:41.353343+0000","last_fullsized":"2024-09-19T07:19:41.353343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369727+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:19:42.369727+0000","last_peered":"2024-09-19T07:19:42.369727+0000","last_clean":"2024-09-19T07:19:42.369727+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:19:42.369727+0000","last_undegraded":"2024-09-19T07:19:42.369727+0000","last_fullsized":"2024-09-19T07:19:42.369727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:48.308137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355901+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:19:41.355901+0000","last_peered":"2024-09-19T07:19:41.355901+0000","last_clean":"2024-09-19T07:19:41.355901+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:19:41.355901+0000","last_undegraded":"2024-09-19T07:19:41.355901+0000","last_fullsized":"2024-09-19T07:19:41.355901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"24'8","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365988+0000","last_change":"2024-09-19T07:19:39.410445+0000","last_active":"2024-09-19T07:19:42.365988+0000","last_peered":"2024-09-19T07:19:42.365988+0000","last_clean":"2024-09-19T07:19:42.365988+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:19:42.365988+0000","last_undegraded":"2024-09-19T07:19:42.365988+0000","last_fullsized":"2024-09-19T07:19:42.365988+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:27.020327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365513+0000","last_change":"2024-09-19T07:19:40.389499+0000","last_active":"2024-09-19T07:19:42.365513+0000","last_peered":"2024-09-19T07:19:42.365513+0000","last_clean":"2024-09-19T07:19:42.365513+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:19:42.365513+0000","last_undegraded":"2024-09-19T07:19:42.365513+0000","last_fullsized":"2024-09-19T07:19:42.365513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.520872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355599+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:19:41.355599+0000","last_peered":"2024-09-19T07:19:41.355599+0000","last_clean":"2024-09-19T07:19:41.355599+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:19:41.355599+0000","last_undegraded":"2024-09-19T07:19:41.355599+0000","last_fullsized":"2024-09-19T07:19:41.355599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.357039+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:19:41.357039+0000","last_peered":"2024-09-19T07:19:41.357039+0000","last_clean":"2024-09-19T07:19:41.357039+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:19:41.357039+0000","last_undegraded":"2024-09-19T07:19:41.357039+0000","last_fullsized":"2024-09-19T07:19:41.357039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368271+0000","last_change":"2024-09-19T07:19:39.416943+0000","last_active":"2024-09-19T07:19:42.368271+0000","last_peered":"2024-09-19T07:19:42.368271+0000","last_clean":"2024-09-19T07:19:42.368271+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:19:42.368271+0000","last_undegraded":"2024-09-19T07:19:42.368271+0000","last_fullsized":"2024-09-19T07:19:42.368271+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:35.555826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356615+0000","last_change":"2024-09-19T07:19:40.394993+0000","last_active":"2024-09-19T07:19:41.356615+0000","last_peered":"2024-09-19T07:19:41.356615+0000","last_clean":"2024-09-19T07:19:41.356615+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:19:41.356615+0000","last_undegraded":"2024-09-19T07:19:41.356615+0000","last_fullsized":"2024-09-19T07:19:41.356615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370672+0000","last_change":"2024-09-19T07:19:39.418885+0000","last_active":"2024-09-19T07:19:42.370672+0000","last_peered":"2024-09-19T07:19:42.370672+0000","last_clean":"2024-09-19T07:19:42.370672+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:19:42.370672+0000","last_undegraded":"2024-09-19T07:19:42.370672+0000","last_fullsized":"2024-09-19T07:19:42.370672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:35:45.790411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366817+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:19:42.366817+0000","last_peered":"2024-09-19T07:19:42.366817+0000","last_clean":"2024-09-19T07:19:42.366817+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:19:42.366817+0000","last_undegraded":"2024-09-19T07:19:42.366817+0000","last_fullsized":"2024-09-19T07:19:42.366817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353835+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:19:41.353835+0000","last_peered":"2024-09-19T07:19:41.353835+0000","last_clean":"2024-09-19T07:19:41.353835+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:19:41.353835+0000","last_undegraded":"2024-09-19T07:19:41.353835+0000","last_fullsized":"2024-09-19T07:19:41.353835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370413+0000","last_change":"2024-09-19T07:19:40.374048+0000","last_active":"2024-09-19T07:19:42.370413+0000","last_peered":"2024-09-19T07:19:42.370413+0000","last_clean":"2024-09-19T07:19:42.370413+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:19:42.370413+0000","last_undegraded":"2024-09-19T07:19:42.370413+0000","last_fullsized":"2024-09-19T07:19:42.370413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:40.084827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"24'4","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356842+0000","last_change":"2024-09-19T07:19:39.432426+0000","last_active":"2024-09-19T07:19:41.356842+0000","last_peered":"2024-09-19T07:19:41.356842+0000","last_clean":"2024-09-19T07:19:41.356842+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:19:41.356842+0000","last_undegraded":"2024-09-19T07:19:41.356842+0000","last_fullsized":"2024-09-19T07:19:41.356842+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366831+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:19:42.366831+0000","last_peered":"2024-09-19T07:19:42.366831+0000","last_clean":"2024-09-19T07:19:42.366831+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:19:42.366831+0000","last_undegraded":"2024-09-19T07:19:42.366831+0000","last_fullsized":"2024-09-19T07:19:42.366831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368900+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:19:42.368900+0000","last_peered":"2024-09-19T07:19:42.368900+0000","last_clean":"2024-09-19T07:19:42.368900+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:19:42.368900+0000","last_undegraded":"2024-09-19T07:19:42.368900+0000","last_fullsized":"2024-09-19T07:19:42.368900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369632+0000","last_change":"2024-09-19T07:19:40.384934+0000","last_active":"2024-09-19T07:19:42.369632+0000","last_peered":"2024-09-19T07:19:42.369632+0000","last_clean":"2024-09-19T07:19:42.369632+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:19:42.369632+0000","last_undegraded":"2024-09-19T07:19:42.369632+0000","last_fullsized":"2024-09-19T07:19:42.369632+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353660+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:19:41.353660+0000","last_peered":"2024-09-19T07:19:41.353660+0000","last_clean":"2024-09-19T07:19:41.353660+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:19:41.353660+0000","last_undegraded":"2024-09-19T07:19:41.353660+0000","last_fullsized":"2024-09-19T07:19:41.353660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368442+0000","last_change":"2024-09-19T07:19:32.289374+0000","last_active":"2024-09-19T07:19:42.368442+0000","last_peered":"2024-09-19T07:19:42.368442+0000","last_clean":"2024-09-19T07:19:42.368442+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:19:42.368442+0000","last_undegraded":"2024-09-19T07:19:42.368442+0000","last_fullsized":"2024-09-19T07:19:42.368442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348715+0000","last_change":"2024-09-19T07:19:39.410582+0000","last_active":"2024-09-19T07:19:41.348715+0000","last_peered":"2024-09-19T07:19:41.348715+0000","last_clean":"2024-09-19T07:19:41.348715+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:19:41.348715+0000","last_undegraded":"2024-09-19T07:19:41.348715+0000","last_fullsized":"2024-09-19T07:19:41.348715+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:41:40.592907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367452+0000","last_change":"2024-09-19T07:19:40.393950+0000","last_active":"2024-09-19T07:19:42.367452+0000","last_peered":"2024-09-19T07:19:42.367452+0000","last_clean":"2024-09-19T07:19:42.367452+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:19:42.367452+0000","last_undegraded":"2024-09-19T07:19:42.367452+0000","last_fullsized":"2024-09-19T07:19:42.367452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371158+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:19:42.371158+0000","last_peered":"2024-09-19T07:19:42.371158+0000","last_clean":"2024-09-19T07:19:42.371158+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:19:42.371158+0000","last_undegraded":"2024-09-19T07:19:42.371158+0000","last_fullsized":"2024-09-19T07:19:42.371158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366780+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:19:42.366780+0000","last_peered":"2024-09-19T07:19:42.366780+0000","last_clean":"2024-09-19T07:19:42.366780+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:19:42.366780+0000","last_undegraded":"2024-09-19T07:19:42.366780+0000","last_fullsized":"2024-09-19T07:19:42.366780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371594+0000","last_change":"2024-09-19T07:19:39.410662+0000","last_active":"2024-09-19T07:19:42.371594+0000","last_peered":"2024-09-19T07:19:42.371594+0000","last_clean":"2024-09-19T07:19:42.371594+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:42.371594+0000","last_undegraded":"2024-09-19T07:19:42.371594+0000","last_fullsized":"2024-09-19T07:19:42.371594+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:19.196074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372306+0000","last_change":"2024-09-19T07:19:40.385581+0000","last_active":"2024-09-19T07:19:42.372306+0000","last_peered":"2024-09-19T07:19:42.372306+0000","last_clean":"2024-09-19T07:19:42.372306+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:19:42.372306+0000","last_undegraded":"2024-09-19T07:19:42.372306+0000","last_fullsized":"2024-09-19T07:19:42.372306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:14:13.749729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353359+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:19:41.353359+0000","last_peered":"2024-09-19T07:19:41.353359+0000","last_clean":"2024-09-19T07:19:41.353359+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:19:41.353359+0000","last_undegraded":"2024-09-19T07:19:41.353359+0000","last_fullsized":"2024-09-19T07:19:41.353359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369574+0000","last_change":"2024-09-19T07:19:40.377608+0000","last_active":"2024-09-19T07:19:42.369574+0000","last_peered":"2024-09-19T07:19:42.369574+0000","last_clean":"2024-09-19T07:19:42.369574+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:19:42.369574+0000","last_undegraded":"2024-09-19T07:19:42.369574+0000","last_fullsized":"2024-09-19T07:19:42.369574+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348053+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:19:41.348053+0000","last_peered":"2024-09-19T07:19:41.348053+0000","last_clean":"2024-09-19T07:19:41.348053+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:19:41.348053+0000","last_undegraded":"2024-09-19T07:19:41.348053+0000","last_fullsized":"2024-09-19T07:19:41.348053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353277+0000","last_change":"2024-09-19T07:19:40.381262+0000","last_active":"2024-09-19T07:19:41.353277+0000","last_peered":"2024-09-19T07:19:41.353277+0000","last_clean":"2024-09-19T07:19:41.353277+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:19:41.353277+0000","last_undegraded":"2024-09-19T07:19:41.353277+0000","last_fullsized":"2024-09-19T07:19:41.353277+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:34.425472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354018+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:19:41.354018+0000","last_peered":"2024-09-19T07:19:41.354018+0000","last_clean":"2024-09-19T07:19:41.354018+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:19:41.354018+0000","last_undegraded":"2024-09-19T07:19:41.354018+0000","last_fullsized":"2024-09-19T07:19:41.354018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369264+0000","last_change":"2024-09-19T07:19:40.386432+0000","last_active":"2024-09-19T07:19:42.369264+0000","last_peered":"2024-09-19T07:19:42.369264+0000","last_clean":"2024-09-19T07:19:42.369264+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:19:42.369264+0000","last_undegraded":"2024-09-19T07:19:42.369264+0000","last_fullsized":"2024-09-19T07:19:42.369264+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368214+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:19:42.368214+0000","last_peered":"2024-09-19T07:19:42.368214+0000","last_clean":"2024-09-19T07:19:42.368214+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:19:42.368214+0000","last_undegraded":"2024-09-19T07:19:42.368214+0000","last_fullsized":"2024-09-19T07:19:42.368214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365212+0000","last_change":"2024-09-19T07:19:40.378630+0000","last_active":"2024-09-19T07:19:42.365212+0000","last_peered":"2024-09-19T07:19:42.365212+0000","last_clean":"2024-09-19T07:19:42.365212+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:19:42.365212+0000","last_undegraded":"2024-09-19T07:19:42.365212+0000","last_fullsized":"2024-09-19T07:19:42.365212+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371622+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:19:42.371622+0000","last_peered":"2024-09-19T07:19:42.371622+0000","last_clean":"2024-09-19T07:19:42.371622+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:19:42.371622+0000","last_undegraded":"2024-09-19T07:19:42.371622+0000","last_fullsized":"2024-09-19T07:19:42.371622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365853+0000","last_change":"2024-09-19T07:19:40.381365+0000","last_active":"2024-09-19T07:19:42.365853+0000","last_peered":"2024-09-19T07:19:42.365853+0000","last_clean":"2024-09-19T07:19:42.365853+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:19:42.365853+0000","last_undegraded":"2024-09-19T07:19:42.365853+0000","last_fullsized":"2024-09-19T07:19:42.365853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370004+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:19:42.370004+0000","last_peered":"2024-09-19T07:19:42.370004+0000","last_clean":"2024-09-19T07:19:42.370004+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:19:42.370004+0000","last_undegraded":"2024-09-19T07:19:42.370004+0000","last_fullsized":"2024-09-19T07:19:42.370004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365148+0000","last_change":"2024-09-19T07:19:40.387116+0000","last_active":"2024-09-19T07:19:42.365148+0000","last_peered":"2024-09-19T07:19:42.365148+0000","last_clean":"2024-09-19T07:19:42.365148+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:19:42.365148+0000","last_undegraded":"2024-09-19T07:19:42.365148+0000","last_fullsized":"2024-09-19T07:19:42.365148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356605+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:19:41.356605+0000","last_peered":"2024-09-19T07:19:41.356605+0000","last_clean":"2024-09-19T07:19:41.356605+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:19:41.356605+0000","last_undegraded":"2024-09-19T07:19:41.356605+0000","last_fullsized":"2024-09-19T07:19:41.356605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352940+0000","last_change":"2024-09-19T07:19:40.357629+0000","last_active":"2024-09-19T07:19:41.352940+0000","last_peered":"2024-09-19T07:19:41.352940+0000","last_clean":"2024-09-19T07:19:41.352940+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:19:41.352940+0000","last_undegraded":"2024-09-19T07:19:41.352940+0000","last_fullsized":"2024-09-19T07:19:41.352940+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371018+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:19:42.371018+0000","last_peered":"2024-09-19T07:19:42.371018+0000","last_clean":"2024-09-19T07:19:42.371018+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:19:42.371018+0000","last_undegraded":"2024-09-19T07:19:42.371018+0000","last_fullsized":"2024-09-19T07:19:42.371018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371827+0000","last_change":"2024-09-19T07:19:40.388597+0000","last_active":"2024-09-19T07:19:42.371827+0000","last_peered":"2024-09-19T07:19:42.371827+0000","last_clean":"2024-09-19T07:19:42.371827+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:19:42.371827+0000","last_undegraded":"2024-09-19T07:19:42.371827+0000","last_fullsized":"2024-09-19T07:19:42.371827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:34:18.402016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356021+0000","last_change":"2024-09-19T07:19:32.273635+0000","last_active":"2024-09-19T07:19:41.356021+0000","last_peered":"2024-09-19T07:19:41.356021+0000","last_clean":"2024-09-19T07:19:41.356021+0000","last_became_active":"2024-09-19T07:19:32.271748+0000","last_became_peered":"2024-09-19T07:19:32.271748+0000","last_unstale":"2024-09-19T07:19:41.356021+0000","last_undegraded":"2024-09-19T07:19:41.356021+0000","last_fullsized":"2024-09-19T07:19:41.356021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:49:41.724626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369047+0000","last_change":"2024-09-19T07:19:40.383779+0000","last_active":"2024-09-19T07:19:42.369047+0000","last_peered":"2024-09-19T07:19:42.369047+0000","last_clean":"2024-09-19T07:19:42.369047+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:19:42.369047+0000","last_undegraded":"2024-09-19T07:19:42.369047+0000","last_fullsized":"2024-09-19T07:19:42.369047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368684+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:19:42.368684+0000","last_peered":"2024-09-19T07:19:42.368684+0000","last_clean":"2024-09-19T07:19:42.368684+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:19:42.368684+0000","last_undegraded":"2024-09-19T07:19:42.368684+0000","last_fullsized":"2024-09-19T07:19:42.368684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348737+0000","last_change":"2024-09-19T07:19:40.376342+0000","last_active":"2024-09-19T07:19:41.348737+0000","last_peered":"2024-09-19T07:19:41.348737+0000","last_clean":"2024-09-19T07:19:41.348737+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:19:41.348737+0000","last_undegraded":"2024-09-19T07:19:41.348737+0000","last_fullsized":"2024-09-19T07:19:41.348737+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:22.997744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353771+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:19:41.353771+0000","last_peered":"2024-09-19T07:19:41.353771+0000","last_clean":"2024-09-19T07:19:41.353771+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:19:41.353771+0000","last_undegraded":"2024-09-19T07:19:41.353771+0000","last_fullsized":"2024-09-19T07:19:41.353771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371199+0000","last_change":"2024-09-19T07:19:40.361504+0000","last_active":"2024-09-19T07:19:42.371199+0000","last_peered":"2024-09-19T07:19:42.371199+0000","last_clean":"2024-09-19T07:19:42.371199+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:19:42.371199+0000","last_undegraded":"2024-09-19T07:19:42.371199+0000","last_fullsized":"2024-09-19T07:19:42.371199+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:30:31.110669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367582+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:19:42.367582+0000","last_peered":"2024-09-19T07:19:42.367582+0000","last_clean":"2024-09-19T07:19:42.367582+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:19:42.367582+0000","last_undegraded":"2024-09-19T07:19:42.367582+0000","last_fullsized":"2024-09-19T07:19:42.367582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:19.033192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365909+0000","last_change":"2024-09-19T07:19:40.390205+0000","last_active":"2024-09-19T07:19:42.365909+0000","last_peered":"2024-09-19T07:19:42.365909+0000","last_clean":"2024-09-19T07:19:42.365909+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:19:42.365909+0000","last_undegraded":"2024-09-19T07:19:42.365909+0000","last_fullsized":"2024-09-19T07:19:42.365909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354180+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:19:41.354180+0000","last_peered":"2024-09-19T07:19:41.354180+0000","last_clean":"2024-09-19T07:19:41.354180+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:19:41.354180+0000","last_undegraded":"2024-09-19T07:19:41.354180+0000","last_fullsized":"2024-09-19T07:19:41.354180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365447+0000","last_change":"2024-09-19T07:19:40.390559+0000","last_active":"2024-09-19T07:19:42.365447+0000","last_peered":"2024-09-19T07:19:42.365447+0000","last_clean":"2024-09-19T07:19:42.365447+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:19:42.365447+0000","last_undegraded":"2024-09-19T07:19:42.365447+0000","last_fullsized":"2024-09-19T07:19:42.365447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367349+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:19:42.367349+0000","last_peered":"2024-09-19T07:19:42.367349+0000","last_clean":"2024-09-19T07:19:42.367349+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:19:42.367349+0000","last_undegraded":"2024-09-19T07:19:42.367349+0000","last_fullsized":"2024-09-19T07:19:42.367349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352820+0000","last_change":"2024-09-19T07:19:40.358433+0000","last_active":"2024-09-19T07:19:41.352820+0000","last_peered":"2024-09-19T07:19:41.352820+0000","last_clean":"2024-09-19T07:19:41.352820+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:19:41.352820+0000","last_undegraded":"2024-09-19T07:19:41.352820+0000","last_fullsized":"2024-09-19T07:19:41.352820+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:10:25.226677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349337+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:19:41.349337+0000","last_peered":"2024-09-19T07:19:41.349337+0000","last_clean":"2024-09-19T07:19:41.349337+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:19:41.349337+0000","last_undegraded":"2024-09-19T07:19:41.349337+0000","last_fullsized":"2024-09-19T07:19:41.349337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:51.710997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365454+0000","last_change":"2024-09-19T07:19:40.374374+0000","last_active":"2024-09-19T07:19:42.365454+0000","last_peered":"2024-09-19T07:19:42.365454+0000","last_clean":"2024-09-19T07:19:42.365454+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:19:42.365454+0000","last_undegraded":"2024-09-19T07:19:42.365454+0000","last_fullsized":"2024-09-19T07:19:42.365454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:34:05.875388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356952+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:19:41.356952+0000","last_peered":"2024-09-19T07:19:41.356952+0000","last_clean":"2024-09-19T07:19:41.356952+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:19:41.356952+0000","last_undegraded":"2024-09-19T07:19:41.356952+0000","last_fullsized":"2024-09-19T07:19:41.356952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368071+0000","last_change":"2024-09-19T07:19:40.387341+0000","last_active":"2024-09-19T07:19:42.368071+0000","last_peered":"2024-09-19T07:19:42.368071+0000","last_clean":"2024-09-19T07:19:42.368071+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:19:42.368071+0000","last_undegraded":"2024-09-19T07:19:42.368071+0000","last_fullsized":"2024-09-19T07:19:42.368071+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:56:40.928844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353563+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:19:41.353563+0000","last_peered":"2024-09-19T07:19:41.353563+0000","last_clean":"2024-09-19T07:19:41.353563+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:19:41.353563+0000","last_undegraded":"2024-09-19T07:19:41.353563+0000","last_fullsized":"2024-09-19T07:19:41.353563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368148+0000","last_change":"2024-09-19T07:19:40.388484+0000","last_active":"2024-09-19T07:19:42.368148+0000","last_peered":"2024-09-19T07:19:42.368148+0000","last_clean":"2024-09-19T07:19:42.368148+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:19:42.368148+0000","last_undegraded":"2024-09-19T07:19:42.368148+0000","last_fullsized":"2024-09-19T07:19:42.368148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366974+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:19:42.366974+0000","last_peered":"2024-09-19T07:19:42.366974+0000","last_clean":"2024-09-19T07:19:42.366974+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:19:42.366974+0000","last_undegraded":"2024-09-19T07:19:42.366974+0000","last_fullsized":"2024-09-19T07:19:42.366974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352836+0000","last_change":"2024-09-19T07:19:40.367578+0000","last_active":"2024-09-19T07:19:41.352836+0000","last_peered":"2024-09-19T07:19:41.352836+0000","last_clean":"2024-09-19T07:19:41.352836+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:19:41.352836+0000","last_undegraded":"2024-09-19T07:19:41.352836+0000","last_fullsized":"2024-09-19T07:19:41.352836+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:00:25.357854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366048+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:19:42.366048+0000","last_peered":"2024-09-19T07:19:42.366048+0000","last_clean":"2024-09-19T07:19:42.366048+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:19:42.366048+0000","last_undegraded":"2024-09-19T07:19:42.366048+0000","last_fullsized":"2024-09-19T07:19:42.366048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:43.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366598+0000","last_change":"2024-09-19T07:19:40.385641+0000","last_active":"2024-09-19T07:19:42.366598+0000","last_peered":"2024-09-19T07:19:42.366598+0000","last_clean":"2024-09-19T07:19:42.366598+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:19:42.366598+0000","last_undegraded":"2024-09-19T07:19:42.366598+0000","last_fullsized":"2024-09-19T07:19:42.366598+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370694+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:19:42.370694+0000","last_peered":"2024-09-19T07:19:42.370694+0000","last_clean":"2024-09-19T07:19:42.370694+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:19:42.370694+0000","last_undegraded":"2024-09-19T07:19:42.370694+0000","last_fullsized":"2024-09-19T07:19:42.370694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369737+0000","last_change":"2024-09-19T07:19:40.367101+0000","last_active":"2024-09-19T07:19:42.369737+0000","last_peered":"2024-09-19T07:19:42.369737+0000","last_clean":"2024-09-19T07:19:42.369737+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:19:42.369737+0000","last_undegraded":"2024-09-19T07:19:42.369737+0000","last_fullsized":"2024-09-19T07:19:42.369737+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368379+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:19:42.368379+0000","last_peered":"2024-09-19T07:19:42.368379+0000","last_clean":"2024-09-19T07:19:42.368379+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:19:42.368379+0000","last_undegraded":"2024-09-19T07:19:42.368379+0000","last_fullsized":"2024-09-19T07:19:42.368379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348413+0000","last_change":"2024-09-19T07:19:40.384212+0000","last_active":"2024-09-19T07:19:41.348413+0000","last_peered":"2024-09-19T07:19:41.348413+0000","last_clean":"2024-09-19T07:19:41.348413+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:19:41.348413+0000","last_undegraded":"2024-09-19T07:19:41.348413+0000","last_fullsized":"2024-09-19T07:19:41.348413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:57.724404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355987+0000","last_change":"2024-09-19T07:19:32.299956+0000","last_active":"2024-09-19T07:19:41.355987+0000","last_peered":"2024-09-19T07:19:41.355987+0000","last_clean":"2024-09-19T07:19:41.355987+0000","last_became_active":"2024-09-19T07:19:32.299168+0000","last_became_peered":"2024-09-19T07:19:32.299168+0000","last_unstale":"2024-09-19T07:19:41.355987+0000","last_undegraded":"2024-09-19T07:19:41.355987+0000","last_fullsized":"2024-09-19T07:19:41.355987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366761+0000","last_change":"2024-09-19T07:19:40.387401+0000","last_active":"2024-09-19T07:19:42.366761+0000","last_peered":"2024-09-19T07:19:42.366761+0000","last_clean":"2024-09-19T07:19:42.366761+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:19:42.366761+0000","last_undegraded":"2024-09-19T07:19:42.366761+0000","last_fullsized":"2024-09-19T07:19:42.366761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356423+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:19:41.356423+0000","last_peered":"2024-09-19T07:19:41.356423+0000","last_clean":"2024-09-19T07:19:41.356423+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:19:41.356423+0000","last_undegraded":"2024-09-19T07:19:41.356423+0000","last_fullsized":"2024-09-19T07:19:41.356423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352920+0000","last_change":"2024-09-19T07:19:40.361430+0000","last_active":"2024-09-19T07:19:41.352920+0000","last_peered":"2024-09-19T07:19:41.352920+0000","last_clean":"2024-09-19T07:19:41.352920+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:19:41.352920+0000","last_undegraded":"2024-09-19T07:19:41.352920+0000","last_fullsized":"2024-09-19T07:19:41.352920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:23:05.185943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369329+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:19:42.369329+0000","last_peered":"2024-09-19T07:19:42.369329+0000","last_clean":"2024-09-19T07:19:42.369329+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:19:42.369329+0000","last_undegraded":"2024-09-19T07:19:42.369329+0000","last_fullsized":"2024-09-19T07:19:42.369329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371825+0000","last_change":"2024-09-19T07:19:40.378172+0000","last_active":"2024-09-19T07:19:42.371825+0000","last_peered":"2024-09-19T07:19:42.371825+0000","last_clean":"2024-09-19T07:19:42.371825+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:19:42.371825+0000","last_undegraded":"2024-09-19T07:19:42.371825+0000","last_fullsized":"2024-09-19T07:19:42.371825+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:52.321790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366422+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:19:42.366422+0000","last_peered":"2024-09-19T07:19:42.366422+0000","last_clean":"2024-09-19T07:19:42.366422+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:19:42.366422+0000","last_undegraded":"2024-09-19T07:19:42.366422+0000","last_fullsized":"2024-09-19T07:19:42.366422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356589+0000","last_change":"2024-09-19T07:19:40.373909+0000","last_active":"2024-09-19T07:19:41.356589+0000","last_peered":"2024-09-19T07:19:41.356589+0000","last_clean":"2024-09-19T07:19:41.356589+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:19:41.356589+0000","last_undegraded":"2024-09-19T07:19:41.356589+0000","last_fullsized":"2024-09-19T07:19:41.356589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366411+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:19:42.366411+0000","last_peered":"2024-09-19T07:19:42.366411+0000","last_clean":"2024-09-19T07:19:42.366411+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:19:42.366411+0000","last_undegraded":"2024-09-19T07:19:42.366411+0000","last_fullsized":"2024-09-19T07:19:42.366411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367511+0000","last_change":"2024-09-19T07:19:40.396146+0000","last_active":"2024-09-19T07:19:42.367511+0000","last_peered":"2024-09-19T07:19:42.367511+0000","last_clean":"2024-09-19T07:19:42.367511+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:19:42.367511+0000","last_undegraded":"2024-09-19T07:19:42.367511+0000","last_fullsized":"2024-09-19T07:19:42.367511+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353532+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:19:41.353532+0000","last_peered":"2024-09-19T07:19:41.353532+0000","last_clean":"2024-09-19T07:19:41.353532+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:19:41.353532+0000","last_undegraded":"2024-09-19T07:19:41.353532+0000","last_fullsized":"2024-09-19T07:19:41.353532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355009+0000","last_change":"2024-09-19T07:19:40.373552+0000","last_active":"2024-09-19T07:19:41.355009+0000","last_peered":"2024-09-19T07:19:41.355009+0000","last_clean":"2024-09-19T07:19:41.355009+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:19:41.355009+0000","last_undegraded":"2024-09-19T07:19:41.355009+0000","last_fullsized":"2024-09-19T07:19:41.355009+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:23:33.629831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366537+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:19:42.366537+0000","last_peered":"2024-09-19T07:19:42.366537+0000","last_clean":"2024-09-19T07:19:42.366537+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:19:42.366537+0000","last_undegraded":"2024-09-19T07:19:42.366537+0000","last_fullsized":"2024-09-19T07:19:42.366537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367996+0000","last_change":"2024-09-19T07:19:40.356989+0000","last_active":"2024-09-19T07:19:42.367996+0000","last_peered":"2024-09-19T07:19:42.367996+0000","last_clean":"2024-09-19T07:19:42.367996+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:19:42.367996+0000","last_undegraded":"2024-09-19T07:19:42.367996+0000","last_fullsized":"2024-09-19T07:19:42.367996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353682+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:19:41.353682+0000","last_peered":"2024-09-19T07:19:41.353682+0000","last_clean":"2024-09-19T07:19:41.353682+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:19:41.353682+0000","last_undegraded":"2024-09-19T07:19:41.353682+0000","last_fullsized":"2024-09-19T07:19:41.353682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348160+0000","last_change":"2024-09-19T07:19:40.376487+0000","last_active":"2024-09-19T07:19:41.348160+0000","last_peered":"2024-09-19T07:19:41.348160+0000","last_clean":"2024-09-19T07:19:41.348160+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:19:41.348160+0000","last_undegraded":"2024-09-19T07:19:41.348160+0000","last_fullsized":"2024-09-19T07:19:41.348160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:05.185451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369628+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:19:42.369628+0000","last_peered":"2024-09-19T07:19:42.369628+0000","last_clean":"2024-09-19T07:19:42.369628+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:19:42.369628+0000","last_undegraded":"2024-09-19T07:19:42.369628+0000","last_fullsized":"2024-09-19T07:19:42.369628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:25.667168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364909+0000","last_change":"2024-09-19T07:19:40.377604+0000","last_active":"2024-09-19T07:19:42.364909+0000","last_peered":"2024-09-19T07:19:42.364909+0000","last_clean":"2024-09-19T07:19:42.364909+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:19:42.364909+0000","last_undegraded":"2024-09-19T07:19:42.364909+0000","last_fullsized":"2024-09-19T07:19:42.364909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368557+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:19:42.368557+0000","last_peered":"2024-09-19T07:19:42.368557+0000","last_clean":"2024-09-19T07:19:42.368557+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:19:42.368557+0000","last_undegraded":"2024-09-19T07:19:42.368557+0000","last_fullsized":"2024-09-19T07:19:42.368557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365847+0000","last_change":"2024-09-19T07:19:40.377580+0000","last_active":"2024-09-19T07:19:42.365847+0000","last_peered":"2024-09-19T07:19:42.365847+0000","last_clean":"2024-09-19T07:19:42.365847+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:19:42.365847+0000","last_undegraded":"2024-09-19T07:19:42.365847+0000","last_fullsized":"2024-09-19T07:19:42.365847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369285+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:19:42.369285+0000","last_peered":"2024-09-19T07:19:42.369285+0000","last_clean":"2024-09-19T07:19:42.369285+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:19:42.369285+0000","last_undegraded":"2024-09-19T07:19:42.369285+0000","last_fullsized":"2024-09-19T07:19:42.369285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":864256,"data_stored":693530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":282624,"data_stored":104718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":103252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":282624,"data_stored":102720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":290816,"data_stored":105736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":102560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542153,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":286720,"data_stored":104284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":868352,"data_stored":693496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:19:51.653 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-19T07:19:51.653 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-19T07:19:51.653 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-19T07:19:51.654 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-19T07:19:51.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.803+0000 7fad220ff640 1 Processor -- start 2024-09-19T07:19:51.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.803+0000 7fad220ff640 1 -- start start 2024-09-19T07:19:51.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.803+0000 7fad220ff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fad1c0c28e0 0x7fad1c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.803+0000 7fad220ff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c06dea0 0x7fad1c06e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.803+0000 7fad220ff640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad1c0a9e60 con 0x7fad1c071100 2024-09-19T07:19:51.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.803+0000 7fad220ff640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fad1c071b50 con 0x7fad1c0c28e0 2024-09-19T07:19:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.803+0000 7fad220ff640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fad1c0c1ff0 con 0x7fad1c06dea0 2024-09-19T07:19:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.803+0000 7fad1bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c06dea0 0x7fad1c06e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.803+0000 7fad1affd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fad1c0c28e0 0x7fad1c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.803+0000 7fad1bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c06dea0 0x7fad1c06e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:56272/0 (socket says 172.21.15.83:56272) 2024-09-19T07:19:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.803+0000 7fad1bfff640 1 -- 172.21.15.83:0/486434357 learned_addr learned my addr 172.21.15.83:0/486434357 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad1affd640 1 -- 172.21.15.83:0/486434357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c06dea0 msgr2=0x7fad1c06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:51.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad1affd640 1 --2- 172.21.15.83:0/486434357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c06dea0 0x7fad1c06e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad1affd640 1 -- 172.21.15.83:0/486434357 >> v1:172.21.15.184:6789/0 conn(0x7fad1c071100 legacy=0x7fad1c150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad1affd640 1 -- 172.21.15.83:0/486434357 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1c0aa140 con 0x7fad1c0c28e0 2024-09-19T07:19:51.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad1bfff640 1 --2- 172.21.15.83:0/486434357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c06dea0 0x7fad1c06e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:51.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad1affd640 1 --2- 172.21.15.83:0/486434357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad1c0c28e0 0x7fad1c1545e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fad0c009d90 tx=0x7fad0c01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea13dfb0d539bcc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:51.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad1a7fc640 1 -- 172.21.15.83:0/486434357 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c0b2070 con 0x7fad1c0c28e0 2024-09-19T07:19:51.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad1a7fc640 1 -- 172.21.15.83:0/486434357 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0c0a5890 con 0x7fad1c0c28e0 2024-09-19T07:19:51.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad1a7fc640 1 -- 172.21.15.83:0/486434357 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c0a5c60 con 0x7fad1c0c28e0 2024-09-19T07:19:51.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad220ff640 1 -- 172.21.15.83:0/486434357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad1c0c28e0 msgr2=0x7fad1c1545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad220ff640 1 --2- 172.21.15.83:0/486434357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad1c0c28e0 0x7fad1c1545e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fad0c009d90 tx=0x7fad0c01c8e0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad220ff640 1 -- 172.21.15.83:0/486434357 shutdown_connections 2024-09-19T07:19:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad220ff640 1 --2- 172.21.15.83:0/486434357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c06dea0 0x7fad1c06e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad220ff640 1 --2- 172.21.15.83:0/486434357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad1c0c28e0 0x7fad1c1545e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad220ff640 1 -- 172.21.15.83:0/486434357 >> 172.21.15.83:0/486434357 conn(0x7fad1c06bc80 msgr2=0x7fad1c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad220ff640 1 -- 172.21.15.83:0/486434357 shutdown_connections 2024-09-19T07:19:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.807+0000 7fad220ff640 1 -- 172.21.15.83:0/486434357 wait complete. 2024-09-19T07:19:51.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad220ff640 1 Processor -- start 2024-09-19T07:19:51.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad220ff640 1 -- start start 2024-09-19T07:19:51.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad220ff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c06dea0 0x7fad1c0aa460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad220ff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad1c071100 0x7fad1c1ee950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad220ff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c0c28e0 0x7fad1c152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad220ff640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c071660 con 0x7fad1c06dea0 2024-09-19T07:19:51.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad220ff640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c0714e0 con 0x7fad1c071100 2024-09-19T07:19:51.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad220ff640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fad1c160c60 con 0x7fad1c0c28e0 2024-09-19T07:19:51.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad1b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c06dea0 0x7fad1c0aa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad1b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c06dea0 0x7fad1c0aa460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54168/0 (socket says 172.21.15.83:54168) 2024-09-19T07:19:51.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad1b7fe640 1 -- 172.21.15.83:0/3251476828 learned_addr learned my addr 172.21.15.83:0/3251476828 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:51.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad1affd640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad1c071100 0x7fad1c1ee950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad1bfff640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c0c28e0 0x7fad1c152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad1b7fe640 1 -- 172.21.15.83:0/3251476828 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c0c28e0 msgr2=0x7fad1c152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad1b7fe640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c0c28e0 0x7fad1c152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad1b7fe640 1 -- 172.21.15.83:0/3251476828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad1c071100 msgr2=0x7fad1c1ee950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:51.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad1b7fe640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad1c071100 0x7fad1c1ee950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:51.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad1b7fe640 1 -- 172.21.15.83:0/3251476828 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1c1ef1a0 con 0x7fad1c06dea0 2024-09-19T07:19:51.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad1bfff640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c0c28e0 0x7fad1c152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:51.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad1b7fe640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c06dea0 0x7fad1c0aa460 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fad0400c450 tx=0x7fad0400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:51.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad13fff640 1 -- 172.21.15.83:0/3251476828 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04016020 con 0x7fad1c06dea0 2024-09-19T07:19:51.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad13fff640 1 -- 172.21.15.83:0/3251476828 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad040052d0 con 0x7fad1c06dea0 2024-09-19T07:19:51.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad220ff640 1 -- 172.21.15.83:0/3251476828 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad1c1ef480 con 0x7fad1c06dea0 2024-09-19T07:19:51.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad13fff640 1 -- 172.21.15.83:0/3251476828 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04007260 con 0x7fad1c06dea0 2024-09-19T07:19:51.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad220ff640 1 -- 172.21.15.83:0/3251476828 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad1c1eee90 con 0x7fad1c06dea0 2024-09-19T07:19:51.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad220ff640 1 -- 172.21.15.83:0/3251476828 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad1c06e330 con 0x7fad1c06dea0 2024-09-19T07:19:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.811+0000 7fad13fff640 1 -- 172.21.15.83:0/3251476828 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fad04007400 con 0x7fad1c06dea0 2024-09-19T07:19:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.815+0000 7fad13fff640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7facfc0678f0 0x7facfc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.815+0000 7fad1affd640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7facfc0678f0 0x7facfc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.815+0000 7fad13fff640 1 -- 172.21.15.83:0/3251476828 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fad040858f0 con 0x7fad1c06dea0 2024-09-19T07:19:51.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.815+0000 7fad1affd640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7facfc0678f0 0x7facfc069db0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0af000 tx=0x7fad0c0a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:51.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.819+0000 7fad13fff640 1 -- 172.21.15.83:0/3251476828 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fad04020020 con 0x7fad1c06dea0 2024-09-19T07:19:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.999+0000 7fad220ff640 1 -- 172.21.15.83:0/3251476828 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fad1c0c4f30 con 0x7fad1c06dea0 2024-09-19T07:19:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:51.999+0000 7fad13fff640 1 -- 172.21.15.83:0/3251476828 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fad04005480 con 0x7fad1c06dea0 2024-09-19T07:19:52.001 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:52.001 INFO:teuthology.orchestra.run.smithi083.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-19T07:19:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.003+0000 7fad220ff640 1 -- 172.21.15.83:0/3251476828 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7facfc0678f0 msgr2=0x7facfc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.003+0000 7fad220ff640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7facfc0678f0 0x7facfc069db0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0af000 tx=0x7fad0c0a7070 comp rx=0 tx=0).stop 2024-09-19T07:19:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.003+0000 7fad220ff640 1 -- 172.21.15.83:0/3251476828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c06dea0 msgr2=0x7fad1c0aa460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.003+0000 7fad220ff640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c06dea0 0x7fad1c0aa460 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fad0400c450 tx=0x7fad0400c910 comp rx=0 tx=0).stop 2024-09-19T07:19:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.003+0000 7fad220ff640 1 -- 172.21.15.83:0/3251476828 shutdown_connections 2024-09-19T07:19:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.003+0000 7fad220ff640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7facfc0678f0 0x7facfc069db0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.003+0000 7fad220ff640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad1c0c28e0 0x7fad1c152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.003+0000 7fad220ff640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad1c071100 0x7fad1c1ee950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.003+0000 7fad220ff640 1 --2- 172.21.15.83:0/3251476828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c06dea0 0x7fad1c0aa460 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.003+0000 7fad220ff640 1 -- 172.21.15.83:0/3251476828 >> 172.21.15.83:0/3251476828 conn(0x7fad1c06bc80 msgr2=0x7fad1c06c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:52.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.003+0000 7fad220ff640 1 -- 172.21.15.83:0/3251476828 shutdown_connections 2024-09-19T07:19:52.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.003+0000 7fad220ff640 1 -- 172.21.15.83:0/3251476828 wait complete. 2024-09-19T07:19:52.019 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-19T07:19:52.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:19:52.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.127+0000 7f4310651640 1 Processor -- start 2024-09-19T07:19:52.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.127+0000 7f4310651640 1 -- start start 2024-09-19T07:19:52.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f4310651640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4308071100 0x7f43081545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f4310651640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f43080c28e0 0x7f4308150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f4310651640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4308151480 0x7f430806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f4310651640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f4308071b50 con 0x7f43080c28e0 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f4310651640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f43080c1ff0 con 0x7f4308071100 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f4310651640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f43080a9e60 con 0x7f4308151480 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430dbec640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f43080c28e0 0x7f4308150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430dbec640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f43080c28e0 0x7f4308150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54174/0 (socket says 172.21.15.83:54174) 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430dbec640 1 -- 172.21.15.83:0/4176846672 learned_addr learned my addr 172.21.15.83:0/4176846672 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430e3ed640 1 --2- 172.21.15.83:0/4176846672 >> v2:172.21.15.184:3300/0 conn(0x7f4308071100 0x7f43081545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430ebee640 1 --2- 172.21.15.83:0/4176846672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4308151480 0x7f430806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430dbec640 1 -- 172.21.15.83:0/4176846672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4308151480 msgr2=0x7f430806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430dbec640 1 --2- 172.21.15.83:0/4176846672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4308151480 0x7f430806dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430dbec640 1 -- 172.21.15.83:0/4176846672 >> v2:172.21.15.184:3300/0 conn(0x7f4308071100 msgr2=0x7f43081545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:52.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430dbec640 1 --2- 172.21.15.83:0/4176846672 >> v2:172.21.15.184:3300/0 conn(0x7f4308071100 0x7f43081545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430dbec640 1 -- 172.21.15.83:0/4176846672 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43080aa140 con 0x7f43080c28e0 2024-09-19T07:19:52.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430ebee640 1 --2- 172.21.15.83:0/4176846672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4308151480 0x7f430806dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:52.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430dbec640 1 --2- 172.21.15.83:0/4176846672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43080c28e0 0x7f4308150f40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f430806f620 tx=0x7f430401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22f23f4e10a2be8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:52.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430d3eb640 1 -- 172.21.15.83:0/4176846672 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43040a55e0 con 0x7f43080c28e0 2024-09-19T07:19:52.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430d3eb640 1 -- 172.21.15.83:0/4176846672 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43040a5780 con 0x7f43080c28e0 2024-09-19T07:19:52.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.131+0000 7f430d3eb640 1 -- 172.21.15.83:0/4176846672 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43040a5b50 con 0x7f43080c28e0 2024-09-19T07:19:52.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 -- 172.21.15.83:0/4176846672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43080c28e0 msgr2=0x7f4308150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 --2- 172.21.15.83:0/4176846672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43080c28e0 0x7f4308150f40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f430806f620 tx=0x7f430401c960 comp rx=0 tx=0).stop 2024-09-19T07:19:52.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 -- 172.21.15.83:0/4176846672 shutdown_connections 2024-09-19T07:19:52.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 --2- 172.21.15.83:0/4176846672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4308151480 0x7f430806dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 --2- 172.21.15.83:0/4176846672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43080c28e0 0x7f4308150f40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 --2- 172.21.15.83:0/4176846672 >> v2:172.21.15.184:3300/0 conn(0x7f4308071100 0x7f43081545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 -- 172.21.15.83:0/4176846672 >> 172.21.15.83:0/4176846672 conn(0x7f430806bc80 msgr2=0x7f430806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:52.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 -- 172.21.15.83:0/4176846672 shutdown_connections 2024-09-19T07:19:52.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 -- 172.21.15.83:0/4176846672 wait complete. 2024-09-19T07:19:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 Processor -- start 2024-09-19T07:19:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 -- start start 2024-09-19T07:19:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4308071100 0x7f4308155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43080c28e0 0x7f430815c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4308151480 0x7f43081ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f43081570d0 con 0x7f4308151480 2024-09-19T07:19:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4308071730 con 0x7f43080c28e0 2024-09-19T07:19:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4308157250 con 0x7f4308071100 2024-09-19T07:19:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f430ebee640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4308151480 0x7f43081ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f430ebee640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4308151480 0x7f43081ee6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54176/0 (socket says 172.21.15.83:54176) 2024-09-19T07:19:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f430ebee640 1 -- 172.21.15.83:0/571503642 learned_addr learned my addr 172.21.15.83:0/571503642 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f430e3ed640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4308071100 0x7f4308155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f430dbec640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43080c28e0 0x7f430815c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f430ebee640 1 -- 172.21.15.83:0/571503642 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4308071100 msgr2=0x7f4308155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f430ebee640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4308071100 0x7f4308155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f430ebee640 1 -- 172.21.15.83:0/571503642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43080c28e0 msgr2=0x7f430815c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f430ebee640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43080c28e0 0x7f430815c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f430ebee640 1 -- 172.21.15.83:0/571503642 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4308155290 con 0x7f4308151480 2024-09-19T07:19:52.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f430ebee640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4308151480 0x7f43081ee6c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f42f800c7f0 tx=0x7f42f800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:52.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f42feffd640 1 -- 172.21.15.83:0/571503642 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8015020 con 0x7f4308151480 2024-09-19T07:19:52.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f42feffd640 1 -- 172.21.15.83:0/571503642 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f8004250 con 0x7f4308151480 2024-09-19T07:19:52.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 -- 172.21.15.83:0/571503642 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43080abe80 con 0x7f4308151480 2024-09-19T07:19:52.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f42feffd640 1 -- 172.21.15.83:0/571503642 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8003620 con 0x7f4308151480 2024-09-19T07:19:52.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.135+0000 7f4310651640 1 -- 172.21.15.83:0/571503642 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4308155500 con 0x7f4308151480 2024-09-19T07:19:52.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.139+0000 7f4310651640 1 -- 172.21.15.83:0/571503642 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4308150f40 con 0x7f4308151480 2024-09-19T07:19:52.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.139+0000 7f42feffd640 1 -- 172.21.15.83:0/571503642 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f42f8018020 con 0x7f4308151480 2024-09-19T07:19:52.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.139+0000 7f42feffd640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f42f40678c0 0x7f42f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.139+0000 7f42feffd640 1 -- 172.21.15.83:0/571503642 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f42f8085e70 con 0x7f4308151480 2024-09-19T07:19:52.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.139+0000 7f430e3ed640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f42f40678c0 0x7f42f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.139+0000 7f430e3ed640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f42f40678c0 0x7f42f4069d80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4300009790 tx=0x7f4300008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:52.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.143+0000 7f42feffd640 1 -- 172.21.15.83:0/571503642 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f42f801f030 con 0x7f4308151480 2024-09-19T07:19:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.323+0000 7f4310651640 1 -- 172.21.15.83:0/571503642 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43080c4f30 con 0x7f4308151480 2024-09-19T07:19:52.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.323+0000 7f42feffd640 1 -- 172.21.15.83:0/571503642 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f42f8056710 con 0x7f4308151480 2024-09-19T07:19:52.325 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:52.325 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:19:52.326 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:19:52.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.327+0000 7f4310651640 1 -- 172.21.15.83:0/571503642 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f42f40678c0 msgr2=0x7f42f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.327+0000 7f4310651640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f42f40678c0 0x7f42f4069d80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4300009790 tx=0x7f4300008040 comp rx=0 tx=0).stop 2024-09-19T07:19:52.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.327+0000 7f4310651640 1 -- 172.21.15.83:0/571503642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4308151480 msgr2=0x7f43081ee6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.327+0000 7f4310651640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4308151480 0x7f43081ee6c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f42f800c7f0 tx=0x7f42f800ccb0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.327+0000 7f4310651640 1 -- 172.21.15.83:0/571503642 shutdown_connections 2024-09-19T07:19:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.327+0000 7f4310651640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f42f40678c0 0x7f42f4069d80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.327+0000 7f4310651640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4308151480 0x7f43081ee6c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.327+0000 7f4310651640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43080c28e0 0x7f430815c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.327+0000 7f4310651640 1 --2- 172.21.15.83:0/571503642 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4308071100 0x7f4308155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.327+0000 7f4310651640 1 -- 172.21.15.83:0/571503642 >> 172.21.15.83:0/571503642 conn(0x7f430806bc80 msgr2=0x7f430815d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.327+0000 7f4310651640 1 -- 172.21.15.83:0/571503642 shutdown_connections 2024-09-19T07:19:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.327+0000 7f4310651640 1 -- 172.21.15.83:0/571503642 wait complete. 2024-09-19T07:19:52.343 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:19:52.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdffaf2640 1 Processor -- start 2024-09-19T07:19:52.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdffaf2640 1 -- start start 2024-09-19T07:19:52.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdffaf2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcdf8071100 0x7fcdf81545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdffaf2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdf80c28e0 0x7fcdf8150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdffaf2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcdf8151480 0x7fcdf806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdffaf2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fcdf8071b50 con 0x7fcdf8071100 2024-09-19T07:19:52.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdffaf2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fcdf80c1ff0 con 0x7fcdf8151480 2024-09-19T07:19:52.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdffaf2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcdf80a9e60 con 0x7fcdf80c28e0 2024-09-19T07:19:52.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfd88e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcdf8071100 0x7fcdf81545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfd88e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcdf8071100 0x7fcdf81545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54182/0 (socket says 172.21.15.83:54182) 2024-09-19T07:19:52.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfd88e640 1 -- 172.21.15.83:0/597698635 learned_addr learned my addr 172.21.15.83:0/597698635 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:52.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfe08f640 1 --2- 172.21.15.83:0/597698635 >> v2:172.21.15.184:3300/0 conn(0x7fcdf8151480 0x7fcdf806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfd08d640 1 --2- 172.21.15.83:0/597698635 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdf80c28e0 0x7fcdf8150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfd88e640 1 -- 172.21.15.83:0/597698635 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdf80c28e0 msgr2=0x7fcdf8150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:52.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfd88e640 1 --2- 172.21.15.83:0/597698635 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdf80c28e0 0x7fcdf8150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfd88e640 1 -- 172.21.15.83:0/597698635 >> v2:172.21.15.184:3300/0 conn(0x7fcdf8151480 msgr2=0x7fcdf806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:52.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfd88e640 1 --2- 172.21.15.83:0/597698635 >> v2:172.21.15.184:3300/0 conn(0x7fcdf8151480 0x7fcdf806dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfd88e640 1 -- 172.21.15.83:0/597698635 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf80aa140 con 0x7fcdf8071100 2024-09-19T07:19:52.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfe08f640 1 --2- 172.21.15.83:0/597698635 >> v2:172.21.15.184:3300/0 conn(0x7fcdf8151480 0x7fcdf806dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:52.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfd88e640 1 --2- 172.21.15.83:0/597698635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdf8071100 0x7fcdf81545e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcdf806cd60 tx=0x7fcdf401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81edeb784e46c981 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:52.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.455+0000 7fcdfc88c640 1 -- 172.21.15.83:0/597698635 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf40b0070 con 0x7fcdf8071100 2024-09-19T07:19:52.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfc88c640 1 -- 172.21.15.83:0/597698635 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdf40ab600 con 0x7fcdf8071100 2024-09-19T07:19:52.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfc88c640 1 -- 172.21.15.83:0/597698635 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf40ab9d0 con 0x7fcdf8071100 2024-09-19T07:19:52.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 -- 172.21.15.83:0/597698635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdf8071100 msgr2=0x7fcdf81545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 --2- 172.21.15.83:0/597698635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdf8071100 0x7fcdf81545e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcdf806cd60 tx=0x7fcdf401c490 comp rx=0 tx=0).stop 2024-09-19T07:19:52.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 -- 172.21.15.83:0/597698635 shutdown_connections 2024-09-19T07:19:52.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 --2- 172.21.15.83:0/597698635 >> v2:172.21.15.184:3300/0 conn(0x7fcdf8151480 0x7fcdf806dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 --2- 172.21.15.83:0/597698635 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdf80c28e0 0x7fcdf8150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 --2- 172.21.15.83:0/597698635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdf8071100 0x7fcdf81545e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 -- 172.21.15.83:0/597698635 >> 172.21.15.83:0/597698635 conn(0x7fcdf806bc80 msgr2=0x7fcdf806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:52.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 -- 172.21.15.83:0/597698635 shutdown_connections 2024-09-19T07:19:52.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 -- 172.21.15.83:0/597698635 wait complete. 2024-09-19T07:19:52.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 Processor -- start 2024-09-19T07:19:52.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 -- start start 2024-09-19T07:19:52.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdf8071100 0x7fcdf8154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdf80c28e0 0x7fcdf81ece30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdf8151480 0x7fcdf814f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf81570d0 con 0x7fcdf80c28e0 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf8071730 con 0x7fcdf8151480 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcdf8157250 con 0x7fcdf8071100 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfd08d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdf80c28e0 0x7fcdf81ece30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfd08d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdf80c28e0 0x7fcdf81ece30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54190/0 (socket says 172.21.15.83:54190) 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfd08d640 1 -- 172.21.15.83:0/206656770 learned_addr learned my addr 172.21.15.83:0/206656770 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfd88e640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdf8071100 0x7fcdf8154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfe08f640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdf8151480 0x7fcdf814f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfd08d640 1 -- 172.21.15.83:0/206656770 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdf8071100 msgr2=0x7fcdf8154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfd08d640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdf8071100 0x7fcdf8154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfd08d640 1 -- 172.21.15.83:0/206656770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdf8151480 msgr2=0x7fcdf814f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfd08d640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdf8151480 0x7fcdf814f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfd08d640 1 -- 172.21.15.83:0/206656770 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf8156260 con 0x7fcdf80c28e0 2024-09-19T07:19:52.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfd88e640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdf8071100 0x7fcdf8154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:52.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdfd08d640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdf80c28e0 0x7fcdf81ece30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcde800c820 tx=0x7fcde800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:52.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdee7fc640 1 -- 172.21.15.83:0/206656770 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde8004230 con 0x7fcdf80c28e0 2024-09-19T07:19:52.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdee7fc640 1 -- 172.21.15.83:0/206656770 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcde80043d0 con 0x7fcdf80c28e0 2024-09-19T07:19:52.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 -- 172.21.15.83:0/206656770 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf8156540 con 0x7fcdf80c28e0 2024-09-19T07:19:52.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdee7fc640 1 -- 172.21.15.83:0/206656770 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde8007a70 con 0x7fcdf80c28e0 2024-09-19T07:19:52.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.459+0000 7fcdffaf2640 1 -- 172.21.15.83:0/206656770 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf8156b00 con 0x7fcdf80c28e0 2024-09-19T07:19:52.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.463+0000 7fcdffaf2640 1 -- 172.21.15.83:0/206656770 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdf8150f40 con 0x7fcdf80c28e0 2024-09-19T07:19:52.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.463+0000 7fcdee7fc640 1 -- 172.21.15.83:0/206656770 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcde8007c10 con 0x7fcdf80c28e0 2024-09-19T07:19:52.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.463+0000 7fcdee7fc640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcddc0678f0 0x7fcddc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.463+0000 7fcdee7fc640 1 -- 172.21.15.83:0/206656770 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fcde8085f40 con 0x7fcdf80c28e0 2024-09-19T07:19:52.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.463+0000 7fcdfd88e640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcddc0678f0 0x7fcddc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.463+0000 7fcdfd88e640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcddc0678f0 0x7fcddc069db0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40ad000 tx=0x7fcdf4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:52.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.467+0000 7fcdee7fc640 1 -- 172.21.15.83:0/206656770 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcde8015020 con 0x7fcdf80c28e0 2024-09-19T07:19:52.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.655+0000 7fcdffaf2640 1 -- 172.21.15.83:0/206656770 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdf80c4f30 con 0x7fcdf80c28e0 2024-09-19T07:19:52.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdee7fc640 1 -- 172.21.15.83:0/206656770 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7fcde80567e0 con 0x7fcdf80c28e0 2024-09-19T07:19:52.658 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:52.658 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:19:52.658 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:19:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdffaf2640 1 -- 172.21.15.83:0/206656770 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcddc0678f0 msgr2=0x7fcddc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdffaf2640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcddc0678f0 0x7fcddc069db0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40ad000 tx=0x7fcdf4008040 comp rx=0 tx=0).stop 2024-09-19T07:19:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdffaf2640 1 -- 172.21.15.83:0/206656770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdf80c28e0 msgr2=0x7fcdf81ece30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdffaf2640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdf80c28e0 0x7fcdf81ece30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcde800c820 tx=0x7fcde800cce0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdffaf2640 1 -- 172.21.15.83:0/206656770 shutdown_connections 2024-09-19T07:19:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdffaf2640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcddc0678f0 0x7fcddc069db0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdffaf2640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdf8151480 0x7fcdf814f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdffaf2640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdf80c28e0 0x7fcdf81ece30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdffaf2640 1 --2- 172.21.15.83:0/206656770 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdf8071100 0x7fcdf8154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdffaf2640 1 -- 172.21.15.83:0/206656770 >> 172.21.15.83:0/206656770 conn(0x7fcdf806bc80 msgr2=0x7fcdf8072f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdffaf2640 1 -- 172.21.15.83:0/206656770 shutdown_connections 2024-09-19T07:19:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.659+0000 7fcdffaf2640 1 -- 172.21.15.83:0/206656770 wait complete. 2024-09-19T07:19:52.675 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:52.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05ef59e640 1 Processor -- start 2024-09-19T07:19:52.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05ef59e640 1 -- start start 2024-09-19T07:19:52.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05ef59e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f05f0151090 0x7f05f00c3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05ef59e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05f00c3b00 con 0x7f05f0154080 2024-09-19T07:19:52.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05ef59e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05f00c3820 con 0x7f05f00ac730 2024-09-19T07:19:52.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05ef59e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f05f0071330 con 0x7f05f0151090 2024-09-19T07:19:52.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05ee59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f05f0151090 0x7f05f00c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05eed9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f05f0154080 0x7f05f015e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35534/0 (socket says 172.21.15.83:35534) 2024-09-19T07:19:52.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05eed9d640 1 -- 172.21.15.83:0/2135852309 learned_addr learned my addr 172.21.15.83:0/2135852309 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:52.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05ee59c640 1 -- 172.21.15.83:0/2135852309 >> v1:172.21.15.184:6789/0 conn(0x7f05f00ac730 legacy=0x7f05f0153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:52.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05ee59c640 1 -- 172.21.15.83:0/2135852309 >> v1:172.21.15.83:6789/0 conn(0x7f05f0154080 legacy=0x7f05f015e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05ee59c640 1 -- 172.21.15.83:0/2135852309 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f0070ce0 con 0x7f05f0151090 2024-09-19T07:19:52.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.783+0000 7f05ed59a640 1 -- 172.21.15.83:0/2135852309 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2078101221 0 0) 0x7f05f00c3b00 con 0x7f05f0154080 2024-09-19T07:19:52.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ee59c640 1 --2- 172.21.15.83:0/2135852309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05f0151090 0x7f05f00c3140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f05e00097c0 tx=0x7f05e001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20be77d6ee99b22 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:52.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ed59a640 1 -- 172.21.15.83:0/2135852309 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e00b0070 con 0x7f05f0151090 2024-09-19T07:19:52.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ed59a640 1 -- 172.21.15.83:0/2135852309 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05e00ab600 con 0x7f05f0151090 2024-09-19T07:19:52.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ed59a640 1 -- 172.21.15.83:0/2135852309 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e00a4e60 con 0x7f05f0151090 2024-09-19T07:19:52.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 -- 172.21.15.83:0/2135852309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05f0151090 msgr2=0x7f05f00c3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 --2- 172.21.15.83:0/2135852309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05f0151090 0x7f05f00c3140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f05e00097c0 tx=0x7f05e001c3d0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 -- 172.21.15.83:0/2135852309 shutdown_connections 2024-09-19T07:19:52.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 --2- 172.21.15.83:0/2135852309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05f0151090 0x7f05f00c3140 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 -- 172.21.15.83:0/2135852309 >> 172.21.15.83:0/2135852309 conn(0x7f05f00c7370 msgr2=0x7f05f00c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:52.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 -- 172.21.15.83:0/2135852309 shutdown_connections 2024-09-19T07:19:52.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 -- 172.21.15.83:0/2135852309 wait complete. 2024-09-19T07:19:52.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 Processor -- start 2024-09-19T07:19:52.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 -- start start 2024-09-19T07:19:52.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f05f00ac730 0x7f05f00c0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f05f0151090 0x7f05f0152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05f0154080 0x7f05f00aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f05f01515f0 con 0x7f05f0154080 2024-09-19T07:19:52.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f05f0151470 con 0x7f05f0151090 2024-09-19T07:19:52.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ef59e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f05f00c18c0 con 0x7f05f00ac730 2024-09-19T07:19:52.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05eed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05f0154080 0x7f05f00aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05eed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05f0154080 0x7f05f00aaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54216/0 (socket says 172.21.15.83:54216) 2024-09-19T07:19:52.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05eed9d640 1 -- 172.21.15.83:0/1651137556 learned_addr learned my addr 172.21.15.83:0/1651137556 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:52.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05edd9b640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f05f0151090 0x7f05f0152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05ee59c640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f05f00ac730 0x7f05f00c0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05eed9d640 1 -- 172.21.15.83:0/1651137556 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f05f00ac730 msgr2=0x7f05f00c0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05eed9d640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f05f00ac730 0x7f05f00c0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05eed9d640 1 -- 172.21.15.83:0/1651137556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f05f0151090 msgr2=0x7f05f0152d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05eed9d640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f05f0151090 0x7f05f0152d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05eed9d640 1 -- 172.21.15.83:0/1651137556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f00ac330 con 0x7f05f0154080 2024-09-19T07:19:52.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.787+0000 7f05edd9b640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f05f0151090 0x7f05f0152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:52.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.791+0000 7f05eed9d640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05f0154080 0x7f05f00aaee0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f05e800c6d0 tx=0x7f05e800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:52.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.791+0000 7f05deffd640 1 -- 172.21.15.83:0/1651137556 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e8018070 con 0x7f05f0154080 2024-09-19T07:19:52.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.791+0000 7f05deffd640 1 -- 172.21.15.83:0/1651137556 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05e8004d70 con 0x7f05f0154080 2024-09-19T07:19:52.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.791+0000 7f05deffd640 1 -- 172.21.15.83:0/1651137556 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e800a880 con 0x7f05f0154080 2024-09-19T07:19:52.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.791+0000 7f05ef59e640 1 -- 172.21.15.83:0/1651137556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05f00ac090 con 0x7f05f0154080 2024-09-19T07:19:52.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.791+0000 7f05ef59e640 1 -- 172.21.15.83:0/1651137556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05f0146670 con 0x7f05f0154080 2024-09-19T07:19:52.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.791+0000 7f05ef59e640 1 -- 172.21.15.83:0/1651137556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05f00c4f60 con 0x7f05f0154080 2024-09-19T07:19:52.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.791+0000 7f05deffd640 1 -- 172.21.15.83:0/1651137556 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f05e800ac30 con 0x7f05f0154080 2024-09-19T07:19:52.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.791+0000 7f05deffd640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05d00678c0 0x7f05d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:52.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.791+0000 7f05ee59c640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05d00678c0 0x7f05d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:52.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.791+0000 7f05deffd640 1 -- 172.21.15.83:0/1651137556 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f05e8085570 con 0x7f05f0154080 2024-09-19T07:19:52.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.791+0000 7f05ee59c640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05d00678c0 0x7f05d0069d80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f05e00ad000 tx=0x7f05e0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:52.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.795+0000 7f05deffd640 1 -- 172.21.15.83:0/1651137556 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f05e8055e10 con 0x7f05f0154080 2024-09-19T07:19:52.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.947+0000 7f05ef59e640 1 -- 172.21.15.83:0/1651137556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05f00c3b00 con 0x7f05f0154080 2024-09-19T07:19:52.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05deffd640 1 -- 172.21.15.83:0/1651137556 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+20451 (secure 0 0 0) 0x7f05e8055710 con 0x7f05f0154080 2024-09-19T07:19:52.951 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:52.951 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:41.334020+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05ef59e640 1 -- 172.21.15.83:0/1651137556 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05d00678c0 msgr2=0x7f05d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05ef59e640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05d00678c0 0x7f05d0069d80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f05e00ad000 tx=0x7f05e0008040 comp rx=0 tx=0).stop 2024-09-19T07:19:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05ef59e640 1 -- 172.21.15.83:0/1651137556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05f0154080 msgr2=0x7f05f00aaee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05ef59e640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05f0154080 0x7f05f00aaee0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f05e800c6d0 tx=0x7f05e800cb90 comp rx=0 tx=0).stop 2024-09-19T07:19:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05ef59e640 1 -- 172.21.15.83:0/1651137556 shutdown_connections 2024-09-19T07:19:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05ef59e640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05d00678c0 0x7f05d0069d80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05ef59e640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05f0154080 0x7f05f00aaee0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05ef59e640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f05f0151090 0x7f05f0152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05ef59e640 1 --2- 172.21.15.83:0/1651137556 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f05f00ac730 0x7f05f00c0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05ef59e640 1 -- 172.21.15.83:0/1651137556 >> 172.21.15.83:0/1651137556 conn(0x7f05f00c7370 msgr2=0x7f05f014eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:52.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05ef59e640 1 -- 172.21.15.83:0/1651137556 shutdown_connections 2024-09-19T07:19:52.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:52.951+0000 7f05ef59e640 1 -- 172.21.15.83:0/1651137556 wait complete. 2024-09-19T07:19:52.968 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:19:53.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.083+0000 7fd321eeb640 1 Processor -- start 2024-09-19T07:19:53.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.083+0000 7fd321eeb640 1 -- start start 2024-09-19T07:19:53.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd321eeb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd31c06de90 0x7fd31c0c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd321eeb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd31c0c4010 0x7fd31c0c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd321eeb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd31c14df80 con 0x7fd31c14dba0 2024-09-19T07:19:53.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd321eeb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd31c0c7510 con 0x7fd31c06de90 2024-09-19T07:19:53.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd321eeb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd31c0c8870 con 0x7fd31c0c4010 2024-09-19T07:19:53.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd31affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd31c06de90 0x7fd31c0c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd31affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd31c06de90 0x7fd31c0c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54232/0 (socket says 172.21.15.83:54232) 2024-09-19T07:19:53.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd31affd640 1 -- 172.21.15.83:0/197754848 learned_addr learned my addr 172.21.15.83:0/197754848 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:53.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd31bfff640 1 --2- 172.21.15.83:0/197754848 >> v2:172.21.15.184:3300/0 conn(0x7fd31c0c4010 0x7fd31c0c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd31affd640 1 -- 172.21.15.83:0/197754848 >> v2:172.21.15.184:3300/0 conn(0x7fd31c0c4010 msgr2=0x7fd31c0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:53.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd31affd640 1 --2- 172.21.15.83:0/197754848 >> v2:172.21.15.184:3300/0 conn(0x7fd31c0c4010 0x7fd31c0c44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd31affd640 1 -- 172.21.15.83:0/197754848 >> v1:172.21.15.184:6789/0 conn(0x7fd31c14dba0 legacy=0x7fd31c154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:53.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd31affd640 1 -- 172.21.15.83:0/197754848 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c0c7740 con 0x7fd31c06de90 2024-09-19T07:19:53.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.087+0000 7fd31bfff640 1 --2- 172.21.15.83:0/197754848 >> v2:172.21.15.184:3300/0 conn(0x7fd31c0c4010 0x7fd31c0c44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:53.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31affd640 1 --2- 172.21.15.83:0/197754848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd31c06de90 0x7fd31c0c8d00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd31c1518d0 tx=0x7fd31001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6ed008ac8092ac server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:53.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31a7fc640 1 -- 172.21.15.83:0/197754848 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3100b2070 con 0x7fd31c06de90 2024-09-19T07:19:53.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31a7fc640 1 -- 172.21.15.83:0/197754848 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3100a9070 con 0x7fd31c06de90 2024-09-19T07:19:53.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31a7fc640 1 -- 172.21.15.83:0/197754848 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3100ad880 con 0x7fd31c06de90 2024-09-19T07:19:53.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 -- 172.21.15.83:0/197754848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd31c06de90 msgr2=0x7fd31c0c8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 --2- 172.21.15.83:0/197754848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd31c06de90 0x7fd31c0c8d00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd31c1518d0 tx=0x7fd31001c450 comp rx=0 tx=0).stop 2024-09-19T07:19:53.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 -- 172.21.15.83:0/197754848 shutdown_connections 2024-09-19T07:19:53.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 --2- 172.21.15.83:0/197754848 >> v2:172.21.15.184:3300/0 conn(0x7fd31c0c4010 0x7fd31c0c44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 --2- 172.21.15.83:0/197754848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd31c06de90 0x7fd31c0c8d00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 -- 172.21.15.83:0/197754848 >> 172.21.15.83:0/197754848 conn(0x7fd31c06ff50 msgr2=0x7fd31c070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:53.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 -- 172.21.15.83:0/197754848 shutdown_connections 2024-09-19T07:19:53.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 -- 172.21.15.83:0/197754848 wait complete. 2024-09-19T07:19:53.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 Processor -- start 2024-09-19T07:19:53.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 -- start start 2024-09-19T07:19:53.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd31c06de90 0x7fd31c146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd31c0c4010 0x7fd31c143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd31c14dba0 0x7fd31c143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c0c3d30 con 0x7fd31c14dba0 2024-09-19T07:19:53.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c0c3bb0 con 0x7fd31c06de90 2024-09-19T07:19:53.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd321eeb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd31c0c9240 con 0x7fd31c0c4010 2024-09-19T07:19:53.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd31c14dba0 0x7fd31c143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd31c14dba0 0x7fd31c143fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54236/0 (socket says 172.21.15.83:54236) 2024-09-19T07:19:53.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31bfff640 1 -- 172.21.15.83:0/2940974361 learned_addr learned my addr 172.21.15.83:0/2940974361 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:53.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31b7fe640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd31c06de90 0x7fd31c146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31affd640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd31c0c4010 0x7fd31c143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31bfff640 1 -- 172.21.15.83:0/2940974361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd31c0c4010 msgr2=0x7fd31c143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31bfff640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd31c0c4010 0x7fd31c143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31bfff640 1 -- 172.21.15.83:0/2940974361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd31c06de90 msgr2=0x7fd31c146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31bfff640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd31c06de90 0x7fd31c146700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31bfff640 1 -- 172.21.15.83:0/2940974361 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c1432d0 con 0x7fd31c14dba0 2024-09-19T07:19:53.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.091+0000 7fd31b7fe640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd31c06de90 0x7fd31c146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:19:53.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.095+0000 7fd31bfff640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd31c14dba0 0x7fd31c143fd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd31400ce70 tx=0x7fd31400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:53.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.095+0000 7fd303fff640 1 -- 172.21.15.83:0/2940974361 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31401a070 con 0x7fd31c14dba0 2024-09-19T07:19:53.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.095+0000 7fd303fff640 1 -- 172.21.15.83:0/2940974361 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd31400d840 con 0x7fd31c14dba0 2024-09-19T07:19:53.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.095+0000 7fd321eeb640 1 -- 172.21.15.83:0/2940974361 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31c142aa0 con 0x7fd31c14dba0 2024-09-19T07:19:53.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.095+0000 7fd303fff640 1 -- 172.21.15.83:0/2940974361 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31400dc30 con 0x7fd31c14dba0 2024-09-19T07:19:53.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.095+0000 7fd321eeb640 1 -- 172.21.15.83:0/2940974361 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31c142d80 con 0x7fd31c14dba0 2024-09-19T07:19:53.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.095+0000 7fd321eeb640 1 -- 172.21.15.83:0/2940974361 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd31c150b80 con 0x7fd31c14dba0 2024-09-19T07:19:53.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.095+0000 7fd303fff640 1 -- 172.21.15.83:0/2940974361 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd31401d070 con 0x7fd31c14dba0 2024-09-19T07:19:53.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.095+0000 7fd303fff640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd3080678c0 0x7fd308069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.095+0000 7fd303fff640 1 -- 172.21.15.83:0/2940974361 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fd3140866f0 con 0x7fd31c14dba0 2024-09-19T07:19:53.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.095+0000 7fd31b7fe640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd3080678c0 0x7fd308069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.095+0000 7fd31b7fe640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd3080678c0 0x7fd308069d80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009890 tx=0x7fd30c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.099+0000 7fd303fff640 1 -- 172.21.15.83:0/2940974361 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd31c150b80 con 0x7fd31c14dba0 2024-09-19T07:19:53.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.295+0000 7fd321eeb640 1 -- 172.21.15.83:0/2940974361 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd31c0c8ce0 con 0x7fd31c14dba0 2024-09-19T07:19:53.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd303fff640 1 -- 172.21.15.83:0/2940974361 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7fd314056f90 con 0x7fd31c14dba0 2024-09-19T07:19:53.298 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:53.298 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:19:53.298 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:19:53.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd321eeb640 1 -- 172.21.15.83:0/2940974361 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd3080678c0 msgr2=0x7fd308069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd321eeb640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd3080678c0 0x7fd308069d80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009890 tx=0x7fd30c008040 comp rx=0 tx=0).stop 2024-09-19T07:19:53.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd321eeb640 1 -- 172.21.15.83:0/2940974361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd31c14dba0 msgr2=0x7fd31c143fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd321eeb640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd31c14dba0 0x7fd31c143fd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd31400ce70 tx=0x7fd31400a710 comp rx=0 tx=0).stop 2024-09-19T07:19:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd321eeb640 1 -- 172.21.15.83:0/2940974361 shutdown_connections 2024-09-19T07:19:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd321eeb640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd3080678c0 0x7fd308069d80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd321eeb640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd31c14dba0 0x7fd31c143fd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd321eeb640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd31c0c4010 0x7fd31c143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd321eeb640 1 --2- 172.21.15.83:0/2940974361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd31c06de90 0x7fd31c146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd321eeb640 1 -- 172.21.15.83:0/2940974361 >> 172.21.15.83:0/2940974361 conn(0x7fd31c06ff50 msgr2=0x7fd31c06e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd321eeb640 1 -- 172.21.15.83:0/2940974361 shutdown_connections 2024-09-19T07:19:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.299+0000 7fd321eeb640 1 -- 172.21.15.83:0/2940974361 wait complete. 2024-09-19T07:19:53.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-19T07:19:31.617640+0000', 'modified': '2024-09-19T07:19:35.648535+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4254}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4254': {'gid': 4254, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6835/1819616410', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 1819616410}, {'type': 'v1', 'addr': '172.21.15.83:6835', 'nonce': 1819616410}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'a', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4254, 'qdb_cluster': [4254]} 2024-09-19T07:19:53.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-19T07:19:53.720 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-19T07:19:53.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d4199640 1 Processor -- start 2024-09-19T07:19:53.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d4199640 1 -- start start 2024-09-19T07:19:53.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d4199640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35cc0c4270 0x7f35cc0c0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d4199640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f35cc0ac3f0 0x7f35cc0c3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d4199640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35cc15a320 con 0x7f35cc155f20 2024-09-19T07:19:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d4199640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f35cc159b00 con 0x7f35cc0ac3f0 2024-09-19T07:19:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d4199640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f35cc159d60 con 0x7f35cc0c4270 2024-09-19T07:19:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d2736640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f35cc0ac3f0 0x7f35cc0c3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d2736640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f35cc0ac3f0 0x7f35cc0c3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54242/0 (socket says 172.21.15.83:54242) 2024-09-19T07:19:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d2736640 1 -- 172.21.15.83:0/992489840 learned_addr learned my addr 172.21.15.83:0/992489840 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d1f35640 1 --2- 172.21.15.83:0/992489840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35cc0c4270 0x7f35cc0c0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d2736640 1 -- 172.21.15.83:0/992489840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35cc0c4270 msgr2=0x7f35cc0c0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d2736640 1 --2- 172.21.15.83:0/992489840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35cc0c4270 0x7f35cc0c0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d2736640 1 -- 172.21.15.83:0/992489840 >> v1:172.21.15.184:6789/0 conn(0x7f35cc155f20 legacy=0x7f35cc1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:53.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.854+0000 7f35d2736640 1 -- 172.21.15.83:0/992489840 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35cc15a040 con 0x7f35cc0ac3f0 2024-09-19T07:19:53.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d2736640 1 --2- 172.21.15.83:0/992489840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc0ac3f0 0x7f35cc0c3480 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009870 tx=0x7f35bc01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf447d9556129f19 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:53.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d0f33640 1 -- 172.21.15.83:0/992489840 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc0b0070 con 0x7f35cc0ac3f0 2024-09-19T07:19:53.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d0f33640 1 -- 172.21.15.83:0/992489840 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35bc0ab600 con 0x7f35cc0ac3f0 2024-09-19T07:19:53.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d0f33640 1 -- 172.21.15.83:0/992489840 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc0ab9d0 con 0x7f35cc0ac3f0 2024-09-19T07:19:53.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d4199640 1 -- 172.21.15.83:0/992489840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc0ac3f0 msgr2=0x7f35cc0c3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d4199640 1 --2- 172.21.15.83:0/992489840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc0ac3f0 0x7f35cc0c3480 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009870 tx=0x7f35bc01c490 comp rx=0 tx=0).stop 2024-09-19T07:19:53.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d4199640 1 -- 172.21.15.83:0/992489840 shutdown_connections 2024-09-19T07:19:53.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d4199640 1 --2- 172.21.15.83:0/992489840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc0ac3f0 0x7f35cc0c3480 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d4199640 1 --2- 172.21.15.83:0/992489840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35cc0c4270 0x7f35cc0c0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d4199640 1 -- 172.21.15.83:0/992489840 >> 172.21.15.83:0/992489840 conn(0x7f35cc06bc80 msgr2=0x7f35cc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:53.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d4199640 1 -- 172.21.15.83:0/992489840 shutdown_connections 2024-09-19T07:19:53.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d4199640 1 -- 172.21.15.83:0/992489840 wait complete. 2024-09-19T07:19:53.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d4199640 1 Processor -- start 2024-09-19T07:19:53.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.858+0000 7f35d4199640 1 -- start start 2024-09-19T07:19:53.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d4199640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35cc0ac3f0 0x7f35cc1f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d4199640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35cc0c4270 0x7f35cc0c25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d4199640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc155f20 0x7f35cc1ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d4199640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc0c3dd0 con 0x7f35cc155f20 2024-09-19T07:19:53.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d4199640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc0c4860 con 0x7f35cc0c4270 2024-09-19T07:19:53.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d4199640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f35cc162fa0 con 0x7f35cc0ac3f0 2024-09-19T07:19:53.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d2736640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc155f20 0x7f35cc1ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d2736640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc155f20 0x7f35cc1ee8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54250/0 (socket says 172.21.15.83:54250) 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d2736640 1 -- 172.21.15.83:0/3902592695 learned_addr learned my addr 172.21.15.83:0/3902592695 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d1734640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35cc0c4270 0x7f35cc0c25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d1f35640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35cc0ac3f0 0x7f35cc1f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d2736640 1 -- 172.21.15.83:0/3902592695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35cc0ac3f0 msgr2=0x7f35cc1f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d2736640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35cc0ac3f0 0x7f35cc1f2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d2736640 1 -- 172.21.15.83:0/3902592695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35cc0c4270 msgr2=0x7f35cc0c25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d2736640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35cc0c4270 0x7f35cc0c25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d2736640 1 -- 172.21.15.83:0/3902592695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35cc153040 con 0x7f35cc155f20 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d1734640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35cc0c4270 0x7f35cc0c25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d2736640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc155f20 0x7f35cc1ee8c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f35bc01ab40 tx=0x7f35bc019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35c27fc640 1 -- 172.21.15.83:0/3902592695 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc0bb070 con 0x7f35cc155f20 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35c27fc640 1 -- 172.21.15.83:0/3902592695 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35bc0b6640 con 0x7f35cc155f20 2024-09-19T07:19:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d4199640 1 -- 172.21.15.83:0/3902592695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35cc153320 con 0x7f35cc155f20 2024-09-19T07:19:53.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35c27fc640 1 -- 172.21.15.83:0/3902592695 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc0b0040 con 0x7f35cc155f20 2024-09-19T07:19:53.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d4199640 1 -- 172.21.15.83:0/3902592695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35cc1f2110 con 0x7f35cc155f20 2024-09-19T07:19:53.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35c27fc640 1 -- 172.21.15.83:0/3902592695 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f35bc0ab4c0 con 0x7f35cc155f20 2024-09-19T07:19:53.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35c27fc640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35b00678c0 0x7f35b0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35c27fc640 1 -- 172.21.15.83:0/3902592695 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f35bc11f920 con 0x7f35cc155f20 2024-09-19T07:19:53.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d1f35640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35b00678c0 0x7f35b0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d1f35640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35b00678c0 0x7f35b0069d80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f35cc153650 tx=0x7f35c4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:53.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d7255000 1 Processor -- start 2024-09-19T07:19:53.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.862+0000 7f35d7255000 1 -- start start 2024-09-19T07:19:53.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a6a2d0 0x5636f7a61630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a61b70 0x5636f7a61f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 0x5636f7a62870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5636f794bb60 con 0x5636f7a61b70 2024-09-19T07:19:53.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5636f78b2c70 con 0x5636f7a6a2d0 2024-09-19T07:19:53.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5636f7858ed0 con 0x5636f7a62490 2024-09-19T07:19:53.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1734640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a61b70 0x5636f7a61f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1734640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a61b70 0x5636f7a61f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54258/0 (socket says 172.21.15.83:54258) 2024-09-19T07:19:53.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1734640 1 -- 172.21.15.83:0/2296019403 learned_addr learned my addr 172.21.15.83:0/2296019403 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:53.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1f35640 1 --2- 172.21.15.83:0/2296019403 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a6a2d0 0x5636f7a61630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d2736640 1 --2- 172.21.15.83:0/2296019403 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 0x5636f7a62870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1734640 1 -- 172.21.15.83:0/2296019403 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 msgr2=0x5636f7a62870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:53.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1734640 1 --2- 172.21.15.83:0/2296019403 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 0x5636f7a62870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1734640 1 -- 172.21.15.83:0/2296019403 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a6a2d0 msgr2=0x5636f7a61630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1734640 1 --2- 172.21.15.83:0/2296019403 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a6a2d0 0x5636f7a61630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1734640 1 -- 172.21.15.83:0/2296019403 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636f786ae50 con 0x5636f7a61b70 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1f35640 1 --2- 172.21.15.83:0/2296019403 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a6a2d0 0x5636f7a61630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1734640 1 --2- 172.21.15.83:0/2296019403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a61b70 0x5636f7a61f50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f35c8004600 tx=0x7f35c800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e82ea9fb002ae4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35af7fe640 1 -- 172.21.15.83:0/2296019403 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8004810 con 0x5636f7a61b70 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35af7fe640 1 -- 172.21.15.83:0/2296019403 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c8006b60 con 0x5636f7a61b70 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35af7fe640 1 -- 172.21.15.83:0/2296019403 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8007030 con 0x5636f7a61b70 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 -- 172.21.15.83:0/2296019403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a61b70 msgr2=0x5636f7a61f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 --2- 172.21.15.83:0/2296019403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a61b70 0x5636f7a61f50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f35c8004600 tx=0x7f35c800d520 comp rx=0 tx=0).stop 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 -- 172.21.15.83:0/2296019403 shutdown_connections 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 --2- 172.21.15.83:0/2296019403 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 0x5636f7a62870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 --2- 172.21.15.83:0/2296019403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a61b70 0x5636f7a61f50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 --2- 172.21.15.83:0/2296019403 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a6a2d0 0x5636f7a61630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 -- 172.21.15.83:0/2296019403 >> 172.21.15.83:0/2296019403 conn(0x5636f7a9f830 msgr2=0x5636f7aa1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:53.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 -- 172.21.15.83:0/2296019403 shutdown_connections 2024-09-19T07:19:53.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 -- 172.21.15.83:0/2296019403 wait complete. 2024-09-19T07:19:53.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 -- start start 2024-09-19T07:19:53.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 Processor -- start 2024-09-19T07:19:53.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 0x5636f7a9dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 0x5636f7a9e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 0x5636f7a9fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5636f7914b00 con 0x5636f7a6a2d0 2024-09-19T07:19:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5636f788d360 con 0x5636f7a61b70 2024-09-19T07:19:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5636f788a2d0 con 0x5636f7a62490 2024-09-19T07:19:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d7255000 10 client.0 _ll_register_callbacks cb 0x5636f7a4f820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T07:19:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d2736640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 0x5636f7a9fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d2736640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 0x5636f7a9fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54268/0 (socket says 172.21.15.83:54268) 2024-09-19T07:19:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d2736640 1 -- 172.21.15.83:0/1032135594 learned_addr learned my addr 172.21.15.83:0/1032135594 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1f35640 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 0x5636f7a9dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d1734640 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 0x5636f7a9e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d2736640 1 -- 172.21.15.83:0/1032135594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 msgr2=0x5636f7a9e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d2736640 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 0x5636f7a9e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d2736640 1 -- 172.21.15.83:0/1032135594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 msgr2=0x5636f7a9dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d2736640 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 0x5636f7a9dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d2736640 1 -- 172.21.15.83:0/1032135594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636f7859120 con 0x5636f7a6a2d0 2024-09-19T07:19:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35d2736640 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 0x5636f7a9fc10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f35bc097e10 tx=0x7f35bc0ef4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9df4b103e20d71 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.866+0000 7f35acff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-19T07:19:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35acff9640 1 -- 172.21.15.83:0/1032135594 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc0f86a0 con 0x5636f7a6a2d0 2024-09-19T07:19:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35acff9640 1 -- 172.21.15.83:0/1032135594 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f35bc0efd20 con 0x5636f7a6a2d0 2024-09-19T07:19:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 -- 172.21.15.83:0/1032135594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35c8006b60 con 0x5636f7a6a2d0 2024-09-19T07:19:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35acff9640 1 -- 172.21.15.83:0/1032135594 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f35bc0eff20 con 0x5636f7a6a2d0 2024-09-19T07:19:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 10 client.4698 fetch_fsmap learned FSMap version 14 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 -- 172.21.15.83:0/1032135594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x5636f7852450 con 0x5636f7a6a2d0 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35acff9640 1 -- 172.21.15.83:0/1032135594 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c8006b60 con 0x5636f7a6a2d0 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35acff9640 1 -- 172.21.15.83:0/1032135594 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 14) ==== 4136+0+0 (secure 0 0 0) 0x7f35bc0b3070 con 0x5636f7a6a2d0 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 10 client.4698 fetch_fsmap finished waiting for FSMap version 14 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 10 client.4698 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 20 client.4698 populate_metadata read hostname 'smithi083' 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.83:6834/1819616410,v1:172.21.15.83:6835/1819616410] conn(0x5636f7aa0620 0x5636f7aa0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 4 client.4698 mds_command: new command op to 4254 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 -- 172.21.15.83:0/1032135594 --> [v2:172.21.15.83:6834/1819616410,v1:172.21.15.83:6835/1819616410] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636f784cc00 con 0x5636f7aa0620 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d1f35640 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.83:6834/1819616410,v1:172.21.15.83:6835/1819616410] conn(0x5636f7aa0620 0x5636f7aa0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d1f35640 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.83:6834/1819616410,v1:172.21.15.83:6835/1819616410] conn(0x5636f7aa0620 0x5636f7aa0a00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d7891add766d83d server_cookie=c3de6b756f4b4838 in_seq=0 out_seq=0 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35acff9640 10 client.4698 ms_handle_connect on v2:172.21.15.83:6834/1819616410 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35acff9640 1 -- 172.21.15.83:0/1032135594 <== mds.0 v2:172.21.15.83:6834/1819616410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5636f784cc00 con 0x5636f7aa0620 2024-09-19T07:19:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35acff9640 10 client.4698 handle_command_reply: tid=0 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 client.4698 shutdown 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 -- 172.21.15.83:0/1032135594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 msgr2=0x5636f7a9fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 0x5636f7a9fc10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f35bc097e10 tx=0x7f35bc0ef4a0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 -- 172.21.15.83:0/1032135594 shutdown_connections 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.83:6834/1819616410,v1:172.21.15.83:6835/1819616410] conn(0x5636f7aa0620 0x5636f7aa0a00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 0x5636f7a9fc10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 0x5636f7a9e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35acff9640 0 client.4698 ms_handle_reset on v2:172.21.15.83:6834/1819616410 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 --2- 172.21.15.83:0/1032135594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 0x5636f7a9dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 -- 172.21.15.83:0/1032135594 >> 172.21.15.83:0/1032135594 conn(0x5636f7a9f830 msgr2=0x5636f7a60d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 -- 172.21.15.83:0/1032135594 shutdown_connections 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 1 -- 172.21.15.83:0/1032135594 wait complete. 2024-09-19T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.870+0000 7f35d7255000 20 client.4698 trim_cache size 0 max 16384 2024-09-19T07:19:53.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 Processor -- start 2024-09-19T07:19:53.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 -- start start 2024-09-19T07:19:53.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 0x5636f7abc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a62490 0x5636f7ab1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a6a2d0 0x5636f7ab2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1fe9b0 con 0x5636f7a62490 2024-09-19T07:19:53.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc14f990 con 0x5636f7a61b70 2024-09-19T07:19:53.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f35cc1fc9e0 con 0x5636f7a6a2d0 2024-09-19T07:19:53.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d1734640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a62490 0x5636f7ab1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d1734640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a62490 0x5636f7ab1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54282/0 (socket says 172.21.15.83:54282) 2024-09-19T07:19:53.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d1734640 1 -- 172.21.15.83:0/1592731641 learned_addr learned my addr 172.21.15.83:0/1592731641 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:53.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d1f35640 1 --2- 172.21.15.83:0/1592731641 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 0x5636f7abc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d2736640 1 --2- 172.21.15.83:0/1592731641 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a6a2d0 0x5636f7ab2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d1734640 1 -- 172.21.15.83:0/1592731641 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a6a2d0 msgr2=0x5636f7ab2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:53.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d1734640 1 --2- 172.21.15.83:0/1592731641 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a6a2d0 0x5636f7ab2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d1734640 1 -- 172.21.15.83:0/1592731641 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 msgr2=0x5636f7abc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:53.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d1734640 1 --2- 172.21.15.83:0/1592731641 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 0x5636f7abc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d1734640 1 -- 172.21.15.83:0/1592731641 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636f7839fd0 con 0x5636f7a62490 2024-09-19T07:19:53.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d1734640 1 --2- 172.21.15.83:0/1592731641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a62490 0x5636f7ab1f40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f35c8007fd0 tx=0x7f35c8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9468d4e68b7738d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:53.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35aeffd640 1 -- 172.21.15.83:0/1592731641 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8006b60 con 0x5636f7a62490 2024-09-19T07:19:53.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35aeffd640 1 -- 172.21.15.83:0/1592731641 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c8006d20 con 0x5636f7a62490 2024-09-19T07:19:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35aeffd640 1 -- 172.21.15.83:0/1592731641 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8004a50 con 0x5636f7a62490 2024-09-19T07:19:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 -- 172.21.15.83:0/1592731641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a62490 msgr2=0x5636f7ab1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 --2- 172.21.15.83:0/1592731641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a62490 0x5636f7ab1f40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f35c8007fd0 tx=0x7f35c8012000 comp rx=0 tx=0).stop 2024-09-19T07:19:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 -- 172.21.15.83:0/1592731641 shutdown_connections 2024-09-19T07:19:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 --2- 172.21.15.83:0/1592731641 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a6a2d0 0x5636f7ab2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 --2- 172.21.15.83:0/1592731641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a62490 0x5636f7ab1f40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 --2- 172.21.15.83:0/1592731641 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 0x5636f7abc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 -- 172.21.15.83:0/1592731641 >> 172.21.15.83:0/1592731641 conn(0x5636f7a9f830 msgr2=0x5636f7a60c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 -- 172.21.15.83:0/1592731641 shutdown_connections 2024-09-19T07:19:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 -- 172.21.15.83:0/1592731641 wait complete. 2024-09-19T07:19:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 -- start start 2024-09-19T07:19:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.890+0000 7f35d7255000 1 Processor -- start 2024-09-19T07:19:53.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 0x5636f7ab2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 0x5636f7ab0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 0x5636f7ab0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5636f78b1f80 con 0x5636f7a6a2d0 2024-09-19T07:19:53.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5636f786aca0 con 0x5636f7a61b70 2024-09-19T07:19:53.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5636f78710f0 con 0x5636f7a62490 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 10 client.0 _ll_register_callbacks cb 0x7f35bc0094d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d2736640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 0x5636f7ab0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d2736640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 0x5636f7ab0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54286/0 (socket says 172.21.15.83:54286) 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d2736640 1 -- 172.21.15.83:0/931898737 learned_addr learned my addr 172.21.15.83:0/931898737 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d1f35640 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 0x5636f7ab2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d1734640 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 0x5636f7ab0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d2736640 1 -- 172.21.15.83:0/931898737 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 msgr2=0x5636f7ab0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d2736640 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 0x5636f7ab0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d2736640 1 -- 172.21.15.83:0/931898737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 msgr2=0x5636f7ab2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d2736640 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 0x5636f7ab2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d2736640 1 -- 172.21.15.83:0/931898737 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636f7935750 con 0x5636f7a6a2d0 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d2736640 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 0x5636f7ab0a40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f35bc01a0c0 tx=0x7f35bc01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940b29130140fbe4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:53.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35acff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-19T07:19:53.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35acff9640 1 -- 172.21.15.83:0/931898737 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc0ef600 con 0x5636f7a6a2d0 2024-09-19T07:19:53.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35acff9640 1 -- 172.21.15.83:0/931898737 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f35bc0eddc0 con 0x5636f7a6a2d0 2024-09-19T07:19:53.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 -- 172.21.15.83:0/931898737 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35c8006d20 con 0x5636f7a6a2d0 2024-09-19T07:19:53.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35acff9640 1 -- 172.21.15.83:0/931898737 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f35bc0f86a0 con 0x5636f7a6a2d0 2024-09-19T07:19:53.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 10 client.4704 fetch_fsmap learned FSMap version 14 2024-09-19T07:19:53.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 -- 172.21.15.83:0/931898737 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x5636f7840a30 con 0x5636f7a6a2d0 2024-09-19T07:19:53.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35acff9640 1 -- 172.21.15.83:0/931898737 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c8006d20 con 0x5636f7a6a2d0 2024-09-19T07:19:53.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35acff9640 1 -- 172.21.15.83:0/931898737 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 14) ==== 4136+0+0 (secure 0 0 0) 0x7f35bc0b3040 con 0x5636f7a6a2d0 2024-09-19T07:19:53.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 10 client.4704 fetch_fsmap finished waiting for FSMap version 14 2024-09-19T07:19:53.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 10 client.4704 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-19T07:19:53.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 20 client.4704 populate_metadata read hostname 'smithi083' 2024-09-19T07:19:53.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.83:6834/1819616410,v1:172.21.15.83:6835/1819616410] conn(0x5636f7ab0f80 0x5636f7ab1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 4 client.4704 mds_command: new command op to 4254 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 -- 172.21.15.83:0/931898737 --> [v2:172.21.15.83:6834/1819616410,v1:172.21.15.83:6835/1819616410] -- command(tid 0: {"prefix": "status"}) -- 0x5636f78c03a0 con 0x5636f7ab0f80 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d1f35640 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.83:6834/1819616410,v1:172.21.15.83:6835/1819616410] conn(0x5636f7ab0f80 0x5636f7ab1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35acff9640 10 client.4704 ms_handle_connect on v2:172.21.15.83:6834/1819616410 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d1f35640 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.83:6834/1819616410,v1:172.21.15.83:6835/1819616410] conn(0x5636f7ab0f80 0x5636f7ab1360 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b389be5c623d2197 server_cookie=9ff4f37e156a8db0 in_seq=0 out_seq=0 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35acff9640 1 -- 172.21.15.83:0/931898737 <== mds.0 v2:172.21.15.83:6834/1819616410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x5636f78c03a0 con 0x5636f7ab0f80 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35acff9640 10 client.4704 handle_command_reply: tid=0 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 client.4704 shutdown 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 -- 172.21.15.83:0/931898737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 msgr2=0x5636f7ab0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 0x5636f7ab0a40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f35bc01a0c0 tx=0x7f35bc01c490 comp rx=0 tx=0).stop 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 -- 172.21.15.83:0/931898737 shutdown_connections 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.83:6834/1819616410,v1:172.21.15.83:6835/1819616410] conn(0x5636f7ab0f80 0x5636f7ab1360 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5636f7a6a2d0 0x5636f7ab0a40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5636f7a62490 0x5636f7ab0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35acff9640 0 client.4704 ms_handle_reset on v2:172.21.15.83:6834/1819616410 2024-09-19T07:19:53.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 --2- 172.21.15.83:0/931898737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5636f7a61b70 0x5636f7ab2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 -- 172.21.15.83:0/931898737 >> 172.21.15.83:0/931898737 conn(0x5636f7a9f830 msgr2=0x5636f7a60c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:53.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 -- 172.21.15.83:0/931898737 shutdown_connections 2024-09-19T07:19:53.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 1 -- 172.21.15.83:0/931898737 wait complete. 2024-09-19T07:19:53.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d7255000 20 client.4704 trim_cache size 0 max 16384 2024-09-19T07:19:53.903 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-19T07:19:53.903 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "86ed7344-2eac-4937-889b-a144de44eb64", 2024-09-19T07:19:53.903 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-09-19T07:19:53.903 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 4254, 2024-09-19T07:19:53.904 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-09-19T07:19:53.904 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-09-19T07:19:53.904 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "a", 2024-09-19T07:19:53.904 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 21.613598307, 2024-09-19T07:19:53.904 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 8, 2024-09-19T07:19:53.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 25, 2024-09-19T07:19:53.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 21, 2024-09-19T07:19:53.904 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 27.713385510999998 2024-09-19T07:19:53.904 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-19T07:19:53.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d4199640 1 -- 172.21.15.83:0/3902592695 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35b00678c0 msgr2=0x7f35b0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d4199640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35b00678c0 0x7f35b0069d80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f35cc153650 tx=0x7f35c4008040 comp rx=0 tx=0).stop 2024-09-19T07:19:53.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d4199640 1 -- 172.21.15.83:0/3902592695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc155f20 msgr2=0x7f35cc1ee8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:53.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d4199640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc155f20 0x7f35cc1ee8c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f35bc01ab40 tx=0x7f35bc019230 comp rx=0 tx=0).stop 2024-09-19T07:19:53.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d4199640 1 -- 172.21.15.83:0/3902592695 shutdown_connections 2024-09-19T07:19:53.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d4199640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35b00678c0 0x7f35b0069d80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d4199640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc155f20 0x7f35cc1ee8c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d4199640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35cc0c4270 0x7f35cc0c25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d4199640 1 --2- 172.21.15.83:0/3902592695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35cc0ac3f0 0x7f35cc1f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:53.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d4199640 1 -- 172.21.15.83:0/3902592695 >> 172.21.15.83:0/3902592695 conn(0x7f35cc06bc80 msgr2=0x7f35cc1593a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:53.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d4199640 1 -- 172.21.15.83:0/3902592695 shutdown_connections 2024-09-19T07:19:53.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:53.894+0000 7f35d4199640 1 -- 172.21.15.83:0/3902592695 wait complete. 2024-09-19T07:19:53.912 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:19:54.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.018+0000 7fbe3b0be640 1 Processor -- start 2024-09-19T07:19:54.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.018+0000 7fbe3b0be640 1 -- start start 2024-09-19T07:19:54.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.018+0000 7fbe3b0be640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe34071100 0x7fbe341545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.018+0000 7fbe3b0be640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbe340c28e0 0x7fbe34150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.018+0000 7fbe3b0be640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbe34151480 0x7fbe3406dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.018+0000 7fbe3b0be640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fbe34071b50 con 0x7fbe340c28e0 2024-09-19T07:19:54.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.018+0000 7fbe3b0be640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbe340c1ff0 con 0x7fbe34151480 2024-09-19T07:19:54.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.018+0000 7fbe3b0be640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbe340a9e60 con 0x7fbe34071100 2024-09-19T07:19:54.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.018+0000 7fbe33fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbe340c28e0 0x7fbe34150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.018+0000 7fbe33fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbe340c28e0 0x7fbe34150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54288/0 (socket says 172.21.15.83:54288) 2024-09-19T07:19:54.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.018+0000 7fbe33fff640 1 -- 172.21.15.83:0/3645906821 learned_addr learned my addr 172.21.15.83:0/3645906821 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:54.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe33fff640 1 -- 172.21.15.83:0/3645906821 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe34071100 msgr2=0x7fbe341545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:54.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe33fff640 1 --2- 172.21.15.83:0/3645906821 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe34071100 0x7fbe341545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe33fff640 1 -- 172.21.15.83:0/3645906821 >> v2:172.21.15.184:3300/0 conn(0x7fbe34151480 msgr2=0x7fbe3406dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:54.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe33fff640 1 --2- 172.21.15.83:0/3645906821 >> v2:172.21.15.184:3300/0 conn(0x7fbe34151480 0x7fbe3406dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe33fff640 1 -- 172.21.15.83:0/3645906821 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe340aa140 con 0x7fbe340c28e0 2024-09-19T07:19:54.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe33fff640 1 --2- 172.21.15.83:0/3645906821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe340c28e0 0x7fbe34150f40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbe3406cd60 tx=0x7fbe2401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2e4f99d8213c43 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:54.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe337fe640 1 -- 172.21.15.83:0/3645906821 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe240a56d0 con 0x7fbe340c28e0 2024-09-19T07:19:54.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe337fe640 1 -- 172.21.15.83:0/3645906821 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe240a5870 con 0x7fbe340c28e0 2024-09-19T07:19:54.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe337fe640 1 -- 172.21.15.83:0/3645906821 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe240a5c40 con 0x7fbe340c28e0 2024-09-19T07:19:54.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe3b0be640 1 -- 172.21.15.83:0/3645906821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe340c28e0 msgr2=0x7fbe34150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe3b0be640 1 --2- 172.21.15.83:0/3645906821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe340c28e0 0x7fbe34150f40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbe3406cd60 tx=0x7fbe2401c960 comp rx=0 tx=0).stop 2024-09-19T07:19:54.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe3b0be640 1 -- 172.21.15.83:0/3645906821 shutdown_connections 2024-09-19T07:19:54.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe3b0be640 1 --2- 172.21.15.83:0/3645906821 >> v2:172.21.15.184:3300/0 conn(0x7fbe34151480 0x7fbe3406dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe3b0be640 1 --2- 172.21.15.83:0/3645906821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe340c28e0 0x7fbe34150f40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe3b0be640 1 --2- 172.21.15.83:0/3645906821 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe34071100 0x7fbe341545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe3b0be640 1 -- 172.21.15.83:0/3645906821 >> 172.21.15.83:0/3645906821 conn(0x7fbe3406bc80 msgr2=0x7fbe3406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:54.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe3b0be640 1 -- 172.21.15.83:0/3645906821 shutdown_connections 2024-09-19T07:19:54.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.022+0000 7fbe3b0be640 1 -- 172.21.15.83:0/3645906821 wait complete. 2024-09-19T07:19:54.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe3b0be640 1 Processor -- start 2024-09-19T07:19:54.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe3b0be640 1 -- start start 2024-09-19T07:19:54.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe3b0be640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe34071100 0x7fbe340c6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe3b0be640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe340c28e0 0x7fbe340bc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe3b0be640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe34151480 0x7fbe34154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe3b0be640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbe341570d0 con 0x7fbe34071100 2024-09-19T07:19:54.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe3b0be640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbe34071730 con 0x7fbe340c28e0 2024-09-19T07:19:54.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe3b0be640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbe34157250 con 0x7fbe34151480 2024-09-19T07:19:54.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe38e5a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe34071100 0x7fbe340c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe38e5a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe34071100 0x7fbe340c6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54292/0 (socket says 172.21.15.83:54292) 2024-09-19T07:19:54.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe38e5a640 1 -- 172.21.15.83:0/1853527053 learned_addr learned my addr 172.21.15.83:0/1853527053 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:54.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe33fff640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe340c28e0 0x7fbe340bc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe3965b640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe34151480 0x7fbe34154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe38e5a640 1 -- 172.21.15.83:0/1853527053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe34151480 msgr2=0x7fbe34154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe38e5a640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe34151480 0x7fbe34154cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe38e5a640 1 -- 172.21.15.83:0/1853527053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe340c28e0 msgr2=0x7fbe340bc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe38e5a640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe340c28e0 0x7fbe340bc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe38e5a640 1 -- 172.21.15.83:0/1853527053 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe340c6c40 con 0x7fbe34071100 2024-09-19T07:19:54.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe33fff640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe340c28e0 0x7fbe340bc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:54.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe38e5a640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe34071100 0x7fbe340c6700 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00b550 tx=0x7fbe1c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:54.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe317fa640 1 -- 172.21.15.83:0/1853527053 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c014070 con 0x7fbe34071100 2024-09-19T07:19:54.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe317fa640 1 -- 172.21.15.83:0/1853527053 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe1c010890 con 0x7fbe34071100 2024-09-19T07:19:54.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe3b0be640 1 -- 172.21.15.83:0/1853527053 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe341505d0 con 0x7fbe34071100 2024-09-19T07:19:54.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe317fa640 1 -- 172.21.15.83:0/1853527053 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c010c80 con 0x7fbe34071100 2024-09-19T07:19:54.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe3b0be640 1 -- 172.21.15.83:0/1853527053 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe3414f7f0 con 0x7fbe34071100 2024-09-19T07:19:54.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe3b0be640 1 -- 172.21.15.83:0/1853527053 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe34150f40 con 0x7fbe34071100 2024-09-19T07:19:54.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe317fa640 1 -- 172.21.15.83:0/1853527053 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbe1c017070 con 0x7fbe34071100 2024-09-19T07:19:54.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe317fa640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe140678c0 0x7fbe14069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe317fa640 1 -- 172.21.15.83:0/1853527053 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fbe1c085d00 con 0x7fbe34071100 2024-09-19T07:19:54.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.026+0000 7fbe33fff640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe140678c0 0x7fbe14069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.030+0000 7fbe33fff640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe140678c0 0x7fbe14069d80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbe240ae000 tx=0x7fbe240a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:54.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.030+0000 7fbe317fa640 1 -- 172.21.15.83:0/1853527053 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbe1c0565a0 con 0x7fbe34071100 2024-09-19T07:19:54.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.222+0000 7fbe3b0be640 1 -- 172.21.15.83:0/1853527053 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe340c4f30 con 0x7fbe34071100 2024-09-19T07:19:54.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.222+0000 7fbe317fa640 1 -- 172.21.15.83:0/1853527053 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7fbe1c010e20 con 0x7fbe34071100 2024-09-19T07:19:54.224 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:54.224 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:19:54.224 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:19:54.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.226+0000 7fbe3b0be640 1 -- 172.21.15.83:0/1853527053 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe140678c0 msgr2=0x7fbe14069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.226+0000 7fbe3b0be640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe140678c0 0x7fbe14069d80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbe240ae000 tx=0x7fbe240a7070 comp rx=0 tx=0).stop 2024-09-19T07:19:54.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.226+0000 7fbe3b0be640 1 -- 172.21.15.83:0/1853527053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe34071100 msgr2=0x7fbe340c6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.226+0000 7fbe3b0be640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe34071100 0x7fbe340c6700 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00b550 tx=0x7fbe1c00ba10 comp rx=0 tx=0).stop 2024-09-19T07:19:54.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.226+0000 7fbe3b0be640 1 -- 172.21.15.83:0/1853527053 shutdown_connections 2024-09-19T07:19:54.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.226+0000 7fbe3b0be640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe140678c0 0x7fbe14069d80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.226+0000 7fbe3b0be640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe34151480 0x7fbe34154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.226+0000 7fbe3b0be640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe340c28e0 0x7fbe340bc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.226+0000 7fbe3b0be640 1 --2- 172.21.15.83:0/1853527053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe34071100 0x7fbe340c6700 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.226+0000 7fbe3b0be640 1 -- 172.21.15.83:0/1853527053 >> 172.21.15.83:0/1853527053 conn(0x7fbe3406bc80 msgr2=0x7fbe34073c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:54.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.226+0000 7fbe3b0be640 1 -- 172.21.15.83:0/1853527053 shutdown_connections 2024-09-19T07:19:54.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.226+0000 7fbe3b0be640 1 -- 172.21.15.83:0/1853527053 wait complete. 2024-09-19T07:19:54.241 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:19:54.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.346+0000 7faf94e12640 1 Processor -- start 2024-09-19T07:19:54.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.346+0000 7faf94e12640 1 -- start start 2024-09-19T07:19:54.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf94e12640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7faf90152b20 0x7faf900c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf94e12640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf900c45c0 0x7faf90153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf94e12640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf90150670 con 0x7faf90150290 2024-09-19T07:19:54.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf94e12640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7faf900716c0 con 0x7faf90152b20 2024-09-19T07:19:54.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf94e12640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faf900c3920 con 0x7faf900c45c0 2024-09-19T07:19:54.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8ed9d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf900c45c0 0x7faf90153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8dd9b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7faf90152b20 0x7faf900c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8e59c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7faf90150290 0x7faf900c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:51174/0 (socket says 172.21.15.83:51174) 2024-09-19T07:19:54.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8dd9b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7faf90152b20 0x7faf900c4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:47238/0 (socket says 172.21.15.83:47238) 2024-09-19T07:19:54.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8e59c640 1 -- 172.21.15.83:0/3221009861 learned_addr learned my addr 172.21.15.83:0/3221009861 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:54.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8d59a640 1 -- 172.21.15.83:0/3221009861 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 513966062 0 0) 0x7faf90150670 con 0x7faf90150290 2024-09-19T07:19:54.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8d59a640 1 -- 172.21.15.83:0/3221009861 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf78003530 con 0x7faf90150290 2024-09-19T07:19:54.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8dd9b640 1 -- 172.21.15.83:0/3221009861 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf900c45c0 msgr2=0x7faf90153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:54.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8dd9b640 1 --2- 172.21.15.83:0/3221009861 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf900c45c0 0x7faf90153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8dd9b640 1 -- 172.21.15.83:0/3221009861 >> v1:172.21.15.184:6789/0 conn(0x7faf90150290 legacy=0x7faf900c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8dd9b640 1 -- 172.21.15.83:0/3221009861 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf900c2880 con 0x7faf90152b20 2024-09-19T07:19:54.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8ed9d640 1 --2- 172.21.15.83:0/3221009861 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf900c45c0 0x7faf90153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:19:54.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.350+0000 7faf8dd9b640 1 --2- 172.21.15.83:0/3221009861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90152b20 0x7faf900c4080 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faf840097f0 tx=0x7faf8401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bc43c6571a3d129 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:54.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8d59a640 1 -- 172.21.15.83:0/3221009861 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf840b0070 con 0x7faf90152b20 2024-09-19T07:19:54.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8d59a640 1 -- 172.21.15.83:0/3221009861 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf840ab600 con 0x7faf90152b20 2024-09-19T07:19:54.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8d59a640 1 -- 172.21.15.83:0/3221009861 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf840ab9b0 con 0x7faf90152b20 2024-09-19T07:19:54.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 -- 172.21.15.83:0/3221009861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90152b20 msgr2=0x7faf900c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 --2- 172.21.15.83:0/3221009861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90152b20 0x7faf900c4080 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faf840097f0 tx=0x7faf8401c4a0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 -- 172.21.15.83:0/3221009861 shutdown_connections 2024-09-19T07:19:54.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 --2- 172.21.15.83:0/3221009861 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf900c45c0 0x7faf90153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 --2- 172.21.15.83:0/3221009861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90152b20 0x7faf900c4080 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 -- 172.21.15.83:0/3221009861 >> 172.21.15.83:0/3221009861 conn(0x7faf9006bc80 msgr2=0x7faf9006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:54.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 -- 172.21.15.83:0/3221009861 shutdown_connections 2024-09-19T07:19:54.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 -- 172.21.15.83:0/3221009861 wait complete. 2024-09-19T07:19:54.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 Processor -- start 2024-09-19T07:19:54.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 -- start start 2024-09-19T07:19:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf900c45c0 0x7faf90149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf90150290 0x7faf901efab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90152b20 0x7faf901efff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf90154980 con 0x7faf900c45c0 2024-09-19T07:19:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faf90154800 con 0x7faf90152b20 2024-09-19T07:19:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faf9006e6b0 con 0x7faf90150290 2024-09-19T07:19:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf900c45c0 0x7faf90149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf900c45c0 0x7faf90149c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54308/0 (socket says 172.21.15.83:54308) 2024-09-19T07:19:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8e59c640 1 -- 172.21.15.83:0/3226864012 learned_addr learned my addr 172.21.15.83:0/3226864012 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8dd9b640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf90150290 0x7faf901efab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8ed9d640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90152b20 0x7faf901efff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8e59c640 1 -- 172.21.15.83:0/3226864012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf90150290 msgr2=0x7faf901efab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8e59c640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf90150290 0x7faf901efab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8e59c640 1 -- 172.21.15.83:0/3226864012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90152b20 msgr2=0x7faf901efff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8e59c640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90152b20 0x7faf901efff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8e59c640 1 -- 172.21.15.83:0/3226864012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf900ac910 con 0x7faf900c45c0 2024-09-19T07:19:54.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8ed9d640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90152b20 0x7faf901efff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:54.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf8e59c640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf900c45c0 0x7faf90149c00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faf8000c4d0 tx=0x7faf8000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:54.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf7effd640 1 -- 172.21.15.83:0/3226864012 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf80017070 con 0x7faf900c45c0 2024-09-19T07:19:54.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf7effd640 1 -- 172.21.15.83:0/3226864012 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf80005cf0 con 0x7faf900c45c0 2024-09-19T07:19:54.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf94e12640 1 -- 172.21.15.83:0/3226864012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf900c5640 con 0x7faf900c45c0 2024-09-19T07:19:54.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.354+0000 7faf7effd640 1 -- 172.21.15.83:0/3226864012 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf80007270 con 0x7faf900c45c0 2024-09-19T07:19:54.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.358+0000 7faf94e12640 1 -- 172.21.15.83:0/3226864012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf901f0530 con 0x7faf900c45c0 2024-09-19T07:19:54.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.358+0000 7faf94e12640 1 -- 172.21.15.83:0/3226864012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf9006ea40 con 0x7faf900c45c0 2024-09-19T07:19:54.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.358+0000 7faf7effd640 1 -- 172.21.15.83:0/3226864012 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf8001a070 con 0x7faf900c45c0 2024-09-19T07:19:54.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.358+0000 7faf7effd640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf700678c0 0x7faf70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.358+0000 7faf7effd640 1 -- 172.21.15.83:0/3226864012 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7faf80085790 con 0x7faf900c45c0 2024-09-19T07:19:54.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.358+0000 7faf8dd9b640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf700678c0 0x7faf70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.358+0000 7faf8dd9b640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf700678c0 0x7faf70069d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faf840ad000 tx=0x7faf84008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:54.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.362+0000 7faf7effd640 1 -- 172.21.15.83:0/3226864012 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7faf800559e0 con 0x7faf900c45c0 2024-09-19T07:19:54.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.526+0000 7faf94e12640 1 -- 172.21.15.83:0/3226864012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf900c53f0 con 0x7faf900c45c0 2024-09-19T07:19:54.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.526+0000 7faf7effd640 1 -- 172.21.15.83:0/3226864012 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+20451 (secure 0 0 0) 0x7faf80059ce0 con 0x7faf900c45c0 2024-09-19T07:19:54.528 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:54.528 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:41.334020+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:19:54.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.530+0000 7faf94e12640 1 -- 172.21.15.83:0/3226864012 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf700678c0 msgr2=0x7faf70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.530+0000 7faf94e12640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf700678c0 0x7faf70069d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faf840ad000 tx=0x7faf84008040 comp rx=0 tx=0).stop 2024-09-19T07:19:54.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.530+0000 7faf94e12640 1 -- 172.21.15.83:0/3226864012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf900c45c0 msgr2=0x7faf90149c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.530+0000 7faf94e12640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf900c45c0 0x7faf90149c00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faf8000c4d0 tx=0x7faf8000c990 comp rx=0 tx=0).stop 2024-09-19T07:19:54.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.530+0000 7faf94e12640 1 -- 172.21.15.83:0/3226864012 shutdown_connections 2024-09-19T07:19:54.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.530+0000 7faf94e12640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf700678c0 0x7faf70069d80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.530+0000 7faf94e12640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf90152b20 0x7faf901efff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.530+0000 7faf94e12640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf90150290 0x7faf901efab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.530+0000 7faf94e12640 1 --2- 172.21.15.83:0/3226864012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf900c45c0 0x7faf90149c00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.530+0000 7faf94e12640 1 -- 172.21.15.83:0/3226864012 >> 172.21.15.83:0/3226864012 conn(0x7faf9006bc80 msgr2=0x7faf9006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:54.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.530+0000 7faf94e12640 1 -- 172.21.15.83:0/3226864012 shutdown_connections 2024-09-19T07:19:54.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.530+0000 7faf94e12640 1 -- 172.21.15.83:0/3226864012 wait complete. 2024-09-19T07:19:54.547 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:19:54.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.670+0000 7fb3d55f7640 1 Processor -- start 2024-09-19T07:19:54.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.670+0000 7fb3d55f7640 1 -- start start 2024-09-19T07:19:54.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.670+0000 7fb3d55f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb3d00c6120 0x7fb3d00c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.670+0000 7fb3d55f7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb3d01592f0 0x7fb3d0157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.670+0000 7fb3d55f7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3d0156200 con 0x7fb3d0155d70 2024-09-19T07:19:54.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.670+0000 7fb3d55f7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb3d00c9930 con 0x7fb3d00c6120 2024-09-19T07:19:54.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.670+0000 7fb3d55f7640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fb3d01564e0 con 0x7fb3d01592f0 2024-09-19T07:19:54.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.670+0000 7fb3ceffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb3d00c6120 0x7fb3d00c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3ceffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb3d00c6120 0x7fb3d00c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54310/0 (socket says 172.21.15.83:54310) 2024-09-19T07:19:54.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3ceffd640 1 -- 172.21.15.83:0/1129117186 learned_addr learned my addr 172.21.15.83:0/1129117186 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:54.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3cf7fe640 1 --2- 172.21.15.83:0/1129117186 >> v2:172.21.15.184:3300/0 conn(0x7fb3d01592f0 0x7fb3d0157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3ceffd640 1 -- 172.21.15.83:0/1129117186 >> v2:172.21.15.184:3300/0 conn(0x7fb3d01592f0 msgr2=0x7fb3d0157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:54.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3ceffd640 1 --2- 172.21.15.83:0/1129117186 >> v2:172.21.15.184:3300/0 conn(0x7fb3d01592f0 0x7fb3d0157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3ceffd640 1 -- 172.21.15.83:0/1129117186 >> v1:172.21.15.184:6789/0 conn(0x7fb3d0155d70 legacy=0x7fb3d01575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:54.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3ceffd640 1 -- 172.21.15.83:0/1129117186 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d01567c0 con 0x7fb3d00c6120 2024-09-19T07:19:54.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3ceffd640 1 --2- 172.21.15.83:0/1129117186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3d00c6120 0x7fb3d00c6500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb3d00b1990 tx=0x7fb3c001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bdad0c0461399d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:54.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3cdffb640 1 -- 172.21.15.83:0/1129117186 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c00b0070 con 0x7fb3d00c6120 2024-09-19T07:19:54.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3cdffb640 1 -- 172.21.15.83:0/1129117186 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c00ab600 con 0x7fb3d00c6120 2024-09-19T07:19:54.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3cdffb640 1 -- 172.21.15.83:0/1129117186 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c00ab9d0 con 0x7fb3d00c6120 2024-09-19T07:19:54.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3d55f7640 1 -- 172.21.15.83:0/1129117186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3d00c6120 msgr2=0x7fb3d00c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3d55f7640 1 --2- 172.21.15.83:0/1129117186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3d00c6120 0x7fb3d00c6500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb3d00b1990 tx=0x7fb3c001c4e0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3d55f7640 1 -- 172.21.15.83:0/1129117186 shutdown_connections 2024-09-19T07:19:54.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3d55f7640 1 --2- 172.21.15.83:0/1129117186 >> v2:172.21.15.184:3300/0 conn(0x7fb3d01592f0 0x7fb3d0157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3d55f7640 1 --2- 172.21.15.83:0/1129117186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3d00c6120 0x7fb3d00c6500 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3d55f7640 1 -- 172.21.15.83:0/1129117186 >> 172.21.15.83:0/1129117186 conn(0x7fb3d006bc80 msgr2=0x7fb3d006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:54.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3d55f7640 1 -- 172.21.15.83:0/1129117186 shutdown_connections 2024-09-19T07:19:54.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.674+0000 7fb3d55f7640 1 -- 172.21.15.83:0/1129117186 wait complete. 2024-09-19T07:19:54.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3d55f7640 1 Processor -- start 2024-09-19T07:19:54.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3d55f7640 1 -- start start 2024-09-19T07:19:54.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3d55f7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3d00c6120 0x7fb3d014ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3d55f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3d01592f0 0x7fb3d01f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3d55f7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3d00c5040 0x7fb3d01f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3d55f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d0153860 con 0x7fb3d01592f0 2024-09-19T07:19:54.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3d55f7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d0162ea0 con 0x7fb3d00c5040 2024-09-19T07:19:54.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3d55f7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb3d00c41c0 con 0x7fb3d00c6120 2024-09-19T07:19:54.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3ce7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3d01592f0 0x7fb3d01f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3ce7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3d01592f0 0x7fb3d01f20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54320/0 (socket says 172.21.15.83:54320) 2024-09-19T07:19:54.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3ce7fc640 1 -- 172.21.15.83:0/3939004717 learned_addr learned my addr 172.21.15.83:0/3939004717 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:54.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3ceffd640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3d00c6120 0x7fb3d014ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3cf7fe640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3d00c5040 0x7fb3d01f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3ce7fc640 1 -- 172.21.15.83:0/3939004717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3d00c6120 msgr2=0x7fb3d014ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3ce7fc640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3d00c6120 0x7fb3d014ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3ce7fc640 1 -- 172.21.15.83:0/3939004717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3d00c5040 msgr2=0x7fb3d01f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3ce7fc640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3d00c5040 0x7fb3d01f25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3ce7fc640 1 -- 172.21.15.83:0/3939004717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d00c2870 con 0x7fb3d01592f0 2024-09-19T07:19:54.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3cf7fe640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3d00c5040 0x7fb3d01f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:54.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3ce7fc640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3d01592f0 0x7fb3d01f20b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400c450 tx=0x7fb3c400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:54.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3b77fe640 1 -- 172.21.15.83:0/3939004717 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4012830 con 0x7fb3d01592f0 2024-09-19T07:19:54.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3b77fe640 1 -- 172.21.15.83:0/3939004717 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c4017070 con 0x7fb3d01592f0 2024-09-19T07:19:54.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3d55f7640 1 -- 172.21.15.83:0/3939004717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d00c59f0 con 0x7fb3d01592f0 2024-09-19T07:19:54.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3b77fe640 1 -- 172.21.15.83:0/3939004717 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4012c30 con 0x7fb3d01592f0 2024-09-19T07:19:54.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3d55f7640 1 -- 172.21.15.83:0/3939004717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d014cf60 con 0x7fb3d01592f0 2024-09-19T07:19:54.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3d55f7640 1 -- 172.21.15.83:0/3939004717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3d00c6500 con 0x7fb3d01592f0 2024-09-19T07:19:54.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.678+0000 7fb3b77fe640 1 -- 172.21.15.83:0/3939004717 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb3c401c460 con 0x7fb3d01592f0 2024-09-19T07:19:54.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.682+0000 7fb3b77fe640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb3bc0678c0 0x7fb3bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:54.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.682+0000 7fb3ceffd640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb3bc0678c0 0x7fb3bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:54.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.682+0000 7fb3b77fe640 1 -- 172.21.15.83:0/3939004717 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fb3c4085ca0 con 0x7fb3d01592f0 2024-09-19T07:19:54.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.682+0000 7fb3ceffd640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb3bc0678c0 0x7fb3bc069d80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00ad000 tx=0x7fb3c00a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:54.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.686+0000 7fb3b77fe640 1 -- 172.21.15.83:0/3939004717 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb3c4056540 con 0x7fb3d01592f0 2024-09-19T07:19:54.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.874+0000 7fb3d55f7640 1 -- 172.21.15.83:0/3939004717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3d0149ab0 con 0x7fb3d01592f0 2024-09-19T07:19:54.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.874+0000 7fb3b77fe640 1 -- 172.21.15.83:0/3939004717 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7fb3c4055e40 con 0x7fb3d01592f0 2024-09-19T07:19:54.876 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:19:54.876 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:19:54.876 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:19:54.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.874+0000 7fb3d55f7640 1 -- 172.21.15.83:0/3939004717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb3bc0678c0 msgr2=0x7fb3bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.874+0000 7fb3d55f7640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb3bc0678c0 0x7fb3bc069d80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00ad000 tx=0x7fb3c00a6000 comp rx=0 tx=0).stop 2024-09-19T07:19:54.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.874+0000 7fb3d55f7640 1 -- 172.21.15.83:0/3939004717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3d01592f0 msgr2=0x7fb3d01f20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:54.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.874+0000 7fb3d55f7640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3d01592f0 0x7fb3d01f20b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400c450 tx=0x7fb3c400c910 comp rx=0 tx=0).stop 2024-09-19T07:19:54.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.878+0000 7fb3d55f7640 1 -- 172.21.15.83:0/3939004717 shutdown_connections 2024-09-19T07:19:54.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.878+0000 7fb3d55f7640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb3bc0678c0 0x7fb3bc069d80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.878+0000 7fb3d55f7640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3d00c5040 0x7fb3d01f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.878+0000 7fb3d55f7640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3d01592f0 0x7fb3d01f20b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.878+0000 7fb3d55f7640 1 --2- 172.21.15.83:0/3939004717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3d00c6120 0x7fb3d014ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:54.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.878+0000 7fb3d55f7640 1 -- 172.21.15.83:0/3939004717 >> 172.21.15.83:0/3939004717 conn(0x7fb3d006bc80 msgr2=0x7fb3d01575d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:54.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.878+0000 7fb3d55f7640 1 -- 172.21.15.83:0/3939004717 shutdown_connections 2024-09-19T07:19:54.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:54.878+0000 7fb3d55f7640 1 -- 172.21.15.83:0/3939004717 wait complete. 2024-09-19T07:19:55.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-19T07:19:39.753527+0000', 'modified': '2024-09-19T07:19:43.800486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4232}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4232': {'gid': 4232, 'name': 'b', 'rank': 0, 'incarnation': 9, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.184:6835/149004692', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6834', 'nonce': 149004692}, {'type': 'v1', 'addr': '172.21.15.184:6835', 'nonce': 149004692}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'b', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4232, 'qdb_cluster': [4232]} 2024-09-19T07:19:55.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-19T07:19:55.296 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-19T07:19:55.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abfbc2640 1 Processor -- start 2024-09-19T07:19:55.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abfbc2640 1 -- start start 2024-09-19T07:19:55.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abfbc2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ab8150060 0x7f1ab80c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abfbc2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1ab81534b0 0x7f1ab80c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abfbc2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1ab80c4550 0x7f1ab80c2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abfbc2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f1ab8150440 con 0x7f1ab80c4550 2024-09-19T07:19:55.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abfbc2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f1ab81506c0 con 0x7f1ab81534b0 2024-09-19T07:19:55.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abfbc2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab81539c0 con 0x7f1ab8150060 2024-09-19T07:19:55.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abe15f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1ab80c4550 0x7f1ab80c2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abe15f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1ab80c4550 0x7f1ab80c2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54328/0 (socket says 172.21.15.83:54328) 2024-09-19T07:19:55.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abe15f640 1 -- 172.21.15.83:0/671652296 learned_addr learned my addr 172.21.15.83:0/671652296 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:55.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abd95e640 1 --2- 172.21.15.83:0/671652296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ab8150060 0x7f1ab80c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abd15d640 1 --2- 172.21.15.83:0/671652296 >> v2:172.21.15.184:3300/0 conn(0x7f1ab81534b0 0x7f1ab80c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abe15f640 1 -- 172.21.15.83:0/671652296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ab8150060 msgr2=0x7f1ab80c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:55.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abe15f640 1 --2- 172.21.15.83:0/671652296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ab8150060 0x7f1ab80c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abe15f640 1 -- 172.21.15.83:0/671652296 >> v2:172.21.15.184:3300/0 conn(0x7f1ab81534b0 msgr2=0x7f1ab80c25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:55.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abe15f640 1 --2- 172.21.15.83:0/671652296 >> v2:172.21.15.184:3300/0 conn(0x7f1ab81534b0 0x7f1ab80c25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abe15f640 1 -- 172.21.15.83:0/671652296 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab80c3810 con 0x7f1ab80c4550 2024-09-19T07:19:55.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.430+0000 7f1abd15d640 1 --2- 172.21.15.83:0/671652296 >> v2:172.21.15.184:3300/0 conn(0x7f1ab81534b0 0x7f1ab80c25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:55.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abe15f640 1 --2- 172.21.15.83:0/671652296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ab80c4550 0x7f1ab80c2b10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1ab806f2f0 tx=0x7f1ab001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1430d643d3c31205 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:55.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abc95c640 1 -- 172.21.15.83:0/671652296 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00b0070 con 0x7f1ab80c4550 2024-09-19T07:19:55.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abc95c640 1 -- 172.21.15.83:0/671652296 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab00ab600 con 0x7f1ab80c4550 2024-09-19T07:19:55.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abc95c640 1 -- 172.21.15.83:0/671652296 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00ab9d0 con 0x7f1ab80c4550 2024-09-19T07:19:55.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abfbc2640 1 -- 172.21.15.83:0/671652296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ab80c4550 msgr2=0x7f1ab80c2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abfbc2640 1 --2- 172.21.15.83:0/671652296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ab80c4550 0x7f1ab80c2b10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1ab806f2f0 tx=0x7f1ab001c490 comp rx=0 tx=0).stop 2024-09-19T07:19:55.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abfbc2640 1 -- 172.21.15.83:0/671652296 shutdown_connections 2024-09-19T07:19:55.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abfbc2640 1 --2- 172.21.15.83:0/671652296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ab80c4550 0x7f1ab80c2b10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abfbc2640 1 --2- 172.21.15.83:0/671652296 >> v2:172.21.15.184:3300/0 conn(0x7f1ab81534b0 0x7f1ab80c25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abfbc2640 1 --2- 172.21.15.83:0/671652296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ab8150060 0x7f1ab80c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abfbc2640 1 -- 172.21.15.83:0/671652296 >> 172.21.15.83:0/671652296 conn(0x7f1ab806bc80 msgr2=0x7f1ab806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:55.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abfbc2640 1 -- 172.21.15.83:0/671652296 shutdown_connections 2024-09-19T07:19:55.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abfbc2640 1 -- 172.21.15.83:0/671652296 wait complete. 2024-09-19T07:19:55.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abfbc2640 1 Processor -- start 2024-09-19T07:19:55.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.434+0000 7f1abfbc2640 1 -- start start 2024-09-19T07:19:55.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abfbc2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ab80c4550 0x7f1ab80c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abfbc2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ab8150060 0x7f1ab81513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abfbc2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ab81534b0 0x7f1ab81f1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abfbc2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab80c1ab0 con 0x7f1ab80c4550 2024-09-19T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abfbc2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab80c3d90 con 0x7f1ab81534b0 2024-09-19T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abfbc2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab80c1c30 con 0x7f1ab8150060 2024-09-19T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd95e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ab80c4550 0x7f1ab80c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd95e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ab80c4550 0x7f1ab80c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54334/0 (socket says 172.21.15.83:54334) 2024-09-19T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd95e640 1 -- 172.21.15.83:0/96329969 learned_addr learned my addr 172.21.15.83:0/96329969 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abe15f640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ab81534b0 0x7f1ab81f1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd15d640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ab8150060 0x7f1ab81513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd95e640 1 -- 172.21.15.83:0/96329969 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ab8150060 msgr2=0x7f1ab81513d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd95e640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ab8150060 0x7f1ab81513d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd95e640 1 -- 172.21.15.83:0/96329969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ab81534b0 msgr2=0x7f1ab81f1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd95e640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ab81534b0 0x7f1ab81f1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd95e640 1 -- 172.21.15.83:0/96329969 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab80c6610 con 0x7f1ab80c4550 2024-09-19T07:19:55.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd15d640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ab8150060 0x7f1ab81513d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:55.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd95e640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ab80c4550 0x7f1ab80c60d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800cb80 tx=0x7f1aa800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:55.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1ab67fc640 1 -- 172.21.15.83:0/96329969 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8015070 con 0x7f1ab80c4550 2024-09-19T07:19:55.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1ab67fc640 1 -- 172.21.15.83:0/96329969 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa8003ba0 con 0x7f1ab80c4550 2024-09-19T07:19:55.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abfbc2640 1 -- 172.21.15.83:0/96329969 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab8152730 con 0x7f1ab80c4550 2024-09-19T07:19:55.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1ab67fc640 1 -- 172.21.15.83:0/96329969 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa800d040 con 0x7f1ab80c4550 2024-09-19T07:19:55.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abfbc2640 1 -- 172.21.15.83:0/96329969 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab80c55c0 con 0x7f1ab80c4550 2024-09-19T07:19:55.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1ab67fc640 1 -- 172.21.15.83:0/96329969 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1aa800a8b0 con 0x7f1ab80c4550 2024-09-19T07:19:55.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1ab67fc640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1aa40678c0 0x7f1aa4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1ab67fc640 1 -- 172.21.15.83:0/96329969 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f1aa8085e70 con 0x7f1ab80c4550 2024-09-19T07:19:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd15d640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1aa40678c0 0x7f1aa4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.438+0000 7f1abd15d640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1aa40678c0 0x7f1aa4069d80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8073b60 tx=0x7f1aac008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 Processor -- start 2024-09-19T07:19:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 -- start start 2024-09-19T07:19:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128df2d0 0x5655128d6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x5655128d6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d7490 0x5655128d7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5655127c0b60 con 0x5655128d7490 2024-09-19T07:19:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x565512727c70 con 0x5655128d6b70 2024-09-19T07:19:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5655126cded0 con 0x5655128df2d0 2024-09-19T07:19:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abe15f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d7490 0x5655128d7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abe15f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d7490 0x5655128d7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54336/0 (socket says 172.21.15.83:54336) 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abe15f640 1 -- 172.21.15.83:0/1920295314 learned_addr learned my addr 172.21.15.83:0/1920295314 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abd95e640 1 --2- 172.21.15.83:0/1920295314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128df2d0 0x5655128d6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abd15d640 1 --2- 172.21.15.83:0/1920295314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x5655128d6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abe15f640 1 -- 172.21.15.83:0/1920295314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128df2d0 msgr2=0x5655128d6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abe15f640 1 --2- 172.21.15.83:0/1920295314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128df2d0 0x5655128d6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abe15f640 1 -- 172.21.15.83:0/1920295314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 msgr2=0x5655128d6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abe15f640 1 --2- 172.21.15.83:0/1920295314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x5655128d6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abe15f640 1 -- 172.21.15.83:0/1920295314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5655126dfe50 con 0x5655128d7490 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abd95e640 1 --2- 172.21.15.83:0/1920295314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128df2d0 0x5655128d6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abd15d640 1 --2- 172.21.15.83:0/1920295314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x5655128d6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1abe15f640 1 --2- 172.21.15.83:0/1920295314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d7490 0x5655128d7870 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1ab001c970 tx=0x7f1ab0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf8715877d4e1ad server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1a9f7fe640 1 -- 172.21.15.83:0/1920295314 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00b0040 con 0x5655128d7490 2024-09-19T07:19:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1a9f7fe640 1 -- 172.21.15.83:0/1920295314 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab00ab700 con 0x5655128d7490 2024-09-19T07:19:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1a9f7fe640 1 -- 172.21.15.83:0/1920295314 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00abab0 con 0x5655128d7490 2024-09-19T07:19:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/1920295314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d7490 msgr2=0x5655128d7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/1920295314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d7490 0x5655128d7870 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1ab001c970 tx=0x7f1ab0004740 comp rx=0 tx=0).stop 2024-09-19T07:19:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/1920295314 shutdown_connections 2024-09-19T07:19:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/1920295314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d7490 0x5655128d7870 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/1920295314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x5655128d6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/1920295314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128df2d0 0x5655128d6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/1920295314 >> 172.21.15.83:0/1920295314 conn(0x565512914830 msgr2=0x565512916c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/1920295314 shutdown_connections 2024-09-19T07:19:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/1920295314 wait complete. 2024-09-19T07:19:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 -- start start 2024-09-19T07:19:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 Processor -- start 2024-09-19T07:19:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x565512912fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.442+0000 7f1ac2c7e000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 0x565512913520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 0x565512913a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x565512789b00 con 0x5655128df2d0 2024-09-19T07:19:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x565512702360 con 0x5655128d6b70 2024-09-19T07:19:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5655126ff2d0 con 0x5655128d7490 2024-09-19T07:19:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 10 client.0 _ll_register_callbacks cb 0x5655128c4820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T07:19:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abe15f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 0x565512913a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abe15f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 0x565512913a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54344/0 (socket says 172.21.15.83:54344) 2024-09-19T07:19:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abe15f640 1 -- 172.21.15.83:0/237302672 learned_addr learned my addr 172.21.15.83:0/237302672 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abd15d640 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 0x565512913520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abd95e640 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x565512912fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abe15f640 1 -- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 msgr2=0x565512913520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abe15f640 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 0x565512913520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abe15f640 1 -- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 msgr2=0x565512912fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abe15f640 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x565512912fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abe15f640 1 -- 172.21.15.83:0/237302672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5655126ce120 con 0x5655128df2d0 2024-09-19T07:19:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abd15d640 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 0x565512913520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abd95e640 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x565512912fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:19:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abe15f640 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 0x565512913a60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1ab001ab40 tx=0x7f1ab001acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a09391c1f0d29d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1a9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-19T07:19:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1a9cff9640 1 -- 172.21.15.83:0/237302672 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00b0040 con 0x5655128df2d0 2024-09-19T07:19:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1a9cff9640 1 -- 172.21.15.83:0/237302672 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f1ab00abab0 con 0x5655128df2d0 2024-09-19T07:19:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/237302672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ab00ab700 con 0x5655128df2d0 2024-09-19T07:19:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1a9cff9640 1 -- 172.21.15.83:0/237302672 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f1ab00cf050 con 0x5655128df2d0 2024-09-19T07:19:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 10 client.4731 fetch_fsmap learned FSMap version 14 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/237302672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x5655126c7450 con 0x5655128df2d0 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1a9cff9640 1 -- 172.21.15.83:0/237302672 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab00ab700 con 0x5655128df2d0 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1a9cff9640 1 -- 172.21.15.83:0/237302672 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 14) ==== 4136+0+0 (secure 0 0 0) 0x7f1ab00d2020 con 0x5655128df2d0 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 10 client.4731 fetch_fsmap finished waiting for FSMap version 14 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 10 client.4731 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 20 client.4731 populate_metadata read hostname 'smithi083' 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:6834/149004692,v1:172.21.15.184:6835/149004692] conn(0x5655129292b0 0x56551292b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 4 client.4731 mds_command: new command op to 4232 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/237302672 --> [v2:172.21.15.184:6834/149004692,v1:172.21.15.184:6835/149004692] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5655126c1c00 con 0x5655129292b0 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abd95e640 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:6834/149004692,v1:172.21.15.184:6835/149004692] conn(0x5655129292b0 0x56551292b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1a9cff9640 10 client.4731 ms_handle_connect on v2:172.21.15.184:6834/149004692 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1abd95e640 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:6834/149004692,v1:172.21.15.184:6835/149004692] conn(0x5655129292b0 0x56551292b6a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a98e77096a1b005c server_cookie=4da2d6d84503f8ab in_seq=0 out_seq=0 2024-09-19T07:19:55.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1a9cff9640 1 -- 172.21.15.83:0/237302672 <== mds.0 v2:172.21.15.184:6834/149004692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5655126c1c00 con 0x5655129292b0 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1a9cff9640 10 client.4731 handle_command_reply: tid=0 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.446+0000 7f1ac2c7e000 1 client.4731 shutdown 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.450+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/237302672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 msgr2=0x565512913a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.450+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 0x565512913a60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1ab001ab40 tx=0x7f1ab001acd0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.450+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/237302672 shutdown_connections 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.450+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:6834/149004692,v1:172.21.15.184:6835/149004692] conn(0x5655129292b0 0x56551292b6a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.450+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 0x565512913a60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.450+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 0x565512913520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.450+0000 7f1a9cff9640 0 client.4731 ms_handle_reset on v2:172.21.15.184:6834/149004692 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.450+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/237302672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x565512912fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.450+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/237302672 >> 172.21.15.83:0/237302672 conn(0x565512914830 msgr2=0x5655128d5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.450+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/237302672 shutdown_connections 2024-09-19T07:19:55.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.450+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/237302672 wait complete. 2024-09-19T07:19:55.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.450+0000 7f1ac2c7e000 20 client.4731 trim_cache size 0 max 16384 2024-09-19T07:19:55.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1ac2c7e000 1 Processor -- start 2024-09-19T07:19:55.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1ac2c7e000 1 -- start start 2024-09-19T07:19:55.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1ac2c7e000 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d6b70 0x5655128d6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1ac2c7e000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 0x565512924110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1ac2c7e000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128df2d0 0x565512924650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1ac2c7e000 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab81fe3e0 con 0x5655128d6b70 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1ac2c7e000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab81fbdd0 con 0x5655128df2d0 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1ac2c7e000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab81fc3f0 con 0x5655128d7490 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1abd95e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d6b70 0x5655128d6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1abd95e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d6b70 0x5655128d6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54352/0 (socket says 172.21.15.83:54352) 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1abd95e640 1 -- 172.21.15.83:0/2531232473 learned_addr learned my addr 172.21.15.83:0/2531232473 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1abd95e640 1 -- 172.21.15.83:0/2531232473 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 msgr2=0x565512924110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1abd95e640 1 --2- 172.21.15.83:0/2531232473 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 0x565512924110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1abd95e640 1 -- 172.21.15.83:0/2531232473 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128df2d0 msgr2=0x565512924650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1abd95e640 1 --2- 172.21.15.83:0/2531232473 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128df2d0 0x565512924650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1abd95e640 1 -- 172.21.15.83:0/2531232473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5655126aefd0 con 0x5655128d6b70 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1abd95e640 1 --2- 172.21.15.83:0/2531232473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d6b70 0x5655128d6520 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8055fa0 tx=0x7f1aa80523b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f21fc00cbc77f541 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1a9effd640 1 -- 172.21.15.83:0/2531232473 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa801e040 con 0x5655128d6b70 2024-09-19T07:19:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1a9effd640 1 -- 172.21.15.83:0/2531232473 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa8054310 con 0x5655128d6b70 2024-09-19T07:19:55.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.466+0000 7f1a9effd640 1 -- 172.21.15.83:0/2531232473 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa800aba0 con 0x5655128d6b70 2024-09-19T07:19:55.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/2531232473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d6b70 msgr2=0x5655128d6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/2531232473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d6b70 0x5655128d6520 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8055fa0 tx=0x7f1aa80523b0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/2531232473 shutdown_connections 2024-09-19T07:19:55.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/2531232473 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128df2d0 0x565512924650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/2531232473 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 0x565512924110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/2531232473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128d6b70 0x5655128d6520 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/2531232473 >> 172.21.15.83:0/2531232473 conn(0x565512914830 msgr2=0x5655128d4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:55.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/2531232473 shutdown_connections 2024-09-19T07:19:55.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/2531232473 wait complete. 2024-09-19T07:19:55.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 -- start start 2024-09-19T07:19:55.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 Processor -- start 2024-09-19T07:19:55.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x5655129519d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 0x565512951f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 0x565512952450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x565512726f80 con 0x5655128df2d0 2024-09-19T07:19:55.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5655126dfca0 con 0x5655128d6b70 2024-09-19T07:19:55.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5655126e60f0 con 0x5655128d7490 2024-09-19T07:19:55.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 10 client.0 _ll_register_callbacks cb 0x7f1ab00096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T07:19:55.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1abe15f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 0x565512952450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1abe15f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 0x565512952450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54368/0 (socket says 172.21.15.83:54368) 2024-09-19T07:19:55.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1abe15f640 1 -- 172.21.15.83:0/3770213610 learned_addr learned my addr 172.21.15.83:0/3770213610 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:55.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1abe15f640 1 -- 172.21.15.83:0/3770213610 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 msgr2=0x565512951f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:55.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1abd95e640 1 --2- 172.21.15.83:0/3770213610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x5655129519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1abe15f640 1 --2- 172.21.15.83:0/3770213610 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 0x565512951f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1abe15f640 1 -- 172.21.15.83:0/3770213610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 msgr2=0x5655129519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:55.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1abe15f640 1 --2- 172.21.15.83:0/3770213610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x5655129519d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1abe15f640 1 -- 172.21.15.83:0/3770213610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5655127aa750 con 0x5655128df2d0 2024-09-19T07:19:55.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1abe15f640 1 --2- 172.21.15.83:0/3770213610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 0x565512952450 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0006e90 tx=0x7f1ab001c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cead8cf80d510c2d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:55.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1a9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-09-19T07:19:55.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1a9cff9640 1 -- 172.21.15.83:0/3770213610 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00b0040 con 0x5655128df2d0 2024-09-19T07:19:55.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1a9cff9640 1 -- 172.21.15.83:0/3770213610 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f1ab00ab6e0 con 0x5655128df2d0 2024-09-19T07:19:55.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/3770213610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1aa8054310 con 0x5655128df2d0 2024-09-19T07:19:55.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1a9cff9640 1 -- 172.21.15.83:0/3770213610 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f1ab00ab8e0 con 0x5655128df2d0 2024-09-19T07:19:55.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 10 client.4737 fetch_fsmap learned FSMap version 14 2024-09-19T07:19:55.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/3770213610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x5655126b5a30 con 0x5655128df2d0 2024-09-19T07:19:55.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1a9cff9640 1 -- 172.21.15.83:0/3770213610 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa8054310 con 0x5655128df2d0 2024-09-19T07:19:55.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1a9cff9640 1 -- 172.21.15.83:0/3770213610 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 14) ==== 4136+0+0 (secure 0 0 0) 0x7f1ab00d2040 con 0x5655128df2d0 2024-09-19T07:19:55.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 10 client.4737 fetch_fsmap finished waiting for FSMap version 14 2024-09-19T07:19:55.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 10 client.4737 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-19T07:19:55.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 20 client.4737 populate_metadata read hostname 'smithi083' 2024-09-19T07:19:55.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/3770213610 >> [v2:172.21.15.184:6834/149004692,v1:172.21.15.184:6835/149004692] conn(0x565512952d80 0x5655129957d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 4 client.4737 mds_command: new command op to 4232 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T07:19:55.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/3770213610 --> [v2:172.21.15.184:6834/149004692,v1:172.21.15.184:6835/149004692] -- command(tid 0: {"prefix": "status"}) -- 0x5655127353a0 con 0x565512952d80 2024-09-19T07:19:55.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1abd95e640 1 --2- 172.21.15.83:0/3770213610 >> [v2:172.21.15.184:6834/149004692,v1:172.21.15.184:6835/149004692] conn(0x565512952d80 0x5655129957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1a9cff9640 10 client.4737 ms_handle_connect on v2:172.21.15.184:6834/149004692 2024-09-19T07:19:55.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.470+0000 7f1abd95e640 1 --2- 172.21.15.83:0/3770213610 >> [v2:172.21.15.184:6834/149004692,v1:172.21.15.184:6835/149004692] conn(0x565512952d80 0x5655129957d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94f62c3207c8c0cb server_cookie=455d9a79731c215f in_seq=0 out_seq=0 2024-09-19T07:19:55.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1a9cff9640 1 -- 172.21.15.83:0/3770213610 <== mds.0 v2:172.21.15.184:6834/149004692 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5655127353a0 con 0x565512952d80 2024-09-19T07:19:55.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1a9cff9640 10 client.4737 handle_command_reply: tid=0 2024-09-19T07:19:55.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1ac2c7e000 1 client.4737 shutdown 2024-09-19T07:19:55.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/3770213610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 msgr2=0x565512952450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/3770213610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 0x565512952450 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0006e90 tx=0x7f1ab001c700 comp rx=0 tx=0).stop 2024-09-19T07:19:55.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/3770213610 shutdown_connections 2024-09-19T07:19:55.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/3770213610 >> [v2:172.21.15.184:6834/149004692,v1:172.21.15.184:6835/149004692] conn(0x565512952d80 0x5655129957d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/3770213610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5655128df2d0 0x565512952450 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/3770213610 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5655128d7490 0x565512951f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1ac2c7e000 1 --2- 172.21.15.83:0/3770213610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5655128d6b70 0x5655129519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1a9cff9640 0 client.4737 ms_handle_reset on v2:172.21.15.184:6834/149004692 2024-09-19T07:19:55.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/3770213610 >> 172.21.15.83:0/3770213610 conn(0x565512914830 msgr2=0x5655128d4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:55.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/3770213610 shutdown_connections 2024-09-19T07:19:55.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1ac2c7e000 1 -- 172.21.15.83:0/3770213610 wait complete. 2024-09-19T07:19:55.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1ac2c7e000 20 client.4737 trim_cache size 0 max 16384 2024-09-19T07:19:55.481 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-19T07:19:55.481 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "86ed7344-2eac-4937-889b-a144de44eb64", 2024-09-19T07:19:55.481 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-09-19T07:19:55.481 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 4232, 2024-09-19T07:19:55.481 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-09-19T07:19:55.481 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-09-19T07:19:55.481 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "b", 2024-09-19T07:19:55.482 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 15.056593512999999, 2024-09-19T07:19:55.482 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 14, 2024-09-19T07:19:55.482 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 25, 2024-09-19T07:19:55.482 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 25, 2024-09-19T07:19:55.482 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 29.257671007999999 2024-09-19T07:19:55.482 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-19T07:19:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1abfbc2640 1 -- 172.21.15.83:0/96329969 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1aa40678c0 msgr2=0x7f1aa4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1abfbc2640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1aa40678c0 0x7f1aa4069d80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8073b60 tx=0x7f1aac008040 comp rx=0 tx=0).stop 2024-09-19T07:19:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1abfbc2640 1 -- 172.21.15.83:0/96329969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ab80c4550 msgr2=0x7f1ab80c60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1abfbc2640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ab80c4550 0x7f1ab80c60d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800cb80 tx=0x7f1aa800cf80 comp rx=0 tx=0).stop 2024-09-19T07:19:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1abfbc2640 1 -- 172.21.15.83:0/96329969 shutdown_connections 2024-09-19T07:19:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1abfbc2640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1aa40678c0 0x7f1aa4069d80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1abfbc2640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ab81534b0 0x7f1ab81f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1abfbc2640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ab8150060 0x7f1ab81513d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1abfbc2640 1 --2- 172.21.15.83:0/96329969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ab80c4550 0x7f1ab80c60d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1abfbc2640 1 -- 172.21.15.83:0/96329969 >> 172.21.15.83:0/96329969 conn(0x7f1ab806bc80 msgr2=0x7f1ab815c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1abfbc2640 1 -- 172.21.15.83:0/96329969 shutdown_connections 2024-09-19T07:19:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.474+0000 7f1abfbc2640 1 -- 172.21.15.83:0/96329969 wait complete. 2024-09-19T07:19:55.491 INFO:teuthology.run_tasks:Running task sequential... 2024-09-19T07:19:55.500 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-19T07:19:55.500 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-19T07:19:55.501 INFO:teuthology.task.print:Enabling mgr modules 2024-09-19T07:19:55.501 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-09-19T07:19:55.525 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-09-19T07:19:55.525 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-09-19T07:19:55.526 INFO:tasks.ceph_fuse:client.0 config is {'cephfs_name': 'a'} 2024-09-19T07:19:55.526 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-19T07:19:55.526 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-19T07:19:55.526 INFO:tasks.ceph_fuse:client.1 config is {'cephfs_name': 'b'} 2024-09-19T07:19:55.526 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-19T07:19:55.526 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-19T07:19:55.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:55.526 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-09-19T07:19:55.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:55.532 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link delete ceph-brx 2024-09-19T07:19:55.587 INFO:teuthology.orchestra.run.smithi083.stderr:Cannot find device "ceph-brx" 2024-09-19T07:19:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:19:55.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:55.589 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-19T07:19:55.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:55.596 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link delete ceph-brx 2024-09-19T07:19:55.655 INFO:teuthology.orchestra.run.smithi184.stderr:Cannot find device "ceph-brx" 2024-09-19T07:19:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:19:55.656 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-09-19T07:19:55.656 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-19T07:19:55.657 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-19T07:19:55.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.774+0000 7fcdc35d3640 1 Processor -- start 2024-09-19T07:19:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.774+0000 7fcdc35d3640 1 -- start start 2024-09-19T07:19:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc35d3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcdbc0c42e0 0x7fcdbc06e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc35d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdbc1566b0 0x7fcdbc0abb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc35d3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcdbc159a80 0x7fcdbc153060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc35d3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fcdbc0ac070 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc35d3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fcdbc1595f0 con 0x7fcdbc159a80 2024-09-19T07:19:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc35d3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcdbc1592c0 con 0x7fcdbc1566b0 2024-09-19T07:19:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc136f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcdbc0c42e0 0x7fcdbc06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc136f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcdbc0c42e0 0x7fcdbc06e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54370/0 (socket says 172.21.15.83:54370) 2024-09-19T07:19:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc136f640 1 -- 172.21.15.83:0/1036198165 learned_addr learned my addr 172.21.15.83:0/1036198165 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc1b70640 1 --2- 172.21.15.83:0/1036198165 >> v2:172.21.15.184:3300/0 conn(0x7fcdbc159a80 0x7fcdbc153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc0b6e640 1 --2- 172.21.15.83:0/1036198165 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdbc1566b0 0x7fcdbc0abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc136f640 1 -- 172.21.15.83:0/1036198165 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdbc1566b0 msgr2=0x7fcdbc0abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc136f640 1 --2- 172.21.15.83:0/1036198165 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdbc1566b0 0x7fcdbc0abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc136f640 1 -- 172.21.15.83:0/1036198165 >> v2:172.21.15.184:3300/0 conn(0x7fcdbc159a80 msgr2=0x7fcdbc153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc136f640 1 --2- 172.21.15.83:0/1036198165 >> v2:172.21.15.184:3300/0 conn(0x7fcdbc159a80 0x7fcdbc153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.778+0000 7fcdc136f640 1 -- 172.21.15.83:0/1036198165 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdbc06e900 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdc136f640 1 --2- 172.21.15.83:0/1036198165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdbc0c42e0 0x7fcdbc06e270 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0c4010 tx=0x7fcdb401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e29a4d5a7649f75 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:55.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdbbfff640 1 -- 172.21.15.83:0/1036198165 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb40a5860 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdbbfff640 1 -- 172.21.15.83:0/1036198165 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdb40a5a00 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdbbfff640 1 -- 172.21.15.83:0/1036198165 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb40a5dd0 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1036198165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdbc0c42e0 msgr2=0x7fcdbc06e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdc35d3640 1 --2- 172.21.15.83:0/1036198165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdbc0c42e0 0x7fcdbc06e270 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0c4010 tx=0x7fcdb401c9c0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1036198165 shutdown_connections 2024-09-19T07:19:55.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdc35d3640 1 --2- 172.21.15.83:0/1036198165 >> v2:172.21.15.184:3300/0 conn(0x7fcdbc159a80 0x7fcdbc153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdc35d3640 1 --2- 172.21.15.83:0/1036198165 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdbc1566b0 0x7fcdbc0abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdc35d3640 1 --2- 172.21.15.83:0/1036198165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdbc0c42e0 0x7fcdbc06e270 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1036198165 >> 172.21.15.83:0/1036198165 conn(0x7fcdbc06bc80 msgr2=0x7fcdbc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:55.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1036198165 shutdown_connections 2024-09-19T07:19:55.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1036198165 wait complete. 2024-09-19T07:19:55.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdc35d3640 1 Processor -- start 2024-09-19T07:19:55.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.782+0000 7fcdc35d3640 1 -- start start 2024-09-19T07:19:55.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc35d3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdbc0c42e0 0x7fcdbc157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc35d3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdbc1566b0 0x7fcdbc158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc35d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdbc159a80 0x7fcdbc0c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc35d3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcdbc0c4840 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc35d3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcdbc0c46c0 con 0x7fcdbc1566b0 2024-09-19T07:19:55.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc35d3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcdbc159e90 con 0x7fcdbc159a80 2024-09-19T07:19:55.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc136f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdbc0c42e0 0x7fcdbc157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc136f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdbc0c42e0 0x7fcdbc157b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54372/0 (socket says 172.21.15.83:54372) 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc136f640 1 -- 172.21.15.83:0/1598270487 learned_addr learned my addr 172.21.15.83:0/1598270487 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc0b6e640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdbc1566b0 0x7fcdbc158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc1b70640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdbc159a80 0x7fcdbc0c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc136f640 1 -- 172.21.15.83:0/1598270487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdbc159a80 msgr2=0x7fcdbc0c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc136f640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdbc159a80 0x7fcdbc0c2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc136f640 1 -- 172.21.15.83:0/1598270487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdbc1566b0 msgr2=0x7fcdbc158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc136f640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdbc1566b0 0x7fcdbc158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc136f640 1 -- 172.21.15.83:0/1598270487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdbc1ef330 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc136f640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdbc0c42e0 0x7fcdbc157b00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40af040 tx=0x7fcdb4019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdb9ffb640 1 -- 172.21.15.83:0/1598270487 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb40b8880 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdb9ffb640 1 -- 172.21.15.83:0/1598270487 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdb40be070 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1598270487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdbc1ef610 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdb9ffb640 1 -- 172.21.15.83:0/1598270487 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb40b8c60 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1598270487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdbc157300 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1598270487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdbc0abb30 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdb9ffb640 1 -- 172.21.15.83:0/1598270487 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcdb40c2410 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdb9ffb640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcda80678f0 0x7fcda8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:55.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdb9ffb640 1 -- 172.21.15.83:0/1598270487 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fcdb4120d90 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc0b6e640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcda80678f0 0x7fcda8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:55.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.786+0000 7fcdc0b6e640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcda80678f0 0x7fcda8069db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0714a0 tx=0x7fcdb0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:55.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.790+0000 7fcdb9ffb640 1 -- 172.21.15.83:0/1598270487 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcdb40c7020 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.990+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1598270487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcdbc06c470 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.990+0000 7fcdb9ffb640 1 -- 172.21.15.83:0/1598270487 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) ==== 53+0+116 (secure 0 0 0) 0x7fcdb40165e0 con 0x7fcdbc0c42e0 2024-09-19T07:19:55.992 INFO:teuthology.orchestra.run.smithi083.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2024-09-19T07:19:55.992 INFO:teuthology.orchestra.run.smithi083.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2024-09-19T07:19:55.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.994+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1598270487 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcda80678f0 msgr2=0x7fcda8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.994+0000 7fcdc35d3640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcda80678f0 0x7fcda8069db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0714a0 tx=0x7fcdb0008040 comp rx=0 tx=0).stop 2024-09-19T07:19:55.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.994+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1598270487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdbc0c42e0 msgr2=0x7fcdbc157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:55.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.994+0000 7fcdc35d3640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdbc0c42e0 0x7fcdbc157b00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40af040 tx=0x7fcdb4019760 comp rx=0 tx=0).stop 2024-09-19T07:19:55.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.994+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1598270487 shutdown_connections 2024-09-19T07:19:55.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.994+0000 7fcdc35d3640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcda80678f0 0x7fcda8069db0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.994+0000 7fcdc35d3640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdbc159a80 0x7fcdbc0c2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.994+0000 7fcdc35d3640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdbc1566b0 0x7fcdbc158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.994+0000 7fcdc35d3640 1 --2- 172.21.15.83:0/1598270487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcdbc0c42e0 0x7fcdbc157b00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:55.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.994+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1598270487 >> 172.21.15.83:0/1598270487 conn(0x7fcdbc06bc80 msgr2=0x7fcdbc15f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:55.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.994+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1598270487 shutdown_connections 2024-09-19T07:19:55.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:55.994+0000 7fcdc35d3640 1 -- 172.21.15.83:0/1598270487 wait complete. 2024-09-19T07:19:56.011 INFO:tasks.cephfs.mount:Mounting Ceph FS a; just confirmed its presence on cluster 2024-09-19T07:19:56.011 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-19T07:19:56.011 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-09-19T07:19:56.011 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-19T07:19:56.012 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-19T07:19:56.012 INFO:tasks.cephfs.mount:self.cephfs_name = a 2024-09-19T07:19:56.012 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-19T07:19:56.012 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-19T07:19:56.012 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-19T07:19:56.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:56.012 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-09-19T07:19:56.017 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-19T07:19:56.017 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-19T07:19:56.017 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-09-19T07:19:56.017 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-09-19T07:19:56.017 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-09-19T07:19:56.017 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-09-19T07:19:56.017 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-19T07:19:56.017 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-09-19T07:19:56.017 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-09-19T07:19:56.017 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-19T07:19:56.017 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42489sec preferred_lft 42489sec 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 02:42:0d:65:1a:cb brd ff:ff:ff:ff:ff:ff 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-19T07:19:56.018 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-09-19T07:19:56.019 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-19T07:19:56.019 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T07:19:56.019 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-09-19T07:19:56.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link add name ceph-brx type bridge 2024-09-19T07:19:56.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip addr flush dev ceph-brx 2024-09-19T07:19:56.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set ceph-brx up 2024-09-19T07:19:56.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-19T07:19:56.019 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-09-19T07:19:56.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T07:19:56.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T07:19:56.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:56.128 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-19T07:19:56.178 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-09-19T07:19:56.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:56.180 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-09-19T07:19:56.247 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-09-19T07:19:56.248 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-19T07:19:56.248 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-19T07:19:56.248 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-19T07:19:56.248 INFO:teuthology.orchestra.run.smithi083.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-19T07:19:56.248 INFO:teuthology.orchestra.run.smithi083.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-19T07:19:56.248 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-19T07:19:56.248 INFO:teuthology.orchestra.run.smithi083.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-19T07:19:56.248 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-19T07:19:56.248 INFO:teuthology.orchestra.run.smithi083.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-19T07:19:56.248 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-19T07:19:56.249 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T07:19:56.249 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-09-19T07:19:56.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-19T07:19:56.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-19T07:19:56.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-19T07:19:56.249 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-09-19T07:19:56.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T07:19:56.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T07:19:56.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:56.361 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-09-19T07:19:56.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:56.414 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list-id 2024-09-19T07:19:56.461 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T07:19:56.462 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-09-19T07:19:56.462 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-19T07:19:56.462 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-19T07:19:56.462 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-09-19T07:19:56.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T07:19:56.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T07:19:56.577 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-19T07:19:56.577 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T07:19:56.577 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-09-19T07:19:56.577 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-19T07:19:56.577 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-19T07:19:56.577 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-19T07:19:56.577 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-19T07:19:56.577 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-19T07:19:56.577 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-09-19T07:19:56.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T07:19:56.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T07:19:56.717 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T07:19:56.717 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-09-19T07:19:56.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-09-19T07:19:56.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set dev brx.0 master ceph-brx 2024-09-19T07:19:56.718 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-09-19T07:19:56.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T07:19:56.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T07:19:56.830 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'cephfs_name': 'a'} 2024-09-19T07:19:56.831 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-19T07:19:56.831 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-19T07:19:56.877 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-19T07:19:56.878 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-19T07:19:56.878 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-19T07:19:56.925 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-09-19T07:19:56.981 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-09-19T07:19:57.031 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-09-19T07:19:57.031 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-09-19T07:19:57.031 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/run/lock 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-09-19T07:19:57.032 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/fs/nfsd 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysusers.service 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/run/rpc_pipefs 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-09-19T07:19:57.033 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-19T07:19:57.034 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-19T07:19:57.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:57.035 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-09-19T07:19:57.083 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-19T07:19:57.083 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_fs=a) 2024-09-19T07:19:57.127 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-09-19T07:19:57.141 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-09-19T07:19:57.195 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-09-19T07:19:57.195 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-09-19T07:19:57.195 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-09-19T07:19:57.195 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-09-19T07:19:57.195 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-09-19T07:19:57.195 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-09-19T07:19:57.195 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-09-19T07:19:57.195 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-09-19T07:19:57.195 INFO:teuthology.orchestra.run.smithi083.stdout:/run/lock 2024-09-19T07:19:57.195 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-09-19T07:19:57.195 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-09-19T07:19:57.195 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/fs/nfsd 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysusers.service 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/run/rpc_pipefs 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-09-19T07:19:57.196 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-09-19T07:19:57.197 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-09-19T07:19:57.197 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-09-19T07:19:57.197 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-09-19T07:19:57.197 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-09-19T07:19:57.197 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-19T07:19:57.197 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-19T07:19:57.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:57.198 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-09-19T07:19:57.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-09-19T07:19:57.210+0000 7f58bb0fb580 -1 init, newargv = 0x556ee9e12330 newargc=15 2024-09-19T07:19:57.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[36570]: starting ceph client 2024-09-19T07:19:57.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[36570]: starting fuse 2024-09-19T07:19:58.316 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-09-19T07:19:58.332 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-09-19T07:19:58.380 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-09-19T07:19:58.380 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-09-19T07:19:58.380 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-09-19T07:19:58.380 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-09-19T07:19:58.380 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-09-19T07:19:58.380 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-09-19T07:19:58.380 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-09-19T07:19:58.380 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-09-19T07:19:58.381 INFO:teuthology.orchestra.run.smithi083.stdout:/run/lock 2024-09-19T07:19:58.381 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-09-19T07:19:58.381 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-09-19T07:19:58.381 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-09-19T07:19:58.381 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-09-19T07:19:58.381 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-09-19T07:19:58.381 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-09-19T07:19:58.381 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-09-19T07:19:58.381 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-09-19T07:19:58.381 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/fs/nfsd 2024-09-19T07:19:58.381 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-09-19T07:19:58.382 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-09-19T07:19:58.382 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysusers.service 2024-09-19T07:19:58.382 INFO:teuthology.orchestra.run.smithi083.stdout:/run/rpc_pipefs 2024-09-19T07:19:58.382 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-09-19T07:19:58.382 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-09-19T07:19:58.382 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-09-19T07:19:58.382 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-09-19T07:19:58.382 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-09-19T07:19:58.382 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-09-19T07:19:58.383 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-09-19T07:19:58.383 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-19T07:19:58.383 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-19T07:19:58.383 INFO:teuthology.orchestra.run.smithi083.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-19T07:19:58.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:58.384 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-09-19T07:19:58.430 INFO:teuthology.orchestra.run.smithi083.stdout:44 2024-09-19T07:19:58.430 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [44] 2024-09-19T07:19:58.431 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-09-19T07:19:58.431 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-09-19T07:19:58.431 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-09-19T07:19:58.431 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-09-19T07:19:58.431 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-09-19T07:19:58.431 DEBUG:teuthology.orchestra.run.smithi083:> 2024-09-19T07:19:58.431 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-09-19T07:19:58.431 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-19T07:19:58.431 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-09-19T07:19:58.431 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-19T07:19:58.431 DEBUG:teuthology.orchestra.run.smithi083:> 2024-09-19T07:19:58.431 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-09-19T07:19:58.432 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-09-19T07:19:58.555 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.36570.asok 2024-09-19T07:19:58.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T07:19:58.567 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.36570.asok 2024-09-19T07:19:58.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:58.567 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.36570.asok status 2024-09-19T07:19:58.724 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-19T07:19:58.724 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-09-19T07:19:58.724 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "2f599a46ad10e37c929e5c43da2f6907ddcd1961", 2024-09-19T07:19:58.724 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable)", 2024-09-19T07:19:58.724 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "36570", 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 4746, 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4746 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-19T07:19:58.725 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3577248012 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3577248012 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.4746 192.168.0.1:0/3577248012", 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/3577248012", 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-09-19T07:19:58.726 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 8, 2024-09-19T07:19:58.727 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 25, 2024-09-19T07:19:58.727 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-09-19T07:19:58.727 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-09-19T07:19:58.727 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "a" 2024-09-19T07:19:58.727 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-19T07:19:58.738 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-19T07:19:58.738 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-19T07:19:58.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.870+0000 7f104bb73640 1 Processor -- start 2024-09-19T07:19:58.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.870+0000 7f104bb73640 1 -- start start 2024-09-19T07:19:58.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104bb73640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f104414f710 0x7f10440728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104bb73640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1044072e30 0x7f104406e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104bb73640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1044156040 con 0x7f104406fe80 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104bb73640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f1044073260 con 0x7f104414f710 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104bb73640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1044156320 con 0x7f1044072e30 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104990f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f104414f710 0x7f10440728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104910e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f104406fe80 0x7f104406da40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35538/0 (socket says 172.21.15.83:35538) 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104910e640 1 -- 172.21.15.83:0/3648195981 learned_addr learned my addr 172.21.15.83:0/3648195981 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104a110640 1 --2- 172.21.15.83:0/3648195981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1044072e30 0x7f104406e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104990f640 1 -- 172.21.15.83:0/3648195981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1044072e30 msgr2=0x7f104406e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104990f640 1 --2- 172.21.15.83:0/3648195981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1044072e30 0x7f104406e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104990f640 1 -- 172.21.15.83:0/3648195981 >> v1:172.21.15.83:6789/0 conn(0x7f104406fe80 legacy=0x7f104406da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104990f640 1 -- 172.21.15.83:0/3648195981 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10440720f0 con 0x7f104414f710 2024-09-19T07:19:58.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104890d640 1 -- 172.21.15.83:0/3648195981 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3680186796 0 0) 0x7f1044156040 con 0x7f104406fe80 2024-09-19T07:19:58.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104990f640 1 --2- 172.21.15.83:0/3648195981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f104414f710 0x7f10440728f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f10400097c0 tx=0x7f104001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72125f41600fcab0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:58.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104890d640 1 -- 172.21.15.83:0/3648195981 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10400b0070 con 0x7f104414f710 2024-09-19T07:19:58.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104890d640 1 -- 172.21.15.83:0/3648195981 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10400ab600 con 0x7f104414f710 2024-09-19T07:19:58.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104890d640 1 -- 172.21.15.83:0/3648195981 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10400ab9d0 con 0x7f104414f710 2024-09-19T07:19:58.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104bb73640 1 -- 172.21.15.83:0/3648195981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f104414f710 msgr2=0x7f10440728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:58.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.874+0000 7f104bb73640 1 --2- 172.21.15.83:0/3648195981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f104414f710 0x7f10440728f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f10400097c0 tx=0x7f104001c410 comp rx=0 tx=0).stop 2024-09-19T07:19:58.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 -- 172.21.15.83:0/3648195981 shutdown_connections 2024-09-19T07:19:58.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 --2- 172.21.15.83:0/3648195981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1044072e30 0x7f104406e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:58.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 --2- 172.21.15.83:0/3648195981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f104414f710 0x7f10440728f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:58.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 -- 172.21.15.83:0/3648195981 >> 172.21.15.83:0/3648195981 conn(0x7f10440c7e40 msgr2=0x7f10440c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:58.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 -- 172.21.15.83:0/3648195981 shutdown_connections 2024-09-19T07:19:58.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 -- 172.21.15.83:0/3648195981 wait complete. 2024-09-19T07:19:58.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 Processor -- start 2024-09-19T07:19:58.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 -- start start 2024-09-19T07:19:58.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f104406fe80 0x7f10440bd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:58.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1044072e30 0x7f1044148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:58.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f104414f710 0x7f10441489a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:58.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f10440c29c0 con 0x7f1044072e30 2024-09-19T07:19:58.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f10440c2840 con 0x7f104414f710 2024-09-19T07:19:58.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1044073a60 con 0x7f104406fe80 2024-09-19T07:19:58.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104910e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1044072e30 0x7f1044148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:58.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104910e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1044072e30 0x7f1044148460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54386/0 (socket says 172.21.15.83:54386) 2024-09-19T07:19:58.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104910e640 1 -- 172.21.15.83:0/2834786844 learned_addr learned my addr 172.21.15.83:0/2834786844 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:19:58.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104a110640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f104414f710 0x7f10441489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:58.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104990f640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f104406fe80 0x7f10440bd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:58.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104910e640 1 -- 172.21.15.83:0/2834786844 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f104406fe80 msgr2=0x7f10440bd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:58.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104910e640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f104406fe80 0x7f10440bd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:58.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104910e640 1 -- 172.21.15.83:0/2834786844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f104414f710 msgr2=0x7f10441489a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:58.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104910e640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f104414f710 0x7f10441489a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:58.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104910e640 1 -- 172.21.15.83:0/2834786844 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10441ed160 con 0x7f1044072e30 2024-09-19T07:19:58.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104910e640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1044072e30 0x7f1044148460 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f103400c450 tx=0x7f103400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:58.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f103a7fc640 1 -- 172.21.15.83:0/2834786844 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10340074e0 con 0x7f1044072e30 2024-09-19T07:19:58.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f103a7fc640 1 -- 172.21.15.83:0/2834786844 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10340047f0 con 0x7f1044072e30 2024-09-19T07:19:58.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f103a7fc640 1 -- 172.21.15.83:0/2834786844 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1034007680 con 0x7f1044072e30 2024-09-19T07:19:58.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 -- 172.21.15.83:0/2834786844 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10440ab590 con 0x7f1044072e30 2024-09-19T07:19:58.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 -- 172.21.15.83:0/2834786844 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1044150d40 con 0x7f1044072e30 2024-09-19T07:19:58.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.878+0000 7f104bb73640 1 -- 172.21.15.83:0/2834786844 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10440728f0 con 0x7f1044072e30 2024-09-19T07:19:58.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.882+0000 7f103a7fc640 1 -- 172.21.15.83:0/2834786844 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1034011070 con 0x7f1044072e30 2024-09-19T07:19:58.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.882+0000 7f103a7fc640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10280678c0 0x7f1028069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:19:58.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.882+0000 7f103a7fc640 1 -- 172.21.15.83:0/2834786844 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f1034085b00 con 0x7f1044072e30 2024-09-19T07:19:58.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.882+0000 7f104990f640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10280678c0 0x7f1028069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:19:58.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.882+0000 7f104990f640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10280678c0 0x7f1028069d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f10400ad000 tx=0x7f1040008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:19:58.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:58.886+0000 7f103a7fc640 1 -- 172.21.15.83:0/2834786844 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f10340563a0 con 0x7f1044072e30 2024-09-19T07:19:59.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.078+0000 7f104bb73640 1 -- 172.21.15.83:0/2834786844 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f10440737f0 con 0x7f1044072e30 2024-09-19T07:19:59.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.078+0000 7f103a7fc640 1 -- 172.21.15.83:0/2834786844 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) ==== 53+0+116 (secure 0 0 0) 0x7f1034055ca0 con 0x7f1044072e30 2024-09-19T07:19:59.082 INFO:teuthology.orchestra.run.smithi083.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2024-09-19T07:19:59.082 INFO:teuthology.orchestra.run.smithi083.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2024-09-19T07:19:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.082+0000 7f104bb73640 1 -- 172.21.15.83:0/2834786844 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10280678c0 msgr2=0x7f1028069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.082+0000 7f104bb73640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10280678c0 0x7f1028069d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f10400ad000 tx=0x7f1040008040 comp rx=0 tx=0).stop 2024-09-19T07:19:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.082+0000 7f104bb73640 1 -- 172.21.15.83:0/2834786844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1044072e30 msgr2=0x7f1044148460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:19:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.082+0000 7f104bb73640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1044072e30 0x7f1044148460 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f103400c450 tx=0x7f103400c910 comp rx=0 tx=0).stop 2024-09-19T07:19:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.082+0000 7f104bb73640 1 -- 172.21.15.83:0/2834786844 shutdown_connections 2024-09-19T07:19:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.082+0000 7f104bb73640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10280678c0 0x7f1028069d80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.082+0000 7f104bb73640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f104414f710 0x7f10441489a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.082+0000 7f104bb73640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1044072e30 0x7f1044148460 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.082+0000 7f104bb73640 1 --2- 172.21.15.83:0/2834786844 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f104406fe80 0x7f10440bd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:19:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.082+0000 7f104bb73640 1 -- 172.21.15.83:0/2834786844 >> 172.21.15.83:0/2834786844 conn(0x7f10440c7e40 msgr2=0x7f104414ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:19:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.082+0000 7f104bb73640 1 -- 172.21.15.83:0/2834786844 shutdown_connections 2024-09-19T07:19:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:19:59.082+0000 7f104bb73640 1 -- 172.21.15.83:0/2834786844 wait complete. 2024-09-19T07:19:59.099 INFO:tasks.cephfs.mount:Mounting Ceph FS b; just confirmed its presence on cluster 2024-09-19T07:19:59.099 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-19T07:19:59.099 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-19T07:19:59.099 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-19T07:19:59.099 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-19T07:19:59.099 INFO:tasks.cephfs.mount:self.cephfs_name = b 2024-09-19T07:19:59.099 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-19T07:19:59.099 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-19T07:19:59.099 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-19T07:19:59.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:59.100 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-19T07:19:59.106 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-19T07:19:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-19T07:19:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-19T07:19:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-19T07:19:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-19T07:19:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42487sec preferred_lft 42487sec 2024-09-19T07:19:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-19T07:19:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-19T07:19:59.108 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-19T07:19:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:ec:eb:88:92 brd ff:ff:ff:ff:ff:ff 2024-09-19T07:19:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-19T07:19:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-19T07:19:59.108 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-19T07:19:59.108 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T07:19:59.109 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-19T07:19:59.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link add name ceph-brx type bridge 2024-09-19T07:19:59.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip addr flush dev ceph-brx 2024-09-19T07:19:59.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set ceph-brx up 2024-09-19T07:19:59.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-19T07:19:59.109 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-19T07:19:59.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T07:19:59.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T07:19:59.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:59.241 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-19T07:19:59.251 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-19T07:19:59.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:59.252 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-19T07:19:59.328 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-19T07:19:59.329 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-19T07:19:59.329 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-19T07:19:59.329 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-19T07:19:59.329 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-19T07:19:59.329 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-19T07:19:59.329 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-19T07:19:59.329 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-19T07:19:59.329 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-19T07:19:59.329 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-19T07:19:59.329 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-19T07:19:59.330 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T07:19:59.330 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-19T07:19:59.330 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-19T07:19:59.330 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-19T07:19:59.330 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-19T07:19:59.330 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-19T07:19:59.415 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T07:19:59.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T07:19:59.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:59.452 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-19T07:19:59.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:19:59.457 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list-id 2024-09-19T07:19:59.503 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T07:19:59.503 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-19T07:19:59.503 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-19T07:19:59.503 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-19T07:19:59.503 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-19T07:19:59.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T07:19:59.617 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T07:19:59.624 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-19T07:19:59.624 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T07:19:59.624 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-19T07:19:59.624 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-19T07:19:59.624 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-19T07:19:59.625 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-19T07:19:59.625 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-19T07:19:59.625 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-19T07:19:59.625 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-19T07:19:59.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T07:19:59.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T07:19:59.725 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T07:19:59.725 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-19T07:19:59.725 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-19T07:19:59.725 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set dev brx.0 master ceph-brx 2024-09-19T07:19:59.725 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-19T07:19:59.783 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:19:59.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:19:59.804 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T07:19:59.820 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:19:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T07:19:59.827 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {'cephfs_name': 'b'} 2024-09-19T07:19:59.827 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-19T07:19:59.827 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-19T07:19:59.885 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-19T07:19:59.886 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-19T07:19:59.886 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-19T07:19:59.930 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-19T07:19:59.989 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-19T07:20:00.037 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-19T07:20:00.037 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-19T07:20:00.037 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-19T07:20:00.037 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-19T07:20:00.037 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-19T07:20:00.038 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-19T07:20:00.039 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-19T07:20:00.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:20:00.041 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-19T07:20:00.090 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-19T07:20:00.090 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_fs=b) 2024-09-19T07:20:00.135 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-19T07:20:00.148 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-19T07:20:00.198 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-19T07:20:00.198 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-19T07:20:00.198 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-19T07:20:00.199 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-19T07:20:00.200 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-19T07:20:00.201 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-19T07:20:00.201 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-19T07:20:00.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:20:00.202 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-19T07:20:00.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-19T07:20:00.206+0000 7f270d201580 -1 init, newargv = 0x5641c6d84450 newargc=15 2024-09-19T07:20:00.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[34414]: starting ceph client 2024-09-19T07:20:00.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[34414]: starting fuse 2024-09-19T07:20:01.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-19T07:20:01.329 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-19T07:20:01.376 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-19T07:20:01.376 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-19T07:20:01.376 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-19T07:20:01.376 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-19T07:20:01.376 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-19T07:20:01.376 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-19T07:20:01.376 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-19T07:20:01.376 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-19T07:20:01.376 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-19T07:20:01.376 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-19T07:20:01.376 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-19T07:20:01.377 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-19T07:20:01.378 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-19T07:20:01.378 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-19T07:20:01.378 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-19T07:20:01.378 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-19T07:20:01.378 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-19T07:20:01.378 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-19T07:20:01.378 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-19T07:20:01.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:20:01.379 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-19T07:20:01.427 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-19T07:20:01.428 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-19T07:20:01.428 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-19T07:20:01.428 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-19T07:20:01.428 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-19T07:20:01.428 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-19T07:20:01.428 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-19T07:20:01.428 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-19T07:20:01.428 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-19T07:20:01.428 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-19T07:20:01.428 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-19T07:20:01.428 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-19T07:20:01.429 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-19T07:20:01.430 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-19T07:20:01.430 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-19T07:20:01.430 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-19T07:20:01.550 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.34414.asok 2024-09-19T07:20:01.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-19T07:20:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T07:20:01.564 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34414.asok 2024-09-19T07:20:01.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:20:01.565 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34414.asok status 2024-09-19T07:20:01.712 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-19T07:20:01.713 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-19T07:20:01.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "2f599a46ad10e37c929e5c43da2f6907ddcd1961", 2024-09-19T07:20:01.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable)", 2024-09-19T07:20:01.713 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-19T07:20:01.713 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-19T07:20:01.713 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-19T07:20:01.713 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "34414", 2024-09-19T07:20:01.713 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-19T07:20:01.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-19T07:20:01.714 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-19T07:20:01.714 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-19T07:20:01.714 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 4454, 2024-09-19T07:20:01.714 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-19T07:20:01.714 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-19T07:20:01.714 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-19T07:20:01.714 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 4454 2024-09-19T07:20:01.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-19T07:20:01.714 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-19T07:20:01.714 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-19T07:20:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-19T07:20:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 792782352 2024-09-19T07:20:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-19T07:20:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-19T07:20:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-19T07:20:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-19T07:20:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-19T07:20:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 792782352 2024-09-19T07:20:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-19T07:20:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.4454 192.168.0.1:0/792782352", 2024-09-19T07:20:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/792782352", 2024-09-19T07:20:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-19T07:20:01.716 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 14, 2024-09-19T07:20:01.716 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 25, 2024-09-19T07:20:01.716 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-19T07:20:01.716 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-19T07:20:01.716 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "b" 2024-09-19T07:20:01.716 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-19T07:20:01.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:20:01.727 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-09-19T07:20:01.727 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-19T07:20:01.735 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-09-19T07:20:01.735 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-19T07:20:01.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:20:01.735 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-19T07:20:01.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:20:01.792 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-19T07:20:01.792 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-19T07:20:01.798 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-19T07:20:01.799 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-19T07:20:01.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:20:01.799 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-19T07:20:01.856 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-09-19T07:20:01.904 INFO:tasks.mds_thrash:mds thrasher using random seed: 1726730401 2024-09-19T07:20:01.905 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:20:02.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d4148f640 1 Processor -- start 2024-09-19T07:20:02.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d4148f640 1 -- start start 2024-09-19T07:20:02.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d4148f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7d3c071100 0x7f7d3c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d4148f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d3c0a9e60 con 0x7f7d3c06dea0 2024-09-19T07:20:02.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d4148f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d3c0aa140 con 0x7f7d3c0c28e0 2024-09-19T07:20:02.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d4148f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f7d3c071b50 con 0x7f7d3c071100 2024-09-19T07:20:02.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d3b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7d3c06dea0 0x7f7d3c06e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49976/0 (socket says 172.21.15.83:49976) 2024-09-19T07:20:02.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d3b7fe640 1 -- 172.21.15.83:0/2346870288 learned_addr learned my addr 172.21.15.83:0/2346870288 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:02.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d3affd640 1 --2- 172.21.15.83:0/2346870288 >> v2:172.21.15.184:3300/0 conn(0x7f7d3c071100 0x7f7d3c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 302175435 0 0) 0x7f7d3c0a9e60 con 0x7f7d3c06dea0 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d28003530 con 0x7f7d3c06dea0 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 143058052 0 0) 0x7f7d28003530 con 0x7f7d3c06dea0 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 >> v2:172.21.15.184:3300/0 conn(0x7f7d3c071100 msgr2=0x7f7d3c1545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 --2- 172.21.15.83:0/2346870288 >> v2:172.21.15.184:3300/0 conn(0x7f7d3c071100 0x7f7d3c1545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 >> v1:172.21.15.184:6789/0 conn(0x7f7d3c0c28e0 legacy=0x7f7d3c150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c0c2240 con 0x7f7d3c06dea0 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f7d34001680 con 0x7f7d3c06dea0 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 >> v1:172.21.15.83:6789/0 conn(0x7f7d3c06dea0 legacy=0x7f7d3c06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 --2- 172.21.15.83:0/2346870288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d280039c0 0x7f7d28003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 --2- 172.21.15.83:0/2346870288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d28004810 0x7f7d28004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 --2- 172.21.15.83:0/2346870288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d280051d0 0x7f7d2800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c071b50 con 0x7f7d280051d0 2024-09-19T07:20:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2800c170 con 0x7f7d280039c0 2024-09-19T07:20:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7d2800c370 con 0x7f7d28004810 2024-09-19T07:20:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d3b7fe640 1 --2- 172.21.15.83:0/2346870288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d280051d0 0x7f7d2800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d3affd640 1 --2- 172.21.15.83:0/2346870288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d280039c0 0x7f7d28003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d3a7fc640 1 --2- 172.21.15.83:0/2346870288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d28004810 0x7f7d28004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d3b7fe640 1 -- 172.21.15.83:0/2346870288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d28004810 msgr2=0x7f7d28004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d3b7fe640 1 --2- 172.21.15.83:0/2346870288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d28004810 0x7f7d28004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d3b7fe640 1 -- 172.21.15.83:0/2346870288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d280039c0 msgr2=0x7f7d28003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d3b7fe640 1 --2- 172.21.15.83:0/2346870288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d280039c0 0x7f7d28003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d3b7fe640 1 -- 172.21.15.83:0/2346870288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d28003530 con 0x7f7d280051d0 2024-09-19T07:20:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.034+0000 7f7d3a7fc640 1 --2- 172.21.15.83:0/2346870288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d28004810 0x7f7d28004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d3b7fe640 1 -- 172.21.15.83:0/2346870288 reap_dead start 2024-09-19T07:20:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d3b7fe640 1 --2- 172.21.15.83:0/2346870288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d280051d0 0x7f7d2800b9d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400c1c0 tx=0x7f7d3401f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1fc706300af98d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:02.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d34007d00 con 0x7f7d280051d0 2024-09-19T07:20:02.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d34005b00 con 0x7f7d280051d0 2024-09-19T07:20:02.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d39ffb640 1 -- 172.21.15.83:0/2346870288 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d34007070 con 0x7f7d280051d0 2024-09-19T07:20:02.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 -- 172.21.15.83:0/2346870288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d280051d0 msgr2=0x7f7d2800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 --2- 172.21.15.83:0/2346870288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d280051d0 0x7f7d2800b9d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400c1c0 tx=0x7f7d3401f3d0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 -- 172.21.15.83:0/2346870288 shutdown_connections 2024-09-19T07:20:02.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 --2- 172.21.15.83:0/2346870288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d280051d0 0x7f7d2800b9d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 -- 172.21.15.83:0/2346870288 >> 172.21.15.83:0/2346870288 conn(0x7f7d3c06bc80 msgr2=0x7f7d3c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:02.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 -- 172.21.15.83:0/2346870288 shutdown_connections 2024-09-19T07:20:02.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 -- 172.21.15.83:0/2346870288 wait complete. 2024-09-19T07:20:02.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 Processor -- start 2024-09-19T07:20:02.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 -- start start 2024-09-19T07:20:02.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d280051d0 0x7f7d3c150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d3c149b40 0x7f7d3c0c6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d3c1e6570 0x7f7d3c1e69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c0714e0 con 0x7f7d3c149b40 2024-09-19T07:20:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c0c1ff0 con 0x7f7d280051d0 2024-09-19T07:20:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d4148f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7d3c071660 con 0x7f7d3c1e6570 2024-09-19T07:20:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d3a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d3c149b40 0x7f7d3c0c6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d3a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d3c149b40 0x7f7d3c0c6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49542/0 (socket says 172.21.15.83:49542) 2024-09-19T07:20:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.038+0000 7f7d3a7fc640 1 -- 172.21.15.83:0/3663303302 learned_addr learned my addr 172.21.15.83:0/3663303302 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d3a7fc640 1 -- 172.21.15.83:0/3663303302 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d3c1e6570 msgr2=0x7f7d3c1e69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d3b7fe640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d3c1e6570 0x7f7d3c1e69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d3affd640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d280051d0 0x7f7d3c150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d3a7fc640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d3c1e6570 0x7f7d3c1e69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d3a7fc640 1 -- 172.21.15.83:0/3663303302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d280051d0 msgr2=0x7f7d3c150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d3a7fc640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d280051d0 0x7f7d3c150500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d3a7fc640 1 -- 172.21.15.83:0/3663303302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c1ed300 con 0x7f7d3c149b40 2024-09-19T07:20:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d3b7fe640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d3c1e6570 0x7f7d3c1e69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d3a7fc640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d3c149b40 0x7f7d3c0c6910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400d8e0 tx=0x7f7d2400dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d337fe640 1 -- 172.21.15.83:0/3663303302 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24003db0 con 0x7f7d3c149b40 2024-09-19T07:20:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d337fe640 1 -- 172.21.15.83:0/3663303302 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d24017070 con 0x7f7d3c149b40 2024-09-19T07:20:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d4148f640 1 -- 172.21.15.83:0/3663303302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3c1ed5e0 con 0x7f7d3c149b40 2024-09-19T07:20:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d337fe640 1 -- 172.21.15.83:0/3663303302 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d240052b0 con 0x7f7d3c149b40 2024-09-19T07:20:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d4148f640 1 -- 172.21.15.83:0/3663303302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3c14c230 con 0x7f7d3c149b40 2024-09-19T07:20:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d4148f640 1 -- 172.21.15.83:0/3663303302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d3c154670 con 0x7f7d3c149b40 2024-09-19T07:20:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d337fe640 1 -- 172.21.15.83:0/3663303302 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7d24005450 con 0x7f7d3c149b40 2024-09-19T07:20:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d337fe640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7d1c0678f0 0x7f7d1c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d337fe640 1 -- 172.21.15.83:0/3663303302 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f7d240861a0 con 0x7f7d3c149b40 2024-09-19T07:20:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d3affd640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7d1c0678f0 0x7f7d1c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.042+0000 7f7d3affd640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7d1c0678f0 0x7f7d1c069db0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0098f0 tx=0x7f7d2c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:02.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.046+0000 7f7d337fe640 1 -- 172.21.15.83:0/3663303302 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7d240569a0 con 0x7f7d3c149b40 2024-09-19T07:20:02.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.210+0000 7f7d4148f640 1 -- 172.21.15.83:0/3663303302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d3c0c4f30 con 0x7f7d3c149b40 2024-09-19T07:20:02.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d337fe640 1 -- 172.21.15.83:0/3663303302 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+20451 (secure 0 0 0) 0x7f7d240562a0 con 0x7f7d3c149b40 2024-09-19T07:20:02.215 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:02.215 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:41.334020+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:20:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d4148f640 1 -- 172.21.15.83:0/3663303302 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7d1c0678f0 msgr2=0x7f7d1c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d4148f640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7d1c0678f0 0x7f7d1c069db0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0098f0 tx=0x7f7d2c008040 comp rx=0 tx=0).stop 2024-09-19T07:20:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d4148f640 1 -- 172.21.15.83:0/3663303302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d3c149b40 msgr2=0x7f7d3c0c6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d4148f640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d3c149b40 0x7f7d3c0c6910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400d8e0 tx=0x7f7d2400dda0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d4148f640 1 -- 172.21.15.83:0/3663303302 shutdown_connections 2024-09-19T07:20:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d4148f640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7d1c0678f0 0x7f7d1c069db0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d4148f640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d3c1e6570 0x7f7d3c1e69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d4148f640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d3c149b40 0x7f7d3c0c6910 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d4148f640 1 --2- 172.21.15.83:0/3663303302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d280051d0 0x7f7d3c150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d4148f640 1 -- 172.21.15.83:0/3663303302 >> 172.21.15.83:0/3663303302 conn(0x7f7d3c06bc80 msgr2=0x7f7d3c06e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:02.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d4148f640 1 -- 172.21.15.83:0/3663303302 shutdown_connections 2024-09-19T07:20:02.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.214+0000 7f7d4148f640 1 -- 172.21.15.83:0/3663303302 wait complete. 2024-09-19T07:20:02.233 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-19T07:19:20.885610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-09-19T07:19:29.461167+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-09-19T07:19:30.589046+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-09-19T07:19:38.325531+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-09-19T07:19:38.737342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-19T07:20:02.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-19T07:20:02.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0ec33640 1 Processor -- start 2024-09-19T07:20:02.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0ec33640 1 -- start start 2024-09-19T07:20:02.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0ec33640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9e0806dc40 0x7f9e0806e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0ec33640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e08073720 0x7f9e0814f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0ec33640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e0814fb60 0x7f9e080c1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0ec33640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f9e08073c90 con 0x7f9e0814fb60 2024-09-19T07:20:02.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0ec33640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f9e08073ee0 con 0x7f9e0806dc40 2024-09-19T07:20:02.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0ec33640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9e080c3700 con 0x7f9e08073720 2024-09-19T07:20:02.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0d1d0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e0814fb60 0x7f9e080c1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0d1d0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e0814fb60 0x7f9e080c1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49552/0 (socket says 172.21.15.83:49552) 2024-09-19T07:20:02.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0d1d0640 1 -- 172.21.15.83:0/370720019 learned_addr learned my addr 172.21.15.83:0/370720019 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:02.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0c9cf640 1 --2- 172.21.15.83:0/370720019 >> v2:172.21.15.184:3300/0 conn(0x7f9e0806dc40 0x7f9e0806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e07fff640 1 --2- 172.21.15.83:0/370720019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e08073720 0x7f9e0814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0d1d0640 1 -- 172.21.15.83:0/370720019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e08073720 msgr2=0x7f9e0814f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:02.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0d1d0640 1 --2- 172.21.15.83:0/370720019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e08073720 0x7f9e0814f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0d1d0640 1 -- 172.21.15.83:0/370720019 >> v2:172.21.15.184:3300/0 conn(0x7f9e0806dc40 msgr2=0x7f9e0806e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:02.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0d1d0640 1 --2- 172.21.15.83:0/370720019 >> v2:172.21.15.184:3300/0 conn(0x7f9e0806dc40 0x7f9e0806e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0d1d0640 1 -- 172.21.15.83:0/370720019 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e080c39e0 con 0x7f9e0814fb60 2024-09-19T07:20:02.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.358+0000 7f9e0c9cf640 1 --2- 172.21.15.83:0/370720019 >> v2:172.21.15.184:3300/0 conn(0x7f9e0806dc40 0x7f9e0806e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:20:02.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0d1d0640 1 --2- 172.21.15.83:0/370720019 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e0814fb60 0x7f9e080c1f00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9e0806f140 tx=0x7f9e0001d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d33b2d545a7849 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:02.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e077fe640 1 -- 172.21.15.83:0/370720019 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e000a6710 con 0x7f9e0814fb60 2024-09-19T07:20:02.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e077fe640 1 -- 172.21.15.83:0/370720019 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e000a68b0 con 0x7f9e0814fb60 2024-09-19T07:20:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e077fe640 1 -- 172.21.15.83:0/370720019 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e000a6c80 con 0x7f9e0814fb60 2024-09-19T07:20:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 -- 172.21.15.83:0/370720019 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e0814fb60 msgr2=0x7f9e080c1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 --2- 172.21.15.83:0/370720019 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e0814fb60 0x7f9e080c1f00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9e0806f140 tx=0x7f9e0001d9e0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 -- 172.21.15.83:0/370720019 shutdown_connections 2024-09-19T07:20:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 --2- 172.21.15.83:0/370720019 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e0814fb60 0x7f9e080c1f00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 --2- 172.21.15.83:0/370720019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e08073720 0x7f9e0814f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 --2- 172.21.15.83:0/370720019 >> v2:172.21.15.184:3300/0 conn(0x7f9e0806dc40 0x7f9e0806e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 -- 172.21.15.83:0/370720019 >> 172.21.15.83:0/370720019 conn(0x7f9e0806bc80 msgr2=0x7f9e0806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 -- 172.21.15.83:0/370720019 shutdown_connections 2024-09-19T07:20:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 -- 172.21.15.83:0/370720019 wait complete. 2024-09-19T07:20:02.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 Processor -- start 2024-09-19T07:20:02.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 -- start start 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e0806dc40 0x7f9e08156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e08073720 0x7f9e08157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e0814fb60 0x7f9e081577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e080c27d0 con 0x7f9e0814fb60 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9e080c2650 con 0x7f9e0806dc40 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9e080c2950 con 0x7f9e08073720 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0d1d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e0814fb60 0x7f9e081577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0d1d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e0814fb60 0x7f9e081577c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49564/0 (socket says 172.21.15.83:49564) 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0d1d0640 1 -- 172.21.15.83:0/3913078847 learned_addr learned my addr 172.21.15.83:0/3913078847 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0c9cf640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e0806dc40 0x7f9e08156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e07fff640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e08073720 0x7f9e08157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0d1d0640 1 -- 172.21.15.83:0/3913078847 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e08073720 msgr2=0x7f9e08157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0d1d0640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e08073720 0x7f9e08157280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0d1d0640 1 -- 172.21.15.83:0/3913078847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e0806dc40 msgr2=0x7f9e08156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0d1d0640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e0806dc40 0x7f9e08156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0d1d0640 1 -- 172.21.15.83:0/3913078847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e081f03a0 con 0x7f9e0814fb60 2024-09-19T07:20:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0c9cf640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e0806dc40 0x7f9e08156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0d1d0640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e0814fb60 0x7f9e081577c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9e000b0040 tx=0x7f9e0001c220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e057fa640 1 -- 172.21.15.83:0/3913078847 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e000bc020 con 0x7f9e0814fb60 2024-09-19T07:20:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e057fa640 1 -- 172.21.15.83:0/3913078847 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e000b8620 con 0x7f9e0814fb60 2024-09-19T07:20:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 -- 172.21.15.83:0/3913078847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e080c0bd0 con 0x7f9e0814fb60 2024-09-19T07:20:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e057fa640 1 -- 172.21.15.83:0/3913078847 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e000b8a10 con 0x7f9e0814fb60 2024-09-19T07:20:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.362+0000 7f9e0ec33640 1 -- 172.21.15.83:0/3913078847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e080726e0 con 0x7f9e0814fb60 2024-09-19T07:20:02.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.366+0000 7f9e0ec33640 1 -- 172.21.15.83:0/3913078847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e0814f620 con 0x7f9e0814fb60 2024-09-19T07:20:02.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.366+0000 7f9e057fa640 1 -- 172.21.15.83:0/3913078847 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9e000bf020 con 0x7f9e0814fb60 2024-09-19T07:20:02.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.366+0000 7f9e057fa640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9dec0678c0 0x7f9dec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.366+0000 7f9e057fa640 1 -- 172.21.15.83:0/3913078847 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f9e001208b0 con 0x7f9e0814fb60 2024-09-19T07:20:02.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.366+0000 7f9e0c9cf640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9dec0678c0 0x7f9dec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.366+0000 7f9e0c9cf640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9dec0678c0 0x7f9dec069d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00d940 tx=0x7f9dfc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:02.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.370+0000 7f9e057fa640 1 -- 172.21.15.83:0/3913078847 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9e000c6030 con 0x7f9e0814fb60 2024-09-19T07:20:02.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 -- 172.21.15.83:0/3913078847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9e0806ebb0 con 0x7f9e0814fb60 2024-09-19T07:20:02.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e057fa640 1 -- 172.21.15.83:0/3913078847 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f9e000176f0 con 0x7f9e0814fb60 2024-09-19T07:20:02.535 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-09-19T07:20:02.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 -- 172.21.15.83:0/3913078847 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9dec0678c0 msgr2=0x7f9dec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9dec0678c0 0x7f9dec069d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00d940 tx=0x7f9dfc008040 comp rx=0 tx=0).stop 2024-09-19T07:20:02.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 -- 172.21.15.83:0/3913078847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e0814fb60 msgr2=0x7f9e081577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e0814fb60 0x7f9e081577c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9e000b0040 tx=0x7f9e0001c220 comp rx=0 tx=0).stop 2024-09-19T07:20:02.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 -- 172.21.15.83:0/3913078847 shutdown_connections 2024-09-19T07:20:02.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9dec0678c0 0x7f9dec069d80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e0814fb60 0x7f9e081577c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e08073720 0x7f9e08157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 --2- 172.21.15.83:0/3913078847 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e0806dc40 0x7f9e08156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 -- 172.21.15.83:0/3913078847 >> 172.21.15.83:0/3913078847 conn(0x7f9e0806bc80 msgr2=0x7f9e081516d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:02.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 -- 172.21.15.83:0/3913078847 shutdown_connections 2024-09-19T07:20:02.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.534+0000 7f9e0ec33640 1 -- 172.21.15.83:0/3913078847 wait complete. 2024-09-19T07:20:02.553 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-09-19T07:20:02.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.662+0000 7fd200138640 1 Processor -- start 2024-09-19T07:20:02.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.662+0000 7fd200138640 1 -- start start 2024-09-19T07:20:02.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd200138640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd1f80685e0 0x7fd1f80c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd200138640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1f80c2a30 0x7fd1f80c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd200138640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1f8074220 con 0x7fd1f80c3d40 2024-09-19T07:20:02.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd200138640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd1f80ad510 con 0x7fd1f80685e0 2024-09-19T07:20:02.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd200138640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd1f80ad030 con 0x7fd1f80c2a30 2024-09-19T07:20:02.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd1fded4640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd1f80c3d40 0x7fd1f80ab0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49986/0 (socket says 172.21.15.83:49986) 2024-09-19T07:20:02.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd1fd6d3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd1f80685e0 0x7fd1f80c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd1fded4640 1 -- 172.21.15.83:0/1925718353 learned_addr learned my addr 172.21.15.83:0/1925718353 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:02.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd1fe6d5640 1 --2- 172.21.15.83:0/1925718353 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1f80c2a30 0x7fd1f80c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd1fd6d3640 1 -- 172.21.15.83:0/1925718353 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1f80c2a30 msgr2=0x7fd1f80c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:02.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd1fd6d3640 1 --2- 172.21.15.83:0/1925718353 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1f80c2a30 0x7fd1f80c2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd1fd6d3640 1 -- 172.21.15.83:0/1925718353 >> v1:172.21.15.83:6789/0 conn(0x7fd1f80c3d40 legacy=0x7fd1f80ab0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd1fd6d3640 1 -- 172.21.15.83:0/1925718353 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f80c8830 con 0x7fd1f80685e0 2024-09-19T07:20:02.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd1fced2640 1 -- 172.21.15.83:0/1925718353 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1863242404 0 0) 0x7fd1f8074220 con 0x7fd1f80c3d40 2024-09-19T07:20:02.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd1fe6d5640 1 --2- 172.21.15.83:0/1925718353 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1f80c2a30 0x7fd1f80c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:02.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.666+0000 7fd1fd6d3640 1 --2- 172.21.15.83:0/1925718353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f80685e0 0x7fd1f80c24f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd1e80097c0 tx=0x7fd1e801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6ca588a41ceee2a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:02.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1fced2640 1 -- 172.21.15.83:0/1925718353 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e80b0070 con 0x7fd1f80685e0 2024-09-19T07:20:02.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1fced2640 1 -- 172.21.15.83:0/1925718353 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1e80ab600 con 0x7fd1f80685e0 2024-09-19T07:20:02.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1fced2640 1 -- 172.21.15.83:0/1925718353 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e80ab9d0 con 0x7fd1f80685e0 2024-09-19T07:20:02.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 -- 172.21.15.83:0/1925718353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f80685e0 msgr2=0x7fd1f80c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 --2- 172.21.15.83:0/1925718353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f80685e0 0x7fd1f80c24f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd1e80097c0 tx=0x7fd1e801c380 comp rx=0 tx=0).stop 2024-09-19T07:20:02.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 -- 172.21.15.83:0/1925718353 shutdown_connections 2024-09-19T07:20:02.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 --2- 172.21.15.83:0/1925718353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f80685e0 0x7fd1f80c24f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 --2- 172.21.15.83:0/1925718353 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1f80c2a30 0x7fd1f80c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 -- 172.21.15.83:0/1925718353 >> 172.21.15.83:0/1925718353 conn(0x7fd1f80c7b70 msgr2=0x7fd1f80c7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:02.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 -- 172.21.15.83:0/1925718353 shutdown_connections 2024-09-19T07:20:02.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 -- 172.21.15.83:0/1925718353 wait complete. 2024-09-19T07:20:02.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 Processor -- start 2024-09-19T07:20:02.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 -- start start 2024-09-19T07:20:02.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f80685e0 0x7fd1f80a8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1f80c2a30 0x7fd1f81eb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1f80c3d40 0x7fd1f81eb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f80a6f70 con 0x7fd1f80685e0 2024-09-19T07:20:02.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f80a6df0 con 0x7fd1f80c2a30 2024-09-19T07:20:02.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd1f80c4120 con 0x7fd1f80c3d40 2024-09-19T07:20:02.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1fded4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f80685e0 0x7fd1f80a8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1fded4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f80685e0 0x7fd1f80a8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49576/0 (socket says 172.21.15.83:49576) 2024-09-19T07:20:02.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1fded4640 1 -- 172.21.15.83:0/89811632 learned_addr learned my addr 172.21.15.83:0/89811632 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:02.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1fded4640 1 -- 172.21.15.83:0/89811632 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1f80c3d40 msgr2=0x7fd1f81eb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1fded4640 1 --2- 172.21.15.83:0/89811632 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1f80c3d40 0x7fd1f81eb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1fded4640 1 -- 172.21.15.83:0/89811632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1f80c2a30 msgr2=0x7fd1f81eb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1fded4640 1 --2- 172.21.15.83:0/89811632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1f80c2a30 0x7fd1f81eb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1fded4640 1 -- 172.21.15.83:0/89811632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f80a90f0 con 0x7fd1f80685e0 2024-09-19T07:20:02.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1fded4640 1 --2- 172.21.15.83:0/89811632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f80685e0 0x7fd1f80a8840 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd1f400c480 tx=0x7fd1f400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:02.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1ee7fc640 1 -- 172.21.15.83:0/89811632 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4018070 con 0x7fd1f80685e0 2024-09-19T07:20:02.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1ee7fc640 1 -- 172.21.15.83:0/89811632 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1f4007500 con 0x7fd1f80685e0 2024-09-19T07:20:02.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 -- 172.21.15.83:0/89811632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f806ec10 con 0x7fd1f80685e0 2024-09-19T07:20:02.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd1ee7fc640 1 -- 172.21.15.83:0/89811632 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4004a30 con 0x7fd1f80685e0 2024-09-19T07:20:02.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.670+0000 7fd200138640 1 -- 172.21.15.83:0/89811632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f806eed0 con 0x7fd1f80685e0 2024-09-19T07:20:02.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.674+0000 7fd200138640 1 -- 172.21.15.83:0/89811632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1f806f5f0 con 0x7fd1f80685e0 2024-09-19T07:20:02.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.674+0000 7fd1ee7fc640 1 -- 172.21.15.83:0/89811632 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd1f401b070 con 0x7fd1f80685e0 2024-09-19T07:20:02.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.674+0000 7fd1ee7fc640 1 --2- 172.21.15.83:0/89811632 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd1dc0678c0 0x7fd1dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.674+0000 7fd1fd6d3640 1 --2- 172.21.15.83:0/89811632 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd1dc0678c0 0x7fd1dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.674+0000 7fd1ee7fc640 1 -- 172.21.15.83:0/89811632 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fd1f4085230 con 0x7fd1f80685e0 2024-09-19T07:20:02.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.674+0000 7fd1fd6d3640 1 --2- 172.21.15.83:0/89811632 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd1dc0678c0 0x7fd1dc069d80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd1e80ad000 tx=0x7fd1e8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:02.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.678+0000 7fd1ee7fc640 1 -- 172.21.15.83:0/89811632 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd1f4055ad0 con 0x7fd1f80685e0 2024-09-19T07:20:02.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.826+0000 7fd200138640 1 -- 172.21.15.83:0/89811632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7fd1f80c24f0 con 0x7fd1f80685e0 2024-09-19T07:20:02.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.826+0000 7fd1ee7fc640 1 -- 172.21.15.83:0/89811632 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v25) ==== 99+0+11 (secure 0 0 0) 0x7fd1f40553d0 con 0x7fd1f80685e0 2024-09-19T07:20:02.828 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-09-19T07:20:02.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.830+0000 7fd200138640 1 -- 172.21.15.83:0/89811632 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd1dc0678c0 msgr2=0x7fd1dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.830+0000 7fd200138640 1 --2- 172.21.15.83:0/89811632 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd1dc0678c0 0x7fd1dc069d80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd1e80ad000 tx=0x7fd1e8008040 comp rx=0 tx=0).stop 2024-09-19T07:20:02.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.830+0000 7fd200138640 1 -- 172.21.15.83:0/89811632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f80685e0 msgr2=0x7fd1f80a8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.830+0000 7fd200138640 1 --2- 172.21.15.83:0/89811632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f80685e0 0x7fd1f80a8840 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd1f400c480 tx=0x7fd1f400c940 comp rx=0 tx=0).stop 2024-09-19T07:20:02.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.830+0000 7fd200138640 1 -- 172.21.15.83:0/89811632 shutdown_connections 2024-09-19T07:20:02.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.830+0000 7fd200138640 1 --2- 172.21.15.83:0/89811632 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd1dc0678c0 0x7fd1dc069d80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.830+0000 7fd200138640 1 --2- 172.21.15.83:0/89811632 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1f80c3d40 0x7fd1f81eb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.830+0000 7fd200138640 1 --2- 172.21.15.83:0/89811632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1f80c2a30 0x7fd1f81eb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.830+0000 7fd200138640 1 --2- 172.21.15.83:0/89811632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1f80685e0 0x7fd1f80a8840 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.830+0000 7fd200138640 1 -- 172.21.15.83:0/89811632 >> 172.21.15.83:0/89811632 conn(0x7fd1f80c7b70 msgr2=0x7fd1f8154f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:02.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.830+0000 7fd200138640 1 -- 172.21.15.83:0/89811632 shutdown_connections 2024-09-19T07:20:02.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.830+0000 7fd200138640 1 -- 172.21.15.83:0/89811632 wait complete. 2024-09-19T07:20:02.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-09-19T07:20:02.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.962+0000 7f979d3b6640 1 Processor -- start 2024-09-19T07:20:02.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.962+0000 7f979d3b6640 1 -- start start 2024-09-19T07:20:02.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f979806f3f0 0x7f979806bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f979814ff70 0x7f9798155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97980c2e90 con 0x7f979806c440 2024-09-19T07:20:02.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f97980c3130 con 0x7f979806f3f0 2024-09-19T07:20:02.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f97980c3410 con 0x7f979814ff70 2024-09-19T07:20:02.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f9796ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f979806f3f0 0x7f979806bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f9796ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f979806f3f0 0x7f979806bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49588/0 (socket says 172.21.15.83:49588) 2024-09-19T07:20:02.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f9796ffd640 1 -- 172.21.15.83:0/2249621184 learned_addr learned my addr 172.21.15.83:0/2249621184 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:02.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f9796ffd640 1 -- 172.21.15.83:0/2249621184 >> v2:172.21.15.184:3300/0 conn(0x7f979814ff70 msgr2=0x7f9798155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:02.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f9796ffd640 1 --2- 172.21.15.83:0/2249621184 >> v2:172.21.15.184:3300/0 conn(0x7f979814ff70 0x7f9798155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f9796ffd640 1 -- 172.21.15.83:0/2249621184 >> v1:172.21.15.184:6789/0 conn(0x7f979806c440 legacy=0x7f979814f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:02.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f9796ffd640 1 -- 172.21.15.83:0/2249621184 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97980c36f0 con 0x7f979806f3f0 2024-09-19T07:20:02.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f9796ffd640 1 --2- 172.21.15.83:0/2249621184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f979806f3f0 0x7f979806bf00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f97980c2c20 tx=0x7f978801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1acbe58a9566d852 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:02.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f97967fc640 1 -- 172.21.15.83:0/2249621184 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97880b0070 con 0x7f979806f3f0 2024-09-19T07:20:02.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f97967fc640 1 -- 172.21.15.83:0/2249621184 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97880ab600 con 0x7f979806f3f0 2024-09-19T07:20:02.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f97967fc640 1 -- 172.21.15.83:0/2249621184 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97880ab9d0 con 0x7f979806f3f0 2024-09-19T07:20:02.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 -- 172.21.15.83:0/2249621184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f979806f3f0 msgr2=0x7f979806bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 --2- 172.21.15.83:0/2249621184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f979806f3f0 0x7f979806bf00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f97980c2c20 tx=0x7f978801c4e0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 -- 172.21.15.83:0/2249621184 shutdown_connections 2024-09-19T07:20:02.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 --2- 172.21.15.83:0/2249621184 >> v2:172.21.15.184:3300/0 conn(0x7f979814ff70 0x7f9798155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 --2- 172.21.15.83:0/2249621184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f979806f3f0 0x7f979806bf00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 -- 172.21.15.83:0/2249621184 >> 172.21.15.83:0/2249621184 conn(0x7f97980c52f0 msgr2=0x7f97980c5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:02.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 -- 172.21.15.83:0/2249621184 shutdown_connections 2024-09-19T07:20:02.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.966+0000 7f979d3b6640 1 -- 172.21.15.83:0/2249621184 wait complete. 2024-09-19T07:20:02.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f979d3b6640 1 Processor -- start 2024-09-19T07:20:02.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f979d3b6640 1 -- start start 2024-09-19T07:20:02.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f979d3b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f979806c440 0x7f97980ac380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f979d3b6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f979806f3f0 0x7f9798147de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f979d3b6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f979814ff70 0x7f9798148320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f979d3b6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9798162640 con 0x7f979806c440 2024-09-19T07:20:02.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f979d3b6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f979806fa20 con 0x7f979814ff70 2024-09-19T07:20:02.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f979d3b6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f97981627c0 con 0x7f979806f3f0 2024-09-19T07:20:02.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f9796ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f979806c440 0x7f97980ac380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f9796ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f979806c440 0x7f97980ac380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49598/0 (socket says 172.21.15.83:49598) 2024-09-19T07:20:02.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f9796ffd640 1 -- 172.21.15.83:0/2762257031 learned_addr learned my addr 172.21.15.83:0/2762257031 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:02.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f9796ffd640 1 -- 172.21.15.83:0/2762257031 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f979806f3f0 msgr2=0x7f9798147de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f9796ffd640 1 --2- 172.21.15.83:0/2762257031 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f979806f3f0 0x7f9798147de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f9796ffd640 1 -- 172.21.15.83:0/2762257031 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f979814ff70 msgr2=0x7f9798148320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:02.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f9796ffd640 1 --2- 172.21.15.83:0/2762257031 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f979814ff70 0x7f9798148320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:02.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f9796ffd640 1 -- 172.21.15.83:0/2762257031 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97980c5bc0 con 0x7f979806c440 2024-09-19T07:20:02.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f9796ffd640 1 --2- 172.21.15.83:0/2762257031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f979806c440 0x7f97980ac380 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f97880ae040 tx=0x7f978801ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:02.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f978ffff640 1 -- 172.21.15.83:0/2762257031 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97880ba020 con 0x7f979806c440 2024-09-19T07:20:02.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f978ffff640 1 -- 172.21.15.83:0/2762257031 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97880b6640 con 0x7f979806c440 2024-09-19T07:20:02.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f979d3b6640 1 -- 172.21.15.83:0/2762257031 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97981edf20 con 0x7f979806c440 2024-09-19T07:20:02.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f978ffff640 1 -- 172.21.15.83:0/2762257031 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97880b0040 con 0x7f979806c440 2024-09-19T07:20:02.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f979d3b6640 1 -- 172.21.15.83:0/2762257031 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97981ee1e0 con 0x7f979806c440 2024-09-19T07:20:02.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f979d3b6640 1 -- 172.21.15.83:0/2762257031 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f979806bf00 con 0x7f979806c440 2024-09-19T07:20:02.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.970+0000 7f978ffff640 1 -- 172.21.15.83:0/2762257031 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f97880bd020 con 0x7f979806c440 2024-09-19T07:20:02.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.974+0000 7f978ffff640 1 --2- 172.21.15.83:0/2762257031 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f97780678c0 0x7f9778069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:02.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.974+0000 7f978ffff640 1 -- 172.21.15.83:0/2762257031 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f9788120310 con 0x7f979806c440 2024-09-19T07:20:02.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.974+0000 7f978e7fc640 1 --2- 172.21.15.83:0/2762257031 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f97780678c0 0x7f9778069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:02.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.974+0000 7f978e7fc640 1 --2- 172.21.15.83:0/2762257031 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f97780678c0 0x7f9778069d80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9780007fd0 tx=0x7f97800079a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:02.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:02.978+0000 7f978ffff640 1 -- 172.21.15.83:0/2762257031 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f97880c4030 con 0x7f979806c440 2024-09-19T07:20:03.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 -- 172.21.15.83:0/2762257031 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7f97980ac8c0 con 0x7f979806c440 2024-09-19T07:20:03.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f978ffff640 1 -- 172.21.15.83:0/2762257031 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v25) ==== 95+0+11 (secure 0 0 0) 0x7f97880160d0 con 0x7f979806c440 2024-09-19T07:20:03.138 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-09-19T07:20:03.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 -- 172.21.15.83:0/2762257031 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f97780678c0 msgr2=0x7f9778069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 --2- 172.21.15.83:0/2762257031 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f97780678c0 0x7f9778069d80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9780007fd0 tx=0x7f97800079a0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 -- 172.21.15.83:0/2762257031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f979806c440 msgr2=0x7f97980ac380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 --2- 172.21.15.83:0/2762257031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f979806c440 0x7f97980ac380 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f97880ae040 tx=0x7f978801ad20 comp rx=0 tx=0).stop 2024-09-19T07:20:03.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 -- 172.21.15.83:0/2762257031 shutdown_connections 2024-09-19T07:20:03.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 --2- 172.21.15.83:0/2762257031 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f97780678c0 0x7f9778069d80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 --2- 172.21.15.83:0/2762257031 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f979814ff70 0x7f9798148320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 --2- 172.21.15.83:0/2762257031 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f979806f3f0 0x7f9798147de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 --2- 172.21.15.83:0/2762257031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f979806c440 0x7f97980ac380 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 -- 172.21.15.83:0/2762257031 >> 172.21.15.83:0/2762257031 conn(0x7f97980c52f0 msgr2=0x7f979814e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:03.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 -- 172.21.15.83:0/2762257031 shutdown_connections 2024-09-19T07:20:03.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.138+0000 7f979d3b6640 1 -- 172.21.15.83:0/2762257031 wait complete. 2024-09-19T07:20:03.156 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-09-19T07:20:03.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc84196a640 1 Processor -- start 2024-09-19T07:20:03.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc84196a640 1 -- start start 2024-09-19T07:20:03.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc84196a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc83c0c32d0 0x7fc83c0c27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc84196a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc83c153450 0x7fc83c0c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc84196a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc83c153e30 0x7fc83c14c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc84196a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc83c0c1d70 con 0x7fc83c0c32d0 2024-09-19T07:20:03.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc84196a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc83c0c7800 con 0x7fc83c153e30 2024-09-19T07:20:03.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc84196a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc83c1538b0 con 0x7fc83c153450 2024-09-19T07:20:03.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc83affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc83c0c32d0 0x7fc83c0c27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc83affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc83c0c32d0 0x7fc83c0c27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49602/0 (socket says 172.21.15.83:49602) 2024-09-19T07:20:03.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc83affd640 1 -- 172.21.15.83:0/3884922794 learned_addr learned my addr 172.21.15.83:0/3884922794 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:03.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc83b7fe640 1 --2- 172.21.15.83:0/3884922794 >> v2:172.21.15.184:3300/0 conn(0x7fc83c153e30 0x7fc83c14c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc83a7fc640 1 --2- 172.21.15.83:0/3884922794 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc83c153450 0x7fc83c0c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc83affd640 1 -- 172.21.15.83:0/3884922794 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc83c153450 msgr2=0x7fc83c0c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:03.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc83affd640 1 --2- 172.21.15.83:0/3884922794 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc83c153450 0x7fc83c0c72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc83affd640 1 -- 172.21.15.83:0/3884922794 >> v2:172.21.15.184:3300/0 conn(0x7fc83c153e30 msgr2=0x7fc83c14c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:03.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc83affd640 1 --2- 172.21.15.83:0/3884922794 >> v2:172.21.15.184:3300/0 conn(0x7fc83c153e30 0x7fc83c14c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc83affd640 1 -- 172.21.15.83:0/3884922794 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c0c7a80 con 0x7fc83c0c32d0 2024-09-19T07:20:03.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.278+0000 7fc83b7fe640 1 --2- 172.21.15.83:0/3884922794 >> v2:172.21.15.184:3300/0 conn(0x7fc83c153e30 0x7fc83c14c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:03.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc83affd640 1 --2- 172.21.15.83:0/3884922794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc83c0c32d0 0x7fc83c0c27c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc83c0aa1b0 tx=0x7fc83001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b43601e1baa2468 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:03.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc839ffb640 1 -- 172.21.15.83:0/3884922794 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8300b0070 con 0x7fc83c0c32d0 2024-09-19T07:20:03.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc839ffb640 1 -- 172.21.15.83:0/3884922794 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8300ab600 con 0x7fc83c0c32d0 2024-09-19T07:20:03.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc839ffb640 1 -- 172.21.15.83:0/3884922794 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8300ab9d0 con 0x7fc83c0c32d0 2024-09-19T07:20:03.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc84196a640 1 -- 172.21.15.83:0/3884922794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc83c0c32d0 msgr2=0x7fc83c0c27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc84196a640 1 --2- 172.21.15.83:0/3884922794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc83c0c32d0 0x7fc83c0c27c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc83c0aa1b0 tx=0x7fc83001c400 comp rx=0 tx=0).stop 2024-09-19T07:20:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc84196a640 1 -- 172.21.15.83:0/3884922794 shutdown_connections 2024-09-19T07:20:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc84196a640 1 --2- 172.21.15.83:0/3884922794 >> v2:172.21.15.184:3300/0 conn(0x7fc83c153e30 0x7fc83c14c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc84196a640 1 --2- 172.21.15.83:0/3884922794 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc83c153450 0x7fc83c0c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc84196a640 1 --2- 172.21.15.83:0/3884922794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc83c0c32d0 0x7fc83c0c27c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc84196a640 1 -- 172.21.15.83:0/3884922794 >> 172.21.15.83:0/3884922794 conn(0x7fc83c06bc80 msgr2=0x7fc83c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc84196a640 1 -- 172.21.15.83:0/3884922794 shutdown_connections 2024-09-19T07:20:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc84196a640 1 -- 172.21.15.83:0/3884922794 wait complete. 2024-09-19T07:20:03.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc84196a640 1 Processor -- start 2024-09-19T07:20:03.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.282+0000 7fc84196a640 1 -- start start 2024-09-19T07:20:03.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc84196a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc83c0c32d0 0x7fc83c14a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc84196a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc83c153450 0x7fc83c14cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc84196a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc83c153e30 0x7fc83c14d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc84196a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c156740 con 0x7fc83c153450 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc84196a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c0c38c0 con 0x7fc83c153e30 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc84196a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc83c1568c0 con 0x7fc83c0c32d0 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc83c153450 0x7fc83c14cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc83c153450 0x7fc83c14cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49614/0 (socket says 172.21.15.83:49614) 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83a7fc640 1 -- 172.21.15.83:0/277280653 learned_addr learned my addr 172.21.15.83:0/277280653 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83b7fe640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc83c153e30 0x7fc83c14d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83affd640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc83c0c32d0 0x7fc83c14a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83a7fc640 1 -- 172.21.15.83:0/277280653 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc83c0c32d0 msgr2=0x7fc83c14a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83a7fc640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc83c0c32d0 0x7fc83c14a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83a7fc640 1 -- 172.21.15.83:0/277280653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc83c153e30 msgr2=0x7fc83c14d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83a7fc640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc83c153e30 0x7fc83c14d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83a7fc640 1 -- 172.21.15.83:0/277280653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c06e410 con 0x7fc83c153450 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83affd640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc83c0c32d0 0x7fc83c14a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83a7fc640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc83c153450 0x7fc83c14cef0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00c820 tx=0x7fc82c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc823fff640 1 -- 172.21.15.83:0/277280653 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c015070 con 0x7fc83c153450 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc823fff640 1 -- 172.21.15.83:0/277280653 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc82c004220 con 0x7fc83c153450 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc84196a640 1 -- 172.21.15.83:0/277280653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83c14aea0 con 0x7fc83c153450 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc823fff640 1 -- 172.21.15.83:0/277280653 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c00ce10 con 0x7fc83c153450 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc84196a640 1 -- 172.21.15.83:0/277280653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc83c14b140 con 0x7fc83c153450 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc84196a640 1 -- 172.21.15.83:0/277280653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc83c156d30 con 0x7fc83c153450 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc823fff640 1 -- 172.21.15.83:0/277280653 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc82c007830 con 0x7fc83c153450 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc823fff640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8280678c0 0x7fc828069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc823fff640 1 -- 172.21.15.83:0/277280653 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fc82c085d30 con 0x7fc83c153450 2024-09-19T07:20:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83affd640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8280678c0 0x7fc828069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.286+0000 7fc83affd640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8280678c0 0x7fc828069d80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc8300ad000 tx=0x7fc830008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.290+0000 7fc823fff640 1 -- 172.21.15.83:0/277280653 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc82c0565d0 con 0x7fc83c153450 2024-09-19T07:20:03.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.454+0000 7fc84196a640 1 -- 172.21.15.83:0/277280653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7fc83c14ab90 con 0x7fc83c153450 2024-09-19T07:20:03.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc823fff640 1 -- 172.21.15.83:0/277280653 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v25) ==== 99+0+11 (secure 0 0 0) 0x7fc82c055ed0 con 0x7fc83c153450 2024-09-19T07:20:03.458 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-09-19T07:20:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc84196a640 1 -- 172.21.15.83:0/277280653 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8280678c0 msgr2=0x7fc828069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc84196a640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8280678c0 0x7fc828069d80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc8300ad000 tx=0x7fc830008040 comp rx=0 tx=0).stop 2024-09-19T07:20:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc84196a640 1 -- 172.21.15.83:0/277280653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc83c153450 msgr2=0x7fc83c14cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc84196a640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc83c153450 0x7fc83c14cef0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00c820 tx=0x7fc82c00cce0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc84196a640 1 -- 172.21.15.83:0/277280653 shutdown_connections 2024-09-19T07:20:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc84196a640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8280678c0 0x7fc828069d80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc84196a640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc83c153e30 0x7fc83c14d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc84196a640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc83c153450 0x7fc83c14cef0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc84196a640 1 --2- 172.21.15.83:0/277280653 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc83c0c32d0 0x7fc83c14a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc84196a640 1 -- 172.21.15.83:0/277280653 >> 172.21.15.83:0/277280653 conn(0x7fc83c06bc80 msgr2=0x7fc83c06c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc84196a640 1 -- 172.21.15.83:0/277280653 shutdown_connections 2024-09-19T07:20:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.458+0000 7fc84196a640 1 -- 172.21.15.83:0/277280653 wait complete. 2024-09-19T07:20:03.477 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-09-19T07:20:03.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f5103ace640 1 Processor -- start 2024-09-19T07:20:03.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f5103ace640 1 -- start start 2024-09-19T07:20:03.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f5103ace640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f50fc0ab8b0 0x7f50fc152e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f5103ace640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50fc156510 con 0x7f50fc0c7520 2024-09-19T07:20:03.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f5103ace640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50fc1567f0 con 0x7f50fc0c09d0 2024-09-19T07:20:03.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f5103ace640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f50fc156ad0 con 0x7f50fc0ab8b0 2024-09-19T07:20:03.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f510186a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f50fc0c7520 0x7f50fc155c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50002/0 (socket says 172.21.15.83:50002) 2024-09-19T07:20:03.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f510186a640 1 -- 172.21.15.83:0/1424116824 learned_addr learned my addr 172.21.15.83:0/1424116824 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:03.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f510206b640 1 --2- 172.21.15.83:0/1424116824 >> v2:172.21.15.83:3300/0 conn(0x7f50fc0ab8b0 0x7f50fc152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f5100868640 1 -- 172.21.15.83:0/1424116824 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1660019990 0 0) 0x7f50fc156510 con 0x7f50fc0c7520 2024-09-19T07:20:03.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f5100868640 1 -- 172.21.15.83:0/1424116824 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50e8003530 con 0x7f50fc0c7520 2024-09-19T07:20:03.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f510206b640 1 -- 172.21.15.83:0/1424116824 >> v1:172.21.15.184:6789/0 conn(0x7f50fc0c09d0 legacy=0x7f50fc06e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f510206b640 1 -- 172.21.15.83:0/1424116824 >> v1:172.21.15.83:6789/0 conn(0x7f50fc0c7520 legacy=0x7f50fc155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.598+0000 7f510206b640 1 -- 172.21.15.83:0/1424116824 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50fc06d9e0 con 0x7f50fc0ab8b0 2024-09-19T07:20:03.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f510206b640 1 --2- 172.21.15.83:0/1424116824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50fc0ab8b0 0x7f50fc152e80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009d90 tx=0x7f50f401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb835f466df7a1a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:03.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5100868640 1 -- 172.21.15.83:0/1424116824 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f40b2070 con 0x7f50fc0ab8b0 2024-09-19T07:20:03.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5100868640 1 -- 172.21.15.83:0/1424116824 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f40a5830 con 0x7f50fc0ab8b0 2024-09-19T07:20:03.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5100868640 1 -- 172.21.15.83:0/1424116824 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f40a5c00 con 0x7f50fc0ab8b0 2024-09-19T07:20:03.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 -- 172.21.15.83:0/1424116824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50fc0ab8b0 msgr2=0x7f50fc152e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 --2- 172.21.15.83:0/1424116824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50fc0ab8b0 0x7f50fc152e80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009d90 tx=0x7f50f401c970 comp rx=0 tx=0).stop 2024-09-19T07:20:03.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 -- 172.21.15.83:0/1424116824 shutdown_connections 2024-09-19T07:20:03.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 --2- 172.21.15.83:0/1424116824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50fc0ab8b0 0x7f50fc152e80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 -- 172.21.15.83:0/1424116824 >> 172.21.15.83:0/1424116824 conn(0x7f50fc06bc80 msgr2=0x7f50fc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:03.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 -- 172.21.15.83:0/1424116824 shutdown_connections 2024-09-19T07:20:03.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 -- 172.21.15.83:0/1424116824 wait complete. 2024-09-19T07:20:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 Processor -- start 2024-09-19T07:20:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 -- start start 2024-09-19T07:20:03.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50fc0ab8b0 0x7f50fc0c39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50fc0c09d0 0x7f50fc072e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50fc0c7520 0x7f50fc0733c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc0c7a80 con 0x7f50fc0c09d0 2024-09-19T07:20:03.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc0c7900 con 0x7f50fc0c7520 2024-09-19T07:20:03.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5103ace640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f50fc163e00 con 0x7f50fc0ab8b0 2024-09-19T07:20:03.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5101069640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50fc0c09d0 0x7f50fc072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5101069640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50fc0c09d0 0x7f50fc072e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49638/0 (socket says 172.21.15.83:49638) 2024-09-19T07:20:03.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5101069640 1 -- 172.21.15.83:0/2808495439 learned_addr learned my addr 172.21.15.83:0/2808495439 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f510186a640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50fc0ab8b0 0x7f50fc0c39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f510206b640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50fc0c7520 0x7f50fc0733c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5101069640 1 -- 172.21.15.83:0/2808495439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50fc0ab8b0 msgr2=0x7f50fc0c39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5101069640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50fc0ab8b0 0x7f50fc0c39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5101069640 1 -- 172.21.15.83:0/2808495439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50fc0c7520 msgr2=0x7f50fc0733c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5101069640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50fc0c7520 0x7f50fc0733c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5101069640 1 -- 172.21.15.83:0/2808495439 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50fc071570 con 0x7f50fc0c09d0 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f510186a640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50fc0ab8b0 0x7f50fc0c39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f510206b640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50fc0c7520 0x7f50fc0733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f5101069640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50fc0c09d0 0x7f50fc072e80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f50ec00c450 tx=0x7f50ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f50f27fc640 1 -- 172.21.15.83:0/2808495439 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec016020 con 0x7f50fc0c09d0 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f50f27fc640 1 -- 172.21.15.83:0/2808495439 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50ec0052d0 con 0x7f50fc0c09d0 2024-09-19T07:20:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.602+0000 7f50f27fc640 1 -- 172.21.15.83:0/2808495439 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec007260 con 0x7f50fc0c09d0 2024-09-19T07:20:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.606+0000 7f5103ace640 1 -- 172.21.15.83:0/2808495439 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50fc071810 con 0x7f50fc0c09d0 2024-09-19T07:20:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.606+0000 7f5103ace640 1 -- 172.21.15.83:0/2808495439 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50fc1ee8f0 con 0x7f50fc0c09d0 2024-09-19T07:20:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.606+0000 7f5103ace640 1 -- 172.21.15.83:0/2808495439 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50fc152e80 con 0x7f50fc0c09d0 2024-09-19T07:20:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.606+0000 7f50f27fc640 1 -- 172.21.15.83:0/2808495439 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f50ec007400 con 0x7f50fc0c09d0 2024-09-19T07:20:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.606+0000 7f50f27fc640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50e00678f0 0x7f50e0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.606+0000 7f50f27fc640 1 -- 172.21.15.83:0/2808495439 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f50ec0858f0 con 0x7f50fc0c09d0 2024-09-19T07:20:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.606+0000 7f510186a640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50e00678f0 0x7f50e0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.606+0000 7f510186a640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50e00678f0 0x7f50e0069db0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f50f80011f0 tx=0x7f50f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.610+0000 7f50f27fc640 1 -- 172.21.15.83:0/2808495439 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f50ec020020 con 0x7f50fc0c09d0 2024-09-19T07:20:03.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.774+0000 7f5103ace640 1 -- 172.21.15.83:0/2808495439 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7f50fc0ac600 con 0x7f50fc0c09d0 2024-09-19T07:20:03.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.774+0000 7f50f27fc640 1 -- 172.21.15.83:0/2808495439 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v25) ==== 95+0+11 (secure 0 0 0) 0x7f50ec005480 con 0x7f50fc0c09d0 2024-09-19T07:20:03.775 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-09-19T07:20:03.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.774+0000 7f5103ace640 1 -- 172.21.15.83:0/2808495439 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50e00678f0 msgr2=0x7f50e0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.774+0000 7f5103ace640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50e00678f0 0x7f50e0069db0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f50f80011f0 tx=0x7f50f8009040 comp rx=0 tx=0).stop 2024-09-19T07:20:03.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.774+0000 7f5103ace640 1 -- 172.21.15.83:0/2808495439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50fc0c09d0 msgr2=0x7f50fc072e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.774+0000 7f5103ace640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50fc0c09d0 0x7f50fc072e80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f50ec00c450 tx=0x7f50ec00c910 comp rx=0 tx=0).stop 2024-09-19T07:20:03.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.774+0000 7f5103ace640 1 -- 172.21.15.83:0/2808495439 shutdown_connections 2024-09-19T07:20:03.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.774+0000 7f5103ace640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50e00678f0 0x7f50e0069db0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.774+0000 7f5103ace640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50fc0c7520 0x7f50fc0733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.774+0000 7f5103ace640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50fc0c09d0 0x7f50fc072e80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.774+0000 7f5103ace640 1 --2- 172.21.15.83:0/2808495439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50fc0ab8b0 0x7f50fc0c39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.774+0000 7f5103ace640 1 -- 172.21.15.83:0/2808495439 >> 172.21.15.83:0/2808495439 conn(0x7f50fc06bc80 msgr2=0x7f50fc155ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:03.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.778+0000 7f5103ace640 1 -- 172.21.15.83:0/2808495439 shutdown_connections 2024-09-19T07:20:03.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.778+0000 7f5103ace640 1 -- 172.21.15.83:0/2808495439 wait complete. 2024-09-19T07:20:03.794 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-09-19T07:20:03.795 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:03.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.906+0000 7f7b1f10b640 1 Processor -- start 2024-09-19T07:20:03.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.906+0000 7f7b1f10b640 1 -- start start 2024-09-19T07:20:03.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1f10b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b18152b20 0x7f7b180c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1f10b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7b180c45c0 0x7f7b18153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1f10b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b18150670 con 0x7f7b18150290 2024-09-19T07:20:03.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1f10b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f7b180716c0 con 0x7f7b180c45c0 2024-09-19T07:20:03.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1f10b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7b180c3920 con 0x7f7b18152b20 2024-09-19T07:20:03.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1cea7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7b18150290 0x7f7b180c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50010/0 (socket says 172.21.15.83:50010) 2024-09-19T07:20:03.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1cea7640 1 -- 172.21.15.83:0/2499462012 learned_addr learned my addr 172.21.15.83:0/2499462012 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:03.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 775379414 0 0) 0x7f7b18150670 con 0x7f7b18150290 2024-09-19T07:20:03.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b04003530 con 0x7f7b18150290 2024-09-19T07:20:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b17fff640 1 --2- 172.21.15.83:0/2499462012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b18152b20 0x7f7b180c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1d6a8640 1 --2- 172.21.15.83:0/2499462012 >> v2:172.21.15.184:3300/0 conn(0x7f7b180c45c0 0x7f7b18153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 870372329 0 0) 0x7f7b04003530 con 0x7f7b18150290 2024-09-19T07:20:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b18152b20 msgr2=0x7f7b180c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 --2- 172.21.15.83:0/2499462012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b18152b20 0x7f7b180c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 >> v2:172.21.15.184:3300/0 conn(0x7f7b180c45c0 msgr2=0x7f7b18153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 --2- 172.21.15.83:0/2499462012 >> v2:172.21.15.184:3300/0 conn(0x7f7b180c45c0 0x7f7b18153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b180c2880 con 0x7f7b18150290 2024-09-19T07:20:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f7b10001a00 con 0x7f7b18150290 2024-09-19T07:20:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1d6a8640 1 --2- 172.21.15.83:0/2499462012 >> v2:172.21.15.184:3300/0 conn(0x7f7b180c45c0 0x7f7b18153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:20:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 >> v1:172.21.15.83:6789/0 conn(0x7f7b18150290 legacy=0x7f7b180c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 --2- 172.21.15.83:0/2499462012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b040039f0 0x7f7b04003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 --2- 172.21.15.83:0/2499462012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b04004810 0x7f7b04004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 --2- 172.21.15.83:0/2499462012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b18152b20 0x7f7b040051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0400ba80 con 0x7f7b040039f0 2024-09-19T07:20:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0400bc00 con 0x7f7b04004810 2024-09-19T07:20:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7b0400be00 con 0x7f7b18152b20 2024-09-19T07:20:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1cea7640 1 --2- 172.21.15.83:0/2499462012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b040039f0 0x7f7b04003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b17fff640 1 --2- 172.21.15.83:0/2499462012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b04004810 0x7f7b04004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1cea7640 1 -- 172.21.15.83:0/2499462012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b18152b20 msgr2=0x7f7b040051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1cea7640 1 --2- 172.21.15.83:0/2499462012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b18152b20 0x7f7b040051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1cea7640 1 -- 172.21.15.83:0/2499462012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b04004810 msgr2=0x7f7b04004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1cea7640 1 --2- 172.21.15.83:0/2499462012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b04004810 0x7f7b04004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1cea7640 1 -- 172.21.15.83:0/2499462012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b04003530 con 0x7f7b040039f0 2024-09-19T07:20:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b17fff640 1 --2- 172.21.15.83:0/2499462012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b04004810 0x7f7b04004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:03.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b1cea7640 1 --2- 172.21.15.83:0/2499462012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b040039f0 0x7f7b04003e10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7b1000a780 tx=0x7f7b1001d280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8744f33562909c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:03.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1000a3d0 con 0x7f7b040039f0 2024-09-19T07:20:03.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b100ad650 con 0x7f7b040039f0 2024-09-19T07:20:03.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.910+0000 7f7b177fe640 1 -- 172.21.15.83:0/2499462012 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b100ada80 con 0x7f7b040039f0 2024-09-19T07:20:03.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 -- 172.21.15.83:0/2499462012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b040039f0 msgr2=0x7f7b04003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 --2- 172.21.15.83:0/2499462012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b040039f0 0x7f7b04003e10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7b1000a780 tx=0x7f7b1001d280 comp rx=0 tx=0).stop 2024-09-19T07:20:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1d6a8640 1 -- 172.21.15.83:0/2499462012 reap_dead start 2024-09-19T07:20:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 -- 172.21.15.83:0/2499462012 shutdown_connections 2024-09-19T07:20:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 -- 172.21.15.83:0/2499462012 >> 172.21.15.83:0/2499462012 conn(0x7f7b1806bc80 msgr2=0x7f7b1806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 -- 172.21.15.83:0/2499462012 shutdown_connections 2024-09-19T07:20:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 -- 172.21.15.83:0/2499462012 wait complete. 2024-09-19T07:20:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 Processor -- start 2024-09-19T07:20:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 -- start start 2024-09-19T07:20:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b040039f0 0x7f7b18144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b18147f60 0x7f7b181ed460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b18147450 0x7f7b18147900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7b18154980 con 0x7f7b18147f60 2024-09-19T07:20:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b18154800 con 0x7f7b18147450 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7b1815db50 con 0x7f7b040039f0 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b18147f60 0x7f7b181ed460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b18147f60 0x7f7b181ed460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49650/0 (socket says 172.21.15.83:49650) 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b17fff640 1 -- 172.21.15.83:0/816678073 learned_addr learned my addr 172.21.15.83:0/816678073 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1cea7640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b040039f0 0x7f7b18144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1d6a8640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b18147450 0x7f7b18147900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b17fff640 1 -- 172.21.15.83:0/816678073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b040039f0 msgr2=0x7f7b18144940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b17fff640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b040039f0 0x7f7b18144940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b17fff640 1 -- 172.21.15.83:0/816678073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b18147450 msgr2=0x7f7b18147900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b17fff640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b18147450 0x7f7b18147900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b17fff640 1 -- 172.21.15.83:0/816678073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b18150cc0 con 0x7f7b18147f60 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1d6a8640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b18147450 0x7f7b18147900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b17fff640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b18147f60 0x7f7b181ed460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7b0000e7e0 tx=0x7f7b0000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b157fa640 1 -- 172.21.15.83:0/816678073 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b00017070 con 0x7f7b18147f60 2024-09-19T07:20:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b157fa640 1 -- 172.21.15.83:0/816678073 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b0000c670 con 0x7f7b18147f60 2024-09-19T07:20:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 -- 172.21.15.83:0/816678073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b181ecbf0 con 0x7f7b18147f60 2024-09-19T07:20:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b157fa640 1 -- 172.21.15.83:0/816678073 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0000f040 con 0x7f7b18147f60 2024-09-19T07:20:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.914+0000 7f7b1f10b640 1 -- 172.21.15.83:0/816678073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b181523d0 con 0x7f7b18147f60 2024-09-19T07:20:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.918+0000 7f7b1f10b640 1 -- 172.21.15.83:0/816678073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b18153eb0 con 0x7f7b18147f60 2024-09-19T07:20:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.918+0000 7f7b157fa640 1 -- 172.21.15.83:0/816678073 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7b0001a070 con 0x7f7b18147f60 2024-09-19T07:20:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.918+0000 7f7b157fa640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7af80678c0 0x7f7af8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.918+0000 7f7b157fa640 1 -- 172.21.15.83:0/816678073 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f7b00085aa0 con 0x7f7b18147f60 2024-09-19T07:20:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.918+0000 7f7b1cea7640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7af80678c0 0x7f7af8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.918+0000 7f7b1cea7640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7af80678c0 0x7f7af8069d80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7b04004790 tx=0x7f7b100a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:03.922+0000 7f7b157fa640 1 -- 172.21.15.83:0/816678073 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7b00056340 con 0x7f7b18147f60 2024-09-19T07:20:04.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.102+0000 7f7b1f10b640 1 -- 172.21.15.83:0/816678073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b181521a0 con 0x7f7b18147f60 2024-09-19T07:20:04.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.102+0000 7f7b157fa640 1 -- 172.21.15.83:0/816678073 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f7b00055c40 con 0x7f7b18147f60 2024-09-19T07:20:04.105 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:04.106 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:04.106 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:04.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.106+0000 7f7b1f10b640 1 -- 172.21.15.83:0/816678073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7af80678c0 msgr2=0x7f7af8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.106+0000 7f7b1f10b640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7af80678c0 0x7f7af8069d80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7b04004790 tx=0x7f7b100a7000 comp rx=0 tx=0).stop 2024-09-19T07:20:04.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.106+0000 7f7b1f10b640 1 -- 172.21.15.83:0/816678073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b18147f60 msgr2=0x7f7b181ed460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.106+0000 7f7b1f10b640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b18147f60 0x7f7b181ed460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7b0000e7e0 tx=0x7f7b0000eca0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.106+0000 7f7b1f10b640 1 -- 172.21.15.83:0/816678073 shutdown_connections 2024-09-19T07:20:04.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.106+0000 7f7b1f10b640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7af80678c0 0x7f7af8069d80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.106+0000 7f7b1f10b640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b18147450 0x7f7b18147900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.106+0000 7f7b1f10b640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b18147f60 0x7f7b181ed460 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.106+0000 7f7b1f10b640 1 --2- 172.21.15.83:0/816678073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b040039f0 0x7f7b18144940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.106+0000 7f7b1f10b640 1 -- 172.21.15.83:0/816678073 >> 172.21.15.83:0/816678073 conn(0x7f7b1806bc80 msgr2=0x7f7b1806e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:04.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.106+0000 7f7b1f10b640 1 -- 172.21.15.83:0/816678073 shutdown_connections 2024-09-19T07:20:04.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.106+0000 7f7b1f10b640 1 -- 172.21.15.83:0/816678073 wait complete. 2024-09-19T07:20:04.123 INFO:tasks.mds_thrash:Ready to start thrashing 2024-09-19T07:20:04.123 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-09-19T07:20:04.123 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:20:04.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.242+0000 7f6bcf910640 1 Processor -- start 2024-09-19T07:20:04.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.242+0000 7f6bcf910640 1 -- start start 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcf910640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6bc8150290 0x7f6bc80c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcf910640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bc8153eb0 0x7f6bc8154340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcf910640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bc8150670 con 0x7f6bc8152b20 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcf910640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6bc80716c0 con 0x7f6bc8150290 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcf910640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6bc80c3920 con 0x7f6bc8153eb0 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcd6ac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6bc8150290 0x7f6bc80c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcceab640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6bc8152b20 0x7f6bc80c4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50020/0 (socket says 172.21.15.83:50020) 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcceab640 1 -- 172.21.15.83:0/3847336834 learned_addr learned my addr 172.21.15.83:0/3847336834 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcdead640 1 --2- 172.21.15.83:0/3847336834 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bc8153eb0 0x7f6bc8154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcd6ac640 1 -- 172.21.15.83:0/3847336834 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bc8153eb0 msgr2=0x7f6bc8154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcd6ac640 1 --2- 172.21.15.83:0/3847336834 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bc8153eb0 0x7f6bc8154340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcd6ac640 1 -- 172.21.15.83:0/3847336834 >> v1:172.21.15.83:6789/0 conn(0x7f6bc8152b20 legacy=0x7f6bc80c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcd6ac640 1 -- 172.21.15.83:0/3847336834 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc80c2880 con 0x7f6bc8150290 2024-09-19T07:20:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcdead640 1 --2- 172.21.15.83:0/3847336834 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bc8153eb0 0x7f6bc8154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bcd6ac640 1 --2- 172.21.15.83:0/3847336834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bc8150290 0x7f6bc80c1ff0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00097c0 tx=0x7f6bc001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4980da4a2f6d54 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bbffff640 1 -- 172.21.15.83:0/3847336834 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc00b2070 con 0x7f6bc8150290 2024-09-19T07:20:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bbffff640 1 -- 172.21.15.83:0/3847336834 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bc00ad640 con 0x7f6bc8150290 2024-09-19T07:20:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.246+0000 7f6bbffff640 1 -- 172.21.15.83:0/3847336834 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc00ada10 con 0x7f6bc8150290 2024-09-19T07:20:04.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 -- 172.21.15.83:0/3847336834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bc8150290 msgr2=0x7f6bc80c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 --2- 172.21.15.83:0/3847336834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bc8150290 0x7f6bc80c1ff0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00097c0 tx=0x7f6bc001c4a0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 -- 172.21.15.83:0/3847336834 shutdown_connections 2024-09-19T07:20:04.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 --2- 172.21.15.83:0/3847336834 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bc8153eb0 0x7f6bc8154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 --2- 172.21.15.83:0/3847336834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bc8150290 0x7f6bc80c1ff0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 -- 172.21.15.83:0/3847336834 >> 172.21.15.83:0/3847336834 conn(0x7f6bc806bc80 msgr2=0x7f6bc806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:04.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 -- 172.21.15.83:0/3847336834 shutdown_connections 2024-09-19T07:20:04.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 -- 172.21.15.83:0/3847336834 wait complete. 2024-09-19T07:20:04.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 Processor -- start 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 -- start start 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bc8150290 0x7f6bc80aa870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bc8152b20 0x7f6bc80c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bc8153eb0 0x7f6bc8152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc80c4790 con 0x7f6bc8152b20 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc80c2530 con 0x7f6bc8150290 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6bc80c4910 con 0x7f6bc8153eb0 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcceab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bc8152b20 0x7f6bc80c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcceab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bc8152b20 0x7f6bc80c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49668/0 (socket says 172.21.15.83:49668) 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcceab640 1 -- 172.21.15.83:0/1676267974 learned_addr learned my addr 172.21.15.83:0/1676267974 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcd6ac640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bc8150290 0x7f6bc80aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcceab640 1 -- 172.21.15.83:0/1676267974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bc8153eb0 msgr2=0x7f6bc8152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcceab640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bc8153eb0 0x7f6bc8152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcceab640 1 -- 172.21.15.83:0/1676267974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bc8150290 msgr2=0x7f6bc80aa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcceab640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bc8150290 0x7f6bc80aa870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcceab640 1 -- 172.21.15.83:0/1676267974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc8073d40 con 0x7f6bc8152b20 2024-09-19T07:20:04.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcceab640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bc8152b20 0x7f6bc80c6d70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6bb800c420 tx=0x7f6bb800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:04.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bbdffb640 1 -- 172.21.15.83:0/1676267974 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8017070 con 0x7f6bc8152b20 2024-09-19T07:20:04.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bbdffb640 1 -- 172.21.15.83:0/1676267974 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bb8005cf0 con 0x7f6bc8152b20 2024-09-19T07:20:04.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 -- 172.21.15.83:0/1676267974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bc8073fd0 con 0x7f6bc8152b20 2024-09-19T07:20:04.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bbdffb640 1 -- 172.21.15.83:0/1676267974 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb80072f0 con 0x7f6bc8152b20 2024-09-19T07:20:04.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 -- 172.21.15.83:0/1676267974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bc814ecb0 con 0x7f6bc8152b20 2024-09-19T07:20:04.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.250+0000 7f6bcf910640 1 -- 172.21.15.83:0/1676267974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bc80c1ff0 con 0x7f6bc8152b20 2024-09-19T07:20:04.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.254+0000 7f6bbdffb640 1 -- 172.21.15.83:0/1676267974 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6bb801a070 con 0x7f6bc8152b20 2024-09-19T07:20:04.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.254+0000 7f6bbdffb640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6bac0678c0 0x7f6bac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:04.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.254+0000 7f6bcd6ac640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6bac0678c0 0x7f6bac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:04.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.254+0000 7f6bbdffb640 1 -- 172.21.15.83:0/1676267974 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f6bb8085300 con 0x7f6bc8152b20 2024-09-19T07:20:04.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.254+0000 7f6bcd6ac640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6bac0678c0 0x7f6bac069d80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00af000 tx=0x7f6bc00a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:04.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.258+0000 7f6bbdffb640 1 -- 172.21.15.83:0/1676267974 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6bb8055ba0 con 0x7f6bc8152b20 2024-09-19T07:20:04.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.422+0000 7f6bcf910640 1 -- 172.21.15.83:0/1676267974 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6bc814b8c0 con 0x7f6bac0678c0 2024-09-19T07:20:04.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.454+0000 7f6bbdffb640 1 -- 172.21.15.83:0/1676267974 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490337 (secure 0 0 0) 0x7f6bc814b8c0 con 0x7f6bac0678c0 2024-09-19T07:20:04.461 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:04.464 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:20:04.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.462+0000 7f6bcf910640 1 -- 172.21.15.83:0/1676267974 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6bac0678c0 msgr2=0x7f6bac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.462+0000 7f6bcf910640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6bac0678c0 0x7f6bac069d80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00af000 tx=0x7f6bc00a7070 comp rx=0 tx=0).stop 2024-09-19T07:20:04.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.462+0000 7f6bcf910640 1 -- 172.21.15.83:0/1676267974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bc8152b20 msgr2=0x7f6bc80c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.462+0000 7f6bcf910640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bc8152b20 0x7f6bc80c6d70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6bb800c420 tx=0x7f6bb800c8e0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.462+0000 7f6bcf910640 1 -- 172.21.15.83:0/1676267974 shutdown_connections 2024-09-19T07:20:04.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.462+0000 7f6bcf910640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6bac0678c0 0x7f6bac069d80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.462+0000 7f6bcf910640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bc8153eb0 0x7f6bc8152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.462+0000 7f6bcf910640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bc8152b20 0x7f6bc80c6d70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.462+0000 7f6bcf910640 1 --2- 172.21.15.83:0/1676267974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bc8150290 0x7f6bc80aa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.462+0000 7f6bcf910640 1 -- 172.21.15.83:0/1676267974 >> 172.21.15.83:0/1676267974 conn(0x7f6bc806bc80 msgr2=0x7f6bc8152fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:04.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.462+0000 7f6bcf910640 1 -- 172.21.15.83:0/1676267974 shutdown_connections 2024-09-19T07:20:04.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.462+0000 7f6bcf910640 1 -- 172.21.15.83:0/1676267974 wait complete. 2024-09-19T07:20:04.481 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-19T07:20:02.820535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596074,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":253,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223724,"kb_used_data":3336,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750996,"statfs":{"total":773094113280,"available":772865019904,"internally_reserved":0,"allocated":3416064,"data_stored":2011625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":886,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003877"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364888+0000","last_change":"2024-09-19T07:19:40.358457+0000","last_active":"2024-09-19T07:19:42.364888+0000","last_peered":"2024-09-19T07:19:42.364888+0000","last_clean":"2024-09-19T07:19:42.364888+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:19:42.364888+0000","last_undegraded":"2024-09-19T07:19:42.364888+0000","last_fullsized":"2024-09-19T07:19:42.364888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370195+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:19:42.370195+0000","last_peered":"2024-09-19T07:19:42.370195+0000","last_clean":"2024-09-19T07:19:42.370195+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:19:42.370195+0000","last_undegraded":"2024-09-19T07:19:42.370195+0000","last_fullsized":"2024-09-19T07:19:42.370195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:57.799442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372168+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:19:42.372168+0000","last_peered":"2024-09-19T07:19:42.372168+0000","last_clean":"2024-09-19T07:19:42.372168+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:19:42.372168+0000","last_undegraded":"2024-09-19T07:19:42.372168+0000","last_fullsized":"2024-09-19T07:19:42.372168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368511+0000","last_change":"2024-09-19T07:19:39.401621+0000","last_active":"2024-09-19T07:19:42.368511+0000","last_peered":"2024-09-19T07:19:42.368511+0000","last_clean":"2024-09-19T07:19:42.368511+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:19:42.368511+0000","last_undegraded":"2024-09-19T07:19:42.368511+0000","last_fullsized":"2024-09-19T07:19:42.368511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:32.095381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366428+0000","last_change":"2024-09-19T07:19:40.366073+0000","last_active":"2024-09-19T07:19:42.366428+0000","last_peered":"2024-09-19T07:19:42.366428+0000","last_clean":"2024-09-19T07:19:42.366428+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:19:42.366428+0000","last_undegraded":"2024-09-19T07:19:42.366428+0000","last_fullsized":"2024-09-19T07:19:42.366428+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372588+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:19:42.372588+0000","last_peered":"2024-09-19T07:19:42.372588+0000","last_clean":"2024-09-19T07:19:42.372588+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:19:42.372588+0000","last_undegraded":"2024-09-19T07:19:42.372588+0000","last_fullsized":"2024-09-19T07:19:42.372588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369500+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:19:42.369500+0000","last_peered":"2024-09-19T07:19:42.369500+0000","last_clean":"2024-09-19T07:19:42.369500+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:19:42.369500+0000","last_undegraded":"2024-09-19T07:19:42.369500+0000","last_fullsized":"2024-09-19T07:19:42.369500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:15.036396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364782+0000","last_change":"2024-09-19T07:19:39.403810+0000","last_active":"2024-09-19T07:19:42.364782+0000","last_peered":"2024-09-19T07:19:42.364782+0000","last_clean":"2024-09-19T07:19:42.364782+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:19:42.364782+0000","last_undegraded":"2024-09-19T07:19:42.364782+0000","last_fullsized":"2024-09-19T07:19:42.364782+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:55.714964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365790+0000","last_change":"2024-09-19T07:19:40.390455+0000","last_active":"2024-09-19T07:19:42.365790+0000","last_peered":"2024-09-19T07:19:42.365790+0000","last_clean":"2024-09-19T07:19:42.365790+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:19:42.365790+0000","last_undegraded":"2024-09-19T07:19:42.365790+0000","last_fullsized":"2024-09-19T07:19:42.365790+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367307+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:19:42.367307+0000","last_peered":"2024-09-19T07:19:42.367307+0000","last_clean":"2024-09-19T07:19:42.367307+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:19:42.367307+0000","last_undegraded":"2024-09-19T07:19:42.367307+0000","last_fullsized":"2024-09-19T07:19:42.367307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353956+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:19:41.353956+0000","last_peered":"2024-09-19T07:19:41.353956+0000","last_clean":"2024-09-19T07:19:41.353956+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:19:41.353956+0000","last_undegraded":"2024-09-19T07:19:41.353956+0000","last_fullsized":"2024-09-19T07:19:41.353956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366091+0000","last_change":"2024-09-19T07:19:39.408309+0000","last_active":"2024-09-19T07:19:42.366091+0000","last_peered":"2024-09-19T07:19:42.366091+0000","last_clean":"2024-09-19T07:19:42.366091+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:19:42.366091+0000","last_undegraded":"2024-09-19T07:19:42.366091+0000","last_fullsized":"2024-09-19T07:19:42.366091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:19.396261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353849+0000","last_change":"2024-09-19T07:19:40.372543+0000","last_active":"2024-09-19T07:19:41.353849+0000","last_peered":"2024-09-19T07:19:41.353849+0000","last_clean":"2024-09-19T07:19:41.353849+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:19:41.353849+0000","last_undegraded":"2024-09-19T07:19:41.353849+0000","last_fullsized":"2024-09-19T07:19:41.353849+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:53:15.682896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367705+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:19:42.367705+0000","last_peered":"2024-09-19T07:19:42.367705+0000","last_clean":"2024-09-19T07:19:42.367705+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:19:42.367705+0000","last_undegraded":"2024-09-19T07:19:42.367705+0000","last_fullsized":"2024-09-19T07:19:42.367705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355253+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:19:41.355253+0000","last_peered":"2024-09-19T07:19:41.355253+0000","last_clean":"2024-09-19T07:19:41.355253+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:19:41.355253+0000","last_undegraded":"2024-09-19T07:19:41.355253+0000","last_fullsized":"2024-09-19T07:19:41.355253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366176+0000","last_change":"2024-09-19T07:19:39.407045+0000","last_active":"2024-09-19T07:19:42.366176+0000","last_peered":"2024-09-19T07:19:42.366176+0000","last_clean":"2024-09-19T07:19:42.366176+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:19:42.366176+0000","last_undegraded":"2024-09-19T07:19:42.366176+0000","last_fullsized":"2024-09-19T07:19:42.366176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:24.846974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367577+0000","last_change":"2024-09-19T07:19:40.396520+0000","last_active":"2024-09-19T07:19:42.367577+0000","last_peered":"2024-09-19T07:19:42.367577+0000","last_clean":"2024-09-19T07:19:42.367577+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:19:42.367577+0000","last_undegraded":"2024-09-19T07:19:42.367577+0000","last_fullsized":"2024-09-19T07:19:42.367577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353504+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:19:41.353504+0000","last_peered":"2024-09-19T07:19:41.353504+0000","last_clean":"2024-09-19T07:19:41.353504+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:19:41.353504+0000","last_undegraded":"2024-09-19T07:19:41.353504+0000","last_fullsized":"2024-09-19T07:19:41.353504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367046+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:19:42.367046+0000","last_peered":"2024-09-19T07:19:42.367046+0000","last_clean":"2024-09-19T07:19:42.367046+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:19:42.367046+0000","last_undegraded":"2024-09-19T07:19:42.367046+0000","last_fullsized":"2024-09-19T07:19:42.367046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368034+0000","last_change":"2024-09-19T07:19:39.410500+0000","last_active":"2024-09-19T07:19:42.368034+0000","last_peered":"2024-09-19T07:19:42.368034+0000","last_clean":"2024-09-19T07:19:42.368034+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:19:42.368034+0000","last_undegraded":"2024-09-19T07:19:42.368034+0000","last_fullsized":"2024-09-19T07:19:42.368034+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:04.182606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349177+0000","last_change":"2024-09-19T07:19:40.381259+0000","last_active":"2024-09-19T07:19:41.349177+0000","last_peered":"2024-09-19T07:19:41.349177+0000","last_clean":"2024-09-19T07:19:41.349177+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:19:41.349177+0000","last_undegraded":"2024-09-19T07:19:41.349177+0000","last_fullsized":"2024-09-19T07:19:41.349177+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:22.017682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370825+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:19:42.370825+0000","last_peered":"2024-09-19T07:19:42.370825+0000","last_clean":"2024-09-19T07:19:42.370825+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:19:42.370825+0000","last_undegraded":"2024-09-19T07:19:42.370825+0000","last_fullsized":"2024-09-19T07:19:42.370825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367918+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:19:42.367918+0000","last_peered":"2024-09-19T07:19:42.367918+0000","last_clean":"2024-09-19T07:19:42.367918+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:19:42.367918+0000","last_undegraded":"2024-09-19T07:19:42.367918+0000","last_fullsized":"2024-09-19T07:19:42.367918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367682+0000","last_change":"2024-09-19T07:19:39.412123+0000","last_active":"2024-09-19T07:19:42.367682+0000","last_peered":"2024-09-19T07:19:42.367682+0000","last_clean":"2024-09-19T07:19:42.367682+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:19:42.367682+0000","last_undegraded":"2024-09-19T07:19:42.367682+0000","last_fullsized":"2024-09-19T07:19:42.367682+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367943+0000","last_change":"2024-09-19T07:19:40.383214+0000","last_active":"2024-09-19T07:19:42.367943+0000","last_peered":"2024-09-19T07:19:42.367943+0000","last_clean":"2024-09-19T07:19:42.367943+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:19:42.367943+0000","last_undegraded":"2024-09-19T07:19:42.367943+0000","last_fullsized":"2024-09-19T07:19:42.367943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367066+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:19:42.367066+0000","last_peered":"2024-09-19T07:19:42.367066+0000","last_clean":"2024-09-19T07:19:42.367066+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:19:42.367066+0000","last_undegraded":"2024-09-19T07:19:42.367066+0000","last_fullsized":"2024-09-19T07:19:42.367066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367094+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:19:42.367094+0000","last_peered":"2024-09-19T07:19:42.367094+0000","last_clean":"2024-09-19T07:19:42.367094+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:19:42.367094+0000","last_undegraded":"2024-09-19T07:19:42.367094+0000","last_fullsized":"2024-09-19T07:19:42.367094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369704+0000","last_change":"2024-09-19T07:19:39.408228+0000","last_active":"2024-09-19T07:19:42.369704+0000","last_peered":"2024-09-19T07:19:42.369704+0000","last_clean":"2024-09-19T07:19:42.369704+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:19:42.369704+0000","last_undegraded":"2024-09-19T07:19:42.369704+0000","last_fullsized":"2024-09-19T07:19:42.369704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355858+0000","last_change":"2024-09-19T07:19:40.395566+0000","last_active":"2024-09-19T07:19:41.355858+0000","last_peered":"2024-09-19T07:19:41.355858+0000","last_clean":"2024-09-19T07:19:41.355858+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:19:41.355858+0000","last_undegraded":"2024-09-19T07:19:41.355858+0000","last_fullsized":"2024-09-19T07:19:41.355858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:54:02.766021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367642+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:19:42.367642+0000","last_peered":"2024-09-19T07:19:42.367642+0000","last_clean":"2024-09-19T07:19:42.367642+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:19:42.367642+0000","last_undegraded":"2024-09-19T07:19:42.367642+0000","last_fullsized":"2024-09-19T07:19:42.367642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366078+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:19:42.366078+0000","last_peered":"2024-09-19T07:19:42.366078+0000","last_clean":"2024-09-19T07:19:42.366078+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:19:42.366078+0000","last_undegraded":"2024-09-19T07:19:42.366078+0000","last_fullsized":"2024-09-19T07:19:42.366078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368203+0000","last_change":"2024-09-19T07:19:39.424661+0000","last_active":"2024-09-19T07:19:42.368203+0000","last_peered":"2024-09-19T07:19:42.368203+0000","last_clean":"2024-09-19T07:19:42.368203+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:19:42.368203+0000","last_undegraded":"2024-09-19T07:19:42.368203+0000","last_fullsized":"2024-09-19T07:19:42.368203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365431+0000","last_change":"2024-09-19T07:19:40.388972+0000","last_active":"2024-09-19T07:19:42.365431+0000","last_peered":"2024-09-19T07:19:42.365431+0000","last_clean":"2024-09-19T07:19:42.365431+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:19:42.365431+0000","last_undegraded":"2024-09-19T07:19:42.365431+0000","last_fullsized":"2024-09-19T07:19:42.365431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371005+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:19:42.371005+0000","last_peered":"2024-09-19T07:19:42.371005+0000","last_clean":"2024-09-19T07:19:42.371005+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:19:42.371005+0000","last_undegraded":"2024-09-19T07:19:42.371005+0000","last_fullsized":"2024-09-19T07:19:42.371005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:23.791880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.357019+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:19:41.357019+0000","last_peered":"2024-09-19T07:19:41.357019+0000","last_clean":"2024-09-19T07:19:41.357019+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:19:41.357019+0000","last_undegraded":"2024-09-19T07:19:41.357019+0000","last_fullsized":"2024-09-19T07:19:41.357019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366707+0000","last_change":"2024-09-19T07:19:39.412192+0000","last_active":"2024-09-19T07:19:42.366707+0000","last_peered":"2024-09-19T07:19:42.366707+0000","last_clean":"2024-09-19T07:19:42.366707+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:19:42.366707+0000","last_undegraded":"2024-09-19T07:19:42.366707+0000","last_fullsized":"2024-09-19T07:19:42.366707+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365056+0000","last_change":"2024-09-19T07:19:40.390217+0000","last_active":"2024-09-19T07:19:42.365056+0000","last_peered":"2024-09-19T07:19:42.365056+0000","last_clean":"2024-09-19T07:19:42.365056+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:19:42.365056+0000","last_undegraded":"2024-09-19T07:19:42.365056+0000","last_fullsized":"2024-09-19T07:19:42.365056+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371269+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:19:42.371269+0000","last_peered":"2024-09-19T07:19:42.371269+0000","last_clean":"2024-09-19T07:19:42.371269+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:19:42.371269+0000","last_undegraded":"2024-09-19T07:19:42.371269+0000","last_fullsized":"2024-09-19T07:19:42.371269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368999+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:19:42.368999+0000","last_peered":"2024-09-19T07:19:42.368999+0000","last_clean":"2024-09-19T07:19:42.368999+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:19:42.368999+0000","last_undegraded":"2024-09-19T07:19:42.368999+0000","last_fullsized":"2024-09-19T07:19:42.368999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:50.700962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365712+0000","last_change":"2024-09-19T07:19:39.434150+0000","last_active":"2024-09-19T07:19:42.365712+0000","last_peered":"2024-09-19T07:19:42.365712+0000","last_clean":"2024-09-19T07:19:42.365712+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:19:42.365712+0000","last_undegraded":"2024-09-19T07:19:42.365712+0000","last_fullsized":"2024-09-19T07:19:42.365712+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:13.654208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352846+0000","last_change":"2024-09-19T07:19:40.380140+0000","last_active":"2024-09-19T07:19:41.352846+0000","last_peered":"2024-09-19T07:19:41.352846+0000","last_clean":"2024-09-19T07:19:41.352846+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:19:41.352846+0000","last_undegraded":"2024-09-19T07:19:41.352846+0000","last_fullsized":"2024-09-19T07:19:41.352846+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368156+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:19:42.368156+0000","last_peered":"2024-09-19T07:19:42.368156+0000","last_clean":"2024-09-19T07:19:42.368156+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:19:42.368156+0000","last_undegraded":"2024-09-19T07:19:42.368156+0000","last_fullsized":"2024-09-19T07:19:42.368156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367593+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:19:42.367593+0000","last_peered":"2024-09-19T07:19:42.367593+0000","last_clean":"2024-09-19T07:19:42.367593+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:19:42.367593+0000","last_undegraded":"2024-09-19T07:19:42.367593+0000","last_fullsized":"2024-09-19T07:19:42.367593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366996+0000","last_change":"2024-09-19T07:19:39.411605+0000","last_active":"2024-09-19T07:19:42.366996+0000","last_peered":"2024-09-19T07:19:42.366996+0000","last_clean":"2024-09-19T07:19:42.366996+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:19:42.366996+0000","last_undegraded":"2024-09-19T07:19:42.366996+0000","last_fullsized":"2024-09-19T07:19:42.366996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:55.654483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367982+0000","last_change":"2024-09-19T07:19:40.372083+0000","last_active":"2024-09-19T07:19:42.367982+0000","last_peered":"2024-09-19T07:19:42.367982+0000","last_clean":"2024-09-19T07:19:42.367982+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:19:42.367982+0000","last_undegraded":"2024-09-19T07:19:42.367982+0000","last_fullsized":"2024-09-19T07:19:42.367982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:39.665475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355342+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:19:41.355342+0000","last_peered":"2024-09-19T07:19:41.355342+0000","last_clean":"2024-09-19T07:19:41.355342+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:19:41.355342+0000","last_undegraded":"2024-09-19T07:19:41.355342+0000","last_fullsized":"2024-09-19T07:19:41.355342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368442+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:19:42.368442+0000","last_peered":"2024-09-19T07:19:42.368442+0000","last_clean":"2024-09-19T07:19:42.368442+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:19:42.368442+0000","last_undegraded":"2024-09-19T07:19:42.368442+0000","last_fullsized":"2024-09-19T07:19:42.368442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367339+0000","last_change":"2024-09-19T07:19:39.420340+0000","last_active":"2024-09-19T07:19:42.367339+0000","last_peered":"2024-09-19T07:19:42.367339+0000","last_clean":"2024-09-19T07:19:42.367339+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:19:42.367339+0000","last_undegraded":"2024-09-19T07:19:42.367339+0000","last_fullsized":"2024-09-19T07:19:42.367339+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:29.338848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369245+0000","last_change":"2024-09-19T07:19:40.383082+0000","last_active":"2024-09-19T07:19:42.369245+0000","last_peered":"2024-09-19T07:19:42.369245+0000","last_clean":"2024-09-19T07:19:42.369245+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:19:42.369245+0000","last_undegraded":"2024-09-19T07:19:42.369245+0000","last_fullsized":"2024-09-19T07:19:42.369245+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:54.436493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367222+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:19:42.367222+0000","last_peered":"2024-09-19T07:19:42.367222+0000","last_clean":"2024-09-19T07:19:42.367222+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:19:42.367222+0000","last_undegraded":"2024-09-19T07:19:42.367222+0000","last_fullsized":"2024-09-19T07:19:42.367222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354505+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:19:41.354505+0000","last_peered":"2024-09-19T07:19:41.354505+0000","last_clean":"2024-09-19T07:19:41.354505+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:19:41.354505+0000","last_undegraded":"2024-09-19T07:19:41.354505+0000","last_fullsized":"2024-09-19T07:19:41.354505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368035+0000","last_change":"2024-09-19T07:19:39.407135+0000","last_active":"2024-09-19T07:19:42.368035+0000","last_peered":"2024-09-19T07:19:42.368035+0000","last_clean":"2024-09-19T07:19:42.368035+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:19:42.368035+0000","last_undegraded":"2024-09-19T07:19:42.368035+0000","last_fullsized":"2024-09-19T07:19:42.368035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:27.890595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349118+0000","last_change":"2024-09-19T07:19:40.368866+0000","last_active":"2024-09-19T07:19:41.349118+0000","last_peered":"2024-09-19T07:19:41.349118+0000","last_clean":"2024-09-19T07:19:41.349118+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:19:41.349118+0000","last_undegraded":"2024-09-19T07:19:41.349118+0000","last_fullsized":"2024-09-19T07:19:41.349118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:28.893439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366065+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:19:42.366065+0000","last_peered":"2024-09-19T07:19:42.366065+0000","last_clean":"2024-09-19T07:19:42.366065+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:19:42.366065+0000","last_undegraded":"2024-09-19T07:19:42.366065+0000","last_fullsized":"2024-09-19T07:19:42.366065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348283+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:19:41.348283+0000","last_peered":"2024-09-19T07:19:41.348283+0000","last_clean":"2024-09-19T07:19:41.348283+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:19:41.348283+0000","last_undegraded":"2024-09-19T07:19:41.348283+0000","last_fullsized":"2024-09-19T07:19:41.348283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369806+0000","last_change":"2024-09-19T07:19:39.420497+0000","last_active":"2024-09-19T07:19:42.369806+0000","last_peered":"2024-09-19T07:19:42.369806+0000","last_clean":"2024-09-19T07:19:42.369806+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:19:42.369806+0000","last_undegraded":"2024-09-19T07:19:42.369806+0000","last_fullsized":"2024-09-19T07:19:42.369806+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:19.547270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356452+0000","last_change":"2024-09-19T07:19:40.387573+0000","last_active":"2024-09-19T07:19:41.356452+0000","last_peered":"2024-09-19T07:19:41.356452+0000","last_clean":"2024-09-19T07:19:41.356452+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:19:41.356452+0000","last_undegraded":"2024-09-19T07:19:41.356452+0000","last_fullsized":"2024-09-19T07:19:41.356452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:20:43.234793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365631+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:19:42.365631+0000","last_peered":"2024-09-19T07:19:42.365631+0000","last_clean":"2024-09-19T07:19:42.365631+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:19:42.365631+0000","last_undegraded":"2024-09-19T07:19:42.365631+0000","last_fullsized":"2024-09-19T07:19:42.365631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:20.824916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366356+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:19:42.366356+0000","last_peered":"2024-09-19T07:19:42.366356+0000","last_clean":"2024-09-19T07:19:42.366356+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:19:42.366356+0000","last_undegraded":"2024-09-19T07:19:42.366356+0000","last_fullsized":"2024-09-19T07:19:42.366356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371671+0000","last_change":"2024-09-19T07:19:39.410752+0000","last_active":"2024-09-19T07:19:42.371671+0000","last_peered":"2024-09-19T07:19:42.371671+0000","last_clean":"2024-09-19T07:19:42.371671+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:42.371671+0000","last_undegraded":"2024-09-19T07:19:42.371671+0000","last_fullsized":"2024-09-19T07:19:42.371671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:47:09.376497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367120+0000","last_change":"2024-09-19T07:19:40.392973+0000","last_active":"2024-09-19T07:19:42.367120+0000","last_peered":"2024-09-19T07:19:42.367120+0000","last_clean":"2024-09-19T07:19:42.367120+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:19:42.367120+0000","last_undegraded":"2024-09-19T07:19:42.367120+0000","last_fullsized":"2024-09-19T07:19:42.367120+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368064+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:19:42.368064+0000","last_peered":"2024-09-19T07:19:42.368064+0000","last_clean":"2024-09-19T07:19:42.368064+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:19:42.368064+0000","last_undegraded":"2024-09-19T07:19:42.368064+0000","last_fullsized":"2024-09-19T07:19:42.368064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354067+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:19:41.354067+0000","last_peered":"2024-09-19T07:19:41.354067+0000","last_clean":"2024-09-19T07:19:41.354067+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:19:41.354067+0000","last_undegraded":"2024-09-19T07:19:41.354067+0000","last_fullsized":"2024-09-19T07:19:41.354067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367007+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:19:42.367007+0000","last_peered":"2024-09-19T07:19:42.367007+0000","last_clean":"2024-09-19T07:19:42.367007+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:19:42.367007+0000","last_undegraded":"2024-09-19T07:19:42.367007+0000","last_fullsized":"2024-09-19T07:19:42.367007+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354541+0000","last_change":"2024-09-19T07:19:39.429085+0000","last_active":"2024-09-19T07:19:41.354541+0000","last_peered":"2024-09-19T07:19:41.354541+0000","last_clean":"2024-09-19T07:19:41.354541+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:19:41.354541+0000","last_undegraded":"2024-09-19T07:19:41.354541+0000","last_fullsized":"2024-09-19T07:19:41.354541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348394+0000","last_change":"2024-09-19T07:19:40.382352+0000","last_active":"2024-09-19T07:19:41.348394+0000","last_peered":"2024-09-19T07:19:41.348394+0000","last_clean":"2024-09-19T07:19:41.348394+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:19:41.348394+0000","last_undegraded":"2024-09-19T07:19:41.348394+0000","last_fullsized":"2024-09-19T07:19:41.348394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:31.024388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367107+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:19:42.367107+0000","last_peered":"2024-09-19T07:19:42.367107+0000","last_clean":"2024-09-19T07:19:42.367107+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:19:42.367107+0000","last_undegraded":"2024-09-19T07:19:42.367107+0000","last_fullsized":"2024-09-19T07:19:42.367107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348973+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:19:41.348973+0000","last_peered":"2024-09-19T07:19:41.348973+0000","last_clean":"2024-09-19T07:19:41.348973+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:19:41.348973+0000","last_undegraded":"2024-09-19T07:19:41.348973+0000","last_fullsized":"2024-09-19T07:19:41.348973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353111+0000","last_change":"2024-09-19T07:19:39.427644+0000","last_active":"2024-09-19T07:19:41.353111+0000","last_peered":"2024-09-19T07:19:41.353111+0000","last_clean":"2024-09-19T07:19:41.353111+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:19:41.353111+0000","last_undegraded":"2024-09-19T07:19:41.353111+0000","last_fullsized":"2024-09-19T07:19:41.353111+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348470+0000","last_change":"2024-09-19T07:19:40.390380+0000","last_active":"2024-09-19T07:19:41.348470+0000","last_peered":"2024-09-19T07:19:41.348470+0000","last_clean":"2024-09-19T07:19:41.348470+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:19:41.348470+0000","last_undegraded":"2024-09-19T07:19:41.348470+0000","last_fullsized":"2024-09-19T07:19:41.348470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:26.385490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354180+0000","last_change":"2024-09-19T07:19:32.276600+0000","last_active":"2024-09-19T07:19:41.354180+0000","last_peered":"2024-09-19T07:19:41.354180+0000","last_clean":"2024-09-19T07:19:41.354180+0000","last_became_active":"2024-09-19T07:19:32.275766+0000","last_became_peered":"2024-09-19T07:19:32.275766+0000","last_unstale":"2024-09-19T07:19:41.354180+0000","last_undegraded":"2024-09-19T07:19:41.354180+0000","last_fullsized":"2024-09-19T07:19:41.354180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:48.506399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370499+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:19:42.370499+0000","last_peered":"2024-09-19T07:19:42.370499+0000","last_clean":"2024-09-19T07:19:42.370499+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:19:42.370499+0000","last_undegraded":"2024-09-19T07:19:42.370499+0000","last_fullsized":"2024-09-19T07:19:42.370499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366597+0000","last_change":"2024-09-19T07:19:39.410638+0000","last_active":"2024-09-19T07:19:42.366597+0000","last_peered":"2024-09-19T07:19:42.366597+0000","last_clean":"2024-09-19T07:19:42.366597+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:19:42.366597+0000","last_undegraded":"2024-09-19T07:19:42.366597+0000","last_fullsized":"2024-09-19T07:19:42.366597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:32.073136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365442+0000","last_change":"2024-09-19T07:19:40.378481+0000","last_active":"2024-09-19T07:19:42.365442+0000","last_peered":"2024-09-19T07:19:42.365442+0000","last_clean":"2024-09-19T07:19:42.365442+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:19:42.365442+0000","last_undegraded":"2024-09-19T07:19:42.365442+0000","last_fullsized":"2024-09-19T07:19:42.365442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:30.585686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369508+0000","last_change":"2024-09-19T07:19:32.292026+0000","last_active":"2024-09-19T07:19:42.369508+0000","last_peered":"2024-09-19T07:19:42.369508+0000","last_clean":"2024-09-19T07:19:42.369508+0000","last_became_active":"2024-09-19T07:19:32.291037+0000","last_became_peered":"2024-09-19T07:19:32.291037+0000","last_unstale":"2024-09-19T07:19:42.369508+0000","last_undegraded":"2024-09-19T07:19:42.369508+0000","last_fullsized":"2024-09-19T07:19:42.369508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:22.190763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349468+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:19:41.349468+0000","last_peered":"2024-09-19T07:19:41.349468+0000","last_clean":"2024-09-19T07:19:41.349468+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:19:41.349468+0000","last_undegraded":"2024-09-19T07:19:41.349468+0000","last_fullsized":"2024-09-19T07:19:41.349468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353302+0000","last_change":"2024-09-19T07:19:39.437656+0000","last_active":"2024-09-19T07:19:41.353302+0000","last_peered":"2024-09-19T07:19:41.353302+0000","last_clean":"2024-09-19T07:19:41.353302+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:19:41.353302+0000","last_undegraded":"2024-09-19T07:19:41.353302+0000","last_fullsized":"2024-09-19T07:19:41.353302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:41.468096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.347816+0000","last_change":"2024-09-19T07:19:40.379289+0000","last_active":"2024-09-19T07:19:41.347816+0000","last_peered":"2024-09-19T07:19:41.347816+0000","last_clean":"2024-09-19T07:19:41.347816+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:19:41.347816+0000","last_undegraded":"2024-09-19T07:19:41.347816+0000","last_fullsized":"2024-09-19T07:19:41.347816+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:20.399106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368075+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:19:42.368075+0000","last_peered":"2024-09-19T07:19:42.368075+0000","last_clean":"2024-09-19T07:19:42.368075+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:19:42.368075+0000","last_undegraded":"2024-09-19T07:19:42.368075+0000","last_fullsized":"2024-09-19T07:19:42.368075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368710+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:19:42.368710+0000","last_peered":"2024-09-19T07:19:42.368710+0000","last_clean":"2024-09-19T07:19:42.368710+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:19:42.368710+0000","last_undegraded":"2024-09-19T07:19:42.368710+0000","last_fullsized":"2024-09-19T07:19:42.368710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366459+0000","last_change":"2024-09-19T07:19:39.413589+0000","last_active":"2024-09-19T07:19:42.366459+0000","last_peered":"2024-09-19T07:19:42.366459+0000","last_clean":"2024-09-19T07:19:42.366459+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:19:42.366459+0000","last_undegraded":"2024-09-19T07:19:42.366459+0000","last_fullsized":"2024-09-19T07:19:42.366459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370515+0000","last_change":"2024-09-19T07:19:40.387159+0000","last_active":"2024-09-19T07:19:42.370515+0000","last_peered":"2024-09-19T07:19:42.370515+0000","last_clean":"2024-09-19T07:19:42.370515+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:19:42.370515+0000","last_undegraded":"2024-09-19T07:19:42.370515+0000","last_fullsized":"2024-09-19T07:19:42.370515+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366844+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:19:42.366844+0000","last_peered":"2024-09-19T07:19:42.366844+0000","last_clean":"2024-09-19T07:19:42.366844+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:19:42.366844+0000","last_undegraded":"2024-09-19T07:19:42.366844+0000","last_fullsized":"2024-09-19T07:19:42.366844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.347952+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:19:41.347952+0000","last_peered":"2024-09-19T07:19:41.347952+0000","last_clean":"2024-09-19T07:19:41.347952+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:19:41.347952+0000","last_undegraded":"2024-09-19T07:19:41.347952+0000","last_fullsized":"2024-09-19T07:19:41.347952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371002+0000","last_change":"2024-09-19T07:19:39.406110+0000","last_active":"2024-09-19T07:19:42.371002+0000","last_peered":"2024-09-19T07:19:42.371002+0000","last_clean":"2024-09-19T07:19:42.371002+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:19:42.371002+0000","last_undegraded":"2024-09-19T07:19:42.371002+0000","last_fullsized":"2024-09-19T07:19:42.371002+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:10:05.224225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355606+0000","last_change":"2024-09-19T07:19:40.365764+0000","last_active":"2024-09-19T07:19:41.355606+0000","last_peered":"2024-09-19T07:19:41.355606+0000","last_clean":"2024-09-19T07:19:41.355606+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:19:41.355606+0000","last_undegraded":"2024-09-19T07:19:41.355606+0000","last_fullsized":"2024-09-19T07:19:41.355606+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:04:03.242292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352283+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:19:41.352283+0000","last_peered":"2024-09-19T07:19:41.352283+0000","last_clean":"2024-09-19T07:19:41.352283+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:19:41.352283+0000","last_undegraded":"2024-09-19T07:19:41.352283+0000","last_fullsized":"2024-09-19T07:19:41.352283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353746+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:19:41.353746+0000","last_peered":"2024-09-19T07:19:41.353746+0000","last_clean":"2024-09-19T07:19:41.353746+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:19:41.353746+0000","last_undegraded":"2024-09-19T07:19:41.353746+0000","last_fullsized":"2024-09-19T07:19:41.353746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370756+0000","last_change":"2024-09-19T07:19:39.429768+0000","last_active":"2024-09-19T07:19:42.370756+0000","last_peered":"2024-09-19T07:19:42.370756+0000","last_clean":"2024-09-19T07:19:42.370756+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:19:42.370756+0000","last_undegraded":"2024-09-19T07:19:42.370756+0000","last_fullsized":"2024-09-19T07:19:42.370756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:47.503631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352914+0000","last_change":"2024-09-19T07:19:40.373542+0000","last_active":"2024-09-19T07:19:41.352914+0000","last_peered":"2024-09-19T07:19:41.352914+0000","last_clean":"2024-09-19T07:19:41.352914+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:19:41.352914+0000","last_undegraded":"2024-09-19T07:19:41.352914+0000","last_fullsized":"2024-09-19T07:19:41.352914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:14.326662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366794+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:19:42.366794+0000","last_peered":"2024-09-19T07:19:42.366794+0000","last_clean":"2024-09-19T07:19:42.366794+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:19:42.366794+0000","last_undegraded":"2024-09-19T07:19:42.366794+0000","last_fullsized":"2024-09-19T07:19:42.366794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354258+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:19:41.354258+0000","last_peered":"2024-09-19T07:19:41.354258+0000","last_clean":"2024-09-19T07:19:41.354258+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:19:41.354258+0000","last_undegraded":"2024-09-19T07:19:41.354258+0000","last_fullsized":"2024-09-19T07:19:41.354258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"25'8","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:20:00.218770+0000","last_change":"2024-09-19T07:19:39.411074+0000","last_active":"2024-09-19T07:20:00.218770+0000","last_peered":"2024-09-19T07:20:00.218770+0000","last_clean":"2024-09-19T07:20:00.218770+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:20:00.218770+0000","last_undegraded":"2024-09-19T07:20:00.218770+0000","last_fullsized":"2024-09-19T07:20:00.218770+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:53.992787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2055,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370720+0000","last_change":"2024-09-19T07:19:40.364814+0000","last_active":"2024-09-19T07:19:42.370720+0000","last_peered":"2024-09-19T07:19:42.370720+0000","last_clean":"2024-09-19T07:19:42.370720+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:19:42.370720+0000","last_undegraded":"2024-09-19T07:19:42.370720+0000","last_fullsized":"2024-09-19T07:19:42.370720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366287+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:19:42.366287+0000","last_peered":"2024-09-19T07:19:42.366287+0000","last_clean":"2024-09-19T07:19:42.366287+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:19:42.366287+0000","last_undegraded":"2024-09-19T07:19:42.366287+0000","last_fullsized":"2024-09-19T07:19:42.366287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:57.131912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368746+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:19:42.368746+0000","last_peered":"2024-09-19T07:19:42.368746+0000","last_clean":"2024-09-19T07:19:42.368746+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:19:42.368746+0000","last_undegraded":"2024-09-19T07:19:42.368746+0000","last_fullsized":"2024-09-19T07:19:42.368746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:23.367365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"24'3","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355049+0000","last_change":"2024-09-19T07:19:39.431616+0000","last_active":"2024-09-19T07:19:41.355049+0000","last_peered":"2024-09-19T07:19:41.355049+0000","last_clean":"2024-09-19T07:19:41.355049+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:19:41.355049+0000","last_undegraded":"2024-09-19T07:19:41.355049+0000","last_fullsized":"2024-09-19T07:19:41.355049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:18.149581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370545+0000","last_change":"2024-09-19T07:19:40.381443+0000","last_active":"2024-09-19T07:19:42.370545+0000","last_peered":"2024-09-19T07:19:42.370545+0000","last_clean":"2024-09-19T07:19:42.370545+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:19:42.370545+0000","last_undegraded":"2024-09-19T07:19:42.370545+0000","last_fullsized":"2024-09-19T07:19:42.370545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366843+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:19:42.366843+0000","last_peered":"2024-09-19T07:19:42.366843+0000","last_clean":"2024-09-19T07:19:42.366843+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:19:42.366843+0000","last_undegraded":"2024-09-19T07:19:42.366843+0000","last_fullsized":"2024-09-19T07:19:42.366843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:57.218063+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:19:57.218063+0000","last_peered":"2024-09-19T07:19:57.218063+0000","last_clean":"2024-09-19T07:19:57.218063+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:19:57.218063+0000","last_undegraded":"2024-09-19T07:19:57.218063+0000","last_fullsized":"2024-09-19T07:19:57.218063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2055,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367083+0000","last_change":"2024-09-19T07:19:39.423972+0000","last_active":"2024-09-19T07:19:42.367083+0000","last_peered":"2024-09-19T07:19:42.367083+0000","last_clean":"2024-09-19T07:19:42.367083+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:19:42.367083+0000","last_undegraded":"2024-09-19T07:19:42.367083+0000","last_fullsized":"2024-09-19T07:19:42.367083+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:45.387040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367684+0000","last_change":"2024-09-19T07:19:40.382092+0000","last_active":"2024-09-19T07:19:42.367684+0000","last_peered":"2024-09-19T07:19:42.367684+0000","last_clean":"2024-09-19T07:19:42.367684+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:19:42.367684+0000","last_undegraded":"2024-09-19T07:19:42.367684+0000","last_fullsized":"2024-09-19T07:19:42.367684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368384+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:19:42.368384+0000","last_peered":"2024-09-19T07:19:42.368384+0000","last_clean":"2024-09-19T07:19:42.368384+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:19:42.368384+0000","last_undegraded":"2024-09-19T07:19:42.368384+0000","last_fullsized":"2024-09-19T07:19:42.368384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370238+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:19:42.370238+0000","last_peered":"2024-09-19T07:19:42.370238+0000","last_clean":"2024-09-19T07:19:42.370238+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:19:42.370238+0000","last_undegraded":"2024-09-19T07:19:42.370238+0000","last_fullsized":"2024-09-19T07:19:42.370238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366357+0000","last_change":"2024-09-19T07:19:39.424879+0000","last_active":"2024-09-19T07:19:42.366357+0000","last_peered":"2024-09-19T07:19:42.366357+0000","last_clean":"2024-09-19T07:19:42.366357+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:19:42.366357+0000","last_undegraded":"2024-09-19T07:19:42.366357+0000","last_fullsized":"2024-09-19T07:19:42.366357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:30.021543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353343+0000","last_change":"2024-09-19T07:19:40.384136+0000","last_active":"2024-09-19T07:19:41.353343+0000","last_peered":"2024-09-19T07:19:41.353343+0000","last_clean":"2024-09-19T07:19:41.353343+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:19:41.353343+0000","last_undegraded":"2024-09-19T07:19:41.353343+0000","last_fullsized":"2024-09-19T07:19:41.353343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369727+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:19:42.369727+0000","last_peered":"2024-09-19T07:19:42.369727+0000","last_clean":"2024-09-19T07:19:42.369727+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:19:42.369727+0000","last_undegraded":"2024-09-19T07:19:42.369727+0000","last_fullsized":"2024-09-19T07:19:42.369727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:48.308137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355901+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:19:41.355901+0000","last_peered":"2024-09-19T07:19:41.355901+0000","last_clean":"2024-09-19T07:19:41.355901+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:19:41.355901+0000","last_undegraded":"2024-09-19T07:19:41.355901+0000","last_fullsized":"2024-09-19T07:19:41.355901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"24'8","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365988+0000","last_change":"2024-09-19T07:19:39.410445+0000","last_active":"2024-09-19T07:19:42.365988+0000","last_peered":"2024-09-19T07:19:42.365988+0000","last_clean":"2024-09-19T07:19:42.365988+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:19:42.365988+0000","last_undegraded":"2024-09-19T07:19:42.365988+0000","last_fullsized":"2024-09-19T07:19:42.365988+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:27.020327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365513+0000","last_change":"2024-09-19T07:19:40.389499+0000","last_active":"2024-09-19T07:19:42.365513+0000","last_peered":"2024-09-19T07:19:42.365513+0000","last_clean":"2024-09-19T07:19:42.365513+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:19:42.365513+0000","last_undegraded":"2024-09-19T07:19:42.365513+0000","last_fullsized":"2024-09-19T07:19:42.365513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.520872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355599+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:19:41.355599+0000","last_peered":"2024-09-19T07:19:41.355599+0000","last_clean":"2024-09-19T07:19:41.355599+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:19:41.355599+0000","last_undegraded":"2024-09-19T07:19:41.355599+0000","last_fullsized":"2024-09-19T07:19:41.355599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.357039+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:19:41.357039+0000","last_peered":"2024-09-19T07:19:41.357039+0000","last_clean":"2024-09-19T07:19:41.357039+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:19:41.357039+0000","last_undegraded":"2024-09-19T07:19:41.357039+0000","last_fullsized":"2024-09-19T07:19:41.357039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368271+0000","last_change":"2024-09-19T07:19:39.416943+0000","last_active":"2024-09-19T07:19:42.368271+0000","last_peered":"2024-09-19T07:19:42.368271+0000","last_clean":"2024-09-19T07:19:42.368271+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:19:42.368271+0000","last_undegraded":"2024-09-19T07:19:42.368271+0000","last_fullsized":"2024-09-19T07:19:42.368271+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:35.555826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356615+0000","last_change":"2024-09-19T07:19:40.394993+0000","last_active":"2024-09-19T07:19:41.356615+0000","last_peered":"2024-09-19T07:19:41.356615+0000","last_clean":"2024-09-19T07:19:41.356615+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:19:41.356615+0000","last_undegraded":"2024-09-19T07:19:41.356615+0000","last_fullsized":"2024-09-19T07:19:41.356615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370672+0000","last_change":"2024-09-19T07:19:39.418885+0000","last_active":"2024-09-19T07:19:42.370672+0000","last_peered":"2024-09-19T07:19:42.370672+0000","last_clean":"2024-09-19T07:19:42.370672+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:19:42.370672+0000","last_undegraded":"2024-09-19T07:19:42.370672+0000","last_fullsized":"2024-09-19T07:19:42.370672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:35:45.790411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366817+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:19:42.366817+0000","last_peered":"2024-09-19T07:19:42.366817+0000","last_clean":"2024-09-19T07:19:42.366817+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:19:42.366817+0000","last_undegraded":"2024-09-19T07:19:42.366817+0000","last_fullsized":"2024-09-19T07:19:42.366817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353835+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:19:41.353835+0000","last_peered":"2024-09-19T07:19:41.353835+0000","last_clean":"2024-09-19T07:19:41.353835+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:19:41.353835+0000","last_undegraded":"2024-09-19T07:19:41.353835+0000","last_fullsized":"2024-09-19T07:19:41.353835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370413+0000","last_change":"2024-09-19T07:19:40.374048+0000","last_active":"2024-09-19T07:19:42.370413+0000","last_peered":"2024-09-19T07:19:42.370413+0000","last_clean":"2024-09-19T07:19:42.370413+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:19:42.370413+0000","last_undegraded":"2024-09-19T07:19:42.370413+0000","last_fullsized":"2024-09-19T07:19:42.370413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:40.084827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"24'4","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356842+0000","last_change":"2024-09-19T07:19:39.432426+0000","last_active":"2024-09-19T07:19:41.356842+0000","last_peered":"2024-09-19T07:19:41.356842+0000","last_clean":"2024-09-19T07:19:41.356842+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:19:41.356842+0000","last_undegraded":"2024-09-19T07:19:41.356842+0000","last_fullsized":"2024-09-19T07:19:41.356842+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366831+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:19:42.366831+0000","last_peered":"2024-09-19T07:19:42.366831+0000","last_clean":"2024-09-19T07:19:42.366831+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:19:42.366831+0000","last_undegraded":"2024-09-19T07:19:42.366831+0000","last_fullsized":"2024-09-19T07:19:42.366831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368900+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:19:42.368900+0000","last_peered":"2024-09-19T07:19:42.368900+0000","last_clean":"2024-09-19T07:19:42.368900+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:19:42.368900+0000","last_undegraded":"2024-09-19T07:19:42.368900+0000","last_fullsized":"2024-09-19T07:19:42.368900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369632+0000","last_change":"2024-09-19T07:19:40.384934+0000","last_active":"2024-09-19T07:19:42.369632+0000","last_peered":"2024-09-19T07:19:42.369632+0000","last_clean":"2024-09-19T07:19:42.369632+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:19:42.369632+0000","last_undegraded":"2024-09-19T07:19:42.369632+0000","last_fullsized":"2024-09-19T07:19:42.369632+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353660+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:19:41.353660+0000","last_peered":"2024-09-19T07:19:41.353660+0000","last_clean":"2024-09-19T07:19:41.353660+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:19:41.353660+0000","last_undegraded":"2024-09-19T07:19:41.353660+0000","last_fullsized":"2024-09-19T07:19:41.353660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368442+0000","last_change":"2024-09-19T07:19:32.289374+0000","last_active":"2024-09-19T07:19:42.368442+0000","last_peered":"2024-09-19T07:19:42.368442+0000","last_clean":"2024-09-19T07:19:42.368442+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:19:42.368442+0000","last_undegraded":"2024-09-19T07:19:42.368442+0000","last_fullsized":"2024-09-19T07:19:42.368442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348715+0000","last_change":"2024-09-19T07:19:39.410582+0000","last_active":"2024-09-19T07:19:41.348715+0000","last_peered":"2024-09-19T07:19:41.348715+0000","last_clean":"2024-09-19T07:19:41.348715+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:19:41.348715+0000","last_undegraded":"2024-09-19T07:19:41.348715+0000","last_fullsized":"2024-09-19T07:19:41.348715+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:41:40.592907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367452+0000","last_change":"2024-09-19T07:19:40.393950+0000","last_active":"2024-09-19T07:19:42.367452+0000","last_peered":"2024-09-19T07:19:42.367452+0000","last_clean":"2024-09-19T07:19:42.367452+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:19:42.367452+0000","last_undegraded":"2024-09-19T07:19:42.367452+0000","last_fullsized":"2024-09-19T07:19:42.367452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371158+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:19:42.371158+0000","last_peered":"2024-09-19T07:19:42.371158+0000","last_clean":"2024-09-19T07:19:42.371158+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:19:42.371158+0000","last_undegraded":"2024-09-19T07:19:42.371158+0000","last_fullsized":"2024-09-19T07:19:42.371158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366780+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:19:42.366780+0000","last_peered":"2024-09-19T07:19:42.366780+0000","last_clean":"2024-09-19T07:19:42.366780+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:19:42.366780+0000","last_undegraded":"2024-09-19T07:19:42.366780+0000","last_fullsized":"2024-09-19T07:19:42.366780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371594+0000","last_change":"2024-09-19T07:19:39.410662+0000","last_active":"2024-09-19T07:19:42.371594+0000","last_peered":"2024-09-19T07:19:42.371594+0000","last_clean":"2024-09-19T07:19:42.371594+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:42.371594+0000","last_undegraded":"2024-09-19T07:19:42.371594+0000","last_fullsized":"2024-09-19T07:19:42.371594+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:19.196074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372306+0000","last_change":"2024-09-19T07:19:40.385581+0000","last_active":"2024-09-19T07:19:42.372306+0000","last_peered":"2024-09-19T07:19:42.372306+0000","last_clean":"2024-09-19T07:19:42.372306+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:19:42.372306+0000","last_undegraded":"2024-09-19T07:19:42.372306+0000","last_fullsized":"2024-09-19T07:19:42.372306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:14:13.749729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353359+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:19:41.353359+0000","last_peered":"2024-09-19T07:19:41.353359+0000","last_clean":"2024-09-19T07:19:41.353359+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:19:41.353359+0000","last_undegraded":"2024-09-19T07:19:41.353359+0000","last_fullsized":"2024-09-19T07:19:41.353359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369574+0000","last_change":"2024-09-19T07:19:40.377608+0000","last_active":"2024-09-19T07:19:42.369574+0000","last_peered":"2024-09-19T07:19:42.369574+0000","last_clean":"2024-09-19T07:19:42.369574+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:19:42.369574+0000","last_undegraded":"2024-09-19T07:19:42.369574+0000","last_fullsized":"2024-09-19T07:19:42.369574+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348053+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:19:41.348053+0000","last_peered":"2024-09-19T07:19:41.348053+0000","last_clean":"2024-09-19T07:19:41.348053+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:19:41.348053+0000","last_undegraded":"2024-09-19T07:19:41.348053+0000","last_fullsized":"2024-09-19T07:19:41.348053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353277+0000","last_change":"2024-09-19T07:19:40.381262+0000","last_active":"2024-09-19T07:19:41.353277+0000","last_peered":"2024-09-19T07:19:41.353277+0000","last_clean":"2024-09-19T07:19:41.353277+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:19:41.353277+0000","last_undegraded":"2024-09-19T07:19:41.353277+0000","last_fullsized":"2024-09-19T07:19:41.353277+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:34.425472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354018+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:19:41.354018+0000","last_peered":"2024-09-19T07:19:41.354018+0000","last_clean":"2024-09-19T07:19:41.354018+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:19:41.354018+0000","last_undegraded":"2024-09-19T07:19:41.354018+0000","last_fullsized":"2024-09-19T07:19:41.354018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369264+0000","last_change":"2024-09-19T07:19:40.386432+0000","last_active":"2024-09-19T07:19:42.369264+0000","last_peered":"2024-09-19T07:19:42.369264+0000","last_clean":"2024-09-19T07:19:42.369264+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:19:42.369264+0000","last_undegraded":"2024-09-19T07:19:42.369264+0000","last_fullsized":"2024-09-19T07:19:42.369264+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368214+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:19:42.368214+0000","last_peered":"2024-09-19T07:19:42.368214+0000","last_clean":"2024-09-19T07:19:42.368214+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:19:42.368214+0000","last_undegraded":"2024-09-19T07:19:42.368214+0000","last_fullsized":"2024-09-19T07:19:42.368214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365212+0000","last_change":"2024-09-19T07:19:40.378630+0000","last_active":"2024-09-19T07:19:42.365212+0000","last_peered":"2024-09-19T07:19:42.365212+0000","last_clean":"2024-09-19T07:19:42.365212+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:19:42.365212+0000","last_undegraded":"2024-09-19T07:19:42.365212+0000","last_fullsized":"2024-09-19T07:19:42.365212+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371622+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:19:42.371622+0000","last_peered":"2024-09-19T07:19:42.371622+0000","last_clean":"2024-09-19T07:19:42.371622+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:19:42.371622+0000","last_undegraded":"2024-09-19T07:19:42.371622+0000","last_fullsized":"2024-09-19T07:19:42.371622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365853+0000","last_change":"2024-09-19T07:19:40.381365+0000","last_active":"2024-09-19T07:19:42.365853+0000","last_peered":"2024-09-19T07:19:42.365853+0000","last_clean":"2024-09-19T07:19:42.365853+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:19:42.365853+0000","last_undegraded":"2024-09-19T07:19:42.365853+0000","last_fullsized":"2024-09-19T07:19:42.365853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370004+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:19:42.370004+0000","last_peered":"2024-09-19T07:19:42.370004+0000","last_clean":"2024-09-19T07:19:42.370004+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:19:42.370004+0000","last_undegraded":"2024-09-19T07:19:42.370004+0000","last_fullsized":"2024-09-19T07:19:42.370004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365148+0000","last_change":"2024-09-19T07:19:40.387116+0000","last_active":"2024-09-19T07:19:42.365148+0000","last_peered":"2024-09-19T07:19:42.365148+0000","last_clean":"2024-09-19T07:19:42.365148+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:19:42.365148+0000","last_undegraded":"2024-09-19T07:19:42.365148+0000","last_fullsized":"2024-09-19T07:19:42.365148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356605+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:19:41.356605+0000","last_peered":"2024-09-19T07:19:41.356605+0000","last_clean":"2024-09-19T07:19:41.356605+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:19:41.356605+0000","last_undegraded":"2024-09-19T07:19:41.356605+0000","last_fullsized":"2024-09-19T07:19:41.356605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352940+0000","last_change":"2024-09-19T07:19:40.357629+0000","last_active":"2024-09-19T07:19:41.352940+0000","last_peered":"2024-09-19T07:19:41.352940+0000","last_clean":"2024-09-19T07:19:41.352940+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:19:41.352940+0000","last_undegraded":"2024-09-19T07:19:41.352940+0000","last_fullsized":"2024-09-19T07:19:41.352940+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371018+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:19:42.371018+0000","last_peered":"2024-09-19T07:19:42.371018+0000","last_clean":"2024-09-19T07:19:42.371018+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:19:42.371018+0000","last_undegraded":"2024-09-19T07:19:42.371018+0000","last_fullsized":"2024-09-19T07:19:42.371018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371827+0000","last_change":"2024-09-19T07:19:40.388597+0000","last_active":"2024-09-19T07:19:42.371827+0000","last_peered":"2024-09-19T07:19:42.371827+0000","last_clean":"2024-09-19T07:19:42.371827+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:19:42.371827+0000","last_undegraded":"2024-09-19T07:19:42.371827+0000","last_fullsized":"2024-09-19T07:19:42.371827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:34:18.402016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356021+0000","last_change":"2024-09-19T07:19:32.273635+0000","last_active":"2024-09-19T07:19:41.356021+0000","last_peered":"2024-09-19T07:19:41.356021+0000","last_clean":"2024-09-19T07:19:41.356021+0000","last_became_active":"2024-09-19T07:19:32.271748+0000","last_became_peered":"2024-09-19T07:19:32.271748+0000","last_unstale":"2024-09-19T07:19:41.356021+0000","last_undegraded":"2024-09-19T07:19:41.356021+0000","last_fullsized":"2024-09-19T07:19:41.356021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:49:41.724626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369047+0000","last_change":"2024-09-19T07:19:40.383779+0000","last_active":"2024-09-19T07:19:42.369047+0000","last_peered":"2024-09-19T07:19:42.369047+0000","last_clean":"2024-09-19T07:19:42.369047+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:19:42.369047+0000","last_undegraded":"2024-09-19T07:19:42.369047+0000","last_fullsized":"2024-09-19T07:19:42.369047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368684+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:19:42.368684+0000","last_peered":"2024-09-19T07:19:42.368684+0000","last_clean":"2024-09-19T07:19:42.368684+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:19:42.368684+0000","last_undegraded":"2024-09-19T07:19:42.368684+0000","last_fullsized":"2024-09-19T07:19:42.368684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348737+0000","last_change":"2024-09-19T07:19:40.376342+0000","last_active":"2024-09-19T07:19:41.348737+0000","last_peered":"2024-09-19T07:19:41.348737+0000","last_clean":"2024-09-19T07:19:41.348737+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:19:41.348737+0000","last_undegraded":"2024-09-19T07:19:41.348737+0000","last_fullsized":"2024-09-19T07:19:41.348737+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:22.997744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353771+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:19:41.353771+0000","last_peered":"2024-09-19T07:19:41.353771+0000","last_clean":"2024-09-19T07:19:41.353771+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:19:41.353771+0000","last_undegraded":"2024-09-19T07:19:41.353771+0000","last_fullsized":"2024-09-19T07:19:41.353771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371199+0000","last_change":"2024-09-19T07:19:40.361504+0000","last_active":"2024-09-19T07:19:42.371199+0000","last_peered":"2024-09-19T07:19:42.371199+0000","last_clean":"2024-09-19T07:19:42.371199+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:19:42.371199+0000","last_undegraded":"2024-09-19T07:19:42.371199+0000","last_fullsized":"2024-09-19T07:19:42.371199+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:30:31.110669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367582+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:19:42.367582+0000","last_peered":"2024-09-19T07:19:42.367582+0000","last_clean":"2024-09-19T07:19:42.367582+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:19:42.367582+0000","last_undegraded":"2024-09-19T07:19:42.367582+0000","last_fullsized":"2024-09-19T07:19:42.367582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:19.033192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365909+0000","last_change":"2024-09-19T07:19:40.390205+0000","last_active":"2024-09-19T07:19:42.365909+0000","last_peered":"2024-09-19T07:19:42.365909+0000","last_clean":"2024-09-19T07:19:42.365909+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:19:42.365909+0000","last_undegraded":"2024-09-19T07:19:42.365909+0000","last_fullsized":"2024-09-19T07:19:42.365909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354180+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:19:41.354180+0000","last_peered":"2024-09-19T07:19:41.354180+0000","last_clean":"2024-09-19T07:19:41.354180+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:19:41.354180+0000","last_undegraded":"2024-09-19T07:19:41.354180+0000","last_fullsized":"2024-09-19T07:19:41.354180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365447+0000","last_change":"2024-09-19T07:19:40.390559+0000","last_active":"2024-09-19T07:19:42.365447+0000","last_peered":"2024-09-19T07:19:42.365447+0000","last_clean":"2024-09-19T07:19:42.365447+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:19:42.365447+0000","last_undegraded":"2024-09-19T07:19:42.365447+0000","last_fullsized":"2024-09-19T07:19:42.365447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367349+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:19:42.367349+0000","last_peered":"2024-09-19T07:19:42.367349+0000","last_clean":"2024-09-19T07:19:42.367349+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:19:42.367349+0000","last_undegraded":"2024-09-19T07:19:42.367349+0000","last_fullsized":"2024-09-19T07:19:42.367349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352820+0000","last_change":"2024-09-19T07:19:40.358433+0000","last_active":"2024-09-19T07:19:41.352820+0000","last_peered":"2024-09-19T07:19:41.352820+0000","last_clean":"2024-09-19T07:19:41.352820+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:19:41.352820+0000","last_undegraded":"2024-09-19T07:19:41.352820+0000","last_fullsized":"2024-09-19T07:19:41.352820+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:10:25.226677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349337+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:19:41.349337+0000","last_peered":"2024-09-19T07:19:41.349337+0000","last_clean":"2024-09-19T07:19:41.349337+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:19:41.349337+0000","last_undegraded":"2024-09-19T07:19:41.349337+0000","last_fullsized":"2024-09-19T07:19:41.349337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:51.710997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365454+0000","last_change":"2024-09-19T07:19:40.374374+0000","last_active":"2024-09-19T07:19:42.365454+0000","last_peered":"2024-09-19T07:19:42.365454+0000","last_clean":"2024-09-19T07:19:42.365454+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:19:42.365454+0000","last_undegraded":"2024-09-19T07:19:42.365454+0000","last_fullsized":"2024-09-19T07:19:42.365454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:34:05.875388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356952+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:19:41.356952+0000","last_peered":"2024-09-19T07:19:41.356952+0000","last_clean":"2024-09-19T07:19:41.356952+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:19:41.356952+0000","last_undegraded":"2024-09-19T07:19:41.356952+0000","last_fullsized":"2024-09-19T07:19:41.356952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368071+0000","last_change":"2024-09-19T07:19:40.387341+0000","last_active":"2024-09-19T07:19:42.368071+0000","last_peered":"2024-09-19T07:19:42.368071+0000","last_clean":"2024-09-19T07:19:42.368071+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:19:42.368071+0000","last_undegraded":"2024-09-19T07:19:42.368071+0000","last_fullsized":"2024-09-19T07:19:42.368071+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:56:40.928844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353563+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:19:41.353563+0000","last_peered":"2024-09-19T07:19:41.353563+0000","last_clean":"2024-09-19T07:19:41.353563+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:19:41.353563+0000","last_undegraded":"2024-09-19T07:19:41.353563+0000","last_fullsized":"2024-09-19T07:19:41.353563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368148+0000","last_change":"2024-09-19T07:19:40.388484+0000","last_active":"2024-09-19T07:19:42.368148+0000","last_peered":"2024-09-19T07:19:42.368148+0000","last_clean":"2024-09-19T07:19:42.368148+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:19:42.368148+0000","last_undegraded":"2024-09-19T07:19:42.368148+0000","last_fullsized":"2024-09-19T07:19:42.368148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366974+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:19:42.366974+0000","last_peered":"2024-09-19T07:19:42.366974+0000","last_clean":"2024-09-19T07:19:42.366974+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:19:42.366974+0000","last_undegraded":"2024-09-19T07:19:42.366974+0000","last_fullsized":"2024-09-19T07:19:42.366974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352836+0000","last_change":"2024-09-19T07:19:40.367578+0000","last_active":"2024-09-19T07:19:41.352836+0000","last_peered":"2024-09-19T07:19:41.352836+0000","last_clean":"2024-09-19T07:19:41.352836+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:19:41.352836+0000","last_undegraded":"2024-09-19T07:19:41.352836+0000","last_fullsized":"2024-09-19T07:19:41.352836+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:00:25.357854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366048+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:19:42.366048+0000","last_peered":"2024-09-19T07:19:42.366048+0000","last_clean":"2024-09-19T07:19:42.366048+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:19:42.366048+0000","last_undegraded":"2024-09-19T07:19:42.366048+0000","last_fullsized":"2024-09-19T07:19:42.366048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:43.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366598+0000","last_change":"2024-09-19T07:19:40.385641+0000","last_active":"2024-09-19T07:19:42.366598+0000","last_peered":"2024-09-19T07:19:42.366598+0000","last_clean":"2024-09-19T07:19:42.366598+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:19:42.366598+0000","last_undegraded":"2024-09-19T07:19:42.366598+0000","last_fullsized":"2024-09-19T07:19:42.366598+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370694+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:19:42.370694+0000","last_peered":"2024-09-19T07:19:42.370694+0000","last_clean":"2024-09-19T07:19:42.370694+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:19:42.370694+0000","last_undegraded":"2024-09-19T07:19:42.370694+0000","last_fullsized":"2024-09-19T07:19:42.370694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369737+0000","last_change":"2024-09-19T07:19:40.367101+0000","last_active":"2024-09-19T07:19:42.369737+0000","last_peered":"2024-09-19T07:19:42.369737+0000","last_clean":"2024-09-19T07:19:42.369737+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:19:42.369737+0000","last_undegraded":"2024-09-19T07:19:42.369737+0000","last_fullsized":"2024-09-19T07:19:42.369737+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368379+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:19:42.368379+0000","last_peered":"2024-09-19T07:19:42.368379+0000","last_clean":"2024-09-19T07:19:42.368379+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:19:42.368379+0000","last_undegraded":"2024-09-19T07:19:42.368379+0000","last_fullsized":"2024-09-19T07:19:42.368379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348413+0000","last_change":"2024-09-19T07:19:40.384212+0000","last_active":"2024-09-19T07:19:41.348413+0000","last_peered":"2024-09-19T07:19:41.348413+0000","last_clean":"2024-09-19T07:19:41.348413+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:19:41.348413+0000","last_undegraded":"2024-09-19T07:19:41.348413+0000","last_fullsized":"2024-09-19T07:19:41.348413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:57.724404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355987+0000","last_change":"2024-09-19T07:19:32.299956+0000","last_active":"2024-09-19T07:19:41.355987+0000","last_peered":"2024-09-19T07:19:41.355987+0000","last_clean":"2024-09-19T07:19:41.355987+0000","last_became_active":"2024-09-19T07:19:32.299168+0000","last_became_peered":"2024-09-19T07:19:32.299168+0000","last_unstale":"2024-09-19T07:19:41.355987+0000","last_undegraded":"2024-09-19T07:19:41.355987+0000","last_fullsized":"2024-09-19T07:19:41.355987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366761+0000","last_change":"2024-09-19T07:19:40.387401+0000","last_active":"2024-09-19T07:19:42.366761+0000","last_peered":"2024-09-19T07:19:42.366761+0000","last_clean":"2024-09-19T07:19:42.366761+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:19:42.366761+0000","last_undegraded":"2024-09-19T07:19:42.366761+0000","last_fullsized":"2024-09-19T07:19:42.366761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356423+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:19:41.356423+0000","last_peered":"2024-09-19T07:19:41.356423+0000","last_clean":"2024-09-19T07:19:41.356423+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:19:41.356423+0000","last_undegraded":"2024-09-19T07:19:41.356423+0000","last_fullsized":"2024-09-19T07:19:41.356423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352920+0000","last_change":"2024-09-19T07:19:40.361430+0000","last_active":"2024-09-19T07:19:41.352920+0000","last_peered":"2024-09-19T07:19:41.352920+0000","last_clean":"2024-09-19T07:19:41.352920+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:19:41.352920+0000","last_undegraded":"2024-09-19T07:19:41.352920+0000","last_fullsized":"2024-09-19T07:19:41.352920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:23:05.185943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369329+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:19:42.369329+0000","last_peered":"2024-09-19T07:19:42.369329+0000","last_clean":"2024-09-19T07:19:42.369329+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:19:42.369329+0000","last_undegraded":"2024-09-19T07:19:42.369329+0000","last_fullsized":"2024-09-19T07:19:42.369329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371825+0000","last_change":"2024-09-19T07:19:40.378172+0000","last_active":"2024-09-19T07:19:42.371825+0000","last_peered":"2024-09-19T07:19:42.371825+0000","last_clean":"2024-09-19T07:19:42.371825+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:19:42.371825+0000","last_undegraded":"2024-09-19T07:19:42.371825+0000","last_fullsized":"2024-09-19T07:19:42.371825+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:52.321790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366422+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:19:42.366422+0000","last_peered":"2024-09-19T07:19:42.366422+0000","last_clean":"2024-09-19T07:19:42.366422+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:19:42.366422+0000","last_undegraded":"2024-09-19T07:19:42.366422+0000","last_fullsized":"2024-09-19T07:19:42.366422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356589+0000","last_change":"2024-09-19T07:19:40.373909+0000","last_active":"2024-09-19T07:19:41.356589+0000","last_peered":"2024-09-19T07:19:41.356589+0000","last_clean":"2024-09-19T07:19:41.356589+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:19:41.356589+0000","last_undegraded":"2024-09-19T07:19:41.356589+0000","last_fullsized":"2024-09-19T07:19:41.356589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366411+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:19:42.366411+0000","last_peered":"2024-09-19T07:19:42.366411+0000","last_clean":"2024-09-19T07:19:42.366411+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:19:42.366411+0000","last_undegraded":"2024-09-19T07:19:42.366411+0000","last_fullsized":"2024-09-19T07:19:42.366411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367511+0000","last_change":"2024-09-19T07:19:40.396146+0000","last_active":"2024-09-19T07:19:42.367511+0000","last_peered":"2024-09-19T07:19:42.367511+0000","last_clean":"2024-09-19T07:19:42.367511+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:19:42.367511+0000","last_undegraded":"2024-09-19T07:19:42.367511+0000","last_fullsized":"2024-09-19T07:19:42.367511+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353532+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:19:41.353532+0000","last_peered":"2024-09-19T07:19:41.353532+0000","last_clean":"2024-09-19T07:19:41.353532+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:19:41.353532+0000","last_undegraded":"2024-09-19T07:19:41.353532+0000","last_fullsized":"2024-09-19T07:19:41.353532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355009+0000","last_change":"2024-09-19T07:19:40.373552+0000","last_active":"2024-09-19T07:19:41.355009+0000","last_peered":"2024-09-19T07:19:41.355009+0000","last_clean":"2024-09-19T07:19:41.355009+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:19:41.355009+0000","last_undegraded":"2024-09-19T07:19:41.355009+0000","last_fullsized":"2024-09-19T07:19:41.355009+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:23:33.629831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366537+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:19:42.366537+0000","last_peered":"2024-09-19T07:19:42.366537+0000","last_clean":"2024-09-19T07:19:42.366537+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:19:42.366537+0000","last_undegraded":"2024-09-19T07:19:42.366537+0000","last_fullsized":"2024-09-19T07:19:42.366537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367996+0000","last_change":"2024-09-19T07:19:40.356989+0000","last_active":"2024-09-19T07:19:42.367996+0000","last_peered":"2024-09-19T07:19:42.367996+0000","last_clean":"2024-09-19T07:19:42.367996+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:19:42.367996+0000","last_undegraded":"2024-09-19T07:19:42.367996+0000","last_fullsized":"2024-09-19T07:19:42.367996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353682+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:19:41.353682+0000","last_peered":"2024-09-19T07:19:41.353682+0000","last_clean":"2024-09-19T07:19:41.353682+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:19:41.353682+0000","last_undegraded":"2024-09-19T07:19:41.353682+0000","last_fullsized":"2024-09-19T07:19:41.353682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348160+0000","last_change":"2024-09-19T07:19:40.376487+0000","last_active":"2024-09-19T07:19:41.348160+0000","last_peered":"2024-09-19T07:19:41.348160+0000","last_clean":"2024-09-19T07:19:41.348160+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:19:41.348160+0000","last_undegraded":"2024-09-19T07:19:41.348160+0000","last_fullsized":"2024-09-19T07:19:41.348160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:05.185451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369628+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:19:42.369628+0000","last_peered":"2024-09-19T07:19:42.369628+0000","last_clean":"2024-09-19T07:19:42.369628+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:19:42.369628+0000","last_undegraded":"2024-09-19T07:19:42.369628+0000","last_fullsized":"2024-09-19T07:19:42.369628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:25.667168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364909+0000","last_change":"2024-09-19T07:19:40.377604+0000","last_active":"2024-09-19T07:19:42.364909+0000","last_peered":"2024-09-19T07:19:42.364909+0000","last_clean":"2024-09-19T07:19:42.364909+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:19:42.364909+0000","last_undegraded":"2024-09-19T07:19:42.364909+0000","last_fullsized":"2024-09-19T07:19:42.364909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368557+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:19:42.368557+0000","last_peered":"2024-09-19T07:19:42.368557+0000","last_clean":"2024-09-19T07:19:42.368557+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:19:42.368557+0000","last_undegraded":"2024-09-19T07:19:42.368557+0000","last_fullsized":"2024-09-19T07:19:42.368557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365847+0000","last_change":"2024-09-19T07:19:40.377580+0000","last_active":"2024-09-19T07:19:42.365847+0000","last_peered":"2024-09-19T07:19:42.365847+0000","last_clean":"2024-09-19T07:19:42.365847+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:19:42.365847+0000","last_undegraded":"2024-09-19T07:19:42.365847+0000","last_fullsized":"2024-09-19T07:19:42.365847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369285+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:19:42.369285+0000","last_peered":"2024-09-19T07:19:42.369285+0000","last_clean":"2024-09-19T07:19:42.369285+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:19:42.369285+0000","last_undegraded":"2024-09-19T07:19:42.369285+0000","last_fullsized":"2024-09-19T07:19:42.369285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2853,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2853,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542156,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":864256,"data_stored":693530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":105161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":103252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":282624,"data_stored":102720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":106179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542157,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":102560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542156,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":104727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542155,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":868352,"data_stored":693496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:20:04.494 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:20:04.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.626+0000 7fb631bec640 1 Processor -- start 2024-09-19T07:20:04.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb631bec640 1 -- start start 2024-09-19T07:20:04.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb631bec640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb62c155d70 0x7fb62c0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:04.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb631bec640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb62c1592f0 0x7fb62c157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:04.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb631bec640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb62c156200 con 0x7fb62c0c6120 2024-09-19T07:20:04.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb631bec640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb62c0c9930 con 0x7fb62c1592f0 2024-09-19T07:20:04.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb631bec640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb62c1564e0 con 0x7fb62c155d70 2024-09-19T07:20:04.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb62bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb62c1592f0 0x7fb62c157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:04.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb62b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb62c0c6120 0x7fb62c1575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50028/0 (socket says 172.21.15.83:50028) 2024-09-19T07:20:04.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb62b7fe640 1 -- 172.21.15.83:0/2998830566 learned_addr learned my addr 172.21.15.83:0/2998830566 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:04.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb62bfff640 1 -- 172.21.15.83:0/2998830566 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb62c155d70 msgr2=0x7fb62c0c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:04.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb62bfff640 1 --2- 172.21.15.83:0/2998830566 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb62c155d70 0x7fb62c0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb62bfff640 1 -- 172.21.15.83:0/2998830566 >> v1:172.21.15.83:6789/0 conn(0x7fb62c0c6120 legacy=0x7fb62c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb62a7fc640 1 -- 172.21.15.83:0/2998830566 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 441232431 0 0) 0x7fb62c156200 con 0x7fb62c0c6120 2024-09-19T07:20:04.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb62bfff640 1 -- 172.21.15.83:0/2998830566 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62c1567c0 con 0x7fb62c1592f0 2024-09-19T07:20:04.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb62bfff640 1 --2- 172.21.15.83:0/2998830566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb62c1592f0 0x7fb62c157ce0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb624009d90 tx=0x7fb62401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6095a03e7c41b73c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:04.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb62a7fc640 1 -- 172.21.15.83:0/2998830566 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6240b2070 con 0x7fb62c1592f0 2024-09-19T07:20:04.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb62a7fc640 1 -- 172.21.15.83:0/2998830566 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6240a5830 con 0x7fb62c1592f0 2024-09-19T07:20:04.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.630+0000 7fb62a7fc640 1 -- 172.21.15.83:0/2998830566 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6240a5c00 con 0x7fb62c1592f0 2024-09-19T07:20:04.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 -- 172.21.15.83:0/2998830566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb62c1592f0 msgr2=0x7fb62c157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 --2- 172.21.15.83:0/2998830566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb62c1592f0 0x7fb62c157ce0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb624009d90 tx=0x7fb62401c970 comp rx=0 tx=0).stop 2024-09-19T07:20:04.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 -- 172.21.15.83:0/2998830566 shutdown_connections 2024-09-19T07:20:04.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 --2- 172.21.15.83:0/2998830566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb62c1592f0 0x7fb62c157ce0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 --2- 172.21.15.83:0/2998830566 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb62c155d70 0x7fb62c0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 -- 172.21.15.83:0/2998830566 >> 172.21.15.83:0/2998830566 conn(0x7fb62c06bc80 msgr2=0x7fb62c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:04.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 -- 172.21.15.83:0/2998830566 shutdown_connections 2024-09-19T07:20:04.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 -- 172.21.15.83:0/2998830566 wait complete. 2024-09-19T07:20:04.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 Processor -- start 2024-09-19T07:20:04.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 -- start start 2024-09-19T07:20:04.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb62c0c6120 0x7fb62c0ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:04.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb62c1592f0 0x7fb62c0c1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:04.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb62c0caeb0 0x7fb62c0cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:04.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c153860 con 0x7fb62c1592f0 2024-09-19T07:20:04.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c162ea0 con 0x7fb62c0c6120 2024-09-19T07:20:04.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb62c146af0 con 0x7fb62c0caeb0 2024-09-19T07:20:04.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb62affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb62c1592f0 0x7fb62c0c1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:04.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb62affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb62c1592f0 0x7fb62c0c1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49684/0 (socket says 172.21.15.83:49684) 2024-09-19T07:20:04.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb62affd640 1 -- 172.21.15.83:0/3643558249 learned_addr learned my addr 172.21.15.83:0/3643558249 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:04.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb62affd640 1 -- 172.21.15.83:0/3643558249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb62c0caeb0 msgr2=0x7fb62c0cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb62affd640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb62c0caeb0 0x7fb62c0cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb62b7fe640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb62c0c6120 0x7fb62c0ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:04.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb62affd640 1 -- 172.21.15.83:0/3643558249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb62c0c6120 msgr2=0x7fb62c0ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb62affd640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb62c0c6120 0x7fb62c0ca5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb62affd640 1 -- 172.21.15.83:0/3643558249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62c146dd0 con 0x7fb62c1592f0 2024-09-19T07:20:04.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb62affd640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb62c1592f0 0x7fb62c0c1bd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb61400b550 tx=0x7fb61400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:04.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb623fff640 1 -- 172.21.15.83:0/3643558249 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614014070 con 0x7fb62c1592f0 2024-09-19T07:20:04.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb623fff640 1 -- 172.21.15.83:0/3643558249 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb614010890 con 0x7fb62c1592f0 2024-09-19T07:20:04.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 -- 172.21.15.83:0/3643558249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62c147cc0 con 0x7fb62c1592f0 2024-09-19T07:20:04.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb623fff640 1 -- 172.21.15.83:0/3643558249 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614010c80 con 0x7fb62c1592f0 2024-09-19T07:20:04.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 -- 172.21.15.83:0/3643558249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb62c148280 con 0x7fb62c1592f0 2024-09-19T07:20:04.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.634+0000 7fb631bec640 1 -- 172.21.15.83:0/3643558249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb62c148480 con 0x7fb62c1592f0 2024-09-19T07:20:04.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.638+0000 7fb623fff640 1 -- 172.21.15.83:0/3643558249 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb614017070 con 0x7fb62c1592f0 2024-09-19T07:20:04.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.638+0000 7fb623fff640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb60c0678c0 0x7fb60c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:04.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.638+0000 7fb62b7fe640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb60c0678c0 0x7fb60c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:04.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.638+0000 7fb623fff640 1 -- 172.21.15.83:0/3643558249 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fb614085e20 con 0x7fb62c1592f0 2024-09-19T07:20:04.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.638+0000 7fb62b7fe640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb60c0678c0 0x7fb60c069d80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb61c000f70 tx=0x7fb61c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:04.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.642+0000 7fb623fff640 1 -- 172.21.15.83:0/3643558249 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb62c148480 con 0x7fb62c1592f0 2024-09-19T07:20:04.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.790+0000 7fb631bec640 1 -- 172.21.15.83:0/3643558249 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb62c0c6500 con 0x7fb60c0678c0 2024-09-19T07:20:04.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.826+0000 7fb623fff640 1 -- 172.21.15.83:0/3643558249 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490337 (secure 0 0 0) 0x7fb62c0c6500 con 0x7fb60c0678c0 2024-09-19T07:20:04.830 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:04.834 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:20:04.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.830+0000 7fb631bec640 1 -- 172.21.15.83:0/3643558249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb60c0678c0 msgr2=0x7fb60c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.830+0000 7fb631bec640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb60c0678c0 0x7fb60c069d80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb61c000f70 tx=0x7fb61c009040 comp rx=0 tx=0).stop 2024-09-19T07:20:04.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.830+0000 7fb631bec640 1 -- 172.21.15.83:0/3643558249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb62c1592f0 msgr2=0x7fb62c0c1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:04.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.830+0000 7fb631bec640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb62c1592f0 0x7fb62c0c1bd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb61400b550 tx=0x7fb61400ba10 comp rx=0 tx=0).stop 2024-09-19T07:20:04.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.830+0000 7fb631bec640 1 -- 172.21.15.83:0/3643558249 shutdown_connections 2024-09-19T07:20:04.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.830+0000 7fb631bec640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb60c0678c0 0x7fb60c069d80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.830+0000 7fb631bec640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb62c0caeb0 0x7fb62c0cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.830+0000 7fb631bec640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb62c1592f0 0x7fb62c0c1bd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.830+0000 7fb631bec640 1 --2- 172.21.15.83:0/3643558249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb62c0c6120 0x7fb62c0ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:04.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.830+0000 7fb631bec640 1 -- 172.21.15.83:0/3643558249 >> 172.21.15.83:0/3643558249 conn(0x7fb62c06bc80 msgr2=0x7fb62c06c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:04.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.830+0000 7fb631bec640 1 -- 172.21.15.83:0/3643558249 shutdown_connections 2024-09-19T07:20:04.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:04.830+0000 7fb631bec640 1 -- 172.21.15.83:0/3643558249 wait complete. 2024-09-19T07:20:04.849 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-19T07:20:02.820535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596074,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":253,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223724,"kb_used_data":3336,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750996,"statfs":{"total":773094113280,"available":772865019904,"internally_reserved":0,"allocated":3416064,"data_stored":2011625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":886,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003877"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364888+0000","last_change":"2024-09-19T07:19:40.358457+0000","last_active":"2024-09-19T07:19:42.364888+0000","last_peered":"2024-09-19T07:19:42.364888+0000","last_clean":"2024-09-19T07:19:42.364888+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:19:42.364888+0000","last_undegraded":"2024-09-19T07:19:42.364888+0000","last_fullsized":"2024-09-19T07:19:42.364888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370195+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:19:42.370195+0000","last_peered":"2024-09-19T07:19:42.370195+0000","last_clean":"2024-09-19T07:19:42.370195+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:19:42.370195+0000","last_undegraded":"2024-09-19T07:19:42.370195+0000","last_fullsized":"2024-09-19T07:19:42.370195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:57.799442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372168+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:19:42.372168+0000","last_peered":"2024-09-19T07:19:42.372168+0000","last_clean":"2024-09-19T07:19:42.372168+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:19:42.372168+0000","last_undegraded":"2024-09-19T07:19:42.372168+0000","last_fullsized":"2024-09-19T07:19:42.372168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368511+0000","last_change":"2024-09-19T07:19:39.401621+0000","last_active":"2024-09-19T07:19:42.368511+0000","last_peered":"2024-09-19T07:19:42.368511+0000","last_clean":"2024-09-19T07:19:42.368511+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:19:42.368511+0000","last_undegraded":"2024-09-19T07:19:42.368511+0000","last_fullsized":"2024-09-19T07:19:42.368511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:32.095381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366428+0000","last_change":"2024-09-19T07:19:40.366073+0000","last_active":"2024-09-19T07:19:42.366428+0000","last_peered":"2024-09-19T07:19:42.366428+0000","last_clean":"2024-09-19T07:19:42.366428+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:19:42.366428+0000","last_undegraded":"2024-09-19T07:19:42.366428+0000","last_fullsized":"2024-09-19T07:19:42.366428+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372588+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:19:42.372588+0000","last_peered":"2024-09-19T07:19:42.372588+0000","last_clean":"2024-09-19T07:19:42.372588+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:19:42.372588+0000","last_undegraded":"2024-09-19T07:19:42.372588+0000","last_fullsized":"2024-09-19T07:19:42.372588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369500+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:19:42.369500+0000","last_peered":"2024-09-19T07:19:42.369500+0000","last_clean":"2024-09-19T07:19:42.369500+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:19:42.369500+0000","last_undegraded":"2024-09-19T07:19:42.369500+0000","last_fullsized":"2024-09-19T07:19:42.369500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:15.036396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364782+0000","last_change":"2024-09-19T07:19:39.403810+0000","last_active":"2024-09-19T07:19:42.364782+0000","last_peered":"2024-09-19T07:19:42.364782+0000","last_clean":"2024-09-19T07:19:42.364782+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:19:42.364782+0000","last_undegraded":"2024-09-19T07:19:42.364782+0000","last_fullsized":"2024-09-19T07:19:42.364782+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:55.714964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365790+0000","last_change":"2024-09-19T07:19:40.390455+0000","last_active":"2024-09-19T07:19:42.365790+0000","last_peered":"2024-09-19T07:19:42.365790+0000","last_clean":"2024-09-19T07:19:42.365790+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:19:42.365790+0000","last_undegraded":"2024-09-19T07:19:42.365790+0000","last_fullsized":"2024-09-19T07:19:42.365790+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367307+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:19:42.367307+0000","last_peered":"2024-09-19T07:19:42.367307+0000","last_clean":"2024-09-19T07:19:42.367307+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:19:42.367307+0000","last_undegraded":"2024-09-19T07:19:42.367307+0000","last_fullsized":"2024-09-19T07:19:42.367307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353956+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:19:41.353956+0000","last_peered":"2024-09-19T07:19:41.353956+0000","last_clean":"2024-09-19T07:19:41.353956+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:19:41.353956+0000","last_undegraded":"2024-09-19T07:19:41.353956+0000","last_fullsized":"2024-09-19T07:19:41.353956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366091+0000","last_change":"2024-09-19T07:19:39.408309+0000","last_active":"2024-09-19T07:19:42.366091+0000","last_peered":"2024-09-19T07:19:42.366091+0000","last_clean":"2024-09-19T07:19:42.366091+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:19:42.366091+0000","last_undegraded":"2024-09-19T07:19:42.366091+0000","last_fullsized":"2024-09-19T07:19:42.366091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:19.396261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353849+0000","last_change":"2024-09-19T07:19:40.372543+0000","last_active":"2024-09-19T07:19:41.353849+0000","last_peered":"2024-09-19T07:19:41.353849+0000","last_clean":"2024-09-19T07:19:41.353849+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:19:41.353849+0000","last_undegraded":"2024-09-19T07:19:41.353849+0000","last_fullsized":"2024-09-19T07:19:41.353849+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:53:15.682896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367705+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:19:42.367705+0000","last_peered":"2024-09-19T07:19:42.367705+0000","last_clean":"2024-09-19T07:19:42.367705+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:19:42.367705+0000","last_undegraded":"2024-09-19T07:19:42.367705+0000","last_fullsized":"2024-09-19T07:19:42.367705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355253+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:19:41.355253+0000","last_peered":"2024-09-19T07:19:41.355253+0000","last_clean":"2024-09-19T07:19:41.355253+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:19:41.355253+0000","last_undegraded":"2024-09-19T07:19:41.355253+0000","last_fullsized":"2024-09-19T07:19:41.355253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366176+0000","last_change":"2024-09-19T07:19:39.407045+0000","last_active":"2024-09-19T07:19:42.366176+0000","last_peered":"2024-09-19T07:19:42.366176+0000","last_clean":"2024-09-19T07:19:42.366176+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:19:42.366176+0000","last_undegraded":"2024-09-19T07:19:42.366176+0000","last_fullsized":"2024-09-19T07:19:42.366176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:24.846974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367577+0000","last_change":"2024-09-19T07:19:40.396520+0000","last_active":"2024-09-19T07:19:42.367577+0000","last_peered":"2024-09-19T07:19:42.367577+0000","last_clean":"2024-09-19T07:19:42.367577+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:19:42.367577+0000","last_undegraded":"2024-09-19T07:19:42.367577+0000","last_fullsized":"2024-09-19T07:19:42.367577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353504+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:19:41.353504+0000","last_peered":"2024-09-19T07:19:41.353504+0000","last_clean":"2024-09-19T07:19:41.353504+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:19:41.353504+0000","last_undegraded":"2024-09-19T07:19:41.353504+0000","last_fullsized":"2024-09-19T07:19:41.353504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367046+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:19:42.367046+0000","last_peered":"2024-09-19T07:19:42.367046+0000","last_clean":"2024-09-19T07:19:42.367046+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:19:42.367046+0000","last_undegraded":"2024-09-19T07:19:42.367046+0000","last_fullsized":"2024-09-19T07:19:42.367046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368034+0000","last_change":"2024-09-19T07:19:39.410500+0000","last_active":"2024-09-19T07:19:42.368034+0000","last_peered":"2024-09-19T07:19:42.368034+0000","last_clean":"2024-09-19T07:19:42.368034+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:19:42.368034+0000","last_undegraded":"2024-09-19T07:19:42.368034+0000","last_fullsized":"2024-09-19T07:19:42.368034+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:04.182606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349177+0000","last_change":"2024-09-19T07:19:40.381259+0000","last_active":"2024-09-19T07:19:41.349177+0000","last_peered":"2024-09-19T07:19:41.349177+0000","last_clean":"2024-09-19T07:19:41.349177+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:19:41.349177+0000","last_undegraded":"2024-09-19T07:19:41.349177+0000","last_fullsized":"2024-09-19T07:19:41.349177+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:22.017682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370825+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:19:42.370825+0000","last_peered":"2024-09-19T07:19:42.370825+0000","last_clean":"2024-09-19T07:19:42.370825+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:19:42.370825+0000","last_undegraded":"2024-09-19T07:19:42.370825+0000","last_fullsized":"2024-09-19T07:19:42.370825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367918+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:19:42.367918+0000","last_peered":"2024-09-19T07:19:42.367918+0000","last_clean":"2024-09-19T07:19:42.367918+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:19:42.367918+0000","last_undegraded":"2024-09-19T07:19:42.367918+0000","last_fullsized":"2024-09-19T07:19:42.367918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367682+0000","last_change":"2024-09-19T07:19:39.412123+0000","last_active":"2024-09-19T07:19:42.367682+0000","last_peered":"2024-09-19T07:19:42.367682+0000","last_clean":"2024-09-19T07:19:42.367682+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:19:42.367682+0000","last_undegraded":"2024-09-19T07:19:42.367682+0000","last_fullsized":"2024-09-19T07:19:42.367682+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367943+0000","last_change":"2024-09-19T07:19:40.383214+0000","last_active":"2024-09-19T07:19:42.367943+0000","last_peered":"2024-09-19T07:19:42.367943+0000","last_clean":"2024-09-19T07:19:42.367943+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:19:42.367943+0000","last_undegraded":"2024-09-19T07:19:42.367943+0000","last_fullsized":"2024-09-19T07:19:42.367943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367066+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:19:42.367066+0000","last_peered":"2024-09-19T07:19:42.367066+0000","last_clean":"2024-09-19T07:19:42.367066+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:19:42.367066+0000","last_undegraded":"2024-09-19T07:19:42.367066+0000","last_fullsized":"2024-09-19T07:19:42.367066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367094+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:19:42.367094+0000","last_peered":"2024-09-19T07:19:42.367094+0000","last_clean":"2024-09-19T07:19:42.367094+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:19:42.367094+0000","last_undegraded":"2024-09-19T07:19:42.367094+0000","last_fullsized":"2024-09-19T07:19:42.367094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369704+0000","last_change":"2024-09-19T07:19:39.408228+0000","last_active":"2024-09-19T07:19:42.369704+0000","last_peered":"2024-09-19T07:19:42.369704+0000","last_clean":"2024-09-19T07:19:42.369704+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:19:42.369704+0000","last_undegraded":"2024-09-19T07:19:42.369704+0000","last_fullsized":"2024-09-19T07:19:42.369704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355858+0000","last_change":"2024-09-19T07:19:40.395566+0000","last_active":"2024-09-19T07:19:41.355858+0000","last_peered":"2024-09-19T07:19:41.355858+0000","last_clean":"2024-09-19T07:19:41.355858+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:19:41.355858+0000","last_undegraded":"2024-09-19T07:19:41.355858+0000","last_fullsized":"2024-09-19T07:19:41.355858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:54:02.766021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367642+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:19:42.367642+0000","last_peered":"2024-09-19T07:19:42.367642+0000","last_clean":"2024-09-19T07:19:42.367642+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:19:42.367642+0000","last_undegraded":"2024-09-19T07:19:42.367642+0000","last_fullsized":"2024-09-19T07:19:42.367642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366078+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:19:42.366078+0000","last_peered":"2024-09-19T07:19:42.366078+0000","last_clean":"2024-09-19T07:19:42.366078+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:19:42.366078+0000","last_undegraded":"2024-09-19T07:19:42.366078+0000","last_fullsized":"2024-09-19T07:19:42.366078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368203+0000","last_change":"2024-09-19T07:19:39.424661+0000","last_active":"2024-09-19T07:19:42.368203+0000","last_peered":"2024-09-19T07:19:42.368203+0000","last_clean":"2024-09-19T07:19:42.368203+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:19:42.368203+0000","last_undegraded":"2024-09-19T07:19:42.368203+0000","last_fullsized":"2024-09-19T07:19:42.368203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365431+0000","last_change":"2024-09-19T07:19:40.388972+0000","last_active":"2024-09-19T07:19:42.365431+0000","last_peered":"2024-09-19T07:19:42.365431+0000","last_clean":"2024-09-19T07:19:42.365431+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:19:42.365431+0000","last_undegraded":"2024-09-19T07:19:42.365431+0000","last_fullsized":"2024-09-19T07:19:42.365431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371005+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:19:42.371005+0000","last_peered":"2024-09-19T07:19:42.371005+0000","last_clean":"2024-09-19T07:19:42.371005+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:19:42.371005+0000","last_undegraded":"2024-09-19T07:19:42.371005+0000","last_fullsized":"2024-09-19T07:19:42.371005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:23.791880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.357019+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:19:41.357019+0000","last_peered":"2024-09-19T07:19:41.357019+0000","last_clean":"2024-09-19T07:19:41.357019+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:19:41.357019+0000","last_undegraded":"2024-09-19T07:19:41.357019+0000","last_fullsized":"2024-09-19T07:19:41.357019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366707+0000","last_change":"2024-09-19T07:19:39.412192+0000","last_active":"2024-09-19T07:19:42.366707+0000","last_peered":"2024-09-19T07:19:42.366707+0000","last_clean":"2024-09-19T07:19:42.366707+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:19:42.366707+0000","last_undegraded":"2024-09-19T07:19:42.366707+0000","last_fullsized":"2024-09-19T07:19:42.366707+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365056+0000","last_change":"2024-09-19T07:19:40.390217+0000","last_active":"2024-09-19T07:19:42.365056+0000","last_peered":"2024-09-19T07:19:42.365056+0000","last_clean":"2024-09-19T07:19:42.365056+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:19:42.365056+0000","last_undegraded":"2024-09-19T07:19:42.365056+0000","last_fullsized":"2024-09-19T07:19:42.365056+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371269+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:19:42.371269+0000","last_peered":"2024-09-19T07:19:42.371269+0000","last_clean":"2024-09-19T07:19:42.371269+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:19:42.371269+0000","last_undegraded":"2024-09-19T07:19:42.371269+0000","last_fullsized":"2024-09-19T07:19:42.371269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368999+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:19:42.368999+0000","last_peered":"2024-09-19T07:19:42.368999+0000","last_clean":"2024-09-19T07:19:42.368999+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:19:42.368999+0000","last_undegraded":"2024-09-19T07:19:42.368999+0000","last_fullsized":"2024-09-19T07:19:42.368999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:50.700962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365712+0000","last_change":"2024-09-19T07:19:39.434150+0000","last_active":"2024-09-19T07:19:42.365712+0000","last_peered":"2024-09-19T07:19:42.365712+0000","last_clean":"2024-09-19T07:19:42.365712+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:19:42.365712+0000","last_undegraded":"2024-09-19T07:19:42.365712+0000","last_fullsized":"2024-09-19T07:19:42.365712+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:13.654208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352846+0000","last_change":"2024-09-19T07:19:40.380140+0000","last_active":"2024-09-19T07:19:41.352846+0000","last_peered":"2024-09-19T07:19:41.352846+0000","last_clean":"2024-09-19T07:19:41.352846+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:19:41.352846+0000","last_undegraded":"2024-09-19T07:19:41.352846+0000","last_fullsized":"2024-09-19T07:19:41.352846+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368156+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:19:42.368156+0000","last_peered":"2024-09-19T07:19:42.368156+0000","last_clean":"2024-09-19T07:19:42.368156+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:19:42.368156+0000","last_undegraded":"2024-09-19T07:19:42.368156+0000","last_fullsized":"2024-09-19T07:19:42.368156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367593+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:19:42.367593+0000","last_peered":"2024-09-19T07:19:42.367593+0000","last_clean":"2024-09-19T07:19:42.367593+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:19:42.367593+0000","last_undegraded":"2024-09-19T07:19:42.367593+0000","last_fullsized":"2024-09-19T07:19:42.367593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366996+0000","last_change":"2024-09-19T07:19:39.411605+0000","last_active":"2024-09-19T07:19:42.366996+0000","last_peered":"2024-09-19T07:19:42.366996+0000","last_clean":"2024-09-19T07:19:42.366996+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:19:42.366996+0000","last_undegraded":"2024-09-19T07:19:42.366996+0000","last_fullsized":"2024-09-19T07:19:42.366996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:55.654483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367982+0000","last_change":"2024-09-19T07:19:40.372083+0000","last_active":"2024-09-19T07:19:42.367982+0000","last_peered":"2024-09-19T07:19:42.367982+0000","last_clean":"2024-09-19T07:19:42.367982+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:19:42.367982+0000","last_undegraded":"2024-09-19T07:19:42.367982+0000","last_fullsized":"2024-09-19T07:19:42.367982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:39.665475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355342+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:19:41.355342+0000","last_peered":"2024-09-19T07:19:41.355342+0000","last_clean":"2024-09-19T07:19:41.355342+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:19:41.355342+0000","last_undegraded":"2024-09-19T07:19:41.355342+0000","last_fullsized":"2024-09-19T07:19:41.355342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368442+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:19:42.368442+0000","last_peered":"2024-09-19T07:19:42.368442+0000","last_clean":"2024-09-19T07:19:42.368442+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:19:42.368442+0000","last_undegraded":"2024-09-19T07:19:42.368442+0000","last_fullsized":"2024-09-19T07:19:42.368442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367339+0000","last_change":"2024-09-19T07:19:39.420340+0000","last_active":"2024-09-19T07:19:42.367339+0000","last_peered":"2024-09-19T07:19:42.367339+0000","last_clean":"2024-09-19T07:19:42.367339+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:19:42.367339+0000","last_undegraded":"2024-09-19T07:19:42.367339+0000","last_fullsized":"2024-09-19T07:19:42.367339+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:29.338848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369245+0000","last_change":"2024-09-19T07:19:40.383082+0000","last_active":"2024-09-19T07:19:42.369245+0000","last_peered":"2024-09-19T07:19:42.369245+0000","last_clean":"2024-09-19T07:19:42.369245+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:19:42.369245+0000","last_undegraded":"2024-09-19T07:19:42.369245+0000","last_fullsized":"2024-09-19T07:19:42.369245+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:54.436493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367222+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:19:42.367222+0000","last_peered":"2024-09-19T07:19:42.367222+0000","last_clean":"2024-09-19T07:19:42.367222+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:19:42.367222+0000","last_undegraded":"2024-09-19T07:19:42.367222+0000","last_fullsized":"2024-09-19T07:19:42.367222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354505+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:19:41.354505+0000","last_peered":"2024-09-19T07:19:41.354505+0000","last_clean":"2024-09-19T07:19:41.354505+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:19:41.354505+0000","last_undegraded":"2024-09-19T07:19:41.354505+0000","last_fullsized":"2024-09-19T07:19:41.354505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368035+0000","last_change":"2024-09-19T07:19:39.407135+0000","last_active":"2024-09-19T07:19:42.368035+0000","last_peered":"2024-09-19T07:19:42.368035+0000","last_clean":"2024-09-19T07:19:42.368035+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:19:42.368035+0000","last_undegraded":"2024-09-19T07:19:42.368035+0000","last_fullsized":"2024-09-19T07:19:42.368035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:27.890595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349118+0000","last_change":"2024-09-19T07:19:40.368866+0000","last_active":"2024-09-19T07:19:41.349118+0000","last_peered":"2024-09-19T07:19:41.349118+0000","last_clean":"2024-09-19T07:19:41.349118+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:19:41.349118+0000","last_undegraded":"2024-09-19T07:19:41.349118+0000","last_fullsized":"2024-09-19T07:19:41.349118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:28.893439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366065+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:19:42.366065+0000","last_peered":"2024-09-19T07:19:42.366065+0000","last_clean":"2024-09-19T07:19:42.366065+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:19:42.366065+0000","last_undegraded":"2024-09-19T07:19:42.366065+0000","last_fullsized":"2024-09-19T07:19:42.366065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348283+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:19:41.348283+0000","last_peered":"2024-09-19T07:19:41.348283+0000","last_clean":"2024-09-19T07:19:41.348283+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:19:41.348283+0000","last_undegraded":"2024-09-19T07:19:41.348283+0000","last_fullsized":"2024-09-19T07:19:41.348283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369806+0000","last_change":"2024-09-19T07:19:39.420497+0000","last_active":"2024-09-19T07:19:42.369806+0000","last_peered":"2024-09-19T07:19:42.369806+0000","last_clean":"2024-09-19T07:19:42.369806+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:19:42.369806+0000","last_undegraded":"2024-09-19T07:19:42.369806+0000","last_fullsized":"2024-09-19T07:19:42.369806+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:19.547270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356452+0000","last_change":"2024-09-19T07:19:40.387573+0000","last_active":"2024-09-19T07:19:41.356452+0000","last_peered":"2024-09-19T07:19:41.356452+0000","last_clean":"2024-09-19T07:19:41.356452+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:19:41.356452+0000","last_undegraded":"2024-09-19T07:19:41.356452+0000","last_fullsized":"2024-09-19T07:19:41.356452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:20:43.234793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365631+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:19:42.365631+0000","last_peered":"2024-09-19T07:19:42.365631+0000","last_clean":"2024-09-19T07:19:42.365631+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:19:42.365631+0000","last_undegraded":"2024-09-19T07:19:42.365631+0000","last_fullsized":"2024-09-19T07:19:42.365631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:20.824916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366356+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:19:42.366356+0000","last_peered":"2024-09-19T07:19:42.366356+0000","last_clean":"2024-09-19T07:19:42.366356+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:19:42.366356+0000","last_undegraded":"2024-09-19T07:19:42.366356+0000","last_fullsized":"2024-09-19T07:19:42.366356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371671+0000","last_change":"2024-09-19T07:19:39.410752+0000","last_active":"2024-09-19T07:19:42.371671+0000","last_peered":"2024-09-19T07:19:42.371671+0000","last_clean":"2024-09-19T07:19:42.371671+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:42.371671+0000","last_undegraded":"2024-09-19T07:19:42.371671+0000","last_fullsized":"2024-09-19T07:19:42.371671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:47:09.376497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367120+0000","last_change":"2024-09-19T07:19:40.392973+0000","last_active":"2024-09-19T07:19:42.367120+0000","last_peered":"2024-09-19T07:19:42.367120+0000","last_clean":"2024-09-19T07:19:42.367120+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:19:42.367120+0000","last_undegraded":"2024-09-19T07:19:42.367120+0000","last_fullsized":"2024-09-19T07:19:42.367120+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368064+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:19:42.368064+0000","last_peered":"2024-09-19T07:19:42.368064+0000","last_clean":"2024-09-19T07:19:42.368064+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:19:42.368064+0000","last_undegraded":"2024-09-19T07:19:42.368064+0000","last_fullsized":"2024-09-19T07:19:42.368064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354067+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:19:41.354067+0000","last_peered":"2024-09-19T07:19:41.354067+0000","last_clean":"2024-09-19T07:19:41.354067+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:19:41.354067+0000","last_undegraded":"2024-09-19T07:19:41.354067+0000","last_fullsized":"2024-09-19T07:19:41.354067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367007+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:19:42.367007+0000","last_peered":"2024-09-19T07:19:42.367007+0000","last_clean":"2024-09-19T07:19:42.367007+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:19:42.367007+0000","last_undegraded":"2024-09-19T07:19:42.367007+0000","last_fullsized":"2024-09-19T07:19:42.367007+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354541+0000","last_change":"2024-09-19T07:19:39.429085+0000","last_active":"2024-09-19T07:19:41.354541+0000","last_peered":"2024-09-19T07:19:41.354541+0000","last_clean":"2024-09-19T07:19:41.354541+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:19:41.354541+0000","last_undegraded":"2024-09-19T07:19:41.354541+0000","last_fullsized":"2024-09-19T07:19:41.354541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348394+0000","last_change":"2024-09-19T07:19:40.382352+0000","last_active":"2024-09-19T07:19:41.348394+0000","last_peered":"2024-09-19T07:19:41.348394+0000","last_clean":"2024-09-19T07:19:41.348394+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:19:41.348394+0000","last_undegraded":"2024-09-19T07:19:41.348394+0000","last_fullsized":"2024-09-19T07:19:41.348394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:31.024388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367107+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:19:42.367107+0000","last_peered":"2024-09-19T07:19:42.367107+0000","last_clean":"2024-09-19T07:19:42.367107+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:19:42.367107+0000","last_undegraded":"2024-09-19T07:19:42.367107+0000","last_fullsized":"2024-09-19T07:19:42.367107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348973+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:19:41.348973+0000","last_peered":"2024-09-19T07:19:41.348973+0000","last_clean":"2024-09-19T07:19:41.348973+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:19:41.348973+0000","last_undegraded":"2024-09-19T07:19:41.348973+0000","last_fullsized":"2024-09-19T07:19:41.348973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353111+0000","last_change":"2024-09-19T07:19:39.427644+0000","last_active":"2024-09-19T07:19:41.353111+0000","last_peered":"2024-09-19T07:19:41.353111+0000","last_clean":"2024-09-19T07:19:41.353111+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:19:41.353111+0000","last_undegraded":"2024-09-19T07:19:41.353111+0000","last_fullsized":"2024-09-19T07:19:41.353111+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348470+0000","last_change":"2024-09-19T07:19:40.390380+0000","last_active":"2024-09-19T07:19:41.348470+0000","last_peered":"2024-09-19T07:19:41.348470+0000","last_clean":"2024-09-19T07:19:41.348470+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:19:41.348470+0000","last_undegraded":"2024-09-19T07:19:41.348470+0000","last_fullsized":"2024-09-19T07:19:41.348470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:26.385490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354180+0000","last_change":"2024-09-19T07:19:32.276600+0000","last_active":"2024-09-19T07:19:41.354180+0000","last_peered":"2024-09-19T07:19:41.354180+0000","last_clean":"2024-09-19T07:19:41.354180+0000","last_became_active":"2024-09-19T07:19:32.275766+0000","last_became_peered":"2024-09-19T07:19:32.275766+0000","last_unstale":"2024-09-19T07:19:41.354180+0000","last_undegraded":"2024-09-19T07:19:41.354180+0000","last_fullsized":"2024-09-19T07:19:41.354180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:48.506399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370499+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:19:42.370499+0000","last_peered":"2024-09-19T07:19:42.370499+0000","last_clean":"2024-09-19T07:19:42.370499+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:19:42.370499+0000","last_undegraded":"2024-09-19T07:19:42.370499+0000","last_fullsized":"2024-09-19T07:19:42.370499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366597+0000","last_change":"2024-09-19T07:19:39.410638+0000","last_active":"2024-09-19T07:19:42.366597+0000","last_peered":"2024-09-19T07:19:42.366597+0000","last_clean":"2024-09-19T07:19:42.366597+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:19:42.366597+0000","last_undegraded":"2024-09-19T07:19:42.366597+0000","last_fullsized":"2024-09-19T07:19:42.366597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:32.073136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365442+0000","last_change":"2024-09-19T07:19:40.378481+0000","last_active":"2024-09-19T07:19:42.365442+0000","last_peered":"2024-09-19T07:19:42.365442+0000","last_clean":"2024-09-19T07:19:42.365442+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:19:42.365442+0000","last_undegraded":"2024-09-19T07:19:42.365442+0000","last_fullsized":"2024-09-19T07:19:42.365442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:30.585686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369508+0000","last_change":"2024-09-19T07:19:32.292026+0000","last_active":"2024-09-19T07:19:42.369508+0000","last_peered":"2024-09-19T07:19:42.369508+0000","last_clean":"2024-09-19T07:19:42.369508+0000","last_became_active":"2024-09-19T07:19:32.291037+0000","last_became_peered":"2024-09-19T07:19:32.291037+0000","last_unstale":"2024-09-19T07:19:42.369508+0000","last_undegraded":"2024-09-19T07:19:42.369508+0000","last_fullsized":"2024-09-19T07:19:42.369508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:22.190763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349468+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:19:41.349468+0000","last_peered":"2024-09-19T07:19:41.349468+0000","last_clean":"2024-09-19T07:19:41.349468+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:19:41.349468+0000","last_undegraded":"2024-09-19T07:19:41.349468+0000","last_fullsized":"2024-09-19T07:19:41.349468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353302+0000","last_change":"2024-09-19T07:19:39.437656+0000","last_active":"2024-09-19T07:19:41.353302+0000","last_peered":"2024-09-19T07:19:41.353302+0000","last_clean":"2024-09-19T07:19:41.353302+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:19:41.353302+0000","last_undegraded":"2024-09-19T07:19:41.353302+0000","last_fullsized":"2024-09-19T07:19:41.353302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:41.468096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.347816+0000","last_change":"2024-09-19T07:19:40.379289+0000","last_active":"2024-09-19T07:19:41.347816+0000","last_peered":"2024-09-19T07:19:41.347816+0000","last_clean":"2024-09-19T07:19:41.347816+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:19:41.347816+0000","last_undegraded":"2024-09-19T07:19:41.347816+0000","last_fullsized":"2024-09-19T07:19:41.347816+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:20.399106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368075+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:19:42.368075+0000","last_peered":"2024-09-19T07:19:42.368075+0000","last_clean":"2024-09-19T07:19:42.368075+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:19:42.368075+0000","last_undegraded":"2024-09-19T07:19:42.368075+0000","last_fullsized":"2024-09-19T07:19:42.368075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368710+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:19:42.368710+0000","last_peered":"2024-09-19T07:19:42.368710+0000","last_clean":"2024-09-19T07:19:42.368710+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:19:42.368710+0000","last_undegraded":"2024-09-19T07:19:42.368710+0000","last_fullsized":"2024-09-19T07:19:42.368710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366459+0000","last_change":"2024-09-19T07:19:39.413589+0000","last_active":"2024-09-19T07:19:42.366459+0000","last_peered":"2024-09-19T07:19:42.366459+0000","last_clean":"2024-09-19T07:19:42.366459+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:19:42.366459+0000","last_undegraded":"2024-09-19T07:19:42.366459+0000","last_fullsized":"2024-09-19T07:19:42.366459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370515+0000","last_change":"2024-09-19T07:19:40.387159+0000","last_active":"2024-09-19T07:19:42.370515+0000","last_peered":"2024-09-19T07:19:42.370515+0000","last_clean":"2024-09-19T07:19:42.370515+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:19:42.370515+0000","last_undegraded":"2024-09-19T07:19:42.370515+0000","last_fullsized":"2024-09-19T07:19:42.370515+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366844+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:19:42.366844+0000","last_peered":"2024-09-19T07:19:42.366844+0000","last_clean":"2024-09-19T07:19:42.366844+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:19:42.366844+0000","last_undegraded":"2024-09-19T07:19:42.366844+0000","last_fullsized":"2024-09-19T07:19:42.366844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.347952+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:19:41.347952+0000","last_peered":"2024-09-19T07:19:41.347952+0000","last_clean":"2024-09-19T07:19:41.347952+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:19:41.347952+0000","last_undegraded":"2024-09-19T07:19:41.347952+0000","last_fullsized":"2024-09-19T07:19:41.347952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371002+0000","last_change":"2024-09-19T07:19:39.406110+0000","last_active":"2024-09-19T07:19:42.371002+0000","last_peered":"2024-09-19T07:19:42.371002+0000","last_clean":"2024-09-19T07:19:42.371002+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:19:42.371002+0000","last_undegraded":"2024-09-19T07:19:42.371002+0000","last_fullsized":"2024-09-19T07:19:42.371002+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:10:05.224225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355606+0000","last_change":"2024-09-19T07:19:40.365764+0000","last_active":"2024-09-19T07:19:41.355606+0000","last_peered":"2024-09-19T07:19:41.355606+0000","last_clean":"2024-09-19T07:19:41.355606+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:19:41.355606+0000","last_undegraded":"2024-09-19T07:19:41.355606+0000","last_fullsized":"2024-09-19T07:19:41.355606+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:04:03.242292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352283+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:19:41.352283+0000","last_peered":"2024-09-19T07:19:41.352283+0000","last_clean":"2024-09-19T07:19:41.352283+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:19:41.352283+0000","last_undegraded":"2024-09-19T07:19:41.352283+0000","last_fullsized":"2024-09-19T07:19:41.352283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353746+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:19:41.353746+0000","last_peered":"2024-09-19T07:19:41.353746+0000","last_clean":"2024-09-19T07:19:41.353746+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:19:41.353746+0000","last_undegraded":"2024-09-19T07:19:41.353746+0000","last_fullsized":"2024-09-19T07:19:41.353746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370756+0000","last_change":"2024-09-19T07:19:39.429768+0000","last_active":"2024-09-19T07:19:42.370756+0000","last_peered":"2024-09-19T07:19:42.370756+0000","last_clean":"2024-09-19T07:19:42.370756+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:19:42.370756+0000","last_undegraded":"2024-09-19T07:19:42.370756+0000","last_fullsized":"2024-09-19T07:19:42.370756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:47.503631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352914+0000","last_change":"2024-09-19T07:19:40.373542+0000","last_active":"2024-09-19T07:19:41.352914+0000","last_peered":"2024-09-19T07:19:41.352914+0000","last_clean":"2024-09-19T07:19:41.352914+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:19:41.352914+0000","last_undegraded":"2024-09-19T07:19:41.352914+0000","last_fullsized":"2024-09-19T07:19:41.352914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:14.326662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366794+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:19:42.366794+0000","last_peered":"2024-09-19T07:19:42.366794+0000","last_clean":"2024-09-19T07:19:42.366794+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:19:42.366794+0000","last_undegraded":"2024-09-19T07:19:42.366794+0000","last_fullsized":"2024-09-19T07:19:42.366794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354258+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:19:41.354258+0000","last_peered":"2024-09-19T07:19:41.354258+0000","last_clean":"2024-09-19T07:19:41.354258+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:19:41.354258+0000","last_undegraded":"2024-09-19T07:19:41.354258+0000","last_fullsized":"2024-09-19T07:19:41.354258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"25'8","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:20:00.218770+0000","last_change":"2024-09-19T07:19:39.411074+0000","last_active":"2024-09-19T07:20:00.218770+0000","last_peered":"2024-09-19T07:20:00.218770+0000","last_clean":"2024-09-19T07:20:00.218770+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:20:00.218770+0000","last_undegraded":"2024-09-19T07:20:00.218770+0000","last_fullsized":"2024-09-19T07:20:00.218770+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:53.992787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2055,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370720+0000","last_change":"2024-09-19T07:19:40.364814+0000","last_active":"2024-09-19T07:19:42.370720+0000","last_peered":"2024-09-19T07:19:42.370720+0000","last_clean":"2024-09-19T07:19:42.370720+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:19:42.370720+0000","last_undegraded":"2024-09-19T07:19:42.370720+0000","last_fullsized":"2024-09-19T07:19:42.370720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366287+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:19:42.366287+0000","last_peered":"2024-09-19T07:19:42.366287+0000","last_clean":"2024-09-19T07:19:42.366287+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:19:42.366287+0000","last_undegraded":"2024-09-19T07:19:42.366287+0000","last_fullsized":"2024-09-19T07:19:42.366287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:57.131912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368746+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:19:42.368746+0000","last_peered":"2024-09-19T07:19:42.368746+0000","last_clean":"2024-09-19T07:19:42.368746+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:19:42.368746+0000","last_undegraded":"2024-09-19T07:19:42.368746+0000","last_fullsized":"2024-09-19T07:19:42.368746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:23.367365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"24'3","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355049+0000","last_change":"2024-09-19T07:19:39.431616+0000","last_active":"2024-09-19T07:19:41.355049+0000","last_peered":"2024-09-19T07:19:41.355049+0000","last_clean":"2024-09-19T07:19:41.355049+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:19:41.355049+0000","last_undegraded":"2024-09-19T07:19:41.355049+0000","last_fullsized":"2024-09-19T07:19:41.355049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:18.149581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370545+0000","last_change":"2024-09-19T07:19:40.381443+0000","last_active":"2024-09-19T07:19:42.370545+0000","last_peered":"2024-09-19T07:19:42.370545+0000","last_clean":"2024-09-19T07:19:42.370545+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:19:42.370545+0000","last_undegraded":"2024-09-19T07:19:42.370545+0000","last_fullsized":"2024-09-19T07:19:42.370545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366843+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:19:42.366843+0000","last_peered":"2024-09-19T07:19:42.366843+0000","last_clean":"2024-09-19T07:19:42.366843+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:19:42.366843+0000","last_undegraded":"2024-09-19T07:19:42.366843+0000","last_fullsized":"2024-09-19T07:19:42.366843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:57.218063+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:19:57.218063+0000","last_peered":"2024-09-19T07:19:57.218063+0000","last_clean":"2024-09-19T07:19:57.218063+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:19:57.218063+0000","last_undegraded":"2024-09-19T07:19:57.218063+0000","last_fullsized":"2024-09-19T07:19:57.218063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2055,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367083+0000","last_change":"2024-09-19T07:19:39.423972+0000","last_active":"2024-09-19T07:19:42.367083+0000","last_peered":"2024-09-19T07:19:42.367083+0000","last_clean":"2024-09-19T07:19:42.367083+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:19:42.367083+0000","last_undegraded":"2024-09-19T07:19:42.367083+0000","last_fullsized":"2024-09-19T07:19:42.367083+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:45.387040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367684+0000","last_change":"2024-09-19T07:19:40.382092+0000","last_active":"2024-09-19T07:19:42.367684+0000","last_peered":"2024-09-19T07:19:42.367684+0000","last_clean":"2024-09-19T07:19:42.367684+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:19:42.367684+0000","last_undegraded":"2024-09-19T07:19:42.367684+0000","last_fullsized":"2024-09-19T07:19:42.367684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368384+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:19:42.368384+0000","last_peered":"2024-09-19T07:19:42.368384+0000","last_clean":"2024-09-19T07:19:42.368384+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:19:42.368384+0000","last_undegraded":"2024-09-19T07:19:42.368384+0000","last_fullsized":"2024-09-19T07:19:42.368384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370238+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:19:42.370238+0000","last_peered":"2024-09-19T07:19:42.370238+0000","last_clean":"2024-09-19T07:19:42.370238+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:19:42.370238+0000","last_undegraded":"2024-09-19T07:19:42.370238+0000","last_fullsized":"2024-09-19T07:19:42.370238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366357+0000","last_change":"2024-09-19T07:19:39.424879+0000","last_active":"2024-09-19T07:19:42.366357+0000","last_peered":"2024-09-19T07:19:42.366357+0000","last_clean":"2024-09-19T07:19:42.366357+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:19:42.366357+0000","last_undegraded":"2024-09-19T07:19:42.366357+0000","last_fullsized":"2024-09-19T07:19:42.366357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:30.021543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353343+0000","last_change":"2024-09-19T07:19:40.384136+0000","last_active":"2024-09-19T07:19:41.353343+0000","last_peered":"2024-09-19T07:19:41.353343+0000","last_clean":"2024-09-19T07:19:41.353343+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:19:41.353343+0000","last_undegraded":"2024-09-19T07:19:41.353343+0000","last_fullsized":"2024-09-19T07:19:41.353343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369727+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:19:42.369727+0000","last_peered":"2024-09-19T07:19:42.369727+0000","last_clean":"2024-09-19T07:19:42.369727+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:19:42.369727+0000","last_undegraded":"2024-09-19T07:19:42.369727+0000","last_fullsized":"2024-09-19T07:19:42.369727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:48.308137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355901+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:19:41.355901+0000","last_peered":"2024-09-19T07:19:41.355901+0000","last_clean":"2024-09-19T07:19:41.355901+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:19:41.355901+0000","last_undegraded":"2024-09-19T07:19:41.355901+0000","last_fullsized":"2024-09-19T07:19:41.355901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"24'8","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365988+0000","last_change":"2024-09-19T07:19:39.410445+0000","last_active":"2024-09-19T07:19:42.365988+0000","last_peered":"2024-09-19T07:19:42.365988+0000","last_clean":"2024-09-19T07:19:42.365988+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:19:42.365988+0000","last_undegraded":"2024-09-19T07:19:42.365988+0000","last_fullsized":"2024-09-19T07:19:42.365988+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:27.020327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365513+0000","last_change":"2024-09-19T07:19:40.389499+0000","last_active":"2024-09-19T07:19:42.365513+0000","last_peered":"2024-09-19T07:19:42.365513+0000","last_clean":"2024-09-19T07:19:42.365513+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:19:42.365513+0000","last_undegraded":"2024-09-19T07:19:42.365513+0000","last_fullsized":"2024-09-19T07:19:42.365513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.520872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355599+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:19:41.355599+0000","last_peered":"2024-09-19T07:19:41.355599+0000","last_clean":"2024-09-19T07:19:41.355599+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:19:41.355599+0000","last_undegraded":"2024-09-19T07:19:41.355599+0000","last_fullsized":"2024-09-19T07:19:41.355599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.357039+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:19:41.357039+0000","last_peered":"2024-09-19T07:19:41.357039+0000","last_clean":"2024-09-19T07:19:41.357039+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:19:41.357039+0000","last_undegraded":"2024-09-19T07:19:41.357039+0000","last_fullsized":"2024-09-19T07:19:41.357039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368271+0000","last_change":"2024-09-19T07:19:39.416943+0000","last_active":"2024-09-19T07:19:42.368271+0000","last_peered":"2024-09-19T07:19:42.368271+0000","last_clean":"2024-09-19T07:19:42.368271+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:19:42.368271+0000","last_undegraded":"2024-09-19T07:19:42.368271+0000","last_fullsized":"2024-09-19T07:19:42.368271+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:35.555826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356615+0000","last_change":"2024-09-19T07:19:40.394993+0000","last_active":"2024-09-19T07:19:41.356615+0000","last_peered":"2024-09-19T07:19:41.356615+0000","last_clean":"2024-09-19T07:19:41.356615+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:19:41.356615+0000","last_undegraded":"2024-09-19T07:19:41.356615+0000","last_fullsized":"2024-09-19T07:19:41.356615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370672+0000","last_change":"2024-09-19T07:19:39.418885+0000","last_active":"2024-09-19T07:19:42.370672+0000","last_peered":"2024-09-19T07:19:42.370672+0000","last_clean":"2024-09-19T07:19:42.370672+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:19:42.370672+0000","last_undegraded":"2024-09-19T07:19:42.370672+0000","last_fullsized":"2024-09-19T07:19:42.370672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:35:45.790411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366817+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:19:42.366817+0000","last_peered":"2024-09-19T07:19:42.366817+0000","last_clean":"2024-09-19T07:19:42.366817+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:19:42.366817+0000","last_undegraded":"2024-09-19T07:19:42.366817+0000","last_fullsized":"2024-09-19T07:19:42.366817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353835+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:19:41.353835+0000","last_peered":"2024-09-19T07:19:41.353835+0000","last_clean":"2024-09-19T07:19:41.353835+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:19:41.353835+0000","last_undegraded":"2024-09-19T07:19:41.353835+0000","last_fullsized":"2024-09-19T07:19:41.353835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370413+0000","last_change":"2024-09-19T07:19:40.374048+0000","last_active":"2024-09-19T07:19:42.370413+0000","last_peered":"2024-09-19T07:19:42.370413+0000","last_clean":"2024-09-19T07:19:42.370413+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:19:42.370413+0000","last_undegraded":"2024-09-19T07:19:42.370413+0000","last_fullsized":"2024-09-19T07:19:42.370413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:40.084827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"24'4","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356842+0000","last_change":"2024-09-19T07:19:39.432426+0000","last_active":"2024-09-19T07:19:41.356842+0000","last_peered":"2024-09-19T07:19:41.356842+0000","last_clean":"2024-09-19T07:19:41.356842+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:19:41.356842+0000","last_undegraded":"2024-09-19T07:19:41.356842+0000","last_fullsized":"2024-09-19T07:19:41.356842+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366831+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:19:42.366831+0000","last_peered":"2024-09-19T07:19:42.366831+0000","last_clean":"2024-09-19T07:19:42.366831+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:19:42.366831+0000","last_undegraded":"2024-09-19T07:19:42.366831+0000","last_fullsized":"2024-09-19T07:19:42.366831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368900+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:19:42.368900+0000","last_peered":"2024-09-19T07:19:42.368900+0000","last_clean":"2024-09-19T07:19:42.368900+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:19:42.368900+0000","last_undegraded":"2024-09-19T07:19:42.368900+0000","last_fullsized":"2024-09-19T07:19:42.368900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369632+0000","last_change":"2024-09-19T07:19:40.384934+0000","last_active":"2024-09-19T07:19:42.369632+0000","last_peered":"2024-09-19T07:19:42.369632+0000","last_clean":"2024-09-19T07:19:42.369632+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:19:42.369632+0000","last_undegraded":"2024-09-19T07:19:42.369632+0000","last_fullsized":"2024-09-19T07:19:42.369632+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353660+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:19:41.353660+0000","last_peered":"2024-09-19T07:19:41.353660+0000","last_clean":"2024-09-19T07:19:41.353660+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:19:41.353660+0000","last_undegraded":"2024-09-19T07:19:41.353660+0000","last_fullsized":"2024-09-19T07:19:41.353660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368442+0000","last_change":"2024-09-19T07:19:32.289374+0000","last_active":"2024-09-19T07:19:42.368442+0000","last_peered":"2024-09-19T07:19:42.368442+0000","last_clean":"2024-09-19T07:19:42.368442+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:19:42.368442+0000","last_undegraded":"2024-09-19T07:19:42.368442+0000","last_fullsized":"2024-09-19T07:19:42.368442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348715+0000","last_change":"2024-09-19T07:19:39.410582+0000","last_active":"2024-09-19T07:19:41.348715+0000","last_peered":"2024-09-19T07:19:41.348715+0000","last_clean":"2024-09-19T07:19:41.348715+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:19:41.348715+0000","last_undegraded":"2024-09-19T07:19:41.348715+0000","last_fullsized":"2024-09-19T07:19:41.348715+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:41:40.592907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367452+0000","last_change":"2024-09-19T07:19:40.393950+0000","last_active":"2024-09-19T07:19:42.367452+0000","last_peered":"2024-09-19T07:19:42.367452+0000","last_clean":"2024-09-19T07:19:42.367452+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:19:42.367452+0000","last_undegraded":"2024-09-19T07:19:42.367452+0000","last_fullsized":"2024-09-19T07:19:42.367452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371158+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:19:42.371158+0000","last_peered":"2024-09-19T07:19:42.371158+0000","last_clean":"2024-09-19T07:19:42.371158+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:19:42.371158+0000","last_undegraded":"2024-09-19T07:19:42.371158+0000","last_fullsized":"2024-09-19T07:19:42.371158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366780+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:19:42.366780+0000","last_peered":"2024-09-19T07:19:42.366780+0000","last_clean":"2024-09-19T07:19:42.366780+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:19:42.366780+0000","last_undegraded":"2024-09-19T07:19:42.366780+0000","last_fullsized":"2024-09-19T07:19:42.366780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371594+0000","last_change":"2024-09-19T07:19:39.410662+0000","last_active":"2024-09-19T07:19:42.371594+0000","last_peered":"2024-09-19T07:19:42.371594+0000","last_clean":"2024-09-19T07:19:42.371594+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:42.371594+0000","last_undegraded":"2024-09-19T07:19:42.371594+0000","last_fullsized":"2024-09-19T07:19:42.371594+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:19.196074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372306+0000","last_change":"2024-09-19T07:19:40.385581+0000","last_active":"2024-09-19T07:19:42.372306+0000","last_peered":"2024-09-19T07:19:42.372306+0000","last_clean":"2024-09-19T07:19:42.372306+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:19:42.372306+0000","last_undegraded":"2024-09-19T07:19:42.372306+0000","last_fullsized":"2024-09-19T07:19:42.372306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:14:13.749729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353359+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:19:41.353359+0000","last_peered":"2024-09-19T07:19:41.353359+0000","last_clean":"2024-09-19T07:19:41.353359+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:19:41.353359+0000","last_undegraded":"2024-09-19T07:19:41.353359+0000","last_fullsized":"2024-09-19T07:19:41.353359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369574+0000","last_change":"2024-09-19T07:19:40.377608+0000","last_active":"2024-09-19T07:19:42.369574+0000","last_peered":"2024-09-19T07:19:42.369574+0000","last_clean":"2024-09-19T07:19:42.369574+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:19:42.369574+0000","last_undegraded":"2024-09-19T07:19:42.369574+0000","last_fullsized":"2024-09-19T07:19:42.369574+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348053+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:19:41.348053+0000","last_peered":"2024-09-19T07:19:41.348053+0000","last_clean":"2024-09-19T07:19:41.348053+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:19:41.348053+0000","last_undegraded":"2024-09-19T07:19:41.348053+0000","last_fullsized":"2024-09-19T07:19:41.348053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353277+0000","last_change":"2024-09-19T07:19:40.381262+0000","last_active":"2024-09-19T07:19:41.353277+0000","last_peered":"2024-09-19T07:19:41.353277+0000","last_clean":"2024-09-19T07:19:41.353277+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:19:41.353277+0000","last_undegraded":"2024-09-19T07:19:41.353277+0000","last_fullsized":"2024-09-19T07:19:41.353277+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:34.425472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354018+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:19:41.354018+0000","last_peered":"2024-09-19T07:19:41.354018+0000","last_clean":"2024-09-19T07:19:41.354018+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:19:41.354018+0000","last_undegraded":"2024-09-19T07:19:41.354018+0000","last_fullsized":"2024-09-19T07:19:41.354018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369264+0000","last_change":"2024-09-19T07:19:40.386432+0000","last_active":"2024-09-19T07:19:42.369264+0000","last_peered":"2024-09-19T07:19:42.369264+0000","last_clean":"2024-09-19T07:19:42.369264+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:19:42.369264+0000","last_undegraded":"2024-09-19T07:19:42.369264+0000","last_fullsized":"2024-09-19T07:19:42.369264+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368214+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:19:42.368214+0000","last_peered":"2024-09-19T07:19:42.368214+0000","last_clean":"2024-09-19T07:19:42.368214+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:19:42.368214+0000","last_undegraded":"2024-09-19T07:19:42.368214+0000","last_fullsized":"2024-09-19T07:19:42.368214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365212+0000","last_change":"2024-09-19T07:19:40.378630+0000","last_active":"2024-09-19T07:19:42.365212+0000","last_peered":"2024-09-19T07:19:42.365212+0000","last_clean":"2024-09-19T07:19:42.365212+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:19:42.365212+0000","last_undegraded":"2024-09-19T07:19:42.365212+0000","last_fullsized":"2024-09-19T07:19:42.365212+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371622+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:19:42.371622+0000","last_peered":"2024-09-19T07:19:42.371622+0000","last_clean":"2024-09-19T07:19:42.371622+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:19:42.371622+0000","last_undegraded":"2024-09-19T07:19:42.371622+0000","last_fullsized":"2024-09-19T07:19:42.371622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365853+0000","last_change":"2024-09-19T07:19:40.381365+0000","last_active":"2024-09-19T07:19:42.365853+0000","last_peered":"2024-09-19T07:19:42.365853+0000","last_clean":"2024-09-19T07:19:42.365853+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:19:42.365853+0000","last_undegraded":"2024-09-19T07:19:42.365853+0000","last_fullsized":"2024-09-19T07:19:42.365853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370004+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:19:42.370004+0000","last_peered":"2024-09-19T07:19:42.370004+0000","last_clean":"2024-09-19T07:19:42.370004+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:19:42.370004+0000","last_undegraded":"2024-09-19T07:19:42.370004+0000","last_fullsized":"2024-09-19T07:19:42.370004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365148+0000","last_change":"2024-09-19T07:19:40.387116+0000","last_active":"2024-09-19T07:19:42.365148+0000","last_peered":"2024-09-19T07:19:42.365148+0000","last_clean":"2024-09-19T07:19:42.365148+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:19:42.365148+0000","last_undegraded":"2024-09-19T07:19:42.365148+0000","last_fullsized":"2024-09-19T07:19:42.365148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356605+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:19:41.356605+0000","last_peered":"2024-09-19T07:19:41.356605+0000","last_clean":"2024-09-19T07:19:41.356605+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:19:41.356605+0000","last_undegraded":"2024-09-19T07:19:41.356605+0000","last_fullsized":"2024-09-19T07:19:41.356605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352940+0000","last_change":"2024-09-19T07:19:40.357629+0000","last_active":"2024-09-19T07:19:41.352940+0000","last_peered":"2024-09-19T07:19:41.352940+0000","last_clean":"2024-09-19T07:19:41.352940+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:19:41.352940+0000","last_undegraded":"2024-09-19T07:19:41.352940+0000","last_fullsized":"2024-09-19T07:19:41.352940+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371018+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:19:42.371018+0000","last_peered":"2024-09-19T07:19:42.371018+0000","last_clean":"2024-09-19T07:19:42.371018+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:19:42.371018+0000","last_undegraded":"2024-09-19T07:19:42.371018+0000","last_fullsized":"2024-09-19T07:19:42.371018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371827+0000","last_change":"2024-09-19T07:19:40.388597+0000","last_active":"2024-09-19T07:19:42.371827+0000","last_peered":"2024-09-19T07:19:42.371827+0000","last_clean":"2024-09-19T07:19:42.371827+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:19:42.371827+0000","last_undegraded":"2024-09-19T07:19:42.371827+0000","last_fullsized":"2024-09-19T07:19:42.371827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:34:18.402016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356021+0000","last_change":"2024-09-19T07:19:32.273635+0000","last_active":"2024-09-19T07:19:41.356021+0000","last_peered":"2024-09-19T07:19:41.356021+0000","last_clean":"2024-09-19T07:19:41.356021+0000","last_became_active":"2024-09-19T07:19:32.271748+0000","last_became_peered":"2024-09-19T07:19:32.271748+0000","last_unstale":"2024-09-19T07:19:41.356021+0000","last_undegraded":"2024-09-19T07:19:41.356021+0000","last_fullsized":"2024-09-19T07:19:41.356021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:49:41.724626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369047+0000","last_change":"2024-09-19T07:19:40.383779+0000","last_active":"2024-09-19T07:19:42.369047+0000","last_peered":"2024-09-19T07:19:42.369047+0000","last_clean":"2024-09-19T07:19:42.369047+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:19:42.369047+0000","last_undegraded":"2024-09-19T07:19:42.369047+0000","last_fullsized":"2024-09-19T07:19:42.369047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368684+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:19:42.368684+0000","last_peered":"2024-09-19T07:19:42.368684+0000","last_clean":"2024-09-19T07:19:42.368684+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:19:42.368684+0000","last_undegraded":"2024-09-19T07:19:42.368684+0000","last_fullsized":"2024-09-19T07:19:42.368684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348737+0000","last_change":"2024-09-19T07:19:40.376342+0000","last_active":"2024-09-19T07:19:41.348737+0000","last_peered":"2024-09-19T07:19:41.348737+0000","last_clean":"2024-09-19T07:19:41.348737+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:19:41.348737+0000","last_undegraded":"2024-09-19T07:19:41.348737+0000","last_fullsized":"2024-09-19T07:19:41.348737+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:22.997744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353771+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:19:41.353771+0000","last_peered":"2024-09-19T07:19:41.353771+0000","last_clean":"2024-09-19T07:19:41.353771+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:19:41.353771+0000","last_undegraded":"2024-09-19T07:19:41.353771+0000","last_fullsized":"2024-09-19T07:19:41.353771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371199+0000","last_change":"2024-09-19T07:19:40.361504+0000","last_active":"2024-09-19T07:19:42.371199+0000","last_peered":"2024-09-19T07:19:42.371199+0000","last_clean":"2024-09-19T07:19:42.371199+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:19:42.371199+0000","last_undegraded":"2024-09-19T07:19:42.371199+0000","last_fullsized":"2024-09-19T07:19:42.371199+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:30:31.110669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367582+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:19:42.367582+0000","last_peered":"2024-09-19T07:19:42.367582+0000","last_clean":"2024-09-19T07:19:42.367582+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:19:42.367582+0000","last_undegraded":"2024-09-19T07:19:42.367582+0000","last_fullsized":"2024-09-19T07:19:42.367582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:19.033192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365909+0000","last_change":"2024-09-19T07:19:40.390205+0000","last_active":"2024-09-19T07:19:42.365909+0000","last_peered":"2024-09-19T07:19:42.365909+0000","last_clean":"2024-09-19T07:19:42.365909+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:19:42.365909+0000","last_undegraded":"2024-09-19T07:19:42.365909+0000","last_fullsized":"2024-09-19T07:19:42.365909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354180+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:19:41.354180+0000","last_peered":"2024-09-19T07:19:41.354180+0000","last_clean":"2024-09-19T07:19:41.354180+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:19:41.354180+0000","last_undegraded":"2024-09-19T07:19:41.354180+0000","last_fullsized":"2024-09-19T07:19:41.354180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365447+0000","last_change":"2024-09-19T07:19:40.390559+0000","last_active":"2024-09-19T07:19:42.365447+0000","last_peered":"2024-09-19T07:19:42.365447+0000","last_clean":"2024-09-19T07:19:42.365447+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:19:42.365447+0000","last_undegraded":"2024-09-19T07:19:42.365447+0000","last_fullsized":"2024-09-19T07:19:42.365447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367349+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:19:42.367349+0000","last_peered":"2024-09-19T07:19:42.367349+0000","last_clean":"2024-09-19T07:19:42.367349+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:19:42.367349+0000","last_undegraded":"2024-09-19T07:19:42.367349+0000","last_fullsized":"2024-09-19T07:19:42.367349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352820+0000","last_change":"2024-09-19T07:19:40.358433+0000","last_active":"2024-09-19T07:19:41.352820+0000","last_peered":"2024-09-19T07:19:41.352820+0000","last_clean":"2024-09-19T07:19:41.352820+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:19:41.352820+0000","last_undegraded":"2024-09-19T07:19:41.352820+0000","last_fullsized":"2024-09-19T07:19:41.352820+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:10:25.226677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349337+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:19:41.349337+0000","last_peered":"2024-09-19T07:19:41.349337+0000","last_clean":"2024-09-19T07:19:41.349337+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:19:41.349337+0000","last_undegraded":"2024-09-19T07:19:41.349337+0000","last_fullsized":"2024-09-19T07:19:41.349337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:51.710997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365454+0000","last_change":"2024-09-19T07:19:40.374374+0000","last_active":"2024-09-19T07:19:42.365454+0000","last_peered":"2024-09-19T07:19:42.365454+0000","last_clean":"2024-09-19T07:19:42.365454+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:19:42.365454+0000","last_undegraded":"2024-09-19T07:19:42.365454+0000","last_fullsized":"2024-09-19T07:19:42.365454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:34:05.875388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356952+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:19:41.356952+0000","last_peered":"2024-09-19T07:19:41.356952+0000","last_clean":"2024-09-19T07:19:41.356952+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:19:41.356952+0000","last_undegraded":"2024-09-19T07:19:41.356952+0000","last_fullsized":"2024-09-19T07:19:41.356952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368071+0000","last_change":"2024-09-19T07:19:40.387341+0000","last_active":"2024-09-19T07:19:42.368071+0000","last_peered":"2024-09-19T07:19:42.368071+0000","last_clean":"2024-09-19T07:19:42.368071+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:19:42.368071+0000","last_undegraded":"2024-09-19T07:19:42.368071+0000","last_fullsized":"2024-09-19T07:19:42.368071+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:56:40.928844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353563+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:19:41.353563+0000","last_peered":"2024-09-19T07:19:41.353563+0000","last_clean":"2024-09-19T07:19:41.353563+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:19:41.353563+0000","last_undegraded":"2024-09-19T07:19:41.353563+0000","last_fullsized":"2024-09-19T07:19:41.353563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368148+0000","last_change":"2024-09-19T07:19:40.388484+0000","last_active":"2024-09-19T07:19:42.368148+0000","last_peered":"2024-09-19T07:19:42.368148+0000","last_clean":"2024-09-19T07:19:42.368148+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:19:42.368148+0000","last_undegraded":"2024-09-19T07:19:42.368148+0000","last_fullsized":"2024-09-19T07:19:42.368148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366974+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:19:42.366974+0000","last_peered":"2024-09-19T07:19:42.366974+0000","last_clean":"2024-09-19T07:19:42.366974+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:19:42.366974+0000","last_undegraded":"2024-09-19T07:19:42.366974+0000","last_fullsized":"2024-09-19T07:19:42.366974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352836+0000","last_change":"2024-09-19T07:19:40.367578+0000","last_active":"2024-09-19T07:19:41.352836+0000","last_peered":"2024-09-19T07:19:41.352836+0000","last_clean":"2024-09-19T07:19:41.352836+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:19:41.352836+0000","last_undegraded":"2024-09-19T07:19:41.352836+0000","last_fullsized":"2024-09-19T07:19:41.352836+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:00:25.357854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366048+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:19:42.366048+0000","last_peered":"2024-09-19T07:19:42.366048+0000","last_clean":"2024-09-19T07:19:42.366048+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:19:42.366048+0000","last_undegraded":"2024-09-19T07:19:42.366048+0000","last_fullsized":"2024-09-19T07:19:42.366048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:43.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366598+0000","last_change":"2024-09-19T07:19:40.385641+0000","last_active":"2024-09-19T07:19:42.366598+0000","last_peered":"2024-09-19T07:19:42.366598+0000","last_clean":"2024-09-19T07:19:42.366598+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:19:42.366598+0000","last_undegraded":"2024-09-19T07:19:42.366598+0000","last_fullsized":"2024-09-19T07:19:42.366598+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370694+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:19:42.370694+0000","last_peered":"2024-09-19T07:19:42.370694+0000","last_clean":"2024-09-19T07:19:42.370694+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:19:42.370694+0000","last_undegraded":"2024-09-19T07:19:42.370694+0000","last_fullsized":"2024-09-19T07:19:42.370694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369737+0000","last_change":"2024-09-19T07:19:40.367101+0000","last_active":"2024-09-19T07:19:42.369737+0000","last_peered":"2024-09-19T07:19:42.369737+0000","last_clean":"2024-09-19T07:19:42.369737+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:19:42.369737+0000","last_undegraded":"2024-09-19T07:19:42.369737+0000","last_fullsized":"2024-09-19T07:19:42.369737+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368379+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:19:42.368379+0000","last_peered":"2024-09-19T07:19:42.368379+0000","last_clean":"2024-09-19T07:19:42.368379+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:19:42.368379+0000","last_undegraded":"2024-09-19T07:19:42.368379+0000","last_fullsized":"2024-09-19T07:19:42.368379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348413+0000","last_change":"2024-09-19T07:19:40.384212+0000","last_active":"2024-09-19T07:19:41.348413+0000","last_peered":"2024-09-19T07:19:41.348413+0000","last_clean":"2024-09-19T07:19:41.348413+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:19:41.348413+0000","last_undegraded":"2024-09-19T07:19:41.348413+0000","last_fullsized":"2024-09-19T07:19:41.348413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:57.724404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355987+0000","last_change":"2024-09-19T07:19:32.299956+0000","last_active":"2024-09-19T07:19:41.355987+0000","last_peered":"2024-09-19T07:19:41.355987+0000","last_clean":"2024-09-19T07:19:41.355987+0000","last_became_active":"2024-09-19T07:19:32.299168+0000","last_became_peered":"2024-09-19T07:19:32.299168+0000","last_unstale":"2024-09-19T07:19:41.355987+0000","last_undegraded":"2024-09-19T07:19:41.355987+0000","last_fullsized":"2024-09-19T07:19:41.355987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366761+0000","last_change":"2024-09-19T07:19:40.387401+0000","last_active":"2024-09-19T07:19:42.366761+0000","last_peered":"2024-09-19T07:19:42.366761+0000","last_clean":"2024-09-19T07:19:42.366761+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:19:42.366761+0000","last_undegraded":"2024-09-19T07:19:42.366761+0000","last_fullsized":"2024-09-19T07:19:42.366761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356423+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:19:41.356423+0000","last_peered":"2024-09-19T07:19:41.356423+0000","last_clean":"2024-09-19T07:19:41.356423+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:19:41.356423+0000","last_undegraded":"2024-09-19T07:19:41.356423+0000","last_fullsized":"2024-09-19T07:19:41.356423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352920+0000","last_change":"2024-09-19T07:19:40.361430+0000","last_active":"2024-09-19T07:19:41.352920+0000","last_peered":"2024-09-19T07:19:41.352920+0000","last_clean":"2024-09-19T07:19:41.352920+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:19:41.352920+0000","last_undegraded":"2024-09-19T07:19:41.352920+0000","last_fullsized":"2024-09-19T07:19:41.352920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:23:05.185943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369329+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:19:42.369329+0000","last_peered":"2024-09-19T07:19:42.369329+0000","last_clean":"2024-09-19T07:19:42.369329+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:19:42.369329+0000","last_undegraded":"2024-09-19T07:19:42.369329+0000","last_fullsized":"2024-09-19T07:19:42.369329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371825+0000","last_change":"2024-09-19T07:19:40.378172+0000","last_active":"2024-09-19T07:19:42.371825+0000","last_peered":"2024-09-19T07:19:42.371825+0000","last_clean":"2024-09-19T07:19:42.371825+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:19:42.371825+0000","last_undegraded":"2024-09-19T07:19:42.371825+0000","last_fullsized":"2024-09-19T07:19:42.371825+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:52.321790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366422+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:19:42.366422+0000","last_peered":"2024-09-19T07:19:42.366422+0000","last_clean":"2024-09-19T07:19:42.366422+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:19:42.366422+0000","last_undegraded":"2024-09-19T07:19:42.366422+0000","last_fullsized":"2024-09-19T07:19:42.366422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356589+0000","last_change":"2024-09-19T07:19:40.373909+0000","last_active":"2024-09-19T07:19:41.356589+0000","last_peered":"2024-09-19T07:19:41.356589+0000","last_clean":"2024-09-19T07:19:41.356589+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:19:41.356589+0000","last_undegraded":"2024-09-19T07:19:41.356589+0000","last_fullsized":"2024-09-19T07:19:41.356589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366411+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:19:42.366411+0000","last_peered":"2024-09-19T07:19:42.366411+0000","last_clean":"2024-09-19T07:19:42.366411+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:19:42.366411+0000","last_undegraded":"2024-09-19T07:19:42.366411+0000","last_fullsized":"2024-09-19T07:19:42.366411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367511+0000","last_change":"2024-09-19T07:19:40.396146+0000","last_active":"2024-09-19T07:19:42.367511+0000","last_peered":"2024-09-19T07:19:42.367511+0000","last_clean":"2024-09-19T07:19:42.367511+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:19:42.367511+0000","last_undegraded":"2024-09-19T07:19:42.367511+0000","last_fullsized":"2024-09-19T07:19:42.367511+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353532+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:19:41.353532+0000","last_peered":"2024-09-19T07:19:41.353532+0000","last_clean":"2024-09-19T07:19:41.353532+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:19:41.353532+0000","last_undegraded":"2024-09-19T07:19:41.353532+0000","last_fullsized":"2024-09-19T07:19:41.353532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355009+0000","last_change":"2024-09-19T07:19:40.373552+0000","last_active":"2024-09-19T07:19:41.355009+0000","last_peered":"2024-09-19T07:19:41.355009+0000","last_clean":"2024-09-19T07:19:41.355009+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:19:41.355009+0000","last_undegraded":"2024-09-19T07:19:41.355009+0000","last_fullsized":"2024-09-19T07:19:41.355009+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:23:33.629831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366537+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:19:42.366537+0000","last_peered":"2024-09-19T07:19:42.366537+0000","last_clean":"2024-09-19T07:19:42.366537+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:19:42.366537+0000","last_undegraded":"2024-09-19T07:19:42.366537+0000","last_fullsized":"2024-09-19T07:19:42.366537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367996+0000","last_change":"2024-09-19T07:19:40.356989+0000","last_active":"2024-09-19T07:19:42.367996+0000","last_peered":"2024-09-19T07:19:42.367996+0000","last_clean":"2024-09-19T07:19:42.367996+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:19:42.367996+0000","last_undegraded":"2024-09-19T07:19:42.367996+0000","last_fullsized":"2024-09-19T07:19:42.367996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353682+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:19:41.353682+0000","last_peered":"2024-09-19T07:19:41.353682+0000","last_clean":"2024-09-19T07:19:41.353682+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:19:41.353682+0000","last_undegraded":"2024-09-19T07:19:41.353682+0000","last_fullsized":"2024-09-19T07:19:41.353682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348160+0000","last_change":"2024-09-19T07:19:40.376487+0000","last_active":"2024-09-19T07:19:41.348160+0000","last_peered":"2024-09-19T07:19:41.348160+0000","last_clean":"2024-09-19T07:19:41.348160+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:19:41.348160+0000","last_undegraded":"2024-09-19T07:19:41.348160+0000","last_fullsized":"2024-09-19T07:19:41.348160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:05.185451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369628+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:19:42.369628+0000","last_peered":"2024-09-19T07:19:42.369628+0000","last_clean":"2024-09-19T07:19:42.369628+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:19:42.369628+0000","last_undegraded":"2024-09-19T07:19:42.369628+0000","last_fullsized":"2024-09-19T07:19:42.369628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:25.667168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364909+0000","last_change":"2024-09-19T07:19:40.377604+0000","last_active":"2024-09-19T07:19:42.364909+0000","last_peered":"2024-09-19T07:19:42.364909+0000","last_clean":"2024-09-19T07:19:42.364909+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:19:42.364909+0000","last_undegraded":"2024-09-19T07:19:42.364909+0000","last_fullsized":"2024-09-19T07:19:42.364909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368557+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:19:42.368557+0000","last_peered":"2024-09-19T07:19:42.368557+0000","last_clean":"2024-09-19T07:19:42.368557+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:19:42.368557+0000","last_undegraded":"2024-09-19T07:19:42.368557+0000","last_fullsized":"2024-09-19T07:19:42.368557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365847+0000","last_change":"2024-09-19T07:19:40.377580+0000","last_active":"2024-09-19T07:19:42.365847+0000","last_peered":"2024-09-19T07:19:42.365847+0000","last_clean":"2024-09-19T07:19:42.365847+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:19:42.365847+0000","last_undegraded":"2024-09-19T07:19:42.365847+0000","last_fullsized":"2024-09-19T07:19:42.365847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369285+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:19:42.369285+0000","last_peered":"2024-09-19T07:19:42.369285+0000","last_clean":"2024-09-19T07:19:42.369285+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:19:42.369285+0000","last_undegraded":"2024-09-19T07:19:42.369285+0000","last_fullsized":"2024-09-19T07:19:42.369285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2853,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2853,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542156,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":864256,"data_stored":693530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":105161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":103252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":282624,"data_stored":102720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":106179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542157,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":102560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542156,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":104727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542155,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":868352,"data_stored":693496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:20:04.859 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-09-19T07:20:04.859 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:20:05.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec4b99c640 1 Processor -- start 2024-09-19T07:20:05.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec4b99c640 1 -- start start 2024-09-19T07:20:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec4b99c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec440c6120 0x7fec440c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec4b99c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fec441592f0 0x7fec44157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec4b99c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec44156200 con 0x7fec44155d70 2024-09-19T07:20:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec4b99c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fec440c9930 con 0x7fec441592f0 2024-09-19T07:20:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec4b99c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fec441564e0 con 0x7fec440c6120 2024-09-19T07:20:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec49f39640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fec441592f0 0x7fec44157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec49f39640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fec441592f0 0x7fec44157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49690/0 (socket says 172.21.15.83:49690) 2024-09-19T07:20:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec49f39640 1 -- 172.21.15.83:0/1359485934 learned_addr learned my addr 172.21.15.83:0/1359485934 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec49738640 1 --2- 172.21.15.83:0/1359485934 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec440c6120 0x7fec440c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec49f39640 1 -- 172.21.15.83:0/1359485934 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec440c6120 msgr2=0x7fec440c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec49f39640 1 --2- 172.21.15.83:0/1359485934 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec440c6120 0x7fec440c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec49f39640 1 -- 172.21.15.83:0/1359485934 >> v1:172.21.15.184:6789/0 conn(0x7fec44155d70 legacy=0x7fec441575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.010+0000 7fec49f39640 1 -- 172.21.15.83:0/1359485934 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec441567c0 con 0x7fec441592f0 2024-09-19T07:20:05.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec49f39640 1 --2- 172.21.15.83:0/1359485934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec441592f0 0x7fec44157ce0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009e50 tx=0x7fec3c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596b370d362b34b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:05.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec3bfff640 1 -- 172.21.15.83:0/1359485934 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c0a5700 con 0x7fec441592f0 2024-09-19T07:20:05.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec3bfff640 1 -- 172.21.15.83:0/1359485934 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec3c0a58a0 con 0x7fec441592f0 2024-09-19T07:20:05.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec3bfff640 1 -- 172.21.15.83:0/1359485934 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c0a5c70 con 0x7fec441592f0 2024-09-19T07:20:05.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec4b99c640 1 -- 172.21.15.83:0/1359485934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec441592f0 msgr2=0x7fec44157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec4b99c640 1 --2- 172.21.15.83:0/1359485934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec441592f0 0x7fec44157ce0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009e50 tx=0x7fec3c01ca30 comp rx=0 tx=0).stop 2024-09-19T07:20:05.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec4b99c640 1 -- 172.21.15.83:0/1359485934 shutdown_connections 2024-09-19T07:20:05.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec4b99c640 1 --2- 172.21.15.83:0/1359485934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec441592f0 0x7fec44157ce0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec4b99c640 1 --2- 172.21.15.83:0/1359485934 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec440c6120 0x7fec440c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec4b99c640 1 -- 172.21.15.83:0/1359485934 >> 172.21.15.83:0/1359485934 conn(0x7fec4406bc80 msgr2=0x7fec4406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:05.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec4b99c640 1 -- 172.21.15.83:0/1359485934 shutdown_connections 2024-09-19T07:20:05.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec4b99c640 1 -- 172.21.15.83:0/1359485934 wait complete. 2024-09-19T07:20:05.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec4b99c640 1 Processor -- start 2024-09-19T07:20:05.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.014+0000 7fec4b99c640 1 -- start start 2024-09-19T07:20:05.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec4b99c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec440c6120 0x7fec440aca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec4b99c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec441592f0 0x7fec440ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec4b99c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec4414f790 0x7fec440abae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec4b99c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fec44153860 con 0x7fec441592f0 2024-09-19T07:20:05.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec4b99c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fec44162ea0 con 0x7fec440c6120 2024-09-19T07:20:05.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec4b99c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fec440acfb0 con 0x7fec4414f790 2024-09-19T07:20:05.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec48f37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec441592f0 0x7fec440ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec48f37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec441592f0 0x7fec440ab5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49698/0 (socket says 172.21.15.83:49698) 2024-09-19T07:20:05.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec48f37640 1 -- 172.21.15.83:0/3486724149 learned_addr learned my addr 172.21.15.83:0/3486724149 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:05.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec49738640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec440c6120 0x7fec440aca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec49f39640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec4414f790 0x7fec440abae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec48f37640 1 -- 172.21.15.83:0/3486724149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec4414f790 msgr2=0x7fec440abae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec48f37640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec4414f790 0x7fec440abae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec48f37640 1 -- 172.21.15.83:0/3486724149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec440c6120 msgr2=0x7fec440aca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec48f37640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec440c6120 0x7fec440aca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec48f37640 1 -- 172.21.15.83:0/3486724149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4414e680 con 0x7fec441592f0 2024-09-19T07:20:05.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec49738640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec440c6120 0x7fec440aca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:05.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec48f37640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec441592f0 0x7fec440ab5a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fec3400c3f0 tx=0x7fec3400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:05.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec39ffb640 1 -- 172.21.15.83:0/3486724149 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec340052d0 con 0x7fec441592f0 2024-09-19T07:20:05.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec39ffb640 1 -- 172.21.15.83:0/3486724149 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec34005490 con 0x7fec441592f0 2024-09-19T07:20:05.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec4b99c640 1 -- 172.21.15.83:0/3486724149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec4414e320 con 0x7fec441592f0 2024-09-19T07:20:05.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec39ffb640 1 -- 172.21.15.83:0/3486724149 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec34007240 con 0x7fec441592f0 2024-09-19T07:20:05.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec4b99c640 1 -- 172.21.15.83:0/3486724149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec4414db20 con 0x7fec441592f0 2024-09-19T07:20:05.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec4b99c640 1 -- 172.21.15.83:0/3486724149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec440c6500 con 0x7fec441592f0 2024-09-19T07:20:05.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec39ffb640 1 -- 172.21.15.83:0/3486724149 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fec340073e0 con 0x7fec441592f0 2024-09-19T07:20:05.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec39ffb640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec280678c0 0x7fec28069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec49738640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec280678c0 0x7fec28069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec39ffb640 1 -- 172.21.15.83:0/3486724149 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7fec340853a0 con 0x7fec441592f0 2024-09-19T07:20:05.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.018+0000 7fec49738640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec280678c0 0x7fec28069d80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fec40009840 tx=0x7fec40008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:05.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.022+0000 7fec39ffb640 1 -- 172.21.15.83:0/3486724149 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fec34055c40 con 0x7fec441592f0 2024-09-19T07:20:05.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.170+0000 7fec4b99c640 1 -- 172.21.15.83:0/3486724149 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fec440c96a0 con 0x7fec280678c0 2024-09-19T07:20:05.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.186+0000 7fec39ffb640 1 -- 172.21.15.83:0/3486724149 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490337 (secure 0 0 0) 0x7fec440c96a0 con 0x7fec280678c0 2024-09-19T07:20:05.191 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:05.197 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:20:05.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.194+0000 7fec4b99c640 1 -- 172.21.15.83:0/3486724149 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec280678c0 msgr2=0x7fec28069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.194+0000 7fec4b99c640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec280678c0 0x7fec28069d80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fec40009840 tx=0x7fec40008040 comp rx=0 tx=0).stop 2024-09-19T07:20:05.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.194+0000 7fec4b99c640 1 -- 172.21.15.83:0/3486724149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec441592f0 msgr2=0x7fec440ab5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.194+0000 7fec4b99c640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec441592f0 0x7fec440ab5a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fec3400c3f0 tx=0x7fec3400c8b0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.194+0000 7fec4b99c640 1 -- 172.21.15.83:0/3486724149 shutdown_connections 2024-09-19T07:20:05.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.194+0000 7fec4b99c640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec280678c0 0x7fec28069d80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.194+0000 7fec4b99c640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec4414f790 0x7fec440abae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.194+0000 7fec4b99c640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec441592f0 0x7fec440ab5a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.194+0000 7fec4b99c640 1 --2- 172.21.15.83:0/3486724149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec440c6120 0x7fec440aca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.194+0000 7fec4b99c640 1 -- 172.21.15.83:0/3486724149 >> 172.21.15.83:0/3486724149 conn(0x7fec4406bc80 msgr2=0x7fec44154050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:05.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.194+0000 7fec4b99c640 1 -- 172.21.15.83:0/3486724149 shutdown_connections 2024-09-19T07:20:05.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.194+0000 7fec4b99c640 1 -- 172.21.15.83:0/3486724149 wait complete. 2024-09-19T07:20:05.211 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-09-19T07:20:04.826984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596074,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":253,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223724,"kb_used_data":3336,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750996,"statfs":{"total":773094113280,"available":772865019904,"internally_reserved":0,"allocated":3416064,"data_stored":2011625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":886,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009904"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364888+0000","last_change":"2024-09-19T07:19:40.358457+0000","last_active":"2024-09-19T07:19:42.364888+0000","last_peered":"2024-09-19T07:19:42.364888+0000","last_clean":"2024-09-19T07:19:42.364888+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:19:42.364888+0000","last_undegraded":"2024-09-19T07:19:42.364888+0000","last_fullsized":"2024-09-19T07:19:42.364888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370195+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:19:42.370195+0000","last_peered":"2024-09-19T07:19:42.370195+0000","last_clean":"2024-09-19T07:19:42.370195+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:19:42.370195+0000","last_undegraded":"2024-09-19T07:19:42.370195+0000","last_fullsized":"2024-09-19T07:19:42.370195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:57.799442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372168+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:19:42.372168+0000","last_peered":"2024-09-19T07:19:42.372168+0000","last_clean":"2024-09-19T07:19:42.372168+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:19:42.372168+0000","last_undegraded":"2024-09-19T07:19:42.372168+0000","last_fullsized":"2024-09-19T07:19:42.372168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368511+0000","last_change":"2024-09-19T07:19:39.401621+0000","last_active":"2024-09-19T07:19:42.368511+0000","last_peered":"2024-09-19T07:19:42.368511+0000","last_clean":"2024-09-19T07:19:42.368511+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:19:42.368511+0000","last_undegraded":"2024-09-19T07:19:42.368511+0000","last_fullsized":"2024-09-19T07:19:42.368511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:32.095381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366428+0000","last_change":"2024-09-19T07:19:40.366073+0000","last_active":"2024-09-19T07:19:42.366428+0000","last_peered":"2024-09-19T07:19:42.366428+0000","last_clean":"2024-09-19T07:19:42.366428+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:19:42.366428+0000","last_undegraded":"2024-09-19T07:19:42.366428+0000","last_fullsized":"2024-09-19T07:19:42.366428+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372588+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:19:42.372588+0000","last_peered":"2024-09-19T07:19:42.372588+0000","last_clean":"2024-09-19T07:19:42.372588+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:19:42.372588+0000","last_undegraded":"2024-09-19T07:19:42.372588+0000","last_fullsized":"2024-09-19T07:19:42.372588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369500+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:19:42.369500+0000","last_peered":"2024-09-19T07:19:42.369500+0000","last_clean":"2024-09-19T07:19:42.369500+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:19:42.369500+0000","last_undegraded":"2024-09-19T07:19:42.369500+0000","last_fullsized":"2024-09-19T07:19:42.369500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:15.036396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364782+0000","last_change":"2024-09-19T07:19:39.403810+0000","last_active":"2024-09-19T07:19:42.364782+0000","last_peered":"2024-09-19T07:19:42.364782+0000","last_clean":"2024-09-19T07:19:42.364782+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:19:42.364782+0000","last_undegraded":"2024-09-19T07:19:42.364782+0000","last_fullsized":"2024-09-19T07:19:42.364782+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:55.714964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365790+0000","last_change":"2024-09-19T07:19:40.390455+0000","last_active":"2024-09-19T07:19:42.365790+0000","last_peered":"2024-09-19T07:19:42.365790+0000","last_clean":"2024-09-19T07:19:42.365790+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:19:42.365790+0000","last_undegraded":"2024-09-19T07:19:42.365790+0000","last_fullsized":"2024-09-19T07:19:42.365790+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367307+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:19:42.367307+0000","last_peered":"2024-09-19T07:19:42.367307+0000","last_clean":"2024-09-19T07:19:42.367307+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:19:42.367307+0000","last_undegraded":"2024-09-19T07:19:42.367307+0000","last_fullsized":"2024-09-19T07:19:42.367307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353956+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:19:41.353956+0000","last_peered":"2024-09-19T07:19:41.353956+0000","last_clean":"2024-09-19T07:19:41.353956+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:19:41.353956+0000","last_undegraded":"2024-09-19T07:19:41.353956+0000","last_fullsized":"2024-09-19T07:19:41.353956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366091+0000","last_change":"2024-09-19T07:19:39.408309+0000","last_active":"2024-09-19T07:19:42.366091+0000","last_peered":"2024-09-19T07:19:42.366091+0000","last_clean":"2024-09-19T07:19:42.366091+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:19:42.366091+0000","last_undegraded":"2024-09-19T07:19:42.366091+0000","last_fullsized":"2024-09-19T07:19:42.366091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:19.396261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353849+0000","last_change":"2024-09-19T07:19:40.372543+0000","last_active":"2024-09-19T07:19:41.353849+0000","last_peered":"2024-09-19T07:19:41.353849+0000","last_clean":"2024-09-19T07:19:41.353849+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:19:41.353849+0000","last_undegraded":"2024-09-19T07:19:41.353849+0000","last_fullsized":"2024-09-19T07:19:41.353849+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:53:15.682896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367705+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:19:42.367705+0000","last_peered":"2024-09-19T07:19:42.367705+0000","last_clean":"2024-09-19T07:19:42.367705+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:19:42.367705+0000","last_undegraded":"2024-09-19T07:19:42.367705+0000","last_fullsized":"2024-09-19T07:19:42.367705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355253+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:19:41.355253+0000","last_peered":"2024-09-19T07:19:41.355253+0000","last_clean":"2024-09-19T07:19:41.355253+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:19:41.355253+0000","last_undegraded":"2024-09-19T07:19:41.355253+0000","last_fullsized":"2024-09-19T07:19:41.355253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366176+0000","last_change":"2024-09-19T07:19:39.407045+0000","last_active":"2024-09-19T07:19:42.366176+0000","last_peered":"2024-09-19T07:19:42.366176+0000","last_clean":"2024-09-19T07:19:42.366176+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:19:42.366176+0000","last_undegraded":"2024-09-19T07:19:42.366176+0000","last_fullsized":"2024-09-19T07:19:42.366176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:24.846974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367577+0000","last_change":"2024-09-19T07:19:40.396520+0000","last_active":"2024-09-19T07:19:42.367577+0000","last_peered":"2024-09-19T07:19:42.367577+0000","last_clean":"2024-09-19T07:19:42.367577+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:19:42.367577+0000","last_undegraded":"2024-09-19T07:19:42.367577+0000","last_fullsized":"2024-09-19T07:19:42.367577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353504+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:19:41.353504+0000","last_peered":"2024-09-19T07:19:41.353504+0000","last_clean":"2024-09-19T07:19:41.353504+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:19:41.353504+0000","last_undegraded":"2024-09-19T07:19:41.353504+0000","last_fullsized":"2024-09-19T07:19:41.353504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367046+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:19:42.367046+0000","last_peered":"2024-09-19T07:19:42.367046+0000","last_clean":"2024-09-19T07:19:42.367046+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:19:42.367046+0000","last_undegraded":"2024-09-19T07:19:42.367046+0000","last_fullsized":"2024-09-19T07:19:42.367046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368034+0000","last_change":"2024-09-19T07:19:39.410500+0000","last_active":"2024-09-19T07:19:42.368034+0000","last_peered":"2024-09-19T07:19:42.368034+0000","last_clean":"2024-09-19T07:19:42.368034+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:19:42.368034+0000","last_undegraded":"2024-09-19T07:19:42.368034+0000","last_fullsized":"2024-09-19T07:19:42.368034+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:04.182606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349177+0000","last_change":"2024-09-19T07:19:40.381259+0000","last_active":"2024-09-19T07:19:41.349177+0000","last_peered":"2024-09-19T07:19:41.349177+0000","last_clean":"2024-09-19T07:19:41.349177+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:19:41.349177+0000","last_undegraded":"2024-09-19T07:19:41.349177+0000","last_fullsized":"2024-09-19T07:19:41.349177+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:22.017682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370825+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:19:42.370825+0000","last_peered":"2024-09-19T07:19:42.370825+0000","last_clean":"2024-09-19T07:19:42.370825+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:19:42.370825+0000","last_undegraded":"2024-09-19T07:19:42.370825+0000","last_fullsized":"2024-09-19T07:19:42.370825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367918+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:19:42.367918+0000","last_peered":"2024-09-19T07:19:42.367918+0000","last_clean":"2024-09-19T07:19:42.367918+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:19:42.367918+0000","last_undegraded":"2024-09-19T07:19:42.367918+0000","last_fullsized":"2024-09-19T07:19:42.367918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367682+0000","last_change":"2024-09-19T07:19:39.412123+0000","last_active":"2024-09-19T07:19:42.367682+0000","last_peered":"2024-09-19T07:19:42.367682+0000","last_clean":"2024-09-19T07:19:42.367682+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:19:42.367682+0000","last_undegraded":"2024-09-19T07:19:42.367682+0000","last_fullsized":"2024-09-19T07:19:42.367682+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367943+0000","last_change":"2024-09-19T07:19:40.383214+0000","last_active":"2024-09-19T07:19:42.367943+0000","last_peered":"2024-09-19T07:19:42.367943+0000","last_clean":"2024-09-19T07:19:42.367943+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:19:42.367943+0000","last_undegraded":"2024-09-19T07:19:42.367943+0000","last_fullsized":"2024-09-19T07:19:42.367943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367066+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:19:42.367066+0000","last_peered":"2024-09-19T07:19:42.367066+0000","last_clean":"2024-09-19T07:19:42.367066+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:19:42.367066+0000","last_undegraded":"2024-09-19T07:19:42.367066+0000","last_fullsized":"2024-09-19T07:19:42.367066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367094+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:19:42.367094+0000","last_peered":"2024-09-19T07:19:42.367094+0000","last_clean":"2024-09-19T07:19:42.367094+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:19:42.367094+0000","last_undegraded":"2024-09-19T07:19:42.367094+0000","last_fullsized":"2024-09-19T07:19:42.367094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369704+0000","last_change":"2024-09-19T07:19:39.408228+0000","last_active":"2024-09-19T07:19:42.369704+0000","last_peered":"2024-09-19T07:19:42.369704+0000","last_clean":"2024-09-19T07:19:42.369704+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:19:42.369704+0000","last_undegraded":"2024-09-19T07:19:42.369704+0000","last_fullsized":"2024-09-19T07:19:42.369704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355858+0000","last_change":"2024-09-19T07:19:40.395566+0000","last_active":"2024-09-19T07:19:41.355858+0000","last_peered":"2024-09-19T07:19:41.355858+0000","last_clean":"2024-09-19T07:19:41.355858+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:19:41.355858+0000","last_undegraded":"2024-09-19T07:19:41.355858+0000","last_fullsized":"2024-09-19T07:19:41.355858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:54:02.766021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367642+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:19:42.367642+0000","last_peered":"2024-09-19T07:19:42.367642+0000","last_clean":"2024-09-19T07:19:42.367642+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:19:42.367642+0000","last_undegraded":"2024-09-19T07:19:42.367642+0000","last_fullsized":"2024-09-19T07:19:42.367642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366078+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:19:42.366078+0000","last_peered":"2024-09-19T07:19:42.366078+0000","last_clean":"2024-09-19T07:19:42.366078+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:19:42.366078+0000","last_undegraded":"2024-09-19T07:19:42.366078+0000","last_fullsized":"2024-09-19T07:19:42.366078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368203+0000","last_change":"2024-09-19T07:19:39.424661+0000","last_active":"2024-09-19T07:19:42.368203+0000","last_peered":"2024-09-19T07:19:42.368203+0000","last_clean":"2024-09-19T07:19:42.368203+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:19:42.368203+0000","last_undegraded":"2024-09-19T07:19:42.368203+0000","last_fullsized":"2024-09-19T07:19:42.368203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365431+0000","last_change":"2024-09-19T07:19:40.388972+0000","last_active":"2024-09-19T07:19:42.365431+0000","last_peered":"2024-09-19T07:19:42.365431+0000","last_clean":"2024-09-19T07:19:42.365431+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:19:42.365431+0000","last_undegraded":"2024-09-19T07:19:42.365431+0000","last_fullsized":"2024-09-19T07:19:42.365431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371005+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:19:42.371005+0000","last_peered":"2024-09-19T07:19:42.371005+0000","last_clean":"2024-09-19T07:19:42.371005+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:19:42.371005+0000","last_undegraded":"2024-09-19T07:19:42.371005+0000","last_fullsized":"2024-09-19T07:19:42.371005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:23.791880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.357019+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:19:41.357019+0000","last_peered":"2024-09-19T07:19:41.357019+0000","last_clean":"2024-09-19T07:19:41.357019+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:19:41.357019+0000","last_undegraded":"2024-09-19T07:19:41.357019+0000","last_fullsized":"2024-09-19T07:19:41.357019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366707+0000","last_change":"2024-09-19T07:19:39.412192+0000","last_active":"2024-09-19T07:19:42.366707+0000","last_peered":"2024-09-19T07:19:42.366707+0000","last_clean":"2024-09-19T07:19:42.366707+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:19:42.366707+0000","last_undegraded":"2024-09-19T07:19:42.366707+0000","last_fullsized":"2024-09-19T07:19:42.366707+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365056+0000","last_change":"2024-09-19T07:19:40.390217+0000","last_active":"2024-09-19T07:19:42.365056+0000","last_peered":"2024-09-19T07:19:42.365056+0000","last_clean":"2024-09-19T07:19:42.365056+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:19:42.365056+0000","last_undegraded":"2024-09-19T07:19:42.365056+0000","last_fullsized":"2024-09-19T07:19:42.365056+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371269+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:19:42.371269+0000","last_peered":"2024-09-19T07:19:42.371269+0000","last_clean":"2024-09-19T07:19:42.371269+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:19:42.371269+0000","last_undegraded":"2024-09-19T07:19:42.371269+0000","last_fullsized":"2024-09-19T07:19:42.371269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368999+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:19:42.368999+0000","last_peered":"2024-09-19T07:19:42.368999+0000","last_clean":"2024-09-19T07:19:42.368999+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:19:42.368999+0000","last_undegraded":"2024-09-19T07:19:42.368999+0000","last_fullsized":"2024-09-19T07:19:42.368999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:50.700962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365712+0000","last_change":"2024-09-19T07:19:39.434150+0000","last_active":"2024-09-19T07:19:42.365712+0000","last_peered":"2024-09-19T07:19:42.365712+0000","last_clean":"2024-09-19T07:19:42.365712+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:19:42.365712+0000","last_undegraded":"2024-09-19T07:19:42.365712+0000","last_fullsized":"2024-09-19T07:19:42.365712+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:13.654208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352846+0000","last_change":"2024-09-19T07:19:40.380140+0000","last_active":"2024-09-19T07:19:41.352846+0000","last_peered":"2024-09-19T07:19:41.352846+0000","last_clean":"2024-09-19T07:19:41.352846+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:19:41.352846+0000","last_undegraded":"2024-09-19T07:19:41.352846+0000","last_fullsized":"2024-09-19T07:19:41.352846+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368156+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:19:42.368156+0000","last_peered":"2024-09-19T07:19:42.368156+0000","last_clean":"2024-09-19T07:19:42.368156+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:19:42.368156+0000","last_undegraded":"2024-09-19T07:19:42.368156+0000","last_fullsized":"2024-09-19T07:19:42.368156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367593+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:19:42.367593+0000","last_peered":"2024-09-19T07:19:42.367593+0000","last_clean":"2024-09-19T07:19:42.367593+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:19:42.367593+0000","last_undegraded":"2024-09-19T07:19:42.367593+0000","last_fullsized":"2024-09-19T07:19:42.367593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366996+0000","last_change":"2024-09-19T07:19:39.411605+0000","last_active":"2024-09-19T07:19:42.366996+0000","last_peered":"2024-09-19T07:19:42.366996+0000","last_clean":"2024-09-19T07:19:42.366996+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:19:42.366996+0000","last_undegraded":"2024-09-19T07:19:42.366996+0000","last_fullsized":"2024-09-19T07:19:42.366996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:55.654483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367982+0000","last_change":"2024-09-19T07:19:40.372083+0000","last_active":"2024-09-19T07:19:42.367982+0000","last_peered":"2024-09-19T07:19:42.367982+0000","last_clean":"2024-09-19T07:19:42.367982+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:19:42.367982+0000","last_undegraded":"2024-09-19T07:19:42.367982+0000","last_fullsized":"2024-09-19T07:19:42.367982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:39.665475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355342+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:19:41.355342+0000","last_peered":"2024-09-19T07:19:41.355342+0000","last_clean":"2024-09-19T07:19:41.355342+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:19:41.355342+0000","last_undegraded":"2024-09-19T07:19:41.355342+0000","last_fullsized":"2024-09-19T07:19:41.355342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368442+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:19:42.368442+0000","last_peered":"2024-09-19T07:19:42.368442+0000","last_clean":"2024-09-19T07:19:42.368442+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:19:42.368442+0000","last_undegraded":"2024-09-19T07:19:42.368442+0000","last_fullsized":"2024-09-19T07:19:42.368442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367339+0000","last_change":"2024-09-19T07:19:39.420340+0000","last_active":"2024-09-19T07:19:42.367339+0000","last_peered":"2024-09-19T07:19:42.367339+0000","last_clean":"2024-09-19T07:19:42.367339+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:19:42.367339+0000","last_undegraded":"2024-09-19T07:19:42.367339+0000","last_fullsized":"2024-09-19T07:19:42.367339+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:29.338848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369245+0000","last_change":"2024-09-19T07:19:40.383082+0000","last_active":"2024-09-19T07:19:42.369245+0000","last_peered":"2024-09-19T07:19:42.369245+0000","last_clean":"2024-09-19T07:19:42.369245+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:19:42.369245+0000","last_undegraded":"2024-09-19T07:19:42.369245+0000","last_fullsized":"2024-09-19T07:19:42.369245+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:54.436493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367222+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:19:42.367222+0000","last_peered":"2024-09-19T07:19:42.367222+0000","last_clean":"2024-09-19T07:19:42.367222+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:19:42.367222+0000","last_undegraded":"2024-09-19T07:19:42.367222+0000","last_fullsized":"2024-09-19T07:19:42.367222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354505+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:19:41.354505+0000","last_peered":"2024-09-19T07:19:41.354505+0000","last_clean":"2024-09-19T07:19:41.354505+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:19:41.354505+0000","last_undegraded":"2024-09-19T07:19:41.354505+0000","last_fullsized":"2024-09-19T07:19:41.354505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368035+0000","last_change":"2024-09-19T07:19:39.407135+0000","last_active":"2024-09-19T07:19:42.368035+0000","last_peered":"2024-09-19T07:19:42.368035+0000","last_clean":"2024-09-19T07:19:42.368035+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:19:42.368035+0000","last_undegraded":"2024-09-19T07:19:42.368035+0000","last_fullsized":"2024-09-19T07:19:42.368035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:27.890595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349118+0000","last_change":"2024-09-19T07:19:40.368866+0000","last_active":"2024-09-19T07:19:41.349118+0000","last_peered":"2024-09-19T07:19:41.349118+0000","last_clean":"2024-09-19T07:19:41.349118+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:19:41.349118+0000","last_undegraded":"2024-09-19T07:19:41.349118+0000","last_fullsized":"2024-09-19T07:19:41.349118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:28.893439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366065+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:19:42.366065+0000","last_peered":"2024-09-19T07:19:42.366065+0000","last_clean":"2024-09-19T07:19:42.366065+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:19:42.366065+0000","last_undegraded":"2024-09-19T07:19:42.366065+0000","last_fullsized":"2024-09-19T07:19:42.366065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348283+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:19:41.348283+0000","last_peered":"2024-09-19T07:19:41.348283+0000","last_clean":"2024-09-19T07:19:41.348283+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:19:41.348283+0000","last_undegraded":"2024-09-19T07:19:41.348283+0000","last_fullsized":"2024-09-19T07:19:41.348283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369806+0000","last_change":"2024-09-19T07:19:39.420497+0000","last_active":"2024-09-19T07:19:42.369806+0000","last_peered":"2024-09-19T07:19:42.369806+0000","last_clean":"2024-09-19T07:19:42.369806+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:19:42.369806+0000","last_undegraded":"2024-09-19T07:19:42.369806+0000","last_fullsized":"2024-09-19T07:19:42.369806+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:19.547270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356452+0000","last_change":"2024-09-19T07:19:40.387573+0000","last_active":"2024-09-19T07:19:41.356452+0000","last_peered":"2024-09-19T07:19:41.356452+0000","last_clean":"2024-09-19T07:19:41.356452+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:19:41.356452+0000","last_undegraded":"2024-09-19T07:19:41.356452+0000","last_fullsized":"2024-09-19T07:19:41.356452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:20:43.234793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365631+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:19:42.365631+0000","last_peered":"2024-09-19T07:19:42.365631+0000","last_clean":"2024-09-19T07:19:42.365631+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:19:42.365631+0000","last_undegraded":"2024-09-19T07:19:42.365631+0000","last_fullsized":"2024-09-19T07:19:42.365631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:20.824916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366356+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:19:42.366356+0000","last_peered":"2024-09-19T07:19:42.366356+0000","last_clean":"2024-09-19T07:19:42.366356+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:19:42.366356+0000","last_undegraded":"2024-09-19T07:19:42.366356+0000","last_fullsized":"2024-09-19T07:19:42.366356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371671+0000","last_change":"2024-09-19T07:19:39.410752+0000","last_active":"2024-09-19T07:19:42.371671+0000","last_peered":"2024-09-19T07:19:42.371671+0000","last_clean":"2024-09-19T07:19:42.371671+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:42.371671+0000","last_undegraded":"2024-09-19T07:19:42.371671+0000","last_fullsized":"2024-09-19T07:19:42.371671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:47:09.376497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367120+0000","last_change":"2024-09-19T07:19:40.392973+0000","last_active":"2024-09-19T07:19:42.367120+0000","last_peered":"2024-09-19T07:19:42.367120+0000","last_clean":"2024-09-19T07:19:42.367120+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:19:42.367120+0000","last_undegraded":"2024-09-19T07:19:42.367120+0000","last_fullsized":"2024-09-19T07:19:42.367120+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368064+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:19:42.368064+0000","last_peered":"2024-09-19T07:19:42.368064+0000","last_clean":"2024-09-19T07:19:42.368064+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:19:42.368064+0000","last_undegraded":"2024-09-19T07:19:42.368064+0000","last_fullsized":"2024-09-19T07:19:42.368064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354067+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:19:41.354067+0000","last_peered":"2024-09-19T07:19:41.354067+0000","last_clean":"2024-09-19T07:19:41.354067+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:19:41.354067+0000","last_undegraded":"2024-09-19T07:19:41.354067+0000","last_fullsized":"2024-09-19T07:19:41.354067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367007+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:19:42.367007+0000","last_peered":"2024-09-19T07:19:42.367007+0000","last_clean":"2024-09-19T07:19:42.367007+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:19:42.367007+0000","last_undegraded":"2024-09-19T07:19:42.367007+0000","last_fullsized":"2024-09-19T07:19:42.367007+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354541+0000","last_change":"2024-09-19T07:19:39.429085+0000","last_active":"2024-09-19T07:19:41.354541+0000","last_peered":"2024-09-19T07:19:41.354541+0000","last_clean":"2024-09-19T07:19:41.354541+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:19:41.354541+0000","last_undegraded":"2024-09-19T07:19:41.354541+0000","last_fullsized":"2024-09-19T07:19:41.354541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348394+0000","last_change":"2024-09-19T07:19:40.382352+0000","last_active":"2024-09-19T07:19:41.348394+0000","last_peered":"2024-09-19T07:19:41.348394+0000","last_clean":"2024-09-19T07:19:41.348394+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:19:41.348394+0000","last_undegraded":"2024-09-19T07:19:41.348394+0000","last_fullsized":"2024-09-19T07:19:41.348394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:31.024388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367107+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:19:42.367107+0000","last_peered":"2024-09-19T07:19:42.367107+0000","last_clean":"2024-09-19T07:19:42.367107+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:19:42.367107+0000","last_undegraded":"2024-09-19T07:19:42.367107+0000","last_fullsized":"2024-09-19T07:19:42.367107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348973+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:19:41.348973+0000","last_peered":"2024-09-19T07:19:41.348973+0000","last_clean":"2024-09-19T07:19:41.348973+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:19:41.348973+0000","last_undegraded":"2024-09-19T07:19:41.348973+0000","last_fullsized":"2024-09-19T07:19:41.348973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353111+0000","last_change":"2024-09-19T07:19:39.427644+0000","last_active":"2024-09-19T07:19:41.353111+0000","last_peered":"2024-09-19T07:19:41.353111+0000","last_clean":"2024-09-19T07:19:41.353111+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:19:41.353111+0000","last_undegraded":"2024-09-19T07:19:41.353111+0000","last_fullsized":"2024-09-19T07:19:41.353111+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348470+0000","last_change":"2024-09-19T07:19:40.390380+0000","last_active":"2024-09-19T07:19:41.348470+0000","last_peered":"2024-09-19T07:19:41.348470+0000","last_clean":"2024-09-19T07:19:41.348470+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:19:41.348470+0000","last_undegraded":"2024-09-19T07:19:41.348470+0000","last_fullsized":"2024-09-19T07:19:41.348470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:26.385490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354180+0000","last_change":"2024-09-19T07:19:32.276600+0000","last_active":"2024-09-19T07:19:41.354180+0000","last_peered":"2024-09-19T07:19:41.354180+0000","last_clean":"2024-09-19T07:19:41.354180+0000","last_became_active":"2024-09-19T07:19:32.275766+0000","last_became_peered":"2024-09-19T07:19:32.275766+0000","last_unstale":"2024-09-19T07:19:41.354180+0000","last_undegraded":"2024-09-19T07:19:41.354180+0000","last_fullsized":"2024-09-19T07:19:41.354180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:48.506399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370499+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:19:42.370499+0000","last_peered":"2024-09-19T07:19:42.370499+0000","last_clean":"2024-09-19T07:19:42.370499+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:19:42.370499+0000","last_undegraded":"2024-09-19T07:19:42.370499+0000","last_fullsized":"2024-09-19T07:19:42.370499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366597+0000","last_change":"2024-09-19T07:19:39.410638+0000","last_active":"2024-09-19T07:19:42.366597+0000","last_peered":"2024-09-19T07:19:42.366597+0000","last_clean":"2024-09-19T07:19:42.366597+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:19:42.366597+0000","last_undegraded":"2024-09-19T07:19:42.366597+0000","last_fullsized":"2024-09-19T07:19:42.366597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:32.073136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365442+0000","last_change":"2024-09-19T07:19:40.378481+0000","last_active":"2024-09-19T07:19:42.365442+0000","last_peered":"2024-09-19T07:19:42.365442+0000","last_clean":"2024-09-19T07:19:42.365442+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:19:42.365442+0000","last_undegraded":"2024-09-19T07:19:42.365442+0000","last_fullsized":"2024-09-19T07:19:42.365442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:30.585686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369508+0000","last_change":"2024-09-19T07:19:32.292026+0000","last_active":"2024-09-19T07:19:42.369508+0000","last_peered":"2024-09-19T07:19:42.369508+0000","last_clean":"2024-09-19T07:19:42.369508+0000","last_became_active":"2024-09-19T07:19:32.291037+0000","last_became_peered":"2024-09-19T07:19:32.291037+0000","last_unstale":"2024-09-19T07:19:42.369508+0000","last_undegraded":"2024-09-19T07:19:42.369508+0000","last_fullsized":"2024-09-19T07:19:42.369508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:22.190763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349468+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:19:41.349468+0000","last_peered":"2024-09-19T07:19:41.349468+0000","last_clean":"2024-09-19T07:19:41.349468+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:19:41.349468+0000","last_undegraded":"2024-09-19T07:19:41.349468+0000","last_fullsized":"2024-09-19T07:19:41.349468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353302+0000","last_change":"2024-09-19T07:19:39.437656+0000","last_active":"2024-09-19T07:19:41.353302+0000","last_peered":"2024-09-19T07:19:41.353302+0000","last_clean":"2024-09-19T07:19:41.353302+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:19:41.353302+0000","last_undegraded":"2024-09-19T07:19:41.353302+0000","last_fullsized":"2024-09-19T07:19:41.353302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:41.468096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.347816+0000","last_change":"2024-09-19T07:19:40.379289+0000","last_active":"2024-09-19T07:19:41.347816+0000","last_peered":"2024-09-19T07:19:41.347816+0000","last_clean":"2024-09-19T07:19:41.347816+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:19:41.347816+0000","last_undegraded":"2024-09-19T07:19:41.347816+0000","last_fullsized":"2024-09-19T07:19:41.347816+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:20.399106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368075+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:19:42.368075+0000","last_peered":"2024-09-19T07:19:42.368075+0000","last_clean":"2024-09-19T07:19:42.368075+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:19:42.368075+0000","last_undegraded":"2024-09-19T07:19:42.368075+0000","last_fullsized":"2024-09-19T07:19:42.368075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368710+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:19:42.368710+0000","last_peered":"2024-09-19T07:19:42.368710+0000","last_clean":"2024-09-19T07:19:42.368710+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:19:42.368710+0000","last_undegraded":"2024-09-19T07:19:42.368710+0000","last_fullsized":"2024-09-19T07:19:42.368710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366459+0000","last_change":"2024-09-19T07:19:39.413589+0000","last_active":"2024-09-19T07:19:42.366459+0000","last_peered":"2024-09-19T07:19:42.366459+0000","last_clean":"2024-09-19T07:19:42.366459+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:19:42.366459+0000","last_undegraded":"2024-09-19T07:19:42.366459+0000","last_fullsized":"2024-09-19T07:19:42.366459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370515+0000","last_change":"2024-09-19T07:19:40.387159+0000","last_active":"2024-09-19T07:19:42.370515+0000","last_peered":"2024-09-19T07:19:42.370515+0000","last_clean":"2024-09-19T07:19:42.370515+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:19:42.370515+0000","last_undegraded":"2024-09-19T07:19:42.370515+0000","last_fullsized":"2024-09-19T07:19:42.370515+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366844+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:19:42.366844+0000","last_peered":"2024-09-19T07:19:42.366844+0000","last_clean":"2024-09-19T07:19:42.366844+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:19:42.366844+0000","last_undegraded":"2024-09-19T07:19:42.366844+0000","last_fullsized":"2024-09-19T07:19:42.366844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.347952+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:19:41.347952+0000","last_peered":"2024-09-19T07:19:41.347952+0000","last_clean":"2024-09-19T07:19:41.347952+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:19:41.347952+0000","last_undegraded":"2024-09-19T07:19:41.347952+0000","last_fullsized":"2024-09-19T07:19:41.347952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371002+0000","last_change":"2024-09-19T07:19:39.406110+0000","last_active":"2024-09-19T07:19:42.371002+0000","last_peered":"2024-09-19T07:19:42.371002+0000","last_clean":"2024-09-19T07:19:42.371002+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:19:42.371002+0000","last_undegraded":"2024-09-19T07:19:42.371002+0000","last_fullsized":"2024-09-19T07:19:42.371002+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:10:05.224225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355606+0000","last_change":"2024-09-19T07:19:40.365764+0000","last_active":"2024-09-19T07:19:41.355606+0000","last_peered":"2024-09-19T07:19:41.355606+0000","last_clean":"2024-09-19T07:19:41.355606+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:19:41.355606+0000","last_undegraded":"2024-09-19T07:19:41.355606+0000","last_fullsized":"2024-09-19T07:19:41.355606+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:04:03.242292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352283+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:19:41.352283+0000","last_peered":"2024-09-19T07:19:41.352283+0000","last_clean":"2024-09-19T07:19:41.352283+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:19:41.352283+0000","last_undegraded":"2024-09-19T07:19:41.352283+0000","last_fullsized":"2024-09-19T07:19:41.352283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353746+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:19:41.353746+0000","last_peered":"2024-09-19T07:19:41.353746+0000","last_clean":"2024-09-19T07:19:41.353746+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:19:41.353746+0000","last_undegraded":"2024-09-19T07:19:41.353746+0000","last_fullsized":"2024-09-19T07:19:41.353746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370756+0000","last_change":"2024-09-19T07:19:39.429768+0000","last_active":"2024-09-19T07:19:42.370756+0000","last_peered":"2024-09-19T07:19:42.370756+0000","last_clean":"2024-09-19T07:19:42.370756+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:19:42.370756+0000","last_undegraded":"2024-09-19T07:19:42.370756+0000","last_fullsized":"2024-09-19T07:19:42.370756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:47.503631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352914+0000","last_change":"2024-09-19T07:19:40.373542+0000","last_active":"2024-09-19T07:19:41.352914+0000","last_peered":"2024-09-19T07:19:41.352914+0000","last_clean":"2024-09-19T07:19:41.352914+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:19:41.352914+0000","last_undegraded":"2024-09-19T07:19:41.352914+0000","last_fullsized":"2024-09-19T07:19:41.352914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:14.326662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366794+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:19:42.366794+0000","last_peered":"2024-09-19T07:19:42.366794+0000","last_clean":"2024-09-19T07:19:42.366794+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:19:42.366794+0000","last_undegraded":"2024-09-19T07:19:42.366794+0000","last_fullsized":"2024-09-19T07:19:42.366794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354258+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:19:41.354258+0000","last_peered":"2024-09-19T07:19:41.354258+0000","last_clean":"2024-09-19T07:19:41.354258+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:19:41.354258+0000","last_undegraded":"2024-09-19T07:19:41.354258+0000","last_fullsized":"2024-09-19T07:19:41.354258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"25'8","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:20:00.218770+0000","last_change":"2024-09-19T07:19:39.411074+0000","last_active":"2024-09-19T07:20:00.218770+0000","last_peered":"2024-09-19T07:20:00.218770+0000","last_clean":"2024-09-19T07:20:00.218770+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:20:00.218770+0000","last_undegraded":"2024-09-19T07:20:00.218770+0000","last_fullsized":"2024-09-19T07:20:00.218770+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:53.992787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2055,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370720+0000","last_change":"2024-09-19T07:19:40.364814+0000","last_active":"2024-09-19T07:19:42.370720+0000","last_peered":"2024-09-19T07:19:42.370720+0000","last_clean":"2024-09-19T07:19:42.370720+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:19:42.370720+0000","last_undegraded":"2024-09-19T07:19:42.370720+0000","last_fullsized":"2024-09-19T07:19:42.370720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366287+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:19:42.366287+0000","last_peered":"2024-09-19T07:19:42.366287+0000","last_clean":"2024-09-19T07:19:42.366287+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:19:42.366287+0000","last_undegraded":"2024-09-19T07:19:42.366287+0000","last_fullsized":"2024-09-19T07:19:42.366287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:57.131912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368746+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:19:42.368746+0000","last_peered":"2024-09-19T07:19:42.368746+0000","last_clean":"2024-09-19T07:19:42.368746+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:19:42.368746+0000","last_undegraded":"2024-09-19T07:19:42.368746+0000","last_fullsized":"2024-09-19T07:19:42.368746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:23.367365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"24'3","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355049+0000","last_change":"2024-09-19T07:19:39.431616+0000","last_active":"2024-09-19T07:19:41.355049+0000","last_peered":"2024-09-19T07:19:41.355049+0000","last_clean":"2024-09-19T07:19:41.355049+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:19:41.355049+0000","last_undegraded":"2024-09-19T07:19:41.355049+0000","last_fullsized":"2024-09-19T07:19:41.355049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:18.149581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370545+0000","last_change":"2024-09-19T07:19:40.381443+0000","last_active":"2024-09-19T07:19:42.370545+0000","last_peered":"2024-09-19T07:19:42.370545+0000","last_clean":"2024-09-19T07:19:42.370545+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:19:42.370545+0000","last_undegraded":"2024-09-19T07:19:42.370545+0000","last_fullsized":"2024-09-19T07:19:42.370545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366843+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:19:42.366843+0000","last_peered":"2024-09-19T07:19:42.366843+0000","last_clean":"2024-09-19T07:19:42.366843+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:19:42.366843+0000","last_undegraded":"2024-09-19T07:19:42.366843+0000","last_fullsized":"2024-09-19T07:19:42.366843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:57.218063+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:19:57.218063+0000","last_peered":"2024-09-19T07:19:57.218063+0000","last_clean":"2024-09-19T07:19:57.218063+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:19:57.218063+0000","last_undegraded":"2024-09-19T07:19:57.218063+0000","last_fullsized":"2024-09-19T07:19:57.218063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2055,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367083+0000","last_change":"2024-09-19T07:19:39.423972+0000","last_active":"2024-09-19T07:19:42.367083+0000","last_peered":"2024-09-19T07:19:42.367083+0000","last_clean":"2024-09-19T07:19:42.367083+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:19:42.367083+0000","last_undegraded":"2024-09-19T07:19:42.367083+0000","last_fullsized":"2024-09-19T07:19:42.367083+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:45.387040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367684+0000","last_change":"2024-09-19T07:19:40.382092+0000","last_active":"2024-09-19T07:19:42.367684+0000","last_peered":"2024-09-19T07:19:42.367684+0000","last_clean":"2024-09-19T07:19:42.367684+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:19:42.367684+0000","last_undegraded":"2024-09-19T07:19:42.367684+0000","last_fullsized":"2024-09-19T07:19:42.367684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368384+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:19:42.368384+0000","last_peered":"2024-09-19T07:19:42.368384+0000","last_clean":"2024-09-19T07:19:42.368384+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:19:42.368384+0000","last_undegraded":"2024-09-19T07:19:42.368384+0000","last_fullsized":"2024-09-19T07:19:42.368384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370238+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:19:42.370238+0000","last_peered":"2024-09-19T07:19:42.370238+0000","last_clean":"2024-09-19T07:19:42.370238+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:19:42.370238+0000","last_undegraded":"2024-09-19T07:19:42.370238+0000","last_fullsized":"2024-09-19T07:19:42.370238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366357+0000","last_change":"2024-09-19T07:19:39.424879+0000","last_active":"2024-09-19T07:19:42.366357+0000","last_peered":"2024-09-19T07:19:42.366357+0000","last_clean":"2024-09-19T07:19:42.366357+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:19:42.366357+0000","last_undegraded":"2024-09-19T07:19:42.366357+0000","last_fullsized":"2024-09-19T07:19:42.366357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:30.021543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353343+0000","last_change":"2024-09-19T07:19:40.384136+0000","last_active":"2024-09-19T07:19:41.353343+0000","last_peered":"2024-09-19T07:19:41.353343+0000","last_clean":"2024-09-19T07:19:41.353343+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:19:41.353343+0000","last_undegraded":"2024-09-19T07:19:41.353343+0000","last_fullsized":"2024-09-19T07:19:41.353343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369727+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:19:42.369727+0000","last_peered":"2024-09-19T07:19:42.369727+0000","last_clean":"2024-09-19T07:19:42.369727+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:19:42.369727+0000","last_undegraded":"2024-09-19T07:19:42.369727+0000","last_fullsized":"2024-09-19T07:19:42.369727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:48.308137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355901+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:19:41.355901+0000","last_peered":"2024-09-19T07:19:41.355901+0000","last_clean":"2024-09-19T07:19:41.355901+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:19:41.355901+0000","last_undegraded":"2024-09-19T07:19:41.355901+0000","last_fullsized":"2024-09-19T07:19:41.355901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"24'8","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365988+0000","last_change":"2024-09-19T07:19:39.410445+0000","last_active":"2024-09-19T07:19:42.365988+0000","last_peered":"2024-09-19T07:19:42.365988+0000","last_clean":"2024-09-19T07:19:42.365988+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:19:42.365988+0000","last_undegraded":"2024-09-19T07:19:42.365988+0000","last_fullsized":"2024-09-19T07:19:42.365988+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:27.020327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365513+0000","last_change":"2024-09-19T07:19:40.389499+0000","last_active":"2024-09-19T07:19:42.365513+0000","last_peered":"2024-09-19T07:19:42.365513+0000","last_clean":"2024-09-19T07:19:42.365513+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:19:42.365513+0000","last_undegraded":"2024-09-19T07:19:42.365513+0000","last_fullsized":"2024-09-19T07:19:42.365513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.520872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355599+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:19:41.355599+0000","last_peered":"2024-09-19T07:19:41.355599+0000","last_clean":"2024-09-19T07:19:41.355599+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:19:41.355599+0000","last_undegraded":"2024-09-19T07:19:41.355599+0000","last_fullsized":"2024-09-19T07:19:41.355599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.357039+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:19:41.357039+0000","last_peered":"2024-09-19T07:19:41.357039+0000","last_clean":"2024-09-19T07:19:41.357039+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:19:41.357039+0000","last_undegraded":"2024-09-19T07:19:41.357039+0000","last_fullsized":"2024-09-19T07:19:41.357039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368271+0000","last_change":"2024-09-19T07:19:39.416943+0000","last_active":"2024-09-19T07:19:42.368271+0000","last_peered":"2024-09-19T07:19:42.368271+0000","last_clean":"2024-09-19T07:19:42.368271+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:19:42.368271+0000","last_undegraded":"2024-09-19T07:19:42.368271+0000","last_fullsized":"2024-09-19T07:19:42.368271+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:35.555826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356615+0000","last_change":"2024-09-19T07:19:40.394993+0000","last_active":"2024-09-19T07:19:41.356615+0000","last_peered":"2024-09-19T07:19:41.356615+0000","last_clean":"2024-09-19T07:19:41.356615+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:19:41.356615+0000","last_undegraded":"2024-09-19T07:19:41.356615+0000","last_fullsized":"2024-09-19T07:19:41.356615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370672+0000","last_change":"2024-09-19T07:19:39.418885+0000","last_active":"2024-09-19T07:19:42.370672+0000","last_peered":"2024-09-19T07:19:42.370672+0000","last_clean":"2024-09-19T07:19:42.370672+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:19:42.370672+0000","last_undegraded":"2024-09-19T07:19:42.370672+0000","last_fullsized":"2024-09-19T07:19:42.370672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:35:45.790411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366817+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:19:42.366817+0000","last_peered":"2024-09-19T07:19:42.366817+0000","last_clean":"2024-09-19T07:19:42.366817+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:19:42.366817+0000","last_undegraded":"2024-09-19T07:19:42.366817+0000","last_fullsized":"2024-09-19T07:19:42.366817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353835+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:19:41.353835+0000","last_peered":"2024-09-19T07:19:41.353835+0000","last_clean":"2024-09-19T07:19:41.353835+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:19:41.353835+0000","last_undegraded":"2024-09-19T07:19:41.353835+0000","last_fullsized":"2024-09-19T07:19:41.353835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370413+0000","last_change":"2024-09-19T07:19:40.374048+0000","last_active":"2024-09-19T07:19:42.370413+0000","last_peered":"2024-09-19T07:19:42.370413+0000","last_clean":"2024-09-19T07:19:42.370413+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:19:42.370413+0000","last_undegraded":"2024-09-19T07:19:42.370413+0000","last_fullsized":"2024-09-19T07:19:42.370413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:40.084827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"24'4","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356842+0000","last_change":"2024-09-19T07:19:39.432426+0000","last_active":"2024-09-19T07:19:41.356842+0000","last_peered":"2024-09-19T07:19:41.356842+0000","last_clean":"2024-09-19T07:19:41.356842+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:19:41.356842+0000","last_undegraded":"2024-09-19T07:19:41.356842+0000","last_fullsized":"2024-09-19T07:19:41.356842+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366831+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:19:42.366831+0000","last_peered":"2024-09-19T07:19:42.366831+0000","last_clean":"2024-09-19T07:19:42.366831+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:19:42.366831+0000","last_undegraded":"2024-09-19T07:19:42.366831+0000","last_fullsized":"2024-09-19T07:19:42.366831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368900+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:19:42.368900+0000","last_peered":"2024-09-19T07:19:42.368900+0000","last_clean":"2024-09-19T07:19:42.368900+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:19:42.368900+0000","last_undegraded":"2024-09-19T07:19:42.368900+0000","last_fullsized":"2024-09-19T07:19:42.368900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369632+0000","last_change":"2024-09-19T07:19:40.384934+0000","last_active":"2024-09-19T07:19:42.369632+0000","last_peered":"2024-09-19T07:19:42.369632+0000","last_clean":"2024-09-19T07:19:42.369632+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:19:42.369632+0000","last_undegraded":"2024-09-19T07:19:42.369632+0000","last_fullsized":"2024-09-19T07:19:42.369632+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353660+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:19:41.353660+0000","last_peered":"2024-09-19T07:19:41.353660+0000","last_clean":"2024-09-19T07:19:41.353660+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:19:41.353660+0000","last_undegraded":"2024-09-19T07:19:41.353660+0000","last_fullsized":"2024-09-19T07:19:41.353660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368442+0000","last_change":"2024-09-19T07:19:32.289374+0000","last_active":"2024-09-19T07:19:42.368442+0000","last_peered":"2024-09-19T07:19:42.368442+0000","last_clean":"2024-09-19T07:19:42.368442+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:19:42.368442+0000","last_undegraded":"2024-09-19T07:19:42.368442+0000","last_fullsized":"2024-09-19T07:19:42.368442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348715+0000","last_change":"2024-09-19T07:19:39.410582+0000","last_active":"2024-09-19T07:19:41.348715+0000","last_peered":"2024-09-19T07:19:41.348715+0000","last_clean":"2024-09-19T07:19:41.348715+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:19:41.348715+0000","last_undegraded":"2024-09-19T07:19:41.348715+0000","last_fullsized":"2024-09-19T07:19:41.348715+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:41:40.592907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367452+0000","last_change":"2024-09-19T07:19:40.393950+0000","last_active":"2024-09-19T07:19:42.367452+0000","last_peered":"2024-09-19T07:19:42.367452+0000","last_clean":"2024-09-19T07:19:42.367452+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:19:42.367452+0000","last_undegraded":"2024-09-19T07:19:42.367452+0000","last_fullsized":"2024-09-19T07:19:42.367452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371158+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:19:42.371158+0000","last_peered":"2024-09-19T07:19:42.371158+0000","last_clean":"2024-09-19T07:19:42.371158+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:19:42.371158+0000","last_undegraded":"2024-09-19T07:19:42.371158+0000","last_fullsized":"2024-09-19T07:19:42.371158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366780+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:19:42.366780+0000","last_peered":"2024-09-19T07:19:42.366780+0000","last_clean":"2024-09-19T07:19:42.366780+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:19:42.366780+0000","last_undegraded":"2024-09-19T07:19:42.366780+0000","last_fullsized":"2024-09-19T07:19:42.366780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"24'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371594+0000","last_change":"2024-09-19T07:19:39.410662+0000","last_active":"2024-09-19T07:19:42.371594+0000","last_peered":"2024-09-19T07:19:42.371594+0000","last_clean":"2024-09-19T07:19:42.371594+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:19:42.371594+0000","last_undegraded":"2024-09-19T07:19:42.371594+0000","last_fullsized":"2024-09-19T07:19:42.371594+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:19.196074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.372306+0000","last_change":"2024-09-19T07:19:40.385581+0000","last_active":"2024-09-19T07:19:42.372306+0000","last_peered":"2024-09-19T07:19:42.372306+0000","last_clean":"2024-09-19T07:19:42.372306+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:19:42.372306+0000","last_undegraded":"2024-09-19T07:19:42.372306+0000","last_fullsized":"2024-09-19T07:19:42.372306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:14:13.749729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353359+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:19:41.353359+0000","last_peered":"2024-09-19T07:19:41.353359+0000","last_clean":"2024-09-19T07:19:41.353359+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:19:41.353359+0000","last_undegraded":"2024-09-19T07:19:41.353359+0000","last_fullsized":"2024-09-19T07:19:41.353359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369574+0000","last_change":"2024-09-19T07:19:40.377608+0000","last_active":"2024-09-19T07:19:42.369574+0000","last_peered":"2024-09-19T07:19:42.369574+0000","last_clean":"2024-09-19T07:19:42.369574+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:19:42.369574+0000","last_undegraded":"2024-09-19T07:19:42.369574+0000","last_fullsized":"2024-09-19T07:19:42.369574+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348053+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:19:41.348053+0000","last_peered":"2024-09-19T07:19:41.348053+0000","last_clean":"2024-09-19T07:19:41.348053+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:19:41.348053+0000","last_undegraded":"2024-09-19T07:19:41.348053+0000","last_fullsized":"2024-09-19T07:19:41.348053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353277+0000","last_change":"2024-09-19T07:19:40.381262+0000","last_active":"2024-09-19T07:19:41.353277+0000","last_peered":"2024-09-19T07:19:41.353277+0000","last_clean":"2024-09-19T07:19:41.353277+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:19:41.353277+0000","last_undegraded":"2024-09-19T07:19:41.353277+0000","last_fullsized":"2024-09-19T07:19:41.353277+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:34.425472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354018+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:19:41.354018+0000","last_peered":"2024-09-19T07:19:41.354018+0000","last_clean":"2024-09-19T07:19:41.354018+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:19:41.354018+0000","last_undegraded":"2024-09-19T07:19:41.354018+0000","last_fullsized":"2024-09-19T07:19:41.354018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369264+0000","last_change":"2024-09-19T07:19:40.386432+0000","last_active":"2024-09-19T07:19:42.369264+0000","last_peered":"2024-09-19T07:19:42.369264+0000","last_clean":"2024-09-19T07:19:42.369264+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:19:42.369264+0000","last_undegraded":"2024-09-19T07:19:42.369264+0000","last_fullsized":"2024-09-19T07:19:42.369264+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368214+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:19:42.368214+0000","last_peered":"2024-09-19T07:19:42.368214+0000","last_clean":"2024-09-19T07:19:42.368214+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:19:42.368214+0000","last_undegraded":"2024-09-19T07:19:42.368214+0000","last_fullsized":"2024-09-19T07:19:42.368214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365212+0000","last_change":"2024-09-19T07:19:40.378630+0000","last_active":"2024-09-19T07:19:42.365212+0000","last_peered":"2024-09-19T07:19:42.365212+0000","last_clean":"2024-09-19T07:19:42.365212+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:19:42.365212+0000","last_undegraded":"2024-09-19T07:19:42.365212+0000","last_fullsized":"2024-09-19T07:19:42.365212+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371622+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:19:42.371622+0000","last_peered":"2024-09-19T07:19:42.371622+0000","last_clean":"2024-09-19T07:19:42.371622+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:19:42.371622+0000","last_undegraded":"2024-09-19T07:19:42.371622+0000","last_fullsized":"2024-09-19T07:19:42.371622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365853+0000","last_change":"2024-09-19T07:19:40.381365+0000","last_active":"2024-09-19T07:19:42.365853+0000","last_peered":"2024-09-19T07:19:42.365853+0000","last_clean":"2024-09-19T07:19:42.365853+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:19:42.365853+0000","last_undegraded":"2024-09-19T07:19:42.365853+0000","last_fullsized":"2024-09-19T07:19:42.365853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370004+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:19:42.370004+0000","last_peered":"2024-09-19T07:19:42.370004+0000","last_clean":"2024-09-19T07:19:42.370004+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:19:42.370004+0000","last_undegraded":"2024-09-19T07:19:42.370004+0000","last_fullsized":"2024-09-19T07:19:42.370004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365148+0000","last_change":"2024-09-19T07:19:40.387116+0000","last_active":"2024-09-19T07:19:42.365148+0000","last_peered":"2024-09-19T07:19:42.365148+0000","last_clean":"2024-09-19T07:19:42.365148+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:19:42.365148+0000","last_undegraded":"2024-09-19T07:19:42.365148+0000","last_fullsized":"2024-09-19T07:19:42.365148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356605+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:19:41.356605+0000","last_peered":"2024-09-19T07:19:41.356605+0000","last_clean":"2024-09-19T07:19:41.356605+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:19:41.356605+0000","last_undegraded":"2024-09-19T07:19:41.356605+0000","last_fullsized":"2024-09-19T07:19:41.356605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352940+0000","last_change":"2024-09-19T07:19:40.357629+0000","last_active":"2024-09-19T07:19:41.352940+0000","last_peered":"2024-09-19T07:19:41.352940+0000","last_clean":"2024-09-19T07:19:41.352940+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:19:41.352940+0000","last_undegraded":"2024-09-19T07:19:41.352940+0000","last_fullsized":"2024-09-19T07:19:41.352940+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371018+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:19:42.371018+0000","last_peered":"2024-09-19T07:19:42.371018+0000","last_clean":"2024-09-19T07:19:42.371018+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:19:42.371018+0000","last_undegraded":"2024-09-19T07:19:42.371018+0000","last_fullsized":"2024-09-19T07:19:42.371018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371827+0000","last_change":"2024-09-19T07:19:40.388597+0000","last_active":"2024-09-19T07:19:42.371827+0000","last_peered":"2024-09-19T07:19:42.371827+0000","last_clean":"2024-09-19T07:19:42.371827+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:19:42.371827+0000","last_undegraded":"2024-09-19T07:19:42.371827+0000","last_fullsized":"2024-09-19T07:19:42.371827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:34:18.402016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356021+0000","last_change":"2024-09-19T07:19:32.273635+0000","last_active":"2024-09-19T07:19:41.356021+0000","last_peered":"2024-09-19T07:19:41.356021+0000","last_clean":"2024-09-19T07:19:41.356021+0000","last_became_active":"2024-09-19T07:19:32.271748+0000","last_became_peered":"2024-09-19T07:19:32.271748+0000","last_unstale":"2024-09-19T07:19:41.356021+0000","last_undegraded":"2024-09-19T07:19:41.356021+0000","last_fullsized":"2024-09-19T07:19:41.356021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:49:41.724626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369047+0000","last_change":"2024-09-19T07:19:40.383779+0000","last_active":"2024-09-19T07:19:42.369047+0000","last_peered":"2024-09-19T07:19:42.369047+0000","last_clean":"2024-09-19T07:19:42.369047+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:19:42.369047+0000","last_undegraded":"2024-09-19T07:19:42.369047+0000","last_fullsized":"2024-09-19T07:19:42.369047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368684+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:19:42.368684+0000","last_peered":"2024-09-19T07:19:42.368684+0000","last_clean":"2024-09-19T07:19:42.368684+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:19:42.368684+0000","last_undegraded":"2024-09-19T07:19:42.368684+0000","last_fullsized":"2024-09-19T07:19:42.368684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348737+0000","last_change":"2024-09-19T07:19:40.376342+0000","last_active":"2024-09-19T07:19:41.348737+0000","last_peered":"2024-09-19T07:19:41.348737+0000","last_clean":"2024-09-19T07:19:41.348737+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:19:41.348737+0000","last_undegraded":"2024-09-19T07:19:41.348737+0000","last_fullsized":"2024-09-19T07:19:41.348737+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:22.997744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353771+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:19:41.353771+0000","last_peered":"2024-09-19T07:19:41.353771+0000","last_clean":"2024-09-19T07:19:41.353771+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:19:41.353771+0000","last_undegraded":"2024-09-19T07:19:41.353771+0000","last_fullsized":"2024-09-19T07:19:41.353771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371199+0000","last_change":"2024-09-19T07:19:40.361504+0000","last_active":"2024-09-19T07:19:42.371199+0000","last_peered":"2024-09-19T07:19:42.371199+0000","last_clean":"2024-09-19T07:19:42.371199+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:19:42.371199+0000","last_undegraded":"2024-09-19T07:19:42.371199+0000","last_fullsized":"2024-09-19T07:19:42.371199+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:30:31.110669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367582+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:19:42.367582+0000","last_peered":"2024-09-19T07:19:42.367582+0000","last_clean":"2024-09-19T07:19:42.367582+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:19:42.367582+0000","last_undegraded":"2024-09-19T07:19:42.367582+0000","last_fullsized":"2024-09-19T07:19:42.367582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:19.033192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365909+0000","last_change":"2024-09-19T07:19:40.390205+0000","last_active":"2024-09-19T07:19:42.365909+0000","last_peered":"2024-09-19T07:19:42.365909+0000","last_clean":"2024-09-19T07:19:42.365909+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:19:42.365909+0000","last_undegraded":"2024-09-19T07:19:42.365909+0000","last_fullsized":"2024-09-19T07:19:42.365909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.354180+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:19:41.354180+0000","last_peered":"2024-09-19T07:19:41.354180+0000","last_clean":"2024-09-19T07:19:41.354180+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:19:41.354180+0000","last_undegraded":"2024-09-19T07:19:41.354180+0000","last_fullsized":"2024-09-19T07:19:41.354180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365447+0000","last_change":"2024-09-19T07:19:40.390559+0000","last_active":"2024-09-19T07:19:42.365447+0000","last_peered":"2024-09-19T07:19:42.365447+0000","last_clean":"2024-09-19T07:19:42.365447+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:19:42.365447+0000","last_undegraded":"2024-09-19T07:19:42.365447+0000","last_fullsized":"2024-09-19T07:19:42.365447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367349+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:19:42.367349+0000","last_peered":"2024-09-19T07:19:42.367349+0000","last_clean":"2024-09-19T07:19:42.367349+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:19:42.367349+0000","last_undegraded":"2024-09-19T07:19:42.367349+0000","last_fullsized":"2024-09-19T07:19:42.367349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352820+0000","last_change":"2024-09-19T07:19:40.358433+0000","last_active":"2024-09-19T07:19:41.352820+0000","last_peered":"2024-09-19T07:19:41.352820+0000","last_clean":"2024-09-19T07:19:41.352820+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:19:41.352820+0000","last_undegraded":"2024-09-19T07:19:41.352820+0000","last_fullsized":"2024-09-19T07:19:41.352820+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:10:25.226677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.349337+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:19:41.349337+0000","last_peered":"2024-09-19T07:19:41.349337+0000","last_clean":"2024-09-19T07:19:41.349337+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:19:41.349337+0000","last_undegraded":"2024-09-19T07:19:41.349337+0000","last_fullsized":"2024-09-19T07:19:41.349337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:51.710997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365454+0000","last_change":"2024-09-19T07:19:40.374374+0000","last_active":"2024-09-19T07:19:42.365454+0000","last_peered":"2024-09-19T07:19:42.365454+0000","last_clean":"2024-09-19T07:19:42.365454+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:19:42.365454+0000","last_undegraded":"2024-09-19T07:19:42.365454+0000","last_fullsized":"2024-09-19T07:19:42.365454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:34:05.875388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356952+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:19:41.356952+0000","last_peered":"2024-09-19T07:19:41.356952+0000","last_clean":"2024-09-19T07:19:41.356952+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:19:41.356952+0000","last_undegraded":"2024-09-19T07:19:41.356952+0000","last_fullsized":"2024-09-19T07:19:41.356952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368071+0000","last_change":"2024-09-19T07:19:40.387341+0000","last_active":"2024-09-19T07:19:42.368071+0000","last_peered":"2024-09-19T07:19:42.368071+0000","last_clean":"2024-09-19T07:19:42.368071+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:19:42.368071+0000","last_undegraded":"2024-09-19T07:19:42.368071+0000","last_fullsized":"2024-09-19T07:19:42.368071+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:56:40.928844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353563+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:19:41.353563+0000","last_peered":"2024-09-19T07:19:41.353563+0000","last_clean":"2024-09-19T07:19:41.353563+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:19:41.353563+0000","last_undegraded":"2024-09-19T07:19:41.353563+0000","last_fullsized":"2024-09-19T07:19:41.353563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368148+0000","last_change":"2024-09-19T07:19:40.388484+0000","last_active":"2024-09-19T07:19:42.368148+0000","last_peered":"2024-09-19T07:19:42.368148+0000","last_clean":"2024-09-19T07:19:42.368148+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:19:42.368148+0000","last_undegraded":"2024-09-19T07:19:42.368148+0000","last_fullsized":"2024-09-19T07:19:42.368148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366974+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:19:42.366974+0000","last_peered":"2024-09-19T07:19:42.366974+0000","last_clean":"2024-09-19T07:19:42.366974+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:19:42.366974+0000","last_undegraded":"2024-09-19T07:19:42.366974+0000","last_fullsized":"2024-09-19T07:19:42.366974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352836+0000","last_change":"2024-09-19T07:19:40.367578+0000","last_active":"2024-09-19T07:19:41.352836+0000","last_peered":"2024-09-19T07:19:41.352836+0000","last_clean":"2024-09-19T07:19:41.352836+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:19:41.352836+0000","last_undegraded":"2024-09-19T07:19:41.352836+0000","last_fullsized":"2024-09-19T07:19:41.352836+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:00:25.357854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366048+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:19:42.366048+0000","last_peered":"2024-09-19T07:19:42.366048+0000","last_clean":"2024-09-19T07:19:42.366048+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:19:42.366048+0000","last_undegraded":"2024-09-19T07:19:42.366048+0000","last_fullsized":"2024-09-19T07:19:42.366048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:43.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366598+0000","last_change":"2024-09-19T07:19:40.385641+0000","last_active":"2024-09-19T07:19:42.366598+0000","last_peered":"2024-09-19T07:19:42.366598+0000","last_clean":"2024-09-19T07:19:42.366598+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:19:42.366598+0000","last_undegraded":"2024-09-19T07:19:42.366598+0000","last_fullsized":"2024-09-19T07:19:42.366598+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.370694+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:19:42.370694+0000","last_peered":"2024-09-19T07:19:42.370694+0000","last_clean":"2024-09-19T07:19:42.370694+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:19:42.370694+0000","last_undegraded":"2024-09-19T07:19:42.370694+0000","last_fullsized":"2024-09-19T07:19:42.370694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369737+0000","last_change":"2024-09-19T07:19:40.367101+0000","last_active":"2024-09-19T07:19:42.369737+0000","last_peered":"2024-09-19T07:19:42.369737+0000","last_clean":"2024-09-19T07:19:42.369737+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:19:42.369737+0000","last_undegraded":"2024-09-19T07:19:42.369737+0000","last_fullsized":"2024-09-19T07:19:42.369737+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368379+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:19:42.368379+0000","last_peered":"2024-09-19T07:19:42.368379+0000","last_clean":"2024-09-19T07:19:42.368379+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:19:42.368379+0000","last_undegraded":"2024-09-19T07:19:42.368379+0000","last_fullsized":"2024-09-19T07:19:42.368379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348413+0000","last_change":"2024-09-19T07:19:40.384212+0000","last_active":"2024-09-19T07:19:41.348413+0000","last_peered":"2024-09-19T07:19:41.348413+0000","last_clean":"2024-09-19T07:19:41.348413+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:19:41.348413+0000","last_undegraded":"2024-09-19T07:19:41.348413+0000","last_fullsized":"2024-09-19T07:19:41.348413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:57.724404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355987+0000","last_change":"2024-09-19T07:19:32.299956+0000","last_active":"2024-09-19T07:19:41.355987+0000","last_peered":"2024-09-19T07:19:41.355987+0000","last_clean":"2024-09-19T07:19:41.355987+0000","last_became_active":"2024-09-19T07:19:32.299168+0000","last_became_peered":"2024-09-19T07:19:32.299168+0000","last_unstale":"2024-09-19T07:19:41.355987+0000","last_undegraded":"2024-09-19T07:19:41.355987+0000","last_fullsized":"2024-09-19T07:19:41.355987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366761+0000","last_change":"2024-09-19T07:19:40.387401+0000","last_active":"2024-09-19T07:19:42.366761+0000","last_peered":"2024-09-19T07:19:42.366761+0000","last_clean":"2024-09-19T07:19:42.366761+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:19:42.366761+0000","last_undegraded":"2024-09-19T07:19:42.366761+0000","last_fullsized":"2024-09-19T07:19:42.366761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356423+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:19:41.356423+0000","last_peered":"2024-09-19T07:19:41.356423+0000","last_clean":"2024-09-19T07:19:41.356423+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:19:41.356423+0000","last_undegraded":"2024-09-19T07:19:41.356423+0000","last_fullsized":"2024-09-19T07:19:41.356423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.352920+0000","last_change":"2024-09-19T07:19:40.361430+0000","last_active":"2024-09-19T07:19:41.352920+0000","last_peered":"2024-09-19T07:19:41.352920+0000","last_clean":"2024-09-19T07:19:41.352920+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:19:41.352920+0000","last_undegraded":"2024-09-19T07:19:41.352920+0000","last_fullsized":"2024-09-19T07:19:41.352920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:23:05.185943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369329+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:19:42.369329+0000","last_peered":"2024-09-19T07:19:42.369329+0000","last_clean":"2024-09-19T07:19:42.369329+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:19:42.369329+0000","last_undegraded":"2024-09-19T07:19:42.369329+0000","last_fullsized":"2024-09-19T07:19:42.369329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.371825+0000","last_change":"2024-09-19T07:19:40.378172+0000","last_active":"2024-09-19T07:19:42.371825+0000","last_peered":"2024-09-19T07:19:42.371825+0000","last_clean":"2024-09-19T07:19:42.371825+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:19:42.371825+0000","last_undegraded":"2024-09-19T07:19:42.371825+0000","last_fullsized":"2024-09-19T07:19:42.371825+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:52.321790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366422+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:19:42.366422+0000","last_peered":"2024-09-19T07:19:42.366422+0000","last_clean":"2024-09-19T07:19:42.366422+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:19:42.366422+0000","last_undegraded":"2024-09-19T07:19:42.366422+0000","last_fullsized":"2024-09-19T07:19:42.366422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.356589+0000","last_change":"2024-09-19T07:19:40.373909+0000","last_active":"2024-09-19T07:19:41.356589+0000","last_peered":"2024-09-19T07:19:41.356589+0000","last_clean":"2024-09-19T07:19:41.356589+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:19:41.356589+0000","last_undegraded":"2024-09-19T07:19:41.356589+0000","last_fullsized":"2024-09-19T07:19:41.356589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366411+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:19:42.366411+0000","last_peered":"2024-09-19T07:19:42.366411+0000","last_clean":"2024-09-19T07:19:42.366411+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:19:42.366411+0000","last_undegraded":"2024-09-19T07:19:42.366411+0000","last_fullsized":"2024-09-19T07:19:42.366411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367511+0000","last_change":"2024-09-19T07:19:40.396146+0000","last_active":"2024-09-19T07:19:42.367511+0000","last_peered":"2024-09-19T07:19:42.367511+0000","last_clean":"2024-09-19T07:19:42.367511+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:19:42.367511+0000","last_undegraded":"2024-09-19T07:19:42.367511+0000","last_fullsized":"2024-09-19T07:19:42.367511+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353532+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:19:41.353532+0000","last_peered":"2024-09-19T07:19:41.353532+0000","last_clean":"2024-09-19T07:19:41.353532+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:19:41.353532+0000","last_undegraded":"2024-09-19T07:19:41.353532+0000","last_fullsized":"2024-09-19T07:19:41.353532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.355009+0000","last_change":"2024-09-19T07:19:40.373552+0000","last_active":"2024-09-19T07:19:41.355009+0000","last_peered":"2024-09-19T07:19:41.355009+0000","last_clean":"2024-09-19T07:19:41.355009+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:19:41.355009+0000","last_undegraded":"2024-09-19T07:19:41.355009+0000","last_fullsized":"2024-09-19T07:19:41.355009+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:23:33.629831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.366537+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:19:42.366537+0000","last_peered":"2024-09-19T07:19:42.366537+0000","last_clean":"2024-09-19T07:19:42.366537+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:19:42.366537+0000","last_undegraded":"2024-09-19T07:19:42.366537+0000","last_fullsized":"2024-09-19T07:19:42.366537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.367996+0000","last_change":"2024-09-19T07:19:40.356989+0000","last_active":"2024-09-19T07:19:42.367996+0000","last_peered":"2024-09-19T07:19:42.367996+0000","last_clean":"2024-09-19T07:19:42.367996+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:19:42.367996+0000","last_undegraded":"2024-09-19T07:19:42.367996+0000","last_fullsized":"2024-09-19T07:19:42.367996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.353682+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:19:41.353682+0000","last_peered":"2024-09-19T07:19:41.353682+0000","last_clean":"2024-09-19T07:19:41.353682+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:19:41.353682+0000","last_undegraded":"2024-09-19T07:19:41.353682+0000","last_fullsized":"2024-09-19T07:19:41.353682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:41.348160+0000","last_change":"2024-09-19T07:19:40.376487+0000","last_active":"2024-09-19T07:19:41.348160+0000","last_peered":"2024-09-19T07:19:41.348160+0000","last_clean":"2024-09-19T07:19:41.348160+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:19:41.348160+0000","last_undegraded":"2024-09-19T07:19:41.348160+0000","last_fullsized":"2024-09-19T07:19:41.348160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:05.185451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369628+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:19:42.369628+0000","last_peered":"2024-09-19T07:19:42.369628+0000","last_clean":"2024-09-19T07:19:42.369628+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:19:42.369628+0000","last_undegraded":"2024-09-19T07:19:42.369628+0000","last_fullsized":"2024-09-19T07:19:42.369628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:25.667168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.364909+0000","last_change":"2024-09-19T07:19:40.377604+0000","last_active":"2024-09-19T07:19:42.364909+0000","last_peered":"2024-09-19T07:19:42.364909+0000","last_clean":"2024-09-19T07:19:42.364909+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:19:42.364909+0000","last_undegraded":"2024-09-19T07:19:42.364909+0000","last_fullsized":"2024-09-19T07:19:42.364909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.368557+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:19:42.368557+0000","last_peered":"2024-09-19T07:19:42.368557+0000","last_clean":"2024-09-19T07:19:42.368557+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:19:42.368557+0000","last_undegraded":"2024-09-19T07:19:42.368557+0000","last_fullsized":"2024-09-19T07:19:42.368557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.365847+0000","last_change":"2024-09-19T07:19:40.377580+0000","last_active":"2024-09-19T07:19:42.365847+0000","last_peered":"2024-09-19T07:19:42.365847+0000","last_clean":"2024-09-19T07:19:42.365847+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:19:42.365847+0000","last_undegraded":"2024-09-19T07:19:42.365847+0000","last_fullsized":"2024-09-19T07:19:42.365847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-19T07:19:42.369285+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:19:42.369285+0000","last_peered":"2024-09-19T07:19:42.369285+0000","last_clean":"2024-09-19T07:19:42.369285+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:19:42.369285+0000","last_undegraded":"2024-09-19T07:19:42.369285+0000","last_fullsized":"2024-09-19T07:19:42.369285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2853,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2853,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542156,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":864256,"data_stored":693530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":105161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":103252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":282624,"data_stored":102720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":106179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542157,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":102560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542156,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":104727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542155,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":868352,"data_stored":693496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:20:05.218 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:05.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fed3c640 1 Processor -- start 2024-09-19T07:20:05.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fed3c640 1 -- start start 2024-09-19T07:20:05.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fed3c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21f8152b20 0x7f21f80c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fed3c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21f8150670 con 0x7f21f80c45c0 2024-09-19T07:20:05.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fed3c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21f80c2880 con 0x7f21f8150290 2024-09-19T07:20:05.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fed3c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f21f80716c0 con 0x7f21f8152b20 2024-09-19T07:20:05.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fd2d9640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f21f80c45c0 0x7f21f8153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50038/0 (socket says 172.21.15.83:50038) 2024-09-19T07:20:05.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fd2d9640 1 -- 172.21.15.83:0/2896153858 learned_addr learned my addr 172.21.15.83:0/2896153858 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:05.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21effff640 1 --2- 172.21.15.83:0/2896153858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21f8152b20 0x7f21f80c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2053024376 0 0) 0x7f21f8150670 con 0x7f21f80c45c0 2024-09-19T07:20:05.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21e0005740 con 0x7f21f80c45c0 2024-09-19T07:20:05.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 246461790 0 0) 0x7f21e0005740 con 0x7f21f80c45c0 2024-09-19T07:20:05.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21f8152b20 msgr2=0x7f21f80c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:05.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 --2- 172.21.15.83:0/2896153858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21f8152b20 0x7f21f80c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 >> v1:172.21.15.184:6789/0 conn(0x7f21f8150290 legacy=0x7f21f80c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f80c2b60 con 0x7f21f80c45c0 2024-09-19T07:20:05.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f21f0001a00 con 0x7f21f80c45c0 2024-09-19T07:20:05.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 >> v1:172.21.15.83:6789/0 conn(0x7f21f80c45c0 legacy=0x7f21f8153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 --2- 172.21.15.83:0/2896153858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21e0005c00 0x7f21e0006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 --2- 172.21.15.83:0/2896153858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21f8152b20 0x7f21e0006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 --2- 172.21.15.83:0/2896153858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21e0006f60 0x7f21e00073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21e000dc90 con 0x7f21e0005c00 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21e000de10 con 0x7f21e0006f60 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f21e000e010 con 0x7f21f8152b20 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fcad8640 1 --2- 172.21.15.83:0/2896153858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21e0005c00 0x7f21e0006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21effff640 1 --2- 172.21.15.83:0/2896153858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21f8152b20 0x7f21e0006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fd2d9640 1 --2- 172.21.15.83:0/2896153858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21e0006f60 0x7f21e00073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fcad8640 1 -- 172.21.15.83:0/2896153858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21f8152b20 msgr2=0x7f21e0006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fcad8640 1 --2- 172.21.15.83:0/2896153858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21f8152b20 0x7f21e0006a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fcad8640 1 -- 172.21.15.83:0/2896153858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21e0006f60 msgr2=0x7f21e00073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fcad8640 1 --2- 172.21.15.83:0/2896153858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21e0006f60 0x7f21e00073e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.370+0000 7f21fcad8640 1 -- 172.21.15.83:0/2896153858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e0005740 con 0x7f21e0005c00 2024-09-19T07:20:05.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fcad8640 1 --2- 172.21.15.83:0/2896153858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21e0005c00 0x7f21e0006020 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f21f4001750 tx=0x7f21f401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0f1c07d5b6e5e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:05.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f40013a0 con 0x7f21e0005c00 2024-09-19T07:20:05.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f40052f0 con 0x7f21e0005c00 2024-09-19T07:20:05.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21ef7fe640 1 -- 172.21.15.83:0/2896153858 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f4003840 con 0x7f21e0005c00 2024-09-19T07:20:05.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 -- 172.21.15.83:0/2896153858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21e0005c00 msgr2=0x7f21e0006020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 --2- 172.21.15.83:0/2896153858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21e0005c00 0x7f21e0006020 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f21f4001750 tx=0x7f21f401c3e0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fd2d9640 1 -- 172.21.15.83:0/2896153858 reap_dead start 2024-09-19T07:20:05.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 -- 172.21.15.83:0/2896153858 shutdown_connections 2024-09-19T07:20:05.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 -- 172.21.15.83:0/2896153858 >> 172.21.15.83:0/2896153858 conn(0x7f21f806bc80 msgr2=0x7f21f806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:05.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 -- 172.21.15.83:0/2896153858 shutdown_connections 2024-09-19T07:20:05.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 -- 172.21.15.83:0/2896153858 wait complete. 2024-09-19T07:20:05.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 Processor -- start 2024-09-19T07:20:05.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 -- start start 2024-09-19T07:20:05.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21e0005c00 0x7f21f80c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21f814d4a0 0x7f21f8148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f814f3f0 0x7f21f81eddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21f806e6b0 con 0x7f21f814d4a0 2024-09-19T07:20:05.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21f80c3490 con 0x7f21f814f3f0 2024-09-19T07:20:05.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fed3c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f21f806e830 con 0x7f21e0005c00 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21effff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21f814d4a0 0x7f21f8148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21effff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21f814d4a0 0x7f21f8148b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49712/0 (socket says 172.21.15.83:49712) 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21effff640 1 -- 172.21.15.83:0/3723011976 learned_addr learned my addr 172.21.15.83:0/3723011976 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fcad8640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21e0005c00 0x7f21f80c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21fd2d9640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f814f3f0 0x7f21f81eddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21effff640 1 -- 172.21.15.83:0/3723011976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21e0005c00 msgr2=0x7f21f80c6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21effff640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21e0005c00 0x7f21f80c6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21effff640 1 -- 172.21.15.83:0/3723011976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f814f3f0 msgr2=0x7f21f81eddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21effff640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f814f3f0 0x7f21f81eddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21effff640 1 -- 172.21.15.83:0/3723011976 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f81ee8a0 con 0x7f21f814d4a0 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.374+0000 7f21effff640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21f814d4a0 0x7f21f8148b10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f21e8010810 tx=0x7f21e8010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.378+0000 7f21ed7fa640 1 -- 172.21.15.83:0/3723011976 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e800f040 con 0x7f21f814d4a0 2024-09-19T07:20:05.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.378+0000 7f21ed7fa640 1 -- 172.21.15.83:0/3723011976 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21e800eb60 con 0x7f21f814d4a0 2024-09-19T07:20:05.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.378+0000 7f21fed3c640 1 -- 172.21.15.83:0/3723011976 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f81eeb80 con 0x7f21f814d4a0 2024-09-19T07:20:05.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.378+0000 7f21ed7fa640 1 -- 172.21.15.83:0/3723011976 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e801c400 con 0x7f21f814d4a0 2024-09-19T07:20:05.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.378+0000 7f21fed3c640 1 -- 172.21.15.83:0/3723011976 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f80ac100 con 0x7f21f814d4a0 2024-09-19T07:20:05.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.378+0000 7f21fed3c640 1 -- 172.21.15.83:0/3723011976 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f814ea60 con 0x7f21f814d4a0 2024-09-19T07:20:05.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.378+0000 7f21ed7fa640 1 -- 172.21.15.83:0/3723011976 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f21e801c5a0 con 0x7f21f814d4a0 2024-09-19T07:20:05.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.378+0000 7f21ed7fa640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21e40678c0 0x7f21e4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.378+0000 7f21ed7fa640 1 -- 172.21.15.83:0/3723011976 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f21e8086510 con 0x7f21f814d4a0 2024-09-19T07:20:05.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.378+0000 7f21fcad8640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21e40678c0 0x7f21e4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.378+0000 7f21fcad8640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21e40678c0 0x7f21e4069d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f21e0007a00 tx=0x7f21f40011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:05.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.382+0000 7f21ed7fa640 1 -- 172.21.15.83:0/3723011976 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f21e8018030 con 0x7f21f814d4a0 2024-09-19T07:20:05.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 -- 172.21.15.83:0/3723011976 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21f806ec80 con 0x7f21f814d4a0 2024-09-19T07:20:05.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21ed7fa640 1 -- 172.21.15.83:0/3723011976 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f21e8056db0 con 0x7f21f814d4a0 2024-09-19T07:20:05.579 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:05.579 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:05.579 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:05.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 -- 172.21.15.83:0/3723011976 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21e40678c0 msgr2=0x7f21e4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21e40678c0 0x7f21e4069d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f21e0007a00 tx=0x7f21f40011a0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 -- 172.21.15.83:0/3723011976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21f814d4a0 msgr2=0x7f21f8148b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21f814d4a0 0x7f21f8148b10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f21e8010810 tx=0x7f21e8010cd0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 -- 172.21.15.83:0/3723011976 shutdown_connections 2024-09-19T07:20:05.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21e40678c0 0x7f21e4069d80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f814f3f0 0x7f21f81eddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21f814d4a0 0x7f21f8148b10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 --2- 172.21.15.83:0/3723011976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21e0005c00 0x7f21f80c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 -- 172.21.15.83:0/3723011976 >> 172.21.15.83:0/3723011976 conn(0x7f21f806bc80 msgr2=0x7f21f8157620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:05.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 -- 172.21.15.83:0/3723011976 shutdown_connections 2024-09-19T07:20:05.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.578+0000 7f21fed3c640 1 -- 172.21.15.83:0/3723011976 wait complete. 2024-09-19T07:20:05.596 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:20:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.714+0000 7ff4b682e640 1 Processor -- start 2024-09-19T07:20:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.714+0000 7ff4b682e640 1 -- start start 2024-09-19T07:20:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4b682e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff4b00ab750 0x7ff4b006eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4b682e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff4b0073a50 0x7ff4b00c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4b682e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b006ddd0 con 0x7ff4b00c3b70 2024-09-19T07:20:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4b682e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff4b006e070 con 0x7ff4b0073a50 2024-09-19T07:20:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4b682e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b006e350 con 0x7ff4b00ab750 2024-09-19T07:20:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4af7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff4b0073a50 0x7ff4b00c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4af7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff4b0073a50 0x7ff4b00c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49716/0 (socket says 172.21.15.83:49716) 2024-09-19T07:20:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4af7fe640 1 -- 172.21.15.83:0/4034374851 learned_addr learned my addr 172.21.15.83:0/4034374851 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4affff640 1 --2- 172.21.15.83:0/4034374851 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff4b00ab750 0x7ff4b006eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4af7fe640 1 -- 172.21.15.83:0/4034374851 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff4b00ab750 msgr2=0x7ff4b006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4af7fe640 1 --2- 172.21.15.83:0/4034374851 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff4b00ab750 0x7ff4b006eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4af7fe640 1 -- 172.21.15.83:0/4034374851 >> v1:172.21.15.184:6789/0 conn(0x7ff4b00c3b70 legacy=0x7ff4b0152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:05.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4af7fe640 1 -- 172.21.15.83:0/4034374851 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b006e630 con 0x7ff4b0073a50 2024-09-19T07:20:05.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4af7fe640 1 --2- 172.21.15.83:0/4034374851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4b0073a50 0x7ff4b00c1ff0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009870 tx=0x7ff4a001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f15299d306bbcb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:05.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4aeffd640 1 -- 172.21.15.83:0/4034374851 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a00b0070 con 0x7ff4b0073a50 2024-09-19T07:20:05.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4aeffd640 1 -- 172.21.15.83:0/4034374851 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a00ab600 con 0x7ff4b0073a50 2024-09-19T07:20:05.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.718+0000 7ff4aeffd640 1 -- 172.21.15.83:0/4034374851 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a00ab9d0 con 0x7ff4b0073a50 2024-09-19T07:20:05.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 -- 172.21.15.83:0/4034374851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4b0073a50 msgr2=0x7ff4b00c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 --2- 172.21.15.83:0/4034374851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4b0073a50 0x7ff4b00c1ff0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009870 tx=0x7ff4a001c490 comp rx=0 tx=0).stop 2024-09-19T07:20:05.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 -- 172.21.15.83:0/4034374851 shutdown_connections 2024-09-19T07:20:05.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 --2- 172.21.15.83:0/4034374851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4b0073a50 0x7ff4b00c1ff0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 --2- 172.21.15.83:0/4034374851 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff4b00ab750 0x7ff4b006eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 -- 172.21.15.83:0/4034374851 >> 172.21.15.83:0/4034374851 conn(0x7ff4b006bc80 msgr2=0x7ff4b006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:05.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 -- 172.21.15.83:0/4034374851 shutdown_connections 2024-09-19T07:20:05.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 -- 172.21.15.83:0/4034374851 wait complete. 2024-09-19T07:20:05.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 Processor -- start 2024-09-19T07:20:05.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 -- start start 2024-09-19T07:20:05.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4b0073a50 0x7ff4b00bd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4b00ab750 0x7ff4b0151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff4b00c3b70 0x7ff4b01efba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b0153060 con 0x7ff4b00ab750 2024-09-19T07:20:05.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b00ab350 con 0x7ff4b0073a50 2024-09-19T07:20:05.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b01531e0 con 0x7ff4b00c3b70 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4af7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4b00ab750 0x7ff4b0151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4af7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4b00ab750 0x7ff4b0151030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49722/0 (socket says 172.21.15.83:49722) 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4af7fe640 1 -- 172.21.15.83:0/3986066621 learned_addr learned my addr 172.21.15.83:0/3986066621 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4affff640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4b0073a50 0x7ff4b00bd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b4dcb640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff4b00c3b70 0x7ff4b01efba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4af7fe640 1 -- 172.21.15.83:0/3986066621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff4b00c3b70 msgr2=0x7ff4b01efba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4af7fe640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff4b00c3b70 0x7ff4b01efba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4af7fe640 1 -- 172.21.15.83:0/3986066621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4b0073a50 msgr2=0x7ff4b00bd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4af7fe640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4b0073a50 0x7ff4b00bd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4af7fe640 1 -- 172.21.15.83:0/3986066621 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b0150b50 con 0x7ff4b00ab750 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b4dcb640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff4b00c3b70 0x7ff4b01efba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4af7fe640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4b00ab750 0x7ff4b0151030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00ae040 tx=0x7ff4b0074160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:05.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4acff9640 1 -- 172.21.15.83:0/3986066621 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a00b0040 con 0x7ff4b00ab750 2024-09-19T07:20:05.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4acff9640 1 -- 172.21.15.83:0/3986066621 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a00b4420 con 0x7ff4b00ab750 2024-09-19T07:20:05.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 -- 172.21.15.83:0/3986066621 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b00bc740 con 0x7ff4b00ab750 2024-09-19T07:20:05.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4acff9640 1 -- 172.21.15.83:0/3986066621 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a00b4810 con 0x7ff4b00ab750 2024-09-19T07:20:05.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.722+0000 7ff4b682e640 1 -- 172.21.15.83:0/3986066621 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b0152b20 con 0x7ff4b00ab750 2024-09-19T07:20:05.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.726+0000 7ff4b682e640 1 -- 172.21.15.83:0/3986066621 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b00abb30 con 0x7ff4b00ab750 2024-09-19T07:20:05.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.726+0000 7ff4acff9640 1 -- 172.21.15.83:0/3986066621 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff4a00b49b0 con 0x7ff4b00ab750 2024-09-19T07:20:05.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.726+0000 7ff4acff9640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff49c0678f0 0x7ff49c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:05.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.726+0000 7ff4acff9640 1 -- 172.21.15.83:0/3986066621 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7ff4a011f9d0 con 0x7ff4b00ab750 2024-09-19T07:20:05.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.726+0000 7ff4affff640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff49c0678f0 0x7ff49c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:05.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.726+0000 7ff4affff640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff49c0678f0 0x7ff49c069db0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff4b00c1710 tx=0x7ff4a4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:05.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.730+0000 7ff4acff9640 1 -- 172.21.15.83:0/3986066621 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff4a0016110 con 0x7ff4b00ab750 2024-09-19T07:20:05.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.894+0000 7ff4b682e640 1 -- 172.21.15.83:0/3986066621 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff4b00abb30 con 0x7ff4b00ab750 2024-09-19T07:20:05.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.898+0000 7ff4acff9640 1 -- 172.21.15.83:0/3986066621 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+20451 (secure 0 0 0) 0x7ff4a00f0390 con 0x7ff4b00ab750 2024-09-19T07:20:05.899 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:05.899 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:41.334020+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:20:05.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.898+0000 7ff4b682e640 1 -- 172.21.15.83:0/3986066621 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff49c0678f0 msgr2=0x7ff49c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.898+0000 7ff4b682e640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff49c0678f0 0x7ff49c069db0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff4b00c1710 tx=0x7ff4a4008040 comp rx=0 tx=0).stop 2024-09-19T07:20:05.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.898+0000 7ff4b682e640 1 -- 172.21.15.83:0/3986066621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4b00ab750 msgr2=0x7ff4b0151030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:05.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.898+0000 7ff4b682e640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4b00ab750 0x7ff4b0151030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00ae040 tx=0x7ff4b0074160 comp rx=0 tx=0).stop 2024-09-19T07:20:05.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.902+0000 7ff4b682e640 1 -- 172.21.15.83:0/3986066621 shutdown_connections 2024-09-19T07:20:05.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.902+0000 7ff4b682e640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff49c0678f0 0x7ff49c069db0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.902+0000 7ff4b682e640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff4b00c3b70 0x7ff4b01efba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.902+0000 7ff4b682e640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4b00ab750 0x7ff4b0151030 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.902+0000 7ff4b682e640 1 --2- 172.21.15.83:0/3986066621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4b0073a50 0x7ff4b00bd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:05.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.902+0000 7ff4b682e640 1 -- 172.21.15.83:0/3986066621 >> 172.21.15.83:0/3986066621 conn(0x7ff4b006bc80 msgr2=0x7ff4b015c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:05.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.902+0000 7ff4b682e640 1 -- 172.21.15.83:0/3986066621 shutdown_connections 2024-09-19T07:20:05.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:05.902+0000 7ff4b682e640 1 -- 172.21.15.83:0/3986066621 wait complete. 2024-09-19T07:20:05.920 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:05.921 INFO:tasks.mds_thrash.fs.[a]:starting mds_do_thrash for fs a 2024-09-19T07:20:05.921 INFO:tasks.mds_thrash.fs.[a]:waiting for 43 secs before thrashing 2024-09-19T07:20:06.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92bb5a5640 1 Processor -- start 2024-09-19T07:20:06.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92bb5a5640 1 -- start start 2024-09-19T07:20:06.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92bb5a5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92b4150290 0x7f92b40c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:06.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92bb5a5640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f92b4152b20 0x7f92b40c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:06.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92bb5a5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92b4150670 con 0x7f92b40c45c0 2024-09-19T07:20:06.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92bb5a5640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f92b40716c0 con 0x7f92b4152b20 2024-09-19T07:20:06.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92bb5a5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f92b40c3920 con 0x7f92b4150290 2024-09-19T07:20:06.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92b8b40640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f92b4152b20 0x7f92b40c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:06.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92b9341640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92b4150290 0x7f92b40c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:06.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92b9b42640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f92b40c45c0 0x7f92b4153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:41324/0 (socket says 172.21.15.83:41324) 2024-09-19T07:20:06.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92b9b42640 1 -- 172.21.15.83:0/898128253 learned_addr learned my addr 172.21.15.83:0/898128253 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:06.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92b9341640 1 -- 172.21.15.83:0/898128253 >> v2:172.21.15.184:3300/0 conn(0x7f92b4152b20 msgr2=0x7f92b40c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:06.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92b9341640 1 --2- 172.21.15.83:0/898128253 >> v2:172.21.15.184:3300/0 conn(0x7f92b4152b20 0x7f92b40c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92b9341640 1 -- 172.21.15.83:0/898128253 >> v1:172.21.15.184:6789/0 conn(0x7f92b40c45c0 legacy=0x7f92b4153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:06.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.034+0000 7f92b9341640 1 -- 172.21.15.83:0/898128253 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b40c2880 con 0x7f92b4150290 2024-09-19T07:20:06.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92b9341640 1 --2- 172.21.15.83:0/898128253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92b4150290 0x7f92b40c1ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f92b00097c0 tx=0x7f92b001e6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4dd0ea76668b9a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:06.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92abfff640 1 -- 172.21.15.83:0/898128253 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b00ac430 con 0x7f92b4150290 2024-09-19T07:20:06.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92abfff640 1 -- 172.21.15.83:0/898128253 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b00b1070 con 0x7f92b4150290 2024-09-19T07:20:06.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92abfff640 1 -- 172.21.15.83:0/898128253 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b00ac7f0 con 0x7f92b4150290 2024-09-19T07:20:06.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 -- 172.21.15.83:0/898128253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92b4150290 msgr2=0x7f92b40c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 --2- 172.21.15.83:0/898128253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92b4150290 0x7f92b40c1ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f92b00097c0 tx=0x7f92b001e6b0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 -- 172.21.15.83:0/898128253 shutdown_connections 2024-09-19T07:20:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 --2- 172.21.15.83:0/898128253 >> v2:172.21.15.184:3300/0 conn(0x7f92b4152b20 0x7f92b40c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 --2- 172.21.15.83:0/898128253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92b4150290 0x7f92b40c1ff0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 -- 172.21.15.83:0/898128253 >> 172.21.15.83:0/898128253 conn(0x7f92b406bc80 msgr2=0x7f92b406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 -- 172.21.15.83:0/898128253 shutdown_connections 2024-09-19T07:20:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 -- 172.21.15.83:0/898128253 wait complete. 2024-09-19T07:20:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 Processor -- start 2024-09-19T07:20:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 -- start start 2024-09-19T07:20:06.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92b40c45c0 0x7f92b414f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:06.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92b4150290 0x7f92b4072380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:06.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92b4152b20 0x7f92b40728c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:06.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f92b40c2530 con 0x7f92b4150290 2024-09-19T07:20:06.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f92b40c3120 con 0x7f92b40c45c0 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.038+0000 7f92bb5a5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f92b406e6b0 con 0x7f92b4152b20 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b8b40640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92b4150290 0x7f92b4072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b8b40640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92b4150290 0x7f92b4072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49732/0 (socket says 172.21.15.83:49732) 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b8b40640 1 -- 172.21.15.83:0/911551189 learned_addr learned my addr 172.21.15.83:0/911551189 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b9b42640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92b4152b20 0x7f92b40728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b9341640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92b40c45c0 0x7f92b414f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b8b40640 1 -- 172.21.15.83:0/911551189 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92b4152b20 msgr2=0x7f92b40728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b8b40640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92b4152b20 0x7f92b40728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b8b40640 1 -- 172.21.15.83:0/911551189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92b40c45c0 msgr2=0x7f92b414f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b8b40640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92b40c45c0 0x7f92b414f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b8b40640 1 -- 172.21.15.83:0/911551189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b41eb1c0 con 0x7f92b4150290 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b8b40640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92b4150290 0x7f92b4072380 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f92a400e840 tx=0x7f92a400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:06.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92a9ffb640 1 -- 172.21.15.83:0/911551189 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a4017070 con 0x7f92b4150290 2024-09-19T07:20:06.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92a9ffb640 1 -- 172.21.15.83:0/911551189 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92a4004020 con 0x7f92b4150290 2024-09-19T07:20:06.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92bb5a5640 1 -- 172.21.15.83:0/911551189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b4073750 con 0x7f92b4150290 2024-09-19T07:20:06.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92a9ffb640 1 -- 172.21.15.83:0/911551189 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a40043f0 con 0x7f92b4150290 2024-09-19T07:20:06.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92bb5a5640 1 -- 172.21.15.83:0/911551189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b4073d10 con 0x7f92b4150290 2024-09-19T07:20:06.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92bb5a5640 1 -- 172.21.15.83:0/911551189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92b406eb20 con 0x7f92b4150290 2024-09-19T07:20:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92a9ffb640 1 -- 172.21.15.83:0/911551189 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f92a400c940 con 0x7f92b4150290 2024-09-19T07:20:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92a9ffb640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f92980678f0 0x7f9298069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92a9ffb640 1 -- 172.21.15.83:0/911551189 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f92a4086100 con 0x7f92b4150290 2024-09-19T07:20:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b9341640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f92980678f0 0x7f9298069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.042+0000 7f92b9341640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f92980678f0 0x7f9298069db0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f92b00097c0 tx=0x7f92b00a9000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:06.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.046+0000 7f92a9ffb640 1 -- 172.21.15.83:0/911551189 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f92a40569a0 con 0x7f92b4150290 2024-09-19T07:20:06.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.230+0000 7f92bb5a5640 1 -- 172.21.15.83:0/911551189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92b414b500 con 0x7f92b4150290 2024-09-19T07:20:06.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.230+0000 7f92a9ffb640 1 -- 172.21.15.83:0/911551189 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f92a40562a0 con 0x7f92b4150290 2024-09-19T07:20:06.234 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:06.234 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:06.234 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:06.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.234+0000 7f92bb5a5640 1 -- 172.21.15.83:0/911551189 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f92980678f0 msgr2=0x7f9298069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:06.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.234+0000 7f92bb5a5640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f92980678f0 0x7f9298069db0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f92b00097c0 tx=0x7f92b00a9000 comp rx=0 tx=0).stop 2024-09-19T07:20:06.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.234+0000 7f92bb5a5640 1 -- 172.21.15.83:0/911551189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92b4150290 msgr2=0x7f92b4072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:06.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.234+0000 7f92bb5a5640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92b4150290 0x7f92b4072380 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f92a400e840 tx=0x7f92a400ed00 comp rx=0 tx=0).stop 2024-09-19T07:20:06.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.234+0000 7f92bb5a5640 1 -- 172.21.15.83:0/911551189 shutdown_connections 2024-09-19T07:20:06.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.234+0000 7f92bb5a5640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f92980678f0 0x7f9298069db0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.234+0000 7f92bb5a5640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92b4152b20 0x7f92b40728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.234+0000 7f92bb5a5640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92b4150290 0x7f92b4072380 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.234+0000 7f92bb5a5640 1 --2- 172.21.15.83:0/911551189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92b40c45c0 0x7f92b414f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.234+0000 7f92bb5a5640 1 -- 172.21.15.83:0/911551189 >> 172.21.15.83:0/911551189 conn(0x7f92b406bc80 msgr2=0x7f92b406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:06.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.234+0000 7f92bb5a5640 1 -- 172.21.15.83:0/911551189 shutdown_connections 2024-09-19T07:20:06.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.234+0000 7f92bb5a5640 1 -- 172.21.15.83:0/911551189 wait complete. 2024-09-19T07:20:06.251 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:20:06.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821f3c3640 1 Processor -- start 2024-09-19T07:20:06.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821f3c3640 1 -- start start 2024-09-19T07:20:06.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821f3c3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f8218155d70 0x7f82180c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:06.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821f3c3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f82181592f0 0x7f8218157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:06.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821f3c3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8218156200 con 0x7f82180c6120 2024-09-19T07:20:06.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821f3c3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f82180c9930 con 0x7f82181592f0 2024-09-19T07:20:06.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821f3c3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f82181564e0 con 0x7f8218155d70 2024-09-19T07:20:06.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821d960640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f82181592f0 0x7f8218157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:06.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821d960640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f82181592f0 0x7f8218157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49746/0 (socket says 172.21.15.83:49746) 2024-09-19T07:20:06.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821d960640 1 -- 172.21.15.83:0/1691724065 learned_addr learned my addr 172.21.15.83:0/1691724065 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:06.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821c95e640 1 --2- 172.21.15.83:0/1691724065 >> v2:172.21.15.184:3300/0 conn(0x7f8218155d70 0x7f82180c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:06.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821d960640 1 -- 172.21.15.83:0/1691724065 >> v2:172.21.15.184:3300/0 conn(0x7f8218155d70 msgr2=0x7f82180c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:06.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821d960640 1 --2- 172.21.15.83:0/1691724065 >> v2:172.21.15.184:3300/0 conn(0x7f8218155d70 0x7f82180c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821d960640 1 -- 172.21.15.83:0/1691724065 >> v1:172.21.15.184:6789/0 conn(0x7f82180c6120 legacy=0x7f82181575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:06.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.370+0000 7f821d960640 1 -- 172.21.15.83:0/1691724065 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82181567c0 con 0x7f82181592f0 2024-09-19T07:20:06.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821d960640 1 --2- 172.21.15.83:0/1691724065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82181592f0 0x7f8218157ce0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f821806cca0 tx=0x7f821001c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab391576b47e9493 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:06.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f820ffff640 1 -- 172.21.15.83:0/1691724065 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82100a5670 con 0x7f82181592f0 2024-09-19T07:20:06.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f820ffff640 1 -- 172.21.15.83:0/1691724065 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82100a5810 con 0x7f82181592f0 2024-09-19T07:20:06.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f820ffff640 1 -- 172.21.15.83:0/1691724065 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82100a5be0 con 0x7f82181592f0 2024-09-19T07:20:06.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 -- 172.21.15.83:0/1691724065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82181592f0 msgr2=0x7f8218157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:06.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 --2- 172.21.15.83:0/1691724065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82181592f0 0x7f8218157ce0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f821806cca0 tx=0x7f821001c9b0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 -- 172.21.15.83:0/1691724065 shutdown_connections 2024-09-19T07:20:06.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 --2- 172.21.15.83:0/1691724065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82181592f0 0x7f8218157ce0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 --2- 172.21.15.83:0/1691724065 >> v2:172.21.15.184:3300/0 conn(0x7f8218155d70 0x7f82180c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 -- 172.21.15.83:0/1691724065 >> 172.21.15.83:0/1691724065 conn(0x7f821806bc80 msgr2=0x7f821806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:06.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 -- 172.21.15.83:0/1691724065 shutdown_connections 2024-09-19T07:20:06.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 -- 172.21.15.83:0/1691724065 wait complete. 2024-09-19T07:20:06.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 Processor -- start 2024-09-19T07:20:06.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 -- start start 2024-09-19T07:20:06.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82180c6120 0x7f821814b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82181592f0 0x7f82181ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82180b1700 0x7f82181eec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8218153860 con 0x7f82181592f0 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8218162ea0 con 0x7f82180b1700 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f821814bd00 con 0x7f82180c6120 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821c95e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82181592f0 0x7f82181ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821c95e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82181592f0 0x7f82181ee6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49754/0 (socket says 172.21.15.83:49754) 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821c95e640 1 -- 172.21.15.83:0/808628840 learned_addr learned my addr 172.21.15.83:0/808628840 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821c95e640 1 -- 172.21.15.83:0/808628840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82180c6120 msgr2=0x7f821814b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821c95e640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82180c6120 0x7f821814b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821c95e640 1 -- 172.21.15.83:0/808628840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82180b1700 msgr2=0x7f82181eec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821d960640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82180b1700 0x7f82181eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821c95e640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82180b1700 0x7f82181eec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821c95e640 1 -- 172.21.15.83:0/808628840 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82180adce0 con 0x7f82181592f0 2024-09-19T07:20:06.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821c95e640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82181592f0 0x7f82181ee6c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f820800d830 tx=0x7f820800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:06.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f820dffb640 1 -- 172.21.15.83:0/808628840 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820800b6c0 con 0x7f82181592f0 2024-09-19T07:20:06.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f820dffb640 1 -- 172.21.15.83:0/808628840 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8208019070 con 0x7f82181592f0 2024-09-19T07:20:06.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f820dffb640 1 -- 172.21.15.83:0/808628840 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820800bae0 con 0x7f82181592f0 2024-09-19T07:20:06.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 -- 172.21.15.83:0/808628840 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82180c0850 con 0x7f82181592f0 2024-09-19T07:20:06.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.374+0000 7f821f3c3640 1 -- 172.21.15.83:0/808628840 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f821814bf00 con 0x7f82181592f0 2024-09-19T07:20:06.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.378+0000 7f821f3c3640 1 -- 172.21.15.83:0/808628840 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82180c0640 con 0x7f82181592f0 2024-09-19T07:20:06.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.378+0000 7f820dffb640 1 -- 172.21.15.83:0/808628840 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f820801e460 con 0x7f82181592f0 2024-09-19T07:20:06.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.378+0000 7f820dffb640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f81fc0678f0 0x7f81fc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:06.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.378+0000 7f820dffb640 1 -- 172.21.15.83:0/808628840 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f82080866e0 con 0x7f82181592f0 2024-09-19T07:20:06.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.378+0000 7f821d15f640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f81fc0678f0 0x7f81fc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:06.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.378+0000 7f821d15f640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f81fc0678f0 0x7f81fc069db0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8214003cf0 tx=0x7f82140079c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:06.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.382+0000 7f820dffb640 1 -- 172.21.15.83:0/808628840 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f8208056ef0 con 0x7f82181592f0 2024-09-19T07:20:06.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.546+0000 7f821f3c3640 1 -- 172.21.15.83:0/808628840 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82180c6500 con 0x7f82181592f0 2024-09-19T07:20:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.546+0000 7f820dffb640 1 -- 172.21.15.83:0/808628840 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+20451 (secure 0 0 0) 0x7f820805d0c0 con 0x7f82181592f0 2024-09-19T07:20:06.548 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:06.548 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:19:41.334020+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:20:06.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.550+0000 7f821f3c3640 1 -- 172.21.15.83:0/808628840 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f81fc0678f0 msgr2=0x7f81fc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:06.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.550+0000 7f821f3c3640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f81fc0678f0 0x7f81fc069db0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8214003cf0 tx=0x7f82140079c0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.550+0000 7f821f3c3640 1 -- 172.21.15.83:0/808628840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82181592f0 msgr2=0x7f82181ee6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:06.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.550+0000 7f821f3c3640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82181592f0 0x7f82181ee6c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f820800d830 tx=0x7f820800dcf0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.550+0000 7f821f3c3640 1 -- 172.21.15.83:0/808628840 shutdown_connections 2024-09-19T07:20:06.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.550+0000 7f821f3c3640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f81fc0678f0 0x7f81fc069db0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.550+0000 7f821f3c3640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82180b1700 0x7f82181eec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.550+0000 7f821f3c3640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82181592f0 0x7f82181ee6c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.550+0000 7f821f3c3640 1 --2- 172.21.15.83:0/808628840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82180c6120 0x7f821814b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:06.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.550+0000 7f821f3c3640 1 -- 172.21.15.83:0/808628840 >> 172.21.15.83:0/808628840 conn(0x7f821806bc80 msgr2=0x7f8218159700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:06.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.550+0000 7f821f3c3640 1 -- 172.21.15.83:0/808628840 shutdown_connections 2024-09-19T07:20:06.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:06.550+0000 7f821f3c3640 1 -- 172.21.15.83:0/808628840 wait complete. 2024-09-19T07:20:06.567 DEBUG:tasks.mds_thrash:Yielding 2024-09-19T07:20:06.568 INFO:teuthology.run_tasks:Running task workunit... 2024-09-19T07:20:06.581 INFO:tasks.workunit:Pulling workunits from ref 6fad5dd73c93545e52f57f681e140a9c3529c11e 2024-09-19T07:20:06.582 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-19T07:20:06.582 INFO:tasks.workunit:timeout=3h 2024-09-19T07:20:06.582 INFO:tasks.workunit:cleanup=True 2024-09-19T07:20:06.582 DEBUG:teuthology.orchestra.run.smithi083:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-19T07:20:06.582 INFO:tasks.mds_thrash.fs.[b]:starting mds_do_thrash for fs b 2024-09-19T07:20:06.583 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2024-09-19T07:20:06.588 INFO:teuthology.orchestra.run.smithi083.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-19T07:20:06.588 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-09-19T07:20:06.588 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 2ch/44d Inode: 1 Links: 2 2024-09-19T07:20:06.588 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:20:06.588 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-19T07:20:06.588 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-19 07:19:32.285386906 +0000 2024-09-19T07:20:06.588 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-19 07:20:01.791860570 +0000 2024-09-19T07:20:06.589 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-09-19T07:20:06.589 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-19T07:20:06.589 DEBUG:teuthology.orchestra.run.smithi083:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-19T07:20:06.646 DEBUG:teuthology.orchestra.run.smithi184:> stat -- /home/ubuntu/cephtest/mnt.1 2024-09-19T07:20:06.655 INFO:teuthology.orchestra.run.smithi184.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-09-19T07:20:06.655 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-09-19T07:20:06.655 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 2dh/45d Inode: 1 Links: 2 2024-09-19T07:20:06.655 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T07:20:06.655 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-19T07:20:06.655 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-19 07:19:40.420182118 +0000 2024-09-19T07:20:06.656 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-19 07:20:01.855294732 +0000 2024-09-19T07:20:06.656 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2024-09-19T07:20:06.656 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-09-19T07:20:06.656 DEBUG:teuthology.orchestra.run.smithi184:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-09-19T07:20:06.723 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 6fad5dd73c93545e52f57f681e140a9c3529c11e 2024-09-19T07:20:06.723 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 6fad5dd73c93545e52f57f681e140a9c3529c11e 2024-09-19T07:20:06.730 INFO:tasks.workunit.client.0.smithi083.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-19T07:20:06.773 INFO:tasks.workunit.client.1.smithi184.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-09-19T07:20:29.839 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:20:29.842 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:20:40.584 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:40.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.713+0000 7fdb84114640 1 Processor -- start 2024-09-19T07:20:40.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.713+0000 7fdb84114640 1 -- start start 2024-09-19T07:20:40.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb84114640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fdb7c06ffe0 0x7fdb7c0c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:40.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb84114640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb7c0c26c0 con 0x7fdb7c159af0 2024-09-19T07:20:40.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb84114640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb7c0c29a0 con 0x7fdb7c1535d0 2024-09-19T07:20:40.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb84114640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fdb7c0707e0 con 0x7fdb7c06ffe0 2024-09-19T07:20:40.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb826b1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdb7c159af0 0x7fdb7c0c2f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43194/0 (socket says 172.21.15.83:43194) 2024-09-19T07:20:40.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb826b1640 1 -- 172.21.15.83:0/605827056 learned_addr learned my addr 172.21.15.83:0/605827056 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:40.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb816af640 1 --2- 172.21.15.83:0/605827056 >> v2:172.21.15.184:3300/0 conn(0x7fdb7c06ffe0 0x7fdb7c0c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:40.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4042121196 0 0) 0x7fdb7c0c26c0 con 0x7fdb7c159af0 2024-09-19T07:20:40.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb68003530 con 0x7fdb7c159af0 2024-09-19T07:20:40.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 66339868 0 0) 0x7fdb68003530 con 0x7fdb7c159af0 2024-09-19T07:20:40.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 >> v2:172.21.15.184:3300/0 conn(0x7fdb7c06ffe0 msgr2=0x7fdb7c0c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:40.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 --2- 172.21.15.83:0/605827056 >> v2:172.21.15.184:3300/0 conn(0x7fdb7c06ffe0 0x7fdb7c0c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:40.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 >> v1:172.21.15.184:6789/0 conn(0x7fdb7c1535d0 legacy=0x7fdb7c155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:40.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c1592f0 con 0x7fdb7c159af0 2024-09-19T07:20:40.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fdb74001a00 con 0x7fdb7c159af0 2024-09-19T07:20:40.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 >> v1:172.21.15.83:6789/0 conn(0x7fdb7c159af0 legacy=0x7fdb7c0c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:40.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 --2- 172.21.15.83:0/605827056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb680039c0 0x7fdb68003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:40.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 --2- 172.21.15.83:0/605827056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb68004810 0x7fdb68004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 --2- 172.21.15.83:0/605827056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb680052f0 0x7fdb6800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6800c190 con 0x7fdb68004810 2024-09-19T07:20:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6800c340 con 0x7fdb680039c0 2024-09-19T07:20:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdb6800c540 con 0x7fdb680052f0 2024-09-19T07:20:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb816af640 1 --2- 172.21.15.83:0/605827056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb68004810 0x7fdb68004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb81eb0640 1 --2- 172.21.15.83:0/605827056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb680039c0 0x7fdb68003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb816af640 1 -- 172.21.15.83:0/605827056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb680052f0 msgr2=0x7fdb6800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb816af640 1 --2- 172.21.15.83:0/605827056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb680052f0 0x7fdb6800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb816af640 1 -- 172.21.15.83:0/605827056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb680039c0 msgr2=0x7fdb68003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb816af640 1 --2- 172.21.15.83:0/605827056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb680039c0 0x7fdb68003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:40.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb816af640 1 -- 172.21.15.83:0/605827056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb68003530 con 0x7fdb68004810 2024-09-19T07:20:40.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.717+0000 7fdb826b1640 1 -- 172.21.15.83:0/605827056 reap_dead start 2024-09-19T07:20:40.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb816af640 1 --2- 172.21.15.83:0/605827056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb68004810 0x7fdb68004c90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c009a90 tx=0x7fdb6c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b247f76e5c81113 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:40.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c009690 con 0x7fdb68004810 2024-09-19T07:20:40.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb6c0a8450 con 0x7fdb68004810 2024-09-19T07:20:40.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb80eae640 1 -- 172.21.15.83:0/605827056 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c0a8820 con 0x7fdb68004810 2024-09-19T07:20:40.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 -- 172.21.15.83:0/605827056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb68004810 msgr2=0x7fdb68004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:40.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 --2- 172.21.15.83:0/605827056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb68004810 0x7fdb68004c90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c009a90 tx=0x7fdb6c01ca10 comp rx=0 tx=0).stop 2024-09-19T07:20:40.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 -- 172.21.15.83:0/605827056 shutdown_connections 2024-09-19T07:20:40.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 --2- 172.21.15.83:0/605827056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb68004810 0x7fdb68004c90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:40.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 -- 172.21.15.83:0/605827056 >> 172.21.15.83:0/605827056 conn(0x7fdb7c0c5060 msgr2=0x7fdb7c0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 -- 172.21.15.83:0/605827056 shutdown_connections 2024-09-19T07:20:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 -- 172.21.15.83:0/605827056 wait complete. 2024-09-19T07:20:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 Processor -- start 2024-09-19T07:20:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 -- start start 2024-09-19T07:20:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb68004810 0x7fdb7c0c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb7c073eb0 0x7fdb7c072720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb7c072c60 0x7fdb7c1f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c0c2c80 con 0x7fdb7c072c60 2024-09-19T07:20:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c15a110 con 0x7fdb7c073eb0 2024-09-19T07:20:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb84114640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdb7c0c3770 con 0x7fdb68004810 2024-09-19T07:20:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb826b1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb7c072c60 0x7fdb7c1f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb826b1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb7c072c60 0x7fdb7c1f1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37104/0 (socket says 172.21.15.83:37104) 2024-09-19T07:20:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb826b1640 1 -- 172.21.15.83:0/713260481 learned_addr learned my addr 172.21.15.83:0/713260481 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb816af640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb7c073eb0 0x7fdb7c072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb81eb0640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb68004810 0x7fdb7c0c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb826b1640 1 -- 172.21.15.83:0/713260481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb68004810 msgr2=0x7fdb7c0c73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb826b1640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb68004810 0x7fdb7c0c73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb826b1640 1 -- 172.21.15.83:0/713260481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb7c073eb0 msgr2=0x7fdb7c072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb826b1640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb7c073eb0 0x7fdb7c072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.721+0000 7fdb826b1640 1 -- 172.21.15.83:0/713260481 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c0ad0b0 con 0x7fdb7c072c60 2024-09-19T07:20:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.725+0000 7fdb826b1640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb7c072c60 0x7fdb7c1f1e60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0c4af0 tx=0x7fdb7400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.725+0000 7fdb727fc640 1 -- 172.21.15.83:0/713260481 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb74004d70 con 0x7fdb7c072c60 2024-09-19T07:20:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.725+0000 7fdb727fc640 1 -- 172.21.15.83:0/713260481 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb7400a610 con 0x7fdb7c072c60 2024-09-19T07:20:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.725+0000 7fdb84114640 1 -- 172.21.15.83:0/713260481 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb7c09c1d0 con 0x7fdb7c072c60 2024-09-19T07:20:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.725+0000 7fdb727fc640 1 -- 172.21.15.83:0/713260481 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7400aa00 con 0x7fdb7c072c60 2024-09-19T07:20:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.725+0000 7fdb84114640 1 -- 172.21.15.83:0/713260481 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb7c075cc0 con 0x7fdb7c072c60 2024-09-19T07:20:40.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.725+0000 7fdb84114640 1 -- 172.21.15.83:0/713260481 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb7c074f90 con 0x7fdb7c072c60 2024-09-19T07:20:40.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.725+0000 7fdb727fc640 1 -- 172.21.15.83:0/713260481 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdb7400aba0 con 0x7fdb7c072c60 2024-09-19T07:20:40.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.725+0000 7fdb727fc640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdb600678f0 0x7fdb60069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:40.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.725+0000 7fdb727fc640 1 -- 172.21.15.83:0/713260481 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7fdb740855c0 con 0x7fdb7c072c60 2024-09-19T07:20:40.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.729+0000 7fdb81eb0640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdb600678f0 0x7fdb60069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:40.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.729+0000 7fdb81eb0640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdb600678f0 0x7fdb60069db0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdb78007fd0 tx=0x7fdb78009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:40.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:40.729+0000 7fdb727fc640 1 -- 172.21.15.83:0/713260481 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdb740557a0 con 0x7fdb7c072c60 2024-09-19T07:20:41.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 -- 172.21.15.83:0/713260481 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb7c0704e0 con 0x7fdb7c072c60 2024-09-19T07:20:41.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb727fc640 1 -- 172.21.15.83:0/713260481 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7fdb74059aa0 con 0x7fdb7c072c60 2024-09-19T07:20:41.013 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:41.013 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:41.015 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:41.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 -- 172.21.15.83:0/713260481 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdb600678f0 msgr2=0x7fdb60069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:41.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdb600678f0 0x7fdb60069db0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdb78007fd0 tx=0x7fdb78009040 comp rx=0 tx=0).stop 2024-09-19T07:20:41.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 -- 172.21.15.83:0/713260481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb7c072c60 msgr2=0x7fdb7c1f1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:41.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb7c072c60 0x7fdb7c1f1e60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0c4af0 tx=0x7fdb7400cb90 comp rx=0 tx=0).stop 2024-09-19T07:20:41.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 -- 172.21.15.83:0/713260481 shutdown_connections 2024-09-19T07:20:41.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdb600678f0 0x7fdb60069db0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb7c072c60 0x7fdb7c1f1e60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb7c073eb0 0x7fdb7c072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 --2- 172.21.15.83:0/713260481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb68004810 0x7fdb7c0c73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 -- 172.21.15.83:0/713260481 >> 172.21.15.83:0/713260481 conn(0x7fdb7c0c5060 msgr2=0x7fdb7c0c5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:41.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 -- 172.21.15.83:0/713260481 shutdown_connections 2024-09-19T07:20:41.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.009+0000 7fdb84114640 1 -- 172.21.15.83:0/713260481 wait complete. 2024-09-19T07:20:41.034 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-09-19T07:20:41.034 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-19T07:20:41.034 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:20:41.046 INFO:tasks.ceph.mds.b:Stopped 2024-09-19T07:20:41.046 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:41.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.145+0000 7f6fe7616640 1 Processor -- start 2024-09-19T07:20:41.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe7616640 1 -- start start 2024-09-19T07:20:41.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe7616640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6fe00c4880 0x7f6fe00c1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:41.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe7616640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c2330 0x7f6fe0152f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe7616640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe0159430 con 0x7f6fe0153470 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe7616640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6fe015a040 con 0x7f6fe00c4880 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe7616640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe00c2c10 con 0x7f6fe00c2330 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe4bb1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c2330 0x7f6fe0152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe5bb3640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f6fe0153470 0x7f6fe0155e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:54320/0 (socket says 172.21.15.83:54320) 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe53b2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6fe00c4880 0x7f6fe00c1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe5bb3640 1 -- 172.21.15.83:0/301204677 learned_addr learned my addr 172.21.15.83:0/301204677 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe4bb1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c2330 0x7f6fe0152f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:37574/0 (socket says 172.21.15.83:37574) 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe4bb1640 1 -- 172.21.15.83:0/301204677 >> v2:172.21.15.184:3300/0 conn(0x7f6fe00c4880 msgr2=0x7f6fe00c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe4bb1640 1 --2- 172.21.15.83:0/301204677 >> v2:172.21.15.184:3300/0 conn(0x7f6fe00c4880 0x7f6fe00c1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe4bb1640 1 -- 172.21.15.83:0/301204677 >> v1:172.21.15.184:6789/0 conn(0x7f6fe0153470 legacy=0x7f6fe0155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe4bb1640 1 -- 172.21.15.83:0/301204677 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe0159710 con 0x7f6fe00c2330 2024-09-19T07:20:41.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe53b2640 1 --2- 172.21.15.83:0/301204677 >> v2:172.21.15.184:3300/0 conn(0x7f6fe00c4880 0x7f6fe00c1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:41.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fe4bb1640 1 --2- 172.21.15.83:0/301204677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c2330 0x7f6fe0152f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00097c0 tx=0x7f6fd001c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b68eb78bcd470c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:41.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fd7fff640 1 -- 172.21.15.83:0/301204677 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd00b0070 con 0x7f6fe00c2330 2024-09-19T07:20:41.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fd7fff640 1 -- 172.21.15.83:0/301204677 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd0004230 con 0x7f6fe00c2330 2024-09-19T07:20:41.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.149+0000 7f6fd7fff640 1 -- 172.21.15.83:0/301204677 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0003620 con 0x7f6fe00c2330 2024-09-19T07:20:41.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 -- 172.21.15.83:0/301204677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c2330 msgr2=0x7f6fe0152f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:41.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 --2- 172.21.15.83:0/301204677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c2330 0x7f6fe0152f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00097c0 tx=0x7f6fd001c350 comp rx=0 tx=0).stop 2024-09-19T07:20:41.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 -- 172.21.15.83:0/301204677 shutdown_connections 2024-09-19T07:20:41.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 --2- 172.21.15.83:0/301204677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c2330 0x7f6fe0152f30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 --2- 172.21.15.83:0/301204677 >> v2:172.21.15.184:3300/0 conn(0x7f6fe00c4880 0x7f6fe00c1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 -- 172.21.15.83:0/301204677 >> 172.21.15.83:0/301204677 conn(0x7f6fe00aa910 msgr2=0x7f6fe00c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:41.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 -- 172.21.15.83:0/301204677 shutdown_connections 2024-09-19T07:20:41.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 -- 172.21.15.83:0/301204677 wait complete. 2024-09-19T07:20:41.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 Processor -- start 2024-09-19T07:20:41.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 -- start start 2024-09-19T07:20:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c4880 0x7f6fe01eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fe0153470 0x7f6fe00acc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fe014e9e0 0x7f6fe01f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe0156ab0 con 0x7f6fe014e9e0 2024-09-19T07:20:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe0156930 con 0x7f6fe0153470 2024-09-19T07:20:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe7616640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe00c4c60 con 0x7f6fe00c4880 2024-09-19T07:20:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe53b2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c4880 0x7f6fe01eebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe5bb3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fe014e9e0 0x7f6fe01f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe53b2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c4880 0x7f6fe01eebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:37584/0 (socket says 172.21.15.83:37584) 2024-09-19T07:20:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe4bb1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fe0153470 0x7f6fe00acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe53b2640 1 -- 172.21.15.83:0/3124981046 learned_addr learned my addr 172.21.15.83:0/3124981046 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe5bb3640 1 -- 172.21.15.83:0/3124981046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c4880 msgr2=0x7f6fe01eebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe5bb3640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c4880 0x7f6fe01eebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe5bb3640 1 -- 172.21.15.83:0/3124981046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fe0153470 msgr2=0x7f6fe00acc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe5bb3640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fe0153470 0x7f6fe00acc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe5bb3640 1 -- 172.21.15.83:0/3124981046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe0157a80 con 0x7f6fe014e9e0 2024-09-19T07:20:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe4bb1640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fe0153470 0x7f6fe00acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:41.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.153+0000 7f6fe5bb3640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fe014e9e0 0x7f6fe01f29d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800ca70 tx=0x7f6fd800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:41.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.157+0000 7f6fd5ffb640 1 -- 172.21.15.83:0/3124981046 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8017070 con 0x7f6fe014e9e0 2024-09-19T07:20:41.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.157+0000 7f6fd5ffb640 1 -- 172.21.15.83:0/3124981046 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd800aa90 con 0x7f6fe014e9e0 2024-09-19T07:20:41.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.157+0000 7f6fe7616640 1 -- 172.21.15.83:0/3124981046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe0157d60 con 0x7f6fe014e9e0 2024-09-19T07:20:41.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.157+0000 7f6fd5ffb640 1 -- 172.21.15.83:0/3124981046 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8012400 con 0x7f6fe014e9e0 2024-09-19T07:20:41.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.157+0000 7f6fe7616640 1 -- 172.21.15.83:0/3124981046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe00a9d70 con 0x7f6fe014e9e0 2024-09-19T07:20:41.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.157+0000 7f6fe7616640 1 -- 172.21.15.83:0/3124981046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe0152f30 con 0x7f6fe014e9e0 2024-09-19T07:20:41.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.157+0000 7f6fd5ffb640 1 -- 172.21.15.83:0/3124981046 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6fd801a070 con 0x7f6fe014e9e0 2024-09-19T07:20:41.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.157+0000 7f6fd5ffb640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6fc40678c0 0x7f6fc4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:41.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.157+0000 7f6fd5ffb640 1 -- 172.21.15.83:0/3124981046 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7f6fd8085b10 con 0x7f6fe014e9e0 2024-09-19T07:20:41.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.157+0000 7f6fe53b2640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6fc40678c0 0x7f6fc4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:41.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.157+0000 7f6fe53b2640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6fc40678c0 0x7f6fc4069d80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00a9cc0 tx=0x7f6fdc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:41.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.165+0000 7f6fd5ffb640 1 -- 172.21.15.83:0/3124981046 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6fd8055cf0 con 0x7f6fe014e9e0 2024-09-19T07:20:41.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.545+0000 7f6fe7616640 1 -- 172.21.15.83:0/3124981046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fe00c20f0 con 0x7f6fe014e9e0 2024-09-19T07:20:41.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fd5ffb640 1 -- 172.21.15.83:0/3124981046 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f6fd8059ff0 con 0x7f6fe014e9e0 2024-09-19T07:20:41.553 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:41.553 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fe7616640 1 -- 172.21.15.83:0/3124981046 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6fc40678c0 msgr2=0x7f6fc4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fe7616640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6fc40678c0 0x7f6fc4069d80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00a9cc0 tx=0x7f6fdc009190 comp rx=0 tx=0).stop 2024-09-19T07:20:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fe7616640 1 -- 172.21.15.83:0/3124981046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fe014e9e0 msgr2=0x7f6fe01f29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fe7616640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fe014e9e0 0x7f6fe01f29d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800ca70 tx=0x7f6fd800cf30 comp rx=0 tx=0).stop 2024-09-19T07:20:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fe7616640 1 -- 172.21.15.83:0/3124981046 shutdown_connections 2024-09-19T07:20:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fe7616640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6fc40678c0 0x7f6fc4069d80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fe7616640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fe014e9e0 0x7f6fe01f29d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fe7616640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fe0153470 0x7f6fe00acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fe7616640 1 --2- 172.21.15.83:0/3124981046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fe00c4880 0x7f6fe01eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fe7616640 1 -- 172.21.15.83:0/3124981046 >> 172.21.15.83:0/3124981046 conn(0x7f6fe00aa910 msgr2=0x7f6fe00a8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fe7616640 1 -- 172.21.15.83:0/3124981046 shutdown_connections 2024-09-19T07:20:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:41.549+0000 7f6fe7616640 1 -- 172.21.15.83:0/3124981046 wait complete. 2024-09-19T07:20:41.573 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:20:43.575 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:43.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.725+0000 7fe614072640 1 Processor -- start 2024-09-19T07:20:43.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe614072640 1 -- start start 2024-09-19T07:20:43.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe614072640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe60c0aa4a0 0x7fe60c157720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:43.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe614072640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60c073d90 con 0x7fe60c14dbe0 2024-09-19T07:20:43.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe614072640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60c153b30 con 0x7fe60c0c3ab0 2024-09-19T07:20:43.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe614072640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe60c154700 con 0x7fe60c0aa4a0 2024-09-19T07:20:43.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe61260f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe60c0aa4a0 0x7fe60c157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:43.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe61260f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe60c0aa4a0 0x7fe60c157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37120/0 (socket says 172.21.15.83:37120) 2024-09-19T07:20:43.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe61260f640 1 -- 172.21.15.83:0/263447612 learned_addr learned my addr 172.21.15.83:0/263447612 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:43.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe610e0c640 1 -- 172.21.15.83:0/263447612 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 387812629 0 0) 0x7fe60c073d90 con 0x7fe60c14dbe0 2024-09-19T07:20:43.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe610e0c640 1 -- 172.21.15.83:0/263447612 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5f8003530 con 0x7fe60c14dbe0 2024-09-19T07:20:43.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe610e0c640 1 -- 172.21.15.83:0/263447612 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102257545 0 0) 0x7fe60c153b30 con 0x7fe60c0c3ab0 2024-09-19T07:20:43.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe610e0c640 1 -- 172.21.15.83:0/263447612 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe60c073d90 con 0x7fe60c0c3ab0 2024-09-19T07:20:43.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe61260f640 1 -- 172.21.15.83:0/263447612 >> v1:172.21.15.184:6789/0 conn(0x7fe60c0c3ab0 legacy=0x7fe60c157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:43.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe61260f640 1 -- 172.21.15.83:0/263447612 >> v1:172.21.15.83:6789/0 conn(0x7fe60c14dbe0 legacy=0x7fe60c0c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:43.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe61260f640 1 -- 172.21.15.83:0/263447612 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60c153dd0 con 0x7fe60c0aa4a0 2024-09-19T07:20:43.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.729+0000 7fe610e0c640 1 -- 172.21.15.83:0/263447612 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 754450564 0 0) 0x7fe5f8003530 con 0x7fe60c14dbe0 2024-09-19T07:20:43.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.733+0000 7fe61260f640 1 --2- 172.21.15.83:0/263447612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe60c0aa4a0 0x7fe60c157720 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe60c154880 tx=0x7fe60401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e63dc2662c3017 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:43.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.733+0000 7fe610e0c640 1 -- 172.21.15.83:0/263447612 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe608001680 con 0x7fe60c0aa4a0 2024-09-19T07:20:43.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.733+0000 7fe610e0c640 1 -- 172.21.15.83:0/263447612 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6040090d0 con 0x7fe60c0aa4a0 2024-09-19T07:20:43.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.733+0000 7fe610e0c640 1 -- 172.21.15.83:0/263447612 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6040a5810 con 0x7fe60c0aa4a0 2024-09-19T07:20:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.733+0000 7fe614072640 1 -- 172.21.15.83:0/263447612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe60c0aa4a0 msgr2=0x7fe60c157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.733+0000 7fe614072640 1 --2- 172.21.15.83:0/263447612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe60c0aa4a0 0x7fe60c157720 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe60c154880 tx=0x7fe60401c8a0 comp rx=0 tx=0).stop 2024-09-19T07:20:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.733+0000 7fe614072640 1 -- 172.21.15.83:0/263447612 shutdown_connections 2024-09-19T07:20:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.733+0000 7fe614072640 1 --2- 172.21.15.83:0/263447612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe60c0aa4a0 0x7fe60c157720 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.733+0000 7fe614072640 1 -- 172.21.15.83:0/263447612 >> 172.21.15.83:0/263447612 conn(0x7fe60c0c95a0 msgr2=0x7fe60c0c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.733+0000 7fe614072640 1 -- 172.21.15.83:0/263447612 shutdown_connections 2024-09-19T07:20:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.733+0000 7fe614072640 1 -- 172.21.15.83:0/263447612 wait complete. 2024-09-19T07:20:43.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe614072640 1 Processor -- start 2024-09-19T07:20:43.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe614072640 1 -- start start 2024-09-19T07:20:43.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe614072640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe60c0aa4a0 0x7fe60c14a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:43.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe614072640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe60c14dbe0 0x7fe60c1ec440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:43.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe614072640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe60c1ec980 0x7fe60c143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:43.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe614072640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c0c46b0 con 0x7fe60c14dbe0 2024-09-19T07:20:43.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe61160d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe60c14dbe0 0x7fe60c1ec440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:43.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe61160d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe60c14dbe0 0x7fe60c1ec440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37124/0 (socket says 172.21.15.83:37124) 2024-09-19T07:20:43.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe61160d640 1 -- 172.21.15.83:0/3156479993 learned_addr learned my addr 172.21.15.83:0/3156479993 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:43.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe611e0e640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe60c0aa4a0 0x7fe60c14a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:43.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe614072640 1 -- 172.21.15.83:0/3156479993 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c0741a0 con 0x7fe60c1ec980 2024-09-19T07:20:43.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe61260f640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe60c1ec980 0x7fe60c143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:43.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe614072640 1 -- 172.21.15.83:0/3156479993 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe60c0c4830 con 0x7fe60c0aa4a0 2024-09-19T07:20:43.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe61160d640 1 -- 172.21.15.83:0/3156479993 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe60c0aa4a0 msgr2=0x7fe60c14a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:43.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe61160d640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe60c0aa4a0 0x7fe60c14a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:43.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe61160d640 1 -- 172.21.15.83:0/3156479993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe60c1ec980 msgr2=0x7fe60c143970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:43.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe61160d640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe60c1ec980 0x7fe60c143970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:43.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.737+0000 7fe61160d640 1 -- 172.21.15.83:0/3156479993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60c0738f0 con 0x7fe60c14dbe0 2024-09-19T07:20:43.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.741+0000 7fe61160d640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe60c14dbe0 0x7fe60c1ec440 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc001e60 tx=0x7fe5fc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:43.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.741+0000 7fe6027fc640 1 -- 172.21.15.83:0/3156479993 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0052d0 con 0x7fe60c14dbe0 2024-09-19T07:20:43.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.741+0000 7fe6027fc640 1 -- 172.21.15.83:0/3156479993 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5fc005490 con 0x7fe60c14dbe0 2024-09-19T07:20:43.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.741+0000 7fe614072640 1 -- 172.21.15.83:0/3156479993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe60c1ebf40 con 0x7fe60c14dbe0 2024-09-19T07:20:43.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.741+0000 7fe6027fc640 1 -- 172.21.15.83:0/3156479993 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc011ab0 con 0x7fe60c14dbe0 2024-09-19T07:20:43.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.741+0000 7fe614072640 1 -- 172.21.15.83:0/3156479993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe60c14a0a0 con 0x7fe60c14dbe0 2024-09-19T07:20:43.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.741+0000 7fe614072640 1 -- 172.21.15.83:0/3156479993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe60c0c3fa0 con 0x7fe60c14dbe0 2024-09-19T07:20:43.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.745+0000 7fe6027fc640 1 -- 172.21.15.83:0/3156479993 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe5fc011c50 con 0x7fe60c14dbe0 2024-09-19T07:20:43.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.745+0000 7fe6027fc640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe5f00678c0 0x7fe5f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:43.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.745+0000 7fe6027fc640 1 -- 172.21.15.83:0/3156479993 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7fe5fc089f00 con 0x7fe60c14dbe0 2024-09-19T07:20:43.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.745+0000 7fe611e0e640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe5f00678c0 0x7fe5f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:43.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.745+0000 7fe611e0e640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe5f00678c0 0x7fe5f0069d80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe608001ec0 tx=0x7fe608001a40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:43.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:43.749+0000 7fe6027fc640 1 -- 172.21.15.83:0/3156479993 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe5fc05a040 con 0x7fe60c14dbe0 2024-09-19T07:20:44.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.129+0000 7fe614072640 1 -- 172.21.15.83:0/3156479993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe60c073f80 con 0x7fe60c14dbe0 2024-09-19T07:20:44.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.129+0000 7fe6027fc640 1 -- 172.21.15.83:0/3156479993 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7fe5fc05e340 con 0x7fe60c14dbe0 2024-09-19T07:20:44.134 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:44.134 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:44.134 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:44.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.133+0000 7fe614072640 1 -- 172.21.15.83:0/3156479993 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe5f00678c0 msgr2=0x7fe5f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:44.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.133+0000 7fe614072640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe5f00678c0 0x7fe5f0069d80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe608001ec0 tx=0x7fe608001a40 comp rx=0 tx=0).stop 2024-09-19T07:20:44.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.133+0000 7fe614072640 1 -- 172.21.15.83:0/3156479993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe60c14dbe0 msgr2=0x7fe60c1ec440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:44.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.133+0000 7fe614072640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe60c14dbe0 0x7fe60c1ec440 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc001e60 tx=0x7fe5fc00c990 comp rx=0 tx=0).stop 2024-09-19T07:20:44.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.133+0000 7fe614072640 1 -- 172.21.15.83:0/3156479993 shutdown_connections 2024-09-19T07:20:44.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.133+0000 7fe614072640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe5f00678c0 0x7fe5f0069d80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:44.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.133+0000 7fe614072640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe60c1ec980 0x7fe60c143970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:44.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.133+0000 7fe614072640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe60c14dbe0 0x7fe60c1ec440 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc001e60 tx=0x7fe5fc00c990 comp rx=0 tx=0).stop 2024-09-19T07:20:44.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.133+0000 7fe614072640 1 --2- 172.21.15.83:0/3156479993 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe60c0aa4a0 0x7fe60c14a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:44.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.133+0000 7fe614072640 1 -- 172.21.15.83:0/3156479993 >> 172.21.15.83:0/3156479993 conn(0x7fe60c0c95a0 msgr2=0x7fe60c0c6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:44.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.133+0000 7fe614072640 1 -- 172.21.15.83:0/3156479993 shutdown_connections 2024-09-19T07:20:44.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:44.133+0000 7fe614072640 1 -- 172.21.15.83:0/3156479993 wait complete. 2024-09-19T07:20:44.159 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:20:46.160 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:46.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.313+0000 7f099df1d640 1 Processor -- start 2024-09-19T07:20:46.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.313+0000 7f099df1d640 1 -- start start 2024-09-19T07:20:46.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f099df1d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09980ad150 0x7f09980c2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:46.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f099df1d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f09980c2d80 0x7f0998150c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:46.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f099df1d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f099806d630 con 0x7f09980abda0 2024-09-19T07:20:46.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f099df1d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f0998154ae0 con 0x7f09980c2d80 2024-09-19T07:20:46.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f099df1d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f09980acc00 con 0x7f09980ad150 2024-09-19T07:20:46.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09977fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f09980abda0 0x7f09980ac180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43206/0 (socket says 172.21.15.83:43206) 2024-09-19T07:20:46.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09977fe640 1 -- 172.21.15.83:0/107391790 learned_addr learned my addr 172.21.15.83:0/107391790 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:46.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f0996ffd640 1 --2- 172.21.15.83:0/107391790 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09980ad150 0x7f09980c2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:46.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f0997fff640 1 --2- 172.21.15.83:0/107391790 >> v2:172.21.15.83:3300/0 conn(0x7f09980c2d80 0x7f0998150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:46.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1755165743 0 0) 0x7f099806d630 con 0x7f09980abda0 2024-09-19T07:20:46.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0980003530 con 0x7f09980abda0 2024-09-19T07:20:46.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1935837737 0 0) 0x7f0980003530 con 0x7f09980abda0 2024-09-19T07:20:46.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09980ad150 msgr2=0x7f09980c2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:46.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 --2- 172.21.15.83:0/107391790 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09980ad150 0x7f09980c2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 >> v2:172.21.15.83:3300/0 conn(0x7f09980c2d80 msgr2=0x7f0998150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 --2- 172.21.15.83:0/107391790 >> v2:172.21.15.83:3300/0 conn(0x7f09980c2d80 0x7f0998150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099806e450 con 0x7f09980abda0 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f0988001680 con 0x7f09980abda0 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f0997fff640 1 --2- 172.21.15.83:0/107391790 >> v2:172.21.15.83:3300/0 conn(0x7f09980c2d80 0x7f0998150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 >> v1:172.21.15.83:6789/0 conn(0x7f09980abda0 legacy=0x7f09980ac180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 --2- 172.21.15.83:0/107391790 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09980ad150 0x7f09800039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 --2- 172.21.15.83:0/107391790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0980003f90 0x7f0980004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 --2- 172.21.15.83:0/107391790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0980004d50 0x7f09800051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0998154ae0 con 0x7f0980004d50 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f09980acc00 con 0x7f0980003f90 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f09800059a0 con 0x7f09980ad150 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f0997fff640 1 --2- 172.21.15.83:0/107391790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0980004d50 0x7f09800051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:46.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f0996ffd640 1 --2- 172.21.15.83:0/107391790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0980003f90 0x7f0980004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f09977fe640 1 --2- 172.21.15.83:0/107391790 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09980ad150 0x7f09800039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f0997fff640 1 -- 172.21.15.83:0/107391790 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09980ad150 msgr2=0x7f09800039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f0997fff640 1 --2- 172.21.15.83:0/107391790 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09980ad150 0x7f09800039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f0997fff640 1 -- 172.21.15.83:0/107391790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0980003f90 msgr2=0x7f0980004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f0997fff640 1 --2- 172.21.15.83:0/107391790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0980003f90 0x7f0980004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:46.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.317+0000 7f0997fff640 1 -- 172.21.15.83:0/107391790 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0980003530 con 0x7f0980004d50 2024-09-19T07:20:46.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.321+0000 7f0997fff640 1 --2- 172.21.15.83:0/107391790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0980004d50 0x7f09800051d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f099000e5b0 tx=0x7f099001d0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11e27f9412f4f8e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:46.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.321+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09900b1070 con 0x7f0980004d50 2024-09-19T07:20:46.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.321+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09900ac600 con 0x7f0980004d50 2024-09-19T07:20:46.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.321+0000 7f09967fc640 1 -- 172.21.15.83:0/107391790 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990009de0 con 0x7f0980004d50 2024-09-19T07:20:46.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.321+0000 7f099df1d640 1 -- 172.21.15.83:0/107391790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0980004d50 msgr2=0x7f09800051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:46.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.321+0000 7f099df1d640 1 --2- 172.21.15.83:0/107391790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0980004d50 0x7f09800051d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f099000e5b0 tx=0x7f099001d0e0 comp rx=0 tx=0).stop 2024-09-19T07:20:46.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.321+0000 7f0997fff640 1 -- 172.21.15.83:0/107391790 reap_dead start 2024-09-19T07:20:46.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.321+0000 7f099df1d640 1 -- 172.21.15.83:0/107391790 shutdown_connections 2024-09-19T07:20:46.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.321+0000 7f099df1d640 1 -- 172.21.15.83:0/107391790 >> 172.21.15.83:0/107391790 conn(0x7f09980c7e40 msgr2=0x7f09980c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:46.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.321+0000 7f099df1d640 1 -- 172.21.15.83:0/107391790 shutdown_connections 2024-09-19T07:20:46.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.321+0000 7f099df1d640 1 -- 172.21.15.83:0/107391790 wait complete. 2024-09-19T07:20:46.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f099df1d640 1 Processor -- start 2024-09-19T07:20:46.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f099df1d640 1 -- start start 2024-09-19T07:20:46.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f099df1d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0998149610 0x7f09980716a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:46.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f099df1d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09980c33f0 0x7f099814a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:46.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f099df1d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09981e7d70 0x7f09981465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:46.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f099df1d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0998154800 con 0x7f09980c33f0 2024-09-19T07:20:46.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f099df1d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0998154520 con 0x7f09981e7d70 2024-09-19T07:20:46.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f0996ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09980c33f0 0x7f099814a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:46.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f0996ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09980c33f0 0x7f099814a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37146/0 (socket says 172.21.15.83:37146) 2024-09-19T07:20:46.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f0996ffd640 1 -- 172.21.15.83:0/2711990514 learned_addr learned my addr 172.21.15.83:0/2711990514 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:46.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f099df1d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f09980c3210 con 0x7f0998149610 2024-09-19T07:20:46.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f09977fe640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0998149610 0x7f09980716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:46.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f0997fff640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09981e7d70 0x7f09981465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:46.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f0996ffd640 1 -- 172.21.15.83:0/2711990514 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0998149610 msgr2=0x7f09980716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:46.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f0996ffd640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0998149610 0x7f09980716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:46.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f0996ffd640 1 -- 172.21.15.83:0/2711990514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09981e7d70 msgr2=0x7f09981465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:46.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f0996ffd640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09981e7d70 0x7f09981465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:46.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f0996ffd640 1 -- 172.21.15.83:0/2711990514 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09981e7590 con 0x7f09980c33f0 2024-09-19T07:20:46.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f0996ffd640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09980c33f0 0x7f099814a710 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f098c011990 tx=0x7f098c011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:46.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f097ffff640 1 -- 172.21.15.83:0/2711990514 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c00fa70 con 0x7f09980c33f0 2024-09-19T07:20:46.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f097ffff640 1 -- 172.21.15.83:0/2711990514 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f098c00fc10 con 0x7f09980c33f0 2024-09-19T07:20:46.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f099df1d640 1 -- 172.21.15.83:0/2711990514 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f099814c2b0 con 0x7f09980c33f0 2024-09-19T07:20:46.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f097ffff640 1 -- 172.21.15.83:0/2711990514 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c014440 con 0x7f09980c33f0 2024-09-19T07:20:46.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.325+0000 7f099df1d640 1 -- 172.21.15.83:0/2711990514 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09981e7850 con 0x7f09980c33f0 2024-09-19T07:20:46.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.329+0000 7f099df1d640 1 -- 172.21.15.83:0/2711990514 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f099806db10 con 0x7f09980c33f0 2024-09-19T07:20:46.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.329+0000 7f097ffff640 1 -- 172.21.15.83:0/2711990514 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f098c012040 con 0x7f09980c33f0 2024-09-19T07:20:46.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.333+0000 7f097ffff640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0984067940 0x7f0984069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:46.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.333+0000 7f097ffff640 1 -- 172.21.15.83:0/2711990514 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7f098c086410 con 0x7f09980c33f0 2024-09-19T07:20:46.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.337+0000 7f09977fe640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0984067940 0x7f0984069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:46.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.337+0000 7f09977fe640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0984067940 0x7f0984069e00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0988004850 tx=0x7f0988009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:46.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.337+0000 7f097ffff640 1 -- 172.21.15.83:0/2711990514 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f098c019030 con 0x7f09980c33f0 2024-09-19T07:20:46.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.721+0000 7f099df1d640 1 -- 172.21.15.83:0/2711990514 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09980c65f0 con 0x7f09980c33f0 2024-09-19T07:20:46.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.721+0000 7f097ffff640 1 -- 172.21.15.83:0/2711990514 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f098c056d20 con 0x7f09980c33f0 2024-09-19T07:20:46.724 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:46.725 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:46.725 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:46.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.725+0000 7f099df1d640 1 -- 172.21.15.83:0/2711990514 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0984067940 msgr2=0x7f0984069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:46.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.725+0000 7f099df1d640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0984067940 0x7f0984069e00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0988004850 tx=0x7f0988009040 comp rx=0 tx=0).stop 2024-09-19T07:20:46.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.725+0000 7f099df1d640 1 -- 172.21.15.83:0/2711990514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09980c33f0 msgr2=0x7f099814a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:46.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.725+0000 7f099df1d640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09980c33f0 0x7f099814a710 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f098c011990 tx=0x7f098c011e50 comp rx=0 tx=0).stop 2024-09-19T07:20:46.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.725+0000 7f099df1d640 1 -- 172.21.15.83:0/2711990514 shutdown_connections 2024-09-19T07:20:46.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.725+0000 7f099df1d640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0984067940 0x7f0984069e00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:46.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.725+0000 7f099df1d640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09981e7d70 0x7f09981465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:46.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.725+0000 7f099df1d640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09980c33f0 0x7f099814a710 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:46.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.725+0000 7f099df1d640 1 --2- 172.21.15.83:0/2711990514 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0998149610 0x7f09980716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:46.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.725+0000 7f099df1d640 1 -- 172.21.15.83:0/2711990514 >> 172.21.15.83:0/2711990514 conn(0x7f09980c7e40 msgr2=0x7f099814fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:46.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.725+0000 7f099df1d640 1 -- 172.21.15.83:0/2711990514 shutdown_connections 2024-09-19T07:20:46.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:46.725+0000 7f099df1d640 1 -- 172.21.15.83:0/2711990514 wait complete. 2024-09-19T07:20:46.748 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:20:48.749 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:48.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.901+0000 7f3f5d394640 1 Processor -- start 2024-09-19T07:20:48.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.901+0000 7f3f5d394640 1 -- start start 2024-09-19T07:20:48.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f5d394640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3f580c3ab0 0x7f3f58157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:48.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f5d394640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f58073d90 con 0x7f3f5814dbe0 2024-09-19T07:20:48.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f5d394640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f58153b30 con 0x7f3f580aa4a0 2024-09-19T07:20:48.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f5d394640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f3f58154700 con 0x7f3f580c3ab0 2024-09-19T07:20:48.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f56ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3f5814dbe0 0x7f3f580c3fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43216/0 (socket says 172.21.15.83:43216) 2024-09-19T07:20:48.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f56ffd640 1 -- 172.21.15.83:0/639295418 learned_addr learned my addr 172.21.15.83:0/639295418 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:48.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f4ffff640 1 --2- 172.21.15.83:0/639295418 >> v2:172.21.15.184:3300/0 conn(0x7f3f580c3ab0 0x7f3f58157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:48.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1504279568 0 0) 0x7f3f58073d90 con 0x7f3f5814dbe0 2024-09-19T07:20:48.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f44003530 con 0x7f3f5814dbe0 2024-09-19T07:20:48.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3044833567 0 0) 0x7f3f58153b30 con 0x7f3f580aa4a0 2024-09-19T07:20:48.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f58073d90 con 0x7f3f580aa4a0 2024-09-19T07:20:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3661596014 0 0) 0x7f3f44003530 con 0x7f3f5814dbe0 2024-09-19T07:20:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 >> v2:172.21.15.184:3300/0 conn(0x7f3f580c3ab0 msgr2=0x7f3f58157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 --2- 172.21.15.83:0/639295418 >> v2:172.21.15.184:3300/0 conn(0x7f3f580c3ab0 0x7f3f58157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 >> v1:172.21.15.184:6789/0 conn(0x7f3f580aa4a0 legacy=0x7f3f58157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f58153dd0 con 0x7f3f5814dbe0 2024-09-19T07:20:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f3f48001680 con 0x7f3f5814dbe0 2024-09-19T07:20:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 >> v1:172.21.15.83:6789/0 conn(0x7f3f5814dbe0 legacy=0x7f3f580c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 --2- 172.21.15.83:0/639295418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f44004bb0 0x7f3f44004fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 --2- 172.21.15.83:0/639295418 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f440037b0 0x7f3f44003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 --2- 172.21.15.83:0/639295418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f44004170 0x7f3f4400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58154700 con 0x7f3f44004bb0 2024-09-19T07:20:48.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4400c170 con 0x7f3f44004170 2024-09-19T07:20:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3f4400c370 con 0x7f3f440037b0 2024-09-19T07:20:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f4ffff640 1 --2- 172.21.15.83:0/639295418 >> v2:172.21.15.184:3300/0 conn(0x7f3f580c3ab0 0x7f3f58157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f56ffd640 1 --2- 172.21.15.83:0/639295418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f44004bb0 0x7f3f44004fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f4ffff640 1 --2- 172.21.15.83:0/639295418 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f440037b0 0x7f3f44003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f577fe640 1 --2- 172.21.15.83:0/639295418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f44004170 0x7f3f4400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f56ffd640 1 -- 172.21.15.83:0/639295418 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f440037b0 msgr2=0x7f3f44003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f56ffd640 1 --2- 172.21.15.83:0/639295418 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f440037b0 0x7f3f44003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f56ffd640 1 -- 172.21.15.83:0/639295418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f44004170 msgr2=0x7f3f4400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f56ffd640 1 --2- 172.21.15.83:0/639295418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f44004170 0x7f3f4400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f56ffd640 1 -- 172.21.15.83:0/639295418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f44003530 con 0x7f3f44004bb0 2024-09-19T07:20:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.905+0000 7f3f577fe640 1 -- 172.21.15.83:0/639295418 reap_dead start 2024-09-19T07:20:48.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.909+0000 7f3f56ffd640 1 --2- 172.21.15.83:0/639295418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f44004bb0 0x7f3f44004fd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3f48007c90 tx=0x7f3f48021050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed66221262f9167f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:48.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.909+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48007cd0 con 0x7f3f44004bb0 2024-09-19T07:20:48.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.909+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f480ad020 con 0x7f3f44004bb0 2024-09-19T07:20:48.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.909+0000 7f3f567fc640 1 -- 172.21.15.83:0/639295418 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f480a9c20 con 0x7f3f44004bb0 2024-09-19T07:20:48.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.909+0000 7f3f5d394640 1 -- 172.21.15.83:0/639295418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f44004bb0 msgr2=0x7f3f44004fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:48.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.909+0000 7f3f5d394640 1 --2- 172.21.15.83:0/639295418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f44004bb0 0x7f3f44004fd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3f48007c90 tx=0x7f3f48021050 comp rx=0 tx=0).stop 2024-09-19T07:20:48.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.909+0000 7f3f5d394640 1 -- 172.21.15.83:0/639295418 shutdown_connections 2024-09-19T07:20:48.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.909+0000 7f3f5d394640 1 --2- 172.21.15.83:0/639295418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f44004bb0 0x7f3f44004fd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:48.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.909+0000 7f3f5d394640 1 -- 172.21.15.83:0/639295418 >> 172.21.15.83:0/639295418 conn(0x7f3f580c95a0 msgr2=0x7f3f580c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:48.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.909+0000 7f3f5d394640 1 -- 172.21.15.83:0/639295418 shutdown_connections 2024-09-19T07:20:48.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.909+0000 7f3f5d394640 1 -- 172.21.15.83:0/639295418 wait complete. 2024-09-19T07:20:48.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f5d394640 1 Processor -- start 2024-09-19T07:20:48.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f5d394640 1 -- start start 2024-09-19T07:20:48.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f5d394640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f44004bb0 0x7f3f581eaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:48.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f5d394640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f58144960 0x7f3f580c8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:48.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f5d394640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f581e74e0 0x7f3f581e7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:48.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f5d394640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58157190 con 0x7f3f58144960 2024-09-19T07:20:48.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f5d394640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58157010 con 0x7f3f581e74e0 2024-09-19T07:20:48.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f5d394640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3f58157310 con 0x7f3f44004bb0 2024-09-19T07:20:48.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f4ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f58144960 0x7f3f580c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:48.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f56ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f44004bb0 0x7f3f581eaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:48.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f56ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f44004bb0 0x7f3f581eaba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:37654/0 (socket says 172.21.15.83:37654) 2024-09-19T07:20:48.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f56ffd640 1 -- 172.21.15.83:0/881425191 learned_addr learned my addr 172.21.15.83:0/881425191 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:48.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f577fe640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f581e74e0 0x7f3f581e7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:48.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f56ffd640 1 -- 172.21.15.83:0/881425191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f581e74e0 msgr2=0x7f3f581e7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:48.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f56ffd640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f581e74e0 0x7f3f581e7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:48.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f56ffd640 1 -- 172.21.15.83:0/881425191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f58144960 msgr2=0x7f3f580c8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:48.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f56ffd640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f58144960 0x7f3f580c8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:48.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f56ffd640 1 -- 172.21.15.83:0/881425191 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f580aef10 con 0x7f3f44004bb0 2024-09-19T07:20:48.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f577fe640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f581e74e0 0x7f3f581e7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:20:48.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f4ffff640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f58144960 0x7f3f580c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:48.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f56ffd640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f44004bb0 0x7f3f581eaba0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800c250 tx=0x7f3f44005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:48.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f4f7fe640 1 -- 172.21.15.83:0/881425191 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f480b8070 con 0x7f3f44004bb0 2024-09-19T07:20:48.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f4f7fe640 1 -- 172.21.15.83:0/881425191 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f48001030 con 0x7f3f44004bb0 2024-09-19T07:20:48.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f4f7fe640 1 -- 172.21.15.83:0/881425191 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48007cd0 con 0x7f3f44004bb0 2024-09-19T07:20:48.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f5d394640 1 -- 172.21.15.83:0/881425191 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f580ade50 con 0x7f3f44004bb0 2024-09-19T07:20:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.913+0000 7f3f5d394640 1 -- 172.21.15.83:0/881425191 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f580c8980 con 0x7f3f44004bb0 2024-09-19T07:20:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.917+0000 7f3f4d7fa640 1 -- 172.21.15.83:0/881425191 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f300052a0 con 0x7f3f44004bb0 2024-09-19T07:20:48.927 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:48.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.917+0000 7f3f4f7fe640 1 -- 172.21.15.83:0/881425191 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3f48007310 con 0x7f3f44004bb0 2024-09-19T07:20:48.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.917+0000 7f3f4f7fe640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3f380678f0 0x7f3f38069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:48.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.917+0000 7f3f4ffff640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3f380678f0 0x7f3f38069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:48.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.917+0000 7f3f4f7fe640 1 -- 172.21.15.83:0/881425191 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7f3f48123a00 con 0x7f3f44004bb0 2024-09-19T07:20:48.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.917+0000 7f3f4ffff640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3f380678f0 0x7f3f38069db0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3f5814c4b0 tx=0x7f3f40009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:48.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:48.925+0000 7f3f4f7fe640 1 -- 172.21.15.83:0/881425191 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3f4801ad60 con 0x7f3f44004bb0 2024-09-19T07:20:49.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.085+0000 7fe63e06a640 1 Processor -- start 2024-09-19T07:20:49.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.085+0000 7fe63e06a640 1 -- start start 2024-09-19T07:20:49.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.085+0000 7fe63e06a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6380c7510 0x7fe63806de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:49.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.085+0000 7fe63e06a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe638154360 0x7fe6380c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:49.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.085+0000 7fe63e06a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe63814ddd0 con 0x7fe63814d9f0 2024-09-19T07:20:49.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.085+0000 7fe63e06a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe6380c8830 con 0x7fe638154360 2024-09-19T07:20:49.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.085+0000 7fe63e06a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe63806fdf0 con 0x7fe6380c7510 2024-09-19T07:20:49.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.089+0000 7fe6377fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe63814d9f0 0x7fe6380c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43220/0 (socket says 172.21.15.83:43220) 2024-09-19T07:20:49.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.089+0000 7fe6377fe640 1 -- 172.21.15.83:0/2593452465 learned_addr learned my addr 172.21.15.83:0/2593452465 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:49.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.089+0000 7fe637fff640 1 --2- 172.21.15.83:0/2593452465 >> v2:172.21.15.184:3300/0 conn(0x7fe638154360 0x7fe6380c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:49.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.089+0000 7fe62ffff640 1 --2- 172.21.15.83:0/2593452465 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6380c7510 0x7fe63806de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:49.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.089+0000 7fe636ffd640 1 -- 172.21.15.83:0/2593452465 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3840797771 0 0) 0x7fe63814ddd0 con 0x7fe63814d9f0 2024-09-19T07:20:49.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.089+0000 7fe636ffd640 1 -- 172.21.15.83:0/2593452465 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe624003530 con 0x7fe63814d9f0 2024-09-19T07:20:49.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.089+0000 7fe62ffff640 1 -- 172.21.15.83:0/2593452465 >> v2:172.21.15.184:3300/0 conn(0x7fe638154360 msgr2=0x7fe6380c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:49.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.089+0000 7fe62ffff640 1 --2- 172.21.15.83:0/2593452465 >> v2:172.21.15.184:3300/0 conn(0x7fe638154360 0x7fe6380c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.089+0000 7fe62ffff640 1 -- 172.21.15.83:0/2593452465 >> v1:172.21.15.83:6789/0 conn(0x7fe63814d9f0 legacy=0x7fe6380c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.089+0000 7fe62ffff640 1 -- 172.21.15.83:0/2593452465 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6380c7f00 con 0x7fe6380c7510 2024-09-19T07:20:49.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.089+0000 7fe637fff640 1 --2- 172.21.15.83:0/2593452465 >> v2:172.21.15.184:3300/0 conn(0x7fe638154360 0x7fe6380c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:49.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.089+0000 7fe62ffff640 1 --2- 172.21.15.83:0/2593452465 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6380c7510 0x7fe63806de90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe6200097c0 tx=0x7fe62001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2a5a2f44646aa8e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:49.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe636ffd640 1 -- 172.21.15.83:0/2593452465 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6200b0070 con 0x7fe6380c7510 2024-09-19T07:20:49.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe636ffd640 1 -- 172.21.15.83:0/2593452465 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6200ab600 con 0x7fe6380c7510 2024-09-19T07:20:49.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe636ffd640 1 -- 172.21.15.83:0/2593452465 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6200ab9d0 con 0x7fe6380c7510 2024-09-19T07:20:49.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe63e06a640 1 -- 172.21.15.83:0/2593452465 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6380c7510 msgr2=0x7fe63806de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe63e06a640 1 --2- 172.21.15.83:0/2593452465 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6380c7510 0x7fe63806de90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe6200097c0 tx=0x7fe62001c410 comp rx=0 tx=0).stop 2024-09-19T07:20:49.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe63e06a640 1 -- 172.21.15.83:0/2593452465 shutdown_connections 2024-09-19T07:20:49.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe63e06a640 1 --2- 172.21.15.83:0/2593452465 >> v2:172.21.15.184:3300/0 conn(0x7fe638154360 0x7fe6380c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe63e06a640 1 --2- 172.21.15.83:0/2593452465 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6380c7510 0x7fe63806de90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe63e06a640 1 -- 172.21.15.83:0/2593452465 >> 172.21.15.83:0/2593452465 conn(0x7fe6380c6110 msgr2=0x7fe6380c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:49.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe63e06a640 1 -- 172.21.15.83:0/2593452465 shutdown_connections 2024-09-19T07:20:49.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe63e06a640 1 -- 172.21.15.83:0/2593452465 wait complete. 2024-09-19T07:20:49.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe63e06a640 1 Processor -- start 2024-09-19T07:20:49.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.093+0000 7fe63e06a640 1 -- start start 2024-09-19T07:20:49.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe63e06a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6380c7510 0x7fe63806d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:49.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe63e06a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe63814d9f0 0x7fe638149960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:49.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe63e06a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe638154360 0x7fe63806c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:49.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe63e06a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe638160c00 con 0x7fe63814d9f0 2024-09-19T07:20:49.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe63e06a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe638160a80 con 0x7fe6380c7510 2024-09-19T07:20:49.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe63e06a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe638160d80 con 0x7fe638154360 2024-09-19T07:20:49.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe6377fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6380c7510 0x7fe63806d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe62ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe63814d9f0 0x7fe638149960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe6377fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6380c7510 0x7fe63806d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:38174/0 (socket says 172.21.15.83:38174) 2024-09-19T07:20:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe62ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe63814d9f0 0x7fe638149960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37164/0 (socket says 172.21.15.83:37164) 2024-09-19T07:20:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe6377fe640 1 -- 172.21.15.83:0/1890206954 learned_addr learned my addr 172.21.15.83:0/1890206954 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe637fff640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe638154360 0x7fe63806c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe6377fe640 1 -- 172.21.15.83:0/1890206954 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe638154360 msgr2=0x7fe63806c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe6377fe640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe638154360 0x7fe63806c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe6377fe640 1 -- 172.21.15.83:0/1890206954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe63814d9f0 msgr2=0x7fe638149960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe6377fe640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe63814d9f0 0x7fe638149960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe6377fe640 1 -- 172.21.15.83:0/1890206954 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe638148e50 con 0x7fe6380c7510 2024-09-19T07:20:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe637fff640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe638154360 0x7fe63806c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe6377fe640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6380c7510 0x7fe63806d550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe62800c420 tx=0x7fe62800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe634ff9640 1 -- 172.21.15.83:0/1890206954 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe628017070 con 0x7fe6380c7510 2024-09-19T07:20:49.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe63e06a640 1 -- 172.21.15.83:0/1890206954 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe638157c90 con 0x7fe6380c7510 2024-09-19T07:20:49.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe63e06a640 1 -- 172.21.15.83:0/1890206954 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe63806da90 con 0x7fe6380c7510 2024-09-19T07:20:49.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe634ff9640 1 -- 172.21.15.83:0/1890206954 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe628005cf0 con 0x7fe6380c7510 2024-09-19T07:20:49.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe634ff9640 1 -- 172.21.15.83:0/1890206954 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6280072d0 con 0x7fe6380c7510 2024-09-19T07:20:49.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.097+0000 7fe63e06a640 1 -- 172.21.15.83:0/1890206954 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe61c0052a0 con 0x7fe6380c7510 2024-09-19T07:20:49.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.101+0000 7fe634ff9640 1 -- 172.21.15.83:0/1890206954 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe62801a070 con 0x7fe6380c7510 2024-09-19T07:20:49.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.101+0000 7fe634ff9640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6180678c0 0x7fe618069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:49.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.101+0000 7fe62ffff640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6180678c0 0x7fe618069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:49.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.101+0000 7fe62ffff640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6180678c0 0x7fe618069d80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe6200ad000 tx=0x7fe6200a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:49.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.101+0000 7fe634ff9640 1 -- 172.21.15.83:0/1890206954 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7fe628085210 con 0x7fe6380c7510 2024-09-19T07:20:49.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.109+0000 7fe634ff9640 1 -- 172.21.15.83:0/1890206954 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe6280553d0 con 0x7fe6380c7510 2024-09-19T07:20:49.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.309+0000 7f3f4d7fa640 1 -- 172.21.15.83:0/881425191 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f30005590 con 0x7f3f44004bb0 2024-09-19T07:20:49.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4f7fe640 1 -- 172.21.15.83:0/881425191 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f3f480f3c60 con 0x7f3f44004bb0 2024-09-19T07:20:49.315 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:49.315 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:49.315 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:49.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4d7fa640 1 -- 172.21.15.83:0/881425191 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3f380678f0 msgr2=0x7f3f38069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4d7fa640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3f380678f0 0x7f3f38069db0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3f5814c4b0 tx=0x7f3f40009190 comp rx=0 tx=0).stop 2024-09-19T07:20:49.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4d7fa640 1 -- 172.21.15.83:0/881425191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f44004bb0 msgr2=0x7f3f581eaba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4d7fa640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f44004bb0 0x7f3f581eaba0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800c250 tx=0x7f3f44005840 comp rx=0 tx=0).stop 2024-09-19T07:20:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4d7fa640 1 -- 172.21.15.83:0/881425191 shutdown_connections 2024-09-19T07:20:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4d7fa640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3f380678f0 0x7f3f38069db0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4d7fa640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f581e74e0 0x7f3f581e7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4d7fa640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f58144960 0x7f3f580c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4d7fa640 1 --2- 172.21.15.83:0/881425191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f44004bb0 0x7f3f581eaba0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4d7fa640 1 -- 172.21.15.83:0/881425191 >> 172.21.15.83:0/881425191 conn(0x7f3f580c95a0 msgr2=0x7f3f580aab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4d7fa640 1 -- 172.21.15.83:0/881425191 shutdown_connections 2024-09-19T07:20:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.313+0000 7f3f4d7fa640 1 -- 172.21.15.83:0/881425191 wait complete. 2024-09-19T07:20:49.340 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:20:49.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.488+0000 7fe63e06a640 1 -- 172.21.15.83:0/1890206954 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe61c005590 con 0x7fe6380c7510 2024-09-19T07:20:49.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe634ff9640 1 -- 172.21.15.83:0/1890206954 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7fe6280596d0 con 0x7fe6380c7510 2024-09-19T07:20:49.495 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:49.495 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe63e06a640 1 -- 172.21.15.83:0/1890206954 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6180678c0 msgr2=0x7fe618069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe63e06a640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6180678c0 0x7fe618069d80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe6200ad000 tx=0x7fe6200a6000 comp rx=0 tx=0).stop 2024-09-19T07:20:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe63e06a640 1 -- 172.21.15.83:0/1890206954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6380c7510 msgr2=0x7fe63806d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe63e06a640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6380c7510 0x7fe63806d550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe62800c420 tx=0x7fe62800c820 comp rx=0 tx=0).stop 2024-09-19T07:20:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe63e06a640 1 -- 172.21.15.83:0/1890206954 shutdown_connections 2024-09-19T07:20:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe63e06a640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6180678c0 0x7fe618069d80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe63e06a640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe638154360 0x7fe63806c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe63e06a640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe63814d9f0 0x7fe638149960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe63e06a640 1 --2- 172.21.15.83:0/1890206954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6380c7510 0x7fe63806d550 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe63e06a640 1 -- 172.21.15.83:0/1890206954 >> 172.21.15.83:0/1890206954 conn(0x7fe6380c6110 msgr2=0x7fe6380c6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe63e06a640 1 -- 172.21.15.83:0/1890206954 shutdown_connections 2024-09-19T07:20:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.492+0000 7fe63e06a640 1 -- 172.21.15.83:0/1890206954 wait complete. 2024-09-19T07:20:49.517 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-19T07:20:49.518 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:20:49.518 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:20:49.530 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:20:49.530 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:49.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.724+0000 7f391432a640 1 Processor -- start 2024-09-19T07:20:49.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.724+0000 7f391432a640 1 -- start start 2024-09-19T07:20:49.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.724+0000 7f391432a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f390c150310 0x7f390c0728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:49.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.724+0000 7f391432a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390c0c2f90 con 0x7f390c06cef0 2024-09-19T07:20:49.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.724+0000 7f391432a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390c0c3210 con 0x7f390c0c4990 2024-09-19T07:20:49.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.724+0000 7f391432a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f390c0c3e20 con 0x7f390c150310 2024-09-19T07:20:49.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.728+0000 7f39118c5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f390c150310 0x7f390c0728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:49.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.728+0000 7f39118c5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f390c150310 0x7f390c0728d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37178/0 (socket says 172.21.15.83:37178) 2024-09-19T07:20:49.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.728+0000 7f39118c5640 1 -- 172.21.15.83:0/2871012487 learned_addr learned my addr 172.21.15.83:0/2871012487 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:49.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.728+0000 7f39118c5640 1 -- 172.21.15.83:0/2871012487 >> v1:172.21.15.184:6789/0 conn(0x7f390c0c4990 legacy=0x7f390c156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.728+0000 7f39110c4640 1 -- 172.21.15.83:0/2871012487 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2505590899 0 0) 0x7f3908001120 con 0x7f390c06cef0 2024-09-19T07:20:49.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.728+0000 7f39118c5640 1 -- 172.21.15.83:0/2871012487 >> v1:172.21.15.83:6789/0 conn(0x7f390c06cef0 legacy=0x7f390c14fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.728+0000 7f39118c5640 1 -- 172.21.15.83:0/2871012487 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390c0c34f0 con 0x7f390c150310 2024-09-19T07:20:49.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.728+0000 7f39118c5640 1 --2- 172.21.15.83:0/2871012487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f390c150310 0x7f390c0728d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f390c0c3fa0 tx=0x7f38fc01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e41181c0e29786 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:49.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.728+0000 7f39110c4640 1 -- 172.21.15.83:0/2871012487 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0b0070 con 0x7f390c150310 2024-09-19T07:20:49.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.728+0000 7f39110c4640 1 -- 172.21.15.83:0/2871012487 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38fc0ab600 con 0x7f390c150310 2024-09-19T07:20:49.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.728+0000 7f39110c4640 1 -- 172.21.15.83:0/2871012487 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0ab9d0 con 0x7f390c150310 2024-09-19T07:20:49.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 -- 172.21.15.83:0/2871012487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f390c150310 msgr2=0x7f390c0728d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 --2- 172.21.15.83:0/2871012487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f390c150310 0x7f390c0728d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f390c0c3fa0 tx=0x7f38fc01c4b0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 -- 172.21.15.83:0/2871012487 shutdown_connections 2024-09-19T07:20:49.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 --2- 172.21.15.83:0/2871012487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f390c150310 0x7f390c0728d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 -- 172.21.15.83:0/2871012487 >> 172.21.15.83:0/2871012487 conn(0x7f390c0c5090 msgr2=0x7f390c0c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:49.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 -- 172.21.15.83:0/2871012487 shutdown_connections 2024-09-19T07:20:49.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 -- 172.21.15.83:0/2871012487 wait complete. 2024-09-19T07:20:49.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 Processor -- start 2024-09-19T07:20:49.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 -- start start 2024-09-19T07:20:49.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f390c06cef0 0x7f390c0c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:49.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f390c0c4990 0x7f390c147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:49.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f390c150310 0x7f390c145490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:49.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f390c06d450 con 0x7f390c06cef0 2024-09-19T07:20:49.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f390c06d2d0 con 0x7f390c0c4990 2024-09-19T07:20:49.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f391432a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f390c0757a0 con 0x7f390c150310 2024-09-19T07:20:49.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f39120c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f390c06cef0 0x7f390c0c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:49.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f39120c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f390c06cef0 0x7f390c0c7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37180/0 (socket says 172.21.15.83:37180) 2024-09-19T07:20:49.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f39120c6640 1 -- 172.21.15.83:0/2621553706 learned_addr learned my addr 172.21.15.83:0/2621553706 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:49.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f39118c5640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f390c0c4990 0x7f390c147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:49.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.732+0000 7f39128c7640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f390c150310 0x7f390c145490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:49.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f39120c6640 1 -- 172.21.15.83:0/2621553706 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f390c150310 msgr2=0x7f390c145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f39120c6640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f390c150310 0x7f390c145490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f39120c6640 1 -- 172.21.15.83:0/2621553706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f390c0c4990 msgr2=0x7f390c147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:49.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f39120c6640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f390c0c4990 0x7f390c147510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:49.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f39120c6640 1 -- 172.21.15.83:0/2621553706 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390c147a50 con 0x7f390c06cef0 2024-09-19T07:20:49.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f39120c6640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f390c06cef0 0x7f390c0c7f40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f390800c7b0 tx=0x7f390800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:49.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f3902ffd640 1 -- 172.21.15.83:0/2621553706 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3908005070 con 0x7f390c06cef0 2024-09-19T07:20:49.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f3902ffd640 1 -- 172.21.15.83:0/2621553706 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3908005230 con 0x7f390c06cef0 2024-09-19T07:20:49.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f3902ffd640 1 -- 172.21.15.83:0/2621553706 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3908003600 con 0x7f390c06cef0 2024-09-19T07:20:49.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f391432a640 1 -- 172.21.15.83:0/2621553706 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f390c1f05e0 con 0x7f390c06cef0 2024-09-19T07:20:49.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f391432a640 1 -- 172.21.15.83:0/2621553706 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f390c0c6d30 con 0x7f390c06cef0 2024-09-19T07:20:49.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f391432a640 1 -- 172.21.15.83:0/2621553706 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f390c075b30 con 0x7f390c06cef0 2024-09-19T07:20:49.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.736+0000 7f3902ffd640 1 -- 172.21.15.83:0/2621553706 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f39080037a0 con 0x7f390c06cef0 2024-09-19T07:20:49.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.740+0000 7f3902ffd640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f38f00678f0 0x7f38f0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:49.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.740+0000 7f3902ffd640 1 -- 172.21.15.83:0/2621553706 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7f3908086520 con 0x7f390c06cef0 2024-09-19T07:20:49.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.740+0000 7f39118c5640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f38f00678f0 0x7f38f0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:49.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.740+0000 7f39118c5640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f38f00678f0 0x7f38f0069db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0ad000 tx=0x7f38fc0a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:49.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:49.748+0000 7f3902ffd640 1 -- 172.21.15.83:0/2621553706 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f390c0c2f90 con 0x7f390c06cef0 2024-09-19T07:20:50.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.128+0000 7f391432a640 1 -- 172.21.15.83:0/2621553706 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f390c0c5620 con 0x7f390c06cef0 2024-09-19T07:20:50.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.128+0000 7f3902ffd640 1 -- 172.21.15.83:0/2621553706 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f3908056750 con 0x7f390c06cef0 2024-09-19T07:20:50.133 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:50.133 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:50.133 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:50.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.132+0000 7f391432a640 1 -- 172.21.15.83:0/2621553706 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f38f00678f0 msgr2=0x7f38f0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:50.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.132+0000 7f391432a640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f38f00678f0 0x7f38f0069db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0ad000 tx=0x7f38fc0a6000 comp rx=0 tx=0).stop 2024-09-19T07:20:50.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.132+0000 7f391432a640 1 -- 172.21.15.83:0/2621553706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f390c06cef0 msgr2=0x7f390c0c7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:50.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.132+0000 7f391432a640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f390c06cef0 0x7f390c0c7f40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f390800c7b0 tx=0x7f390800cc70 comp rx=0 tx=0).stop 2024-09-19T07:20:50.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.132+0000 7f391432a640 1 -- 172.21.15.83:0/2621553706 shutdown_connections 2024-09-19T07:20:50.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.132+0000 7f391432a640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f38f00678f0 0x7f38f0069db0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:50.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.132+0000 7f391432a640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f390c150310 0x7f390c145490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:50.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.132+0000 7f391432a640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f390c0c4990 0x7f390c147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:50.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.132+0000 7f391432a640 1 --2- 172.21.15.83:0/2621553706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f390c06cef0 0x7f390c0c7f40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:50.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.132+0000 7f391432a640 1 -- 172.21.15.83:0/2621553706 >> 172.21.15.83:0/2621553706 conn(0x7f390c0c5090 msgr2=0x7f390c14d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:50.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.132+0000 7f391432a640 1 -- 172.21.15.83:0/2621553706 shutdown_connections 2024-09-19T07:20:50.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:50.132+0000 7f391432a640 1 -- 172.21.15.83:0/2621553706 wait complete. 2024-09-19T07:20:50.156 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:20:51.342 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:51.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb115298640 1 Processor -- start 2024-09-19T07:20:51.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb115298640 1 -- start start 2024-09-19T07:20:51.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb115298640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb110152b20 0x7fb1100c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:51.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb115298640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb110150670 con 0x7fb1100c45c0 2024-09-19T07:20:51.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb115298640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1100c2880 con 0x7fb110150290 2024-09-19T07:20:51.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb10f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb1100c45c0 0x7fb110153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42066/0 (socket says 172.21.15.83:42066) 2024-09-19T07:20:51.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb10f7fe640 1 -- 172.21.15.83:0/863194089 learned_addr learned my addr 172.21.15.83:0/863194089 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:51.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb107fff640 1 --2- 172.21.15.83:0/863194089 >> v2:172.21.15.83:3300/0 conn(0x7fb110152b20 0x7fb1100c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:51.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb115298640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb1100716c0 con 0x7fb110152b20 2024-09-19T07:20:51.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb10e7fc640 1 -- 172.21.15.83:0/863194089 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3513836363 0 0) 0x7fb110150670 con 0x7fb1100c45c0 2024-09-19T07:20:51.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb10e7fc640 1 -- 172.21.15.83:0/863194089 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0fc003530 con 0x7fb1100c45c0 2024-09-19T07:20:51.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb10e7fc640 1 -- 172.21.15.83:0/863194089 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3279146966 0 0) 0x7fb1100c2880 con 0x7fb110150290 2024-09-19T07:20:51.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb107fff640 1 -- 172.21.15.83:0/863194089 >> v1:172.21.15.184:6789/0 conn(0x7fb110150290 legacy=0x7fb1100c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:51.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb107fff640 1 -- 172.21.15.83:0/863194089 >> v1:172.21.15.83:6789/0 conn(0x7fb1100c45c0 legacy=0x7fb110153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:51.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.496+0000 7fb107fff640 1 -- 172.21.15.83:0/863194089 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1100c2b60 con 0x7fb110152b20 2024-09-19T07:20:51.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.500+0000 7fb107fff640 1 --2- 172.21.15.83:0/863194089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb110152b20 0x7fb1100c4080 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80097c0 tx=0x7fb0f801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6b9e3549117ac5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:51.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.500+0000 7fb10e7fc640 1 -- 172.21.15.83:0/863194089 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f80b0070 con 0x7fb110152b20 2024-09-19T07:20:51.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.500+0000 7fb10e7fc640 1 -- 172.21.15.83:0/863194089 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f80ab600 con 0x7fb110152b20 2024-09-19T07:20:51.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.500+0000 7fb10e7fc640 1 -- 172.21.15.83:0/863194089 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f80ab9d0 con 0x7fb110152b20 2024-09-19T07:20:51.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.500+0000 7fb115298640 1 -- 172.21.15.83:0/863194089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb110152b20 msgr2=0x7fb1100c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:51.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.500+0000 7fb115298640 1 --2- 172.21.15.83:0/863194089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb110152b20 0x7fb1100c4080 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80097c0 tx=0x7fb0f801c4a0 comp rx=0 tx=0).stop 2024-09-19T07:20:51.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.500+0000 7fb115298640 1 -- 172.21.15.83:0/863194089 shutdown_connections 2024-09-19T07:20:51.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.500+0000 7fb115298640 1 --2- 172.21.15.83:0/863194089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb110152b20 0x7fb1100c4080 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:51.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.500+0000 7fb115298640 1 -- 172.21.15.83:0/863194089 >> 172.21.15.83:0/863194089 conn(0x7fb11006bc80 msgr2=0x7fb11006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:51.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.500+0000 7fb115298640 1 -- 172.21.15.83:0/863194089 shutdown_connections 2024-09-19T07:20:51.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.500+0000 7fb115298640 1 -- 172.21.15.83:0/863194089 wait complete. 2024-09-19T07:20:51.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb115298640 1 Processor -- start 2024-09-19T07:20:51.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb115298640 1 -- start start 2024-09-19T07:20:51.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb115298640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100c45c0 0x7fb11014f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:51.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb115298640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb110150290 0x7fb110150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:51.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb115298640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb110152b20 0x7fb1101467b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:51.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb10effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100c45c0 0x7fb11014f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:51.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb10effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100c45c0 0x7fb11014f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43140/0 (socket says 172.21.15.83:43140) 2024-09-19T07:20:51.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb107fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb110150290 0x7fb110150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:51.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb10f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb110152b20 0x7fb1101467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:51.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb107fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb110150290 0x7fb110150940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:56438/0 (socket says 172.21.15.83:56438) 2024-09-19T07:20:51.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb10f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb110152b20 0x7fb1101467b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:57550/0 (socket says 172.21.15.83:57550) 2024-09-19T07:20:51.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb10effd640 1 -- 172.21.15.83:0/1037681166 learned_addr learned my addr 172.21.15.83:0/1037681166 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:51.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb110160cf0 con 0x7fb1100c45c0 2024-09-19T07:20:51.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb110160b70 con 0x7fb110152b20 2024-09-19T07:20:51.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb110160e70 con 0x7fb110150290 2024-09-19T07:20:51.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb10effd640 1 -- 172.21.15.83:0/1037681166 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb110150290 msgr2=0x7fb110150940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:51.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb10effd640 1 --2- 172.21.15.83:0/1037681166 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb110150290 0x7fb110150940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:51.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb10effd640 1 -- 172.21.15.83:0/1037681166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb110152b20 msgr2=0x7fb1101467b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:51.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb10effd640 1 --2- 172.21.15.83:0/1037681166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb110152b20 0x7fb1101467b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:51.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb10effd640 1 -- 172.21.15.83:0/1037681166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1100c20a0 con 0x7fb1100c45c0 2024-09-19T07:20:51.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb10effd640 1 --2- 172.21.15.83:0/1037681166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100c45c0 0x7fb11014f4c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb10000c480 tx=0x7fb10000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:51.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb1077fe640 1 -- 172.21.15.83:0/1037681166 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb100005480 con 0x7fb1100c45c0 2024-09-19T07:20:51.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb1077fe640 1 -- 172.21.15.83:0/1037681166 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb100003600 con 0x7fb1100c45c0 2024-09-19T07:20:51.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1100c66b0 con 0x7fb1100c45c0 2024-09-19T07:20:51.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb1077fe640 1 -- 172.21.15.83:0/1037681166 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1000039f0 con 0x7fb1100c45c0 2024-09-19T07:20:51.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.504+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb110152570 con 0x7fb1100c45c0 2024-09-19T07:20:51.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.508+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb110155fc0 con 0x7fb1100c45c0 2024-09-19T07:20:51.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.508+0000 7fb1077fe640 1 -- 172.21.15.83:0/1037681166 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb100003b90 con 0x7fb1100c45c0 2024-09-19T07:20:51.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.508+0000 7fb1077fe640 1 --2- 172.21.15.83:0/1037681166 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb0f0067990 0x7fb0f0069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:51.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.508+0000 7fb1077fe640 1 -- 172.21.15.83:0/1037681166 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7fb100085be0 con 0x7fb1100c45c0 2024-09-19T07:20:51.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.508+0000 7fb107fff640 1 --2- 172.21.15.83:0/1037681166 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb0f0067990 0x7fb0f0069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:51.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.508+0000 7fb107fff640 1 --2- 172.21.15.83:0/1037681166 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb0f0067990 0x7fb0f0069e50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80ad000 tx=0x7fb0f80a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:51.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.516+0000 7fb1077fe640 1 -- 172.21.15.83:0/1037681166 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb100055da0 con 0x7fb1100c45c0 2024-09-19T07:20:51.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb110068290 con 0x7fb1100c45c0 2024-09-19T07:20:51.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb1077fe640 1 -- 172.21.15.83:0/1037681166 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7fb10005a0a0 con 0x7fb1100c45c0 2024-09-19T07:20:51.903 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:51.903 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:51.903 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:51.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb0f0067990 msgr2=0x7fb0f0069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:51.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 --2- 172.21.15.83:0/1037681166 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb0f0067990 0x7fb0f0069e50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80ad000 tx=0x7fb0f80a6000 comp rx=0 tx=0).stop 2024-09-19T07:20:51.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100c45c0 msgr2=0x7fb11014f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:51.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 --2- 172.21.15.83:0/1037681166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100c45c0 0x7fb11014f4c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb10000c480 tx=0x7fb10000c940 comp rx=0 tx=0).stop 2024-09-19T07:20:51.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 shutdown_connections 2024-09-19T07:20:51.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 --2- 172.21.15.83:0/1037681166 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb0f0067990 0x7fb0f0069e50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:51.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 --2- 172.21.15.83:0/1037681166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb110152b20 0x7fb1101467b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:51.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 --2- 172.21.15.83:0/1037681166 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb110150290 0x7fb110150940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:51.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 --2- 172.21.15.83:0/1037681166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100c45c0 0x7fb11014f4c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:51.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 >> 172.21.15.83:0/1037681166 conn(0x7fb11006bc80 msgr2=0x7fb11014cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:51.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 shutdown_connections 2024-09-19T07:20:51.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:51.900+0000 7fb115298640 1 -- 172.21.15.83:0/1037681166 wait complete. 2024-09-19T07:20:51.926 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:20:52.158 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:52.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f4312224640 1 Processor -- start 2024-09-19T07:20:52.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f4312224640 1 -- start start 2024-09-19T07:20:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f4312224640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f430c0c3ab0 0x7f430c157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f4312224640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f430c0aa4a0 0x7f430c157550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f4312224640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f430c073d90 con 0x7f430c14dbe0 2024-09-19T07:20:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f4312224640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f430c154700 con 0x7f430c0aa4a0 2024-09-19T07:20:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f4312224640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f430c153b30 con 0x7f430c0c3ab0 2024-09-19T07:20:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f430bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f430c0aa4a0 0x7f430c157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f430bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f430c0aa4a0 0x7f430c157550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43150/0 (socket says 172.21.15.83:43150) 2024-09-19T07:20:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f430bfff640 1 -- 172.21.15.83:0/662541948 learned_addr learned my addr 172.21.15.83:0/662541948 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f4303fff640 1 --2- 172.21.15.83:0/662541948 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f430c0c3ab0 0x7f430c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f430bfff640 1 -- 172.21.15.83:0/662541948 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f430c0c3ab0 msgr2=0x7f430c157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f430bfff640 1 --2- 172.21.15.83:0/662541948 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f430c0c3ab0 0x7f430c157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f430bfff640 1 -- 172.21.15.83:0/662541948 >> v1:172.21.15.184:6789/0 conn(0x7f430c14dbe0 legacy=0x7f430c0c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f430bfff640 1 -- 172.21.15.83:0/662541948 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f430c153dd0 con 0x7f430c0aa4a0 2024-09-19T07:20:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.324+0000 7f430affd640 1 -- 172.21.15.83:0/662541948 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 471974125 0 0) 0x7f430c073d90 con 0x7f430c14dbe0 2024-09-19T07:20:52.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.328+0000 7f430bfff640 1 --2- 172.21.15.83:0/662541948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c0aa4a0 0x7f430c157550 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4304009e50 tx=0x7f430401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46193435290184d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:52.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.328+0000 7f430affd640 1 -- 172.21.15.83:0/662541948 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43040a5770 con 0x7f430c0aa4a0 2024-09-19T07:20:52.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.328+0000 7f430affd640 1 -- 172.21.15.83:0/662541948 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43040a5910 con 0x7f430c0aa4a0 2024-09-19T07:20:52.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.328+0000 7f430affd640 1 -- 172.21.15.83:0/662541948 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43040a5ce0 con 0x7f430c0aa4a0 2024-09-19T07:20:52.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.328+0000 7f4312224640 1 -- 172.21.15.83:0/662541948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c0aa4a0 msgr2=0x7f430c157550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:52.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.328+0000 7f4312224640 1 --2- 172.21.15.83:0/662541948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c0aa4a0 0x7f430c157550 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4304009e50 tx=0x7f430401c9a0 comp rx=0 tx=0).stop 2024-09-19T07:20:52.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.328+0000 7f4312224640 1 -- 172.21.15.83:0/662541948 shutdown_connections 2024-09-19T07:20:52.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.328+0000 7f4312224640 1 --2- 172.21.15.83:0/662541948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c0aa4a0 0x7f430c157550 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:52.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.328+0000 7f4312224640 1 --2- 172.21.15.83:0/662541948 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f430c0c3ab0 0x7f430c157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:52.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.328+0000 7f4312224640 1 -- 172.21.15.83:0/662541948 >> 172.21.15.83:0/662541948 conn(0x7f430c0c95a0 msgr2=0x7f430c0c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:52.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.328+0000 7f4312224640 1 -- 172.21.15.83:0/662541948 shutdown_connections 2024-09-19T07:20:52.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.328+0000 7f4312224640 1 -- 172.21.15.83:0/662541948 wait complete. 2024-09-19T07:20:52.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4312224640 1 Processor -- start 2024-09-19T07:20:52.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4312224640 1 -- start start 2024-09-19T07:20:52.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4312224640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f430c0aa4a0 0x7f430c1ece30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:52.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4312224640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c14dbe0 0x7f430c1ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:52.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4312224640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f430c0c8de0 0x7f430c152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:52.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4312224640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f430c0c4830 con 0x7f430c14dbe0 2024-09-19T07:20:52.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4312224640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f430c0c46b0 con 0x7f430c0aa4a0 2024-09-19T07:20:52.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4312224640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f430c160bc0 con 0x7f430c0c8de0 2024-09-19T07:20:52.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4303fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c14dbe0 0x7f430c1ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:52.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f430b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f430c0aa4a0 0x7f430c1ece30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:52.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4303fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c14dbe0 0x7f430c1ef9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43162/0 (socket says 172.21.15.83:43162) 2024-09-19T07:20:52.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f430bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f430c0c8de0 0x7f430c152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:52.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f430b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f430c0aa4a0 0x7f430c1ece30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:57562/0 (socket says 172.21.15.83:57562) 2024-09-19T07:20:52.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4303fff640 1 -- 172.21.15.83:0/2962803707 learned_addr learned my addr 172.21.15.83:0/2962803707 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:52.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f430b7fe640 1 -- 172.21.15.83:0/2962803707 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f430c0c8de0 msgr2=0x7f430c152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:52.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f430b7fe640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f430c0c8de0 0x7f430c152b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:52.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f430b7fe640 1 -- 172.21.15.83:0/2962803707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c14dbe0 msgr2=0x7f430c1ef9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:52.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f430b7fe640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c14dbe0 0x7f430c1ef9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:52.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f430b7fe640 1 -- 172.21.15.83:0/2962803707 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f430c1ed370 con 0x7f430c0aa4a0 2024-09-19T07:20:52.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f430bfff640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f430c0c8de0 0x7f430c152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:52.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4303fff640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c14dbe0 0x7f430c1ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:52.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f430b7fe640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f430c0aa4a0 0x7f430c1ece30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f42fc0011f0 tx=0x7f42fc00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:52.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4308ff9640 1 -- 172.21.15.83:0/2962803707 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc017070 con 0x7f430c0aa4a0 2024-09-19T07:20:52.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4312224640 1 -- 172.21.15.83:0/2962803707 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f430c1efef0 con 0x7f430c0aa4a0 2024-09-19T07:20:52.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4308ff9640 1 -- 172.21.15.83:0/2962803707 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42fc003600 con 0x7f430c0aa4a0 2024-09-19T07:20:52.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4312224640 1 -- 172.21.15.83:0/2962803707 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f430c0c8780 con 0x7f430c0aa4a0 2024-09-19T07:20:52.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.332+0000 7f4308ff9640 1 -- 172.21.15.83:0/2962803707 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc003a30 con 0x7f430c0aa4a0 2024-09-19T07:20:52.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.336+0000 7f4312224640 1 -- 172.21.15.83:0/2962803707 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f430c0c89c0 con 0x7f430c0aa4a0 2024-09-19T07:20:52.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.336+0000 7f4308ff9640 1 -- 172.21.15.83:0/2962803707 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f42fc01a070 con 0x7f430c0aa4a0 2024-09-19T07:20:52.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.336+0000 7f4308ff9640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f42ec0678c0 0x7f42ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:52.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.336+0000 7f4308ff9640 1 -- 172.21.15.83:0/2962803707 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7f42fc0852b0 con 0x7f430c0aa4a0 2024-09-19T07:20:52.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.336+0000 7f4303fff640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f42ec0678c0 0x7f42ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:52.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.336+0000 7f4303fff640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f42ec0678c0 0x7f42ec069d80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f430c153a60 tx=0x7f42f4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:52.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.344+0000 7f4308ff9640 1 -- 172.21.15.83:0/2962803707 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f430c0c89c0 con 0x7f430c0aa4a0 2024-09-19T07:20:52.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.724+0000 7f4312224640 1 -- 172.21.15.83:0/2962803707 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f430c0c3fa0 con 0x7f430c0aa4a0 2024-09-19T07:20:52.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.724+0000 7f4308ff9640 1 -- 172.21.15.83:0/2962803707 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f42fc0553f0 con 0x7f430c0aa4a0 2024-09-19T07:20:52.730 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:52.730 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:52.730 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:52.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.728+0000 7f4312224640 1 -- 172.21.15.83:0/2962803707 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f42ec0678c0 msgr2=0x7f42ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:52.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.728+0000 7f4312224640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f42ec0678c0 0x7f42ec069d80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f430c153a60 tx=0x7f42f4009190 comp rx=0 tx=0).stop 2024-09-19T07:20:52.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.728+0000 7f4312224640 1 -- 172.21.15.83:0/2962803707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f430c0aa4a0 msgr2=0x7f430c1ece30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:52.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.728+0000 7f4312224640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f430c0aa4a0 0x7f430c1ece30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f42fc0011f0 tx=0x7f42fc00c8c0 comp rx=0 tx=0).stop 2024-09-19T07:20:52.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.728+0000 7f4312224640 1 -- 172.21.15.83:0/2962803707 shutdown_connections 2024-09-19T07:20:52.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.728+0000 7f4312224640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f42ec0678c0 0x7f42ec069d80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:52.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.728+0000 7f4312224640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f430c0c8de0 0x7f430c152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:52.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.728+0000 7f4312224640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f430c14dbe0 0x7f430c1ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:52.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.728+0000 7f4312224640 1 --2- 172.21.15.83:0/2962803707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f430c0aa4a0 0x7f430c1ece30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:52.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.728+0000 7f4312224640 1 -- 172.21.15.83:0/2962803707 >> 172.21.15.83:0/2962803707 conn(0x7f430c0c95a0 msgr2=0x7f430c14c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:52.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.728+0000 7f4312224640 1 -- 172.21.15.83:0/2962803707 shutdown_connections 2024-09-19T07:20:52.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:52.728+0000 7f4312224640 1 -- 172.21.15.83:0/2962803707 wait complete. 2024-09-19T07:20:52.755 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:20:53.927 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:54.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.080+0000 7f6f59a49640 1 Processor -- start 2024-09-19T07:20:54.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.080+0000 7f6f59a49640 1 -- start start 2024-09-19T07:20:54.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.080+0000 7f6f59a49640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6f540c3ab0 0x7f6f54157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:54.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.080+0000 7f6f59a49640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f54073d90 con 0x7f6f540aa4a0 2024-09-19T07:20:54.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.080+0000 7f6f59a49640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f54153b30 con 0x7f6f5414dbe0 2024-09-19T07:20:54.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.080+0000 7f6f59a49640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6f54154700 con 0x7f6f540c3ab0 2024-09-19T07:20:54.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.080+0000 7f6f537fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6f540aa4a0 0x7f6f54157550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42082/0 (socket says 172.21.15.83:42082) 2024-09-19T07:20:54.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.080+0000 7f6f537fe640 1 -- 172.21.15.83:0/4172629878 learned_addr learned my addr 172.21.15.83:0/4172629878 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:54.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.080+0000 7f6f4bfff640 1 --2- 172.21.15.83:0/4172629878 >> v2:172.21.15.184:3300/0 conn(0x7f6f540c3ab0 0x7f6f54157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:54.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.080+0000 7f6f527fc640 1 -- 172.21.15.83:0/4172629878 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3160585188 0 0) 0x7f6f54073d90 con 0x7f6f540aa4a0 2024-09-19T07:20:54.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.080+0000 7f6f527fc640 1 -- 172.21.15.83:0/4172629878 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f40003530 con 0x7f6f540aa4a0 2024-09-19T07:20:54.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.080+0000 7f6f527fc640 1 -- 172.21.15.83:0/4172629878 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1538461032 0 0) 0x7f6f54153b30 con 0x7f6f5414dbe0 2024-09-19T07:20:54.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.084+0000 7f6f527fc640 1 -- 172.21.15.83:0/4172629878 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f54073d90 con 0x7f6f5414dbe0 2024-09-19T07:20:54.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.084+0000 7f6f4bfff640 1 -- 172.21.15.83:0/4172629878 >> v1:172.21.15.184:6789/0 conn(0x7f6f5414dbe0 legacy=0x7f6f540c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:54.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.084+0000 7f6f4bfff640 1 -- 172.21.15.83:0/4172629878 >> v1:172.21.15.83:6789/0 conn(0x7f6f540aa4a0 legacy=0x7f6f54157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:54.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.084+0000 7f6f4bfff640 1 -- 172.21.15.83:0/4172629878 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f54153dd0 con 0x7f6f540c3ab0 2024-09-19T07:20:54.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.084+0000 7f6f4bfff640 1 --2- 172.21.15.83:0/4172629878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f540c3ab0 0x7f6f54157010 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6f54154880 tx=0x7f6f3c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e382f8da49373885 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:54.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.084+0000 7f6f527fc640 1 -- 172.21.15.83:0/4172629878 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c0b0070 con 0x7f6f540c3ab0 2024-09-19T07:20:54.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.084+0000 7f6f527fc640 1 -- 172.21.15.83:0/4172629878 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f3c0ab600 con 0x7f6f540c3ab0 2024-09-19T07:20:54.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.084+0000 7f6f527fc640 1 -- 172.21.15.83:0/4172629878 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c0ab9d0 con 0x7f6f540c3ab0 2024-09-19T07:20:54.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 -- 172.21.15.83:0/4172629878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f540c3ab0 msgr2=0x7f6f54157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:54.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 --2- 172.21.15.83:0/4172629878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f540c3ab0 0x7f6f54157010 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6f54154880 tx=0x7f6f3c01c410 comp rx=0 tx=0).stop 2024-09-19T07:20:54.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 -- 172.21.15.83:0/4172629878 shutdown_connections 2024-09-19T07:20:54.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 --2- 172.21.15.83:0/4172629878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f540c3ab0 0x7f6f54157010 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:54.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 -- 172.21.15.83:0/4172629878 >> 172.21.15.83:0/4172629878 conn(0x7f6f540c95a0 msgr2=0x7f6f540c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:54.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 -- 172.21.15.83:0/4172629878 shutdown_connections 2024-09-19T07:20:54.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 -- 172.21.15.83:0/4172629878 wait complete. 2024-09-19T07:20:54.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 Processor -- start 2024-09-19T07:20:54.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 -- start start 2024-09-19T07:20:54.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f540aa4a0 0x7f6f54147040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:54.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f5414dbe0 0x7f6f540c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:54.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f541553d0 0x7f6f54155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:54.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f540c46b0 con 0x7f6f541553d0 2024-09-19T07:20:54.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6f540741a0 con 0x7f6f540aa4a0 2024-09-19T07:20:54.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f59a49640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6f540c4830 con 0x7f6f5414dbe0 2024-09-19T07:20:54.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f537fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f541553d0 0x7f6f54155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:54.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f537fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f541553d0 0x7f6f54155860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43174/0 (socket says 172.21.15.83:43174) 2024-09-19T07:20:54.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f537fe640 1 -- 172.21.15.83:0/70292205 learned_addr learned my addr 172.21.15.83:0/70292205 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:54.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f52ffd640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f540aa4a0 0x7f6f54147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:54.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.088+0000 7f6f4bfff640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f5414dbe0 0x7f6f540c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:54.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f52ffd640 1 -- 172.21.15.83:0/70292205 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f5414dbe0 msgr2=0x7f6f540c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:54.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f52ffd640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f5414dbe0 0x7f6f540c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:54.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f52ffd640 1 -- 172.21.15.83:0/70292205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f541553d0 msgr2=0x7f6f54155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:54.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f52ffd640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f541553d0 0x7f6f54155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:54.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f52ffd640 1 -- 172.21.15.83:0/70292205 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f54147580 con 0x7f6f540aa4a0 2024-09-19T07:20:54.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f537fe640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f541553d0 0x7f6f54155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:54.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f4bfff640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f5414dbe0 0x7f6f540c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:20:54.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f52ffd640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f540aa4a0 0x7f6f54147040 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6f4400c3c0 tx=0x7f6f4400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:54.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f50ff9640 1 -- 172.21.15.83:0/70292205 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f44017070 con 0x7f6f540aa4a0 2024-09-19T07:20:54.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f50ff9640 1 -- 172.21.15.83:0/70292205 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f440052d0 con 0x7f6f540aa4a0 2024-09-19T07:20:54.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f59a49640 1 -- 172.21.15.83:0/70292205 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f5414fbe0 con 0x7f6f540aa4a0 2024-09-19T07:20:54.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f50ff9640 1 -- 172.21.15.83:0/70292205 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f440054c0 con 0x7f6f540aa4a0 2024-09-19T07:20:54.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f59a49640 1 -- 172.21.15.83:0/70292205 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f541501a0 con 0x7f6f540aa4a0 2024-09-19T07:20:54.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f59a49640 1 -- 172.21.15.83:0/70292205 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f5406f3e0 con 0x7f6f540aa4a0 2024-09-19T07:20:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f50ff9640 1 -- 172.21.15.83:0/70292205 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6f4401a070 con 0x7f6f540aa4a0 2024-09-19T07:20:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f50ff9640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f340678f0 0x7f6f34069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.092+0000 7f6f50ff9640 1 -- 172.21.15.83:0/70292205 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7f6f44085390 con 0x7f6f540aa4a0 2024-09-19T07:20:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.096+0000 7f6f4bfff640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f340678f0 0x7f6f34069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.096+0000 7f6f4bfff640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f340678f0 0x7f6f34069db0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0ad000 tx=0x7f6f3c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:54.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.104+0000 7f6f50ff9640 1 -- 172.21.15.83:0/70292205 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6f54147580 con 0x7f6f540aa4a0 2024-09-19T07:20:54.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.484+0000 7f6f59a49640 1 -- 172.21.15.83:0/70292205 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f540c3fa0 con 0x7f6f540aa4a0 2024-09-19T07:20:54.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.484+0000 7f6f50ff9640 1 -- 172.21.15.83:0/70292205 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f6f440553b0 con 0x7f6f540aa4a0 2024-09-19T07:20:54.490 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:54.490 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:54.490 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:54.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.488+0000 7f6f59a49640 1 -- 172.21.15.83:0/70292205 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f340678f0 msgr2=0x7f6f34069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:54.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.488+0000 7f6f59a49640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f340678f0 0x7f6f34069db0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0ad000 tx=0x7f6f3c008040 comp rx=0 tx=0).stop 2024-09-19T07:20:54.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.488+0000 7f6f59a49640 1 -- 172.21.15.83:0/70292205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f540aa4a0 msgr2=0x7f6f54147040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:54.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.488+0000 7f6f59a49640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f540aa4a0 0x7f6f54147040 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6f4400c3c0 tx=0x7f6f4400c880 comp rx=0 tx=0).stop 2024-09-19T07:20:54.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.488+0000 7f6f59a49640 1 -- 172.21.15.83:0/70292205 shutdown_connections 2024-09-19T07:20:54.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.488+0000 7f6f59a49640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f340678f0 0x7f6f34069db0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:54.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.488+0000 7f6f59a49640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f541553d0 0x7f6f54155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:54.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.488+0000 7f6f59a49640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f5414dbe0 0x7f6f540c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:54.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.488+0000 7f6f59a49640 1 --2- 172.21.15.83:0/70292205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f540aa4a0 0x7f6f54147040 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:54.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.488+0000 7f6f59a49640 1 -- 172.21.15.83:0/70292205 >> 172.21.15.83:0/70292205 conn(0x7f6f540c95a0 msgr2=0x7f6f540ab8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:54.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.488+0000 7f6f59a49640 1 -- 172.21.15.83:0/70292205 shutdown_connections 2024-09-19T07:20:54.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.488+0000 7f6f59a49640 1 -- 172.21.15.83:0/70292205 wait complete. 2024-09-19T07:20:54.513 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:20:54.755 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:54.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.908+0000 7fcc1b4a6640 1 Processor -- start 2024-09-19T07:20:54.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.908+0000 7fcc1b4a6640 1 -- start start 2024-09-19T07:20:54.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.908+0000 7fcc1b4a6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc141592f0 0x7fcc140c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:54.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.908+0000 7fcc1b4a6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc14156200 con 0x7fcc140c6120 2024-09-19T07:20:54.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.908+0000 7fcc1b4a6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc141564e0 con 0x7fcc14155d70 2024-09-19T07:20:54.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.908+0000 7fcc1b4a6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc140c9930 con 0x7fcc141592f0 2024-09-19T07:20:54.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.912+0000 7fcc19242640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcc140c6120 0x7fcc141575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42088/0 (socket says 172.21.15.83:42088) 2024-09-19T07:20:54.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.912+0000 7fcc19242640 1 -- 172.21.15.83:0/72641279 learned_addr learned my addr 172.21.15.83:0/72641279 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:54.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.912+0000 7fcc19a43640 1 --2- 172.21.15.83:0/72641279 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc141592f0 0x7fcc140c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:54.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.912+0000 7fcc0bfff640 1 -- 172.21.15.83:0/72641279 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1917869674 0 0) 0x7fcc14156200 con 0x7fcc140c6120 2024-09-19T07:20:54.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.912+0000 7fcc0bfff640 1 -- 172.21.15.83:0/72641279 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc00003530 con 0x7fcc140c6120 2024-09-19T07:20:54.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.912+0000 7fcc0bfff640 1 -- 172.21.15.83:0/72641279 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3484066047 0 0) 0x7fcc141564e0 con 0x7fcc14155d70 2024-09-19T07:20:54.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.912+0000 7fcc0bfff640 1 -- 172.21.15.83:0/72641279 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc14156200 con 0x7fcc14155d70 2024-09-19T07:20:54.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.912+0000 7fcc19a43640 1 -- 172.21.15.83:0/72641279 >> v1:172.21.15.184:6789/0 conn(0x7fcc14155d70 legacy=0x7fcc14157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:54.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.912+0000 7fcc19a43640 1 -- 172.21.15.83:0/72641279 >> v1:172.21.15.83:6789/0 conn(0x7fcc140c6120 legacy=0x7fcc141575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:54.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.912+0000 7fcc19a43640 1 -- 172.21.15.83:0/72641279 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc141567c0 con 0x7fcc141592f0 2024-09-19T07:20:54.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.912+0000 7fcc19a43640 1 --2- 172.21.15.83:0/72641279 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc141592f0 0x7fcc140c6500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009dc0 tx=0x7fcc0c01c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ba8a769cc6e0dda server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:54.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.916+0000 7fcc0bfff640 1 -- 172.21.15.83:0/72641279 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c0b2070 con 0x7fcc141592f0 2024-09-19T07:20:54.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.916+0000 7fcc0bfff640 1 -- 172.21.15.83:0/72641279 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c0a57c0 con 0x7fcc141592f0 2024-09-19T07:20:54.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.916+0000 7fcc0bfff640 1 -- 172.21.15.83:0/72641279 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c0a5b90 con 0x7fcc141592f0 2024-09-19T07:20:54.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.916+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/72641279 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc141592f0 msgr2=0x7fcc140c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:54.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.916+0000 7fcc1b4a6640 1 --2- 172.21.15.83:0/72641279 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc141592f0 0x7fcc140c6500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009dc0 tx=0x7fcc0c01c900 comp rx=0 tx=0).stop 2024-09-19T07:20:54.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.916+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/72641279 shutdown_connections 2024-09-19T07:20:54.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.916+0000 7fcc1b4a6640 1 --2- 172.21.15.83:0/72641279 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc141592f0 0x7fcc140c6500 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:54.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.916+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/72641279 >> 172.21.15.83:0/72641279 conn(0x7fcc1406bc80 msgr2=0x7fcc1406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:54.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.916+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/72641279 shutdown_connections 2024-09-19T07:20:54.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.916+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/72641279 wait complete. 2024-09-19T07:20:54.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.916+0000 7fcc1b4a6640 1 Processor -- start 2024-09-19T07:20:54.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.916+0000 7fcc1b4a6640 1 -- start start 2024-09-19T07:20:54.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc1b4a6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc140c6120 0x7fcc140c18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:54.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc1b4a6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc141592f0 0x7fcc141f1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:54.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc1b4a6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc140ad5e0 0x7fcc141f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:54.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc1b4a6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14164030 con 0x7fcc140c6120 2024-09-19T07:20:54.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc1b4a6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14163eb0 con 0x7fcc140ad5e0 2024-09-19T07:20:54.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc1b4a6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc141641b0 con 0x7fcc141592f0 2024-09-19T07:20:54.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc19242640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc140c6120 0x7fcc140c18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:54.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc19242640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc140c6120 0x7fcc140c18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43176/0 (socket says 172.21.15.83:43176) 2024-09-19T07:20:54.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc19242640 1 -- 172.21.15.83:0/3103070481 learned_addr learned my addr 172.21.15.83:0/3103070481 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:54.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc19a43640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc140ad5e0 0x7fcc141f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:54.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc18a41640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc141592f0 0x7fcc141f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:54.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc19a43640 1 -- 172.21.15.83:0/3103070481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc141592f0 msgr2=0x7fcc141f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:54.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc19a43640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc141592f0 0x7fcc141f1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:54.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc19a43640 1 -- 172.21.15.83:0/3103070481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc140c6120 msgr2=0x7fcc140c18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:54.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc19a43640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc140c6120 0x7fcc140c18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:54.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc19a43640 1 -- 172.21.15.83:0/3103070481 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc140ab590 con 0x7fcc140ad5e0 2024-09-19T07:20:54.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc19a43640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc140ad5e0 0x7fcc141f1d50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0ac000 tx=0x7fcc0c0196a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:54.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc09ffb640 1 -- 172.21.15.83:0/3103070481 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c0b2050 con 0x7fcc140ad5e0 2024-09-19T07:20:54.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc09ffb640 1 -- 172.21.15.83:0/3103070481 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c0be070 con 0x7fcc140ad5e0 2024-09-19T07:20:54.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/3103070481 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1414a5a0 con 0x7fcc140ad5e0 2024-09-19T07:20:54.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc09ffb640 1 -- 172.21.15.83:0/3103070481 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c0ad890 con 0x7fcc140ad5e0 2024-09-19T07:20:54.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/3103070481 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc140cb370 con 0x7fcc140ad5e0 2024-09-19T07:20:54.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/3103070481 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc140c6500 con 0x7fcc140ad5e0 2024-09-19T07:20:54.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.920+0000 7fcc09ffb640 1 -- 172.21.15.83:0/3103070481 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcc0c0ada30 con 0x7fcc140ad5e0 2024-09-19T07:20:54.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.924+0000 7fcc09ffb640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcbf80678c0 0x7fcbf8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:54.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.924+0000 7fcc19242640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcbf80678c0 0x7fcbf8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:54.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.924+0000 7fcc09ffb640 1 -- 172.21.15.83:0/3103070481 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7fcc0c121100 con 0x7fcc140ad5e0 2024-09-19T07:20:54.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.924+0000 7fcc19242640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcbf80678c0 0x7fcbf8069d80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcc10007fd0 tx=0x7fcc10009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:54.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:54.928+0000 7fcc09ffb640 1 -- 172.21.15.83:0/3103070481 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcc0c016580 con 0x7fcc140ad5e0 2024-09-19T07:20:55.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.312+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/3103070481 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc140c96a0 con 0x7fcc140ad5e0 2024-09-19T07:20:55.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.312+0000 7fcc09ffb640 1 -- 172.21.15.83:0/3103070481 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7fcc0c0f13e0 con 0x7fcc140ad5e0 2024-09-19T07:20:55.317 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:55.318 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:55.318 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:55.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.316+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/3103070481 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcbf80678c0 msgr2=0x7fcbf8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:55.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.316+0000 7fcc1b4a6640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcbf80678c0 0x7fcbf8069d80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcc10007fd0 tx=0x7fcc10009040 comp rx=0 tx=0).stop 2024-09-19T07:20:55.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.316+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/3103070481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc140ad5e0 msgr2=0x7fcc141f1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:55.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.316+0000 7fcc1b4a6640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc140ad5e0 0x7fcc141f1d50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0ac000 tx=0x7fcc0c0196a0 comp rx=0 tx=0).stop 2024-09-19T07:20:55.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.316+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/3103070481 shutdown_connections 2024-09-19T07:20:55.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.316+0000 7fcc1b4a6640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcbf80678c0 0x7fcbf8069d80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:55.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.316+0000 7fcc1b4a6640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc140ad5e0 0x7fcc141f1d50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:55.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.316+0000 7fcc1b4a6640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc141592f0 0x7fcc141f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:55.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.316+0000 7fcc1b4a6640 1 --2- 172.21.15.83:0/3103070481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc140c6120 0x7fcc140c18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:55.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.316+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/3103070481 >> 172.21.15.83:0/3103070481 conn(0x7fcc1406bc80 msgr2=0x7fcc141597d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:55.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.316+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/3103070481 shutdown_connections 2024-09-19T07:20:55.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:55.316+0000 7fcc1b4a6640 1 -- 172.21.15.83:0/3103070481 wait complete. 2024-09-19T07:20:55.342 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:20:56.515 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:56.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f046d176640 1 Processor -- start 2024-09-19T07:20:56.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f046d176640 1 -- start start 2024-09-19T07:20:56.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f046d176640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f046814dbe0 0x7f04680c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:56.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f046d176640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04680c3ab0 0x7f0468157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:56.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f046d176640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0468073d90 con 0x7f04680c44e0 2024-09-19T07:20:56.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f046d176640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f0468154700 con 0x7f046814dbe0 2024-09-19T07:20:56.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f046d176640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0468153b30 con 0x7f04680c3ab0 2024-09-19T07:20:56.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f0466d9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f046814dbe0 0x7f04680c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:56.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f0466d9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f046814dbe0 0x7f04680c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:57596/0 (socket says 172.21.15.83:57596) 2024-09-19T07:20:56.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f0466d9d640 1 -- 172.21.15.83:0/657728687 learned_addr learned my addr 172.21.15.83:0/657728687 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:56.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f045e59c640 1 --2- 172.21.15.83:0/657728687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04680c3ab0 0x7f0468157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:56.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f046759e640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f04680c44e0 0x7f0468157550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:32982/0 (socket says 172.21.15.83:32982) 2024-09-19T07:20:56.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f0466d9d640 1 -- 172.21.15.83:0/657728687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04680c3ab0 msgr2=0x7f0468157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:56.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f0466d9d640 1 --2- 172.21.15.83:0/657728687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04680c3ab0 0x7f0468157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:56.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f0466d9d640 1 -- 172.21.15.83:0/657728687 >> v1:172.21.15.184:6789/0 conn(0x7f04680c44e0 legacy=0x7f0468157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:56.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f0466d9d640 1 -- 172.21.15.83:0/657728687 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0468153dd0 con 0x7f046814dbe0 2024-09-19T07:20:56.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.668+0000 7f045e59c640 1 --2- 172.21.15.83:0/657728687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04680c3ab0 0x7f0468157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:56.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.672+0000 7f0466d9d640 1 --2- 172.21.15.83:0/657728687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f046814dbe0 0x7f04680c3fa0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0458009870 tx=0x7f045801c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=819afc57c5f298d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:56.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.672+0000 7f046659c640 1 -- 172.21.15.83:0/657728687 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04580b1070 con 0x7f046814dbe0 2024-09-19T07:20:56.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.672+0000 7f046659c640 1 -- 172.21.15.83:0/657728687 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04580a8050 con 0x7f046814dbe0 2024-09-19T07:20:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.672+0000 7f046659c640 1 -- 172.21.15.83:0/657728687 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04580ac860 con 0x7f046814dbe0 2024-09-19T07:20:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.672+0000 7f046d176640 1 -- 172.21.15.83:0/657728687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f046814dbe0 msgr2=0x7f04680c3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.672+0000 7f046d176640 1 --2- 172.21.15.83:0/657728687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f046814dbe0 0x7f04680c3fa0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0458009870 tx=0x7f045801c4d0 comp rx=0 tx=0).stop 2024-09-19T07:20:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.672+0000 7f046d176640 1 -- 172.21.15.83:0/657728687 shutdown_connections 2024-09-19T07:20:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.672+0000 7f046d176640 1 --2- 172.21.15.83:0/657728687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04680c3ab0 0x7f0468157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.672+0000 7f046d176640 1 --2- 172.21.15.83:0/657728687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f046814dbe0 0x7f04680c3fa0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.672+0000 7f046d176640 1 -- 172.21.15.83:0/657728687 >> 172.21.15.83:0/657728687 conn(0x7f04680c95a0 msgr2=0x7f04680c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.672+0000 7f046d176640 1 -- 172.21.15.83:0/657728687 shutdown_connections 2024-09-19T07:20:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.672+0000 7f046d176640 1 -- 172.21.15.83:0/657728687 wait complete. 2024-09-19T07:20:56.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f046d176640 1 Processor -- start 2024-09-19T07:20:56.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f046d176640 1 -- start start 2024-09-19T07:20:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f046d176640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04680c44e0 0x7f04680aed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f046d176640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f046814dbe0 0x7f0468148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f046d176640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04681472a0 0x7f04681487b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f0466d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04680c44e0 0x7f04680aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f0466d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04680c44e0 0x7f04680aed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:57612/0 (socket says 172.21.15.83:57612) 2024-09-19T07:20:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f045e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f046814dbe0 0x7f0468148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f045e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f046814dbe0 0x7f0468148270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43178/0 (socket says 172.21.15.83:43178) 2024-09-19T07:20:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f045e59c640 1 -- 172.21.15.83:0/4035286389 learned_addr learned my addr 172.21.15.83:0/4035286389 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0468160bc0 con 0x7f046814dbe0 2024-09-19T07:20:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f04680741a0 con 0x7f04680c44e0 2024-09-19T07:20:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0468160d40 con 0x7f04681472a0 2024-09-19T07:20:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f0466d9d640 1 -- 172.21.15.83:0/4035286389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04681472a0 msgr2=0x7f04681487b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f046759e640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04681472a0 0x7f04681487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f0466d9d640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04681472a0 0x7f04681487b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f0466d9d640 1 -- 172.21.15.83:0/4035286389 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f046814dbe0 msgr2=0x7f0468148270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f0466d9d640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f046814dbe0 0x7f0468148270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f0466d9d640 1 -- 172.21.15.83:0/4035286389 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04680ac640 con 0x7f04680c44e0 2024-09-19T07:20:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f045e59c640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f046814dbe0 0x7f0468148270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f046759e640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04681472a0 0x7f04681487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f0466d9d640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04680c44e0 0x7f04680aed30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f04580ac000 tx=0x7f0458019270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f045ffff640 1 -- 172.21.15.83:0/4035286389 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04580bc070 con 0x7f04680c44e0 2024-09-19T07:20:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f045ffff640 1 -- 172.21.15.83:0/4035286389 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04580acdd0 con 0x7f04680c44e0 2024-09-19T07:20:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04680ac920 con 0x7f04680c44e0 2024-09-19T07:20:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.676+0000 7f045ffff640 1 -- 172.21.15.83:0/4035286389 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04580b1040 con 0x7f04680c44e0 2024-09-19T07:20:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.680+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04680af270 con 0x7f04680c44e0 2024-09-19T07:20:56.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.680+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04680c3fa0 con 0x7f04680c44e0 2024-09-19T07:20:56.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.680+0000 7f045ffff640 1 -- 172.21.15.83:0/4035286389 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f04580bf070 con 0x7f04680c44e0 2024-09-19T07:20:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.680+0000 7f045ffff640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f04480678c0 0x7f0448069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.680+0000 7f045ffff640 1 -- 172.21.15.83:0/4035286389 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7f045811f700 con 0x7f04680c44e0 2024-09-19T07:20:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.680+0000 7f045e59c640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f04480678c0 0x7f0448069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.680+0000 7f045e59c640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f04480678c0 0x7f0448069d80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0468149c60 tx=0x7f0450009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:56.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:56.688+0000 7f045ffff640 1 -- 172.21.15.83:0/4035286389 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f0458016150 con 0x7f04680c44e0 2024-09-19T07:20:57.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.072+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0468073f80 con 0x7f04680c44e0 2024-09-19T07:20:57.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.072+0000 7f045ffff640 1 -- 172.21.15.83:0/4035286389 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f04580ef960 con 0x7f04680c44e0 2024-09-19T07:20:57.075 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:57.076 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:57.076 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:57.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.072+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f04480678c0 msgr2=0x7f0448069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:57.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.072+0000 7f046d176640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f04480678c0 0x7f0448069d80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0468149c60 tx=0x7f0450009190 comp rx=0 tx=0).stop 2024-09-19T07:20:57.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.072+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04680c44e0 msgr2=0x7f04680aed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:57.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.072+0000 7f046d176640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04680c44e0 0x7f04680aed30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f04580ac000 tx=0x7f0458019270 comp rx=0 tx=0).stop 2024-09-19T07:20:57.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.076+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 shutdown_connections 2024-09-19T07:20:57.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.076+0000 7f046d176640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f04480678c0 0x7f0448069d80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.076+0000 7f046d176640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04681472a0 0x7f04681487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.076+0000 7f046d176640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f046814dbe0 0x7f0468148270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.076+0000 7f046d176640 1 --2- 172.21.15.83:0/4035286389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04680c44e0 0x7f04680aed30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.076+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 >> 172.21.15.83:0/4035286389 conn(0x7f04680c95a0 msgr2=0x7f046814c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:57.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.076+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 shutdown_connections 2024-09-19T07:20:57.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.076+0000 7f046d176640 1 -- 172.21.15.83:0/4035286389 wait complete. 2024-09-19T07:20:57.099 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:20:57.343 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:57.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddeb64640 1 Processor -- start 2024-09-19T07:20:57.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddeb64640 1 -- start start 2024-09-19T07:20:57.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddeb64640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2dd81513e0 0x7f2dd80c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddeb64640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2dd806f690 0x7f2dd806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddeb64640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dd80c3830 con 0x7f2dd806d1d0 2024-09-19T07:20:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddeb64640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f2dd80703c0 con 0x7f2dd81513e0 2024-09-19T07:20:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddeb64640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2dd80c3120 con 0x7f2dd806f690 2024-09-19T07:20:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddd101640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2dd806d1d0 0x7f2dd8157090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42102/0 (socket says 172.21.15.83:42102) 2024-09-19T07:20:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddd101640 1 -- 172.21.15.83:0/1667255663 learned_addr learned my addr 172.21.15.83:0/1667255663 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2dd7fff640 1 --2- 172.21.15.83:0/1667255663 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2dd806f690 0x7f2dd806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddc900640 1 --2- 172.21.15.83:0/1667255663 >> v2:172.21.15.83:3300/0 conn(0x7f2dd81513e0 0x7f2dd80c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:57.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddc900640 1 -- 172.21.15.83:0/1667255663 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2dd806f690 msgr2=0x7f2dd806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddc900640 1 --2- 172.21.15.83:0/1667255663 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2dd806f690 0x7f2dd806cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddc900640 1 -- 172.21.15.83:0/1667255663 >> v1:172.21.15.83:6789/0 conn(0x7f2dd806d1d0 legacy=0x7f2dd8157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.496+0000 7f2ddc900640 1 -- 172.21.15.83:0/1667255663 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd81577a0 con 0x7f2dd81513e0 2024-09-19T07:20:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.500+0000 7f2ddc900640 1 --2- 172.21.15.83:0/1667255663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2dd81513e0 0x7f2dd80c45f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0097c0 tx=0x7f2dcc01c2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b40250c7381e1c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.500+0000 7f2dd77fe640 1 -- 172.21.15.83:0/1667255663 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc0b0070 con 0x7f2dd81513e0 2024-09-19T07:20:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.500+0000 7f2dd77fe640 1 -- 172.21.15.83:0/1667255663 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dcc0ab600 con 0x7f2dd81513e0 2024-09-19T07:20:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.500+0000 7f2dd77fe640 1 -- 172.21.15.83:0/1667255663 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc0ab9d0 con 0x7f2dd81513e0 2024-09-19T07:20:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.500+0000 7f2ddeb64640 1 -- 172.21.15.83:0/1667255663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2dd81513e0 msgr2=0x7f2dd80c45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.500+0000 7f2ddeb64640 1 --2- 172.21.15.83:0/1667255663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2dd81513e0 0x7f2dd80c45f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0097c0 tx=0x7f2dcc01c2a0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.500+0000 7f2ddeb64640 1 -- 172.21.15.83:0/1667255663 shutdown_connections 2024-09-19T07:20:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.500+0000 7f2ddeb64640 1 --2- 172.21.15.83:0/1667255663 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2dd806f690 0x7f2dd806cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.500+0000 7f2ddeb64640 1 --2- 172.21.15.83:0/1667255663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2dd81513e0 0x7f2dd80c45f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.500+0000 7f2ddeb64640 1 -- 172.21.15.83:0/1667255663 >> 172.21.15.83:0/1667255663 conn(0x7f2dd80c5090 msgr2=0x7f2dd80c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:57.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.500+0000 7f2ddeb64640 1 -- 172.21.15.83:0/1667255663 shutdown_connections 2024-09-19T07:20:57.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.500+0000 7f2ddeb64640 1 -- 172.21.15.83:0/1667255663 wait complete. 2024-09-19T07:20:57.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddeb64640 1 Processor -- start 2024-09-19T07:20:57.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddeb64640 1 -- start start 2024-09-19T07:20:57.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddeb64640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2dd806d1d0 0x7f2dd8148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:57.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddeb64640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2dd806f690 0x7f2dd815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:57.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddeb64640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2dd81513e0 0x7f2dd80733f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:57.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddd101640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2dd81513e0 0x7f2dd80733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:57.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddc900640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2dd806d1d0 0x7f2dd8148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:57.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddd101640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2dd81513e0 0x7f2dd80733f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43186/0 (socket says 172.21.15.83:43186) 2024-09-19T07:20:57.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddd101640 1 -- 172.21.15.83:0/4008397463 learned_addr learned my addr 172.21.15.83:0/4008397463 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:57.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd815e320 con 0x7f2dd81513e0 2024-09-19T07:20:57.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd8157cd0 con 0x7f2dd806f690 2024-09-19T07:20:57.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2dd815e4a0 con 0x7f2dd806d1d0 2024-09-19T07:20:57.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2dd7fff640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2dd806f690 0x7f2dd815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:57.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddc900640 1 -- 172.21.15.83:0/4008397463 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2dd806f690 msgr2=0x7f2dd815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:57.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddc900640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2dd806f690 0x7f2dd815cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddc900640 1 -- 172.21.15.83:0/4008397463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2dd81513e0 msgr2=0x7f2dd80733f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:57.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddc900640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2dd81513e0 0x7f2dd80733f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddc900640 1 -- 172.21.15.83:0/4008397463 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd8072f10 con 0x7f2dd806d1d0 2024-09-19T07:20:57.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddd101640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2dd81513e0 0x7f2dd80733f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:57.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddc900640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2dd806d1d0 0x7f2dd8148d00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0ab000 tx=0x7f2dcc019040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:57.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2dd57fa640 1 -- 172.21.15.83:0/4008397463 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc0b8020 con 0x7f2dd806d1d0 2024-09-19T07:20:57.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2dd57fa640 1 -- 172.21.15.83:0/4008397463 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dcc0aa7d0 con 0x7f2dd806d1d0 2024-09-19T07:20:57.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2dd57fa640 1 -- 172.21.15.83:0/4008397463 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc0b0040 con 0x7f2dd806d1d0 2024-09-19T07:20:57.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd8149240 con 0x7f2dd806d1d0 2024-09-19T07:20:57.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.504+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd81ebb40 con 0x7f2dd806d1d0 2024-09-19T07:20:57.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.508+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd806cc90 con 0x7f2dd806d1d0 2024-09-19T07:20:57.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.508+0000 7f2dd57fa640 1 -- 172.21.15.83:0/4008397463 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2dcc0bb020 con 0x7f2dd806d1d0 2024-09-19T07:20:57.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.508+0000 7f2dd57fa640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2dbc0678c0 0x7f2dbc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:57.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.508+0000 7f2dd57fa640 1 -- 172.21.15.83:0/4008397463 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6637+0+0 (secure 0 0 0) 0x7f2dcc11fef0 con 0x7f2dd806d1d0 2024-09-19T07:20:57.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.508+0000 7f2dd7fff640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2dbc0678c0 0x7f2dbc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:57.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.508+0000 7f2dd7fff640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2dbc0678c0 0x7f2dbc069d80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2dd815cad0 tx=0x7f2dc4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:57.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.516+0000 7f2dd57fa640 1 -- 172.21.15.83:0/4008397463 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2dcc0c2030 con 0x7f2dd806d1d0 2024-09-19T07:20:57.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.900+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2dd806f470 con 0x7f2dd806d1d0 2024-09-19T07:20:57.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.900+0000 7f2dd57fa640 1 -- 172.21.15.83:0/4008397463 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6527 (secure 0 0 0) 0x7f2dcc015fb0 con 0x7f2dd806d1d0 2024-09-19T07:20:57.905 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:57.905 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"btime":"2024-09-19T07:19:44:439777+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:19:43.800486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/149004692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":149004692},{"type":"v1","addr":"172.21.15.184:6835","nonce":149004692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4232,"qdb_cluster":[4232]},"id":2}]} 2024-09-19T07:20:57.905 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 14 2024-09-19T07:20:57.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.904+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2dbc0678c0 msgr2=0x7f2dbc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:57.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.904+0000 7f2ddeb64640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2dbc0678c0 0x7f2dbc069d80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2dd815cad0 tx=0x7f2dc4009190 comp rx=0 tx=0).stop 2024-09-19T07:20:57.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.904+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2dd806d1d0 msgr2=0x7f2dd8148d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:57.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.904+0000 7f2ddeb64640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2dd806d1d0 0x7f2dd8148d00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0ab000 tx=0x7f2dcc019040 comp rx=0 tx=0).stop 2024-09-19T07:20:57.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.904+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 shutdown_connections 2024-09-19T07:20:57.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.904+0000 7f2ddeb64640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2dbc0678c0 0x7f2dbc069d80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.904+0000 7f2ddeb64640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2dd81513e0 0x7f2dd80733f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.904+0000 7f2ddeb64640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2dd806f690 0x7f2dd815cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.904+0000 7f2ddeb64640 1 --2- 172.21.15.83:0/4008397463 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2dd806d1d0 0x7f2dd8148d00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:57.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.904+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 >> 172.21.15.83:0/4008397463 conn(0x7f2dd80c5090 msgr2=0x7f2dd80ab430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:57.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.904+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 shutdown_connections 2024-09-19T07:20:57.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:57.904+0000 7f2ddeb64640 1 -- 172.21.15.83:0/4008397463 wait complete. 2024-09-19T07:20:57.932 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:20:59.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:59.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.252+0000 7fbae6654640 1 Processor -- start 2024-09-19T07:20:59.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.252+0000 7fbae6654640 1 -- start start 2024-09-19T07:20:59.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbae6654640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbae0070140 0x7fbae00c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbae6654640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbae01579a0 0x7fbae0150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbae006cef0 0x7fbae0154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42108/0 (socket says 172.21.15.83:42108) 2024-09-19T07:20:59.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbae6654640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae0070830 con 0x7fbae006cef0 2024-09-19T07:20:59.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadffff640 1 -- 172.21.15.83:0/4227317704 learned_addr learned my addr 172.21.15.83:0/4227317704 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:59.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbae6654640 1 -- 172.21.15.83:0/4227317704 --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbae0153cf0 con 0x7fbae01579a0 2024-09-19T07:20:59.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbae6654640 1 -- 172.21.15.83:0/4227317704 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbae0157e50 con 0x7fbae0070140 2024-09-19T07:20:59.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadf7fe640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbae0070140 0x7fbae00c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbae4bf1640 1 --2- 172.21.15.83:0/4227317704 >> v2:172.21.15.184:3300/0 conn(0x7fbae01579a0 0x7fbae0150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 188243541 0 0) 0x7fbae0070830 con 0x7fbae006cef0 2024-09-19T07:20:59.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbac8003530 con 0x7fbae006cef0 2024-09-19T07:20:59.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1853179655 0 0) 0x7fbac8003530 con 0x7fbae006cef0 2024-09-19T07:20:59.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbae0070140 msgr2=0x7fbae00c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:59.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbae0070140 0x7fbae00c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 >> v2:172.21.15.184:3300/0 conn(0x7fbae01579a0 msgr2=0x7fbae0150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:59.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 --2- 172.21.15.83:0/4227317704 >> v2:172.21.15.184:3300/0 conn(0x7fbae01579a0 0x7fbae0150ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae01539a0 con 0x7fbae006cef0 2024-09-19T07:20:59.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fbad0001680 con 0x7fbae006cef0 2024-09-19T07:20:59.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 >> v1:172.21.15.83:6789/0 conn(0x7fbae006cef0 legacy=0x7fbae0154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac80039c0 0x7fbac8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac80043b0 0x7fbac8004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbac8004fa0 0x7fbac800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbae0153cf0 con 0x7fbac8004fa0 2024-09-19T07:20:59.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbae0157e50 con 0x7fbac80043b0 2024-09-19T07:20:59.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbac800bf40 con 0x7fbac80039c0 2024-09-19T07:20:59.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadffff640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac80039c0 0x7fbac8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbae4bf1640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbac8004fa0 0x7fbac800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadffff640 1 -- 172.21.15.83:0/4227317704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac80043b0 msgr2=0x7fbac8004890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:20:59.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadf7fe640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac80043b0 0x7fbac8004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadffff640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac80043b0 0x7fbac8004890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadffff640 1 -- 172.21.15.83:0/4227317704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbac8004fa0 msgr2=0x7fbac800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:59.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadffff640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbac8004fa0 0x7fbac800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbadffff640 1 -- 172.21.15.83:0/4227317704 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac8003530 con 0x7fbac80039c0 2024-09-19T07:20:59.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.256+0000 7fbae4bf1640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbac8004fa0 0x7fbac800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:59.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.260+0000 7fbadffff640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac80039c0 0x7fbac8003de0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbad000c240 tx=0x7fbad001f3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e00b32467e0cca9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:59.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.260+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad0007d70 con 0x7fbac80039c0 2024-09-19T07:20:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.260+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbad0005ae0 con 0x7fbac80039c0 2024-09-19T07:20:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.260+0000 7fbadeffd640 1 -- 172.21.15.83:0/4227317704 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad0007070 con 0x7fbac80039c0 2024-09-19T07:20:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.260+0000 7fbae6654640 1 -- 172.21.15.83:0/4227317704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac80039c0 msgr2=0x7fbac8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.260+0000 7fbae6654640 1 --2- 172.21.15.83:0/4227317704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac80039c0 0x7fbac8003de0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbad000c240 tx=0x7fbad001f3d0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.260+0000 7fbae4bf1640 1 -- 172.21.15.83:0/4227317704 reap_dead start 2024-09-19T07:20:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.260+0000 7fbae6654640 1 -- 172.21.15.83:0/4227317704 shutdown_connections 2024-09-19T07:20:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.260+0000 7fbae6654640 1 -- 172.21.15.83:0/4227317704 >> 172.21.15.83:0/4227317704 conn(0x7fbae00c5090 msgr2=0x7fbae00c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.260+0000 7fbae6654640 1 -- 172.21.15.83:0/4227317704 shutdown_connections 2024-09-19T07:20:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.260+0000 7fbae6654640 1 -- 172.21.15.83:0/4227317704 wait complete. 2024-09-19T07:20:59.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbae6654640 1 Processor -- start 2024-09-19T07:20:59.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbae6654640 1 -- start start 2024-09-19T07:20:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbae6654640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac80039c0 0x7fbae0155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbae6654640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbae0156350 0x7fbae00bdf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbae6654640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbae00c6580 0x7fbae00be490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadffff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac80039c0 0x7fbae0155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadffff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac80039c0 0x7fbae0155e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:56522/0 (socket says 172.21.15.83:56522) 2024-09-19T07:20:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadffff640 1 -- 172.21.15.83:0/896806942 learned_addr learned my addr 172.21.15.83:0/896806942 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbae4bf1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbae00c6580 0x7fbae00be490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbae6654640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbae006f5c0 con 0x7fbae00c6580 2024-09-19T07:20:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbae6654640 1 -- 172.21.15.83:0/896806942 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbae006f2e0 con 0x7fbae0156350 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbae6654640 1 -- 172.21.15.83:0/896806942 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbae006f8a0 con 0x7fbac80039c0 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadf7fe640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbae0156350 0x7fbae00bdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadffff640 1 -- 172.21.15.83:0/896806942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbae0156350 msgr2=0x7fbae00bdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadffff640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbae0156350 0x7fbae00bdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadffff640 1 -- 172.21.15.83:0/896806942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbae00c6580 msgr2=0x7fbae00be490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadffff640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbae00c6580 0x7fbae00be490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadffff640 1 -- 172.21.15.83:0/896806942 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae0155920 con 0x7fbac80039c0 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadf7fe640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbae0156350 0x7fbae00bdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbae4bf1640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbae00c6580 0x7fbae00be490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadffff640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac80039c0 0x7fbae0155e10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbad001fc70 tx=0x7fbac8005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadcff9640 1 -- 172.21.15.83:0/896806942 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad00b4070 con 0x7fbac80039c0 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadcff9640 1 -- 172.21.15.83:0/896806942 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbad0005c90 con 0x7fbac80039c0 2024-09-19T07:20:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbae6654640 1 -- 172.21.15.83:0/896806942 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae01eeb90 con 0x7fbac80039c0 2024-09-19T07:20:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbadcff9640 1 -- 172.21.15.83:0/896806942 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad0004a30 con 0x7fbac80039c0 2024-09-19T07:20:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.264+0000 7fbae6654640 1 -- 172.21.15.83:0/896806942 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae00bf610 con 0x7fbac80039c0 2024-09-19T07:20:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.268+0000 7fbae6654640 1 -- 172.21.15.83:0/896806942 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae01eedb0 con 0x7fbac80039c0 2024-09-19T07:20:59.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.268+0000 7fbadcff9640 1 -- 172.21.15.83:0/896806942 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbad0004bd0 con 0x7fbac80039c0 2024-09-19T07:20:59.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.268+0000 7fbadcff9640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbacc0678f0 0x7fbacc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.268+0000 7fbadcff9640 1 -- 172.21.15.83:0/896806942 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6723+0+0 (secure 0 0 0) 0x7fbad0122d00 con 0x7fbac80039c0 2024-09-19T07:20:59.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.268+0000 7fbadf7fe640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbacc0678f0 0x7fbacc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.268+0000 7fbadf7fe640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbacc0678f0 0x7fbacc069db0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbae01ec6b0 tx=0x7fbad4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:59.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.276+0000 7fbadcff9640 1 -- 172.21.15.83:0/896806942 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbad0019140 con 0x7fbac80039c0 2024-09-19T07:20:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.520+0000 7fbae6654640 1 -- 172.21.15.83:0/896806942 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbae0068e20 con 0x7fbac80039c0 2024-09-19T07:20:59.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.520+0000 7fbadcff9640 1 -- 172.21.15.83:0/896806942 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5702 (secure 0 0 0) 0x7fbad00f2e80 con 0x7fbac80039c0 2024-09-19T07:20:59.525 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:59.525 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":16,"btime":"2024-09-19T07:20:58:929925+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:20:57.964644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:20:59.526 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 16 2024-09-19T07:20:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.524+0000 7fbae6654640 1 -- 172.21.15.83:0/896806942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbacc0678f0 msgr2=0x7fbacc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.524+0000 7fbae6654640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbacc0678f0 0x7fbacc069db0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbae01ec6b0 tx=0x7fbad4008040 comp rx=0 tx=0).stop 2024-09-19T07:20:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.524+0000 7fbae6654640 1 -- 172.21.15.83:0/896806942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac80039c0 msgr2=0x7fbae0155e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.524+0000 7fbae6654640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac80039c0 0x7fbae0155e10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbad001fc70 tx=0x7fbac8005840 comp rx=0 tx=0).stop 2024-09-19T07:20:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.524+0000 7fbae6654640 1 -- 172.21.15.83:0/896806942 shutdown_connections 2024-09-19T07:20:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.524+0000 7fbae6654640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbacc0678f0 0x7fbacc069db0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.524+0000 7fbae6654640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbae00c6580 0x7fbae00be490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.524+0000 7fbae6654640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbae0156350 0x7fbae00bdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.524+0000 7fbae6654640 1 --2- 172.21.15.83:0/896806942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac80039c0 0x7fbae0155e10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.524+0000 7fbae6654640 1 -- 172.21.15.83:0/896806942 >> 172.21.15.83:0/896806942 conn(0x7fbae00c5090 msgr2=0x7fbae0152c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.524+0000 7fbae6654640 1 -- 172.21.15.83:0/896806942 shutdown_connections 2024-09-19T07:20:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.524+0000 7fbae6654640 1 -- 172.21.15.83:0/896806942 wait complete. 2024-09-19T07:20:59.543 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-09-19T07:20:59.543 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:20:59.543 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:59.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbea09bb640 1 Processor -- start 2024-09-19T07:20:59.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbea09bb640 1 -- start start 2024-09-19T07:20:59.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbea09bb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbe9c0c4450 0x7fbe9c1594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbea09bb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe9c155ee0 con 0x7fbe9c0c3a20 2024-09-19T07:20:59.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbea09bb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe9c0684e0 con 0x7fbe9c06f3f0 2024-09-19T07:20:59.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbea09bb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbe9c068290 con 0x7fbe9c0c4450 2024-09-19T07:20:59.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9a59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbe9c0c3a20 0x7fbe9c1566c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42112/0 (socket says 172.21.15.83:42112) 2024-09-19T07:20:59.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9a59c640 1 -- 172.21.15.83:0/723191148 learned_addr learned my addr 172.21.15.83:0/723191148 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:20:59.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe99d9b640 1 --2- 172.21.15.83:0/723191148 >> v2:172.21.15.184:3300/0 conn(0x7fbe9c0c4450 0x7fbe9c1594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3271522219 0 0) 0x7fbe9c155ee0 con 0x7fbe9c0c3a20 2024-09-19T07:20:59.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe84003530 con 0x7fbe9c0c3a20 2024-09-19T07:20:59.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2105158307 0 0) 0x7fbe84003530 con 0x7fbe9c0c3a20 2024-09-19T07:20:59.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 >> v2:172.21.15.184:3300/0 conn(0x7fbe9c0c4450 msgr2=0x7fbe9c1594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:59.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 --2- 172.21.15.83:0/723191148 >> v2:172.21.15.184:3300/0 conn(0x7fbe9c0c4450 0x7fbe9c1594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 >> v1:172.21.15.184:6789/0 conn(0x7fbe9c06f3f0 legacy=0x7fbe9c14c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9c0687c0 con 0x7fbe9c0c3a20 2024-09-19T07:20:59.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fbe8c001680 con 0x7fbe9c0c3a20 2024-09-19T07:20:59.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 >> v1:172.21.15.83:6789/0 conn(0x7fbe9c0c3a20 legacy=0x7fbe9c1566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 --2- 172.21.15.83:0/723191148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 0x7fbe84003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 --2- 172.21.15.83:0/723191148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe84004810 0x7fbe84004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 --2- 172.21.15.83:0/723191148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe84005220 0x7fbe8400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c068290 con 0x7fbe840039c0 2024-09-19T07:20:59.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8400c150 con 0x7fbe84005220 2024-09-19T07:20:59.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbe8400c350 con 0x7fbe84004810 2024-09-19T07:20:59.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9a59c640 1 --2- 172.21.15.83:0/723191148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 0x7fbe84003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe99d9b640 1 --2- 172.21.15.83:0/723191148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe84004810 0x7fbe84004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9ad9d640 1 --2- 172.21.15.83:0/723191148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe84005220 0x7fbe8400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9a59c640 1 -- 172.21.15.83:0/723191148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe84004810 msgr2=0x7fbe84004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:59.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9a59c640 1 --2- 172.21.15.83:0/723191148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe84004810 0x7fbe84004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9a59c640 1 -- 172.21.15.83:0/723191148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe84005220 msgr2=0x7fbe8400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:20:59.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9a59c640 1 --2- 172.21.15.83:0/723191148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe84005220 0x7fbe8400b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9ad9d640 1 -- 172.21.15.83:0/723191148 reap_dead start 2024-09-19T07:20:59.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.664+0000 7fbe9a59c640 1 -- 172.21.15.83:0/723191148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe84003530 con 0x7fbe840039c0 2024-09-19T07:20:59.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9a59c640 1 --2- 172.21.15.83:0/723191148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 0x7fbe84003e00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00c1c0 tx=0x7fbe8c01cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc98a3f920281d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:59.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c007d00 con 0x7fbe840039c0 2024-09-19T07:20:59.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe8c0a9020 con 0x7fbe840039c0 2024-09-19T07:20:59.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9959a640 1 -- 172.21.15.83:0/723191148 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c005b00 con 0x7fbe840039c0 2024-09-19T07:20:59.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 -- 172.21.15.83:0/723191148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 msgr2=0x7fbe84003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 --2- 172.21.15.83:0/723191148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 0x7fbe84003e00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00c1c0 tx=0x7fbe8c01cc30 comp rx=0 tx=0).stop 2024-09-19T07:20:59.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 -- 172.21.15.83:0/723191148 shutdown_connections 2024-09-19T07:20:59.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 --2- 172.21.15.83:0/723191148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 0x7fbe84003e00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 -- 172.21.15.83:0/723191148 >> 172.21.15.83:0/723191148 conn(0x7fbe9c0c48b0 msgr2=0x7fbe9c0c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:59.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 -- 172.21.15.83:0/723191148 shutdown_connections 2024-09-19T07:20:59.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 -- 172.21.15.83:0/723191148 wait complete. 2024-09-19T07:20:59.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 Processor -- start 2024-09-19T07:20:59.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 -- start start 2024-09-19T07:20:59.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 0x7fbe9c0aa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe9c071850 0x7fbe9c1f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe9c0acae0 0x7fbe9c0acf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c06fcd0 con 0x7fbe840039c0 2024-09-19T07:20:59.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c0c4010 con 0x7fbe9c0acae0 2024-09-19T07:20:59.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbe9c14c410 con 0x7fbe9c071850 2024-09-19T07:20:59.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 0x7fbe9c0aa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 0x7fbe9c0aa7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43238/0 (socket says 172.21.15.83:43238) 2024-09-19T07:20:59.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9a59c640 1 -- 172.21.15.83:0/126822067 learned_addr learned my addr 172.21.15.83:0/126822067 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:20:59.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe99d9b640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe9c071850 0x7fbe9c1f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9ad9d640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe9c0acae0 0x7fbe9c0acf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9a59c640 1 -- 172.21.15.83:0/126822067 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe9c071850 msgr2=0x7fbe9c1f1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9a59c640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe9c071850 0x7fbe9c1f1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9a59c640 1 -- 172.21.15.83:0/126822067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe9c0acae0 msgr2=0x7fbe9c0acf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9a59c640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe9c0acae0 0x7fbe9c0acf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9a59c640 1 -- 172.21.15.83:0/126822067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9c0c5520 con 0x7fbe840039c0 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9ad9d640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe9c0acae0 0x7fbe9c0acf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe9a59c640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 0x7fbe9c0aa7f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c01ca60 tx=0x7fbe84005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe8affd640 1 -- 172.21.15.83:0/126822067 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c0a5a20 con 0x7fbe840039c0 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe8affd640 1 -- 172.21.15.83:0/126822067 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe8c0b2070 con 0x7fbe840039c0 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 -- 172.21.15.83:0/126822067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe9c0735d0 con 0x7fbe840039c0 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbe8affd640 1 -- 172.21.15.83:0/126822067 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c0a5de0 con 0x7fbe840039c0 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.668+0000 7fbea09bb640 1 -- 172.21.15.83:0/126822067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe9c0732f0 con 0x7fbe840039c0 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.672+0000 7fbea09bb640 1 -- 172.21.15.83:0/126822067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe9c14c7a0 con 0x7fbe840039c0 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.672+0000 7fbe8affd640 1 -- 172.21.15.83:0/126822067 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbe8c0047f0 con 0x7fbe840039c0 2024-09-19T07:20:59.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.672+0000 7fbe8affd640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe7c0678c0 0x7fbe7c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:20:59.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.672+0000 7fbe8affd640 1 -- 172.21.15.83:0/126822067 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6723+0+0 (secure 0 0 0) 0x7fbe8c11f280 con 0x7fbe840039c0 2024-09-19T07:20:59.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.672+0000 7fbe99d9b640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe7c0678c0 0x7fbe7c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:20:59.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.672+0000 7fbe99d9b640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe7c0678c0 0x7fbe7c069d80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c14c3c0 tx=0x7fbe90008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:20:59.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.676+0000 7fbe8affd640 1 -- 172.21.15.83:0/126822067 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbe8c016970 con 0x7fbe840039c0 2024-09-19T07:20:59.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 -- 172.21.15.83:0/126822067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe9c0c67e0 con 0x7fbe840039c0 2024-09-19T07:20:59.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbe8affd640 1 -- 172.21.15.83:0/126822067 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5702 (secure 0 0 0) 0x7fbe8c0ef400 con 0x7fbe840039c0 2024-09-19T07:20:59.868 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:20:59.868 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":16,"btime":"2024-09-19T07:20:58:929925+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:20:57.964644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:20:59.868 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 16 2024-09-19T07:20:59.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 -- 172.21.15.83:0/126822067 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe7c0678c0 msgr2=0x7fbe7c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe7c0678c0 0x7fbe7c069d80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c14c3c0 tx=0x7fbe90008040 comp rx=0 tx=0).stop 2024-09-19T07:20:59.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 -- 172.21.15.83:0/126822067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 msgr2=0x7fbe9c0aa7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:20:59.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 0x7fbe9c0aa7f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c01ca60 tx=0x7fbe84005840 comp rx=0 tx=0).stop 2024-09-19T07:20:59.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 -- 172.21.15.83:0/126822067 shutdown_connections 2024-09-19T07:20:59.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe7c0678c0 0x7fbe7c069d80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe9c0acae0 0x7fbe9c0acf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe9c071850 0x7fbe9c1f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 --2- 172.21.15.83:0/126822067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe840039c0 0x7fbe9c0aa7f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:20:59.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 -- 172.21.15.83:0/126822067 >> 172.21.15.83:0/126822067 conn(0x7fbe9c0c48b0 msgr2=0x7fbe9c1574f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:20:59.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 -- 172.21.15.83:0/126822067 shutdown_connections 2024-09-19T07:20:59.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:20:59.864+0000 7fbea09bb640 1 -- 172.21.15.83:0/126822067 wait complete. 2024-09-19T07:20:59.888 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:20:58:929925+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 16, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 8, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4254": { "addr": "172.21.15.83:6835/1819616410", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 1819616410, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 1819616410, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4254, "incarnation": 4, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:19:35.648535+0000", "qdb_cluster": [ 4254 ], "qdb_leader": 4254, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4254 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 16, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4229": { "addr": "172.21.15.184:6833/1773885233", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 1773885233, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 1773885233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 15, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:20:57.964644+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4229 } } } ], "standbys": [ { "addr": "172.21.15.83:6837/4265456507", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 4265456507, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 4265456507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:20:59.932 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:20:59.965 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:20:59.979 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:21:00.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc33d50b640 1 Processor -- start 2024-09-19T07:21:00.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc33d50b640 1 -- start start 2024-09-19T07:21:00.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc33d50b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc33814a500 0x7fc3380c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:00.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc33d50b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc338154540 0x7fc338157490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:00.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc33d50b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc3380c4550 0x7fc3381579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:00.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc33d50b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc338151750 con 0x7fc33814a500 2024-09-19T07:21:00.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc33d50b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc338157f10 con 0x7fc3380c4550 2024-09-19T07:21:00.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc33d50b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc3380c7e90 con 0x7fc338154540 2024-09-19T07:21:00.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc336ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc33814a500 0x7fc3380c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:00.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc336ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc33814a500 0x7fc3380c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43244/0 (socket says 172.21.15.83:43244) 2024-09-19T07:21:00.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc336ffd640 1 -- 172.21.15.83:0/2917555200 learned_addr learned my addr 172.21.15.83:0/2917555200 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:00.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc32e7fc640 1 --2- 172.21.15.83:0/2917555200 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc338154540 0x7fc338157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:00.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc3377fe640 1 --2- 172.21.15.83:0/2917555200 >> v2:172.21.15.184:3300/0 conn(0x7fc3380c4550 0x7fc3381579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:00.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc336ffd640 1 -- 172.21.15.83:0/2917555200 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc338154540 msgr2=0x7fc338157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:00.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc336ffd640 1 --2- 172.21.15.83:0/2917555200 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc338154540 0x7fc338157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:00.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc336ffd640 1 -- 172.21.15.83:0/2917555200 >> v2:172.21.15.184:3300/0 conn(0x7fc3380c4550 msgr2=0x7fc3381579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:00.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc336ffd640 1 --2- 172.21.15.83:0/2917555200 >> v2:172.21.15.184:3300/0 conn(0x7fc3380c4550 0x7fc3381579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:00.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc336ffd640 1 -- 172.21.15.83:0/2917555200 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc338073110 con 0x7fc33814a500 2024-09-19T07:21:00.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.036+0000 7fc3377fe640 1 --2- 172.21.15.83:0/2917555200 >> v2:172.21.15.184:3300/0 conn(0x7fc3380c4550 0x7fc3381579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:21:00.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc336ffd640 1 --2- 172.21.15.83:0/2917555200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc33814a500 0x7fc3380c4010 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc3380c3c10 tx=0x7fc32801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d443093fa91f9512 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:00.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc3367fc640 1 -- 172.21.15.83:0/2917555200 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3280aa610 con 0x7fc33814a500 2024-09-19T07:21:00.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc3367fc640 1 -- 172.21.15.83:0/2917555200 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3280af070 con 0x7fc33814a500 2024-09-19T07:21:00.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc3367fc640 1 -- 172.21.15.83:0/2917555200 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3280aa9d0 con 0x7fc33814a500 2024-09-19T07:21:00.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 -- 172.21.15.83:0/2917555200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc33814a500 msgr2=0x7fc3380c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:00.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 --2- 172.21.15.83:0/2917555200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc33814a500 0x7fc3380c4010 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc3380c3c10 tx=0x7fc32801c4f0 comp rx=0 tx=0).stop 2024-09-19T07:21:00.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 -- 172.21.15.83:0/2917555200 shutdown_connections 2024-09-19T07:21:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 --2- 172.21.15.83:0/2917555200 >> v2:172.21.15.184:3300/0 conn(0x7fc3380c4550 0x7fc3381579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 --2- 172.21.15.83:0/2917555200 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc338154540 0x7fc338157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 --2- 172.21.15.83:0/2917555200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc33814a500 0x7fc3380c4010 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 -- 172.21.15.83:0/2917555200 >> 172.21.15.83:0/2917555200 conn(0x7fc3380c9610 msgr2=0x7fc3380c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 -- 172.21.15.83:0/2917555200 shutdown_connections 2024-09-19T07:21:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 -- 172.21.15.83:0/2917555200 wait complete. 2024-09-19T07:21:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 Processor -- start 2024-09-19T07:21:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 -- start start 2024-09-19T07:21:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3380c4550 0x7fc338159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc33814a500 0x7fc3380acbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc338154540 0x7fc3380ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc3380c76d0 con 0x7fc33814a500 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc3380741f0 con 0x7fc3380c4550 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc3380c7850 con 0x7fc338154540 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc32e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc33814a500 0x7fc3380acbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc32e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc33814a500 0x7fc3380acbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43248/0 (socket says 172.21.15.83:43248) 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc32e7fc640 1 -- 172.21.15.83:0/2235679186 learned_addr learned my addr 172.21.15.83:0/2235679186 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc3377fe640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc338154540 0x7fc3380ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc336ffd640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3380c4550 0x7fc338159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc32e7fc640 1 -- 172.21.15.83:0/2235679186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc338154540 msgr2=0x7fc3380ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc32e7fc640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc338154540 0x7fc3380ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc32e7fc640 1 -- 172.21.15.83:0/2235679186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3380c4550 msgr2=0x7fc338159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc32e7fc640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3380c4550 0x7fc338159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc32e7fc640 1 -- 172.21.15.83:0/2235679186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc338144f70 con 0x7fc33814a500 2024-09-19T07:21:00.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc32e7fc640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc33814a500 0x7fc3380acbd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc32000d5d0 tx=0x7fc32000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:00.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc32ffff640 1 -- 172.21.15.83:0/2235679186 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320004240 con 0x7fc33814a500 2024-09-19T07:21:00.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc32ffff640 1 -- 172.21.15.83:0/2235679186 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3200043e0 con 0x7fc33814a500 2024-09-19T07:21:00.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 -- 172.21.15.83:0/2235679186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3380af170 con 0x7fc33814a500 2024-09-19T07:21:00.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc32ffff640 1 -- 172.21.15.83:0/2235679186 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320003840 con 0x7fc33814a500 2024-09-19T07:21:00.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.040+0000 7fc33d50b640 1 -- 172.21.15.83:0/2235679186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3380ad110 con 0x7fc33814a500 2024-09-19T07:21:00.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.044+0000 7fc33d50b640 1 -- 172.21.15.83:0/2235679186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3380c4010 con 0x7fc33814a500 2024-09-19T07:21:00.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.044+0000 7fc32ffff640 1 -- 172.21.15.83:0/2235679186 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc3200039e0 con 0x7fc33814a500 2024-09-19T07:21:00.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.044+0000 7fc32ffff640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc3180678c0 0x7fc318069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:00.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.044+0000 7fc336ffd640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc3180678c0 0x7fc318069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:00.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.044+0000 7fc32ffff640 1 -- 172.21.15.83:0/2235679186 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6723+0+0 (secure 0 0 0) 0x7fc320086200 con 0x7fc33814a500 2024-09-19T07:21:00.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.044+0000 7fc336ffd640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc3180678c0 0x7fc318069d80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc3280ae000 tx=0x7fc3280a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:00.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.048+0000 7fc32ffff640 1 -- 172.21.15.83:0/2235679186 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc3200569d0 con 0x7fc33814a500 2024-09-19T07:21:00.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.232+0000 7fc33d50b640 1 -- 172.21.15.83:0/2235679186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc338144790 con 0x7fc33814a500 2024-09-19T07:21:00.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc32ffff640 1 -- 172.21.15.83:0/2235679186 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5699 (secure 0 0 0) 0x7fc3200562d0 con 0x7fc33814a500 2024-09-19T07:21:00.239 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:21:00.239 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":17,"btime":"2024-09-19T07:20:59:931341+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:20:58.934447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:21:00.239 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 17 2024-09-19T07:21:00.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc33d50b640 1 -- 172.21.15.83:0/2235679186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc3180678c0 msgr2=0x7fc318069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:00.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc33d50b640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc3180678c0 0x7fc318069d80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc3280ae000 tx=0x7fc3280a7000 comp rx=0 tx=0).stop 2024-09-19T07:21:00.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc33d50b640 1 -- 172.21.15.83:0/2235679186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc33814a500 msgr2=0x7fc3380acbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:00.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc33d50b640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc33814a500 0x7fc3380acbd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc32000d5d0 tx=0x7fc32000da90 comp rx=0 tx=0).stop 2024-09-19T07:21:00.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc33d50b640 1 -- 172.21.15.83:0/2235679186 shutdown_connections 2024-09-19T07:21:00.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc33d50b640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc3180678c0 0x7fc318069d80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:00.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc33d50b640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc338154540 0x7fc3380ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:00.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc33d50b640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc33814a500 0x7fc3380acbd0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:00.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc33d50b640 1 --2- 172.21.15.83:0/2235679186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3380c4550 0x7fc338159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:00.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc33d50b640 1 -- 172.21.15.83:0/2235679186 >> 172.21.15.83:0/2235679186 conn(0x7fc3380c9610 msgr2=0x7fc338155e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:00.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc33d50b640 1 -- 172.21.15.83:0/2235679186 shutdown_connections 2024-09-19T07:21:00.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:00.236+0000 7fc33d50b640 1 -- 172.21.15.83:0/2235679186 wait complete. 2024-09-19T07:21:00.256 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:21:01.606 INFO:tasks.workunit.client.1.smithi184.stderr:Updating files: 96% (12387/12896) Updating files: 97% (12510/12896) Updating files: 98% (12639/12896) Updating files: 99% (12768/12896) Updating files: 100% (12896/12896) Updating files: 100% (12896/12896), done. 2024-09-19T07:21:01.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:21:02.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc576858640 1 Processor -- start 2024-09-19T07:21:02.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc576858640 1 -- start start 2024-09-19T07:21:02.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc576858640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc570070140 0x7fc5700c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc576858640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc570070830 con 0x7fc57006cef0 2024-09-19T07:21:02.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc576858640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc570157df0 con 0x7fc5701579a0 2024-09-19T07:21:02.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc576858640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc570153cf0 con 0x7fc570070140 2024-09-19T07:21:02.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc57006cef0 0x7fc570154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33508/0 (socket says 172.21.15.83:33508) 2024-09-19T07:21:02.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56ffff640 1 -- 172.21.15.83:0/3849655022 learned_addr learned my addr 172.21.15.83:0/3849655022 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:21:02.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc567fff640 1 --2- 172.21.15.83:0/3849655022 >> v2:172.21.15.184:3300/0 conn(0x7fc570070140 0x7fc5700c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 931020233 0 0) 0x7fc570070830 con 0x7fc57006cef0 2024-09-19T07:21:02.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc55c003530 con 0x7fc57006cef0 2024-09-19T07:21:02.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 73714700 0 0) 0x7fc55c003530 con 0x7fc57006cef0 2024-09-19T07:21:02.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 >> v2:172.21.15.184:3300/0 conn(0x7fc570070140 msgr2=0x7fc5700c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:02.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 --2- 172.21.15.83:0/3849655022 >> v2:172.21.15.184:3300/0 conn(0x7fc570070140 0x7fc5700c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 >> v1:172.21.15.184:6789/0 conn(0x7fc5701579a0 legacy=0x7fc570150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5701539a0 con 0x7fc57006cef0 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fc560001680 con 0x7fc57006cef0 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 >> v1:172.21.15.83:6789/0 conn(0x7fc57006cef0 legacy=0x7fc570154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 --2- 172.21.15.83:0/3849655022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc55c0039c0 0x7fc55c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 --2- 172.21.15.83:0/3849655022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc55c004810 0x7fc55c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 --2- 172.21.15.83:0/3849655022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc55c0052f0 0x7fc55c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc570153cf0 con 0x7fc55c0052f0 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc55c00c1c0 con 0x7fc55c004810 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc55c00c3c0 con 0x7fc55c0039c0 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc567fff640 1 --2- 172.21.15.83:0/3849655022 >> v2:172.21.15.184:3300/0 conn(0x7fc570070140 0x7fc5700c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc574df5640 1 --2- 172.21.15.83:0/3849655022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc55c0052f0 0x7fc55c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc56ffff640 1 --2- 172.21.15.83:0/3849655022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc55c0039c0 0x7fc55c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc567fff640 1 --2- 172.21.15.83:0/3849655022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc55c004810 0x7fc55c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc574df5640 1 -- 172.21.15.83:0/3849655022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc55c0039c0 msgr2=0x7fc55c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:02.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc574df5640 1 --2- 172.21.15.83:0/3849655022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc55c0039c0 0x7fc55c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc574df5640 1 -- 172.21.15.83:0/3849655022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc55c004810 msgr2=0x7fc55c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:02.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc574df5640 1 --2- 172.21.15.83:0/3849655022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc55c004810 0x7fc55c004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.000+0000 7fc574df5640 1 -- 172.21.15.83:0/3849655022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55c003530 con 0x7fc55c0052f0 2024-09-19T07:21:02.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc574df5640 1 -- 172.21.15.83:0/3849655022 reap_dead start 2024-09-19T07:21:02.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc574df5640 1 --2- 172.21.15.83:0/3849655022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc55c0052f0 0x7fc55c00b9d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc568001e40 tx=0x7fc56801c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b36f4cb9d289b8b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:02.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568001a90 con 0x7fc55c0052f0 2024-09-19T07:21:02.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc568003b80 con 0x7fc55c0052f0 2024-09-19T07:21:02.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc56f7fe640 1 -- 172.21.15.83:0/3849655022 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5680050d0 con 0x7fc55c0052f0 2024-09-19T07:21:02.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 -- 172.21.15.83:0/3849655022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc55c0052f0 msgr2=0x7fc55c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 --2- 172.21.15.83:0/3849655022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc55c0052f0 0x7fc55c00b9d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc568001e40 tx=0x7fc56801c760 comp rx=0 tx=0).stop 2024-09-19T07:21:02.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 -- 172.21.15.83:0/3849655022 shutdown_connections 2024-09-19T07:21:02.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 --2- 172.21.15.83:0/3849655022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc55c0052f0 0x7fc55c00b9d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 -- 172.21.15.83:0/3849655022 >> 172.21.15.83:0/3849655022 conn(0x7fc5700c5090 msgr2=0x7fc5700c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:02.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 -- 172.21.15.83:0/3849655022 shutdown_connections 2024-09-19T07:21:02.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 -- 172.21.15.83:0/3849655022 wait complete. 2024-09-19T07:21:02.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 Processor -- start 2024-09-19T07:21:02.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 -- start start 2024-09-19T07:21:02.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc55c0052f0 0x7fc5700c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5700c19c0 0x7fc5700c1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc57014a4d0 0x7fc57014a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc57006d2d0 con 0x7fc57014a4d0 2024-09-19T07:21:02.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc57006f8a0 con 0x7fc5700c19c0 2024-09-19T07:21:02.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc576858640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc57006d450 con 0x7fc55c0052f0 2024-09-19T07:21:02.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc574df5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc57014a4d0 0x7fc57014a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc574df5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc57014a4d0 0x7fc57014a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50196/0 (socket says 172.21.15.83:50196) 2024-09-19T07:21:02.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.004+0000 7fc574df5640 1 -- 172.21.15.83:0/1560849288 learned_addr learned my addr 172.21.15.83:0/1560849288 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:02.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc567fff640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5700c19c0 0x7fc5700c1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc56ffff640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc55c0052f0 0x7fc5700c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc574df5640 1 -- 172.21.15.83:0/1560849288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc55c0052f0 msgr2=0x7fc5700c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc574df5640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc55c0052f0 0x7fc5700c7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc574df5640 1 -- 172.21.15.83:0/1560849288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5700c19c0 msgr2=0x7fc5700c1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc574df5640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5700c19c0 0x7fc5700c1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc574df5640 1 -- 172.21.15.83:0/1560849288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5700c6820 con 0x7fc57014a4d0 2024-09-19T07:21:02.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc574df5640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc57014a4d0 0x7fc57014a950 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc56801c7a0 tx=0x7fc55c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:02.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc56d7fa640 1 -- 172.21.15.83:0/1560849288 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568007dc0 con 0x7fc57014a4d0 2024-09-19T07:21:02.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc56d7fa640 1 -- 172.21.15.83:0/1560849288 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc568001260 con 0x7fc57014a4d0 2024-09-19T07:21:02.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc56d7fa640 1 -- 172.21.15.83:0/1560849288 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568006b60 con 0x7fc57014a4d0 2024-09-19T07:21:02.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc576858640 1 -- 172.21.15.83:0/1560849288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5700c6b00 con 0x7fc57014a4d0 2024-09-19T07:21:02.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc576858640 1 -- 172.21.15.83:0/1560849288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5700c70c0 con 0x7fc57014a4d0 2024-09-19T07:21:02.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc576858640 1 -- 172.21.15.83:0/1560849288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5700c2f90 con 0x7fc57014a4d0 2024-09-19T07:21:02.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc56d7fa640 1 -- 172.21.15.83:0/1560849288 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc5680a9430 con 0x7fc57014a4d0 2024-09-19T07:21:02.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc56d7fa640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5500678f0 0x7fc550069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc56d7fa640 1 -- 172.21.15.83:0/1560849288 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6723+0+0 (secure 0 0 0) 0x7fc56811f350 con 0x7fc57014a4d0 2024-09-19T07:21:02.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc56ffff640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5500678f0 0x7fc550069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.008+0000 7fc56ffff640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5500678f0 0x7fc550069db0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc560001ec0 tx=0x7fc560009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:02.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.016+0000 7fc56d7fa640 1 -- 172.21.15.83:0/1560849288 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc568016470 con 0x7fc57014a4d0 2024-09-19T07:21:02.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.212+0000 7fc576858640 1 -- 172.21.15.83:0/1560849288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5700c3740 con 0x7fc57014a4d0 2024-09-19T07:21:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.212+0000 7fc56d7fa640 1 -- 172.21.15.83:0/1560849288 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5706 (secure 0 0 0) 0x7fc5680ef3c0 con 0x7fc57014a4d0 2024-09-19T07:21:02.217 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:21:02.218 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":18,"btime":"2024-09-19T07:21:00:932940+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:21:02.218 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 18 2024-09-19T07:21:02.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.216+0000 7fc576858640 1 -- 172.21.15.83:0/1560849288 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5500678f0 msgr2=0x7fc550069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.216+0000 7fc576858640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5500678f0 0x7fc550069db0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc560001ec0 tx=0x7fc560009040 comp rx=0 tx=0).stop 2024-09-19T07:21:02.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.216+0000 7fc576858640 1 -- 172.21.15.83:0/1560849288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc57014a4d0 msgr2=0x7fc57014a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.216+0000 7fc576858640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc57014a4d0 0x7fc57014a950 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc56801c7a0 tx=0x7fc55c005840 comp rx=0 tx=0).stop 2024-09-19T07:21:02.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.216+0000 7fc576858640 1 -- 172.21.15.83:0/1560849288 shutdown_connections 2024-09-19T07:21:02.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.216+0000 7fc576858640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5500678f0 0x7fc550069db0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.216+0000 7fc576858640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc57014a4d0 0x7fc57014a950 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.216+0000 7fc576858640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5700c19c0 0x7fc5700c1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.216+0000 7fc576858640 1 --2- 172.21.15.83:0/1560849288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc55c0052f0 0x7fc5700c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.216+0000 7fc576858640 1 -- 172.21.15.83:0/1560849288 >> 172.21.15.83:0/1560849288 conn(0x7fc5700c5090 msgr2=0x7fc570150b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:02.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.216+0000 7fc576858640 1 -- 172.21.15.83:0/1560849288 shutdown_connections 2024-09-19T07:21:02.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.216+0000 7fc576858640 1 -- 172.21.15.83:0/1560849288 wait complete. 2024-09-19T07:21:02.223 INFO:tasks.ceph.mds.d.smithi184.stderr:2024-09-19T07:21:02.220+0000 7f8116a60640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:21:02.235 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=4232 2024-09-19T07:21:02.236 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.b 2024-09-19T07:21:02.257 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:21:02.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6962df5640 1 Processor -- start 2024-09-19T07:21:02.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6962df5640 1 -- start start 2024-09-19T07:21:02.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6962df5640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f695c06cef0 0x7f695c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6962df5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f695c0c6fc0 0x7f695c157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6962df5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f695c0c7de0 con 0x7f695c0ab610 2024-09-19T07:21:02.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6962df5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f695c0c6a90 con 0x7f695c0c6fc0 2024-09-19T07:21:02.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6962df5640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f695c0c8360 con 0x7f695c06cef0 2024-09-19T07:21:02.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6961392640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f695c0c6fc0 0x7f695c157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f695bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f695c0ab610 0x7f695c0c3970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33512/0 (socket says 172.21.15.83:33512) 2024-09-19T07:21:02.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f695bfff640 1 -- 172.21.15.83:0/1806598816 learned_addr learned my addr 172.21.15.83:0/1806598816 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:21:02.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6960b91640 1 --2- 172.21.15.83:0/1806598816 >> v2:172.21.15.184:3300/0 conn(0x7f695c06cef0 0x7f695c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6961392640 1 -- 172.21.15.83:0/1806598816 >> v2:172.21.15.184:3300/0 conn(0x7f695c06cef0 msgr2=0x7f695c0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:02.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6961392640 1 --2- 172.21.15.83:0/1806598816 >> v2:172.21.15.184:3300/0 conn(0x7f695c06cef0 0x7f695c0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6961392640 1 -- 172.21.15.83:0/1806598816 >> v1:172.21.15.83:6789/0 conn(0x7f695c0ab610 legacy=0x7f695c0c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6961392640 1 -- 172.21.15.83:0/1806598816 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695c0c8080 con 0x7f695c0c6fc0 2024-09-19T07:21:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.360+0000 7f6961392640 1 --2- 172.21.15.83:0/1806598816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f695c0c6fc0 0x7f695c157890 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f695c0744a0 tx=0x7f695401c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e149bdb9dd4f29 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f695b7fe640 1 -- 172.21.15.83:0/1806598816 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69540b2070 con 0x7f695c0c6fc0 2024-09-19T07:21:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f695b7fe640 1 -- 172.21.15.83:0/1806598816 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69540a57e0 con 0x7f695c0c6fc0 2024-09-19T07:21:02.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f695b7fe640 1 -- 172.21.15.83:0/1806598816 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69540a5bb0 con 0x7f695c0c6fc0 2024-09-19T07:21:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 -- 172.21.15.83:0/1806598816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f695c0c6fc0 msgr2=0x7f695c157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 --2- 172.21.15.83:0/1806598816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f695c0c6fc0 0x7f695c157890 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f695c0744a0 tx=0x7f695401c810 comp rx=0 tx=0).stop 2024-09-19T07:21:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 -- 172.21.15.83:0/1806598816 shutdown_connections 2024-09-19T07:21:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 --2- 172.21.15.83:0/1806598816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f695c0c6fc0 0x7f695c157890 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 --2- 172.21.15.83:0/1806598816 >> v2:172.21.15.184:3300/0 conn(0x7f695c06cef0 0x7f695c0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 -- 172.21.15.83:0/1806598816 >> 172.21.15.83:0/1806598816 conn(0x7f695c0aa910 msgr2=0x7f695c0c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:02.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 -- 172.21.15.83:0/1806598816 shutdown_connections 2024-09-19T07:21:02.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 -- 172.21.15.83:0/1806598816 wait complete. 2024-09-19T07:21:02.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 Processor -- start 2024-09-19T07:21:02.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 -- start start 2024-09-19T07:21:02.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f695c06cef0 0x7f695c1f0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f695c0ab610 0x7f695c14b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f695c0c6fc0 0x7f695c14b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f695c1581e0 con 0x7f695c0c6fc0 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f695c06d520 con 0x7f695c0ab610 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f695c160e20 con 0x7f695c06cef0 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6961392640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f695c0c6fc0 0x7f695c14b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6961392640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f695c0c6fc0 0x7f695c14b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50214/0 (socket says 172.21.15.83:50214) 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6961392640 1 -- 172.21.15.83:0/2376761287 learned_addr learned my addr 172.21.15.83:0/2376761287 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6960b91640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f695c06cef0 0x7f695c1f0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f695bfff640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f695c0ab610 0x7f695c14b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6961392640 1 -- 172.21.15.83:0/2376761287 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f695c06cef0 msgr2=0x7f695c1f0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6961392640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f695c06cef0 0x7f695c1f0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6961392640 1 -- 172.21.15.83:0/2376761287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f695c0ab610 msgr2=0x7f695c14b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6961392640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f695c0ab610 0x7f695c14b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6961392640 1 -- 172.21.15.83:0/2376761287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695c1eded0 con 0x7f695c0c6fc0 2024-09-19T07:21:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6961392640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f695c0c6fc0 0x7f695c14b830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f69540b0040 tx=0x7f69540195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:02.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f69597fa640 1 -- 172.21.15.83:0/2376761287 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69540b2050 con 0x7f695c0c6fc0 2024-09-19T07:21:02.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f69597fa640 1 -- 172.21.15.83:0/2376761287 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69540be070 con 0x7f695c0c6fc0 2024-09-19T07:21:02.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 -- 172.21.15.83:0/2376761287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f695c1f0ae0 con 0x7f695c0c6fc0 2024-09-19T07:21:02.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f69597fa640 1 -- 172.21.15.83:0/2376761287 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69540ad870 con 0x7f695c0c6fc0 2024-09-19T07:21:02.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.364+0000 7f6962df5640 1 -- 172.21.15.83:0/2376761287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f695c1f0dc0 con 0x7f695c0c6fc0 2024-09-19T07:21:02.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.368+0000 7f6962df5640 1 -- 172.21.15.83:0/2376761287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f695c0abe80 con 0x7f695c0c6fc0 2024-09-19T07:21:02.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.368+0000 7f69597fa640 1 -- 172.21.15.83:0/2376761287 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f69540ada10 con 0x7f695c0c6fc0 2024-09-19T07:21:02.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.368+0000 7f69597fa640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69400678c0 0x7f6940069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:02.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.368+0000 7f6960b91640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69400678c0 0x7f6940069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:02.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.368+0000 7f69597fa640 1 -- 172.21.15.83:0/2376761287 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6723+0+0 (secure 0 0 0) 0x7f6954121120 con 0x7f695c0c6fc0 2024-09-19T07:21:02.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.368+0000 7f6960b91640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69400678c0 0x7f6940069d80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f69500079c0 tx=0x7f6950008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:02.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.372+0000 7f69597fa640 1 -- 172.21.15.83:0/2376761287 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6954016520 con 0x7f695c0c6fc0 2024-09-19T07:21:02.382 INFO:tasks.workunit.client.1.smithi184.stderr:Note: switching to '6fad5dd73c93545e52f57f681e140a9c3529c11e'. 2024-09-19T07:21:02.382 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-09-19T07:21:02.382 INFO:tasks.workunit.client.1.smithi184.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-19T07:21:02.382 INFO:tasks.workunit.client.1.smithi184.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-19T07:21:02.382 INFO:tasks.workunit.client.1.smithi184.stderr:state without impacting any branches by switching back to a branch. 2024-09-19T07:21:02.382 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-09-19T07:21:02.382 INFO:tasks.workunit.client.1.smithi184.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-19T07:21:02.383 INFO:tasks.workunit.client.1.smithi184.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-19T07:21:02.383 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-09-19T07:21:02.383 INFO:tasks.workunit.client.1.smithi184.stderr: git switch -c 2024-09-19T07:21:02.383 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-09-19T07:21:02.383 INFO:tasks.workunit.client.1.smithi184.stderr:Or undo this operation with: 2024-09-19T07:21:02.383 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-09-19T07:21:02.383 INFO:tasks.workunit.client.1.smithi184.stderr: git switch - 2024-09-19T07:21:02.383 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-09-19T07:21:02.383 INFO:tasks.workunit.client.1.smithi184.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-19T07:21:02.383 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-09-19T07:21:02.383 INFO:tasks.workunit.client.1.smithi184.stderr:HEAD is now at 6fad5dd73c9 Merge pull request #59812 from neha-ojha/wip-rtd-squid 2024-09-19T07:21:02.389 DEBUG:teuthology.orchestra.run.smithi184:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-09-19T07:21:02.396 INFO:tasks.workunit.client.1.smithi184.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-19T07:21:02.399 INFO:tasks.workunit.client.1.smithi184.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-19T07:21:02.399 INFO:tasks.workunit.client.1.smithi184.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-19T07:21:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.556+0000 7f6962df5640 1 -- 172.21.15.83:0/2376761287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f695c0ada50 con 0x7f695c0c6fc0 2024-09-19T07:21:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.556+0000 7f69597fa640 1 -- 172.21.15.83:0/2376761287 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5706 (secure 0 0 0) 0x7f69540add00 con 0x7f695c0c6fc0 2024-09-19T07:21:02.562 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:21:02.562 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":18,"btime":"2024-09-19T07:21:00:932940+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:21:02.562 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 18 2024-09-19T07:21:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.560+0000 7f6962df5640 1 -- 172.21.15.83:0/2376761287 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69400678c0 msgr2=0x7f6940069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.560+0000 7f6962df5640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69400678c0 0x7f6940069d80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f69500079c0 tx=0x7f6950008040 comp rx=0 tx=0).stop 2024-09-19T07:21:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.560+0000 7f6962df5640 1 -- 172.21.15.83:0/2376761287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f695c0c6fc0 msgr2=0x7f695c14b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.560+0000 7f6962df5640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f695c0c6fc0 0x7f695c14b830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f69540b0040 tx=0x7f69540195b0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.560+0000 7f6962df5640 1 -- 172.21.15.83:0/2376761287 shutdown_connections 2024-09-19T07:21:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.560+0000 7f6962df5640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69400678c0 0x7f6940069d80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.560+0000 7f6962df5640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f695c0c6fc0 0x7f695c14b830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.560+0000 7f6962df5640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f695c0ab610 0x7f695c14b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.560+0000 7f6962df5640 1 --2- 172.21.15.83:0/2376761287 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f695c06cef0 0x7f695c1f0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.560+0000 7f6962df5640 1 -- 172.21.15.83:0/2376761287 >> 172.21.15.83:0/2376761287 conn(0x7f695c0aa910 msgr2=0x7f695c074f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.560+0000 7f6962df5640 1 -- 172.21.15.83:0/2376761287 shutdown_connections 2024-09-19T07:21:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:02.560+0000 7f6962df5640 1 -- 172.21.15.83:0/2376761287 wait complete. 2024-09-19T07:21:02.580 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:21:02.955 INFO:tasks.workunit.client.0.smithi083.stderr:Note: switching to '6fad5dd73c93545e52f57f681e140a9c3529c11e'. 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr:state without impacting any branches by switching back to a branch. 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr: git switch -c 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr:Or undo this operation with: 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-19T07:21:02.956 INFO:tasks.workunit.client.0.smithi083.stderr: git switch - 2024-09-19T07:21:02.957 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-19T07:21:02.957 INFO:tasks.workunit.client.0.smithi083.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-19T07:21:02.957 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-19T07:21:02.957 INFO:tasks.workunit.client.0.smithi083.stderr:HEAD is now at 6fad5dd73c9 Merge pull request #59812 from neha-ojha/wip-rtd-squid 2024-09-19T07:21:02.963 DEBUG:teuthology.orchestra.run.smithi083:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-19T07:21:02.969 INFO:tasks.workunit.client.0.smithi083.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-19T07:21:02.973 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-19T07:21:02.973 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-19T07:21:03.192 INFO:tasks.workunit.client.1.smithi184.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-19T07:21:03.272 INFO:tasks.workunit.client.1.smithi184.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-19T07:21:03.316 INFO:tasks.workunit.client.1.smithi184.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-19T07:21:03.318 INFO:tasks.workunit.client.1.smithi184.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-19T07:21:03.318 INFO:tasks.workunit.client.1.smithi184.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-19T07:21:03.365 INFO:tasks.workunit.client.1.smithi184.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-19T07:21:03.369 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:21:03.369 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-09-19T07:21:03.416 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.1... 2024-09-19T07:21:03.417 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-09-19T07:21:03.417 DEBUG:teuthology.orchestra.run.smithi184:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6fad5dd73c93545e52f57f681e140a9c3529c11e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/ffsb.sh 2024-09-19T07:21:03.476 INFO:tasks.workunit.client.1.smithi184.stderr:++ dirname /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/ffsb.sh 2024-09-19T07:21:03.477 INFO:tasks.workunit.client.1.smithi184.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.1/qa/workunits/suites 2024-09-19T07:21:03.477 INFO:tasks.workunit.client.1.smithi184.stderr:+ trap -- retry EXIT 2024-09-19T07:21:03.477 INFO:tasks.workunit.client.1.smithi184.stderr:+ rm -rf ffsb 2024-09-19T07:21:03.478 INFO:tasks.workunit.client.1.smithi184.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-09-19T07:21:03.483 INFO:tasks.workunit.client.1.smithi184.stderr:Cloning into 'ffsb'... 2024-09-19T07:21:03.784 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-19T07:21:03.873 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-19T07:21:03.933 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-19T07:21:03.935 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-19T07:21:03.935 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-19T07:21:03.986 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-19T07:21:03.989 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:21:03.989 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-19T07:21:04.034 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-09-19T07:21:04.036 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-09-19T07:21:04.036 DEBUG:teuthology.orchestra.run.smithi083:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6fad5dd73c93545e52f57f681e140a9c3529c11e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-09-19T07:21:04.581 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:21:04.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c40b84640 1 Processor -- start 2024-09-19T07:21:04.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c40b84640 1 -- start start 2024-09-19T07:21:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c40b84640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c3c0ac290 0x7f1c3c0c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c40b84640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1c3c0c3a50 0x7f1c3c1563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c40b84640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c3c0ac6d0 con 0x7f1c3c0c3130 2024-09-19T07:21:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c40b84640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f1c3c06e110 con 0x7f1c3c0c3a50 2024-09-19T07:21:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c40b84640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1c3c0c4990 con 0x7f1c3c0ac290 2024-09-19T07:21:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c3ad9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1c3c0c3a50 0x7f1c3c1563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c3ad9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1c3c0c3a50 0x7f1c3c1563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:40368/0 (socket says 172.21.15.83:40368) 2024-09-19T07:21:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c3ad9d640 1 -- 172.21.15.83:0/574799229 learned_addr learned my addr 172.21.15.83:0/574799229 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c3a59c640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f1c3c0c3130 0x7f1c3c06da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:59332/0 (socket says 172.21.15.83:59332) 2024-09-19T07:21:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c33fff640 1 --2- 172.21.15.83:0/574799229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c3c0ac290 0x7f1c3c0c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c33fff640 1 -- 172.21.15.83:0/574799229 >> v2:172.21.15.184:3300/0 conn(0x7f1c3c0c3a50 msgr2=0x7f1c3c1563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c33fff640 1 --2- 172.21.15.83:0/574799229 >> v2:172.21.15.184:3300/0 conn(0x7f1c3c0c3a50 0x7f1c3c1563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c33fff640 1 -- 172.21.15.83:0/574799229 >> v1:172.21.15.184:6789/0 conn(0x7f1c3c0c3130 legacy=0x7f1c3c06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c33fff640 1 -- 172.21.15.83:0/574799229 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c3c0c4c70 con 0x7f1c3c0ac290 2024-09-19T07:21:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.716+0000 7f1c3ad9d640 1 --2- 172.21.15.83:0/574799229 >> v2:172.21.15.184:3300/0 conn(0x7f1c3c0c3a50 0x7f1c3c1563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:21:04.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c33fff640 1 --2- 172.21.15.83:0/574799229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c3c0ac290 0x7f1c3c0c3510 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c240097c0 tx=0x7f1c2401c150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d26725b19a34719 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c39d9b640 1 -- 172.21.15.83:0/574799229 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c240b0070 con 0x7f1c3c0ac290 2024-09-19T07:21:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c39d9b640 1 -- 172.21.15.83:0/574799229 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c240ab600 con 0x7f1c3c0ac290 2024-09-19T07:21:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c39d9b640 1 -- 172.21.15.83:0/574799229 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c240ab9d0 con 0x7f1c3c0ac290 2024-09-19T07:21:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c40b84640 1 -- 172.21.15.83:0/574799229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c3c0ac290 msgr2=0x7f1c3c0c3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c40b84640 1 --2- 172.21.15.83:0/574799229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c3c0ac290 0x7f1c3c0c3510 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c240097c0 tx=0x7f1c2401c150 comp rx=0 tx=0).stop 2024-09-19T07:21:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c40b84640 1 -- 172.21.15.83:0/574799229 shutdown_connections 2024-09-19T07:21:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c40b84640 1 --2- 172.21.15.83:0/574799229 >> v2:172.21.15.184:3300/0 conn(0x7f1c3c0c3a50 0x7f1c3c1563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c40b84640 1 --2- 172.21.15.83:0/574799229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c3c0ac290 0x7f1c3c0c3510 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c40b84640 1 -- 172.21.15.83:0/574799229 >> 172.21.15.83:0/574799229 conn(0x7f1c3c0c5060 msgr2=0x7f1c3c0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c40b84640 1 -- 172.21.15.83:0/574799229 shutdown_connections 2024-09-19T07:21:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c40b84640 1 -- 172.21.15.83:0/574799229 wait complete. 2024-09-19T07:21:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c40b84640 1 Processor -- start 2024-09-19T07:21:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.720+0000 7f1c40b84640 1 -- start start 2024-09-19T07:21:04.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c40b84640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c3c0ac290 0x7f1c3c15a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:04.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c40b84640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c3c0c3130 0x7f1c3c06ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:04.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c40b84640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c3c0c3a50 0x7f1c3c1ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:04.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c40b84640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3c0ad520 con 0x7f1c3c0c3130 2024-09-19T07:21:04.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c40b84640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3c0c82a0 con 0x7f1c3c0c3a50 2024-09-19T07:21:04.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c40b84640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1c3c0ad6a0 con 0x7f1c3c0ac290 2024-09-19T07:21:04.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c33fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c3c0c3130 0x7f1c3c06ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:04.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c33fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c3c0c3130 0x7f1c3c06ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50216/0 (socket says 172.21.15.83:50216) 2024-09-19T07:21:04.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c33fff640 1 -- 172.21.15.83:0/2297696845 learned_addr learned my addr 172.21.15.83:0/2297696845 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:04.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c3a59c640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c3c0ac290 0x7f1c3c15a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:04.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c3ad9d640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c3c0c3a50 0x7f1c3c1ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:04.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c33fff640 1 -- 172.21.15.83:0/2297696845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c3c0ac290 msgr2=0x7f1c3c15a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:04.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c33fff640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c3c0ac290 0x7f1c3c15a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:04.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c33fff640 1 -- 172.21.15.83:0/2297696845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c3c0c3a50 msgr2=0x7f1c3c1ee1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:04.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c33fff640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c3c0c3a50 0x7f1c3c1ee1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:04.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c33fff640 1 -- 172.21.15.83:0/2297696845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c3c15c470 con 0x7f1c3c0c3130 2024-09-19T07:21:04.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c33fff640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c3c0c3130 0x7f1c3c06ed90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1c240ae040 tx=0x7f1c24018ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:04.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c32ffd640 1 -- 172.21.15.83:0/2297696845 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c240b0040 con 0x7f1c3c0c3130 2024-09-19T07:21:04.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c32ffd640 1 -- 172.21.15.83:0/2297696845 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c240b43f0 con 0x7f1c3c0c3130 2024-09-19T07:21:04.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c32ffd640 1 -- 172.21.15.83:0/2297696845 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c240b47c0 con 0x7f1c3c0c3130 2024-09-19T07:21:04.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c40b84640 1 -- 172.21.15.83:0/2297696845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c3c15c750 con 0x7f1c3c0c3130 2024-09-19T07:21:04.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c40b84640 1 -- 172.21.15.83:0/2297696845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c3c0bdf90 con 0x7f1c3c0c3130 2024-09-19T07:21:04.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c40b84640 1 -- 172.21.15.83:0/2297696845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c3c15c920 con 0x7f1c3c0c3130 2024-09-19T07:21:04.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c32ffd640 1 -- 172.21.15.83:0/2297696845 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1c240b4960 con 0x7f1c3c0c3130 2024-09-19T07:21:04.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.724+0000 7f1c32ffd640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c1c0678c0 0x7f1c1c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.728+0000 7f1c32ffd640 1 -- 172.21.15.83:0/2297696845 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6723+0+0 (secure 0 0 0) 0x7f1c2411f960 con 0x7f1c3c0c3130 2024-09-19T07:21:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.728+0000 7f1c3a59c640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c1c0678c0 0x7f1c1c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.728+0000 7f1c3a59c640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c1c0678c0 0x7f1c1c069d80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c000f70 tx=0x7f1c2c007e50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.732+0000 7f1c32ffd640 1 -- 172.21.15.83:0/2297696845 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1c24016090 con 0x7f1c3c0c3130 2024-09-19T07:21:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.944+0000 7f1c40b84640 1 -- 172.21.15.83:0/2297696845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c3c0c3510 con 0x7f1c3c0c3130 2024-09-19T07:21:04.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.944+0000 7f1c32ffd640 1 -- 172.21.15.83:0/2297696845 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5706 (secure 0 0 0) 0x7f1c240efae0 con 0x7f1c3c0c3130 2024-09-19T07:21:04.948 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:21:04.948 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":18,"btime":"2024-09-19T07:21:00:932940+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:21:04.949 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 18 2024-09-19T07:21:04.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.944+0000 7f1c40b84640 1 -- 172.21.15.83:0/2297696845 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c1c0678c0 msgr2=0x7f1c1c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:04.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.944+0000 7f1c40b84640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c1c0678c0 0x7f1c1c069d80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c000f70 tx=0x7f1c2c007e50 comp rx=0 tx=0).stop 2024-09-19T07:21:04.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.944+0000 7f1c40b84640 1 -- 172.21.15.83:0/2297696845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c3c0c3130 msgr2=0x7f1c3c06ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:04.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.944+0000 7f1c40b84640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c3c0c3130 0x7f1c3c06ed90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1c240ae040 tx=0x7f1c24018ef0 comp rx=0 tx=0).stop 2024-09-19T07:21:04.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.948+0000 7f1c40b84640 1 -- 172.21.15.83:0/2297696845 shutdown_connections 2024-09-19T07:21:04.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.948+0000 7f1c40b84640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c1c0678c0 0x7f1c1c069d80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:04.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.948+0000 7f1c40b84640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c3c0c3a50 0x7f1c3c1ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:04.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.948+0000 7f1c40b84640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c3c0c3130 0x7f1c3c06ed90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:04.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.948+0000 7f1c40b84640 1 --2- 172.21.15.83:0/2297696845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c3c0ac290 0x7f1c3c15a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:04.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.948+0000 7f1c40b84640 1 -- 172.21.15.83:0/2297696845 >> 172.21.15.83:0/2297696845 conn(0x7f1c3c0c5060 msgr2=0x7f1c3c0c5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:04.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.948+0000 7f1c40b84640 1 -- 172.21.15.83:0/2297696845 shutdown_connections 2024-09-19T07:21:04.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:04.948+0000 7f1c40b84640 1 -- 172.21.15.83:0/2297696845 wait complete. 2024-09-19T07:21:04.968 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:21:05.504 INFO:tasks.workunit.client.1.smithi184.stderr:+ trap - EXIT 2024-09-19T07:21:05.505 INFO:tasks.workunit.client.1.smithi184.stderr:+ cd ffsb 2024-09-19T07:21:05.505 INFO:tasks.workunit.client.1.smithi184.stderr:+ ./configure 2024-09-19T07:21:05.670 INFO:tasks.workunit.client.1.smithi184.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-19T07:21:06.685 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether build environment is sane... yes 2024-09-19T07:21:06.704 INFO:tasks.workunit.client.1.smithi184.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-19T07:21:06.705 INFO:tasks.workunit.client.1.smithi184.stdout:checking for gawk... gawk 2024-09-19T07:21:06.729 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether make sets $(MAKE)... yes 2024-09-19T07:21:06.775 INFO:tasks.workunit.client.1.smithi184.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-19T07:21:06.776 INFO:tasks.workunit.client.1.smithi184.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-19T07:21:06.798 INFO:tasks.workunit.client.1.smithi184.stdout:checking for style of include used by make... GNU 2024-09-19T07:21:06.803 INFO:tasks.workunit.client.1.smithi184.stdout:checking for gcc... gcc 2024-09-19T07:21:06.900 INFO:tasks.workunit.client.1.smithi184.stdout:checking for C compiler default output file name... a.out 2024-09-19T07:21:06.903 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether the C compiler works... yes 2024-09-19T07:21:06.907 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether we are cross compiling... no 2024-09-19T07:21:06.969 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:21:06.971 INFO:tasks.workunit.client.1.smithi184.stdout:checking for suffix of executables... 2024-09-19T07:21:07.017 INFO:tasks.workunit.client.1.smithi184.stdout:checking for suffix of object files... o 2024-09-19T07:21:07.058 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether we are using the GNU C compiler... yes 2024-09-19T07:21:07.100 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether gcc accepts -g... yes 2024-09-19T07:21:07.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.096+0000 7fc38048b640 1 Processor -- start 2024-09-19T07:21:07.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.096+0000 7fc38048b640 1 -- start start 2024-09-19T07:21:07.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc38048b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc378071100 0x7fc3781545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:07.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc38048b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc3780c28e0 0x7fc378150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:07.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc38048b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3780a9e60 con 0x7fc378151480 2024-09-19T07:21:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc38048b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc378071b50 con 0x7fc3780c28e0 2024-09-19T07:21:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc38048b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc3780c1ff0 con 0x7fc378071100 2024-09-19T07:21:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37da26640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc3780c28e0 0x7fc378150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37ea28640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fc378151480 0x7fc37806dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:59340/0 (socket says 172.21.15.83:59340) 2024-09-19T07:21:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37da26640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc3780c28e0 0x7fc378150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50218/0 (socket says 172.21.15.83:50218) 2024-09-19T07:21:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37ea28640 1 -- 172.21.15.83:0/142693917 learned_addr learned my addr 172.21.15.83:0/142693917 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:21:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37e227640 1 --2- 172.21.15.83:0/142693917 >> v2:172.21.15.184:3300/0 conn(0x7fc378071100 0x7fc3781545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37da26640 1 -- 172.21.15.83:0/142693917 >> v2:172.21.15.184:3300/0 conn(0x7fc378071100 msgr2=0x7fc3781545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37da26640 1 --2- 172.21.15.83:0/142693917 >> v2:172.21.15.184:3300/0 conn(0x7fc378071100 0x7fc3781545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37da26640 1 -- 172.21.15.83:0/142693917 >> v1:172.21.15.184:6789/0 conn(0x7fc378151480 legacy=0x7fc37806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37da26640 1 -- 172.21.15.83:0/142693917 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3780aa140 con 0x7fc3780c28e0 2024-09-19T07:21:07.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37da26640 1 --2- 172.21.15.83:0/142693917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3780c28e0 0x7fc378150f40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc37806f620 tx=0x7fc37401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f418e3014b64d3c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:07.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37d225640 1 -- 172.21.15.83:0/142693917 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3740b0070 con 0x7fc3780c28e0 2024-09-19T07:21:07.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37d225640 1 -- 172.21.15.83:0/142693917 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3740ab600 con 0x7fc3780c28e0 2024-09-19T07:21:07.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.100+0000 7fc37d225640 1 -- 172.21.15.83:0/142693917 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3740ab9d0 con 0x7fc3780c28e0 2024-09-19T07:21:07.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 -- 172.21.15.83:0/142693917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3780c28e0 msgr2=0x7fc378150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:07.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 --2- 172.21.15.83:0/142693917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3780c28e0 0x7fc378150f40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc37806f620 tx=0x7fc37401c3f0 comp rx=0 tx=0).stop 2024-09-19T07:21:07.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 -- 172.21.15.83:0/142693917 shutdown_connections 2024-09-19T07:21:07.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 --2- 172.21.15.83:0/142693917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3780c28e0 0x7fc378150f40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:07.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 --2- 172.21.15.83:0/142693917 >> v2:172.21.15.184:3300/0 conn(0x7fc378071100 0x7fc3781545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:07.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 -- 172.21.15.83:0/142693917 >> 172.21.15.83:0/142693917 conn(0x7fc37806bc80 msgr2=0x7fc37806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:07.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 -- 172.21.15.83:0/142693917 shutdown_connections 2024-09-19T07:21:07.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 -- 172.21.15.83:0/142693917 wait complete. 2024-09-19T07:21:07.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 Processor -- start 2024-09-19T07:21:07.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 -- start start 2024-09-19T07:21:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc378071100 0x7fc378153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3780c28e0 0x7fc378142790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc378151480 0x7fc3780c4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc378071660 con 0x7fc3780c28e0 2024-09-19T07:21:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc3780714e0 con 0x7fc378151480 2024-09-19T07:21:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc378160c60 con 0x7fc378071100 2024-09-19T07:21:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc37da26640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3780c28e0 0x7fc378142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc37da26640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3780c28e0 0x7fc378142790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50228/0 (socket says 172.21.15.83:50228) 2024-09-19T07:21:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc37da26640 1 -- 172.21.15.83:0/3428295295 learned_addr learned my addr 172.21.15.83:0/3428295295 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc37e227640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc378071100 0x7fc378153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:07.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc37ea28640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc378151480 0x7fc3780c4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:07.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc37da26640 1 -- 172.21.15.83:0/3428295295 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc378071100 msgr2=0x7fc378153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:07.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc37da26640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc378071100 0x7fc378153880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:07.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc37da26640 1 -- 172.21.15.83:0/3428295295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc378151480 msgr2=0x7fc3780c4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:07.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc37da26640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc378151480 0x7fc3780c4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:07.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc37da26640 1 -- 172.21.15.83:0/3428295295 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc378152b60 con 0x7fc3780c28e0 2024-09-19T07:21:07.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc37da26640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3780c28e0 0x7fc378142790 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc374019000 tx=0x7fc374019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:07.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc36effd640 1 -- 172.21.15.83:0/3428295295 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3740b0040 con 0x7fc3780c28e0 2024-09-19T07:21:07.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc36effd640 1 -- 172.21.15.83:0/3428295295 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3740b9070 con 0x7fc3780c28e0 2024-09-19T07:21:07.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc36effd640 1 -- 172.21.15.83:0/3428295295 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3740b4650 con 0x7fc3780c28e0 2024-09-19T07:21:07.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 -- 172.21.15.83:0/3428295295 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc378142cd0 con 0x7fc3780c28e0 2024-09-19T07:21:07.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.104+0000 7fc38048b640 1 -- 172.21.15.83:0/3428295295 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc378146d10 con 0x7fc3780c28e0 2024-09-19T07:21:07.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.108+0000 7fc38048b640 1 -- 172.21.15.83:0/3428295295 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3781545e0 con 0x7fc3780c28e0 2024-09-19T07:21:07.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.108+0000 7fc36effd640 1 -- 172.21.15.83:0/3428295295 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc3740b47f0 con 0x7fc3780c28e0 2024-09-19T07:21:07.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.108+0000 7fc36effd640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc35c0678c0 0x7fc35c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:07.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.108+0000 7fc37e227640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc35c0678c0 0x7fc35c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:07.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.108+0000 7fc36effd640 1 -- 172.21.15.83:0/3428295295 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6723+0+0 (secure 0 0 0) 0x7fc37411fc60 con 0x7fc3780c28e0 2024-09-19T07:21:07.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.108+0000 7fc37e227640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc35c0678c0 0x7fc35c069d80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc378147970 tx=0x7fc368009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.112+0000 7fc36effd640 1 -- 172.21.15.83:0/3428295295 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc374016070 con 0x7fc3780c28e0 2024-09-19T07:21:07.180 INFO:tasks.workunit.client.1.smithi184.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-19T07:21:07.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.320+0000 7fc38048b640 1 -- 172.21.15.83:0/3428295295 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3780c4f30 con 0x7fc3780c28e0 2024-09-19T07:21:07.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.320+0000 7fc36effd640 1 -- 172.21.15.83:0/3428295295 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5706 (secure 0 0 0) 0x7fc37411f070 con 0x7fc3780c28e0 2024-09-19T07:21:07.330 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:21:07.330 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":18,"btime":"2024-09-19T07:21:00:932940+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:19:35.648535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/1819616410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1819616410},{"type":"v1","addr":"172.21.15.83:6835","nonce":1819616410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:21:07.331 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 18 2024-09-19T07:21:07.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.324+0000 7fc38048b640 1 -- 172.21.15.83:0/3428295295 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc35c0678c0 msgr2=0x7fc35c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:07.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.324+0000 7fc38048b640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc35c0678c0 0x7fc35c069d80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc378147970 tx=0x7fc368009190 comp rx=0 tx=0).stop 2024-09-19T07:21:07.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.324+0000 7fc38048b640 1 -- 172.21.15.83:0/3428295295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3780c28e0 msgr2=0x7fc378142790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:07.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.324+0000 7fc38048b640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3780c28e0 0x7fc378142790 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc374019000 tx=0x7fc374019190 comp rx=0 tx=0).stop 2024-09-19T07:21:07.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.324+0000 7fc38048b640 1 -- 172.21.15.83:0/3428295295 shutdown_connections 2024-09-19T07:21:07.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.324+0000 7fc38048b640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc35c0678c0 0x7fc35c069d80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:07.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.324+0000 7fc38048b640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc378151480 0x7fc3780c4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.324+0000 7fc38048b640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3780c28e0 0x7fc378142790 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.324+0000 7fc38048b640 1 --2- 172.21.15.83:0/3428295295 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc378071100 0x7fc378153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.324+0000 7fc38048b640 1 -- 172.21.15.83:0/3428295295 >> 172.21.15.83:0/3428295295 conn(0x7fc37806bc80 msgr2=0x7fc37806dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.324+0000 7fc38048b640 1 -- 172.21.15.83:0/3428295295 shutdown_connections 2024-09-19T07:21:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:07.324+0000 7fc38048b640 1 -- 172.21.15.83:0/3428295295 wait complete. 2024-09-19T07:21:07.341 INFO:tasks.workunit.client.1.smithi184.stdout:checking dependency style of gcc... gcc3 2024-09-19T07:21:07.345 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:21:07.433 INFO:tasks.workunit.client.1.smithi184.stdout:checking how to run the C preprocessor... gcc -E 2024-09-19T07:21:07.486 INFO:tasks.workunit.client.1.smithi184.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-19T07:21:07.487 INFO:tasks.workunit.client.1.smithi184.stdout:checking for egrep... /usr/bin/grep -E 2024-09-19T07:21:07.499 INFO:tasks.workunit.client.1.smithi184.stdout:checking for AIX... no 2024-09-19T07:21:07.525 INFO:tasks.workunit.client.1.smithi184.stdout:checking for gcc... (cached) gcc 2024-09-19T07:21:07.537 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-19T07:21:07.538 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-19T07:21:07.539 INFO:tasks.workunit.client.1.smithi184.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-19T07:21:07.540 INFO:tasks.workunit.client.1.smithi184.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-19T07:21:07.634 INFO:tasks.workunit.client.1.smithi184.stdout:checking for main in -lm... yes 2024-09-19T07:21:07.710 INFO:tasks.workunit.client.1.smithi184.stdout:checking for main in -lpthread... yes 2024-09-19T07:21:07.902 INFO:tasks.workunit.client.1.smithi184.stdout:checking for ANSI C header files... yes 2024-09-19T07:21:07.966 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-19T07:21:08.022 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/types.h... yes 2024-09-19T07:21:08.084 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/stat.h... yes 2024-09-19T07:21:08.147 INFO:tasks.workunit.client.1.smithi184.stdout:checking for stdlib.h... yes 2024-09-19T07:21:08.210 INFO:tasks.workunit.client.1.smithi184.stdout:checking for string.h... yes 2024-09-19T07:21:08.282 INFO:tasks.workunit.client.1.smithi184.stdout:checking for memory.h... yes 2024-09-19T07:21:08.348 INFO:tasks.workunit.client.1.smithi184.stdout:checking for strings.h... yes 2024-09-19T07:21:08.413 INFO:tasks.workunit.client.1.smithi184.stdout:checking for inttypes.h... yes 2024-09-19T07:21:08.476 INFO:tasks.workunit.client.1.smithi184.stdout:checking for stdint.h... yes 2024-09-19T07:21:08.553 INFO:tasks.workunit.client.1.smithi184.stdout:checking for unistd.h... yes 2024-09-19T07:21:08.625 INFO:tasks.workunit.client.1.smithi184.stdout:checking pthread.h usability... yes 2024-09-19T07:21:08.651 INFO:tasks.workunit.client.1.smithi184.stdout:checking pthread.h presence... yes 2024-09-19T07:21:08.652 INFO:tasks.workunit.client.1.smithi184.stdout:checking for pthread.h... yes 2024-09-19T07:21:08.728 INFO:tasks.workunit.client.1.smithi184.stdout:checking fcntl.h usability... yes 2024-09-19T07:21:08.753 INFO:tasks.workunit.client.1.smithi184.stdout:checking fcntl.h presence... yes 2024-09-19T07:21:08.754 INFO:tasks.workunit.client.1.smithi184.stdout:checking for fcntl.h... yes 2024-09-19T07:21:08.825 INFO:tasks.workunit.client.1.smithi184.stdout:checking limits.h usability... yes 2024-09-19T07:21:08.849 INFO:tasks.workunit.client.1.smithi184.stdout:checking limits.h presence... yes 2024-09-19T07:21:08.849 INFO:tasks.workunit.client.1.smithi184.stdout:checking for limits.h... yes 2024-09-19T07:21:08.878 INFO:tasks.workunit.client.1.smithi184.stdout:checking for stdint.h... (cached) yes 2024-09-19T07:21:08.936 INFO:tasks.workunit.client.1.smithi184.stdout:checking sys/time.h usability... yes 2024-09-19T07:21:08.963 INFO:tasks.workunit.client.1.smithi184.stdout:checking sys/time.h presence... yes 2024-09-19T07:21:08.964 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/time.h... yes 2024-09-19T07:21:08.988 INFO:tasks.workunit.client.1.smithi184.stdout:checking for unistd.h... (cached) yes 2024-09-19T07:21:09.045 INFO:tasks.workunit.client.1.smithi184.stdout:checking sys/vfs.h usability... yes 2024-09-19T07:21:09.069 INFO:tasks.workunit.client.1.smithi184.stdout:checking sys/vfs.h presence... yes 2024-09-19T07:21:09.069 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/vfs.h... yes 2024-09-19T07:21:09.237 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-09-19T07:21:09.237 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-19T07:21:09.237 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-19T07:21:09.240 INFO:tasks.ceph.mds.b:Started 2024-09-19T07:21:09.240 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:21:09.300 INFO:tasks.workunit.client.1.smithi184.stdout:checking sys/limits.h usability... no 2024-09-19T07:21:09.314 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-19T07:21:09.332 INFO:tasks.workunit.client.1.smithi184.stdout:checking sys/limits.h presence... no 2024-09-19T07:21:09.333 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/limits.h... no 2024-09-19T07:21:09.346 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:21:09.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe3286fe640 1 Processor -- start 2024-09-19T07:21:09.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe3286fe640 1 -- start start 2024-09-19T07:21:09.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe3286fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe3200c6360 0x7fe320153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe3286fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe3200bee30 0x7fe3201570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe3286fe640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3200c5b90 con 0x7fe3200c9680 2024-09-19T07:21:09.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe3286fe640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe32015a050 con 0x7fe3200bee30 2024-09-19T07:21:09.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe3286fe640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe32006c950 con 0x7fe3200c6360 2024-09-19T07:21:09.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe325c99640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe3200bee30 0x7fe3201570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe325c99640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe3200bee30 0x7fe3201570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50238/0 (socket says 172.21.15.83:50238) 2024-09-19T07:21:09.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe325c99640 1 -- 172.21.15.83:0/2725610625 learned_addr learned my addr 172.21.15.83:0/2725610625 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:09.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe32649a640 1 --2- 172.21.15.83:0/2725610625 >> v2:172.21.15.184:3300/0 conn(0x7fe3200c6360 0x7fe320153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe325c99640 1 -- 172.21.15.83:0/2725610625 >> v2:172.21.15.184:3300/0 conn(0x7fe3200c6360 msgr2=0x7fe320153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:09.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe325c99640 1 --2- 172.21.15.83:0/2725610625 >> v2:172.21.15.184:3300/0 conn(0x7fe3200c6360 0x7fe320153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe325498640 1 -- 172.21.15.83:0/2725610625 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1035751487 0 0) 0x7fe3200c5b90 con 0x7fe3200c9680 2024-09-19T07:21:09.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe325c99640 1 -- 172.21.15.83:0/2725610625 >> v1:172.21.15.184:6789/0 conn(0x7fe3200c9680 legacy=0x7fe320157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.368+0000 7fe325c99640 1 -- 172.21.15.83:0/2725610625 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32006c670 con 0x7fe3200bee30 2024-09-19T07:21:09.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe325c99640 1 --2- 172.21.15.83:0/2725610625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3200bee30 0x7fe3201570f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe3200ae3c0 tx=0x7fe31c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94391884100f477 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:09.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe325498640 1 -- 172.21.15.83:0/2725610625 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c0b0070 con 0x7fe3200bee30 2024-09-19T07:21:09.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe325498640 1 -- 172.21.15.83:0/2725610625 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe31c0ab600 con 0x7fe3200bee30 2024-09-19T07:21:09.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe325498640 1 -- 172.21.15.83:0/2725610625 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c0ab9d0 con 0x7fe3200bee30 2024-09-19T07:21:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe3286fe640 1 -- 172.21.15.83:0/2725610625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3200bee30 msgr2=0x7fe3201570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe3286fe640 1 --2- 172.21.15.83:0/2725610625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3200bee30 0x7fe3201570f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe3200ae3c0 tx=0x7fe31c01c400 comp rx=0 tx=0).stop 2024-09-19T07:21:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe3286fe640 1 -- 172.21.15.83:0/2725610625 shutdown_connections 2024-09-19T07:21:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe3286fe640 1 --2- 172.21.15.83:0/2725610625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3200bee30 0x7fe3201570f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe3286fe640 1 --2- 172.21.15.83:0/2725610625 >> v2:172.21.15.184:3300/0 conn(0x7fe3200c6360 0x7fe320153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe3286fe640 1 -- 172.21.15.83:0/2725610625 >> 172.21.15.83:0/2725610625 conn(0x7fe3200722a0 msgr2=0x7fe3200726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe3286fe640 1 -- 172.21.15.83:0/2725610625 shutdown_connections 2024-09-19T07:21:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe3286fe640 1 -- 172.21.15.83:0/2725610625 wait complete. 2024-09-19T07:21:09.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe3286fe640 1 Processor -- start 2024-09-19T07:21:09.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.372+0000 7fe3286fe640 1 -- start start 2024-09-19T07:21:09.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe3286fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3200bee30 0x7fe320156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe3286fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3200c6360 0x7fe32014c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe3286fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3200c9680 0x7fe32014ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe3286fe640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe3200c68c0 con 0x7fe3200c6360 2024-09-19T07:21:09.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe3286fe640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe3200c6740 con 0x7fe3200bee30 2024-09-19T07:21:09.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe3286fe640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe320162da0 con 0x7fe3200c9680 2024-09-19T07:21:09.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe325c99640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3200c6360 0x7fe32014c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe325c99640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3200c6360 0x7fe32014c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50250/0 (socket says 172.21.15.83:50250) 2024-09-19T07:21:09.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe325c99640 1 -- 172.21.15.83:0/2560001580 learned_addr learned my addr 172.21.15.83:0/2560001580 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:09.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe32649a640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3200bee30 0x7fe320156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe326c9b640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3200c9680 0x7fe32014ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe325c99640 1 -- 172.21.15.83:0/2560001580 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3200c9680 msgr2=0x7fe32014ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe325c99640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3200c9680 0x7fe32014ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe325c99640 1 -- 172.21.15.83:0/2560001580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3200bee30 msgr2=0x7fe320156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe325c99640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3200bee30 0x7fe320156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe325c99640 1 -- 172.21.15.83:0/2560001580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3201f2330 con 0x7fe3200c6360 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe326c9b640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3200c9680 0x7fe32014ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe325c99640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3200c6360 0x7fe32014c910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe31c01a030 tx=0x7fe31c01ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe316ffd640 1 -- 172.21.15.83:0/2560001580 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c0ba070 con 0x7fe3200c6360 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe316ffd640 1 -- 172.21.15.83:0/2560001580 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe31c004020 con 0x7fe3200c6360 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe316ffd640 1 -- 172.21.15.83:0/2560001580 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c0b0040 con 0x7fe3200c6360 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe3286fe640 1 -- 172.21.15.83:0/2560001580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3201f25f0 con 0x7fe3200c6360 2024-09-19T07:21:09.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe3286fe640 1 -- 172.21.15.83:0/2560001580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3200c3d20 con 0x7fe3200c6360 2024-09-19T07:21:09.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe3286fe640 1 -- 172.21.15.83:0/2560001580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe320153050 con 0x7fe3200c6360 2024-09-19T07:21:09.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe316ffd640 1 -- 172.21.15.83:0/2560001580 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe31c003620 con 0x7fe3200c6360 2024-09-19T07:21:09.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe316ffd640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe3040678c0 0x7fe304069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe316ffd640 1 -- 172.21.15.83:0/2560001580 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7fe31c1200a0 con 0x7fe3200c6360 2024-09-19T07:21:09.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe32649a640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe3040678c0 0x7fe304069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.376+0000 7fe32649a640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe3040678c0 0x7fe304069d80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe3200cbf60 tx=0x7fe310009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:09.385 INFO:tasks.workunit.client.1.smithi184.stdout:checking for an ANSI C-conforming const... yes 2024-09-19T07:21:09.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.384+0000 7fe316ffd640 1 -- 172.21.15.83:0/2560001580 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe31c016110 con 0x7fe3200c6360 2024-09-19T07:21:09.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.476+0000 7f715cfcd640 1 Processor -- start 2024-09-19T07:21:09.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.476+0000 7f715cfcd640 1 -- start start 2024-09-19T07:21:09.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f715cfcd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f71580c4a20 0x7f71580715f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f715cfcd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f715806f680 0x7f715806fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f715cfcd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71580c3ee0 con 0x7f71580c7750 2024-09-19T07:21:09.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f715cfcd640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f7158070050 con 0x7f71580c4a20 2024-09-19T07:21:09.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f715cfcd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f71580c41c0 con 0x7f715806f680 2024-09-19T07:21:09.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f714ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f71580c7750 0x7f71580c2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33524/0 (socket says 172.21.15.83:33524) 2024-09-19T07:21:09.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f714ffff640 1 -- 172.21.15.83:0/379409071 learned_addr learned my addr 172.21.15.83:0/379409071 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:21:09.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f7156d9d640 1 --2- 172.21.15.83:0/379409071 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f715806f680 0x7f715806fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f715659c640 1 --2- 172.21.15.83:0/379409071 >> v2:172.21.15.184:3300/0 conn(0x7f71580c4a20 0x7f71580715f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f7155d9b640 1 -- 172.21.15.83:0/379409071 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2174862935 0 0) 0x7f71580c3ee0 con 0x7f71580c7750 2024-09-19T07:21:09.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f7155d9b640 1 -- 172.21.15.83:0/379409071 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7144004640 con 0x7f71580c7750 2024-09-19T07:21:09.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f7156d9d640 1 -- 172.21.15.83:0/379409071 >> v2:172.21.15.184:3300/0 conn(0x7f71580c4a20 msgr2=0x7f71580715f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:09.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f7156d9d640 1 --2- 172.21.15.83:0/379409071 >> v2:172.21.15.184:3300/0 conn(0x7f71580c4a20 0x7f71580715f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f7156d9d640 1 -- 172.21.15.83:0/379409071 >> v1:172.21.15.83:6789/0 conn(0x7f71580c7750 legacy=0x7f71580c2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f7156d9d640 1 -- 172.21.15.83:0/379409071 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71580c44a0 con 0x7f715806f680 2024-09-19T07:21:09.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f7156d9d640 1 --2- 172.21.15.83:0/379409071 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f715806f680 0x7f715806fb10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7150009d90 tx=0x7f715001c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcb35d01547375af server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:09.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f7155d9b640 1 -- 172.21.15.83:0/379409071 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71500b2070 con 0x7f715806f680 2024-09-19T07:21:09.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f7155d9b640 1 -- 172.21.15.83:0/379409071 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71500a5830 con 0x7f715806f680 2024-09-19T07:21:09.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.480+0000 7f7155d9b640 1 -- 172.21.15.83:0/379409071 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71500a5c00 con 0x7f715806f680 2024-09-19T07:21:09.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 -- 172.21.15.83:0/379409071 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f715806f680 msgr2=0x7f715806fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 --2- 172.21.15.83:0/379409071 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f715806f680 0x7f715806fb10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7150009d90 tx=0x7f715001c8e0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 -- 172.21.15.83:0/379409071 shutdown_connections 2024-09-19T07:21:09.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 --2- 172.21.15.83:0/379409071 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f715806f680 0x7f715806fb10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 --2- 172.21.15.83:0/379409071 >> v2:172.21.15.184:3300/0 conn(0x7f71580c4a20 0x7f71580715f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 -- 172.21.15.83:0/379409071 >> 172.21.15.83:0/379409071 conn(0x7f71580c7e00 msgr2=0x7f71580c8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:09.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 -- 172.21.15.83:0/379409071 shutdown_connections 2024-09-19T07:21:09.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 -- 172.21.15.83:0/379409071 wait complete. 2024-09-19T07:21:09.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 Processor -- start 2024-09-19T07:21:09.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 -- start start 2024-09-19T07:21:09.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f715806f680 0x7f71580724b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71580c4a20 0x7f71580c1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71580bf460 0x7f71580bf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f71580c3650 con 0x7f71580c4a20 2024-09-19T07:21:09.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f71580a7470 con 0x7f715806f680 2024-09-19T07:21:09.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715cfcd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f71580c37d0 con 0x7f71580bf460 2024-09-19T07:21:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f714ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71580c4a20 0x7f71580c1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f714ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71580c4a20 0x7f71580c1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50266/0 (socket says 172.21.15.83:50266) 2024-09-19T07:21:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f714ffff640 1 -- 172.21.15.83:0/150017361 learned_addr learned my addr 172.21.15.83:0/150017361 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715659c640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f715806f680 0x7f71580724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f7156d9d640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71580bf460 0x7f71580bf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f714ffff640 1 -- 172.21.15.83:0/150017361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71580bf460 msgr2=0x7f71580bf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f714ffff640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71580bf460 0x7f71580bf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f714ffff640 1 -- 172.21.15.83:0/150017361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f715806f680 msgr2=0x7f71580724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f714ffff640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f715806f680 0x7f71580724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f714ffff640 1 -- 172.21.15.83:0/150017361 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71581ee0e0 con 0x7f71580c4a20 2024-09-19T07:21:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f715659c640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f715806f680 0x7f71580724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:21:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f714ffff640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71580c4a20 0x7f71580c1340 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f714000c420 tx=0x7f714000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f714effd640 1 -- 172.21.15.83:0/150017361 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7140017070 con 0x7f71580c4a20 2024-09-19T07:21:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.484+0000 7f714effd640 1 -- 172.21.15.83:0/150017361 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7140005d10 con 0x7f71580c4a20 2024-09-19T07:21:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.488+0000 7f715cfcd640 1 -- 172.21.15.83:0/150017361 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71580bfe50 con 0x7f71580c4a20 2024-09-19T07:21:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.488+0000 7f714effd640 1 -- 172.21.15.83:0/150017361 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7140007310 con 0x7f71580c4a20 2024-09-19T07:21:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.488+0000 7f715cfcd640 1 -- 172.21.15.83:0/150017361 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71581ee380 con 0x7f71580c4a20 2024-09-19T07:21:09.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.488+0000 7f715cfcd640 1 -- 172.21.15.83:0/150017361 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71580c4e00 con 0x7f71580c4a20 2024-09-19T07:21:09.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.488+0000 7f714effd640 1 -- 172.21.15.83:0/150017361 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f714001a070 con 0x7f71580c4a20 2024-09-19T07:21:09.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.488+0000 7f714effd640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71380678c0 0x7f7138069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.488+0000 7f714effd640 1 -- 172.21.15.83:0/150017361 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7f7140085880 con 0x7f71580c4a20 2024-09-19T07:21:09.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.488+0000 7f715659c640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71380678c0 0x7f7138069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.488+0000 7f715659c640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71380678c0 0x7f7138069d80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7148009f50 tx=0x7f7148009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.492+0000 7f714effd640 1 -- 172.21.15.83:0/150017361 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7140021020 con 0x7f71580c4a20 2024-09-19T07:21:09.509 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-19T07:21:09.511 INFO:tasks.workunit.client.1.smithi184.stdout:checking for function prototypes... yes 2024-09-19T07:21:09.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.604+0000 7fe3286fe640 1 -- 172.21.15.83:0/2560001580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe320072a70 con 0x7fe3200c6360 2024-09-19T07:21:09.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe316ffd640 1 -- 172.21.15.83:0/2560001580 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+4883 (secure 0 0 0) 0x7fe31c0f07a0 con 0x7fe3200c6360 2024-09-19T07:21:09.610 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:21:09.611 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"btime":"2024-09-19T07:21:08:933579+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:07.955083+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:21:09.611 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 20 2024-09-19T07:21:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe3286fe640 1 -- 172.21.15.83:0/2560001580 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe3040678c0 msgr2=0x7fe304069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe3286fe640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe3040678c0 0x7fe304069d80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe3200cbf60 tx=0x7fe310009190 comp rx=0 tx=0).stop 2024-09-19T07:21:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe3286fe640 1 -- 172.21.15.83:0/2560001580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3200c6360 msgr2=0x7fe32014c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe3286fe640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3200c6360 0x7fe32014c910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe31c01a030 tx=0x7fe31c01ac40 comp rx=0 tx=0).stop 2024-09-19T07:21:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe3286fe640 1 -- 172.21.15.83:0/2560001580 shutdown_connections 2024-09-19T07:21:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe3286fe640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe3040678c0 0x7fe304069d80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe3286fe640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3200c9680 0x7fe32014ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe3286fe640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3200c6360 0x7fe32014c910 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe3286fe640 1 --2- 172.21.15.83:0/2560001580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3200bee30 0x7fe320156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe3286fe640 1 -- 172.21.15.83:0/2560001580 >> 172.21.15.83:0/2560001580 conn(0x7fe3200722a0 msgr2=0x7fe320156ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:09.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe3286fe640 1 -- 172.21.15.83:0/2560001580 shutdown_connections 2024-09-19T07:21:09.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.608+0000 7fe3286fe640 1 -- 172.21.15.83:0/2560001580 wait complete. 2024-09-19T07:21:09.629 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-19T07:21:09.660 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether setvbuf arguments are reversed... no 2024-09-19T07:21:09.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.692+0000 7f715cfcd640 1 -- 172.21.15.83:0/150017361 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f715806cc90 con 0x7f71580c4a20 2024-09-19T07:21:09.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f714effd640 1 -- 172.21.15.83:0/150017361 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+4883 (secure 0 0 0) 0x7f7140055a40 con 0x7f71580c4a20 2024-09-19T07:21:09.699 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:21:09.699 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"btime":"2024-09-19T07:21:08:933579+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:07.955083+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:21:09.699 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 20 2024-09-19T07:21:09.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f715cfcd640 1 -- 172.21.15.83:0/150017361 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71380678c0 msgr2=0x7f7138069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f715cfcd640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71380678c0 0x7f7138069d80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7148009f50 tx=0x7f7148009190 comp rx=0 tx=0).stop 2024-09-19T07:21:09.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f715cfcd640 1 -- 172.21.15.83:0/150017361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71580c4a20 msgr2=0x7f71580c1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f715cfcd640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71580c4a20 0x7f71580c1340 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f714000c420 tx=0x7f714000c8e0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f715cfcd640 1 -- 172.21.15.83:0/150017361 shutdown_connections 2024-09-19T07:21:09.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f715cfcd640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71380678c0 0x7f7138069d80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f715cfcd640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71580bf460 0x7f71580bf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f715cfcd640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71580c4a20 0x7f71580c1340 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f715cfcd640 1 --2- 172.21.15.83:0/150017361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f715806f680 0x7f71580724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f715cfcd640 1 -- 172.21.15.83:0/150017361 >> 172.21.15.83:0/150017361 conn(0x7f71580c7e00 msgr2=0x7f71580c6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:09.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f715cfcd640 1 -- 172.21.15.83:0/150017361 shutdown_connections 2024-09-19T07:21:09.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.696+0000 7f715cfcd640 1 -- 172.21.15.83:0/150017361 wait complete. 2024-09-19T07:21:09.718 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-19T07:21:09.718 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:21:09.718 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:21:09.841 INFO:tasks.workunit.client.1.smithi184.stdout:checking for system... yes 2024-09-19T07:21:09.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f041671d640 1 Processor -- start 2024-09-19T07:21:09.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f041671d640 1 -- start start 2024-09-19T07:21:09.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f041671d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041006f950 0x7f04100716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f041671d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f04101515e0 0x7f04101575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f041671d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04100c37c0 con 0x7f0410071e20 2024-09-19T07:21:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f041671d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f04100c4670 con 0x7f04101515e0 2024-09-19T07:21:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f041671d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f04100c3a60 con 0x7f041006f950 2024-09-19T07:21:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0410071e20 0x7f0410150ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33528/0 (socket says 172.21.15.83:33528) 2024-09-19T07:21:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040f7fe640 1 -- 172.21.15.83:0/3009574460 learned_addr learned my addr 172.21.15.83:0/3009574460 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:21:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f0414cba640 1 --2- 172.21.15.83:0/3009574460 >> v2:172.21.15.184:3300/0 conn(0x7f04101515e0 0x7f04101575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040ffff640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041006f950 0x7f04100716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 691845867 0 0) 0x7f04100c37c0 con 0x7f0410071e20 2024-09-19T07:21:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03f8003530 con 0x7f0410071e20 2024-09-19T07:21:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 240942219 0 0) 0x7f03f8003530 con 0x7f0410071e20 2024-09-19T07:21:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041006f950 msgr2=0x7f04100716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041006f950 0x7f04100716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 >> v2:172.21.15.184:3300/0 conn(0x7f04101515e0 msgr2=0x7f04101575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 --2- 172.21.15.83:0/3009574460 >> v2:172.21.15.184:3300/0 conn(0x7f04101515e0 0x7f04101575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04100c3d40 con 0x7f0410071e20 2024-09-19T07:21:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f0404001a20 con 0x7f0410071e20 2024-09-19T07:21:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 >> v1:172.21.15.83:6789/0 conn(0x7f0410071e20 legacy=0x7f0410150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03f80039c0 0x7f03f8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03f8004810 0x7f03f8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041006f950 0x7f03f80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f04100c4670 con 0x7f03f80039c0 2024-09-19T07:21:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f04100c3a60 con 0x7f03f8004810 2024-09-19T07:21:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f03f800b9d0 con 0x7f041006f950 2024-09-19T07:21:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040ffff640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03f80039c0 0x7f03f8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040f7fe640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03f8004810 0x7f03f8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f0414cba640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041006f950 0x7f03f80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040ffff640 1 -- 172.21.15.83:0/3009574460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041006f950 msgr2=0x7f03f80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040ffff640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041006f950 0x7f03f80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040ffff640 1 -- 172.21.15.83:0/3009574460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03f8004810 msgr2=0x7f03f8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040ffff640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03f8004810 0x7f03f8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f040ffff640 1 -- 172.21.15.83:0/3009574460 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f8003530 con 0x7f03f80039c0 2024-09-19T07:21:09.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.840+0000 7f0414cba640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041006f950 0x7f03f80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:21:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f040ffff640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03f80039c0 0x7f03f8003de0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0400009b10 tx=0x7f040001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f65945e0e4baddd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04000096b0 con 0x7f03f80039c0 2024-09-19T07:21:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04000a9470 con 0x7f03f80039c0 2024-09-19T07:21:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f040effd640 1 -- 172.21.15.83:0/3009574460 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04000a9860 con 0x7f03f80039c0 2024-09-19T07:21:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 -- 172.21.15.83:0/3009574460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03f80039c0 msgr2=0x7f03f8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 --2- 172.21.15.83:0/3009574460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03f80039c0 0x7f03f8003de0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0400009b10 tx=0x7f040001ca70 comp rx=0 tx=0).stop 2024-09-19T07:21:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f0414cba640 1 -- 172.21.15.83:0/3009574460 reap_dead start 2024-09-19T07:21:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 -- 172.21.15.83:0/3009574460 shutdown_connections 2024-09-19T07:21:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 -- 172.21.15.83:0/3009574460 >> 172.21.15.83:0/3009574460 conn(0x7f04100c48b0 msgr2=0x7f04100c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 -- 172.21.15.83:0/3009574460 shutdown_connections 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 -- 172.21.15.83:0/3009574460 wait complete. 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 Processor -- start 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 -- start start 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03f80039c0 0x7f04101437f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0410142810 0x7f0410142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f041014fc10 0x7f0410150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f041015e680 con 0x7f041014fc10 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f041006ff80 con 0x7f03f80039c0 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f041671d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f041015e800 con 0x7f0410142810 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f0414cba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f041014fc10 0x7f0410150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f0414cba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f041014fc10 0x7f0410150090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50284/0 (socket says 172.21.15.83:50284) 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f0414cba640 1 -- 172.21.15.83:0/2066287124 learned_addr learned my addr 172.21.15.83:0/2066287124 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f040f7fe640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0410142810 0x7f0410142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.844+0000 7f040ffff640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03f80039c0 0x7f04101437f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f0414cba640 1 -- 172.21.15.83:0/2066287124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0410142810 msgr2=0x7f0410142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f0414cba640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0410142810 0x7f0410142c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f0414cba640 1 -- 172.21.15.83:0/2066287124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03f80039c0 msgr2=0x7f04101437f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:09.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f0414cba640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03f80039c0 0x7f04101437f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:09.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f0414cba640 1 -- 172.21.15.83:0/2066287124 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04100c6bb0 con 0x7f041014fc10 2024-09-19T07:21:09.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f0414cba640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f041014fc10 0x7f0410150090 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0408011490 tx=0x7f0408011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:09.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f040cff9640 1 -- 172.21.15.83:0/2066287124 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0408004230 con 0x7f041014fc10 2024-09-19T07:21:09.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f040cff9640 1 -- 172.21.15.83:0/2066287124 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04080043f0 con 0x7f041014fc10 2024-09-19T07:21:09.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f041671d640 1 -- 172.21.15.83:0/2066287124 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04101435a0 con 0x7f041014fc10 2024-09-19T07:21:09.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f040cff9640 1 -- 172.21.15.83:0/2066287124 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0408015970 con 0x7f041014fc10 2024-09-19T07:21:09.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f041671d640 1 -- 172.21.15.83:0/2066287124 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04100c6e70 con 0x7f041014fc10 2024-09-19T07:21:09.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f041671d640 1 -- 172.21.15.83:0/2066287124 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04101575b0 con 0x7f041014fc10 2024-09-19T07:21:09.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f040cff9640 1 -- 172.21.15.83:0/2066287124 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0408015b10 con 0x7f041014fc10 2024-09-19T07:21:09.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f040cff9640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f03fc0678f0 0x7f03fc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:09.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f040cff9640 1 -- 172.21.15.83:0/2066287124 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7f0408086230 con 0x7f041014fc10 2024-09-19T07:21:09.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f040ffff640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f03fc0678f0 0x7f03fc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:09.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.848+0000 7f040ffff640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f03fc0678f0 0x7f03fc069db0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f04000ae000 tx=0x7f04000b3040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:09.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:09.852+0000 7f040cff9640 1 -- 172.21.15.83:0/2066287124 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f04100c6bb0 con 0x7f041014fc10 2024-09-19T07:21:10.021 INFO:tasks.workunit.client.1.smithi184.stdout:checking for gettimeofday... yes 2024-09-19T07:21:10.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.068+0000 7f041671d640 1 -- 172.21.15.83:0/2066287124 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04100c2720 con 0x7f041014fc10 2024-09-19T07:21:10.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.068+0000 7f040cff9640 1 -- 172.21.15.83:0/2066287124 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5707 (secure 0 0 0) 0x7f0408019020 con 0x7f041014fc10 2024-09-19T07:21:10.072 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:21:10.073 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":21,"btime":"2024-09-19T07:21:09:935339+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:08.940896+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:21:10.073 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 21 2024-09-19T07:21:10.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.072+0000 7f041671d640 1 -- 172.21.15.83:0/2066287124 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f03fc0678f0 msgr2=0x7f03fc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:10.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.072+0000 7f041671d640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f03fc0678f0 0x7f03fc069db0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f04000ae000 tx=0x7f04000b3040 comp rx=0 tx=0).stop 2024-09-19T07:21:10.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.072+0000 7f041671d640 1 -- 172.21.15.83:0/2066287124 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f041014fc10 msgr2=0x7f0410150090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:10.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.072+0000 7f041671d640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f041014fc10 0x7f0410150090 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0408011490 tx=0x7f0408011890 comp rx=0 tx=0).stop 2024-09-19T07:21:10.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.072+0000 7f041671d640 1 -- 172.21.15.83:0/2066287124 shutdown_connections 2024-09-19T07:21:10.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.072+0000 7f041671d640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f03fc0678f0 0x7f03fc069db0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:10.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.072+0000 7f041671d640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f041014fc10 0x7f0410150090 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:10.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.072+0000 7f041671d640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0410142810 0x7f0410142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:10.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.072+0000 7f041671d640 1 --2- 172.21.15.83:0/2066287124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03f80039c0 0x7f04101437f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:10.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.072+0000 7f041671d640 1 -- 172.21.15.83:0/2066287124 >> 172.21.15.83:0/2066287124 conn(0x7f04100c48b0 msgr2=0x7f04100c2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:10.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.072+0000 7f041671d640 1 -- 172.21.15.83:0/2066287124 shutdown_connections 2024-09-19T07:21:10.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:10.072+0000 7f041671d640 1 -- 172.21.15.83:0/2066287124 wait complete. 2024-09-19T07:21:10.093 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:21:09:935339+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4220": { "addr": "172.21.15.83:6837/4265456507", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 4265456507, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 4265456507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 19, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 28 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:21:08.940896+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4220 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 18, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4229": { "addr": "172.21.15.184:6833/1773885233", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 1773885233, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 1773885233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 15, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:21:00.932928+0000", "qdb_cluster": [ 4229 ], "qdb_leader": 4229, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4229 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/4115191178", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 4115191178, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 4115191178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 21, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4786, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:21:10.190 INFO:tasks.workunit.client.1.smithi184.stdout:checking for mkdir... yes 2024-09-19T07:21:10.292 INFO:tasks.workunit.client.1.smithi184.stdout:checking for strerror... yes 2024-09-19T07:21:10.391 INFO:tasks.workunit.client.1.smithi184.stdout:checking for open64... yes 2024-09-19T07:21:10.485 INFO:tasks.workunit.client.1.smithi184.stdout:checking for stat64... yes 2024-09-19T07:21:10.577 INFO:tasks.workunit.client.1.smithi184.stdout:checking for fseeko64... yes 2024-09-19T07:21:10.669 INFO:tasks.workunit.client.1.smithi184.stdout:checking for lrand48_r... yes 2024-09-19T07:21:10.766 INFO:tasks.workunit.client.1.smithi184.stdout:checking for srand48_r... yes 2024-09-19T07:21:10.813 INFO:tasks.workunit.client.1.smithi184.stdout:configure: creating ./config.status 2024-09-19T07:21:10.944 INFO:tasks.workunit.client.1.smithi184.stdout:config.status: creating Makefile 2024-09-19T07:21:10.966 INFO:tasks.workunit.client.0.smithi083.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-09-19T07:21:10.967 INFO:tasks.workunit.client.0.smithi083.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-09-19T07:21:10.967 INFO:tasks.workunit.client.0.smithi083.stderr:+ trap -- retry EXIT 2024-09-19T07:21:10.967 INFO:tasks.workunit.client.0.smithi083.stderr:+ rm -rf ffsb 2024-09-19T07:21:10.968 INFO:tasks.workunit.client.0.smithi083.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-09-19T07:21:10.971 INFO:tasks.workunit.client.1.smithi184.stdout:config.status: creating config.h 2024-09-19T07:21:10.972 INFO:tasks.workunit.client.0.smithi083.stderr:Cloning into 'ffsb'... 2024-09-19T07:21:10.996 INFO:tasks.workunit.client.1.smithi184.stdout:config.status: executing depfiles commands 2024-09-19T07:21:11.066 INFO:tasks.workunit.client.1.smithi184.stdout:config.status: executing default commands 2024-09-19T07:21:11.132 INFO:tasks.workunit.client.1.smithi184.stderr:+ make 2024-09-19T07:21:11.162 INFO:tasks.workunit.client.1.smithi184.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run aclocal-1.10 2024-09-19T07:21:11.168 INFO:tasks.workunit.client.1.smithi184.stderr:/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-09-19T07:21:11.169 INFO:tasks.workunit.client.1.smithi184.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-09-19T07:21:11.170 INFO:tasks.workunit.client.1.smithi184.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-09-19T07:21:11.170 INFO:tasks.workunit.client.1.smithi184.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-09-19T07:21:11.170 INFO:tasks.workunit.client.1.smithi184.stderr: any GNU archive site. 2024-09-19T07:21:11.173 INFO:tasks.workunit.client.1.smithi184.stdout: cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run automake-1.10 --gnu 2024-09-19T07:21:11.177 INFO:tasks.workunit.client.1.smithi184.stderr:/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-09-19T07:21:11.178 INFO:tasks.workunit.client.1.smithi184.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-09-19T07:21:11.178 INFO:tasks.workunit.client.1.smithi184.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-09-19T07:21:11.179 INFO:tasks.workunit.client.1.smithi184.stderr: You might want to install the `Automake' and `Perl' packages. 2024-09-19T07:21:11.179 INFO:tasks.workunit.client.1.smithi184.stderr: Grab them from any GNU archive site. 2024-09-19T07:21:11.371 INFO:tasks.workunit.client.1.smithi184.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run autoconf 2024-09-19T07:21:11.510 INFO:tasks.workunit.client.1.smithi184.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-19T07:21:11.510 INFO:tasks.workunit.client.1.smithi184.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-19T07:21:11.511 INFO:tasks.workunit.client.1.smithi184.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-19T07:21:11.511 INFO:tasks.workunit.client.1.smithi184.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-19T07:21:11.607 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:4: warning: The macro `AC_FOREACH' is obsolete. 2024-09-19T07:21:11.607 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:4: You should run autoupdate. 2024-09-19T07:21:11.607 INFO:tasks.workunit.client.1.smithi184.stderr:./lib/autoconf/general.m4:191: AC_FOREACH is expanded from... 2024-09-19T07:21:11.607 INFO:tasks.workunit.client.1.smithi184.stderr:aclocal.m4:684: _AM_SET_OPTIONS is expanded from... 2024-09-19T07:21:11.607 INFO:tasks.workunit.client.1.smithi184.stderr:aclocal.m4:410: AM_INIT_AUTOMAKE is expanded from... 2024-09-19T07:21:11.607 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:4: the top level 2024-09-19T07:21:11.607 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:6: warning: The macro `AM_CONFIG_HEADER' is obsolete. 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:6: You should run autoupdate. 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:aclocal.m4:383: AM_CONFIG_HEADER is expanded from... 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:6: the top level 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:10: warning: The macro `AC_AIX' is obsolete. 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:10: You should run autoupdate. 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:./lib/autoconf/specific.m4:540: AC_AIX is expanded from... 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:10: the top level 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:36: warning: The macro `AC_HEADER_STDC' is obsolete. 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:36: You should run autoupdate. 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from... 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:36: the top level 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:42: warning: The macro `AC_HEADER_TIME' is obsolete. 2024-09-19T07:21:11.608 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:42: You should run autoupdate. 2024-09-19T07:21:11.609 INFO:tasks.workunit.client.1.smithi184.stderr:./lib/autoconf/headers.m4:743: AC_HEADER_TIME is expanded from... 2024-09-19T07:21:11.609 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:42: the top level 2024-09-19T07:21:11.609 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:45: warning: The macro `AC_FUNC_SETVBUF_REVERSED' is obsolete. Remove it and all references to SETVBUF_REVERSED. 2024-09-19T07:21:11.609 INFO:tasks.workunit.client.1.smithi184.stderr:./lib/autoconf/functions.m4:1766: AC_FUNC_SETVBUF_REVERSED is expanded from... 2024-09-19T07:21:11.609 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:45: the top level 2024-09-19T07:21:11.609 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:50: warning: AC_OUTPUT should be used without arguments. 2024-09-19T07:21:11.609 INFO:tasks.workunit.client.1.smithi184.stderr:configure.in:50: You should run autoupdate. 2024-09-19T07:21:11.631 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:21:11.674 INFO:tasks.workunit.client.1.smithi184.stdout:/bin/bash ./config.status --recheck 2024-09-19T07:21:11.733 INFO:tasks.workunit.client.1.smithi184.stdout:running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion 2024-09-19T07:21:11.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdec1535640 1 Processor -- start 2024-09-19T07:21:11.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdec1535640 1 -- start start 2024-09-19T07:21:11.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdec1535640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdebc0ac290 0x7fdebc0c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:11.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdec1535640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdebc06e110 con 0x7fdebc0c3a50 2024-09-19T07:21:11.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdec1535640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdebc0c4990 con 0x7fdebc0c3130 2024-09-19T07:21:11.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdec1535640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fdebc0ac700 con 0x7fdebc0ac290 2024-09-19T07:21:11.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdeba7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdebc0ac290 0x7fdebc0c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:11.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdebb7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdebc0c3a50 0x7fdebc1563a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42262/0 (socket says 172.21.15.83:42262) 2024-09-19T07:21:11.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdebb7fe640 1 -- 172.21.15.83:0/1868642073 learned_addr learned my addr 172.21.15.83:0/1868642073 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:21:11.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdeb9ffb640 1 -- 172.21.15.83:0/1868642073 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 824242016 0 0) 0x7fdebc06e110 con 0x7fdebc0c3a50 2024-09-19T07:21:11.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdeb9ffb640 1 -- 172.21.15.83:0/1868642073 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea4003530 con 0x7fdebc0c3a50 2024-09-19T07:21:11.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdeb9ffb640 1 -- 172.21.15.83:0/1868642073 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1985909366 0 0) 0x7fdebc0c4990 con 0x7fdebc0c3130 2024-09-19T07:21:11.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdeb9ffb640 1 -- 172.21.15.83:0/1868642073 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdebc06e110 con 0x7fdebc0c3130 2024-09-19T07:21:11.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdeba7fc640 1 -- 172.21.15.83:0/1868642073 >> v1:172.21.15.184:6789/0 conn(0x7fdebc0c3130 legacy=0x7fdebc06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:11.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdeba7fc640 1 -- 172.21.15.83:0/1868642073 >> v1:172.21.15.83:6789/0 conn(0x7fdebc0c3a50 legacy=0x7fdebc1563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:11.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.744+0000 7fdeba7fc640 1 -- 172.21.15.83:0/1868642073 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc0c4c70 con 0x7fdebc0ac290 2024-09-19T07:21:11.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.748+0000 7fdeba7fc640 1 --2- 172.21.15.83:0/1868642073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdebc0ac290 0x7fdebc0c3510 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00097c0 tx=0x7fdeb001e620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20beca1f1894eed9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:11.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.748+0000 7fdeb9ffb640 1 -- 172.21.15.83:0/1868642073 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb00ad640 con 0x7fdebc0ac290 2024-09-19T07:21:11.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.748+0000 7fdeb9ffb640 1 -- 172.21.15.83:0/1868642073 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb00b2070 con 0x7fdebc0ac290 2024-09-19T07:21:11.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.748+0000 7fdeb9ffb640 1 -- 172.21.15.83:0/1868642073 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb00ad9e0 con 0x7fdebc0ac290 2024-09-19T07:21:11.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.748+0000 7fdec1535640 1 -- 172.21.15.83:0/1868642073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdebc0ac290 msgr2=0x7fdebc0c3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:11.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.748+0000 7fdec1535640 1 --2- 172.21.15.83:0/1868642073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdebc0ac290 0x7fdebc0c3510 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00097c0 tx=0x7fdeb001e620 comp rx=0 tx=0).stop 2024-09-19T07:21:11.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.748+0000 7fdec1535640 1 -- 172.21.15.83:0/1868642073 shutdown_connections 2024-09-19T07:21:11.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.748+0000 7fdec1535640 1 --2- 172.21.15.83:0/1868642073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdebc0ac290 0x7fdebc0c3510 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:11.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.748+0000 7fdec1535640 1 -- 172.21.15.83:0/1868642073 >> 172.21.15.83:0/1868642073 conn(0x7fdebc0c5060 msgr2=0x7fdebc0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:11.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.748+0000 7fdec1535640 1 -- 172.21.15.83:0/1868642073 shutdown_connections 2024-09-19T07:21:11.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.748+0000 7fdec1535640 1 -- 172.21.15.83:0/1868642073 wait complete. 2024-09-19T07:21:11.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdec1535640 1 Processor -- start 2024-09-19T07:21:11.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdec1535640 1 -- start start 2024-09-19T07:21:11.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdec1535640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdebc0ac290 0x7fdebc1538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:11.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdec1535640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdebc0c3130 0x7fdebc153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:11.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdec1535640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdebc0c3a50 0x7fdebc072bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:11.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdec1535640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc0ad520 con 0x7fdebc0c3130 2024-09-19T07:21:11.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdec1535640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc0c82a0 con 0x7fdebc0ac290 2024-09-19T07:21:11.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdec1535640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdebc0ad6a0 con 0x7fdebc0c3a50 2024-09-19T07:21:11.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdeba7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdebc0c3130 0x7fdebc153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:11.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdeba7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdebc0c3130 0x7fdebc153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32772/0 (socket says 172.21.15.83:32772) 2024-09-19T07:21:11.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdeba7fc640 1 -- 172.21.15.83:0/268755771 learned_addr learned my addr 172.21.15.83:0/268755771 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:11.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdebaffd640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdebc0ac290 0x7fdebc1538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:11.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdebb7fe640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdebc0c3a50 0x7fdebc072bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:11.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdeba7fc640 1 -- 172.21.15.83:0/268755771 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdebc0c3a50 msgr2=0x7fdebc072bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:11.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdeba7fc640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdebc0c3a50 0x7fdebc072bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:11.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdeba7fc640 1 -- 172.21.15.83:0/268755771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdebc0ac290 msgr2=0x7fdebc1538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:11.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdeba7fc640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdebc0ac290 0x7fdebc1538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:11.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdeba7fc640 1 -- 172.21.15.83:0/268755771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc155070 con 0x7fdebc0c3130 2024-09-19T07:21:11.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdeba7fc640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdebc0c3130 0x7fdebc153de0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00af040 tx=0x7fdeb001b3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:11.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdea37fe640 1 -- 172.21.15.83:0/268755771 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0003620 con 0x7fdebc0c3130 2024-09-19T07:21:11.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdea37fe640 1 -- 172.21.15.83:0/268755771 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb00037c0 con 0x7fdebc0c3130 2024-09-19T07:21:11.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdec1535640 1 -- 172.21.15.83:0/268755771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdebc14fe50 con 0x7fdebc0c3130 2024-09-19T07:21:11.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdea37fe640 1 -- 172.21.15.83:0/268755771 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0003b90 con 0x7fdebc0c3130 2024-09-19T07:21:11.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdec1535640 1 -- 172.21.15.83:0/268755771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdebc151100 con 0x7fdebc0c3130 2024-09-19T07:21:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.752+0000 7fdec1535640 1 -- 172.21.15.83:0/268755771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdebc0c3510 con 0x7fdebc0c3130 2024-09-19T07:21:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.756+0000 7fdea37fe640 1 -- 172.21.15.83:0/268755771 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdeb00c5060 con 0x7fdebc0c3130 2024-09-19T07:21:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.756+0000 7fdea37fe640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea80678c0 0x7fdea8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.756+0000 7fdea37fe640 1 -- 172.21.15.83:0/268755771 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7fdeb0121e70 con 0x7fdebc0c3130 2024-09-19T07:21:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.756+0000 7fdebaffd640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea80678c0 0x7fdea8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.756+0000 7fdebaffd640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea80678c0 0x7fdea8069d80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdeac0017c0 tx=0x7fdeac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:11.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:11.760+0000 7fdea37fe640 1 -- 172.21.15.83:0/268755771 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdeb00182a0 con 0x7fdebc0c3130 2024-09-19T07:21:11.871 INFO:tasks.workunit.client.1.smithi184.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-19T07:21:12.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 -- 172.21.15.83:0/268755771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdebc06dd00 con 0x7fdebc0c3130 2024-09-19T07:21:12.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdea37fe640 1 -- 172.21.15.83:0/268755771 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5714 (secure 0 0 0) 0x7fdeb00f1f10 con 0x7fdebc0c3130 2024-09-19T07:21:12.044 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:21:12.044 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":22,"btime":"2024-09-19T07:21:10:937450+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:21:12.044 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 22 2024-09-19T07:21:12.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 -- 172.21.15.83:0/268755771 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea80678c0 msgr2=0x7fdea8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:12.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea80678c0 0x7fdea8069d80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdeac0017c0 tx=0x7fdeac009040 comp rx=0 tx=0).stop 2024-09-19T07:21:12.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 -- 172.21.15.83:0/268755771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdebc0c3130 msgr2=0x7fdebc153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:12.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdebc0c3130 0x7fdebc153de0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00af040 tx=0x7fdeb001b3c0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 -- 172.21.15.83:0/268755771 shutdown_connections 2024-09-19T07:21:12.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea80678c0 0x7fdea8069d80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdebc0c3a50 0x7fdebc072bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdebc0c3130 0x7fdebc153de0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 --2- 172.21.15.83:0/268755771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdebc0ac290 0x7fdebc1538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 -- 172.21.15.83:0/268755771 >> 172.21.15.83:0/268755771 conn(0x7fdebc0c5060 msgr2=0x7fdebc06bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:12.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 -- 172.21.15.83:0/268755771 shutdown_connections 2024-09-19T07:21:12.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.040+0000 7fdec1535640 1 -- 172.21.15.83:0/268755771 wait complete. 2024-09-19T07:21:12.062 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-09-19T07:21:12.062 INFO:tasks.mds_thrash.fs.[b]:waiting for 59 secs before thrashing 2024-09-19T07:21:12.094 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:21:12.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.208+0000 7fab04205640 1 Processor -- start 2024-09-19T07:21:12.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.208+0000 7fab04205640 1 -- start start 2024-09-19T07:21:12.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab04205640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faafc073830 0x7faafc154050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab04205640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faafc1570c0 0x7faafc154590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab04205640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faafc154ad0 con 0x7faafc0c43c0 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab04205640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7faafc153b50 con 0x7faafc1570c0 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab04205640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faafc1575c0 con 0x7faafc073830 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab027a2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faafc1570c0 0x7faafc154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab027a2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faafc1570c0 0x7faafc154590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32786/0 (socket says 172.21.15.83:32786) 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab027a2640 1 -- 172.21.15.83:0/3348394909 learned_addr learned my addr 172.21.15.83:0/3348394909 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab027a2640 1 -- 172.21.15.83:0/3348394909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faafc073830 msgr2=0x7faafc154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab017a0640 1 --2- 172.21.15.83:0/3348394909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faafc073830 0x7faafc154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab027a2640 1 --2- 172.21.15.83:0/3348394909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faafc073830 0x7faafc154050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab027a2640 1 -- 172.21.15.83:0/3348394909 >> v1:172.21.15.184:6789/0 conn(0x7faafc0c43c0 legacy=0x7faafc06cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab027a2640 1 -- 172.21.15.83:0/3348394909 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faafc1578a0 con 0x7faafc1570c0 2024-09-19T07:21:12.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab017a0640 1 --2- 172.21.15.83:0/3348394909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faafc073830 0x7faafc154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:21:12.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab027a2640 1 --2- 172.21.15.83:0/3348394909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc1570c0 0x7faafc154590 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7faaf4009e50 tx=0x7faaf401c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e8e37ed9bbb55cb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:12.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab00f9f640 1 -- 172.21.15.83:0/3348394909 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf40090d0 con 0x7faafc1570c0 2024-09-19T07:21:12.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab00f9f640 1 -- 172.21.15.83:0/3348394909 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaf40a6500 con 0x7faafc1570c0 2024-09-19T07:21:12.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.212+0000 7fab00f9f640 1 -- 172.21.15.83:0/3348394909 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4009270 con 0x7faafc1570c0 2024-09-19T07:21:12.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 -- 172.21.15.83:0/3348394909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc1570c0 msgr2=0x7faafc154590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:12.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 --2- 172.21.15.83:0/3348394909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc1570c0 0x7faafc154590 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7faaf4009e50 tx=0x7faaf401c780 comp rx=0 tx=0).stop 2024-09-19T07:21:12.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 -- 172.21.15.83:0/3348394909 shutdown_connections 2024-09-19T07:21:12.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 --2- 172.21.15.83:0/3348394909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc1570c0 0x7faafc154590 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 --2- 172.21.15.83:0/3348394909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faafc073830 0x7faafc154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 -- 172.21.15.83:0/3348394909 >> 172.21.15.83:0/3348394909 conn(0x7faafc0c5060 msgr2=0x7faafc0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:12.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 -- 172.21.15.83:0/3348394909 shutdown_connections 2024-09-19T07:21:12.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 -- 172.21.15.83:0/3348394909 wait complete. 2024-09-19T07:21:12.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 Processor -- start 2024-09-19T07:21:12.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 -- start start 2024-09-19T07:21:12.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc073830 0x7faafc152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:12.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faafc0c43c0 0x7faafc06f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:12.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faafc1570c0 0x7faafc148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:12.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faafc06d520 con 0x7faafc073830 2024-09-19T07:21:12.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faafc06d3a0 con 0x7faafc1570c0 2024-09-19T07:21:12.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faafc15ec00 con 0x7faafc0c43c0 2024-09-19T07:21:12.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab01fa1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc073830 0x7faafc152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:12.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab01fa1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc073830 0x7faafc152ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32802/0 (socket says 172.21.15.83:32802) 2024-09-19T07:21:12.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab01fa1640 1 -- 172.21.15.83:0/1588054324 learned_addr learned my addr 172.21.15.83:0/1588054324 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:12.222 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fb55b91a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:21:12.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab017a0640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faafc0c43c0 0x7faafc06f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:12.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab027a2640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faafc1570c0 0x7faafc148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:12.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab01fa1640 1 -- 172.21.15.83:0/1588054324 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faafc0c43c0 msgr2=0x7faafc06f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:12.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab01fa1640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faafc0c43c0 0x7faafc06f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab01fa1640 1 -- 172.21.15.83:0/1588054324 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faafc1570c0 msgr2=0x7faafc148810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab01fa1640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faafc1570c0 0x7faafc148810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab01fa1640 1 -- 172.21.15.83:0/1588054324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faafc14e250 con 0x7faafc073830 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab01fa1640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc073830 0x7faafc152ac0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7faaf800c450 tx=0x7faaf800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7faaf27fc640 1 -- 172.21.15.83:0/1588054324 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf8012830 con 0x7faafc073830 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7faaf27fc640 1 -- 172.21.15.83:0/1588054324 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaf8017070 con 0x7faafc073830 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 -- 172.21.15.83:0/1588054324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faafc14e530 con 0x7faafc073830 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7faaf27fc640 1 -- 172.21.15.83:0/1588054324 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf8012c30 con 0x7faafc073830 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.216+0000 7fab04205640 1 -- 172.21.15.83:0/1588054324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faafc15d300 con 0x7faafc073830 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.220+0000 7fab04205640 1 -- 172.21.15.83:0/1588054324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faafc06cc90 con 0x7faafc073830 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.220+0000 7faaf27fc640 1 -- 172.21.15.83:0/1588054324 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faaf801c460 con 0x7faafc073830 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.220+0000 7faaf27fc640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faae00678c0 0x7faae0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.220+0000 7faaf27fc640 1 -- 172.21.15.83:0/1588054324 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7faaf8085e30 con 0x7faafc073830 2024-09-19T07:21:12.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.220+0000 7fab017a0640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faae00678c0 0x7faae0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:12.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.220+0000 7fab017a0640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faae00678c0 0x7faae0069d80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faafc0c27e0 tx=0x7faaec00d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:12.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.224+0000 7faaf27fc640 1 -- 172.21.15.83:0/1588054324 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7faaf80565b0 con 0x7faafc073830 2024-09-19T07:21:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.460+0000 7fab04205640 1 -- 172.21.15.83:0/1588054324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faafc151ac0 con 0x7faafc073830 2024-09-19T07:21:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.460+0000 7faaf27fc640 1 -- 172.21.15.83:0/1588054324 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5714 (secure 0 0 0) 0x7faaf8055eb0 con 0x7faafc073830 2024-09-19T07:21:12.466 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:21:12.466 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":22,"btime":"2024-09-19T07:21:10:937450+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:21:12.466 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 22 2024-09-19T07:21:12.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.464+0000 7fab04205640 1 -- 172.21.15.83:0/1588054324 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faae00678c0 msgr2=0x7faae0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.464+0000 7fab04205640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faae00678c0 0x7faae0069d80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faafc0c27e0 tx=0x7faaec00d040 comp rx=0 tx=0).stop 2024-09-19T07:21:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.464+0000 7fab04205640 1 -- 172.21.15.83:0/1588054324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc073830 msgr2=0x7faafc152ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.464+0000 7fab04205640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc073830 0x7faafc152ac0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7faaf800c450 tx=0x7faaf800c910 comp rx=0 tx=0).stop 2024-09-19T07:21:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.464+0000 7fab04205640 1 -- 172.21.15.83:0/1588054324 shutdown_connections 2024-09-19T07:21:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.464+0000 7fab04205640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faae00678c0 0x7faae0069d80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.464+0000 7fab04205640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faafc1570c0 0x7faafc148810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.464+0000 7fab04205640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faafc0c43c0 0x7faafc06f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.464+0000 7fab04205640 1 --2- 172.21.15.83:0/1588054324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc073830 0x7faafc152ac0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.464+0000 7fab04205640 1 -- 172.21.15.83:0/1588054324 >> 172.21.15.83:0/1588054324 conn(0x7faafc0c5060 msgr2=0x7faafc0c5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.464+0000 7fab04205640 1 -- 172.21.15.83:0/1588054324 shutdown_connections 2024-09-19T07:21:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:12.464+0000 7fab04205640 1 -- 172.21.15.83:0/1588054324 wait complete. 2024-09-19T07:21:12.487 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=4254 2024-09-19T07:21:12.487 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before reviving mds.a 2024-09-19T07:21:12.708 INFO:tasks.workunit.client.0.smithi083.stderr:+ trap - EXIT 2024-09-19T07:21:12.709 INFO:tasks.workunit.client.0.smithi083.stderr:+ cd ffsb 2024-09-19T07:21:12.709 INFO:tasks.workunit.client.0.smithi083.stderr:+ ./configure 2024-09-19T07:21:12.854 INFO:tasks.workunit.client.0.smithi083.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-19T07:21:12.886 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether build environment is sane... yes 2024-09-19T07:21:12.899 INFO:tasks.workunit.client.1.smithi184.stdout:checking for a race-free mkdir -p... /usr/bin/mkdir -p 2024-09-19T07:21:12.900 INFO:tasks.workunit.client.1.smithi184.stdout:checking for gawk... gawk 2024-09-19T07:21:12.923 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether make sets $(MAKE)... yes 2024-09-19T07:21:12.982 INFO:tasks.workunit.client.1.smithi184.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-19T07:21:12.982 INFO:tasks.workunit.client.1.smithi184.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-19T07:21:13.005 INFO:tasks.workunit.client.1.smithi184.stdout:checking for style of include used by make... GNU 2024-09-19T07:21:13.010 INFO:tasks.workunit.client.1.smithi184.stdout:checking for gcc... gcc 2024-09-19T07:21:13.157 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether the C compiler works... yes 2024-09-19T07:21:13.158 INFO:tasks.workunit.client.1.smithi184.stdout:checking for C compiler default output file name... a.out 2024-09-19T07:21:13.217 INFO:tasks.workunit.client.1.smithi184.stdout:checking for suffix of executables... 2024-09-19T07:21:13.277 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether we are cross compiling... no 2024-09-19T07:21:13.328 INFO:tasks.workunit.client.1.smithi184.stdout:checking for suffix of object files... o 2024-09-19T07:21:13.360 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether the compiler supports GNU C... yes 2024-09-19T07:21:13.395 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether gcc accepts -g... yes 2024-09-19T07:21:13.482 INFO:tasks.workunit.client.1.smithi184.stdout:checking for gcc option to enable C11 features... none needed 2024-09-19T07:21:13.645 INFO:tasks.workunit.client.1.smithi184.stdout:checking dependency style of gcc... gcc3 2024-09-19T07:21:13.680 INFO:tasks.workunit.client.1.smithi184.stdout:checking for stdio.h... yes 2024-09-19T07:21:13.729 INFO:tasks.workunit.client.1.smithi184.stdout:checking for stdlib.h... yes 2024-09-19T07:21:13.772 INFO:tasks.workunit.client.1.smithi184.stdout:checking for string.h... yes 2024-09-19T07:21:13.814 INFO:tasks.workunit.client.1.smithi184.stdout:checking for inttypes.h... yes 2024-09-19T07:21:13.854 INFO:tasks.workunit.client.1.smithi184.stdout:checking for stdint.h... yes 2024-09-19T07:21:13.870 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether build environment is sane... yes 2024-09-19T07:21:13.889 INFO:tasks.workunit.client.0.smithi083.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-19T07:21:13.891 INFO:tasks.workunit.client.0.smithi083.stdout:checking for gawk... gawk 2024-09-19T07:21:13.892 INFO:tasks.workunit.client.1.smithi184.stdout:checking for strings.h... yes 2024-09-19T07:21:13.916 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether make sets $(MAKE)... yes 2024-09-19T07:21:13.931 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/stat.h... yes 2024-09-19T07:21:13.970 INFO:tasks.workunit.client.0.smithi083.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-19T07:21:13.970 INFO:tasks.workunit.client.0.smithi083.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-19T07:21:13.975 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/types.h... yes 2024-09-19T07:21:13.990 INFO:tasks.workunit.client.0.smithi083.stdout:checking for style of include used by make... GNU 2024-09-19T07:21:13.995 INFO:tasks.workunit.client.0.smithi083.stdout:checking for gcc... gcc 2024-09-19T07:21:14.020 INFO:tasks.workunit.client.1.smithi184.stdout:checking for unistd.h... yes 2024-09-19T07:21:14.068 INFO:tasks.workunit.client.1.smithi184.stdout:checking for wchar.h... yes 2024-09-19T07:21:14.090 INFO:tasks.workunit.client.0.smithi083.stdout:checking for C compiler default output file name... a.out 2024-09-19T07:21:14.092 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether the C compiler works... yes 2024-09-19T07:21:14.095 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether we are cross compiling... no 2024-09-19T07:21:14.123 INFO:tasks.workunit.client.1.smithi184.stdout:checking for minix/config.h... no 2024-09-19T07:21:14.153 INFO:tasks.workunit.client.0.smithi083.stdout:checking for suffix of executables... 2024-09-19T07:21:14.165 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/time.h... yes 2024-09-19T07:21:14.199 INFO:tasks.workunit.client.0.smithi083.stdout:checking for suffix of object files... o 2024-09-19T07:21:14.218 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-09-19T07:21:14.240 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether we are using the GNU C compiler... yes 2024-09-19T07:21:14.255 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether _XOPEN_SOURCE should be defined... no 2024-09-19T07:21:14.265 INFO:tasks.workunit.client.1.smithi184.stdout:checking for gcc... (cached) gcc 2024-09-19T07:21:14.284 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether gcc accepts -g... yes 2024-09-19T07:21:14.324 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether the compiler supports GNU C... (cached) yes 2024-09-19T07:21:14.324 INFO:tasks.workunit.client.1.smithi184.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-19T07:21:14.324 INFO:tasks.workunit.client.1.smithi184.stdout:checking for gcc option to enable C11 features... (cached) none needed 2024-09-19T07:21:14.325 INFO:tasks.workunit.client.1.smithi184.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-19T07:21:14.383 INFO:tasks.workunit.client.1.smithi184.stdout:checking for main in -lm... yes 2024-09-19T07:21:14.421 INFO:tasks.workunit.client.0.smithi083.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-19T07:21:14.447 INFO:tasks.workunit.client.1.smithi184.stdout:checking for main in -lpthread... yes 2024-09-19T07:21:14.450 INFO:tasks.workunit.client.1.smithi184.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-19T07:21:14.451 INFO:tasks.workunit.client.1.smithi184.stdout:checking for egrep... /usr/bin/grep -E 2024-09-19T07:21:14.494 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-19T07:21:14.545 INFO:tasks.workunit.client.1.smithi184.stdout:checking for pthread.h... yes 2024-09-19T07:21:14.592 INFO:tasks.workunit.client.1.smithi184.stdout:checking for fcntl.h... yes 2024-09-19T07:21:14.597 INFO:tasks.workunit.client.0.smithi083.stdout:checking dependency style of gcc... gcc3 2024-09-19T07:21:14.640 INFO:tasks.workunit.client.1.smithi184.stdout:checking for limits.h... yes 2024-09-19T07:21:14.642 INFO:tasks.workunit.client.1.smithi184.stdout:checking for stdint.h... (cached) yes 2024-09-19T07:21:14.643 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/time.h... (cached) yes 2024-09-19T07:21:14.644 INFO:tasks.workunit.client.1.smithi184.stdout:checking for unistd.h... (cached) yes 2024-09-19T07:21:14.686 INFO:tasks.workunit.client.0.smithi083.stdout:checking how to run the C preprocessor... gcc -E 2024-09-19T07:21:14.691 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/vfs.h... yes 2024-09-19T07:21:14.748 INFO:tasks.workunit.client.0.smithi083.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-19T07:21:14.748 INFO:tasks.workunit.client.1.smithi184.stdout:checking for sys/limits.h... no 2024-09-19T07:21:14.749 INFO:tasks.workunit.client.0.smithi083.stdout:checking for egrep... /usr/bin/grep -E 2024-09-19T07:21:14.762 INFO:tasks.workunit.client.0.smithi083.stdout:checking for AIX... no 2024-09-19T07:21:14.779 INFO:tasks.workunit.client.1.smithi184.stdout:checking for an ANSI C-conforming const... yes 2024-09-19T07:21:14.785 INFO:tasks.workunit.client.0.smithi083.stdout:checking for gcc... (cached) gcc 2024-09-19T07:21:14.799 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-19T07:21:14.800 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-19T07:21:14.801 INFO:tasks.workunit.client.0.smithi083.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-19T07:21:14.802 INFO:tasks.workunit.client.0.smithi083.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-19T07:21:14.847 INFO:tasks.workunit.client.1.smithi184.stdout:checking for system... yes 2024-09-19T07:21:14.890 INFO:tasks.workunit.client.0.smithi083.stdout:checking for main in -lm... yes 2024-09-19T07:21:14.912 INFO:tasks.workunit.client.1.smithi184.stdout:checking for gettimeofday... yes 2024-09-19T07:21:14.977 INFO:tasks.workunit.client.0.smithi083.stdout:checking for main in -lpthread... yes 2024-09-19T07:21:14.982 INFO:tasks.workunit.client.1.smithi184.stdout:checking for mkdir... yes 2024-09-19T07:21:15.047 INFO:tasks.workunit.client.1.smithi184.stdout:checking for strerror... yes 2024-09-19T07:21:15.113 INFO:tasks.workunit.client.1.smithi184.stdout:checking for open64... yes 2024-09-19T07:21:15.178 INFO:tasks.workunit.client.1.smithi184.stdout:checking for stat64... yes 2024-09-19T07:21:15.198 INFO:tasks.workunit.client.0.smithi083.stdout:checking for ANSI C header files... yes 2024-09-19T07:21:15.244 INFO:tasks.workunit.client.1.smithi184.stdout:checking for fseeko64... yes 2024-09-19T07:21:15.256 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-19T07:21:15.311 INFO:tasks.workunit.client.1.smithi184.stdout:checking for lrand48_r... yes 2024-09-19T07:21:15.311 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/types.h... yes 2024-09-19T07:21:15.372 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/stat.h... yes 2024-09-19T07:21:15.375 INFO:tasks.workunit.client.1.smithi184.stdout:checking for srand48_r... yes 2024-09-19T07:21:15.396 INFO:tasks.workunit.client.1.smithi184.stdout:configure: creating ./config.status 2024-09-19T07:21:15.436 INFO:tasks.workunit.client.0.smithi083.stdout:checking for stdlib.h... yes 2024-09-19T07:21:15.497 INFO:tasks.workunit.client.0.smithi083.stdout:checking for string.h... yes 2024-09-19T07:21:15.500 INFO:tasks.workunit.client.1.smithi184.stdout: /bin/bash ./config.status 2024-09-19T07:21:15.558 INFO:tasks.workunit.client.1.smithi184.stdout:config.status: creating Makefile 2024-09-19T07:21:15.559 INFO:tasks.workunit.client.0.smithi083.stdout:checking for memory.h... yes 2024-09-19T07:21:15.575 INFO:tasks.workunit.client.1.smithi184.stdout:config.status: creating config.h 2024-09-19T07:21:15.604 INFO:tasks.workunit.client.1.smithi184.stdout:config.status: executing depfiles commands 2024-09-19T07:21:15.621 INFO:tasks.workunit.client.0.smithi083.stdout:checking for strings.h... yes 2024-09-19T07:21:15.622 INFO:tasks.workunit.client.1.smithi184.stdout:config.status: executing default commands 2024-09-19T07:21:15.679 INFO:tasks.workunit.client.1.smithi184.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run autoheader 2024-09-19T07:21:15.685 INFO:tasks.workunit.client.0.smithi083.stdout:checking for inttypes.h... yes 2024-09-19T07:21:15.736 INFO:tasks.workunit.client.1.smithi184.stderr:autoheader: warning: autoconf input should be named 'configure.ac', not 'configure.in' 2024-09-19T07:21:15.744 INFO:tasks.workunit.client.0.smithi083.stdout:checking for stdint.h... yes 2024-09-19T07:21:15.808 INFO:tasks.workunit.client.0.smithi083.stdout:checking for unistd.h... yes 2024-09-19T07:21:15.809 INFO:tasks.workunit.client.1.smithi184.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-19T07:21:15.809 INFO:tasks.workunit.client.1.smithi184.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-19T07:21:15.809 INFO:tasks.workunit.client.1.smithi184.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-19T07:21:15.809 INFO:tasks.workunit.client.1.smithi184.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-19T07:21:15.878 INFO:tasks.workunit.client.0.smithi083.stdout:checking pthread.h usability... yes 2024-09-19T07:21:15.906 INFO:tasks.workunit.client.0.smithi083.stdout:checking pthread.h presence... yes 2024-09-19T07:21:15.906 INFO:tasks.workunit.client.0.smithi083.stdout:checking for pthread.h... yes 2024-09-19T07:21:15.935 INFO:tasks.workunit.client.1.smithi184.stdout:rm -f stamp-h1 2024-09-19T07:21:15.939 INFO:tasks.workunit.client.1.smithi184.stdout:touch config.h.in 2024-09-19T07:21:15.948 INFO:tasks.workunit.client.1.smithi184.stdout:cd . && /bin/bash ./config.status config.h 2024-09-19T07:21:15.990 INFO:tasks.workunit.client.0.smithi083.stdout:checking fcntl.h usability... yes 2024-09-19T07:21:16.009 INFO:tasks.workunit.client.1.smithi184.stdout:config.status: creating config.h 2024-09-19T07:21:16.015 INFO:tasks.workunit.client.0.smithi083.stdout:checking fcntl.h presence... yes 2024-09-19T07:21:16.016 INFO:tasks.workunit.client.0.smithi083.stdout:checking for fcntl.h... yes 2024-09-19T07:21:16.045 INFO:tasks.workunit.client.1.smithi184.stdout:make all-am 2024-09-19T07:21:16.072 INFO:tasks.workunit.client.1.smithi184.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb' 2024-09-19T07:21:16.072 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-09-19T07:21:16.085 INFO:tasks.workunit.client.0.smithi083.stdout:checking limits.h usability... yes 2024-09-19T07:21:16.110 INFO:tasks.workunit.client.0.smithi083.stdout:checking limits.h presence... yes 2024-09-19T07:21:16.111 INFO:tasks.workunit.client.0.smithi083.stdout:checking for limits.h... yes 2024-09-19T07:21:16.137 INFO:tasks.workunit.client.0.smithi083.stdout:checking for stdint.h... (cached) yes 2024-09-19T07:21:16.203 INFO:tasks.workunit.client.0.smithi083.stdout:checking sys/time.h usability... yes 2024-09-19T07:21:16.228 INFO:tasks.workunit.client.0.smithi083.stdout:checking sys/time.h presence... yes 2024-09-19T07:21:16.229 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/time.h... yes 2024-09-19T07:21:16.251 INFO:tasks.workunit.client.0.smithi083.stdout:checking for unistd.h... (cached) yes 2024-09-19T07:21:16.293 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-09-19T07:21:16.303 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-09-19T07:21:16.325 INFO:tasks.workunit.client.0.smithi083.stdout:checking sys/vfs.h usability... yes 2024-09-19T07:21:16.350 INFO:tasks.workunit.client.0.smithi083.stdout:checking sys/vfs.h presence... yes 2024-09-19T07:21:16.350 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/vfs.h... yes 2024-09-19T07:21:16.416 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-09-19T07:21:16.423 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-09-19T07:21:16.427 INFO:tasks.workunit.client.0.smithi083.stdout:checking sys/limits.h usability... no 2024-09-19T07:21:16.458 INFO:tasks.workunit.client.0.smithi083.stdout:checking sys/limits.h presence... no 2024-09-19T07:21:16.458 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/limits.h... no 2024-09-19T07:21:16.555 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-09-19T07:21:16.556 INFO:tasks.workunit.client.0.smithi083.stdout:checking for an ANSI C-conforming const... yes 2024-09-19T07:21:16.564 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-09-19T07:21:16.702 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-09-19T07:21:16.712 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-09-19T07:21:16.717 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-19T07:21:16.721 INFO:tasks.workunit.client.0.smithi083.stdout:checking for function prototypes... yes 2024-09-19T07:21:16.888 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-09-19T07:21:16.898 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-09-19T07:21:16.925 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether setvbuf arguments are reversed... no 2024-09-19T07:21:17.003 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-09-19T07:21:17.010 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-09-19T07:21:17.064 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-09-19T07:21:17.071 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-09-19T07:21:17.140 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-09-19T07:21:17.149 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-09-19T07:21:17.188 INFO:tasks.workunit.client.0.smithi083.stdout:checking for system... yes 2024-09-19T07:21:17.293 INFO:tasks.workunit.client.0.smithi083.stdout:checking for gettimeofday... yes 2024-09-19T07:21:17.393 INFO:tasks.workunit.client.0.smithi083.stdout:checking for mkdir... yes 2024-09-19T07:21:17.405 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-09-19T07:21:17.415 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-09-19T07:21:17.485 INFO:tasks.workunit.client.0.smithi083.stdout:checking for strerror... yes 2024-09-19T07:21:17.578 INFO:tasks.workunit.client.0.smithi083.stdout:checking for open64... yes 2024-09-19T07:21:17.632 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-09-19T07:21:17.644 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-09-19T07:21:17.668 INFO:tasks.workunit.client.0.smithi083.stdout:checking for stat64... yes 2024-09-19T07:21:17.757 INFO:tasks.workunit.client.0.smithi083.stdout:checking for fseeko64... yes 2024-09-19T07:21:17.847 INFO:tasks.workunit.client.0.smithi083.stdout:checking for lrand48_r... yes 2024-09-19T07:21:17.875 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-09-19T07:21:17.885 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-09-19T07:21:17.941 INFO:tasks.workunit.client.0.smithi083.stdout:checking for srand48_r... yes 2024-09-19T07:21:17.993 INFO:tasks.workunit.client.0.smithi083.stdout:configure: creating ./config.status 2024-09-19T07:21:18.005 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-09-19T07:21:18.013 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-09-19T07:21:18.070 INFO:tasks.workunit.client.1.smithi184.stderr:ffsb_op.c: In function 'print_op_results': 2024-09-19T07:21:18.071 INFO:tasks.workunit.client.1.smithi184.stderr:ffsb_op.c:83:37: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-09-19T07:21:18.071 INFO:tasks.workunit.client.1.smithi184.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-09-19T07:21:18.071 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~ 2024-09-19T07:21:18.071 INFO:tasks.workunit.client.1.smithi184.stderr:ffsb_op.c:86:33: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-09-19T07:21:18.071 INFO:tasks.workunit.client.1.smithi184.stderr: 86 | sprintf(buf, "NA\0"); 2024-09-19T07:21:18.071 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~ 2024-09-19T07:21:18.086 INFO:tasks.workunit.client.1.smithi184.stderr:ffsb_op.c: In function 'print_results': 2024-09-19T07:21:18.086 INFO:tasks.workunit.client.1.smithi184.stderr:ffsb_op.c:83:33: warning: '/sec' directive writing 4 bytes into a region of size between 1 and 256 [-Wformat-overflow=] 2024-09-19T07:21:18.086 INFO:tasks.workunit.client.1.smithi184.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-09-19T07:21:18.086 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~ 2024-09-19T07:21:18.087 INFO:tasks.workunit.client.1.smithi184.stderr:In file included from /usr/include/stdio.h:894, 2024-09-19T07:21:18.087 INFO:tasks.workunit.client.1.smithi184.stderr: from ffsb_op.c:20: 2024-09-19T07:21:18.087 INFO:tasks.workunit.client.1.smithi184.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 5 and 260 bytes into a destination of size 256 2024-09-19T07:21:18.087 INFO:tasks.workunit.client.1.smithi184.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, 2024-09-19T07:21:18.087 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.087 INFO:tasks.workunit.client.1.smithi184.stderr: 39 | __glibc_objsize (__s), __fmt, 2024-09-19T07:21:18.088 INFO:tasks.workunit.client.1.smithi184.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.088 INFO:tasks.workunit.client.1.smithi184.stderr: 40 | __va_arg_pack ()); 2024-09-19T07:21:18.088 INFO:tasks.workunit.client.1.smithi184.stderr: | ~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.123 INFO:tasks.workunit.client.0.smithi083.stdout:config.status: creating Makefile 2024-09-19T07:21:18.134 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-09-19T07:21:18.142 INFO:tasks.workunit.client.0.smithi083.stdout:config.status: creating config.h 2024-09-19T07:21:18.146 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-09-19T07:21:18.166 INFO:tasks.workunit.client.0.smithi083.stdout:config.status: executing depfiles commands 2024-09-19T07:21:18.233 INFO:tasks.workunit.client.0.smithi083.stdout:config.status: executing default commands 2024-09-19T07:21:18.328 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-09-19T07:21:18.331 INFO:tasks.workunit.client.0.smithi083.stderr:+ make 2024-09-19T07:21:18.338 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-09-19T07:21:18.362 INFO:tasks.workunit.client.0.smithi083.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-09-19T07:21:18.368 INFO:tasks.workunit.client.0.smithi083.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-09-19T07:21:18.369 INFO:tasks.workunit.client.0.smithi083.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-09-19T07:21:18.369 INFO:tasks.workunit.client.0.smithi083.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-09-19T07:21:18.369 INFO:tasks.workunit.client.0.smithi083.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-09-19T07:21:18.369 INFO:tasks.workunit.client.0.smithi083.stderr: any GNU archive site. 2024-09-19T07:21:18.374 INFO:tasks.workunit.client.0.smithi083.stdout: cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-09-19T07:21:18.380 INFO:tasks.workunit.client.0.smithi083.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-09-19T07:21:18.380 INFO:tasks.workunit.client.0.smithi083.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-09-19T07:21:18.381 INFO:tasks.workunit.client.0.smithi083.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-09-19T07:21:18.381 INFO:tasks.workunit.client.0.smithi083.stderr: You might want to install the `Automake' and `Perl' packages. 2024-09-19T07:21:18.381 INFO:tasks.workunit.client.0.smithi083.stderr: Grab them from any GNU archive site. 2024-09-19T07:21:18.406 INFO:tasks.workunit.client.1.smithi184.stderr:In file included from parser.h:22, 2024-09-19T07:21:18.406 INFO:tasks.workunit.client.1.smithi184.stderr: from parser.c:25: 2024-09-19T07:21:18.406 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c: In function 'init_filesys': 2024-09-19T07:21:18.406 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:18.406 INFO:tasks.workunit.client.1.smithi184.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:18.406 INFO:tasks.workunit.client.1.smithi184.stderr: | ^ 2024-09-19T07:21:18.406 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:18.406 INFO:tasks.workunit.client.1.smithi184.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:18.406 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~ 2024-09-19T07:21:18.407 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:18.407 INFO:tasks.workunit.client.1.smithi184.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:18.407 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:18.407 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:18.407 INFO:tasks.workunit.client.1.smithi184.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-19T07:21:18.407 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:18.407 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:18.407 INFO:tasks.workunit.client.1.smithi184.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-19T07:21:18.407 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.407 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:18.407 INFO:tasks.workunit.client.1.smithi184.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:18.408 INFO:tasks.workunit.client.1.smithi184.stderr: | ^ 2024-09-19T07:21:18.408 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:18.408 INFO:tasks.workunit.client.1.smithi184.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:18.408 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~ 2024-09-19T07:21:18.408 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:18.408 INFO:tasks.workunit.client.1.smithi184.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:18.408 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:18.408 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:18.408 INFO:tasks.workunit.client.1.smithi184.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-19T07:21:18.409 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:18.409 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:18.409 INFO:tasks.workunit.client.1.smithi184.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-19T07:21:18.409 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.409 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:18.409 INFO:tasks.workunit.client.1.smithi184.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:18.409 INFO:tasks.workunit.client.1.smithi184.stderr: | ^ 2024-09-19T07:21:18.409 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:18.409 INFO:tasks.workunit.client.1.smithi184.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:18.409 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~ 2024-09-19T07:21:18.409 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:18.409 INFO:tasks.workunit.client.1.smithi184.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:18.410 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:18.410 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:18.410 INFO:tasks.workunit.client.1.smithi184.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-19T07:21:18.410 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:18.410 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:18.410 INFO:tasks.workunit.client.1.smithi184.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-19T07:21:18.410 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.410 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:18.410 INFO:tasks.workunit.client.1.smithi184.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:18.410 INFO:tasks.workunit.client.1.smithi184.stderr: | ^ 2024-09-19T07:21:18.410 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:18.410 INFO:tasks.workunit.client.1.smithi184.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:18.411 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~ 2024-09-19T07:21:18.411 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:18.411 INFO:tasks.workunit.client.1.smithi184.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:18.411 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:18.411 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:18.411 INFO:tasks.workunit.client.1.smithi184.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-19T07:21:18.411 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:18.412 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:18.412 INFO:tasks.workunit.client.1.smithi184.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-19T07:21:18.412 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.412 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c: In function 'init_tg_stats': 2024-09-19T07:21:18.412 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:18.412 INFO:tasks.workunit.client.1.smithi184.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:18.412 INFO:tasks.workunit.client.1.smithi184.stderr: | ^ 2024-09-19T07:21:18.412 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:18.412 INFO:tasks.workunit.client.1.smithi184.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:18.412 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~ 2024-09-19T07:21:18.413 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:18.413 INFO:tasks.workunit.client.1.smithi184.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:18.413 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:18.413 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:18.413 INFO:tasks.workunit.client.1.smithi184.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-19T07:21:18.413 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:18.413 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:18.413 INFO:tasks.workunit.client.1.smithi184.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-19T07:21:18.413 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.413 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:18.413 INFO:tasks.workunit.client.1.smithi184.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:18.414 INFO:tasks.workunit.client.1.smithi184.stderr: | ^ 2024-09-19T07:21:18.414 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:18.414 INFO:tasks.workunit.client.1.smithi184.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:18.414 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~ 2024-09-19T07:21:18.414 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:18.414 INFO:tasks.workunit.client.1.smithi184.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:18.414 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:18.414 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:18.414 INFO:tasks.workunit.client.1.smithi184.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-19T07:21:18.414 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:18.414 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:18.415 INFO:tasks.workunit.client.1.smithi184.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-19T07:21:18.415 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.415 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:18.415 INFO:tasks.workunit.client.1.smithi184.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:18.415 INFO:tasks.workunit.client.1.smithi184.stderr: | ^ 2024-09-19T07:21:18.415 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:18.415 INFO:tasks.workunit.client.1.smithi184.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:18.415 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~ 2024-09-19T07:21:18.415 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr: | ^ 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:18.416 INFO:tasks.workunit.client.1.smithi184.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:18.417 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~ 2024-09-19T07:21:18.417 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:18.417 INFO:tasks.workunit.client.1.smithi184.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:18.417 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:18.417 INFO:tasks.workunit.client.1.smithi184.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:18.417 INFO:tasks.workunit.client.1.smithi184.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-19T07:21:18.417 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:18.417 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:18.417 INFO:tasks.workunit.client.1.smithi184.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-19T07:21:18.417 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.464 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c: In function 'get_optstr': 2024-09-19T07:21:18.464 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c:206:23: warning: 'strnlen' specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-09-19T07:21:18.464 INFO:tasks.workunit.client.1.smithi184.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-09-19T07:21:18.465 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.465 INFO:tasks.workunit.client.1.smithi184.stderr:parser.c:200:14: note: source object declared here 2024-09-19T07:21:18.465 INFO:tasks.workunit.client.1.smithi184.stderr: 200 | char temp[BUFSIZE]; 2024-09-19T07:21:18.465 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~ 2024-09-19T07:21:18.481 INFO:tasks.workunit.client.1.smithi184.stderr:In file included from /usr/include/string.h:535, 2024-09-19T07:21:18.481 INFO:tasks.workunit.client.1.smithi184.stderr: from parser.c:19: 2024-09-19T07:21:18.481 INFO:tasks.workunit.client.1.smithi184.stderr:In function 'memcpy', 2024-09-19T07:21:18.481 INFO:tasks.workunit.client.1.smithi184.stderr: inlined from 'size64_convert' at parser.c:119:2: 2024-09-19T07:21:18.481 INFO:tasks.workunit.client.1.smithi184.stderr:/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10: warning: '__builtin_memcpy' reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-09-19T07:21:18.482 INFO:tasks.workunit.client.1.smithi184.stderr: 29 | return __builtin___memcpy_chk (__dest, __src, __len, 2024-09-19T07:21:18.482 INFO:tasks.workunit.client.1.smithi184.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.482 INFO:tasks.workunit.client.1.smithi184.stderr: 30 | __glibc_objsize0 (__dest)); 2024-09-19T07:21:18.482 INFO:tasks.workunit.client.1.smithi184.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:18.488 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-19T07:21:18.488 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:21:18.488 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:21:18.490 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:21:18.490 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:21:18.553 INFO:tasks.workunit.client.0.smithi083.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-09-19T07:21:18.555 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:21:18.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.596+0000 7f16424cb640 1 Processor -- start 2024-09-19T07:21:18.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.596+0000 7f16424cb640 1 -- start start 2024-09-19T07:21:18.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f16424cb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f163c0c3a20 0x7f163c1594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:18.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f16424cb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f163c0c4450 0x7f163c1566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:18.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f16424cb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f163c155ee0 con 0x7f163c156c00 2024-09-19T07:21:18.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f16424cb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f163c068290 con 0x7f163c0c3a20 2024-09-19T07:21:18.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f16424cb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f163c0684e0 con 0x7f163c0c4450 2024-09-19T07:21:18.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f163bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f163c0c3a20 0x7f163c1594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:18.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f1640a68640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f163c156c00 0x7f163c14c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42274/0 (socket says 172.21.15.83:42274) 2024-09-19T07:21:18.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f1640a68640 1 -- 172.21.15.83:0/3716330592 learned_addr learned my addr 172.21.15.83:0/3716330592 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:21:18.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f163b7fe640 1 --2- 172.21.15.83:0/3716330592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f163c0c4450 0x7f163c1566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:18.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f163bfff640 1 -- 172.21.15.83:0/3716330592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f163c0c4450 msgr2=0x7f163c1566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:18.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f163bfff640 1 --2- 172.21.15.83:0/3716330592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f163c0c4450 0x7f163c1566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:18.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f163bfff640 1 -- 172.21.15.83:0/3716330592 >> v1:172.21.15.83:6789/0 conn(0x7f163c156c00 legacy=0x7f163c14c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:18.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f163affd640 1 -- 172.21.15.83:0/3716330592 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4177763678 0 0) 0x7f163c155ee0 con 0x7f163c156c00 2024-09-19T07:21:18.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f163bfff640 1 -- 172.21.15.83:0/3716330592 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163c0687c0 con 0x7f163c0c3a20 2024-09-19T07:21:18.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f163bfff640 1 --2- 172.21.15.83:0/3716330592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f163c0c3a20 0x7f163c1594d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f162c0097c0 tx=0x7f162c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51923830670669ca server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:18.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f163affd640 1 -- 172.21.15.83:0/3716330592 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c06fcd0 con 0x7f163c0c3a20 2024-09-19T07:21:18.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f163affd640 1 -- 172.21.15.83:0/3716330592 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f162c0b0070 con 0x7f163c0c3a20 2024-09-19T07:21:18.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.600+0000 7f163affd640 1 -- 172.21.15.83:0/3716330592 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f162c0ab820 con 0x7f163c0c3a20 2024-09-19T07:21:18.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 -- 172.21.15.83:0/3716330592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f163c0c3a20 msgr2=0x7f163c1594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:18.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 --2- 172.21.15.83:0/3716330592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f163c0c3a20 0x7f163c1594d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f162c0097c0 tx=0x7f162c01c3e0 comp rx=0 tx=0).stop 2024-09-19T07:21:18.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 -- 172.21.15.83:0/3716330592 shutdown_connections 2024-09-19T07:21:18.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 --2- 172.21.15.83:0/3716330592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f163c0c4450 0x7f163c1566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:18.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 --2- 172.21.15.83:0/3716330592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f163c0c3a20 0x7f163c1594d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:18.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 -- 172.21.15.83:0/3716330592 >> 172.21.15.83:0/3716330592 conn(0x7f163c0c48b0 msgr2=0x7f163c0c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:18.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 -- 172.21.15.83:0/3716330592 shutdown_connections 2024-09-19T07:21:18.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 -- 172.21.15.83:0/3716330592 wait complete. 2024-09-19T07:21:18.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 Processor -- start 2024-09-19T07:21:18.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 -- start start 2024-09-19T07:21:18.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f163c0c3a20 0x7f163c14fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:18.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f163c0c4450 0x7f163c14d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:18.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f163c156c00 0x7f163c146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:18.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f163c163080 con 0x7f163c156c00 2024-09-19T07:21:18.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f163c0c4010 con 0x7f163c0c3a20 2024-09-19T07:21:18.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f163c163200 con 0x7f163c0c4450 2024-09-19T07:21:18.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f1640a68640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f163c156c00 0x7f163c146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:18.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f1640a68640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f163c156c00 0x7f163c146210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32826/0 (socket says 172.21.15.83:32826) 2024-09-19T07:21:18.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f1640a68640 1 -- 172.21.15.83:0/2232867664 learned_addr learned my addr 172.21.15.83:0/2232867664 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:18.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f163bfff640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f163c0c3a20 0x7f163c14fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:18.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f163b7fe640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f163c0c4450 0x7f163c14d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:18.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f1640a68640 1 -- 172.21.15.83:0/2232867664 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f163c0c4450 msgr2=0x7f163c14d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:18.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f1640a68640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f163c0c4450 0x7f163c14d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:18.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f1640a68640 1 -- 172.21.15.83:0/2232867664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f163c0c3a20 msgr2=0x7f163c14fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:18.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f1640a68640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f163c0c3a20 0x7f163c14fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:18.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f1640a68640 1 -- 172.21.15.83:0/2232867664 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163c0735d0 con 0x7f163c156c00 2024-09-19T07:21:18.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f163bfff640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f163c0c3a20 0x7f163c14fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:21:18.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f163b7fe640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f163c0c4450 0x7f163c14d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:21:18.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f1640a68640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f163c156c00 0x7f163c146210 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f163400c730 tx=0x7f163400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:18.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f1638ff9640 1 -- 172.21.15.83:0/2232867664 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1634018070 con 0x7f163c156c00 2024-09-19T07:21:18.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f1638ff9640 1 -- 172.21.15.83:0/2232867664 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f163400ce40 con 0x7f163c156c00 2024-09-19T07:21:18.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 -- 172.21.15.83:0/2232867664 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f163c1f27e0 con 0x7f163c156c00 2024-09-19T07:21:18.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f1638ff9640 1 -- 172.21.15.83:0/2232867664 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163400a880 con 0x7f163c156c00 2024-09-19T07:21:18.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 -- 172.21.15.83:0/2232867664 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f163c1557c0 con 0x7f163c156c00 2024-09-19T07:21:18.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.604+0000 7f16424cb640 1 -- 172.21.15.83:0/2232867664 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f163c1594d0 con 0x7f163c156c00 2024-09-19T07:21:18.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.608+0000 7f1638ff9640 1 -- 172.21.15.83:0/2232867664 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f163401b070 con 0x7f163c156c00 2024-09-19T07:21:18.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.608+0000 7f1638ff9640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16280678c0 0x7f1628069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:18.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.608+0000 7f1638ff9640 1 -- 172.21.15.83:0/2232867664 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7f16340855b0 con 0x7f163c156c00 2024-09-19T07:21:18.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.608+0000 7f163bfff640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16280678c0 0x7f1628069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:18.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.608+0000 7f163bfff640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16280678c0 0x7f1628069d80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f162c0af000 tx=0x7f162c0a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:18.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.612+0000 7f1638ff9640 1 -- 172.21.15.83:0/2232867664 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1634055d30 con 0x7f163c156c00 2024-09-19T07:21:18.734 INFO:tasks.workunit.client.0.smithi083.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-19T07:21:18.734 INFO:tasks.workunit.client.0.smithi083.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-19T07:21:18.735 INFO:tasks.workunit.client.0.smithi083.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-19T07:21:18.735 INFO:tasks.workunit.client.0.smithi083.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-19T07:21:18.818 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-09-19T07:21:18.828 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-09-19T07:21:18.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.832+0000 7f16424cb640 1 -- 172.21.15.83:0/2232867664 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f163c06e9a0 con 0x7f163c156c00 2024-09-19T07:21:18.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.832+0000 7f1638ff9640 1 -- 172.21.15.83:0/2232867664 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5714 (secure 0 0 0) 0x7f1634055630 con 0x7f163c156c00 2024-09-19T07:21:18.836 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:21:18.836 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":22,"btime":"2024-09-19T07:21:10:937450+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:21:18.837 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 22 2024-09-19T07:21:18.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.832+0000 7f16424cb640 1 -- 172.21.15.83:0/2232867664 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16280678c0 msgr2=0x7f1628069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:18.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.832+0000 7f16424cb640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16280678c0 0x7f1628069d80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f162c0af000 tx=0x7f162c0a7000 comp rx=0 tx=0).stop 2024-09-19T07:21:18.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.832+0000 7f16424cb640 1 -- 172.21.15.83:0/2232867664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f163c156c00 msgr2=0x7f163c146210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:18.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.832+0000 7f16424cb640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f163c156c00 0x7f163c146210 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f163400c730 tx=0x7f163400cbf0 comp rx=0 tx=0).stop 2024-09-19T07:21:18.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.832+0000 7f16424cb640 1 -- 172.21.15.83:0/2232867664 shutdown_connections 2024-09-19T07:21:18.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.832+0000 7f16424cb640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16280678c0 0x7f1628069d80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:18.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.832+0000 7f16424cb640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f163c156c00 0x7f163c146210 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:18.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.832+0000 7f16424cb640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f163c0c4450 0x7f163c14d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:18.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.832+0000 7f16424cb640 1 --2- 172.21.15.83:0/2232867664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f163c0c3a20 0x7f163c14fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:18.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.836+0000 7f16424cb640 1 -- 172.21.15.83:0/2232867664 >> 172.21.15.83:0/2232867664 conn(0x7f163c0c48b0 msgr2=0x7f163c15ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:18.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.836+0000 7f16424cb640 1 -- 172.21.15.83:0/2232867664 shutdown_connections 2024-09-19T07:21:18.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:18.836+0000 7f16424cb640 1 -- 172.21.15.83:0/2232867664 wait complete. 2024-09-19T07:21:18.840 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:4: warning: The macro `AC_FOREACH' is obsolete. 2024-09-19T07:21:18.840 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:4: You should run autoupdate. 2024-09-19T07:21:18.841 INFO:tasks.workunit.client.0.smithi083.stderr:./lib/autoconf/general.m4:191: AC_FOREACH is expanded from... 2024-09-19T07:21:18.841 INFO:tasks.workunit.client.0.smithi083.stderr:aclocal.m4:684: _AM_SET_OPTIONS is expanded from... 2024-09-19T07:21:18.841 INFO:tasks.workunit.client.0.smithi083.stderr:aclocal.m4:410: AM_INIT_AUTOMAKE is expanded from... 2024-09-19T07:21:18.841 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:4: the top level 2024-09-19T07:21:18.841 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:6: warning: The macro `AM_CONFIG_HEADER' is obsolete. 2024-09-19T07:21:18.841 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:6: You should run autoupdate. 2024-09-19T07:21:18.841 INFO:tasks.workunit.client.0.smithi083.stderr:aclocal.m4:383: AM_CONFIG_HEADER is expanded from... 2024-09-19T07:21:18.841 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:6: the top level 2024-09-19T07:21:18.841 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:10: warning: The macro `AC_AIX' is obsolete. 2024-09-19T07:21:18.841 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:10: You should run autoupdate. 2024-09-19T07:21:18.841 INFO:tasks.workunit.client.0.smithi083.stderr:./lib/autoconf/specific.m4:540: AC_AIX is expanded from... 2024-09-19T07:21:18.842 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:10: the top level 2024-09-19T07:21:18.842 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:36: warning: The macro `AC_HEADER_STDC' is obsolete. 2024-09-19T07:21:18.842 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:36: You should run autoupdate. 2024-09-19T07:21:18.842 INFO:tasks.workunit.client.0.smithi083.stderr:./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from... 2024-09-19T07:21:18.842 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:36: the top level 2024-09-19T07:21:18.842 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:42: warning: The macro `AC_HEADER_TIME' is obsolete. 2024-09-19T07:21:18.842 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:42: You should run autoupdate. 2024-09-19T07:21:18.842 INFO:tasks.workunit.client.0.smithi083.stderr:./lib/autoconf/headers.m4:743: AC_HEADER_TIME is expanded from... 2024-09-19T07:21:18.842 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:42: the top level 2024-09-19T07:21:18.842 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:45: warning: The macro `AC_FUNC_SETVBUF_REVERSED' is obsolete. Remove it and all references to SETVBUF_REVERSED. 2024-09-19T07:21:18.842 INFO:tasks.workunit.client.0.smithi083.stderr:./lib/autoconf/functions.m4:1766: AC_FUNC_SETVBUF_REVERSED is expanded from... 2024-09-19T07:21:18.842 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:45: the top level 2024-09-19T07:21:18.843 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:50: warning: AC_OUTPUT should be used without arguments. 2024-09-19T07:21:18.843 INFO:tasks.workunit.client.0.smithi083.stderr:configure.in:50: You should run autoupdate. 2024-09-19T07:21:18.854 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-19T07:21:18.919 INFO:tasks.workunit.client.0.smithi083.stdout:/bin/bash ./config.status --recheck 2024-09-19T07:21:18.940 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-09-19T07:21:18.949 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-09-19T07:21:18.990 INFO:tasks.workunit.client.0.smithi083.stdout:running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion 2024-09-19T07:21:19.092 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-09-19T07:21:19.103 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-09-19T07:21:19.124 INFO:tasks.workunit.client.0.smithi083.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-19T07:21:19.155 INFO:tasks.workunit.client.1.smithi184.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-09-19T07:21:19.167 INFO:tasks.workunit.client.1.smithi184.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-09-19T07:21:19.320 INFO:tasks.workunit.client.1.smithi184.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb' 2024-09-19T07:21:19.320 INFO:tasks.workunit.client.1.smithi184.stderr:+ cd .. 2024-09-19T07:21:19.322 INFO:tasks.workunit.client.1.smithi184.stderr:+ mkdir tmp 2024-09-19T07:21:19.326 INFO:tasks.workunit.client.1.smithi184.stderr:+ cd tmp 2024-09-19T07:21:19.327 INFO:tasks.workunit.client.1.smithi184.stderr:+ for f in $mydir/*.ffsb 2024-09-19T07:21:19.327 INFO:tasks.workunit.client.1.smithi184.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/random_write.32.ffsb 2024-09-19T07:21:19.328 INFO:tasks.workunit.client.1.smithi184.stdout:FFSB version 6.0-RC2 started 2024-09-19T07:21:19.329 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:21:19.333 INFO:tasks.workunit.client.1.smithi184.stdout:benchmark time = 300 2024-09-19T07:21:19.333 INFO:tasks.workunit.client.1.smithi184.stdout:ThreadGroup 0 2024-09-19T07:21:19.333 INFO:tasks.workunit.client.1.smithi184.stdout:================ 2024-09-19T07:21:19.333 INFO:tasks.workunit.client.1.smithi184.stdout: num_threads = 32 2024-09-19T07:21:19.333 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:21:19.333 INFO:tasks.workunit.client.1.smithi184.stdout: read_random = off 2024-09-19T07:21:19.333 INFO:tasks.workunit.client.1.smithi184.stdout: read_size = 0 (0B) 2024-09-19T07:21:19.333 INFO:tasks.workunit.client.1.smithi184.stdout: read_blocksize = 0 (0B) 2024-09-19T07:21:19.333 INFO:tasks.workunit.client.1.smithi184.stdout: read_skip = off 2024-09-19T07:21:19.333 INFO:tasks.workunit.client.1.smithi184.stdout: read_skipsize = 0 (0B) 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: write_random = on 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: write_size = 5242880 (5MB) 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: fsync_file = 0 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: write_blocksize = 4096 (4KB) 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: wait time = 0 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: op weights 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: read = 0 (0.00%) 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: readall = 0 (0.00%) 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: write = 1 (100.00%) 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: create = 0 (0.00%) 2024-09-19T07:21:19.334 INFO:tasks.workunit.client.1.smithi184.stdout: append = 0 (0.00%) 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout: delete = 0 (0.00%) 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout: metaop = 0 (0.00%) 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout: createdir = 0 (0.00%) 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout: stat = 0 (0.00%) 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout: writeall = 0 (0.00%) 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout: writeall_fsync = 0 (0.00%) 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout: open_close = 0 (0.00%) 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout: write_fsync = 0 (0.00%) 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout: create_fsync = 0 (0.00%) 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout: append_fsync = 0 (0.00%) 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout:FileSystem . 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout:========== 2024-09-19T07:21:19.335 INFO:tasks.workunit.client.1.smithi184.stdout: num_dirs = 0 2024-09-19T07:21:19.336 INFO:tasks.workunit.client.1.smithi184.stdout: starting files = 128 2024-09-19T07:21:19.336 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:21:19.336 INFO:tasks.workunit.client.1.smithi184.stdout: min file size = 104857600 (100MB) 2024-09-19T07:21:19.336 INFO:tasks.workunit.client.1.smithi184.stdout: max file size = 104857600 (100MB) 2024-09-19T07:21:19.336 INFO:tasks.workunit.client.1.smithi184.stdout: directio = off 2024-09-19T07:21:19.336 INFO:tasks.workunit.client.1.smithi184.stdout: alignedio = on 2024-09-19T07:21:19.336 INFO:tasks.workunit.client.1.smithi184.stdout: bufferedio = off 2024-09-19T07:21:19.336 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:21:19.336 INFO:tasks.workunit.client.1.smithi184.stdout: aging is off 2024-09-19T07:21:19.336 INFO:tasks.workunit.client.1.smithi184.stdout: current utilization = 0.00% 2024-09-19T07:21:19.336 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:21:19.336 INFO:tasks.workunit.client.1.smithi184.stdout:checking existing fs: . 2024-09-19T07:21:19.337 INFO:tasks.workunit.client.1.smithi184.stderr:opendir: No such file or directory 2024-09-19T07:21:19.337 INFO:tasks.workunit.client.1.smithi184.stdout:recreating new fileset 2024-09-19T07:21:20.140 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether build environment is sane... yes 2024-09-19T07:21:20.153 INFO:tasks.workunit.client.0.smithi083.stdout:checking for a race-free mkdir -p... /usr/bin/mkdir -p 2024-09-19T07:21:20.154 INFO:tasks.workunit.client.0.smithi083.stdout:checking for gawk... gawk 2024-09-19T07:21:20.183 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether make sets $(MAKE)... yes 2024-09-19T07:21:20.235 INFO:tasks.workunit.client.0.smithi083.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-19T07:21:20.235 INFO:tasks.workunit.client.0.smithi083.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-19T07:21:20.255 INFO:tasks.workunit.client.0.smithi083.stdout:checking for style of include used by make... GNU 2024-09-19T07:21:20.262 INFO:tasks.workunit.client.0.smithi083.stdout:checking for gcc... gcc 2024-09-19T07:21:20.414 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether the C compiler works... yes 2024-09-19T07:21:20.414 INFO:tasks.workunit.client.0.smithi083.stdout:checking for C compiler default output file name... a.out 2024-09-19T07:21:20.474 INFO:tasks.workunit.client.0.smithi083.stdout:checking for suffix of executables... 2024-09-19T07:21:20.535 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether we are cross compiling... no 2024-09-19T07:21:20.582 INFO:tasks.workunit.client.0.smithi083.stdout:checking for suffix of object files... o 2024-09-19T07:21:20.613 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether the compiler supports GNU C... yes 2024-09-19T07:21:20.644 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether gcc accepts -g... yes 2024-09-19T07:21:20.707 INFO:tasks.workunit.client.0.smithi083.stdout:checking for gcc option to enable C11 features... none needed 2024-09-19T07:21:20.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:21:20.864 INFO:tasks.workunit.client.0.smithi083.stdout:checking dependency style of gcc... gcc3 2024-09-19T07:21:20.897 INFO:tasks.workunit.client.0.smithi083.stdout:checking for stdio.h... yes 2024-09-19T07:21:20.944 INFO:tasks.workunit.client.0.smithi083.stdout:checking for stdlib.h... yes 2024-09-19T07:21:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.980+0000 7f5f102d4640 1 Processor -- start 2024-09-19T07:21:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.980+0000 7f5f102d4640 1 -- start start 2024-09-19T07:21:20.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f102d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f08153300 0x7f5f0806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:20.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f102d4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f08155bb0 0x7f5f08156060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:20.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f102d4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f081593f0 con 0x7f5f0806d1d0 2024-09-19T07:21:20.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f102d4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5f081591a0 con 0x7f5f08153300 2024-09-19T07:21:20.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f102d4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5f081596d0 con 0x7f5f08155bb0 2024-09-19T07:21:20.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f0e070640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f08153300 0x7f5f0806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:20.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f0e070640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f08153300 0x7f5f0806cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60680/0 (socket says 172.21.15.83:60680) 2024-09-19T07:21:20.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f0e070640 1 -- 172.21.15.83:0/4217228921 learned_addr learned my addr 172.21.15.83:0/4217228921 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:20.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f0e871640 1 --2- 172.21.15.83:0/4217228921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f08155bb0 0x7f5f08156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:20.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f0e070640 1 -- 172.21.15.83:0/4217228921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f08155bb0 msgr2=0x7f5f08156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:21:20.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f0e070640 1 --2- 172.21.15.83:0/4217228921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f08155bb0 0x7f5f08156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:20.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f0e070640 1 -- 172.21.15.83:0/4217228921 >> v1:172.21.15.184:6789/0 conn(0x7f5f0806d1d0 legacy=0x7f5f080c23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:20.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f0e070640 1 -- 172.21.15.83:0/4217228921 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f081599b0 con 0x7f5f08153300 2024-09-19T07:21:20.989 INFO:tasks.workunit.client.0.smithi083.stdout:checking for string.h... yes 2024-09-19T07:21:20.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.984+0000 7f5f0e070640 1 --2- 172.21.15.83:0/4217228921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f08153300 0x7f5f0806cc90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5f04009870 tx=0x7f5f0401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c90b67f8f3c999 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:20.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f0d06e640 1 -- 172.21.15.83:0/4217228921 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f040b0070 con 0x7f5f08153300 2024-09-19T07:21:20.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f0d06e640 1 -- 172.21.15.83:0/4217228921 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f04004230 con 0x7f5f08153300 2024-09-19T07:21:20.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f0d06e640 1 -- 172.21.15.83:0/4217228921 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f04003620 con 0x7f5f08153300 2024-09-19T07:21:20.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f102d4640 1 -- 172.21.15.83:0/4217228921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f08153300 msgr2=0x7f5f0806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:20.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f102d4640 1 --2- 172.21.15.83:0/4217228921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f08153300 0x7f5f0806cc90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5f04009870 tx=0x7f5f0401c440 comp rx=0 tx=0).stop 2024-09-19T07:21:20.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f102d4640 1 -- 172.21.15.83:0/4217228921 shutdown_connections 2024-09-19T07:21:20.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f102d4640 1 --2- 172.21.15.83:0/4217228921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f08155bb0 0x7f5f08156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:20.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f102d4640 1 --2- 172.21.15.83:0/4217228921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f08153300 0x7f5f0806cc90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:20.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f102d4640 1 -- 172.21.15.83:0/4217228921 >> 172.21.15.83:0/4217228921 conn(0x7f5f080c5090 msgr2=0x7f5f080c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:20.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f102d4640 1 -- 172.21.15.83:0/4217228921 shutdown_connections 2024-09-19T07:21:20.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f102d4640 1 -- 172.21.15.83:0/4217228921 wait complete. 2024-09-19T07:21:20.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f102d4640 1 Processor -- start 2024-09-19T07:21:20.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.988+0000 7f5f102d4640 1 -- start start 2024-09-19T07:21:20.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f102d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0806d1d0 0x7f5f081ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:20.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f102d4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f08155bb0 0x7f5f08158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:20.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f102d4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f080751f0 0x7f5f080c3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:20.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f102d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5f08153860 con 0x7f5f0806d1d0 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f102d4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5f081536e0 con 0x7f5f08155bb0 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f102d4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5f081565a0 con 0x7f5f080751f0 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f0e070640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0806d1d0 0x7f5f081ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f0e070640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0806d1d0 0x7f5f081ee210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60686/0 (socket says 172.21.15.83:60686) 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f0e070640 1 -- 172.21.15.83:0/715498768 learned_addr learned my addr 172.21.15.83:0/715498768 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f0d86f640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f08155bb0 0x7f5f08158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f0e871640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f080751f0 0x7f5f080c3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f0e070640 1 -- 172.21.15.83:0/715498768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f080751f0 msgr2=0x7f5f080c3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f0e070640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f080751f0 0x7f5f080c3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f0e070640 1 -- 172.21.15.83:0/715498768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f08155bb0 msgr2=0x7f5f08158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f0e070640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f08155bb0 0x7f5f08158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f0e070640 1 -- 172.21.15.83:0/715498768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f081f2160 con 0x7f5f0806d1d0 2024-09-19T07:21:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f0e070640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0806d1d0 0x7f5f081ee210 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5f040ae040 tx=0x7f5f040191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:20.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5efeffd640 1 -- 172.21.15.83:0/715498768 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f040b9020 con 0x7f5f0806d1d0 2024-09-19T07:21:20.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5efeffd640 1 -- 172.21.15.83:0/715498768 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f040b1040 con 0x7f5f0806d1d0 2024-09-19T07:21:20.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5efeffd640 1 -- 172.21.15.83:0/715498768 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f040b0050 con 0x7f5f0806d1d0 2024-09-19T07:21:20.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f102d4640 1 -- 172.21.15.83:0/715498768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f08150770 con 0x7f5f0806d1d0 2024-09-19T07:21:20.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f102d4640 1 -- 172.21.15.83:0/715498768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f081570d0 con 0x7f5f0806d1d0 2024-09-19T07:21:20.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5f102d4640 1 -- 172.21.15.83:0/715498768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f0806cc90 con 0x7f5f0806d1d0 2024-09-19T07:21:20.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.992+0000 7f5efeffd640 1 -- 172.21.15.83:0/715498768 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5f040bc020 con 0x7f5f0806d1d0 2024-09-19T07:21:20.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.996+0000 7f5efeffd640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5eec0678c0 0x7f5eec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:21:20.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.996+0000 7f5efeffd640 1 -- 172.21.15.83:0/715498768 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7f5f0411fa90 con 0x7f5f0806d1d0 2024-09-19T07:21:20.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.996+0000 7f5f0d86f640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5eec0678c0 0x7f5eec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:21:20.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:20.996+0000 7f5f0d86f640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5eec0678c0 0x7f5eec069d80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5f0806f9d0 tx=0x7f5ef8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:21:21.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.000+0000 7f5efeffd640 1 -- 172.21.15.83:0/715498768 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5f040c3030 con 0x7f5f0806d1d0 2024-09-19T07:21:21.030 INFO:tasks.workunit.client.0.smithi083.stdout:checking for inttypes.h... yes 2024-09-19T07:21:21.070 INFO:tasks.workunit.client.0.smithi083.stdout:checking for stdint.h... yes 2024-09-19T07:21:21.113 INFO:tasks.workunit.client.0.smithi083.stdout:checking for strings.h... yes 2024-09-19T07:21:21.155 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/stat.h... yes 2024-09-19T07:21:21.198 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/types.h... yes 2024-09-19T07:21:21.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.220+0000 7f5f102d4640 1 -- 172.21.15.83:0/715498768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f0806f720 con 0x7f5f0806d1d0 2024-09-19T07:21:21.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.220+0000 7f5efeffd640 1 -- 172.21.15.83:0/715498768 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6539 (secure 0 0 0) 0x7f5f081f2160 con 0x7f5f0806d1d0 2024-09-19T07:21:21.226 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:21:21.226 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":23,"btime":"2024-09-19T07:21:18:974034+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5025,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:21:21.227 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 23 2024-09-19T07:21:21.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.224+0000 7f5f102d4640 1 -- 172.21.15.83:0/715498768 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5eec0678c0 msgr2=0x7f5eec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:21.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.224+0000 7f5f102d4640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5eec0678c0 0x7f5eec069d80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5f0806f9d0 tx=0x7f5ef8009040 comp rx=0 tx=0).stop 2024-09-19T07:21:21.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.224+0000 7f5f102d4640 1 -- 172.21.15.83:0/715498768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0806d1d0 msgr2=0x7f5f081ee210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:21:21.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.224+0000 7f5f102d4640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0806d1d0 0x7f5f081ee210 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5f040ae040 tx=0x7f5f040191e0 comp rx=0 tx=0).stop 2024-09-19T07:21:21.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.224+0000 7f5f102d4640 1 -- 172.21.15.83:0/715498768 shutdown_connections 2024-09-19T07:21:21.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.224+0000 7f5f102d4640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5eec0678c0 0x7f5eec069d80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:21.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.224+0000 7f5f102d4640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f080751f0 0x7f5f080c3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:21.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.224+0000 7f5f102d4640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f08155bb0 0x7f5f08158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:21.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.224+0000 7f5f102d4640 1 --2- 172.21.15.83:0/715498768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0806d1d0 0x7f5f081ee210 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:21:21.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.224+0000 7f5f102d4640 1 -- 172.21.15.83:0/715498768 >> 172.21.15.83:0/715498768 conn(0x7f5f080c5090 msgr2=0x7f5f0814f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:21:21.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.224+0000 7f5f102d4640 1 -- 172.21.15.83:0/715498768 shutdown_connections 2024-09-19T07:21:21.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:21:21.224+0000 7f5f102d4640 1 -- 172.21.15.83:0/715498768 wait complete. 2024-09-19T07:21:21.245 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-19T07:21:21.246 INFO:tasks.mds_thrash.fs.[a]:waiting for 56 secs before thrashing 2024-09-19T07:21:21.250 INFO:tasks.workunit.client.0.smithi083.stdout:checking for unistd.h... yes 2024-09-19T07:21:21.300 INFO:tasks.workunit.client.0.smithi083.stdout:checking for wchar.h... yes 2024-09-19T07:21:21.355 INFO:tasks.workunit.client.0.smithi083.stdout:checking for minix/config.h... no 2024-09-19T07:21:21.396 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/time.h... yes 2024-09-19T07:21:21.449 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-09-19T07:21:21.483 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether _XOPEN_SOURCE should be defined... no 2024-09-19T07:21:21.505 INFO:tasks.workunit.client.0.smithi083.stdout:checking for gcc... (cached) gcc 2024-09-19T07:21:21.566 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether the compiler supports GNU C... (cached) yes 2024-09-19T07:21:21.567 INFO:tasks.workunit.client.0.smithi083.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-19T07:21:21.567 INFO:tasks.workunit.client.0.smithi083.stdout:checking for gcc option to enable C11 features... (cached) none needed 2024-09-19T07:21:21.568 INFO:tasks.workunit.client.0.smithi083.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-19T07:21:21.625 INFO:tasks.workunit.client.0.smithi083.stdout:checking for main in -lm... yes 2024-09-19T07:21:21.688 INFO:tasks.workunit.client.0.smithi083.stdout:checking for main in -lpthread... yes 2024-09-19T07:21:21.692 INFO:tasks.workunit.client.0.smithi083.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-19T07:21:21.693 INFO:tasks.workunit.client.0.smithi083.stdout:checking for egrep... /usr/bin/grep -E 2024-09-19T07:21:21.733 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-19T07:21:21.784 INFO:tasks.workunit.client.0.smithi083.stdout:checking for pthread.h... yes 2024-09-19T07:21:21.830 INFO:tasks.workunit.client.0.smithi083.stdout:checking for fcntl.h... yes 2024-09-19T07:21:21.881 INFO:tasks.workunit.client.0.smithi083.stdout:checking for limits.h... yes 2024-09-19T07:21:21.881 INFO:tasks.workunit.client.0.smithi083.stdout:checking for stdint.h... (cached) yes 2024-09-19T07:21:21.881 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/time.h... (cached) yes 2024-09-19T07:21:21.882 INFO:tasks.workunit.client.0.smithi083.stdout:checking for unistd.h... (cached) yes 2024-09-19T07:21:21.930 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/vfs.h... yes 2024-09-19T07:21:21.992 INFO:tasks.workunit.client.0.smithi083.stdout:checking for sys/limits.h... no 2024-09-19T07:21:22.022 INFO:tasks.workunit.client.0.smithi083.stdout:checking for an ANSI C-conforming const... yes 2024-09-19T07:21:22.088 INFO:tasks.workunit.client.0.smithi083.stdout:checking for system... yes 2024-09-19T07:21:22.152 INFO:tasks.workunit.client.0.smithi083.stdout:checking for gettimeofday... yes 2024-09-19T07:21:22.213 INFO:tasks.workunit.client.0.smithi083.stdout:checking for mkdir... yes 2024-09-19T07:21:22.278 INFO:tasks.workunit.client.0.smithi083.stdout:checking for strerror... yes 2024-09-19T07:21:22.339 INFO:tasks.workunit.client.0.smithi083.stdout:checking for open64... yes 2024-09-19T07:21:22.403 INFO:tasks.workunit.client.0.smithi083.stdout:checking for stat64... yes 2024-09-19T07:21:22.465 INFO:tasks.workunit.client.0.smithi083.stdout:checking for fseeko64... yes 2024-09-19T07:21:22.531 INFO:tasks.workunit.client.0.smithi083.stdout:checking for lrand48_r... yes 2024-09-19T07:21:22.596 INFO:tasks.workunit.client.0.smithi083.stdout:checking for srand48_r... yes 2024-09-19T07:21:22.618 INFO:tasks.workunit.client.0.smithi083.stdout:configure: creating ./config.status 2024-09-19T07:21:22.725 INFO:tasks.workunit.client.0.smithi083.stdout: /bin/bash ./config.status 2024-09-19T07:21:22.785 INFO:tasks.workunit.client.0.smithi083.stdout:config.status: creating Makefile 2024-09-19T07:21:22.801 INFO:tasks.workunit.client.0.smithi083.stdout:config.status: creating config.h 2024-09-19T07:21:22.828 INFO:tasks.workunit.client.0.smithi083.stdout:config.status: executing depfiles commands 2024-09-19T07:21:22.846 INFO:tasks.workunit.client.0.smithi083.stdout:config.status: executing default commands 2024-09-19T07:21:22.898 INFO:tasks.workunit.client.0.smithi083.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-09-19T07:21:22.953 INFO:tasks.workunit.client.0.smithi083.stderr:autoheader: warning: autoconf input should be named 'configure.ac', not 'configure.in' 2024-09-19T07:21:23.032 INFO:tasks.workunit.client.0.smithi083.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-19T07:21:23.170 INFO:tasks.workunit.client.0.smithi083.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-19T07:21:23.170 INFO:tasks.workunit.client.0.smithi083.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-19T07:21:23.170 INFO:tasks.workunit.client.0.smithi083.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-19T07:21:23.171 INFO:tasks.workunit.client.0.smithi083.stdout:rm -f stamp-h1 2024-09-19T07:21:23.172 INFO:tasks.workunit.client.0.smithi083.stdout:touch config.h.in 2024-09-19T07:21:23.179 INFO:tasks.workunit.client.0.smithi083.stdout:cd . && /bin/bash ./config.status config.h 2024-09-19T07:21:23.232 INFO:tasks.workunit.client.0.smithi083.stdout:config.status: creating config.h 2024-09-19T07:21:23.268 INFO:tasks.workunit.client.0.smithi083.stdout:make all-am 2024-09-19T07:21:23.296 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-09-19T07:21:23.296 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-09-19T07:21:23.486 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-09-19T07:21:23.496 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-09-19T07:21:23.607 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-09-19T07:21:23.615 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-09-19T07:21:23.743 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-09-19T07:21:23.752 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-09-19T07:21:23.906 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-09-19T07:21:23.920 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-09-19T07:21:24.113 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-09-19T07:21:24.128 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-09-19T07:21:24.218 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-09-19T07:21:24.226 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-09-19T07:21:24.293 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-09-19T07:21:24.300 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-09-19T07:21:24.375 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-09-19T07:21:24.383 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-09-19T07:21:24.649 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-09-19T07:21:24.662 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-09-19T07:21:24.872 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-09-19T07:21:24.885 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-09-19T07:21:25.148 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-09-19T07:21:25.157 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-09-19T07:21:25.274 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-09-19T07:21:25.284 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-09-19T07:21:25.345 INFO:tasks.workunit.client.0.smithi083.stderr:ffsb_op.c: In function 'print_op_results': 2024-09-19T07:21:25.345 INFO:tasks.workunit.client.0.smithi083.stderr:ffsb_op.c:83:37: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-09-19T07:21:25.345 INFO:tasks.workunit.client.0.smithi083.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-09-19T07:21:25.345 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~ 2024-09-19T07:21:25.345 INFO:tasks.workunit.client.0.smithi083.stderr:ffsb_op.c:86:33: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-09-19T07:21:25.345 INFO:tasks.workunit.client.0.smithi083.stderr: 86 | sprintf(buf, "NA\0"); 2024-09-19T07:21:25.345 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~ 2024-09-19T07:21:25.365 INFO:tasks.workunit.client.0.smithi083.stderr:ffsb_op.c: In function 'print_results': 2024-09-19T07:21:25.365 INFO:tasks.workunit.client.0.smithi083.stderr:ffsb_op.c:83:33: warning: '/sec' directive writing 4 bytes into a region of size between 1 and 256 [-Wformat-overflow=] 2024-09-19T07:21:25.365 INFO:tasks.workunit.client.0.smithi083.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-09-19T07:21:25.366 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~ 2024-09-19T07:21:25.366 INFO:tasks.workunit.client.0.smithi083.stderr:In file included from /usr/include/stdio.h:894, 2024-09-19T07:21:25.366 INFO:tasks.workunit.client.0.smithi083.stderr: from ffsb_op.c:20: 2024-09-19T07:21:25.366 INFO:tasks.workunit.client.0.smithi083.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 5 and 260 bytes into a destination of size 256 2024-09-19T07:21:25.366 INFO:tasks.workunit.client.0.smithi083.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, 2024-09-19T07:21:25.366 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.366 INFO:tasks.workunit.client.0.smithi083.stderr: 39 | __glibc_objsize (__s), __fmt, 2024-09-19T07:21:25.366 INFO:tasks.workunit.client.0.smithi083.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.366 INFO:tasks.workunit.client.0.smithi083.stderr: 40 | __va_arg_pack ()); 2024-09-19T07:21:25.366 INFO:tasks.workunit.client.0.smithi083.stderr: | ~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.416 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-09-19T07:21:25.427 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-09-19T07:21:25.603 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-09-19T07:21:25.613 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-09-19T07:21:25.679 INFO:tasks.workunit.client.0.smithi083.stderr:In file included from parser.h:22, 2024-09-19T07:21:25.679 INFO:tasks.workunit.client.0.smithi083.stderr: from parser.c:25: 2024-09-19T07:21:25.679 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c: In function 'init_filesys': 2024-09-19T07:21:25.680 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:25.681 INFO:tasks.workunit.client.0.smithi083.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:25.681 INFO:tasks.workunit.client.0.smithi083.stderr: | ^ 2024-09-19T07:21:25.681 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:25.681 INFO:tasks.workunit.client.0.smithi083.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:25.681 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~ 2024-09-19T07:21:25.681 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:25.681 INFO:tasks.workunit.client.0.smithi083.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:25.681 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:25.681 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:25.681 INFO:tasks.workunit.client.0.smithi083.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-19T07:21:25.681 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:25.682 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:25.682 INFO:tasks.workunit.client.0.smithi083.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-19T07:21:25.682 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.683 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:25.683 INFO:tasks.workunit.client.0.smithi083.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:25.683 INFO:tasks.workunit.client.0.smithi083.stderr: | ^ 2024-09-19T07:21:25.683 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:25.684 INFO:tasks.workunit.client.0.smithi083.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:25.684 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~ 2024-09-19T07:21:25.684 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:25.684 INFO:tasks.workunit.client.0.smithi083.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:25.684 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:25.684 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:25.684 INFO:tasks.workunit.client.0.smithi083.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-19T07:21:25.684 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:25.685 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:25.685 INFO:tasks.workunit.client.0.smithi083.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-19T07:21:25.685 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.685 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:25.685 INFO:tasks.workunit.client.0.smithi083.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:25.685 INFO:tasks.workunit.client.0.smithi083.stderr: | ^ 2024-09-19T07:21:25.685 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:25.685 INFO:tasks.workunit.client.0.smithi083.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:25.685 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~ 2024-09-19T07:21:25.685 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:25.685 INFO:tasks.workunit.client.0.smithi083.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:25.685 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:25.686 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:25.686 INFO:tasks.workunit.client.0.smithi083.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-19T07:21:25.686 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:25.686 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:25.686 INFO:tasks.workunit.client.0.smithi083.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-19T07:21:25.686 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.686 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:25.686 INFO:tasks.workunit.client.0.smithi083.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:25.686 INFO:tasks.workunit.client.0.smithi083.stderr: | ^ 2024-09-19T07:21:25.686 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:25.687 INFO:tasks.workunit.client.0.smithi083.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:25.687 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~ 2024-09-19T07:21:25.687 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:25.687 INFO:tasks.workunit.client.0.smithi083.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:25.687 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:25.687 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:25.687 INFO:tasks.workunit.client.0.smithi083.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-19T07:21:25.687 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:25.687 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:25.687 INFO:tasks.workunit.client.0.smithi083.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-19T07:21:25.687 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.687 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c: In function 'init_tg_stats': 2024-09-19T07:21:25.688 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:25.688 INFO:tasks.workunit.client.0.smithi083.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:25.688 INFO:tasks.workunit.client.0.smithi083.stderr: | ^ 2024-09-19T07:21:25.688 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:25.688 INFO:tasks.workunit.client.0.smithi083.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:25.688 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~ 2024-09-19T07:21:25.688 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:25.688 INFO:tasks.workunit.client.0.smithi083.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:25.688 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:25.688 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:25.688 INFO:tasks.workunit.client.0.smithi083.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-19T07:21:25.689 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:25.689 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:25.689 INFO:tasks.workunit.client.0.smithi083.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-19T07:21:25.689 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.689 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:25.689 INFO:tasks.workunit.client.0.smithi083.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:25.689 INFO:tasks.workunit.client.0.smithi083.stderr: | ^ 2024-09-19T07:21:25.689 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:25.689 INFO:tasks.workunit.client.0.smithi083.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:25.689 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~ 2024-09-19T07:21:25.689 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:25.689 INFO:tasks.workunit.client.0.smithi083.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:25.690 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:25.690 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:25.690 INFO:tasks.workunit.client.0.smithi083.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-19T07:21:25.690 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:25.690 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:25.690 INFO:tasks.workunit.client.0.smithi083.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-19T07:21:25.690 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.690 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:25.690 INFO:tasks.workunit.client.0.smithi083.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:25.690 INFO:tasks.workunit.client.0.smithi083.stderr: | ^ 2024-09-19T07:21:25.690 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~ 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-19T07:21:25.691 INFO:tasks.workunit.client.0.smithi083.stderr: | ^ 2024-09-19T07:21:25.692 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-19T07:21:25.692 INFO:tasks.workunit.client.0.smithi083.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-19T07:21:25.692 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~ 2024-09-19T07:21:25.692 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-19T07:21:25.692 INFO:tasks.workunit.client.0.smithi083.stderr: 36 | container_of(ptr, type, member) 2024-09-19T07:21:25.692 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~ 2024-09-19T07:21:25.692 INFO:tasks.workunit.client.0.smithi083.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-19T07:21:25.692 INFO:tasks.workunit.client.0.smithi083.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-19T07:21:25.692 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~ 2024-09-19T07:21:25.693 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-09-19T07:21:25.693 INFO:tasks.workunit.client.0.smithi083.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-19T07:21:25.693 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.743 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c: In function 'get_optstr': 2024-09-19T07:21:25.743 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c:206:23: warning: 'strnlen' specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-09-19T07:21:25.743 INFO:tasks.workunit.client.0.smithi083.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-09-19T07:21:25.743 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.743 INFO:tasks.workunit.client.0.smithi083.stderr:parser.c:200:14: note: source object declared here 2024-09-19T07:21:25.743 INFO:tasks.workunit.client.0.smithi083.stderr: 200 | char temp[BUFSIZE]; 2024-09-19T07:21:25.744 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~ 2024-09-19T07:21:25.761 INFO:tasks.workunit.client.0.smithi083.stderr:In file included from /usr/include/string.h:535, 2024-09-19T07:21:25.761 INFO:tasks.workunit.client.0.smithi083.stderr: from parser.c:19: 2024-09-19T07:21:25.761 INFO:tasks.workunit.client.0.smithi083.stderr:In function 'memcpy', 2024-09-19T07:21:25.762 INFO:tasks.workunit.client.0.smithi083.stderr: inlined from 'size64_convert' at parser.c:119:2: 2024-09-19T07:21:25.762 INFO:tasks.workunit.client.0.smithi083.stderr:/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10: warning: '__builtin_memcpy' reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-09-19T07:21:25.762 INFO:tasks.workunit.client.0.smithi083.stderr: 29 | return __builtin___memcpy_chk (__dest, __src, __len, 2024-09-19T07:21:25.762 INFO:tasks.workunit.client.0.smithi083.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:25.762 INFO:tasks.workunit.client.0.smithi083.stderr: 30 | __glibc_objsize0 (__dest)); 2024-09-19T07:21:25.762 INFO:tasks.workunit.client.0.smithi083.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-19T07:21:26.109 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-09-19T07:21:26.119 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-09-19T07:21:26.239 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-09-19T07:21:26.250 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-09-19T07:21:26.388 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-09-19T07:21:26.397 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-09-19T07:21:26.447 INFO:tasks.workunit.client.0.smithi083.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-09-19T07:21:26.459 INFO:tasks.workunit.client.0.smithi083.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-09-19T07:21:26.618 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-09-19T07:21:26.619 INFO:tasks.workunit.client.0.smithi083.stderr:+ cd .. 2024-09-19T07:21:26.621 INFO:tasks.workunit.client.0.smithi083.stderr:+ mkdir tmp 2024-09-19T07:21:26.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ cd tmp 2024-09-19T07:21:26.625 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in $mydir/*.ffsb 2024-09-19T07:21:26.625 INFO:tasks.workunit.client.0.smithi083.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-09-19T07:21:26.626 INFO:tasks.workunit.client.0.smithi083.stdout:FFSB version 6.0-RC2 started 2024-09-19T07:21:26.626 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:21:26.630 INFO:tasks.workunit.client.0.smithi083.stdout:benchmark time = 300 2024-09-19T07:21:26.630 INFO:tasks.workunit.client.0.smithi083.stdout:ThreadGroup 0 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout:================ 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout: num_threads = 32 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout: read_random = off 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout: read_size = 0 (0B) 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout: read_blocksize = 0 (0B) 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout: read_skip = off 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout: read_skipsize = 0 (0B) 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout: write_random = on 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout: write_size = 5242880 (5MB) 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout: fsync_file = 0 2024-09-19T07:21:26.631 INFO:tasks.workunit.client.0.smithi083.stdout: write_blocksize = 4096 (4KB) 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: wait time = 0 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: op weights 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: read = 0 (0.00%) 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: readall = 0 (0.00%) 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: write = 1 (100.00%) 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: create = 0 (0.00%) 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: append = 0 (0.00%) 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: delete = 0 (0.00%) 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: metaop = 0 (0.00%) 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: createdir = 0 (0.00%) 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: stat = 0 (0.00%) 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: writeall = 0 (0.00%) 2024-09-19T07:21:26.632 INFO:tasks.workunit.client.0.smithi083.stdout: writeall_fsync = 0 (0.00%) 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout: open_close = 0 (0.00%) 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout: write_fsync = 0 (0.00%) 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout: create_fsync = 0 (0.00%) 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout: append_fsync = 0 (0.00%) 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout:FileSystem . 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout:========== 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout: num_dirs = 0 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout: starting files = 128 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout: min file size = 104857600 (100MB) 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout: max file size = 104857600 (100MB) 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout: directio = off 2024-09-19T07:21:26.633 INFO:tasks.workunit.client.0.smithi083.stdout: alignedio = on 2024-09-19T07:21:26.634 INFO:tasks.workunit.client.0.smithi083.stdout: bufferedio = off 2024-09-19T07:21:26.634 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:21:26.634 INFO:tasks.workunit.client.0.smithi083.stdout: aging is off 2024-09-19T07:21:26.634 INFO:tasks.workunit.client.0.smithi083.stdout: current utilization = 0.00% 2024-09-19T07:21:26.634 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:21:26.634 INFO:tasks.workunit.client.0.smithi083.stdout:checking existing fs: . 2024-09-19T07:21:26.635 INFO:tasks.workunit.client.0.smithi083.stdout:recreating new fileset 2024-09-19T07:21:26.635 INFO:tasks.workunit.client.0.smithi083.stderr:opendir: No such file or directory 2024-09-19T07:21:30.485 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:21:30.489 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:22:00.596 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:22:00.599 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:22:11.062 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:11.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8d13a640 1 Processor -- start 2024-09-19T07:22:11.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8d13a640 1 -- start start 2024-09-19T07:22:11.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8d13a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4c881535d0 0x7f4c880c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:11.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8d13a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4c8806ffe0 0x7f4c88155c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8d13a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c88156190 con 0x7f4c88159af0 2024-09-19T07:22:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8d13a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f4c880707e0 con 0x7f4c8806ffe0 2024-09-19T07:22:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8d13a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4c880c2c80 con 0x7f4c881535d0 2024-09-19T07:22:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4c8806ffe0 0x7f4c88155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8759e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4c88159af0 0x7f4c880c2f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56026/0 (socket says 172.21.15.83:56026) 2024-09-19T07:22:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8759e640 1 -- 172.21.15.83:0/2195587514 learned_addr learned my addr 172.21.15.83:0/2195587514 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c86d9d640 1 --2- 172.21.15.83:0/2195587514 >> v2:172.21.15.184:3300/0 conn(0x7f4c881535d0 0x7f4c880c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:11.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8659c640 1 -- 172.21.15.83:0/2195587514 >> v2:172.21.15.184:3300/0 conn(0x7f4c881535d0 msgr2=0x7f4c880c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:11.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8659c640 1 --2- 172.21.15.83:0/2195587514 >> v2:172.21.15.184:3300/0 conn(0x7f4c881535d0 0x7f4c880c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8659c640 1 -- 172.21.15.83:0/2195587514 >> v1:172.21.15.83:6789/0 conn(0x7f4c88159af0 legacy=0x7f4c880c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:11.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.207+0000 7f4c8659c640 1 -- 172.21.15.83:0/2195587514 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c880c26c0 con 0x7f4c8806ffe0 2024-09-19T07:22:11.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8659c640 1 --2- 172.21.15.83:0/2195587514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c8806ffe0 0x7f4c88155c50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4c880c2e00 tx=0x7f4c7c01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71887159a3734d12 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:11.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c85d9b640 1 -- 172.21.15.83:0/2195587514 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c0b0070 con 0x7f4c8806ffe0 2024-09-19T07:22:11.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c85d9b640 1 -- 172.21.15.83:0/2195587514 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c7c0ab600 con 0x7f4c8806ffe0 2024-09-19T07:22:11.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c85d9b640 1 -- 172.21.15.83:0/2195587514 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c0ab9d0 con 0x7f4c8806ffe0 2024-09-19T07:22:11.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 -- 172.21.15.83:0/2195587514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c8806ffe0 msgr2=0x7f4c88155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:11.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 --2- 172.21.15.83:0/2195587514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c8806ffe0 0x7f4c88155c50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4c880c2e00 tx=0x7f4c7c01c340 comp rx=0 tx=0).stop 2024-09-19T07:22:11.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 -- 172.21.15.83:0/2195587514 shutdown_connections 2024-09-19T07:22:11.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 --2- 172.21.15.83:0/2195587514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c8806ffe0 0x7f4c88155c50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 --2- 172.21.15.83:0/2195587514 >> v2:172.21.15.184:3300/0 conn(0x7f4c881535d0 0x7f4c880c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 -- 172.21.15.83:0/2195587514 >> 172.21.15.83:0/2195587514 conn(0x7f4c880c5060 msgr2=0x7f4c880c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:11.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 -- 172.21.15.83:0/2195587514 shutdown_connections 2024-09-19T07:22:11.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 -- 172.21.15.83:0/2195587514 wait complete. 2024-09-19T07:22:11.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 Processor -- start 2024-09-19T07:22:11.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 -- start start 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c8806ffe0 0x7f4c88157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c881535d0 0x7f4c8815dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c88159af0 0x7f4c8815e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4c880c37f0 con 0x7f4c88159af0 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4c880c3670 con 0x7f4c881535d0 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8d13a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4c88159f00 con 0x7f4c8806ffe0 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c88159af0 0x7f4c8815e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c88159af0 0x7f4c8815e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34090/0 (socket says 172.21.15.83:34090) 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8759e640 1 -- 172.21.15.83:0/885088360 learned_addr learned my addr 172.21.15.83:0/885088360 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c86d9d640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c8806ffe0 0x7f4c88157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8759e640 1 -- 172.21.15.83:0/885088360 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c8806ffe0 msgr2=0x7f4c88157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8759e640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c8806ffe0 0x7f4c88157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8759e640 1 -- 172.21.15.83:0/885088360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c881535d0 msgr2=0x7f4c8815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:11.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8759e640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c881535d0 0x7f4c8815dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.211+0000 7f4c8759e640 1 -- 172.21.15.83:0/885088360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c881eefd0 con 0x7f4c88159af0 2024-09-19T07:22:11.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.215+0000 7f4c8759e640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c88159af0 0x7f4c8815e220 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000cbc0 tx=0x7f4c8000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:11.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.215+0000 7f4c6f7fe640 1 -- 172.21.15.83:0/885088360 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c80017070 con 0x7f4c88159af0 2024-09-19T07:22:11.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.215+0000 7f4c6f7fe640 1 -- 172.21.15.83:0/885088360 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c8000ad40 con 0x7f4c88159af0 2024-09-19T07:22:11.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.215+0000 7f4c8d13a640 1 -- 172.21.15.83:0/885088360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c881ef2b0 con 0x7f4c88159af0 2024-09-19T07:22:11.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.215+0000 7f4c6f7fe640 1 -- 172.21.15.83:0/885088360 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c80012600 con 0x7f4c88159af0 2024-09-19T07:22:11.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.215+0000 7f4c8d13a640 1 -- 172.21.15.83:0/885088360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c8815ee60 con 0x7f4c88159af0 2024-09-19T07:22:11.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.215+0000 7f4c8d13a640 1 -- 172.21.15.83:0/885088360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c88155c50 con 0x7f4c88159af0 2024-09-19T07:22:11.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.215+0000 7f4c6f7fe640 1 -- 172.21.15.83:0/885088360 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4c8001a070 con 0x7f4c88159af0 2024-09-19T07:22:11.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.215+0000 7f4c6f7fe640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c740678c0 0x7f4c74069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:11.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.215+0000 7f4c6f7fe640 1 -- 172.21.15.83:0/885088360 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7f4c80085cb0 con 0x7f4c88159af0 2024-09-19T07:22:11.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.215+0000 7f4c86d9d640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c740678c0 0x7f4c74069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:11.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.215+0000 7f4c86d9d640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c740678c0 0x7f4c74069d80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800b9d0 tx=0x7f4c78008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:11.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.219+0000 7f4c6f7fe640 1 -- 172.21.15.83:0/885088360 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4c80055de0 con 0x7f4c88159af0 2024-09-19T07:22:11.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.431+0000 7f4c8d13a640 1 -- 172.21.15.83:0/885088360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c880adaf0 con 0x7f4c88159af0 2024-09-19T07:22:11.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.431+0000 7f4c6f7fe640 1 -- 172.21.15.83:0/885088360 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6539 (secure 0 0 0) 0x7f4c80085070 con 0x7f4c88159af0 2024-09-19T07:22:11.432 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:11.432 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":23,"btime":"2024-09-19T07:21:18:974034+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5025,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:22:11.433 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 23 2024-09-19T07:22:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.435+0000 7f4c8d13a640 1 -- 172.21.15.83:0/885088360 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c740678c0 msgr2=0x7f4c74069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.435+0000 7f4c8d13a640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c740678c0 0x7f4c74069d80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800b9d0 tx=0x7f4c78008040 comp rx=0 tx=0).stop 2024-09-19T07:22:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.435+0000 7f4c8d13a640 1 -- 172.21.15.83:0/885088360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c88159af0 msgr2=0x7f4c8815e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.435+0000 7f4c8d13a640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c88159af0 0x7f4c8815e220 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000cbc0 tx=0x7f4c8000a430 comp rx=0 tx=0).stop 2024-09-19T07:22:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.435+0000 7f4c8d13a640 1 -- 172.21.15.83:0/885088360 shutdown_connections 2024-09-19T07:22:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.435+0000 7f4c8d13a640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c740678c0 0x7f4c74069d80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.435+0000 7f4c8d13a640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c88159af0 0x7f4c8815e220 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.435+0000 7f4c8d13a640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c881535d0 0x7f4c8815dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.435+0000 7f4c8d13a640 1 --2- 172.21.15.83:0/885088360 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c8806ffe0 0x7f4c88157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.435+0000 7f4c8d13a640 1 -- 172.21.15.83:0/885088360 >> 172.21.15.83:0/885088360 conn(0x7f4c880c5060 msgr2=0x7f4c880c2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:11.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.435+0000 7f4c8d13a640 1 -- 172.21.15.83:0/885088360 shutdown_connections 2024-09-19T07:22:11.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.435+0000 7f4c8d13a640 1 -- 172.21.15.83:0/885088360 wait complete. 2024-09-19T07:22:11.453 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-19T07:22:11.453 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-19T07:22:11.453 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:22:11.476 INFO:tasks.ceph.mds.d:Stopped 2024-09-19T07:22:11.476 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:11.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.595+0000 7f33de9e0640 1 Processor -- start 2024-09-19T07:22:11.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.595+0000 7f33de9e0640 1 -- start start 2024-09-19T07:22:11.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.595+0000 7f33de9e0640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f33d8153490 0x7f33d8159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:11.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.595+0000 7f33de9e0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33d80c89f0 0x7f33d81565e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:11.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.595+0000 7f33de9e0640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33d80cac10 con 0x7f33d80c8580 2024-09-19T07:22:11.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.595+0000 7f33de9e0640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f33d806db30 con 0x7f33d8153490 2024-09-19T07:22:11.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.595+0000 7f33de9e0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f33d815a1f0 con 0x7f33d80c89f0 2024-09-19T07:22:11.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.595+0000 7f33cffff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f33d80c8580 0x7f33d8155ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:35158/0 (socket says 172.21.15.83:35158) 2024-09-19T07:22:11.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.595+0000 7f33cffff640 1 -- 172.21.15.83:0/480832322 learned_addr learned my addr 172.21.15.83:0/480832322 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:11.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.595+0000 7f33dcf7d640 1 --2- 172.21.15.83:0/480832322 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33d80c89f0 0x7f33d81565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:11.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.595+0000 7f33d7fff640 1 --2- 172.21.15.83:0/480832322 >> v2:172.21.15.184:3300/0 conn(0x7f33d8153490 0x7f33d8159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:11.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33d7fff640 1 -- 172.21.15.83:0/480832322 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33d80c89f0 msgr2=0x7f33d81565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:11.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33d7fff640 1 --2- 172.21.15.83:0/480832322 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33d80c89f0 0x7f33d81565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33d7fff640 1 -- 172.21.15.83:0/480832322 >> v1:172.21.15.184:6789/0 conn(0x7f33d80c8580 legacy=0x7f33d8155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:11.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33d7fff640 1 -- 172.21.15.83:0/480832322 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d80caeb0 con 0x7f33d8153490 2024-09-19T07:22:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33d7fff640 1 --2- 172.21.15.83:0/480832322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33d8153490 0x7f33d8159cb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f33c80097c0 tx=0x7f33c801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=347fd04314853584 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:11.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33d77fe640 1 -- 172.21.15.83:0/480832322 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c80b0070 con 0x7f33d8153490 2024-09-19T07:22:11.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33d77fe640 1 -- 172.21.15.83:0/480832322 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c80ab600 con 0x7f33d8153490 2024-09-19T07:22:11.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33d77fe640 1 -- 172.21.15.83:0/480832322 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c80ab9d0 con 0x7f33d8153490 2024-09-19T07:22:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33de9e0640 1 -- 172.21.15.83:0/480832322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33d8153490 msgr2=0x7f33d8159cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33de9e0640 1 --2- 172.21.15.83:0/480832322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33d8153490 0x7f33d8159cb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f33c80097c0 tx=0x7f33c801c410 comp rx=0 tx=0).stop 2024-09-19T07:22:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33de9e0640 1 -- 172.21.15.83:0/480832322 shutdown_connections 2024-09-19T07:22:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33de9e0640 1 --2- 172.21.15.83:0/480832322 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33d80c89f0 0x7f33d81565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33de9e0640 1 --2- 172.21.15.83:0/480832322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33d8153490 0x7f33d8159cb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33de9e0640 1 -- 172.21.15.83:0/480832322 >> 172.21.15.83:0/480832322 conn(0x7f33d80ad7d0 msgr2=0x7f33d80adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33de9e0640 1 -- 172.21.15.83:0/480832322 shutdown_connections 2024-09-19T07:22:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.599+0000 7f33de9e0640 1 -- 172.21.15.83:0/480832322 wait complete. 2024-09-19T07:22:11.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33de9e0640 1 Processor -- start 2024-09-19T07:22:11.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33de9e0640 1 -- start start 2024-09-19T07:22:11.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33de9e0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33d80c8580 0x7f33d80af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:11.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33de9e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33d80c89f0 0x7f33d80af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:11.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33de9e0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33d8153490 0x7f33d8147bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:11.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33de9e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f33d8153870 con 0x7f33d80c89f0 2024-09-19T07:22:11.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33de9e0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f33d8156b20 con 0x7f33d8153490 2024-09-19T07:22:11.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33de9e0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f33d806bc80 con 0x7f33d80c8580 2024-09-19T07:22:11.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33cffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33d80c89f0 0x7f33d80af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:11.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33cffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33d80c89f0 0x7f33d80af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34106/0 (socket says 172.21.15.83:34106) 2024-09-19T07:22:11.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33cffff640 1 -- 172.21.15.83:0/1754037022 learned_addr learned my addr 172.21.15.83:0/1754037022 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:11.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33dcf7d640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33d8153490 0x7f33d8147bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:11.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33d7fff640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33d80c8580 0x7f33d80af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:11.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33cffff640 1 -- 172.21.15.83:0/1754037022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33d80c8580 msgr2=0x7f33d80af0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:11.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33cffff640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33d80c8580 0x7f33d80af0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33cffff640 1 -- 172.21.15.83:0/1754037022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33d8153490 msgr2=0x7f33d8147bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:11.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33cffff640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33d8153490 0x7f33d8147bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33cffff640 1 -- 172.21.15.83:0/1754037022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d8159020 con 0x7f33d80c89f0 2024-09-19T07:22:11.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33cffff640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33d80c89f0 0x7f33d80af9c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f33c000c370 tx=0x7f33c000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:11.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33d57fa640 1 -- 172.21.15.83:0/1754037022 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c0017070 con 0x7f33d80c89f0 2024-09-19T07:22:11.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33d57fa640 1 -- 172.21.15.83:0/1754037022 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c0005cf0 con 0x7f33d80c89f0 2024-09-19T07:22:11.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33de9e0640 1 -- 172.21.15.83:0/1754037022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d8158d40 con 0x7f33d80c89f0 2024-09-19T07:22:11.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33d57fa640 1 -- 172.21.15.83:0/1754037022 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c00072f0 con 0x7f33d80c89f0 2024-09-19T07:22:11.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33de9e0640 1 -- 172.21.15.83:0/1754037022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d80ae8d0 con 0x7f33d80c89f0 2024-09-19T07:22:11.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.603+0000 7f33de9e0640 1 -- 172.21.15.83:0/1754037022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33d8159cb0 con 0x7f33d80c89f0 2024-09-19T07:22:11.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.607+0000 7f33d57fa640 1 -- 172.21.15.83:0/1754037022 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f33c001a070 con 0x7f33d80c89f0 2024-09-19T07:22:11.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.607+0000 7f33d57fa640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f33b80678c0 0x7f33b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:11.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.607+0000 7f33d57fa640 1 -- 172.21.15.83:0/1754037022 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7f33c0085360 con 0x7f33d80c89f0 2024-09-19T07:22:11.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.607+0000 7f33d7fff640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f33b80678c0 0x7f33b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:11.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.607+0000 7f33d7fff640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f33b80678c0 0x7f33b8069d80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f33c80ad000 tx=0x7f33c80a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:11.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.611+0000 7f33d57fa640 1 -- 172.21.15.83:0/1754037022 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f33c0055410 con 0x7f33d80c89f0 2024-09-19T07:22:11.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 -- 172.21.15.83:0/1754037022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33d80ca9f0 con 0x7f33d80c89f0 2024-09-19T07:22:11.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33d57fa640 1 -- 172.21.15.83:0/1754037022 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6539 (secure 0 0 0) 0x7f33c0059710 con 0x7f33d80c89f0 2024-09-19T07:22:11.851 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:11.851 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":23,"btime":"2024-09-19T07:21:18:974034+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5025,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:22:11.851 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 23 2024-09-19T07:22:11.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 -- 172.21.15.83:0/1754037022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f33b80678c0 msgr2=0x7f33b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:11.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f33b80678c0 0x7f33b8069d80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f33c80ad000 tx=0x7f33c80a6000 comp rx=0 tx=0).stop 2024-09-19T07:22:11.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 -- 172.21.15.83:0/1754037022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33d80c89f0 msgr2=0x7f33d80af9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:11.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33d80c89f0 0x7f33d80af9c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f33c000c370 tx=0x7f33c000c830 comp rx=0 tx=0).stop 2024-09-19T07:22:11.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 -- 172.21.15.83:0/1754037022 shutdown_connections 2024-09-19T07:22:11.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f33b80678c0 0x7f33b8069d80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33d8153490 0x7f33d8147bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33d80c89f0 0x7f33d80af9c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 --2- 172.21.15.83:0/1754037022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33d80c8580 0x7f33d80af0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:11.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 -- 172.21.15.83:0/1754037022 >> 172.21.15.83:0/1754037022 conn(0x7f33d80ad7d0 msgr2=0x7f33d8150b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:11.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 -- 172.21.15.83:0/1754037022 shutdown_connections 2024-09-19T07:22:11.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:11.851+0000 7f33de9e0640 1 -- 172.21.15.83:0/1754037022 wait complete. 2024-09-19T07:22:11.869 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:22:13.870 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:14.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66d5d50640 1 Processor -- start 2024-09-19T07:22:14.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66d5d50640 1 -- start start 2024-09-19T07:22:14.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66d5d50640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f66d01494a0 0x7f66d006e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:14.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66d5d50640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66d0155fa0 con 0x7f66d0153390 2024-09-19T07:22:14.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66d5d50640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66d00c3850 con 0x7f66d0068700 2024-09-19T07:22:14.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66d5d50640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f66d0156e40 con 0x7f66d01494a0 2024-09-19T07:22:14.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66cffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f66d0153390 0x7f66d006d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56038/0 (socket says 172.21.15.83:56038) 2024-09-19T07:22:14.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66cffff640 1 -- 172.21.15.83:0/1974547268 learned_addr learned my addr 172.21.15.83:0/1974547268 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:14.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66cf7fe640 1 --2- 172.21.15.83:0/1974547268 >> v2:172.21.15.184:3300/0 conn(0x7f66d01494a0 0x7f66d006e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:14.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1125160631 0 0) 0x7f66d0155fa0 con 0x7f66d0153390 2024-09-19T07:22:14.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66bc003530 con 0x7f66d0153390 2024-09-19T07:22:14.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3708765141 0 0) 0x7f66bc003530 con 0x7f66d0153390 2024-09-19T07:22:14.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 >> v2:172.21.15.184:3300/0 conn(0x7f66d01494a0 msgr2=0x7f66d006e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:14.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 --2- 172.21.15.83:0/1974547268 >> v2:172.21.15.184:3300/0 conn(0x7f66d01494a0 0x7f66d006e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:14.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 >> v1:172.21.15.184:6789/0 conn(0x7f66d0068700 legacy=0x7f66d0156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:14.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d00c3b10 con 0x7f66d0153390 2024-09-19T07:22:14.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f66c8001a00 con 0x7f66d0153390 2024-09-19T07:22:14.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 >> v1:172.21.15.83:6789/0 conn(0x7f66d0153390 legacy=0x7f66d006d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:14.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 --2- 172.21.15.83:0/1974547268 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66bc0039c0 0x7f66bc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:14.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 --2- 172.21.15.83:0/1974547268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 0x7f66bc004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:14.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 --2- 172.21.15.83:0/1974547268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66bc005090 0x7f66bc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:14.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f66d0156e40 con 0x7f66bc0043c0 2024-09-19T07:22:14.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f66bc00bff0 con 0x7f66bc005090 2024-09-19T07:22:14.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f66bc00c1f0 con 0x7f66bc0039c0 2024-09-19T07:22:14.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66c7fff640 1 --2- 172.21.15.83:0/1974547268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 0x7f66bc004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:14.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66cffff640 1 --2- 172.21.15.83:0/1974547268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66bc005090 0x7f66bc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:14.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66cf7fe640 1 --2- 172.21.15.83:0/1974547268 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66bc0039c0 0x7f66bc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:14.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66c7fff640 1 -- 172.21.15.83:0/1974547268 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66bc0039c0 msgr2=0x7f66bc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:14.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66c7fff640 1 --2- 172.21.15.83:0/1974547268 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66bc0039c0 0x7f66bc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:14.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66c7fff640 1 -- 172.21.15.83:0/1974547268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66bc005090 msgr2=0x7f66bc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:14.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66c7fff640 1 --2- 172.21.15.83:0/1974547268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66bc005090 0x7f66bc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:14.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66c7fff640 1 -- 172.21.15.83:0/1974547268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66bc003530 con 0x7f66bc0043c0 2024-09-19T07:22:14.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.019+0000 7f66cffff640 1 -- 172.21.15.83:0/1974547268 reap_dead start 2024-09-19T07:22:14.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.023+0000 7f66c7fff640 1 --2- 172.21.15.83:0/1974547268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 0x7f66bc004890 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f66b8001860 tx=0x7f66b801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa18b878d18db7ff server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:14.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.023+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b80b1070 con 0x7f66bc0043c0 2024-09-19T07:22:14.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.023+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b80ac640 con 0x7f66bc0043c0 2024-09-19T07:22:14.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.023+0000 7f66ceffd640 1 -- 172.21.15.83:0/1974547268 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8001460 con 0x7f66bc0043c0 2024-09-19T07:22:14.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.023+0000 7f66d5d50640 1 -- 172.21.15.83:0/1974547268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 msgr2=0x7f66bc004890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:14.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.023+0000 7f66d5d50640 1 --2- 172.21.15.83:0/1974547268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 0x7f66bc004890 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f66b8001860 tx=0x7f66b801c4e0 comp rx=0 tx=0).stop 2024-09-19T07:22:14.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.023+0000 7f66d5d50640 1 -- 172.21.15.83:0/1974547268 shutdown_connections 2024-09-19T07:22:14.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.023+0000 7f66d5d50640 1 --2- 172.21.15.83:0/1974547268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 0x7f66bc004890 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:14.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.023+0000 7f66d5d50640 1 -- 172.21.15.83:0/1974547268 >> 172.21.15.83:0/1974547268 conn(0x7f66d00c4940 msgr2=0x7f66d00c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:14.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.023+0000 7f66d5d50640 1 -- 172.21.15.83:0/1974547268 shutdown_connections 2024-09-19T07:22:14.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.023+0000 7f66d5d50640 1 -- 172.21.15.83:0/1974547268 wait complete. 2024-09-19T07:22:14.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66d5d50640 1 Processor -- start 2024-09-19T07:22:14.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66d5d50640 1 -- start start 2024-09-19T07:22:14.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66d5d50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 0x7f66d01499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:14.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66d5d50640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66d014fcc0 0x7f66d00c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:14.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66d5d50640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66d01e0930 0x7f66d0148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:14.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66d5d50640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f66d01537a0 con 0x7f66bc0043c0 2024-09-19T07:22:14.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66d5d50640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f66d006dc80 con 0x7f66d014fcc0 2024-09-19T07:22:14.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66d5d50640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f66d00c40d0 con 0x7f66d01e0930 2024-09-19T07:22:14.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66cf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 0x7f66d01499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:14.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66cf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 0x7f66d01499a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34134/0 (socket says 172.21.15.83:34134) 2024-09-19T07:22:14.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66cf7fe640 1 -- 172.21.15.83:0/2394985386 learned_addr learned my addr 172.21.15.83:0/2394985386 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:14.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66cffff640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66d01e0930 0x7f66d0148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:14.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66c7fff640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66d014fcc0 0x7f66d00c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:14.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66cf7fe640 1 -- 172.21.15.83:0/2394985386 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66d01e0930 msgr2=0x7f66d0148580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:14.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66cf7fe640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66d01e0930 0x7f66d0148580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:14.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66cf7fe640 1 -- 172.21.15.83:0/2394985386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66d014fcc0 msgr2=0x7f66d00c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:14.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66cf7fe640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66d014fcc0 0x7f66d00c6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:14.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66cf7fe640 1 -- 172.21.15.83:0/2394985386 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d01502c0 con 0x7f66bc0043c0 2024-09-19T07:22:14.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66cffff640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66d01e0930 0x7f66d0148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:22:14.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66cf7fe640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 0x7f66d01499a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f66c0011660 tx=0x7f66c0011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:14.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66ccff9640 1 -- 172.21.15.83:0/2394985386 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c000f9c0 con 0x7f66bc0043c0 2024-09-19T07:22:14.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66ccff9640 1 -- 172.21.15.83:0/2394985386 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66c000fb60 con 0x7f66bc0043c0 2024-09-19T07:22:14.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66d5d50640 1 -- 172.21.15.83:0/2394985386 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66d014ca20 con 0x7f66bc0043c0 2024-09-19T07:22:14.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66ccff9640 1 -- 172.21.15.83:0/2394985386 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0015460 con 0x7f66bc0043c0 2024-09-19T07:22:14.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.027+0000 7f66d5d50640 1 -- 172.21.15.83:0/2394985386 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66d014acc0 con 0x7f66bc0043c0 2024-09-19T07:22:14.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.031+0000 7f66d5d50640 1 -- 172.21.15.83:0/2394985386 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66d00c5a30 con 0x7f66bc0043c0 2024-09-19T07:22:14.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.031+0000 7f66ccff9640 1 -- 172.21.15.83:0/2394985386 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f66c0015600 con 0x7f66bc0043c0 2024-09-19T07:22:14.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.031+0000 7f66ccff9640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66b00678c0 0x7f66b0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:14.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.031+0000 7f66ccff9640 1 -- 172.21.15.83:0/2394985386 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7f66c0086b40 con 0x7f66bc0043c0 2024-09-19T07:22:14.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.031+0000 7f66c7fff640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66b00678c0 0x7f66b0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:14.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.031+0000 7f66c7fff640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66b00678c0 0x7f66b0069d80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f66b80ae000 tx=0x7f66b80a6070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:14.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.035+0000 7f66ccff9640 1 -- 172.21.15.83:0/2394985386 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f66d01502c0 con 0x7f66bc0043c0 2024-09-19T07:22:14.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.275+0000 7f66d5d50640 1 -- 172.21.15.83:0/2394985386 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66d0070060 con 0x7f66bc0043c0 2024-09-19T07:22:14.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.275+0000 7f66ccff9640 1 -- 172.21.15.83:0/2394985386 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6539 (secure 0 0 0) 0x7f66c0019030 con 0x7f66bc0043c0 2024-09-19T07:22:14.278 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:14.278 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":23,"btime":"2024-09-19T07:21:18:974034+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5025,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:22:14.278 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 23 2024-09-19T07:22:14.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.279+0000 7f66d5d50640 1 -- 172.21.15.83:0/2394985386 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66b00678c0 msgr2=0x7f66b0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:14.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.279+0000 7f66d5d50640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66b00678c0 0x7f66b0069d80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f66b80ae000 tx=0x7f66b80a6070 comp rx=0 tx=0).stop 2024-09-19T07:22:14.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.279+0000 7f66d5d50640 1 -- 172.21.15.83:0/2394985386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 msgr2=0x7f66d01499a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:14.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.279+0000 7f66d5d50640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 0x7f66d01499a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f66c0011660 tx=0x7f66c0011b20 comp rx=0 tx=0).stop 2024-09-19T07:22:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.279+0000 7f66d5d50640 1 -- 172.21.15.83:0/2394985386 shutdown_connections 2024-09-19T07:22:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.279+0000 7f66d5d50640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66b00678c0 0x7f66b0069d80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.279+0000 7f66d5d50640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66d01e0930 0x7f66d0148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.279+0000 7f66d5d50640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66d014fcc0 0x7f66d00c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.279+0000 7f66d5d50640 1 --2- 172.21.15.83:0/2394985386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66bc0043c0 0x7f66d01499a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.279+0000 7f66d5d50640 1 -- 172.21.15.83:0/2394985386 >> 172.21.15.83:0/2394985386 conn(0x7f66d00c4940 msgr2=0x7f66d01540e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.279+0000 7f66d5d50640 1 -- 172.21.15.83:0/2394985386 shutdown_connections 2024-09-19T07:22:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:14.279+0000 7f66d5d50640 1 -- 172.21.15.83:0/2394985386 wait complete. 2024-09-19T07:22:14.298 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:22:16.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:16.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.439+0000 7feb852dd640 1 Processor -- start 2024-09-19T07:22:16.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.439+0000 7feb852dd640 1 -- start start 2024-09-19T07:22:16.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb852dd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb800c28e0 0x7feb801545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:16.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb852dd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb800a9e60 con 0x7feb8006dea0 2024-09-19T07:22:16.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb852dd640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb800aa140 con 0x7feb80071100 2024-09-19T07:22:16.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb852dd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feb80071b50 con 0x7feb800c28e0 2024-09-19T07:22:16.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7feb8006dea0 0x7feb8006e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56040/0 (socket says 172.21.15.83:56040) 2024-09-19T07:22:16.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7f7fe640 1 -- 172.21.15.83:0/2308139036 learned_addr learned my addr 172.21.15.83:0/2308139036 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:16.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2190628498 0 0) 0x7feb800a9e60 con 0x7feb8006dea0 2024-09-19T07:22:16.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb68003530 con 0x7feb8006dea0 2024-09-19T07:22:16.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1001680653 0 0) 0x7feb68003530 con 0x7feb8006dea0 2024-09-19T07:22:16.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb800c28e0 msgr2=0x7feb801545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:16.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 --2- 172.21.15.83:0/2308139036 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb800c28e0 0x7feb801545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:16.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 >> v1:172.21.15.184:6789/0 conn(0x7feb80071100 legacy=0x7feb80150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:16.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb800c2240 con 0x7feb8006dea0 2024-09-19T07:22:16.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7feb78001a00 con 0x7feb8006dea0 2024-09-19T07:22:16.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 >> v1:172.21.15.83:6789/0 conn(0x7feb8006dea0 legacy=0x7feb8006e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:16.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 --2- 172.21.15.83:0/2308139036 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb800c28e0 0x7feb680039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:16.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 --2- 172.21.15.83:0/2308139036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb68004100 0x7feb68004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:16.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 --2- 172.21.15.83:0/2308139036 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb68004e70 0x7feb6800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:16.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feb80071b50 con 0x7feb68004100 2024-09-19T07:22:16.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feb680055f0 con 0x7feb68004e70 2024-09-19T07:22:16.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feb6800bf10 con 0x7feb800c28e0 2024-09-19T07:22:16.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7e7fc640 1 --2- 172.21.15.83:0/2308139036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb68004100 0x7feb68004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:16.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7effd640 1 --2- 172.21.15.83:0/2308139036 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb800c28e0 0x7feb680039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:16.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7e7fc640 1 -- 172.21.15.83:0/2308139036 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb800c28e0 msgr2=0x7feb680039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:16.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7e7fc640 1 --2- 172.21.15.83:0/2308139036 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb800c28e0 0x7feb680039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:16.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7e7fc640 1 -- 172.21.15.83:0/2308139036 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb68004e70 msgr2=0x7feb6800b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:22:16.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7e7fc640 1 --2- 172.21.15.83:0/2308139036 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb68004e70 0x7feb6800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:16.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7e7fc640 1 -- 172.21.15.83:0/2308139036 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb68003530 con 0x7feb68004100 2024-09-19T07:22:16.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.443+0000 7feb7effd640 1 --2- 172.21.15.83:0/2308139036 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb800c28e0 0x7feb680039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:16.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7e7fc640 1 --2- 172.21.15.83:0/2308139036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb68004100 0x7feb68004810 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7feb74009860 tx=0x7feb7401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f57b0a58eca81b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:16.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb740b0070 con 0x7feb68004100 2024-09-19T07:22:16.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb740ab620 con 0x7feb68004100 2024-09-19T07:22:16.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7dffb640 1 -- 172.21.15.83:0/2308139036 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb74009460 con 0x7feb68004100 2024-09-19T07:22:16.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 -- 172.21.15.83:0/2308139036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb68004100 msgr2=0x7feb68004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:16.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 --2- 172.21.15.83:0/2308139036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb68004100 0x7feb68004810 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7feb74009860 tx=0x7feb7401c340 comp rx=0 tx=0).stop 2024-09-19T07:22:16.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7f7fe640 1 -- 172.21.15.83:0/2308139036 reap_dead start 2024-09-19T07:22:16.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 -- 172.21.15.83:0/2308139036 shutdown_connections 2024-09-19T07:22:16.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 -- 172.21.15.83:0/2308139036 >> 172.21.15.83:0/2308139036 conn(0x7feb8006bc80 msgr2=0x7feb8006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:16.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 -- 172.21.15.83:0/2308139036 shutdown_connections 2024-09-19T07:22:16.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 -- 172.21.15.83:0/2308139036 wait complete. 2024-09-19T07:22:16.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 Processor -- start 2024-09-19T07:22:16.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 -- start start 2024-09-19T07:22:16.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb68004100 0x7feb801ef8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:16.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb80157180 0x7feb801ecd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb8014ffc0 0x7feb80150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feb801517d0 con 0x7feb80157180 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feb80151650 con 0x7feb68004100 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb852dd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feb800714e0 con 0x7feb8014ffc0 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb80157180 0x7feb801ecd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb80157180 0x7feb801ecd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34148/0 (socket says 172.21.15.83:34148) 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7e7fc640 1 -- 172.21.15.83:0/3127815859 learned_addr learned my addr 172.21.15.83:0/3127815859 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7f7fe640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb8014ffc0 0x7feb80150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7effd640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb68004100 0x7feb801ef8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7e7fc640 1 -- 172.21.15.83:0/3127815859 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb8014ffc0 msgr2=0x7feb80150440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7e7fc640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb8014ffc0 0x7feb80150440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7e7fc640 1 -- 172.21.15.83:0/3127815859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb68004100 msgr2=0x7feb801ef8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:16.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7e7fc640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb68004100 0x7feb801ef8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:16.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7e7fc640 1 -- 172.21.15.83:0/3127815859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb800c6f20 con 0x7feb80157180 2024-09-19T07:22:16.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.447+0000 7feb7e7fc640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb80157180 0x7feb801ecd20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7feb74000c10 tx=0x7feb68005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:16.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.451+0000 7feb677fe640 1 -- 172.21.15.83:0/3127815859 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb740bb070 con 0x7feb80157180 2024-09-19T07:22:16.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.451+0000 7feb677fe640 1 -- 172.21.15.83:0/3127815859 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb740b7460 con 0x7feb80157180 2024-09-19T07:22:16.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.451+0000 7feb852dd640 1 -- 172.21.15.83:0/3127815859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb801ec520 con 0x7feb80157180 2024-09-19T07:22:16.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.451+0000 7feb677fe640 1 -- 172.21.15.83:0/3127815859 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb740b0040 con 0x7feb80157180 2024-09-19T07:22:16.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.451+0000 7feb852dd640 1 -- 172.21.15.83:0/3127815859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb801ed260 con 0x7feb80157180 2024-09-19T07:22:16.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.451+0000 7feb852dd640 1 -- 172.21.15.83:0/3127815859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb8006e350 con 0x7feb80157180 2024-09-19T07:22:16.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.451+0000 7feb677fe640 1 -- 172.21.15.83:0/3127815859 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feb740be070 con 0x7feb80157180 2024-09-19T07:22:16.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.451+0000 7feb677fe640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb6c067910 0x7feb6c069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:16.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.451+0000 7feb7effd640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb6c067910 0x7feb6c069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:16.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.451+0000 7feb677fe640 1 -- 172.21.15.83:0/3127815859 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7feb741208b0 con 0x7feb80157180 2024-09-19T07:22:16.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.451+0000 7feb7effd640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb6c067910 0x7feb6c069dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feb70001760 tx=0x7feb70009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:16.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.455+0000 7feb677fe640 1 -- 172.21.15.83:0/3127815859 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7feb74016050 con 0x7feb80157180 2024-09-19T07:22:16.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.691+0000 7feb852dd640 1 -- 172.21.15.83:0/3127815859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb800c4f30 con 0x7feb80157180 2024-09-19T07:22:16.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.691+0000 7feb677fe640 1 -- 172.21.15.83:0/3127815859 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6539 (secure 0 0 0) 0x7feb740f0ac0 con 0x7feb80157180 2024-09-19T07:22:16.692 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:16.693 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":23,"btime":"2024-09-19T07:21:18:974034+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5025,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:22:16.693 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 23 2024-09-19T07:22:16.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.695+0000 7feb852dd640 1 -- 172.21.15.83:0/3127815859 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb6c067910 msgr2=0x7feb6c069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:16.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.695+0000 7feb852dd640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb6c067910 0x7feb6c069dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feb70001760 tx=0x7feb70009040 comp rx=0 tx=0).stop 2024-09-19T07:22:16.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.695+0000 7feb852dd640 1 -- 172.21.15.83:0/3127815859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb80157180 msgr2=0x7feb801ecd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:16.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.695+0000 7feb852dd640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb80157180 0x7feb801ecd20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7feb74000c10 tx=0x7feb68005840 comp rx=0 tx=0).stop 2024-09-19T07:22:16.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.695+0000 7feb852dd640 1 -- 172.21.15.83:0/3127815859 shutdown_connections 2024-09-19T07:22:16.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.695+0000 7feb852dd640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb6c067910 0x7feb6c069dd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:16.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.695+0000 7feb852dd640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb8014ffc0 0x7feb80150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:16.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.695+0000 7feb852dd640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb80157180 0x7feb801ecd20 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:16.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.695+0000 7feb852dd640 1 --2- 172.21.15.83:0/3127815859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb68004100 0x7feb801ef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:16.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.695+0000 7feb852dd640 1 -- 172.21.15.83:0/3127815859 >> 172.21.15.83:0/3127815859 conn(0x7feb8006bc80 msgr2=0x7feb80073030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:16.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.695+0000 7feb852dd640 1 -- 172.21.15.83:0/3127815859 shutdown_connections 2024-09-19T07:22:16.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:16.695+0000 7feb852dd640 1 -- 172.21.15.83:0/3127815859 wait complete. 2024-09-19T07:22:16.712 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:22:17.247 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f5048657640 1 Processor -- start 2024-09-19T07:22:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f5048657640 1 -- start start 2024-09-19T07:22:17.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f5048657640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5040155e10 0x7f5040156280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f5048657640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5040159430 con 0x7f5040152f30 2024-09-19T07:22:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f5048657640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5040159710 con 0x7f50400c4880 2024-09-19T07:22:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f5048657640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f504015a040 con 0x7f5040155e10 2024-09-19T07:22:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f5045bf2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5040152f30 0x7f50401533a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56048/0 (socket says 172.21.15.83:56048) 2024-09-19T07:22:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f5045bf2640 1 -- 172.21.15.83:0/2032325451 learned_addr learned my addr 172.21.15.83:0/2032325451 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4114832884 0 0) 0x7f5040159430 con 0x7f5040152f30 2024-09-19T07:22:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f502c003530 con 0x7f5040152f30 2024-09-19T07:22:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3582367493 0 0) 0x7f502c003530 con 0x7f5040152f30 2024-09-19T07:22:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 >> v2:172.21.15.184:3300/0 conn(0x7f5040155e10 msgr2=0x7f5040156280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:22:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f50453f1640 1 --2- 172.21.15.83:0/2032325451 >> v2:172.21.15.184:3300/0 conn(0x7f5040155e10 0x7f5040156280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 >> v1:172.21.15.184:6789/0 conn(0x7f50400c4880 legacy=0x7f50400c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50401599f0 con 0x7f5040152f30 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f5030001680 con 0x7f5040152f30 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.371+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 >> v1:172.21.15.83:6789/0 conn(0x7f5040152f30 legacy=0x7f50401533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50453f1640 1 --2- 172.21.15.83:0/2032325451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f502c0039c0 0x7f502c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50453f1640 1 --2- 172.21.15.83:0/2032325451 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f502c004810 0x7f502c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50453f1640 1 --2- 172.21.15.83:0/2032325451 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f502c0053d0 0x7f502c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f504015a040 con 0x7f502c0039c0 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f502c00c310 con 0x7f502c004810 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f502c00c510 con 0x7f502c0053d0 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50463f3640 1 --2- 172.21.15.83:0/2032325451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f502c0039c0 0x7f502c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50463f3640 1 -- 172.21.15.83:0/2032325451 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f502c0053d0 msgr2=0x7f502c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50463f3640 1 --2- 172.21.15.83:0/2032325451 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f502c0053d0 0x7f502c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50463f3640 1 -- 172.21.15.83:0/2032325451 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f502c004810 msgr2=0x7f502c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50463f3640 1 --2- 172.21.15.83:0/2032325451 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f502c004810 0x7f502c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50463f3640 1 -- 172.21.15.83:0/2032325451 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502c003530 con 0x7f502c0039c0 2024-09-19T07:22:17.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f5046bf4640 1 -- 172.21.15.83:0/2032325451 reap_dead start 2024-09-19T07:22:17.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50463f3640 1 --2- 172.21.15.83:0/2032325451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f502c0039c0 0x7f502c003dc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f503c001d50 tx=0x7f503c01c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bafd22b495f17451 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:17.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c001950 con 0x7f502c0039c0 2024-09-19T07:22:17.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503c003ba0 con 0x7f502c0039c0 2024-09-19T07:22:17.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f50453f1640 1 -- 172.21.15.83:0/2032325451 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c005070 con 0x7f502c0039c0 2024-09-19T07:22:17.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f5048657640 1 -- 172.21.15.83:0/2032325451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f502c0039c0 msgr2=0x7f502c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:17.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.375+0000 7f5048657640 1 --2- 172.21.15.83:0/2032325451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f502c0039c0 0x7f502c003dc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f503c001d50 tx=0x7f503c01c660 comp rx=0 tx=0).stop 2024-09-19T07:22:17.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 -- 172.21.15.83:0/2032325451 shutdown_connections 2024-09-19T07:22:17.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 --2- 172.21.15.83:0/2032325451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f502c0039c0 0x7f502c003dc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 -- 172.21.15.83:0/2032325451 >> 172.21.15.83:0/2032325451 conn(0x7f50400aa910 msgr2=0x7f50400c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:17.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 -- 172.21.15.83:0/2032325451 shutdown_connections 2024-09-19T07:22:17.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 -- 172.21.15.83:0/2032325451 wait complete. 2024-09-19T07:22:17.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 Processor -- start 2024-09-19T07:22:17.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 -- start start 2024-09-19T07:22:17.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f502c0039c0 0x7f50400c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f504006cf50 0x7f5040154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50401ee5f0 0x7f50401eea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5040156240 con 0x7f50401ee5f0 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50400c4e70 con 0x7f502c0039c0 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f50401563c0 con 0x7f504006cf50 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5046bf4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50401ee5f0 0x7f50401eea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5046bf4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50401ee5f0 0x7f50401eea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34154/0 (socket says 172.21.15.83:34154) 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5046bf4640 1 -- 172.21.15.83:0/1924924184 learned_addr learned my addr 172.21.15.83:0/1924924184 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f50463f3640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f502c0039c0 0x7f50400c7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5045bf2640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f504006cf50 0x7f5040154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5046bf4640 1 -- 172.21.15.83:0/1924924184 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f504006cf50 msgr2=0x7f5040154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5046bf4640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f504006cf50 0x7f5040154310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5046bf4640 1 -- 172.21.15.83:0/1924924184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f502c0039c0 msgr2=0x7f50400c7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5046bf4640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f502c0039c0 0x7f50400c7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5046bf4640 1 -- 172.21.15.83:0/1924924184 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50400c8220 con 0x7f50401ee5f0 2024-09-19T07:22:17.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f50463f3640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f502c0039c0 0x7f50400c7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:17.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5046bf4640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50401ee5f0 0x7f50401eea80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f50400c6100 tx=0x7f5038001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:17.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5036ffd640 1 -- 172.21.15.83:0/1924924184 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5038005b10 con 0x7f50401ee5f0 2024-09-19T07:22:17.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5036ffd640 1 -- 172.21.15.83:0/1924924184 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5038005cb0 con 0x7f50401ee5f0 2024-09-19T07:22:17.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 -- 172.21.15.83:0/1924924184 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50401f2a30 con 0x7f50401ee5f0 2024-09-19T07:22:17.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5036ffd640 1 -- 172.21.15.83:0/1924924184 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5038005e60 con 0x7f50401ee5f0 2024-09-19T07:22:17.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 -- 172.21.15.83:0/1924924184 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50401f1f50 con 0x7f50401ee5f0 2024-09-19T07:22:17.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.379+0000 7f5048657640 1 -- 172.21.15.83:0/1924924184 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5040156750 con 0x7f50401ee5f0 2024-09-19T07:22:17.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.383+0000 7f5036ffd640 1 -- 172.21.15.83:0/1924924184 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5038011070 con 0x7f50401ee5f0 2024-09-19T07:22:17.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.383+0000 7f5036ffd640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50240678f0 0x7f5024069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.383+0000 7f50463f3640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50240678f0 0x7f5024069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:17.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.383+0000 7f5036ffd640 1 -- 172.21.15.83:0/1924924184 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7f5038085610 con 0x7f50401ee5f0 2024-09-19T07:22:17.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.383+0000 7f50463f3640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50240678f0 0x7f5024069db0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f503c0ab000 tx=0x7f503c005040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:17.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.387+0000 7f5036ffd640 1 -- 172.21.15.83:0/1924924184 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f503801a060 con 0x7f50401ee5f0 2024-09-19T07:22:17.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.615+0000 7f5048657640 1 -- 172.21.15.83:0/1924924184 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5040154850 con 0x7f50401ee5f0 2024-09-19T07:22:17.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.615+0000 7f5036ffd640 1 -- 172.21.15.83:0/1924924184 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6539 (secure 0 0 0) 0x7f5038055e70 con 0x7f50401ee5f0 2024-09-19T07:22:17.616 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:17.616 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":23,"btime":"2024-09-19T07:21:18:974034+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5025,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:22:17.616 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 23 2024-09-19T07:22:17.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.615+0000 7f5048657640 1 -- 172.21.15.83:0/1924924184 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50240678f0 msgr2=0x7f5024069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:17.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.615+0000 7f5048657640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50240678f0 0x7f5024069db0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f503c0ab000 tx=0x7f503c005040 comp rx=0 tx=0).stop 2024-09-19T07:22:17.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.615+0000 7f5048657640 1 -- 172.21.15.83:0/1924924184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50401ee5f0 msgr2=0x7f50401eea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:17.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.615+0000 7f5048657640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50401ee5f0 0x7f50401eea80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f50400c6100 tx=0x7f5038001770 comp rx=0 tx=0).stop 2024-09-19T07:22:17.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.615+0000 7f5048657640 1 -- 172.21.15.83:0/1924924184 shutdown_connections 2024-09-19T07:22:17.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.615+0000 7f5048657640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50240678f0 0x7f5024069db0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.615+0000 7f5048657640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50401ee5f0 0x7f50401eea80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.615+0000 7f5048657640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f504006cf50 0x7f5040154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.615+0000 7f5048657640 1 --2- 172.21.15.83:0/1924924184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f502c0039c0 0x7f50400c7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.615+0000 7f5048657640 1 -- 172.21.15.83:0/1924924184 >> 172.21.15.83:0/1924924184 conn(0x7f50400aa910 msgr2=0x7f50401552d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:17.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.619+0000 7f5048657640 1 -- 172.21.15.83:0/1924924184 shutdown_connections 2024-09-19T07:22:17.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.619+0000 7f5048657640 1 -- 172.21.15.83:0/1924924184 wait complete. 2024-09-19T07:22:17.637 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-19T07:22:17.637 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:22:17.637 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:22:17.648 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:22:17.648 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:17.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.807+0000 7fc6de30d640 1 Processor -- start 2024-09-19T07:22:17.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.807+0000 7fc6de30d640 1 -- start start 2024-09-19T07:22:17.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6de30d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc6d8151170 0x7fc6d8153a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6de30d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6d80ada90 0x7fc6d8073020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6de30d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc6d8073560 0x7fc6d806e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6de30d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc6d80adf70 con 0x7fc6d8151170 2024-09-19T07:22:17.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6de30d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc6d80ae250 con 0x7fc6d8073560 2024-09-19T07:22:17.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6de30d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d80ae530 con 0x7fc6d80ada90 2024-09-19T07:22:17.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6d7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc6d8151170 0x7fc6d8153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:17.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6d7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc6d8151170 0x7fc6d8153a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34162/0 (socket says 172.21.15.83:34162) 2024-09-19T07:22:17.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6d7fff640 1 -- 172.21.15.83:0/2307670286 learned_addr learned my addr 172.21.15.83:0/2307670286 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:17.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6d7fff640 1 -- 172.21.15.83:0/2307670286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6d80ada90 msgr2=0x7fc6d8073020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:17.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6d7fff640 1 --2- 172.21.15.83:0/2307670286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6d80ada90 0x7fc6d8073020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6dc8aa640 1 --2- 172.21.15.83:0/2307670286 >> v2:172.21.15.184:3300/0 conn(0x7fc6d8073560 0x7fc6d806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:17.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6d7fff640 1 -- 172.21.15.83:0/2307670286 >> v2:172.21.15.184:3300/0 conn(0x7fc6d8073560 msgr2=0x7fc6d806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:17.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6d7fff640 1 --2- 172.21.15.83:0/2307670286 >> v2:172.21.15.184:3300/0 conn(0x7fc6d8073560 0x7fc6d806e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6d7fff640 1 -- 172.21.15.83:0/2307670286 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d80ae810 con 0x7fc6d8151170 2024-09-19T07:22:17.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6dc8aa640 1 --2- 172.21.15.83:0/2307670286 >> v2:172.21.15.184:3300/0 conn(0x7fc6d8073560 0x7fc6d806e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:17.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6d7fff640 1 --2- 172.21.15.83:0/2307670286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d8151170 0x7fc6d8153a60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80740e0 tx=0x7fc6c801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fc89e641949c89 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:17.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6d77fe640 1 -- 172.21.15.83:0/2307670286 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80b2070 con 0x7fc6d8151170 2024-09-19T07:22:17.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6d77fe640 1 -- 172.21.15.83:0/2307670286 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6c80ad680 con 0x7fc6d8151170 2024-09-19T07:22:17.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.811+0000 7fc6d77fe640 1 -- 172.21.15.83:0/2307670286 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80ada50 con 0x7fc6d8151170 2024-09-19T07:22:17.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 -- 172.21.15.83:0/2307670286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d8151170 msgr2=0x7fc6d8153a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:17.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 --2- 172.21.15.83:0/2307670286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d8151170 0x7fc6d8153a60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80740e0 tx=0x7fc6c801c4b0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 -- 172.21.15.83:0/2307670286 shutdown_connections 2024-09-19T07:22:17.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 --2- 172.21.15.83:0/2307670286 >> v2:172.21.15.184:3300/0 conn(0x7fc6d8073560 0x7fc6d806e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 --2- 172.21.15.83:0/2307670286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6d80ada90 0x7fc6d8073020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 --2- 172.21.15.83:0/2307670286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d8151170 0x7fc6d8153a60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 -- 172.21.15.83:0/2307670286 >> 172.21.15.83:0/2307670286 conn(0x7fc6d806fff0 msgr2=0x7fc6d8070400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:17.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 -- 172.21.15.83:0/2307670286 shutdown_connections 2024-09-19T07:22:17.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 -- 172.21.15.83:0/2307670286 wait complete. 2024-09-19T07:22:17.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 Processor -- start 2024-09-19T07:22:17.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 -- start start 2024-09-19T07:22:17.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6d8073560 0x7fc6d80a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6d80ada90 0x7fc6d8153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d8151170 0x7fc6d81ecaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d81573c0 con 0x7fc6d8151170 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d8157240 con 0x7fc6d8073560 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6de30d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d8157540 con 0x7fc6d80ada90 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6dc8aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d8151170 0x7fc6d81ecaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6dc8aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d8151170 0x7fc6d81ecaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34176/0 (socket says 172.21.15.83:34176) 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6dc8aa640 1 -- 172.21.15.83:0/1516377956 learned_addr learned my addr 172.21.15.83:0/1516377956 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6cffff640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6d80ada90 0x7fc6d8153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6dc8aa640 1 -- 172.21.15.83:0/1516377956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6d80ada90 msgr2=0x7fc6d8153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6dc8aa640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6d80ada90 0x7fc6d8153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6dc8aa640 1 -- 172.21.15.83:0/1516377956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6d8073560 msgr2=0x7fc6d80a9a20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6dc8aa640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6d8073560 0x7fc6d80a9a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.815+0000 7fc6dc8aa640 1 -- 172.21.15.83:0/1516377956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d806f710 con 0x7fc6d8151170 2024-09-19T07:22:17.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.819+0000 7fc6dc8aa640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d8151170 0x7fc6d81ecaf0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000ce40 tx=0x7fc6d000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:17.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.819+0000 7fc6d57fa640 1 -- 172.21.15.83:0/1516377956 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d001a070 con 0x7fc6d8151170 2024-09-19T07:22:17.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.819+0000 7fc6d57fa640 1 -- 172.21.15.83:0/1516377956 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d000d840 con 0x7fc6d8151170 2024-09-19T07:22:17.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.819+0000 7fc6de30d640 1 -- 172.21.15.83:0/1516377956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d80a9f60 con 0x7fc6d8151170 2024-09-19T07:22:17.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.819+0000 7fc6d57fa640 1 -- 172.21.15.83:0/1516377956 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d000dc30 con 0x7fc6d8151170 2024-09-19T07:22:17.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.819+0000 7fc6de30d640 1 -- 172.21.15.83:0/1516377956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d80ab700 con 0x7fc6d8151170 2024-09-19T07:22:17.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.819+0000 7fc6de30d640 1 -- 172.21.15.83:0/1516377956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d80ab8d0 con 0x7fc6d8151170 2024-09-19T07:22:17.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.819+0000 7fc6d57fa640 1 -- 172.21.15.83:0/1516377956 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc6d001d070 con 0x7fc6d8151170 2024-09-19T07:22:17.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.819+0000 7fc6d57fa640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6b80678c0 0x7fc6b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:17.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.819+0000 7fc6d57fa640 1 -- 172.21.15.83:0/1516377956 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7fc6d0086be0 con 0x7fc6d8151170 2024-09-19T07:22:17.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.819+0000 7fc6d7fff640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6b80678c0 0x7fc6b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:17.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.823+0000 7fc6d7fff640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6b80678c0 0x7fc6b8069d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80af000 tx=0x7fc6c8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:17.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:17.823+0000 7fc6d57fa640 1 -- 172.21.15.83:0/1516377956 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc6d0024020 con 0x7fc6d8151170 2024-09-19T07:22:18.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 -- 172.21.15.83:0/1516377956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6d806e040 con 0x7fc6d8151170 2024-09-19T07:22:18.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6d57fa640 1 -- 172.21.15.83:0/1516377956 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6539 (secure 0 0 0) 0x7fc6d0057440 con 0x7fc6d8151170 2024-09-19T07:22:18.111 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:18.111 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":23,"btime":"2024-09-19T07:21:18:974034+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5025,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:22:18.111 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 23 2024-09-19T07:22:18.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 -- 172.21.15.83:0/1516377956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6b80678c0 msgr2=0x7fc6b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:18.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6b80678c0 0x7fc6b8069d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80af000 tx=0x7fc6c8008040 comp rx=0 tx=0).stop 2024-09-19T07:22:18.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 -- 172.21.15.83:0/1516377956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d8151170 msgr2=0x7fc6d81ecaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:18.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d8151170 0x7fc6d81ecaf0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000ce40 tx=0x7fc6d000a710 comp rx=0 tx=0).stop 2024-09-19T07:22:18.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 -- 172.21.15.83:0/1516377956 shutdown_connections 2024-09-19T07:22:18.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6b80678c0 0x7fc6b8069d80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:18.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6d8151170 0x7fc6d81ecaf0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:18.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6d80ada90 0x7fc6d8153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:18.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 --2- 172.21.15.83:0/1516377956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6d8073560 0x7fc6d80a9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:18.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 -- 172.21.15.83:0/1516377956 >> 172.21.15.83:0/1516377956 conn(0x7fc6d806fff0 msgr2=0x7fc6d814c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:18.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 -- 172.21.15.83:0/1516377956 shutdown_connections 2024-09-19T07:22:18.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.111+0000 7fc6de30d640 1 -- 172.21.15.83:0/1516377956 wait complete. 2024-09-19T07:22:18.130 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:22:18.714 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:18.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f522853d640 1 Processor -- start 2024-09-19T07:22:18.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f522853d640 1 -- start start 2024-09-19T07:22:18.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f522853d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f52201535d0 0x7f52200c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:18.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f522853d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5220159af0 0x7f52200c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:18.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f522853d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52200c26c0 con 0x7f522006ffe0 2024-09-19T07:22:18.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f522853d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f52200707e0 con 0x7f52201535d0 2024-09-19T07:22:18.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f522853d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f52200c2c80 con 0x7f5220159af0 2024-09-19T07:22:18.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f5225ad8640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f522006ffe0 0x7f5220155c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56064/0 (socket says 172.21.15.83:56064) 2024-09-19T07:22:18.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f5225ad8640 1 -- 172.21.15.83:0/3976927756 learned_addr learned my addr 172.21.15.83:0/3976927756 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:18.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 613111305 0 0) 0x7f52200c26c0 con 0x7f522006ffe0 2024-09-19T07:22:18.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f520c003530 con 0x7f522006ffe0 2024-09-19T07:22:18.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f5226ada640 1 --2- 172.21.15.83:0/3976927756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5220159af0 0x7f52200c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:18.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52262d9640 1 --2- 172.21.15.83:0/3976927756 >> v2:172.21.15.184:3300/0 conn(0x7f52201535d0 0x7f52200c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:18.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 64969028 0 0) 0x7f520c003530 con 0x7f522006ffe0 2024-09-19T07:22:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5220159af0 msgr2=0x7f52200c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 --2- 172.21.15.83:0/3976927756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5220159af0 0x7f52200c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 >> v2:172.21.15.184:3300/0 conn(0x7f52201535d0 msgr2=0x7f52200c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 --2- 172.21.15.83:0/3976927756 >> v2:172.21.15.184:3300/0 conn(0x7f52201535d0 0x7f52200c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52200c29a0 con 0x7f522006ffe0 2024-09-19T07:22:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f5210001680 con 0x7f522006ffe0 2024-09-19T07:22:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52262d9640 1 --2- 172.21.15.83:0/3976927756 >> v2:172.21.15.184:3300/0 conn(0x7f52201535d0 0x7f52200c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:22:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 >> v1:172.21.15.83:6789/0 conn(0x7f522006ffe0 legacy=0x7f5220155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 --2- 172.21.15.83:0/3976927756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f520c0039c0 0x7f520c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 --2- 172.21.15.83:0/3976927756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5220159af0 0x7f520c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 --2- 172.21.15.83:0/3976927756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f520c004d50 0x7f520c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f52200c2c80 con 0x7f520c0039c0 2024-09-19T07:22:18.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f520c00bb60 con 0x7f520c004d50 2024-09-19T07:22:18.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f520c00bd60 con 0x7f5220159af0 2024-09-19T07:22:18.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52262d9640 1 --2- 172.21.15.83:0/3976927756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f520c0039c0 0x7f520c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:18.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f5226ada640 1 --2- 172.21.15.83:0/3976927756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f520c004d50 0x7f520c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:18.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52262d9640 1 -- 172.21.15.83:0/3976927756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5220159af0 msgr2=0x7f520c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:18.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52262d9640 1 --2- 172.21.15.83:0/3976927756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5220159af0 0x7f520c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:18.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52262d9640 1 -- 172.21.15.83:0/3976927756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f520c004d50 msgr2=0x7f520c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:18.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52262d9640 1 --2- 172.21.15.83:0/3976927756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f520c004d50 0x7f520c0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:18.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.855+0000 7f52262d9640 1 -- 172.21.15.83:0/3976927756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520c003530 con 0x7f520c0039c0 2024-09-19T07:22:18.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.859+0000 7f52262d9640 1 --2- 172.21.15.83:0/3976927756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f520c0039c0 0x7f520c003de0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f521c0098d0 tx=0x7f521c01c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16bf01acd1125bfe server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:18.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.859+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c00f040 con 0x7f520c0039c0 2024-09-19T07:22:18.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.859+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f521c0a9450 con 0x7f520c0039c0 2024-09-19T07:22:18.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.859+0000 7f52252d7640 1 -- 172.21.15.83:0/3976927756 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c0a9800 con 0x7f520c0039c0 2024-09-19T07:22:18.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.859+0000 7f522853d640 1 -- 172.21.15.83:0/3976927756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f520c0039c0 msgr2=0x7f520c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:18.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.859+0000 7f522853d640 1 --2- 172.21.15.83:0/3976927756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f520c0039c0 0x7f520c003de0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f521c0098d0 tx=0x7f521c01c650 comp rx=0 tx=0).stop 2024-09-19T07:22:18.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.859+0000 7f5226ada640 1 -- 172.21.15.83:0/3976927756 reap_dead start 2024-09-19T07:22:18.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.859+0000 7f522853d640 1 -- 172.21.15.83:0/3976927756 shutdown_connections 2024-09-19T07:22:18.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.859+0000 7f522853d640 1 -- 172.21.15.83:0/3976927756 >> 172.21.15.83:0/3976927756 conn(0x7f52200c5060 msgr2=0x7f52200c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:18.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.859+0000 7f522853d640 1 -- 172.21.15.83:0/3976927756 shutdown_connections 2024-09-19T07:22:18.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.859+0000 7f522853d640 1 -- 172.21.15.83:0/3976927756 wait complete. 2024-09-19T07:22:18.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f522853d640 1 Processor -- start 2024-09-19T07:22:18.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f522853d640 1 -- start start 2024-09-19T07:22:18.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f522853d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f520c0039c0 0x7f52201f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:18.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f522853d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5220152a60 0x7f522006f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:18.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f522853d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52201ee710 0x7f52201eeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:18.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f522853d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5220160670 con 0x7f520c0039c0 2024-09-19T07:22:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f522853d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f52201604f0 con 0x7f5220152a60 2024-09-19T07:22:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f522853d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f52201607f0 con 0x7f52201ee710 2024-09-19T07:22:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f52262d9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f520c0039c0 0x7f52201f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f52262d9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f520c0039c0 0x7f52201f2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34194/0 (socket says 172.21.15.83:34194) 2024-09-19T07:22:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f52262d9640 1 -- 172.21.15.83:0/851110480 learned_addr learned my addr 172.21.15.83:0/851110480 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f5225ad8640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5220152a60 0x7f522006f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f5226ada640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52201ee710 0x7f52201eeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f52262d9640 1 -- 172.21.15.83:0/851110480 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52201ee710 msgr2=0x7f52201eeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f52262d9640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52201ee710 0x7f52201eeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f52262d9640 1 -- 172.21.15.83:0/851110480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5220152a60 msgr2=0x7f522006f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f52262d9640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5220152a60 0x7f522006f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f52262d9640 1 -- 172.21.15.83:0/851110480 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5220156110 con 0x7f520c0039c0 2024-09-19T07:22:18.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f5226ada640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52201ee710 0x7f52201eeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:18.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f52262d9640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f520c0039c0 0x7f52201f2800 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f521c01c480 tx=0x7f520c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:18.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f5216ffd640 1 -- 172.21.15.83:0/851110480 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c0a9c60 con 0x7f520c0039c0 2024-09-19T07:22:18.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f5216ffd640 1 -- 172.21.15.83:0/851110480 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f521c0a9e00 con 0x7f520c0039c0 2024-09-19T07:22:18.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f522853d640 1 -- 172.21.15.83:0/851110480 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522006fd10 con 0x7f520c0039c0 2024-09-19T07:22:18.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f5216ffd640 1 -- 172.21.15.83:0/851110480 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c0bc660 con 0x7f520c0039c0 2024-09-19T07:22:18.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f522853d640 1 -- 172.21.15.83:0/851110480 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52201f1d20 con 0x7f520c0039c0 2024-09-19T07:22:18.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.863+0000 7f522853d640 1 -- 172.21.15.83:0/851110480 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52200c6460 con 0x7f520c0039c0 2024-09-19T07:22:18.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.867+0000 7f5216ffd640 1 -- 172.21.15.83:0/851110480 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f521c0bc800 con 0x7f520c0039c0 2024-09-19T07:22:18.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.867+0000 7f5216ffd640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52040678c0 0x7f5204069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:18.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.867+0000 7f5216ffd640 1 -- 172.21.15.83:0/851110480 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7f521c120130 con 0x7f520c0039c0 2024-09-19T07:22:18.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.867+0000 7f5225ad8640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52040678c0 0x7f5204069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:18.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.867+0000 7f5225ad8640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52040678c0 0x7f5204069d80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f52100017f0 tx=0x7f52100015e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:18.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:18.871+0000 7f5216ffd640 1 -- 172.21.15.83:0/851110480 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f521c0c3030 con 0x7f520c0039c0 2024-09-19T07:22:19.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.099+0000 7f522853d640 1 -- 172.21.15.83:0/851110480 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52200adaf0 con 0x7f520c0039c0 2024-09-19T07:22:19.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.099+0000 7f5216ffd640 1 -- 172.21.15.83:0/851110480 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6539 (secure 0 0 0) 0x7f521c0ad020 con 0x7f520c0039c0 2024-09-19T07:22:19.100 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:19.100 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":23,"btime":"2024-09-19T07:21:18:974034+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5025,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:22:19.100 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 23 2024-09-19T07:22:19.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.103+0000 7f522853d640 1 -- 172.21.15.83:0/851110480 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52040678c0 msgr2=0x7f5204069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:19.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.103+0000 7f522853d640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52040678c0 0x7f5204069d80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f52100017f0 tx=0x7f52100015e0 comp rx=0 tx=0).stop 2024-09-19T07:22:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.103+0000 7f522853d640 1 -- 172.21.15.83:0/851110480 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f520c0039c0 msgr2=0x7f52201f2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.103+0000 7f522853d640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f520c0039c0 0x7f52201f2800 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f521c01c480 tx=0x7f520c005840 comp rx=0 tx=0).stop 2024-09-19T07:22:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.103+0000 7f522853d640 1 -- 172.21.15.83:0/851110480 shutdown_connections 2024-09-19T07:22:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.103+0000 7f522853d640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52040678c0 0x7f5204069d80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.103+0000 7f522853d640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52201ee710 0x7f52201eeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.103+0000 7f522853d640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5220152a60 0x7f522006f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.103+0000 7f522853d640 1 --2- 172.21.15.83:0/851110480 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f520c0039c0 0x7f52201f2800 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.103+0000 7f522853d640 1 -- 172.21.15.83:0/851110480 >> 172.21.15.83:0/851110480 conn(0x7f52200c5060 msgr2=0x7f52201586a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.103+0000 7f522853d640 1 -- 172.21.15.83:0/851110480 shutdown_connections 2024-09-19T07:22:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:19.103+0000 7f522853d640 1 -- 172.21.15.83:0/851110480 wait complete. 2024-09-19T07:22:19.119 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:22:20.131 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:20.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.259+0000 7fbac8e47640 1 Processor -- start 2024-09-19T07:22:20.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.259+0000 7fbac8e47640 1 -- start start 2024-09-19T07:22:20.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac8e47640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac4071100 0x7fbac41545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:20.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac8e47640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbac40c28e0 0x7fbac4150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:20.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac8e47640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbac40a9e60 con 0x7fbac4151480 2024-09-19T07:22:20.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac8e47640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbac4071b50 con 0x7fbac40c28e0 2024-09-19T07:22:20.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac8e47640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbac40c1ff0 con 0x7fbac4071100 2024-09-19T07:22:20.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac1d9b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbac40c28e0 0x7fbac4150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:20.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac259c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac4071100 0x7fbac41545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:20.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac1d9b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbac40c28e0 0x7fbac4150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:55860/0 (socket says 172.21.15.83:55860) 2024-09-19T07:22:20.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac1d9b640 1 -- 172.21.15.83:0/157635289 learned_addr learned my addr 172.21.15.83:0/157635289 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:20.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac1d9b640 1 -- 172.21.15.83:0/157635289 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac4071100 msgr2=0x7fbac41545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:20.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac1d9b640 1 --2- 172.21.15.83:0/157635289 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac4071100 0x7fbac41545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:20.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac1d9b640 1 -- 172.21.15.83:0/157635289 >> v1:172.21.15.184:6789/0 conn(0x7fbac4151480 legacy=0x7fbac406dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:20.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac1d9b640 1 -- 172.21.15.83:0/157635289 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac40aa140 con 0x7fbac40c28e0 2024-09-19T07:22:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.263+0000 7fbac1d9b640 1 --2- 172.21.15.83:0/157635289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac40c28e0 0x7fbac4150f40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009870 tx=0x7fbaac01c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f88ea0fcd5b9347 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac159a640 1 -- 172.21.15.83:0/157635289 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0b1070 con 0x7fbac40c28e0 2024-09-19T07:22:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac159a640 1 -- 172.21.15.83:0/157635289 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaac0a8050 con 0x7fbac40c28e0 2024-09-19T07:22:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac159a640 1 -- 172.21.15.83:0/157635289 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0ac860 con 0x7fbac40c28e0 2024-09-19T07:22:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 -- 172.21.15.83:0/157635289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac40c28e0 msgr2=0x7fbac4150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 --2- 172.21.15.83:0/157635289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac40c28e0 0x7fbac4150f40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009870 tx=0x7fbaac01c4d0 comp rx=0 tx=0).stop 2024-09-19T07:22:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 -- 172.21.15.83:0/157635289 shutdown_connections 2024-09-19T07:22:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 --2- 172.21.15.83:0/157635289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac40c28e0 0x7fbac4150f40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 --2- 172.21.15.83:0/157635289 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac4071100 0x7fbac41545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 -- 172.21.15.83:0/157635289 >> 172.21.15.83:0/157635289 conn(0x7fbac406bc80 msgr2=0x7fbac406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 -- 172.21.15.83:0/157635289 shutdown_connections 2024-09-19T07:22:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 -- 172.21.15.83:0/157635289 wait complete. 2024-09-19T07:22:20.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 Processor -- start 2024-09-19T07:22:20.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 -- start start 2024-09-19T07:22:20.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac4071100 0x7fbac4152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:20.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbac40c28e0 0x7fbac40c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:20.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac4151480 0x7fbac4153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbac4071660 con 0x7fbac40c28e0 2024-09-19T07:22:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbac40714e0 con 0x7fbac4151480 2024-09-19T07:22:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbac4160c60 con 0x7fbac4071100 2024-09-19T07:22:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbac40c28e0 0x7fbac40c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbac40c28e0 0x7fbac40c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46298/0 (socket says 172.21.15.83:46298) 2024-09-19T07:22:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac1d9b640 1 -- 172.21.15.83:0/3531093995 learned_addr learned my addr 172.21.15.83:0/3531093995 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac259c640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac4071100 0x7fbac4152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac2d9d640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac4151480 0x7fbac4153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac1d9b640 1 -- 172.21.15.83:0/3531093995 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac4071100 msgr2=0x7fbac4152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac1d9b640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac4071100 0x7fbac4152cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac1d9b640 1 -- 172.21.15.83:0/3531093995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac4151480 msgr2=0x7fbac4153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac1d9b640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac4151480 0x7fbac4153720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac1d9b640 1 -- 172.21.15.83:0/3531093995 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac4153c60 con 0x7fbac40c28e0 2024-09-19T07:22:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac1d9b640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbac40c28e0 0x7fbac40c53f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fbaac01a100 tx=0x7fbaac019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbabaffd640 1 -- 172.21.15.83:0/3531093995 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0b1040 con 0x7fbac40c28e0 2024-09-19T07:22:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbabaffd640 1 -- 172.21.15.83:0/3531093995 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaac0bb070 con 0x7fbac40c28e0 2024-09-19T07:22:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbac8e47640 1 -- 172.21.15.83:0/3531093995 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac41462a0 con 0x7fbac40c28e0 2024-09-19T07:22:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.267+0000 7fbabaffd640 1 -- 172.21.15.83:0/3531093995 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0b5410 con 0x7fbac40c28e0 2024-09-19T07:22:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.271+0000 7fbac8e47640 1 -- 172.21.15.83:0/3531093995 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac40c1bf0 con 0x7fbac40c28e0 2024-09-19T07:22:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.271+0000 7fbac8e47640 1 -- 172.21.15.83:0/3531093995 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac41545e0 con 0x7fbac40c28e0 2024-09-19T07:22:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.271+0000 7fbabaffd640 1 -- 172.21.15.83:0/3531093995 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbaac0b55b0 con 0x7fbac40c28e0 2024-09-19T07:22:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.271+0000 7fbabaffd640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbaa40678f0 0x7fbaa4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.271+0000 7fbabaffd640 1 -- 172.21.15.83:0/3531093995 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7fbaac11fcc0 con 0x7fbac40c28e0 2024-09-19T07:22:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.271+0000 7fbac259c640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbaa40678f0 0x7fbaa4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.271+0000 7fbac259c640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbaa40678f0 0x7fbaa4069db0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009a50 tx=0x7fbab4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:20.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.275+0000 7fbabaffd640 1 -- 172.21.15.83:0/3531093995 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbaac016150 con 0x7fbac40c28e0 2024-09-19T07:22:20.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.479+0000 7fbac8e47640 1 -- 172.21.15.83:0/3531093995 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbac40c4f30 con 0x7fbac40c28e0 2024-09-19T07:22:20.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.479+0000 7fbabaffd640 1 -- 172.21.15.83:0/3531093995 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6539 (secure 0 0 0) 0x7fbaac0f04d0 con 0x7fbac40c28e0 2024-09-19T07:22:20.481 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:20.481 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":23,"btime":"2024-09-19T07:21:18:974034+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5025,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:22:20.482 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 23 2024-09-19T07:22:20.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.483+0000 7fbac8e47640 1 -- 172.21.15.83:0/3531093995 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbaa40678f0 msgr2=0x7fbaa4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:20.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.483+0000 7fbac8e47640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbaa40678f0 0x7fbaa4069db0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009a50 tx=0x7fbab4008040 comp rx=0 tx=0).stop 2024-09-19T07:22:20.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.483+0000 7fbac8e47640 1 -- 172.21.15.83:0/3531093995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbac40c28e0 msgr2=0x7fbac40c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:20.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.483+0000 7fbac8e47640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbac40c28e0 0x7fbac40c53f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fbaac01a100 tx=0x7fbaac019270 comp rx=0 tx=0).stop 2024-09-19T07:22:20.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.483+0000 7fbac8e47640 1 -- 172.21.15.83:0/3531093995 shutdown_connections 2024-09-19T07:22:20.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.483+0000 7fbac8e47640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbaa40678f0 0x7fbaa4069db0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:20.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.483+0000 7fbac8e47640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbac4151480 0x7fbac4153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:20.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.483+0000 7fbac8e47640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbac40c28e0 0x7fbac40c53f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:20.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.483+0000 7fbac8e47640 1 --2- 172.21.15.83:0/3531093995 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbac4071100 0x7fbac4152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:20.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.483+0000 7fbac8e47640 1 -- 172.21.15.83:0/3531093995 >> 172.21.15.83:0/3531093995 conn(0x7fbac406bc80 msgr2=0x7fbac4073000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:20.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.483+0000 7fbac8e47640 1 -- 172.21.15.83:0/3531093995 shutdown_connections 2024-09-19T07:22:20.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:20.483+0000 7fbac8e47640 1 -- 172.21.15.83:0/3531093995 wait complete. 2024-09-19T07:22:20.500 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:22:21.120 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:21.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f065b30b640 1 Processor -- start 2024-09-19T07:22:21.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f065b30b640 1 -- start start 2024-09-19T07:22:21.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f065b30b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0654155d70 0x7f06540c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:21.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f065b30b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f06541592f0 0x7f0654157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:21.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f065b30b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0654156200 con 0x7f06540c6120 2024-09-19T07:22:21.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f065b30b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f06540c9930 con 0x7f06541592f0 2024-09-19T07:22:21.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f065b30b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f06541564e0 con 0x7f0654155d70 2024-09-19T07:22:21.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f06590a7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f06540c6120 0x7f06541575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48618/0 (socket says 172.21.15.83:48618) 2024-09-19T07:22:21.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f06590a7640 1 -- 172.21.15.83:0/2092259625 learned_addr learned my addr 172.21.15.83:0/2092259625 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:21.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f06598a8640 1 --2- 172.21.15.83:0/2092259625 >> v2:172.21.15.83:3300/0 conn(0x7f06541592f0 0x7f0654157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:21.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f064bfff640 1 -- 172.21.15.83:0/2092259625 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1088639674 0 0) 0x7f0654156200 con 0x7f06540c6120 2024-09-19T07:22:21.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f064bfff640 1 -- 172.21.15.83:0/2092259625 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0640003530 con 0x7f06540c6120 2024-09-19T07:22:21.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f06588a6640 1 --2- 172.21.15.83:0/2092259625 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0654155d70 0x7f06540c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:21.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f06598a8640 1 -- 172.21.15.83:0/2092259625 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0654155d70 msgr2=0x7f06540c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:21.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f06598a8640 1 --2- 172.21.15.83:0/2092259625 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0654155d70 0x7f06540c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:21.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f06598a8640 1 -- 172.21.15.83:0/2092259625 >> v1:172.21.15.83:6789/0 conn(0x7f06540c6120 legacy=0x7f06541575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:21.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.251+0000 7f06598a8640 1 -- 172.21.15.83:0/2092259625 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06541567c0 con 0x7f06541592f0 2024-09-19T07:22:21.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06598a8640 1 --2- 172.21.15.83:0/2092259625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06541592f0 0x7f0654157ce0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f064c009d90 tx=0x7f064c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ab2e26081a87e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:21.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f064bfff640 1 -- 172.21.15.83:0/2092259625 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c0b2070 con 0x7f06541592f0 2024-09-19T07:22:21.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f064bfff640 1 -- 172.21.15.83:0/2092259625 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f064c0a5830 con 0x7f06541592f0 2024-09-19T07:22:21.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f064bfff640 1 -- 172.21.15.83:0/2092259625 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c0a5c00 con 0x7f06541592f0 2024-09-19T07:22:21.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 -- 172.21.15.83:0/2092259625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06541592f0 msgr2=0x7f0654157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:21.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 --2- 172.21.15.83:0/2092259625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06541592f0 0x7f0654157ce0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f064c009d90 tx=0x7f064c01c970 comp rx=0 tx=0).stop 2024-09-19T07:22:21.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 -- 172.21.15.83:0/2092259625 shutdown_connections 2024-09-19T07:22:21.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 --2- 172.21.15.83:0/2092259625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06541592f0 0x7f0654157ce0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:21.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 --2- 172.21.15.83:0/2092259625 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0654155d70 0x7f06540c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:21.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 -- 172.21.15.83:0/2092259625 >> 172.21.15.83:0/2092259625 conn(0x7f065406bc80 msgr2=0x7f065406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:21.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 -- 172.21.15.83:0/2092259625 shutdown_connections 2024-09-19T07:22:21.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 -- 172.21.15.83:0/2092259625 wait complete. 2024-09-19T07:22:21.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 Processor -- start 2024-09-19T07:22:21.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 -- start start 2024-09-19T07:22:21.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06540c6120 0x7f06540c2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:21.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06541592f0 0x7f06540c10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:21.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06540c0c00 0x7f06540c1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:21.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0654153860 con 0x7f06541592f0 2024-09-19T07:22:21.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0654162ea0 con 0x7f06540c0c00 2024-09-19T07:22:21.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f065b30b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0654148d30 con 0x7f06540c6120 2024-09-19T07:22:21.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06588a6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06541592f0 0x7f06540c10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:21.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06588a6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06541592f0 0x7f06540c10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46320/0 (socket says 172.21.15.83:46320) 2024-09-19T07:22:21.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06588a6640 1 -- 172.21.15.83:0/1090344406 learned_addr learned my addr 172.21.15.83:0/1090344406 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:21.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06598a8640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06540c0c00 0x7f06540c1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:21.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06588a6640 1 -- 172.21.15.83:0/1090344406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06540c6120 msgr2=0x7f06540c2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:21.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06590a7640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06540c6120 0x7f06540c2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:21.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06588a6640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06540c6120 0x7f06540c2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:21.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06588a6640 1 -- 172.21.15.83:0/1090344406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06540c0c00 msgr2=0x7f06540c1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:21.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06588a6640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06540c0c00 0x7f06540c1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:21.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06588a6640 1 -- 172.21.15.83:0/1090344406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06540c2540 con 0x7f06541592f0 2024-09-19T07:22:21.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06598a8640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06540c0c00 0x7f06540c1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:21.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f06588a6640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06541592f0 0x7f06540c10f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f064400c7f0 tx=0x7f064400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:21.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f0649ffb640 1 -- 172.21.15.83:0/1090344406 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0644015070 con 0x7f06541592f0 2024-09-19T07:22:21.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f0649ffb640 1 -- 172.21.15.83:0/1090344406 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0644004240 con 0x7f06541592f0 2024-09-19T07:22:21.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.255+0000 7f0649ffb640 1 -- 172.21.15.83:0/1090344406 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0644003620 con 0x7f06541592f0 2024-09-19T07:22:21.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.259+0000 7f065b30b640 1 -- 172.21.15.83:0/1090344406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0654149320 con 0x7f06541592f0 2024-09-19T07:22:21.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.259+0000 7f065b30b640 1 -- 172.21.15.83:0/1090344406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06540c9ee0 con 0x7f06541592f0 2024-09-19T07:22:21.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.259+0000 7f065b30b640 1 -- 172.21.15.83:0/1090344406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06540c6500 con 0x7f06541592f0 2024-09-19T07:22:21.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.259+0000 7f0649ffb640 1 -- 172.21.15.83:0/1090344406 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f06440039d0 con 0x7f06541592f0 2024-09-19T07:22:21.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.259+0000 7f0649ffb640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f06380678c0 0x7f0638069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:21.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.259+0000 7f0649ffb640 1 -- 172.21.15.83:0/1090344406 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7f06440860b0 con 0x7f06541592f0 2024-09-19T07:22:21.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.259+0000 7f06590a7640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f06380678c0 0x7f0638069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:21.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.259+0000 7f06590a7640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f06380678c0 0x7f0638069d80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0650007fd0 tx=0x7f0650009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:21.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.263+0000 7f0649ffb640 1 -- 172.21.15.83:0/1090344406 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f0644056830 con 0x7f06541592f0 2024-09-19T07:22:21.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.459+0000 7f065b30b640 1 -- 172.21.15.83:0/1090344406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06540c96a0 con 0x7f06541592f0 2024-09-19T07:22:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.459+0000 7f0649ffb640 1 -- 172.21.15.83:0/1090344406 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6539 (secure 0 0 0) 0x7f0644056130 con 0x7f06541592f0 2024-09-19T07:22:21.461 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:21.461 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":23,"btime":"2024-09-19T07:21:18:974034+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5025,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:22:21.462 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 23 2024-09-19T07:22:21.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.463+0000 7f065b30b640 1 -- 172.21.15.83:0/1090344406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f06380678c0 msgr2=0x7f0638069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:21.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.463+0000 7f065b30b640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f06380678c0 0x7f0638069d80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0650007fd0 tx=0x7f0650009040 comp rx=0 tx=0).stop 2024-09-19T07:22:21.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.463+0000 7f065b30b640 1 -- 172.21.15.83:0/1090344406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06541592f0 msgr2=0x7f06540c10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:21.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.463+0000 7f065b30b640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06541592f0 0x7f06540c10f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f064400c7f0 tx=0x7f064400ccb0 comp rx=0 tx=0).stop 2024-09-19T07:22:21.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.463+0000 7f065b30b640 1 -- 172.21.15.83:0/1090344406 shutdown_connections 2024-09-19T07:22:21.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.463+0000 7f065b30b640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f06380678c0 0x7f0638069d80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:21.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.463+0000 7f065b30b640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06540c0c00 0x7f06540c1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:21.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.463+0000 7f065b30b640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06541592f0 0x7f06540c10f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:21.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.463+0000 7f065b30b640 1 --2- 172.21.15.83:0/1090344406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06540c6120 0x7f06540c2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:21.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.463+0000 7f065b30b640 1 -- 172.21.15.83:0/1090344406 >> 172.21.15.83:0/1090344406 conn(0x7f065406bc80 msgr2=0x7f065415ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:21.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.463+0000 7f065b30b640 1 -- 172.21.15.83:0/1090344406 shutdown_connections 2024-09-19T07:22:21.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:21.463+0000 7f065b30b640 1 -- 172.21.15.83:0/1090344406 wait complete. 2024-09-19T07:22:21.481 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:22:22.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:22.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f7b87b640 1 Processor -- start 2024-09-19T07:22:22.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f7b87b640 1 -- start start 2024-09-19T07:22:22.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f7b87b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 0x7f9f741545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f7b87b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9f740c28e0 0x7f9f74150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f7b87b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f740a9e60 con 0x7f9f74151480 2024-09-19T07:22:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f7b87b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f9f74071b50 con 0x7f9f740c28e0 2024-09-19T07:22:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f7b87b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9f740c1ff0 con 0x7f9f74071100 2024-09-19T07:22:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f78e16640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9f740c28e0 0x7f9f74150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f79617640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 0x7f9f741545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f79e18640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f9f74151480 0x7f9f7406dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:44838/0 (socket says 172.21.15.83:44838) 2024-09-19T07:22:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f79617640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 0x7f9f741545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:52056/0 (socket says 172.21.15.83:52056) 2024-09-19T07:22:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f78e16640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9f740c28e0 0x7f9f74150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:55882/0 (socket says 172.21.15.83:55882) 2024-09-19T07:22:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f79e18640 1 -- 172.21.15.83:0/2526634152 learned_addr learned my addr 172.21.15.83:0/2526634152 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f79617640 1 -- 172.21.15.83:0/2526634152 >> v2:172.21.15.184:3300/0 conn(0x7f9f740c28e0 msgr2=0x7f9f74150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f79617640 1 --2- 172.21.15.83:0/2526634152 >> v2:172.21.15.184:3300/0 conn(0x7f9f740c28e0 0x7f9f74150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f79617640 1 -- 172.21.15.83:0/2526634152 >> v1:172.21.15.184:6789/0 conn(0x7f9f74151480 legacy=0x7f9f7406dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.630+0000 7f9f79617640 1 -- 172.21.15.83:0/2526634152 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f740aa140 con 0x7f9f74071100 2024-09-19T07:22:22.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f79617640 1 --2- 172.21.15.83:0/2526634152 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 0x7f9f741545e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9f640097c0 tx=0x7f9f6401c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6dc32e7e9b62769 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:22.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f6bfff640 1 -- 172.21.15.83:0/2526634152 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f640b0070 con 0x7f9f74071100 2024-09-19T07:22:22.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f6bfff640 1 -- 172.21.15.83:0/2526634152 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f640ab600 con 0x7f9f74071100 2024-09-19T07:22:22.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f6bfff640 1 -- 172.21.15.83:0/2526634152 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f640ab9d0 con 0x7f9f74071100 2024-09-19T07:22:22.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2526634152 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 msgr2=0x7f9f741545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:22.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f7b87b640 1 --2- 172.21.15.83:0/2526634152 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 0x7f9f741545e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9f640097c0 tx=0x7f9f6401c410 comp rx=0 tx=0).stop 2024-09-19T07:22:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2526634152 shutdown_connections 2024-09-19T07:22:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f7b87b640 1 --2- 172.21.15.83:0/2526634152 >> v2:172.21.15.184:3300/0 conn(0x7f9f740c28e0 0x7f9f74150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f7b87b640 1 --2- 172.21.15.83:0/2526634152 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 0x7f9f741545e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2526634152 >> 172.21.15.83:0/2526634152 conn(0x7f9f7406bc80 msgr2=0x7f9f7406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2526634152 shutdown_connections 2024-09-19T07:22:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2526634152 wait complete. 2024-09-19T07:22:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f7b87b640 1 Processor -- start 2024-09-19T07:22:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.634+0000 7f9f7b87b640 1 -- start start 2024-09-19T07:22:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f7b87b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 0x7f9f741537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f7b87b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f740c28e0 0x7f9f74148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f7b87b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f74151480 0x7f9f741ecd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f7b87b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9f74071660 con 0x7f9f740c28e0 2024-09-19T07:22:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f7b87b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9f740714e0 con 0x7f9f74151480 2024-09-19T07:22:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f7b87b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9f74160c60 con 0x7f9f74071100 2024-09-19T07:22:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f78e16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f740c28e0 0x7f9f74148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:22.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f78e16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f740c28e0 0x7f9f74148d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46328/0 (socket says 172.21.15.83:46328) 2024-09-19T07:22:22.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f78e16640 1 -- 172.21.15.83:0/2580625318 learned_addr learned my addr 172.21.15.83:0/2580625318 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:22.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f79e18640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f74151480 0x7f9f741ecd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:22.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f79617640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 0x7f9f741537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:22.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f78e16640 1 -- 172.21.15.83:0/2580625318 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 msgr2=0x7f9f741537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:22.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f78e16640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 0x7f9f741537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:22.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f78e16640 1 -- 172.21.15.83:0/2580625318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f74151480 msgr2=0x7f9f741ecd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:22.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f78e16640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f74151480 0x7f9f741ecd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:22.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f78e16640 1 -- 172.21.15.83:0/2580625318 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7414a7c0 con 0x7f9f740c28e0 2024-09-19T07:22:22.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f79617640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 0x7f9f741537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:22.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f78e16640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f740c28e0 0x7f9f74148d90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000e900 tx=0x7f9f7000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:22.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f69ffb640 1 -- 172.21.15.83:0/2580625318 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f70017070 con 0x7f9f740c28e0 2024-09-19T07:22:22.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f69ffb640 1 -- 172.21.15.83:0/2580625318 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f7000ca00 con 0x7f9f740c28e0 2024-09-19T07:22:22.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2580625318 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f7414aaa0 con 0x7f9f740c28e0 2024-09-19T07:22:22.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f69ffb640 1 -- 172.21.15.83:0/2580625318 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7000cdd0 con 0x7f9f740c28e0 2024-09-19T07:22:22.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2580625318 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f741492d0 con 0x7f9f740c28e0 2024-09-19T07:22:22.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2580625318 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f7414acc0 con 0x7f9f740c28e0 2024-09-19T07:22:22.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f69ffb640 1 -- 172.21.15.83:0/2580625318 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9f7001a070 con 0x7f9f740c28e0 2024-09-19T07:22:22.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f69ffb640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9f580678c0 0x7f9f58069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:22.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f69ffb640 1 -- 172.21.15.83:0/2580625318 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6809+0+0 (secure 0 0 0) 0x7f9f70086400 con 0x7f9f740c28e0 2024-09-19T07:22:22.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f79617640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9f580678c0 0x7f9f58069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:22.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.638+0000 7f9f79617640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9f580678c0 0x7f9f58069d80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9f640ad000 tx=0x7f9f64008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:22.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.642+0000 7f9f69ffb640 1 -- 172.21.15.83:0/2580625318 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9f70056b80 con 0x7f9f740c28e0 2024-09-19T07:22:22.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.838+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2580625318 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f741ec330 con 0x7f9f740c28e0 2024-09-19T07:22:22.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f69ffb640 1 -- 172.21.15.83:0/2580625318 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6539 (secure 0 0 0) 0x7f9f70056480 con 0x7f9f740c28e0 2024-09-19T07:22:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:22.843 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":23,"btime":"2024-09-19T07:21:18:974034+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5025,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:21:00.932928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6833/1773885233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1773885233},{"type":"v1","addr":"172.21.15.184:6833","nonce":1773885233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4229,"qdb_cluster":[4229]},"id":2}]} 2024-09-19T07:22:22.843 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 23 2024-09-19T07:22:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2580625318 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9f580678c0 msgr2=0x7f9f58069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f7b87b640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9f580678c0 0x7f9f58069d80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9f640ad000 tx=0x7f9f64008040 comp rx=0 tx=0).stop 2024-09-19T07:22:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2580625318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f740c28e0 msgr2=0x7f9f74148d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f7b87b640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f740c28e0 0x7f9f74148d90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000e900 tx=0x7f9f7000edc0 comp rx=0 tx=0).stop 2024-09-19T07:22:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2580625318 shutdown_connections 2024-09-19T07:22:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f7b87b640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9f580678c0 0x7f9f58069d80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f7b87b640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f74151480 0x7f9f741ecd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f7b87b640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f740c28e0 0x7f9f74148d90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f7b87b640 1 --2- 172.21.15.83:0/2580625318 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f74071100 0x7f9f741537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2580625318 >> 172.21.15.83:0/2580625318 conn(0x7f9f7406bc80 msgr2=0x7f9f7414f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2580625318 shutdown_connections 2024-09-19T07:22:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:22.842+0000 7f9f7b87b640 1 -- 172.21.15.83:0/2580625318 wait complete. 2024-09-19T07:22:22.862 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:22:23.482 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:23.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f6224a0a640 1 Processor -- start 2024-09-19T07:22:23.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f6224a0a640 1 -- start start 2024-09-19T07:22:23.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f6224a0a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f62200c3ab0 0x7f6220157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:23.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f6224a0a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f62200aa4a0 0x7f6220157550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:23.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f6224a0a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6220073d90 con 0x7f622014dbe0 2024-09-19T07:22:23.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f6224a0a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6220154700 con 0x7f62200aa4a0 2024-09-19T07:22:23.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f6224a0a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6220153b30 con 0x7f62200c3ab0 2024-09-19T07:22:23.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f621e59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f622014dbe0 0x7f62200c3fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48628/0 (socket says 172.21.15.83:48628) 2024-09-19T07:22:23.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f621e59c640 1 -- 172.21.15.83:0/2954015097 learned_addr learned my addr 172.21.15.83:0/2954015097 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:23.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f621ed9d640 1 --2- 172.21.15.83:0/2954015097 >> v2:172.21.15.83:3300/0 conn(0x7f62200aa4a0 0x7f6220157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:23.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f621dd9b640 1 --2- 172.21.15.83:0/2954015097 >> v2:172.21.15.184:3300/0 conn(0x7f62200c3ab0 0x7f6220157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:23.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f621ed9d640 1 -- 172.21.15.83:0/2954015097 >> v2:172.21.15.184:3300/0 conn(0x7f62200c3ab0 msgr2=0x7f6220157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:23.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f621ed9d640 1 --2- 172.21.15.83:0/2954015097 >> v2:172.21.15.184:3300/0 conn(0x7f62200c3ab0 0x7f6220157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f621ed9d640 1 -- 172.21.15.83:0/2954015097 >> v1:172.21.15.83:6789/0 conn(0x7f622014dbe0 legacy=0x7f62200c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:23.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f621ed9d640 1 -- 172.21.15.83:0/2954015097 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6220153dd0 con 0x7f62200aa4a0 2024-09-19T07:22:23.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.610+0000 7f621dd9b640 1 --2- 172.21.15.83:0/2954015097 >> v2:172.21.15.184:3300/0 conn(0x7f62200c3ab0 0x7f6220157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:23.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621ed9d640 1 --2- 172.21.15.83:0/2954015097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62200aa4a0 0x7f6220157550 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f62200c3830 tx=0x7f621801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f606eabca8ac99 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:23.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621d59a640 1 -- 172.21.15.83:0/2954015097 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62180b2070 con 0x7f62200aa4a0 2024-09-19T07:22:23.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621d59a640 1 -- 172.21.15.83:0/2954015097 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62180a5870 con 0x7f62200aa4a0 2024-09-19T07:22:23.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621d59a640 1 -- 172.21.15.83:0/2954015097 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62180a5c40 con 0x7f62200aa4a0 2024-09-19T07:22:23.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 -- 172.21.15.83:0/2954015097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62200aa4a0 msgr2=0x7f6220157550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:23.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 --2- 172.21.15.83:0/2954015097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62200aa4a0 0x7f6220157550 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f62200c3830 tx=0x7f621801c8a0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 -- 172.21.15.83:0/2954015097 shutdown_connections 2024-09-19T07:22:23.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 --2- 172.21.15.83:0/2954015097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62200aa4a0 0x7f6220157550 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 --2- 172.21.15.83:0/2954015097 >> v2:172.21.15.184:3300/0 conn(0x7f62200c3ab0 0x7f6220157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 -- 172.21.15.83:0/2954015097 >> 172.21.15.83:0/2954015097 conn(0x7f62200c95a0 msgr2=0x7f62200c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:23.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 -- 172.21.15.83:0/2954015097 shutdown_connections 2024-09-19T07:22:23.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 -- 172.21.15.83:0/2954015097 wait complete. 2024-09-19T07:22:23.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 Processor -- start 2024-09-19T07:22:23.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 -- start start 2024-09-19T07:22:23.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62200aa4a0 0x7f62200ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:23.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f622014dbe0 0x7f62200ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:23.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62200af350 0x7f6220144ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:23.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f62200c4830 con 0x7f62200aa4a0 2024-09-19T07:22:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62200c46b0 con 0x7f62200af350 2024-09-19T07:22:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f6224a0a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6220160bc0 con 0x7f622014dbe0 2024-09-19T07:22:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62200aa4a0 0x7f62200ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62200aa4a0 0x7f62200ac6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46346/0 (socket says 172.21.15.83:46346) 2024-09-19T07:22:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621e59c640 1 -- 172.21.15.83:0/199361126 learned_addr learned my addr 172.21.15.83:0/199361126 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621ed9d640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62200af350 0x7f6220144ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621dd9b640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f622014dbe0 0x7f62200ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621e59c640 1 -- 172.21.15.83:0/199361126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f622014dbe0 msgr2=0x7f62200ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621e59c640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f622014dbe0 0x7f62200ad560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621e59c640 1 -- 172.21.15.83:0/199361126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62200af350 msgr2=0x7f6220144ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621e59c640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62200af350 0x7f6220144ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.614+0000 7f621e59c640 1 -- 172.21.15.83:0/199361126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6220145f60 con 0x7f62200aa4a0 2024-09-19T07:22:23.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.618+0000 7f621e59c640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62200aa4a0 0x7f62200ac6a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f621400c3a0 tx=0x7f621400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:23.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.618+0000 7f620effd640 1 -- 172.21.15.83:0/199361126 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214017070 con 0x7f62200aa4a0 2024-09-19T07:22:23.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.618+0000 7f620effd640 1 -- 172.21.15.83:0/199361126 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6214005cf0 con 0x7f62200aa4a0 2024-09-19T07:22:23.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.618+0000 7f620effd640 1 -- 172.21.15.83:0/199361126 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62140072f0 con 0x7f62200aa4a0 2024-09-19T07:22:23.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.618+0000 7f6224a0a640 1 -- 172.21.15.83:0/199361126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62200c7230 con 0x7f62200aa4a0 2024-09-19T07:22:23.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.618+0000 7f6224a0a640 1 -- 172.21.15.83:0/199361126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62200aeeb0 con 0x7f62200aa4a0 2024-09-19T07:22:23.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.618+0000 7f6224a0a640 1 -- 172.21.15.83:0/199361126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62200aeca0 con 0x7f62200aa4a0 2024-09-19T07:22:23.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.618+0000 7f620effd640 1 -- 172.21.15.83:0/199361126 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f621401a070 con 0x7f62200aa4a0 2024-09-19T07:22:23.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.618+0000 7f620effd640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62000678c0 0x7f6200069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:23.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.618+0000 7f620effd640 1 -- 172.21.15.83:0/199361126 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6895+0+0 (secure 0 0 0) 0x7f62140854a0 con 0x7f62200aa4a0 2024-09-19T07:22:23.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.618+0000 7f621dd9b640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62000678c0 0x7f6200069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:23.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.618+0000 7f621dd9b640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62000678c0 0x7f6200069d80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6210006fd0 tx=0x7f6210008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:23.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.622+0000 7f620effd640 1 -- 172.21.15.83:0/199361126 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6214055b50 con 0x7f62200aa4a0 2024-09-19T07:22:23.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.822+0000 7f6224a0a640 1 -- 172.21.15.83:0/199361126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62200c3fa0 con 0x7f62200aa4a0 2024-09-19T07:22:23.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f620effd640 1 -- 172.21.15.83:0/199361126 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+5703 (secure 0 0 0) 0x7f6214055450 con 0x7f62200aa4a0 2024-09-19T07:22:23.827 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:23.827 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":24,"btime":"2024-09-19T07:22:22:946115+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:22.946106+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:22:23.827 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 24 2024-09-19T07:22:23.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f6224a0a640 1 -- 172.21.15.83:0/199361126 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62000678c0 msgr2=0x7f6200069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:23.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f6224a0a640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62000678c0 0x7f6200069d80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6210006fd0 tx=0x7f6210008040 comp rx=0 tx=0).stop 2024-09-19T07:22:23.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f6224a0a640 1 -- 172.21.15.83:0/199361126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62200aa4a0 msgr2=0x7f62200ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:23.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f6224a0a640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62200aa4a0 0x7f62200ac6a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f621400c3a0 tx=0x7f621400c860 comp rx=0 tx=0).stop 2024-09-19T07:22:23.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f6224a0a640 1 -- 172.21.15.83:0/199361126 shutdown_connections 2024-09-19T07:22:23.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f6224a0a640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62000678c0 0x7f6200069d80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f6224a0a640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62200af350 0x7f6220144ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f6224a0a640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f622014dbe0 0x7f62200ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f6224a0a640 1 --2- 172.21.15.83:0/199361126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62200aa4a0 0x7f62200ac6a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f6224a0a640 1 -- 172.21.15.83:0/199361126 >> 172.21.15.83:0/199361126 conn(0x7f62200c95a0 msgr2=0x7f62201496b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:23.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f6224a0a640 1 -- 172.21.15.83:0/199361126 shutdown_connections 2024-09-19T07:22:23.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.826+0000 7f6224a0a640 1 -- 172.21.15.83:0/199361126 wait complete. 2024-09-19T07:22:23.846 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-19T07:22:23.846 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:22:23.846 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f367acef640 1 Processor -- start 2024-09-19T07:22:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f367acef640 1 -- start start 2024-09-19T07:22:23.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f367acef640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f367414dbe0 0x7f36740c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:23.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f367acef640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36740c44e0 0x7f3674157720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:23.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f367acef640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3674073d90 con 0x7f36740c3ab0 2024-09-19T07:22:23.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f367acef640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f3674154700 con 0x7f367414dbe0 2024-09-19T07:22:23.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f367acef640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3674153b30 con 0x7f36740c44e0 2024-09-19T07:22:23.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f367928c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36740c44e0 0x7f3674157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:23.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f3678a8b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f367414dbe0 0x7f36740c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:23.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f3678a8b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f367414dbe0 0x7f36740c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:55926/0 (socket says 172.21.15.83:55926) 2024-09-19T07:22:23.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f3673fff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f36740c3ab0 0x7f3674157010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:44854/0 (socket says 172.21.15.83:44854) 2024-09-19T07:22:23.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f3678a8b640 1 -- 172.21.15.83:0/1602154202 learned_addr learned my addr 172.21.15.83:0/1602154202 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:23.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f367928c640 1 -- 172.21.15.83:0/1602154202 >> v2:172.21.15.184:3300/0 conn(0x7f367414dbe0 msgr2=0x7f36740c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:23.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f367928c640 1 --2- 172.21.15.83:0/1602154202 >> v2:172.21.15.184:3300/0 conn(0x7f367414dbe0 0x7f36740c3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f367928c640 1 -- 172.21.15.83:0/1602154202 >> v1:172.21.15.184:6789/0 conn(0x7f36740c3ab0 legacy=0x7f3674157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:23.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f367928c640 1 -- 172.21.15.83:0/1602154202 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3674153dd0 con 0x7f36740c44e0 2024-09-19T07:22:23.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.966+0000 7f3678a8b640 1 --2- 172.21.15.83:0/1602154202 >> v2:172.21.15.184:3300/0 conn(0x7f367414dbe0 0x7f36740c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:22:23.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367928c640 1 --2- 172.21.15.83:0/1602154202 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36740c44e0 0x7f3674157720 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3674154880 tx=0x7f366c01c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0bb59c4f51b8b03 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f36737fe640 1 -- 172.21.15.83:0/1602154202 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c0b2070 con 0x7f36740c44e0 2024-09-19T07:22:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f36737fe640 1 -- 172.21.15.83:0/1602154202 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f366c0a5890 con 0x7f36740c44e0 2024-09-19T07:22:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f36737fe640 1 -- 172.21.15.83:0/1602154202 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c0a5c60 con 0x7f36740c44e0 2024-09-19T07:22:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 -- 172.21.15.83:0/1602154202 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36740c44e0 msgr2=0x7f3674157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 --2- 172.21.15.83:0/1602154202 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36740c44e0 0x7f3674157720 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3674154880 tx=0x7f366c01c8c0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 -- 172.21.15.83:0/1602154202 shutdown_connections 2024-09-19T07:22:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 --2- 172.21.15.83:0/1602154202 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36740c44e0 0x7f3674157720 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 --2- 172.21.15.83:0/1602154202 >> v2:172.21.15.184:3300/0 conn(0x7f367414dbe0 0x7f36740c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 -- 172.21.15.83:0/1602154202 >> 172.21.15.83:0/1602154202 conn(0x7f36740c95a0 msgr2=0x7f36740c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 -- 172.21.15.83:0/1602154202 shutdown_connections 2024-09-19T07:22:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 -- 172.21.15.83:0/1602154202 wait complete. 2024-09-19T07:22:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 Processor -- start 2024-09-19T07:22:23.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 -- start start 2024-09-19T07:22:23.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36740c44e0 0x7f36741ef3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:23.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f367414dbe0 0x7f36740aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:23.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36740ad190 0x7f3674146d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:23.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3674160bc0 con 0x7f36740ad190 2024-09-19T07:22:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f36740741a0 con 0x7f367414dbe0 2024-09-19T07:22:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.970+0000 7f367acef640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3674160d40 con 0x7f36740c44e0 2024-09-19T07:22:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f367928c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36740ad190 0x7f3674146d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f367928c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36740ad190 0x7f3674146d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46360/0 (socket says 172.21.15.83:46360) 2024-09-19T07:22:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f367928c640 1 -- 172.21.15.83:0/1019419979 learned_addr learned my addr 172.21.15.83:0/1019419979 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f3678a8b640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36740c44e0 0x7f36741ef3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f3673fff640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f367414dbe0 0x7f36740aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f367928c640 1 -- 172.21.15.83:0/1019419979 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36740c44e0 msgr2=0x7f36741ef3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f367928c640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36740c44e0 0x7f36741ef3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f367928c640 1 -- 172.21.15.83:0/1019419979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f367414dbe0 msgr2=0x7f36740aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f367928c640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f367414dbe0 0x7f36740aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:23.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f367928c640 1 -- 172.21.15.83:0/1019419979 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36740ac440 con 0x7f36740ad190 2024-09-19T07:22:23.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f367928c640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36740ad190 0x7f3674146d80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f366c0b0040 tx=0x7f366c019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:23.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f36717fa640 1 -- 172.21.15.83:0/1019419979 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c0b2050 con 0x7f36740ad190 2024-09-19T07:22:23.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f367acef640 1 -- 172.21.15.83:0/1019419979 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36740ac720 con 0x7f36740ad190 2024-09-19T07:22:23.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f36717fa640 1 -- 172.21.15.83:0/1019419979 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f366c0be070 con 0x7f36740ad190 2024-09-19T07:22:23.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f36717fa640 1 -- 172.21.15.83:0/1019419979 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c0ada90 con 0x7f36740ad190 2024-09-19T07:22:23.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f367acef640 1 -- 172.21.15.83:0/1019419979 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36740ad6f0 con 0x7f36740ad190 2024-09-19T07:22:23.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f367acef640 1 -- 172.21.15.83:0/1019419979 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36740ad8f0 con 0x7f36740ad190 2024-09-19T07:22:23.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f36717fa640 1 -- 172.21.15.83:0/1019419979 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f366c0a5710 con 0x7f36740ad190 2024-09-19T07:22:23.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f36717fa640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f36600678f0 0x7f3660069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:23.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f36717fa640 1 -- 172.21.15.83:0/1019419979 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6895+0+0 (secure 0 0 0) 0x7f366c121670 con 0x7f36740ad190 2024-09-19T07:22:23.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f3678a8b640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f36600678f0 0x7f3660069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:23.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.974+0000 7f3678a8b640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f36600678f0 0x7f3660069db0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3668009af0 tx=0x7f3668008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:23.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:23.978+0000 7f36717fa640 1 -- 172.21.15.83:0/1019419979 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f36740ad8f0 con 0x7f36740ad190 2024-09-19T07:22:24.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 -- 172.21.15.83:0/1019419979 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36740aba90 con 0x7f36740ad190 2024-09-19T07:22:24.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f36717fa640 1 -- 172.21.15.83:0/1019419979 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5707 (secure 0 0 0) 0x7f366c0c7020 con 0x7f36740ad190 2024-09-19T07:22:24.183 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:24.184 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"btime":"2024-09-19T07:22:24:137376+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:23.197612+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:22:24.184 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 25 2024-09-19T07:22:24.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 -- 172.21.15.83:0/1019419979 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f36600678f0 msgr2=0x7f3660069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:24.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f36600678f0 0x7f3660069db0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3668009af0 tx=0x7f3668008040 comp rx=0 tx=0).stop 2024-09-19T07:22:24.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 -- 172.21.15.83:0/1019419979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36740ad190 msgr2=0x7f3674146d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:24.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36740ad190 0x7f3674146d80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f366c0b0040 tx=0x7f366c019660 comp rx=0 tx=0).stop 2024-09-19T07:22:24.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 -- 172.21.15.83:0/1019419979 shutdown_connections 2024-09-19T07:22:24.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f36600678f0 0x7f3660069db0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:24.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36740ad190 0x7f3674146d80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:24.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f367414dbe0 0x7f36740aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:24.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 --2- 172.21.15.83:0/1019419979 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36740c44e0 0x7f36741ef3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:24.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 -- 172.21.15.83:0/1019419979 >> 172.21.15.83:0/1019419979 conn(0x7f36740c95a0 msgr2=0x7f367414c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:24.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 -- 172.21.15.83:0/1019419979 shutdown_connections 2024-09-19T07:22:24.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.182+0000 7f367acef640 1 -- 172.21.15.83:0/1019419979 wait complete. 2024-09-19T07:22:24.203 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:22:24:137376+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 25, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4220": { "addr": "172.21.15.83:6837/4265456507", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 4265456507, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 4265456507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 19, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:21:10.937435+0000", "qdb_cluster": [ 4220 ], "qdb_leader": 4220, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4220 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 25, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5025": { "addr": "172.21.15.83:6835/1774585869", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 1774585869, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 1774585869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5025, "incarnation": 24, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:22:23.197612+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5025 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/4115191178", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 4115191178, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 4115191178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 21, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4786, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:22:24.862 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:24.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf7311a640 1 Processor -- start 2024-09-19T07:22:24.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf7311a640 1 -- start start 2024-09-19T07:22:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf7311a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbf6c0ab060 0x7fbf6c1528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf7311a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf6c0c3b70 0x7fbf6c152e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf7311a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf6c06de90 con 0x7fbf6c0ab750 2024-09-19T07:22:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf7311a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fbf6c06e170 con 0x7fbf6c0c3b70 2024-09-19T07:22:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf7311a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbf6c06e450 con 0x7fbf6c0ab060 2024-09-19T07:22:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf716b7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf6c0c3b70 0x7fbf6c152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf716b7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf6c0c3b70 0x7fbf6c152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46364/0 (socket says 172.21.15.83:46364) 2024-09-19T07:22:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf716b7640 1 -- 172.21.15.83:0/98232276 learned_addr learned my addr 172.21.15.83:0/98232276 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf63fff640 1 --2- 172.21.15.83:0/98232276 >> v2:172.21.15.184:3300/0 conn(0x7fbf6c0ab060 0x7fbf6c1528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf716b7640 1 -- 172.21.15.83:0/98232276 >> v2:172.21.15.184:3300/0 conn(0x7fbf6c0ab060 msgr2=0x7fbf6c1528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf716b7640 1 --2- 172.21.15.83:0/98232276 >> v2:172.21.15.184:3300/0 conn(0x7fbf6c0ab060 0x7fbf6c1528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:24.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf716b7640 1 -- 172.21.15.83:0/98232276 >> v1:172.21.15.184:6789/0 conn(0x7fbf6c0ab750 legacy=0x7fbf6c0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:24.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf716b7640 1 -- 172.21.15.83:0/98232276 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf6c06e730 con 0x7fbf6c0c3b70 2024-09-19T07:22:24.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.994+0000 7fbf716b7640 1 --2- 172.21.15.83:0/98232276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf6c0c3b70 0x7fbf6c152e00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c156930 tx=0x7fbf6401d820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e927612a8c3f6a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:24.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf637fe640 1 -- 172.21.15.83:0/98232276 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf640ad070 con 0x7fbf6c0c3b70 2024-09-19T07:22:24.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf637fe640 1 -- 172.21.15.83:0/98232276 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf640a7680 con 0x7fbf6c0c3b70 2024-09-19T07:22:24.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf637fe640 1 -- 172.21.15.83:0/98232276 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6400b040 con 0x7fbf6c0c3b70 2024-09-19T07:22:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 -- 172.21.15.83:0/98232276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf6c0c3b70 msgr2=0x7fbf6c152e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 --2- 172.21.15.83:0/98232276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf6c0c3b70 0x7fbf6c152e00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c156930 tx=0x7fbf6401d820 comp rx=0 tx=0).stop 2024-09-19T07:22:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 -- 172.21.15.83:0/98232276 shutdown_connections 2024-09-19T07:22:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 --2- 172.21.15.83:0/98232276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf6c0c3b70 0x7fbf6c152e00 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 --2- 172.21.15.83:0/98232276 >> v2:172.21.15.184:3300/0 conn(0x7fbf6c0ab060 0x7fbf6c1528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 -- 172.21.15.83:0/98232276 >> 172.21.15.83:0/98232276 conn(0x7fbf6c06bc80 msgr2=0x7fbf6c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 -- 172.21.15.83:0/98232276 shutdown_connections 2024-09-19T07:22:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 -- 172.21.15.83:0/98232276 wait complete. 2024-09-19T07:22:25.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 Processor -- start 2024-09-19T07:22:25.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 -- start start 2024-09-19T07:22:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf6c0ab060 0x7fbf6c0bdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf6c0ab750 0x7fbf6c1f06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf6c0c3b70 0x7fbf6c0c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c15ebd0 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c0abd40 con 0x7fbf6c0ab750 2024-09-19T07:22:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf7311a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbf6c15ed50 con 0x7fbf6c0ab060 2024-09-19T07:22:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf716b7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf6c0c3b70 0x7fbf6c0c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf716b7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf6c0c3b70 0x7fbf6c0c66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46368/0 (socket says 172.21.15.83:46368) 2024-09-19T07:22:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf716b7640 1 -- 172.21.15.83:0/4255362035 learned_addr learned my addr 172.21.15.83:0/4255362035 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf63fff640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf6c0ab750 0x7fbf6c1f06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf70eb6640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf6c0ab060 0x7fbf6c0bdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf716b7640 1 -- 172.21.15.83:0/4255362035 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf6c0ab060 msgr2=0x7fbf6c0bdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf716b7640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf6c0ab060 0x7fbf6c0bdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf716b7640 1 -- 172.21.15.83:0/4255362035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf6c0ab750 msgr2=0x7fbf6c1f06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf716b7640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf6c0ab750 0x7fbf6c1f06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf716b7640 1 -- 172.21.15.83:0/4255362035 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf6c0c0d80 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf70eb6640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf6c0ab060 0x7fbf6c0bdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:24.998+0000 7fbf716b7640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf6c0c3b70 0x7fbf6c0c66a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbf6401d290 tx=0x7fbf6401a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.002+0000 7fbf617fa640 1 -- 172.21.15.83:0/4255362035 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf640ad040 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.002+0000 7fbf617fa640 1 -- 172.21.15.83:0/4255362035 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf640b9070 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.002+0000 7fbf7311a640 1 -- 172.21.15.83:0/4255362035 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf6c0be600 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.002+0000 7fbf617fa640 1 -- 172.21.15.83:0/4255362035 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf640a8870 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.002+0000 7fbf7311a640 1 -- 172.21.15.83:0/4255362035 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf6c0c07c0 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.002+0000 7fbf7311a640 1 -- 172.21.15.83:0/4255362035 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf6c0c1ff0 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.002+0000 7fbf617fa640 1 -- 172.21.15.83:0/4255362035 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbf640b0070 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.002+0000 7fbf617fa640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf580678c0 0x7fbf58069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.002+0000 7fbf617fa640 1 -- 172.21.15.83:0/4255362035 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6895+0+0 (secure 0 0 0) 0x7fbf6411fcd0 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.002+0000 7fbf70eb6640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf580678c0 0x7fbf58069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.002+0000 7fbf70eb6640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf580678c0 0x7fbf58069d80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbf68007fd0 tx=0x7fbf68009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:25.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.006+0000 7fbf617fa640 1 -- 172.21.15.83:0/4255362035 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbf640174a0 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.202+0000 7fbf7311a640 1 -- 172.21.15.83:0/4255362035 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf6c0c2490 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.202+0000 7fbf617fa640 1 -- 172.21.15.83:0/4255362035 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5704 (secure 0 0 0) 0x7fbf640f0360 con 0x7fbf6c0c3b70 2024-09-19T07:22:25.205 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:25.205 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":26,"btime":"2024-09-19T07:22:25:139652+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:24.145751+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:22:25.206 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 26 2024-09-19T07:22:25.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.206+0000 7fbf7311a640 1 -- 172.21.15.83:0/4255362035 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf580678c0 msgr2=0x7fbf58069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:25.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.206+0000 7fbf7311a640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf580678c0 0x7fbf58069d80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbf68007fd0 tx=0x7fbf68009040 comp rx=0 tx=0).stop 2024-09-19T07:22:25.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.206+0000 7fbf7311a640 1 -- 172.21.15.83:0/4255362035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf6c0c3b70 msgr2=0x7fbf6c0c66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:25.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.206+0000 7fbf7311a640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf6c0c3b70 0x7fbf6c0c66a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbf6401d290 tx=0x7fbf6401a5c0 comp rx=0 tx=0).stop 2024-09-19T07:22:25.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.206+0000 7fbf7311a640 1 -- 172.21.15.83:0/4255362035 shutdown_connections 2024-09-19T07:22:25.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.206+0000 7fbf7311a640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf580678c0 0x7fbf58069d80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:25.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.206+0000 7fbf7311a640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf6c0c3b70 0x7fbf6c0c66a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:25.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.206+0000 7fbf7311a640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf6c0ab750 0x7fbf6c1f06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:25.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.206+0000 7fbf7311a640 1 --2- 172.21.15.83:0/4255362035 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf6c0ab060 0x7fbf6c0bdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:25.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.206+0000 7fbf7311a640 1 -- 172.21.15.83:0/4255362035 >> 172.21.15.83:0/4255362035 conn(0x7fbf6c06bc80 msgr2=0x7fbf6c151e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:25.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.206+0000 7fbf7311a640 1 -- 172.21.15.83:0/4255362035 shutdown_connections 2024-09-19T07:22:25.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:25.206+0000 7fbf7311a640 1 -- 172.21.15.83:0/4255362035 wait complete. 2024-09-19T07:22:25.222 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:22:26.204 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:26.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.326+0000 7f951ba34640 1 Processor -- start 2024-09-19T07:22:26.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.326+0000 7f951ba34640 1 -- start start 2024-09-19T07:22:26.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f951ba34640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9514071100 0x7f95141545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:26.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f951ba34640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f951406dea0 0x7f951406e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:26.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f951ba34640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95140a9e60 con 0x7f95140c28e0 2024-09-19T07:22:26.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f951ba34640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f9514071b50 con 0x7f9514071100 2024-09-19T07:22:26.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f951ba34640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f95140c1ff0 con 0x7f951406dea0 2024-09-19T07:22:26.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f95197d0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9514071100 0x7f95141545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:26.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f95197d0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9514071100 0x7f95141545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46376/0 (socket says 172.21.15.83:46376) 2024-09-19T07:22:26.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f95197d0640 1 -- 172.21.15.83:0/1503933286 learned_addr learned my addr 172.21.15.83:0/1503933286 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:26.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f9519fd1640 1 --2- 172.21.15.83:0/1503933286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f951406dea0 0x7f951406e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:26.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f95197d0640 1 -- 172.21.15.83:0/1503933286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f951406dea0 msgr2=0x7f951406e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:26.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f95197d0640 1 --2- 172.21.15.83:0/1503933286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f951406dea0 0x7f951406e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f95197d0640 1 -- 172.21.15.83:0/1503933286 >> v1:172.21.15.184:6789/0 conn(0x7f95140c28e0 legacy=0x7f9514150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:26.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f95197d0640 1 -- 172.21.15.83:0/1503933286 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95140aa140 con 0x7f9514071100 2024-09-19T07:22:26.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.330+0000 7f95197d0640 1 --2- 172.21.15.83:0/1503933286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9514071100 0x7f95141545e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9508009e50 tx=0x7f950801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=658872782c062f14 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:26.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f9507fff640 1 -- 172.21.15.83:0/1503933286 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95080a56d0 con 0x7f9514071100 2024-09-19T07:22:26.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f9507fff640 1 -- 172.21.15.83:0/1503933286 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95080a5870 con 0x7f9514071100 2024-09-19T07:22:26.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f9507fff640 1 -- 172.21.15.83:0/1503933286 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95080a5c40 con 0x7f9514071100 2024-09-19T07:22:26.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 -- 172.21.15.83:0/1503933286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9514071100 msgr2=0x7f95141545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:26.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 --2- 172.21.15.83:0/1503933286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9514071100 0x7f95141545e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9508009e50 tx=0x7f950801c960 comp rx=0 tx=0).stop 2024-09-19T07:22:26.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 -- 172.21.15.83:0/1503933286 shutdown_connections 2024-09-19T07:22:26.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 --2- 172.21.15.83:0/1503933286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f951406dea0 0x7f951406e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 --2- 172.21.15.83:0/1503933286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9514071100 0x7f95141545e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 -- 172.21.15.83:0/1503933286 >> 172.21.15.83:0/1503933286 conn(0x7f951406bc80 msgr2=0x7f951406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:26.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 -- 172.21.15.83:0/1503933286 shutdown_connections 2024-09-19T07:22:26.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 -- 172.21.15.83:0/1503933286 wait complete. 2024-09-19T07:22:26.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 Processor -- start 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 -- start start 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f951406dea0 0x7f95140c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9514071100 0x7f9514150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95140c28e0 0x7f9514145810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9514071660 con 0x7f951406dea0 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f95140714e0 con 0x7f9514071100 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9514160c60 con 0x7f95140c28e0 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f95197d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f951406dea0 0x7f95140c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f95197d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f951406dea0 0x7f95140c5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46384/0 (socket says 172.21.15.83:46384) 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f95197d0640 1 -- 172.21.15.83:0/3900273786 learned_addr learned my addr 172.21.15.83:0/3900273786 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f9519fd1640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95140c28e0 0x7f9514145810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f9518fcf640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9514071100 0x7f9514150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:26.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f95197d0640 1 -- 172.21.15.83:0/3900273786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95140c28e0 msgr2=0x7f9514145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:26.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f95197d0640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95140c28e0 0x7f9514145810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f95197d0640 1 -- 172.21.15.83:0/3900273786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9514071100 msgr2=0x7f9514150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:26.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f95197d0640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9514071100 0x7f9514150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f95197d0640 1 -- 172.21.15.83:0/3900273786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f951406cb00 con 0x7f951406dea0 2024-09-19T07:22:26.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f95197d0640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f951406dea0 0x7f95140c5be0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f95080af040 tx=0x7f9508019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:26.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f9505ffb640 1 -- 172.21.15.83:0/3900273786 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95080bc020 con 0x7f951406dea0 2024-09-19T07:22:26.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f9505ffb640 1 -- 172.21.15.83:0/3900273786 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95080b8670 con 0x7f951406dea0 2024-09-19T07:22:26.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 -- 172.21.15.83:0/3900273786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95140c61c0 con 0x7f951406dea0 2024-09-19T07:22:26.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f9505ffb640 1 -- 172.21.15.83:0/3900273786 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95080b8a60 con 0x7f951406dea0 2024-09-19T07:22:26.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.334+0000 7f951ba34640 1 -- 172.21.15.83:0/3900273786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9514148030 con 0x7f951406dea0 2024-09-19T07:22:26.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.338+0000 7f951ba34640 1 -- 172.21.15.83:0/3900273786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f951406e330 con 0x7f951406dea0 2024-09-19T07:22:26.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.338+0000 7f9505ffb640 1 -- 172.21.15.83:0/3900273786 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f95080bf020 con 0x7f951406dea0 2024-09-19T07:22:26.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.338+0000 7f9505ffb640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94f80678c0 0x7f94f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:26.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.338+0000 7f9505ffb640 1 -- 172.21.15.83:0/3900273786 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6895+0+0 (secure 0 0 0) 0x7f9508121120 con 0x7f951406dea0 2024-09-19T07:22:26.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.338+0000 7f9518fcf640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94f80678c0 0x7f94f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:26.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.338+0000 7f9518fcf640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94f80678c0 0x7f94f8069d80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f95141509a0 tx=0x7f950c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.342+0000 7f9505ffb640 1 -- 172.21.15.83:0/3900273786 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f95080c6030 con 0x7f951406dea0 2024-09-19T07:22:26.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.538+0000 7f951ba34640 1 -- 172.21.15.83:0/3900273786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95141519d0 con 0x7f951406dea0 2024-09-19T07:22:26.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.538+0000 7f9505ffb640 1 -- 172.21.15.83:0/3900273786 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5711 (secure 0 0 0) 0x7f95080165e0 con 0x7f951406dea0 2024-09-19T07:22:26.542 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:26.542 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":27,"btime":"2024-09-19T07:22:26:150322+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:26.544 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 27 2024-09-19T07:22:26.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.542+0000 7f951ba34640 1 -- 172.21.15.83:0/3900273786 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94f80678c0 msgr2=0x7f94f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:26.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.542+0000 7f951ba34640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94f80678c0 0x7f94f8069d80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f95141509a0 tx=0x7f950c008040 comp rx=0 tx=0).stop 2024-09-19T07:22:26.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.542+0000 7f951ba34640 1 -- 172.21.15.83:0/3900273786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f951406dea0 msgr2=0x7f95140c5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:26.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.542+0000 7f951ba34640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f951406dea0 0x7f95140c5be0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f95080af040 tx=0x7f9508019700 comp rx=0 tx=0).stop 2024-09-19T07:22:26.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.542+0000 7f951ba34640 1 -- 172.21.15.83:0/3900273786 shutdown_connections 2024-09-19T07:22:26.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.542+0000 7f951ba34640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94f80678c0 0x7f94f8069d80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.542+0000 7f951ba34640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95140c28e0 0x7f9514145810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.542+0000 7f951ba34640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9514071100 0x7f9514150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.542+0000 7f951ba34640 1 --2- 172.21.15.83:0/3900273786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f951406dea0 0x7f95140c5be0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.542+0000 7f951ba34640 1 -- 172.21.15.83:0/3900273786 >> 172.21.15.83:0/3900273786 conn(0x7f951406bc80 msgr2=0x7f9514150f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:26.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.542+0000 7f951ba34640 1 -- 172.21.15.83:0/3900273786 shutdown_connections 2024-09-19T07:22:26.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.542+0000 7f951ba34640 1 -- 172.21.15.83:0/3900273786 wait complete. 2024-09-19T07:22:26.560 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=4229 2024-09-19T07:22:26.561 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.d 2024-09-19T07:22:26.561 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-19T07:22:26.561 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-19T07:22:26.562 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-19T07:22:26.565 INFO:tasks.ceph.mds.d:Started 2024-09-19T07:22:26.565 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:26.647 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-19T07:22:26.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.682+0000 7fb4cad7d640 1 Processor -- start 2024-09-19T07:22:26.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.682+0000 7fb4cad7d640 1 -- start start 2024-09-19T07:22:26.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4cad7d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c4071100 0x7fb4c41545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:26.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4cad7d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4c406dea0 0x7fb4c406e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:26.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4cad7d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4c40a9e60 con 0x7fb4c40c28e0 2024-09-19T07:22:26.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4cad7d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb4c4071b50 con 0x7fb4c406dea0 2024-09-19T07:22:26.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4cad7d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb4c40c1ff0 con 0x7fb4c4071100 2024-09-19T07:22:26.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c931a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4c406dea0 0x7fb4c406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:26.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb4c40c28e0 0x7fb4c4150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48636/0 (socket says 172.21.15.83:48636) 2024-09-19T07:22:26.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c931a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4c406dea0 0x7fb4c406e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46400/0 (socket says 172.21.15.83:46400) 2024-09-19T07:22:26.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c931a640 1 -- 172.21.15.83:0/391512796 learned_addr learned my addr 172.21.15.83:0/391512796 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:26.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c8b19640 1 --2- 172.21.15.83:0/391512796 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c4071100 0x7fb4c41545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:26.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c931a640 1 -- 172.21.15.83:0/391512796 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c4071100 msgr2=0x7fb4c41545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:26.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c931a640 1 --2- 172.21.15.83:0/391512796 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c4071100 0x7fb4c41545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c931a640 1 -- 172.21.15.83:0/391512796 >> v1:172.21.15.83:6789/0 conn(0x7fb4c40c28e0 legacy=0x7fb4c4150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:26.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c931a640 1 -- 172.21.15.83:0/391512796 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c40aa140 con 0x7fb4c406dea0 2024-09-19T07:22:26.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c37fe640 1 -- 172.21.15.83:0/391512796 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1047107745 0 0) 0x7fb4c40a9e60 con 0x7fb4c40c28e0 2024-09-19T07:22:26.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c931a640 1 --2- 172.21.15.83:0/391512796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c406dea0 0x7fb4c406e350 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009e50 tx=0x7fb4bc01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a863e4b84aecff1a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:26.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c37fe640 1 -- 172.21.15.83:0/391512796 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc0a5660 con 0x7fb4c406dea0 2024-09-19T07:22:26.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c37fe640 1 -- 172.21.15.83:0/391512796 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4bc0a5800 con 0x7fb4c406dea0 2024-09-19T07:22:26.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.686+0000 7fb4c37fe640 1 -- 172.21.15.83:0/391512796 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc0a5bd0 con 0x7fb4c406dea0 2024-09-19T07:22:26.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 -- 172.21.15.83:0/391512796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c406dea0 msgr2=0x7fb4c406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:26.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 --2- 172.21.15.83:0/391512796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c406dea0 0x7fb4c406e350 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009e50 tx=0x7fb4bc01c9a0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 -- 172.21.15.83:0/391512796 shutdown_connections 2024-09-19T07:22:26.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 --2- 172.21.15.83:0/391512796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c406dea0 0x7fb4c406e350 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 --2- 172.21.15.83:0/391512796 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c4071100 0x7fb4c41545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 -- 172.21.15.83:0/391512796 >> 172.21.15.83:0/391512796 conn(0x7fb4c406bc80 msgr2=0x7fb4c406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:26.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 -- 172.21.15.83:0/391512796 shutdown_connections 2024-09-19T07:22:26.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 -- 172.21.15.83:0/391512796 wait complete. 2024-09-19T07:22:26.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 Processor -- start 2024-09-19T07:22:26.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 -- start start 2024-09-19T07:22:26.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c406dea0 0x7fb4c4156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:26.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c4071100 0x7fb4c415c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:26.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c40c28e0 0x7fb4c415cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:26.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c4071660 con 0x7fb4c406dea0 2024-09-19T07:22:26.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c40714e0 con 0x7fb4c40c28e0 2024-09-19T07:22:26.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb4c4160c60 con 0x7fb4c4071100 2024-09-19T07:22:26.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c8b19640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c406dea0 0x7fb4c4156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:26.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c8b19640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c406dea0 0x7fb4c4156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46410/0 (socket says 172.21.15.83:46410) 2024-09-19T07:22:26.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c8b19640 1 -- 172.21.15.83:0/2358457492 learned_addr learned my addr 172.21.15.83:0/2358457492 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:26.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c3fff640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c4071100 0x7fb4c415c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:26.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c931a640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c40c28e0 0x7fb4c415cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:26.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c8b19640 1 -- 172.21.15.83:0/2358457492 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c4071100 msgr2=0x7fb4c415c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:26.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c8b19640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c4071100 0x7fb4c415c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c8b19640 1 -- 172.21.15.83:0/2358457492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c40c28e0 msgr2=0x7fb4c415cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:26.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c8b19640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c40c28e0 0x7fb4c415cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c8b19640 1 -- 172.21.15.83:0/2358457492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c41ef0c0 con 0x7fb4c406dea0 2024-09-19T07:22:26.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c931a640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c40c28e0 0x7fb4c415cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:22:26.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c8b19640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c406dea0 0x7fb4c4156940 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400e960 tx=0x7fb4b400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:26.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c17fa640 1 -- 172.21.15.83:0/2358457492 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4017070 con 0x7fb4c406dea0 2024-09-19T07:22:26.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c17fa640 1 -- 172.21.15.83:0/2358457492 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4b400ca00 con 0x7fb4c406dea0 2024-09-19T07:22:26.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4c17fa640 1 -- 172.21.15.83:0/2358457492 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b400cdb0 con 0x7fb4c406dea0 2024-09-19T07:22:26.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 -- 172.21.15.83:0/2358457492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c40ac200 con 0x7fb4c406dea0 2024-09-19T07:22:26.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 -- 172.21.15.83:0/2358457492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c40ac7c0 con 0x7fb4c406dea0 2024-09-19T07:22:26.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.690+0000 7fb4cad7d640 1 -- 172.21.15.83:0/2358457492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4c406e350 con 0x7fb4c406dea0 2024-09-19T07:22:26.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.694+0000 7fb4c17fa640 1 -- 172.21.15.83:0/2358457492 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb4b401a070 con 0x7fb4c406dea0 2024-09-19T07:22:26.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.694+0000 7fb4c17fa640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4b00678c0 0x7fb4b0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:26.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.694+0000 7fb4c17fa640 1 -- 172.21.15.83:0/2358457492 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6895+0+0 (secure 0 0 0) 0x7fb4b4086020 con 0x7fb4c406dea0 2024-09-19T07:22:26.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.694+0000 7fb4c3fff640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4b00678c0 0x7fb4b0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:26.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.694+0000 7fb4c3fff640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4b00678c0 0x7fb4b0069d80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb4c41ebff0 tx=0x7fb4b8007df0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:26.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.698+0000 7fb4c17fa640 1 -- 172.21.15.83:0/2358457492 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb4b40560e0 con 0x7fb4c406dea0 2024-09-19T07:22:26.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 -- 172.21.15.83:0/2358457492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4c40c4f30 con 0x7fb4c406dea0 2024-09-19T07:22:26.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4c17fa640 1 -- 172.21.15.83:0/2358457492 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5711 (secure 0 0 0) 0x7fb4b405a3e0 con 0x7fb4c406dea0 2024-09-19T07:22:26.895 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:26.895 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":27,"btime":"2024-09-19T07:22:26:150322+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:26.896 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 27 2024-09-19T07:22:26.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 -- 172.21.15.83:0/2358457492 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4b00678c0 msgr2=0x7fb4b0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:26.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4b00678c0 0x7fb4b0069d80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb4c41ebff0 tx=0x7fb4b8007df0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 -- 172.21.15.83:0/2358457492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c406dea0 msgr2=0x7fb4c4156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:26.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c406dea0 0x7fb4c4156940 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400e960 tx=0x7fb4b400ee20 comp rx=0 tx=0).stop 2024-09-19T07:22:26.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 -- 172.21.15.83:0/2358457492 shutdown_connections 2024-09-19T07:22:26.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4b00678c0 0x7fb4b0069d80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c40c28e0 0x7fb4c415cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c4071100 0x7fb4c415c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 --2- 172.21.15.83:0/2358457492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c406dea0 0x7fb4c4156940 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:26.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 -- 172.21.15.83:0/2358457492 >> 172.21.15.83:0/2358457492 conn(0x7fb4c406bc80 msgr2=0x7fb4c4150f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:26.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 -- 172.21.15.83:0/2358457492 shutdown_connections 2024-09-19T07:22:26.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:26.894+0000 7fb4cad7d640 1 -- 172.21.15.83:0/2358457492 wait complete. 2024-09-19T07:22:26.914 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-19T07:22:27.222 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:27.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.350+0000 7fd329457640 1 Processor -- start 2024-09-19T07:22:27.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd329457640 1 -- start start 2024-09-19T07:22:27.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd329457640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd324071100 0x7fd3241545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:27.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd329457640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd32406dea0 0x7fd32406e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:27.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd329457640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3240a9e60 con 0x7fd3240c28e0 2024-09-19T07:22:27.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd329457640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd324071b50 con 0x7fd32406dea0 2024-09-19T07:22:27.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd329457640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd3240c1ff0 con 0x7fd324071100 2024-09-19T07:22:27.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd322ffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd324071100 0x7fd3241545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:27.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd3237fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd32406dea0 0x7fd32406e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:27.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd322ffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd324071100 0x7fd3241545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:55962/0 (socket says 172.21.15.83:55962) 2024-09-19T07:22:27.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd3227fc640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fd3240c28e0 0x7fd324150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:44876/0 (socket says 172.21.15.83:44876) 2024-09-19T07:22:27.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd3237fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd32406dea0 0x7fd32406e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46420/0 (socket says 172.21.15.83:46420) 2024-09-19T07:22:27.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd322ffd640 1 -- 172.21.15.83:0/1808529453 learned_addr learned my addr 172.21.15.83:0/1808529453 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:27.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd3237fe640 1 -- 172.21.15.83:0/1808529453 >> v2:172.21.15.184:3300/0 conn(0x7fd324071100 msgr2=0x7fd3241545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:27.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd3237fe640 1 --2- 172.21.15.83:0/1808529453 >> v2:172.21.15.184:3300/0 conn(0x7fd324071100 0x7fd3241545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:27.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd3237fe640 1 -- 172.21.15.83:0/1808529453 >> v1:172.21.15.184:6789/0 conn(0x7fd3240c28e0 legacy=0x7fd324150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:27.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd3237fe640 1 -- 172.21.15.83:0/1808529453 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3240aa140 con 0x7fd32406dea0 2024-09-19T07:22:27.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.354+0000 7fd3237fe640 1 --2- 172.21.15.83:0/1808529453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32406dea0 0x7fd32406e330 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd32406f620 tx=0x7fd31c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0421d396814f9d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:27.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd321ffb640 1 -- 172.21.15.83:0/1808529453 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c0b2070 con 0x7fd32406dea0 2024-09-19T07:22:27.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd321ffb640 1 -- 172.21.15.83:0/1808529453 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd31c0a5850 con 0x7fd32406dea0 2024-09-19T07:22:27.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd321ffb640 1 -- 172.21.15.83:0/1808529453 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c0a5c20 con 0x7fd32406dea0 2024-09-19T07:22:27.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 -- 172.21.15.83:0/1808529453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32406dea0 msgr2=0x7fd32406e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:27.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 --2- 172.21.15.83:0/1808529453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32406dea0 0x7fd32406e330 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd32406f620 tx=0x7fd31c01c8a0 comp rx=0 tx=0).stop 2024-09-19T07:22:27.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 -- 172.21.15.83:0/1808529453 shutdown_connections 2024-09-19T07:22:27.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 --2- 172.21.15.83:0/1808529453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32406dea0 0x7fd32406e330 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:27.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 --2- 172.21.15.83:0/1808529453 >> v2:172.21.15.184:3300/0 conn(0x7fd324071100 0x7fd3241545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:27.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 -- 172.21.15.83:0/1808529453 >> 172.21.15.83:0/1808529453 conn(0x7fd32406bc80 msgr2=0x7fd32406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:27.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 -- 172.21.15.83:0/1808529453 shutdown_connections 2024-09-19T07:22:27.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 -- 172.21.15.83:0/1808529453 wait complete. 2024-09-19T07:22:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 Processor -- start 2024-09-19T07:22:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 -- start start 2024-09-19T07:22:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32406dea0 0x7fd324149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd324071100 0x7fd32415c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3240c28e0 0x7fd3241f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd324071660 con 0x7fd32406dea0 2024-09-19T07:22:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd3240714e0 con 0x7fd3240c28e0 2024-09-19T07:22:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd324160c60 con 0x7fd324071100 2024-09-19T07:22:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd322ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32406dea0 0x7fd324149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd322ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32406dea0 0x7fd324149f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46424/0 (socket says 172.21.15.83:46424) 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd322ffd640 1 -- 172.21.15.83:0/3420752319 learned_addr learned my addr 172.21.15.83:0/3420752319 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd3227fc640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd324071100 0x7fd32415c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd3237fe640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3240c28e0 0x7fd3241f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd322ffd640 1 -- 172.21.15.83:0/3420752319 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd324071100 msgr2=0x7fd32415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd322ffd640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd324071100 0x7fd32415c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd322ffd640 1 -- 172.21.15.83:0/3420752319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3240c28e0 msgr2=0x7fd3241f1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd322ffd640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3240c28e0 0x7fd3241f1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd322ffd640 1 -- 172.21.15.83:0/3420752319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd32414a520 con 0x7fd32406dea0 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd322ffd640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32406dea0 0x7fd324149f40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd31400e890 tx=0x7fd31400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd30b7fe640 1 -- 172.21.15.83:0/3420752319 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd314017070 con 0x7fd32406dea0 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd30b7fe640 1 -- 172.21.15.83:0/3420752319 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd31400c940 con 0x7fd32406dea0 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 -- 172.21.15.83:0/3420752319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3241ee890 con 0x7fd32406dea0 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd30b7fe640 1 -- 172.21.15.83:0/3420752319 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31400cd30 con 0x7fd32406dea0 2024-09-19T07:22:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.358+0000 7fd329457640 1 -- 172.21.15.83:0/3420752319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd32415cdb0 con 0x7fd32406dea0 2024-09-19T07:22:27.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.362+0000 7fd329457640 1 -- 172.21.15.83:0/3420752319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd32406e330 con 0x7fd32406dea0 2024-09-19T07:22:27.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.362+0000 7fd30b7fe640 1 -- 172.21.15.83:0/3420752319 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd31401a070 con 0x7fd32406dea0 2024-09-19T07:22:27.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.362+0000 7fd30b7fe640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd3100678f0 0x7fd310069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:27.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.362+0000 7fd3227fc640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd3100678f0 0x7fd310069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:27.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.362+0000 7fd30b7fe640 1 -- 172.21.15.83:0/3420752319 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6895+0+0 (secure 0 0 0) 0x7fd314086300 con 0x7fd32406dea0 2024-09-19T07:22:27.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.362+0000 7fd3227fc640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd3100678f0 0x7fd310069db0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd324157ef0 tx=0x7fd318007e90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:27.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.366+0000 7fd30b7fe640 1 -- 172.21.15.83:0/3420752319 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd3140563e0 con 0x7fd32406dea0 2024-09-19T07:22:27.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.558+0000 7fd329457640 1 -- 172.21.15.83:0/3420752319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd324069840 con 0x7fd32406dea0 2024-09-19T07:22:27.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.558+0000 7fd30b7fe640 1 -- 172.21.15.83:0/3420752319 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6539 (secure 0 0 0) 0x7fd31405a6e0 con 0x7fd32406dea0 2024-09-19T07:22:27.561 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:27.561 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":28,"btime":"2024-09-19T07:22:27:162033+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:27.562 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 28 2024-09-19T07:22:27.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.562+0000 7fd329457640 1 -- 172.21.15.83:0/3420752319 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd3100678f0 msgr2=0x7fd310069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:27.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.562+0000 7fd329457640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd3100678f0 0x7fd310069db0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd324157ef0 tx=0x7fd318007e90 comp rx=0 tx=0).stop 2024-09-19T07:22:27.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.562+0000 7fd329457640 1 -- 172.21.15.83:0/3420752319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32406dea0 msgr2=0x7fd324149f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:27.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.562+0000 7fd329457640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32406dea0 0x7fd324149f40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd31400e890 tx=0x7fd31400ed50 comp rx=0 tx=0).stop 2024-09-19T07:22:27.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.562+0000 7fd329457640 1 -- 172.21.15.83:0/3420752319 shutdown_connections 2024-09-19T07:22:27.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.562+0000 7fd329457640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd3100678f0 0x7fd310069db0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:27.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.562+0000 7fd329457640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3240c28e0 0x7fd3241f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:27.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.562+0000 7fd329457640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd324071100 0x7fd32415c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:27.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.562+0000 7fd329457640 1 --2- 172.21.15.83:0/3420752319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32406dea0 0x7fd324149f40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:27.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.562+0000 7fd329457640 1 -- 172.21.15.83:0/3420752319 >> 172.21.15.83:0/3420752319 conn(0x7fd32406bc80 msgr2=0x7fd32406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:27.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.562+0000 7fd329457640 1 -- 172.21.15.83:0/3420752319 shutdown_connections 2024-09-19T07:22:27.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:27.562+0000 7fd329457640 1 -- 172.21.15.83:0/3420752319 wait complete. 2024-09-19T07:22:27.581 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:22:28.915 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:28.984 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:22:28.982+0000 7fdaf24f0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:22:29.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.046+0000 7f4683760640 1 Processor -- start 2024-09-19T07:22:29.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.046+0000 7f4683760640 1 -- start start 2024-09-19T07:22:29.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.046+0000 7f4683760640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f467c06d6f0 0x7f467c152b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:29.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.046+0000 7f4683760640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f467c14f110 0x7f467c14f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:29.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.046+0000 7f4683760640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467c14fa70 con 0x7f467c0c2fb0 2024-09-19T07:22:29.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.046+0000 7f4683760640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f467c06dff0 con 0x7f467c06d6f0 2024-09-19T07:22:29.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.046+0000 7f4683760640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f467c0c2840 con 0x7f467c14f110 2024-09-19T07:22:29.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.046+0000 7f46814fc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f467c06d6f0 0x7f467c152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.046+0000 7f4680cfb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f467c14f110 0x7f467c14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.046+0000 7f46814fc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f467c06d6f0 0x7f467c152b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:55980/0 (socket says 172.21.15.83:55980) 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.046+0000 7f4681cfd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f467c0c2fb0 0x7f467c0c3460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:44890/0 (socket says 172.21.15.83:44890) 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.046+0000 7f46814fc640 1 -- 172.21.15.83:0/2243127702 learned_addr learned my addr 172.21.15.83:0/2243127702 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.050+0000 7f4680cfb640 1 -- 172.21.15.83:0/2243127702 >> v2:172.21.15.184:3300/0 conn(0x7f467c06d6f0 msgr2=0x7f467c152b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.050+0000 7f4680cfb640 1 --2- 172.21.15.83:0/2243127702 >> v2:172.21.15.184:3300/0 conn(0x7f467c06d6f0 0x7f467c152b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.050+0000 7f4680cfb640 1 -- 172.21.15.83:0/2243127702 >> v1:172.21.15.184:6789/0 conn(0x7f467c0c2fb0 legacy=0x7f467c0c3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.050+0000 7f4680cfb640 1 -- 172.21.15.83:0/2243127702 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467c14fcf0 con 0x7f467c14f110 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.050+0000 7f4673fff640 1 -- 172.21.15.83:0/2243127702 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 90406098 0 0) 0x7f467c14fa70 con 0x7f467c0c2fb0 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.050+0000 7f46814fc640 1 --2- 172.21.15.83:0/2243127702 >> v2:172.21.15.184:3300/0 conn(0x7f467c06d6f0 0x7f467c152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.050+0000 7f4680cfb640 1 --2- 172.21.15.83:0/2243127702 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f467c14f110 0x7f467c14f530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f466c0097c0 tx=0x7f466c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1e41956c4e9ede0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.050+0000 7f4673fff640 1 -- 172.21.15.83:0/2243127702 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c0b0070 con 0x7f467c14f110 2024-09-19T07:22:29.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.050+0000 7f4673fff640 1 -- 172.21.15.83:0/2243127702 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f466c0ab600 con 0x7f467c14f110 2024-09-19T07:22:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.050+0000 7f4673fff640 1 -- 172.21.15.83:0/2243127702 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c0ab9d0 con 0x7f467c14f110 2024-09-19T07:22:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 -- 172.21.15.83:0/2243127702 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f467c14f110 msgr2=0x7f467c14f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 --2- 172.21.15.83:0/2243127702 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f467c14f110 0x7f467c14f530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f466c0097c0 tx=0x7f466c01c4a0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 -- 172.21.15.83:0/2243127702 shutdown_connections 2024-09-19T07:22:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 --2- 172.21.15.83:0/2243127702 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f467c14f110 0x7f467c14f530 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 --2- 172.21.15.83:0/2243127702 >> v2:172.21.15.184:3300/0 conn(0x7f467c06d6f0 0x7f467c152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 -- 172.21.15.83:0/2243127702 >> 172.21.15.83:0/2243127702 conn(0x7f467c0c7e40 msgr2=0x7f467c0c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 -- 172.21.15.83:0/2243127702 shutdown_connections 2024-09-19T07:22:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 -- 172.21.15.83:0/2243127702 wait complete. 2024-09-19T07:22:29.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 Processor -- start 2024-09-19T07:22:29.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 -- start start 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c06d6f0 0x7f467c15dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f467c0c2fb0 0x7f467c1ec0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f467c14f110 0x7f467c1ec610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f467c06fef0 con 0x7f467c06d6f0 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f467c0c2a90 con 0x7f467c14f110 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4683760640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f467c0c3b70 con 0x7f467c0c2fb0 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f46814fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c06d6f0 0x7f467c15dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f46814fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c06d6f0 0x7f467c15dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46428/0 (socket says 172.21.15.83:46428) 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f46814fc640 1 -- 172.21.15.83:0/3184941239 learned_addr learned my addr 172.21.15.83:0/3184941239 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4680cfb640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f467c0c2fb0 0x7f467c1ec0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f46814fc640 1 -- 172.21.15.83:0/3184941239 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f467c0c2fb0 msgr2=0x7f467c1ec0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f46814fc640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f467c0c2fb0 0x7f467c1ec0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f46814fc640 1 -- 172.21.15.83:0/3184941239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f467c14f110 msgr2=0x7f467c1ec610 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:22:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f46814fc640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f467c14f110 0x7f467c1ec610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f46814fc640 1 -- 172.21.15.83:0/3184941239 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467c0c37b0 con 0x7f467c06d6f0 2024-09-19T07:22:29.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f4680cfb640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f467c0c2fb0 0x7f467c1ec0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:22:29.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.054+0000 7f46814fc640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c06d6f0 0x7f467c15dcc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f467800e9d0 tx=0x7f467800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:29.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.058+0000 7f4671ffb640 1 -- 172.21.15.83:0/3184941239 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678017070 con 0x7f467c06d6f0 2024-09-19T07:22:29.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.058+0000 7f4671ffb640 1 -- 172.21.15.83:0/3184941239 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f467800ca60 con 0x7f467c06d6f0 2024-09-19T07:22:29.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.058+0000 7f4683760640 1 -- 172.21.15.83:0/3184941239 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f467c0ac120 con 0x7f467c06d6f0 2024-09-19T07:22:29.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.058+0000 7f4671ffb640 1 -- 172.21.15.83:0/3184941239 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467800ce10 con 0x7f467c06d6f0 2024-09-19T07:22:29.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.058+0000 7f4683760640 1 -- 172.21.15.83:0/3184941239 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f467c1f01f0 con 0x7f467c06d6f0 2024-09-19T07:22:29.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.058+0000 7f4683760640 1 -- 172.21.15.83:0/3184941239 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f467c06dad0 con 0x7f467c06d6f0 2024-09-19T07:22:29.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.058+0000 7f4671ffb640 1 -- 172.21.15.83:0/3184941239 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f467801a070 con 0x7f467c06d6f0 2024-09-19T07:22:29.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.058+0000 7f4671ffb640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f46600678c0 0x7f4660069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:29.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.058+0000 7f4671ffb640 1 -- 172.21.15.83:0/3184941239 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6895+0+0 (secure 0 0 0) 0x7f4678086010 con 0x7f467c06d6f0 2024-09-19T07:22:29.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.058+0000 7f4680cfb640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f46600678c0 0x7f4660069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:29.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.058+0000 7f4680cfb640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f46600678c0 0x7f4660069d80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f466c0ad000 tx=0x7f466c0a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:29.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.062+0000 7f4671ffb640 1 -- 172.21.15.83:0/3184941239 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4678056740 con 0x7f467c06d6f0 2024-09-19T07:22:29.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.262+0000 7f4683760640 1 -- 172.21.15.83:0/3184941239 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f467c0c8630 con 0x7f467c06d6f0 2024-09-19T07:22:29.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.262+0000 7f4671ffb640 1 -- 172.21.15.83:0/3184941239 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6539 (secure 0 0 0) 0x7f4678056040 con 0x7f467c06d6f0 2024-09-19T07:22:29.264 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:29.264 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":28,"btime":"2024-09-19T07:22:27:162033+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:29.265 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 28 2024-09-19T07:22:29.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.266+0000 7f4683760640 1 -- 172.21.15.83:0/3184941239 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f46600678c0 msgr2=0x7f4660069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:29.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.266+0000 7f4683760640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f46600678c0 0x7f4660069d80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f466c0ad000 tx=0x7f466c0a6000 comp rx=0 tx=0).stop 2024-09-19T07:22:29.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.266+0000 7f4683760640 1 -- 172.21.15.83:0/3184941239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c06d6f0 msgr2=0x7f467c15dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:29.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.266+0000 7f4683760640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c06d6f0 0x7f467c15dcc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f467800e9d0 tx=0x7f467800ee90 comp rx=0 tx=0).stop 2024-09-19T07:22:29.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.266+0000 7f4683760640 1 -- 172.21.15.83:0/3184941239 shutdown_connections 2024-09-19T07:22:29.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.266+0000 7f4683760640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f46600678c0 0x7f4660069d80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.266+0000 7f4683760640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f467c14f110 0x7f467c1ec610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.266+0000 7f4683760640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f467c0c2fb0 0x7f467c1ec0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.266+0000 7f4683760640 1 --2- 172.21.15.83:0/3184941239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c06d6f0 0x7f467c15dcc0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.266+0000 7f4683760640 1 -- 172.21.15.83:0/3184941239 >> 172.21.15.83:0/3184941239 conn(0x7f467c0c7e40 msgr2=0x7f467c14cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:29.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.266+0000 7f4683760640 1 -- 172.21.15.83:0/3184941239 shutdown_connections 2024-09-19T07:22:29.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.266+0000 7f4683760640 1 -- 172.21.15.83:0/3184941239 wait complete. 2024-09-19T07:22:29.283 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-19T07:22:29.283 INFO:tasks.mds_thrash.fs.[b]:waiting for 20 secs before thrashing 2024-09-19T07:22:29.583 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:29.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89b5df2640 1 Processor -- start 2024-09-19T07:22:29.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89b5df2640 1 -- start start 2024-09-19T07:22:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89b5df2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f89b01545e0 0x7f89b006e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89b5df2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89b0154a70 con 0x7f89b00c28e0 2024-09-19T07:22:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89b5df2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89b00a9e60 con 0x7f89b0071100 2024-09-19T07:22:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89b5df2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f89b0071b50 con 0x7f89b01545e0 2024-09-19T07:22:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89affff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f89b01545e0 0x7f89b006e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89aeffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f89b00c28e0 0x7f89b006dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48650/0 (socket says 172.21.15.83:48650) 2024-09-19T07:22:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89aeffd640 1 -- 172.21.15.83:0/3934547742 learned_addr learned my addr 172.21.15.83:0/3934547742 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89affff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f89b01545e0 0x7f89b006e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46432/0 (socket says 172.21.15.83:46432) 2024-09-19T07:22:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89affff640 1 -- 172.21.15.83:0/3934547742 >> v1:172.21.15.184:6789/0 conn(0x7f89b0071100 legacy=0x7f89b0150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89affff640 1 -- 172.21.15.83:0/3934547742 >> v1:172.21.15.83:6789/0 conn(0x7f89b00c28e0 legacy=0x7f89b006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.714+0000 7f89affff640 1 -- 172.21.15.83:0/3934547742 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b00aa140 con 0x7f89b01545e0 2024-09-19T07:22:29.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89affff640 1 --2- 172.21.15.83:0/3934547742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89b01545e0 0x7f89b006e5b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f89a8009d90 tx=0x7f89a801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f548ef24b978dd6f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:29.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89ae7fc640 1 -- 172.21.15.83:0/3934547742 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a80b2070 con 0x7f89b01545e0 2024-09-19T07:22:29.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89ae7fc640 1 -- 172.21.15.83:0/3934547742 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a80a5850 con 0x7f89b01545e0 2024-09-19T07:22:29.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89ae7fc640 1 -- 172.21.15.83:0/3934547742 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a80a5c20 con 0x7f89b01545e0 2024-09-19T07:22:29.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 -- 172.21.15.83:0/3934547742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89b01545e0 msgr2=0x7f89b006e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:29.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 --2- 172.21.15.83:0/3934547742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89b01545e0 0x7f89b006e5b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f89a8009d90 tx=0x7f89a801c8a0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 -- 172.21.15.83:0/3934547742 shutdown_connections 2024-09-19T07:22:29.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 --2- 172.21.15.83:0/3934547742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89b01545e0 0x7f89b006e5b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 -- 172.21.15.83:0/3934547742 >> 172.21.15.83:0/3934547742 conn(0x7f89b006bc80 msgr2=0x7f89b006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:29.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 -- 172.21.15.83:0/3934547742 shutdown_connections 2024-09-19T07:22:29.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 -- 172.21.15.83:0/3934547742 wait complete. 2024-09-19T07:22:29.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 Processor -- start 2024-09-19T07:22:29.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 -- start start 2024-09-19T07:22:29.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89b0071100 0x7f89b014d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:29.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89b00c28e0 0x7f89b01efa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:29.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89b01545e0 0x7f89b01effb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:29.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89b01517d0 con 0x7f89b0071100 2024-09-19T07:22:29.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f89b0151650 con 0x7f89b01545e0 2024-09-19T07:22:29.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89b5df2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f89b00714e0 con 0x7f89b00c28e0 2024-09-19T07:22:29.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89af7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89b0071100 0x7f89b014d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:29.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89af7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89b0071100 0x7f89b014d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46444/0 (socket says 172.21.15.83:46444) 2024-09-19T07:22:29.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.718+0000 7f89af7fe640 1 -- 172.21.15.83:0/40446792 learned_addr learned my addr 172.21.15.83:0/40446792 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:29.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f89affff640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89b01545e0 0x7f89b01effb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:29.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f89af7fe640 1 -- 172.21.15.83:0/40446792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89b00c28e0 msgr2=0x7f89b01efa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:29.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f89af7fe640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89b00c28e0 0x7f89b01efa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f89af7fe640 1 -- 172.21.15.83:0/40446792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89b01545e0 msgr2=0x7f89b01effb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:29.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f89af7fe640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89b01545e0 0x7f89b01effb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f89af7fe640 1 -- 172.21.15.83:0/40446792 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b0155ca0 con 0x7f89b0071100 2024-09-19T07:22:29.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f89af7fe640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89b0071100 0x7f89b014d9d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f89a400b440 tx=0x7f89a400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:29.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f8997fff640 1 -- 172.21.15.83:0/40446792 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a4015020 con 0x7f89b0071100 2024-09-19T07:22:29.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f8997fff640 1 -- 172.21.15.83:0/40446792 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a4011890 con 0x7f89b0071100 2024-09-19T07:22:29.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f89b5df2640 1 -- 172.21.15.83:0/40446792 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b0155090 con 0x7f89b0071100 2024-09-19T07:22:29.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f8997fff640 1 -- 172.21.15.83:0/40446792 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a4011c80 con 0x7f89b0071100 2024-09-19T07:22:29.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f89b5df2640 1 -- 172.21.15.83:0/40446792 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b0155930 con 0x7f89b0071100 2024-09-19T07:22:29.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f89b5df2640 1 -- 172.21.15.83:0/40446792 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89b0150f40 con 0x7f89b0071100 2024-09-19T07:22:29.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f8997fff640 1 -- 172.21.15.83:0/40446792 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f89a4018020 con 0x7f89b0071100 2024-09-19T07:22:29.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f8997fff640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f899c0678f0 0x7f899c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:29.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f89aeffd640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f899c0678f0 0x7f899c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:29.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f8997fff640 1 -- 172.21.15.83:0/40446792 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6895+0+0 (secure 0 0 0) 0x7f89a4085ef0 con 0x7f89b0071100 2024-09-19T07:22:29.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.722+0000 7f89aeffd640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f899c0678f0 0x7f899c069db0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f89a0000f70 tx=0x7f89a0007a90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:29.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.726+0000 7f8997fff640 1 -- 172.21.15.83:0/40446792 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f89a401f030 con 0x7f89b0071100 2024-09-19T07:22:29.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.922+0000 7f89b5df2640 1 -- 172.21.15.83:0/40446792 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89b00c4f30 con 0x7f89b0071100 2024-09-19T07:22:29.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.922+0000 7f8997fff640 1 -- 172.21.15.83:0/40446792 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6539 (secure 0 0 0) 0x7f89a400d040 con 0x7f89b0071100 2024-09-19T07:22:29.924 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:29.924 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":28,"btime":"2024-09-19T07:22:27:162033+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:29.924 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 28 2024-09-19T07:22:29.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.926+0000 7f89b5df2640 1 -- 172.21.15.83:0/40446792 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f899c0678f0 msgr2=0x7f899c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:29.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.926+0000 7f89b5df2640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f899c0678f0 0x7f899c069db0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f89a0000f70 tx=0x7f89a0007a90 comp rx=0 tx=0).stop 2024-09-19T07:22:29.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.926+0000 7f89b5df2640 1 -- 172.21.15.83:0/40446792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89b0071100 msgr2=0x7f89b014d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:29.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.926+0000 7f89b5df2640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89b0071100 0x7f89b014d9d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f89a400b440 tx=0x7f89a400b900 comp rx=0 tx=0).stop 2024-09-19T07:22:29.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.926+0000 7f89b5df2640 1 -- 172.21.15.83:0/40446792 shutdown_connections 2024-09-19T07:22:29.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.926+0000 7f89b5df2640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f899c0678f0 0x7f899c069db0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.926+0000 7f89b5df2640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89b01545e0 0x7f89b01effb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.926+0000 7f89b5df2640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89b00c28e0 0x7f89b01efa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.926+0000 7f89b5df2640 1 --2- 172.21.15.83:0/40446792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89b0071100 0x7f89b014d9d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:29.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.926+0000 7f89b5df2640 1 -- 172.21.15.83:0/40446792 >> 172.21.15.83:0/40446792 conn(0x7f89b006bc80 msgr2=0x7f89b006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:29.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.926+0000 7f89b5df2640 1 -- 172.21.15.83:0/40446792 shutdown_connections 2024-09-19T07:22:29.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:29.926+0000 7f89b5df2640 1 -- 172.21.15.83:0/40446792 wait complete. 2024-09-19T07:22:29.943 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:22:30.715 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:22:30.718 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:22:31.944 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:32.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe314f12640 1 Processor -- start 2024-09-19T07:22:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe314f12640 1 -- start start 2024-09-19T07:22:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe314f12640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe3100c6120 0x7fe3100c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe314f12640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3101592f0 0x7fe310157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe314f12640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe310156200 con 0x7fe310155d70 2024-09-19T07:22:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe314f12640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe3100c9930 con 0x7fe3100c6120 2024-09-19T07:22:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe314f12640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe3101564e0 con 0x7fe3101592f0 2024-09-19T07:22:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe30dd9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe310155d70 0x7fe3101575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36570/0 (socket says 172.21.15.83:36570) 2024-09-19T07:22:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe30e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe3100c6120 0x7fe3100c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe30dd9b640 1 -- 172.21.15.83:0/2656741379 learned_addr learned my addr 172.21.15.83:0/2656741379 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe30ed9d640 1 --2- 172.21.15.83:0/2656741379 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3101592f0 0x7fe310157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe30e59c640 1 -- 172.21.15.83:0/2656741379 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3101592f0 msgr2=0x7fe310157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe30e59c640 1 --2- 172.21.15.83:0/2656741379 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3101592f0 0x7fe310157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe30e59c640 1 -- 172.21.15.83:0/2656741379 >> v1:172.21.15.83:6789/0 conn(0x7fe310155d70 legacy=0x7fe3101575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.074+0000 7fe30e59c640 1 -- 172.21.15.83:0/2656741379 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3101567c0 con 0x7fe3100c6120 2024-09-19T07:22:32.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30e59c640 1 --2- 172.21.15.83:0/2656741379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3100c6120 0x7fe3100c6500 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe3000097c0 tx=0x7fe30001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ad63c5b43093e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:32.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30d59a640 1 -- 172.21.15.83:0/2656741379 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3000b0070 con 0x7fe3100c6120 2024-09-19T07:22:32.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30d59a640 1 -- 172.21.15.83:0/2656741379 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3000ab600 con 0x7fe3100c6120 2024-09-19T07:22:32.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30d59a640 1 -- 172.21.15.83:0/2656741379 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3000ab9d0 con 0x7fe3100c6120 2024-09-19T07:22:32.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 -- 172.21.15.83:0/2656741379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3100c6120 msgr2=0x7fe3100c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:32.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 --2- 172.21.15.83:0/2656741379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3100c6120 0x7fe3100c6500 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe3000097c0 tx=0x7fe30001c4a0 comp rx=0 tx=0).stop 2024-09-19T07:22:32.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 -- 172.21.15.83:0/2656741379 shutdown_connections 2024-09-19T07:22:32.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 --2- 172.21.15.83:0/2656741379 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3101592f0 0x7fe310157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:32.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 --2- 172.21.15.83:0/2656741379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3100c6120 0x7fe3100c6500 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:32.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 -- 172.21.15.83:0/2656741379 >> 172.21.15.83:0/2656741379 conn(0x7fe31006bc80 msgr2=0x7fe31006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:32.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 -- 172.21.15.83:0/2656741379 shutdown_connections 2024-09-19T07:22:32.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 -- 172.21.15.83:0/2656741379 wait complete. 2024-09-19T07:22:32.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 Processor -- start 2024-09-19T07:22:32.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 -- start start 2024-09-19T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3100c6120 0x7fe3100c1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3101592f0 0x7fe3101481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3101486e0 0x7fe31014c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe310153860 con 0x7fe3101592f0 2024-09-19T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe310162ea0 con 0x7fe3100c6120 2024-09-19T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe314f12640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe3100ab600 con 0x7fe3101486e0 2024-09-19T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3101592f0 0x7fe3101481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:32.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3101592f0 0x7fe3101481a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41348/0 (socket says 172.21.15.83:41348) 2024-09-19T07:22:32.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30dd9b640 1 -- 172.21.15.83:0/2913090222 learned_addr learned my addr 172.21.15.83:0/2913090222 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:32.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30ed9d640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3101486e0 0x7fe31014c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:32.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30e59c640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3100c6120 0x7fe3100c1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:32.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30dd9b640 1 -- 172.21.15.83:0/2913090222 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3101486e0 msgr2=0x7fe31014c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:32.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30dd9b640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3101486e0 0x7fe31014c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:32.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30dd9b640 1 -- 172.21.15.83:0/2913090222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3100c6120 msgr2=0x7fe3100c1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:32.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30dd9b640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3100c6120 0x7fe3100c1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:32.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30dd9b640 1 -- 172.21.15.83:0/2913090222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe310148fa0 con 0x7fe3101592f0 2024-09-19T07:22:32.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30e59c640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3100c6120 0x7fe3100c1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:32.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe30dd9b640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3101592f0 0x7fe3101481a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe30400c530 tx=0x7fe30400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:32.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.078+0000 7fe2feffd640 1 -- 172.21.15.83:0/2913090222 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304017070 con 0x7fe3101592f0 2024-09-19T07:22:32.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.082+0000 7fe2feffd640 1 -- 172.21.15.83:0/2913090222 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe304005cf0 con 0x7fe3101592f0 2024-09-19T07:22:32.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.082+0000 7fe314f12640 1 -- 172.21.15.83:0/2913090222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31014a3c0 con 0x7fe3101592f0 2024-09-19T07:22:32.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.082+0000 7fe2feffd640 1 -- 172.21.15.83:0/2913090222 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304007270 con 0x7fe3101592f0 2024-09-19T07:22:32.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.082+0000 7fe314f12640 1 -- 172.21.15.83:0/2913090222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe31014a980 con 0x7fe3101592f0 2024-09-19T07:22:32.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.082+0000 7fe314f12640 1 -- 172.21.15.83:0/2913090222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3100c6500 con 0x7fe3101592f0 2024-09-19T07:22:32.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.082+0000 7fe2feffd640 1 -- 172.21.15.83:0/2913090222 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe30401a070 con 0x7fe3101592f0 2024-09-19T07:22:32.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.082+0000 7fe2feffd640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe2f00678c0 0x7fe2f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:32.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.082+0000 7fe2feffd640 1 -- 172.21.15.83:0/2913090222 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6895+0+0 (secure 0 0 0) 0x7fe3040852f0 con 0x7fe3101592f0 2024-09-19T07:22:32.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.082+0000 7fe30e59c640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe2f00678c0 0x7fe2f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:32.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.082+0000 7fe30e59c640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe2f00678c0 0x7fe2f0069d80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe3000ad000 tx=0x7fe300008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:32.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.086+0000 7fe2feffd640 1 -- 172.21.15.83:0/2913090222 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe3040553d0 con 0x7fe3101592f0 2024-09-19T07:22:32.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 -- 172.21.15.83:0/2913090222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe31015a1f0 con 0x7fe3101592f0 2024-09-19T07:22:32.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe2feffd640 1 -- 172.21.15.83:0/2913090222 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6539 (secure 0 0 0) 0x7fe3040596d0 con 0x7fe3101592f0 2024-09-19T07:22:32.287 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:32.287 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":28,"btime":"2024-09-19T07:22:27:162033+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:21:10.937435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4220},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4220":{"gid":4220,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":29,"addr":"172.21.15.83:6837/4265456507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":4265456507},{"type":"v1","addr":"172.21.15.83:6837","nonce":4265456507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4220,"qdb_cluster":[4220]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:32.287 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 28 2024-09-19T07:22:32.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 -- 172.21.15.83:0/2913090222 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe2f00678c0 msgr2=0x7fe2f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:32.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe2f00678c0 0x7fe2f0069d80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe3000ad000 tx=0x7fe300008040 comp rx=0 tx=0).stop 2024-09-19T07:22:32.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 -- 172.21.15.83:0/2913090222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3101592f0 msgr2=0x7fe3101481a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:32.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3101592f0 0x7fe3101481a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe30400c530 tx=0x7fe30400c9f0 comp rx=0 tx=0).stop 2024-09-19T07:22:32.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 -- 172.21.15.83:0/2913090222 shutdown_connections 2024-09-19T07:22:32.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe2f00678c0 0x7fe2f0069d80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:32.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3101486e0 0x7fe31014c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:32.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3101592f0 0x7fe3101481a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:32.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 --2- 172.21.15.83:0/2913090222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3100c6120 0x7fe3100c1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:32.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 -- 172.21.15.83:0/2913090222 >> 172.21.15.83:0/2913090222 conn(0x7fe31006bc80 msgr2=0x7fe3101575d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:32.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 -- 172.21.15.83:0/2913090222 shutdown_connections 2024-09-19T07:22:32.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:32.286+0000 7fe314f12640 1 -- 172.21.15.83:0/2913090222 wait complete. 2024-09-19T07:22:32.305 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:22:34.306 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd81ee29640 1 Processor -- start 2024-09-19T07:22:34.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd81ee29640 1 -- start start 2024-09-19T07:22:34.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd81ee29640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd8180c28e0 0x7fd8181545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd81ee29640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd81806dea0 0x7fd81806e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd81ee29640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8180a9e60 con 0x7fd818071100 2024-09-19T07:22:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd81ee29640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd818071b50 con 0x7fd8180c28e0 2024-09-19T07:22:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd81ee29640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd8180c1ff0 con 0x7fd81806dea0 2024-09-19T07:22:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd817fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd8180c28e0 0x7fd8181545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd81cbc5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd818071100 0x7fd818150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36586/0 (socket says 172.21.15.83:36586) 2024-09-19T07:22:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd81cbc5640 1 -- 172.21.15.83:0/3820574482 learned_addr learned my addr 172.21.15.83:0/3820574482 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd81d3c6640 1 --2- 172.21.15.83:0/3820574482 >> v2:172.21.15.184:3300/0 conn(0x7fd81806dea0 0x7fd81806e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd817fff640 1 -- 172.21.15.83:0/3820574482 >> v2:172.21.15.184:3300/0 conn(0x7fd81806dea0 msgr2=0x7fd81806e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd817fff640 1 --2- 172.21.15.83:0/3820574482 >> v2:172.21.15.184:3300/0 conn(0x7fd81806dea0 0x7fd81806e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd817fff640 1 -- 172.21.15.83:0/3820574482 >> v1:172.21.15.83:6789/0 conn(0x7fd818071100 legacy=0x7fd818150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:34.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd817fff640 1 -- 172.21.15.83:0/3820574482 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8180aa140 con 0x7fd8180c28e0 2024-09-19T07:22:34.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.438+0000 7fd817fff640 1 --2- 172.21.15.83:0/3820574482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8180c28e0 0x7fd8181545e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd8180c4f00 tx=0x7fd80001c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ae3150c99978e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:34.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd8177fe640 1 -- 172.21.15.83:0/3820574482 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8000b0070 con 0x7fd8180c28e0 2024-09-19T07:22:34.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd8177fe640 1 -- 172.21.15.83:0/3820574482 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8000ab600 con 0x7fd8180c28e0 2024-09-19T07:22:34.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd8177fe640 1 -- 172.21.15.83:0/3820574482 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8000ab9d0 con 0x7fd8180c28e0 2024-09-19T07:22:34.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 -- 172.21.15.83:0/3820574482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8180c28e0 msgr2=0x7fd8181545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:34.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 --2- 172.21.15.83:0/3820574482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8180c28e0 0x7fd8181545e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd8180c4f00 tx=0x7fd80001c280 comp rx=0 tx=0).stop 2024-09-19T07:22:34.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 -- 172.21.15.83:0/3820574482 shutdown_connections 2024-09-19T07:22:34.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 --2- 172.21.15.83:0/3820574482 >> v2:172.21.15.184:3300/0 conn(0x7fd81806dea0 0x7fd81806e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 --2- 172.21.15.83:0/3820574482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8180c28e0 0x7fd8181545e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 -- 172.21.15.83:0/3820574482 >> 172.21.15.83:0/3820574482 conn(0x7fd81806bc80 msgr2=0x7fd81806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:34.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 -- 172.21.15.83:0/3820574482 shutdown_connections 2024-09-19T07:22:34.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 -- 172.21.15.83:0/3820574482 wait complete. 2024-09-19T07:22:34.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 Processor -- start 2024-09-19T07:22:34.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 -- start start 2024-09-19T07:22:34.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd81806dea0 0x7fd8181ecf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:34.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd818071100 0x7fd818157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:34.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8180c28e0 0x7fd81815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:34.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd818071660 con 0x7fd818071100 2024-09-19T07:22:34.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd8180714e0 con 0x7fd8180c28e0 2024-09-19T07:22:34.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd818160c60 con 0x7fd81806dea0 2024-09-19T07:22:34.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd817fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd818071100 0x7fd818157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:34.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd817fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd818071100 0x7fd818157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41374/0 (socket says 172.21.15.83:41374) 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd817fff640 1 -- 172.21.15.83:0/2427644157 learned_addr learned my addr 172.21.15.83:0/2427644157 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81d3c6640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8180c28e0 0x7fd81815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd817fff640 1 -- 172.21.15.83:0/2427644157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd81806dea0 msgr2=0x7fd8181ecf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd817fff640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd81806dea0 0x7fd8181ecf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd817fff640 1 -- 172.21.15.83:0/2427644157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8180c28e0 msgr2=0x7fd81815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd817fff640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8180c28e0 0x7fd81815c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd817fff640 1 -- 172.21.15.83:0/2427644157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd818153d00 con 0x7fd818071100 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd817fff640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd818071100 0x7fd818157930 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd800019eb0 tx=0x7fd80001aac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd8157fa640 1 -- 172.21.15.83:0/2427644157 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8000b0050 con 0x7fd818071100 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd8157fa640 1 -- 172.21.15.83:0/2427644157 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8000bc070 con 0x7fd818071100 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 -- 172.21.15.83:0/2427644157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd818156ef0 con 0x7fd818071100 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd8157fa640 1 -- 172.21.15.83:0/2427644157 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8000b5810 con 0x7fd818071100 2024-09-19T07:22:34.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.442+0000 7fd81ee29640 1 -- 172.21.15.83:0/2427644157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd818158200 con 0x7fd818071100 2024-09-19T07:22:34.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.446+0000 7fd81ee29640 1 -- 172.21.15.83:0/2427644157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd81806e330 con 0x7fd818071100 2024-09-19T07:22:34.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.446+0000 7fd8157fa640 1 -- 172.21.15.83:0/2427644157 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd8000b59b0 con 0x7fd818071100 2024-09-19T07:22:34.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.446+0000 7fd8157fa640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7f80678f0 0x7fd7f8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:34.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.446+0000 7fd8157fa640 1 -- 172.21.15.83:0/2427644157 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7fd800120960 con 0x7fd818071100 2024-09-19T07:22:34.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.446+0000 7fd81cbc5640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7f80678f0 0x7fd7f8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:34.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.446+0000 7fd81cbc5640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7f80678f0 0x7fd7f8069db0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd8100017a0 tx=0x7fd8100079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:34.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.450+0000 7fd8157fa640 1 -- 172.21.15.83:0/2427644157 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd800015f90 con 0x7fd818071100 2024-09-19T07:22:34.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.646+0000 7fd81ee29640 1 -- 172.21.15.83:0/2427644157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8180c4f30 con 0x7fd818071100 2024-09-19T07:22:34.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd8157fa640 1 -- 172.21.15.83:0/2427644157 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5710 (secure 0 0 0) 0x7fd8000f1030 con 0x7fd818071100 2024-09-19T07:22:34.651 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:34.651 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":30,"btime":"2024-09-19T07:22:34:188763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:33.238328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:34.651 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 30 2024-09-19T07:22:34.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd81ee29640 1 -- 172.21.15.83:0/2427644157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7f80678f0 msgr2=0x7fd7f8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:34.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd81ee29640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7f80678f0 0x7fd7f8069db0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd8100017a0 tx=0x7fd8100079b0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd81ee29640 1 -- 172.21.15.83:0/2427644157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd818071100 msgr2=0x7fd818157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:34.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd81ee29640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd818071100 0x7fd818157930 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd800019eb0 tx=0x7fd80001aac0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd81ee29640 1 -- 172.21.15.83:0/2427644157 shutdown_connections 2024-09-19T07:22:34.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd81ee29640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7f80678f0 0x7fd7f8069db0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd81ee29640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8180c28e0 0x7fd81815c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd81ee29640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd818071100 0x7fd818157930 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd81ee29640 1 --2- 172.21.15.83:0/2427644157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd81806dea0 0x7fd8181ecf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd81ee29640 1 -- 172.21.15.83:0/2427644157 >> 172.21.15.83:0/2427644157 conn(0x7fd81806bc80 msgr2=0x7fd81814f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:34.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd81ee29640 1 -- 172.21.15.83:0/2427644157 shutdown_connections 2024-09-19T07:22:34.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.650+0000 7fd81ee29640 1 -- 172.21.15.83:0/2427644157 wait complete. 2024-09-19T07:22:34.670 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-19T07:22:34.670 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:22:34.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:34.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.782+0000 7fefb8fdf640 1 Processor -- start 2024-09-19T07:22:34.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.782+0000 7fefb8fdf640 1 -- start start 2024-09-19T07:22:34.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb8fdf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fefb4150290 0x7fefb40c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:34.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb8fdf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fefb4153eb0 0x7fefb4154340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:34.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb8fdf640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefb4150670 con 0x7fefb4152b20 2024-09-19T07:22:34.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb8fdf640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fefb40716c0 con 0x7fefb4150290 2024-09-19T07:22:34.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb8fdf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fefb40c3920 con 0x7fefb4153eb0 2024-09-19T07:22:34.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fefb4150290 0x7fefb40c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:34.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fefb4150290 0x7fefb40c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41386/0 (socket says 172.21.15.83:41386) 2024-09-19T07:22:34.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb259c640 1 -- 172.21.15.83:0/942938583 learned_addr learned my addr 172.21.15.83:0/942938583 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:34.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb2d9d640 1 --2- 172.21.15.83:0/942938583 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fefb4153eb0 0x7fefb4154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:34.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb259c640 1 -- 172.21.15.83:0/942938583 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fefb4153eb0 msgr2=0x7fefb4154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:34.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb259c640 1 --2- 172.21.15.83:0/942938583 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fefb4153eb0 0x7fefb4154340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb259c640 1 -- 172.21.15.83:0/942938583 >> v1:172.21.15.184:6789/0 conn(0x7fefb4152b20 legacy=0x7fefb40c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:34.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb259c640 1 -- 172.21.15.83:0/942938583 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefb40c2880 con 0x7fefb4150290 2024-09-19T07:22:34.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb259c640 1 --2- 172.21.15.83:0/942938583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb4150290 0x7fefb40c1ff0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fefa4009870 tx=0x7fefa401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb3836dfc281899 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:34.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb159a640 1 -- 172.21.15.83:0/942938583 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa40b1070 con 0x7fefb4150290 2024-09-19T07:22:34.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb159a640 1 -- 172.21.15.83:0/942938583 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefa40ac630 con 0x7fefb4150290 2024-09-19T07:22:34.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.786+0000 7fefb159a640 1 -- 172.21.15.83:0/942938583 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa40a8050 con 0x7fefb4150290 2024-09-19T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 -- 172.21.15.83:0/942938583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb4150290 msgr2=0x7fefb40c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 --2- 172.21.15.83:0/942938583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb4150290 0x7fefb40c1ff0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fefa4009870 tx=0x7fefa401c520 comp rx=0 tx=0).stop 2024-09-19T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 -- 172.21.15.83:0/942938583 shutdown_connections 2024-09-19T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 --2- 172.21.15.83:0/942938583 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fefb4153eb0 0x7fefb4154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 --2- 172.21.15.83:0/942938583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb4150290 0x7fefb40c1ff0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 -- 172.21.15.83:0/942938583 >> 172.21.15.83:0/942938583 conn(0x7fefb406bc80 msgr2=0x7fefb406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 -- 172.21.15.83:0/942938583 shutdown_connections 2024-09-19T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 -- 172.21.15.83:0/942938583 wait complete. 2024-09-19T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 Processor -- start 2024-09-19T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 -- start start 2024-09-19T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefb4150290 0x7fefb40aa490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb4152b20 0x7fefb414d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fefb4153eb0 0x7fefb40bda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fefb40c4790 con 0x7fefb4152b20 2024-09-19T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fefb40c2530 con 0x7fefb4150290 2024-09-19T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fefb40c4910 con 0x7fefb4153eb0 2024-09-19T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb4152b20 0x7fefb414d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb4152b20 0x7fefb414d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41396/0 (socket says 172.21.15.83:41396) 2024-09-19T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb1d9b640 1 -- 172.21.15.83:0/3234506673 learned_addr learned my addr 172.21.15.83:0/3234506673 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb259c640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefb4150290 0x7fefb40aa490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb2d9d640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fefb4153eb0 0x7fefb40bda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb1d9b640 1 -- 172.21.15.83:0/3234506673 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fefb4153eb0 msgr2=0x7fefb40bda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb1d9b640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fefb4153eb0 0x7fefb40bda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb1d9b640 1 -- 172.21.15.83:0/3234506673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefb4150290 msgr2=0x7fefb40aa490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb1d9b640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefb4150290 0x7fefb40aa490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb1d9b640 1 -- 172.21.15.83:0/3234506673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefb414e4d0 con 0x7fefb4152b20 2024-09-19T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb259c640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefb4150290 0x7fefb40aa490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb1d9b640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb4152b20 0x7fefb414d870 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00c450 tx=0x7fef9c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefaaffd640 1 -- 172.21.15.83:0/3234506673 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c012830 con 0x7fefb4152b20 2024-09-19T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefaaffd640 1 -- 172.21.15.83:0/3234506673 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef9c017070 con 0x7fefb4152b20 2024-09-19T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 -- 172.21.15.83:0/3234506673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefb414e7b0 con 0x7fefb4152b20 2024-09-19T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefaaffd640 1 -- 172.21.15.83:0/3234506673 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c012c30 con 0x7fefb4152b20 2024-09-19T07:22:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 -- 172.21.15.83:0/3234506673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefb414ed70 con 0x7fefb4152b20 2024-09-19T07:22:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.790+0000 7fefb8fdf640 1 -- 172.21.15.83:0/3234506673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefb40c1ff0 con 0x7fefb4152b20 2024-09-19T07:22:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.794+0000 7fefaaffd640 1 -- 172.21.15.83:0/3234506673 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fef9c01c460 con 0x7fefb4152b20 2024-09-19T07:22:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.794+0000 7fefaaffd640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef940678c0 0x7fef94069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.794+0000 7fefaaffd640 1 -- 172.21.15.83:0/3234506673 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7fef9c085f00 con 0x7fefb4152b20 2024-09-19T07:22:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.794+0000 7fefb259c640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef940678c0 0x7fef94069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:34.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.794+0000 7fefb259c640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef940678c0 0x7fef94069d80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fefa40ae000 tx=0x7fefa40a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.798+0000 7fefaaffd640 1 -- 172.21.15.83:0/3234506673 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fef9c0565d0 con 0x7fefb4152b20 2024-09-19T07:22:34.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.990+0000 7fefb8fdf640 1 -- 172.21.15.83:0/3234506673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefb41546c0 con 0x7fefb4152b20 2024-09-19T07:22:34.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.990+0000 7fefaaffd640 1 -- 172.21.15.83:0/3234506673 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5710 (secure 0 0 0) 0x7fef9c055ed0 con 0x7fefb4152b20 2024-09-19T07:22:34.992 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:34.992 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":30,"btime":"2024-09-19T07:22:34:188763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:33.238328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:34.992 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 30 2024-09-19T07:22:34.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.994+0000 7fefb8fdf640 1 -- 172.21.15.83:0/3234506673 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef940678c0 msgr2=0x7fef94069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:34.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.994+0000 7fefb8fdf640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef940678c0 0x7fef94069d80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fefa40ae000 tx=0x7fefa40a7000 comp rx=0 tx=0).stop 2024-09-19T07:22:34.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.994+0000 7fefb8fdf640 1 -- 172.21.15.83:0/3234506673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb4152b20 msgr2=0x7fefb414d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:34.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.994+0000 7fefb8fdf640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb4152b20 0x7fefb414d870 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00c450 tx=0x7fef9c00c910 comp rx=0 tx=0).stop 2024-09-19T07:22:34.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.994+0000 7fefb8fdf640 1 -- 172.21.15.83:0/3234506673 shutdown_connections 2024-09-19T07:22:34.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.994+0000 7fefb8fdf640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef940678c0 0x7fef94069d80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.994+0000 7fefb8fdf640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fefb4153eb0 0x7fefb40bda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.994+0000 7fefb8fdf640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb4152b20 0x7fefb414d870 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.994+0000 7fefb8fdf640 1 --2- 172.21.15.83:0/3234506673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefb4150290 0x7fefb40aa490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:34.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.994+0000 7fefb8fdf640 1 -- 172.21.15.83:0/3234506673 >> 172.21.15.83:0/3234506673 conn(0x7fefb406bc80 msgr2=0x7fefb406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:34.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.994+0000 7fefb8fdf640 1 -- 172.21.15.83:0/3234506673 shutdown_connections 2024-09-19T07:22:34.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:34.994+0000 7fefb8fdf640 1 -- 172.21.15.83:0/3234506673 wait complete. 2024-09-19T07:22:35.010 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:22:34:188763+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 30, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4786": { "addr": "172.21.15.184:6835/4115191178", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 4115191178, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 4115191178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4786, "incarnation": 29, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:22:33.238328+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4786 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5025": { "addr": "172.21.15.83:6835/1774585869", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 1774585869, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 1774585869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5025, "incarnation": 24, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:22:26.150314+0000", "qdb_cluster": [ 5025 ], "qdb_leader": 5025, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5025 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 28, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:22:37.012 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:37.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe570d55640 1 Processor -- start 2024-09-19T07:22:37.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe570d55640 1 -- start start 2024-09-19T07:22:37.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe570d55640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe56c14ca60 0x7fe56c159490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:37.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe570d55640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe56c0cbb00 0x7fe56c0cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:37.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe570d55640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe56c1599d0 con 0x7fe56c0bef00 2024-09-19T07:22:37.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe570d55640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe56c0ca240 con 0x7fe56c0cbb00 2024-09-19T07:22:37.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe570d55640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe56c0c9c80 con 0x7fe56c14ca60 2024-09-19T07:22:37.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe56ad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe56c0cbb00 0x7fe56c0cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:37.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe569d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe56c0bef00 0x7fe56c156840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36602/0 (socket says 172.21.15.83:36602) 2024-09-19T07:22:37.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe569d9b640 1 -- 172.21.15.83:0/2652780840 learned_addr learned my addr 172.21.15.83:0/2652780840 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:37.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe56a59c640 1 --2- 172.21.15.83:0/2652780840 >> v2:172.21.15.184:3300/0 conn(0x7fe56c14ca60 0x7fe56c159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:37.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe56ad9d640 1 -- 172.21.15.83:0/2652780840 >> v2:172.21.15.184:3300/0 conn(0x7fe56c14ca60 msgr2=0x7fe56c159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:37.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe56ad9d640 1 --2- 172.21.15.83:0/2652780840 >> v2:172.21.15.184:3300/0 conn(0x7fe56c14ca60 0x7fe56c159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe56ad9d640 1 -- 172.21.15.83:0/2652780840 >> v1:172.21.15.83:6789/0 conn(0x7fe56c0bef00 legacy=0x7fe56c156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.146+0000 7fe56ad9d640 1 -- 172.21.15.83:0/2652780840 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe56c0c9f60 con 0x7fe56c0cbb00 2024-09-19T07:22:37.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.150+0000 7fe56ad9d640 1 --2- 172.21.15.83:0/2652780840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56c0cbb00 0x7fe56c0cbf20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe56c073250 tx=0x7fe56401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e4af7987e99e71 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:37.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.150+0000 7fe56959a640 1 -- 172.21.15.83:0/2652780840 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5640b2070 con 0x7fe56c0cbb00 2024-09-19T07:22:37.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.150+0000 7fe56959a640 1 -- 172.21.15.83:0/2652780840 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5640a57f0 con 0x7fe56c0cbb00 2024-09-19T07:22:37.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.150+0000 7fe56959a640 1 -- 172.21.15.83:0/2652780840 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5640a5bc0 con 0x7fe56c0cbb00 2024-09-19T07:22:37.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.150+0000 7fe570d55640 1 -- 172.21.15.83:0/2652780840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56c0cbb00 msgr2=0x7fe56c0cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.150+0000 7fe570d55640 1 --2- 172.21.15.83:0/2652780840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56c0cbb00 0x7fe56c0cbf20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe56c073250 tx=0x7fe56401c930 comp rx=0 tx=0).stop 2024-09-19T07:22:37.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.150+0000 7fe570d55640 1 -- 172.21.15.83:0/2652780840 shutdown_connections 2024-09-19T07:22:37.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.150+0000 7fe570d55640 1 --2- 172.21.15.83:0/2652780840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56c0cbb00 0x7fe56c0cbf20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.150+0000 7fe570d55640 1 --2- 172.21.15.83:0/2652780840 >> v2:172.21.15.184:3300/0 conn(0x7fe56c14ca60 0x7fe56c159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.150+0000 7fe570d55640 1 -- 172.21.15.83:0/2652780840 >> 172.21.15.83:0/2652780840 conn(0x7fe56c072470 msgr2=0x7fe56c072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:37.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.150+0000 7fe570d55640 1 -- 172.21.15.83:0/2652780840 shutdown_connections 2024-09-19T07:22:37.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.150+0000 7fe570d55640 1 -- 172.21.15.83:0/2652780840 wait complete. 2024-09-19T07:22:37.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe570d55640 1 Processor -- start 2024-09-19T07:22:37.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe570d55640 1 -- start start 2024-09-19T07:22:37.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe570d55640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe56c0bef00 0x7fe56c151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:37.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe570d55640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe56c0cbb00 0x7fe56c1f03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:37.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe570d55640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56c14ca60 0x7fe56c1f08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:37.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe570d55640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe56c0c98c0 con 0x7fe56c14ca60 2024-09-19T07:22:37.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe570d55640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe56c153a50 con 0x7fe56c0bef00 2024-09-19T07:22:37.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe570d55640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe56c0c9a40 con 0x7fe56c0cbb00 2024-09-19T07:22:37.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe56ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56c14ca60 0x7fe56c1f08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:37.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe56ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56c14ca60 0x7fe56c1f08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41422/0 (socket says 172.21.15.83:41422) 2024-09-19T07:22:37.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe56ad9d640 1 -- 172.21.15.83:0/1642734090 learned_addr learned my addr 172.21.15.83:0/1642734090 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:37.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe56a59c640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe56c0bef00 0x7fe56c151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:37.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe569d9b640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe56c0cbb00 0x7fe56c1f03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:37.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe56ad9d640 1 -- 172.21.15.83:0/1642734090 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe56c0cbb00 msgr2=0x7fe56c1f03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe56ad9d640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe56c0cbb00 0x7fe56c1f03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe56ad9d640 1 -- 172.21.15.83:0/1642734090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe56c0bef00 msgr2=0x7fe56c151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe56ad9d640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe56c0bef00 0x7fe56c151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe56ad9d640 1 -- 172.21.15.83:0/1642734090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe56c0c8a70 con 0x7fe56c14ca60 2024-09-19T07:22:37.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe56ad9d640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56c14ca60 0x7fe56c1f08e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe5640b0040 tx=0x7fe5640196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:37.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe55affd640 1 -- 172.21.15.83:0/1642734090 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5640b2050 con 0x7fe56c14ca60 2024-09-19T07:22:37.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe55affd640 1 -- 172.21.15.83:0/1642734090 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5640be070 con 0x7fe56c14ca60 2024-09-19T07:22:37.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe570d55640 1 -- 172.21.15.83:0/1642734090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe56c153460 con 0x7fe56c14ca60 2024-09-19T07:22:37.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe55affd640 1 -- 172.21.15.83:0/1642734090 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5640ad870 con 0x7fe56c14ca60 2024-09-19T07:22:37.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe570d55640 1 -- 172.21.15.83:0/1642734090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe56c1519a0 con 0x7fe56c14ca60 2024-09-19T07:22:37.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.154+0000 7fe570d55640 1 -- 172.21.15.83:0/1642734090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe56c0cbf20 con 0x7fe56c14ca60 2024-09-19T07:22:37.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.158+0000 7fe55affd640 1 -- 172.21.15.83:0/1642734090 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe5640ada10 con 0x7fe56c14ca60 2024-09-19T07:22:37.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.158+0000 7fe55affd640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe54c0678c0 0x7fe54c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:37.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.158+0000 7fe55affd640 1 -- 172.21.15.83:0/1642734090 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7fe564121b10 con 0x7fe56c14ca60 2024-09-19T07:22:37.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.158+0000 7fe56a59c640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe54c0678c0 0x7fe54c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:37.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.158+0000 7fe56a59c640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe54c0678c0 0x7fe54c069d80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009ac0 tx=0x7fe55c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:37.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.162+0000 7fe55affd640 1 -- 172.21.15.83:0/1642734090 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe564016520 con 0x7fe56c14ca60 2024-09-19T07:22:37.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.414+0000 7fe570d55640 1 -- 172.21.15.83:0/1642734090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe56c15f840 con 0x7fe56c14ca60 2024-09-19T07:22:37.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.414+0000 7fe55affd640 1 -- 172.21.15.83:0/1642734090 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5714 (secure 0 0 0) 0x7fe5640add00 con 0x7fe56c14ca60 2024-09-19T07:22:37.418 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:37.418 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"btime":"2024-09-19T07:22:36:207741+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:37.418 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-09-19T07:22:37.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.418+0000 7fe570d55640 1 -- 172.21.15.83:0/1642734090 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe54c0678c0 msgr2=0x7fe54c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.418+0000 7fe570d55640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe54c0678c0 0x7fe54c069d80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009ac0 tx=0x7fe55c009190 comp rx=0 tx=0).stop 2024-09-19T07:22:37.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.418+0000 7fe570d55640 1 -- 172.21.15.83:0/1642734090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56c14ca60 msgr2=0x7fe56c1f08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.418+0000 7fe570d55640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56c14ca60 0x7fe56c1f08e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe5640b0040 tx=0x7fe5640196d0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.418+0000 7fe570d55640 1 -- 172.21.15.83:0/1642734090 shutdown_connections 2024-09-19T07:22:37.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.418+0000 7fe570d55640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe54c0678c0 0x7fe54c069d80 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009ac0 tx=0x7fe55c009190 comp rx=0 tx=0).stop 2024-09-19T07:22:37.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.418+0000 7fe570d55640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56c14ca60 0x7fe56c1f08e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.418+0000 7fe570d55640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe56c0cbb00 0x7fe56c1f03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.418+0000 7fe570d55640 1 --2- 172.21.15.83:0/1642734090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe56c0bef00 0x7fe56c151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.418+0000 7fe570d55640 1 -- 172.21.15.83:0/1642734090 >> 172.21.15.83:0/1642734090 conn(0x7fe56c072470 msgr2=0x7fe56c153d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:37.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.418+0000 7fe570d55640 1 -- 172.21.15.83:0/1642734090 shutdown_connections 2024-09-19T07:22:37.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.418+0000 7fe570d55640 1 -- 172.21.15.83:0/1642734090 wait complete. 2024-09-19T07:22:37.449 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=4220 2024-09-19T07:22:37.449 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.c 2024-09-19T07:22:37.449 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-19T07:22:37.449 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:22:37.449 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:22:37.451 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:22:37.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:37.517 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:22:37.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f6119284640 1 Processor -- start 2024-09-19T07:22:37.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f6119284640 1 -- start start 2024-09-19T07:22:37.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f6119284640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f61140c3130 0x7f61140ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:37.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f6119284640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61140c3c20 con 0x7f611406da00 2024-09-19T07:22:37.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f6119284640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61140c4990 con 0x7f611406de70 2024-09-19T07:22:37.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f6119284640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f61140c4c70 con 0x7f61140c3130 2024-09-19T07:22:37.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f6112ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f61140c3130 0x7f61140ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:37.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f61127fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f611406da00 0x7f61140c3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36628/0 (socket says 172.21.15.83:36628) 2024-09-19T07:22:37.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f61127fc640 1 -- 172.21.15.83:0/1695886050 learned_addr learned my addr 172.21.15.83:0/1695886050 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:37.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f6112ffd640 1 -- 172.21.15.83:0/1695886050 >> v1:172.21.15.184:6789/0 conn(0x7f611406de70 legacy=0x7f61141563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f6112ffd640 1 -- 172.21.15.83:0/1695886050 >> v1:172.21.15.83:6789/0 conn(0x7f611406da00 legacy=0x7f61140c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f6111ffb640 1 -- 172.21.15.83:0/1695886050 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3922255015 0 0) 0x7f61140c3c20 con 0x7f611406da00 2024-09-19T07:22:37.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.578+0000 7f6112ffd640 1 -- 172.21.15.83:0/1695886050 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61140ab7b0 con 0x7f61140c3130 2024-09-19T07:22:37.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.582+0000 7f6112ffd640 1 --2- 172.21.15.83:0/1695886050 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61140c3130 0x7f61140ac290 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f61040097c0 tx=0x7f610401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc69b4ea8e2519bd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:37.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.582+0000 7f6111ffb640 1 -- 172.21.15.83:0/1695886050 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61040b0070 con 0x7f61140c3130 2024-09-19T07:22:37.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.582+0000 7f6111ffb640 1 -- 172.21.15.83:0/1695886050 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61040ab600 con 0x7f61140c3130 2024-09-19T07:22:37.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.582+0000 7f6111ffb640 1 -- 172.21.15.83:0/1695886050 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61040ab9d0 con 0x7f61140c3130 2024-09-19T07:22:37.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.582+0000 7f6119284640 1 -- 172.21.15.83:0/1695886050 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61140c3130 msgr2=0x7f61140ac290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.582+0000 7f6119284640 1 --2- 172.21.15.83:0/1695886050 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61140c3130 0x7f61140ac290 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f61040097c0 tx=0x7f610401c410 comp rx=0 tx=0).stop 2024-09-19T07:22:37.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.582+0000 7f6119284640 1 -- 172.21.15.83:0/1695886050 shutdown_connections 2024-09-19T07:22:37.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.582+0000 7f6119284640 1 --2- 172.21.15.83:0/1695886050 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61140c3130 0x7f61140ac290 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.582+0000 7f6119284640 1 -- 172.21.15.83:0/1695886050 >> 172.21.15.83:0/1695886050 conn(0x7f61140c5060 msgr2=0x7f61140c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:37.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.582+0000 7f6119284640 1 -- 172.21.15.83:0/1695886050 shutdown_connections 2024-09-19T07:22:37.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.582+0000 7f6119284640 1 -- 172.21.15.83:0/1695886050 wait complete. 2024-09-19T07:22:37.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6119284640 1 Processor -- start 2024-09-19T07:22:37.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6119284640 1 -- start start 2024-09-19T07:22:37.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6119284640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f611406da00 0x7f61141502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:37.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6119284640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f611406de70 0x7f61140beb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:37.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6119284640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61140c3130 0x7f611414b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6119284640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f61140ad6a0 con 0x7f611406de70 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6119284640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f61140ad520 con 0x7f61140c3130 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6119284640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6114160820 con 0x7f611406da00 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f61127fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f611406de70 0x7f61140beb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f61127fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f611406de70 0x7f61140beb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41464/0 (socket says 172.21.15.83:41464) 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f61127fc640 1 -- 172.21.15.83:0/1942728998 learned_addr learned my addr 172.21.15.83:0/1942728998 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f61137fe640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61140c3130 0x7f611414b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6112ffd640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f611406da00 0x7f61141502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f61127fc640 1 -- 172.21.15.83:0/1942728998 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f611406da00 msgr2=0x7f61141502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f61127fc640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f611406da00 0x7f61141502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f61127fc640 1 -- 172.21.15.83:0/1942728998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61140c3130 msgr2=0x7f611414b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f61127fc640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61140c3130 0x7f611414b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f61127fc640 1 -- 172.21.15.83:0/1942728998 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61141ef510 con 0x7f611406de70 2024-09-19T07:22:37.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6112ffd640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f611406da00 0x7f61141502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:37.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f61127fc640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f611406de70 0x7f61140beb40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f610800c4d0 tx=0x7f610800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:37.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f60fb7fe640 1 -- 172.21.15.83:0/1942728998 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6108018070 con 0x7f611406de70 2024-09-19T07:22:37.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f60fb7fe640 1 -- 172.21.15.83:0/1942728998 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61080074e0 con 0x7f611406de70 2024-09-19T07:22:37.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6119284640 1 -- 172.21.15.83:0/1942728998 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61141ef7d0 con 0x7f611406de70 2024-09-19T07:22:37.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f60fb7fe640 1 -- 172.21.15.83:0/1942728998 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6108004a30 con 0x7f611406de70 2024-09-19T07:22:37.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6119284640 1 -- 172.21.15.83:0/1942728998 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61141efd90 con 0x7f611406de70 2024-09-19T07:22:37.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.586+0000 7f6119284640 1 -- 172.21.15.83:0/1942728998 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61140ac290 con 0x7f611406de70 2024-09-19T07:22:37.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.590+0000 7f60fb7fe640 1 -- 172.21.15.83:0/1942728998 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f610801b070 con 0x7f611406de70 2024-09-19T07:22:37.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.590+0000 7f60fb7fe640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61000678c0 0x7f6100069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:37.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.590+0000 7f60fb7fe640 1 -- 172.21.15.83:0/1942728998 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7f6108085440 con 0x7f611406de70 2024-09-19T07:22:37.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.590+0000 7f6112ffd640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61000678c0 0x7f6100069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:37.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.590+0000 7f6112ffd640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61000678c0 0x7f6100069d80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f61040ad000 tx=0x7f6104008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:37.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.594+0000 7f60fb7fe640 1 -- 172.21.15.83:0/1942728998 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6108055b10 con 0x7f611406de70 2024-09-19T07:22:37.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 -- 172.21.15.83:0/1942728998 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6114075b40 con 0x7f611406de70 2024-09-19T07:22:37.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f60fb7fe640 1 -- 172.21.15.83:0/1942728998 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5714 (secure 0 0 0) 0x7f6108055410 con 0x7f611406de70 2024-09-19T07:22:37.831 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:37.831 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"btime":"2024-09-19T07:22:36:207741+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:37.832 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-09-19T07:22:37.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 -- 172.21.15.83:0/1942728998 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61000678c0 msgr2=0x7f6100069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61000678c0 0x7f6100069d80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f61040ad000 tx=0x7f6104008040 comp rx=0 tx=0).stop 2024-09-19T07:22:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 -- 172.21.15.83:0/1942728998 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f611406de70 msgr2=0x7f61140beb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f611406de70 0x7f61140beb40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f610800c4d0 tx=0x7f610800c990 comp rx=0 tx=0).stop 2024-09-19T07:22:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 -- 172.21.15.83:0/1942728998 shutdown_connections 2024-09-19T07:22:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61000678c0 0x7f6100069d80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61140c3130 0x7f611414b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f611406de70 0x7f61140beb40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 --2- 172.21.15.83:0/1942728998 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f611406da00 0x7f61141502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 -- 172.21.15.83:0/1942728998 >> 172.21.15.83:0/1942728998 conn(0x7f61140c5060 msgr2=0x7f611406e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 -- 172.21.15.83:0/1942728998 shutdown_connections 2024-09-19T07:22:37.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:37.830+0000 7f6119284640 1 -- 172.21.15.83:0/1942728998 wait complete. 2024-09-19T07:22:37.850 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-19T07:22:39.851 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:39.946 INFO:tasks.ceph.mds.b.smithi184.stderr:2024-09-19T07:22:39.947+0000 7f179ee74640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:22:39.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.990+0000 7fa12e89d640 1 Processor -- start 2024-09-19T07:22:39.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12e89d640 1 -- start start 2024-09-19T07:22:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12e89d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fa1280c2c30 0x7fa128152e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12e89d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1280aaa60 0x7fa1281533d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12e89d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa128159c60 con 0x7fa12806f070 2024-09-19T07:22:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12e89d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fa12815a160 con 0x7fa1280c2c30 2024-09-19T07:22:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12e89d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa1280c4940 con 0x7fa1280aaa60 2024-09-19T07:22:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12ce3a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa12806f070 0x7fa128155ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36636/0 (socket says 172.21.15.83:36636) 2024-09-19T07:22:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12ce3a640 1 -- 172.21.15.83:0/2672216728 learned_addr learned my addr 172.21.15.83:0/2672216728 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:39.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa1277fe640 1 --2- 172.21.15.83:0/2672216728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1280aaa60 0x7fa1281533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:39.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690677068 0 0) 0x7fa128159c60 con 0x7fa12806f070 2024-09-19T07:22:39.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa110003530 con 0x7fa12806f070 2024-09-19T07:22:39.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1115731495 0 0) 0x7fa110003530 con 0x7fa12806f070 2024-09-19T07:22:39.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1280aaa60 msgr2=0x7fa1281533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:39.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 --2- 172.21.15.83:0/2672216728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1280aaa60 0x7fa1281533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:39.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 >> v2:172.21.15.184:3300/0 conn(0x7fa1280c2c30 msgr2=0x7fa128152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:39.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 --2- 172.21.15.83:0/2672216728 >> v2:172.21.15.184:3300/0 conn(0x7fa1280c2c30 0x7fa128152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:39.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1280c4b90 con 0x7fa12806f070 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fa120001a00 con 0x7fa12806f070 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 >> v1:172.21.15.83:6789/0 conn(0x7fa12806f070 legacy=0x7fa128155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 --2- 172.21.15.83:0/2672216728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1100039c0 0x7fa110003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 --2- 172.21.15.83:0/2672216728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1280aaa60 0x7fa110004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 --2- 172.21.15.83:0/2672216728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa110004d50 0x7fa1100051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa12815a160 con 0x7fa110004d50 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa1280c4940 con 0x7fa1100039c0 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa11000bb90 con 0x7fa1280aaa60 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12ce3a640 1 --2- 172.21.15.83:0/2672216728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa110004d50 0x7fa1100051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12ce3a640 1 -- 172.21.15.83:0/2672216728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1280aaa60 msgr2=0x7fa110004810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12ce3a640 1 --2- 172.21.15.83:0/2672216728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1280aaa60 0x7fa110004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12ce3a640 1 -- 172.21.15.83:0/2672216728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1100039c0 msgr2=0x7fa110003e00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:22:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12ce3a640 1 --2- 172.21.15.83:0/2672216728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1100039c0 0x7fa110003e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.994+0000 7fa12ce3a640 1 -- 172.21.15.83:0/2672216728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa110003530 con 0x7fa110004d50 2024-09-19T07:22:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa12ce3a640 1 --2- 172.21.15.83:0/2672216728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa110004d50 0x7fa1100051d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa12000a850 tx=0x7fa12001fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb346b45f00fa80b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12000a4a0 con 0x7fa110004d50 2024-09-19T07:22:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1200aa730 con 0x7fa110004d50 2024-09-19T07:22:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa126ffd640 1 -- 172.21.15.83:0/2672216728 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1200aab20 con 0x7fa110004d50 2024-09-19T07:22:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa12e89d640 1 -- 172.21.15.83:0/2672216728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa110004d50 msgr2=0x7fa1100051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa12e89d640 1 --2- 172.21.15.83:0/2672216728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa110004d50 0x7fa1100051d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa12000a850 tx=0x7fa12001fae0 comp rx=0 tx=0).stop 2024-09-19T07:22:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa12ce3a640 1 -- 172.21.15.83:0/2672216728 reap_dead start 2024-09-19T07:22:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa12e89d640 1 -- 172.21.15.83:0/2672216728 shutdown_connections 2024-09-19T07:22:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa12e89d640 1 -- 172.21.15.83:0/2672216728 >> 172.21.15.83:0/2672216728 conn(0x7fa1280c95a0 msgr2=0x7fa1280c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa12e89d640 1 -- 172.21.15.83:0/2672216728 shutdown_connections 2024-09-19T07:22:40.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa12e89d640 1 -- 172.21.15.83:0/2672216728 wait complete. 2024-09-19T07:22:40.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa12e89d640 1 Processor -- start 2024-09-19T07:22:40.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:39.998+0000 7fa12e89d640 1 -- start start 2024-09-19T07:22:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12e89d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa12806e500 0x7fa128153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12e89d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1281e2e30 0x7fa128154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12e89d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa128156870 0x7fa1281eee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12e89d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa1280c3a90 con 0x7fa128156870 2024-09-19T07:22:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12e89d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa1280aa780 con 0x7fa12806e500 2024-09-19T07:22:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12e89d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa1280c3c10 con 0x7fa1281e2e30 2024-09-19T07:22:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12ce3a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa128156870 0x7fa1281eee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12ce3a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa128156870 0x7fa1281eee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41484/0 (socket says 172.21.15.83:41484) 2024-09-19T07:22:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12ce3a640 1 -- 172.21.15.83:0/3609937461 learned_addr learned my addr 172.21.15.83:0/3609937461 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa127fff640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa12806e500 0x7fa128153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa1277fe640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1281e2e30 0x7fa128154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12ce3a640 1 -- 172.21.15.83:0/3609937461 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1281e2e30 msgr2=0x7fa128154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12ce3a640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1281e2e30 0x7fa128154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12ce3a640 1 -- 172.21.15.83:0/3609937461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa12806e500 msgr2=0x7fa128153fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12ce3a640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa12806e500 0x7fa128153fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12ce3a640 1 -- 172.21.15.83:0/3609937461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa128155580 con 0x7fa128156870 2024-09-19T07:22:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12ce3a640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa128156870 0x7fa1281eee20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa1200211c0 tx=0x7fa110005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa124ff9640 1 -- 172.21.15.83:0/3609937461 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa120002000 con 0x7fa128156870 2024-09-19T07:22:40.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa124ff9640 1 -- 172.21.15.83:0/3609937461 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1200aa8e0 con 0x7fa128156870 2024-09-19T07:22:40.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12e89d640 1 -- 172.21.15.83:0/3609937461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa128155860 con 0x7fa128156870 2024-09-19T07:22:40.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa124ff9640 1 -- 172.21.15.83:0/3609937461 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1200aac90 con 0x7fa128156870 2024-09-19T07:22:40.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12e89d640 1 -- 172.21.15.83:0/3609937461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa12806e8e0 con 0x7fa128156870 2024-09-19T07:22:40.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa12e89d640 1 -- 172.21.15.83:0/3609937461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1280c3130 con 0x7fa128156870 2024-09-19T07:22:40.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa124ff9640 1 -- 172.21.15.83:0/3609937461 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa1200b2400 con 0x7fa128156870 2024-09-19T07:22:40.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa124ff9640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1140678c0 0x7fa114069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:40.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa124ff9640 1 -- 172.21.15.83:0/3609937461 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7fa1201237c0 con 0x7fa128156870 2024-09-19T07:22:40.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa127fff640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1140678c0 0x7fa114069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:40.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.002+0000 7fa127fff640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1140678c0 0x7fa114069d80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa118006fd0 tx=0x7fa1180090e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:40.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.010+0000 7fa124ff9640 1 -- 172.21.15.83:0/3609937461 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa1200c1030 con 0x7fa128156870 2024-09-19T07:22:40.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.246+0000 7fa12e89d640 1 -- 172.21.15.83:0/3609937461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1280c3130 con 0x7fa128156870 2024-09-19T07:22:40.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.250+0000 7fa124ff9640 1 -- 172.21.15.83:0/3609937461 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6539 (secure 0 0 0) 0x7fa1200ae030 con 0x7fa128156870 2024-09-19T07:22:40.253 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:40.253 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"btime":"2024-09-19T07:22:38:226763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:40.253 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 33 2024-09-19T07:22:40.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.254+0000 7fa12e89d640 1 -- 172.21.15.83:0/3609937461 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1140678c0 msgr2=0x7fa114069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:40.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.254+0000 7fa12e89d640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1140678c0 0x7fa114069d80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa118006fd0 tx=0x7fa1180090e0 comp rx=0 tx=0).stop 2024-09-19T07:22:40.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.254+0000 7fa12e89d640 1 -- 172.21.15.83:0/3609937461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa128156870 msgr2=0x7fa1281eee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:40.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.254+0000 7fa12e89d640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa128156870 0x7fa1281eee20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa1200211c0 tx=0x7fa110005840 comp rx=0 tx=0).stop 2024-09-19T07:22:40.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.254+0000 7fa12e89d640 1 -- 172.21.15.83:0/3609937461 shutdown_connections 2024-09-19T07:22:40.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.254+0000 7fa12e89d640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1140678c0 0x7fa114069d80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:40.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.254+0000 7fa12e89d640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa128156870 0x7fa1281eee20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:40.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.254+0000 7fa12e89d640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1281e2e30 0x7fa128154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:40.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.254+0000 7fa12e89d640 1 --2- 172.21.15.83:0/3609937461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa12806e500 0x7fa128153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:40.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.254+0000 7fa12e89d640 1 -- 172.21.15.83:0/3609937461 >> 172.21.15.83:0/3609937461 conn(0x7fa1280c95a0 msgr2=0x7fa1280ac0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:40.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.254+0000 7fa12e89d640 1 -- 172.21.15.83:0/3609937461 shutdown_connections 2024-09-19T07:22:40.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:40.254+0000 7fa12e89d640 1 -- 172.21.15.83:0/3609937461 wait complete. 2024-09-19T07:22:40.274 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-19T07:22:40.274 INFO:tasks.mds_thrash.fs.[a]:waiting for 56 secs before thrashing 2024-09-19T07:22:49.284 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:49.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.442+0000 7f619803f640 1 Processor -- start 2024-09-19T07:22:49.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.442+0000 7f619803f640 1 -- start start 2024-09-19T07:22:49.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f619803f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6190070140 0x7f61900c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:49.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f619803f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f61901579a0 0x7f6190150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:49.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f619803f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6190070830 con 0x7f619006cef0 2024-09-19T07:22:49.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f619803f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6190153cf0 con 0x7f61901579a0 2024-09-19T07:22:49.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f619803f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6190157df0 con 0x7f6190070140 2024-09-19T07:22:49.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f61965dc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f61901579a0 0x7f6190150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:49.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f6195ddb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f619006cef0 0x7f6190154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33466/0 (socket says 172.21.15.83:33466) 2024-09-19T07:22:49.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f6195ddb640 1 -- 172.21.15.83:0/152821348 learned_addr learned my addr 172.21.15.83:0/152821348 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:49.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f61955da640 1 --2- 172.21.15.83:0/152821348 >> v2:172.21.15.184:3300/0 conn(0x7f6190070140 0x7f61900c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:49.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f61965dc640 1 -- 172.21.15.83:0/152821348 >> v2:172.21.15.184:3300/0 conn(0x7f6190070140 msgr2=0x7f61900c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:49.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f61965dc640 1 --2- 172.21.15.83:0/152821348 >> v2:172.21.15.184:3300/0 conn(0x7f6190070140 0x7f61900c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f61965dc640 1 -- 172.21.15.83:0/152821348 >> v1:172.21.15.83:6789/0 conn(0x7f619006cef0 legacy=0x7f6190154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:49.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f61965dc640 1 -- 172.21.15.83:0/152821348 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61901539a0 con 0x7f61901579a0 2024-09-19T07:22:49.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f6194dd9640 1 -- 172.21.15.83:0/152821348 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3526188398 0 0) 0x7f6190070830 con 0x7f619006cef0 2024-09-19T07:22:49.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f61965dc640 1 --2- 172.21.15.83:0/152821348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61901579a0 0x7f6190150ba0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f61900c2920 tx=0x7f618801c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc895f13cb608cf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:49.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f6194dd9640 1 -- 172.21.15.83:0/152821348 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61880b2070 con 0x7f61901579a0 2024-09-19T07:22:49.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f6194dd9640 1 -- 172.21.15.83:0/152821348 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61880a57c0 con 0x7f61901579a0 2024-09-19T07:22:49.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f6194dd9640 1 -- 172.21.15.83:0/152821348 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61880a5b90 con 0x7f61901579a0 2024-09-19T07:22:49.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f619803f640 1 -- 172.21.15.83:0/152821348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61901579a0 msgr2=0x7f6190150ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:49.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.446+0000 7f619803f640 1 --2- 172.21.15.83:0/152821348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61901579a0 0x7f6190150ba0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f61900c2920 tx=0x7f618801c810 comp rx=0 tx=0).stop 2024-09-19T07:22:49.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 -- 172.21.15.83:0/152821348 shutdown_connections 2024-09-19T07:22:49.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 --2- 172.21.15.83:0/152821348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61901579a0 0x7f6190150ba0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 --2- 172.21.15.83:0/152821348 >> v2:172.21.15.184:3300/0 conn(0x7f6190070140 0x7f61900c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 -- 172.21.15.83:0/152821348 >> 172.21.15.83:0/152821348 conn(0x7f61900c5090 msgr2=0x7f61900c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:49.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 -- 172.21.15.83:0/152821348 shutdown_connections 2024-09-19T07:22:49.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 -- 172.21.15.83:0/152821348 wait complete. 2024-09-19T07:22:49.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 Processor -- start 2024-09-19T07:22:49.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 -- start start 2024-09-19T07:22:49.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f619006cef0 0x7f6190148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6190070140 0x7f61900c3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61901579a0 0x7f61900c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f619006d2d0 con 0x7f61901579a0 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f619006f8a0 con 0x7f6190070140 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f619006d450 con 0x7f619006cef0 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61965dc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61901579a0 0x7f61900c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61965dc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61901579a0 0x7f61900c73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45650/0 (socket says 172.21.15.83:45650) 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61965dc640 1 -- 172.21.15.83:0/999846160 learned_addr learned my addr 172.21.15.83:0/999846160 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f6195ddb640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f619006cef0 0x7f6190148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61955da640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6190070140 0x7f61900c3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61965dc640 1 -- 172.21.15.83:0/999846160 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f619006cef0 msgr2=0x7f6190148af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61965dc640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f619006cef0 0x7f6190148af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61965dc640 1 -- 172.21.15.83:0/999846160 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6190070140 msgr2=0x7f61900c3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:49.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61965dc640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6190070140 0x7f61900c3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61965dc640 1 -- 172.21.15.83:0/999846160 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61901efd20 con 0x7f61901579a0 2024-09-19T07:22:49.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f6195ddb640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f619006cef0 0x7f6190148af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:22:49.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61965dc640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61901579a0 0x7f61900c73d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6188019420 tx=0x7f61880195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:49.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61867fc640 1 -- 172.21.15.83:0/999846160 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61880b2050 con 0x7f61901579a0 2024-09-19T07:22:49.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61867fc640 1 -- 172.21.15.83:0/999846160 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61880b3040 con 0x7f61901579a0 2024-09-19T07:22:49.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f61867fc640 1 -- 172.21.15.83:0/999846160 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6188009450 con 0x7f61901579a0 2024-09-19T07:22:49.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 -- 172.21.15.83:0/999846160 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61901f0000 con 0x7f61901579a0 2024-09-19T07:22:49.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.450+0000 7f619803f640 1 -- 172.21.15.83:0/999846160 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61901f05c0 con 0x7f61901579a0 2024-09-19T07:22:49.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.454+0000 7f619803f640 1 -- 172.21.15.83:0/999846160 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61900c2f90 con 0x7f61901579a0 2024-09-19T07:22:49.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.454+0000 7f61867fc640 1 -- 172.21.15.83:0/999846160 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f61880c0020 con 0x7f61901579a0 2024-09-19T07:22:49.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.454+0000 7f61867fc640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61740678f0 0x7f6174069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:49.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.454+0000 7f61867fc640 1 -- 172.21.15.83:0/999846160 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7f6188120b60 con 0x7f61901579a0 2024-09-19T07:22:49.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.454+0000 7f6195ddb640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61740678f0 0x7f6174069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:49.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.454+0000 7f6195ddb640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61740678f0 0x7f6174069db0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f618c000f70 tx=0x7f618c007cd0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:49.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.458+0000 7f61867fc640 1 -- 172.21.15.83:0/999846160 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6188016520 con 0x7f61901579a0 2024-09-19T07:22:49.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.682+0000 7f619803f640 1 -- 172.21.15.83:0/999846160 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f619015e100 con 0x7f61901579a0 2024-09-19T07:22:49.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.682+0000 7f61867fc640 1 -- 172.21.15.83:0/999846160 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6539 (secure 0 0 0) 0x7f61880f1350 con 0x7f61901579a0 2024-09-19T07:22:49.684 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:49.684 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"btime":"2024-09-19T07:22:38:226763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:49.685 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 33 2024-09-19T07:22:49.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.686+0000 7f619803f640 1 -- 172.21.15.83:0/999846160 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61740678f0 msgr2=0x7f6174069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:49.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.686+0000 7f619803f640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61740678f0 0x7f6174069db0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f618c000f70 tx=0x7f618c007cd0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.686+0000 7f619803f640 1 -- 172.21.15.83:0/999846160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61901579a0 msgr2=0x7f61900c73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:49.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.686+0000 7f619803f640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61901579a0 0x7f61900c73d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6188019420 tx=0x7f61880195b0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.686+0000 7f619803f640 1 -- 172.21.15.83:0/999846160 shutdown_connections 2024-09-19T07:22:49.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.686+0000 7f619803f640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61740678f0 0x7f6174069db0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.686+0000 7f619803f640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61901579a0 0x7f61900c73d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.686+0000 7f619803f640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6190070140 0x7f61900c3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.686+0000 7f619803f640 1 --2- 172.21.15.83:0/999846160 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f619006cef0 0x7f6190148af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.686+0000 7f619803f640 1 -- 172.21.15.83:0/999846160 >> 172.21.15.83:0/999846160 conn(0x7f61900c5090 msgr2=0x7f6190152270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:49.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.686+0000 7f619803f640 1 -- 172.21.15.83:0/999846160 shutdown_connections 2024-09-19T07:22:49.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.686+0000 7f619803f640 1 -- 172.21.15.83:0/999846160 wait complete. 2024-09-19T07:22:49.709 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-19T07:22:49.710 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:22:49.710 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:22:49.721 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:22:49.721 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:49.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.886+0000 7fb30693e640 1 Processor -- start 2024-09-19T07:22:49.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.886+0000 7fb30693e640 1 -- start start 2024-09-19T07:22:49.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.886+0000 7fb30693e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb3000c7d70 0x7fb300157c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:49.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb30693e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3000c74f0 con 0x7fb300072d00 2024-09-19T07:22:49.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb30693e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb300073690 con 0x7fb3001547b0 2024-09-19T07:22:49.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb30693e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb300157160 con 0x7fb3000c7d70 2024-09-19T07:22:49.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb2fffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb3000c7d70 0x7fb300157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:49.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb2fffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb3000c7d70 0x7fb300157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45664/0 (socket says 172.21.15.83:45664) 2024-09-19T07:22:49.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb2fffff640 1 -- 172.21.15.83:0/1190659367 learned_addr learned my addr 172.21.15.83:0/1190659367 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:49.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb2fffff640 1 -- 172.21.15.83:0/1190659367 >> v1:172.21.15.184:6789/0 conn(0x7fb3001547b0 legacy=0x7fb3000c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:49.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb2fffff640 1 -- 172.21.15.83:0/1190659367 >> v1:172.21.15.83:6789/0 conn(0x7fb300072d00 legacy=0x7fb30006cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:49.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb2fffff640 1 -- 172.21.15.83:0/1190659367 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb300157440 con 0x7fb3000c7d70 2024-09-19T07:22:49.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb2fffff640 1 --2- 172.21.15.83:0/1190659367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3000c7d70 0x7fb300157c40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009870 tx=0x7fb2f001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9273f210aa64cc09 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:49.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb2ff7fe640 1 -- 172.21.15.83:0/1190659367 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f00ab600 con 0x7fb3000c7d70 2024-09-19T07:22:49.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb2ff7fe640 1 -- 172.21.15.83:0/1190659367 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2f00b0070 con 0x7fb3000c7d70 2024-09-19T07:22:49.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb2ff7fe640 1 -- 172.21.15.83:0/1190659367 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f00ab9c0 con 0x7fb3000c7d70 2024-09-19T07:22:49.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb30693e640 1 -- 172.21.15.83:0/1190659367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3000c7d70 msgr2=0x7fb300157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:49.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb30693e640 1 --2- 172.21.15.83:0/1190659367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3000c7d70 0x7fb300157c40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009870 tx=0x7fb2f001c4a0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb30693e640 1 -- 172.21.15.83:0/1190659367 shutdown_connections 2024-09-19T07:22:49.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb30693e640 1 --2- 172.21.15.83:0/1190659367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3000c7d70 0x7fb300157c40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb30693e640 1 -- 172.21.15.83:0/1190659367 >> 172.21.15.83:0/1190659367 conn(0x7fb3000c5060 msgr2=0x7fb3000c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:49.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.890+0000 7fb30693e640 1 -- 172.21.15.83:0/1190659367 shutdown_connections 2024-09-19T07:22:49.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb30693e640 1 -- 172.21.15.83:0/1190659367 wait complete. 2024-09-19T07:22:49.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb30693e640 1 Processor -- start 2024-09-19T07:22:49.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb30693e640 1 -- start start 2024-09-19T07:22:49.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb30693e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb300072d00 0x7fb3000c1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:49.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb30693e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3000c7d70 0x7fb3000a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:49.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb30693e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3001547b0 0x7fb3000a9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:49.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb30693e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb3000c82d0 con 0x7fb300072d00 2024-09-19T07:22:49.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb30693e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb3000c8150 con 0x7fb3000c7d70 2024-09-19T07:22:49.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb30693e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb300070230 con 0x7fb3001547b0 2024-09-19T07:22:49.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb300072d00 0x7fb3000c1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:49.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb300072d00 0x7fb3000c1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45678/0 (socket says 172.21.15.83:45678) 2024-09-19T07:22:49.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2fffff640 1 -- 172.21.15.83:0/247730856 learned_addr learned my addr 172.21.15.83:0/247730856 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:49.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2f7fff640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3000c7d70 0x7fb3000a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:49.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb304edb640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3001547b0 0x7fb3000a9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:49.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2fffff640 1 -- 172.21.15.83:0/247730856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3001547b0 msgr2=0x7fb3000a9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:49.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2fffff640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3001547b0 0x7fb3000a9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2fffff640 1 -- 172.21.15.83:0/247730856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3000c7d70 msgr2=0x7fb3000a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:49.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2fffff640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3000c7d70 0x7fb3000a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:49.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2fffff640 1 -- 172.21.15.83:0/247730856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3000c2000 con 0x7fb300072d00 2024-09-19T07:22:49.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2fffff640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb300072d00 0x7fb3000c1990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb2f00190b0 tx=0x7fb2f001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:49.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2fd7fa640 1 -- 172.21.15.83:0/247730856 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f00be5f0 con 0x7fb300072d00 2024-09-19T07:22:49.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2fd7fa640 1 -- 172.21.15.83:0/247730856 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2f00be7b0 con 0x7fb300072d00 2024-09-19T07:22:49.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb30693e640 1 -- 172.21.15.83:0/247730856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3000c22e0 con 0x7fb300072d00 2024-09-19T07:22:49.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb2fd7fa640 1 -- 172.21.15.83:0/247730856 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f00beba0 con 0x7fb300072d00 2024-09-19T07:22:49.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb30693e640 1 -- 172.21.15.83:0/247730856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3000a8460 con 0x7fb300072d00 2024-09-19T07:22:49.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.894+0000 7fb30693e640 1 -- 172.21.15.83:0/247730856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3000705f0 con 0x7fb300072d00 2024-09-19T07:22:49.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.898+0000 7fb2fd7fa640 1 -- 172.21.15.83:0/247730856 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb2f00aa070 con 0x7fb300072d00 2024-09-19T07:22:49.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.898+0000 7fb2fd7fa640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2e00678c0 0x7fb2e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:49.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.898+0000 7fb2f7fff640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2e00678c0 0x7fb2e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:49.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.898+0000 7fb2fd7fa640 1 -- 172.21.15.83:0/247730856 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7fb2f0120900 con 0x7fb300072d00 2024-09-19T07:22:49.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.898+0000 7fb2f7fff640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2e00678c0 0x7fb2e0069d80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb300154070 tx=0x7fb2e8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:49.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:49.902+0000 7fb2fd7fa640 1 -- 172.21.15.83:0/247730856 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb2f0016150 con 0x7fb300072d00 2024-09-19T07:22:50.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.186+0000 7fb30693e640 1 -- 172.21.15.83:0/247730856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3000705f0 con 0x7fb300072d00 2024-09-19T07:22:50.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.186+0000 7fb2fd7fa640 1 -- 172.21.15.83:0/247730856 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6539 (secure 0 0 0) 0x7fb2f00f0fd0 con 0x7fb300072d00 2024-09-19T07:22:50.188 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:50.189 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"btime":"2024-09-19T07:22:38:226763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:50.189 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 33 2024-09-19T07:22:50.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.190+0000 7fb30693e640 1 -- 172.21.15.83:0/247730856 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2e00678c0 msgr2=0x7fb2e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:50.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.190+0000 7fb30693e640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2e00678c0 0x7fb2e0069d80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb300154070 tx=0x7fb2e8009040 comp rx=0 tx=0).stop 2024-09-19T07:22:50.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.190+0000 7fb30693e640 1 -- 172.21.15.83:0/247730856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb300072d00 msgr2=0x7fb3000c1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.190+0000 7fb30693e640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb300072d00 0x7fb3000c1990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb2f00190b0 tx=0x7fb2f001ace0 comp rx=0 tx=0).stop 2024-09-19T07:22:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.190+0000 7fb30693e640 1 -- 172.21.15.83:0/247730856 shutdown_connections 2024-09-19T07:22:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.190+0000 7fb30693e640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb2e00678c0 0x7fb2e0069d80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.190+0000 7fb30693e640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3001547b0 0x7fb3000a9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.190+0000 7fb30693e640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb3000c7d70 0x7fb3000a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.190+0000 7fb30693e640 1 --2- 172.21.15.83:0/247730856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb300072d00 0x7fb3000c1990 secure :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb2f00190b0 tx=0x7fb2f001ace0 comp rx=0 tx=0).stop 2024-09-19T07:22:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.190+0000 7fb30693e640 1 -- 172.21.15.83:0/247730856 >> 172.21.15.83:0/247730856 conn(0x7fb3000c5060 msgr2=0x7fb3000c5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.190+0000 7fb30693e640 1 -- 172.21.15.83:0/247730856 shutdown_connections 2024-09-19T07:22:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:50.190+0000 7fb30693e640 1 -- 172.21.15.83:0/247730856 wait complete. 2024-09-19T07:22:50.211 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:22:52.213 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:52.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf57b21640 1 Processor -- start 2024-09-19T07:22:52.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf57b21640 1 -- start start 2024-09-19T07:22:52.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf57b21640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf500c7510 0x7fcf5006de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf57b21640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf50154360 0x7fcf500c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf57b21640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5014ddd0 con 0x7fcf5014d9f0 2024-09-19T07:22:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf57b21640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fcf500c8830 con 0x7fcf500c7510 2024-09-19T07:22:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf57b21640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcf5006fdf0 con 0x7fcf50154360 2024-09-19T07:22:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf550bc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf500c7510 0x7fcf5006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf558bd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcf5014d9f0 0x7fcf500c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45852/0 (socket says 172.21.15.83:45852) 2024-09-19T07:22:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf558bd640 1 -- 172.21.15.83:0/2238685193 learned_addr learned my addr 172.21.15.83:0/2238685193 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf550bc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf500c7510 0x7fcf5006de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39098/0 (socket says 172.21.15.83:39098) 2024-09-19T07:22:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf560be640 1 --2- 172.21.15.83:0/2238685193 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf50154360 0x7fcf500c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf550bc640 1 -- 172.21.15.83:0/2238685193 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf50154360 msgr2=0x7fcf500c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf550bc640 1 --2- 172.21.15.83:0/2238685193 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf50154360 0x7fcf500c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf550bc640 1 -- 172.21.15.83:0/2238685193 >> v1:172.21.15.83:6789/0 conn(0x7fcf5014d9f0 legacy=0x7fcf500c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:52.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf548bb640 1 -- 172.21.15.83:0/2238685193 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4169314617 0 0) 0x7fcf5014ddd0 con 0x7fcf5014d9f0 2024-09-19T07:22:52.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.354+0000 7fcf550bc640 1 -- 172.21.15.83:0/2238685193 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf500c7f00 con 0x7fcf500c7510 2024-09-19T07:22:52.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf550bc640 1 --2- 172.21.15.83:0/2238685193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf500c7510 0x7fcf5006de90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcf440097c0 tx=0x7fcf4401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e191c5aecc61a03 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:52.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf548bb640 1 -- 172.21.15.83:0/2238685193 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf440b0070 con 0x7fcf500c7510 2024-09-19T07:22:52.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf548bb640 1 -- 172.21.15.83:0/2238685193 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf440ab600 con 0x7fcf500c7510 2024-09-19T07:22:52.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf548bb640 1 -- 172.21.15.83:0/2238685193 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf440ab9d0 con 0x7fcf500c7510 2024-09-19T07:22:52.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 -- 172.21.15.83:0/2238685193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf500c7510 msgr2=0x7fcf5006de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:52.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 --2- 172.21.15.83:0/2238685193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf500c7510 0x7fcf5006de90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcf440097c0 tx=0x7fcf4401c410 comp rx=0 tx=0).stop 2024-09-19T07:22:52.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 -- 172.21.15.83:0/2238685193 shutdown_connections 2024-09-19T07:22:52.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 --2- 172.21.15.83:0/2238685193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf500c7510 0x7fcf5006de90 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:52.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 --2- 172.21.15.83:0/2238685193 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf50154360 0x7fcf500c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:52.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 -- 172.21.15.83:0/2238685193 >> 172.21.15.83:0/2238685193 conn(0x7fcf500c6110 msgr2=0x7fcf500c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 -- 172.21.15.83:0/2238685193 shutdown_connections 2024-09-19T07:22:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 -- 172.21.15.83:0/2238685193 wait complete. 2024-09-19T07:22:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 Processor -- start 2024-09-19T07:22:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 -- start start 2024-09-19T07:22:52.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf500c7510 0x7fcf50150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf5014d9f0 0x7fcf500c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf50154360 0x7fcf501f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcf50160c00 con 0x7fcf500c7510 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcf50160a80 con 0x7fcf50154360 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.358+0000 7fcf57b21640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcf50160d80 con 0x7fcf5014d9f0 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf558bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf500c7510 0x7fcf50150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf558bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf500c7510 0x7fcf50150550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39108/0 (socket says 172.21.15.83:39108) 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf558bd640 1 -- 172.21.15.83:0/3972198926 learned_addr learned my addr 172.21.15.83:0/3972198926 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf550bc640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf5014d9f0 0x7fcf500c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf560be640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf50154360 0x7fcf501f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf558bd640 1 -- 172.21.15.83:0/3972198926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf5014d9f0 msgr2=0x7fcf500c2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf558bd640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf5014d9f0 0x7fcf500c2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:52.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf558bd640 1 -- 172.21.15.83:0/3972198926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf50154360 msgr2=0x7fcf501f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:52.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf558bd640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf50154360 0x7fcf501f1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:52.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf558bd640 1 -- 172.21.15.83:0/3972198926 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5014b6c0 con 0x7fcf500c7510 2024-09-19T07:22:52.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf558bd640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf500c7510 0x7fcf50150550 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000c7f0 tx=0x7fcf4000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:52.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf4e7fc640 1 -- 172.21.15.83:0/3972198926 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf40018070 con 0x7fcf500c7510 2024-09-19T07:22:52.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf4e7fc640 1 -- 172.21.15.83:0/3972198926 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf4000a640 con 0x7fcf500c7510 2024-09-19T07:22:52.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf57b21640 1 -- 172.21.15.83:0/3972198926 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5014b9a0 con 0x7fcf500c7510 2024-09-19T07:22:52.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf4e7fc640 1 -- 172.21.15.83:0/3972198926 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4000aa30 con 0x7fcf500c7510 2024-09-19T07:22:52.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf57b21640 1 -- 172.21.15.83:0/3972198926 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf5014e150 con 0x7fcf500c7510 2024-09-19T07:22:52.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf57b21640 1 -- 172.21.15.83:0/3972198926 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf5006de90 con 0x7fcf500c7510 2024-09-19T07:22:52.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf4e7fc640 1 -- 172.21.15.83:0/3972198926 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcf4001b070 con 0x7fcf500c7510 2024-09-19T07:22:52.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf4e7fc640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf3c0678c0 0x7fcf3c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:52.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf4e7fc640 1 -- 172.21.15.83:0/3972198926 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7fcf40085780 con 0x7fcf500c7510 2024-09-19T07:22:52.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf550bc640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf3c0678c0 0x7fcf3c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:52.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.362+0000 7fcf550bc640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf3c0678c0 0x7fcf3c069d80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcf440ad000 tx=0x7fcf440a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:52.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.370+0000 7fcf4e7fc640 1 -- 172.21.15.83:0/3972198926 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcf40055e50 con 0x7fcf500c7510 2024-09-19T07:22:52.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.610+0000 7fcf57b21640 1 -- 172.21.15.83:0/3972198926 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf500c4010 con 0x7fcf500c7510 2024-09-19T07:22:52.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.610+0000 7fcf4e7fc640 1 -- 172.21.15.83:0/3972198926 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6539 (secure 0 0 0) 0x7fcf40055750 con 0x7fcf500c7510 2024-09-19T07:22:52.613 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:52.613 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"btime":"2024-09-19T07:22:38:226763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:52.613 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 33 2024-09-19T07:22:52.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.614+0000 7fcf57b21640 1 -- 172.21.15.83:0/3972198926 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf3c0678c0 msgr2=0x7fcf3c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:52.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.614+0000 7fcf57b21640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf3c0678c0 0x7fcf3c069d80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcf440ad000 tx=0x7fcf440a6000 comp rx=0 tx=0).stop 2024-09-19T07:22:52.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.614+0000 7fcf57b21640 1 -- 172.21.15.83:0/3972198926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf500c7510 msgr2=0x7fcf50150550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:52.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.614+0000 7fcf57b21640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf500c7510 0x7fcf50150550 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000c7f0 tx=0x7fcf4000ccb0 comp rx=0 tx=0).stop 2024-09-19T07:22:52.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.614+0000 7fcf57b21640 1 -- 172.21.15.83:0/3972198926 shutdown_connections 2024-09-19T07:22:52.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.614+0000 7fcf57b21640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf3c0678c0 0x7fcf3c069d80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:52.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.614+0000 7fcf57b21640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf50154360 0x7fcf501f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:52.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.614+0000 7fcf57b21640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf5014d9f0 0x7fcf500c2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:52.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.614+0000 7fcf57b21640 1 --2- 172.21.15.83:0/3972198926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf500c7510 0x7fcf50150550 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:52.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.614+0000 7fcf57b21640 1 -- 172.21.15.83:0/3972198926 >> 172.21.15.83:0/3972198926 conn(0x7fcf500c6110 msgr2=0x7fcf500c64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:52.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.614+0000 7fcf57b21640 1 -- 172.21.15.83:0/3972198926 shutdown_connections 2024-09-19T07:22:52.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:52.614+0000 7fcf57b21640 1 -- 172.21.15.83:0/3972198926 wait complete. 2024-09-19T07:22:52.635 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:22:54.636 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:54.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.782+0000 7fd1ece13640 1 Processor -- start 2024-09-19T07:22:54.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.782+0000 7fd1ece13640 1 -- start start 2024-09-19T07:22:54.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.782+0000 7fd1ece13640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1e80c3ab0 0x7fd1e8157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:54.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.782+0000 7fd1ece13640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e8073d90 con 0x7fd1e814dbe0 2024-09-19T07:22:54.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.782+0000 7fd1ece13640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e8153b30 con 0x7fd1e80aa4a0 2024-09-19T07:22:54.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.782+0000 7fd1ece13640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e8154700 con 0x7fd1e80c3ab0 2024-09-19T07:22:54.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.782+0000 7fd1e659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd1e814dbe0 0x7fd1e80c3fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45866/0 (socket says 172.21.15.83:45866) 2024-09-19T07:22:54.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.782+0000 7fd1e659c640 1 -- 172.21.15.83:0/2339761533 learned_addr learned my addr 172.21.15.83:0/2339761533 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:22:54.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1dffff640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1e80c3ab0 0x7fd1e8157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:54.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 388881489 0 0) 0x7fd1e8073d90 con 0x7fd1e814dbe0 2024-09-19T07:22:54.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1d4003530 con 0x7fd1e814dbe0 2024-09-19T07:22:54.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1358959890 0 0) 0x7fd1d4003530 con 0x7fd1e814dbe0 2024-09-19T07:22:54.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1e80c3ab0 msgr2=0x7fd1e8157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:54.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1e80c3ab0 0x7fd1e8157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:54.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 >> v1:172.21.15.184:6789/0 conn(0x7fd1e80aa4a0 legacy=0x7fd1e8157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:54.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e8153dd0 con 0x7fd1e814dbe0 2024-09-19T07:22:54.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fd1d8001680 con 0x7fd1e814dbe0 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 >> v1:172.21.15.83:6789/0 conn(0x7fd1e814dbe0 legacy=0x7fd1e80c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1d40039f0 0x7fd1d4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d4004810 0x7fd1d4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d4005220 0x7fd1d400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e8154700 con 0x7fd1d40039f0 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d400bf90 con 0x7fd1d4004810 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d400c190 con 0x7fd1d4005220 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e659c640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1d40039f0 0x7fd1d4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1dffff640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d4004810 0x7fd1d4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e6d9d640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d4005220 0x7fd1d400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e659c640 1 -- 172.21.15.83:0/2339761533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d4005220 msgr2=0x7fd1d400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e659c640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d4005220 0x7fd1d400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:54.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e659c640 1 -- 172.21.15.83:0/2339761533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d4004810 msgr2=0x7fd1d4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:54.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e659c640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d4004810 0x7fd1d4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:54.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e659c640 1 -- 172.21.15.83:0/2339761533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d4003530 con 0x7fd1d40039f0 2024-09-19T07:22:54.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1dffff640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d4004810 0x7fd1d4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:22:54.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e659c640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1d40039f0 0x7fd1d4003e30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800c0e0 tx=0x7fd1d801cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feebc74e57289fd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:54.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8007c50 con 0x7fd1d40039f0 2024-09-19T07:22:54.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d80a9070 con 0x7fd1d40039f0 2024-09-19T07:22:54.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.786+0000 7fd1e5d9b640 1 -- 172.21.15.83:0/2339761533 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8005d40 con 0x7fd1d40039f0 2024-09-19T07:22:54.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 -- 172.21.15.83:0/2339761533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1d40039f0 msgr2=0x7fd1d4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:54.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 --2- 172.21.15.83:0/2339761533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1d40039f0 0x7fd1d4003e30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800c0e0 tx=0x7fd1d801cac0 comp rx=0 tx=0).stop 2024-09-19T07:22:54.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1e6d9d640 1 -- 172.21.15.83:0/2339761533 reap_dead start 2024-09-19T07:22:54.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 -- 172.21.15.83:0/2339761533 shutdown_connections 2024-09-19T07:22:54.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 -- 172.21.15.83:0/2339761533 >> 172.21.15.83:0/2339761533 conn(0x7fd1e80c95a0 msgr2=0x7fd1e80c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:54.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 -- 172.21.15.83:0/2339761533 shutdown_connections 2024-09-19T07:22:54.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 -- 172.21.15.83:0/2339761533 wait complete. 2024-09-19T07:22:54.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 Processor -- start 2024-09-19T07:22:54.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 -- start start 2024-09-19T07:22:54.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1e814f5f0 0x7fd1e814e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:54.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1e814e620 0x7fd1e8146780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:54.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1e814eaa0 0x7fd1e8146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:54.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e80c46b0 con 0x7fd1e814e620 2024-09-19T07:22:54.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e80741a0 con 0x7fd1e814eaa0 2024-09-19T07:22:54.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e80c4830 con 0x7fd1e814f5f0 2024-09-19T07:22:54.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1dffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1e814e620 0x7fd1e8146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:54.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1dffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1e814e620 0x7fd1e8146780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39140/0 (socket says 172.21.15.83:39140) 2024-09-19T07:22:54.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1dffff640 1 -- 172.21.15.83:0/1482518190 learned_addr learned my addr 172.21.15.83:0/1482518190 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:54.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1e6d9d640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1e814eaa0 0x7fd1e8146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1e659c640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1e814f5f0 0x7fd1e814e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1dffff640 1 -- 172.21.15.83:0/1482518190 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1e814f5f0 msgr2=0x7fd1e814e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1dffff640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1e814f5f0 0x7fd1e814e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1dffff640 1 -- 172.21.15.83:0/1482518190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1e814eaa0 msgr2=0x7fd1e8146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1dffff640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1e814eaa0 0x7fd1e8146cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1dffff640 1 -- 172.21.15.83:0/1482518190 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e81595f0 con 0x7fd1e814e620 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1dffff640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1e814e620 0x7fd1e8146780 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00116a0 tx=0x7fd1d0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1deffd640 1 -- 172.21.15.83:0/1482518190 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0004220 con 0x7fd1e814e620 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1deffd640 1 -- 172.21.15.83:0/1482518190 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d00043c0 con 0x7fd1e814e620 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 -- 172.21.15.83:0/1482518190 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e81ea2a0 con 0x7fd1e814e620 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1deffd640 1 -- 172.21.15.83:0/1482518190 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d000f960 con 0x7fd1e814e620 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.790+0000 7fd1ece13640 1 -- 172.21.15.83:0/1482518190 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e81ea840 con 0x7fd1e814e620 2024-09-19T07:22:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.794+0000 7fd1ece13640 1 -- 172.21.15.83:0/1482518190 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e80c3ab0 con 0x7fd1e814e620 2024-09-19T07:22:54.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.794+0000 7fd1deffd640 1 -- 172.21.15.83:0/1482518190 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd1d000fb00 con 0x7fd1e814e620 2024-09-19T07:22:54.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.794+0000 7fd1deffd640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd1c80678f0 0x7fd1c8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:54.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.794+0000 7fd1deffd640 1 -- 172.21.15.83:0/1482518190 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7fd1d0086630 con 0x7fd1e814e620 2024-09-19T07:22:54.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.794+0000 7fd1e659c640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd1c80678f0 0x7fd1c8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:54.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.794+0000 7fd1e659c640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd1c80678f0 0x7fd1c8069db0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80ab000 tx=0x7fd1d80b0040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:54.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:54.798+0000 7fd1deffd640 1 -- 172.21.15.83:0/1482518190 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd1d0019030 con 0x7fd1e814e620 2024-09-19T07:22:55.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.010+0000 7fd1ece13640 1 -- 172.21.15.83:0/1482518190 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1e8073f80 con 0x7fd1e814e620 2024-09-19T07:22:55.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1deffd640 1 -- 172.21.15.83:0/1482518190 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6539 (secure 0 0 0) 0x7fd1d0056d00 con 0x7fd1e814e620 2024-09-19T07:22:55.014 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:55.014 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"btime":"2024-09-19T07:22:38:226763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:55.015 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 33 2024-09-19T07:22:55.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1ece13640 1 -- 172.21.15.83:0/1482518190 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd1c80678f0 msgr2=0x7fd1c8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:55.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1ece13640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd1c80678f0 0x7fd1c8069db0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80ab000 tx=0x7fd1d80b0040 comp rx=0 tx=0).stop 2024-09-19T07:22:55.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1ece13640 1 -- 172.21.15.83:0/1482518190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1e814e620 msgr2=0x7fd1e8146780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:55.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1ece13640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1e814e620 0x7fd1e8146780 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00116a0 tx=0x7fd1d0011b60 comp rx=0 tx=0).stop 2024-09-19T07:22:55.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1ece13640 1 -- 172.21.15.83:0/1482518190 shutdown_connections 2024-09-19T07:22:55.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1ece13640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd1c80678f0 0x7fd1c8069db0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:55.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1ece13640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1e814eaa0 0x7fd1e8146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:55.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1ece13640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1e814e620 0x7fd1e8146780 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:55.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1ece13640 1 --2- 172.21.15.83:0/1482518190 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1e814f5f0 0x7fd1e814e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:55.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1ece13640 1 -- 172.21.15.83:0/1482518190 >> 172.21.15.83:0/1482518190 conn(0x7fd1e80c95a0 msgr2=0x7fd1e8072eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:55.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1ece13640 1 -- 172.21.15.83:0/1482518190 shutdown_connections 2024-09-19T07:22:55.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:55.014+0000 7fd1ece13640 1 -- 172.21.15.83:0/1482518190 wait complete. 2024-09-19T07:22:55.033 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:22:57.034 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:57.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.174+0000 7fc795973640 1 Processor -- start 2024-09-19T07:22:57.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.174+0000 7fc795973640 1 -- start start 2024-09-19T07:22:57.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc795973640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc790153450 0x7fc7900ae840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:57.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc795973640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc7901594f0 0x7fc790159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:57.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc795973640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7900ad380 con 0x7fc7900c40a0 2024-09-19T07:22:57.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc795973640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc790155d50 con 0x7fc7901594f0 2024-09-19T07:22:57.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc795973640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc790156010 con 0x7fc790153450 2024-09-19T07:22:57.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc78f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc7901594f0 0x7fc790159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:57.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc78f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc7901594f0 0x7fc790159980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39154/0 (socket says 172.21.15.83:39154) 2024-09-19T07:22:57.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc78f7fe640 1 -- 172.21.15.83:0/3967315360 learned_addr learned my addr 172.21.15.83:0/3967315360 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:57.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc78effd640 1 --2- 172.21.15.83:0/3967315360 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc790153450 0x7fc7900ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:57.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc78f7fe640 1 -- 172.21.15.83:0/3967315360 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc790153450 msgr2=0x7fc7900ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:57.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc78f7fe640 1 --2- 172.21.15.83:0/3967315360 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc790153450 0x7fc7900ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:57.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc78f7fe640 1 -- 172.21.15.83:0/3967315360 >> v1:172.21.15.184:6789/0 conn(0x7fc7900c40a0 legacy=0x7fc790073020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:57.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc78f7fe640 1 -- 172.21.15.83:0/3967315360 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7901562f0 con 0x7fc7901594f0 2024-09-19T07:22:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc78f7fe640 1 --2- 172.21.15.83:0/3967315360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7901594f0 0x7fc790159980 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc788009e50 tx=0x7fc78801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8624534f5d977f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc78dffb640 1 -- 172.21.15.83:0/3967315360 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7880a5810 con 0x7fc7901594f0 2024-09-19T07:22:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc78dffb640 1 -- 172.21.15.83:0/3967315360 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7880a59b0 con 0x7fc7901594f0 2024-09-19T07:22:57.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.178+0000 7fc78dffb640 1 -- 172.21.15.83:0/3967315360 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7880a5d80 con 0x7fc7901594f0 2024-09-19T07:22:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 -- 172.21.15.83:0/3967315360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7901594f0 msgr2=0x7fc790159980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 --2- 172.21.15.83:0/3967315360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7901594f0 0x7fc790159980 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc788009e50 tx=0x7fc78801ca30 comp rx=0 tx=0).stop 2024-09-19T07:22:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 -- 172.21.15.83:0/3967315360 shutdown_connections 2024-09-19T07:22:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 --2- 172.21.15.83:0/3967315360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7901594f0 0x7fc790159980 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 --2- 172.21.15.83:0/3967315360 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc790153450 0x7fc7900ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 -- 172.21.15.83:0/3967315360 >> 172.21.15.83:0/3967315360 conn(0x7fc79006fff0 msgr2=0x7fc790070400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 -- 172.21.15.83:0/3967315360 shutdown_connections 2024-09-19T07:22:57.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 -- 172.21.15.83:0/3967315360 wait complete. 2024-09-19T07:22:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 Processor -- start 2024-09-19T07:22:57.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 -- start start 2024-09-19T07:22:57.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7900c40a0 0x7fc79014e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:57.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc790153450 0x7fc79015f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:57.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7901594f0 0x7fc79015fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:57.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc7901539b0 con 0x7fc7900c40a0 2024-09-19T07:22:57.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc790153830 con 0x7fc7901594f0 2024-09-19T07:22:57.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc790162ef0 con 0x7fc790153450 2024-09-19T07:22:57.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc78effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7900c40a0 0x7fc79014e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:57.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc78effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7900c40a0 0x7fc79014e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39166/0 (socket says 172.21.15.83:39166) 2024-09-19T07:22:57.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc78effd640 1 -- 172.21.15.83:0/2488926356 learned_addr learned my addr 172.21.15.83:0/2488926356 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:57.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc78e7fc640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc790153450 0x7fc79015f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:57.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc78f7fe640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7901594f0 0x7fc79015fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:57.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc78effd640 1 -- 172.21.15.83:0/2488926356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc790153450 msgr2=0x7fc79015f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:57.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc78effd640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc790153450 0x7fc79015f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:57.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc78effd640 1 -- 172.21.15.83:0/2488926356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7901594f0 msgr2=0x7fc79015fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:57.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc78effd640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7901594f0 0x7fc79015fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:57.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc78effd640 1 -- 172.21.15.83:0/2488926356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7901602b0 con 0x7fc7900c40a0 2024-09-19T07:22:57.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc78effd640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7900c40a0 0x7fc79014e470 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc78000d830 tx=0x7fc78000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:57.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc777fff640 1 -- 172.21.15.83:0/2488926356 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78000b6c0 con 0x7fc7900c40a0 2024-09-19T07:22:57.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc777fff640 1 -- 172.21.15.83:0/2488926356 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc780019070 con 0x7fc7900c40a0 2024-09-19T07:22:57.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 -- 172.21.15.83:0/2488926356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7900c7ce0 con 0x7fc7900c40a0 2024-09-19T07:22:57.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc777fff640 1 -- 172.21.15.83:0/2488926356 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78000ba60 con 0x7fc7900c40a0 2024-09-19T07:22:57.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.182+0000 7fc795973640 1 -- 172.21.15.83:0/2488926356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79006bc80 con 0x7fc7900c40a0 2024-09-19T07:22:57.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.186+0000 7fc795973640 1 -- 172.21.15.83:0/2488926356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7900ae840 con 0x7fc7900c40a0 2024-09-19T07:22:57.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.186+0000 7fc777fff640 1 -- 172.21.15.83:0/2488926356 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc78001e460 con 0x7fc7900c40a0 2024-09-19T07:22:57.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.186+0000 7fc777fff640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc77c0678c0 0x7fc77c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:57.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.186+0000 7fc777fff640 1 -- 172.21.15.83:0/2488926356 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7fc780086840 con 0x7fc7900c40a0 2024-09-19T07:22:57.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.186+0000 7fc78e7fc640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc77c0678c0 0x7fc77c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:57.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.186+0000 7fc78e7fc640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc77c0678c0 0x7fc77c069d80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc79014ed30 tx=0x7fc784009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:57.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.190+0000 7fc777fff640 1 -- 172.21.15.83:0/2488926356 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc7901602b0 con 0x7fc7900c40a0 2024-09-19T07:22:57.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.394+0000 7fc795973640 1 -- 172.21.15.83:0/2488926356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc790073ff0 con 0x7fc7900c40a0 2024-09-19T07:22:57.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.394+0000 7fc777fff640 1 -- 172.21.15.83:0/2488926356 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6539 (secure 0 0 0) 0x7fc780056e90 con 0x7fc7900c40a0 2024-09-19T07:22:57.397 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:57.397 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"btime":"2024-09-19T07:22:38:226763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:57.397 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 33 2024-09-19T07:22:57.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.398+0000 7fc795973640 1 -- 172.21.15.83:0/2488926356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc77c0678c0 msgr2=0x7fc77c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:57.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.398+0000 7fc795973640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc77c0678c0 0x7fc77c069d80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc79014ed30 tx=0x7fc784009040 comp rx=0 tx=0).stop 2024-09-19T07:22:57.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.398+0000 7fc795973640 1 -- 172.21.15.83:0/2488926356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7900c40a0 msgr2=0x7fc79014e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:57.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.398+0000 7fc795973640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7900c40a0 0x7fc79014e470 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc78000d830 tx=0x7fc78000dcf0 comp rx=0 tx=0).stop 2024-09-19T07:22:57.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.398+0000 7fc795973640 1 -- 172.21.15.83:0/2488926356 shutdown_connections 2024-09-19T07:22:57.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.398+0000 7fc795973640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc77c0678c0 0x7fc77c069d80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:57.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.398+0000 7fc795973640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7901594f0 0x7fc79015fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:57.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.398+0000 7fc795973640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc790153450 0x7fc79015f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:57.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.398+0000 7fc795973640 1 --2- 172.21.15.83:0/2488926356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7900c40a0 0x7fc79014e470 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:57.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.398+0000 7fc795973640 1 -- 172.21.15.83:0/2488926356 >> 172.21.15.83:0/2488926356 conn(0x7fc79006fff0 msgr2=0x7fc7900acc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:57.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.398+0000 7fc795973640 1 -- 172.21.15.83:0/2488926356 shutdown_connections 2024-09-19T07:22:57.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:57.398+0000 7fc795973640 1 -- 172.21.15.83:0/2488926356 wait complete. 2024-09-19T07:22:57.417 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:22:59.419 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:22:59.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.570+0000 7f06b9004640 1 Processor -- start 2024-09-19T07:22:59.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.570+0000 7f06b9004640 1 -- start start 2024-09-19T07:22:59.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.570+0000 7f06b9004640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06b406cef0 0x7f06b40abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:59.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.570+0000 7f06b9004640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f06b40ab610 0x7f06b40c3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:59.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.570+0000 7f06b9004640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b40c7de0 con 0x7f06b40c6fc0 2024-09-19T07:22:59.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.570+0000 7f06b9004640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f06b40c6a90 con 0x7f06b40ab610 2024-09-19T07:22:59.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.570+0000 7f06b9004640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f06b40c8360 con 0x7f06b406cef0 2024-09-19T07:22:59.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.570+0000 7f06b259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f06b40ab610 0x7f06b40c3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:59.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.570+0000 7f06b259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f06b40ab610 0x7f06b40c3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39174/0 (socket says 172.21.15.83:39174) 2024-09-19T07:22:59.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.570+0000 7f06b259c640 1 -- 172.21.15.83:0/1762323325 learned_addr learned my addr 172.21.15.83:0/1762323325 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:59.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.570+0000 7f06b359e640 1 --1- 172.21.15.83:0/1762323325 >> v1:172.21.15.83:6789/0 conn(0x7f06b40c6fc0 0x7f06b4157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45878/0 (socket says 172.21.15.83:45878) 2024-09-19T07:22:59.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b2d9d640 1 --2- 172.21.15.83:0/1762323325 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06b406cef0 0x7f06b40abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:59.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b259c640 1 -- 172.21.15.83:0/1762323325 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06b406cef0 msgr2=0x7f06b40abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:22:59.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b259c640 1 --2- 172.21.15.83:0/1762323325 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06b406cef0 0x7f06b40abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:59.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b259c640 1 -- 172.21.15.83:0/1762323325 >> v1:172.21.15.83:6789/0 conn(0x7f06b40c6fc0 legacy=0x7f06b4157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:59.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b1d9b640 1 -- 172.21.15.83:0/1762323325 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 695950715 0 0) 0x7f06b40c7de0 con 0x7f06b40c6fc0 2024-09-19T07:22:59.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b259c640 1 -- 172.21.15.83:0/1762323325 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b40c8080 con 0x7f06b40ab610 2024-09-19T07:22:59.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b2d9d640 1 --2- 172.21.15.83:0/1762323325 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06b406cef0 0x7f06b40abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:59.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b259c640 1 --2- 172.21.15.83:0/1762323325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b40ab610 0x7f06b40c3970 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009870 tx=0x7f06a801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10765c5553af2e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:59.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b1d9b640 1 -- 172.21.15.83:0/1762323325 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a80ab430 con 0x7f06b40ab610 2024-09-19T07:22:59.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b1d9b640 1 -- 172.21.15.83:0/1762323325 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a80b0070 con 0x7f06b40ab610 2024-09-19T07:22:59.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b1d9b640 1 -- 172.21.15.83:0/1762323325 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a80ab7f0 con 0x7f06b40ab610 2024-09-19T07:22:59.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b9004640 1 -- 172.21.15.83:0/1762323325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b40ab610 msgr2=0x7f06b40c3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:59.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b9004640 1 --2- 172.21.15.83:0/1762323325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b40ab610 0x7f06b40c3970 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009870 tx=0x7f06a801c410 comp rx=0 tx=0).stop 2024-09-19T07:22:59.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b9004640 1 -- 172.21.15.83:0/1762323325 shutdown_connections 2024-09-19T07:22:59.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b9004640 1 --2- 172.21.15.83:0/1762323325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b40ab610 0x7f06b40c3970 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:59.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b9004640 1 --2- 172.21.15.83:0/1762323325 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06b406cef0 0x7f06b40abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:59.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b9004640 1 -- 172.21.15.83:0/1762323325 >> 172.21.15.83:0/1762323325 conn(0x7f06b40aa910 msgr2=0x7f06b40c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:59.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b9004640 1 -- 172.21.15.83:0/1762323325 shutdown_connections 2024-09-19T07:22:59.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.574+0000 7f06b9004640 1 -- 172.21.15.83:0/1762323325 wait complete. 2024-09-19T07:22:59.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b9004640 1 Processor -- start 2024-09-19T07:22:59.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b9004640 1 -- start start 2024-09-19T07:22:59.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b9004640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06b406cef0 0x7f06b414f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:59.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b9004640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b40ab610 0x7f06b40be9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:59.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b9004640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06b40c6fc0 0x7f06b40bef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b9004640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f06b406d450 con 0x7f06b40ab610 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b9004640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f06b406d2d0 con 0x7f06b406cef0 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b9004640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f06b4157fa0 con 0x7f06b40c6fc0 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b40ab610 0x7f06b40be9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b40ab610 0x7f06b40be9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39180/0 (socket says 172.21.15.83:39180) 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b259c640 1 -- 172.21.15.83:0/616835926 learned_addr learned my addr 172.21.15.83:0/616835926 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b359e640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06b40c6fc0 0x7f06b40bef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b2d9d640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06b406cef0 0x7f06b414f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b259c640 1 -- 172.21.15.83:0/616835926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06b40c6fc0 msgr2=0x7f06b40bef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b259c640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06b40c6fc0 0x7f06b40bef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b259c640 1 -- 172.21.15.83:0/616835926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06b406cef0 msgr2=0x7f06b414f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b259c640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06b406cef0 0x7f06b414f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:59.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b259c640 1 -- 172.21.15.83:0/616835926 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b4151b40 con 0x7f06b40ab610 2024-09-19T07:22:59.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b2d9d640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06b406cef0 0x7f06b414f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:22:59.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b259c640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b40ab610 0x7f06b40be9e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f06a80ad040 tx=0x7f06a80191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:59.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f069b7fe640 1 -- 172.21.15.83:0/616835926 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a80ba070 con 0x7f06b40ab610 2024-09-19T07:22:59.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f069b7fe640 1 -- 172.21.15.83:0/616835926 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a80b4420 con 0x7f06b40ab610 2024-09-19T07:22:59.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f069b7fe640 1 -- 172.21.15.83:0/616835926 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a80b4830 con 0x7f06b40ab610 2024-09-19T07:22:59.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b9004640 1 -- 172.21.15.83:0/616835926 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b4151e20 con 0x7f06b40ab610 2024-09-19T07:22:59.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.578+0000 7f06b9004640 1 -- 172.21.15.83:0/616835926 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b4073c20 con 0x7f06b40ab610 2024-09-19T07:22:59.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.582+0000 7f06b9004640 1 -- 172.21.15.83:0/616835926 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b40c7410 con 0x7f06b40ab610 2024-09-19T07:22:59.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.582+0000 7f069b7fe640 1 -- 172.21.15.83:0/616835926 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f06a80bd070 con 0x7f06b40ab610 2024-09-19T07:22:59.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.582+0000 7f069b7fe640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f06a00678c0 0x7f06a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:22:59.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.582+0000 7f069b7fe640 1 -- 172.21.15.83:0/616835926 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7f06a81203f0 con 0x7f06b40ab610 2024-09-19T07:22:59.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.582+0000 7f06b2d9d640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f06a00678c0 0x7f06a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:22:59.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.582+0000 7f06b2d9d640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f06a00678c0 0x7f06a0069d80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f06b406c350 tx=0x7f06a4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:22:59.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.586+0000 7f069b7fe640 1 -- 172.21.15.83:0/616835926 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f06a8016150 con 0x7f06b40ab610 2024-09-19T07:22:59.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.822+0000 7f06b9004640 1 -- 172.21.15.83:0/616835926 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06b406c730 con 0x7f06b40ab610 2024-09-19T07:22:59.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.822+0000 7f069b7fe640 1 -- 172.21.15.83:0/616835926 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6539 (secure 0 0 0) 0x7f06a80f0ac0 con 0x7f06b40ab610 2024-09-19T07:22:59.825 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:22:59.825 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"btime":"2024-09-19T07:22:38:226763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:22:59.825 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 33 2024-09-19T07:22:59.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.826+0000 7f06b9004640 1 -- 172.21.15.83:0/616835926 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f06a00678c0 msgr2=0x7f06a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:59.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.826+0000 7f06b9004640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f06a00678c0 0x7f06a0069d80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f06b406c350 tx=0x7f06a4008040 comp rx=0 tx=0).stop 2024-09-19T07:22:59.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.826+0000 7f06b9004640 1 -- 172.21.15.83:0/616835926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b40ab610 msgr2=0x7f06b40be9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:22:59.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.826+0000 7f06b9004640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b40ab610 0x7f06b40be9e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f06a80ad040 tx=0x7f06a80191b0 comp rx=0 tx=0).stop 2024-09-19T07:22:59.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.826+0000 7f06b9004640 1 -- 172.21.15.83:0/616835926 shutdown_connections 2024-09-19T07:22:59.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.826+0000 7f06b9004640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f06a00678c0 0x7f06a0069d80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:59.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.826+0000 7f06b9004640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06b40c6fc0 0x7f06b40bef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:59.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.826+0000 7f06b9004640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b40ab610 0x7f06b40be9e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:59.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.826+0000 7f06b9004640 1 --2- 172.21.15.83:0/616835926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06b406cef0 0x7f06b414f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:22:59.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.826+0000 7f06b9004640 1 -- 172.21.15.83:0/616835926 >> 172.21.15.83:0/616835926 conn(0x7f06b40aa910 msgr2=0x7f06b40c41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:22:59.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.826+0000 7f06b9004640 1 -- 172.21.15.83:0/616835926 shutdown_connections 2024-09-19T07:22:59.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:22:59.826+0000 7f06b9004640 1 -- 172.21.15.83:0/616835926 wait complete. 2024-09-19T07:22:59.849 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:23:00.854 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:23:00.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:23:01.849 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:01.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.986+0000 7fc0b14ef640 1 Processor -- start 2024-09-19T07:23:01.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.986+0000 7fc0b14ef640 1 -- start start 2024-09-19T07:23:01.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.986+0000 7fc0b14ef640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc0ac06d6f0 0x7fc0ac152b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.986+0000 7fc0b14ef640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0ac14fcf0 con 0x7fc0ac0c2fb0 2024-09-19T07:23:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.986+0000 7fc0b14ef640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0ac06e240 con 0x7fc0ac14f110 2024-09-19T07:23:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.986+0000 7fc0b14ef640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc0ac06dff0 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.986+0000 7fc0aaffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc0ac06d6f0 0x7fc0ac152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.986+0000 7fc0aaffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc0ac06d6f0 0x7fc0ac152b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56764/0 (socket says 172.21.15.83:56764) 2024-09-19T07:23:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.986+0000 7fc0aaffd640 1 -- 172.21.15.83:0/3205905547 learned_addr learned my addr 172.21.15.83:0/3205905547 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.986+0000 7fc0aaffd640 1 -- 172.21.15.83:0/3205905547 >> v1:172.21.15.184:6789/0 conn(0x7fc0ac14f110 legacy=0x7fc0ac14f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.986+0000 7fc0aaffd640 1 -- 172.21.15.83:0/3205905547 >> v1:172.21.15.83:6789/0 conn(0x7fc0ac0c2fb0 legacy=0x7fc0ac0c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.986+0000 7fc0aaffd640 1 -- 172.21.15.83:0/3205905547 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ac06e520 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.990+0000 7fc0aaffd640 1 --2- 172.21.15.83:0/3205905547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0ac06d6f0 0x7fc0ac152b50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac06e170 tx=0x7fc09c01e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2333953194e3c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.990+0000 7fc0aa7fc640 1 -- 172.21.15.83:0/3205905547 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c0ab680 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.990+0000 7fc0aa7fc640 1 -- 172.21.15.83:0/3205905547 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc09c0ab820 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.990+0000 7fc0aa7fc640 1 -- 172.21.15.83:0/3205905547 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c0abbf0 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.990+0000 7fc0b14ef640 1 -- 172.21.15.83:0/3205905547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0ac06d6f0 msgr2=0x7fc0ac152b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:01.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.990+0000 7fc0b14ef640 1 --2- 172.21.15.83:0/3205905547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0ac06d6f0 0x7fc0ac152b50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac06e170 tx=0x7fc09c01e720 comp rx=0 tx=0).stop 2024-09-19T07:23:01.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.990+0000 7fc0b14ef640 1 -- 172.21.15.83:0/3205905547 shutdown_connections 2024-09-19T07:23:01.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.990+0000 7fc0b14ef640 1 --2- 172.21.15.83:0/3205905547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0ac06d6f0 0x7fc0ac152b50 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:01.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.990+0000 7fc0b14ef640 1 -- 172.21.15.83:0/3205905547 >> 172.21.15.83:0/3205905547 conn(0x7fc0ac0c7e40 msgr2=0x7fc0ac0c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:01.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.990+0000 7fc0b14ef640 1 -- 172.21.15.83:0/3205905547 shutdown_connections 2024-09-19T07:23:01.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.990+0000 7fc0b14ef640 1 -- 172.21.15.83:0/3205905547 wait complete. 2024-09-19T07:23:01.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0b14ef640 1 Processor -- start 2024-09-19T07:23:01.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0b14ef640 1 -- start start 2024-09-19T07:23:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0b14ef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0ac06d6f0 0x7fc0ac14c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0b14ef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0ac0c2fb0 0x7fc0ac1ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0b14ef640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0ac14f110 0x7fc0ac1efdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0b14ef640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ac0c3b30 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0b14ef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ac0c2840 con 0x7fc0ac0c2fb0 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0b14ef640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc0ac0c3cb0 con 0x7fc0ac14f110 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0ac06d6f0 0x7fc0ac14c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0ac06d6f0 0x7fc0ac14c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56770/0 (socket says 172.21.15.83:56770) 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0aaffd640 1 -- 172.21.15.83:0/1027661097 learned_addr learned my addr 172.21.15.83:0/1027661097 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0ab7fe640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0ac14f110 0x7fc0ac1efdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0a3fff640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0ac0c2fb0 0x7fc0ac1ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0aaffd640 1 -- 172.21.15.83:0/1027661097 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0ac14f110 msgr2=0x7fc0ac1efdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0aaffd640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0ac14f110 0x7fc0ac1efdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0aaffd640 1 -- 172.21.15.83:0/1027661097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0ac0c2fb0 msgr2=0x7fc0ac1ef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0aaffd640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0ac0c2fb0 0x7fc0ac1ef890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0aaffd640 1 -- 172.21.15.83:0/1027661097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ac152530 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0a3fff640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0ac0c2fb0 0x7fc0ac1ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0ab7fe640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0ac14f110 0x7fc0ac1efdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0aaffd640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0ac06d6f0 0x7fc0ac14c930 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc09c024d30 tx=0x7fc09c01b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0a37fe640 1 -- 172.21.15.83:0/1027661097 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c0aa870 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0a37fe640 1 -- 172.21.15.83:0/1027661097 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc09c0aaa10 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0b14ef640 1 -- 172.21.15.83:0/1027661097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0ac14ac50 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0a37fe640 1 -- 172.21.15.83:0/1027661097 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c0aade0 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0b14ef640 1 -- 172.21.15.83:0/1027661097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0ac150fd0 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.994+0000 7fc0b14ef640 1 -- 172.21.15.83:0/1027661097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0ac06db80 con 0x7fc0ac06d6f0 2024-09-19T07:23:01.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.998+0000 7fc0a37fe640 1 -- 172.21.15.83:0/1027661097 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc09c0a9070 con 0x7fc0ac06d6f0 2024-09-19T07:23:02.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.998+0000 7fc0a37fe640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc08c067910 0x7fc08c069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:02.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.998+0000 7fc0a37fe640 1 -- 172.21.15.83:0/1027661097 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7fc09c1205b0 con 0x7fc0ac06d6f0 2024-09-19T07:23:02.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.998+0000 7fc0a3fff640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc08c067910 0x7fc08c069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:02.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:01.998+0000 7fc0a3fff640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc08c067910 0x7fc08c069dd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac150de0 tx=0x7fc094009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:02.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.002+0000 7fc0a37fe640 1 -- 172.21.15.83:0/1027661097 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc09c0af020 con 0x7fc0ac06d6f0 2024-09-19T07:23:02.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.262+0000 7fc0b14ef640 1 -- 172.21.15.83:0/1027661097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0ac0c84d0 con 0x7fc0ac06d6f0 2024-09-19T07:23:02.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.262+0000 7fc0a37fe640 1 -- 172.21.15.83:0/1027661097 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6539 (secure 0 0 0) 0x7fc09c018310 con 0x7fc0ac06d6f0 2024-09-19T07:23:02.263 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:02.263 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"btime":"2024-09-19T07:22:38:226763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:23:02.264 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 33 2024-09-19T07:23:02.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.266+0000 7fc0b14ef640 1 -- 172.21.15.83:0/1027661097 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc08c067910 msgr2=0x7fc08c069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:02.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.266+0000 7fc0b14ef640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc08c067910 0x7fc08c069dd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac150de0 tx=0x7fc094009040 comp rx=0 tx=0).stop 2024-09-19T07:23:02.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.266+0000 7fc0b14ef640 1 -- 172.21.15.83:0/1027661097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0ac06d6f0 msgr2=0x7fc0ac14c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:02.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.266+0000 7fc0b14ef640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0ac06d6f0 0x7fc0ac14c930 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc09c024d30 tx=0x7fc09c01b4c0 comp rx=0 tx=0).stop 2024-09-19T07:23:02.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.266+0000 7fc0b14ef640 1 -- 172.21.15.83:0/1027661097 shutdown_connections 2024-09-19T07:23:02.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.266+0000 7fc0b14ef640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc08c067910 0x7fc08c069dd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:02.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.266+0000 7fc0b14ef640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0ac14f110 0x7fc0ac1efdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:02.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.266+0000 7fc0b14ef640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0ac0c2fb0 0x7fc0ac1ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:02.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.266+0000 7fc0b14ef640 1 --2- 172.21.15.83:0/1027661097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0ac06d6f0 0x7fc0ac14c930 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:02.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.266+0000 7fc0b14ef640 1 -- 172.21.15.83:0/1027661097 >> 172.21.15.83:0/1027661097 conn(0x7fc0ac0c7e40 msgr2=0x7fc0ac14f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:02.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.266+0000 7fc0b14ef640 1 -- 172.21.15.83:0/1027661097 shutdown_connections 2024-09-19T07:23:02.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:02.266+0000 7fc0b14ef640 1 -- 172.21.15.83:0/1027661097 wait complete. 2024-09-19T07:23:02.283 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:23:04.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:04.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c19bdf640 1 Processor -- start 2024-09-19T07:23:04.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c19bdf640 1 -- start start 2024-09-19T07:23:04.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c19bdf640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1c140c2240 0x7f1c14156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:04.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c19bdf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c140c4570 0x7f1c140c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:04.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c19bdf640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c1406d250 con 0x7f1c140c3110 2024-09-19T07:23:04.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c19bdf640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f1c14156eb0 con 0x7f1c140c2240 2024-09-19T07:23:04.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c19bdf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1c14153970 con 0x7f1c140c4570 2024-09-19T07:23:04.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c12ffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1c140c2240 0x7f1c14156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:04.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c13fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c140c4570 0x7f1c140c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:04.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c137fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f1c140c3110 0x7f1c14153260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:54138/0 (socket says 172.21.15.83:54138) 2024-09-19T07:23:04.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c137fe640 1 -- 172.21.15.83:0/2881910178 learned_addr learned my addr 172.21.15.83:0/2881910178 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:04.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c12ffd640 1 -- 172.21.15.83:0/2881910178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c140c4570 msgr2=0x7f1c140c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:04.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c12ffd640 1 --2- 172.21.15.83:0/2881910178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c140c4570 0x7f1c140c4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:04.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c12ffd640 1 -- 172.21.15.83:0/2881910178 >> v1:172.21.15.184:6789/0 conn(0x7f1c140c3110 legacy=0x7f1c14153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:04.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c12ffd640 1 -- 172.21.15.83:0/2881910178 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1406d530 con 0x7f1c140c2240 2024-09-19T07:23:04.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.422+0000 7f1c13fff640 1 --2- 172.21.15.83:0/2881910178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c140c4570 0x7f1c140c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:23:04.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.426+0000 7f1c12ffd640 1 --2- 172.21.15.83:0/2881910178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c140c2240 0x7f1c14156970 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1c080097c0 tx=0x7f1c0801c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0657959ca7198ae server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:04.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.426+0000 7f1c127fc640 1 -- 172.21.15.83:0/2881910178 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c080b0070 con 0x7f1c140c2240 2024-09-19T07:23:04.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.426+0000 7f1c127fc640 1 -- 172.21.15.83:0/2881910178 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c080ab600 con 0x7f1c140c2240 2024-09-19T07:23:04.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.426+0000 7f1c127fc640 1 -- 172.21.15.83:0/2881910178 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c080ab9d0 con 0x7f1c140c2240 2024-09-19T07:23:04.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.426+0000 7f1c19bdf640 1 -- 172.21.15.83:0/2881910178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c140c2240 msgr2=0x7f1c14156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:04.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.426+0000 7f1c19bdf640 1 --2- 172.21.15.83:0/2881910178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c140c2240 0x7f1c14156970 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1c080097c0 tx=0x7f1c0801c380 comp rx=0 tx=0).stop 2024-09-19T07:23:04.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.426+0000 7f1c19bdf640 1 -- 172.21.15.83:0/2881910178 shutdown_connections 2024-09-19T07:23:04.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.426+0000 7f1c19bdf640 1 --2- 172.21.15.83:0/2881910178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c140c4570 0x7f1c140c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:04.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.426+0000 7f1c19bdf640 1 --2- 172.21.15.83:0/2881910178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c140c2240 0x7f1c14156970 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:04.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.426+0000 7f1c19bdf640 1 -- 172.21.15.83:0/2881910178 >> 172.21.15.83:0/2881910178 conn(0x7f1c1406c080 msgr2=0x7f1c1406c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:04.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.426+0000 7f1c19bdf640 1 -- 172.21.15.83:0/2881910178 shutdown_connections 2024-09-19T07:23:04.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.426+0000 7f1c19bdf640 1 -- 172.21.15.83:0/2881910178 wait complete. 2024-09-19T07:23:04.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c19bdf640 1 Processor -- start 2024-09-19T07:23:04.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c19bdf640 1 -- start start 2024-09-19T07:23:04.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c19bdf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c140c2240 0x7f1c14073b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:04.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c19bdf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c140c3110 0x7f1c1415a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:04.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c19bdf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c140c4570 0x7f1c1415a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:04.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c19bdf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c141529f0 con 0x7f1c140c4570 2024-09-19T07:23:04.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c19bdf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1c140c34f0 con 0x7f1c140c2240 2024-09-19T07:23:04.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c19bdf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1c140c3f00 con 0x7f1c140c3110 2024-09-19T07:23:04.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c13fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c140c4570 0x7f1c1415a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:04.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c13fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c140c4570 0x7f1c1415a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56782/0 (socket says 172.21.15.83:56782) 2024-09-19T07:23:04.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c13fff640 1 -- 172.21.15.83:0/586643180 learned_addr learned my addr 172.21.15.83:0/586643180 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:04.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c137fe640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c140c2240 0x7f1c14073b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:04.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c12ffd640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c140c3110 0x7f1c1415a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:04.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c13fff640 1 -- 172.21.15.83:0/586643180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c140c3110 msgr2=0x7f1c1415a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:04.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c13fff640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c140c3110 0x7f1c1415a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:04.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c13fff640 1 -- 172.21.15.83:0/586643180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c140c2240 msgr2=0x7f1c14073b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:04.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c13fff640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c140c2240 0x7f1c14073b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:04.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c13fff640 1 -- 172.21.15.83:0/586643180 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1415c340 con 0x7f1c140c4570 2024-09-19T07:23:04.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c13fff640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c140c4570 0x7f1c1415a6c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00efc0 tx=0x7f1c0c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:04.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1bfbfff640 1 -- 172.21.15.83:0/586643180 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c018030 con 0x7f1c140c4570 2024-09-19T07:23:04.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1bfbfff640 1 -- 172.21.15.83:0/586643180 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c0c00cde0 con 0x7f1c140c4570 2024-09-19T07:23:04.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c19bdf640 1 -- 172.21.15.83:0/586643180 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c141f0280 con 0x7f1c140c4570 2024-09-19T07:23:04.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1bfbfff640 1 -- 172.21.15.83:0/586643180 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c003bc0 con 0x7f1c140c4570 2024-09-19T07:23:04.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c19bdf640 1 -- 172.21.15.83:0/586643180 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c140c1540 con 0x7f1c140c4570 2024-09-19T07:23:04.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.430+0000 7f1c19bdf640 1 -- 172.21.15.83:0/586643180 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c1406c5d0 con 0x7f1c140c4570 2024-09-19T07:23:04.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.434+0000 7f1bfbfff640 1 -- 172.21.15.83:0/586643180 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1c0c01b070 con 0x7f1c140c4570 2024-09-19T07:23:04.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.434+0000 7f1bfbfff640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c000678c0 0x7f1c00069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:04.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.434+0000 7f1bfbfff640 1 -- 172.21.15.83:0/586643180 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7f1c0c087270 con 0x7f1c140c4570 2024-09-19T07:23:04.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.434+0000 7f1c137fe640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c000678c0 0x7f1c00069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:04.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.434+0000 7f1c137fe640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c000678c0 0x7f1c00069d80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1c04000f70 tx=0x7f1c04007e90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:04.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.438+0000 7f1bfbfff640 1 -- 172.21.15.83:0/586643180 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1c0c022070 con 0x7f1c140c4570 2024-09-19T07:23:04.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.670+0000 7f1c19bdf640 1 -- 172.21.15.83:0/586643180 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c14156950 con 0x7f1c140c4570 2024-09-19T07:23:04.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.670+0000 7f1bfbfff640 1 -- 172.21.15.83:0/586643180 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6539 (secure 0 0 0) 0x7f1c0c010040 con 0x7f1c140c4570 2024-09-19T07:23:04.671 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:04.672 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"btime":"2024-09-19T07:22:38:226763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:23:04.672 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 33 2024-09-19T07:23:04.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.674+0000 7f1c19bdf640 1 -- 172.21.15.83:0/586643180 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c000678c0 msgr2=0x7f1c00069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:04.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.674+0000 7f1c19bdf640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c000678c0 0x7f1c00069d80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1c04000f70 tx=0x7f1c04007e90 comp rx=0 tx=0).stop 2024-09-19T07:23:04.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.674+0000 7f1c19bdf640 1 -- 172.21.15.83:0/586643180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c140c4570 msgr2=0x7f1c1415a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:04.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.674+0000 7f1c19bdf640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c140c4570 0x7f1c1415a6c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00efc0 tx=0x7f1c0c009f90 comp rx=0 tx=0).stop 2024-09-19T07:23:04.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.674+0000 7f1c19bdf640 1 -- 172.21.15.83:0/586643180 shutdown_connections 2024-09-19T07:23:04.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.674+0000 7f1c19bdf640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c000678c0 0x7f1c00069d80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:04.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.674+0000 7f1c19bdf640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c140c4570 0x7f1c1415a6c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:04.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.674+0000 7f1c19bdf640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c140c3110 0x7f1c1415a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:04.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.674+0000 7f1c19bdf640 1 --2- 172.21.15.83:0/586643180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c140c2240 0x7f1c14073b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:04.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.674+0000 7f1c19bdf640 1 -- 172.21.15.83:0/586643180 >> 172.21.15.83:0/586643180 conn(0x7f1c1406c080 msgr2=0x7f1c14150d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:04.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.674+0000 7f1c19bdf640 1 -- 172.21.15.83:0/586643180 shutdown_connections 2024-09-19T07:23:04.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:04.674+0000 7f1c19bdf640 1 -- 172.21.15.83:0/586643180 wait complete. 2024-09-19T07:23:04.694 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:23:06.695 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:06.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.854+0000 7fd219347640 1 Processor -- start 2024-09-19T07:23:06.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.854+0000 7fd219347640 1 -- start start 2024-09-19T07:23:06.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.858+0000 7fd219347640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd21414d9f0 0x7fd21406de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:06.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.858+0000 7fd219347640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2140c7510 0x7fd2140c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:06.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.858+0000 7fd219347640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd21414ddd0 con 0x7fd2140c9200 2024-09-19T07:23:06.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.858+0000 7fd219347640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd2140c8830 con 0x7fd21414d9f0 2024-09-19T07:23:06.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.858+0000 7fd219347640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd21406fdf0 con 0x7fd2140c7510 2024-09-19T07:23:06.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.858+0000 7fd212ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd21414d9f0 0x7fd21406de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:06.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.858+0000 7fd212ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd21414d9f0 0x7fd21406de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56788/0 (socket says 172.21.15.83:56788) 2024-09-19T07:23:06.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.858+0000 7fd212ffd640 1 -- 172.21.15.83:0/3022055097 learned_addr learned my addr 172.21.15.83:0/3022055097 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:06.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.858+0000 7fd212ffd640 1 -- 172.21.15.83:0/3022055097 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2140c7510 msgr2=0x7fd2140c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:06.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.858+0000 7fd212ffd640 1 --2- 172.21.15.83:0/3022055097 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2140c7510 0x7fd2140c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:06.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.858+0000 7fd212ffd640 1 -- 172.21.15.83:0/3022055097 >> v1:172.21.15.184:6789/0 conn(0x7fd2140c9200 legacy=0x7fd2140c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:06.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.858+0000 7fd212ffd640 1 -- 172.21.15.83:0/3022055097 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2140c7f00 con 0x7fd21414d9f0 2024-09-19T07:23:06.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.862+0000 7fd212ffd640 1 --2- 172.21.15.83:0/3022055097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd21414d9f0 0x7fd21406de90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd204009870 tx=0x7fd20401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83cfa11cd9787a7c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:06.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.862+0000 7fd211ffb640 1 -- 172.21.15.83:0/3022055097 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2040b0070 con 0x7fd21414d9f0 2024-09-19T07:23:06.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.862+0000 7fd211ffb640 1 -- 172.21.15.83:0/3022055097 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2040ab600 con 0x7fd21414d9f0 2024-09-19T07:23:06.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.862+0000 7fd211ffb640 1 -- 172.21.15.83:0/3022055097 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2040ab9d0 con 0x7fd21414d9f0 2024-09-19T07:23:06.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.862+0000 7fd219347640 1 -- 172.21.15.83:0/3022055097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd21414d9f0 msgr2=0x7fd21406de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:06.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.862+0000 7fd219347640 1 --2- 172.21.15.83:0/3022055097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd21414d9f0 0x7fd21406de90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd204009870 tx=0x7fd20401c490 comp rx=0 tx=0).stop 2024-09-19T07:23:06.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.862+0000 7fd219347640 1 -- 172.21.15.83:0/3022055097 shutdown_connections 2024-09-19T07:23:06.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.862+0000 7fd219347640 1 --2- 172.21.15.83:0/3022055097 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2140c7510 0x7fd2140c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:06.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.862+0000 7fd219347640 1 --2- 172.21.15.83:0/3022055097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd21414d9f0 0x7fd21406de90 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:06.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.862+0000 7fd219347640 1 -- 172.21.15.83:0/3022055097 >> 172.21.15.83:0/3022055097 conn(0x7fd2140c6110 msgr2=0x7fd2140c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:06.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.862+0000 7fd219347640 1 -- 172.21.15.83:0/3022055097 shutdown_connections 2024-09-19T07:23:06.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.862+0000 7fd219347640 1 -- 172.21.15.83:0/3022055097 wait complete. 2024-09-19T07:23:06.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd219347640 1 Processor -- start 2024-09-19T07:23:06.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd219347640 1 -- start start 2024-09-19T07:23:06.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd219347640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2140c7510 0x7fd214143a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:06.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd219347640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2140c9200 0x7fd2141efb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:06.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd219347640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd21414d9f0 0x7fd2141f0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:06.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd219347640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd214160c00 con 0x7fd2140c7510 2024-09-19T07:23:06.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd219347640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd214160a80 con 0x7fd2140c9200 2024-09-19T07:23:06.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd219347640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd214160d80 con 0x7fd21414d9f0 2024-09-19T07:23:06.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd212ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2140c7510 0x7fd214143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:06.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd212ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2140c7510 0x7fd214143a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56796/0 (socket says 172.21.15.83:56796) 2024-09-19T07:23:06.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd212ffd640 1 -- 172.21.15.83:0/1988027752 learned_addr learned my addr 172.21.15.83:0/1988027752 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:06.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd2127fc640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2140c9200 0x7fd2141efb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:06.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd2137fe640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd21414d9f0 0x7fd2141f0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:06.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd212ffd640 1 -- 172.21.15.83:0/1988027752 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd21414d9f0 msgr2=0x7fd2141f0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:06.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd212ffd640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd21414d9f0 0x7fd2141f0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:06.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd212ffd640 1 -- 172.21.15.83:0/1988027752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2140c9200 msgr2=0x7fd2141efb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:06.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd212ffd640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2140c9200 0x7fd2141efb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:06.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd212ffd640 1 -- 172.21.15.83:0/1988027752 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd21414c180 con 0x7fd2140c7510 2024-09-19T07:23:06.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd212ffd640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2140c7510 0x7fd214143a60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd2040ae040 tx=0x7fd20401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:06.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd1fb7fe640 1 -- 172.21.15.83:0/1988027752 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2040c1070 con 0x7fd2140c7510 2024-09-19T07:23:06.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd1fb7fe640 1 -- 172.21.15.83:0/1988027752 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2040b43f0 con 0x7fd2140c7510 2024-09-19T07:23:06.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd219347640 1 -- 172.21.15.83:0/1988027752 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd21414c500 con 0x7fd2140c7510 2024-09-19T07:23:06.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd1fb7fe640 1 -- 172.21.15.83:0/1988027752 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2040b0040 con 0x7fd2140c7510 2024-09-19T07:23:06.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd219347640 1 -- 172.21.15.83:0/1988027752 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd21414cac0 con 0x7fd2140c7510 2024-09-19T07:23:06.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd219347640 1 -- 172.21.15.83:0/1988027752 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2140c8cc0 con 0x7fd2140c7510 2024-09-19T07:23:06.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.866+0000 7fd1fb7fe640 1 -- 172.21.15.83:0/1988027752 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd2040c4070 con 0x7fd2140c7510 2024-09-19T07:23:06.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.870+0000 7fd1fb7fe640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd2000678c0 0x7fd200069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:06.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.870+0000 7fd1fb7fe640 1 -- 172.21.15.83:0/1988027752 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6981+0+0 (secure 0 0 0) 0x7fd2041286c0 con 0x7fd2140c7510 2024-09-19T07:23:06.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.870+0000 7fd2127fc640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd2000678c0 0x7fd200069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:06.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.870+0000 7fd2127fc640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd2000678c0 0x7fd200069d80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd208006fd0 tx=0x7fd208008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:06.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:06.874+0000 7fd1fb7fe640 1 -- 172.21.15.83:0/1988027752 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd204016110 con 0x7fd2140c7510 2024-09-19T07:23:07.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.134+0000 7fd219347640 1 -- 172.21.15.83:0/1988027752 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd21406faf0 con 0x7fd2140c7510 2024-09-19T07:23:07.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.134+0000 7fd1fb7fe640 1 -- 172.21.15.83:0/1988027752 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6539 (secure 0 0 0) 0x7fd2040f8e30 con 0x7fd2140c7510 2024-09-19T07:23:07.137 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:07.138 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"btime":"2024-09-19T07:22:38:226763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:22:26.150314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5025":{"gid":5025,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/1774585869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1774585869},{"type":"v1","addr":"172.21.15.83:6835","nonce":1774585869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5025,"qdb_cluster":[5025]},"id":2}]} 2024-09-19T07:23:07.138 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 33 2024-09-19T07:23:07.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.138+0000 7fd219347640 1 -- 172.21.15.83:0/1988027752 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd2000678c0 msgr2=0x7fd200069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:07.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.138+0000 7fd219347640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd2000678c0 0x7fd200069d80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd208006fd0 tx=0x7fd208008040 comp rx=0 tx=0).stop 2024-09-19T07:23:07.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.138+0000 7fd219347640 1 -- 172.21.15.83:0/1988027752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2140c7510 msgr2=0x7fd214143a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.138+0000 7fd219347640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2140c7510 0x7fd214143a60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd2040ae040 tx=0x7fd20401acd0 comp rx=0 tx=0).stop 2024-09-19T07:23:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.138+0000 7fd219347640 1 -- 172.21.15.83:0/1988027752 shutdown_connections 2024-09-19T07:23:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.138+0000 7fd219347640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd2000678c0 0x7fd200069d80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.138+0000 7fd219347640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd21414d9f0 0x7fd2141f0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.138+0000 7fd219347640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2140c9200 0x7fd2141efb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.138+0000 7fd219347640 1 --2- 172.21.15.83:0/1988027752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2140c7510 0x7fd214143a60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.138+0000 7fd219347640 1 -- 172.21.15.83:0/1988027752 >> 172.21.15.83:0/1988027752 conn(0x7fd2140c6110 msgr2=0x7fd2140c64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.138+0000 7fd219347640 1 -- 172.21.15.83:0/1988027752 shutdown_connections 2024-09-19T07:23:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:07.138+0000 7fd219347640 1 -- 172.21.15.83:0/1988027752 wait complete. 2024-09-19T07:23:07.160 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:23:09.161 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:09.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.302+0000 7f3229fb3640 1 Processor -- start 2024-09-19T07:23:09.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.302+0000 7f3229fb3640 1 -- start start 2024-09-19T07:23:09.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3229fb3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f32240c4860 0x7f32240c2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3229fb3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32240c1e80 con 0x7f32240c2b70 2024-09-19T07:23:09.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3229fb3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32240a7350 con 0x7f32240715f0 2024-09-19T07:23:09.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3229fb3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f32240c3070 con 0x7f32240c4860 2024-09-19T07:23:09.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3222ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f32240c2b70 0x7f32240c1060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57106/0 (socket says 172.21.15.83:57106) 2024-09-19T07:23:09.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3222ffd640 1 -- 172.21.15.83:0/77059608 learned_addr learned my addr 172.21.15.83:0/77059608 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:09.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4241623625 0 0) 0x7f32240c1e80 con 0x7f32240c2b70 2024-09-19T07:23:09.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f320c003530 con 0x7f32240c2b70 2024-09-19T07:23:09.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32237fe640 1 --2- 172.21.15.83:0/77059608 >> v2:172.21.15.184:3300/0 conn(0x7f32240c4860 0x7f32240c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1054358803 0 0) 0x7f320c003530 con 0x7f32240c2b70 2024-09-19T07:23:09.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 >> v2:172.21.15.184:3300/0 conn(0x7f32240c4860 msgr2=0x7f32240c2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:09.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 --2- 172.21.15.83:0/77059608 >> v2:172.21.15.184:3300/0 conn(0x7f32240c4860 0x7f32240c2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 >> v1:172.21.15.184:6789/0 conn(0x7f32240715f0 legacy=0x7f32240c1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32240c3350 con 0x7f32240c2b70 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f3218001680 con 0x7f32240c2b70 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 >> v1:172.21.15.83:6789/0 conn(0x7f32240c2b70 legacy=0x7f32240c1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 --2- 172.21.15.83:0/77059608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f320c0039c0 0x7f320c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 --2- 172.21.15.83:0/77059608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c004810 0x7f320c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 --2- 172.21.15.83:0/77059608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 0x7f320c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f32240c3070 con 0x7f320c0051d0 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f320c00c170 con 0x7f320c0039c0 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f320c00c370 con 0x7f320c004810 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3223fff640 1 --2- 172.21.15.83:0/77059608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 0x7f320c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f32237fe640 1 --2- 172.21.15.83:0/77059608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f320c0039c0 0x7f320c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3222ffd640 1 --2- 172.21.15.83:0/77059608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c004810 0x7f320c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3223fff640 1 -- 172.21.15.83:0/77059608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c004810 msgr2=0x7f320c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:09.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3223fff640 1 --2- 172.21.15.83:0/77059608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c004810 0x7f320c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3223fff640 1 -- 172.21.15.83:0/77059608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f320c0039c0 msgr2=0x7f320c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:09.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3223fff640 1 --2- 172.21.15.83:0/77059608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f320c0039c0 0x7f320c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.306+0000 7f3223fff640 1 -- 172.21.15.83:0/77059608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f320c003530 con 0x7f320c0051d0 2024-09-19T07:23:09.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3223fff640 1 -- 172.21.15.83:0/77059608 reap_dead start 2024-09-19T07:23:09.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3223fff640 1 --2- 172.21.15.83:0/77059608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 0x7f320c00b9d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f321c001e40 tx=0x7f321c01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700ad5006e228ed1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:09.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c001a90 con 0x7f320c0051d0 2024-09-19T07:23:09.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f321c003b80 con 0x7f320c0051d0 2024-09-19T07:23:09.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f32227fc640 1 -- 172.21.15.83:0/77059608 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c0050d0 con 0x7f320c0051d0 2024-09-19T07:23:09.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 -- 172.21.15.83:0/77059608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 msgr2=0x7f320c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 --2- 172.21.15.83:0/77059608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 0x7f320c00b9d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f321c001e40 tx=0x7f321c01c760 comp rx=0 tx=0).stop 2024-09-19T07:23:09.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 -- 172.21.15.83:0/77059608 shutdown_connections 2024-09-19T07:23:09.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 --2- 172.21.15.83:0/77059608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 0x7f320c00b9d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 -- 172.21.15.83:0/77059608 >> 172.21.15.83:0/77059608 conn(0x7f32240c7e00 msgr2=0x7f32240c8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:09.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 -- 172.21.15.83:0/77059608 shutdown_connections 2024-09-19T07:23:09.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 -- 172.21.15.83:0/77059608 wait complete. 2024-09-19T07:23:09.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 Processor -- start 2024-09-19T07:23:09.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 -- start start 2024-09-19T07:23:09.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 0x7f32241dbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f322406f1c0 0x7f32241ee800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32241eed40 0x7f32241d7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f32240a7620 con 0x7f320c0051d0 2024-09-19T07:23:09.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f32240a7060 con 0x7f32241eed40 2024-09-19T07:23:09.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3229fb3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f32240a77a0 con 0x7f322406f1c0 2024-09-19T07:23:09.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f32237fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 0x7f32241dbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f32237fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 0x7f32241dbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56804/0 (socket says 172.21.15.83:56804) 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f32237fe640 1 -- 172.21.15.83:0/901625984 learned_addr learned my addr 172.21.15.83:0/901625984 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.310+0000 7f3222ffd640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f322406f1c0 0x7f32241ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f3223fff640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32241eed40 0x7f32241d7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f32237fe640 1 -- 172.21.15.83:0/901625984 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f322406f1c0 msgr2=0x7f32241ee800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f32237fe640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f322406f1c0 0x7f32241ee800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f32237fe640 1 -- 172.21.15.83:0/901625984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32241eed40 msgr2=0x7f32241d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f32237fe640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32241eed40 0x7f32241d7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f32237fe640 1 -- 172.21.15.83:0/901625984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32241d88b0 con 0x7f320c0051d0 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f3223fff640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32241eed40 0x7f32241d7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f32237fe640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 0x7f32241dbd40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3214011560 tx=0x7f3214011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f320bfff640 1 -- 172.21.15.83:0/901625984 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3214015930 con 0x7f320c0051d0 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f320bfff640 1 -- 172.21.15.83:0/901625984 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3214015ad0 con 0x7f320c0051d0 2024-09-19T07:23:09.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f3229fb3640 1 -- 172.21.15.83:0/901625984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32241e15f0 con 0x7f320c0051d0 2024-09-19T07:23:09.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f320bfff640 1 -- 172.21.15.83:0/901625984 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321400f460 con 0x7f320c0051d0 2024-09-19T07:23:09.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f3229fb3640 1 -- 172.21.15.83:0/901625984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32241dc280 con 0x7f320c0051d0 2024-09-19T07:23:09.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f3229fb3640 1 -- 172.21.15.83:0/901625984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32240aace0 con 0x7f320c0051d0 2024-09-19T07:23:09.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f320bfff640 1 -- 172.21.15.83:0/901625984 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f321400f600 con 0x7f320c0051d0 2024-09-19T07:23:09.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f320bfff640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f32100678f0 0x7f3210069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f320bfff640 1 -- 172.21.15.83:0/901625984 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f3214086a90 con 0x7f320c0051d0 2024-09-19T07:23:09.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f3222ffd640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f32100678f0 0x7f3210069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.314+0000 7f3222ffd640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f32100678f0 0x7f3210069db0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f32241dbab0 tx=0x7f3218009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:09.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.318+0000 7f320bfff640 1 -- 172.21.15.83:0/901625984 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3214019030 con 0x7f320c0051d0 2024-09-19T07:23:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.550+0000 7f3229fb3640 1 -- 172.21.15.83:0/901625984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32240733b0 con 0x7f320c0051d0 2024-09-19T07:23:09.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f320bfff640 1 -- 172.21.15.83:0/901625984 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5709 (secure 0 0 0) 0x7f3214057110 con 0x7f320c0051d0 2024-09-19T07:23:09.554 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:09.554 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":35,"btime":"2024-09-19T07:23:08:982843+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:08.227375+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:23:09.554 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 35 2024-09-19T07:23:09.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f3229fb3640 1 -- 172.21.15.83:0/901625984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f32100678f0 msgr2=0x7f3210069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f3229fb3640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f32100678f0 0x7f3210069db0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f32241dbab0 tx=0x7f3218009040 comp rx=0 tx=0).stop 2024-09-19T07:23:09.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f3229fb3640 1 -- 172.21.15.83:0/901625984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 msgr2=0x7f32241dbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f3229fb3640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 0x7f32241dbd40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3214011560 tx=0x7f3214011a20 comp rx=0 tx=0).stop 2024-09-19T07:23:09.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f3229fb3640 1 -- 172.21.15.83:0/901625984 shutdown_connections 2024-09-19T07:23:09.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f3229fb3640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f32100678f0 0x7f3210069db0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f3229fb3640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32241eed40 0x7f32241d7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f3229fb3640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f322406f1c0 0x7f32241ee800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f3229fb3640 1 --2- 172.21.15.83:0/901625984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0051d0 0x7f32241dbd40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f3229fb3640 1 -- 172.21.15.83:0/901625984 >> 172.21.15.83:0/901625984 conn(0x7f32240c7e00 msgr2=0x7f32240c81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:09.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f3229fb3640 1 -- 172.21.15.83:0/901625984 shutdown_connections 2024-09-19T07:23:09.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.554+0000 7f3229fb3640 1 -- 172.21.15.83:0/901625984 wait complete. 2024-09-19T07:23:09.573 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-19T07:23:09.573 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:23:09.573 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:09.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.682+0000 7f21d0e3f640 1 Processor -- start 2024-09-19T07:23:09.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.682+0000 7f21d0e3f640 1 -- start start 2024-09-19T07:23:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.682+0000 7f21d0e3f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21cc14a500 0x7f21cc0c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.682+0000 7f21d0e3f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21cc151750 con 0x7f21cc154540 2024-09-19T07:23:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.682+0000 7f21d0e3f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21cc0c7e90 con 0x7f21cc0c4550 2024-09-19T07:23:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.682+0000 7f21d0e3f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f21cc073110 con 0x7f21cc14a500 2024-09-19T07:23:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.682+0000 7f21ca59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21cc14a500 0x7f21cc0c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.682+0000 7f21ca59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21cc14a500 0x7f21cc0c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56814/0 (socket says 172.21.15.83:56814) 2024-09-19T07:23:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.682+0000 7f21ca59c640 1 -- 172.21.15.83:0/2767818237 learned_addr learned my addr 172.21.15.83:0/2767818237 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21ca59c640 1 -- 172.21.15.83:0/2767818237 >> v1:172.21.15.184:6789/0 conn(0x7f21cc0c4550 legacy=0x7f21cc15e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21ca59c640 1 -- 172.21.15.83:0/2767818237 >> v1:172.21.15.83:6789/0 conn(0x7f21cc154540 legacy=0x7f21cc157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21ca59c640 1 -- 172.21.15.83:0/2767818237 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21cc0733f0 con 0x7f21cc14a500 2024-09-19T07:23:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21ca59c640 1 --2- 172.21.15.83:0/2767818237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21cc14a500 0x7f21cc0c4010 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009870 tx=0x7f21bc01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd26c167a22f4157 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:09.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21c959a640 1 -- 172.21.15.83:0/2767818237 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0ab600 con 0x7f21cc14a500 2024-09-19T07:23:09.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21c959a640 1 -- 172.21.15.83:0/2767818237 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21bc0b0070 con 0x7f21cc14a500 2024-09-19T07:23:09.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21c959a640 1 -- 172.21.15.83:0/2767818237 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0ab9c0 con 0x7f21cc14a500 2024-09-19T07:23:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21d0e3f640 1 -- 172.21.15.83:0/2767818237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21cc14a500 msgr2=0x7f21cc0c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21d0e3f640 1 --2- 172.21.15.83:0/2767818237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21cc14a500 0x7f21cc0c4010 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009870 tx=0x7f21bc01c4f0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21d0e3f640 1 -- 172.21.15.83:0/2767818237 shutdown_connections 2024-09-19T07:23:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21d0e3f640 1 --2- 172.21.15.83:0/2767818237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21cc14a500 0x7f21cc0c4010 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21d0e3f640 1 -- 172.21.15.83:0/2767818237 >> 172.21.15.83:0/2767818237 conn(0x7f21cc0c9610 msgr2=0x7f21cc0c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21d0e3f640 1 -- 172.21.15.83:0/2767818237 shutdown_connections 2024-09-19T07:23:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.686+0000 7f21d0e3f640 1 -- 172.21.15.83:0/2767818237 wait complete. 2024-09-19T07:23:09.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21d0e3f640 1 Processor -- start 2024-09-19T07:23:09.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21d0e3f640 1 -- start start 2024-09-19T07:23:09.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21d0e3f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21cc0c4550 0x7f21cc1432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21d0e3f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21cc14a500 0x7f21cc1f02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21d0e3f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21cc154540 0x7f21cc14bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21d0e3f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc0c7850 con 0x7f21cc0c4550 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21d0e3f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc0c76d0 con 0x7f21cc14a500 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21d0e3f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f21cc162790 con 0x7f21cc154540 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21ca59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21cc0c4550 0x7f21cc1432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21ca59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21cc0c4550 0x7f21cc1432d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56826/0 (socket says 172.21.15.83:56826) 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21ca59c640 1 -- 172.21.15.83:0/883891407 learned_addr learned my addr 172.21.15.83:0/883891407 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21cad9d640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21cc154540 0x7f21cc14bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21c9d9b640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21cc14a500 0x7f21cc1f02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21ca59c640 1 -- 172.21.15.83:0/883891407 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21cc154540 msgr2=0x7f21cc14bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21ca59c640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21cc154540 0x7f21cc14bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21ca59c640 1 -- 172.21.15.83:0/883891407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21cc14a500 msgr2=0x7f21cc1f02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21ca59c640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21cc14a500 0x7f21cc1f02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21ca59c640 1 -- 172.21.15.83:0/883891407 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21cc0c6fd0 con 0x7f21cc0c4550 2024-09-19T07:23:09.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21cad9d640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21cc154540 0x7f21cc14bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:09.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21c9d9b640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21cc14a500 0x7f21cc1f02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:09.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21ca59c640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21cc0c4550 0x7f21cc1432d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0ad040 tx=0x7f21bc01ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:09.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21baffd640 1 -- 172.21.15.83:0/883891407 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0b4800 con 0x7f21cc0c4550 2024-09-19T07:23:09.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21baffd640 1 -- 172.21.15.83:0/883891407 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21bc0b49c0 con 0x7f21cc0c4550 2024-09-19T07:23:09.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21d0e3f640 1 -- 172.21.15.83:0/883891407 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21cc1edcc0 con 0x7f21cc0c4550 2024-09-19T07:23:09.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21baffd640 1 -- 172.21.15.83:0/883891407 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0b4db0 con 0x7f21cc0c4550 2024-09-19T07:23:09.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21d0e3f640 1 -- 172.21.15.83:0/883891407 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21cc1ed9e0 con 0x7f21cc0c4550 2024-09-19T07:23:09.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.690+0000 7f21baffd640 1 -- 172.21.15.83:0/883891407 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f21bc0aa070 con 0x7f21cc0c4550 2024-09-19T07:23:09.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.694+0000 7f21d0e3f640 1 -- 172.21.15.83:0/883891407 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21cc0c4010 con 0x7f21cc0c4550 2024-09-19T07:23:09.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.694+0000 7f21baffd640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21ac0678f0 0x7f21ac069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:09.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.694+0000 7f21c9d9b640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21ac0678f0 0x7f21ac069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:09.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.694+0000 7f21baffd640 1 -- 172.21.15.83:0/883891407 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f21bc120470 con 0x7f21cc0c4550 2024-09-19T07:23:09.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.694+0000 7f21c9d9b640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21ac0678f0 0x7f21ac069db0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f21cc1f13c0 tx=0x7f21c0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:09.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.698+0000 7f21baffd640 1 -- 172.21.15.83:0/883891407 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f21bc0c4060 con 0x7f21cc0c4550 2024-09-19T07:23:09.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.922+0000 7f21d0e3f640 1 -- 172.21.15.83:0/883891407 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21cc0c4010 con 0x7f21cc0c4550 2024-09-19T07:23:09.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.922+0000 7f21baffd640 1 -- 172.21.15.83:0/883891407 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5709 (secure 0 0 0) 0x7f21bc016110 con 0x7f21cc0c4550 2024-09-19T07:23:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:09.925 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":35,"btime":"2024-09-19T07:23:08:982843+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:08.227375+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:23:09.926 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 35 2024-09-19T07:23:09.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.926+0000 7f21d0e3f640 1 -- 172.21.15.83:0/883891407 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21ac0678f0 msgr2=0x7f21ac069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.926+0000 7f21d0e3f640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21ac0678f0 0x7f21ac069db0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f21cc1f13c0 tx=0x7f21c0009040 comp rx=0 tx=0).stop 2024-09-19T07:23:09.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.926+0000 7f21d0e3f640 1 -- 172.21.15.83:0/883891407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21cc0c4550 msgr2=0x7f21cc1432d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:09.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.926+0000 7f21d0e3f640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21cc0c4550 0x7f21cc1432d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0ad040 tx=0x7f21bc01ad30 comp rx=0 tx=0).stop 2024-09-19T07:23:09.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.926+0000 7f21d0e3f640 1 -- 172.21.15.83:0/883891407 shutdown_connections 2024-09-19T07:23:09.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.926+0000 7f21d0e3f640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21ac0678f0 0x7f21ac069db0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.926+0000 7f21d0e3f640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21cc154540 0x7f21cc14bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.926+0000 7f21d0e3f640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21cc14a500 0x7f21cc1f02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.926+0000 7f21d0e3f640 1 --2- 172.21.15.83:0/883891407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21cc0c4550 0x7f21cc1432d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:09.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.926+0000 7f21d0e3f640 1 -- 172.21.15.83:0/883891407 >> 172.21.15.83:0/883891407 conn(0x7f21cc0c9610 msgr2=0x7f21cc0c68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:09.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.926+0000 7f21d0e3f640 1 -- 172.21.15.83:0/883891407 shutdown_connections 2024-09-19T07:23:09.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:09.926+0000 7f21d0e3f640 1 -- 172.21.15.83:0/883891407 wait complete. 2024-09-19T07:23:09.944 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:23:08:982843+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 35, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4786": { "addr": "172.21.15.184:6835/4115191178", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 4115191178, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 4115191178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4786, "incarnation": 29, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:22:36.207732+0000", "qdb_cluster": [ 4786 ], "qdb_leader": 4786, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4786 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 35, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5196": { "addr": "172.21.15.83:6837/1049139637", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 1049139637, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 1049139637, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5196, "incarnation": 34, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:23:08.227375+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5196 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 28, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:23:11.945 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:12.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.070+0000 7f59a48f7640 1 Processor -- start 2024-09-19T07:23:12.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.070+0000 7f59a48f7640 1 -- start start 2024-09-19T07:23:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.070+0000 7f59a48f7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f59a00c7d70 0x7f59a0157c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.070+0000 7f59a48f7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59a01547b0 0x7f59a00c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.070+0000 7f59a48f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f59a0072d00 0x7f59a006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.070+0000 7f59a48f7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f59a0073690 con 0x7f59a0072d00 2024-09-19T07:23:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.070+0000 7f59a48f7640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f59a0157160 con 0x7f59a00c7d70 2024-09-19T07:23:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.070+0000 7f59a48f7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f59a0157440 con 0x7f59a01547b0 2024-09-19T07:23:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.070+0000 7f599ed9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f59a0072d00 0x7f59a006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f599ed9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f59a0072d00 0x7f59a006cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59142/0 (socket says 172.21.15.83:59142) 2024-09-19T07:23:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f599ed9d640 1 -- 172.21.15.83:0/1061236402 learned_addr learned my addr 172.21.15.83:0/1061236402 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f599e59c640 1 --2- 172.21.15.83:0/1061236402 >> v2:172.21.15.184:3300/0 conn(0x7f59a00c7d70 0x7f59a0157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f5997fff640 1 --2- 172.21.15.83:0/1061236402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59a01547b0 0x7f59a00c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f599ed9d640 1 -- 172.21.15.83:0/1061236402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59a01547b0 msgr2=0x7f59a00c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:12.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f599ed9d640 1 --2- 172.21.15.83:0/1061236402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59a01547b0 0x7f59a00c6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f599ed9d640 1 -- 172.21.15.83:0/1061236402 >> v2:172.21.15.184:3300/0 conn(0x7f59a00c7d70 msgr2=0x7f59a0157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:12.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f599ed9d640 1 --2- 172.21.15.83:0/1061236402 >> v2:172.21.15.184:3300/0 conn(0x7f59a00c7d70 0x7f59a0157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f599ed9d640 1 -- 172.21.15.83:0/1061236402 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a0157720 con 0x7f59a0072d00 2024-09-19T07:23:12.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f599ed9d640 1 --2- 172.21.15.83:0/1061236402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a0072d00 0x7f59a006cc90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f59a0076000 tx=0x7f599801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760c04bc45e79552 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:12.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f599dd9b640 1 -- 172.21.15.83:0/1061236402 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59980a5630 con 0x7f59a0072d00 2024-09-19T07:23:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f599dd9b640 1 -- 172.21.15.83:0/1061236402 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59980a57d0 con 0x7f59a0072d00 2024-09-19T07:23:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f599dd9b640 1 -- 172.21.15.83:0/1061236402 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59980a5ba0 con 0x7f59a0072d00 2024-09-19T07:23:12.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f59a48f7640 1 -- 172.21.15.83:0/1061236402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a0072d00 msgr2=0x7f59a006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:12.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.074+0000 7f59a48f7640 1 --2- 172.21.15.83:0/1061236402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a0072d00 0x7f59a006cc90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f59a0076000 tx=0x7f599801c8a0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 -- 172.21.15.83:0/1061236402 shutdown_connections 2024-09-19T07:23:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 --2- 172.21.15.83:0/1061236402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a0072d00 0x7f59a006cc90 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 --2- 172.21.15.83:0/1061236402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59a01547b0 0x7f59a00c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 --2- 172.21.15.83:0/1061236402 >> v2:172.21.15.184:3300/0 conn(0x7f59a00c7d70 0x7f59a0157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 -- 172.21.15.83:0/1061236402 >> 172.21.15.83:0/1061236402 conn(0x7f59a00c5060 msgr2=0x7f59a00c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 -- 172.21.15.83:0/1061236402 shutdown_connections 2024-09-19T07:23:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 -- 172.21.15.83:0/1061236402 wait complete. 2024-09-19T07:23:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 Processor -- start 2024-09-19T07:23:12.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 -- start start 2024-09-19T07:23:12.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a0072d00 0x7f59a00c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59a00c7d70 0x7f59a01ef5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59a01547b0 0x7f59a014fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f59a0070230 con 0x7f59a0072d00 2024-09-19T07:23:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f59a00703b0 con 0x7f59a00c7d70 2024-09-19T07:23:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f59a0070530 con 0x7f59a01547b0 2024-09-19T07:23:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f599e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a0072d00 0x7f59a00c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f599e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a0072d00 0x7f59a00c34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59158/0 (socket says 172.21.15.83:59158) 2024-09-19T07:23:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f599e59c640 1 -- 172.21.15.83:0/4181359314 learned_addr learned my addr 172.21.15.83:0/4181359314 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f5997fff640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59a00c7d70 0x7f59a01ef5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f599ed9d640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59a01547b0 0x7f59a014fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f599e59c640 1 -- 172.21.15.83:0/4181359314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59a01547b0 msgr2=0x7f59a014fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f599e59c640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59a01547b0 0x7f59a014fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f599e59c640 1 -- 172.21.15.83:0/4181359314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59a00c7d70 msgr2=0x7f59a01ef5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f599e59c640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59a00c7d70 0x7f59a01ef5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f599e59c640 1 -- 172.21.15.83:0/4181359314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a01eeb60 con 0x7f59a0072d00 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f599e59c640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a0072d00 0x7f59a00c34b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f599000d800 tx=0x7f599000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f5996ffd640 1 -- 172.21.15.83:0/4181359314 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599000b6c0 con 0x7f59a0072d00 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f5996ffd640 1 -- 172.21.15.83:0/4181359314 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5990019070 con 0x7f59a0072d00 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 -- 172.21.15.83:0/4181359314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59a00c3a90 con 0x7f59a0072d00 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f5996ffd640 1 -- 172.21.15.83:0/4181359314 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599000ba60 con 0x7f59a0072d00 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.078+0000 7f59a48f7640 1 -- 172.21.15.83:0/4181359314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59a01eede0 con 0x7f59a0072d00 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.082+0000 7f59a48f7640 1 -- 172.21.15.83:0/4181359314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59a0157c40 con 0x7f59a0072d00 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.082+0000 7f5996ffd640 1 -- 172.21.15.83:0/4181359314 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f599000bc00 con 0x7f59a0072d00 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.082+0000 7f5996ffd640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59800678c0 0x7f5980069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.082+0000 7f5996ffd640 1 -- 172.21.15.83:0/4181359314 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f5990086680 con 0x7f59a0072d00 2024-09-19T07:23:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.082+0000 7f5997fff640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59800678c0 0x7f5980069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.082+0000 7f5997fff640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59800678c0 0x7f5980069d80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f59a00a80d0 tx=0x7f5988008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:12.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.086+0000 7f5996ffd640 1 -- 172.21.15.83:0/4181359314 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f59a01eeb60 con 0x7f59a0072d00 2024-09-19T07:23:12.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.298+0000 7f59a48f7640 1 -- 172.21.15.83:0/4181359314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59a00c2c40 con 0x7f59a0072d00 2024-09-19T07:23:12.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.298+0000 7f5996ffd640 1 -- 172.21.15.83:0/4181359314 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5714 (secure 0 0 0) 0x7f5990056c70 con 0x7f59a0072d00 2024-09-19T07:23:12.300 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:12.301 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":37,"btime":"2024-09-19T07:23:10:999297+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:12.301 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 37 2024-09-19T07:23:12.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.302+0000 7f59a48f7640 1 -- 172.21.15.83:0/4181359314 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59800678c0 msgr2=0x7f5980069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:12.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.302+0000 7f59a48f7640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59800678c0 0x7f5980069d80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f59a00a80d0 tx=0x7f5988008040 comp rx=0 tx=0).stop 2024-09-19T07:23:12.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.302+0000 7f59a48f7640 1 -- 172.21.15.83:0/4181359314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a0072d00 msgr2=0x7f59a00c34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:12.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.302+0000 7f59a48f7640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a0072d00 0x7f59a00c34b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f599000d800 tx=0x7f599000dcc0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.302+0000 7f59a48f7640 1 -- 172.21.15.83:0/4181359314 shutdown_connections 2024-09-19T07:23:12.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.302+0000 7f59a48f7640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59800678c0 0x7f5980069d80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.302+0000 7f59a48f7640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59a01547b0 0x7f59a014fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.302+0000 7f59a48f7640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59a00c7d70 0x7f59a01ef5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.302+0000 7f59a48f7640 1 --2- 172.21.15.83:0/4181359314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a0072d00 0x7f59a00c34b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.302+0000 7f59a48f7640 1 -- 172.21.15.83:0/4181359314 >> 172.21.15.83:0/4181359314 conn(0x7f59a00c5060 msgr2=0x7f59a006e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:12.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.302+0000 7f59a48f7640 1 -- 172.21.15.83:0/4181359314 shutdown_connections 2024-09-19T07:23:12.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.302+0000 7f59a48f7640 1 -- 172.21.15.83:0/4181359314 wait complete. 2024-09-19T07:23:12.323 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5025 2024-09-19T07:23:12.323 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.a 2024-09-19T07:23:12.323 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-19T07:23:12.324 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:23:12.324 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:23:12.326 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:23:12.326 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:12.392 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:23:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf8054640 1 Processor -- start 2024-09-19T07:23:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf8054640 1 -- start start 2024-09-19T07:23:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf8054640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3cf01494a0 0x7f3cf006e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf8054640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3cf0068700 0x7f3cf0153390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf8054640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf0155fa0 con 0x7f3cf0156730 2024-09-19T07:23:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf8054640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3cf00c3850 con 0x7f3cf01494a0 2024-09-19T07:23:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf8054640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf00c3b10 con 0x7f3cf0068700 2024-09-19T07:23:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf5df0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3cf01494a0 0x7f3cf006e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf65f1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3cf0156730 0x7f3cf006d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41526/0 (socket says 172.21.15.83:41526) 2024-09-19T07:23:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf65f1640 1 -- 172.21.15.83:0/2388689988 learned_addr learned my addr 172.21.15.83:0/2388689988 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf55ef640 1 --2- 172.21.15.83:0/2388689988 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3cf0068700 0x7f3cf0153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf5df0640 1 -- 172.21.15.83:0/2388689988 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3cf0068700 msgr2=0x7f3cf0153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf5df0640 1 --2- 172.21.15.83:0/2388689988 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3cf0068700 0x7f3cf0153390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf5df0640 1 -- 172.21.15.83:0/2388689988 >> v1:172.21.15.83:6789/0 conn(0x7f3cf0156730 legacy=0x7f3cf006d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.462+0000 7f3cf5df0640 1 -- 172.21.15.83:0/2388689988 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf00c3df0 con 0x7f3cf01494a0 2024-09-19T07:23:12.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.466+0000 7f3cf5df0640 1 --2- 172.21.15.83:0/2388689988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3cf01494a0 0x7f3cf006e010 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0097c0 tx=0x7f3cec01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791e95a9296ec3fb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:12.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.466+0000 7f3cf4dee640 1 -- 172.21.15.83:0/2388689988 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec0b1070 con 0x7f3cf01494a0 2024-09-19T07:23:12.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.466+0000 7f3cf4dee640 1 -- 172.21.15.83:0/2388689988 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cec0ac410 con 0x7f3cf01494a0 2024-09-19T07:23:12.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.466+0000 7f3cf4dee640 1 -- 172.21.15.83:0/2388689988 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec0ac7e0 con 0x7f3cf01494a0 2024-09-19T07:23:12.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.466+0000 7f3cf8054640 1 -- 172.21.15.83:0/2388689988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3cf01494a0 msgr2=0x7f3cf006e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:12.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.466+0000 7f3cf8054640 1 --2- 172.21.15.83:0/2388689988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3cf01494a0 0x7f3cf006e010 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0097c0 tx=0x7f3cec01c4a0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.466+0000 7f3cf8054640 1 -- 172.21.15.83:0/2388689988 shutdown_connections 2024-09-19T07:23:12.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.466+0000 7f3cf8054640 1 --2- 172.21.15.83:0/2388689988 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3cf0068700 0x7f3cf0153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.466+0000 7f3cf8054640 1 --2- 172.21.15.83:0/2388689988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3cf01494a0 0x7f3cf006e010 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.466+0000 7f3cf8054640 1 -- 172.21.15.83:0/2388689988 >> 172.21.15.83:0/2388689988 conn(0x7f3cf00c4940 msgr2=0x7f3cf00c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.466+0000 7f3cf8054640 1 -- 172.21.15.83:0/2388689988 shutdown_connections 2024-09-19T07:23:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.466+0000 7f3cf8054640 1 -- 172.21.15.83:0/2388689988 wait complete. 2024-09-19T07:23:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf8054640 1 Processor -- start 2024-09-19T07:23:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf8054640 1 -- start start 2024-09-19T07:23:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf8054640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3cf0068700 0x7f3cf01446c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf8054640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3cf01494a0 0x7f3cf0142500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf8054640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3cf0156730 0x7f3cf0142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf8054640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf015dc20 con 0x7f3cf01494a0 2024-09-19T07:23:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf8054640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf015daa0 con 0x7f3cf0068700 2024-09-19T07:23:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf8054640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf015dda0 con 0x7f3cf0156730 2024-09-19T07:23:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf55ef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3cf01494a0 0x7f3cf0142500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf55ef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3cf01494a0 0x7f3cf0142500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59194/0 (socket says 172.21.15.83:59194) 2024-09-19T07:23:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf55ef640 1 -- 172.21.15.83:0/3688416562 learned_addr learned my addr 172.21.15.83:0/3688416562 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf65f1640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3cf0156730 0x7f3cf0142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf5df0640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3cf0068700 0x7f3cf01446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf55ef640 1 -- 172.21.15.83:0/3688416562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3cf0156730 msgr2=0x7f3cf0142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf55ef640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3cf0156730 0x7f3cf0142a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf55ef640 1 -- 172.21.15.83:0/3688416562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3cf0068700 msgr2=0x7f3cf01446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf55ef640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3cf0068700 0x7f3cf01446c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf55ef640 1 -- 172.21.15.83:0/3688416562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf0154eb0 con 0x7f3cf01494a0 2024-09-19T07:23:12.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf55ef640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3cf01494a0 0x7f3cf0142500 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3ce000d800 tx=0x7f3ce000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:12.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3ce67fc640 1 -- 172.21.15.83:0/3688416562 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce000b6c0 con 0x7f3cf01494a0 2024-09-19T07:23:12.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3ce67fc640 1 -- 172.21.15.83:0/3688416562 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce0019070 con 0x7f3cf01494a0 2024-09-19T07:23:12.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3ce67fc640 1 -- 172.21.15.83:0/3688416562 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce000ba60 con 0x7f3cf01494a0 2024-09-19T07:23:12.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf8054640 1 -- 172.21.15.83:0/3688416562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf00ad1f0 con 0x7f3cf01494a0 2024-09-19T07:23:12.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf8054640 1 -- 172.21.15.83:0/3688416562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf01520d0 con 0x7f3cf01494a0 2024-09-19T07:23:12.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.470+0000 7f3cf8054640 1 -- 172.21.15.83:0/3688416562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf006e010 con 0x7f3cf01494a0 2024-09-19T07:23:12.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.474+0000 7f3ce67fc640 1 -- 172.21.15.83:0/3688416562 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3ce000bc00 con 0x7f3cf01494a0 2024-09-19T07:23:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.474+0000 7f3ce67fc640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3cd40678c0 0x7f3cd4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.474+0000 7f3ce67fc640 1 -- 172.21.15.83:0/3688416562 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f3ce00865c0 con 0x7f3cf01494a0 2024-09-19T07:23:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.474+0000 7f3cf5df0640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3cd40678c0 0x7f3cd4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.474+0000 7f3cf5df0640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3cd40678c0 0x7f3cd4069d80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0ae000 tx=0x7f3cec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.478+0000 7f3ce67fc640 1 -- 172.21.15.83:0/3688416562 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3ce0056c40 con 0x7f3cf01494a0 2024-09-19T07:23:12.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 -- 172.21.15.83:0/3688416562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3cf00c21e0 con 0x7f3cf01494a0 2024-09-19T07:23:12.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3ce67fc640 1 -- 172.21.15.83:0/3688416562 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5714 (secure 0 0 0) 0x7f3ce0056540 con 0x7f3cf01494a0 2024-09-19T07:23:12.703 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:12.703 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":37,"btime":"2024-09-19T07:23:10:999297+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:12.703 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 37 2024-09-19T07:23:12.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 -- 172.21.15.83:0/3688416562 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3cd40678c0 msgr2=0x7f3cd4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:12.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3cd40678c0 0x7f3cd4069d80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0ae000 tx=0x7f3cec008040 comp rx=0 tx=0).stop 2024-09-19T07:23:12.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 -- 172.21.15.83:0/3688416562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3cf01494a0 msgr2=0x7f3cf0142500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:12.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3cf01494a0 0x7f3cf0142500 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3ce000d800 tx=0x7f3ce000dcc0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 -- 172.21.15.83:0/3688416562 shutdown_connections 2024-09-19T07:23:12.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3cd40678c0 0x7f3cd4069d80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3cf0156730 0x7f3cf0142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3cf01494a0 0x7f3cf0142500 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 --2- 172.21.15.83:0/3688416562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3cf0068700 0x7f3cf01446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:12.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 -- 172.21.15.83:0/3688416562 >> 172.21.15.83:0/3688416562 conn(0x7f3cf00c4940 msgr2=0x7f3cf00c4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:12.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 -- 172.21.15.83:0/3688416562 shutdown_connections 2024-09-19T07:23:12.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:12.702+0000 7f3cf8054640 1 -- 172.21.15.83:0/3688416562 wait complete. 2024-09-19T07:23:12.721 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-19T07:23:13.243 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:23:13.242+0000 7f2e7abd5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:23:14.722 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:14.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5c2aaf640 1 Processor -- start 2024-09-19T07:23:14.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5c2aaf640 1 -- start start 2024-09-19T07:23:14.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5c2aaf640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb5bc0c3a20 0x7fb5bc152e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:14.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5c2aaf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb5bc072890 0x7fb5bc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:14.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5c2aaf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5bc073980 con 0x7fb5bc0723d0 2024-09-19T07:23:14.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5c2aaf640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb5bc0716a0 con 0x7fb5bc072890 2024-09-19T07:23:14.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5c2aaf640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fb5bc072d70 con 0x7fb5bc0c3a20 2024-09-19T07:23:14.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5bbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb5bc072890 0x7fb5bc06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:14.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5c104c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb5bc0723d0 0x7fb5bc148920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41536/0 (socket says 172.21.15.83:41536) 2024-09-19T07:23:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5c104c640 1 -- 172.21.15.83:0/3671749149 learned_addr learned my addr 172.21.15.83:0/3671749149 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5bb7fe640 1 -- 172.21.15.83:0/3671749149 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4264256984 0 0) 0x7fb5bc073980 con 0x7fb5bc0723d0 2024-09-19T07:23:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5bb7fe640 1 -- 172.21.15.83:0/3671749149 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5ac003530 con 0x7fb5bc0723d0 2024-09-19T07:23:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5bbfff640 1 -- 172.21.15.83:0/3671749149 >> v2:172.21.15.184:3300/0 conn(0x7fb5bc0c3a20 msgr2=0x7fb5bc152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5bbfff640 1 --2- 172.21.15.83:0/3671749149 >> v2:172.21.15.184:3300/0 conn(0x7fb5bc0c3a20 0x7fb5bc152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5bbfff640 1 -- 172.21.15.83:0/3671749149 >> v1:172.21.15.83:6789/0 conn(0x7fb5bc0723d0 legacy=0x7fb5bc148920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.858+0000 7fb5bbfff640 1 -- 172.21.15.83:0/3671749149 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5bc0718f0 con 0x7fb5bc072890 2024-09-19T07:23:14.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5bbfff640 1 --2- 172.21.15.83:0/3671749149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5bc072890 0x7fb5bc06d3e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc071820 tx=0x7fb5a801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae37fdf483264612 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:14.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5bb7fe640 1 -- 172.21.15.83:0/3671749149 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a80b0070 con 0x7fb5bc072890 2024-09-19T07:23:14.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5bb7fe640 1 -- 172.21.15.83:0/3671749149 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a8004230 con 0x7fb5bc072890 2024-09-19T07:23:14.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5bb7fe640 1 -- 172.21.15.83:0/3671749149 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8003620 con 0x7fb5bc072890 2024-09-19T07:23:14.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/3671749149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5bc072890 msgr2=0x7fb5bc06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:14.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 --2- 172.21.15.83:0/3671749149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5bc072890 0x7fb5bc06d3e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc071820 tx=0x7fb5a801c460 comp rx=0 tx=0).stop 2024-09-19T07:23:14.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/3671749149 shutdown_connections 2024-09-19T07:23:14.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 --2- 172.21.15.83:0/3671749149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5bc072890 0x7fb5bc06d3e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:14.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 --2- 172.21.15.83:0/3671749149 >> v2:172.21.15.184:3300/0 conn(0x7fb5bc0c3a20 0x7fb5bc152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:14.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/3671749149 >> 172.21.15.83:0/3671749149 conn(0x7fb5bc0c48b0 msgr2=0x7fb5bc0c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:14.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/3671749149 shutdown_connections 2024-09-19T07:23:14.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/3671749149 wait complete. 2024-09-19T07:23:14.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 Processor -- start 2024-09-19T07:23:14.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 -- start start 2024-09-19T07:23:14.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5bc0723d0 0x7fb5bc1ef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5bc072890 0x7fb5bc156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5bc0c3a20 0x7fb5bc145530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc15cd50 con 0x7fb5bc072890 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc0c4010 con 0x7fb5bc0723d0 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.862+0000 7fb5c2aaf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb5bc15ced0 con 0x7fb5bc0c3a20 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5bc072890 0x7fb5bc156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5bc072890 0x7fb5bc156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59204/0 (socket says 172.21.15.83:59204) 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5bbfff640 1 -- 172.21.15.83:0/689504181 learned_addr learned my addr 172.21.15.83:0/689504181 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5c084b640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5bc0723d0 0x7fb5bc1ef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5c104c640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5bc0c3a20 0x7fb5bc145530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5bbfff640 1 -- 172.21.15.83:0/689504181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5bc0c3a20 msgr2=0x7fb5bc145530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5bbfff640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5bc0c3a20 0x7fb5bc145530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:14.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5bbfff640 1 -- 172.21.15.83:0/689504181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5bc0723d0 msgr2=0x7fb5bc1ef2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:14.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5bbfff640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5bc0723d0 0x7fb5bc1ef2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:14.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5bbfff640 1 -- 172.21.15.83:0/689504181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5bc0c1310 con 0x7fb5bc072890 2024-09-19T07:23:14.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5c104c640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5bc0c3a20 0x7fb5bc145530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:14.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5bbfff640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5bc072890 0x7fb5bc156040 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80ae040 tx=0x7fb5a8019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:14.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5b97fa640 1 -- 172.21.15.83:0/689504181 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a80b9020 con 0x7fb5bc072890 2024-09-19T07:23:14.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5b97fa640 1 -- 172.21.15.83:0/689504181 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a8003df0 con 0x7fb5bc072890 2024-09-19T07:23:14.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/689504181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5bc0c15f0 con 0x7fb5bc072890 2024-09-19T07:23:14.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5b97fa640 1 -- 172.21.15.83:0/689504181 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a80b0040 con 0x7fb5bc072890 2024-09-19T07:23:14.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/689504181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5bc154be0 con 0x7fb5bc072890 2024-09-19T07:23:14.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/689504181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5bc06d3e0 con 0x7fb5bc072890 2024-09-19T07:23:14.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5b97fa640 1 -- 172.21.15.83:0/689504181 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb5a80bc020 con 0x7fb5bc072890 2024-09-19T07:23:14.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5b97fa640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5a00678c0 0x7fb5a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:14.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5b97fa640 1 -- 172.21.15.83:0/689504181 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7fb5a8120140 con 0x7fb5bc072890 2024-09-19T07:23:14.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5c084b640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5a00678c0 0x7fb5a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:14.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.866+0000 7fb5c084b640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5a00678c0 0x7fb5a0069d80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0006fd0 tx=0x7fb5b0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:14.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:14.874+0000 7fb5b97fa640 1 -- 172.21.15.83:0/689504181 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb5a80c3030 con 0x7fb5bc072890 2024-09-19T07:23:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/689504181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5bc0c4330 con 0x7fb5bc072890 2024-09-19T07:23:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5b97fa640 1 -- 172.21.15.83:0/689504181 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6539 (secure 0 0 0) 0x7fb5a8016050 con 0x7fb5bc072890 2024-09-19T07:23:15.115 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:15.115 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":38,"btime":"2024-09-19T07:23:13:002854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5304,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 38 2024-09-19T07:23:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/689504181 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5a00678c0 msgr2=0x7fb5a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5a00678c0 0x7fb5a0069d80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0006fd0 tx=0x7fb5b0008040 comp rx=0 tx=0).stop 2024-09-19T07:23:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/689504181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5bc072890 msgr2=0x7fb5bc156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5bc072890 0x7fb5bc156040 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80ae040 tx=0x7fb5a8019200 comp rx=0 tx=0).stop 2024-09-19T07:23:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/689504181 shutdown_connections 2024-09-19T07:23:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5a00678c0 0x7fb5a0069d80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5bc0c3a20 0x7fb5bc145530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5bc072890 0x7fb5bc156040 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 --2- 172.21.15.83:0/689504181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5bc0723d0 0x7fb5bc1ef2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/689504181 >> 172.21.15.83:0/689504181 conn(0x7fb5bc0c48b0 msgr2=0x7fb5bc146d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/689504181 shutdown_connections 2024-09-19T07:23:15.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:15.114+0000 7fb5c2aaf640 1 -- 172.21.15.83:0/689504181 wait complete. 2024-09-19T07:23:15.132 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-19T07:23:15.132 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before thrashing 2024-09-19T07:23:18.133 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:18.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.266+0000 7f4feb388640 1 Processor -- start 2024-09-19T07:23:18.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.266+0000 7f4feb388640 1 -- start start 2024-09-19T07:23:18.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.266+0000 7f4feb388640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4fe4151310 0x7f4fe406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.266+0000 7f4feb388640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4fe4074f90 0x7f4fe40753c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.266+0000 7f4feb388640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fe4157fc0 con 0x7f4fe406bc80 2024-09-19T07:23:18.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.266+0000 7f4feb388640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f4fe406d5a0 con 0x7f4fe4074f90 2024-09-19T07:23:18.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.266+0000 7f4feb388640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4fe4157b30 con 0x7f4fe4151310 2024-09-19T07:23:18.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.266+0000 7f4fe9925640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4fe406bc80 0x7f4fe406c0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41538/0 (socket says 172.21.15.83:41538) 2024-09-19T07:23:18.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.266+0000 7f4fe8923640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4fe4074f90 0x7f4fe40753c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.270+0000 7f4fe9925640 1 -- 172.21.15.83:0/3691494032 learned_addr learned my addr 172.21.15.83:0/3691494032 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:18.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.270+0000 7f4fe9124640 1 --2- 172.21.15.83:0/3691494032 >> v2:172.21.15.184:3300/0 conn(0x7f4fe4151310 0x7f4fe406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.270+0000 7f4fe8923640 1 -- 172.21.15.83:0/3691494032 >> v2:172.21.15.184:3300/0 conn(0x7f4fe4151310 msgr2=0x7f4fe406ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:18.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.270+0000 7f4fe8923640 1 --2- 172.21.15.83:0/3691494032 >> v2:172.21.15.184:3300/0 conn(0x7f4fe4151310 0x7f4fe406ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.270+0000 7f4fe8923640 1 -- 172.21.15.83:0/3691494032 >> v1:172.21.15.83:6789/0 conn(0x7f4fe406bc80 legacy=0x7f4fe406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:18.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.270+0000 7f4fdbfff640 1 -- 172.21.15.83:0/3691494032 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3151247743 0 0) 0x7f4fe4157fc0 con 0x7f4fe406bc80 2024-09-19T07:23:18.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.270+0000 7f4fe8923640 1 -- 172.21.15.83:0/3691494032 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fe4157200 con 0x7f4fe4074f90 2024-09-19T07:23:18.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.270+0000 7f4fe8923640 1 --2- 172.21.15.83:0/3691494032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fe4074f90 0x7f4fe40753c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40adc40 tx=0x7f4fd401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b33a5928aa893df server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:18.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.270+0000 7f4fdbfff640 1 -- 172.21.15.83:0/3691494032 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd40ab5f0 con 0x7f4fe4074f90 2024-09-19T07:23:18.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.270+0000 7f4fdbfff640 1 -- 172.21.15.83:0/3691494032 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd40b0070 con 0x7f4fe4074f90 2024-09-19T07:23:18.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.270+0000 7f4fdbfff640 1 -- 172.21.15.83:0/3691494032 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd40ab9b0 con 0x7f4fe4074f90 2024-09-19T07:23:18.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 -- 172.21.15.83:0/3691494032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fe4074f90 msgr2=0x7f4fe40753c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:18.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 --2- 172.21.15.83:0/3691494032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fe4074f90 0x7f4fe40753c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40adc40 tx=0x7f4fd401c3f0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 -- 172.21.15.83:0/3691494032 shutdown_connections 2024-09-19T07:23:18.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 --2- 172.21.15.83:0/3691494032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fe4074f90 0x7f4fe40753c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 --2- 172.21.15.83:0/3691494032 >> v2:172.21.15.184:3300/0 conn(0x7f4fe4151310 0x7f4fe406ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 -- 172.21.15.83:0/3691494032 >> 172.21.15.83:0/3691494032 conn(0x7f4fe40c90e0 msgr2=0x7f4fe40aa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:18.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 -- 172.21.15.83:0/3691494032 shutdown_connections 2024-09-19T07:23:18.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 -- 172.21.15.83:0/3691494032 wait complete. 2024-09-19T07:23:18.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 Processor -- start 2024-09-19T07:23:18.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 -- start start 2024-09-19T07:23:18.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fe406bc80 0x7f4fe41457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fe4074f90 0x7f4fe40c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fe4151310 0x7f4fe40c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe40c3c70 con 0x7f4fe406bc80 2024-09-19T07:23:18.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe40c3af0 con 0x7f4fe4151310 2024-09-19T07:23:18.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4feb388640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4fe406d130 con 0x7f4fe4074f90 2024-09-19T07:23:18.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fe9124640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fe406bc80 0x7f4fe41457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fe9124640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fe406bc80 0x7f4fe41457a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59230/0 (socket says 172.21.15.83:59230) 2024-09-19T07:23:18.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fe9124640 1 -- 172.21.15.83:0/731684506 learned_addr learned my addr 172.21.15.83:0/731684506 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:18.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fe9925640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fe4151310 0x7f4fe40c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fe8923640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fe4074f90 0x7f4fe40c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fe9124640 1 -- 172.21.15.83:0/731684506 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fe4074f90 msgr2=0x7f4fe40c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:18.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fe9124640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fe4074f90 0x7f4fe40c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fe9124640 1 -- 172.21.15.83:0/731684506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fe4151310 msgr2=0x7f4fe40c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:18.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fe9124640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fe4151310 0x7f4fe40c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fe9124640 1 -- 172.21.15.83:0/731684506 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fe41ece30 con 0x7f4fe406bc80 2024-09-19T07:23:18.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fe9925640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fe4151310 0x7f4fe40c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:23:18.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fe9124640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fe406bc80 0x7f4fe41457a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4fe000e870 tx=0x7f4fe000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:18.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fd9ffb640 1 -- 172.21.15.83:0/731684506 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0017070 con 0x7f4fe406bc80 2024-09-19T07:23:18.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fd9ffb640 1 -- 172.21.15.83:0/731684506 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe000c940 con 0x7f4fe406bc80 2024-09-19T07:23:18.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.274+0000 7f4fd9ffb640 1 -- 172.21.15.83:0/731684506 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe000ccf0 con 0x7f4fe406bc80 2024-09-19T07:23:18.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.278+0000 7f4feb388640 1 -- 172.21.15.83:0/731684506 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fe40c72c0 con 0x7f4fe406bc80 2024-09-19T07:23:18.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.278+0000 7f4feb388640 1 -- 172.21.15.83:0/731684506 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fe41ec350 con 0x7f4fe406bc80 2024-09-19T07:23:18.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.278+0000 7f4feb388640 1 -- 172.21.15.83:0/731684506 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fe406ed70 con 0x7f4fe406bc80 2024-09-19T07:23:18.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.278+0000 7f4fd9ffb640 1 -- 172.21.15.83:0/731684506 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4fe001a070 con 0x7f4fe406bc80 2024-09-19T07:23:18.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.278+0000 7f4fd9ffb640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4fc80678c0 0x7f4fc8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.278+0000 7f4fd9ffb640 1 -- 172.21.15.83:0/731684506 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f4fe0085f90 con 0x7f4fe406bc80 2024-09-19T07:23:18.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.278+0000 7f4fe8923640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4fc80678c0 0x7f4fc8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.278+0000 7f4fe8923640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4fc80678c0 0x7f4fc8069d80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4fd40af000 tx=0x7f4fd40a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:18.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.282+0000 7f4fd9ffb640 1 -- 172.21.15.83:0/731684506 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4fe0055fc0 con 0x7f4fe406bc80 2024-09-19T07:23:18.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.526+0000 7f4feb388640 1 -- 172.21.15.83:0/731684506 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fe40742b0 con 0x7f4fe406bc80 2024-09-19T07:23:18.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.526+0000 7f4fd9ffb640 1 -- 172.21.15.83:0/731684506 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6539 (secure 0 0 0) 0x7f4fe005a2c0 con 0x7f4fe406bc80 2024-09-19T07:23:18.529 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:18.529 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":38,"btime":"2024-09-19T07:23:13:002854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5304,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:18.530 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 38 2024-09-19T07:23:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.530+0000 7f4feb388640 1 -- 172.21.15.83:0/731684506 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4fc80678c0 msgr2=0x7f4fc8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.530+0000 7f4feb388640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4fc80678c0 0x7f4fc8069d80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4fd40af000 tx=0x7f4fd40a7000 comp rx=0 tx=0).stop 2024-09-19T07:23:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.530+0000 7f4feb388640 1 -- 172.21.15.83:0/731684506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fe406bc80 msgr2=0x7f4fe41457a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.530+0000 7f4feb388640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fe406bc80 0x7f4fe41457a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4fe000e870 tx=0x7f4fe000ec70 comp rx=0 tx=0).stop 2024-09-19T07:23:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.530+0000 7f4feb388640 1 -- 172.21.15.83:0/731684506 shutdown_connections 2024-09-19T07:23:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.530+0000 7f4feb388640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4fc80678c0 0x7f4fc8069d80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.530+0000 7f4feb388640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fe4151310 0x7f4fe40c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.530+0000 7f4feb388640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fe4074f90 0x7f4fe40c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.530+0000 7f4feb388640 1 --2- 172.21.15.83:0/731684506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fe406bc80 0x7f4fe41457a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.530+0000 7f4feb388640 1 -- 172.21.15.83:0/731684506 >> 172.21.15.83:0/731684506 conn(0x7f4fe40c90e0 msgr2=0x7f4fe414e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:18.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.530+0000 7f4feb388640 1 -- 172.21.15.83:0/731684506 shutdown_connections 2024-09-19T07:23:18.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.530+0000 7f4feb388640 1 -- 172.21.15.83:0/731684506 wait complete. 2024-09-19T07:23:18.552 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-19T07:23:18.553 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:23:18.553 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:23:18.565 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:23:18.565 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:18.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b3b913640 1 Processor -- start 2024-09-19T07:23:18.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b3b913640 1 -- start start 2024-09-19T07:23:18.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b3b913640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6b34155d70 0x7f6b340c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b3b913640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b34156200 con 0x7f6b341592f0 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b3b913640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b341564e0 con 0x7f6b340c6120 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b3b913640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6b340c9930 con 0x7f6b34155d70 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b39eb0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6b341592f0 0x7f6b34157ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41544/0 (socket says 172.21.15.83:41544) 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b39eb0640 1 -- 172.21.15.83:0/566320258 learned_addr learned my addr 172.21.15.83:0/566320258 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b38eae640 1 --2- 172.21.15.83:0/566320258 >> v2:172.21.15.184:3300/0 conn(0x7f6b34155d70 0x7f6b340c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2259600209 0 0) 0x7f6b34156200 con 0x7f6b341592f0 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b20003530 con 0x7f6b341592f0 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 926758687 0 0) 0x7f6b20003530 con 0x7f6b341592f0 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 >> v2:172.21.15.184:3300/0 conn(0x7f6b34155d70 msgr2=0x7f6b340c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 --2- 172.21.15.83:0/566320258 >> v2:172.21.15.184:3300/0 conn(0x7f6b34155d70 0x7f6b340c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 >> v1:172.21.15.184:6789/0 conn(0x7f6b340c6120 legacy=0x7f6b341575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b341567c0 con 0x7f6b341592f0 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f6b2c001a00 con 0x7f6b341592f0 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 >> v1:172.21.15.83:6789/0 conn(0x7f6b341592f0 legacy=0x7f6b34157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 --2- 172.21.15.83:0/566320258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b200039c0 0x7f6b20003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 --2- 172.21.15.83:0/566320258 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b20004810 0x7f6b20004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 --2- 172.21.15.83:0/566320258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b200051d0 0x7f6b2000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2000c140 con 0x7f6b200039c0 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2000c2f0 con 0x7f6b200051d0 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6b2000c4f0 con 0x7f6b20004810 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b396af640 1 --2- 172.21.15.83:0/566320258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b200039c0 0x7f6b20003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b39eb0640 1 --2- 172.21.15.83:0/566320258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b200051d0 0x7f6b2000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b396af640 1 -- 172.21.15.83:0/566320258 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b20004810 msgr2=0x7f6b20004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b396af640 1 --2- 172.21.15.83:0/566320258 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b20004810 0x7f6b20004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b396af640 1 -- 172.21.15.83:0/566320258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b200051d0 msgr2=0x7f6b2000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b396af640 1 --2- 172.21.15.83:0/566320258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b200051d0 0x7f6b2000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b396af640 1 -- 172.21.15.83:0/566320258 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b20003530 con 0x7f6b200039c0 2024-09-19T07:23:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b39eb0640 1 --2- 172.21.15.83:0/566320258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b200051d0 0x7f6b2000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.738+0000 7f6b39eb0640 1 -- 172.21.15.83:0/566320258 reap_dead start 2024-09-19T07:23:18.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b396af640 1 --2- 172.21.15.83:0/566320258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b200039c0 0x7f6b20003de0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6b30001780 tx=0x7f6b3001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=662b70cac347eb14 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:18.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b300013a0 con 0x7f6b200039c0 2024-09-19T07:23:18.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b300052f0 con 0x7f6b200039c0 2024-09-19T07:23:18.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b2bfff640 1 -- 172.21.15.83:0/566320258 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b30003840 con 0x7f6b200039c0 2024-09-19T07:23:18.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b3b913640 1 -- 172.21.15.83:0/566320258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b200039c0 msgr2=0x7f6b20003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:18.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b3b913640 1 --2- 172.21.15.83:0/566320258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b200039c0 0x7f6b20003de0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6b30001780 tx=0x7f6b3001c400 comp rx=0 tx=0).stop 2024-09-19T07:23:18.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b3b913640 1 -- 172.21.15.83:0/566320258 shutdown_connections 2024-09-19T07:23:18.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b3b913640 1 --2- 172.21.15.83:0/566320258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b200039c0 0x7f6b20003de0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b3b913640 1 -- 172.21.15.83:0/566320258 >> 172.21.15.83:0/566320258 conn(0x7f6b3406bc80 msgr2=0x7f6b3406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:18.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b3b913640 1 -- 172.21.15.83:0/566320258 shutdown_connections 2024-09-19T07:23:18.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b3b913640 1 -- 172.21.15.83:0/566320258 wait complete. 2024-09-19T07:23:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b3b913640 1 Processor -- start 2024-09-19T07:23:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.742+0000 7f6b3b913640 1 -- start start 2024-09-19T07:23:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b3b913640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b200039c0 0x7f6b341ef980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b3b913640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b340ad080 0x7f6b34151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b3b913640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b340ad4e0 0x7f6b341f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b3b913640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6b34153860 con 0x7f6b340ad4e0 2024-09-19T07:23:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b3b913640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6b34157de0 con 0x7f6b340ad080 2024-09-19T07:23:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b3b913640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6b34154630 con 0x7f6b200039c0 2024-09-19T07:23:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b39eb0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b340ad4e0 0x7f6b341f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b39eb0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b340ad4e0 0x7f6b341f28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59260/0 (socket says 172.21.15.83:59260) 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b39eb0640 1 -- 172.21.15.83:0/930268091 learned_addr learned my addr 172.21.15.83:0/930268091 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b396af640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b200039c0 0x7f6b341ef980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b38eae640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b340ad080 0x7f6b34151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b39eb0640 1 -- 172.21.15.83:0/930268091 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b200039c0 msgr2=0x7f6b341ef980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b39eb0640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b200039c0 0x7f6b341ef980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b39eb0640 1 -- 172.21.15.83:0/930268091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b340ad080 msgr2=0x7f6b34151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b39eb0640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b340ad080 0x7f6b34151b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b39eb0640 1 -- 172.21.15.83:0/930268091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b341f25d0 con 0x7f6b340ad4e0 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b396af640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b200039c0 0x7f6b341ef980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b39eb0640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b340ad4e0 0x7f6b341f28b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6b3408d440 tx=0x7f6b2c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b29ffb640 1 -- 172.21.15.83:0/930268091 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c004d70 con 0x7f6b340ad4e0 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b29ffb640 1 -- 172.21.15.83:0/930268091 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b2c00bb80 con 0x7f6b340ad4e0 2024-09-19T07:23:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b3b913640 1 -- 172.21.15.83:0/930268091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b341ef180 con 0x7f6b340ad4e0 2024-09-19T07:23:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b29ffb640 1 -- 172.21.15.83:0/930268091 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c007070 con 0x7f6b340ad4e0 2024-09-19T07:23:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b3b913640 1 -- 172.21.15.83:0/930268091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b3414d110 con 0x7f6b340ad4e0 2024-09-19T07:23:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b3b913640 1 -- 172.21.15.83:0/930268091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b34158110 con 0x7f6b340ad4e0 2024-09-19T07:23:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.746+0000 7f6b29ffb640 1 -- 172.21.15.83:0/930268091 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6b2c01b020 con 0x7f6b340ad4e0 2024-09-19T07:23:18.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.750+0000 7f6b29ffb640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6b180678f0 0x7f6b18069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:18.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.750+0000 7f6b29ffb640 1 -- 172.21.15.83:0/930268091 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f6b2c086ab0 con 0x7f6b340ad4e0 2024-09-19T07:23:18.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.750+0000 7f6b396af640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6b180678f0 0x7f6b18069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:18.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.750+0000 7f6b396af640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6b180678f0 0x7f6b18069db0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6b30001750 tx=0x7f6b300011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:18.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.754+0000 7f6b29ffb640 1 -- 172.21.15.83:0/930268091 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6b2c022070 con 0x7f6b340ad4e0 2024-09-19T07:23:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.994+0000 7f6b3b913640 1 -- 172.21.15.83:0/930268091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b340c96a0 con 0x7f6b340ad4e0 2024-09-19T07:23:18.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b29ffb640 1 -- 172.21.15.83:0/930268091 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6539 (secure 0 0 0) 0x7f6b2c0570f0 con 0x7f6b340ad4e0 2024-09-19T07:23:18.998 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:18.998 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":38,"btime":"2024-09-19T07:23:13:002854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5304,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:18.998 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 38 2024-09-19T07:23:19.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b3b913640 1 -- 172.21.15.83:0/930268091 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6b180678f0 msgr2=0x7f6b18069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:19.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b3b913640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6b180678f0 0x7f6b18069db0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6b30001750 tx=0x7f6b300011a0 comp rx=0 tx=0).stop 2024-09-19T07:23:19.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b3b913640 1 -- 172.21.15.83:0/930268091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b340ad4e0 msgr2=0x7f6b341f28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:19.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b3b913640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b340ad4e0 0x7f6b341f28b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6b3408d440 tx=0x7f6b2c00b4a0 comp rx=0 tx=0).stop 2024-09-19T07:23:19.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b3b913640 1 -- 172.21.15.83:0/930268091 shutdown_connections 2024-09-19T07:23:19.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b3b913640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6b180678f0 0x7f6b18069db0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:19.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b3b913640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b340ad4e0 0x7f6b341f28b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:19.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b3b913640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b340ad080 0x7f6b34151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:19.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b3b913640 1 --2- 172.21.15.83:0/930268091 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b200039c0 0x7f6b341ef980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:19.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b3b913640 1 -- 172.21.15.83:0/930268091 >> 172.21.15.83:0/930268091 conn(0x7f6b3406bc80 msgr2=0x7f6b341549c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:19.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b3b913640 1 -- 172.21.15.83:0/930268091 shutdown_connections 2024-09-19T07:23:19.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:18.998+0000 7f6b3b913640 1 -- 172.21.15.83:0/930268091 wait complete. 2024-09-19T07:23:19.018 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:23:21.020 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:21.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc69e174640 1 Processor -- start 2024-09-19T07:23:21.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc69e174640 1 -- start start 2024-09-19T07:23:21.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc69e174640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc69814dbe0 0x7fc6980c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:21.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc69e174640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6980c3ab0 0x7fc698157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:21.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc69e174640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc698073d90 con 0x7fc6980c44e0 2024-09-19T07:23:21.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc69e174640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc698154700 con 0x7fc69814dbe0 2024-09-19T07:23:21.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc69e174640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc698153b30 con 0x7fc6980c3ab0 2024-09-19T07:23:21.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc696ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6980c3ab0 0x7fc698157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:21.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc696ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6980c3ab0 0x7fc698157010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:41842/0 (socket says 172.21.15.83:41842) 2024-09-19T07:23:21.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc696ffd640 1 -- 172.21.15.83:0/812959218 learned_addr learned my addr 172.21.15.83:0/812959218 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:21.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc6977fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc69814dbe0 0x7fc6980c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:21.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc696ffd640 1 -- 172.21.15.83:0/812959218 >> v2:172.21.15.184:3300/0 conn(0x7fc69814dbe0 msgr2=0x7fc6980c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:21.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc696ffd640 1 --2- 172.21.15.83:0/812959218 >> v2:172.21.15.184:3300/0 conn(0x7fc69814dbe0 0x7fc6980c3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:21.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc696ffd640 1 -- 172.21.15.83:0/812959218 >> v1:172.21.15.184:6789/0 conn(0x7fc6980c44e0 legacy=0x7fc698157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:21.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.154+0000 7fc696ffd640 1 -- 172.21.15.83:0/812959218 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc698153dd0 con 0x7fc6980c3ab0 2024-09-19T07:23:21.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.158+0000 7fc696ffd640 1 --2- 172.21.15.83:0/812959218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6980c3ab0 0x7fc698157010 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc680009870 tx=0x7fc68001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca98789395c9b570 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:21.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.158+0000 7fc6967fc640 1 -- 172.21.15.83:0/812959218 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6800b1070 con 0x7fc6980c3ab0 2024-09-19T07:23:21.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.158+0000 7fc6967fc640 1 -- 172.21.15.83:0/812959218 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6800a8050 con 0x7fc6980c3ab0 2024-09-19T07:23:21.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.158+0000 7fc6967fc640 1 -- 172.21.15.83:0/812959218 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6800ac860 con 0x7fc6980c3ab0 2024-09-19T07:23:21.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.158+0000 7fc69e174640 1 -- 172.21.15.83:0/812959218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6980c3ab0 msgr2=0x7fc698157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:21.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.158+0000 7fc69e174640 1 --2- 172.21.15.83:0/812959218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6980c3ab0 0x7fc698157010 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc680009870 tx=0x7fc68001c4d0 comp rx=0 tx=0).stop 2024-09-19T07:23:21.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.158+0000 7fc69e174640 1 -- 172.21.15.83:0/812959218 shutdown_connections 2024-09-19T07:23:21.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.158+0000 7fc69e174640 1 --2- 172.21.15.83:0/812959218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6980c3ab0 0x7fc698157010 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:21.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.158+0000 7fc69e174640 1 --2- 172.21.15.83:0/812959218 >> v2:172.21.15.184:3300/0 conn(0x7fc69814dbe0 0x7fc6980c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:21.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.158+0000 7fc69e174640 1 -- 172.21.15.83:0/812959218 >> 172.21.15.83:0/812959218 conn(0x7fc6980c95a0 msgr2=0x7fc6980c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:21.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.158+0000 7fc69e174640 1 -- 172.21.15.83:0/812959218 shutdown_connections 2024-09-19T07:23:21.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.158+0000 7fc69e174640 1 -- 172.21.15.83:0/812959218 wait complete. 2024-09-19T07:23:21.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc69e174640 1 Processor -- start 2024-09-19T07:23:21.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc69e174640 1 -- start start 2024-09-19T07:23:21.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc69e174640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6980c44e0 0x7fc6980c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:21.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc69e174640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc69814dbe0 0x7fc69813f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:21.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc69e174640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc69813f790 0x7fc6981f2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:21.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc69e174640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc698160bc0 con 0x7fc69814dbe0 2024-09-19T07:23:21.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc69e174640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc6980741a0 con 0x7fc6980c44e0 2024-09-19T07:23:21.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc69e174640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc698160d40 con 0x7fc69813f790 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc696ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc69814dbe0 0x7fc69813f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc696ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc69814dbe0 0x7fc69813f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59618/0 (socket says 172.21.15.83:59618) 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc696ffd640 1 -- 172.21.15.83:0/3515608894 learned_addr learned my addr 172.21.15.83:0/3515608894 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc6977fe640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6980c44e0 0x7fc6980c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc697fff640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc69813f790 0x7fc6981f2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc696ffd640 1 -- 172.21.15.83:0/3515608894 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc69813f790 msgr2=0x7fc6981f2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc696ffd640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc69813f790 0x7fc6981f2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc696ffd640 1 -- 172.21.15.83:0/3515608894 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6980c44e0 msgr2=0x7fc6980c2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc696ffd640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6980c44e0 0x7fc6980c2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc696ffd640 1 -- 172.21.15.83:0/3515608894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69806d750 con 0x7fc69814dbe0 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc697fff640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc69813f790 0x7fc6981f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc696ffd640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc69814dbe0 0x7fc69813f250 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc680000c10 tx=0x7fc68001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:21.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc68ffff640 1 -- 172.21.15.83:0/3515608894 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6800b1040 con 0x7fc69814dbe0 2024-09-19T07:23:21.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc68ffff640 1 -- 172.21.15.83:0/3515608894 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6800bc070 con 0x7fc69814dbe0 2024-09-19T07:23:21.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc69e174640 1 -- 172.21.15.83:0/3515608894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc698152500 con 0x7fc69814dbe0 2024-09-19T07:23:21.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc68ffff640 1 -- 172.21.15.83:0/3515608894 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6800b7690 con 0x7fc69814dbe0 2024-09-19T07:23:21.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc69e174640 1 -- 172.21.15.83:0/3515608894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6980af680 con 0x7fc69814dbe0 2024-09-19T07:23:21.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc69e174640 1 -- 172.21.15.83:0/3515608894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6980c3fa0 con 0x7fc69814dbe0 2024-09-19T07:23:21.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.162+0000 7fc68ffff640 1 -- 172.21.15.83:0/3515608894 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc6800b7830 con 0x7fc69814dbe0 2024-09-19T07:23:21.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.166+0000 7fc68ffff640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6780678c0 0x7fc678069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:21.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.166+0000 7fc68ffff640 1 -- 172.21.15.83:0/3515608894 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7fc680120240 con 0x7fc69814dbe0 2024-09-19T07:23:21.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.166+0000 7fc6977fe640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6780678c0 0x7fc678069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:21.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.166+0000 7fc6977fe640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6780678c0 0x7fc678069d80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc688009af0 tx=0x7fc688008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:21.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.170+0000 7fc68ffff640 1 -- 172.21.15.83:0/3515608894 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc69806d750 con 0x7fc69814dbe0 2024-09-19T07:23:21.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.410+0000 7fc69e174640 1 -- 172.21.15.83:0/3515608894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6981f20a0 con 0x7fc69814dbe0 2024-09-19T07:23:21.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.410+0000 7fc68ffff640 1 -- 172.21.15.83:0/3515608894 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6539 (secure 0 0 0) 0x7fc680016150 con 0x7fc69814dbe0 2024-09-19T07:23:21.411 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:21.411 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":38,"btime":"2024-09-19T07:23:13:002854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5304,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:21.411 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 38 2024-09-19T07:23:21.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.410+0000 7fc69e174640 1 -- 172.21.15.83:0/3515608894 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6780678c0 msgr2=0x7fc678069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:21.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.410+0000 7fc69e174640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6780678c0 0x7fc678069d80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc688009af0 tx=0x7fc688008040 comp rx=0 tx=0).stop 2024-09-19T07:23:21.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.410+0000 7fc69e174640 1 -- 172.21.15.83:0/3515608894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc69814dbe0 msgr2=0x7fc69813f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:21.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.410+0000 7fc69e174640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc69814dbe0 0x7fc69813f250 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc680000c10 tx=0x7fc68001ad10 comp rx=0 tx=0).stop 2024-09-19T07:23:21.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.410+0000 7fc69e174640 1 -- 172.21.15.83:0/3515608894 shutdown_connections 2024-09-19T07:23:21.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.410+0000 7fc69e174640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6780678c0 0x7fc678069d80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:21.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.410+0000 7fc69e174640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc69813f790 0x7fc6981f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:21.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.410+0000 7fc69e174640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc69814dbe0 0x7fc69813f250 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:21.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.410+0000 7fc69e174640 1 --2- 172.21.15.83:0/3515608894 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6980c44e0 0x7fc6980c2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:21.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.414+0000 7fc69e174640 1 -- 172.21.15.83:0/3515608894 >> 172.21.15.83:0/3515608894 conn(0x7fc6980c95a0 msgr2=0x7fc69815d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:21.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.414+0000 7fc69e174640 1 -- 172.21.15.83:0/3515608894 shutdown_connections 2024-09-19T07:23:21.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:21.414+0000 7fc69e174640 1 -- 172.21.15.83:0/3515608894 wait complete. 2024-09-19T07:23:21.434 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:23:23.435 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:23.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f7396897640 1 Processor -- start 2024-09-19T07:23:23.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f7396897640 1 -- start start 2024-09-19T07:23:23.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f7396897640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7390150600 0x7f7390156220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:23.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f7396897640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7390068290 con 0x7f7390150220 2024-09-19T07:23:23.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f7396897640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7390068570 con 0x7f739006d3e0 2024-09-19T07:23:23.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f7396897640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7390152d60 con 0x7f7390150600 2024-09-19T07:23:23.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7390150220 0x7f73900ad2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59866/0 (socket says 172.21.15.83:59866) 2024-09-19T07:23:23.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738ffff640 1 -- 172.21.15.83:0/3136911608 learned_addr learned my addr 172.21.15.83:0/3136911608 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:23.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738f7fe640 1 --2- 172.21.15.83:0/3136911608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7390150600 0x7f7390156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:23.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845995634 0 0) 0x7f7390068290 con 0x7f7390150220 2024-09-19T07:23:23.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7378003530 con 0x7f7390150220 2024-09-19T07:23:23.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 785033396 0 0) 0x7f7378003530 con 0x7f7390150220 2024-09-19T07:23:23.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7390150600 msgr2=0x7f7390156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:23.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 --2- 172.21.15.83:0/3136911608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7390150600 0x7f7390156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:23.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 >> v1:172.21.15.184:6789/0 conn(0x7f739006d3e0 legacy=0x7f7390153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:23.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7390153040 con 0x7f7390150220 2024-09-19T07:23:23.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f7380001680 con 0x7f7390150220 2024-09-19T07:23:23.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 >> v1:172.21.15.83:6789/0 conn(0x7f7390150220 legacy=0x7f73900ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:23.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 --2- 172.21.15.83:0/3136911608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7390150600 0x7f73780039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:23.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 --2- 172.21.15.83:0/3136911608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7378003f70 0x7f7378004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:23.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 --2- 172.21.15.83:0/3136911608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7378004d50 0x7f73780051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:23.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7390152d60 con 0x7f7378003f70 2024-09-19T07:23:23.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f737800bc10 con 0x7f7378004d50 2024-09-19T07:23:23.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f737800be10 con 0x7f7390150600 2024-09-19T07:23:23.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738f7fe640 1 --2- 172.21.15.83:0/3136911608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7378003f70 0x7f7378004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:23.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738ffff640 1 --2- 172.21.15.83:0/3136911608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7390150600 0x7f73780039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:23.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738f7fe640 1 -- 172.21.15.83:0/3136911608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7390150600 msgr2=0x7f73780039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:23.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738f7fe640 1 --2- 172.21.15.83:0/3136911608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7390150600 0x7f73780039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:23.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738f7fe640 1 -- 172.21.15.83:0/3136911608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7378004d50 msgr2=0x7f73780051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:23.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738f7fe640 1 --2- 172.21.15.83:0/3136911608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7378004d50 0x7f73780051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:23.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.570+0000 7f738f7fe640 1 -- 172.21.15.83:0/3136911608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7378003530 con 0x7f7378003f70 2024-09-19T07:23:23.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f738f7fe640 1 --2- 172.21.15.83:0/3136911608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7378003f70 0x7f7378004810 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7384009ae0 tx=0x7f738401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da6f8169b3781731 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:23.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73840096b0 con 0x7f7378003f70 2024-09-19T07:23:23.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73840a64a0 con 0x7f7378003f70 2024-09-19T07:23:23.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f738effd640 1 -- 172.21.15.83:0/3136911608 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73840a6890 con 0x7f7378003f70 2024-09-19T07:23:23.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f7396897640 1 -- 172.21.15.83:0/3136911608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7378003f70 msgr2=0x7f7378004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:23.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f7396897640 1 --2- 172.21.15.83:0/3136911608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7378003f70 0x7f7378004810 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7384009ae0 tx=0x7f738401c9b0 comp rx=0 tx=0).stop 2024-09-19T07:23:23.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f7394e34640 1 -- 172.21.15.83:0/3136911608 reap_dead start 2024-09-19T07:23:23.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f7396897640 1 -- 172.21.15.83:0/3136911608 shutdown_connections 2024-09-19T07:23:23.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f7396897640 1 -- 172.21.15.83:0/3136911608 >> 172.21.15.83:0/3136911608 conn(0x7f73900c48b0 msgr2=0x7f73900c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:23.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f7396897640 1 -- 172.21.15.83:0/3136911608 shutdown_connections 2024-09-19T07:23:23.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f7396897640 1 -- 172.21.15.83:0/3136911608 wait complete. 2024-09-19T07:23:23.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f7396897640 1 Processor -- start 2024-09-19T07:23:23.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.574+0000 7f7396897640 1 -- start start 2024-09-19T07:23:23.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f7396897640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7378003f70 0x7f7390154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:23.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f7396897640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390148cb0 0x7f73901ed400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:23.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f7396897640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f739014f510 0x7f7390155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:23.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f7396897640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7390071a30 con 0x7f7390148cb0 2024-09-19T07:23:23.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f7396897640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f73900718b0 con 0x7f7378003f70 2024-09-19T07:23:23.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f7396897640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7390071bb0 con 0x7f739014f510 2024-09-19T07:23:23.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390148cb0 0x7f73901ed400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:23.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390148cb0 0x7f73901ed400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59628/0 (socket says 172.21.15.83:59628) 2024-09-19T07:23:23.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738f7fe640 1 -- 172.21.15.83:0/2036933824 learned_addr learned my addr 172.21.15.83:0/2036933824 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:23.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f7394e34640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f739014f510 0x7f7390155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:23.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738ffff640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7378003f70 0x7f7390154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:23.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738f7fe640 1 -- 172.21.15.83:0/2036933824 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f739014f510 msgr2=0x7f7390155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:23.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738f7fe640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f739014f510 0x7f7390155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:23.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738f7fe640 1 -- 172.21.15.83:0/2036933824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7378003f70 msgr2=0x7f7390154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:23.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738f7fe640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7378003f70 0x7f7390154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:23.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738f7fe640 1 -- 172.21.15.83:0/2036933824 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73901552f0 con 0x7f7390148cb0 2024-09-19T07:23:23.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738f7fe640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390148cb0 0x7f73901ed400 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7384009ae0 tx=0x7f7378005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:23.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738cff9640 1 -- 172.21.15.83:0/2036933824 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73840be070 con 0x7f7390148cb0 2024-09-19T07:23:23.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738cff9640 1 -- 172.21.15.83:0/2036933824 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73840a6cd0 con 0x7f7390148cb0 2024-09-19T07:23:23.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f7396897640 1 -- 172.21.15.83:0/2036933824 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73901ec940 con 0x7f7390148cb0 2024-09-19T07:23:23.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738cff9640 1 -- 172.21.15.83:0/2036933824 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73840a76a0 con 0x7f7390148cb0 2024-09-19T07:23:23.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f7396897640 1 -- 172.21.15.83:0/2036933824 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73901f0c40 con 0x7f7390148cb0 2024-09-19T07:23:23.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f7396897640 1 -- 172.21.15.83:0/2036933824 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73900ad2a0 con 0x7f7390148cb0 2024-09-19T07:23:23.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.578+0000 7f738cff9640 1 -- 172.21.15.83:0/2036933824 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f73840c1070 con 0x7f7390148cb0 2024-09-19T07:23:23.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.582+0000 7f738cff9640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f737c0678c0 0x7f737c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:23.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.582+0000 7f738ffff640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f737c0678c0 0x7f737c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:23.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.582+0000 7f738cff9640 1 -- 172.21.15.83:0/2036933824 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f738411fe20 con 0x7f7390148cb0 2024-09-19T07:23:23.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.582+0000 7f738ffff640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f737c0678c0 0x7f737c069d80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7380001000 tx=0x7f7380009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:23.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.586+0000 7f738cff9640 1 -- 172.21.15.83:0/2036933824 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f73840ac020 con 0x7f7390148cb0 2024-09-19T07:23:23.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.818+0000 7f7396897640 1 -- 172.21.15.83:0/2036933824 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73900c54d0 con 0x7f7390148cb0 2024-09-19T07:23:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.818+0000 7f738cff9640 1 -- 172.21.15.83:0/2036933824 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6539 (secure 0 0 0) 0x7f7384016780 con 0x7f7390148cb0 2024-09-19T07:23:23.820 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:23.820 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":38,"btime":"2024-09-19T07:23:13:002854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5304,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:23.820 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 38 2024-09-19T07:23:23.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.822+0000 7f7396897640 1 -- 172.21.15.83:0/2036933824 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f737c0678c0 msgr2=0x7f737c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.822+0000 7f7396897640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f737c0678c0 0x7f737c069d80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7380001000 tx=0x7f7380009040 comp rx=0 tx=0).stop 2024-09-19T07:23:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.822+0000 7f7396897640 1 -- 172.21.15.83:0/2036933824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390148cb0 msgr2=0x7f73901ed400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.822+0000 7f7396897640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390148cb0 0x7f73901ed400 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7384009ae0 tx=0x7f7378005840 comp rx=0 tx=0).stop 2024-09-19T07:23:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.822+0000 7f7396897640 1 -- 172.21.15.83:0/2036933824 shutdown_connections 2024-09-19T07:23:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.822+0000 7f7396897640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f737c0678c0 0x7f737c069d80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.822+0000 7f7396897640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f739014f510 0x7f7390155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:23.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.822+0000 7f7396897640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390148cb0 0x7f73901ed400 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:23.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.822+0000 7f7396897640 1 --2- 172.21.15.83:0/2036933824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7378003f70 0x7f7390154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:23.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.822+0000 7f7396897640 1 -- 172.21.15.83:0/2036933824 >> 172.21.15.83:0/2036933824 conn(0x7f73900c48b0 msgr2=0x7f73900721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:23.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.822+0000 7f7396897640 1 -- 172.21.15.83:0/2036933824 shutdown_connections 2024-09-19T07:23:23.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:23.822+0000 7f7396897640 1 -- 172.21.15.83:0/2036933824 wait complete. 2024-09-19T07:23:23.842 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:23:25.843 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:25.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.978+0000 7f1006380640 1 Processor -- start 2024-09-19T07:23:25.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.978+0000 7f1006380640 1 -- start start 2024-09-19T07:23:25.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f1006380640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10000c6120 0x7f10000c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:25.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f1006380640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f10001592f0 0x7f1000157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:25.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f1006380640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1000156200 con 0x7f1000155d70 2024-09-19T07:23:25.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f1006380640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f10000c9930 con 0x7f10001592f0 2024-09-19T07:23:25.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f1006380640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f10001564e0 con 0x7f10000c6120 2024-09-19T07:23:25.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f100491d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f10001592f0 0x7f1000157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:25.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f100491d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f10001592f0 0x7f1000157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59638/0 (socket says 172.21.15.83:59638) 2024-09-19T07:23:25.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f100491d640 1 -- 172.21.15.83:0/3654683474 learned_addr learned my addr 172.21.15.83:0/3654683474 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:25.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f0ffffff640 1 --2- 172.21.15.83:0/3654683474 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10000c6120 0x7f10000c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:25.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f100491d640 1 -- 172.21.15.83:0/3654683474 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10000c6120 msgr2=0x7f10000c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:25.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f100491d640 1 --2- 172.21.15.83:0/3654683474 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10000c6120 0x7f10000c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:25.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f100491d640 1 -- 172.21.15.83:0/3654683474 >> v1:172.21.15.184:6789/0 conn(0x7f1000155d70 legacy=0x7f10001575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:25.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f100491d640 1 -- 172.21.15.83:0/3654683474 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10001567c0 con 0x7f10001592f0 2024-09-19T07:23:25.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f100491d640 1 --2- 172.21.15.83:0/3654683474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10001592f0 0x7f1000157ce0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009870 tx=0x7f0ff801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ab7f3d25b2bc5a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:25.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f0ffeffd640 1 -- 172.21.15.83:0/3654683474 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff80b0070 con 0x7f10001592f0 2024-09-19T07:23:25.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f0ffeffd640 1 -- 172.21.15.83:0/3654683474 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ff80ab600 con 0x7f10001592f0 2024-09-19T07:23:25.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f0ffeffd640 1 -- 172.21.15.83:0/3654683474 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff80ab9d0 con 0x7f10001592f0 2024-09-19T07:23:25.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f1006380640 1 -- 172.21.15.83:0/3654683474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10001592f0 msgr2=0x7f1000157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:25.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.982+0000 7f1006380640 1 --2- 172.21.15.83:0/3654683474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10001592f0 0x7f1000157ce0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009870 tx=0x7f0ff801c520 comp rx=0 tx=0).stop 2024-09-19T07:23:25.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 -- 172.21.15.83:0/3654683474 shutdown_connections 2024-09-19T07:23:25.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 --2- 172.21.15.83:0/3654683474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10001592f0 0x7f1000157ce0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:25.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 --2- 172.21.15.83:0/3654683474 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10000c6120 0x7f10000c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:25.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 -- 172.21.15.83:0/3654683474 >> 172.21.15.83:0/3654683474 conn(0x7f100006bc80 msgr2=0x7f100006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:25.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 -- 172.21.15.83:0/3654683474 shutdown_connections 2024-09-19T07:23:25.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 -- 172.21.15.83:0/3654683474 wait complete. 2024-09-19T07:23:25.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 Processor -- start 2024-09-19T07:23:25.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 -- start start 2024-09-19T07:23:25.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10000c6120 0x7f10000c18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:25.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10001592f0 0x7f100014eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:25.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f100014a260 0x7f100014a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:25.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1000153860 con 0x7f10001592f0 2024-09-19T07:23:25.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1000162ea0 con 0x7f10000c6120 2024-09-19T07:23:25.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f10001f1c60 con 0x7f100014a260 2024-09-19T07:23:25.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0fff7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10001592f0 0x7f100014eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:25.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0fff7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10001592f0 0x7f100014eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59654/0 (socket says 172.21.15.83:59654) 2024-09-19T07:23:25.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0fff7fe640 1 -- 172.21.15.83:0/2822134610 learned_addr learned my addr 172.21.15.83:0/2822134610 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:25.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f100491d640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f100014a260 0x7f100014a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0ffffff640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10000c6120 0x7f10000c18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0fff7fe640 1 -- 172.21.15.83:0/2822134610 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f100014a260 msgr2=0x7f100014a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0fff7fe640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f100014a260 0x7f100014a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0fff7fe640 1 -- 172.21.15.83:0/2822134610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10000c6120 msgr2=0x7f10000c18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0fff7fe640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10000c6120 0x7f10000c18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0fff7fe640 1 -- 172.21.15.83:0/2822134610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10001f1f40 con 0x7f10001592f0 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f100491d640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f100014a260 0x7f100014a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0ffffff640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10000c6120 0x7f10000c18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0fff7fe640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10001592f0 0x7f100014eff0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0fe800c410 tx=0x7f0fe800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0ffcff9640 1 -- 172.21.15.83:0/2822134610 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8005020 con 0x7f10001592f0 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0ffcff9640 1 -- 172.21.15.83:0/2822134610 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fe80051c0 con 0x7f10001592f0 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 -- 172.21.15.83:0/2822134610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10001f2220 con 0x7f10001592f0 2024-09-19T07:23:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f0ffcff9640 1 -- 172.21.15.83:0/2822134610 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8007020 con 0x7f10001592f0 2024-09-19T07:23:25.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.986+0000 7f1006380640 1 -- 172.21.15.83:0/2822134610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10001f27e0 con 0x7f10001592f0 2024-09-19T07:23:25.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.990+0000 7f1006380640 1 -- 172.21.15.83:0/2822134610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10000c6500 con 0x7f10001592f0 2024-09-19T07:23:25.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.990+0000 7f0ffcff9640 1 -- 172.21.15.83:0/2822134610 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0fe8019030 con 0x7f10001592f0 2024-09-19T07:23:25.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.990+0000 7f0ffcff9640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0fe00678c0 0x7f0fe0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:25.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.990+0000 7f0ffcff9640 1 -- 172.21.15.83:0/2822134610 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f0fe8085860 con 0x7f10001592f0 2024-09-19T07:23:25.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.990+0000 7f0ffffff640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0fe00678c0 0x7f0fe0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:25.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.990+0000 7f0ffffff640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0fe00678c0 0x7f0fe0069d80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0007f50 tx=0x7f0ff00074a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:25.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:25.994+0000 7f0ffcff9640 1 -- 172.21.15.83:0/2822134610 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f0fe8020020 con 0x7f10001592f0 2024-09-19T07:23:26.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.218+0000 7f1006380640 1 -- 172.21.15.83:0/2822134610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10000c96a0 con 0x7f10001592f0 2024-09-19T07:23:26.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.218+0000 7f0ffcff9640 1 -- 172.21.15.83:0/2822134610 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6539 (secure 0 0 0) 0x7f0fe8055ee0 con 0x7f10001592f0 2024-09-19T07:23:26.220 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:26.220 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":38,"btime":"2024-09-19T07:23:13:002854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5304,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:26.220 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 38 2024-09-19T07:23:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.222+0000 7f1006380640 1 -- 172.21.15.83:0/2822134610 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0fe00678c0 msgr2=0x7f0fe0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.222+0000 7f1006380640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0fe00678c0 0x7f0fe0069d80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0007f50 tx=0x7f0ff00074a0 comp rx=0 tx=0).stop 2024-09-19T07:23:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.222+0000 7f1006380640 1 -- 172.21.15.83:0/2822134610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10001592f0 msgr2=0x7f100014eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.222+0000 7f1006380640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10001592f0 0x7f100014eff0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0fe800c410 tx=0x7f0fe800c8d0 comp rx=0 tx=0).stop 2024-09-19T07:23:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.222+0000 7f1006380640 1 -- 172.21.15.83:0/2822134610 shutdown_connections 2024-09-19T07:23:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.222+0000 7f1006380640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0fe00678c0 0x7f0fe0069d80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.222+0000 7f1006380640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f100014a260 0x7f100014a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.222+0000 7f1006380640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10001592f0 0x7f100014eff0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.222+0000 7f1006380640 1 --2- 172.21.15.83:0/2822134610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10000c6120 0x7f10000c18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:26.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.222+0000 7f1006380640 1 -- 172.21.15.83:0/2822134610 >> 172.21.15.83:0/2822134610 conn(0x7f100006bc80 msgr2=0x7f1000154fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:26.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.222+0000 7f1006380640 1 -- 172.21.15.83:0/2822134610 shutdown_connections 2024-09-19T07:23:26.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:26.222+0000 7f1006380640 1 -- 172.21.15.83:0/2822134610 wait complete. 2024-09-19T07:23:26.239 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:23:28.240 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:28.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c73b73640 1 Processor -- start 2024-09-19T07:23:28.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c73b73640 1 -- start start 2024-09-19T07:23:28.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c73b73640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4c6c0c6360 0x7f4c6c153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:28.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c73b73640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4c6c0bee30 0x7f4c6c1570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:28.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c73b73640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c6c0c5b90 con 0x7f4c6c0c9680 2024-09-19T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c73b73640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f4c6c15a050 con 0x7f4c6c0bee30 2024-09-19T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c73b73640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4c6c06c950 con 0x7f4c6c0c6360 2024-09-19T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c7110e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4c6c0bee30 0x7f4c6c1570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c7110e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4c6c0bee30 0x7f4c6c1570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59658/0 (socket says 172.21.15.83:59658) 2024-09-19T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c7110e640 1 -- 172.21.15.83:0/2739745391 learned_addr learned my addr 172.21.15.83:0/2739745391 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c7110e640 1 -- 172.21.15.83:0/2739745391 >> v2:172.21.15.184:3300/0 conn(0x7f4c6c0c6360 msgr2=0x7f4c6c153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c7190f640 1 --2- 172.21.15.83:0/2739745391 >> v2:172.21.15.184:3300/0 conn(0x7f4c6c0c6360 0x7f4c6c153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c7110e640 1 --2- 172.21.15.83:0/2739745391 >> v2:172.21.15.184:3300/0 conn(0x7f4c6c0c6360 0x7f4c6c153050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c7110e640 1 -- 172.21.15.83:0/2739745391 >> v1:172.21.15.184:6789/0 conn(0x7f4c6c0c9680 legacy=0x7f4c6c157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.390+0000 7f4c7110e640 1 -- 172.21.15.83:0/2739745391 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6c06c670 con 0x7f4c6c0bee30 2024-09-19T07:23:28.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c7110e640 1 --2- 172.21.15.83:0/2739745391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c6c0bee30 0x7f4c6c1570f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c0ae3c0 tx=0x7f4c5c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cad59413ec912f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:28.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c7090d640 1 -- 172.21.15.83:0/2739745391 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c0b0070 con 0x7f4c6c0bee30 2024-09-19T07:23:28.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c7090d640 1 -- 172.21.15.83:0/2739745391 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c5c0ab600 con 0x7f4c6c0bee30 2024-09-19T07:23:28.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c7090d640 1 -- 172.21.15.83:0/2739745391 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c0ab9d0 con 0x7f4c6c0bee30 2024-09-19T07:23:28.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c73b73640 1 -- 172.21.15.83:0/2739745391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c6c0bee30 msgr2=0x7f4c6c1570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:28.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c73b73640 1 --2- 172.21.15.83:0/2739745391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c6c0bee30 0x7f4c6c1570f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c0ae3c0 tx=0x7f4c5c01c3c0 comp rx=0 tx=0).stop 2024-09-19T07:23:28.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c73b73640 1 -- 172.21.15.83:0/2739745391 shutdown_connections 2024-09-19T07:23:28.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c73b73640 1 --2- 172.21.15.83:0/2739745391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c6c0bee30 0x7f4c6c1570f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:28.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c73b73640 1 --2- 172.21.15.83:0/2739745391 >> v2:172.21.15.184:3300/0 conn(0x7f4c6c0c6360 0x7f4c6c153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:28.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c73b73640 1 -- 172.21.15.83:0/2739745391 >> 172.21.15.83:0/2739745391 conn(0x7f4c6c0722a0 msgr2=0x7f4c6c0726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:28.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c73b73640 1 -- 172.21.15.83:0/2739745391 shutdown_connections 2024-09-19T07:23:28.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c73b73640 1 -- 172.21.15.83:0/2739745391 wait complete. 2024-09-19T07:23:28.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c73b73640 1 Processor -- start 2024-09-19T07:23:28.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.394+0000 7f4c73b73640 1 -- start start 2024-09-19T07:23:28.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c73b73640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c6c0bee30 0x7f4c6c0bfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:28.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c73b73640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c6c0c6360 0x7f4c6c14cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:28.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c73b73640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c6c0c9680 0x7f4c6c14d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:28.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c73b73640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6c0c68c0 con 0x7f4c6c0bee30 2024-09-19T07:23:28.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c73b73640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6c0c6740 con 0x7f4c6c0c9680 2024-09-19T07:23:28.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c73b73640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4c6c162da0 con 0x7f4c6c0c6360 2024-09-19T07:23:28.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c7190f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c6c0bee30 0x7f4c6c0bfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:28.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c7190f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c6c0bee30 0x7f4c6c0bfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59666/0 (socket says 172.21.15.83:59666) 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c7190f640 1 -- 172.21.15.83:0/1212968460 learned_addr learned my addr 172.21.15.83:0/1212968460 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c7110e640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c6c0c6360 0x7f4c6c14cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c72110640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c6c0c9680 0x7f4c6c14d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c7190f640 1 -- 172.21.15.83:0/1212968460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c6c0c6360 msgr2=0x7f4c6c14cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c7190f640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c6c0c6360 0x7f4c6c14cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c7190f640 1 -- 172.21.15.83:0/1212968460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c6c0c9680 msgr2=0x7f4c6c14d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c7190f640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c6c0c9680 0x7f4c6c14d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c7190f640 1 -- 172.21.15.83:0/1212968460 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6c0c4430 con 0x7f4c6c0bee30 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c7110e640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c6c0c6360 0x7f4c6c14cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c7190f640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c6c0bee30 0x7f4c6c0bfb20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4c6800c7f0 tx=0x7f4c6800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c627fc640 1 -- 172.21.15.83:0/1212968460 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c68015070 con 0x7f4c6c0bee30 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c627fc640 1 -- 172.21.15.83:0/1212968460 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c68007820 con 0x7f4c6c0bee30 2024-09-19T07:23:28.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c73b73640 1 -- 172.21.15.83:0/1212968460 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c6c0c4710 con 0x7f4c6c0bee30 2024-09-19T07:23:28.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c627fc640 1 -- 172.21.15.83:0/1212968460 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c68007c10 con 0x7f4c6c0bee30 2024-09-19T07:23:28.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c73b73640 1 -- 172.21.15.83:0/1212968460 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c6c0c4cd0 con 0x7f4c6c0bee30 2024-09-19T07:23:28.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c73b73640 1 -- 172.21.15.83:0/1212968460 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c6c153050 con 0x7f4c6c0bee30 2024-09-19T07:23:28.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c627fc640 1 -- 172.21.15.83:0/1212968460 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4c68018070 con 0x7f4c6c0bee30 2024-09-19T07:23:28.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c627fc640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c500678c0 0x7f4c50069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:28.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c627fc640 1 -- 172.21.15.83:0/1212968460 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f4c68086660 con 0x7f4c6c0bee30 2024-09-19T07:23:28.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.398+0000 7f4c7110e640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c500678c0 0x7f4c50069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:28.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.402+0000 7f4c7110e640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c500678c0 0x7f4c50069d80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0ad000 tx=0x7f4c5c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:28.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.406+0000 7f4c627fc640 1 -- 172.21.15.83:0/1212968460 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4c68056dc0 con 0x7f4c6c0bee30 2024-09-19T07:23:28.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.614+0000 7f4c73b73640 1 -- 172.21.15.83:0/1212968460 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c6c153d30 con 0x7f4c6c0bee30 2024-09-19T07:23:28.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c627fc640 1 -- 172.21.15.83:0/1212968460 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6539 (secure 0 0 0) 0x7f4c680566c0 con 0x7f4c6c0bee30 2024-09-19T07:23:28.617 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:28.618 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":38,"btime":"2024-09-19T07:23:13:002854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5304,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:28.618 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 38 2024-09-19T07:23:28.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c73b73640 1 -- 172.21.15.83:0/1212968460 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c500678c0 msgr2=0x7f4c50069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:28.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c73b73640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c500678c0 0x7f4c50069d80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0ad000 tx=0x7f4c5c008040 comp rx=0 tx=0).stop 2024-09-19T07:23:28.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c73b73640 1 -- 172.21.15.83:0/1212968460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c6c0bee30 msgr2=0x7f4c6c0bfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:28.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c73b73640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c6c0bee30 0x7f4c6c0bfb20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4c6800c7f0 tx=0x7f4c6800ccb0 comp rx=0 tx=0).stop 2024-09-19T07:23:28.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c73b73640 1 -- 172.21.15.83:0/1212968460 shutdown_connections 2024-09-19T07:23:28.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c73b73640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c500678c0 0x7f4c50069d80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:28.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c73b73640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c6c0c9680 0x7f4c6c14d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:28.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c73b73640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c6c0c6360 0x7f4c6c14cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:28.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c73b73640 1 --2- 172.21.15.83:0/1212968460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c6c0bee30 0x7f4c6c0bfb20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:28.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c73b73640 1 -- 172.21.15.83:0/1212968460 >> 172.21.15.83:0/1212968460 conn(0x7f4c6c0722a0 msgr2=0x7f4c6c156ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:28.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c73b73640 1 -- 172.21.15.83:0/1212968460 shutdown_connections 2024-09-19T07:23:28.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:28.618+0000 7f4c73b73640 1 -- 172.21.15.83:0/1212968460 wait complete. 2024-09-19T07:23:28.637 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:23:30.639 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:30.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49b0306640 1 Processor -- start 2024-09-19T07:23:30.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49b0306640 1 -- start start 2024-09-19T07:23:30.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49b0306640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f49a80c3a20 0x7f49a81594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:30.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49b0306640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a80c4450 0x7f49a81566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:30.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49b0306640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f49a8156c00 0x7f49a814c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:30.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49b0306640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f49a8068290 con 0x7f49a80c3a20 2024-09-19T07:23:30.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49b0306640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f49a8155ee0 con 0x7f49a8156c00 2024-09-19T07:23:30.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49b0306640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f49a80684e0 con 0x7f49a80c4450 2024-09-19T07:23:30.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49ae0a2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f49a80c3a20 0x7f49a81594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:30.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49ae0a2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f49a80c3a20 0x7f49a81594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44998/0 (socket says 172.21.15.83:44998) 2024-09-19T07:23:30.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49ae0a2640 1 -- 172.21.15.83:0/2400250410 learned_addr learned my addr 172.21.15.83:0/2400250410 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:30.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49ad8a1640 1 --2- 172.21.15.83:0/2400250410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a80c4450 0x7f49a81566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:30.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49ae8a3640 1 --2- 172.21.15.83:0/2400250410 >> v2:172.21.15.184:3300/0 conn(0x7f49a8156c00 0x7f49a814c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:30.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49ae0a2640 1 -- 172.21.15.83:0/2400250410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a80c4450 msgr2=0x7f49a81566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:30.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49ae0a2640 1 --2- 172.21.15.83:0/2400250410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a80c4450 0x7f49a81566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:30.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49ae0a2640 1 -- 172.21.15.83:0/2400250410 >> v2:172.21.15.184:3300/0 conn(0x7f49a8156c00 msgr2=0x7f49a814c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:30.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49ae0a2640 1 --2- 172.21.15.83:0/2400250410 >> v2:172.21.15.184:3300/0 conn(0x7f49a8156c00 0x7f49a814c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:30.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49ae0a2640 1 -- 172.21.15.83:0/2400250410 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a80687c0 con 0x7f49a80c3a20 2024-09-19T07:23:30.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.798+0000 7f49ae8a3640 1 --2- 172.21.15.83:0/2400250410 >> v2:172.21.15.184:3300/0 conn(0x7f49a8156c00 0x7f49a814c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:30.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ae0a2640 1 --2- 172.21.15.83:0/2400250410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a80c3a20 0x7f49a81594d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f49a80c5860 tx=0x7f49a401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e292e7192abc52c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:30.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ad0a0640 1 -- 172.21.15.83:0/2400250410 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a806fcd0 con 0x7f49a80c3a20 2024-09-19T07:23:30.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ad0a0640 1 -- 172.21.15.83:0/2400250410 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49a40b0070 con 0x7f49a80c3a20 2024-09-19T07:23:30.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ad0a0640 1 -- 172.21.15.83:0/2400250410 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a40ab860 con 0x7f49a80c3a20 2024-09-19T07:23:30.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 -- 172.21.15.83:0/2400250410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a80c3a20 msgr2=0x7f49a81594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:30.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 --2- 172.21.15.83:0/2400250410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a80c3a20 0x7f49a81594d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f49a80c5860 tx=0x7f49a401c460 comp rx=0 tx=0).stop 2024-09-19T07:23:30.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 -- 172.21.15.83:0/2400250410 shutdown_connections 2024-09-19T07:23:30.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 --2- 172.21.15.83:0/2400250410 >> v2:172.21.15.184:3300/0 conn(0x7f49a8156c00 0x7f49a814c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:30.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 --2- 172.21.15.83:0/2400250410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a80c4450 0x7f49a81566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:30.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 --2- 172.21.15.83:0/2400250410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a80c3a20 0x7f49a81594d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:30.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 -- 172.21.15.83:0/2400250410 >> 172.21.15.83:0/2400250410 conn(0x7f49a80c48b0 msgr2=0x7f49a80c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:30.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 -- 172.21.15.83:0/2400250410 shutdown_connections 2024-09-19T07:23:30.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 -- 172.21.15.83:0/2400250410 wait complete. 2024-09-19T07:23:30.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 Processor -- start 2024-09-19T07:23:30.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 -- start start 2024-09-19T07:23:30.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a80c3a20 0x7f49a8158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:30.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a80c4450 0x7f49a8072e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:30.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a8156c00 0x7f49a8140710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:30.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f49a80c3f80 con 0x7f49a8156c00 2024-09-19T07:23:30.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49a80c3e00 con 0x7f49a80c4450 2024-09-19T07:23:30.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49b0306640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f49a80c1a80 con 0x7f49a80c3a20 2024-09-19T07:23:30.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ae8a3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a8156c00 0x7f49a8140710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:30.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ae8a3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a8156c00 0x7f49a8140710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45010/0 (socket says 172.21.15.83:45010) 2024-09-19T07:23:30.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ae8a3640 1 -- 172.21.15.83:0/2282196958 learned_addr learned my addr 172.21.15.83:0/2282196958 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:30.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ae0a2640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a80c3a20 0x7f49a8158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:30.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ad8a1640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a80c4450 0x7f49a8072e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:30.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ae8a3640 1 -- 172.21.15.83:0/2282196958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a80c3a20 msgr2=0x7f49a8158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:30.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ae8a3640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a80c3a20 0x7f49a8158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:30.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ae8a3640 1 -- 172.21.15.83:0/2282196958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a80c4450 msgr2=0x7f49a8072e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:30.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ae8a3640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a80c4450 0x7f49a8072e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:30.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.802+0000 7f49ae8a3640 1 -- 172.21.15.83:0/2282196958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a81572d0 con 0x7f49a8156c00 2024-09-19T07:23:30.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.806+0000 7f49ae8a3640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a8156c00 0x7f49a8140710 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f49a000cd80 tx=0x7f49a000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:30.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.806+0000 7f499effd640 1 -- 172.21.15.83:0/2282196958 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a000f680 con 0x7f49a8156c00 2024-09-19T07:23:30.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.806+0000 7f499effd640 1 -- 172.21.15.83:0/2282196958 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49a000f840 con 0x7f49a8156c00 2024-09-19T07:23:30.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.806+0000 7f49b0306640 1 -- 172.21.15.83:0/2282196958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a8157590 con 0x7f49a8156c00 2024-09-19T07:23:30.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.806+0000 7f499effd640 1 -- 172.21.15.83:0/2282196958 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a000fc30 con 0x7f49a8156c00 2024-09-19T07:23:30.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.806+0000 7f49b0306640 1 -- 172.21.15.83:0/2282196958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a81f2a30 con 0x7f49a8156c00 2024-09-19T07:23:30.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.806+0000 7f499effd640 1 -- 172.21.15.83:0/2282196958 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f49a00213f0 con 0x7f49a8156c00 2024-09-19T07:23:30.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.806+0000 7f499cff9640 1 -- 172.21.15.83:0/2282196958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a80c1ec0 con 0x7f49a8156c00 2024-09-19T07:23:30.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.806+0000 7f499effd640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f498c0678c0 0x7f498c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:30.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.806+0000 7f499effd640 1 -- 172.21.15.83:0/2282196958 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f49a0086ed0 con 0x7f49a8156c00 2024-09-19T07:23:30.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.806+0000 7f49ae0a2640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f498c0678c0 0x7f498c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:30.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.806+0000 7f49ae0a2640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f498c0678c0 0x7f498c069d80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f49a40af000 tx=0x7f49a40a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:30.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:30.810+0000 7f499effd640 1 -- 172.21.15.83:0/2282196958 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f49a0014030 con 0x7f49a8156c00 2024-09-19T07:23:30.984 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:23:30.987 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:23:31.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.046+0000 7f499cff9640 1 -- 172.21.15.83:0/2282196958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49a81595f0 con 0x7f49a8156c00 2024-09-19T07:23:31.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499effd640 1 -- 172.21.15.83:0/2282196958 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6539 (secure 0 0 0) 0x7f49a0057550 con 0x7f49a8156c00 2024-09-19T07:23:31.049 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:31.049 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":38,"btime":"2024-09-19T07:23:13:002854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5304,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:31.050 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 38 2024-09-19T07:23:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499cff9640 1 -- 172.21.15.83:0/2282196958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f498c0678c0 msgr2=0x7f498c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499cff9640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f498c0678c0 0x7f498c069d80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f49a40af000 tx=0x7f49a40a7000 comp rx=0 tx=0).stop 2024-09-19T07:23:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499cff9640 1 -- 172.21.15.83:0/2282196958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a8156c00 msgr2=0x7f49a8140710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499cff9640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a8156c00 0x7f49a8140710 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f49a000cd80 tx=0x7f49a000a650 comp rx=0 tx=0).stop 2024-09-19T07:23:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499cff9640 1 -- 172.21.15.83:0/2282196958 shutdown_connections 2024-09-19T07:23:31.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499cff9640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f498c0678c0 0x7f498c069d80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:31.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499cff9640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a8156c00 0x7f49a8140710 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:31.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499cff9640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a80c4450 0x7f49a8072e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:31.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499cff9640 1 --2- 172.21.15.83:0/2282196958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a80c3a20 0x7f49a8158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:31.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499cff9640 1 -- 172.21.15.83:0/2282196958 >> 172.21.15.83:0/2282196958 conn(0x7f49a80c48b0 msgr2=0x7f49a814b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:31.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499cff9640 1 -- 172.21.15.83:0/2282196958 shutdown_connections 2024-09-19T07:23:31.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:31.050+0000 7f499cff9640 1 -- 172.21.15.83:0/2282196958 wait complete. 2024-09-19T07:23:31.070 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:23:33.070 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:33.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.202+0000 7f29b5638640 1 Processor -- start 2024-09-19T07:23:33.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.202+0000 7f29b5638640 1 -- start start 2024-09-19T07:23:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29b5638640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29b00c3130 0x7f29b00ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29b5638640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f29b006de40 0x7f29b01563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29b5638640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29b00c3c20 con 0x7f29b006da00 2024-09-19T07:23:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29b5638640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f29b00c4990 con 0x7f29b006de40 2024-09-19T07:23:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29b5638640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f29b00c4c70 con 0x7f29b00c3130 2024-09-19T07:23:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29af7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f29b006de40 0x7f29b01563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29af7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f29b006de40 0x7f29b01563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45026/0 (socket says 172.21.15.83:45026) 2024-09-19T07:23:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29af7fe640 1 -- 172.21.15.83:0/3471087424 learned_addr learned my addr 172.21.15.83:0/3471087424 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29aeffd640 1 --2- 172.21.15.83:0/3471087424 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29b00c3130 0x7f29b00ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29af7fe640 1 -- 172.21.15.83:0/3471087424 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29b00c3130 msgr2=0x7f29b00ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:33.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29af7fe640 1 --2- 172.21.15.83:0/3471087424 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29b00c3130 0x7f29b00ac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:33.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29af7fe640 1 -- 172.21.15.83:0/3471087424 >> v1:172.21.15.184:6789/0 conn(0x7f29b006da00 legacy=0x7f29b00c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:33.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29af7fe640 1 -- 172.21.15.83:0/3471087424 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b00ab7b0 con 0x7f29b006de40 2024-09-19T07:23:33.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29af7fe640 1 --2- 172.21.15.83:0/3471087424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29b006de40 0x7f29b01563a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f29a8009e50 tx=0x7f29a801c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3d1ba62cd3550c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:33.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29adffb640 1 -- 172.21.15.83:0/3471087424 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a80ac020 con 0x7f29b006de40 2024-09-19T07:23:33.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29adffb640 1 -- 172.21.15.83:0/3471087424 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29a80a64a0 con 0x7f29b006de40 2024-09-19T07:23:33.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29adffb640 1 -- 172.21.15.83:0/3471087424 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a80a6870 con 0x7f29b006de40 2024-09-19T07:23:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29b5638640 1 -- 172.21.15.83:0/3471087424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29b006de40 msgr2=0x7f29b01563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29b5638640 1 --2- 172.21.15.83:0/3471087424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29b006de40 0x7f29b01563a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f29a8009e50 tx=0x7f29a801c6e0 comp rx=0 tx=0).stop 2024-09-19T07:23:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29b5638640 1 -- 172.21.15.83:0/3471087424 shutdown_connections 2024-09-19T07:23:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29b5638640 1 --2- 172.21.15.83:0/3471087424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29b006de40 0x7f29b01563a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29b5638640 1 --2- 172.21.15.83:0/3471087424 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29b00c3130 0x7f29b00ac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.206+0000 7f29b5638640 1 -- 172.21.15.83:0/3471087424 >> 172.21.15.83:0/3471087424 conn(0x7f29b00c5060 msgr2=0x7f29b00c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29b5638640 1 -- 172.21.15.83:0/3471087424 shutdown_connections 2024-09-19T07:23:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29b5638640 1 -- 172.21.15.83:0/3471087424 wait complete. 2024-09-19T07:23:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29b5638640 1 Processor -- start 2024-09-19T07:23:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29b5638640 1 -- start start 2024-09-19T07:23:33.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29b5638640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29b006da00 0x7f29b01ebe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:33.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29b5638640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29b006de40 0x7f29b014dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:33.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29b5638640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29b00c3130 0x7f29b014b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:33.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29b5638640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29b00ad6a0 con 0x7f29b006da00 2024-09-19T07:23:33.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29b5638640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f29b00ad520 con 0x7f29b00c3130 2024-09-19T07:23:33.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29b5638640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f29b015f810 con 0x7f29b006de40 2024-09-19T07:23:33.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29aeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29b006da00 0x7f29b01ebe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:33.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29aeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29b006da00 0x7f29b01ebe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45040/0 (socket says 172.21.15.83:45040) 2024-09-19T07:23:33.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29aeffd640 1 -- 172.21.15.83:0/1804236899 learned_addr learned my addr 172.21.15.83:0/1804236899 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:33.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29ae7fc640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29b006de40 0x7f29b014dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:33.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29af7fe640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29b00c3130 0x7f29b014b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:33.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29aeffd640 1 -- 172.21.15.83:0/1804236899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29b006de40 msgr2=0x7f29b014dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:33.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29aeffd640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29b006de40 0x7f29b014dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:33.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29aeffd640 1 -- 172.21.15.83:0/1804236899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29b00c3130 msgr2=0x7f29b014b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:33.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29aeffd640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29b00c3130 0x7f29b014b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:33.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29aeffd640 1 -- 172.21.15.83:0/1804236899 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b01489b0 con 0x7f29b006da00 2024-09-19T07:23:33.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29af7fe640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29b00c3130 0x7f29b014b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:23:33.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29aeffd640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29b006da00 0x7f29b01ebe60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f29a000d5f0 tx=0x7f29a000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:33.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29977fe640 1 -- 172.21.15.83:0/1804236899 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a00125f0 con 0x7f29b006da00 2024-09-19T07:23:33.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29977fe640 1 -- 172.21.15.83:0/1804236899 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29a0019070 con 0x7f29b006da00 2024-09-19T07:23:33.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29b5638640 1 -- 172.21.15.83:0/1804236899 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b014fab0 con 0x7f29b006da00 2024-09-19T07:23:33.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29977fe640 1 -- 172.21.15.83:0/1804236899 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a00129b0 con 0x7f29b006da00 2024-09-19T07:23:33.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.210+0000 7f29b5638640 1 -- 172.21.15.83:0/1804236899 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b0150070 con 0x7f29b006da00 2024-09-19T07:23:33.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.214+0000 7f29b5638640 1 -- 172.21.15.83:0/1804236899 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b01563a0 con 0x7f29b006da00 2024-09-19T07:23:33.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.214+0000 7f29977fe640 1 -- 172.21.15.83:0/1804236899 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f29a0012b50 con 0x7f29b006da00 2024-09-19T07:23:33.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.214+0000 7f29977fe640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f299c0678c0 0x7f299c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:33.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.214+0000 7f29ae7fc640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f299c0678c0 0x7f299c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:33.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.214+0000 7f29977fe640 1 -- 172.21.15.83:0/1804236899 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f29a0086480 con 0x7f29b006da00 2024-09-19T07:23:33.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.214+0000 7f29ae7fc640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f299c0678c0 0x7f299c069d80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29a4007fd0 tx=0x7f29a4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:33.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.222+0000 7f29977fe640 1 -- 172.21.15.83:0/1804236899 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f29a0056b00 con 0x7f29b006da00 2024-09-19T07:23:33.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.446+0000 7f29b5638640 1 -- 172.21.15.83:0/1804236899 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29b006c830 con 0x7f29b006da00 2024-09-19T07:23:33.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.446+0000 7f29977fe640 1 -- 172.21.15.83:0/1804236899 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6539 (secure 0 0 0) 0x7f29a0056400 con 0x7f29b006da00 2024-09-19T07:23:33.448 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:33.448 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":38,"btime":"2024-09-19T07:23:13:002854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5304,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:33.448 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 38 2024-09-19T07:23:33.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.450+0000 7f29b5638640 1 -- 172.21.15.83:0/1804236899 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f299c0678c0 msgr2=0x7f299c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:33.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.450+0000 7f29b5638640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f299c0678c0 0x7f299c069d80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29a4007fd0 tx=0x7f29a4009040 comp rx=0 tx=0).stop 2024-09-19T07:23:33.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.450+0000 7f29b5638640 1 -- 172.21.15.83:0/1804236899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29b006da00 msgr2=0x7f29b01ebe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:33.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.450+0000 7f29b5638640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29b006da00 0x7f29b01ebe60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f29a000d5f0 tx=0x7f29a000dab0 comp rx=0 tx=0).stop 2024-09-19T07:23:33.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.450+0000 7f29b5638640 1 -- 172.21.15.83:0/1804236899 shutdown_connections 2024-09-19T07:23:33.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.450+0000 7f29b5638640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f299c0678c0 0x7f299c069d80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:33.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.450+0000 7f29b5638640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29b00c3130 0x7f29b014b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:33.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.450+0000 7f29b5638640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29b006de40 0x7f29b014dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:33.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.450+0000 7f29b5638640 1 --2- 172.21.15.83:0/1804236899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29b006da00 0x7f29b01ebe60 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:33.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.450+0000 7f29b5638640 1 -- 172.21.15.83:0/1804236899 >> 172.21.15.83:0/1804236899 conn(0x7f29b00c5060 msgr2=0x7f29b00c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:33.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.450+0000 7f29b5638640 1 -- 172.21.15.83:0/1804236899 shutdown_connections 2024-09-19T07:23:33.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:33.450+0000 7f29b5638640 1 -- 172.21.15.83:0/1804236899 wait complete. 2024-09-19T07:23:33.467 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:23:35.468 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:35.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f4412be3640 1 Processor -- start 2024-09-19T07:23:35.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f4412be3640 1 -- start start 2024-09-19T07:23:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f4412be3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f440c14a500 0x7f440c0c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f4412be3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f440c0c4550 0x7f440c157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f4412be3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f440c151750 con 0x7f440c154540 2024-09-19T07:23:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f4412be3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f440c0c7e90 con 0x7f440c0c4550 2024-09-19T07:23:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f4412be3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f440c073110 con 0x7f440c14a500 2024-09-19T07:23:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f440c154540 0x7f440c157490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56294/0 (socket says 172.21.15.83:56294) 2024-09-19T07:23:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440bfff640 1 -- 172.21.15.83:0/4027091667 learned_addr learned my addr 172.21.15.83:0/4027091667 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f441097f640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f440c14a500 0x7f440c0c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326654561 0 0) 0x7f43f8001120 con 0x7f440c154540 2024-09-19T07:23:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f4411180640 1 --2- 172.21.15.83:0/4027091667 >> v2:172.21.15.184:3300/0 conn(0x7f440c0c4550 0x7f440c157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43fc003530 con 0x7f440c154540 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 853622186 0 0) 0x7f43fc003530 con 0x7f440c154540 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f440c14a500 msgr2=0x7f440c0c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f440c14a500 0x7f440c0c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 >> v2:172.21.15.184:3300/0 conn(0x7f440c0c4550 msgr2=0x7f440c157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 --2- 172.21.15.83:0/4027091667 >> v2:172.21.15.184:3300/0 conn(0x7f440c0c4550 0x7f440c157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c0733f0 con 0x7f440c154540 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f43f8001850 con 0x7f440c154540 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f4411180640 1 --2- 172.21.15.83:0/4027091667 >> v2:172.21.15.184:3300/0 conn(0x7f440c0c4550 0x7f440c157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 >> v1:172.21.15.83:6789/0 conn(0x7f440c154540 legacy=0x7f440c157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43fc0039c0 0x7f43fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f440c14a500 0x7f43fc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43fc004d50 0x7f43fc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f440c0c7e90 con 0x7f43fc0039c0 2024-09-19T07:23:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f440c073110 con 0x7f43fc004d50 2024-09-19T07:23:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f43fc00b9d0 con 0x7f440c14a500 2024-09-19T07:23:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f441097f640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43fc0039c0 0x7f43fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f4411180640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43fc004d50 0x7f43fc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f440bfff640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f440c14a500 0x7f43fc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f441097f640 1 -- 172.21.15.83:0/4027091667 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f440c14a500 msgr2=0x7f43fc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f441097f640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f440c14a500 0x7f43fc004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f441097f640 1 -- 172.21.15.83:0/4027091667 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43fc004d50 msgr2=0x7f43fc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f441097f640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43fc004d50 0x7f43fc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:35.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f441097f640 1 -- 172.21.15.83:0/4027091667 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc003530 con 0x7f43fc0039c0 2024-09-19T07:23:35.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.830+0000 7f4411180640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43fc004d50 0x7f43fc0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:35.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f441097f640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43fc0039c0 0x7f43fc003de0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4400009b10 tx=0x7f440001ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0cf3152a225427e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:35.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44000b2070 con 0x7f43fc0039c0 2024-09-19T07:23:35.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44000a5520 con 0x7f43fc0039c0 2024-09-19T07:23:35.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f440b7fe640 1 -- 172.21.15.83:0/4027091667 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44000096b0 con 0x7f43fc0039c0 2024-09-19T07:23:35.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f4412be3640 1 -- 172.21.15.83:0/4027091667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43fc0039c0 msgr2=0x7f43fc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:35.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f4412be3640 1 --2- 172.21.15.83:0/4027091667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43fc0039c0 0x7f43fc003de0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4400009b10 tx=0x7f440001ca40 comp rx=0 tx=0).stop 2024-09-19T07:23:35.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f4411180640 1 -- 172.21.15.83:0/4027091667 reap_dead start 2024-09-19T07:23:35.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f4412be3640 1 -- 172.21.15.83:0/4027091667 shutdown_connections 2024-09-19T07:23:35.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f4412be3640 1 -- 172.21.15.83:0/4027091667 >> 172.21.15.83:0/4027091667 conn(0x7f440c0c9610 msgr2=0x7f440c0c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:35.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f4412be3640 1 -- 172.21.15.83:0/4027091667 shutdown_connections 2024-09-19T07:23:35.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f4412be3640 1 -- 172.21.15.83:0/4027091667 wait complete. 2024-09-19T07:23:35.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f4412be3640 1 Processor -- start 2024-09-19T07:23:35.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.834+0000 7f4412be3640 1 -- start start 2024-09-19T07:23:35.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f4412be3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43fc0039c0 0x7f440c14d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f4412be3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f440c145da0 0x7f440c0c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f4412be3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f440c06c7b0 0x7f440c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f4412be3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f440c0c7850 con 0x7f440c145da0 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f4412be3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f440c0c76d0 con 0x7f440c06c7b0 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f4412be3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f440c06c490 con 0x7f43fc0039c0 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f440bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f440c145da0 0x7f440c0c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f440bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f440c145da0 0x7f440c0c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45062/0 (socket says 172.21.15.83:45062) 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f440bfff640 1 -- 172.21.15.83:0/3229357244 learned_addr learned my addr 172.21.15.83:0/3229357244 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f441097f640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43fc0039c0 0x7f440c14d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f4411180640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f440c06c7b0 0x7f440c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f440bfff640 1 -- 172.21.15.83:0/3229357244 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43fc0039c0 msgr2=0x7f440c14d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f440bfff640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43fc0039c0 0x7f440c14d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f440bfff640 1 -- 172.21.15.83:0/3229357244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f440c06c7b0 msgr2=0x7f440c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:35.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f440bfff640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f440c06c7b0 0x7f440c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:35.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f440bfff640 1 -- 172.21.15.83:0/3229357244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c06e960 con 0x7f440c145da0 2024-09-19T07:23:35.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f440bfff640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f440c145da0 0x7f440c0c8810 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f43f800d8e0 tx=0x7f43f800dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:35.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f44097fa640 1 -- 172.21.15.83:0/3229357244 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f8019070 con 0x7f440c145da0 2024-09-19T07:23:35.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f44097fa640 1 -- 172.21.15.83:0/3229357244 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f80074e0 con 0x7f440c145da0 2024-09-19T07:23:35.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f4412be3640 1 -- 172.21.15.83:0/3229357244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f440c1ee820 con 0x7f440c145da0 2024-09-19T07:23:35.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f44097fa640 1 -- 172.21.15.83:0/3229357244 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f8004a30 con 0x7f440c145da0 2024-09-19T07:23:35.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f4412be3640 1 -- 172.21.15.83:0/3229357244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440c0c8d50 con 0x7f440c145da0 2024-09-19T07:23:35.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f4412be3640 1 -- 172.21.15.83:0/3229357244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f440c157490 con 0x7f440c145da0 2024-09-19T07:23:35.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.838+0000 7f44097fa640 1 -- 172.21.15.83:0/3229357244 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f43f801c070 con 0x7f440c145da0 2024-09-19T07:23:35.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.842+0000 7f44097fa640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f43f00678c0 0x7f43f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:35.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.842+0000 7f44097fa640 1 -- 172.21.15.83:0/3229357244 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f43f8085fe0 con 0x7f440c145da0 2024-09-19T07:23:35.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.842+0000 7f441097f640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f43f00678c0 0x7f43f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:35.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.842+0000 7f441097f640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f43f00678c0 0x7f43f0069d80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f44000af000 tx=0x7f44000046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:35.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:35.842+0000 7f44097fa640 1 -- 172.21.15.83:0/3229357244 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f43f8023020 con 0x7f440c145da0 2024-09-19T07:23:36.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.050+0000 7f4412be3640 1 -- 172.21.15.83:0/3229357244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f440c073fd0 con 0x7f440c145da0 2024-09-19T07:23:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.050+0000 7f44097fa640 1 -- 172.21.15.83:0/3229357244 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6539 (secure 0 0 0) 0x7f440c151750 con 0x7f440c145da0 2024-09-19T07:23:36.051 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:36.051 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":38,"btime":"2024-09-19T07:23:13:002854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5304,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 38 2024-09-19T07:23:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.054+0000 7f4412be3640 1 -- 172.21.15.83:0/3229357244 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f43f00678c0 msgr2=0x7f43f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.054+0000 7f4412be3640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f43f00678c0 0x7f43f0069d80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f44000af000 tx=0x7f44000046d0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.054+0000 7f4412be3640 1 -- 172.21.15.83:0/3229357244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f440c145da0 msgr2=0x7f440c0c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.054+0000 7f4412be3640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f440c145da0 0x7f440c0c8810 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f43f800d8e0 tx=0x7f43f800dda0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.054+0000 7f4412be3640 1 -- 172.21.15.83:0/3229357244 shutdown_connections 2024-09-19T07:23:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.054+0000 7f4412be3640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f43f00678c0 0x7f43f0069d80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.054+0000 7f4412be3640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f440c06c7b0 0x7f440c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.054+0000 7f4412be3640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f440c145da0 0x7f440c0c8810 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.054+0000 7f4412be3640 1 --2- 172.21.15.83:0/3229357244 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43fc0039c0 0x7f440c14d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.054+0000 7f4412be3640 1 -- 172.21.15.83:0/3229357244 >> 172.21.15.83:0/3229357244 conn(0x7f440c0c9610 msgr2=0x7f440c147d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.054+0000 7f4412be3640 1 -- 172.21.15.83:0/3229357244 shutdown_connections 2024-09-19T07:23:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.054+0000 7f4412be3640 1 -- 172.21.15.83:0/3229357244 wait complete. 2024-09-19T07:23:36.070 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:23:36.275 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:36.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f8d5bf640 1 Processor -- start 2024-09-19T07:23:36.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f8d5bf640 1 -- start start 2024-09-19T07:23:36.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f8d5bf640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1f880c7510 0x7f1f8806de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f8d5bf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8814ddd0 con 0x7f1f88154360 2024-09-19T07:23:36.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f8d5bf640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8806fdf0 con 0x7f1f8814d9f0 2024-09-19T07:23:36.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f8d5bf640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f1f880c8830 con 0x7f1f880c7510 2024-09-19T07:23:36.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f877fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1f88154360 0x7f1f880c4010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56296/0 (socket says 172.21.15.83:56296) 2024-09-19T07:23:36.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f877fe640 1 -- 172.21.15.83:0/2770398038 learned_addr learned my addr 172.21.15.83:0/2770398038 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:36.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 345939925 0 0) 0x7f1f8814ddd0 con 0x7f1f88154360 2024-09-19T07:23:36.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f74003530 con 0x7f1f88154360 2024-09-19T07:23:36.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f7ffff640 1 --2- 172.21.15.83:0/2770398038 >> v2:172.21.15.184:3300/0 conn(0x7f1f880c7510 0x7f1f8806de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:36.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 180104161 0 0) 0x7f1f74003530 con 0x7f1f88154360 2024-09-19T07:23:36.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 >> v2:172.21.15.184:3300/0 conn(0x7f1f880c7510 msgr2=0x7f1f8806de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:36.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 --2- 172.21.15.83:0/2770398038 >> v2:172.21.15.184:3300/0 conn(0x7f1f880c7510 0x7f1f8806de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 >> v1:172.21.15.184:6789/0 conn(0x7f1f8814d9f0 legacy=0x7f1f880c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f880c7f00 con 0x7f1f88154360 2024-09-19T07:23:36.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f1f80001a00 con 0x7f1f88154360 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 >> v1:172.21.15.83:6789/0 conn(0x7f1f88154360 legacy=0x7f1f880c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 --2- 172.21.15.83:0/2770398038 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f740039c0 0x7f1f74003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 --2- 172.21.15.83:0/2770398038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f74004810 0x7f1f74004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 --2- 172.21.15.83:0/2770398038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 0x7f1f7400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1f880c8830 con 0x7f1f740053d0 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7400c260 con 0x7f1f74004810 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f7400c460 con 0x7f1f740039c0 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f877fe640 1 --2- 172.21.15.83:0/2770398038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 0x7f1f7400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f7ffff640 1 --2- 172.21.15.83:0/2770398038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f74004810 0x7f1f74004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f86ffd640 1 --2- 172.21.15.83:0/2770398038 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f740039c0 0x7f1f74003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f877fe640 1 -- 172.21.15.83:0/2770398038 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f740039c0 msgr2=0x7f1f74003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f877fe640 1 --2- 172.21.15.83:0/2770398038 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f740039c0 0x7f1f74003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f877fe640 1 -- 172.21.15.83:0/2770398038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f74004810 msgr2=0x7f1f74004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:36.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.398+0000 7f1f877fe640 1 --2- 172.21.15.83:0/2770398038 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f74004810 0x7f1f74004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f877fe640 1 -- 172.21.15.83:0/2770398038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f74003530 con 0x7f1f740053d0 2024-09-19T07:23:36.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f877fe640 1 -- 172.21.15.83:0/2770398038 reap_dead start 2024-09-19T07:23:36.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f877fe640 1 --2- 172.21.15.83:0/2770398038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 0x7f1f7400b9d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1f8000a850 tx=0x7f1f8001d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b012d26c8720d91 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:36.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f8000a4a0 con 0x7f1f740053d0 2024-09-19T07:23:36.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f800ad650 con 0x7f1f740053d0 2024-09-19T07:23:36.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f867fc640 1 -- 172.21.15.83:0/2770398038 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f800ada40 con 0x7f1f740053d0 2024-09-19T07:23:36.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/2770398038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 msgr2=0x7f1f7400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f8d5bf640 1 --2- 172.21.15.83:0/2770398038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 0x7f1f7400b9d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1f8000a850 tx=0x7f1f8001d3a0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/2770398038 shutdown_connections 2024-09-19T07:23:36.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f8d5bf640 1 --2- 172.21.15.83:0/2770398038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 0x7f1f7400b9d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/2770398038 >> 172.21.15.83:0/2770398038 conn(0x7f1f880c6110 msgr2=0x7f1f880c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:36.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/2770398038 shutdown_connections 2024-09-19T07:23:36.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.402+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/2770398038 wait complete. 2024-09-19T07:23:36.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f8d5bf640 1 Processor -- start 2024-09-19T07:23:36.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f8d5bf640 1 -- start start 2024-09-19T07:23:36.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f8d5bf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 0x7f1f881ef880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f8d5bf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f880ab5d0 0x7f1f880ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f8d5bf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f880aa4a0 0x7f1f880ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f8d5bf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8806e220 con 0x7f1f740053d0 2024-09-19T07:23:36.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f8d5bf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8806e0a0 con 0x7f1f880ab5d0 2024-09-19T07:23:36.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f8d5bf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f880c8ca0 con 0x7f1f880aa4a0 2024-09-19T07:23:36.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f86ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 0x7f1f881ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:36.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f86ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 0x7f1f881ef880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45076/0 (socket says 172.21.15.83:45076) 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f86ffd640 1 -- 172.21.15.83:0/1192765427 learned_addr learned my addr 172.21.15.83:0/1192765427 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f877fe640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f880aa4a0 0x7f1f880ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f7ffff640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f880ab5d0 0x7f1f880ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f86ffd640 1 -- 172.21.15.83:0/1192765427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f880aa4a0 msgr2=0x7f1f880ae790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f86ffd640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f880aa4a0 0x7f1f880ae790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f86ffd640 1 -- 172.21.15.83:0/1192765427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f880ab5d0 msgr2=0x7f1f880ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f86ffd640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f880ab5d0 0x7f1f880ae250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f86ffd640 1 -- 172.21.15.83:0/1192765427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f881f00d0 con 0x7f1f740053d0 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f7ffff640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f880ab5d0 0x7f1f880ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f86ffd640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 0x7f1f881ef880 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800e4b0 tx=0x7f1f7800e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f7f7fe640 1 -- 172.21.15.83:0/1192765427 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f780052e0 con 0x7f1f740053d0 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f7f7fe640 1 -- 172.21.15.83:0/1192765427 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f78018070 con 0x7f1f740053d0 2024-09-19T07:23:36.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/1192765427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f881f0340 con 0x7f1f740053d0 2024-09-19T07:23:36.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f7f7fe640 1 -- 172.21.15.83:0/1192765427 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f78005480 con 0x7f1f740053d0 2024-09-19T07:23:36.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/1192765427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f88157d40 con 0x7f1f740053d0 2024-09-19T07:23:36.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/1192765427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f880c9030 con 0x7f1f740053d0 2024-09-19T07:23:36.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.406+0000 7f1f7f7fe640 1 -- 172.21.15.83:0/1192765427 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1f78003840 con 0x7f1f740053d0 2024-09-19T07:23:36.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.410+0000 7f1f7f7fe640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f680678f0 0x7f1f68069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.410+0000 7f1f7f7fe640 1 -- 172.21.15.83:0/1192765427 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f1f78085a60 con 0x7f1f740053d0 2024-09-19T07:23:36.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.410+0000 7f1f7ffff640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f680678f0 0x7f1f68069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:36.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.410+0000 7f1f7ffff640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f680678f0 0x7f1f68069db0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1f880ab9b0 tx=0x7f1f70008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:36.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.414+0000 7f1f7f7fe640 1 -- 172.21.15.83:0/1192765427 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1f780560e0 con 0x7f1f740053d0 2024-09-19T07:23:36.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.642+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/1192765427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f8806faf0 con 0x7f1f740053d0 2024-09-19T07:23:36.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.642+0000 7f1f7f7fe640 1 -- 172.21.15.83:0/1192765427 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6539 (secure 0 0 0) 0x7f1f780559e0 con 0x7f1f740053d0 2024-09-19T07:23:36.643 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:36.644 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":38,"btime":"2024-09-19T07:23:13:002854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5304,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:36.644 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 38 2024-09-19T07:23:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.642+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/1192765427 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f680678f0 msgr2=0x7f1f68069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.642+0000 7f1f8d5bf640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f680678f0 0x7f1f68069db0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1f880ab9b0 tx=0x7f1f70008040 comp rx=0 tx=0).stop 2024-09-19T07:23:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.642+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/1192765427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 msgr2=0x7f1f881ef880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.642+0000 7f1f8d5bf640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 0x7f1f881ef880 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800e4b0 tx=0x7f1f7800e970 comp rx=0 tx=0).stop 2024-09-19T07:23:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.642+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/1192765427 shutdown_connections 2024-09-19T07:23:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.642+0000 7f1f8d5bf640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f680678f0 0x7f1f68069db0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.646+0000 7f1f8d5bf640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f880aa4a0 0x7f1f880ae790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.646+0000 7f1f8d5bf640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f880ab5d0 0x7f1f880ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.646+0000 7f1f8d5bf640 1 --2- 172.21.15.83:0/1192765427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f740053d0 0x7f1f881ef880 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.646+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/1192765427 >> 172.21.15.83:0/1192765427 conn(0x7f1f880c6110 msgr2=0x7f1f8806e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:36.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.646+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/1192765427 shutdown_connections 2024-09-19T07:23:36.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.646+0000 7f1f8d5bf640 1 -- 172.21.15.83:0/1192765427 wait complete. 2024-09-19T07:23:36.663 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-09-19T07:23:36.663 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-19T07:23:36.664 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:23:36.678 INFO:tasks.ceph.mds.b:Stopped 2024-09-19T07:23:36.679 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:36.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f6bfcb640 1 Processor -- start 2024-09-19T07:23:36.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f6bfcb640 1 -- start start 2024-09-19T07:23:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f6bfcb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6f640c7d70 0x7f6f64157c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f6bfcb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6f64072d00 0x7f6f6406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f6bfcb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f640c74f0 con 0x7f6f641547b0 2024-09-19T07:23:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f6bfcb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6f64073690 con 0x7f6f640c7d70 2024-09-19T07:23:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f6bfcb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6f64157160 con 0x7f6f64072d00 2024-09-19T07:23:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f69d67640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6f640c7d70 0x7f6f64157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f69d67640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6f640c7d70 0x7f6f64157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45082/0 (socket says 172.21.15.83:45082) 2024-09-19T07:23:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f69566640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6f641547b0 0x7f6f640c6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56312/0 (socket says 172.21.15.83:56312) 2024-09-19T07:23:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f69d67640 1 -- 172.21.15.83:0/1076566599 learned_addr learned my addr 172.21.15.83:0/1076566599 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f69d67640 1 -- 172.21.15.83:0/1076566599 >> v2:172.21.15.184:3300/0 conn(0x7f6f64072d00 msgr2=0x7f6f6406cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f69d67640 1 --2- 172.21.15.83:0/1076566599 >> v2:172.21.15.184:3300/0 conn(0x7f6f64072d00 0x7f6f6406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f69d67640 1 -- 172.21.15.83:0/1076566599 >> v1:172.21.15.83:6789/0 conn(0x7f6f641547b0 legacy=0x7f6f640c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.786+0000 7f6f69d67640 1 -- 172.21.15.83:0/1076566599 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f64157440 con 0x7f6f640c7d70 2024-09-19T07:23:36.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.790+0000 7f6f69d67640 1 --2- 172.21.15.83:0/1076566599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f640c7d70 0x7f6f64157c40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6f64071da0 tx=0x7f6f6001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ead0a584a519e62 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:36.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.790+0000 7f6f68d65640 1 -- 172.21.15.83:0/1076566599 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f600b1070 con 0x7f6f640c7d70 2024-09-19T07:23:36.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.790+0000 7f6f68d65640 1 -- 172.21.15.83:0/1076566599 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f600ac620 con 0x7f6f640c7d70 2024-09-19T07:23:36.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.790+0000 7f6f68d65640 1 -- 172.21.15.83:0/1076566599 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f600ac9f0 con 0x7f6f640c7d70 2024-09-19T07:23:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.790+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1076566599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f640c7d70 msgr2=0x7f6f64157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.790+0000 7f6f6bfcb640 1 --2- 172.21.15.83:0/1076566599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f640c7d70 0x7f6f64157c40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6f64071da0 tx=0x7f6f6001c460 comp rx=0 tx=0).stop 2024-09-19T07:23:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.790+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1076566599 shutdown_connections 2024-09-19T07:23:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.790+0000 7f6f6bfcb640 1 --2- 172.21.15.83:0/1076566599 >> v2:172.21.15.184:3300/0 conn(0x7f6f64072d00 0x7f6f6406cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.790+0000 7f6f6bfcb640 1 --2- 172.21.15.83:0/1076566599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f640c7d70 0x7f6f64157c40 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.790+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1076566599 >> 172.21.15.83:0/1076566599 conn(0x7f6f640c5060 msgr2=0x7f6f640c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.790+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1076566599 shutdown_connections 2024-09-19T07:23:36.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.790+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1076566599 wait complete. 2024-09-19T07:23:36.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f6bfcb640 1 Processor -- start 2024-09-19T07:23:36.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f6bfcb640 1 -- start start 2024-09-19T07:23:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f6bfcb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f64072d00 0x7f6f640a9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f6bfcb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f640c7d70 0x7f6f641eeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f6bfcb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f641547b0 0x7f6f640c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f6bfcb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f640703b0 con 0x7f6f64072d00 2024-09-19T07:23:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f6bfcb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6f64070230 con 0x7f6f641547b0 2024-09-19T07:23:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f6bfcb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6f64070530 con 0x7f6f640c7d70 2024-09-19T07:23:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f69d67640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f64072d00 0x7f6f640a9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f69d67640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f64072d00 0x7f6f640a9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45090/0 (socket says 172.21.15.83:45090) 2024-09-19T07:23:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f69d67640 1 -- 172.21.15.83:0/1961662960 learned_addr learned my addr 172.21.15.83:0/1961662960 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f69d67640 1 -- 172.21.15.83:0/1961662960 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f640c7d70 msgr2=0x7f6f641eeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f69d67640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f640c7d70 0x7f6f641eeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f69d67640 1 -- 172.21.15.83:0/1961662960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f641547b0 msgr2=0x7f6f640c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f6a568640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f641547b0 0x7f6f640c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f69d67640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f641547b0 0x7f6f640c1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f69d67640 1 -- 172.21.15.83:0/1961662960 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f641ef1b0 con 0x7f6f64072d00 2024-09-19T07:23:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f69d67640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f64072d00 0x7f6f640a9330 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6f600af040 tx=0x7f6f6001aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f5a7fc640 1 -- 172.21.15.83:0/1961662960 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f600bb020 con 0x7f6f64072d00 2024-09-19T07:23:36.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f5a7fc640 1 -- 172.21.15.83:0/1961662960 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f600b7620 con 0x7f6f64072d00 2024-09-19T07:23:36.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f5a7fc640 1 -- 172.21.15.83:0/1961662960 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f600b1040 con 0x7f6f64072d00 2024-09-19T07:23:36.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1961662960 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f641ef490 con 0x7f6f64072d00 2024-09-19T07:23:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1961662960 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f64154d50 con 0x7f6f64072d00 2024-09-19T07:23:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.794+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1961662960 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f6406cc90 con 0x7f6f64072d00 2024-09-19T07:23:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.798+0000 7f6f5a7fc640 1 -- 172.21.15.83:0/1961662960 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6f600be020 con 0x7f6f64072d00 2024-09-19T07:23:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.798+0000 7f6f5a7fc640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f480678f0 0x7f6f48069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.798+0000 7f6f5a7fc640 1 -- 172.21.15.83:0/1961662960 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7067+0+0 (secure 0 0 0) 0x7f6f601205e0 con 0x7f6f64072d00 2024-09-19T07:23:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.798+0000 7f6f69566640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f480678f0 0x7f6f48069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.798+0000 7f6f69566640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f480678f0 0x7f6f48069db0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6f54000f70 tx=0x7f6f54007a10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:36.802+0000 7f6f5a7fc640 1 -- 172.21.15.83:0/1961662960 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6f600c5030 con 0x7f6f64072d00 2024-09-19T07:23:37.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.002+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1961662960 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f6415ef90 con 0x7f6f64072d00 2024-09-19T07:23:37.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.002+0000 7f6f5a7fc640 1 -- 172.21.15.83:0/1961662960 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6539 (secure 0 0 0) 0x7f6f60016110 con 0x7f6f64072d00 2024-09-19T07:23:37.004 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:37.004 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":38,"btime":"2024-09-19T07:23:13:002854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5304,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:10.999287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":34,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/1049139637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1049139637},{"type":"v1","addr":"172.21.15.83:6837","nonce":1049139637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-09-19T07:23:37.004 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 38 2024-09-19T07:23:37.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.006+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1961662960 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f480678f0 msgr2=0x7f6f48069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:37.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.006+0000 7f6f6bfcb640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f480678f0 0x7f6f48069db0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6f54000f70 tx=0x7f6f54007a10 comp rx=0 tx=0).stop 2024-09-19T07:23:37.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.006+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1961662960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f64072d00 msgr2=0x7f6f640a9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:37.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.006+0000 7f6f6bfcb640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f64072d00 0x7f6f640a9330 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6f600af040 tx=0x7f6f6001aca0 comp rx=0 tx=0).stop 2024-09-19T07:23:37.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.006+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1961662960 shutdown_connections 2024-09-19T07:23:37.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.006+0000 7f6f6bfcb640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f480678f0 0x7f6f48069db0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:37.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.006+0000 7f6f6bfcb640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f641547b0 0x7f6f640c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:37.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.006+0000 7f6f6bfcb640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f640c7d70 0x7f6f641eeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:37.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.006+0000 7f6f6bfcb640 1 --2- 172.21.15.83:0/1961662960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f64072d00 0x7f6f640a9330 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:37.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.006+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1961662960 >> 172.21.15.83:0/1961662960 conn(0x7f6f640c5060 msgr2=0x7f6f6406da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:37.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.006+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1961662960 shutdown_connections 2024-09-19T07:23:37.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:37.006+0000 7f6f6bfcb640 1 -- 172.21.15.83:0/1961662960 wait complete. 2024-09-19T07:23:37.022 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:23:38.071 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.186+0000 7fe1ece9f640 1 Processor -- start 2024-09-19T07:23:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.186+0000 7fe1ece9f640 1 -- start start 2024-09-19T07:23:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.186+0000 7fe1ece9f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe1e8155d70 0x7fe1e80c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.186+0000 7fe1ece9f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1e81592f0 0x7fe1e8157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.186+0000 7fe1ece9f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1e8156200 con 0x7fe1e80c6120 2024-09-19T07:23:38.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.186+0000 7fe1ece9f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe1e80c9930 con 0x7fe1e81592f0 2024-09-19T07:23:38.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.186+0000 7fe1ece9f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe1e81564e0 con 0x7fe1e8155d70 2024-09-19T07:23:38.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.186+0000 7fe1e659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe1e80c6120 0x7fe1e81575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56326/0 (socket says 172.21.15.83:56326) 2024-09-19T07:23:38.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.186+0000 7fe1e659c640 1 -- 172.21.15.83:0/454627511 learned_addr learned my addr 172.21.15.83:0/454627511 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:38.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.186+0000 7fe1e6d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1e81592f0 0x7fe1e8157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.190+0000 7fe1e6d9d640 1 -- 172.21.15.83:0/454627511 >> v2:172.21.15.184:3300/0 conn(0x7fe1e8155d70 msgr2=0x7fe1e80c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.190+0000 7fe1e5d9b640 1 --2- 172.21.15.83:0/454627511 >> v2:172.21.15.184:3300/0 conn(0x7fe1e8155d70 0x7fe1e80c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.190+0000 7fe1e6d9d640 1 --2- 172.21.15.83:0/454627511 >> v2:172.21.15.184:3300/0 conn(0x7fe1e8155d70 0x7fe1e80c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.190+0000 7fe1e559a640 1 -- 172.21.15.83:0/454627511 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 553651311 0 0) 0x7fe1e8156200 con 0x7fe1e80c6120 2024-09-19T07:23:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.190+0000 7fe1e6d9d640 1 -- 172.21.15.83:0/454627511 >> v1:172.21.15.83:6789/0 conn(0x7fe1e80c6120 legacy=0x7fe1e81575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.190+0000 7fe1e6d9d640 1 -- 172.21.15.83:0/454627511 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e81567c0 con 0x7fe1e81592f0 2024-09-19T07:23:38.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.190+0000 7fe1e6d9d640 1 --2- 172.21.15.83:0/454627511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1e81592f0 0x7fe1e8157ce0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe1e806cca0 tx=0x7fe1d801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b425e03f74be8d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:38.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.190+0000 7fe1e559a640 1 -- 172.21.15.83:0/454627511 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80b0070 con 0x7fe1e81592f0 2024-09-19T07:23:38.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.190+0000 7fe1e559a640 1 -- 172.21.15.83:0/454627511 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1d80ab600 con 0x7fe1e81592f0 2024-09-19T07:23:38.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.190+0000 7fe1e559a640 1 -- 172.21.15.83:0/454627511 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80ab9d0 con 0x7fe1e81592f0 2024-09-19T07:23:38.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 -- 172.21.15.83:0/454627511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1e81592f0 msgr2=0x7fe1e8157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:38.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 --2- 172.21.15.83:0/454627511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1e81592f0 0x7fe1e8157ce0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe1e806cca0 tx=0x7fe1d801c480 comp rx=0 tx=0).stop 2024-09-19T07:23:38.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 -- 172.21.15.83:0/454627511 shutdown_connections 2024-09-19T07:23:38.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 --2- 172.21.15.83:0/454627511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1e81592f0 0x7fe1e8157ce0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 --2- 172.21.15.83:0/454627511 >> v2:172.21.15.184:3300/0 conn(0x7fe1e8155d70 0x7fe1e80c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 -- 172.21.15.83:0/454627511 >> 172.21.15.83:0/454627511 conn(0x7fe1e806bc80 msgr2=0x7fe1e806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:38.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 -- 172.21.15.83:0/454627511 shutdown_connections 2024-09-19T07:23:38.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 -- 172.21.15.83:0/454627511 wait complete. 2024-09-19T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 Processor -- start 2024-09-19T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 -- start start 2024-09-19T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe1e80c6120 0x7fe1e80ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1e81592f0 0x7fe1e8148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1e80abff0 0x7fe1e814e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e8153860 con 0x7fe1e80abff0 2024-09-19T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e8162ea0 con 0x7fe1e80c6120 2024-09-19T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe1e814b4a0 con 0x7fe1e81592f0 2024-09-19T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1e6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1e80abff0 0x7fe1e814e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1e6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1e80abff0 0x7fe1e814e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45116/0 (socket says 172.21.15.83:45116) 2024-09-19T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1e6d9d640 1 -- 172.21.15.83:0/225511185 learned_addr learned my addr 172.21.15.83:0/225511185 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1e659c640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe1e80c6120 0x7fe1e80ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1e5d9b640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1e81592f0 0x7fe1e8148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1e6d9d640 1 -- 172.21.15.83:0/225511185 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1e81592f0 msgr2=0x7fe1e8148f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1e6d9d640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1e81592f0 0x7fe1e8148f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1e6d9d640 1 -- 172.21.15.83:0/225511185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe1e80c6120 msgr2=0x7fe1e80ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1e6d9d640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe1e80c6120 0x7fe1e80ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1e6d9d640 1 -- 172.21.15.83:0/225511185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e80cb990 con 0x7fe1e80abff0 2024-09-19T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1e6d9d640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1e80abff0 0x7fe1e814e760 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe1d801a0b0 tx=0x7fe1d801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1deffd640 1 -- 172.21.15.83:0/225511185 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80bc070 con 0x7fe1e80abff0 2024-09-19T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1deffd640 1 -- 172.21.15.83:0/225511185 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1d80b1040 con 0x7fe1e80abff0 2024-09-19T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1ece9f640 1 -- 172.21.15.83:0/225511185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e80cbc70 con 0x7fe1e80abff0 2024-09-19T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.194+0000 7fe1deffd640 1 -- 172.21.15.83:0/225511185 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80b0050 con 0x7fe1e80abff0 2024-09-19T07:23:38.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.198+0000 7fe1ece9f640 1 -- 172.21.15.83:0/225511185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e814b130 con 0x7fe1e80abff0 2024-09-19T07:23:38.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.198+0000 7fe1ece9f640 1 -- 172.21.15.83:0/225511185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1e80c6500 con 0x7fe1e80abff0 2024-09-19T07:23:38.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.198+0000 7fe1deffd640 1 -- 172.21.15.83:0/225511185 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe1d80bf070 con 0x7fe1e80abff0 2024-09-19T07:23:38.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.198+0000 7fe1deffd640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe1c80678c0 0x7fe1c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.198+0000 7fe1e659c640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe1c80678c0 0x7fe1c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.198+0000 7fe1deffd640 1 -- 172.21.15.83:0/225511185 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7153+0+0 (secure 0 0 0) 0x7fe1d811fe70 con 0x7fe1e80abff0 2024-09-19T07:23:38.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.198+0000 7fe1e659c640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe1c80678c0 0x7fe1c8069d80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0007fd0 tx=0x7fe1e0007ee0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:38.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.202+0000 7fe1deffd640 1 -- 172.21.15.83:0/225511185 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe1d8016070 con 0x7fe1e80abff0 2024-09-19T07:23:38.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 -- 172.21.15.83:0/225511185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1e80c96a0 con 0x7fe1e80abff0 2024-09-19T07:23:38.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1deffd640 1 -- 172.21.15.83:0/225511185 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+5706 (secure 0 0 0) 0x7fe1d80f0490 con 0x7fe1e80abff0 2024-09-19T07:23:38.402 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:38.403 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":39,"btime":"2024-09-19T07:23:37:966484+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:37.966475+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:23:38.403 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 39 2024-09-19T07:23:38.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 -- 172.21.15.83:0/225511185 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe1c80678c0 msgr2=0x7fe1c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:38.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe1c80678c0 0x7fe1c8069d80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0007fd0 tx=0x7fe1e0007ee0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 -- 172.21.15.83:0/225511185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1e80abff0 msgr2=0x7fe1e814e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:38.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1e80abff0 0x7fe1e814e760 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe1d801a0b0 tx=0x7fe1d801acc0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 -- 172.21.15.83:0/225511185 shutdown_connections 2024-09-19T07:23:38.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe1c80678c0 0x7fe1c8069d80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1e80abff0 0x7fe1e814e760 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1e81592f0 0x7fe1e8148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 --2- 172.21.15.83:0/225511185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe1e80c6120 0x7fe1e80ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 -- 172.21.15.83:0/225511185 >> 172.21.15.83:0/225511185 conn(0x7fe1e806bc80 msgr2=0x7fe1e8153b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:38.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 -- 172.21.15.83:0/225511185 shutdown_connections 2024-09-19T07:23:38.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.402+0000 7fe1ece9f640 1 -- 172.21.15.83:0/225511185 wait complete. 2024-09-19T07:23:38.419 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-19T07:23:38.420 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:23:38.420 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:38.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1397eb6640 1 Processor -- start 2024-09-19T07:23:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1397eb6640 1 -- start start 2024-09-19T07:23:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1397eb6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f13900b0d00 0x7f1390159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1397eb6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13900c9660 0x7f13901563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1397eb6640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1390156910 0x7f13900c8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1397eb6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f13900c69b0 con 0x7f13900b0d00 2024-09-19T07:23:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1397eb6640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f13900c6180 con 0x7f1390156910 2024-09-19T07:23:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1397eb6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f139015a130 con 0x7f13900c9660 2024-09-19T07:23:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1395c52640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f13900b0d00 0x7f1390159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1395c52640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f13900b0d00 0x7f1390159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45132/0 (socket says 172.21.15.83:45132) 2024-09-19T07:23:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1395c52640 1 -- 172.21.15.83:0/3613855962 learned_addr learned my addr 172.21.15.83:0/3613855962 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1396453640 1 --2- 172.21.15.83:0/3613855962 >> v2:172.21.15.184:3300/0 conn(0x7f1390156910 0x7f13900c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1395451640 1 --2- 172.21.15.83:0/3613855962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13900c9660 0x7f13901563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1395c52640 1 -- 172.21.15.83:0/3613855962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13900c9660 msgr2=0x7f13901563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1395c52640 1 --2- 172.21.15.83:0/3613855962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13900c9660 0x7f13901563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1395c52640 1 -- 172.21.15.83:0/3613855962 >> v2:172.21.15.184:3300/0 conn(0x7f1390156910 msgr2=0x7f13900c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1395c52640 1 --2- 172.21.15.83:0/3613855962 >> v2:172.21.15.184:3300/0 conn(0x7f1390156910 0x7f13900c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1395c52640 1 -- 172.21.15.83:0/3613855962 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1390155bf0 con 0x7f13900b0d00 2024-09-19T07:23:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.526+0000 7f1396453640 1 --2- 172.21.15.83:0/3613855962 >> v2:172.21.15.184:3300/0 conn(0x7f1390156910 0x7f13900c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:23:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1395c52640 1 --2- 172.21.15.83:0/3613855962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13900b0d00 0x7f1390159bf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f13900c6f90 tx=0x7f138c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c56b7957103e69e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1394c50640 1 -- 172.21.15.83:0/3613855962 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c0b0070 con 0x7f13900b0d00 2024-09-19T07:23:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1394c50640 1 -- 172.21.15.83:0/3613855962 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f138c0ab600 con 0x7f13900b0d00 2024-09-19T07:23:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1394c50640 1 -- 172.21.15.83:0/3613855962 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c0ab9d0 con 0x7f13900b0d00 2024-09-19T07:23:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1397eb6640 1 -- 172.21.15.83:0/3613855962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13900b0d00 msgr2=0x7f1390159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1397eb6640 1 --2- 172.21.15.83:0/3613855962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13900b0d00 0x7f1390159bf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f13900c6f90 tx=0x7f138c01c490 comp rx=0 tx=0).stop 2024-09-19T07:23:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1397eb6640 1 -- 172.21.15.83:0/3613855962 shutdown_connections 2024-09-19T07:23:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1397eb6640 1 --2- 172.21.15.83:0/3613855962 >> v2:172.21.15.184:3300/0 conn(0x7f1390156910 0x7f13900c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1397eb6640 1 --2- 172.21.15.83:0/3613855962 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13900c9660 0x7f13901563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1397eb6640 1 --2- 172.21.15.83:0/3613855962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13900b0d00 0x7f1390159bf0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1397eb6640 1 -- 172.21.15.83:0/3613855962 >> 172.21.15.83:0/3613855962 conn(0x7f13900722a0 msgr2=0x7f13900726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1397eb6640 1 -- 172.21.15.83:0/3613855962 shutdown_connections 2024-09-19T07:23:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.530+0000 7f1397eb6640 1 -- 172.21.15.83:0/3613855962 wait complete. 2024-09-19T07:23:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1397eb6640 1 Processor -- start 2024-09-19T07:23:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1397eb6640 1 -- start start 2024-09-19T07:23:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1397eb6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13900b0d00 0x7f13900afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1397eb6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13900c9660 0x7f1390158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1397eb6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13900c2060 0x7f1390159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1397eb6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13900cbf00 con 0x7f13900c9660 2024-09-19T07:23:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1397eb6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13900b10e0 con 0x7f13900b0d00 2024-09-19T07:23:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1397eb6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f13900cc080 con 0x7f13900c2060 2024-09-19T07:23:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1395451640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13900c9660 0x7f1390158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1395451640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13900c9660 0x7f1390158c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45144/0 (socket says 172.21.15.83:45144) 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1395451640 1 -- 172.21.15.83:0/1001130869 learned_addr learned my addr 172.21.15.83:0/1001130869 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1395451640 1 -- 172.21.15.83:0/1001130869 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13900c2060 msgr2=0x7f1390159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1395c52640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13900b0d00 0x7f13900afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1396453640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13900c2060 0x7f1390159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1395451640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13900c2060 0x7f1390159180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1395451640 1 -- 172.21.15.83:0/1001130869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13900b0d00 msgr2=0x7f13900afe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1395451640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13900b0d00 0x7f13900afe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1395451640 1 -- 172.21.15.83:0/1001130869 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13901596c0 con 0x7f13900c9660 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1395451640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13900c9660 0x7f1390158c40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f138000e8b0 tx=0x7f138000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f13867fc640 1 -- 172.21.15.83:0/1001130869 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1380017070 con 0x7f13900c9660 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f13867fc640 1 -- 172.21.15.83:0/1001130869 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f138000c8e0 con 0x7f13900c9660 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1397eb6640 1 -- 172.21.15.83:0/1001130869 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13901f20c0 con 0x7f13900c9660 2024-09-19T07:23:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f13867fc640 1 -- 172.21.15.83:0/1001130869 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138000ccb0 con 0x7f13900c9660 2024-09-19T07:23:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1397eb6640 1 -- 172.21.15.83:0/1001130869 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13901eef40 con 0x7f13900c9660 2024-09-19T07:23:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f1397eb6640 1 -- 172.21.15.83:0/1001130869 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13900cc410 con 0x7f13900c9660 2024-09-19T07:23:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.534+0000 7f13867fc640 1 -- 172.21.15.83:0/1001130869 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f138001a070 con 0x7f13900c9660 2024-09-19T07:23:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.538+0000 7f13867fc640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13740678c0 0x7f1374069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.538+0000 7f13867fc640 1 -- 172.21.15.83:0/1001130869 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7153+0+0 (secure 0 0 0) 0x7f13800865d0 con 0x7f13900c9660 2024-09-19T07:23:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.538+0000 7f1395c52640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13740678c0 0x7f1374069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:38.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.538+0000 7f1395c52640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13740678c0 0x7f1374069d80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f138c0ad000 tx=0x7f138c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:38.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.542+0000 7f13867fc640 1 -- 172.21.15.83:0/1001130869 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1380056b60 con 0x7f13900c9660 2024-09-19T07:23:38.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 -- 172.21.15.83:0/1001130869 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1390155190 con 0x7f13900c9660 2024-09-19T07:23:38.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f13867fc640 1 -- 172.21.15.83:0/1001130869 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+5706 (secure 0 0 0) 0x7f1380056460 con 0x7f13900c9660 2024-09-19T07:23:38.722 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:38.723 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":39,"btime":"2024-09-19T07:23:37:966484+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:37.966475+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:23:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 39 2024-09-19T07:23:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 -- 172.21.15.83:0/1001130869 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13740678c0 msgr2=0x7f1374069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13740678c0 0x7f1374069d80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f138c0ad000 tx=0x7f138c008040 comp rx=0 tx=0).stop 2024-09-19T07:23:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 -- 172.21.15.83:0/1001130869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13900c9660 msgr2=0x7f1390158c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13900c9660 0x7f1390158c40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f138000e8b0 tx=0x7f138000ed70 comp rx=0 tx=0).stop 2024-09-19T07:23:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 -- 172.21.15.83:0/1001130869 shutdown_connections 2024-09-19T07:23:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13740678c0 0x7f1374069d80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13900c2060 0x7f1390159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13900c9660 0x7f1390158c40 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 --2- 172.21.15.83:0/1001130869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13900b0d00 0x7f13900afe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 -- 172.21.15.83:0/1001130869 >> 172.21.15.83:0/1001130869 conn(0x7f13900722a0 msgr2=0x7f1390152e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 -- 172.21.15.83:0/1001130869 shutdown_connections 2024-09-19T07:23:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:38.722+0000 7f1397eb6640 1 -- 172.21.15.83:0/1001130869 wait complete. 2024-09-19T07:23:38.740 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:23:37:966484+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 39, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4786": { "addr": "172.21.15.184:6835/4115191178", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 4115191178, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 4115191178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4786, "incarnation": 29, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:22:36.207732+0000", "qdb_cluster": [ 4786 ], "qdb_leader": 4786, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4786 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 39, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5304": { "addr": "172.21.15.83:6835/3442755673", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 3442755673, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 3442755673, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5304, "incarnation": 39, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:23:37.966475+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5304 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 28, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:23:39.023 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:39.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa497656640 1 Processor -- start 2024-09-19T07:23:39.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa497656640 1 -- start start 2024-09-19T07:23:39.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa497656640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa4900c6120 0x7fa4900c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:39.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa497656640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fa490155d70 0x7fa4901575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:39.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa497656640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa490156200 con 0x7fa4901592f0 2024-09-19T07:23:39.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa497656640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fa4900c9930 con 0x7fa490155d70 2024-09-19T07:23:39.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa497656640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa4901564e0 con 0x7fa4900c6120 2024-09-19T07:23:39.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa4953f2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa4900c6120 0x7fa4900c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:39.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa4953f2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa4900c6120 0x7fa4900c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:42040/0 (socket says 172.21.15.83:42040) 2024-09-19T07:23:39.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa4953f2640 1 -- 172.21.15.83:0/877878226 learned_addr learned my addr 172.21.15.83:0/877878226 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:39.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa494bf1640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fa490155d70 0x7fa4901575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:39.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa4953f2640 1 -- 172.21.15.83:0/877878226 >> v2:172.21.15.184:3300/0 conn(0x7fa490155d70 msgr2=0x7fa4901575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:39.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa4953f2640 1 --2- 172.21.15.83:0/877878226 >> v2:172.21.15.184:3300/0 conn(0x7fa490155d70 0x7fa4901575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:39.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa4953f2640 1 -- 172.21.15.83:0/877878226 >> v1:172.21.15.184:6789/0 conn(0x7fa4901592f0 legacy=0x7fa490157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:39.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.146+0000 7fa4953f2640 1 -- 172.21.15.83:0/877878226 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4901567c0 con 0x7fa4900c6120 2024-09-19T07:23:39.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa4953f2640 1 --2- 172.21.15.83:0/877878226 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa4900c6120 0x7fa4900c6500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa48c009870 tx=0x7fa48c01c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff3c89bbc1fedc4a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:39.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa483fff640 1 -- 172.21.15.83:0/877878226 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c0b2070 con 0x7fa4900c6120 2024-09-19T07:23:39.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa483fff640 1 -- 172.21.15.83:0/877878226 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa48c0a9070 con 0x7fa4900c6120 2024-09-19T07:23:39.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa483fff640 1 -- 172.21.15.83:0/877878226 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c0ad880 con 0x7fa4900c6120 2024-09-19T07:23:39.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 -- 172.21.15.83:0/877878226 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa4900c6120 msgr2=0x7fa4900c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:39.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 --2- 172.21.15.83:0/877878226 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa4900c6120 0x7fa4900c6500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa48c009870 tx=0x7fa48c01c560 comp rx=0 tx=0).stop 2024-09-19T07:23:39.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 -- 172.21.15.83:0/877878226 shutdown_connections 2024-09-19T07:23:39.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 --2- 172.21.15.83:0/877878226 >> v2:172.21.15.184:3300/0 conn(0x7fa490155d70 0x7fa4901575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:39.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 --2- 172.21.15.83:0/877878226 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa4900c6120 0x7fa4900c6500 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:39.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 -- 172.21.15.83:0/877878226 >> 172.21.15.83:0/877878226 conn(0x7fa49006bc80 msgr2=0x7fa49006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:39.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 -- 172.21.15.83:0/877878226 shutdown_connections 2024-09-19T07:23:39.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 -- 172.21.15.83:0/877878226 wait complete. 2024-09-19T07:23:39.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 Processor -- start 2024-09-19T07:23:39.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 -- start start 2024-09-19T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4900c6120 0x7fa49014cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4901592f0 0x7fa4900c0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa490150630 0x7fa4900abe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa490153860 con 0x7fa4900c6120 2024-09-19T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa490162ea0 con 0x7fa4901592f0 2024-09-19T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa497656640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa4900ae0d0 con 0x7fa490150630 2024-09-19T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa4953f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4900c6120 0x7fa49014cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa4953f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4900c6120 0x7fa49014cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45148/0 (socket says 172.21.15.83:45148) 2024-09-19T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa4953f2640 1 -- 172.21.15.83:0/9204763 learned_addr learned my addr 172.21.15.83:0/9204763 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa494bf1640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4901592f0 0x7fa4900c0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa495bf3640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa490150630 0x7fa4900abe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa4953f2640 1 -- 172.21.15.83:0/9204763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa490150630 msgr2=0x7fa4900abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa4953f2640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa490150630 0x7fa4900abe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa4953f2640 1 -- 172.21.15.83:0/9204763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4901592f0 msgr2=0x7fa4900c0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa4953f2640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4901592f0 0x7fa4900c0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa4953f2640 1 -- 172.21.15.83:0/9204763 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4900ab4e0 con 0x7fa4900c6120 2024-09-19T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.150+0000 7fa495bf3640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa490150630 0x7fa4900abe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.154+0000 7fa4953f2640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4900c6120 0x7fa49014cc60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0b0040 tx=0x7fa48c01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.154+0000 7fa481ffb640 1 -- 172.21.15.83:0/9204763 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c0bc020 con 0x7fa4900c6120 2024-09-19T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.154+0000 7fa481ffb640 1 -- 172.21.15.83:0/9204763 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa48c0adda0 con 0x7fa4900c6120 2024-09-19T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.154+0000 7fa497656640 1 -- 172.21.15.83:0/9204763 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4900add60 con 0x7fa4900c6120 2024-09-19T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.154+0000 7fa481ffb640 1 -- 172.21.15.83:0/9204763 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c0b2040 con 0x7fa4900c6120 2024-09-19T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.154+0000 7fa497656640 1 -- 172.21.15.83:0/9204763 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4900ac3d0 con 0x7fa4900c6120 2024-09-19T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.154+0000 7fa497656640 1 -- 172.21.15.83:0/9204763 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4900c6500 con 0x7fa4900c6120 2024-09-19T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.154+0000 7fa481ffb640 1 -- 172.21.15.83:0/9204763 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa48c0bf020 con 0x7fa4900c6120 2024-09-19T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.154+0000 7fa481ffb640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa4740678f0 0x7fa474069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.154+0000 7fa481ffb640 1 -- 172.21.15.83:0/9204763 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7153+0+0 (secure 0 0 0) 0x7fa48c120320 con 0x7fa4900c6120 2024-09-19T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.154+0000 7fa494bf1640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa4740678f0 0x7fa474069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.154+0000 7fa494bf1640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa4740678f0 0x7fa474069db0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa4900c2560 tx=0x7fa484009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:39.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.158+0000 7fa481ffb640 1 -- 172.21.15.83:0/9204763 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa48c0c6030 con 0x7fa4900c6120 2024-09-19T07:23:39.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.354+0000 7fa497656640 1 -- 172.21.15.83:0/9204763 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4900c96a0 con 0x7fa4900c6120 2024-09-19T07:23:39.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.354+0000 7fa481ffb640 1 -- 172.21.15.83:0/9204763 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+5709 (secure 0 0 0) 0x7fa48c016150 con 0x7fa4900c6120 2024-09-19T07:23:39.355 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:39.355 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":40,"btime":"2024-09-19T07:23:39:093338+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:38.228983+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:23:39.356 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 40 2024-09-19T07:23:39.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.358+0000 7fa497656640 1 -- 172.21.15.83:0/9204763 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa4740678f0 msgr2=0x7fa474069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:39.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.358+0000 7fa497656640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa4740678f0 0x7fa474069db0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa4900c2560 tx=0x7fa484009190 comp rx=0 tx=0).stop 2024-09-19T07:23:39.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.358+0000 7fa497656640 1 -- 172.21.15.83:0/9204763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4900c6120 msgr2=0x7fa49014cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:39.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.358+0000 7fa497656640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4900c6120 0x7fa49014cc60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0b0040 tx=0x7fa48c01ada0 comp rx=0 tx=0).stop 2024-09-19T07:23:39.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.358+0000 7fa497656640 1 -- 172.21.15.83:0/9204763 shutdown_connections 2024-09-19T07:23:39.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.358+0000 7fa497656640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa4740678f0 0x7fa474069db0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:39.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.358+0000 7fa497656640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa490150630 0x7fa4900abe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:39.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.358+0000 7fa497656640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4901592f0 0x7fa4900c0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:39.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.358+0000 7fa497656640 1 --2- 172.21.15.83:0/9204763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4900c6120 0x7fa49014cc60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:39.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.358+0000 7fa497656640 1 -- 172.21.15.83:0/9204763 >> 172.21.15.83:0/9204763 conn(0x7fa49006bc80 msgr2=0x7fa4901597d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:39.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.358+0000 7fa497656640 1 -- 172.21.15.83:0/9204763 shutdown_connections 2024-09-19T07:23:39.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:39.358+0000 7fa497656640 1 -- 172.21.15.83:0/9204763 wait complete. 2024-09-19T07:23:39.372 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:23:40.743 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:40.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc6109bb640 1 Processor -- start 2024-09-19T07:23:40.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc6109bb640 1 -- start start 2024-09-19T07:23:40.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc6109bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc60c152b20 0x7fc60c0c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:40.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc6109bb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc60c0c45c0 0x7fc60c153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:40.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc6109bb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc60c150670 con 0x7fc60c150290 2024-09-19T07:23:40.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc6109bb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc60c0716c0 con 0x7fc60c0c45c0 2024-09-19T07:23:40.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc6109bb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc60c0c3920 con 0x7fc60c152b20 2024-09-19T07:23:40.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc60a59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc60c150290 0x7fc60c0c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33864/0 (socket says 172.21.15.83:33864) 2024-09-19T07:23:40.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc60ad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc60c0c45c0 0x7fc60c153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:40.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc60a59c640 1 -- 172.21.15.83:0/3728969981 learned_addr learned my addr 172.21.15.83:0/3728969981 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:40.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc609d9b640 1 --2- 172.21.15.83:0/3728969981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc60c152b20 0x7fc60c0c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:40.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc60ad9d640 1 -- 172.21.15.83:0/3728969981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc60c152b20 msgr2=0x7fc60c0c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc60ad9d640 1 --2- 172.21.15.83:0/3728969981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc60c152b20 0x7fc60c0c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc60959a640 1 -- 172.21.15.83:0/3728969981 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3383619091 0 0) 0x7fc60c150670 con 0x7fc60c150290 2024-09-19T07:23:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc60ad9d640 1 -- 172.21.15.83:0/3728969981 >> v1:172.21.15.83:6789/0 conn(0x7fc60c150290 legacy=0x7fc60c0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc60ad9d640 1 -- 172.21.15.83:0/3728969981 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60c0c2880 con 0x7fc60c0c45c0 2024-09-19T07:23:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.870+0000 7fc609d9b640 1 --2- 172.21.15.83:0/3728969981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc60c152b20 0x7fc60c0c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.874+0000 7fc60ad9d640 1 --2- 172.21.15.83:0/3728969981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc60c0c45c0 0x7fc60c153eb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc604009dc0 tx=0x7fc60401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3151a79a8a5f380c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.874+0000 7fc60959a640 1 -- 172.21.15.83:0/3728969981 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6040b2070 con 0x7fc60c0c45c0 2024-09-19T07:23:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.874+0000 7fc60959a640 1 -- 172.21.15.83:0/3728969981 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6040a5900 con 0x7fc60c0c45c0 2024-09-19T07:23:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.874+0000 7fc60959a640 1 -- 172.21.15.83:0/3728969981 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6040a5cd0 con 0x7fc60c0c45c0 2024-09-19T07:23:40.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.874+0000 7fc6109bb640 1 -- 172.21.15.83:0/3728969981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc60c0c45c0 msgr2=0x7fc60c153eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:40.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.874+0000 7fc6109bb640 1 --2- 172.21.15.83:0/3728969981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc60c0c45c0 0x7fc60c153eb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc604009dc0 tx=0x7fc60401c950 comp rx=0 tx=0).stop 2024-09-19T07:23:40.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.874+0000 7fc6109bb640 1 -- 172.21.15.83:0/3728969981 shutdown_connections 2024-09-19T07:23:40.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.874+0000 7fc6109bb640 1 --2- 172.21.15.83:0/3728969981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc60c0c45c0 0x7fc60c153eb0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:40.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.874+0000 7fc6109bb640 1 --2- 172.21.15.83:0/3728969981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc60c152b20 0x7fc60c0c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:40.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.874+0000 7fc6109bb640 1 -- 172.21.15.83:0/3728969981 >> 172.21.15.83:0/3728969981 conn(0x7fc60c06bc80 msgr2=0x7fc60c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:40.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.874+0000 7fc6109bb640 1 -- 172.21.15.83:0/3728969981 shutdown_connections 2024-09-19T07:23:40.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.874+0000 7fc6109bb640 1 -- 172.21.15.83:0/3728969981 wait complete. 2024-09-19T07:23:40.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc6109bb640 1 Processor -- start 2024-09-19T07:23:40.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc6109bb640 1 -- start start 2024-09-19T07:23:40.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc6109bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc60c0c45c0 0x7fc60c0bd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:40.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc6109bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc60c150290 0x7fc60c14e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:40.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc6109bb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc60c152b20 0x7fc60c1498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:40.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc6109bb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c154980 con 0x7fc60c0c45c0 2024-09-19T07:23:40.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc6109bb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c154800 con 0x7fc60c152b20 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc6109bb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc60c15fb60 con 0x7fc60c150290 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc60a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc60c0c45c0 0x7fc60c0bd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc60a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc60c0c45c0 0x7fc60c0bd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38544/0 (socket says 172.21.15.83:38544) 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc60a59c640 1 -- 172.21.15.83:0/795937805 learned_addr learned my addr 172.21.15.83:0/795937805 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc609d9b640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc60c150290 0x7fc60c14e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc60a59c640 1 -- 172.21.15.83:0/795937805 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc60c150290 msgr2=0x7fc60c14e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc60a59c640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc60c150290 0x7fc60c14e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc60a59c640 1 -- 172.21.15.83:0/795937805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc60c152b20 msgr2=0x7fc60c1498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc60a59c640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc60c152b20 0x7fc60c1498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc60a59c640 1 -- 172.21.15.83:0/795937805 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60c073e00 con 0x7fc60c0c45c0 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc60a59c640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc60c0c45c0 0x7fc60c0bd530 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0011f0 tx=0x7fc5fc00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc5faffd640 1 -- 172.21.15.83:0/795937805 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc018070 con 0x7fc60c0c45c0 2024-09-19T07:23:40.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc5faffd640 1 -- 172.21.15.83:0/795937805 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5fc0074e0 con 0x7fc60c0c45c0 2024-09-19T07:23:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc6109bb640 1 -- 172.21.15.83:0/795937805 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc60c0bc7b0 con 0x7fc60c0c45c0 2024-09-19T07:23:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc5faffd640 1 -- 172.21.15.83:0/795937805 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc004a10 con 0x7fc60c0c45c0 2024-09-19T07:23:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc6109bb640 1 -- 172.21.15.83:0/795937805 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc60c1ecd60 con 0x7fc60c0c45c0 2024-09-19T07:23:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc6109bb640 1 -- 172.21.15.83:0/795937805 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60c0bc980 con 0x7fc60c0c45c0 2024-09-19T07:23:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc5faffd640 1 -- 172.21.15.83:0/795937805 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc5fc01b070 con 0x7fc60c0c45c0 2024-09-19T07:23:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc5faffd640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5ec0678c0 0x7fc5ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc5faffd640 1 -- 172.21.15.83:0/795937805 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7153+0+0 (secure 0 0 0) 0x7fc5fc085510 con 0x7fc60c0c45c0 2024-09-19T07:23:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.878+0000 7fc609d9b640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5ec0678c0 0x7fc5ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:40.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.882+0000 7fc609d9b640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5ec0678c0 0x7fc5ec069d80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc600006fd0 tx=0x7fc600008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:40.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:40.886+0000 7fc5faffd640 1 -- 172.21.15.83:0/795937805 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc5fc055b30 con 0x7fc60c0c45c0 2024-09-19T07:23:41.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.082+0000 7fc6109bb640 1 -- 172.21.15.83:0/795937805 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc60c153eb0 con 0x7fc60c0c45c0 2024-09-19T07:23:41.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.082+0000 7fc5faffd640 1 -- 172.21.15.83:0/795937805 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+5706 (secure 0 0 0) 0x7fc5fc055430 con 0x7fc60c0c45c0 2024-09-19T07:23:41.083 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:41.084 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":41,"btime":"2024-09-19T07:23:40:102372+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:39.108158+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:23:41.084 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 41 2024-09-19T07:23:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.086+0000 7fc6109bb640 1 -- 172.21.15.83:0/795937805 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5ec0678c0 msgr2=0x7fc5ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.086+0000 7fc6109bb640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5ec0678c0 0x7fc5ec069d80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc600006fd0 tx=0x7fc600008040 comp rx=0 tx=0).stop 2024-09-19T07:23:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.086+0000 7fc6109bb640 1 -- 172.21.15.83:0/795937805 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc60c0c45c0 msgr2=0x7fc60c0bd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.086+0000 7fc6109bb640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc60c0c45c0 0x7fc60c0bd530 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0011f0 tx=0x7fc5fc00c9b0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.086+0000 7fc6109bb640 1 -- 172.21.15.83:0/795937805 shutdown_connections 2024-09-19T07:23:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.086+0000 7fc6109bb640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5ec0678c0 0x7fc5ec069d80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.086+0000 7fc6109bb640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc60c152b20 0x7fc60c1498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.086+0000 7fc6109bb640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc60c150290 0x7fc60c14e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.086+0000 7fc6109bb640 1 --2- 172.21.15.83:0/795937805 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc60c0c45c0 0x7fc60c0bd530 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.086+0000 7fc6109bb640 1 -- 172.21.15.83:0/795937805 >> 172.21.15.83:0/795937805 conn(0x7fc60c06bc80 msgr2=0x7fc60c14bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:41.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.086+0000 7fc6109bb640 1 -- 172.21.15.83:0/795937805 shutdown_connections 2024-09-19T07:23:41.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.086+0000 7fc6109bb640 1 -- 172.21.15.83:0/795937805 wait complete. 2024-09-19T07:23:41.102 INFO:tasks.mds_thrash.fs.[b]:no change 2024-09-19T07:23:41.373 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:41.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f609e7c8640 1 Processor -- start 2024-09-19T07:23:41.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f609e7c8640 1 -- start start 2024-09-19T07:23:41.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f609e7c8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60980c28e0 0x7f60981545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f609e7c8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f609806dea0 0x7f609806e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f609e7c8640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60980a9e60 con 0x7f6098071100 2024-09-19T07:23:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f609e7c8640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6098071b50 con 0x7f60980c28e0 2024-09-19T07:23:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f609e7c8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f60980c1ff0 con 0x7f609806dea0 2024-09-19T07:23:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f60977fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60980c28e0 0x7f60981545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f60977fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60980c28e0 0x7f60981545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38548/0 (socket says 172.21.15.83:38548) 2024-09-19T07:23:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f60977fe640 1 -- 172.21.15.83:0/2401230115 learned_addr learned my addr 172.21.15.83:0/2401230115 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f609cd65640 1 --2- 172.21.15.83:0/2401230115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f609806dea0 0x7f609806e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f60977fe640 1 -- 172.21.15.83:0/2401230115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f609806dea0 msgr2=0x7f609806e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f60977fe640 1 --2- 172.21.15.83:0/2401230115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f609806dea0 0x7f609806e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f60977fe640 1 -- 172.21.15.83:0/2401230115 >> v1:172.21.15.184:6789/0 conn(0x7f6098071100 legacy=0x7f6098150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f60977fe640 1 -- 172.21.15.83:0/2401230115 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60980aa140 con 0x7f60980c28e0 2024-09-19T07:23:41.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.506+0000 7f60977fe640 1 --2- 172.21.15.83:0/2401230115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60980c28e0 0x7f60981545e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6090009e50 tx=0x7f609001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d809215f1ebab08c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:41.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6096ffd640 1 -- 172.21.15.83:0/2401230115 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60900a5750 con 0x7f60980c28e0 2024-09-19T07:23:41.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6096ffd640 1 -- 172.21.15.83:0/2401230115 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60900a58f0 con 0x7f60980c28e0 2024-09-19T07:23:41.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6096ffd640 1 -- 172.21.15.83:0/2401230115 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60900a5cc0 con 0x7f60980c28e0 2024-09-19T07:23:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 -- 172.21.15.83:0/2401230115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60980c28e0 msgr2=0x7f60981545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 --2- 172.21.15.83:0/2401230115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60980c28e0 0x7f60981545e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6090009e50 tx=0x7f609001c9a0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 -- 172.21.15.83:0/2401230115 shutdown_connections 2024-09-19T07:23:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 --2- 172.21.15.83:0/2401230115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f609806dea0 0x7f609806e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 --2- 172.21.15.83:0/2401230115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60980c28e0 0x7f60981545e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 -- 172.21.15.83:0/2401230115 >> 172.21.15.83:0/2401230115 conn(0x7f609806bc80 msgr2=0x7f609806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:41.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 -- 172.21.15.83:0/2401230115 shutdown_connections 2024-09-19T07:23:41.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 -- 172.21.15.83:0/2401230115 wait complete. 2024-09-19T07:23:41.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 Processor -- start 2024-09-19T07:23:41.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 -- start start 2024-09-19T07:23:41.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f609806dea0 0x7f6098146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:41.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6098071100 0x7f6098147960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:41.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60980c28e0 0x7f6098153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:41.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6098071660 con 0x7f609806dea0 2024-09-19T07:23:41.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f60980714e0 con 0x7f60980c28e0 2024-09-19T07:23:41.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6098160c60 con 0x7f6098071100 2024-09-19T07:23:41.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6097fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f609806dea0 0x7f6098146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:41.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6097fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f609806dea0 0x7f6098146ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38550/0 (socket says 172.21.15.83:38550) 2024-09-19T07:23:41.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6097fff640 1 -- 172.21.15.83:0/346074919 learned_addr learned my addr 172.21.15.83:0/346074919 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:41.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609cd65640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60980c28e0 0x7f6098153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:41.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f60977fe640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6098071100 0x7f6098147960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:41.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6097fff640 1 -- 172.21.15.83:0/346074919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6098071100 msgr2=0x7f6098147960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:41.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6097fff640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6098071100 0x7f6098147960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6097fff640 1 -- 172.21.15.83:0/346074919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60980c28e0 msgr2=0x7f6098153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:41.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6097fff640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60980c28e0 0x7f6098153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6097fff640 1 -- 172.21.15.83:0/346074919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60981ed390 con 0x7f609806dea0 2024-09-19T07:23:41.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6097fff640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f609806dea0 0x7f6098146ea0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f608000c420 tx=0x7f608000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:41.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6094ff9640 1 -- 172.21.15.83:0/346074919 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60800052d0 con 0x7f609806dea0 2024-09-19T07:23:41.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6094ff9640 1 -- 172.21.15.83:0/346074919 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6080005470 con 0x7f609806dea0 2024-09-19T07:23:41.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 -- 172.21.15.83:0/346074919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60980c5520 con 0x7f609806dea0 2024-09-19T07:23:41.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f6094ff9640 1 -- 172.21.15.83:0/346074919 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080007260 con 0x7f609806dea0 2024-09-19T07:23:41.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.510+0000 7f609e7c8640 1 -- 172.21.15.83:0/346074919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6098148450 con 0x7f609806dea0 2024-09-19T07:23:41.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.514+0000 7f609e7c8640 1 -- 172.21.15.83:0/346074919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60981ecee0 con 0x7f609806dea0 2024-09-19T07:23:41.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.514+0000 7f6094ff9640 1 -- 172.21.15.83:0/346074919 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6080007400 con 0x7f609806dea0 2024-09-19T07:23:41.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.514+0000 7f6094ff9640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60780678c0 0x7f6078069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:41.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.514+0000 7f6094ff9640 1 -- 172.21.15.83:0/346074919 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7153+0+0 (secure 0 0 0) 0x7f60800856f0 con 0x7f609806dea0 2024-09-19T07:23:41.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.514+0000 7f60977fe640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60780678c0 0x7f6078069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:41.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.514+0000 7f60977fe640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60780678c0 0x7f6078069d80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f60900ae000 tx=0x7f60900a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:41.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.518+0000 7f6094ff9640 1 -- 172.21.15.83:0/346074919 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6080055d10 con 0x7f609806dea0 2024-09-19T07:23:41.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.714+0000 7f609e7c8640 1 -- 172.21.15.83:0/346074919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f609806e330 con 0x7f609806dea0 2024-09-19T07:23:41.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.714+0000 7f6094ff9640 1 -- 172.21.15.83:0/346074919 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5714 (secure 0 0 0) 0x7f6080055610 con 0x7f609806dea0 2024-09-19T07:23:41.717 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:41.717 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":42,"btime":"2024-09-19T07:23:41:116081+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:41.717 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 42 2024-09-19T07:23:41.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.718+0000 7f609e7c8640 1 -- 172.21.15.83:0/346074919 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60780678c0 msgr2=0x7f6078069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:41.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.718+0000 7f609e7c8640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60780678c0 0x7f6078069d80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f60900ae000 tx=0x7f60900a7070 comp rx=0 tx=0).stop 2024-09-19T07:23:41.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.718+0000 7f609e7c8640 1 -- 172.21.15.83:0/346074919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f609806dea0 msgr2=0x7f6098146ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:41.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.718+0000 7f609e7c8640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f609806dea0 0x7f6098146ea0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f608000c420 tx=0x7f608000c8e0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.718+0000 7f609e7c8640 1 -- 172.21.15.83:0/346074919 shutdown_connections 2024-09-19T07:23:41.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.718+0000 7f609e7c8640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60780678c0 0x7f6078069d80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.718+0000 7f609e7c8640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60980c28e0 0x7f6098153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.718+0000 7f609e7c8640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6098071100 0x7f6098147960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.718+0000 7f609e7c8640 1 --2- 172.21.15.83:0/346074919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f609806dea0 0x7f6098146ea0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:41.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.718+0000 7f609e7c8640 1 -- 172.21.15.83:0/346074919 >> 172.21.15.83:0/346074919 conn(0x7f609806bc80 msgr2=0x7f609806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:41.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.718+0000 7f609e7c8640 1 -- 172.21.15.83:0/346074919 shutdown_connections 2024-09-19T07:23:41.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:41.718+0000 7f609e7c8640 1 -- 172.21.15.83:0/346074919 wait complete. 2024-09-19T07:23:41.736 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:23:43.017 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:23:43.014+0000 7ff6d00ff640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:23:43.104 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:43.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.234+0000 7fa6b70ad640 1 Processor -- start 2024-09-19T07:23:43.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.234+0000 7fa6b70ad640 1 -- start start 2024-09-19T07:23:43.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.234+0000 7fa6b70ad640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa6b01545e0 0x7fa6b006e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.234+0000 7fa6b70ad640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6b0154a70 con 0x7fa6b00c28e0 2024-09-19T07:23:43.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.234+0000 7fa6b70ad640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6b00a9e60 con 0x7fa6b0071100 2024-09-19T07:23:43.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.234+0000 7fa6b70ad640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa6b0071b50 con 0x7fa6b01545e0 2024-09-19T07:23:43.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.234+0000 7fa6b564a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa6b01545e0 0x7fa6b006e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.234+0000 7fa6affff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa6b00c28e0 0x7fa6b006dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33880/0 (socket says 172.21.15.83:33880) 2024-09-19T07:23:43.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.234+0000 7fa6affff640 1 -- 172.21.15.83:0/967229806 learned_addr learned my addr 172.21.15.83:0/967229806 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:43.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6b564a640 1 -- 172.21.15.83:0/967229806 >> v1:172.21.15.184:6789/0 conn(0x7fa6b0071100 legacy=0x7fa6b0150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:43.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6b564a640 1 -- 172.21.15.83:0/967229806 >> v1:172.21.15.83:6789/0 conn(0x7fa6b00c28e0 legacy=0x7fa6b006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:43.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6af7fe640 1 -- 172.21.15.83:0/967229806 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1127482998 0 0) 0x7fa6b0154a70 con 0x7fa6b00c28e0 2024-09-19T07:23:43.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6b564a640 1 -- 172.21.15.83:0/967229806 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b00aa140 con 0x7fa6b01545e0 2024-09-19T07:23:43.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6b564a640 1 --2- 172.21.15.83:0/967229806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa6b01545e0 0x7fa6b006e5b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa6a80097c0 tx=0x7fa6a801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db95a5a70d0cdcd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:43.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6af7fe640 1 -- 172.21.15.83:0/967229806 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a80b0070 con 0x7fa6b01545e0 2024-09-19T07:23:43.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6af7fe640 1 -- 172.21.15.83:0/967229806 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6a80ab600 con 0x7fa6b01545e0 2024-09-19T07:23:43.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6af7fe640 1 -- 172.21.15.83:0/967229806 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a80ab9d0 con 0x7fa6b01545e0 2024-09-19T07:23:43.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6b70ad640 1 -- 172.21.15.83:0/967229806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa6b01545e0 msgr2=0x7fa6b006e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:43.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6b70ad640 1 --2- 172.21.15.83:0/967229806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa6b01545e0 0x7fa6b006e5b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa6a80097c0 tx=0x7fa6a801c410 comp rx=0 tx=0).stop 2024-09-19T07:23:43.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6b70ad640 1 -- 172.21.15.83:0/967229806 shutdown_connections 2024-09-19T07:23:43.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6b70ad640 1 --2- 172.21.15.83:0/967229806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa6b01545e0 0x7fa6b006e5b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.238+0000 7fa6b70ad640 1 -- 172.21.15.83:0/967229806 >> 172.21.15.83:0/967229806 conn(0x7fa6b006bc80 msgr2=0x7fa6b006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:43.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 -- 172.21.15.83:0/967229806 shutdown_connections 2024-09-19T07:23:43.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 -- 172.21.15.83:0/967229806 wait complete. 2024-09-19T07:23:43.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 Processor -- start 2024-09-19T07:23:43.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 -- start start 2024-09-19T07:23:43.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa6b0071100 0x7fa6b0140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6b00c28e0 0x7fa6b0150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa6b01545e0 0x7fa6b01505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b01517d0 con 0x7fa6b0071100 2024-09-19T07:23:43.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b0151650 con 0x7fa6b00c28e0 2024-09-19T07:23:43.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa6b00714e0 con 0x7fa6b01545e0 2024-09-19T07:23:43.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b4e49640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa6b0071100 0x7fa6b0140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b4e49640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa6b0071100 0x7fa6b0140cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38574/0 (socket says 172.21.15.83:38574) 2024-09-19T07:23:43.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b4e49640 1 -- 172.21.15.83:0/952761538 learned_addr learned my addr 172.21.15.83:0/952761538 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:43.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b564a640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa6b01545e0 0x7fa6b01505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6affff640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6b00c28e0 0x7fa6b0150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b4e49640 1 -- 172.21.15.83:0/952761538 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa6b01545e0 msgr2=0x7fa6b01505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:43.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b4e49640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa6b01545e0 0x7fa6b01505d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b4e49640 1 -- 172.21.15.83:0/952761538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6b00c28e0 msgr2=0x7fa6b0150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:43.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b4e49640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6b00c28e0 0x7fa6b0150090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b4e49640 1 -- 172.21.15.83:0/952761538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b014b3f0 con 0x7fa6b0071100 2024-09-19T07:23:43.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6affff640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6b00c28e0 0x7fa6b0150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:43.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b4e49640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa6b0071100 0x7fa6b0140cf0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000c7d0 tx=0x7fa6a000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:43.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6ad7fa640 1 -- 172.21.15.83:0/952761538 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0003b80 con 0x7fa6b0071100 2024-09-19T07:23:43.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6ad7fa640 1 -- 172.21.15.83:0/952761538 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6a0003d20 con 0x7fa6b0071100 2024-09-19T07:23:43.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 -- 172.21.15.83:0/952761538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6b014ec90 con 0x7fa6b0071100 2024-09-19T07:23:43.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6ad7fa640 1 -- 172.21.15.83:0/952761538 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a000aa90 con 0x7fa6b0071100 2024-09-19T07:23:43.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 -- 172.21.15.83:0/952761538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6b00c5eb0 con 0x7fa6b0071100 2024-09-19T07:23:43.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.242+0000 7fa6b70ad640 1 -- 172.21.15.83:0/952761538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6b0150f40 con 0x7fa6b0071100 2024-09-19T07:23:43.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.246+0000 7fa6ad7fa640 1 -- 172.21.15.83:0/952761538 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa6a000ac30 con 0x7fa6b0071100 2024-09-19T07:23:43.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.246+0000 7fa6ad7fa640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa69c0678f0 0x7fa69c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.246+0000 7fa6ad7fa640 1 -- 172.21.15.83:0/952761538 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7153+0+0 (secure 0 0 0) 0x7fa6a00860e0 con 0x7fa6b0071100 2024-09-19T07:23:43.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.246+0000 7fa6affff640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa69c0678f0 0x7fa69c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.246+0000 7fa6affff640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa69c0678f0 0x7fa69c069db0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0150bc0 tx=0x7fa6a4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:43.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.250+0000 7fa6ad7fa640 1 -- 172.21.15.83:0/952761538 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa6a0056700 con 0x7fa6b0071100 2024-09-19T07:23:43.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.446+0000 7fa6b70ad640 1 -- 172.21.15.83:0/952761538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6b00c5980 con 0x7fa6b0071100 2024-09-19T07:23:43.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.446+0000 7fa6ad7fa640 1 -- 172.21.15.83:0/952761538 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5714 (secure 0 0 0) 0x7fa6a0056000 con 0x7fa6b0071100 2024-09-19T07:23:43.447 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:43.447 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":42,"btime":"2024-09-19T07:23:41:116081+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:43.447 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 42 2024-09-19T07:23:43.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.446+0000 7fa6b70ad640 1 -- 172.21.15.83:0/952761538 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa69c0678f0 msgr2=0x7fa69c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:43.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.446+0000 7fa6b70ad640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa69c0678f0 0x7fa69c069db0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0150bc0 tx=0x7fa6a4009040 comp rx=0 tx=0).stop 2024-09-19T07:23:43.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.446+0000 7fa6b70ad640 1 -- 172.21.15.83:0/952761538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa6b0071100 msgr2=0x7fa6b0140cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:43.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.450+0000 7fa6b70ad640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa6b0071100 0x7fa6b0140cf0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000c7d0 tx=0x7fa6a000cc90 comp rx=0 tx=0).stop 2024-09-19T07:23:43.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.450+0000 7fa6b70ad640 1 -- 172.21.15.83:0/952761538 shutdown_connections 2024-09-19T07:23:43.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.450+0000 7fa6b70ad640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa69c0678f0 0x7fa69c069db0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.450+0000 7fa6b70ad640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa6b01545e0 0x7fa6b01505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.450+0000 7fa6b70ad640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6b00c28e0 0x7fa6b0150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.450+0000 7fa6b70ad640 1 --2- 172.21.15.83:0/952761538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa6b0071100 0x7fa6b0140cf0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.450+0000 7fa6b70ad640 1 -- 172.21.15.83:0/952761538 >> 172.21.15.83:0/952761538 conn(0x7fa6b006bc80 msgr2=0x7fa6b0147780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:43.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.450+0000 7fa6b70ad640 1 -- 172.21.15.83:0/952761538 shutdown_connections 2024-09-19T07:23:43.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.450+0000 7fa6b70ad640 1 -- 172.21.15.83:0/952761538 wait complete. 2024-09-19T07:23:43.466 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5196 2024-09-19T07:23:43.466 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.c 2024-09-19T07:23:43.737 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.866+0000 7f355c1de640 1 Processor -- start 2024-09-19T07:23:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.866+0000 7f355c1de640 1 -- start start 2024-09-19T07:23:43.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f355c1de640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f35540c42e0 0x7f355406e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f355c1de640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35541566b0 0x7f35540abb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f355c1de640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35541592c0 con 0x7f3554159a80 2024-09-19T07:23:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f355c1de640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f35540ac070 con 0x7f35540c42e0 2024-09-19T07:23:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f355c1de640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f35541595f0 con 0x7f35541566b0 2024-09-19T07:23:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f355a77b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3554159a80 0x7f3554153060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33896/0 (socket says 172.21.15.83:33896) 2024-09-19T07:23:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f355a77b640 1 -- 172.21.15.83:0/3203033537 learned_addr learned my addr 172.21.15.83:0/3203033537 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3559f7a640 1 --2- 172.21.15.83:0/3203033537 >> v2:172.21.15.184:3300/0 conn(0x7f35540c42e0 0x7f355406e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3559779640 1 --2- 172.21.15.83:0/3203033537 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35541566b0 0x7f35540abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1456941039 0 0) 0x7f35541592c0 con 0x7f3554159a80 2024-09-19T07:23:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3540003530 con 0x7f3554159a80 2024-09-19T07:23:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1326151801 0 0) 0x7f3540003530 con 0x7f3554159a80 2024-09-19T07:23:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35541566b0 msgr2=0x7f35540abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 --2- 172.21.15.83:0/3203033537 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35541566b0 0x7f35540abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 >> v2:172.21.15.184:3300/0 conn(0x7f35540c42e0 msgr2=0x7f355406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 --2- 172.21.15.83:0/3203033537 >> v2:172.21.15.184:3300/0 conn(0x7f35540c42e0 0x7f355406e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355406e900 con 0x7f3554159a80 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f354c001a00 con 0x7f3554159a80 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 >> v1:172.21.15.83:6789/0 conn(0x7f3554159a80 legacy=0x7f3554153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 --2- 172.21.15.83:0/3203033537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35400039c0 0x7f3540003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 --2- 172.21.15.83:0/3203033537 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35541566b0 0x7f3540004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 --2- 172.21.15.83:0/3203033537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3540004d50 0x7f35400051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f35541595f0 con 0x7f35400039c0 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f354000b9d0 con 0x7f3540004d50 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f354000bb50 con 0x7f35541566b0 2024-09-19T07:23:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3559f7a640 1 --2- 172.21.15.83:0/3203033537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35400039c0 0x7f3540003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3559779640 1 --2- 172.21.15.83:0/3203033537 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35541566b0 0x7f3540004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3559f7a640 1 -- 172.21.15.83:0/3203033537 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35541566b0 msgr2=0x7f3540004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3559f7a640 1 --2- 172.21.15.83:0/3203033537 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35541566b0 0x7f3540004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3559f7a640 1 -- 172.21.15.83:0/3203033537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3540004d50 msgr2=0x7f35400051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3559f7a640 1 --2- 172.21.15.83:0/3203033537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3540004d50 0x7f35400051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.870+0000 7f3559f7a640 1 -- 172.21.15.83:0/3203033537 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3540003530 con 0x7f35400039c0 2024-09-19T07:23:43.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f3559f7a640 1 --2- 172.21.15.83:0/3203033537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35400039c0 0x7f3540003de0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3544009b10 tx=0x7f354401ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31274f15d2d5d414 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:43.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35440b2070 con 0x7f35400039c0 2024-09-19T07:23:43.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35440a5560 con 0x7f35400039c0 2024-09-19T07:23:43.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f3558f78640 1 -- 172.21.15.83:0/3203033537 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35440096b0 con 0x7f35400039c0 2024-09-19T07:23:43.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 -- 172.21.15.83:0/3203033537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35400039c0 msgr2=0x7f3540003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:43.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 --2- 172.21.15.83:0/3203033537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35400039c0 0x7f3540003de0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3544009b10 tx=0x7f354401ca80 comp rx=0 tx=0).stop 2024-09-19T07:23:43.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355a77b640 1 -- 172.21.15.83:0/3203033537 reap_dead start 2024-09-19T07:23:43.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 -- 172.21.15.83:0/3203033537 shutdown_connections 2024-09-19T07:23:43.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 -- 172.21.15.83:0/3203033537 >> 172.21.15.83:0/3203033537 conn(0x7f355406bc80 msgr2=0x7f355406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:43.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 -- 172.21.15.83:0/3203033537 shutdown_connections 2024-09-19T07:23:43.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 -- 172.21.15.83:0/3203033537 wait complete. 2024-09-19T07:23:43.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 Processor -- start 2024-09-19T07:23:43.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 -- start start 2024-09-19T07:23:43.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35400039c0 0x7f35541511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35541499d0 0x7f35541520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3554147a30 0x7f3554153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f35541538f0 con 0x7f35541499d0 2024-09-19T07:23:43.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3554153770 con 0x7f35400039c0 2024-09-19T07:23:43.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f355c1de640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f35540c46c0 con 0x7f3554147a30 2024-09-19T07:23:43.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f3559779640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35541499d0 0x7f35541520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f3559779640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35541499d0 0x7f35541520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38594/0 (socket says 172.21.15.83:38594) 2024-09-19T07:23:43.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.874+0000 7f3559779640 1 -- 172.21.15.83:0/938760942 learned_addr learned my addr 172.21.15.83:0/938760942 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:43.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f3559f7a640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35400039c0 0x7f35541511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f355a77b640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3554147a30 0x7f3554153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f3559779640 1 -- 172.21.15.83:0/938760942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3554147a30 msgr2=0x7f3554153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:43.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f3559779640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3554147a30 0x7f3554153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f3559779640 1 -- 172.21.15.83:0/938760942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35400039c0 msgr2=0x7f35541511f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:43.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f3559779640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35400039c0 0x7f35541511f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:43.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f3559779640 1 -- 172.21.15.83:0/938760942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3554152610 con 0x7f35541499d0 2024-09-19T07:23:43.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f355a77b640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3554147a30 0x7f3554153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:43.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f3559f7a640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35400039c0 0x7f35541511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:43.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f3559779640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35541499d0 0x7f35541520d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3550011630 tx=0x7f3550011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:43.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f354a7fc640 1 -- 172.21.15.83:0/938760942 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355000f920 con 0x7f35541499d0 2024-09-19T07:23:43.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f354a7fc640 1 -- 172.21.15.83:0/938760942 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f355000fac0 con 0x7f35541499d0 2024-09-19T07:23:43.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f355c1de640 1 -- 172.21.15.83:0/938760942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3554155570 con 0x7f35541499d0 2024-09-19T07:23:43.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f354a7fc640 1 -- 172.21.15.83:0/938760942 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3550015460 con 0x7f35541499d0 2024-09-19T07:23:43.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f355c1de640 1 -- 172.21.15.83:0/938760942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f355414b690 con 0x7f35541499d0 2024-09-19T07:23:43.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f355c1de640 1 -- 172.21.15.83:0/938760942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3554155a20 con 0x7f35541499d0 2024-09-19T07:23:43.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f354a7fc640 1 -- 172.21.15.83:0/938760942 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3550015600 con 0x7f35541499d0 2024-09-19T07:23:43.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f354a7fc640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35380678c0 0x7f3538069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:43.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f354a7fc640 1 -- 172.21.15.83:0/938760942 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7153+0+0 (secure 0 0 0) 0x7f3550086b20 con 0x7f35541499d0 2024-09-19T07:23:43.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f3559f7a640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35380678c0 0x7f3538069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:43.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.878+0000 7f3559f7a640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35380678c0 0x7f3538069d80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f35440af000 tx=0x7f35440046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:43.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:43.882+0000 7f354a7fc640 1 -- 172.21.15.83:0/938760942 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3550019030 con 0x7f35541499d0 2024-09-19T07:23:44.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.082+0000 7f355c1de640 1 -- 172.21.15.83:0/938760942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3554159eb0 con 0x7f35541499d0 2024-09-19T07:23:44.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.082+0000 7f354a7fc640 1 -- 172.21.15.83:0/938760942 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5714 (secure 0 0 0) 0x7f3550057140 con 0x7f35541499d0 2024-09-19T07:23:44.083 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:44.083 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":42,"btime":"2024-09-19T07:23:41:116081+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:44.083 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 42 2024-09-19T07:23:44.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.082+0000 7f355c1de640 1 -- 172.21.15.83:0/938760942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35380678c0 msgr2=0x7f3538069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:44.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.082+0000 7f355c1de640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35380678c0 0x7f3538069d80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f35440af000 tx=0x7f35440046d0 comp rx=0 tx=0).stop 2024-09-19T07:23:44.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.082+0000 7f355c1de640 1 -- 172.21.15.83:0/938760942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35541499d0 msgr2=0x7f35541520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:44.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.082+0000 7f355c1de640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35541499d0 0x7f35541520d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3550011630 tx=0x7f3550011af0 comp rx=0 tx=0).stop 2024-09-19T07:23:44.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.086+0000 7f355c1de640 1 -- 172.21.15.83:0/938760942 shutdown_connections 2024-09-19T07:23:44.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.086+0000 7f355c1de640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35380678c0 0x7f3538069d80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:44.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.086+0000 7f355c1de640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3554147a30 0x7f3554153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:44.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.086+0000 7f355c1de640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35541499d0 0x7f35541520d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:44.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.086+0000 7f355c1de640 1 --2- 172.21.15.83:0/938760942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35400039c0 0x7f35541511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:44.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.086+0000 7f355c1de640 1 -- 172.21.15.83:0/938760942 >> 172.21.15.83:0/938760942 conn(0x7f355406bc80 msgr2=0x7f3554153040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:44.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.086+0000 7f355c1de640 1 -- 172.21.15.83:0/938760942 shutdown_connections 2024-09-19T07:23:44.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:44.086+0000 7f355c1de640 1 -- 172.21.15.83:0/938760942 wait complete. 2024-09-19T07:23:44.102 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:23:46.103 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:46.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1853c9f640 1 Processor -- start 2024-09-19T07:23:46.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1853c9f640 1 -- start start 2024-09-19T07:23:46.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1853c9f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f184c0ab750 0x7f184c0c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:46.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1853c9f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f184c0c2530 0x7f184c1528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:46.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1853c9f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f184c0ab060 0x7f184c152e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:46.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1853c9f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f184c06de90 con 0x7f184c0ab750 2024-09-19T07:23:46.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1853c9f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f184c06e170 con 0x7f184c0c2530 2024-09-19T07:23:46.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1853c9f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f184c06e450 con 0x7f184c0ab060 2024-09-19T07:23:46.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1851a3b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f184c0ab750 0x7f184c0c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:46.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1851a3b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f184c0ab750 0x7f184c0c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38596/0 (socket says 172.21.15.83:38596) 2024-09-19T07:23:46.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1851a3b640 1 -- 172.21.15.83:0/3654003148 learned_addr learned my addr 172.21.15.83:0/3654003148 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:46.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f185223c640 1 --2- 172.21.15.83:0/3654003148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f184c0ab060 0x7f184c152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:46.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f185123a640 1 --2- 172.21.15.83:0/3654003148 >> v2:172.21.15.184:3300/0 conn(0x7f184c0c2530 0x7f184c1528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:46.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1851a3b640 1 -- 172.21.15.83:0/3654003148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f184c0ab060 msgr2=0x7f184c152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:46.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1851a3b640 1 --2- 172.21.15.83:0/3654003148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f184c0ab060 0x7f184c152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:46.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1851a3b640 1 -- 172.21.15.83:0/3654003148 >> v2:172.21.15.184:3300/0 conn(0x7f184c0c2530 msgr2=0x7f184c1528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:46.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1851a3b640 1 --2- 172.21.15.83:0/3654003148 >> v2:172.21.15.184:3300/0 conn(0x7f184c0c2530 0x7f184c1528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:46.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.238+0000 7f1851a3b640 1 -- 172.21.15.83:0/3654003148 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184c06e730 con 0x7f184c0ab750 2024-09-19T07:23:46.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1851a3b640 1 --2- 172.21.15.83:0/3654003148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f184c0ab750 0x7f184c0c1ff0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f184c06cd40 tx=0x7f184801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=876b4302884dfcac server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:46.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1850a39640 1 -- 172.21.15.83:0/3654003148 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18480b0070 con 0x7f184c0ab750 2024-09-19T07:23:46.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1850a39640 1 -- 172.21.15.83:0/3654003148 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18480ab600 con 0x7f184c0ab750 2024-09-19T07:23:46.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1850a39640 1 -- 172.21.15.83:0/3654003148 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18480ab9d0 con 0x7f184c0ab750 2024-09-19T07:23:46.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 -- 172.21.15.83:0/3654003148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f184c0ab750 msgr2=0x7f184c0c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:46.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 --2- 172.21.15.83:0/3654003148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f184c0ab750 0x7f184c0c1ff0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f184c06cd40 tx=0x7f184801c490 comp rx=0 tx=0).stop 2024-09-19T07:23:46.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 -- 172.21.15.83:0/3654003148 shutdown_connections 2024-09-19T07:23:46.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 --2- 172.21.15.83:0/3654003148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f184c0ab060 0x7f184c152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:46.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 --2- 172.21.15.83:0/3654003148 >> v2:172.21.15.184:3300/0 conn(0x7f184c0c2530 0x7f184c1528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:46.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 --2- 172.21.15.83:0/3654003148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f184c0ab750 0x7f184c0c1ff0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:46.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 -- 172.21.15.83:0/3654003148 >> 172.21.15.83:0/3654003148 conn(0x7f184c06bc80 msgr2=0x7f184c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:46.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 -- 172.21.15.83:0/3654003148 shutdown_connections 2024-09-19T07:23:46.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 -- 172.21.15.83:0/3654003148 wait complete. 2024-09-19T07:23:46.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 Processor -- start 2024-09-19T07:23:46.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 -- start start 2024-09-19T07:23:46.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f184c0ab060 0x7f184c1f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f184c0ab750 0x7f184c0bf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f184c0c2530 0x7f184c0bf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f184c155c20 con 0x7f184c0c2530 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f184c155aa0 con 0x7f184c0ab060 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1853c9f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f184c155da0 con 0x7f184c0ab750 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f185223c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f184c0c2530 0x7f184c0bf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f185223c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f184c0c2530 0x7f184c0bf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38598/0 (socket says 172.21.15.83:38598) 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f185223c640 1 -- 172.21.15.83:0/3462013908 learned_addr learned my addr 172.21.15.83:0/3462013908 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1851a3b640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f184c0ab060 0x7f184c1f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f185123a640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f184c0ab750 0x7f184c0bf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f185223c640 1 -- 172.21.15.83:0/3462013908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f184c0ab750 msgr2=0x7f184c0bf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f185223c640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f184c0ab750 0x7f184c0bf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f185223c640 1 -- 172.21.15.83:0/3462013908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f184c0ab060 msgr2=0x7f184c1f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:46.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f185223c640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f184c0ab060 0x7f184c1f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:46.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f185223c640 1 -- 172.21.15.83:0/3462013908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184c14fb80 con 0x7f184c0c2530 2024-09-19T07:23:46.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f1851a3b640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f184c0ab060 0x7f184c1f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:46.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.242+0000 7f185223c640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f184c0c2530 0x7f184c0bf7d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1844007e90 tx=0x7f184400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:46.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.246+0000 7f18427fc640 1 -- 172.21.15.83:0/3462013908 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844014420 con 0x7f184c0c2530 2024-09-19T07:23:46.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.246+0000 7f18427fc640 1 -- 172.21.15.83:0/3462013908 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1844019070 con 0x7f184c0c2530 2024-09-19T07:23:46.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.246+0000 7f1853c9f640 1 -- 172.21.15.83:0/3462013908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f184c1f2e40 con 0x7f184c0c2530 2024-09-19T07:23:46.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.246+0000 7f18427fc640 1 -- 172.21.15.83:0/3462013908 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844014820 con 0x7f184c0c2530 2024-09-19T07:23:46.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.246+0000 7f1853c9f640 1 -- 172.21.15.83:0/3462013908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f184c157600 con 0x7f184c0c2530 2024-09-19T07:23:46.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.246+0000 7f1853c9f640 1 -- 172.21.15.83:0/3462013908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f184c0c1ff0 con 0x7f184c0c2530 2024-09-19T07:23:46.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.246+0000 7f18427fc640 1 -- 172.21.15.83:0/3462013908 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f18440149c0 con 0x7f184c0c2530 2024-09-19T07:23:46.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.246+0000 7f18427fc640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f18300678c0 0x7f1830069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:46.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.246+0000 7f18427fc640 1 -- 172.21.15.83:0/3462013908 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7153+0+0 (secure 0 0 0) 0x7f1844086b10 con 0x7f184c0c2530 2024-09-19T07:23:46.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.246+0000 7f1851a3b640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f18300678c0 0x7f1830069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:46.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.246+0000 7f1851a3b640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f18300678c0 0x7f1830069d80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f18480ad000 tx=0x7f1848008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.250+0000 7f18427fc640 1 -- 172.21.15.83:0/3462013908 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1844057090 con 0x7f184c0c2530 2024-09-19T07:23:46.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.446+0000 7f1853c9f640 1 -- 172.21.15.83:0/3462013908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f184c073880 con 0x7f184c0c2530 2024-09-19T07:23:46.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.446+0000 7f18427fc640 1 -- 172.21.15.83:0/3462013908 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5714 (secure 0 0 0) 0x7f1844056990 con 0x7f184c0c2530 2024-09-19T07:23:46.449 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:46.449 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":42,"btime":"2024-09-19T07:23:41:116081+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 42 2024-09-19T07:23:46.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.450+0000 7f1853c9f640 1 -- 172.21.15.83:0/3462013908 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f18300678c0 msgr2=0x7f1830069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:46.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.450+0000 7f1853c9f640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f18300678c0 0x7f1830069d80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f18480ad000 tx=0x7f1848008040 comp rx=0 tx=0).stop 2024-09-19T07:23:46.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.450+0000 7f1853c9f640 1 -- 172.21.15.83:0/3462013908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f184c0c2530 msgr2=0x7f184c0bf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:46.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.450+0000 7f1853c9f640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f184c0c2530 0x7f184c0bf7d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1844007e90 tx=0x7f184400e4b0 comp rx=0 tx=0).stop 2024-09-19T07:23:46.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.450+0000 7f1853c9f640 1 -- 172.21.15.83:0/3462013908 shutdown_connections 2024-09-19T07:23:46.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.450+0000 7f1853c9f640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f18300678c0 0x7f1830069d80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:46.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.450+0000 7f1853c9f640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f184c0c2530 0x7f184c0bf7d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:46.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.450+0000 7f1853c9f640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f184c0ab750 0x7f184c0bf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:46.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.450+0000 7f1853c9f640 1 --2- 172.21.15.83:0/3462013908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f184c0ab060 0x7f184c1f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:46.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.450+0000 7f1853c9f640 1 -- 172.21.15.83:0/3462013908 >> 172.21.15.83:0/3462013908 conn(0x7f184c06bc80 msgr2=0x7f184c15ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:46.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.450+0000 7f1853c9f640 1 -- 172.21.15.83:0/3462013908 shutdown_connections 2024-09-19T07:23:46.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:46.450+0000 7f1853c9f640 1 -- 172.21.15.83:0/3462013908 wait complete. 2024-09-19T07:23:46.470 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:23:48.471 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:48.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.602+0000 7f560d72c640 1 Processor -- start 2024-09-19T07:23:48.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.602+0000 7f560d72c640 1 -- start start 2024-09-19T07:23:48.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f560d72c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5608071100 0x7f56081545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:48.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f560d72c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56080c28e0 0x7f5608150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:48.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f560d72c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56080a9e60 con 0x7f5608151480 2024-09-19T07:23:48.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f560d72c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5608071b50 con 0x7f5608071100 2024-09-19T07:23:48.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f560d72c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f56080c1ff0 con 0x7f56080c28e0 2024-09-19T07:23:48.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f56077fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5608151480 0x7f560806dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33908/0 (socket says 172.21.15.83:33908) 2024-09-19T07:23:48.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f56077fe640 1 -- 172.21.15.83:0/4047801763 learned_addr learned my addr 172.21.15.83:0/4047801763 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:48.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f56067fc640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56080c28e0 0x7f5608150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:48.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5606ffd640 1 --2- 172.21.15.83:0/4047801763 >> v2:172.21.15.184:3300/0 conn(0x7f5608071100 0x7f56081545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:48.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 191266262 0 0) 0x7f56080a9e60 con 0x7f5608151480 2024-09-19T07:23:48.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55f0003530 con 0x7f5608151480 2024-09-19T07:23:48.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 68009951 0 0) 0x7f55f0003530 con 0x7f5608151480 2024-09-19T07:23:48.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56080c28e0 msgr2=0x7f5608150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:48.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56080c28e0 0x7f5608150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:48.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 >> v2:172.21.15.184:3300/0 conn(0x7f5608071100 msgr2=0x7f56081545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:48.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 --2- 172.21.15.83:0/4047801763 >> v2:172.21.15.184:3300/0 conn(0x7f5608071100 0x7f56081545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:48.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56080aa140 con 0x7f5608151480 2024-09-19T07:23:48.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f5600001a00 con 0x7f5608151480 2024-09-19T07:23:48.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 >> v1:172.21.15.83:6789/0 conn(0x7f5608151480 legacy=0x7f560806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:48.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56080c28e0 0x7f55f00039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:48.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55f0003f40 0x7f55f0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:48.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55f0004d50 0x7f55f00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:48.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5608071b50 con 0x7f55f0003f40 2024-09-19T07:23:48.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f56080c1ff0 con 0x7f55f0004d50 2024-09-19T07:23:48.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f55f000ba20 con 0x7f56080c28e0 2024-09-19T07:23:48.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f56067fc640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55f0003f40 0x7f55f0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:48.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f5606ffd640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56080c28e0 0x7f55f00039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:48.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f56077fe640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55f0004d50 0x7f55f00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:48.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f56067fc640 1 -- 172.21.15.83:0/4047801763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56080c28e0 msgr2=0x7f55f00039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:48.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f56067fc640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56080c28e0 0x7f55f00039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:48.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f56067fc640 1 -- 172.21.15.83:0/4047801763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55f0004d50 msgr2=0x7f55f00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:48.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f56067fc640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55f0004d50 0x7f55f00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:48.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f56067fc640 1 -- 172.21.15.83:0/4047801763 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f0003530 con 0x7f55f0003f40 2024-09-19T07:23:48.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.606+0000 7f56077fe640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55f0004d50 0x7f55f00051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:48.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f56067fc640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55f0003f40 0x7f55f0004810 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f55fc009ae0 tx=0x7f55fc01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda1a81009bdd5fa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:48.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc0096b0 con 0x7f55f0003f40 2024-09-19T07:23:48.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55fc0a6410 con 0x7f55f0003f40 2024-09-19T07:23:48.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f5605ffb640 1 -- 172.21.15.83:0/4047801763 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc0a6800 con 0x7f55f0003f40 2024-09-19T07:23:48.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 -- 172.21.15.83:0/4047801763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55f0003f40 msgr2=0x7f55f0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:48.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 --2- 172.21.15.83:0/4047801763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55f0003f40 0x7f55f0004810 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f55fc009ae0 tx=0x7f55fc01ca30 comp rx=0 tx=0).stop 2024-09-19T07:23:48.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f56077fe640 1 -- 172.21.15.83:0/4047801763 reap_dead start 2024-09-19T07:23:48.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 -- 172.21.15.83:0/4047801763 shutdown_connections 2024-09-19T07:23:48.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 -- 172.21.15.83:0/4047801763 >> 172.21.15.83:0/4047801763 conn(0x7f560806bc80 msgr2=0x7f560806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:48.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 -- 172.21.15.83:0/4047801763 shutdown_connections 2024-09-19T07:23:48.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 -- 172.21.15.83:0/4047801763 wait complete. 2024-09-19T07:23:48.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 Processor -- start 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 -- start start 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55f0003f40 0x7f56081f0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56081ecfd0 0x7f56081f0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5608152d20 0x7f56081531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f56080714e0 con 0x7f56081ecfd0 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f560806ebd0 con 0x7f5608152d20 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5608071660 con 0x7f55f0003f40 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f56067fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56081ecfd0 0x7f56081f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f56067fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56081ecfd0 0x7f56081f0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38604/0 (socket says 172.21.15.83:38604) 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f56067fc640 1 -- 172.21.15.83:0/3161979148 learned_addr learned my addr 172.21.15.83:0/3161979148 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f5606ffd640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55f0003f40 0x7f56081f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f56077fe640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5608152d20 0x7f56081531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f56067fc640 1 -- 172.21.15.83:0/3161979148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55f0003f40 msgr2=0x7f56081f0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f56067fc640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55f0003f40 0x7f56081f0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f56067fc640 1 -- 172.21.15.83:0/3161979148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5608152d20 msgr2=0x7f56081531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f56067fc640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5608152d20 0x7f56081531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f56067fc640 1 -- 172.21.15.83:0/3161979148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56081ed4b0 con 0x7f56081ecfd0 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f56067fc640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56081ecfd0 0x7f56081f0640 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f55fc01c860 tx=0x7f55f0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f55ef7fe640 1 -- 172.21.15.83:0/3161979148 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc00f000 con 0x7f56081ecfd0 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f55ef7fe640 1 -- 172.21.15.83:0/3161979148 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55fc0a6bc0 con 0x7f56081ecfd0 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 -- 172.21.15.83:0/3161979148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5608154e30 con 0x7f56081ecfd0 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f55ef7fe640 1 -- 172.21.15.83:0/3161979148 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc0ba420 con 0x7f56081ecfd0 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.610+0000 7f560d72c640 1 -- 172.21.15.83:0/3161979148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56081efbe0 con 0x7f56081ecfd0 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.614+0000 7f560d72c640 1 -- 172.21.15.83:0/3161979148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56081545e0 con 0x7f56081ecfd0 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.614+0000 7f55ef7fe640 1 -- 172.21.15.83:0/3161979148 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f55fc0c3440 con 0x7f56081ecfd0 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.614+0000 7f55ef7fe640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f55f40678f0 0x7f55f4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.614+0000 7f55ef7fe640 1 -- 172.21.15.83:0/3161979148 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7153+0+0 (secure 0 0 0) 0x7f55fc11fef0 con 0x7f56081ecfd0 2024-09-19T07:23:48.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.614+0000 7f5606ffd640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f55f40678f0 0x7f55f4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:48.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.614+0000 7f5606ffd640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f55f40678f0 0x7f55f4069db0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009ab0 tx=0x7f55f8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:48.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.618+0000 7f55ef7fe640 1 -- 172.21.15.83:0/3161979148 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f55fc0ac020 con 0x7f56081ecfd0 2024-09-19T07:23:48.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.814+0000 7f560d72c640 1 -- 172.21.15.83:0/3161979148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56080c4f30 con 0x7f56081ecfd0 2024-09-19T07:23:48.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.814+0000 7f55ef7fe640 1 -- 172.21.15.83:0/3161979148 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5714 (secure 0 0 0) 0x7f55fc016780 con 0x7f56081ecfd0 2024-09-19T07:23:48.816 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:48.816 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":42,"btime":"2024-09-19T07:23:41:116081+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:48.816 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 42 2024-09-19T07:23:48.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.818+0000 7f560d72c640 1 -- 172.21.15.83:0/3161979148 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f55f40678f0 msgr2=0x7f55f4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:48.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.818+0000 7f560d72c640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f55f40678f0 0x7f55f4069db0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009ab0 tx=0x7f55f8008040 comp rx=0 tx=0).stop 2024-09-19T07:23:48.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.818+0000 7f560d72c640 1 -- 172.21.15.83:0/3161979148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56081ecfd0 msgr2=0x7f56081f0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:48.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.818+0000 7f560d72c640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56081ecfd0 0x7f56081f0640 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f55fc01c860 tx=0x7f55f0005840 comp rx=0 tx=0).stop 2024-09-19T07:23:48.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.818+0000 7f560d72c640 1 -- 172.21.15.83:0/3161979148 shutdown_connections 2024-09-19T07:23:48.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.818+0000 7f560d72c640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f55f40678f0 0x7f55f4069db0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:48.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.818+0000 7f560d72c640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5608152d20 0x7f56081531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:48.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.818+0000 7f560d72c640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56081ecfd0 0x7f56081f0640 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:48.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.818+0000 7f560d72c640 1 --2- 172.21.15.83:0/3161979148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55f0003f40 0x7f56081f0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:48.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.818+0000 7f560d72c640 1 -- 172.21.15.83:0/3161979148 >> 172.21.15.83:0/3161979148 conn(0x7f560806bc80 msgr2=0x7f5608159550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:48.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.818+0000 7f560d72c640 1 -- 172.21.15.83:0/3161979148 shutdown_connections 2024-09-19T07:23:48.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:48.818+0000 7f560d72c640 1 -- 172.21.15.83:0/3161979148 wait complete. 2024-09-19T07:23:48.836 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:23:50.837 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:50.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4db3ea640 1 Processor -- start 2024-09-19T07:23:50.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4db3ea640 1 -- start start 2024-09-19T07:23:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4db3ea640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd4d414dbe0 0x7fd4d40c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4db3ea640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4d40c3ab0 0x7fd4d4157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4db3ea640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4d4073d90 con 0x7fd4d40c44e0 2024-09-19T07:23:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4db3ea640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d4154700 con 0x7fd4d414dbe0 2024-09-19T07:23:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4db3ea640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d4153b30 con 0x7fd4d40c3ab0 2024-09-19T07:23:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4d9186640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd4d414dbe0 0x7fd4d40c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4d9186640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd4d414dbe0 0x7fd4d40c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48932/0 (socket says 172.21.15.83:48932) 2024-09-19T07:23:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4d9186640 1 -- 172.21.15.83:0/1646853845 learned_addr learned my addr 172.21.15.83:0/1646853845 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4d8985640 1 --2- 172.21.15.83:0/1646853845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4d40c3ab0 0x7fd4d4157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4d9186640 1 -- 172.21.15.83:0/1646853845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4d40c3ab0 msgr2=0x7fd4d4157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4d9186640 1 --2- 172.21.15.83:0/1646853845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4d40c3ab0 0x7fd4d4157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4d9186640 1 -- 172.21.15.83:0/1646853845 >> v1:172.21.15.184:6789/0 conn(0x7fd4d40c44e0 legacy=0x7fd4d4157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.966+0000 7fd4d9186640 1 -- 172.21.15.83:0/1646853845 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d4153dd0 con 0x7fd4d414dbe0 2024-09-19T07:23:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4d9186640 1 --2- 172.21.15.83:0/1646853845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4d414dbe0 0x7fd4d40c3fa0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0009870 tx=0x7fd4d001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be04da6569303e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4cbfff640 1 -- 172.21.15.83:0/1646853845 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d00b2070 con 0x7fd4d414dbe0 2024-09-19T07:23:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4cbfff640 1 -- 172.21.15.83:0/1646853845 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d00a9070 con 0x7fd4d414dbe0 2024-09-19T07:23:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4cbfff640 1 -- 172.21.15.83:0/1646853845 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d00ad880 con 0x7fd4d414dbe0 2024-09-19T07:23:50.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 -- 172.21.15.83:0/1646853845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4d414dbe0 msgr2=0x7fd4d40c3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:50.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 --2- 172.21.15.83:0/1646853845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4d414dbe0 0x7fd4d40c3fa0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0009870 tx=0x7fd4d001c490 comp rx=0 tx=0).stop 2024-09-19T07:23:50.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 -- 172.21.15.83:0/1646853845 shutdown_connections 2024-09-19T07:23:50.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 --2- 172.21.15.83:0/1646853845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4d40c3ab0 0x7fd4d4157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:50.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 --2- 172.21.15.83:0/1646853845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4d414dbe0 0x7fd4d40c3fa0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:50.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 -- 172.21.15.83:0/1646853845 >> 172.21.15.83:0/1646853845 conn(0x7fd4d40c95a0 msgr2=0x7fd4d40c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:50.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 -- 172.21.15.83:0/1646853845 shutdown_connections 2024-09-19T07:23:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 -- 172.21.15.83:0/1646853845 wait complete. 2024-09-19T07:23:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 Processor -- start 2024-09-19T07:23:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 -- start start 2024-09-19T07:23:50.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4d40c44e0 0x7fd4d41f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:50.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4d414dbe0 0x7fd4d4149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:50.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4d41ecc40 0x7fd4d40c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:50.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d4160bc0 con 0x7fd4d40c44e0 2024-09-19T07:23:50.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d40741a0 con 0x7fd4d41ecc40 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4db3ea640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d4160d40 con 0x7fd4d414dbe0 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4d9186640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4d40c44e0 0x7fd4d41f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4d9186640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4d40c44e0 0x7fd4d41f0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48942/0 (socket says 172.21.15.83:48942) 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4d9186640 1 -- 172.21.15.83:0/2870079958 learned_addr learned my addr 172.21.15.83:0/2870079958 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4d9186640 1 -- 172.21.15.83:0/2870079958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4d414dbe0 msgr2=0x7fd4d4149bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4d9186640 1 --2- 172.21.15.83:0/2870079958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4d414dbe0 0x7fd4d4149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4d9186640 1 -- 172.21.15.83:0/2870079958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4d41ecc40 msgr2=0x7fd4d40c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4d9186640 1 --2- 172.21.15.83:0/2870079958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4d41ecc40 0x7fd4d40c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4d9186640 1 -- 172.21.15.83:0/2870079958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d414efe0 con 0x7fd4d40c44e0 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.970+0000 7fd4d9186640 1 --2- 172.21.15.83:0/2870079958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4d40c44e0 0x7fd4d41f0c90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00b0040 tx=0x7fd4d0019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.974+0000 7fd4c9ffb640 1 -- 172.21.15.83:0/2870079958 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d00bb070 con 0x7fd4d40c44e0 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.974+0000 7fd4c9ffb640 1 -- 172.21.15.83:0/2870079958 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d00addd0 con 0x7fd4d40c44e0 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.974+0000 7fd4db3ea640 1 -- 172.21.15.83:0/2870079958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d40ac9b0 con 0x7fd4d40c44e0 2024-09-19T07:23:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.974+0000 7fd4c9ffb640 1 -- 172.21.15.83:0/2870079958 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d00b2040 con 0x7fd4d40c44e0 2024-09-19T07:23:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.974+0000 7fd4db3ea640 1 -- 172.21.15.83:0/2870079958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d41f01b0 con 0x7fd4d40c44e0 2024-09-19T07:23:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.974+0000 7fd4db3ea640 1 -- 172.21.15.83:0/2870079958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d40c3fa0 con 0x7fd4d40c44e0 2024-09-19T07:23:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.974+0000 7fd4c9ffb640 1 -- 172.21.15.83:0/2870079958 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd4d00be070 con 0x7fd4d40c44e0 2024-09-19T07:23:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.974+0000 7fd4c9ffb640 1 --2- 172.21.15.83:0/2870079958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4b80678c0 0x7fd4b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.974+0000 7fd4d8985640 1 --2- 172.21.15.83:0/2870079958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4b80678c0 0x7fd4b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.974+0000 7fd4c9ffb640 1 -- 172.21.15.83:0/2870079958 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7153+0+0 (secure 0 0 0) 0x7fd4d011f240 con 0x7fd4d40c44e0 2024-09-19T07:23:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.974+0000 7fd4d8985640 1 --2- 172.21.15.83:0/2870079958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4b80678c0 0x7fd4b8069d80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40aec10 tx=0x7fd4c4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:50.978+0000 7fd4c9ffb640 1 -- 172.21.15.83:0/2870079958 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd4d0016110 con 0x7fd4d40c44e0 2024-09-19T07:23:51.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 -- 172.21.15.83:0/2870079958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4d4073f80 con 0x7fd4d40c44e0 2024-09-19T07:23:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4c9ffb640 1 -- 172.21.15.83:0/2870079958 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5714 (secure 0 0 0) 0x7fd4d00ef980 con 0x7fd4d40c44e0 2024-09-19T07:23:51.162 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:51.162 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":42,"btime":"2024-09-19T07:23:41:116081+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:51.163 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 42 2024-09-19T07:23:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 -- 172.21.15.83:0/2870079958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4b80678c0 msgr2=0x7fd4b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 --2- 172.21.15.83:0/2870079958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4b80678c0 0x7fd4b8069d80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40aec10 tx=0x7fd4c4008040 comp rx=0 tx=0).stop 2024-09-19T07:23:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 -- 172.21.15.83:0/2870079958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4d40c44e0 msgr2=0x7fd4d41f0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 --2- 172.21.15.83:0/2870079958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4d40c44e0 0x7fd4d41f0c90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00b0040 tx=0x7fd4d0019230 comp rx=0 tx=0).stop 2024-09-19T07:23:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 -- 172.21.15.83:0/2870079958 shutdown_connections 2024-09-19T07:23:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 --2- 172.21.15.83:0/2870079958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4b80678c0 0x7fd4b8069d80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 --2- 172.21.15.83:0/2870079958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4d41ecc40 0x7fd4d40c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 --2- 172.21.15.83:0/2870079958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4d414dbe0 0x7fd4d4149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 --2- 172.21.15.83:0/2870079958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4d40c44e0 0x7fd4d41f0c90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 -- 172.21.15.83:0/2870079958 >> 172.21.15.83:0/2870079958 conn(0x7fd4d40c95a0 msgr2=0x7fd4d40c6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 -- 172.21.15.83:0/2870079958 shutdown_connections 2024-09-19T07:23:51.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.162+0000 7fd4db3ea640 1 -- 172.21.15.83:0/2870079958 wait complete. 2024-09-19T07:23:51.179 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:23:51.467 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-19T07:23:51.468 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:23:51.468 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:23:51.471 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:23:51.471 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:51.539 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:23:51.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a43a3640 1 Processor -- start 2024-09-19T07:23:51.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a43a3640 1 -- start start 2024-09-19T07:23:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a43a3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f299c0bef00 0x7f299c156840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a43a3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f299c0ca240 con 0x7f299c0cbb00 2024-09-19T07:23:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a43a3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f299c0c9c80 con 0x7f299c14ca60 2024-09-19T07:23:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a43a3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f299c0c9f60 con 0x7f299c0bef00 2024-09-19T07:23:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a193e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f299c0bef00 0x7f299c156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a2940640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f299c0cbb00 0x7f299c160820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51412/0 (socket says 172.21.15.83:51412) 2024-09-19T07:23:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a2940640 1 -- 172.21.15.83:0/2326874916 learned_addr learned my addr 172.21.15.83:0/2326874916 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a193e640 1 -- 172.21.15.83:0/2326874916 >> v1:172.21.15.184:6789/0 conn(0x7f299c14ca60 legacy=0x7f299c159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a193e640 1 -- 172.21.15.83:0/2326874916 >> v1:172.21.15.83:6789/0 conn(0x7f299c0cbb00 legacy=0x7f299c160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a113d640 1 -- 172.21.15.83:0/2326874916 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 259594330 0 0) 0x7f299c0ca240 con 0x7f299c0cbb00 2024-09-19T07:23:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.586+0000 7f29a193e640 1 -- 172.21.15.83:0/2326874916 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299c156040 con 0x7f299c0bef00 2024-09-19T07:23:51.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a193e640 1 --2- 172.21.15.83:0/2326874916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299c0bef00 0x7f299c156840 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f298c0097c0 tx=0x7f298c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464823cfbb2ed72b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:51.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a113d640 1 -- 172.21.15.83:0/2326874916 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c0b0070 con 0x7f299c0bef00 2024-09-19T07:23:51.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a113d640 1 -- 172.21.15.83:0/2326874916 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f298c0ab600 con 0x7f299c0bef00 2024-09-19T07:23:51.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a113d640 1 -- 172.21.15.83:0/2326874916 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c0ab9d0 con 0x7f299c0bef00 2024-09-19T07:23:51.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a43a3640 1 -- 172.21.15.83:0/2326874916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299c0bef00 msgr2=0x7f299c156840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:51.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a43a3640 1 --2- 172.21.15.83:0/2326874916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299c0bef00 0x7f299c156840 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f298c0097c0 tx=0x7f298c01c460 comp rx=0 tx=0).stop 2024-09-19T07:23:51.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a43a3640 1 -- 172.21.15.83:0/2326874916 shutdown_connections 2024-09-19T07:23:51.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a43a3640 1 --2- 172.21.15.83:0/2326874916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299c0bef00 0x7f299c156840 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:51.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a43a3640 1 -- 172.21.15.83:0/2326874916 >> 172.21.15.83:0/2326874916 conn(0x7f299c072470 msgr2=0x7f299c072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:51.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a43a3640 1 -- 172.21.15.83:0/2326874916 shutdown_connections 2024-09-19T07:23:51.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a43a3640 1 -- 172.21.15.83:0/2326874916 wait complete. 2024-09-19T07:23:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a43a3640 1 Processor -- start 2024-09-19T07:23:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.590+0000 7f29a43a3640 1 -- start start 2024-09-19T07:23:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a43a3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299c0bef00 0x7f299c1ef550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a43a3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f299c0cbb00 0x7f299c1492e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a43a3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f299c14ca60 0x7f299c14b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a43a3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f299c153a50 con 0x7f299c0bef00 2024-09-19T07:23:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a43a3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f299c06ccb0 con 0x7f299c0cbb00 2024-09-19T07:23:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a43a3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f299c0c98c0 con 0x7f299c14ca60 2024-09-19T07:23:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a213f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299c0bef00 0x7f299c1ef550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a213f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299c0bef00 0x7f299c1ef550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48988/0 (socket says 172.21.15.83:48988) 2024-09-19T07:23:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a213f640 1 -- 172.21.15.83:0/435276907 learned_addr learned my addr 172.21.15.83:0/435276907 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a2940640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f299c14ca60 0x7f299c14b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a193e640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f299c0cbb00 0x7f299c1492e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a213f640 1 -- 172.21.15.83:0/435276907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f299c14ca60 msgr2=0x7f299c14b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a213f640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f299c14ca60 0x7f299c14b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a213f640 1 -- 172.21.15.83:0/435276907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f299c0cbb00 msgr2=0x7f299c1492e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a213f640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f299c0cbb00 0x7f299c1492e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a213f640 1 -- 172.21.15.83:0/435276907 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299c0c2450 con 0x7f299c0bef00 2024-09-19T07:23:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a2940640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f299c14ca60 0x7f299c14b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a213f640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299c0bef00 0x7f299c1ef550 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f299800b410 tx=0x7f299800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f2992ffd640 1 -- 172.21.15.83:0/435276907 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998015030 con 0x7f299c0bef00 2024-09-19T07:23:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f2992ffd640 1 -- 172.21.15.83:0/435276907 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2998004020 con 0x7f299c0bef00 2024-09-19T07:23:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a43a3640 1 -- 172.21.15.83:0/435276907 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299c14af50 con 0x7f299c0bef00 2024-09-19T07:23:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f2992ffd640 1 -- 172.21.15.83:0/435276907 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998004450 con 0x7f299c0bef00 2024-09-19T07:23:51.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a43a3640 1 -- 172.21.15.83:0/435276907 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f299c1f21a0 con 0x7f299c0bef00 2024-09-19T07:23:51.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a43a3640 1 -- 172.21.15.83:0/435276907 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f299c06ca90 con 0x7f299c0bef00 2024-09-19T07:23:51.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f2992ffd640 1 -- 172.21.15.83:0/435276907 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2998018070 con 0x7f299c0bef00 2024-09-19T07:23:51.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f2992ffd640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29800678c0 0x7f2980069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:51.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f2992ffd640 1 -- 172.21.15.83:0/435276907 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7153+0+0 (secure 0 0 0) 0x7f2998085dc0 con 0x7f299c0bef00 2024-09-19T07:23:51.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a193e640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29800678c0 0x7f2980069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:51.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.594+0000 7f29a193e640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29800678c0 0x7f2980069d80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f298c0ad000 tx=0x7f298c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:51.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.598+0000 7f2992ffd640 1 -- 172.21.15.83:0/435276907 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f29980563e0 con 0x7f299c0bef00 2024-09-19T07:23:51.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.786+0000 7f29a43a3640 1 -- 172.21.15.83:0/435276907 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f299c072aa0 con 0x7f299c0bef00 2024-09-19T07:23:51.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.786+0000 7f2992ffd640 1 -- 172.21.15.83:0/435276907 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5714 (secure 0 0 0) 0x7f2998055ce0 con 0x7f299c0bef00 2024-09-19T07:23:51.789 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:51.789 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":42,"btime":"2024-09-19T07:23:41:116081+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:22:36.207732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4786":{"gid":4786,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":24,"addr":"172.21.15.184:6835/4115191178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4115191178},{"type":"v1","addr":"172.21.15.184:6835","nonce":4115191178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4786,"qdb_cluster":[4786]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:51.789 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 42 2024-09-19T07:23:51.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.790+0000 7f29a43a3640 1 -- 172.21.15.83:0/435276907 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29800678c0 msgr2=0x7f2980069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:51.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.790+0000 7f29a43a3640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29800678c0 0x7f2980069d80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f298c0ad000 tx=0x7f298c008040 comp rx=0 tx=0).stop 2024-09-19T07:23:51.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.790+0000 7f29a43a3640 1 -- 172.21.15.83:0/435276907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299c0bef00 msgr2=0x7f299c1ef550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:51.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.790+0000 7f29a43a3640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299c0bef00 0x7f299c1ef550 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f299800b410 tx=0x7f299800b8d0 comp rx=0 tx=0).stop 2024-09-19T07:23:51.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.790+0000 7f29a43a3640 1 -- 172.21.15.83:0/435276907 shutdown_connections 2024-09-19T07:23:51.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.790+0000 7f29a43a3640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29800678c0 0x7f2980069d80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:51.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.790+0000 7f29a43a3640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f299c14ca60 0x7f299c14b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:51.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.790+0000 7f29a43a3640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f299c0cbb00 0x7f299c1492e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:51.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.790+0000 7f29a43a3640 1 --2- 172.21.15.83:0/435276907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299c0bef00 0x7f299c1ef550 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:51.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.790+0000 7f29a43a3640 1 -- 172.21.15.83:0/435276907 >> 172.21.15.83:0/435276907 conn(0x7f299c072470 msgr2=0x7f299c155840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:51.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.790+0000 7f29a43a3640 1 -- 172.21.15.83:0/435276907 shutdown_connections 2024-09-19T07:23:51.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:51.790+0000 7f29a43a3640 1 -- 172.21.15.83:0/435276907 wait complete. 2024-09-19T07:23:51.805 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-19T07:23:53.181 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:53.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.306+0000 7f3e0e1cc640 1 Processor -- start 2024-09-19T07:23:53.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e0e1cc640 1 -- start start 2024-09-19T07:23:53.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e0e1cc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e08152b20 0x7f3e080c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e0e1cc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e080c45c0 0x7f3e08153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e0e1cc640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e08150670 con 0x7f3e08150290 2024-09-19T07:23:53.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e0e1cc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3e080716c0 con 0x7f3e080c45c0 2024-09-19T07:23:53.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e0e1cc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3e080c3920 con 0x7f3e08152b20 2024-09-19T07:23:53.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e07fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e080c45c0 0x7f3e08153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e07fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e080c45c0 0x7f3e08153eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48994/0 (socket says 172.21.15.83:48994) 2024-09-19T07:23:53.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e07fff640 1 -- 172.21.15.83:0/2043895755 learned_addr learned my addr 172.21.15.83:0/2043895755 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:53.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e06ffd640 1 --2- 172.21.15.83:0/2043895755 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e08152b20 0x7f3e080c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e07fff640 1 -- 172.21.15.83:0/2043895755 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e08152b20 msgr2=0x7f3e080c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:53.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e07fff640 1 --2- 172.21.15.83:0/2043895755 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e08152b20 0x7f3e080c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e07fff640 1 -- 172.21.15.83:0/2043895755 >> v1:172.21.15.184:6789/0 conn(0x7f3e08150290 legacy=0x7f3e080c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e07fff640 1 -- 172.21.15.83:0/2043895755 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e080c2880 con 0x7f3e080c45c0 2024-09-19T07:23:53.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e07fff640 1 --2- 172.21.15.83:0/2043895755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e080c45c0 0x7f3e08153eb0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009e80 tx=0x7f3e0001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d420af460baab6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:53.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e067fc640 1 -- 172.21.15.83:0/2043895755 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e000a57d0 con 0x7f3e080c45c0 2024-09-19T07:23:53.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e067fc640 1 -- 172.21.15.83:0/2043895755 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e000a5970 con 0x7f3e080c45c0 2024-09-19T07:23:53.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.310+0000 7f3e067fc640 1 -- 172.21.15.83:0/2043895755 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e000a5d40 con 0x7f3e080c45c0 2024-09-19T07:23:53.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/2043895755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e080c45c0 msgr2=0x7f3e08153eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 --2- 172.21.15.83:0/2043895755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e080c45c0 0x7f3e08153eb0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009e80 tx=0x7f3e0001ca10 comp rx=0 tx=0).stop 2024-09-19T07:23:53.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/2043895755 shutdown_connections 2024-09-19T07:23:53.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 --2- 172.21.15.83:0/2043895755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e080c45c0 0x7f3e08153eb0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 --2- 172.21.15.83:0/2043895755 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e08152b20 0x7f3e080c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/2043895755 >> 172.21.15.83:0/2043895755 conn(0x7f3e0806bc80 msgr2=0x7f3e0806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:53.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/2043895755 shutdown_connections 2024-09-19T07:23:53.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/2043895755 wait complete. 2024-09-19T07:23:53.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 Processor -- start 2024-09-19T07:23:53.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 -- start start 2024-09-19T07:23:53.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e080c45c0 0x7f3e081f06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e08150290 0x7f3e0806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e08152b20 0x7f3e08072480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3e08154980 con 0x7f3e080c45c0 2024-09-19T07:23:53.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3e08154800 con 0x7f3e08152b20 2024-09-19T07:23:53.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3e0815fb60 con 0x7f3e08150290 2024-09-19T07:23:53.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e080c45c0 0x7f3e081f06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e080c45c0 0x7f3e081f06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49010/0 (socket says 172.21.15.83:49010) 2024-09-19T07:23:53.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e077fe640 1 -- 172.21.15.83:0/4200915596 learned_addr learned my addr 172.21.15.83:0/4200915596 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:53.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e06ffd640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e08150290 0x7f3e0806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e07fff640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e08152b20 0x7f3e08072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e077fe640 1 -- 172.21.15.83:0/4200915596 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e08150290 msgr2=0x7f3e0806e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e077fe640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e08150290 0x7f3e0806e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e077fe640 1 -- 172.21.15.83:0/4200915596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e08152b20 msgr2=0x7f3e08072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e077fe640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e08152b20 0x7f3e08072480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e077fe640 1 -- 172.21.15.83:0/4200915596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e08073df0 con 0x7f3e080c45c0 2024-09-19T07:23:53.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e077fe640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e080c45c0 0x7f3e081f06d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3df000c530 tx=0x7f3df000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:53.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3dfffff640 1 -- 172.21.15.83:0/4200915596 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df0016020 con 0x7f3e080c45c0 2024-09-19T07:23:53.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3dfffff640 1 -- 172.21.15.83:0/4200915596 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df0005220 con 0x7f3e080c45c0 2024-09-19T07:23:53.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/4200915596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e080740d0 con 0x7f3e080c45c0 2024-09-19T07:23:53.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3dfffff640 1 -- 172.21.15.83:0/4200915596 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df0007030 con 0x7f3e080c45c0 2024-09-19T07:23:53.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.314+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/4200915596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e081f01b0 con 0x7f3e080c45c0 2024-09-19T07:23:53.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.318+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/4200915596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e08153eb0 con 0x7f3e080c45c0 2024-09-19T07:23:53.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.318+0000 7f3dfffff640 1 -- 172.21.15.83:0/4200915596 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3df00071d0 con 0x7f3e080c45c0 2024-09-19T07:23:53.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.318+0000 7f3dfffff640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3de80678c0 0x7f3de8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.318+0000 7f3dfffff640 1 -- 172.21.15.83:0/4200915596 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7f3df0085c10 con 0x7f3e080c45c0 2024-09-19T07:23:53.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.318+0000 7f3e06ffd640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3de80678c0 0x7f3de8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.318+0000 7f3e06ffd640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3de80678c0 0x7f3de8069d80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3e0814ffb0 tx=0x7f3df8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:53.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.322+0000 7f3dfffff640 1 -- 172.21.15.83:0/4200915596 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3df0020020 con 0x7f3e080c45c0 2024-09-19T07:23:53.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.514+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/4200915596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e080c3c70 con 0x7f3e080c45c0 2024-09-19T07:23:53.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.514+0000 7f3dfffff640 1 -- 172.21.15.83:0/4200915596 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5700 (secure 0 0 0) 0x7f3df00074c0 con 0x7f3e080c45c0 2024-09-19T07:23:53.517 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:53.517 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":45,"btime":"2024-09-19T07:23:53:168427+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:52.469054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:53.517 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 45 2024-09-19T07:23:53.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.518+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/4200915596 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3de80678c0 msgr2=0x7f3de8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.518+0000 7f3e0e1cc640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3de80678c0 0x7f3de8069d80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3e0814ffb0 tx=0x7f3df8008040 comp rx=0 tx=0).stop 2024-09-19T07:23:53.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.518+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/4200915596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e080c45c0 msgr2=0x7f3e081f06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.518+0000 7f3e0e1cc640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e080c45c0 0x7f3e081f06d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3df000c530 tx=0x7f3df000c9f0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.518+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/4200915596 shutdown_connections 2024-09-19T07:23:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.518+0000 7f3e0e1cc640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3de80678c0 0x7f3de8069d80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.518+0000 7f3e0e1cc640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e08152b20 0x7f3e08072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.518+0000 7f3e0e1cc640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e08150290 0x7f3e0806e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.518+0000 7f3e0e1cc640 1 --2- 172.21.15.83:0/4200915596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e080c45c0 0x7f3e081f06d0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.518+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/4200915596 >> 172.21.15.83:0/4200915596 conn(0x7f3e0806bc80 msgr2=0x7f3e08155e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.518+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/4200915596 shutdown_connections 2024-09-19T07:23:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.518+0000 7f3e0e1cc640 1 -- 172.21.15.83:0/4200915596 wait complete. 2024-09-19T07:23:53.535 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-09-19T07:23:53.535 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:23:53.535 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:53.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfe2e43640 1 Processor -- start 2024-09-19T07:23:53.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfe2e43640 1 -- start start 2024-09-19T07:23:53.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfe2e43640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdfdc152b20 0x7fdfdc0c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfe2e43640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfdc0c45c0 0x7fdfdc153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfe2e43640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfdc150670 con 0x7fdfdc150290 2024-09-19T07:23:53.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfe2e43640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fdfdc0716c0 con 0x7fdfdc152b20 2024-09-19T07:23:53.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfe2e43640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdfdc0c3920 con 0x7fdfdc0c45c0 2024-09-19T07:23:53.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfdbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdfdc152b20 0x7fdfdc0c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfe0bdf640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdfdc150290 0x7fdfdc0c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51420/0 (socket says 172.21.15.83:51420) 2024-09-19T07:23:53.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfe0bdf640 1 -- 172.21.15.83:0/473686989 learned_addr learned my addr 172.21.15.83:0/473686989 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:53.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfe13e0640 1 --2- 172.21.15.83:0/473686989 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfdc0c45c0 0x7fdfdc153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfdbfff640 1 -- 172.21.15.83:0/473686989 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfdc0c45c0 msgr2=0x7fdfdc153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:53.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfdbfff640 1 --2- 172.21.15.83:0/473686989 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfdc0c45c0 0x7fdfdc153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfdbfff640 1 -- 172.21.15.83:0/473686989 >> v1:172.21.15.83:6789/0 conn(0x7fdfdc150290 legacy=0x7fdfdc0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfdbfff640 1 -- 172.21.15.83:0/473686989 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfdc0c2880 con 0x7fdfdc152b20 2024-09-19T07:23:53.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.658+0000 7fdfdb7fe640 1 -- 172.21.15.83:0/473686989 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3674479768 0 0) 0x7fdfdc150670 con 0x7fdfdc150290 2024-09-19T07:23:53.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdbfff640 1 --2- 172.21.15.83:0/473686989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfdc152b20 0x7fdfdc0c4080 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0097f0 tx=0x7fdfcc01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c047504c138c48 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:53.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdb7fe640 1 -- 172.21.15.83:0/473686989 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc0b0070 con 0x7fdfdc152b20 2024-09-19T07:23:53.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdb7fe640 1 -- 172.21.15.83:0/473686989 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfcc0ab600 con 0x7fdfdc152b20 2024-09-19T07:23:53.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdb7fe640 1 -- 172.21.15.83:0/473686989 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc0ab9d0 con 0x7fdfdc152b20 2024-09-19T07:23:53.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 -- 172.21.15.83:0/473686989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfdc152b20 msgr2=0x7fdfdc0c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 --2- 172.21.15.83:0/473686989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfdc152b20 0x7fdfdc0c4080 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0097f0 tx=0x7fdfcc01c4a0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 -- 172.21.15.83:0/473686989 shutdown_connections 2024-09-19T07:23:53.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 --2- 172.21.15.83:0/473686989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfdc152b20 0x7fdfdc0c4080 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 --2- 172.21.15.83:0/473686989 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfdc0c45c0 0x7fdfdc153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 -- 172.21.15.83:0/473686989 >> 172.21.15.83:0/473686989 conn(0x7fdfdc06bc80 msgr2=0x7fdfdc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:53.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 -- 172.21.15.83:0/473686989 shutdown_connections 2024-09-19T07:23:53.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 -- 172.21.15.83:0/473686989 wait complete. 2024-09-19T07:23:53.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 Processor -- start 2024-09-19T07:23:53.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 -- start start 2024-09-19T07:23:53.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfdc0c45c0 0x7fdfdc150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfdc150290 0x7fdfdc0c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfdc152b20 0x7fdfdc1f1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc154980 con 0x7fdfdc150290 2024-09-19T07:23:53.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc154800 con 0x7fdfdc152b20 2024-09-19T07:23:53.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdfdc06e6b0 con 0x7fdfdc0c45c0 2024-09-19T07:23:53.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfdc150290 0x7fdfdc0c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfdc150290 0x7fdfdc0c0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49024/0 (socket says 172.21.15.83:49024) 2024-09-19T07:23:53.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdbfff640 1 -- 172.21.15.83:0/3074615728 learned_addr learned my addr 172.21.15.83:0/3074615728 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe0bdf640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfdc0c45c0 0x7fdfdc150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdbfff640 1 -- 172.21.15.83:0/3074615728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfdc0c45c0 msgr2=0x7fdfdc150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe13e0640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfdc152b20 0x7fdfdc1f1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdbfff640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfdc0c45c0 0x7fdfdc150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdbfff640 1 -- 172.21.15.83:0/3074615728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfdc152b20 msgr2=0x7fdfdc1f1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdbfff640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfdc152b20 0x7fdfdc1f1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdbfff640 1 -- 172.21.15.83:0/3074615728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfdc15f380 con 0x7fdfdc150290 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe0bdf640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfdc0c45c0 0x7fdfdc150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfdbfff640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfdc150290 0x7fdfdc0c0c80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0ae040 tx=0x7fdfcc01ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfd97fa640 1 -- 172.21.15.83:0/3074615728 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc0b0050 con 0x7fdfdc150290 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfd97fa640 1 -- 172.21.15.83:0/3074615728 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfcc0b1040 con 0x7fdfdc150290 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 -- 172.21.15.83:0/3074615728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfdc15ee00 con 0x7fdfdc150290 2024-09-19T07:23:53.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfd97fa640 1 -- 172.21.15.83:0/3074615728 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc004260 con 0x7fdfdc150290 2024-09-19T07:23:53.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.662+0000 7fdfe2e43640 1 -- 172.21.15.83:0/3074615728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfdc15f0a0 con 0x7fdfdc150290 2024-09-19T07:23:53.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.666+0000 7fdfe2e43640 1 -- 172.21.15.83:0/3074615728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfdc0c3e90 con 0x7fdfdc150290 2024-09-19T07:23:53.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.666+0000 7fdfd97fa640 1 -- 172.21.15.83:0/3074615728 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdfcc0be060 con 0x7fdfdc150290 2024-09-19T07:23:53.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.666+0000 7fdfd97fa640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdfc80678c0 0x7fdfc8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.666+0000 7fdfe0bdf640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdfc80678c0 0x7fdfc8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.666+0000 7fdfd97fa640 1 -- 172.21.15.83:0/3074615728 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7fdfcc11fc70 con 0x7fdfdc150290 2024-09-19T07:23:53.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.666+0000 7fdfe0bdf640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdfc80678c0 0x7fdfc8069d80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0000f70 tx=0x7fdfd0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:53.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.670+0000 7fdfd97fa640 1 -- 172.21.15.83:0/3074615728 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdfcc016090 con 0x7fdfdc150290 2024-09-19T07:23:53.807 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:53.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.866+0000 7fdfe2e43640 1 -- 172.21.15.83:0/3074615728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfdc06ea80 con 0x7fdfdc150290 2024-09-19T07:23:53.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.866+0000 7fdfd97fa640 1 -- 172.21.15.83:0/3074615728 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5700 (secure 0 0 0) 0x7fdfcc0f01a0 con 0x7fdfdc150290 2024-09-19T07:23:53.868 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:53.868 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":45,"btime":"2024-09-19T07:23:53:168427+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:52.469054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:53.868 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 45 2024-09-19T07:23:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.870+0000 7fdfe2e43640 1 -- 172.21.15.83:0/3074615728 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdfc80678c0 msgr2=0x7fdfc8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.870+0000 7fdfe2e43640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdfc80678c0 0x7fdfc8069d80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0000f70 tx=0x7fdfd0009040 comp rx=0 tx=0).stop 2024-09-19T07:23:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.870+0000 7fdfe2e43640 1 -- 172.21.15.83:0/3074615728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfdc150290 msgr2=0x7fdfdc0c0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.870+0000 7fdfe2e43640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfdc150290 0x7fdfdc0c0c80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0ae040 tx=0x7fdfcc01ace0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.870+0000 7fdfe2e43640 1 -- 172.21.15.83:0/3074615728 shutdown_connections 2024-09-19T07:23:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.870+0000 7fdfe2e43640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdfc80678c0 0x7fdfc8069d80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.870+0000 7fdfe2e43640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfdc152b20 0x7fdfdc1f1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.870+0000 7fdfe2e43640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfdc150290 0x7fdfdc0c0c80 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.870+0000 7fdfe2e43640 1 --2- 172.21.15.83:0/3074615728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfdc0c45c0 0x7fdfdc150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.870+0000 7fdfe2e43640 1 -- 172.21.15.83:0/3074615728 >> 172.21.15.83:0/3074615728 conn(0x7fdfdc06bc80 msgr2=0x7fdfdc14ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.870+0000 7fdfe2e43640 1 -- 172.21.15.83:0/3074615728 shutdown_connections 2024-09-19T07:23:53.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.870+0000 7fdfe2e43640 1 -- 172.21.15.83:0/3074615728 wait complete. 2024-09-19T07:23:53.888 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:23:53:168427+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 45, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 45, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5463": { "addr": "172.21.15.83:6837/26497755", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 26497755, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 26497755, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5463, "incarnation": 44, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:23:52.469054+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5463 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 42, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5304": { "addr": "172.21.15.83:6835/3442755673", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 3442755673, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 3442755673, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5304, "incarnation": 39, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:23:41.116073+0000", "qdb_cluster": [ 5304 ], "qdb_leader": 5304, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5304 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 28, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:23:53.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.922+0000 7f15fdc23640 1 Processor -- start 2024-09-19T07:23:53.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.922+0000 7f15fdc23640 1 -- start start 2024-09-19T07:23:53.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.922+0000 7f15fdc23640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15f80c6120 0x7f15f80c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.922+0000 7f15fdc23640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f15f81592f0 0x7f15f8157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.922+0000 7f15fdc23640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15f8156200 con 0x7f15f8155d70 2024-09-19T07:23:53.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.922+0000 7f15fdc23640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f15f80c9930 con 0x7f15f81592f0 2024-09-19T07:23:53.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.922+0000 7f15fdc23640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f15f81564e0 con 0x7f15f80c6120 2024-09-19T07:23:53.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.922+0000 7f15f7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f15f81592f0 0x7f15f8157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.922+0000 7f15f7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f15f81592f0 0x7f15f8157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49026/0 (socket says 172.21.15.83:49026) 2024-09-19T07:23:53.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.922+0000 7f15f7fff640 1 -- 172.21.15.83:0/3363762847 learned_addr learned my addr 172.21.15.83:0/3363762847 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:53.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15f7fff640 1 -- 172.21.15.83:0/3363762847 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15f80c6120 msgr2=0x7f15f80c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:53.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15f77fe640 1 --2- 172.21.15.83:0/3363762847 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15f80c6120 0x7f15f80c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15f7fff640 1 --2- 172.21.15.83:0/3363762847 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15f80c6120 0x7f15f80c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15f7fff640 1 -- 172.21.15.83:0/3363762847 >> v1:172.21.15.184:6789/0 conn(0x7f15f8155d70 legacy=0x7f15f81575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:53.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15f7fff640 1 -- 172.21.15.83:0/3363762847 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f81567c0 con 0x7f15f81592f0 2024-09-19T07:23:53.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15f7fff640 1 --2- 172.21.15.83:0/3363762847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15f81592f0 0x7f15f8157ce0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009e50 tx=0x7f15f001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a224b526a751c423 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:53.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15f6ffd640 1 -- 172.21.15.83:0/3363762847 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f00a5670 con 0x7f15f81592f0 2024-09-19T07:23:53.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15f6ffd640 1 -- 172.21.15.83:0/3363762847 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15f00a5810 con 0x7f15f81592f0 2024-09-19T07:23:53.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15f6ffd640 1 -- 172.21.15.83:0/3363762847 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f00a5be0 con 0x7f15f81592f0 2024-09-19T07:23:53.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15fdc23640 1 -- 172.21.15.83:0/3363762847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15f81592f0 msgr2=0x7f15f8157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15fdc23640 1 --2- 172.21.15.83:0/3363762847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15f81592f0 0x7f15f8157ce0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009e50 tx=0x7f15f001c9f0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15fdc23640 1 -- 172.21.15.83:0/3363762847 shutdown_connections 2024-09-19T07:23:53.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15fdc23640 1 --2- 172.21.15.83:0/3363762847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15f81592f0 0x7f15f8157ce0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15fdc23640 1 --2- 172.21.15.83:0/3363762847 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15f80c6120 0x7f15f80c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15fdc23640 1 -- 172.21.15.83:0/3363762847 >> 172.21.15.83:0/3363762847 conn(0x7f15f806bc80 msgr2=0x7f15f806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:53.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15fdc23640 1 -- 172.21.15.83:0/3363762847 shutdown_connections 2024-09-19T07:23:53.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.926+0000 7f15fdc23640 1 -- 172.21.15.83:0/3363762847 wait complete. 2024-09-19T07:23:53.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15fdc23640 1 Processor -- start 2024-09-19T07:23:53.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15fdc23640 1 -- start start 2024-09-19T07:23:53.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15fdc23640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15f80c6120 0x7f15f80aca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15fdc23640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15f81592f0 0x7f15f80ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15fdc23640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15f814f790 0x7f15f80abae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15fdc23640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f15f8153860 con 0x7f15f81592f0 2024-09-19T07:23:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15fdc23640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f15f8162ea0 con 0x7f15f814f790 2024-09-19T07:23:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15fdc23640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f15f80acfb0 con 0x7f15f80c6120 2024-09-19T07:23:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15effff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15f81592f0 0x7f15f80ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15effff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15f81592f0 0x7f15f80ab5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49028/0 (socket says 172.21.15.83:49028) 2024-09-19T07:23:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15effff640 1 -- 172.21.15.83:0/1563053257 learned_addr learned my addr 172.21.15.83:0/1563053257 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15effff640 1 -- 172.21.15.83:0/1563053257 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15f80c6120 msgr2=0x7f15f80aca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15effff640 1 --2- 172.21.15.83:0/1563053257 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15f80c6120 0x7f15f80aca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15effff640 1 -- 172.21.15.83:0/1563053257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15f814f790 msgr2=0x7f15f80abae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15effff640 1 --2- 172.21.15.83:0/1563053257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15f814f790 0x7f15f80abae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15effff640 1 -- 172.21.15.83:0/1563053257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f814e680 con 0x7f15f81592f0 2024-09-19T07:23:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15effff640 1 --2- 172.21.15.83:0/1563053257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15f81592f0 0x7f15f80ab5a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f15e000c440 tx=0x7f15e000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15f4ff9640 1 -- 172.21.15.83:0/1563053257 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e0016030 con 0x7f15f81592f0 2024-09-19T07:23:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15f4ff9640 1 -- 172.21.15.83:0/1563053257 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15e0005020 con 0x7f15f81592f0 2024-09-19T07:23:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15f4ff9640 1 -- 172.21.15.83:0/1563053257 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e00053f0 con 0x7f15f81592f0 2024-09-19T07:23:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15fdc23640 1 -- 172.21.15.83:0/1563053257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f814e320 con 0x7f15f81592f0 2024-09-19T07:23:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15fdc23640 1 -- 172.21.15.83:0/1563053257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f814db20 con 0x7f15f81592f0 2024-09-19T07:23:53.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15fdc23640 1 -- 172.21.15.83:0/1563053257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15f80c6500 con 0x7f15f81592f0 2024-09-19T07:23:53.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.930+0000 7f15f4ff9640 1 -- 172.21.15.83:0/1563053257 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f15e0007020 con 0x7f15f81592f0 2024-09-19T07:23:53.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.934+0000 7f15f4ff9640 1 --2- 172.21.15.83:0/1563053257 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15d80678c0 0x7f15d8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:53.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.934+0000 7f15f4ff9640 1 -- 172.21.15.83:0/1563053257 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7f15e0085c40 con 0x7f15f81592f0 2024-09-19T07:23:53.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.934+0000 7f15f77fe640 1 --2- 172.21.15.83:0/1563053257 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15d80678c0 0x7f15d8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:53.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.934+0000 7f15f77fe640 1 --2- 172.21.15.83:0/1563053257 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15d80678c0 0x7f15d8069d80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f15e800a8b0 tx=0x7f15e8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:53.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:53.938+0000 7f15f4ff9640 1 -- 172.21.15.83:0/1563053257 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f15e0056210 con 0x7f15f81592f0 2024-09-19T07:23:54.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15fdc23640 1 -- 172.21.15.83:0/1563053257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15f80c96a0 con 0x7f15f81592f0 2024-09-19T07:23:54.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15f4ff9640 1 -- 172.21.15.83:0/1563053257 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5700 (secure 0 0 0) 0x7f15e0055b10 con 0x7f15f81592f0 2024-09-19T07:23:54.127 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:54.127 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":45,"btime":"2024-09-19T07:23:53:168427+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:52.469054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:54.127 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 45 2024-09-19T07:23:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15fdc23640 1 -- 172.21.15.83:0/1563053257 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15d80678c0 msgr2=0x7f15d8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15fdc23640 1 --2- 172.21.15.83:0/1563053257 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15d80678c0 0x7f15d8069d80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f15e800a8b0 tx=0x7f15e8008040 comp rx=0 tx=0).stop 2024-09-19T07:23:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15fdc23640 1 -- 172.21.15.83:0/1563053257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15f81592f0 msgr2=0x7f15f80ab5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15fdc23640 1 --2- 172.21.15.83:0/1563053257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15f81592f0 0x7f15f80ab5a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f15e000c440 tx=0x7f15e000c900 comp rx=0 tx=0).stop 2024-09-19T07:23:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15fdc23640 1 -- 172.21.15.83:0/1563053257 shutdown_connections 2024-09-19T07:23:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15fdc23640 1 --2- 172.21.15.83:0/1563053257 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15d80678c0 0x7f15d8069d80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15fdc23640 1 --2- 172.21.15.83:0/1563053257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15f814f790 0x7f15f80abae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15fdc23640 1 --2- 172.21.15.83:0/1563053257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15f81592f0 0x7f15f80ab5a0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15fdc23640 1 --2- 172.21.15.83:0/1563053257 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15f80c6120 0x7f15f80aca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15fdc23640 1 -- 172.21.15.83:0/1563053257 >> 172.21.15.83:0/1563053257 conn(0x7f15f806bc80 msgr2=0x7f15f8154050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.125+0000 7f15fdc23640 1 -- 172.21.15.83:0/1563053257 shutdown_connections 2024-09-19T07:23:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:54.129+0000 7f15fdc23640 1 -- 172.21.15.83:0/1563053257 wait complete. 2024-09-19T07:23:54.144 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-19T07:23:55.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:56.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fedb3640 1 Processor -- start 2024-09-19T07:23:56.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fedb3640 1 -- start start 2024-09-19T07:23:56.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fedb3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc9f80c3fa0 0x7fc9f8070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fedb3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9f80aa230 con 0x7fc9f80c3360 2024-09-19T07:23:56.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fedb3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9f80c28a0 con 0x7fc9f80ab3c0 2024-09-19T07:23:56.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fedb3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc9f8074190 con 0x7fc9f80c3fa0 2024-09-19T07:23:56.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fd350640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc9f80c3360 0x7fc9f8166330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51424/0 (socket says 172.21.15.83:51424) 2024-09-19T07:23:56.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fd350640 1 -- 172.21.15.83:0/1684902756 learned_addr learned my addr 172.21.15.83:0/1684902756 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:23:56.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fcb4f640 1 --2- 172.21.15.83:0/1684902756 >> v2:172.21.15.184:3300/0 conn(0x7fc9f80c3fa0 0x7fc9f8070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560811394 0 0) 0x7fc9f80aa230 con 0x7fc9f80c3360 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e0003530 con 0x7fc9f80c3360 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 606517383 0 0) 0x7fc9e0003530 con 0x7fc9f80c3360 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 >> v2:172.21.15.184:3300/0 conn(0x7fc9f80c3fa0 msgr2=0x7fc9f8070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 --2- 172.21.15.83:0/1684902756 >> v2:172.21.15.184:3300/0 conn(0x7fc9f80c3fa0 0x7fc9f8070ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 >> v1:172.21.15.184:6789/0 conn(0x7fc9f80ab3c0 legacy=0x7fc9f80c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f80c2240 con 0x7fc9f80c3360 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fc9e8001320 con 0x7fc9f80c3360 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 >> v1:172.21.15.83:6789/0 conn(0x7fc9f80c3360 legacy=0x7fc9f8166330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e00039c0 0x7fc9e0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9e0004810 0x7fc9e0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e00051d0 0x7fc9e000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e000c140 con 0x7fc9e00051d0 2024-09-19T07:23:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e000c2f0 con 0x7fc9e00039c0 2024-09-19T07:23:56.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e000c4f0 con 0x7fc9e0004810 2024-09-19T07:23:56.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fd350640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e00051d0 0x7fc9e000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fcb4f640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e00039c0 0x7fc9e0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f7fff640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9e0004810 0x7fc9e0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fd350640 1 -- 172.21.15.83:0/1684902756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9e0004810 msgr2=0x7fc9e0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:56.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fd350640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9e0004810 0x7fc9e0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fd350640 1 -- 172.21.15.83:0/1684902756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e00039c0 msgr2=0x7fc9e0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:56.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fd350640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e00039c0 0x7fc9e0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fd350640 1 -- 172.21.15.83:0/1684902756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e0003530 con 0x7fc9e00051d0 2024-09-19T07:23:56.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9f7fff640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9e0004810 0x7fc9e0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:23:56.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.017+0000 7fc9fcb4f640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e00039c0 0x7fc9e0003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:23:56.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.021+0000 7fc9fd350640 1 -- 172.21.15.83:0/1684902756 reap_dead start 2024-09-19T07:23:56.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.021+0000 7fc9fd350640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e00051d0 0x7fc9e000b9d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800c4e0 tx=0x7fc9e8021910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ead31acbe8e16c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:56.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.021+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e800c100 con 0x7fc9e00051d0 2024-09-19T07:23:56.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.021+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e80af6a0 con 0x7fc9e00051d0 2024-09-19T07:23:56.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.021+0000 7fc9f77fe640 1 -- 172.21.15.83:0/1684902756 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80afa90 con 0x7fc9e00051d0 2024-09-19T07:23:56.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.021+0000 7fc9fedb3640 1 -- 172.21.15.83:0/1684902756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e00051d0 msgr2=0x7fc9e000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.021+0000 7fc9fedb3640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e00051d0 0x7fc9e000b9d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800c4e0 tx=0x7fc9e8021910 comp rx=0 tx=0).stop 2024-09-19T07:23:56.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.021+0000 7fc9fedb3640 1 -- 172.21.15.83:0/1684902756 shutdown_connections 2024-09-19T07:23:56.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.021+0000 7fc9fedb3640 1 --2- 172.21.15.83:0/1684902756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e00051d0 0x7fc9e000b9d0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.021+0000 7fc9fedb3640 1 -- 172.21.15.83:0/1684902756 >> 172.21.15.83:0/1684902756 conn(0x7fc9f8075390 msgr2=0x7fc9f806bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:56.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.021+0000 7fc9fedb3640 1 -- 172.21.15.83:0/1684902756 shutdown_connections 2024-09-19T07:23:56.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.021+0000 7fc9fedb3640 1 -- 172.21.15.83:0/1684902756 wait complete. 2024-09-19T07:23:56.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fedb3640 1 Processor -- start 2024-09-19T07:23:56.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fedb3640 1 -- start start 2024-09-19T07:23:56.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fedb3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9e00051d0 0x7fc9f8161ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fedb3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9f80c0c80 0x7fc9f8082370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fedb3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f80828b0 0x7fc9f8071d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fedb3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f80c2e40 con 0x7fc9f80828b0 2024-09-19T07:23:56.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fedb3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f80c2b60 con 0x7fc9f80c0c80 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fedb3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f80c1ff0 con 0x7fc9e00051d0 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fd350640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f80828b0 0x7fc9f8071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fd350640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f80828b0 0x7fc9f8071d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49054/0 (socket says 172.21.15.83:49054) 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fd350640 1 -- 172.21.15.83:0/2274344390 learned_addr learned my addr 172.21.15.83:0/2274344390 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fcb4f640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9e00051d0 0x7fc9f8161ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9f7fff640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9f80c0c80 0x7fc9f8082370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fd350640 1 -- 172.21.15.83:0/2274344390 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9e00051d0 msgr2=0x7fc9f8161ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fd350640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9e00051d0 0x7fc9f8161ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fd350640 1 -- 172.21.15.83:0/2274344390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9f80c0c80 msgr2=0x7fc9f8082370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fd350640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9f80c0c80 0x7fc9f8082370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fd350640 1 -- 172.21.15.83:0/2274344390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f81623e0 con 0x7fc9f80828b0 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9f7fff640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9f80c0c80 0x7fc9f8082370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:23:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fd350640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f80828b0 0x7fc9f8071d00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80221b0 tx=0x7fc9e0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:56.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9f57fa640 1 -- 172.21.15.83:0/2274344390 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e800c100 con 0x7fc9f80828b0 2024-09-19T07:23:56.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9f57fa640 1 -- 172.21.15.83:0/2274344390 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e80bd070 con 0x7fc9f80828b0 2024-09-19T07:23:56.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9f57fa640 1 -- 172.21.15.83:0/2274344390 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80b0ec0 con 0x7fc9f80828b0 2024-09-19T07:23:56.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fedb3640 1 -- 172.21.15.83:0/2274344390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f81fc000 con 0x7fc9f80828b0 2024-09-19T07:23:56.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fedb3640 1 -- 172.21.15.83:0/2274344390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f8071220 con 0x7fc9f80828b0 2024-09-19T07:23:56.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.025+0000 7fc9fedb3640 1 -- 172.21.15.83:0/2274344390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f80c3790 con 0x7fc9f80828b0 2024-09-19T07:23:56.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.029+0000 7fc9f57fa640 1 -- 172.21.15.83:0/2274344390 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc9e8001120 con 0x7fc9f80828b0 2024-09-19T07:23:56.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.029+0000 7fc9f57fa640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9e4067a90 0x7fc9e4069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.029+0000 7fc9f57fa640 1 -- 172.21.15.83:0/2274344390 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7fc9e8123dd0 con 0x7fc9f80828b0 2024-09-19T07:23:56.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.029+0000 7fc9fcb4f640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9e4067a90 0x7fc9e4069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.029+0000 7fc9fcb4f640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9e4067a90 0x7fc9e4069f50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000a6d0 tx=0x7fc9f000c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:56.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.033+0000 7fc9f57fa640 1 -- 172.21.15.83:0/2274344390 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc9e801b620 con 0x7fc9f80828b0 2024-09-19T07:23:56.145 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:56.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3fca7640 1 Processor -- start 2024-09-19T07:23:56.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3fca7640 1 -- start start 2024-09-19T07:23:56.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3fca7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb380c6120 0x7feb380c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3fca7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feb38155d70 0x7feb381575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3fca7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb38156200 con 0x7feb381592f0 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3fca7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7feb380c9930 con 0x7feb38155d70 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3fca7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feb381564e0 con 0x7feb380c6120 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3d242640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feb38155d70 0x7feb381575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3d242640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feb38155d70 0x7feb381575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49056/0 (socket says 172.21.15.83:49056) 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3e244640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7feb381592f0 0x7feb38157b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51426/0 (socket says 172.21.15.83:51426) 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3d242640 1 -- 172.21.15.83:0/2628189839 learned_addr learned my addr 172.21.15.83:0/2628189839 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3da43640 1 --2- 172.21.15.83:0/2628189839 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb380c6120 0x7feb380c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3d242640 1 -- 172.21.15.83:0/2628189839 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb380c6120 msgr2=0x7feb380c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3d242640 1 --2- 172.21.15.83:0/2628189839 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb380c6120 0x7feb380c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3d242640 1 -- 172.21.15.83:0/2628189839 >> v1:172.21.15.83:6789/0 conn(0x7feb381592f0 legacy=0x7feb38157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3d242640 1 -- 172.21.15.83:0/2628189839 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb381567c0 con 0x7feb38155d70 2024-09-19T07:23:56.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.281+0000 7feb3ca41640 1 -- 172.21.15.83:0/2628189839 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 909627533 0 0) 0x7feb38156200 con 0x7feb381592f0 2024-09-19T07:23:56.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7feb3d242640 1 --2- 172.21.15.83:0/2628189839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb38155d70 0x7feb381575d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7feb28009870 tx=0x7feb2801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616dc0ffd37e3489 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:56.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7feb3ca41640 1 -- 172.21.15.83:0/2628189839 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb280b2070 con 0x7feb38155d70 2024-09-19T07:23:56.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7feb3ca41640 1 -- 172.21.15.83:0/2628189839 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb280a9070 con 0x7feb38155d70 2024-09-19T07:23:56.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7feb3ca41640 1 -- 172.21.15.83:0/2628189839 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb280ad880 con 0x7feb38155d70 2024-09-19T07:23:56.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7fc9fedb3640 1 -- 172.21.15.83:0/2274344390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9f80c7b70 con 0x7fc9f80828b0 2024-09-19T07:23:56.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7feb3fca7640 1 -- 172.21.15.83:0/2628189839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb38155d70 msgr2=0x7feb381575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7feb3fca7640 1 --2- 172.21.15.83:0/2628189839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb38155d70 0x7feb381575d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7feb28009870 tx=0x7feb2801c560 comp rx=0 tx=0).stop 2024-09-19T07:23:56.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7feb3fca7640 1 -- 172.21.15.83:0/2628189839 shutdown_connections 2024-09-19T07:23:56.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7feb3fca7640 1 --2- 172.21.15.83:0/2628189839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb38155d70 0x7feb381575d0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7feb3fca7640 1 --2- 172.21.15.83:0/2628189839 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb380c6120 0x7feb380c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7feb3fca7640 1 -- 172.21.15.83:0/2628189839 >> 172.21.15.83:0/2628189839 conn(0x7feb3806bc80 msgr2=0x7feb3806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:56.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7fc9f57fa640 1 -- 172.21.15.83:0/2274344390 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+5704 (secure 0 0 0) 0x7fc9e8125020 con 0x7fc9f80828b0 2024-09-19T07:23:56.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7feb3fca7640 1 -- 172.21.15.83:0/2628189839 shutdown_connections 2024-09-19T07:23:56.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.285+0000 7feb3fca7640 1 -- 172.21.15.83:0/2628189839 wait complete. 2024-09-19T07:23:56.289 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:56.289 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":47,"btime":"2024-09-19T07:23:55:188288+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:56.289 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 47 2024-09-19T07:23:56.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3fca7640 1 Processor -- start 2024-09-19T07:23:56.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3fca7640 1 -- start start 2024-09-19T07:23:56.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7fc9fedb3640 1 -- 172.21.15.83:0/2274344390 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9e4067a90 msgr2=0x7fc9e4069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7fc9fedb3640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9e4067a90 0x7fc9e4069f50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000a6d0 tx=0x7fc9f000c040 comp rx=0 tx=0).stop 2024-09-19T07:23:56.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7fc9fedb3640 1 -- 172.21.15.83:0/2274344390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f80828b0 msgr2=0x7fc9f8071d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7fc9fedb3640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f80828b0 0x7fc9f8071d00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80221b0 tx=0x7fc9e0005840 comp rx=0 tx=0).stop 2024-09-19T07:23:56.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3fca7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb380c6120 0x7feb380c1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3fca7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb381592f0 0x7feb38149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3fca7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3814baa0 0x7feb3814a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3fca7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feb38153860 con 0x7feb3814baa0 2024-09-19T07:23:56.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3fca7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feb38162ea0 con 0x7feb381592f0 2024-09-19T07:23:56.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3fca7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feb3814b3e0 con 0x7feb380c6120 2024-09-19T07:23:56.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7fc9fedb3640 1 -- 172.21.15.83:0/2274344390 shutdown_connections 2024-09-19T07:23:56.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7fc9fedb3640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9e4067a90 0x7fc9e4069f50 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7fc9fedb3640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f80828b0 0x7fc9f8071d00 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7fc9fedb3640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9f80c0c80 0x7fc9f8082370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7fc9fedb3640 1 --2- 172.21.15.83:0/2274344390 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9e00051d0 0x7fc9f8161ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7fc9fedb3640 1 -- 172.21.15.83:0/2274344390 >> 172.21.15.83:0/2274344390 conn(0x7fc9f8075390 msgr2=0x7fc9f8073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:56.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7fc9fedb3640 1 -- 172.21.15.83:0/2274344390 shutdown_connections 2024-09-19T07:23:56.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7fc9fedb3640 1 -- 172.21.15.83:0/2274344390 wait complete. 2024-09-19T07:23:56.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3e244640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3814baa0 0x7feb3814a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3da43640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb380c6120 0x7feb380c1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3e244640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3814baa0 0x7feb3814a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49064/0 (socket says 172.21.15.83:49064) 2024-09-19T07:23:56.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3e244640 1 -- 172.21.15.83:0/43043394 learned_addr learned my addr 172.21.15.83:0/43043394 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:56.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3d242640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb381592f0 0x7feb38149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3e244640 1 -- 172.21.15.83:0/43043394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb380c6120 msgr2=0x7feb380c1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3e244640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb380c6120 0x7feb380c1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3e244640 1 -- 172.21.15.83:0/43043394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb381592f0 msgr2=0x7feb38149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3e244640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb381592f0 0x7feb38149e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3e244640 1 -- 172.21.15.83:0/43043394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3814fb20 con 0x7feb3814baa0 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3e244640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3814baa0 0x7feb3814a340 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7feb3000c700 tx=0x7feb3000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb2e7fc640 1 -- 172.21.15.83:0/43043394 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30018070 con 0x7feb3814baa0 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb2e7fc640 1 -- 172.21.15.83:0/43043394 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb3000ce10 con 0x7feb3814baa0 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb2e7fc640 1 -- 172.21.15.83:0/43043394 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3000a880 con 0x7feb3814baa0 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3fca7640 1 -- 172.21.15.83:0/43043394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb381ef1e0 con 0x7feb3814baa0 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.289+0000 7feb3fca7640 1 -- 172.21.15.83:0/43043394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb380c23d0 con 0x7feb3814baa0 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.293+0000 7feb3fca7640 1 -- 172.21.15.83:0/43043394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb380ada60 con 0x7feb3814baa0 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.293+0000 7feb2e7fc640 1 -- 172.21.15.83:0/43043394 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feb3000aa20 con 0x7feb3814baa0 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.293+0000 7feb2e7fc640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb1c0678f0 0x7feb1c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.293+0000 7feb2e7fc640 1 -- 172.21.15.83:0/43043394 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7feb30085670 con 0x7feb3814baa0 2024-09-19T07:23:56.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.293+0000 7feb3da43640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb1c0678f0 0x7feb1c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:56.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.293+0000 7feb3da43640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb1c0678f0 0x7feb1c069db0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feb3400b9d0 tx=0x7feb34008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:56.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.297+0000 7feb2e7fc640 1 -- 172.21.15.83:0/43043394 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7feb30055c40 con 0x7feb3814baa0 2024-09-19T07:23:56.309 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=4786 2024-09-19T07:23:56.309 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.b 2024-09-19T07:23:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.525+0000 7feb3fca7640 1 -- 172.21.15.83:0/43043394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb380c6500 con 0x7feb3814baa0 2024-09-19T07:23:56.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.525+0000 7feb2e7fc640 1 -- 172.21.15.83:0/43043394 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+5704 (secure 0 0 0) 0x7feb30055540 con 0x7feb3814baa0 2024-09-19T07:23:56.526 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:56.526 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":47,"btime":"2024-09-19T07:23:55:188288+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:56.527 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 47 2024-09-19T07:23:56.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.525+0000 7feb3fca7640 1 -- 172.21.15.83:0/43043394 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb1c0678f0 msgr2=0x7feb1c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.525+0000 7feb3fca7640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb1c0678f0 0x7feb1c069db0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feb3400b9d0 tx=0x7feb34008040 comp rx=0 tx=0).stop 2024-09-19T07:23:56.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.525+0000 7feb3fca7640 1 -- 172.21.15.83:0/43043394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3814baa0 msgr2=0x7feb3814a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:56.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.525+0000 7feb3fca7640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3814baa0 0x7feb3814a340 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7feb3000c700 tx=0x7feb3000cbc0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.525+0000 7feb3fca7640 1 -- 172.21.15.83:0/43043394 shutdown_connections 2024-09-19T07:23:56.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.525+0000 7feb3fca7640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb1c0678f0 0x7feb1c069db0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.525+0000 7feb3fca7640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3814baa0 0x7feb3814a340 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.525+0000 7feb3fca7640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb381592f0 0x7feb38149e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.525+0000 7feb3fca7640 1 --2- 172.21.15.83:0/43043394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb380c6120 0x7feb380c1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:56.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.525+0000 7feb3fca7640 1 -- 172.21.15.83:0/43043394 >> 172.21.15.83:0/43043394 conn(0x7feb3806bc80 msgr2=0x7feb381574d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:56.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.529+0000 7feb3fca7640 1 -- 172.21.15.83:0/43043394 shutdown_connections 2024-09-19T07:23:56.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:56.529+0000 7feb3fca7640 1 -- 172.21.15.83:0/43043394 wait complete. 2024-09-19T07:23:56.549 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:active state 2024-09-19T07:23:56.549 INFO:tasks.mds_thrash.fs.[b]:waiting for 32 secs before thrashing 2024-09-19T07:23:57.169 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:23:57.169+0000 7efd9f754640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:23:57.310 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-09-19T07:23:57.311 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-19T07:23:57.311 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-19T07:23:57.313 INFO:tasks.ceph.mds.b:Started 2024-09-19T07:23:57.314 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:57.391 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-19T07:23:57.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.441+0000 7f6182e18640 1 Processor -- start 2024-09-19T07:23:57.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.441+0000 7f6182e18640 1 -- start start 2024-09-19T07:23:57.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f6182e18640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f617c06d6f0 0x7f617c154490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:57.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f6182e18640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f617c0abad0 0x7f617c06f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:57.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f6182e18640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f617c151650 con 0x7f617c06f690 2024-09-19T07:23:57.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f6182e18640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f617c06fc10 con 0x7f617c0abad0 2024-09-19T07:23:57.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f6182e18640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f617c06fef0 con 0x7f617c06d6f0 2024-09-19T07:23:57.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f617bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f617c0abad0 0x7f617c06f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:57.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f617bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f617c0abad0 0x7f617c06f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49070/0 (socket says 172.21.15.83:49070) 2024-09-19T07:23:57.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f617bfff640 1 -- 172.21.15.83:0/3199093627 learned_addr learned my addr 172.21.15.83:0/3199093627 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:57.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f6180bb4640 1 --2- 172.21.15.83:0/3199093627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f617c06d6f0 0x7f617c154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:57.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f617bfff640 1 -- 172.21.15.83:0/3199093627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f617c06d6f0 msgr2=0x7f617c154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:57.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f617bfff640 1 --2- 172.21.15.83:0/3199093627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f617c06d6f0 0x7f617c154490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:57.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f617bfff640 1 -- 172.21.15.83:0/3199093627 >> v1:172.21.15.184:6789/0 conn(0x7f617c06f690 legacy=0x7f617c150a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:57.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f617bfff640 1 -- 172.21.15.83:0/3199093627 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617c06e670 con 0x7f617c0abad0 2024-09-19T07:23:57.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f6180bb4640 1 --2- 172.21.15.83:0/3199093627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f617c06d6f0 0x7f617c154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:23:57.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f617bfff640 1 --2- 172.21.15.83:0/3199093627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c0abad0 0x7f617c06f150 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6168009870 tx=0x7f616801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a4f4e0f50158b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:57.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f617b7fe640 1 -- 172.21.15.83:0/3199093627 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61680b1070 con 0x7f617c0abad0 2024-09-19T07:23:57.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f617b7fe640 1 -- 172.21.15.83:0/3199093627 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61680ac620 con 0x7f617c0abad0 2024-09-19T07:23:57.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f617b7fe640 1 -- 172.21.15.83:0/3199093627 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61680ac9f0 con 0x7f617c0abad0 2024-09-19T07:23:57.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f6182e18640 1 -- 172.21.15.83:0/3199093627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c0abad0 msgr2=0x7f617c06f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:57.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f6182e18640 1 --2- 172.21.15.83:0/3199093627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c0abad0 0x7f617c06f150 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6168009870 tx=0x7f616801c4d0 comp rx=0 tx=0).stop 2024-09-19T07:23:57.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f6182e18640 1 -- 172.21.15.83:0/3199093627 shutdown_connections 2024-09-19T07:23:57.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f6182e18640 1 --2- 172.21.15.83:0/3199093627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c0abad0 0x7f617c06f150 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:57.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.445+0000 7f6182e18640 1 --2- 172.21.15.83:0/3199093627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f617c06d6f0 0x7f617c154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:57.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 -- 172.21.15.83:0/3199093627 >> 172.21.15.83:0/3199093627 conn(0x7f617c0c7e40 msgr2=0x7f617c0c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:57.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 -- 172.21.15.83:0/3199093627 shutdown_connections 2024-09-19T07:23:57.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 -- 172.21.15.83:0/3199093627 wait complete. 2024-09-19T07:23:57.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 Processor -- start 2024-09-19T07:23:57.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 -- start start 2024-09-19T07:23:57.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c06d6f0 0x7f617c1ecdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:57.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c06f690 0x7f617c146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:57.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f617c0abad0 0x7f617c1480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:57.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f617c06dad0 con 0x7f617c06f690 2024-09-19T07:23:57.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f617c068810 con 0x7f617c06d6f0 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f617c06dc50 con 0x7f617c0abad0 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f617bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c06f690 0x7f617c146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f617bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c06f690 0x7f617c146d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49072/0 (socket says 172.21.15.83:49072) 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f617bfff640 1 -- 172.21.15.83:0/2470241568 learned_addr learned my addr 172.21.15.83:0/2470241568 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6180bb4640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c06d6f0 0x7f617c1ecdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f61813b5640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f617c0abad0 0x7f617c1480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f617bfff640 1 -- 172.21.15.83:0/2470241568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f617c0abad0 msgr2=0x7f617c1480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f617bfff640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f617c0abad0 0x7f617c1480f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f617bfff640 1 -- 172.21.15.83:0/2470241568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c06d6f0 msgr2=0x7f617c1ecdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f617bfff640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c06d6f0 0x7f617c1ecdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f617bfff640 1 -- 172.21.15.83:0/2470241568 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617c1ec360 con 0x7f617c06f690 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6180bb4640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c06d6f0 0x7f617c1ecdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:23:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f617bfff640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c06f690 0x7f617c146d90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f61680af040 tx=0x7f6168019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f61797fa640 1 -- 172.21.15.83:0/2470241568 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61680b1040 con 0x7f617c06f690 2024-09-19T07:23:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f61797fa640 1 -- 172.21.15.83:0/2470241568 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61680ba070 con 0x7f617c06f690 2024-09-19T07:23:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f61797fa640 1 -- 172.21.15.83:0/2470241568 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61680b5680 con 0x7f617c06f690 2024-09-19T07:23:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 -- 172.21.15.83:0/2470241568 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f617c1523d0 con 0x7f617c06f690 2024-09-19T07:23:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.449+0000 7f6182e18640 1 -- 172.21.15.83:0/2470241568 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f617c1526b0 con 0x7f617c06f690 2024-09-19T07:23:57.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.453+0000 7f6182e18640 1 -- 172.21.15.83:0/2470241568 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f617c154490 con 0x7f617c06f690 2024-09-19T07:23:57.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.453+0000 7f61797fa640 1 -- 172.21.15.83:0/2470241568 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f61680b5820 con 0x7f617c06f690 2024-09-19T07:23:57.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.453+0000 7f61797fa640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61600678c0 0x7f6160069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:57.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.453+0000 7f61797fa640 1 -- 172.21.15.83:0/2470241568 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7f6168120050 con 0x7f617c06f690 2024-09-19T07:23:57.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.453+0000 7f6180bb4640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61600678c0 0x7f6160069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:57.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.453+0000 7f6180bb4640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61600678c0 0x7f6160069d80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f617c15d380 tx=0x7f6170008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:57.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.457+0000 7f61797fa640 1 -- 172.21.15.83:0/2470241568 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6168016150 con 0x7f617c06f690 2024-09-19T07:23:57.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.733+0000 7f6182e18640 1 -- 172.21.15.83:0/2470241568 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f617c06e240 con 0x7f617c06f690 2024-09-19T07:23:57.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.733+0000 7f61797fa640 1 -- 172.21.15.83:0/2470241568 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+5704 (secure 0 0 0) 0x7f61680f05a0 con 0x7f617c06f690 2024-09-19T07:23:57.737 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:23:57.737 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":47,"btime":"2024-09-19T07:23:55:188288+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:23:57.737 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 47 2024-09-19T07:23:57.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.737+0000 7f6182e18640 1 -- 172.21.15.83:0/2470241568 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61600678c0 msgr2=0x7f6160069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:57.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.737+0000 7f6182e18640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61600678c0 0x7f6160069d80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f617c15d380 tx=0x7f6170008040 comp rx=0 tx=0).stop 2024-09-19T07:23:57.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.737+0000 7f6182e18640 1 -- 172.21.15.83:0/2470241568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c06f690 msgr2=0x7f617c146d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:57.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.737+0000 7f6182e18640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c06f690 0x7f617c146d90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f61680af040 tx=0x7f6168019270 comp rx=0 tx=0).stop 2024-09-19T07:23:57.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.737+0000 7f6182e18640 1 -- 172.21.15.83:0/2470241568 shutdown_connections 2024-09-19T07:23:57.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.737+0000 7f6182e18640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61600678c0 0x7f6160069d80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:57.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.737+0000 7f6182e18640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f617c0abad0 0x7f617c1480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:57.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.737+0000 7f6182e18640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c06f690 0x7f617c146d90 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:57.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.737+0000 7f6182e18640 1 --2- 172.21.15.83:0/2470241568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f617c06d6f0 0x7f617c1ecdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:57.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.737+0000 7f6182e18640 1 -- 172.21.15.83:0/2470241568 >> 172.21.15.83:0/2470241568 conn(0x7f617c0c7e40 msgr2=0x7f617c150800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:57.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.737+0000 7f6182e18640 1 -- 172.21.15.83:0/2470241568 shutdown_connections 2024-09-19T07:23:57.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:57.737+0000 7f6182e18640 1 -- 172.21.15.83:0/2470241568 wait complete. 2024-09-19T07:23:57.756 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-19T07:23:59.756 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:23:59.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.885+0000 7fc218082640 1 Processor -- start 2024-09-19T07:23:59.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.885+0000 7fc218082640 1 -- start start 2024-09-19T07:23:59.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc218082640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2100c7d70 0x7fc210157c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:59.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc218082640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc2101547b0 0x7fc2100c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:59.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc218082640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc210073690 con 0x7fc210072d00 2024-09-19T07:23:59.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc218082640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc210157160 con 0x7fc2101547b0 2024-09-19T07:23:59.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc218082640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc210157440 con 0x7fc2100c7d70 2024-09-19T07:23:59.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc21561d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc2101547b0 0x7fc2100c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:59.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc21561d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc2101547b0 0x7fc2100c6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:36112/0 (socket says 172.21.15.83:36112) 2024-09-19T07:23:59.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc21561d640 1 -- 172.21.15.83:0/1523755816 learned_addr learned my addr 172.21.15.83:0/1523755816 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:59.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc215e1e640 1 --2- 172.21.15.83:0/1523755816 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2100c7d70 0x7fc210157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:59.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc215e1e640 1 -- 172.21.15.83:0/1523755816 >> v2:172.21.15.184:3300/0 conn(0x7fc2101547b0 msgr2=0x7fc2100c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:23:59.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc215e1e640 1 --2- 172.21.15.83:0/1523755816 >> v2:172.21.15.184:3300/0 conn(0x7fc2101547b0 0x7fc2100c6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:59.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc215e1e640 1 -- 172.21.15.83:0/1523755816 >> v1:172.21.15.184:6789/0 conn(0x7fc210072d00 legacy=0x7fc21006cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:59.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc215e1e640 1 -- 172.21.15.83:0/1523755816 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc210157720 con 0x7fc2100c7d70 2024-09-19T07:23:59.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.889+0000 7fc21561d640 1 --2- 172.21.15.83:0/1523755816 >> v2:172.21.15.184:3300/0 conn(0x7fc2101547b0 0x7fc2100c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:23:59.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc215e1e640 1 --2- 172.21.15.83:0/1523755816 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2100c7d70 0x7fc210157c40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0097c0 tx=0x7fc20c01c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9279b6031f5487a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:59.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc214e1c640 1 -- 172.21.15.83:0/1523755816 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c0b0070 con 0x7fc2100c7d70 2024-09-19T07:23:59.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc214e1c640 1 -- 172.21.15.83:0/1523755816 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc20c0ab600 con 0x7fc2100c7d70 2024-09-19T07:23:59.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc214e1c640 1 -- 172.21.15.83:0/1523755816 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c0ab9d0 con 0x7fc2100c7d70 2024-09-19T07:23:59.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc218082640 1 -- 172.21.15.83:0/1523755816 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2100c7d70 msgr2=0x7fc210157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:59.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc218082640 1 --2- 172.21.15.83:0/1523755816 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2100c7d70 0x7fc210157c40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0097c0 tx=0x7fc20c01c350 comp rx=0 tx=0).stop 2024-09-19T07:23:59.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc218082640 1 -- 172.21.15.83:0/1523755816 shutdown_connections 2024-09-19T07:23:59.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc218082640 1 --2- 172.21.15.83:0/1523755816 >> v2:172.21.15.184:3300/0 conn(0x7fc2101547b0 0x7fc2100c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:59.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc218082640 1 --2- 172.21.15.83:0/1523755816 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2100c7d70 0x7fc210157c40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:59.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc218082640 1 -- 172.21.15.83:0/1523755816 >> 172.21.15.83:0/1523755816 conn(0x7fc2100c5060 msgr2=0x7fc2100c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:23:59.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc218082640 1 -- 172.21.15.83:0/1523755816 shutdown_connections 2024-09-19T07:23:59.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc218082640 1 -- 172.21.15.83:0/1523755816 wait complete. 2024-09-19T07:23:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc218082640 1 Processor -- start 2024-09-19T07:23:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.893+0000 7fc218082640 1 -- start start 2024-09-19T07:23:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc218082640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc210072d00 0x7fc210153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc218082640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100c7d70 0x7fc210156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc218082640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2101547b0 0x7fc2101482c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc218082640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2100c74a0 con 0x7fc2100c7d70 2024-09-19T07:23:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc218082640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc2100c7320 con 0x7fc210072d00 2024-09-19T07:23:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc218082640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc210070230 con 0x7fc2101547b0 2024-09-19T07:23:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc21561d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100c7d70 0x7fc210156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc21561d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100c7d70 0x7fc210156000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49078/0 (socket says 172.21.15.83:49078) 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc21561d640 1 -- 172.21.15.83:0/2160365357 learned_addr learned my addr 172.21.15.83:0/2160365357 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc215e1e640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc210072d00 0x7fc210153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc21661f640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2101547b0 0x7fc2101482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc21561d640 1 -- 172.21.15.83:0/2160365357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2101547b0 msgr2=0x7fc2101482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc21561d640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2101547b0 0x7fc2101482c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc21561d640 1 -- 172.21.15.83:0/2160365357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc210072d00 msgr2=0x7fc210153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc21561d640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc210072d00 0x7fc210153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc21561d640 1 -- 172.21.15.83:0/2160365357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc210156f20 con 0x7fc2100c7d70 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc21561d640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100c7d70 0x7fc210156000 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc20000e9b0 tx=0x7fc20000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc2067fc640 1 -- 172.21.15.83:0/2160365357 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc200017070 con 0x7fc2100c7d70 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc2067fc640 1 -- 172.21.15.83:0/2160365357 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc20000c9a0 con 0x7fc2100c7d70 2024-09-19T07:23:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc218082640 1 -- 172.21.15.83:0/2160365357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2101eca60 con 0x7fc2100c7d70 2024-09-19T07:23:59.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc2067fc640 1 -- 172.21.15.83:0/2160365357 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20000cd90 con 0x7fc2100c7d70 2024-09-19T07:23:59.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc218082640 1 -- 172.21.15.83:0/2160365357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2101ed000 con 0x7fc2100c7d70 2024-09-19T07:23:59.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc218082640 1 -- 172.21.15.83:0/2160365357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc210070640 con 0x7fc2100c7d70 2024-09-19T07:23:59.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc2067fc640 1 -- 172.21.15.83:0/2160365357 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc20001a070 con 0x7fc2100c7d70 2024-09-19T07:23:59.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc2067fc640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1f40678c0 0x7fc1f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:23:59.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc2067fc640 1 -- 172.21.15.83:0/2160365357 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7fc200086090 con 0x7fc2100c7d70 2024-09-19T07:23:59.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.897+0000 7fc215e1e640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1f40678c0 0x7fc1f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:23:59.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.901+0000 7fc215e1e640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1f40678c0 0x7fc1f4069d80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0ad000 tx=0x7fc20c0a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:23:59.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:23:59.905+0000 7fc2067fc640 1 -- 172.21.15.83:0/2160365357 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc200056010 con 0x7fc2100c7d70 2024-09-19T07:24:00.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.137+0000 7fc218082640 1 -- 172.21.15.83:0/2160365357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc210073c40 con 0x7fc2100c7d70 2024-09-19T07:24:00.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.137+0000 7fc2067fc640 1 -- 172.21.15.83:0/2160365357 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6529 (secure 0 0 0) 0x7fc20005a310 con 0x7fc2100c7d70 2024-09-19T07:24:00.140 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:00.140 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":48,"btime":"2024-09-19T07:23:58:201892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:24:00.140 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 48 2024-09-19T07:24:00.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.141+0000 7fc218082640 1 -- 172.21.15.83:0/2160365357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1f40678c0 msgr2=0x7fc1f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:00.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.141+0000 7fc218082640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1f40678c0 0x7fc1f4069d80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0ad000 tx=0x7fc20c0a6000 comp rx=0 tx=0).stop 2024-09-19T07:24:00.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.141+0000 7fc218082640 1 -- 172.21.15.83:0/2160365357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100c7d70 msgr2=0x7fc210156000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:00.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.141+0000 7fc218082640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100c7d70 0x7fc210156000 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc20000e9b0 tx=0x7fc20000ee70 comp rx=0 tx=0).stop 2024-09-19T07:24:00.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.141+0000 7fc218082640 1 -- 172.21.15.83:0/2160365357 shutdown_connections 2024-09-19T07:24:00.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.141+0000 7fc218082640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1f40678c0 0x7fc1f4069d80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:00.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.141+0000 7fc218082640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2101547b0 0x7fc2101482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:00.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.141+0000 7fc218082640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100c7d70 0x7fc210156000 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:00.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.141+0000 7fc218082640 1 --2- 172.21.15.83:0/2160365357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc210072d00 0x7fc210153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:00.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.141+0000 7fc218082640 1 -- 172.21.15.83:0/2160365357 >> 172.21.15.83:0/2160365357 conn(0x7fc2100c5060 msgr2=0x7fc21006f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:00.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.141+0000 7fc218082640 1 -- 172.21.15.83:0/2160365357 shutdown_connections 2024-09-19T07:24:00.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:00.141+0000 7fc218082640 1 -- 172.21.15.83:0/2160365357 wait complete. 2024-09-19T07:24:00.164 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-09-19T07:24:00.165 INFO:tasks.mds_thrash.fs.[a]:waiting for 70 secs before thrashing 2024-09-19T07:24:01.090 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:24:01.092 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:24:28.550 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:28.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.693+0000 7ff8459b7640 1 Processor -- start 2024-09-19T07:24:28.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.693+0000 7ff8459b7640 1 -- start start 2024-09-19T07:24:28.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.693+0000 7ff8459b7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff8400c6460 0x7ff8400c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:28.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.693+0000 7ff8459b7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8400c26c0 con 0x7ff84006ffe0 2024-09-19T07:24:28.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.693+0000 7ff8459b7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8400c29a0 con 0x7ff8401535d0 2024-09-19T07:24:28.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.693+0000 7ff8459b7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff8400707e0 con 0x7ff8400c6460 2024-09-19T07:24:28.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.693+0000 7ff83f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff8400c6460 0x7ff8400c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:28.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.693+0000 7ff83e7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff84006ffe0 0x7ff840159af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45914/0 (socket says 172.21.15.83:45914) 2024-09-19T07:24:28.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.693+0000 7ff83e7fc640 1 -- 172.21.15.83:0/2054205889 learned_addr learned my addr 172.21.15.83:0/2054205889 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:24:28.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.693+0000 7ff83f7fe640 1 -- 172.21.15.83:0/2054205889 >> v1:172.21.15.184:6789/0 conn(0x7ff8401535d0 legacy=0x7ff840155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:28.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.693+0000 7ff83f7fe640 1 -- 172.21.15.83:0/2054205889 >> v1:172.21.15.83:6789/0 conn(0x7ff84006ffe0 legacy=0x7ff840159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:28.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.693+0000 7ff83f7fe640 1 -- 172.21.15.83:0/2054205889 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8401592f0 con 0x7ff8400c6460 2024-09-19T07:24:28.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.697+0000 7ff83f7fe640 1 --2- 172.21.15.83:0/2054205889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8400c6460 0x7ff8400c2f60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff838009d90 tx=0x7ff83801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7c85eeb2891a92 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:28.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.697+0000 7ff83dffb640 1 -- 172.21.15.83:0/2054205889 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8380b2070 con 0x7ff8400c6460 2024-09-19T07:24:28.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.697+0000 7ff83dffb640 1 -- 172.21.15.83:0/2054205889 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8380a57a0 con 0x7ff8400c6460 2024-09-19T07:24:28.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.697+0000 7ff83dffb640 1 -- 172.21.15.83:0/2054205889 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8380a5b70 con 0x7ff8400c6460 2024-09-19T07:24:28.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.697+0000 7ff8459b7640 1 -- 172.21.15.83:0/2054205889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8400c6460 msgr2=0x7ff8400c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:28.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.697+0000 7ff8459b7640 1 --2- 172.21.15.83:0/2054205889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8400c6460 0x7ff8400c2f60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff838009d90 tx=0x7ff83801c8e0 comp rx=0 tx=0).stop 2024-09-19T07:24:28.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.697+0000 7ff8459b7640 1 -- 172.21.15.83:0/2054205889 shutdown_connections 2024-09-19T07:24:28.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.697+0000 7ff8459b7640 1 --2- 172.21.15.83:0/2054205889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8400c6460 0x7ff8400c2f60 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:28.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.697+0000 7ff8459b7640 1 -- 172.21.15.83:0/2054205889 >> 172.21.15.83:0/2054205889 conn(0x7ff8400c5060 msgr2=0x7ff8400c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:28.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.697+0000 7ff8459b7640 1 -- 172.21.15.83:0/2054205889 shutdown_connections 2024-09-19T07:24:28.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.697+0000 7ff8459b7640 1 -- 172.21.15.83:0/2054205889 wait complete. 2024-09-19T07:24:28.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff8459b7640 1 Processor -- start 2024-09-19T07:24:28.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff8459b7640 1 -- start start 2024-09-19T07:24:28.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff8459b7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff84006ffe0 0x7ff84015a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:28.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff8459b7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8400c6460 0x7ff84015b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:28.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff8459b7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8401535d0 0x7ff84015b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:28.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff8459b7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff8400c2c80 con 0x7ff8401535d0 2024-09-19T07:24:28.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff8459b7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff8400c6890 con 0x7ff8400c6460 2024-09-19T07:24:28.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff8459b7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff840163f30 con 0x7ff84006ffe0 2024-09-19T07:24:28.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff83f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8401535d0 0x7ff84015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:28.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff83f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8401535d0 0x7ff84015b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54776/0 (socket says 172.21.15.83:54776) 2024-09-19T07:24:28.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff83f7fe640 1 -- 172.21.15.83:0/2842285689 learned_addr learned my addr 172.21.15.83:0/2842285689 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:28.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff83e7fc640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8400c6460 0x7ff84015b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff83effd640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff84006ffe0 0x7ff84015a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff83f7fe640 1 -- 172.21.15.83:0/2842285689 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff84006ffe0 msgr2=0x7ff84015a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff83f7fe640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff84006ffe0 0x7ff84015a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff83f7fe640 1 -- 172.21.15.83:0/2842285689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8400c6460 msgr2=0x7ff84015b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff83f7fe640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8400c6460 0x7ff84015b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff83f7fe640 1 -- 172.21.15.83:0/2842285689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8401529d0 con 0x7ff8401535d0 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff83e7fc640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8400c6460 0x7ff84015b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff83f7fe640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8401535d0 0x7ff84015b740 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff8380b0040 tx=0x7ff838019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff827fff640 1 -- 172.21.15.83:0/2842285689 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8380b2050 con 0x7ff8401535d0 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff827fff640 1 -- 172.21.15.83:0/2842285689 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8380be070 con 0x7ff8401535d0 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff827fff640 1 -- 172.21.15.83:0/2842285689 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff838009450 con 0x7ff8401535d0 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff8459b7640 1 -- 172.21.15.83:0/2842285689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff840153d10 con 0x7ff8401535d0 2024-09-19T07:24:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff8459b7640 1 -- 172.21.15.83:0/2842285689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8401545b0 con 0x7ff8401535d0 2024-09-19T07:24:28.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.701+0000 7ff8459b7640 1 -- 172.21.15.83:0/2842285689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff840154100 con 0x7ff8401535d0 2024-09-19T07:24:28.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.705+0000 7ff827fff640 1 -- 172.21.15.83:0/2842285689 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff8380adcc0 con 0x7ff8401535d0 2024-09-19T07:24:28.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.705+0000 7ff827fff640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff82c0678f0 0x7ff82c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:28.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.705+0000 7ff827fff640 1 -- 172.21.15.83:0/2842285689 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7ff838121540 con 0x7ff8401535d0 2024-09-19T07:24:28.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.705+0000 7ff83effd640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff82c0678f0 0x7ff82c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:28.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.705+0000 7ff83effd640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff82c0678f0 0x7ff82c069db0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff830001770 tx=0x7ff830009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.709+0000 7ff827fff640 1 -- 172.21.15.83:0/2842285689 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff838016560 con 0x7ff8401535d0 2024-09-19T07:24:28.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.949+0000 7ff8459b7640 1 -- 172.21.15.83:0/2842285689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff840153ee0 con 0x7ff8401535d0 2024-09-19T07:24:28.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.949+0000 7ff827fff640 1 -- 172.21.15.83:0/2842285689 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6529 (secure 0 0 0) 0x7ff8380f1b10 con 0x7ff8401535d0 2024-09-19T07:24:28.951 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:28.951 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":48,"btime":"2024-09-19T07:23:58:201892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:24:28.952 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 48 2024-09-19T07:24:28.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.953+0000 7ff8459b7640 1 -- 172.21.15.83:0/2842285689 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff82c0678f0 msgr2=0x7ff82c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:28.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.953+0000 7ff8459b7640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff82c0678f0 0x7ff82c069db0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff830001770 tx=0x7ff830009040 comp rx=0 tx=0).stop 2024-09-19T07:24:28.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.953+0000 7ff8459b7640 1 -- 172.21.15.83:0/2842285689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8401535d0 msgr2=0x7ff84015b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:28.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.953+0000 7ff8459b7640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8401535d0 0x7ff84015b740 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff8380b0040 tx=0x7ff838019680 comp rx=0 tx=0).stop 2024-09-19T07:24:28.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.953+0000 7ff8459b7640 1 -- 172.21.15.83:0/2842285689 shutdown_connections 2024-09-19T07:24:28.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.953+0000 7ff8459b7640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff82c0678f0 0x7ff82c069db0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:28.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.953+0000 7ff8459b7640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8401535d0 0x7ff84015b740 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:28.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.953+0000 7ff8459b7640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8400c6460 0x7ff84015b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:28.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.953+0000 7ff8459b7640 1 --2- 172.21.15.83:0/2842285689 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff84006ffe0 0x7ff84015a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:28.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.953+0000 7ff8459b7640 1 -- 172.21.15.83:0/2842285689 >> 172.21.15.83:0/2842285689 conn(0x7ff8400c5060 msgr2=0x7ff8400c7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:28.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.953+0000 7ff8459b7640 1 -- 172.21.15.83:0/2842285689 shutdown_connections 2024-09-19T07:24:28.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:28.953+0000 7ff8459b7640 1 -- 172.21.15.83:0/2842285689 wait complete. 2024-09-19T07:24:28.972 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-19T07:24:28.972 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:24:28.972 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:24:28.984 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:24:28.984 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:29.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.153+0000 7effc09cb640 1 Processor -- start 2024-09-19T07:24:29.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.153+0000 7effc09cb640 1 -- start start 2024-09-19T07:24:29.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.153+0000 7effc09cb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7effbc0ac290 0x7effbc0c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:29.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.153+0000 7effc09cb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7effbc0c3a50 0x7effbc1563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:29.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.153+0000 7effc09cb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effbc06e110 con 0x7effbc0c3130 2024-09-19T07:24:29.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.153+0000 7effc09cb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7effbc0ac700 con 0x7effbc0c3a50 2024-09-19T07:24:29.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.153+0000 7effc09cb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7effbc0c4990 con 0x7effbc0ac290 2024-09-19T07:24:29.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.157+0000 7effbad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7effbc0c3a50 0x7effbc1563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:29.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.157+0000 7effbad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7effbc0c3a50 0x7effbc1563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54790/0 (socket says 172.21.15.83:54790) 2024-09-19T07:24:29.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.157+0000 7effbad9d640 1 -- 172.21.15.83:0/3464334745 learned_addr learned my addr 172.21.15.83:0/3464334745 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:29.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.157+0000 7effb9d9b640 1 --2- 172.21.15.83:0/3464334745 >> v2:172.21.15.184:3300/0 conn(0x7effbc0ac290 0x7effbc0c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:29.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.157+0000 7effbad9d640 1 -- 172.21.15.83:0/3464334745 >> v2:172.21.15.184:3300/0 conn(0x7effbc0ac290 msgr2=0x7effbc0c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:29.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.157+0000 7effbad9d640 1 --2- 172.21.15.83:0/3464334745 >> v2:172.21.15.184:3300/0 conn(0x7effbc0ac290 0x7effbc0c3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:29.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.157+0000 7effbad9d640 1 -- 172.21.15.83:0/3464334745 >> v1:172.21.15.184:6789/0 conn(0x7effbc0c3130 legacy=0x7effbc06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:29.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.157+0000 7effbad9d640 1 -- 172.21.15.83:0/3464334745 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effbc0c4c70 con 0x7effbc0c3a50 2024-09-19T07:24:29.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.157+0000 7effbad9d640 1 --2- 172.21.15.83:0/3464334745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effbc0c3a50 0x7effbc1563a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7effbc0a80b0 tx=0x7effb401c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c720aa0f49ddfd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:29.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.157+0000 7effb959a640 1 -- 172.21.15.83:0/3464334745 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb40ac070 con 0x7effbc0c3a50 2024-09-19T07:24:29.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.157+0000 7effb959a640 1 -- 172.21.15.83:0/3464334745 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effb40a6460 con 0x7effbc0c3a50 2024-09-19T07:24:29.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.157+0000 7effb959a640 1 -- 172.21.15.83:0/3464334745 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb40a6830 con 0x7effbc0c3a50 2024-09-19T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 -- 172.21.15.83:0/3464334745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effbc0c3a50 msgr2=0x7effbc1563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 --2- 172.21.15.83:0/3464334745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effbc0c3a50 0x7effbc1563a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7effbc0a80b0 tx=0x7effb401c6a0 comp rx=0 tx=0).stop 2024-09-19T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 -- 172.21.15.83:0/3464334745 shutdown_connections 2024-09-19T07:24:29.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 --2- 172.21.15.83:0/3464334745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effbc0c3a50 0x7effbc1563a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:29.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 --2- 172.21.15.83:0/3464334745 >> v2:172.21.15.184:3300/0 conn(0x7effbc0ac290 0x7effbc0c3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:29.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 -- 172.21.15.83:0/3464334745 >> 172.21.15.83:0/3464334745 conn(0x7effbc0c5060 msgr2=0x7effbc0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:29.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 -- 172.21.15.83:0/3464334745 shutdown_connections 2024-09-19T07:24:29.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 -- 172.21.15.83:0/3464334745 wait complete. 2024-09-19T07:24:29.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 Processor -- start 2024-09-19T07:24:29.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 -- start start 2024-09-19T07:24:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effbc0ac290 0x7effbc1506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effbc0c3130 0x7effbc072050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effbc0c3a50 0x7effbc072590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7effbc0ad520 con 0x7effbc0c3130 2024-09-19T07:24:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7effbc0c82a0 con 0x7effbc0c3a50 2024-09-19T07:24:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7effbc0ad6a0 con 0x7effbc0ac290 2024-09-19T07:24:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effb9d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effbc0c3130 0x7effbc072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effb9d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effbc0c3130 0x7effbc072050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54792/0 (socket says 172.21.15.83:54792) 2024-09-19T07:24:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effb9d9b640 1 -- 172.21.15.83:0/1230794093 learned_addr learned my addr 172.21.15.83:0/1230794093 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effbad9d640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effbc0c3a50 0x7effbc072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effba59c640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effbc0ac290 0x7effbc1506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effb9d9b640 1 -- 172.21.15.83:0/1230794093 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effbc0ac290 msgr2=0x7effbc1506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:29.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effb9d9b640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effbc0ac290 0x7effbc1506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:29.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effb9d9b640 1 -- 172.21.15.83:0/1230794093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effbc0c3a50 msgr2=0x7effbc072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:29.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effb9d9b640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effbc0c3a50 0x7effbc072590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:29.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effb9d9b640 1 -- 172.21.15.83:0/1230794093 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effbc0c54b0 con 0x7effbc0c3130 2024-09-19T07:24:29.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effbad9d640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effbc0c3a50 0x7effbc072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:24:29.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effb9d9b640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effbc0c3130 0x7effbc072050 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7effb000d800 tx=0x7effb000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:29.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effaaffd640 1 -- 172.21.15.83:0/1230794093 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb000b6c0 con 0x7effbc0c3130 2024-09-19T07:24:29.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effaaffd640 1 -- 172.21.15.83:0/1230794093 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effb0019070 con 0x7effbc0c3130 2024-09-19T07:24:29.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 -- 172.21.15.83:0/1230794093 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effbc073a60 con 0x7effbc0c3130 2024-09-19T07:24:29.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effaaffd640 1 -- 172.21.15.83:0/1230794093 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb000ba60 con 0x7effbc0c3130 2024-09-19T07:24:29.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.161+0000 7effc09cb640 1 -- 172.21.15.83:0/1230794093 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effbc14fe10 con 0x7effbc0c3130 2024-09-19T07:24:29.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.165+0000 7effc09cb640 1 -- 172.21.15.83:0/1230794093 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effbc0c3510 con 0x7effbc0c3130 2024-09-19T07:24:29.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.165+0000 7effaaffd640 1 -- 172.21.15.83:0/1230794093 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7effb000bc00 con 0x7effbc0c3130 2024-09-19T07:24:29.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.165+0000 7effaaffd640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff9c0678f0 0x7eff9c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:29.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.165+0000 7effaaffd640 1 -- 172.21.15.83:0/1230794093 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7effb00866e0 con 0x7effbc0c3130 2024-09-19T07:24:29.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.165+0000 7effba59c640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff9c0678f0 0x7eff9c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.165+0000 7effba59c640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff9c0678f0 0x7eff9c069db0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7effac001650 tx=0x7effac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:29.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.169+0000 7effaaffd640 1 -- 172.21.15.83:0/1230794093 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7effb0056c20 con 0x7effbc0c3130 2024-09-19T07:24:29.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.437+0000 7effc09cb640 1 -- 172.21.15.83:0/1230794093 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effbc15d5e0 con 0x7effbc0c3130 2024-09-19T07:24:29.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effaaffd640 1 -- 172.21.15.83:0/1230794093 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6529 (secure 0 0 0) 0x7effb0056520 con 0x7effbc0c3130 2024-09-19T07:24:29.441 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:29.441 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":48,"btime":"2024-09-19T07:23:58:201892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:24:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 48 2024-09-19T07:24:29.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effc09cb640 1 -- 172.21.15.83:0/1230794093 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff9c0678f0 msgr2=0x7eff9c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:29.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effc09cb640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff9c0678f0 0x7eff9c069db0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7effac001650 tx=0x7effac009040 comp rx=0 tx=0).stop 2024-09-19T07:24:29.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effc09cb640 1 -- 172.21.15.83:0/1230794093 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effbc0c3130 msgr2=0x7effbc072050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:29.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effc09cb640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effbc0c3130 0x7effbc072050 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7effb000d800 tx=0x7effb000dcc0 comp rx=0 tx=0).stop 2024-09-19T07:24:29.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effc09cb640 1 -- 172.21.15.83:0/1230794093 shutdown_connections 2024-09-19T07:24:29.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effc09cb640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff9c0678f0 0x7eff9c069db0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:29.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effc09cb640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effbc0c3a50 0x7effbc072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:29.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effc09cb640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effbc0c3130 0x7effbc072050 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:29.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effc09cb640 1 --2- 172.21.15.83:0/1230794093 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effbc0ac290 0x7effbc1506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:29.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effc09cb640 1 -- 172.21.15.83:0/1230794093 >> 172.21.15.83:0/1230794093 conn(0x7effbc0c5060 msgr2=0x7effbc15bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:29.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effc09cb640 1 -- 172.21.15.83:0/1230794093 shutdown_connections 2024-09-19T07:24:29.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:29.441+0000 7effc09cb640 1 -- 172.21.15.83:0/1230794093 wait complete. 2024-09-19T07:24:29.461 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:24:31.136 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:24:31.139 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:24:31.461 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:31.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.605+0000 7f1924910640 1 Processor -- start 2024-09-19T07:24:31.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.605+0000 7f1924910640 1 -- start start 2024-09-19T07:24:31.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f1924910640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f19200b0d00 0x7f1920159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:31.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f1924910640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f192006c600 0x7f19200c8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:31.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f1924910640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19200c6180 con 0x7f19200c9660 2024-09-19T07:24:31.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f1924910640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f19200c69b0 con 0x7f192006c600 2024-09-19T07:24:31.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f1924910640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f192015a130 con 0x7f19200b0d00 2024-09-19T07:24:31.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191ed9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f192006c600 0x7f19200c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:31.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191ed9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f192006c600 0x7f19200c8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58044/0 (socket says 172.21.15.83:58044) 2024-09-19T07:24:31.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191ed9d640 1 -- 172.21.15.83:0/3038138143 learned_addr learned my addr 172.21.15.83:0/3038138143 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:31.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191e59c640 1 --2- 172.21.15.83:0/3038138143 >> v2:172.21.15.184:3300/0 conn(0x7f19200b0d00 0x7f1920159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:31.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191ed9d640 1 -- 172.21.15.83:0/3038138143 >> v2:172.21.15.184:3300/0 conn(0x7f19200b0d00 msgr2=0x7f1920159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:31.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191ed9d640 1 --2- 172.21.15.83:0/3038138143 >> v2:172.21.15.184:3300/0 conn(0x7f19200b0d00 0x7f1920159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:31.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191ed9d640 1 -- 172.21.15.83:0/3038138143 >> v1:172.21.15.83:6789/0 conn(0x7f19200c9660 legacy=0x7f19201563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:31.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191ed9d640 1 -- 172.21.15.83:0/3038138143 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1920155bf0 con 0x7f192006c600 2024-09-19T07:24:31.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191e59c640 1 --2- 172.21.15.83:0/3038138143 >> v2:172.21.15.184:3300/0 conn(0x7f19200b0d00 0x7f1920159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:24:31.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191ed9d640 1 --2- 172.21.15.83:0/3038138143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f192006c600 0x7f19200c8210 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f19200c80c0 tx=0x7f191801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9568b68bc9c12d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:31.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191d59a640 1 -- 172.21.15.83:0/3038138143 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19180a5620 con 0x7f192006c600 2024-09-19T07:24:31.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191d59a640 1 -- 172.21.15.83:0/3038138143 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19180a57c0 con 0x7f192006c600 2024-09-19T07:24:31.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.609+0000 7f191d59a640 1 -- 172.21.15.83:0/3038138143 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19180a5b90 con 0x7f192006c600 2024-09-19T07:24:31.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 -- 172.21.15.83:0/3038138143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f192006c600 msgr2=0x7f19200c8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:31.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 --2- 172.21.15.83:0/3038138143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f192006c600 0x7f19200c8210 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f19200c80c0 tx=0x7f191801c960 comp rx=0 tx=0).stop 2024-09-19T07:24:31.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 -- 172.21.15.83:0/3038138143 shutdown_connections 2024-09-19T07:24:31.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 --2- 172.21.15.83:0/3038138143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f192006c600 0x7f19200c8210 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:31.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 --2- 172.21.15.83:0/3038138143 >> v2:172.21.15.184:3300/0 conn(0x7f19200b0d00 0x7f1920159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:31.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 -- 172.21.15.83:0/3038138143 >> 172.21.15.83:0/3038138143 conn(0x7f19200722a0 msgr2=0x7f19200726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:31.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 -- 172.21.15.83:0/3038138143 shutdown_connections 2024-09-19T07:24:31.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 -- 172.21.15.83:0/3038138143 wait complete. 2024-09-19T07:24:31.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 Processor -- start 2024-09-19T07:24:31.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 -- start start 2024-09-19T07:24:31.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f192006c600 0x7f19201f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:31.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f19200b0d00 0x7f19201f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:31.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19200c9660 0x7f19201f0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:31.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f19200b10e0 con 0x7f19200c9660 2024-09-19T07:24:31.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1920155eb0 con 0x7f192006c600 2024-09-19T07:24:31.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1920162d90 con 0x7f19200b0d00 2024-09-19T07:24:31.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f191ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19200c9660 0x7f19201f0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:31.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f191ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19200c9660 0x7f19201f0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58060/0 (socket says 172.21.15.83:58060) 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f191ed9d640 1 -- 172.21.15.83:0/220925368 learned_addr learned my addr 172.21.15.83:0/220925368 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f191e59c640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f192006c600 0x7f19201f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f191dd9b640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f19200b0d00 0x7f19201f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f191ed9d640 1 -- 172.21.15.83:0/220925368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f19200b0d00 msgr2=0x7f19201f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f191ed9d640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f19200b0d00 0x7f19201f2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f191ed9d640 1 -- 172.21.15.83:0/220925368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f192006c600 msgr2=0x7f19201f2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f191ed9d640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f192006c600 0x7f19201f2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f191ed9d640 1 -- 172.21.15.83:0/220925368 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19200aef30 con 0x7f19200c9660 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f191ed9d640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19200c9660 0x7f19201f0600 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f19180af040 tx=0x7f1918019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f190effd640 1 -- 172.21.15.83:0/220925368 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19180bc020 con 0x7f19200c9660 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f190effd640 1 -- 172.21.15.83:0/220925368 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19180b8880 con 0x7f19200c9660 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 -- 172.21.15.83:0/220925368 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1920149a50 con 0x7f19200c9660 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f190effd640 1 -- 172.21.15.83:0/220925368 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1918009510 con 0x7f19200c9660 2024-09-19T07:24:31.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.613+0000 7f1924910640 1 -- 172.21.15.83:0/220925368 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19201f2d90 con 0x7f19200c9660 2024-09-19T07:24:31.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.617+0000 7f1924910640 1 -- 172.21.15.83:0/220925368 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19200c8210 con 0x7f19200c9660 2024-09-19T07:24:31.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.617+0000 7f190effd640 1 -- 172.21.15.83:0/220925368 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f19180bf020 con 0x7f19200c9660 2024-09-19T07:24:31.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.617+0000 7f190effd640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f19000678f0 0x7f1900069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:31.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.617+0000 7f190effd640 1 -- 172.21.15.83:0/220925368 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7f19181214a0 con 0x7f19200c9660 2024-09-19T07:24:31.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.617+0000 7f191e59c640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f19000678f0 0x7f1900069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:31.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.617+0000 7f191e59c640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f19000678f0 0x7f1900069db0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f191000a9a0 tx=0x7f1910008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:31.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.625+0000 7f190effd640 1 -- 172.21.15.83:0/220925368 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f19180c6030 con 0x7f19200c9660 2024-09-19T07:24:31.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.913+0000 7f1924910640 1 -- 172.21.15.83:0/220925368 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1920072a50 con 0x7f19200c9660 2024-09-19T07:24:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.913+0000 7f190effd640 1 -- 172.21.15.83:0/220925368 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6529 (secure 0 0 0) 0x7f19180165e0 con 0x7f19200c9660 2024-09-19T07:24:31.915 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:31.915 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":48,"btime":"2024-09-19T07:23:58:201892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:24:31.916 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 48 2024-09-19T07:24:31.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.917+0000 7f1924910640 1 -- 172.21.15.83:0/220925368 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f19000678f0 msgr2=0x7f1900069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:31.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.917+0000 7f1924910640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f19000678f0 0x7f1900069db0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f191000a9a0 tx=0x7f1910008040 comp rx=0 tx=0).stop 2024-09-19T07:24:31.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.917+0000 7f1924910640 1 -- 172.21.15.83:0/220925368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19200c9660 msgr2=0x7f19201f0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:31.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.917+0000 7f1924910640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19200c9660 0x7f19201f0600 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f19180af040 tx=0x7f1918019700 comp rx=0 tx=0).stop 2024-09-19T07:24:31.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.917+0000 7f1924910640 1 -- 172.21.15.83:0/220925368 shutdown_connections 2024-09-19T07:24:31.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.917+0000 7f1924910640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f19000678f0 0x7f1900069db0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:31.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.917+0000 7f1924910640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19200c9660 0x7f19201f0600 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:31.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.917+0000 7f1924910640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f19200b0d00 0x7f19201f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:31.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.917+0000 7f1924910640 1 --2- 172.21.15.83:0/220925368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f192006c600 0x7f19201f2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:31.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.917+0000 7f1924910640 1 -- 172.21.15.83:0/220925368 >> 172.21.15.83:0/220925368 conn(0x7f19200722a0 msgr2=0x7f19200ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:31.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.917+0000 7f1924910640 1 -- 172.21.15.83:0/220925368 shutdown_connections 2024-09-19T07:24:31.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:31.917+0000 7f1924910640 1 -- 172.21.15.83:0/220925368 wait complete. 2024-09-19T07:24:31.935 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:24:33.936 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:34.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.065+0000 7fc8ccd84640 1 Processor -- start 2024-09-19T07:24:34.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8ccd84640 1 -- start start 2024-09-19T07:24:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8ccd84640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc8c8159280 0x7fc8c80c0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8ccd84640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8c806d8f0 0x7fc8c80ca730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8ccd84640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8c80c0ac0 con 0x7fc8c814f640 2024-09-19T07:24:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8ccd84640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc8c80ca2a0 con 0x7fc8c8159280 2024-09-19T07:24:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8ccd84640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc8c806ceb0 con 0x7fc8c806d8f0 2024-09-19T07:24:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8c77fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc8c8159280 0x7fc8c80c0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8c77fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc8c8159280 0x7fc8c80c0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58068/0 (socket says 172.21.15.83:58068) 2024-09-19T07:24:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8c77fe640 1 -- 172.21.15.83:0/2937458581 learned_addr learned my addr 172.21.15.83:0/2937458581 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8c6ffd640 1 --2- 172.21.15.83:0/2937458581 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8c806d8f0 0x7fc8c80ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8c77fe640 1 -- 172.21.15.83:0/2937458581 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8c806d8f0 msgr2=0x7fc8c80ca730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8c77fe640 1 --2- 172.21.15.83:0/2937458581 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8c806d8f0 0x7fc8c80ca730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8c77fe640 1 -- 172.21.15.83:0/2937458581 >> v1:172.21.15.184:6789/0 conn(0x7fc8c814f640 legacy=0x7fc8c8159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:34.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8c77fe640 1 -- 172.21.15.83:0/2937458581 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c80cac70 con 0x7fc8c8159280 2024-09-19T07:24:34.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8c77fe640 1 --2- 172.21.15.83:0/2937458581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8c8159280 0x7fc8c80c0580 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009870 tx=0x7fc8b801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd48718f491117ec server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:34.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8c67fc640 1 -- 172.21.15.83:0/2937458581 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80b0070 con 0x7fc8c8159280 2024-09-19T07:24:34.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8c67fc640 1 -- 172.21.15.83:0/2937458581 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8b80ab600 con 0x7fc8c8159280 2024-09-19T07:24:34.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.069+0000 7fc8c67fc640 1 -- 172.21.15.83:0/2937458581 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80ab9d0 con 0x7fc8c8159280 2024-09-19T07:24:34.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2937458581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8c8159280 msgr2=0x7fc8c80c0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:34.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 --2- 172.21.15.83:0/2937458581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8c8159280 0x7fc8c80c0580 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009870 tx=0x7fc8b801c460 comp rx=0 tx=0).stop 2024-09-19T07:24:34.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2937458581 shutdown_connections 2024-09-19T07:24:34.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 --2- 172.21.15.83:0/2937458581 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8c806d8f0 0x7fc8c80ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:34.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 --2- 172.21.15.83:0/2937458581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8c8159280 0x7fc8c80c0580 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:34.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2937458581 >> 172.21.15.83:0/2937458581 conn(0x7fc8c80c8060 msgr2=0x7fc8c80c8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2937458581 shutdown_connections 2024-09-19T07:24:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2937458581 wait complete. 2024-09-19T07:24:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 Processor -- start 2024-09-19T07:24:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 -- start start 2024-09-19T07:24:34.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8c806d8f0 0x7fc8c80c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:34.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8c814f640 0x7fc8c80c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:34.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8c8159280 0x7fc8c81413c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:34.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c8159ef0 con 0x7fc8c814f640 2024-09-19T07:24:34.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c8159d70 con 0x7fc8c8159280 2024-09-19T07:24:34.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc8c8162bd0 con 0x7fc8c806d8f0 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8c6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8c814f640 0x7fc8c80c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8c6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8c814f640 0x7fc8c80c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58070/0 (socket says 172.21.15.83:58070) 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8c6ffd640 1 -- 172.21.15.83:0/2570517054 learned_addr learned my addr 172.21.15.83:0/2570517054 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8c77fe640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8c806d8f0 0x7fc8c80c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8c7fff640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8c8159280 0x7fc8c81413c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8c6ffd640 1 -- 172.21.15.83:0/2570517054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8c806d8f0 msgr2=0x7fc8c80c7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8c6ffd640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8c806d8f0 0x7fc8c80c7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8c6ffd640 1 -- 172.21.15.83:0/2570517054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8c8159280 msgr2=0x7fc8c81413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8c6ffd640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8c8159280 0x7fc8c81413c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8c6ffd640 1 -- 172.21.15.83:0/2570517054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c8158d30 con 0x7fc8c814f640 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8c6ffd640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8c814f640 0x7fc8c80c5eb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc00d7d0 tx=0x7fc8bc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8affff640 1 -- 172.21.15.83:0/2570517054 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc004030 con 0x7fc8c814f640 2024-09-19T07:24:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8affff640 1 -- 172.21.15.83:0/2570517054 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8bc004210 con 0x7fc8c814f640 2024-09-19T07:24:34.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2570517054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8c81ee5f0 con 0x7fc8c814f640 2024-09-19T07:24:34.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8affff640 1 -- 172.21.15.83:0/2570517054 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc00b6c0 con 0x7fc8c814f640 2024-09-19T07:24:34.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.073+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2570517054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8c81eebb0 con 0x7fc8c814f640 2024-09-19T07:24:34.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.077+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2570517054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8c81eedb0 con 0x7fc8c814f640 2024-09-19T07:24:34.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.077+0000 7fc8affff640 1 -- 172.21.15.83:0/2570517054 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc8bc00b860 con 0x7fc8c814f640 2024-09-19T07:24:34.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.077+0000 7fc8affff640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8b40678f0 0x7fc8b4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:34.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.077+0000 7fc8affff640 1 -- 172.21.15.83:0/2570517054 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7fc8bc0866b0 con 0x7fc8c814f640 2024-09-19T07:24:34.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.077+0000 7fc8c77fe640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8b40678f0 0x7fc8b4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:34.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.077+0000 7fc8c77fe640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8b40678f0 0x7fc8b4069db0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80ad000 tx=0x7fc8b80a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:34.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.081+0000 7fc8affff640 1 -- 172.21.15.83:0/2570517054 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc8c81eedb0 con 0x7fc8c814f640 2024-09-19T07:24:34.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.345+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2570517054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8c80c3160 con 0x7fc8c814f640 2024-09-19T07:24:34.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.345+0000 7fc8affff640 1 -- 172.21.15.83:0/2570517054 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6529 (secure 0 0 0) 0x7fc8bc01c360 con 0x7fc8c814f640 2024-09-19T07:24:34.348 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:34.348 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":48,"btime":"2024-09-19T07:23:58:201892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:24:34.348 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 48 2024-09-19T07:24:34.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.349+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2570517054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8b40678f0 msgr2=0x7fc8b4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:34.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.349+0000 7fc8ccd84640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8b40678f0 0x7fc8b4069db0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80ad000 tx=0x7fc8b80a6000 comp rx=0 tx=0).stop 2024-09-19T07:24:34.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.349+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2570517054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8c814f640 msgr2=0x7fc8c80c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:34.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.349+0000 7fc8ccd84640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8c814f640 0x7fc8c80c5eb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc00d7d0 tx=0x7fc8bc00dc90 comp rx=0 tx=0).stop 2024-09-19T07:24:34.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.349+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2570517054 shutdown_connections 2024-09-19T07:24:34.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.349+0000 7fc8ccd84640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8b40678f0 0x7fc8b4069db0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:34.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.349+0000 7fc8ccd84640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8c8159280 0x7fc8c81413c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:34.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.349+0000 7fc8ccd84640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8c814f640 0x7fc8c80c5eb0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:34.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.349+0000 7fc8ccd84640 1 --2- 172.21.15.83:0/2570517054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8c806d8f0 0x7fc8c80c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:34.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.349+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2570517054 >> 172.21.15.83:0/2570517054 conn(0x7fc8c80c8060 msgr2=0x7fc8c80c8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:34.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.349+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2570517054 shutdown_connections 2024-09-19T07:24:34.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:34.349+0000 7fc8ccd84640 1 -- 172.21.15.83:0/2570517054 wait complete. 2024-09-19T07:24:34.371 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:24:36.371 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e385d4640 1 Processor -- start 2024-09-19T07:24:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e385d4640 1 -- start start 2024-09-19T07:24:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e385d4640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1e30154540 0x7f1e30157490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e385d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1e301579d0 0x7f1e3015e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e385d4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e30151750 con 0x7f1e3014a500 2024-09-19T07:24:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e385d4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f1e300c7e90 con 0x7f1e301579d0 2024-09-19T07:24:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e385d4640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f1e30073110 con 0x7f1e30154540 2024-09-19T07:24:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e36b71640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1e301579d0 0x7f1e3015e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e36b71640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1e301579d0 0x7f1e3015e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58086/0 (socket says 172.21.15.83:58086) 2024-09-19T07:24:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e36b71640 1 -- 172.21.15.83:0/2887015119 learned_addr learned my addr 172.21.15.83:0/2887015119 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e36b71640 1 -- 172.21.15.83:0/2887015119 >> v2:172.21.15.184:3300/0 conn(0x7f1e30154540 msgr2=0x7f1e30157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e36b71640 1 --2- 172.21.15.83:0/2887015119 >> v2:172.21.15.184:3300/0 conn(0x7f1e30154540 0x7f1e30157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e36b71640 1 -- 172.21.15.83:0/2887015119 >> v1:172.21.15.83:6789/0 conn(0x7f1e3014a500 legacy=0x7f1e300c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.513+0000 7f1e36b71640 1 -- 172.21.15.83:0/2887015119 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e300733f0 con 0x7f1e301579d0 2024-09-19T07:24:36.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e36b71640 1 --2- 172.21.15.83:0/2887015119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e301579d0 0x7f1e3015e440 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1e300c94f0 tx=0x7f1e2801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf291b728b227a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e3536e640 1 -- 172.21.15.83:0/2887015119 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e280a5840 con 0x7f1e301579d0 2024-09-19T07:24:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e3536e640 1 -- 172.21.15.83:0/2887015119 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e280a59e0 con 0x7f1e301579d0 2024-09-19T07:24:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e3536e640 1 -- 172.21.15.83:0/2887015119 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e280a5db0 con 0x7f1e301579d0 2024-09-19T07:24:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e385d4640 1 -- 172.21.15.83:0/2887015119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e301579d0 msgr2=0x7f1e3015e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e385d4640 1 --2- 172.21.15.83:0/2887015119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e301579d0 0x7f1e3015e440 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1e300c94f0 tx=0x7f1e2801c9c0 comp rx=0 tx=0).stop 2024-09-19T07:24:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e385d4640 1 -- 172.21.15.83:0/2887015119 shutdown_connections 2024-09-19T07:24:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e385d4640 1 --2- 172.21.15.83:0/2887015119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e301579d0 0x7f1e3015e440 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e385d4640 1 --2- 172.21.15.83:0/2887015119 >> v2:172.21.15.184:3300/0 conn(0x7f1e30154540 0x7f1e30157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e385d4640 1 -- 172.21.15.83:0/2887015119 >> 172.21.15.83:0/2887015119 conn(0x7f1e300c9610 msgr2=0x7f1e300c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e385d4640 1 -- 172.21.15.83:0/2887015119 shutdown_connections 2024-09-19T07:24:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e385d4640 1 -- 172.21.15.83:0/2887015119 wait complete. 2024-09-19T07:24:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e385d4640 1 Processor -- start 2024-09-19T07:24:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.517+0000 7f1e385d4640 1 -- start start 2024-09-19T07:24:36.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e385d4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e3014a500 0x7f1e30156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e385d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e30154540 0x7f1e3013e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e385d4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e301579d0 0x7f1e3013ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e385d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1e300c7850 con 0x7f1e30154540 2024-09-19T07:24:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e385d4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1e300c76d0 con 0x7f1e3014a500 2024-09-19T07:24:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e385d4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1e300c4720 con 0x7f1e301579d0 2024-09-19T07:24:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e35b6f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e30154540 0x7f1e3013e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e35b6f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e30154540 0x7f1e3013e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58102/0 (socket says 172.21.15.83:58102) 2024-09-19T07:24:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e35b6f640 1 -- 172.21.15.83:0/3127499328 learned_addr learned my addr 172.21.15.83:0/3127499328 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:36.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e35b6f640 1 -- 172.21.15.83:0/3127499328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e301579d0 msgr2=0x7f1e3013ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:36.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e35b6f640 1 --2- 172.21.15.83:0/3127499328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e301579d0 0x7f1e3013ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:36.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e35b6f640 1 -- 172.21.15.83:0/3127499328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e3014a500 msgr2=0x7f1e30156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:36.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e35b6f640 1 --2- 172.21.15.83:0/3127499328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e3014a500 0x7f1e30156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:36.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e35b6f640 1 -- 172.21.15.83:0/3127499328 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e301ec3e0 con 0x7f1e30154540 2024-09-19T07:24:36.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e35b6f640 1 --2- 172.21.15.83:0/3127499328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e30154540 0x7f1e3013e950 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000b410 tx=0x7f1e2000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:36.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e26ffd640 1 -- 172.21.15.83:0/3127499328 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e20015020 con 0x7f1e30154540 2024-09-19T07:24:36.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e26ffd640 1 -- 172.21.15.83:0/3127499328 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e20011830 con 0x7f1e30154540 2024-09-19T07:24:36.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e26ffd640 1 -- 172.21.15.83:0/3127499328 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e20011c20 con 0x7f1e30154540 2024-09-19T07:24:36.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e385d4640 1 -- 172.21.15.83:0/3127499328 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e301ec6a0 con 0x7f1e30154540 2024-09-19T07:24:36.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e385d4640 1 -- 172.21.15.83:0/3127499328 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e301ecc60 con 0x7f1e30154540 2024-09-19T07:24:36.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.521+0000 7f1e385d4640 1 -- 172.21.15.83:0/3127499328 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e30157e20 con 0x7f1e30154540 2024-09-19T07:24:36.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.525+0000 7f1e26ffd640 1 -- 172.21.15.83:0/3127499328 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1e20018020 con 0x7f1e30154540 2024-09-19T07:24:36.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.525+0000 7f1e26ffd640 1 --2- 172.21.15.83:0/3127499328 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1e140678c0 0x7f1e14069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:36.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.525+0000 7f1e26ffd640 1 -- 172.21.15.83:0/3127499328 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7f1e20086740 con 0x7f1e30154540 2024-09-19T07:24:36.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.529+0000 7f1e36370640 1 --2- 172.21.15.83:0/3127499328 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1e140678c0 0x7f1e14069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:36.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.533+0000 7f1e26ffd640 1 -- 172.21.15.83:0/3127499328 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1e2001f020 con 0x7f1e30154540 2024-09-19T07:24:36.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.533+0000 7f1e36370640 1 --2- 172.21.15.83:0/3127499328 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1e140678c0 0x7f1e14069d80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c000f70 tx=0x7f1e2c0079e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.761+0000 7f1e385d4640 1 -- 172.21.15.83:0/3127499328 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e30073fd0 con 0x7f1e30154540 2024-09-19T07:24:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.761+0000 7f1e26ffd640 1 -- 172.21.15.83:0/3127499328 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6529 (secure 0 0 0) 0x7f1e2000f2e0 con 0x7f1e30154540 2024-09-19T07:24:36.764 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:36.764 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":48,"btime":"2024-09-19T07:23:58:201892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:24:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 48 2024-09-19T07:24:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.765+0000 7f1e385d4640 1 -- 172.21.15.83:0/3127499328 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1e140678c0 msgr2=0x7f1e14069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.765+0000 7f1e385d4640 1 --2- 172.21.15.83:0/3127499328 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1e140678c0 0x7f1e14069d80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c000f70 tx=0x7f1e2c0079e0 comp rx=0 tx=0).stop 2024-09-19T07:24:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.765+0000 7f1e385d4640 1 -- 172.21.15.83:0/3127499328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e30154540 msgr2=0x7f1e3013e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.765+0000 7f1e385d4640 1 --2- 172.21.15.83:0/3127499328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e30154540 0x7f1e3013e950 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000b410 tx=0x7f1e2000b8d0 comp rx=0 tx=0).stop 2024-09-19T07:24:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.765+0000 7f1e385d4640 1 -- 172.21.15.83:0/3127499328 shutdown_connections 2024-09-19T07:24:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.765+0000 7f1e385d4640 1 --2- 172.21.15.83:0/3127499328 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1e140678c0 0x7f1e14069d80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.765+0000 7f1e385d4640 1 --2- 172.21.15.83:0/3127499328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e301579d0 0x7f1e3013ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.765+0000 7f1e385d4640 1 --2- 172.21.15.83:0/3127499328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e30154540 0x7f1e3013e950 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.765+0000 7f1e385d4640 1 --2- 172.21.15.83:0/3127499328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e3014a500 0x7f1e30156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.765+0000 7f1e385d4640 1 -- 172.21.15.83:0/3127499328 >> 172.21.15.83:0/3127499328 conn(0x7f1e300c9610 msgr2=0x7f1e30152a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.765+0000 7f1e385d4640 1 -- 172.21.15.83:0/3127499328 shutdown_connections 2024-09-19T07:24:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:36.765+0000 7f1e385d4640 1 -- 172.21.15.83:0/3127499328 wait complete. 2024-09-19T07:24:36.785 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:24:38.786 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:38.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.921+0000 7f40bf353640 1 Processor -- start 2024-09-19T07:24:38.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.921+0000 7f40bf353640 1 -- start start 2024-09-19T07:24:38.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.921+0000 7f40bf353640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f40b8152f30 0x7f40b81533a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:38.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bf353640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b81538e0 con 0x7f40b8155e10 2024-09-19T07:24:38.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bf353640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b8159430 con 0x7f40b80c4880 2024-09-19T07:24:38.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bf353640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f40b815a040 con 0x7f40b8152f30 2024-09-19T07:24:38.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bc8ee640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f40b8152f30 0x7f40b81533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:38.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bc8ee640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f40b8152f30 0x7f40b81533a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58110/0 (socket says 172.21.15.83:58110) 2024-09-19T07:24:38.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bc8ee640 1 -- 172.21.15.83:0/3466896332 learned_addr learned my addr 172.21.15.83:0/3466896332 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:38.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bc8ee640 1 -- 172.21.15.83:0/3466896332 >> v1:172.21.15.184:6789/0 conn(0x7f40b80c4880 legacy=0x7f40b80c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:38.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bc8ee640 1 -- 172.21.15.83:0/3466896332 >> v1:172.21.15.83:6789/0 conn(0x7f40b8155e10 legacy=0x7f40b8156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:38.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bc8ee640 1 -- 172.21.15.83:0/3466896332 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b8159710 con 0x7f40b8152f30 2024-09-19T07:24:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bc8ee640 1 --2- 172.21.15.83:0/3466896332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40b8152f30 0x7f40b81533a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f40a8009870 tx=0x7f40a801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb51bf1ada6fd252 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40affff640 1 -- 172.21.15.83:0/3466896332 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80b0070 con 0x7f40b8152f30 2024-09-19T07:24:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40affff640 1 -- 172.21.15.83:0/3466896332 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a80ab600 con 0x7f40b8152f30 2024-09-19T07:24:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40affff640 1 -- 172.21.15.83:0/3466896332 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80ab9d0 con 0x7f40b8152f30 2024-09-19T07:24:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bf353640 1 -- 172.21.15.83:0/3466896332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40b8152f30 msgr2=0x7f40b81533a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bf353640 1 --2- 172.21.15.83:0/3466896332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40b8152f30 0x7f40b81533a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f40a8009870 tx=0x7f40a801c400 comp rx=0 tx=0).stop 2024-09-19T07:24:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bf353640 1 -- 172.21.15.83:0/3466896332 shutdown_connections 2024-09-19T07:24:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bf353640 1 --2- 172.21.15.83:0/3466896332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40b8152f30 0x7f40b81533a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bf353640 1 -- 172.21.15.83:0/3466896332 >> 172.21.15.83:0/3466896332 conn(0x7f40b80aa910 msgr2=0x7f40b80c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:38.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.925+0000 7f40bf353640 1 -- 172.21.15.83:0/3466896332 shutdown_connections 2024-09-19T07:24:38.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bf353640 1 -- 172.21.15.83:0/3466896332 wait complete. 2024-09-19T07:24:38.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bf353640 1 Processor -- start 2024-09-19T07:24:38.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bf353640 1 -- start start 2024-09-19T07:24:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bf353640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40b80c4880 0x7f40b80ad070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bf353640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40b8152f30 0x7f40b8158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bf353640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40b8155e10 0x7f40b80ab710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bf353640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f40b80c2680 con 0x7f40b80c4880 2024-09-19T07:24:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bf353640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f40b80c2500 con 0x7f40b8152f30 2024-09-19T07:24:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bf353640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f40b80c4c60 con 0x7f40b8155e10 2024-09-19T07:24:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bd0ef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40b80c4880 0x7f40b80ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bd0ef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40b80c4880 0x7f40b80ad070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58112/0 (socket says 172.21.15.83:58112) 2024-09-19T07:24:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bd0ef640 1 -- 172.21.15.83:0/3585754047 learned_addr learned my addr 172.21.15.83:0/3585754047 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bc8ee640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40b8152f30 0x7f40b8158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bd8f0640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40b8155e10 0x7f40b80ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bd0ef640 1 -- 172.21.15.83:0/3585754047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40b8155e10 msgr2=0x7f40b80ab710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bd0ef640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40b8155e10 0x7f40b80ab710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bd0ef640 1 -- 172.21.15.83:0/3585754047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40b8152f30 msgr2=0x7f40b8158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bd0ef640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40b8152f30 0x7f40b8158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bd0ef640 1 -- 172.21.15.83:0/3585754047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b814fcd0 con 0x7f40b80c4880 2024-09-19T07:24:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bc8ee640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40b8152f30 0x7f40b8158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:24:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bd0ef640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40b80c4880 0x7f40b80ad070 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f40b400b440 tx=0x7f40b400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40adffb640 1 -- 172.21.15.83:0/3585754047 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4015020 con 0x7f40b80c4880 2024-09-19T07:24:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bf353640 1 -- 172.21.15.83:0/3585754047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b806e2f0 con 0x7f40b80c4880 2024-09-19T07:24:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40bf353640 1 -- 172.21.15.83:0/3585754047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b81589b0 con 0x7f40b80c4880 2024-09-19T07:24:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40adffb640 1 -- 172.21.15.83:0/3585754047 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b4004020 con 0x7f40b80c4880 2024-09-19T07:24:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.929+0000 7f40adffb640 1 -- 172.21.15.83:0/3585754047 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4004410 con 0x7f40b80c4880 2024-09-19T07:24:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.933+0000 7f40bf353640 1 -- 172.21.15.83:0/3585754047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40980052a0 con 0x7f40b80c4880 2024-09-19T07:24:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.933+0000 7f40adffb640 1 -- 172.21.15.83:0/3585754047 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f40b4018020 con 0x7f40b80c4880 2024-09-19T07:24:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.933+0000 7f40adffb640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f409c0678c0 0x7f409c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.933+0000 7f40adffb640 1 -- 172.21.15.83:0/3585754047 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7f40b4086010 con 0x7f40b80c4880 2024-09-19T07:24:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.933+0000 7f40bc8ee640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f409c0678c0 0x7f409c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.933+0000 7f40bc8ee640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f409c0678c0 0x7f409c069d80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f40a80ad000 tx=0x7f40a8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:38.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:38.937+0000 7f40adffb640 1 -- 172.21.15.83:0/3585754047 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f40b401f030 con 0x7f40b80c4880 2024-09-19T07:24:39.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.153+0000 7f40bf353640 1 -- 172.21.15.83:0/3585754047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4098005860 con 0x7f40b80c4880 2024-09-19T07:24:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.153+0000 7f40adffb640 1 -- 172.21.15.83:0/3585754047 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6529 (secure 0 0 0) 0x7f40b400d040 con 0x7f40b80c4880 2024-09-19T07:24:39.154 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:39.154 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":48,"btime":"2024-09-19T07:23:58:201892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:24:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 48 2024-09-19T07:24:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.157+0000 7f40bf353640 1 -- 172.21.15.83:0/3585754047 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f409c0678c0 msgr2=0x7f409c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.157+0000 7f40bf353640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f409c0678c0 0x7f409c069d80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f40a80ad000 tx=0x7f40a8008040 comp rx=0 tx=0).stop 2024-09-19T07:24:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.157+0000 7f40bf353640 1 -- 172.21.15.83:0/3585754047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40b80c4880 msgr2=0x7f40b80ad070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.157+0000 7f40bf353640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40b80c4880 0x7f40b80ad070 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f40b400b440 tx=0x7f40b400b900 comp rx=0 tx=0).stop 2024-09-19T07:24:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.157+0000 7f40bf353640 1 -- 172.21.15.83:0/3585754047 shutdown_connections 2024-09-19T07:24:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.157+0000 7f40bf353640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f409c0678c0 0x7f409c069d80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.157+0000 7f40bf353640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40b8155e10 0x7f40b80ab710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.157+0000 7f40bf353640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40b8152f30 0x7f40b8158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.157+0000 7f40bf353640 1 --2- 172.21.15.83:0/3585754047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40b80c4880 0x7f40b80ad070 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.157+0000 7f40bf353640 1 -- 172.21.15.83:0/3585754047 >> 172.21.15.83:0/3585754047 conn(0x7f40b80aa910 msgr2=0x7f40b8154620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:39.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.157+0000 7f40bf353640 1 -- 172.21.15.83:0/3585754047 shutdown_connections 2024-09-19T07:24:39.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:39.157+0000 7f40bf353640 1 -- 172.21.15.83:0/3585754047 wait complete. 2024-09-19T07:24:39.174 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:24:41.175 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:41.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.317+0000 7f6d9b6f0640 1 Processor -- start 2024-09-19T07:24:41.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.317+0000 7f6d9b6f0640 1 -- start start 2024-09-19T07:24:41.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d9b6f0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6d940c6360 0x7f6d94153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:41.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d9b6f0640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6d940c9680 0x7f6d94157800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:41.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d9b6f0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d940c5b90 con 0x7f6d940bee30 2024-09-19T07:24:41.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d9b6f0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6d9415a050 con 0x7f6d940c6360 2024-09-19T07:24:41.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d9b6f0640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6d9406c950 con 0x7f6d940c9680 2024-09-19T07:24:41.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d9948c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6d940c6360 0x7f6d94153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:41.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d9948c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6d940c6360 0x7f6d94153050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53730/0 (socket says 172.21.15.83:53730) 2024-09-19T07:24:41.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d9948c640 1 -- 172.21.15.83:0/4266916872 learned_addr learned my addr 172.21.15.83:0/4266916872 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:41.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d99c8d640 1 --2- 172.21.15.83:0/4266916872 >> v2:172.21.15.184:3300/0 conn(0x7f6d940c9680 0x7f6d94157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:41.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d9948c640 1 -- 172.21.15.83:0/4266916872 >> v2:172.21.15.184:3300/0 conn(0x7f6d940c9680 msgr2=0x7f6d94157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:41.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d9948c640 1 --2- 172.21.15.83:0/4266916872 >> v2:172.21.15.184:3300/0 conn(0x7f6d940c9680 0x7f6d94157800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:41.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d9948c640 1 -- 172.21.15.83:0/4266916872 >> v1:172.21.15.83:6789/0 conn(0x7f6d940bee30 legacy=0x7f6d941570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:41.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.321+0000 7f6d9948c640 1 -- 172.21.15.83:0/4266916872 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d9406c670 con 0x7f6d940c6360 2024-09-19T07:24:41.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9948c640 1 --2- 172.21.15.83:0/4266916872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d940c6360 0x7f6d94153050 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6d9406cdf0 tx=0x7f6d9001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d527ae38ecf06 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:41.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d8bfff640 1 -- 172.21.15.83:0/4266916872 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d900b0070 con 0x7f6d940c6360 2024-09-19T07:24:41.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d8bfff640 1 -- 172.21.15.83:0/4266916872 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d900ab600 con 0x7f6d940c6360 2024-09-19T07:24:41.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d8bfff640 1 -- 172.21.15.83:0/4266916872 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d900ab9d0 con 0x7f6d940c6360 2024-09-19T07:24:41.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/4266916872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d940c6360 msgr2=0x7f6d94153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:41.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 --2- 172.21.15.83:0/4266916872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d940c6360 0x7f6d94153050 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6d9406cdf0 tx=0x7f6d9001c400 comp rx=0 tx=0).stop 2024-09-19T07:24:41.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/4266916872 shutdown_connections 2024-09-19T07:24:41.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 --2- 172.21.15.83:0/4266916872 >> v2:172.21.15.184:3300/0 conn(0x7f6d940c9680 0x7f6d94157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:41.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 --2- 172.21.15.83:0/4266916872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d940c6360 0x7f6d94153050 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:41.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/4266916872 >> 172.21.15.83:0/4266916872 conn(0x7f6d940722a0 msgr2=0x7f6d940726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:41.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/4266916872 shutdown_connections 2024-09-19T07:24:41.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/4266916872 wait complete. 2024-09-19T07:24:41.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 Processor -- start 2024-09-19T07:24:41.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 -- start start 2024-09-19T07:24:41.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d940bee30 0x7f6d9414c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:41.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d940c6360 0x7f6d940cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:41.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d940c9680 0x7f6d94145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:41.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6d940c68c0 con 0x7f6d940c6360 2024-09-19T07:24:41.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6d940c6740 con 0x7f6d940c9680 2024-09-19T07:24:41.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9b6f0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6d94162da0 con 0x7f6d940bee30 2024-09-19T07:24:41.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d98c8b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d940c6360 0x7f6d940cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:41.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d98c8b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d940c6360 0x7f6d940cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53736/0 (socket says 172.21.15.83:53736) 2024-09-19T07:24:41.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d98c8b640 1 -- 172.21.15.83:0/1245899974 learned_addr learned my addr 172.21.15.83:0/1245899974 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:41.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d9948c640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d940bee30 0x7f6d9414c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:41.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.325+0000 7f6d99c8d640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d940c9680 0x7f6d94145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:41.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d98c8b640 1 -- 172.21.15.83:0/1245899974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d940bee30 msgr2=0x7f6d9414c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:41.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d98c8b640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d940bee30 0x7f6d9414c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:41.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d98c8b640 1 -- 172.21.15.83:0/1245899974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d940c9680 msgr2=0x7f6d94145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:41.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d98c8b640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d940c9680 0x7f6d94145bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:41.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d98c8b640 1 -- 172.21.15.83:0/1245899974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d940cac00 con 0x7f6d940c6360 2024-09-19T07:24:41.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d9948c640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d940bee30 0x7f6d9414c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:24:41.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d98c8b640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d940c6360 0x7f6d940cb290 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6d8400c340 tx=0x7f6d8400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:41.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d89ffb640 1 -- 172.21.15.83:0/1245899974 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d840052d0 con 0x7f6d940c6360 2024-09-19T07:24:41.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d89ffb640 1 -- 172.21.15.83:0/1245899974 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d84005470 con 0x7f6d940c6360 2024-09-19T07:24:41.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/1245899974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d94146b90 con 0x7f6d940c6360 2024-09-19T07:24:41.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d89ffb640 1 -- 172.21.15.83:0/1245899974 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d84007260 con 0x7f6d940c6360 2024-09-19T07:24:41.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/1245899974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d940ca920 con 0x7f6d940c6360 2024-09-19T07:24:41.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/1245899974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d94153050 con 0x7f6d940c6360 2024-09-19T07:24:41.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d89ffb640 1 -- 172.21.15.83:0/1245899974 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6d84007400 con 0x7f6d940c6360 2024-09-19T07:24:41.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d89ffb640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6d780678c0 0x7f6d78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:41.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d89ffb640 1 -- 172.21.15.83:0/1245899974 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7f6d84085710 con 0x7f6d940c6360 2024-09-19T07:24:41.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d9948c640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6d780678c0 0x7f6d78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:41.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.329+0000 7f6d9948c640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6d780678c0 0x7f6d78069d80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6d900ad000 tx=0x7f6d90008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:41.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.333+0000 7f6d89ffb640 1 -- 172.21.15.83:0/1245899974 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6d84055c60 con 0x7f6d940c6360 2024-09-19T07:24:41.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.557+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/1245899974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d94157e10 con 0x7f6d940c6360 2024-09-19T07:24:41.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.557+0000 7f6d89ffb640 1 -- 172.21.15.83:0/1245899974 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6529 (secure 0 0 0) 0x7f6d84055560 con 0x7f6d940c6360 2024-09-19T07:24:41.558 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:41.558 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":48,"btime":"2024-09-19T07:23:58:201892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:24:41.558 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 48 2024-09-19T07:24:41.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.561+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/1245899974 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6d780678c0 msgr2=0x7f6d78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:41.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.561+0000 7f6d9b6f0640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6d780678c0 0x7f6d78069d80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6d900ad000 tx=0x7f6d90008040 comp rx=0 tx=0).stop 2024-09-19T07:24:41.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.561+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/1245899974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d940c6360 msgr2=0x7f6d940cb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:41.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.561+0000 7f6d9b6f0640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d940c6360 0x7f6d940cb290 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6d8400c340 tx=0x7f6d8400c800 comp rx=0 tx=0).stop 2024-09-19T07:24:41.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.561+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/1245899974 shutdown_connections 2024-09-19T07:24:41.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.561+0000 7f6d9b6f0640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6d780678c0 0x7f6d78069d80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:41.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.561+0000 7f6d9b6f0640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d940c9680 0x7f6d94145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:41.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.561+0000 7f6d9b6f0640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d940c6360 0x7f6d940cb290 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:41.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.561+0000 7f6d9b6f0640 1 --2- 172.21.15.83:0/1245899974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d940bee30 0x7f6d9414c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:41.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.561+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/1245899974 >> 172.21.15.83:0/1245899974 conn(0x7f6d940722a0 msgr2=0x7f6d941588f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:41.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.561+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/1245899974 shutdown_connections 2024-09-19T07:24:41.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:41.561+0000 7f6d9b6f0640 1 -- 172.21.15.83:0/1245899974 wait complete. 2024-09-19T07:24:41.581 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:24:43.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:43.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.717+0000 7f00a9aa0640 1 Processor -- start 2024-09-19T07:24:43.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.717+0000 7f00a9aa0640 1 -- start start 2024-09-19T07:24:43.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a9aa0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f00a4068700 0x7f00a406e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:43.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a9aa0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a4153390 0x7f00a406d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:43.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a9aa0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00a4155fa0 con 0x7f00a41494a0 2024-09-19T07:24:43.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a9aa0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f00a4156e40 con 0x7f00a4068700 2024-09-19T07:24:43.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a9aa0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f00a40c3850 con 0x7f00a4153390 2024-09-19T07:24:43.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f00a4068700 0x7f00a406e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:43.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f00a4068700 0x7f00a406e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53740/0 (socket says 172.21.15.83:53740) 2024-09-19T07:24:43.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a2ffd640 1 -- 172.21.15.83:0/3392248549 learned_addr learned my addr 172.21.15.83:0/3392248549 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:43.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a3fff640 1 --2- 172.21.15.83:0/3392248549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a4153390 0x7f00a406d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:43.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a2ffd640 1 -- 172.21.15.83:0/3392248549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a4153390 msgr2=0x7f00a406d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:43.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a2ffd640 1 --2- 172.21.15.83:0/3392248549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a4153390 0x7f00a406d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:43.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a2ffd640 1 -- 172.21.15.83:0/3392248549 >> v1:172.21.15.83:6789/0 conn(0x7f00a41494a0 legacy=0x7f00a4156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:43.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a2ffd640 1 -- 172.21.15.83:0/3392248549 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a40c3b10 con 0x7f00a4068700 2024-09-19T07:24:43.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a2ffd640 1 --2- 172.21.15.83:0/3392248549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4068700 0x7f00a406e010 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f009800dc90 tx=0x7f0098020870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4a274b498e1fbe server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:43.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a27fc640 1 -- 172.21.15.83:0/3392248549 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009800d2d0 con 0x7f00a4068700 2024-09-19T07:24:43.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a27fc640 1 -- 172.21.15.83:0/3392248549 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00980a9550 con 0x7f00a4068700 2024-09-19T07:24:43.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.721+0000 7f00a27fc640 1 -- 172.21.15.83:0/3392248549 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00980a9920 con 0x7f00a4068700 2024-09-19T07:24:43.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3392248549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4068700 msgr2=0x7f00a406e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:43.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 --2- 172.21.15.83:0/3392248549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4068700 0x7f00a406e010 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f009800dc90 tx=0x7f0098020870 comp rx=0 tx=0).stop 2024-09-19T07:24:43.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3392248549 shutdown_connections 2024-09-19T07:24:43.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 --2- 172.21.15.83:0/3392248549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4068700 0x7f00a406e010 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:43.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 --2- 172.21.15.83:0/3392248549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a4153390 0x7f00a406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:43.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3392248549 >> 172.21.15.83:0/3392248549 conn(0x7f00a40c4940 msgr2=0x7f00a40c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:43.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3392248549 shutdown_connections 2024-09-19T07:24:43.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3392248549 wait complete. 2024-09-19T07:24:43.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 Processor -- start 2024-09-19T07:24:43.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 -- start start 2024-09-19T07:24:43.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a4068700 0x7f00a40c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:43.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a41494a0 0x7f00a4150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:43.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4153390 0x7f00a41435e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:43.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00a4071eb0 con 0x7f00a4153390 2024-09-19T07:24:43.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f00a40c40d0 con 0x7f00a4068700 2024-09-19T07:24:43.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a9aa0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f00a4072030 con 0x7f00a41494a0 2024-09-19T07:24:43.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4153390 0x7f00a41435e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:43.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4153390 0x7f00a41435e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53756/0 (socket says 172.21.15.83:53756) 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a3fff640 1 -- 172.21.15.83:0/3362879931 learned_addr learned my addr 172.21.15.83:0/3362879931 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a2ffd640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a41494a0 0x7f00a4150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a37fe640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a4068700 0x7f00a40c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a3fff640 1 -- 172.21.15.83:0/3362879931 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a41494a0 msgr2=0x7f00a4150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a3fff640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a41494a0 0x7f00a4150780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a3fff640 1 -- 172.21.15.83:0/3362879931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a4068700 msgr2=0x7f00a40c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a3fff640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a4068700 0x7f00a40c59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a3fff640 1 -- 172.21.15.83:0/3362879931 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a40c5f10 con 0x7f00a4153390 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.725+0000 7f00a2ffd640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a41494a0 0x7f00a4150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.729+0000 7f00a3fff640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4153390 0x7f00a41435e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f009c007e90 tx=0x7f009c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.729+0000 7f008bfff640 1 -- 172.21.15.83:0/3362879931 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c014420 con 0x7f00a4153390 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.729+0000 7f008bfff640 1 -- 172.21.15.83:0/3362879931 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f009c019070 con 0x7f00a4153390 2024-09-19T07:24:43.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.729+0000 7f008bfff640 1 -- 172.21.15.83:0/3362879931 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c014800 con 0x7f00a4153390 2024-09-19T07:24:43.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.729+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3362879931 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a41464a0 con 0x7f00a4153390 2024-09-19T07:24:43.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.729+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3362879931 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a4150cc0 con 0x7f00a4153390 2024-09-19T07:24:43.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.729+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3362879931 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a4072360 con 0x7f00a4153390 2024-09-19T07:24:43.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.729+0000 7f008bfff640 1 -- 172.21.15.83:0/3362879931 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f009c0149a0 con 0x7f00a4153390 2024-09-19T07:24:43.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.729+0000 7f008bfff640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00900678c0 0x7f0090069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:43.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.729+0000 7f00a37fe640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00900678c0 0x7f0090069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:43.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.729+0000 7f008bfff640 1 -- 172.21.15.83:0/3362879931 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7f009c086ac0 con 0x7f00a4153390 2024-09-19T07:24:43.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.729+0000 7f00a37fe640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00900678c0 0x7f0090069d80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0094001650 tx=0x7f0094009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:43.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:43.737+0000 7f008bfff640 1 -- 172.21.15.83:0/3362879931 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f009c011260 con 0x7f00a4153390 2024-09-19T07:24:44.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.013+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3362879931 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00a4070060 con 0x7f00a4153390 2024-09-19T07:24:44.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f008bfff640 1 -- 172.21.15.83:0/3362879931 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6529 (secure 0 0 0) 0x7f009c057090 con 0x7f00a4153390 2024-09-19T07:24:44.016 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:44.016 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":48,"btime":"2024-09-19T07:23:58:201892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:24:44.017 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 48 2024-09-19T07:24:44.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3362879931 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00900678c0 msgr2=0x7f0090069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:44.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f00a9aa0640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00900678c0 0x7f0090069d80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0094001650 tx=0x7f0094009040 comp rx=0 tx=0).stop 2024-09-19T07:24:44.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3362879931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4153390 msgr2=0x7f00a41435e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f00a9aa0640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4153390 0x7f00a41435e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f009c007e90 tx=0x7f009c00e4b0 comp rx=0 tx=0).stop 2024-09-19T07:24:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3362879931 shutdown_connections 2024-09-19T07:24:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f00a9aa0640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00900678c0 0x7f0090069d80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f00a9aa0640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4153390 0x7f00a41435e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f00a9aa0640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a41494a0 0x7f00a4150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f00a9aa0640 1 --2- 172.21.15.83:0/3362879931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a4068700 0x7f00a40c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3362879931 >> 172.21.15.83:0/3362879931 conn(0x7f00a40c4940 msgr2=0x7f00a40c4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3362879931 shutdown_connections 2024-09-19T07:24:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:44.017+0000 7f00a9aa0640 1 -- 172.21.15.83:0/3362879931 wait complete. 2024-09-19T07:24:44.036 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:24:46.037 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:46.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.181+0000 7f762415c640 1 Processor -- start 2024-09-19T07:24:46.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.181+0000 7f762415c640 1 -- start start 2024-09-19T07:24:46.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f762415c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f761c1494a0 0x7f761c06e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:46.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f762415c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f761c153390 0x7f761c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:46.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f762415c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f761c155fa0 con 0x7f761c068700 2024-09-19T07:24:46.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f762415c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f761c156e40 con 0x7f761c1494a0 2024-09-19T07:24:46.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f762415c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f761c0c3850 con 0x7f761c153390 2024-09-19T07:24:46.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f7621ef8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f761c1494a0 0x7f761c06e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:46.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f7621ef8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f761c1494a0 0x7f761c06e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53768/0 (socket says 172.21.15.83:53768) 2024-09-19T07:24:46.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f7621ef8640 1 -- 172.21.15.83:0/2165920213 learned_addr learned my addr 172.21.15.83:0/2165920213 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:46.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f76226f9640 1 --2- 172.21.15.83:0/2165920213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f761c153390 0x7f761c06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:46.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f7621ef8640 1 -- 172.21.15.83:0/2165920213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f761c153390 msgr2=0x7f761c06d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:46.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f7621ef8640 1 --2- 172.21.15.83:0/2165920213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f761c153390 0x7f761c06d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:46.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f7621ef8640 1 -- 172.21.15.83:0/2165920213 >> v1:172.21.15.184:6789/0 conn(0x7f761c068700 legacy=0x7f761c156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:46.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.185+0000 7f7621ef8640 1 -- 172.21.15.83:0/2165920213 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761c0c3b10 con 0x7f761c1494a0 2024-09-19T07:24:46.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f7621ef8640 1 --2- 172.21.15.83:0/2165920213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f761c1494a0 0x7f761c06e010 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f761800bb10 tx=0x7f761801ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e47f229f1e117e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:46.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f7620ef6640 1 -- 172.21.15.83:0/2165920213 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76180b4070 con 0x7f761c1494a0 2024-09-19T07:24:46.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f7620ef6640 1 -- 172.21.15.83:0/2165920213 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76180a7a90 con 0x7f761c1494a0 2024-09-19T07:24:46.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f7620ef6640 1 -- 172.21.15.83:0/2165920213 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76180a7e60 con 0x7f761c1494a0 2024-09-19T07:24:46.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 -- 172.21.15.83:0/2165920213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f761c1494a0 msgr2=0x7f761c06e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:46.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 --2- 172.21.15.83:0/2165920213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f761c1494a0 0x7f761c06e010 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f761800bb10 tx=0x7f761801ebf0 comp rx=0 tx=0).stop 2024-09-19T07:24:46.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 -- 172.21.15.83:0/2165920213 shutdown_connections 2024-09-19T07:24:46.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 --2- 172.21.15.83:0/2165920213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f761c153390 0x7f761c06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:46.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 --2- 172.21.15.83:0/2165920213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f761c1494a0 0x7f761c06e010 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:46.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 -- 172.21.15.83:0/2165920213 >> 172.21.15.83:0/2165920213 conn(0x7f761c0c4940 msgr2=0x7f761c0c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:46.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 -- 172.21.15.83:0/2165920213 shutdown_connections 2024-09-19T07:24:46.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 -- 172.21.15.83:0/2165920213 wait complete. 2024-09-19T07:24:46.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 Processor -- start 2024-09-19T07:24:46.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 -- start start 2024-09-19T07:24:46.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f761c068700 0x7f761c1ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:46.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f761c1494a0 0x7f761c151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.189+0000 7f762415c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f761c153390 0x7f761c1542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f762415c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f761c071eb0 con 0x7f761c1494a0 2024-09-19T07:24:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f762415c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f761c0c40d0 con 0x7f761c153390 2024-09-19T07:24:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f762415c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f761c072030 con 0x7f761c068700 2024-09-19T07:24:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76216f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f761c1494a0 0x7f761c151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76216f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f761c1494a0 0x7f761c151a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53782/0 (socket says 172.21.15.83:53782) 2024-09-19T07:24:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76216f7640 1 -- 172.21.15.83:0/1796470518 learned_addr learned my addr 172.21.15.83:0/1796470518 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f7621ef8640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f761c068700 0x7f761c1ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76226f9640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f761c153390 0x7f761c1542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76216f7640 1 -- 172.21.15.83:0/1796470518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f761c068700 msgr2=0x7f761c1ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76216f7640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f761c068700 0x7f761c1ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:46.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76216f7640 1 -- 172.21.15.83:0/1796470518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f761c153390 msgr2=0x7f761c1542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:46.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76216f7640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f761c153390 0x7f761c1542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:46.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76216f7640 1 -- 172.21.15.83:0/1796470518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761c0c6fc0 con 0x7f761c1494a0 2024-09-19T07:24:46.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f7621ef8640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f761c068700 0x7f761c1ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:24:46.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76216f7640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f761c1494a0 0x7f761c151a30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f760c00c450 tx=0x7f760c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:46.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76127fc640 1 -- 172.21.15.83:0/1796470518 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c012830 con 0x7f761c1494a0 2024-09-19T07:24:46.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76127fc640 1 -- 172.21.15.83:0/1796470518 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f760c017070 con 0x7f761c1494a0 2024-09-19T07:24:46.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76127fc640 1 -- 172.21.15.83:0/1796470518 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c012c30 con 0x7f761c1494a0 2024-09-19T07:24:46.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f762415c640 1 -- 172.21.15.83:0/1796470518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f761c0c72a0 con 0x7f761c1494a0 2024-09-19T07:24:46.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f762415c640 1 -- 172.21.15.83:0/1796470518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f761c0c7860 con 0x7f761c1494a0 2024-09-19T07:24:46.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f762415c640 1 -- 172.21.15.83:0/1796470518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f761c072400 con 0x7f761c1494a0 2024-09-19T07:24:46.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76127fc640 1 -- 172.21.15.83:0/1796470518 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f760c01c460 con 0x7f761c1494a0 2024-09-19T07:24:46.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76127fc640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f76000678c0 0x7f7600069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:46.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f7621ef8640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f76000678c0 0x7f7600069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:46.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f76127fc640 1 -- 172.21.15.83:0/1796470518 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7239+0+0 (secure 0 0 0) 0x7f760c085fb0 con 0x7f761c1494a0 2024-09-19T07:24:46.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.193+0000 7f7621ef8640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f76000678c0 0x7f7600069d80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f76180b1000 tx=0x7f76180a9070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:46.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.201+0000 7f76127fc640 1 -- 172.21.15.83:0/1796470518 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f760c056580 con 0x7f761c1494a0 2024-09-19T07:24:46.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.457+0000 7f762415c640 1 -- 172.21.15.83:0/1796470518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f761c070060 con 0x7f761c1494a0 2024-09-19T07:24:46.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f76127fc640 1 -- 172.21.15.83:0/1796470518 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6529 (secure 0 0 0) 0x7f760c0599a0 con 0x7f761c1494a0 2024-09-19T07:24:46.461 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:46.461 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":48,"btime":"2024-09-19T07:23:58:201892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:23:41.116073+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5304":{"gid":5304,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/3442755673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3442755673},{"type":"v1","addr":"172.21.15.83:6835","nonce":3442755673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5304,"qdb_cluster":[5304]},"id":2}]} 2024-09-19T07:24:46.461 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 48 2024-09-19T07:24:46.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f762415c640 1 -- 172.21.15.83:0/1796470518 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f76000678c0 msgr2=0x7f7600069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:46.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f762415c640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f76000678c0 0x7f7600069d80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f76180b1000 tx=0x7f76180a9070 comp rx=0 tx=0).stop 2024-09-19T07:24:46.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f762415c640 1 -- 172.21.15.83:0/1796470518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f761c1494a0 msgr2=0x7f761c151a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:46.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f762415c640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f761c1494a0 0x7f761c151a30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f760c00c450 tx=0x7f760c00c910 comp rx=0 tx=0).stop 2024-09-19T07:24:46.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f762415c640 1 -- 172.21.15.83:0/1796470518 shutdown_connections 2024-09-19T07:24:46.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f762415c640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f76000678c0 0x7f7600069d80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:46.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f762415c640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f761c153390 0x7f761c1542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:46.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f762415c640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f761c1494a0 0x7f761c151a30 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:46.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f762415c640 1 --2- 172.21.15.83:0/1796470518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f761c068700 0x7f761c1ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:46.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f762415c640 1 -- 172.21.15.83:0/1796470518 >> 172.21.15.83:0/1796470518 conn(0x7f761c0c4940 msgr2=0x7f761c15c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:46.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f762415c640 1 -- 172.21.15.83:0/1796470518 shutdown_connections 2024-09-19T07:24:46.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:46.461+0000 7f762415c640 1 -- 172.21.15.83:0/1796470518 wait complete. 2024-09-19T07:24:46.481 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:24:48.482 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:48.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.617+0000 7faac179e640 1 Processor -- start 2024-09-19T07:24:48.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.617+0000 7faac179e640 1 -- start start 2024-09-19T07:24:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.617+0000 7faac179e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7faabc071100 0x7faabc1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.617+0000 7faac179e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faabc06dea0 0x7faabc06e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.617+0000 7faac179e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faabc0a9e60 con 0x7faabc0c28e0 2024-09-19T07:24:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.617+0000 7faac179e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7faabc071b50 con 0x7faabc071100 2024-09-19T07:24:48.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.617+0000 7faac179e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faabc0c1ff0 con 0x7faabc06dea0 2024-09-19T07:24:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faabaffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7faabc071100 0x7faabc1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faabb7fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faabc06dea0 0x7faabc06e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faaba7fc640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7faabc0c28e0 0x7faabc150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:52262/0 (socket says 172.21.15.83:52262) 2024-09-19T07:24:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faaba7fc640 1 -- 172.21.15.83:0/2607083430 learned_addr learned my addr 172.21.15.83:0/2607083430 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:24:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faabaffd640 1 -- 172.21.15.83:0/2607083430 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faabc06dea0 msgr2=0x7faabc06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faabaffd640 1 --2- 172.21.15.83:0/2607083430 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faabc06dea0 0x7faabc06e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faabaffd640 1 -- 172.21.15.83:0/2607083430 >> v1:172.21.15.184:6789/0 conn(0x7faabc0c28e0 legacy=0x7faabc150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faabaffd640 1 -- 172.21.15.83:0/2607083430 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faabc0aa140 con 0x7faabc071100 2024-09-19T07:24:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faabb7fe640 1 --2- 172.21.15.83:0/2607083430 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faabc06dea0 0x7faabc06e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:24:48.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faabaffd640 1 --2- 172.21.15.83:0/2607083430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faabc071100 0x7faabc1545e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faaac0097c0 tx=0x7faaac01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f66a8bf4c06ea2b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:48.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faab9ffb640 1 -- 172.21.15.83:0/2607083430 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac0b0070 con 0x7faabc071100 2024-09-19T07:24:48.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faab9ffb640 1 -- 172.21.15.83:0/2607083430 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaac0ab600 con 0x7faabc071100 2024-09-19T07:24:48.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.621+0000 7faab9ffb640 1 -- 172.21.15.83:0/2607083430 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac0ab9d0 con 0x7faabc071100 2024-09-19T07:24:48.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 -- 172.21.15.83:0/2607083430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faabc071100 msgr2=0x7faabc1545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:48.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 --2- 172.21.15.83:0/2607083430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faabc071100 0x7faabc1545e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faaac0097c0 tx=0x7faaac01c410 comp rx=0 tx=0).stop 2024-09-19T07:24:48.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 -- 172.21.15.83:0/2607083430 shutdown_connections 2024-09-19T07:24:48.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 --2- 172.21.15.83:0/2607083430 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faabc06dea0 0x7faabc06e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:48.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 --2- 172.21.15.83:0/2607083430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faabc071100 0x7faabc1545e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:48.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 -- 172.21.15.83:0/2607083430 >> 172.21.15.83:0/2607083430 conn(0x7faabc06bc80 msgr2=0x7faabc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:48.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 -- 172.21.15.83:0/2607083430 shutdown_connections 2024-09-19T07:24:48.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 -- 172.21.15.83:0/2607083430 wait complete. 2024-09-19T07:24:48.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 Processor -- start 2024-09-19T07:24:48.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 -- start start 2024-09-19T07:24:48.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faabc06dea0 0x7faabc152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:48.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faabc071100 0x7faabc156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:48.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faabc0c28e0 0x7faabc155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:48.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faabc071660 con 0x7faabc0c28e0 2024-09-19T07:24:48.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faabc0714e0 con 0x7faabc06dea0 2024-09-19T07:24:48.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faac179e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faabc160c60 con 0x7faabc071100 2024-09-19T07:24:48.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faabb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faabc0c28e0 0x7faabc155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:48.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faabb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faabc0c28e0 0x7faabc155dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53786/0 (socket says 172.21.15.83:53786) 2024-09-19T07:24:48.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.625+0000 7faabb7fe640 1 -- 172.21.15.83:0/1151068089 learned_addr learned my addr 172.21.15.83:0/1151068089 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:48.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faabaffd640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faabc06dea0 0x7faabc152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:48.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faaba7fc640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faabc071100 0x7faabc156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:48.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faabb7fe640 1 -- 172.21.15.83:0/1151068089 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faabc071100 msgr2=0x7faabc156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:48.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faabb7fe640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faabc071100 0x7faabc156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faabb7fe640 1 -- 172.21.15.83:0/1151068089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faabc06dea0 msgr2=0x7faabc152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faabb7fe640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faabc06dea0 0x7faabc152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faabb7fe640 1 -- 172.21.15.83:0/1151068089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faabc154270 con 0x7faabc0c28e0 2024-09-19T07:24:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faabb7fe640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faabc0c28e0 0x7faabc155dc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7faab400efc0 tx=0x7faab4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faaa37fe640 1 -- 172.21.15.83:0/1151068089 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab4018030 con 0x7faabc0c28e0 2024-09-19T07:24:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faaa37fe640 1 -- 172.21.15.83:0/1151068089 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab400cde0 con 0x7faabc0c28e0 2024-09-19T07:24:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faac179e640 1 -- 172.21.15.83:0/1151068089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faabc1535e0 con 0x7faabc0c28e0 2024-09-19T07:24:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faaa37fe640 1 -- 172.21.15.83:0/1151068089 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab4003bc0 con 0x7faabc0c28e0 2024-09-19T07:24:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faac179e640 1 -- 172.21.15.83:0/1151068089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faabc1edad0 con 0x7faabc0c28e0 2024-09-19T07:24:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faac179e640 1 -- 172.21.15.83:0/1151068089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faabc1ed900 con 0x7faabc0c28e0 2024-09-19T07:24:48.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faaa37fe640 1 -- 172.21.15.83:0/1151068089 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faab401b070 con 0x7faabc0c28e0 2024-09-19T07:24:48.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faaa37fe640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faaa80678f0 0x7faaa8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faaa37fe640 1 -- 172.21.15.83:0/1151068089 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7faab4086be0 con 0x7faabc0c28e0 2024-09-19T07:24:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.629+0000 7faabaffd640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faaa80678f0 0x7faaa8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.633+0000 7faabaffd640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faaa80678f0 0x7faaa8069db0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faaac0ad000 tx=0x7faaac0a4f90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:48.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.637+0000 7faaa37fe640 1 -- 172.21.15.83:0/1151068089 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7faab4022070 con 0x7faabc0c28e0 2024-09-19T07:24:48.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.869+0000 7faac179e640 1 -- 172.21.15.83:0/1151068089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faabc153ae0 con 0x7faabc0c28e0 2024-09-19T07:24:48.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faaa37fe640 1 -- 172.21.15.83:0/1151068089 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5696 (secure 0 0 0) 0x7faab4010040 con 0x7faabc0c28e0 2024-09-19T07:24:48.872 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:48.872 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":49,"btime":"2024-09-19T07:24:47:982813+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:47.982803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:24:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 49 2024-09-19T07:24:48.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faac179e640 1 -- 172.21.15.83:0/1151068089 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faaa80678f0 msgr2=0x7faaa8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:48.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faac179e640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faaa80678f0 0x7faaa8069db0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faaac0ad000 tx=0x7faaac0a4f90 comp rx=0 tx=0).stop 2024-09-19T07:24:48.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faac179e640 1 -- 172.21.15.83:0/1151068089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faabc0c28e0 msgr2=0x7faabc155dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faac179e640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faabc0c28e0 0x7faabc155dc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7faab400efc0 tx=0x7faab4009f90 comp rx=0 tx=0).stop 2024-09-19T07:24:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faac179e640 1 -- 172.21.15.83:0/1151068089 shutdown_connections 2024-09-19T07:24:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faac179e640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faaa80678f0 0x7faaa8069db0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faac179e640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faabc0c28e0 0x7faabc155dc0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faac179e640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faabc071100 0x7faabc156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faac179e640 1 --2- 172.21.15.83:0/1151068089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faabc06dea0 0x7faabc152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faac179e640 1 -- 172.21.15.83:0/1151068089 >> 172.21.15.83:0/1151068089 conn(0x7faabc06bc80 msgr2=0x7faabc150f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faac179e640 1 -- 172.21.15.83:0/1151068089 shutdown_connections 2024-09-19T07:24:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:48.873+0000 7faac179e640 1 -- 172.21.15.83:0/1151068089 wait complete. 2024-09-19T07:24:48.895 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-19T07:24:48.895 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:24:48.895 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:49.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.021+0000 7f787a46f640 1 Processor -- start 2024-09-19T07:24:49.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.021+0000 7f787a46f640 1 -- start start 2024-09-19T07:24:49.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f787a46f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f78740c4690 0x7f78740aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:49.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f787a46f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f78740c3f70 0x7f787415e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:49.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f787a46f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7874073e00 con 0x7f787414dac0 2024-09-19T07:24:49.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f787a46f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7874154700 con 0x7f78740c4690 2024-09-19T07:24:49.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f787a46f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f78740740a0 con 0x7f78740c3f70 2024-09-19T07:24:49.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f7878c6c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f78740c4690 0x7f78740aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:49.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f7878c6c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f78740c4690 0x7f78740aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53790/0 (socket says 172.21.15.83:53790) 2024-09-19T07:24:49.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f7878c6c640 1 -- 172.21.15.83:0/554008529 learned_addr learned my addr 172.21.15.83:0/554008529 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:49.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f7879c6e640 1 --2- 172.21.15.83:0/554008529 >> v2:172.21.15.184:3300/0 conn(0x7f78740c3f70 0x7f787415e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:49.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f7878c6c640 1 -- 172.21.15.83:0/554008529 >> v2:172.21.15.184:3300/0 conn(0x7f78740c3f70 msgr2=0x7f787415e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:49.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f7878c6c640 1 --2- 172.21.15.83:0/554008529 >> v2:172.21.15.184:3300/0 conn(0x7f78740c3f70 0x7f787415e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:49.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f7878c6c640 1 -- 172.21.15.83:0/554008529 >> v1:172.21.15.83:6789/0 conn(0x7f787414dac0 legacy=0x7f78740ac620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:49.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f7878c6c640 1 -- 172.21.15.83:0/554008529 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78740ae680 con 0x7f78740c4690 2024-09-19T07:24:49.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.025+0000 7f7878c6c640 1 --2- 172.21.15.83:0/554008529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78740c4690 0x7f78740aeba0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7874157ec0 tx=0x7f786401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5232e81f4f809285 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:49.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.029+0000 7f786bfff640 1 -- 172.21.15.83:0/554008529 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78640b0070 con 0x7f78740c4690 2024-09-19T07:24:49.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.029+0000 7f786bfff640 1 -- 172.21.15.83:0/554008529 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78640ab600 con 0x7f78740c4690 2024-09-19T07:24:49.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.029+0000 7f786bfff640 1 -- 172.21.15.83:0/554008529 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78640ab9d0 con 0x7f78740c4690 2024-09-19T07:24:49.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.029+0000 7f787a46f640 1 -- 172.21.15.83:0/554008529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78740c4690 msgr2=0x7f78740aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:49.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.029+0000 7f787a46f640 1 --2- 172.21.15.83:0/554008529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78740c4690 0x7f78740aeba0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7874157ec0 tx=0x7f786401c490 comp rx=0 tx=0).stop 2024-09-19T07:24:49.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.029+0000 7f787a46f640 1 -- 172.21.15.83:0/554008529 shutdown_connections 2024-09-19T07:24:49.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.029+0000 7f787a46f640 1 --2- 172.21.15.83:0/554008529 >> v2:172.21.15.184:3300/0 conn(0x7f78740c3f70 0x7f787415e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:49.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.029+0000 7f787a46f640 1 --2- 172.21.15.83:0/554008529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78740c4690 0x7f78740aeba0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:49.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.029+0000 7f787a46f640 1 -- 172.21.15.83:0/554008529 >> 172.21.15.83:0/554008529 conn(0x7f78740c6070 msgr2=0x7f78740c6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:49.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.029+0000 7f787a46f640 1 -- 172.21.15.83:0/554008529 shutdown_connections 2024-09-19T07:24:49.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.029+0000 7f787a46f640 1 -- 172.21.15.83:0/554008529 wait complete. 2024-09-19T07:24:49.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f787a46f640 1 Processor -- start 2024-09-19T07:24:49.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f787a46f640 1 -- start start 2024-09-19T07:24:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f787a46f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78740c3f70 0x7f78741eccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f787a46f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78740c4690 0x7f78741439c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f787a46f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f787414dac0 0x7f7874157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f787a46f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7874161380 con 0x7f78740c4690 2024-09-19T07:24:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f787a46f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7874161200 con 0x7f787414dac0 2024-09-19T07:24:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f787a46f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7874161500 con 0x7f78740c3f70 2024-09-19T07:24:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7878c6c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78740c4690 0x7f78741439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7878c6c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78740c4690 0x7f78741439c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53792/0 (socket says 172.21.15.83:53792) 2024-09-19T07:24:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7878c6c640 1 -- 172.21.15.83:0/640457125 learned_addr learned my addr 172.21.15.83:0/640457125 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f787946d640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78740c3f70 0x7f78741eccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7879c6e640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f787414dac0 0x7f7874157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7878c6c640 1 -- 172.21.15.83:0/640457125 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78740c3f70 msgr2=0x7f78741eccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7878c6c640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78740c3f70 0x7f78741eccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7878c6c640 1 -- 172.21.15.83:0/640457125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f787414dac0 msgr2=0x7f7874157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7878c6c640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f787414dac0 0x7f7874157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7878c6c640 1 -- 172.21.15.83:0/640457125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78741f0790 con 0x7f78740c4690 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7878c6c640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78740c4690 0x7f78741439c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f78640ae040 tx=0x7f7864019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7869ffb640 1 -- 172.21.15.83:0/640457125 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78640b9070 con 0x7f78740c4690 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7869ffb640 1 -- 172.21.15.83:0/640457125 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78640b4400 con 0x7f78740c4690 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7869ffb640 1 -- 172.21.15.83:0/640457125 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78640b0040 con 0x7f78740c4690 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f787a46f640 1 -- 172.21.15.83:0/640457125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78741f0a70 con 0x7f78740c4690 2024-09-19T07:24:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f787a46f640 1 -- 172.21.15.83:0/640457125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78741f1030 con 0x7f78740c4690 2024-09-19T07:24:49.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f787a46f640 1 -- 172.21.15.83:0/640457125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78740aeba0 con 0x7f78740c4690 2024-09-19T07:24:49.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7869ffb640 1 -- 172.21.15.83:0/640457125 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f78640b33a0 con 0x7f78740c4690 2024-09-19T07:24:49.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.033+0000 7f7869ffb640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78580678c0 0x7f7858069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:49.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.037+0000 7f7869ffb640 1 -- 172.21.15.83:0/640457125 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7f786411f470 con 0x7f78740c4690 2024-09-19T07:24:49.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.037+0000 7f787946d640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78580678c0 0x7f7858069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:49.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.037+0000 7f787946d640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78580678c0 0x7f7858069d80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7870007fd0 tx=0x7f7870009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.041+0000 7f7869ffb640 1 -- 172.21.15.83:0/640457125 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7864016110 con 0x7f78740c4690 2024-09-19T07:24:49.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.277+0000 7f787a46f640 1 -- 172.21.15.83:0/640457125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78740c3790 con 0x7f78740c4690 2024-09-19T07:24:49.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f7869ffb640 1 -- 172.21.15.83:0/640457125 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5700 (secure 0 0 0) 0x7f78640ef3a0 con 0x7f78740c4690 2024-09-19T07:24:49.281 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:49.281 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":50,"btime":"2024-09-19T07:24:49:024071+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:48.258789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:24:49.281 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 50 2024-09-19T07:24:49.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f787a46f640 1 -- 172.21.15.83:0/640457125 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78580678c0 msgr2=0x7f7858069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:49.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f787a46f640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78580678c0 0x7f7858069d80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7870007fd0 tx=0x7f7870009040 comp rx=0 tx=0).stop 2024-09-19T07:24:49.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f787a46f640 1 -- 172.21.15.83:0/640457125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78740c4690 msgr2=0x7f78741439c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:49.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f787a46f640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78740c4690 0x7f78741439c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f78640ae040 tx=0x7f7864019230 comp rx=0 tx=0).stop 2024-09-19T07:24:49.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f787a46f640 1 -- 172.21.15.83:0/640457125 shutdown_connections 2024-09-19T07:24:49.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f787a46f640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78580678c0 0x7f7858069d80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:49.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f787a46f640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f787414dac0 0x7f7874157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:49.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f787a46f640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78740c4690 0x7f78741439c0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:49.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f787a46f640 1 --2- 172.21.15.83:0/640457125 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78740c3f70 0x7f78741eccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:49.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f787a46f640 1 -- 172.21.15.83:0/640457125 >> 172.21.15.83:0/640457125 conn(0x7f78740c6070 msgr2=0x7f78740c7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:49.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f787a46f640 1 -- 172.21.15.83:0/640457125 shutdown_connections 2024-09-19T07:24:49.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:49.281+0000 7f787a46f640 1 -- 172.21.15.83:0/640457125 wait complete. 2024-09-19T07:24:49.302 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:24:49:024071+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 47, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5463": { "addr": "172.21.15.83:6837/26497755", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 26497755, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 26497755, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5463, "incarnation": 44, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:23:55.188280+0000", "qdb_cluster": [ 5463 ], "qdb_leader": 5463, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5463 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5098": { "addr": "172.21.15.184:6835/144714632", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 144714632, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 144714632, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5098, "incarnation": 49, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:24:48.258789+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5098 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 28, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:24:51.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:51.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa30e2640 1 Processor -- start 2024-09-19T07:24:51.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa30e2640 1 -- start start 2024-09-19T07:24:51.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa30e2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fea9c0c6120 0x7fea9c0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:51.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa30e2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fea9c155d70 0x7fea9c1575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:51.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa30e2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea9c156200 con 0x7fea9c1592f0 2024-09-19T07:24:51.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa30e2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fea9c0c9930 con 0x7fea9c0c6120 2024-09-19T07:24:51.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa30e2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fea9c1564e0 con 0x7fea9c155d70 2024-09-19T07:24:51.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa0e7e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fea9c0c6120 0x7fea9c0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:51.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa0e7e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fea9c0c6120 0x7fea9c0c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47120/0 (socket says 172.21.15.83:47120) 2024-09-19T07:24:51.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa0e7e640 1 -- 172.21.15.83:0/111246462 learned_addr learned my addr 172.21.15.83:0/111246462 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:51.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7fea9bfff640 1 --2- 172.21.15.83:0/111246462 >> v2:172.21.15.184:3300/0 conn(0x7fea9c155d70 0x7fea9c1575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:51.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa0e7e640 1 -- 172.21.15.83:0/111246462 >> v2:172.21.15.184:3300/0 conn(0x7fea9c155d70 msgr2=0x7fea9c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:51.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa0e7e640 1 --2- 172.21.15.83:0/111246462 >> v2:172.21.15.184:3300/0 conn(0x7fea9c155d70 0x7fea9c1575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:51.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa0e7e640 1 -- 172.21.15.83:0/111246462 >> v1:172.21.15.83:6789/0 conn(0x7fea9c1592f0 legacy=0x7fea9c157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:51.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.445+0000 7feaa0e7e640 1 -- 172.21.15.83:0/111246462 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9c1567c0 con 0x7fea9c0c6120 2024-09-19T07:24:51.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.449+0000 7feaa0e7e640 1 --2- 172.21.15.83:0/111246462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea9c0c6120 0x7fea9c0c6500 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fea9c06cca0 tx=0x7fea9001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2931c5dee0a4fd04 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.449+0000 7fea9b7fe640 1 -- 172.21.15.83:0/111246462 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea900b0070 con 0x7fea9c0c6120 2024-09-19T07:24:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.449+0000 7fea9b7fe640 1 -- 172.21.15.83:0/111246462 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea900ab600 con 0x7fea9c0c6120 2024-09-19T07:24:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.449+0000 7fea9b7fe640 1 -- 172.21.15.83:0/111246462 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea900ab9d0 con 0x7fea9c0c6120 2024-09-19T07:24:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.449+0000 7feaa30e2640 1 -- 172.21.15.83:0/111246462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea9c0c6120 msgr2=0x7fea9c0c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.449+0000 7feaa30e2640 1 --2- 172.21.15.83:0/111246462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea9c0c6120 0x7fea9c0c6500 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fea9c06cca0 tx=0x7fea9001c520 comp rx=0 tx=0).stop 2024-09-19T07:24:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.449+0000 7feaa30e2640 1 -- 172.21.15.83:0/111246462 shutdown_connections 2024-09-19T07:24:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.449+0000 7feaa30e2640 1 --2- 172.21.15.83:0/111246462 >> v2:172.21.15.184:3300/0 conn(0x7fea9c155d70 0x7fea9c1575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.449+0000 7feaa30e2640 1 --2- 172.21.15.83:0/111246462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea9c0c6120 0x7fea9c0c6500 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.449+0000 7feaa30e2640 1 -- 172.21.15.83:0/111246462 >> 172.21.15.83:0/111246462 conn(0x7fea9c06bc80 msgr2=0x7fea9c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:51.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.449+0000 7feaa30e2640 1 -- 172.21.15.83:0/111246462 shutdown_connections 2024-09-19T07:24:51.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.449+0000 7feaa30e2640 1 -- 172.21.15.83:0/111246462 wait complete. 2024-09-19T07:24:51.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa30e2640 1 Processor -- start 2024-09-19T07:24:51.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa30e2640 1 -- start start 2024-09-19T07:24:51.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa30e2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea9c0c6120 0x7fea9c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:51.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa30e2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea9c1592f0 0x7fea9c149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:51.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa30e2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea9c0ab450 0x7fea9c149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:51.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa30e2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c153860 con 0x7fea9c0c6120 2024-09-19T07:24:51.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa30e2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c162ea0 con 0x7fea9c1592f0 2024-09-19T07:24:51.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa30e2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fea9c15f4a0 con 0x7fea9c0ab450 2024-09-19T07:24:51.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa0e7e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea9c0c6120 0x7fea9c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:51.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa0e7e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea9c0c6120 0x7fea9c0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47134/0 (socket says 172.21.15.83:47134) 2024-09-19T07:24:51.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa0e7e640 1 -- 172.21.15.83:0/2590969728 learned_addr learned my addr 172.21.15.83:0/2590969728 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:51.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa167f640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea9c0ab450 0x7fea9c149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:51.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7fea9bfff640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea9c1592f0 0x7fea9c149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa0e7e640 1 -- 172.21.15.83:0/2590969728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea9c0ab450 msgr2=0x7fea9c149550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa0e7e640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea9c0ab450 0x7fea9c149550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa0e7e640 1 -- 172.21.15.83:0/2590969728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea9c1592f0 msgr2=0x7fea9c149010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa0e7e640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea9c1592f0 0x7fea9c149010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa0e7e640 1 -- 172.21.15.83:0/2590969728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9c0c1ec0 con 0x7fea9c0c6120 2024-09-19T07:24:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa167f640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea9c0ab450 0x7fea9c149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:24:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7fea9bfff640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea9c1592f0 0x7fea9c149010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:24:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa0e7e640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea9c0c6120 0x7fea9c0cae00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fea9001abd0 tx=0x7fea9001ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7fea997fa640 1 -- 172.21.15.83:0/2590969728 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea900ba020 con 0x7fea9c0c6120 2024-09-19T07:24:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7fea997fa640 1 -- 172.21.15.83:0/2590969728 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea900b6620 con 0x7fea9c0c6120 2024-09-19T07:24:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa30e2640 1 -- 172.21.15.83:0/2590969728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea9c0c21a0 con 0x7fea9c0c6120 2024-09-19T07:24:51.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa30e2640 1 -- 172.21.15.83:0/2590969728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea9c1486d0 con 0x7fea9c0c6120 2024-09-19T07:24:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7fea997fa640 1 -- 172.21.15.83:0/2590969728 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea900b0040 con 0x7fea9c0c6120 2024-09-19T07:24:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7feaa30e2640 1 -- 172.21.15.83:0/2590969728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea9c0c6500 con 0x7fea9c0c6120 2024-09-19T07:24:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.453+0000 7fea997fa640 1 -- 172.21.15.83:0/2590969728 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fea900bd020 con 0x7fea9c0c6120 2024-09-19T07:24:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.457+0000 7fea997fa640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fea800678c0 0x7fea80069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.457+0000 7fea997fa640 1 -- 172.21.15.83:0/2590969728 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7fea901204d0 con 0x7fea9c0c6120 2024-09-19T07:24:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.457+0000 7fea9bfff640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fea800678c0 0x7fea80069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.457+0000 7fea9bfff640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fea800678c0 0x7fea80069d80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0c6780 tx=0x7fea88008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:51.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.461+0000 7fea997fa640 1 -- 172.21.15.83:0/2590969728 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fea900c4030 con 0x7fea9c0c6120 2024-09-19T07:24:51.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.677+0000 7feaa30e2640 1 -- 172.21.15.83:0/2590969728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea9c0c2370 con 0x7fea9c0c6120 2024-09-19T07:24:51.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.677+0000 7fea997fa640 1 -- 172.21.15.83:0/2590969728 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5704 (secure 0 0 0) 0x7fea90016110 con 0x7fea9c0c6120 2024-09-19T07:24:51.678 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:51.678 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":52,"btime":"2024-09-19T07:24:51:040265+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:24:51.679 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 52 2024-09-19T07:24:51.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.677+0000 7feaa30e2640 1 -- 172.21.15.83:0/2590969728 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fea800678c0 msgr2=0x7fea80069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:51.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.677+0000 7feaa30e2640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fea800678c0 0x7fea80069d80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0c6780 tx=0x7fea88008040 comp rx=0 tx=0).stop 2024-09-19T07:24:51.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.677+0000 7feaa30e2640 1 -- 172.21.15.83:0/2590969728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea9c0c6120 msgr2=0x7fea9c0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:51.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.677+0000 7feaa30e2640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea9c0c6120 0x7fea9c0cae00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fea9001abd0 tx=0x7fea9001ad60 comp rx=0 tx=0).stop 2024-09-19T07:24:51.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.677+0000 7feaa30e2640 1 -- 172.21.15.83:0/2590969728 shutdown_connections 2024-09-19T07:24:51.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.677+0000 7feaa30e2640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fea800678c0 0x7fea80069d80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:51.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.677+0000 7feaa30e2640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea9c0ab450 0x7fea9c149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:51.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.677+0000 7feaa30e2640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea9c1592f0 0x7fea9c149010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:51.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.677+0000 7feaa30e2640 1 --2- 172.21.15.83:0/2590969728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea9c0c6120 0x7fea9c0cae00 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:51.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.677+0000 7feaa30e2640 1 -- 172.21.15.83:0/2590969728 >> 172.21.15.83:0/2590969728 conn(0x7fea9c06bc80 msgr2=0x7fea9c15ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:51.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.681+0000 7feaa30e2640 1 -- 172.21.15.83:0/2590969728 shutdown_connections 2024-09-19T07:24:51.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:51.681+0000 7feaa30e2640 1 -- 172.21.15.83:0/2590969728 wait complete. 2024-09-19T07:24:51.699 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=5304 2024-09-19T07:24:51.699 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.a 2024-09-19T07:24:53.211 INFO:tasks.ceph.mds.b.smithi184.stderr:2024-09-19T07:24:53.209+0000 7f91b7de6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:24:58.700 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-19T07:24:58.700 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:24:58.701 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:24:58.704 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:24:58.704 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:24:58.783 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:24:58.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f652e128640 1 Processor -- start 2024-09-19T07:24:58.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f652e128640 1 -- start start 2024-09-19T07:24:58.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f652e128640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6528155d70 0x7f65280c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:58.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f652e128640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f65281592f0 0x7f6528157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f652e128640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6528156200 con 0x7f65280c6120 2024-09-19T07:24:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f652e128640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f65280c9930 con 0x7f6528155d70 2024-09-19T07:24:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f652e128640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f65281564e0 con 0x7f65281592f0 2024-09-19T07:24:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f651ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6528155d70 0x7f65280c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f65277fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f65280c6120 0x7f65281575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51194/0 (socket says 172.21.15.83:51194) 2024-09-19T07:24:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f65277fe640 1 -- 172.21.15.83:0/1378805227 learned_addr learned my addr 172.21.15.83:0/1378805227 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:24:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f6527fff640 1 --2- 172.21.15.83:0/1378805227 >> v2:172.21.15.184:3300/0 conn(0x7f65281592f0 0x7f6528157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f651ffff640 1 -- 172.21.15.83:0/1378805227 >> v2:172.21.15.184:3300/0 conn(0x7f65281592f0 msgr2=0x7f6528157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:24:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f6526ffd640 1 -- 172.21.15.83:0/1378805227 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3806582694 0 0) 0x7f6528156200 con 0x7f65280c6120 2024-09-19T07:24:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f651ffff640 1 --2- 172.21.15.83:0/1378805227 >> v2:172.21.15.184:3300/0 conn(0x7f65281592f0 0x7f6528157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f651ffff640 1 -- 172.21.15.83:0/1378805227 >> v1:172.21.15.83:6789/0 conn(0x7f65280c6120 legacy=0x7f65281575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.841+0000 7f651ffff640 1 -- 172.21.15.83:0/1378805227 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65281567c0 con 0x7f6528155d70 2024-09-19T07:24:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.845+0000 7f651ffff640 1 --2- 172.21.15.83:0/1378805227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6528155d70 0x7f65280c6500 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f65280b1990 tx=0x7f651001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8934e46bba2f61db server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.845+0000 7f6526ffd640 1 -- 172.21.15.83:0/1378805227 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65100b0070 con 0x7f6528155d70 2024-09-19T07:24:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.845+0000 7f6526ffd640 1 -- 172.21.15.83:0/1378805227 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65100ab600 con 0x7f6528155d70 2024-09-19T07:24:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.845+0000 7f6526ffd640 1 -- 172.21.15.83:0/1378805227 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65100ab9d0 con 0x7f6528155d70 2024-09-19T07:24:58.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.845+0000 7f652e128640 1 -- 172.21.15.83:0/1378805227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6528155d70 msgr2=0x7f65280c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:58.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.845+0000 7f652e128640 1 --2- 172.21.15.83:0/1378805227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6528155d70 0x7f65280c6500 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f65280b1990 tx=0x7f651001c480 comp rx=0 tx=0).stop 2024-09-19T07:24:58.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.845+0000 7f652e128640 1 -- 172.21.15.83:0/1378805227 shutdown_connections 2024-09-19T07:24:58.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.845+0000 7f652e128640 1 --2- 172.21.15.83:0/1378805227 >> v2:172.21.15.184:3300/0 conn(0x7f65281592f0 0x7f6528157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:58.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.845+0000 7f652e128640 1 --2- 172.21.15.83:0/1378805227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6528155d70 0x7f65280c6500 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:58.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.845+0000 7f652e128640 1 -- 172.21.15.83:0/1378805227 >> 172.21.15.83:0/1378805227 conn(0x7f652806bc80 msgr2=0x7f652806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:58.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.845+0000 7f652e128640 1 -- 172.21.15.83:0/1378805227 shutdown_connections 2024-09-19T07:24:58.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.845+0000 7f652e128640 1 -- 172.21.15.83:0/1378805227 wait complete. 2024-09-19T07:24:58.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f652e128640 1 Processor -- start 2024-09-19T07:24:58.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f652e128640 1 -- start start 2024-09-19T07:24:58.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f652e128640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280c6120 0x7f65280c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:58.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f652e128640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65281592f0 0x7f65281f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:58.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f652e128640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6528147540 0x7f652814a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f652e128640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6528153860 con 0x7f6528147540 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f652e128640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6528162ea0 con 0x7f65280c6120 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f652e128640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f65281f2a50 con 0x7f65281592f0 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f6527fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6528147540 0x7f652814a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f6527fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6528147540 0x7f652814a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47172/0 (socket says 172.21.15.83:47172) 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f6527fff640 1 -- 172.21.15.83:0/601983238 learned_addr learned my addr 172.21.15.83:0/601983238 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f651ffff640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65281592f0 0x7f65281f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f65277fe640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280c6120 0x7f65280c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f6527fff640 1 -- 172.21.15.83:0/601983238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65281592f0 msgr2=0x7f65281f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f6527fff640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65281592f0 0x7f65281f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f6527fff640 1 -- 172.21.15.83:0/601983238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280c6120 msgr2=0x7f65280c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f6527fff640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280c6120 0x7f65280c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:58.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f6527fff640 1 -- 172.21.15.83:0/601983238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65280c2260 con 0x7f6528147540 2024-09-19T07:24:58.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f6527fff640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6528147540 0x7f652814a490 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f6520007e60 tx=0x7f652000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:58.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f6524ff9640 1 -- 172.21.15.83:0/601983238 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520014420 con 0x7f6528147540 2024-09-19T07:24:58.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f6524ff9640 1 -- 172.21.15.83:0/601983238 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6520019070 con 0x7f6528147540 2024-09-19T07:24:58.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f652e128640 1 -- 172.21.15.83:0/601983238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65281ee8a0 con 0x7f6528147540 2024-09-19T07:24:58.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f6524ff9640 1 -- 172.21.15.83:0/601983238 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520014820 con 0x7f6528147540 2024-09-19T07:24:58.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f652e128640 1 -- 172.21.15.83:0/601983238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65281eee60 con 0x7f6528147540 2024-09-19T07:24:58.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.849+0000 7f652e128640 1 -- 172.21.15.83:0/601983238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65280c6500 con 0x7f6528147540 2024-09-19T07:24:58.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.853+0000 7f6524ff9640 1 -- 172.21.15.83:0/601983238 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f65200149c0 con 0x7f6528147540 2024-09-19T07:24:58.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.853+0000 7f6524ff9640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65080678f0 0x7f6508069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:24:58.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.853+0000 7f6524ff9640 1 -- 172.21.15.83:0/601983238 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7f65200869e0 con 0x7f6528147540 2024-09-19T07:24:58.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.853+0000 7f65277fe640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65080678f0 0x7f6508069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:24:58.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.853+0000 7f65277fe640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65080678f0 0x7f6508069db0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6518000f70 tx=0x7f6518007b60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:24:58.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:58.857+0000 7f6524ff9640 1 -- 172.21.15.83:0/601983238 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6520056f60 con 0x7f6528147540 2024-09-19T07:24:59.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 -- 172.21.15.83:0/601983238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65280c5ad0 con 0x7f6528147540 2024-09-19T07:24:59.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f6524ff9640 1 -- 172.21.15.83:0/601983238 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7f6520056860 con 0x7f6528147540 2024-09-19T07:24:59.077 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:24:59.077 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:24:59.077 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:24:59.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 -- 172.21.15.83:0/601983238 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65080678f0 msgr2=0x7f6508069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:59.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65080678f0 0x7f6508069db0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6518000f70 tx=0x7f6518007b60 comp rx=0 tx=0).stop 2024-09-19T07:24:59.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 -- 172.21.15.83:0/601983238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6528147540 msgr2=0x7f652814a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:24:59.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6528147540 0x7f652814a490 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f6520007e60 tx=0x7f652000e4b0 comp rx=0 tx=0).stop 2024-09-19T07:24:59.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 -- 172.21.15.83:0/601983238 shutdown_connections 2024-09-19T07:24:59.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65080678f0 0x7f6508069db0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:59.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6528147540 0x7f652814a490 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:59.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65281592f0 0x7f65281f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:59.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 --2- 172.21.15.83:0/601983238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280c6120 0x7f65280c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:24:59.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 -- 172.21.15.83:0/601983238 >> 172.21.15.83:0/601983238 conn(0x7f652806bc80 msgr2=0x7f6528153b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:24:59.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 -- 172.21.15.83:0/601983238 shutdown_connections 2024-09-19T07:24:59.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:24:59.077+0000 7f652e128640 1 -- 172.21.15.83:0/601983238 wait complete. 2024-09-19T07:24:59.096 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-19T07:24:59.097 INFO:tasks.mds_thrash.fs.[b]:waiting for 21 secs before thrashing 2024-09-19T07:25:01.509 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:25:01.511 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:25:10.166 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:10.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.305+0000 7f8f62c71640 1 Processor -- start 2024-09-19T07:25:10.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.305+0000 7f8f62c71640 1 -- start start 2024-09-19T07:25:10.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.305+0000 7f8f62c71640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8f5c1535d0 0x7f8f5c0c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.305+0000 7f8f62c71640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f5c0c26c0 con 0x7f8f5c159af0 2024-09-19T07:25:10.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.305+0000 7f8f62c71640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f5c0c29a0 con 0x7f8f5c06ffe0 2024-09-19T07:25:10.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.305+0000 7f8f62c71640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f8f5c0707e0 con 0x7f8f5c1535d0 2024-09-19T07:25:10.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.305+0000 7f8f60a0d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8f5c1535d0 0x7f8f5c0c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.305+0000 7f8f6120e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8f5c159af0 0x7f8f5c0c2f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41936/0 (socket says 172.21.15.83:41936) 2024-09-19T07:25:10.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.305+0000 7f8f6120e640 1 -- 172.21.15.83:0/1543719468 learned_addr learned my addr 172.21.15.83:0/1543719468 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:10.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.305+0000 7f8f60a0d640 1 -- 172.21.15.83:0/1543719468 >> v1:172.21.15.184:6789/0 conn(0x7f8f5c06ffe0 legacy=0x7f8f5c155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:10.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.305+0000 7f8f60a0d640 1 -- 172.21.15.83:0/1543719468 >> v1:172.21.15.83:6789/0 conn(0x7f8f5c159af0 legacy=0x7f8f5c0c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:10.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.305+0000 7f8f60a0d640 1 -- 172.21.15.83:0/1543719468 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5c1592f0 con 0x7f8f5c1535d0 2024-09-19T07:25:10.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.309+0000 7f8f60a0d640 1 --2- 172.21.15.83:0/1543719468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f5c1535d0 0x7f8f5c0c6460 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8f500097c0 tx=0x7f8f5001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff56b997303688d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:10.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.309+0000 7f8f5b7fe640 1 -- 172.21.15.83:0/1543719468 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f500b0070 con 0x7f8f5c1535d0 2024-09-19T07:25:10.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.309+0000 7f8f5b7fe640 1 -- 172.21.15.83:0/1543719468 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f500ab600 con 0x7f8f5c1535d0 2024-09-19T07:25:10.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.309+0000 7f8f5b7fe640 1 -- 172.21.15.83:0/1543719468 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f500ab9d0 con 0x7f8f5c1535d0 2024-09-19T07:25:10.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.309+0000 7f8f62c71640 1 -- 172.21.15.83:0/1543719468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f5c1535d0 msgr2=0x7f8f5c0c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:10.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.309+0000 7f8f62c71640 1 --2- 172.21.15.83:0/1543719468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f5c1535d0 0x7f8f5c0c6460 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8f500097c0 tx=0x7f8f5001c380 comp rx=0 tx=0).stop 2024-09-19T07:25:10.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.309+0000 7f8f62c71640 1 -- 172.21.15.83:0/1543719468 shutdown_connections 2024-09-19T07:25:10.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.309+0000 7f8f62c71640 1 --2- 172.21.15.83:0/1543719468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f5c1535d0 0x7f8f5c0c6460 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:10.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.309+0000 7f8f62c71640 1 -- 172.21.15.83:0/1543719468 >> 172.21.15.83:0/1543719468 conn(0x7f8f5c0c5060 msgr2=0x7f8f5c0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:10.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.309+0000 7f8f62c71640 1 -- 172.21.15.83:0/1543719468 shutdown_connections 2024-09-19T07:25:10.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.309+0000 7f8f62c71640 1 -- 172.21.15.83:0/1543719468 wait complete. 2024-09-19T07:25:10.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f62c71640 1 Processor -- start 2024-09-19T07:25:10.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f62c71640 1 -- start start 2024-09-19T07:25:10.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f62c71640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f5c06ffe0 0x7f8f5c15b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f62c71640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f5c1535d0 0x7f8f5c15c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f62c71640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f5c159af0 0x7f8f5c154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f62c71640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c163d20 con 0x7f8f5c159af0 2024-09-19T07:25:10.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f62c71640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c0c2c80 con 0x7f8f5c06ffe0 2024-09-19T07:25:10.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f62c71640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8f5c163ea0 con 0x7f8f5c1535d0 2024-09-19T07:25:10.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f6120e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f5c159af0 0x7f8f5c154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f6120e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f5c159af0 0x7f8f5c154d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34706/0 (socket says 172.21.15.83:34706) 2024-09-19T07:25:10.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f6120e640 1 -- 172.21.15.83:0/469995426 learned_addr learned my addr 172.21.15.83:0/469995426 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:10.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f60a0d640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f5c06ffe0 0x7f8f5c15b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f5bfff640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f5c1535d0 0x7f8f5c15c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f6120e640 1 -- 172.21.15.83:0/469995426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f5c1535d0 msgr2=0x7f8f5c15c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:10.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f6120e640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f5c1535d0 0x7f8f5c15c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f6120e640 1 -- 172.21.15.83:0/469995426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f5c06ffe0 msgr2=0x7f8f5c15b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f6120e640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f5c06ffe0 0x7f8f5c15b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f6120e640 1 -- 172.21.15.83:0/469995426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5c15b570 con 0x7f8f5c159af0 2024-09-19T07:25:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f6120e640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f5c159af0 0x7f8f5c154d70 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400cbf0 tx=0x7f8f5400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f597fa640 1 -- 172.21.15.83:0/469995426 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f54017070 con 0x7f8f5c159af0 2024-09-19T07:25:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f597fa640 1 -- 172.21.15.83:0/469995426 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f5400ad40 con 0x7f8f5c159af0 2024-09-19T07:25:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f62c71640 1 -- 172.21.15.83:0/469995426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f5c15c630 con 0x7f8f5c159af0 2024-09-19T07:25:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f597fa640 1 -- 172.21.15.83:0/469995426 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f54012600 con 0x7f8f5c159af0 2024-09-19T07:25:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f62c71640 1 -- 172.21.15.83:0/469995426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f5c155db0 con 0x7f8f5c159af0 2024-09-19T07:25:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.313+0000 7f8f62c71640 1 -- 172.21.15.83:0/469995426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f5c0c6460 con 0x7f8f5c159af0 2024-09-19T07:25:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.317+0000 7f8f597fa640 1 -- 172.21.15.83:0/469995426 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8f5401a070 con 0x7f8f5c159af0 2024-09-19T07:25:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.317+0000 7f8f597fa640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8f400678f0 0x7f8f40069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.317+0000 7f8f597fa640 1 -- 172.21.15.83:0/469995426 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7f8f540862f0 con 0x7f8f5c159af0 2024-09-19T07:25:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.317+0000 7f8f60a0d640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8f400678f0 0x7f8f40069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.317+0000 7f8f60a0d640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8f400678f0 0x7f8f40069db0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8f500ad000 tx=0x7f8f50008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.321+0000 7f8f597fa640 1 -- 172.21.15.83:0/469995426 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f8f54056300 con 0x7f8f5c159af0 2024-09-19T07:25:10.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.569+0000 7f8f62c71640 1 -- 172.21.15.83:0/469995426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f5c06fd70 con 0x7f8f5c159af0 2024-09-19T07:25:10.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.569+0000 7f8f597fa640 1 -- 172.21.15.83:0/469995426 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7f8f5405a600 con 0x7f8f5c159af0 2024-09-19T07:25:10.572 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:10.572 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:10.572 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:10.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.573+0000 7f8f62c71640 1 -- 172.21.15.83:0/469995426 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8f400678f0 msgr2=0x7f8f40069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:10.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.573+0000 7f8f62c71640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8f400678f0 0x7f8f40069db0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8f500ad000 tx=0x7f8f50008040 comp rx=0 tx=0).stop 2024-09-19T07:25:10.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.573+0000 7f8f62c71640 1 -- 172.21.15.83:0/469995426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f5c159af0 msgr2=0x7f8f5c154d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:10.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.573+0000 7f8f62c71640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f5c159af0 0x7f8f5c154d70 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400cbf0 tx=0x7f8f5400a430 comp rx=0 tx=0).stop 2024-09-19T07:25:10.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.573+0000 7f8f62c71640 1 -- 172.21.15.83:0/469995426 shutdown_connections 2024-09-19T07:25:10.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.573+0000 7f8f62c71640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8f400678f0 0x7f8f40069db0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:10.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.573+0000 7f8f62c71640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f5c159af0 0x7f8f5c154d70 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:10.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.573+0000 7f8f62c71640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f5c1535d0 0x7f8f5c15c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:10.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.573+0000 7f8f62c71640 1 --2- 172.21.15.83:0/469995426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f5c06ffe0 0x7f8f5c15b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:10.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.573+0000 7f8f62c71640 1 -- 172.21.15.83:0/469995426 >> 172.21.15.83:0/469995426 conn(0x7f8f5c0c5060 msgr2=0x7f8f5c1586a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:10.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.573+0000 7f8f62c71640 1 -- 172.21.15.83:0/469995426 shutdown_connections 2024-09-19T07:25:10.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.573+0000 7f8f62c71640 1 -- 172.21.15.83:0/469995426 wait complete. 2024-09-19T07:25:10.593 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-19T07:25:10.593 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:25:10.593 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:25:10.605 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:25:10.605 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:10.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.777+0000 7fbfabe05640 1 Processor -- start 2024-09-19T07:25:10.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.777+0000 7fbfabe05640 1 -- start start 2024-09-19T07:25:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfabe05640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa40c2c30 0x7fbfa4152e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfabe05640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa4159c60 con 0x7fbfa40aaa60 2024-09-19T07:25:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfabe05640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa40c4b90 con 0x7fbfa406f070 2024-09-19T07:25:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfabe05640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa415a160 con 0x7fbfa40c2c30 2024-09-19T07:25:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa93a0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbfa40aaa60 0x7fbfa41533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41950/0 (socket says 172.21.15.83:41950) 2024-09-19T07:25:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa93a0640 1 -- 172.21.15.83:0/3168299045 learned_addr learned my addr 172.21.15.83:0/3168299045 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa9ba1640 1 --2- 172.21.15.83:0/3168299045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa40c2c30 0x7fbfa4152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1608682848 0 0) 0x7fbfa4159c60 con 0x7fbfa40aaa60 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf90003530 con 0x7fbfa40aaa60 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3034277447 0 0) 0x7fbf90003530 con 0x7fbfa40aaa60 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa40c2c30 msgr2=0x7fbfa4152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 --2- 172.21.15.83:0/3168299045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa40c2c30 0x7fbfa4152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 >> v1:172.21.15.184:6789/0 conn(0x7fbfa406f070 legacy=0x7fbfa4155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa40aa4a0 con 0x7fbfa40aaa60 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fbf94001680 con 0x7fbfa40aaa60 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 >> v1:172.21.15.83:6789/0 conn(0x7fbfa40aaa60 legacy=0x7fbfa41533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 --2- 172.21.15.83:0/3168299045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa40c2c30 0x7fbf900039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 --2- 172.21.15.83:0/3168299045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf90003f70 0x7fbf90004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 --2- 172.21.15.83:0/3168299045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf90004d50 0x7fbf900051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa415a160 con 0x7fbf90004d50 2024-09-19T07:25:10.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9000bb60 con 0x7fbf90003f70 2024-09-19T07:25:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbf9000bd60 con 0x7fbfa40c2c30 2024-09-19T07:25:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfaa3a2640 1 --2- 172.21.15.83:0/3168299045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf90004d50 0x7fbf900051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa9ba1640 1 --2- 172.21.15.83:0/3168299045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa40c2c30 0x7fbf900039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfa93a0640 1 --2- 172.21.15.83:0/3168299045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf90003f70 0x7fbf90004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfaa3a2640 1 -- 172.21.15.83:0/3168299045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa40c2c30 msgr2=0x7fbf900039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfaa3a2640 1 --2- 172.21.15.83:0/3168299045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa40c2c30 0x7fbf900039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfaa3a2640 1 -- 172.21.15.83:0/3168299045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf90003f70 msgr2=0x7fbf90004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfaa3a2640 1 --2- 172.21.15.83:0/3168299045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf90003f70 0x7fbf90004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.781+0000 7fbfaa3a2640 1 -- 172.21.15.83:0/3168299045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf90003530 con 0x7fbf90004d50 2024-09-19T07:25:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfaa3a2640 1 --2- 172.21.15.83:0/3168299045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf90004d50 0x7fbf900051d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c001e40 tx=0x7fbf9c01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20fb28690e480e7b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c001a90 con 0x7fbf90004d50 2024-09-19T07:25:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf9c003b80 con 0x7fbf90004d50 2024-09-19T07:25:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfa8b9f640 1 -- 172.21.15.83:0/3168299045 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c005080 con 0x7fbf90004d50 2024-09-19T07:25:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfabe05640 1 -- 172.21.15.83:0/3168299045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf90004d50 msgr2=0x7fbf900051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:10.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfabe05640 1 --2- 172.21.15.83:0/3168299045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf90004d50 0x7fbf900051d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c001e40 tx=0x7fbf9c01c760 comp rx=0 tx=0).stop 2024-09-19T07:25:10.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfaa3a2640 1 -- 172.21.15.83:0/3168299045 reap_dead start 2024-09-19T07:25:10.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfabe05640 1 -- 172.21.15.83:0/3168299045 shutdown_connections 2024-09-19T07:25:10.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfabe05640 1 -- 172.21.15.83:0/3168299045 >> 172.21.15.83:0/3168299045 conn(0x7fbfa40c95a0 msgr2=0x7fbfa40c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:10.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfabe05640 1 -- 172.21.15.83:0/3168299045 shutdown_connections 2024-09-19T07:25:10.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfabe05640 1 -- 172.21.15.83:0/3168299045 wait complete. 2024-09-19T07:25:10.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfabe05640 1 Processor -- start 2024-09-19T07:25:10.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfabe05640 1 -- start start 2024-09-19T07:25:10.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.785+0000 7fbfabe05640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfa40aba60 0x7fbfa41dfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfabe05640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfa40abe40 0x7fbfa4158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfabe05640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa41ee750 0x7fbfa41489c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfabe05640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa40c3a90 con 0x7fbfa40abe40 2024-09-19T07:25:10.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfabe05640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa40c4940 con 0x7fbfa40aba60 2024-09-19T07:25:10.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfabe05640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa40c3c10 con 0x7fbfa41ee750 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfa93a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfa40abe40 0x7fbfa4158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfa93a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfa40abe40 0x7fbfa4158920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34726/0 (socket says 172.21.15.83:34726) 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfa93a0640 1 -- 172.21.15.83:0/849936525 learned_addr learned my addr 172.21.15.83:0/849936525 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfa9ba1640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfa40aba60 0x7fbfa41dfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfaa3a2640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa41ee750 0x7fbfa41489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfa93a0640 1 -- 172.21.15.83:0/849936525 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa41ee750 msgr2=0x7fbfa41489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfa93a0640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa41ee750 0x7fbfa41489c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfa93a0640 1 -- 172.21.15.83:0/849936525 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfa40aba60 msgr2=0x7fbfa41dfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfa93a0640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfa40aba60 0x7fbfa41dfcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfa93a0640 1 -- 172.21.15.83:0/849936525 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa41579b0 con 0x7fbfa40abe40 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfa93a0640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfa40abe40 0x7fbfa4158920 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbf94011980 tx=0x7fbf94011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbf9a7fc640 1 -- 172.21.15.83:0/849936525 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94010040 con 0x7fbfa40abe40 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbf9a7fc640 1 -- 172.21.15.83:0/849936525 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf9401a070 con 0x7fbfa40abe40 2024-09-19T07:25:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfabe05640 1 -- 172.21.15.83:0/849936525 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa4157c90 con 0x7fbfa40abe40 2024-09-19T07:25:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbf9a7fc640 1 -- 172.21.15.83:0/849936525 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94015410 con 0x7fbfa40abe40 2024-09-19T07:25:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfabe05640 1 -- 172.21.15.83:0/849936525 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa41df4b0 con 0x7fbfa40abe40 2024-09-19T07:25:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfabe05640 1 -- 172.21.15.83:0/849936525 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa41df260 con 0x7fbfa40abe40 2024-09-19T07:25:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbf9a7fc640 1 -- 172.21.15.83:0/849936525 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbf940155b0 con 0x7fbfa40abe40 2024-09-19T07:25:10.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbf9a7fc640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf880678c0 0x7fbf88069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:10.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbf9a7fc640 1 -- 172.21.15.83:0/849936525 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7fbf940895e0 con 0x7fbfa40abe40 2024-09-19T07:25:10.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfa9ba1640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf880678c0 0x7fbf88069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:10.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.789+0000 7fbfa9ba1640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf880678c0 0x7fbf88069d80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009b20 tx=0x7fbfa0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:10.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:10.793+0000 7fbf9a7fc640 1 -- 172.21.15.83:0/849936525 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbf94059b60 con 0x7fbfa40abe40 2024-09-19T07:25:11.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 -- 172.21.15.83:0/849936525 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfa4158e60 con 0x7fbfa40abe40 2024-09-19T07:25:11.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbf9a7fc640 1 -- 172.21.15.83:0/849936525 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7fbf94059460 con 0x7fbfa40abe40 2024-09-19T07:25:11.009 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:11.009 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:11.009 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:11.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 -- 172.21.15.83:0/849936525 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf880678c0 msgr2=0x7fbf88069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:11.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf880678c0 0x7fbf88069d80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009b20 tx=0x7fbfa0008040 comp rx=0 tx=0).stop 2024-09-19T07:25:11.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 -- 172.21.15.83:0/849936525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfa40abe40 msgr2=0x7fbfa4158920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:11.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfa40abe40 0x7fbfa4158920 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbf94011980 tx=0x7fbf94011e40 comp rx=0 tx=0).stop 2024-09-19T07:25:11.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 -- 172.21.15.83:0/849936525 shutdown_connections 2024-09-19T07:25:11.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf880678c0 0x7fbf88069d80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:11.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa41ee750 0x7fbfa41489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:11.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfa40abe40 0x7fbfa4158920 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:11.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 --2- 172.21.15.83:0/849936525 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfa40aba60 0x7fbfa41dfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:11.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 -- 172.21.15.83:0/849936525 >> 172.21.15.83:0/849936525 conn(0x7fbfa40c95a0 msgr2=0x7fbfa40adc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:11.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 -- 172.21.15.83:0/849936525 shutdown_connections 2024-09-19T07:25:11.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:11.009+0000 7fbfabe05640 1 -- 172.21.15.83:0/849936525 wait complete. 2024-09-19T07:25:11.030 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:25:13.032 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:13.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.161+0000 7f6c03037640 1 Processor -- start 2024-09-19T07:25:13.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.161+0000 7f6c03037640 1 -- start start 2024-09-19T07:25:13.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.161+0000 7f6c03037640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bfc150290 0x7f6bfc0c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:13.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.161+0000 7f6c03037640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6bfc152b20 0x7f6bfc0c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:13.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.161+0000 7f6c03037640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6bfc0c45c0 0x7f6bfc153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:13.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.161+0000 7f6c03037640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6bfc0716c0 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c03037640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6bfc0c3920 con 0x7f6bfc152b20 2024-09-19T07:25:13.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c03037640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6bfc0c3490 con 0x7f6bfc150290 2024-09-19T07:25:13.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c015d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6bfc0c45c0 0x7f6bfc153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:13.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c015d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6bfc0c45c0 0x7f6bfc153eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34736/0 (socket says 172.21.15.83:34736) 2024-09-19T07:25:13.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c015d4640 1 -- 172.21.15.83:0/3641338434 learned_addr learned my addr 172.21.15.83:0/3641338434 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:13.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c00dd3640 1 --2- 172.21.15.83:0/3641338434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bfc150290 0x7f6bfc0c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:13.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6bfbfff640 1 --2- 172.21.15.83:0/3641338434 >> v2:172.21.15.184:3300/0 conn(0x7f6bfc152b20 0x7f6bfc0c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:13.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c015d4640 1 -- 172.21.15.83:0/3641338434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bfc150290 msgr2=0x7f6bfc0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:13.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c015d4640 1 --2- 172.21.15.83:0/3641338434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bfc150290 0x7f6bfc0c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:13.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c015d4640 1 -- 172.21.15.83:0/3641338434 >> v2:172.21.15.184:3300/0 conn(0x7f6bfc152b20 msgr2=0x7f6bfc0c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:13.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c015d4640 1 --2- 172.21.15.83:0/3641338434 >> v2:172.21.15.184:3300/0 conn(0x7f6bfc152b20 0x7f6bfc0c4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:13.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c015d4640 1 -- 172.21.15.83:0/3641338434 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc150670 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c00dd3640 1 --2- 172.21.15.83:0/3641338434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bfc150290 0x7f6bfc0c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:13.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c015d4640 1 --2- 172.21.15.83:0/3641338434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bfc0c45c0 0x7f6bfc153eb0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc06cae0 tx=0x7f6bec01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ad1396cd0f5970 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:13.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6bfb7fe640 1 -- 172.21.15.83:0/3641338434 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec0a5840 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6bfb7fe640 1 -- 172.21.15.83:0/3641338434 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bec0a59e0 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6bfb7fe640 1 -- 172.21.15.83:0/3641338434 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec0a5db0 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c03037640 1 -- 172.21.15.83:0/3641338434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bfc0c45c0 msgr2=0x7f6bfc153eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:13.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c03037640 1 --2- 172.21.15.83:0/3641338434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bfc0c45c0 0x7f6bfc153eb0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc06cae0 tx=0x7f6bec01c9c0 comp rx=0 tx=0).stop 2024-09-19T07:25:13.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c03037640 1 -- 172.21.15.83:0/3641338434 shutdown_connections 2024-09-19T07:25:13.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c03037640 1 --2- 172.21.15.83:0/3641338434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bfc0c45c0 0x7f6bfc153eb0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:13.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c03037640 1 --2- 172.21.15.83:0/3641338434 >> v2:172.21.15.184:3300/0 conn(0x7f6bfc152b20 0x7f6bfc0c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:13.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c03037640 1 --2- 172.21.15.83:0/3641338434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bfc150290 0x7f6bfc0c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:13.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c03037640 1 -- 172.21.15.83:0/3641338434 >> 172.21.15.83:0/3641338434 conn(0x7f6bfc06bc80 msgr2=0x7f6bfc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:13.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c03037640 1 -- 172.21.15.83:0/3641338434 shutdown_connections 2024-09-19T07:25:13.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.165+0000 7f6c03037640 1 -- 172.21.15.83:0/3641338434 wait complete. 2024-09-19T07:25:13.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c03037640 1 Processor -- start 2024-09-19T07:25:13.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c03037640 1 -- start start 2024-09-19T07:25:13.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c03037640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bfc0c45c0 0x7f6bfc0735b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:13.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c03037640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bfc150290 0x7f6bfc0c0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:13.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c03037640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bfc152b20 0x7f6bfc1ee250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:13.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c03037640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc0c2530 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c03037640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc0c3120 con 0x7f6bfc150290 2024-09-19T07:25:13.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c03037640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6bfc154800 con 0x7f6bfc152b20 2024-09-19T07:25:13.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c00dd3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bfc0c45c0 0x7f6bfc0735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:13.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c00dd3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bfc0c45c0 0x7f6bfc0735b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34750/0 (socket says 172.21.15.83:34750) 2024-09-19T07:25:13.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c00dd3640 1 -- 172.21.15.83:0/1334245445 learned_addr learned my addr 172.21.15.83:0/1334245445 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:13.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6bfbfff640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bfc150290 0x7f6bfc0c0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:13.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c015d4640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bfc152b20 0x7f6bfc1ee250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:13.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c00dd3640 1 -- 172.21.15.83:0/1334245445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bfc152b20 msgr2=0x7f6bfc1ee250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:13.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c00dd3640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bfc152b20 0x7f6bfc1ee250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:13.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c00dd3640 1 -- 172.21.15.83:0/1334245445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bfc150290 msgr2=0x7f6bfc0c0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:13.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c00dd3640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bfc150290 0x7f6bfc0c0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:13.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c00dd3640 1 -- 172.21.15.83:0/1334245445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc1f20d0 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c00dd3640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bfc0c45c0 0x7f6bfc0735b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400c7c0 tx=0x7f6bf400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:13.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6bf97fa640 1 -- 172.21.15.83:0/1334245445 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4015070 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6bf97fa640 1 -- 172.21.15.83:0/1334245445 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bf4007a30 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c03037640 1 -- 172.21.15.83:0/1334245445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bfc1f23b0 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6bf97fa640 1 -- 172.21.15.83:0/1334245445 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4007e00 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c03037640 1 -- 172.21.15.83:0/1334245445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bfc151be0 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.169+0000 7f6c03037640 1 -- 172.21.15.83:0/1334245445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bfc0c1ff0 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.173+0000 7f6bf97fa640 1 -- 172.21.15.83:0/1334245445 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6bf4018070 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.173+0000 7f6bf97fa640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6bdc0678c0 0x7f6bdc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:13.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.173+0000 7f6bf97fa640 1 -- 172.21.15.83:0/1334245445 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7f6bf4086240 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.173+0000 7f6bfbfff640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6bdc0678c0 0x7f6bdc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:13.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.173+0000 7f6bfbfff640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6bdc0678c0 0x7f6bdc069d80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc074040 tx=0x7f6be4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:13.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.177+0000 7f6bf97fa640 1 -- 172.21.15.83:0/1334245445 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6bf4056730 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.373+0000 7f6c03037640 1 -- 172.21.15.83:0/1334245445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6bfc0acab0 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.373+0000 7f6bf97fa640 1 -- 172.21.15.83:0/1334245445 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7f6bf4056030 con 0x7f6bfc0c45c0 2024-09-19T07:25:13.376 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:13.376 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:13.376 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:13.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.377+0000 7f6c03037640 1 -- 172.21.15.83:0/1334245445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6bdc0678c0 msgr2=0x7f6bdc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.377+0000 7f6c03037640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6bdc0678c0 0x7f6bdc069d80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc074040 tx=0x7f6be4008040 comp rx=0 tx=0).stop 2024-09-19T07:25:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.377+0000 7f6c03037640 1 -- 172.21.15.83:0/1334245445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bfc0c45c0 msgr2=0x7f6bfc0735b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.377+0000 7f6c03037640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bfc0c45c0 0x7f6bfc0735b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400c7c0 tx=0x7f6bf400cc80 comp rx=0 tx=0).stop 2024-09-19T07:25:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.377+0000 7f6c03037640 1 -- 172.21.15.83:0/1334245445 shutdown_connections 2024-09-19T07:25:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.377+0000 7f6c03037640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6bdc0678c0 0x7f6bdc069d80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.377+0000 7f6c03037640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6bfc152b20 0x7f6bfc1ee250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.377+0000 7f6c03037640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6bfc150290 0x7f6bfc0c0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.377+0000 7f6c03037640 1 --2- 172.21.15.83:0/1334245445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bfc0c45c0 0x7f6bfc0735b0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.377+0000 7f6c03037640 1 -- 172.21.15.83:0/1334245445 >> 172.21.15.83:0/1334245445 conn(0x7f6bfc06bc80 msgr2=0x7f6bfc153ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.377+0000 7f6c03037640 1 -- 172.21.15.83:0/1334245445 shutdown_connections 2024-09-19T07:25:13.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:13.377+0000 7f6c03037640 1 -- 172.21.15.83:0/1334245445 wait complete. 2024-09-19T07:25:13.393 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:25:15.394 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:15.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49e60c6640 1 Processor -- start 2024-09-19T07:25:15.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49e60c6640 1 -- start start 2024-09-19T07:25:15.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49e60c6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49e0071100 0x7f49e01545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:15.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49e60c6640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f49e00c28e0 0x7f49e0150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:15.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49e60c6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f49e0151480 0x7f49e006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:15.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49e60c6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f49e0071b50 con 0x7f49e0151480 2024-09-19T07:25:15.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49e60c6640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f49e00c1ff0 con 0x7f49e00c28e0 2024-09-19T07:25:15.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49e60c6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f49e00a9e60 con 0x7f49e0071100 2024-09-19T07:25:15.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49dffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f49e0151480 0x7f49e006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:15.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49dffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f49e0151480 0x7f49e006dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34762/0 (socket says 172.21.15.83:34762) 2024-09-19T07:25:15.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49dffff640 1 -- 172.21.15.83:0/996086393 learned_addr learned my addr 172.21.15.83:0/996086393 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:15.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49df7fe640 1 --2- 172.21.15.83:0/996086393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49e0071100 0x7f49e01545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:15.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49d6ffd640 1 --2- 172.21.15.83:0/996086393 >> v2:172.21.15.184:3300/0 conn(0x7f49e00c28e0 0x7f49e0150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:15.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49dffff640 1 -- 172.21.15.83:0/996086393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49e0071100 msgr2=0x7f49e01545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:15.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49dffff640 1 --2- 172.21.15.83:0/996086393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49e0071100 0x7f49e01545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:15.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49dffff640 1 -- 172.21.15.83:0/996086393 >> v2:172.21.15.184:3300/0 conn(0x7f49e00c28e0 msgr2=0x7f49e0150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:15.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49dffff640 1 --2- 172.21.15.83:0/996086393 >> v2:172.21.15.184:3300/0 conn(0x7f49e00c28e0 0x7f49e0150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:15.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49dffff640 1 -- 172.21.15.83:0/996086393 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e00aa140 con 0x7f49e0151480 2024-09-19T07:25:15.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49dffff640 1 --2- 172.21.15.83:0/996086393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e0151480 0x7f49e006dea0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f49e006ca00 tx=0x7f49d801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e71d950ef285a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:15.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.529+0000 7f49deffd640 1 -- 172.21.15.83:0/996086393 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d80a55e0 con 0x7f49e0151480 2024-09-19T07:25:15.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49deffd640 1 -- 172.21.15.83:0/996086393 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d80a5780 con 0x7f49e0151480 2024-09-19T07:25:15.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49deffd640 1 -- 172.21.15.83:0/996086393 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d80a5b50 con 0x7f49e0151480 2024-09-19T07:25:15.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 -- 172.21.15.83:0/996086393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e0151480 msgr2=0x7f49e006dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:15.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 --2- 172.21.15.83:0/996086393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e0151480 0x7f49e006dea0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f49e006ca00 tx=0x7f49d801c960 comp rx=0 tx=0).stop 2024-09-19T07:25:15.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 -- 172.21.15.83:0/996086393 shutdown_connections 2024-09-19T07:25:15.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 --2- 172.21.15.83:0/996086393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e0151480 0x7f49e006dea0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:15.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 --2- 172.21.15.83:0/996086393 >> v2:172.21.15.184:3300/0 conn(0x7f49e00c28e0 0x7f49e0150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:15.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 --2- 172.21.15.83:0/996086393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49e0071100 0x7f49e01545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:15.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 -- 172.21.15.83:0/996086393 >> 172.21.15.83:0/996086393 conn(0x7f49e006bc80 msgr2=0x7f49e006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:15.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 -- 172.21.15.83:0/996086393 shutdown_connections 2024-09-19T07:25:15.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 -- 172.21.15.83:0/996086393 wait complete. 2024-09-19T07:25:15.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 Processor -- start 2024-09-19T07:25:15.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 -- start start 2024-09-19T07:25:15.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49e0071100 0x7f49e00c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:15.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e00c28e0 0x7f49e014d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:15.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49e0151480 0x7f49e014dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:15.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f49e01570d0 con 0x7f49e00c28e0 2024-09-19T07:25:15.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49e0071730 con 0x7f49e0151480 2024-09-19T07:25:15.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f49e0157250 con 0x7f49e0071100 2024-09-19T07:25:15.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49d6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e00c28e0 0x7f49e014d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:15.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49d6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e00c28e0 0x7f49e014d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34776/0 (socket says 172.21.15.83:34776) 2024-09-19T07:25:15.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49d6ffd640 1 -- 172.21.15.83:0/3210036914 learned_addr learned my addr 172.21.15.83:0/3210036914 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:15.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49df7fe640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49e0071100 0x7f49e00c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:15.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49dffff640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49e0151480 0x7f49e014dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:15.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49d6ffd640 1 -- 172.21.15.83:0/3210036914 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49e0071100 msgr2=0x7f49e00c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:15.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49d6ffd640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49e0071100 0x7f49e00c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:15.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49d6ffd640 1 -- 172.21.15.83:0/3210036914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49e0151480 msgr2=0x7f49e014dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:15.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49d6ffd640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49e0151480 0x7f49e014dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:15.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49d6ffd640 1 -- 172.21.15.83:0/3210036914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e01f08e0 con 0x7f49e00c28e0 2024-09-19T07:25:15.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49dffff640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49e0151480 0x7f49e014dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:15.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49d6ffd640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e00c28e0 0x7f49e014d980 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f49c800d7d0 tx=0x7f49c800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:15.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49dcff9640 1 -- 172.21.15.83:0/3210036914 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c8004020 con 0x7f49e00c28e0 2024-09-19T07:25:15.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49dcff9640 1 -- 172.21.15.83:0/3210036914 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c80041c0 con 0x7f49e00c28e0 2024-09-19T07:25:15.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 -- 172.21.15.83:0/3210036914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49e014ba10 con 0x7f49e00c28e0 2024-09-19T07:25:15.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49dcff9640 1 -- 172.21.15.83:0/3210036914 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c80043b0 con 0x7f49e00c28e0 2024-09-19T07:25:15.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.533+0000 7f49e60c6640 1 -- 172.21.15.83:0/3210036914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49e014e400 con 0x7f49e00c28e0 2024-09-19T07:25:15.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.537+0000 7f49e60c6640 1 -- 172.21.15.83:0/3210036914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49e014b800 con 0x7f49e00c28e0 2024-09-19T07:25:15.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.537+0000 7f49dcff9640 1 -- 172.21.15.83:0/3210036914 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f49c801b020 con 0x7f49e00c28e0 2024-09-19T07:25:15.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.537+0000 7f49dcff9640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f49c00678c0 0x7f49c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:15.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.537+0000 7f49dcff9640 1 -- 172.21.15.83:0/3210036914 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7f49c8086090 con 0x7f49e00c28e0 2024-09-19T07:25:15.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.537+0000 7f49df7fe640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f49c00678c0 0x7f49c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:15.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.537+0000 7f49df7fe640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f49c00678c0 0x7f49c0069d80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f49d000b9d0 tx=0x7f49d0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:15.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.541+0000 7f49dcff9640 1 -- 172.21.15.83:0/3210036914 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f49c8056570 con 0x7f49e00c28e0 2024-09-19T07:25:15.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.737+0000 7f49e60c6640 1 -- 172.21.15.83:0/3210036914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49e0150f40 con 0x7f49e00c28e0 2024-09-19T07:25:15.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.737+0000 7f49dcff9640 1 -- 172.21.15.83:0/3210036914 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7f49c8055e70 con 0x7f49e00c28e0 2024-09-19T07:25:15.739 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:15.739 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:15.739 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:15.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.741+0000 7f49e60c6640 1 -- 172.21.15.83:0/3210036914 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f49c00678c0 msgr2=0x7f49c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:15.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.741+0000 7f49e60c6640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f49c00678c0 0x7f49c0069d80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f49d000b9d0 tx=0x7f49d0008040 comp rx=0 tx=0).stop 2024-09-19T07:25:15.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.741+0000 7f49e60c6640 1 -- 172.21.15.83:0/3210036914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e00c28e0 msgr2=0x7f49e014d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:15.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.741+0000 7f49e60c6640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e00c28e0 0x7f49e014d980 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f49c800d7d0 tx=0x7f49c800dc90 comp rx=0 tx=0).stop 2024-09-19T07:25:15.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.741+0000 7f49e60c6640 1 -- 172.21.15.83:0/3210036914 shutdown_connections 2024-09-19T07:25:15.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.741+0000 7f49e60c6640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f49c00678c0 0x7f49c0069d80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:15.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.741+0000 7f49e60c6640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49e0151480 0x7f49e014dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:15.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.741+0000 7f49e60c6640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e00c28e0 0x7f49e014d980 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:15.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.741+0000 7f49e60c6640 1 --2- 172.21.15.83:0/3210036914 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49e0071100 0x7f49e00c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:15.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.741+0000 7f49e60c6640 1 -- 172.21.15.83:0/3210036914 >> 172.21.15.83:0/3210036914 conn(0x7f49e006bc80 msgr2=0x7f49e00c3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:15.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.741+0000 7f49e60c6640 1 -- 172.21.15.83:0/3210036914 shutdown_connections 2024-09-19T07:25:15.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:15.741+0000 7f49e60c6640 1 -- 172.21.15.83:0/3210036914 wait complete. 2024-09-19T07:25:15.757 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:25:17.758 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:17.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.889+0000 7fd8756e5640 1 Processor -- start 2024-09-19T07:25:17.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.889+0000 7fd8756e5640 1 -- start start 2024-09-19T07:25:17.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd8756e5640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd870153eb0 0x7fd870154340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd8756e5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd870150670 con 0x7fd870152b20 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd8756e5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8700c2880 con 0x7fd870150290 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd8756e5640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd8700716c0 con 0x7fd870153eb0 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86e7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd870152b20 0x7fd8700c4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41952/0 (socket says 172.21.15.83:41952) 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86e7fc640 1 -- 172.21.15.83:0/1218942758 learned_addr learned my addr 172.21.15.83:0/1218942758 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86f7fe640 1 --2- 172.21.15.83:0/1218942758 >> v2:172.21.15.184:3300/0 conn(0x7fd870153eb0 0x7fd870154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1105148183 0 0) 0x7fd870150670 con 0x7fd870152b20 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd85c003530 con 0x7fd870152b20 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2550897110 0 0) 0x7fd85c003530 con 0x7fd870152b20 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 >> v2:172.21.15.184:3300/0 conn(0x7fd870153eb0 msgr2=0x7fd870154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 --2- 172.21.15.83:0/1218942758 >> v2:172.21.15.184:3300/0 conn(0x7fd870153eb0 0x7fd870154340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 >> v1:172.21.15.184:6789/0 conn(0x7fd870150290 legacy=0x7fd8700c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:17.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8700c2b60 con 0x7fd870152b20 2024-09-19T07:25:17.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fd860001680 con 0x7fd870152b20 2024-09-19T07:25:17.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 >> v1:172.21.15.83:6789/0 conn(0x7fd870152b20 legacy=0x7fd8700c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:17.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 --2- 172.21.15.83:0/1218942758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd85c0039c0 0x7fd85c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:17.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 --2- 172.21.15.83:0/1218942758 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd85c004810 0x7fd85c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:17.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 --2- 172.21.15.83:0/1218942758 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd85c0051d0 0x7fd85c00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:17.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd85c00e250 con 0x7fd85c0039c0 2024-09-19T07:25:17.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd85c00e400 con 0x7fd85c004810 2024-09-19T07:25:17.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd85c00e600 con 0x7fd85c0051d0 2024-09-19T07:25:17.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86effd640 1 --2- 172.21.15.83:0/1218942758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd85c0039c0 0x7fd85c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:17.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86e7fc640 1 --2- 172.21.15.83:0/1218942758 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd85c004810 0x7fd85c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:17.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86f7fe640 1 --2- 172.21.15.83:0/1218942758 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd85c0051d0 0x7fd85c00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:17.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86effd640 1 -- 172.21.15.83:0/1218942758 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd85c0051d0 msgr2=0x7fd85c00dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:17.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86effd640 1 --2- 172.21.15.83:0/1218942758 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd85c0051d0 0x7fd85c00dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:17.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86effd640 1 -- 172.21.15.83:0/1218942758 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd85c004810 msgr2=0x7fd85c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:17.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86effd640 1 --2- 172.21.15.83:0/1218942758 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd85c004810 0x7fd85c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:17.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86effd640 1 -- 172.21.15.83:0/1218942758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd85c003530 con 0x7fd85c0039c0 2024-09-19T07:25:17.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.893+0000 7fd86f7fe640 1 -- 172.21.15.83:0/1218942758 reap_dead start 2024-09-19T07:25:17.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86effd640 1 --2- 172.21.15.83:0/1218942758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd85c0039c0 0x7fd85c003de0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd858001780 tx=0x7fd85801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa8dc835da7c7fb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:17.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8580013a0 con 0x7fd85c0039c0 2024-09-19T07:25:17.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8580052f0 con 0x7fd85c0039c0 2024-09-19T07:25:17.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86dffb640 1 -- 172.21.15.83:0/1218942758 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd858003840 con 0x7fd85c0039c0 2024-09-19T07:25:17.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 -- 172.21.15.83:0/1218942758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd85c0039c0 msgr2=0x7fd85c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:17.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 --2- 172.21.15.83:0/1218942758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd85c0039c0 0x7fd85c003de0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd858001780 tx=0x7fd85801c400 comp rx=0 tx=0).stop 2024-09-19T07:25:17.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 -- 172.21.15.83:0/1218942758 shutdown_connections 2024-09-19T07:25:17.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 --2- 172.21.15.83:0/1218942758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd85c0039c0 0x7fd85c003de0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:17.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 -- 172.21.15.83:0/1218942758 >> 172.21.15.83:0/1218942758 conn(0x7fd87006bc80 msgr2=0x7fd87006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:17.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 -- 172.21.15.83:0/1218942758 shutdown_connections 2024-09-19T07:25:17.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 -- 172.21.15.83:0/1218942758 wait complete. 2024-09-19T07:25:17.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 Processor -- start 2024-09-19T07:25:17.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 -- start start 2024-09-19T07:25:17.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd85c0039c0 0x7fd8700c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:17.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd870071ed0 0x7fd870143c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:17.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd870144180 0x7fd87014e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:17.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd8700c4080 con 0x7fd870071ed0 2024-09-19T07:25:17.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd8700c2610 con 0x7fd870144180 2024-09-19T07:25:17.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd8756e5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd8700c4200 con 0x7fd85c0039c0 2024-09-19T07:25:17.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd870071ed0 0x7fd870143c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:17.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd870071ed0 0x7fd870143c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34788/0 (socket says 172.21.15.83:34788) 2024-09-19T07:25:17.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86e7fc640 1 -- 172.21.15.83:0/3321356008 learned_addr learned my addr 172.21.15.83:0/3321356008 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:17.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86effd640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd85c0039c0 0x7fd8700c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:17.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86f7fe640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd870144180 0x7fd87014e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:17.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86e7fc640 1 -- 172.21.15.83:0/3321356008 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd85c0039c0 msgr2=0x7fd8700c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:17.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86e7fc640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd85c0039c0 0x7fd8700c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:17.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86e7fc640 1 -- 172.21.15.83:0/3321356008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd870144180 msgr2=0x7fd87014e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:17.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86e7fc640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd870144180 0x7fd87014e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:17.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86e7fc640 1 -- 172.21.15.83:0/3321356008 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87014b310 con 0x7fd870071ed0 2024-09-19T07:25:17.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86f7fe640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd870144180 0x7fd87014e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:17.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.897+0000 7fd86e7fc640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd870071ed0 0x7fd870143c40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd86000e780 tx=0x7fd86000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:17.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.901+0000 7fd8677fe640 1 -- 172.21.15.83:0/3321356008 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86000c660 con 0x7fd870071ed0 2024-09-19T07:25:17.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.901+0000 7fd8677fe640 1 -- 172.21.15.83:0/3321356008 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd860018070 con 0x7fd870071ed0 2024-09-19T07:25:17.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.901+0000 7fd8756e5640 1 -- 172.21.15.83:0/3321356008 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8701ed760 con 0x7fd870071ed0 2024-09-19T07:25:17.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.901+0000 7fd8677fe640 1 -- 172.21.15.83:0/3321356008 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86000ca20 con 0x7fd870071ed0 2024-09-19T07:25:17.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.901+0000 7fd8756e5640 1 -- 172.21.15.83:0/3321356008 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8701edd20 con 0x7fd870071ed0 2024-09-19T07:25:17.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.901+0000 7fd8756e5640 1 -- 172.21.15.83:0/3321356008 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8700c3e90 con 0x7fd870071ed0 2024-09-19T07:25:17.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.901+0000 7fd8677fe640 1 -- 172.21.15.83:0/3321356008 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd86000cbc0 con 0x7fd870071ed0 2024-09-19T07:25:17.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.901+0000 7fd8677fe640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8500678f0 0x7fd850069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:17.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.901+0000 7fd8677fe640 1 -- 172.21.15.83:0/3321356008 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7fd860085ca0 con 0x7fd870071ed0 2024-09-19T07:25:17.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.901+0000 7fd86effd640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8500678f0 0x7fd850069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:17.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.901+0000 7fd86effd640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8500678f0 0x7fd850069db0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd858001750 tx=0x7fd8580011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:17.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:17.905+0000 7fd8677fe640 1 -- 172.21.15.83:0/3321356008 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd860056220 con 0x7fd870071ed0 2024-09-19T07:25:18.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.101+0000 7fd8756e5640 1 -- 172.21.15.83:0/3321356008 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8700c45f0 con 0x7fd870071ed0 2024-09-19T07:25:18.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.105+0000 7fd8677fe640 1 -- 172.21.15.83:0/3321356008 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7fd860055b20 con 0x7fd870071ed0 2024-09-19T07:25:18.106 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:18.106 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:18.106 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:18.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.105+0000 7fd8756e5640 1 -- 172.21.15.83:0/3321356008 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8500678f0 msgr2=0x7fd850069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:18.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.105+0000 7fd8756e5640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8500678f0 0x7fd850069db0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd858001750 tx=0x7fd8580011a0 comp rx=0 tx=0).stop 2024-09-19T07:25:18.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.105+0000 7fd8756e5640 1 -- 172.21.15.83:0/3321356008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd870071ed0 msgr2=0x7fd870143c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:18.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.105+0000 7fd8756e5640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd870071ed0 0x7fd870143c40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd86000e780 tx=0x7fd86000ec40 comp rx=0 tx=0).stop 2024-09-19T07:25:18.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.105+0000 7fd8756e5640 1 -- 172.21.15.83:0/3321356008 shutdown_connections 2024-09-19T07:25:18.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.105+0000 7fd8756e5640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8500678f0 0x7fd850069db0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:18.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.105+0000 7fd8756e5640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd870144180 0x7fd87014e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:18.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.105+0000 7fd8756e5640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd870071ed0 0x7fd870143c40 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:18.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.105+0000 7fd8756e5640 1 --2- 172.21.15.83:0/3321356008 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd85c0039c0 0x7fd8700c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:18.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.105+0000 7fd8756e5640 1 -- 172.21.15.83:0/3321356008 >> 172.21.15.83:0/3321356008 conn(0x7fd87006bc80 msgr2=0x7fd8700c75d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:18.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.109+0000 7fd8756e5640 1 -- 172.21.15.83:0/3321356008 shutdown_connections 2024-09-19T07:25:18.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:18.109+0000 7fd8756e5640 1 -- 172.21.15.83:0/3321356008 wait complete. 2024-09-19T07:25:18.124 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:25:20.098 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:20.124 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:20.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.225+0000 7fe6df7aa640 1 Processor -- start 2024-09-19T07:25:20.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.225+0000 7fe6df7aa640 1 -- start start 2024-09-19T07:25:20.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6df7aa640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8150310 0x7fe6d80728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6df7aa640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d80c2f90 con 0x7fe6d806cef0 2024-09-19T07:25:20.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6df7aa640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d80c3210 con 0x7fe6d80c4990 2024-09-19T07:25:20.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6df7aa640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe6d80c3e20 con 0x7fe6d8150310 2024-09-19T07:25:20.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6dd546640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe6d806cef0 0x7fe6d814fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53098/0 (socket says 172.21.15.83:53098) 2024-09-19T07:25:20.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6dd546640 1 -- 172.21.15.83:0/4104802940 learned_addr learned my addr 172.21.15.83:0/4104802940 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:20.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2500734417 0 0) 0x7fe6d0001120 con 0x7fe6d806cef0 2024-09-19T07:25:20.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6c4003530 con 0x7fe6d806cef0 2024-09-19T07:25:20.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6dcd45640 1 --2- 172.21.15.83:0/4104802940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8150310 0x7fe6d80728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 787072662 0 0) 0x7fe6c4003530 con 0x7fe6d806cef0 2024-09-19T07:25:20.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8150310 msgr2=0x7fe6d80728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:20.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 --2- 172.21.15.83:0/4104802940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8150310 0x7fe6d80728d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 >> v1:172.21.15.184:6789/0 conn(0x7fe6d80c4990 legacy=0x7fe6d8156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d80c34f0 con 0x7fe6d806cef0 2024-09-19T07:25:20.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fe6d0001870 con 0x7fe6d806cef0 2024-09-19T07:25:20.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 >> v1:172.21.15.83:6789/0 conn(0x7fe6d806cef0 legacy=0x7fe6d814fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 --2- 172.21.15.83:0/4104802940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6c40039f0 0x7fe6c4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 --2- 172.21.15.83:0/4104802940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8150310 0x7fe6c4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 --2- 172.21.15.83:0/4104802940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6c4004d50 0x7fe6c40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d80c3e20 con 0x7fe6c4004d50 2024-09-19T07:25:20.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c400bbe0 con 0x7fe6c40039f0 2024-09-19T07:25:20.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe6c400dff0 con 0x7fe6d8150310 2024-09-19T07:25:20.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6ddd47640 1 --2- 172.21.15.83:0/4104802940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6c4004d50 0x7fe6c40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6dd546640 1 --2- 172.21.15.83:0/4104802940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6c40039f0 0x7fe6c4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6dcd45640 1 --2- 172.21.15.83:0/4104802940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8150310 0x7fe6c4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6ddd47640 1 -- 172.21.15.83:0/4104802940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8150310 msgr2=0x7fe6c4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:20.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6ddd47640 1 --2- 172.21.15.83:0/4104802940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8150310 0x7fe6c4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6ddd47640 1 -- 172.21.15.83:0/4104802940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6c40039f0 msgr2=0x7fe6c4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:20.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6ddd47640 1 --2- 172.21.15.83:0/4104802940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6c40039f0 0x7fe6c4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.229+0000 7fe6ddd47640 1 -- 172.21.15.83:0/4104802940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c4003530 con 0x7fe6c4004d50 2024-09-19T07:25:20.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6ddd47640 1 --2- 172.21.15.83:0/4104802940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6c4004d50 0x7fe6c40051d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4001e70 tx=0x7fe6d401c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc535d9fb90510ae server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:20.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4001a90 con 0x7fe6c4004d50 2024-09-19T07:25:20.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6d4003b80 con 0x7fe6c4004d50 2024-09-19T07:25:20.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6cffff640 1 -- 172.21.15.83:0/4104802940 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4005080 con 0x7fe6c4004d50 2024-09-19T07:25:20.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 -- 172.21.15.83:0/4104802940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6c4004d50 msgr2=0x7fe6c40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 --2- 172.21.15.83:0/4104802940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6c4004d50 0x7fe6c40051d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4001e70 tx=0x7fe6d401c780 comp rx=0 tx=0).stop 2024-09-19T07:25:20.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6ddd47640 1 -- 172.21.15.83:0/4104802940 reap_dead start 2024-09-19T07:25:20.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 -- 172.21.15.83:0/4104802940 shutdown_connections 2024-09-19T07:25:20.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 -- 172.21.15.83:0/4104802940 >> 172.21.15.83:0/4104802940 conn(0x7fe6d80c5090 msgr2=0x7fe6d80c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:20.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 -- 172.21.15.83:0/4104802940 shutdown_connections 2024-09-19T07:25:20.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 -- 172.21.15.83:0/4104802940 wait complete. 2024-09-19T07:25:20.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 Processor -- start 2024-09-19T07:25:20.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 -- start start 2024-09-19T07:25:20.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8152210 0x7fe6d81506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6d81e8400 0x7fe6d81525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6d81473e0 0x7fe6d81531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d806d2d0 con 0x7fe6d81473e0 2024-09-19T07:25:20.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d80c3ab0 con 0x7fe6d81e8400 2024-09-19T07:25:20.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6df7aa640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe6d806d450 con 0x7fe6d8152210 2024-09-19T07:25:20.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.233+0000 7fe6ddd47640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6d81473e0 0x7fe6d81531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6ddd47640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6d81473e0 0x7fe6d81531b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45040/0 (socket says 172.21.15.83:45040) 2024-09-19T07:25:20.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6ddd47640 1 -- 172.21.15.83:0/420220592 learned_addr learned my addr 172.21.15.83:0/420220592 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:20.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6dd546640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8152210 0x7fe6d81506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6dcd45640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6d81e8400 0x7fe6d81525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6ddd47640 1 -- 172.21.15.83:0/420220592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8152210 msgr2=0x7fe6d81506f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6ddd47640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8152210 0x7fe6d81506f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6ddd47640 1 -- 172.21.15.83:0/420220592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6d81e8400 msgr2=0x7fe6d81525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6ddd47640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6d81e8400 0x7fe6d81525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6ddd47640 1 -- 172.21.15.83:0/420220592 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d80addd0 con 0x7fe6d81473e0 2024-09-19T07:25:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6dcd45640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6d81e8400 0x7fe6d81525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:25:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6ddd47640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6d81473e0 0x7fe6d81531b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4005690 tx=0x7fe6c4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6cdffb640 1 -- 172.21.15.83:0/420220592 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d40b7070 con 0x7fe6d81473e0 2024-09-19T07:25:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6cdffb640 1 -- 172.21.15.83:0/420220592 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6d40057f0 con 0x7fe6d81473e0 2024-09-19T07:25:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6df7aa640 1 -- 172.21.15.83:0/420220592 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d8147100 con 0x7fe6d81473e0 2024-09-19T07:25:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6cdffb640 1 -- 172.21.15.83:0/420220592 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4007dc0 con 0x7fe6d81473e0 2024-09-19T07:25:20.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6df7aa640 1 -- 172.21.15.83:0/420220592 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d80ae090 con 0x7fe6d81473e0 2024-09-19T07:25:20.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6df7aa640 1 -- 172.21.15.83:0/420220592 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d80728d0 con 0x7fe6d81473e0 2024-09-19T07:25:20.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6cdffb640 1 -- 172.21.15.83:0/420220592 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe6d40ba070 con 0x7fe6d81473e0 2024-09-19T07:25:20.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6cdffb640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6bc0678c0 0x7fe6bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6cdffb640 1 -- 172.21.15.83:0/420220592 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7fe6d411f3b0 con 0x7fe6d81473e0 2024-09-19T07:25:20.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6dd546640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6bc0678c0 0x7fe6bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.237+0000 7fe6dd546640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6bc0678c0 0x7fe6bc069d80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0001060 tx=0x7fe6d0001120 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:20.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.241+0000 7fe6cdffb640 1 -- 172.21.15.83:0/420220592 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe6d40ac020 con 0x7fe6d81473e0 2024-09-19T07:25:20.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfeb63640 1 Processor -- start 2024-09-19T07:25:20.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfeb63640 1 -- start start 2024-09-19T07:25:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfeb63640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbcf81535d0 0x7fbcf80c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfeb63640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbcf806ffe0 0x7fbcf8155c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfeb63640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcf8156190 con 0x7fbcf8159af0 2024-09-19T07:25:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfeb63640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fbcf80707e0 con 0x7fbcf81535d0 2024-09-19T07:25:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfeb63640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbcf80c2c80 con 0x7fbcf806ffe0 2024-09-19T07:25:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfc8ff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbcf81535d0 0x7fbcf80c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfc8ff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbcf81535d0 0x7fbcf80c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45054/0 (socket says 172.21.15.83:45054) 2024-09-19T07:25:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfc8ff640 1 -- 172.21.15.83:0/4010028894 learned_addr learned my addr 172.21.15.83:0/4010028894 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcf7fff640 1 --2- 172.21.15.83:0/4010028894 >> v2:172.21.15.184:3300/0 conn(0x7fbcf806ffe0 0x7fbcf8155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfc8ff640 1 -- 172.21.15.83:0/4010028894 >> v2:172.21.15.184:3300/0 conn(0x7fbcf806ffe0 msgr2=0x7fbcf8155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfc8ff640 1 --2- 172.21.15.83:0/4010028894 >> v2:172.21.15.184:3300/0 conn(0x7fbcf806ffe0 0x7fbcf8155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfc8ff640 1 -- 172.21.15.83:0/4010028894 >> v1:172.21.15.184:6789/0 conn(0x7fbcf8159af0 legacy=0x7fbcf80c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfc8ff640 1 -- 172.21.15.83:0/4010028894 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf80c26c0 con 0x7fbcf81535d0 2024-09-19T07:25:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcf7fff640 1 --2- 172.21.15.83:0/4010028894 >> v2:172.21.15.184:3300/0 conn(0x7fbcf806ffe0 0x7fbcf8155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.265+0000 7fbcfc8ff640 1 --2- 172.21.15.83:0/4010028894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf81535d0 0x7fbcf80c6460 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80c2e00 tx=0x7fbce401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377abefa5936201e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcf77fe640 1 -- 172.21.15.83:0/4010028894 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce40b0070 con 0x7fbcf81535d0 2024-09-19T07:25:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcf77fe640 1 -- 172.21.15.83:0/4010028894 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbce40ab600 con 0x7fbcf81535d0 2024-09-19T07:25:20.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcf77fe640 1 -- 172.21.15.83:0/4010028894 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce40ab9d0 con 0x7fbcf81535d0 2024-09-19T07:25:20.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 -- 172.21.15.83:0/4010028894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf81535d0 msgr2=0x7fbcf80c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 --2- 172.21.15.83:0/4010028894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf81535d0 0x7fbcf80c6460 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80c2e00 tx=0x7fbce401c3c0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 -- 172.21.15.83:0/4010028894 shutdown_connections 2024-09-19T07:25:20.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 --2- 172.21.15.83:0/4010028894 >> v2:172.21.15.184:3300/0 conn(0x7fbcf806ffe0 0x7fbcf8155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 --2- 172.21.15.83:0/4010028894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf81535d0 0x7fbcf80c6460 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 -- 172.21.15.83:0/4010028894 >> 172.21.15.83:0/4010028894 conn(0x7fbcf80c5060 msgr2=0x7fbcf80c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:20.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 -- 172.21.15.83:0/4010028894 shutdown_connections 2024-09-19T07:25:20.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 -- 172.21.15.83:0/4010028894 wait complete. 2024-09-19T07:25:20.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 Processor -- start 2024-09-19T07:25:20.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 -- start start 2024-09-19T07:25:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf806ffe0 0x7fbcf815ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbcf81535d0 0x7fbcf81f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcf8159af0 0x7fbcf81f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf80c37f0 con 0x7fbcf806ffe0 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf80c3670 con 0x7fbcf8159af0 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbcf8159f20 con 0x7fbcf81535d0 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfc8ff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf806ffe0 0x7fbcf815ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfc8ff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf806ffe0 0x7fbcf815ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45070/0 (socket says 172.21.15.83:45070) 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfc8ff640 1 -- 172.21.15.83:0/2995737294 learned_addr learned my addr 172.21.15.83:0/2995737294 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcf7fff640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbcf81535d0 0x7fbcf81f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfc8ff640 1 -- 172.21.15.83:0/2995737294 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbcf81535d0 msgr2=0x7fbcf81f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfc8ff640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbcf81535d0 0x7fbcf81f1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfc8ff640 1 -- 172.21.15.83:0/2995737294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcf8159af0 msgr2=0x7fbcf81f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfd100640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcf8159af0 0x7fbcf81f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfc8ff640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcf8159af0 0x7fbcf81f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfc8ff640 1 -- 172.21.15.83:0/2995737294 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf81f2830 con 0x7fbcf806ffe0 2024-09-19T07:25:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfd100640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcf8159af0 0x7fbcf81f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfc8ff640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf806ffe0 0x7fbcf815ed40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbce40ae040 tx=0x7fbce401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcf57fa640 1 -- 172.21.15.83:0/2995737294 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce40ba020 con 0x7fbcf806ffe0 2024-09-19T07:25:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcf57fa640 1 -- 172.21.15.83:0/2995737294 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbce40b6640 con 0x7fbcf806ffe0 2024-09-19T07:25:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 -- 172.21.15.83:0/2995737294 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf806f9d0 con 0x7fbcf806ffe0 2024-09-19T07:25:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcfeb63640 1 -- 172.21.15.83:0/2995737294 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf80ad780 con 0x7fbcf806ffe0 2024-09-19T07:25:20.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.269+0000 7fbcf57fa640 1 -- 172.21.15.83:0/2995737294 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce40b0040 con 0x7fbcf806ffe0 2024-09-19T07:25:20.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.273+0000 7fbcfeb63640 1 -- 172.21.15.83:0/2995737294 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcd80052a0 con 0x7fbcf806ffe0 2024-09-19T07:25:20.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.273+0000 7fbcf57fa640 1 -- 172.21.15.83:0/2995737294 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbce40bd020 con 0x7fbcf806ffe0 2024-09-19T07:25:20.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.273+0000 7fbcf57fa640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbcdc0678c0 0x7fbcdc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.273+0000 7fbcf57fa640 1 -- 172.21.15.83:0/2995737294 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7fbce4120860 con 0x7fbcf806ffe0 2024-09-19T07:25:20.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.273+0000 7fbcf7fff640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbcdc0678c0 0x7fbcdc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.273+0000 7fbcf7fff640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbcdc0678c0 0x7fbcdc069d80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbcec006fd0 tx=0x7fbcec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:20.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.281+0000 7fbcf57fa640 1 -- 172.21.15.83:0/2995737294 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbce40c4070 con 0x7fbcf806ffe0 2024-09-19T07:25:20.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.441+0000 7fe6df7aa640 1 -- 172.21.15.83:0/420220592 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6d814d240 con 0x7fe6d81473e0 2024-09-19T07:25:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6cdffb640 1 -- 172.21.15.83:0/420220592 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7fe6d4016490 con 0x7fe6d81473e0 2024-09-19T07:25:20.444 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:20.444 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6df7aa640 1 -- 172.21.15.83:0/420220592 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6bc0678c0 msgr2=0x7fe6bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6df7aa640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6bc0678c0 0x7fe6bc069d80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0001060 tx=0x7fe6d0001120 comp rx=0 tx=0).stop 2024-09-19T07:25:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6df7aa640 1 -- 172.21.15.83:0/420220592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6d81473e0 msgr2=0x7fe6d81531b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6df7aa640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6d81473e0 0x7fe6d81531b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4005690 tx=0x7fe6c4005840 comp rx=0 tx=0).stop 2024-09-19T07:25:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6df7aa640 1 -- 172.21.15.83:0/420220592 shutdown_connections 2024-09-19T07:25:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6df7aa640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6bc0678c0 0x7fe6bc069d80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6df7aa640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6d81473e0 0x7fe6d81531b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6df7aa640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6d81e8400 0x7fe6d81525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6df7aa640 1 --2- 172.21.15.83:0/420220592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6d8152210 0x7fe6d81506f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6df7aa640 1 -- 172.21.15.83:0/420220592 >> 172.21.15.83:0/420220592 conn(0x7fe6d80c5090 msgr2=0x7fe6d814d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6df7aa640 1 -- 172.21.15.83:0/420220592 shutdown_connections 2024-09-19T07:25:20.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.445+0000 7fe6df7aa640 1 -- 172.21.15.83:0/420220592 wait complete. 2024-09-19T07:25:20.461 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-09-19T07:25:20.462 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-19T07:25:20.462 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:25:20.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.477+0000 7fbcfeb63640 1 -- 172.21.15.83:0/2995737294 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbcd8005590 con 0x7fbcf806ffe0 2024-09-19T07:25:20.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.477+0000 7fbcf57fa640 1 -- 172.21.15.83:0/2995737294 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7fbce40f5e20 con 0x7fbcf806ffe0 2024-09-19T07:25:20.479 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:20.480 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:20.480 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:20.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.481+0000 7fbcfeb63640 1 -- 172.21.15.83:0/2995737294 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbcdc0678c0 msgr2=0x7fbcdc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.481+0000 7fbcfeb63640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbcdc0678c0 0x7fbcdc069d80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbcec006fd0 tx=0x7fbcec008040 comp rx=0 tx=0).stop 2024-09-19T07:25:20.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.481+0000 7fbcfeb63640 1 -- 172.21.15.83:0/2995737294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf806ffe0 msgr2=0x7fbcf815ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.481+0000 7fbcfeb63640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf806ffe0 0x7fbcf815ed40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbce40ae040 tx=0x7fbce401ac00 comp rx=0 tx=0).stop 2024-09-19T07:25:20.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.481+0000 7fbcfeb63640 1 -- 172.21.15.83:0/2995737294 shutdown_connections 2024-09-19T07:25:20.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.481+0000 7fbcfeb63640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbcdc0678c0 0x7fbcdc069d80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.481+0000 7fbcfeb63640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcf8159af0 0x7fbcf81f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.481+0000 7fbcfeb63640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbcf81535d0 0x7fbcf81f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.481+0000 7fbcfeb63640 1 --2- 172.21.15.83:0/2995737294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf806ffe0 0x7fbcf815ed40 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.481+0000 7fbcfeb63640 1 -- 172.21.15.83:0/2995737294 >> 172.21.15.83:0/2995737294 conn(0x7fbcf80c5060 msgr2=0x7fbcf80a8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:20.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.481+0000 7fbcfeb63640 1 -- 172.21.15.83:0/2995737294 shutdown_connections 2024-09-19T07:25:20.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.481+0000 7fbcfeb63640 1 -- 172.21.15.83:0/2995737294 wait complete. 2024-09-19T07:25:20.485 INFO:tasks.ceph.mds.b:Stopped 2024-09-19T07:25:20.485 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:20.498 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:25:20.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc70257640 1 Processor -- start 2024-09-19T07:25:20.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc70257640 1 -- start start 2024-09-19T07:25:20.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc70257640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdc680751f0 0x7fdc6806cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc70257640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc6806f5e0 0x7fdc68152f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc70257640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc6806ef10 con 0x7fdc6806d100 2024-09-19T07:25:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc70257640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fdc6806c3a0 con 0x7fdc680751f0 2024-09-19T07:25:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc70257640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdc6806f1f0 con 0x7fdc6806f5e0 2024-09-19T07:25:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc6dff3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdc680751f0 0x7fdc6806cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc6dff3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdc680751f0 0x7fdc6806cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45074/0 (socket says 172.21.15.83:45074) 2024-09-19T07:25:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc6dff3640 1 -- 172.21.15.83:0/154529785 learned_addr learned my addr 172.21.15.83:0/154529785 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc6e7f4640 1 --2- 172.21.15.83:0/154529785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc6806f5e0 0x7fdc68152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc6dff3640 1 -- 172.21.15.83:0/154529785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc6806f5e0 msgr2=0x7fdc68152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc6dff3640 1 --2- 172.21.15.83:0/154529785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc6806f5e0 0x7fdc68152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc6dff3640 1 -- 172.21.15.83:0/154529785 >> v1:172.21.15.184:6789/0 conn(0x7fdc6806d100 legacy=0x7fdc680c4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.597+0000 7fdc6dff3640 1 -- 172.21.15.83:0/154529785 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6806c5f0 con 0x7fdc680751f0 2024-09-19T07:25:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6dff3640 1 --2- 172.21.15.83:0/154529785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc680751f0 0x7fdc6806cb10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009870 tx=0x7fdc5801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44b7e389f6d2224 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6cff1640 1 -- 172.21.15.83:0/154529785 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc580b1070 con 0x7fdc680751f0 2024-09-19T07:25:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6cff1640 1 -- 172.21.15.83:0/154529785 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc580ac640 con 0x7fdc680751f0 2024-09-19T07:25:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6cff1640 1 -- 172.21.15.83:0/154529785 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc580aca10 con 0x7fdc680751f0 2024-09-19T07:25:20.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 -- 172.21.15.83:0/154529785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc680751f0 msgr2=0x7fdc6806cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 --2- 172.21.15.83:0/154529785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc680751f0 0x7fdc6806cb10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009870 tx=0x7fdc5801c440 comp rx=0 tx=0).stop 2024-09-19T07:25:20.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 -- 172.21.15.83:0/154529785 shutdown_connections 2024-09-19T07:25:20.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 --2- 172.21.15.83:0/154529785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc6806f5e0 0x7fdc68152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 --2- 172.21.15.83:0/154529785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc680751f0 0x7fdc6806cb10 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 -- 172.21.15.83:0/154529785 >> 172.21.15.83:0/154529785 conn(0x7fdc680c90e0 msgr2=0x7fdc680aa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:20.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 -- 172.21.15.83:0/154529785 shutdown_connections 2024-09-19T07:25:20.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 -- 172.21.15.83:0/154529785 wait complete. 2024-09-19T07:25:20.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 Processor -- start 2024-09-19T07:25:20.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 -- start start 2024-09-19T07:25:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc6806d100 0x7fdc681ef030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc6806f5e0 0x7fdc6814f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc680751f0 0x7fdc6814dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdc68162d10 con 0x7fdc6806d100 2024-09-19T07:25:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdc680757e0 con 0x7fdc6806f5e0 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc70257640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdc68162e90 con 0x7fdc680751f0 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6dff3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc6806d100 0x7fdc681ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6dff3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc6806d100 0x7fdc681ef030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45080/0 (socket says 172.21.15.83:45080) 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6dff3640 1 -- 172.21.15.83:0/3988824891 learned_addr learned my addr 172.21.15.83:0/3988824891 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6d7f2640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc6806f5e0 0x7fdc6814f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6e7f4640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc680751f0 0x7fdc6814dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6dff3640 1 -- 172.21.15.83:0/3988824891 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc680751f0 msgr2=0x7fdc6814dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6dff3640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc680751f0 0x7fdc6814dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6dff3640 1 -- 172.21.15.83:0/3988824891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc6806f5e0 msgr2=0x7fdc6814f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6dff3640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc6806f5e0 0x7fdc6814f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6dff3640 1 -- 172.21.15.83:0/3988824891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc680ad5d0 con 0x7fdc6806d100 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6d7f2640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc6806f5e0 0x7fdc6814f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:25:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.601+0000 7fdc6dff3640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc6806d100 0x7fdc681ef030 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdc5801aaf0 tx=0x7fdc580191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:20.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.605+0000 7fdc5e7fc640 1 -- 172.21.15.83:0/3988824891 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc580b9020 con 0x7fdc6806d100 2024-09-19T07:25:20.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.605+0000 7fdc5e7fc640 1 -- 172.21.15.83:0/3988824891 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc580b5410 con 0x7fdc6806d100 2024-09-19T07:25:20.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.605+0000 7fdc70257640 1 -- 172.21.15.83:0/3988824891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc680ad2f0 con 0x7fdc6806d100 2024-09-19T07:25:20.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.605+0000 7fdc5e7fc640 1 -- 172.21.15.83:0/3988824891 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc580b1040 con 0x7fdc6806d100 2024-09-19T07:25:20.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.605+0000 7fdc70257640 1 -- 172.21.15.83:0/3988824891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc681ee850 con 0x7fdc6806d100 2024-09-19T07:25:20.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.605+0000 7fdc70257640 1 -- 172.21.15.83:0/3988824891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc68152f10 con 0x7fdc6806d100 2024-09-19T07:25:20.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.605+0000 7fdc5e7fc640 1 -- 172.21.15.83:0/3988824891 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdc580bc020 con 0x7fdc6806d100 2024-09-19T07:25:20.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.605+0000 7fdc5e7fc640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdc4c0678c0 0x7fdc4c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:20.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.605+0000 7fdc5e7fc640 1 -- 172.21.15.83:0/3988824891 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7fdc5811fee0 con 0x7fdc6806d100 2024-09-19T07:25:20.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.605+0000 7fdc6d7f2640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdc4c0678c0 0x7fdc4c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:20.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.605+0000 7fdc6d7f2640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdc4c0678c0 0x7fdc4c069d80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdc680c52e0 tx=0x7fdc64009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.609+0000 7fdc5e7fc640 1 -- 172.21.15.83:0/3988824891 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdc580c3030 con 0x7fdc6806d100 2024-09-19T07:25:20.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.793+0000 7fdc70257640 1 -- 172.21.15.83:0/3988824891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc68068900 con 0x7fdc6806d100 2024-09-19T07:25:20.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.793+0000 7fdc5e7fc640 1 -- 172.21.15.83:0/3988824891 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7fdc58016150 con 0x7fdc6806d100 2024-09-19T07:25:20.795 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:20.795 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:20.795 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:20.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.797+0000 7fdc70257640 1 -- 172.21.15.83:0/3988824891 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdc4c0678c0 msgr2=0x7fdc4c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.797+0000 7fdc70257640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdc4c0678c0 0x7fdc4c069d80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdc680c52e0 tx=0x7fdc64009040 comp rx=0 tx=0).stop 2024-09-19T07:25:20.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.797+0000 7fdc70257640 1 -- 172.21.15.83:0/3988824891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc6806d100 msgr2=0x7fdc681ef030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:20.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.797+0000 7fdc70257640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc6806d100 0x7fdc681ef030 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdc5801aaf0 tx=0x7fdc580191e0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.797+0000 7fdc70257640 1 -- 172.21.15.83:0/3988824891 shutdown_connections 2024-09-19T07:25:20.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.797+0000 7fdc70257640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdc4c0678c0 0x7fdc4c069d80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.797+0000 7fdc70257640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc680751f0 0x7fdc6814dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.797+0000 7fdc70257640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc6806f5e0 0x7fdc6814f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.797+0000 7fdc70257640 1 --2- 172.21.15.83:0/3988824891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc6806d100 0x7fdc681ef030 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:20.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.797+0000 7fdc70257640 1 -- 172.21.15.83:0/3988824891 >> 172.21.15.83:0/3988824891 conn(0x7fdc680c90e0 msgr2=0x7fdc680c58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:20.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.797+0000 7fdc70257640 1 -- 172.21.15.83:0/3988824891 shutdown_connections 2024-09-19T07:25:20.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:20.797+0000 7fdc70257640 1 -- 172.21.15.83:0/3988824891 wait complete. 2024-09-19T07:25:20.813 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:25:22.498 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:22.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f9740929640 1 Processor -- start 2024-09-19T07:25:22.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f9740929640 1 -- start start 2024-09-19T07:25:22.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f9740929640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f973c1566b0 0x7f973c0abb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:22.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f9740929640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f973c1592c0 con 0x7f973c06e270 2024-09-19T07:25:22.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f9740929640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f973c06e900 con 0x7f973c0c42e0 2024-09-19T07:25:22.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f9740929640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f973c06e6e0 con 0x7f973c1566b0 2024-09-19T07:25:22.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f973ad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f973c1566b0 0x7f973c0abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:22.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f9739d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f973c06e270 0x7f973c153060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53104/0 (socket says 172.21.15.83:53104) 2024-09-19T07:25:22.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f9739d9b640 1 -- 172.21.15.83:0/508246017 learned_addr learned my addr 172.21.15.83:0/508246017 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:22.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f973ad9d640 1 -- 172.21.15.83:0/508246017 >> v1:172.21.15.184:6789/0 conn(0x7f973c0c42e0 legacy=0x7f973c159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:22.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f973ad9d640 1 -- 172.21.15.83:0/508246017 >> v1:172.21.15.83:6789/0 conn(0x7f973c06e270 legacy=0x7f973c153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:22.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f973ad9d640 1 -- 172.21.15.83:0/508246017 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973c06ebe0 con 0x7f973c1566b0 2024-09-19T07:25:22.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.629+0000 7f973959a640 1 -- 172.21.15.83:0/508246017 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 941496987 0 0) 0x7f973c1592c0 con 0x7f973c06e270 2024-09-19T07:25:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f973ad9d640 1 --2- 172.21.15.83:0/508246017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f973c1566b0 0x7f973c0abb30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9734009d90 tx=0x7f973401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac21e58b9a457a94 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f973959a640 1 -- 172.21.15.83:0/508246017 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97340b2070 con 0x7f973c1566b0 2024-09-19T07:25:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f973959a640 1 -- 172.21.15.83:0/508246017 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97340a57f0 con 0x7f973c1566b0 2024-09-19T07:25:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f973959a640 1 -- 172.21.15.83:0/508246017 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97340a5bc0 con 0x7f973c1566b0 2024-09-19T07:25:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 -- 172.21.15.83:0/508246017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f973c1566b0 msgr2=0x7f973c0abb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 --2- 172.21.15.83:0/508246017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f973c1566b0 0x7f973c0abb30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9734009d90 tx=0x7f973401c930 comp rx=0 tx=0).stop 2024-09-19T07:25:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 -- 172.21.15.83:0/508246017 shutdown_connections 2024-09-19T07:25:22.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 --2- 172.21.15.83:0/508246017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f973c1566b0 0x7f973c0abb30 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 -- 172.21.15.83:0/508246017 >> 172.21.15.83:0/508246017 conn(0x7f973c06bc80 msgr2=0x7f973c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:22.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 -- 172.21.15.83:0/508246017 shutdown_connections 2024-09-19T07:25:22.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 -- 172.21.15.83:0/508246017 wait complete. 2024-09-19T07:25:22.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 Processor -- start 2024-09-19T07:25:22.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 -- start start 2024-09-19T07:25:22.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f973c06e270 0x7f973c154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:22.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f973c0c42e0 0x7f973c0c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:22.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f973c1566b0 0x7f973c1f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:22.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f973c0c4840 con 0x7f973c0c42e0 2024-09-19T07:25:22.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f973c0c46c0 con 0x7f973c06e270 2024-09-19T07:25:22.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9740929640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f973c163de0 con 0x7f973c1566b0 2024-09-19T07:25:22.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9739d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f973c0c42e0 0x7f973c0c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9739d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f973c0c42e0 0x7f973c0c6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45096/0 (socket says 172.21.15.83:45096) 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9739d9b640 1 -- 172.21.15.83:0/4071619123 learned_addr learned my addr 172.21.15.83:0/4071619123 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f973ad9d640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f973c1566b0 0x7f973c1f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f973a59c640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f973c06e270 0x7f973c154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9739d9b640 1 -- 172.21.15.83:0/4071619123 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f973c1566b0 msgr2=0x7f973c1f03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9739d9b640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f973c1566b0 0x7f973c1f03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9739d9b640 1 -- 172.21.15.83:0/4071619123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f973c06e270 msgr2=0x7f973c154e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9739d9b640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f973c06e270 0x7f973c154e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f9739d9b640 1 -- 172.21.15.83:0/4071619123 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973c0c8270 con 0x7f973c0c42e0 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.633+0000 7f973a59c640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f973c06e270 0x7f973c154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.637+0000 7f9739d9b640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f973c0c42e0 0x7f973c0c6b80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f972c007fd0 tx=0x7f972c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.637+0000 7f972affd640 1 -- 172.21.15.83:0/4071619123 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972c017070 con 0x7f973c0c42e0 2024-09-19T07:25:22.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.637+0000 7f972affd640 1 -- 172.21.15.83:0/4071619123 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f972c005d10 con 0x7f973c0c42e0 2024-09-19T07:25:22.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.637+0000 7f9740929640 1 -- 172.21.15.83:0/4071619123 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f973c14cc20 con 0x7f973c0c42e0 2024-09-19T07:25:22.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.637+0000 7f972affd640 1 -- 172.21.15.83:0/4071619123 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972c007310 con 0x7f973c0c42e0 2024-09-19T07:25:22.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.637+0000 7f9740929640 1 -- 172.21.15.83:0/4071619123 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f973c155870 con 0x7f973c0c42e0 2024-09-19T07:25:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.637+0000 7f9740929640 1 -- 172.21.15.83:0/4071619123 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f973c0abb30 con 0x7f973c0c42e0 2024-09-19T07:25:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.637+0000 7f972affd640 1 -- 172.21.15.83:0/4071619123 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f972c01a070 con 0x7f973c0c42e0 2024-09-19T07:25:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.637+0000 7f972affd640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f971c0678f0 0x7f971c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.637+0000 7f972affd640 1 -- 172.21.15.83:0/4071619123 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7f972c085640 con 0x7f973c0c42e0 2024-09-19T07:25:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.637+0000 7f973a59c640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f971c0678f0 0x7f971c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:22.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.637+0000 7f973a59c640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f971c0678f0 0x7f971c069db0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f973c06c5f0 tx=0x7f9730008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:22.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.641+0000 7f972affd640 1 -- 172.21.15.83:0/4071619123 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f972c055bc0 con 0x7f973c0c42e0 2024-09-19T07:25:22.814 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:22.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.825+0000 7f9740929640 1 -- 172.21.15.83:0/4071619123 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f973c06e050 con 0x7f973c0c42e0 2024-09-19T07:25:22.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.825+0000 7f972affd640 1 -- 172.21.15.83:0/4071619123 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7f972c0554c0 con 0x7f973c0c42e0 2024-09-19T07:25:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:22.826 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:22.826 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:22.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.829+0000 7f9740929640 1 -- 172.21.15.83:0/4071619123 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f971c0678f0 msgr2=0x7f971c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:22.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.829+0000 7f9740929640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f971c0678f0 0x7f971c069db0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f973c06c5f0 tx=0x7f9730008040 comp rx=0 tx=0).stop 2024-09-19T07:25:22.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.829+0000 7f9740929640 1 -- 172.21.15.83:0/4071619123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f973c0c42e0 msgr2=0x7f973c0c6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:22.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.829+0000 7f9740929640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f973c0c42e0 0x7f973c0c6b80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f972c007fd0 tx=0x7f972c00c8e0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.829+0000 7f9740929640 1 -- 172.21.15.83:0/4071619123 shutdown_connections 2024-09-19T07:25:22.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.829+0000 7f9740929640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f971c0678f0 0x7f971c069db0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.829+0000 7f9740929640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f973c1566b0 0x7f973c1f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.829+0000 7f9740929640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f973c0c42e0 0x7f973c0c6b80 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.829+0000 7f9740929640 1 --2- 172.21.15.83:0/4071619123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f973c06e270 0x7f973c154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.829+0000 7f9740929640 1 -- 172.21.15.83:0/4071619123 >> 172.21.15.83:0/4071619123 conn(0x7f973c06bc80 msgr2=0x7f973c0ac200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:22.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.829+0000 7f9740929640 1 -- 172.21.15.83:0/4071619123 shutdown_connections 2024-09-19T07:25:22.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.829+0000 7f9740929640 1 -- 172.21.15.83:0/4071619123 wait complete. 2024-09-19T07:25:22.845 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:25:22.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.929+0000 7f21b267c640 1 Processor -- start 2024-09-19T07:25:22.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.929+0000 7f21b267c640 1 -- start start 2024-09-19T07:25:22.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21ac0c6120 0x7f21ac0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:22.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21ac1592f0 0x7f21ac157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:22.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21ac156200 con 0x7f21ac155d70 2024-09-19T07:25:22.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f21ac0c9930 con 0x7f21ac1592f0 2024-09-19T07:25:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f21ac1564e0 con 0x7f21ac0c6120 2024-09-19T07:25:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b0c19640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21ac1592f0 0x7f21ac157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b0c19640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21ac1592f0 0x7f21ac157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45112/0 (socket says 172.21.15.83:45112) 2024-09-19T07:25:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b0c19640 1 -- 172.21.15.83:0/4228379069 learned_addr learned my addr 172.21.15.83:0/4228379069 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21abfff640 1 --2- 172.21.15.83:0/4228379069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21ac0c6120 0x7f21ac0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b0c19640 1 -- 172.21.15.83:0/4228379069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21ac0c6120 msgr2=0x7f21ac0c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b0c19640 1 --2- 172.21.15.83:0/4228379069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21ac0c6120 0x7f21ac0c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b0c19640 1 -- 172.21.15.83:0/4228379069 >> v1:172.21.15.184:6789/0 conn(0x7f21ac155d70 legacy=0x7f21ac1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b0c19640 1 -- 172.21.15.83:0/4228379069 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ac1567c0 con 0x7f21ac1592f0 2024-09-19T07:25:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21abfff640 1 --2- 172.21.15.83:0/4228379069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21ac0c6120 0x7f21ac0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:22.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b0c19640 1 --2- 172.21.15.83:0/4228379069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21ac1592f0 0x7f21ac157ce0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009e50 tx=0x7f21a401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2fe2389c9d5cad server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:22.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21ab7fe640 1 -- 172.21.15.83:0/4228379069 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a40a5670 con 0x7f21ac1592f0 2024-09-19T07:25:22.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21ab7fe640 1 -- 172.21.15.83:0/4228379069 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a40a5810 con 0x7f21ac1592f0 2024-09-19T07:25:22.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21ab7fe640 1 -- 172.21.15.83:0/4228379069 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a40a5be0 con 0x7f21ac1592f0 2024-09-19T07:25:22.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 -- 172.21.15.83:0/4228379069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21ac1592f0 msgr2=0x7f21ac157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:22.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 --2- 172.21.15.83:0/4228379069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21ac1592f0 0x7f21ac157ce0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009e50 tx=0x7f21a401c9f0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 -- 172.21.15.83:0/4228379069 shutdown_connections 2024-09-19T07:25:22.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 --2- 172.21.15.83:0/4228379069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21ac1592f0 0x7f21ac157ce0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 --2- 172.21.15.83:0/4228379069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21ac0c6120 0x7f21ac0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 -- 172.21.15.83:0/4228379069 >> 172.21.15.83:0/4228379069 conn(0x7f21ac06bc80 msgr2=0x7f21ac06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:22.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 -- 172.21.15.83:0/4228379069 shutdown_connections 2024-09-19T07:25:22.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.933+0000 7f21b267c640 1 -- 172.21.15.83:0/4228379069 wait complete. 2024-09-19T07:25:22.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21b267c640 1 Processor -- start 2024-09-19T07:25:22.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21b267c640 1 -- start start 2024-09-19T07:25:22.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21b267c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21ac0c6120 0x7f21ac148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:22.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21b267c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21ac1592f0 0x7f21ac14def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:22.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21b267c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21ac148560 0x7f21ac14f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:22.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21b267c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac153860 con 0x7f21ac1592f0 2024-09-19T07:25:22.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21b267c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac162ea0 con 0x7f21ac148560 2024-09-19T07:25:22.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21b267c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f21ac14da70 con 0x7f21ac0c6120 2024-09-19T07:25:22.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21ac1592f0 0x7f21ac14def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:22.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21ac1592f0 0x7f21ac14def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45118/0 (socket says 172.21.15.83:45118) 2024-09-19T07:25:22.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21a3fff640 1 -- 172.21.15.83:0/3232483399 learned_addr learned my addr 172.21.15.83:0/3232483399 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:22.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21abfff640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21ac0c6120 0x7f21ac148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:22.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21b0c19640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21ac148560 0x7f21ac14f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:22.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21a3fff640 1 -- 172.21.15.83:0/3232483399 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21ac0c6120 msgr2=0x7f21ac148c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21a3fff640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21ac0c6120 0x7f21ac148c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21a3fff640 1 -- 172.21.15.83:0/3232483399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21ac148560 msgr2=0x7f21ac14f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21a3fff640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21ac148560 0x7f21ac14f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21a3fff640 1 -- 172.21.15.83:0/3232483399 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ac145f50 con 0x7f21ac1592f0 2024-09-19T07:25:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21a3fff640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21ac1592f0 0x7f21ac14def0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f219c00c410 tx=0x7f219c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21a97fa640 1 -- 172.21.15.83:0/3232483399 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c005020 con 0x7f21ac1592f0 2024-09-19T07:25:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21a97fa640 1 -- 172.21.15.83:0/3232483399 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f219c0051c0 con 0x7f21ac1592f0 2024-09-19T07:25:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21b267c640 1 -- 172.21.15.83:0/3232483399 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21ac14d790 con 0x7f21ac1592f0 2024-09-19T07:25:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21a97fa640 1 -- 172.21.15.83:0/3232483399 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c007020 con 0x7f21ac1592f0 2024-09-19T07:25:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21b267c640 1 -- 172.21.15.83:0/3232483399 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21ac14e430 con 0x7f21ac1592f0 2024-09-19T07:25:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.937+0000 7f21b267c640 1 -- 172.21.15.83:0/3232483399 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21ac0c6500 con 0x7f21ac1592f0 2024-09-19T07:25:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.941+0000 7f21a97fa640 1 -- 172.21.15.83:0/3232483399 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f219c019030 con 0x7f21ac1592f0 2024-09-19T07:25:22.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.941+0000 7f21a97fa640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21900678c0 0x7f2190069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:22.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.941+0000 7f21abfff640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21900678c0 0x7f2190069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:22.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.941+0000 7f21a97fa640 1 -- 172.21.15.83:0/3232483399 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7f219c085960 con 0x7f21ac1592f0 2024-09-19T07:25:22.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.941+0000 7f21abfff640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21900678c0 0x7f2190069d80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f21ac159d30 tx=0x7f2194008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:22.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:22.945+0000 7f21a97fa640 1 -- 172.21.15.83:0/3232483399 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f219c020020 con 0x7f21ac1592f0 2024-09-19T07:25:23.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.133+0000 7f21b267c640 1 -- 172.21.15.83:0/3232483399 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21ac069630 con 0x7f21ac1592f0 2024-09-19T07:25:23.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.137+0000 7f21a97fa640 1 -- 172.21.15.83:0/3232483399 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7f219c055ee0 con 0x7f21ac1592f0 2024-09-19T07:25:23.138 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:23.138 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:23.138 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:23.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.141+0000 7f21b267c640 1 -- 172.21.15.83:0/3232483399 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21900678c0 msgr2=0x7f2190069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:23.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.141+0000 7f21b267c640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21900678c0 0x7f2190069d80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f21ac159d30 tx=0x7f2194008040 comp rx=0 tx=0).stop 2024-09-19T07:25:23.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.141+0000 7f21b267c640 1 -- 172.21.15.83:0/3232483399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21ac1592f0 msgr2=0x7f21ac14def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:23.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.141+0000 7f21b267c640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21ac1592f0 0x7f21ac14def0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f219c00c410 tx=0x7f219c00c8d0 comp rx=0 tx=0).stop 2024-09-19T07:25:23.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.141+0000 7f21b267c640 1 -- 172.21.15.83:0/3232483399 shutdown_connections 2024-09-19T07:25:23.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.141+0000 7f21b267c640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21900678c0 0x7f2190069d80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:23.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.141+0000 7f21b267c640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21ac148560 0x7f21ac14f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:23.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.141+0000 7f21b267c640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21ac1592f0 0x7f21ac14def0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:23.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.141+0000 7f21b267c640 1 --2- 172.21.15.83:0/3232483399 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21ac0c6120 0x7f21ac148c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:23.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.141+0000 7f21b267c640 1 -- 172.21.15.83:0/3232483399 >> 172.21.15.83:0/3232483399 conn(0x7f21ac06bc80 msgr2=0x7f21ac14cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:23.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.141+0000 7f21b267c640 1 -- 172.21.15.83:0/3232483399 shutdown_connections 2024-09-19T07:25:23.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:23.141+0000 7f21b267c640 1 -- 172.21.15.83:0/3232483399 wait complete. 2024-09-19T07:25:23.156 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:25:24.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:24.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26f942640 1 Processor -- start 2024-09-19T07:25:24.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26f942640 1 -- start start 2024-09-19T07:25:24.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26f942640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd268071100 0x7fd2681545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:24.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26f942640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2680a9e60 con 0x7fd2680c28e0 2024-09-19T07:25:24.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26f942640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2680aa140 con 0x7fd26806dea0 2024-09-19T07:25:24.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26f942640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd268071b50 con 0x7fd268071100 2024-09-19T07:25:24.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26cedd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd2680c28e0 0x7fd268150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53108/0 (socket says 172.21.15.83:53108) 2024-09-19T07:25:24.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26cedd640 1 -- 172.21.15.83:0/4098202534 learned_addr learned my addr 172.21.15.83:0/4098202534 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:24.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 359695840 0 0) 0x7fd2680a9e60 con 0x7fd2680c28e0 2024-09-19T07:25:24.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd254003530 con 0x7fd2680c28e0 2024-09-19T07:25:24.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26d6de640 1 --2- 172.21.15.83:0/4098202534 >> v2:172.21.15.184:3300/0 conn(0x7fd268071100 0x7fd2681545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:24.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2160013193 0 0) 0x7fd254003530 con 0x7fd2680c28e0 2024-09-19T07:25:24.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 >> v2:172.21.15.184:3300/0 conn(0x7fd268071100 msgr2=0x7fd2681545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:24.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 --2- 172.21.15.83:0/4098202534 >> v2:172.21.15.184:3300/0 conn(0x7fd268071100 0x7fd2681545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:24.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 >> v1:172.21.15.184:6789/0 conn(0x7fd26806dea0 legacy=0x7fd26806e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:24.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2680c2240 con 0x7fd2680c28e0 2024-09-19T07:25:24.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fd264001a00 con 0x7fd2680c28e0 2024-09-19T07:25:24.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 >> v1:172.21.15.83:6789/0 conn(0x7fd2680c28e0 legacy=0x7fd268150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:24.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 --2- 172.21.15.83:0/4098202534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2540039c0 0x7fd254003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:24.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 --2- 172.21.15.83:0/4098202534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd254004810 0x7fd254004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:24.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 --2- 172.21.15.83:0/4098202534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2540052f0 0x7fd25400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:24.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd268071b50 con 0x7fd254004810 2024-09-19T07:25:24.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd25400c1c0 con 0x7fd2540039c0 2024-09-19T07:25:24.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd25400c3c0 con 0x7fd2540052f0 2024-09-19T07:25:24.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26cedd640 1 --2- 172.21.15.83:0/4098202534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd254004810 0x7fd254004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:24.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26d6de640 1 --2- 172.21.15.83:0/4098202534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2540039c0 0x7fd254003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:24.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26dedf640 1 --2- 172.21.15.83:0/4098202534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2540052f0 0x7fd25400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:24.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26cedd640 1 -- 172.21.15.83:0/4098202534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2540052f0 msgr2=0x7fd25400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:24.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26cedd640 1 --2- 172.21.15.83:0/4098202534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2540052f0 0x7fd25400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:24.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26cedd640 1 -- 172.21.15.83:0/4098202534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2540039c0 msgr2=0x7fd254003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:24.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26cedd640 1 --2- 172.21.15.83:0/4098202534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2540039c0 0x7fd254003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:24.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26cedd640 1 -- 172.21.15.83:0/4098202534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd254003530 con 0x7fd254004810 2024-09-19T07:25:24.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26dedf640 1 --2- 172.21.15.83:0/4098202534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2540052f0 0x7fd25400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:24.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.977+0000 7fd26dedf640 1 -- 172.21.15.83:0/4098202534 reap_dead start 2024-09-19T07:25:24.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26cedd640 1 --2- 172.21.15.83:0/4098202534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd254004810 0x7fd254004c90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd26400a810 tx=0x7fd26401d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86371ff6014c7453 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:24.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26400a460 con 0x7fd254004810 2024-09-19T07:25:24.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2640ad650 con 0x7fd254004810 2024-09-19T07:25:24.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd25ffff640 1 -- 172.21.15.83:0/4098202534 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2640ada80 con 0x7fd254004810 2024-09-19T07:25:24.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 -- 172.21.15.83:0/4098202534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd254004810 msgr2=0x7fd254004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:24.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 --2- 172.21.15.83:0/4098202534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd254004810 0x7fd254004c90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd26400a810 tx=0x7fd26401d320 comp rx=0 tx=0).stop 2024-09-19T07:25:24.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 -- 172.21.15.83:0/4098202534 shutdown_connections 2024-09-19T07:25:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 --2- 172.21.15.83:0/4098202534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd254004810 0x7fd254004c90 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 -- 172.21.15.83:0/4098202534 >> 172.21.15.83:0/4098202534 conn(0x7fd26806bc80 msgr2=0x7fd26806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 -- 172.21.15.83:0/4098202534 shutdown_connections 2024-09-19T07:25:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 -- 172.21.15.83:0/4098202534 wait complete. 2024-09-19T07:25:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 Processor -- start 2024-09-19T07:25:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 -- start start 2024-09-19T07:25:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd254004810 0x7fd2680c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2680c1870 0x7fd2681efd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2681f0260 0x7fd26814e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd26806e280 con 0x7fd2680c1870 2024-09-19T07:25:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd2680716f0 con 0x7fd254004810 2024-09-19T07:25:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26f942640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd26806e400 con 0x7fd2681f0260 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26cedd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2680c1870 0x7fd2681efd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26cedd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2680c1870 0x7fd2681efd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45146/0 (socket says 172.21.15.83:45146) 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26cedd640 1 -- 172.21.15.83:0/1266885820 learned_addr learned my addr 172.21.15.83:0/1266885820 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26dedf640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2681f0260 0x7fd26814e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26d6de640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd254004810 0x7fd2680c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26cedd640 1 -- 172.21.15.83:0/1266885820 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2681f0260 msgr2=0x7fd26814e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26cedd640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2681f0260 0x7fd26814e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26cedd640 1 -- 172.21.15.83:0/1266885820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd254004810 msgr2=0x7fd2680c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26cedd640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd254004810 0x7fd2680c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.981+0000 7fd26cedd640 1 -- 172.21.15.83:0/1266885820 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2680c1d00 con 0x7fd2680c1870 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.985+0000 7fd26cedd640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2680c1870 0x7fd2681efd20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd264001d00 tx=0x7fd254005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.985+0000 7fd25dffb640 1 -- 172.21.15.83:0/1266885820 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2640bb070 con 0x7fd2680c1870 2024-09-19T07:25:24.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.985+0000 7fd25dffb640 1 -- 172.21.15.83:0/1266885820 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2640ad430 con 0x7fd2680c1870 2024-09-19T07:25:24.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.985+0000 7fd26f942640 1 -- 172.21.15.83:0/1266885820 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd26814ff60 con 0x7fd2680c1870 2024-09-19T07:25:24.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.985+0000 7fd25dffb640 1 -- 172.21.15.83:0/1266885820 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2640019e0 con 0x7fd2680c1870 2024-09-19T07:25:24.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.985+0000 7fd26f942640 1 -- 172.21.15.83:0/1266885820 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd268150520 con 0x7fd2680c1870 2024-09-19T07:25:24.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.985+0000 7fd26f942640 1 -- 172.21.15.83:0/1266885820 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd26806e790 con 0x7fd2680c1870 2024-09-19T07:25:24.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.985+0000 7fd25dffb640 1 -- 172.21.15.83:0/1266885820 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd2640be070 con 0x7fd2680c1870 2024-09-19T07:25:24.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.985+0000 7fd25dffb640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd24c0678c0 0x7fd24c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:24.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.985+0000 7fd25dffb640 1 -- 172.21.15.83:0/1266885820 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7fd264120360 con 0x7fd2680c1870 2024-09-19T07:25:24.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.985+0000 7fd26d6de640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd24c0678c0 0x7fd24c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:24.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.985+0000 7fd26d6de640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd24c0678c0 0x7fd24c069d80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd26814ad60 tx=0x7fd260008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:24.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:24.989+0000 7fd25dffb640 1 -- 172.21.15.83:0/1266885820 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd264017030 con 0x7fd2680c1870 2024-09-19T07:25:25.157 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:25.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 -- 172.21.15.83:0/1266885820 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2680c4f30 con 0x7fd2680c1870 2024-09-19T07:25:25.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd25dffb640 1 -- 172.21.15.83:0/1266885820 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7fd2640f08e0 con 0x7fd2680c1870 2024-09-19T07:25:25.188 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:25.189 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:25.189 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:25.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 -- 172.21.15.83:0/1266885820 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd24c0678c0 msgr2=0x7fd24c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:25.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd24c0678c0 0x7fd24c069d80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd26814ad60 tx=0x7fd260008040 comp rx=0 tx=0).stop 2024-09-19T07:25:25.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 -- 172.21.15.83:0/1266885820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2680c1870 msgr2=0x7fd2681efd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:25.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2680c1870 0x7fd2681efd20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd264001d00 tx=0x7fd254005840 comp rx=0 tx=0).stop 2024-09-19T07:25:25.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 -- 172.21.15.83:0/1266885820 shutdown_connections 2024-09-19T07:25:25.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd24c0678c0 0x7fd24c069d80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2681f0260 0x7fd26814e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2680c1870 0x7fd2681efd20 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 --2- 172.21.15.83:0/1266885820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd254004810 0x7fd2680c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 -- 172.21.15.83:0/1266885820 >> 172.21.15.83:0/1266885820 conn(0x7fd26806bc80 msgr2=0x7fd2681519d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:25.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 -- 172.21.15.83:0/1266885820 shutdown_connections 2024-09-19T07:25:25.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.189+0000 7fd26f942640 1 -- 172.21.15.83:0/1266885820 wait complete. 2024-09-19T07:25:25.206 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:25:25.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b9b0f9640 1 Processor -- start 2024-09-19T07:25:25.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b9b0f9640 1 -- start start 2024-09-19T07:25:25.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b9b0f9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b940ab750 0x7f1b940c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:25.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b9b0f9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1b940ab060 0x7f1b9415cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:25.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b9b0f9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b9406de90 con 0x7f1b940c2530 2024-09-19T07:25:25.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b9b0f9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f1b9406e170 con 0x7f1b940ab060 2024-09-19T07:25:25.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b9b0f9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1b9406e450 con 0x7f1b940ab750 2024-09-19T07:25:25.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b99696640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1b940ab060 0x7f1b9415cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:25.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b93fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1b940c2530 0x7f1b941528c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53120/0 (socket says 172.21.15.83:53120) 2024-09-19T07:25:25.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b93fff640 1 -- 172.21.15.83:0/3335584325 learned_addr learned my addr 172.21.15.83:0/3335584325 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:25.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b98e95640 1 --2- 172.21.15.83:0/3335584325 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b940ab750 0x7f1b940c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:25.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b99696640 1 -- 172.21.15.83:0/3335584325 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b940ab750 msgr2=0x7f1b940c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:25.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b99696640 1 --2- 172.21.15.83:0/3335584325 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b940ab750 0x7f1b940c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b99696640 1 -- 172.21.15.83:0/3335584325 >> v1:172.21.15.83:6789/0 conn(0x7f1b940c2530 legacy=0x7f1b941528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:25.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b937fe640 1 -- 172.21.15.83:0/3335584325 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4127592956 0 0) 0x7f1b9406de90 con 0x7f1b940c2530 2024-09-19T07:25:25.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.277+0000 7f1b99696640 1 -- 172.21.15.83:0/3335584325 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b9406e730 con 0x7f1b940ab060 2024-09-19T07:25:25.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b99696640 1 --2- 172.21.15.83:0/3335584325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b940ab060 0x7f1b9415cbc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009d90 tx=0x7f1b8c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a22cd9dfd47f831 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:25.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b937fe640 1 -- 172.21.15.83:0/3335584325 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c0b2070 con 0x7f1b940ab060 2024-09-19T07:25:25.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b937fe640 1 -- 172.21.15.83:0/3335584325 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b8c0a57a0 con 0x7f1b940ab060 2024-09-19T07:25:25.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b937fe640 1 -- 172.21.15.83:0/3335584325 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c0a5b70 con 0x7f1b940ab060 2024-09-19T07:25:25.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/3335584325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b940ab060 msgr2=0x7f1b9415cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:25.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b9b0f9640 1 --2- 172.21.15.83:0/3335584325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b940ab060 0x7f1b9415cbc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009d90 tx=0x7f1b8c01c8e0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/3335584325 shutdown_connections 2024-09-19T07:25:25.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b9b0f9640 1 --2- 172.21.15.83:0/3335584325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b940ab060 0x7f1b9415cbc0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b9b0f9640 1 --2- 172.21.15.83:0/3335584325 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b940ab750 0x7f1b940c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/3335584325 >> 172.21.15.83:0/3335584325 conn(0x7f1b9406bc80 msgr2=0x7f1b9406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:25.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/3335584325 shutdown_connections 2024-09-19T07:25:25.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/3335584325 wait complete. 2024-09-19T07:25:25.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b9b0f9640 1 Processor -- start 2024-09-19T07:25:25.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.281+0000 7f1b9b0f9640 1 -- start start 2024-09-19T07:25:25.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b9b0f9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b940ab060 0x7f1b941f0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:25.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b9b0f9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b940ab750 0x7f1b941ed7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:25.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b9b0f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b940c2530 0x7f1b9414dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:25.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b9b0f9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1b940abd40 con 0x7f1b940c2530 2024-09-19T07:25:25.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b9b0f9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9406ee60 con 0x7f1b940ab750 2024-09-19T07:25:25.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b9b0f9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1b9415f110 con 0x7f1b940ab060 2024-09-19T07:25:25.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b99696640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b940c2530 0x7f1b9414dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b99696640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b940c2530 0x7f1b9414dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45166/0 (socket says 172.21.15.83:45166) 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b99696640 1 -- 172.21.15.83:0/2338860488 learned_addr learned my addr 172.21.15.83:0/2338860488 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b98e95640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b940ab060 0x7f1b941f0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b99696640 1 -- 172.21.15.83:0/2338860488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b940ab060 msgr2=0x7f1b941f0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b99696640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b940ab060 0x7f1b941f0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b99696640 1 -- 172.21.15.83:0/2338860488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b940ab750 msgr2=0x7f1b941ed7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b99696640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b940ab750 0x7f1b941ed7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b99696640 1 -- 172.21.15.83:0/2338860488 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b941edd10 con 0x7f1b940c2530 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b98e95640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b940ab060 0x7f1b941f0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b99696640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b940c2530 0x7f1b9414dac0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0b0040 tx=0x7f1b8c019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b917fa640 1 -- 172.21.15.83:0/2338860488 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c0b2050 con 0x7f1b940c2530 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b917fa640 1 -- 172.21.15.83:0/2338860488 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b8c0b3040 con 0x7f1b940c2530 2024-09-19T07:25:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/2338860488 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b940c5e30 con 0x7f1b940c2530 2024-09-19T07:25:25.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b917fa640 1 -- 172.21.15.83:0/2338860488 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c0ad650 con 0x7f1b940c2530 2024-09-19T07:25:25.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/2338860488 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b941f06c0 con 0x7f1b940c2530 2024-09-19T07:25:25.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/2338860488 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b94073880 con 0x7f1b940c2530 2024-09-19T07:25:25.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b917fa640 1 -- 172.21.15.83:0/2338860488 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1b8c0c0020 con 0x7f1b940c2530 2024-09-19T07:25:25.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b917fa640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1b800678c0 0x7f1b80069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:25.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b917fa640 1 -- 172.21.15.83:0/2338860488 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7f1b8c120c60 con 0x7f1b940c2530 2024-09-19T07:25:25.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b98e95640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1b800678c0 0x7f1b80069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:25.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.285+0000 7f1b98e95640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1b800678c0 0x7f1b80069d80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009840 tx=0x7f1b84008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.289+0000 7f1b917fa640 1 -- 172.21.15.83:0/2338860488 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1b941edd10 con 0x7f1b940c2530 2024-09-19T07:25:25.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.485+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/2338860488 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b94068290 con 0x7f1b940c2530 2024-09-19T07:25:25.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.485+0000 7f1b917fa640 1 -- 172.21.15.83:0/2338860488 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7f1b8c016560 con 0x7f1b940c2530 2024-09-19T07:25:25.486 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:25.486 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:25.486 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:25.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.489+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/2338860488 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1b800678c0 msgr2=0x7f1b80069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:25.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.489+0000 7f1b9b0f9640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1b800678c0 0x7f1b80069d80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009840 tx=0x7f1b84008040 comp rx=0 tx=0).stop 2024-09-19T07:25:25.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.489+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/2338860488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b940c2530 msgr2=0x7f1b9414dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:25.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.489+0000 7f1b9b0f9640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b940c2530 0x7f1b9414dac0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0b0040 tx=0x7f1b8c019680 comp rx=0 tx=0).stop 2024-09-19T07:25:25.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.489+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/2338860488 shutdown_connections 2024-09-19T07:25:25.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.489+0000 7f1b9b0f9640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1b800678c0 0x7f1b80069d80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.489+0000 7f1b9b0f9640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b940c2530 0x7f1b9414dac0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.489+0000 7f1b9b0f9640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b940ab750 0x7f1b941ed7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.489+0000 7f1b9b0f9640 1 --2- 172.21.15.83:0/2338860488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b940ab060 0x7f1b941f0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:25.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.489+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/2338860488 >> 172.21.15.83:0/2338860488 conn(0x7f1b9406bc80 msgr2=0x7f1b941528a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:25.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.489+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/2338860488 shutdown_connections 2024-09-19T07:25:25.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:25.489+0000 7f1b9b0f9640 1 -- 172.21.15.83:0/2338860488 wait complete. 2024-09-19T07:25:25.503 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:25:27.207 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:27.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.333+0000 7fd7ac23d640 1 Processor -- start 2024-09-19T07:25:27.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.333+0000 7fd7ac23d640 1 -- start start 2024-09-19T07:25:27.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7ac23d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd7a4071100 0x7fd7a41545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:27.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7ac23d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd7a406dea0 0x7fd7a406e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:27.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7ac23d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7a40a9e60 con 0x7fd7a40c28e0 2024-09-19T07:25:27.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7ac23d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd7a4071b50 con 0x7fd7a4071100 2024-09-19T07:25:27.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7ac23d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd7a40c1ff0 con 0x7fd7a406dea0 2024-09-19T07:25:27.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7a9fd9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd7a4071100 0x7fd7a41545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:27.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7a9fd9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd7a4071100 0x7fd7a41545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45172/0 (socket says 172.21.15.83:45172) 2024-09-19T07:25:27.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7a9fd9640 1 -- 172.21.15.83:0/2298327586 learned_addr learned my addr 172.21.15.83:0/2298327586 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:27.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7a9fd9640 1 -- 172.21.15.83:0/2298327586 >> v2:172.21.15.184:3300/0 conn(0x7fd7a406dea0 msgr2=0x7fd7a406e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:27.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7a9fd9640 1 --2- 172.21.15.83:0/2298327586 >> v2:172.21.15.184:3300/0 conn(0x7fd7a406dea0 0x7fd7a406e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7a9fd9640 1 -- 172.21.15.83:0/2298327586 >> v1:172.21.15.184:6789/0 conn(0x7fd7a40c28e0 legacy=0x7fd7a4150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:27.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7a9fd9640 1 -- 172.21.15.83:0/2298327586 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a40aa140 con 0x7fd7a4071100 2024-09-19T07:25:27.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7a9fd9640 1 --2- 172.21.15.83:0/2298327586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7a4071100 0x7fd7a41545e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40c4f00 tx=0x7fd7a001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9736eaf2d0498a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:27.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7a8fd7640 1 -- 172.21.15.83:0/2298327586 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a00b0070 con 0x7fd7a4071100 2024-09-19T07:25:27.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7a8fd7640 1 -- 172.21.15.83:0/2298327586 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a00ab600 con 0x7fd7a4071100 2024-09-19T07:25:27.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7a8fd7640 1 -- 172.21.15.83:0/2298327586 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a00ab9d0 con 0x7fd7a4071100 2024-09-19T07:25:27.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2298327586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7a4071100 msgr2=0x7fd7a41545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:27.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7ac23d640 1 --2- 172.21.15.83:0/2298327586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7a4071100 0x7fd7a41545e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40c4f00 tx=0x7fd7a001c450 comp rx=0 tx=0).stop 2024-09-19T07:25:27.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2298327586 shutdown_connections 2024-09-19T07:25:27.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7ac23d640 1 --2- 172.21.15.83:0/2298327586 >> v2:172.21.15.184:3300/0 conn(0x7fd7a406dea0 0x7fd7a406e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.337+0000 7fd7ac23d640 1 --2- 172.21.15.83:0/2298327586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7a4071100 0x7fd7a41545e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2298327586 >> 172.21.15.83:0/2298327586 conn(0x7fd7a406bc80 msgr2=0x7fd7a406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:27.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2298327586 shutdown_connections 2024-09-19T07:25:27.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2298327586 wait complete. 2024-09-19T07:25:27.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 Processor -- start 2024-09-19T07:25:27.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 -- start start 2024-09-19T07:25:27.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7a406dea0 0x7fd7a40c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:27.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7a4071100 0x7fd7a4147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:27.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7a40c28e0 0x7fd7a4147b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:27.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a4071660 con 0x7fd7a4071100 2024-09-19T07:25:27.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a40714e0 con 0x7fd7a40c28e0 2024-09-19T07:25:27.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd7a4160c60 con 0x7fd7a406dea0 2024-09-19T07:25:27.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7a97d8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7a4071100 0x7fd7a4147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:27.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7a97d8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7a4071100 0x7fd7a4147160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45178/0 (socket says 172.21.15.83:45178) 2024-09-19T07:25:27.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7a97d8640 1 -- 172.21.15.83:0/2376924353 learned_addr learned my addr 172.21.15.83:0/2376924353 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:27.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7a97d8640 1 -- 172.21.15.83:0/2376924353 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7a406dea0 msgr2=0x7fd7a40c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:27.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7a97d8640 1 --2- 172.21.15.83:0/2376924353 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7a406dea0 0x7fd7a40c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7a97d8640 1 -- 172.21.15.83:0/2376924353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7a40c28e0 msgr2=0x7fd7a4147b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:27.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7a97d8640 1 --2- 172.21.15.83:0/2376924353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7a40c28e0 0x7fd7a4147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7a97d8640 1 -- 172.21.15.83:0/2376924353 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a41519d0 con 0x7fd7a4071100 2024-09-19T07:25:27.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7a97d8640 1 --2- 172.21.15.83:0/2376924353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7a4071100 0x7fd7a4147160 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd79400b3e0 tx=0x7fd79400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:27.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd79a7fc640 1 -- 172.21.15.83:0/2376924353 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794015020 con 0x7fd7a4071100 2024-09-19T07:25:27.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd79a7fc640 1 -- 172.21.15.83:0/2376924353 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd794011850 con 0x7fd7a4071100 2024-09-19T07:25:27.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2376924353 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a4151cb0 con 0x7fd7a4071100 2024-09-19T07:25:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd79a7fc640 1 -- 172.21.15.83:0/2376924353 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794011c40 con 0x7fd7a4071100 2024-09-19T07:25:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2376924353 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a41485d0 con 0x7fd7a4071100 2024-09-19T07:25:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.341+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2376924353 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7a406e330 con 0x7fd7a4071100 2024-09-19T07:25:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.345+0000 7fd79a7fc640 1 -- 172.21.15.83:0/2376924353 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd794018020 con 0x7fd7a4071100 2024-09-19T07:25:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.345+0000 7fd79a7fc640 1 --2- 172.21.15.83:0/2376924353 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7880678c0 0x7fd788069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.345+0000 7fd79a7fc640 1 -- 172.21.15.83:0/2376924353 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7fd794086230 con 0x7fd7a4071100 2024-09-19T07:25:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.345+0000 7fd7a9fd9640 1 --2- 172.21.15.83:0/2376924353 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7880678c0 0x7fd788069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.345+0000 7fd7a9fd9640 1 --2- 172.21.15.83:0/2376924353 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7880678c0 0x7fd788069d80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40c2890 tx=0x7fd7a0004740 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:27.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.349+0000 7fd79a7fc640 1 -- 172.21.15.83:0/2376924353 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd79401f020 con 0x7fd7a4071100 2024-09-19T07:25:27.504 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:27.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.549+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2376924353 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7a4151e80 con 0x7fd7a4071100 2024-09-19T07:25:27.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.549+0000 7fd79a7fc640 1 -- 172.21.15.83:0/2376924353 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7fd7940172c0 con 0x7fd7a4071100 2024-09-19T07:25:27.549 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:27.549 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:27.550 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:27.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.549+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2376924353 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7880678c0 msgr2=0x7fd788069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:27.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.549+0000 7fd7ac23d640 1 --2- 172.21.15.83:0/2376924353 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7880678c0 0x7fd788069d80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40c2890 tx=0x7fd7a0004740 comp rx=0 tx=0).stop 2024-09-19T07:25:27.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.549+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2376924353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7a4071100 msgr2=0x7fd7a4147160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:27.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.549+0000 7fd7ac23d640 1 --2- 172.21.15.83:0/2376924353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7a4071100 0x7fd7a4147160 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd79400b3e0 tx=0x7fd79400b8a0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.549+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2376924353 shutdown_connections 2024-09-19T07:25:27.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.553+0000 7fd7ac23d640 1 --2- 172.21.15.83:0/2376924353 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7880678c0 0x7fd788069d80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.553+0000 7fd7ac23d640 1 --2- 172.21.15.83:0/2376924353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7a40c28e0 0x7fd7a4147b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.553+0000 7fd7ac23d640 1 --2- 172.21.15.83:0/2376924353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7a4071100 0x7fd7a4147160 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.553+0000 7fd7ac23d640 1 --2- 172.21.15.83:0/2376924353 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7a406dea0 0x7fd7a40c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.553+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2376924353 >> 172.21.15.83:0/2376924353 conn(0x7fd7a406bc80 msgr2=0x7fd7a4073030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:27.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.553+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2376924353 shutdown_connections 2024-09-19T07:25:27.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.553+0000 7fd7ac23d640 1 -- 172.21.15.83:0/2376924353 wait complete. 2024-09-19T07:25:27.569 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:25:27.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.625+0000 7f8738c6b640 1 Processor -- start 2024-09-19T07:25:27.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.625+0000 7f8738c6b640 1 -- start start 2024-09-19T07:25:27.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f8738c6b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87340c6120 0x7f87340c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:27.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f8738c6b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87341592f0 0x7f8734157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:27.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f8738c6b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8734156200 con 0x7f8734155d70 2024-09-19T07:25:27.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f8738c6b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f87340c9930 con 0x7f87340c6120 2024-09-19T07:25:27.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f8738c6b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f87341564e0 con 0x7f87341592f0 2024-09-19T07:25:27.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f873259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87340c6120 0x7f87340c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:27.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f873259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87340c6120 0x7f87340c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45194/0 (socket says 172.21.15.83:45194) 2024-09-19T07:25:27.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f873259c640 1 -- 172.21.15.83:0/713533986 learned_addr learned my addr 172.21.15.83:0/713533986 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:27.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f8732d9d640 1 --2- 172.21.15.83:0/713533986 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87341592f0 0x7f8734157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:27.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f873259c640 1 -- 172.21.15.83:0/713533986 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87341592f0 msgr2=0x7f8734157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:27.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f873259c640 1 --2- 172.21.15.83:0/713533986 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87341592f0 0x7f8734157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f873259c640 1 -- 172.21.15.83:0/713533986 >> v1:172.21.15.184:6789/0 conn(0x7f8734155d70 legacy=0x7f87341575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:27.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f873259c640 1 -- 172.21.15.83:0/713533986 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87341567c0 con 0x7f87340c6120 2024-09-19T07:25:27.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f8732d9d640 1 --2- 172.21.15.83:0/713533986 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87341592f0 0x7f8734157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f873259c640 1 --2- 172.21.15.83:0/713533986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87340c6120 0x7f87340c6500 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8724009870 tx=0x7f872401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e8e57850c1c0fd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f873159a640 1 -- 172.21.15.83:0/713533986 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87240b0070 con 0x7f87340c6120 2024-09-19T07:25:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f873159a640 1 -- 172.21.15.83:0/713533986 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87240ab600 con 0x7f87340c6120 2024-09-19T07:25:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.629+0000 7f873159a640 1 -- 172.21.15.83:0/713533986 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87240ab9d0 con 0x7f87340c6120 2024-09-19T07:25:27.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 -- 172.21.15.83:0/713533986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87340c6120 msgr2=0x7f87340c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:27.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 --2- 172.21.15.83:0/713533986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87340c6120 0x7f87340c6500 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8724009870 tx=0x7f872401c520 comp rx=0 tx=0).stop 2024-09-19T07:25:27.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 -- 172.21.15.83:0/713533986 shutdown_connections 2024-09-19T07:25:27.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 --2- 172.21.15.83:0/713533986 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87341592f0 0x7f8734157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 --2- 172.21.15.83:0/713533986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87340c6120 0x7f87340c6500 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 -- 172.21.15.83:0/713533986 >> 172.21.15.83:0/713533986 conn(0x7f873406bc80 msgr2=0x7f873406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:27.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 -- 172.21.15.83:0/713533986 shutdown_connections 2024-09-19T07:25:27.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 -- 172.21.15.83:0/713533986 wait complete. 2024-09-19T07:25:27.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 Processor -- start 2024-09-19T07:25:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 -- start start 2024-09-19T07:25:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87340c6120 0x7f87340c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87341592f0 0x7f873415f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87340ac390 0x7f87340c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8734153860 con 0x7f87340c6120 2024-09-19T07:25:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8734162ea0 con 0x7f87341592f0 2024-09-19T07:25:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f87341ef0d0 con 0x7f87340ac390 2024-09-19T07:25:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f873259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87340c6120 0x7f87340c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f873259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87340c6120 0x7f87340c3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45208/0 (socket says 172.21.15.83:45208) 2024-09-19T07:25:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f873259c640 1 -- 172.21.15.83:0/3458758033 learned_addr learned my addr 172.21.15.83:0/3458758033 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f873259c640 1 -- 172.21.15.83:0/3458758033 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87340ac390 msgr2=0x7f87340c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f873259c640 1 --2- 172.21.15.83:0/3458758033 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87340ac390 0x7f87340c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f873259c640 1 -- 172.21.15.83:0/3458758033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87341592f0 msgr2=0x7f873415f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f873259c640 1 --2- 172.21.15.83:0/3458758033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87341592f0 0x7f873415f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f873259c640 1 -- 172.21.15.83:0/3458758033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87340c2ff0 con 0x7f87340c6120 2024-09-19T07:25:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f873259c640 1 --2- 172.21.15.83:0/3458758033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87340c6120 0x7f87340c3b00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f87240ae040 tx=0x7f872401ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8722ffd640 1 -- 172.21.15.83:0/3458758033 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87240ba020 con 0x7f87340c6120 2024-09-19T07:25:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8722ffd640 1 -- 172.21.15.83:0/3458758033 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87240b6640 con 0x7f87340c6120 2024-09-19T07:25:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 -- 172.21.15.83:0/3458758033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87341ef320 con 0x7f87340c6120 2024-09-19T07:25:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8722ffd640 1 -- 172.21.15.83:0/3458758033 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87240b0040 con 0x7f87340c6120 2024-09-19T07:25:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 -- 172.21.15.83:0/3458758033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87340c2c80 con 0x7f87340c6120 2024-09-19T07:25:27.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.633+0000 7f8738c6b640 1 -- 172.21.15.83:0/3458758033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87340c6500 con 0x7f87340c6120 2024-09-19T07:25:27.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.637+0000 7f8722ffd640 1 -- 172.21.15.83:0/3458758033 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f87240bd020 con 0x7f87340c6120 2024-09-19T07:25:27.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.637+0000 7f8722ffd640 1 --2- 172.21.15.83:0/3458758033 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87140678c0 0x7f8714069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:27.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.637+0000 7f8722ffd640 1 -- 172.21.15.83:0/3458758033 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7325+0+0 (secure 0 0 0) 0x7f8724120540 con 0x7f87340c6120 2024-09-19T07:25:27.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.637+0000 7f8731d9b640 1 --2- 172.21.15.83:0/3458758033 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87140678c0 0x7f8714069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:27.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.637+0000 7f8731d9b640 1 --2- 172.21.15.83:0/3458758033 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87140678c0 0x7f8714069d80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8728007fd0 tx=0x7f87280079a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:27.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.641+0000 7f8722ffd640 1 -- 172.21.15.83:0/3458758033 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f87240c4030 con 0x7f87340c6120 2024-09-19T07:25:27.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.829+0000 7f8738c6b640 1 -- 172.21.15.83:0/3458758033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87340c5ad0 con 0x7f87340c6120 2024-09-19T07:25:27.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8722ffd640 1 -- 172.21.15.83:0/3458758033 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6529 (secure 0 0 0) 0x7f8724016110 con 0x7f87340c6120 2024-09-19T07:25:27.832 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:27.832 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"btime":"2024-09-19T07:24:59:056411+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:23:55.188280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/26497755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":26497755},{"type":"v1","addr":"172.21.15.83:6837","nonce":26497755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5463,"qdb_cluster":[5463]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:27.833 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-09-19T07:25:27.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8738c6b640 1 -- 172.21.15.83:0/3458758033 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87140678c0 msgr2=0x7f8714069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:27.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8738c6b640 1 --2- 172.21.15.83:0/3458758033 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87140678c0 0x7f8714069d80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8728007fd0 tx=0x7f87280079a0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8738c6b640 1 -- 172.21.15.83:0/3458758033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87340c6120 msgr2=0x7f87340c3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:27.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8738c6b640 1 --2- 172.21.15.83:0/3458758033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87340c6120 0x7f87340c3b00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f87240ae040 tx=0x7f872401ad60 comp rx=0 tx=0).stop 2024-09-19T07:25:27.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8738c6b640 1 -- 172.21.15.83:0/3458758033 shutdown_connections 2024-09-19T07:25:27.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8738c6b640 1 --2- 172.21.15.83:0/3458758033 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87140678c0 0x7f8714069d80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8738c6b640 1 --2- 172.21.15.83:0/3458758033 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87340ac390 0x7f87340c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8738c6b640 1 --2- 172.21.15.83:0/3458758033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87341592f0 0x7f873415f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8738c6b640 1 --2- 172.21.15.83:0/3458758033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87340c6120 0x7f87340c3b00 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:27.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8738c6b640 1 -- 172.21.15.83:0/3458758033 >> 172.21.15.83:0/3458758033 conn(0x7f873406bc80 msgr2=0x7f87341549f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:27.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8738c6b640 1 -- 172.21.15.83:0/3458758033 shutdown_connections 2024-09-19T07:25:27.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:27.833+0000 7f8738c6b640 1 -- 172.21.15.83:0/3458758033 wait complete. 2024-09-19T07:25:27.849 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:25:29.570 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:29.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb1e621640 1 Processor -- start 2024-09-19T07:25:29.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb1e621640 1 -- start start 2024-09-19T07:25:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb1e621640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdb18071100 0x7fdb181545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb1e621640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fdb180c28e0 0x7fdb18150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb1e621640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb180a9e60 con 0x7fdb18151480 2024-09-19T07:25:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb1e621640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fdb18071b50 con 0x7fdb18071100 2024-09-19T07:25:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb1e621640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fdb180c1ff0 con 0x7fdb180c28e0 2024-09-19T07:25:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb17fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdb18071100 0x7fdb181545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb17fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdb18071100 0x7fdb181545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45212/0 (socket says 172.21.15.83:45212) 2024-09-19T07:25:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb17fff640 1 -- 172.21.15.83:0/2326406444 learned_addr learned my addr 172.21.15.83:0/2326406444 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb177fe640 1 --2- 172.21.15.83:0/2326406444 >> v2:172.21.15.184:3300/0 conn(0x7fdb180c28e0 0x7fdb18150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb17fff640 1 -- 172.21.15.83:0/2326406444 >> v2:172.21.15.184:3300/0 conn(0x7fdb180c28e0 msgr2=0x7fdb18150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb17fff640 1 --2- 172.21.15.83:0/2326406444 >> v2:172.21.15.184:3300/0 conn(0x7fdb180c28e0 0x7fdb18150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb17fff640 1 -- 172.21.15.83:0/2326406444 >> v1:172.21.15.184:6789/0 conn(0x7fdb18151480 legacy=0x7fdb1806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.701+0000 7fdb17fff640 1 -- 172.21.15.83:0/2326406444 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb180aa140 con 0x7fdb18071100 2024-09-19T07:25:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb17fff640 1 --2- 172.21.15.83:0/2326406444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb18071100 0x7fdb181545e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdb180c2890 tx=0x7fdb0801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35bdf06abaeefa5b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb16ffd640 1 -- 172.21.15.83:0/2326406444 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb080b0070 con 0x7fdb18071100 2024-09-19T07:25:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb16ffd640 1 -- 172.21.15.83:0/2326406444 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb080ab600 con 0x7fdb18071100 2024-09-19T07:25:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb16ffd640 1 -- 172.21.15.83:0/2326406444 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb080ab9d0 con 0x7fdb18071100 2024-09-19T07:25:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 -- 172.21.15.83:0/2326406444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb18071100 msgr2=0x7fdb181545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 --2- 172.21.15.83:0/2326406444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb18071100 0x7fdb181545e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdb180c2890 tx=0x7fdb0801c450 comp rx=0 tx=0).stop 2024-09-19T07:25:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 -- 172.21.15.83:0/2326406444 shutdown_connections 2024-09-19T07:25:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 --2- 172.21.15.83:0/2326406444 >> v2:172.21.15.184:3300/0 conn(0x7fdb180c28e0 0x7fdb18150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 --2- 172.21.15.83:0/2326406444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb18071100 0x7fdb181545e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 -- 172.21.15.83:0/2326406444 >> 172.21.15.83:0/2326406444 conn(0x7fdb1806bc80 msgr2=0x7fdb1806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 -- 172.21.15.83:0/2326406444 shutdown_connections 2024-09-19T07:25:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 -- 172.21.15.83:0/2326406444 wait complete. 2024-09-19T07:25:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 Processor -- start 2024-09-19T07:25:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 -- start start 2024-09-19T07:25:29.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb18071100 0x7fdb1814c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb180c28e0 0x7fdb18146df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb18151480 0x7fdb18151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb18071660 con 0x7fdb180c28e0 2024-09-19T07:25:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdb180714e0 con 0x7fdb18151480 2024-09-19T07:25:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdb18160c60 con 0x7fdb18071100 2024-09-19T07:25:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb177fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb180c28e0 0x7fdb18146df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb177fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb180c28e0 0x7fdb18146df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45228/0 (socket says 172.21.15.83:45228) 2024-09-19T07:25:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb177fe640 1 -- 172.21.15.83:0/780745565 learned_addr learned my addr 172.21.15.83:0/780745565 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb17fff640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb18071100 0x7fdb1814c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1cbbe640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb18151480 0x7fdb18151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb177fe640 1 -- 172.21.15.83:0/780745565 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb18071100 msgr2=0x7fdb1814c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb177fe640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb18071100 0x7fdb1814c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb177fe640 1 -- 172.21.15.83:0/780745565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb18151480 msgr2=0x7fdb18151ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:29.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb177fe640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb18151480 0x7fdb18151ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb177fe640 1 -- 172.21.15.83:0/780745565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1814d220 con 0x7fdb180c28e0 2024-09-19T07:25:29.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb177fe640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb180c28e0 0x7fdb18146df0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fdb0000d800 tx=0x7fdb0000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:29.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb14ff9640 1 -- 172.21.15.83:0/780745565 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0000b660 con 0x7fdb180c28e0 2024-09-19T07:25:29.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb14ff9640 1 -- 172.21.15.83:0/780745565 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb00019070 con 0x7fdb180c28e0 2024-09-19T07:25:29.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 -- 172.21.15.83:0/780745565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb181f0450 con 0x7fdb180c28e0 2024-09-19T07:25:29.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb14ff9640 1 -- 172.21.15.83:0/780745565 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0000ba00 con 0x7fdb180c28e0 2024-09-19T07:25:29.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.705+0000 7fdb1e621640 1 -- 172.21.15.83:0/780745565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb180c5490 con 0x7fdb180c28e0 2024-09-19T07:25:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.709+0000 7fdb1e621640 1 -- 172.21.15.83:0/780745565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb181545e0 con 0x7fdb180c28e0 2024-09-19T07:25:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.709+0000 7fdb14ff9640 1 -- 172.21.15.83:0/780745565 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdb0000bba0 con 0x7fdb180c28e0 2024-09-19T07:25:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.709+0000 7fdb14ff9640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdaf80678c0 0x7fdaf8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.709+0000 7fdb14ff9640 1 -- 172.21.15.83:0/780745565 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7411+0+0 (secure 0 0 0) 0x7fdb00086700 con 0x7fdb180c28e0 2024-09-19T07:25:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.709+0000 7fdb17fff640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdaf80678c0 0x7fdaf8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.709+0000 7fdb17fff640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdaf80678c0 0x7fdaf8069d80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdb180c4f00 tx=0x7fdb08008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:29.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.713+0000 7fdb14ff9640 1 -- 172.21.15.83:0/780745565 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdb00056c20 con 0x7fdb180c28e0 2024-09-19T07:25:29.849 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:29.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 -- 172.21.15.83:0/780745565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb180c4f30 con 0x7fdb180c28e0 2024-09-19T07:25:29.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb14ff9640 1 -- 172.21.15.83:0/780745565 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5706 (secure 0 0 0) 0x7fdb00056520 con 0x7fdb180c28e0 2024-09-19T07:25:29.898 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:29.898 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":55,"btime":"2024-09-19T07:25:29:146685+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:28.262937+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:29.898 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 55 2024-09-19T07:25:29.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 -- 172.21.15.83:0/780745565 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdaf80678c0 msgr2=0x7fdaf8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:29.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdaf80678c0 0x7fdaf8069d80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdb180c4f00 tx=0x7fdb08008040 comp rx=0 tx=0).stop 2024-09-19T07:25:29.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 -- 172.21.15.83:0/780745565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb180c28e0 msgr2=0x7fdb18146df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:29.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb180c28e0 0x7fdb18146df0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fdb0000d800 tx=0x7fdb0000dcc0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 -- 172.21.15.83:0/780745565 shutdown_connections 2024-09-19T07:25:29.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdaf80678c0 0x7fdaf8069d80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb18151480 0x7fdb18151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb180c28e0 0x7fdb18146df0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 --2- 172.21.15.83:0/780745565 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb18071100 0x7fdb1814c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 -- 172.21.15.83:0/780745565 >> 172.21.15.83:0/780745565 conn(0x7fdb1806bc80 msgr2=0x7fdb1806dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:29.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 -- 172.21.15.83:0/780745565 shutdown_connections 2024-09-19T07:25:29.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.897+0000 7fdb1e621640 1 -- 172.21.15.83:0/780745565 wait complete. 2024-09-19T07:25:29.916 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-19T07:25:29.916 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:25:29.916 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:29.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b99d3640 1 Processor -- start 2024-09-19T07:25:29.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b99d3640 1 -- start start 2024-09-19T07:25:29.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b99d3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60b40abda0 0x7f60b40ac180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:29.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b99d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60b40c2f50 0x7f60b4150c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:29.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b99d3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60b406d630 con 0x7f60b40ad150 2024-09-19T07:25:29.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b99d3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f60b4154ae0 con 0x7f60b40abda0 2024-09-19T07:25:29.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b99d3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f60b40acc00 con 0x7f60b40c2f50 2024-09-19T07:25:29.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60b40abda0 0x7f60b40ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:29.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60b40abda0 0x7f60b40ac180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45236/0 (socket says 172.21.15.83:45236) 2024-09-19T07:25:29.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b27fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f60b40ad150 0x7f60b40c2840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53128/0 (socket says 172.21.15.83:53128) 2024-09-19T07:25:29.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b2ffd640 1 -- 172.21.15.83:0/2866664140 learned_addr learned my addr 172.21.15.83:0/2866664140 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:29.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b37fe640 1 --2- 172.21.15.83:0/2866664140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60b40c2f50 0x7f60b4150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:29.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b2ffd640 1 -- 172.21.15.83:0/2866664140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60b40c2f50 msgr2=0x7f60b4150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:29.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b2ffd640 1 --2- 172.21.15.83:0/2866664140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60b40c2f50 0x7f60b4150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b2ffd640 1 -- 172.21.15.83:0/2866664140 >> v1:172.21.15.83:6789/0 conn(0x7f60b40ad150 legacy=0x7f60b40c2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:29.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b2ffd640 1 -- 172.21.15.83:0/2866664140 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b406e450 con 0x7f60b40abda0 2024-09-19T07:25:29.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.969+0000 7f60b37fe640 1 --2- 172.21.15.83:0/2866664140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60b40c2f50 0x7f60b4150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:29.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b2ffd640 1 --2- 172.21.15.83:0/2866664140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60b40abda0 0x7f60b40ac180 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f60b40acd80 tx=0x7f60ac01c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00c967254a3dcf7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:29.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b1ffb640 1 -- 172.21.15.83:0/2866664140 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0a5730 con 0x7f60b40abda0 2024-09-19T07:25:29.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b1ffb640 1 -- 172.21.15.83:0/2866664140 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60ac0a58d0 con 0x7f60b40abda0 2024-09-19T07:25:29.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b1ffb640 1 -- 172.21.15.83:0/2866664140 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0a5ca0 con 0x7f60b40abda0 2024-09-19T07:25:29.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 -- 172.21.15.83:0/2866664140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60b40abda0 msgr2=0x7f60b40ac180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:29.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 --2- 172.21.15.83:0/2866664140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60b40abda0 0x7f60b40ac180 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f60b40acd80 tx=0x7f60ac01c980 comp rx=0 tx=0).stop 2024-09-19T07:25:29.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 -- 172.21.15.83:0/2866664140 shutdown_connections 2024-09-19T07:25:29.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 --2- 172.21.15.83:0/2866664140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60b40c2f50 0x7f60b4150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 --2- 172.21.15.83:0/2866664140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60b40abda0 0x7f60b40ac180 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 -- 172.21.15.83:0/2866664140 >> 172.21.15.83:0/2866664140 conn(0x7f60b40c7e40 msgr2=0x7f60b40c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:29.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 -- 172.21.15.83:0/2866664140 shutdown_connections 2024-09-19T07:25:29.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 -- 172.21.15.83:0/2866664140 wait complete. 2024-09-19T07:25:29.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 Processor -- start 2024-09-19T07:25:29.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 -- start start 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60b40abda0 0x7f60b40718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60b40ad150 0x7f60b40c14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60b40c2f50 0x7f60b41479b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f60b40ac6c0 con 0x7f60b40ad150 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f60b406db10 con 0x7f60b40abda0 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.973+0000 7f60b99d3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f60b40ac840 con 0x7f60b40c2f50 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60b40ad150 0x7f60b40c14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60b40ad150 0x7f60b40c14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45248/0 (socket says 172.21.15.83:45248) 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b27fc640 1 -- 172.21.15.83:0/3475567423 learned_addr learned my addr 172.21.15.83:0/3475567423 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b37fe640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60b40c2f50 0x7f60b41479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b27fc640 1 -- 172.21.15.83:0/3475567423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60b40c2f50 msgr2=0x7f60b41479b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b27fc640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60b40c2f50 0x7f60b41479b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b2ffd640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60b40abda0 0x7f60b40718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b27fc640 1 -- 172.21.15.83:0/3475567423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60b40abda0 msgr2=0x7f60b40718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b27fc640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60b40abda0 0x7f60b40718b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b27fc640 1 -- 172.21.15.83:0/3475567423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b4151910 con 0x7f60b40ad150 2024-09-19T07:25:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b37fe640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60b40c2f50 0x7f60b41479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b27fc640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60b40ad150 0x7f60b40c14d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f609c00c420 tx=0x7f609c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60abfff640 1 -- 172.21.15.83:0/3475567423 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c0074e0 con 0x7f60b40ad150 2024-09-19T07:25:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60abfff640 1 -- 172.21.15.83:0/3475567423 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f609c007680 con 0x7f60b40ad150 2024-09-19T07:25:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60abfff640 1 -- 172.21.15.83:0/3475567423 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c004a30 con 0x7f60b40ad150 2024-09-19T07:25:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b99d3640 1 -- 172.21.15.83:0/3475567423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b4073370 con 0x7f60b40ad150 2024-09-19T07:25:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b99d3640 1 -- 172.21.15.83:0/3475567423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b4072600 con 0x7f60b40ad150 2024-09-19T07:25:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b99d3640 1 -- 172.21.15.83:0/3475567423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60b41515a0 con 0x7f60b40ad150 2024-09-19T07:25:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60abfff640 1 -- 172.21.15.83:0/3475567423 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f609c01a020 con 0x7f60b40ad150 2024-09-19T07:25:29.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60abfff640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60940678c0 0x7f6094069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:29.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60abfff640 1 -- 172.21.15.83:0/3475567423 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7411+0+0 (secure 0 0 0) 0x7f609c086240 con 0x7f60b40ad150 2024-09-19T07:25:29.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b2ffd640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60940678c0 0x7f6094069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:29.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.977+0000 7f60b2ffd640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60940678c0 0x7f6094069d80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0ae000 tx=0x7f60ac0a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:29.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:29.981+0000 7f60abfff640 1 -- 172.21.15.83:0/3475567423 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f609c021070 con 0x7f60b40ad150 2024-09-19T07:25:30.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb738e3c640 1 Processor -- start 2024-09-19T07:25:30.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb738e3c640 1 -- start start 2024-09-19T07:25:30.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb738e3c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb7340c3130 0x7fb7340ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:30.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb738e3c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb73406da00 0x7fb7340c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:30.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb738e3c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7340c4990 con 0x7fb7340c3a50 2024-09-19T07:25:30.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb738e3c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb7340c4c70 con 0x7fb7340c3130 2024-09-19T07:25:30.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb738e3c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb7340ab7b0 con 0x7fb73406da00 2024-09-19T07:25:30.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb732d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb7340c3a50 0x7fb7340c6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53136/0 (socket says 172.21.15.83:53136) 2024-09-19T07:25:30.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb73259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb7340c3130 0x7fb7340ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:30.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb732d9d640 1 -- 172.21.15.83:0/2872296217 learned_addr learned my addr 172.21.15.83:0/2872296217 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:30.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb731d9b640 1 --2- 172.21.15.83:0/2872296217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb73406da00 0x7fb7340c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:30.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb73259c640 1 -- 172.21.15.83:0/2872296217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb73406da00 msgr2=0x7fb7340c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:30.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb73259c640 1 --2- 172.21.15.83:0/2872296217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb73406da00 0x7fb7340c3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb73159a640 1 -- 172.21.15.83:0/2872296217 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 641375331 0 0) 0x7fb7340c4990 con 0x7fb7340c3a50 2024-09-19T07:25:30.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb73259c640 1 -- 172.21.15.83:0/2872296217 >> v1:172.21.15.83:6789/0 conn(0x7fb7340c3a50 legacy=0x7fb7340c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:30.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb73259c640 1 -- 172.21.15.83:0/2872296217 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7340aba90 con 0x7fb7340c3130 2024-09-19T07:25:30.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.021+0000 7fb731d9b640 1 --2- 172.21.15.83:0/2872296217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb73406da00 0x7fb7340c3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:30.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb73259c640 1 --2- 172.21.15.83:0/2872296217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7340c3130 0x7fb7340ac290 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb72c00bfa0 tx=0x7fb72c01eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66667de80c4a049 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:30.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb73159a640 1 -- 172.21.15.83:0/2872296217 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c00b2d0 con 0x7fb7340c3130 2024-09-19T07:25:30.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb73159a640 1 -- 172.21.15.83:0/2872296217 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb72c0a77b0 con 0x7fb7340c3130 2024-09-19T07:25:30.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb73159a640 1 -- 172.21.15.83:0/2872296217 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0a7b80 con 0x7fb7340c3130 2024-09-19T07:25:30.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 -- 172.21.15.83:0/2872296217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7340c3130 msgr2=0x7fb7340ac290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:30.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 --2- 172.21.15.83:0/2872296217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7340c3130 0x7fb7340ac290 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb72c00bfa0 tx=0x7fb72c01eae0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 -- 172.21.15.83:0/2872296217 shutdown_connections 2024-09-19T07:25:30.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 --2- 172.21.15.83:0/2872296217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb73406da00 0x7fb7340c3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 --2- 172.21.15.83:0/2872296217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7340c3130 0x7fb7340ac290 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 -- 172.21.15.83:0/2872296217 >> 172.21.15.83:0/2872296217 conn(0x7fb7340c5060 msgr2=0x7fb7340c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:30.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 -- 172.21.15.83:0/2872296217 shutdown_connections 2024-09-19T07:25:30.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 -- 172.21.15.83:0/2872296217 wait complete. 2024-09-19T07:25:30.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 Processor -- start 2024-09-19T07:25:30.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 -- start start 2024-09-19T07:25:30.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb73406da00 0x7fb73415be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:30.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7340c3130 0x7fb73414d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:30.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7340c3a50 0x7fb73414dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:30.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb734074f90 con 0x7fb73406da00 2024-09-19T07:25:30.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb7340ad730 con 0x7fb7340c3130 2024-09-19T07:25:30.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb738e3c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb734075110 con 0x7fb7340c3a50 2024-09-19T07:25:30.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb73259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb73406da00 0x7fb73415be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:30.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb73259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb73406da00 0x7fb73415be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45268/0 (socket says 172.21.15.83:45268) 2024-09-19T07:25:30.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.025+0000 7fb73259c640 1 -- 172.21.15.83:0/1996745022 learned_addr learned my addr 172.21.15.83:0/1996745022 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:30.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb732d9d640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7340c3a50 0x7fb73414dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:30.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb73259c640 1 -- 172.21.15.83:0/1996745022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7340c3a50 msgr2=0x7fb73414dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:30.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb731d9b640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7340c3130 0x7fb73414d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:30.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb73259c640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7340c3a50 0x7fb73414dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb73259c640 1 -- 172.21.15.83:0/1996745022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7340c3130 msgr2=0x7fb73414d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:30.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb73259c640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7340c3130 0x7fb73414d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb73259c640 1 -- 172.21.15.83:0/1996745022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7341ee530 con 0x7fb73406da00 2024-09-19T07:25:30.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb73259c640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb73406da00 0x7fb73415be40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0b1040 tx=0x7fb72c01b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:30.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb722ffd640 1 -- 172.21.15.83:0/1996745022 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c003ba0 con 0x7fb73406da00 2024-09-19T07:25:30.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb722ffd640 1 -- 172.21.15.83:0/1996745022 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb72c003d40 con 0x7fb73406da00 2024-09-19T07:25:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb722ffd640 1 -- 172.21.15.83:0/1996745022 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c004230 con 0x7fb73406da00 2024-09-19T07:25:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb738e3c640 1 -- 172.21.15.83:0/1996745022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7341ee810 con 0x7fb73406da00 2024-09-19T07:25:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb738e3c640 1 -- 172.21.15.83:0/1996745022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7341eedd0 con 0x7fb73406da00 2024-09-19T07:25:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb738e3c640 1 -- 172.21.15.83:0/1996745022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7340c3510 con 0x7fb73406da00 2024-09-19T07:25:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb722ffd640 1 -- 172.21.15.83:0/1996745022 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb72c0b6070 con 0x7fb73406da00 2024-09-19T07:25:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb722ffd640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb7140678c0 0x7fb714069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb722ffd640 1 -- 172.21.15.83:0/1996745022 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7411+0+0 (secure 0 0 0) 0x7fb72c121040 con 0x7fb73406da00 2024-09-19T07:25:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb731d9b640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb7140678c0 0x7fb714069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.029+0000 7fb731d9b640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb7140678c0 0x7fb714069d80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb728006fd0 tx=0x7fb728008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.033+0000 7fb722ffd640 1 -- 172.21.15.83:0/1996745022 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb72c018760 con 0x7fb73406da00 2024-09-19T07:25:30.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.185+0000 7f60b99d3640 1 -- 172.21.15.83:0/3475567423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60b4073540 con 0x7f60b40ad150 2024-09-19T07:25:30.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.185+0000 7f60abfff640 1 -- 172.21.15.83:0/3475567423 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+5704 (secure 0 0 0) 0x7f609c056840 con 0x7f60b40ad150 2024-09-19T07:25:30.186 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:30.186 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":56,"btime":"2024-09-19T07:25:30:156510+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:29.160751+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:30.186 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 56 2024-09-19T07:25:30.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.189+0000 7f60b99d3640 1 -- 172.21.15.83:0/3475567423 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60940678c0 msgr2=0x7f6094069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:30.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.189+0000 7f60b99d3640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60940678c0 0x7f6094069d80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0ae000 tx=0x7f60ac0a7070 comp rx=0 tx=0).stop 2024-09-19T07:25:30.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.189+0000 7f60b99d3640 1 -- 172.21.15.83:0/3475567423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60b40ad150 msgr2=0x7f60b40c14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:30.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.189+0000 7f60b99d3640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60b40ad150 0x7f60b40c14d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f609c00c420 tx=0x7f609c00c8e0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.189+0000 7f60b99d3640 1 -- 172.21.15.83:0/3475567423 shutdown_connections 2024-09-19T07:25:30.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.189+0000 7f60b99d3640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60940678c0 0x7f6094069d80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.189+0000 7f60b99d3640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60b40c2f50 0x7f60b41479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.189+0000 7f60b99d3640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60b40ad150 0x7f60b40c14d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.189+0000 7f60b99d3640 1 --2- 172.21.15.83:0/3475567423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60b40abda0 0x7f60b40718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.189+0000 7f60b99d3640 1 -- 172.21.15.83:0/3475567423 >> 172.21.15.83:0/3475567423 conn(0x7f60b40c7e40 msgr2=0x7f60b415dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:30.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.189+0000 7f60b99d3640 1 -- 172.21.15.83:0/3475567423 shutdown_connections 2024-09-19T07:25:30.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.189+0000 7f60b99d3640 1 -- 172.21.15.83:0/3475567423 wait complete. 2024-09-19T07:25:30.205 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:25:30.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.237+0000 7fb738e3c640 1 -- 172.21.15.83:0/1996745022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb73406fac0 con 0x7fb73406da00 2024-09-19T07:25:30.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.237+0000 7fb722ffd640 1 -- 172.21.15.83:0/1996745022 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+5704 (secure 0 0 0) 0x7fb72c0f1560 con 0x7fb73406da00 2024-09-19T07:25:30.238 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:30.238 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":56,"btime":"2024-09-19T07:25:30:156510+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:29.160751+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:30.238 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 56 2024-09-19T07:25:30.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.241+0000 7fb738e3c640 1 -- 172.21.15.83:0/1996745022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb7140678c0 msgr2=0x7fb714069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:30.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.241+0000 7fb738e3c640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb7140678c0 0x7fb714069d80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb728006fd0 tx=0x7fb728008040 comp rx=0 tx=0).stop 2024-09-19T07:25:30.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.241+0000 7fb738e3c640 1 -- 172.21.15.83:0/1996745022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb73406da00 msgr2=0x7fb73415be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:30.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.241+0000 7fb738e3c640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb73406da00 0x7fb73415be40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0b1040 tx=0x7fb72c01b880 comp rx=0 tx=0).stop 2024-09-19T07:25:30.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.241+0000 7fb738e3c640 1 -- 172.21.15.83:0/1996745022 shutdown_connections 2024-09-19T07:25:30.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.241+0000 7fb738e3c640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb7140678c0 0x7fb714069d80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.241+0000 7fb738e3c640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7340c3a50 0x7fb73414dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.241+0000 7fb738e3c640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7340c3130 0x7fb73414d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.241+0000 7fb738e3c640 1 --2- 172.21.15.83:0/1996745022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb73406da00 0x7fb73415be40 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:30.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.241+0000 7fb738e3c640 1 -- 172.21.15.83:0/1996745022 >> 172.21.15.83:0/1996745022 conn(0x7fb7340c5060 msgr2=0x7fb7340a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:30.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.241+0000 7fb738e3c640 1 -- 172.21.15.83:0/1996745022 shutdown_connections 2024-09-19T07:25:30.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:30.241+0000 7fb738e3c640 1 -- 172.21.15.83:0/1996745022 wait complete. 2024-09-19T07:25:30.257 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:25:30:156510+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 56, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 56, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5607": { "addr": "172.21.15.83:6835/3763485069", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 3763485069, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 3763485069, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5607, "incarnation": 54, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:25:29.160751+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5607 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 52, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5098": { "addr": "172.21.15.184:6835/144714632", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 144714632, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 144714632, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5098, "incarnation": 49, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:24:51.040252+0000", "qdb_cluster": [ 5098 ], "qdb_leader": 5098, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5098 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 28, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:25:31.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:25:31.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:25:32.206 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:32.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:32.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.352+0000 7f89280d3640 1 Processor -- start 2024-09-19T07:25:32.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.352+0000 7f89280d3640 1 -- start start 2024-09-19T07:25:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f89280d3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f89200aaa60 0x7f89201535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f89280d3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f892006f070 0x7f8920155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f89280d3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8920159c60 con 0x7f89200c2c30 2024-09-19T07:25:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f89280d3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f892015a160 con 0x7f89200aaa60 2024-09-19T07:25:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f89280d3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f89200c4940 con 0x7f892006f070 2024-09-19T07:25:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f892566e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f89200aaa60 0x7f89201535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f8925e6f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f89200c2c30 0x7f8920152e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43588/0 (socket says 172.21.15.83:43588) 2024-09-19T07:25:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f8925e6f640 1 -- 172.21.15.83:0/2412083086 learned_addr learned my addr 172.21.15.83:0/2412083086 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f8926670640 1 --2- 172.21.15.83:0/2412083086 >> v2:172.21.15.184:3300/0 conn(0x7f892006f070 0x7f8920155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f8924e6d640 1 -- 172.21.15.83:0/2412083086 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1484973895 0 0) 0x7f8920159c60 con 0x7f89200c2c30 2024-09-19T07:25:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f8924e6d640 1 -- 172.21.15.83:0/2412083086 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f890c003530 con 0x7f89200c2c30 2024-09-19T07:25:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f892566e640 1 -- 172.21.15.83:0/2412083086 >> v2:172.21.15.184:3300/0 conn(0x7f892006f070 msgr2=0x7f8920155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:32.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f892566e640 1 --2- 172.21.15.83:0/2412083086 >> v2:172.21.15.184:3300/0 conn(0x7f892006f070 0x7f8920155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f892566e640 1 -- 172.21.15.83:0/2412083086 >> v1:172.21.15.83:6789/0 conn(0x7f89200c2c30 legacy=0x7f8920152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:32.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f892566e640 1 -- 172.21.15.83:0/2412083086 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89200c4b90 con 0x7f89200aaa60 2024-09-19T07:25:32.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f892566e640 1 --2- 172.21.15.83:0/2412083086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89200aaa60 0x7f89201535a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f89200c4ac0 tx=0x7f891001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3cb76d676bd3c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:32.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.356+0000 7f8924e6d640 1 -- 172.21.15.83:0/2412083086 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89100b1070 con 0x7f89200aaa60 2024-09-19T07:25:32.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f8924e6d640 1 -- 172.21.15.83:0/2412083086 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89100ac620 con 0x7f89200aaa60 2024-09-19T07:25:32.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f8924e6d640 1 -- 172.21.15.83:0/2412083086 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89100aca10 con 0x7f89200aaa60 2024-09-19T07:25:32.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 -- 172.21.15.83:0/2412083086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89200aaa60 msgr2=0x7f89201535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:32.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 --2- 172.21.15.83:0/2412083086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89200aaa60 0x7f89201535a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f89200c4ac0 tx=0x7f891001c3c0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 -- 172.21.15.83:0/2412083086 shutdown_connections 2024-09-19T07:25:32.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 --2- 172.21.15.83:0/2412083086 >> v2:172.21.15.184:3300/0 conn(0x7f892006f070 0x7f8920155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 --2- 172.21.15.83:0/2412083086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89200aaa60 0x7f89201535a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 -- 172.21.15.83:0/2412083086 >> 172.21.15.83:0/2412083086 conn(0x7f89200c95a0 msgr2=0x7f89200c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:32.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 -- 172.21.15.83:0/2412083086 shutdown_connections 2024-09-19T07:25:32.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 -- 172.21.15.83:0/2412083086 wait complete. 2024-09-19T07:25:32.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 Processor -- start 2024-09-19T07:25:32.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 -- start start 2024-09-19T07:25:32.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f892006f070 0x7f89200c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:32.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89200aaa60 0x7f89200ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:32.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89200c2c30 0x7f89200c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:32.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89200c3010 con 0x7f89200aaa60 2024-09-19T07:25:32.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f89200c3ca0 con 0x7f89200c2c30 2024-09-19T07:25:32.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f89280d3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f89200c3190 con 0x7f892006f070 2024-09-19T07:25:32.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f892566e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89200aaa60 0x7f89200ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:32.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f892566e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89200aaa60 0x7f89200ae4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35112/0 (socket says 172.21.15.83:35112) 2024-09-19T07:25:32.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f892566e640 1 -- 172.21.15.83:0/3918273126 learned_addr learned my addr 172.21.15.83:0/3918273126 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:32.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f8926670640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89200c2c30 0x7f89200c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:32.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f8925e6f640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f892006f070 0x7f89200c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:32.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f892566e640 1 -- 172.21.15.83:0/3918273126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f892006f070 msgr2=0x7f89200c8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:32.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f892566e640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f892006f070 0x7f89200c8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f892566e640 1 -- 172.21.15.83:0/3918273126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89200c2c30 msgr2=0x7f89200c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:32.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f892566e640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89200c2c30 0x7f89200c7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f892566e640 1 -- 172.21.15.83:0/3918273126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89200c7990 con 0x7f89200aaa60 2024-09-19T07:25:32.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.360+0000 7f892566e640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89200aaa60 0x7f89200ae4b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f89100af040 tx=0x7f8910019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:32.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.364+0000 7f89167fc640 1 -- 172.21.15.83:0/3918273126 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89100b9070 con 0x7f89200aaa60 2024-09-19T07:25:32.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.364+0000 7f89280d3640 1 -- 172.21.15.83:0/3918273126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f892014db80 con 0x7f89200aaa60 2024-09-19T07:25:32.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.364+0000 7f89280d3640 1 -- 172.21.15.83:0/3918273126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89200ad2a0 con 0x7f89200aaa60 2024-09-19T07:25:32.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.364+0000 7f89167fc640 1 -- 172.21.15.83:0/3918273126 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89100ac620 con 0x7f89200aaa60 2024-09-19T07:25:32.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.364+0000 7f89167fc640 1 -- 172.21.15.83:0/3918273126 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89100c3060 con 0x7f89200aaa60 2024-09-19T07:25:32.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.364+0000 7f89280d3640 1 -- 172.21.15.83:0/3918273126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89080052a0 con 0x7f89200aaa60 2024-09-19T07:25:32.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.364+0000 7f89167fc640 1 -- 172.21.15.83:0/3918273126 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f89100bc070 con 0x7f89200aaa60 2024-09-19T07:25:32.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.364+0000 7f89167fc640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f89040678c0 0x7f8904069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:32.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.364+0000 7f89167fc640 1 -- 172.21.15.83:0/3918273126 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7411+0+0 (secure 0 0 0) 0x7f89101203d0 con 0x7f89200aaa60 2024-09-19T07:25:32.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.364+0000 7f8925e6f640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f89040678c0 0x7f8904069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:32.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.364+0000 7f8925e6f640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f89040678c0 0x7f8904069d80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f891c005890 tx=0x7f891c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.372+0000 7f89167fc640 1 -- 172.21.15.83:0/3918273126 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f8910016070 con 0x7f89200aaa60 2024-09-19T07:25:32.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.400+0000 7ffb6bd6e640 1 Processor -- start 2024-09-19T07:25:32.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.400+0000 7ffb6bd6e640 1 -- start start 2024-09-19T07:25:32.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.400+0000 7ffb6bd6e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ffb640bee30 0x7ffb641570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:32.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.400+0000 7ffb6bd6e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640c9680 0x7ffb64157630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:32.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.400+0000 7ffb6bd6e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb64157b70 con 0x7ffb640c6360 2024-09-19T07:25:32.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.400+0000 7ffb6bd6e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ffb640c5b90 con 0x7ffb640bee30 2024-09-19T07:25:32.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.400+0000 7ffb6bd6e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6415a050 con 0x7ffb640c9680 2024-09-19T07:25:32.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.400+0000 7ffb6a30b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640c9680 0x7ffb64157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:32.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.400+0000 7ffb69309640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ffb640bee30 0x7ffb641570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:32.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.400+0000 7ffb6a30b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640c9680 0x7ffb64157630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:54184/0 (socket says 172.21.15.83:54184) 2024-09-19T07:25:32.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.400+0000 7ffb6a30b640 1 -- 172.21.15.83:0/2699366609 learned_addr learned my addr 172.21.15.83:0/2699366609 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:32.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.400+0000 7ffb69b0a640 1 --1- 172.21.15.83:0/2699366609 >> v1:172.21.15.184:6789/0 conn(0x7ffb640c6360 0x7ffb64153050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:36584/0 (socket says 172.21.15.83:36584) 2024-09-19T07:25:32.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.404+0000 7ffb6a30b640 1 -- 172.21.15.83:0/2699366609 >> v2:172.21.15.184:3300/0 conn(0x7ffb640bee30 msgr2=0x7ffb641570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:32.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.404+0000 7ffb6a30b640 1 --2- 172.21.15.83:0/2699366609 >> v2:172.21.15.184:3300/0 conn(0x7ffb640bee30 0x7ffb641570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.404+0000 7ffb6a30b640 1 -- 172.21.15.83:0/2699366609 >> v1:172.21.15.184:6789/0 conn(0x7ffb640c6360 legacy=0x7ffb64153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:32.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.404+0000 7ffb6a30b640 1 -- 172.21.15.83:0/2699366609 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6406c950 con 0x7ffb640c9680 2024-09-19T07:25:32.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.404+0000 7ffb69309640 1 --2- 172.21.15.83:0/2699366609 >> v2:172.21.15.184:3300/0 conn(0x7ffb640bee30 0x7ffb641570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:25:32.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.404+0000 7ffb6a30b640 1 --2- 172.21.15.83:0/2699366609 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640c9680 0x7ffb64157630 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c009e50 tx=0x7ffb5c01c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a0c60a65d098000 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:32.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.404+0000 7ffb68b08640 1 -- 172.21.15.83:0/2699366609 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c0a5750 con 0x7ffb640c9680 2024-09-19T07:25:32.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.404+0000 7ffb68b08640 1 -- 172.21.15.83:0/2699366609 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb5c0a58f0 con 0x7ffb640c9680 2024-09-19T07:25:32.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.404+0000 7ffb68b08640 1 -- 172.21.15.83:0/2699366609 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c0a5cc0 con 0x7ffb640c9680 2024-09-19T07:25:32.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/2699366609 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640c9680 msgr2=0x7ffb64157630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:32.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 --2- 172.21.15.83:0/2699366609 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640c9680 0x7ffb64157630 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c009e50 tx=0x7ffb5c01c9a0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/2699366609 shutdown_connections 2024-09-19T07:25:32.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 --2- 172.21.15.83:0/2699366609 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640c9680 0x7ffb64157630 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 --2- 172.21.15.83:0/2699366609 >> v2:172.21.15.184:3300/0 conn(0x7ffb640bee30 0x7ffb641570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/2699366609 >> 172.21.15.83:0/2699366609 conn(0x7ffb640722a0 msgr2=0x7ffb640726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:32.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/2699366609 shutdown_connections 2024-09-19T07:25:32.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/2699366609 wait complete. 2024-09-19T07:25:32.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 Processor -- start 2024-09-19T07:25:32.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 -- start start 2024-09-19T07:25:32.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640bee30 0x7ffb641f1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:32.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb640c6360 0x7ffb641558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:32.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb640c9680 0x7ffb640ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:32.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffb64153760 con 0x7ffb640c6360 2024-09-19T07:25:32.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffb64159850 con 0x7ffb640c9680 2024-09-19T07:25:32.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ffb641538e0 con 0x7ffb640bee30 2024-09-19T07:25:32.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb69309640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb640c6360 0x7ffb641558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:32.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb69309640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb640c6360 0x7ffb641558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35128/0 (socket says 172.21.15.83:35128) 2024-09-19T07:25:32.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb69309640 1 -- 172.21.15.83:0/1728184539 learned_addr learned my addr 172.21.15.83:0/1728184539 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:32.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb69b0a640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640bee30 0x7ffb641f1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:32.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6a30b640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb640c9680 0x7ffb640ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:32.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb69309640 1 -- 172.21.15.83:0/1728184539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640bee30 msgr2=0x7ffb641f1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:32.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb69309640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640bee30 0x7ffb641f1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb69309640 1 -- 172.21.15.83:0/1728184539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb640c9680 msgr2=0x7ffb640ab820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:32.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb69309640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb640c9680 0x7ffb640ab820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb69309640 1 -- 172.21.15.83:0/1728184539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb641ee1f0 con 0x7ffb640c6360 2024-09-19T07:25:32.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb69b0a640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640bee30 0x7ffb641f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:25:32.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6a30b640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb640c9680 0x7ffb640ab820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:25:32.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb69309640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb640c6360 0x7ffb641558c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400e9b0 tx=0x7ffb5400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:32.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb5a7fc640 1 -- 172.21.15.83:0/1728184539 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54017070 con 0x7ffb640c6360 2024-09-19T07:25:32.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb5a7fc640 1 -- 172.21.15.83:0/1728184539 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb5400c9a0 con 0x7ffb640c6360 2024-09-19T07:25:32.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb5a7fc640 1 -- 172.21.15.83:0/1728184539 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5400cd90 con 0x7ffb640c6360 2024-09-19T07:25:32.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/1728184539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb641f17a0 con 0x7ffb640c6360 2024-09-19T07:25:32.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.408+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/1728184539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb640c25c0 con 0x7ffb640c6360 2024-09-19T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.412+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/1728184539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb640c67d0 con 0x7ffb640c6360 2024-09-19T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.412+0000 7ffb5a7fc640 1 -- 172.21.15.83:0/1728184539 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffb5401a070 con 0x7ffb640c6360 2024-09-19T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.412+0000 7ffb5a7fc640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ffb480678c0 0x7ffb48069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.412+0000 7ffb5a7fc640 1 -- 172.21.15.83:0/1728184539 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7411+0+0 (secure 0 0 0) 0x7ffb54086260 con 0x7ffb640c6360 2024-09-19T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.412+0000 7ffb69b0a640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ffb480678c0 0x7ffb48069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.412+0000 7ffb69b0a640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ffb480678c0 0x7ffb48069d80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ffb640bec00 tx=0x7ffb60007d90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:32.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.416+0000 7ffb5a7fc640 1 -- 172.21.15.83:0/1728184539 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ffb54056700 con 0x7ffb640c6360 2024-09-19T07:25:32.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 -- 172.21.15.83:0/3918273126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8908005860 con 0x7f89200aaa60 2024-09-19T07:25:32.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89167fc640 1 -- 172.21.15.83:0/3918273126 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5711 (secure 0 0 0) 0x7f89100f08b0 con 0x7f89200aaa60 2024-09-19T07:25:32.660 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:32.661 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":57,"btime":"2024-09-19T07:25:31:169387+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:32.661 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 57 2024-09-19T07:25:32.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 -- 172.21.15.83:0/3918273126 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f89040678c0 msgr2=0x7f8904069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:32.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f89040678c0 0x7f8904069d80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f891c005890 tx=0x7f891c009040 comp rx=0 tx=0).stop 2024-09-19T07:25:32.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 -- 172.21.15.83:0/3918273126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89200aaa60 msgr2=0x7f89200ae4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:32.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89200aaa60 0x7f89200ae4b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f89100af040 tx=0x7f8910019160 comp rx=0 tx=0).stop 2024-09-19T07:25:32.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 -- 172.21.15.83:0/3918273126 shutdown_connections 2024-09-19T07:25:32.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f89040678c0 0x7f8904069d80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89200c2c30 0x7f89200c7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89200aaa60 0x7f89200ae4b0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 --2- 172.21.15.83:0/3918273126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f892006f070 0x7f89200c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 -- 172.21.15.83:0/3918273126 >> 172.21.15.83:0/3918273126 conn(0x7f89200c95a0 msgr2=0x7f89200c6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:32.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 -- 172.21.15.83:0/3918273126 shutdown_connections 2024-09-19T07:25:32.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.660+0000 7f89280d3640 1 -- 172.21.15.83:0/3918273126 wait complete. 2024-09-19T07:25:32.684 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:25:32.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/1728184539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb64150df0 con 0x7ffb640c6360 2024-09-19T07:25:32.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb5a7fc640 1 -- 172.21.15.83:0/1728184539 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5711 (secure 0 0 0) 0x7ffb54056000 con 0x7ffb640c6360 2024-09-19T07:25:32.705 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:32.705 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":57,"btime":"2024-09-19T07:25:31:169387+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:32.705 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 57 2024-09-19T07:25:32.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/1728184539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ffb480678c0 msgr2=0x7ffb48069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:32.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ffb480678c0 0x7ffb48069d80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ffb640bec00 tx=0x7ffb60007d90 comp rx=0 tx=0).stop 2024-09-19T07:25:32.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/1728184539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb640c6360 msgr2=0x7ffb641558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:32.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb640c6360 0x7ffb641558c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400e9b0 tx=0x7ffb5400ee70 comp rx=0 tx=0).stop 2024-09-19T07:25:32.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/1728184539 shutdown_connections 2024-09-19T07:25:32.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ffb480678c0 0x7ffb48069d80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb640c9680 0x7ffb640ab820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb640c6360 0x7ffb641558c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 --2- 172.21.15.83:0/1728184539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffb640bee30 0x7ffb641f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:32.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/1728184539 >> 172.21.15.83:0/1728184539 conn(0x7ffb640722a0 msgr2=0x7ffb640caaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:32.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/1728184539 shutdown_connections 2024-09-19T07:25:32.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:32.704+0000 7ffb6bd6e640 1 -- 172.21.15.83:0/1728184539 wait complete. 2024-09-19T07:25:32.725 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=5463 2024-09-19T07:25:32.725 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.c 2024-09-19T07:25:34.067 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:25:34.064+0000 7f780fa50640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:25:34.685 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:34.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f6288875640 1 Processor -- start 2024-09-19T07:25:34.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f6288875640 1 -- start start 2024-09-19T07:25:34.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f6288875640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f62840c6360 0x7f6284153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:34.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f6288875640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f62840c9680 0x7f6284157800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:34.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f6288875640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62840c5b90 con 0x7f62840bee30 2024-09-19T07:25:34.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f6288875640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f628415a050 con 0x7f62840c6360 2024-09-19T07:25:34.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f6288875640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f628406c950 con 0x7f62840c9680 2024-09-19T07:25:34.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f628259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f62840c6360 0x7f6284153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:34.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f628259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f62840c6360 0x7f6284153050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35134/0 (socket says 172.21.15.83:35134) 2024-09-19T07:25:34.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f628259c640 1 -- 172.21.15.83:0/354044692 learned_addr learned my addr 172.21.15.83:0/354044692 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:34.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f6282d9d640 1 --2- 172.21.15.83:0/354044692 >> v2:172.21.15.184:3300/0 conn(0x7f62840c9680 0x7f6284157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:34.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f628259c640 1 -- 172.21.15.83:0/354044692 >> v2:172.21.15.184:3300/0 conn(0x7f62840c9680 msgr2=0x7f6284157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:34.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f628259c640 1 --2- 172.21.15.83:0/354044692 >> v2:172.21.15.184:3300/0 conn(0x7f62840c9680 0x7f6284157800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:34.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f628259c640 1 -- 172.21.15.83:0/354044692 >> v1:172.21.15.184:6789/0 conn(0x7f62840bee30 legacy=0x7f62841570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:34.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f628259c640 1 -- 172.21.15.83:0/354044692 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628406c670 con 0x7f62840c6360 2024-09-19T07:25:34.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.820+0000 7f6282d9d640 1 --2- 172.21.15.83:0/354044692 >> v2:172.21.15.184:3300/0 conn(0x7f62840c9680 0x7f6284157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:34.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.824+0000 7f628259c640 1 --2- 172.21.15.83:0/354044692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62840c6360 0x7f6284153050 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f628406cdf0 tx=0x7f627401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ecbc1726afe2cd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:34.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.824+0000 7f628159a640 1 -- 172.21.15.83:0/354044692 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62740b0070 con 0x7f62840c6360 2024-09-19T07:25:34.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.824+0000 7f628159a640 1 -- 172.21.15.83:0/354044692 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62740ab600 con 0x7f62840c6360 2024-09-19T07:25:34.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.824+0000 7f628159a640 1 -- 172.21.15.83:0/354044692 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62740ab9d0 con 0x7f62840c6360 2024-09-19T07:25:34.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.824+0000 7f6288875640 1 -- 172.21.15.83:0/354044692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62840c6360 msgr2=0x7f6284153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:34.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.824+0000 7f6288875640 1 --2- 172.21.15.83:0/354044692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62840c6360 0x7f6284153050 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f628406cdf0 tx=0x7f627401c3c0 comp rx=0 tx=0).stop 2024-09-19T07:25:34.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.824+0000 7f6288875640 1 -- 172.21.15.83:0/354044692 shutdown_connections 2024-09-19T07:25:34.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.824+0000 7f6288875640 1 --2- 172.21.15.83:0/354044692 >> v2:172.21.15.184:3300/0 conn(0x7f62840c9680 0x7f6284157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:34.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.824+0000 7f6288875640 1 --2- 172.21.15.83:0/354044692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62840c6360 0x7f6284153050 secure :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f628406cdf0 tx=0x7f627401c3c0 comp rx=0 tx=0).stop 2024-09-19T07:25:34.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.824+0000 7f6288875640 1 -- 172.21.15.83:0/354044692 >> 172.21.15.83:0/354044692 conn(0x7f62840722a0 msgr2=0x7f62840726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:34.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.824+0000 7f6288875640 1 -- 172.21.15.83:0/354044692 shutdown_connections 2024-09-19T07:25:34.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.824+0000 7f6288875640 1 -- 172.21.15.83:0/354044692 wait complete. 2024-09-19T07:25:34.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6288875640 1 Processor -- start 2024-09-19T07:25:34.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6288875640 1 -- start start 2024-09-19T07:25:34.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6288875640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62840bee30 0x7f62840cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:34.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6288875640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62840c9680 0x7f62841480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:34.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6288875640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f628414a780 0x7f6284148600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:34.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6288875640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f62840c68c0 con 0x7f62840bee30 2024-09-19T07:25:34.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6288875640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62840c6740 con 0x7f628414a780 2024-09-19T07:25:34.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6288875640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6284162da0 con 0x7f62840c9680 2024-09-19T07:25:34.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f628259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62840bee30 0x7f62840cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:34.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f628259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62840bee30 0x7f62840cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35138/0 (socket says 172.21.15.83:35138) 2024-09-19T07:25:34.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f628259c640 1 -- 172.21.15.83:0/3848451110 learned_addr learned my addr 172.21.15.83:0/3848451110 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:34.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6282d9d640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f628414a780 0x7f6284148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:34.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f628259c640 1 -- 172.21.15.83:0/3848451110 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62840c9680 msgr2=0x7f62841480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:34.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6281d9b640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62840c9680 0x7f62841480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:34.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f628259c640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62840c9680 0x7f62841480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:34.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f628259c640 1 -- 172.21.15.83:0/3848451110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f628414a780 msgr2=0x7f6284148600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:34.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f628259c640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f628414a780 0x7f6284148600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:34.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f628259c640 1 -- 172.21.15.83:0/3848451110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62840b1020 con 0x7f62840bee30 2024-09-19T07:25:34.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f628259c640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62840bee30 0x7f62840cb3e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f6274018fd0 tx=0x7f6274019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:34.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6272ffd640 1 -- 172.21.15.83:0/3848451110 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62740b9070 con 0x7f62840bee30 2024-09-19T07:25:34.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6272ffd640 1 -- 172.21.15.83:0/3848451110 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62740b4400 con 0x7f62840bee30 2024-09-19T07:25:34.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6288875640 1 -- 172.21.15.83:0/3848451110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62840b1300 con 0x7f62840bee30 2024-09-19T07:25:34.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6272ffd640 1 -- 172.21.15.83:0/3848451110 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62740b0040 con 0x7f62840bee30 2024-09-19T07:25:34.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6288875640 1 -- 172.21.15.83:0/3848451110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62840cb920 con 0x7f62840bee30 2024-09-19T07:25:34.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.828+0000 7f6288875640 1 -- 172.21.15.83:0/3848451110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6284153050 con 0x7f62840bee30 2024-09-19T07:25:34.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.832+0000 7f6272ffd640 1 -- 172.21.15.83:0/3848451110 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f62740bc070 con 0x7f62840bee30 2024-09-19T07:25:34.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.832+0000 7f6272ffd640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62640678c0 0x7f6264069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:34.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.832+0000 7f6272ffd640 1 -- 172.21.15.83:0/3848451110 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7411+0+0 (secure 0 0 0) 0x7f627411f510 con 0x7f62840bee30 2024-09-19T07:25:34.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.832+0000 7f6281d9b640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62640678c0 0x7f6264069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:34.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.832+0000 7f6281d9b640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62640678c0 0x7f6264069d80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6284146d90 tx=0x7f6278008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:34.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:34.836+0000 7f6272ffd640 1 -- 172.21.15.83:0/3848451110 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f62740160d0 con 0x7f62840bee30 2024-09-19T07:25:35.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.088+0000 7f6288875640 1 -- 172.21.15.83:0/3848451110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6284146dc0 con 0x7f62840bee30 2024-09-19T07:25:35.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.088+0000 7f6272ffd640 1 -- 172.21.15.83:0/3848451110 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5711 (secure 0 0 0) 0x7f62740efa30 con 0x7f62840bee30 2024-09-19T07:25:35.089 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:35.089 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":57,"btime":"2024-09-19T07:25:31:169387+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:35.090 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 57 2024-09-19T07:25:35.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.092+0000 7f6288875640 1 -- 172.21.15.83:0/3848451110 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62640678c0 msgr2=0x7f6264069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:35.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.092+0000 7f6288875640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62640678c0 0x7f6264069d80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6284146d90 tx=0x7f6278008040 comp rx=0 tx=0).stop 2024-09-19T07:25:35.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.092+0000 7f6288875640 1 -- 172.21.15.83:0/3848451110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62840bee30 msgr2=0x7f62840cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:35.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.092+0000 7f6288875640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62840bee30 0x7f62840cb3e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f6274018fd0 tx=0x7f6274019160 comp rx=0 tx=0).stop 2024-09-19T07:25:35.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.092+0000 7f6288875640 1 -- 172.21.15.83:0/3848451110 shutdown_connections 2024-09-19T07:25:35.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.092+0000 7f6288875640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62640678c0 0x7f6264069d80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:35.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.092+0000 7f6288875640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f628414a780 0x7f6284148600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:35.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.092+0000 7f6288875640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62840c9680 0x7f62841480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:35.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.092+0000 7f6288875640 1 --2- 172.21.15.83:0/3848451110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62840bee30 0x7f62840cb3e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:35.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.092+0000 7f6288875640 1 -- 172.21.15.83:0/3848451110 >> 172.21.15.83:0/3848451110 conn(0x7f62840722a0 msgr2=0x7f62840c8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:35.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.092+0000 7f6288875640 1 -- 172.21.15.83:0/3848451110 shutdown_connections 2024-09-19T07:25:35.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.092+0000 7f6288875640 1 -- 172.21.15.83:0/3848451110 wait complete. 2024-09-19T07:25:35.112 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:25:35.726 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-19T07:25:35.726 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:25:35.726 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:25:35.729 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:25:35.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:35.804 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:25:35.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377cd45640 1 Processor -- start 2024-09-19T07:25:35.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377cd45640 1 -- start start 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377cd45640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3778155e10 0x7f3778156280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377cd45640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3778159430 con 0x7f3778152f30 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377cd45640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3778159710 con 0x7f37780c4880 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377cd45640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f377815a040 con 0x7f3778155e10 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f3775d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3778152f30 0x7f37781533a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43612/0 (socket says 172.21.15.83:43612) 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f3775d9b640 1 -- 172.21.15.83:0/442206471 learned_addr learned my addr 172.21.15.83:0/442206471 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f3776d9d640 1 --2- 172.21.15.83:0/442206471 >> v2:172.21.15.184:3300/0 conn(0x7f3778155e10 0x7f3778156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2167274560 0 0) 0x7f3778159430 con 0x7f3778152f30 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3764003530 con 0x7f3778152f30 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 597570006 0 0) 0x7f3764003530 con 0x7f3778152f30 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 >> v2:172.21.15.184:3300/0 conn(0x7f3778155e10 msgr2=0x7f3778156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 --2- 172.21.15.83:0/442206471 >> v2:172.21.15.184:3300/0 conn(0x7f3778155e10 0x7f3778156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 >> v1:172.21.15.184:6789/0 conn(0x7f37780c4880 legacy=0x7f37780c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:35.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37781599f0 con 0x7f3778152f30 2024-09-19T07:25:35.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f3760001680 con 0x7f3778152f30 2024-09-19T07:25:35.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 >> v1:172.21.15.83:6789/0 conn(0x7f3778152f30 legacy=0x7f37781533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:35.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 --2- 172.21.15.83:0/442206471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37640039c0 0x7f3764003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:35.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 --2- 172.21.15.83:0/442206471 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3764004810 0x7f3764004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:35.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 --2- 172.21.15.83:0/442206471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37640052f0 0x7f376400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:35.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f377815a040 con 0x7f37640052f0 2024-09-19T07:25:35.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f376400c1c0 con 0x7f37640039c0 2024-09-19T07:25:35.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f376400c3c0 con 0x7f3764004810 2024-09-19T07:25:35.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f3776d9d640 1 --2- 172.21.15.83:0/442206471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37640052f0 0x7f376400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:35.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f377659c640 1 --2- 172.21.15.83:0/442206471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37640039c0 0x7f3764003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:35.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f3775d9b640 1 --2- 172.21.15.83:0/442206471 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3764004810 0x7f3764004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:35.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f3776d9d640 1 -- 172.21.15.83:0/442206471 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3764004810 msgr2=0x7f3764004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:35.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f3776d9d640 1 --2- 172.21.15.83:0/442206471 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3764004810 0x7f3764004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:35.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f3776d9d640 1 -- 172.21.15.83:0/442206471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37640039c0 msgr2=0x7f3764003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:35.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f3776d9d640 1 --2- 172.21.15.83:0/442206471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37640039c0 0x7f3764003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:35.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.860+0000 7f3776d9d640 1 -- 172.21.15.83:0/442206471 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3764003530 con 0x7f37640052f0 2024-09-19T07:25:35.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f3776d9d640 1 -- 172.21.15.83:0/442206471 reap_dead start 2024-09-19T07:25:35.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f3776d9d640 1 --2- 172.21.15.83:0/442206471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37640052f0 0x7f376400b9d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3770009f70 tx=0x7f377001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a8a965d973eb4c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37700b2070 con 0x7f37640052f0 2024-09-19T07:25:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37700a5c20 con 0x7f37640052f0 2024-09-19T07:25:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f377559a640 1 -- 172.21.15.83:0/442206471 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770009bf0 con 0x7f37640052f0 2024-09-19T07:25:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f377cd45640 1 -- 172.21.15.83:0/442206471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37640052f0 msgr2=0x7f376400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f377cd45640 1 --2- 172.21.15.83:0/442206471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37640052f0 0x7f376400b9d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3770009f70 tx=0x7f377001cc60 comp rx=0 tx=0).stop 2024-09-19T07:25:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f377cd45640 1 -- 172.21.15.83:0/442206471 shutdown_connections 2024-09-19T07:25:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f377cd45640 1 --2- 172.21.15.83:0/442206471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37640052f0 0x7f376400b9d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f377cd45640 1 -- 172.21.15.83:0/442206471 >> 172.21.15.83:0/442206471 conn(0x7f37780aa910 msgr2=0x7f37780c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f377cd45640 1 -- 172.21.15.83:0/442206471 shutdown_connections 2024-09-19T07:25:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f377cd45640 1 -- 172.21.15.83:0/442206471 wait complete. 2024-09-19T07:25:35.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f377cd45640 1 Processor -- start 2024-09-19T07:25:35.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.864+0000 7f377cd45640 1 -- start start 2024-09-19T07:25:35.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f377cd45640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37640052f0 0x7f37781ee080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:35.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f377cd45640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780ac4d0 0x7f37780c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:35.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f377cd45640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37780c7bd0 0x7f3778150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:35.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f377cd45640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3778159290 con 0x7f37780ac4d0 2024-09-19T07:25:35.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f377cd45640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f37780c4e70 con 0x7f37640052f0 2024-09-19T07:25:35.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f377cd45640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3778156220 con 0x7f37780c7bd0 2024-09-19T07:25:35.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f3775d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780ac4d0 0x7f37780c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:35.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f3775d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780ac4d0 0x7f37780c7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35174/0 (socket says 172.21.15.83:35174) 2024-09-19T07:25:35.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f3775d9b640 1 -- 172.21.15.83:0/1272989985 learned_addr learned my addr 172.21.15.83:0/1272989985 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:35.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f377659c640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37640052f0 0x7f37781ee080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:35.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f3776d9d640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37780c7bd0 0x7f3778150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:35.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f3775d9b640 1 -- 172.21.15.83:0/1272989985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37780c7bd0 msgr2=0x7f3778150a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:35.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f3775d9b640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37780c7bd0 0x7f3778150a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:35.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f3775d9b640 1 -- 172.21.15.83:0/1272989985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37640052f0 msgr2=0x7f37781ee080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:35.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f3775d9b640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37640052f0 0x7f37781ee080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:35.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f3775d9b640 1 -- 172.21.15.83:0/1272989985 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f377806f170 con 0x7f37780ac4d0 2024-09-19T07:25:35.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f377659c640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37640052f0 0x7f37781ee080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:25:35.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f3775d9b640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780ac4d0 0x7f37780c7690 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f37600010f0 tx=0x7f376000eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:35.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f376effd640 1 -- 172.21.15.83:0/1272989985 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376000c6c0 con 0x7f37780ac4d0 2024-09-19T07:25:35.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f376effd640 1 -- 172.21.15.83:0/1272989985 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3760018070 con 0x7f37780ac4d0 2024-09-19T07:25:35.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f377cd45640 1 -- 172.21.15.83:0/1272989985 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37781613f0 con 0x7f37780ac4d0 2024-09-19T07:25:35.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f376effd640 1 -- 172.21.15.83:0/1272989985 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376000cac0 con 0x7f37780ac4d0 2024-09-19T07:25:35.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f377cd45640 1 -- 172.21.15.83:0/1272989985 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37780c8260 con 0x7f37780ac4d0 2024-09-19T07:25:35.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f377cd45640 1 -- 172.21.15.83:0/1272989985 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f377806f5e0 con 0x7f37780ac4d0 2024-09-19T07:25:35.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f376effd640 1 -- 172.21.15.83:0/1272989985 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f376000cc60 con 0x7f37780ac4d0 2024-09-19T07:25:35.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.868+0000 7f376effd640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37580678f0 0x7f3758069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:35.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.872+0000 7f376effd640 1 -- 172.21.15.83:0/1272989985 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7411+0+0 (secure 0 0 0) 0x7f37600860e0 con 0x7f37780ac4d0 2024-09-19T07:25:35.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.872+0000 7f377659c640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37580678f0 0x7f3758069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:35.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.872+0000 7f377659c640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37580678f0 0x7f3758069db0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3768000c10 tx=0x7f3768009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:35.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:35.876+0000 7f376effd640 1 -- 172.21.15.83:0/1272989985 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3760056570 con 0x7f37780ac4d0 2024-09-19T07:25:36.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 -- 172.21.15.83:0/1272989985 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37781565b0 con 0x7f37780ac4d0 2024-09-19T07:25:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f376effd640 1 -- 172.21.15.83:0/1272989985 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5711 (secure 0 0 0) 0x7f3760055e70 con 0x7f37780ac4d0 2024-09-19T07:25:36.152 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:36.152 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":57,"btime":"2024-09-19T07:25:31:169387+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:24:51.040252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5098":{"gid":5098,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6835/144714632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":144714632},{"type":"v1","addr":"172.21.15.184:6835","nonce":144714632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5098,"qdb_cluster":[5098]},"id":2}]} 2024-09-19T07:25:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 57 2024-09-19T07:25:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 -- 172.21.15.83:0/1272989985 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37580678f0 msgr2=0x7f3758069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37580678f0 0x7f3758069db0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3768000c10 tx=0x7f3768009040 comp rx=0 tx=0).stop 2024-09-19T07:25:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 -- 172.21.15.83:0/1272989985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780ac4d0 msgr2=0x7f37780c7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780ac4d0 0x7f37780c7690 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f37600010f0 tx=0x7f376000eb60 comp rx=0 tx=0).stop 2024-09-19T07:25:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 -- 172.21.15.83:0/1272989985 shutdown_connections 2024-09-19T07:25:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37580678f0 0x7f3758069db0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37780c7bd0 0x7f3778150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780ac4d0 0x7f37780c7690 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 --2- 172.21.15.83:0/1272989985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37640052f0 0x7f37781ee080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 -- 172.21.15.83:0/1272989985 >> 172.21.15.83:0/1272989985 conn(0x7f37780aa910 msgr2=0x7f37780a9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 -- 172.21.15.83:0/1272989985 shutdown_connections 2024-09-19T07:25:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:36.152+0000 7f377cd45640 1 -- 172.21.15.83:0/1272989985 wait complete. 2024-09-19T07:25:36.174 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-19T07:25:37.113 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:37.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.228+0000 7f3ebcfc4640 1 Processor -- start 2024-09-19T07:25:37.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.228+0000 7f3ebcfc4640 1 -- start start 2024-09-19T07:25:37.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.228+0000 7f3ebcfc4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb81592f0 0x7f3eb80c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.228+0000 7f3ebcfc4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3eb8156200 con 0x7f3eb8155d70 2024-09-19T07:25:37.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.228+0000 7f3ebcfc4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3eb81564e0 con 0x7f3eb80c6120 2024-09-19T07:25:37.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.228+0000 7f3ebcfc4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3eb80c9930 con 0x7f3eb81592f0 2024-09-19T07:25:37.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.228+0000 7f3eb5d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3eb8155d70 0x7f3eb8157ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43616/0 (socket says 172.21.15.83:43616) 2024-09-19T07:25:37.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.228+0000 7f3eb5d9b640 1 -- 172.21.15.83:0/2152152099 learned_addr learned my addr 172.21.15.83:0/2152152099 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:37.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.228+0000 7f3eb6d9d640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb81592f0 0x7f3eb80c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:37.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2409681532 0 0) 0x7f3eb8156200 con 0x7f3eb8155d70 2024-09-19T07:25:37.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ea0003530 con 0x7f3eb8155d70 2024-09-19T07:25:37.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1112558062 0 0) 0x7f3ea0003530 con 0x7f3eb8155d70 2024-09-19T07:25:37.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb81592f0 msgr2=0x7f3eb80c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:37.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb81592f0 0x7f3eb80c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 >> v1:172.21.15.184:6789/0 conn(0x7f3eb80c6120 legacy=0x7f3eb81575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb81567c0 con 0x7f3eb8155d70 2024-09-19T07:25:37.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f3eac001680 con 0x7f3eb8155d70 2024-09-19T07:25:37.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb6d9d640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb81592f0 0x7f3eb80c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:25:37.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 >> v1:172.21.15.83:6789/0 conn(0x7f3eb8155d70 legacy=0x7f3eb8157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ea00039f0 0x7f3ea0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb81592f0 0x7f3ea0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ea0004d50 0x7f3ea00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea000ba80 con 0x7f3ea0004d50 2024-09-19T07:25:37.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea000bc30 con 0x7f3ea00039f0 2024-09-19T07:25:37.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3ea000be30 con 0x7f3eb81592f0 2024-09-19T07:25:37.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb6d9d640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ea0004d50 0x7f3ea00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:37.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb659c640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ea00039f0 0x7f3ea0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:37.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb5d9b640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb81592f0 0x7f3ea0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:37.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb6d9d640 1 -- 172.21.15.83:0/2152152099 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb81592f0 msgr2=0x7f3ea0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:37.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb6d9d640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb81592f0 0x7f3ea0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb6d9d640 1 -- 172.21.15.83:0/2152152099 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ea00039f0 msgr2=0x7f3ea0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:37.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb6d9d640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ea00039f0 0x7f3ea0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb6d9d640 1 -- 172.21.15.83:0/2152152099 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea0003530 con 0x7f3ea0004d50 2024-09-19T07:25:37.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb6d9d640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ea0004d50 0x7f3ea00051d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3eb000e350 tx=0x7f3eb001ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c266f9d0e39d3faa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:37.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00b2070 con 0x7f3ea0004d50 2024-09-19T07:25:37.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eb00a5e10 con 0x7f3ea0004d50 2024-09-19T07:25:37.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.232+0000 7f3eb559a640 1 -- 172.21.15.83:0/2152152099 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0009de0 con 0x7f3ea0004d50 2024-09-19T07:25:37.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/2152152099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ea0004d50 msgr2=0x7f3ea00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 --2- 172.21.15.83:0/2152152099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ea0004d50 0x7f3ea00051d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3eb000e350 tx=0x7f3eb001ce00 comp rx=0 tx=0).stop 2024-09-19T07:25:37.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3eb6d9d640 1 -- 172.21.15.83:0/2152152099 reap_dead start 2024-09-19T07:25:37.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/2152152099 shutdown_connections 2024-09-19T07:25:37.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/2152152099 >> 172.21.15.83:0/2152152099 conn(0x7f3eb806bc80 msgr2=0x7f3eb806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:37.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/2152152099 shutdown_connections 2024-09-19T07:25:37.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/2152152099 wait complete. 2024-09-19T07:25:37.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 Processor -- start 2024-09-19T07:25:37.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 -- start start 2024-09-19T07:25:37.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3eb81eef00 0x7f3eb80c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb80abcf0 0x7f3eb80ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3eb80adfb0 0x7f3eb814c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb8153860 con 0x7f3eb81eef00 2024-09-19T07:25:37.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb80cc630 con 0x7f3eb80adfb0 2024-09-19T07:25:37.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3eb80ad300 con 0x7f3eb80abcf0 2024-09-19T07:25:37.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3eb659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3eb81eef00 0x7f3eb80c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:37.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3eb659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3eb81eef00 0x7f3eb80c40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35186/0 (socket says 172.21.15.83:35186) 2024-09-19T07:25:37.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3eb659c640 1 -- 172.21.15.83:0/3600749409 learned_addr learned my addr 172.21.15.83:0/3600749409 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:37.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3eb659c640 1 -- 172.21.15.83:0/3600749409 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb80abcf0 msgr2=0x7f3eb80ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3eb659c640 1 --2- 172.21.15.83:0/3600749409 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb80abcf0 0x7f3eb80ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3eb659c640 1 -- 172.21.15.83:0/3600749409 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3eb80adfb0 msgr2=0x7f3eb814c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3eb659c640 1 --2- 172.21.15.83:0/3600749409 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3eb80adfb0 0x7f3eb814c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3eb659c640 1 -- 172.21.15.83:0/3600749409 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb80c3a30 con 0x7f3eb81eef00 2024-09-19T07:25:37.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3eb659c640 1 --2- 172.21.15.83:0/3600749409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3eb81eef00 0x7f3eb80c40c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800f7f0 tx=0x7f3ea800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:37.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ea6ffd640 1 -- 172.21.15.83:0/3600749409 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80050d0 con 0x7f3eb81eef00 2024-09-19T07:25:37.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ea6ffd640 1 -- 172.21.15.83:0/3600749409 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ea8018070 con 0x7f3eb81eef00 2024-09-19T07:25:37.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/3600749409 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb80aba10 con 0x7f3eb81eef00 2024-09-19T07:25:37.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ea6ffd640 1 -- 172.21.15.83:0/3600749409 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea800d6b0 con 0x7f3eb81eef00 2024-09-19T07:25:37.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.236+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/3600749409 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb81ef750 con 0x7f3eb81eef00 2024-09-19T07:25:37.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.240+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/3600749409 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3eb80c6500 con 0x7f3eb81eef00 2024-09-19T07:25:37.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.240+0000 7f3ea6ffd640 1 -- 172.21.15.83:0/3600749409 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3ea800d850 con 0x7f3eb81eef00 2024-09-19T07:25:37.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.240+0000 7f3ea6ffd640 1 --2- 172.21.15.83:0/3600749409 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e980678f0 0x7f3e98069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.240+0000 7f3ea6ffd640 1 -- 172.21.15.83:0/3600749409 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7f3ea80861e0 con 0x7f3eb81eef00 2024-09-19T07:25:37.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.240+0000 7f3eb5d9b640 1 --2- 172.21.15.83:0/3600749409 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e980678f0 0x7f3e98069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:37.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.240+0000 7f3eb5d9b640 1 --2- 172.21.15.83:0/3600749409 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e980678f0 0x7f3e98069db0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3eb80cbc90 tx=0x7f3eac001440 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:37.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.248+0000 7f3ea6ffd640 1 -- 172.21.15.83:0/3600749409 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3ea8056600 con 0x7f3eb81eef00 2024-09-19T07:25:37.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.468+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/3600749409 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3eb80c96a0 con 0x7f3eb81eef00 2024-09-19T07:25:37.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.468+0000 7f3ea6ffd640 1 -- 172.21.15.83:0/3600749409 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5703 (secure 0 0 0) 0x7f3ea8055f00 con 0x7f3eb81eef00 2024-09-19T07:25:37.470 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:37.470 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":60,"btime":"2024-09-19T07:25:37:208561+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:36.528033+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:25:37.471 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 60 2024-09-19T07:25:37.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.472+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/3600749409 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e980678f0 msgr2=0x7f3e98069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.472+0000 7f3ebcfc4640 1 --2- 172.21.15.83:0/3600749409 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e980678f0 0x7f3e98069db0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3eb80cbc90 tx=0x7f3eac001440 comp rx=0 tx=0).stop 2024-09-19T07:25:37.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.472+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/3600749409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3eb81eef00 msgr2=0x7f3eb80c40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.472+0000 7f3ebcfc4640 1 --2- 172.21.15.83:0/3600749409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3eb81eef00 0x7f3eb80c40c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800f7f0 tx=0x7f3ea800fcb0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.472+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/3600749409 shutdown_connections 2024-09-19T07:25:37.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.472+0000 7f3ebcfc4640 1 --2- 172.21.15.83:0/3600749409 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e980678f0 0x7f3e98069db0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.472+0000 7f3ebcfc4640 1 --2- 172.21.15.83:0/3600749409 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3eb80adfb0 0x7f3eb814c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.472+0000 7f3ebcfc4640 1 --2- 172.21.15.83:0/3600749409 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3eb80abcf0 0x7f3eb80ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.472+0000 7f3ebcfc4640 1 --2- 172.21.15.83:0/3600749409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3eb81eef00 0x7f3eb80c40c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.472+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/3600749409 >> 172.21.15.83:0/3600749409 conn(0x7f3eb806bc80 msgr2=0x7f3eb8152890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:37.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.472+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/3600749409 shutdown_connections 2024-09-19T07:25:37.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.472+0000 7f3ebcfc4640 1 -- 172.21.15.83:0/3600749409 wait complete. 2024-09-19T07:25:37.493 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-09-19T07:25:37.493 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:25:37.493 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:37.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.596+0000 7fec40f1e640 1 Processor -- start 2024-09-19T07:25:37.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.596+0000 7fec40f1e640 1 -- start start 2024-09-19T07:25:37.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec40f1e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fec3c155d70 0x7fec3c0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec40f1e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec3c1592f0 0x7fec3c157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec40f1e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec3c156200 con 0x7fec3c0c6120 2024-09-19T07:25:37.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec40f1e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fec3c0c9930 con 0x7fec3c155d70 2024-09-19T07:25:37.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec40f1e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fec3c1564e0 con 0x7fec3c1592f0 2024-09-19T07:25:37.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec39d9b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fec3c155d70 0x7fec3c0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:37.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec3ad9d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec3c1592f0 0x7fec3c157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:37.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec3ad9d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec3c1592f0 0x7fec3c157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:54284/0 (socket says 172.21.15.83:54284) 2024-09-19T07:25:37.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec3ad9d640 1 -- 172.21.15.83:0/954955186 learned_addr learned my addr 172.21.15.83:0/954955186 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:37.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec3ad9d640 1 -- 172.21.15.83:0/954955186 >> v2:172.21.15.184:3300/0 conn(0x7fec3c155d70 msgr2=0x7fec3c0c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:37.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec3ad9d640 1 --2- 172.21.15.83:0/954955186 >> v2:172.21.15.184:3300/0 conn(0x7fec3c155d70 0x7fec3c0c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec3ad9d640 1 -- 172.21.15.83:0/954955186 >> v1:172.21.15.184:6789/0 conn(0x7fec3c0c6120 legacy=0x7fec3c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec3ad9d640 1 -- 172.21.15.83:0/954955186 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3c1567c0 con 0x7fec3c1592f0 2024-09-19T07:25:37.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.600+0000 7fec39d9b640 1 --2- 172.21.15.83:0/954955186 >> v2:172.21.15.184:3300/0 conn(0x7fec3c155d70 0x7fec3c0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:25:37.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec3ad9d640 1 --2- 172.21.15.83:0/954955186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec3c1592f0 0x7fec3c157ce0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fec34009e50 tx=0x7fec3401ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f21750da6920f91f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:37.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec3959a640 1 -- 172.21.15.83:0/954955186 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec340a5700 con 0x7fec3c1592f0 2024-09-19T07:25:37.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec3959a640 1 -- 172.21.15.83:0/954955186 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec340a58a0 con 0x7fec3c1592f0 2024-09-19T07:25:37.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec3959a640 1 -- 172.21.15.83:0/954955186 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec340a5c70 con 0x7fec3c1592f0 2024-09-19T07:25:37.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec40f1e640 1 -- 172.21.15.83:0/954955186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec3c1592f0 msgr2=0x7fec3c157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec40f1e640 1 --2- 172.21.15.83:0/954955186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec3c1592f0 0x7fec3c157ce0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fec34009e50 tx=0x7fec3401ca30 comp rx=0 tx=0).stop 2024-09-19T07:25:37.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec40f1e640 1 -- 172.21.15.83:0/954955186 shutdown_connections 2024-09-19T07:25:37.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec40f1e640 1 --2- 172.21.15.83:0/954955186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec3c1592f0 0x7fec3c157ce0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec40f1e640 1 --2- 172.21.15.83:0/954955186 >> v2:172.21.15.184:3300/0 conn(0x7fec3c155d70 0x7fec3c0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec40f1e640 1 -- 172.21.15.83:0/954955186 >> 172.21.15.83:0/954955186 conn(0x7fec3c06bc80 msgr2=0x7fec3c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:37.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec40f1e640 1 -- 172.21.15.83:0/954955186 shutdown_connections 2024-09-19T07:25:37.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec40f1e640 1 -- 172.21.15.83:0/954955186 wait complete. 2024-09-19T07:25:37.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec40f1e640 1 Processor -- start 2024-09-19T07:25:37.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.604+0000 7fec40f1e640 1 -- start start 2024-09-19T07:25:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec40f1e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec3c0c6120 0x7fec3c0ac900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec40f1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec3c1592f0 0x7fec3c0ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec40f1e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec3c0caab0 0x7fec3c159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec40f1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c153860 con 0x7fec3c1592f0 2024-09-19T07:25:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec40f1e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c162ea0 con 0x7fec3c0c6120 2024-09-19T07:25:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec40f1e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fec3c0abe40 con 0x7fec3c0caab0 2024-09-19T07:25:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec39d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec3c1592f0 0x7fec3c0ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec39d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec3c1592f0 0x7fec3c0ab900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35198/0 (socket says 172.21.15.83:35198) 2024-09-19T07:25:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec39d9b640 1 -- 172.21.15.83:0/523433178 learned_addr learned my addr 172.21.15.83:0/523433178 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec3ad9d640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec3c0caab0 0x7fec3c159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec3a59c640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec3c0c6120 0x7fec3c0ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec39d9b640 1 -- 172.21.15.83:0/523433178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec3c0caab0 msgr2=0x7fec3c159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec39d9b640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec3c0caab0 0x7fec3c159a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec39d9b640 1 -- 172.21.15.83:0/523433178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec3c0c6120 msgr2=0x7fec3c0ac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec39d9b640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec3c0c6120 0x7fec3c0ac900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec39d9b640 1 -- 172.21.15.83:0/523433178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3c0c9f80 con 0x7fec3c1592f0 2024-09-19T07:25:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec3a59c640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec3c0c6120 0x7fec3c0ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec39d9b640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec3c1592f0 0x7fec3c0ab900 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fec3000ebe0 tx=0x7fec3000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec2affd640 1 -- 172.21.15.83:0/523433178 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec30017070 con 0x7fec3c1592f0 2024-09-19T07:25:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec2affd640 1 -- 172.21.15.83:0/523433178 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec3000cbd0 con 0x7fec3c1592f0 2024-09-19T07:25:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec2affd640 1 -- 172.21.15.83:0/523433178 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec30013470 con 0x7fec3c1592f0 2024-09-19T07:25:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec40f1e640 1 -- 172.21.15.83:0/523433178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec3c0ca260 con 0x7fec3c1592f0 2024-09-19T07:25:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec40f1e640 1 -- 172.21.15.83:0/523433178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec3c0ca820 con 0x7fec3c1592f0 2024-09-19T07:25:37.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec40f1e640 1 -- 172.21.15.83:0/523433178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec3c0c6500 con 0x7fec3c1592f0 2024-09-19T07:25:37.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec2affd640 1 -- 172.21.15.83:0/523433178 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fec3001a070 con 0x7fec3c1592f0 2024-09-19T07:25:37.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec2affd640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec1c0678f0 0x7fec1c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:37.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec2affd640 1 -- 172.21.15.83:0/523433178 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7fec300865e0 con 0x7fec3c1592f0 2024-09-19T07:25:37.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.608+0000 7fec3a59c640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec1c0678f0 0x7fec1c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:37.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.612+0000 7fec3a59c640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec1c0678f0 0x7fec1c069db0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fec2c000f70 tx=0x7fec2c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:37.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.616+0000 7fec2affd640 1 -- 172.21.15.83:0/523433178 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fec30056ab0 con 0x7fec3c1592f0 2024-09-19T07:25:37.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.836+0000 7fec40f1e640 1 -- 172.21.15.83:0/523433178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec3c0c5ad0 con 0x7fec3c1592f0 2024-09-19T07:25:37.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec2affd640 1 -- 172.21.15.83:0/523433178 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5703 (secure 0 0 0) 0x7fec300563b0 con 0x7fec3c1592f0 2024-09-19T07:25:37.840 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:37.840 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":60,"btime":"2024-09-19T07:25:37:208561+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:36.528033+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:25:37.841 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 60 2024-09-19T07:25:37.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec40f1e640 1 -- 172.21.15.83:0/523433178 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec1c0678f0 msgr2=0x7fec1c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec40f1e640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec1c0678f0 0x7fec1c069db0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fec2c000f70 tx=0x7fec2c009040 comp rx=0 tx=0).stop 2024-09-19T07:25:37.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec40f1e640 1 -- 172.21.15.83:0/523433178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec3c1592f0 msgr2=0x7fec3c0ab900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:37.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec40f1e640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec3c1592f0 0x7fec3c0ab900 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fec3000ebe0 tx=0x7fec3000c520 comp rx=0 tx=0).stop 2024-09-19T07:25:37.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec40f1e640 1 -- 172.21.15.83:0/523433178 shutdown_connections 2024-09-19T07:25:37.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec40f1e640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec1c0678f0 0x7fec1c069db0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec40f1e640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec3c0caab0 0x7fec3c159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec40f1e640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec3c1592f0 0x7fec3c0ab900 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec40f1e640 1 --2- 172.21.15.83:0/523433178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec3c0c6120 0x7fec3c0ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:37.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec40f1e640 1 -- 172.21.15.83:0/523433178 >> 172.21.15.83:0/523433178 conn(0x7fec3c06bc80 msgr2=0x7fec3c1550d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:37.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec40f1e640 1 -- 172.21.15.83:0/523433178 shutdown_connections 2024-09-19T07:25:37.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:37.840+0000 7fec40f1e640 1 -- 172.21.15.83:0/523433178 wait complete. 2024-09-19T07:25:37.859 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:25:37:208561+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 60, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 57, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5607": { "addr": "172.21.15.83:6835/3763485069", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 3763485069, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 3763485069, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5607, "incarnation": 54, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:25:31.169380+0000", "qdb_cluster": [ 5607 ], "qdb_leader": 5607, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5607 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 60, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5757": { "addr": "172.21.15.83:6837/455754420", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 455754420, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 455754420, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5757, "incarnation": 59, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:25:36.528033+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5757 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 28, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:25:38.175 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:38.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.292+0000 7f5750dee640 1 Processor -- start 2024-09-19T07:25:38.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.292+0000 7f5750dee640 1 -- start start 2024-09-19T07:25:38.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f5750dee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f574c153570 0x7f574c0c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:38.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f5750dee640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f574c0c4130 0x7f574c159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:38.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f5750dee640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f574c0c3c20 con 0x7f574c0c5110 2024-09-19T07:25:38.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f5750dee640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f574c0c89f0 con 0x7f574c0c4130 2024-09-19T07:25:38.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f5750dee640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f574c0c8cd0 con 0x7f574c153570 2024-09-19T07:25:38.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f574ad9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f574c0c4130 0x7f574c159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:38.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f574a59c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f574c153570 0x7f574c0c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:38.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f574ad9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f574c0c4130 0x7f574c159980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:41594/0 (socket says 172.21.15.83:41594) 2024-09-19T07:25:38.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f574ad9d640 1 -- 172.21.15.83:0/1782646252 learned_addr learned my addr 172.21.15.83:0/1782646252 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:38.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f574ad9d640 1 -- 172.21.15.83:0/1782646252 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f574c153570 msgr2=0x7f574c0c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:38.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f574ad9d640 1 --2- 172.21.15.83:0/1782646252 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f574c153570 0x7f574c0c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:38.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f574ad9d640 1 -- 172.21.15.83:0/1782646252 >> v1:172.21.15.184:6789/0 conn(0x7f574c0c5110 legacy=0x7f574c159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:38.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f574ad9d640 1 -- 172.21.15.83:0/1782646252 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574c0c8fb0 con 0x7f574c0c4130 2024-09-19T07:25:38.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f574a59c640 1 --2- 172.21.15.83:0/1782646252 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f574c153570 0x7f574c0c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:25:38.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.296+0000 7f574ad9d640 1 --2- 172.21.15.83:0/1782646252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f574c0c4130 0x7f574c159980 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5744009e80 tx=0x7f574401c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fd0e7be78fbd7a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:38.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574959a640 1 -- 172.21.15.83:0/1782646252 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57440a57d0 con 0x7f574c0c4130 2024-09-19T07:25:38.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574959a640 1 -- 172.21.15.83:0/1782646252 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57440a5970 con 0x7f574c0c4130 2024-09-19T07:25:38.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574959a640 1 -- 172.21.15.83:0/1782646252 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57440a5d40 con 0x7f574c0c4130 2024-09-19T07:25:38.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 -- 172.21.15.83:0/1782646252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f574c0c4130 msgr2=0x7f574c159980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:38.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 --2- 172.21.15.83:0/1782646252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f574c0c4130 0x7f574c159980 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5744009e80 tx=0x7f574401c8f0 comp rx=0 tx=0).stop 2024-09-19T07:25:38.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 -- 172.21.15.83:0/1782646252 shutdown_connections 2024-09-19T07:25:38.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 --2- 172.21.15.83:0/1782646252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f574c0c4130 0x7f574c159980 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:38.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 --2- 172.21.15.83:0/1782646252 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f574c153570 0x7f574c0c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:38.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 -- 172.21.15.83:0/1782646252 >> 172.21.15.83:0/1782646252 conn(0x7f574c0c6110 msgr2=0x7f574c0c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:38.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 -- 172.21.15.83:0/1782646252 shutdown_connections 2024-09-19T07:25:38.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 -- 172.21.15.83:0/1782646252 wait complete. 2024-09-19T07:25:38.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 Processor -- start 2024-09-19T07:25:38.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 -- start start 2024-09-19T07:25:38.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f574c0c4130 0x7f574c155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f574c0c5110 0x7f574c0aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574c153570 0x7f574c1f2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f574c163070 con 0x7f574c153570 2024-09-19T07:25:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f574c162ef0 con 0x7f574c0c4130 2024-09-19T07:25:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f574c1631f0 con 0x7f574c0c5110 2024-09-19T07:25:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574c153570 0x7f574c1f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574c153570 0x7f574c1f2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35202/0 (socket says 172.21.15.83:35202) 2024-09-19T07:25:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574ad9d640 1 -- 172.21.15.83:0/3226500385 learned_addr learned my addr 172.21.15.83:0/3226500385 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5749d9b640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f574c0c5110 0x7f574c0aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574a59c640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f574c0c4130 0x7f574c155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574ad9d640 1 -- 172.21.15.83:0/3226500385 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f574c0c5110 msgr2=0x7f574c0aacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574ad9d640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f574c0c5110 0x7f574c0aacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574ad9d640 1 -- 172.21.15.83:0/3226500385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f574c0c4130 msgr2=0x7f574c155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574ad9d640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f574c0c4130 0x7f574c155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574ad9d640 1 -- 172.21.15.83:0/3226500385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574c153a10 con 0x7f574c153570 2024-09-19T07:25:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f574ad9d640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574c153570 0x7f574c1f2410 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f574401a520 tx=0x7f5744019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f573affd640 1 -- 172.21.15.83:0/3226500385 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57440bc020 con 0x7f574c153570 2024-09-19T07:25:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f573affd640 1 -- 172.21.15.83:0/3226500385 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57440b8670 con 0x7f574c153570 2024-09-19T07:25:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f573affd640 1 -- 172.21.15.83:0/3226500385 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57440b8a60 con 0x7f574c153570 2024-09-19T07:25:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 -- 172.21.15.83:0/3226500385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f574c148380 con 0x7f574c153570 2024-09-19T07:25:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.300+0000 7f5750dee640 1 -- 172.21.15.83:0/3226500385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f574c0aa6f0 con 0x7f574c153570 2024-09-19T07:25:38.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.304+0000 7f5750dee640 1 -- 172.21.15.83:0/3226500385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f574c153be0 con 0x7f574c153570 2024-09-19T07:25:38.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.304+0000 7f573affd640 1 -- 172.21.15.83:0/3226500385 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f57440bf020 con 0x7f574c153570 2024-09-19T07:25:38.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.304+0000 7f573affd640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f572c0678c0 0x7f572c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:38.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.304+0000 7f574a59c640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f572c0678c0 0x7f572c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:38.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.304+0000 7f573affd640 1 -- 172.21.15.83:0/3226500385 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7f57441210d0 con 0x7f574c153570 2024-09-19T07:25:38.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.304+0000 7f574a59c640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f572c0678c0 0x7f572c069d80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f573c009580 tx=0x7f573c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.308+0000 7f573affd640 1 -- 172.21.15.83:0/3226500385 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f57440c6030 con 0x7f574c153570 2024-09-19T07:25:38.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.504+0000 7f5750dee640 1 -- 172.21.15.83:0/3226500385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f574c0c4bd0 con 0x7f574c153570 2024-09-19T07:25:38.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.508+0000 7f573affd640 1 -- 172.21.15.83:0/3226500385 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5700 (secure 0 0 0) 0x7f5744016600 con 0x7f574c153570 2024-09-19T07:25:38.508 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:38.508 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":61,"btime":"2024-09-19T07:25:38:216508+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:37.221845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:25:38.508 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 61 2024-09-19T07:25:38.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.508+0000 7f5750dee640 1 -- 172.21.15.83:0/3226500385 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f572c0678c0 msgr2=0x7f572c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:38.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.508+0000 7f5750dee640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f572c0678c0 0x7f572c069d80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f573c009580 tx=0x7f573c008040 comp rx=0 tx=0).stop 2024-09-19T07:25:38.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.508+0000 7f5750dee640 1 -- 172.21.15.83:0/3226500385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574c153570 msgr2=0x7f574c1f2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:38.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.508+0000 7f5750dee640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574c153570 0x7f574c1f2410 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f574401a520 tx=0x7f5744019690 comp rx=0 tx=0).stop 2024-09-19T07:25:38.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.508+0000 7f5750dee640 1 -- 172.21.15.83:0/3226500385 shutdown_connections 2024-09-19T07:25:38.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.508+0000 7f5750dee640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f572c0678c0 0x7f572c069d80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:38.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.508+0000 7f5750dee640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574c153570 0x7f574c1f2410 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:38.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.508+0000 7f5750dee640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f574c0c5110 0x7f574c0aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:38.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.508+0000 7f5750dee640 1 --2- 172.21.15.83:0/3226500385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f574c0c4130 0x7f574c155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:38.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.508+0000 7f5750dee640 1 -- 172.21.15.83:0/3226500385 >> 172.21.15.83:0/3226500385 conn(0x7f574c0c6110 msgr2=0x7f574c1518a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:38.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.512+0000 7f5750dee640 1 -- 172.21.15.83:0/3226500385 shutdown_connections 2024-09-19T07:25:38.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:38.512+0000 7f5750dee640 1 -- 172.21.15.83:0/3226500385 wait complete. 2024-09-19T07:25:38.528 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-19T07:25:39.860 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:39.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.984+0000 7f2b3aa8f640 1 Processor -- start 2024-09-19T07:25:39.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b3aa8f640 1 -- start start 2024-09-19T07:25:39.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b3aa8f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2b3414f9f0 0x7f2b340c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:39.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b3aa8f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b3406ed40 0x7f2b340c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:39.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b3aa8f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b34153820 con 0x7f2b3414fdd0 2024-09-19T07:25:39.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b3aa8f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f2b340c2890 con 0x7f2b3414f9f0 2024-09-19T07:25:39.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b3aa8f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2b340c2b70 con 0x7f2b3406ed40 2024-09-19T07:25:39.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b3882b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2b3414f9f0 0x7f2b340c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:39.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b33fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2b3414fdd0 0x7f2b34155b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43628/0 (socket says 172.21.15.83:43628) 2024-09-19T07:25:39.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b33fff640 1 -- 172.21.15.83:0/3038354845 learned_addr learned my addr 172.21.15.83:0/3038354845 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:39.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b3902c640 1 --2- 172.21.15.83:0/3038354845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b3406ed40 0x7f2b340c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:39.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b3882b640 1 -- 172.21.15.83:0/3038354845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b3406ed40 msgr2=0x7f2b340c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:39.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b3882b640 1 --2- 172.21.15.83:0/3038354845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b3406ed40 0x7f2b340c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:39.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b3882b640 1 -- 172.21.15.83:0/3038354845 >> v1:172.21.15.83:6789/0 conn(0x7f2b3414fdd0 legacy=0x7f2b34155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:39.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b337fe640 1 -- 172.21.15.83:0/3038354845 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3671311009 0 0) 0x7f2b20001120 con 0x7f2b3414fdd0 2024-09-19T07:25:39.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.988+0000 7f2b3882b640 1 -- 172.21.15.83:0/3038354845 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b34159220 con 0x7f2b3414f9f0 2024-09-19T07:25:39.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.992+0000 7f2b3882b640 1 --2- 172.21.15.83:0/3038354845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b3414f9f0 0x7f2b340c2190 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2b280097c0 tx=0x7f2b2801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c63507606e41011 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:39.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.992+0000 7f2b337fe640 1 -- 172.21.15.83:0/3038354845 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b280b0070 con 0x7f2b3414f9f0 2024-09-19T07:25:39.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.992+0000 7f2b337fe640 1 -- 172.21.15.83:0/3038354845 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b280ab600 con 0x7f2b3414f9f0 2024-09-19T07:25:39.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.992+0000 7f2b337fe640 1 -- 172.21.15.83:0/3038354845 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b280ab9d0 con 0x7f2b3414f9f0 2024-09-19T07:25:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.992+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/3038354845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b3414f9f0 msgr2=0x7f2b340c2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.992+0000 7f2b3aa8f640 1 --2- 172.21.15.83:0/3038354845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b3414f9f0 0x7f2b340c2190 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2b280097c0 tx=0x7f2b2801c410 comp rx=0 tx=0).stop 2024-09-19T07:25:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.992+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/3038354845 shutdown_connections 2024-09-19T07:25:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.992+0000 7f2b3aa8f640 1 --2- 172.21.15.83:0/3038354845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b3406ed40 0x7f2b340c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:39.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.992+0000 7f2b3aa8f640 1 --2- 172.21.15.83:0/3038354845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b3414f9f0 0x7f2b340c2190 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:39.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.992+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/3038354845 >> 172.21.15.83:0/3038354845 conn(0x7f2b340c5090 msgr2=0x7f2b340c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:39.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.992+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/3038354845 shutdown_connections 2024-09-19T07:25:39.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/3038354845 wait complete. 2024-09-19T07:25:39.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3aa8f640 1 Processor -- start 2024-09-19T07:25:39.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3aa8f640 1 -- start start 2024-09-19T07:25:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3aa8f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b3406ed40 0x7f2b34072360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3aa8f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b3414f9f0 0x7f2b341f05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3aa8f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b340ae440 0x7f2b341f0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3aa8f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2b34162eb0 con 0x7f2b3406ed40 2024-09-19T07:25:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3aa8f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2b34162d30 con 0x7f2b3414f9f0 2024-09-19T07:25:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3aa8f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2b34163030 con 0x7f2b340ae440 2024-09-19T07:25:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3882b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b3406ed40 0x7f2b34072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3882b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b3406ed40 0x7f2b34072360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35226/0 (socket says 172.21.15.83:35226) 2024-09-19T07:25:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3882b640 1 -- 172.21.15.83:0/665350533 learned_addr learned my addr 172.21.15.83:0/665350533 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3902c640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b340ae440 0x7f2b341f0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b33fff640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b3414f9f0 0x7f2b341f05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3882b640 1 -- 172.21.15.83:0/665350533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b340ae440 msgr2=0x7f2b341f0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:40.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3882b640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b340ae440 0x7f2b341f0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3882b640 1 -- 172.21.15.83:0/665350533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b3414f9f0 msgr2=0x7f2b341f05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:40.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3882b640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b3414f9f0 0x7f2b341f05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:39.996+0000 7f2b3882b640 1 -- 172.21.15.83:0/665350533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3414f220 con 0x7f2b3406ed40 2024-09-19T07:25:40.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.000+0000 7f2b3882b640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b3406ed40 0x7f2b34072360 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2b280ae040 tx=0x7f2b2801ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:40.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.000+0000 7f2b317fa640 1 -- 172.21.15.83:0/665350533 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b280bb070 con 0x7f2b3406ed40 2024-09-19T07:25:40.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.000+0000 7f2b317fa640 1 -- 172.21.15.83:0/665350533 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b28004020 con 0x7f2b3406ed40 2024-09-19T07:25:40.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.000+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/665350533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b34150260 con 0x7f2b3406ed40 2024-09-19T07:25:40.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.000+0000 7f2b317fa640 1 -- 172.21.15.83:0/665350533 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b280b0040 con 0x7f2b3406ed40 2024-09-19T07:25:40.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.000+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/665350533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b341f31e0 con 0x7f2b3406ed40 2024-09-19T07:25:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.000+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/665350533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b340c2f90 con 0x7f2b3406ed40 2024-09-19T07:25:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.000+0000 7f2b317fa640 1 -- 172.21.15.83:0/665350533 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2b28003620 con 0x7f2b3406ed40 2024-09-19T07:25:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.000+0000 7f2b317fa640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2b180678f0 0x7f2b18069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.000+0000 7f2b317fa640 1 -- 172.21.15.83:0/665350533 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7f2b28120750 con 0x7f2b3406ed40 2024-09-19T07:25:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.000+0000 7f2b33fff640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2b180678f0 0x7f2b18069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.000+0000 7f2b33fff640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2b180678f0 0x7f2b18069db0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2b20007fd0 tx=0x7f2b20007980 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:40.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.008+0000 7f2b317fa640 1 -- 172.21.15.83:0/665350533 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2b28016090 con 0x7f2b3406ed40 2024-09-19T07:25:40.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.236+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/665350533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b34158440 con 0x7f2b3406ed40 2024-09-19T07:25:40.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.236+0000 7f2b317fa640 1 -- 172.21.15.83:0/665350533 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+5707 (secure 0 0 0) 0x7f2b280f0c20 con 0x7f2b3406ed40 2024-09-19T07:25:40.239 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:40.239 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":62,"btime":"2024-09-19T07:25:39:224539+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:25:40.239 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 62 2024-09-19T07:25:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.240+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/665350533 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2b180678f0 msgr2=0x7f2b18069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.240+0000 7f2b3aa8f640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2b180678f0 0x7f2b18069db0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2b20007fd0 tx=0x7f2b20007980 comp rx=0 tx=0).stop 2024-09-19T07:25:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.240+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/665350533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b3406ed40 msgr2=0x7f2b34072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.240+0000 7f2b3aa8f640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b3406ed40 0x7f2b34072360 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2b280ae040 tx=0x7f2b2801ac50 comp rx=0 tx=0).stop 2024-09-19T07:25:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.240+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/665350533 shutdown_connections 2024-09-19T07:25:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.240+0000 7f2b3aa8f640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2b180678f0 0x7f2b18069db0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.240+0000 7f2b3aa8f640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b340ae440 0x7f2b341f0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.240+0000 7f2b3aa8f640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b3414f9f0 0x7f2b341f05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.240+0000 7f2b3aa8f640 1 --2- 172.21.15.83:0/665350533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b3406ed40 0x7f2b34072360 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.240+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/665350533 >> 172.21.15.83:0/665350533 conn(0x7f2b340c5090 msgr2=0x7f2b34073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:40.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.240+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/665350533 shutdown_connections 2024-09-19T07:25:40.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.240+0000 7f2b3aa8f640 1 -- 172.21.15.83:0/665350533 wait complete. 2024-09-19T07:25:40.264 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5098 2024-09-19T07:25:40.265 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.b 2024-09-19T07:25:40.529 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:40.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.668+0000 7f931a77f640 1 Processor -- start 2024-09-19T07:25:40.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.668+0000 7f931a77f640 1 -- start start 2024-09-19T07:25:40.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.668+0000 7f931a77f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9314150140 0x7f931406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:40.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.668+0000 7f931a77f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93140c4f90 con 0x7f931406f3b0 2024-09-19T07:25:40.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.668+0000 7f931a77f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93140c3840 con 0x7f93140c3c60 2024-09-19T07:25:40.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.668+0000 7f931a77f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f93140c34f0 con 0x7f9314150140 2024-09-19T07:25:40.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.668+0000 7f9318d1c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f931406f3b0 0x7f931415d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55154/0 (socket says 172.21.15.83:55154) 2024-09-19T07:25:40.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.668+0000 7f9318d1c640 1 -- 172.21.15.83:0/3268958402 learned_addr learned my addr 172.21.15.83:0/3268958402 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:40.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.668+0000 7f9313fff640 1 --2- 172.21.15.83:0/3268958402 >> v2:172.21.15.184:3300/0 conn(0x7f9314150140 0x7f931406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:40.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.668+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2595733927 0 0) 0x7f930c0014a0 con 0x7f931406f3b0 2024-09-19T07:25:40.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.668+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9300003530 con 0x7f931406f3b0 2024-09-19T07:25:40.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 372827799 0 0) 0x7f9300003530 con 0x7f931406f3b0 2024-09-19T07:25:40.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 >> v2:172.21.15.184:3300/0 conn(0x7f9314150140 msgr2=0x7f931406ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:40.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 --2- 172.21.15.83:0/3268958402 >> v2:172.21.15.184:3300/0 conn(0x7f9314150140 0x7f931406ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 >> v1:172.21.15.184:6789/0 conn(0x7f93140c3c60 legacy=0x7f9314155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:40.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93140c3210 con 0x7f931406f3b0 2024-09-19T07:25:40.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f930c001bd0 con 0x7f931406f3b0 2024-09-19T07:25:40.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 >> v1:172.21.15.83:6789/0 conn(0x7f931406f3b0 legacy=0x7f931415d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:40.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 --2- 172.21.15.83:0/3268958402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93000039c0 0x7f9300003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:40.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 --2- 172.21.15.83:0/3268958402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9300004810 0x7f9300004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:40.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 --2- 172.21.15.83:0/3268958402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93000052f0 0x7f930000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:40.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f93140c34f0 con 0x7f93000052f0 2024-09-19T07:25:40.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f930000c1c0 con 0x7f9300004810 2024-09-19T07:25:40.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f930000c3c0 con 0x7f93000039c0 2024-09-19T07:25:40.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f9318d1c640 1 --2- 172.21.15.83:0/3268958402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93000052f0 0x7f930000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:40.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f930b7fe640 1 --2- 172.21.15.83:0/3268958402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9300004810 0x7f9300004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:40.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f9318d1c640 1 -- 172.21.15.83:0/3268958402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93000039c0 msgr2=0x7f9300003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:40.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f9313fff640 1 --2- 172.21.15.83:0/3268958402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93000039c0 0x7f9300003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:40.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f9318d1c640 1 --2- 172.21.15.83:0/3268958402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93000039c0 0x7f9300003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f9318d1c640 1 -- 172.21.15.83:0/3268958402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9300004810 msgr2=0x7f9300004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:40.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f9318d1c640 1 --2- 172.21.15.83:0/3268958402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9300004810 0x7f9300004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f9318d1c640 1 -- 172.21.15.83:0/3268958402 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9300003530 con 0x7f93000052f0 2024-09-19T07:25:40.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f930b7fe640 1 --2- 172.21.15.83:0/3268958402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9300004810 0x7f9300004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:25:40.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f9318d1c640 1 -- 172.21.15.83:0/3268958402 reap_dead start 2024-09-19T07:25:40.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f9318d1c640 1 --2- 172.21.15.83:0/3268958402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93000052f0 0x7f930000b9d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f930c00aa00 tx=0x7f930c01d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3cf736bc20a775 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:40.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c00a650 con 0x7f93000052f0 2024-09-19T07:25:40.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f930c0ad650 con 0x7f93000052f0 2024-09-19T07:25:40.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f93137fe640 1 -- 172.21.15.83:0/3268958402 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c0ada40 con 0x7f93000052f0 2024-09-19T07:25:40.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f931a77f640 1 -- 172.21.15.83:0/3268958402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93000052f0 msgr2=0x7f930000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:40.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f931a77f640 1 --2- 172.21.15.83:0/3268958402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93000052f0 0x7f930000b9d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f930c00aa00 tx=0x7f930c01d340 comp rx=0 tx=0).stop 2024-09-19T07:25:40.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f931a77f640 1 -- 172.21.15.83:0/3268958402 shutdown_connections 2024-09-19T07:25:40.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f931a77f640 1 --2- 172.21.15.83:0/3268958402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93000052f0 0x7f930000b9d0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f931a77f640 1 -- 172.21.15.83:0/3268958402 >> 172.21.15.83:0/3268958402 conn(0x7f93140aa5f0 msgr2=0x7f93140aaa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:40.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f931a77f640 1 -- 172.21.15.83:0/3268958402 shutdown_connections 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.672+0000 7f931a77f640 1 -- 172.21.15.83:0/3268958402 wait complete. 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f931a77f640 1 Processor -- start 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f931a77f640 1 -- start start 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f931a77f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93000052f0 0x7f9314149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f931a77f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93140c1d00 0x7f9314148ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f931a77f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9314149220 0x7f93140ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f931a77f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f931406eff0 con 0x7f93140c1d00 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f931a77f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f931406ee70 con 0x7f9314149220 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f931a77f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f931406f170 con 0x7f93000052f0 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f930b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93140c1d00 0x7f9314148ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f930b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93140c1d00 0x7f9314148ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56666/0 (socket says 172.21.15.83:56666) 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f930b7fe640 1 -- 172.21.15.83:0/3115005786 learned_addr learned my addr 172.21.15.83:0/3115005786 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:40.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f9313fff640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93000052f0 0x7f9314149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f9318d1c640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9314149220 0x7f93140ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f930b7fe640 1 -- 172.21.15.83:0/3115005786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93000052f0 msgr2=0x7f9314149a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f930b7fe640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93000052f0 0x7f9314149a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f930b7fe640 1 -- 172.21.15.83:0/3115005786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9314149220 msgr2=0x7f93140ab820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f930b7fe640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9314149220 0x7f93140ab820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f930b7fe640 1 -- 172.21.15.83:0/3115005786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931414ee30 con 0x7f93140c1d00 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f930b7fe640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93140c1d00 0x7f9314148ce0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00d540 tx=0x7f92fc00da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f93117fa640 1 -- 172.21.15.83:0/3115005786 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc005480 con 0x7f93140c1d00 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f93117fa640 1 -- 172.21.15.83:0/3115005786 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92fc017070 con 0x7f93140c1d00 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f931a77f640 1 -- 172.21.15.83:0/3115005786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f931414f110 con 0x7f93140c1d00 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f93117fa640 1 -- 172.21.15.83:0/3115005786 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc003840 con 0x7f93140c1d00 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f931a77f640 1 -- 172.21.15.83:0/3115005786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f931414f6d0 con 0x7f93140c1d00 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.676+0000 7f931a77f640 1 -- 172.21.15.83:0/3115005786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9314155eb0 con 0x7f93140c1d00 2024-09-19T07:25:40.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.680+0000 7f93117fa640 1 -- 172.21.15.83:0/3115005786 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f92fc0039e0 con 0x7f93140c1d00 2024-09-19T07:25:40.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.680+0000 7f93117fa640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f92f40678c0 0x7f92f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:40.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.680+0000 7f93117fa640 1 -- 172.21.15.83:0/3115005786 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7f92fc086150 con 0x7f93140c1d00 2024-09-19T07:25:40.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.680+0000 7f9313fff640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f92f40678c0 0x7f92f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:40.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.680+0000 7f9313fff640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f92f40678c0 0x7f92f4069d80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9304009a30 tx=0x7f9304008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:40.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.684+0000 7f93117fa640 1 -- 172.21.15.83:0/3115005786 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f92fc019310 con 0x7f93140c1d00 2024-09-19T07:25:40.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.908+0000 7f931a77f640 1 -- 172.21.15.83:0/3115005786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93140aaad0 con 0x7f93140c1d00 2024-09-19T07:25:40.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f93117fa640 1 -- 172.21.15.83:0/3115005786 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+5707 (secure 0 0 0) 0x7f92fc056620 con 0x7f93140c1d00 2024-09-19T07:25:40.912 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:40.912 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":62,"btime":"2024-09-19T07:25:39:224539+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:25:40.912 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 62 2024-09-19T07:25:40.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f931a77f640 1 -- 172.21.15.83:0/3115005786 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f92f40678c0 msgr2=0x7f92f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:40.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f931a77f640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f92f40678c0 0x7f92f4069d80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9304009a30 tx=0x7f9304008040 comp rx=0 tx=0).stop 2024-09-19T07:25:40.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f931a77f640 1 -- 172.21.15.83:0/3115005786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93140c1d00 msgr2=0x7f9314148ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:40.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f931a77f640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93140c1d00 0x7f9314148ce0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00d540 tx=0x7f92fc00da00 comp rx=0 tx=0).stop 2024-09-19T07:25:40.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f931a77f640 1 -- 172.21.15.83:0/3115005786 shutdown_connections 2024-09-19T07:25:40.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f931a77f640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f92f40678c0 0x7f92f4069d80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f931a77f640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9314149220 0x7f93140ab820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f931a77f640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93140c1d00 0x7f9314148ce0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f931a77f640 1 --2- 172.21.15.83:0/3115005786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93000052f0 0x7f9314149a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:40.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f931a77f640 1 -- 172.21.15.83:0/3115005786 >> 172.21.15.83:0/3115005786 conn(0x7f93140aa5f0 msgr2=0x7f931414cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:40.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f931a77f640 1 -- 172.21.15.83:0/3115005786 shutdown_connections 2024-09-19T07:25:40.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:40.912+0000 7f931a77f640 1 -- 172.21.15.83:0/3115005786 wait complete. 2024-09-19T07:25:40.930 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:active state 2024-09-19T07:25:40.930 INFO:tasks.mds_thrash.fs.[a]:waiting for 80 secs before thrashing 2024-09-19T07:25:41.205 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:25:41.204+0000 7ff81b2f8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:25:48.265 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-09-19T07:25:48.420 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-19T07:25:48.420 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-19T07:25:48.423 INFO:tasks.ceph.mds.b:Started 2024-09-19T07:25:48.423 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:48.510 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-19T07:25:48.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.564+0000 7f4852e3d640 1 Processor -- start 2024-09-19T07:25:48.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.564+0000 7f4852e3d640 1 -- start start 2024-09-19T07:25:48.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.564+0000 7f4852e3d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f484c0aa4a0 0x7f484c157720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:48.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.564+0000 7f4852e3d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f484c073d90 con 0x7f484c0c3ab0 2024-09-19T07:25:48.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.564+0000 7f4852e3d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f484c153b30 con 0x7f484c14dbe0 2024-09-19T07:25:48.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.564+0000 7f4852e3d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f484c154700 con 0x7f484c0aa4a0 2024-09-19T07:25:48.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.564+0000 7f48513da640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f484c0aa4a0 0x7f484c157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:48.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.564+0000 7f484bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f484c0c3ab0 0x7f484c157010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55162/0 (socket says 172.21.15.83:55162) 2024-09-19T07:25:48.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.564+0000 7f484bfff640 1 -- 172.21.15.83:0/1719818729 learned_addr learned my addr 172.21.15.83:0/1719818729 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:48.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.568+0000 7f48513da640 1 -- 172.21.15.83:0/1719818729 >> v1:172.21.15.184:6789/0 conn(0x7f484c14dbe0 legacy=0x7f484c0c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:48.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.568+0000 7f48513da640 1 -- 172.21.15.83:0/1719818729 >> v1:172.21.15.83:6789/0 conn(0x7f484c0c3ab0 legacy=0x7f484c157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:48.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.568+0000 7f484b7fe640 1 -- 172.21.15.83:0/1719818729 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3210552327 0 0) 0x7f484c073d90 con 0x7f484c0c3ab0 2024-09-19T07:25:48.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.568+0000 7f48513da640 1 -- 172.21.15.83:0/1719818729 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484c153dd0 con 0x7f484c0aa4a0 2024-09-19T07:25:48.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.568+0000 7f48513da640 1 --2- 172.21.15.83:0/1719818729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f484c0aa4a0 0x7f484c157720 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f484c154880 tx=0x7f484401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e804dc6f8e84f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:48.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.568+0000 7f484b7fe640 1 -- 172.21.15.83:0/1719818729 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48440b2070 con 0x7f484c0aa4a0 2024-09-19T07:25:48.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.568+0000 7f484b7fe640 1 -- 172.21.15.83:0/1719818729 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48440a5890 con 0x7f484c0aa4a0 2024-09-19T07:25:48.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.568+0000 7f484b7fe640 1 -- 172.21.15.83:0/1719818729 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48440a5c60 con 0x7f484c0aa4a0 2024-09-19T07:25:48.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.568+0000 7f4852e3d640 1 -- 172.21.15.83:0/1719818729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f484c0aa4a0 msgr2=0x7f484c157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:48.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.568+0000 7f4852e3d640 1 --2- 172.21.15.83:0/1719818729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f484c0aa4a0 0x7f484c157720 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f484c154880 tx=0x7f484401c8c0 comp rx=0 tx=0).stop 2024-09-19T07:25:48.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 -- 172.21.15.83:0/1719818729 shutdown_connections 2024-09-19T07:25:48.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 --2- 172.21.15.83:0/1719818729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f484c0aa4a0 0x7f484c157720 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:48.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 -- 172.21.15.83:0/1719818729 >> 172.21.15.83:0/1719818729 conn(0x7f484c0c95a0 msgr2=0x7f484c0c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:48.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 -- 172.21.15.83:0/1719818729 shutdown_connections 2024-09-19T07:25:48.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 -- 172.21.15.83:0/1719818729 wait complete. 2024-09-19T07:25:48.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 Processor -- start 2024-09-19T07:25:48.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 -- start start 2024-09-19T07:25:48.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f484c0aa4a0 0x7f484c0ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:48.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f484c14dbe0 0x7f484c143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:48.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f484c147a00 0x7f484c143eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:48.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f484c0c46b0 con 0x7f484c14dbe0 2024-09-19T07:25:48.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f484c0741a0 con 0x7f484c0aa4a0 2024-09-19T07:25:48.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f484c0c4830 con 0x7f484c147a00 2024-09-19T07:25:48.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f484bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f484c14dbe0 0x7f484c143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:48.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f484bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f484c14dbe0 0x7f484c143970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56684/0 (socket says 172.21.15.83:56684) 2024-09-19T07:25:48.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f484bfff640 1 -- 172.21.15.83:0/1237555254 learned_addr learned my addr 172.21.15.83:0/1237555254 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:48.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4850bd9640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f484c0aa4a0 0x7f484c0ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:48.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f48513da640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f484c147a00 0x7f484c143eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:48.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f484bfff640 1 -- 172.21.15.83:0/1237555254 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f484c147a00 msgr2=0x7f484c143eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:48.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f484bfff640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f484c147a00 0x7f484c143eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:48.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f484bfff640 1 -- 172.21.15.83:0/1237555254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f484c0aa4a0 msgr2=0x7f484c0ada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:48.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f484bfff640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f484c0aa4a0 0x7f484c0ada60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:48.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f484bfff640 1 -- 172.21.15.83:0/1237555254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484c0af260 con 0x7f484c14dbe0 2024-09-19T07:25:48.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f484bfff640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f484c14dbe0 0x7f484c143970 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f483800c4a0 tx=0x7f483800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:48.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f48497fa640 1 -- 172.21.15.83:0/1237555254 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4838018070 con 0x7f484c14dbe0 2024-09-19T07:25:48.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f48497fa640 1 -- 172.21.15.83:0/1237555254 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48380074e0 con 0x7f484c14dbe0 2024-09-19T07:25:48.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 -- 172.21.15.83:0/1237555254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f484c147e70 con 0x7f484c14dbe0 2024-09-19T07:25:48.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f48497fa640 1 -- 172.21.15.83:0/1237555254 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4838004a30 con 0x7f484c14dbe0 2024-09-19T07:25:48.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.572+0000 7f4852e3d640 1 -- 172.21.15.83:0/1237555254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f484c14a410 con 0x7f484c14dbe0 2024-09-19T07:25:48.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.576+0000 7f4852e3d640 1 -- 172.21.15.83:0/1237555254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f484c14a610 con 0x7f484c14dbe0 2024-09-19T07:25:48.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.576+0000 7f48497fa640 1 -- 172.21.15.83:0/1237555254 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f483801b070 con 0x7f484c14dbe0 2024-09-19T07:25:48.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.576+0000 7f48497fa640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f48300678f0 0x7f4830069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:48.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.576+0000 7f48497fa640 1 -- 172.21.15.83:0/1237555254 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7f48380857b0 con 0x7f484c14dbe0 2024-09-19T07:25:48.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.576+0000 7f4850bd9640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f48300678f0 0x7f4830069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:48.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.576+0000 7f4850bd9640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f48300678f0 0x7f4830069db0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f484c14cb90 tx=0x7f4840009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:48.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.580+0000 7f48497fa640 1 -- 172.21.15.83:0/1237555254 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4838055710 con 0x7f484c14dbe0 2024-09-19T07:25:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.800+0000 7f4852e3d640 1 -- 172.21.15.83:0/1237555254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f484c0c3fa0 con 0x7f484c14dbe0 2024-09-19T07:25:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.800+0000 7f48497fa640 1 -- 172.21.15.83:0/1237555254 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+5707 (secure 0 0 0) 0x7f4838059a10 con 0x7f484c14dbe0 2024-09-19T07:25:48.803 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:48.803 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":62,"btime":"2024-09-19T07:25:39:224539+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:25:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 62 2024-09-19T07:25:48.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.804+0000 7f4852e3d640 1 -- 172.21.15.83:0/1237555254 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f48300678f0 msgr2=0x7f4830069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:48.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.804+0000 7f4852e3d640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f48300678f0 0x7f4830069db0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f484c14cb90 tx=0x7f4840009040 comp rx=0 tx=0).stop 2024-09-19T07:25:48.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.804+0000 7f4852e3d640 1 -- 172.21.15.83:0/1237555254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f484c14dbe0 msgr2=0x7f484c143970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.804+0000 7f4852e3d640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f484c14dbe0 0x7f484c143970 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f483800c4a0 tx=0x7f483800c960 comp rx=0 tx=0).stop 2024-09-19T07:25:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.804+0000 7f4852e3d640 1 -- 172.21.15.83:0/1237555254 shutdown_connections 2024-09-19T07:25:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.804+0000 7f4852e3d640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f48300678f0 0x7f4830069db0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.804+0000 7f4852e3d640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f484c147a00 0x7f484c143eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.804+0000 7f4852e3d640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f484c14dbe0 0x7f484c143970 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.804+0000 7f4852e3d640 1 --2- 172.21.15.83:0/1237555254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f484c0aa4a0 0x7f484c0ada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.804+0000 7f4852e3d640 1 -- 172.21.15.83:0/1237555254 >> 172.21.15.83:0/1237555254 conn(0x7f484c0c95a0 msgr2=0x7f484c151890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.804+0000 7f4852e3d640 1 -- 172.21.15.83:0/1237555254 shutdown_connections 2024-09-19T07:25:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:48.804+0000 7f4852e3d640 1 -- 172.21.15.83:0/1237555254 wait complete. 2024-09-19T07:25:48.826 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-19T07:25:50.826 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:25:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc56fbb640 1 Processor -- start 2024-09-19T07:25:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc56fbb640 1 -- start start 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc56fbb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efc500c3ab0 0x7efc50157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc56fbb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7efc500aa4a0 0x7efc50157550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc56fbb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc50073d90 con 0x7efc5014dbe0 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc56fbb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7efc50154700 con 0x7efc500c3ab0 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc56fbb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7efc50153b30 con 0x7efc500aa4a0 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc4ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efc500c3ab0 0x7efc50157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc54d57640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7efc5014dbe0 0x7efc500c3fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42392/0 (socket says 172.21.15.83:42392) 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc54d57640 1 -- 172.21.15.83:0/3747977543 learned_addr learned my addr 172.21.15.83:0/3747977543 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc55558640 1 --2- 172.21.15.83:0/3747977543 >> v2:172.21.15.184:3300/0 conn(0x7efc500aa4a0 0x7efc50157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc4ffff640 1 -- 172.21.15.83:0/3747977543 >> v2:172.21.15.184:3300/0 conn(0x7efc500aa4a0 msgr2=0x7efc50157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc4ffff640 1 --2- 172.21.15.83:0/3747977543 >> v2:172.21.15.184:3300/0 conn(0x7efc500aa4a0 0x7efc50157550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc4ffff640 1 -- 172.21.15.83:0/3747977543 >> v1:172.21.15.83:6789/0 conn(0x7efc5014dbe0 legacy=0x7efc500c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc4ffff640 1 -- 172.21.15.83:0/3747977543 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc50153dd0 con 0x7efc500c3ab0 2024-09-19T07:25:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.968+0000 7efc4f7fe640 1 -- 172.21.15.83:0/3747977543 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4140860319 0 0) 0x7efc50073d90 con 0x7efc5014dbe0 2024-09-19T07:25:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.972+0000 7efc4ffff640 1 --2- 172.21.15.83:0/3747977543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc500c3ab0 0x7efc50157010 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7efc500ca2e0 tx=0x7efc3c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654a2d26bc8d70c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.972+0000 7efc4f7fe640 1 -- 172.21.15.83:0/3747977543 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0b0070 con 0x7efc500c3ab0 2024-09-19T07:25:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.972+0000 7efc4f7fe640 1 -- 172.21.15.83:0/3747977543 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc3c0ab600 con 0x7efc500c3ab0 2024-09-19T07:25:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.972+0000 7efc4f7fe640 1 -- 172.21.15.83:0/3747977543 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0ab9d0 con 0x7efc500c3ab0 2024-09-19T07:25:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.972+0000 7efc56fbb640 1 -- 172.21.15.83:0/3747977543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc500c3ab0 msgr2=0x7efc50157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.972+0000 7efc56fbb640 1 --2- 172.21.15.83:0/3747977543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc500c3ab0 0x7efc50157010 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7efc500ca2e0 tx=0x7efc3c01c3f0 comp rx=0 tx=0).stop 2024-09-19T07:25:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.972+0000 7efc56fbb640 1 -- 172.21.15.83:0/3747977543 shutdown_connections 2024-09-19T07:25:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.972+0000 7efc56fbb640 1 --2- 172.21.15.83:0/3747977543 >> v2:172.21.15.184:3300/0 conn(0x7efc500aa4a0 0x7efc50157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.972+0000 7efc56fbb640 1 --2- 172.21.15.83:0/3747977543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc500c3ab0 0x7efc50157010 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.972+0000 7efc56fbb640 1 -- 172.21.15.83:0/3747977543 >> 172.21.15.83:0/3747977543 conn(0x7efc500c95a0 msgr2=0x7efc500c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.972+0000 7efc56fbb640 1 -- 172.21.15.83:0/3747977543 shutdown_connections 2024-09-19T07:25:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.972+0000 7efc56fbb640 1 -- 172.21.15.83:0/3747977543 wait complete. 2024-09-19T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc56fbb640 1 Processor -- start 2024-09-19T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc56fbb640 1 -- start start 2024-09-19T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc56fbb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc500aa4a0 0x7efc500c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc56fbb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc5014dbe0 0x7efc50149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc56fbb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc501504e0 0x7efc50150990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc56fbb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efc500c4830 con 0x7efc501504e0 2024-09-19T07:25:50.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc56fbb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efc500c46b0 con 0x7efc500aa4a0 2024-09-19T07:25:50.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc56fbb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efc50160bc0 con 0x7efc5014dbe0 2024-09-19T07:25:50.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc55558640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc501504e0 0x7efc50150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:50.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc55558640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc501504e0 0x7efc50150990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46574/0 (socket says 172.21.15.83:46574) 2024-09-19T07:25:50.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc55558640 1 -- 172.21.15.83:0/1865258880 learned_addr learned my addr 172.21.15.83:0/1865258880 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:25:50.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc54d57640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc500aa4a0 0x7efc500c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:50.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc4ffff640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc5014dbe0 0x7efc50149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:50.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc55558640 1 -- 172.21.15.83:0/1865258880 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc5014dbe0 msgr2=0x7efc50149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:50.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc55558640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc5014dbe0 0x7efc50149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:50.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc55558640 1 -- 172.21.15.83:0/1865258880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc500aa4a0 msgr2=0x7efc500c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:50.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc55558640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc500aa4a0 0x7efc500c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:50.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc55558640 1 -- 172.21.15.83:0/1865258880 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc50150ed0 con 0x7efc501504e0 2024-09-19T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc55558640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc501504e0 0x7efc50150990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7efc48007e60 tx=0x7efc4800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc4d7fa640 1 -- 172.21.15.83:0/1865258880 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48014420 con 0x7efc501504e0 2024-09-19T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc4d7fa640 1 -- 172.21.15.83:0/1865258880 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc48019070 con 0x7efc501504e0 2024-09-19T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc56fbb640 1 -- 172.21.15.83:0/1865258880 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc5006d980 con 0x7efc501504e0 2024-09-19T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc4d7fa640 1 -- 172.21.15.83:0/1865258880 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48014820 con 0x7efc501504e0 2024-09-19T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc56fbb640 1 -- 172.21.15.83:0/1865258880 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc500c3160 con 0x7efc501504e0 2024-09-19T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc3affd640 1 -- 172.21.15.83:0/1865258880 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc280052a0 con 0x7efc501504e0 2024-09-19T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.976+0000 7efc4d7fa640 1 -- 172.21.15.83:0/1865258880 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efc480149c0 con 0x7efc501504e0 2024-09-19T07:25:50.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.980+0000 7efc4d7fa640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc340678c0 0x7efc34069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:25:50.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.980+0000 7efc4d7fa640 1 -- 172.21.15.83:0/1865258880 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7efc48086ac0 con 0x7efc501504e0 2024-09-19T07:25:50.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.980+0000 7efc54d57640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc340678c0 0x7efc34069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:25:50.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.980+0000 7efc54d57640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc340678c0 0x7efc34069d80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7efc44007fd0 tx=0x7efc44007d60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:50.984+0000 7efc4d7fa640 1 -- 172.21.15.83:0/1865258880 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7efc48056f90 con 0x7efc501504e0 2024-09-19T07:25:51.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.196+0000 7efc3affd640 1 -- 172.21.15.83:0/1865258880 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc28005860 con 0x7efc501504e0 2024-09-19T07:25:51.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc4d7fa640 1 -- 172.21.15.83:0/1865258880 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6532 (secure 0 0 0) 0x7efc48056890 con 0x7efc501504e0 2024-09-19T07:25:51.199 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:25:51.200 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":63,"btime":"2024-09-19T07:25:49:257173+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:25:51.200 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 63 2024-09-19T07:25:51.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc3affd640 1 -- 172.21.15.83:0/1865258880 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc340678c0 msgr2=0x7efc34069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:51.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc3affd640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc340678c0 0x7efc34069d80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7efc44007fd0 tx=0x7efc44007d60 comp rx=0 tx=0).stop 2024-09-19T07:25:51.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc3affd640 1 -- 172.21.15.83:0/1865258880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc501504e0 msgr2=0x7efc50150990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:25:51.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc3affd640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc501504e0 0x7efc50150990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7efc48007e60 tx=0x7efc4800e4b0 comp rx=0 tx=0).stop 2024-09-19T07:25:51.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc3affd640 1 -- 172.21.15.83:0/1865258880 shutdown_connections 2024-09-19T07:25:51.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc3affd640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc340678c0 0x7efc34069d80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:51.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc3affd640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc501504e0 0x7efc50150990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:51.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc3affd640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc5014dbe0 0x7efc50149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:51.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc3affd640 1 --2- 172.21.15.83:0/1865258880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc500aa4a0 0x7efc500c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:25:51.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc3affd640 1 -- 172.21.15.83:0/1865258880 >> 172.21.15.83:0/1865258880 conn(0x7efc500c95a0 msgr2=0x7efc5014b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:25:51.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc3affd640 1 -- 172.21.15.83:0/1865258880 shutdown_connections 2024-09-19T07:25:51.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:25:51.200+0000 7efc3affd640 1 -- 172.21.15.83:0/1865258880 wait complete. 2024-09-19T07:25:51.223 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-09-19T07:25:51.223 INFO:tasks.mds_thrash.fs.[b]:waiting for 88 secs before thrashing 2024-09-19T07:26:01.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:26:01.815 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:26:31.866 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:26:31.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:27:00.931 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:01.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cfd8da640 1 Processor -- start 2024-09-19T07:27:01.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cfd8da640 1 -- start start 2024-09-19T07:27:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cfd8da640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7cf8068700 0x7f7cf806e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cfd8da640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7cf8153390 0x7f7cf806d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cfd8da640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cf8155fa0 con 0x7f7cf81494a0 2024-09-19T07:27:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cfd8da640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7cf8156e40 con 0x7f7cf8068700 2024-09-19T07:27:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cfd8da640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f7cf80c3850 con 0x7f7cf8153390 2024-09-19T07:27:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cf67fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7cf8068700 0x7f7cf806e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cf67fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7cf8068700 0x7f7cf806e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34114/0 (socket says 172.21.15.83:34114) 2024-09-19T07:27:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cf67fc640 1 -- 172.21.15.83:0/2586092318 learned_addr learned my addr 172.21.15.83:0/2586092318 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cf77fe640 1 --2- 172.21.15.83:0/2586092318 >> v2:172.21.15.184:3300/0 conn(0x7f7cf8153390 0x7f7cf806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cf67fc640 1 -- 172.21.15.83:0/2586092318 >> v2:172.21.15.184:3300/0 conn(0x7f7cf8153390 msgr2=0x7f7cf806d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cf67fc640 1 --2- 172.21.15.83:0/2586092318 >> v2:172.21.15.184:3300/0 conn(0x7f7cf8153390 0x7f7cf806d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cf67fc640 1 -- 172.21.15.83:0/2586092318 >> v1:172.21.15.184:6789/0 conn(0x7f7cf81494a0 legacy=0x7f7cf8156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cf67fc640 1 -- 172.21.15.83:0/2586092318 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf80c3b10 con 0x7f7cf8068700 2024-09-19T07:27:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.068+0000 7f7cf77fe640 1 --2- 172.21.15.83:0/2586092318 >> v2:172.21.15.184:3300/0 conn(0x7f7cf8153390 0x7f7cf806d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.072+0000 7f7cf67fc640 1 --2- 172.21.15.83:0/2586092318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cf8068700 0x7f7cf806e010 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8150450 tx=0x7f7cec01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5a2e4ff5c8f03 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.072+0000 7f7cf5ffb640 1 -- 172.21.15.83:0/2586092318 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec0ab5f0 con 0x7f7cf8068700 2024-09-19T07:27:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.072+0000 7f7cf5ffb640 1 -- 172.21.15.83:0/2586092318 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cec0b0070 con 0x7f7cf8068700 2024-09-19T07:27:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.072+0000 7f7cf5ffb640 1 -- 172.21.15.83:0/2586092318 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec0ab9b0 con 0x7f7cf8068700 2024-09-19T07:27:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.072+0000 7f7cfd8da640 1 -- 172.21.15.83:0/2586092318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cf8068700 msgr2=0x7f7cf806e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.072+0000 7f7cfd8da640 1 --2- 172.21.15.83:0/2586092318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cf8068700 0x7f7cf806e010 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8150450 tx=0x7f7cec01c4e0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.072+0000 7f7cfd8da640 1 -- 172.21.15.83:0/2586092318 shutdown_connections 2024-09-19T07:27:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.072+0000 7f7cfd8da640 1 --2- 172.21.15.83:0/2586092318 >> v2:172.21.15.184:3300/0 conn(0x7f7cf8153390 0x7f7cf806d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.072+0000 7f7cfd8da640 1 --2- 172.21.15.83:0/2586092318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cf8068700 0x7f7cf806e010 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.072+0000 7f7cfd8da640 1 -- 172.21.15.83:0/2586092318 >> 172.21.15.83:0/2586092318 conn(0x7f7cf80c4940 msgr2=0x7f7cf80c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.072+0000 7f7cfd8da640 1 -- 172.21.15.83:0/2586092318 shutdown_connections 2024-09-19T07:27:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.072+0000 7f7cfd8da640 1 -- 172.21.15.83:0/2586092318 wait complete. 2024-09-19T07:27:01.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cfd8da640 1 Processor -- start 2024-09-19T07:27:01.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cfd8da640 1 -- start start 2024-09-19T07:27:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cfd8da640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7cf8068700 0x7f7cf80c14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cfd8da640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cf81494a0 0x7f7cf81438f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cfd8da640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7cf8153390 0x7f7cf8143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cfd8da640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf815fab0 con 0x7f7cf81494a0 2024-09-19T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cfd8da640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf80c40d0 con 0x7f7cf8153390 2024-09-19T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cfd8da640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf815fc30 con 0x7f7cf8068700 2024-09-19T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cf67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cf81494a0 0x7f7cf81438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cf67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cf81494a0 0x7f7cf81438f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34120/0 (socket says 172.21.15.83:34120) 2024-09-19T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cf67fc640 1 -- 172.21.15.83:0/3414532979 learned_addr learned my addr 172.21.15.83:0/3414532979 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cf77fe640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7cf8153390 0x7f7cf8143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cf6ffd640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7cf8068700 0x7f7cf80c14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cf67fc640 1 -- 172.21.15.83:0/3414532979 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7cf8068700 msgr2=0x7f7cf80c14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cf67fc640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7cf8068700 0x7f7cf80c14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cf67fc640 1 -- 172.21.15.83:0/3414532979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7cf8153390 msgr2=0x7f7cf8143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cf67fc640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7cf8153390 0x7f7cf8143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cf67fc640 1 -- 172.21.15.83:0/3414532979 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf80c5810 con 0x7f7cf81494a0 2024-09-19T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cf77fe640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7cf8153390 0x7f7cf8143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cf67fc640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cf81494a0 0x7f7cf81438f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7cec0ad040 tx=0x7f7cec01ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cdffff640 1 -- 172.21.15.83:0/3414532979 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec0be5f0 con 0x7f7cf81494a0 2024-09-19T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cdffff640 1 -- 172.21.15.83:0/3414532979 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cec0be7b0 con 0x7f7cf81494a0 2024-09-19T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cdffff640 1 -- 172.21.15.83:0/3414532979 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec0beba0 con 0x7f7cf81494a0 2024-09-19T07:27:01.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cfd8da640 1 -- 172.21.15.83:0/3414532979 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf8148a90 con 0x7f7cf81494a0 2024-09-19T07:27:01.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cfd8da640 1 -- 172.21.15.83:0/3414532979 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf80c1a10 con 0x7f7cf81494a0 2024-09-19T07:27:01.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.076+0000 7f7cfd8da640 1 -- 172.21.15.83:0/3414532979 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf806d470 con 0x7f7cf81494a0 2024-09-19T07:27:01.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.080+0000 7f7cdffff640 1 -- 172.21.15.83:0/3414532979 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7cec0b43f0 con 0x7f7cf81494a0 2024-09-19T07:27:01.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.080+0000 7f7cdffff640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7ce40678c0 0x7f7ce4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:01.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.080+0000 7f7cdffff640 1 -- 172.21.15.83:0/3414532979 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7f7cec120d20 con 0x7f7cf81494a0 2024-09-19T07:27:01.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.080+0000 7f7cf6ffd640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7ce40678c0 0x7f7ce4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:01.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.080+0000 7f7cf6ffd640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7ce40678c0 0x7f7ce4069d80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80ad480 tx=0x7f7ce8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:01.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.084+0000 7f7cdffff640 1 -- 172.21.15.83:0/3414532979 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7cec0160d0 con 0x7f7cf81494a0 2024-09-19T07:27:01.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.304+0000 7f7cfd8da640 1 -- 172.21.15.83:0/3414532979 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7cf8070060 con 0x7f7cf81494a0 2024-09-19T07:27:01.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.304+0000 7f7cdffff640 1 -- 172.21.15.83:0/3414532979 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6532 (secure 0 0 0) 0x7f7cec0f1310 con 0x7f7cf81494a0 2024-09-19T07:27:01.306 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:01.306 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":63,"btime":"2024-09-19T07:25:49:257173+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:01.306 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 63 2024-09-19T07:27:01.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.308+0000 7f7cfd8da640 1 -- 172.21.15.83:0/3414532979 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7ce40678c0 msgr2=0x7f7ce4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:01.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.308+0000 7f7cfd8da640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7ce40678c0 0x7f7ce4069d80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80ad480 tx=0x7f7ce8009040 comp rx=0 tx=0).stop 2024-09-19T07:27:01.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.308+0000 7f7cfd8da640 1 -- 172.21.15.83:0/3414532979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cf81494a0 msgr2=0x7f7cf81438f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:01.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.308+0000 7f7cfd8da640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cf81494a0 0x7f7cf81438f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7cec0ad040 tx=0x7f7cec01ad20 comp rx=0 tx=0).stop 2024-09-19T07:27:01.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.308+0000 7f7cfd8da640 1 -- 172.21.15.83:0/3414532979 shutdown_connections 2024-09-19T07:27:01.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.308+0000 7f7cfd8da640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7ce40678c0 0x7f7ce4069d80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.308+0000 7f7cfd8da640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7cf8153390 0x7f7cf8143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.308+0000 7f7cfd8da640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cf81494a0 0x7f7cf81438f0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.308+0000 7f7cfd8da640 1 --2- 172.21.15.83:0/3414532979 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7cf8068700 0x7f7cf80c14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.308+0000 7f7cfd8da640 1 -- 172.21.15.83:0/3414532979 >> 172.21.15.83:0/3414532979 conn(0x7f7cf80c4940 msgr2=0x7f7cf8146a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:01.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.308+0000 7f7cfd8da640 1 -- 172.21.15.83:0/3414532979 shutdown_connections 2024-09-19T07:27:01.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.308+0000 7f7cfd8da640 1 -- 172.21.15.83:0/3414532979 wait complete. 2024-09-19T07:27:01.326 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-19T07:27:01.326 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:27:01.326 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:27:01.338 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:27:01.338 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:01.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.508+0000 7f4161f09640 1 Processor -- start 2024-09-19T07:27:01.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.508+0000 7f4161f09640 1 -- start start 2024-09-19T07:27:01.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.508+0000 7f4161f09640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f415c0c6360 0x7f415c153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.508+0000 7f4161f09640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f415c0bee30 0x7f415c1570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.508+0000 7f4161f09640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f415c0c5b90 con 0x7f415c0c9680 2024-09-19T07:27:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.508+0000 7f4161f09640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f415c15a050 con 0x7f415c0c6360 2024-09-19T07:27:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.508+0000 7f4161f09640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f415c06c950 con 0x7f415c0bee30 2024-09-19T07:27:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.508+0000 7f415b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f415c0c6360 0x7f415c153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.508+0000 7f415b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f415c0c6360 0x7f415c153050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34130/0 (socket says 172.21.15.83:34130) 2024-09-19T07:27:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.508+0000 7f415b7fe640 1 -- 172.21.15.83:0/23629068 learned_addr learned my addr 172.21.15.83:0/23629068 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f4153fff640 1 --2- 172.21.15.83:0/23629068 >> v2:172.21.15.184:3300/0 conn(0x7f415c0bee30 0x7f415c1570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f415b7fe640 1 -- 172.21.15.83:0/23629068 >> v2:172.21.15.184:3300/0 conn(0x7f415c0bee30 msgr2=0x7f415c1570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f415b7fe640 1 --2- 172.21.15.83:0/23629068 >> v2:172.21.15.184:3300/0 conn(0x7f415c0bee30 0x7f415c1570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f415b7fe640 1 -- 172.21.15.83:0/23629068 >> v1:172.21.15.83:6789/0 conn(0x7f415c0c9680 legacy=0x7f415c157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f415b7fe640 1 -- 172.21.15.83:0/23629068 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415c06c670 con 0x7f415c0c6360 2024-09-19T07:27:01.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f415b7fe640 1 --2- 172.21.15.83:0/23629068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f415c0c6360 0x7f415c153050 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f415c0c2650 tx=0x7f414c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf0050b65f6b622 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:01.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f415affd640 1 -- 172.21.15.83:0/23629068 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c0b0070 con 0x7f415c0c6360 2024-09-19T07:27:01.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f415affd640 1 -- 172.21.15.83:0/23629068 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f414c0ab600 con 0x7f415c0c6360 2024-09-19T07:27:01.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f415affd640 1 -- 172.21.15.83:0/23629068 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c0ab9d0 con 0x7f415c0c6360 2024-09-19T07:27:01.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f4161f09640 1 -- 172.21.15.83:0/23629068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f415c0c6360 msgr2=0x7f415c153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:01.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f4161f09640 1 --2- 172.21.15.83:0/23629068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f415c0c6360 0x7f415c153050 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f415c0c2650 tx=0x7f414c01c400 comp rx=0 tx=0).stop 2024-09-19T07:27:01.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f4161f09640 1 -- 172.21.15.83:0/23629068 shutdown_connections 2024-09-19T07:27:01.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f4161f09640 1 --2- 172.21.15.83:0/23629068 >> v2:172.21.15.184:3300/0 conn(0x7f415c0bee30 0x7f415c1570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f4161f09640 1 --2- 172.21.15.83:0/23629068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f415c0c6360 0x7f415c153050 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f4161f09640 1 -- 172.21.15.83:0/23629068 >> 172.21.15.83:0/23629068 conn(0x7f415c0722a0 msgr2=0x7f415c0726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:01.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f4161f09640 1 -- 172.21.15.83:0/23629068 shutdown_connections 2024-09-19T07:27:01.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.512+0000 7f4161f09640 1 -- 172.21.15.83:0/23629068 wait complete. 2024-09-19T07:27:01.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4161f09640 1 Processor -- start 2024-09-19T07:27:01.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4161f09640 1 -- start start 2024-09-19T07:27:01.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4161f09640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f415c0bee30 0x7f415c14c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:01.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4161f09640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f415c0c6360 0x7f415c0ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:01.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4161f09640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f415c0c9680 0x7f415c0c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:01.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4161f09640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f415c0c68c0 con 0x7f415c0c9680 2024-09-19T07:27:01.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4161f09640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f415c0c6740 con 0x7f415c0bee30 2024-09-19T07:27:01.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4161f09640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f415c162da0 con 0x7f415c0c6360 2024-09-19T07:27:01.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f415bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f415c0c9680 0x7f415c0c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:01.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f415bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f415c0c9680 0x7f415c0c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34144/0 (socket says 172.21.15.83:34144) 2024-09-19T07:27:01.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f415bfff640 1 -- 172.21.15.83:0/4259007966 learned_addr learned my addr 172.21.15.83:0/4259007966 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:01.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4153fff640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f415c0c6360 0x7f415c0ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f415b7fe640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f415c0bee30 0x7f415c14c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f415bfff640 1 -- 172.21.15.83:0/4259007966 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f415c0c6360 msgr2=0x7f415c0ae6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f415bfff640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f415c0c6360 0x7f415c0ae6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f415bfff640 1 -- 172.21.15.83:0/4259007966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f415c0bee30 msgr2=0x7f415c14c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f415bfff640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f415c0bee30 0x7f415c14c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f415bfff640 1 -- 172.21.15.83:0/4259007966 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415c146e00 con 0x7f415c0c9680 2024-09-19T07:27:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f415bfff640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f415c0c9680 0x7f415c0c9b90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f415400c730 tx=0x7f415400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4158ff9640 1 -- 172.21.15.83:0/4259007966 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4154016020 con 0x7f415c0c9680 2024-09-19T07:27:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4158ff9640 1 -- 172.21.15.83:0/4259007966 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4154006d60 con 0x7f415c0c9680 2024-09-19T07:27:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4161f09640 1 -- 172.21.15.83:0/4259007966 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f415c0ae440 con 0x7f415c0c9680 2024-09-19T07:27:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4158ff9640 1 -- 172.21.15.83:0/4259007966 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4154005260 con 0x7f415c0c9680 2024-09-19T07:27:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4161f09640 1 -- 172.21.15.83:0/4259007966 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f415c1f2540 con 0x7f415c0c9680 2024-09-19T07:27:01.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.516+0000 7f4161f09640 1 -- 172.21.15.83:0/4259007966 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f415c153050 con 0x7f415c0c9680 2024-09-19T07:27:01.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.520+0000 7f4158ff9640 1 -- 172.21.15.83:0/4259007966 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4154005400 con 0x7f415c0c9680 2024-09-19T07:27:01.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.520+0000 7f4158ff9640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f413c0678c0 0x7f413c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:01.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.520+0000 7f4158ff9640 1 -- 172.21.15.83:0/4259007966 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7f4154085ed0 con 0x7f415c0c9680 2024-09-19T07:27:01.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.520+0000 7f415b7fe640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f413c0678c0 0x7f413c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:01.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.520+0000 7f415b7fe640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f413c0678c0 0x7f413c069d80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f414c0ad000 tx=0x7f414c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:01.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.524+0000 7f4158ff9640 1 -- 172.21.15.83:0/4259007966 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4154020020 con 0x7f415c0c9680 2024-09-19T07:27:01.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.764+0000 7f4161f09640 1 -- 172.21.15.83:0/4259007966 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f415c0bec10 con 0x7f415c0c9680 2024-09-19T07:27:01.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.764+0000 7f4158ff9640 1 -- 172.21.15.83:0/4259007966 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6532 (secure 0 0 0) 0x7f41540563a0 con 0x7f415c0c9680 2024-09-19T07:27:01.764 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:01.764 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":63,"btime":"2024-09-19T07:25:49:257173+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:01.764 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 63 2024-09-19T07:27:01.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.768+0000 7f4161f09640 1 -- 172.21.15.83:0/4259007966 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f413c0678c0 msgr2=0x7f413c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:01.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.768+0000 7f4161f09640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f413c0678c0 0x7f413c069d80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f414c0ad000 tx=0x7f414c008040 comp rx=0 tx=0).stop 2024-09-19T07:27:01.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.768+0000 7f4161f09640 1 -- 172.21.15.83:0/4259007966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f415c0c9680 msgr2=0x7f415c0c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:01.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.768+0000 7f4161f09640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f415c0c9680 0x7f415c0c9b90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f415400c730 tx=0x7f415400cbf0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.768+0000 7f4161f09640 1 -- 172.21.15.83:0/4259007966 shutdown_connections 2024-09-19T07:27:01.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.768+0000 7f4161f09640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f413c0678c0 0x7f413c069d80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.768+0000 7f4161f09640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f415c0c9680 0x7f415c0c9b90 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.768+0000 7f4161f09640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f415c0c6360 0x7f415c0ae6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.768+0000 7f4161f09640 1 --2- 172.21.15.83:0/4259007966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f415c0bee30 0x7f415c14c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:01.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.768+0000 7f4161f09640 1 -- 172.21.15.83:0/4259007966 >> 172.21.15.83:0/4259007966 conn(0x7f415c0722a0 msgr2=0x7f415c0c8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:01.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.768+0000 7f4161f09640 1 -- 172.21.15.83:0/4259007966 shutdown_connections 2024-09-19T07:27:01.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:01.768+0000 7f4161f09640 1 -- 172.21.15.83:0/4259007966 wait complete. 2024-09-19T07:27:01.784 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:27:01.991 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:27:01.994 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:27:03.785 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:03.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f71bd5640 1 Processor -- start 2024-09-19T07:27:03.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f71bd5640 1 -- start start 2024-09-19T07:27:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f71bd5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1545e0 0x7f5f6c06e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f71bd5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f6c154a70 con 0x7f5f6c071100 2024-09-19T07:27:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f71bd5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f6c0a9e60 con 0x7f5f6c0c28e0 2024-09-19T07:27:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f71bd5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6c071b50 con 0x7f5f6c1545e0 2024-09-19T07:27:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5f6c071100 0x7f5f6c150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36852/0 (socket says 172.21.15.83:36852) 2024-09-19T07:27:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6b7fe640 1 -- 172.21.15.83:0/1283876789 learned_addr learned my addr 172.21.15.83:0/1283876789 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1431251456 0 0) 0x7f5f6c154a70 con 0x7f5f6c071100 2024-09-19T07:27:03.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6bfff640 1 --2- 172.21.15.83:0/1283876789 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1545e0 0x7f5f6c06e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f58003530 con 0x7f5f6c071100 2024-09-19T07:27:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 483434956 0 0) 0x7f5f58003530 con 0x7f5f6c071100 2024-09-19T07:27:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1545e0 msgr2=0x7f5f6c06e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 --2- 172.21.15.83:0/1283876789 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1545e0 0x7f5f6c06e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 >> v1:172.21.15.184:6789/0 conn(0x7f5f6c0c28e0 legacy=0x7f5f6c06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c0aa140 con 0x7f5f6c071100 2024-09-19T07:27:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f5f54001680 con 0x7f5f6c071100 2024-09-19T07:27:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 >> v1:172.21.15.83:6789/0 conn(0x7f5f6c071100 legacy=0x7f5f6c150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 --2- 172.21.15.83:0/1283876789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f580039f0 0x7f5f58003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 --2- 172.21.15.83:0/1283876789 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1545e0 0x7f5f58004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 --2- 172.21.15.83:0/1283876789 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f58004d50 0x7f5f580051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c071b50 con 0x7f5f58004d50 2024-09-19T07:27:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5800bbe0 con 0x7f5f580039f0 2024-09-19T07:27:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5f5800bde0 con 0x7f5f6c1545e0 2024-09-19T07:27:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6bfff640 1 --2- 172.21.15.83:0/1283876789 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f58004d50 0x7f5f580051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f62ffd640 1 --2- 172.21.15.83:0/1283876789 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1545e0 0x7f5f58004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6b7fe640 1 --2- 172.21.15.83:0/1283876789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f580039f0 0x7f5f58003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6bfff640 1 -- 172.21.15.83:0/1283876789 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1545e0 msgr2=0x7f5f58004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6bfff640 1 --2- 172.21.15.83:0/1283876789 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1545e0 0x7f5f58004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6bfff640 1 -- 172.21.15.83:0/1283876789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f580039f0 msgr2=0x7f5f58003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6bfff640 1 --2- 172.21.15.83:0/1283876789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f580039f0 0x7f5f58003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:03.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.920+0000 7f5f6bfff640 1 -- 172.21.15.83:0/1283876789 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f58003530 con 0x7f5f58004d50 2024-09-19T07:27:03.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f6bfff640 1 --2- 172.21.15.83:0/1283876789 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f58004d50 0x7f5f580051d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5f64007360 tx=0x7f5f6401ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2488fe96aa17e2f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:03.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f640b2070 con 0x7f5f58004d50 2024-09-19T07:27:03.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f640a5c30 con 0x7f5f58004d50 2024-09-19T07:27:03.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f6affd640 1 -- 172.21.15.83:0/1283876789 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64009be0 con 0x7f5f58004d50 2024-09-19T07:27:03.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 -- 172.21.15.83:0/1283876789 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f58004d50 msgr2=0x7f5f580051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:03.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 --2- 172.21.15.83:0/1283876789 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f58004d50 0x7f5f580051d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5f64007360 tx=0x7f5f6401ce40 comp rx=0 tx=0).stop 2024-09-19T07:27:03.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f6bfff640 1 -- 172.21.15.83:0/1283876789 reap_dead start 2024-09-19T07:27:03.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 -- 172.21.15.83:0/1283876789 shutdown_connections 2024-09-19T07:27:03.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 -- 172.21.15.83:0/1283876789 >> 172.21.15.83:0/1283876789 conn(0x7f5f6c06bc80 msgr2=0x7f5f6c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:03.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 -- 172.21.15.83:0/1283876789 shutdown_connections 2024-09-19T07:27:03.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 -- 172.21.15.83:0/1283876789 wait complete. 2024-09-19T07:27:03.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 Processor -- start 2024-09-19T07:27:03.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 -- start start 2024-09-19T07:27:03.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f58004d50 0x7f5f6c1ddb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:03.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f6c152db0 0x7f5f6c1de800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:03.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1ded40 0x7f5f6c1eca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:03.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c0c2520 con 0x7f5f58004d50 2024-09-19T07:27:03.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c0c1ff0 con 0x7f5f6c152db0 2024-09-19T07:27:03.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f71bd5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6c151650 con 0x7f5f6c1ded40 2024-09-19T07:27:03.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f6b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f58004d50 0x7f5f6c1ddb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:03.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f6b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f58004d50 0x7f5f6c1ddb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34172/0 (socket says 172.21.15.83:34172) 2024-09-19T07:27:03.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f6b7fe640 1 -- 172.21.15.83:0/4268342836 learned_addr learned my addr 172.21.15.83:0/4268342836 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f6bfff640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1ded40 0x7f5f6c1eca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.924+0000 7f5f62ffd640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f6c152db0 0x7f5f6c1de800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f6b7fe640 1 -- 172.21.15.83:0/4268342836 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1ded40 msgr2=0x7f5f6c1eca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f6b7fe640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1ded40 0x7f5f6c1eca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f6b7fe640 1 -- 172.21.15.83:0/4268342836 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f6c152db0 msgr2=0x7f5f6c1de800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f6b7fe640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f6c152db0 0x7f5f6c1de800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f6b7fe640 1 -- 172.21.15.83:0/4268342836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c149210 con 0x7f5f58004d50 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f6b7fe640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f58004d50 0x7f5f6c1ddb30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5f54011950 tx=0x7f5f54011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f68ff9640 1 -- 172.21.15.83:0/4268342836 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f54010040 con 0x7f5f58004d50 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f68ff9640 1 -- 172.21.15.83:0/4268342836 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f5401a070 con 0x7f5f58004d50 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f71bd5640 1 -- 172.21.15.83:0/4268342836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6c0c7f10 con 0x7f5f58004d50 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f68ff9640 1 -- 172.21.15.83:0/4268342836 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5400fd80 con 0x7f5f58004d50 2024-09-19T07:27:03.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f71bd5640 1 -- 172.21.15.83:0/4268342836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f6c1de070 con 0x7f5f58004d50 2024-09-19T07:27:03.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f71bd5640 1 -- 172.21.15.83:0/4268342836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f6c0714e0 con 0x7f5f58004d50 2024-09-19T07:27:03.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f68ff9640 1 -- 172.21.15.83:0/4268342836 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5f5401f460 con 0x7f5f58004d50 2024-09-19T07:27:03.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f68ff9640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5f4c0678f0 0x7f5f4c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:03.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f68ff9640 1 -- 172.21.15.83:0/4268342836 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7f5f54089620 con 0x7f5f58004d50 2024-09-19T07:27:03.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f62ffd640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5f4c0678f0 0x7f5f4c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:03.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.928+0000 7f5f62ffd640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5f4c0678f0 0x7f5f4c069db0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c148e70 tx=0x7f5f5c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:03.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:03.932+0000 7f5f68ff9640 1 -- 172.21.15.83:0/4268342836 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5f54059a60 con 0x7f5f58004d50 2024-09-19T07:27:04.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 -- 172.21.15.83:0/4268342836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f6c0c4f30 con 0x7f5f58004d50 2024-09-19T07:27:04.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f68ff9640 1 -- 172.21.15.83:0/4268342836 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6532 (secure 0 0 0) 0x7f5f54059360 con 0x7f5f58004d50 2024-09-19T07:27:04.132 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:04.132 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":63,"btime":"2024-09-19T07:25:49:257173+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 63 2024-09-19T07:27:04.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 -- 172.21.15.83:0/4268342836 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5f4c0678f0 msgr2=0x7f5f4c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:04.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5f4c0678f0 0x7f5f4c069db0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c148e70 tx=0x7f5f5c009040 comp rx=0 tx=0).stop 2024-09-19T07:27:04.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 -- 172.21.15.83:0/4268342836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f58004d50 msgr2=0x7f5f6c1ddb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:04.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f58004d50 0x7f5f6c1ddb30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5f54011950 tx=0x7f5f54011e10 comp rx=0 tx=0).stop 2024-09-19T07:27:04.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 -- 172.21.15.83:0/4268342836 shutdown_connections 2024-09-19T07:27:04.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5f4c0678f0 0x7f5f4c069db0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:04.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c1ded40 0x7f5f6c1eca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:04.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f6c152db0 0x7f5f6c1de800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:04.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 --2- 172.21.15.83:0/4268342836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f58004d50 0x7f5f6c1ddb30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:04.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 -- 172.21.15.83:0/4268342836 >> 172.21.15.83:0/4268342836 conn(0x7f5f6c06bc80 msgr2=0x7f5f6c14ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:04.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 -- 172.21.15.83:0/4268342836 shutdown_connections 2024-09-19T07:27:04.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:04.132+0000 7f5f71bd5640 1 -- 172.21.15.83:0/4268342836 wait complete. 2024-09-19T07:27:04.150 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:27:06.151 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:06.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.280+0000 7fa21bacc640 1 Processor -- start 2024-09-19T07:27:06.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.280+0000 7fa21bacc640 1 -- start start 2024-09-19T07:27:06.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa21bacc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa21406e270 0x7fa2141566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:06.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa21bacc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2140abb30 0x7fa214153060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:06.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa21bacc640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2141592c0 con 0x7fa2140c42e0 2024-09-19T07:27:06.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa21bacc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa21406e6e0 con 0x7fa21406e270 2024-09-19T07:27:06.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa21bacc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa2141595f0 con 0x7fa2140abb30 2024-09-19T07:27:06.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa219067640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa21406e270 0x7fa2141566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:06.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa219067640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa21406e270 0x7fa2141566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34186/0 (socket says 172.21.15.83:34186) 2024-09-19T07:27:06.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa219067640 1 -- 172.21.15.83:0/1464111076 learned_addr learned my addr 172.21.15.83:0/1464111076 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:06.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa21a069640 1 --2- 172.21.15.83:0/1464111076 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2140abb30 0x7fa214153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:06.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa219067640 1 -- 172.21.15.83:0/1464111076 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2140abb30 msgr2=0x7fa214153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:06.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa219067640 1 --2- 172.21.15.83:0/1464111076 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2140abb30 0x7fa214153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:06.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa219067640 1 -- 172.21.15.83:0/1464111076 >> v1:172.21.15.184:6789/0 conn(0x7fa2140c42e0 legacy=0x7fa214159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:06.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa219067640 1 -- 172.21.15.83:0/1464111076 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa21406e900 con 0x7fa21406e270 2024-09-19T07:27:06.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa219067640 1 --2- 172.21.15.83:0/1464111076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa21406e270 0x7fa2141566b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa214159770 tx=0x7fa21001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8056f29e974b8c81 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:06.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa218866640 1 -- 172.21.15.83:0/1464111076 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2100b1070 con 0x7fa21406e270 2024-09-19T07:27:06.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa218866640 1 -- 172.21.15.83:0/1464111076 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2100ac630 con 0x7fa21406e270 2024-09-19T07:27:06.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.284+0000 7fa218866640 1 -- 172.21.15.83:0/1464111076 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2100a8050 con 0x7fa21406e270 2024-09-19T07:27:06.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 -- 172.21.15.83:0/1464111076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa21406e270 msgr2=0x7fa2141566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:06.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 --2- 172.21.15.83:0/1464111076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa21406e270 0x7fa2141566b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa214159770 tx=0x7fa21001c500 comp rx=0 tx=0).stop 2024-09-19T07:27:06.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 -- 172.21.15.83:0/1464111076 shutdown_connections 2024-09-19T07:27:06.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 --2- 172.21.15.83:0/1464111076 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2140abb30 0x7fa214153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:06.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 --2- 172.21.15.83:0/1464111076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa21406e270 0x7fa2141566b0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:06.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 -- 172.21.15.83:0/1464111076 >> 172.21.15.83:0/1464111076 conn(0x7fa21406bc80 msgr2=0x7fa21406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:06.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 -- 172.21.15.83:0/1464111076 shutdown_connections 2024-09-19T07:27:06.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 -- 172.21.15.83:0/1464111076 wait complete. 2024-09-19T07:27:06.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 Processor -- start 2024-09-19T07:27:06.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 -- start start 2024-09-19T07:27:06.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa21406e270 0x7fa2141ef6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:06.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2140abb30 0x7fa2141456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:06.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2140c42e0 0x7fa214147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:06.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa2140c4840 con 0x7fa2140c42e0 2024-09-19T07:27:06.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa2140c46c0 con 0x7fa21406e270 2024-09-19T07:27:06.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa214162dd0 con 0x7fa2140abb30 2024-09-19T07:27:06.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21a069640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2140c42e0 0x7fa214147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:06.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21a069640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2140c42e0 0x7fa214147ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34192/0 (socket says 172.21.15.83:34192) 2024-09-19T07:27:06.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21a069640 1 -- 172.21.15.83:0/1969886960 learned_addr learned my addr 172.21.15.83:0/1969886960 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa219067640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2140abb30 0x7fa2141456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa219868640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa21406e270 0x7fa2141ef6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21a069640 1 -- 172.21.15.83:0/1969886960 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2140abb30 msgr2=0x7fa2141456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21a069640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2140abb30 0x7fa2141456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21a069640 1 -- 172.21.15.83:0/1969886960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa21406e270 msgr2=0x7fa2141ef6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21a069640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa21406e270 0x7fa2141ef6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21a069640 1 -- 172.21.15.83:0/1969886960 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2141efc20 con 0x7fa2140c42e0 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21a069640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2140c42e0 0x7fa214147ce0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa20c007e40 tx=0x7fa20c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa2067fc640 1 -- 172.21.15.83:0/1969886960 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20c014620 con 0x7fa2140c42e0 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa2067fc640 1 -- 172.21.15.83:0/1969886960 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa20c019070 con 0x7fa2140c42e0 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 -- 172.21.15.83:0/1969886960 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa214145c30 con 0x7fa2140c42e0 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa2067fc640 1 -- 172.21.15.83:0/1969886960 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20c014a20 con 0x7fa2140c42e0 2024-09-19T07:27:06.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 -- 172.21.15.83:0/1969886960 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa214144df0 con 0x7fa2140c42e0 2024-09-19T07:27:06.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.288+0000 7fa21bacc640 1 -- 172.21.15.83:0/1969886960 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2141566b0 con 0x7fa2140c42e0 2024-09-19T07:27:06.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.292+0000 7fa2067fc640 1 -- 172.21.15.83:0/1969886960 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa20c014bc0 con 0x7fa2140c42e0 2024-09-19T07:27:06.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.292+0000 7fa2067fc640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1f80678c0 0x7fa1f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:06.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.292+0000 7fa2067fc640 1 -- 172.21.15.83:0/1969886960 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7fa20c086c10 con 0x7fa2140c42e0 2024-09-19T07:27:06.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.292+0000 7fa219868640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1f80678c0 0x7fa1f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:06.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.292+0000 7fa219868640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1f80678c0 0x7fa1f8069d80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa21414f130 tx=0x7fa208009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:06.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.296+0000 7fa2067fc640 1 -- 172.21.15.83:0/1969886960 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa20c0570e0 con 0x7fa2140c42e0 2024-09-19T07:27:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.492+0000 7fa21bacc640 1 -- 172.21.15.83:0/1969886960 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa21406e050 con 0x7fa2140c42e0 2024-09-19T07:27:06.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.492+0000 7fa2067fc640 1 -- 172.21.15.83:0/1969886960 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6532 (secure 0 0 0) 0x7fa20c0569e0 con 0x7fa2140c42e0 2024-09-19T07:27:06.493 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:06.493 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":63,"btime":"2024-09-19T07:25:49:257173+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:06.493 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 63 2024-09-19T07:27:06.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.496+0000 7fa21bacc640 1 -- 172.21.15.83:0/1969886960 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1f80678c0 msgr2=0x7fa1f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:06.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.496+0000 7fa21bacc640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1f80678c0 0x7fa1f8069d80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa21414f130 tx=0x7fa208009040 comp rx=0 tx=0).stop 2024-09-19T07:27:06.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.496+0000 7fa21bacc640 1 -- 172.21.15.83:0/1969886960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2140c42e0 msgr2=0x7fa214147ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:06.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.496+0000 7fa21bacc640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2140c42e0 0x7fa214147ce0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa20c007e40 tx=0x7fa20c00e4b0 comp rx=0 tx=0).stop 2024-09-19T07:27:06.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.496+0000 7fa21bacc640 1 -- 172.21.15.83:0/1969886960 shutdown_connections 2024-09-19T07:27:06.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.496+0000 7fa21bacc640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1f80678c0 0x7fa1f8069d80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:06.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.496+0000 7fa21bacc640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2140c42e0 0x7fa214147ce0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:06.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.496+0000 7fa21bacc640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2140abb30 0x7fa2141456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:06.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.496+0000 7fa21bacc640 1 --2- 172.21.15.83:0/1969886960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa21406e270 0x7fa2141ef6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:06.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.496+0000 7fa21bacc640 1 -- 172.21.15.83:0/1969886960 >> 172.21.15.83:0/1969886960 conn(0x7fa21406bc80 msgr2=0x7fa2140c23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:06.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.496+0000 7fa21bacc640 1 -- 172.21.15.83:0/1969886960 shutdown_connections 2024-09-19T07:27:06.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:06.496+0000 7fa21bacc640 1 -- 172.21.15.83:0/1969886960 wait complete. 2024-09-19T07:27:06.511 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:27:08.511 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:08.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.644+0000 7f605cc7e640 1 Processor -- start 2024-09-19T07:27:08.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.644+0000 7f605cc7e640 1 -- start start 2024-09-19T07:27:08.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605cc7e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6058152b20 0x7f60580c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:08.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605cc7e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6058150670 con 0x7f6058150290 2024-09-19T07:27:08.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605cc7e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60580c2880 con 0x7f60580c45c0 2024-09-19T07:27:08.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605cc7e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f60580716c0 con 0x7f6058152b20 2024-09-19T07:27:08.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6058150290 0x7f60580c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36864/0 (socket says 172.21.15.83:36864) 2024-09-19T07:27:08.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605659c640 1 -- 172.21.15.83:0/3503941443 learned_addr learned my addr 172.21.15.83:0/3503941443 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:08.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f6055d9b640 1 --2- 172.21.15.83:0/3503941443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6058152b20 0x7f60580c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:08.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565022244 0 0) 0x7f6058150670 con 0x7f6058150290 2024-09-19T07:27:08.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6040003530 con 0x7f6058150290 2024-09-19T07:27:08.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1319828737 0 0) 0x7f6040003530 con 0x7f6058150290 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6058152b20 msgr2=0x7f60580c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 --2- 172.21.15.83:0/3503941443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6058152b20 0x7f60580c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 >> v1:172.21.15.184:6789/0 conn(0x7f60580c45c0 legacy=0x7f6058153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60580c2b60 con 0x7f6058150290 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f604c001680 con 0x7f6058150290 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 >> v1:172.21.15.83:6789/0 conn(0x7f6058150290 legacy=0x7f60580c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 --2- 172.21.15.83:0/3503941443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6058152b20 0x7f60400039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 --2- 172.21.15.83:0/3503941443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6040003f70 0x7f6040004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 --2- 172.21.15.83:0/3503941443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6040004d50 0x7f60400051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f604000ba80 con 0x7f6040004d50 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f604000bc30 con 0x7f6040003f70 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f604000be30 con 0x7f6058152b20 2024-09-19T07:27:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f6056d9d640 1 --2- 172.21.15.83:0/3503941443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6040004d50 0x7f60400051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:08.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f6055d9b640 1 --2- 172.21.15.83:0/3503941443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6040003f70 0x7f6040004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:08.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f6056d9d640 1 -- 172.21.15.83:0/3503941443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6058152b20 msgr2=0x7f60400039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:08.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f6056d9d640 1 --2- 172.21.15.83:0/3503941443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6058152b20 0x7f60400039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:08.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f6056d9d640 1 -- 172.21.15.83:0/3503941443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6040003f70 msgr2=0x7f6040004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:08.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f6056d9d640 1 --2- 172.21.15.83:0/3503941443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6040003f70 0x7f6040004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:08.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f6056d9d640 1 -- 172.21.15.83:0/3503941443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6040003530 con 0x7f6040004d50 2024-09-19T07:27:08.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.648+0000 7f6055d9b640 1 --2- 172.21.15.83:0/3503941443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6040003f70 0x7f6040004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:08.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6056d9d640 1 --2- 172.21.15.83:0/3503941443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6040004d50 0x7f60400051d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6050001a50 tx=0x7f605001e9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b1b125e503b1f47 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:08.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050001a90 con 0x7f6040004d50 2024-09-19T07:27:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6050003b80 con 0x7f6040004d50 2024-09-19T07:27:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605559a640 1 -- 172.21.15.83:0/3503941443 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60500050d0 con 0x7f6040004d50 2024-09-19T07:27:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 -- 172.21.15.83:0/3503941443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6040004d50 msgr2=0x7f60400051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 --2- 172.21.15.83:0/3503941443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6040004d50 0x7f60400051d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6050001a50 tx=0x7f605001e9b0 comp rx=0 tx=0).stop 2024-09-19T07:27:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6056d9d640 1 -- 172.21.15.83:0/3503941443 reap_dead start 2024-09-19T07:27:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 -- 172.21.15.83:0/3503941443 shutdown_connections 2024-09-19T07:27:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 -- 172.21.15.83:0/3503941443 >> 172.21.15.83:0/3503941443 conn(0x7f605806bc80 msgr2=0x7f605806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 -- 172.21.15.83:0/3503941443 shutdown_connections 2024-09-19T07:27:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 -- 172.21.15.83:0/3503941443 wait complete. 2024-09-19T07:27:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 Processor -- start 2024-09-19T07:27:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 -- start start 2024-09-19T07:27:08.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60580c7650 0x7f60580c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:08.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60580c5610 0x7f60581eaff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:08.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60581eb530 0x7f60581496c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:08.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f60580c3490 con 0x7f60580c5610 2024-09-19T07:27:08.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f60580c3120 con 0x7f60581eb530 2024-09-19T07:27:08.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605cc7e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f60580c4080 con 0x7f60580c7650 2024-09-19T07:27:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6055d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60580c5610 0x7f60581eaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6055d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60580c5610 0x7f60581eaff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34212/0 (socket says 172.21.15.83:34212) 2024-09-19T07:27:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6055d9b640 1 -- 172.21.15.83:0/4130214046 learned_addr learned my addr 172.21.15.83:0/4130214046 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6056d9d640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60581eb530 0x7f60581496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f605659c640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60580c7650 0x7f60580c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6055d9b640 1 -- 172.21.15.83:0/4130214046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60580c7650 msgr2=0x7f60580c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6055d9b640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60580c7650 0x7f60580c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6055d9b640 1 -- 172.21.15.83:0/4130214046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60581eb530 msgr2=0x7f60581496c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6055d9b640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60581eb530 0x7f60581496c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6055d9b640 1 -- 172.21.15.83:0/4130214046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60580aa1a0 con 0x7f60580c5610 2024-09-19T07:27:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6056d9d640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60581eb530 0x7f60581496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:27:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6055d9b640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60580c5610 0x7f60581eaff0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f60480107e0 tx=0x7f6048010ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.652+0000 7f6046ffd640 1 -- 172.21.15.83:0/4130214046 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604800f040 con 0x7f60580c5610 2024-09-19T07:27:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.656+0000 7f6046ffd640 1 -- 172.21.15.83:0/4130214046 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f604800eb00 con 0x7f60580c5610 2024-09-19T07:27:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.656+0000 7f605cc7e640 1 -- 172.21.15.83:0/4130214046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6058073cb0 con 0x7f60580c5610 2024-09-19T07:27:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.656+0000 7f6046ffd640 1 -- 172.21.15.83:0/4130214046 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604801c400 con 0x7f60580c5610 2024-09-19T07:27:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.656+0000 7f605cc7e640 1 -- 172.21.15.83:0/4130214046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60581eaad0 con 0x7f60580c5610 2024-09-19T07:27:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.656+0000 7f605cc7e640 1 -- 172.21.15.83:0/4130214046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60581ea8c0 con 0x7f60580c5610 2024-09-19T07:27:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.656+0000 7f6046ffd640 1 -- 172.21.15.83:0/4130214046 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f60480253f0 con 0x7f60580c5610 2024-09-19T07:27:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.656+0000 7f6046ffd640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60380678c0 0x7f6038069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.656+0000 7f6046ffd640 1 -- 172.21.15.83:0/4130214046 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7f6048086a70 con 0x7f60580c5610 2024-09-19T07:27:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.656+0000 7f605659c640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60380678c0 0x7f6038069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:08.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.656+0000 7f605659c640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60380678c0 0x7f6038069d80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f604c001ec0 tx=0x7f604c001600 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:08.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.660+0000 7f6046ffd640 1 -- 172.21.15.83:0/4130214046 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6048018030 con 0x7f60580c5610 2024-09-19T07:27:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.860+0000 7f605cc7e640 1 -- 172.21.15.83:0/4130214046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60580c1ff0 con 0x7f60580c5610 2024-09-19T07:27:08.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f6046ffd640 1 -- 172.21.15.83:0/4130214046 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6532 (secure 0 0 0) 0x7f6048056e10 con 0x7f60580c5610 2024-09-19T07:27:08.863 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:08.863 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":63,"btime":"2024-09-19T07:25:49:257173+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:08.863 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 63 2024-09-19T07:27:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f605cc7e640 1 -- 172.21.15.83:0/4130214046 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60380678c0 msgr2=0x7f6038069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f605cc7e640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60380678c0 0x7f6038069d80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f604c001ec0 tx=0x7f604c001600 comp rx=0 tx=0).stop 2024-09-19T07:27:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f605cc7e640 1 -- 172.21.15.83:0/4130214046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60580c5610 msgr2=0x7f60581eaff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f605cc7e640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60580c5610 0x7f60581eaff0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f60480107e0 tx=0x7f6048010ca0 comp rx=0 tx=0).stop 2024-09-19T07:27:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f605cc7e640 1 -- 172.21.15.83:0/4130214046 shutdown_connections 2024-09-19T07:27:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f605cc7e640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60380678c0 0x7f6038069d80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f605cc7e640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60581eb530 0x7f60581496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f605cc7e640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60580c5610 0x7f60581eaff0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f605cc7e640 1 --2- 172.21.15.83:0/4130214046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60580c7650 0x7f60580c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f605cc7e640 1 -- 172.21.15.83:0/4130214046 >> 172.21.15.83:0/4130214046 conn(0x7f605806bc80 msgr2=0x7f6058153710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f605cc7e640 1 -- 172.21.15.83:0/4130214046 shutdown_connections 2024-09-19T07:27:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:08.864+0000 7f605cc7e640 1 -- 172.21.15.83:0/4130214046 wait complete. 2024-09-19T07:27:08.882 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:27:10.883 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:11.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.016+0000 7f7913d06640 1 Processor -- start 2024-09-19T07:27:11.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.016+0000 7f7913d06640 1 -- start start 2024-09-19T07:27:11.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7913d06640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f790c1592f0 0x7f790c0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:11.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7913d06640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f790c156200 con 0x7f790c155d70 2024-09-19T07:27:11.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7913d06640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f790c1564e0 con 0x7f790c0c6120 2024-09-19T07:27:11.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7913d06640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f790c0c9930 con 0x7f790c1592f0 2024-09-19T07:27:11.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f79112a1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f790c155d70 0x7f790c157ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59752/0 (socket says 172.21.15.83:59752) 2024-09-19T07:27:11.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f79112a1640 1 -- 172.21.15.83:0/1651261068 learned_addr learned my addr 172.21.15.83:0/1651261068 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:11.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f79122a3640 1 --2- 172.21.15.83:0/1651261068 >> v2:172.21.15.83:3300/0 conn(0x7f790c1592f0 0x7f790c0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:11.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f79122a3640 1 -- 172.21.15.83:0/1651261068 >> v1:172.21.15.184:6789/0 conn(0x7f790c0c6120 legacy=0x7f790c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:11.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f79122a3640 1 -- 172.21.15.83:0/1651261068 >> v1:172.21.15.83:6789/0 conn(0x7f790c155d70 legacy=0x7f790c157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:11.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f79122a3640 1 -- 172.21.15.83:0/1651261068 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790c1567c0 con 0x7f790c1592f0 2024-09-19T07:27:11.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f79122a3640 1 --2- 172.21.15.83:0/1651261068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c1592f0 0x7f790c0c6500 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7904009d90 tx=0x7f790401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3b6339f6f5223e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:11.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7910aa0640 1 -- 172.21.15.83:0/1651261068 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79040b2070 con 0x7f790c1592f0 2024-09-19T07:27:11.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7910aa0640 1 -- 172.21.15.83:0/1651261068 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79040a5760 con 0x7f790c1592f0 2024-09-19T07:27:11.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7910aa0640 1 -- 172.21.15.83:0/1651261068 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79040a5b30 con 0x7f790c1592f0 2024-09-19T07:27:11.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7913d06640 1 -- 172.21.15.83:0/1651261068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c1592f0 msgr2=0x7f790c0c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:11.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7913d06640 1 --2- 172.21.15.83:0/1651261068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c1592f0 0x7f790c0c6500 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7904009d90 tx=0x7f790401c8a0 comp rx=0 tx=0).stop 2024-09-19T07:27:11.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7913d06640 1 -- 172.21.15.83:0/1651261068 shutdown_connections 2024-09-19T07:27:11.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7913d06640 1 --2- 172.21.15.83:0/1651261068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c1592f0 0x7f790c0c6500 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:11.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7913d06640 1 -- 172.21.15.83:0/1651261068 >> 172.21.15.83:0/1651261068 conn(0x7f790c06bc80 msgr2=0x7f790c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:11.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7913d06640 1 -- 172.21.15.83:0/1651261068 shutdown_connections 2024-09-19T07:27:11.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.020+0000 7f7913d06640 1 -- 172.21.15.83:0/1651261068 wait complete. 2024-09-19T07:27:11.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7913d06640 1 Processor -- start 2024-09-19T07:27:11.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7913d06640 1 -- start start 2024-09-19T07:27:11.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7913d06640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0c6120 0x7f790c148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:11.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7913d06640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f790c1592f0 0x7f790c145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:11.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7913d06640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f790c14f850 0x7f790c146490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:11.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7913d06640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f790c164030 con 0x7f790c0c6120 2024-09-19T07:27:11.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7913d06640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f790c163eb0 con 0x7f790c14f850 2024-09-19T07:27:11.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7913d06640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f790c1641b0 con 0x7f790c1592f0 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7911aa2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0c6120 0x7f790c148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7911aa2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0c6120 0x7f790c148080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39614/0 (socket says 172.21.15.83:39614) 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7911aa2640 1 -- 172.21.15.83:0/3823858223 learned_addr learned my addr 172.21.15.83:0/3823858223 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f79112a1640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f790c1592f0 0x7f790c145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7911aa2640 1 -- 172.21.15.83:0/3823858223 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f790c1592f0 msgr2=0x7f790c145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7911aa2640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f790c1592f0 0x7f790c145f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7911aa2640 1 -- 172.21.15.83:0/3823858223 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f790c14f850 msgr2=0x7f790c146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7911aa2640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f790c14f850 0x7f790c146490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7911aa2640 1 -- 172.21.15.83:0/3823858223 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790c1469d0 con 0x7f790c0c6120 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f79112a1640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f790c1592f0 0x7f790c145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7911aa2640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0c6120 0x7f790c148080 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f790800c480 tx=0x7f790800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f79027fc640 1 -- 172.21.15.83:0/3823858223 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7908012830 con 0x7f790c0c6120 2024-09-19T07:27:11.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f79027fc640 1 -- 172.21.15.83:0/3823858223 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7908017070 con 0x7f790c0c6120 2024-09-19T07:27:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7913d06640 1 -- 172.21.15.83:0/3823858223 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f790c0c1b00 con 0x7f790c0c6120 2024-09-19T07:27:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f79027fc640 1 -- 172.21.15.83:0/3823858223 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7908012c30 con 0x7f790c0c6120 2024-09-19T07:27:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7913d06640 1 -- 172.21.15.83:0/3823858223 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f790c0c20c0 con 0x7f790c0c6120 2024-09-19T07:27:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.024+0000 7f7913d06640 1 -- 172.21.15.83:0/3823858223 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f790c0c6500 con 0x7f790c0c6120 2024-09-19T07:27:11.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.028+0000 7f79027fc640 1 -- 172.21.15.83:0/3823858223 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f790801c460 con 0x7f790c0c6120 2024-09-19T07:27:11.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.028+0000 7f79027fc640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78f00678f0 0x7f78f0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:11.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.028+0000 7f79112a1640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78f00678f0 0x7f78f0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:11.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.028+0000 7f79027fc640 1 -- 172.21.15.83:0/3823858223 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7f79080860d0 con 0x7f790c0c6120 2024-09-19T07:27:11.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.028+0000 7f79112a1640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78f00678f0 0x7f78f0069db0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f790c149e00 tx=0x7f78fc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:11.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.032+0000 7f79027fc640 1 -- 172.21.15.83:0/3823858223 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f79080565a0 con 0x7f790c0c6120 2024-09-19T07:27:11.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 -- 172.21.15.83:0/3823858223 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f790c0c96a0 con 0x7f790c0c6120 2024-09-19T07:27:11.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f79027fc640 1 -- 172.21.15.83:0/3823858223 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6532 (secure 0 0 0) 0x7f7908055ea0 con 0x7f790c0c6120 2024-09-19T07:27:11.227 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:11.227 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":63,"btime":"2024-09-19T07:25:49:257173+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:11.227 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 63 2024-09-19T07:27:11.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 -- 172.21.15.83:0/3823858223 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78f00678f0 msgr2=0x7f78f0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:11.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78f00678f0 0x7f78f0069db0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f790c149e00 tx=0x7f78fc009040 comp rx=0 tx=0).stop 2024-09-19T07:27:11.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 -- 172.21.15.83:0/3823858223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0c6120 msgr2=0x7f790c148080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:11.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0c6120 0x7f790c148080 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f790800c480 tx=0x7f790800c940 comp rx=0 tx=0).stop 2024-09-19T07:27:11.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 -- 172.21.15.83:0/3823858223 shutdown_connections 2024-09-19T07:27:11.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78f00678f0 0x7f78f0069db0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:11.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f790c14f850 0x7f790c146490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:11.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f790c1592f0 0x7f790c145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:11.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 --2- 172.21.15.83:0/3823858223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0c6120 0x7f790c148080 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:11.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 -- 172.21.15.83:0/3823858223 >> 172.21.15.83:0/3823858223 conn(0x7f790c06bc80 msgr2=0x7f790c159700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:11.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 -- 172.21.15.83:0/3823858223 shutdown_connections 2024-09-19T07:27:11.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:11.228+0000 7f7913d06640 1 -- 172.21.15.83:0/3823858223 wait complete. 2024-09-19T07:27:11.244 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:27:13.245 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdb5d3d640 1 Processor -- start 2024-09-19T07:27:13.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdb5d3d640 1 -- start start 2024-09-19T07:27:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdb5d3d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbdb00c42e0 0x7fbdb006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdb5d3d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbdb00abb30 0x7fbdb0153060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdb5d3d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdb01592c0 con 0x7fbdb01566b0 2024-09-19T07:27:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdb5d3d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fbdb01595f0 con 0x7fbdb00abb30 2024-09-19T07:27:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdb5d3d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbdb006e900 con 0x7fbdb00c42e0 2024-09-19T07:27:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdaffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbdb00abb30 0x7fbdb0153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdaffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbdb00abb30 0x7fbdb0153060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39626/0 (socket says 172.21.15.83:39626) 2024-09-19T07:27:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdaeffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbdb01566b0 0x7fbdb0159a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59760/0 (socket says 172.21.15.83:59760) 2024-09-19T07:27:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdaffff640 1 -- 172.21.15.83:0/1611583263 learned_addr learned my addr 172.21.15.83:0/1611583263 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdaf7fe640 1 --2- 172.21.15.83:0/1611583263 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbdb00c42e0 0x7fbdb006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdaffff640 1 -- 172.21.15.83:0/1611583263 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbdb00c42e0 msgr2=0x7fbdb006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdaffff640 1 --2- 172.21.15.83:0/1611583263 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbdb00c42e0 0x7fbdb006e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:13.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdaffff640 1 -- 172.21.15.83:0/1611583263 >> v1:172.21.15.83:6789/0 conn(0x7fbdb01566b0 legacy=0x7fbdb0159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:13.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.379+0000 7fbdaffff640 1 -- 172.21.15.83:0/1611583263 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb006ebe0 con 0x7fbdb00abb30 2024-09-19T07:27:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdaffff640 1 --2- 172.21.15.83:0/1611583263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdb00abb30 0x7fbdb0153060 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fbda8009e50 tx=0x7fbda801ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6feb8ee5643879e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdae7fc640 1 -- 172.21.15.83:0/1611583263 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda80a58f0 con 0x7fbdb00abb30 2024-09-19T07:27:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdae7fc640 1 -- 172.21.15.83:0/1611583263 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbda80a5a90 con 0x7fbdb00abb30 2024-09-19T07:27:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdae7fc640 1 -- 172.21.15.83:0/1611583263 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda80a5e60 con 0x7fbdb00abb30 2024-09-19T07:27:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/1611583263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdb00abb30 msgr2=0x7fbdb0153060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 --2- 172.21.15.83:0/1611583263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdb00abb30 0x7fbdb0153060 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fbda8009e50 tx=0x7fbda801ca00 comp rx=0 tx=0).stop 2024-09-19T07:27:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/1611583263 shutdown_connections 2024-09-19T07:27:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 --2- 172.21.15.83:0/1611583263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdb00abb30 0x7fbdb0153060 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 --2- 172.21.15.83:0/1611583263 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbdb00c42e0 0x7fbdb006e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/1611583263 >> 172.21.15.83:0/1611583263 conn(0x7fbdb006bc80 msgr2=0x7fbdb006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/1611583263 shutdown_connections 2024-09-19T07:27:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/1611583263 wait complete. 2024-09-19T07:27:13.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 Processor -- start 2024-09-19T07:27:13.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 -- start start 2024-09-19T07:27:13.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdb00abb30 0x7fbdb0154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:13.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdb00c42e0 0x7fbdb00c6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:13.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbdb01566b0 0x7fbdb00c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:13.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb00c4840 con 0x7fbdb00c42e0 2024-09-19T07:27:13.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb00c46c0 con 0x7fbdb00abb30 2024-09-19T07:27:13.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdb5d3d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbdb0162dd0 con 0x7fbdb01566b0 2024-09-19T07:27:13.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdaeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdb00c42e0 0x7fbdb00c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:13.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdaeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdb00c42e0 0x7fbdb00c6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39636/0 (socket says 172.21.15.83:39636) 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.383+0000 7fbdaeffd640 1 -- 172.21.15.83:0/3041743094 learned_addr learned my addr 172.21.15.83:0/3041743094 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdaffff640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbdb01566b0 0x7fbdb00c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdaf7fe640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdb00abb30 0x7fbdb0154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdaeffd640 1 -- 172.21.15.83:0/3041743094 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbdb01566b0 msgr2=0x7fbdb00c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdaeffd640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbdb01566b0 0x7fbdb00c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdaeffd640 1 -- 172.21.15.83:0/3041743094 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdb00abb30 msgr2=0x7fbdb0154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdaeffd640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdb00abb30 0x7fbdb0154d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdaeffd640 1 -- 172.21.15.83:0/3041743094 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb00714b0 con 0x7fbdb00c42e0 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdaeffd640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdb00c42e0 0x7fbdb00c6960 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fbda400c370 tx=0x7fbda400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbd97fff640 1 -- 172.21.15.83:0/3041743094 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda40074e0 con 0x7fbdb00c42e0 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbd97fff640 1 -- 172.21.15.83:0/3041743094 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbda4007680 con 0x7fbdb00c42e0 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/3041743094 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdb00710f0 con 0x7fbdb00c42e0 2024-09-19T07:27:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbd97fff640 1 -- 172.21.15.83:0/3041743094 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda4004a30 con 0x7fbdb00c42e0 2024-09-19T07:27:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/3041743094 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdb0155920 con 0x7fbdb00c42e0 2024-09-19T07:27:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/3041743094 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdb0070e80 con 0x7fbdb00c42e0 2024-09-19T07:27:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbd97fff640 1 -- 172.21.15.83:0/3041743094 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbda401a020 con 0x7fbdb00c42e0 2024-09-19T07:27:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbd97fff640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbd9c0678c0 0x7fbd9c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbd97fff640 1 -- 172.21.15.83:0/3041743094 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7fbda4085a90 con 0x7fbdb00c42e0 2024-09-19T07:27:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdaf7fe640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbd9c0678c0 0x7fbd9c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.387+0000 7fbdaf7fe640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbd9c0678c0 0x7fbd9c069d80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0151d50 tx=0x7fbda0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:13.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.391+0000 7fbd97fff640 1 -- 172.21.15.83:0/3041743094 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbda4055f60 con 0x7fbdb00c42e0 2024-09-19T07:27:13.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.591+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/3041743094 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdb0153060 con 0x7fbdb00c42e0 2024-09-19T07:27:13.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.591+0000 7fbd97fff640 1 -- 172.21.15.83:0/3041743094 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6532 (secure 0 0 0) 0x7fbda4055860 con 0x7fbdb00c42e0 2024-09-19T07:27:13.593 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:13.593 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":63,"btime":"2024-09-19T07:25:49:257173+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:13.593 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 63 2024-09-19T07:27:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.595+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/3041743094 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbd9c0678c0 msgr2=0x7fbd9c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.595+0000 7fbdb5d3d640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbd9c0678c0 0x7fbd9c069d80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0151d50 tx=0x7fbda0008040 comp rx=0 tx=0).stop 2024-09-19T07:27:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.595+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/3041743094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdb00c42e0 msgr2=0x7fbdb00c6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.595+0000 7fbdb5d3d640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdb00c42e0 0x7fbdb00c6960 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fbda400c370 tx=0x7fbda400c830 comp rx=0 tx=0).stop 2024-09-19T07:27:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.595+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/3041743094 shutdown_connections 2024-09-19T07:27:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.595+0000 7fbdb5d3d640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbd9c0678c0 0x7fbd9c069d80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.595+0000 7fbdb5d3d640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbdb01566b0 0x7fbdb00c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.595+0000 7fbdb5d3d640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdb00c42e0 0x7fbdb00c6960 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.595+0000 7fbdb5d3d640 1 --2- 172.21.15.83:0/3041743094 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbdb00abb30 0x7fbdb0154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.595+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/3041743094 >> 172.21.15.83:0/3041743094 conn(0x7fbdb006bc80 msgr2=0x7fbdb00ac200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.595+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/3041743094 shutdown_connections 2024-09-19T07:27:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:13.595+0000 7fbdb5d3d640 1 -- 172.21.15.83:0/3041743094 wait complete. 2024-09-19T07:27:13.613 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:27:15.614 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:15.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff19077c640 1 Processor -- start 2024-09-19T07:27:15.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff19077c640 1 -- start start 2024-09-19T07:27:15.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff19077c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff188071100 0x7ff1881545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:15.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff19077c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff18806dea0 0x7ff18806e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:15.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff19077c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1880a9e60 con 0x7ff1880c28e0 2024-09-19T07:27:15.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff19077c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff188071b50 con 0x7ff18806dea0 2024-09-19T07:27:15.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff19077c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff1880c1ff0 con 0x7ff188071100 2024-09-19T07:27:15.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff18ed19640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff18806dea0 0x7ff18806e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:15.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff18ed19640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff18806dea0 0x7ff18806e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39644/0 (socket says 172.21.15.83:39644) 2024-09-19T07:27:15.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff18ed19640 1 -- 172.21.15.83:0/4053916894 learned_addr learned my addr 172.21.15.83:0/4053916894 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:15.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff18ed19640 1 -- 172.21.15.83:0/4053916894 >> v2:172.21.15.184:3300/0 conn(0x7ff188071100 msgr2=0x7ff1881545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:15.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff18ed19640 1 --2- 172.21.15.83:0/4053916894 >> v2:172.21.15.184:3300/0 conn(0x7ff188071100 0x7ff1881545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:15.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff18ed19640 1 -- 172.21.15.83:0/4053916894 >> v1:172.21.15.184:6789/0 conn(0x7ff1880c28e0 legacy=0x7ff188150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:15.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff18ed19640 1 -- 172.21.15.83:0/4053916894 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1880aa140 con 0x7ff18806dea0 2024-09-19T07:27:15.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.747+0000 7ff18ed19640 1 --2- 172.21.15.83:0/4053916894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff18806dea0 0x7ff18806e330 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff18806f620 tx=0x7ff18001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffe33dfb407b5c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:15.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18d516640 1 -- 172.21.15.83:0/4053916894 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1800a56d0 con 0x7ff18806dea0 2024-09-19T07:27:15.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18d516640 1 -- 172.21.15.83:0/4053916894 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1800a5870 con 0x7ff18806dea0 2024-09-19T07:27:15.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18d516640 1 -- 172.21.15.83:0/4053916894 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1800a5c40 con 0x7ff18806dea0 2024-09-19T07:27:15.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 -- 172.21.15.83:0/4053916894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff18806dea0 msgr2=0x7ff18806e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:15.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 --2- 172.21.15.83:0/4053916894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff18806dea0 0x7ff18806e330 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff18806f620 tx=0x7ff18001c920 comp rx=0 tx=0).stop 2024-09-19T07:27:15.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 -- 172.21.15.83:0/4053916894 shutdown_connections 2024-09-19T07:27:15.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 --2- 172.21.15.83:0/4053916894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff18806dea0 0x7ff18806e330 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:15.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 --2- 172.21.15.83:0/4053916894 >> v2:172.21.15.184:3300/0 conn(0x7ff188071100 0x7ff1881545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:15.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 -- 172.21.15.83:0/4053916894 >> 172.21.15.83:0/4053916894 conn(0x7ff18806bc80 msgr2=0x7ff18806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:15.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 -- 172.21.15.83:0/4053916894 shutdown_connections 2024-09-19T07:27:15.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 -- 172.21.15.83:0/4053916894 wait complete. 2024-09-19T07:27:15.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 Processor -- start 2024-09-19T07:27:15.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 -- start start 2024-09-19T07:27:15.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff18806dea0 0x7ff18814bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:15.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff188071100 0x7ff18814c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:15.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1880c28e0 0x7ff18815c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:15.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff188071660 con 0x7ff1880c28e0 2024-09-19T07:27:15.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff1880714e0 con 0x7ff18806dea0 2024-09-19T07:27:15.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff188160c60 con 0x7ff188071100 2024-09-19T07:27:15.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18ed19640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1880c28e0 0x7ff18815c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18ed19640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1880c28e0 0x7ff18815c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39646/0 (socket says 172.21.15.83:39646) 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18ed19640 1 -- 172.21.15.83:0/3239862478 learned_addr learned my addr 172.21.15.83:0/3239862478 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18e518640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff18806dea0 0x7ff18814bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18dd17640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff188071100 0x7ff18814c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18ed19640 1 -- 172.21.15.83:0/3239862478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff188071100 msgr2=0x7ff18814c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18ed19640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff188071100 0x7ff18814c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18ed19640 1 -- 172.21.15.83:0/3239862478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff18806dea0 msgr2=0x7ff18814bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18ed19640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff18806dea0 0x7ff18814bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18ed19640 1 -- 172.21.15.83:0/3239862478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff18814f160 con 0x7ff1880c28e0 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18dd17640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff188071100 0x7ff18814c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff18ed19640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1880c28e0 0x7ff18815c9b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff18001a550 tx=0x7ff1800196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff17effd640 1 -- 172.21.15.83:0/3239862478 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1800bc020 con 0x7ff1880c28e0 2024-09-19T07:27:15.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff17effd640 1 -- 172.21.15.83:0/3239862478 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1800b8670 con 0x7ff1880c28e0 2024-09-19T07:27:15.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 -- 172.21.15.83:0/3239862478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff18814cc90 con 0x7ff1880c28e0 2024-09-19T07:27:15.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff17effd640 1 -- 172.21.15.83:0/3239862478 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1800b8a60 con 0x7ff1880c28e0 2024-09-19T07:27:15.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.751+0000 7ff19077c640 1 -- 172.21.15.83:0/3239862478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff188157680 con 0x7ff1880c28e0 2024-09-19T07:27:15.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.755+0000 7ff19077c640 1 -- 172.21.15.83:0/3239862478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff18806e330 con 0x7ff1880c28e0 2024-09-19T07:27:15.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.755+0000 7ff17effd640 1 -- 172.21.15.83:0/3239862478 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff1800bf020 con 0x7ff1880c28e0 2024-09-19T07:27:15.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.755+0000 7ff17effd640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff16c0678c0 0x7ff16c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:15.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.755+0000 7ff17effd640 1 -- 172.21.15.83:0/3239862478 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7497+0+0 (secure 0 0 0) 0x7ff180121550 con 0x7ff1880c28e0 2024-09-19T07:27:15.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.755+0000 7ff18e518640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff16c0678c0 0x7ff16c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:15.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.755+0000 7ff18e518640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff16c0678c0 0x7ff16c069d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff178006fd0 tx=0x7ff178008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:15.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.759+0000 7ff17effd640 1 -- 172.21.15.83:0/3239862478 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff1800c6030 con 0x7ff1880c28e0 2024-09-19T07:27:15.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.955+0000 7ff19077c640 1 -- 172.21.15.83:0/3239862478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1880c4f30 con 0x7ff1880c28e0 2024-09-19T07:27:15.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff17effd640 1 -- 172.21.15.83:0/3239862478 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6532 (secure 0 0 0) 0x7ff18814f160 con 0x7ff1880c28e0 2024-09-19T07:27:15.959 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:15.959 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":63,"btime":"2024-09-19T07:25:49:257173+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:25:31.169380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5607},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5607":{"gid":5607,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6835/3763485069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3763485069},{"type":"v1","addr":"172.21.15.83:6835","nonce":3763485069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5607,"qdb_cluster":[5607]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:15.960 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 63 2024-09-19T07:27:15.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff19077c640 1 -- 172.21.15.83:0/3239862478 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff16c0678c0 msgr2=0x7ff16c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:15.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff19077c640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff16c0678c0 0x7ff16c069d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff178006fd0 tx=0x7ff178008040 comp rx=0 tx=0).stop 2024-09-19T07:27:15.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff19077c640 1 -- 172.21.15.83:0/3239862478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1880c28e0 msgr2=0x7ff18815c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:15.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff19077c640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1880c28e0 0x7ff18815c9b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff18001a550 tx=0x7ff1800196c0 comp rx=0 tx=0).stop 2024-09-19T07:27:15.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff19077c640 1 -- 172.21.15.83:0/3239862478 shutdown_connections 2024-09-19T07:27:15.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff19077c640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff16c0678c0 0x7ff16c069d80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:15.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff19077c640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1880c28e0 0x7ff18815c9b0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:15.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff19077c640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff188071100 0x7ff18814c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:15.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff19077c640 1 --2- 172.21.15.83:0/3239862478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff18806dea0 0x7ff18814bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:15.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff19077c640 1 -- 172.21.15.83:0/3239862478 >> 172.21.15.83:0/3239862478 conn(0x7ff18806bc80 msgr2=0x7ff18814f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:15.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff19077c640 1 -- 172.21.15.83:0/3239862478 shutdown_connections 2024-09-19T07:27:15.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:15.959+0000 7ff19077c640 1 -- 172.21.15.83:0/3239862478 wait complete. 2024-09-19T07:27:15.979 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:27:17.979 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb49dd7640 1 Processor -- start 2024-09-19T07:27:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb49dd7640 1 -- start start 2024-09-19T07:27:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb49dd7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcb44155d70 0x7fcb440c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb49dd7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcb441592f0 0x7fcb44157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb49dd7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb44156200 con 0x7fcb440c6120 2024-09-19T07:27:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb49dd7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fcb440c9930 con 0x7fcb441592f0 2024-09-19T07:27:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb49dd7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcb441564e0 con 0x7fcb44155d70 2024-09-19T07:27:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb43fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcb441592f0 0x7fcb44157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb43fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcb441592f0 0x7fcb44157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39660/0 (socket says 172.21.15.83:39660) 2024-09-19T07:27:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb43fff640 1 -- 172.21.15.83:0/4079506282 learned_addr learned my addr 172.21.15.83:0/4079506282 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb42ffd640 1 --2- 172.21.15.83:0/4079506282 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcb44155d70 0x7fcb440c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb43fff640 1 -- 172.21.15.83:0/4079506282 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcb44155d70 msgr2=0x7fcb440c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb43fff640 1 --2- 172.21.15.83:0/4079506282 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcb44155d70 0x7fcb440c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb43fff640 1 -- 172.21.15.83:0/4079506282 >> v1:172.21.15.83:6789/0 conn(0x7fcb440c6120 legacy=0x7fcb441575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:18.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.123+0000 7fcb43fff640 1 -- 172.21.15.83:0/4079506282 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb441567c0 con 0x7fcb441592f0 2024-09-19T07:27:18.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb43fff640 1 --2- 172.21.15.83:0/4079506282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb441592f0 0x7fcb44157ce0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c009e50 tx=0x7fcb3c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ba029c611cca1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:18.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb427fc640 1 -- 172.21.15.83:0/4079506282 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c0a5700 con 0x7fcb441592f0 2024-09-19T07:27:18.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb427fc640 1 -- 172.21.15.83:0/4079506282 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb3c0a58a0 con 0x7fcb441592f0 2024-09-19T07:27:18.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb427fc640 1 -- 172.21.15.83:0/4079506282 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c0a5c70 con 0x7fcb441592f0 2024-09-19T07:27:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 -- 172.21.15.83:0/4079506282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb441592f0 msgr2=0x7fcb44157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 --2- 172.21.15.83:0/4079506282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb441592f0 0x7fcb44157ce0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c009e50 tx=0x7fcb3c01ca30 comp rx=0 tx=0).stop 2024-09-19T07:27:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 -- 172.21.15.83:0/4079506282 shutdown_connections 2024-09-19T07:27:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 --2- 172.21.15.83:0/4079506282 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb441592f0 0x7fcb44157ce0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 --2- 172.21.15.83:0/4079506282 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcb44155d70 0x7fcb440c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 -- 172.21.15.83:0/4079506282 >> 172.21.15.83:0/4079506282 conn(0x7fcb4406bc80 msgr2=0x7fcb4406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 -- 172.21.15.83:0/4079506282 shutdown_connections 2024-09-19T07:27:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 -- 172.21.15.83:0/4079506282 wait complete. 2024-09-19T07:27:18.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 Processor -- start 2024-09-19T07:27:18.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 -- start start 2024-09-19T07:27:18.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb440c6120 0x7fcb440cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:18.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcb441592f0 0x7fcb440ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb441494f0 0x7fcb441f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcb44153860 con 0x7fcb441494f0 2024-09-19T07:27:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcb44162ea0 con 0x7fcb440c6120 2024-09-19T07:27:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcb440cc120 con 0x7fcb441592f0 2024-09-19T07:27:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb43fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb441494f0 0x7fcb441f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb43fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb441494f0 0x7fcb441f2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39670/0 (socket says 172.21.15.83:39670) 2024-09-19T07:27:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb43fff640 1 -- 172.21.15.83:0/1255957511 learned_addr learned my addr 172.21.15.83:0/1255957511 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb42ffd640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcb441592f0 0x7fcb440ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb437fe640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb440c6120 0x7fcb440cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb43fff640 1 -- 172.21.15.83:0/1255957511 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcb441592f0 msgr2=0x7fcb440ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb43fff640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcb441592f0 0x7fcb440ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb43fff640 1 -- 172.21.15.83:0/1255957511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb440c6120 msgr2=0x7fcb440cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb43fff640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb440c6120 0x7fcb440cbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb43fff640 1 -- 172.21.15.83:0/1255957511 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb440cadf0 con 0x7fcb441494f0 2024-09-19T07:27:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb43fff640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb441494f0 0x7fcb441f2050 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0af040 tx=0x7fcb3c0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb2bfff640 1 -- 172.21.15.83:0/1255957511 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c0bc020 con 0x7fcb441494f0 2024-09-19T07:27:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb2bfff640 1 -- 172.21.15.83:0/1255957511 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb3c0b8670 con 0x7fcb441494f0 2024-09-19T07:27:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 -- 172.21.15.83:0/1255957511 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb4414afa0 con 0x7fcb441494f0 2024-09-19T07:27:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb2bfff640 1 -- 172.21.15.83:0/1255957511 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c009510 con 0x7fcb441494f0 2024-09-19T07:27:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.127+0000 7fcb49dd7640 1 -- 172.21.15.83:0/1255957511 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb440c2330 con 0x7fcb441494f0 2024-09-19T07:27:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.131+0000 7fcb49dd7640 1 -- 172.21.15.83:0/1255957511 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb440c2570 con 0x7fcb441494f0 2024-09-19T07:27:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.131+0000 7fcb2bfff640 1 -- 172.21.15.83:0/1255957511 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcb3c0bf020 con 0x7fcb441494f0 2024-09-19T07:27:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.131+0000 7fcb2bfff640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcb300678f0 0x7fcb30069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.131+0000 7fcb437fe640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcb300678f0 0x7fcb30069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.131+0000 7fcb2bfff640 1 -- 172.21.15.83:0/1255957511 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7fcb3c121080 con 0x7fcb441494f0 2024-09-19T07:27:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.131+0000 7fcb437fe640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcb300678f0 0x7fcb30069db0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcb34001810 tx=0x7fcb34009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:18.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.135+0000 7fcb2bfff640 1 -- 172.21.15.83:0/1255957511 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcb440c2570 con 0x7fcb441494f0 2024-09-19T07:27:18.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.339+0000 7fcb49dd7640 1 -- 172.21.15.83:0/1255957511 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb440c6500 con 0x7fcb441494f0 2024-09-19T07:27:18.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.339+0000 7fcb2bfff640 1 -- 172.21.15.83:0/1255957511 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5699 (secure 0 0 0) 0x7fcb3c0c6030 con 0x7fcb441494f0 2024-09-19T07:27:18.340 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:18.340 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":64,"btime":"2024-09-19T07:27:18:018915+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:18.018910+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:replay","state_seq":1,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:18.341 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 64 2024-09-19T07:27:18.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.343+0000 7fcb49dd7640 1 -- 172.21.15.83:0/1255957511 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcb300678f0 msgr2=0x7fcb30069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:18.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.343+0000 7fcb49dd7640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcb300678f0 0x7fcb30069db0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcb34001810 tx=0x7fcb34009040 comp rx=0 tx=0).stop 2024-09-19T07:27:18.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.343+0000 7fcb49dd7640 1 -- 172.21.15.83:0/1255957511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb441494f0 msgr2=0x7fcb441f2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:18.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.343+0000 7fcb49dd7640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb441494f0 0x7fcb441f2050 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0af040 tx=0x7fcb3c0197d0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.343+0000 7fcb49dd7640 1 -- 172.21.15.83:0/1255957511 shutdown_connections 2024-09-19T07:27:18.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.343+0000 7fcb49dd7640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcb300678f0 0x7fcb30069db0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.343+0000 7fcb49dd7640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb441494f0 0x7fcb441f2050 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.343+0000 7fcb49dd7640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcb441592f0 0x7fcb440ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.343+0000 7fcb49dd7640 1 --2- 172.21.15.83:0/1255957511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb440c6120 0x7fcb440cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.343+0000 7fcb49dd7640 1 -- 172.21.15.83:0/1255957511 >> 172.21.15.83:0/1255957511 conn(0x7fcb4406bc80 msgr2=0x7fcb441574d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:18.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.343+0000 7fcb49dd7640 1 -- 172.21.15.83:0/1255957511 shutdown_connections 2024-09-19T07:27:18.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.343+0000 7fcb49dd7640 1 -- 172.21.15.83:0/1255957511 wait complete. 2024-09-19T07:27:18.360 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-19T07:27:18.360 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:27:18.360 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:18.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.483+0000 7fbe32602640 1 Processor -- start 2024-09-19T07:27:18.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe32602640 1 -- start start 2024-09-19T07:27:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe32602640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbe2c0c79d0 0x7fbe2c06ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe32602640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe2c1534f0 0x7fbe2c14fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe32602640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe2c0c6b70 con 0x7fbe2c06cef0 2024-09-19T07:27:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe32602640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fbe2c06f730 con 0x7fbe2c0c79d0 2024-09-19T07:27:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe32602640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbe2c153980 con 0x7fbe2c1534f0 2024-09-19T07:27:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbe2c06cef0 0x7fbe2c0c6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59776/0 (socket says 172.21.15.83:59776) 2024-09-19T07:27:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2bfff640 1 -- 172.21.15.83:0/3987945765 learned_addr learned my addr 172.21.15.83:0/3987945765 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2b7fe640 1 --2- 172.21.15.83:0/3987945765 >> v2:172.21.15.83:3300/0 conn(0x7fbe2c0c79d0 0x7fbe2c06ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe30b9f640 1 --2- 172.21.15.83:0/3987945765 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe2c1534f0 0x7fbe2c14fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:18.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2affd640 1 -- 172.21.15.83:0/3987945765 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3255742089 0 0) 0x7fbe2c0c6b70 con 0x7fbe2c06cef0 2024-09-19T07:27:18.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2affd640 1 -- 172.21.15.83:0/3987945765 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe18003530 con 0x7fbe2c06cef0 2024-09-19T07:27:18.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2b7fe640 1 -- 172.21.15.83:0/3987945765 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe2c1534f0 msgr2=0x7fbe2c14fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:18.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2b7fe640 1 --2- 172.21.15.83:0/3987945765 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe2c1534f0 0x7fbe2c14fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2b7fe640 1 -- 172.21.15.83:0/3987945765 >> v1:172.21.15.83:6789/0 conn(0x7fbe2c06cef0 legacy=0x7fbe2c0c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:18.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2b7fe640 1 -- 172.21.15.83:0/3987945765 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2c0c7fb0 con 0x7fbe2c0c79d0 2024-09-19T07:27:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2b7fe640 1 --2- 172.21.15.83:0/3987945765 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe2c0c79d0 0x7fbe2c06ddd0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fbe140097c0 tx=0x7fbe1401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8df6f2a467cb34 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2affd640 1 -- 172.21.15.83:0/3987945765 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe140b0070 con 0x7fbe2c0c79d0 2024-09-19T07:27:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2affd640 1 -- 172.21.15.83:0/3987945765 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe140ab600 con 0x7fbe2c0c79d0 2024-09-19T07:27:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.487+0000 7fbe2affd640 1 -- 172.21.15.83:0/3987945765 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe140ab9d0 con 0x7fbe2c0c79d0 2024-09-19T07:27:18.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 -- 172.21.15.83:0/3987945765 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe2c0c79d0 msgr2=0x7fbe2c06ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:18.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 --2- 172.21.15.83:0/3987945765 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe2c0c79d0 0x7fbe2c06ddd0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fbe140097c0 tx=0x7fbe1401c380 comp rx=0 tx=0).stop 2024-09-19T07:27:18.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 -- 172.21.15.83:0/3987945765 shutdown_connections 2024-09-19T07:27:18.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 --2- 172.21.15.83:0/3987945765 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe2c0c79d0 0x7fbe2c06ddd0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 --2- 172.21.15.83:0/3987945765 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe2c1534f0 0x7fbe2c14fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 -- 172.21.15.83:0/3987945765 >> 172.21.15.83:0/3987945765 conn(0x7fbe2c0aa910 msgr2=0x7fbe2c0c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:18.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 -- 172.21.15.83:0/3987945765 shutdown_connections 2024-09-19T07:27:18.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 -- 172.21.15.83:0/3987945765 wait complete. 2024-09-19T07:27:18.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 Processor -- start 2024-09-19T07:27:18.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 -- start start 2024-09-19T07:27:18.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe2c06cef0 0x7fbe2c1ecb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:18.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe2c0c79d0 0x7fbe2c15bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:18.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe2c1534f0 0x7fbe2c0bd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:18.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c15fc10 con 0x7fbe2c1534f0 2024-09-19T07:27:18.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c150420 con 0x7fbe2c0c79d0 2024-09-19T07:27:18.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbe2c15fd90 con 0x7fbe2c06cef0 2024-09-19T07:27:18.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe30b9f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe2c1534f0 0x7fbe2c0bd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:18.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe30b9f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe2c1534f0 0x7fbe2c0bd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39700/0 (socket says 172.21.15.83:39700) 2024-09-19T07:27:18.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe30b9f640 1 -- 172.21.15.83:0/3418534982 learned_addr learned my addr 172.21.15.83:0/3418534982 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:18.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe2bfff640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe2c06cef0 0x7fbe2c1ecb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe2b7fe640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe2c0c79d0 0x7fbe2c15bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe30b9f640 1 -- 172.21.15.83:0/3418534982 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe2c06cef0 msgr2=0x7fbe2c1ecb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe30b9f640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe2c06cef0 0x7fbe2c1ecb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe30b9f640 1 -- 172.21.15.83:0/3418534982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe2c0c79d0 msgr2=0x7fbe2c15bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe30b9f640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe2c0c79d0 0x7fbe2c15bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe30b9f640 1 -- 172.21.15.83:0/3418534982 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2c0ad160 con 0x7fbe2c1534f0 2024-09-19T07:27:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe30b9f640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe2c1534f0 0x7fbe2c0bd580 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00e930 tx=0x7fbe1c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe28ff9640 1 -- 172.21.15.83:0/3418534982 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c017070 con 0x7fbe2c1534f0 2024-09-19T07:27:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe28ff9640 1 -- 172.21.15.83:0/3418534982 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe1c00c9a0 con 0x7fbe2c1534f0 2024-09-19T07:27:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 -- 172.21.15.83:0/3418534982 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2c1ec340 con 0x7fbe2c1534f0 2024-09-19T07:27:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe28ff9640 1 -- 172.21.15.83:0/3418534982 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c00cd70 con 0x7fbe2c1534f0 2024-09-19T07:27:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.491+0000 7fbe32602640 1 -- 172.21.15.83:0/3418534982 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe2c0bdac0 con 0x7fbe2c1534f0 2024-09-19T07:27:18.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.495+0000 7fbe32602640 1 -- 172.21.15.83:0/3418534982 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe2c06ddd0 con 0x7fbe2c1534f0 2024-09-19T07:27:18.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.495+0000 7fbe28ff9640 1 -- 172.21.15.83:0/3418534982 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbe1c01a070 con 0x7fbe2c1534f0 2024-09-19T07:27:18.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.495+0000 7fbe28ff9640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe0c0678f0 0x7fbe0c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:18.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.495+0000 7fbe28ff9640 1 -- 172.21.15.83:0/3418534982 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7fbe1c086710 con 0x7fbe2c1534f0 2024-09-19T07:27:18.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.495+0000 7fbe2bfff640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe0c0678f0 0x7fbe0c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:18.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.495+0000 7fbe2bfff640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe0c0678f0 0x7fbe0c069db0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbe24005c10 tx=0x7fbe24009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:18.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.499+0000 7fbe28ff9640 1 -- 172.21.15.83:0/3418534982 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbe1c056560 con 0x7fbe2c1534f0 2024-09-19T07:27:18.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.699+0000 7fbe32602640 1 -- 172.21.15.83:0/3418534982 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe2c0ad330 con 0x7fbe2c1534f0 2024-09-19T07:27:18.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.699+0000 7fbe28ff9640 1 -- 172.21.15.83:0/3418534982 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5699 (secure 0 0 0) 0x7fbe1c05a860 con 0x7fbe2c1534f0 2024-09-19T07:27:18.700 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:18.700 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":64,"btime":"2024-09-19T07:27:18:018915+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:18.018910+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:replay","state_seq":1,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:18.701 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 64 2024-09-19T07:27:18.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.703+0000 7fbe32602640 1 -- 172.21.15.83:0/3418534982 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe0c0678f0 msgr2=0x7fbe0c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:18.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.703+0000 7fbe32602640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe0c0678f0 0x7fbe0c069db0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbe24005c10 tx=0x7fbe24009040 comp rx=0 tx=0).stop 2024-09-19T07:27:18.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.703+0000 7fbe32602640 1 -- 172.21.15.83:0/3418534982 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe2c1534f0 msgr2=0x7fbe2c0bd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:18.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.703+0000 7fbe32602640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe2c1534f0 0x7fbe2c0bd580 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00e930 tx=0x7fbe1c00edf0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.703+0000 7fbe32602640 1 -- 172.21.15.83:0/3418534982 shutdown_connections 2024-09-19T07:27:18.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.703+0000 7fbe32602640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbe0c0678f0 0x7fbe0c069db0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.703+0000 7fbe32602640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe2c1534f0 0x7fbe2c0bd580 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.703+0000 7fbe32602640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe2c0c79d0 0x7fbe2c15bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.703+0000 7fbe32602640 1 --2- 172.21.15.83:0/3418534982 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbe2c06cef0 0x7fbe2c1ecb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:18.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.703+0000 7fbe32602640 1 -- 172.21.15.83:0/3418534982 >> 172.21.15.83:0/3418534982 conn(0x7fbe2c0aa910 msgr2=0x7fbe2c14d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:18.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.703+0000 7fbe32602640 1 -- 172.21.15.83:0/3418534982 shutdown_connections 2024-09-19T07:27:18.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:18.703+0000 7fbe32602640 1 -- 172.21.15.83:0/3418534982 wait complete. 2024-09-19T07:27:18.721 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:27:18:018915+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 64, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 64, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4979": { "addr": "172.21.15.184:6835/872719174", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 872719174, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 872719174, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4979, "incarnation": 64, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:27:18.018910+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4979 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 62, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5757": { "addr": "172.21.15.83:6837/455754420", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 455754420, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 455754420, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5757, "incarnation": 59, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:25:39.224528+0000", "qdb_cluster": [ 5757 ], "qdb_leader": 5757, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5757 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 28, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:27:19.224 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:19.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf17f06640 1 Processor -- start 2024-09-19T07:27:19.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf17f06640 1 -- start start 2024-09-19T07:27:19.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf17f06640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbf10071100 0x7fbf101545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf17f06640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf100a9e60 con 0x7fbf1006dea0 2024-09-19T07:27:19.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf17f06640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf100aa140 con 0x7fbf100c28e0 2024-09-19T07:27:19.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf17f06640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbf10071b50 con 0x7fbf10071100 2024-09-19T07:27:19.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf164a3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbf1006dea0 0x7fbf1006e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59786/0 (socket says 172.21.15.83:59786) 2024-09-19T07:27:19.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf164a3640 1 -- 172.21.15.83:0/4056437376 learned_addr learned my addr 172.21.15.83:0/4056437376 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:19.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf15ca2640 1 --2- 172.21.15.83:0/4056437376 >> v2:172.21.15.184:3300/0 conn(0x7fbf10071100 0x7fbf101545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3676196199 0 0) 0x7fbf100a9e60 con 0x7fbf1006dea0 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbefc003530 con 0x7fbf1006dea0 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3149262446 0 0) 0x7fbefc003530 con 0x7fbf1006dea0 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 >> v2:172.21.15.184:3300/0 conn(0x7fbf10071100 msgr2=0x7fbf101545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 --2- 172.21.15.83:0/4056437376 >> v2:172.21.15.184:3300/0 conn(0x7fbf10071100 0x7fbf101545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 >> v1:172.21.15.184:6789/0 conn(0x7fbf100c28e0 legacy=0x7fbf10150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf100c2240 con 0x7fbf1006dea0 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fbf08001a00 con 0x7fbf1006dea0 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 >> v1:172.21.15.83:6789/0 conn(0x7fbf1006dea0 legacy=0x7fbf1006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 --2- 172.21.15.83:0/4056437376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbefc0039c0 0x7fbefc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 --2- 172.21.15.83:0/4056437376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 0x7fbefc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 --2- 172.21.15.83:0/4056437376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbefc0051d0 0x7fbefc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbf10071b50 con 0x7fbefc004810 2024-09-19T07:27:19.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc00c170 con 0x7fbefc0039c0 2024-09-19T07:27:19.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbefc00c370 con 0x7fbefc0051d0 2024-09-19T07:27:19.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf154a1640 1 --2- 172.21.15.83:0/4056437376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 0x7fbefc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf15ca2640 1 --2- 172.21.15.83:0/4056437376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbefc0039c0 0x7fbefc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf164a3640 1 --2- 172.21.15.83:0/4056437376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbefc0051d0 0x7fbefc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf154a1640 1 -- 172.21.15.83:0/4056437376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbefc0051d0 msgr2=0x7fbefc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:19.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf154a1640 1 --2- 172.21.15.83:0/4056437376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbefc0051d0 0x7fbefc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf154a1640 1 -- 172.21.15.83:0/4056437376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbefc0039c0 msgr2=0x7fbefc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:19.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf154a1640 1 --2- 172.21.15.83:0/4056437376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbefc0039c0 0x7fbefc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf154a1640 1 -- 172.21.15.83:0/4056437376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbefc003530 con 0x7fbefc004810 2024-09-19T07:27:19.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf164a3640 1 --2- 172.21.15.83:0/4056437376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbefc0051d0 0x7fbefc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:27:19.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.359+0000 7fbf164a3640 1 -- 172.21.15.83:0/4056437376 reap_dead start 2024-09-19T07:27:19.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf154a1640 1 --2- 172.21.15.83:0/4056437376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 0x7fbefc004c90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbf00001780 tx=0x7fbf0001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9321130c74514993 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:19.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf000013a0 con 0x7fbefc004810 2024-09-19T07:27:19.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf000052f0 con 0x7fbefc004810 2024-09-19T07:27:19.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf14ca0640 1 -- 172.21.15.83:0/4056437376 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf00003840 con 0x7fbefc004810 2024-09-19T07:27:19.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 -- 172.21.15.83:0/4056437376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 msgr2=0x7fbefc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 --2- 172.21.15.83:0/4056437376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 0x7fbefc004c90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbf00001780 tx=0x7fbf0001c400 comp rx=0 tx=0).stop 2024-09-19T07:27:19.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 -- 172.21.15.83:0/4056437376 shutdown_connections 2024-09-19T07:27:19.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 --2- 172.21.15.83:0/4056437376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 0x7fbefc004c90 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 -- 172.21.15.83:0/4056437376 >> 172.21.15.83:0/4056437376 conn(0x7fbf1006bc80 msgr2=0x7fbf1006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:19.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 -- 172.21.15.83:0/4056437376 shutdown_connections 2024-09-19T07:27:19.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 -- 172.21.15.83:0/4056437376 wait complete. 2024-09-19T07:27:19.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 Processor -- start 2024-09-19T07:27:19.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 -- start start 2024-09-19T07:27:19.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 0x7fbf100c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf100ac1c0 0x7fbf1014cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf101ef3d0 0x7fbf1014c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1006e280 con 0x7fbefc004810 2024-09-19T07:27:19.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbf100716f0 con 0x7fbf101ef3d0 2024-09-19T07:27:19.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf17f06640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbf1006e400 con 0x7fbf100ac1c0 2024-09-19T07:27:19.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf15ca2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 0x7fbf100c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf15ca2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 0x7fbf100c2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39714/0 (socket says 172.21.15.83:39714) 2024-09-19T07:27:19.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.363+0000 7fbf15ca2640 1 -- 172.21.15.83:0/3031414913 learned_addr learned my addr 172.21.15.83:0/3031414913 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:19.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf154a1640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf100ac1c0 0x7fbf1014cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf164a3640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf101ef3d0 0x7fbf1014c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf15ca2640 1 -- 172.21.15.83:0/3031414913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf100ac1c0 msgr2=0x7fbf1014cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf15ca2640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf100ac1c0 0x7fbf1014cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf15ca2640 1 -- 172.21.15.83:0/3031414913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf101ef3d0 msgr2=0x7fbf1014c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf15ca2640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf101ef3d0 0x7fbf1014c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf15ca2640 1 -- 172.21.15.83:0/3031414913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf100c1b20 con 0x7fbefc004810 2024-09-19T07:27:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf154a1640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf100ac1c0 0x7fbf1014cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:27:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf15ca2640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 0x7fbf100c2fc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c011660 tx=0x7fbf0c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf067fc640 1 -- 172.21.15.83:0/3031414913 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c00f960 con 0x7fbefc004810 2024-09-19T07:27:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf067fc640 1 -- 172.21.15.83:0/3031414913 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf0c00fb00 con 0x7fbefc004810 2024-09-19T07:27:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf17f06640 1 -- 172.21.15.83:0/3031414913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf101e09d0 con 0x7fbefc004810 2024-09-19T07:27:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf067fc640 1 -- 172.21.15.83:0/3031414913 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c015460 con 0x7fbefc004810 2024-09-19T07:27:19.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf17f06640 1 -- 172.21.15.83:0/3031414913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf101e0f90 con 0x7fbefc004810 2024-09-19T07:27:19.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf17f06640 1 -- 172.21.15.83:0/3031414913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf1006e790 con 0x7fbefc004810 2024-09-19T07:27:19.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf067fc640 1 -- 172.21.15.83:0/3031414913 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbf0c015600 con 0x7fbefc004810 2024-09-19T07:27:19.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf067fc640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbef40678f0 0x7fbef4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf067fc640 1 -- 172.21.15.83:0/3031414913 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7fbf0c086db0 con 0x7fbefc004810 2024-09-19T07:27:19.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf154a1640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbef40678f0 0x7fbef4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.367+0000 7fbf154a1640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbef40678f0 0x7fbef4069db0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbf00001750 tx=0x7fbf000011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:19.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.371+0000 7fbf067fc640 1 -- 172.21.15.83:0/3031414913 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbf0c019030 con 0x7fbefc004810 2024-09-19T07:27:19.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.571+0000 7fbf17f06640 1 -- 172.21.15.83:0/3031414913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf100c4f30 con 0x7fbefc004810 2024-09-19T07:27:19.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.571+0000 7fbf067fc640 1 -- 172.21.15.83:0/3031414913 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+5703 (secure 0 0 0) 0x7fbf0c057310 con 0x7fbefc004810 2024-09-19T07:27:19.572 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:19.572 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":65,"btime":"2024-09-19T07:27:19:183284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:18.363326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:19.572 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 65 2024-09-19T07:27:19.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.571+0000 7fbf17f06640 1 -- 172.21.15.83:0/3031414913 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbef40678f0 msgr2=0x7fbef4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.571+0000 7fbf17f06640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbef40678f0 0x7fbef4069db0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbf00001750 tx=0x7fbf000011a0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.571+0000 7fbf17f06640 1 -- 172.21.15.83:0/3031414913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 msgr2=0x7fbf100c2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.571+0000 7fbf17f06640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 0x7fbf100c2fc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c011660 tx=0x7fbf0c011b20 comp rx=0 tx=0).stop 2024-09-19T07:27:19.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.571+0000 7fbf17f06640 1 -- 172.21.15.83:0/3031414913 shutdown_connections 2024-09-19T07:27:19.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.571+0000 7fbf17f06640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbef40678f0 0x7fbef4069db0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.571+0000 7fbf17f06640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf101ef3d0 0x7fbf1014c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.571+0000 7fbf17f06640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf100ac1c0 0x7fbf1014cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.571+0000 7fbf17f06640 1 --2- 172.21.15.83:0/3031414913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbefc004810 0x7fbf100c2fc0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.571+0000 7fbf17f06640 1 -- 172.21.15.83:0/3031414913 >> 172.21.15.83:0/3031414913 conn(0x7fbf1006bc80 msgr2=0x7fbf101519d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:19.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.575+0000 7fbf17f06640 1 -- 172.21.15.83:0/3031414913 shutdown_connections 2024-09-19T07:27:19.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.575+0000 7fbf17f06640 1 -- 172.21.15.83:0/3031414913 wait complete. 2024-09-19T07:27:19.591 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-19T07:27:19.592 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:27:19.592 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:27:19.602 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:27:19.602 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:19.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.759+0000 7f83f9c8c640 1 Processor -- start 2024-09-19T07:27:19.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.759+0000 7f83f9c8c640 1 -- start start 2024-09-19T07:27:19.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f9c8c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f83f40c28e0 0x7f83f41545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f9c8c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f83f406dea0 0x7f83f406e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f9c8c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83f40a9e60 con 0x7f83f4071100 2024-09-19T07:27:19.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f9c8c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f83f4071b50 con 0x7f83f40c28e0 2024-09-19T07:27:19.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f9c8c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f83f40c1ff0 con 0x7f83f406dea0 2024-09-19T07:27:19.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f2ffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f83f40c28e0 0x7f83f41545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f3fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f83f406dea0 0x7f83f406e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f2ffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f83f40c28e0 0x7f83f41545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:49536/0 (socket says 172.21.15.83:49536) 2024-09-19T07:27:19.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f2ffd640 1 -- 172.21.15.83:0/3179483188 learned_addr learned my addr 172.21.15.83:0/3179483188 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:19.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f37fe640 1 --1- 172.21.15.83:0/3179483188 >> v1:172.21.15.184:6789/0 conn(0x7f83f4071100 0x7f83f4150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:49330/0 (socket says 172.21.15.83:49330) 2024-09-19T07:27:19.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f2ffd640 1 -- 172.21.15.83:0/3179483188 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f83f406dea0 msgr2=0x7f83f406e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:19.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f2ffd640 1 --2- 172.21.15.83:0/3179483188 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f83f406dea0 0x7f83f406e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f2ffd640 1 -- 172.21.15.83:0/3179483188 >> v1:172.21.15.184:6789/0 conn(0x7f83f4071100 legacy=0x7f83f4150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.763+0000 7f83f2ffd640 1 -- 172.21.15.83:0/3179483188 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f40aa140 con 0x7f83f40c28e0 2024-09-19T07:27:19.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f2ffd640 1 --2- 172.21.15.83:0/3179483188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f83f40c28e0 0x7f83f41545e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009870 tx=0x7f83e801c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=606f74dc690606c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:19.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f27fc640 1 -- 172.21.15.83:0/3179483188 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e80b1070 con 0x7f83f40c28e0 2024-09-19T07:27:19.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f27fc640 1 -- 172.21.15.83:0/3179483188 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e80ac620 con 0x7f83f40c28e0 2024-09-19T07:27:19.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f27fc640 1 -- 172.21.15.83:0/3179483188 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e80ac9f0 con 0x7f83f40c28e0 2024-09-19T07:27:19.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3179483188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f83f40c28e0 msgr2=0x7f83f41545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f9c8c640 1 --2- 172.21.15.83:0/3179483188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f83f40c28e0 0x7f83f41545e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009870 tx=0x7f83e801c4d0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3179483188 shutdown_connections 2024-09-19T07:27:19.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f9c8c640 1 --2- 172.21.15.83:0/3179483188 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f83f406dea0 0x7f83f406e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f9c8c640 1 --2- 172.21.15.83:0/3179483188 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f83f40c28e0 0x7f83f41545e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3179483188 >> 172.21.15.83:0/3179483188 conn(0x7f83f406bc80 msgr2=0x7f83f406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:19.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3179483188 shutdown_connections 2024-09-19T07:27:19.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3179483188 wait complete. 2024-09-19T07:27:19.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f9c8c640 1 Processor -- start 2024-09-19T07:27:19.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.767+0000 7f83f9c8c640 1 -- start start 2024-09-19T07:27:19.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f9c8c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f83f406dea0 0x7f83f414ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f9c8c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4071100 0x7f83f415c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f9c8c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f83f40c28e0 0x7f83f41f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f9c8c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f83f4071660 con 0x7f83f4071100 2024-09-19T07:27:19.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f9c8c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f83f40714e0 con 0x7f83f406dea0 2024-09-19T07:27:19.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f9c8c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f83f4160c60 con 0x7f83f40c28e0 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4071100 0x7f83f415c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4071100 0x7f83f415c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39716/0 (socket says 172.21.15.83:39716) 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f2ffd640 1 -- 172.21.15.83:0/3040011004 learned_addr learned my addr 172.21.15.83:0/3040011004 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f37fe640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f83f406dea0 0x7f83f414ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f3fff640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f83f40c28e0 0x7f83f41f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f2ffd640 1 -- 172.21.15.83:0/3040011004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f83f40c28e0 msgr2=0x7f83f41f2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f2ffd640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f83f40c28e0 0x7f83f41f2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f2ffd640 1 -- 172.21.15.83:0/3040011004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f83f406dea0 msgr2=0x7f83f414ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f2ffd640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f83f406dea0 0x7f83f414ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f2ffd640 1 -- 172.21.15.83:0/3040011004 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f4146210 con 0x7f83f4071100 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f2ffd640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4071100 0x7f83f415c610 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f83e801a100 tx=0x7f83e8019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83dbfff640 1 -- 172.21.15.83:0/3040011004 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e80b1040 con 0x7f83f4071100 2024-09-19T07:27:19.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83dbfff640 1 -- 172.21.15.83:0/3040011004 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e80ba070 con 0x7f83f4071100 2024-09-19T07:27:19.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83dbfff640 1 -- 172.21.15.83:0/3040011004 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e80b5660 con 0x7f83f4071100 2024-09-19T07:27:19.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3040011004 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83f40ab760 con 0x7f83f4071100 2024-09-19T07:27:19.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3040011004 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83f40abd20 con 0x7f83f4071100 2024-09-19T07:27:19.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3040011004 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83f40aa360 con 0x7f83f4071100 2024-09-19T07:27:19.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83dbfff640 1 -- 172.21.15.83:0/3040011004 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f83e80b5800 con 0x7f83f4071100 2024-09-19T07:27:19.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83dbfff640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f83e00678c0 0x7f83e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:19.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83dbfff640 1 -- 172.21.15.83:0/3040011004 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7f83e811ff90 con 0x7f83f4071100 2024-09-19T07:27:19.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.771+0000 7f83f37fe640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f83e00678c0 0x7f83e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:19.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.775+0000 7f83f37fe640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f83e00678c0 0x7f83e0069d80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f83f414d540 tx=0x7f83e4007e90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.779+0000 7f83dbfff640 1 -- 172.21.15.83:0/3040011004 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f83e8016150 con 0x7f83f4071100 2024-09-19T07:27:19.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.979+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3040011004 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83f40c4f30 con 0x7f83f4071100 2024-09-19T07:27:19.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83dbfff640 1 -- 172.21.15.83:0/3040011004 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+5703 (secure 0 0 0) 0x7f83e80efd30 con 0x7f83f4071100 2024-09-19T07:27:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:19.983 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":65,"btime":"2024-09-19T07:27:19:183284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:18.363326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:19.983 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 65 2024-09-19T07:27:19.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3040011004 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f83e00678c0 msgr2=0x7f83e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83f9c8c640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f83e00678c0 0x7f83e0069d80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f83f414d540 tx=0x7f83e4007e90 comp rx=0 tx=0).stop 2024-09-19T07:27:19.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3040011004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4071100 msgr2=0x7f83f415c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:19.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83f9c8c640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4071100 0x7f83f415c610 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f83e801a100 tx=0x7f83e8019270 comp rx=0 tx=0).stop 2024-09-19T07:27:19.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3040011004 shutdown_connections 2024-09-19T07:27:19.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83f9c8c640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f83e00678c0 0x7f83e0069d80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83f9c8c640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f83f40c28e0 0x7f83f41f2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83f9c8c640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4071100 0x7f83f415c610 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83f9c8c640 1 --2- 172.21.15.83:0/3040011004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f83f406dea0 0x7f83f414ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:19.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3040011004 >> 172.21.15.83:0/3040011004 conn(0x7f83f406bc80 msgr2=0x7f83f4150f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:19.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3040011004 shutdown_connections 2024-09-19T07:27:19.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:19.983+0000 7f83f9c8c640 1 -- 172.21.15.83:0/3040011004 wait complete. 2024-09-19T07:27:20.005 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:27:20.722 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.851+0000 7ff54515e640 1 Processor -- start 2024-09-19T07:27:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.851+0000 7ff54515e640 1 -- start start 2024-09-19T07:27:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.851+0000 7ff54515e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff540153eb0 0x7ff540154340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.851+0000 7ff54515e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff540150670 con 0x7ff540150290 2024-09-19T07:27:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.851+0000 7ff54515e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5400c2880 con 0x7ff540152b20 2024-09-19T07:27:20.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.851+0000 7ff54515e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff5400716c0 con 0x7ff540153eb0 2024-09-19T07:27:20.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.851+0000 7ff53f59e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff540153eb0 0x7ff540154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:20.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.851+0000 7ff53ed9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff540150290 0x7ff5400c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33710/0 (socket says 172.21.15.83:33710) 2024-09-19T07:27:20.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.851+0000 7ff53ed9d640 1 -- 172.21.15.83:0/111422205 learned_addr learned my addr 172.21.15.83:0/111422205 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:20.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff53f59e640 1 -- 172.21.15.83:0/111422205 >> v1:172.21.15.184:6789/0 conn(0x7ff540152b20 legacy=0x7ff5400c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:20.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff53f59e640 1 -- 172.21.15.83:0/111422205 >> v1:172.21.15.83:6789/0 conn(0x7ff540150290 legacy=0x7ff5400c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:20.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff53f59e640 1 -- 172.21.15.83:0/111422205 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5400c2b60 con 0x7ff540153eb0 2024-09-19T07:27:20.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff53f59e640 1 --2- 172.21.15.83:0/111422205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff540153eb0 0x7ff540154340 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff5300097c0 tx=0x7ff53001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfe7040aaffcf49 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:20.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff53dd9b640 1 -- 172.21.15.83:0/111422205 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5300b0070 con 0x7ff540153eb0 2024-09-19T07:27:20.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff53dd9b640 1 -- 172.21.15.83:0/111422205 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5300ab600 con 0x7ff540153eb0 2024-09-19T07:27:20.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff53dd9b640 1 -- 172.21.15.83:0/111422205 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5300ab9d0 con 0x7ff540153eb0 2024-09-19T07:27:20.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff54515e640 1 -- 172.21.15.83:0/111422205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff540153eb0 msgr2=0x7ff540154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff54515e640 1 --2- 172.21.15.83:0/111422205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff540153eb0 0x7ff540154340 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff5300097c0 tx=0x7ff53001c460 comp rx=0 tx=0).stop 2024-09-19T07:27:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff54515e640 1 -- 172.21.15.83:0/111422205 shutdown_connections 2024-09-19T07:27:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff54515e640 1 --2- 172.21.15.83:0/111422205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff540153eb0 0x7ff540154340 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff54515e640 1 -- 172.21.15.83:0/111422205 >> 172.21.15.83:0/111422205 conn(0x7ff54006bc80 msgr2=0x7ff54006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff54515e640 1 -- 172.21.15.83:0/111422205 shutdown_connections 2024-09-19T07:27:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.855+0000 7ff54515e640 1 -- 172.21.15.83:0/111422205 wait complete. 2024-09-19T07:27:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff54515e640 1 Processor -- start 2024-09-19T07:27:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff54515e640 1 -- start start 2024-09-19T07:27:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff54515e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff540150290 0x7ff5400bc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff54515e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff540152b20 0x7ff540149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff54515e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff540153eb0 0x7ff5400c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff54515e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff540154a00 con 0x7ff540153eb0 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff54515e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff540154880 con 0x7ff540152b20 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff54515e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff54006e6b0 con 0x7ff540150290 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff53f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff540153eb0 0x7ff5400c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff53f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff540153eb0 0x7ff5400c6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34042/0 (socket says 172.21.15.83:34042) 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff53f59e640 1 -- 172.21.15.83:0/1283111603 learned_addr learned my addr 172.21.15.83:0/1283111603 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff53ed9d640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff540150290 0x7ff5400bc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff53e59c640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff540152b20 0x7ff540149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff53f59e640 1 -- 172.21.15.83:0/1283111603 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff540150290 msgr2=0x7ff5400bc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff53f59e640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff540150290 0x7ff5400bc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff53f59e640 1 -- 172.21.15.83:0/1283111603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff540152b20 msgr2=0x7ff540149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff53f59e640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff540152b20 0x7ff540149800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff53f59e640 1 -- 172.21.15.83:0/1283111603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5400c14d0 con 0x7ff540153eb0 2024-09-19T07:27:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff53f59e640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff540153eb0 0x7ff5400c6570 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff530019070 tx=0x7ff530019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:20.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff5377fe640 1 -- 172.21.15.83:0/1283111603 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5300b0040 con 0x7ff540153eb0 2024-09-19T07:27:20.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff5377fe640 1 -- 172.21.15.83:0/1283111603 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5300b9070 con 0x7ff540153eb0 2024-09-19T07:27:20.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff54515e640 1 -- 172.21.15.83:0/1283111603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff540151d70 con 0x7ff540153eb0 2024-09-19T07:27:20.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff5377fe640 1 -- 172.21.15.83:0/1283111603 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5300b4630 con 0x7ff540153eb0 2024-09-19T07:27:20.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff54515e640 1 -- 172.21.15.83:0/1283111603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff54014b0b0 con 0x7ff540153eb0 2024-09-19T07:27:20.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff54515e640 1 -- 172.21.15.83:0/1283111603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5400c3e90 con 0x7ff540153eb0 2024-09-19T07:27:20.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff5377fe640 1 -- 172.21.15.83:0/1283111603 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff5300b47d0 con 0x7ff540153eb0 2024-09-19T07:27:20.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.859+0000 7ff5377fe640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5240678c0 0x7ff524069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:20.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.863+0000 7ff5377fe640 1 -- 172.21.15.83:0/1283111603 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7ff530120110 con 0x7ff540153eb0 2024-09-19T07:27:20.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.863+0000 7ff53ed9d640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5240678c0 0x7ff524069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:20.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.863+0000 7ff53ed9d640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5240678c0 0x7ff524069d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff54014b510 tx=0x7ff538009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:20.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:20.867+0000 7ff5377fe640 1 -- 172.21.15.83:0/1283111603 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff530016050 con 0x7ff540153eb0 2024-09-19T07:27:21.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.055+0000 7ff54515e640 1 -- 172.21.15.83:0/1283111603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff54006ea10 con 0x7ff540153eb0 2024-09-19T07:27:21.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff5377fe640 1 -- 172.21.15.83:0/1283111603 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5700 (secure 0 0 0) 0x7ff5300f0590 con 0x7ff540153eb0 2024-09-19T07:27:21.058 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:21.059 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"btime":"2024-09-19T07:27:20:192646+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:19.199039+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:21.059 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-09-19T07:27:21.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff54515e640 1 -- 172.21.15.83:0/1283111603 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5240678c0 msgr2=0x7ff524069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:21.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff54515e640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5240678c0 0x7ff524069d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff54014b510 tx=0x7ff538009040 comp rx=0 tx=0).stop 2024-09-19T07:27:21.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff54515e640 1 -- 172.21.15.83:0/1283111603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff540153eb0 msgr2=0x7ff5400c6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:21.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff54515e640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff540153eb0 0x7ff5400c6570 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff530019070 tx=0x7ff530019200 comp rx=0 tx=0).stop 2024-09-19T07:27:21.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff54515e640 1 -- 172.21.15.83:0/1283111603 shutdown_connections 2024-09-19T07:27:21.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff54515e640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5240678c0 0x7ff524069d80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:21.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff54515e640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff540153eb0 0x7ff5400c6570 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:21.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff54515e640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff540152b20 0x7ff540149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:21.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff54515e640 1 --2- 172.21.15.83:0/1283111603 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff540150290 0x7ff5400bc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:21.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff54515e640 1 -- 172.21.15.83:0/1283111603 >> 172.21.15.83:0/1283111603 conn(0x7ff54006bc80 msgr2=0x7ff540155e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:21.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff54515e640 1 -- 172.21.15.83:0/1283111603 shutdown_connections 2024-09-19T07:27:21.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:21.059+0000 7ff54515e640 1 -- 172.21.15.83:0/1283111603 wait complete. 2024-09-19T07:27:21.077 INFO:tasks.mds_thrash.fs.[a]:no change 2024-09-19T07:27:22.006 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:22.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7c0a4640 1 Processor -- start 2024-09-19T07:27:22.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7c0a4640 1 -- start start 2024-09-19T07:27:22.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7c0a4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab740c6360 0x7fab74153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7c0a4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab740bee30 0x7fab741570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7c0a4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab740c5b90 con 0x7fab740c9680 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7c0a4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fab7415a050 con 0x7fab740bee30 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7c0a4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fab7406c950 con 0x7fab740c6360 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7963f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab740bee30 0x7fab741570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7963f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab740bee30 0x7fab741570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34052/0 (socket says 172.21.15.83:34052) 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7963f640 1 -- 172.21.15.83:0/2323244028 learned_addr learned my addr 172.21.15.83:0/2323244028 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab79e40640 1 --2- 172.21.15.83:0/2323244028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab740c6360 0x7fab74153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7963f640 1 -- 172.21.15.83:0/2323244028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab740c6360 msgr2=0x7fab74153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7963f640 1 --2- 172.21.15.83:0/2323244028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab740c6360 0x7fab74153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7963f640 1 -- 172.21.15.83:0/2323244028 >> v1:172.21.15.184:6789/0 conn(0x7fab740c9680 legacy=0x7fab74157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab7963f640 1 -- 172.21.15.83:0/2323244028 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7406c670 con 0x7fab740bee30 2024-09-19T07:27:22.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.139+0000 7fab79e40640 1 --2- 172.21.15.83:0/2323244028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab740c6360 0x7fab74153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:22.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7963f640 1 --2- 172.21.15.83:0/2323244028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab740bee30 0x7fab741570f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fab64009870 tx=0x7fab6401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ba6a79ac62a0fd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:22.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab78e3e640 1 -- 172.21.15.83:0/2323244028 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab640b0070 con 0x7fab740bee30 2024-09-19T07:27:22.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab78e3e640 1 -- 172.21.15.83:0/2323244028 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab640ab600 con 0x7fab740bee30 2024-09-19T07:27:22.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab78e3e640 1 -- 172.21.15.83:0/2323244028 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab640ab9d0 con 0x7fab740bee30 2024-09-19T07:27:22.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 -- 172.21.15.83:0/2323244028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab740bee30 msgr2=0x7fab741570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:22.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 --2- 172.21.15.83:0/2323244028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab740bee30 0x7fab741570f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fab64009870 tx=0x7fab6401c490 comp rx=0 tx=0).stop 2024-09-19T07:27:22.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 -- 172.21.15.83:0/2323244028 shutdown_connections 2024-09-19T07:27:22.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 --2- 172.21.15.83:0/2323244028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab740bee30 0x7fab741570f0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:22.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 --2- 172.21.15.83:0/2323244028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab740c6360 0x7fab74153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:22.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 -- 172.21.15.83:0/2323244028 >> 172.21.15.83:0/2323244028 conn(0x7fab740722a0 msgr2=0x7fab740726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:22.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 -- 172.21.15.83:0/2323244028 shutdown_connections 2024-09-19T07:27:22.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 -- 172.21.15.83:0/2323244028 wait complete. 2024-09-19T07:27:22.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 Processor -- start 2024-09-19T07:27:22.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 -- start start 2024-09-19T07:27:22.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab740bee30 0x7fab74151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:22.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab740c6360 0x7fab7414c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:22.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab740c9680 0x7fab7414ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:22.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fab740c68c0 con 0x7fab740c6360 2024-09-19T07:27:22.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab740c6740 con 0x7fab740bee30 2024-09-19T07:27:22.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7c0a4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fab74162da0 con 0x7fab740c9680 2024-09-19T07:27:22.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.143+0000 7fab7963f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab740c6360 0x7fab7414c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:22.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab7963f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab740c6360 0x7fab7414c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34066/0 (socket says 172.21.15.83:34066) 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab7963f640 1 -- 172.21.15.83:0/4253192231 learned_addr learned my addr 172.21.15.83:0/4253192231 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab79e40640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab740bee30 0x7fab74151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab7a641640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab740c9680 0x7fab7414ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab7963f640 1 -- 172.21.15.83:0/4253192231 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab740c9680 msgr2=0x7fab7414ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab7963f640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab740c9680 0x7fab7414ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab7963f640 1 -- 172.21.15.83:0/4253192231 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab740bee30 msgr2=0x7fab74151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab7963f640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab740bee30 0x7fab74151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab7963f640 1 -- 172.21.15.83:0/4253192231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab741ee5f0 con 0x7fab740c6360 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab79e40640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab740bee30 0x7fab74151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab7963f640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab740c6360 0x7fab7414c930 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fab6401ab40 tx=0x7fab64019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab6a7fc640 1 -- 172.21.15.83:0/4253192231 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab640b9070 con 0x7fab740c6360 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab6a7fc640 1 -- 172.21.15.83:0/4253192231 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab640b4620 con 0x7fab740c6360 2024-09-19T07:27:22.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab7c0a4640 1 -- 172.21.15.83:0/4253192231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab741f1ba0 con 0x7fab740c6360 2024-09-19T07:27:22.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab6a7fc640 1 -- 172.21.15.83:0/4253192231 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab640b0040 con 0x7fab740c6360 2024-09-19T07:27:22.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab7c0a4640 1 -- 172.21.15.83:0/4253192231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab741f2160 con 0x7fab740c6360 2024-09-19T07:27:22.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab7c0a4640 1 -- 172.21.15.83:0/4253192231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab74153050 con 0x7fab740c6360 2024-09-19T07:27:22.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab6a7fc640 1 -- 172.21.15.83:0/4253192231 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fab640bc070 con 0x7fab740c6360 2024-09-19T07:27:22.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab6a7fc640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab580678c0 0x7fab58069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:22.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab6a7fc640 1 -- 172.21.15.83:0/4253192231 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7fab6411f660 con 0x7fab740c6360 2024-09-19T07:27:22.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab79e40640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab580678c0 0x7fab58069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:22.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.147+0000 7fab79e40640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab580678c0 0x7fab58069d80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fab74073410 tx=0x7fab70008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:22.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.151+0000 7fab6a7fc640 1 -- 172.21.15.83:0/4253192231 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fab64016110 con 0x7fab740c6360 2024-09-19T07:27:22.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.391+0000 7fab7c0a4640 1 -- 172.21.15.83:0/4253192231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab740cbc70 con 0x7fab740c6360 2024-09-19T07:27:22.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab6a7fc640 1 -- 172.21.15.83:0/4253192231 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5707 (secure 0 0 0) 0x7fab640efae0 con 0x7fab740c6360 2024-09-19T07:27:22.395 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:22.395 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":67,"btime":"2024-09-19T07:27:21:202637+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:22.395 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 67 2024-09-19T07:27:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab7c0a4640 1 -- 172.21.15.83:0/4253192231 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab580678c0 msgr2=0x7fab58069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab7c0a4640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab580678c0 0x7fab58069d80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fab74073410 tx=0x7fab70008040 comp rx=0 tx=0).stop 2024-09-19T07:27:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab7c0a4640 1 -- 172.21.15.83:0/4253192231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab740c6360 msgr2=0x7fab7414c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab7c0a4640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab740c6360 0x7fab7414c930 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fab6401ab40 tx=0x7fab64019230 comp rx=0 tx=0).stop 2024-09-19T07:27:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab7c0a4640 1 -- 172.21.15.83:0/4253192231 shutdown_connections 2024-09-19T07:27:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab7c0a4640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab580678c0 0x7fab58069d80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab7c0a4640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab740c9680 0x7fab7414ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab7c0a4640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab740c6360 0x7fab7414c930 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab7c0a4640 1 --2- 172.21.15.83:0/4253192231 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab740bee30 0x7fab74151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:22.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab7c0a4640 1 -- 172.21.15.83:0/4253192231 >> 172.21.15.83:0/4253192231 conn(0x7fab740722a0 msgr2=0x7fab7415fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:22.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab7c0a4640 1 -- 172.21.15.83:0/4253192231 shutdown_connections 2024-09-19T07:27:22.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:22.395+0000 7fab7c0a4640 1 -- 172.21.15.83:0/4253192231 wait complete. 2024-09-19T07:27:22.416 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:27:23.078 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:23.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.203+0000 7f207ce3c640 1 Processor -- start 2024-09-19T07:27:23.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.203+0000 7f207ce3c640 1 -- start start 2024-09-19T07:27:23.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.203+0000 7f207ce3c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f20781535e0 0x7f20780af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:23.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.203+0000 7f207ce3c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f20780c8690 0x7f20780c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:23.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.203+0000 7f207ce3c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20780c4e50 con 0x7f20780aee90 2024-09-19T07:27:23.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.203+0000 7f207ce3c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f20780c4920 con 0x7f20780c8690 2024-09-19T07:27:23.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.203+0000 7f207ce3c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f20780c7510 con 0x7f20781535e0 2024-09-19T07:27:23.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.203+0000 7f2076d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f20780c8690 0x7f20780c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:23.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.203+0000 7f2076d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f20780c8690 0x7f20780c8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34072/0 (socket says 172.21.15.83:34072) 2024-09-19T07:27:23.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.203+0000 7f2076d9d640 1 -- 172.21.15.83:0/2688694056 learned_addr learned my addr 172.21.15.83:0/2688694056 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:23.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.207+0000 7f207659c640 1 --2- 172.21.15.83:0/2688694056 >> v2:172.21.15.184:3300/0 conn(0x7f20781535e0 0x7f20780af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:23.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.207+0000 7f2076d9d640 1 -- 172.21.15.83:0/2688694056 >> v2:172.21.15.184:3300/0 conn(0x7f20781535e0 msgr2=0x7f20780af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:23.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.207+0000 7f2076d9d640 1 --2- 172.21.15.83:0/2688694056 >> v2:172.21.15.184:3300/0 conn(0x7f20781535e0 0x7f20780af3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:23.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.207+0000 7f2076d9d640 1 -- 172.21.15.83:0/2688694056 >> v1:172.21.15.184:6789/0 conn(0x7f20780aee90 legacy=0x7f20780c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:23.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.207+0000 7f2076d9d640 1 -- 172.21.15.83:0/2688694056 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20780c77f0 con 0x7f20780c8690 2024-09-19T07:27:23.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.207+0000 7f2076d9d640 1 --2- 172.21.15.83:0/2688694056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20780c8690 0x7f20780c8ae0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f20780744a0 tx=0x7f207001c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532f48423da646a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:23.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207559a640 1 -- 172.21.15.83:0/2688694056 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20700a5670 con 0x7f20780c8690 2024-09-19T07:27:23.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207559a640 1 -- 172.21.15.83:0/2688694056 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20700a5810 con 0x7f20780c8690 2024-09-19T07:27:23.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207559a640 1 -- 172.21.15.83:0/2688694056 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20700a5be0 con 0x7f20780c8690 2024-09-19T07:27:23.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 -- 172.21.15.83:0/2688694056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20780c8690 msgr2=0x7f20780c8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:23.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 --2- 172.21.15.83:0/2688694056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20780c8690 0x7f20780c8ae0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f20780744a0 tx=0x7f207001c9b0 comp rx=0 tx=0).stop 2024-09-19T07:27:23.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 -- 172.21.15.83:0/2688694056 shutdown_connections 2024-09-19T07:27:23.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 --2- 172.21.15.83:0/2688694056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20780c8690 0x7f20780c8ae0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:23.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 --2- 172.21.15.83:0/2688694056 >> v2:172.21.15.184:3300/0 conn(0x7f20781535e0 0x7f20780af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:23.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 -- 172.21.15.83:0/2688694056 >> 172.21.15.83:0/2688694056 conn(0x7f20780c9610 msgr2=0x7f20780c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:23.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 -- 172.21.15.83:0/2688694056 shutdown_connections 2024-09-19T07:27:23.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 -- 172.21.15.83:0/2688694056 wait complete. 2024-09-19T07:27:23.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 Processor -- start 2024-09-19T07:27:23.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 -- start start 2024-09-19T07:27:23.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20780aee90 0x7f20781ef080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:23.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20780c8690 0x7f207814c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:23.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20781535e0 0x7f20780ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:23.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2078163050 con 0x7f20780c8690 2024-09-19T07:27:23.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2078162ed0 con 0x7f20781535e0 2024-09-19T07:27:23.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207ce3c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f20781631d0 con 0x7f20780aee90 2024-09-19T07:27:23.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f2075d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20780c8690 0x7f207814c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:23.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f2075d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20780c8690 0x7f207814c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34086/0 (socket says 172.21.15.83:34086) 2024-09-19T07:27:23.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f2075d9b640 1 -- 172.21.15.83:0/3103821302 learned_addr learned my addr 172.21.15.83:0/3103821302 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:23.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f2076d9d640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20781535e0 0x7f20780ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:23.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.211+0000 7f207659c640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20780aee90 0x7f20781ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:23.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f2075d9b640 1 -- 172.21.15.83:0/3103821302 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20780aee90 msgr2=0x7f20781ef080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:23.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f2075d9b640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20780aee90 0x7f20781ef080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:23.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f2075d9b640 1 -- 172.21.15.83:0/3103821302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20781535e0 msgr2=0x7f20780ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:23.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f2075d9b640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20781535e0 0x7f20780ad1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:23.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f2075d9b640 1 -- 172.21.15.83:0/3103821302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20780adef0 con 0x7f20780c8690 2024-09-19T07:27:23.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f2075d9b640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20780c8690 0x7f207814c3e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f206c00c450 tx=0x7f206c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:23.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f2066ffd640 1 -- 172.21.15.83:0/3103821302 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c012830 con 0x7f20780c8690 2024-09-19T07:27:23.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f2066ffd640 1 -- 172.21.15.83:0/3103821302 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f206c017070 con 0x7f20780c8690 2024-09-19T07:27:23.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f207ce3c640 1 -- 172.21.15.83:0/3103821302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20780ae1d0 con 0x7f20780c8690 2024-09-19T07:27:23.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f2066ffd640 1 -- 172.21.15.83:0/3103821302 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c012c30 con 0x7f20780c8690 2024-09-19T07:27:23.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f207ce3c640 1 -- 172.21.15.83:0/3103821302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20780ae790 con 0x7f20780c8690 2024-09-19T07:27:23.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f207ce3c640 1 -- 172.21.15.83:0/3103821302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f205c0052a0 con 0x7f20780c8690 2024-09-19T07:27:23.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f2066ffd640 1 -- 172.21.15.83:0/3103821302 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f206c01c460 con 0x7f20780c8690 2024-09-19T07:27:23.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f2066ffd640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f20580678c0 0x7f2058069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:23.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f2066ffd640 1 -- 172.21.15.83:0/3103821302 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7f206c086040 con 0x7f20780c8690 2024-09-19T07:27:23.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f207659c640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f20580678c0 0x7f2058069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:23.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.215+0000 7f207659c640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f20580678c0 0x7f2058069d80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f206800b930 tx=0x7f2068008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:23.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.219+0000 7f2066ffd640 1 -- 172.21.15.83:0/3103821302 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f206c0564c0 con 0x7f20780c8690 2024-09-19T07:27:23.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.451+0000 7f207ce3c640 1 -- 172.21.15.83:0/3103821302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f205c005860 con 0x7f20780c8690 2024-09-19T07:27:23.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.451+0000 7f2066ffd640 1 -- 172.21.15.83:0/3103821302 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5707 (secure 0 0 0) 0x7f206c055dc0 con 0x7f20780c8690 2024-09-19T07:27:23.451 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:23.451 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":67,"btime":"2024-09-19T07:27:21:202637+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:23.451 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 67 2024-09-19T07:27:23.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.451+0000 7f207ce3c640 1 -- 172.21.15.83:0/3103821302 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f20580678c0 msgr2=0x7f2058069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:23.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.451+0000 7f207ce3c640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f20580678c0 0x7f2058069d80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f206800b930 tx=0x7f2068008040 comp rx=0 tx=0).stop 2024-09-19T07:27:23.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.451+0000 7f207ce3c640 1 -- 172.21.15.83:0/3103821302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20780c8690 msgr2=0x7f207814c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:23.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.451+0000 7f207ce3c640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20780c8690 0x7f207814c3e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f206c00c450 tx=0x7f206c00c910 comp rx=0 tx=0).stop 2024-09-19T07:27:23.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.451+0000 7f207ce3c640 1 -- 172.21.15.83:0/3103821302 shutdown_connections 2024-09-19T07:27:23.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.451+0000 7f207ce3c640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f20580678c0 0x7f2058069d80 secure :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f206800b930 tx=0x7f2068008040 comp rx=0 tx=0).stop 2024-09-19T07:27:23.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.451+0000 7f207ce3c640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20781535e0 0x7f20780ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:23.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.451+0000 7f207ce3c640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20780c8690 0x7f207814c3e0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:23.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.451+0000 7f207ce3c640 1 --2- 172.21.15.83:0/3103821302 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20780aee90 0x7f20781ef080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:23.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.451+0000 7f207ce3c640 1 -- 172.21.15.83:0/3103821302 >> 172.21.15.83:0/3103821302 conn(0x7f20780c9610 msgr2=0x7f207814fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:23.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.455+0000 7f207ce3c640 1 -- 172.21.15.83:0/3103821302 shutdown_connections 2024-09-19T07:27:23.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:23.455+0000 7f207ce3c640 1 -- 172.21.15.83:0/3103821302 wait complete. 2024-09-19T07:27:23.473 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=5607 2024-09-19T07:27:23.473 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before reviving mds.a 2024-09-19T07:27:24.270 INFO:tasks.ceph.mds.b.smithi184.stderr:2024-09-19T07:27:24.271+0000 7f5829f47640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:27:24.416 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:24.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff28b7b5640 1 Processor -- start 2024-09-19T07:27:24.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff28b7b5640 1 -- start start 2024-09-19T07:27:24.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff28b7b5640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff2840c3a20 0x7ff2841594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:24.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff28b7b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2840c4450 0x7ff2841566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:24.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff28b7b5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff284155ee0 con 0x7ff284156c00 2024-09-19T07:27:24.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff28b7b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff284068290 con 0x7ff2840c4450 2024-09-19T07:27:24.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff28b7b5640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff2840684e0 con 0x7ff2840c3a20 2024-09-19T07:27:24.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff288d50640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2840c4450 0x7ff2841566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:24.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff288d50640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2840c4450 0x7ff2841566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34092/0 (socket says 172.21.15.83:34092) 2024-09-19T07:27:24.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff288d50640 1 -- 172.21.15.83:0/14075360 learned_addr learned my addr 172.21.15.83:0/14075360 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:24.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff288d50640 1 -- 172.21.15.83:0/14075360 >> v2:172.21.15.184:3300/0 conn(0x7ff2840c3a20 msgr2=0x7ff2841594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:24.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff289551640 1 --2- 172.21.15.83:0/14075360 >> v2:172.21.15.184:3300/0 conn(0x7ff2840c3a20 0x7ff2841594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:24.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff288d50640 1 --2- 172.21.15.83:0/14075360 >> v2:172.21.15.184:3300/0 conn(0x7ff2840c3a20 0x7ff2841594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:24.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff288d50640 1 -- 172.21.15.83:0/14075360 >> v1:172.21.15.184:6789/0 conn(0x7ff284156c00 legacy=0x7ff28414c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:24.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.563+0000 7ff288d50640 1 -- 172.21.15.83:0/14075360 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2840687c0 con 0x7ff2840c4450 2024-09-19T07:27:24.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff288d50640 1 --2- 172.21.15.83:0/14075360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2840c4450 0x7ff2841566c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ff284071270 tx=0x7ff27401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e3c9ab120b7177 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:24.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff27bfff640 1 -- 172.21.15.83:0/14075360 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28406fcd0 con 0x7ff2840c4450 2024-09-19T07:27:24.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff27bfff640 1 -- 172.21.15.83:0/14075360 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2740b0070 con 0x7ff2840c4450 2024-09-19T07:27:24.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff27bfff640 1 -- 172.21.15.83:0/14075360 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740ab860 con 0x7ff2840c4450 2024-09-19T07:27:24.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff28b7b5640 1 -- 172.21.15.83:0/14075360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2840c4450 msgr2=0x7ff2841566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:24.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff28b7b5640 1 --2- 172.21.15.83:0/14075360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2840c4450 0x7ff2841566c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ff284071270 tx=0x7ff27401c420 comp rx=0 tx=0).stop 2024-09-19T07:27:24.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff28b7b5640 1 -- 172.21.15.83:0/14075360 shutdown_connections 2024-09-19T07:27:24.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff28b7b5640 1 --2- 172.21.15.83:0/14075360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2840c4450 0x7ff2841566c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:24.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff28b7b5640 1 --2- 172.21.15.83:0/14075360 >> v2:172.21.15.184:3300/0 conn(0x7ff2840c3a20 0x7ff2841594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:24.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff28b7b5640 1 -- 172.21.15.83:0/14075360 >> 172.21.15.83:0/14075360 conn(0x7ff2840c48b0 msgr2=0x7ff2840c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:24.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff28b7b5640 1 -- 172.21.15.83:0/14075360 shutdown_connections 2024-09-19T07:27:24.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff28b7b5640 1 -- 172.21.15.83:0/14075360 wait complete. 2024-09-19T07:27:24.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.567+0000 7ff28b7b5640 1 Processor -- start 2024-09-19T07:27:24.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff28b7b5640 1 -- start start 2024-09-19T07:27:24.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff28b7b5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2840c3a20 0x7ff2841f1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:24.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff28b7b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2840c4450 0x7ff28414eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:24.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff28b7b5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff284156c00 0x7ff28414f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:24.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff28b7b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff284163080 con 0x7ff2840c4450 2024-09-19T07:27:24.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff28b7b5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff2840c4010 con 0x7ff284156c00 2024-09-19T07:27:24.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff28b7b5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff284163200 con 0x7ff2840c3a20 2024-09-19T07:27:24.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff288d50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2840c4450 0x7ff28414eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:24.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff288d50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2840c4450 0x7ff28414eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34096/0 (socket says 172.21.15.83:34096) 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff288d50640 1 -- 172.21.15.83:0/2993976939 learned_addr learned my addr 172.21.15.83:0/2993976939 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff289551640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2840c3a20 0x7ff2841f1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff289d52640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff284156c00 0x7ff28414f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff288d50640 1 -- 172.21.15.83:0/2993976939 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2840c3a20 msgr2=0x7ff2841f1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff288d50640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2840c3a20 0x7ff2841f1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff288d50640 1 -- 172.21.15.83:0/2993976939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff284156c00 msgr2=0x7ff28414f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff288d50640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff284156c00 0x7ff28414f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff288d50640 1 -- 172.21.15.83:0/2993976939 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2840aa860 con 0x7ff2840c4450 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff288d50640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2840c4450 0x7ff28414eef0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff2740ad040 tx=0x7ff27401ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff279ffb640 1 -- 172.21.15.83:0/2993976939 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740be5f0 con 0x7ff2840c4450 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff279ffb640 1 -- 172.21.15.83:0/2993976939 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2740be7b0 con 0x7ff2840c4450 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff28b7b5640 1 -- 172.21.15.83:0/2993976939 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2841f2880 con 0x7ff2840c4450 2024-09-19T07:27:24.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff279ffb640 1 -- 172.21.15.83:0/2993976939 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740beba0 con 0x7ff2840c4450 2024-09-19T07:27:24.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff28b7b5640 1 -- 172.21.15.83:0/2993976939 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff284150970 con 0x7ff2840c4450 2024-09-19T07:27:24.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff28b7b5640 1 -- 172.21.15.83:0/2993976939 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2841594d0 con 0x7ff2840c4450 2024-09-19T07:27:24.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.571+0000 7ff279ffb640 1 -- 172.21.15.83:0/2993976939 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff2740b43f0 con 0x7ff2840c4450 2024-09-19T07:27:24.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.575+0000 7ff279ffb640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2680678f0 0x7ff268069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:24.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.575+0000 7ff279ffb640 1 -- 172.21.15.83:0/2993976939 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7ff274120e90 con 0x7ff2840c4450 2024-09-19T07:27:24.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.575+0000 7ff289551640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2680678f0 0x7ff268069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:24.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.575+0000 7ff289551640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2680678f0 0x7ff268069db0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff28000a970 tx=0x7ff280008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:24.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.579+0000 7ff279ffb640 1 -- 172.21.15.83:0/2993976939 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff2740160d0 con 0x7ff2840c4450 2024-09-19T07:27:24.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.787+0000 7ff28b7b5640 1 -- 172.21.15.83:0/2993976939 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff28414c690 con 0x7ff2840c4450 2024-09-19T07:27:24.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.787+0000 7ff279ffb640 1 -- 172.21.15.83:0/2993976939 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5707 (secure 0 0 0) 0x7ff2740f1310 con 0x7ff2840c4450 2024-09-19T07:27:24.790 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:24.790 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":67,"btime":"2024-09-19T07:27:21:202637+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:24.790 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 67 2024-09-19T07:27:24.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.791+0000 7ff28b7b5640 1 -- 172.21.15.83:0/2993976939 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2680678f0 msgr2=0x7ff268069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:24.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.791+0000 7ff28b7b5640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2680678f0 0x7ff268069db0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff28000a970 tx=0x7ff280008040 comp rx=0 tx=0).stop 2024-09-19T07:27:24.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.791+0000 7ff28b7b5640 1 -- 172.21.15.83:0/2993976939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2840c4450 msgr2=0x7ff28414eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:24.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.791+0000 7ff28b7b5640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2840c4450 0x7ff28414eef0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff2740ad040 tx=0x7ff27401ac60 comp rx=0 tx=0).stop 2024-09-19T07:27:24.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.791+0000 7ff28b7b5640 1 -- 172.21.15.83:0/2993976939 shutdown_connections 2024-09-19T07:27:24.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.791+0000 7ff28b7b5640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2680678f0 0x7ff268069db0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:24.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.791+0000 7ff28b7b5640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff284156c00 0x7ff28414f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:24.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.791+0000 7ff28b7b5640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2840c4450 0x7ff28414eef0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:24.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.791+0000 7ff28b7b5640 1 --2- 172.21.15.83:0/2993976939 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2840c3a20 0x7ff2841f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:24.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.791+0000 7ff28b7b5640 1 -- 172.21.15.83:0/2993976939 >> 172.21.15.83:0/2993976939 conn(0x7ff2840c48b0 msgr2=0x7ff2840c1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:24.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.791+0000 7ff28b7b5640 1 -- 172.21.15.83:0/2993976939 shutdown_connections 2024-09-19T07:27:24.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:24.791+0000 7ff28b7b5640 1 -- 172.21.15.83:0/2993976939 wait complete. 2024-09-19T07:27:24.814 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:27:26.816 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:26.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.939+0000 7fbf656c7640 1 Processor -- start 2024-09-19T07:27:26.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.939+0000 7fbf656c7640 1 -- start start 2024-09-19T07:27:26.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.939+0000 7fbf656c7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbf6014a500 0x7fbf600c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:26.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.939+0000 7fbf656c7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf600c4550 0x7fbf60157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:26.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.939+0000 7fbf656c7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf60151750 con 0x7fbf60154540 2024-09-19T07:27:26.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.939+0000 7fbf656c7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fbf600c7e90 con 0x7fbf600c4550 2024-09-19T07:27:26.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.939+0000 7fbf656c7640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbf60073110 con 0x7fbf6014a500 2024-09-19T07:27:26.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.939+0000 7fbf5f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf600c4550 0x7fbf60157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:26.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.939+0000 7fbf5f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf600c4550 0x7fbf60157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34112/0 (socket says 172.21.15.83:34112) 2024-09-19T07:27:26.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.939+0000 7fbf5f7fe640 1 -- 172.21.15.83:0/4145750113 learned_addr learned my addr 172.21.15.83:0/4145750113 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:26.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf5effd640 1 --2- 172.21.15.83:0/4145750113 >> v2:172.21.15.184:3300/0 conn(0x7fbf6014a500 0x7fbf600c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:26.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf5f7fe640 1 -- 172.21.15.83:0/4145750113 >> v2:172.21.15.184:3300/0 conn(0x7fbf6014a500 msgr2=0x7fbf600c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:26.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf5f7fe640 1 --2- 172.21.15.83:0/4145750113 >> v2:172.21.15.184:3300/0 conn(0x7fbf6014a500 0x7fbf600c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:26.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf5f7fe640 1 -- 172.21.15.83:0/4145750113 >> v1:172.21.15.184:6789/0 conn(0x7fbf60154540 legacy=0x7fbf60157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:26.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf5f7fe640 1 -- 172.21.15.83:0/4145750113 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf600733f0 con 0x7fbf600c4550 2024-09-19T07:27:26.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf5f7fe640 1 --2- 172.21.15.83:0/4145750113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf600c4550 0x7fbf60157ba0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fbf60072c20 tx=0x7fbf5801c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77cb6c02b91484fe server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:26.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf5dffb640 1 -- 172.21.15.83:0/4145750113 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf580a5620 con 0x7fbf600c4550 2024-09-19T07:27:26.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf5dffb640 1 -- 172.21.15.83:0/4145750113 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf580a57c0 con 0x7fbf600c4550 2024-09-19T07:27:26.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf5dffb640 1 -- 172.21.15.83:0/4145750113 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf580a5b90 con 0x7fbf600c4550 2024-09-19T07:27:26.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf656c7640 1 -- 172.21.15.83:0/4145750113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf600c4550 msgr2=0x7fbf60157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:26.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf656c7640 1 --2- 172.21.15.83:0/4145750113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf600c4550 0x7fbf60157ba0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fbf60072c20 tx=0x7fbf5801c7a0 comp rx=0 tx=0).stop 2024-09-19T07:27:26.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf656c7640 1 -- 172.21.15.83:0/4145750113 shutdown_connections 2024-09-19T07:27:26.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf656c7640 1 --2- 172.21.15.83:0/4145750113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf600c4550 0x7fbf60157ba0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:26.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf656c7640 1 --2- 172.21.15.83:0/4145750113 >> v2:172.21.15.184:3300/0 conn(0x7fbf6014a500 0x7fbf600c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:26.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf656c7640 1 -- 172.21.15.83:0/4145750113 >> 172.21.15.83:0/4145750113 conn(0x7fbf600c9610 msgr2=0x7fbf600c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:26.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf656c7640 1 -- 172.21.15.83:0/4145750113 shutdown_connections 2024-09-19T07:27:26.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.943+0000 7fbf656c7640 1 -- 172.21.15.83:0/4145750113 wait complete. 2024-09-19T07:27:26.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf656c7640 1 Processor -- start 2024-09-19T07:27:26.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf656c7640 1 -- start start 2024-09-19T07:27:26.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf656c7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf600c4550 0x7fbf60151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf656c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf6014a500 0x7fbf600c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf656c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf60154540 0x7fbf6014eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf656c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbf600c7850 con 0x7fbf60154540 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf656c7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbf600c76d0 con 0x7fbf6014a500 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf656c7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbf60160c60 con 0x7fbf600c4550 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf5f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf60154540 0x7fbf6014eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf5f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf60154540 0x7fbf6014eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34116/0 (socket says 172.21.15.83:34116) 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf5f7fe640 1 -- 172.21.15.83:0/3096801298 learned_addr learned my addr 172.21.15.83:0/3096801298 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf5effd640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf600c4550 0x7fbf60151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf5f7fe640 1 -- 172.21.15.83:0/3096801298 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf600c4550 msgr2=0x7fbf60151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf5f7fe640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf600c4550 0x7fbf60151110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf5f7fe640 1 -- 172.21.15.83:0/3096801298 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf6014a500 msgr2=0x7fbf600c2ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:27:26.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf5f7fe640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf6014a500 0x7fbf600c2ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:26.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf5f7fe640 1 -- 172.21.15.83:0/3096801298 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf600c8c90 con 0x7fbf60154540 2024-09-19T07:27:26.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf5f7fe640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf60154540 0x7fbf6014eb20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fbf5801ae50 tx=0x7fbf58019540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:26.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf477fe640 1 -- 172.21.15.83:0/3096801298 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf580bc020 con 0x7fbf60154540 2024-09-19T07:27:26.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf477fe640 1 -- 172.21.15.83:0/3096801298 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf580b8670 con 0x7fbf60154540 2024-09-19T07:27:26.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf656c7640 1 -- 172.21.15.83:0/3096801298 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf6014e770 con 0x7fbf60154540 2024-09-19T07:27:26.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf477fe640 1 -- 172.21.15.83:0/3096801298 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf58009510 con 0x7fbf60154540 2024-09-19T07:27:26.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.947+0000 7fbf656c7640 1 -- 172.21.15.83:0/3096801298 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf60150bf0 con 0x7fbf60154540 2024-09-19T07:27:26.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.951+0000 7fbf656c7640 1 -- 172.21.15.83:0/3096801298 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf600c4010 con 0x7fbf60154540 2024-09-19T07:27:26.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.951+0000 7fbf477fe640 1 -- 172.21.15.83:0/3096801298 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbf580bf020 con 0x7fbf60154540 2024-09-19T07:27:26.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.951+0000 7fbf477fe640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf4c0678c0 0x7fbf4c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:26.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.951+0000 7fbf477fe640 1 -- 172.21.15.83:0/3096801298 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7fbf58121180 con 0x7fbf60154540 2024-09-19T07:27:26.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.951+0000 7fbf5effd640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf4c0678c0 0x7fbf4c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:26.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.951+0000 7fbf5effd640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf4c0678c0 0x7fbf4c069d80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbf500079c0 tx=0x7fbf50008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:26.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:26.955+0000 7fbf477fe640 1 -- 172.21.15.83:0/3096801298 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbf580c6030 con 0x7fbf60154540 2024-09-19T07:27:27.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.187+0000 7fbf656c7640 1 -- 172.21.15.83:0/3096801298 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf600af9a0 con 0x7fbf60154540 2024-09-19T07:27:27.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.187+0000 7fbf477fe640 1 -- 172.21.15.83:0/3096801298 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5707 (secure 0 0 0) 0x7fbf60151750 con 0x7fbf60154540 2024-09-19T07:27:27.189 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:27.189 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":67,"btime":"2024-09-19T07:27:21:202637+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:27.189 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 67 2024-09-19T07:27:27.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.191+0000 7fbf656c7640 1 -- 172.21.15.83:0/3096801298 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf4c0678c0 msgr2=0x7fbf4c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:27.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.191+0000 7fbf656c7640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf4c0678c0 0x7fbf4c069d80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbf500079c0 tx=0x7fbf50008040 comp rx=0 tx=0).stop 2024-09-19T07:27:27.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.191+0000 7fbf656c7640 1 -- 172.21.15.83:0/3096801298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf60154540 msgr2=0x7fbf6014eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:27.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.191+0000 7fbf656c7640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf60154540 0x7fbf6014eb20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fbf5801ae50 tx=0x7fbf58019540 comp rx=0 tx=0).stop 2024-09-19T07:27:27.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.191+0000 7fbf656c7640 1 -- 172.21.15.83:0/3096801298 shutdown_connections 2024-09-19T07:27:27.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.191+0000 7fbf656c7640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf4c0678c0 0x7fbf4c069d80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:27.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.191+0000 7fbf656c7640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf60154540 0x7fbf6014eb20 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:27.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.191+0000 7fbf656c7640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf6014a500 0x7fbf600c2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:27.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.191+0000 7fbf656c7640 1 --2- 172.21.15.83:0/3096801298 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf600c4550 0x7fbf60151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:27.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.191+0000 7fbf656c7640 1 -- 172.21.15.83:0/3096801298 >> 172.21.15.83:0/3096801298 conn(0x7fbf600c9610 msgr2=0x7fbf601532f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:27.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.191+0000 7fbf656c7640 1 -- 172.21.15.83:0/3096801298 shutdown_connections 2024-09-19T07:27:27.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:27.191+0000 7fbf656c7640 1 -- 172.21.15.83:0/3096801298 wait complete. 2024-09-19T07:27:27.210 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:27:28.474 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-19T07:27:28.475 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:27:28.475 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:27:28.477 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:27:28.478 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:28.558 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:27:28.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63ca4fb640 1 Processor -- start 2024-09-19T07:27:28.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63ca4fb640 1 -- start start 2024-09-19T07:27:28.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63ca4fb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f63c40c6120 0x7f63c40c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:28.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63ca4fb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f63c41592f0 0x7f63c4157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:28.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63ca4fb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c4156200 con 0x7f63c4155d70 2024-09-19T07:27:28.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63ca4fb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f63c40c9930 con 0x7f63c41592f0 2024-09-19T07:27:28.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63ca4fb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f63c41564e0 con 0x7f63c40c6120 2024-09-19T07:27:28.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63c8a98640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f63c41592f0 0x7f63c4157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:28.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63c8a98640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f63c41592f0 0x7f63c4157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34138/0 (socket says 172.21.15.83:34138) 2024-09-19T07:27:28.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63c8a98640 1 -- 172.21.15.83:0/1604249010 learned_addr learned my addr 172.21.15.83:0/1604249010 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:28.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63c3fff640 1 --2- 172.21.15.83:0/1604249010 >> v2:172.21.15.184:3300/0 conn(0x7f63c40c6120 0x7f63c40c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:28.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63c8a98640 1 -- 172.21.15.83:0/1604249010 >> v2:172.21.15.184:3300/0 conn(0x7f63c40c6120 msgr2=0x7f63c40c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:28.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63c8a98640 1 --2- 172.21.15.83:0/1604249010 >> v2:172.21.15.184:3300/0 conn(0x7f63c40c6120 0x7f63c40c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:28.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63c8a98640 1 -- 172.21.15.83:0/1604249010 >> v1:172.21.15.184:6789/0 conn(0x7f63c4155d70 legacy=0x7f63c41575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:28.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.623+0000 7f63c8a98640 1 -- 172.21.15.83:0/1604249010 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c41567c0 con 0x7f63c41592f0 2024-09-19T07:27:28.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.627+0000 7f63c8a98640 1 --2- 172.21.15.83:0/1604249010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c41592f0 0x7f63c4157ce0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f63c408d540 tx=0x7f63bc01c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73a2f81dd245cc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:28.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.627+0000 7f63c37fe640 1 -- 172.21.15.83:0/1604249010 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0a5670 con 0x7f63c41592f0 2024-09-19T07:27:28.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.627+0000 7f63c37fe640 1 -- 172.21.15.83:0/1604249010 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63bc0a5810 con 0x7f63c41592f0 2024-09-19T07:27:28.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.627+0000 7f63c37fe640 1 -- 172.21.15.83:0/1604249010 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0a5be0 con 0x7f63c41592f0 2024-09-19T07:27:28.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.627+0000 7f63ca4fb640 1 -- 172.21.15.83:0/1604249010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c41592f0 msgr2=0x7f63c4157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:28.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.627+0000 7f63ca4fb640 1 --2- 172.21.15.83:0/1604249010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c41592f0 0x7f63c4157ce0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f63c408d540 tx=0x7f63bc01c9b0 comp rx=0 tx=0).stop 2024-09-19T07:27:28.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.627+0000 7f63ca4fb640 1 -- 172.21.15.83:0/1604249010 shutdown_connections 2024-09-19T07:27:28.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.627+0000 7f63ca4fb640 1 --2- 172.21.15.83:0/1604249010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c41592f0 0x7f63c4157ce0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:28.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.627+0000 7f63ca4fb640 1 --2- 172.21.15.83:0/1604249010 >> v2:172.21.15.184:3300/0 conn(0x7f63c40c6120 0x7f63c40c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:28.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.627+0000 7f63ca4fb640 1 -- 172.21.15.83:0/1604249010 >> 172.21.15.83:0/1604249010 conn(0x7f63c406bc80 msgr2=0x7f63c406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:28.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.627+0000 7f63ca4fb640 1 -- 172.21.15.83:0/1604249010 shutdown_connections 2024-09-19T07:27:28.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.627+0000 7f63ca4fb640 1 -- 172.21.15.83:0/1604249010 wait complete. 2024-09-19T07:27:28.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63ca4fb640 1 Processor -- start 2024-09-19T07:27:28.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63ca4fb640 1 -- start start 2024-09-19T07:27:28.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63ca4fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c40c6120 0x7f63c40ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:28.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63ca4fb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63c41592f0 0x7f63c4147e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:28.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63ca4fb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63c40ae000 0x7f63c41483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:28.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63ca4fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f63c4153860 con 0x7f63c40c6120 2024-09-19T07:27:28.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63ca4fb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f63c4162ea0 con 0x7f63c40ae000 2024-09-19T07:27:28.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63ca4fb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f63c41f1b00 con 0x7f63c41592f0 2024-09-19T07:27:28.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c40c6120 0x7f63c40ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:28.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c40c6120 0x7f63c40ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34148/0 (socket says 172.21.15.83:34148) 2024-09-19T07:27:28.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c3fff640 1 -- 172.21.15.83:0/3850171522 learned_addr learned my addr 172.21.15.83:0/3850171522 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:28.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c3fff640 1 -- 172.21.15.83:0/3850171522 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63c41592f0 msgr2=0x7f63c4147e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:28.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c3fff640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63c41592f0 0x7f63c4147e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:28.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c8a98640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63c40ae000 0x7f63c41483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:28.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c3fff640 1 -- 172.21.15.83:0/3850171522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63c40ae000 msgr2=0x7f63c41483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:28.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c3fff640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63c40ae000 0x7f63c41483b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:28.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c3fff640 1 -- 172.21.15.83:0/3850171522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c414a1a0 con 0x7f63c40c6120 2024-09-19T07:27:28.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c3fff640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c40c6120 0x7f63c40ca140 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f63b400d800 tx=0x7f63b400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:28.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c17fa640 1 -- 172.21.15.83:0/3850171522 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b400b660 con 0x7f63c40c6120 2024-09-19T07:27:28.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c17fa640 1 -- 172.21.15.83:0/3850171522 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63b4019070 con 0x7f63c40c6120 2024-09-19T07:27:28.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63ca4fb640 1 -- 172.21.15.83:0/3850171522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c414a480 con 0x7f63c40c6120 2024-09-19T07:27:28.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c17fa640 1 -- 172.21.15.83:0/3850171522 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b400ba00 con 0x7f63c40c6120 2024-09-19T07:27:28.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63ca4fb640 1 -- 172.21.15.83:0/3850171522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c40cbdc0 con 0x7f63c40c6120 2024-09-19T07:27:28.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63ca4fb640 1 -- 172.21.15.83:0/3850171522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c4149f90 con 0x7f63c40c6120 2024-09-19T07:27:28.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c17fa640 1 -- 172.21.15.83:0/3850171522 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f63b400bba0 con 0x7f63c40c6120 2024-09-19T07:27:28.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.631+0000 7f63c17fa640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f63a40678c0 0x7f63a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:28.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.635+0000 7f63c17fa640 1 -- 172.21.15.83:0/3850171522 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7f63b40867d0 con 0x7f63c40c6120 2024-09-19T07:27:28.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.635+0000 7f63bb7fe640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f63a40678c0 0x7f63a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:28.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.635+0000 7f63bb7fe640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f63a40678c0 0x7f63a4069d80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f63c415fb10 tx=0x7f63ac0079c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:28.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.639+0000 7f63c17fa640 1 -- 172.21.15.83:0/3850171522 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f63b4056c50 con 0x7f63c40c6120 2024-09-19T07:27:28.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 -- 172.21.15.83:0/3850171522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63c40c6500 con 0x7f63c40c6120 2024-09-19T07:27:28.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63c17fa640 1 -- 172.21.15.83:0/3850171522 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5707 (secure 0 0 0) 0x7f63b4056550 con 0x7f63c40c6120 2024-09-19T07:27:28.863 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:28.863 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":67,"btime":"2024-09-19T07:27:21:202637+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:28.864 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 67 2024-09-19T07:27:28.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 -- 172.21.15.83:0/3850171522 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f63a40678c0 msgr2=0x7f63a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:28.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f63a40678c0 0x7f63a4069d80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f63c415fb10 tx=0x7f63ac0079c0 comp rx=0 tx=0).stop 2024-09-19T07:27:28.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 -- 172.21.15.83:0/3850171522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c40c6120 msgr2=0x7f63c40ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:28.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c40c6120 0x7f63c40ca140 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f63b400d800 tx=0x7f63b400dcc0 comp rx=0 tx=0).stop 2024-09-19T07:27:28.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 -- 172.21.15.83:0/3850171522 shutdown_connections 2024-09-19T07:27:28.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f63a40678c0 0x7f63a4069d80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:28.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63c40ae000 0x7f63c41483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:28.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63c41592f0 0x7f63c4147e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:28.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 --2- 172.21.15.83:0/3850171522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c40c6120 0x7f63c40ca140 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:28.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 -- 172.21.15.83:0/3850171522 >> 172.21.15.83:0/3850171522 conn(0x7f63c406bc80 msgr2=0x7f63c406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:28.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 -- 172.21.15.83:0/3850171522 shutdown_connections 2024-09-19T07:27:28.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:28.863+0000 7f63ca4fb640 1 -- 172.21.15.83:0/3850171522 wait complete. 2024-09-19T07:27:28.883 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-19T07:27:29.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:29.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f061cc3f640 1 Processor -- start 2024-09-19T07:27:29.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f061cc3f640 1 -- start start 2024-09-19T07:27:29.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f061cc3f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0618074000 0x7f06180c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:29.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f061cc3f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f06181573a0 0x7f0618157830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:29.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f061cc3f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0618153a50 con 0x7f06181546a0 2024-09-19T07:27:29.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f061cc3f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f06180730e0 con 0x7f0618074000 2024-09-19T07:27:29.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f061cc3f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f06180733c0 con 0x7f06181573a0 2024-09-19T07:27:29.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f0615d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0618074000 0x7f06180c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:29.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f0615d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0618074000 0x7f06180c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34158/0 (socket says 172.21.15.83:34158) 2024-09-19T07:27:29.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f0615d9b640 1 -- 172.21.15.83:0/2707781905 learned_addr learned my addr 172.21.15.83:0/2707781905 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:29.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f0615d9b640 1 -- 172.21.15.83:0/2707781905 >> v2:172.21.15.184:3300/0 conn(0x7f06181573a0 msgr2=0x7f0618157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:29.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f0616d9d640 1 --2- 172.21.15.83:0/2707781905 >> v2:172.21.15.184:3300/0 conn(0x7f06181573a0 0x7f0618157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:29.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f0615d9b640 1 --2- 172.21.15.83:0/2707781905 >> v2:172.21.15.184:3300/0 conn(0x7f06181573a0 0x7f0618157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:29.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f0615d9b640 1 -- 172.21.15.83:0/2707781905 >> v1:172.21.15.184:6789/0 conn(0x7f06181546a0 legacy=0x7f061806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:29.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.355+0000 7f0615d9b640 1 -- 172.21.15.83:0/2707781905 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06180736a0 con 0x7f0618074000 2024-09-19T07:27:29.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f0615d9b640 1 --2- 172.21.15.83:0/2707781905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0618074000 0x7f06180c70f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f06180ab6a0 tx=0x7f060c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587fa381e5922af4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:29.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061559a640 1 -- 172.21.15.83:0/2707781905 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c0ab5f0 con 0x7f0618074000 2024-09-19T07:27:29.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061559a640 1 -- 172.21.15.83:0/2707781905 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060c0b0070 con 0x7f0618074000 2024-09-19T07:27:29.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061559a640 1 -- 172.21.15.83:0/2707781905 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c0ab9b0 con 0x7f0618074000 2024-09-19T07:27:29.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061cc3f640 1 -- 172.21.15.83:0/2707781905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0618074000 msgr2=0x7f06180c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:29.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061cc3f640 1 --2- 172.21.15.83:0/2707781905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0618074000 0x7f06180c70f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f06180ab6a0 tx=0x7f060c01c4e0 comp rx=0 tx=0).stop 2024-09-19T07:27:29.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061cc3f640 1 -- 172.21.15.83:0/2707781905 shutdown_connections 2024-09-19T07:27:29.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061cc3f640 1 --2- 172.21.15.83:0/2707781905 >> v2:172.21.15.184:3300/0 conn(0x7f06181573a0 0x7f0618157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:29.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061cc3f640 1 --2- 172.21.15.83:0/2707781905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0618074000 0x7f06180c70f0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:29.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061cc3f640 1 -- 172.21.15.83:0/2707781905 >> 172.21.15.83:0/2707781905 conn(0x7f06180c5060 msgr2=0x7f06180c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:29.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061cc3f640 1 -- 172.21.15.83:0/2707781905 shutdown_connections 2024-09-19T07:27:29.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061cc3f640 1 -- 172.21.15.83:0/2707781905 wait complete. 2024-09-19T07:27:29.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061cc3f640 1 Processor -- start 2024-09-19T07:27:29.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.359+0000 7f061cc3f640 1 -- start start 2024-09-19T07:27:29.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f061cc3f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0618074000 0x7f0618154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:29.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f061cc3f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06181546a0 0x7f06181f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:29.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f061cc3f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06181573a0 0x7f06181f2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:29.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f061cc3f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0618157ef0 con 0x7f06181546a0 2024-09-19T07:27:29.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f061cc3f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0618157d70 con 0x7f0618074000 2024-09-19T07:27:29.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f061cc3f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0618070230 con 0x7f06181573a0 2024-09-19T07:27:29.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0615d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06181546a0 0x7f06181f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:29.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0615d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06181546a0 0x7f06181f1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34160/0 (socket says 172.21.15.83:34160) 2024-09-19T07:27:29.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0615d9b640 1 -- 172.21.15.83:0/1062123700 learned_addr learned my addr 172.21.15.83:0/1062123700 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:29.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0616d9d640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06181573a0 0x7f06181f2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:29.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f061659c640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0618074000 0x7f0618154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:29.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0615d9b640 1 -- 172.21.15.83:0/1062123700 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06181573a0 msgr2=0x7f06181f2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:29.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0615d9b640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06181573a0 0x7f06181f2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:29.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0615d9b640 1 -- 172.21.15.83:0/1062123700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0618074000 msgr2=0x7f0618154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:29.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0615d9b640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0618074000 0x7f0618154e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:29.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0615d9b640 1 -- 172.21.15.83:0/1062123700 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0618156fe0 con 0x7f06181546a0 2024-09-19T07:27:29.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f061659c640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0618074000 0x7f0618154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:29.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0615d9b640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06181546a0 0x7f06181f1c50 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f060c01a110 tx=0x7f060c01ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:29.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0606ffd640 1 -- 172.21.15.83:0/1062123700 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c0be5f0 con 0x7f06181546a0 2024-09-19T07:27:29.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0606ffd640 1 -- 172.21.15.83:0/1062123700 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060c0be790 con 0x7f06181546a0 2024-09-19T07:27:29.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0606ffd640 1 -- 172.21.15.83:0/1062123700 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c0beb80 con 0x7f06181546a0 2024-09-19T07:27:29.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f061cc3f640 1 -- 172.21.15.83:0/1062123700 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06180c4280 con 0x7f06181546a0 2024-09-19T07:27:29.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f061cc3f640 1 -- 172.21.15.83:0/1062123700 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0618155350 con 0x7f06181546a0 2024-09-19T07:27:29.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f061cc3f640 1 -- 172.21.15.83:0/1062123700 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06180705f0 con 0x7f06181546a0 2024-09-19T07:27:29.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0606ffd640 1 -- 172.21.15.83:0/1062123700 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f060c0b43f0 con 0x7f06181546a0 2024-09-19T07:27:29.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0606ffd640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05f80678c0 0x7f05f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:29.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.363+0000 7f0606ffd640 1 -- 172.21.15.83:0/1062123700 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7f060c120b90 con 0x7f06181546a0 2024-09-19T07:27:29.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.367+0000 7f061659c640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05f80678c0 0x7f05f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:29.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.367+0000 7f061659c640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05f80678c0 0x7f05f8069d80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f06080016e0 tx=0x7f0608009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:29.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.371+0000 7f0606ffd640 1 -- 172.21.15.83:0/1062123700 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f060c0160d0 con 0x7f06181546a0 2024-09-19T07:27:29.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.591+0000 7f061cc3f640 1 -- 172.21.15.83:0/1062123700 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06181f19b0 con 0x7f06181546a0 2024-09-19T07:27:29.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f0606ffd640 1 -- 172.21.15.83:0/1062123700 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6529 (secure 0 0 0) 0x7f060c0f0ae0 con 0x7f06181546a0 2024-09-19T07:27:29.594 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:29.594 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":68,"btime":"2024-09-19T07:27:29:234190+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5934,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:29.594 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 68 2024-09-19T07:27:29.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f061cc3f640 1 -- 172.21.15.83:0/1062123700 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05f80678c0 msgr2=0x7f05f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:29.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f061cc3f640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05f80678c0 0x7f05f8069d80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f06080016e0 tx=0x7f0608009040 comp rx=0 tx=0).stop 2024-09-19T07:27:29.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f061cc3f640 1 -- 172.21.15.83:0/1062123700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06181546a0 msgr2=0x7f06181f1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:29.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f061cc3f640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06181546a0 0x7f06181f1c50 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f060c01a110 tx=0x7f060c01ad20 comp rx=0 tx=0).stop 2024-09-19T07:27:29.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f061cc3f640 1 -- 172.21.15.83:0/1062123700 shutdown_connections 2024-09-19T07:27:29.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f061cc3f640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05f80678c0 0x7f05f8069d80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:29.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f061cc3f640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06181573a0 0x7f06181f2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:29.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f061cc3f640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06181546a0 0x7f06181f1c50 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:29.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f061cc3f640 1 --2- 172.21.15.83:0/1062123700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0618074000 0x7f0618154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:29.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f061cc3f640 1 -- 172.21.15.83:0/1062123700 >> 172.21.15.83:0/1062123700 conn(0x7f06180c5060 msgr2=0x7f061806e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:29.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f061cc3f640 1 -- 172.21.15.83:0/1062123700 shutdown_connections 2024-09-19T07:27:29.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:29.595+0000 7f061cc3f640 1 -- 172.21.15.83:0/1062123700 wait complete. 2024-09-19T07:27:29.615 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:27:30.884 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:31.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0ed540640 1 Processor -- start 2024-09-19T07:27:31.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0ed540640 1 -- start start 2024-09-19T07:27:31.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0ed540640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe0e806f950 0x7fe0e80716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:31.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0ed540640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe0e8071e20 0x7fe0e8150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:31.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0ed540640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0e80c37c0 con 0x7fe0e8151410 2024-09-19T07:27:31.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0ed540640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe0e80c4670 con 0x7fe0e8071e20 2024-09-19T07:27:31.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0ed540640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe0e80c3a60 con 0x7fe0e806f950 2024-09-19T07:27:31.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0dffff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe0e8071e20 0x7fe0e8150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:31.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0e6ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe0e806f950 0x7fe0e80716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:31.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0dffff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe0e8071e20 0x7fe0e8150ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:45466/0 (socket says 172.21.15.83:45466) 2024-09-19T07:27:31.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0e77fe640 1 --1- 172.21.15.83:0/4257185898 >> v1:172.21.15.184:6789/0 conn(0x7fe0e8151410 0x7fe0e81575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:34044/0 (socket says 172.21.15.83:34044) 2024-09-19T07:27:31.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0dffff640 1 -- 172.21.15.83:0/4257185898 learned_addr learned my addr 172.21.15.83:0/4257185898 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:31.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0dffff640 1 -- 172.21.15.83:0/4257185898 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe0e806f950 msgr2=0x7fe0e80716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:31.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0dffff640 1 --2- 172.21.15.83:0/4257185898 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe0e806f950 0x7fe0e80716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0dffff640 1 -- 172.21.15.83:0/4257185898 >> v1:172.21.15.184:6789/0 conn(0x7fe0e8151410 legacy=0x7fe0e81575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:31.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.035+0000 7fe0dffff640 1 -- 172.21.15.83:0/4257185898 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e80c3d40 con 0x7fe0e8071e20 2024-09-19T07:27:31.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0dffff640 1 --2- 172.21.15.83:0/4257185898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe0e8071e20 0x7fe0e8150ed0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0009870 tx=0x7fe0d001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efccda3224e27090 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:31.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0e67fc640 1 -- 172.21.15.83:0/4257185898 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00aa600 con 0x7fe0e8071e20 2024-09-19T07:27:31.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0e67fc640 1 -- 172.21.15.83:0/4257185898 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0d00af070 con 0x7fe0e8071e20 2024-09-19T07:27:31.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0e67fc640 1 -- 172.21.15.83:0/4257185898 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00aa9c0 con 0x7fe0e8071e20 2024-09-19T07:27:31.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0ed540640 1 -- 172.21.15.83:0/4257185898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe0e8071e20 msgr2=0x7fe0e8150ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:31.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0ed540640 1 --2- 172.21.15.83:0/4257185898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe0e8071e20 0x7fe0e8150ed0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0009870 tx=0x7fe0d001c4a0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0ed540640 1 -- 172.21.15.83:0/4257185898 shutdown_connections 2024-09-19T07:27:31.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0ed540640 1 --2- 172.21.15.83:0/4257185898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe0e8071e20 0x7fe0e8150ed0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0ed540640 1 --2- 172.21.15.83:0/4257185898 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe0e806f950 0x7fe0e80716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0ed540640 1 -- 172.21.15.83:0/4257185898 >> 172.21.15.83:0/4257185898 conn(0x7fe0e80c48b0 msgr2=0x7fe0e80c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:31.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0ed540640 1 -- 172.21.15.83:0/4257185898 shutdown_connections 2024-09-19T07:27:31.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0ed540640 1 -- 172.21.15.83:0/4257185898 wait complete. 2024-09-19T07:27:31.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0ed540640 1 Processor -- start 2024-09-19T07:27:31.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.039+0000 7fe0ed540640 1 -- start start 2024-09-19T07:27:31.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0ed540640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0e806f950 0x7fe0e81eef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:31.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0ed540640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe0e8071e20 0x7fe0e81533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:31.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0ed540640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe0e8151410 0x7fe0e81446b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:31.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0ed540640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e81519a0 con 0x7fe0e806f950 2024-09-19T07:27:31.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0ed540640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e8151820 con 0x7fe0e8071e20 2024-09-19T07:27:31.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0ed540640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe0e806fd30 con 0x7fe0e8151410 2024-09-19T07:27:31.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0e6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0e806f950 0x7fe0e81eef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:31.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0e6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0e806f950 0x7fe0e81eef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36482/0 (socket says 172.21.15.83:36482) 2024-09-19T07:27:31.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0e6ffd640 1 -- 172.21.15.83:0/1794492956 learned_addr learned my addr 172.21.15.83:0/1794492956 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:31.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0e77fe640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe0e8151410 0x7fe0e81446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:31.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0dffff640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe0e8071e20 0x7fe0e81533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:31.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0e6ffd640 1 -- 172.21.15.83:0/1794492956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe0e8151410 msgr2=0x7fe0e81446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:31.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0e6ffd640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe0e8151410 0x7fe0e81446b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0e6ffd640 1 -- 172.21.15.83:0/1794492956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe0e8071e20 msgr2=0x7fe0e81533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:31.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0e6ffd640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe0e8071e20 0x7fe0e81533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0e6ffd640 1 -- 172.21.15.83:0/1794492956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e806d4e0 con 0x7fe0e806f950 2024-09-19T07:27:31.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0dffff640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe0e8071e20 0x7fe0e81533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:31.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0e77fe640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe0e8151410 0x7fe0e81446b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:31.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0e6ffd640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0e806f950 0x7fe0e81eef00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800e960 tx=0x7fe0d800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:31.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0df7fe640 1 -- 172.21.15.83:0/1794492956 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8017070 con 0x7fe0e806f950 2024-09-19T07:27:31.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0df7fe640 1 -- 172.21.15.83:0/1794492956 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0d800ca00 con 0x7fe0e806f950 2024-09-19T07:27:31.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0ed540640 1 -- 172.21.15.83:0/1794492956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e80c5ed0 con 0x7fe0e806f950 2024-09-19T07:27:31.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0df7fe640 1 -- 172.21.15.83:0/1794492956 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d800cdb0 con 0x7fe0e806f950 2024-09-19T07:27:31.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0ed540640 1 -- 172.21.15.83:0/1794492956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e80c6490 con 0x7fe0e806f950 2024-09-19T07:27:31.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0ed540640 1 -- 172.21.15.83:0/1794492956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e8071680 con 0x7fe0e806f950 2024-09-19T07:27:31.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0df7fe640 1 -- 172.21.15.83:0/1794492956 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe0d801a070 con 0x7fe0e806f950 2024-09-19T07:27:31.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0df7fe640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe0c80678f0 0x7fe0c8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:31.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.043+0000 7fe0df7fe640 1 -- 172.21.15.83:0/1794492956 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7fe0d8086370 con 0x7fe0e806f950 2024-09-19T07:27:31.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.047+0000 7fe0dffff640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe0c80678f0 0x7fe0c8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:31.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.047+0000 7fe0dffff640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe0c80678f0 0x7fe0c8069db0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe0d00ae000 tx=0x7fe0d0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:31.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.051+0000 7fe0df7fe640 1 -- 172.21.15.83:0/1794492956 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe0d80567f0 con 0x7fe0e806f950 2024-09-19T07:27:31.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.283+0000 7fe0ed540640 1 -- 172.21.15.83:0/1794492956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0e8072cb0 con 0x7fe0e806f950 2024-09-19T07:27:31.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.283+0000 7fe0df7fe640 1 -- 172.21.15.83:0/1794492956 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6529 (secure 0 0 0) 0x7fe0d80560f0 con 0x7fe0e806f950 2024-09-19T07:27:31.282 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:31.282 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":68,"btime":"2024-09-19T07:27:29:234190+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5934,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:31.282 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 68 2024-09-19T07:27:31.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.287+0000 7fe0ed540640 1 -- 172.21.15.83:0/1794492956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe0c80678f0 msgr2=0x7fe0c8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:31.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.287+0000 7fe0ed540640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe0c80678f0 0x7fe0c8069db0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe0d00ae000 tx=0x7fe0d0008040 comp rx=0 tx=0).stop 2024-09-19T07:27:31.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.287+0000 7fe0ed540640 1 -- 172.21.15.83:0/1794492956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0e806f950 msgr2=0x7fe0e81eef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:31.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.287+0000 7fe0ed540640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0e806f950 0x7fe0e81eef00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800e960 tx=0x7fe0d800ee20 comp rx=0 tx=0).stop 2024-09-19T07:27:31.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.287+0000 7fe0ed540640 1 -- 172.21.15.83:0/1794492956 shutdown_connections 2024-09-19T07:27:31.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.287+0000 7fe0ed540640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe0c80678f0 0x7fe0c8069db0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.287+0000 7fe0ed540640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe0e8151410 0x7fe0e81446b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.287+0000 7fe0ed540640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe0e8071e20 0x7fe0e81533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.287+0000 7fe0ed540640 1 --2- 172.21.15.83:0/1794492956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0e806f950 0x7fe0e81eef00 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.287+0000 7fe0ed540640 1 -- 172.21.15.83:0/1794492956 >> 172.21.15.83:0/1794492956 conn(0x7fe0e80c48b0 msgr2=0x7fe0e8072200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:31.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.287+0000 7fe0ed540640 1 -- 172.21.15.83:0/1794492956 shutdown_connections 2024-09-19T07:27:31.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.287+0000 7fe0ed540640 1 -- 172.21.15.83:0/1794492956 wait complete. 2024-09-19T07:27:31.304 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-19T07:27:31.304 INFO:tasks.mds_thrash.fs.[a]:waiting for 23 secs before thrashing 2024-09-19T07:27:31.616 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:31.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f53983b2640 1 Processor -- start 2024-09-19T07:27:31.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f53983b2640 1 -- start start 2024-09-19T07:27:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f53983b2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5390153570 0x7f53900c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f53983b2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53900c5110 0x7f5390159270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f53983b2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53900c3c20 con 0x7f53900c4130 2024-09-19T07:27:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f53983b2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f53900c89f0 con 0x7f5390153570 2024-09-19T07:27:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f53983b2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f53900c8cd0 con 0x7f53900c5110 2024-09-19T07:27:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f539614e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5390153570 0x7f53900c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f539594d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53900c5110 0x7f5390159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f539614e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5390153570 0x7f53900c4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:45488/0 (socket says 172.21.15.83:45488) 2024-09-19T07:27:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f539614e640 1 -- 172.21.15.83:0/3740149449 learned_addr learned my addr 172.21.15.83:0/3740149449 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f539594d640 1 -- 172.21.15.83:0/3740149449 >> v2:172.21.15.184:3300/0 conn(0x7f5390153570 msgr2=0x7f53900c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f539594d640 1 --2- 172.21.15.83:0/3740149449 >> v2:172.21.15.184:3300/0 conn(0x7f5390153570 0x7f53900c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f539594d640 1 -- 172.21.15.83:0/3740149449 >> v1:172.21.15.184:6789/0 conn(0x7f53900c4130 legacy=0x7f53901597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f539594d640 1 -- 172.21.15.83:0/3740149449 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53900c8fb0 con 0x7f53900c5110 2024-09-19T07:27:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.743+0000 7f539614e640 1 --2- 172.21.15.83:0/3740149449 >> v2:172.21.15.184:3300/0 conn(0x7f5390153570 0x7f53900c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:27:31.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.747+0000 7f539594d640 1 --2- 172.21.15.83:0/3740149449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53900c5110 0x7f5390159270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f53800097f0 tx=0x7f538001c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a407c51f645041 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:31.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.747+0000 7f539514c640 1 -- 172.21.15.83:0/3740149449 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53800b1070 con 0x7f53900c5110 2024-09-19T07:27:31.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.747+0000 7f539514c640 1 -- 172.21.15.83:0/3740149449 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53800ac600 con 0x7f53900c5110 2024-09-19T07:27:31.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.747+0000 7f539514c640 1 -- 172.21.15.83:0/3740149449 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53800ac9d0 con 0x7f53900c5110 2024-09-19T07:27:31.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.747+0000 7f53983b2640 1 -- 172.21.15.83:0/3740149449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53900c5110 msgr2=0x7f5390159270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:31.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.747+0000 7f53983b2640 1 --2- 172.21.15.83:0/3740149449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53900c5110 0x7f5390159270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f53800097f0 tx=0x7f538001c380 comp rx=0 tx=0).stop 2024-09-19T07:27:31.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.747+0000 7f53983b2640 1 -- 172.21.15.83:0/3740149449 shutdown_connections 2024-09-19T07:27:31.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.747+0000 7f53983b2640 1 --2- 172.21.15.83:0/3740149449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53900c5110 0x7f5390159270 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.747+0000 7f53983b2640 1 --2- 172.21.15.83:0/3740149449 >> v2:172.21.15.184:3300/0 conn(0x7f5390153570 0x7f53900c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.747+0000 7f53983b2640 1 -- 172.21.15.83:0/3740149449 >> 172.21.15.83:0/3740149449 conn(0x7f53900c6110 msgr2=0x7f53900c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:31.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.747+0000 7f53983b2640 1 -- 172.21.15.83:0/3740149449 shutdown_connections 2024-09-19T07:27:31.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.747+0000 7f53983b2640 1 -- 172.21.15.83:0/3740149449 wait complete. 2024-09-19T07:27:31.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f53983b2640 1 Processor -- start 2024-09-19T07:27:31.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f53983b2640 1 -- start start 2024-09-19T07:27:31.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f53983b2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53900c4130 0x7f53901f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:31.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f53983b2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53900c5110 0x7f53901f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:31.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f53983b2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5390153570 0x7f539014bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:31.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f53983b2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5390163070 con 0x7f53900c4130 2024-09-19T07:27:31.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f53983b2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5390162ef0 con 0x7f53900c5110 2024-09-19T07:27:31.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f53983b2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f53901631f0 con 0x7f5390153570 2024-09-19T07:27:31.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f539614e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53900c4130 0x7f53901f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f539614e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53900c4130 0x7f53901f1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36496/0 (socket says 172.21.15.83:36496) 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f539614e640 1 -- 172.21.15.83:0/2582145152 learned_addr learned my addr 172.21.15.83:0/2582145152 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f539594d640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53900c5110 0x7f53901f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f539694f640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5390153570 0x7f539014bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f539614e640 1 -- 172.21.15.83:0/2582145152 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5390153570 msgr2=0x7f539014bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f539614e640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5390153570 0x7f539014bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f539614e640 1 -- 172.21.15.83:0/2582145152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53900c5110 msgr2=0x7f53901f2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f539614e640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53900c5110 0x7f53901f2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f539614e640 1 -- 172.21.15.83:0/2582145152 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53901f28a0 con 0x7f53900c4130 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f539594d640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53900c5110 0x7f53901f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f539614e640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53900c4130 0x7f53901f1d20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f538c00ea00 tx=0x7f538c00eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f5386ffd640 1 -- 172.21.15.83:0/2582145152 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c017070 con 0x7f53900c4130 2024-09-19T07:27:31.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f5386ffd640 1 -- 172.21.15.83:0/2582145152 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f538c00ca60 con 0x7f53900c4130 2024-09-19T07:27:31.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f53983b2640 1 -- 172.21.15.83:0/2582145152 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f539014a7e0 con 0x7f53900c4130 2024-09-19T07:27:31.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f5386ffd640 1 -- 172.21.15.83:0/2582145152 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c00ce10 con 0x7f53900c4130 2024-09-19T07:27:31.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f53983b2640 1 -- 172.21.15.83:0/2582145152 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53901efa20 con 0x7f53900c4130 2024-09-19T07:27:31.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.751+0000 7f53983b2640 1 -- 172.21.15.83:0/2582145152 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53900c4bd0 con 0x7f53900c4130 2024-09-19T07:27:31.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.755+0000 7f5386ffd640 1 -- 172.21.15.83:0/2582145152 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f538c01a070 con 0x7f53900c4130 2024-09-19T07:27:31.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.755+0000 7f5386ffd640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53740678c0 0x7f5374069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:31.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.755+0000 7f5386ffd640 1 -- 172.21.15.83:0/2582145152 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7f538c086a00 con 0x7f53900c4130 2024-09-19T07:27:31.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.755+0000 7f539594d640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53740678c0 0x7f5374069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:31.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.755+0000 7f539594d640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53740678c0 0x7f5374069d80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f53800ae000 tx=0x7f5380008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:31.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.759+0000 7f5386ffd640 1 -- 172.21.15.83:0/2582145152 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f538c056f60 con 0x7f53900c4130 2024-09-19T07:27:31.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 -- 172.21.15.83:0/2582145152 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5390068290 con 0x7f53900c4130 2024-09-19T07:27:31.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f5386ffd640 1 -- 172.21.15.83:0/2582145152 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6529 (secure 0 0 0) 0x7f538c056860 con 0x7f53900c4130 2024-09-19T07:27:31.963 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:31.963 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":68,"btime":"2024-09-19T07:27:29:234190+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5934,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:31.963 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 68 2024-09-19T07:27:31.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 -- 172.21.15.83:0/2582145152 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53740678c0 msgr2=0x7f5374069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:31.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53740678c0 0x7f5374069d80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f53800ae000 tx=0x7f5380008040 comp rx=0 tx=0).stop 2024-09-19T07:27:31.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 -- 172.21.15.83:0/2582145152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53900c4130 msgr2=0x7f53901f1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:31.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53900c4130 0x7f53901f1d20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f538c00ea00 tx=0x7f538c00eec0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 -- 172.21.15.83:0/2582145152 shutdown_connections 2024-09-19T07:27:31.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53740678c0 0x7f5374069d80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5390153570 0x7f539014bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53900c5110 0x7f53901f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 --2- 172.21.15.83:0/2582145152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53900c4130 0x7f53901f1d20 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:31.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 -- 172.21.15.83:0/2582145152 >> 172.21.15.83:0/2582145152 conn(0x7f53900c6110 msgr2=0x7f539014ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:31.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 -- 172.21.15.83:0/2582145152 shutdown_connections 2024-09-19T07:27:31.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:31.967+0000 7f53983b2640 1 -- 172.21.15.83:0/2582145152 wait complete. 2024-09-19T07:27:31.980 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:27:32.055 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:27:32.057 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:27:33.981 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:34.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef1c6da640 1 Processor -- start 2024-09-19T07:27:34.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef1c6da640 1 -- start start 2024-09-19T07:27:34.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef1c6da640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef140c6120 0x7fef140c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:34.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef1c6da640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fef14155d70 0x7fef141575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:34.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef1c6da640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef14156200 con 0x7fef141592f0 2024-09-19T07:27:34.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef1c6da640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fef140c9930 con 0x7fef14155d70 2024-09-19T07:27:34.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef1c6da640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef141564e0 con 0x7fef140c6120 2024-09-19T07:27:34.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef19c75640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fef14155d70 0x7fef141575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:34.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef19c75640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fef14155d70 0x7fef141575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36506/0 (socket says 172.21.15.83:36506) 2024-09-19T07:27:34.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef19c75640 1 -- 172.21.15.83:0/3601867299 learned_addr learned my addr 172.21.15.83:0/3601867299 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:34.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef1a476640 1 --2- 172.21.15.83:0/3601867299 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef140c6120 0x7fef140c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:34.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef19c75640 1 -- 172.21.15.83:0/3601867299 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef140c6120 msgr2=0x7fef140c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:34.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef19c75640 1 --2- 172.21.15.83:0/3601867299 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef140c6120 0x7fef140c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:34.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef19c75640 1 -- 172.21.15.83:0/3601867299 >> v1:172.21.15.184:6789/0 conn(0x7fef141592f0 legacy=0x7fef14157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:34.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.127+0000 7fef19c75640 1 -- 172.21.15.83:0/3601867299 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef141567c0 con 0x7fef14155d70 2024-09-19T07:27:34.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef19c75640 1 --2- 172.21.15.83:0/3601867299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef14155d70 0x7fef141575d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fef04009870 tx=0x7fef0401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad993f8d99ef73c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:34.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef19474640 1 -- 172.21.15.83:0/3601867299 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef040b0070 con 0x7fef14155d70 2024-09-19T07:27:34.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef19474640 1 -- 172.21.15.83:0/3601867299 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef040ab600 con 0x7fef14155d70 2024-09-19T07:27:34.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef19474640 1 -- 172.21.15.83:0/3601867299 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef040ab9d0 con 0x7fef14155d70 2024-09-19T07:27:34.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 -- 172.21.15.83:0/3601867299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef14155d70 msgr2=0x7fef141575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:34.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 --2- 172.21.15.83:0/3601867299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef14155d70 0x7fef141575d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fef04009870 tx=0x7fef0401c520 comp rx=0 tx=0).stop 2024-09-19T07:27:34.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 -- 172.21.15.83:0/3601867299 shutdown_connections 2024-09-19T07:27:34.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 --2- 172.21.15.83:0/3601867299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef14155d70 0x7fef141575d0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:34.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 --2- 172.21.15.83:0/3601867299 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef140c6120 0x7fef140c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:34.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 -- 172.21.15.83:0/3601867299 >> 172.21.15.83:0/3601867299 conn(0x7fef1406bc80 msgr2=0x7fef1406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:34.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 -- 172.21.15.83:0/3601867299 shutdown_connections 2024-09-19T07:27:34.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 -- 172.21.15.83:0/3601867299 wait complete. 2024-09-19T07:27:34.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 Processor -- start 2024-09-19T07:27:34.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 -- start start 2024-09-19T07:27:34.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef140c6120 0x7fef140cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:34.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef141592f0 0x7fef1414ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:34.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1414bfc0 0x7fef14145b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:34.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fef14153860 con 0x7fef1414bfc0 2024-09-19T07:27:34.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fef14162ea0 con 0x7fef141592f0 2024-09-19T07:27:34.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1c6da640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef1414ee50 con 0x7fef140c6120 2024-09-19T07:27:34.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1ac77640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1414bfc0 0x7fef14145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:34.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1ac77640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1414bfc0 0x7fef14145b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36508/0 (socket says 172.21.15.83:36508) 2024-09-19T07:27:34.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1ac77640 1 -- 172.21.15.83:0/973751633 learned_addr learned my addr 172.21.15.83:0/973751633 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:34.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef1a476640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef140c6120 0x7fef140cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:34.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.131+0000 7fef19c75640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef141592f0 0x7fef1414ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:34.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef1ac77640 1 -- 172.21.15.83:0/973751633 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef140c6120 msgr2=0x7fef140cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:34.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef1ac77640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef140c6120 0x7fef140cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:34.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef1ac77640 1 -- 172.21.15.83:0/973751633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef141592f0 msgr2=0x7fef1414ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:34.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef1ac77640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef141592f0 0x7fef1414ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:34.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef1ac77640 1 -- 172.21.15.83:0/973751633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1414a250 con 0x7fef1414bfc0 2024-09-19T07:27:34.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef19c75640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef141592f0 0x7fef1414ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:27:34.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef1ac77640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1414bfc0 0x7fef14145b90 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00b9e0 tx=0x7fef0c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:34.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef0affd640 1 -- 172.21.15.83:0/973751633 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c003b80 con 0x7fef1414bfc0 2024-09-19T07:27:34.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef0affd640 1 -- 172.21.15.83:0/973751633 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef0c003d20 con 0x7fef1414bfc0 2024-09-19T07:27:34.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef1c6da640 1 -- 172.21.15.83:0/973751633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef1414a530 con 0x7fef1414bfc0 2024-09-19T07:27:34.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef0affd640 1 -- 172.21.15.83:0/973751633 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c00cb60 con 0x7fef1414bfc0 2024-09-19T07:27:34.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef1c6da640 1 -- 172.21.15.83:0/973751633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef1414aaf0 con 0x7fef1414bfc0 2024-09-19T07:27:34.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef1c6da640 1 -- 172.21.15.83:0/973751633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef140c6500 con 0x7fef1414bfc0 2024-09-19T07:27:34.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef0affd640 1 -- 172.21.15.83:0/973751633 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fef0c006d10 con 0x7fef1414bfc0 2024-09-19T07:27:34.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef0affd640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feef80678f0 0x7feef8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:34.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef0affd640 1 -- 172.21.15.83:0/973751633 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7fef0c085e90 con 0x7fef1414bfc0 2024-09-19T07:27:34.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef1a476640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feef80678f0 0x7feef8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:34.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.135+0000 7fef1a476640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feef80678f0 0x7feef8069db0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fef1000a970 tx=0x7fef10008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:34.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.143+0000 7fef0affd640 1 -- 172.21.15.83:0/973751633 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fef0c056310 con 0x7fef1414bfc0 2024-09-19T07:27:34.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.399+0000 7fef1c6da640 1 -- 172.21.15.83:0/973751633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef14160460 con 0x7fef1414bfc0 2024-09-19T07:27:34.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.399+0000 7fef0affd640 1 -- 172.21.15.83:0/973751633 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6529 (secure 0 0 0) 0x7fef0c055c10 con 0x7fef1414bfc0 2024-09-19T07:27:34.398 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:34.398 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":68,"btime":"2024-09-19T07:27:29:234190+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5934,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:34.398 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 68 2024-09-19T07:27:34.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.403+0000 7fef1c6da640 1 -- 172.21.15.83:0/973751633 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feef80678f0 msgr2=0x7feef8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:34.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.403+0000 7fef1c6da640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feef80678f0 0x7feef8069db0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fef1000a970 tx=0x7fef10008040 comp rx=0 tx=0).stop 2024-09-19T07:27:34.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.403+0000 7fef1c6da640 1 -- 172.21.15.83:0/973751633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1414bfc0 msgr2=0x7fef14145b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:34.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.403+0000 7fef1c6da640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1414bfc0 0x7fef14145b90 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00b9e0 tx=0x7fef0c00bea0 comp rx=0 tx=0).stop 2024-09-19T07:27:34.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.403+0000 7fef1c6da640 1 -- 172.21.15.83:0/973751633 shutdown_connections 2024-09-19T07:27:34.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.403+0000 7fef1c6da640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feef80678f0 0x7feef8069db0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:34.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.403+0000 7fef1c6da640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1414bfc0 0x7fef14145b90 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:34.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.403+0000 7fef1c6da640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef141592f0 0x7fef1414ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:34.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.403+0000 7fef1c6da640 1 --2- 172.21.15.83:0/973751633 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef140c6120 0x7fef140cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:34.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.403+0000 7fef1c6da640 1 -- 172.21.15.83:0/973751633 >> 172.21.15.83:0/973751633 conn(0x7fef1406bc80 msgr2=0x7fef1406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:34.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.403+0000 7fef1c6da640 1 -- 172.21.15.83:0/973751633 shutdown_connections 2024-09-19T07:27:34.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:34.403+0000 7fef1c6da640 1 -- 172.21.15.83:0/973751633 wait complete. 2024-09-19T07:27:34.419 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:27:36.421 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:36.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a07614640 1 Processor -- start 2024-09-19T07:27:36.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a07614640 1 -- start start 2024-09-19T07:27:36.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a07614640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a000c3ab0 0x7f0a00157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:36.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a07614640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a00073d90 con 0x7f0a000aa4a0 2024-09-19T07:27:36.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a07614640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a00153b30 con 0x7f0a0014dbe0 2024-09-19T07:27:36.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a07614640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0a00154700 con 0x7f0a000c3ab0 2024-09-19T07:27:36.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a05bb1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0a000aa4a0 0x7f0a00157550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59682/0 (socket says 172.21.15.83:59682) 2024-09-19T07:27:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a05bb1640 1 -- 172.21.15.83:0/4036537394 learned_addr learned my addr 172.21.15.83:0/4036537394 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a04baf640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a000c3ab0 0x7f0a00157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2466041687 0 0) 0x7f0a00073d90 con 0x7f0a000aa4a0 2024-09-19T07:27:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09ec003530 con 0x7f0a000aa4a0 2024-09-19T07:27:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 84792232 0 0) 0x7f09ec003530 con 0x7f0a000aa4a0 2024-09-19T07:27:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a000c3ab0 msgr2=0x7f0a00157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a000c3ab0 0x7f0a00157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 >> v1:172.21.15.184:6789/0 conn(0x7f0a0014dbe0 legacy=0x7f0a000c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a00153dd0 con 0x7f0a000aa4a0 2024-09-19T07:27:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f09f8001a00 con 0x7f0a000aa4a0 2024-09-19T07:27:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 >> v1:172.21.15.83:6789/0 conn(0x7f0a000aa4a0 legacy=0x7f0a00157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09ec0039f0 0x7f09ec003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09ec004370 0x7f09ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09ec005050 0x7f09ec00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0a00154700 con 0x7f09ec005050 2024-09-19T07:27:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f09ec00bf70 con 0x7f09ec004370 2024-09-19T07:27:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f09ec00c170 con 0x7f09ec0039f0 2024-09-19T07:27:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a05bb1640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09ec005050 0x7f09ec00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a053b0640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09ec0039f0 0x7f09ec003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a04baf640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09ec004370 0x7f09ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a05bb1640 1 -- 172.21.15.83:0/4036537394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09ec0039f0 msgr2=0x7f09ec003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a05bb1640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09ec0039f0 0x7f09ec003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a05bb1640 1 -- 172.21.15.83:0/4036537394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09ec004370 msgr2=0x7f09ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a05bb1640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09ec004370 0x7f09ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a05bb1640 1 -- 172.21.15.83:0/4036537394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ec003530 con 0x7f09ec005050 2024-09-19T07:27:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.567+0000 7f0a053b0640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09ec0039f0 0x7f09ec003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:27:36.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f0a05bb1640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09ec005050 0x7f09ec00b9d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f09f800a830 tx=0x7f09f801fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe8a6aede8e8986 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:36.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f800a480 con 0x7f09ec005050 2024-09-19T07:27:36.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09f80aa6f0 con 0x7f09ec005050 2024-09-19T07:27:36.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f09f7fff640 1 -- 172.21.15.83:0/4036537394 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f80aab60 con 0x7f09ec005050 2024-09-19T07:27:36.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f0a07614640 1 -- 172.21.15.83:0/4036537394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09ec005050 msgr2=0x7f09ec00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:36.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f0a07614640 1 --2- 172.21.15.83:0/4036537394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09ec005050 0x7f09ec00b9d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f09f800a830 tx=0x7f09f801fac0 comp rx=0 tx=0).stop 2024-09-19T07:27:36.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f0a05bb1640 1 -- 172.21.15.83:0/4036537394 reap_dead start 2024-09-19T07:27:36.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f0a07614640 1 -- 172.21.15.83:0/4036537394 shutdown_connections 2024-09-19T07:27:36.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f0a07614640 1 -- 172.21.15.83:0/4036537394 >> 172.21.15.83:0/4036537394 conn(0x7f0a000c95a0 msgr2=0x7f0a000c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:36.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f0a07614640 1 -- 172.21.15.83:0/4036537394 shutdown_connections 2024-09-19T07:27:36.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f0a07614640 1 -- 172.21.15.83:0/4036537394 wait complete. 2024-09-19T07:27:36.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f0a07614640 1 Processor -- start 2024-09-19T07:27:36.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.571+0000 7f0a07614640 1 -- start start 2024-09-19T07:27:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a07614640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a001e87e0 0x7f0a001ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a07614640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a001e8bc0 0x7f0a00149130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a07614640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a0006d9d0 0x7f0a00149670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a07614640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0a00157c60 con 0x7f0a001e8bc0 2024-09-19T07:27:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a07614640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0a00154390 con 0x7f0a001e87e0 2024-09-19T07:27:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a07614640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0a000741a0 con 0x7f0a0006d9d0 2024-09-19T07:27:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a04baf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a001e8bc0 0x7f0a00149130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a04baf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a001e8bc0 0x7f0a00149130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36528/0 (socket says 172.21.15.83:36528) 2024-09-19T07:27:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a04baf640 1 -- 172.21.15.83:0/3365526225 learned_addr learned my addr 172.21.15.83:0/3365526225 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a053b0640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a001e87e0 0x7f0a001ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a05bb1640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a0006d9d0 0x7f0a00149670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a04baf640 1 -- 172.21.15.83:0/3365526225 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a0006d9d0 msgr2=0x7f0a00149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a04baf640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a0006d9d0 0x7f0a00149670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a04baf640 1 -- 172.21.15.83:0/3365526225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a001e87e0 msgr2=0x7f0a001ee9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a04baf640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a001e87e0 0x7f0a001ee9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a04baf640 1 -- 172.21.15.83:0/3365526225 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a001eb5f0 con 0x7f0a001e8bc0 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a053b0640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a001e87e0 0x7f0a001ee9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a04baf640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a001e8bc0 0x7f0a00149130 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f09f0011560 tx=0x7f09f0011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f09f5ffb640 1 -- 172.21.15.83:0/3365526225 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f0015870 con 0x7f0a001e8bc0 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f09f5ffb640 1 -- 172.21.15.83:0/3365526225 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09f0015a10 con 0x7f0a001e8bc0 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a07614640 1 -- 172.21.15.83:0/3365526225 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a0006cea0 con 0x7f0a001e8bc0 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f09f5ffb640 1 -- 172.21.15.83:0/3365526225 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f0015dc0 con 0x7f0a001e8bc0 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a07614640 1 -- 172.21.15.83:0/3365526225 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a0006bd10 con 0x7f0a001e8bc0 2024-09-19T07:27:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a07614640 1 -- 172.21.15.83:0/3365526225 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a001eb3e0 con 0x7f0a001e8bc0 2024-09-19T07:27:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f09f5ffb640 1 -- 172.21.15.83:0/3365526225 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f09f0013400 con 0x7f0a001e8bc0 2024-09-19T07:27:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f09f5ffb640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09e40678f0 0x7f09e4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f0a053b0640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09e40678f0 0x7f09e4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.575+0000 7f09f5ffb640 1 -- 172.21.15.83:0/3365526225 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7583+0+0 (secure 0 0 0) 0x7f09f0086ba0 con 0x7f0a001e8bc0 2024-09-19T07:27:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.579+0000 7f0a053b0640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09e40678f0 0x7f09e4069db0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0a001e9220 tx=0x7f09fc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:36.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.583+0000 7f09f5ffb640 1 -- 172.21.15.83:0/3365526225 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f09f0019030 con 0x7f0a001e8bc0 2024-09-19T07:27:36.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.811+0000 7f0a07614640 1 -- 172.21.15.83:0/3365526225 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a000c3ab0 con 0x7f0a001e8bc0 2024-09-19T07:27:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.811+0000 7f09f5ffb640 1 -- 172.21.15.83:0/3365526225 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6529 (secure 0 0 0) 0x7f09f0057020 con 0x7f0a001e8bc0 2024-09-19T07:27:36.808 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:36.808 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":68,"btime":"2024-09-19T07:27:29:234190+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5934,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:25:39.224528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5757},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5757":{"gid":5757,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/455754420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":455754420},{"type":"v1","addr":"172.21.15.83:6837","nonce":455754420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5757,"qdb_cluster":[5757]},"id":2}]} 2024-09-19T07:27:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 68 2024-09-19T07:27:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.815+0000 7f0a07614640 1 -- 172.21.15.83:0/3365526225 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09e40678f0 msgr2=0x7f09e4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.815+0000 7f0a07614640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09e40678f0 0x7f09e4069db0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0a001e9220 tx=0x7f09fc009040 comp rx=0 tx=0).stop 2024-09-19T07:27:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.815+0000 7f0a07614640 1 -- 172.21.15.83:0/3365526225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a001e8bc0 msgr2=0x7f0a00149130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.815+0000 7f0a07614640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a001e8bc0 0x7f0a00149130 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f09f0011560 tx=0x7f09f0011a20 comp rx=0 tx=0).stop 2024-09-19T07:27:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.815+0000 7f0a07614640 1 -- 172.21.15.83:0/3365526225 shutdown_connections 2024-09-19T07:27:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.815+0000 7f0a07614640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09e40678f0 0x7f09e4069db0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.815+0000 7f0a07614640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a0006d9d0 0x7f0a00149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.815+0000 7f0a07614640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a001e8bc0 0x7f0a00149130 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.815+0000 7f0a07614640 1 --2- 172.21.15.83:0/3365526225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a001e87e0 0x7f0a001ee9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.815+0000 7f0a07614640 1 -- 172.21.15.83:0/3365526225 >> 172.21.15.83:0/3365526225 conn(0x7f0a000c95a0 msgr2=0x7f0a00153220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.815+0000 7f0a07614640 1 -- 172.21.15.83:0/3365526225 shutdown_connections 2024-09-19T07:27:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:36.815+0000 7f0a07614640 1 -- 172.21.15.83:0/3365526225 wait complete. 2024-09-19T07:27:36.829 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:27:38.830 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3c6a5640 1 Processor -- start 2024-09-19T07:27:38.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3c6a5640 1 -- start start 2024-09-19T07:27:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3c6a5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf340c6360 0x7fcf34153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3c6a5640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcf340bee30 0x7fcf341570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3c6a5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf340c5b90 con 0x7fcf340c9680 2024-09-19T07:27:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3c6a5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fcf3415a050 con 0x7fcf340c6360 2024-09-19T07:27:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3c6a5640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fcf3406c950 con 0x7fcf340bee30 2024-09-19T07:27:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3a441640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf340c6360 0x7fcf34153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3ac42640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcf340c9680 0x7fcf34157630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59690/0 (socket says 172.21.15.83:59690) 2024-09-19T07:27:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3ac42640 1 -- 172.21.15.83:0/1594130250 learned_addr learned my addr 172.21.15.83:0/1594130250 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:38.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf39c40640 1 --2- 172.21.15.83:0/1594130250 >> v2:172.21.15.184:3300/0 conn(0x7fcf340bee30 0x7fcf341570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:38.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3a441640 1 -- 172.21.15.83:0/1594130250 >> v2:172.21.15.184:3300/0 conn(0x7fcf340bee30 msgr2=0x7fcf341570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:38.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3a441640 1 --2- 172.21.15.83:0/1594130250 >> v2:172.21.15.184:3300/0 conn(0x7fcf340bee30 0x7fcf341570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:38.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3a441640 1 -- 172.21.15.83:0/1594130250 >> v1:172.21.15.83:6789/0 conn(0x7fcf340c9680 legacy=0x7fcf34157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:38.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3a441640 1 -- 172.21.15.83:0/1594130250 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf3406c670 con 0x7fcf340c6360 2024-09-19T07:27:38.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.967+0000 7fcf3943f640 1 -- 172.21.15.83:0/1594130250 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4152095099 0 0) 0x7fcf340c5b90 con 0x7fcf340c9680 2024-09-19T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3a441640 1 --2- 172.21.15.83:0/1594130250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf340c6360 0x7fcf34153050 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcf340c2650 tx=0x7fcf3001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b5d6bb5571fb2bf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3943f640 1 -- 172.21.15.83:0/1594130250 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf300b0070 con 0x7fcf340c6360 2024-09-19T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3943f640 1 -- 172.21.15.83:0/1594130250 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf300ab600 con 0x7fcf340c6360 2024-09-19T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3943f640 1 -- 172.21.15.83:0/1594130250 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf300ab9d0 con 0x7fcf340c6360 2024-09-19T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/1594130250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf340c6360 msgr2=0x7fcf34153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3c6a5640 1 --2- 172.21.15.83:0/1594130250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf340c6360 0x7fcf34153050 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcf340c2650 tx=0x7fcf3001c360 comp rx=0 tx=0).stop 2024-09-19T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/1594130250 shutdown_connections 2024-09-19T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3c6a5640 1 --2- 172.21.15.83:0/1594130250 >> v2:172.21.15.184:3300/0 conn(0x7fcf340bee30 0x7fcf341570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3c6a5640 1 --2- 172.21.15.83:0/1594130250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf340c6360 0x7fcf34153050 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/1594130250 >> 172.21.15.83:0/1594130250 conn(0x7fcf340722a0 msgr2=0x7fcf340726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:38.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/1594130250 shutdown_connections 2024-09-19T07:27:38.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/1594130250 wait complete. 2024-09-19T07:27:38.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3c6a5640 1 Processor -- start 2024-09-19T07:27:38.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.971+0000 7fcf3c6a5640 1 -- start start 2024-09-19T07:27:38.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf3c6a5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf340bee30 0x7fcf3414a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf3c6a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf340c6360 0x7fcf340ab690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf3c6a5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf340c9680 0x7fcf340b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf3c6a5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcf340c68c0 con 0x7fcf340c6360 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf3c6a5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcf340c6740 con 0x7fcf340bee30 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf3c6a5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcf34162da0 con 0x7fcf340c9680 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf39c40640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf340c6360 0x7fcf340ab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf39c40640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf340c6360 0x7fcf340ab690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36544/0 (socket says 172.21.15.83:36544) 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf39c40640 1 -- 172.21.15.83:0/3274665878 learned_addr learned my addr 172.21.15.83:0/3274665878 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf3a441640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf340bee30 0x7fcf3414a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf3ac42640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf340c9680 0x7fcf340b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf39c40640 1 -- 172.21.15.83:0/3274665878 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf340c9680 msgr2=0x7fcf340b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf39c40640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf340c9680 0x7fcf340b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:38.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf39c40640 1 -- 172.21.15.83:0/3274665878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf340bee30 msgr2=0x7fcf3414a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:38.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf39c40640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf340bee30 0x7fcf3414a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:38.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf39c40640 1 -- 172.21.15.83:0/3274665878 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf34156610 con 0x7fcf340c6360 2024-09-19T07:27:38.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf3a441640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf340bee30 0x7fcf3414a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:38.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf39c40640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf340c6360 0x7fcf340ab690 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fcf2400d7d0 tx=0x7fcf2400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:38.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf2affd640 1 -- 172.21.15.83:0/3274665878 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2400b6c0 con 0x7fcf340c6360 2024-09-19T07:27:38.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf2affd640 1 -- 172.21.15.83:0/3274665878 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf24019070 con 0x7fcf340c6360 2024-09-19T07:27:38.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf2affd640 1 -- 172.21.15.83:0/3274665878 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2400ba80 con 0x7fcf340c6360 2024-09-19T07:27:38.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/3274665878 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf341ee5f0 con 0x7fcf340c6360 2024-09-19T07:27:38.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/3274665878 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf341eebb0 con 0x7fcf340c6360 2024-09-19T07:27:38.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/3274665878 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf34153050 con 0x7fcf340c6360 2024-09-19T07:27:38.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf2affd640 1 -- 172.21.15.83:0/3274665878 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcf2400bc20 con 0x7fcf340c6360 2024-09-19T07:27:38.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.975+0000 7fcf2affd640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf180678c0 0x7fcf18069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:38.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.979+0000 7fcf2affd640 1 -- 172.21.15.83:0/3274665878 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7fcf24086750 con 0x7fcf340c6360 2024-09-19T07:27:38.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.979+0000 7fcf3a441640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf180678c0 0x7fcf18069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:38.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.979+0000 7fcf3a441640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf180678c0 0x7fcf18069d80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcf300ad000 tx=0x7fcf30008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:38.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:38.983+0000 7fcf2affd640 1 -- 172.21.15.83:0/3274665878 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcf24056b70 con 0x7fcf340c6360 2024-09-19T07:27:39.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.195+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/3274665878 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf34152440 con 0x7fcf340c6360 2024-09-19T07:27:39.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf2affd640 1 -- 172.21.15.83:0/3274665878 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5700 (secure 0 0 0) 0x7fcf24056470 con 0x7fcf340c6360 2024-09-19T07:27:39.194 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:39.194 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":69,"btime":"2024-09-19T07:27:38:023594+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:38.023583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:27:39.194 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 69 2024-09-19T07:27:39.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/3274665878 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf180678c0 msgr2=0x7fcf18069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:39.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf3c6a5640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf180678c0 0x7fcf18069d80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcf300ad000 tx=0x7fcf30008040 comp rx=0 tx=0).stop 2024-09-19T07:27:39.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/3274665878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf340c6360 msgr2=0x7fcf340ab690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:39.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf3c6a5640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf340c6360 0x7fcf340ab690 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fcf2400d7d0 tx=0x7fcf2400dc90 comp rx=0 tx=0).stop 2024-09-19T07:27:39.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/3274665878 shutdown_connections 2024-09-19T07:27:39.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf3c6a5640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf180678c0 0x7fcf18069d80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf3c6a5640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf340c9680 0x7fcf340b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf3c6a5640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf340c6360 0x7fcf340ab690 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf3c6a5640 1 --2- 172.21.15.83:0/3274665878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf340bee30 0x7fcf3414a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/3274665878 >> 172.21.15.83:0/3274665878 conn(0x7fcf340722a0 msgr2=0x7fcf34157d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:39.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/3274665878 shutdown_connections 2024-09-19T07:27:39.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.199+0000 7fcf3c6a5640 1 -- 172.21.15.83:0/3274665878 wait complete. 2024-09-19T07:27:39.214 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-19T07:27:39.214 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:27:39.214 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f716a15d640 1 Processor -- start 2024-09-19T07:27:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f716a15d640 1 -- start start 2024-09-19T07:27:39.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f716a15d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f716406f2f0 0x7f71641570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:39.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f716a15d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71640c57d0 con 0x7f716406e460 2024-09-19T07:27:39.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f716a15d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71640c5ab0 con 0x7f716414dcc0 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f716a15d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7164154b20 con 0x7f716406f2f0 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f7163fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f716406e460 0x7f71640c60a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59692/0 (socket says 172.21.15.83:59692) 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f7163fff640 1 -- 172.21.15.83:0/930823685 learned_addr learned my addr 172.21.15.83:0/930823685 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2663439806 0 0) 0x7f71640c57d0 con 0x7f716406e460 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f714c003530 con 0x7f716406e460 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f7162ffd640 1 --2- 172.21.15.83:0/930823685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f716406f2f0 0x7f71641570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 529703747 0 0) 0x7f714c003530 con 0x7f716406e460 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f716406f2f0 msgr2=0x7f71641570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 --2- 172.21.15.83:0/930823685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f716406f2f0 0x7f71641570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 >> v1:172.21.15.184:6789/0 conn(0x7f716414dcc0 legacy=0x7f7164154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71640c5d90 con 0x7f716406e460 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f715c001a00 con 0x7f716406e460 2024-09-19T07:27:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 >> v1:172.21.15.83:6789/0 conn(0x7f716406e460 legacy=0x7f71640c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:39.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 --2- 172.21.15.83:0/930823685 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f714c0039f0 0x7f714c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:39.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 --2- 172.21.15.83:0/930823685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f714c004810 0x7f714c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:39.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 --2- 172.21.15.83:0/930823685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f716406f2f0 0x7f714c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:39.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7164154b20 con 0x7f714c004810 2024-09-19T07:27:39.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f714c00bb00 con 0x7f714c0039f0 2024-09-19T07:27:39.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f714c00bd00 con 0x7f716406f2f0 2024-09-19T07:27:39.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f7162ffd640 1 --2- 172.21.15.83:0/930823685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f714c004810 0x7f714c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:39.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f71637fe640 1 --2- 172.21.15.83:0/930823685 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f714c0039f0 0x7f714c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:39.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f7163fff640 1 --2- 172.21.15.83:0/930823685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f716406f2f0 0x7f714c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:39.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f7162ffd640 1 -- 172.21.15.83:0/930823685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f716406f2f0 msgr2=0x7f714c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:39.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f7162ffd640 1 --2- 172.21.15.83:0/930823685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f716406f2f0 0x7f714c005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f7162ffd640 1 -- 172.21.15.83:0/930823685 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f714c0039f0 msgr2=0x7f714c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:39.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f7162ffd640 1 --2- 172.21.15.83:0/930823685 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f714c0039f0 0x7f714c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.331+0000 7f7162ffd640 1 -- 172.21.15.83:0/930823685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714c003530 con 0x7f714c004810 2024-09-19T07:27:39.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f7162ffd640 1 --2- 172.21.15.83:0/930823685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f714c004810 0x7f714c004c90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f71640c5ff0 tx=0x7f715801c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e928bb393da4184 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:39.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715800b610 con 0x7f714c004810 2024-09-19T07:27:39.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71580b0070 con 0x7f714c004810 2024-09-19T07:27:39.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f71627fc640 1 -- 172.21.15.83:0/930823685 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71580ab670 con 0x7f714c004810 2024-09-19T07:27:39.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f716a15d640 1 -- 172.21.15.83:0/930823685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f714c004810 msgr2=0x7f714c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:39.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f716a15d640 1 --2- 172.21.15.83:0/930823685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f714c004810 0x7f714c004c90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f71640c5ff0 tx=0x7f715801c620 comp rx=0 tx=0).stop 2024-09-19T07:27:39.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f7163fff640 1 -- 172.21.15.83:0/930823685 reap_dead start 2024-09-19T07:27:39.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f716a15d640 1 -- 172.21.15.83:0/930823685 shutdown_connections 2024-09-19T07:27:39.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f716a15d640 1 -- 172.21.15.83:0/930823685 >> 172.21.15.83:0/930823685 conn(0x7f716406ffc0 msgr2=0x7f71640703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:39.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f716a15d640 1 -- 172.21.15.83:0/930823685 shutdown_connections 2024-09-19T07:27:39.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f716a15d640 1 -- 172.21.15.83:0/930823685 wait complete. 2024-09-19T07:27:39.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f716a15d640 1 Processor -- start 2024-09-19T07:27:39.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.335+0000 7f716a15d640 1 -- start start 2024-09-19T07:27:39.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f716a15d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f714c004810 0x7f71641ed790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:39.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f716a15d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7164149d20 0x7f71640c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:39.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f716a15d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71641e9ec0 0x7f71641ea340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:39.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f716a15d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f71640c67b0 con 0x7f714c004810 2024-09-19T07:27:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f716a15d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7164153f80 con 0x7f71641e9ec0 2024-09-19T07:27:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f716a15d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f71640c6930 con 0x7f7164149d20 2024-09-19T07:27:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f71637fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f714c004810 0x7f71641ed790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f71637fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f714c004810 0x7f71641ed790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36566/0 (socket says 172.21.15.83:36566) 2024-09-19T07:27:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f71637fe640 1 -- 172.21.15.83:0/1263416030 learned_addr learned my addr 172.21.15.83:0/1263416030 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f7162ffd640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7164149d20 0x7f71640c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f7163fff640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71641e9ec0 0x7f71641ea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f71637fe640 1 -- 172.21.15.83:0/1263416030 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7164149d20 msgr2=0x7f71640c9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f71637fe640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7164149d20 0x7f71640c9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f71637fe640 1 -- 172.21.15.83:0/1263416030 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71641e9ec0 msgr2=0x7f71641ea340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f71637fe640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71641e9ec0 0x7f71641ea340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f71637fe640 1 -- 172.21.15.83:0/1263416030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716414ea40 con 0x7f714c004810 2024-09-19T07:27:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f71637fe640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f714c004810 0x7f71641ed790 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f715400e4d0 tx=0x7f715400e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f714bfff640 1 -- 172.21.15.83:0/1263416030 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71540053a0 con 0x7f714c004810 2024-09-19T07:27:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f716a15d640 1 -- 172.21.15.83:0/1263416030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716414e480 con 0x7f714c004810 2024-09-19T07:27:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f714bfff640 1 -- 172.21.15.83:0/1263416030 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7154018070 con 0x7f714c004810 2024-09-19T07:27:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f716a15d640 1 -- 172.21.15.83:0/1263416030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71641df800 con 0x7f714c004810 2024-09-19T07:27:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f714bfff640 1 -- 172.21.15.83:0/1263416030 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7154003820 con 0x7f714c004810 2024-09-19T07:27:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f716a15d640 1 -- 172.21.15.83:0/1263416030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71440052a0 con 0x7f714c004810 2024-09-19T07:27:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.339+0000 7f714bfff640 1 -- 172.21.15.83:0/1263416030 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f71540039c0 con 0x7f714c004810 2024-09-19T07:27:39.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.343+0000 7f714bfff640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71500678f0 0x7f7150069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:39.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.343+0000 7f714bfff640 1 -- 172.21.15.83:0/1263416030 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7f7154086140 con 0x7f714c004810 2024-09-19T07:27:39.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.343+0000 7f7162ffd640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71500678f0 0x7f7150069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:39.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.343+0000 7f7162ffd640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71500678f0 0x7f7150069db0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f71580af000 tx=0x7f71580046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:39.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.347+0000 7f714bfff640 1 -- 172.21.15.83:0/1263416030 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f71540564c0 con 0x7f714c004810 2024-09-19T07:27:39.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.623+0000 7f716a15d640 1 -- 172.21.15.83:0/1263416030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7144005590 con 0x7f714c004810 2024-09-19T07:27:39.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.623+0000 7f714bfff640 1 -- 172.21.15.83:0/1263416030 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5703 (secure 0 0 0) 0x7f7154055dc0 con 0x7f714c004810 2024-09-19T07:27:39.619 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:39.619 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":70,"btime":"2024-09-19T07:27:39:274032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:38.403038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:27:39.619 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 70 2024-09-19T07:27:39.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.627+0000 7f716a15d640 1 -- 172.21.15.83:0/1263416030 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71500678f0 msgr2=0x7f7150069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:39.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.627+0000 7f716a15d640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71500678f0 0x7f7150069db0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f71580af000 tx=0x7f71580046d0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.627+0000 7f716a15d640 1 -- 172.21.15.83:0/1263416030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f714c004810 msgr2=0x7f71641ed790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:39.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.627+0000 7f716a15d640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f714c004810 0x7f71641ed790 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f715400e4d0 tx=0x7f715400e990 comp rx=0 tx=0).stop 2024-09-19T07:27:39.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.627+0000 7f716a15d640 1 -- 172.21.15.83:0/1263416030 shutdown_connections 2024-09-19T07:27:39.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.627+0000 7f716a15d640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71500678f0 0x7f7150069db0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.627+0000 7f716a15d640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71641e9ec0 0x7f71641ea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.627+0000 7f716a15d640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7164149d20 0x7f71640c9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.627+0000 7f716a15d640 1 --2- 172.21.15.83:0/1263416030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f714c004810 0x7f71641ed790 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:39.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.627+0000 7f716a15d640 1 -- 172.21.15.83:0/1263416030 >> 172.21.15.83:0/1263416030 conn(0x7f716406ffc0 msgr2=0x7f71640c60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:39.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.627+0000 7f716a15d640 1 -- 172.21.15.83:0/1263416030 shutdown_connections 2024-09-19T07:27:39.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:39.627+0000 7f716a15d640 1 -- 172.21.15.83:0/1263416030 wait complete. 2024-09-19T07:27:39.640 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:27:39:274032+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 70, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 67, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4979": { "addr": "172.21.15.184:6835/872719174", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 872719174, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 872719174, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4979, "incarnation": 64, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:27:21.202626+0000", "qdb_cluster": [ 4979 ], "qdb_leader": 4979, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4979 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 70, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5934": { "addr": "172.21.15.83:6835/104517500", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 104517500, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 104517500, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5934, "incarnation": 69, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:27:38.403038+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5934 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 28, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:27:41.642 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:41.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.771+0000 7effa2ec4640 1 Processor -- start 2024-09-19T07:27:41.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.771+0000 7effa2ec4640 1 -- start start 2024-09-19T07:27:41.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.771+0000 7effa2ec4640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7eff9c14fe10 0x7eff9c0c3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:41.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.771+0000 7effa2ec4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff9c06d550 0x7eff9c1605a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:41.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.771+0000 7effa2ec4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff9c071b80 con 0x7eff9c0716a0 2024-09-19T07:27:41.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.771+0000 7effa2ec4640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7eff9c06ecc0 con 0x7eff9c14fe10 2024-09-19T07:27:41.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.771+0000 7effa2ec4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7eff9c06eef0 con 0x7eff9c06d550 2024-09-19T07:27:41.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.775+0000 7effa1461640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff9c06d550 0x7eff9c1605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:41.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.775+0000 7eff9bfff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7eff9c0716a0 0x7eff9c155c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:57318/0 (socket says 172.21.15.83:57318) 2024-09-19T07:27:41.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.775+0000 7effa0c60640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7eff9c14fe10 0x7eff9c0c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:41.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.775+0000 7eff9bfff640 1 -- 172.21.15.83:0/215046338 learned_addr learned my addr 172.21.15.83:0/215046338 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:41.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.775+0000 7effa1461640 1 -- 172.21.15.83:0/215046338 >> v2:172.21.15.184:3300/0 conn(0x7eff9c14fe10 msgr2=0x7eff9c0c3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:41.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.775+0000 7effa1461640 1 --2- 172.21.15.83:0/215046338 >> v2:172.21.15.184:3300/0 conn(0x7eff9c14fe10 0x7eff9c0c3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:41.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.775+0000 7effa1461640 1 -- 172.21.15.83:0/215046338 >> v1:172.21.15.184:6789/0 conn(0x7eff9c0716a0 legacy=0x7eff9c155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:41.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.775+0000 7effa1461640 1 -- 172.21.15.83:0/215046338 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9c06f1d0 con 0x7eff9c06d550 2024-09-19T07:27:41.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.775+0000 7effa1461640 1 --2- 172.21.15.83:0/215046338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff9c06d550 0x7eff9c1605a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7eff94009d90 tx=0x7eff9401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdd25734016fa0ee server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:41.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.775+0000 7eff9b7fe640 1 -- 172.21.15.83:0/215046338 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff940b2070 con 0x7eff9c06d550 2024-09-19T07:27:41.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.775+0000 7eff9b7fe640 1 -- 172.21.15.83:0/215046338 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff940a5a20 con 0x7eff9c06d550 2024-09-19T07:27:41.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.775+0000 7eff9b7fe640 1 -- 172.21.15.83:0/215046338 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff940a5df0 con 0x7eff9c06d550 2024-09-19T07:27:41.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 -- 172.21.15.83:0/215046338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff9c06d550 msgr2=0x7eff9c1605a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:41.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 --2- 172.21.15.83:0/215046338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff9c06d550 0x7eff9c1605a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7eff94009d90 tx=0x7eff9401c940 comp rx=0 tx=0).stop 2024-09-19T07:27:41.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 -- 172.21.15.83:0/215046338 shutdown_connections 2024-09-19T07:27:41.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 --2- 172.21.15.83:0/215046338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff9c06d550 0x7eff9c1605a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:41.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 --2- 172.21.15.83:0/215046338 >> v2:172.21.15.184:3300/0 conn(0x7eff9c14fe10 0x7eff9c0c3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:41.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 -- 172.21.15.83:0/215046338 >> 172.21.15.83:0/215046338 conn(0x7eff9c0c4940 msgr2=0x7eff9c0c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:41.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 -- 172.21.15.83:0/215046338 shutdown_connections 2024-09-19T07:27:41.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 -- 172.21.15.83:0/215046338 wait complete. 2024-09-19T07:27:41.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 Processor -- start 2024-09-19T07:27:41.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 -- start start 2024-09-19T07:27:41.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff9c06d550 0x7eff9c0ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:41.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff9c0716a0 0x7eff9c1f1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:41.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff9c14fe10 0x7eff9c1f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:41.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c163300 con 0x7eff9c14fe10 2024-09-19T07:27:41.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c068870 con 0x7eff9c0716a0 2024-09-19T07:27:41.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa2ec4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7eff9c163480 con 0x7eff9c06d550 2024-09-19T07:27:41.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa1461640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff9c14fe10 0x7eff9c1f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:41.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa1461640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff9c14fe10 0x7eff9c1f23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47302/0 (socket says 172.21.15.83:47302) 2024-09-19T07:27:41.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa1461640 1 -- 172.21.15.83:0/2147787253 learned_addr learned my addr 172.21.15.83:0/2147787253 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:41.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa0c60640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff9c06d550 0x7eff9c0ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:41.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7eff9bfff640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff9c0716a0 0x7eff9c1f1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:41.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa1461640 1 -- 172.21.15.83:0/2147787253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff9c06d550 msgr2=0x7eff9c0ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:41.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa1461640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff9c06d550 0x7eff9c0ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:41.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa1461640 1 -- 172.21.15.83:0/2147787253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff9c0716a0 msgr2=0x7eff9c1f1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:41.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa1461640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff9c0716a0 0x7eff9c1f1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:41.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7effa1461640 1 -- 172.21.15.83:0/2147787253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9c150580 con 0x7eff9c14fe10 2024-09-19T07:27:41.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.779+0000 7eff9bfff640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff9c0716a0 0x7eff9c1f1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:27:41.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.783+0000 7effa1461640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff9c14fe10 0x7eff9c1f23f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7eff940b0040 tx=0x7eff940196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:41.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.783+0000 7eff997fa640 1 -- 172.21.15.83:0/2147787253 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff940b2050 con 0x7eff9c14fe10 2024-09-19T07:27:41.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.783+0000 7eff997fa640 1 -- 172.21.15.83:0/2147787253 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff940be070 con 0x7eff9c14fe10 2024-09-19T07:27:41.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.783+0000 7eff997fa640 1 -- 172.21.15.83:0/2147787253 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff940adb10 con 0x7eff9c14fe10 2024-09-19T07:27:41.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.783+0000 7effa2ec4640 1 -- 172.21.15.83:0/2147787253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9c153440 con 0x7eff9c14fe10 2024-09-19T07:27:41.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.783+0000 7effa2ec4640 1 -- 172.21.15.83:0/2147787253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff9c14d020 con 0x7eff9c14fe10 2024-09-19T07:27:41.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.783+0000 7effa2ec4640 1 -- 172.21.15.83:0/2147787253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff9c153230 con 0x7eff9c14fe10 2024-09-19T07:27:41.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.783+0000 7eff997fa640 1 -- 172.21.15.83:0/2147787253 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7eff940adcb0 con 0x7eff9c14fe10 2024-09-19T07:27:41.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.783+0000 7eff997fa640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff800678c0 0x7eff80069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:41.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.783+0000 7eff997fa640 1 -- 172.21.15.83:0/2147787253 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7eff94121780 con 0x7eff9c14fe10 2024-09-19T07:27:41.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.783+0000 7effa0c60640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff800678c0 0x7eff80069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:41.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.783+0000 7effa0c60640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff800678c0 0x7eff80069d80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7eff9c1ec780 tx=0x7eff90009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:41.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:41.787+0000 7eff997fa640 1 -- 172.21.15.83:0/2147787253 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7eff94016560 con 0x7eff9c14fe10 2024-09-19T07:27:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 -- 172.21.15.83:0/2147787253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff9c0c3850 con 0x7eff9c14fe10 2024-09-19T07:27:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7eff997fa640 1 -- 172.21.15.83:0/2147787253 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5706 (secure 0 0 0) 0x7eff940f1b20 con 0x7eff9c14fe10 2024-09-19T07:27:42.007 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:42.007 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":72,"btime":"2024-09-19T07:27:41:289676+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:40.293654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:clientreplay","state_seq":6,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:27:42.007 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 72 2024-09-19T07:27:42.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 -- 172.21.15.83:0/2147787253 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff800678c0 msgr2=0x7eff80069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:42.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff800678c0 0x7eff80069d80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7eff9c1ec780 tx=0x7eff90009190 comp rx=0 tx=0).stop 2024-09-19T07:27:42.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 -- 172.21.15.83:0/2147787253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff9c14fe10 msgr2=0x7eff9c1f23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:42.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff9c14fe10 0x7eff9c1f23f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7eff940b0040 tx=0x7eff940196e0 comp rx=0 tx=0).stop 2024-09-19T07:27:42.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 -- 172.21.15.83:0/2147787253 shutdown_connections 2024-09-19T07:27:42.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff800678c0 0x7eff80069d80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:42.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff9c14fe10 0x7eff9c1f23f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:42.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff9c0716a0 0x7eff9c1f1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:42.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 --2- 172.21.15.83:0/2147787253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff9c06d550 0x7eff9c0ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:42.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 -- 172.21.15.83:0/2147787253 >> 172.21.15.83:0/2147787253 conn(0x7eff9c0c4940 msgr2=0x7eff9c155aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:42.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 -- 172.21.15.83:0/2147787253 shutdown_connections 2024-09-19T07:27:42.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:42.011+0000 7effa2ec4640 1 -- 172.21.15.83:0/2147787253 wait complete. 2024-09-19T07:27:42.028 INFO:tasks.mds_thrash.fs.[b]:no change 2024-09-19T07:27:44.029 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:44.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c8a632640 1 Processor -- start 2024-09-19T07:27:44.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c8a632640 1 -- start start 2024-09-19T07:27:44.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c8a632640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c840c4990 0x7f4c840728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c8a632640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c840c2f90 con 0x7f4c84150310 2024-09-19T07:27:44.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c8a632640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c840c3210 con 0x7f4c8406cef0 2024-09-19T07:27:44.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c8a632640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4c840c3e20 con 0x7f4c840c4990 2024-09-19T07:27:44.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c7bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4c84150310 0x7f4c84156070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51542/0 (socket says 172.21.15.83:51542) 2024-09-19T07:27:44.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c7bfff640 1 -- 172.21.15.83:0/4222902614 learned_addr learned my addr 172.21.15.83:0/4222902614 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:44.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c88bcf640 1 --2- 172.21.15.83:0/4222902614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c840c4990 0x7f4c840728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823348800 0 0) 0x7f4c6c001120 con 0x7f4c84150310 2024-09-19T07:27:44.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c70003530 con 0x7f4c84150310 2024-09-19T07:27:44.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 751497728 0 0) 0x7f4c70003530 con 0x7f4c84150310 2024-09-19T07:27:44.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c840c4990 msgr2=0x7f4c840728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:44.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 --2- 172.21.15.83:0/4222902614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c840c4990 0x7f4c840728d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 >> v1:172.21.15.184:6789/0 conn(0x7f4c8406cef0 legacy=0x7f4c8414fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c840c34f0 con 0x7f4c84150310 2024-09-19T07:27:44.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f4c6c001870 con 0x7f4c84150310 2024-09-19T07:27:44.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 >> v1:172.21.15.83:6789/0 conn(0x7f4c84150310 legacy=0x7f4c84156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 --2- 172.21.15.83:0/4222902614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c700039f0 0x7f4c70003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 --2- 172.21.15.83:0/4222902614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c70004810 0x7f4c70004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 --2- 172.21.15.83:0/4222902614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c840c4990 0x7f4c70005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4c840c3e20 con 0x7f4c70004810 2024-09-19T07:27:44.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7000bb00 con 0x7f4c700039f0 2024-09-19T07:27:44.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4c7000bd00 con 0x7f4c840c4990 2024-09-19T07:27:44.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.179+0000 7f4c7bfff640 1 --2- 172.21.15.83:0/4222902614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c70004810 0x7f4c70004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c88bcf640 1 --2- 172.21.15.83:0/4222902614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c840c4990 0x7f4c70005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c83fff640 1 --2- 172.21.15.83:0/4222902614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c700039f0 0x7f4c70003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c7bfff640 1 -- 172.21.15.83:0/4222902614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c840c4990 msgr2=0x7f4c70005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:44.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c7bfff640 1 --2- 172.21.15.83:0/4222902614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c840c4990 0x7f4c70005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c7bfff640 1 -- 172.21.15.83:0/4222902614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c700039f0 msgr2=0x7f4c70003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:44.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c7bfff640 1 --2- 172.21.15.83:0/4222902614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c700039f0 0x7f4c70003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c7bfff640 1 -- 172.21.15.83:0/4222902614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c70003530 con 0x7f4c70004810 2024-09-19T07:27:44.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c7bfff640 1 --2- 172.21.15.83:0/4222902614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c70004810 0x7f4c70004c90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00c490 tx=0x7f4c6c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d0651499f96064 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:44.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c00c0e0 con 0x7f4c70004810 2024-09-19T07:27:44.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c6c0a9070 con 0x7f4c70004810 2024-09-19T07:27:44.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c837fe640 1 -- 172.21.15.83:0/4222902614 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c005d20 con 0x7f4c70004810 2024-09-19T07:27:44.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c8a632640 1 -- 172.21.15.83:0/4222902614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c70004810 msgr2=0x7f4c70004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c8a632640 1 --2- 172.21.15.83:0/4222902614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c70004810 0x7f4c70004c90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00c490 tx=0x7f4c6c01cc60 comp rx=0 tx=0).stop 2024-09-19T07:27:44.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c88bcf640 1 -- 172.21.15.83:0/4222902614 reap_dead start 2024-09-19T07:27:44.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c8a632640 1 -- 172.21.15.83:0/4222902614 shutdown_connections 2024-09-19T07:27:44.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c8a632640 1 -- 172.21.15.83:0/4222902614 >> 172.21.15.83:0/4222902614 conn(0x7f4c840c5090 msgr2=0x7f4c840c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:44.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c8a632640 1 -- 172.21.15.83:0/4222902614 shutdown_connections 2024-09-19T07:27:44.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.183+0000 7f4c8a632640 1 -- 172.21.15.83:0/4222902614 wait complete. 2024-09-19T07:27:44.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c8a632640 1 Processor -- start 2024-09-19T07:27:44.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c8a632640 1 -- start start 2024-09-19T07:27:44.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c8a632640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c70004810 0x7f4c84145200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c8a632640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c840c7e80 0x7f4c84142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c8a632640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c840c7930 0x7f4c84143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c8a632640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4c84156d10 con 0x7f4c840c7e80 2024-09-19T07:27:44.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c8a632640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4c84156b90 con 0x7f4c840c7930 2024-09-19T07:27:44.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c8a632640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4c8406d2d0 con 0x7f4c70004810 2024-09-19T07:27:44.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c7bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c840c7e80 0x7f4c84142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c7bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c840c7e80 0x7f4c84142de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47330/0 (socket says 172.21.15.83:47330) 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c7bfff640 1 -- 172.21.15.83:0/4129839313 learned_addr learned my addr 172.21.15.83:0/4129839313 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c88bcf640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c840c7930 0x7f4c84143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c83fff640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c70004810 0x7f4c84145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c7bfff640 1 -- 172.21.15.83:0/4129839313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c70004810 msgr2=0x7f4c84145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c7bfff640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c70004810 0x7f4c84145200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c7bfff640 1 -- 172.21.15.83:0/4129839313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c840c7930 msgr2=0x7f4c84143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c7bfff640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c840c7930 0x7f4c84143320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c7bfff640 1 -- 172.21.15.83:0/4129839313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c841f2210 con 0x7f4c840c7e80 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c7bfff640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c840c7e80 0x7f4c84142de0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c01c6d0 tx=0x7f4c70005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c817fa640 1 -- 172.21.15.83:0/4129839313 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c001120 con 0x7f4c840c7e80 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c817fa640 1 -- 172.21.15.83:0/4129839313 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c6c0a5850 con 0x7f4c840c7e80 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c8a632640 1 -- 172.21.15.83:0/4129839313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c841f24f0 con 0x7f4c840c7e80 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c817fa640 1 -- 172.21.15.83:0/4129839313 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c0a5c00 con 0x7f4c840c7e80 2024-09-19T07:27:44.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c8a632640 1 -- 172.21.15.83:0/4129839313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c841f2ab0 con 0x7f4c840c7e80 2024-09-19T07:27:44.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.187+0000 7f4c8a632640 1 -- 172.21.15.83:0/4129839313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c8414fc00 con 0x7f4c840c7e80 2024-09-19T07:27:44.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.191+0000 7f4c817fa640 1 -- 172.21.15.83:0/4129839313 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4c6c0047f0 con 0x7f4c840c7e80 2024-09-19T07:27:44.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.191+0000 7f4c817fa640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c640678c0 0x7f4c64069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.191+0000 7f4c817fa640 1 -- 172.21.15.83:0/4129839313 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7f4c6c11fce0 con 0x7f4c840c7e80 2024-09-19T07:27:44.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.191+0000 7f4c83fff640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c640678c0 0x7f4c64069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.191+0000 7f4c83fff640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c640678c0 0x7f4c64069d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4c74005490 tx=0x7f4c74009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:44.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.195+0000 7f4c817fa640 1 -- 172.21.15.83:0/4129839313 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4c840c2f90 con 0x7f4c840c7e80 2024-09-19T07:27:44.246 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:27:44.251+0000 7f0aa3819640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:27:44.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.471+0000 7f4c8a632640 1 -- 172.21.15.83:0/4129839313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c84073d60 con 0x7f4c840c7e80 2024-09-19T07:27:44.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.471+0000 7f4c817fa640 1 -- 172.21.15.83:0/4129839313 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5707 (secure 0 0 0) 0x7f4c6c0effd0 con 0x7f4c840c7e80 2024-09-19T07:27:44.468 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:44.469 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":73,"btime":"2024-09-19T07:27:42:299328+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:27:44.469 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 73 2024-09-19T07:27:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.475+0000 7f4c8a632640 1 -- 172.21.15.83:0/4129839313 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c640678c0 msgr2=0x7f4c64069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.475+0000 7f4c8a632640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c640678c0 0x7f4c64069d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4c74005490 tx=0x7f4c74009040 comp rx=0 tx=0).stop 2024-09-19T07:27:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.475+0000 7f4c8a632640 1 -- 172.21.15.83:0/4129839313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c840c7e80 msgr2=0x7f4c84142de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.475+0000 7f4c8a632640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c840c7e80 0x7f4c84142de0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c01c6d0 tx=0x7f4c70005840 comp rx=0 tx=0).stop 2024-09-19T07:27:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.475+0000 7f4c8a632640 1 -- 172.21.15.83:0/4129839313 shutdown_connections 2024-09-19T07:27:44.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.475+0000 7f4c8a632640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4c640678c0 0x7f4c64069d80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.475+0000 7f4c8a632640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c840c7930 0x7f4c84143320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.475+0000 7f4c8a632640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c840c7e80 0x7f4c84142de0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.475+0000 7f4c8a632640 1 --2- 172.21.15.83:0/4129839313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c70004810 0x7f4c84145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.475+0000 7f4c8a632640 1 -- 172.21.15.83:0/4129839313 >> 172.21.15.83:0/4129839313 conn(0x7f4c840c5090 msgr2=0x7f4c84149400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:44.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.475+0000 7f4c8a632640 1 -- 172.21.15.83:0/4129839313 shutdown_connections 2024-09-19T07:27:44.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.475+0000 7f4c8a632640 1 -- 172.21.15.83:0/4129839313 wait complete. 2024-09-19T07:27:44.492 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5757 2024-09-19T07:27:44.492 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.c 2024-09-19T07:27:44.493 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-19T07:27:44.493 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:27:44.493 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:27:44.495 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:27:44.495 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:44.566 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:27:44.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f0790359640 1 Processor -- start 2024-09-19T07:27:44.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f0790359640 1 -- start start 2024-09-19T07:27:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f0790359640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f078806cef0 0x7f07880c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f0790359640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07881534f0 0x7f078814fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f0790359640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07880c6b70 con 0x7f078806ddd0 2024-09-19T07:27:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f0790359640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f078806f730 con 0x7f078806cef0 2024-09-19T07:27:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f0790359640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0788153980 con 0x7f07881534f0 2024-09-19T07:27:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f078e0f5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f078806cef0 0x7f07880c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f078e0f5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f078806cef0 0x7f07880c79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47338/0 (socket says 172.21.15.83:47338) 2024-09-19T07:27:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f078e0f5640 1 -- 172.21.15.83:0/2540753794 learned_addr learned my addr 172.21.15.83:0/2540753794 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f078e8f6640 1 --2- 172.21.15.83:0/2540753794 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07881534f0 0x7f078814fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f078e0f5640 1 -- 172.21.15.83:0/2540753794 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07881534f0 msgr2=0x7f078814fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:44.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f078e0f5640 1 --2- 172.21.15.83:0/2540753794 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07881534f0 0x7f078814fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f078e0f5640 1 -- 172.21.15.83:0/2540753794 >> v1:172.21.15.184:6789/0 conn(0x7f078806ddd0 legacy=0x7f07880c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.627+0000 7f078e0f5640 1 -- 172.21.15.83:0/2540753794 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078806e210 con 0x7f078806cef0 2024-09-19T07:27:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f078e0f5640 1 --2- 172.21.15.83:0/2540753794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f078806cef0 0x7f07880c79d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f078400bb10 tx=0x7f078401eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d519580691623074 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f078d0f3640 1 -- 172.21.15.83:0/2540753794 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07840b4070 con 0x7f078806cef0 2024-09-19T07:27:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f078d0f3640 1 -- 172.21.15.83:0/2540753794 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07840a78e0 con 0x7f078806cef0 2024-09-19T07:27:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f078d0f3640 1 -- 172.21.15.83:0/2540753794 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07840a7cb0 con 0x7f078806cef0 2024-09-19T07:27:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 -- 172.21.15.83:0/2540753794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f078806cef0 msgr2=0x7f07880c79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 --2- 172.21.15.83:0/2540753794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f078806cef0 0x7f07880c79d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f078400bb10 tx=0x7f078401eb10 comp rx=0 tx=0).stop 2024-09-19T07:27:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 -- 172.21.15.83:0/2540753794 shutdown_connections 2024-09-19T07:27:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 --2- 172.21.15.83:0/2540753794 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07881534f0 0x7f078814fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 --2- 172.21.15.83:0/2540753794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f078806cef0 0x7f07880c79d0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 -- 172.21.15.83:0/2540753794 >> 172.21.15.83:0/2540753794 conn(0x7f07880aa910 msgr2=0x7f07880c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 -- 172.21.15.83:0/2540753794 shutdown_connections 2024-09-19T07:27:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 -- 172.21.15.83:0/2540753794 wait complete. 2024-09-19T07:27:44.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 Processor -- start 2024-09-19T07:27:44.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 -- start start 2024-09-19T07:27:44.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f078806cef0 0x7f07880aca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f078806ddd0 0x7f07880acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07881534f0 0x7f07881ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f078806d2d0 con 0x7f078806ddd0 2024-09-19T07:27:44.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0788150420 con 0x7f078806cef0 2024-09-19T07:27:44.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.631+0000 7f0790359640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f078806d450 con 0x7f07881534f0 2024-09-19T07:27:44.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078d8f4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f078806ddd0 0x7f07880acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078d8f4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f078806ddd0 0x7f07880acf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47354/0 (socket says 172.21.15.83:47354) 2024-09-19T07:27:44.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078d8f4640 1 -- 172.21.15.83:0/3864489711 learned_addr learned my addr 172.21.15.83:0/3864489711 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:44.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078e0f5640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f078806cef0 0x7f07880aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078e8f6640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07881534f0 0x7f07881ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078d8f4640 1 -- 172.21.15.83:0/3864489711 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07881534f0 msgr2=0x7f07881ea6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078d8f4640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07881534f0 0x7f07881ea6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078d8f4640 1 -- 172.21.15.83:0/3864489711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f078806cef0 msgr2=0x7f07880aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078d8f4640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f078806cef0 0x7f07880aca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078d8f4640 1 -- 172.21.15.83:0/3864489711 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078814f910 con 0x7f078806ddd0 2024-09-19T07:27:44.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078e8f6640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07881534f0 0x7f07881ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:27:44.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078d8f4640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f078806ddd0 0x7f07880acf50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f077800c370 tx=0x7f077800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:44.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f077effd640 1 -- 172.21.15.83:0/3864489711 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07780052d0 con 0x7f078806ddd0 2024-09-19T07:27:44.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f077effd640 1 -- 172.21.15.83:0/3864489711 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0778005470 con 0x7f078806ddd0 2024-09-19T07:27:44.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f077effd640 1 -- 172.21.15.83:0/3864489711 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778007260 con 0x7f078806ddd0 2024-09-19T07:27:44.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f0790359640 1 -- 172.21.15.83:0/3864489711 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07880c4e20 con 0x7f078806ddd0 2024-09-19T07:27:44.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f0790359640 1 -- 172.21.15.83:0/3864489711 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07880ac320 con 0x7f078806ddd0 2024-09-19T07:27:44.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f0790359640 1 -- 172.21.15.83:0/3864489711 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07880c79d0 con 0x7f078806ddd0 2024-09-19T07:27:44.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f077effd640 1 -- 172.21.15.83:0/3864489711 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0778007400 con 0x7f078806ddd0 2024-09-19T07:27:44.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f077effd640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f076c0678c0 0x7f076c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:44.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f077effd640 1 -- 172.21.15.83:0/3864489711 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7f07780857a0 con 0x7f078806ddd0 2024-09-19T07:27:44.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078e0f5640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f076c0678c0 0x7f076c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:44.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.635+0000 7f078e0f5640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f076c0678c0 0x7f076c069d80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f07840b1000 tx=0x7f07840a9070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:44.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.639+0000 7f077effd640 1 -- 172.21.15.83:0/3864489711 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f0778055570 con 0x7f078806ddd0 2024-09-19T07:27:44.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.863+0000 7f0790359640 1 -- 172.21.15.83:0/3864489711 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0788067b10 con 0x7f078806ddd0 2024-09-19T07:27:44.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f077effd640 1 -- 172.21.15.83:0/3864489711 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5707 (secure 0 0 0) 0x7f0778059870 con 0x7f078806ddd0 2024-09-19T07:27:44.862 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:44.862 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":73,"btime":"2024-09-19T07:27:42:299328+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:27:44.862 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 73 2024-09-19T07:27:44.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f0790359640 1 -- 172.21.15.83:0/3864489711 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f076c0678c0 msgr2=0x7f076c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f0790359640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f076c0678c0 0x7f076c069d80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f07840b1000 tx=0x7f07840a9070 comp rx=0 tx=0).stop 2024-09-19T07:27:44.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f0790359640 1 -- 172.21.15.83:0/3864489711 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f078806ddd0 msgr2=0x7f07880acf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:44.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f0790359640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f078806ddd0 0x7f07880acf50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f077800c370 tx=0x7f077800c830 comp rx=0 tx=0).stop 2024-09-19T07:27:44.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f0790359640 1 -- 172.21.15.83:0/3864489711 shutdown_connections 2024-09-19T07:27:44.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f0790359640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f076c0678c0 0x7f076c069d80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f0790359640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07881534f0 0x7f07881ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f0790359640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f078806ddd0 0x7f07880acf50 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f0790359640 1 --2- 172.21.15.83:0/3864489711 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f078806cef0 0x7f07880aca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:44.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f0790359640 1 -- 172.21.15.83:0/3864489711 >> 172.21.15.83:0/3864489711 conn(0x7f07880aa910 msgr2=0x7f078815c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:44.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f0790359640 1 -- 172.21.15.83:0/3864489711 shutdown_connections 2024-09-19T07:27:44.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:44.867+0000 7f0790359640 1 -- 172.21.15.83:0/3864489711 wait complete. 2024-09-19T07:27:44.886 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-19T07:27:46.888 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:47.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.031+0000 7f3eebd19640 1 Processor -- start 2024-09-19T07:27:47.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.031+0000 7f3eebd19640 1 -- start start 2024-09-19T07:27:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.031+0000 7f3eebd19640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ee414ffb0 0x7f3ee40701b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.031+0000 7f3eebd19640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3ee406bc80 0x7f3ee40ace50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.031+0000 7f3eebd19640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ee41532e0 con 0x7f3ee406c080 2024-09-19T07:27:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.031+0000 7f3eebd19640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3ee40c2990 con 0x7f3ee406bc80 2024-09-19T07:27:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.031+0000 7f3eebd19640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3ee40c2c70 con 0x7f3ee414ffb0 2024-09-19T07:27:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.031+0000 7f3eea2b6640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3ee406c080 0x7f3ee40ad390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51546/0 (socket says 172.21.15.83:51546) 2024-09-19T07:27:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3ee92b4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3ee406bc80 0x7f3ee40ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3eea2b6640 1 -- 172.21.15.83:0/4150992310 learned_addr learned my addr 172.21.15.83:0/4150992310 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:47.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3ee9ab5640 1 --2- 172.21.15.83:0/4150992310 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ee414ffb0 0x7f3ee40701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:47.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3ee8ab3640 1 -- 172.21.15.83:0/4150992310 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3138697837 0 0) 0x7f3ee41532e0 con 0x7f3ee406c080 2024-09-19T07:27:47.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3ee8ab3640 1 -- 172.21.15.83:0/4150992310 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ed0003530 con 0x7f3ee406c080 2024-09-19T07:27:47.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3ee92b4640 1 -- 172.21.15.83:0/4150992310 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ee414ffb0 msgr2=0x7f3ee40701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:47.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3ee92b4640 1 --2- 172.21.15.83:0/4150992310 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ee414ffb0 0x7f3ee40701b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:47.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3ee92b4640 1 -- 172.21.15.83:0/4150992310 >> v1:172.21.15.83:6789/0 conn(0x7f3ee406c080 legacy=0x7f3ee40ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:47.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3ee92b4640 1 -- 172.21.15.83:0/4150992310 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee41535a0 con 0x7f3ee406bc80 2024-09-19T07:27:47.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3ee92b4640 1 --2- 172.21.15.83:0/4150992310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ee406bc80 0x7f3ee40ace50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3ed40097c0 tx=0x7f3ed401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80875f4e6d1315f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:47.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3ee8ab3640 1 -- 172.21.15.83:0/4150992310 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40b0070 con 0x7f3ee406bc80 2024-09-19T07:27:47.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3ee8ab3640 1 -- 172.21.15.83:0/4150992310 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ed40ab600 con 0x7f3ee406bc80 2024-09-19T07:27:47.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.035+0000 7f3ee8ab3640 1 -- 172.21.15.83:0/4150992310 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40ab9d0 con 0x7f3ee406bc80 2024-09-19T07:27:47.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 -- 172.21.15.83:0/4150992310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ee406bc80 msgr2=0x7f3ee40ace50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:47.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 --2- 172.21.15.83:0/4150992310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ee406bc80 0x7f3ee40ace50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3ed40097c0 tx=0x7f3ed401c380 comp rx=0 tx=0).stop 2024-09-19T07:27:47.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 -- 172.21.15.83:0/4150992310 shutdown_connections 2024-09-19T07:27:47.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 --2- 172.21.15.83:0/4150992310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ee406bc80 0x7f3ee40ace50 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:47.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 --2- 172.21.15.83:0/4150992310 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ee414ffb0 0x7f3ee40701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:47.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 -- 172.21.15.83:0/4150992310 >> 172.21.15.83:0/4150992310 conn(0x7f3ee40c90e0 msgr2=0x7f3ee40aa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:47.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 -- 172.21.15.83:0/4150992310 shutdown_connections 2024-09-19T07:27:47.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 -- 172.21.15.83:0/4150992310 wait complete. 2024-09-19T07:27:47.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 Processor -- start 2024-09-19T07:27:47.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 -- start start 2024-09-19T07:27:47.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ee406bc80 0x7f3ee40be580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ee406c080 0x7f3ee414ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ee414ffb0 0x7f3ee4151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee406f080 con 0x7f3ee406c080 2024-09-19T07:27:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee406ef00 con 0x7f3ee414ffb0 2024-09-19T07:27:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eebd19640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3ee415fa00 con 0x7f3ee406bc80 2024-09-19T07:27:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3ee92b4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ee406c080 0x7f3ee414ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3ee92b4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ee406c080 0x7f3ee414ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47366/0 (socket says 172.21.15.83:47366) 2024-09-19T07:27:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3ee92b4640 1 -- 172.21.15.83:0/1154096354 learned_addr learned my addr 172.21.15.83:0/1154096354 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3ee9ab5640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ee406bc80 0x7f3ee40be580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eea2b6640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ee414ffb0 0x7f3ee4151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3ee92b4640 1 -- 172.21.15.83:0/1154096354 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ee406bc80 msgr2=0x7f3ee40be580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:47.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3ee92b4640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ee406bc80 0x7f3ee40be580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:47.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3ee92b4640 1 -- 172.21.15.83:0/1154096354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ee414ffb0 msgr2=0x7f3ee4151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:47.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3ee92b4640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ee414ffb0 0x7f3ee4151bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:47.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3ee92b4640 1 -- 172.21.15.83:0/1154096354 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee40abd30 con 0x7f3ee406c080 2024-09-19T07:27:47.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eea2b6640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ee414ffb0 0x7f3ee4151bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:47.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3ee92b4640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ee406c080 0x7f3ee414ccf0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3ed40ae040 tx=0x7f3ed4019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:47.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.039+0000 7f3eda7fc640 1 -- 172.21.15.83:0/1154096354 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40b0040 con 0x7f3ee406c080 2024-09-19T07:27:47.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.043+0000 7f3eda7fc640 1 -- 172.21.15.83:0/1154096354 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ed40b9070 con 0x7f3ee406c080 2024-09-19T07:27:47.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.043+0000 7f3eebd19640 1 -- 172.21.15.83:0/1154096354 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee41507d0 con 0x7f3ee406c080 2024-09-19T07:27:47.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.043+0000 7f3eda7fc640 1 -- 172.21.15.83:0/1154096354 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40b4670 con 0x7f3ee406c080 2024-09-19T07:27:47.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.043+0000 7f3eebd19640 1 -- 172.21.15.83:0/1154096354 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ee40bd730 con 0x7f3ee406c080 2024-09-19T07:27:47.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.043+0000 7f3eebd19640 1 -- 172.21.15.83:0/1154096354 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ee40701b0 con 0x7f3ee406c080 2024-09-19T07:27:47.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.043+0000 7f3eda7fc640 1 -- 172.21.15.83:0/1154096354 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3ed40b4810 con 0x7f3ee406c080 2024-09-19T07:27:47.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.043+0000 7f3eda7fc640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3ec80678c0 0x7f3ec8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:47.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.043+0000 7f3ee9ab5640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3ec80678c0 0x7f3ec8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:47.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.043+0000 7f3eda7fc640 1 -- 172.21.15.83:0/1154096354 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7f3ed4120000 con 0x7f3ee406c080 2024-09-19T07:27:47.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.043+0000 7f3ee9ab5640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3ec80678c0 0x7f3ec8069d80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009a50 tx=0x7f3ee0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:47.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.051+0000 7f3eda7fc640 1 -- 172.21.15.83:0/1154096354 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3ed4016090 con 0x7f3ee406c080 2024-09-19T07:27:47.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 -- 172.21.15.83:0/1154096354 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ee415cfc0 con 0x7f3ee406c080 2024-09-19T07:27:47.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eda7fc640 1 -- 172.21.15.83:0/1154096354 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+6532 (secure 0 0 0) 0x7f3ed40f0420 con 0x7f3ee406c080 2024-09-19T07:27:47.315 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:47.315 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":74,"btime":"2024-09-19T07:27:45:324647+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5991,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:27:47.316 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 74 2024-09-19T07:27:47.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 -- 172.21.15.83:0/1154096354 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3ec80678c0 msgr2=0x7f3ec8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:47.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3ec80678c0 0x7f3ec8069d80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009a50 tx=0x7f3ee0008040 comp rx=0 tx=0).stop 2024-09-19T07:27:47.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 -- 172.21.15.83:0/1154096354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ee406c080 msgr2=0x7f3ee414ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:47.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ee406c080 0x7f3ee414ccf0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3ed40ae040 tx=0x7f3ed4019120 comp rx=0 tx=0).stop 2024-09-19T07:27:47.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 -- 172.21.15.83:0/1154096354 shutdown_connections 2024-09-19T07:27:47.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3ec80678c0 0x7f3ec8069d80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:47.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ee414ffb0 0x7f3ee4151bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:47.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ee406c080 0x7f3ee414ccf0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:47.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 --2- 172.21.15.83:0/1154096354 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ee406bc80 0x7f3ee40be580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:47.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 -- 172.21.15.83:0/1154096354 >> 172.21.15.83:0/1154096354 conn(0x7f3ee40c90e0 msgr2=0x7f3ee40c7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:47.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 -- 172.21.15.83:0/1154096354 shutdown_connections 2024-09-19T07:27:47.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:47.319+0000 7f3eebd19640 1 -- 172.21.15.83:0/1154096354 wait complete. 2024-09-19T07:27:47.338 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-19T07:27:47.338 INFO:tasks.mds_thrash.fs.[b]:waiting for 37 secs before thrashing 2024-09-19T07:27:54.305 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:54.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84c63b7640 1 Processor -- start 2024-09-19T07:27:54.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84c63b7640 1 -- start start 2024-09-19T07:27:54.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84c63b7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84c00c6120 0x7f84c00c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84c63b7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84c0155d70 0x7f84c01575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84c63b7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f84c01592f0 0x7f84c0157b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84c63b7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f84c00c9930 con 0x7f84c0155d70 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84c63b7640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f84c0156200 con 0x7f84c01592f0 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84c63b7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f84c01564e0 con 0x7f84c00c6120 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84bf7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84c0155d70 0x7f84c01575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84bf7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84c0155d70 0x7f84c01575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38738/0 (socket says 172.21.15.83:38738) 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84bf7fe640 1 -- 172.21.15.83:0/1331770658 learned_addr learned my addr 172.21.15.83:0/1331770658 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84bffff640 1 --2- 172.21.15.83:0/1331770658 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84c00c6120 0x7f84c00c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84c4954640 1 --2- 172.21.15.83:0/1331770658 >> v2:172.21.15.184:3300/0 conn(0x7f84c01592f0 0x7f84c0157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84bf7fe640 1 -- 172.21.15.83:0/1331770658 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84c00c6120 msgr2=0x7f84c00c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84bf7fe640 1 --2- 172.21.15.83:0/1331770658 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84c00c6120 0x7f84c00c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84bf7fe640 1 -- 172.21.15.83:0/1331770658 >> v2:172.21.15.184:3300/0 conn(0x7f84c01592f0 msgr2=0x7f84c0157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84bf7fe640 1 --2- 172.21.15.83:0/1331770658 >> v2:172.21.15.184:3300/0 conn(0x7f84c01592f0 0x7f84c0157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.439+0000 7f84bf7fe640 1 -- 172.21.15.83:0/1331770658 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c01567c0 con 0x7f84c0155d70 2024-09-19T07:27:54.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84bf7fe640 1 --2- 172.21.15.83:0/1331770658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84c0155d70 0x7f84c01575d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f84c008d2e0 tx=0x7f84b401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2040c7987919201 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:54.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84beffd640 1 -- 172.21.15.83:0/1331770658 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b40b0070 con 0x7f84c0155d70 2024-09-19T07:27:54.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84beffd640 1 -- 172.21.15.83:0/1331770658 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84b40ab600 con 0x7f84c0155d70 2024-09-19T07:27:54.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84beffd640 1 -- 172.21.15.83:0/1331770658 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b40ab9d0 con 0x7f84c0155d70 2024-09-19T07:27:54.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84c63b7640 1 -- 172.21.15.83:0/1331770658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84c0155d70 msgr2=0x7f84c01575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:54.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84c63b7640 1 --2- 172.21.15.83:0/1331770658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84c0155d70 0x7f84c01575d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f84c008d2e0 tx=0x7f84b401c520 comp rx=0 tx=0).stop 2024-09-19T07:27:54.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84c63b7640 1 -- 172.21.15.83:0/1331770658 shutdown_connections 2024-09-19T07:27:54.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84c63b7640 1 --2- 172.21.15.83:0/1331770658 >> v2:172.21.15.184:3300/0 conn(0x7f84c01592f0 0x7f84c0157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84c63b7640 1 --2- 172.21.15.83:0/1331770658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84c0155d70 0x7f84c01575d0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84c63b7640 1 --2- 172.21.15.83:0/1331770658 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84c00c6120 0x7f84c00c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84c63b7640 1 -- 172.21.15.83:0/1331770658 >> 172.21.15.83:0/1331770658 conn(0x7f84c006bc80 msgr2=0x7f84c006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:54.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84c63b7640 1 -- 172.21.15.83:0/1331770658 shutdown_connections 2024-09-19T07:27:54.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.443+0000 7f84c63b7640 1 -- 172.21.15.83:0/1331770658 wait complete. 2024-09-19T07:27:54.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c63b7640 1 Processor -- start 2024-09-19T07:27:54.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c63b7640 1 -- start start 2024-09-19T07:27:54.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c63b7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84c00c6120 0x7f84c014a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c63b7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84c01592f0 0x7f84c014a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c63b7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84c014c750 0x7f84c014cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c63b7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f84c00c8b40 con 0x7f84c014c750 2024-09-19T07:27:54.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c63b7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f84c00c89c0 con 0x7f84c00c6120 2024-09-19T07:27:54.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c63b7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f84c00c8cc0 con 0x7f84c01592f0 2024-09-19T07:27:54.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c4954640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84c014c750 0x7f84c014cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c4954640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84c014c750 0x7f84c014cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38750/0 (socket says 172.21.15.83:38750) 2024-09-19T07:27:54.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c4954640 1 -- 172.21.15.83:0/2899005848 learned_addr learned my addr 172.21.15.83:0/2899005848 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:54.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84bffff640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84c00c6120 0x7f84c014a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84bf7fe640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84c01592f0 0x7f84c014a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c4954640 1 -- 172.21.15.83:0/2899005848 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84c01592f0 msgr2=0x7f84c014a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:54.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c4954640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84c01592f0 0x7f84c014a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c4954640 1 -- 172.21.15.83:0/2899005848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84c00c6120 msgr2=0x7f84c014a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:54.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c4954640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84c00c6120 0x7f84c014a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c4954640 1 -- 172.21.15.83:0/2899005848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c00adf90 con 0x7f84c014c750 2024-09-19T07:27:54.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c4954640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84c014c750 0x7f84c014cbe0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f84b800efc0 tx=0x7f84b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:54.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84bcff9640 1 -- 172.21.15.83:0/2899005848 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b8018020 con 0x7f84c014c750 2024-09-19T07:27:54.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84bcff9640 1 -- 172.21.15.83:0/2899005848 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84b8014480 con 0x7f84c014c750 2024-09-19T07:27:54.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c63b7640 1 -- 172.21.15.83:0/2899005848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c01f29e0 con 0x7f84c014c750 2024-09-19T07:27:54.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84bcff9640 1 -- 172.21.15.83:0/2899005848 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b8009e60 con 0x7f84c014c750 2024-09-19T07:27:54.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c63b7640 1 -- 172.21.15.83:0/2899005848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c0148d50 con 0x7f84c014c750 2024-09-19T07:27:54.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.447+0000 7f84c63b7640 1 -- 172.21.15.83:0/2899005848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84c01f2810 con 0x7f84c014c750 2024-09-19T07:27:54.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.451+0000 7f84bcff9640 1 -- 172.21.15.83:0/2899005848 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f84b801b020 con 0x7f84c014c750 2024-09-19T07:27:54.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.451+0000 7f84bcff9640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f84ac0678c0 0x7f84ac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.451+0000 7f84bcff9640 1 -- 172.21.15.83:0/2899005848 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7f84b8087460 con 0x7f84c014c750 2024-09-19T07:27:54.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.451+0000 7f84bffff640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f84ac0678c0 0x7f84ac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.451+0000 7f84bffff640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f84ac0678c0 0x7f84ac069d80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f84c014c440 tx=0x7f84b0009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:54.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.455+0000 7f84bcff9640 1 -- 172.21.15.83:0/2899005848 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f84b8022030 con 0x7f84c014c750 2024-09-19T07:27:54.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.723+0000 7f84c63b7640 1 -- 172.21.15.83:0/2899005848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84c00c6500 con 0x7f84c014c750 2024-09-19T07:27:54.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.723+0000 7f84bcff9640 1 -- 172.21.15.83:0/2899005848 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+6532 (secure 0 0 0) 0x7f84b8010040 con 0x7f84c014c750 2024-09-19T07:27:54.721 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:54.721 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":74,"btime":"2024-09-19T07:27:45:324647+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5991,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:27:54.721 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 74 2024-09-19T07:27:54.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.727+0000 7f84c63b7640 1 -- 172.21.15.83:0/2899005848 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f84ac0678c0 msgr2=0x7f84ac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:54.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.727+0000 7f84c63b7640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f84ac0678c0 0x7f84ac069d80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f84c014c440 tx=0x7f84b0009190 comp rx=0 tx=0).stop 2024-09-19T07:27:54.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.727+0000 7f84c63b7640 1 -- 172.21.15.83:0/2899005848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84c014c750 msgr2=0x7f84c014cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:54.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.727+0000 7f84c63b7640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84c014c750 0x7f84c014cbe0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f84b800efc0 tx=0x7f84b800c490 comp rx=0 tx=0).stop 2024-09-19T07:27:54.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.727+0000 7f84c63b7640 1 -- 172.21.15.83:0/2899005848 shutdown_connections 2024-09-19T07:27:54.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.727+0000 7f84c63b7640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f84ac0678c0 0x7f84ac069d80 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.727+0000 7f84c63b7640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84c014c750 0x7f84c014cbe0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.727+0000 7f84c63b7640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84c01592f0 0x7f84c014a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.727+0000 7f84c63b7640 1 --2- 172.21.15.83:0/2899005848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84c00c6120 0x7f84c014a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.727+0000 7f84c63b7640 1 -- 172.21.15.83:0/2899005848 >> 172.21.15.83:0/2899005848 conn(0x7f84c006bc80 msgr2=0x7f84c0154a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:54.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.727+0000 7f84c63b7640 1 -- 172.21.15.83:0/2899005848 shutdown_connections 2024-09-19T07:27:54.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.727+0000 7f84c63b7640 1 -- 172.21.15.83:0/2899005848 wait complete. 2024-09-19T07:27:54.740 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-09-19T07:27:54.741 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-19T07:27:54.741 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:27:54.754 INFO:tasks.ceph.mds.b:Stopped 2024-09-19T07:27:54.755 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.887+0000 7f454fcf5640 1 Processor -- start 2024-09-19T07:27:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.887+0000 7f454fcf5640 1 -- start start 2024-09-19T07:27:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454fcf5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4548153300 0x7f454806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454fcf5640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4548155bb0 0x7f4548156060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454fcf5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45481593f0 con 0x7f454806d1d0 2024-09-19T07:27:54.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454fcf5640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f45481591a0 con 0x7f4548155bb0 2024-09-19T07:27:54.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454fcf5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f45481596d0 con 0x7f4548153300 2024-09-19T07:27:54.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454d290640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f454806d1d0 0x7f45480c23d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56412/0 (socket says 172.21.15.83:56412) 2024-09-19T07:27:54.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454d290640 1 -- 172.21.15.83:0/3628013533 learned_addr learned my addr 172.21.15.83:0/3628013533 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:54.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454e292640 1 --2- 172.21.15.83:0/3628013533 >> v2:172.21.15.184:3300/0 conn(0x7f4548155bb0 0x7f4548156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454da91640 1 --2- 172.21.15.83:0/3628013533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4548153300 0x7f454806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454ca8f640 1 -- 172.21.15.83:0/3628013533 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1502317534 0 0) 0x7f45481593f0 con 0x7f454806d1d0 2024-09-19T07:27:54.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454ca8f640 1 -- 172.21.15.83:0/3628013533 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4534003530 con 0x7f454806d1d0 2024-09-19T07:27:54.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454da91640 1 -- 172.21.15.83:0/3628013533 >> v2:172.21.15.184:3300/0 conn(0x7f4548155bb0 msgr2=0x7f4548156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:54.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454da91640 1 --2- 172.21.15.83:0/3628013533 >> v2:172.21.15.184:3300/0 conn(0x7f4548155bb0 0x7f4548156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454da91640 1 -- 172.21.15.83:0/3628013533 >> v1:172.21.15.83:6789/0 conn(0x7f454806d1d0 legacy=0x7f45480c23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:54.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454da91640 1 -- 172.21.15.83:0/3628013533 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45481599b0 con 0x7f4548153300 2024-09-19T07:27:54.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454da91640 1 --2- 172.21.15.83:0/3628013533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4548153300 0x7f454806cc90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f45440097c0 tx=0x7f454401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcc4d23e487e96fb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:54.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454ca8f640 1 -- 172.21.15.83:0/3628013533 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45440b0070 con 0x7f4548153300 2024-09-19T07:27:54.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454ca8f640 1 -- 172.21.15.83:0/3628013533 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45440ab600 con 0x7f4548153300 2024-09-19T07:27:54.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.891+0000 7f454ca8f640 1 -- 172.21.15.83:0/3628013533 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45440ab9d0 con 0x7f4548153300 2024-09-19T07:27:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.895+0000 7f454fcf5640 1 -- 172.21.15.83:0/3628013533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4548153300 msgr2=0x7f454806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.895+0000 7f454fcf5640 1 --2- 172.21.15.83:0/3628013533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4548153300 0x7f454806cc90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f45440097c0 tx=0x7f454401c380 comp rx=0 tx=0).stop 2024-09-19T07:27:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.895+0000 7f454fcf5640 1 -- 172.21.15.83:0/3628013533 shutdown_connections 2024-09-19T07:27:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.895+0000 7f454fcf5640 1 --2- 172.21.15.83:0/3628013533 >> v2:172.21.15.184:3300/0 conn(0x7f4548155bb0 0x7f4548156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.895+0000 7f454fcf5640 1 --2- 172.21.15.83:0/3628013533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4548153300 0x7f454806cc90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.895+0000 7f454fcf5640 1 -- 172.21.15.83:0/3628013533 >> 172.21.15.83:0/3628013533 conn(0x7f45480c5090 msgr2=0x7f45480c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.895+0000 7f454fcf5640 1 -- 172.21.15.83:0/3628013533 shutdown_connections 2024-09-19T07:27:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.895+0000 7f454fcf5640 1 -- 172.21.15.83:0/3628013533 wait complete. 2024-09-19T07:27:54.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.895+0000 7f454fcf5640 1 Processor -- start 2024-09-19T07:27:54.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.895+0000 7f454fcf5640 1 -- start start 2024-09-19T07:27:54.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454fcf5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f454806d1d0 0x7f45481ee9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454fcf5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548153300 0x7f4548150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454fcf5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4548155bb0 0x7f454814d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454fcf5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4548153860 con 0x7f4548153300 2024-09-19T07:27:54.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454fcf5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f45481536e0 con 0x7f4548155bb0 2024-09-19T07:27:54.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454fcf5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f45481565a0 con 0x7f454806d1d0 2024-09-19T07:27:54.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454d290640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548153300 0x7f4548150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454d290640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548153300 0x7f4548150c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38754/0 (socket says 172.21.15.83:38754) 2024-09-19T07:27:54.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454d290640 1 -- 172.21.15.83:0/517905550 learned_addr learned my addr 172.21.15.83:0/517905550 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:54.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454da91640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f454806d1d0 0x7f45481ee9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454e292640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4548155bb0 0x7f454814d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454d290640 1 -- 172.21.15.83:0/517905550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f454806d1d0 msgr2=0x7f45481ee9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:54.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454d290640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f454806d1d0 0x7f45481ee9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454d290640 1 -- 172.21.15.83:0/517905550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4548155bb0 msgr2=0x7f454814d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:54.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454d290640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4548155bb0 0x7f454814d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:54.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454d290640 1 -- 172.21.15.83:0/517905550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4548151190 con 0x7f4548153300 2024-09-19T07:27:54.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454d290640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548153300 0x7f4548150c50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f453800c3c0 tx=0x7f453800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:54.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f453e7fc640 1 -- 172.21.15.83:0/517905550 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4538017070 con 0x7f4548153300 2024-09-19T07:27:54.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f453e7fc640 1 -- 172.21.15.83:0/517905550 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4538005d10 con 0x7f4548153300 2024-09-19T07:27:54.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454fcf5640 1 -- 172.21.15.83:0/517905550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45480c8140 con 0x7f4548153300 2024-09-19T07:27:54.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f453e7fc640 1 -- 172.21.15.83:0/517905550 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4538007310 con 0x7f4548153300 2024-09-19T07:27:54.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454fcf5640 1 -- 172.21.15.83:0/517905550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4548152200 con 0x7f4548153300 2024-09-19T07:27:54.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454fcf5640 1 -- 172.21.15.83:0/517905550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f454806cc90 con 0x7f4548153300 2024-09-19T07:27:54.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f453e7fc640 1 -- 172.21.15.83:0/517905550 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f453801a070 con 0x7f4548153300 2024-09-19T07:27:54.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f453e7fc640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f452c0678c0 0x7f452c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:54.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f453e7fc640 1 -- 172.21.15.83:0/517905550 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7f45380856a0 con 0x7f4548153300 2024-09-19T07:27:54.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454da91640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f452c0678c0 0x7f452c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:54.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.899+0000 7f454da91640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f452c0678c0 0x7f452c069d80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f45440ad000 tx=0x7f4544008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:54.903+0000 7f453e7fc640 1 -- 172.21.15.83:0/517905550 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4538055530 con 0x7f4548153300 2024-09-19T07:27:55.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.111+0000 7f454fcf5640 1 -- 172.21.15.83:0/517905550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f454814dc40 con 0x7f4548153300 2024-09-19T07:27:55.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.111+0000 7f453e7fc640 1 -- 172.21.15.83:0/517905550 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+6532 (secure 0 0 0) 0x7f4538059830 con 0x7f4548153300 2024-09-19T07:27:55.109 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:55.110 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":74,"btime":"2024-09-19T07:27:45:324647+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5991,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:27:55.110 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 74 2024-09-19T07:27:55.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.115+0000 7f454fcf5640 1 -- 172.21.15.83:0/517905550 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f452c0678c0 msgr2=0x7f452c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:55.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.115+0000 7f454fcf5640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f452c0678c0 0x7f452c069d80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f45440ad000 tx=0x7f4544008040 comp rx=0 tx=0).stop 2024-09-19T07:27:55.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.115+0000 7f454fcf5640 1 -- 172.21.15.83:0/517905550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548153300 msgr2=0x7f4548150c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:55.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.115+0000 7f454fcf5640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548153300 0x7f4548150c50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f453800c3c0 tx=0x7f453800c880 comp rx=0 tx=0).stop 2024-09-19T07:27:55.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.115+0000 7f454fcf5640 1 -- 172.21.15.83:0/517905550 shutdown_connections 2024-09-19T07:27:55.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.115+0000 7f454fcf5640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f452c0678c0 0x7f452c069d80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:55.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.115+0000 7f454fcf5640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4548155bb0 0x7f454814d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:55.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.115+0000 7f454fcf5640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548153300 0x7f4548150c50 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:55.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.115+0000 7f454fcf5640 1 --2- 172.21.15.83:0/517905550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f454806d1d0 0x7f45481ee9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:55.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.115+0000 7f454fcf5640 1 -- 172.21.15.83:0/517905550 >> 172.21.15.83:0/517905550 conn(0x7f45480c5090 msgr2=0x7f4548155360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:55.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.115+0000 7f454fcf5640 1 -- 172.21.15.83:0/517905550 shutdown_connections 2024-09-19T07:27:55.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:55.115+0000 7f454fcf5640 1 -- 172.21.15.83:0/517905550 wait complete. 2024-09-19T07:27:55.128 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:27:57.130 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:57.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.263+0000 7ff6bae73640 1 Processor -- start 2024-09-19T07:27:57.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.263+0000 7ff6bae73640 1 -- start start 2024-09-19T07:27:57.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6bae73640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6b4153eb0 0x7ff6b4154340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:57.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6bae73640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6b4150670 con 0x7ff6b4152b20 2024-09-19T07:27:57.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6bae73640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6b40c2880 con 0x7ff6b4150290 2024-09-19T07:27:57.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6bae73640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff6b40716c0 con 0x7ff6b4153eb0 2024-09-19T07:27:57.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff6b4152b20 0x7ff6b40c4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56420/0 (socket says 172.21.15.83:56420) 2024-09-19T07:27:57.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b3fff640 1 -- 172.21.15.83:0/650772892 learned_addr learned my addr 172.21.15.83:0/650772892 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:27:57.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b9410640 1 --2- 172.21.15.83:0/650772892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6b4153eb0 0x7ff6b4154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:57.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1593857986 0 0) 0x7ff6b4150670 con 0x7ff6b4152b20 2024-09-19T07:27:57.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6a0003530 con 0x7ff6b4152b20 2024-09-19T07:27:57.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2859341672 0 0) 0x7ff6a0003530 con 0x7ff6b4152b20 2024-09-19T07:27:57.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6b4153eb0 msgr2=0x7ff6b4154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:57.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 --2- 172.21.15.83:0/650772892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6b4153eb0 0x7ff6b4154340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:57.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 >> v1:172.21.15.184:6789/0 conn(0x7ff6b4150290 legacy=0x7ff6b40c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:57.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b40c2b60 con 0x7ff6b4152b20 2024-09-19T07:27:57.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7ff6a4001680 con 0x7ff6b4152b20 2024-09-19T07:27:57.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 >> v1:172.21.15.83:6789/0 conn(0x7ff6b4152b20 legacy=0x7ff6b40c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:57.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 --2- 172.21.15.83:0/650772892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6a00039f0 0x7ff6a0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:57.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 --2- 172.21.15.83:0/650772892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6b4153eb0 0x7ff6a0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:57.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 --2- 172.21.15.83:0/650772892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6a0004d50 0x7ff6a00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:57.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a000ba80 con 0x7ff6a00039f0 2024-09-19T07:27:57.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a000bc30 con 0x7ff6a0004d50 2024-09-19T07:27:57.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a000be30 con 0x7ff6b4153eb0 2024-09-19T07:27:57.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b8c0f640 1 --2- 172.21.15.83:0/650772892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6a00039f0 0x7ff6a0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:57.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b9410640 1 --2- 172.21.15.83:0/650772892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6a0004d50 0x7ff6a00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:57.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b8c0f640 1 -- 172.21.15.83:0/650772892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6b4153eb0 msgr2=0x7ff6a0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:57.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b8c0f640 1 --2- 172.21.15.83:0/650772892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6b4153eb0 0x7ff6a0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:57.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b8c0f640 1 -- 172.21.15.83:0/650772892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6a0004d50 msgr2=0x7ff6a00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:57.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b8c0f640 1 --2- 172.21.15.83:0/650772892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6a0004d50 0x7ff6a00051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:57.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b8c0f640 1 -- 172.21.15.83:0/650772892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a0003530 con 0x7ff6a00039f0 2024-09-19T07:27:57.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.267+0000 7ff6b8c0f640 1 --2- 172.21.15.83:0/650772892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6a00039f0 0x7ff6a0003e10 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ff69c001750 tx=0x7ff69c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6234aa1bd646eb6d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:57.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c0013a0 con 0x7ff6a00039f0 2024-09-19T07:27:57.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff69c0052f0 con 0x7ff6a00039f0 2024-09-19T07:27:57.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b37fe640 1 -- 172.21.15.83:0/650772892 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c003840 con 0x7ff6a00039f0 2024-09-19T07:27:57.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 -- 172.21.15.83:0/650772892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6a00039f0 msgr2=0x7ff6a0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:57.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 --2- 172.21.15.83:0/650772892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6a00039f0 0x7ff6a0003e10 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ff69c001750 tx=0x7ff69c01c3e0 comp rx=0 tx=0).stop 2024-09-19T07:27:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b9410640 1 -- 172.21.15.83:0/650772892 reap_dead start 2024-09-19T07:27:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 -- 172.21.15.83:0/650772892 shutdown_connections 2024-09-19T07:27:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 -- 172.21.15.83:0/650772892 >> 172.21.15.83:0/650772892 conn(0x7ff6b406bc80 msgr2=0x7ff6b406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 -- 172.21.15.83:0/650772892 shutdown_connections 2024-09-19T07:27:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 -- 172.21.15.83:0/650772892 wait complete. 2024-09-19T07:27:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 Processor -- start 2024-09-19T07:27:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 -- start start 2024-09-19T07:27:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6a00039f0 0x7ff6b414e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6b4149830 0x7ff6b4149cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6b4158820 0x7ff6b4158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b415e4d0 con 0x7ff6b4158820 2024-09-19T07:27:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b415e350 con 0x7ff6b4149830 2024-09-19T07:27:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff6b415e650 con 0x7ff6a00039f0 2024-09-19T07:27:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b9410640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6b4158820 0x7ff6b4158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b9410640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6b4158820 0x7ff6b4158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38772/0 (socket says 172.21.15.83:38772) 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b9410640 1 -- 172.21.15.83:0/2629041655 learned_addr learned my addr 172.21.15.83:0/2629041655 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b8c0f640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6a00039f0 0x7ff6b414e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b3fff640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6b4149830 0x7ff6b4149cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b9410640 1 -- 172.21.15.83:0/2629041655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6a00039f0 msgr2=0x7ff6b414e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b9410640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6a00039f0 0x7ff6b414e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b9410640 1 -- 172.21.15.83:0/2629041655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6b4149830 msgr2=0x7ff6b4149cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b9410640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6b4149830 0x7ff6b4149cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b9410640 1 -- 172.21.15.83:0/2629041655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b414a1f0 con 0x7ff6b4158820 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b3fff640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6b4149830 0x7ff6b4149cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b9410640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6b4158820 0x7ff6b4158ca0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40c1cb0 tx=0x7ff6ac00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b17fa640 1 -- 172.21.15.83:0/2629041655 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac0039a0 con 0x7ff6b4158820 2024-09-19T07:27:57.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b17fa640 1 -- 172.21.15.83:0/2629041655 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6ac003b40 con 0x7ff6b4158820 2024-09-19T07:27:57.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 -- 172.21.15.83:0/2629041655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b4158090 con 0x7ff6b4158820 2024-09-19T07:27:57.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6b17fa640 1 -- 172.21.15.83:0/2629041655 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac004020 con 0x7ff6b4158820 2024-09-19T07:27:57.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.271+0000 7ff6bae73640 1 -- 172.21.15.83:0/2629041655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6b41f03d0 con 0x7ff6b4158820 2024-09-19T07:27:57.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.275+0000 7ff6bae73640 1 -- 172.21.15.83:0/2629041655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6b40c1ff0 con 0x7ff6b4158820 2024-09-19T07:27:57.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.275+0000 7ff6b17fa640 1 -- 172.21.15.83:0/2629041655 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff6ac0041c0 con 0x7ff6b4158820 2024-09-19T07:27:57.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.275+0000 7ff6b17fa640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6940678c0 0x7ff694069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:57.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.275+0000 7ff6b8c0f640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6940678c0 0x7ff694069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:57.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.275+0000 7ff6b17fa640 1 -- 172.21.15.83:0/2629041655 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7ff6ac086900 con 0x7ff6b4158820 2024-09-19T07:27:57.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.275+0000 7ff6b8c0f640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6940678c0 0x7ff694069d80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00057f0 tx=0x7ff69c0011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:57.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.279+0000 7ff6b17fa640 1 -- 172.21.15.83:0/2629041655 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff6ac012070 con 0x7ff6b4158820 2024-09-19T07:27:57.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.483+0000 7ff6bae73640 1 -- 172.21.15.83:0/2629041655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6b4157a00 con 0x7ff6b4158820 2024-09-19T07:27:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.483+0000 7ff6b17fa640 1 -- 172.21.15.83:0/2629041655 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+6532 (secure 0 0 0) 0x7ff6ac056d20 con 0x7ff6b4158820 2024-09-19T07:27:57.479 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:57.480 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":74,"btime":"2024-09-19T07:27:45:324647+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5991,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:27:57.480 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 74 2024-09-19T07:27:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.487+0000 7ff6bae73640 1 -- 172.21.15.83:0/2629041655 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6940678c0 msgr2=0x7ff694069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.487+0000 7ff6bae73640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6940678c0 0x7ff694069d80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00057f0 tx=0x7ff69c0011a0 comp rx=0 tx=0).stop 2024-09-19T07:27:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.487+0000 7ff6bae73640 1 -- 172.21.15.83:0/2629041655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6b4158820 msgr2=0x7ff6b4158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.487+0000 7ff6bae73640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6b4158820 0x7ff6b4158ca0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40c1cb0 tx=0x7ff6ac00f4a0 comp rx=0 tx=0).stop 2024-09-19T07:27:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.487+0000 7ff6bae73640 1 -- 172.21.15.83:0/2629041655 shutdown_connections 2024-09-19T07:27:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.487+0000 7ff6bae73640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6940678c0 0x7ff694069d80 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.487+0000 7ff6bae73640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6b4158820 0x7ff6b4158ca0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.487+0000 7ff6bae73640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6b4149830 0x7ff6b4149cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.487+0000 7ff6bae73640 1 --2- 172.21.15.83:0/2629041655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6a00039f0 0x7ff6b414e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.487+0000 7ff6bae73640 1 -- 172.21.15.83:0/2629041655 >> 172.21.15.83:0/2629041655 conn(0x7ff6b406bc80 msgr2=0x7ff6b406e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.487+0000 7ff6bae73640 1 -- 172.21.15.83:0/2629041655 shutdown_connections 2024-09-19T07:27:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:57.487+0000 7ff6bae73640 1 -- 172.21.15.83:0/2629041655 wait complete. 2024-09-19T07:27:57.500 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:27:59.501 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:27:59.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304907f640 1 Processor -- start 2024-09-19T07:27:59.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304907f640 1 -- start start 2024-09-19T07:27:59.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304907f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3044071100 0x7f30441545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:59.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304907f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30440c28e0 0x7f3044150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:59.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304907f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3044151480 0x7f304406dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:59.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304907f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3044071b50 con 0x7f3044151480 2024-09-19T07:27:59.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304907f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f30440c1ff0 con 0x7f3044071100 2024-09-19T07:27:59.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304907f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f30440a9e60 con 0x7f30440c28e0 2024-09-19T07:27:59.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304359e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3044151480 0x7f304406dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:59.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304359e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3044151480 0x7f304406dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38776/0 (socket says 172.21.15.83:38776) 2024-09-19T07:27:59.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304359e640 1 -- 172.21.15.83:0/4237193215 learned_addr learned my addr 172.21.15.83:0/4237193215 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:59.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304259c640 1 --2- 172.21.15.83:0/4237193215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30440c28e0 0x7f3044150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:59.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f3042d9d640 1 --2- 172.21.15.83:0/4237193215 >> v2:172.21.15.184:3300/0 conn(0x7f3044071100 0x7f30441545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:59.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304359e640 1 -- 172.21.15.83:0/4237193215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30440c28e0 msgr2=0x7f3044150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:59.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304359e640 1 --2- 172.21.15.83:0/4237193215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30440c28e0 0x7f3044150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:59.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304359e640 1 -- 172.21.15.83:0/4237193215 >> v2:172.21.15.184:3300/0 conn(0x7f3044071100 msgr2=0x7f30441545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:27:59.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304359e640 1 --2- 172.21.15.83:0/4237193215 >> v2:172.21.15.184:3300/0 conn(0x7f3044071100 0x7f30441545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:59.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304359e640 1 -- 172.21.15.83:0/4237193215 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30440aa140 con 0x7f3044151480 2024-09-19T07:27:59.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.635+0000 7f304259c640 1 --2- 172.21.15.83:0/4237193215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30440c28e0 0x7f3044150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:27:59.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304359e640 1 --2- 172.21.15.83:0/4237193215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3044151480 0x7f304406dea0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f304406f620 tx=0x7f303c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0408dfc9b7acbd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:59.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f3041d9b640 1 -- 172.21.15.83:0/4237193215 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c0a55e0 con 0x7f3044151480 2024-09-19T07:27:59.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f3041d9b640 1 -- 172.21.15.83:0/4237193215 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f303c0a5780 con 0x7f3044151480 2024-09-19T07:27:59.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f3041d9b640 1 -- 172.21.15.83:0/4237193215 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c0a5b50 con 0x7f3044151480 2024-09-19T07:27:59.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 -- 172.21.15.83:0/4237193215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3044151480 msgr2=0x7f304406dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:59.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 --2- 172.21.15.83:0/4237193215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3044151480 0x7f304406dea0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f304406f620 tx=0x7f303c01c960 comp rx=0 tx=0).stop 2024-09-19T07:27:59.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 -- 172.21.15.83:0/4237193215 shutdown_connections 2024-09-19T07:27:59.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 --2- 172.21.15.83:0/4237193215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3044151480 0x7f304406dea0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:59.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 --2- 172.21.15.83:0/4237193215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30440c28e0 0x7f3044150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:59.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 --2- 172.21.15.83:0/4237193215 >> v2:172.21.15.184:3300/0 conn(0x7f3044071100 0x7f30441545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:59.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 -- 172.21.15.83:0/4237193215 >> 172.21.15.83:0/4237193215 conn(0x7f304406bc80 msgr2=0x7f304406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:59.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 -- 172.21.15.83:0/4237193215 shutdown_connections 2024-09-19T07:27:59.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 -- 172.21.15.83:0/4237193215 wait complete. 2024-09-19T07:27:59.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 Processor -- start 2024-09-19T07:27:59.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 -- start start 2024-09-19T07:27:59.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3044071100 0x7f30440ab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:59.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30440c28e0 0x7f304414f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:59.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3044151480 0x7f304414fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:59.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f30441570d0 con 0x7f30440c28e0 2024-09-19T07:27:59.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3044071730 con 0x7f3044151480 2024-09-19T07:27:59.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304907f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3044157250 con 0x7f3044071100 2024-09-19T07:27:59.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30440c28e0 0x7f304414f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:59.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30440c28e0 0x7f304414f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38780/0 (socket says 172.21.15.83:38780) 2024-09-19T07:27:59.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304259c640 1 -- 172.21.15.83:0/2910741928 learned_addr learned my addr 172.21.15.83:0/2910741928 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:27:59.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f3042d9d640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3044071100 0x7f30440ab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:59.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304359e640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3044151480 0x7f304414fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:59.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304259c640 1 -- 172.21.15.83:0/2910741928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3044071100 msgr2=0x7f30440ab3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:59.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304259c640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3044071100 0x7f30440ab3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:59.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304259c640 1 -- 172.21.15.83:0/2910741928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3044151480 msgr2=0x7f304414fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:59.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304259c640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3044151480 0x7f304414fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:59.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304259c640 1 -- 172.21.15.83:0/2910741928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30441564e0 con 0x7f30440c28e0 2024-09-19T07:27:59.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f304259c640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30440c28e0 0x7f304414f730 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f303400c7f0 tx=0x7f303400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:59.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.639+0000 7f302b7fe640 1 -- 172.21.15.83:0/2910741928 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034015070 con 0x7f30440c28e0 2024-09-19T07:27:59.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.643+0000 7f302b7fe640 1 -- 172.21.15.83:0/2910741928 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3034007a30 con 0x7f30440c28e0 2024-09-19T07:27:59.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.643+0000 7f304907f640 1 -- 172.21.15.83:0/2910741928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3044147b70 con 0x7f30440c28e0 2024-09-19T07:27:59.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.643+0000 7f302b7fe640 1 -- 172.21.15.83:0/2910741928 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034007e20 con 0x7f30440c28e0 2024-09-19T07:27:59.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.643+0000 7f304907f640 1 -- 172.21.15.83:0/2910741928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30440c6440 con 0x7f30440c28e0 2024-09-19T07:27:59.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.643+0000 7f304907f640 1 -- 172.21.15.83:0/2910741928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3044150f40 con 0x7f30440c28e0 2024-09-19T07:27:59.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.643+0000 7f302b7fe640 1 -- 172.21.15.83:0/2910741928 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3034018070 con 0x7f30440c28e0 2024-09-19T07:27:59.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.643+0000 7f302b7fe640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f30300678c0 0x7f3030069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:27:59.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.643+0000 7f302b7fe640 1 -- 172.21.15.83:0/2910741928 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7f3034086710 con 0x7f30440c28e0 2024-09-19T07:27:59.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.643+0000 7f3042d9d640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f30300678c0 0x7f3030069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:27:59.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.643+0000 7f3042d9d640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f30300678c0 0x7f3030069d80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f30440c1b10 tx=0x7f3038008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:27:59.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.647+0000 7f302b7fe640 1 -- 172.21.15.83:0/2910741928 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3034056b80 con 0x7f30440c28e0 2024-09-19T07:27:59.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.843+0000 7f304907f640 1 -- 172.21.15.83:0/2910741928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30440c4f30 con 0x7f30440c28e0 2024-09-19T07:27:59.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f302b7fe640 1 -- 172.21.15.83:0/2910741928 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+6532 (secure 0 0 0) 0x7f3034056480 con 0x7f30440c28e0 2024-09-19T07:27:59.842 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:27:59.842 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":74,"btime":"2024-09-19T07:27:45:324647+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5991,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:27:59.842 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 74 2024-09-19T07:27:59.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f304907f640 1 -- 172.21.15.83:0/2910741928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f30300678c0 msgr2=0x7f3030069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:59.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f304907f640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f30300678c0 0x7f3030069d80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f30440c1b10 tx=0x7f3038008040 comp rx=0 tx=0).stop 2024-09-19T07:27:59.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f304907f640 1 -- 172.21.15.83:0/2910741928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30440c28e0 msgr2=0x7f304414f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:27:59.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f304907f640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30440c28e0 0x7f304414f730 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f303400c7f0 tx=0x7f303400ccb0 comp rx=0 tx=0).stop 2024-09-19T07:27:59.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f304907f640 1 -- 172.21.15.83:0/2910741928 shutdown_connections 2024-09-19T07:27:59.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f304907f640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f30300678c0 0x7f3030069d80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:59.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f304907f640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3044151480 0x7f304414fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:59.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f304907f640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30440c28e0 0x7f304414f730 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:59.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f304907f640 1 --2- 172.21.15.83:0/2910741928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3044071100 0x7f30440ab3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:27:59.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f304907f640 1 -- 172.21.15.83:0/2910741928 >> 172.21.15.83:0/2910741928 conn(0x7f304406bc80 msgr2=0x7f304406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:27:59.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f304907f640 1 -- 172.21.15.83:0/2910741928 shutdown_connections 2024-09-19T07:27:59.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:27:59.847+0000 7f304907f640 1 -- 172.21.15.83:0/2910741928 wait complete. 2024-09-19T07:27:59.861 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:28:01.862 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:01.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.995+0000 7ff1cf03c640 1 Processor -- start 2024-09-19T07:28:01.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.995+0000 7ff1cf03c640 1 -- start start 2024-09-19T07:28:01.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1cf03c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff1c80ab750 0x7ff1c80c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:01.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1cf03c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff1c80c2530 0x7ff1c81528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1cf03c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1c80ab060 0x7ff1c8152e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1cf03c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff1c806de90 con 0x7ff1c80c2530 2024-09-19T07:28:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1cf03c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff1c806e170 con 0x7ff1c80ab750 2024-09-19T07:28:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1cf03c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff1c806e450 con 0x7ff1c80ab060 2024-09-19T07:28:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1c7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff1c80c2530 0x7ff1c81528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1cd5d9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1c80ab060 0x7ff1c8152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1c7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff1c80c2530 0x7ff1c81528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34190/0 (socket says 172.21.15.83:34190) 2024-09-19T07:28:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1ccdd8640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff1c80ab750 0x7ff1c80c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1c7fff640 1 -- 172.21.15.83:0/1271057935 learned_addr learned my addr 172.21.15.83:0/1271057935 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1c7fff640 1 -- 172.21.15.83:0/1271057935 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1c80ab060 msgr2=0x7ff1c8152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1c7fff640 1 --2- 172.21.15.83:0/1271057935 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1c80ab060 0x7ff1c8152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1c7fff640 1 -- 172.21.15.83:0/1271057935 >> v2:172.21.15.184:3300/0 conn(0x7ff1c80ab750 msgr2=0x7ff1c80c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1c7fff640 1 --2- 172.21.15.83:0/1271057935 >> v2:172.21.15.184:3300/0 conn(0x7ff1c80ab750 0x7ff1c80c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1c7fff640 1 -- 172.21.15.83:0/1271057935 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c806e730 con 0x7ff1c80c2530 2024-09-19T07:28:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1c7fff640 1 --2- 172.21.15.83:0/1271057935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c80c2530 0x7ff1c81528c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80ad7d0 tx=0x7ff1bc01e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c11f0bdeb7895d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1c77fe640 1 -- 172.21.15.83:0/1271057935 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0b4070 con 0x7ff1c80c2530 2024-09-19T07:28:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1c77fe640 1 -- 172.21.15.83:0/1271057935 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1bc0a7800 con 0x7ff1c80c2530 2024-09-19T07:28:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:01.999+0000 7ff1c77fe640 1 -- 172.21.15.83:0/1271057935 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc00b040 con 0x7ff1c80c2530 2024-09-19T07:28:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 -- 172.21.15.83:0/1271057935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c80c2530 msgr2=0x7ff1c81528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 --2- 172.21.15.83:0/1271057935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c80c2530 0x7ff1c81528c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80ad7d0 tx=0x7ff1bc01e970 comp rx=0 tx=0).stop 2024-09-19T07:28:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 -- 172.21.15.83:0/1271057935 shutdown_connections 2024-09-19T07:28:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 --2- 172.21.15.83:0/1271057935 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1c80ab060 0x7ff1c8152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:01.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 --2- 172.21.15.83:0/1271057935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c80c2530 0x7ff1c81528c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:01.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 --2- 172.21.15.83:0/1271057935 >> v2:172.21.15.184:3300/0 conn(0x7ff1c80ab750 0x7ff1c80c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:01.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 -- 172.21.15.83:0/1271057935 >> 172.21.15.83:0/1271057935 conn(0x7ff1c806bc80 msgr2=0x7ff1c806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:01.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 -- 172.21.15.83:0/1271057935 shutdown_connections 2024-09-19T07:28:01.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 -- 172.21.15.83:0/1271057935 wait complete. 2024-09-19T07:28:02.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 Processor -- start 2024-09-19T07:28:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 -- start start 2024-09-19T07:28:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1c80ab060 0x7ff1c81500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1c80ab750 0x7ff1c8149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c80c2530 0x7ff1c81471a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c8155c20 con 0x7ff1c80c2530 2024-09-19T07:28:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c8155aa0 con 0x7ff1c80ab750 2024-09-19T07:28:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff1c8155da0 con 0x7ff1c80ab060 2024-09-19T07:28:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cd5d9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c80c2530 0x7ff1c81471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cd5d9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c80c2530 0x7ff1c81471a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34206/0 (socket says 172.21.15.83:34206) 2024-09-19T07:28:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cd5d9640 1 -- 172.21.15.83:0/2084340478 learned_addr learned my addr 172.21.15.83:0/2084340478 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1ccdd8640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1c80ab060 0x7ff1c81500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1c7fff640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1c80ab750 0x7ff1c8149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:02.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cd5d9640 1 -- 172.21.15.83:0/2084340478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1c80ab060 msgr2=0x7ff1c81500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:02.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cd5d9640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1c80ab060 0x7ff1c81500b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:02.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cd5d9640 1 -- 172.21.15.83:0/2084340478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1c80ab750 msgr2=0x7ff1c8149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:02.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cd5d9640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1c80ab750 0x7ff1c8149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:02.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cd5d9640 1 -- 172.21.15.83:0/2084340478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c81505f0 con 0x7ff1c80c2530 2024-09-19T07:28:02.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1c7fff640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1c80ab750 0x7ff1c8149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:28:02.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cd5d9640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c80c2530 0x7ff1c81471a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000eed0 tx=0x7ff1c000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:02.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1c57fa640 1 -- 172.21.15.83:0/2084340478 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0018020 con 0x7ff1c80c2530 2024-09-19T07:28:02.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1c57fa640 1 -- 172.21.15.83:0/2084340478 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1c0014400 con 0x7ff1c80c2530 2024-09-19T07:28:02.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1c57fa640 1 -- 172.21.15.83:0/2084340478 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c00147f0 con 0x7ff1c80c2530 2024-09-19T07:28:02.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 -- 172.21.15.83:0/2084340478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1c80c7010 con 0x7ff1c80c2530 2024-09-19T07:28:02.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 -- 172.21.15.83:0/2084340478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1c80730a0 con 0x7ff1c80c2530 2024-09-19T07:28:02.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.003+0000 7ff1cf03c640 1 -- 172.21.15.83:0/2084340478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1c80c1ff0 con 0x7ff1c80c2530 2024-09-19T07:28:02.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.007+0000 7ff1c57fa640 1 -- 172.21.15.83:0/2084340478 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff1c001b020 con 0x7ff1c80c2530 2024-09-19T07:28:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.007+0000 7ff1c57fa640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1b40678c0 0x7ff1b4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.007+0000 7ff1ccdd8640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1b40678c0 0x7ff1b4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.007+0000 7ff1c57fa640 1 -- 172.21.15.83:0/2084340478 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7ff1c0086a80 con 0x7ff1c80c2530 2024-09-19T07:28:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.007+0000 7ff1ccdd8640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1b40678c0 0x7ff1b4069d80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8147e60 tx=0x7ff1b8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:02.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.011+0000 7ff1c57fa640 1 -- 172.21.15.83:0/2084340478 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff1c0022070 con 0x7ff1c80c2530 2024-09-19T07:28:02.163 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:28:02.165 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:28:02.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.211+0000 7ff1cf03c640 1 -- 172.21.15.83:0/2084340478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1c8073880 con 0x7ff1c80c2530 2024-09-19T07:28:02.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.211+0000 7ff1c57fa640 1 -- 172.21.15.83:0/2084340478 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+6532 (secure 0 0 0) 0x7ff1c0056ea0 con 0x7ff1c80c2530 2024-09-19T07:28:02.210 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:02.210 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":74,"btime":"2024-09-19T07:27:45:324647+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5991,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:02.210 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 74 2024-09-19T07:28:02.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.215+0000 7ff1cf03c640 1 -- 172.21.15.83:0/2084340478 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1b40678c0 msgr2=0x7ff1b4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:02.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.215+0000 7ff1cf03c640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1b40678c0 0x7ff1b4069d80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8147e60 tx=0x7ff1b8008040 comp rx=0 tx=0).stop 2024-09-19T07:28:02.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.215+0000 7ff1cf03c640 1 -- 172.21.15.83:0/2084340478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c80c2530 msgr2=0x7ff1c81471a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:02.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.215+0000 7ff1cf03c640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c80c2530 0x7ff1c81471a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000eed0 tx=0x7ff1c000b480 comp rx=0 tx=0).stop 2024-09-19T07:28:02.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.215+0000 7ff1cf03c640 1 -- 172.21.15.83:0/2084340478 shutdown_connections 2024-09-19T07:28:02.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.215+0000 7ff1cf03c640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1b40678c0 0x7ff1b4069d80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:02.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.215+0000 7ff1cf03c640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1c80c2530 0x7ff1c81471a0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:02.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.215+0000 7ff1cf03c640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1c80ab750 0x7ff1c8149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:02.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.215+0000 7ff1cf03c640 1 --2- 172.21.15.83:0/2084340478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1c80ab060 0x7ff1c81500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:02.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.215+0000 7ff1cf03c640 1 -- 172.21.15.83:0/2084340478 >> 172.21.15.83:0/2084340478 conn(0x7ff1c806bc80 msgr2=0x7ff1c80abd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:02.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.215+0000 7ff1cf03c640 1 -- 172.21.15.83:0/2084340478 shutdown_connections 2024-09-19T07:28:02.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:02.215+0000 7ff1cf03c640 1 -- 172.21.15.83:0/2084340478 wait complete. 2024-09-19T07:28:02.229 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:28:04.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff383d88640 1 Processor -- start 2024-09-19T07:28:04.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff383d88640 1 -- start start 2024-09-19T07:28:04.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff383d88640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff37c155d70 0x7ff37c0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:04.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff383d88640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff37c156200 con 0x7ff37c1592f0 2024-09-19T07:28:04.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff383d88640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff37c1564e0 con 0x7ff37c0c6120 2024-09-19T07:28:04.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff383d88640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff37c0c9930 con 0x7ff37c155d70 2024-09-19T07:28:04.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff382325640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff37c1592f0 0x7ff37c157ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54154/0 (socket says 172.21.15.83:54154) 2024-09-19T07:28:04.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff382325640 1 -- 172.21.15.83:0/2111957249 learned_addr learned my addr 172.21.15.83:0/2111957249 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:04.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff381323640 1 --2- 172.21.15.83:0/2111957249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff37c155d70 0x7ff37c0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:04.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3911267958 0 0) 0x7ff37c156200 con 0x7ff37c1592f0 2024-09-19T07:28:04.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff368003530 con 0x7ff37c1592f0 2024-09-19T07:28:04.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1681448517 0 0) 0x7ff368003530 con 0x7ff37c1592f0 2024-09-19T07:28:04.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff37c155d70 msgr2=0x7ff37c0c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:04.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 --2- 172.21.15.83:0/2111957249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff37c155d70 0x7ff37c0c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:04.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 >> v1:172.21.15.184:6789/0 conn(0x7ff37c0c6120 legacy=0x7ff37c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:04.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c1567c0 con 0x7ff37c1592f0 2024-09-19T07:28:04.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7ff374001a00 con 0x7ff37c1592f0 2024-09-19T07:28:04.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 >> v1:172.21.15.83:6789/0 conn(0x7ff37c1592f0 legacy=0x7ff37c157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:04.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 --2- 172.21.15.83:0/2111957249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3680039f0 0x7ff368003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:04.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 --2- 172.21.15.83:0/2111957249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff368004810 0x7ff368004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:04.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 --2- 172.21.15.83:0/2111957249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff368005190 0x7ff36800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:04.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff36800bf10 con 0x7ff368005190 2024-09-19T07:28:04.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff36800c090 con 0x7ff3680039f0 2024-09-19T07:28:04.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff36800c260 con 0x7ff368004810 2024-09-19T07:28:04.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff382325640 1 --2- 172.21.15.83:0/2111957249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff368005190 0x7ff36800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:04.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff381323640 1 --2- 172.21.15.83:0/2111957249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff368004810 0x7ff368004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:04.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff381b24640 1 --2- 172.21.15.83:0/2111957249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3680039f0 0x7ff368003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:04.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff382325640 1 -- 172.21.15.83:0/2111957249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff368004810 msgr2=0x7ff368004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:04.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff382325640 1 --2- 172.21.15.83:0/2111957249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff368004810 0x7ff368004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:04.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff382325640 1 -- 172.21.15.83:0/2111957249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3680039f0 msgr2=0x7ff368003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:04.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff382325640 1 --2- 172.21.15.83:0/2111957249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3680039f0 0x7ff368003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:04.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.363+0000 7ff382325640 1 -- 172.21.15.83:0/2111957249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff368003530 con 0x7ff368005190 2024-09-19T07:28:04.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff382325640 1 --2- 172.21.15.83:0/2111957249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff368005190 0x7ff36800b9d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff37400a860 tx=0x7ff37401fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0993babf608844 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:04.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37400a480 con 0x7ff368005190 2024-09-19T07:28:04.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3740aa710 con 0x7ff368005190 2024-09-19T07:28:04.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff380b22640 1 -- 172.21.15.83:0/2111957249 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3740aab40 con 0x7ff368005190 2024-09-19T07:28:04.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff383d88640 1 -- 172.21.15.83:0/2111957249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff368005190 msgr2=0x7ff36800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:04.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff383d88640 1 --2- 172.21.15.83:0/2111957249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff368005190 0x7ff36800b9d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff37400a860 tx=0x7ff37401fae0 comp rx=0 tx=0).stop 2024-09-19T07:28:04.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff382325640 1 -- 172.21.15.83:0/2111957249 reap_dead start 2024-09-19T07:28:04.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff383d88640 1 -- 172.21.15.83:0/2111957249 shutdown_connections 2024-09-19T07:28:04.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff383d88640 1 -- 172.21.15.83:0/2111957249 >> 172.21.15.83:0/2111957249 conn(0x7ff37c06bc80 msgr2=0x7ff37c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:04.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff383d88640 1 -- 172.21.15.83:0/2111957249 shutdown_connections 2024-09-19T07:28:04.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff383d88640 1 -- 172.21.15.83:0/2111957249 wait complete. 2024-09-19T07:28:04.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff383d88640 1 Processor -- start 2024-09-19T07:28:04.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.367+0000 7ff383d88640 1 -- start start 2024-09-19T07:28:04.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff383d88640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff37c0c4de0 0x7ff37c0ad260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:04.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff383d88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff37c155b80 0x7ff37c153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:04.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff383d88640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff37c0ab340 0x7ff37c154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:04.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff383d88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c153860 con 0x7ff37c155b80 2024-09-19T07:28:04.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff383d88640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c0cc630 con 0x7ff37c0c4de0 2024-09-19T07:28:04.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff383d88640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff37c0c2e80 con 0x7ff37c0ab340 2024-09-19T07:28:04.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff381323640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff37c155b80 0x7ff37c153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:04.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff381323640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff37c155b80 0x7ff37c153c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34224/0 (socket says 172.21.15.83:34224) 2024-09-19T07:28:04.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff381323640 1 -- 172.21.15.83:0/651855264 learned_addr learned my addr 172.21.15.83:0/651855264 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:04.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff381b24640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff37c0c4de0 0x7ff37c0ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff382325640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff37c0ab340 0x7ff37c154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff381323640 1 -- 172.21.15.83:0/651855264 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff37c0ab340 msgr2=0x7ff37c154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff381323640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff37c0ab340 0x7ff37c154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff381323640 1 -- 172.21.15.83:0/651855264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff37c0c4de0 msgr2=0x7ff37c0ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff381323640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff37c0c4de0 0x7ff37c0ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff381323640 1 -- 172.21.15.83:0/651855264 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c156060 con 0x7ff37c155b80 2024-09-19T07:28:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff381323640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff37c155b80 0x7ff37c153c00 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff36c0115e0 tx=0x7ff36c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff3727fc640 1 -- 172.21.15.83:0/651855264 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c00f8f0 con 0x7ff37c155b80 2024-09-19T07:28:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff3727fc640 1 -- 172.21.15.83:0/651855264 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff36c00fa90 con 0x7ff37c155b80 2024-09-19T07:28:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff383d88640 1 -- 172.21.15.83:0/651855264 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff37c0ac420 con 0x7ff37c155b80 2024-09-19T07:28:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff3727fc640 1 -- 172.21.15.83:0/651855264 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c00fe60 con 0x7ff37c155b80 2024-09-19T07:28:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff383d88640 1 -- 172.21.15.83:0/651855264 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff37c0ac9e0 con 0x7ff37c155b80 2024-09-19T07:28:04.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff383d88640 1 -- 172.21.15.83:0/651855264 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff37c1539e0 con 0x7ff37c155b80 2024-09-19T07:28:04.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff3727fc640 1 -- 172.21.15.83:0/651855264 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff36c015460 con 0x7ff37c155b80 2024-09-19T07:28:04.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff3727fc640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff3600678c0 0x7ff360069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:04.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff3727fc640 1 -- 172.21.15.83:0/651855264 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7ff36c086d80 con 0x7ff37c155b80 2024-09-19T07:28:04.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff381b24640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff3600678c0 0x7ff360069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:04.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.371+0000 7ff381b24640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff3600678c0 0x7ff360069d80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff378005490 tx=0x7ff378009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:04.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.379+0000 7ff3727fc640 1 -- 172.21.15.83:0/651855264 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff36c019030 con 0x7ff37c155b80 2024-09-19T07:28:04.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.579+0000 7ff383d88640 1 -- 172.21.15.83:0/651855264 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff37c0c6500 con 0x7ff37c155b80 2024-09-19T07:28:04.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.579+0000 7ff3727fc640 1 -- 172.21.15.83:0/651855264 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+6532 (secure 0 0 0) 0x7ff36c056b50 con 0x7ff37c155b80 2024-09-19T07:28:04.577 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:04.577 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":74,"btime":"2024-09-19T07:27:45:324647+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5991,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:04.577 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 74 2024-09-19T07:28:04.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.583+0000 7ff383d88640 1 -- 172.21.15.83:0/651855264 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff3600678c0 msgr2=0x7ff360069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:04.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.583+0000 7ff383d88640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff3600678c0 0x7ff360069d80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff378005490 tx=0x7ff378009040 comp rx=0 tx=0).stop 2024-09-19T07:28:04.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.583+0000 7ff383d88640 1 -- 172.21.15.83:0/651855264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff37c155b80 msgr2=0x7ff37c153c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:04.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.583+0000 7ff383d88640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff37c155b80 0x7ff37c153c00 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff36c0115e0 tx=0x7ff36c011aa0 comp rx=0 tx=0).stop 2024-09-19T07:28:04.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.583+0000 7ff383d88640 1 -- 172.21.15.83:0/651855264 shutdown_connections 2024-09-19T07:28:04.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.583+0000 7ff383d88640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff3600678c0 0x7ff360069d80 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:04.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.583+0000 7ff383d88640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff37c0ab340 0x7ff37c154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:04.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.583+0000 7ff383d88640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff37c155b80 0x7ff37c153c00 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:04.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.583+0000 7ff383d88640 1 --2- 172.21.15.83:0/651855264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff37c0c4de0 0x7ff37c0ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:04.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.583+0000 7ff383d88640 1 -- 172.21.15.83:0/651855264 >> 172.21.15.83:0/651855264 conn(0x7ff37c06bc80 msgr2=0x7ff37c150630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:04.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.583+0000 7ff383d88640 1 -- 172.21.15.83:0/651855264 shutdown_connections 2024-09-19T07:28:04.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:04.583+0000 7ff383d88640 1 -- 172.21.15.83:0/651855264 wait complete. 2024-09-19T07:28:04.595 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:28:06.596 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:06.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.723+0000 7ff63fcc8640 1 Processor -- start 2024-09-19T07:28:06.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.723+0000 7ff63fcc8640 1 -- start start 2024-09-19T07:28:06.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63fcc8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff638071100 0x7ff6381545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:06.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63fcc8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6380c28e0 0x7ff638150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:06.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63fcc8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6380a9e60 con 0x7ff638151480 2024-09-19T07:28:06.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63fcc8640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff638071b50 con 0x7ff638071100 2024-09-19T07:28:06.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63fcc8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff6380c1ff0 con 0x7ff6380c28e0 2024-09-19T07:28:06.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63e265640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff638151480 0x7ff63806dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54158/0 (socket says 172.21.15.83:54158) 2024-09-19T07:28:06.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63e265640 1 -- 172.21.15.83:0/3108933791 learned_addr learned my addr 172.21.15.83:0/3108933791 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:06.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63da64640 1 --2- 172.21.15.83:0/3108933791 >> v2:172.21.15.83:3300/0 conn(0x7ff638071100 0x7ff6381545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:06.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63d263640 1 --2- 172.21.15.83:0/3108933791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6380c28e0 0x7ff638150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:06.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63ca62640 1 -- 172.21.15.83:0/3108933791 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2880590837 0 0) 0x7ff6380a9e60 con 0x7ff638151480 2024-09-19T07:28:06.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63ca62640 1 -- 172.21.15.83:0/3108933791 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff624003530 con 0x7ff638151480 2024-09-19T07:28:06.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63da64640 1 -- 172.21.15.83:0/3108933791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6380c28e0 msgr2=0x7ff638150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:06.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63da64640 1 --2- 172.21.15.83:0/3108933791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6380c28e0 0x7ff638150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:06.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63da64640 1 -- 172.21.15.83:0/3108933791 >> v1:172.21.15.83:6789/0 conn(0x7ff638151480 legacy=0x7ff63806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:06.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63da64640 1 -- 172.21.15.83:0/3108933791 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6380aa140 con 0x7ff638071100 2024-09-19T07:28:06.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63da64640 1 --2- 172.21.15.83:0/3108933791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff638071100 0x7ff6381545e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff62c009d90 tx=0x7ff62c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94fc3a4422bc0805 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:06.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63ca62640 1 -- 172.21.15.83:0/3108933791 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c0b2070 con 0x7ff638071100 2024-09-19T07:28:06.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63ca62640 1 -- 172.21.15.83:0/3108933791 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff62c0a58b0 con 0x7ff638071100 2024-09-19T07:28:06.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.727+0000 7ff63ca62640 1 -- 172.21.15.83:0/3108933791 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c0a5c60 con 0x7ff638071100 2024-09-19T07:28:06.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 -- 172.21.15.83:0/3108933791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff638071100 msgr2=0x7ff6381545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:06.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 --2- 172.21.15.83:0/3108933791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff638071100 0x7ff6381545e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff62c009d90 tx=0x7ff62c01c8e0 comp rx=0 tx=0).stop 2024-09-19T07:28:06.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 -- 172.21.15.83:0/3108933791 shutdown_connections 2024-09-19T07:28:06.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 --2- 172.21.15.83:0/3108933791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6380c28e0 0x7ff638150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:06.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 --2- 172.21.15.83:0/3108933791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff638071100 0x7ff6381545e0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:06.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 -- 172.21.15.83:0/3108933791 >> 172.21.15.83:0/3108933791 conn(0x7ff63806bc80 msgr2=0x7ff63806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:06.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 -- 172.21.15.83:0/3108933791 shutdown_connections 2024-09-19T07:28:06.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 -- 172.21.15.83:0/3108933791 wait complete. 2024-09-19T07:28:06.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 Processor -- start 2024-09-19T07:28:06.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 -- start start 2024-09-19T07:28:06.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff638071100 0x7ff6381412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:06.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6380c28e0 0x7ff6381523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:06.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff638151480 0x7ff6381528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:06.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff638071660 con 0x7ff638071100 2024-09-19T07:28:06.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff6380714e0 con 0x7ff638151480 2024-09-19T07:28:06.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff638160c60 con 0x7ff6380c28e0 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63da64640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff638071100 0x7ff6381412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63da64640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff638071100 0x7ff6381412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34242/0 (socket says 172.21.15.83:34242) 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63da64640 1 -- 172.21.15.83:0/2001044303 learned_addr learned my addr 172.21.15.83:0/2001044303 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63d263640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6380c28e0 0x7ff6381523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63da64640 1 -- 172.21.15.83:0/2001044303 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6380c28e0 msgr2=0x7ff6381523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63da64640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6380c28e0 0x7ff6381523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63da64640 1 -- 172.21.15.83:0/2001044303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff638151480 msgr2=0x7ff6381528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63da64640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff638151480 0x7ff6381528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63da64640 1 -- 172.21.15.83:0/2001044303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff638153040 con 0x7ff638071100 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63da64640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff638071100 0x7ff6381412f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff62c01a510 tx=0x7ff62c019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff62a7fc640 1 -- 172.21.15.83:0/2001044303 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c0bd020 con 0x7ff638071100 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff62a7fc640 1 -- 172.21.15.83:0/2001044303 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff62c0b8470 con 0x7ff638071100 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 -- 172.21.15.83:0/2001044303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6381f0780 con 0x7ff638071100 2024-09-19T07:28:06.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff62a7fc640 1 -- 172.21.15.83:0/2001044303 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c0b2040 con 0x7ff638071100 2024-09-19T07:28:06.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.731+0000 7ff63fcc8640 1 -- 172.21.15.83:0/2001044303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6380c6280 con 0x7ff638071100 2024-09-19T07:28:06.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.735+0000 7ff63fcc8640 1 -- 172.21.15.83:0/2001044303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6380c5dd0 con 0x7ff638071100 2024-09-19T07:28:06.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.735+0000 7ff62a7fc640 1 -- 172.21.15.83:0/2001044303 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff62c0c0020 con 0x7ff638071100 2024-09-19T07:28:06.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.735+0000 7ff62a7fc640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff61c0678f0 0x7ff61c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:06.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.735+0000 7ff62a7fc640 1 -- 172.21.15.83:0/2001044303 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7669+0+0 (secure 0 0 0) 0x7ff62c120e30 con 0x7ff638071100 2024-09-19T07:28:06.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.735+0000 7ff63d263640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff61c0678f0 0x7ff61c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:06.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.735+0000 7ff63d263640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff61c0678f0 0x7ff61c069db0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff638147fd0 tx=0x7ff630008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:06.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.739+0000 7ff62a7fc640 1 -- 172.21.15.83:0/2001044303 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff62c0c7070 con 0x7ff638071100 2024-09-19T07:28:06.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.931+0000 7ff63fcc8640 1 -- 172.21.15.83:0/2001044303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6381545e0 con 0x7ff638071100 2024-09-19T07:28:06.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.931+0000 7ff62a7fc640 1 -- 172.21.15.83:0/2001044303 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+6532 (secure 0 0 0) 0x7ff62c016560 con 0x7ff638071100 2024-09-19T07:28:06.929 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:06.929 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":74,"btime":"2024-09-19T07:27:45:324647+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5991,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:27:21.202626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4979":{"gid":4979,"name":"b","rank":0,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.184:6835/872719174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":872719174},{"type":"v1","addr":"172.21.15.184:6835","nonce":872719174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4979,"qdb_cluster":[4979]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:06.930 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 74 2024-09-19T07:28:06.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.935+0000 7ff63fcc8640 1 -- 172.21.15.83:0/2001044303 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff61c0678f0 msgr2=0x7ff61c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:06.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.935+0000 7ff63fcc8640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff61c0678f0 0x7ff61c069db0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff638147fd0 tx=0x7ff630008040 comp rx=0 tx=0).stop 2024-09-19T07:28:06.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.935+0000 7ff63fcc8640 1 -- 172.21.15.83:0/2001044303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff638071100 msgr2=0x7ff6381412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:06.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.935+0000 7ff63fcc8640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff638071100 0x7ff6381412f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff62c01a510 tx=0x7ff62c019680 comp rx=0 tx=0).stop 2024-09-19T07:28:06.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.935+0000 7ff63fcc8640 1 -- 172.21.15.83:0/2001044303 shutdown_connections 2024-09-19T07:28:06.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.935+0000 7ff63fcc8640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff61c0678f0 0x7ff61c069db0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:06.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.935+0000 7ff63fcc8640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff638151480 0x7ff6381528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:06.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.935+0000 7ff63fcc8640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6380c28e0 0x7ff6381523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:06.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.935+0000 7ff63fcc8640 1 --2- 172.21.15.83:0/2001044303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff638071100 0x7ff6381412f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:06.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.935+0000 7ff63fcc8640 1 -- 172.21.15.83:0/2001044303 >> 172.21.15.83:0/2001044303 conn(0x7ff63806bc80 msgr2=0x7ff638073000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:06.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.935+0000 7ff63fcc8640 1 -- 172.21.15.83:0/2001044303 shutdown_connections 2024-09-19T07:28:06.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:06.935+0000 7ff63fcc8640 1 -- 172.21.15.83:0/2001044303 wait complete. 2024-09-19T07:28:06.947 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:28:08.948 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:09.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb011a8a640 1 Processor -- start 2024-09-19T07:28:09.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb011a8a640 1 -- start start 2024-09-19T07:28:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb011a8a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb00c150290 0x7fb00c0c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb011a8a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb00c152b20 0x7fb00c0c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb011a8a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb00c150670 con 0x7fb00c0c45c0 2024-09-19T07:28:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb011a8a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb00c0716c0 con 0x7fb00c152b20 2024-09-19T07:28:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb011a8a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fb00c0c3920 con 0x7fb00c150290 2024-09-19T07:28:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb00affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb00c152b20 0x7fb00c0c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb00affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb00c152b20 0x7fb00c0c4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34258/0 (socket says 172.21.15.83:34258) 2024-09-19T07:28:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb00affd640 1 -- 172.21.15.83:0/3758560546 learned_addr learned my addr 172.21.15.83:0/3758560546 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb00b7fe640 1 --2- 172.21.15.83:0/3758560546 >> v2:172.21.15.184:3300/0 conn(0x7fb00c150290 0x7fb00c0c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb00affd640 1 -- 172.21.15.83:0/3758560546 >> v2:172.21.15.184:3300/0 conn(0x7fb00c150290 msgr2=0x7fb00c0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb00affd640 1 --2- 172.21.15.83:0/3758560546 >> v2:172.21.15.184:3300/0 conn(0x7fb00c150290 0x7fb00c0c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb00affd640 1 -- 172.21.15.83:0/3758560546 >> v1:172.21.15.184:6789/0 conn(0x7fb00c0c45c0 legacy=0x7fb00c153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb00affd640 1 -- 172.21.15.83:0/3758560546 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00c0c2880 con 0x7fb00c152b20 2024-09-19T07:28:09.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb00affd640 1 --2- 172.21.15.83:0/3758560546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c152b20 0x7fb00c0c4080 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb00c06f590 tx=0x7faff401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495c78e9069b8033 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:09.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb00a7fc640 1 -- 172.21.15.83:0/3758560546 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff40ab5f0 con 0x7fb00c152b20 2024-09-19T07:28:09.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb00a7fc640 1 -- 172.21.15.83:0/3758560546 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faff40b0070 con 0x7fb00c152b20 2024-09-19T07:28:09.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.083+0000 7fb00a7fc640 1 -- 172.21.15.83:0/3758560546 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff40ab9b0 con 0x7fb00c152b20 2024-09-19T07:28:09.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 -- 172.21.15.83:0/3758560546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c152b20 msgr2=0x7fb00c0c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:09.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 --2- 172.21.15.83:0/3758560546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c152b20 0x7fb00c0c4080 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb00c06f590 tx=0x7faff401c4e0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 -- 172.21.15.83:0/3758560546 shutdown_connections 2024-09-19T07:28:09.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 --2- 172.21.15.83:0/3758560546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c152b20 0x7fb00c0c4080 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 --2- 172.21.15.83:0/3758560546 >> v2:172.21.15.184:3300/0 conn(0x7fb00c150290 0x7fb00c0c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 -- 172.21.15.83:0/3758560546 >> 172.21.15.83:0/3758560546 conn(0x7fb00c06bc80 msgr2=0x7fb00c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:09.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 -- 172.21.15.83:0/3758560546 shutdown_connections 2024-09-19T07:28:09.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 -- 172.21.15.83:0/3758560546 wait complete. 2024-09-19T07:28:09.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 Processor -- start 2024-09-19T07:28:09.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 -- start start 2024-09-19T07:28:09.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb00c0c45c0 0x7fb00c14c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c150290 0x7fb00c0721d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb00c152b20 0x7fb00c072710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c0c2530 con 0x7fb00c150290 2024-09-19T07:28:09.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c0c3120 con 0x7fb00c0c45c0 2024-09-19T07:28:09.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb00c15fb60 con 0x7fb00c152b20 2024-09-19T07:28:09.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb00affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c150290 0x7fb00c0721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb00affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c150290 0x7fb00c0721d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34274/0 (socket says 172.21.15.83:34274) 2024-09-19T07:28:09.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb00affd640 1 -- 172.21.15.83:0/3852192219 learned_addr learned my addr 172.21.15.83:0/3852192219 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:09.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb00b7fe640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb00c0c45c0 0x7fb00c14c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb00bfff640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb00c152b20 0x7fb00c072710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb00affd640 1 -- 172.21.15.83:0/3852192219 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb00c152b20 msgr2=0x7fb00c072710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:09.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb00affd640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb00c152b20 0x7fb00c072710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb00affd640 1 -- 172.21.15.83:0/3852192219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb00c0c45c0 msgr2=0x7fb00c14c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:09.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb00affd640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb00c0c45c0 0x7fb00c14c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb00affd640 1 -- 172.21.15.83:0/3852192219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00c157240 con 0x7fb00c150290 2024-09-19T07:28:09.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb00affd640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c150290 0x7fb00c0721d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7faff401c9c0 tx=0x7faff401ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:09.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb003fff640 1 -- 172.21.15.83:0/3852192219 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff40be5f0 con 0x7fb00c150290 2024-09-19T07:28:09.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb003fff640 1 -- 172.21.15.83:0/3852192219 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faff40be790 con 0x7fb00c150290 2024-09-19T07:28:09.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 -- 172.21.15.83:0/3852192219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb00c157520 con 0x7fb00c150290 2024-09-19T07:28:09.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb003fff640 1 -- 172.21.15.83:0/3852192219 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff40beb80 con 0x7fb00c150290 2024-09-19T07:28:09.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.087+0000 7fb011a8a640 1 -- 172.21.15.83:0/3852192219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb00c14aeb0 con 0x7fb00c150290 2024-09-19T07:28:09.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.091+0000 7fb011a8a640 1 -- 172.21.15.83:0/3852192219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb00c0c1ff0 con 0x7fb00c150290 2024-09-19T07:28:09.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.091+0000 7fb003fff640 1 -- 172.21.15.83:0/3852192219 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faff40b43f0 con 0x7fb00c150290 2024-09-19T07:28:09.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.091+0000 7fb003fff640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fafec0678c0 0x7fafec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.091+0000 7fb003fff640 1 -- 172.21.15.83:0/3852192219 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7755+0+0 (secure 0 0 0) 0x7faff4120ce0 con 0x7fb00c150290 2024-09-19T07:28:09.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.091+0000 7fb00b7fe640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fafec0678c0 0x7fafec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.091+0000 7fb00b7fe640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fafec0678c0 0x7fafec069d80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faffc003fd0 tx=0x7faffc0039a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:09.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.095+0000 7fb003fff640 1 -- 172.21.15.83:0/3852192219 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7faff40160d0 con 0x7fb00c150290 2024-09-19T07:28:09.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.299+0000 7fb011a8a640 1 -- 172.21.15.83:0/3852192219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb00c0c3e90 con 0x7fb00c150290 2024-09-19T07:28:09.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.299+0000 7fb003fff640 1 -- 172.21.15.83:0/3852192219 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5702 (secure 0 0 0) 0x7faff40abe20 con 0x7fb00c150290 2024-09-19T07:28:09.296 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:09.296 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":76,"btime":"2024-09-19T07:28:09:143001+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:08.331475+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:09.297 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 76 2024-09-19T07:28:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.303+0000 7fb011a8a640 1 -- 172.21.15.83:0/3852192219 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fafec0678c0 msgr2=0x7fafec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.303+0000 7fb011a8a640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fafec0678c0 0x7fafec069d80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faffc003fd0 tx=0x7faffc0039a0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.303+0000 7fb011a8a640 1 -- 172.21.15.83:0/3852192219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c150290 msgr2=0x7fb00c0721d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.303+0000 7fb011a8a640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c150290 0x7fb00c0721d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7faff401c9c0 tx=0x7faff401ad20 comp rx=0 tx=0).stop 2024-09-19T07:28:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.303+0000 7fb011a8a640 1 -- 172.21.15.83:0/3852192219 shutdown_connections 2024-09-19T07:28:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.303+0000 7fb011a8a640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fafec0678c0 0x7fafec069d80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.303+0000 7fb011a8a640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb00c152b20 0x7fb00c072710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.303+0000 7fb011a8a640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c150290 0x7fb00c0721d0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.303+0000 7fb011a8a640 1 --2- 172.21.15.83:0/3852192219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb00c0c45c0 0x7fb00c14c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.303+0000 7fb011a8a640 1 -- 172.21.15.83:0/3852192219 >> 172.21.15.83:0/3852192219 conn(0x7fb00c06bc80 msgr2=0x7fb00c153ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.303+0000 7fb011a8a640 1 -- 172.21.15.83:0/3852192219 shutdown_connections 2024-09-19T07:28:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.303+0000 7fb011a8a640 1 -- 172.21.15.83:0/3852192219 wait complete. 2024-09-19T07:28:09.315 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-09-19T07:28:09.315 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:28:09.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:09.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a3325b640 1 Processor -- start 2024-09-19T07:28:09.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a3325b640 1 -- start start 2024-09-19T07:28:09.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a3325b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a2c1545e0 0x7f4a2c06e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a3325b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a2c154a70 con 0x7f4a2c071100 2024-09-19T07:28:09.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a3325b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a2c0a9e60 con 0x7f4a2c0c28e0 2024-09-19T07:28:09.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a3325b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4a2c071b50 con 0x7f4a2c1545e0 2024-09-19T07:28:09.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a30ff7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4a2c071100 0x7f4a2c150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54160/0 (socket says 172.21.15.83:54160) 2024-09-19T07:28:09.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a30ff7640 1 -- 172.21.15.83:0/484059662 learned_addr learned my addr 172.21.15.83:0/484059662 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:09.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a317f8640 1 --2- 172.21.15.83:0/484059662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a2c1545e0 0x7f4a2c06e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1552230446 0 0) 0x7f4a2c154a70 con 0x7f4a2c071100 2024-09-19T07:28:09.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a1c003530 con 0x7f4a2c071100 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1788483980 0 0) 0x7f4a1c003530 con 0x7f4a2c071100 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a2c1545e0 msgr2=0x7f4a2c06e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 --2- 172.21.15.83:0/484059662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a2c1545e0 0x7f4a2c06e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 >> v1:172.21.15.184:6789/0 conn(0x7f4a2c0c28e0 legacy=0x7f4a2c06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2c0aa140 con 0x7f4a2c071100 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f4a18001680 con 0x7f4a2c071100 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 >> v1:172.21.15.83:6789/0 conn(0x7f4a2c071100 legacy=0x7f4a2c150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 --2- 172.21.15.83:0/484059662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a1c0039f0 0x7f4a1c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 --2- 172.21.15.83:0/484059662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a1c004810 0x7f4a1c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 --2- 172.21.15.83:0/484059662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a2c1545e0 0x7f4a1c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c071b50 con 0x7f4a1c0039f0 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c00bb30 con 0x7f4a1c004810 2024-09-19T07:28:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4a1c00bd30 con 0x7f4a2c1545e0 2024-09-19T07:28:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a30ff7640 1 --2- 172.21.15.83:0/484059662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a1c0039f0 0x7f4a1c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a317f8640 1 --2- 172.21.15.83:0/484059662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a2c1545e0 0x7f4a1c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a30ff7640 1 -- 172.21.15.83:0/484059662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a2c1545e0 msgr2=0x7f4a1c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a30ff7640 1 --2- 172.21.15.83:0/484059662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a2c1545e0 0x7f4a1c005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a2bfff640 1 --2- 172.21.15.83:0/484059662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a1c004810 0x7f4a1c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a30ff7640 1 -- 172.21.15.83:0/484059662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a1c004810 msgr2=0x7f4a1c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a30ff7640 1 --2- 172.21.15.83:0/484059662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a1c004810 0x7f4a1c004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.443+0000 7f4a30ff7640 1 -- 172.21.15.83:0/484059662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1c003530 con 0x7f4a1c0039f0 2024-09-19T07:28:09.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a30ff7640 1 --2- 172.21.15.83:0/484059662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a1c0039f0 0x7f4a1c003e10 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800c1c0 tx=0x7f4a1801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6de290f2b3de7d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:09.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a18007d00 con 0x7f4a1c0039f0 2024-09-19T07:28:09.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a180a9020 con 0x7f4a1c0039f0 2024-09-19T07:28:09.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a2b7fe640 1 -- 172.21.15.83:0/484059662 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a18005b00 con 0x7f4a1c0039f0 2024-09-19T07:28:09.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 -- 172.21.15.83:0/484059662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a1c0039f0 msgr2=0x7f4a1c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:09.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 --2- 172.21.15.83:0/484059662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a1c0039f0 0x7f4a1c003e10 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800c1c0 tx=0x7f4a1801cc60 comp rx=0 tx=0).stop 2024-09-19T07:28:09.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a317f8640 1 -- 172.21.15.83:0/484059662 reap_dead start 2024-09-19T07:28:09.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 -- 172.21.15.83:0/484059662 shutdown_connections 2024-09-19T07:28:09.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 -- 172.21.15.83:0/484059662 >> 172.21.15.83:0/484059662 conn(0x7f4a2c06bc80 msgr2=0x7f4a2c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:09.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 -- 172.21.15.83:0/484059662 shutdown_connections 2024-09-19T07:28:09.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 -- 172.21.15.83:0/484059662 wait complete. 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 Processor -- start 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 -- start start 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a1c0039f0 0x7f4a2c0c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a2c0c1990 0x7f4a2c14c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a2c0ac3b0 0x7f4a2c14cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c06eb40 con 0x7f4a2c0c1990 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c06e9c0 con 0x7f4a2c0ac3b0 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a3325b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4a2c0714e0 con 0x7f4a1c0039f0 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a2bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a2c0c1990 0x7f4a2c14c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a2bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a2c0c1990 0x7f4a2c14c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34288/0 (socket says 172.21.15.83:34288) 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a2bfff640 1 -- 172.21.15.83:0/3436645831 learned_addr learned my addr 172.21.15.83:0/3436645831 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a30ff7640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a1c0039f0 0x7f4a2c0c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a317f8640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a2c0ac3b0 0x7f4a2c14cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a2bfff640 1 -- 172.21.15.83:0/3436645831 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a1c0039f0 msgr2=0x7f4a2c0c0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a2bfff640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a1c0039f0 0x7f4a2c0c0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a2bfff640 1 -- 172.21.15.83:0/3436645831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a2c0ac3b0 msgr2=0x7f4a2c14cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a2bfff640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a2c0ac3b0 0x7f4a2c14cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a2bfff640 1 -- 172.21.15.83:0/3436645831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2c0ab1b0 con 0x7f4a2c0c1990 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a30ff7640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a1c0039f0 0x7f4a2c0c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a317f8640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a2c0ac3b0 0x7f4a2c14cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.447+0000 7f4a2bfff640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a2c0c1990 0x7f4a2c14c660 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000d4c0 tx=0x7f4a2000d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.451+0000 7f4a297fa640 1 -- 172.21.15.83:0/3436645831 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a20005480 con 0x7f4a2c0c1990 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.451+0000 7f4a297fa640 1 -- 172.21.15.83:0/3436645831 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a20017070 con 0x7f4a2c0c1990 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.451+0000 7f4a3325b640 1 -- 172.21.15.83:0/3436645831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2c0ab490 con 0x7f4a2c0c1990 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.451+0000 7f4a297fa640 1 -- 172.21.15.83:0/3436645831 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a20003840 con 0x7f4a2c0c1990 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.451+0000 7f4a3325b640 1 -- 172.21.15.83:0/3436645831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a2c15bd70 con 0x7f4a2c0c1990 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.451+0000 7f4a3325b640 1 -- 172.21.15.83:0/3436645831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a2c0ab840 con 0x7f4a2c0c1990 2024-09-19T07:28:09.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.451+0000 7f4a297fa640 1 -- 172.21.15.83:0/3436645831 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4a200039e0 con 0x7f4a2c0c1990 2024-09-19T07:28:09.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.451+0000 7f4a297fa640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4a100678f0 0x7f4a10069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:09.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.451+0000 7f4a297fa640 1 -- 172.21.15.83:0/3436645831 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7755+0+0 (secure 0 0 0) 0x7f4a200861d0 con 0x7f4a2c0c1990 2024-09-19T07:28:09.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.451+0000 7f4a30ff7640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4a100678f0 0x7f4a10069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:09.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.451+0000 7f4a30ff7640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4a100678f0 0x7f4a10069db0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4a180aa000 tx=0x7f4a2c072fc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:09.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.455+0000 7f4a297fa640 1 -- 172.21.15.83:0/3436645831 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4a2000f2f0 con 0x7f4a2c0c1990 2024-09-19T07:28:09.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.655+0000 7f4a3325b640 1 -- 172.21.15.83:0/3436645831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a2c150f40 con 0x7f4a2c0c1990 2024-09-19T07:28:09.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.655+0000 7f4a297fa640 1 -- 172.21.15.83:0/3436645831 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5702 (secure 0 0 0) 0x7f4a200565a0 con 0x7f4a2c0c1990 2024-09-19T07:28:09.652 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:09.653 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":76,"btime":"2024-09-19T07:28:09:143001+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:08.331475+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:09.653 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 76 2024-09-19T07:28:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.659+0000 7f4a3325b640 1 -- 172.21.15.83:0/3436645831 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4a100678f0 msgr2=0x7f4a10069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.659+0000 7f4a3325b640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4a100678f0 0x7f4a10069db0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4a180aa000 tx=0x7f4a2c072fc0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.659+0000 7f4a3325b640 1 -- 172.21.15.83:0/3436645831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a2c0c1990 msgr2=0x7f4a2c14c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.659+0000 7f4a3325b640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a2c0c1990 0x7f4a2c14c660 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000d4c0 tx=0x7f4a2000d980 comp rx=0 tx=0).stop 2024-09-19T07:28:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.659+0000 7f4a3325b640 1 -- 172.21.15.83:0/3436645831 shutdown_connections 2024-09-19T07:28:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.659+0000 7f4a3325b640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4a100678f0 0x7f4a10069db0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.659+0000 7f4a3325b640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a2c0ac3b0 0x7f4a2c14cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.659+0000 7f4a3325b640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a2c0c1990 0x7f4a2c14c660 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.659+0000 7f4a3325b640 1 --2- 172.21.15.83:0/3436645831 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a1c0039f0 0x7f4a2c0c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.659+0000 7f4a3325b640 1 -- 172.21.15.83:0/3436645831 >> 172.21.15.83:0/3436645831 conn(0x7f4a2c06bc80 msgr2=0x7f4a2c14f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:09.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.659+0000 7f4a3325b640 1 -- 172.21.15.83:0/3436645831 shutdown_connections 2024-09-19T07:28:09.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:09.659+0000 7f4a3325b640 1 -- 172.21.15.83:0/3436645831 wait complete. 2024-09-19T07:28:09.670 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:28:09:143001+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 76, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 76, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5991": { "addr": "172.21.15.83:6837/1229829432", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 1229829432, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 1229829432, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5991, "incarnation": 75, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:28:08.331475+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5991 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 73, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5934": { "addr": "172.21.15.83:6835/104517500", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 104517500, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 104517500, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5934, "incarnation": 69, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:27:42.299316+0000", "qdb_cluster": [ 5934 ], "qdb_leader": 5934, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5934 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 28, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:28:11.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:11.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.815+0000 7f50c22e4640 1 Processor -- start 2024-09-19T07:28:11.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.815+0000 7f50c22e4640 1 -- start start 2024-09-19T07:28:11.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50c22e4640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f50bc0c43c0 0x7f50bc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:11.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50c22e4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50bc1570c0 0x7f50bc154760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:11.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50c22e4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50bc153b50 con 0x7f50bc073830 2024-09-19T07:28:11.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50c22e4640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f50bc1575c0 con 0x7f50bc0c43c0 2024-09-19T07:28:11.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50c22e4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f50bc1578a0 con 0x7f50bc1570c0 2024-09-19T07:28:11.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50bbfff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f50bc0c43c0 0x7f50bc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:11.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50c0881640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50bc1570c0 0x7f50bc154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:11.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50bbfff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f50bc0c43c0 0x7f50bc06cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:60930/0 (socket says 172.21.15.83:60930) 2024-09-19T07:28:11.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50bb7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f50bc073830 0x7f50bc154050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:47052/0 (socket says 172.21.15.83:47052) 2024-09-19T07:28:11.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50bbfff640 1 -- 172.21.15.83:0/1129528706 learned_addr learned my addr 172.21.15.83:0/1129528706 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:11.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50bbfff640 1 -- 172.21.15.83:0/1129528706 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50bc1570c0 msgr2=0x7f50bc154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:11.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50bbfff640 1 --2- 172.21.15.83:0/1129528706 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50bc1570c0 0x7f50bc154760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:11.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50bbfff640 1 -- 172.21.15.83:0/1129528706 >> v1:172.21.15.184:6789/0 conn(0x7f50bc073830 legacy=0x7f50bc154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:11.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50bbfff640 1 -- 172.21.15.83:0/1129528706 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50bc157b80 con 0x7f50bc0c43c0 2024-09-19T07:28:11.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.819+0000 7f50baffd640 1 -- 172.21.15.83:0/1129528706 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 752579969 0 0) 0x7f50bc153b50 con 0x7f50bc073830 2024-09-19T07:28:11.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50bbfff640 1 --2- 172.21.15.83:0/1129528706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc0c43c0 0x7f50bc06cc90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009870 tx=0x7f50ac01c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d7d5c780c8f78c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:11.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50baffd640 1 -- 172.21.15.83:0/1129528706 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac0b1070 con 0x7f50bc0c43c0 2024-09-19T07:28:11.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50baffd640 1 -- 172.21.15.83:0/1129528706 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50ac0ac600 con 0x7f50bc0c43c0 2024-09-19T07:28:11.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50baffd640 1 -- 172.21.15.83:0/1129528706 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac0ac9d0 con 0x7f50bc0c43c0 2024-09-19T07:28:11.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 -- 172.21.15.83:0/1129528706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc0c43c0 msgr2=0x7f50bc06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:11.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 --2- 172.21.15.83:0/1129528706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc0c43c0 0x7f50bc06cc90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009870 tx=0x7f50ac01c4d0 comp rx=0 tx=0).stop 2024-09-19T07:28:11.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 -- 172.21.15.83:0/1129528706 shutdown_connections 2024-09-19T07:28:11.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 --2- 172.21.15.83:0/1129528706 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50bc1570c0 0x7f50bc154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:11.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 --2- 172.21.15.83:0/1129528706 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc0c43c0 0x7f50bc06cc90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:11.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 -- 172.21.15.83:0/1129528706 >> 172.21.15.83:0/1129528706 conn(0x7f50bc0c5060 msgr2=0x7f50bc0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:11.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 -- 172.21.15.83:0/1129528706 shutdown_connections 2024-09-19T07:28:11.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 -- 172.21.15.83:0/1129528706 wait complete. 2024-09-19T07:28:11.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 Processor -- start 2024-09-19T07:28:11.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 -- start start 2024-09-19T07:28:11.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50bc073830 0x7f50bc15c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:11.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc0c43c0 0x7f50bc0c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:11.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50bc1570c0 0x7f50bc0c2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:11.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc160c10 con 0x7f50bc1570c0 2024-09-19T07:28:11.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc070680 con 0x7f50bc0c43c0 2024-09-19T07:28:11.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c22e4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f50bc160d90 con 0x7f50bc073830 2024-09-19T07:28:11.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c0881640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50bc1570c0 0x7f50bc0c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:11.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c0881640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50bc1570c0 0x7f50bc0c2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50718/0 (socket says 172.21.15.83:50718) 2024-09-19T07:28:11.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50c0881640 1 -- 172.21.15.83:0/2822777749 learned_addr learned my addr 172.21.15.83:0/2822777749 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:11.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50bbfff640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50bc073830 0x7f50bc15c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:11.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.823+0000 7f50bb7fe640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc0c43c0 0x7f50bc0c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:11.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50c0881640 1 -- 172.21.15.83:0/2822777749 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50bc073830 msgr2=0x7f50bc15c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:11.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50c0881640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50bc073830 0x7f50bc15c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:11.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50c0881640 1 -- 172.21.15.83:0/2822777749 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc0c43c0 msgr2=0x7f50bc0c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:11.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50c0881640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc0c43c0 0x7f50bc0c1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:11.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50c0881640 1 -- 172.21.15.83:0/2822777749 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50bc0c2a60 con 0x7f50bc1570c0 2024-09-19T07:28:11.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50c0881640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50bc1570c0 0x7f50bc0c2520 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f50b4007d80 tx=0x7f50b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:11.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50b8ff9640 1 -- 172.21.15.83:0/2822777749 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b4014420 con 0x7f50bc1570c0 2024-09-19T07:28:11.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50b8ff9640 1 -- 172.21.15.83:0/2822777749 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50b4019070 con 0x7f50bc1570c0 2024-09-19T07:28:11.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50c22e4640 1 -- 172.21.15.83:0/2822777749 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50bc1efe50 con 0x7f50bc1570c0 2024-09-19T07:28:11.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50b8ff9640 1 -- 172.21.15.83:0/2822777749 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b4014800 con 0x7f50bc1570c0 2024-09-19T07:28:11.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50c22e4640 1 -- 172.21.15.83:0/2822777749 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50bc0be310 con 0x7f50bc1570c0 2024-09-19T07:28:11.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50c22e4640 1 -- 172.21.15.83:0/2822777749 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50bc06cc90 con 0x7f50bc1570c0 2024-09-19T07:28:11.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50b8ff9640 1 -- 172.21.15.83:0/2822777749 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f50b40149a0 con 0x7f50bc1570c0 2024-09-19T07:28:11.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50b8ff9640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50a80678c0 0x7f50a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:11.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50b8ff9640 1 -- 172.21.15.83:0/2822777749 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7755+0+0 (secure 0 0 0) 0x7f50b4086a90 con 0x7f50bc1570c0 2024-09-19T07:28:11.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50bbfff640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50a80678c0 0x7f50a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:11.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.827+0000 7f50bbfff640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50a80678c0 0x7f50a8069d80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0ae000 tx=0x7f50ac0a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:11.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:11.831+0000 7f50b8ff9640 1 -- 172.21.15.83:0/2822777749 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f50b4056810 con 0x7f50bc1570c0 2024-09-19T07:28:12.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.079+0000 7f50c22e4640 1 -- 172.21.15.83:0/2822777749 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50bc0c6d50 con 0x7f50bc1570c0 2024-09-19T07:28:12.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.079+0000 7f50b8ff9640 1 -- 172.21.15.83:0/2822777749 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) ==== 94+0+5706 (secure 0 0 0) 0x7f50b405ab10 con 0x7f50bc1570c0 2024-09-19T07:28:12.075 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:12.075 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":78,"btime":"2024-09-19T07:28:11:164141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 78 2024-09-19T07:28:12.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.079+0000 7f50c22e4640 1 -- 172.21.15.83:0/2822777749 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50a80678c0 msgr2=0x7f50a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:12.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.079+0000 7f50c22e4640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50a80678c0 0x7f50a8069d80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0ae000 tx=0x7f50ac0a7000 comp rx=0 tx=0).stop 2024-09-19T07:28:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.079+0000 7f50c22e4640 1 -- 172.21.15.83:0/2822777749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50bc1570c0 msgr2=0x7f50bc0c2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.079+0000 7f50c22e4640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50bc1570c0 0x7f50bc0c2520 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f50b4007d80 tx=0x7f50b400e4b0 comp rx=0 tx=0).stop 2024-09-19T07:28:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.079+0000 7f50c22e4640 1 -- 172.21.15.83:0/2822777749 shutdown_connections 2024-09-19T07:28:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.079+0000 7f50c22e4640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50a80678c0 0x7f50a8069d80 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.079+0000 7f50c22e4640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50bc1570c0 0x7f50bc0c2520 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.079+0000 7f50c22e4640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50bc0c43c0 0x7f50bc0c1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.079+0000 7f50c22e4640 1 --2- 172.21.15.83:0/2822777749 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50bc073830 0x7f50bc15c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.079+0000 7f50c22e4640 1 -- 172.21.15.83:0/2822777749 >> 172.21.15.83:0/2822777749 conn(0x7f50bc0c5060 msgr2=0x7f50bc155fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.083+0000 7f50c22e4640 1 -- 172.21.15.83:0/2822777749 shutdown_connections 2024-09-19T07:28:12.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:12.083+0000 7f50c22e4640 1 -- 172.21.15.83:0/2822777749 wait complete. 2024-09-19T07:28:12.095 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=4979 2024-09-19T07:28:12.096 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.b 2024-09-19T07:28:15.097 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-09-19T07:28:15.097 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-19T07:28:15.097 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-19T07:28:15.100 INFO:tasks.ceph.mds.b:Started 2024-09-19T07:28:15.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:15.173 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-19T07:28:15.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.255+0000 7f6a74310640 1 Processor -- start 2024-09-19T07:28:15.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.255+0000 7f6a74310640 1 -- start start 2024-09-19T07:28:15.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a74310640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6a6c06cef0 0x7f6a6c0c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:15.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a74310640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a6c070140 0x7f6a6c1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a74310640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a6c070830 con 0x7f6a6c154180 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a74310640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6a6c153cf0 con 0x7f6a6c06cef0 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a74310640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6a6c1539a0 con 0x7f6a6c070140 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a720ac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6a6c06cef0 0x7f6a6c0c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a728ad640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6a6c154180 0x7f6a6c150ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39088/0 (socket says 172.21.15.83:39088) 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a728ad640 1 -- 172.21.15.83:0/3061291291 learned_addr learned my addr 172.21.15.83:0/3061291291 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a718ab640 1 --2- 172.21.15.83:0/3061291291 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a6c070140 0x7f6a6c1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a720ac640 1 -- 172.21.15.83:0/3061291291 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a6c070140 msgr2=0x7f6a6c1579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a720ac640 1 --2- 172.21.15.83:0/3061291291 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a6c070140 0x7f6a6c1579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a720ac640 1 -- 172.21.15.83:0/3061291291 >> v1:172.21.15.83:6789/0 conn(0x7f6a6c154180 legacy=0x7f6a6c150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a720ac640 1 -- 172.21.15.83:0/3061291291 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6c06f2e0 con 0x7f6a6c06cef0 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.259+0000 7f6a710aa640 1 -- 172.21.15.83:0/3061291291 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1589844971 0 0) 0x7f6a6c070830 con 0x7f6a6c154180 2024-09-19T07:28:15.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a720ac640 1 --2- 172.21.15.83:0/3061291291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a6c06cef0 0x7f6a6c0c2f90 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6a680097c0 tx=0x7f6a6801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2112469fe96cadd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a710aa640 1 -- 172.21.15.83:0/3061291291 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a680b0070 con 0x7f6a6c06cef0 2024-09-19T07:28:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a710aa640 1 -- 172.21.15.83:0/3061291291 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a680ab600 con 0x7f6a6c06cef0 2024-09-19T07:28:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a710aa640 1 -- 172.21.15.83:0/3061291291 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a680ab9d0 con 0x7f6a6c06cef0 2024-09-19T07:28:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 -- 172.21.15.83:0/3061291291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a6c06cef0 msgr2=0x7f6a6c0c2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 --2- 172.21.15.83:0/3061291291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a6c06cef0 0x7f6a6c0c2f90 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6a680097c0 tx=0x7f6a6801c410 comp rx=0 tx=0).stop 2024-09-19T07:28:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 -- 172.21.15.83:0/3061291291 shutdown_connections 2024-09-19T07:28:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 --2- 172.21.15.83:0/3061291291 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a6c070140 0x7f6a6c1579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 --2- 172.21.15.83:0/3061291291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a6c06cef0 0x7f6a6c0c2f90 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 -- 172.21.15.83:0/3061291291 >> 172.21.15.83:0/3061291291 conn(0x7f6a6c0c5090 msgr2=0x7f6a6c0c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 -- 172.21.15.83:0/3061291291 shutdown_connections 2024-09-19T07:28:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 -- 172.21.15.83:0/3061291291 wait complete. 2024-09-19T07:28:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 Processor -- start 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 -- start start 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a6c06cef0 0x7f6a6c0c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a6c070140 0x7f6a6c13d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a6c154180 0x7f6a6c138760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6c06d450 con 0x7f6a6c070140 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6c06d2d0 con 0x7f6a6c154180 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a74310640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6a6c160b40 con 0x7f6a6c06cef0 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a718ab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a6c070140 0x7f6a6c13d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a718ab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a6c070140 0x7f6a6c13d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50738/0 (socket says 172.21.15.83:50738) 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a718ab640 1 -- 172.21.15.83:0/1482686529 learned_addr learned my addr 172.21.15.83:0/1482686529 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a720ac640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a6c06cef0 0x7f6a6c0c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a728ad640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a6c154180 0x7f6a6c138760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a718ab640 1 -- 172.21.15.83:0/1482686529 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a6c06cef0 msgr2=0x7f6a6c0c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a718ab640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a6c06cef0 0x7f6a6c0c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a718ab640 1 -- 172.21.15.83:0/1482686529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a6c154180 msgr2=0x7f6a6c138760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a718ab640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a6c154180 0x7f6a6c138760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a718ab640 1 -- 172.21.15.83:0/1482686529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6c149300 con 0x7f6a6c070140 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.263+0000 7f6a720ac640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a6c06cef0 0x7f6a6c0c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.267+0000 7f6a718ab640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a6c070140 0x7f6a6c13d0b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00e930 tx=0x7f6a5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.267+0000 7f6a62ffd640 1 -- 172.21.15.83:0/1482686529 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c017070 con 0x7f6a6c070140 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.267+0000 7f6a62ffd640 1 -- 172.21.15.83:0/1482686529 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a5c00ca00 con 0x7f6a6c070140 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.267+0000 7f6a62ffd640 1 -- 172.21.15.83:0/1482686529 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c00cdd0 con 0x7f6a6c070140 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.267+0000 7f6a74310640 1 -- 172.21.15.83:0/1482686529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a6c14a890 con 0x7f6a6c070140 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.267+0000 7f6a74310640 1 -- 172.21.15.83:0/1482686529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a6c149850 con 0x7f6a6c070140 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.267+0000 7f6a74310640 1 -- 172.21.15.83:0/1482686529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a6c0c2f90 con 0x7f6a6c070140 2024-09-19T07:28:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.267+0000 7f6a62ffd640 1 -- 172.21.15.83:0/1482686529 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6a5c01a070 con 0x7f6a6c070140 2024-09-19T07:28:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.267+0000 7f6a62ffd640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6a500678c0 0x7f6a50069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.267+0000 7f6a62ffd640 1 -- 172.21.15.83:0/1482686529 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7755+0+0 (secure 0 0 0) 0x7f6a5c086450 con 0x7f6a6c070140 2024-09-19T07:28:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.267+0000 7f6a720ac640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6a500678c0 0x7f6a50069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.267+0000 7f6a720ac640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6a500678c0 0x7f6a50069d80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6a680ad000 tx=0x7f6a680a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.271+0000 7f6a62ffd640 1 -- 172.21.15.83:0/1482686529 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6a5c0568e0 con 0x7f6a6c070140 2024-09-19T07:28:15.420 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:28:15.335+0000 7f8437757640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:28:15.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 -- 172.21.15.83:0/1482686529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a6c068900 con 0x7f6a6c070140 2024-09-19T07:28:15.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a62ffd640 1 -- 172.21.15.83:0/1482686529 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) ==== 94+0+5706 (secure 0 0 0) 0x7f6a5c0561e0 con 0x7f6a6c070140 2024-09-19T07:28:15.487 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:15.487 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":78,"btime":"2024-09-19T07:28:11:164141+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:15.487 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 78 2024-09-19T07:28:15.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 -- 172.21.15.83:0/1482686529 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6a500678c0 msgr2=0x7f6a50069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:15.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6a500678c0 0x7f6a50069d80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6a680ad000 tx=0x7f6a680a6000 comp rx=0 tx=0).stop 2024-09-19T07:28:15.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 -- 172.21.15.83:0/1482686529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a6c070140 msgr2=0x7f6a6c13d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:15.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a6c070140 0x7f6a6c13d0b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00e930 tx=0x7f6a5c00edf0 comp rx=0 tx=0).stop 2024-09-19T07:28:15.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 -- 172.21.15.83:0/1482686529 shutdown_connections 2024-09-19T07:28:15.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6a500678c0 0x7f6a50069d80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:15.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a6c154180 0x7f6a6c138760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:15.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a6c070140 0x7f6a6c13d0b0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 --2- 172.21.15.83:0/1482686529 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a6c06cef0 0x7f6a6c0c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 -- 172.21.15.83:0/1482686529 >> 172.21.15.83:0/1482686529 conn(0x7f6a6c0c5090 msgr2=0x7f6a6c0757b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 -- 172.21.15.83:0/1482686529 shutdown_connections 2024-09-19T07:28:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:15.491+0000 7f6a74310640 1 -- 172.21.15.83:0/1482686529 wait complete. 2024-09-19T07:28:15.507 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-19T07:28:17.508 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:17.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90cb05a640 1 Processor -- start 2024-09-19T07:28:17.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90cb05a640 1 -- start start 2024-09-19T07:28:17.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90cb05a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90c40ab610 0x7f90c40abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:17.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90cb05a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f90c40c6fc0 0x7f90c4157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:17.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90cb05a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90c40c7de0 con 0x7f90c406cef0 2024-09-19T07:28:17.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90cb05a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f90c40c6a90 con 0x7f90c40c6fc0 2024-09-19T07:28:17.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90cb05a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f90c40c8360 con 0x7f90c40ab610 2024-09-19T07:28:17.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90c95f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f90c40c6fc0 0x7f90c4157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:17.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90c8df6640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f90c406cef0 0x7f90c40c3970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39104/0 (socket says 172.21.15.83:39104) 2024-09-19T07:28:17.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90c8df6640 1 -- 172.21.15.83:0/3171673863 learned_addr learned my addr 172.21.15.83:0/3171673863 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:17.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90c3fff640 1 --2- 172.21.15.83:0/3171673863 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90c40ab610 0x7f90c40abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:17.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90c95f7640 1 -- 172.21.15.83:0/3171673863 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90c40ab610 msgr2=0x7f90c40abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:17.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90c95f7640 1 --2- 172.21.15.83:0/3171673863 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90c40ab610 0x7f90c40abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:17.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90c95f7640 1 -- 172.21.15.83:0/3171673863 >> v1:172.21.15.83:6789/0 conn(0x7f90c406cef0 legacy=0x7f90c40c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:17.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.639+0000 7f90c95f7640 1 -- 172.21.15.83:0/3171673863 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c40c8080 con 0x7f90c40c6fc0 2024-09-19T07:28:17.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90c95f7640 1 --2- 172.21.15.83:0/3171673863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90c40c6fc0 0x7f90c4157890 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009d90 tx=0x7f90bc01c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647987d9bba740ad server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:17.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90c37fe640 1 -- 172.21.15.83:0/3171673863 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc0b2070 con 0x7f90c40c6fc0 2024-09-19T07:28:17.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90c37fe640 1 -- 172.21.15.83:0/3171673863 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90bc0a5820 con 0x7f90c40c6fc0 2024-09-19T07:28:17.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90c37fe640 1 -- 172.21.15.83:0/3171673863 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc0a5bf0 con 0x7f90c40c6fc0 2024-09-19T07:28:17.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90cb05a640 1 -- 172.21.15.83:0/3171673863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90c40c6fc0 msgr2=0x7f90c4157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:17.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90cb05a640 1 --2- 172.21.15.83:0/3171673863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90c40c6fc0 0x7f90c4157890 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009d90 tx=0x7f90bc01c850 comp rx=0 tx=0).stop 2024-09-19T07:28:17.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90cb05a640 1 -- 172.21.15.83:0/3171673863 shutdown_connections 2024-09-19T07:28:17.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90cb05a640 1 --2- 172.21.15.83:0/3171673863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90c40c6fc0 0x7f90c4157890 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:17.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90cb05a640 1 --2- 172.21.15.83:0/3171673863 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90c40ab610 0x7f90c40abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:17.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90cb05a640 1 -- 172.21.15.83:0/3171673863 >> 172.21.15.83:0/3171673863 conn(0x7f90c40aa910 msgr2=0x7f90c40c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:17.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90cb05a640 1 -- 172.21.15.83:0/3171673863 shutdown_connections 2024-09-19T07:28:17.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90cb05a640 1 -- 172.21.15.83:0/3171673863 wait complete. 2024-09-19T07:28:17.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90cb05a640 1 Processor -- start 2024-09-19T07:28:17.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.643+0000 7f90cb05a640 1 -- start start 2024-09-19T07:28:17.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90cb05a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90c406cef0 0x7f90c41f0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:17.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90cb05a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90c40ab610 0x7f90c4150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90cb05a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90c40c6fc0 0x7f90c4150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90cb05a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f90c41581e0 con 0x7f90c406cef0 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90cb05a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90c406d520 con 0x7f90c40c6fc0 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90cb05a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f90c4160e20 con 0x7f90c40ab610 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c8df6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90c406cef0 0x7f90c41f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c8df6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90c406cef0 0x7f90c41f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50750/0 (socket says 172.21.15.83:50750) 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c8df6640 1 -- 172.21.15.83:0/4043421179 learned_addr learned my addr 172.21.15.83:0/4043421179 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c3fff640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90c40ab610 0x7f90c4150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c95f7640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90c40c6fc0 0x7f90c4150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c8df6640 1 -- 172.21.15.83:0/4043421179 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90c40ab610 msgr2=0x7f90c4150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c8df6640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90c40ab610 0x7f90c4150910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c8df6640 1 -- 172.21.15.83:0/4043421179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90c40c6fc0 msgr2=0x7f90c4150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c8df6640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90c40c6fc0 0x7f90c4150e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c8df6640 1 -- 172.21.15.83:0/4043421179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c40c5e10 con 0x7f90c406cef0 2024-09-19T07:28:17.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c8df6640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90c406cef0 0x7f90c41f0430 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f90b800c370 tx=0x7f90b800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:17.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c17fa640 1 -- 172.21.15.83:0/4043421179 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8017070 con 0x7f90c406cef0 2024-09-19T07:28:17.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c17fa640 1 -- 172.21.15.83:0/4043421179 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90b8005cf0 con 0x7f90c406cef0 2024-09-19T07:28:17.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90cb05a640 1 -- 172.21.15.83:0/4043421179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90c41f0e30 con 0x7f90c406cef0 2024-09-19T07:28:17.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c17fa640 1 -- 172.21.15.83:0/4043421179 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b80072f0 con 0x7f90c406cef0 2024-09-19T07:28:17.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90cb05a640 1 -- 172.21.15.83:0/4043421179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90c40c23b0 con 0x7f90c406cef0 2024-09-19T07:28:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90cb05a640 1 -- 172.21.15.83:0/4043421179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90c4150600 con 0x7f90c406cef0 2024-09-19T07:28:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c17fa640 1 -- 172.21.15.83:0/4043421179 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f90b801a070 con 0x7f90c406cef0 2024-09-19T07:28:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c17fa640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90a80678c0 0x7f90a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c17fa640 1 -- 172.21.15.83:0/4043421179 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7755+0+0 (secure 0 0 0) 0x7f90b8085760 con 0x7f90c406cef0 2024-09-19T07:28:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.647+0000 7f90c3fff640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90a80678c0 0x7f90a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.651+0000 7f90c3fff640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90a80678c0 0x7f90a8069d80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f90c40c1d60 tx=0x7f90b0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:17.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.651+0000 7f90c17fa640 1 -- 172.21.15.83:0/4043421179 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f90b8055b30 con 0x7f90c406cef0 2024-09-19T07:28:17.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.927+0000 7f90cb05a640 1 -- 172.21.15.83:0/4043421179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90c4152af0 con 0x7f90c406cef0 2024-09-19T07:28:17.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.927+0000 7f90c17fa640 1 -- 172.21.15.83:0/4043421179 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6534 (secure 0 0 0) 0x7f90b8055430 con 0x7f90c406cef0 2024-09-19T07:28:17.924 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:17.924 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":79,"btime":"2024-09-19T07:28:16:180103+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:17.925 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 79 2024-09-19T07:28:17.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.931+0000 7f90cb05a640 1 -- 172.21.15.83:0/4043421179 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90a80678c0 msgr2=0x7f90a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:17.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.931+0000 7f90cb05a640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90a80678c0 0x7f90a8069d80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f90c40c1d60 tx=0x7f90b0008040 comp rx=0 tx=0).stop 2024-09-19T07:28:17.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.931+0000 7f90cb05a640 1 -- 172.21.15.83:0/4043421179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90c406cef0 msgr2=0x7f90c41f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:17.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.931+0000 7f90cb05a640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90c406cef0 0x7f90c41f0430 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f90b800c370 tx=0x7f90b800c830 comp rx=0 tx=0).stop 2024-09-19T07:28:17.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.931+0000 7f90cb05a640 1 -- 172.21.15.83:0/4043421179 shutdown_connections 2024-09-19T07:28:17.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.931+0000 7f90cb05a640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90a80678c0 0x7f90a8069d80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:17.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.931+0000 7f90cb05a640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90c40c6fc0 0x7f90c4150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:17.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.931+0000 7f90cb05a640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90c40ab610 0x7f90c4150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:17.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.931+0000 7f90cb05a640 1 --2- 172.21.15.83:0/4043421179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90c406cef0 0x7f90c41f0430 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:17.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.931+0000 7f90cb05a640 1 -- 172.21.15.83:0/4043421179 >> 172.21.15.83:0/4043421179 conn(0x7f90c40aa910 msgr2=0x7f90c4074f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:17.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.931+0000 7f90cb05a640 1 -- 172.21.15.83:0/4043421179 shutdown_connections 2024-09-19T07:28:17.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:17.931+0000 7f90cb05a640 1 -- 172.21.15.83:0/4043421179 wait complete. 2024-09-19T07:28:17.943 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-09-19T07:28:17.944 INFO:tasks.mds_thrash.fs.[a]:waiting for 72 secs before thrashing 2024-09-19T07:28:24.339 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:24.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.487+0000 7f6d24e97640 1 Processor -- start 2024-09-19T07:28:24.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.487+0000 7f6d24e97640 1 -- start start 2024-09-19T07:28:24.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.487+0000 7f6d24e97640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6d200c27f0 0x7f6d201562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.487+0000 7f6d24e97640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d200c2c10 con 0x7f6d200c23d0 2024-09-19T07:28:24.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.487+0000 7f6d24e97640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d201593f0 con 0x7f6d20153300 2024-09-19T07:28:24.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.487+0000 7f6d24e97640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6d201591a0 con 0x7f6d200c27f0 2024-09-19T07:28:24.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.487+0000 7f6d1dd9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6d200c23d0 0x7f6d20155bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51314/0 (socket says 172.21.15.83:51314) 2024-09-19T07:28:24.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1dd9b640 1 -- 172.21.15.83:0/1180425691 learned_addr learned my addr 172.21.15.83:0/1180425691 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:24.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1ed9d640 1 --2- 172.21.15.83:0/1180425691 >> v2:172.21.15.184:3300/0 conn(0x7f6d200c27f0 0x7f6d201562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2122145345 0 0) 0x7f6d200c2c10 con 0x7f6d200c23d0 2024-09-19T07:28:24.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d08003530 con 0x7f6d200c23d0 2024-09-19T07:28:24.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2464076083 0 0) 0x7f6d08003530 con 0x7f6d200c23d0 2024-09-19T07:28:24.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 >> v2:172.21.15.184:3300/0 conn(0x7f6d200c27f0 msgr2=0x7f6d201562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:24.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 --2- 172.21.15.83:0/1180425691 >> v2:172.21.15.184:3300/0 conn(0x7f6d200c27f0 0x7f6d201562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 >> v1:172.21.15.184:6789/0 conn(0x7f6d20153300 legacy=0x7f6d2006cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:24.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d201596d0 con 0x7f6d200c23d0 2024-09-19T07:28:24.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f6d14001680 con 0x7f6d200c23d0 2024-09-19T07:28:24.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 >> v1:172.21.15.83:6789/0 conn(0x7f6d200c23d0 legacy=0x7f6d20155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:24.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 --2- 172.21.15.83:0/1180425691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d08003a10 0x7f6d08003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 --2- 172.21.15.83:0/1180425691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d08004810 0x7f6d08004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 --2- 172.21.15.83:0/1180425691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d080052f0 0x7f6d0800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201591a0 con 0x7f6d08004810 2024-09-19T07:28:24.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0800c1f0 con 0x7f6d080052f0 2024-09-19T07:28:24.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0800c3f0 con 0x7f6d08003a10 2024-09-19T07:28:24.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1dd9b640 1 --2- 172.21.15.83:0/1180425691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d08004810 0x7f6d08004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1dd9b640 1 -- 172.21.15.83:0/1180425691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d08003a10 msgr2=0x7f6d08003e30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:28:24.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1ed9d640 1 --2- 172.21.15.83:0/1180425691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d080052f0 0x7f6d0800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1dd9b640 1 --2- 172.21.15.83:0/1180425691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d08003a10 0x7f6d08003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1dd9b640 1 -- 172.21.15.83:0/1180425691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d080052f0 msgr2=0x7f6d0800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:24.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1dd9b640 1 --2- 172.21.15.83:0/1180425691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d080052f0 0x7f6d0800b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1dd9b640 1 -- 172.21.15.83:0/1180425691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d08003530 con 0x7f6d08004810 2024-09-19T07:28:24.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1ed9d640 1 -- 172.21.15.83:0/1180425691 reap_dead start 2024-09-19T07:28:24.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1dd9b640 1 --2- 172.21.15.83:0/1180425691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d08004810 0x7f6d08004c90 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6d1400c110 tx=0x7f6d1401cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7641c64e577f3fbd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:24.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d14007c50 con 0x7f6d08004810 2024-09-19T07:28:24.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d140a9070 con 0x7f6d08004810 2024-09-19T07:28:24.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.491+0000 7f6d1d59a640 1 -- 172.21.15.83:0/1180425691 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d14005d80 con 0x7f6d08004810 2024-09-19T07:28:24.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 -- 172.21.15.83:0/1180425691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d08004810 msgr2=0x7f6d08004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:24.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 --2- 172.21.15.83:0/1180425691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d08004810 0x7f6d08004c90 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6d1400c110 tx=0x7f6d1401cbc0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 -- 172.21.15.83:0/1180425691 shutdown_connections 2024-09-19T07:28:24.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 --2- 172.21.15.83:0/1180425691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d08004810 0x7f6d08004c90 secure :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6d1400c110 tx=0x7f6d1401cbc0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 -- 172.21.15.83:0/1180425691 >> 172.21.15.83:0/1180425691 conn(0x7f6d200c5090 msgr2=0x7f6d200c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:24.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 -- 172.21.15.83:0/1180425691 shutdown_connections 2024-09-19T07:28:24.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 -- 172.21.15.83:0/1180425691 wait complete. 2024-09-19T07:28:24.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 Processor -- start 2024-09-19T07:28:24.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 -- start start 2024-09-19T07:28:24.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d200ad780 0x7f6d200acad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d20156fa0 0x7f6d200adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d200ab3e0 0x7f6d200ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6d20153860 con 0x7f6d200ad780 2024-09-19T07:28:24.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201536e0 con 0x7f6d20156fa0 2024-09-19T07:28:24.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6d20155bb0 con 0x7f6d200ab3e0 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d1e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d200ad780 0x7f6d200acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d1e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d200ad780 0x7f6d200acad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43356/0 (socket says 172.21.15.83:43356) 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d1e59c640 1 -- 172.21.15.83:0/2713756175 learned_addr learned my addr 172.21.15.83:0/2713756175 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d1dd9b640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d20156fa0 0x7f6d200adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d1ed9d640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d200ab3e0 0x7f6d200ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d1e59c640 1 -- 172.21.15.83:0/2713756175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d200ab3e0 msgr2=0x7f6d200ab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d1e59c640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d200ab3e0 0x7f6d200ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d1e59c640 1 -- 172.21.15.83:0/2713756175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d20156fa0 msgr2=0x7f6d200adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d1e59c640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d20156fa0 0x7f6d200adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d1e59c640 1 -- 172.21.15.83:0/2713756175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d200c7c00 con 0x7f6d200ad780 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d1e59c640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d200ad780 0x7f6d200acad0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000c390 tx=0x7f6d1000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d0effd640 1 -- 172.21.15.83:0/2713756175 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d100052d0 con 0x7f6d200ad780 2024-09-19T07:28:24.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d0effd640 1 -- 172.21.15.83:0/2713756175 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d10005470 con 0x7f6d200ad780 2024-09-19T07:28:24.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d24e97640 1 -- 172.21.15.83:0/2713756175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d200c7ee0 con 0x7f6d200ad780 2024-09-19T07:28:24.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.495+0000 7f6d0effd640 1 -- 172.21.15.83:0/2713756175 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10007240 con 0x7f6d200ad780 2024-09-19T07:28:24.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.499+0000 7f6d24e97640 1 -- 172.21.15.83:0/2713756175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d200c6f70 con 0x7f6d200ad780 2024-09-19T07:28:24.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.499+0000 7f6d24e97640 1 -- 172.21.15.83:0/2713756175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d200c7170 con 0x7f6d200ad780 2024-09-19T07:28:24.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.499+0000 7f6d0effd640 1 -- 172.21.15.83:0/2713756175 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6d100073e0 con 0x7f6d200ad780 2024-09-19T07:28:24.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.499+0000 7f6d0effd640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6d000678c0 0x7f6d00069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.499+0000 7f6d0effd640 1 -- 172.21.15.83:0/2713756175 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7755+0+0 (secure 0 0 0) 0x7f6d10085860 con 0x7f6d200ad780 2024-09-19T07:28:24.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.499+0000 7f6d1dd9b640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6d000678c0 0x7f6d00069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.499+0000 7f6d1dd9b640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6d000678c0 0x7f6d00069d80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6d140ab000 tx=0x7f6d140b0040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:24.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.503+0000 7f6d0effd640 1 -- 172.21.15.83:0/2713756175 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6d200c7170 con 0x7f6d200ad780 2024-09-19T07:28:24.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.719+0000 7f6d24e97640 1 -- 172.21.15.83:0/2713756175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d20155f40 con 0x7f6d200ad780 2024-09-19T07:28:24.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.719+0000 7f6d0effd640 1 -- 172.21.15.83:0/2713756175 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6534 (secure 0 0 0) 0x7f6d10055c30 con 0x7f6d200ad780 2024-09-19T07:28:24.717 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:24.718 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":79,"btime":"2024-09-19T07:28:16:180103+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:24.718 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 79 2024-09-19T07:28:24.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.723+0000 7f6d24e97640 1 -- 172.21.15.83:0/2713756175 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6d000678c0 msgr2=0x7f6d00069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:24.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.723+0000 7f6d24e97640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6d000678c0 0x7f6d00069d80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6d140ab000 tx=0x7f6d140b0040 comp rx=0 tx=0).stop 2024-09-19T07:28:24.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.723+0000 7f6d24e97640 1 -- 172.21.15.83:0/2713756175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d200ad780 msgr2=0x7f6d200acad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:24.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.723+0000 7f6d24e97640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d200ad780 0x7f6d200acad0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000c390 tx=0x7f6d1000c850 comp rx=0 tx=0).stop 2024-09-19T07:28:24.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.723+0000 7f6d24e97640 1 -- 172.21.15.83:0/2713756175 shutdown_connections 2024-09-19T07:28:24.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.723+0000 7f6d24e97640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6d000678c0 0x7f6d00069d80 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.723+0000 7f6d24e97640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d200ab3e0 0x7f6d200ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.723+0000 7f6d24e97640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d20156fa0 0x7f6d200adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.723+0000 7f6d24e97640 1 --2- 172.21.15.83:0/2713756175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6d200ad780 0x7f6d200acad0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.723+0000 7f6d24e97640 1 -- 172.21.15.83:0/2713756175 >> 172.21.15.83:0/2713756175 conn(0x7f6d200c5090 msgr2=0x7f6d20071e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:24.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.723+0000 7f6d24e97640 1 -- 172.21.15.83:0/2713756175 shutdown_connections 2024-09-19T07:28:24.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.723+0000 7f6d24e97640 1 -- 172.21.15.83:0/2713756175 wait complete. 2024-09-19T07:28:24.737 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-19T07:28:24.737 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:28:24.737 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:28:24.748 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:28:24.748 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:24.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.915+0000 7f4deb0bf640 1 Processor -- start 2024-09-19T07:28:24.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.915+0000 7f4deb0bf640 1 -- start start 2024-09-19T07:28:24.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4deb0bf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de406ffe0 0x7f4de40c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4deb0bf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4de40c26c0 con 0x7f4de41535d0 2024-09-19T07:28:24.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4deb0bf640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4de40c29a0 con 0x7f4de4159af0 2024-09-19T07:28:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4deb0bf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4de40707e0 con 0x7f4de406ffe0 2024-09-19T07:28:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de8e5b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4de41535d0 0x7f4de4155c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51322/0 (socket says 172.21.15.83:51322) 2024-09-19T07:28:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de8e5b640 1 -- 172.21.15.83:0/1090235115 learned_addr learned my addr 172.21.15.83:0/1090235115 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de3fff640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de406ffe0 0x7f4de40c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1667500318 0 0) 0x7f4de40c26c0 con 0x7f4de41535d0 2024-09-19T07:28:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dd4003530 con 0x7f4de41535d0 2024-09-19T07:28:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2537878305 0 0) 0x7f4dd4003530 con 0x7f4de41535d0 2024-09-19T07:28:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de406ffe0 msgr2=0x7f4de40c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de406ffe0 0x7f4de40c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 >> v1:172.21.15.184:6789/0 conn(0x7f4de4159af0 legacy=0x7f4de40c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de41592f0 con 0x7f4de41535d0 2024-09-19T07:28:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f4dd8001680 con 0x7f4de41535d0 2024-09-19T07:28:24.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 >> v1:172.21.15.83:6789/0 conn(0x7f4de41535d0 legacy=0x7f4de4155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:24.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4dd40039f0 0x7f4dd4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4dd4004810 0x7f4dd4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de406ffe0 0x7f4dd400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd4005520 con 0x7f4dd40039f0 2024-09-19T07:28:24.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd400bf10 con 0x7f4dd4004810 2024-09-19T07:28:24.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4dd400c090 con 0x7f4de406ffe0 2024-09-19T07:28:24.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de8e5b640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4dd40039f0 0x7f4dd4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de3fff640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4dd4004810 0x7f4dd4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de965c640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de406ffe0 0x7f4dd400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de8e5b640 1 -- 172.21.15.83:0/1090235115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de406ffe0 msgr2=0x7f4dd400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:24.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de8e5b640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de406ffe0 0x7f4dd400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de8e5b640 1 -- 172.21.15.83:0/1090235115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4dd4004810 msgr2=0x7f4dd4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:24.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de8e5b640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4dd4004810 0x7f4dd4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de8e5b640 1 -- 172.21.15.83:0/1090235115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd4003530 con 0x7f4dd40039f0 2024-09-19T07:28:24.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.919+0000 7f4de3fff640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4dd4004810 0x7f4dd4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:28:24.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4de8e5b640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4dd40039f0 0x7f4dd4003e10 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800c240 tx=0x7f4dd801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f4d35608ea47ffd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:24.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8007d20 con 0x7f4dd40039f0 2024-09-19T07:28:24.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dd80a9070 con 0x7f4dd40039f0 2024-09-19T07:28:24.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4de37fe640 1 -- 172.21.15.83:0/1090235115 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8005d60 con 0x7f4dd40039f0 2024-09-19T07:28:24.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 -- 172.21.15.83:0/1090235115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4dd40039f0 msgr2=0x7f4dd4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:24.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 --2- 172.21.15.83:0/1090235115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4dd40039f0 0x7f4dd4003e10 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800c240 tx=0x7f4dd801cc80 comp rx=0 tx=0).stop 2024-09-19T07:28:24.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4de965c640 1 -- 172.21.15.83:0/1090235115 reap_dead start 2024-09-19T07:28:24.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 -- 172.21.15.83:0/1090235115 shutdown_connections 2024-09-19T07:28:24.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 -- 172.21.15.83:0/1090235115 >> 172.21.15.83:0/1090235115 conn(0x7f4de40c5060 msgr2=0x7f4de40c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:24.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 -- 172.21.15.83:0/1090235115 shutdown_connections 2024-09-19T07:28:24.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 -- 172.21.15.83:0/1090235115 wait complete. 2024-09-19T07:28:24.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 Processor -- start 2024-09-19T07:28:24.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 -- start start 2024-09-19T07:28:24.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4dd40039f0 0x7f4de40720b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de40ad6d0 0x7f4de40725f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de40ad040 0x7f4de40ab870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4de40c2c80 con 0x7f4de40ad6d0 2024-09-19T07:28:24.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4de415a110 con 0x7f4dd40039f0 2024-09-19T07:28:24.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4deb0bf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4de4161d10 con 0x7f4de40ad040 2024-09-19T07:28:24.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4de3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de40ad6d0 0x7f4de40725f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4de3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de40ad6d0 0x7f4de40725f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43374/0 (socket says 172.21.15.83:43374) 2024-09-19T07:28:24.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4de3fff640 1 -- 172.21.15.83:0/3874666456 learned_addr learned my addr 172.21.15.83:0/3874666456 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:24.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.923+0000 7f4de8e5b640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4dd40039f0 0x7f4de40720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de965c640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de40ad040 0x7f4de40ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de3fff640 1 -- 172.21.15.83:0/3874666456 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de40ad040 msgr2=0x7f4de40ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:24.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de3fff640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de40ad040 0x7f4de40ab870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de3fff640 1 -- 172.21.15.83:0/3874666456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4dd40039f0 msgr2=0x7f4de40720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:24.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de3fff640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4dd40039f0 0x7f4de40720b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:24.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de3fff640 1 -- 172.21.15.83:0/3874666456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de4152c00 con 0x7f4de40ad6d0 2024-09-19T07:28:24.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de3fff640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de40ad6d0 0x7f4de40725f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000f540 tx=0x7f4dd000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:24.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de17fa640 1 -- 172.21.15.83:0/3874666456 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0019020 con 0x7f4de40ad6d0 2024-09-19T07:28:24.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de17fa640 1 -- 172.21.15.83:0/3874666456 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dd0014870 con 0x7f4de40ad6d0 2024-09-19T07:28:24.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4deb0bf640 1 -- 172.21.15.83:0/3874666456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4de41ef940 con 0x7f4de40ad6d0 2024-09-19T07:28:24.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de17fa640 1 -- 172.21.15.83:0/3874666456 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0014c40 con 0x7f4de40ad6d0 2024-09-19T07:28:24.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4deb0bf640 1 -- 172.21.15.83:0/3874666456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4de41530b0 con 0x7f4de40ad6d0 2024-09-19T07:28:24.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4deb0bf640 1 -- 172.21.15.83:0/3874666456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4de41ef770 con 0x7f4de40ad6d0 2024-09-19T07:28:24.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de17fa640 1 -- 172.21.15.83:0/3874666456 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4dd001c020 con 0x7f4de40ad6d0 2024-09-19T07:28:24.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de17fa640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4dc80678c0 0x7f4dc8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:24.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de17fa640 1 -- 172.21.15.83:0/3874666456 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7755+0+0 (secure 0 0 0) 0x7f4dd0086b10 con 0x7f4de40ad6d0 2024-09-19T07:28:24.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de8e5b640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4dc80678c0 0x7f4dc8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:24.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.927+0000 7f4de8e5b640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4dc80678c0 0x7f4dc8069d80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80ab000 tx=0x7f4dd80b1040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:24.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:24.935+0000 7f4de17fa640 1 -- 172.21.15.83:0/3874666456 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4dd0023020 con 0x7f4de40ad6d0 2024-09-19T07:28:25.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 -- 172.21.15.83:0/3874666456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4de4155c50 con 0x7f4de40ad6d0 2024-09-19T07:28:25.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4de17fa640 1 -- 172.21.15.83:0/3874666456 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6534 (secure 0 0 0) 0x7f4dd0026350 con 0x7f4de40ad6d0 2024-09-19T07:28:25.190 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:25.190 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":79,"btime":"2024-09-19T07:28:16:180103+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:25.190 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 79 2024-09-19T07:28:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 -- 172.21.15.83:0/3874666456 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4dc80678c0 msgr2=0x7f4dc8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4dc80678c0 0x7f4dc8069d80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80ab000 tx=0x7f4dd80b1040 comp rx=0 tx=0).stop 2024-09-19T07:28:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 -- 172.21.15.83:0/3874666456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de40ad6d0 msgr2=0x7f4de40725f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de40ad6d0 0x7f4de40725f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000f540 tx=0x7f4dd000fa00 comp rx=0 tx=0).stop 2024-09-19T07:28:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 -- 172.21.15.83:0/3874666456 shutdown_connections 2024-09-19T07:28:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4dc80678c0 0x7f4dc8069d80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4de40ad040 0x7f4de40ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de40ad6d0 0x7f4de40725f0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 --2- 172.21.15.83:0/3874666456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4dd40039f0 0x7f4de40720b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 -- 172.21.15.83:0/3874666456 >> 172.21.15.83:0/3874666456 conn(0x7f4de40c5060 msgr2=0x7f4de40c2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:25.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 -- 172.21.15.83:0/3874666456 shutdown_connections 2024-09-19T07:28:25.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:25.195+0000 7f4deb0bf640 1 -- 172.21.15.83:0/3874666456 wait complete. 2024-09-19T07:28:25.211 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:28:27.213 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:27.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.371+0000 7f26cb328640 1 Processor -- start 2024-09-19T07:28:27.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.371+0000 7f26cb328640 1 -- start start 2024-09-19T07:28:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.371+0000 7f26cb328640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f26c4150220 0x7f26c4156220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.371+0000 7f26cb328640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26c4068290 con 0x7f26c406d3e0 2024-09-19T07:28:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.371+0000 7f26cb328640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26c4068570 con 0x7f26c4150600 2024-09-19T07:28:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.371+0000 7f26cb328640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f26c4152d60 con 0x7f26c4150220 2024-09-19T07:28:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.371+0000 7f26c98c5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f26c406d3e0 0x7f26c4153560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51324/0 (socket says 172.21.15.83:51324) 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.371+0000 7f26c98c5640 1 -- 172.21.15.83:0/313488292 learned_addr learned my addr 172.21.15.83:0/313488292 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.371+0000 7f26c90c4640 1 --2- 172.21.15.83:0/313488292 >> v2:172.21.15.184:3300/0 conn(0x7f26c4150220 0x7f26c4156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.371+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 772061451 0 0) 0x7f26c4068290 con 0x7f26c406d3e0 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.371+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26b0003530 con 0x7f26c406d3e0 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 691356446 0 0) 0x7f26c4068570 con 0x7f26c4150600 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26c4068290 con 0x7f26c4150600 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 953426698 0 0) 0x7f26b0003530 con 0x7f26c406d3e0 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 >> v2:172.21.15.184:3300/0 conn(0x7f26c4150220 msgr2=0x7f26c4156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 --2- 172.21.15.83:0/313488292 >> v2:172.21.15.184:3300/0 conn(0x7f26c4150220 0x7f26c4156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 >> v1:172.21.15.184:6789/0 conn(0x7f26c4150600 legacy=0x7f26c40ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c4153040 con 0x7f26c406d3e0 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f26bc001a00 con 0x7f26c406d3e0 2024-09-19T07:28:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 >> v1:172.21.15.83:6789/0 conn(0x7f26c406d3e0 legacy=0x7f26c4153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:27.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 --2- 172.21.15.83:0/313488292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b0003b50 0x7f26b0003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:27.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 --2- 172.21.15.83:0/313488292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26b0004810 0x7f26b0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:27.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 --2- 172.21.15.83:0/313488292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26b0005240 0x7f26b000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:27.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f26c4152d60 con 0x7f26b0004810 2024-09-19T07:28:27.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26b000c1a0 con 0x7f26b0003b50 2024-09-19T07:28:27.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f26b000e5b0 con 0x7f26b0005240 2024-09-19T07:28:27.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26c88c3640 1 --2- 172.21.15.83:0/313488292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26b0004810 0x7f26b0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:27.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26c90c4640 1 --2- 172.21.15.83:0/313488292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b0003b50 0x7f26b0003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:27.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26c98c5640 1 --2- 172.21.15.83:0/313488292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26b0005240 0x7f26b000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:27.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26c88c3640 1 -- 172.21.15.83:0/313488292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26b0005240 msgr2=0x7f26b000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:27.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26c88c3640 1 --2- 172.21.15.83:0/313488292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26b0005240 0x7f26b000b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:27.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26c88c3640 1 -- 172.21.15.83:0/313488292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b0003b50 msgr2=0x7f26b0003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:27.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26c88c3640 1 --2- 172.21.15.83:0/313488292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26b0003b50 0x7f26b0003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:27.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26c88c3640 1 -- 172.21.15.83:0/313488292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b0003530 con 0x7f26b0004810 2024-09-19T07:28:27.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26c98c5640 1 -- 172.21.15.83:0/313488292 reap_dead start 2024-09-19T07:28:27.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.375+0000 7f26c88c3640 1 --2- 172.21.15.83:0/313488292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26b0004810 0x7f26b0004c90 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f26b40018d0 tx=0x7f26b401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c40c95a71c4345a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:27.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4001520 con 0x7f26b0004810 2024-09-19T07:28:27.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26b40a4e50 con 0x7f26b0004810 2024-09-19T07:28:27.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26bbfff640 1 -- 172.21.15.83:0/313488292 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4003600 con 0x7f26b0004810 2024-09-19T07:28:27.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 -- 172.21.15.83:0/313488292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26b0004810 msgr2=0x7f26b0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:27.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 --2- 172.21.15.83:0/313488292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26b0004810 0x7f26b0004c90 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f26b40018d0 tx=0x7f26b401c4a0 comp rx=0 tx=0).stop 2024-09-19T07:28:27.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 -- 172.21.15.83:0/313488292 shutdown_connections 2024-09-19T07:28:27.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 --2- 172.21.15.83:0/313488292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26b0004810 0x7f26b0004c90 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:27.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 -- 172.21.15.83:0/313488292 >> 172.21.15.83:0/313488292 conn(0x7f26c40c48b0 msgr2=0x7f26c40c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:27.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 -- 172.21.15.83:0/313488292 shutdown_connections 2024-09-19T07:28:27.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 -- 172.21.15.83:0/313488292 wait complete. 2024-09-19T07:28:27.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 Processor -- start 2024-09-19T07:28:27.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 -- start start 2024-09-19T07:28:27.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26b0004810 0x7f26c4154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:27.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26c41545a0 0x7f26c41defa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:27.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26c4155160 0x7f26c41df4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:27.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f26c40c3b50 con 0x7f26c41545a0 2024-09-19T07:28:27.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26c40c39d0 con 0x7f26c4155160 2024-09-19T07:28:27.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26cb328640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f26c40c3cd0 con 0x7f26b0004810 2024-09-19T07:28:27.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26c88c3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26c41545a0 0x7f26c41defa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:27.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26c88c3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26c41545a0 0x7f26c41defa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43404/0 (socket says 172.21.15.83:43404) 2024-09-19T07:28:27.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26c88c3640 1 -- 172.21.15.83:0/746033092 learned_addr learned my addr 172.21.15.83:0/746033092 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:27.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26c90c4640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26b0004810 0x7f26c4154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:27.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.379+0000 7f26c98c5640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26c4155160 0x7f26c41df4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:27.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26c88c3640 1 -- 172.21.15.83:0/746033092 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26b0004810 msgr2=0x7f26c4154c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:27.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26c88c3640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26b0004810 0x7f26c4154c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:27.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26c88c3640 1 -- 172.21.15.83:0/746033092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26c4155160 msgr2=0x7f26c41df4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:27.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26c88c3640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26c4155160 0x7f26c41df4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:27.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26c88c3640 1 -- 172.21.15.83:0/746033092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c41540c0 con 0x7f26c41545a0 2024-09-19T07:28:27.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26c88c3640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26c41545a0 0x7f26c41defa0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f26b4005040 tx=0x7f26b0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:27.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26b9ffb640 1 -- 172.21.15.83:0/746033092 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4003a10 con 0x7f26c41545a0 2024-09-19T07:28:27.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26b9ffb640 1 -- 172.21.15.83:0/746033092 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26b4003bd0 con 0x7f26c41545a0 2024-09-19T07:28:27.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26cb328640 1 -- 172.21.15.83:0/746033092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26c41e02d0 con 0x7f26c41545a0 2024-09-19T07:28:27.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26b9ffb640 1 -- 172.21.15.83:0/746033092 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4001500 con 0x7f26c41545a0 2024-09-19T07:28:27.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26cb328640 1 -- 172.21.15.83:0/746033092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26c414b660 con 0x7f26c41545a0 2024-09-19T07:28:27.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26cb328640 1 -- 172.21.15.83:0/746033092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26c4157580 con 0x7f26c41545a0 2024-09-19T07:28:27.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26b9ffb640 1 -- 172.21.15.83:0/746033092 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f26b40067e0 con 0x7f26c41545a0 2024-09-19T07:28:27.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26b9ffb640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26a80678f0 0x7f26a8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:27.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26b9ffb640 1 -- 172.21.15.83:0/746033092 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7755+0+0 (secure 0 0 0) 0x7f26b41204d0 con 0x7f26c41545a0 2024-09-19T07:28:27.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26c90c4640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26a80678f0 0x7f26a8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:27.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.383+0000 7f26c90c4640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26a80678f0 0x7f26a8069db0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f26c00099d0 tx=0x7f26c0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:27.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.391+0000 7f26b9ffb640 1 -- 172.21.15.83:0/746033092 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f26b40b8070 con 0x7f26c41545a0 2024-09-19T07:28:27.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 -- 172.21.15.83:0/746033092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26c40c54d0 con 0x7f26c41545a0 2024-09-19T07:28:27.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26b9ffb640 1 -- 172.21.15.83:0/746033092 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6534 (secure 0 0 0) 0x7f26b4016270 con 0x7f26c41545a0 2024-09-19T07:28:27.618 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:27.618 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":79,"btime":"2024-09-19T07:28:16:180103+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 79 2024-09-19T07:28:27.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 -- 172.21.15.83:0/746033092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26a80678f0 msgr2=0x7f26a8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:27.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26a80678f0 0x7f26a8069db0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f26c00099d0 tx=0x7f26c0008040 comp rx=0 tx=0).stop 2024-09-19T07:28:27.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 -- 172.21.15.83:0/746033092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26c41545a0 msgr2=0x7f26c41defa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:27.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26c41545a0 0x7f26c41defa0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f26b4005040 tx=0x7f26b0005840 comp rx=0 tx=0).stop 2024-09-19T07:28:27.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 -- 172.21.15.83:0/746033092 shutdown_connections 2024-09-19T07:28:27.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26a80678f0 0x7f26a8069db0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:27.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26c4155160 0x7f26c41df4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:27.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26c41545a0 0x7f26c41defa0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:27.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 --2- 172.21.15.83:0/746033092 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26b0004810 0x7f26c4154c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:27.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 -- 172.21.15.83:0/746033092 >> 172.21.15.83:0/746033092 conn(0x7f26c40c48b0 msgr2=0x7f26c414f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:27.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 -- 172.21.15.83:0/746033092 shutdown_connections 2024-09-19T07:28:27.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:27.623+0000 7f26cb328640 1 -- 172.21.15.83:0/746033092 wait complete. 2024-09-19T07:28:27.641 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:28:29.642 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:29.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4d031cf640 1 Processor -- start 2024-09-19T07:28:29.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4d031cf640 1 -- start start 2024-09-19T07:28:29.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4d031cf640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4cfc06cef0 0x7f4cfc0c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:29.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4d031cf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1534f0 0x7f4cfc14fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:29.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4d031cf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cfc0c6b70 con 0x7f4cfc06ddd0 2024-09-19T07:28:29.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4d031cf640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4cfc06f730 con 0x7f4cfc06cef0 2024-09-19T07:28:29.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4d031cf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4cfc153980 con 0x7f4cfc1534f0 2024-09-19T07:28:29.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4cfbfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4cfc06ddd0 0x7f4cfc0c6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51338/0 (socket says 172.21.15.83:51338) 2024-09-19T07:28:29.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4cfbfff640 1 -- 172.21.15.83:0/3203514007 learned_addr learned my addr 172.21.15.83:0/3203514007 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:29.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2566195988 0 0) 0x7f4cfc0c6b70 con 0x7f4cfc06ddd0 2024-09-19T07:28:29.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cec003530 con 0x7f4cfc06ddd0 2024-09-19T07:28:29.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4d0176c640 1 --2- 172.21.15.83:0/3203514007 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1534f0 0x7f4cfc14fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:29.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4d00f6b640 1 --2- 172.21.15.83:0/3203514007 >> v2:172.21.15.184:3300/0 conn(0x7f4cfc06cef0 0x7f4cfc0c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:29.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2715639125 0 0) 0x7f4cec003530 con 0x7f4cfc06ddd0 2024-09-19T07:28:29.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1534f0 msgr2=0x7f4cfc14fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4cfb7fe640 1 --2- 172.21.15.83:0/3203514007 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1534f0 0x7f4cfc14fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 >> v2:172.21.15.184:3300/0 conn(0x7f4cfc06cef0 msgr2=0x7f4cfc0c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4cfb7fe640 1 --2- 172.21.15.83:0/3203514007 >> v2:172.21.15.184:3300/0 conn(0x7f4cfc06cef0 0x7f4cfc0c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.787+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cfc06e210 con 0x7f4cfc06ddd0 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f4ce8001680 con 0x7f4cfc06ddd0 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 >> v1:172.21.15.83:6789/0 conn(0x7f4cfc06ddd0 legacy=0x7f4cfc0c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfb7fe640 1 --2- 172.21.15.83:0/3203514007 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1534f0 0x7f4cec0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfb7fe640 1 --2- 172.21.15.83:0/3203514007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cec003ff0 0x7f4cec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfb7fe640 1 --2- 172.21.15.83:0/3203514007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cec004d50 0x7f4cec0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc06f730 con 0x7f4cec003ff0 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc153980 con 0x7f4cec004d50 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4cec00bc40 con 0x7f4cfc1534f0 2024-09-19T07:28:29.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfbfff640 1 --2- 172.21.15.83:0/3203514007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cec003ff0 0x7f4cec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:29.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4d0176c640 1 --2- 172.21.15.83:0/3203514007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cec004d50 0x7f4cec0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:29.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4d00f6b640 1 --2- 172.21.15.83:0/3203514007 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1534f0 0x7f4cec0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:29.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfbfff640 1 -- 172.21.15.83:0/3203514007 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1534f0 msgr2=0x7f4cec0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:29.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfbfff640 1 --2- 172.21.15.83:0/3203514007 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1534f0 0x7f4cec0039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:29.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfbfff640 1 -- 172.21.15.83:0/3203514007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cec004d50 msgr2=0x7f4cec0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:29.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfbfff640 1 --2- 172.21.15.83:0/3203514007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cec004d50 0x7f4cec0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:29.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfbfff640 1 -- 172.21.15.83:0/3203514007 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cec003530 con 0x7f4cec003ff0 2024-09-19T07:28:29.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfbfff640 1 --2- 172.21.15.83:0/3203514007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cec003ff0 0x7f4cec004810 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800c210 tx=0x7f4ce801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbd0e66eb0540fb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:29.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8007d60 con 0x7f4cec003ff0 2024-09-19T07:28:29.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce80a9020 con 0x7f4cec003ff0 2024-09-19T07:28:29.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4cfb7fe640 1 -- 172.21.15.83:0/3203514007 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8005b00 con 0x7f4cec003ff0 2024-09-19T07:28:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4d031cf640 1 -- 172.21.15.83:0/3203514007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cec003ff0 msgr2=0x7f4cec004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4d031cf640 1 --2- 172.21.15.83:0/3203514007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cec003ff0 0x7f4cec004810 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800c210 tx=0x7f4ce801ccc0 comp rx=0 tx=0).stop 2024-09-19T07:28:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.791+0000 7f4d0176c640 1 -- 172.21.15.83:0/3203514007 reap_dead start 2024-09-19T07:28:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 -- 172.21.15.83:0/3203514007 shutdown_connections 2024-09-19T07:28:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 -- 172.21.15.83:0/3203514007 >> 172.21.15.83:0/3203514007 conn(0x7f4cfc0aa910 msgr2=0x7f4cfc0c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 -- 172.21.15.83:0/3203514007 shutdown_connections 2024-09-19T07:28:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 -- 172.21.15.83:0/3203514007 wait complete. 2024-09-19T07:28:29.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 Processor -- start 2024-09-19T07:28:29.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 -- start start 2024-09-19T07:28:29.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cec003ff0 0x7f4cfc0bde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1f1d00 0x7f4cfc1efd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cfc1f02c0 0x7f4cfc0c2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc06d450 con 0x7f4cfc1f02c0 2024-09-19T07:28:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc06d2d0 con 0x7f4cec003ff0 2024-09-19T07:28:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4cfc074f90 con 0x7f4cfc1f1d00 2024-09-19T07:28:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d0176c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cfc1f02c0 0x7f4cfc0c2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d0176c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cfc1f02c0 0x7f4cfc0c2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43408/0 (socket says 172.21.15.83:43408) 2024-09-19T07:28:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d0176c640 1 -- 172.21.15.83:0/223077147 learned_addr learned my addr 172.21.15.83:0/223077147 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d00f6b640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cec003ff0 0x7f4cfc0bde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4cfbfff640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1f1d00 0x7f4cfc1efd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d0176c640 1 -- 172.21.15.83:0/223077147 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1f1d00 msgr2=0x7f4cfc1efd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d0176c640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1f1d00 0x7f4cfc1efd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:29.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d0176c640 1 -- 172.21.15.83:0/223077147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cec003ff0 msgr2=0x7f4cfc0bde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:29.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d0176c640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cec003ff0 0x7f4cfc0bde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:29.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d0176c640 1 -- 172.21.15.83:0/223077147 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cfc0c50e0 con 0x7f4cfc1f02c0 2024-09-19T07:28:29.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d0176c640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cfc1f02c0 0x7f4cfc0c2e60 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4cec004790 tx=0x7f4cf4009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:29.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4cf97fa640 1 -- 172.21.15.83:0/223077147 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf4007cf0 con 0x7f4cfc1f02c0 2024-09-19T07:28:29.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4cf97fa640 1 -- 172.21.15.83:0/223077147 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cf4019070 con 0x7f4cfc1f02c0 2024-09-19T07:28:29.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 -- 172.21.15.83:0/223077147 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cfc1f2550 con 0x7f4cfc1f02c0 2024-09-19T07:28:29.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4cf97fa640 1 -- 172.21.15.83:0/223077147 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf4014650 con 0x7f4cfc1f02c0 2024-09-19T07:28:29.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.795+0000 7f4d031cf640 1 -- 172.21.15.83:0/223077147 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cfc0ad1f0 con 0x7f4cfc1f02c0 2024-09-19T07:28:29.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.799+0000 7f4d031cf640 1 -- 172.21.15.83:0/223077147 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cfc0c79d0 con 0x7f4cfc1f02c0 2024-09-19T07:28:29.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.799+0000 7f4cf97fa640 1 -- 172.21.15.83:0/223077147 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4cf40147f0 con 0x7f4cfc1f02c0 2024-09-19T07:28:29.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.799+0000 7f4cf97fa640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ce00678c0 0x7f4ce0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:29.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.799+0000 7f4cf97fa640 1 -- 172.21.15.83:0/223077147 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7755+0+0 (secure 0 0 0) 0x7f4cf4086920 con 0x7f4cfc1f02c0 2024-09-19T07:28:29.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.799+0000 7f4d00f6b640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ce00678c0 0x7f4ce0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:29.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.799+0000 7f4d00f6b640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ce00678c0 0x7f4ce0069d80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc14d1a0 tx=0x7f4cf0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:29.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:29.803+0000 7f4cf97fa640 1 -- 172.21.15.83:0/223077147 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4cf4056c50 con 0x7f4cfc1f02c0 2024-09-19T07:28:30.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.039+0000 7f4d031cf640 1 -- 172.21.15.83:0/223077147 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cfc14d800 con 0x7f4cfc1f02c0 2024-09-19T07:28:30.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4cf97fa640 1 -- 172.21.15.83:0/223077147 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6534 (secure 0 0 0) 0x7f4cf4056550 con 0x7f4cfc1f02c0 2024-09-19T07:28:30.038 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:30.038 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":79,"btime":"2024-09-19T07:28:16:180103+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:30.038 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 79 2024-09-19T07:28:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4d031cf640 1 -- 172.21.15.83:0/223077147 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ce00678c0 msgr2=0x7f4ce0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4d031cf640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ce00678c0 0x7f4ce0069d80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc14d1a0 tx=0x7f4cf0008040 comp rx=0 tx=0).stop 2024-09-19T07:28:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4d031cf640 1 -- 172.21.15.83:0/223077147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cfc1f02c0 msgr2=0x7f4cfc0c2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4d031cf640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cfc1f02c0 0x7f4cfc0c2e60 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4cec004790 tx=0x7f4cf4009d40 comp rx=0 tx=0).stop 2024-09-19T07:28:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4d031cf640 1 -- 172.21.15.83:0/223077147 shutdown_connections 2024-09-19T07:28:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4d031cf640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ce00678c0 0x7f4ce0069d80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4d031cf640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cfc1f02c0 0x7f4cfc0c2e60 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4d031cf640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cfc1f1d00 0x7f4cfc1efd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4d031cf640 1 --2- 172.21.15.83:0/223077147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cec003ff0 0x7f4cfc0bde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4d031cf640 1 -- 172.21.15.83:0/223077147 >> 172.21.15.83:0/223077147 conn(0x7f4cfc0aa910 msgr2=0x7f4cfc151a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4d031cf640 1 -- 172.21.15.83:0/223077147 shutdown_connections 2024-09-19T07:28:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:30.043+0000 7f4d031cf640 1 -- 172.21.15.83:0/223077147 wait complete. 2024-09-19T07:28:30.059 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:28:32.060 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:32.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.203+0000 7f864b754640 1 Processor -- start 2024-09-19T07:28:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.203+0000 7f864b754640 1 -- start start 2024-09-19T07:28:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f864b754640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8644153480 0x7f8644155e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f864b754640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8644153860 0x7f8644156380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f864b754640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8644159dc0 con 0x7f864406be10 2024-09-19T07:28:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f864b754640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f8644070040 con 0x7f8644153480 2024-09-19T07:28:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f864b754640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f86440c1d80 con 0x7f8644153860 2024-09-19T07:28:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f86494f0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8644153480 0x7f8644155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f86494f0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8644153480 0x7f8644155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46850/0 (socket says 172.21.15.83:46850) 2024-09-19T07:28:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f86494f0640 1 -- 172.21.15.83:0/496766410 learned_addr learned my addr 172.21.15.83:0/496766410 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f86494f0640 1 -- 172.21.15.83:0/496766410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8644153860 msgr2=0x7f8644156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f86494f0640 1 --2- 172.21.15.83:0/496766410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8644153860 0x7f8644156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f86494f0640 1 -- 172.21.15.83:0/496766410 >> v1:172.21.15.184:6789/0 conn(0x7f864406be10 legacy=0x7f864406c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f86494f0640 1 -- 172.21.15.83:0/496766410 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86440c2bf0 con 0x7f8644153480 2024-09-19T07:28:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f86494f0640 1 --2- 172.21.15.83:0/496766410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8644153480 0x7f8644155e40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8640009870 tx=0x7f864001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577d16faccec75e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f863bfff640 1 -- 172.21.15.83:0/496766410 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86400b0070 con 0x7f8644153480 2024-09-19T07:28:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f863bfff640 1 -- 172.21.15.83:0/496766410 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86400ab600 con 0x7f8644153480 2024-09-19T07:28:32.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.207+0000 7f863bfff640 1 -- 172.21.15.83:0/496766410 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86400ab9d0 con 0x7f8644153480 2024-09-19T07:28:32.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 -- 172.21.15.83:0/496766410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8644153480 msgr2=0x7f8644155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:32.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 --2- 172.21.15.83:0/496766410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8644153480 0x7f8644155e40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8640009870 tx=0x7f864001c520 comp rx=0 tx=0).stop 2024-09-19T07:28:32.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 -- 172.21.15.83:0/496766410 shutdown_connections 2024-09-19T07:28:32.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 --2- 172.21.15.83:0/496766410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8644153860 0x7f8644156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:32.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 --2- 172.21.15.83:0/496766410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8644153480 0x7f8644155e40 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:32.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 -- 172.21.15.83:0/496766410 >> 172.21.15.83:0/496766410 conn(0x7f86440c8840 msgr2=0x7f86440c8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:32.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 -- 172.21.15.83:0/496766410 shutdown_connections 2024-09-19T07:28:32.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 -- 172.21.15.83:0/496766410 wait complete. 2024-09-19T07:28:32.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 Processor -- start 2024-09-19T07:28:32.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 -- start start 2024-09-19T07:28:32.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f864406be10 0x7f864415ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:32.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8644153480 0x7f864415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:32.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8644153860 0x7f864415b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:32.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f864406ce80 con 0x7f8644153860 2024-09-19T07:28:32.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f86440c29a0 con 0x7f8644153480 2024-09-19T07:28:32.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f864b754640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f864406d000 con 0x7f864406be10 2024-09-19T07:28:32.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f8649cf1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8644153860 0x7f864415b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:32.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f8649cf1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8644153860 0x7f864415b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46858/0 (socket says 172.21.15.83:46858) 2024-09-19T07:28:32.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f8649cf1640 1 -- 172.21.15.83:0/1134117593 learned_addr learned my addr 172.21.15.83:0/1134117593 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:32.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f8648cef640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8644153480 0x7f864415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:32.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f8649cf1640 1 -- 172.21.15.83:0/1134117593 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f864406be10 msgr2=0x7f864415ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:32.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f8649cf1640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f864406be10 0x7f864415ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:32.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f8649cf1640 1 -- 172.21.15.83:0/1134117593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8644153480 msgr2=0x7f864415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:32.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f8649cf1640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8644153480 0x7f864415b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:32.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f8649cf1640 1 -- 172.21.15.83:0/1134117593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864415f7f0 con 0x7f8644153860 2024-09-19T07:28:32.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f8648cef640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8644153480 0x7f864415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:28:32.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.211+0000 7f8649cf1640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8644153860 0x7f864415b830 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f863c00b9a0 tx=0x7f863c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:32.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.215+0000 7f8639ffb640 1 -- 172.21.15.83:0/1134117593 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c019070 con 0x7f8644153860 2024-09-19T07:28:32.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.215+0000 7f8639ffb640 1 -- 172.21.15.83:0/1134117593 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f863c00cba0 con 0x7f8644153860 2024-09-19T07:28:32.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.215+0000 7f8639ffb640 1 -- 172.21.15.83:0/1134117593 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c014440 con 0x7f8644153860 2024-09-19T07:28:32.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.215+0000 7f864b754640 1 -- 172.21.15.83:0/1134117593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86440adac0 con 0x7f8644153860 2024-09-19T07:28:32.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.215+0000 7f864b754640 1 -- 172.21.15.83:0/1134117593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f864415a5b0 con 0x7f8644153860 2024-09-19T07:28:32.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.215+0000 7f864b754640 1 -- 172.21.15.83:0/1134117593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f864406c360 con 0x7f8644153860 2024-09-19T07:28:32.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.215+0000 7f8639ffb640 1 -- 172.21.15.83:0/1134117593 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f863c01c070 con 0x7f8644153860 2024-09-19T07:28:32.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.215+0000 7f8639ffb640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8628067910 0x7f8628069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:32.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.215+0000 7f8639ffb640 1 -- 172.21.15.83:0/1134117593 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7755+0+0 (secure 0 0 0) 0x7f863c085ff0 con 0x7f8644153860 2024-09-19T07:28:32.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.215+0000 7f86494f0640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8628067910 0x7f8628069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:32.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.215+0000 7f86494f0640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8628067910 0x7f8628069dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f86400ad000 tx=0x7f8640008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:32.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.219+0000 7f8639ffb640 1 -- 172.21.15.83:0/1134117593 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f863c0564a0 con 0x7f8644153860 2024-09-19T07:28:32.247 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:28:32.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:28:32.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.483+0000 7f864b754640 1 -- 172.21.15.83:0/1134117593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86440c4cc0 con 0x7f8644153860 2024-09-19T07:28:32.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f8639ffb640 1 -- 172.21.15.83:0/1134117593 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6534 (secure 0 0 0) 0x7f863c055da0 con 0x7f8644153860 2024-09-19T07:28:32.481 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:32.482 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":79,"btime":"2024-09-19T07:28:16:180103+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:32.482 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 79 2024-09-19T07:28:32.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f864b754640 1 -- 172.21.15.83:0/1134117593 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8628067910 msgr2=0x7f8628069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:32.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f864b754640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8628067910 0x7f8628069dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f86400ad000 tx=0x7f8640008040 comp rx=0 tx=0).stop 2024-09-19T07:28:32.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f864b754640 1 -- 172.21.15.83:0/1134117593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8644153860 msgr2=0x7f864415b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:32.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f864b754640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8644153860 0x7f864415b830 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f863c00b9a0 tx=0x7f863c00be60 comp rx=0 tx=0).stop 2024-09-19T07:28:32.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f864b754640 1 -- 172.21.15.83:0/1134117593 shutdown_connections 2024-09-19T07:28:32.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f864b754640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8628067910 0x7f8628069dd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:32.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f864b754640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8644153860 0x7f864415b830 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:32.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f864b754640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8644153480 0x7f864415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f864b754640 1 --2- 172.21.15.83:0/1134117593 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f864406be10 0x7f864415ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f864b754640 1 -- 172.21.15.83:0/1134117593 >> 172.21.15.83:0/1134117593 conn(0x7f86440c8840 msgr2=0x7f86440aa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f864b754640 1 -- 172.21.15.83:0/1134117593 shutdown_connections 2024-09-19T07:28:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:32.487+0000 7f864b754640 1 -- 172.21.15.83:0/1134117593 wait complete. 2024-09-19T07:28:32.504 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:28:34.505 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:34.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.671+0000 7ff5bdda3640 1 Processor -- start 2024-09-19T07:28:34.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.671+0000 7ff5bdda3640 1 -- start start 2024-09-19T07:28:34.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5bdda3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff5b80c3130 0x7ff5b80ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:34.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5bdda3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff5b806da00 0x7ff5b80c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:34.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5bdda3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5b80c4990 con 0x7ff5b80c3a50 2024-09-19T07:28:34.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5bdda3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff5b80c4c70 con 0x7ff5b806da00 2024-09-19T07:28:34.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5bdda3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff5b80ab7b0 con 0x7ff5b80c3130 2024-09-19T07:28:34.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5b6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff5b806da00 0x7ff5b80c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:34.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5b6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff5b806da00 0x7ff5b80c3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46874/0 (socket says 172.21.15.83:46874) 2024-09-19T07:28:34.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5b6ffd640 1 -- 172.21.15.83:0/322328289 learned_addr learned my addr 172.21.15.83:0/322328289 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:34.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5b6ffd640 1 -- 172.21.15.83:0/322328289 >> v2:172.21.15.184:3300/0 conn(0x7ff5b80c3130 msgr2=0x7ff5b80ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:34.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5b6ffd640 1 --2- 172.21.15.83:0/322328289 >> v2:172.21.15.184:3300/0 conn(0x7ff5b80c3130 0x7ff5b80ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:34.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5b6ffd640 1 -- 172.21.15.83:0/322328289 >> v1:172.21.15.184:6789/0 conn(0x7ff5b80c3a50 legacy=0x7ff5b80c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:34.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5b6ffd640 1 -- 172.21.15.83:0/322328289 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b80aba90 con 0x7ff5b806da00 2024-09-19T07:28:34.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5b6ffd640 1 --2- 172.21.15.83:0/322328289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b806da00 0x7ff5b80c3510 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff5b80c5eb0 tx=0x7ff5ac01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27202f2cc77213cb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:34.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5b67fc640 1 -- 172.21.15.83:0/322328289 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac0b2070 con 0x7ff5b806da00 2024-09-19T07:28:34.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5b67fc640 1 -- 172.21.15.83:0/322328289 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5ac0a9070 con 0x7ff5b806da00 2024-09-19T07:28:34.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.675+0000 7ff5b67fc640 1 -- 172.21.15.83:0/322328289 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac0ad880 con 0x7ff5b806da00 2024-09-19T07:28:34.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 -- 172.21.15.83:0/322328289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b806da00 msgr2=0x7ff5b80c3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:34.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 --2- 172.21.15.83:0/322328289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b806da00 0x7ff5b80c3510 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff5b80c5eb0 tx=0x7ff5ac01c450 comp rx=0 tx=0).stop 2024-09-19T07:28:34.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 -- 172.21.15.83:0/322328289 shutdown_connections 2024-09-19T07:28:34.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 --2- 172.21.15.83:0/322328289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b806da00 0x7ff5b80c3510 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:34.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 --2- 172.21.15.83:0/322328289 >> v2:172.21.15.184:3300/0 conn(0x7ff5b80c3130 0x7ff5b80ac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:34.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 -- 172.21.15.83:0/322328289 >> 172.21.15.83:0/322328289 conn(0x7ff5b80c5060 msgr2=0x7ff5b80c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:34.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 -- 172.21.15.83:0/322328289 shutdown_connections 2024-09-19T07:28:34.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 -- 172.21.15.83:0/322328289 wait complete. 2024-09-19T07:28:34.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 Processor -- start 2024-09-19T07:28:34.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 -- start start 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5b806da00 0x7ff5b8153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5b80c3130 0x7ff5b814e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b80c3a50 0x7ff5b814f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b815f810 con 0x7ff5b80c3a50 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b80ad730 con 0x7ff5b80c3130 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff5b815f990 con 0x7ff5b806da00 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5b7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b80c3a50 0x7ff5b814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5b7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b80c3a50 0x7ff5b814f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46886/0 (socket says 172.21.15.83:46886) 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5b7fff640 1 -- 172.21.15.83:0/1899217740 learned_addr learned my addr 172.21.15.83:0/1899217740 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5b77fe640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5b806da00 0x7ff5b8153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5b6ffd640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5b80c3130 0x7ff5b814e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5b7fff640 1 -- 172.21.15.83:0/1899217740 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5b806da00 msgr2=0x7ff5b8153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5b7fff640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5b806da00 0x7ff5b8153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5b7fff640 1 -- 172.21.15.83:0/1899217740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5b80c3130 msgr2=0x7ff5b814e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5b7fff640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5b80c3130 0x7ff5b814e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5b7fff640 1 -- 172.21.15.83:0/1899217740 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b81f1d50 con 0x7ff5b80c3a50 2024-09-19T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5b77fe640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5b806da00 0x7ff5b8153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5b7fff640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b80c3a50 0x7ff5b814f850 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff5b000ba10 tx=0x7ff5b000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff59ffff640 1 -- 172.21.15.83:0/1899217740 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0019070 con 0x7ff5b80c3a50 2024-09-19T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff59ffff640 1 -- 172.21.15.83:0/1899217740 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5b000cba0 con 0x7ff5b80c3a50 2024-09-19T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 -- 172.21.15.83:0/1899217740 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b814c880 con 0x7ff5b80c3a50 2024-09-19T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff59ffff640 1 -- 172.21.15.83:0/1899217740 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0014440 con 0x7ff5b80c3a50 2024-09-19T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.679+0000 7ff5bdda3640 1 -- 172.21.15.83:0/1899217740 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b8073bb0 con 0x7ff5b80c3a50 2024-09-19T07:28:34.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.683+0000 7ff5bdda3640 1 -- 172.21.15.83:0/1899217740 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5b80c3510 con 0x7ff5b80c3a50 2024-09-19T07:28:34.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.683+0000 7ff59ffff640 1 -- 172.21.15.83:0/1899217740 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff5b001c070 con 0x7ff5b80c3a50 2024-09-19T07:28:34.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.683+0000 7ff59ffff640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5a40678c0 0x7ff5a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:34.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.683+0000 7ff59ffff640 1 -- 172.21.15.83:0/1899217740 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7755+0+0 (secure 0 0 0) 0x7ff5b0086150 con 0x7ff5b80c3a50 2024-09-19T07:28:34.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.683+0000 7ff5b77fe640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5a40678c0 0x7ff5a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:34.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.683+0000 7ff5b77fe640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5a40678c0 0x7ff5a4069d80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8006fd0 tx=0x7ff5a8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:34.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.687+0000 7ff59ffff640 1 -- 172.21.15.83:0/1899217740 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff5b0056520 con 0x7ff5b80c3a50 2024-09-19T07:28:34.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.927+0000 7ff5bdda3640 1 -- 172.21.15.83:0/1899217740 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5b80ac330 con 0x7ff5b80c3a50 2024-09-19T07:28:34.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.927+0000 7ff59ffff640 1 -- 172.21.15.83:0/1899217740 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6534 (secure 0 0 0) 0x7ff5b0055e20 con 0x7ff5b80c3a50 2024-09-19T07:28:34.924 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:34.925 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":79,"btime":"2024-09-19T07:28:16:180103+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:34.925 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 79 2024-09-19T07:28:34.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.931+0000 7ff5bdda3640 1 -- 172.21.15.83:0/1899217740 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5a40678c0 msgr2=0x7ff5a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:34.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.931+0000 7ff5bdda3640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5a40678c0 0x7ff5a4069d80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8006fd0 tx=0x7ff5a8008040 comp rx=0 tx=0).stop 2024-09-19T07:28:34.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.931+0000 7ff5bdda3640 1 -- 172.21.15.83:0/1899217740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b80c3a50 msgr2=0x7ff5b814f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:34.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.931+0000 7ff5bdda3640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b80c3a50 0x7ff5b814f850 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff5b000ba10 tx=0x7ff5b000bed0 comp rx=0 tx=0).stop 2024-09-19T07:28:34.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.931+0000 7ff5bdda3640 1 -- 172.21.15.83:0/1899217740 shutdown_connections 2024-09-19T07:28:34.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.931+0000 7ff5bdda3640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5a40678c0 0x7ff5a4069d80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:34.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.931+0000 7ff5bdda3640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5b80c3a50 0x7ff5b814f850 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:34.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.931+0000 7ff5bdda3640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5b80c3130 0x7ff5b814e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:34.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.931+0000 7ff5bdda3640 1 --2- 172.21.15.83:0/1899217740 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5b806da00 0x7ff5b8153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:34.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.931+0000 7ff5bdda3640 1 -- 172.21.15.83:0/1899217740 >> 172.21.15.83:0/1899217740 conn(0x7ff5b80c5060 msgr2=0x7ff5b8067b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:34.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.931+0000 7ff5bdda3640 1 -- 172.21.15.83:0/1899217740 shutdown_connections 2024-09-19T07:28:34.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:34.931+0000 7ff5bdda3640 1 -- 172.21.15.83:0/1899217740 wait complete. 2024-09-19T07:28:34.944 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:28:36.945 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:37.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.083+0000 7f3ce514e640 1 Processor -- start 2024-09-19T07:28:37.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.083+0000 7f3ce514e640 1 -- start start 2024-09-19T07:28:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3ce514e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ce00701b0 0x7f3ce00ace50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3ce514e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3ce00ad390 0x7f3ce00ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3ce514e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce006c390 con 0x7f3ce014ffb0 2024-09-19T07:28:37.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3ce514e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce00705b0 con 0x7f3ce00ad390 2024-09-19T07:28:37.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3ce514e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce01532e0 con 0x7f3ce00701b0 2024-09-19T07:28:37.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3cd7fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ce00701b0 0x7f3ce00ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:37.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3cdf59e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3ce00ad390 0x7f3ce00ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:37.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3cd7fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ce00701b0 0x7f3ce00ace50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:58374/0 (socket says 172.21.15.83:58374) 2024-09-19T07:28:37.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3cd7fff640 1 -- 172.21.15.83:0/4002561402 learned_addr learned my addr 172.21.15.83:0/4002561402 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:37.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3cded9d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f3ce014ffb0 0x7f3ce006bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:55070/0 (socket says 172.21.15.83:55070) 2024-09-19T07:28:37.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3cd7fff640 1 -- 172.21.15.83:0/4002561402 >> v2:172.21.15.184:3300/0 conn(0x7f3ce00ad390 msgr2=0x7f3ce00ad7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:37.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3cd7fff640 1 --2- 172.21.15.83:0/4002561402 >> v2:172.21.15.184:3300/0 conn(0x7f3ce00ad390 0x7f3ce00ad7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:37.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3cd7fff640 1 -- 172.21.15.83:0/4002561402 >> v1:172.21.15.184:6789/0 conn(0x7f3ce014ffb0 legacy=0x7f3ce006bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:37.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.087+0000 7f3cd7fff640 1 -- 172.21.15.83:0/4002561402 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00c2990 con 0x7f3ce00701b0 2024-09-19T07:28:37.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3cd7fff640 1 --2- 172.21.15.83:0/4002561402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ce00701b0 0x7f3ce00ace50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8009870 tx=0x7f3cc801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36c9fe38337b2155 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3cde59c640 1 -- 172.21.15.83:0/4002561402 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc80b0070 con 0x7f3ce00701b0 2024-09-19T07:28:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3cde59c640 1 -- 172.21.15.83:0/4002561402 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cc80ab600 con 0x7f3ce00701b0 2024-09-19T07:28:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3cde59c640 1 -- 172.21.15.83:0/4002561402 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc80ab9d0 con 0x7f3ce00701b0 2024-09-19T07:28:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 -- 172.21.15.83:0/4002561402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ce00701b0 msgr2=0x7f3ce00ace50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 --2- 172.21.15.83:0/4002561402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ce00701b0 0x7f3ce00ace50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8009870 tx=0x7f3cc801c440 comp rx=0 tx=0).stop 2024-09-19T07:28:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 -- 172.21.15.83:0/4002561402 shutdown_connections 2024-09-19T07:28:37.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 --2- 172.21.15.83:0/4002561402 >> v2:172.21.15.184:3300/0 conn(0x7f3ce00ad390 0x7f3ce00ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:37.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 --2- 172.21.15.83:0/4002561402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ce00701b0 0x7f3ce00ace50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:37.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 -- 172.21.15.83:0/4002561402 >> 172.21.15.83:0/4002561402 conn(0x7f3ce00c90e0 msgr2=0x7f3ce00aa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:37.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 -- 172.21.15.83:0/4002561402 shutdown_connections 2024-09-19T07:28:37.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 -- 172.21.15.83:0/4002561402 wait complete. 2024-09-19T07:28:37.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 Processor -- start 2024-09-19T07:28:37.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 -- start start 2024-09-19T07:28:37.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ce00701b0 0x7f3ce00c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:37.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ce00ad390 0x7f3ce015aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:37.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce014ffb0 0x7f3ce015b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00c3260 con 0x7f3ce014ffb0 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00c30e0 con 0x7f3ce00701b0 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3ce514e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce006ef00 con 0x7f3ce00ad390 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3cdf59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce014ffb0 0x7f3ce015b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3cdf59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce014ffb0 0x7f3ce015b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46892/0 (socket says 172.21.15.83:46892) 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3cdf59e640 1 -- 172.21.15.83:0/3594734642 learned_addr learned my addr 172.21.15.83:0/3594734642 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3cd7fff640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ce00ad390 0x7f3ce015aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.091+0000 7f3cded9d640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ce00701b0 0x7f3ce00c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cdf59e640 1 -- 172.21.15.83:0/3594734642 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ce00ad390 msgr2=0x7f3ce015aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cdf59e640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ce00ad390 0x7f3ce015aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cdf59e640 1 -- 172.21.15.83:0/3594734642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ce00701b0 msgr2=0x7f3ce00c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cdf59e640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ce00701b0 0x7f3ce00c7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cdf59e640 1 -- 172.21.15.83:0/3594734642 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00736d0 con 0x7f3ce014ffb0 2024-09-19T07:28:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cdf59e640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce014ffb0 0x7f3ce015b410 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800efc0 tx=0x7f3cd800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cd77fe640 1 -- 172.21.15.83:0/3594734642 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8018020 con 0x7f3ce014ffb0 2024-09-19T07:28:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cd77fe640 1 -- 172.21.15.83:0/3594734642 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd8014480 con 0x7f3ce014ffb0 2024-09-19T07:28:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3ce514e640 1 -- 172.21.15.83:0/3594734642 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce015b950 con 0x7f3ce014ffb0 2024-09-19T07:28:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cd77fe640 1 -- 172.21.15.83:0/3594734642 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8014870 con 0x7f3ce014ffb0 2024-09-19T07:28:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3ce514e640 1 -- 172.21.15.83:0/3594734642 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce0073970 con 0x7f3ce014ffb0 2024-09-19T07:28:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3ce514e640 1 -- 172.21.15.83:0/3594734642 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce00ace50 con 0x7f3ce014ffb0 2024-09-19T07:28:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cd77fe640 1 -- 172.21.15.83:0/3594734642 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3cd801b020 con 0x7f3ce014ffb0 2024-09-19T07:28:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cd77fe640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3cc00678c0 0x7f3cc0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cded9d640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3cc00678c0 0x7f3cc0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cd77fe640 1 -- 172.21.15.83:0/3594734642 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7755+0+0 (secure 0 0 0) 0x7f3cd8087160 con 0x7f3ce014ffb0 2024-09-19T07:28:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.095+0000 7f3cded9d640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3cc00678c0 0x7f3cc0069d80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0000f70 tx=0x7f3cd0007e90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.099+0000 7f3cd77fe640 1 -- 172.21.15.83:0/3594734642 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3cd8022030 con 0x7f3ce014ffb0 2024-09-19T07:28:37.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.351+0000 7f3ce514e640 1 -- 172.21.15.83:0/3594734642 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ce00ac2b0 con 0x7f3ce014ffb0 2024-09-19T07:28:37.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.351+0000 7f3cd77fe640 1 -- 172.21.15.83:0/3594734642 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6534 (secure 0 0 0) 0x7f3cd8010040 con 0x7f3ce014ffb0 2024-09-19T07:28:37.347 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:37.347 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":79,"btime":"2024-09-19T07:28:16:180103+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:27:42.299316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/104517500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":104517500},{"type":"v1","addr":"172.21.15.83:6835","nonce":104517500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5934,"qdb_cluster":[5934]},"id":2}]} 2024-09-19T07:28:37.347 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 79 2024-09-19T07:28:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.355+0000 7f3ce514e640 1 -- 172.21.15.83:0/3594734642 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3cc00678c0 msgr2=0x7f3cc0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.355+0000 7f3ce514e640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3cc00678c0 0x7f3cc0069d80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0000f70 tx=0x7f3cd0007e90 comp rx=0 tx=0).stop 2024-09-19T07:28:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.355+0000 7f3ce514e640 1 -- 172.21.15.83:0/3594734642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce014ffb0 msgr2=0x7f3ce015b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.355+0000 7f3ce514e640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce014ffb0 0x7f3ce015b410 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800efc0 tx=0x7f3cd800c490 comp rx=0 tx=0).stop 2024-09-19T07:28:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.355+0000 7f3ce514e640 1 -- 172.21.15.83:0/3594734642 shutdown_connections 2024-09-19T07:28:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.355+0000 7f3ce514e640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3cc00678c0 0x7f3cc0069d80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.355+0000 7f3ce514e640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce014ffb0 0x7f3ce015b410 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.355+0000 7f3ce514e640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ce00ad390 0x7f3ce015aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.355+0000 7f3ce514e640 1 --2- 172.21.15.83:0/3594734642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ce00701b0 0x7f3ce00c7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.355+0000 7f3ce514e640 1 -- 172.21.15.83:0/3594734642 >> 172.21.15.83:0/3594734642 conn(0x7f3ce00c90e0 msgr2=0x7f3ce006bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.355+0000 7f3ce514e640 1 -- 172.21.15.83:0/3594734642 shutdown_connections 2024-09-19T07:28:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:37.355+0000 7f3ce514e640 1 -- 172.21.15.83:0/3594734642 wait complete. 2024-09-19T07:28:37.367 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:28:39.368 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:39.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.519+0000 7f2fbdfc4640 1 Processor -- start 2024-09-19T07:28:39.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.519+0000 7f2fbdfc4640 1 -- start start 2024-09-19T07:28:39.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fbdfc4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2fb8152b20 0x7f2fb80c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fbdfc4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fb80c45c0 0x7f2fb8153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fbdfc4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb8150670 con 0x7f2fb8150290 2024-09-19T07:28:39.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fbdfc4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb80716c0 con 0x7f2fb8152b20 2024-09-19T07:28:39.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fbdfc4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb80c3920 con 0x7f2fb80c45c0 2024-09-19T07:28:39.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2fb8152b20 0x7f2fb80c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2fb8152b20 0x7f2fb80c4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46906/0 (socket says 172.21.15.83:46906) 2024-09-19T07:28:39.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb6ffd640 1 -- 172.21.15.83:0/1548301883 learned_addr learned my addr 172.21.15.83:0/1548301883 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:39.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb7fff640 1 --2- 172.21.15.83:0/1548301883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fb80c45c0 0x7f2fb8153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb6ffd640 1 -- 172.21.15.83:0/1548301883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fb80c45c0 msgr2=0x7f2fb8153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:39.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb6ffd640 1 --2- 172.21.15.83:0/1548301883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fb80c45c0 0x7f2fb8153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb6ffd640 1 -- 172.21.15.83:0/1548301883 >> v1:172.21.15.83:6789/0 conn(0x7f2fb8150290 legacy=0x7f2fb80c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:39.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb6ffd640 1 -- 172.21.15.83:0/1548301883 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb80c2880 con 0x7f2fb8152b20 2024-09-19T07:28:39.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb7fff640 1 --2- 172.21.15.83:0/1548301883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fb80c45c0 0x7f2fb8153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:28:39.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb6ffd640 1 --2- 172.21.15.83:0/1548301883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb8152b20 0x7f2fb80c4080 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0098a0 tx=0x7f2fac01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f899a0ddf9bdeb4f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:39.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb67fc640 1 -- 172.21.15.83:0/1548301883 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fac0ab430 con 0x7f2fb8152b20 2024-09-19T07:28:39.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb67fc640 1 -- 172.21.15.83:0/1548301883 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fac0b0070 con 0x7f2fb8152b20 2024-09-19T07:28:39.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.523+0000 7f2fb67fc640 1 -- 172.21.15.83:0/1548301883 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fac0ab7f0 con 0x7f2fb8152b20 2024-09-19T07:28:39.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/1548301883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb8152b20 msgr2=0x7f2fb80c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:39.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 --2- 172.21.15.83:0/1548301883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb8152b20 0x7f2fb80c4080 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0098a0 tx=0x7f2fac01c540 comp rx=0 tx=0).stop 2024-09-19T07:28:39.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/1548301883 shutdown_connections 2024-09-19T07:28:39.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 --2- 172.21.15.83:0/1548301883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb8152b20 0x7f2fb80c4080 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 --2- 172.21.15.83:0/1548301883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fb80c45c0 0x7f2fb8153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/1548301883 >> 172.21.15.83:0/1548301883 conn(0x7f2fb806bc80 msgr2=0x7f2fb806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:39.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/1548301883 shutdown_connections 2024-09-19T07:28:39.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/1548301883 wait complete. 2024-09-19T07:28:39.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 Processor -- start 2024-09-19T07:28:39.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 -- start start 2024-09-19T07:28:39.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb80c45c0 0x7f2fb814edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fb8150290 0x7f2fb80c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fb8152b20 0x7f2fb81497c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb8154980 con 0x7f2fb80c45c0 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb8154800 con 0x7f2fb8150290 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb815fb60 con 0x7f2fb8152b20 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fb77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb80c45c0 0x7f2fb814edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fb77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb80c45c0 0x7f2fb814edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46916/0 (socket says 172.21.15.83:46916) 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fb77fe640 1 -- 172.21.15.83:0/3014608166 learned_addr learned my addr 172.21.15.83:0/3014608166 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fb7fff640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fb8152b20 0x7f2fb81497c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fb6ffd640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fb8150290 0x7f2fb80c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fb77fe640 1 -- 172.21.15.83:0/3014608166 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fb8152b20 msgr2=0x7f2fb81497c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fb77fe640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fb8152b20 0x7f2fb81497c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fb77fe640 1 -- 172.21.15.83:0/3014608166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fb8150290 msgr2=0x7f2fb80c0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fb77fe640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fb8150290 0x7f2fb80c0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fb77fe640 1 -- 172.21.15.83:0/3014608166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb814f560 con 0x7f2fb80c45c0 2024-09-19T07:28:39.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fb77fe640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb80c45c0 0x7f2fb814edf0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800c450 tx=0x7f2fa800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:39.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2f9ffff640 1 -- 172.21.15.83:0/3014608166 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8016020 con 0x7f2fb80c45c0 2024-09-19T07:28:39.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2f9ffff640 1 -- 172.21.15.83:0/3014608166 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa80052d0 con 0x7f2fb80c45c0 2024-09-19T07:28:39.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/3014608166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb80733f0 con 0x7f2fb80c45c0 2024-09-19T07:28:39.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2f9ffff640 1 -- 172.21.15.83:0/3014608166 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8007260 con 0x7f2fb80c45c0 2024-09-19T07:28:39.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.527+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/3014608166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb80739b0 con 0x7f2fb80c45c0 2024-09-19T07:28:39.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.531+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/3014608166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb8153eb0 con 0x7f2fb80c45c0 2024-09-19T07:28:39.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.531+0000 7f2f9ffff640 1 -- 172.21.15.83:0/3014608166 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2fa8007400 con 0x7f2fb80c45c0 2024-09-19T07:28:39.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.531+0000 7f2f9ffff640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2fa40678c0 0x7f2fa4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.531+0000 7f2f9ffff640 1 -- 172.21.15.83:0/3014608166 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7f2fa8085e20 con 0x7f2fb80c45c0 2024-09-19T07:28:39.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.531+0000 7f2fb6ffd640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2fa40678c0 0x7f2fa4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.531+0000 7f2fb6ffd640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2fa40678c0 0x7f2fa4069d80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0af000 tx=0x7f2fac008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:39.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.539+0000 7f2f9ffff640 1 -- 172.21.15.83:0/3014608166 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2fa8020020 con 0x7f2fb80c45c0 2024-09-19T07:28:39.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/3014608166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fb806c3a0 con 0x7f2fb80c45c0 2024-09-19T07:28:39.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2f9ffff640 1 -- 172.21.15.83:0/3014608166 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5708 (secure 0 0 0) 0x7f2fa8005480 con 0x7f2fb80c45c0 2024-09-19T07:28:39.782 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:39.782 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"btime":"2024-09-19T07:28:39:280656+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:38.332104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:28:39.782 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-09-19T07:28:39.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/3014608166 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2fa40678c0 msgr2=0x7f2fa4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:39.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2fa40678c0 0x7f2fa4069d80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0af000 tx=0x7f2fac008040 comp rx=0 tx=0).stop 2024-09-19T07:28:39.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/3014608166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb80c45c0 msgr2=0x7f2fb814edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:39.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb80c45c0 0x7f2fb814edf0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800c450 tx=0x7f2fa800c910 comp rx=0 tx=0).stop 2024-09-19T07:28:39.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/3014608166 shutdown_connections 2024-09-19T07:28:39.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2fa40678c0 0x7f2fa4069d80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fb8152b20 0x7f2fb81497c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fb8150290 0x7f2fb80c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 --2- 172.21.15.83:0/3014608166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fb80c45c0 0x7f2fb814edf0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/3014608166 >> 172.21.15.83:0/3014608166 conn(0x7f2fb806bc80 msgr2=0x7f2fb814c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:39.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/3014608166 shutdown_connections 2024-09-19T07:28:39.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.787+0000 7f2fbdfc4640 1 -- 172.21.15.83:0/3014608166 wait complete. 2024-09-19T07:28:39.801 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-19T07:28:39.801 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:28:39.801 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:39.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847b2e2640 1 Processor -- start 2024-09-19T07:28:39.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847b2e2640 1 -- start start 2024-09-19T07:28:39.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847b2e2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8474155d70 0x7f84740c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847b2e2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f84741592f0 0x7f8474157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847b2e2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8474156200 con 0x7f84740c6120 2024-09-19T07:28:39.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847b2e2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f84740c9930 con 0x7f84741592f0 2024-09-19T07:28:39.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847b2e2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f84741564e0 con 0x7f8474155d70 2024-09-19T07:28:39.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847907e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f84740c6120 0x7f84741575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60756/0 (socket says 172.21.15.83:60756) 2024-09-19T07:28:39.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847907e640 1 -- 172.21.15.83:0/2978275627 learned_addr learned my addr 172.21.15.83:0/2978275627 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:39.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1269349579 0 0) 0x7f8474156200 con 0x7f84740c6120 2024-09-19T07:28:39.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8460003530 con 0x7f84740c6120 2024-09-19T07:28:39.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847987f640 1 --2- 172.21.15.83:0/2978275627 >> v2:172.21.15.184:3300/0 conn(0x7f84741592f0 0x7f8474157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847887d640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8474155d70 0x7f84740c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3139653292 0 0) 0x7f8460003530 con 0x7f84740c6120 2024-09-19T07:28:39.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8474155d70 msgr2=0x7f84740c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:39.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8474155d70 0x7f84740c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 >> v2:172.21.15.184:3300/0 conn(0x7f84741592f0 msgr2=0x7f8474157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:39.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 --2- 172.21.15.83:0/2978275627 >> v2:172.21.15.184:3300/0 conn(0x7f84741592f0 0x7f8474157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84741567c0 con 0x7f84740c6120 2024-09-19T07:28:39.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f8470001680 con 0x7f84740c6120 2024-09-19T07:28:39.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847887d640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8474155d70 0x7f84740c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:28:39.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 >> v1:172.21.15.83:6789/0 conn(0x7f84740c6120 legacy=0x7f84741575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:39.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84600039c0 0x7f8460003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8460004330 0x7f8460004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8460004f30 0x7f846000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f84741564e0 con 0x7f8460004f30 2024-09-19T07:28:39.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f846000bf10 con 0x7f8460004330 2024-09-19T07:28:39.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f846000c090 con 0x7f84600039c0 2024-09-19T07:28:39.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847987f640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8460004f30 0x7f846000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847907e640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84600039c0 0x7f8460003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847887d640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8460004330 0x7f8460004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847987f640 1 -- 172.21.15.83:0/2978275627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84600039c0 msgr2=0x7f8460003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:39.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847987f640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84600039c0 0x7f8460003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847987f640 1 -- 172.21.15.83:0/2978275627 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8460004330 msgr2=0x7f8460004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:39.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847987f640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8460004330 0x7f8460004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.939+0000 7f847987f640 1 -- 172.21.15.83:0/2978275627 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8460003530 con 0x7f8460004f30 2024-09-19T07:28:39.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.943+0000 7f847987f640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8460004f30 0x7f846000b9d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f846c009fc0 tx=0x7f846c01ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c043313cf5a186b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:39.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.943+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c0b2070 con 0x7f8460004f30 2024-09-19T07:28:39.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.943+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f846c0a5cd0 con 0x7f8460004f30 2024-09-19T07:28:39.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.943+0000 7f846bfff640 1 -- 172.21.15.83:0/2978275627 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c009c10 con 0x7f8460004f30 2024-09-19T07:28:39.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.943+0000 7f847b2e2640 1 -- 172.21.15.83:0/2978275627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8460004f30 msgr2=0x7f846000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:39.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.943+0000 7f847b2e2640 1 --2- 172.21.15.83:0/2978275627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8460004f30 0x7f846000b9d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f846c009fc0 tx=0x7f846c01ccc0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.943+0000 7f847987f640 1 -- 172.21.15.83:0/2978275627 reap_dead start 2024-09-19T07:28:39.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.943+0000 7f847b2e2640 1 -- 172.21.15.83:0/2978275627 shutdown_connections 2024-09-19T07:28:39.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.943+0000 7f847b2e2640 1 -- 172.21.15.83:0/2978275627 >> 172.21.15.83:0/2978275627 conn(0x7f847406bc80 msgr2=0x7f847406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:39.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.943+0000 7f847b2e2640 1 -- 172.21.15.83:0/2978275627 shutdown_connections 2024-09-19T07:28:39.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.943+0000 7f847b2e2640 1 -- 172.21.15.83:0/2978275627 wait complete. 2024-09-19T07:28:39.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847b2e2640 1 Processor -- start 2024-09-19T07:28:39.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847b2e2640 1 -- start start 2024-09-19T07:28:39.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847b2e2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8460004f30 0x7f84740cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847b2e2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84740cbad0 0x7f847414b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847b2e2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f847414b600 0x7f84741ec2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847b2e2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f84740cc630 con 0x7f84740cbad0 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847b2e2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8474158220 con 0x7f847414b600 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847b2e2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8474153860 con 0x7f8460004f30 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847887d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84740cbad0 0x7f847414b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847887d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84740cbad0 0x7f847414b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46942/0 (socket says 172.21.15.83:46942) 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847887d640 1 -- 172.21.15.83:0/983834494 learned_addr learned my addr 172.21.15.83:0/983834494 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847907e640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8460004f30 0x7f84740cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847987f640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f847414b600 0x7f84741ec2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847887d640 1 -- 172.21.15.83:0/983834494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8460004f30 msgr2=0x7f84740cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847887d640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8460004f30 0x7f84740cb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847887d640 1 -- 172.21.15.83:0/983834494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f847414b600 msgr2=0x7f84741ec2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:39.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847887d640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f847414b600 0x7f84741ec2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:39.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847887d640 1 -- 172.21.15.83:0/983834494 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847414ba80 con 0x7f84740cbad0 2024-09-19T07:28:39.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847887d640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84740cbad0 0x7f847414b0c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f84640108c0 tx=0x7f8464010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:39.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f8469ffb640 1 -- 172.21.15.83:0/983834494 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846401a070 con 0x7f84740cbad0 2024-09-19T07:28:39.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f8469ffb640 1 -- 172.21.15.83:0/983834494 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f846400cb40 con 0x7f84740cbad0 2024-09-19T07:28:39.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847b2e2640 1 -- 172.21.15.83:0/983834494 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84740ab470 con 0x7f84740cbad0 2024-09-19T07:28:39.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f8469ffb640 1 -- 172.21.15.83:0/983834494 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846400f040 con 0x7f84740cbad0 2024-09-19T07:28:39.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847b2e2640 1 -- 172.21.15.83:0/983834494 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84740ac6c0 con 0x7f84740cbad0 2024-09-19T07:28:39.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.947+0000 7f847b2e2640 1 -- 172.21.15.83:0/983834494 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8474155d70 con 0x7f84740cbad0 2024-09-19T07:28:39.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.951+0000 7f8469ffb640 1 -- 172.21.15.83:0/983834494 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f846401d070 con 0x7f84740cbad0 2024-09-19T07:28:39.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.951+0000 7f8469ffb640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f84580678c0 0x7f8458069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:39.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.951+0000 7f8469ffb640 1 -- 172.21.15.83:0/983834494 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7f8464086de0 con 0x7f84740cbad0 2024-09-19T07:28:39.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.951+0000 7f847907e640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f84580678c0 0x7f8458069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:39.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.951+0000 7f847907e640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f84580678c0 0x7f8458069d80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8470001060 tx=0x7f8470009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:39.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:39.955+0000 7f8469ffb640 1 -- 172.21.15.83:0/983834494 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f847414ba80 con 0x7f84740cbad0 2024-09-19T07:28:40.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 -- 172.21.15.83:0/983834494 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84740c96a0 con 0x7f84740cbad0 2024-09-19T07:28:40.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f8469ffb640 1 -- 172.21.15.83:0/983834494 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5708 (secure 0 0 0) 0x7f84640571a0 con 0x7f84740cbad0 2024-09-19T07:28:40.191 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:40.191 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"btime":"2024-09-19T07:28:39:280656+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:38.332104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:28:40.191 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-09-19T07:28:40.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 -- 172.21.15.83:0/983834494 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f84580678c0 msgr2=0x7f8458069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:40.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f84580678c0 0x7f8458069d80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8470001060 tx=0x7f8470009040 comp rx=0 tx=0).stop 2024-09-19T07:28:40.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 -- 172.21.15.83:0/983834494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84740cbad0 msgr2=0x7f847414b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:40.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84740cbad0 0x7f847414b0c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f84640108c0 tx=0x7f8464010d80 comp rx=0 tx=0).stop 2024-09-19T07:28:40.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 -- 172.21.15.83:0/983834494 shutdown_connections 2024-09-19T07:28:40.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f84580678c0 0x7f8458069d80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:40.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f847414b600 0x7f84741ec2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:40.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84740cbad0 0x7f847414b0c0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:40.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 --2- 172.21.15.83:0/983834494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8460004f30 0x7f84740cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:40.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 -- 172.21.15.83:0/983834494 >> 172.21.15.83:0/983834494 conn(0x7f847406bc80 msgr2=0x7f84741574d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:40.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 -- 172.21.15.83:0/983834494 shutdown_connections 2024-09-19T07:28:40.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:40.195+0000 7f847b2e2640 1 -- 172.21.15.83:0/983834494 wait complete. 2024-09-19T07:28:40.209 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:28:39:280656+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 81, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 78, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5991": { "addr": "172.21.15.83:6837/1229829432", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 1229829432, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 1229829432, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5991, "incarnation": 75, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:28:11.164127+0000", "qdb_cluster": [ 5991 ], "qdb_leader": 5991, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5991 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 81, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5476": { "addr": "172.21.15.184:6835/3841282254", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3841282254, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3841282254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5476, "incarnation": 80, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:28:38.332104+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5476 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 28, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:28:42.210 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:42.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f8722983640 1 Processor -- start 2024-09-19T07:28:42.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f8722983640 1 -- start start 2024-09-19T07:28:42.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f8722983640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f871c06f950 0x7f871c0716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f8722983640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f871c0c37c0 con 0x7f871c071e20 2024-09-19T07:28:42.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f8722983640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f871c0c4670 con 0x7f871c1515e0 2024-09-19T07:28:42.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f8722983640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f871c1519f0 con 0x7f871c06f950 2024-09-19T07:28:42.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f87137fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f871c071e20 0x7f871c150ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48422/0 (socket says 172.21.15.83:48422) 2024-09-19T07:28:42.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f87137fe640 1 -- 172.21.15.83:0/2382758047 learned_addr learned my addr 172.21.15.83:0/2382758047 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:42.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871bfff640 1 --2- 172.21.15.83:0/2382758047 >> v2:172.21.15.184:3300/0 conn(0x7f871c06f950 0x7f871c0716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1851874097 0 0) 0x7f871c0c37c0 con 0x7f871c071e20 2024-09-19T07:28:42.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8708003530 con 0x7f871c071e20 2024-09-19T07:28:42.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4185252277 0 0) 0x7f8708003530 con 0x7f871c071e20 2024-09-19T07:28:42.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 >> v2:172.21.15.184:3300/0 conn(0x7f871c06f950 msgr2=0x7f871c0716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:42.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 --2- 172.21.15.83:0/2382758047 >> v2:172.21.15.184:3300/0 conn(0x7f871c06f950 0x7f871c0716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 >> v1:172.21.15.184:6789/0 conn(0x7f871c1515e0 legacy=0x7f871c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871c0c3a60 con 0x7f871c071e20 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f8704001680 con 0x7f871c071e20 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 >> v1:172.21.15.83:6789/0 conn(0x7f871c071e20 legacy=0x7f871c150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 --2- 172.21.15.83:0/2382758047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 0x7f8708003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 --2- 172.21.15.83:0/2382758047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8708004810 0x7f8708004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 --2- 172.21.15.83:0/2382758047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87080051d0 0x7f870800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f871c1519f0 con 0x7f87080039c0 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f870800c170 con 0x7f8708004810 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f870800c370 con 0x7f87080051d0 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871bfff640 1 --2- 172.21.15.83:0/2382758047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 0x7f8708003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871bfff640 1 -- 172.21.15.83:0/2382758047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87080051d0 msgr2=0x7f870800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871bfff640 1 --2- 172.21.15.83:0/2382758047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87080051d0 0x7f870800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871bfff640 1 -- 172.21.15.83:0/2382758047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8708004810 msgr2=0x7f8708004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:42.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871bfff640 1 --2- 172.21.15.83:0/2382758047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8708004810 0x7f8708004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f871bfff640 1 -- 172.21.15.83:0/2382758047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8708003530 con 0x7f87080039c0 2024-09-19T07:28:42.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.351+0000 7f8720f20640 1 -- 172.21.15.83:0/2382758047 reap_dead start 2024-09-19T07:28:42.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.355+0000 7f871bfff640 1 --2- 172.21.15.83:0/2382758047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 0x7f8708003de0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f870c009fc0 tx=0x7f870c01cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66793e5d29f74f3d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:42.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.355+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c0b2070 con 0x7f87080039c0 2024-09-19T07:28:42.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.355+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f870c0a5b50 con 0x7f87080039c0 2024-09-19T07:28:42.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.355+0000 7f871b7fe640 1 -- 172.21.15.83:0/2382758047 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c009c10 con 0x7f87080039c0 2024-09-19T07:28:42.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.355+0000 7f8722983640 1 -- 172.21.15.83:0/2382758047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 msgr2=0x7f8708003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.355+0000 7f8722983640 1 --2- 172.21.15.83:0/2382758047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 0x7f8708003de0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f870c009fc0 tx=0x7f870c01cc50 comp rx=0 tx=0).stop 2024-09-19T07:28:42.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.355+0000 7f8722983640 1 -- 172.21.15.83:0/2382758047 shutdown_connections 2024-09-19T07:28:42.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.355+0000 7f8722983640 1 --2- 172.21.15.83:0/2382758047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 0x7f8708003de0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.355+0000 7f8722983640 1 -- 172.21.15.83:0/2382758047 >> 172.21.15.83:0/2382758047 conn(0x7f871c0c48b0 msgr2=0x7f871c0c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.355+0000 7f8722983640 1 -- 172.21.15.83:0/2382758047 shutdown_connections 2024-09-19T07:28:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.355+0000 7f8722983640 1 -- 172.21.15.83:0/2382758047 wait complete. 2024-09-19T07:28:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f8722983640 1 Processor -- start 2024-09-19T07:28:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f8722983640 1 -- start start 2024-09-19T07:28:42.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f8722983640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 0x7f871c1f0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f8722983640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f871c1e0d30 0x7f871c152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f8722983640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f871c0c1120 0x7f871c0c15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f8722983640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f871c151460 con 0x7f87080039c0 2024-09-19T07:28:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f8722983640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f871c1512e0 con 0x7f871c1e0d30 2024-09-19T07:28:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f8722983640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f871c06fd30 con 0x7f871c0c1120 2024-09-19T07:28:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f871bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 0x7f871c1f0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f871bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 0x7f871c1f0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51208/0 (socket says 172.21.15.83:51208) 2024-09-19T07:28:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f871bfff640 1 -- 172.21.15.83:0/279810908 learned_addr learned my addr 172.21.15.83:0/279810908 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f87137fe640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f871c1e0d30 0x7f871c152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f871bfff640 1 -- 172.21.15.83:0/279810908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f871c0c1120 msgr2=0x7f871c0c15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f8720f20640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f871c0c1120 0x7f871c0c15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f871bfff640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f871c0c1120 0x7f871c0c15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f871bfff640 1 -- 172.21.15.83:0/279810908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f871c1e0d30 msgr2=0x7f871c152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f871bfff640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f871c1e0d30 0x7f871c152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f871bfff640 1 -- 172.21.15.83:0/279810908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871c1f0bb0 con 0x7f87080039c0 2024-09-19T07:28:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f87137fe640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f871c1e0d30 0x7f871c152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:28:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f871bfff640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 0x7f871c1f0670 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f870c098670 tx=0x7f870c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f87197fa640 1 -- 172.21.15.83:0/279810908 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c0bc020 con 0x7f87080039c0 2024-09-19T07:28:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f87197fa640 1 -- 172.21.15.83:0/279810908 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f870c0ada40 con 0x7f87080039c0 2024-09-19T07:28:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f8722983640 1 -- 172.21.15.83:0/279810908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f871c1e12f0 con 0x7f87080039c0 2024-09-19T07:28:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f87197fa640 1 -- 172.21.15.83:0/279810908 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c0b2040 con 0x7f87080039c0 2024-09-19T07:28:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f8722983640 1 -- 172.21.15.83:0/279810908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f871c0bd470 con 0x7f87080039c0 2024-09-19T07:28:42.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.359+0000 7f8722983640 1 -- 172.21.15.83:0/279810908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f871c0bcfb0 con 0x7f87080039c0 2024-09-19T07:28:42.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.363+0000 7f87197fa640 1 -- 172.21.15.83:0/279810908 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f870c0b6410 con 0x7f87080039c0 2024-09-19T07:28:42.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.363+0000 7f87197fa640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f86fc0678c0 0x7f86fc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.363+0000 7f87197fa640 1 -- 172.21.15.83:0/279810908 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7f870c120af0 con 0x7f87080039c0 2024-09-19T07:28:42.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.363+0000 7f87137fe640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f86fc0678c0 0x7f86fc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.363+0000 7f87137fe640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f86fc0678c0 0x7f86fc069d80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8704001ec0 tx=0x7f8704009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:42.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.367+0000 7f87197fa640 1 -- 172.21.15.83:0/279810908 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f871c0bcfb0 con 0x7f87080039c0 2024-09-19T07:28:42.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.583+0000 7f8722983640 1 -- 172.21.15.83:0/279810908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f871c157dc0 con 0x7f87080039c0 2024-09-19T07:28:42.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.583+0000 7f87197fa640 1 -- 172.21.15.83:0/279810908 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5712 (secure 0 0 0) 0x7f870c0c6060 con 0x7f87080039c0 2024-09-19T07:28:42.578 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:42.578 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"btime":"2024-09-19T07:28:41:296963+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:28:42.578 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-09-19T07:28:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.583+0000 7f8722983640 1 -- 172.21.15.83:0/279810908 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f86fc0678c0 msgr2=0x7f86fc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.583+0000 7f8722983640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f86fc0678c0 0x7f86fc069d80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8704001ec0 tx=0x7f8704009040 comp rx=0 tx=0).stop 2024-09-19T07:28:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.583+0000 7f8722983640 1 -- 172.21.15.83:0/279810908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 msgr2=0x7f871c1f0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.583+0000 7f8722983640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 0x7f871c1f0670 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f870c098670 tx=0x7f870c009750 comp rx=0 tx=0).stop 2024-09-19T07:28:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.583+0000 7f8722983640 1 -- 172.21.15.83:0/279810908 shutdown_connections 2024-09-19T07:28:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.583+0000 7f8722983640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f86fc0678c0 0x7f86fc069d80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.583+0000 7f8722983640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f871c0c1120 0x7f871c0c15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.583+0000 7f8722983640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f871c1e0d30 0x7f871c152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.583+0000 7f8722983640 1 --2- 172.21.15.83:0/279810908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87080039c0 0x7f871c1f0670 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.587+0000 7f8722983640 1 -- 172.21.15.83:0/279810908 >> 172.21.15.83:0/279810908 conn(0x7f871c0c48b0 msgr2=0x7f871c1545d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:42.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.587+0000 7f8722983640 1 -- 172.21.15.83:0/279810908 shutdown_connections 2024-09-19T07:28:42.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.587+0000 7f8722983640 1 -- 172.21.15.83:0/279810908 wait complete. 2024-09-19T07:28:42.600 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=5934 2024-09-19T07:28:42.601 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.a 2024-09-19T07:28:42.601 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-19T07:28:42.601 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:28:42.601 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:28:42.603 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:28:42.603 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:42.676 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:28:42.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.735+0000 7f1f0f076640 1 Processor -- start 2024-09-19T07:28:42.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.735+0000 7f1f0f076640 1 -- start start 2024-09-19T07:28:42.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.735+0000 7f1f0f076640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0806cef0 0x7f1f080abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.735+0000 7f1f0f076640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1f080ab610 0x7f1f080c3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.735+0000 7f1f0f076640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f080c7de0 con 0x7f1f080c6fc0 2024-09-19T07:28:42.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.735+0000 7f1f0f076640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f1f080c6a90 con 0x7f1f080ab610 2024-09-19T07:28:42.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.735+0000 7f1f0f076640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f080c8360 con 0x7f1f0806cef0 2024-09-19T07:28:42.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.735+0000 7f1f0d613640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1f080c6fc0 0x7f1f08157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48450/0 (socket says 172.21.15.83:48450) 2024-09-19T07:28:42.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.735+0000 7f1f0d613640 1 -- 172.21.15.83:0/3437775850 learned_addr learned my addr 172.21.15.83:0/3437775850 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:42.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.735+0000 7f1f0ce12640 1 --2- 172.21.15.83:0/3437775850 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0806cef0 0x7f1f080abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.735+0000 7f1f07fff640 1 --2- 172.21.15.83:0/3437775850 >> v2:172.21.15.184:3300/0 conn(0x7f1f080ab610 0x7f1f080c3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 388466193 0 0) 0x7f1f080c7de0 con 0x7f1f080c6fc0 2024-09-19T07:28:42.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ef8003530 con 0x7f1f080c6fc0 2024-09-19T07:28:42.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1501249422 0 0) 0x7f1ef8003530 con 0x7f1f080c6fc0 2024-09-19T07:28:42.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0806cef0 msgr2=0x7f1f080abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:42.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 --2- 172.21.15.83:0/3437775850 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0806cef0 0x7f1f080abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 >> v2:172.21.15.184:3300/0 conn(0x7f1f080ab610 msgr2=0x7f1f080c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:42.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 --2- 172.21.15.83:0/3437775850 >> v2:172.21.15.184:3300/0 conn(0x7f1f080ab610 0x7f1f080c3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f080c8080 con 0x7f1f080c6fc0 2024-09-19T07:28:42.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f1f00001a00 con 0x7f1f080c6fc0 2024-09-19T07:28:42.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 >> v1:172.21.15.83:6789/0 conn(0x7f1f080c6fc0 legacy=0x7f1f08157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 --2- 172.21.15.83:0/3437775850 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0806cef0 0x7f1ef80039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 --2- 172.21.15.83:0/3437775850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ef8003f40 0x7f1ef8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 --2- 172.21.15.83:0/3437775850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ef8004d50 0x7f1ef80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1f080c8360 con 0x7f1ef8004d50 2024-09-19T07:28:42.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef800ba30 con 0x7f1ef8003f40 2024-09-19T07:28:42.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1ef800bc30 con 0x7f1f0806cef0 2024-09-19T07:28:42.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f0d613640 1 --2- 172.21.15.83:0/3437775850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ef8004d50 0x7f1ef80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f0ce12640 1 --2- 172.21.15.83:0/3437775850 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0806cef0 0x7f1ef80039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f07fff640 1 --2- 172.21.15.83:0/3437775850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ef8003f40 0x7f1ef8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f0d613640 1 -- 172.21.15.83:0/3437775850 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0806cef0 msgr2=0x7f1ef80039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:42.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f0d613640 1 --2- 172.21.15.83:0/3437775850 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0806cef0 0x7f1ef80039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f0d613640 1 -- 172.21.15.83:0/3437775850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ef8003f40 msgr2=0x7f1ef8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:42.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f0d613640 1 --2- 172.21.15.83:0/3437775850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ef8003f40 0x7f1ef8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f0d613640 1 -- 172.21.15.83:0/3437775850 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef8003530 con 0x7f1ef8004d50 2024-09-19T07:28:42.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f0d613640 1 --2- 172.21.15.83:0/3437775850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ef8004d50 0x7f1ef80051d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000a8a0 tx=0x7f1f0001d390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0fd8d20c8a2de7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:42.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0000a4c0 con 0x7f1ef8004d50 2024-09-19T07:28:42.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f000ad630 con 0x7f1ef8004d50 2024-09-19T07:28:42.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.739+0000 7f1f077fe640 1 -- 172.21.15.83:0/3437775850 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f000ada20 con 0x7f1ef8004d50 2024-09-19T07:28:42.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 -- 172.21.15.83:0/3437775850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ef8004d50 msgr2=0x7f1ef80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 --2- 172.21.15.83:0/3437775850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ef8004d50 0x7f1ef80051d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000a8a0 tx=0x7f1f0001d390 comp rx=0 tx=0).stop 2024-09-19T07:28:42.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0d613640 1 -- 172.21.15.83:0/3437775850 reap_dead start 2024-09-19T07:28:42.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 -- 172.21.15.83:0/3437775850 shutdown_connections 2024-09-19T07:28:42.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 -- 172.21.15.83:0/3437775850 >> 172.21.15.83:0/3437775850 conn(0x7f1f080aa910 msgr2=0x7f1f080c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:42.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 -- 172.21.15.83:0/3437775850 shutdown_connections 2024-09-19T07:28:42.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 -- 172.21.15.83:0/3437775850 wait complete. 2024-09-19T07:28:42.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 Processor -- start 2024-09-19T07:28:42.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 -- start start 2024-09-19T07:28:42.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0814fd00 0x7f1f081500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f080a7e30 0x7f1f080a9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f0815a8c0 0x7f1f081ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0806d450 con 0x7f1f080a7e30 2024-09-19T07:28:42.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0806d2d0 con 0x7f1f0815a8c0 2024-09-19T07:28:42.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f08157fa0 con 0x7f1f0814fd00 2024-09-19T07:28:42.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f07fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f080a7e30 0x7f1f080a9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f07fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f080a7e30 0x7f1f080a9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51246/0 (socket says 172.21.15.83:51246) 2024-09-19T07:28:42.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f07fff640 1 -- 172.21.15.83:0/3091247707 learned_addr learned my addr 172.21.15.83:0/3091247707 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:42.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0ce12640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0814fd00 0x7f1f081500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0d613640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f0815a8c0 0x7f1f081ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f07fff640 1 -- 172.21.15.83:0/3091247707 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0814fd00 msgr2=0x7f1f081500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f07fff640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0814fd00 0x7f1f081500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f07fff640 1 -- 172.21.15.83:0/3091247707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f0815a8c0 msgr2=0x7f1f081ef5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f07fff640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f0815a8c0 0x7f1f081ef5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f07fff640 1 -- 172.21.15.83:0/3091247707 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0814e320 con 0x7f1f080a7e30 2024-09-19T07:28:42.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0ce12640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0814fd00 0x7f1f081500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:28:42.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f07fff640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f080a7e30 0x7f1f080a9aa0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40115c0 tx=0x7f1ef4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:42.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f057fa640 1 -- 172.21.15.83:0/3091247707 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4019070 con 0x7f1f080a7e30 2024-09-19T07:28:42.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f057fa640 1 -- 172.21.15.83:0/3091247707 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ef4003620 con 0x7f1f080a7e30 2024-09-19T07:28:42.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 -- 172.21.15.83:0/3091247707 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f0814e040 con 0x7f1f080a7e30 2024-09-19T07:28:42.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f057fa640 1 -- 172.21.15.83:0/3091247707 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4003a10 con 0x7f1f080a7e30 2024-09-19T07:28:42.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.743+0000 7f1f0f076640 1 -- 172.21.15.83:0/3091247707 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f081f00e0 con 0x7f1f080a7e30 2024-09-19T07:28:42.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.747+0000 7f1f0f076640 1 -- 172.21.15.83:0/3091247707 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f080c7d40 con 0x7f1f080a7e30 2024-09-19T07:28:42.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.747+0000 7f1f057fa640 1 -- 172.21.15.83:0/3091247707 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1ef401c070 con 0x7f1f080a7e30 2024-09-19T07:28:42.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.747+0000 7f1f057fa640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1eec0678f0 0x7f1eec069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:42.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.747+0000 7f1f057fa640 1 -- 172.21.15.83:0/3091247707 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7f1ef40864b0 con 0x7f1f080a7e30 2024-09-19T07:28:42.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.747+0000 7f1f0ce12640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1eec0678f0 0x7f1eec069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:42.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.747+0000 7f1f0ce12640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1eec0678f0 0x7f1eec069db0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1efc009af0 tx=0x7f1efc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:42.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.751+0000 7f1f057fa640 1 -- 172.21.15.83:0/3091247707 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1ef4056820 con 0x7f1f080a7e30 2024-09-19T07:28:42.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.987+0000 7f1f0f076640 1 -- 172.21.15.83:0/3091247707 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f08155b90 con 0x7f1f080a7e30 2024-09-19T07:28:42.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.987+0000 7f1f057fa640 1 -- 172.21.15.83:0/3091247707 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5712 (secure 0 0 0) 0x7f1ef4056120 con 0x7f1f080a7e30 2024-09-19T07:28:42.983 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:42.983 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"btime":"2024-09-19T07:28:41:296963+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:28:42.984 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-09-19T07:28:42.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.991+0000 7f1f0f076640 1 -- 172.21.15.83:0/3091247707 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1eec0678f0 msgr2=0x7f1eec069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.991+0000 7f1f0f076640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1eec0678f0 0x7f1eec069db0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1efc009af0 tx=0x7f1efc008040 comp rx=0 tx=0).stop 2024-09-19T07:28:42.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.991+0000 7f1f0f076640 1 -- 172.21.15.83:0/3091247707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f080a7e30 msgr2=0x7f1f080a9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:42.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.991+0000 7f1f0f076640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f080a7e30 0x7f1f080a9aa0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40115c0 tx=0x7f1ef4009f90 comp rx=0 tx=0).stop 2024-09-19T07:28:42.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.991+0000 7f1f0f076640 1 -- 172.21.15.83:0/3091247707 shutdown_connections 2024-09-19T07:28:42.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.991+0000 7f1f0f076640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1eec0678f0 0x7f1eec069db0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.991+0000 7f1f0f076640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f0815a8c0 0x7f1f081ef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.991+0000 7f1f0f076640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f080a7e30 0x7f1f080a9aa0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.991+0000 7f1f0f076640 1 --2- 172.21.15.83:0/3091247707 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f0814fd00 0x7f1f081500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:42.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.991+0000 7f1f0f076640 1 -- 172.21.15.83:0/3091247707 >> 172.21.15.83:0/3091247707 conn(0x7f1f080aa910 msgr2=0x7f1f080c41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:42.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.991+0000 7f1f0f076640 1 -- 172.21.15.83:0/3091247707 shutdown_connections 2024-09-19T07:28:42.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:42.991+0000 7f1f0f076640 1 -- 172.21.15.83:0/3091247707 wait complete. 2024-09-19T07:28:43.002 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-19T07:28:45.003 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:28:45.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f13492640 1 Processor -- start 2024-09-19T07:28:45.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f13492640 1 -- start start 2024-09-19T07:28:45.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f13492640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6f0c0c7d70 0x7f6f0c157c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:45.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f13492640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6f0c072d00 0x7f6f0c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:45.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f13492640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0c0c74f0 con 0x7f6f0c1547b0 2024-09-19T07:28:45.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f13492640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6f0c073690 con 0x7f6f0c072d00 2024-09-19T07:28:45.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f13492640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6f0c157160 con 0x7f6f0c0c7d70 2024-09-19T07:28:45.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f10a2d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6f0c1547b0 0x7f6f0c0c6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48460/0 (socket says 172.21.15.83:48460) 2024-09-19T07:28:45.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f10a2d640 1 -- 172.21.15.83:0/244150168 learned_addr learned my addr 172.21.15.83:0/244150168 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:28:45.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f11a2f640 1 --2- 172.21.15.83:0/244150168 >> v2:172.21.15.83:3300/0 conn(0x7f6f0c072d00 0x7f6f0c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:45.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f1122e640 1 --2- 172.21.15.83:0/244150168 >> v2:172.21.15.184:3300/0 conn(0x7f6f0c0c7d70 0x7f6f0c157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:45.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f11a2f640 1 -- 172.21.15.83:0/244150168 >> v2:172.21.15.184:3300/0 conn(0x7f6f0c0c7d70 msgr2=0x7f6f0c157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:28:45.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f11a2f640 1 --2- 172.21.15.83:0/244150168 >> v2:172.21.15.184:3300/0 conn(0x7f6f0c0c7d70 0x7f6f0c157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:45.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f11a2f640 1 -- 172.21.15.83:0/244150168 >> v1:172.21.15.83:6789/0 conn(0x7f6f0c1547b0 legacy=0x7f6f0c0c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:45.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f11a2f640 1 -- 172.21.15.83:0/244150168 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0c157440 con 0x7f6f0c072d00 2024-09-19T07:28:45.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.147+0000 7f6f1122e640 1 --2- 172.21.15.83:0/244150168 >> v2:172.21.15.184:3300/0 conn(0x7f6f0c0c7d70 0x7f6f0c157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:28:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f11a2f640 1 --2- 172.21.15.83:0/244150168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f0c072d00 0x7f6f0c06cc90 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c076000 tx=0x7f6f0401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd0386e3f124452 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f03fff640 1 -- 172.21.15.83:0/244150168 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f040b2070 con 0x7f6f0c072d00 2024-09-19T07:28:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f03fff640 1 -- 172.21.15.83:0/244150168 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f040a5890 con 0x7f6f0c072d00 2024-09-19T07:28:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f03fff640 1 -- 172.21.15.83:0/244150168 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f040a5c60 con 0x7f6f0c072d00 2024-09-19T07:28:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f13492640 1 -- 172.21.15.83:0/244150168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f0c072d00 msgr2=0x7f6f0c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f13492640 1 --2- 172.21.15.83:0/244150168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f0c072d00 0x7f6f0c06cc90 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c076000 tx=0x7f6f0401c870 comp rx=0 tx=0).stop 2024-09-19T07:28:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f13492640 1 -- 172.21.15.83:0/244150168 shutdown_connections 2024-09-19T07:28:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f13492640 1 --2- 172.21.15.83:0/244150168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f0c072d00 0x7f6f0c06cc90 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f13492640 1 --2- 172.21.15.83:0/244150168 >> v2:172.21.15.184:3300/0 conn(0x7f6f0c0c7d70 0x7f6f0c157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f13492640 1 -- 172.21.15.83:0/244150168 >> 172.21.15.83:0/244150168 conn(0x7f6f0c0c5060 msgr2=0x7f6f0c0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:45.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f13492640 1 -- 172.21.15.83:0/244150168 shutdown_connections 2024-09-19T07:28:45.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f13492640 1 -- 172.21.15.83:0/244150168 wait complete. 2024-09-19T07:28:45.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f13492640 1 Processor -- start 2024-09-19T07:28:45.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.151+0000 7f6f13492640 1 -- start start 2024-09-19T07:28:45.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f13492640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f0c072d00 0x7f6f0c14fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f13492640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f0c0c7d70 0x7f6f0c150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f13492640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f0c1547b0 0x7f6f0c1517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f13492640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c0703b0 con 0x7f6f0c072d00 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f13492640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c070230 con 0x7f6f0c0c7d70 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f13492640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0c070530 con 0x7f6f0c1547b0 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f1122e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f0c072d00 0x7f6f0c14fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f1122e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f0c072d00 0x7f6f0c14fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51258/0 (socket says 172.21.15.83:51258) 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f1122e640 1 -- 172.21.15.83:0/1144055385 learned_addr learned my addr 172.21.15.83:0/1144055385 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f11a2f640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f0c1547b0 0x7f6f0c1517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f10a2d640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f0c0c7d70 0x7f6f0c150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f1122e640 1 -- 172.21.15.83:0/1144055385 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f0c1547b0 msgr2=0x7f6f0c1517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f1122e640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f0c1547b0 0x7f6f0c1517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:45.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f1122e640 1 -- 172.21.15.83:0/1144055385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f0c0c7d70 msgr2=0x7f6f0c150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:45.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f1122e640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f0c0c7d70 0x7f6f0c150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:45.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f1122e640 1 -- 172.21.15.83:0/1144055385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0c1ecd90 con 0x7f6f0c072d00 2024-09-19T07:28:45.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f1122e640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f0c072d00 0x7f6f0c14fcf0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800c7f0 tx=0x7f6f0800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:45.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f01ffb640 1 -- 172.21.15.83:0/1144055385 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f08007a30 con 0x7f6f0c072d00 2024-09-19T07:28:45.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f01ffb640 1 -- 172.21.15.83:0/1144055385 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f08007bd0 con 0x7f6f0c072d00 2024-09-19T07:28:45.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f13492640 1 -- 172.21.15.83:0/1144055385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0c152ee0 con 0x7f6f0c072d00 2024-09-19T07:28:45.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f01ffb640 1 -- 172.21.15.83:0/1144055385 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f080113f0 con 0x7f6f0c072d00 2024-09-19T07:28:45.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f13492640 1 -- 172.21.15.83:0/1144055385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f0c150fb0 con 0x7f6f0c072d00 2024-09-19T07:28:45.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f13492640 1 -- 172.21.15.83:0/1144055385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f0c150da0 con 0x7f6f0c072d00 2024-09-19T07:28:45.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f01ffb640 1 -- 172.21.15.83:0/1144055385 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6f08011590 con 0x7f6f0c072d00 2024-09-19T07:28:45.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f01ffb640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6ef00678c0 0x7f6ef0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:28:45.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f10a2d640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6ef00678c0 0x7f6ef0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:28:45.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.155+0000 7f6f01ffb640 1 -- 172.21.15.83:0/1144055385 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7f6f08086640 con 0x7f6f0c072d00 2024-09-19T07:28:45.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.159+0000 7f6f10a2d640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6ef00678c0 0x7f6ef0069d80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c1540c0 tx=0x7f6efc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:28:45.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.163+0000 7f6f01ffb640 1 -- 172.21.15.83:0/1144055385 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6f08015020 con 0x7f6f0c072d00 2024-09-19T07:28:45.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.423+0000 7f6f13492640 1 -- 172.21.15.83:0/1144055385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f0c06cc90 con 0x7f6f0c072d00 2024-09-19T07:28:45.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f01ffb640 1 -- 172.21.15.83:0/1144055385 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+6537 (secure 0 0 0) 0x7f6f08056990 con 0x7f6f0c072d00 2024-09-19T07:28:45.424 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:28:45.425 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":84,"btime":"2024-09-19T07:28:43:303745+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:28:45.425 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 84 2024-09-19T07:28:45.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f13492640 1 -- 172.21.15.83:0/1144055385 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6ef00678c0 msgr2=0x7f6ef0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:45.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f13492640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6ef00678c0 0x7f6ef0069d80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c1540c0 tx=0x7f6efc009040 comp rx=0 tx=0).stop 2024-09-19T07:28:45.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f13492640 1 -- 172.21.15.83:0/1144055385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f0c072d00 msgr2=0x7f6f0c14fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:28:45.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f13492640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f0c072d00 0x7f6f0c14fcf0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800c7f0 tx=0x7f6f0800ccb0 comp rx=0 tx=0).stop 2024-09-19T07:28:45.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f13492640 1 -- 172.21.15.83:0/1144055385 shutdown_connections 2024-09-19T07:28:45.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f13492640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6ef00678c0 0x7f6ef0069d80 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:45.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f13492640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f0c1547b0 0x7f6f0c1517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:45.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f13492640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f0c0c7d70 0x7f6f0c150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:45.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f13492640 1 --2- 172.21.15.83:0/1144055385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f0c072d00 0x7f6f0c14fcf0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:28:45.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f13492640 1 -- 172.21.15.83:0/1144055385 >> 172.21.15.83:0/1144055385 conn(0x7f6f0c0c5060 msgr2=0x7f6f0c0c5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:28:45.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f13492640 1 -- 172.21.15.83:0/1144055385 shutdown_connections 2024-09-19T07:28:45.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:28:45.427+0000 7f6f13492640 1 -- 172.21.15.83:0/1144055385 wait complete. 2024-09-19T07:28:45.442 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-19T07:28:45.442 INFO:tasks.mds_thrash.fs.[b]:waiting for 104 secs before thrashing 2024-09-19T07:28:46.195 INFO:tasks.ceph.mds.b.smithi184.stderr:2024-09-19T07:28:46.198+0000 7fee378a7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:29:02.351 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:29:02.354 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:29:29.945 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:30.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff37a08640 1 Processor -- start 2024-09-19T07:29:30.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff37a08640 1 -- start start 2024-09-19T07:29:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff37a08640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7eff3006ffe0 0x7eff300c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff37a08640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff300c26c0 con 0x7eff30159af0 2024-09-19T07:29:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff37a08640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff300c29a0 con 0x7eff301535d0 2024-09-19T07:29:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff37a08640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7eff300707e0 con 0x7eff3006ffe0 2024-09-19T07:29:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff35fa5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7eff30159af0 0x7eff300c2f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60388/0 (socket says 172.21.15.83:60388) 2024-09-19T07:29:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff35fa5640 1 -- 172.21.15.83:0/3116436594 learned_addr learned my addr 172.21.15.83:0/3116436594 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:29:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff34fa3640 1 --2- 172.21.15.83:0/3116436594 >> v2:172.21.15.83:3300/0 conn(0x7eff3006ffe0 0x7eff300c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff27fff640 1 -- 172.21.15.83:0/3116436594 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890713155 0 0) 0x7eff300c26c0 con 0x7eff30159af0 2024-09-19T07:29:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff27fff640 1 -- 172.21.15.83:0/3116436594 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff1c003530 con 0x7eff30159af0 2024-09-19T07:29:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff34fa3640 1 -- 172.21.15.83:0/3116436594 >> v1:172.21.15.184:6789/0 conn(0x7eff301535d0 legacy=0x7eff30155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff34fa3640 1 -- 172.21.15.83:0/3116436594 >> v1:172.21.15.83:6789/0 conn(0x7eff30159af0 legacy=0x7eff300c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.094+0000 7eff34fa3640 1 -- 172.21.15.83:0/3116436594 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff301592f0 con 0x7eff3006ffe0 2024-09-19T07:29:30.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff34fa3640 1 --2- 172.21.15.83:0/3116436594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff3006ffe0 0x7eff300c6460 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7eff200097c0 tx=0x7eff2001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1fcf0f7626f11ce server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:30.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff27fff640 1 -- 172.21.15.83:0/3116436594 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff200b0070 con 0x7eff3006ffe0 2024-09-19T07:29:30.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff27fff640 1 -- 172.21.15.83:0/3116436594 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff200ab600 con 0x7eff3006ffe0 2024-09-19T07:29:30.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff27fff640 1 -- 172.21.15.83:0/3116436594 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff200ab9d0 con 0x7eff3006ffe0 2024-09-19T07:29:30.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff37a08640 1 -- 172.21.15.83:0/3116436594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff3006ffe0 msgr2=0x7eff300c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:30.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff37a08640 1 --2- 172.21.15.83:0/3116436594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff3006ffe0 0x7eff300c6460 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7eff200097c0 tx=0x7eff2001c340 comp rx=0 tx=0).stop 2024-09-19T07:29:30.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff37a08640 1 -- 172.21.15.83:0/3116436594 shutdown_connections 2024-09-19T07:29:30.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff37a08640 1 --2- 172.21.15.83:0/3116436594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff3006ffe0 0x7eff300c6460 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff37a08640 1 -- 172.21.15.83:0/3116436594 >> 172.21.15.83:0/3116436594 conn(0x7eff300c5060 msgr2=0x7eff300c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:30.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff37a08640 1 -- 172.21.15.83:0/3116436594 shutdown_connections 2024-09-19T07:29:30.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff37a08640 1 -- 172.21.15.83:0/3116436594 wait complete. 2024-09-19T07:29:30.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff37a08640 1 Processor -- start 2024-09-19T07:29:30.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.098+0000 7eff37a08640 1 -- start start 2024-09-19T07:29:30.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff37a08640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff3006ffe0 0x7eff3006f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:30.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff37a08640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff301535d0 0x7eff30155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:30.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff37a08640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff30159af0 0x7eff30155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:30.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff37a08640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7eff30163d20 con 0x7eff30159af0 2024-09-19T07:29:30.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff37a08640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7eff300c2c80 con 0x7eff3006ffe0 2024-09-19T07:29:30.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff37a08640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7eff30163ea0 con 0x7eff301535d0 2024-09-19T07:29:30.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff35fa5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff30159af0 0x7eff30155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:30.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff35fa5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff30159af0 0x7eff30155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48904/0 (socket says 172.21.15.83:48904) 2024-09-19T07:29:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff35fa5640 1 -- 172.21.15.83:0/330086990 learned_addr learned my addr 172.21.15.83:0/330086990 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff357a4640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff3006ffe0 0x7eff3006f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff34fa3640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff301535d0 0x7eff30155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff35fa5640 1 -- 172.21.15.83:0/330086990 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff301535d0 msgr2=0x7eff30155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff35fa5640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff301535d0 0x7eff30155710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff35fa5640 1 -- 172.21.15.83:0/330086990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff3006ffe0 msgr2=0x7eff3006f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff35fa5640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff3006ffe0 0x7eff3006f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff35fa5640 1 -- 172.21.15.83:0/330086990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff300ac5b0 con 0x7eff30159af0 2024-09-19T07:29:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff35fa5640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff30159af0 0x7eff30155c50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7eff2800cb70 tx=0x7eff2800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff25ffb640 1 -- 172.21.15.83:0/330086990 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28017070 con 0x7eff30159af0 2024-09-19T07:29:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff25ffb640 1 -- 172.21.15.83:0/330086990 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2800ad40 con 0x7eff30159af0 2024-09-19T07:29:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff37a08640 1 -- 172.21.15.83:0/330086990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff301ee4d0 con 0x7eff30159af0 2024-09-19T07:29:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff25ffb640 1 -- 172.21.15.83:0/330086990 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28012620 con 0x7eff30159af0 2024-09-19T07:29:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff37a08640 1 -- 172.21.15.83:0/330086990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff301eea70 con 0x7eff30159af0 2024-09-19T07:29:30.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff37a08640 1 -- 172.21.15.83:0/330086990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff300c6460 con 0x7eff30159af0 2024-09-19T07:29:30.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff25ffb640 1 -- 172.21.15.83:0/330086990 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7eff2801a070 con 0x7eff30159af0 2024-09-19T07:29:30.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff25ffb640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff140678c0 0x7eff14069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:30.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff25ffb640 1 -- 172.21.15.83:0/330086990 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7eff280862c0 con 0x7eff30159af0 2024-09-19T07:29:30.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.102+0000 7eff357a4640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff140678c0 0x7eff14069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:30.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.106+0000 7eff357a4640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff140678c0 0x7eff14069d80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7eff2c004e30 tx=0x7eff2c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:30.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.110+0000 7eff25ffb640 1 -- 172.21.15.83:0/330086990 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7eff28056630 con 0x7eff30159af0 2024-09-19T07:29:30.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.338+0000 7eff37a08640 1 -- 172.21.15.83:0/330086990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff30158af0 con 0x7eff30159af0 2024-09-19T07:29:30.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.338+0000 7eff25ffb640 1 -- 172.21.15.83:0/330086990 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+6537 (secure 0 0 0) 0x7eff28055f30 con 0x7eff30159af0 2024-09-19T07:29:30.335 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:30.335 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":84,"btime":"2024-09-19T07:28:43:303745+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 84 2024-09-19T07:29:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.342+0000 7eff37a08640 1 -- 172.21.15.83:0/330086990 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff140678c0 msgr2=0x7eff14069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.342+0000 7eff37a08640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff140678c0 0x7eff14069d80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7eff2c004e30 tx=0x7eff2c009040 comp rx=0 tx=0).stop 2024-09-19T07:29:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.342+0000 7eff37a08640 1 -- 172.21.15.83:0/330086990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff30159af0 msgr2=0x7eff30155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.342+0000 7eff37a08640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff30159af0 0x7eff30155c50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7eff2800cb70 tx=0x7eff2800a430 comp rx=0 tx=0).stop 2024-09-19T07:29:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.342+0000 7eff37a08640 1 -- 172.21.15.83:0/330086990 shutdown_connections 2024-09-19T07:29:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.342+0000 7eff37a08640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff140678c0 0x7eff14069d80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.342+0000 7eff37a08640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff30159af0 0x7eff30155c50 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.342+0000 7eff37a08640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff301535d0 0x7eff30155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.342+0000 7eff37a08640 1 --2- 172.21.15.83:0/330086990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff3006ffe0 0x7eff3006f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.342+0000 7eff37a08640 1 -- 172.21.15.83:0/330086990 >> 172.21.15.83:0/330086990 conn(0x7eff300c5060 msgr2=0x7eff300c2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.342+0000 7eff37a08640 1 -- 172.21.15.83:0/330086990 shutdown_connections 2024-09-19T07:29:30.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.342+0000 7eff37a08640 1 -- 172.21.15.83:0/330086990 wait complete. 2024-09-19T07:29:30.356 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-19T07:29:30.356 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:29:30.357 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:29:30.370 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:29:30.370 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:30.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.526+0000 7f2200b1e640 1 Processor -- start 2024-09-19T07:29:30.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.526+0000 7f2200b1e640 1 -- start start 2024-09-19T07:29:30.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f2200b1e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21fc153300 0x7f21fc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:30.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f2200b1e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21fc06d1d0 0x7f21fc0c23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:30.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f2200b1e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21fc1593f0 con 0x7f21fc0c2910 2024-09-19T07:29:30.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f2200b1e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f21fc1591a0 con 0x7f21fc153300 2024-09-19T07:29:30.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f2200b1e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f21fc1596d0 con 0x7f21fc06d1d0 2024-09-19T07:29:30.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f21fa59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21fc153300 0x7f21fc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:30.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f21fa59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21fc153300 0x7f21fc06cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44654/0 (socket says 172.21.15.83:44654) 2024-09-19T07:29:30.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f21fa59c640 1 -- 172.21.15.83:0/69725840 learned_addr learned my addr 172.21.15.83:0/69725840 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:30.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f21f9d9b640 1 --2- 172.21.15.83:0/69725840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21fc06d1d0 0x7f21fc0c23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:30.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f21fa59c640 1 -- 172.21.15.83:0/69725840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21fc06d1d0 msgr2=0x7f21fc0c23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:30.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f21fa59c640 1 --2- 172.21.15.83:0/69725840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21fc06d1d0 0x7f21fc0c23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f21fa59c640 1 -- 172.21.15.83:0/69725840 >> v1:172.21.15.184:6789/0 conn(0x7f21fc0c2910 legacy=0x7f21fc155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:30.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f21fa59c640 1 -- 172.21.15.83:0/69725840 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21fc1599b0 con 0x7f21fc153300 2024-09-19T07:29:30.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.530+0000 7f21fa59c640 1 --2- 172.21.15.83:0/69725840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21fc153300 0x7f21fc06cc90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009870 tx=0x7f21ec01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a2b4fccba6347e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:30.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f21f959a640 1 -- 172.21.15.83:0/69725840 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec0b0070 con 0x7f21fc153300 2024-09-19T07:29:30.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f21f959a640 1 -- 172.21.15.83:0/69725840 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21ec004230 con 0x7f21fc153300 2024-09-19T07:29:30.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f21f959a640 1 -- 172.21.15.83:0/69725840 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec003620 con 0x7f21fc153300 2024-09-19T07:29:30.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f2200b1e640 1 -- 172.21.15.83:0/69725840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21fc153300 msgr2=0x7f21fc06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:30.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f2200b1e640 1 --2- 172.21.15.83:0/69725840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21fc153300 0x7f21fc06cc90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009870 tx=0x7f21ec01c440 comp rx=0 tx=0).stop 2024-09-19T07:29:30.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f2200b1e640 1 -- 172.21.15.83:0/69725840 shutdown_connections 2024-09-19T07:29:30.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f2200b1e640 1 --2- 172.21.15.83:0/69725840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21fc06d1d0 0x7f21fc0c23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f2200b1e640 1 --2- 172.21.15.83:0/69725840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21fc153300 0x7f21fc06cc90 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f2200b1e640 1 -- 172.21.15.83:0/69725840 >> 172.21.15.83:0/69725840 conn(0x7f21fc0c5090 msgr2=0x7f21fc0c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:30.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f2200b1e640 1 -- 172.21.15.83:0/69725840 shutdown_connections 2024-09-19T07:29:30.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f2200b1e640 1 -- 172.21.15.83:0/69725840 wait complete. 2024-09-19T07:29:30.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f2200b1e640 1 Processor -- start 2024-09-19T07:29:30.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.534+0000 7f2200b1e640 1 -- start start 2024-09-19T07:29:30.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f2200b1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21fc06d1d0 0x7f21fc154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:30.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f2200b1e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21fc0c2910 0x7f21fc14cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:30.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f2200b1e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21fc153300 0x7f21fc149200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:30.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f2200b1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc153860 con 0x7f21fc06d1d0 2024-09-19T07:29:30.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f2200b1e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1536e0 con 0x7f21fc153300 2024-09-19T07:29:30.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f2200b1e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f21fc162d50 con 0x7f21fc0c2910 2024-09-19T07:29:30.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21fa59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21fc06d1d0 0x7f21fc154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:30.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21fa59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21fc06d1d0 0x7f21fc154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44662/0 (socket says 172.21.15.83:44662) 2024-09-19T07:29:30.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21fa59c640 1 -- 172.21.15.83:0/568841375 learned_addr learned my addr 172.21.15.83:0/568841375 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:30.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21f9d9b640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21fc0c2910 0x7f21fc14cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:30.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21fad9d640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21fc153300 0x7f21fc149200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:30.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21fa59c640 1 -- 172.21.15.83:0/568841375 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21fc0c2910 msgr2=0x7f21fc14cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:30.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21fa59c640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21fc0c2910 0x7f21fc14cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21fa59c640 1 -- 172.21.15.83:0/568841375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21fc153300 msgr2=0x7f21fc149200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:30.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21fa59c640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21fc153300 0x7f21fc149200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21fa59c640 1 -- 172.21.15.83:0/568841375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21fc149f90 con 0x7f21fc06d1d0 2024-09-19T07:29:30.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21fa59c640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21fc06d1d0 0x7f21fc154610 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f21ec01a070 tx=0x7f21ec0191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:30.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21eaffd640 1 -- 172.21.15.83:0/568841375 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec0b0040 con 0x7f21fc06d1d0 2024-09-19T07:29:30.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21eaffd640 1 -- 172.21.15.83:0/568841375 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21ec0bb070 con 0x7f21fc06d1d0 2024-09-19T07:29:30.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f2200b1e640 1 -- 172.21.15.83:0/568841375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21fc0c6830 con 0x7f21fc06d1d0 2024-09-19T07:29:30.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f21eaffd640 1 -- 172.21.15.83:0/568841375 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec0043e0 con 0x7f21fc06d1d0 2024-09-19T07:29:30.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f2200b1e640 1 -- 172.21.15.83:0/568841375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21fc0c6df0 con 0x7f21fc06d1d0 2024-09-19T07:29:30.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.538+0000 7f2200b1e640 1 -- 172.21.15.83:0/568841375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21fc0c23d0 con 0x7f21fc06d1d0 2024-09-19T07:29:30.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.542+0000 7f21eaffd640 1 -- 172.21.15.83:0/568841375 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f21ec0c0460 con 0x7f21fc06d1d0 2024-09-19T07:29:30.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.542+0000 7f21eaffd640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21dc0678f0 0x7f21dc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:30.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.542+0000 7f21eaffd640 1 -- 172.21.15.83:0/568841375 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7f21ec121060 con 0x7f21fc06d1d0 2024-09-19T07:29:30.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.542+0000 7f21f9d9b640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21dc0678f0 0x7f21dc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:30.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.542+0000 7f21f9d9b640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21dc0678f0 0x7f21dc069db0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f21fc14a920 tx=0x7f21f0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:30.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.546+0000 7f21eaffd640 1 -- 172.21.15.83:0/568841375 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f21ec016150 con 0x7f21fc06d1d0 2024-09-19T07:29:30.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.782+0000 7f2200b1e640 1 -- 172.21.15.83:0/568841375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21fc0c82c0 con 0x7f21fc06d1d0 2024-09-19T07:29:30.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f21eaffd640 1 -- 172.21.15.83:0/568841375 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+6537 (secure 0 0 0) 0x7f21ec0f14b0 con 0x7f21fc06d1d0 2024-09-19T07:29:30.781 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:30.781 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":84,"btime":"2024-09-19T07:28:43:303745+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:30.781 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 84 2024-09-19T07:29:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f2200b1e640 1 -- 172.21.15.83:0/568841375 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21dc0678f0 msgr2=0x7f21dc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f2200b1e640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21dc0678f0 0x7f21dc069db0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f21fc14a920 tx=0x7f21f0008040 comp rx=0 tx=0).stop 2024-09-19T07:29:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f2200b1e640 1 -- 172.21.15.83:0/568841375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21fc06d1d0 msgr2=0x7f21fc154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f2200b1e640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21fc06d1d0 0x7f21fc154610 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f21ec01a070 tx=0x7f21ec0191e0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f2200b1e640 1 -- 172.21.15.83:0/568841375 shutdown_connections 2024-09-19T07:29:30.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f2200b1e640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21dc0678f0 0x7f21dc069db0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f2200b1e640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21fc153300 0x7f21fc149200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f2200b1e640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21fc0c2910 0x7f21fc14cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f2200b1e640 1 --2- 172.21.15.83:0/568841375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21fc06d1d0 0x7f21fc154610 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:30.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f2200b1e640 1 -- 172.21.15.83:0/568841375 >> 172.21.15.83:0/568841375 conn(0x7f21fc0c5090 msgr2=0x7f21fc06f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:30.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f2200b1e640 1 -- 172.21.15.83:0/568841375 shutdown_connections 2024-09-19T07:29:30.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:30.786+0000 7f2200b1e640 1 -- 172.21.15.83:0/568841375 wait complete. 2024-09-19T07:29:30.799 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:29:32.498 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:29:32.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:29:32.800 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:32.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f900e640 1 Processor -- start 2024-09-19T07:29:32.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f900e640 1 -- start start 2024-09-19T07:29:32.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f900e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40ab060 0x7f68f41528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:32.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f900e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f68f40c3b70 0x7f68f4152e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:32.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f900e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f406de90 con 0x7f68f40ab750 2024-09-19T07:29:32.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f900e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f68f406e170 con 0x7f68f40c3b70 2024-09-19T07:29:32.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f900e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f68f406e450 con 0x7f68f40ab060 2024-09-19T07:29:32.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f2d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f68f40ab750 0x7f68f40c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38312/0 (socket says 172.21.15.83:38312) 2024-09-19T07:29:32.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f2d9d640 1 -- 172.21.15.83:0/783672195 learned_addr learned my addr 172.21.15.83:0/783672195 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:29:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f259c640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40ab060 0x7f68f41528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f359e640 1 --2- 172.21.15.83:0/783672195 >> v2:172.21.15.184:3300/0 conn(0x7f68f40c3b70 0x7f68f4152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3670929389 0 0) 0x7f68f406de90 con 0x7f68f40ab750 2024-09-19T07:29:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68dc003530 con 0x7f68f40ab750 2024-09-19T07:29:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1000262404 0 0) 0x7f68dc003530 con 0x7f68f40ab750 2024-09-19T07:29:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40ab060 msgr2=0x7f68f41528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40ab060 0x7f68f41528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 >> v2:172.21.15.184:3300/0 conn(0x7f68f40c3b70 msgr2=0x7f68f4152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 --2- 172.21.15.83:0/783672195 >> v2:172.21.15.184:3300/0 conn(0x7f68f40c3b70 0x7f68f4152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f406e730 con 0x7f68f40ab750 2024-09-19T07:29:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f68e4001680 con 0x7f68f40ab750 2024-09-19T07:29:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 >> v1:172.21.15.83:6789/0 conn(0x7f68f40ab750 legacy=0x7f68f40c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40ab060 0x7f68dc0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68dc003f40 0x7f68dc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68dc004d50 0x7f68dc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f68f406e170 con 0x7f68dc003f40 2024-09-19T07:29:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f68f406e450 con 0x7f68dc004d50 2024-09-19T07:29:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f68dc00ba20 con 0x7f68f40ab060 2024-09-19T07:29:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f259c640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68dc003f40 0x7f68dc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f2d9d640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40ab060 0x7f68dc0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f359e640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68dc004d50 0x7f68dc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:32.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f259c640 1 -- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40ab060 msgr2=0x7f68dc0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f259c640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40ab060 0x7f68dc0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f259c640 1 -- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68dc004d50 msgr2=0x7f68dc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f259c640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68dc004d50 0x7f68dc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f259c640 1 -- 172.21.15.83:0/783672195 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68dc003530 con 0x7f68dc003f40 2024-09-19T07:29:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f359e640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68dc004d50 0x7f68dc0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:29:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.934+0000 7f68f2d9d640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40ab060 0x7f68dc0039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:29:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f259c640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68dc003f40 0x7f68dc004810 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f68e800bc70 tx=0x7f68e801cd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d3c76ce445ca1e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e800b810 con 0x7f68dc003f40 2024-09-19T07:29:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e80a5bf0 con 0x7f68dc003f40 2024-09-19T07:29:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f1d9b640 1 -- 172.21.15.83:0/783672195 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e80ad610 con 0x7f68dc003f40 2024-09-19T07:29:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 -- 172.21.15.83:0/783672195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68dc003f40 msgr2=0x7f68dc004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 --2- 172.21.15.83:0/783672195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68dc003f40 0x7f68dc004810 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f68e800bc70 tx=0x7f68e801cd40 comp rx=0 tx=0).stop 2024-09-19T07:29:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f359e640 1 -- 172.21.15.83:0/783672195 reap_dead start 2024-09-19T07:29:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 -- 172.21.15.83:0/783672195 shutdown_connections 2024-09-19T07:29:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 -- 172.21.15.83:0/783672195 >> 172.21.15.83:0/783672195 conn(0x7f68f406bc80 msgr2=0x7f68f406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 -- 172.21.15.83:0/783672195 shutdown_connections 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 -- 172.21.15.83:0/783672195 wait complete. 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 Processor -- start 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 -- start start 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68f40729f0 0x7f68f414f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40c7140 0x7f68f414cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68f4148a40 0x7f68f414d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f68f415cbc0 con 0x7f68f40729f0 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f68f40abd40 con 0x7f68f4148a40 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f900e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f68f415cd40 con 0x7f68f40c7140 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68f40729f0 0x7f68f414f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68f40729f0 0x7f68f414f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44674/0 (socket says 172.21.15.83:44674) 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.938+0000 7f68f2d9d640 1 -- 172.21.15.83:0/1760484338 learned_addr learned my addr 172.21.15.83:0/1760484338 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f259c640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40c7140 0x7f68f414cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f359e640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68f4148a40 0x7f68f414d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f2d9d640 1 -- 172.21.15.83:0/1760484338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40c7140 msgr2=0x7f68f414cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f2d9d640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40c7140 0x7f68f414cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f2d9d640 1 -- 172.21.15.83:0/1760484338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68f4148a40 msgr2=0x7f68f414d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f2d9d640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68f4148a40 0x7f68f414d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f2d9d640 1 -- 172.21.15.83:0/1760484338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f40c5e00 con 0x7f68f40729f0 2024-09-19T07:29:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f2d9d640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68f40729f0 0x7f68f414f940 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f68e400d720 tx=0x7f68e400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68db7fe640 1 -- 172.21.15.83:0/1760484338 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e4019070 con 0x7f68f40729f0 2024-09-19T07:29:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68db7fe640 1 -- 172.21.15.83:0/1760484338 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e4004cc0 con 0x7f68f40729f0 2024-09-19T07:29:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68db7fe640 1 -- 172.21.15.83:0/1760484338 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e40072d0 con 0x7f68f40729f0 2024-09-19T07:29:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f900e640 1 -- 172.21.15.83:0/1760484338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f40c60e0 con 0x7f68f40729f0 2024-09-19T07:29:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f900e640 1 -- 172.21.15.83:0/1760484338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f4072dd0 con 0x7f68f40729f0 2024-09-19T07:29:32.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f900e640 1 -- 172.21.15.83:0/1760484338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f40c1ff0 con 0x7f68f40729f0 2024-09-19T07:29:32.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68db7fe640 1 -- 172.21.15.83:0/1760484338 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f68e401c070 con 0x7f68f40729f0 2024-09-19T07:29:32.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68db7fe640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f68e00678c0 0x7f68e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:32.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68db7fe640 1 -- 172.21.15.83:0/1760484338 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7f68e4085ea0 con 0x7f68f40729f0 2024-09-19T07:29:32.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f259c640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f68e00678c0 0x7f68e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:32.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.942+0000 7f68f259c640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f68e00678c0 0x7f68e0069d80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f68e80ae000 tx=0x7f68e801b5e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:32.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:32.946+0000 7f68db7fe640 1 -- 172.21.15.83:0/1760484338 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f68e4056210 con 0x7f68f40729f0 2024-09-19T07:29:33.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.146+0000 7f68f900e640 1 -- 172.21.15.83:0/1760484338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68f40c4a60 con 0x7f68f40729f0 2024-09-19T07:29:33.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.146+0000 7f68db7fe640 1 -- 172.21.15.83:0/1760484338 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+6537 (secure 0 0 0) 0x7f68e4055b10 con 0x7f68f40729f0 2024-09-19T07:29:33.142 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:33.142 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":84,"btime":"2024-09-19T07:28:43:303745+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:33.143 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 84 2024-09-19T07:29:33.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.150+0000 7f68f900e640 1 -- 172.21.15.83:0/1760484338 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f68e00678c0 msgr2=0x7f68e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:33.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.150+0000 7f68f900e640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f68e00678c0 0x7f68e0069d80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f68e80ae000 tx=0x7f68e801b5e0 comp rx=0 tx=0).stop 2024-09-19T07:29:33.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.150+0000 7f68f900e640 1 -- 172.21.15.83:0/1760484338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68f40729f0 msgr2=0x7f68f414f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:33.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.150+0000 7f68f900e640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68f40729f0 0x7f68f414f940 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f68e400d720 tx=0x7f68e400dbe0 comp rx=0 tx=0).stop 2024-09-19T07:29:33.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.150+0000 7f68f900e640 1 -- 172.21.15.83:0/1760484338 shutdown_connections 2024-09-19T07:29:33.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.150+0000 7f68f900e640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f68e00678c0 0x7f68e0069d80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:33.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.150+0000 7f68f900e640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68f4148a40 0x7f68f414d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:33.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.150+0000 7f68f900e640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68f40c7140 0x7f68f414cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:33.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.150+0000 7f68f900e640 1 --2- 172.21.15.83:0/1760484338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68f40729f0 0x7f68f414f940 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:33.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.150+0000 7f68f900e640 1 -- 172.21.15.83:0/1760484338 >> 172.21.15.83:0/1760484338 conn(0x7f68f406bc80 msgr2=0x7f68f40c2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:33.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.150+0000 7f68f900e640 1 -- 172.21.15.83:0/1760484338 shutdown_connections 2024-09-19T07:29:33.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:33.150+0000 7f68f900e640 1 -- 172.21.15.83:0/1760484338 wait complete. 2024-09-19T07:29:33.161 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:29:35.163 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:35.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.298+0000 7f0b45a62640 1 Processor -- start 2024-09-19T07:29:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.298+0000 7f0b45a62640 1 -- start start 2024-09-19T07:29:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b45a62640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0b4006ffe0 0x7f0b400c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b45a62640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b400c26c0 con 0x7f0b401535d0 2024-09-19T07:29:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b45a62640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b400c29a0 con 0x7f0b40159af0 2024-09-19T07:29:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b45a62640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f0b400707e0 con 0x7f0b4006ffe0 2024-09-19T07:29:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b3f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0b401535d0 0x7f0b40155c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38324/0 (socket says 172.21.15.83:38324) 2024-09-19T07:29:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b3f7fe640 1 -- 172.21.15.83:0/3268605691 learned_addr learned my addr 172.21.15.83:0/3268605691 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:29:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b3effd640 1 --2- 172.21.15.83:0/3268605691 >> v2:172.21.15.83:3300/0 conn(0x7f0b4006ffe0 0x7f0b400c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b3effd640 1 -- 172.21.15.83:0/3268605691 >> v1:172.21.15.184:6789/0 conn(0x7f0b40159af0 legacy=0x7f0b400c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b3effd640 1 -- 172.21.15.83:0/3268605691 >> v1:172.21.15.83:6789/0 conn(0x7f0b401535d0 legacy=0x7f0b40155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b3e7fc640 1 -- 172.21.15.83:0/3268605691 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4008989237 0 0) 0x7f0b400c26c0 con 0x7f0b401535d0 2024-09-19T07:29:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b3effd640 1 -- 172.21.15.83:0/3268605691 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b401592f0 con 0x7f0b4006ffe0 2024-09-19T07:29:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b3effd640 1 --2- 172.21.15.83:0/3268605691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b4006ffe0 0x7f0b400c6460 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f0b340097c0 tx=0x7f0b3401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4073e633704dec73 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b3e7fc640 1 -- 172.21.15.83:0/3268605691 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b340b0070 con 0x7f0b4006ffe0 2024-09-19T07:29:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b3e7fc640 1 -- 172.21.15.83:0/3268605691 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b340ab600 con 0x7f0b4006ffe0 2024-09-19T07:29:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.302+0000 7f0b3e7fc640 1 -- 172.21.15.83:0/3268605691 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b340ab9d0 con 0x7f0b4006ffe0 2024-09-19T07:29:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 -- 172.21.15.83:0/3268605691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b4006ffe0 msgr2=0x7f0b400c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 --2- 172.21.15.83:0/3268605691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b4006ffe0 0x7f0b400c6460 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f0b340097c0 tx=0x7f0b3401c340 comp rx=0 tx=0).stop 2024-09-19T07:29:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 -- 172.21.15.83:0/3268605691 shutdown_connections 2024-09-19T07:29:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 --2- 172.21.15.83:0/3268605691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b4006ffe0 0x7f0b400c6460 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 -- 172.21.15.83:0/3268605691 >> 172.21.15.83:0/3268605691 conn(0x7f0b400c5060 msgr2=0x7f0b400c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 -- 172.21.15.83:0/3268605691 shutdown_connections 2024-09-19T07:29:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 -- 172.21.15.83:0/3268605691 wait complete. 2024-09-19T07:29:35.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 Processor -- start 2024-09-19T07:29:35.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 -- start start 2024-09-19T07:29:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0b4006ffe0 0x7f0b401f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b401535d0 0x7f0b4015b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0b40159af0 0x7f0b4015bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0b40163d20 con 0x7f0b401535d0 2024-09-19T07:29:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0b400c2c80 con 0x7f0b40159af0 2024-09-19T07:29:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0b40163ea0 con 0x7f0b4006ffe0 2024-09-19T07:29:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b3effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b401535d0 0x7f0b4015b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b3effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b401535d0 0x7f0b4015b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44704/0 (socket says 172.21.15.83:44704) 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b3effd640 1 -- 172.21.15.83:0/20955695 learned_addr learned my addr 172.21.15.83:0/20955695 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b3f7fe640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0b4006ffe0 0x7f0b401f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b3ffff640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0b40159af0 0x7f0b4015bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b3effd640 1 -- 172.21.15.83:0/20955695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0b4006ffe0 msgr2=0x7f0b401f18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b3effd640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0b4006ffe0 0x7f0b401f18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b3effd640 1 -- 172.21.15.83:0/20955695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0b40159af0 msgr2=0x7f0b4015bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b3effd640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0b40159af0 0x7f0b4015bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b3effd640 1 -- 172.21.15.83:0/20955695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4015b220 con 0x7f0b401535d0 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b3effd640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b401535d0 0x7f0b4015b700 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0b340097c0 tx=0x7f0b3401ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b27fff640 1 -- 172.21.15.83:0/20955695 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b340b0040 con 0x7f0b401535d0 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b27fff640 1 -- 172.21.15.83:0/20955695 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b34004030 con 0x7f0b401535d0 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b27fff640 1 -- 172.21.15.83:0/20955695 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34004400 con 0x7f0b401535d0 2024-09-19T07:29:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 -- 172.21.15.83:0/20955695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b40152f30 con 0x7f0b401535d0 2024-09-19T07:29:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.306+0000 7f0b45a62640 1 -- 172.21.15.83:0/20955695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b401f2500 con 0x7f0b401535d0 2024-09-19T07:29:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.310+0000 7f0b45a62640 1 -- 172.21.15.83:0/20955695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b4015b3f0 con 0x7f0b401535d0 2024-09-19T07:29:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.310+0000 7f0b27fff640 1 -- 172.21.15.83:0/20955695 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0b34003620 con 0x7f0b401535d0 2024-09-19T07:29:35.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.310+0000 7f0b27fff640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0b2c0678c0 0x7f0b2c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:35.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.310+0000 7f0b27fff640 1 -- 172.21.15.83:0/20955695 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7f0b34121320 con 0x7f0b401535d0 2024-09-19T07:29:35.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.310+0000 7f0b3f7fe640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0b2c0678c0 0x7f0b2c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:35.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.310+0000 7f0b3f7fe640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0b2c0678c0 0x7f0b2c069d80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0b38007fd0 tx=0x7f0b38007f00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:35.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.314+0000 7f0b27fff640 1 -- 172.21.15.83:0/20955695 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f0b34016050 con 0x7f0b401535d0 2024-09-19T07:29:35.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.510+0000 7f0b45a62640 1 -- 172.21.15.83:0/20955695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b400adaa0 con 0x7f0b401535d0 2024-09-19T07:29:35.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b27fff640 1 -- 172.21.15.83:0/20955695 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+6537 (secure 0 0 0) 0x7f0b340f17b0 con 0x7f0b401535d0 2024-09-19T07:29:35.508 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:35.508 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":84,"btime":"2024-09-19T07:28:43:303745+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:35.509 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 84 2024-09-19T07:29:35.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b45a62640 1 -- 172.21.15.83:0/20955695 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0b2c0678c0 msgr2=0x7f0b2c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:35.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b45a62640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0b2c0678c0 0x7f0b2c069d80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0b38007fd0 tx=0x7f0b38007f00 comp rx=0 tx=0).stop 2024-09-19T07:29:35.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b45a62640 1 -- 172.21.15.83:0/20955695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b401535d0 msgr2=0x7f0b4015b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:35.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b45a62640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b401535d0 0x7f0b4015b700 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0b340097c0 tx=0x7f0b3401ab80 comp rx=0 tx=0).stop 2024-09-19T07:29:35.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b45a62640 1 -- 172.21.15.83:0/20955695 shutdown_connections 2024-09-19T07:29:35.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b45a62640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0b2c0678c0 0x7f0b2c069d80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:35.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b45a62640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0b40159af0 0x7f0b4015bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:35.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b45a62640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b401535d0 0x7f0b4015b700 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:35.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b45a62640 1 --2- 172.21.15.83:0/20955695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0b4006ffe0 0x7f0b401f18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:35.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b45a62640 1 -- 172.21.15.83:0/20955695 >> 172.21.15.83:0/20955695 conn(0x7f0b400c5060 msgr2=0x7f0b40072580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:35.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b45a62640 1 -- 172.21.15.83:0/20955695 shutdown_connections 2024-09-19T07:29:35.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:35.514+0000 7f0b45a62640 1 -- 172.21.15.83:0/20955695 wait complete. 2024-09-19T07:29:35.527 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:29:37.529 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efec2954640 1 Processor -- start 2024-09-19T07:29:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efec2954640 1 -- start start 2024-09-19T07:29:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efec2954640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efebc06e270 0x7efebc1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efec2954640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efebc1592c0 con 0x7efebc0c42e0 2024-09-19T07:29:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efec2954640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efebc06e900 con 0x7efebc0abb30 2024-09-19T07:29:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efec2954640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7efebc06e6e0 con 0x7efebc06e270 2024-09-19T07:29:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efebb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efebc06e270 0x7efebc1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efebbfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7efebc0c42e0 0x7efebc159a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38332/0 (socket says 172.21.15.83:38332) 2024-09-19T07:29:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efebbfff640 1 -- 172.21.15.83:0/4229348407 learned_addr learned my addr 172.21.15.83:0/4229348407 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:29:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efebaffd640 1 -- 172.21.15.83:0/4229348407 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 532039547 0 0) 0x7efebc1592c0 con 0x7efebc0c42e0 2024-09-19T07:29:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efebaffd640 1 -- 172.21.15.83:0/4229348407 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efea4003530 con 0x7efebc0c42e0 2024-09-19T07:29:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efebb7fe640 1 -- 172.21.15.83:0/4229348407 >> v1:172.21.15.184:6789/0 conn(0x7efebc0abb30 legacy=0x7efebc153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efebb7fe640 1 -- 172.21.15.83:0/4229348407 >> v1:172.21.15.83:6789/0 conn(0x7efebc0c42e0 legacy=0x7efebc159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.666+0000 7efebb7fe640 1 -- 172.21.15.83:0/4229348407 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efebc06ebe0 con 0x7efebc06e270 2024-09-19T07:29:37.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebb7fe640 1 --2- 172.21.15.83:0/4229348407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efebc06e270 0x7efebc1566b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7efeb00097c0 tx=0x7efeb001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c4d5f2a4612ada8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:37.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebaffd640 1 -- 172.21.15.83:0/4229348407 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb00b0070 con 0x7efebc06e270 2024-09-19T07:29:37.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebaffd640 1 -- 172.21.15.83:0/4229348407 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb00ab600 con 0x7efebc06e270 2024-09-19T07:29:37.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebaffd640 1 -- 172.21.15.83:0/4229348407 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb00ab9d0 con 0x7efebc06e270 2024-09-19T07:29:37.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 -- 172.21.15.83:0/4229348407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efebc06e270 msgr2=0x7efebc1566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:37.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 --2- 172.21.15.83:0/4229348407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efebc06e270 0x7efebc1566b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7efeb00097c0 tx=0x7efeb001c460 comp rx=0 tx=0).stop 2024-09-19T07:29:37.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 -- 172.21.15.83:0/4229348407 shutdown_connections 2024-09-19T07:29:37.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 --2- 172.21.15.83:0/4229348407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efebc06e270 0x7efebc1566b0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:37.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 -- 172.21.15.83:0/4229348407 >> 172.21.15.83:0/4229348407 conn(0x7efebc06bc80 msgr2=0x7efebc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:37.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 -- 172.21.15.83:0/4229348407 shutdown_connections 2024-09-19T07:29:37.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 -- 172.21.15.83:0/4229348407 wait complete. 2024-09-19T07:29:37.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 Processor -- start 2024-09-19T07:29:37.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 -- start start 2024-09-19T07:29:37.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efebc06e270 0x7efebc154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:37.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efebc0abb30 0x7efebc1ee510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:37.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efebc0c42e0 0x7efebc1eea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:37.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efebc0c4840 con 0x7efebc06e270 2024-09-19T07:29:37.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efebc0c46c0 con 0x7efebc0c42e0 2024-09-19T07:29:37.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec2954640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efebc163de0 con 0x7efebc0abb30 2024-09-19T07:29:37.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efebc06e270 0x7efebc154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:37.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efebc06e270 0x7efebc154d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44712/0 (socket says 172.21.15.83:44712) 2024-09-19T07:29:37.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebbfff640 1 -- 172.21.15.83:0/3177136693 learned_addr learned my addr 172.21.15.83:0/3177136693 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:37.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebb7fe640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efebc0abb30 0x7efebc1ee510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:37.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efec0ef1640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efebc0c42e0 0x7efebc1eea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:37.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebbfff640 1 -- 172.21.15.83:0/3177136693 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efebc0abb30 msgr2=0x7efebc1ee510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:37.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebbfff640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efebc0abb30 0x7efebc1ee510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:37.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebbfff640 1 -- 172.21.15.83:0/3177136693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efebc0c42e0 msgr2=0x7efebc1eea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:37.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebbfff640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efebc0c42e0 0x7efebc1eea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:37.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.670+0000 7efebbfff640 1 -- 172.21.15.83:0/3177136693 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efebc0c6e30 con 0x7efebc06e270 2024-09-19T07:29:37.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.674+0000 7efebbfff640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efebc06e270 0x7efebc154d50 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7efeac0010f0 tx=0x7efeac00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:37.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.674+0000 7efeb8ff9640 1 -- 172.21.15.83:0/3177136693 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac017070 con 0x7efebc06e270 2024-09-19T07:29:37.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.674+0000 7efeb8ff9640 1 -- 172.21.15.83:0/3177136693 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeac005d10 con 0x7efebc06e270 2024-09-19T07:29:37.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.674+0000 7efec2954640 1 -- 172.21.15.83:0/3177136693 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efebc155e40 con 0x7efebc06e270 2024-09-19T07:29:37.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.674+0000 7efeb8ff9640 1 -- 172.21.15.83:0/3177136693 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac007310 con 0x7efebc06e270 2024-09-19T07:29:37.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.674+0000 7efec2954640 1 -- 172.21.15.83:0/3177136693 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efebc151ce0 con 0x7efebc06e270 2024-09-19T07:29:37.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.674+0000 7efec2954640 1 -- 172.21.15.83:0/3177136693 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efebc1566b0 con 0x7efebc06e270 2024-09-19T07:29:37.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.674+0000 7efeb8ff9640 1 -- 172.21.15.83:0/3177136693 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efeac01a070 con 0x7efebc06e270 2024-09-19T07:29:37.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.674+0000 7efeb8ff9640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efea80678c0 0x7efea8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:37.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.674+0000 7efebb7fe640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efea80678c0 0x7efea8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:37.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.674+0000 7efeb8ff9640 1 -- 172.21.15.83:0/3177136693 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7efeac085840 con 0x7efebc06e270 2024-09-19T07:29:37.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.674+0000 7efebb7fe640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efea80678c0 0x7efea8069d80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7efeb00ad000 tx=0x7efeb0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:37.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.678+0000 7efeb8ff9640 1 -- 172.21.15.83:0/3177136693 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7efeac055bb0 con 0x7efebc06e270 2024-09-19T07:29:37.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.874+0000 7efec2954640 1 -- 172.21.15.83:0/3177136693 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efebc06e050 con 0x7efebc06e270 2024-09-19T07:29:37.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efeb8ff9640 1 -- 172.21.15.83:0/3177136693 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+6537 (secure 0 0 0) 0x7efeac0554b0 con 0x7efebc06e270 2024-09-19T07:29:37.873 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:37.873 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":84,"btime":"2024-09-19T07:28:43:303745+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:37.873 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 84 2024-09-19T07:29:37.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efec2954640 1 -- 172.21.15.83:0/3177136693 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efea80678c0 msgr2=0x7efea8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:37.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efec2954640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efea80678c0 0x7efea8069d80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7efeb00ad000 tx=0x7efeb0008040 comp rx=0 tx=0).stop 2024-09-19T07:29:37.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efec2954640 1 -- 172.21.15.83:0/3177136693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efebc06e270 msgr2=0x7efebc154d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:37.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efec2954640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efebc06e270 0x7efebc154d50 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7efeac0010f0 tx=0x7efeac00c890 comp rx=0 tx=0).stop 2024-09-19T07:29:37.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efec2954640 1 -- 172.21.15.83:0/3177136693 shutdown_connections 2024-09-19T07:29:37.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efec2954640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efea80678c0 0x7efea8069d80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:37.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efec2954640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efebc0c42e0 0x7efebc1eea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:37.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efec2954640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efebc0abb30 0x7efebc1ee510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:37.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efec2954640 1 --2- 172.21.15.83:0/3177136693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efebc06e270 0x7efebc154d50 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:37.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efec2954640 1 -- 172.21.15.83:0/3177136693 >> 172.21.15.83:0/3177136693 conn(0x7efebc06bc80 msgr2=0x7efebc14ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:37.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efec2954640 1 -- 172.21.15.83:0/3177136693 shutdown_connections 2024-09-19T07:29:37.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:37.878+0000 7efec2954640 1 -- 172.21.15.83:0/3177136693 wait complete. 2024-09-19T07:29:37.892 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:29:39.893 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:40.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.026+0000 7f98e233a640 1 Processor -- start 2024-09-19T07:29:40.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.026+0000 7f98e233a640 1 -- start start 2024-09-19T07:29:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98e233a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f98dc0ab750 0x7f98dc0c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98e233a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98dc0c2530 0x7f98dc1528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98e233a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98dc06de90 con 0x7f98dc0ab060 2024-09-19T07:29:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98e233a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f98dc06e170 con 0x7f98dc0ab750 2024-09-19T07:29:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98e233a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f98dc06e450 con 0x7f98dc0c2530 2024-09-19T07:29:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98dbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f98dc0ab750 0x7f98dc0c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:40.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98e08d7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f98dc0ab060 0x7f98dc155890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38348/0 (socket says 172.21.15.83:38348) 2024-09-19T07:29:40.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98e08d7640 1 -- 172.21.15.83:0/354184452 learned_addr learned my addr 172.21.15.83:0/354184452 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:29:40.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98db7fe640 1 --2- 172.21.15.83:0/354184452 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98dc0c2530 0x7f98dc1528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:40.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98dbfff640 1 -- 172.21.15.83:0/354184452 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98dc0c2530 msgr2=0x7f98dc1528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:40.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98dbfff640 1 --2- 172.21.15.83:0/354184452 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98dc0c2530 0x7f98dc1528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:40.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98dbfff640 1 -- 172.21.15.83:0/354184452 >> v1:172.21.15.83:6789/0 conn(0x7f98dc0ab060 legacy=0x7f98dc155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:40.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98dbfff640 1 -- 172.21.15.83:0/354184452 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98dc06e730 con 0x7f98dc0ab750 2024-09-19T07:29:40.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98daffd640 1 -- 172.21.15.83:0/354184452 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 380223447 0 0) 0x7f98dc06de90 con 0x7f98dc0ab060 2024-09-19T07:29:40.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98dbfff640 1 --2- 172.21.15.83:0/354184452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98dc0ab750 0x7f98dc0c1ff0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f98c40097c0 tx=0x7f98c401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c062b81c240c85 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:40.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98daffd640 1 -- 172.21.15.83:0/354184452 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c40b0070 con 0x7f98dc0ab750 2024-09-19T07:29:40.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98daffd640 1 -- 172.21.15.83:0/354184452 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98c40ab600 con 0x7f98dc0ab750 2024-09-19T07:29:40.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.030+0000 7f98daffd640 1 -- 172.21.15.83:0/354184452 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c40ab9d0 con 0x7f98dc0ab750 2024-09-19T07:29:40.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 -- 172.21.15.83:0/354184452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98dc0ab750 msgr2=0x7f98dc0c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:40.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 --2- 172.21.15.83:0/354184452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98dc0ab750 0x7f98dc0c1ff0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f98c40097c0 tx=0x7f98c401c410 comp rx=0 tx=0).stop 2024-09-19T07:29:40.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 -- 172.21.15.83:0/354184452 shutdown_connections 2024-09-19T07:29:40.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 --2- 172.21.15.83:0/354184452 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98dc0c2530 0x7f98dc1528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:40.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 --2- 172.21.15.83:0/354184452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98dc0ab750 0x7f98dc0c1ff0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:40.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 -- 172.21.15.83:0/354184452 >> 172.21.15.83:0/354184452 conn(0x7f98dc06bc80 msgr2=0x7f98dc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:40.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 -- 172.21.15.83:0/354184452 shutdown_connections 2024-09-19T07:29:40.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 -- 172.21.15.83:0/354184452 wait complete. 2024-09-19T07:29:40.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 Processor -- start 2024-09-19T07:29:40.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 -- start start 2024-09-19T07:29:40.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98dc0ab060 0x7f98dc0bd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:40.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98dc0ab750 0x7f98dc0c0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:40.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98dc0c2530 0x7f98dc0c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:40.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f98dc153190 con 0x7f98dc0ab750 2024-09-19T07:29:40.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f98dc153010 con 0x7f98dc0c2530 2024-09-19T07:29:40.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f98dc0abd40 con 0x7f98dc0ab060 2024-09-19T07:29:40.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98db7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98dc0ab750 0x7f98dc0c0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:40.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98db7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98dc0ab750 0x7f98dc0c0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44734/0 (socket says 172.21.15.83:44734) 2024-09-19T07:29:40.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98db7fe640 1 -- 172.21.15.83:0/3647953412 learned_addr learned my addr 172.21.15.83:0/3647953412 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:40.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98dbfff640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98dc0ab060 0x7f98dc0bd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:40.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e08d7640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98dc0c2530 0x7f98dc0c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:40.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98db7fe640 1 -- 172.21.15.83:0/3647953412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98dc0ab060 msgr2=0x7f98dc0bd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:40.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98db7fe640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98dc0ab060 0x7f98dc0bd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:40.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98db7fe640 1 -- 172.21.15.83:0/3647953412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98dc0c2530 msgr2=0x7f98dc0c09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:40.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98db7fe640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98dc0c2530 0x7f98dc0c09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:40.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98db7fe640 1 -- 172.21.15.83:0/3647953412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98dc1eda50 con 0x7f98dc0ab750 2024-09-19T07:29:40.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98db7fe640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98dc0ab750 0x7f98dc0c0490 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f98cc007e70 tx=0x7f98cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:40.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98d8ff9640 1 -- 172.21.15.83:0/3647953412 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc014620 con 0x7f98dc0ab750 2024-09-19T07:29:40.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98d8ff9640 1 -- 172.21.15.83:0/3647953412 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98cc019070 con 0x7f98dc0ab750 2024-09-19T07:29:40.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 -- 172.21.15.83:0/3647953412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98dc0c1500 con 0x7f98dc0ab750 2024-09-19T07:29:40.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98d8ff9640 1 -- 172.21.15.83:0/3647953412 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc014a00 con 0x7f98dc0ab750 2024-09-19T07:29:40.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.034+0000 7f98e233a640 1 -- 172.21.15.83:0/3647953412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98dc1ecc90 con 0x7f98dc0ab750 2024-09-19T07:29:40.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.038+0000 7f98e233a640 1 -- 172.21.15.83:0/3647953412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98dc0c1ff0 con 0x7f98dc0ab750 2024-09-19T07:29:40.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.038+0000 7f98d8ff9640 1 -- 172.21.15.83:0/3647953412 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f98cc014ba0 con 0x7f98dc0ab750 2024-09-19T07:29:40.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.038+0000 7f98d8ff9640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98bc0678c0 0x7f98bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:40.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.038+0000 7f98d8ff9640 1 -- 172.21.15.83:0/3647953412 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7f98cc086d10 con 0x7f98dc0ab750 2024-09-19T07:29:40.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.038+0000 7f98dbfff640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98bc0678c0 0x7f98bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:40.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.038+0000 7f98dbfff640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98bc0678c0 0x7f98bc069d80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f98c40ad000 tx=0x7f98c40a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:40.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.042+0000 7f98d8ff9640 1 -- 172.21.15.83:0/3647953412 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f98cc011260 con 0x7f98dc0ab750 2024-09-19T07:29:40.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.242+0000 7f98e233a640 1 -- 172.21.15.83:0/3647953412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98dc06c260 con 0x7f98dc0ab750 2024-09-19T07:29:40.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98d8ff9640 1 -- 172.21.15.83:0/3647953412 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+6537 (secure 0 0 0) 0x7f98cc057080 con 0x7f98dc0ab750 2024-09-19T07:29:40.241 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:40.241 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":84,"btime":"2024-09-19T07:28:43:303745+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 84 2024-09-19T07:29:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98e233a640 1 -- 172.21.15.83:0/3647953412 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98bc0678c0 msgr2=0x7f98bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98e233a640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98bc0678c0 0x7f98bc069d80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f98c40ad000 tx=0x7f98c40a6000 comp rx=0 tx=0).stop 2024-09-19T07:29:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98e233a640 1 -- 172.21.15.83:0/3647953412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98dc0ab750 msgr2=0x7f98dc0c0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98e233a640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98dc0ab750 0x7f98dc0c0490 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f98cc007e70 tx=0x7f98cc00e4b0 comp rx=0 tx=0).stop 2024-09-19T07:29:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98e233a640 1 -- 172.21.15.83:0/3647953412 shutdown_connections 2024-09-19T07:29:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98e233a640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98bc0678c0 0x7f98bc069d80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:40.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98e233a640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98dc0c2530 0x7f98dc0c09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:40.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98e233a640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98dc0ab750 0x7f98dc0c0490 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:40.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98e233a640 1 --2- 172.21.15.83:0/3647953412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98dc0ab060 0x7f98dc0bd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:40.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98e233a640 1 -- 172.21.15.83:0/3647953412 >> 172.21.15.83:0/3647953412 conn(0x7f98dc06bc80 msgr2=0x7f98dc071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:40.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98e233a640 1 -- 172.21.15.83:0/3647953412 shutdown_connections 2024-09-19T07:29:40.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:40.246+0000 7f98e233a640 1 -- 172.21.15.83:0/3647953412 wait complete. 2024-09-19T07:29:40.260 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:29:42.260 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:42.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96c0d6d640 1 Processor -- start 2024-09-19T07:29:42.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96c0d6d640 1 -- start start 2024-09-19T07:29:42.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96c0d6d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1545e0 0x7f96bc06e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96c0d6d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96bc154a70 con 0x7f96bc071100 2024-09-19T07:29:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96c0d6d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96bc0a9e60 con 0x7f96bc0c28e0 2024-09-19T07:29:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96c0d6d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f96bc071b50 con 0x7f96bc1545e0 2024-09-19T07:29:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96ba59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f96bc071100 0x7f96bc150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52844/0 (socket says 172.21.15.83:52844) 2024-09-19T07:29:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96ba59c640 1 -- 172.21.15.83:0/3328984876 learned_addr learned my addr 172.21.15.83:0/3328984876 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:29:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96bad9d640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1545e0 0x7f96bc06e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1733455548 0 0) 0x7f96bc154a70 con 0x7f96bc071100 2024-09-19T07:29:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96a4003530 con 0x7f96bc071100 2024-09-19T07:29:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3592700098 0 0) 0x7f96a4003530 con 0x7f96bc071100 2024-09-19T07:29:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1545e0 msgr2=0x7f96bc06e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1545e0 0x7f96bc06e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 >> v1:172.21.15.184:6789/0 conn(0x7f96bc0c28e0 legacy=0x7f96bc06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96bc0aa140 con 0x7f96bc071100 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f96b0001680 con 0x7f96bc071100 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 >> v1:172.21.15.83:6789/0 conn(0x7f96bc071100 legacy=0x7f96bc150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1545e0 0x7f96a40039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96a4004020 0x7f96a4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96a4004d50 0x7f96a40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc071b50 con 0x7f96a4004d50 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f96a400bc10 con 0x7f96a4004020 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f96a400be10 con 0x7f96bc1545e0 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96bad9d640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96a4004d50 0x7f96a40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96ba59c640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1545e0 0x7f96a40039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b9d9b640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96a4004020 0x7f96a4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96bad9d640 1 -- 172.21.15.83:0/3328984876 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1545e0 msgr2=0x7f96a40039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96bad9d640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1545e0 0x7f96a40039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96bad9d640 1 -- 172.21.15.83:0/3328984876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96a4004020 msgr2=0x7f96a4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96bad9d640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96a4004020 0x7f96a4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96bad9d640 1 -- 172.21.15.83:0/3328984876 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a4003530 con 0x7f96a4004d50 2024-09-19T07:29:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.394+0000 7f96b9d9b640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96a4004020 0x7f96a4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96bad9d640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96a4004d50 0x7f96a40051d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f96b4007360 tx=0x7f96b401ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c191332f4140acf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b40b2070 con 0x7f96a4004d50 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96b40a5c30 con 0x7f96a4004d50 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96b959a640 1 -- 172.21.15.83:0/3328984876 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b4009be0 con 0x7f96a4004d50 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 -- 172.21.15.83:0/3328984876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96a4004d50 msgr2=0x7f96a40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 --2- 172.21.15.83:0/3328984876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96a4004d50 0x7f96a40051d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f96b4007360 tx=0x7f96b401ce40 comp rx=0 tx=0).stop 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96bad9d640 1 -- 172.21.15.83:0/3328984876 reap_dead start 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 -- 172.21.15.83:0/3328984876 shutdown_connections 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 -- 172.21.15.83:0/3328984876 >> 172.21.15.83:0/3328984876 conn(0x7f96bc06bc80 msgr2=0x7f96bc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 -- 172.21.15.83:0/3328984876 shutdown_connections 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 -- 172.21.15.83:0/3328984876 wait complete. 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 Processor -- start 2024-09-19T07:29:42.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 -- start start 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96bc149000 0x7f96bc1dd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96bc1493e0 0x7f96bc0c1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1e9460 0x7f96bc1e9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc0c2520 con 0x7f96bc1493e0 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc0c1ff0 con 0x7f96bc149000 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96c0d6d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f96bc151650 con 0x7f96bc1e9460 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96b9d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96bc1493e0 0x7f96bc0c1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96b9d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96bc1493e0 0x7f96bc0c1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53100/0 (socket says 172.21.15.83:53100) 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.398+0000 7f96b9d9b640 1 -- 172.21.15.83:0/1027126192 learned_addr learned my addr 172.21.15.83:0/1027126192 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96bad9d640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1e9460 0x7f96bc1e9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96b9d9b640 1 -- 172.21.15.83:0/1027126192 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1e9460 msgr2=0x7f96bc1e9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96b9d9b640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1e9460 0x7f96bc1e9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:42.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96b9d9b640 1 -- 172.21.15.83:0/1027126192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96bc149000 msgr2=0x7f96bc1dd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:42.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96b9d9b640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96bc149000 0x7f96bc1dd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:42.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96b9d9b640 1 -- 172.21.15.83:0/1027126192 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96bc1ed3b0 con 0x7f96bc1493e0 2024-09-19T07:29:42.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96bad9d640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1e9460 0x7f96bc1e9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:29:42.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96b9d9b640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96bc1493e0 0x7f96bc0c1a00 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f96ac00d7d0 tx=0x7f96ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:42.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96aaffd640 1 -- 172.21.15.83:0/1027126192 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac005270 con 0x7f96bc1493e0 2024-09-19T07:29:42.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96aaffd640 1 -- 172.21.15.83:0/1027126192 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96ac017070 con 0x7f96bc1493e0 2024-09-19T07:29:42.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96aaffd640 1 -- 172.21.15.83:0/1027126192 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac00b6b0 con 0x7f96bc1493e0 2024-09-19T07:29:42.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96c0d6d640 1 -- 172.21.15.83:0/1027126192 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96bc1ed630 con 0x7f96bc1493e0 2024-09-19T07:29:42.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96c0d6d640 1 -- 172.21.15.83:0/1027126192 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96bc0c6ac0 con 0x7f96bc1493e0 2024-09-19T07:29:42.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96c0d6d640 1 -- 172.21.15.83:0/1027126192 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96bc0714e0 con 0x7f96bc1493e0 2024-09-19T07:29:42.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96aaffd640 1 -- 172.21.15.83:0/1027126192 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f96ac00b870 con 0x7f96bc1493e0 2024-09-19T07:29:42.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96aaffd640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f969c0678f0 0x7f969c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:42.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96ba59c640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f969c0678f0 0x7f969c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:42.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96aaffd640 1 -- 172.21.15.83:0/1027126192 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7f96ac086260 con 0x7f96bc1493e0 2024-09-19T07:29:42.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.402+0000 7f96ba59c640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f969c0678f0 0x7f969c069db0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f96b0004850 tx=0x7f96b0001c40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:42.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.406+0000 7f96aaffd640 1 -- 172.21.15.83:0/1027126192 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f96bc1ed3b0 con 0x7f96bc1493e0 2024-09-19T07:29:42.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.606+0000 7f96c0d6d640 1 -- 172.21.15.83:0/1027126192 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96bc0c4f30 con 0x7f96bc1493e0 2024-09-19T07:29:42.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.606+0000 7f96aaffd640 1 -- 172.21.15.83:0/1027126192 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+6537 (secure 0 0 0) 0x7f96ac0565d0 con 0x7f96bc1493e0 2024-09-19T07:29:42.603 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:42.603 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":84,"btime":"2024-09-19T07:28:43:303745+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:42.603 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 84 2024-09-19T07:29:42.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.610+0000 7f96c0d6d640 1 -- 172.21.15.83:0/1027126192 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f969c0678f0 msgr2=0x7f969c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:42.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.610+0000 7f96c0d6d640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f969c0678f0 0x7f969c069db0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f96b0004850 tx=0x7f96b0001c40 comp rx=0 tx=0).stop 2024-09-19T07:29:42.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.610+0000 7f96c0d6d640 1 -- 172.21.15.83:0/1027126192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96bc1493e0 msgr2=0x7f96bc0c1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:42.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.610+0000 7f96c0d6d640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96bc1493e0 0x7f96bc0c1a00 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f96ac00d7d0 tx=0x7f96ac00dc90 comp rx=0 tx=0).stop 2024-09-19T07:29:42.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.610+0000 7f96c0d6d640 1 -- 172.21.15.83:0/1027126192 shutdown_connections 2024-09-19T07:29:42.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.610+0000 7f96c0d6d640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f969c0678f0 0x7f969c069db0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:42.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.610+0000 7f96c0d6d640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f96bc1e9460 0x7f96bc1e9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:42.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.610+0000 7f96c0d6d640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96bc1493e0 0x7f96bc0c1a00 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:42.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.610+0000 7f96c0d6d640 1 --2- 172.21.15.83:0/1027126192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96bc149000 0x7f96bc1dd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:42.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.610+0000 7f96c0d6d640 1 -- 172.21.15.83:0/1027126192 >> 172.21.15.83:0/1027126192 conn(0x7f96bc06bc80 msgr2=0x7f96bc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:42.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.610+0000 7f96c0d6d640 1 -- 172.21.15.83:0/1027126192 shutdown_connections 2024-09-19T07:29:42.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:42.610+0000 7f96c0d6d640 1 -- 172.21.15.83:0/1027126192 wait complete. 2024-09-19T07:29:42.621 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:29:44.622 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:44.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058c4b9640 1 Processor -- start 2024-09-19T07:29:44.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058c4b9640 1 -- start start 2024-09-19T07:29:44.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058c4b9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0584150290 0x7f05840c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:44.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058c4b9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0584153eb0 0x7f0584154340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:44.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058c4b9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0584150670 con 0x7f0584152b20 2024-09-19T07:29:44.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058c4b9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f05840716c0 con 0x7f0584150290 2024-09-19T07:29:44.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058c4b9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f05840c3920 con 0x7f0584153eb0 2024-09-19T07:29:44.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058a255640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0584150290 0x7f05840c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:44.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f0589a54640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0584152b20 0x7f05840c4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52852/0 (socket says 172.21.15.83:52852) 2024-09-19T07:29:44.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f0589a54640 1 -- 172.21.15.83:0/3601912661 learned_addr learned my addr 172.21.15.83:0/3601912661 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:29:44.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058a255640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0584150290 0x7f05840c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53102/0 (socket says 172.21.15.83:53102) 2024-09-19T07:29:44.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058aa56640 1 --2- 172.21.15.83:0/3601912661 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0584153eb0 0x7f0584154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:44.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058a255640 1 -- 172.21.15.83:0/3601912661 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0584153eb0 msgr2=0x7f0584154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:44.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058a255640 1 --2- 172.21.15.83:0/3601912661 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0584153eb0 0x7f0584154340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:44.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058a255640 1 -- 172.21.15.83:0/3601912661 >> v1:172.21.15.83:6789/0 conn(0x7f0584152b20 legacy=0x7f05840c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:44.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f058a255640 1 -- 172.21.15.83:0/3601912661 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05840c2880 con 0x7f0584150290 2024-09-19T07:29:44.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.758+0000 7f0589253640 1 -- 172.21.15.83:0/3601912661 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2846749042 0 0) 0x7f0584150670 con 0x7f0584152b20 2024-09-19T07:29:44.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058a255640 1 --2- 172.21.15.83:0/3601912661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0584150290 0x7f05840c1ff0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f05800097c0 tx=0x7f058001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d1cd067490f708 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:44.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f0589253640 1 -- 172.21.15.83:0/3601912661 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05800b2070 con 0x7f0584150290 2024-09-19T07:29:44.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f0589253640 1 -- 172.21.15.83:0/3601912661 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05800ad640 con 0x7f0584150290 2024-09-19T07:29:44.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f0589253640 1 -- 172.21.15.83:0/3601912661 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05800ada10 con 0x7f0584150290 2024-09-19T07:29:44.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 -- 172.21.15.83:0/3601912661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0584150290 msgr2=0x7f05840c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:44.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 --2- 172.21.15.83:0/3601912661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0584150290 0x7f05840c1ff0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f05800097c0 tx=0x7f058001c4a0 comp rx=0 tx=0).stop 2024-09-19T07:29:44.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 -- 172.21.15.83:0/3601912661 shutdown_connections 2024-09-19T07:29:44.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 --2- 172.21.15.83:0/3601912661 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0584153eb0 0x7f0584154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:44.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 --2- 172.21.15.83:0/3601912661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0584150290 0x7f05840c1ff0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:44.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 -- 172.21.15.83:0/3601912661 >> 172.21.15.83:0/3601912661 conn(0x7f058406bc80 msgr2=0x7f058406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:44.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 -- 172.21.15.83:0/3601912661 shutdown_connections 2024-09-19T07:29:44.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 -- 172.21.15.83:0/3601912661 wait complete. 2024-09-19T07:29:44.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 Processor -- start 2024-09-19T07:29:44.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 -- start start 2024-09-19T07:29:44.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0584150290 0x7f058414e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0584152b20 0x7f05841ee410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0584153eb0 0x7f05840c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f05840c4790 con 0x7f0584153eb0 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f05840c2530 con 0x7f0584152b20 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f05840c4910 con 0x7f0584150290 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058aa56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0584153eb0 0x7f05840c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058aa56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0584153eb0 0x7f05840c0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53106/0 (socket says 172.21.15.83:53106) 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058aa56640 1 -- 172.21.15.83:0/3577730577 learned_addr learned my addr 172.21.15.83:0/3577730577 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058a255640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0584150290 0x7f058414e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f0589a54640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0584152b20 0x7f05841ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058aa56640 1 -- 172.21.15.83:0/3577730577 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0584150290 msgr2=0x7f058414e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058aa56640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0584150290 0x7f058414e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058aa56640 1 -- 172.21.15.83:0/3577730577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0584152b20 msgr2=0x7f05841ee410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058aa56640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0584152b20 0x7f05841ee410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058aa56640 1 -- 172.21.15.83:0/3577730577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058414f9f0 con 0x7f0584153eb0 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058a255640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0584150290 0x7f058414e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058aa56640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0584153eb0 0x7f05840c0c80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f057c007e90 tx=0x7f057c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f057affd640 1 -- 172.21.15.83:0/3577730577 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c014420 con 0x7f0584153eb0 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f057affd640 1 -- 172.21.15.83:0/3577730577 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f057c019070 con 0x7f0584153eb0 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 -- 172.21.15.83:0/3577730577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058414eea0 con 0x7f0584153eb0 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f057affd640 1 -- 172.21.15.83:0/3577730577 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c014800 con 0x7f0584153eb0 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.762+0000 7f058c4b9640 1 -- 172.21.15.83:0/3577730577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f058414f460 con 0x7f0584153eb0 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.766+0000 7f058c4b9640 1 -- 172.21.15.83:0/3577730577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05840c1ff0 con 0x7f0584153eb0 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.766+0000 7f057affd640 1 -- 172.21.15.83:0/3577730577 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f057c0149a0 con 0x7f0584153eb0 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.766+0000 7f057affd640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05680678c0 0x7f0568069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.766+0000 7f057affd640 1 -- 172.21.15.83:0/3577730577 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7f057c086cc0 con 0x7f0584153eb0 2024-09-19T07:29:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.766+0000 7f058a255640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05680678c0 0x7f0568069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:44.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.766+0000 7f058a255640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05680678c0 0x7f0568069d80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f05800af000 tx=0x7f0580008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:44.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.770+0000 7f057affd640 1 -- 172.21.15.83:0/3577730577 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f057c011260 con 0x7f0584153eb0 2024-09-19T07:29:44.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.970+0000 7f058c4b9640 1 -- 172.21.15.83:0/3577730577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f058414ccc0 con 0x7f0584153eb0 2024-09-19T07:29:44.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.970+0000 7f057affd640 1 -- 172.21.15.83:0/3577730577 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+6537 (secure 0 0 0) 0x7f057c057030 con 0x7f0584153eb0 2024-09-19T07:29:44.966 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:44.966 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":84,"btime":"2024-09-19T07:28:43:303745+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 84 2024-09-19T07:29:44.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.974+0000 7f058c4b9640 1 -- 172.21.15.83:0/3577730577 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05680678c0 msgr2=0x7f0568069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:44.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.974+0000 7f058c4b9640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05680678c0 0x7f0568069d80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f05800af000 tx=0x7f0580008040 comp rx=0 tx=0).stop 2024-09-19T07:29:44.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.974+0000 7f058c4b9640 1 -- 172.21.15.83:0/3577730577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0584153eb0 msgr2=0x7f05840c0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:44.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.974+0000 7f058c4b9640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0584153eb0 0x7f05840c0c80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f057c007e90 tx=0x7f057c00e4b0 comp rx=0 tx=0).stop 2024-09-19T07:29:44.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.974+0000 7f058c4b9640 1 -- 172.21.15.83:0/3577730577 shutdown_connections 2024-09-19T07:29:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.974+0000 7f058c4b9640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f05680678c0 0x7f0568069d80 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.974+0000 7f058c4b9640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0584153eb0 0x7f05840c0c80 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.974+0000 7f058c4b9640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0584152b20 0x7f05841ee410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.974+0000 7f058c4b9640 1 --2- 172.21.15.83:0/3577730577 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0584150290 0x7f058414e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.974+0000 7f058c4b9640 1 -- 172.21.15.83:0/3577730577 >> 172.21.15.83:0/3577730577 conn(0x7f058406bc80 msgr2=0x7f058406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.974+0000 7f058c4b9640 1 -- 172.21.15.83:0/3577730577 shutdown_connections 2024-09-19T07:29:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:44.974+0000 7f058c4b9640 1 -- 172.21.15.83:0/3577730577 wait complete. 2024-09-19T07:29:44.984 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:29:46.985 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:47.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.122+0000 7fce60b94640 1 Processor -- start 2024-09-19T07:29:47.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.122+0000 7fce60b94640 1 -- start start 2024-09-19T07:29:47.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce60b94640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c0ab750 0x7fce5c0c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:47.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce60b94640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fce5c0c2530 0x7fce5c1528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:47.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce60b94640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fce5c0ab060 0x7fce5c152e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:47.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce60b94640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fce5c06de90 con 0x7fce5c0c2530 2024-09-19T07:29:47.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce60b94640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fce5c06e170 con 0x7fce5c0ab060 2024-09-19T07:29:47.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce60b94640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fce5c06e450 con 0x7fce5c0ab750 2024-09-19T07:29:47.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce51d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fce5c0c2530 0x7fce5c1528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:47.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce51d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fce5c0c2530 0x7fce5c1528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53110/0 (socket says 172.21.15.83:53110) 2024-09-19T07:29:47.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce51d9b640 1 -- 172.21.15.83:0/233062643 learned_addr learned my addr 172.21.15.83:0/233062643 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:47.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce5a59c640 1 --2- 172.21.15.83:0/233062643 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c0ab750 0x7fce5c0c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:47.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce5ad9d640 1 --2- 172.21.15.83:0/233062643 >> v2:172.21.15.184:3300/0 conn(0x7fce5c0ab060 0x7fce5c152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:47.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce51d9b640 1 -- 172.21.15.83:0/233062643 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c0ab750 msgr2=0x7fce5c0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:47.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce51d9b640 1 --2- 172.21.15.83:0/233062643 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c0ab750 0x7fce5c0c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:47.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce51d9b640 1 -- 172.21.15.83:0/233062643 >> v2:172.21.15.184:3300/0 conn(0x7fce5c0ab060 msgr2=0x7fce5c152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:47.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce51d9b640 1 --2- 172.21.15.83:0/233062643 >> v2:172.21.15.184:3300/0 conn(0x7fce5c0ab060 0x7fce5c152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:47.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce51d9b640 1 -- 172.21.15.83:0/233062643 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c06e730 con 0x7fce5c0c2530 2024-09-19T07:29:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce51d9b640 1 --2- 172.21.15.83:0/233062643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c0c2530 0x7fce5c1528c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fce5c070c60 tx=0x7fce4401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dbdef79ea73597c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce59d9b640 1 -- 172.21.15.83:0/233062643 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce440b0070 con 0x7fce5c0c2530 2024-09-19T07:29:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce59d9b640 1 -- 172.21.15.83:0/233062643 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce440ab600 con 0x7fce5c0c2530 2024-09-19T07:29:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce59d9b640 1 -- 172.21.15.83:0/233062643 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce440ab9d0 con 0x7fce5c0c2530 2024-09-19T07:29:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce60b94640 1 -- 172.21.15.83:0/233062643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c0c2530 msgr2=0x7fce5c1528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.126+0000 7fce60b94640 1 --2- 172.21.15.83:0/233062643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c0c2530 0x7fce5c1528c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fce5c070c60 tx=0x7fce4401c490 comp rx=0 tx=0).stop 2024-09-19T07:29:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 -- 172.21.15.83:0/233062643 shutdown_connections 2024-09-19T07:29:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 --2- 172.21.15.83:0/233062643 >> v2:172.21.15.184:3300/0 conn(0x7fce5c0ab060 0x7fce5c152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 --2- 172.21.15.83:0/233062643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c0c2530 0x7fce5c1528c0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 --2- 172.21.15.83:0/233062643 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c0ab750 0x7fce5c0c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 -- 172.21.15.83:0/233062643 >> 172.21.15.83:0/233062643 conn(0x7fce5c06bc80 msgr2=0x7fce5c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 -- 172.21.15.83:0/233062643 shutdown_connections 2024-09-19T07:29:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 -- 172.21.15.83:0/233062643 wait complete. 2024-09-19T07:29:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 Processor -- start 2024-09-19T07:29:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 -- start start 2024-09-19T07:29:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c0ab060 0x7fce5c15aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce5c0ab750 0x7fce5c09cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c0c2530 0x7fce5c09d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c156430 con 0x7fce5c0ab060 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c1562b0 con 0x7fce5c0ab750 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fce5c1565b0 con 0x7fce5c0c2530 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce5a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c0ab060 0x7fce5c15aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce5a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c0ab060 0x7fce5c15aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53120/0 (socket says 172.21.15.83:53120) 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce5a59c640 1 -- 172.21.15.83:0/3565216937 learned_addr learned my addr 172.21.15.83:0/3565216937 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce5ad9d640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c0c2530 0x7fce5c09d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce51d9b640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce5c0ab750 0x7fce5c09cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce5a59c640 1 -- 172.21.15.83:0/3565216937 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c0c2530 msgr2=0x7fce5c09d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce5a59c640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c0c2530 0x7fce5c09d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce5a59c640 1 -- 172.21.15.83:0/3565216937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce5c0ab750 msgr2=0x7fce5c09cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce5a59c640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce5c0ab750 0x7fce5c09cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:47.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce5a59c640 1 -- 172.21.15.83:0/3565216937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c0be470 con 0x7fce5c0ab060 2024-09-19T07:29:47.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce5a59c640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c0ab060 0x7fce5c15aff0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00c820 tx=0x7fce4c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:47.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce537fe640 1 -- 172.21.15.83:0/3565216937 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c015070 con 0x7fce5c0ab060 2024-09-19T07:29:47.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce537fe640 1 -- 172.21.15.83:0/3565216937 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce4c004240 con 0x7fce5c0ab060 2024-09-19T07:29:47.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 -- 172.21.15.83:0/3565216937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5c1ef8e0 con 0x7fce5c0ab060 2024-09-19T07:29:47.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce537fe640 1 -- 172.21.15.83:0/3565216937 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c00ce10 con 0x7fce5c0ab060 2024-09-19T07:29:47.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 -- 172.21.15.83:0/3565216937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce5c09d6e0 con 0x7fce5c0ab060 2024-09-19T07:29:47.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.130+0000 7fce60b94640 1 -- 172.21.15.83:0/3565216937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce5c0c4cc0 con 0x7fce5c0ab060 2024-09-19T07:29:47.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.134+0000 7fce537fe640 1 -- 172.21.15.83:0/3565216937 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fce4c007830 con 0x7fce5c0ab060 2024-09-19T07:29:47.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.134+0000 7fce537fe640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce3c0678f0 0x7fce3c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:47.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.134+0000 7fce537fe640 1 -- 172.21.15.83:0/3565216937 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7841+0+0 (secure 0 0 0) 0x7fce4c086780 con 0x7fce5c0ab060 2024-09-19T07:29:47.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.134+0000 7fce51d9b640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce3c0678f0 0x7fce3c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:47.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.134+0000 7fce51d9b640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce3c0678f0 0x7fce3c069db0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fce440ad000 tx=0x7fce440a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.138+0000 7fce537fe640 1 -- 172.21.15.83:0/3565216937 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fce4c056ad0 con 0x7fce5c0ab060 2024-09-19T07:29:47.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 -- 172.21.15.83:0/3565216937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce5c0c1ff0 con 0x7fce5c0ab060 2024-09-19T07:29:47.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce537fe640 1 -- 172.21.15.83:0/3565216937 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+6537 (secure 0 0 0) 0x7fce4c0563d0 con 0x7fce5c0ab060 2024-09-19T07:29:47.333 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:47.333 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":84,"btime":"2024-09-19T07:28:43:303745+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:28:11.164127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5991},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5991":{"gid":5991,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6837/1229829432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1229829432},{"type":"v1","addr":"172.21.15.83:6837","nonce":1229829432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5991,"qdb_cluster":[5991]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 84 2024-09-19T07:29:47.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 -- 172.21.15.83:0/3565216937 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce3c0678f0 msgr2=0x7fce3c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:47.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce3c0678f0 0x7fce3c069db0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fce440ad000 tx=0x7fce440a6000 comp rx=0 tx=0).stop 2024-09-19T07:29:47.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 -- 172.21.15.83:0/3565216937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c0ab060 msgr2=0x7fce5c15aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:47.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c0ab060 0x7fce5c15aff0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00c820 tx=0x7fce4c00cce0 comp rx=0 tx=0).stop 2024-09-19T07:29:47.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 -- 172.21.15.83:0/3565216937 shutdown_connections 2024-09-19T07:29:47.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce3c0678f0 0x7fce3c069db0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:47.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c0c2530 0x7fce5c09d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:47.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce5c0ab750 0x7fce5c09cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:47.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 --2- 172.21.15.83:0/3565216937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c0ab060 0x7fce5c15aff0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:47.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 -- 172.21.15.83:0/3565216937 >> 172.21.15.83:0/3565216937 conn(0x7fce5c06bc80 msgr2=0x7fce5c0abd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:47.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 -- 172.21.15.83:0/3565216937 shutdown_connections 2024-09-19T07:29:47.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:47.338+0000 7fce60b94640 1 -- 172.21.15.83:0/3565216937 wait complete. 2024-09-19T07:29:47.353 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:29:49.353 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:49.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9515c48640 1 Processor -- start 2024-09-19T07:29:49.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9515c48640 1 -- start start 2024-09-19T07:29:49.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9515c48640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9510150290 0x7f95100c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9515c48640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9510150670 con 0x7f9510152b20 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9515c48640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95100c2880 con 0x7f9510153eb0 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9515c48640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f95100716c0 con 0x7f9510150290 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9507fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9510152b20 0x7f95100c4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52864/0 (socket says 172.21.15.83:52864) 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9507fff640 1 -- 172.21.15.83:0/1983799571 learned_addr learned my addr 172.21.15.83:0/1983799571 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950f7fe640 1 --2- 172.21.15.83:0/1983799571 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9510150290 0x7f95100c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1076929002 0 0) 0x7f9510150670 con 0x7f9510152b20 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94fc003530 con 0x7f9510152b20 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3317664586 0 0) 0x7f94fc003530 con 0x7f9510152b20 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9510150290 msgr2=0x7f95100c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 --2- 172.21.15.83:0/1983799571 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9510150290 0x7f95100c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 >> v1:172.21.15.184:6789/0 conn(0x7f9510153eb0 legacy=0x7f9510154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95100c2b60 con 0x7f9510152b20 2024-09-19T07:29:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f94f8001680 con 0x7f9510152b20 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 >> v1:172.21.15.83:6789/0 conn(0x7f9510152b20 legacy=0x7f95100c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 --2- 172.21.15.83:0/1983799571 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9510150290 0x7f94fc0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 --2- 172.21.15.83:0/1983799571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94fc003f70 0x7f94fc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 --2- 172.21.15.83:0/1983799571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94fc004d50 0x7f94fc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f94fc00ba80 con 0x7f94fc003f70 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f94fc00bc30 con 0x7f94fc004d50 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f94fc00be30 con 0x7f9510150290 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9507fff640 1 --2- 172.21.15.83:0/1983799571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94fc003f70 0x7f94fc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f950ffff640 1 --2- 172.21.15.83:0/1983799571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94fc004d50 0x7f94fc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9507fff640 1 -- 172.21.15.83:0/1983799571 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9510150290 msgr2=0x7f94fc0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9507fff640 1 --2- 172.21.15.83:0/1983799571 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9510150290 0x7f94fc0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9507fff640 1 -- 172.21.15.83:0/1983799571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94fc004d50 msgr2=0x7f94fc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9507fff640 1 --2- 172.21.15.83:0/1983799571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94fc004d50 0x7f94fc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.490+0000 7f9507fff640 1 -- 172.21.15.83:0/1983799571 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94fc003530 con 0x7f94fc003f70 2024-09-19T07:29:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9507fff640 1 --2- 172.21.15.83:0/1983799571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94fc003f70 0x7f94fc004810 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9510070820 tx=0x7f94f801f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fca123817bd96b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8007d40 con 0x7f94fc003f70 2024-09-19T07:29:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94f8005b00 con 0x7f94fc003f70 2024-09-19T07:29:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950effd640 1 -- 172.21.15.83:0/1983799571 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8007070 con 0x7f94fc003f70 2024-09-19T07:29:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 -- 172.21.15.83:0/1983799571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94fc003f70 msgr2=0x7f94fc004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 --2- 172.21.15.83:0/1983799571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94fc003f70 0x7f94fc004810 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9510070820 tx=0x7f94f801f3d0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950ffff640 1 -- 172.21.15.83:0/1983799571 reap_dead start 2024-09-19T07:29:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 -- 172.21.15.83:0/1983799571 shutdown_connections 2024-09-19T07:29:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 -- 172.21.15.83:0/1983799571 >> 172.21.15.83:0/1983799571 conn(0x7f951006bc80 msgr2=0x7f951006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 -- 172.21.15.83:0/1983799571 shutdown_connections 2024-09-19T07:29:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 -- 172.21.15.83:0/1983799571 wait complete. 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 Processor -- start 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 -- start start 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94fc003f70 0x7f95100abd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95100ac5a0 0x7f95101de8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95101520d0 0x7f95101dc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f95100c3490 con 0x7f94fc003f70 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f95100c3120 con 0x7f95100ac5a0 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9515c48640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f95100c4790 con 0x7f95101520d0 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94fc003f70 0x7f95100abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94fc003f70 0x7f95100abd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53138/0 (socket says 172.21.15.83:53138) 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950f7fe640 1 -- 172.21.15.83:0/63582470 learned_addr learned my addr 172.21.15.83:0/63582470 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950ffff640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95101520d0 0x7f95101dc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9507fff640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95100ac5a0 0x7f95101de8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950f7fe640 1 -- 172.21.15.83:0/63582470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95101520d0 msgr2=0x7f95101dc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:49.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950f7fe640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95101520d0 0x7f95101dc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950f7fe640 1 -- 172.21.15.83:0/63582470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95100ac5a0 msgr2=0x7f95101de8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:49.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950f7fe640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95100ac5a0 0x7f95101de8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950f7fe640 1 -- 172.21.15.83:0/63582470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95101eb370 con 0x7f94fc003f70 2024-09-19T07:29:49.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950ffff640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95101520d0 0x7f95101dc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:29:49.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f9507fff640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95100ac5a0 0x7f95101de8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:29:49.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.494+0000 7f950f7fe640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94fc003f70 0x7f95100abd50 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f950000e820 tx=0x7f950000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:49.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.498+0000 7f950cff9640 1 -- 172.21.15.83:0/63582470 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500017070 con 0x7f94fc003f70 2024-09-19T07:29:49.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.498+0000 7f950cff9640 1 -- 172.21.15.83:0/63582470 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f950000c940 con 0x7f94fc003f70 2024-09-19T07:29:49.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.498+0000 7f950cff9640 1 -- 172.21.15.83:0/63582470 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950000cd30 con 0x7f94fc003f70 2024-09-19T07:29:49.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.498+0000 7f9515c48640 1 -- 172.21.15.83:0/63582470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95101eb650 con 0x7f94fc003f70 2024-09-19T07:29:49.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.498+0000 7f9515c48640 1 -- 172.21.15.83:0/63582470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9510152550 con 0x7f94fc003f70 2024-09-19T07:29:49.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.498+0000 7f9515c48640 1 -- 172.21.15.83:0/63582470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95101dee30 con 0x7f94fc003f70 2024-09-19T07:29:49.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.498+0000 7f950cff9640 1 -- 172.21.15.83:0/63582470 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f950001a070 con 0x7f94fc003f70 2024-09-19T07:29:49.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.498+0000 7f950cff9640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94f00678f0 0x7f94f0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.498+0000 7f950cff9640 1 -- 172.21.15.83:0/63582470 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f95000866f0 con 0x7f94fc003f70 2024-09-19T07:29:49.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.498+0000 7f9507fff640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94f00678f0 0x7f94f0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.498+0000 7f9507fff640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94f00678f0 0x7f94f0069db0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f94f80af000 tx=0x7f94f80aa000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:49.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.502+0000 7f950cff9640 1 -- 172.21.15.83:0/63582470 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f95000569f0 con 0x7f94fc003f70 2024-09-19T07:29:49.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.702+0000 7f9515c48640 1 -- 172.21.15.83:0/63582470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f951014ca50 con 0x7f94fc003f70 2024-09-19T07:29:49.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.702+0000 7f950cff9640 1 -- 172.21.15.83:0/63582470 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5709 (secure 0 0 0) 0x7f95000562f0 con 0x7f94fc003f70 2024-09-19T07:29:49.697 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:49.698 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":86,"btime":"2024-09-19T07:29:49:237214+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:48.391668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:49.698 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 86 2024-09-19T07:29:49.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.702+0000 7f9515c48640 1 -- 172.21.15.83:0/63582470 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94f00678f0 msgr2=0x7f94f0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:49.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.702+0000 7f9515c48640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94f00678f0 0x7f94f0069db0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f94f80af000 tx=0x7f94f80aa000 comp rx=0 tx=0).stop 2024-09-19T07:29:49.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.702+0000 7f9515c48640 1 -- 172.21.15.83:0/63582470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94fc003f70 msgr2=0x7f95100abd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:49.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.702+0000 7f9515c48640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94fc003f70 0x7f95100abd50 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f950000e820 tx=0x7f950000ece0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.702+0000 7f9515c48640 1 -- 172.21.15.83:0/63582470 shutdown_connections 2024-09-19T07:29:49.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.706+0000 7f9515c48640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94f00678f0 0x7f94f0069db0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.706+0000 7f9515c48640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95101520d0 0x7f95101dc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.706+0000 7f9515c48640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95100ac5a0 0x7f95101de8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.706+0000 7f9515c48640 1 --2- 172.21.15.83:0/63582470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94fc003f70 0x7f95100abd50 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.706+0000 7f9515c48640 1 -- 172.21.15.83:0/63582470 >> 172.21.15.83:0/63582470 conn(0x7f951006bc80 msgr2=0x7f951014d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:49.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.706+0000 7f9515c48640 1 -- 172.21.15.83:0/63582470 shutdown_connections 2024-09-19T07:29:49.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.706+0000 7f9515c48640 1 -- 172.21.15.83:0/63582470 wait complete. 2024-09-19T07:29:49.717 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-19T07:29:49.717 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:29:49.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:49.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.842+0000 7fba71633640 1 Processor -- start 2024-09-19T07:29:49.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.842+0000 7fba71633640 1 -- start start 2024-09-19T07:29:49.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba71633640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fba6c071100 0x7fba6c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba71633640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fba6c0c28e0 0x7fba6c150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba71633640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba6c0a9e60 con 0x7fba6c151480 2024-09-19T07:29:49.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba71633640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fba6c071b50 con 0x7fba6c0c28e0 2024-09-19T07:29:49.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba71633640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fba6c0c1ff0 con 0x7fba6c071100 2024-09-19T07:29:49.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba627fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fba6c0c28e0 0x7fba6c150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba627fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fba6c0c28e0 0x7fba6c150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53150/0 (socket says 172.21.15.83:53150) 2024-09-19T07:29:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba627fc640 1 -- 172.21.15.83:0/3082918075 learned_addr learned my addr 172.21.15.83:0/3082918075 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba6affd640 1 --2- 172.21.15.83:0/3082918075 >> v2:172.21.15.184:3300/0 conn(0x7fba6c071100 0x7fba6c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba627fc640 1 -- 172.21.15.83:0/3082918075 >> v2:172.21.15.184:3300/0 conn(0x7fba6c071100 msgr2=0x7fba6c1545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba627fc640 1 --2- 172.21.15.83:0/3082918075 >> v2:172.21.15.184:3300/0 conn(0x7fba6c071100 0x7fba6c1545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba627fc640 1 -- 172.21.15.83:0/3082918075 >> v1:172.21.15.184:6789/0 conn(0x7fba6c151480 legacy=0x7fba6c06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba627fc640 1 -- 172.21.15.83:0/3082918075 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6c0aa140 con 0x7fba6c0c28e0 2024-09-19T07:29:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba627fc640 1 --2- 172.21.15.83:0/3082918075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba6c0c28e0 0x7fba6c150f40 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fba6c06f620 tx=0x7fba5401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4c3f7ca689aeb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba6a7fc640 1 -- 172.21.15.83:0/3082918075 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba540b0070 con 0x7fba6c0c28e0 2024-09-19T07:29:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba6a7fc640 1 -- 172.21.15.83:0/3082918075 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba540ab600 con 0x7fba6c0c28e0 2024-09-19T07:29:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.846+0000 7fba6a7fc640 1 -- 172.21.15.83:0/3082918075 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba540ab9d0 con 0x7fba6c0c28e0 2024-09-19T07:29:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 -- 172.21.15.83:0/3082918075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba6c0c28e0 msgr2=0x7fba6c150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 --2- 172.21.15.83:0/3082918075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba6c0c28e0 0x7fba6c150f40 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fba6c06f620 tx=0x7fba5401c450 comp rx=0 tx=0).stop 2024-09-19T07:29:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 -- 172.21.15.83:0/3082918075 shutdown_connections 2024-09-19T07:29:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 --2- 172.21.15.83:0/3082918075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba6c0c28e0 0x7fba6c150f40 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 --2- 172.21.15.83:0/3082918075 >> v2:172.21.15.184:3300/0 conn(0x7fba6c071100 0x7fba6c1545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 -- 172.21.15.83:0/3082918075 >> 172.21.15.83:0/3082918075 conn(0x7fba6c06bc80 msgr2=0x7fba6c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:49.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 -- 172.21.15.83:0/3082918075 shutdown_connections 2024-09-19T07:29:49.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 -- 172.21.15.83:0/3082918075 wait complete. 2024-09-19T07:29:49.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 Processor -- start 2024-09-19T07:29:49.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 -- start start 2024-09-19T07:29:49.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba6c071100 0x7fba6c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba6c0c28e0 0x7fba6c15c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba6c151480 0x7fba6c1eef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c071660 con 0x7fba6c151480 2024-09-19T07:29:49.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c0714e0 con 0x7fba6c0c28e0 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fba6c160c60 con 0x7fba6c071100 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba6b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba6c151480 0x7fba6c1eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba6b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba6c151480 0x7fba6c1eef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53156/0 (socket says 172.21.15.83:53156) 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba6b7fe640 1 -- 172.21.15.83:0/2826999092 learned_addr learned my addr 172.21.15.83:0/2826999092 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba627fc640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba6c0c28e0 0x7fba6c15c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba6affd640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba6c071100 0x7fba6c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba6b7fe640 1 -- 172.21.15.83:0/2826999092 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba6c071100 msgr2=0x7fba6c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba6b7fe640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba6c071100 0x7fba6c155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba6b7fe640 1 -- 172.21.15.83:0/2826999092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba6c0c28e0 msgr2=0x7fba6c15c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba6b7fe640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba6c0c28e0 0x7fba6c15c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba6b7fe640 1 -- 172.21.15.83:0/2826999092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6c1ee790 con 0x7fba6c151480 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba627fc640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba6c0c28e0 0x7fba6c15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:29:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba6b7fe640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba6c151480 0x7fba6c1eef20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fba6400b970 tx=0x7fba6400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba63fff640 1 -- 172.21.15.83:0/2826999092 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba64019070 con 0x7fba6c151480 2024-09-19T07:29:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba63fff640 1 -- 172.21.15.83:0/2826999092 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba6400cb40 con 0x7fba6c151480 2024-09-19T07:29:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 -- 172.21.15.83:0/2826999092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba6c1578a0 con 0x7fba6c151480 2024-09-19T07:29:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba63fff640 1 -- 172.21.15.83:0/2826999092 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba64014440 con 0x7fba6c151480 2024-09-19T07:29:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 -- 172.21.15.83:0/2826999092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba6c15cdb0 con 0x7fba6c151480 2024-09-19T07:29:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.850+0000 7fba71633640 1 -- 172.21.15.83:0/2826999092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba6c1545e0 con 0x7fba6c151480 2024-09-19T07:29:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.854+0000 7fba63fff640 1 -- 172.21.15.83:0/2826999092 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fba6401c070 con 0x7fba6c151480 2024-09-19T07:29:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.854+0000 7fba63fff640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba4c0678c0 0x7fba4c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.854+0000 7fba63fff640 1 -- 172.21.15.83:0/2826999092 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7fba640861d0 con 0x7fba6c151480 2024-09-19T07:29:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.854+0000 7fba6affd640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba4c0678c0 0x7fba4c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.854+0000 7fba6affd640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba4c0678c0 0x7fba4c069d80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009790 tx=0x7fba5c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:49.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:49.858+0000 7fba63fff640 1 -- 172.21.15.83:0/2826999092 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fba64056440 con 0x7fba6c151480 2024-09-19T07:29:50.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.042+0000 7fba71633640 1 -- 172.21.15.83:0/2826999092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba6c0c4f30 con 0x7fba6c151480 2024-09-19T07:29:50.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.042+0000 7fba63fff640 1 -- 172.21.15.83:0/2826999092 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5709 (secure 0 0 0) 0x7fba64055d40 con 0x7fba6c151480 2024-09-19T07:29:50.039 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:50.039 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":86,"btime":"2024-09-19T07:29:49:237214+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:48.391668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:50.040 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 86 2024-09-19T07:29:50.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.046+0000 7fba71633640 1 -- 172.21.15.83:0/2826999092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba4c0678c0 msgr2=0x7fba4c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:50.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.046+0000 7fba71633640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba4c0678c0 0x7fba4c069d80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009790 tx=0x7fba5c008040 comp rx=0 tx=0).stop 2024-09-19T07:29:50.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.046+0000 7fba71633640 1 -- 172.21.15.83:0/2826999092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba6c151480 msgr2=0x7fba6c1eef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:50.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.046+0000 7fba71633640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba6c151480 0x7fba6c1eef20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fba6400b970 tx=0x7fba6400be30 comp rx=0 tx=0).stop 2024-09-19T07:29:50.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.046+0000 7fba71633640 1 -- 172.21.15.83:0/2826999092 shutdown_connections 2024-09-19T07:29:50.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.046+0000 7fba71633640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fba4c0678c0 0x7fba4c069d80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:50.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.046+0000 7fba71633640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba6c151480 0x7fba6c1eef20 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:50.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.046+0000 7fba71633640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba6c0c28e0 0x7fba6c15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:50.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.046+0000 7fba71633640 1 --2- 172.21.15.83:0/2826999092 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba6c071100 0x7fba6c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:50.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.046+0000 7fba71633640 1 -- 172.21.15.83:0/2826999092 >> 172.21.15.83:0/2826999092 conn(0x7fba6c06bc80 msgr2=0x7fba6c06c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:50.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.046+0000 7fba71633640 1 -- 172.21.15.83:0/2826999092 shutdown_connections 2024-09-19T07:29:50.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:50.046+0000 7fba71633640 1 -- 172.21.15.83:0/2826999092 wait complete. 2024-09-19T07:29:50.057 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:29:49:237214+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 86, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 86, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6159": { "addr": "172.21.15.83:6835/2004814399", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 2004814399, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 2004814399, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6159, "incarnation": 85, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:29:48.391668+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6159 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 83, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5476": { "addr": "172.21.15.184:6835/3841282254", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3841282254, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3841282254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5476, "incarnation": 80, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:28:41.296935+0000", "qdb_cluster": [ 5476 ], "qdb_leader": 5476, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5476 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 28, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:29:52.058 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:52.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5b019f5640 1 Processor -- start 2024-09-19T07:29:52.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5b019f5640 1 -- start start 2024-09-19T07:29:52.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5b019f5640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5afc1547b0 0x7f5afc157c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:52.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5b019f5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc072d00 0x7f5afc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:52.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5b019f5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5afc0c74f0 con 0x7f5afc0c7d70 2024-09-19T07:29:52.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5b019f5640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5afc073690 con 0x7f5afc1547b0 2024-09-19T07:29:52.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5b019f5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5afc157160 con 0x7f5afc072d00 2024-09-19T07:29:52.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5afaffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5afc0c7d70 0x7f5afc0c6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:32848/0 (socket says 172.21.15.83:32848) 2024-09-19T07:29:52.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5afaffd640 1 -- 172.21.15.83:0/390624968 learned_addr learned my addr 172.21.15.83:0/390624968 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:29:52.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5afa7fc640 1 --2- 172.21.15.83:0/390624968 >> v2:172.21.15.184:3300/0 conn(0x7f5afc1547b0 0x7f5afc157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:52.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5afb7fe640 1 --2- 172.21.15.83:0/390624968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc072d00 0x7f5afc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:52.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3275329529 0 0) 0x7f5afc0c74f0 con 0x7f5afc0c7d70 2024-09-19T07:29:52.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ae4003530 con 0x7f5afc0c7d70 2024-09-19T07:29:52.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 819583325 0 0) 0x7f5ae4003530 con 0x7f5afc0c7d70 2024-09-19T07:29:52.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc072d00 msgr2=0x7f5afc06cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:52.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5af9ffb640 1 --2- 172.21.15.83:0/390624968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc072d00 0x7f5afc06cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 >> v2:172.21.15.184:3300/0 conn(0x7f5afc1547b0 msgr2=0x7f5afc157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5af9ffb640 1 --2- 172.21.15.83:0/390624968 >> v2:172.21.15.184:3300/0 conn(0x7f5afc1547b0 0x7f5afc157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.194+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5afc157440 con 0x7f5afc0c7d70 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f5aec001680 con 0x7f5afc0c7d70 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 >> v1:172.21.15.83:6789/0 conn(0x7f5afc0c7d70 legacy=0x7f5afc0c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5af9ffb640 1 --2- 172.21.15.83:0/390624968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ae40039c0 0x7f5ae4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5af9ffb640 1 --2- 172.21.15.83:0/390624968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc072d00 0x7f5ae4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5af9ffb640 1 --2- 172.21.15.83:0/390624968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ae4004d50 0x7f5ae40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc073690 con 0x7f5ae4004d50 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc157160 con 0x7f5ae40039c0 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae400b9d0 con 0x7f5afc072d00 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5afb7fe640 1 --2- 172.21.15.83:0/390624968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ae4004d50 0x7f5ae40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5afaffd640 1 --2- 172.21.15.83:0/390624968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ae40039c0 0x7f5ae4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:52.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5afa7fc640 1 --2- 172.21.15.83:0/390624968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc072d00 0x7f5ae4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:52.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5afb7fe640 1 -- 172.21.15.83:0/390624968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc072d00 msgr2=0x7f5ae4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:52.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5afb7fe640 1 --2- 172.21.15.83:0/390624968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc072d00 0x7f5ae4004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:52.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5afb7fe640 1 -- 172.21.15.83:0/390624968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ae40039c0 msgr2=0x7f5ae4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:52.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5afb7fe640 1 --2- 172.21.15.83:0/390624968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ae40039c0 0x7f5ae4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:52.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5afb7fe640 1 -- 172.21.15.83:0/390624968 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5afc157440 con 0x7f5ae4004d50 2024-09-19T07:29:52.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5afb7fe640 1 --2- 172.21.15.83:0/390624968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ae4004d50 0x7f5ae40051d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009fa0 tx=0x7f5af401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b23bf082360cb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:52.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af40b2070 con 0x7f5ae4004d50 2024-09-19T07:29:52.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af40a5d10 con 0x7f5ae4004d50 2024-09-19T07:29:52.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5af9ffb640 1 -- 172.21.15.83:0/390624968 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af4009bf0 con 0x7f5ae4004d50 2024-09-19T07:29:52.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5b019f5640 1 -- 172.21.15.83:0/390624968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ae4004d50 msgr2=0x7f5ae40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:52.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5b019f5640 1 --2- 172.21.15.83:0/390624968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ae4004d50 0x7f5ae40051d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009fa0 tx=0x7f5af401cc40 comp rx=0 tx=0).stop 2024-09-19T07:29:52.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5afb7fe640 1 -- 172.21.15.83:0/390624968 reap_dead start 2024-09-19T07:29:52.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5b019f5640 1 -- 172.21.15.83:0/390624968 shutdown_connections 2024-09-19T07:29:52.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5b019f5640 1 -- 172.21.15.83:0/390624968 >> 172.21.15.83:0/390624968 conn(0x7f5afc0c5060 msgr2=0x7f5afc0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:52.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5b019f5640 1 -- 172.21.15.83:0/390624968 shutdown_connections 2024-09-19T07:29:52.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.198+0000 7f5b019f5640 1 -- 172.21.15.83:0/390624968 wait complete. 2024-09-19T07:29:52.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5b019f5640 1 Processor -- start 2024-09-19T07:29:52.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5b019f5640 1 -- start start 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5b019f5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5afc155d00 0x7f5afc1e8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5b019f5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc152db0 0x7f5afc1560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5b019f5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc1517b0 0x7f5afc150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5b019f5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc0c8150 con 0x7f5afc155d00 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5b019f5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc073970 con 0x7f5afc1517b0 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5b019f5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5afc0c82d0 con 0x7f5afc152db0 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5afaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5afc155d00 0x7f5afc1e8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5afaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5afc155d00 0x7f5afc1e8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37452/0 (socket says 172.21.15.83:37452) 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5afaffd640 1 -- 172.21.15.83:0/2201252214 learned_addr learned my addr 172.21.15.83:0/2201252214 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5afa7fc640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc152db0 0x7f5afc1560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5afb7fe640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc1517b0 0x7f5afc150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5afaffd640 1 -- 172.21.15.83:0/2201252214 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc152db0 msgr2=0x7f5afc1560e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5afaffd640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc152db0 0x7f5afc1560e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:52.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5afaffd640 1 -- 172.21.15.83:0/2201252214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc1517b0 msgr2=0x7f5afc150340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:52.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5afaffd640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc1517b0 0x7f5afc150340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:52.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5afaffd640 1 -- 172.21.15.83:0/2201252214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5afc153320 con 0x7f5afc155d00 2024-09-19T07:29:52.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5afaffd640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5afc155d00 0x7f5afc1e8090 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5aec001210 tx=0x7f5aec011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:52.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5ae3fff640 1 -- 172.21.15.83:0/2201252214 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec010040 con 0x7f5afc155d00 2024-09-19T07:29:52.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5ae3fff640 1 -- 172.21.15.83:0/2201252214 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aec01a070 con 0x7f5afc155d00 2024-09-19T07:29:52.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5b019f5640 1 -- 172.21.15.83:0/2201252214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5afc1e75b0 con 0x7f5afc155d00 2024-09-19T07:29:52.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5ae3fff640 1 -- 172.21.15.83:0/2201252214 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec00fe40 con 0x7f5afc155d00 2024-09-19T07:29:52.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5b019f5640 1 -- 172.21.15.83:0/2201252214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5afc1e7b70 con 0x7f5afc155d00 2024-09-19T07:29:52.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.202+0000 7f5b019f5640 1 -- 172.21.15.83:0/2201252214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5afc073190 con 0x7f5afc155d00 2024-09-19T07:29:52.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.206+0000 7f5ae3fff640 1 -- 172.21.15.83:0/2201252214 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5aec01f460 con 0x7f5afc155d00 2024-09-19T07:29:52.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.206+0000 7f5ae3fff640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5ae80678c0 0x7f5ae8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:52.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.206+0000 7f5afa7fc640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5ae80678c0 0x7f5ae8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:52.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.206+0000 7f5ae3fff640 1 -- 172.21.15.83:0/2201252214 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f5aec089800 con 0x7f5afc155d00 2024-09-19T07:29:52.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.206+0000 7f5afa7fc640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5ae80678c0 0x7f5ae8069d80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009ab0 tx=0x7f5af0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:52.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.210+0000 7f5ae3fff640 1 -- 172.21.15.83:0/2201252214 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5aec059b20 con 0x7f5afc155d00 2024-09-19T07:29:52.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.454+0000 7f5b019f5640 1 -- 172.21.15.83:0/2201252214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5afc0c2c40 con 0x7f5afc155d00 2024-09-19T07:29:52.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.454+0000 7f5ae3fff640 1 -- 172.21.15.83:0/2201252214 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5713 (secure 0 0 0) 0x7f5aec059420 con 0x7f5afc155d00 2024-09-19T07:29:52.451 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:52.452 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":88,"btime":"2024-09-19T07:29:51:259408+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:52.452 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 88 2024-09-19T07:29:52.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.458+0000 7f5b019f5640 1 -- 172.21.15.83:0/2201252214 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5ae80678c0 msgr2=0x7f5ae8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:52.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.458+0000 7f5b019f5640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5ae80678c0 0x7f5ae8069d80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009ab0 tx=0x7f5af0008040 comp rx=0 tx=0).stop 2024-09-19T07:29:52.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.458+0000 7f5b019f5640 1 -- 172.21.15.83:0/2201252214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5afc155d00 msgr2=0x7f5afc1e8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:52.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.458+0000 7f5b019f5640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5afc155d00 0x7f5afc1e8090 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5aec001210 tx=0x7f5aec011dc0 comp rx=0 tx=0).stop 2024-09-19T07:29:52.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.458+0000 7f5b019f5640 1 -- 172.21.15.83:0/2201252214 shutdown_connections 2024-09-19T07:29:52.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.458+0000 7f5b019f5640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5ae80678c0 0x7f5ae8069d80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:52.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.458+0000 7f5b019f5640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5afc1517b0 0x7f5afc150340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:52.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.458+0000 7f5b019f5640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5afc152db0 0x7f5afc1560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:52.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.458+0000 7f5b019f5640 1 --2- 172.21.15.83:0/2201252214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5afc155d00 0x7f5afc1e8090 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:52.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.458+0000 7f5b019f5640 1 -- 172.21.15.83:0/2201252214 >> 172.21.15.83:0/2201252214 conn(0x7f5afc0c5060 msgr2=0x7f5afc0a8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:52.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.458+0000 7f5b019f5640 1 -- 172.21.15.83:0/2201252214 shutdown_connections 2024-09-19T07:29:52.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:52.458+0000 7f5b019f5640 1 -- 172.21.15.83:0/2201252214 wait complete. 2024-09-19T07:29:52.470 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=5991 2024-09-19T07:29:52.471 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.c 2024-09-19T07:29:53.313 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:29:53.318+0000 7fb51914d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:29:56.472 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-19T07:29:56.472 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:29:56.472 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:29:56.475 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:29:56.475 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:56.552 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:29:56.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7daf500640 1 Processor -- start 2024-09-19T07:29:56.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7daf500640 1 -- start start 2024-09-19T07:29:56.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7daf500640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7da8153570 0x7f7da80c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:56.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7daf500640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7da80c5110 0x7f7da8159270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7daf500640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da80c3c20 con 0x7f7da80c4130 2024-09-19T07:29:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7daf500640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7da80c89f0 con 0x7f7da80c5110 2024-09-19T07:29:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7daf500640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f7da80c8cd0 con 0x7f7da8153570 2024-09-19T07:29:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7daca9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7da80c5110 0x7f7da8159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7dada9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7da80c4130 0x7f7da81597b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:32858/0 (socket says 172.21.15.83:32858) 2024-09-19T07:29:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7dada9d640 1 -- 172.21.15.83:0/3390336107 learned_addr learned my addr 172.21.15.83:0/3390336107 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:29:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7dad29c640 1 --2- 172.21.15.83:0/3390336107 >> v2:172.21.15.184:3300/0 conn(0x7f7da8153570 0x7f7da80c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7daca9b640 1 -- 172.21.15.83:0/3390336107 >> v2:172.21.15.184:3300/0 conn(0x7f7da8153570 msgr2=0x7f7da80c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7daca9b640 1 --2- 172.21.15.83:0/3390336107 >> v2:172.21.15.184:3300/0 conn(0x7f7da8153570 0x7f7da80c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7daca9b640 1 -- 172.21.15.83:0/3390336107 >> v1:172.21.15.83:6789/0 conn(0x7f7da80c4130 legacy=0x7f7da81597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7d9ffff640 1 -- 172.21.15.83:0/3390336107 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3536730917 0 0) 0x7f7da80c3c20 con 0x7f7da80c4130 2024-09-19T07:29:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.614+0000 7f7daca9b640 1 -- 172.21.15.83:0/3390336107 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da80c8fb0 con 0x7f7da80c5110 2024-09-19T07:29:56.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daca9b640 1 --2- 172.21.15.83:0/3390336107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7da80c5110 0x7f7da8159270 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7da80c3ee0 tx=0x7f7d9801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b942eb4ecbc43d0f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:56.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7d9ffff640 1 -- 172.21.15.83:0/3390336107 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d980b2070 con 0x7f7da80c5110 2024-09-19T07:29:56.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7d9ffff640 1 -- 172.21.15.83:0/3390336107 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d980ad680 con 0x7f7da80c5110 2024-09-19T07:29:56.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7d9ffff640 1 -- 172.21.15.83:0/3390336107 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d980ada50 con 0x7f7da80c5110 2024-09-19T07:29:56.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 -- 172.21.15.83:0/3390336107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7da80c5110 msgr2=0x7f7da8159270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:56.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 --2- 172.21.15.83:0/3390336107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7da80c5110 0x7f7da8159270 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7da80c3ee0 tx=0x7f7d9801c360 comp rx=0 tx=0).stop 2024-09-19T07:29:56.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 -- 172.21.15.83:0/3390336107 shutdown_connections 2024-09-19T07:29:56.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 --2- 172.21.15.83:0/3390336107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7da80c5110 0x7f7da8159270 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:56.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 --2- 172.21.15.83:0/3390336107 >> v2:172.21.15.184:3300/0 conn(0x7f7da8153570 0x7f7da80c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:56.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 -- 172.21.15.83:0/3390336107 >> 172.21.15.83:0/3390336107 conn(0x7f7da80c6110 msgr2=0x7f7da80c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 -- 172.21.15.83:0/3390336107 shutdown_connections 2024-09-19T07:29:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 -- 172.21.15.83:0/3390336107 wait complete. 2024-09-19T07:29:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 Processor -- start 2024-09-19T07:29:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 -- start start 2024-09-19T07:29:56.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7da80c4130 0x7f7da8158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:56.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7da80c5110 0x7f7da814ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:56.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7da8153570 0x7f7da80adc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:56.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7da8163070 con 0x7f7da8153570 2024-09-19T07:29:56.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7da8162ef0 con 0x7f7da80c5110 2024-09-19T07:29:56.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daf500640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7da81631f0 con 0x7f7da80c4130 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7dada9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7da8153570 0x7f7da80adc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7dada9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7da8153570 0x7f7da80adc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37498/0 (socket says 172.21.15.83:37498) 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7daca9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7da80c5110 0x7f7da814ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7dada9d640 1 -- 172.21.15.83:0/216558173 learned_addr learned my addr 172.21.15.83:0/216558173 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7dad29c640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7da80c4130 0x7f7da8158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7dada9d640 1 -- 172.21.15.83:0/216558173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7da80c4130 msgr2=0x7f7da8158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7dada9d640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7da80c4130 0x7f7da8158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7dada9d640 1 -- 172.21.15.83:0/216558173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7da80c5110 msgr2=0x7f7da814ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7dada9d640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7da80c5110 0x7f7da814ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.618+0000 7f7dada9d640 1 -- 172.21.15.83:0/216558173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da8152060 con 0x7f7da8153570 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.622+0000 7f7dada9d640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7da8153570 0x7f7da80adc90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7da000c700 tx=0x7f7da000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.622+0000 7f7d9dffb640 1 -- 172.21.15.83:0/216558173 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da0018070 con 0x7f7da8153570 2024-09-19T07:29:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.622+0000 7f7d9dffb640 1 -- 172.21.15.83:0/216558173 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da000ce10 con 0x7f7da8153570 2024-09-19T07:29:56.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.622+0000 7f7d9dffb640 1 -- 172.21.15.83:0/216558173 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da000a880 con 0x7f7da8153570 2024-09-19T07:29:56.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.622+0000 7f7daf500640 1 -- 172.21.15.83:0/216558173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da81468a0 con 0x7f7da8153570 2024-09-19T07:29:56.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.622+0000 7f7daf500640 1 -- 172.21.15.83:0/216558173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da8157ce0 con 0x7f7da8153570 2024-09-19T07:29:56.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.622+0000 7f7daf500640 1 -- 172.21.15.83:0/216558173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7da80c4bd0 con 0x7f7da8153570 2024-09-19T07:29:56.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.622+0000 7f7d9dffb640 1 -- 172.21.15.83:0/216558173 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7da000aa20 con 0x7f7da8153570 2024-09-19T07:29:56.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.622+0000 7f7d9dffb640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7d8c0678c0 0x7f7d8c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:56.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.622+0000 7f7d9dffb640 1 -- 172.21.15.83:0/216558173 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f7da00859b0 con 0x7f7da8153570 2024-09-19T07:29:56.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.622+0000 7f7dad29c640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7d8c0678c0 0x7f7d8c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:56.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.622+0000 7f7dad29c640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7d8c0678c0 0x7f7d8c069d80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7da40079c0 tx=0x7f7da4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:56.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.626+0000 7f7d9dffb640 1 -- 172.21.15.83:0/216558173 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7da00555f0 con 0x7f7da8153570 2024-09-19T07:29:56.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.906+0000 7f7daf500640 1 -- 172.21.15.83:0/216558173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7da8150520 con 0x7f7da8153570 2024-09-19T07:29:56.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.906+0000 7f7d9dffb640 1 -- 172.21.15.83:0/216558173 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5713 (secure 0 0 0) 0x7f7da00598f0 con 0x7f7da8153570 2024-09-19T07:29:56.903 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:56.903 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":88,"btime":"2024-09-19T07:29:51:259408+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:56.903 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 88 2024-09-19T07:29:56.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.910+0000 7f7daf500640 1 -- 172.21.15.83:0/216558173 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7d8c0678c0 msgr2=0x7f7d8c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:56.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.910+0000 7f7daf500640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7d8c0678c0 0x7f7d8c069d80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7da40079c0 tx=0x7f7da4008040 comp rx=0 tx=0).stop 2024-09-19T07:29:56.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.910+0000 7f7daf500640 1 -- 172.21.15.83:0/216558173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7da8153570 msgr2=0x7f7da80adc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:56.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.910+0000 7f7daf500640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7da8153570 0x7f7da80adc90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7da000c700 tx=0x7f7da000cbc0 comp rx=0 tx=0).stop 2024-09-19T07:29:56.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.910+0000 7f7daf500640 1 -- 172.21.15.83:0/216558173 shutdown_connections 2024-09-19T07:29:56.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.910+0000 7f7daf500640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7d8c0678c0 0x7f7d8c069d80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:56.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.910+0000 7f7daf500640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7da8153570 0x7f7da80adc90 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:56.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.910+0000 7f7daf500640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7da80c5110 0x7f7da814ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:56.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.910+0000 7f7daf500640 1 --2- 172.21.15.83:0/216558173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7da80c4130 0x7f7da8158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:56.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.910+0000 7f7daf500640 1 -- 172.21.15.83:0/216558173 >> 172.21.15.83:0/216558173 conn(0x7f7da80c6110 msgr2=0x7f7da815f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:56.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.910+0000 7f7daf500640 1 -- 172.21.15.83:0/216558173 shutdown_connections 2024-09-19T07:29:56.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:56.910+0000 7f7daf500640 1 -- 172.21.15.83:0/216558173 wait complete. 2024-09-19T07:29:56.922 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-19T07:29:58.924 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:29:59.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.066+0000 7fd5f6695640 1 Processor -- start 2024-09-19T07:29:59.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.066+0000 7fd5f6695640 1 -- start start 2024-09-19T07:29:59.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.066+0000 7fd5f6695640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5f00c6360 0x7fd5f0153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:59.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.066+0000 7fd5f6695640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd5f00bee30 0x7fd5f01570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:59.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f6695640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd5f00c9680 0x7fd5f0157630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:59.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f6695640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd5f0157b70 con 0x7fd5f00c9680 2024-09-19T07:29:59.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f6695640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd5f00c5b90 con 0x7fd5f00bee30 2024-09-19T07:29:59.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f6695640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f015a050 con 0x7fd5f00c6360 2024-09-19T07:29:59.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f4c32640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd5f00c9680 0x7fd5f0157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:59.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f4c32640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd5f00c9680 0x7fd5f0157630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37506/0 (socket says 172.21.15.83:37506) 2024-09-19T07:29:59.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f4c32640 1 -- 172.21.15.83:0/452495048 learned_addr learned my addr 172.21.15.83:0/452495048 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:59.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5e7fff640 1 --2- 172.21.15.83:0/452495048 >> v2:172.21.15.184:3300/0 conn(0x7fd5f00bee30 0x7fd5f01570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:59.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5effff640 1 --2- 172.21.15.83:0/452495048 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5f00c6360 0x7fd5f0153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:59.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f4c32640 1 -- 172.21.15.83:0/452495048 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5f00c6360 msgr2=0x7fd5f0153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:59.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f4c32640 1 --2- 172.21.15.83:0/452495048 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5f00c6360 0x7fd5f0153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:59.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f4c32640 1 -- 172.21.15.83:0/452495048 >> v2:172.21.15.184:3300/0 conn(0x7fd5f00bee30 msgr2=0x7fd5f01570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:29:59.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f4c32640 1 --2- 172.21.15.83:0/452495048 >> v2:172.21.15.184:3300/0 conn(0x7fd5f00bee30 0x7fd5f01570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:59.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f4c32640 1 -- 172.21.15.83:0/452495048 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f006c950 con 0x7fd5f00c9680 2024-09-19T07:29:59.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f4c32640 1 --2- 172.21.15.83:0/452495048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5f00c9680 0x7fd5f0157630 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd5f00cc880 tx=0x7fd5e801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c98e95443c816d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:59.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5ef7fe640 1 -- 172.21.15.83:0/452495048 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e80a56d0 con 0x7fd5f00c9680 2024-09-19T07:29:59.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5ef7fe640 1 -- 172.21.15.83:0/452495048 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5e80a5870 con 0x7fd5f00c9680 2024-09-19T07:29:59.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5ef7fe640 1 -- 172.21.15.83:0/452495048 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e80a5c40 con 0x7fd5f00c9680 2024-09-19T07:29:59.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f6695640 1 -- 172.21.15.83:0/452495048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5f00c9680 msgr2=0x7fd5f0157630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:59.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.070+0000 7fd5f6695640 1 --2- 172.21.15.83:0/452495048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5f00c9680 0x7fd5f0157630 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd5f00cc880 tx=0x7fd5e801c960 comp rx=0 tx=0).stop 2024-09-19T07:29:59.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 -- 172.21.15.83:0/452495048 shutdown_connections 2024-09-19T07:29:59.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 --2- 172.21.15.83:0/452495048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5f00c9680 0x7fd5f0157630 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:59.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 --2- 172.21.15.83:0/452495048 >> v2:172.21.15.184:3300/0 conn(0x7fd5f00bee30 0x7fd5f01570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:59.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 --2- 172.21.15.83:0/452495048 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5f00c6360 0x7fd5f0153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:59.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 -- 172.21.15.83:0/452495048 >> 172.21.15.83:0/452495048 conn(0x7fd5f00722a0 msgr2=0x7fd5f00726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:59.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 -- 172.21.15.83:0/452495048 shutdown_connections 2024-09-19T07:29:59.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 -- 172.21.15.83:0/452495048 wait complete. 2024-09-19T07:29:59.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 Processor -- start 2024-09-19T07:29:59.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 -- start start 2024-09-19T07:29:59.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5f00bee30 0x7fd5f0148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:59.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5f00c6360 0x7fd5f0156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:59.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5f00c9680 0x7fd5f01efeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:59.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f00c6740 con 0x7fd5f00c9680 2024-09-19T07:29:59.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f0159850 con 0x7fd5f00bee30 2024-09-19T07:29:59.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f00c68c0 con 0x7fd5f00c6360 2024-09-19T07:29:59.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f4c32640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5f00c9680 0x7fd5f01efeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:59.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f4c32640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5f00c9680 0x7fd5f01efeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37510/0 (socket says 172.21.15.83:37510) 2024-09-19T07:29:59.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f4c32640 1 -- 172.21.15.83:0/52584113 learned_addr learned my addr 172.21.15.83:0/52584113 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:29:59.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5e7fff640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5f00c6360 0x7fd5f0156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:59.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5effff640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5f00bee30 0x7fd5f0148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:59.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f4c32640 1 -- 172.21.15.83:0/52584113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5f00c6360 msgr2=0x7fd5f0156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:59.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f4c32640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5f00c6360 0x7fd5f0156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:59.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f4c32640 1 -- 172.21.15.83:0/52584113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5f00bee30 msgr2=0x7fd5f0148770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:59.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f4c32640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5f00bee30 0x7fd5f0148770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:59.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f4c32640 1 -- 172.21.15.83:0/52584113 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f0148f00 con 0x7fd5f00c9680 2024-09-19T07:29:59.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f4c32640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5f00c9680 0x7fd5f01efeb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd5e80af040 tx=0x7fd5f0156c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:59.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5ed7fa640 1 -- 172.21.15.83:0/52584113 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e80bc030 con 0x7fd5f00c9680 2024-09-19T07:29:59.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5ed7fa640 1 -- 172.21.15.83:0/52584113 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5e80b8470 con 0x7fd5f00c9680 2024-09-19T07:29:59.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 -- 172.21.15.83:0/52584113 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f00abde0 con 0x7fd5f00c9680 2024-09-19T07:29:59.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5ed7fa640 1 -- 172.21.15.83:0/52584113 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e80b8880 con 0x7fd5f00c9680 2024-09-19T07:29:59.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.074+0000 7fd5f6695640 1 -- 172.21.15.83:0/52584113 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f0151bf0 con 0x7fd5f00c9680 2024-09-19T07:29:59.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.078+0000 7fd5f6695640 1 -- 172.21.15.83:0/52584113 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f0153050 con 0x7fd5f00c9680 2024-09-19T07:29:59.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.078+0000 7fd5ed7fa640 1 -- 172.21.15.83:0/52584113 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd5e80bf070 con 0x7fd5f00c9680 2024-09-19T07:29:59.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.078+0000 7fd5ed7fa640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5d00678c0 0x7fd5d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:29:59.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.078+0000 7fd5ed7fa640 1 -- 172.21.15.83:0/52584113 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7fd5e8120fc0 con 0x7fd5f00c9680 2024-09-19T07:29:59.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.078+0000 7fd5effff640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5d00678c0 0x7fd5d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:29:59.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.078+0000 7fd5effff640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5d00678c0 0x7fd5d0069d80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00079c0 tx=0x7fd5e0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:29:59.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.082+0000 7fd5ed7fa640 1 -- 172.21.15.83:0/52584113 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd5e80c6070 con 0x7fd5f00c9680 2024-09-19T07:29:59.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.330+0000 7fd5f6695640 1 -- 172.21.15.83:0/52584113 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5f00c5970 con 0x7fd5f00c9680 2024-09-19T07:29:59.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.330+0000 7fd5ed7fa640 1 -- 172.21.15.83:0/52584113 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7fd5e80f7030 con 0x7fd5f00c9680 2024-09-19T07:29:59.325 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:29:59.326 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:29:59.326 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:29:59.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.334+0000 7fd5f6695640 1 -- 172.21.15.83:0/52584113 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5d00678c0 msgr2=0x7fd5d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:59.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.334+0000 7fd5f6695640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5d00678c0 0x7fd5d0069d80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00079c0 tx=0x7fd5e0008040 comp rx=0 tx=0).stop 2024-09-19T07:29:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.334+0000 7fd5f6695640 1 -- 172.21.15.83:0/52584113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5f00c9680 msgr2=0x7fd5f01efeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:29:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.334+0000 7fd5f6695640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5f00c9680 0x7fd5f01efeb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd5e80af040 tx=0x7fd5f0156c30 comp rx=0 tx=0).stop 2024-09-19T07:29:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.334+0000 7fd5f6695640 1 -- 172.21.15.83:0/52584113 shutdown_connections 2024-09-19T07:29:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.334+0000 7fd5f6695640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5d00678c0 0x7fd5d0069d80 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.334+0000 7fd5f6695640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5f00c9680 0x7fd5f01efeb0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.334+0000 7fd5f6695640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5f00c6360 0x7fd5f0156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.334+0000 7fd5f6695640 1 --2- 172.21.15.83:0/52584113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5f00bee30 0x7fd5f0148770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:29:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.334+0000 7fd5f6695640 1 -- 172.21.15.83:0/52584113 >> 172.21.15.83:0/52584113 conn(0x7fd5f00722a0 msgr2=0x7fd5f00cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:29:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.334+0000 7fd5f6695640 1 -- 172.21.15.83:0/52584113 shutdown_connections 2024-09-19T07:29:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:29:59.334+0000 7fd5f6695640 1 -- 172.21.15.83:0/52584113 wait complete. 2024-09-19T07:29:59.347 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-19T07:29:59.347 INFO:tasks.mds_thrash.fs.[a]:waiting for 31 secs before thrashing 2024-09-19T07:30:02.586 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:30:02.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:30:29.443 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:29.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.581+0000 7fb803812640 1 Processor -- start 2024-09-19T07:30:29.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.581+0000 7fb803812640 1 -- start start 2024-09-19T07:30:29.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb803812640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb7fc154540 0x7fb7fc157490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:29.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb803812640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb7fc1579d0 0x7fb7fc15e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:29.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb803812640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7fc151750 con 0x7fb7fc14a500 2024-09-19T07:30:29.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb803812640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb7fc0c7e90 con 0x7fb7fc1579d0 2024-09-19T07:30:29.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb803812640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fb7fc073110 con 0x7fb7fc154540 2024-09-19T07:30:29.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb801daf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb7fc1579d0 0x7fb7fc15e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:29.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb801daf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb7fc1579d0 0x7fb7fc15e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50704/0 (socket says 172.21.15.83:50704) 2024-09-19T07:30:29.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb801daf640 1 -- 172.21.15.83:0/3711240949 learned_addr learned my addr 172.21.15.83:0/3711240949 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:29.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb800dad640 1 --2- 172.21.15.83:0/3711240949 >> v2:172.21.15.184:3300/0 conn(0x7fb7fc154540 0x7fb7fc157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:29.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb801daf640 1 -- 172.21.15.83:0/3711240949 >> v2:172.21.15.184:3300/0 conn(0x7fb7fc154540 msgr2=0x7fb7fc157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:29.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb801daf640 1 --2- 172.21.15.83:0/3711240949 >> v2:172.21.15.184:3300/0 conn(0x7fb7fc154540 0x7fb7fc157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb801daf640 1 -- 172.21.15.83:0/3711240949 >> v1:172.21.15.184:6789/0 conn(0x7fb7fc14a500 legacy=0x7fb7fc0c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:29.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb801daf640 1 -- 172.21.15.83:0/3711240949 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7fc0733f0 con 0x7fb7fc1579d0 2024-09-19T07:30:29.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb801daf640 1 --2- 172.21.15.83:0/3711240949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7fc1579d0 0x7fb7fc15e440 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0c94f0 tx=0x7fb7f401c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7474a13b58cb76 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:29.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.585+0000 7fb7f3fff640 1 -- 172.21.15.83:0/3711240949 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f40a5620 con 0x7fb7fc1579d0 2024-09-19T07:30:29.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb7f3fff640 1 -- 172.21.15.83:0/3711240949 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7f40a57c0 con 0x7fb7fc1579d0 2024-09-19T07:30:29.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb7f3fff640 1 -- 172.21.15.83:0/3711240949 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f40a5b90 con 0x7fb7fc1579d0 2024-09-19T07:30:29.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 -- 172.21.15.83:0/3711240949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7fc1579d0 msgr2=0x7fb7fc15e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:29.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 --2- 172.21.15.83:0/3711240949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7fc1579d0 0x7fb7fc15e440 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0c94f0 tx=0x7fb7f401c7a0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 -- 172.21.15.83:0/3711240949 shutdown_connections 2024-09-19T07:30:29.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 --2- 172.21.15.83:0/3711240949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7fc1579d0 0x7fb7fc15e440 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 --2- 172.21.15.83:0/3711240949 >> v2:172.21.15.184:3300/0 conn(0x7fb7fc154540 0x7fb7fc157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 -- 172.21.15.83:0/3711240949 >> 172.21.15.83:0/3711240949 conn(0x7fb7fc0c9610 msgr2=0x7fb7fc0c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:29.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 -- 172.21.15.83:0/3711240949 shutdown_connections 2024-09-19T07:30:29.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 -- 172.21.15.83:0/3711240949 wait complete. 2024-09-19T07:30:29.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 Processor -- start 2024-09-19T07:30:29.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 -- start start 2024-09-19T07:30:29.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7fc14a500 0x7fb7fc14c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:29.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7fc154540 0x7fb7fc155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:29.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7fc1579d0 0x7fb7fc156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:29.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc0c7850 con 0x7fb7fc154540 2024-09-19T07:30:29.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc0c76d0 con 0x7fb7fc1579d0 2024-09-19T07:30:29.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb7fc0c4720 con 0x7fb7fc14a500 2024-09-19T07:30:29.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb800dad640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7fc154540 0x7fb7fc155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:29.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb800dad640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7fc154540 0x7fb7fc155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50706/0 (socket says 172.21.15.83:50706) 2024-09-19T07:30:29.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb800dad640 1 -- 172.21.15.83:0/1601442798 learned_addr learned my addr 172.21.15.83:0/1601442798 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:29.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb801daf640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7fc1579d0 0x7fb7fc156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:29.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb8015ae640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7fc14a500 0x7fb7fc14c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:29.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb800dad640 1 -- 172.21.15.83:0/1601442798 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7fc14a500 msgr2=0x7fb7fc14c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:29.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb800dad640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7fc14a500 0x7fb7fc14c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb800dad640 1 -- 172.21.15.83:0/1601442798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7fc1579d0 msgr2=0x7fb7fc156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:29.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb800dad640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7fc1579d0 0x7fb7fc156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb800dad640 1 -- 172.21.15.83:0/1601442798 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7fc14af80 con 0x7fb7fc154540 2024-09-19T07:30:29.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb801daf640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7fc1579d0 0x7fb7fc156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:30:29.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb800dad640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7fc154540 0x7fb7fc155190 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00d830 tx=0x7fb7ec00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:29.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb7f1ffb640 1 -- 172.21.15.83:0/1601442798 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec00b6c0 con 0x7fb7fc154540 2024-09-19T07:30:29.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb7f1ffb640 1 -- 172.21.15.83:0/1601442798 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ec019070 con 0x7fb7fc154540 2024-09-19T07:30:29.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 -- 172.21.15.83:0/1601442798 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7fc14d7d0 con 0x7fb7fc154540 2024-09-19T07:30:29.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb7f1ffb640 1 -- 172.21.15.83:0/1601442798 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec00ba60 con 0x7fb7fc154540 2024-09-19T07:30:29.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.589+0000 7fb803812640 1 -- 172.21.15.83:0/1601442798 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7fc06dc40 con 0x7fb7fc154540 2024-09-19T07:30:29.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.593+0000 7fb803812640 1 -- 172.21.15.83:0/1601442798 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7fc157e20 con 0x7fb7fc154540 2024-09-19T07:30:29.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.593+0000 7fb7f1ffb640 1 -- 172.21.15.83:0/1601442798 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb7ec01e460 con 0x7fb7fc154540 2024-09-19T07:30:29.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.593+0000 7fb7f1ffb640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb7e00678c0 0x7fb7e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:29.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.593+0000 7fb7f1ffb640 1 -- 172.21.15.83:0/1601442798 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7fb7ec086bb0 con 0x7fb7fc154540 2024-09-19T07:30:29.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.593+0000 7fb8015ae640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb7e00678c0 0x7fb7e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:29.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.593+0000 7fb8015ae640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb7e00678c0 0x7fb7e0069d80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc071070 tx=0x7fb7f8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:29.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.601+0000 7fb7f1ffb640 1 -- 172.21.15.83:0/1601442798 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb7ec056ed0 con 0x7fb7fc154540 2024-09-19T07:30:29.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.821+0000 7fb803812640 1 -- 172.21.15.83:0/1601442798 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7fc073fd0 con 0x7fb7fc154540 2024-09-19T07:30:29.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.821+0000 7fb7f1ffb640 1 -- 172.21.15.83:0/1601442798 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7fb7ec0567d0 con 0x7fb7fc154540 2024-09-19T07:30:29.818 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:29.818 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:29.818 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:29.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.825+0000 7fb803812640 1 -- 172.21.15.83:0/1601442798 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb7e00678c0 msgr2=0x7fb7e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:29.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.825+0000 7fb803812640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb7e00678c0 0x7fb7e0069d80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc071070 tx=0x7fb7f8008040 comp rx=0 tx=0).stop 2024-09-19T07:30:29.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.825+0000 7fb803812640 1 -- 172.21.15.83:0/1601442798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7fc154540 msgr2=0x7fb7fc155190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:29.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.825+0000 7fb803812640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7fc154540 0x7fb7fc155190 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00d830 tx=0x7fb7ec00dcf0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.825+0000 7fb803812640 1 -- 172.21.15.83:0/1601442798 shutdown_connections 2024-09-19T07:30:29.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.825+0000 7fb803812640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb7e00678c0 0x7fb7e0069d80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.825+0000 7fb803812640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7fc1579d0 0x7fb7fc156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.825+0000 7fb803812640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7fc154540 0x7fb7fc155190 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.825+0000 7fb803812640 1 --2- 172.21.15.83:0/1601442798 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7fc14a500 0x7fb7fc14c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.825+0000 7fb803812640 1 -- 172.21.15.83:0/1601442798 >> 172.21.15.83:0/1601442798 conn(0x7fb7fc0c9610 msgr2=0x7fb7fc0c8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:29.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.825+0000 7fb803812640 1 -- 172.21.15.83:0/1601442798 shutdown_connections 2024-09-19T07:30:29.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.825+0000 7fb803812640 1 -- 172.21.15.83:0/1601442798 wait complete. 2024-09-19T07:30:29.837 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-09-19T07:30:29.838 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-19T07:30:29.838 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:30:29.855 INFO:tasks.ceph.mds.b:Stopped 2024-09-19T07:30:29.855 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:29.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.981+0000 7fc8f2a26640 1 Processor -- start 2024-09-19T07:30:29.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.981+0000 7fc8f2a26640 1 -- start start 2024-09-19T07:30:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.981+0000 7fc8f2a26640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc8ec0c2c30 0x7fc8ec152e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.981+0000 7fc8f2a26640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ec159c60 con 0x7fc8ec0aaa60 2024-09-19T07:30:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.981+0000 7fc8f2a26640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ec0c4b90 con 0x7fc8ec06f070 2024-09-19T07:30:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.981+0000 7fc8f2a26640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc8ec15a160 con 0x7fc8ec0c2c30 2024-09-19T07:30:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8ebfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc8ec0c2c30 0x7fc8ec152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8ebfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc8ec0c2c30 0x7fc8ec152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50712/0 (socket says 172.21.15.83:50712) 2024-09-19T07:30:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8ebfff640 1 -- 172.21.15.83:0/3138278759 learned_addr learned my addr 172.21.15.83:0/3138278759 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8ebfff640 1 -- 172.21.15.83:0/3138278759 >> v1:172.21.15.184:6789/0 conn(0x7fc8ec06f070 legacy=0x7fc8ec155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8ebfff640 1 -- 172.21.15.83:0/3138278759 >> v1:172.21.15.83:6789/0 conn(0x7fc8ec0aaa60 legacy=0x7fc8ec1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:29.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8ebfff640 1 -- 172.21.15.83:0/3138278759 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ec0aa4a0 con 0x7fc8ec0c2c30 2024-09-19T07:30:29.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8eb7fe640 1 -- 172.21.15.83:0/3138278759 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 319506411 0 0) 0x7fc8ec159c60 con 0x7fc8ec0aaa60 2024-09-19T07:30:29.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8ebfff640 1 --2- 172.21.15.83:0/3138278759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8ec0c2c30 0x7fc8ec152e90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0098a0 tx=0x7fc8dc01cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292ab007be6b6631 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:29.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8eb7fe640 1 -- 172.21.15.83:0/3138278759 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc0b1070 con 0x7fc8ec0c2c30 2024-09-19T07:30:29.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8eb7fe640 1 -- 172.21.15.83:0/3138278759 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8dc0ac620 con 0x7fc8ec0c2c30 2024-09-19T07:30:29.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8eb7fe640 1 -- 172.21.15.83:0/3138278759 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc0ac9f0 con 0x7fc8ec0c2c30 2024-09-19T07:30:29.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8f2a26640 1 -- 172.21.15.83:0/3138278759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8ec0c2c30 msgr2=0x7fc8ec152e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:29.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8f2a26640 1 --2- 172.21.15.83:0/3138278759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8ec0c2c30 0x7fc8ec152e90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0098a0 tx=0x7fc8dc01cf10 comp rx=0 tx=0).stop 2024-09-19T07:30:29.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8f2a26640 1 -- 172.21.15.83:0/3138278759 shutdown_connections 2024-09-19T07:30:29.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8f2a26640 1 --2- 172.21.15.83:0/3138278759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8ec0c2c30 0x7fc8ec152e90 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8f2a26640 1 -- 172.21.15.83:0/3138278759 >> 172.21.15.83:0/3138278759 conn(0x7fc8ec0c95a0 msgr2=0x7fc8ec0c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:29.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8f2a26640 1 -- 172.21.15.83:0/3138278759 shutdown_connections 2024-09-19T07:30:29.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.985+0000 7fc8f2a26640 1 -- 172.21.15.83:0/3138278759 wait complete. 2024-09-19T07:30:29.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8f2a26640 1 Processor -- start 2024-09-19T07:30:29.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8f2a26640 1 -- start start 2024-09-19T07:30:29.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8f2a26640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8ec06f070 0x7fc8ec0ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:29.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8f2a26640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8ec0aaa60 0x7fc8ec06e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:29.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8f2a26640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8ec0c2c30 0x7fc8ec15a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8f2a26640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ec0c3010 con 0x7fc8ec06f070 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8f2a26640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ec0c3ca0 con 0x7fc8ec0aaa60 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8f2a26640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc8ec0c3190 con 0x7fc8ec0c2c30 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8ebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8ec06f070 0x7fc8ec0ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8ebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8ec06f070 0x7fc8ec0ad300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50726/0 (socket says 172.21.15.83:50726) 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8ebfff640 1 -- 172.21.15.83:0/1069327040 learned_addr learned my addr 172.21.15.83:0/1069327040 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8f0fc3640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8ec0c2c30 0x7fc8ec15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8ebfff640 1 -- 172.21.15.83:0/1069327040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8ec0c2c30 msgr2=0x7fc8ec15a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8ebfff640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8ec0c2c30 0x7fc8ec15a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8ebfff640 1 -- 172.21.15.83:0/1069327040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8ec0aaa60 msgr2=0x7fc8ec06e540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8ebfff640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8ec0aaa60 0x7fc8ec06e540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8ebfff640 1 -- 172.21.15.83:0/1069327040 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ec06bed0 con 0x7fc8ec06f070 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8ebfff640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8ec06f070 0x7fc8ec0ad300 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0af040 tx=0x7fc8dc01acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:29.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8e97fa640 1 -- 172.21.15.83:0/1069327040 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc0bb020 con 0x7fc8ec06f070 2024-09-19T07:30:29.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8e97fa640 1 -- 172.21.15.83:0/1069327040 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8dc0b7620 con 0x7fc8ec06f070 2024-09-19T07:30:29.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8f2a26640 1 -- 172.21.15.83:0/1069327040 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8ec06c1b0 con 0x7fc8ec06f070 2024-09-19T07:30:29.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8e97fa640 1 -- 172.21.15.83:0/1069327040 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc0b1040 con 0x7fc8ec06f070 2024-09-19T07:30:29.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.989+0000 7fc8f2a26640 1 -- 172.21.15.83:0/1069327040 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8ec15ca30 con 0x7fc8ec06f070 2024-09-19T07:30:29.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.993+0000 7fc8f2a26640 1 -- 172.21.15.83:0/1069327040 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8ec06cc70 con 0x7fc8ec06f070 2024-09-19T07:30:29.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.993+0000 7fc8e97fa640 1 -- 172.21.15.83:0/1069327040 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc8dc0be020 con 0x7fc8ec06f070 2024-09-19T07:30:29.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.993+0000 7fc8e97fa640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8cc0678f0 0x7fc8cc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:29.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.993+0000 7fc8e97fa640 1 -- 172.21.15.83:0/1069327040 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7fc8dc120820 con 0x7fc8ec06f070 2024-09-19T07:30:29.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.993+0000 7fc8e3fff640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8cc0678f0 0x7fc8cc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:29.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.993+0000 7fc8e3fff640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8cc0678f0 0x7fc8cc069db0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4000f70 tx=0x7fc8d40079c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:29.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:29.997+0000 7fc8e97fa640 1 -- 172.21.15.83:0/1069327040 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc8ec06cc70 con 0x7fc8ec06f070 2024-09-19T07:30:30.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.229+0000 7fc8f2a26640 1 -- 172.21.15.83:0/1069327040 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8ec155ce0 con 0x7fc8ec06f070 2024-09-19T07:30:30.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.229+0000 7fc8e97fa640 1 -- 172.21.15.83:0/1069327040 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7fc8dc0c5030 con 0x7fc8ec06f070 2024-09-19T07:30:30.226 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:30.226 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:30.227 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:30.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.233+0000 7fc8f2a26640 1 -- 172.21.15.83:0/1069327040 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8cc0678f0 msgr2=0x7fc8cc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:30.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.233+0000 7fc8f2a26640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8cc0678f0 0x7fc8cc069db0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4000f70 tx=0x7fc8d40079c0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.233+0000 7fc8f2a26640 1 -- 172.21.15.83:0/1069327040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8ec06f070 msgr2=0x7fc8ec0ad300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:30.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.233+0000 7fc8f2a26640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8ec06f070 0x7fc8ec0ad300 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0af040 tx=0x7fc8dc01acf0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.233+0000 7fc8f2a26640 1 -- 172.21.15.83:0/1069327040 shutdown_connections 2024-09-19T07:30:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.233+0000 7fc8f2a26640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8cc0678f0 0x7fc8cc069db0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.233+0000 7fc8f2a26640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8ec0c2c30 0x7fc8ec15a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.233+0000 7fc8f2a26640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8ec0aaa60 0x7fc8ec06e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.233+0000 7fc8f2a26640 1 --2- 172.21.15.83:0/1069327040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8ec06f070 0x7fc8ec0ad300 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.233+0000 7fc8f2a26640 1 -- 172.21.15.83:0/1069327040 >> 172.21.15.83:0/1069327040 conn(0x7fc8ec0c95a0 msgr2=0x7fc8ec0c9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.233+0000 7fc8f2a26640 1 -- 172.21.15.83:0/1069327040 shutdown_connections 2024-09-19T07:30:30.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.233+0000 7fc8f2a26640 1 -- 172.21.15.83:0/1069327040 wait complete. 2024-09-19T07:30:30.245 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:30:30.348 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:30.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.469+0000 7f7ed9020640 1 Processor -- start 2024-09-19T07:30:30.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.469+0000 7f7ed9020640 1 -- start start 2024-09-19T07:30:30.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ed9020640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7ed4070140 0x7f7ed40c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ed9020640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7ed41579a0 0x7f7ed4150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ed9020640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ed4070830 con 0x7f7ed406cef0 2024-09-19T07:30:30.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ed9020640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7ed4153cf0 con 0x7f7ed4070140 2024-09-19T07:30:30.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ed9020640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f7ed4157df0 con 0x7f7ed41579a0 2024-09-19T07:30:30.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ecbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7ed4070140 0x7f7ed40c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ecbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7ed4070140 0x7f7ed40c2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39710/0 (socket says 172.21.15.83:39710) 2024-09-19T07:30:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ecbfff640 1 -- 172.21.15.83:0/3614134582 learned_addr learned my addr 172.21.15.83:0/3614134582 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ed359e640 1 --2- 172.21.15.83:0/3614134582 >> v2:172.21.15.184:3300/0 conn(0x7f7ed41579a0 0x7f7ed4150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ecbfff640 1 -- 172.21.15.83:0/3614134582 >> v2:172.21.15.184:3300/0 conn(0x7f7ed41579a0 msgr2=0x7f7ed4150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ecbfff640 1 --2- 172.21.15.83:0/3614134582 >> v2:172.21.15.184:3300/0 conn(0x7f7ed41579a0 0x7f7ed4150ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ecbfff640 1 -- 172.21.15.83:0/3614134582 >> v1:172.21.15.184:6789/0 conn(0x7f7ed406cef0 legacy=0x7f7ed4154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ecbfff640 1 -- 172.21.15.83:0/3614134582 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed41539a0 con 0x7f7ed4070140 2024-09-19T07:30:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ecbfff640 1 --2- 172.21.15.83:0/3614134582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ed4070140 0x7f7ed40c2f90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7ed406d730 tx=0x7f7ebc01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2504959e0c1aa3c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:30.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ed259c640 1 -- 172.21.15.83:0/3614134582 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc0b0070 con 0x7f7ed4070140 2024-09-19T07:30:30.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ed259c640 1 -- 172.21.15.83:0/3614134582 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ebc0ab600 con 0x7f7ed4070140 2024-09-19T07:30:30.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.473+0000 7f7ed259c640 1 -- 172.21.15.83:0/3614134582 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc0ab9d0 con 0x7f7ed4070140 2024-09-19T07:30:30.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 -- 172.21.15.83:0/3614134582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ed4070140 msgr2=0x7f7ed40c2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:30.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 --2- 172.21.15.83:0/3614134582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ed4070140 0x7f7ed40c2f90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7ed406d730 tx=0x7f7ebc01c3c0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 -- 172.21.15.83:0/3614134582 shutdown_connections 2024-09-19T07:30:30.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 --2- 172.21.15.83:0/3614134582 >> v2:172.21.15.184:3300/0 conn(0x7f7ed41579a0 0x7f7ed4150ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 --2- 172.21.15.83:0/3614134582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ed4070140 0x7f7ed40c2f90 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 -- 172.21.15.83:0/3614134582 >> 172.21.15.83:0/3614134582 conn(0x7f7ed40c5090 msgr2=0x7f7ed40c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:30.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 -- 172.21.15.83:0/3614134582 shutdown_connections 2024-09-19T07:30:30.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 -- 172.21.15.83:0/3614134582 wait complete. 2024-09-19T07:30:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 Processor -- start 2024-09-19T07:30:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 -- start start 2024-09-19T07:30:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ed406cef0 0x7f7ed40c1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ed4070140 0x7f7ed41ef080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ed41579a0 0x7f7ed41ef5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed406d2d0 con 0x7f7ed41579a0 2024-09-19T07:30:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed406f8a0 con 0x7f7ed4070140 2024-09-19T07:30:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed406d450 con 0x7f7ed406cef0 2024-09-19T07:30:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ed41579a0 0x7f7ed41ef5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ed41579a0 0x7f7ed41ef5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39720/0 (socket says 172.21.15.83:39720) 2024-09-19T07:30:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed359e640 1 -- 172.21.15.83:0/2145517596 learned_addr learned my addr 172.21.15.83:0/2145517596 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed2d9d640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ed406cef0 0x7f7ed40c1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ecbfff640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ed4070140 0x7f7ed41ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed359e640 1 -- 172.21.15.83:0/2145517596 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ed406cef0 msgr2=0x7f7ed40c1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed359e640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ed406cef0 0x7f7ed40c1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed359e640 1 -- 172.21.15.83:0/2145517596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ed4070140 msgr2=0x7f7ed41ef080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed359e640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ed4070140 0x7f7ed41ef080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed359e640 1 -- 172.21.15.83:0/2145517596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed4068690 con 0x7f7ed41579a0 2024-09-19T07:30:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed359e640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ed41579a0 0x7f7ed41ef5c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc007dc0 tx=0x7f7ecc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ecb7fe640 1 -- 172.21.15.83:0/2145517596 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc014420 con 0x7f7ed41579a0 2024-09-19T07:30:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ecb7fe640 1 -- 172.21.15.83:0/2145517596 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ecc019070 con 0x7f7ed41579a0 2024-09-19T07:30:30.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ecb7fe640 1 -- 172.21.15.83:0/2145517596 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc014820 con 0x7f7ed41579a0 2024-09-19T07:30:30.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 -- 172.21.15.83:0/2145517596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed40c8280 con 0x7f7ed41579a0 2024-09-19T07:30:30.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 -- 172.21.15.83:0/2145517596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed40c7cc0 con 0x7f7ed41579a0 2024-09-19T07:30:30.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.477+0000 7f7ed9020640 1 -- 172.21.15.83:0/2145517596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed40c2f90 con 0x7f7ed41579a0 2024-09-19T07:30:30.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.481+0000 7f7ecb7fe640 1 -- 172.21.15.83:0/2145517596 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7ecc0149c0 con 0x7f7ed41579a0 2024-09-19T07:30:30.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.481+0000 7f7ecb7fe640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7eb40678c0 0x7f7eb4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.481+0000 7f7ecb7fe640 1 -- 172.21.15.83:0/2145517596 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f7ecc086d50 con 0x7f7ed41579a0 2024-09-19T07:30:30.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.481+0000 7f7ed2d9d640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7eb40678c0 0x7f7eb4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.481+0000 7f7ed2d9d640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7eb40678c0 0x7f7eb4069d80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400b810 tx=0x7f7ec4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.485+0000 7f7ecb7fe640 1 -- 172.21.15.83:0/2145517596 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7ecc08a030 con 0x7f7ed41579a0 2024-09-19T07:30:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.753+0000 7f7ed9020640 1 -- 172.21.15.83:0/2145517596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ed4050050 con 0x7f7ed41579a0 2024-09-19T07:30:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.753+0000 7f7ecb7fe640 1 -- 172.21.15.83:0/2145517596 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7f7ecc05be20 con 0x7f7ed41579a0 2024-09-19T07:30:30.750 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:30.750 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:30.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.757+0000 7f7ed9020640 1 -- 172.21.15.83:0/2145517596 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7eb40678c0 msgr2=0x7f7eb4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:30.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.757+0000 7f7ed9020640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7eb40678c0 0x7f7eb4069d80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400b810 tx=0x7f7ec4008040 comp rx=0 tx=0).stop 2024-09-19T07:30:30.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.757+0000 7f7ed9020640 1 -- 172.21.15.83:0/2145517596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ed41579a0 msgr2=0x7f7ed41ef5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:30.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.757+0000 7f7ed9020640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ed41579a0 0x7f7ed41ef5c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc007dc0 tx=0x7f7ecc00e4b0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.757+0000 7f7ed9020640 1 -- 172.21.15.83:0/2145517596 shutdown_connections 2024-09-19T07:30:30.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.757+0000 7f7ed9020640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7eb40678c0 0x7f7eb4069d80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.757+0000 7f7ed9020640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ed41579a0 0x7f7ed41ef5c0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.757+0000 7f7ed9020640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ed4070140 0x7f7ed41ef080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.757+0000 7f7ed9020640 1 --2- 172.21.15.83:0/2145517596 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ed406cef0 0x7f7ed40c1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.757+0000 7f7ed9020640 1 -- 172.21.15.83:0/2145517596 >> 172.21.15.83:0/2145517596 conn(0x7f7ed40c5090 msgr2=0x7f7ed4154180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:30.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.757+0000 7f7ed9020640 1 -- 172.21.15.83:0/2145517596 shutdown_connections 2024-09-19T07:30:30.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.757+0000 7f7ed9020640 1 -- 172.21.15.83:0/2145517596 wait complete. 2024-09-19T07:30:30.771 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-19T07:30:30.771 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:30:30.771 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:30:30.780 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:30:30.780 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:30.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf52844640 1 Processor -- start 2024-09-19T07:30:30.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf52844640 1 -- start start 2024-09-19T07:30:30.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf52844640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcf4c06cef0 0x7fcf4c0c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf52844640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf4c06ddd0 0x7fcf4c1534f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf52844640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4c0c6460 0x7fcf4c14fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf52844640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fcf4c06f730 con 0x7fcf4c06ddd0 2024-09-19T07:30:30.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf52844640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fcf4c06e210 con 0x7fcf4c06cef0 2024-09-19T07:30:30.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf52844640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcf4c0c7fb0 con 0x7fcf4c0c6460 2024-09-19T07:30:30.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf4b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf4c06ddd0 0x7fcf4c1534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf4b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf4c06ddd0 0x7fcf4c1534f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39722/0 (socket says 172.21.15.83:39722) 2024-09-19T07:30:30.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf4b7fe640 1 -- 172.21.15.83:0/566350573 learned_addr learned my addr 172.21.15.83:0/566350573 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:30.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf50de1640 1 --2- 172.21.15.83:0/566350573 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4c0c6460 0x7fcf4c14fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf4bfff640 1 --2- 172.21.15.83:0/566350573 >> v2:172.21.15.184:3300/0 conn(0x7fcf4c06cef0 0x7fcf4c0c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf4b7fe640 1 -- 172.21.15.83:0/566350573 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4c0c6460 msgr2=0x7fcf4c14fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:30.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf4b7fe640 1 --2- 172.21.15.83:0/566350573 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4c0c6460 0x7fcf4c14fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf4b7fe640 1 -- 172.21.15.83:0/566350573 >> v2:172.21.15.184:3300/0 conn(0x7fcf4c06cef0 msgr2=0x7fcf4c0c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:30.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf4b7fe640 1 --2- 172.21.15.83:0/566350573 >> v2:172.21.15.184:3300/0 conn(0x7fcf4c06cef0 0x7fcf4c0c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf4b7fe640 1 -- 172.21.15.83:0/566350573 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4c06f9f0 con 0x7fcf4c06ddd0 2024-09-19T07:30:30.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.957+0000 7fcf50de1640 1 --2- 172.21.15.83:0/566350573 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4c0c6460 0x7fcf4c14fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:30:30.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf4b7fe640 1 --2- 172.21.15.83:0/566350573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf4c06ddd0 0x7fcf4c1534f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c0c58d0 tx=0x7fcf4001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84468aecd57d8116 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:30.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf4affd640 1 -- 172.21.15.83:0/566350573 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf400b0070 con 0x7fcf4c06ddd0 2024-09-19T07:30:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf4affd640 1 -- 172.21.15.83:0/566350573 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf400a4e50 con 0x7fcf4c06ddd0 2024-09-19T07:30:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf4affd640 1 -- 172.21.15.83:0/566350573 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf400ab820 con 0x7fcf4c06ddd0 2024-09-19T07:30:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 -- 172.21.15.83:0/566350573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf4c06ddd0 msgr2=0x7fcf4c1534f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 --2- 172.21.15.83:0/566350573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf4c06ddd0 0x7fcf4c1534f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c0c58d0 tx=0x7fcf4001c400 comp rx=0 tx=0).stop 2024-09-19T07:30:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 -- 172.21.15.83:0/566350573 shutdown_connections 2024-09-19T07:30:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 --2- 172.21.15.83:0/566350573 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4c0c6460 0x7fcf4c14fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 --2- 172.21.15.83:0/566350573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf4c06ddd0 0x7fcf4c1534f0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 --2- 172.21.15.83:0/566350573 >> v2:172.21.15.184:3300/0 conn(0x7fcf4c06cef0 0x7fcf4c0c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 -- 172.21.15.83:0/566350573 >> 172.21.15.83:0/566350573 conn(0x7fcf4c0aa910 msgr2=0x7fcf4c0c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 -- 172.21.15.83:0/566350573 shutdown_connections 2024-09-19T07:30:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 -- 172.21.15.83:0/566350573 wait complete. 2024-09-19T07:30:30.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 Processor -- start 2024-09-19T07:30:30.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 -- start start 2024-09-19T07:30:30.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf4c06cef0 0x7fcf4c0aca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4c06ddd0 0x7fcf4c151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf4c0c6460 0x7fcf4c1523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c156c40 con 0x7fcf4c06cef0 2024-09-19T07:30:30.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c156ac0 con 0x7fcf4c0c6460 2024-09-19T07:30:30.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf52844640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcf4c156dc0 con 0x7fcf4c06ddd0 2024-09-19T07:30:30.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf4bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf4c06cef0 0x7fcf4c0aca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf4bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf4c06cef0 0x7fcf4c0aca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39738/0 (socket says 172.21.15.83:39738) 2024-09-19T07:30:30.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.961+0000 7fcf4bfff640 1 -- 172.21.15.83:0/1354962249 learned_addr learned my addr 172.21.15.83:0/1354962249 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:30.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf4b7fe640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4c06ddd0 0x7fcf4c151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf50de1640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf4c0c6460 0x7fcf4c1523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf4bfff640 1 -- 172.21.15.83:0/1354962249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4c06ddd0 msgr2=0x7fcf4c151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:30.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf4bfff640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4c06ddd0 0x7fcf4c151e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf4bfff640 1 -- 172.21.15.83:0/1354962249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf4c0c6460 msgr2=0x7fcf4c1523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:30.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf4bfff640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf4c0c6460 0x7fcf4c1523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:30.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf4bfff640 1 -- 172.21.15.83:0/1354962249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4c14ce50 con 0x7fcf4c06cef0 2024-09-19T07:30:30.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf50de1640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf4c0c6460 0x7fcf4c1523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:30:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf4bfff640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf4c06cef0 0x7fcf4c0aca90 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00e870 tx=0x7fcf3c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf48ff9640 1 -- 172.21.15.83:0/1354962249 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c017070 con 0x7fcf4c06cef0 2024-09-19T07:30:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf48ff9640 1 -- 172.21.15.83:0/1354962249 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf3c00ee60 con 0x7fcf4c06cef0 2024-09-19T07:30:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf52844640 1 -- 172.21.15.83:0/1354962249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf4c148bd0 con 0x7fcf4c06cef0 2024-09-19T07:30:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf48ff9640 1 -- 172.21.15.83:0/1354962249 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c00cb80 con 0x7fcf4c06cef0 2024-09-19T07:30:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf52844640 1 -- 172.21.15.83:0/1354962249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf4c0acfd0 con 0x7fcf4c06cef0 2024-09-19T07:30:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf52844640 1 -- 172.21.15.83:0/1354962249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf4c0c79d0 con 0x7fcf4c06cef0 2024-09-19T07:30:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf48ff9640 1 -- 172.21.15.83:0/1354962249 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcf3c01a070 con 0x7fcf4c06cef0 2024-09-19T07:30:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf48ff9640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf380678c0 0x7fcf38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf4b7fe640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf380678c0 0x7fcf38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf48ff9640 1 -- 172.21.15.83:0/1354962249 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7fcf3c086650 con 0x7fcf4c06cef0 2024-09-19T07:30:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.965+0000 7fcf4b7fe640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf380678c0 0x7fcf38069d80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcf400ad000 tx=0x7fcf40008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:30.969+0000 7fcf48ff9640 1 -- 172.21.15.83:0/1354962249 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcf3c056320 con 0x7fcf4c06cef0 2024-09-19T07:30:31.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.201+0000 7fcf52844640 1 -- 172.21.15.83:0/1354962249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf4c155590 con 0x7fcf4c06cef0 2024-09-19T07:30:31.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.201+0000 7fcf48ff9640 1 -- 172.21.15.83:0/1354962249 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7fcf3c05a620 con 0x7fcf4c06cef0 2024-09-19T07:30:31.199 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:31.199 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:31.200 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.205+0000 7fcf52844640 1 -- 172.21.15.83:0/1354962249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf380678c0 msgr2=0x7fcf38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.205+0000 7fcf52844640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf380678c0 0x7fcf38069d80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcf400ad000 tx=0x7fcf40008040 comp rx=0 tx=0).stop 2024-09-19T07:30:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.205+0000 7fcf52844640 1 -- 172.21.15.83:0/1354962249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf4c06cef0 msgr2=0x7fcf4c0aca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.205+0000 7fcf52844640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf4c06cef0 0x7fcf4c0aca90 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00e870 tx=0x7fcf3c00ed30 comp rx=0 tx=0).stop 2024-09-19T07:30:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.205+0000 7fcf52844640 1 -- 172.21.15.83:0/1354962249 shutdown_connections 2024-09-19T07:30:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.205+0000 7fcf52844640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf380678c0 0x7fcf38069d80 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.205+0000 7fcf52844640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf4c0c6460 0x7fcf4c1523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.205+0000 7fcf52844640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4c06ddd0 0x7fcf4c151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.205+0000 7fcf52844640 1 --2- 172.21.15.83:0/1354962249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf4c06cef0 0x7fcf4c0aca90 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.205+0000 7fcf52844640 1 -- 172.21.15.83:0/1354962249 >> 172.21.15.83:0/1354962249 conn(0x7fcf4c0aa910 msgr2=0x7fcf4c15cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.205+0000 7fcf52844640 1 -- 172.21.15.83:0/1354962249 shutdown_connections 2024-09-19T07:30:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:31.205+0000 7fcf52844640 1 -- 172.21.15.83:0/1354962249 wait complete. 2024-09-19T07:30:31.219 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:30:32.246 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.373+0000 7f013207a640 1 Processor -- start 2024-09-19T07:30:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.373+0000 7f013207a640 1 -- start start 2024-09-19T07:30:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.373+0000 7f013207a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f012c0b0d00 0x7f012c159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.373+0000 7f013207a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f012c0c9660 0x7f012c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.373+0000 7f013207a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f012c0c6180 con 0x7f012c156910 2024-09-19T07:30:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.373+0000 7f013207a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f012c0c69b0 con 0x7f012c0b0d00 2024-09-19T07:30:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.373+0000 7f013207a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f012c15a130 con 0x7f012c0c9660 2024-09-19T07:30:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.373+0000 7f012b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f012c0b0d00 0x7f012c159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f012b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f012c0b0d00 0x7f012c159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39752/0 (socket says 172.21.15.83:39752) 2024-09-19T07:30:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f012b7fe640 1 -- 172.21.15.83:0/1490240083 learned_addr learned my addr 172.21.15.83:0/1490240083 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f012affd640 1 --2- 172.21.15.83:0/1490240083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f012c0c9660 0x7f012c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:32.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f012b7fe640 1 -- 172.21.15.83:0/1490240083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f012c0c9660 msgr2=0x7f012c1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:32.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f012b7fe640 1 --2- 172.21.15.83:0/1490240083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f012c0c9660 0x7f012c1563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:32.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f012b7fe640 1 -- 172.21.15.83:0/1490240083 >> v1:172.21.15.83:6789/0 conn(0x7f012c156910 legacy=0x7f012c0c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:32.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f012b7fe640 1 -- 172.21.15.83:0/1490240083 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012c155bf0 con 0x7f012c0b0d00 2024-09-19T07:30:32.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f012b7fe640 1 --2- 172.21.15.83:0/1490240083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f012c0b0d00 0x7f012c159bf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0124009e50 tx=0x7f012401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46e5eede27ea530 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:32.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f012a7fc640 1 -- 172.21.15.83:0/1490240083 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01240a5660 con 0x7f012c0b0d00 2024-09-19T07:30:32.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f012a7fc640 1 -- 172.21.15.83:0/1490240083 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01240a5800 con 0x7f012c0b0d00 2024-09-19T07:30:32.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f012a7fc640 1 -- 172.21.15.83:0/1490240083 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01240a5bd0 con 0x7f012c0b0d00 2024-09-19T07:30:32.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f013207a640 1 -- 172.21.15.83:0/1490240083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f012c0b0d00 msgr2=0x7f012c159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:32.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f013207a640 1 --2- 172.21.15.83:0/1490240083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f012c0b0d00 0x7f012c159bf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0124009e50 tx=0x7f012401c9a0 comp rx=0 tx=0).stop 2024-09-19T07:30:32.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f013207a640 1 -- 172.21.15.83:0/1490240083 shutdown_connections 2024-09-19T07:30:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f013207a640 1 --2- 172.21.15.83:0/1490240083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f012c0c9660 0x7f012c1563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f013207a640 1 --2- 172.21.15.83:0/1490240083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f012c0b0d00 0x7f012c159bf0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f013207a640 1 -- 172.21.15.83:0/1490240083 >> 172.21.15.83:0/1490240083 conn(0x7f012c0722a0 msgr2=0x7f012c0726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f013207a640 1 -- 172.21.15.83:0/1490240083 shutdown_connections 2024-09-19T07:30:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.377+0000 7f013207a640 1 -- 172.21.15.83:0/1490240083 wait complete. 2024-09-19T07:30:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f013207a640 1 Processor -- start 2024-09-19T07:30:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f013207a640 1 -- start start 2024-09-19T07:30:32.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f013207a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f012c0b0d00 0x7f012c0c5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:32.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f013207a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f012c0c9660 0x7f012c154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:32.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f013207a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f012c0acb40 0x7f012c15eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:32.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f013207a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f012c0c8920 con 0x7f012c0c9660 2024-09-19T07:30:32.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f013207a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f012c0b10e0 con 0x7f012c0acb40 2024-09-19T07:30:32.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f013207a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f012c0c8aa0 con 0x7f012c0b0d00 2024-09-19T07:30:32.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f012affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f012c0c9660 0x7f012c154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:32.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f012affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f012c0c9660 0x7f012c154be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39768/0 (socket says 172.21.15.83:39768) 2024-09-19T07:30:32.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f012affd640 1 -- 172.21.15.83:0/3188355131 learned_addr learned my addr 172.21.15.83:0/3188355131 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:32.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f012affd640 1 -- 172.21.15.83:0/3188355131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f012c0b0d00 msgr2=0x7f012c0c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:32.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f012b7fe640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f012c0b0d00 0x7f012c0c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:32.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f012affd640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f012c0b0d00 0x7f012c0c5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:32.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f012affd640 1 -- 172.21.15.83:0/3188355131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f012c0acb40 msgr2=0x7f012c15eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:32.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f012bfff640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f012c0acb40 0x7f012c15eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:32.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f012affd640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f012c0acb40 0x7f012c15eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:32.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f012affd640 1 -- 172.21.15.83:0/3188355131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012c0ad110 con 0x7f012c0c9660 2024-09-19T07:30:32.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f012affd640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f012c0c9660 0x7f012c154be0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f011400d830 tx=0x7f011400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:32.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f0123fff640 1 -- 172.21.15.83:0/3188355131 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01140059a0 con 0x7f012c0c9660 2024-09-19T07:30:32.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f0123fff640 1 -- 172.21.15.83:0/3188355131 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0114005b40 con 0x7f012c0c9660 2024-09-19T07:30:32.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f013207a640 1 -- 172.21.15.83:0/3188355131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f012c0ad3f0 con 0x7f012c0c9660 2024-09-19T07:30:32.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f0123fff640 1 -- 172.21.15.83:0/3188355131 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0114005d30 con 0x7f012c0c9660 2024-09-19T07:30:32.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f013207a640 1 -- 172.21.15.83:0/3188355131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f012c0adc90 con 0x7f012c0c9660 2024-09-19T07:30:32.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.381+0000 7f013207a640 1 -- 172.21.15.83:0/3188355131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f012c159bf0 con 0x7f012c0c9660 2024-09-19T07:30:32.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.385+0000 7f0123fff640 1 -- 172.21.15.83:0/3188355131 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f011401b020 con 0x7f012c0c9660 2024-09-19T07:30:32.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.385+0000 7f0123fff640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f010c0678c0 0x7f010c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:32.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.385+0000 7f0123fff640 1 -- 172.21.15.83:0/3188355131 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f0114086540 con 0x7f012c0c9660 2024-09-19T07:30:32.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.385+0000 7f012b7fe640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f010c0678c0 0x7f010c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:32.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.385+0000 7f012b7fe640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f010c0678c0 0x7f010c069d80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f01240ae000 tx=0x7f01240a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:32.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.389+0000 7f0123fff640 1 -- 172.21.15.83:0/3188355131 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f0114056860 con 0x7f012c0c9660 2024-09-19T07:30:32.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.569+0000 7f013207a640 1 -- 172.21.15.83:0/3188355131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f012c1546a0 con 0x7f012c0c9660 2024-09-19T07:30:32.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.569+0000 7f0123fff640 1 -- 172.21.15.83:0/3188355131 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7f0114056160 con 0x7f012c0c9660 2024-09-19T07:30:32.566 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:32.566 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:32.566 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:32.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.573+0000 7f013207a640 1 -- 172.21.15.83:0/3188355131 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f010c0678c0 msgr2=0x7f010c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:32.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.573+0000 7f013207a640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f010c0678c0 0x7f010c069d80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f01240ae000 tx=0x7f01240a7070 comp rx=0 tx=0).stop 2024-09-19T07:30:32.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.573+0000 7f013207a640 1 -- 172.21.15.83:0/3188355131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f012c0c9660 msgr2=0x7f012c154be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:32.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.573+0000 7f013207a640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f012c0c9660 0x7f012c154be0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f011400d830 tx=0x7f011400dcf0 comp rx=0 tx=0).stop 2024-09-19T07:30:32.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.573+0000 7f013207a640 1 -- 172.21.15.83:0/3188355131 shutdown_connections 2024-09-19T07:30:32.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.573+0000 7f013207a640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f010c0678c0 0x7f010c069d80 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:32.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.573+0000 7f013207a640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f012c0acb40 0x7f012c15eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:32.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.573+0000 7f013207a640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f012c0c9660 0x7f012c154be0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:32.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.573+0000 7f013207a640 1 --2- 172.21.15.83:0/3188355131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f012c0b0d00 0x7f012c0c5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:32.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.573+0000 7f013207a640 1 -- 172.21.15.83:0/3188355131 >> 172.21.15.83:0/3188355131 conn(0x7f012c0722a0 msgr2=0x7f012c072680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:32.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.573+0000 7f013207a640 1 -- 172.21.15.83:0/3188355131 shutdown_connections 2024-09-19T07:30:32.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:32.573+0000 7f013207a640 1 -- 172.21.15.83:0/3188355131 wait complete. 2024-09-19T07:30:32.582 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:30:32.728 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:30:32.730 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:30:33.220 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:33.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f018999f640 1 Processor -- start 2024-09-19T07:30:33.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f018999f640 1 -- start start 2024-09-19T07:30:33.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f018999f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f0184155d70 0x7f01840c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:33.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f018999f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f01841592f0 0x7f0184157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:33.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f018999f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0184156200 con 0x7f01840c6120 2024-09-19T07:30:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f018999f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f01840c9930 con 0x7f01841592f0 2024-09-19T07:30:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f018999f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f01841564e0 con 0x7f0184155d70 2024-09-19T07:30:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f01837fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f01841592f0 0x7f0184157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f01837fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f01841592f0 0x7f0184157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39772/0 (socket says 172.21.15.83:39772) 2024-09-19T07:30:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f01837fe640 1 -- 172.21.15.83:0/1475126753 learned_addr learned my addr 172.21.15.83:0/1475126753 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f01827fc640 1 --2- 172.21.15.83:0/1475126753 >> v2:172.21.15.184:3300/0 conn(0x7f0184155d70 0x7f01840c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:33.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f01837fe640 1 -- 172.21.15.83:0/1475126753 >> v2:172.21.15.184:3300/0 conn(0x7f0184155d70 msgr2=0x7f01840c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:33.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f01837fe640 1 --2- 172.21.15.83:0/1475126753 >> v2:172.21.15.184:3300/0 conn(0x7f0184155d70 0x7f01840c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:33.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f01837fe640 1 -- 172.21.15.83:0/1475126753 >> v1:172.21.15.184:6789/0 conn(0x7f01840c6120 legacy=0x7f01841575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:33.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.353+0000 7f01837fe640 1 -- 172.21.15.83:0/1475126753 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01841567c0 con 0x7f01841592f0 2024-09-19T07:30:33.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f01837fe640 1 --2- 172.21.15.83:0/1475126753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01841592f0 0x7f0184157ce0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f018406cca0 tx=0x7f017c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb9c5df73f7f291 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:33.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f0181ffb640 1 -- 172.21.15.83:0/1475126753 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017c0a56c0 con 0x7f01841592f0 2024-09-19T07:30:33.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f0181ffb640 1 -- 172.21.15.83:0/1475126753 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f017c0a5860 con 0x7f01841592f0 2024-09-19T07:30:33.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f0181ffb640 1 -- 172.21.15.83:0/1475126753 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017c0a5c30 con 0x7f01841592f0 2024-09-19T07:30:33.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 -- 172.21.15.83:0/1475126753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01841592f0 msgr2=0x7f0184157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:33.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 --2- 172.21.15.83:0/1475126753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01841592f0 0x7f0184157ce0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f018406cca0 tx=0x7f017c01c9f0 comp rx=0 tx=0).stop 2024-09-19T07:30:33.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 -- 172.21.15.83:0/1475126753 shutdown_connections 2024-09-19T07:30:33.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 --2- 172.21.15.83:0/1475126753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01841592f0 0x7f0184157ce0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:33.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 --2- 172.21.15.83:0/1475126753 >> v2:172.21.15.184:3300/0 conn(0x7f0184155d70 0x7f01840c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:33.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 -- 172.21.15.83:0/1475126753 >> 172.21.15.83:0/1475126753 conn(0x7f018406bc80 msgr2=0x7f018406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:33.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 -- 172.21.15.83:0/1475126753 shutdown_connections 2024-09-19T07:30:33.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 -- 172.21.15.83:0/1475126753 wait complete. 2024-09-19T07:30:33.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 Processor -- start 2024-09-19T07:30:33.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 -- start start 2024-09-19T07:30:33.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01840c6120 0x7f0184146ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01841592f0 0x7f018414b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01841470e0 0x7f018414b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0184153860 con 0x7f01841592f0 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0184162ea0 con 0x7f01841470e0 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f018999f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f01840ae120 con 0x7f01840c6120 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f01827fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01841592f0 0x7f018414b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f01827fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01841592f0 0x7f018414b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39776/0 (socket says 172.21.15.83:39776) 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.357+0000 7f01827fc640 1 -- 172.21.15.83:0/1457347476 learned_addr learned my addr 172.21.15.83:0/1457347476 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f0182ffd640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01840c6120 0x7f0184146ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f01837fe640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01841470e0 0x7f018414b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f01827fc640 1 -- 172.21.15.83:0/1457347476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01840c6120 msgr2=0x7f0184146ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f01827fc640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01840c6120 0x7f0184146ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f01827fc640 1 -- 172.21.15.83:0/1457347476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01841470e0 msgr2=0x7f018414b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f01827fc640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01841470e0 0x7f018414b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f01827fc640 1 -- 172.21.15.83:0/1457347476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01840b1970 con 0x7f01841592f0 2024-09-19T07:30:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f01837fe640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01841470e0 0x7f018414b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f01827fc640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01841592f0 0x7f018414b2a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f017800e930 tx=0x7f017800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f016bfff640 1 -- 172.21.15.83:0/1457347476 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178017070 con 0x7f01841592f0 2024-09-19T07:30:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f016bfff640 1 -- 172.21.15.83:0/1457347476 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f017800ca00 con 0x7f01841592f0 2024-09-19T07:30:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f018999f640 1 -- 172.21.15.83:0/1457347476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018414fa70 con 0x7f01841592f0 2024-09-19T07:30:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f016bfff640 1 -- 172.21.15.83:0/1457347476 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017800cdb0 con 0x7f01841592f0 2024-09-19T07:30:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f018999f640 1 -- 172.21.15.83:0/1457347476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f018414e330 con 0x7f01841592f0 2024-09-19T07:30:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f018999f640 1 -- 172.21.15.83:0/1457347476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01840c6500 con 0x7f01841592f0 2024-09-19T07:30:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f016bfff640 1 -- 172.21.15.83:0/1457347476 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f017801a070 con 0x7f01841592f0 2024-09-19T07:30:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f016bfff640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01700678c0 0x7f0170069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f016bfff640 1 -- 172.21.15.83:0/1457347476 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f0178086910 con 0x7f01841592f0 2024-09-19T07:30:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f0182ffd640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01700678c0 0x7f0170069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.361+0000 7f0182ffd640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01700678c0 0x7f0170069d80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0174001650 tx=0x7f0174009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:33.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.365+0000 7f016bfff640 1 -- 172.21.15.83:0/1457347476 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f0178056c60 con 0x7f01841592f0 2024-09-19T07:30:33.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.557+0000 7f018999f640 1 -- 172.21.15.83:0/1457347476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01840c96a0 con 0x7f01841592f0 2024-09-19T07:30:33.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f016bfff640 1 -- 172.21.15.83:0/1457347476 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7f0178056560 con 0x7f01841592f0 2024-09-19T07:30:33.556 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:33.556 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:33.557 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:33.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f018999f640 1 -- 172.21.15.83:0/1457347476 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01700678c0 msgr2=0x7f0170069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:33.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f018999f640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01700678c0 0x7f0170069d80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0174001650 tx=0x7f0174009040 comp rx=0 tx=0).stop 2024-09-19T07:30:33.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f018999f640 1 -- 172.21.15.83:0/1457347476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01841592f0 msgr2=0x7f018414b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:33.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f018999f640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01841592f0 0x7f018414b2a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f017800e930 tx=0x7f017800edf0 comp rx=0 tx=0).stop 2024-09-19T07:30:33.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f018999f640 1 -- 172.21.15.83:0/1457347476 shutdown_connections 2024-09-19T07:30:33.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f018999f640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01700678c0 0x7f0170069d80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:33.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f018999f640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01841470e0 0x7f018414b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:33.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f018999f640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01841592f0 0x7f018414b2a0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:33.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f018999f640 1 --2- 172.21.15.83:0/1457347476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01840c6120 0x7f0184146ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:33.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f018999f640 1 -- 172.21.15.83:0/1457347476 >> 172.21.15.83:0/1457347476 conn(0x7f018406bc80 msgr2=0x7f0184155650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:33.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f018999f640 1 -- 172.21.15.83:0/1457347476 shutdown_connections 2024-09-19T07:30:33.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:33.561+0000 7f018999f640 1 -- 172.21.15.83:0/1457347476 wait complete. 2024-09-19T07:30:33.573 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:30:34.583 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:34.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e92bb640 1 Processor -- start 2024-09-19T07:30:34.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e92bb640 1 -- start start 2024-09-19T07:30:34.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e92bb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f60e4150290 0x7f60e40c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:34.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e92bb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60e4153eb0 0x7f60e4154340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:34.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e92bb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60e4150670 con 0x7f60e4152b20 2024-09-19T07:30:34.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e92bb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f60e40716c0 con 0x7f60e4153eb0 2024-09-19T07:30:34.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e92bb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f60e40c3920 con 0x7f60e4150290 2024-09-19T07:30:34.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60e4153eb0 0x7f60e4154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:34.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60e4153eb0 0x7f60e4154340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39782/0 (socket says 172.21.15.83:39782) 2024-09-19T07:30:34.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e37fe640 1 -- 172.21.15.83:0/1731266684 learned_addr learned my addr 172.21.15.83:0/1731266684 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:34.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e37fe640 1 -- 172.21.15.83:0/1731266684 >> v2:172.21.15.184:3300/0 conn(0x7f60e4150290 msgr2=0x7f60e40c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:34.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e37fe640 1 --2- 172.21.15.83:0/1731266684 >> v2:172.21.15.184:3300/0 conn(0x7f60e4150290 0x7f60e40c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:34.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e37fe640 1 -- 172.21.15.83:0/1731266684 >> v1:172.21.15.184:6789/0 conn(0x7f60e4152b20 legacy=0x7f60e40c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:34.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e37fe640 1 -- 172.21.15.83:0/1731266684 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e40c2880 con 0x7f60e4153eb0 2024-09-19T07:30:34.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e37fe640 1 --2- 172.21.15.83:0/1731266684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e4153eb0 0x7f60e4154340 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f60e406f590 tx=0x7f60dc01c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4e00a70ce50961 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:34.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e27fc640 1 -- 172.21.15.83:0/1731266684 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc0a5760 con 0x7f60e4153eb0 2024-09-19T07:30:34.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e27fc640 1 -- 172.21.15.83:0/1731266684 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60dc0a5900 con 0x7f60e4153eb0 2024-09-19T07:30:34.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.717+0000 7f60e27fc640 1 -- 172.21.15.83:0/1731266684 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc0a5cd0 con 0x7f60e4153eb0 2024-09-19T07:30:34.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 -- 172.21.15.83:0/1731266684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e4153eb0 msgr2=0x7f60e4154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:34.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 --2- 172.21.15.83:0/1731266684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e4153eb0 0x7f60e4154340 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f60e406f590 tx=0x7f60dc01c9b0 comp rx=0 tx=0).stop 2024-09-19T07:30:34.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 -- 172.21.15.83:0/1731266684 shutdown_connections 2024-09-19T07:30:34.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 --2- 172.21.15.83:0/1731266684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e4153eb0 0x7f60e4154340 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:34.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 --2- 172.21.15.83:0/1731266684 >> v2:172.21.15.184:3300/0 conn(0x7f60e4150290 0x7f60e40c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:34.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 -- 172.21.15.83:0/1731266684 >> 172.21.15.83:0/1731266684 conn(0x7f60e406bc80 msgr2=0x7f60e406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:34.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 -- 172.21.15.83:0/1731266684 shutdown_connections 2024-09-19T07:30:34.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 -- 172.21.15.83:0/1731266684 wait complete. 2024-09-19T07:30:34.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 Processor -- start 2024-09-19T07:30:34.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 -- start start 2024-09-19T07:30:34.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e4150290 0x7f60e41512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:34.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e4152b20 0x7f60e414d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:34.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e4153eb0 0x7f60e40c0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:34.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f60e40c4790 con 0x7f60e4152b20 2024-09-19T07:30:34.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f60e40c2530 con 0x7f60e4153eb0 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f60e40c4910 con 0x7f60e4150290 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60da7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e4152b20 0x7f60e414d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60da7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e4152b20 0x7f60e414d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39788/0 (socket says 172.21.15.83:39788) 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60da7fc640 1 -- 172.21.15.83:0/505231928 learned_addr learned my addr 172.21.15.83:0/505231928 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e2ffd640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e4150290 0x7f60e41512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e37fe640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e4153eb0 0x7f60e40c0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60da7fc640 1 -- 172.21.15.83:0/505231928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e4150290 msgr2=0x7f60e41512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60da7fc640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e4150290 0x7f60e41512f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60da7fc640 1 -- 172.21.15.83:0/505231928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e4153eb0 msgr2=0x7f60e40c0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60da7fc640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e4153eb0 0x7f60e40c0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60da7fc640 1 -- 172.21.15.83:0/505231928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e4152400 con 0x7f60e4152b20 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e2ffd640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e4150290 0x7f60e41512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:34.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60da7fc640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e4152b20 0x7f60e414d170 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00c4c0 tx=0x7f60cc001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:34.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60dbfff640 1 -- 172.21.15.83:0/505231928 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc017070 con 0x7f60e4152b20 2024-09-19T07:30:34.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60dbfff640 1 -- 172.21.15.83:0/505231928 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60cc003600 con 0x7f60e4152b20 2024-09-19T07:30:34.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 -- 172.21.15.83:0/505231928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e41526e0 con 0x7f60e4152b20 2024-09-19T07:30:34.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60dbfff640 1 -- 172.21.15.83:0/505231928 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0039f0 con 0x7f60e4152b20 2024-09-19T07:30:34.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.721+0000 7f60e92bb640 1 -- 172.21.15.83:0/505231928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e40c11f0 con 0x7f60e4152b20 2024-09-19T07:30:34.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.725+0000 7f60e92bb640 1 -- 172.21.15.83:0/505231928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e40c1ff0 con 0x7f60e4152b20 2024-09-19T07:30:34.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.725+0000 7f60dbfff640 1 -- 172.21.15.83:0/505231928 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f60cc01a070 con 0x7f60e4152b20 2024-09-19T07:30:34.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.725+0000 7f60dbfff640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60c40678c0 0x7f60c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:34.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.725+0000 7f60dbfff640 1 -- 172.21.15.83:0/505231928 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f60cc085840 con 0x7f60e4152b20 2024-09-19T07:30:34.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.725+0000 7f60e2ffd640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60c40678c0 0x7f60c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:34.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.725+0000 7f60e2ffd640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60c40678c0 0x7f60c4069d80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f60d400a8b0 tx=0x7f60d4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:34.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.729+0000 7f60dbfff640 1 -- 172.21.15.83:0/505231928 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f60cc055b60 con 0x7f60e4152b20 2024-09-19T07:30:34.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.925+0000 7f60e92bb640 1 -- 172.21.15.83:0/505231928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60e41528b0 con 0x7f60e4152b20 2024-09-19T07:30:34.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.925+0000 7f60dbfff640 1 -- 172.21.15.83:0/505231928 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7f60cc055460 con 0x7f60e4152b20 2024-09-19T07:30:34.923 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:34.923 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:34.923 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:34.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.929+0000 7f60e92bb640 1 -- 172.21.15.83:0/505231928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60c40678c0 msgr2=0x7f60c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:34.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.929+0000 7f60e92bb640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60c40678c0 0x7f60c4069d80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f60d400a8b0 tx=0x7f60d4008040 comp rx=0 tx=0).stop 2024-09-19T07:30:34.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.929+0000 7f60e92bb640 1 -- 172.21.15.83:0/505231928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e4152b20 msgr2=0x7f60e414d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:34.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.929+0000 7f60e92bb640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e4152b20 0x7f60e414d170 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00c4c0 tx=0x7f60cc001e80 comp rx=0 tx=0).stop 2024-09-19T07:30:34.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.929+0000 7f60e92bb640 1 -- 172.21.15.83:0/505231928 shutdown_connections 2024-09-19T07:30:34.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.929+0000 7f60e92bb640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f60c40678c0 0x7f60c4069d80 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:34.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.929+0000 7f60e92bb640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60e4153eb0 0x7f60e40c0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:34.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.929+0000 7f60e92bb640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e4152b20 0x7f60e414d170 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:34.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.929+0000 7f60e92bb640 1 --2- 172.21.15.83:0/505231928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f60e4150290 0x7f60e41512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:34.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.929+0000 7f60e92bb640 1 -- 172.21.15.83:0/505231928 >> 172.21.15.83:0/505231928 conn(0x7f60e406bc80 msgr2=0x7f60e406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:34.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.929+0000 7f60e92bb640 1 -- 172.21.15.83:0/505231928 shutdown_connections 2024-09-19T07:30:34.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:34.929+0000 7f60e92bb640 1 -- 172.21.15.83:0/505231928 wait complete. 2024-09-19T07:30:34.941 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:30:35.574 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ed20d1640 1 Processor -- start 2024-09-19T07:30:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ed20d1640 1 -- start start 2024-09-19T07:30:35.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ed20d1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2ecc0c28e0 0x7f2ecc1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:35.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ed20d1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ecc0a9e60 con 0x7f2ecc071100 2024-09-19T07:30:35.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ed20d1640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ecc0aa140 con 0x7f2ecc06dea0 2024-09-19T07:30:35.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ed20d1640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f2ecc071b50 con 0x7f2ecc0c28e0 2024-09-19T07:30:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ecaffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2ecc0c28e0 0x7f2ecc1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ecaffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2ecc0c28e0 0x7f2ecc1545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39804/0 (socket says 172.21.15.83:39804) 2024-09-19T07:30:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ecaffd640 1 -- 172.21.15.83:0/1108901900 learned_addr learned my addr 172.21.15.83:0/1108901900 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ecb7fe640 1 --1- 172.21.15.83:0/1108901900 >> v1:172.21.15.83:6789/0 conn(0x7f2ecc071100 0x7f2ecc150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45150/0 (socket says 172.21.15.83:45150) 2024-09-19T07:30:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ecaffd640 1 -- 172.21.15.83:0/1108901900 >> v1:172.21.15.184:6789/0 conn(0x7f2ecc06dea0 legacy=0x7f2ecc06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ecaffd640 1 -- 172.21.15.83:0/1108901900 >> v1:172.21.15.83:6789/0 conn(0x7f2ecc071100 legacy=0x7f2ecc150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.709+0000 7f2ecaffd640 1 -- 172.21.15.83:0/1108901900 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ecc0c2240 con 0x7f2ecc0c28e0 2024-09-19T07:30:35.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.713+0000 7f2ecaffd640 1 --2- 172.21.15.83:0/1108901900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ecc0c28e0 0x7f2ecc1545e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc009870 tx=0x7f2ebc01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1868861a8b11a90 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:35.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.713+0000 7f2eca7fc640 1 -- 172.21.15.83:0/1108901900 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc0b0070 con 0x7f2ecc0c28e0 2024-09-19T07:30:35.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.713+0000 7f2eca7fc640 1 -- 172.21.15.83:0/1108901900 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ebc0ab600 con 0x7f2ecc0c28e0 2024-09-19T07:30:35.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.713+0000 7f2eca7fc640 1 -- 172.21.15.83:0/1108901900 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc0ab9d0 con 0x7f2ecc0c28e0 2024-09-19T07:30:35.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.713+0000 7f2ed20d1640 1 -- 172.21.15.83:0/1108901900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ecc0c28e0 msgr2=0x7f2ecc1545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.713+0000 7f2ed20d1640 1 --2- 172.21.15.83:0/1108901900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ecc0c28e0 0x7f2ecc1545e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc009870 tx=0x7f2ebc01c490 comp rx=0 tx=0).stop 2024-09-19T07:30:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.713+0000 7f2ed20d1640 1 -- 172.21.15.83:0/1108901900 shutdown_connections 2024-09-19T07:30:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.713+0000 7f2ed20d1640 1 --2- 172.21.15.83:0/1108901900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ecc0c28e0 0x7f2ecc1545e0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.713+0000 7f2ed20d1640 1 -- 172.21.15.83:0/1108901900 >> 172.21.15.83:0/1108901900 conn(0x7f2ecc06bc80 msgr2=0x7f2ecc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.713+0000 7f2ed20d1640 1 -- 172.21.15.83:0/1108901900 shutdown_connections 2024-09-19T07:30:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.713+0000 7f2ed20d1640 1 -- 172.21.15.83:0/1108901900 wait complete. 2024-09-19T07:30:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ed20d1640 1 Processor -- start 2024-09-19T07:30:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ed20d1640 1 -- start start 2024-09-19T07:30:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ed20d1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ecc06dea0 0x7f2ecc147de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ed20d1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ecc071100 0x7f2ecc0c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ed20d1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ecc0c28e0 0x7f2ecc1ef810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ed20d1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2ecc071660 con 0x7f2ecc071100 2024-09-19T07:30:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ed20d1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2ecc0714e0 con 0x7f2ecc0c28e0 2024-09-19T07:30:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ed20d1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2ecc161c70 con 0x7f2ecc06dea0 2024-09-19T07:30:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ecc071100 0x7f2ecc0c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ecc071100 0x7f2ecc0c63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39818/0 (socket says 172.21.15.83:39818) 2024-09-19T07:30:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecaffd640 1 -- 172.21.15.83:0/2406311212 learned_addr learned my addr 172.21.15.83:0/2406311212 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecb7fe640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ecc06dea0 0x7f2ecc147de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecbfff640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ecc0c28e0 0x7f2ecc1ef810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecaffd640 1 -- 172.21.15.83:0/2406311212 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ecc06dea0 msgr2=0x7f2ecc147de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecaffd640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ecc06dea0 0x7f2ecc147de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecaffd640 1 -- 172.21.15.83:0/2406311212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ecc0c28e0 msgr2=0x7f2ecc1ef810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecaffd640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ecc0c28e0 0x7f2ecc1ef810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecaffd640 1 -- 172.21.15.83:0/2406311212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ecc0c7110 con 0x7f2ecc071100 2024-09-19T07:30:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecb7fe640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ecc06dea0 0x7f2ecc147de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecaffd640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ecc071100 0x7f2ecc0c63c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0ae040 tx=0x7f2ebc019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2eb3fff640 1 -- 172.21.15.83:0/2406311212 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc0b0040 con 0x7f2ecc071100 2024-09-19T07:30:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2eb3fff640 1 -- 172.21.15.83:0/2406311212 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ebc0b9070 con 0x7f2ecc071100 2024-09-19T07:30:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ed20d1640 1 -- 172.21.15.83:0/2406311212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ecc145c30 con 0x7f2ecc071100 2024-09-19T07:30:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2eb3fff640 1 -- 172.21.15.83:0/2406311212 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc0b4680 con 0x7f2ecc071100 2024-09-19T07:30:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ed20d1640 1 -- 172.21.15.83:0/2406311212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ecc1461f0 con 0x7f2ecc071100 2024-09-19T07:30:35.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ed20d1640 1 -- 172.21.15.83:0/2406311212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ecc1545e0 con 0x7f2ecc071100 2024-09-19T07:30:35.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2eb3fff640 1 -- 172.21.15.83:0/2406311212 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2ebc0b4820 con 0x7f2ecc071100 2024-09-19T07:30:35.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2eb3fff640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2eb80678f0 0x7f2eb8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:35.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2eb3fff640 1 -- 172.21.15.83:0/2406311212 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f2ebc1202c0 con 0x7f2ecc071100 2024-09-19T07:30:35.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.717+0000 7f2ecb7fe640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2eb80678f0 0x7f2eb8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:35.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.721+0000 7f2ecb7fe640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2eb80678f0 0x7f2eb8069db0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0007fd0 tx=0x7f2ec0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.725+0000 7f2eb3fff640 1 -- 172.21.15.83:0/2406311212 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2ebc016110 con 0x7f2ecc071100 2024-09-19T07:30:35.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.921+0000 7f2ed20d1640 1 -- 172.21.15.83:0/2406311212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ecc0c4f30 con 0x7f2ecc071100 2024-09-19T07:30:35.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.921+0000 7f2eb3fff640 1 -- 172.21.15.83:0/2406311212 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7f2ebc0f04c0 con 0x7f2ecc071100 2024-09-19T07:30:35.917 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:35.917 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:35.917 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:35.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.921+0000 7f2ed20d1640 1 -- 172.21.15.83:0/2406311212 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2eb80678f0 msgr2=0x7f2eb8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:35.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.921+0000 7f2ed20d1640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2eb80678f0 0x7f2eb8069db0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0007fd0 tx=0x7f2ec0009040 comp rx=0 tx=0).stop 2024-09-19T07:30:35.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.921+0000 7f2ed20d1640 1 -- 172.21.15.83:0/2406311212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ecc071100 msgr2=0x7f2ecc0c63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:35.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.921+0000 7f2ed20d1640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ecc071100 0x7f2ecc0c63c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0ae040 tx=0x7f2ebc019230 comp rx=0 tx=0).stop 2024-09-19T07:30:35.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.921+0000 7f2ed20d1640 1 -- 172.21.15.83:0/2406311212 shutdown_connections 2024-09-19T07:30:35.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.921+0000 7f2ed20d1640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2eb80678f0 0x7f2eb8069db0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:35.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.921+0000 7f2ed20d1640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ecc0c28e0 0x7f2ecc1ef810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:35.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.921+0000 7f2ed20d1640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ecc071100 0x7f2ecc0c63c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:35.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.921+0000 7f2ed20d1640 1 --2- 172.21.15.83:0/2406311212 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ecc06dea0 0x7f2ecc147de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:35.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.921+0000 7f2ed20d1640 1 -- 172.21.15.83:0/2406311212 >> 172.21.15.83:0/2406311212 conn(0x7f2ecc06bc80 msgr2=0x7f2ecc06c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:35.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.925+0000 7f2ed20d1640 1 -- 172.21.15.83:0/2406311212 shutdown_connections 2024-09-19T07:30:35.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:35.925+0000 7f2ed20d1640 1 -- 172.21.15.83:0/2406311212 wait complete. 2024-09-19T07:30:35.935 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:30:36.942 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:37.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.073+0000 7f379da22640 1 Processor -- start 2024-09-19T07:30:37.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f379da22640 1 -- start start 2024-09-19T07:30:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f379da22640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f37981592f0 0x7f37980c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f379da22640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3798156200 con 0x7f37980c6120 2024-09-19T07:30:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f379da22640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37981564e0 con 0x7f3798155d70 2024-09-19T07:30:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f379da22640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f37980c9930 con 0x7f37981592f0 2024-09-19T07:30:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f37977fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f37981592f0 0x7f37980c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f37977fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f37981592f0 0x7f37980c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39822/0 (socket says 172.21.15.83:39822) 2024-09-19T07:30:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f37977fe640 1 -- 172.21.15.83:0/2902536207 learned_addr learned my addr 172.21.15.83:0/2902536207 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f3796ffd640 1 --1- 172.21.15.83:0/2902536207 >> v1:172.21.15.83:6789/0 conn(0x7f37980c6120 0x7f37981575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45166/0 (socket says 172.21.15.83:45166) 2024-09-19T07:30:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f37977fe640 1 -- 172.21.15.83:0/2902536207 >> v1:172.21.15.184:6789/0 conn(0x7f3798155d70 legacy=0x7f3798157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f37977fe640 1 -- 172.21.15.83:0/2902536207 >> v1:172.21.15.83:6789/0 conn(0x7f37980c6120 legacy=0x7f37981575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:37.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f37977fe640 1 -- 172.21.15.83:0/2902536207 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37981567c0 con 0x7f37981592f0 2024-09-19T07:30:37.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.077+0000 7f37977fe640 1 --2- 172.21.15.83:0/2902536207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37981592f0 0x7f37980c6500 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3790009870 tx=0x7f379001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a857c62795722a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:37.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f3795ffb640 1 -- 172.21.15.83:0/2902536207 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37900b0070 con 0x7f37981592f0 2024-09-19T07:30:37.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f3795ffb640 1 -- 172.21.15.83:0/2902536207 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37900ab600 con 0x7f37981592f0 2024-09-19T07:30:37.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f3795ffb640 1 -- 172.21.15.83:0/2902536207 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37900ab9d0 con 0x7f37981592f0 2024-09-19T07:30:37.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 -- 172.21.15.83:0/2902536207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37981592f0 msgr2=0x7f37980c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:37.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 --2- 172.21.15.83:0/2902536207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37981592f0 0x7f37980c6500 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3790009870 tx=0x7f379001c490 comp rx=0 tx=0).stop 2024-09-19T07:30:37.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 -- 172.21.15.83:0/2902536207 shutdown_connections 2024-09-19T07:30:37.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 --2- 172.21.15.83:0/2902536207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37981592f0 0x7f37980c6500 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:37.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 -- 172.21.15.83:0/2902536207 >> 172.21.15.83:0/2902536207 conn(0x7f379806bc80 msgr2=0x7f379806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:37.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 -- 172.21.15.83:0/2902536207 shutdown_connections 2024-09-19T07:30:37.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 -- 172.21.15.83:0/2902536207 wait complete. 2024-09-19T07:30:37.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 Processor -- start 2024-09-19T07:30:37.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 -- start start 2024-09-19T07:30:37.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37980c6120 0x7f3798154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:37.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37981592f0 0x7f37980c1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:37.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37980c1590 0x7f37981490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:37.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3798164030 con 0x7f37981592f0 2024-09-19T07:30:37.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3798163eb0 con 0x7f37980c1590 2024-09-19T07:30:37.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f37981641b0 con 0x7f37980c6120 2024-09-19T07:30:37.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f37967fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37981592f0 0x7f37980c1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:37.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f37967fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37981592f0 0x7f37980c1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39836/0 (socket says 172.21.15.83:39836) 2024-09-19T07:30:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f37967fc640 1 -- 172.21.15.83:0/1778421585 learned_addr learned my addr 172.21.15.83:0/1778421585 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f37977fe640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37980c1590 0x7f37981490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f3796ffd640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37980c6120 0x7f3798154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f37967fc640 1 -- 172.21.15.83:0/1778421585 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37980c6120 msgr2=0x7f3798154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f37967fc640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37980c6120 0x7f3798154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f37967fc640 1 -- 172.21.15.83:0/1778421585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37980c1590 msgr2=0x7f37981490a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f37967fc640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37980c1590 0x7f37981490a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f37967fc640 1 -- 172.21.15.83:0/1778421585 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37980c1d30 con 0x7f37981592f0 2024-09-19T07:30:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f37967fc640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37981592f0 0x7f37980c1050 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f378800c4c0 tx=0x7f3788001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f378ffff640 1 -- 172.21.15.83:0/1778421585 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788017070 con 0x7f37981592f0 2024-09-19T07:30:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f378ffff640 1 -- 172.21.15.83:0/1778421585 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3788003600 con 0x7f37981592f0 2024-09-19T07:30:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 -- 172.21.15.83:0/1778421585 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37980c2010 con 0x7f37981592f0 2024-09-19T07:30:37.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f378ffff640 1 -- 172.21.15.83:0/1778421585 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37880039f0 con 0x7f37981592f0 2024-09-19T07:30:37.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.081+0000 7f379da22640 1 -- 172.21.15.83:0/1778421585 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37981eff30 con 0x7f37981592f0 2024-09-19T07:30:37.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.085+0000 7f379da22640 1 -- 172.21.15.83:0/1778421585 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37980c2530 con 0x7f37981592f0 2024-09-19T07:30:37.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.085+0000 7f378ffff640 1 -- 172.21.15.83:0/1778421585 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f378801a070 con 0x7f37981592f0 2024-09-19T07:30:37.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.085+0000 7f378ffff640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37780678f0 0x7f3778069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:37.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.085+0000 7f378ffff640 1 -- 172.21.15.83:0/1778421585 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f37880857f0 con 0x7f37981592f0 2024-09-19T07:30:37.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.085+0000 7f3796ffd640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37780678f0 0x7f3778069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:37.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.085+0000 7f3796ffd640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37780678f0 0x7f3778069db0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3780007fd0 tx=0x7f3780005b80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:37.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.089+0000 7f378ffff640 1 -- 172.21.15.83:0/1778421585 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f37980c2530 con 0x7f37981592f0 2024-09-19T07:30:37.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.285+0000 7f379da22640 1 -- 172.21.15.83:0/1778421585 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37980c6500 con 0x7f37981592f0 2024-09-19T07:30:37.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.285+0000 7f378ffff640 1 -- 172.21.15.83:0/1778421585 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7f3788055a80 con 0x7f37981592f0 2024-09-19T07:30:37.283 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:37.283 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:37.283 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:37.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.289+0000 7f379da22640 1 -- 172.21.15.83:0/1778421585 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37780678f0 msgr2=0x7f3778069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:37.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.289+0000 7f379da22640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37780678f0 0x7f3778069db0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3780007fd0 tx=0x7f3780005b80 comp rx=0 tx=0).stop 2024-09-19T07:30:37.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.289+0000 7f379da22640 1 -- 172.21.15.83:0/1778421585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37981592f0 msgr2=0x7f37980c1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:37.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.289+0000 7f379da22640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37981592f0 0x7f37980c1050 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f378800c4c0 tx=0x7f3788001e80 comp rx=0 tx=0).stop 2024-09-19T07:30:37.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.289+0000 7f379da22640 1 -- 172.21.15.83:0/1778421585 shutdown_connections 2024-09-19T07:30:37.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.289+0000 7f379da22640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37780678f0 0x7f3778069db0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:37.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.289+0000 7f379da22640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37980c1590 0x7f37981490a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:37.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.289+0000 7f379da22640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37981592f0 0x7f37980c1050 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:37.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.289+0000 7f379da22640 1 --2- 172.21.15.83:0/1778421585 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37980c6120 0x7f3798154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:37.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.289+0000 7f379da22640 1 -- 172.21.15.83:0/1778421585 >> 172.21.15.83:0/1778421585 conn(0x7f379806bc80 msgr2=0x7f3798152630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:37.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.289+0000 7f379da22640 1 -- 172.21.15.83:0/1778421585 shutdown_connections 2024-09-19T07:30:37.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:37.289+0000 7f379da22640 1 -- 172.21.15.83:0/1778421585 wait complete. 2024-09-19T07:30:37.302 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:30:37.936 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.069+0000 7f522cab0640 1 Processor -- start 2024-09-19T07:30:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.069+0000 7f522cab0640 1 -- start start 2024-09-19T07:30:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f522806cef0 0x7f52280c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5228070140 0x7f52281579a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5228070830 con 0x7f5228154180 2024-09-19T07:30:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5228153cf0 con 0x7f5228070140 2024-09-19T07:30:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f52281539a0 con 0x7f522806cef0 2024-09-19T07:30:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f5225d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5228070140 0x7f52281579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f5225d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5228070140 0x7f52281579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39848/0 (socket says 172.21.15.83:39848) 2024-09-19T07:30:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f5225d9b640 1 -- 172.21.15.83:0/3273818971 learned_addr learned my addr 172.21.15.83:0/3273818971 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522659c640 1 --2- 172.21.15.83:0/3273818971 >> v2:172.21.15.184:3300/0 conn(0x7f522806cef0 0x7f52280c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f5225d9b640 1 -- 172.21.15.83:0/3273818971 >> v2:172.21.15.184:3300/0 conn(0x7f522806cef0 msgr2=0x7f52280c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f5225d9b640 1 --2- 172.21.15.83:0/3273818971 >> v2:172.21.15.184:3300/0 conn(0x7f522806cef0 0x7f52280c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f5225d9b640 1 -- 172.21.15.83:0/3273818971 >> v1:172.21.15.184:6789/0 conn(0x7f5228154180 legacy=0x7f5228150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f5225d9b640 1 -- 172.21.15.83:0/3273818971 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522806f2e0 con 0x7f5228070140 2024-09-19T07:30:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522659c640 1 --2- 172.21.15.83:0/3273818971 >> v2:172.21.15.184:3300/0 conn(0x7f522806cef0 0x7f52280c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f5225d9b640 1 --2- 172.21.15.83:0/3273818971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5228070140 0x7f52281579a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f52280c6060 tx=0x7f522001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af351a05d1ad58a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522559a640 1 -- 172.21.15.83:0/3273818971 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52200b0070 con 0x7f5228070140 2024-09-19T07:30:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522559a640 1 -- 172.21.15.83:0/3273818971 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52200ab600 con 0x7f5228070140 2024-09-19T07:30:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522559a640 1 -- 172.21.15.83:0/3273818971 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52200ab9d0 con 0x7f5228070140 2024-09-19T07:30:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 -- 172.21.15.83:0/3273818971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5228070140 msgr2=0x7f52281579a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 --2- 172.21.15.83:0/3273818971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5228070140 0x7f52281579a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f52280c6060 tx=0x7f522001c3c0 comp rx=0 tx=0).stop 2024-09-19T07:30:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 -- 172.21.15.83:0/3273818971 shutdown_connections 2024-09-19T07:30:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 --2- 172.21.15.83:0/3273818971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5228070140 0x7f52281579a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 --2- 172.21.15.83:0/3273818971 >> v2:172.21.15.184:3300/0 conn(0x7f522806cef0 0x7f52280c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 -- 172.21.15.83:0/3273818971 >> 172.21.15.83:0/3273818971 conn(0x7f52280c5090 msgr2=0x7f52280c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 -- 172.21.15.83:0/3273818971 shutdown_connections 2024-09-19T07:30:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.073+0000 7f522cab0640 1 -- 172.21.15.83:0/3273818971 wait complete. 2024-09-19T07:30:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522cab0640 1 Processor -- start 2024-09-19T07:30:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522cab0640 1 -- start start 2024-09-19T07:30:38.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522cab0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f522806cef0 0x7f522814f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:38.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522cab0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5228070140 0x7f5228148b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:38.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522cab0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5228154180 0x7f52281490d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:38.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522cab0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f522806d450 con 0x7f522806cef0 2024-09-19T07:30:38.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522cab0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f522806d2d0 con 0x7f5228070140 2024-09-19T07:30:38.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522cab0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5228160b40 con 0x7f5228154180 2024-09-19T07:30:38.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f522806cef0 0x7f522814f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:38.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f522806cef0 0x7f522814f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39852/0 (socket says 172.21.15.83:39852) 2024-09-19T07:30:38.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522659c640 1 -- 172.21.15.83:0/2890136257 learned_addr learned my addr 172.21.15.83:0/2890136257 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:38.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f5226d9d640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5228154180 0x7f52281490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:38.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522659c640 1 -- 172.21.15.83:0/2890136257 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5228154180 msgr2=0x7f52281490d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:38.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522659c640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5228154180 0x7f52281490d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:38.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522659c640 1 -- 172.21.15.83:0/2890136257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5228070140 msgr2=0x7f5228148b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:38.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f5225d9b640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5228070140 0x7f5228148b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:38.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522659c640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5228070140 0x7f5228148b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:38.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522659c640 1 -- 172.21.15.83:0/2890136257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52280c6b80 con 0x7f522806cef0 2024-09-19T07:30:38.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f5225d9b640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5228070140 0x7f5228148b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:38.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522659c640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f522806cef0 0x7f522814f400 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f521800cf60 tx=0x7f521800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:38.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f521effd640 1 -- 172.21.15.83:0/2890136257 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521801a070 con 0x7f522806cef0 2024-09-19T07:30:38.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f521effd640 1 -- 172.21.15.83:0/2890136257 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f521800d840 con 0x7f522806cef0 2024-09-19T07:30:38.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522cab0640 1 -- 172.21.15.83:0/2890136257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52280c63d0 con 0x7f522806cef0 2024-09-19T07:30:38.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f521effd640 1 -- 172.21.15.83:0/2890136257 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521800dc10 con 0x7f522806cef0 2024-09-19T07:30:38.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.077+0000 7f522cab0640 1 -- 172.21.15.83:0/2890136257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52280ad7b0 con 0x7f522806cef0 2024-09-19T07:30:38.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.081+0000 7f522cab0640 1 -- 172.21.15.83:0/2890136257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52280c2f90 con 0x7f522806cef0 2024-09-19T07:30:38.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.081+0000 7f521effd640 1 -- 172.21.15.83:0/2890136257 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f521801d070 con 0x7f522806cef0 2024-09-19T07:30:38.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.081+0000 7f521effd640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52080678c0 0x7f5208069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:38.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.081+0000 7f521effd640 1 -- 172.21.15.83:0/2890136257 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f5218086cf0 con 0x7f522806cef0 2024-09-19T07:30:38.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.081+0000 7f5225d9b640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52080678c0 0x7f5208069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:38.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.081+0000 7f5225d9b640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52080678c0 0x7f5208069d80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f52200ad000 tx=0x7f5220008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:38.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.085+0000 7f521effd640 1 -- 172.21.15.83:0/2890136257 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f521808b070 con 0x7f522806cef0 2024-09-19T07:30:38.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.273+0000 7f522cab0640 1 -- 172.21.15.83:0/2890136257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52280c4600 con 0x7f522806cef0 2024-09-19T07:30:38.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.277+0000 7f521effd640 1 -- 172.21.15.83:0/2890136257 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7f52180569c0 con 0x7f522806cef0 2024-09-19T07:30:38.272 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:38.272 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:38.272 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.281+0000 7f522cab0640 1 -- 172.21.15.83:0/2890136257 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52080678c0 msgr2=0x7f5208069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.281+0000 7f522cab0640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52080678c0 0x7f5208069d80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f52200ad000 tx=0x7f5220008040 comp rx=0 tx=0).stop 2024-09-19T07:30:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.281+0000 7f522cab0640 1 -- 172.21.15.83:0/2890136257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f522806cef0 msgr2=0x7f522814f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.281+0000 7f522cab0640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f522806cef0 0x7f522814f400 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f521800cf60 tx=0x7f521800cf90 comp rx=0 tx=0).stop 2024-09-19T07:30:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.281+0000 7f522cab0640 1 -- 172.21.15.83:0/2890136257 shutdown_connections 2024-09-19T07:30:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.281+0000 7f522cab0640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52080678c0 0x7f5208069d80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.281+0000 7f522cab0640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5228154180 0x7f52281490d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.281+0000 7f522cab0640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5228070140 0x7f5228148b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.281+0000 7f522cab0640 1 --2- 172.21.15.83:0/2890136257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f522806cef0 0x7f522814f400 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.281+0000 7f522cab0640 1 -- 172.21.15.83:0/2890136257 >> 172.21.15.83:0/2890136257 conn(0x7f52280c5090 msgr2=0x7f5228150b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:38.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.281+0000 7f522cab0640 1 -- 172.21.15.83:0/2890136257 shutdown_connections 2024-09-19T07:30:38.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:38.281+0000 7f522cab0640 1 -- 172.21.15.83:0/2890136257 wait complete. 2024-09-19T07:30:38.292 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:30:39.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:39.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbfcb41640 1 Processor -- start 2024-09-19T07:30:39.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbfcb41640 1 -- start start 2024-09-19T07:30:39.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbfcb41640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fdbf80ab750 0x7fdbf80c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:39.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbfcb41640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbf806de90 con 0x7fdbf80ab060 2024-09-19T07:30:39.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbfcb41640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbf806e170 con 0x7fdbf80c2530 2024-09-19T07:30:39.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbfcb41640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fdbf806e450 con 0x7fdbf80ab750 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf6d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdbf80ab060 0x7fdbf815cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45170/0 (socket says 172.21.15.83:45170) 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf6d9d640 1 -- 172.21.15.83:0/1355966218 learned_addr learned my addr 172.21.15.83:0/1355966218 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744117536 0 0) 0x7fdbf806de90 con 0x7fdbf80ab060 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbe4003530 con 0x7fdbf80ab060 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf659c640 1 --2- 172.21.15.83:0/1355966218 >> v2:172.21.15.184:3300/0 conn(0x7fdbf80ab750 0x7fdbf80c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2904176993 0 0) 0x7fdbe4003530 con 0x7fdbf80ab060 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 >> v2:172.21.15.184:3300/0 conn(0x7fdbf80ab750 msgr2=0x7fdbf80c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 --2- 172.21.15.83:0/1355966218 >> v2:172.21.15.184:3300/0 conn(0x7fdbf80ab750 0x7fdbf80c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 >> v1:172.21.15.184:6789/0 conn(0x7fdbf80c2530 legacy=0x7fdbf81528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf806e730 con 0x7fdbf80ab060 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fdbf0001a00 con 0x7fdbf80ab060 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 >> v1:172.21.15.83:6789/0 conn(0x7fdbf80ab060 legacy=0x7fdbf815cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:39.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbe40039c0 0x7fdbe4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbe4004810 0x7fdbe4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbe40052f0 0x7fdbe400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf806e450 con 0x7fdbe40039c0 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe400e3d0 con 0x7fdbe4004810 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdbe400e5d0 con 0x7fdbe40052f0 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf659c640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbe40039c0 0x7fdbe4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf5d9b640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbe4004810 0x7fdbe4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf6d9d640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbe40052f0 0x7fdbe400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf659c640 1 -- 172.21.15.83:0/1355966218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbe40052f0 msgr2=0x7fdbe400dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf659c640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbe40052f0 0x7fdbe400dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf659c640 1 -- 172.21.15.83:0/1355966218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbe4004810 msgr2=0x7fdbe4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf659c640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbe4004810 0x7fdbe4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:39.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf659c640 1 -- 172.21.15.83:0/1355966218 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe4003530 con 0x7fdbe40039c0 2024-09-19T07:30:39.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf5d9b640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbe4004810 0x7fdbe4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:30:39.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf6d9d640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbe40052f0 0x7fdbe400dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:39.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.441+0000 7fdbf6d9d640 1 -- 172.21.15.83:0/1355966218 reap_dead start 2024-09-19T07:30:39.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbf659c640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbe40039c0 0x7fdbe4003de0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fdbe000ba00 tx=0x7fdbe001c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ca06fe55fa9d2c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:39.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe000b600 con 0x7fdbe40039c0 2024-09-19T07:30:39.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbe00af620 con 0x7fdbe40039c0 2024-09-19T07:30:39.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbf559a640 1 -- 172.21.15.83:0/1355966218 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe00afa10 con 0x7fdbe40039c0 2024-09-19T07:30:39.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1355966218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbe40039c0 msgr2=0x7fdbe4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:39.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbe40039c0 0x7fdbe4003de0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fdbe000ba00 tx=0x7fdbe001c640 comp rx=0 tx=0).stop 2024-09-19T07:30:39.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1355966218 shutdown_connections 2024-09-19T07:30:39.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 --2- 172.21.15.83:0/1355966218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbe40039c0 0x7fdbe4003de0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:39.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1355966218 >> 172.21.15.83:0/1355966218 conn(0x7fdbf806bc80 msgr2=0x7fdbf806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:39.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1355966218 shutdown_connections 2024-09-19T07:30:39.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1355966218 wait complete. 2024-09-19T07:30:39.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 Processor -- start 2024-09-19T07:30:39.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 -- start start 2024-09-19T07:30:39.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbe40039c0 0x7fdbf81deb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:39.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbf81558b0 0x7fdbf81e2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:39.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbf81ebf90 0x7fdbf814b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:39.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf80c2380 con 0x7fdbf81558b0 2024-09-19T07:30:39.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf80c2200 con 0x7fdbe40039c0 2024-09-19T07:30:39.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbfcb41640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdbf80abd40 con 0x7fdbf81ebf90 2024-09-19T07:30:39.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbf5d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbf81558b0 0x7fdbf81e2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:39.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbf5d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbf81558b0 0x7fdbf81e2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39868/0 (socket says 172.21.15.83:39868) 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbf5d9b640 1 -- 172.21.15.83:0/1990249428 learned_addr learned my addr 172.21.15.83:0/1990249428 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbf659c640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbe40039c0 0x7fdbf81deb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.445+0000 7fdbf6d9d640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbf81ebf90 0x7fdbf814b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbf5d9b640 1 -- 172.21.15.83:0/1990249428 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbf81ebf90 msgr2=0x7fdbf814b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbf5d9b640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbf81ebf90 0x7fdbf814b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbf5d9b640 1 -- 172.21.15.83:0/1990249428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbe40039c0 msgr2=0x7fdbf81deb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbf5d9b640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbe40039c0 0x7fdbf81deb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbf5d9b640 1 -- 172.21.15.83:0/1990249428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf814c4b0 con 0x7fdbf81558b0 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbf659c640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbe40039c0 0x7fdbf81deb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbf5d9b640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbf81558b0 0x7fdbf81e2080 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fdbe800e550 tx=0x7fdbe800ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbeeffd640 1 -- 172.21.15.83:0/1990249428 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe80052e0 con 0x7fdbf81558b0 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbeeffd640 1 -- 172.21.15.83:0/1990249428 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbe8018070 con 0x7fdbf81558b0 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1990249428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbf81de360 con 0x7fdbf81558b0 2024-09-19T07:30:39.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbeeffd640 1 -- 172.21.15.83:0/1990249428 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8003820 con 0x7fdbf81558b0 2024-09-19T07:30:39.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1990249428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbf8156450 con 0x7fdbf81558b0 2024-09-19T07:30:39.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1990249428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbf81528c0 con 0x7fdbf81558b0 2024-09-19T07:30:39.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbeeffd640 1 -- 172.21.15.83:0/1990249428 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdbe80039c0 con 0x7fdbf81558b0 2024-09-19T07:30:39.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbeeffd640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdbd80678f0 0x7fdbd8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:39.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbf659c640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdbd80678f0 0x7fdbd8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:39.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbeeffd640 1 -- 172.21.15.83:0/1990249428 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7fdbe8086240 con 0x7fdbf81558b0 2024-09-19T07:30:39.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.449+0000 7fdbf659c640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdbd80678f0 0x7fdbd8069db0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdbe00b0000 tx=0x7fdbe00046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:39.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.453+0000 7fdbeeffd640 1 -- 172.21.15.83:0/1990249428 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdbe8056560 con 0x7fdbf81558b0 2024-09-19T07:30:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1990249428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdbf8073880 con 0x7fdbf81558b0 2024-09-19T07:30:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbeeffd640 1 -- 172.21.15.83:0/1990249428 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7fdbe8055e60 con 0x7fdbf81558b0 2024-09-19T07:30:39.645 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:39.645 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:39.646 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:39.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1990249428 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdbd80678f0 msgr2=0x7fdbd8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:39.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbfcb41640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdbd80678f0 0x7fdbd8069db0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdbe00b0000 tx=0x7fdbe00046d0 comp rx=0 tx=0).stop 2024-09-19T07:30:39.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1990249428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbf81558b0 msgr2=0x7fdbf81e2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbfcb41640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbf81558b0 0x7fdbf81e2080 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fdbe800e550 tx=0x7fdbe800ea10 comp rx=0 tx=0).stop 2024-09-19T07:30:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1990249428 shutdown_connections 2024-09-19T07:30:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbfcb41640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdbd80678f0 0x7fdbd8069db0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbfcb41640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbf81ebf90 0x7fdbf814b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbfcb41640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbf81558b0 0x7fdbf81e2080 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbfcb41640 1 --2- 172.21.15.83:0/1990249428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbe40039c0 0x7fdbf81deb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1990249428 >> 172.21.15.83:0/1990249428 conn(0x7fdbf806bc80 msgr2=0x7fdbf80727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.649+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1990249428 shutdown_connections 2024-09-19T07:30:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:39.653+0000 7fdbfcb41640 1 -- 172.21.15.83:0/1990249428 wait complete. 2024-09-19T07:30:39.663 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:30:40.292 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:40.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f108199c640 1 Processor -- start 2024-09-19T07:30:40.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f108199c640 1 -- start start 2024-09-19T07:30:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f108199c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f107c150290 0x7f107c0c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f108199c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f107c153eb0 0x7f107c154360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f108199c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f107c150670 con 0x7f107c152b20 2024-09-19T07:30:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f108199c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f107c0716c0 con 0x7f107c153eb0 2024-09-19T07:30:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f108199c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f107c0c3920 con 0x7f107c150290 2024-09-19T07:30:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f107b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f107c153eb0 0x7f107c154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f107a7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f107c152b20 0x7f107c0c4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57182/0 (socket says 172.21.15.83:57182) 2024-09-19T07:30:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f107affd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f107c150290 0x7f107c0c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f107b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f107c153eb0 0x7f107c154360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46532/0 (socket says 172.21.15.83:46532) 2024-09-19T07:30:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f107a7fc640 1 -- 172.21.15.83:0/2957154237 learned_addr learned my addr 172.21.15.83:0/2957154237 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:30:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f107b7fe640 1 -- 172.21.15.83:0/2957154237 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f107c150290 msgr2=0x7f107c0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f107b7fe640 1 --2- 172.21.15.83:0/2957154237 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f107c150290 0x7f107c0c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:40.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f107b7fe640 1 -- 172.21.15.83:0/2957154237 >> v1:172.21.15.83:6789/0 conn(0x7f107c152b20 legacy=0x7f107c0c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:40.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.425+0000 7f107b7fe640 1 -- 172.21.15.83:0/2957154237 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107c0c2880 con 0x7f107c153eb0 2024-09-19T07:30:40.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f107b7fe640 1 --2- 172.21.15.83:0/2957154237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f107c153eb0 0x7f107c154360 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1074009d90 tx=0x7f107401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4dd1f096f0220c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:40.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f1079ffb640 1 -- 172.21.15.83:0/2957154237 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10740b2070 con 0x7f107c153eb0 2024-09-19T07:30:40.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f1079ffb640 1 -- 172.21.15.83:0/2957154237 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10740a5920 con 0x7f107c153eb0 2024-09-19T07:30:40.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f1079ffb640 1 -- 172.21.15.83:0/2957154237 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10740a5cf0 con 0x7f107c153eb0 2024-09-19T07:30:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 -- 172.21.15.83:0/2957154237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f107c153eb0 msgr2=0x7f107c154360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 --2- 172.21.15.83:0/2957154237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f107c153eb0 0x7f107c154360 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1074009d90 tx=0x7f107401c970 comp rx=0 tx=0).stop 2024-09-19T07:30:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 -- 172.21.15.83:0/2957154237 shutdown_connections 2024-09-19T07:30:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 --2- 172.21.15.83:0/2957154237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f107c153eb0 0x7f107c154360 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:40.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 --2- 172.21.15.83:0/2957154237 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f107c150290 0x7f107c0c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:40.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 -- 172.21.15.83:0/2957154237 >> 172.21.15.83:0/2957154237 conn(0x7f107c06bc80 msgr2=0x7f107c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:40.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 -- 172.21.15.83:0/2957154237 shutdown_connections 2024-09-19T07:30:40.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 -- 172.21.15.83:0/2957154237 wait complete. 2024-09-19T07:30:40.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 Processor -- start 2024-09-19T07:30:40.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 -- start start 2024-09-19T07:30:40.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f107c150290 0x7f107c0ab730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f107c152b20 0x7f107c1f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f107c153eb0 0x7f107c1edfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f107c0c4790 con 0x7f107c153eb0 2024-09-19T07:30:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f107c0c2530 con 0x7f107c152b20 2024-09-19T07:30:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f108199c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f107c0c4910 con 0x7f107c150290 2024-09-19T07:30:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f107b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f107c153eb0 0x7f107c1edfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f107b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f107c153eb0 0x7f107c1edfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46536/0 (socket says 172.21.15.83:46536) 2024-09-19T07:30:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f107b7fe640 1 -- 172.21.15.83:0/3114534497 learned_addr learned my addr 172.21.15.83:0/3114534497 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f107affd640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f107c150290 0x7f107c0ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f107b7fe640 1 -- 172.21.15.83:0/3114534497 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f107c150290 msgr2=0x7f107c0ab730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f107b7fe640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f107c150290 0x7f107c0ab730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f107a7fc640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f107c152b20 0x7f107c1f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:40.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f107b7fe640 1 -- 172.21.15.83:0/3114534497 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f107c152b20 msgr2=0x7f107c1f1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:40.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f107b7fe640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f107c152b20 0x7f107c1f1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:40.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f107b7fe640 1 -- 172.21.15.83:0/3114534497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107c1f2410 con 0x7f107c153eb0 2024-09-19T07:30:40.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.429+0000 7f107affd640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f107c150290 0x7f107c0ab730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:40.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.433+0000 7f107b7fe640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f107c153eb0 0x7f107c1edfe0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1074009d90 tx=0x7f1074019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:40.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.433+0000 7f1063fff640 1 -- 172.21.15.83:0/3114534497 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10740b2050 con 0x7f107c153eb0 2024-09-19T07:30:40.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.433+0000 7f1063fff640 1 -- 172.21.15.83:0/3114534497 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10740be070 con 0x7f107c153eb0 2024-09-19T07:30:40.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.433+0000 7f1063fff640 1 -- 172.21.15.83:0/3114534497 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10740ada70 con 0x7f107c153eb0 2024-09-19T07:30:40.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.433+0000 7f108199c640 1 -- 172.21.15.83:0/3114534497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f107c0abd30 con 0x7f107c153eb0 2024-09-19T07:30:40.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.433+0000 7f108199c640 1 -- 172.21.15.83:0/3114534497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f107c1f16d0 con 0x7f107c153eb0 2024-09-19T07:30:40.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.433+0000 7f108199c640 1 -- 172.21.15.83:0/3114534497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f107c0c3e90 con 0x7f107c153eb0 2024-09-19T07:30:40.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.433+0000 7f1063fff640 1 -- 172.21.15.83:0/3114534497 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f10740a57a0 con 0x7f107c153eb0 2024-09-19T07:30:40.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.433+0000 7f1063fff640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10680678f0 0x7f1068069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:40.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.433+0000 7f1063fff640 1 -- 172.21.15.83:0/3114534497 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f1074121610 con 0x7f107c153eb0 2024-09-19T07:30:40.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.433+0000 7f107affd640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10680678f0 0x7f1068069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:40.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.433+0000 7f107affd640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10680678f0 0x7f1068069db0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f106c009ae0 tx=0x7f106c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:40.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.437+0000 7f1063fff640 1 -- 172.21.15.83:0/3114534497 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1074016560 con 0x7f107c153eb0 2024-09-19T07:30:40.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 -- 172.21.15.83:0/3114534497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f107c0c1ff0 con 0x7f107c153eb0 2024-09-19T07:30:40.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f1063fff640 1 -- 172.21.15.83:0/3114534497 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7f10740f1a50 con 0x7f107c153eb0 2024-09-19T07:30:40.617 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:40.617 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:40.617 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:40.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 -- 172.21.15.83:0/3114534497 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10680678f0 msgr2=0x7f1068069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:40.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10680678f0 0x7f1068069db0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f106c009ae0 tx=0x7f106c008040 comp rx=0 tx=0).stop 2024-09-19T07:30:40.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 -- 172.21.15.83:0/3114534497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f107c153eb0 msgr2=0x7f107c1edfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:40.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f107c153eb0 0x7f107c1edfe0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1074009d90 tx=0x7f1074019710 comp rx=0 tx=0).stop 2024-09-19T07:30:40.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 -- 172.21.15.83:0/3114534497 shutdown_connections 2024-09-19T07:30:40.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f10680678f0 0x7f1068069db0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:40.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f107c153eb0 0x7f107c1edfe0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:40.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f107c152b20 0x7f107c1f1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:40.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 --2- 172.21.15.83:0/3114534497 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f107c150290 0x7f107c0ab730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:40.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 -- 172.21.15.83:0/3114534497 >> 172.21.15.83:0/3114534497 conn(0x7f107c06bc80 msgr2=0x7f107c15c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:40.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 -- 172.21.15.83:0/3114534497 shutdown_connections 2024-09-19T07:30:40.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:40.621+0000 7f108199c640 1 -- 172.21.15.83:0/3114534497 wait complete. 2024-09-19T07:30:40.634 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:30:41.664 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:41.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.797+0000 7ff3f90e0640 1 Processor -- start 2024-09-19T07:30:41.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.797+0000 7ff3f90e0640 1 -- start start 2024-09-19T07:30:41.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f90e0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4150290 0x7ff3f40c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:41.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f90e0640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff3f4153eb0 0x7ff3f4154360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:41.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f90e0640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3f4150670 con 0x7ff3f4152b20 2024-09-19T07:30:41.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f90e0640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff3f40716c0 con 0x7ff3f4153eb0 2024-09-19T07:30:41.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f90e0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff3f40c3920 con 0x7ff3f4150290 2024-09-19T07:30:41.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f359e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff3f4153eb0 0x7ff3f4154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:41.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f2d9d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4150290 0x7ff3f40c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:41.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f2d9d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4150290 0x7ff3f40c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:51676/0 (socket says 172.21.15.83:51676) 2024-09-19T07:30:41.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f2d9d640 1 -- 172.21.15.83:0/2631875581 learned_addr learned my addr 172.21.15.83:0/2631875581 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:41.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f2d9d640 1 -- 172.21.15.83:0/2631875581 >> v2:172.21.15.184:3300/0 conn(0x7ff3f4153eb0 msgr2=0x7ff3f4154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:41.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f2d9d640 1 --2- 172.21.15.83:0/2631875581 >> v2:172.21.15.184:3300/0 conn(0x7ff3f4153eb0 0x7ff3f4154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:41.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f2d9d640 1 -- 172.21.15.83:0/2631875581 >> v1:172.21.15.184:6789/0 conn(0x7ff3f4152b20 legacy=0x7ff3f40c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:41.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f2d9d640 1 -- 172.21.15.83:0/2631875581 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f40c2880 con 0x7ff3f4150290 2024-09-19T07:30:41.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.801+0000 7ff3f359e640 1 --2- 172.21.15.83:0/2631875581 >> v2:172.21.15.184:3300/0 conn(0x7ff3f4153eb0 0x7ff3f4154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:30:41.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f2d9d640 1 --2- 172.21.15.83:0/2631875581 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4150290 0x7ff3f40c1ff0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009e50 tx=0x7ff3e401ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c196cadb9f684d03 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:41.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f1d9b640 1 -- 172.21.15.83:0/2631875581 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e40a57f0 con 0x7ff3f4150290 2024-09-19T07:30:41.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f1d9b640 1 -- 172.21.15.83:0/2631875581 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3e40a5990 con 0x7ff3f4150290 2024-09-19T07:30:41.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f1d9b640 1 -- 172.21.15.83:0/2631875581 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e40a5d60 con 0x7ff3f4150290 2024-09-19T07:30:41.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 -- 172.21.15.83:0/2631875581 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4150290 msgr2=0x7ff3f40c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:41.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 --2- 172.21.15.83:0/2631875581 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4150290 0x7ff3f40c1ff0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009e50 tx=0x7ff3e401ca30 comp rx=0 tx=0).stop 2024-09-19T07:30:41.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 -- 172.21.15.83:0/2631875581 shutdown_connections 2024-09-19T07:30:41.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 --2- 172.21.15.83:0/2631875581 >> v2:172.21.15.184:3300/0 conn(0x7ff3f4153eb0 0x7ff3f4154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:41.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 --2- 172.21.15.83:0/2631875581 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4150290 0x7ff3f40c1ff0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:41.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 -- 172.21.15.83:0/2631875581 >> 172.21.15.83:0/2631875581 conn(0x7ff3f406bc80 msgr2=0x7ff3f406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:41.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 -- 172.21.15.83:0/2631875581 shutdown_connections 2024-09-19T07:30:41.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 -- 172.21.15.83:0/2631875581 wait complete. 2024-09-19T07:30:41.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 Processor -- start 2024-09-19T07:30:41.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 -- start start 2024-09-19T07:30:41.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3f4150290 0x7ff3f414ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:41.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3f4152b20 0x7ff3f40abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:41.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4153eb0 0x7ff3f4071f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:41.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f40c4790 con 0x7ff3f4152b20 2024-09-19T07:30:41.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f40c2530 con 0x7ff3f4150290 2024-09-19T07:30:41.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f90e0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff3f40c4910 con 0x7ff3f4153eb0 2024-09-19T07:30:41.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3f4152b20 0x7ff3f40abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:41.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3f4152b20 0x7ff3f40abbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46544/0 (socket says 172.21.15.83:46544) 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f259c640 1 -- 172.21.15.83:0/1096095505 learned_addr learned my addr 172.21.15.83:0/1096095505 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f359e640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4153eb0 0x7ff3f4071f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f2d9d640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3f4150290 0x7ff3f414ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f259c640 1 -- 172.21.15.83:0/1096095505 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4153eb0 msgr2=0x7ff3f4071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f259c640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4153eb0 0x7ff3f4071f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f259c640 1 -- 172.21.15.83:0/1096095505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3f4150290 msgr2=0x7ff3f414ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f259c640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3f4150290 0x7ff3f414ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f259c640 1 -- 172.21.15.83:0/1096095505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f40ab760 con 0x7ff3f4152b20 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.805+0000 7ff3f359e640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4153eb0 0x7ff3f4071f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.809+0000 7ff3f259c640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3f4152b20 0x7ff3f40abbf0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800c370 tx=0x7ff3e800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.809+0000 7ff3db7fe640 1 -- 172.21.15.83:0/1096095505 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e80052d0 con 0x7ff3f4152b20 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.809+0000 7ff3db7fe640 1 -- 172.21.15.83:0/1096095505 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3e8005470 con 0x7ff3f4152b20 2024-09-19T07:30:41.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.809+0000 7ff3f90e0640 1 -- 172.21.15.83:0/1096095505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f4073b60 con 0x7ff3f4152b20 2024-09-19T07:30:41.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.809+0000 7ff3db7fe640 1 -- 172.21.15.83:0/1096095505 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8007260 con 0x7ff3f4152b20 2024-09-19T07:30:41.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.809+0000 7ff3f90e0640 1 -- 172.21.15.83:0/1096095505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f4072fd0 con 0x7ff3f4152b20 2024-09-19T07:30:41.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.809+0000 7ff3f90e0640 1 -- 172.21.15.83:0/1096095505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3f40c1ff0 con 0x7ff3f4152b20 2024-09-19T07:30:41.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.809+0000 7ff3db7fe640 1 -- 172.21.15.83:0/1096095505 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff3e8007400 con 0x7ff3f4152b20 2024-09-19T07:30:41.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.809+0000 7ff3db7fe640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff3e00678f0 0x7ff3e0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:41.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.809+0000 7ff3db7fe640 1 -- 172.21.15.83:0/1096095505 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7ff3e8085da0 con 0x7ff3f4152b20 2024-09-19T07:30:41.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.809+0000 7ff3f2d9d640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff3e00678f0 0x7ff3e0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:41.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.809+0000 7ff3f2d9d640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff3e00678f0 0x7ff3e0069db0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40ae000 tx=0x7ff3e40a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:41.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:41.813+0000 7ff3db7fe640 1 -- 172.21.15.83:0/1096095505 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff3e80561a0 con 0x7ff3f4152b20 2024-09-19T07:30:41.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 -- 172.21.15.83:0/1096095505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3f4069f70 con 0x7ff3f4152b20 2024-09-19T07:30:41.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3db7fe640 1 -- 172.21.15.83:0/1096095505 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7ff3e8055aa0 con 0x7ff3f4152b20 2024-09-19T07:30:41.997 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:41.997 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:41.997 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:41.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 -- 172.21.15.83:0/1096095505 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff3e00678f0 msgr2=0x7ff3e0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:41.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff3e00678f0 0x7ff3e0069db0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40ae000 tx=0x7ff3e40a7070 comp rx=0 tx=0).stop 2024-09-19T07:30:41.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 -- 172.21.15.83:0/1096095505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3f4152b20 msgr2=0x7ff3f40abbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:41.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3f4152b20 0x7ff3f40abbf0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800c370 tx=0x7ff3e800c830 comp rx=0 tx=0).stop 2024-09-19T07:30:41.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 -- 172.21.15.83:0/1096095505 shutdown_connections 2024-09-19T07:30:41.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff3e00678f0 0x7ff3e0069db0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:41.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3f4153eb0 0x7ff3f4071f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:41.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3f4152b20 0x7ff3f40abbf0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:41.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 --2- 172.21.15.83:0/1096095505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3f4150290 0x7ff3f414ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:42.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 -- 172.21.15.83:0/1096095505 >> 172.21.15.83:0/1096095505 conn(0x7ff3f406bc80 msgr2=0x7ff3f414c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:42.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 -- 172.21.15.83:0/1096095505 shutdown_connections 2024-09-19T07:30:42.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.001+0000 7ff3f90e0640 1 -- 172.21.15.83:0/1096095505 wait complete. 2024-09-19T07:30:42.015 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:30:42.635 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d1607b640 1 Processor -- start 2024-09-19T07:30:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d1607b640 1 -- start start 2024-09-19T07:30:42.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d1607b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2d10071100 0x7f2d101545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:42.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d1607b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d100a9e60 con 0x7f2d1006dea0 2024-09-19T07:30:42.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d1607b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d100aa140 con 0x7f2d100c28e0 2024-09-19T07:30:42.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d1607b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f2d10071b50 con 0x7f2d10071100 2024-09-19T07:30:42.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2d1006dea0 0x7f2d1006e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57186/0 (socket says 172.21.15.83:57186) 2024-09-19T07:30:42.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0ffff640 1 -- 172.21.15.83:0/3551502452 learned_addr learned my addr 172.21.15.83:0/3551502452 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:30:42.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0f7fe640 1 --2- 172.21.15.83:0/3551502452 >> v2:172.21.15.184:3300/0 conn(0x7f2d10071100 0x7f2d101545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:42.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 269187453 0 0) 0x7f2d100a9e60 con 0x7f2d1006dea0 2024-09-19T07:30:42.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cf8003530 con 0x7f2d1006dea0 2024-09-19T07:30:42.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4024948583 0 0) 0x7f2cf8003530 con 0x7f2d1006dea0 2024-09-19T07:30:42.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 >> v2:172.21.15.184:3300/0 conn(0x7f2d10071100 msgr2=0x7f2d101545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:42.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 --2- 172.21.15.83:0/3551502452 >> v2:172.21.15.184:3300/0 conn(0x7f2d10071100 0x7f2d101545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:42.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 >> v1:172.21.15.184:6789/0 conn(0x7f2d100c28e0 legacy=0x7f2d10150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:42.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d100c2240 con 0x7f2d1006dea0 2024-09-19T07:30:42.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f2d08001a00 con 0x7f2d1006dea0 2024-09-19T07:30:42.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 >> v1:172.21.15.83:6789/0 conn(0x7f2d1006dea0 legacy=0x7f2d1006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 --2- 172.21.15.83:0/3551502452 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2cf80039c0 0x7f2cf8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 --2- 172.21.15.83:0/3551502452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2cf8004810 0x7f2cf8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 --2- 172.21.15.83:0/3551502452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf80051d0 0x7f2cf800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d10071b50 con 0x7f2cf80051d0 2024-09-19T07:30:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf800c170 con 0x7f2cf8004810 2024-09-19T07:30:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf800c370 con 0x7f2cf80039c0 2024-09-19T07:30:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0ffff640 1 --2- 172.21.15.83:0/3551502452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf80051d0 0x7f2cf800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0f7fe640 1 --2- 172.21.15.83:0/3551502452 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2cf80039c0 0x7f2cf8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0effd640 1 --2- 172.21.15.83:0/3551502452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2cf8004810 0x7f2cf8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0ffff640 1 -- 172.21.15.83:0/3551502452 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2cf80039c0 msgr2=0x7f2cf8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0ffff640 1 --2- 172.21.15.83:0/3551502452 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2cf80039c0 0x7f2cf8003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:42.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0ffff640 1 -- 172.21.15.83:0/3551502452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2cf8004810 msgr2=0x7f2cf8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:42.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0ffff640 1 --2- 172.21.15.83:0/3551502452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2cf8004810 0x7f2cf8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:42.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0ffff640 1 -- 172.21.15.83:0/3551502452 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf8003530 con 0x7f2cf80051d0 2024-09-19T07:30:42.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.769+0000 7f2d0effd640 1 --2- 172.21.15.83:0/3551502452 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2cf8004810 0x7f2cf8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:42.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d0ffff640 1 -- 172.21.15.83:0/3551502452 reap_dead start 2024-09-19T07:30:42.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d0ffff640 1 --2- 172.21.15.83:0/3551502452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf80051d0 0x7f2cf800b9d0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800a840 tx=0x7f2d0801fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3d7f438de9c45c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:42.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0800a460 con 0x7f2cf80051d0 2024-09-19T07:30:42.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d080aa710 con 0x7f2cf80051d0 2024-09-19T07:30:42.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d0e7fc640 1 -- 172.21.15.83:0/3551502452 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d080aab00 con 0x7f2cf80051d0 2024-09-19T07:30:42.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 -- 172.21.15.83:0/3551502452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf80051d0 msgr2=0x7f2cf800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:42.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 --2- 172.21.15.83:0/3551502452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf80051d0 0x7f2cf800b9d0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800a840 tx=0x7f2d0801fac0 comp rx=0 tx=0).stop 2024-09-19T07:30:42.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 -- 172.21.15.83:0/3551502452 shutdown_connections 2024-09-19T07:30:42.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 --2- 172.21.15.83:0/3551502452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf80051d0 0x7f2cf800b9d0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:42.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 -- 172.21.15.83:0/3551502452 >> 172.21.15.83:0/3551502452 conn(0x7f2d1006bc80 msgr2=0x7f2d1006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:42.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 -- 172.21.15.83:0/3551502452 shutdown_connections 2024-09-19T07:30:42.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 -- 172.21.15.83:0/3551502452 wait complete. 2024-09-19T07:30:42.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 Processor -- start 2024-09-19T07:30:42.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 -- start start 2024-09-19T07:30:42.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2cf80051d0 0x7f2d10149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1014aac0 0x7f2d1014af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d101ea680 0x7f2d101eab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d100714e0 con 0x7f2d101ea680 2024-09-19T07:30:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2d100c1ff0 con 0x7f2cf80051d0 2024-09-19T07:30:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d1607b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2d10071660 con 0x7f2d1014aac0 2024-09-19T07:30:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d0ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d101ea680 0x7f2d101eab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d0ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d101ea680 0x7f2d101eab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46552/0 (socket says 172.21.15.83:46552) 2024-09-19T07:30:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d0ffff640 1 -- 172.21.15.83:0/18682864 learned_addr learned my addr 172.21.15.83:0/18682864 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.773+0000 7f2d0f7fe640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2cf80051d0 0x7f2d10149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d0effd640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1014aac0 0x7f2d1014af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d0ffff640 1 -- 172.21.15.83:0/18682864 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1014aac0 msgr2=0x7f2d1014af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d0ffff640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1014aac0 0x7f2d1014af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d0ffff640 1 -- 172.21.15.83:0/18682864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2cf80051d0 msgr2=0x7f2d10149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d0ffff640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2cf80051d0 0x7f2d10149f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d0ffff640 1 -- 172.21.15.83:0/18682864 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1014b480 con 0x7f2d101ea680 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d0effd640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1014aac0 0x7f2d1014af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d0ffff640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d101ea680 0x7f2d101eab10 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2d0801f8f0 tx=0x7f2cf8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2cf7fff640 1 -- 172.21.15.83:0/18682864 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d080bb070 con 0x7f2d101ea680 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2cf7fff640 1 -- 172.21.15.83:0/18682864 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d08001260 con 0x7f2d101ea680 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d1607b640 1 -- 172.21.15.83:0/18682864 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1014eb10 con 0x7f2d101ea680 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2cf7fff640 1 -- 172.21.15.83:0/18682864 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d080aaac0 con 0x7f2d101ea680 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d1607b640 1 -- 172.21.15.83:0/18682864 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d100c6000 con 0x7f2d101ea680 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d1607b640 1 -- 172.21.15.83:0/18682864 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d10154670 con 0x7f2d101ea680 2024-09-19T07:30:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2cf7fff640 1 -- 172.21.15.83:0/18682864 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2d080be070 con 0x7f2d101ea680 2024-09-19T07:30:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2cf7fff640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2cfc0678c0 0x7f2cfc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2cf7fff640 1 -- 172.21.15.83:0/18682864 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f2d08122d50 con 0x7f2d101ea680 2024-09-19T07:30:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d0f7fe640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2cfc0678c0 0x7f2cfc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.777+0000 7f2d0f7fe640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2cfc0678c0 0x7f2cfc069d80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2d1014e130 tx=0x7f2d00008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.781+0000 7f2cf7fff640 1 -- 172.21.15.83:0/18682864 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2d08126030 con 0x7f2d101ea680 2024-09-19T07:30:42.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.981+0000 7f2d1607b640 1 -- 172.21.15.83:0/18682864 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d100c4f30 con 0x7f2d101ea680 2024-09-19T07:30:42.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.981+0000 7f2cf7fff640 1 -- 172.21.15.83:0/18682864 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7f2d1014b480 con 0x7f2d101ea680 2024-09-19T07:30:42.977 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:42.978 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:42.978 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:42.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.981+0000 7f2d1607b640 1 -- 172.21.15.83:0/18682864 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2cfc0678c0 msgr2=0x7f2cfc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:42.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.981+0000 7f2d1607b640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2cfc0678c0 0x7f2cfc069d80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2d1014e130 tx=0x7f2d00008040 comp rx=0 tx=0).stop 2024-09-19T07:30:42.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.981+0000 7f2d1607b640 1 -- 172.21.15.83:0/18682864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d101ea680 msgr2=0x7f2d101eab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:42.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.981+0000 7f2d1607b640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d101ea680 0x7f2d101eab10 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2d0801f8f0 tx=0x7f2cf8005840 comp rx=0 tx=0).stop 2024-09-19T07:30:42.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.985+0000 7f2d1607b640 1 -- 172.21.15.83:0/18682864 shutdown_connections 2024-09-19T07:30:42.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.985+0000 7f2d1607b640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2cfc0678c0 0x7f2cfc069d80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:42.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.985+0000 7f2d1607b640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d101ea680 0x7f2d101eab10 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:42.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.985+0000 7f2d1607b640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1014aac0 0x7f2d1014af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:42.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.985+0000 7f2d1607b640 1 --2- 172.21.15.83:0/18682864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2cf80051d0 0x7f2d10149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:42.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.985+0000 7f2d1607b640 1 -- 172.21.15.83:0/18682864 >> 172.21.15.83:0/18682864 conn(0x7f2d1006bc80 msgr2=0x7f2d10073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:42.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.985+0000 7f2d1607b640 1 -- 172.21.15.83:0/18682864 shutdown_connections 2024-09-19T07:30:42.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:42.985+0000 7f2d1607b640 1 -- 172.21.15.83:0/18682864 wait complete. 2024-09-19T07:30:42.997 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:30:44.015 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5f6f5640 1 Processor -- start 2024-09-19T07:30:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5f6f5640 1 -- start start 2024-09-19T07:30:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5f6f5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fac580ab060 0x7fac581528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5f6f5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac5806de90 con 0x7fac580c3b70 2024-09-19T07:30:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5f6f5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac5806e170 con 0x7fac580ab750 2024-09-19T07:30:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5f6f5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fac5806e450 con 0x7fac580ab060 2024-09-19T07:30:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5cc90640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fac580ab060 0x7fac581528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5cc90640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fac580ab060 0x7fac581528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46562/0 (socket says 172.21.15.83:46562) 2024-09-19T07:30:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5cc90640 1 -- 172.21.15.83:0/1367079430 learned_addr learned my addr 172.21.15.83:0/1367079430 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5dc92640 1 --1- 172.21.15.83:0/1367079430 >> v1:172.21.15.83:6789/0 conn(0x7fac580c3b70 0x7fac5815cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57202/0 (socket says 172.21.15.83:57202) 2024-09-19T07:30:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5cc90640 1 -- 172.21.15.83:0/1367079430 >> v1:172.21.15.184:6789/0 conn(0x7fac580ab750 legacy=0x7fac580c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5cc90640 1 -- 172.21.15.83:0/1367079430 >> v1:172.21.15.83:6789/0 conn(0x7fac580c3b70 legacy=0x7fac5815cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5cc90640 1 -- 172.21.15.83:0/1367079430 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5806e730 con 0x7fac580ab060 2024-09-19T07:30:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.149+0000 7fac5cc90640 1 --2- 172.21.15.83:0/1367079430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac580ab060 0x7fac581528c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fac4c006fd0 tx=0x7fac4c01e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce42d0e1d0376b64 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac57fff640 1 -- 172.21.15.83:0/1367079430 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c0ad070 con 0x7fac580ab060 2024-09-19T07:30:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac57fff640 1 -- 172.21.15.83:0/1367079430 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac4c004230 con 0x7fac580ab060 2024-09-19T07:30:44.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac57fff640 1 -- 172.21.15.83:0/1367079430 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c003620 con 0x7fac580ab060 2024-09-19T07:30:44.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 -- 172.21.15.83:0/1367079430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac580ab060 msgr2=0x7fac581528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:44.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 --2- 172.21.15.83:0/1367079430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac580ab060 0x7fac581528c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fac4c006fd0 tx=0x7fac4c01e920 comp rx=0 tx=0).stop 2024-09-19T07:30:44.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 -- 172.21.15.83:0/1367079430 shutdown_connections 2024-09-19T07:30:44.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 --2- 172.21.15.83:0/1367079430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac580ab060 0x7fac581528c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:44.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 -- 172.21.15.83:0/1367079430 >> 172.21.15.83:0/1367079430 conn(0x7fac5806bc80 msgr2=0x7fac5806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:44.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 -- 172.21.15.83:0/1367079430 shutdown_connections 2024-09-19T07:30:44.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 -- 172.21.15.83:0/1367079430 wait complete. 2024-09-19T07:30:44.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 Processor -- start 2024-09-19T07:30:44.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 -- start start 2024-09-19T07:30:44.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac580ab060 0x7fac580c07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:44.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac580ab750 0x7fac580bcd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:44.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac580c3b70 0x7fac5814e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:44.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fac580abcb0 con 0x7fac580ab750 2024-09-19T07:30:44.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fac580abb30 con 0x7fac580ab060 2024-09-19T07:30:44.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5f6f5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fac581602f0 con 0x7fac580c3b70 2024-09-19T07:30:44.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5cc90640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac580ab750 0x7fac580bcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:44.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5cc90640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac580ab750 0x7fac580bcd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46564/0 (socket says 172.21.15.83:46564) 2024-09-19T07:30:44.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5cc90640 1 -- 172.21.15.83:0/3252862870 learned_addr learned my addr 172.21.15.83:0/3252862870 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:44.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5cc90640 1 -- 172.21.15.83:0/3252862870 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac580c3b70 msgr2=0x7fac5814e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:44.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5cc90640 1 --2- 172.21.15.83:0/3252862870 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac580c3b70 0x7fac5814e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:44.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5cc90640 1 -- 172.21.15.83:0/3252862870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac580ab060 msgr2=0x7fac580c07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:44.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5cc90640 1 --2- 172.21.15.83:0/3252862870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac580ab060 0x7fac580c07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:44.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5cc90640 1 -- 172.21.15.83:0/3252862870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac580bd8a0 con 0x7fac580ab750 2024-09-19T07:30:44.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac5cc90640 1 --2- 172.21.15.83:0/3252862870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac580ab750 0x7fac580bcd60 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fac4c000c10 tx=0x7fac4c01b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:44.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac55ffb640 1 -- 172.21.15.83:0/3252862870 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c0c0070 con 0x7fac580ab750 2024-09-19T07:30:44.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac55ffb640 1 -- 172.21.15.83:0/3252862870 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac4c004030 con 0x7fac580ab750 2024-09-19T07:30:44.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.153+0000 7fac55ffb640 1 -- 172.21.15.83:0/3252862870 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c0ad040 con 0x7fac580ab750 2024-09-19T07:30:44.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.157+0000 7fac5f6f5640 1 -- 172.21.15.83:0/3252862870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac580c11b0 con 0x7fac580ab750 2024-09-19T07:30:44.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.157+0000 7fac5f6f5640 1 -- 172.21.15.83:0/3252862870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5814f590 con 0x7fac580ab750 2024-09-19T07:30:44.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.157+0000 7fac5f6f5640 1 -- 172.21.15.83:0/3252862870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac581528c0 con 0x7fac580ab750 2024-09-19T07:30:44.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.157+0000 7fac55ffb640 1 -- 172.21.15.83:0/3252862870 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fac4c0a7850 con 0x7fac580ab750 2024-09-19T07:30:44.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.157+0000 7fac55ffb640 1 --2- 172.21.15.83:0/3252862870 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fac440678c0 0x7fac44069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:44.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.157+0000 7fac55ffb640 1 -- 172.21.15.83:0/3252862870 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7fac4c122540 con 0x7fac580ab750 2024-09-19T07:30:44.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.157+0000 7fac5d491640 1 --2- 172.21.15.83:0/3252862870 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fac440678c0 0x7fac44069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:44.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.157+0000 7fac5d491640 1 --2- 172.21.15.83:0/3252862870 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fac440678c0 0x7fac44069d80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fac50007fd0 tx=0x7fac50007d40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:44.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.161+0000 7fac55ffb640 1 -- 172.21.15.83:0/3252862870 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fac4c0185a0 con 0x7fac580ab750 2024-09-19T07:30:44.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.357+0000 7fac5f6f5640 1 -- 172.21.15.83:0/3252862870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac58070f40 con 0x7fac580ab750 2024-09-19T07:30:44.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.357+0000 7fac55ffb640 1 -- 172.21.15.83:0/3252862870 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7fac4c0f2860 con 0x7fac580ab750 2024-09-19T07:30:44.355 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:44.355 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.361+0000 7fac5f6f5640 1 -- 172.21.15.83:0/3252862870 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fac440678c0 msgr2=0x7fac44069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.361+0000 7fac5f6f5640 1 --2- 172.21.15.83:0/3252862870 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fac440678c0 0x7fac44069d80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fac50007fd0 tx=0x7fac50007d40 comp rx=0 tx=0).stop 2024-09-19T07:30:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.361+0000 7fac5f6f5640 1 -- 172.21.15.83:0/3252862870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac580ab750 msgr2=0x7fac580bcd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.361+0000 7fac5f6f5640 1 --2- 172.21.15.83:0/3252862870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac580ab750 0x7fac580bcd60 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fac4c000c10 tx=0x7fac4c01b6c0 comp rx=0 tx=0).stop 2024-09-19T07:30:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.361+0000 7fac5f6f5640 1 -- 172.21.15.83:0/3252862870 shutdown_connections 2024-09-19T07:30:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.361+0000 7fac5f6f5640 1 --2- 172.21.15.83:0/3252862870 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fac440678c0 0x7fac44069d80 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.361+0000 7fac5f6f5640 1 --2- 172.21.15.83:0/3252862870 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac580c3b70 0x7fac5814e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.361+0000 7fac5f6f5640 1 --2- 172.21.15.83:0/3252862870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fac580ab750 0x7fac580bcd60 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.361+0000 7fac5f6f5640 1 --2- 172.21.15.83:0/3252862870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac580ab060 0x7fac580c07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.361+0000 7fac5f6f5640 1 -- 172.21.15.83:0/3252862870 >> 172.21.15.83:0/3252862870 conn(0x7fac5806bc80 msgr2=0x7fac5806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.361+0000 7fac5f6f5640 1 -- 172.21.15.83:0/3252862870 shutdown_connections 2024-09-19T07:30:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:44.361+0000 7fac5f6f5640 1 -- 172.21.15.83:0/3252862870 wait complete. 2024-09-19T07:30:44.372 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:30:44.998 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:45.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff596512640 1 Processor -- start 2024-09-19T07:30:45.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff596512640 1 -- start start 2024-09-19T07:30:45.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff596512640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff590073110 0x7ff59015d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:45.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff596512640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff59015a100 con 0x7ff590157b70 2024-09-19T07:30:45.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff596512640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff59006f040 con 0x7ff59015ae80 2024-09-19T07:30:45.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff596512640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff59015a3a0 con 0x7ff590073110 2024-09-19T07:30:45.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff58ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff590073110 0x7ff59015d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:45.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff594aaf640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff590157b70 0x7ff590164a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57214/0 (socket says 172.21.15.83:57214) 2024-09-19T07:30:45.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff58ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff590073110 0x7ff59015d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46574/0 (socket says 172.21.15.83:46574) 2024-09-19T07:30:45.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff594aaf640 1 -- 172.21.15.83:0/1649384166 learned_addr learned my addr 172.21.15.83:0/1649384166 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:30:45.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff58ffff640 1 -- 172.21.15.83:0/1649384166 >> v1:172.21.15.184:6789/0 conn(0x7ff59015ae80 legacy=0x7ff590157460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:45.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff58ffff640 1 -- 172.21.15.83:0/1649384166 >> v1:172.21.15.83:6789/0 conn(0x7ff590157b70 legacy=0x7ff590164a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:45.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.133+0000 7ff58ffff640 1 -- 172.21.15.83:0/1649384166 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59015a680 con 0x7ff590073110 2024-09-19T07:30:45.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff58ffff640 1 --2- 172.21.15.83:0/1649384166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff590073110 0x7ff59015d6d0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff580009d90 tx=0x7ff58001c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17adff58e2316e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:45.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff58effd640 1 -- 172.21.15.83:0/1649384166 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5800a6700 con 0x7ff590073110 2024-09-19T07:30:45.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff58effd640 1 -- 172.21.15.83:0/1649384166 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5800a68a0 con 0x7ff590073110 2024-09-19T07:30:45.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff58effd640 1 -- 172.21.15.83:0/1649384166 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5800a6c70 con 0x7ff590073110 2024-09-19T07:30:45.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 -- 172.21.15.83:0/1649384166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff590073110 msgr2=0x7ff59015d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:45.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 --2- 172.21.15.83:0/1649384166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff590073110 0x7ff59015d6d0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff580009d90 tx=0x7ff58001c810 comp rx=0 tx=0).stop 2024-09-19T07:30:45.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 -- 172.21.15.83:0/1649384166 shutdown_connections 2024-09-19T07:30:45.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 --2- 172.21.15.83:0/1649384166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff590073110 0x7ff59015d6d0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:45.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 -- 172.21.15.83:0/1649384166 >> 172.21.15.83:0/1649384166 conn(0x7ff5900c95a0 msgr2=0x7ff5900c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:45.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 -- 172.21.15.83:0/1649384166 shutdown_connections 2024-09-19T07:30:45.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 -- 172.21.15.83:0/1649384166 wait complete. 2024-09-19T07:30:45.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 Processor -- start 2024-09-19T07:30:45.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 -- start start 2024-09-19T07:30:45.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff590073110 0x7ff59014e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:45.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff590157b70 0x7ff59015bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:45.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff59015ae80 0x7ff59015cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:45.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff59015dc10 con 0x7ff590157b70 2024-09-19T07:30:45.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff590073740 con 0x7ff590073110 2024-09-19T07:30:45.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff596512640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff59015dd90 con 0x7ff59015ae80 2024-09-19T07:30:45.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff58f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff590157b70 0x7ff59015bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:45.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff58f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff590157b70 0x7ff59015bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46578/0 (socket says 172.21.15.83:46578) 2024-09-19T07:30:45.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff58f7fe640 1 -- 172.21.15.83:0/3570418179 learned_addr learned my addr 172.21.15.83:0/3570418179 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:45.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.137+0000 7ff594aaf640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff59015ae80 0x7ff59015cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:45.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58ffff640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff590073110 0x7ff59014e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:45.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58f7fe640 1 -- 172.21.15.83:0/3570418179 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff59015ae80 msgr2=0x7ff59015cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:45.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58f7fe640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff59015ae80 0x7ff59015cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:45.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58f7fe640 1 -- 172.21.15.83:0/3570418179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff590073110 msgr2=0x7ff59014e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:45.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58f7fe640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff590073110 0x7ff59014e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:45.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58f7fe640 1 -- 172.21.15.83:0/3570418179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59014dd20 con 0x7ff590157b70 2024-09-19T07:30:45.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58f7fe640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff590157b70 0x7ff59015bc30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff58800b440 tx=0x7ff58800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:45.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58cff9640 1 -- 172.21.15.83:0/3570418179 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff588015020 con 0x7ff590157b70 2024-09-19T07:30:45.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58cff9640 1 -- 172.21.15.83:0/3570418179 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff588011890 con 0x7ff590157b70 2024-09-19T07:30:45.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff596512640 1 -- 172.21.15.83:0/3570418179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59014d410 con 0x7ff590157b70 2024-09-19T07:30:45.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58cff9640 1 -- 172.21.15.83:0/3570418179 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff588011c80 con 0x7ff590157b70 2024-09-19T07:30:45.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff596512640 1 -- 172.21.15.83:0/3570418179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff59014d6d0 con 0x7ff590157b70 2024-09-19T07:30:45.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff596512640 1 -- 172.21.15.83:0/3570418179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff59015b990 con 0x7ff590157b70 2024-09-19T07:30:45.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58cff9640 1 -- 172.21.15.83:0/3570418179 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff588018020 con 0x7ff590157b70 2024-09-19T07:30:45.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58cff9640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5700678c0 0x7ff570069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:45.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58ffff640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5700678c0 0x7ff570069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:45.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58cff9640 1 -- 172.21.15.83:0/3570418179 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7ff588086340 con 0x7ff590157b70 2024-09-19T07:30:45.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.141+0000 7ff58ffff640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5700678c0 0x7ff570069d80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff5800af000 tx=0x7ff5800a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:45.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.145+0000 7ff58cff9640 1 -- 172.21.15.83:0/3570418179 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff58801f030 con 0x7ff590157b70 2024-09-19T07:30:45.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 -- 172.21.15.83:0/3570418179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff59015e310 con 0x7ff590157b70 2024-09-19T07:30:45.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff58cff9640 1 -- 172.21.15.83:0/3570418179 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7ff58800d040 con 0x7ff590157b70 2024-09-19T07:30:45.334 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:45.334 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:45.334 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:45.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 -- 172.21.15.83:0/3570418179 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5700678c0 msgr2=0x7ff570069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:45.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5700678c0 0x7ff570069d80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff5800af000 tx=0x7ff5800a7000 comp rx=0 tx=0).stop 2024-09-19T07:30:45.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 -- 172.21.15.83:0/3570418179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff590157b70 msgr2=0x7ff59015bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:45.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff590157b70 0x7ff59015bc30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff58800b440 tx=0x7ff58800b900 comp rx=0 tx=0).stop 2024-09-19T07:30:45.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 -- 172.21.15.83:0/3570418179 shutdown_connections 2024-09-19T07:30:45.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5700678c0 0x7ff570069d80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:45.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff59015ae80 0x7ff59015cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:45.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff590157b70 0x7ff59015bc30 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:45.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 --2- 172.21.15.83:0/3570418179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff590073110 0x7ff59014e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:45.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 -- 172.21.15.83:0/3570418179 >> 172.21.15.83:0/3570418179 conn(0x7ff5900c95a0 msgr2=0x7ff5900c3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:45.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 -- 172.21.15.83:0/3570418179 shutdown_connections 2024-09-19T07:30:45.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:45.337+0000 7ff596512640 1 -- 172.21.15.83:0/3570418179 wait complete. 2024-09-19T07:30:45.351 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:30:46.373 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:46.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bf05d640 1 Processor -- start 2024-09-19T07:30:46.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bf05d640 1 -- start start 2024-09-19T07:30:46.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bf05d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21b8155d70 0x7f21b80c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:46.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bf05d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21b8156200 con 0x7f21b81592f0 2024-09-19T07:30:46.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bf05d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21b81564e0 con 0x7f21b80c6120 2024-09-19T07:30:46.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bf05d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f21b80c9930 con 0x7f21b8155d70 2024-09-19T07:30:46.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bd5fa640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f21b81592f0 0x7f21b8157ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57224/0 (socket says 172.21.15.83:57224) 2024-09-19T07:30:46.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bd5fa640 1 -- 172.21.15.83:0/2596676612 learned_addr learned my addr 172.21.15.83:0/2596676612 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:30:46.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b7fff640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21b8155d70 0x7f21b80c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:46.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1494731428 0 0) 0x7f21b8156200 con 0x7f21b81592f0 2024-09-19T07:30:46.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21a4003530 con 0x7f21b81592f0 2024-09-19T07:30:46.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3831403163 0 0) 0x7f21a4003530 con 0x7f21b81592f0 2024-09-19T07:30:46.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21b8155d70 msgr2=0x7f21b80c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:46.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21b8155d70 0x7f21b80c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:46.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 >> v1:172.21.15.184:6789/0 conn(0x7f21b80c6120 legacy=0x7f21b81575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:46.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b81567c0 con 0x7f21b81592f0 2024-09-19T07:30:46.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f21b0001a00 con 0x7f21b81592f0 2024-09-19T07:30:46.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 >> v1:172.21.15.83:6789/0 conn(0x7f21b81592f0 legacy=0x7f21b8157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:46.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21a40039f0 0x7f21a4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a4004810 0x7f21a4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21a40051d0 0x7f21a400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21a400bf10 con 0x7f21a40039f0 2024-09-19T07:30:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21a400c090 con 0x7f21a4004810 2024-09-19T07:30:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f21a400c260 con 0x7f21a40051d0 2024-09-19T07:30:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bcdf9640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21a40039f0 0x7f21a4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b7fff640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a4004810 0x7f21a4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bd5fa640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21a40051d0 0x7f21a400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bcdf9640 1 -- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21a40051d0 msgr2=0x7f21a400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bcdf9640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21a40051d0 0x7f21a400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bcdf9640 1 -- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a4004810 msgr2=0x7f21a4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bcdf9640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a4004810 0x7f21a4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bcdf9640 1 -- 172.21.15.83:0/2596676612 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a4003530 con 0x7f21a40039f0 2024-09-19T07:30:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21b7fff640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21a4004810 0x7f21a4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.509+0000 7f21bd5fa640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21a40051d0 0x7f21a400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:46.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21bcdf9640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21a40039f0 0x7f21a4003e10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f21a8001780 tx=0x7f21a801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255ba4f6b1e87c5f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:46.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a80013a0 con 0x7f21a40039f0 2024-09-19T07:30:46.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a80052f0 con 0x7f21a40039f0 2024-09-19T07:30:46.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21b77fe640 1 -- 172.21.15.83:0/2596676612 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a8003840 con 0x7f21a40039f0 2024-09-19T07:30:46.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21bf05d640 1 -- 172.21.15.83:0/2596676612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21a40039f0 msgr2=0x7f21a4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:46.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21bf05d640 1 --2- 172.21.15.83:0/2596676612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21a40039f0 0x7f21a4003e10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f21a8001780 tx=0x7f21a801c420 comp rx=0 tx=0).stop 2024-09-19T07:30:46.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21bd5fa640 1 -- 172.21.15.83:0/2596676612 reap_dead start 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21bf05d640 1 -- 172.21.15.83:0/2596676612 shutdown_connections 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21bf05d640 1 -- 172.21.15.83:0/2596676612 >> 172.21.15.83:0/2596676612 conn(0x7f21b806bc80 msgr2=0x7f21b806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21bf05d640 1 -- 172.21.15.83:0/2596676612 shutdown_connections 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21bf05d640 1 -- 172.21.15.83:0/2596676612 wait complete. 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21bf05d640 1 Processor -- start 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.513+0000 7f21bf05d640 1 -- start start 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bf05d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21a40039f0 0x7f21b80cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bf05d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21b8154e10 0x7f21b81f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bf05d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21b80c0640 0x7f21b814be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bf05d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21b8153860 con 0x7f21a40039f0 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bf05d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21b8161ea0 con 0x7f21b8154e10 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bf05d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f21b80cbee0 con 0x7f21b80c0640 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bcdf9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21a40039f0 0x7f21b80cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:46.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bcdf9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21a40039f0 0x7f21b80cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46598/0 (socket says 172.21.15.83:46598) 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bcdf9640 1 -- 172.21.15.83:0/2507383890 learned_addr learned my addr 172.21.15.83:0/2507383890 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bd5fa640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21b80c0640 0x7f21b814be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21b7fff640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21b8154e10 0x7f21b81f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bcdf9640 1 -- 172.21.15.83:0/2507383890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21b80c0640 msgr2=0x7f21b814be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bcdf9640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21b80c0640 0x7f21b814be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bcdf9640 1 -- 172.21.15.83:0/2507383890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21b8154e10 msgr2=0x7f21b81f23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bcdf9640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21b8154e10 0x7f21b81f23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bcdf9640 1 -- 172.21.15.83:0/2507383890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b8154930 con 0x7f21a40039f0 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bd5fa640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21b80c0640 0x7f21b814be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21b7fff640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21b8154e10 0x7f21b81f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bcdf9640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21a40039f0 0x7f21b80cb9a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f21a801c250 tx=0x7f21a4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21b57fa640 1 -- 172.21.15.83:0/2507383890 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a80b8070 con 0x7f21a40039f0 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21b57fa640 1 -- 172.21.15.83:0/2507383890 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a80054a0 con 0x7f21a40039f0 2024-09-19T07:30:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21b57fa640 1 -- 172.21.15.83:0/2507383890 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a8007a40 con 0x7f21a40039f0 2024-09-19T07:30:46.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bf05d640 1 -- 172.21.15.83:0/2507383890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b80c24d0 con 0x7f21a40039f0 2024-09-19T07:30:46.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bf05d640 1 -- 172.21.15.83:0/2507383890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b81f2d10 con 0x7f21a40039f0 2024-09-19T07:30:46.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21bf05d640 1 -- 172.21.15.83:0/2507383890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21b8155d70 con 0x7f21a40039f0 2024-09-19T07:30:46.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21b57fa640 1 -- 172.21.15.83:0/2507383890 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f21a8007470 con 0x7f21a40039f0 2024-09-19T07:30:46.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21b57fa640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21980678c0 0x7f2198069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:46.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21b57fa640 1 -- 172.21.15.83:0/2507383890 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f21a8120830 con 0x7f21a40039f0 2024-09-19T07:30:46.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21b7fff640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21980678c0 0x7f2198069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:46.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.517+0000 7f21b7fff640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21980678c0 0x7f2198069d80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f21b80c21c0 tx=0x7f21a0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:46.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.525+0000 7f21b57fa640 1 -- 172.21.15.83:0/2507383890 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f21a8016130 con 0x7f21a40039f0 2024-09-19T07:30:46.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.709+0000 7f21bf05d640 1 -- 172.21.15.83:0/2507383890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21b80c96a0 con 0x7f21a40039f0 2024-09-19T07:30:46.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21b57fa640 1 -- 172.21.15.83:0/2507383890 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7f21a80f0b50 con 0x7f21a40039f0 2024-09-19T07:30:46.708 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:46.708 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:46.709 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:46.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21bf05d640 1 -- 172.21.15.83:0/2507383890 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21980678c0 msgr2=0x7f2198069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:46.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21bf05d640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21980678c0 0x7f2198069d80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f21b80c21c0 tx=0x7f21a0008040 comp rx=0 tx=0).stop 2024-09-19T07:30:46.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21bf05d640 1 -- 172.21.15.83:0/2507383890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21a40039f0 msgr2=0x7f21b80cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:46.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21bf05d640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21a40039f0 0x7f21b80cb9a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f21a801c250 tx=0x7f21a4005840 comp rx=0 tx=0).stop 2024-09-19T07:30:46.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21bf05d640 1 -- 172.21.15.83:0/2507383890 shutdown_connections 2024-09-19T07:30:46.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21bf05d640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f21980678c0 0x7f2198069d80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:46.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21bf05d640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f21b80c0640 0x7f21b814be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:46.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21bf05d640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21b8154e10 0x7f21b81f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:46.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21bf05d640 1 --2- 172.21.15.83:0/2507383890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21a40039f0 0x7f21b80cb9a0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:46.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21bf05d640 1 -- 172.21.15.83:0/2507383890 >> 172.21.15.83:0/2507383890 conn(0x7f21b806bc80 msgr2=0x7f21b8152980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:46.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21bf05d640 1 -- 172.21.15.83:0/2507383890 shutdown_connections 2024-09-19T07:30:46.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:46.713+0000 7f21bf05d640 1 -- 172.21.15.83:0/2507383890 wait complete. 2024-09-19T07:30:46.726 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:30:47.352 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:47.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.485+0000 7f48f18a4640 1 Processor -- start 2024-09-19T07:30:47.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.485+0000 7f48f18a4640 1 -- start start 2024-09-19T07:30:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48f18a4640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f48ec0c28e0 0x7f48ec1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48f18a4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48ec0a9e60 con 0x7f48ec06dea0 2024-09-19T07:30:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48f18a4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48ec0aa140 con 0x7f48ec071100 2024-09-19T07:30:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48f18a4640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f48ec071b50 con 0x7f48ec0c28e0 2024-09-19T07:30:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48eb7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f48ec06dea0 0x7f48ec06e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57230/0 (socket says 172.21.15.83:57230) 2024-09-19T07:30:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48eb7fe640 1 -- 172.21.15.83:0/1882844307 learned_addr learned my addr 172.21.15.83:0/1882844307 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:30:47.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48ea7fc640 1 --2- 172.21.15.83:0/1882844307 >> v2:172.21.15.184:3300/0 conn(0x7f48ec0c28e0 0x7f48ec1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:47.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2753361970 0 0) 0x7f48ec0a9e60 con 0x7f48ec06dea0 2024-09-19T07:30:47.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48d4003530 con 0x7f48ec06dea0 2024-09-19T07:30:47.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1360147039 0 0) 0x7f48d4003530 con 0x7f48ec06dea0 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 >> v2:172.21.15.184:3300/0 conn(0x7f48ec0c28e0 msgr2=0x7f48ec1545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 --2- 172.21.15.83:0/1882844307 >> v2:172.21.15.184:3300/0 conn(0x7f48ec0c28e0 0x7f48ec1545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 >> v1:172.21.15.184:6789/0 conn(0x7f48ec071100 legacy=0x7f48ec150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ec0c2240 con 0x7f48ec06dea0 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f48e4001a00 con 0x7f48ec06dea0 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 >> v1:172.21.15.83:6789/0 conn(0x7f48ec06dea0 legacy=0x7f48ec06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 --2- 172.21.15.83:0/1882844307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48d40039c0 0x7f48d4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 --2- 172.21.15.83:0/1882844307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f48d4004810 0x7f48d4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 --2- 172.21.15.83:0/1882844307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48d40051d0 0x7f48d400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f48ec071b50 con 0x7f48d40039c0 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f48d400c170 con 0x7f48d40051d0 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f48d400c370 con 0x7f48d4004810 2024-09-19T07:30:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48eaffd640 1 --2- 172.21.15.83:0/1882844307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48d40039c0 0x7f48d4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:47.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48ea7fc640 1 --2- 172.21.15.83:0/1882844307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f48d4004810 0x7f48d4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:47.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48eb7fe640 1 --2- 172.21.15.83:0/1882844307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48d40051d0 0x7f48d400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:47.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48eaffd640 1 -- 172.21.15.83:0/1882844307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f48d4004810 msgr2=0x7f48d4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:47.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48eaffd640 1 --2- 172.21.15.83:0/1882844307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f48d4004810 0x7f48d4004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:47.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48eaffd640 1 -- 172.21.15.83:0/1882844307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48d40051d0 msgr2=0x7f48d400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:47.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48eaffd640 1 --2- 172.21.15.83:0/1882844307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48d40051d0 0x7f48d400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:47.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48eaffd640 1 -- 172.21.15.83:0/1882844307 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d4003530 con 0x7f48d40039c0 2024-09-19T07:30:47.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48eb7fe640 1 --2- 172.21.15.83:0/1882844307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48d40051d0 0x7f48d400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:47.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48eb7fe640 1 -- 172.21.15.83:0/1882844307 reap_dead start 2024-09-19T07:30:47.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.489+0000 7f48eaffd640 1 --2- 172.21.15.83:0/1882844307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48d40039c0 0x7f48d4003de0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f48dc001780 tx=0x7f48dc01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a77b41ee7c857f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:47.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc0013a0 con 0x7f48d40039c0 2024-09-19T07:30:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48dc0052f0 con 0x7f48d40039c0 2024-09-19T07:30:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48e9ffb640 1 -- 172.21.15.83:0/1882844307 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc003840 con 0x7f48d40039c0 2024-09-19T07:30:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 -- 172.21.15.83:0/1882844307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48d40039c0 msgr2=0x7f48d4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 --2- 172.21.15.83:0/1882844307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48d40039c0 0x7f48d4003de0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f48dc001780 tx=0x7f48dc01c400 comp rx=0 tx=0).stop 2024-09-19T07:30:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 -- 172.21.15.83:0/1882844307 shutdown_connections 2024-09-19T07:30:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 --2- 172.21.15.83:0/1882844307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48d40039c0 0x7f48d4003de0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 -- 172.21.15.83:0/1882844307 >> 172.21.15.83:0/1882844307 conn(0x7f48ec06bc80 msgr2=0x7f48ec06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 -- 172.21.15.83:0/1882844307 shutdown_connections 2024-09-19T07:30:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 -- 172.21.15.83:0/1882844307 wait complete. 2024-09-19T07:30:47.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 Processor -- start 2024-09-19T07:30:47.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 -- start start 2024-09-19T07:30:47.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f48d40039c0 0x7f48ec1e2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:47.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48ec157460 0x7f48ec1df130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:47.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48ec1df670 0x7f48ec1e1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:47.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f48ec06e280 con 0x7f48ec1df670 2024-09-19T07:30:47.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f48ec0716f0 con 0x7f48ec157460 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f48ec06e400 con 0x7f48d40039c0 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48eb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48ec1df670 0x7f48ec1e1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48eb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48ec1df670 0x7f48ec1e1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46614/0 (socket says 172.21.15.83:46614) 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48eb7fe640 1 -- 172.21.15.83:0/3651124054 learned_addr learned my addr 172.21.15.83:0/3651124054 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48eaffd640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f48d40039c0 0x7f48ec1e2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48ea7fc640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48ec157460 0x7f48ec1df130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48eb7fe640 1 -- 172.21.15.83:0/3651124054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f48d40039c0 msgr2=0x7f48ec1e2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48eb7fe640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f48d40039c0 0x7f48ec1e2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48eb7fe640 1 -- 172.21.15.83:0/3651124054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48ec157460 msgr2=0x7f48ec1df130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48eb7fe640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48ec157460 0x7f48ec1df130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48eb7fe640 1 -- 172.21.15.83:0/3651124054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ec0ab480 con 0x7f48ec1df670 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48eb7fe640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48ec1df670 0x7f48ec1e1480 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f48ec1545b0 tx=0x7f48e400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:47.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48d3fff640 1 -- 172.21.15.83:0/3651124054 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e4004d70 con 0x7f48ec1df670 2024-09-19T07:30:47.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48d3fff640 1 -- 172.21.15.83:0/3651124054 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48e400bb20 con 0x7f48ec1df670 2024-09-19T07:30:47.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 -- 172.21.15.83:0/3651124054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48ec0ac560 con 0x7f48ec1df670 2024-09-19T07:30:47.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48d3fff640 1 -- 172.21.15.83:0/3651124054 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e4007070 con 0x7f48ec1df670 2024-09-19T07:30:47.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.493+0000 7f48f18a4640 1 -- 172.21.15.83:0/3651124054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48ec14d0d0 con 0x7f48ec1df670 2024-09-19T07:30:47.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.497+0000 7f48f18a4640 1 -- 172.21.15.83:0/3651124054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48ec06e790 con 0x7f48ec1df670 2024-09-19T07:30:47.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.497+0000 7f48d3fff640 1 -- 172.21.15.83:0/3651124054 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f48e401b020 con 0x7f48ec1df670 2024-09-19T07:30:47.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.497+0000 7f48d3fff640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f48d80678f0 0x7f48d8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:47.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.497+0000 7f48d3fff640 1 -- 172.21.15.83:0/3651124054 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7927+0+0 (secure 0 0 0) 0x7f48e40865c0 con 0x7f48ec1df670 2024-09-19T07:30:47.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.497+0000 7f48eaffd640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f48d80678f0 0x7f48d8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:47.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.497+0000 7f48eaffd640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f48d80678f0 0x7f48d8069db0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f48dc001750 tx=0x7f48dc0011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:47.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.501+0000 7f48d3fff640 1 -- 172.21.15.83:0/3651124054 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f48e40568e0 con 0x7f48ec1df670 2024-09-19T07:30:47.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.681+0000 7f48f18a4640 1 -- 172.21.15.83:0/3651124054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48ec0c4f30 con 0x7f48ec1df670 2024-09-19T07:30:47.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48d3fff640 1 -- 172.21.15.83:0/3651124054 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6538 (secure 0 0 0) 0x7f48e40561e0 con 0x7f48ec1df670 2024-09-19T07:30:47.680 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:47.680 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":89,"btime":"2024-09-19T07:29:57:285334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6267,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:29:51.259398+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6159":{"gid":6159,"name":"a","rank":0,"incarnation":85,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6835/2004814399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2004814399},{"type":"v1","addr":"172.21.15.83:6835","nonce":2004814399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6159,"qdb_cluster":[6159]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:28:41.296935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5476":{"gid":5476,"name":"b","rank":0,"incarnation":80,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6835/3841282254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3841282254},{"type":"v1","addr":"172.21.15.184:6835","nonce":3841282254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5476,"qdb_cluster":[5476]},"id":2}]} 2024-09-19T07:30:47.680 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 89 2024-09-19T07:30:47.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48f18a4640 1 -- 172.21.15.83:0/3651124054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f48d80678f0 msgr2=0x7f48d8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:47.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48f18a4640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f48d80678f0 0x7f48d8069db0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f48dc001750 tx=0x7f48dc0011a0 comp rx=0 tx=0).stop 2024-09-19T07:30:47.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48f18a4640 1 -- 172.21.15.83:0/3651124054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48ec1df670 msgr2=0x7f48ec1e1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:47.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48f18a4640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48ec1df670 0x7f48ec1e1480 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f48ec1545b0 tx=0x7f48e400b4a0 comp rx=0 tx=0).stop 2024-09-19T07:30:47.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48f18a4640 1 -- 172.21.15.83:0/3651124054 shutdown_connections 2024-09-19T07:30:47.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48f18a4640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f48d80678f0 0x7f48d8069db0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:47.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48f18a4640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48ec1df670 0x7f48ec1e1480 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:47.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48f18a4640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48ec157460 0x7f48ec1df130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:47.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48f18a4640 1 --2- 172.21.15.83:0/3651124054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f48d40039c0 0x7f48ec1e2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:47.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48f18a4640 1 -- 172.21.15.83:0/3651124054 >> 172.21.15.83:0/3651124054 conn(0x7f48ec06bc80 msgr2=0x7f48ec06c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:47.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48f18a4640 1 -- 172.21.15.83:0/3651124054 shutdown_connections 2024-09-19T07:30:47.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:47.685+0000 7f48f18a4640 1 -- 172.21.15.83:0/3651124054 wait complete. 2024-09-19T07:30:47.699 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:30:48.727 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:48.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.861+0000 7f2c99489640 1 Processor -- start 2024-09-19T07:30:48.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.861+0000 7f2c99489640 1 -- start start 2024-09-19T07:30:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c99489640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c941545e0 0x7f2c9406e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c99489640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c94154a70 con 0x7f2c940c28e0 2024-09-19T07:30:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c99489640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c940a9e60 con 0x7f2c94071100 2024-09-19T07:30:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c99489640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2c94071b50 con 0x7f2c941545e0 2024-09-19T07:30:48.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c927fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2c940c28e0 0x7f2c9406dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57242/0 (socket says 172.21.15.83:57242) 2024-09-19T07:30:48.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c927fc640 1 -- 172.21.15.83:0/2851568120 learned_addr learned my addr 172.21.15.83:0/2851568120 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:30:48.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 917908085 0 0) 0x7f2c94154a70 con 0x7f2c940c28e0 2024-09-19T07:30:48.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c80003530 con 0x7f2c940c28e0 2024-09-19T07:30:48.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c937fe640 1 --2- 172.21.15.83:0/2851568120 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c941545e0 0x7f2c9406e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:48.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2619538621 0 0) 0x7f2c80003530 con 0x7f2c940c28e0 2024-09-19T07:30:48.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c941545e0 msgr2=0x7f2c9406e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:48.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 --2- 172.21.15.83:0/2851568120 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c941545e0 0x7f2c9406e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:48.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 >> v1:172.21.15.184:6789/0 conn(0x7f2c94071100 legacy=0x7f2c94150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:48.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c940aa140 con 0x7f2c940c28e0 2024-09-19T07:30:48.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f2c7c001680 con 0x7f2c940c28e0 2024-09-19T07:30:48.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 >> v1:172.21.15.83:6789/0 conn(0x7f2c940c28e0 legacy=0x7f2c9406dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:48.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 --2- 172.21.15.83:0/2851568120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c800039f0 0x7f2c80003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 --2- 172.21.15.83:0/2851568120 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c941545e0 0x7f2c80004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 --2- 172.21.15.83:0/2851568120 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c80004d50 0x7f2c800051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2c94071b50 con 0x7f2c800039f0 2024-09-19T07:30:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8000bb30 con 0x7f2c80004d50 2024-09-19T07:30:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2c8000bd30 con 0x7f2c941545e0 2024-09-19T07:30:48.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c92ffd640 1 --2- 172.21.15.83:0/2851568120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c800039f0 0x7f2c80003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c927fc640 1 --2- 172.21.15.83:0/2851568120 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c941545e0 0x7f2c80004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c937fe640 1 --2- 172.21.15.83:0/2851568120 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c80004d50 0x7f2c800051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c92ffd640 1 -- 172.21.15.83:0/2851568120 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c941545e0 msgr2=0x7f2c80004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c92ffd640 1 --2- 172.21.15.83:0/2851568120 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c941545e0 0x7f2c80004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c92ffd640 1 -- 172.21.15.83:0/2851568120 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c80004d50 msgr2=0x7f2c800051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:48.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c92ffd640 1 --2- 172.21.15.83:0/2851568120 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c80004d50 0x7f2c800051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.865+0000 7f2c92ffd640 1 -- 172.21.15.83:0/2851568120 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c80003530 con 0x7f2c800039f0 2024-09-19T07:30:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c92ffd640 1 --2- 172.21.15.83:0/2851568120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c800039f0 0x7f2c80003e10 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f2c84001e70 tx=0x7f2c8401c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa235e9667dea59 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c840af070 con 0x7f2c800039f0 2024-09-19T07:30:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c840a6040 con 0x7f2c800039f0 2024-09-19T07:30:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c91ffb640 1 -- 172.21.15.83:0/2851568120 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c84001a90 con 0x7f2c800039f0 2024-09-19T07:30:48.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 -- 172.21.15.83:0/2851568120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c800039f0 msgr2=0x7f2c80003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:48.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 --2- 172.21.15.83:0/2851568120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c800039f0 0x7f2c80003e10 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f2c84001e70 tx=0x7f2c8401c6e0 comp rx=0 tx=0).stop 2024-09-19T07:30:48.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c937fe640 1 -- 172.21.15.83:0/2851568120 reap_dead start 2024-09-19T07:30:48.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 -- 172.21.15.83:0/2851568120 shutdown_connections 2024-09-19T07:30:48.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 -- 172.21.15.83:0/2851568120 >> 172.21.15.83:0/2851568120 conn(0x7f2c9406bc80 msgr2=0x7f2c9406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:48.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 -- 172.21.15.83:0/2851568120 shutdown_connections 2024-09-19T07:30:48.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 -- 172.21.15.83:0/2851568120 wait complete. 2024-09-19T07:30:48.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 Processor -- start 2024-09-19T07:30:48.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 -- start start 2024-09-19T07:30:48.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c800039f0 0x7f2c94157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:48.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c941efb50 0x7f2c941effd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:48.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c940c7410 0x7f2c940c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:48.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9406eb40 con 0x7f2c941efb50 2024-09-19T07:30:48.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9406e9c0 con 0x7f2c800039f0 2024-09-19T07:30:48.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c99489640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2c940714e0 con 0x7f2c940c7410 2024-09-19T07:30:48.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c927fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c941efb50 0x7f2c941effd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:48.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c927fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c941efb50 0x7f2c941effd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46626/0 (socket says 172.21.15.83:46626) 2024-09-19T07:30:48.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c927fc640 1 -- 172.21.15.83:0/2209422651 learned_addr learned my addr 172.21.15.83:0/2209422651 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:48.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c92ffd640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c800039f0 0x7f2c94157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:48.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c937fe640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c940c7410 0x7f2c940c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:48.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c927fc640 1 -- 172.21.15.83:0/2209422651 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c940c7410 msgr2=0x7f2c940c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:48.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c927fc640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c940c7410 0x7f2c940c7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:48.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c927fc640 1 -- 172.21.15.83:0/2209422651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c800039f0 msgr2=0x7f2c94157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:48.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c927fc640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c800039f0 0x7f2c94157440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:48.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c927fc640 1 -- 172.21.15.83:0/2209422651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c941487e0 con 0x7f2c941efb50 2024-09-19T07:30:48.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.869+0000 7f2c937fe640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c940c7410 0x7f2c940c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:48.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.873+0000 7f2c927fc640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c941efb50 0x7f2c941effd0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c007340 tx=0x7f2c7c007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:48.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.873+0000 7f2c8b7fe640 1 -- 172.21.15.83:0/2209422651 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c00f990 con 0x7f2c941efb50 2024-09-19T07:30:48.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.873+0000 7f2c8b7fe640 1 -- 172.21.15.83:0/2209422651 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c7c018070 con 0x7f2c941efb50 2024-09-19T07:30:48.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.873+0000 7f2c99489640 1 -- 172.21.15.83:0/2209422651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c94148ac0 con 0x7f2c941efb50 2024-09-19T07:30:48.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.873+0000 7f2c8b7fe640 1 -- 172.21.15.83:0/2209422651 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c00fd70 con 0x7f2c941efb50 2024-09-19T07:30:48.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.873+0000 7f2c99489640 1 -- 172.21.15.83:0/2209422651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c940735d0 con 0x7f2c941efb50 2024-09-19T07:30:48.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.873+0000 7f2c99489640 1 -- 172.21.15.83:0/2209422651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c94150f40 con 0x7f2c941efb50 2024-09-19T07:30:48.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.873+0000 7f2c8b7fe640 1 -- 172.21.15.83:0/2209422651 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2c7c01d460 con 0x7f2c941efb50 2024-09-19T07:30:48.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.873+0000 7f2c8b7fe640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2c740678c0 0x7f2c74069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:48.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.873+0000 7f2c92ffd640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2c740678c0 0x7f2c74069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:48.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.873+0000 7f2c8b7fe640 1 -- 172.21.15.83:0/2209422651 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f2c7c087100 con 0x7f2c941efb50 2024-09-19T07:30:48.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.873+0000 7f2c92ffd640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2c740678c0 0x7f2c74069d80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2c840ac000 tx=0x7f2c84001520 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:48.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:48.877+0000 7f2c8b7fe640 1 -- 172.21.15.83:0/2209422651 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2c7c057450 con 0x7f2c941efb50 2024-09-19T07:30:49.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.073+0000 7f2c99489640 1 -- 172.21.15.83:0/2209422651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c940c4f30 con 0x7f2c941efb50 2024-09-19T07:30:49.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c8b7fe640 1 -- 172.21.15.83:0/2209422651 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+4871 (secure 0 0 0) 0x7f2c7c056d50 con 0x7f2c941efb50 2024-09-19T07:30:49.073 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":90,"btime":"2024-09-19T07:30:48:076693+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:48.076684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:48.076685+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:30:49.073 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 90 2024-09-19T07:30:49.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c99489640 1 -- 172.21.15.83:0/2209422651 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2c740678c0 msgr2=0x7f2c74069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c99489640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2c740678c0 0x7f2c74069d80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2c840ac000 tx=0x7f2c84001520 comp rx=0 tx=0).stop 2024-09-19T07:30:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c99489640 1 -- 172.21.15.83:0/2209422651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c941efb50 msgr2=0x7f2c941effd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c99489640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c941efb50 0x7f2c941effd0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c007340 tx=0x7f2c7c007800 comp rx=0 tx=0).stop 2024-09-19T07:30:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c99489640 1 -- 172.21.15.83:0/2209422651 shutdown_connections 2024-09-19T07:30:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c99489640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2c740678c0 0x7f2c74069d80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c99489640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2c940c7410 0x7f2c940c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c99489640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c941efb50 0x7f2c941effd0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c99489640 1 --2- 172.21.15.83:0/2209422651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2c800039f0 0x7f2c94157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c99489640 1 -- 172.21.15.83:0/2209422651 >> 172.21.15.83:0/2209422651 conn(0x7f2c9406bc80 msgr2=0x7f2c94150200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:49.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c99489640 1 -- 172.21.15.83:0/2209422651 shutdown_connections 2024-09-19T07:30:49.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.077+0000 7f2c99489640 1 -- 172.21.15.83:0/2209422651 wait complete. 2024-09-19T07:30:49.094 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-09-19T07:30:49.467 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:30:49.467 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:49.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc908a10640 1 Processor -- start 2024-09-19T07:30:49.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc908a10640 1 -- start start 2024-09-19T07:30:49.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc908a10640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc9040c28e0 0x7fc9041545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:49.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc908a10640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc90406dea0 0x7fc90406e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:49.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc908a10640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9040a9e60 con 0x7fc904071100 2024-09-19T07:30:49.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc908a10640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc904071b50 con 0x7fc90406dea0 2024-09-19T07:30:49.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc908a10640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc9040c1ff0 con 0x7fc9040c28e0 2024-09-19T07:30:49.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc902d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc90406dea0 0x7fc90406e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:49.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc902d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc90406dea0 0x7fc90406e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46642/0 (socket says 172.21.15.83:46642) 2024-09-19T07:30:49.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc902d9d640 1 -- 172.21.15.83:0/3139507996 learned_addr learned my addr 172.21.15.83:0/3139507996 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:49.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc901d9b640 1 --2- 172.21.15.83:0/3139507996 >> v2:172.21.15.184:3300/0 conn(0x7fc9040c28e0 0x7fc9041545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:49.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc902d9d640 1 -- 172.21.15.83:0/3139507996 >> v2:172.21.15.184:3300/0 conn(0x7fc9040c28e0 msgr2=0x7fc9041545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:49.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc902d9d640 1 --2- 172.21.15.83:0/3139507996 >> v2:172.21.15.184:3300/0 conn(0x7fc9040c28e0 0x7fc9041545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc902d9d640 1 -- 172.21.15.83:0/3139507996 >> v1:172.21.15.184:6789/0 conn(0x7fc904071100 legacy=0x7fc904150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:49.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.601+0000 7fc902d9d640 1 -- 172.21.15.83:0/3139507996 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9040aa140 con 0x7fc90406dea0 2024-09-19T07:30:49.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc902d9d640 1 --2- 172.21.15.83:0/3139507996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc90406dea0 0x7fc90406e330 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fc9040c2890 tx=0x7fc8fc01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f33da59ede4509 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:49.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc90159a640 1 -- 172.21.15.83:0/3139507996 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc0a5710 con 0x7fc90406dea0 2024-09-19T07:30:49.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc90159a640 1 -- 172.21.15.83:0/3139507996 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8fc0a58b0 con 0x7fc90406dea0 2024-09-19T07:30:49.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc90159a640 1 -- 172.21.15.83:0/3139507996 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc0a5c80 con 0x7fc90406dea0 2024-09-19T07:30:49.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 -- 172.21.15.83:0/3139507996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc90406dea0 msgr2=0x7fc90406e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:49.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 --2- 172.21.15.83:0/3139507996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc90406dea0 0x7fc90406e330 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fc9040c2890 tx=0x7fc8fc01c960 comp rx=0 tx=0).stop 2024-09-19T07:30:49.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 -- 172.21.15.83:0/3139507996 shutdown_connections 2024-09-19T07:30:49.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 --2- 172.21.15.83:0/3139507996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc90406dea0 0x7fc90406e330 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 --2- 172.21.15.83:0/3139507996 >> v2:172.21.15.184:3300/0 conn(0x7fc9040c28e0 0x7fc9041545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 -- 172.21.15.83:0/3139507996 >> 172.21.15.83:0/3139507996 conn(0x7fc90406bc80 msgr2=0x7fc90406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:49.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 -- 172.21.15.83:0/3139507996 shutdown_connections 2024-09-19T07:30:49.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 -- 172.21.15.83:0/3139507996 wait complete. 2024-09-19T07:30:49.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 Processor -- start 2024-09-19T07:30:49.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 -- start start 2024-09-19T07:30:49.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc90406dea0 0x7fc904146b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:49.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc904071100 0x7fc90415c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:49.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9040c28e0 0x7fc9041f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:49.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc904071660 con 0x7fc904071100 2024-09-19T07:30:49.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc9040714e0 con 0x7fc9040c28e0 2024-09-19T07:30:49.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc908a10640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc904160c60 con 0x7fc90406dea0 2024-09-19T07:30:49.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc901d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc904071100 0x7fc90415c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:49.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc901d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc904071100 0x7fc90415c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46646/0 (socket says 172.21.15.83:46646) 2024-09-19T07:30:49.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc901d9b640 1 -- 172.21.15.83:0/3258555731 learned_addr learned my addr 172.21.15.83:0/3258555731 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:49.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc902d9d640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9040c28e0 0x7fc9041f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc90259c640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc90406dea0 0x7fc904146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc901d9b640 1 -- 172.21.15.83:0/3258555731 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc90406dea0 msgr2=0x7fc904146b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc901d9b640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc90406dea0 0x7fc904146b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc901d9b640 1 -- 172.21.15.83:0/3258555731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9040c28e0 msgr2=0x7fc9041f1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc901d9b640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9040c28e0 0x7fc9041f1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc901d9b640 1 -- 172.21.15.83:0/3258555731 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9041472d0 con 0x7fc904071100 2024-09-19T07:30:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.605+0000 7fc901d9b640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc904071100 0x7fc90415c870 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400d5f0 tx=0x7fc8f400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.609+0000 7fc8f2ffd640 1 -- 172.21.15.83:0/3258555731 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f40125f0 con 0x7fc904071100 2024-09-19T07:30:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.609+0000 7fc8f2ffd640 1 -- 172.21.15.83:0/3258555731 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8f4019070 con 0x7fc904071100 2024-09-19T07:30:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.609+0000 7fc8f2ffd640 1 -- 172.21.15.83:0/3258555731 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f40129d0 con 0x7fc904071100 2024-09-19T07:30:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.609+0000 7fc908a10640 1 -- 172.21.15.83:0/3258555731 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9040c5490 con 0x7fc904071100 2024-09-19T07:30:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.609+0000 7fc908a10640 1 -- 172.21.15.83:0/3258555731 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc90415cdb0 con 0x7fc904071100 2024-09-19T07:30:49.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.609+0000 7fc908a10640 1 -- 172.21.15.83:0/3258555731 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc90406e330 con 0x7fc904071100 2024-09-19T07:30:49.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.609+0000 7fc8f2ffd640 1 -- 172.21.15.83:0/3258555731 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc8f4012b70 con 0x7fc904071100 2024-09-19T07:30:49.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.609+0000 7fc8f2ffd640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8e40678c0 0x7fc8e4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:49.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.609+0000 7fc90259c640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8e40678c0 0x7fc8e4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:49.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.609+0000 7fc8f2ffd640 1 -- 172.21.15.83:0/3258555731 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7fc8f4087420 con 0x7fc904071100 2024-09-19T07:30:49.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.609+0000 7fc90259c640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8e40678c0 0x7fc8e4069d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8001650 tx=0x7fc8f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:49.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.613+0000 7fc8f2ffd640 1 -- 172.21.15.83:0/3258555731 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc8f4057770 con 0x7fc904071100 2024-09-19T07:30:49.700 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:49.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 -- 172.21.15.83:0/3258555731 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9041548d0 con 0x7fc904071100 2024-09-19T07:30:49.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc8f2ffd640 1 -- 172.21.15.83:0/3258555731 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+4880 (secure 0 0 0) 0x7fc8f4057070 con 0x7fc904071100 2024-09-19T07:30:49.809 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:49.809 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":91,"btime":"2024-09-19T07:30:49:079314+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:48.375911+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:reconnect","state_seq":127,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:48.409430+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:30:49.809 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 91 2024-09-19T07:30:49.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 -- 172.21.15.83:0/3258555731 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8e40678c0 msgr2=0x7fc8e4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:49.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8e40678c0 0x7fc8e4069d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8001650 tx=0x7fc8f8009040 comp rx=0 tx=0).stop 2024-09-19T07:30:49.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 -- 172.21.15.83:0/3258555731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc904071100 msgr2=0x7fc90415c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:49.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc904071100 0x7fc90415c870 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400d5f0 tx=0x7fc8f400dab0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 -- 172.21.15.83:0/3258555731 shutdown_connections 2024-09-19T07:30:49.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8e40678c0 0x7fc8e4069d80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9040c28e0 0x7fc9041f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc904071100 0x7fc90415c870 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 --2- 172.21.15.83:0/3258555731 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc90406dea0 0x7fc904146b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 -- 172.21.15.83:0/3258555731 >> 172.21.15.83:0/3258555731 conn(0x7fc90406bc80 msgr2=0x7fc9040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:49.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 -- 172.21.15.83:0/3258555731 shutdown_connections 2024-09-19T07:30:49.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.813+0000 7fc908a10640 1 -- 172.21.15.83:0/3258555731 wait complete. 2024-09-19T07:30:49.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.821+0000 7f33d29c4640 1 Processor -- start 2024-09-19T07:30:49.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d29c4640 1 -- start start 2024-09-19T07:30:49.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d29c4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33cc14d9f0 0x7f33cc06de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:49.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d29c4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f33cc154360 0x7f33cc0c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:49.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d29c4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33cc14ddd0 con 0x7f33cc0c7510 2024-09-19T07:30:49.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d29c4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f33cc0c8830 con 0x7f33cc154360 2024-09-19T07:30:49.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d29c4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f33cc06fdf0 con 0x7f33cc14d9f0 2024-09-19T07:30:49.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d0f61640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f33cc154360 0x7f33cc0c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:49.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d0f61640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f33cc154360 0x7f33cc0c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46648/0 (socket says 172.21.15.83:46648) 2024-09-19T07:30:49.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d0f61640 1 -- 172.21.15.83:0/320106290 learned_addr learned my addr 172.21.15.83:0/320106290 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:49.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d0f61640 1 -- 172.21.15.83:0/320106290 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33cc14d9f0 msgr2=0x7f33cc06de90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:30:49.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d0f61640 1 --2- 172.21.15.83:0/320106290 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33cc14d9f0 0x7f33cc06de90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d0f61640 1 -- 172.21.15.83:0/320106290 >> v1:172.21.15.184:6789/0 conn(0x7f33cc0c7510 legacy=0x7f33cc0c8cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:30:49.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.825+0000 7f33d0f61640 1 -- 172.21.15.83:0/320106290 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33cc0c7f00 con 0x7f33cc154360 2024-09-19T07:30:49.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33d0f61640 1 --2- 172.21.15.83:0/320106290 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33cc154360 0x7f33cc0c4010 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009e50 tx=0x7f33c401c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50880baae58b489 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:49.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33caffd640 1 -- 172.21.15.83:0/320106290 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c40a5600 con 0x7f33cc154360 2024-09-19T07:30:49.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33caffd640 1 -- 172.21.15.83:0/320106290 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c40a57a0 con 0x7f33cc154360 2024-09-19T07:30:49.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33caffd640 1 -- 172.21.15.83:0/320106290 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c40a5b70 con 0x7f33cc154360 2024-09-19T07:30:49.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33d29c4640 1 -- 172.21.15.83:0/320106290 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33cc154360 msgr2=0x7f33cc0c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:49.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33d29c4640 1 --2- 172.21.15.83:0/320106290 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33cc154360 0x7f33cc0c4010 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009e50 tx=0x7f33c401c980 comp rx=0 tx=0).stop 2024-09-19T07:30:49.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33d29c4640 1 -- 172.21.15.83:0/320106290 shutdown_connections 2024-09-19T07:30:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33d29c4640 1 --2- 172.21.15.83:0/320106290 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33cc154360 0x7f33cc0c4010 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33d29c4640 1 --2- 172.21.15.83:0/320106290 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33cc14d9f0 0x7f33cc06de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33d29c4640 1 -- 172.21.15.83:0/320106290 >> 172.21.15.83:0/320106290 conn(0x7f33cc0c6110 msgr2=0x7f33cc0c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33d29c4640 1 -- 172.21.15.83:0/320106290 shutdown_connections 2024-09-19T07:30:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33d29c4640 1 -- 172.21.15.83:0/320106290 wait complete. 2024-09-19T07:30:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33d29c4640 1 Processor -- start 2024-09-19T07:30:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.829+0000 7f33d29c4640 1 -- start start 2024-09-19T07:30:49.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33d29c4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33cc0c7510 0x7f33cc155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:49.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33d29c4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33cc14d9f0 0x7f33cc14c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:49.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33d29c4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33cc154360 0x7f33cc145aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:49.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33d29c4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc160c00 con 0x7f33cc14d9f0 2024-09-19T07:30:49.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33d29c4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc160a80 con 0x7f33cc0c7510 2024-09-19T07:30:49.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33d29c4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f33cc160d80 con 0x7f33cc154360 2024-09-19T07:30:49.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33cb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33cc14d9f0 0x7f33cc14c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:49.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33cb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33cc14d9f0 0x7f33cc14c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46654/0 (socket says 172.21.15.83:46654) 2024-09-19T07:30:49.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33cb7fe640 1 -- 172.21.15.83:0/3369911993 learned_addr learned my addr 172.21.15.83:0/3369911993 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:49.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33cbfff640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33cc0c7510 0x7f33cc155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:49.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33cb7fe640 1 -- 172.21.15.83:0/3369911993 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33cc154360 msgr2=0x7f33cc145aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:30:49.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33cb7fe640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33cc154360 0x7f33cc145aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33cb7fe640 1 -- 172.21.15.83:0/3369911993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33cc0c7510 msgr2=0x7f33cc155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:49.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33cb7fe640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33cc0c7510 0x7f33cc155100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:49.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33cb7fe640 1 -- 172.21.15.83:0/3369911993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33cc155640 con 0x7f33cc14d9f0 2024-09-19T07:30:49.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33cb7fe640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33cc14d9f0 0x7f33cc14c5e0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f33c000b3e0 tx=0x7f33c000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:49.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33c8ff9640 1 -- 172.21.15.83:0/3369911993 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c0015020 con 0x7f33cc14d9f0 2024-09-19T07:30:49.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33c8ff9640 1 -- 172.21.15.83:0/3369911993 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c0011830 con 0x7f33cc14d9f0 2024-09-19T07:30:49.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33d29c4640 1 -- 172.21.15.83:0/3369911993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33cc155910 con 0x7f33cc14d9f0 2024-09-19T07:30:49.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33c8ff9640 1 -- 172.21.15.83:0/3369911993 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c0011c40 con 0x7f33cc14d9f0 2024-09-19T07:30:49.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33d29c4640 1 -- 172.21.15.83:0/3369911993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33cc1f0250 con 0x7f33cc14d9f0 2024-09-19T07:30:49.832 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:30:49:079314+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 91, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4652": { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 90, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 127 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:30:48.375911+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4652 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6267": { "addr": "172.21.15.83:6837/2581473241", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 2581473241, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 2581473241, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6267, "incarnation": 90, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:30:48.409430+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6267 } } } ], "standbys": [] } 2024-09-19T07:30:49.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33d29c4640 1 -- 172.21.15.83:0/3369911993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33cc06de90 con 0x7f33cc14d9f0 2024-09-19T07:30:49.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33c8ff9640 1 -- 172.21.15.83:0/3369911993 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f33c0018020 con 0x7f33cc14d9f0 2024-09-19T07:30:49.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33c8ff9640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f33b80678c0 0x7f33b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:49.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33cbfff640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f33b80678c0 0x7f33b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:49.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33c8ff9640 1 -- 172.21.15.83:0/3369911993 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f33c0086550 con 0x7f33cc14d9f0 2024-09-19T07:30:49.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.833+0000 7f33cbfff640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f33b80678c0 0x7f33b8069d80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00b810 tx=0x7f33bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:49.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:49.837+0000 7f33c8ff9640 1 -- 172.21.15.83:0/3369911993 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f33c001f020 con 0x7f33cc14d9f0 2024-09-19T07:30:50.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.057+0000 7f33d29c4640 1 -- 172.21.15.83:0/3369911993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33cc06faf0 con 0x7f33cc14d9f0 2024-09-19T07:30:50.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.057+0000 7f33c8ff9640 1 -- 172.21.15.83:0/3369911993 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+4880 (secure 0 0 0) 0x7f33c00172c0 con 0x7f33cc14d9f0 2024-09-19T07:30:50.054 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:50.054 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":91,"btime":"2024-09-19T07:30:49:079314+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:48.375911+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:reconnect","state_seq":127,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:48.409430+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:30:50.054 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 91 2024-09-19T07:30:50.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.061+0000 7f33d29c4640 1 -- 172.21.15.83:0/3369911993 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f33b80678c0 msgr2=0x7f33b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:50.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.061+0000 7f33d29c4640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f33b80678c0 0x7f33b8069d80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00b810 tx=0x7f33bc008040 comp rx=0 tx=0).stop 2024-09-19T07:30:50.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.061+0000 7f33d29c4640 1 -- 172.21.15.83:0/3369911993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33cc14d9f0 msgr2=0x7f33cc14c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:50.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.061+0000 7f33d29c4640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33cc14d9f0 0x7f33cc14c5e0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f33c000b3e0 tx=0x7f33c000b8a0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.061+0000 7f33d29c4640 1 -- 172.21.15.83:0/3369911993 shutdown_connections 2024-09-19T07:30:50.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.061+0000 7f33d29c4640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f33b80678c0 0x7f33b8069d80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.061+0000 7f33d29c4640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33cc154360 0x7f33cc145aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.061+0000 7f33d29c4640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33cc14d9f0 0x7f33cc14c5e0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.061+0000 7f33d29c4640 1 --2- 172.21.15.83:0/3369911993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33cc0c7510 0x7f33cc155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.061+0000 7f33d29c4640 1 -- 172.21.15.83:0/3369911993 >> 172.21.15.83:0/3369911993 conn(0x7f33cc0c6110 msgr2=0x7f33cc144d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:50.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.061+0000 7f33d29c4640 1 -- 172.21.15.83:0/3369911993 shutdown_connections 2024-09-19T07:30:50.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.061+0000 7f33d29c4640 1 -- 172.21.15.83:0/3369911993 wait complete. 2024-09-19T07:30:50.079 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-19T07:30:50.079 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:30:50.079 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:50.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.245+0000 7f7342124640 1 Processor -- start 2024-09-19T07:30:50.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.245+0000 7f7342124640 1 -- start start 2024-09-19T07:30:50.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f733c0c28e0 0x7f733c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:50.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f733c06dea0 0x7f733c06e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:50.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f733c0a9e60 con 0x7f733c071100 2024-09-19T07:30:50.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f733c071b50 con 0x7f733c0c28e0 2024-09-19T07:30:50.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f733c0c1ff0 con 0x7f733c06dea0 2024-09-19T07:30:50.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f733affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f733c0c28e0 0x7f733c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:50.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f733affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f733c0c28e0 0x7f733c1545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54972/0 (socket says 172.21.15.83:54972) 2024-09-19T07:30:50.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f733affd640 1 -- 172.21.15.83:0/3437688265 learned_addr learned my addr 172.21.15.83:0/3437688265 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:50.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f733affd640 1 -- 172.21.15.83:0/3437688265 >> v2:172.21.15.184:3300/0 conn(0x7f733c06dea0 msgr2=0x7f733c06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:50.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f733affd640 1 --2- 172.21.15.83:0/3437688265 >> v2:172.21.15.184:3300/0 conn(0x7f733c06dea0 0x7f733c06e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f733affd640 1 -- 172.21.15.83:0/3437688265 >> v1:172.21.15.83:6789/0 conn(0x7f733c071100 legacy=0x7f733c150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:50.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f733affd640 1 -- 172.21.15.83:0/3437688265 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733c0aa140 con 0x7f733c0c28e0 2024-09-19T07:30:50.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f733affd640 1 --2- 172.21.15.83:0/3437688265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733c0c28e0 0x7f733c1545e0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f733c0c4f00 tx=0x7f732c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2670c306d0659fd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:50.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f733a7fc640 1 -- 172.21.15.83:0/3437688265 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732c0b2070 con 0x7f733c0c28e0 2024-09-19T07:30:50.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f733a7fc640 1 -- 172.21.15.83:0/3437688265 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f732c0ad680 con 0x7f733c0c28e0 2024-09-19T07:30:50.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f733a7fc640 1 -- 172.21.15.83:0/3437688265 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732c0ada50 con 0x7f733c0c28e0 2024-09-19T07:30:50.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 -- 172.21.15.83:0/3437688265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733c0c28e0 msgr2=0x7f733c1545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:50.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 --2- 172.21.15.83:0/3437688265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733c0c28e0 0x7f733c1545e0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f733c0c4f00 tx=0x7f732c01c490 comp rx=0 tx=0).stop 2024-09-19T07:30:50.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 -- 172.21.15.83:0/3437688265 shutdown_connections 2024-09-19T07:30:50.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 --2- 172.21.15.83:0/3437688265 >> v2:172.21.15.184:3300/0 conn(0x7f733c06dea0 0x7f733c06e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 --2- 172.21.15.83:0/3437688265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733c0c28e0 0x7f733c1545e0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 -- 172.21.15.83:0/3437688265 >> 172.21.15.83:0/3437688265 conn(0x7f733c06bc80 msgr2=0x7f733c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:50.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 -- 172.21.15.83:0/3437688265 shutdown_connections 2024-09-19T07:30:50.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.249+0000 7f7342124640 1 -- 172.21.15.83:0/3437688265 wait complete. 2024-09-19T07:30:50.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7342124640 1 Processor -- start 2024-09-19T07:30:50.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7342124640 1 -- start start 2024-09-19T07:30:50.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7342124640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733c06dea0 0x7f733c15ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:50.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7342124640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f733c071100 0x7f733c1ed290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:50.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7342124640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f733c0c28e0 0x7f733c1ed7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7342124640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f733c071660 con 0x7f733c06dea0 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7342124640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f733c0714e0 con 0x7f733c071100 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7342124640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f733c160c60 con 0x7f733c0c28e0 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f733b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733c06dea0 0x7f733c15ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f733b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733c06dea0 0x7f733c15ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54978/0 (socket says 172.21.15.83:54978) 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f733b7fe640 1 -- 172.21.15.83:0/2129712388 learned_addr learned my addr 172.21.15.83:0/2129712388 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f733affd640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f733c071100 0x7f733c1ed290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f733bfff640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f733c0c28e0 0x7f733c1ed7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f733b7fe640 1 -- 172.21.15.83:0/2129712388 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f733c0c28e0 msgr2=0x7f733c1ed7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f733b7fe640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f733c0c28e0 0x7f733c1ed7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f733b7fe640 1 -- 172.21.15.83:0/2129712388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f733c071100 msgr2=0x7f733c1ed290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f733b7fe640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f733c071100 0x7f733c1ed290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f733b7fe640 1 -- 172.21.15.83:0/2129712388 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733c15c7a0 con 0x7f733c06dea0 2024-09-19T07:30:50.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f733b7fe640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733c06dea0 0x7f733c15ceb0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f733000c420 tx=0x7f733000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:50.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7323fff640 1 -- 172.21.15.83:0/2129712388 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73300052d0 con 0x7f733c06dea0 2024-09-19T07:30:50.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7323fff640 1 -- 172.21.15.83:0/2129712388 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7330005470 con 0x7f733c06dea0 2024-09-19T07:30:50.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7323fff640 1 -- 172.21.15.83:0/2129712388 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330007260 con 0x7f733c06dea0 2024-09-19T07:30:50.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7342124640 1 -- 172.21.15.83:0/2129712388 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f733c158da0 con 0x7f733c06dea0 2024-09-19T07:30:50.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7342124640 1 -- 172.21.15.83:0/2129712388 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f733c1587e0 con 0x7f733c06dea0 2024-09-19T07:30:50.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.253+0000 7f7342124640 1 -- 172.21.15.83:0/2129712388 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f733c06e330 con 0x7f733c06dea0 2024-09-19T07:30:50.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.257+0000 7f7323fff640 1 -- 172.21.15.83:0/2129712388 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7330007400 con 0x7f733c06dea0 2024-09-19T07:30:50.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.257+0000 7f7323fff640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f73280678c0 0x7f7328069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:50.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.257+0000 7f7323fff640 1 -- 172.21.15.83:0/2129712388 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f7330085aa0 con 0x7f733c06dea0 2024-09-19T07:30:50.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.257+0000 7f733affd640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f73280678c0 0x7f7328069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:50.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.257+0000 7f733affd640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f73280678c0 0x7f7328069d80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f732c0af000 tx=0x7f732c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:50.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.261+0000 7f7323fff640 1 -- 172.21.15.83:0/2129712388 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7330055d10 con 0x7f733c06dea0 2024-09-19T07:30:50.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.441+0000 7f7342124640 1 -- 172.21.15.83:0/2129712388 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f733c0c1dd0 con 0x7f733c06dea0 2024-09-19T07:30:50.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.441+0000 7f7323fff640 1 -- 172.21.15.83:0/2129712388 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+4874 (secure 0 0 0) 0x7f7330055610 con 0x7f733c06dea0 2024-09-19T07:30:50.439 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:50.439 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":92,"btime":"2024-09-19T07:30:50:082010+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:49.090197+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:rejoin","state_seq":128,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:49.089123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:30:50.439 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 92 2024-09-19T07:30:50.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.445+0000 7f7342124640 1 -- 172.21.15.83:0/2129712388 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f73280678c0 msgr2=0x7f7328069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:50.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.445+0000 7f7342124640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f73280678c0 0x7f7328069d80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f732c0af000 tx=0x7f732c008040 comp rx=0 tx=0).stop 2024-09-19T07:30:50.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.445+0000 7f7342124640 1 -- 172.21.15.83:0/2129712388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733c06dea0 msgr2=0x7f733c15ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:50.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.445+0000 7f7342124640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733c06dea0 0x7f733c15ceb0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f733000c420 tx=0x7f733000c8e0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.445+0000 7f7342124640 1 -- 172.21.15.83:0/2129712388 shutdown_connections 2024-09-19T07:30:50.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.445+0000 7f7342124640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f73280678c0 0x7f7328069d80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.445+0000 7f7342124640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f733c0c28e0 0x7f733c1ed7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.445+0000 7f7342124640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f733c071100 0x7f733c1ed290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.445+0000 7f7342124640 1 --2- 172.21.15.83:0/2129712388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733c06dea0 0x7f733c15ceb0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:50.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.445+0000 7f7342124640 1 -- 172.21.15.83:0/2129712388 >> 172.21.15.83:0/2129712388 conn(0x7f733c06bc80 msgr2=0x7f733c073030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:50.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.445+0000 7f7342124640 1 -- 172.21.15.83:0/2129712388 shutdown_connections 2024-09-19T07:30:50.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:50.445+0000 7f7342124640 1 -- 172.21.15.83:0/2129712388 wait complete. 2024-09-19T07:30:50.457 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:30:50:082010+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 92, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 92, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4652": { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 90, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 128 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:30:49.090197+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4652 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 92, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6267": { "addr": "172.21.15.83:6837/2581473241", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 2581473241, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 2581473241, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6267, "incarnation": 90, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:30:49.089123+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6267 } } } ], "standbys": [] } 2024-09-19T07:30:51.833 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:51.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9498223640 1 Processor -- start 2024-09-19T07:30:51.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9498223640 1 -- start start 2024-09-19T07:30:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9498223640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9490150f40 0x7f94900733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9498223640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94901539a0 0x7f9490153e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9498223640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94900c3aa0 con 0x7f94900adae0 2024-09-19T07:30:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9498223640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f94900c3850 con 0x7f9490150f40 2024-09-19T07:30:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9498223640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f94900c43d0 con 0x7f94901539a0 2024-09-19T07:30:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f94957be640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f94900adae0 0x7f949006f780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33732/0 (socket says 172.21.15.83:33732) 2024-09-19T07:30:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f94957be640 1 -- 172.21.15.83:0/2330302983 learned_addr learned my addr 172.21.15.83:0/2330302983 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:30:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f94967c0640 1 --2- 172.21.15.83:0/2330302983 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94901539a0 0x7f9490153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9495fbf640 1 --2- 172.21.15.83:0/2330302983 >> v2:172.21.15.184:3300/0 conn(0x7f9490150f40 0x7f94900733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1190071469 0 0) 0x7f94900c3aa0 con 0x7f94900adae0 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f947c003530 con 0x7f94900adae0 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2692682217 0 0) 0x7f947c003530 con 0x7f94900adae0 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94901539a0 msgr2=0x7f9490153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 --2- 172.21.15.83:0/2330302983 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94901539a0 0x7f9490153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 >> v2:172.21.15.184:3300/0 conn(0x7f9490150f40 msgr2=0x7f94900733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 --2- 172.21.15.83:0/2330302983 >> v2:172.21.15.184:3300/0 conn(0x7f9490150f40 0x7f94900733d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94900c3d80 con 0x7f94900adae0 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f9480001680 con 0x7f94900adae0 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 >> v1:172.21.15.83:6789/0 conn(0x7f94900adae0 legacy=0x7f949006f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 --2- 172.21.15.83:0/2330302983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f947c0039c0 0x7f947c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 --2- 172.21.15.83:0/2330302983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f947c004810 0x7f947c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 --2- 172.21.15.83:0/2330302983 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94901539a0 0x7f947c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f94900c3850 con 0x7f947c0039c0 2024-09-19T07:30:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f94900c43d0 con 0x7f947c004810 2024-09-19T07:30:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f947c00b9d0 con 0x7f94901539a0 2024-09-19T07:30:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9495fbf640 1 --2- 172.21.15.83:0/2330302983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f947c0039c0 0x7f947c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f94957be640 1 --2- 172.21.15.83:0/2330302983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f947c004810 0x7f947c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9495fbf640 1 -- 172.21.15.83:0/2330302983 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94901539a0 msgr2=0x7f947c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9495fbf640 1 --2- 172.21.15.83:0/2330302983 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94901539a0 0x7f947c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9495fbf640 1 -- 172.21.15.83:0/2330302983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f947c004810 msgr2=0x7f947c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9495fbf640 1 --2- 172.21.15.83:0/2330302983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f947c004810 0x7f947c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f9495fbf640 1 -- 172.21.15.83:0/2330302983 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947c003530 con 0x7f947c0039c0 2024-09-19T07:30:51.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.953+0000 7f94957be640 1 --2- 172.21.15.83:0/2330302983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f947c004810 0x7f947c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:30:51.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.957+0000 7f9495fbf640 1 --2- 172.21.15.83:0/2330302983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f947c0039c0 0x7f947c003de0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f948c009ae0 tx=0x7f948c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a006c709b9d35ee server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:51.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.957+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c0096b0 con 0x7f947c0039c0 2024-09-19T07:30:51.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.957+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f948c0a6430 con 0x7f947c0039c0 2024-09-19T07:30:51.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.957+0000 7f9494fbd640 1 -- 172.21.15.83:0/2330302983 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c0a6820 con 0x7f947c0039c0 2024-09-19T07:30:51.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.957+0000 7f9498223640 1 -- 172.21.15.83:0/2330302983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f947c0039c0 msgr2=0x7f947c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:51.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.957+0000 7f9498223640 1 --2- 172.21.15.83:0/2330302983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f947c0039c0 0x7f947c003de0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f948c009ae0 tx=0x7f948c01ca50 comp rx=0 tx=0).stop 2024-09-19T07:30:51.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.957+0000 7f94967c0640 1 -- 172.21.15.83:0/2330302983 reap_dead start 2024-09-19T07:30:51.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.957+0000 7f9498223640 1 -- 172.21.15.83:0/2330302983 shutdown_connections 2024-09-19T07:30:51.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.957+0000 7f9498223640 1 -- 172.21.15.83:0/2330302983 >> 172.21.15.83:0/2330302983 conn(0x7f94900c4940 msgr2=0x7f94900c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:51.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.957+0000 7f9498223640 1 -- 172.21.15.83:0/2330302983 shutdown_connections 2024-09-19T07:30:51.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.957+0000 7f9498223640 1 -- 172.21.15.83:0/2330302983 wait complete. 2024-09-19T07:30:51.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9498223640 1 Processor -- start 2024-09-19T07:30:51.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9498223640 1 -- start start 2024-09-19T07:30:51.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9498223640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f947c0039c0 0x7f94901effe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:51.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9498223640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9490150470 0x7f9490144740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9498223640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9490144c80 0x7f9490152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9498223640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9490073a90 con 0x7f947c0039c0 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9498223640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9490073910 con 0x7f9490150470 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9498223640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f949015e320 con 0x7f9490144c80 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9495fbf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f947c0039c0 0x7f94901effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9495fbf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f947c0039c0 0x7f94901effe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54982/0 (socket says 172.21.15.83:54982) 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9495fbf640 1 -- 172.21.15.83:0/3754080826 learned_addr learned my addr 172.21.15.83:0/3754080826 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f94957be640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9490150470 0x7f9490144740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9495fbf640 1 -- 172.21.15.83:0/3754080826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9490144c80 msgr2=0x7f9490152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9495fbf640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9490144c80 0x7f9490152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9495fbf640 1 -- 172.21.15.83:0/3754080826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9490150470 msgr2=0x7f9490144740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9495fbf640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9490150470 0x7f9490144740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9495fbf640 1 -- 172.21.15.83:0/3754080826 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949014b600 con 0x7f947c0039c0 2024-09-19T07:30:51.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9495fbf640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f947c0039c0 0x7f94901effe0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f948c01c880 tx=0x7f947c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:51.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f94867fc640 1 -- 172.21.15.83:0/3754080826 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c0c0070 con 0x7f947c0039c0 2024-09-19T07:30:51.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f94867fc640 1 -- 172.21.15.83:0/3754080826 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f948c00f000 con 0x7f947c0039c0 2024-09-19T07:30:51.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f94867fc640 1 -- 172.21.15.83:0/3754080826 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c0a6d40 con 0x7f947c0039c0 2024-09-19T07:30:51.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9498223640 1 -- 172.21.15.83:0/3754080826 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94901f08c0 con 0x7f947c0039c0 2024-09-19T07:30:51.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9498223640 1 -- 172.21.15.83:0/3754080826 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94900c5d70 con 0x7f947c0039c0 2024-09-19T07:30:51.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f9498223640 1 -- 172.21.15.83:0/3754080826 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94900733d0 con 0x7f947c0039c0 2024-09-19T07:30:51.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f94867fc640 1 -- 172.21.15.83:0/3754080826 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f948c0c3070 con 0x7f947c0039c0 2024-09-19T07:30:51.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.961+0000 7f94867fc640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94740678f0 0x7f9474069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:51.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.965+0000 7f94957be640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94740678f0 0x7f9474069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:51.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.965+0000 7f94867fc640 1 -- 172.21.15.83:0/3754080826 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f948c1201b0 con 0x7f947c0039c0 2024-09-19T07:30:51.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.965+0000 7f94957be640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94740678f0 0x7f9474069db0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9480001ec0 tx=0x7f9480009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:51.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:51.969+0000 7f94867fc640 1 -- 172.21.15.83:0/3754080826 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f948c0ac020 con 0x7f947c0039c0 2024-09-19T07:30:52.190 INFO:tasks.ceph.mds.d.smithi184.stderr:2024-09-19T07:30:52.193+0000 7f638ec10640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:30:52.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.205+0000 7f9498223640 1 -- 172.21.15.83:0/3754080826 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f949014e550 con 0x7f947c0039c0 2024-09-19T07:30:52.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.205+0000 7f94867fc640 1 -- 172.21.15.83:0/3754080826 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+4888 (secure 0 0 0) 0x7f948c0167c0 con 0x7f947c0039c0 2024-09-19T07:30:52.203 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:52.203 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":93,"btime":"2024-09-19T07:30:51:094252+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:30:52.204 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 93 2024-09-19T07:30:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.209+0000 7f9498223640 1 -- 172.21.15.83:0/3754080826 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94740678f0 msgr2=0x7f9474069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.209+0000 7f9498223640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94740678f0 0x7f9474069db0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9480001ec0 tx=0x7f9480009040 comp rx=0 tx=0).stop 2024-09-19T07:30:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.209+0000 7f9498223640 1 -- 172.21.15.83:0/3754080826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f947c0039c0 msgr2=0x7f94901effe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.209+0000 7f9498223640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f947c0039c0 0x7f94901effe0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f948c01c880 tx=0x7f947c005840 comp rx=0 tx=0).stop 2024-09-19T07:30:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.209+0000 7f9498223640 1 -- 172.21.15.83:0/3754080826 shutdown_connections 2024-09-19T07:30:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.209+0000 7f9498223640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94740678f0 0x7f9474069db0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.209+0000 7f9498223640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9490144c80 0x7f9490152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.209+0000 7f9498223640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9490150470 0x7f9490144740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.209+0000 7f9498223640 1 --2- 172.21.15.83:0/3754080826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f947c0039c0 0x7f94901effe0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.209+0000 7f9498223640 1 -- 172.21.15.83:0/3754080826 >> 172.21.15.83:0/3754080826 conn(0x7f94900c4940 msgr2=0x7f9490072430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:52.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.209+0000 7f9498223640 1 -- 172.21.15.83:0/3754080826 shutdown_connections 2024-09-19T07:30:52.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.209+0000 7f9498223640 1 -- 172.21.15.83:0/3754080826 wait complete. 2024-09-19T07:30:52.221 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5476 2024-09-19T07:30:52.222 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before reviving mds.b 2024-09-19T07:30:52.292 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:30:52.297+0000 7f70b70af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:30:52.457 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:52.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.597+0000 7f271c442640 1 Processor -- start 2024-09-19T07:30:52.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.597+0000 7f271c442640 1 -- start start 2024-09-19T07:30:52.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.597+0000 7f271c442640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2714150040 0x7f271406d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:52.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.597+0000 7f271c442640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f27140ad010 0x7f2714152930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:52.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.597+0000 7f271c442640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2714153240 con 0x7f2714071ea0 2024-09-19T07:30:52.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.597+0000 7f271c442640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f2714153520 con 0x7f27140ad010 2024-09-19T07:30:52.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.597+0000 7f271c442640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f2714153800 con 0x7f2714150040 2024-09-19T07:30:52.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f27199dd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f27140ad010 0x7f2714152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:52.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f271a9df640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2714071ea0 0x7f27140c3950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33736/0 (socket says 172.21.15.83:33736) 2024-09-19T07:30:52.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f27199dd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f27140ad010 0x7f2714152930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54984/0 (socket says 172.21.15.83:54984) 2024-09-19T07:30:52.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f271a9df640 1 -- 172.21.15.83:0/4043680041 learned_addr learned my addr 172.21.15.83:0/4043680041 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:30:52.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f271a1de640 1 --2- 172.21.15.83:0/4043680041 >> v2:172.21.15.184:3300/0 conn(0x7f2714150040 0x7f271406d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:52.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f27199dd640 1 -- 172.21.15.83:0/4043680041 >> v2:172.21.15.184:3300/0 conn(0x7f2714150040 msgr2=0x7f271406d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:52.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f27199dd640 1 --2- 172.21.15.83:0/4043680041 >> v2:172.21.15.184:3300/0 conn(0x7f2714150040 0x7f271406d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f27199dd640 1 -- 172.21.15.83:0/4043680041 >> v1:172.21.15.83:6789/0 conn(0x7f2714071ea0 legacy=0x7f27140c3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:52.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f27199dd640 1 -- 172.21.15.83:0/4043680041 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27140716a0 con 0x7f27140ad010 2024-09-19T07:30:52.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f27199dd640 1 --2- 172.21.15.83:0/4043680041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27140ad010 0x7f2714152930 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f27140c8df0 tx=0x7f270401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9ee78d4c6ec293 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:52.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f27191dc640 1 -- 172.21.15.83:0/4043680041 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27040b0070 con 0x7f27140ad010 2024-09-19T07:30:52.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f27191dc640 1 -- 172.21.15.83:0/4043680041 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27040ab600 con 0x7f27140ad010 2024-09-19T07:30:52.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.601+0000 7f27191dc640 1 -- 172.21.15.83:0/4043680041 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27040ab9d0 con 0x7f27140ad010 2024-09-19T07:30:52.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 -- 172.21.15.83:0/4043680041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27140ad010 msgr2=0x7f2714152930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:52.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 --2- 172.21.15.83:0/4043680041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27140ad010 0x7f2714152930 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f27140c8df0 tx=0x7f270401c3f0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 -- 172.21.15.83:0/4043680041 shutdown_connections 2024-09-19T07:30:52.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 --2- 172.21.15.83:0/4043680041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27140ad010 0x7f2714152930 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 --2- 172.21.15.83:0/4043680041 >> v2:172.21.15.184:3300/0 conn(0x7f2714150040 0x7f271406d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 -- 172.21.15.83:0/4043680041 >> 172.21.15.83:0/4043680041 conn(0x7f27140c7e40 msgr2=0x7f27140c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:52.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 -- 172.21.15.83:0/4043680041 shutdown_connections 2024-09-19T07:30:52.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 -- 172.21.15.83:0/4043680041 wait complete. 2024-09-19T07:30:52.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 Processor -- start 2024-09-19T07:30:52.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 -- start start 2024-09-19T07:30:52.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2714071ea0 0x7f271415b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:52.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27140ad010 0x7f27140c2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:52.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2714150040 0x7f27141ecbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:52.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f27141505a0 con 0x7f2714150040 2024-09-19T07:30:52.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2714150420 con 0x7f27140ad010 2024-09-19T07:30:52.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f271415fad0 con 0x7f2714071ea0 2024-09-19T07:30:52.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271a9df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2714150040 0x7f27141ecbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:52.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271a9df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2714150040 0x7f27141ecbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54988/0 (socket says 172.21.15.83:54988) 2024-09-19T07:30:52.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271a9df640 1 -- 172.21.15.83:0/356839423 learned_addr learned my addr 172.21.15.83:0/356839423 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:52.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f27199dd640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27140ad010 0x7f27140c2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271a1de640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2714071ea0 0x7f271415b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271a9df640 1 -- 172.21.15.83:0/356839423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2714071ea0 msgr2=0x7f271415b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271a9df640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2714071ea0 0x7f271415b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271a9df640 1 -- 172.21.15.83:0/356839423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27140ad010 msgr2=0x7f27140c2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271a9df640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27140ad010 0x7f27140c2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271a9df640 1 -- 172.21.15.83:0/356839423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27141ef830 con 0x7f2714150040 2024-09-19T07:30:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f27199dd640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27140ad010 0x7f27140c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:30:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271a9df640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2714150040 0x7f27141ecbc0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f270c00cbc0 tx=0x7f270c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f270affd640 1 -- 172.21.15.83:0/356839423 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c017070 con 0x7f2714150040 2024-09-19T07:30:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f270affd640 1 -- 172.21.15.83:0/356839423 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f270c00ad40 con 0x7f2714150040 2024-09-19T07:30:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 -- 172.21.15.83:0/356839423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27141ec3c0 con 0x7f2714150040 2024-09-19T07:30:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f270affd640 1 -- 172.21.15.83:0/356839423 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c012600 con 0x7f2714150040 2024-09-19T07:30:52.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.605+0000 7f271c442640 1 -- 172.21.15.83:0/356839423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f271415cee0 con 0x7f2714150040 2024-09-19T07:30:52.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.609+0000 7f271c442640 1 -- 172.21.15.83:0/356839423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2714152930 con 0x7f2714150040 2024-09-19T07:30:52.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.609+0000 7f270affd640 1 -- 172.21.15.83:0/356839423 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f270c01a070 con 0x7f2714150040 2024-09-19T07:30:52.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.609+0000 7f270affd640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26f80678c0 0x7f26f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:52.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.609+0000 7f270affd640 1 -- 172.21.15.83:0/356839423 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f270c0861b0 con 0x7f2714150040 2024-09-19T07:30:52.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.609+0000 7f271a1de640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26f80678c0 0x7f26f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:52.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.609+0000 7f271a1de640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26f80678c0 0x7f26f8069d80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2714156e70 tx=0x7f2710008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:52.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.613+0000 7f270affd640 1 -- 172.21.15.83:0/356839423 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f270c055dd0 con 0x7f2714150040 2024-09-19T07:30:52.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.845+0000 7f271c442640 1 -- 172.21.15.83:0/356839423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2714150d60 con 0x7f2714150040 2024-09-19T07:30:52.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.845+0000 7f270affd640 1 -- 172.21.15.83:0/356839423 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+4888 (secure 0 0 0) 0x7f270c05a0d0 con 0x7f2714150040 2024-09-19T07:30:52.844 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:52.844 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":93,"btime":"2024-09-19T07:30:51:094252+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:30:52.844 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 93 2024-09-19T07:30:52.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.849+0000 7f271c442640 1 -- 172.21.15.83:0/356839423 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26f80678c0 msgr2=0x7f26f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:52.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.849+0000 7f271c442640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26f80678c0 0x7f26f8069d80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2714156e70 tx=0x7f2710008040 comp rx=0 tx=0).stop 2024-09-19T07:30:52.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.849+0000 7f271c442640 1 -- 172.21.15.83:0/356839423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2714150040 msgr2=0x7f27141ecbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:52.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.849+0000 7f271c442640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2714150040 0x7f27141ecbc0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f270c00cbc0 tx=0x7f270c00a430 comp rx=0 tx=0).stop 2024-09-19T07:30:52.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.849+0000 7f271c442640 1 -- 172.21.15.83:0/356839423 shutdown_connections 2024-09-19T07:30:52.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.849+0000 7f271c442640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26f80678c0 0x7f26f8069d80 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.849+0000 7f271c442640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2714150040 0x7f27141ecbc0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.849+0000 7f271c442640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27140ad010 0x7f27140c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.849+0000 7f271c442640 1 --2- 172.21.15.83:0/356839423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2714071ea0 0x7f271415b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:52.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.849+0000 7f271c442640 1 -- 172.21.15.83:0/356839423 >> 172.21.15.83:0/356839423 conn(0x7f27140c7e40 msgr2=0x7f271406da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:52.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.849+0000 7f271c442640 1 -- 172.21.15.83:0/356839423 shutdown_connections 2024-09-19T07:30:52.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:52.849+0000 7f271c442640 1 -- 172.21.15.83:0/356839423 wait complete. 2024-09-19T07:30:52.868 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=6159 2024-09-19T07:30:52.868 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.a 2024-09-19T07:30:54.870 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-19T07:30:54.870 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:30:54.870 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:30:54.873 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:30:54.873 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:54.958 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:30:55.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51df413640 1 Processor -- start 2024-09-19T07:30:55.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51df413640 1 -- start start 2024-09-19T07:30:55.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51df413640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51d8150140 0x7f51d806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:55.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51df413640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f51d80c3c60 0x7f51d8155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:55.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51df413640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f51d806f3b0 0x7f51d81563f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:55.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51df413640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f51d80c34f0 con 0x7f51d806f3b0 2024-09-19T07:30:55.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51df413640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f51d80c4b00 con 0x7f51d80c3c60 2024-09-19T07:30:55.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51df413640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f51d80c4f90 con 0x7f51d8150140 2024-09-19T07:30:55.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51dd9b0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f51d806f3b0 0x7f51d81563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:55.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51dd9b0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f51d806f3b0 0x7f51d81563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54996/0 (socket says 172.21.15.83:54996) 2024-09-19T07:30:55.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51dd9b0640 1 -- 172.21.15.83:0/887208468 learned_addr learned my addr 172.21.15.83:0/887208468 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:55.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51dd1af640 1 --2- 172.21.15.83:0/887208468 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51d8150140 0x7f51d806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:55.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51dc9ae640 1 --2- 172.21.15.83:0/887208468 >> v2:172.21.15.184:3300/0 conn(0x7f51d80c3c60 0x7f51d8155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:55.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51dd9b0640 1 -- 172.21.15.83:0/887208468 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51d8150140 msgr2=0x7f51d806ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:55.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51dd9b0640 1 --2- 172.21.15.83:0/887208468 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51d8150140 0x7f51d806ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:55.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51dd9b0640 1 -- 172.21.15.83:0/887208468 >> v2:172.21.15.184:3300/0 conn(0x7f51d80c3c60 msgr2=0x7f51d8155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:55.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51dd9b0640 1 --2- 172.21.15.83:0/887208468 >> v2:172.21.15.184:3300/0 conn(0x7f51d80c3c60 0x7f51d8155eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:55.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51dd9b0640 1 -- 172.21.15.83:0/887208468 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d80c3840 con 0x7f51d806f3b0 2024-09-19T07:30:55.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.029+0000 7f51dc9ae640 1 --2- 172.21.15.83:0/887208468 >> v2:172.21.15.184:3300/0 conn(0x7f51d80c3c60 0x7f51d8155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:30:55.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51dd9b0640 1 --2- 172.21.15.83:0/887208468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d806f3b0 0x7f51d81563f0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f51d80c3a10 tx=0x7f51d001ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b3ed747886f1be server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:55.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51cffff640 1 -- 172.21.15.83:0/887208468 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d000b2d0 con 0x7f51d806f3b0 2024-09-19T07:30:55.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51cffff640 1 -- 172.21.15.83:0/887208468 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51d000b470 con 0x7f51d806f3b0 2024-09-19T07:30:55.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51cffff640 1 -- 172.21.15.83:0/887208468 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d00a7a10 con 0x7f51d806f3b0 2024-09-19T07:30:55.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51df413640 1 -- 172.21.15.83:0/887208468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d806f3b0 msgr2=0x7f51d81563f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:55.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51df413640 1 --2- 172.21.15.83:0/887208468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d806f3b0 0x7f51d81563f0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f51d80c3a10 tx=0x7f51d001ead0 comp rx=0 tx=0).stop 2024-09-19T07:30:55.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51df413640 1 -- 172.21.15.83:0/887208468 shutdown_connections 2024-09-19T07:30:55.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51df413640 1 --2- 172.21.15.83:0/887208468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d806f3b0 0x7f51d81563f0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:55.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51df413640 1 --2- 172.21.15.83:0/887208468 >> v2:172.21.15.184:3300/0 conn(0x7f51d80c3c60 0x7f51d8155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:55.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51df413640 1 --2- 172.21.15.83:0/887208468 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51d8150140 0x7f51d806ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:55.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51df413640 1 -- 172.21.15.83:0/887208468 >> 172.21.15.83:0/887208468 conn(0x7f51d80aa5f0 msgr2=0x7f51d80aaa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:55.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51df413640 1 -- 172.21.15.83:0/887208468 shutdown_connections 2024-09-19T07:30:55.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.033+0000 7f51df413640 1 -- 172.21.15.83:0/887208468 wait complete. 2024-09-19T07:30:55.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51df413640 1 Processor -- start 2024-09-19T07:30:55.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51df413640 1 -- start start 2024-09-19T07:30:55.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51df413640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51d806f3b0 0x7f51d81e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:55.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51df413640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51d80c3c60 0x7f51d80ac4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:55.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51df413640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d8150140 0x7f51d80ab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:55.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51df413640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f51d8072b30 con 0x7f51d8150140 2024-09-19T07:30:55.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51df413640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f51d80729b0 con 0x7f51d80c3c60 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51df413640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f51d806c490 con 0x7f51d806f3b0 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51dd9b0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d8150140 0x7f51d80ab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51dd9b0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d8150140 0x7f51d80ab4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55010/0 (socket says 172.21.15.83:55010) 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51dd9b0640 1 -- 172.21.15.83:0/3305568356 learned_addr learned my addr 172.21.15.83:0/3305568356 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51dd1af640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51d806f3b0 0x7f51d81e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51dc9ae640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51d80c3c60 0x7f51d80ac4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51dd9b0640 1 -- 172.21.15.83:0/3305568356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51d806f3b0 msgr2=0x7f51d81e8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51dd9b0640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51d806f3b0 0x7f51d81e8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51dd9b0640 1 -- 172.21.15.83:0/3305568356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51d80c3c60 msgr2=0x7f51d80ac4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51dd9b0640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51d80c3c60 0x7f51d80ac4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51dd9b0640 1 -- 172.21.15.83:0/3305568356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d80ae4c0 con 0x7f51d8150140 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51dd1af640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51d806f3b0 0x7f51d81e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51dd9b0640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d8150140 0x7f51d80ab4b0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f51d00b1040 tx=0x7f51d001b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:55.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51cdffb640 1 -- 172.21.15.83:0/3305568356 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d00be020 con 0x7f51d8150140 2024-09-19T07:30:55.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51cdffb640 1 -- 172.21.15.83:0/3305568356 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51d0009800 con 0x7f51d8150140 2024-09-19T07:30:55.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51cdffb640 1 -- 172.21.15.83:0/3305568356 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d000b720 con 0x7f51d8150140 2024-09-19T07:30:55.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51df413640 1 -- 172.21.15.83:0/3305568356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d81eafb0 con 0x7f51d8150140 2024-09-19T07:30:55.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.037+0000 7f51df413640 1 -- 172.21.15.83:0/3305568356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d80aca10 con 0x7f51d8150140 2024-09-19T07:30:55.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.041+0000 7f51df413640 1 -- 172.21.15.83:0/3305568356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51d806c800 con 0x7f51d8150140 2024-09-19T07:30:55.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.041+0000 7f51cdffb640 1 -- 172.21.15.83:0/3305568356 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f51d00c1020 con 0x7f51d8150140 2024-09-19T07:30:55.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.041+0000 7f51cdffb640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f51bc0678c0 0x7f51bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:55.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.041+0000 7f51dd1af640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f51bc0678c0 0x7f51bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:55.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.041+0000 7f51cdffb640 1 -- 172.21.15.83:0/3305568356 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f51d0123620 con 0x7f51d8150140 2024-09-19T07:30:55.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.041+0000 7f51dd1af640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f51bc0678c0 0x7f51bc069d80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f51d4009b10 tx=0x7f51d4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:55.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.049+0000 7f51cdffb640 1 -- 172.21.15.83:0/3305568356 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f51d00c8030 con 0x7f51d8150140 2024-09-19T07:30:55.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.281+0000 7f51df413640 1 -- 172.21.15.83:0/3305568356 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51d80c2100 con 0x7f51d8150140 2024-09-19T07:30:55.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51cdffb640 1 -- 172.21.15.83:0/3305568356 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+5712 (secure 0 0 0) 0x7f51d00187e0 con 0x7f51d8150140 2024-09-19T07:30:55.280 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:55.280 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":94,"btime":"2024-09-19T07:30:55:042498+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6447,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:30:55.281 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 94 2024-09-19T07:30:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51df413640 1 -- 172.21.15.83:0/3305568356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f51bc0678c0 msgr2=0x7f51bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51df413640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f51bc0678c0 0x7f51bc069d80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f51d4009b10 tx=0x7f51d4008040 comp rx=0 tx=0).stop 2024-09-19T07:30:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51df413640 1 -- 172.21.15.83:0/3305568356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d8150140 msgr2=0x7f51d80ab4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51df413640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d8150140 0x7f51d80ab4b0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f51d00b1040 tx=0x7f51d001b870 comp rx=0 tx=0).stop 2024-09-19T07:30:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51df413640 1 -- 172.21.15.83:0/3305568356 shutdown_connections 2024-09-19T07:30:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51df413640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f51bc0678c0 0x7f51bc069d80 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51df413640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d8150140 0x7f51d80ab4b0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51df413640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51d80c3c60 0x7f51d80ac4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51df413640 1 --2- 172.21.15.83:0/3305568356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51d806f3b0 0x7f51d81e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51df413640 1 -- 172.21.15.83:0/3305568356 >> 172.21.15.83:0/3305568356 conn(0x7f51d80aa5f0 msgr2=0x7f51d814dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51df413640 1 -- 172.21.15.83:0/3305568356 shutdown_connections 2024-09-19T07:30:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:55.285+0000 7f51df413640 1 -- 172.21.15.83:0/3305568356 wait complete. 2024-09-19T07:30:55.300 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-19T07:30:55.300 INFO:tasks.mds_thrash.fs.[a]:waiting for 84 secs before thrashing 2024-09-19T07:30:57.223 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-09-19T07:30:57.223 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-19T07:30:57.223 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-19T07:30:57.226 INFO:tasks.ceph.mds.b:Started 2024-09-19T07:30:57.226 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:57.295 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-19T07:30:57.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.361+0000 7f7c4751f640 1 Processor -- start 2024-09-19T07:30:57.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.361+0000 7f7c4751f640 1 -- start start 2024-09-19T07:30:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c4751f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7c400c2ca0 0x7f7c40155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c4751f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c400c7860 con 0x7f7c400af370 2024-09-19T07:30:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c4751f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c40159d50 con 0x7f7c400aee20 2024-09-19T07:30:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c4751f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7c400c4aa0 con 0x7f7c400c2ca0 2024-09-19T07:30:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c452bb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7c400c2ca0 0x7f7c40155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:57.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c44aba640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7c400af370 0x7f7c4006ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33754/0 (socket says 172.21.15.83:33754) 2024-09-19T07:30:57.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c44aba640 1 -- 172.21.15.83:0/3128535181 learned_addr learned my addr 172.21.15.83:0/3128535181 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:30:57.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c452bb640 1 -- 172.21.15.83:0/3128535181 >> v1:172.21.15.184:6789/0 conn(0x7f7c400aee20 legacy=0x7f7c401532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:57.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c452bb640 1 -- 172.21.15.83:0/3128535181 >> v1:172.21.15.83:6789/0 conn(0x7f7c400af370 legacy=0x7f7c4006ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:57.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c37fff640 1 -- 172.21.15.83:0/3128535181 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1407733026 0 0) 0x7f7c400c7860 con 0x7f7c400af370 2024-09-19T07:30:57.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c452bb640 1 -- 172.21.15.83:0/3128535181 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4015a030 con 0x7f7c400c2ca0 2024-09-19T07:30:57.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c452bb640 1 --2- 172.21.15.83:0/3128535181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c400c2ca0 0x7f7c40155dd0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c0097c0 tx=0x7f7c3c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb0564df52315aba server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:57.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c37fff640 1 -- 172.21.15.83:0/3128535181 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c0ab430 con 0x7f7c400c2ca0 2024-09-19T07:30:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c37fff640 1 -- 172.21.15.83:0/3128535181 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c3c0b0070 con 0x7f7c400c2ca0 2024-09-19T07:30:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c37fff640 1 -- 172.21.15.83:0/3128535181 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c0ab7f0 con 0x7f7c400c2ca0 2024-09-19T07:30:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c4751f640 1 -- 172.21.15.83:0/3128535181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c400c2ca0 msgr2=0x7f7c40155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c4751f640 1 --2- 172.21.15.83:0/3128535181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c400c2ca0 0x7f7c40155dd0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c0097c0 tx=0x7f7c3c01c410 comp rx=0 tx=0).stop 2024-09-19T07:30:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c4751f640 1 -- 172.21.15.83:0/3128535181 shutdown_connections 2024-09-19T07:30:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c4751f640 1 --2- 172.21.15.83:0/3128535181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c400c2ca0 0x7f7c40155dd0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.365+0000 7f7c4751f640 1 -- 172.21.15.83:0/3128535181 >> 172.21.15.83:0/3128535181 conn(0x7f7c400c9610 msgr2=0x7f7c400c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c4751f640 1 -- 172.21.15.83:0/3128535181 shutdown_connections 2024-09-19T07:30:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c4751f640 1 -- 172.21.15.83:0/3128535181 wait complete. 2024-09-19T07:30:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c4751f640 1 Processor -- start 2024-09-19T07:30:57.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c4751f640 1 -- start start 2024-09-19T07:30:57.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c4751f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c400aee20 0x7f7c401eee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:57.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c4751f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c400af370 0x7f7c40156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c4751f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c400c2ca0 0x7f7c4014ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c4751f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7c40164020 con 0x7f7c400aee20 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c4751f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c40163ea0 con 0x7f7c400af370 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c4751f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7c401641a0 con 0x7f7c400c2ca0 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c452bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c400aee20 0x7f7c401eee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c452bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c400aee20 0x7f7c401eee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55032/0 (socket says 172.21.15.83:55032) 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c452bb640 1 -- 172.21.15.83:0/2943636863 learned_addr learned my addr 172.21.15.83:0/2943636863 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c45abc640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c400c2ca0 0x7f7c4014ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c44aba640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c400af370 0x7f7c40156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c452bb640 1 -- 172.21.15.83:0/2943636863 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c400c2ca0 msgr2=0x7f7c4014ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c452bb640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c400c2ca0 0x7f7c4014ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c452bb640 1 -- 172.21.15.83:0/2943636863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c400af370 msgr2=0x7f7c40156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c452bb640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c400af370 0x7f7c40156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c452bb640 1 -- 172.21.15.83:0/2943636863 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c401ef3c0 con 0x7f7c400aee20 2024-09-19T07:30:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c452bb640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c400aee20 0x7f7c401eee80 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c0ad040 tx=0x7f7c3c01ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c35ffb640 1 -- 172.21.15.83:0/2943636863 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c0bf610 con 0x7f7c400aee20 2024-09-19T07:30:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c35ffb640 1 -- 172.21.15.83:0/2943636863 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c3c0bf7f0 con 0x7f7c400aee20 2024-09-19T07:30:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c4751f640 1 -- 172.21.15.83:0/2943636863 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c40157470 con 0x7f7c400aee20 2024-09-19T07:30:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c35ffb640 1 -- 172.21.15.83:0/2943636863 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c0bfbe0 con 0x7f7c400aee20 2024-09-19T07:30:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.369+0000 7f7c4751f640 1 -- 172.21.15.83:0/2943636863 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c401514f0 con 0x7f7c400aee20 2024-09-19T07:30:57.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.373+0000 7f7c4751f640 1 -- 172.21.15.83:0/2943636863 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c400c7860 con 0x7f7c400aee20 2024-09-19T07:30:57.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.373+0000 7f7c35ffb640 1 -- 172.21.15.83:0/2943636863 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7c3c0bd020 con 0x7f7c400aee20 2024-09-19T07:30:57.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.373+0000 7f7c35ffb640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c240678c0 0x7f7c24069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:57.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.373+0000 7f7c35ffb640 1 -- 172.21.15.83:0/2943636863 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f7c3c0a8030 con 0x7f7c400aee20 2024-09-19T07:30:57.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.373+0000 7f7c44aba640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c240678c0 0x7f7c24069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:57.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.373+0000 7f7c44aba640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c240678c0 0x7f7c24069d80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7c401f1940 tx=0x7f7c30007240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:57.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.377+0000 7f7c35ffb640 1 -- 172.21.15.83:0/2943636863 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7c3c0c4060 con 0x7f7c400aee20 2024-09-19T07:30:57.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.649+0000 7f7c4751f640 1 -- 172.21.15.83:0/2943636863 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c400c9b90 con 0x7f7c400aee20 2024-09-19T07:30:57.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c35ffb640 1 -- 172.21.15.83:0/2943636863 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+5712 (secure 0 0 0) 0x7f7c3c016090 con 0x7f7c400aee20 2024-09-19T07:30:57.648 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:30:57.648 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":94,"btime":"2024-09-19T07:30:55:042498+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6447,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:30:57.648 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 94 2024-09-19T07:30:57.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c4751f640 1 -- 172.21.15.83:0/2943636863 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c240678c0 msgr2=0x7f7c24069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:57.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c4751f640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c240678c0 0x7f7c24069d80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7c401f1940 tx=0x7f7c30007240 comp rx=0 tx=0).stop 2024-09-19T07:30:57.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c4751f640 1 -- 172.21.15.83:0/2943636863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c400aee20 msgr2=0x7f7c401eee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:57.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c4751f640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c400aee20 0x7f7c401eee80 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c0ad040 tx=0x7f7c3c01ac50 comp rx=0 tx=0).stop 2024-09-19T07:30:57.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c4751f640 1 -- 172.21.15.83:0/2943636863 shutdown_connections 2024-09-19T07:30:57.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c4751f640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c240678c0 0x7f7c24069d80 secure :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7c401f1940 tx=0x7f7c30007240 comp rx=0 tx=0).stop 2024-09-19T07:30:57.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c4751f640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c400c2ca0 0x7f7c4014ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:57.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c4751f640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c400af370 0x7f7c40156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:57.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c4751f640 1 --2- 172.21.15.83:0/2943636863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c400aee20 0x7f7c401eee80 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:57.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c4751f640 1 -- 172.21.15.83:0/2943636863 >> 172.21.15.83:0/2943636863 conn(0x7f7c400c9610 msgr2=0x7f7c40152640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:57.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c4751f640 1 -- 172.21.15.83:0/2943636863 shutdown_connections 2024-09-19T07:30:57.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:57.653+0000 7f7c4751f640 1 -- 172.21.15.83:0/2943636863 wait complete. 2024-09-19T07:30:57.668 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-19T07:30:59.670 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:30:59.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.809+0000 7f16a370d640 1 Processor -- start 2024-09-19T07:30:59.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.809+0000 7f16a370d640 1 -- start start 2024-09-19T07:30:59.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a370d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f169c0c6120 0x7f169c0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:59.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a370d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f169c155d70 0x7f169c1575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:59.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a370d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169c156200 con 0x7f169c1592f0 2024-09-19T07:30:59.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a370d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f169c0c9930 con 0x7f169c155d70 2024-09-19T07:30:59.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a370d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f169c1564e0 con 0x7f169c0c6120 2024-09-19T07:30:59.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a0ca8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f169c155d70 0x7f169c1575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:59.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a1caa640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f169c1592f0 0x7f169c157b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33768/0 (socket says 172.21.15.83:33768) 2024-09-19T07:30:59.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a1caa640 1 -- 172.21.15.83:0/4209725231 learned_addr learned my addr 172.21.15.83:0/4209725231 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:30:59.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a0ca8640 1 -- 172.21.15.83:0/4209725231 >> v2:172.21.15.184:3300/0 conn(0x7f169c0c6120 msgr2=0x7f169c0c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:30:59.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a0ca8640 1 --2- 172.21.15.83:0/4209725231 >> v2:172.21.15.184:3300/0 conn(0x7f169c0c6120 0x7f169c0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:59.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a0ca8640 1 -- 172.21.15.83:0/4209725231 >> v1:172.21.15.83:6789/0 conn(0x7f169c1592f0 legacy=0x7f169c157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:59.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a0ca8640 1 -- 172.21.15.83:0/4209725231 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c1567c0 con 0x7f169c155d70 2024-09-19T07:30:59.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.813+0000 7f16a0ca8640 1 --2- 172.21.15.83:0/4209725231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169c155d70 0x7f169c1575d0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f169c08d540 tx=0x7f168c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d75c26a86887b38 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:59.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f1693fff640 1 -- 172.21.15.83:0/4209725231 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c0b0070 con 0x7f169c155d70 2024-09-19T07:30:59.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f1693fff640 1 -- 172.21.15.83:0/4209725231 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f168c0ab600 con 0x7f169c155d70 2024-09-19T07:30:59.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f1693fff640 1 -- 172.21.15.83:0/4209725231 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c0ab9d0 con 0x7f169c155d70 2024-09-19T07:30:59.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 -- 172.21.15.83:0/4209725231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169c155d70 msgr2=0x7f169c1575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:59.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 --2- 172.21.15.83:0/4209725231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169c155d70 0x7f169c1575d0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f169c08d540 tx=0x7f168c01c480 comp rx=0 tx=0).stop 2024-09-19T07:30:59.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 -- 172.21.15.83:0/4209725231 shutdown_connections 2024-09-19T07:30:59.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 --2- 172.21.15.83:0/4209725231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169c155d70 0x7f169c1575d0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:59.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 --2- 172.21.15.83:0/4209725231 >> v2:172.21.15.184:3300/0 conn(0x7f169c0c6120 0x7f169c0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:59.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 -- 172.21.15.83:0/4209725231 >> 172.21.15.83:0/4209725231 conn(0x7f169c06bc80 msgr2=0x7f169c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:30:59.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 -- 172.21.15.83:0/4209725231 shutdown_connections 2024-09-19T07:30:59.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 -- 172.21.15.83:0/4209725231 wait complete. 2024-09-19T07:30:59.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 Processor -- start 2024-09-19T07:30:59.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 -- start start 2024-09-19T07:30:59.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f169c0c6120 0x7f169c1ef110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:59.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169c1592f0 0x7f169c0c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:59.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f169c14f880 0x7f169c0cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:59.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f169c153860 con 0x7f169c1592f0 2024-09-19T07:30:59.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f169c162ea0 con 0x7f169c0c6120 2024-09-19T07:30:59.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f169c144220 con 0x7f169c14f880 2024-09-19T07:30:59.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a0ca8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169c1592f0 0x7f169c0c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:59.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a0ca8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169c1592f0 0x7f169c0c1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55060/0 (socket says 172.21.15.83:55060) 2024-09-19T07:30:59.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a0ca8640 1 -- 172.21.15.83:0/2551717052 learned_addr learned my addr 172.21.15.83:0/2551717052 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:30:59.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a14a9640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f169c0c6120 0x7f169c1ef110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:59.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a1caa640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f169c14f880 0x7f169c0cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:59.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a0ca8640 1 -- 172.21.15.83:0/2551717052 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f169c14f880 msgr2=0x7f169c0cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:59.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a0ca8640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f169c14f880 0x7f169c0cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:59.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a0ca8640 1 -- 172.21.15.83:0/2551717052 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f169c0c6120 msgr2=0x7f169c1ef110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:30:59.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a0ca8640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f169c0c6120 0x7f169c1ef110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:30:59.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a0ca8640 1 -- 172.21.15.83:0/2551717052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c1ef650 con 0x7f169c1592f0 2024-09-19T07:30:59.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a14a9640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f169c0c6120 0x7f169c1ef110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:30:59.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a0ca8640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169c1592f0 0x7f169c0c1d20 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f168c019090 tx=0x7f168c01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:59.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f1691ffb640 1 -- 172.21.15.83:0/2551717052 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c0b0050 con 0x7f169c1592f0 2024-09-19T07:30:59.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f1691ffb640 1 -- 172.21.15.83:0/2551717052 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f168c0bc070 con 0x7f169c1592f0 2024-09-19T07:30:59.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 -- 172.21.15.83:0/2551717052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f169c14e290 con 0x7f169c1592f0 2024-09-19T07:30:59.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f1691ffb640 1 -- 172.21.15.83:0/2551717052 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c0b5a70 con 0x7f169c1592f0 2024-09-19T07:30:59.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.817+0000 7f16a370d640 1 -- 172.21.15.83:0/2551717052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f169c14eab0 con 0x7f169c1592f0 2024-09-19T07:30:59.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.821+0000 7f16a370d640 1 -- 172.21.15.83:0/2551717052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f169c0c6500 con 0x7f169c1592f0 2024-09-19T07:30:59.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.821+0000 7f1691ffb640 1 -- 172.21.15.83:0/2551717052 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f168c0ab480 con 0x7f169c1592f0 2024-09-19T07:30:59.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.821+0000 7f1691ffb640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16800678c0 0x7f1680069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:30:59.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.821+0000 7f1691ffb640 1 -- 172.21.15.83:0/2551717052 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f168c121940 con 0x7f169c1592f0 2024-09-19T07:30:59.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.821+0000 7f16a14a9640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16800678c0 0x7f1680069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:30:59.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.821+0000 7f16a14a9640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16800678c0 0x7f1680069d80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1698006fd0 tx=0x7f1698008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:30:59.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:30:59.825+0000 7f1691ffb640 1 -- 172.21.15.83:0/2551717052 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f168c016070 con 0x7f169c1592f0 2024-09-19T07:31:00.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.089+0000 7f16a370d640 1 -- 172.21.15.83:0/2551717052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f169c0c96a0 con 0x7f169c1592f0 2024-09-19T07:31:00.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.089+0000 7f1691ffb640 1 -- 172.21.15.83:0/2551717052 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+6540 (secure 0 0 0) 0x7f168c0aa3a0 con 0x7f169c1592f0 2024-09-19T07:31:00.090 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:00.090 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":95,"btime":"2024-09-19T07:30:58:147417+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6447,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:31:00.091 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 95 2024-09-19T07:31:00.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.093+0000 7f16a370d640 1 -- 172.21.15.83:0/2551717052 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16800678c0 msgr2=0x7f1680069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:00.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.093+0000 7f16a370d640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16800678c0 0x7f1680069d80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1698006fd0 tx=0x7f1698008040 comp rx=0 tx=0).stop 2024-09-19T07:31:00.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.093+0000 7f16a370d640 1 -- 172.21.15.83:0/2551717052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169c1592f0 msgr2=0x7f169c0c1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:00.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.093+0000 7f16a370d640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169c1592f0 0x7f169c0c1d20 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f168c019090 tx=0x7f168c01acc0 comp rx=0 tx=0).stop 2024-09-19T07:31:00.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.093+0000 7f16a370d640 1 -- 172.21.15.83:0/2551717052 shutdown_connections 2024-09-19T07:31:00.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.093+0000 7f16a370d640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16800678c0 0x7f1680069d80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:00.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.093+0000 7f16a370d640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f169c14f880 0x7f169c0cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:00.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.093+0000 7f16a370d640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169c1592f0 0x7f169c0c1d20 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:00.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.093+0000 7f16a370d640 1 --2- 172.21.15.83:0/2551717052 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f169c0c6120 0x7f169c1ef110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:00.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.093+0000 7f16a370d640 1 -- 172.21.15.83:0/2551717052 >> 172.21.15.83:0/2551717052 conn(0x7f169c06bc80 msgr2=0x7f169c151e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:00.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.093+0000 7f16a370d640 1 -- 172.21.15.83:0/2551717052 shutdown_connections 2024-09-19T07:31:00.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:00.093+0000 7f16a370d640 1 -- 172.21.15.83:0/2551717052 wait complete. 2024-09-19T07:31:00.107 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-09-19T07:31:00.108 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before thrashing 2024-09-19T07:31:02.109 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:02.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.269+0000 7f80abec3640 1 Processor -- start 2024-09-19T07:31:02.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.269+0000 7f80abec3640 1 -- start start 2024-09-19T07:31:02.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.269+0000 7f80abec3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f80a40c3a20 0x7f80a41594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:02.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.269+0000 7f80abec3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f80a40c4450 0x7f80a41566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:02.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.269+0000 7f80abec3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80a4155ee0 con 0x7f80a4156c00 2024-09-19T07:31:02.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.269+0000 7f80abec3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f80a4068290 con 0x7f80a40c3a20 2024-09-19T07:31:02.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.269+0000 7f80abec3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f80a40684e0 con 0x7f80a40c4450 2024-09-19T07:31:02.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.269+0000 7f80a9c5f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f80a40c3a20 0x7f80a41594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:02.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.273+0000 7f80a9c5f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f80a40c3a20 0x7f80a41594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43934/0 (socket says 172.21.15.83:43934) 2024-09-19T07:31:02.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.273+0000 7f80a9c5f640 1 -- 172.21.15.83:0/696700563 learned_addr learned my addr 172.21.15.83:0/696700563 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:02.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.273+0000 7f80a945e640 1 --2- 172.21.15.83:0/696700563 >> v2:172.21.15.184:3300/0 conn(0x7f80a40c4450 0x7f80a41566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:02.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.273+0000 7f80a9c5f640 1 -- 172.21.15.83:0/696700563 >> v2:172.21.15.184:3300/0 conn(0x7f80a40c4450 msgr2=0x7f80a41566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:02.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.273+0000 7f80a9c5f640 1 --2- 172.21.15.83:0/696700563 >> v2:172.21.15.184:3300/0 conn(0x7f80a40c4450 0x7f80a41566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.273+0000 7f80a9c5f640 1 -- 172.21.15.83:0/696700563 >> v1:172.21.15.184:6789/0 conn(0x7f80a4156c00 legacy=0x7f80a414c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:02.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.273+0000 7f80a9c5f640 1 -- 172.21.15.83:0/696700563 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a40687c0 con 0x7f80a40c3a20 2024-09-19T07:31:02.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.273+0000 7f80a9c5f640 1 --2- 172.21.15.83:0/696700563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80a40c3a20 0x7f80a41594d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f80a40c2680 tx=0x7f80a001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7210cc5791a1e56 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:02.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.273+0000 7f80a8c5d640 1 -- 172.21.15.83:0/696700563 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a406fcd0 con 0x7f80a40c3a20 2024-09-19T07:31:02.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.273+0000 7f80a8c5d640 1 -- 172.21.15.83:0/696700563 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80a00b0070 con 0x7f80a40c3a20 2024-09-19T07:31:02.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.273+0000 7f80a8c5d640 1 -- 172.21.15.83:0/696700563 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a00ab860 con 0x7f80a40c3a20 2024-09-19T07:31:02.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 -- 172.21.15.83:0/696700563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80a40c3a20 msgr2=0x7f80a41594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:02.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 --2- 172.21.15.83:0/696700563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80a40c3a20 0x7f80a41594d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f80a40c2680 tx=0x7f80a001c420 comp rx=0 tx=0).stop 2024-09-19T07:31:02.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 -- 172.21.15.83:0/696700563 shutdown_connections 2024-09-19T07:31:02.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 --2- 172.21.15.83:0/696700563 >> v2:172.21.15.184:3300/0 conn(0x7f80a40c4450 0x7f80a41566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 --2- 172.21.15.83:0/696700563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80a40c3a20 0x7f80a41594d0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 -- 172.21.15.83:0/696700563 >> 172.21.15.83:0/696700563 conn(0x7f80a40c48b0 msgr2=0x7f80a40c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:02.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 -- 172.21.15.83:0/696700563 shutdown_connections 2024-09-19T07:31:02.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 -- 172.21.15.83:0/696700563 wait complete. 2024-09-19T07:31:02.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 Processor -- start 2024-09-19T07:31:02.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 -- start start 2024-09-19T07:31:02.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80a40c3a20 0x7f80a4152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:02.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80a40c4450 0x7f80a40acf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:02.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80a4156c00 0x7f80a40ad450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:02.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f80a4163080 con 0x7f80a40c4450 2024-09-19T07:31:02.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f80a40c4010 con 0x7f80a4156c00 2024-09-19T07:31:02.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f80a4163200 con 0x7f80a40c3a20 2024-09-19T07:31:02.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80a945e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80a40c4450 0x7f80a40acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:02.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80a945e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80a40c4450 0x7f80a40acf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43936/0 (socket says 172.21.15.83:43936) 2024-09-19T07:31:02.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80a945e640 1 -- 172.21.15.83:0/2340410994 learned_addr learned my addr 172.21.15.83:0/2340410994 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:02.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80a9c5f640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80a40c3a20 0x7f80a4152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:02.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80aa460640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80a4156c00 0x7f80a40ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:02.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80a945e640 1 -- 172.21.15.83:0/2340410994 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80a40c3a20 msgr2=0x7f80a4152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:02.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80a945e640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80a40c3a20 0x7f80a4152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80a945e640 1 -- 172.21.15.83:0/2340410994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80a4156c00 msgr2=0x7f80a40ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:02.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80a945e640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80a4156c00 0x7f80a40ad450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80a945e640 1 -- 172.21.15.83:0/2340410994 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a41ee940 con 0x7f80a40c4450 2024-09-19T07:31:02.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80aa460640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80a4156c00 0x7f80a40ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:31:02.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80a945e640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80a40c4450 0x7f80a40acf10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f809400c7f0 tx=0x7f809400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:02.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f809a7fc640 1 -- 172.21.15.83:0/2340410994 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8094015070 con 0x7f80a40c4450 2024-09-19T07:31:02.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f809a7fc640 1 -- 172.21.15.83:0/2340410994 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8094007820 con 0x7f80a40c4450 2024-09-19T07:31:02.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 -- 172.21.15.83:0/2340410994 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80a41eec00 con 0x7f80a40c4450 2024-09-19T07:31:02.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f809a7fc640 1 -- 172.21.15.83:0/2340410994 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8094007c50 con 0x7f80a40c4450 2024-09-19T07:31:02.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.277+0000 7f80abec3640 1 -- 172.21.15.83:0/2340410994 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80a41ef1c0 con 0x7f80a40c4450 2024-09-19T07:31:02.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.281+0000 7f80abec3640 1 -- 172.21.15.83:0/2340410994 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80a41594d0 con 0x7f80a40c4450 2024-09-19T07:31:02.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.281+0000 7f809a7fc640 1 -- 172.21.15.83:0/2340410994 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8094018070 con 0x7f80a40c4450 2024-09-19T07:31:02.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.281+0000 7f809a7fc640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80880678c0 0x7f8088069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:02.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.281+0000 7f809a7fc640 1 -- 172.21.15.83:0/2340410994 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f8094086620 con 0x7f80a40c4450 2024-09-19T07:31:02.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.281+0000 7f80a9c5f640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80880678c0 0x7f8088069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:02.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.281+0000 7f80a9c5f640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80880678c0 0x7f8088069d80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f80a00af000 tx=0x7f80a00a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:02.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.285+0000 7f809a7fc640 1 -- 172.21.15.83:0/2340410994 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f8094056240 con 0x7f80a40c4450 2024-09-19T07:31:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 -- 172.21.15.83:0/2340410994 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80a406f040 con 0x7f80a40c4450 2024-09-19T07:31:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f809a7fc640 1 -- 172.21.15.83:0/2340410994 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+6540 (secure 0 0 0) 0x7f809405a540 con 0x7f80a40c4450 2024-09-19T07:31:02.568 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:02.569 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":95,"btime":"2024-09-19T07:30:58:147417+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6447,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:31:02.569 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 95 2024-09-19T07:31:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 -- 172.21.15.83:0/2340410994 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80880678c0 msgr2=0x7f8088069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80880678c0 0x7f8088069d80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f80a00af000 tx=0x7f80a00a7000 comp rx=0 tx=0).stop 2024-09-19T07:31:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 -- 172.21.15.83:0/2340410994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80a40c4450 msgr2=0x7f80a40acf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80a40c4450 0x7f80a40acf10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f809400c7f0 tx=0x7f809400ccb0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 -- 172.21.15.83:0/2340410994 shutdown_connections 2024-09-19T07:31:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80880678c0 0x7f8088069d80 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80a4156c00 0x7f80a40ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80a40c4450 0x7f80a40acf10 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 --2- 172.21.15.83:0/2340410994 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80a40c3a20 0x7f80a4152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 -- 172.21.15.83:0/2340410994 >> 172.21.15.83:0/2340410994 conn(0x7f80a40c48b0 msgr2=0x7f80a40c1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 -- 172.21.15.83:0/2340410994 shutdown_connections 2024-09-19T07:31:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.573+0000 7f80abec3640 1 -- 172.21.15.83:0/2340410994 wait complete. 2024-09-19T07:31:02.588 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-19T07:31:02.589 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:31:02.589 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:31:02.600 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:31:02.600 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:02.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3cfda640 1 Processor -- start 2024-09-19T07:31:02.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3cfda640 1 -- start start 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3cfda640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4b3806cef0 0x7f4b380abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3cfda640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4b380ab610 0x7f4b380c3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3cfda640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b380c7de0 con 0x7f4b380c6fc0 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3cfda640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f4b380c6a90 con 0x7f4b3806cef0 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3cfda640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4b380c8360 con 0x7f4b380ab610 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4b3806cef0 0x7f4b380abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4b3806cef0 0x7f4b380abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43944/0 (socket says 172.21.15.83:43944) 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3659c640 1 -- 172.21.15.83:0/3903081311 learned_addr learned my addr 172.21.15.83:0/3903081311 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b2dd9b640 1 --2- 172.21.15.83:0/3903081311 >> v2:172.21.15.184:3300/0 conn(0x7f4b380ab610 0x7f4b380c3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3659c640 1 -- 172.21.15.83:0/3903081311 >> v2:172.21.15.184:3300/0 conn(0x7f4b380ab610 msgr2=0x7f4b380c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3659c640 1 --2- 172.21.15.83:0/3903081311 >> v2:172.21.15.184:3300/0 conn(0x7f4b380ab610 0x7f4b380c3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3659c640 1 -- 172.21.15.83:0/3903081311 >> v1:172.21.15.184:6789/0 conn(0x7f4b380c6fc0 legacy=0x7f4b38157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:02.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.773+0000 7f4b3659c640 1 -- 172.21.15.83:0/3903081311 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b380c8080 con 0x7f4b3806cef0 2024-09-19T07:31:02.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b3659c640 1 --2- 172.21.15.83:0/3903081311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b3806cef0 0x7f4b380abe80 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4b380a8250 tx=0x7f4b2801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6d8cc45ab5ac5a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:02.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b35d9b640 1 -- 172.21.15.83:0/3903081311 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b280ab5f0 con 0x7f4b3806cef0 2024-09-19T07:31:02.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b35d9b640 1 -- 172.21.15.83:0/3903081311 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b280b0070 con 0x7f4b3806cef0 2024-09-19T07:31:02.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b35d9b640 1 -- 172.21.15.83:0/3903081311 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b280ab9b0 con 0x7f4b3806cef0 2024-09-19T07:31:02.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b3cfda640 1 -- 172.21.15.83:0/3903081311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b3806cef0 msgr2=0x7f4b380abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:02.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b3cfda640 1 --2- 172.21.15.83:0/3903081311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b3806cef0 0x7f4b380abe80 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4b380a8250 tx=0x7f4b2801c3d0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b3cfda640 1 -- 172.21.15.83:0/3903081311 shutdown_connections 2024-09-19T07:31:02.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b3cfda640 1 --2- 172.21.15.83:0/3903081311 >> v2:172.21.15.184:3300/0 conn(0x7f4b380ab610 0x7f4b380c3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b3cfda640 1 --2- 172.21.15.83:0/3903081311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b3806cef0 0x7f4b380abe80 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b3cfda640 1 -- 172.21.15.83:0/3903081311 >> 172.21.15.83:0/3903081311 conn(0x7f4b380aa910 msgr2=0x7f4b380c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:02.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b3cfda640 1 -- 172.21.15.83:0/3903081311 shutdown_connections 2024-09-19T07:31:02.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b3cfda640 1 -- 172.21.15.83:0/3903081311 wait complete. 2024-09-19T07:31:02.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b3cfda640 1 Processor -- start 2024-09-19T07:31:02.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.777+0000 7f4b3cfda640 1 -- start start 2024-09-19T07:31:02.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b3cfda640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b380ab610 0x7f4b381538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:02.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b3cfda640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b380c6fc0 0x7f4b3814c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:02.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b3cfda640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b3814f8f0 0x7f4b3814cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:02.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b3cfda640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3806d450 con 0x7f4b380c6fc0 2024-09-19T07:31:02.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b3cfda640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3806d2d0 con 0x7f4b380ab610 2024-09-19T07:31:02.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b3cfda640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4b38157fa0 con 0x7f4b3814f8f0 2024-09-19T07:31:02.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b380c6fc0 0x7f4b3814c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:02.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b380c6fc0 0x7f4b3814c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43948/0 (socket says 172.21.15.83:43948) 2024-09-19T07:31:02.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2dd9b640 1 -- 172.21.15.83:0/4237185343 learned_addr learned my addr 172.21.15.83:0/4237185343 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:02.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b3659c640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b380ab610 0x7f4b381538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:02.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b36d9d640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b3814f8f0 0x7f4b3814cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:02.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2dd9b640 1 -- 172.21.15.83:0/4237185343 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b3814f8f0 msgr2=0x7f4b3814cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:02.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2dd9b640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b3814f8f0 0x7f4b3814cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2dd9b640 1 -- 172.21.15.83:0/4237185343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b380ab610 msgr2=0x7f4b381538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:02.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2dd9b640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b380ab610 0x7f4b381538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:02.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2dd9b640 1 -- 172.21.15.83:0/4237185343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b380be230 con 0x7f4b380c6fc0 2024-09-19T07:31:02.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2dd9b640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b380c6fc0 0x7f4b3814c6c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f4b2000d800 tx=0x7f4b2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:02.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2f7fe640 1 -- 172.21.15.83:0/4237185343 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2000b6c0 con 0x7f4b380c6fc0 2024-09-19T07:31:02.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b3cfda640 1 -- 172.21.15.83:0/4237185343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b380be850 con 0x7f4b380c6fc0 2024-09-19T07:31:02.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2f7fe640 1 -- 172.21.15.83:0/4237185343 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b20019070 con 0x7f4b380c6fc0 2024-09-19T07:31:02.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2f7fe640 1 -- 172.21.15.83:0/4237185343 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2000ba60 con 0x7f4b380c6fc0 2024-09-19T07:31:02.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b3cfda640 1 -- 172.21.15.83:0/4237185343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b380bdf40 con 0x7f4b380c6fc0 2024-09-19T07:31:02.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b3cfda640 1 -- 172.21.15.83:0/4237185343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b380c73d0 con 0x7f4b380c6fc0 2024-09-19T07:31:02.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2f7fe640 1 -- 172.21.15.83:0/4237185343 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4b2000bc00 con 0x7f4b380c6fc0 2024-09-19T07:31:02.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.781+0000 7f4b2f7fe640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4b180678c0 0x7f4b18069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:02.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.785+0000 7f4b2f7fe640 1 -- 172.21.15.83:0/4237185343 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f4b20086910 con 0x7f4b380c6fc0 2024-09-19T07:31:02.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.785+0000 7f4b3659c640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4b180678c0 0x7f4b18069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:02.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.785+0000 7f4b3659c640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4b180678c0 0x7f4b18069d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4b280af000 tx=0x7f4b280a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:02.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:02.789+0000 7f4b2f7fe640 1 -- 172.21.15.83:0/4237185343 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4b200564b0 con 0x7f4b380c6fc0 2024-09-19T07:31:02.859 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:31:02.861 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:31:03.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.081+0000 7f4b3cfda640 1 -- 172.21.15.83:0/4237185343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b381540f0 con 0x7f4b380c6fc0 2024-09-19T07:31:03.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.081+0000 7f4b2f7fe640 1 -- 172.21.15.83:0/4237185343 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+6540 (secure 0 0 0) 0x7f4b2005a7b0 con 0x7f4b380c6fc0 2024-09-19T07:31:03.077 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:03.078 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":95,"btime":"2024-09-19T07:30:58:147417+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6447,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:31:03.078 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 95 2024-09-19T07:31:03.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.085+0000 7f4b3cfda640 1 -- 172.21.15.83:0/4237185343 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4b180678c0 msgr2=0x7f4b18069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.085+0000 7f4b3cfda640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4b180678c0 0x7f4b18069d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4b280af000 tx=0x7f4b280a7000 comp rx=0 tx=0).stop 2024-09-19T07:31:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.085+0000 7f4b3cfda640 1 -- 172.21.15.83:0/4237185343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b380c6fc0 msgr2=0x7f4b3814c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.085+0000 7f4b3cfda640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b380c6fc0 0x7f4b3814c6c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f4b2000d800 tx=0x7f4b2000dcc0 comp rx=0 tx=0).stop 2024-09-19T07:31:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.085+0000 7f4b3cfda640 1 -- 172.21.15.83:0/4237185343 shutdown_connections 2024-09-19T07:31:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.085+0000 7f4b3cfda640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4b180678c0 0x7f4b18069d80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.085+0000 7f4b3cfda640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b3814f8f0 0x7f4b3814cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.085+0000 7f4b3cfda640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b380c6fc0 0x7f4b3814c6c0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.085+0000 7f4b3cfda640 1 --2- 172.21.15.83:0/4237185343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b380ab610 0x7f4b381538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.085+0000 7f4b3cfda640 1 -- 172.21.15.83:0/4237185343 >> 172.21.15.83:0/4237185343 conn(0x7f4b380aa910 msgr2=0x7f4b380abdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.085+0000 7f4b3cfda640 1 -- 172.21.15.83:0/4237185343 shutdown_connections 2024-09-19T07:31:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:03.085+0000 7f4b3cfda640 1 -- 172.21.15.83:0/4237185343 wait complete. 2024-09-19T07:31:03.100 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:31:05.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:05.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.257+0000 7f3e7837c640 1 Processor -- start 2024-09-19T07:31:05.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.257+0000 7f3e7837c640 1 -- start start 2024-09-19T07:31:05.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e7837c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e70154540 0x7f3e70157490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:05.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e7837c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e70151750 con 0x7f3e7014a500 2024-09-19T07:31:05.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e7837c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e700c7e90 con 0x7f3e701579d0 2024-09-19T07:31:05.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e7837c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3e70073110 con 0x7f3e70154540 2024-09-19T07:31:05.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e75917640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e70154540 0x7f3e70157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:05.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e76118640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3e7014a500 0x7f3e700c4010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37028/0 (socket says 172.21.15.83:37028) 2024-09-19T07:31:05.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e76118640 1 -- 172.21.15.83:0/2963564277 learned_addr learned my addr 172.21.15.83:0/2963564277 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:31:05.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e75917640 1 -- 172.21.15.83:0/2963564277 >> v1:172.21.15.184:6789/0 conn(0x7f3e701579d0 legacy=0x7f3e7015e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:05.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e75917640 1 -- 172.21.15.83:0/2963564277 >> v1:172.21.15.83:6789/0 conn(0x7f3e7014a500 legacy=0x7f3e700c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:05.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e75917640 1 -- 172.21.15.83:0/2963564277 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e700733f0 con 0x7f3e70154540 2024-09-19T07:31:05.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e75116640 1 -- 172.21.15.83:0/2963564277 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2509638735 0 0) 0x7f3e6c001120 con 0x7f3e7014a500 2024-09-19T07:31:05.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e75917640 1 --2- 172.21.15.83:0/2963564277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e70154540 0x7f3e70157490 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f3e600097c0 tx=0x7f3e6001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc754826bdd642e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e75116640 1 -- 172.21.15.83:0/2963564277 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e600ab5f0 con 0x7f3e70154540 2024-09-19T07:31:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e75116640 1 -- 172.21.15.83:0/2963564277 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e600b0070 con 0x7f3e70154540 2024-09-19T07:31:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.261+0000 7f3e75116640 1 -- 172.21.15.83:0/2963564277 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e600ab9b0 con 0x7f3e70154540 2024-09-19T07:31:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 -- 172.21.15.83:0/2963564277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e70154540 msgr2=0x7f3e70157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 --2- 172.21.15.83:0/2963564277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e70154540 0x7f3e70157490 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f3e600097c0 tx=0x7f3e6001c430 comp rx=0 tx=0).stop 2024-09-19T07:31:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 -- 172.21.15.83:0/2963564277 shutdown_connections 2024-09-19T07:31:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 --2- 172.21.15.83:0/2963564277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e70154540 0x7f3e70157490 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 -- 172.21.15.83:0/2963564277 >> 172.21.15.83:0/2963564277 conn(0x7f3e700c9610 msgr2=0x7f3e700c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 -- 172.21.15.83:0/2963564277 shutdown_connections 2024-09-19T07:31:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 -- 172.21.15.83:0/2963564277 wait complete. 2024-09-19T07:31:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 Processor -- start 2024-09-19T07:31:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 -- start start 2024-09-19T07:31:05.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7014a500 0x7f3e7014d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:05.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e70154540 0x7f3e70156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:05.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e701579d0 0x7f3e701ecac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:05.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3e700c7850 con 0x7f3e7014a500 2024-09-19T07:31:05.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3e700c76d0 con 0x7f3e701579d0 2024-09-19T07:31:05.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e7837c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3e700c4720 con 0x7f3e70154540 2024-09-19T07:31:05.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e76118640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7014a500 0x7f3e7014d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:05.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e76118640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7014a500 0x7f3e7014d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43966/0 (socket says 172.21.15.83:43966) 2024-09-19T07:31:05.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e76118640 1 -- 172.21.15.83:0/3939867496 learned_addr learned my addr 172.21.15.83:0/3939867496 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:05.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e76919640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e701579d0 0x7f3e701ecac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:05.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e75917640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e70154540 0x7f3e70156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:05.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e76118640 1 -- 172.21.15.83:0/3939867496 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e70154540 msgr2=0x7f3e70156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:05.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e76118640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e70154540 0x7f3e70156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:05.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e76118640 1 -- 172.21.15.83:0/3939867496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e701579d0 msgr2=0x7f3e701ecac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:05.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e76118640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e701579d0 0x7f3e701ecac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:05.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e76118640 1 -- 172.21.15.83:0/3939867496 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e701f0e90 con 0x7f3e7014a500 2024-09-19T07:31:05.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.265+0000 7f3e76118640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7014a500 0x7f3e7014d720 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00c780 tx=0x7f3e6c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:05.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.269+0000 7f3e66ffd640 1 -- 172.21.15.83:0/3939867496 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c018070 con 0x7f3e7014a500 2024-09-19T07:31:05.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.269+0000 7f3e66ffd640 1 -- 172.21.15.83:0/3939867496 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e6c0072f0 con 0x7f3e7014a500 2024-09-19T07:31:05.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.269+0000 7f3e66ffd640 1 -- 172.21.15.83:0/3939867496 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c0047f0 con 0x7f3e7014a500 2024-09-19T07:31:05.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.269+0000 7f3e7837c640 1 -- 172.21.15.83:0/3939867496 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e7014ae30 con 0x7f3e7014a500 2024-09-19T07:31:05.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.269+0000 7f3e7837c640 1 -- 172.21.15.83:0/3939867496 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e701f00d0 con 0x7f3e7014a500 2024-09-19T07:31:05.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.269+0000 7f3e7837c640 1 -- 172.21.15.83:0/3939867496 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e70157e20 con 0x7f3e7014a500 2024-09-19T07:31:05.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.269+0000 7f3e66ffd640 1 -- 172.21.15.83:0/3939867496 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3e6c004990 con 0x7f3e7014a500 2024-09-19T07:31:05.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.269+0000 7f3e66ffd640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e540678c0 0x7f3e54069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:05.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.269+0000 7f3e66ffd640 1 -- 172.21.15.83:0/3939867496 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f3e6c0859f0 con 0x7f3e7014a500 2024-09-19T07:31:05.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.269+0000 7f3e75917640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e540678c0 0x7f3e54069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:05.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.273+0000 7f3e75917640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e540678c0 0x7f3e54069d80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3e600af000 tx=0x7f3e600a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:05.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.277+0000 7f3e66ffd640 1 -- 172.21.15.83:0/3939867496 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3e70151750 con 0x7f3e7014a500 2024-09-19T07:31:05.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.505+0000 7f3e7837c640 1 -- 172.21.15.83:0/3939867496 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e70147e30 con 0x7f3e7014a500 2024-09-19T07:31:05.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e66ffd640 1 -- 172.21.15.83:0/3939867496 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+6540 (secure 0 0 0) 0x7f3e6c055c60 con 0x7f3e7014a500 2024-09-19T07:31:05.504 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:05.504 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":95,"btime":"2024-09-19T07:30:58:147417+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6447,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:31:05.505 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 95 2024-09-19T07:31:05.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e7837c640 1 -- 172.21.15.83:0/3939867496 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e540678c0 msgr2=0x7f3e54069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:05.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e7837c640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e540678c0 0x7f3e54069d80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3e600af000 tx=0x7f3e600a7000 comp rx=0 tx=0).stop 2024-09-19T07:31:05.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e7837c640 1 -- 172.21.15.83:0/3939867496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7014a500 msgr2=0x7f3e7014d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:05.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e7837c640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7014a500 0x7f3e7014d720 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00c780 tx=0x7f3e6c00cc40 comp rx=0 tx=0).stop 2024-09-19T07:31:05.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e7837c640 1 -- 172.21.15.83:0/3939867496 shutdown_connections 2024-09-19T07:31:05.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e7837c640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e540678c0 0x7f3e54069d80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:05.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e7837c640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e701579d0 0x7f3e701ecac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:05.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e7837c640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e70154540 0x7f3e70156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:05.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e7837c640 1 --2- 172.21.15.83:0/3939867496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e7014a500 0x7f3e7014d720 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:05.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e7837c640 1 -- 172.21.15.83:0/3939867496 >> 172.21.15.83:0/3939867496 conn(0x7f3e700c9610 msgr2=0x7f3e700c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:05.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e7837c640 1 -- 172.21.15.83:0/3939867496 shutdown_connections 2024-09-19T07:31:05.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:05.509+0000 7f3e7837c640 1 -- 172.21.15.83:0/3939867496 wait complete. 2024-09-19T07:31:05.525 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:31:07.526 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:07.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c9417640 1 Processor -- start 2024-09-19T07:31:07.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c9417640 1 -- start start 2024-09-19T07:31:07.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c9417640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f56c40c8580 0x7f56c40c89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:07.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c9417640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f56c4155ed0 0x7f56c41562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:07.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c9417640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56c40cac10 con 0x7f56c4153490 2024-09-19T07:31:07.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c9417640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f56c406db30 con 0x7f56c4155ed0 2024-09-19T07:31:07.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c9417640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f56c40caeb0 con 0x7f56c40c8580 2024-09-19T07:31:07.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f56c4155ed0 0x7f56c41562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:07.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f56c4155ed0 0x7f56c41562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43978/0 (socket says 172.21.15.83:43978) 2024-09-19T07:31:07.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c37fe640 1 -- 172.21.15.83:0/2801774896 learned_addr learned my addr 172.21.15.83:0/2801774896 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:07.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c27fc640 1 --2- 172.21.15.83:0/2801774896 >> v2:172.21.15.184:3300/0 conn(0x7f56c40c8580 0x7f56c40c89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:07.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c37fe640 1 -- 172.21.15.83:0/2801774896 >> v2:172.21.15.184:3300/0 conn(0x7f56c40c8580 msgr2=0x7f56c40c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:07.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c37fe640 1 --2- 172.21.15.83:0/2801774896 >> v2:172.21.15.184:3300/0 conn(0x7f56c40c8580 0x7f56c40c89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:07.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c37fe640 1 -- 172.21.15.83:0/2801774896 >> v1:172.21.15.184:6789/0 conn(0x7f56c4153490 legacy=0x7f56c4159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:07.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c37fe640 1 -- 172.21.15.83:0/2801774896 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c40cb190 con 0x7f56c4155ed0 2024-09-19T07:31:07.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.685+0000 7f56c27fc640 1 --2- 172.21.15.83:0/2801774896 >> v2:172.21.15.184:3300/0 conn(0x7f56c40c8580 0x7f56c40c89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:31:07.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c37fe640 1 --2- 172.21.15.83:0/2801774896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c4155ed0 0x7f56c41562e0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f56c40cb610 tx=0x7f56bc01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e91e53e1eb3986 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:07.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c1ffb640 1 -- 172.21.15.83:0/2801774896 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc0a5530 con 0x7f56c4155ed0 2024-09-19T07:31:07.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c1ffb640 1 -- 172.21.15.83:0/2801774896 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56bc0a56d0 con 0x7f56c4155ed0 2024-09-19T07:31:07.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c1ffb640 1 -- 172.21.15.83:0/2801774896 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc0a5aa0 con 0x7f56c4155ed0 2024-09-19T07:31:07.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c9417640 1 -- 172.21.15.83:0/2801774896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c4155ed0 msgr2=0x7f56c41562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:07.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c9417640 1 --2- 172.21.15.83:0/2801774896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c4155ed0 0x7f56c41562e0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f56c40cb610 tx=0x7f56bc01c870 comp rx=0 tx=0).stop 2024-09-19T07:31:07.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c9417640 1 -- 172.21.15.83:0/2801774896 shutdown_connections 2024-09-19T07:31:07.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c9417640 1 --2- 172.21.15.83:0/2801774896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c4155ed0 0x7f56c41562e0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:07.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c9417640 1 --2- 172.21.15.83:0/2801774896 >> v2:172.21.15.184:3300/0 conn(0x7f56c40c8580 0x7f56c40c89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:07.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c9417640 1 -- 172.21.15.83:0/2801774896 >> 172.21.15.83:0/2801774896 conn(0x7f56c40ad7d0 msgr2=0x7f56c40adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:07.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c9417640 1 -- 172.21.15.83:0/2801774896 shutdown_connections 2024-09-19T07:31:07.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c9417640 1 -- 172.21.15.83:0/2801774896 wait complete. 2024-09-19T07:31:07.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c9417640 1 Processor -- start 2024-09-19T07:31:07.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.689+0000 7f56c9417640 1 -- start start 2024-09-19T07:31:07.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c9417640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56c40c8580 0x7f56c40ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:07.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c9417640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c4153490 0x7f56c40ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:07.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c9417640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56c4155ed0 0x7f56c40aec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:07.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c9417640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f56c406be00 con 0x7f56c4153490 2024-09-19T07:31:07.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c9417640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f56c406bc80 con 0x7f56c4155ed0 2024-09-19T07:31:07.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c9417640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f56c4162e50 con 0x7f56c40c8580 2024-09-19T07:31:07.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c4153490 0x7f56c40ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:07.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c4153490 0x7f56c40ae6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43986/0 (socket says 172.21.15.83:43986) 2024-09-19T07:31:07.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c27fc640 1 -- 172.21.15.83:0/1596052256 learned_addr learned my addr 172.21.15.83:0/1596052256 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c37fe640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56c4155ed0 0x7f56c40aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c2ffd640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56c40c8580 0x7f56c40ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c27fc640 1 -- 172.21.15.83:0/1596052256 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56c40c8580 msgr2=0x7f56c40ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c27fc640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56c40c8580 0x7f56c40ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c27fc640 1 -- 172.21.15.83:0/1596052256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56c4155ed0 msgr2=0x7f56c40aec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c27fc640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56c4155ed0 0x7f56c40aec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c27fc640 1 -- 172.21.15.83:0/1596052256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c4153d70 con 0x7f56c4153490 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c27fc640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c4153490 0x7f56c40ae6e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f56b800c820 tx=0x7f56b800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56ab7fe640 1 -- 172.21.15.83:0/1596052256 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8015070 con 0x7f56c4153490 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56ab7fe640 1 -- 172.21.15.83:0/1596052256 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b80079d0 con 0x7f56c4153490 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c9417640 1 -- 172.21.15.83:0/1596052256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c41557b0 con 0x7f56c4153490 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56ab7fe640 1 -- 172.21.15.83:0/1596052256 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8007de0 con 0x7f56c4153490 2024-09-19T07:31:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c9417640 1 -- 172.21.15.83:0/1596052256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c40c7e70 con 0x7f56c4153490 2024-09-19T07:31:07.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c9417640 1 -- 172.21.15.83:0/1596052256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c40c89c0 con 0x7f56c4153490 2024-09-19T07:31:07.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56ab7fe640 1 -- 172.21.15.83:0/1596052256 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f56b8018070 con 0x7f56c4153490 2024-09-19T07:31:07.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56ab7fe640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f56b00678c0 0x7f56b0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:07.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56ab7fe640 1 -- 172.21.15.83:0/1596052256 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f56b8086430 con 0x7f56c4153490 2024-09-19T07:31:07.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.693+0000 7f56c2ffd640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f56b00678c0 0x7f56b0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:07.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.697+0000 7f56c2ffd640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f56b00678c0 0x7f56b0069d80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f56c41604f0 tx=0x7f56b4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:07.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.701+0000 7f56ab7fe640 1 -- 172.21.15.83:0/1596052256 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f56b80566a0 con 0x7f56c4153490 2024-09-19T07:31:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.917+0000 7f56c9417640 1 -- 172.21.15.83:0/1596052256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56c40ca9f0 con 0x7f56c4153490 2024-09-19T07:31:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.917+0000 7f56ab7fe640 1 -- 172.21.15.83:0/1596052256 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+6540 (secure 0 0 0) 0x7f56b8055fa0 con 0x7f56c4153490 2024-09-19T07:31:07.913 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:07.913 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":95,"btime":"2024-09-19T07:30:58:147417+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6447,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:31:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 95 2024-09-19T07:31:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.921+0000 7f56c9417640 1 -- 172.21.15.83:0/1596052256 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f56b00678c0 msgr2=0x7f56b0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.921+0000 7f56c9417640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f56b00678c0 0x7f56b0069d80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f56c41604f0 tx=0x7f56b4008040 comp rx=0 tx=0).stop 2024-09-19T07:31:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.921+0000 7f56c9417640 1 -- 172.21.15.83:0/1596052256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c4153490 msgr2=0x7f56c40ae6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.921+0000 7f56c9417640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c4153490 0x7f56c40ae6e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f56b800c820 tx=0x7f56b800cce0 comp rx=0 tx=0).stop 2024-09-19T07:31:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.921+0000 7f56c9417640 1 -- 172.21.15.83:0/1596052256 shutdown_connections 2024-09-19T07:31:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.921+0000 7f56c9417640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f56b00678c0 0x7f56b0069d80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.921+0000 7f56c9417640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56c4155ed0 0x7f56c40aec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.921+0000 7f56c9417640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56c4153490 0x7f56c40ae6e0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.921+0000 7f56c9417640 1 --2- 172.21.15.83:0/1596052256 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56c40c8580 0x7f56c40ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.921+0000 7f56c9417640 1 -- 172.21.15.83:0/1596052256 >> 172.21.15.83:0/1596052256 conn(0x7f56c40ad7d0 msgr2=0x7f56c414e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.921+0000 7f56c9417640 1 -- 172.21.15.83:0/1596052256 shutdown_connections 2024-09-19T07:31:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:07.921+0000 7f56c9417640 1 -- 172.21.15.83:0/1596052256 wait complete. 2024-09-19T07:31:07.936 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:31:09.938 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:10.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd693088640 1 Processor -- start 2024-09-19T07:31:10.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd693088640 1 -- start start 2024-09-19T07:31:10.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd693088640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd68c06cef0 0x7fd68c0728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:10.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd693088640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd68c0c4990 0x7fd68c14fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:10.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd693088640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd68c150140 0x7fd68c156070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:10.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd693088640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd68c0c3e20 con 0x7fd68c06cef0 2024-09-19T07:31:10.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd693088640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd68c0c2f90 con 0x7fd68c0c4990 2024-09-19T07:31:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd693088640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd68c0c3210 con 0x7fd68c150140 2024-09-19T07:31:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd690e24640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd68c06cef0 0x7fd68c0728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd690e24640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd68c06cef0 0x7fd68c0728d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43996/0 (socket says 172.21.15.83:43996) 2024-09-19T07:31:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd690e24640 1 -- 172.21.15.83:0/3287826358 learned_addr learned my addr 172.21.15.83:0/3287826358 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd691625640 1 --2- 172.21.15.83:0/3287826358 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd68c150140 0x7fd68c156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd68bfff640 1 --2- 172.21.15.83:0/3287826358 >> v2:172.21.15.184:3300/0 conn(0x7fd68c0c4990 0x7fd68c14fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd690e24640 1 -- 172.21.15.83:0/3287826358 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd68c150140 msgr2=0x7fd68c156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd690e24640 1 --2- 172.21.15.83:0/3287826358 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd68c150140 0x7fd68c156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd690e24640 1 -- 172.21.15.83:0/3287826358 >> v2:172.21.15.184:3300/0 conn(0x7fd68c0c4990 msgr2=0x7fd68c14fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd690e24640 1 --2- 172.21.15.83:0/3287826358 >> v2:172.21.15.184:3300/0 conn(0x7fd68c0c4990 0x7fd68c14fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd690e24640 1 -- 172.21.15.83:0/3287826358 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68c0c34f0 con 0x7fd68c06cef0 2024-09-19T07:31:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.080+0000 7fd68bfff640 1 --2- 172.21.15.83:0/3287826358 >> v2:172.21.15.184:3300/0 conn(0x7fd68c0c4990 0x7fd68c14fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:31:10.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd690e24640 1 --2- 172.21.15.83:0/3287826358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd68c06cef0 0x7fd68c0728d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd68c073370 tx=0x7fd68401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b52e68e11f315ca server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:10.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd68b7fe640 1 -- 172.21.15.83:0/3287826358 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6840b2070 con 0x7fd68c06cef0 2024-09-19T07:31:10.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd68b7fe640 1 -- 172.21.15.83:0/3287826358 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6840a9070 con 0x7fd68c06cef0 2024-09-19T07:31:10.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd68b7fe640 1 -- 172.21.15.83:0/3287826358 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6840ad880 con 0x7fd68c06cef0 2024-09-19T07:31:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd693088640 1 -- 172.21.15.83:0/3287826358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd68c06cef0 msgr2=0x7fd68c0728d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd693088640 1 --2- 172.21.15.83:0/3287826358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd68c06cef0 0x7fd68c0728d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd68c073370 tx=0x7fd68401c490 comp rx=0 tx=0).stop 2024-09-19T07:31:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd693088640 1 -- 172.21.15.83:0/3287826358 shutdown_connections 2024-09-19T07:31:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd693088640 1 --2- 172.21.15.83:0/3287826358 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd68c150140 0x7fd68c156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd693088640 1 --2- 172.21.15.83:0/3287826358 >> v2:172.21.15.184:3300/0 conn(0x7fd68c0c4990 0x7fd68c14fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:10.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd693088640 1 --2- 172.21.15.83:0/3287826358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd68c06cef0 0x7fd68c0728d0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:10.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd693088640 1 -- 172.21.15.83:0/3287826358 >> 172.21.15.83:0/3287826358 conn(0x7fd68c0c5090 msgr2=0x7fd68c0c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:10.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd693088640 1 -- 172.21.15.83:0/3287826358 shutdown_connections 2024-09-19T07:31:10.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.084+0000 7fd693088640 1 -- 172.21.15.83:0/3287826358 wait complete. 2024-09-19T07:31:10.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd693088640 1 Processor -- start 2024-09-19T07:31:10.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd693088640 1 -- start start 2024-09-19T07:31:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd693088640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd68c06cef0 0x7fd68c14b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd693088640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd68c0c4990 0x7fd68c0751f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd693088640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd68c150140 0x7fd68c1f2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd693088640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c152880 con 0x7fd68c06cef0 2024-09-19T07:31:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd693088640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c06d520 con 0x7fd68c0c4990 2024-09-19T07:31:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd693088640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd68c152a00 con 0x7fd68c150140 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd690e24640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd68c06cef0 0x7fd68c14b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd690e24640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd68c06cef0 0x7fd68c14b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44012/0 (socket says 172.21.15.83:44012) 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd690e24640 1 -- 172.21.15.83:0/625125404 learned_addr learned my addr 172.21.15.83:0/625125404 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd690e24640 1 -- 172.21.15.83:0/625125404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd68c150140 msgr2=0x7fd68c1f2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd68bfff640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd68c0c4990 0x7fd68c0751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd691625640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd68c150140 0x7fd68c1f2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd690e24640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd68c150140 0x7fd68c1f2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd690e24640 1 -- 172.21.15.83:0/625125404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd68c0c4990 msgr2=0x7fd68c0751f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd690e24640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd68c0c4990 0x7fd68c0751f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd690e24640 1 -- 172.21.15.83:0/625125404 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68c149730 con 0x7fd68c06cef0 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd68bfff640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd68c0c4990 0x7fd68c0751f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd690e24640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd68c06cef0 0x7fd68c14b7a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd684000c10 tx=0x7fd68401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd6897fa640 1 -- 172.21.15.83:0/625125404 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6840bc020 con 0x7fd68c06cef0 2024-09-19T07:31:10.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd6897fa640 1 -- 172.21.15.83:0/625125404 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6840b8420 con 0x7fd68c06cef0 2024-09-19T07:31:10.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd693088640 1 -- 172.21.15.83:0/625125404 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd68c15b8d0 con 0x7fd68c06cef0 2024-09-19T07:31:10.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd6897fa640 1 -- 172.21.15.83:0/625125404 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6840b2040 con 0x7fd68c06cef0 2024-09-19T07:31:10.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd693088640 1 -- 172.21.15.83:0/625125404 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd68c1f1c40 con 0x7fd68c06cef0 2024-09-19T07:31:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.088+0000 7fd693088640 1 -- 172.21.15.83:0/625125404 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd68c14fc00 con 0x7fd68c06cef0 2024-09-19T07:31:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.092+0000 7fd6897fa640 1 -- 172.21.15.83:0/625125404 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd6840bf020 con 0x7fd68c06cef0 2024-09-19T07:31:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.092+0000 7fd6897fa640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd6780678c0 0x7fd678069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.092+0000 7fd6897fa640 1 -- 172.21.15.83:0/625125404 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7fd6841208d0 con 0x7fd68c06cef0 2024-09-19T07:31:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.092+0000 7fd68bfff640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd6780678c0 0x7fd678069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.092+0000 7fd68bfff640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd6780678c0 0x7fd678069d80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd68c1f2d20 tx=0x7fd680008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:10.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.096+0000 7fd6897fa640 1 -- 172.21.15.83:0/625125404 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd6840c6030 con 0x7fd68c06cef0 2024-09-19T07:31:10.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.340+0000 7fd693088640 1 -- 172.21.15.83:0/625125404 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd68c0c2a50 con 0x7fd68c06cef0 2024-09-19T07:31:10.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.340+0000 7fd6897fa640 1 -- 172.21.15.83:0/625125404 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+6540 (secure 0 0 0) 0x7fd68c149730 con 0x7fd68c06cef0 2024-09-19T07:31:10.339 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:10.339 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":95,"btime":"2024-09-19T07:30:58:147417+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6447,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:31:10.339 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 95 2024-09-19T07:31:10.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.344+0000 7fd693088640 1 -- 172.21.15.83:0/625125404 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd6780678c0 msgr2=0x7fd678069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:10.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.344+0000 7fd693088640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd6780678c0 0x7fd678069d80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd68c1f2d20 tx=0x7fd680008040 comp rx=0 tx=0).stop 2024-09-19T07:31:10.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.344+0000 7fd693088640 1 -- 172.21.15.83:0/625125404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd68c06cef0 msgr2=0x7fd68c14b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:10.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.344+0000 7fd693088640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd68c06cef0 0x7fd68c14b7a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd684000c10 tx=0x7fd68401acd0 comp rx=0 tx=0).stop 2024-09-19T07:31:10.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.344+0000 7fd693088640 1 -- 172.21.15.83:0/625125404 shutdown_connections 2024-09-19T07:31:10.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.344+0000 7fd693088640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd6780678c0 0x7fd678069d80 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:10.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.344+0000 7fd693088640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd68c150140 0x7fd68c1f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:10.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.344+0000 7fd693088640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd68c0c4990 0x7fd68c0751f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:10.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.344+0000 7fd693088640 1 --2- 172.21.15.83:0/625125404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd68c06cef0 0x7fd68c14b7a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:10.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.344+0000 7fd693088640 1 -- 172.21.15.83:0/625125404 >> 172.21.15.83:0/625125404 conn(0x7fd68c0c5090 msgr2=0x7fd68c150f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:10.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.344+0000 7fd693088640 1 -- 172.21.15.83:0/625125404 shutdown_connections 2024-09-19T07:31:10.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:10.344+0000 7fd693088640 1 -- 172.21.15.83:0/625125404 wait complete. 2024-09-19T07:31:10.364 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:31:12.365 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:12.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.512+0000 7fc2e38c4640 1 Processor -- start 2024-09-19T07:31:12.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.512+0000 7fc2e38c4640 1 -- start start 2024-09-19T07:31:12.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e38c4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc2dc153570 0x7fc2dc0c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e38c4640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc2dc0c4130 0x7fc2dc159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e38c4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2dc0c3c20 con 0x7fc2dc0c5110 2024-09-19T07:31:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e38c4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc2dc0c89f0 con 0x7fc2dc153570 2024-09-19T07:31:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e38c4640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc2dc0c8cd0 con 0x7fc2dc0c4130 2024-09-19T07:31:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e0e5f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc2dc0c5110 0x7fc2dc159270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45594/0 (socket says 172.21.15.83:45594) 2024-09-19T07:31:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e0e5f640 1 -- 172.21.15.83:0/3850353548 learned_addr learned my addr 172.21.15.83:0/3850353548 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:31:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e1660640 1 --2- 172.21.15.83:0/3850353548 >> v2:172.21.15.83:3300/0 conn(0x7fc2dc153570 0x7fc2dc0c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e1e61640 1 --2- 172.21.15.83:0/3850353548 >> v2:172.21.15.184:3300/0 conn(0x7fc2dc0c4130 0x7fc2dc159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2d3fff640 1 -- 172.21.15.83:0/3850353548 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1540643953 0 0) 0x7fc2dc0c3c20 con 0x7fc2dc0c5110 2024-09-19T07:31:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2d3fff640 1 -- 172.21.15.83:0/3850353548 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2c8003530 con 0x7fc2dc0c5110 2024-09-19T07:31:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e1660640 1 -- 172.21.15.83:0/3850353548 >> v2:172.21.15.184:3300/0 conn(0x7fc2dc0c4130 msgr2=0x7fc2dc159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e1660640 1 --2- 172.21.15.83:0/3850353548 >> v2:172.21.15.184:3300/0 conn(0x7fc2dc0c4130 0x7fc2dc159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:12.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e1660640 1 -- 172.21.15.83:0/3850353548 >> v1:172.21.15.83:6789/0 conn(0x7fc2dc0c5110 legacy=0x7fc2dc159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:12.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.516+0000 7fc2e1660640 1 -- 172.21.15.83:0/3850353548 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc0c8fb0 con 0x7fc2dc153570 2024-09-19T07:31:12.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e1660640 1 --2- 172.21.15.83:0/3850353548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2dc153570 0x7fc2dc0c4bd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc071ba0 tx=0x7fc2d801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69d362c1c2deb68 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:12.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2d3fff640 1 -- 172.21.15.83:0/3850353548 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80b1070 con 0x7fc2dc153570 2024-09-19T07:31:12.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2d3fff640 1 -- 172.21.15.83:0/3850353548 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d80ac600 con 0x7fc2dc153570 2024-09-19T07:31:12.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2d3fff640 1 -- 172.21.15.83:0/3850353548 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80aca10 con 0x7fc2dc153570 2024-09-19T07:31:12.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 -- 172.21.15.83:0/3850353548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2dc153570 msgr2=0x7fc2dc0c4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:12.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 --2- 172.21.15.83:0/3850353548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2dc153570 0x7fc2dc0c4bd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc071ba0 tx=0x7fc2d801c380 comp rx=0 tx=0).stop 2024-09-19T07:31:12.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 -- 172.21.15.83:0/3850353548 shutdown_connections 2024-09-19T07:31:12.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 --2- 172.21.15.83:0/3850353548 >> v2:172.21.15.184:3300/0 conn(0x7fc2dc0c4130 0x7fc2dc159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:12.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 --2- 172.21.15.83:0/3850353548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2dc153570 0x7fc2dc0c4bd0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:12.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 -- 172.21.15.83:0/3850353548 >> 172.21.15.83:0/3850353548 conn(0x7fc2dc0c6110 msgr2=0x7fc2dc0c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:12.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 -- 172.21.15.83:0/3850353548 shutdown_connections 2024-09-19T07:31:12.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 -- 172.21.15.83:0/3850353548 wait complete. 2024-09-19T07:31:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 Processor -- start 2024-09-19T07:31:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 -- start start 2024-09-19T07:31:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2dc0c4130 0x7fc2dc0ab2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2dc0c5110 0x7fc2dc1465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2dc153570 0x7fc2dc14bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc163070 con 0x7fc2dc0c4130 2024-09-19T07:31:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc162ef0 con 0x7fc2dc153570 2024-09-19T07:31:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e38c4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc2dc1631f0 con 0x7fc2dc0c5110 2024-09-19T07:31:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e1660640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2dc0c4130 0x7fc2dc0ab2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e1660640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2dc0c4130 0x7fc2dc0ab2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45520/0 (socket says 172.21.15.83:45520) 2024-09-19T07:31:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e1660640 1 -- 172.21.15.83:0/336052756 learned_addr learned my addr 172.21.15.83:0/336052756 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:12.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e0e5f640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2dc0c5110 0x7fc2dc1465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:12.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e1660640 1 -- 172.21.15.83:0/336052756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2dc0c5110 msgr2=0x7fc2dc1465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:12.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e1e61640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2dc153570 0x7fc2dc14bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:12.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e1660640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2dc0c5110 0x7fc2dc1465c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:12.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e1660640 1 -- 172.21.15.83:0/336052756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2dc153570 msgr2=0x7fc2dc14bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:12.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e1660640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2dc153570 0x7fc2dc14bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:12.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.520+0000 7fc2e1660640 1 -- 172.21.15.83:0/336052756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc145b00 con 0x7fc2dc0c4130 2024-09-19T07:31:12.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.524+0000 7fc2e1660640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2dc0c4130 0x7fc2dc0ab2c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8019fb0 tx=0x7fc2d801abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:12.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.524+0000 7fc2d1ffb640 1 -- 172.21.15.83:0/336052756 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80b1040 con 0x7fc2dc0c4130 2024-09-19T07:31:12.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.524+0000 7fc2d1ffb640 1 -- 172.21.15.83:0/336052756 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d80bc070 con 0x7fc2dc0c4130 2024-09-19T07:31:12.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.524+0000 7fc2e38c4640 1 -- 172.21.15.83:0/336052756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2dc0aa7f0 con 0x7fc2dc0c4130 2024-09-19T07:31:12.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.524+0000 7fc2d1ffb640 1 -- 172.21.15.83:0/336052756 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80b7670 con 0x7fc2dc0c4130 2024-09-19T07:31:12.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.524+0000 7fc2e38c4640 1 -- 172.21.15.83:0/336052756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2dc1460a0 con 0x7fc2dc0c4130 2024-09-19T07:31:12.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.524+0000 7fc2e38c4640 1 -- 172.21.15.83:0/336052756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2dc0c4bd0 con 0x7fc2dc0c4130 2024-09-19T07:31:12.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.524+0000 7fc2d1ffb640 1 -- 172.21.15.83:0/336052756 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc2d80b7810 con 0x7fc2dc0c4130 2024-09-19T07:31:12.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.524+0000 7fc2d1ffb640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2c00678c0 0x7fc2c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:12.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.524+0000 7fc2d1ffb640 1 -- 172.21.15.83:0/336052756 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7fc2d8120640 con 0x7fc2dc0c4130 2024-09-19T07:31:12.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.524+0000 7fc2e0e5f640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2c00678c0 0x7fc2c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:12.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.524+0000 7fc2e0e5f640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2c00678c0 0x7fc2c0069d80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc007fd0 tx=0x7fc2cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:12.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.528+0000 7fc2d1ffb640 1 -- 172.21.15.83:0/336052756 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc2d8016090 con 0x7fc2dc0c4130 2024-09-19T07:31:12.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.760+0000 7fc2e38c4640 1 -- 172.21.15.83:0/336052756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2dc0c32f0 con 0x7fc2dc0c4130 2024-09-19T07:31:12.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.760+0000 7fc2d1ffb640 1 -- 172.21.15.83:0/336052756 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+6540 (secure 0 0 0) 0x7fc2d80f0930 con 0x7fc2dc0c4130 2024-09-19T07:31:12.759 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:12.759 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":95,"btime":"2024-09-19T07:30:58:147417+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6447,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:31:12.759 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 95 2024-09-19T07:31:12.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.764+0000 7fc2e38c4640 1 -- 172.21.15.83:0/336052756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2c00678c0 msgr2=0x7fc2c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:12.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.764+0000 7fc2e38c4640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2c00678c0 0x7fc2c0069d80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc007fd0 tx=0x7fc2cc009040 comp rx=0 tx=0).stop 2024-09-19T07:31:12.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.764+0000 7fc2e38c4640 1 -- 172.21.15.83:0/336052756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2dc0c4130 msgr2=0x7fc2dc0ab2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:12.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.764+0000 7fc2e38c4640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2dc0c4130 0x7fc2dc0ab2c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8019fb0 tx=0x7fc2d801abc0 comp rx=0 tx=0).stop 2024-09-19T07:31:12.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.764+0000 7fc2e38c4640 1 -- 172.21.15.83:0/336052756 shutdown_connections 2024-09-19T07:31:12.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.764+0000 7fc2e38c4640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2c00678c0 0x7fc2c0069d80 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:12.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.764+0000 7fc2e38c4640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2dc153570 0x7fc2dc14bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:12.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.764+0000 7fc2e38c4640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2dc0c5110 0x7fc2dc1465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:12.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.764+0000 7fc2e38c4640 1 --2- 172.21.15.83:0/336052756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2dc0c4130 0x7fc2dc0ab2c0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:12.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.764+0000 7fc2e38c4640 1 -- 172.21.15.83:0/336052756 >> 172.21.15.83:0/336052756 conn(0x7fc2dc0c6110 msgr2=0x7fc2dc15fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:12.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.764+0000 7fc2e38c4640 1 -- 172.21.15.83:0/336052756 shutdown_connections 2024-09-19T07:31:12.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:12.764+0000 7fc2e38c4640 1 -- 172.21.15.83:0/336052756 wait complete. 2024-09-19T07:31:12.778 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:31:14.779 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:14.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f355d640 1 Processor -- start 2024-09-19T07:31:14.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f355d640 1 -- start start 2024-09-19T07:31:14.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f355d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f14ec153600 0x7f14ec0c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:14.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f355d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f14ec0c91b0 0x7f14ec159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:14.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f355d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ec0c4f10 con 0x7f14ec0aee20 2024-09-19T07:31:14.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f355d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f14ec0c7ad0 con 0x7f14ec153600 2024-09-19T07:31:14.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f355d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f14ec0c8380 con 0x7f14ec0c91b0 2024-09-19T07:31:14.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f12f9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f14ec153600 0x7f14ec0c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:14.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f12f9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f14ec153600 0x7f14ec0c49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45526/0 (socket says 172.21.15.83:45526) 2024-09-19T07:31:14.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f12f9640 1 -- 172.21.15.83:0/1954835473 learned_addr learned my addr 172.21.15.83:0/1954835473 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:14.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f0af8640 1 --2- 172.21.15.83:0/1954835473 >> v2:172.21.15.184:3300/0 conn(0x7f14ec0c91b0 0x7f14ec159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:14.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f12f9640 1 -- 172.21.15.83:0/1954835473 >> v2:172.21.15.184:3300/0 conn(0x7f14ec0c91b0 msgr2=0x7f14ec159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:14.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f12f9640 1 --2- 172.21.15.83:0/1954835473 >> v2:172.21.15.184:3300/0 conn(0x7f14ec0c91b0 0x7f14ec159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:14.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f12f9640 1 -- 172.21.15.83:0/1954835473 >> v1:172.21.15.184:6789/0 conn(0x7f14ec0aee20 legacy=0x7f14ec0c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:14.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f12f9640 1 -- 172.21.15.83:0/1954835473 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ec0c7510 con 0x7f14ec153600 2024-09-19T07:31:14.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.912+0000 7f14f0af8640 1 --2- 172.21.15.83:0/1954835473 >> v2:172.21.15.184:3300/0 conn(0x7f14ec0c91b0 0x7f14ec159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:31:14.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f12f9640 1 --2- 172.21.15.83:0/1954835473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14ec153600 0x7f14ec0c49d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0aedd0 tx=0x7f14e001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6c7d460f780d36 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:14.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14dffff640 1 -- 172.21.15.83:0/1954835473 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e00aa430 con 0x7f14ec153600 2024-09-19T07:31:14.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14dffff640 1 -- 172.21.15.83:0/1954835473 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e00af070 con 0x7f14ec153600 2024-09-19T07:31:14.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14dffff640 1 -- 172.21.15.83:0/1954835473 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e00aa7f0 con 0x7f14ec153600 2024-09-19T07:31:14.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 -- 172.21.15.83:0/1954835473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14ec153600 msgr2=0x7f14ec0c49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:14.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 --2- 172.21.15.83:0/1954835473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14ec153600 0x7f14ec0c49d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0aedd0 tx=0x7f14e001c4b0 comp rx=0 tx=0).stop 2024-09-19T07:31:14.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 -- 172.21.15.83:0/1954835473 shutdown_connections 2024-09-19T07:31:14.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 --2- 172.21.15.83:0/1954835473 >> v2:172.21.15.184:3300/0 conn(0x7f14ec0c91b0 0x7f14ec159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:14.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 --2- 172.21.15.83:0/1954835473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14ec153600 0x7f14ec0c49d0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:14.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 -- 172.21.15.83:0/1954835473 >> 172.21.15.83:0/1954835473 conn(0x7f14ec0c9610 msgr2=0x7f14ec0c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:14.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 -- 172.21.15.83:0/1954835473 shutdown_connections 2024-09-19T07:31:14.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 -- 172.21.15.83:0/1954835473 wait complete. 2024-09-19T07:31:14.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 Processor -- start 2024-09-19T07:31:14.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 -- start start 2024-09-19T07:31:14.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14ec0aee20 0x7f14ec14b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:14.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14ec153600 0x7f14ec148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14ec156f30 0x7f14ec149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f14ec0c32d0 con 0x7f14ec0aee20 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f14ec0c3150 con 0x7f14ec156f30 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f355d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f14ec159880 con 0x7f14ec153600 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f12f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14ec0aee20 0x7f14ec14b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f12f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14ec0aee20 0x7f14ec14b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45542/0 (socket says 172.21.15.83:45542) 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.916+0000 7f14f12f9640 1 -- 172.21.15.83:0/220732618 learned_addr learned my addr 172.21.15.83:0/220732618 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f1afa640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14ec156f30 0x7f14ec149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f0af8640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14ec153600 0x7f14ec148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f12f9640 1 -- 172.21.15.83:0/220732618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14ec153600 msgr2=0x7f14ec148d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f12f9640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14ec153600 0x7f14ec148d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f12f9640 1 -- 172.21.15.83:0/220732618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14ec156f30 msgr2=0x7f14ec149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:14.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f12f9640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14ec156f30 0x7f14ec149290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:14.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f12f9640 1 -- 172.21.15.83:0/220732618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ec0ab760 con 0x7f14ec0aee20 2024-09-19T07:31:14.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f0af8640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14ec153600 0x7f14ec148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:31:14.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f12f9640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14ec0aee20 0x7f14ec14b280 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f14e001ab60 tx=0x7f14e001acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:14.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14ddffb640 1 -- 172.21.15.83:0/220732618 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e00b3600 con 0x7f14ec0aee20 2024-09-19T07:31:14.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14ddffb640 1 -- 172.21.15.83:0/220732618 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e00b37c0 con 0x7f14ec0aee20 2024-09-19T07:31:14.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f355d640 1 -- 172.21.15.83:0/220732618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14ec0ab4c0 con 0x7f14ec0aee20 2024-09-19T07:31:14.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14ddffb640 1 -- 172.21.15.83:0/220732618 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e00b3bb0 con 0x7f14ec0aee20 2024-09-19T07:31:14.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f355d640 1 -- 172.21.15.83:0/220732618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14ec0ac2f0 con 0x7f14ec0aee20 2024-09-19T07:31:14.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f355d640 1 -- 172.21.15.83:0/220732618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14ec159340 con 0x7f14ec0aee20 2024-09-19T07:31:14.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14ddffb640 1 -- 172.21.15.83:0/220732618 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f14e00c6390 con 0x7f14ec0aee20 2024-09-19T07:31:14.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14ddffb640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f14d00678c0 0x7f14d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:14.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14ddffb640 1 -- 172.21.15.83:0/220732618 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7f14e0120a60 con 0x7f14ec0aee20 2024-09-19T07:31:14.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f0af8640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f14d00678c0 0x7f14d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:14.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.920+0000 7f14f0af8640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f14d00678c0 0x7f14d0069d80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f14ec150a70 tx=0x7f14e8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:14.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:14.924+0000 7f14ddffb640 1 -- 172.21.15.83:0/220732618 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f14e00c3060 con 0x7f14ec0aee20 2024-09-19T07:31:15.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.140+0000 7f14f355d640 1 -- 172.21.15.83:0/220732618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14ec159340 con 0x7f14ec0aee20 2024-09-19T07:31:15.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.140+0000 7f14ddffb640 1 -- 172.21.15.83:0/220732618 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+6540 (secure 0 0 0) 0x7f14e00160d0 con 0x7f14ec0aee20 2024-09-19T07:31:15.139 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:15.139 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":95,"btime":"2024-09-19T07:30:58:147417+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6447,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:31:15.139 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 95 2024-09-19T07:31:15.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.144+0000 7f14f355d640 1 -- 172.21.15.83:0/220732618 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f14d00678c0 msgr2=0x7f14d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:15.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.144+0000 7f14f355d640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f14d00678c0 0x7f14d0069d80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f14ec150a70 tx=0x7f14e8008040 comp rx=0 tx=0).stop 2024-09-19T07:31:15.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.144+0000 7f14f355d640 1 -- 172.21.15.83:0/220732618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14ec0aee20 msgr2=0x7f14ec14b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:15.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.144+0000 7f14f355d640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14ec0aee20 0x7f14ec14b280 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f14e001ab60 tx=0x7f14e001acf0 comp rx=0 tx=0).stop 2024-09-19T07:31:15.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.144+0000 7f14f355d640 1 -- 172.21.15.83:0/220732618 shutdown_connections 2024-09-19T07:31:15.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.144+0000 7f14f355d640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f14d00678c0 0x7f14d0069d80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:15.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.144+0000 7f14f355d640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14ec156f30 0x7f14ec149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:15.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.144+0000 7f14f355d640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14ec153600 0x7f14ec148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:15.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.144+0000 7f14f355d640 1 --2- 172.21.15.83:0/220732618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14ec0aee20 0x7f14ec14b280 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:15.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.144+0000 7f14f355d640 1 -- 172.21.15.83:0/220732618 >> 172.21.15.83:0/220732618 conn(0x7f14ec0c9610 msgr2=0x7f14ec150e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:15.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.144+0000 7f14f355d640 1 -- 172.21.15.83:0/220732618 shutdown_connections 2024-09-19T07:31:15.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:15.144+0000 7f14f355d640 1 -- 172.21.15.83:0/220732618 wait complete. 2024-09-19T07:31:15.161 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:31:17.162 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:17.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5db40a640 1 Processor -- start 2024-09-19T07:31:17.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5db40a640 1 -- start start 2024-09-19T07:31:17.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5db40a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb5d4153300 0x7fb5d406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:17.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5db40a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb5d4155bb0 0x7fb5d4156020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:17.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5db40a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5d41593f0 con 0x7fb5d406d1d0 2024-09-19T07:31:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5db40a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb5d41591a0 con 0x7fb5d4153300 2024-09-19T07:31:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5db40a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fb5d41596d0 con 0x7fb5d4155bb0 2024-09-19T07:31:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5d91a6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb5d4153300 0x7fb5d406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5d91a6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb5d4153300 0x7fb5d406cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45546/0 (socket says 172.21.15.83:45546) 2024-09-19T07:31:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5d91a6640 1 -- 172.21.15.83:0/3192915458 learned_addr learned my addr 172.21.15.83:0/3192915458 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5d99a7640 1 --2- 172.21.15.83:0/3192915458 >> v2:172.21.15.184:3300/0 conn(0x7fb5d4155bb0 0x7fb5d4156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5d91a6640 1 -- 172.21.15.83:0/3192915458 >> v2:172.21.15.184:3300/0 conn(0x7fb5d4155bb0 msgr2=0x7fb5d4156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5d91a6640 1 --2- 172.21.15.83:0/3192915458 >> v2:172.21.15.184:3300/0 conn(0x7fb5d4155bb0 0x7fb5d4156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5d91a6640 1 -- 172.21.15.83:0/3192915458 >> v1:172.21.15.184:6789/0 conn(0x7fb5d406d1d0 legacy=0x7fb5d40c23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.316+0000 7fb5d91a6640 1 -- 172.21.15.83:0/3192915458 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d41599b0 con 0x7fb5d4153300 2024-09-19T07:31:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5d91a6640 1 --2- 172.21.15.83:0/3192915458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5d4153300 0x7fb5d406cc90 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4071e50 tx=0x7fb5d001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804b404397a5f08 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5cbfff640 1 -- 172.21.15.83:0/3192915458 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00b0070 con 0x7fb5d4153300 2024-09-19T07:31:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5cbfff640 1 -- 172.21.15.83:0/3192915458 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5d00ab600 con 0x7fb5d4153300 2024-09-19T07:31:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5cbfff640 1 -- 172.21.15.83:0/3192915458 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00ab9d0 con 0x7fb5d4153300 2024-09-19T07:31:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 -- 172.21.15.83:0/3192915458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5d4153300 msgr2=0x7fb5d406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 --2- 172.21.15.83:0/3192915458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5d4153300 0x7fb5d406cc90 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4071e50 tx=0x7fb5d001c3c0 comp rx=0 tx=0).stop 2024-09-19T07:31:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 -- 172.21.15.83:0/3192915458 shutdown_connections 2024-09-19T07:31:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 --2- 172.21.15.83:0/3192915458 >> v2:172.21.15.184:3300/0 conn(0x7fb5d4155bb0 0x7fb5d4156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 --2- 172.21.15.83:0/3192915458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5d4153300 0x7fb5d406cc90 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 -- 172.21.15.83:0/3192915458 >> 172.21.15.83:0/3192915458 conn(0x7fb5d40c5090 msgr2=0x7fb5d40c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 -- 172.21.15.83:0/3192915458 shutdown_connections 2024-09-19T07:31:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 -- 172.21.15.83:0/3192915458 wait complete. 2024-09-19T07:31:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 Processor -- start 2024-09-19T07:31:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 -- start start 2024-09-19T07:31:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5d406d1d0 0x7fb5d41ef1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5d4153300 0x7fb5d4144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d4155bb0 0x7fb5d40ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d4153860 con 0x7fb5d406d1d0 2024-09-19T07:31:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d41536e0 con 0x7fb5d4155bb0 2024-09-19T07:31:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5db40a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb5d4162d50 con 0x7fb5d4153300 2024-09-19T07:31:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5d91a6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5d406d1d0 0x7fb5d41ef1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5d91a6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5d406d1d0 0x7fb5d41ef1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45562/0 (socket says 172.21.15.83:45562) 2024-09-19T07:31:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.320+0000 7fb5d91a6640 1 -- 172.21.15.83:0/775608387 learned_addr learned my addr 172.21.15.83:0/775608387 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5d99a7640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d4155bb0 0x7fb5d40ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5d89a5640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5d4153300 0x7fb5d4144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5d91a6640 1 -- 172.21.15.83:0/775608387 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5d4153300 msgr2=0x7fb5d4144940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5d91a6640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5d4153300 0x7fb5d4144940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5d91a6640 1 -- 172.21.15.83:0/775608387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d4155bb0 msgr2=0x7fb5d40ab430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5d91a6640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d4155bb0 0x7fb5d40ab430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5d91a6640 1 -- 172.21.15.83:0/775608387 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d40adf50 con 0x7fb5d406d1d0 2024-09-19T07:31:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5d99a7640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d4155bb0 0x7fb5d40ab430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:31:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5d89a5640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5d4153300 0x7fb5d4144940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:31:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5d91a6640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5d406d1d0 0x7fb5d41ef1d0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb5d00ae040 tx=0x7fb5d001ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5c9ffb640 1 -- 172.21.15.83:0/775608387 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00ba020 con 0x7fb5d406d1d0 2024-09-19T07:31:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5c9ffb640 1 -- 172.21.15.83:0/775608387 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5d00b6640 con 0x7fb5d406d1d0 2024-09-19T07:31:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5db40a640 1 -- 172.21.15.83:0/775608387 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d41ee6f0 con 0x7fb5d406d1d0 2024-09-19T07:31:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5c9ffb640 1 -- 172.21.15.83:0/775608387 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00b0040 con 0x7fb5d406d1d0 2024-09-19T07:31:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5db40a640 1 -- 172.21.15.83:0/775608387 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d41eecb0 con 0x7fb5d406d1d0 2024-09-19T07:31:17.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5db40a640 1 -- 172.21.15.83:0/775608387 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5d406cc90 con 0x7fb5d406d1d0 2024-09-19T07:31:17.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5c9ffb640 1 -- 172.21.15.83:0/775608387 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb5d00bd020 con 0x7fb5d406d1d0 2024-09-19T07:31:17.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5c9ffb640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5b80678c0 0x7fb5b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:17.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5c9ffb640 1 -- 172.21.15.83:0/775608387 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8099+0+0 (secure 0 0 0) 0x7fb5d0120840 con 0x7fb5d406d1d0 2024-09-19T07:31:17.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5d89a5640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5b80678c0 0x7fb5b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:17.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.324+0000 7fb5d89a5640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5b80678c0 0x7fb5b8069d80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4153050 tx=0x7fb5c4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:17.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.332+0000 7fb5c9ffb640 1 -- 172.21.15.83:0/775608387 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb5d00c4030 con 0x7fb5d406d1d0 2024-09-19T07:31:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.584+0000 7fb5db40a640 1 -- 172.21.15.83:0/775608387 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5d406f980 con 0x7fb5d406d1d0 2024-09-19T07:31:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5c9ffb640 1 -- 172.21.15.83:0/775608387 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+6540 (secure 0 0 0) 0x7fb5d00160d0 con 0x7fb5d406d1d0 2024-09-19T07:31:17.584 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:17.584 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":95,"btime":"2024-09-19T07:30:58:147417+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6447,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:30:51.094245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6267":{"gid":6267,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6837/2581473241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2581473241},{"type":"v1","addr":"172.21.15.83:6837","nonce":2581473241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6267,"qdb_cluster":[6267]},"id":2}]} 2024-09-19T07:31:17.584 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 95 2024-09-19T07:31:17.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5db40a640 1 -- 172.21.15.83:0/775608387 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5b80678c0 msgr2=0x7fb5b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:17.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5db40a640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5b80678c0 0x7fb5b8069d80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4153050 tx=0x7fb5c4008040 comp rx=0 tx=0).stop 2024-09-19T07:31:17.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5db40a640 1 -- 172.21.15.83:0/775608387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5d406d1d0 msgr2=0x7fb5d41ef1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:17.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5db40a640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5d406d1d0 0x7fb5d41ef1d0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb5d00ae040 tx=0x7fb5d001ac00 comp rx=0 tx=0).stop 2024-09-19T07:31:17.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5db40a640 1 -- 172.21.15.83:0/775608387 shutdown_connections 2024-09-19T07:31:17.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5db40a640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5b80678c0 0x7fb5b8069d80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:17.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5db40a640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5d4155bb0 0x7fb5d40ab430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:17.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5db40a640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5d4153300 0x7fb5d4144940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:17.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5db40a640 1 --2- 172.21.15.83:0/775608387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5d406d1d0 0x7fb5d41ef1d0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:17.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5db40a640 1 -- 172.21.15.83:0/775608387 >> 172.21.15.83:0/775608387 conn(0x7fb5d40c5090 msgr2=0x7fb5d40c2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:17.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5db40a640 1 -- 172.21.15.83:0/775608387 shutdown_connections 2024-09-19T07:31:17.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:17.588+0000 7fb5db40a640 1 -- 172.21.15.83:0/775608387 wait complete. 2024-09-19T07:31:17.603 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:31:19.604 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:19.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.748+0000 7ff630442640 1 Processor -- start 2024-09-19T07:31:19.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.748+0000 7ff630442640 1 -- start start 2024-09-19T07:31:19.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.748+0000 7ff630442640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff6280c3130 0x7ff6280ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:19.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.748+0000 7ff630442640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62806de70 0x7ff6281563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:19.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.748+0000 7ff630442640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6280c3c20 con 0x7ff62806da00 2024-09-19T07:31:19.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.748+0000 7ff630442640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff6280c4990 con 0x7ff6280c3130 2024-09-19T07:31:19.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.748+0000 7ff630442640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff6280c4c70 con 0x7ff62806de70 2024-09-19T07:31:19.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.752+0000 7ff62e1de640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff6280c3130 0x7ff6280ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:19.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.752+0000 7ff62d9dd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff62806da00 0x7ff6280c3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45598/0 (socket says 172.21.15.83:45598) 2024-09-19T07:31:19.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.752+0000 7ff62d9dd640 1 -- 172.21.15.83:0/1370536878 learned_addr learned my addr 172.21.15.83:0/1370536878 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:31:19.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.752+0000 7ff62e1de640 1 -- 172.21.15.83:0/1370536878 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62806de70 msgr2=0x7ff6281563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:19.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.752+0000 7ff62e1de640 1 --2- 172.21.15.83:0/1370536878 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62806de70 0x7ff6281563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:19.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.752+0000 7ff62e1de640 1 -- 172.21.15.83:0/1370536878 >> v1:172.21.15.83:6789/0 conn(0x7ff62806da00 legacy=0x7ff6280c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:19.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.752+0000 7ff62d1dc640 1 -- 172.21.15.83:0/1370536878 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 687148784 0 0) 0x7ff6280c3c20 con 0x7ff62806da00 2024-09-19T07:31:19.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.752+0000 7ff62e1de640 1 -- 172.21.15.83:0/1370536878 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6280ab7b0 con 0x7ff6280c3130 2024-09-19T07:31:19.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.752+0000 7ff62e1de640 1 --2- 172.21.15.83:0/1370536878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6280c3130 0x7ff6280ac290 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ff62400ba60 tx=0x7ff62401e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c60d3fd407a6bba server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:19.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.752+0000 7ff62d1dc640 1 -- 172.21.15.83:0/1370536878 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6240b4070 con 0x7ff6280c3130 2024-09-19T07:31:19.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.752+0000 7ff62d1dc640 1 -- 172.21.15.83:0/1370536878 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6240a77e0 con 0x7ff6280c3130 2024-09-19T07:31:19.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.752+0000 7ff62d1dc640 1 -- 172.21.15.83:0/1370536878 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6240a7bb0 con 0x7ff6280c3130 2024-09-19T07:31:19.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 -- 172.21.15.83:0/1370536878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6280c3130 msgr2=0x7ff6280ac290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:19.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 --2- 172.21.15.83:0/1370536878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6280c3130 0x7ff6280ac290 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ff62400ba60 tx=0x7ff62401e820 comp rx=0 tx=0).stop 2024-09-19T07:31:19.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 -- 172.21.15.83:0/1370536878 shutdown_connections 2024-09-19T07:31:19.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 --2- 172.21.15.83:0/1370536878 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62806de70 0x7ff6281563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:19.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 --2- 172.21.15.83:0/1370536878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6280c3130 0x7ff6280ac290 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:19.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 -- 172.21.15.83:0/1370536878 >> 172.21.15.83:0/1370536878 conn(0x7ff6280c5060 msgr2=0x7ff6280c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:19.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 -- 172.21.15.83:0/1370536878 shutdown_connections 2024-09-19T07:31:19.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 -- 172.21.15.83:0/1370536878 wait complete. 2024-09-19T07:31:19.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 Processor -- start 2024-09-19T07:31:19.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 -- start start 2024-09-19T07:31:19.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62806da00 0x7ff6281ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:19.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff62806de70 0x7ff6281f23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:19.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6280c3130 0x7ff62814f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:19.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff6280ad6a0 con 0x7ff62806da00 2024-09-19T07:31:19.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff6280ad520 con 0x7ff62806de70 2024-09-19T07:31:19.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff628074f90 con 0x7ff6280c3130 2024-09-19T07:31:19.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff62e1de640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62806da00 0x7ff6281ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:19.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff62e1de640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62806da00 0x7ff6281ee6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45584/0 (socket says 172.21.15.83:45584) 2024-09-19T07:31:19.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff62e1de640 1 -- 172.21.15.83:0/1129060210 learned_addr learned my addr 172.21.15.83:0/1129060210 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:19.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff62e9df640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6280c3130 0x7ff62814f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:19.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff62e1de640 1 -- 172.21.15.83:0/1129060210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6280c3130 msgr2=0x7ff62814f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:19.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff62e1de640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6280c3130 0x7ff62814f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:19.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff62e1de640 1 -- 172.21.15.83:0/1129060210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff62806de70 msgr2=0x7ff6281f23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:19.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff62e1de640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff62806de70 0x7ff6281f23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:19.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff62e1de640 1 -- 172.21.15.83:0/1129060210 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff628155cb0 con 0x7ff62806da00 2024-09-19T07:31:19.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff62e9df640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6280c3130 0x7ff62814f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:31:19.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff62e1de640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62806da00 0x7ff6281ee6c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7ff62400ba60 tx=0x7ff62401d060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:19.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff61effd640 1 -- 172.21.15.83:0/1129060210 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6240b4040 con 0x7ff62806da00 2024-09-19T07:31:19.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff61effd640 1 -- 172.21.15.83:0/1129060210 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff624004220 con 0x7ff62806da00 2024-09-19T07:31:19.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 -- 172.21.15.83:0/1129060210 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff628155f90 con 0x7ff62806da00 2024-09-19T07:31:19.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff61effd640 1 -- 172.21.15.83:0/1129060210 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624003620 con 0x7ff62806da00 2024-09-19T07:31:19.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.756+0000 7ff630442640 1 -- 172.21.15.83:0/1129060210 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6281500f0 con 0x7ff62806da00 2024-09-19T07:31:19.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.760+0000 7ff630442640 1 -- 172.21.15.83:0/1129060210 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6281f2050 con 0x7ff62806da00 2024-09-19T07:31:19.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.760+0000 7ff61effd640 1 -- 172.21.15.83:0/1129060210 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff6240c1060 con 0x7ff62806da00 2024-09-19T07:31:19.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.760+0000 7ff61effd640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff60c0678c0 0x7ff60c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:19.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.760+0000 7ff61effd640 1 -- 172.21.15.83:0/1129060210 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7ff624120580 con 0x7ff62806da00 2024-09-19T07:31:19.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.760+0000 7ff62d9dd640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff60c0678c0 0x7ff60c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:19.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.760+0000 7ff62d9dd640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff60c0678c0 0x7ff60c069d80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff618007fd0 tx=0x7ff618007960 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:19.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.764+0000 7ff61effd640 1 -- 172.21.15.83:0/1129060210 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff624018760 con 0x7ff62806da00 2024-09-19T07:31:19.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 -- 172.21.15.83:0/1129060210 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6280c5720 con 0x7ff62806da00 2024-09-19T07:31:19.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff61effd640 1 -- 172.21.15.83:0/1129060210 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5710 (secure 0 0 0) 0x7ff6240f0750 con 0x7ff62806da00 2024-09-19T07:31:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:19.980 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":97,"btime":"2024-09-19T07:31:19:217731+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:18.441353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:31:19.981 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 97 2024-09-19T07:31:19.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 -- 172.21.15.83:0/1129060210 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff60c0678c0 msgr2=0x7ff60c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:19.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff60c0678c0 0x7ff60c069d80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff618007fd0 tx=0x7ff618007960 comp rx=0 tx=0).stop 2024-09-19T07:31:19.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 -- 172.21.15.83:0/1129060210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62806da00 msgr2=0x7ff6281ee6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:19.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62806da00 0x7ff6281ee6c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7ff62400ba60 tx=0x7ff62401d060 comp rx=0 tx=0).stop 2024-09-19T07:31:19.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 -- 172.21.15.83:0/1129060210 shutdown_connections 2024-09-19T07:31:19.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff60c0678c0 0x7ff60c069d80 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:19.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6280c3130 0x7ff62814f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:19.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff62806de70 0x7ff6281f23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:19.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 --2- 172.21.15.83:0/1129060210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62806da00 0x7ff6281ee6c0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:19.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 -- 172.21.15.83:0/1129060210 >> 172.21.15.83:0/1129060210 conn(0x7ff6280c5060 msgr2=0x7ff6280ac290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:19.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 -- 172.21.15.83:0/1129060210 shutdown_connections 2024-09-19T07:31:19.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:19.984+0000 7ff630442640 1 -- 172.21.15.83:0/1129060210 wait complete. 2024-09-19T07:31:20.000 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-19T07:31:20.000 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:31:20.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:20.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa912cb3640 1 Processor -- start 2024-09-19T07:31:20.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa912cb3640 1 -- start start 2024-09-19T07:31:20.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa912cb3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa90c153480 0x7fa90c155e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:20.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa912cb3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c153860 0x7fa90c156380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:20.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa912cb3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90c159dc0 con 0x7fa90c06be10 2024-09-19T07:31:20.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa912cb3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa90c070040 con 0x7fa90c153480 2024-09-19T07:31:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa912cb3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa90c0c1d80 con 0x7fa90c153860 2024-09-19T07:31:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa910a4f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa90c153480 0x7fa90c155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa910a4f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa90c153480 0x7fa90c155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45586/0 (socket says 172.21.15.83:45586) 2024-09-19T07:31:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa911250640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa90c06be10 0x7fa90c06c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45612/0 (socket says 172.21.15.83:45612) 2024-09-19T07:31:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa910a4f640 1 -- 172.21.15.83:0/3303679672 learned_addr learned my addr 172.21.15.83:0/3303679672 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa90bfff640 1 --2- 172.21.15.83:0/3303679672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c153860 0x7fa90c156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa910a4f640 1 -- 172.21.15.83:0/3303679672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c153860 msgr2=0x7fa90c156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa910a4f640 1 --2- 172.21.15.83:0/3303679672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c153860 0x7fa90c156380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa910a4f640 1 -- 172.21.15.83:0/3303679672 >> v1:172.21.15.83:6789/0 conn(0x7fa90c06be10 legacy=0x7fa90c06c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.112+0000 7fa910a4f640 1 -- 172.21.15.83:0/3303679672 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c0c2bf0 con 0x7fa90c153480 2024-09-19T07:31:20.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa910a4f640 1 --2- 172.21.15.83:0/3303679672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa90c153480 0x7fa90c155e40 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fa900009870 tx=0x7fa90001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5dbb98dc17ab8e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:20.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa90b7fe640 1 -- 172.21.15.83:0/3303679672 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9000b2070 con 0x7fa90c153480 2024-09-19T07:31:20.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa90b7fe640 1 -- 172.21.15.83:0/3303679672 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9000a9070 con 0x7fa90c153480 2024-09-19T07:31:20.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa90b7fe640 1 -- 172.21.15.83:0/3303679672 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9000ad880 con 0x7fa90c153480 2024-09-19T07:31:20.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 -- 172.21.15.83:0/3303679672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa90c153480 msgr2=0x7fa90c155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:20.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 --2- 172.21.15.83:0/3303679672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa90c153480 0x7fa90c155e40 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fa900009870 tx=0x7fa90001c560 comp rx=0 tx=0).stop 2024-09-19T07:31:20.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 -- 172.21.15.83:0/3303679672 shutdown_connections 2024-09-19T07:31:20.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 --2- 172.21.15.83:0/3303679672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c153860 0x7fa90c156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:20.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 --2- 172.21.15.83:0/3303679672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa90c153480 0x7fa90c155e40 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:20.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 -- 172.21.15.83:0/3303679672 >> 172.21.15.83:0/3303679672 conn(0x7fa90c0c8840 msgr2=0x7fa90c0c8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:20.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 -- 172.21.15.83:0/3303679672 shutdown_connections 2024-09-19T07:31:20.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 -- 172.21.15.83:0/3303679672 wait complete. 2024-09-19T07:31:20.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 Processor -- start 2024-09-19T07:31:20.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 -- start start 2024-09-19T07:31:20.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa90c06be10 0x7fa90c15dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:20.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa90c153480 0x7fa90c0acb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:20.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c153860 0x7fa90c1f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:20.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c06ce80 con 0x7fa90c153480 2024-09-19T07:31:20.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c0c29a0 con 0x7fa90c06be10 2024-09-19T07:31:20.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa912cb3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa90c06d000 con 0x7fa90c153860 2024-09-19T07:31:20.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa90bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa90c153480 0x7fa90c0acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:20.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa90bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa90c153480 0x7fa90c0acb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48132/0 (socket says 172.21.15.83:48132) 2024-09-19T07:31:20.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa90bfff640 1 -- 172.21.15.83:0/3605803613 learned_addr learned my addr 172.21.15.83:0/3605803613 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:20.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa911250640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c153860 0x7fa90c1f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:20.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa910a4f640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa90c06be10 0x7fa90c15dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:20.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa90bfff640 1 -- 172.21.15.83:0/3605803613 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c153860 msgr2=0x7fa90c1f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:20.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa90bfff640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c153860 0x7fa90c1f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:20.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa90bfff640 1 -- 172.21.15.83:0/3605803613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa90c06be10 msgr2=0x7fa90c15dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:20.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa90bfff640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa90c06be10 0x7fa90c15dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:20.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa90bfff640 1 -- 172.21.15.83:0/3605803613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c06def0 con 0x7fa90c153480 2024-09-19T07:31:20.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa911250640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c153860 0x7fa90c1f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:31:20.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.116+0000 7fa90bfff640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa90c153480 0x7fa90c0acb90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800c7f0 tx=0x7fa8f800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:20.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.120+0000 7fa9097fa640 1 -- 172.21.15.83:0/3605803613 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8004230 con 0x7fa90c153480 2024-09-19T07:31:20.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.120+0000 7fa9097fa640 1 -- 172.21.15.83:0/3605803613 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8f80043d0 con 0x7fa90c153480 2024-09-19T07:31:20.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.120+0000 7fa9097fa640 1 -- 172.21.15.83:0/3605803613 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8003860 con 0x7fa90c153480 2024-09-19T07:31:20.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.120+0000 7fa912cb3640 1 -- 172.21.15.83:0/3605803613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa90c15e230 con 0x7fa90c153480 2024-09-19T07:31:20.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.120+0000 7fa912cb3640 1 -- 172.21.15.83:0/3605803613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa90c1f2220 con 0x7fa90c153480 2024-09-19T07:31:20.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.120+0000 7fa912cb3640 1 -- 172.21.15.83:0/3605803613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa90c06c360 con 0x7fa90c153480 2024-09-19T07:31:20.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.120+0000 7fa9097fa640 1 -- 172.21.15.83:0/3605803613 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa8f8003c10 con 0x7fa90c153480 2024-09-19T07:31:20.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.120+0000 7fa9097fa640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa8f00678c0 0x7fa8f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:20.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.120+0000 7fa9097fa640 1 -- 172.21.15.83:0/3605803613 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7fa8f80868b0 con 0x7fa90c153480 2024-09-19T07:31:20.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.120+0000 7fa910a4f640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa8f00678c0 0x7fa8f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:20.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.120+0000 7fa910a4f640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa8f00678c0 0x7fa8f0069d80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa9000af000 tx=0x7fa900008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:20.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.124+0000 7fa9097fa640 1 -- 172.21.15.83:0/3605803613 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa8f8015020 con 0x7fa90c153480 2024-09-19T07:31:20.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.348+0000 7fa912cb3640 1 -- 172.21.15.83:0/3605803613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa90c0c8da0 con 0x7fa90c153480 2024-09-19T07:31:20.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa9097fa640 1 -- 172.21.15.83:0/3605803613 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) ==== 94+0+5707 (secure 0 0 0) 0x7fa8f80569f0 con 0x7fa90c153480 2024-09-19T07:31:20.348 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:20.348 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":98,"btime":"2024-09-19T07:31:20:225874+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:19.229616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:31:20.348 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 98 2024-09-19T07:31:20.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa912cb3640 1 -- 172.21.15.83:0/3605803613 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa8f00678c0 msgr2=0x7fa8f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:20.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa912cb3640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa8f00678c0 0x7fa8f0069d80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa9000af000 tx=0x7fa900008040 comp rx=0 tx=0).stop 2024-09-19T07:31:20.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa912cb3640 1 -- 172.21.15.83:0/3605803613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa90c153480 msgr2=0x7fa90c0acb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:20.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa912cb3640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa90c153480 0x7fa90c0acb90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800c7f0 tx=0x7fa8f800ccb0 comp rx=0 tx=0).stop 2024-09-19T07:31:20.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa912cb3640 1 -- 172.21.15.83:0/3605803613 shutdown_connections 2024-09-19T07:31:20.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa912cb3640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa8f00678c0 0x7fa8f0069d80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:20.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa912cb3640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c153860 0x7fa90c1f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:20.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa912cb3640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa90c153480 0x7fa90c0acb90 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:20.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa912cb3640 1 --2- 172.21.15.83:0/3605803613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa90c06be10 0x7fa90c15dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:20.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa912cb3640 1 -- 172.21.15.83:0/3605803613 >> 172.21.15.83:0/3605803613 conn(0x7fa90c0c8840 msgr2=0x7fa90c0aacf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:20.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa912cb3640 1 -- 172.21.15.83:0/3605803613 shutdown_connections 2024-09-19T07:31:20.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:20.352+0000 7fa912cb3640 1 -- 172.21.15.83:0/3605803613 wait complete. 2024-09-19T07:31:20.367 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:31:20:225874+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 98, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4652": { "addr": "172.21.15.184:6833/2993984999", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2993984999, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2993984999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4652, "incarnation": 90, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 129 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:30:51.094244+0000", "qdb_cluster": [ 4652 ], "qdb_leader": 4652, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4652 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 98, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6447": { "addr": "172.21.15.83:6835/1072903448", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 1072903448, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 1072903448, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6447, "incarnation": 96, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:31:19.229616+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6447 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/2112478738", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 2112478738, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 2112478738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 95, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5722, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:31:22.368 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:22.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.504+0000 7f6264cd9640 1 Processor -- start 2024-09-19T07:31:22.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.504+0000 7f6264cd9640 1 -- start start 2024-09-19T07:31:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f6264cd9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f62600c7d70 0x7f6260157c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f6264cd9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6260072d00 0x7f626006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f6264cd9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62600c74f0 con 0x7f62601547b0 2024-09-19T07:31:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f6264cd9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6260073690 con 0x7f62600c7d70 2024-09-19T07:31:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f6264cd9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6260157160 con 0x7f6260072d00 2024-09-19T07:31:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f625e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f62600c7d70 0x7f6260157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f625e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f62600c7d70 0x7f6260157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48134/0 (socket says 172.21.15.83:48134) 2024-09-19T07:31:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f625e59c640 1 -- 172.21.15.83:0/1190598565 learned_addr learned my addr 172.21.15.83:0/1190598565 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:22.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f625ed9d640 1 --2- 172.21.15.83:0/1190598565 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6260072d00 0x7f626006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:22.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f625e59c640 1 -- 172.21.15.83:0/1190598565 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6260072d00 msgr2=0x7f626006cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:22.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f625e59c640 1 --2- 172.21.15.83:0/1190598565 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6260072d00 0x7f626006cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:22.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f625e59c640 1 -- 172.21.15.83:0/1190598565 >> v1:172.21.15.184:6789/0 conn(0x7f62601547b0 legacy=0x7f62600c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:22.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f625e59c640 1 -- 172.21.15.83:0/1190598565 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6260157440 con 0x7f62600c7d70 2024-09-19T07:31:22.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f625e59c640 1 --2- 172.21.15.83:0/1190598565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62600c7d70 0x7f6260157c40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6250009870 tx=0x7f625001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a363f1d75fa7a1ef server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:22.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f625d59a640 1 -- 172.21.15.83:0/1190598565 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62500b2070 con 0x7f62600c7d70 2024-09-19T07:31:22.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f625d59a640 1 -- 172.21.15.83:0/1190598565 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62500ad680 con 0x7f62600c7d70 2024-09-19T07:31:22.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f625d59a640 1 -- 172.21.15.83:0/1190598565 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62500ada50 con 0x7f62600c7d70 2024-09-19T07:31:22.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f6264cd9640 1 -- 172.21.15.83:0/1190598565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62600c7d70 msgr2=0x7f6260157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:22.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.508+0000 7f6264cd9640 1 --2- 172.21.15.83:0/1190598565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62600c7d70 0x7f6260157c40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6250009870 tx=0x7f625001c460 comp rx=0 tx=0).stop 2024-09-19T07:31:22.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 -- 172.21.15.83:0/1190598565 shutdown_connections 2024-09-19T07:31:22.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 --2- 172.21.15.83:0/1190598565 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6260072d00 0x7f626006cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:22.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 --2- 172.21.15.83:0/1190598565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62600c7d70 0x7f6260157c40 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:22.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 -- 172.21.15.83:0/1190598565 >> 172.21.15.83:0/1190598565 conn(0x7f62600c5060 msgr2=0x7f62600c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:22.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 -- 172.21.15.83:0/1190598565 shutdown_connections 2024-09-19T07:31:22.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 -- 172.21.15.83:0/1190598565 wait complete. 2024-09-19T07:31:22.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 Processor -- start 2024-09-19T07:31:22.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 -- start start 2024-09-19T07:31:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6260072d00 0x7f626015a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62600c7d70 0x7f6260155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62601547b0 0x7f62601583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f62600703b0 con 0x7f62600c7d70 2024-09-19T07:31:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6260070230 con 0x7f6260072d00 2024-09-19T07:31:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6260070530 con 0x7f62601547b0 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f625dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62600c7d70 0x7f6260155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f625dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62600c7d70 0x7f6260155400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48146/0 (socket says 172.21.15.83:48146) 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f625dd9b640 1 -- 172.21.15.83:0/3770371390 learned_addr learned my addr 172.21.15.83:0/3770371390 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f625e59c640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6260072d00 0x7f626015a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f625ed9d640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62601547b0 0x7f62601583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f625dd9b640 1 -- 172.21.15.83:0/3770371390 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62601547b0 msgr2=0x7f62601583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f625dd9b640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62601547b0 0x7f62601583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f625dd9b640 1 -- 172.21.15.83:0/3770371390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6260072d00 msgr2=0x7f626015a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f625dd9b640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6260072d00 0x7f626015a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f625dd9b640 1 -- 172.21.15.83:0/3770371390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62601581a0 con 0x7f62600c7d70 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f625dd9b640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62600c7d70 0x7f6260155400 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f625400c450 tx=0x7f625400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f624effd640 1 -- 172.21.15.83:0/3770371390 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6254011790 con 0x7f62600c7d70 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f624effd640 1 -- 172.21.15.83:0/3770371390 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6254017070 con 0x7f62600c7d70 2024-09-19T07:31:22.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 -- 172.21.15.83:0/3770371390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62601f2090 con 0x7f62600c7d70 2024-09-19T07:31:22.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f624effd640 1 -- 172.21.15.83:0/3770371390 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6254011b90 con 0x7f62600c7d70 2024-09-19T07:31:22.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.512+0000 7f6264cd9640 1 -- 172.21.15.83:0/3770371390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62601eef70 con 0x7f62600c7d70 2024-09-19T07:31:22.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.516+0000 7f6264cd9640 1 -- 172.21.15.83:0/3770371390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f626006cc90 con 0x7f62600c7d70 2024-09-19T07:31:22.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.516+0000 7f624effd640 1 -- 172.21.15.83:0/3770371390 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6254011d30 con 0x7f62600c7d70 2024-09-19T07:31:22.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.516+0000 7f624effd640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62400678c0 0x7f6240069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:22.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.516+0000 7f624effd640 1 -- 172.21.15.83:0/3770371390 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7f62540863f0 con 0x7f62600c7d70 2024-09-19T07:31:22.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.516+0000 7f625e59c640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62400678c0 0x7f6240069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:22.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.516+0000 7f625e59c640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62400678c0 0x7f6240069d80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f62500af000 tx=0x7f62500a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:22.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.520+0000 7f624effd640 1 -- 172.21.15.83:0/3770371390 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f62540565c0 con 0x7f62600c7d70 2024-09-19T07:31:22.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.728+0000 7f6264cd9640 1 -- 172.21.15.83:0/3770371390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62600c2c40 con 0x7f62600c7d70 2024-09-19T07:31:22.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.728+0000 7f624effd640 1 -- 172.21.15.83:0/3770371390 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5714 (secure 0 0 0) 0x7f6254055ec0 con 0x7f62600c7d70 2024-09-19T07:31:22.726 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:22.726 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":99,"btime":"2024-09-19T07:31:21:234495+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:31:22.726 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 99 2024-09-19T07:31:22.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.732+0000 7f6264cd9640 1 -- 172.21.15.83:0/3770371390 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62400678c0 msgr2=0x7f6240069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:22.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.732+0000 7f6264cd9640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62400678c0 0x7f6240069d80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f62500af000 tx=0x7f62500a7000 comp rx=0 tx=0).stop 2024-09-19T07:31:22.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.732+0000 7f6264cd9640 1 -- 172.21.15.83:0/3770371390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62600c7d70 msgr2=0x7f6260155400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:22.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.732+0000 7f6264cd9640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62600c7d70 0x7f6260155400 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f625400c450 tx=0x7f625400c910 comp rx=0 tx=0).stop 2024-09-19T07:31:22.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.732+0000 7f6264cd9640 1 -- 172.21.15.83:0/3770371390 shutdown_connections 2024-09-19T07:31:22.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.732+0000 7f6264cd9640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f62400678c0 0x7f6240069d80 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:22.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.732+0000 7f6264cd9640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62601547b0 0x7f62601583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:22.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.732+0000 7f6264cd9640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62600c7d70 0x7f6260155400 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:22.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.732+0000 7f6264cd9640 1 --2- 172.21.15.83:0/3770371390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6260072d00 0x7f626015a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:22.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.732+0000 7f6264cd9640 1 -- 172.21.15.83:0/3770371390 >> 172.21.15.83:0/3770371390 conn(0x7f62600c5060 msgr2=0x7f6260156840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:22.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.732+0000 7f6264cd9640 1 -- 172.21.15.83:0/3770371390 shutdown_connections 2024-09-19T07:31:22.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:22.732+0000 7f6264cd9640 1 -- 172.21.15.83:0/3770371390 wait complete. 2024-09-19T07:31:22.745 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=6267 2024-09-19T07:31:22.745 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.c 2024-09-19T07:31:24.746 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-19T07:31:24.746 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:31:24.747 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:31:24.749 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:31:24.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:24.827 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:31:24.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f6a06929640 1 Processor -- start 2024-09-19T07:31:24.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f6a06929640 1 -- start start 2024-09-19T07:31:24.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f6a06929640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a000c23d0 0x7f6a000c27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:24.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f6a06929640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6a00155bb0 0x7f6a00156040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:24.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f6a06929640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a001593f0 con 0x7f6a00153300 2024-09-19T07:31:24.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f6a06929640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6a001591a0 con 0x7f6a00155bb0 2024-09-19T07:31:24.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f6a06929640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6a001596d0 con 0x7f6a000c23d0 2024-09-19T07:31:24.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f69fffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6a00153300 0x7f6a0006cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41804/0 (socket says 172.21.15.83:41804) 2024-09-19T07:31:24.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f69fffff640 1 -- 172.21.15.83:0/3749120382 learned_addr learned my addr 172.21.15.83:0/3749120382 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:31:24.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f69ff7fe640 1 --2- 172.21.15.83:0/3749120382 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a000c23d0 0x7f6a000c27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:24.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f6a04ec6640 1 --2- 172.21.15.83:0/3749120382 >> v2:172.21.15.184:3300/0 conn(0x7f6a00155bb0 0x7f6a00156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:24.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f69feffd640 1 -- 172.21.15.83:0/3749120382 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 609580566 0 0) 0x7f6a001593f0 con 0x7f6a00153300 2024-09-19T07:31:24.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f69feffd640 1 -- 172.21.15.83:0/3749120382 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69e8003530 con 0x7f6a00153300 2024-09-19T07:31:24.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f69ff7fe640 1 -- 172.21.15.83:0/3749120382 >> v2:172.21.15.184:3300/0 conn(0x7f6a00155bb0 msgr2=0x7f6a00156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:24.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f69ff7fe640 1 --2- 172.21.15.83:0/3749120382 >> v2:172.21.15.184:3300/0 conn(0x7f6a00155bb0 0x7f6a00156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:24.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f69ff7fe640 1 -- 172.21.15.83:0/3749120382 >> v1:172.21.15.83:6789/0 conn(0x7f6a00153300 legacy=0x7f6a0006cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:24.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.888+0000 7f69ff7fe640 1 -- 172.21.15.83:0/3749120382 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a001599b0 con 0x7f6a000c23d0 2024-09-19T07:31:24.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f69ff7fe640 1 --2- 172.21.15.83:0/3749120382 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a000c23d0 0x7f6a000c27f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f69f40097c0 tx=0x7f69f401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d676d13bcb6d9bd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:24.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f69feffd640 1 -- 172.21.15.83:0/3749120382 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f40b0070 con 0x7f6a000c23d0 2024-09-19T07:31:24.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f69feffd640 1 -- 172.21.15.83:0/3749120382 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69f40ab600 con 0x7f6a000c23d0 2024-09-19T07:31:24.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f69feffd640 1 -- 172.21.15.83:0/3749120382 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f40ab9b0 con 0x7f6a000c23d0 2024-09-19T07:31:24.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f6a06929640 1 -- 172.21.15.83:0/3749120382 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a000c23d0 msgr2=0x7f6a000c27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:24.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f6a06929640 1 --2- 172.21.15.83:0/3749120382 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a000c23d0 0x7f6a000c27f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f69f40097c0 tx=0x7f69f401c380 comp rx=0 tx=0).stop 2024-09-19T07:31:24.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f6a06929640 1 -- 172.21.15.83:0/3749120382 shutdown_connections 2024-09-19T07:31:24.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f6a06929640 1 --2- 172.21.15.83:0/3749120382 >> v2:172.21.15.184:3300/0 conn(0x7f6a00155bb0 0x7f6a00156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:24.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f6a06929640 1 --2- 172.21.15.83:0/3749120382 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a000c23d0 0x7f6a000c27f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:24.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f6a06929640 1 -- 172.21.15.83:0/3749120382 >> 172.21.15.83:0/3749120382 conn(0x7f6a000c5090 msgr2=0x7f6a000c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:24.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f6a06929640 1 -- 172.21.15.83:0/3749120382 shutdown_connections 2024-09-19T07:31:24.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f6a06929640 1 -- 172.21.15.83:0/3749120382 wait complete. 2024-09-19T07:31:24.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f6a06929640 1 Processor -- start 2024-09-19T07:31:24.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.892+0000 7f6a06929640 1 -- start start 2024-09-19T07:31:24.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f6a06929640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a000c23d0 0x7f6a001eef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:24.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f6a06929640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a00153300 0x7f6a00150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:24.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f6a06929640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a00155bb0 0x7f6a0014fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:24.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f6a06929640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6a00153860 con 0x7f6a000c23d0 2024-09-19T07:31:24.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f6a06929640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6a001536e0 con 0x7f6a00155bb0 2024-09-19T07:31:24.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f6a06929640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6a00162d50 con 0x7f6a00153300 2024-09-19T07:31:24.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a000c23d0 0x7f6a001eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:24.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a000c23d0 0x7f6a001eef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48162/0 (socket says 172.21.15.83:48162) 2024-09-19T07:31:24.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fffff640 1 -- 172.21.15.83:0/3829729890 learned_addr learned my addr 172.21.15.83:0/3829729890 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:24.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f6a04ec6640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a00155bb0 0x7f6a0014fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:24.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69ff7fe640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a00153300 0x7f6a00150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:24.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fffff640 1 -- 172.21.15.83:0/3829729890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a00153300 msgr2=0x7f6a00150b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:24.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fffff640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a00153300 0x7f6a00150b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:24.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fffff640 1 -- 172.21.15.83:0/3829729890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a00155bb0 msgr2=0x7f6a0014fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:24.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fffff640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a00155bb0 0x7f6a0014fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:24.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fffff640 1 -- 172.21.15.83:0/3829729890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a001584b0 con 0x7f6a000c23d0 2024-09-19T07:31:24.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fffff640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a000c23d0 0x7f6a001eef80 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f69f000c480 tx=0x7f69f000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:24.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fcff9640 1 -- 172.21.15.83:0/3829729890 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f0017070 con 0x7f6a000c23d0 2024-09-19T07:31:24.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fcff9640 1 -- 172.21.15.83:0/3829729890 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69f0005cf0 con 0x7f6a000c23d0 2024-09-19T07:31:24.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f6a06929640 1 -- 172.21.15.83:0/3829729890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a00158790 con 0x7f6a000c23d0 2024-09-19T07:31:24.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fcff9640 1 -- 172.21.15.83:0/3829729890 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f0007290 con 0x7f6a000c23d0 2024-09-19T07:31:24.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f6a06929640 1 -- 172.21.15.83:0/3829729890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a00157420 con 0x7f6a000c23d0 2024-09-19T07:31:24.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f6a06929640 1 -- 172.21.15.83:0/3829729890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a000c27f0 con 0x7f6a000c23d0 2024-09-19T07:31:24.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fcff9640 1 -- 172.21.15.83:0/3829729890 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f69f001a070 con 0x7f6a000c23d0 2024-09-19T07:31:24.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.896+0000 7f69fcff9640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69ec0678c0 0x7f69ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:24.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.900+0000 7f69fcff9640 1 -- 172.21.15.83:0/3829729890 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7f69f0085a10 con 0x7f6a000c23d0 2024-09-19T07:31:24.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.900+0000 7f69ff7fe640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69ec0678c0 0x7f69ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:24.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.900+0000 7f69ff7fe640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69ec0678c0 0x7f69ec069d80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f69f40ad000 tx=0x7f69f4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:24.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:24.904+0000 7f69fcff9640 1 -- 172.21.15.83:0/3829729890 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f69f0055590 con 0x7f6a000c23d0 2024-09-19T07:31:25.044 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:31:25.048+0000 7fd099499640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:31:25.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 -- 172.21.15.83:0/3829729890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a00158960 con 0x7f6a000c23d0 2024-09-19T07:31:25.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f69fcff9640 1 -- 172.21.15.83:0/3829729890 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5714 (secure 0 0 0) 0x7f69f0059890 con 0x7f6a000c23d0 2024-09-19T07:31:25.200 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:25.200 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":99,"btime":"2024-09-19T07:31:21:234495+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:31:25.201 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 99 2024-09-19T07:31:25.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 -- 172.21.15.83:0/3829729890 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69ec0678c0 msgr2=0x7f69ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69ec0678c0 0x7f69ec069d80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f69f40ad000 tx=0x7f69f4008040 comp rx=0 tx=0).stop 2024-09-19T07:31:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 -- 172.21.15.83:0/3829729890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a000c23d0 msgr2=0x7f6a001eef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a000c23d0 0x7f6a001eef80 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f69f000c480 tx=0x7f69f000c940 comp rx=0 tx=0).stop 2024-09-19T07:31:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 -- 172.21.15.83:0/3829729890 shutdown_connections 2024-09-19T07:31:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69ec0678c0 0x7f69ec069d80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a00155bb0 0x7f6a0014fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a00153300 0x7f6a00150b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 --2- 172.21.15.83:0/3829729890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a000c23d0 0x7f6a001eef80 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 -- 172.21.15.83:0/3829729890 >> 172.21.15.83:0/3829729890 conn(0x7f6a000c5090 msgr2=0x7f6a0006cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 -- 172.21.15.83:0/3829729890 shutdown_connections 2024-09-19T07:31:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:25.204+0000 7f6a06929640 1 -- 172.21.15.83:0/3829729890 wait complete. 2024-09-19T07:31:25.219 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-19T07:31:27.220 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:31:27.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f16725ca640 1 Processor -- start 2024-09-19T07:31:27.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f16725ca640 1 -- start start 2024-09-19T07:31:27.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f16725ca640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f166c071e20 0x7f166c0716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:27.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f16725ca640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f166c1515e0 0x7f166c1575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:27.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f16725ca640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f166c0c37c0 con 0x7f166c06f950 2024-09-19T07:31:27.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f16725ca640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f166c0c4670 con 0x7f166c071e20 2024-09-19T07:31:27.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f16725ca640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f166c0c3a60 con 0x7f166c1515e0 2024-09-19T07:31:27.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f166b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f166c071e20 0x7f166c0716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:27.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f166b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f166c071e20 0x7f166c0716a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48172/0 (socket says 172.21.15.83:48172) 2024-09-19T07:31:27.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f166b7fe640 1 -- 172.21.15.83:0/1911425422 learned_addr learned my addr 172.21.15.83:0/1911425422 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:27.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f1670b67640 1 --2- 172.21.15.83:0/1911425422 >> v2:172.21.15.184:3300/0 conn(0x7f166c1515e0 0x7f166c1575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:27.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f166b7fe640 1 -- 172.21.15.83:0/1911425422 >> v2:172.21.15.184:3300/0 conn(0x7f166c1515e0 msgr2=0x7f166c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:31:27.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f166b7fe640 1 --2- 172.21.15.83:0/1911425422 >> v2:172.21.15.184:3300/0 conn(0x7f166c1515e0 0x7f166c1575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:27.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f166b7fe640 1 -- 172.21.15.83:0/1911425422 >> v1:172.21.15.83:6789/0 conn(0x7f166c06f950 legacy=0x7f166c150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:27.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f166affd640 1 -- 172.21.15.83:0/1911425422 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2786471793 0 0) 0x7f166c0c37c0 con 0x7f166c06f950 2024-09-19T07:31:27.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.356+0000 7f166b7fe640 1 -- 172.21.15.83:0/1911425422 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166c0c3d40 con 0x7f166c071e20 2024-09-19T07:31:27.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f166b7fe640 1 --2- 172.21.15.83:0/1911425422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c071e20 0x7f166c0716a0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f166c0ab690 tx=0x7f166001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9200afb08e1c7bd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:27.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f166affd640 1 -- 172.21.15.83:0/1911425422 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16600b0070 con 0x7f166c071e20 2024-09-19T07:31:27.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f166affd640 1 -- 172.21.15.83:0/1911425422 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16600ab600 con 0x7f166c071e20 2024-09-19T07:31:27.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f166affd640 1 -- 172.21.15.83:0/1911425422 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16600ab9d0 con 0x7f166c071e20 2024-09-19T07:31:27.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f16725ca640 1 -- 172.21.15.83:0/1911425422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c071e20 msgr2=0x7f166c0716a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:27.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f16725ca640 1 --2- 172.21.15.83:0/1911425422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c071e20 0x7f166c0716a0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f166c0ab690 tx=0x7f166001c4b0 comp rx=0 tx=0).stop 2024-09-19T07:31:27.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f16725ca640 1 -- 172.21.15.83:0/1911425422 shutdown_connections 2024-09-19T07:31:27.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f16725ca640 1 --2- 172.21.15.83:0/1911425422 >> v2:172.21.15.184:3300/0 conn(0x7f166c1515e0 0x7f166c1575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:27.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f16725ca640 1 --2- 172.21.15.83:0/1911425422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c071e20 0x7f166c0716a0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:27.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f16725ca640 1 -- 172.21.15.83:0/1911425422 >> 172.21.15.83:0/1911425422 conn(0x7f166c0c48b0 msgr2=0x7f166c0c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:27.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f16725ca640 1 -- 172.21.15.83:0/1911425422 shutdown_connections 2024-09-19T07:31:27.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f16725ca640 1 -- 172.21.15.83:0/1911425422 wait complete. 2024-09-19T07:31:27.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f16725ca640 1 Processor -- start 2024-09-19T07:31:27.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.360+0000 7f16725ca640 1 -- start start 2024-09-19T07:31:27.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f16725ca640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c06f950 0x7f166c1587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:27.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f16725ca640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f166c1515e0 0x7f166c14d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:27.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f16725ca640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c1ed8b0 0x7f166c1edd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:27.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f16725ca640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f166c160690 con 0x7f166c06f950 2024-09-19T07:31:27.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f16725ca640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f166c06ff40 con 0x7f166c1ed8b0 2024-09-19T07:31:27.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f16725ca640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f166c160810 con 0x7f166c1515e0 2024-09-19T07:31:27.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f166bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c06f950 0x7f166c1587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:27.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f166bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c06f950 0x7f166c1587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48176/0 (socket says 172.21.15.83:48176) 2024-09-19T07:31:27.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f166bfff640 1 -- 172.21.15.83:0/3238421495 learned_addr learned my addr 172.21.15.83:0/3238421495 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:31:27.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f166b7fe640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f166c1515e0 0x7f166c14d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:27.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f1670b67640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c1ed8b0 0x7f166c1edd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:27.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f166bfff640 1 -- 172.21.15.83:0/3238421495 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f166c1515e0 msgr2=0x7f166c14d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f166bfff640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f166c1515e0 0x7f166c14d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f166bfff640 1 -- 172.21.15.83:0/3238421495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c1ed8b0 msgr2=0x7f166c1edd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f166bfff640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c1ed8b0 0x7f166c1edd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f166bfff640 1 -- 172.21.15.83:0/3238421495 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166c158cf0 con 0x7f166c06f950 2024-09-19T07:31:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f166b7fe640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f166c1515e0 0x7f166c14d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:31:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f166bfff640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c06f950 0x7f166c1587b0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f165c00c450 tx=0x7f165c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f1668ff9640 1 -- 172.21.15.83:0/3238421495 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c017070 con 0x7f166c06f950 2024-09-19T07:31:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f1668ff9640 1 -- 172.21.15.83:0/3238421495 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f165c005480 con 0x7f166c06f950 2024-09-19T07:31:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f16725ca640 1 -- 172.21.15.83:0/3238421495 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166c14bdf0 con 0x7f166c06f950 2024-09-19T07:31:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f1668ff9640 1 -- 172.21.15.83:0/3238421495 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c003840 con 0x7f166c06f950 2024-09-19T07:31:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f16725ca640 1 -- 172.21.15.83:0/3238421495 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f166c14c3b0 con 0x7f166c06f950 2024-09-19T07:31:27.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f16527fc640 1 -- 172.21.15.83:0/3238421495 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16440052a0 con 0x7f166c06f950 2024-09-19T07:31:27.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f1668ff9640 1 -- 172.21.15.83:0/3238421495 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f165c01a070 con 0x7f166c06f950 2024-09-19T07:31:27.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f1668ff9640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16580678c0 0x7f1658069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:31:27.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f1668ff9640 1 -- 172.21.15.83:0/3238421495 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7f165c0859b0 con 0x7f166c06f950 2024-09-19T07:31:27.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.364+0000 7f166b7fe640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16580678c0 0x7f1658069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:31:27.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.368+0000 7f166b7fe640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16580678c0 0x7f1658069d80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f16600ad000 tx=0x7f16600a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:31:27.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.372+0000 7f1668ff9640 1 -- 172.21.15.83:0/3238421495 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f165c055b00 con 0x7f166c06f950 2024-09-19T07:31:27.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.600+0000 7f16527fc640 1 -- 172.21.15.83:0/3238421495 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1644005590 con 0x7f166c06f950 2024-09-19T07:31:27.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f1668ff9640 1 -- 172.21.15.83:0/3238421495 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+6541 (secure 0 0 0) 0x7f165c055400 con 0x7f166c06f950 2024-09-19T07:31:27.599 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:31:27.600 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":100,"btime":"2024-09-19T07:31:25:264553+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6561,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:31:27.600 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 100 2024-09-19T07:31:27.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f16527fc640 1 -- 172.21.15.83:0/3238421495 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16580678c0 msgr2=0x7f1658069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:27.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f16527fc640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16580678c0 0x7f1658069d80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f16600ad000 tx=0x7f16600a6000 comp rx=0 tx=0).stop 2024-09-19T07:31:27.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f16527fc640 1 -- 172.21.15.83:0/3238421495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c06f950 msgr2=0x7f166c1587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:31:27.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f16527fc640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c06f950 0x7f166c1587b0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f165c00c450 tx=0x7f165c00c910 comp rx=0 tx=0).stop 2024-09-19T07:31:27.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f16527fc640 1 -- 172.21.15.83:0/3238421495 shutdown_connections 2024-09-19T07:31:27.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f16527fc640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16580678c0 0x7f1658069d80 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:27.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f16527fc640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f166c1ed8b0 0x7f166c1edd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:27.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f16527fc640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f166c1515e0 0x7f166c14d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:27.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f16527fc640 1 --2- 172.21.15.83:0/3238421495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c06f950 0x7f166c1587b0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:31:27.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f16527fc640 1 -- 172.21.15.83:0/3238421495 >> 172.21.15.83:0/3238421495 conn(0x7f166c0c48b0 msgr2=0x7f166c0c26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:31:27.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f16527fc640 1 -- 172.21.15.83:0/3238421495 shutdown_connections 2024-09-19T07:31:27.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:31:27.604+0000 7f16527fc640 1 -- 172.21.15.83:0/3238421495 wait complete. 2024-09-19T07:31:27.621 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-19T07:31:27.621 INFO:tasks.mds_thrash.fs.[b]:waiting for 105 secs before thrashing 2024-09-19T07:31:32.935 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:31:32.937 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:32:03.251 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:32:03.255 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:32:19.302 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:19.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff303b54640 1 Processor -- start 2024-09-19T07:32:19.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff303b54640 1 -- start start 2024-09-19T07:32:19.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff303b54640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2fc0c7d70 0x7ff2fc157c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff303b54640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2fc072d00 0x7ff2fc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff303b54640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2fc0c74f0 con 0x7ff2fc1547b0 2024-09-19T07:32:19.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff303b54640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff2fc073690 con 0x7ff2fc0c7d70 2024-09-19T07:32:19.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff303b54640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff2fc157160 con 0x7ff2fc072d00 2024-09-19T07:32:19.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff3018f0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2fc0c7d70 0x7ff2fc157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff3010ef640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff2fc1547b0 0x7ff2fc0c6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47944/0 (socket says 172.21.15.83:47944) 2024-09-19T07:32:19.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff3010ef640 1 -- 172.21.15.83:0/372766272 learned_addr learned my addr 172.21.15.83:0/372766272 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:32:19.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff3020f1640 1 --2- 172.21.15.83:0/372766272 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2fc072d00 0x7ff2fc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff3018f0640 1 -- 172.21.15.83:0/372766272 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2fc072d00 msgr2=0x7ff2fc06cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:19.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff3018f0640 1 --2- 172.21.15.83:0/372766272 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2fc072d00 0x7ff2fc06cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff3018f0640 1 -- 172.21.15.83:0/372766272 >> v1:172.21.15.83:6789/0 conn(0x7ff2fc1547b0 legacy=0x7ff2fc0c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:19.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff3018f0640 1 -- 172.21.15.83:0/372766272 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2fc157440 con 0x7ff2fc0c7d70 2024-09-19T07:32:19.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.447+0000 7ff3008ee640 1 -- 172.21.15.83:0/372766272 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288022761 0 0) 0x7ff2fc0c74f0 con 0x7ff2fc1547b0 2024-09-19T07:32:19.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff3018f0640 1 --2- 172.21.15.83:0/372766272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2fc0c7d70 0x7ff2fc157c40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80097c0 tx=0x7ff2f801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b840bd6263a9dcb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:19.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff3008ee640 1 -- 172.21.15.83:0/372766272 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80b0070 con 0x7ff2fc0c7d70 2024-09-19T07:32:19.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff3008ee640 1 -- 172.21.15.83:0/372766272 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2f80ab600 con 0x7ff2fc0c7d70 2024-09-19T07:32:19.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff3008ee640 1 -- 172.21.15.83:0/372766272 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80ab9d0 con 0x7ff2fc0c7d70 2024-09-19T07:32:19.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff303b54640 1 -- 172.21.15.83:0/372766272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2fc0c7d70 msgr2=0x7ff2fc157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:19.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff303b54640 1 --2- 172.21.15.83:0/372766272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2fc0c7d70 0x7ff2fc157c40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80097c0 tx=0x7ff2f801c3e0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff303b54640 1 -- 172.21.15.83:0/372766272 shutdown_connections 2024-09-19T07:32:19.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff303b54640 1 --2- 172.21.15.83:0/372766272 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2fc072d00 0x7ff2fc06cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff303b54640 1 --2- 172.21.15.83:0/372766272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2fc0c7d70 0x7ff2fc157c40 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff303b54640 1 -- 172.21.15.83:0/372766272 >> 172.21.15.83:0/372766272 conn(0x7ff2fc0c5060 msgr2=0x7ff2fc0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:19.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff303b54640 1 -- 172.21.15.83:0/372766272 shutdown_connections 2024-09-19T07:32:19.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff303b54640 1 -- 172.21.15.83:0/372766272 wait complete. 2024-09-19T07:32:19.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff303b54640 1 Processor -- start 2024-09-19T07:32:19.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.451+0000 7ff303b54640 1 -- start start 2024-09-19T07:32:19.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff303b54640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2fc072d00 0x7ff2fc1f08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff303b54640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2fc0c7d70 0x7ff2fc0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff303b54640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2fc1547b0 0x7ff2fc14fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff303b54640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc0703b0 con 0x7ff2fc1547b0 2024-09-19T07:32:19.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff303b54640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc070230 con 0x7ff2fc072d00 2024-09-19T07:32:19.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff303b54640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff2fc070530 con 0x7ff2fc0c7d70 2024-09-19T07:32:19.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff3020f1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2fc1547b0 0x7ff2fc14fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff3020f1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2fc1547b0 0x7ff2fc14fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37246/0 (socket says 172.21.15.83:37246) 2024-09-19T07:32:19.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff3020f1640 1 -- 172.21.15.83:0/3177259795 learned_addr learned my addr 172.21.15.83:0/3177259795 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:19.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff3010ef640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2fc0c7d70 0x7ff2fc0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff3018f0640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2fc072d00 0x7ff2fc1f08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff3020f1640 1 -- 172.21.15.83:0/3177259795 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2fc0c7d70 msgr2=0x7ff2fc0c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:19.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff3020f1640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2fc0c7d70 0x7ff2fc0c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff3020f1640 1 -- 172.21.15.83:0/3177259795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2fc072d00 msgr2=0x7ff2fc1f08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:19.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff3020f1640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2fc072d00 0x7ff2fc1f08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff3020f1640 1 -- 172.21.15.83:0/3177259795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2fc1f0df0 con 0x7ff2fc1547b0 2024-09-19T07:32:19.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff3020f1640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2fc1547b0 0x7ff2fc14fce0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4007e70 tx=0x7ff2f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:19.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff2f27fc640 1 -- 172.21.15.83:0/3177259795 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4014620 con 0x7ff2fc1547b0 2024-09-19T07:32:19.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff2f27fc640 1 -- 172.21.15.83:0/3177259795 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2f4019070 con 0x7ff2fc1547b0 2024-09-19T07:32:19.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff303b54640 1 -- 172.21.15.83:0/3177259795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2fc0c28c0 con 0x7ff2fc1547b0 2024-09-19T07:32:19.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff2f27fc640 1 -- 172.21.15.83:0/3177259795 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4014a20 con 0x7ff2fc1547b0 2024-09-19T07:32:19.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff303b54640 1 -- 172.21.15.83:0/3177259795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2fc14d690 con 0x7ff2fc1547b0 2024-09-19T07:32:19.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.455+0000 7ff303b54640 1 -- 172.21.15.83:0/3177259795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2fc06cc90 con 0x7ff2fc1547b0 2024-09-19T07:32:19.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.459+0000 7ff2f27fc640 1 -- 172.21.15.83:0/3177259795 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff2f4014bc0 con 0x7ff2fc1547b0 2024-09-19T07:32:19.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.459+0000 7ff2f27fc640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2e00678c0 0x7ff2e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.459+0000 7ff3018f0640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2e00678c0 0x7ff2e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.459+0000 7ff2f27fc640 1 -- 172.21.15.83:0/3177259795 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7ff2f4087a00 con 0x7ff2fc1547b0 2024-09-19T07:32:19.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.459+0000 7ff3018f0640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2e00678c0 0x7ff2e0069d80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc151780 tx=0x7ff2f80a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.463+0000 7ff2f27fc640 1 -- 172.21.15.83:0/3177259795 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff2f4057c50 con 0x7ff2fc1547b0 2024-09-19T07:32:19.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 -- 172.21.15.83:0/3177259795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2fc15f200 con 0x7ff2fc1547b0 2024-09-19T07:32:19.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff2f27fc640 1 -- 172.21.15.83:0/3177259795 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+6541 (secure 0 0 0) 0x7ff2f4057550 con 0x7ff2fc1547b0 2024-09-19T07:32:19.692 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:19.692 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":100,"btime":"2024-09-19T07:31:25:264553+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6561,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:19.692 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 100 2024-09-19T07:32:19.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 -- 172.21.15.83:0/3177259795 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2e00678c0 msgr2=0x7ff2e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:19.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2e00678c0 0x7ff2e0069d80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc151780 tx=0x7ff2f80a6000 comp rx=0 tx=0).stop 2024-09-19T07:32:19.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 -- 172.21.15.83:0/3177259795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2fc1547b0 msgr2=0x7ff2fc14fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:19.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2fc1547b0 0x7ff2fc14fce0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4007e70 tx=0x7ff2f400e4b0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 -- 172.21.15.83:0/3177259795 shutdown_connections 2024-09-19T07:32:19.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2e00678c0 0x7ff2e0069d80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2fc1547b0 0x7ff2fc14fce0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2fc0c7d70 0x7ff2fc0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 --2- 172.21.15.83:0/3177259795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2fc072d00 0x7ff2fc1f08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 -- 172.21.15.83:0/3177259795 >> 172.21.15.83:0/3177259795 conn(0x7ff2fc0c5060 msgr2=0x7ff2fc0c6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:19.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 -- 172.21.15.83:0/3177259795 shutdown_connections 2024-09-19T07:32:19.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.699+0000 7ff303b54640 1 -- 172.21.15.83:0/3177259795 wait complete. 2024-09-19T07:32:19.715 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-19T07:32:19.715 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-19T07:32:19.716 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:32:19.735 INFO:tasks.ceph.mds.d:Stopped 2024-09-19T07:32:19.735 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153ff6e640 1 Processor -- start 2024-09-19T07:32:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153ff6e640 1 -- start start 2024-09-19T07:32:19.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153ff6e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1538155d70 0x7f15380c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153ff6e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1538156200 con 0x7f15380c6120 2024-09-19T07:32:19.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153ff6e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15381564e0 con 0x7f15381592f0 2024-09-19T07:32:19.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153ff6e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f15380c9930 con 0x7f1538155d70 2024-09-19T07:32:19.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153dd0a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f15380c6120 0x7f15381575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47950/0 (socket says 172.21.15.83:47950) 2024-09-19T07:32:19.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153dd0a640 1 -- 172.21.15.83:0/3269438387 learned_addr learned my addr 172.21.15.83:0/3269438387 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:32:19.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153d509640 1 --2- 172.21.15.83:0/3269438387 >> v2:172.21.15.184:3300/0 conn(0x7f1538155d70 0x7f15380c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875838869 0 0) 0x7f1538156200 con 0x7f15380c6120 2024-09-19T07:32:19.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1520003530 con 0x7f15380c6120 2024-09-19T07:32:19.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2600481066 0 0) 0x7f1520003530 con 0x7f15380c6120 2024-09-19T07:32:19.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 >> v2:172.21.15.184:3300/0 conn(0x7f1538155d70 msgr2=0x7f15380c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:19.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 --2- 172.21.15.83:0/3269438387 >> v2:172.21.15.184:3300/0 conn(0x7f1538155d70 0x7f15380c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 >> v1:172.21.15.184:6789/0 conn(0x7f15381592f0 legacy=0x7f1538157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:19.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15381567c0 con 0x7f15380c6120 2024-09-19T07:32:19.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f1528001a00 con 0x7f15380c6120 2024-09-19T07:32:19.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 >> v1:172.21.15.83:6789/0 conn(0x7f15380c6120 legacy=0x7f15381575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:19.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 0x7f1520003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1520004810 0x7f1520004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1520005220 0x7f152000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f152000c140 con 0x7f1520003a10 2024-09-19T07:32:19.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f152000c2f0 con 0x7f1520005220 2024-09-19T07:32:19.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.871+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f152000c4f0 con 0x7f1520004810 2024-09-19T07:32:19.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153dd0a640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 0x7f1520003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153d509640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1520004810 0x7f1520004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153e50b640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1520005220 0x7f152000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153dd0a640 1 -- 172.21.15.83:0/3269438387 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1520004810 msgr2=0x7f1520004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:19.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153dd0a640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1520004810 0x7f1520004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153dd0a640 1 -- 172.21.15.83:0/3269438387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1520005220 msgr2=0x7f152000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:19.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153dd0a640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1520005220 0x7f152000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153dd0a640 1 -- 172.21.15.83:0/3269438387 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1520003530 con 0x7f1520003a10 2024-09-19T07:32:19.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153d509640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1520004810 0x7f1520004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:32:19.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153e50b640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1520005220 0x7f152000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:32:19.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153e50b640 1 -- 172.21.15.83:0/3269438387 reap_dead start 2024-09-19T07:32:19.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153dd0a640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 0x7f1520003e30 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f152800a760 tx=0x7f152801d260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3329caad2cccdb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:19.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152800a3b0 con 0x7f1520003a10 2024-09-19T07:32:19.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15280ad650 con 0x7f1520003a10 2024-09-19T07:32:19.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153cd08640 1 -- 172.21.15.83:0/3269438387 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15280ada80 con 0x7f1520003a10 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153ff6e640 1 -- 172.21.15.83:0/3269438387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 msgr2=0x7f1520003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153ff6e640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 0x7f1520003e30 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f152800a760 tx=0x7f152801d260 comp rx=0 tx=0).stop 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153ff6e640 1 -- 172.21.15.83:0/3269438387 shutdown_connections 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153ff6e640 1 --2- 172.21.15.83:0/3269438387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 0x7f1520003e30 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153ff6e640 1 -- 172.21.15.83:0/3269438387 >> 172.21.15.83:0/3269438387 conn(0x7f153806bc80 msgr2=0x7f153806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153ff6e640 1 -- 172.21.15.83:0/3269438387 shutdown_connections 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.875+0000 7f153ff6e640 1 -- 172.21.15.83:0/3269438387 wait complete. 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153ff6e640 1 Processor -- start 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153ff6e640 1 -- start start 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153ff6e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 0x7f15380c18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153ff6e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15381ee9a0 0x7f15381eee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153ff6e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15381ef360 0x7f15380c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153ff6e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1538153860 con 0x7f1520003a10 2024-09-19T07:32:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153ff6e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1538157de0 con 0x7f15381ee9a0 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153ff6e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f15380cb5a0 con 0x7f15381ef360 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153dd0a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 0x7f15380c18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153dd0a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 0x7f15380c18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37258/0 (socket says 172.21.15.83:37258) 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153dd0a640 1 -- 172.21.15.83:0/3781938096 learned_addr learned my addr 172.21.15.83:0/3781938096 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153d509640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15381ee9a0 0x7f15381eee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153e50b640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15381ef360 0x7f15380c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153dd0a640 1 -- 172.21.15.83:0/3781938096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15381ef360 msgr2=0x7f15380c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153dd0a640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15381ef360 0x7f15380c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153dd0a640 1 -- 172.21.15.83:0/3781938096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15381ee9a0 msgr2=0x7f15381eee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153dd0a640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15381ee9a0 0x7f15381eee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153dd0a640 1 -- 172.21.15.83:0/3781938096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15381f2850 con 0x7f1520003a10 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153e50b640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15381ef360 0x7f15380c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:32:19.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153dd0a640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 0x7f15380c18f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f152801d090 tx=0x7f1520005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:19.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f15367fc640 1 -- 172.21.15.83:0/3781938096 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15280bb070 con 0x7f1520003a10 2024-09-19T07:32:19.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f15367fc640 1 -- 172.21.15.83:0/3781938096 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15280ad430 con 0x7f1520003a10 2024-09-19T07:32:19.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153ff6e640 1 -- 172.21.15.83:0/3781938096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15381f2af0 con 0x7f1520003a10 2024-09-19T07:32:19.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f15367fc640 1 -- 172.21.15.83:0/3781938096 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1528001a00 con 0x7f1520003a10 2024-09-19T07:32:19.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153ff6e640 1 -- 172.21.15.83:0/3781938096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15380cbad0 con 0x7f1520003a10 2024-09-19T07:32:19.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f153ff6e640 1 -- 172.21.15.83:0/3781938096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1538158110 con 0x7f1520003a10 2024-09-19T07:32:19.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.879+0000 7f15367fc640 1 -- 172.21.15.83:0/3781938096 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f15280be070 con 0x7f1520003a10 2024-09-19T07:32:19.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.883+0000 7f15367fc640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15240678f0 0x7f1524069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:19.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.883+0000 7f153d509640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15240678f0 0x7f1524069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:19.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.883+0000 7f15367fc640 1 -- 172.21.15.83:0/3781938096 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7f15281205f0 con 0x7f1520003a10 2024-09-19T07:32:19.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.883+0000 7f153d509640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15240678f0 0x7f1524069db0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f153814d2b0 tx=0x7f152c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:19.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:19.887+0000 7f15367fc640 1 -- 172.21.15.83:0/3781938096 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1528016f70 con 0x7f1520003a10 2024-09-19T07:32:20.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.083+0000 7f153ff6e640 1 -- 172.21.15.83:0/3781938096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15380c96a0 con 0x7f1520003a10 2024-09-19T07:32:20.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f15367fc640 1 -- 172.21.15.83:0/3781938096 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+6541 (secure 0 0 0) 0x7f15280f0170 con 0x7f1520003a10 2024-09-19T07:32:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:20.080 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":100,"btime":"2024-09-19T07:31:25:264553+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6561,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:20.080 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 100 2024-09-19T07:32:20.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f153ff6e640 1 -- 172.21.15.83:0/3781938096 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15240678f0 msgr2=0x7f1524069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:20.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f153ff6e640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15240678f0 0x7f1524069db0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f153814d2b0 tx=0x7f152c008040 comp rx=0 tx=0).stop 2024-09-19T07:32:20.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f153ff6e640 1 -- 172.21.15.83:0/3781938096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 msgr2=0x7f15380c18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:20.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f153ff6e640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 0x7f15380c18f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f152801d090 tx=0x7f1520005840 comp rx=0 tx=0).stop 2024-09-19T07:32:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f153ff6e640 1 -- 172.21.15.83:0/3781938096 shutdown_connections 2024-09-19T07:32:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f153ff6e640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15240678f0 0x7f1524069db0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f153ff6e640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15381ef360 0x7f15380c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f153ff6e640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15381ee9a0 0x7f15381eee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f153ff6e640 1 --2- 172.21.15.83:0/3781938096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1520003a10 0x7f15380c18f0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f153ff6e640 1 -- 172.21.15.83:0/3781938096 >> 172.21.15.83:0/3781938096 conn(0x7f153806bc80 msgr2=0x7f15381508f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f153ff6e640 1 -- 172.21.15.83:0/3781938096 shutdown_connections 2024-09-19T07:32:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:20.087+0000 7f153ff6e640 1 -- 172.21.15.83:0/3781938096 wait complete. 2024-09-19T07:32:20.100 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:32:22.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:22.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.239+0000 7f0c9ec9c640 1 Processor -- start 2024-09-19T07:32:22.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.239+0000 7f0c9ec9c640 1 -- start start 2024-09-19T07:32:22.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ec9c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c980c42e0 0x7f0c9806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:22.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ec9c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f0c981566b0 0x7f0c980abb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:22.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ec9c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c981592c0 con 0x7f0c98159a80 2024-09-19T07:32:22.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ec9c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f0c980ac070 con 0x7f0c980c42e0 2024-09-19T07:32:22.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ec9c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f0c981595f0 con 0x7f0c981566b0 2024-09-19T07:32:22.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ca38640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c980c42e0 0x7f0c9806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:22.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ca38640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c980c42e0 0x7f0c9806e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41082/0 (socket says 172.21.15.83:41082) 2024-09-19T07:32:22.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ca38640 1 -- 172.21.15.83:0/3959035080 learned_addr learned my addr 172.21.15.83:0/3959035080 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:22.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c97fff640 1 --2- 172.21.15.83:0/3959035080 >> v2:172.21.15.184:3300/0 conn(0x7f0c981566b0 0x7f0c980abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:22.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ca38640 1 -- 172.21.15.83:0/3959035080 >> v2:172.21.15.184:3300/0 conn(0x7f0c981566b0 msgr2=0x7f0c980abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:22.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ca38640 1 --2- 172.21.15.83:0/3959035080 >> v2:172.21.15.184:3300/0 conn(0x7f0c981566b0 0x7f0c980abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:22.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ca38640 1 -- 172.21.15.83:0/3959035080 >> v1:172.21.15.184:6789/0 conn(0x7f0c98159a80 legacy=0x7f0c98153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:22.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ca38640 1 -- 172.21.15.83:0/3959035080 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9806e900 con 0x7f0c980c42e0 2024-09-19T07:32:22.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c9ca38640 1 --2- 172.21.15.83:0/3959035080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c980c42e0 0x7f0c9806e270 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f0c980c51b0 tx=0x7f0c9001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdfc0c35e51f0045 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:22.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c977fe640 1 -- 172.21.15.83:0/3959035080 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c900aa610 con 0x7f0c980c42e0 2024-09-19T07:32:22.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c977fe640 1 -- 172.21.15.83:0/3959035080 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c900af070 con 0x7f0c980c42e0 2024-09-19T07:32:22.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.243+0000 7f0c977fe640 1 -- 172.21.15.83:0/3959035080 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c900aa9d0 con 0x7f0c980c42e0 2024-09-19T07:32:22.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/3959035080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c980c42e0 msgr2=0x7f0c9806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:22.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 --2- 172.21.15.83:0/3959035080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c980c42e0 0x7f0c9806e270 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f0c980c51b0 tx=0x7f0c9001c490 comp rx=0 tx=0).stop 2024-09-19T07:32:22.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/3959035080 shutdown_connections 2024-09-19T07:32:22.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 --2- 172.21.15.83:0/3959035080 >> v2:172.21.15.184:3300/0 conn(0x7f0c981566b0 0x7f0c980abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:22.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 --2- 172.21.15.83:0/3959035080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c980c42e0 0x7f0c9806e270 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:22.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/3959035080 >> 172.21.15.83:0/3959035080 conn(0x7f0c9806bc80 msgr2=0x7f0c9806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:22.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/3959035080 shutdown_connections 2024-09-19T07:32:22.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/3959035080 wait complete. 2024-09-19T07:32:22.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 Processor -- start 2024-09-19T07:32:22.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 -- start start 2024-09-19T07:32:22.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c980c42e0 0x7f0c980c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:22.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c981566b0 0x7f0c980c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:22.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c98159a80 0x7f0c98154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:22.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0c981538f0 con 0x7f0c981566b0 2024-09-19T07:32:22.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0c98153770 con 0x7f0c980c42e0 2024-09-19T07:32:22.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0c980c46c0 con 0x7f0c98159a80 2024-09-19T07:32:22.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c97fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c981566b0 0x7f0c980c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c97fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c981566b0 0x7f0c980c6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41092/0 (socket says 172.21.15.83:41092) 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c97fff640 1 -- 172.21.15.83:0/1869941401 learned_addr learned my addr 172.21.15.83:0/1869941401 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ca38640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c980c42e0 0x7f0c980c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9d239640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c98159a80 0x7f0c98154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c97fff640 1 -- 172.21.15.83:0/1869941401 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c98159a80 msgr2=0x7f0c98154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c97fff640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c98159a80 0x7f0c98154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c97fff640 1 -- 172.21.15.83:0/1869941401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c980c42e0 msgr2=0x7f0c980c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c97fff640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c980c42e0 0x7f0c980c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c97fff640 1 -- 172.21.15.83:0/1869941401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c980c8180 con 0x7f0c981566b0 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ca38640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c980c42e0 0x7f0c980c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c97fff640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c981566b0 0x7f0c980c6ce0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c007e70 tx=0x7f0c8c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c957fa640 1 -- 172.21.15.83:0/1869941401 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c014620 con 0x7f0c981566b0 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c957fa640 1 -- 172.21.15.83:0/1869941401 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c8c019070 con 0x7f0c981566b0 2024-09-19T07:32:22.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/1869941401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c981f1b80 con 0x7f0c981566b0 2024-09-19T07:32:22.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c957fa640 1 -- 172.21.15.83:0/1869941401 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c014a00 con 0x7f0c981566b0 2024-09-19T07:32:22.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/1869941401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c98155ef0 con 0x7f0c981566b0 2024-09-19T07:32:22.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.247+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/1869941401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c98159eb0 con 0x7f0c981566b0 2024-09-19T07:32:22.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.251+0000 7f0c957fa640 1 -- 172.21.15.83:0/1869941401 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0c8c014ba0 con 0x7f0c981566b0 2024-09-19T07:32:22.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.251+0000 7f0c957fa640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0c840678c0 0x7f0c84069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:22.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.251+0000 7f0c957fa640 1 -- 172.21.15.83:0/1869941401 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7f0c8c086eb0 con 0x7f0c981566b0 2024-09-19T07:32:22.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.251+0000 7f0c9ca38640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0c840678c0 0x7f0c84069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:22.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.251+0000 7f0c9ca38640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0c840678c0 0x7f0c84069d80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0c900ae000 tx=0x7f0c900a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:22.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.255+0000 7f0c957fa640 1 -- 172.21.15.83:0/1869941401 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f0c8c011260 con 0x7f0c981566b0 2024-09-19T07:32:22.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.459+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/1869941401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c980abb10 con 0x7f0c981566b0 2024-09-19T07:32:22.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.459+0000 7f0c957fa640 1 -- 172.21.15.83:0/1869941401 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+6541 (secure 0 0 0) 0x7f0c8c057080 con 0x7f0c981566b0 2024-09-19T07:32:22.453 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:22.453 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":100,"btime":"2024-09-19T07:31:25:264553+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6561,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:22.453 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 100 2024-09-19T07:32:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.463+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/1869941401 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0c840678c0 msgr2=0x7f0c84069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.463+0000 7f0c9ec9c640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0c840678c0 0x7f0c84069d80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0c900ae000 tx=0x7f0c900a7000 comp rx=0 tx=0).stop 2024-09-19T07:32:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.463+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/1869941401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c981566b0 msgr2=0x7f0c980c6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.463+0000 7f0c9ec9c640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c981566b0 0x7f0c980c6ce0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c007e70 tx=0x7f0c8c00e4b0 comp rx=0 tx=0).stop 2024-09-19T07:32:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.463+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/1869941401 shutdown_connections 2024-09-19T07:32:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.463+0000 7f0c9ec9c640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0c840678c0 0x7f0c84069d80 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.463+0000 7f0c9ec9c640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c98159a80 0x7f0c98154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.463+0000 7f0c9ec9c640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c981566b0 0x7f0c980c6ce0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.463+0000 7f0c9ec9c640 1 --2- 172.21.15.83:0/1869941401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c980c42e0 0x7f0c980c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.463+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/1869941401 >> 172.21.15.83:0/1869941401 conn(0x7f0c9806bc80 msgr2=0x7f0c98153040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.463+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/1869941401 shutdown_connections 2024-09-19T07:32:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:22.463+0000 7f0c9ec9c640 1 -- 172.21.15.83:0/1869941401 wait complete. 2024-09-19T07:32:22.472 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:32:24.473 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:24.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7cefff640 1 Processor -- start 2024-09-19T07:32:24.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7cefff640 1 -- start start 2024-09-19T07:32:24.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7cefff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa7c8150290 0x7fa7c80c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:24.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7cefff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fa7c8153eb0 0x7fa7c8154340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:24.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7cefff640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c8150670 con 0x7fa7c8152b20 2024-09-19T07:32:24.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7cefff640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa7c80716c0 con 0x7fa7c8150290 2024-09-19T07:32:24.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7cefff640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fa7c80c3920 con 0x7fa7c8153eb0 2024-09-19T07:32:24.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7ccd9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa7c8150290 0x7fa7c80c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:24.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7ccd9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa7c8150290 0x7fa7c80c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41096/0 (socket says 172.21.15.83:41096) 2024-09-19T07:32:24.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7ccd9b640 1 -- 172.21.15.83:0/580824254 learned_addr learned my addr 172.21.15.83:0/580824254 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:24.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7cd59c640 1 --2- 172.21.15.83:0/580824254 >> v2:172.21.15.184:3300/0 conn(0x7fa7c8153eb0 0x7fa7c8154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:24.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7ccd9b640 1 -- 172.21.15.83:0/580824254 >> v2:172.21.15.184:3300/0 conn(0x7fa7c8153eb0 msgr2=0x7fa7c8154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:24.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7ccd9b640 1 --2- 172.21.15.83:0/580824254 >> v2:172.21.15.184:3300/0 conn(0x7fa7c8153eb0 0x7fa7c8154340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:24.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7ccd9b640 1 -- 172.21.15.83:0/580824254 >> v1:172.21.15.184:6789/0 conn(0x7fa7c8152b20 legacy=0x7fa7c80c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:24.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.611+0000 7fa7ccd9b640 1 -- 172.21.15.83:0/580824254 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c80c2880 con 0x7fa7c8150290 2024-09-19T07:32:24.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7ccd9b640 1 --2- 172.21.15.83:0/580824254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7c8150290 0x7fa7c80c1ff0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fa7c80c18a0 tx=0x7fa7b801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aaf638d76a6999d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:24.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c77fe640 1 -- 172.21.15.83:0/580824254 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80b1070 con 0x7fa7c8150290 2024-09-19T07:32:24.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c77fe640 1 -- 172.21.15.83:0/580824254 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7b80ac630 con 0x7fa7c8150290 2024-09-19T07:32:24.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c77fe640 1 -- 172.21.15.83:0/580824254 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80a8050 con 0x7fa7c8150290 2024-09-19T07:32:24.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 -- 172.21.15.83:0/580824254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7c8150290 msgr2=0x7fa7c80c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:24.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 --2- 172.21.15.83:0/580824254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7c8150290 0x7fa7c80c1ff0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fa7c80c18a0 tx=0x7fa7b801c4e0 comp rx=0 tx=0).stop 2024-09-19T07:32:24.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 -- 172.21.15.83:0/580824254 shutdown_connections 2024-09-19T07:32:24.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 --2- 172.21.15.83:0/580824254 >> v2:172.21.15.184:3300/0 conn(0x7fa7c8153eb0 0x7fa7c8154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:24.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 --2- 172.21.15.83:0/580824254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7c8150290 0x7fa7c80c1ff0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:24.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 -- 172.21.15.83:0/580824254 >> 172.21.15.83:0/580824254 conn(0x7fa7c806bc80 msgr2=0x7fa7c806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:24.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 -- 172.21.15.83:0/580824254 shutdown_connections 2024-09-19T07:32:24.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 -- 172.21.15.83:0/580824254 wait complete. 2024-09-19T07:32:24.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 Processor -- start 2024-09-19T07:32:24.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 -- start start 2024-09-19T07:32:24.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa7c8150290 0x7fa7c806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:24.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7c8152b20 0x7fa7c81457f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:24.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7c8153eb0 0x7fa7c8145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:24.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c80c4790 con 0x7fa7c8152b20 2024-09-19T07:32:24.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c80c2530 con 0x7fa7c8153eb0 2024-09-19T07:32:24.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa7c80c4910 con 0x7fa7c8150290 2024-09-19T07:32:24.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7c8152b20 0x7fa7c81457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:24.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7c8152b20 0x7fa7c81457f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41102/0 (socket says 172.21.15.83:41102) 2024-09-19T07:32:24.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c7fff640 1 -- 172.21.15.83:0/3826626718 learned_addr learned my addr 172.21.15.83:0/3826626718 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:24.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7ccd9b640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa7c8150290 0x7fa7c806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:24.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cd59c640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7c8153eb0 0x7fa7c8145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:24.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c7fff640 1 -- 172.21.15.83:0/3826626718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa7c8150290 msgr2=0x7fa7c806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:24.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c7fff640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa7c8150290 0x7fa7c806e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:24.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c7fff640 1 -- 172.21.15.83:0/3826626718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7c8153eb0 msgr2=0x7fa7c8145d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:24.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c7fff640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7c8153eb0 0x7fa7c8145d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:24.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c7fff640 1 -- 172.21.15.83:0/3826626718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c806ec50 con 0x7fa7c8152b20 2024-09-19T07:32:24.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cd59c640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7c8153eb0 0x7fa7c8145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:32:24.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c7fff640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7c8152b20 0x7fa7c81457f0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00c450 tx=0x7fa7bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:24.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c57fa640 1 -- 172.21.15.83:0/3826626718 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc012830 con 0x7fa7c8152b20 2024-09-19T07:32:24.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c57fa640 1 -- 172.21.15.83:0/3826626718 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7bc017070 con 0x7fa7c8152b20 2024-09-19T07:32:24.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 -- 172.21.15.83:0/3826626718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c8150990 con 0x7fa7c8152b20 2024-09-19T07:32:24.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7c57fa640 1 -- 172.21.15.83:0/3826626718 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc012c30 con 0x7fa7c8152b20 2024-09-19T07:32:24.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.615+0000 7fa7cefff640 1 -- 172.21.15.83:0/3826626718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7c8150d70 con 0x7fa7c8152b20 2024-09-19T07:32:24.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.619+0000 7fa7cefff640 1 -- 172.21.15.83:0/3826626718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7c80c1ff0 con 0x7fa7c8152b20 2024-09-19T07:32:24.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.619+0000 7fa7c57fa640 1 -- 172.21.15.83:0/3826626718 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa7bc01c460 con 0x7fa7c8152b20 2024-09-19T07:32:24.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.619+0000 7fa7c57fa640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa7b40678c0 0x7fa7b4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:24.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.619+0000 7fa7c57fa640 1 -- 172.21.15.83:0/3826626718 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7fa7bc086380 con 0x7fa7c8152b20 2024-09-19T07:32:24.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.619+0000 7fa7ccd9b640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa7b40678c0 0x7fa7b4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:24.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.619+0000 7fa7ccd9b640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa7b40678c0 0x7fa7b4069d80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80ae000 tx=0x7fa7b8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:24.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.623+0000 7fa7c57fa640 1 -- 172.21.15.83:0/3826626718 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa7bc056550 con 0x7fa7c8152b20 2024-09-19T07:32:24.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.819+0000 7fa7cefff640 1 -- 172.21.15.83:0/3826626718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7c80c1a90 con 0x7fa7c8152b20 2024-09-19T07:32:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.819+0000 7fa7c57fa640 1 -- 172.21.15.83:0/3826626718 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+6541 (secure 0 0 0) 0x7fa7bc055e50 con 0x7fa7c8152b20 2024-09-19T07:32:24.814 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:24.814 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":100,"btime":"2024-09-19T07:31:25:264553+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6561,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 100 2024-09-19T07:32:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.823+0000 7fa7cefff640 1 -- 172.21.15.83:0/3826626718 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa7b40678c0 msgr2=0x7fa7b4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:24.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.823+0000 7fa7cefff640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa7b40678c0 0x7fa7b4069d80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80ae000 tx=0x7fa7b8008040 comp rx=0 tx=0).stop 2024-09-19T07:32:24.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.823+0000 7fa7cefff640 1 -- 172.21.15.83:0/3826626718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7c8152b20 msgr2=0x7fa7c81457f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:24.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.823+0000 7fa7cefff640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7c8152b20 0x7fa7c81457f0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00c450 tx=0x7fa7bc00c910 comp rx=0 tx=0).stop 2024-09-19T07:32:24.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.823+0000 7fa7cefff640 1 -- 172.21.15.83:0/3826626718 shutdown_connections 2024-09-19T07:32:24.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.823+0000 7fa7cefff640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa7b40678c0 0x7fa7b4069d80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:24.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.823+0000 7fa7cefff640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7c8153eb0 0x7fa7c8145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:24.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.823+0000 7fa7cefff640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7c8152b20 0x7fa7c81457f0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:24.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.823+0000 7fa7cefff640 1 --2- 172.21.15.83:0/3826626718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa7c8150290 0x7fa7c806e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:24.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.823+0000 7fa7cefff640 1 -- 172.21.15.83:0/3826626718 >> 172.21.15.83:0/3826626718 conn(0x7fa7c806bc80 msgr2=0x7fa7c814abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:24.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.823+0000 7fa7cefff640 1 -- 172.21.15.83:0/3826626718 shutdown_connections 2024-09-19T07:32:24.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:24.823+0000 7fa7cefff640 1 -- 172.21.15.83:0/3826626718 wait complete. 2024-09-19T07:32:24.832 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:32:26.833 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:26.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa446242640 1 Processor -- start 2024-09-19T07:32:26.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa446242640 1 -- start start 2024-09-19T07:32:26.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa446242640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa440071100 0x7fa4401545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:26.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa446242640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fa4400c28e0 0x7fa440150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:26.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa446242640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa440151480 0x7fa44006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:26.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa446242640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa440071b50 con 0x7fa440151480 2024-09-19T07:32:26.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa446242640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fa4400c1ff0 con 0x7fa4400c28e0 2024-09-19T07:32:26.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa446242640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa4400a9e60 con 0x7fa440071100 2024-09-19T07:32:26.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa43ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa440151480 0x7fa44006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:26.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa43ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa440151480 0x7fa44006dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41114/0 (socket says 172.21.15.83:41114) 2024-09-19T07:32:26.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa43ffff640 1 -- 172.21.15.83:0/1103113585 learned_addr learned my addr 172.21.15.83:0/1103113585 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:26.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa43ffff640 1 -- 172.21.15.83:0/1103113585 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa440071100 msgr2=0x7fa4401545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:26.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa43ffff640 1 --2- 172.21.15.83:0/1103113585 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa440071100 0x7fa4401545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:26.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa43ffff640 1 -- 172.21.15.83:0/1103113585 >> v2:172.21.15.184:3300/0 conn(0x7fa4400c28e0 msgr2=0x7fa440150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:26.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa43ffff640 1 --2- 172.21.15.83:0/1103113585 >> v2:172.21.15.184:3300/0 conn(0x7fa4400c28e0 0x7fa440150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:26.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.971+0000 7fa43ffff640 1 -- 172.21.15.83:0/1103113585 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4400aa140 con 0x7fa440151480 2024-09-19T07:32:26.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa43ffff640 1 --2- 172.21.15.83:0/1103113585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa440151480 0x7fa44006dea0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa44006ca00 tx=0x7fa43801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89c59a4bf14d7c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:26.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa43e7fc640 1 -- 172.21.15.83:0/1103113585 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4380a56d0 con 0x7fa440151480 2024-09-19T07:32:26.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa43e7fc640 1 -- 172.21.15.83:0/1103113585 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4380a5870 con 0x7fa440151480 2024-09-19T07:32:26.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa43e7fc640 1 -- 172.21.15.83:0/1103113585 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4380a5c40 con 0x7fa440151480 2024-09-19T07:32:26.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa446242640 1 -- 172.21.15.83:0/1103113585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa440151480 msgr2=0x7fa44006dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:26.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa446242640 1 --2- 172.21.15.83:0/1103113585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa440151480 0x7fa44006dea0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa44006ca00 tx=0x7fa43801c960 comp rx=0 tx=0).stop 2024-09-19T07:32:26.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa446242640 1 -- 172.21.15.83:0/1103113585 shutdown_connections 2024-09-19T07:32:26.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa446242640 1 --2- 172.21.15.83:0/1103113585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa440151480 0x7fa44006dea0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:26.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa446242640 1 --2- 172.21.15.83:0/1103113585 >> v2:172.21.15.184:3300/0 conn(0x7fa4400c28e0 0x7fa440150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:26.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa446242640 1 --2- 172.21.15.83:0/1103113585 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa440071100 0x7fa4401545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:26.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa446242640 1 -- 172.21.15.83:0/1103113585 >> 172.21.15.83:0/1103113585 conn(0x7fa44006bc80 msgr2=0x7fa44006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:26.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa446242640 1 -- 172.21.15.83:0/1103113585 shutdown_connections 2024-09-19T07:32:26.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa446242640 1 -- 172.21.15.83:0/1103113585 wait complete. 2024-09-19T07:32:26.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa446242640 1 Processor -- start 2024-09-19T07:32:26.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.975+0000 7fa446242640 1 -- start start 2024-09-19T07:32:26.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa446242640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa440071100 0x7fa440154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:26.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa446242640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4400c28e0 0x7fa4401ed9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:26.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa446242640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa440151480 0x7fa4400c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa446242640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa4401570d0 con 0x7fa4400c28e0 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa446242640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa440071730 con 0x7fa440071100 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa446242640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa440157250 con 0x7fa440151480 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4400c28e0 0x7fa4401ed9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4400c28e0 0x7fa4401ed9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41130/0 (socket says 172.21.15.83:41130) 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43effd640 1 -- 172.21.15.83:0/645173944 learned_addr learned my addr 172.21.15.83:0/645173944 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43f7fe640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa440071100 0x7fa440154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43ffff640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa440151480 0x7fa4400c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43effd640 1 -- 172.21.15.83:0/645173944 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa440151480 msgr2=0x7fa4400c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43effd640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa440151480 0x7fa4400c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43effd640 1 -- 172.21.15.83:0/645173944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa440071100 msgr2=0x7fa440154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43effd640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa440071100 0x7fa440154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:26.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43effd640 1 -- 172.21.15.83:0/645173944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4400c7300 con 0x7fa4400c28e0 2024-09-19T07:32:26.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43ffff640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa440151480 0x7fa4400c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:32:26.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43effd640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4400c28e0 0x7fa4401ed9c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa42800b480 tx=0x7fa42800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:26.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa437fff640 1 -- 172.21.15.83:0/645173944 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa428014070 con 0x7fa4400c28e0 2024-09-19T07:32:26.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa437fff640 1 -- 172.21.15.83:0/645173944 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa428010890 con 0x7fa4400c28e0 2024-09-19T07:32:26.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa446242640 1 -- 172.21.15.83:0/645173944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4400c1b50 con 0x7fa4400c28e0 2024-09-19T07:32:26.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa437fff640 1 -- 172.21.15.83:0/645173944 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa428010c80 con 0x7fa4400c28e0 2024-09-19T07:32:26.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa446242640 1 -- 172.21.15.83:0/645173944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4401efed0 con 0x7fa4400c28e0 2024-09-19T07:32:26.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa446242640 1 -- 172.21.15.83:0/645173944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa440150f40 con 0x7fa4400c28e0 2024-09-19T07:32:26.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa437fff640 1 -- 172.21.15.83:0/645173944 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa428017070 con 0x7fa4400c28e0 2024-09-19T07:32:26.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa437fff640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa4200678c0 0x7fa420069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:26.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa437fff640 1 -- 172.21.15.83:0/645173944 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7fa428086840 con 0x7fa4400c28e0 2024-09-19T07:32:26.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.979+0000 7fa43f7fe640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa4200678c0 0x7fa420069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:26.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.983+0000 7fa43f7fe640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa4200678c0 0x7fa420069d80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa430006fd0 tx=0x7fa430008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:26.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:26.987+0000 7fa437fff640 1 -- 172.21.15.83:0/645173944 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa428056a90 con 0x7fa4400c28e0 2024-09-19T07:32:27.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.175+0000 7fa446242640 1 -- 172.21.15.83:0/645173944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa44015e220 con 0x7fa4400c28e0 2024-09-19T07:32:27.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa437fff640 1 -- 172.21.15.83:0/645173944 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+6541 (secure 0 0 0) 0x7fa428010e20 con 0x7fa4400c28e0 2024-09-19T07:32:27.171 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:27.171 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":100,"btime":"2024-09-19T07:31:25:264553+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6561,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:27.172 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 100 2024-09-19T07:32:27.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa446242640 1 -- 172.21.15.83:0/645173944 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa4200678c0 msgr2=0x7fa420069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa446242640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa4200678c0 0x7fa420069d80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa430006fd0 tx=0x7fa430008040 comp rx=0 tx=0).stop 2024-09-19T07:32:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa446242640 1 -- 172.21.15.83:0/645173944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4400c28e0 msgr2=0x7fa4401ed9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa446242640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4400c28e0 0x7fa4401ed9c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa42800b480 tx=0x7fa42800b940 comp rx=0 tx=0).stop 2024-09-19T07:32:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa446242640 1 -- 172.21.15.83:0/645173944 shutdown_connections 2024-09-19T07:32:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa446242640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa4200678c0 0x7fa420069d80 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa446242640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa440151480 0x7fa4400c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa446242640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4400c28e0 0x7fa4401ed9c0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa446242640 1 --2- 172.21.15.83:0/645173944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa440071100 0x7fa440154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa446242640 1 -- 172.21.15.83:0/645173944 >> 172.21.15.83:0/645173944 conn(0x7fa44006bc80 msgr2=0x7fa4400c3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa446242640 1 -- 172.21.15.83:0/645173944 shutdown_connections 2024-09-19T07:32:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:27.179+0000 7fa446242640 1 -- 172.21.15.83:0/645173944 wait complete. 2024-09-19T07:32:27.188 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:32:29.189 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:29.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13f2d0f640 1 Processor -- start 2024-09-19T07:32:29.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13f2d0f640 1 -- start start 2024-09-19T07:32:29.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13f2d0f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec0abe80 0x7f13ec157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:29.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13f2d0f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13ec0c76d0 con 0x7f13ec0ab610 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13f2d0f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13ec0c7de0 con 0x7f13ec06cef0 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13f2d0f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f13ec0c6a90 con 0x7f13ec0abe80 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13ebfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f13ec0ab610 0x7f13ec0c6fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45938/0 (socket says 172.21.15.83:45938) 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13ebfff640 1 -- 172.21.15.83:0/1153486284 learned_addr learned my addr 172.21.15.83:0/1153486284 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13f12ac640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec0abe80 0x7f13ec157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 134187126 0 0) 0x7f13e0001120 con 0x7f13ec0ab610 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d4003530 con 0x7f13ec0ab610 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2547354562 0 0) 0x7f13ec0c7de0 con 0x7f13ec06cef0 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13e0001120 con 0x7f13ec06cef0 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1719107832 0 0) 0x7f13d4003530 con 0x7f13ec0ab610 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec0abe80 msgr2=0x7f13ec157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec0abe80 0x7f13ec157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 >> v1:172.21.15.184:6789/0 conn(0x7f13ec06cef0 legacy=0x7f13ec0c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:29.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec0c8080 con 0x7f13ec0ab610 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f13e0001850 con 0x7f13ec0ab610 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 >> v1:172.21.15.83:6789/0 conn(0x7f13ec0ab610 legacy=0x7f13ec0c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec0abe80 0x7f13d4003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13d4004130 0x7f13d4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13d4004d50 0x7f13d40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13d400bc00 con 0x7f13d4004130 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13d400bdb0 con 0x7f13d4004d50 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f13d400bfb0 con 0x7f13ec0abe80 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13ebfff640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13d4004130 0x7f13d4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13f0aab640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec0abe80 0x7f13d4003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13f12ac640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13d4004d50 0x7f13d40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13ebfff640 1 -- 172.21.15.83:0/1153486284 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec0abe80 msgr2=0x7f13d4003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:29.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13ebfff640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec0abe80 0x7f13d4003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:29.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13ebfff640 1 -- 172.21.15.83:0/1153486284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13d4004d50 msgr2=0x7f13d40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:29.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13ebfff640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13d4004d50 0x7f13d40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:29.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13ebfff640 1 -- 172.21.15.83:0/1153486284 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d4003530 con 0x7f13d4004130 2024-09-19T07:32:29.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.327+0000 7f13f0aab640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec0abe80 0x7f13d4003b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:32:29.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13ebfff640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13d4004130 0x7f13d4004810 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f13e000c320 tx=0x7f13e001cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5723361834e05ec server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:29.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e0007df0 con 0x7f13d4004130 2024-09-19T07:32:29.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13e00a9070 con 0x7f13d4004130 2024-09-19T07:32:29.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13eb7fe640 1 -- 172.21.15.83:0/1153486284 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e0005b00 con 0x7f13d4004130 2024-09-19T07:32:29.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13f2d0f640 1 -- 172.21.15.83:0/1153486284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13d4004130 msgr2=0x7f13d4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:29.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13f2d0f640 1 --2- 172.21.15.83:0/1153486284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13d4004130 0x7f13d4004810 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f13e000c320 tx=0x7f13e001cb40 comp rx=0 tx=0).stop 2024-09-19T07:32:29.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13f12ac640 1 -- 172.21.15.83:0/1153486284 reap_dead start 2024-09-19T07:32:29.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13f2d0f640 1 -- 172.21.15.83:0/1153486284 shutdown_connections 2024-09-19T07:32:29.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13f2d0f640 1 -- 172.21.15.83:0/1153486284 >> 172.21.15.83:0/1153486284 conn(0x7f13ec0aa910 msgr2=0x7f13ec0c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:29.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13f2d0f640 1 -- 172.21.15.83:0/1153486284 shutdown_connections 2024-09-19T07:32:29.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13f2d0f640 1 -- 172.21.15.83:0/1153486284 wait complete. 2024-09-19T07:32:29.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13f2d0f640 1 Processor -- start 2024-09-19T07:32:29.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.331+0000 7f13f2d0f640 1 -- start start 2024-09-19T07:32:29.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f2d0f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13d4004130 0x7f13ec15c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:29.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f2d0f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec156460 0x7f13ec072cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:29.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f2d0f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ec1ea6a0 0x7f13ec14c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:29.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f2d0f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec06d450 con 0x7f13d4004130 2024-09-19T07:32:29.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f2d0f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec06d2d0 con 0x7f13ec1ea6a0 2024-09-19T07:32:29.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f2d0f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f13ec15d600 con 0x7f13ec156460 2024-09-19T07:32:29.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f0aab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13d4004130 0x7f13ec15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f0aab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13d4004130 0x7f13ec15c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41152/0 (socket says 172.21.15.83:41152) 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f0aab640 1 -- 172.21.15.83:0/3631173853 learned_addr learned my addr 172.21.15.83:0/3631173853 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f12ac640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ec1ea6a0 0x7f13ec14c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f0aab640 1 -- 172.21.15.83:0/3631173853 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec156460 msgr2=0x7f13ec072cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f0aab640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec156460 0x7f13ec072cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f0aab640 1 -- 172.21.15.83:0/3631173853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ec1ea6a0 msgr2=0x7f13ec14c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f0aab640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ec1ea6a0 0x7f13ec14c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f0aab640 1 -- 172.21.15.83:0/3631173853 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec14cdc0 con 0x7f13d4004130 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f0aab640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13d4004130 0x7f13ec15c030 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00e530 tx=0x7f13dc00e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13e97fa640 1 -- 172.21.15.83:0/3631173853 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc0052e0 con 0x7f13d4004130 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13e97fa640 1 -- 172.21.15.83:0/3631173853 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13dc005480 con 0x7f13d4004130 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f2d0f640 1 -- 172.21.15.83:0/3631173853 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ec1556c0 con 0x7f13d4004130 2024-09-19T07:32:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13e97fa640 1 -- 172.21.15.83:0/3631173853 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc007020 con 0x7f13d4004130 2024-09-19T07:32:29.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f2d0f640 1 -- 172.21.15.83:0/3631173853 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ec15bb10 con 0x7f13d4004130 2024-09-19T07:32:29.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13f2d0f640 1 -- 172.21.15.83:0/3631173853 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13ec15bce0 con 0x7f13d4004130 2024-09-19T07:32:29.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13e97fa640 1 -- 172.21.15.83:0/3631173853 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f13dc01a020 con 0x7f13d4004130 2024-09-19T07:32:29.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13e97fa640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13d80678f0 0x7f13d8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:29.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13ebfff640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13d80678f0 0x7f13d8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:29.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13e97fa640 1 -- 172.21.15.83:0/3631173853 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7f13dc085da0 con 0x7f13d4004130 2024-09-19T07:32:29.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.335+0000 7f13ebfff640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13d80678f0 0x7f13d8069db0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f13e00ab000 tx=0x7f13e0001b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:29.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.339+0000 7f13e97fa640 1 -- 172.21.15.83:0/3631173853 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f13dc055ee0 con 0x7f13d4004130 2024-09-19T07:32:29.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.547+0000 7f13f2d0f640 1 -- 172.21.15.83:0/3631173853 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13ec0a9e50 con 0x7f13d4004130 2024-09-19T07:32:29.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.547+0000 7f13e97fa640 1 -- 172.21.15.83:0/3631173853 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+6541 (secure 0 0 0) 0x7f13dc0557e0 con 0x7f13d4004130 2024-09-19T07:32:29.542 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:29.542 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":100,"btime":"2024-09-19T07:31:25:264553+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6561,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:29.542 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 100 2024-09-19T07:32:29.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.551+0000 7f13f2d0f640 1 -- 172.21.15.83:0/3631173853 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13d80678f0 msgr2=0x7f13d8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:29.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.551+0000 7f13f2d0f640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13d80678f0 0x7f13d8069db0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f13e00ab000 tx=0x7f13e0001b50 comp rx=0 tx=0).stop 2024-09-19T07:32:29.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.551+0000 7f13f2d0f640 1 -- 172.21.15.83:0/3631173853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13d4004130 msgr2=0x7f13ec15c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:29.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.551+0000 7f13f2d0f640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13d4004130 0x7f13ec15c030 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00e530 tx=0x7f13dc00e9f0 comp rx=0 tx=0).stop 2024-09-19T07:32:29.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.551+0000 7f13f2d0f640 1 -- 172.21.15.83:0/3631173853 shutdown_connections 2024-09-19T07:32:29.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.551+0000 7f13f2d0f640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13d80678f0 0x7f13d8069db0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:29.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.551+0000 7f13f2d0f640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ec1ea6a0 0x7f13ec14c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:29.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.551+0000 7f13f2d0f640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ec156460 0x7f13ec072cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:29.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.551+0000 7f13f2d0f640 1 --2- 172.21.15.83:0/3631173853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13d4004130 0x7f13ec15c030 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:29.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.551+0000 7f13f2d0f640 1 -- 172.21.15.83:0/3631173853 >> 172.21.15.83:0/3631173853 conn(0x7f13ec0aa910 msgr2=0x7f13ec156b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:29.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.551+0000 7f13f2d0f640 1 -- 172.21.15.83:0/3631173853 shutdown_connections 2024-09-19T07:32:29.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:29.551+0000 7f13f2d0f640 1 -- 172.21.15.83:0/3631173853 wait complete. 2024-09-19T07:32:29.560 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:32:31.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.699+0000 7f75d0000640 1 Processor -- start 2024-09-19T07:32:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.699+0000 7f75d0000640 1 -- start start 2024-09-19T07:32:31.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75d0000640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75c80c4b00 0x7f75c80728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:31.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75d0000640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f75c8150210 0x7f75c80c3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:31.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75d0000640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75c8152cd0 con 0x7f75c806cef0 2024-09-19T07:32:31.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75d0000640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f75c8152fb0 con 0x7f75c8150210 2024-09-19T07:32:31.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75d0000640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f75c8153290 con 0x7f75c80c4b00 2024-09-19T07:32:31.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75ce59d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f75c8150210 0x7f75c80c3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:31.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75ce59d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f75c8150210 0x7f75c80c3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58290/0 (socket says 172.21.15.83:58290) 2024-09-19T07:32:31.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75ce59d640 1 -- 172.21.15.83:0/3178686439 learned_addr learned my addr 172.21.15.83:0/3178686439 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:31.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75cd59b640 1 --2- 172.21.15.83:0/3178686439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75c80c4b00 0x7f75c80728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:31.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75ce59d640 1 -- 172.21.15.83:0/3178686439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75c80c4b00 msgr2=0x7f75c80728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:31.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75ce59d640 1 --2- 172.21.15.83:0/3178686439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75c80c4b00 0x7f75c80728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:31.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75ce59d640 1 -- 172.21.15.83:0/3178686439 >> v1:172.21.15.184:6789/0 conn(0x7f75c806cef0 legacy=0x7f75c814fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:31.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75ce59d640 1 -- 172.21.15.83:0/3178686439 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c8153570 con 0x7f75c8150210 2024-09-19T07:32:31.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.703+0000 7f75ce59d640 1 --2- 172.21.15.83:0/3178686439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75c8150210 0x7f75c80c3370 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f75c4009870 tx=0x7f75c401e6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b089a23f75ec0da server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:31.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75ccd9a640 1 -- 172.21.15.83:0/3178686439 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c40ad5f0 con 0x7f75c8150210 2024-09-19T07:32:31.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75ccd9a640 1 -- 172.21.15.83:0/3178686439 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c40b2070 con 0x7f75c8150210 2024-09-19T07:32:31.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75ccd9a640 1 -- 172.21.15.83:0/3178686439 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c40ad9b0 con 0x7f75c8150210 2024-09-19T07:32:31.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 -- 172.21.15.83:0/3178686439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75c8150210 msgr2=0x7f75c80c3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:31.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 --2- 172.21.15.83:0/3178686439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75c8150210 0x7f75c80c3370 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f75c4009870 tx=0x7f75c401e6e0 comp rx=0 tx=0).stop 2024-09-19T07:32:31.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 -- 172.21.15.83:0/3178686439 shutdown_connections 2024-09-19T07:32:31.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 --2- 172.21.15.83:0/3178686439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75c8150210 0x7f75c80c3370 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:31.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 --2- 172.21.15.83:0/3178686439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75c80c4b00 0x7f75c80728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:31.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 -- 172.21.15.83:0/3178686439 >> 172.21.15.83:0/3178686439 conn(0x7f75c80c5090 msgr2=0x7f75c80c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:31.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 -- 172.21.15.83:0/3178686439 shutdown_connections 2024-09-19T07:32:31.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 -- 172.21.15.83:0/3178686439 wait complete. 2024-09-19T07:32:31.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 Processor -- start 2024-09-19T07:32:31.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 -- start start 2024-09-19T07:32:31.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75c806cef0 0x7f75c814c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:31.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75c80c4b00 0x7f75c806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75c8150210 0x7f75c8155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f75c806d4e0 con 0x7f75c80c4b00 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f75c80c3ed0 con 0x7f75c8150210 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75d0000640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f75c80757a0 con 0x7f75c806cef0 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75cd59b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75c80c4b00 0x7f75c806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75cd59b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75c80c4b00 0x7f75c806ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58296/0 (socket says 172.21.15.83:58296) 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75cd59b640 1 -- 172.21.15.83:0/1734936328 learned_addr learned my addr 172.21.15.83:0/1734936328 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75ce59d640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75c8150210 0x7f75c8155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75cdd9c640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75c806cef0 0x7f75c814c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75cd59b640 1 -- 172.21.15.83:0/1734936328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75c806cef0 msgr2=0x7f75c814c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75cd59b640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75c806cef0 0x7f75c814c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75cd59b640 1 -- 172.21.15.83:0/1734936328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75c8150210 msgr2=0x7f75c8155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75cd59b640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75c8150210 0x7f75c8155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75cd59b640 1 -- 172.21.15.83:0/1734936328 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c81ede40 con 0x7f75c80c4b00 2024-09-19T07:32:31.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75cdd9c640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75c806cef0 0x7f75c814c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:32:31.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.707+0000 7f75cd59b640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75c80c4b00 0x7f75c806ed40 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f75b800d800 tx=0x7f75b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:31.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.711+0000 7f75be7fc640 1 -- 172.21.15.83:0/1734936328 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b800b6c0 con 0x7f75c80c4b00 2024-09-19T07:32:31.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.711+0000 7f75be7fc640 1 -- 172.21.15.83:0/1734936328 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75b8019070 con 0x7f75c80c4b00 2024-09-19T07:32:31.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.711+0000 7f75d0000640 1 -- 172.21.15.83:0/1734936328 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c81ee120 con 0x7f75c80c4b00 2024-09-19T07:32:31.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.711+0000 7f75be7fc640 1 -- 172.21.15.83:0/1734936328 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b800ba60 con 0x7f75c80c4b00 2024-09-19T07:32:31.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.711+0000 7f75d0000640 1 -- 172.21.15.83:0/1734936328 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c80c74b0 con 0x7f75c80c4b00 2024-09-19T07:32:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.711+0000 7f75d0000640 1 -- 172.21.15.83:0/1734936328 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c8075b60 con 0x7f75c80c4b00 2024-09-19T07:32:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.711+0000 7f75be7fc640 1 -- 172.21.15.83:0/1734936328 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f75b800bc00 con 0x7f75c80c4b00 2024-09-19T07:32:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.711+0000 7f75be7fc640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f75ac0678c0 0x7f75ac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.711+0000 7f75be7fc640 1 -- 172.21.15.83:0/1734936328 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7f75b8086b80 con 0x7f75c80c4b00 2024-09-19T07:32:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.711+0000 7f75cdd9c640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f75ac0678c0 0x7f75ac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.711+0000 7f75cdd9c640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f75ac0678c0 0x7f75ac069d80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f75c0007fd0 tx=0x7f75c0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:31.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.715+0000 7f75be7fc640 1 -- 172.21.15.83:0/1734936328 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f75b8056cc0 con 0x7f75c80c4b00 2024-09-19T07:32:31.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.915+0000 7f75d0000640 1 -- 172.21.15.83:0/1734936328 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75c8068900 con 0x7f75c80c4b00 2024-09-19T07:32:31.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.915+0000 7f75be7fc640 1 -- 172.21.15.83:0/1734936328 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+6541 (secure 0 0 0) 0x7f75b80565c0 con 0x7f75c80c4b00 2024-09-19T07:32:31.909 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:31.909 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":100,"btime":"2024-09-19T07:31:25:264553+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6561,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:31.909 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 100 2024-09-19T07:32:31.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.919+0000 7f75d0000640 1 -- 172.21.15.83:0/1734936328 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f75ac0678c0 msgr2=0x7f75ac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:31.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.919+0000 7f75d0000640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f75ac0678c0 0x7f75ac069d80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f75c0007fd0 tx=0x7f75c0009040 comp rx=0 tx=0).stop 2024-09-19T07:32:31.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.919+0000 7f75d0000640 1 -- 172.21.15.83:0/1734936328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75c80c4b00 msgr2=0x7f75c806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:31.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.919+0000 7f75d0000640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75c80c4b00 0x7f75c806ed40 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f75b800d800 tx=0x7f75b800dcc0 comp rx=0 tx=0).stop 2024-09-19T07:32:31.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.919+0000 7f75d0000640 1 -- 172.21.15.83:0/1734936328 shutdown_connections 2024-09-19T07:32:31.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.919+0000 7f75d0000640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f75ac0678c0 0x7f75ac069d80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:31.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.919+0000 7f75d0000640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75c8150210 0x7f75c8155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:31.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.919+0000 7f75d0000640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75c80c4b00 0x7f75c806ed40 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:31.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.919+0000 7f75d0000640 1 --2- 172.21.15.83:0/1734936328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75c806cef0 0x7f75c814c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:31.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.919+0000 7f75d0000640 1 -- 172.21.15.83:0/1734936328 >> 172.21.15.83:0/1734936328 conn(0x7f75c80c5090 msgr2=0x7f75c814fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:31.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.919+0000 7f75d0000640 1 -- 172.21.15.83:0/1734936328 shutdown_connections 2024-09-19T07:32:31.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:31.919+0000 7f75d0000640 1 -- 172.21.15.83:0/1734936328 wait complete. 2024-09-19T07:32:31.929 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:32:33.304 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:32:33.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:32:33.930 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:34.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.071+0000 7fe92cb05640 1 Processor -- start 2024-09-19T07:32:34.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.071+0000 7fe92cb05640 1 -- start start 2024-09-19T07:32:34.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9280adae0 0x7fe92806f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:34.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe92806fcc0 0x7fe9281539a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:34.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9280c3aa0 con 0x7fe928150f40 2024-09-19T07:32:34.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe9280c3850 con 0x7fe92806fcc0 2024-09-19T07:32:34.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe9280c43d0 con 0x7fe9280adae0 2024-09-19T07:32:34.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe926d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe92806fcc0 0x7fe9281539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:34.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe926d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe92806fcc0 0x7fe9281539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58304/0 (socket says 172.21.15.83:58304) 2024-09-19T07:32:34.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe926d9d640 1 -- 172.21.15.83:0/125314313 learned_addr learned my addr 172.21.15.83:0/125314313 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:34.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe926d9d640 1 -- 172.21.15.83:0/125314313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9280adae0 msgr2=0x7fe92806f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:34.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe926d9d640 1 --2- 172.21.15.83:0/125314313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9280adae0 0x7fe92806f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:34.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe926d9d640 1 -- 172.21.15.83:0/125314313 >> v1:172.21.15.184:6789/0 conn(0x7fe928150f40 legacy=0x7fe9280733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:34.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe926d9d640 1 -- 172.21.15.83:0/125314313 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9280c3d80 con 0x7fe92806fcc0 2024-09-19T07:32:34.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe926d9d640 1 --2- 172.21.15.83:0/125314313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe92806fcc0 0x7fe9281539a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe920009e50 tx=0x7fe92001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155b3617590c00da server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92559a640 1 -- 172.21.15.83:0/125314313 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9200a5590 con 0x7fe92806fcc0 2024-09-19T07:32:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92559a640 1 -- 172.21.15.83:0/125314313 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9200a5730 con 0x7fe92806fcc0 2024-09-19T07:32:34.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92559a640 1 -- 172.21.15.83:0/125314313 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9200a5b00 con 0x7fe92806fcc0 2024-09-19T07:32:34.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 -- 172.21.15.83:0/125314313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe92806fcc0 msgr2=0x7fe9281539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:34.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 --2- 172.21.15.83:0/125314313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe92806fcc0 0x7fe9281539a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe920009e50 tx=0x7fe92001c9f0 comp rx=0 tx=0).stop 2024-09-19T07:32:34.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 -- 172.21.15.83:0/125314313 shutdown_connections 2024-09-19T07:32:34.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 --2- 172.21.15.83:0/125314313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe92806fcc0 0x7fe9281539a0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:34.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 --2- 172.21.15.83:0/125314313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9280adae0 0x7fe92806f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:34.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 -- 172.21.15.83:0/125314313 >> 172.21.15.83:0/125314313 conn(0x7fe9280c4940 msgr2=0x7fe9280c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:34.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 -- 172.21.15.83:0/125314313 shutdown_connections 2024-09-19T07:32:34.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.075+0000 7fe92cb05640 1 -- 172.21.15.83:0/125314313 wait complete. 2024-09-19T07:32:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92cb05640 1 Processor -- start 2024-09-19T07:32:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92cb05640 1 -- start start 2024-09-19T07:32:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92cb05640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe92806fcc0 0x7fe92814d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92cb05640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9280adae0 0x7fe928147360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92cb05640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe928150f40 0x7fe9281478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92cb05640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9281606c0 con 0x7fe92806fcc0 2024-09-19T07:32:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92cb05640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe928160540 con 0x7fe9280adae0 2024-09-19T07:32:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92cb05640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe928160840 con 0x7fe928150f40 2024-09-19T07:32:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe92806fcc0 0x7fe92814d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe92806fcc0 0x7fe92814d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58306/0 (socket says 172.21.15.83:58306) 2024-09-19T07:32:34.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92659c640 1 -- 172.21.15.83:0/106442909 learned_addr learned my addr 172.21.15.83:0/106442909 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:34.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe926d9d640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe928150f40 0x7fe9281478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:34.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe925d9b640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9280adae0 0x7fe928147360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:34.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92659c640 1 -- 172.21.15.83:0/106442909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe928150f40 msgr2=0x7fe9281478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:34.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92659c640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe928150f40 0x7fe9281478a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:34.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92659c640 1 -- 172.21.15.83:0/106442909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9280adae0 msgr2=0x7fe928147360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:34.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92659c640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9280adae0 0x7fe928147360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:34.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92659c640 1 -- 172.21.15.83:0/106442909 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe928146ba0 con 0x7fe92806fcc0 2024-09-19T07:32:34.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92659c640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe92806fcc0 0x7fe92814d220 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe91800c4c0 tx=0x7fe918001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:34.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe916ffd640 1 -- 172.21.15.83:0/106442909 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe918017070 con 0x7fe92806fcc0 2024-09-19T07:32:34.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe916ffd640 1 -- 172.21.15.83:0/106442909 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe918003600 con 0x7fe92806fcc0 2024-09-19T07:32:34.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe916ffd640 1 -- 172.21.15.83:0/106442909 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9180039f0 con 0x7fe92806fcc0 2024-09-19T07:32:34.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92cb05640 1 -- 172.21.15.83:0/106442909 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9281483e0 con 0x7fe92806fcc0 2024-09-19T07:32:34.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92cb05640 1 -- 172.21.15.83:0/106442909 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe92814d760 con 0x7fe92806fcc0 2024-09-19T07:32:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.079+0000 7fe92cb05640 1 -- 172.21.15.83:0/106442909 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9281490c0 con 0x7fe92806fcc0 2024-09-19T07:32:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.083+0000 7fe916ffd640 1 -- 172.21.15.83:0/106442909 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe918003b90 con 0x7fe92806fcc0 2024-09-19T07:32:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.083+0000 7fe916ffd640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe9080678f0 0x7fe908069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.083+0000 7fe916ffd640 1 -- 172.21.15.83:0/106442909 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7fe918085760 con 0x7fe92806fcc0 2024-09-19T07:32:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.083+0000 7fe925d9b640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe9080678f0 0x7fe908069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:34.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.083+0000 7fe925d9b640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe9080678f0 0x7fe908069db0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe92814b5b0 tx=0x7fe91c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:34.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.087+0000 7fe916ffd640 1 -- 172.21.15.83:0/106442909 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe918055930 con 0x7fe92806fcc0 2024-09-19T07:32:34.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.283+0000 7fe92cb05640 1 -- 172.21.15.83:0/106442909 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe928153980 con 0x7fe92806fcc0 2024-09-19T07:32:34.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe916ffd640 1 -- 172.21.15.83:0/106442909 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+6541 (secure 0 0 0) 0x7fe918055230 con 0x7fe92806fcc0 2024-09-19T07:32:34.280 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:34.280 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":100,"btime":"2024-09-19T07:31:25:264553+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6561,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:34.281 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 100 2024-09-19T07:32:34.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe92cb05640 1 -- 172.21.15.83:0/106442909 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe9080678f0 msgr2=0x7fe908069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:34.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe92cb05640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe9080678f0 0x7fe908069db0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe92814b5b0 tx=0x7fe91c008040 comp rx=0 tx=0).stop 2024-09-19T07:32:34.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe92cb05640 1 -- 172.21.15.83:0/106442909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe92806fcc0 msgr2=0x7fe92814d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:34.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe92cb05640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe92806fcc0 0x7fe92814d220 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe91800c4c0 tx=0x7fe918001e80 comp rx=0 tx=0).stop 2024-09-19T07:32:34.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe92cb05640 1 -- 172.21.15.83:0/106442909 shutdown_connections 2024-09-19T07:32:34.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe92cb05640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe9080678f0 0x7fe908069db0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:34.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe92cb05640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe928150f40 0x7fe9281478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:34.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe92cb05640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9280adae0 0x7fe928147360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:34.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe92cb05640 1 --2- 172.21.15.83:0/106442909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe92806fcc0 0x7fe92814d220 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:34.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe92cb05640 1 -- 172.21.15.83:0/106442909 >> 172.21.15.83:0/106442909 conn(0x7fe9280c4940 msgr2=0x7fe9280715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:34.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe92cb05640 1 -- 172.21.15.83:0/106442909 shutdown_connections 2024-09-19T07:32:34.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:34.287+0000 7fe92cb05640 1 -- 172.21.15.83:0/106442909 wait complete. 2024-09-19T07:32:34.298 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:32:36.299 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc791af0640 1 Processor -- start 2024-09-19T07:32:36.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc791af0640 1 -- start start 2024-09-19T07:32:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc791af0640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc78c0abe80 0x7fc78c157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc791af0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc78c0c76d0 con 0x7fc78c0ab610 2024-09-19T07:32:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc791af0640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc78c0ac2b0 con 0x7fc78c06cef0 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc791af0640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc78c0c6a90 con 0x7fc78c0abe80 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc78c0ab610 0x7fc78c0c6fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36006/0 (socket says 172.21.15.83:36006) 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78affd640 1 -- 172.21.15.83:0/246071075 learned_addr learned my addr 172.21.15.83:0/246071075 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78bfff640 1 --2- 172.21.15.83:0/246071075 >> v2:172.21.15.184:3300/0 conn(0x7fc78c0abe80 0x7fc78c157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 357552464 0 0) 0x7fc780001120 con 0x7fc78c0ab610 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc774003530 con 0x7fc78c0ab610 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2086156891 0 0) 0x7fc774003530 con 0x7fc78c0ab610 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 >> v2:172.21.15.184:3300/0 conn(0x7fc78c0abe80 msgr2=0x7fc78c157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 --2- 172.21.15.83:0/246071075 >> v2:172.21.15.184:3300/0 conn(0x7fc78c0abe80 0x7fc78c157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 >> v1:172.21.15.184:6789/0 conn(0x7fc78c06cef0 legacy=0x7fc78c0c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc78c0c7de0 con 0x7fc78c0ab610 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fc780001850 con 0x7fc78c0ab610 2024-09-19T07:32:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 >> v1:172.21.15.83:6789/0 conn(0x7fc78c0ab610 legacy=0x7fc78c0c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 --2- 172.21.15.83:0/246071075 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7740039c0 0x7fc774003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 --2- 172.21.15.83:0/246071075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc774004810 0x7fc774004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 --2- 172.21.15.83:0/246071075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7740052f0 0x7fc77400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc77400c190 con 0x7fc774004810 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc77400c340 con 0x7fc7740052f0 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc77400c540 con 0x7fc7740039c0 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78affd640 1 --2- 172.21.15.83:0/246071075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc774004810 0x7fc774004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78b7fe640 1 --2- 172.21.15.83:0/246071075 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7740039c0 0x7fc774003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78bfff640 1 --2- 172.21.15.83:0/246071075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7740052f0 0x7fc77400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78affd640 1 -- 172.21.15.83:0/246071075 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7740039c0 msgr2=0x7fc774003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78affd640 1 --2- 172.21.15.83:0/246071075 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7740039c0 0x7fc774003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78affd640 1 -- 172.21.15.83:0/246071075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7740052f0 msgr2=0x7fc77400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78affd640 1 --2- 172.21.15.83:0/246071075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7740052f0 0x7fc77400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78affd640 1 -- 172.21.15.83:0/246071075 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc774003530 con 0x7fc774004810 2024-09-19T07:32:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78bfff640 1 --2- 172.21.15.83:0/246071075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7740052f0 0x7fc77400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:32:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.447+0000 7fc78bfff640 1 -- 172.21.15.83:0/246071075 reap_dead start 2024-09-19T07:32:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc78affd640 1 --2- 172.21.15.83:0/246071075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc774004810 0x7fc774004c90 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc78000c4c0 tx=0x7fc78001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9698680185832f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78000c0e0 con 0x7fc774004810 2024-09-19T07:32:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7800a9070 con 0x7fc774004810 2024-09-19T07:32:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc78a7fc640 1 -- 172.21.15.83:0/246071075 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc780005d80 con 0x7fc774004810 2024-09-19T07:32:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 -- 172.21.15.83:0/246071075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc774004810 msgr2=0x7fc774004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 --2- 172.21.15.83:0/246071075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc774004810 0x7fc774004c90 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc78000c4c0 tx=0x7fc78001cc60 comp rx=0 tx=0).stop 2024-09-19T07:32:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 -- 172.21.15.83:0/246071075 shutdown_connections 2024-09-19T07:32:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 --2- 172.21.15.83:0/246071075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc774004810 0x7fc774004c90 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 -- 172.21.15.83:0/246071075 >> 172.21.15.83:0/246071075 conn(0x7fc78c0aa910 msgr2=0x7fc78c0c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 -- 172.21.15.83:0/246071075 shutdown_connections 2024-09-19T07:32:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 -- 172.21.15.83:0/246071075 wait complete. 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 Processor -- start 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 -- start start 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc774004810 0x7fc78c1f0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc78c14e3c0 0x7fc78c1ec740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c1ecc80 0x7fc78c1df500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c0c7550 con 0x7fc78c1ecc80 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c0c73d0 con 0x7fc78c14e3c0 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc791af0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc78c06d2d0 con 0x7fc774004810 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc78bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c1ecc80 0x7fc78c1df500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc78bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c1ecc80 0x7fc78c1df500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58312/0 (socket says 172.21.15.83:58312) 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc78bfff640 1 -- 172.21.15.83:0/231510102 learned_addr learned my addr 172.21.15.83:0/231510102 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc78affd640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc78c14e3c0 0x7fc78c1ec740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.451+0000 7fc78b7fe640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc774004810 0x7fc78c1f0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc78bfff640 1 -- 172.21.15.83:0/231510102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc774004810 msgr2=0x7fc78c1f0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc78bfff640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc774004810 0x7fc78c1f0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc78bfff640 1 -- 172.21.15.83:0/231510102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc78c14e3c0 msgr2=0x7fc78c1ec740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc78bfff640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc78c14e3c0 0x7fc78c1ec740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc78bfff640 1 -- 172.21.15.83:0/231510102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc78c14e910 con 0x7fc78c1ecc80 2024-09-19T07:32:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc78bfff640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c1ecc80 0x7fc78c1df500 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc78c06fe00 tx=0x7fc784010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc773fff640 1 -- 172.21.15.83:0/231510102 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78400f040 con 0x7fc78c1ecc80 2024-09-19T07:32:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc773fff640 1 -- 172.21.15.83:0/231510102 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc78400e7c0 con 0x7fc78c1ecc80 2024-09-19T07:32:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc791af0640 1 -- 172.21.15.83:0/231510102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc78c1efd50 con 0x7fc78c1ecc80 2024-09-19T07:32:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc773fff640 1 -- 172.21.15.83:0/231510102 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78400eb70 con 0x7fc78c1ecc80 2024-09-19T07:32:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc791af0640 1 -- 172.21.15.83:0/231510102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc78c1f0310 con 0x7fc78c1ecc80 2024-09-19T07:32:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc791af0640 1 -- 172.21.15.83:0/231510102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc78c157890 con 0x7fc78c1ecc80 2024-09-19T07:32:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc773fff640 1 -- 172.21.15.83:0/231510102 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc78400ed10 con 0x7fc78c1ecc80 2024-09-19T07:32:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc773fff640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc7780678c0 0x7fc778069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc773fff640 1 -- 172.21.15.83:0/231510102 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8185+0+0 (secure 0 0 0) 0x7fc784086920 con 0x7fc78c1ecc80 2024-09-19T07:32:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc78b7fe640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc7780678c0 0x7fc778069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.455+0000 7fc78b7fe640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc7780678c0 0x7fc778069d80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc77c001de0 tx=0x7fc77c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.459+0000 7fc773fff640 1 -- 172.21.15.83:0/231510102 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc784018030 con 0x7fc78c1ecc80 2024-09-19T07:32:36.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.659+0000 7fc791af0640 1 -- 172.21.15.83:0/231510102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc78c1f04e0 con 0x7fc78c1ecc80 2024-09-19T07:32:36.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.659+0000 7fc773fff640 1 -- 172.21.15.83:0/231510102 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+6541 (secure 0 0 0) 0x7fc7840564a0 con 0x7fc78c1ecc80 2024-09-19T07:32:36.654 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:36.654 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":100,"btime":"2024-09-19T07:31:25:264553+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6561,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:30:51.094244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4652":{"gid":4652,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":129,"addr":"172.21.15.184:6833/2993984999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2993984999},{"type":"v1","addr":"172.21.15.184:6833","nonce":2993984999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4652,"qdb_cluster":[4652]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:36.654 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 100 2024-09-19T07:32:36.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.663+0000 7fc791af0640 1 -- 172.21.15.83:0/231510102 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc7780678c0 msgr2=0x7fc778069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:36.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.663+0000 7fc791af0640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc7780678c0 0x7fc778069d80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc77c001de0 tx=0x7fc77c009040 comp rx=0 tx=0).stop 2024-09-19T07:32:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.663+0000 7fc791af0640 1 -- 172.21.15.83:0/231510102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c1ecc80 msgr2=0x7fc78c1df500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.663+0000 7fc791af0640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c1ecc80 0x7fc78c1df500 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc78c06fe00 tx=0x7fc784010c20 comp rx=0 tx=0).stop 2024-09-19T07:32:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.663+0000 7fc791af0640 1 -- 172.21.15.83:0/231510102 shutdown_connections 2024-09-19T07:32:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.663+0000 7fc791af0640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc7780678c0 0x7fc778069d80 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.663+0000 7fc791af0640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c1ecc80 0x7fc78c1df500 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.663+0000 7fc791af0640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc78c14e3c0 0x7fc78c1ec740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.663+0000 7fc791af0640 1 --2- 172.21.15.83:0/231510102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc774004810 0x7fc78c1f0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.663+0000 7fc791af0640 1 -- 172.21.15.83:0/231510102 >> 172.21.15.83:0/231510102 conn(0x7fc78c0aa910 msgr2=0x7fc78c072750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.663+0000 7fc791af0640 1 -- 172.21.15.83:0/231510102 shutdown_connections 2024-09-19T07:32:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:36.663+0000 7fc791af0640 1 -- 172.21.15.83:0/231510102 wait complete. 2024-09-19T07:32:36.672 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:32:38.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:38.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377f67e640 1 Processor -- start 2024-09-19T07:32:38.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377f67e640 1 -- start start 2024-09-19T07:32:38.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377f67e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f37780c6120 0x7f37780c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:38.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377f67e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f37781592f0 0x7f3778157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:38.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377f67e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3778156200 con 0x7f3778155d70 2024-09-19T07:32:38.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377f67e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f37780c9930 con 0x7f37780c6120 2024-09-19T07:32:38.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377f67e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f37781564e0 con 0x7f37781592f0 2024-09-19T07:32:38.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377d41a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f37780c6120 0x7f37780c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:38.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377cc19640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3778155d70 0x7f37781575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36022/0 (socket says 172.21.15.83:36022) 2024-09-19T07:32:38.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377cc19640 1 -- 172.21.15.83:0/3185578581 learned_addr learned my addr 172.21.15.83:0/3185578581 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:32:38.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377dc1b640 1 --2- 172.21.15.83:0/3185578581 >> v2:172.21.15.184:3300/0 conn(0x7f37781592f0 0x7f3778157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:38.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f3777fff640 1 -- 172.21.15.83:0/3185578581 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2540114261 0 0) 0x7f3778156200 con 0x7f3778155d70 2024-09-19T07:32:38.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f3777fff640 1 -- 172.21.15.83:0/3185578581 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3760003530 con 0x7f3778155d70 2024-09-19T07:32:38.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377d41a640 1 -- 172.21.15.83:0/3185578581 >> v2:172.21.15.184:3300/0 conn(0x7f37781592f0 msgr2=0x7f3778157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:38.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377d41a640 1 --2- 172.21.15.83:0/3185578581 >> v2:172.21.15.184:3300/0 conn(0x7f37781592f0 0x7f3778157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:38.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377d41a640 1 -- 172.21.15.83:0/3185578581 >> v1:172.21.15.83:6789/0 conn(0x7f3778155d70 legacy=0x7f37781575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:38.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.807+0000 7f377d41a640 1 -- 172.21.15.83:0/3185578581 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37781567c0 con 0x7f37780c6120 2024-09-19T07:32:38.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.811+0000 7f377d41a640 1 --2- 172.21.15.83:0/3185578581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780c6120 0x7f37780c6500 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f37780b1990 tx=0x7f376801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a32fad0e94a82c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:38.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.811+0000 7f3777fff640 1 -- 172.21.15.83:0/3185578581 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37680b0070 con 0x7f37780c6120 2024-09-19T07:32:38.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.811+0000 7f3777fff640 1 -- 172.21.15.83:0/3185578581 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37680ab600 con 0x7f37780c6120 2024-09-19T07:32:38.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.811+0000 7f3777fff640 1 -- 172.21.15.83:0/3185578581 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37680ab9d0 con 0x7f37780c6120 2024-09-19T07:32:38.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.811+0000 7f377f67e640 1 -- 172.21.15.83:0/3185578581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780c6120 msgr2=0x7f37780c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:38.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.811+0000 7f377f67e640 1 --2- 172.21.15.83:0/3185578581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780c6120 0x7f37780c6500 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f37780b1990 tx=0x7f376801c480 comp rx=0 tx=0).stop 2024-09-19T07:32:38.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.811+0000 7f377f67e640 1 -- 172.21.15.83:0/3185578581 shutdown_connections 2024-09-19T07:32:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.811+0000 7f377f67e640 1 --2- 172.21.15.83:0/3185578581 >> v2:172.21.15.184:3300/0 conn(0x7f37781592f0 0x7f3778157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.811+0000 7f377f67e640 1 --2- 172.21.15.83:0/3185578581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780c6120 0x7f37780c6500 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.811+0000 7f377f67e640 1 -- 172.21.15.83:0/3185578581 >> 172.21.15.83:0/3185578581 conn(0x7f377806bc80 msgr2=0x7f377806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.811+0000 7f377f67e640 1 -- 172.21.15.83:0/3185578581 shutdown_connections 2024-09-19T07:32:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.811+0000 7f377f67e640 1 -- 172.21.15.83:0/3185578581 wait complete. 2024-09-19T07:32:38.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377f67e640 1 Processor -- start 2024-09-19T07:32:38.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377f67e640 1 -- start start 2024-09-19T07:32:38.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377f67e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37780c6120 0x7f37780c1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:38.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377f67e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37781592f0 0x7f37780c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377f67e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780c0640 0x7f377814fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377f67e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3778153860 con 0x7f37780c0640 2024-09-19T07:32:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377f67e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3778162ea0 con 0x7f37780c6120 2024-09-19T07:32:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377f67e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f377814d1b0 con 0x7f37781592f0 2024-09-19T07:32:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377dc1b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780c0640 0x7f377814fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377dc1b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780c0640 0x7f377814fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58338/0 (socket says 172.21.15.83:58338) 2024-09-19T07:32:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377dc1b640 1 -- 172.21.15.83:0/4072939475 learned_addr learned my addr 172.21.15.83:0/4072939475 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377d41a640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37780c6120 0x7f37780c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377cc19640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37781592f0 0x7f37780c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377dc1b640 1 -- 172.21.15.83:0/4072939475 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37781592f0 msgr2=0x7f37780c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377dc1b640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37781592f0 0x7f37780c9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377dc1b640 1 -- 172.21.15.83:0/4072939475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37780c6120 msgr2=0x7f37780c1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377dc1b640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37780c6120 0x7f37780c1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377dc1b640 1 -- 172.21.15.83:0/4072939475 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f377814ceb0 con 0x7f37780c0640 2024-09-19T07:32:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377d41a640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37780c6120 0x7f37780c1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:32:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377dc1b640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780c0640 0x7f377814fe90 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f377000efc0 tx=0x7f377000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f3775ffb640 1 -- 172.21.15.83:0/4072939475 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770018020 con 0x7f37780c0640 2024-09-19T07:32:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f3775ffb640 1 -- 172.21.15.83:0/4072939475 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3770014480 con 0x7f37780c0640 2024-09-19T07:32:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377f67e640 1 -- 172.21.15.83:0/4072939475 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37780c1660 con 0x7f37780c0640 2024-09-19T07:32:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f3775ffb640 1 -- 172.21.15.83:0/4072939475 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770014870 con 0x7f37780c0640 2024-09-19T07:32:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377f67e640 1 -- 172.21.15.83:0/4072939475 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37780ca4b0 con 0x7f37780c0640 2024-09-19T07:32:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377f67e640 1 -- 172.21.15.83:0/4072939475 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37780c6500 con 0x7f37780c0640 2024-09-19T07:32:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f3775ffb640 1 -- 172.21.15.83:0/4072939475 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f377001b020 con 0x7f37780c0640 2024-09-19T07:32:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f3775ffb640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37640678c0 0x7f3764069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f3775ffb640 1 -- 172.21.15.83:0/4072939475 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7f37700871d0 con 0x7f37780c0640 2024-09-19T07:32:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.815+0000 7f377d41a640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37640678c0 0x7f3764069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.819+0000 7f377d41a640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37640678c0 0x7f3764069d80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f37680ad000 tx=0x7f37680a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:38.823+0000 7f3775ffb640 1 -- 172.21.15.83:0/4072939475 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3770022070 con 0x7f37780c0640 2024-09-19T07:32:39.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.023+0000 7f377f67e640 1 -- 172.21.15.83:0/4072939475 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37780c96a0 con 0x7f37780c0640 2024-09-19T07:32:39.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.023+0000 7f3775ffb640 1 -- 172.21.15.83:0/4072939475 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+5705 (secure 0 0 0) 0x7f3770010040 con 0x7f37780c0640 2024-09-19T07:32:39.018 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:39.018 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":101,"btime":"2024-09-19T07:32:38:105469+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:38.105460+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:39.018 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 101 2024-09-19T07:32:39.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.027+0000 7f377f67e640 1 -- 172.21.15.83:0/4072939475 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37640678c0 msgr2=0x7f3764069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:39.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.027+0000 7f377f67e640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37640678c0 0x7f3764069d80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f37680ad000 tx=0x7f37680a6000 comp rx=0 tx=0).stop 2024-09-19T07:32:39.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.027+0000 7f377f67e640 1 -- 172.21.15.83:0/4072939475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780c0640 msgr2=0x7f377814fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:39.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.027+0000 7f377f67e640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780c0640 0x7f377814fe90 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f377000efc0 tx=0x7f377000c490 comp rx=0 tx=0).stop 2024-09-19T07:32:39.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.027+0000 7f377f67e640 1 -- 172.21.15.83:0/4072939475 shutdown_connections 2024-09-19T07:32:39.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.027+0000 7f377f67e640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37640678c0 0x7f3764069d80 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.027+0000 7f377f67e640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37780c0640 0x7f377814fe90 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.027+0000 7f377f67e640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37781592f0 0x7f37780c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.027+0000 7f377f67e640 1 --2- 172.21.15.83:0/4072939475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37780c6120 0x7f37780c1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.027+0000 7f377f67e640 1 -- 172.21.15.83:0/4072939475 >> 172.21.15.83:0/4072939475 conn(0x7f377806bc80 msgr2=0x7f37781575d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:39.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.027+0000 7f377f67e640 1 -- 172.21.15.83:0/4072939475 shutdown_connections 2024-09-19T07:32:39.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.027+0000 7f377f67e640 1 -- 172.21.15.83:0/4072939475 wait complete. 2024-09-19T07:32:39.037 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-19T07:32:39.038 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:32:39.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.159+0000 7f2228a90640 1 Processor -- start 2024-09-19T07:32:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.159+0000 7f2228a90640 1 -- start start 2024-09-19T07:32:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2228a90640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f22240c42e0 0x7f222406e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2228a90640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22241566b0 0x7f22240abb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2228a90640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22241592c0 con 0x7f2224159a80 2024-09-19T07:32:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2228a90640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f22240ac070 con 0x7f22240c42e0 2024-09-19T07:32:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2228a90640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f22241595f0 con 0x7f22241566b0 2024-09-19T07:32:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2221d9b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22241566b0 0x7f22240abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2222d9d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f2224159a80 0x7f2224153060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:50210/0 (socket says 172.21.15.83:50210) 2024-09-19T07:32:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2222d9d640 1 -- 172.21.15.83:0/1899134704 learned_addr learned my addr 172.21.15.83:0/1899134704 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:32:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f222259c640 1 --2- 172.21.15.83:0/1899134704 >> v2:172.21.15.184:3300/0 conn(0x7f22240c42e0 0x7f222406e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f222159a640 1 -- 172.21.15.83:0/1899134704 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 891080756 0 0) 0x7f22241592c0 con 0x7f2224159a80 2024-09-19T07:32:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f222159a640 1 -- 172.21.15.83:0/1899134704 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2210003530 con 0x7f2224159a80 2024-09-19T07:32:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2221d9b640 1 -- 172.21.15.83:0/1899134704 >> v2:172.21.15.184:3300/0 conn(0x7f22240c42e0 msgr2=0x7f222406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2221d9b640 1 --2- 172.21.15.83:0/1899134704 >> v2:172.21.15.184:3300/0 conn(0x7f22240c42e0 0x7f222406e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2221d9b640 1 -- 172.21.15.83:0/1899134704 >> v1:172.21.15.184:6789/0 conn(0x7f2224159a80 legacy=0x7f2224153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:39.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2221d9b640 1 -- 172.21.15.83:0/1899134704 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222406e900 con 0x7f22241566b0 2024-09-19T07:32:39.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.163+0000 7f2221d9b640 1 --2- 172.21.15.83:0/1899134704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22241566b0 0x7f22240abb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2214009d90 tx=0x7f221401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba925eeed4e91b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:39.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f222159a640 1 -- 172.21.15.83:0/1899134704 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22140b2070 con 0x7f22241566b0 2024-09-19T07:32:39.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f222159a640 1 -- 172.21.15.83:0/1899134704 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22140a5a20 con 0x7f22241566b0 2024-09-19T07:32:39.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f222159a640 1 -- 172.21.15.83:0/1899134704 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22140a5dd0 con 0x7f22241566b0 2024-09-19T07:32:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 -- 172.21.15.83:0/1899134704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22241566b0 msgr2=0x7f22240abb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 --2- 172.21.15.83:0/1899134704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22241566b0 0x7f22240abb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2214009d90 tx=0x7f221401c940 comp rx=0 tx=0).stop 2024-09-19T07:32:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 -- 172.21.15.83:0/1899134704 shutdown_connections 2024-09-19T07:32:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 --2- 172.21.15.83:0/1899134704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22241566b0 0x7f22240abb30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 --2- 172.21.15.83:0/1899134704 >> v2:172.21.15.184:3300/0 conn(0x7f22240c42e0 0x7f222406e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 -- 172.21.15.83:0/1899134704 >> 172.21.15.83:0/1899134704 conn(0x7f222406bc80 msgr2=0x7f222406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 -- 172.21.15.83:0/1899134704 shutdown_connections 2024-09-19T07:32:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 -- 172.21.15.83:0/1899134704 wait complete. 2024-09-19T07:32:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 Processor -- start 2024-09-19T07:32:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 -- start start 2024-09-19T07:32:39.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22240c42e0 0x7f22241501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:39.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22241566b0 0x7f22241ee9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:39.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2224159a80 0x7f22241eef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:39.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f22241538f0 con 0x7f2224159a80 2024-09-19T07:32:39.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2224153770 con 0x7f22241566b0 2024-09-19T07:32:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f22240c46c0 con 0x7f22240c42e0 2024-09-19T07:32:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2222d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2224159a80 0x7f22241eef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2222d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2224159a80 0x7f22241eef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58350/0 (socket says 172.21.15.83:58350) 2024-09-19T07:32:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2222d9d640 1 -- 172.21.15.83:0/1131840484 learned_addr learned my addr 172.21.15.83:0/1131840484 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f222259c640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22240c42e0 0x7f22241501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2221d9b640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22241566b0 0x7f22241ee9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2222d9d640 1 -- 172.21.15.83:0/1131840484 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22240c42e0 msgr2=0x7f22241501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2222d9d640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22240c42e0 0x7f22241501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2222d9d640 1 -- 172.21.15.83:0/1131840484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22241566b0 msgr2=0x7f22241ee9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2222d9d640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22241566b0 0x7f22241ee9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2222d9d640 1 -- 172.21.15.83:0/1131840484 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22241ee390 con 0x7f2224159a80 2024-09-19T07:32:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2222d9d640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2224159a80 0x7f22241eef30 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f221c00c3f0 tx=0x7f221c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:39.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f221affd640 1 -- 172.21.15.83:0/1131840484 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221c017070 con 0x7f2224159a80 2024-09-19T07:32:39.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f221affd640 1 -- 172.21.15.83:0/1131840484 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f221c005cf0 con 0x7f2224159a80 2024-09-19T07:32:39.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 -- 172.21.15.83:0/1131840484 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22240c5ff0 con 0x7f2224159a80 2024-09-19T07:32:39.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f221affd640 1 -- 172.21.15.83:0/1131840484 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221c007310 con 0x7f2224159a80 2024-09-19T07:32:39.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.167+0000 7f2228a90640 1 -- 172.21.15.83:0/1131840484 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22240713d0 con 0x7f2224159a80 2024-09-19T07:32:39.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.171+0000 7f2228a90640 1 -- 172.21.15.83:0/1131840484 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2224159e90 con 0x7f2224159a80 2024-09-19T07:32:39.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.171+0000 7f221affd640 1 -- 172.21.15.83:0/1131840484 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f221c0074b0 con 0x7f2224159a80 2024-09-19T07:32:39.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.171+0000 7f221affd640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f22040678c0 0x7f2204069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:39.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.171+0000 7f221affd640 1 -- 172.21.15.83:0/1131840484 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7f221c0859b0 con 0x7f2224159a80 2024-09-19T07:32:39.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.171+0000 7f222259c640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f22040678c0 0x7f2204069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:39.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.171+0000 7f222259c640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f22040678c0 0x7f2204069d80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2224155b60 tx=0x7f220c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:39.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.175+0000 7f221affd640 1 -- 172.21.15.83:0/1131840484 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f221c055460 con 0x7f2224159a80 2024-09-19T07:32:39.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.375+0000 7f2228a90640 1 -- 172.21.15.83:0/1131840484 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f222406e050 con 0x7f2224159a80 2024-09-19T07:32:39.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.375+0000 7f221affd640 1 -- 172.21.15.83:0/1131840484 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5709 (secure 0 0 0) 0x7f221c059760 con 0x7f2224159a80 2024-09-19T07:32:39.369 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:39.369 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":102,"btime":"2024-09-19T07:32:39:289543+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:38.431666+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:39.370 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 102 2024-09-19T07:32:39.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.379+0000 7f2228a90640 1 -- 172.21.15.83:0/1131840484 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f22040678c0 msgr2=0x7f2204069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:39.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.379+0000 7f2228a90640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f22040678c0 0x7f2204069d80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2224155b60 tx=0x7f220c009190 comp rx=0 tx=0).stop 2024-09-19T07:32:39.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.379+0000 7f2228a90640 1 -- 172.21.15.83:0/1131840484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2224159a80 msgr2=0x7f22241eef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:39.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.379+0000 7f2228a90640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2224159a80 0x7f22241eef30 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f221c00c3f0 tx=0x7f221c00c8b0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.379+0000 7f2228a90640 1 -- 172.21.15.83:0/1131840484 shutdown_connections 2024-09-19T07:32:39.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.379+0000 7f2228a90640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f22040678c0 0x7f2204069d80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.379+0000 7f2228a90640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2224159a80 0x7f22241eef30 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.379+0000 7f2228a90640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22241566b0 0x7f22241ee9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.379+0000 7f2228a90640 1 --2- 172.21.15.83:0/1131840484 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22240c42e0 0x7f22241501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.379+0000 7f2228a90640 1 -- 172.21.15.83:0/1131840484 >> 172.21.15.83:0/1131840484 conn(0x7f222406bc80 msgr2=0x7f2224151770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.379+0000 7f2228a90640 1 -- 172.21.15.83:0/1131840484 shutdown_connections 2024-09-19T07:32:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:39.379+0000 7f2228a90640 1 -- 172.21.15.83:0/1131840484 wait complete. 2024-09-19T07:32:39.390 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:32:39:289543+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 102, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 102, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6561": { "addr": "172.21.15.83:6837/1203406606", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 1203406606, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 1203406606, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6561, "incarnation": 101, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:32:38.431666+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6561 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 99, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6447": { "addr": "172.21.15.83:6835/1072903448", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 1072903448, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 1072903448, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6447, "incarnation": 96, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:31:21.234481+0000", "qdb_cluster": [ 6447 ], "qdb_leader": 6447, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6447 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/2112478738", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 2112478738, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 2112478738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 95, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5722, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:32:41.391 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe878667640 1 Processor -- start 2024-09-19T07:32:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe878667640 1 -- start start 2024-09-19T07:32:41.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe878667640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe87006ffe0 0x7fe8700c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:41.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe878667640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe870159af0 0x7fe8700c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:41.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe878667640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8700c26c0 con 0x7fe8701535d0 2024-09-19T07:32:41.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe878667640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe8700707e0 con 0x7fe87006ffe0 2024-09-19T07:32:41.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe878667640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe8700c2c80 con 0x7fe870159af0 2024-09-19T07:32:41.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe875c02640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe87006ffe0 0x7fe8700c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:41.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe876403640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe8701535d0 0x7fe870155c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49976/0 (socket says 172.21.15.83:49976) 2024-09-19T07:32:41.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe876403640 1 -- 172.21.15.83:0/1208022278 learned_addr learned my addr 172.21.15.83:0/1208022278 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:32:41.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe876c04640 1 --2- 172.21.15.83:0/1208022278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe870159af0 0x7fe8700c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:41.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe875c02640 1 -- 172.21.15.83:0/1208022278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe870159af0 msgr2=0x7fe8700c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:41.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe875c02640 1 --2- 172.21.15.83:0/1208022278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe870159af0 0x7fe8700c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:41.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe875c02640 1 -- 172.21.15.83:0/1208022278 >> v1:172.21.15.83:6789/0 conn(0x7fe8701535d0 legacy=0x7fe870155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:41.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe875c02640 1 -- 172.21.15.83:0/1208022278 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8700c29a0 con 0x7fe87006ffe0 2024-09-19T07:32:41.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.547+0000 7fe876c04640 1 --2- 172.21.15.83:0/1208022278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe870159af0 0x7fe8700c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:32:41.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe875c02640 1 --2- 172.21.15.83:0/1208022278 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe87006ffe0 0x7fe8700c6460 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fe8600097c0 tx=0x7fe86001bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8327f9345e15f71b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:41.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe875401640 1 -- 172.21.15.83:0/1208022278 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87015a110 con 0x7fe87006ffe0 2024-09-19T07:32:41.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe875401640 1 -- 172.21.15.83:0/1208022278 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8600a85f0 con 0x7fe87006ffe0 2024-09-19T07:32:41.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe875401640 1 -- 172.21.15.83:0/1208022278 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8600a89c0 con 0x7fe87006ffe0 2024-09-19T07:32:41.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 -- 172.21.15.83:0/1208022278 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe87006ffe0 msgr2=0x7fe8700c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:41.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 --2- 172.21.15.83:0/1208022278 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe87006ffe0 0x7fe8700c6460 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fe8600097c0 tx=0x7fe86001bf80 comp rx=0 tx=0).stop 2024-09-19T07:32:41.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 -- 172.21.15.83:0/1208022278 shutdown_connections 2024-09-19T07:32:41.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 --2- 172.21.15.83:0/1208022278 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe87006ffe0 0x7fe8700c6460 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:41.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 --2- 172.21.15.83:0/1208022278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe870159af0 0x7fe8700c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:41.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 -- 172.21.15.83:0/1208022278 >> 172.21.15.83:0/1208022278 conn(0x7fe8700c5060 msgr2=0x7fe8700c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:41.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 -- 172.21.15.83:0/1208022278 shutdown_connections 2024-09-19T07:32:41.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 -- 172.21.15.83:0/1208022278 wait complete. 2024-09-19T07:32:41.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 Processor -- start 2024-09-19T07:32:41.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 -- start start 2024-09-19T07:32:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe87006ffe0 0x7fe870153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8701535d0 0x7fe8701546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe870159af0 0x7fe87015edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701638c0 con 0x7fe87006ffe0 2024-09-19T07:32:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe870163740 con 0x7fe8701535d0 2024-09-19T07:32:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe878667640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe870163a40 con 0x7fe870159af0 2024-09-19T07:32:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe876403640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe87006ffe0 0x7fe870153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe876403640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe87006ffe0 0x7fe870153c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56114/0 (socket says 172.21.15.83:56114) 2024-09-19T07:32:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe876403640 1 -- 172.21.15.83:0/3405505479 learned_addr learned my addr 172.21.15.83:0/3405505479 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe876c04640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe870159af0 0x7fe87015edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe875c02640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8701535d0 0x7fe8701546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe876403640 1 -- 172.21.15.83:0/3405505479 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe870159af0 msgr2=0x7fe87015edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe876403640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe870159af0 0x7fe87015edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe876403640 1 -- 172.21.15.83:0/3405505479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8701535d0 msgr2=0x7fe8701546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe876403640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8701535d0 0x7fe8701546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe876403640 1 -- 172.21.15.83:0/3405505479 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe870155cc0 con 0x7fe87006ffe0 2024-09-19T07:32:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe876c04640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe870159af0 0x7fe87015edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:32:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.551+0000 7fe875c02640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8701535d0 0x7fe8701546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:32:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.555+0000 7fe876403640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe87006ffe0 0x7fe870153c70 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00c780 tx=0x7fe86c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.555+0000 7fe866ffd640 1 -- 172.21.15.83:0/3405505479 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c018070 con 0x7fe87006ffe0 2024-09-19T07:32:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.555+0000 7fe866ffd640 1 -- 172.21.15.83:0/3405505479 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c0072d0 con 0x7fe87006ffe0 2024-09-19T07:32:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.555+0000 7fe878667640 1 -- 172.21.15.83:0/3405505479 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8701efc30 con 0x7fe87006ffe0 2024-09-19T07:32:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.555+0000 7fe866ffd640 1 -- 172.21.15.83:0/3405505479 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0047f0 con 0x7fe87006ffe0 2024-09-19T07:32:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.555+0000 7fe878667640 1 -- 172.21.15.83:0/3405505479 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8701eff10 con 0x7fe87006ffe0 2024-09-19T07:32:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.555+0000 7fe878667640 1 -- 172.21.15.83:0/3405505479 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8700c6460 con 0x7fe87006ffe0 2024-09-19T07:32:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.555+0000 7fe866ffd640 1 -- 172.21.15.83:0/3405505479 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe86c01b070 con 0x7fe87006ffe0 2024-09-19T07:32:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.555+0000 7fe866ffd640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe8540678c0 0x7fe854069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.555+0000 7fe875c02640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe8540678c0 0x7fe854069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.555+0000 7fe866ffd640 1 -- 172.21.15.83:0/3405505479 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7fe86c085b50 con 0x7fe87006ffe0 2024-09-19T07:32:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.555+0000 7fe875c02640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe8540678c0 0x7fe854069d80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe8600ad000 tx=0x7fe860008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.559+0000 7fe866ffd640 1 -- 172.21.15.83:0/3405505479 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe86c055cd0 con 0x7fe87006ffe0 2024-09-19T07:32:41.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 -- 172.21.15.83:0/3405505479 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8700adaf0 con 0x7fe87006ffe0 2024-09-19T07:32:41.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe866ffd640 1 -- 172.21.15.83:0/3405505479 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+5713 (secure 0 0 0) 0x7fe86c0555d0 con 0x7fe87006ffe0 2024-09-19T07:32:41.796 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:41.797 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":104,"btime":"2024-09-19T07:32:41:308712+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:41.797 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 104 2024-09-19T07:32:41.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 -- 172.21.15.83:0/3405505479 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe8540678c0 msgr2=0x7fe854069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:41.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe8540678c0 0x7fe854069d80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe8600ad000 tx=0x7fe860008040 comp rx=0 tx=0).stop 2024-09-19T07:32:41.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 -- 172.21.15.83:0/3405505479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe87006ffe0 msgr2=0x7fe870153c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:41.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe87006ffe0 0x7fe870153c70 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00c780 tx=0x7fe86c00cc40 comp rx=0 tx=0).stop 2024-09-19T07:32:41.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 -- 172.21.15.83:0/3405505479 shutdown_connections 2024-09-19T07:32:41.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe8540678c0 0x7fe854069d80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:41.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe870159af0 0x7fe87015edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:41.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8701535d0 0x7fe8701546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:41.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 --2- 172.21.15.83:0/3405505479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe87006ffe0 0x7fe870153c70 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:41.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 -- 172.21.15.83:0/3405505479 >> 172.21.15.83:0/3405505479 conn(0x7fe8700c5060 msgr2=0x7fe870158600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:41.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 -- 172.21.15.83:0/3405505479 shutdown_connections 2024-09-19T07:32:41.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:41.803+0000 7fe878667640 1 -- 172.21.15.83:0/3405505479 wait complete. 2024-09-19T07:32:41.818 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=4652 2024-09-19T07:32:41.818 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.d 2024-09-19T07:32:43.819 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-19T07:32:43.819 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-19T07:32:43.819 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-19T07:32:43.822 INFO:tasks.ceph.mds.d:Started 2024-09-19T07:32:43.822 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:43.907 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-19T07:32:43.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c3c1b640 1 Processor -- start 2024-09-19T07:32:43.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c3c1b640 1 -- start start 2024-09-19T07:32:43.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c3c1b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe8bc0c6120 0x7fe8bc0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:43.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c3c1b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe8bc155d70 0x7fe8bc1575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c3c1b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe8bc1592f0 0x7fe8bc157b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c3c1b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe8bc0c9930 con 0x7fe8bc0c6120 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c3c1b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe8bc156200 con 0x7fe8bc1592f0 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c3c1b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe8bc1564e0 con 0x7fe8bc155d70 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c19b7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe8bc0c6120 0x7fe8bc0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c19b7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe8bc0c6120 0x7fe8bc0c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56128/0 (socket says 172.21.15.83:56128) 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c19b7640 1 -- 172.21.15.83:0/4139064768 learned_addr learned my addr 172.21.15.83:0/4139064768 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c21b8640 1 --2- 172.21.15.83:0/4139064768 >> v2:172.21.15.184:3300/0 conn(0x7fe8bc1592f0 0x7fe8bc157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c11b6640 1 --2- 172.21.15.83:0/4139064768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe8bc155d70 0x7fe8bc1575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c19b7640 1 -- 172.21.15.83:0/4139064768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe8bc155d70 msgr2=0x7fe8bc1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c19b7640 1 --2- 172.21.15.83:0/4139064768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe8bc155d70 0x7fe8bc1575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c19b7640 1 -- 172.21.15.83:0/4139064768 >> v2:172.21.15.184:3300/0 conn(0x7fe8bc1592f0 msgr2=0x7fe8bc157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c19b7640 1 --2- 172.21.15.83:0/4139064768 >> v2:172.21.15.184:3300/0 conn(0x7fe8bc1592f0 0x7fe8bc157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:43.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.963+0000 7fe8c19b7640 1 -- 172.21.15.83:0/4139064768 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8bc1567c0 con 0x7fe8bc0c6120 2024-09-19T07:32:43.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c19b7640 1 --2- 172.21.15.83:0/4139064768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8bc0c6120 0x7fe8bc0c6500 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc08d2e0 tx=0x7fe8ac01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63ee0635b0395fc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:43.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c09b5640 1 -- 172.21.15.83:0/4139064768 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac0a5670 con 0x7fe8bc0c6120 2024-09-19T07:32:43.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c09b5640 1 -- 172.21.15.83:0/4139064768 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8ac0a5810 con 0x7fe8bc0c6120 2024-09-19T07:32:43.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c09b5640 1 -- 172.21.15.83:0/4139064768 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac0a5be0 con 0x7fe8bc0c6120 2024-09-19T07:32:43.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/4139064768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8bc0c6120 msgr2=0x7fe8bc0c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:43.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c3c1b640 1 --2- 172.21.15.83:0/4139064768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8bc0c6120 0x7fe8bc0c6500 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc08d2e0 tx=0x7fe8ac01c9f0 comp rx=0 tx=0).stop 2024-09-19T07:32:43.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/4139064768 shutdown_connections 2024-09-19T07:32:43.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c3c1b640 1 --2- 172.21.15.83:0/4139064768 >> v2:172.21.15.184:3300/0 conn(0x7fe8bc1592f0 0x7fe8bc157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:43.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c3c1b640 1 --2- 172.21.15.83:0/4139064768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe8bc155d70 0x7fe8bc1575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:43.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c3c1b640 1 --2- 172.21.15.83:0/4139064768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8bc0c6120 0x7fe8bc0c6500 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:43.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/4139064768 >> 172.21.15.83:0/4139064768 conn(0x7fe8bc06bc80 msgr2=0x7fe8bc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:43.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/4139064768 shutdown_connections 2024-09-19T07:32:43.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/4139064768 wait complete. 2024-09-19T07:32:43.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c3c1b640 1 Processor -- start 2024-09-19T07:32:43.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.967+0000 7fe8c3c1b640 1 -- start start 2024-09-19T07:32:43.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c3c1b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe8bc0c6120 0x7fe8bc0cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:43.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c3c1b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8bc1592f0 0x7fe8bc159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:43.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c3c1b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8bc14ae10 0x7fe8bc14b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:43.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c3c1b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc0c8b40 con 0x7fe8bc14ae10 2024-09-19T07:32:43.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c3c1b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc0c89c0 con 0x7fe8bc1592f0 2024-09-19T07:32:43.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c3c1b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe8bc0c8cc0 con 0x7fe8bc0c6120 2024-09-19T07:32:43.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c21b8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8bc14ae10 0x7fe8bc14b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:43.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c21b8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8bc14ae10 0x7fe8bc14b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56144/0 (socket says 172.21.15.83:56144) 2024-09-19T07:32:43.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c21b8640 1 -- 172.21.15.83:0/3266462862 learned_addr learned my addr 172.21.15.83:0/3266462862 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:43.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c11b6640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8bc1592f0 0x7fe8bc159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:43.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c19b7640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe8bc0c6120 0x7fe8bc0cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:43.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c21b8640 1 -- 172.21.15.83:0/3266462862 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe8bc0c6120 msgr2=0x7fe8bc0cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:43.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c21b8640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe8bc0c6120 0x7fe8bc0cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:43.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c21b8640 1 -- 172.21.15.83:0/3266462862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8bc1592f0 msgr2=0x7fe8bc159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:43.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c21b8640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8bc1592f0 0x7fe8bc159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:43.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c21b8640 1 -- 172.21.15.83:0/3266462862 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8bc1f2c60 con 0x7fe8bc14ae10 2024-09-19T07:32:43.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c21b8640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8bc14ae10 0x7fe8bc14b2c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe8b400e820 tx=0x7fe8b400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:43.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8ba7fc640 1 -- 172.21.15.83:0/3266462862 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4017070 con 0x7fe8bc14ae10 2024-09-19T07:32:43.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8ba7fc640 1 -- 172.21.15.83:0/3266462862 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8b400c940 con 0x7fe8bc14ae10 2024-09-19T07:32:43.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/3266462862 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8bc0cbca0 con 0x7fe8bc14ae10 2024-09-19T07:32:43.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8ba7fc640 1 -- 172.21.15.83:0/3266462862 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b400cd10 con 0x7fe8bc14ae10 2024-09-19T07:32:43.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/3266462862 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8bc1f2180 con 0x7fe8bc14ae10 2024-09-19T07:32:43.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/3266462862 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8bc0c6500 con 0x7fe8bc14ae10 2024-09-19T07:32:43.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8ba7fc640 1 -- 172.21.15.83:0/3266462862 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe8b401a070 con 0x7fe8bc14ae10 2024-09-19T07:32:43.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.971+0000 7fe8ba7fc640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe8a80678c0 0x7fe8a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:43.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.975+0000 7fe8ba7fc640 1 -- 172.21.15.83:0/3266462862 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7fe8b4086bc0 con 0x7fe8bc14ae10 2024-09-19T07:32:43.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.975+0000 7fe8c19b7640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe8a80678c0 0x7fe8a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:43.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.975+0000 7fe8c19b7640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe8a80678c0 0x7fe8a8069d80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0ae000 tx=0x7fe8ac0a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:43.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:43.979+0000 7fe8ba7fc640 1 -- 172.21.15.83:0/3266462862 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe8b4056770 con 0x7fe8bc14ae10 2024-09-19T07:32:44.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.195+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/3266462862 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8bc0c96a0 con 0x7fe8bc14ae10 2024-09-19T07:32:44.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8ba7fc640 1 -- 172.21.15.83:0/3266462862 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+5713 (secure 0 0 0) 0x7fe8b405aa70 con 0x7fe8bc14ae10 2024-09-19T07:32:44.192 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:44.192 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":104,"btime":"2024-09-19T07:32:41:308712+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:44.192 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 104 2024-09-19T07:32:44.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/3266462862 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe8a80678c0 msgr2=0x7fe8a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:44.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8c3c1b640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe8a80678c0 0x7fe8a8069d80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0ae000 tx=0x7fe8ac0a7070 comp rx=0 tx=0).stop 2024-09-19T07:32:44.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/3266462862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8bc14ae10 msgr2=0x7fe8bc14b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:44.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8c3c1b640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8bc14ae10 0x7fe8bc14b2c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe8b400e820 tx=0x7fe8b400ece0 comp rx=0 tx=0).stop 2024-09-19T07:32:44.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/3266462862 shutdown_connections 2024-09-19T07:32:44.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8c3c1b640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe8a80678c0 0x7fe8a8069d80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:44.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8c3c1b640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8bc14ae10 0x7fe8bc14b2c0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:44.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8c3c1b640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8bc1592f0 0x7fe8bc159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:44.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8c3c1b640 1 --2- 172.21.15.83:0/3266462862 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe8bc0c6120 0x7fe8bc0cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:44.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/3266462862 >> 172.21.15.83:0/3266462862 conn(0x7fe8bc06bc80 msgr2=0x7fe8bc154650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:44.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/3266462862 shutdown_connections 2024-09-19T07:32:44.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:44.199+0000 7fe8c3c1b640 1 -- 172.21.15.83:0/3266462862 wait complete. 2024-09-19T07:32:44.213 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-19T07:32:45.276 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:32:45.283+0000 7f89c5067640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:32:46.214 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:32:46.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.347+0000 7f2ac152f640 1 Processor -- start 2024-09-19T07:32:46.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.347+0000 7f2ac152f640 1 -- start start 2024-09-19T07:32:46.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.347+0000 7f2ac152f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2abc0c91b0 0x7f2abc159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:46.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.347+0000 7f2ac152f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2abc0c7510 con 0x7f2abc153600 2024-09-19T07:32:46.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.347+0000 7f2ac152f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2abc0c77f0 con 0x7f2abc0aee20 2024-09-19T07:32:46.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.347+0000 7f2ac152f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f2abc0c7ad0 con 0x7f2abc0c91b0 2024-09-19T07:32:46.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.347+0000 7f2abaffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2abc153600 0x7f2abc0c49d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49982/0 (socket says 172.21.15.83:49982) 2024-09-19T07:32:46.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.347+0000 7f2abaffd640 1 -- 172.21.15.83:0/4268932195 learned_addr learned my addr 172.21.15.83:0/4268932195 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:32:46.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.347+0000 7f2aba7fc640 1 --2- 172.21.15.83:0/4268932195 >> v2:172.21.15.184:3300/0 conn(0x7f2abc0c91b0 0x7f2abc159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:46.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.351+0000 7f2ab9ffb640 1 -- 172.21.15.83:0/4268932195 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2210606805 0 0) 0x7f2abc0c7510 con 0x7f2abc153600 2024-09-19T07:32:46.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.351+0000 7f2ab9ffb640 1 -- 172.21.15.83:0/4268932195 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa4003530 con 0x7f2abc153600 2024-09-19T07:32:46.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.351+0000 7f2aba7fc640 1 -- 172.21.15.83:0/4268932195 >> v1:172.21.15.184:6789/0 conn(0x7f2abc0aee20 legacy=0x7f2abc0c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:46.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.351+0000 7f2aba7fc640 1 -- 172.21.15.83:0/4268932195 >> v1:172.21.15.83:6789/0 conn(0x7f2abc153600 legacy=0x7f2abc0c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:46.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.351+0000 7f2aba7fc640 1 -- 172.21.15.83:0/4268932195 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc0c8010 con 0x7f2abc0c91b0 2024-09-19T07:32:46.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.351+0000 7f2aba7fc640 1 --2- 172.21.15.83:0/4268932195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2abc0c91b0 0x7f2abc159340 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00097c0 tx=0x7f2ab001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=650cd9c84929fbb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:46.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.351+0000 7f2ab9ffb640 1 -- 172.21.15.83:0/4268932195 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab00b1070 con 0x7f2abc0c91b0 2024-09-19T07:32:46.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.351+0000 7f2ab9ffb640 1 -- 172.21.15.83:0/4268932195 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab00ac410 con 0x7f2abc0c91b0 2024-09-19T07:32:46.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.351+0000 7f2ab9ffb640 1 -- 172.21.15.83:0/4268932195 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab00ac7e0 con 0x7f2abc0c91b0 2024-09-19T07:32:46.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 -- 172.21.15.83:0/4268932195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2abc0c91b0 msgr2=0x7f2abc159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:46.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 --2- 172.21.15.83:0/4268932195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2abc0c91b0 0x7f2abc159340 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00097c0 tx=0x7f2ab001c4a0 comp rx=0 tx=0).stop 2024-09-19T07:32:46.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 -- 172.21.15.83:0/4268932195 shutdown_connections 2024-09-19T07:32:46.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 --2- 172.21.15.83:0/4268932195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2abc0c91b0 0x7f2abc159340 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:46.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 -- 172.21.15.83:0/4268932195 >> 172.21.15.83:0/4268932195 conn(0x7f2abc0c9610 msgr2=0x7f2abc0c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:46.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 -- 172.21.15.83:0/4268932195 shutdown_connections 2024-09-19T07:32:46.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 -- 172.21.15.83:0/4268932195 wait complete. 2024-09-19T07:32:46.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 Processor -- start 2024-09-19T07:32:46.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 -- start start 2024-09-19T07:32:46.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc0aee20 0x7f2abc0abac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:46.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2abc153600 0x7f2abc147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:46.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2abc0ac2d0 0x7f2abc1478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:46.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc164020 con 0x7f2abc0aee20 2024-09-19T07:32:46.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc163ea0 con 0x7f2abc0ac2d0 2024-09-19T07:32:46.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2ac152f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2abc1641a0 con 0x7f2abc153600 2024-09-19T07:32:46.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2abaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc0aee20 0x7f2abc0abac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:46.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2abaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc0aee20 0x7f2abc0abac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56150/0 (socket says 172.21.15.83:56150) 2024-09-19T07:32:46.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.355+0000 7f2abaffd640 1 -- 172.21.15.83:0/783637077 learned_addr learned my addr 172.21.15.83:0/783637077 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:32:46.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2abb7fe640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2abc0ac2d0 0x7f2abc1478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2abaffd640 1 -- 172.21.15.83:0/783637077 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2abc153600 msgr2=0x7f2abc147390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2abaffd640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2abc153600 0x7f2abc147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2abaffd640 1 -- 172.21.15.83:0/783637077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2abc0ac2d0 msgr2=0x7f2abc1478d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2abaffd640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2abc0ac2d0 0x7f2abc1478d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2abaffd640 1 -- 172.21.15.83:0/783637077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc0ad9d0 con 0x7f2abc0aee20 2024-09-19T07:32:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2abaffd640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc0aee20 0x7f2abc0abac0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00cbc0 tx=0x7f2aac00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2aa37fe640 1 -- 172.21.15.83:0/783637077 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac017070 con 0x7f2abc0aee20 2024-09-19T07:32:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2aa37fe640 1 -- 172.21.15.83:0/783637077 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aac00ad40 con 0x7f2abc0aee20 2024-09-19T07:32:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2ac152f640 1 -- 172.21.15.83:0/783637077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2abc158e80 con 0x7f2abc0aee20 2024-09-19T07:32:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2aa37fe640 1 -- 172.21.15.83:0/783637077 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac012600 con 0x7f2abc0aee20 2024-09-19T07:32:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2ac152f640 1 -- 172.21.15.83:0/783637077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2abc0aacc0 con 0x7f2abc0aee20 2024-09-19T07:32:46.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.359+0000 7f2ac152f640 1 -- 172.21.15.83:0/783637077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2abc159340 con 0x7f2abc0aee20 2024-09-19T07:32:46.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.363+0000 7f2aa37fe640 1 -- 172.21.15.83:0/783637077 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2aac01a070 con 0x7f2abc0aee20 2024-09-19T07:32:46.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.363+0000 7f2aa37fe640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2aa80678f0 0x7f2aa8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:32:46.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.363+0000 7f2aa37fe640 1 -- 172.21.15.83:0/783637077 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7f2aac086890 con 0x7f2abc0aee20 2024-09-19T07:32:46.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.363+0000 7f2aba7fc640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2aa80678f0 0x7f2aa8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:32:46.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.363+0000 7f2aba7fc640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2aa80678f0 0x7f2aa8069db0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00ae000 tx=0x7f2ab0004740 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:32:46.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.367+0000 7f2aa37fe640 1 -- 172.21.15.83:0/783637077 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2aac056440 con 0x7f2abc0aee20 2024-09-19T07:32:46.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.651+0000 7f2ac152f640 1 -- 172.21.15.83:0/783637077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2abc0c4b70 con 0x7f2abc0aee20 2024-09-19T07:32:46.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.651+0000 7f2aa37fe640 1 -- 172.21.15.83:0/783637077 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+6542 (secure 0 0 0) 0x7f2aac05a740 con 0x7f2abc0aee20 2024-09-19T07:32:46.645 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:32:46.645 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":105,"btime":"2024-09-19T07:32:44:320183+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:32:46.645 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 105 2024-09-19T07:32:46.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.655+0000 7f2ac152f640 1 -- 172.21.15.83:0/783637077 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2aa80678f0 msgr2=0x7f2aa8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:46.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.655+0000 7f2ac152f640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2aa80678f0 0x7f2aa8069db0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00ae000 tx=0x7f2ab0004740 comp rx=0 tx=0).stop 2024-09-19T07:32:46.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.655+0000 7f2ac152f640 1 -- 172.21.15.83:0/783637077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc0aee20 msgr2=0x7f2abc0abac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:32:46.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.655+0000 7f2ac152f640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc0aee20 0x7f2abc0abac0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00cbc0 tx=0x7f2aac00a430 comp rx=0 tx=0).stop 2024-09-19T07:32:46.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.655+0000 7f2ac152f640 1 -- 172.21.15.83:0/783637077 shutdown_connections 2024-09-19T07:32:46.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.655+0000 7f2ac152f640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2aa80678f0 0x7f2aa8069db0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:46.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.655+0000 7f2ac152f640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2abc0ac2d0 0x7f2abc1478d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:46.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.655+0000 7f2ac152f640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2abc153600 0x7f2abc147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:46.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.655+0000 7f2ac152f640 1 --2- 172.21.15.83:0/783637077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc0aee20 0x7f2abc0abac0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:32:46.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.655+0000 7f2ac152f640 1 -- 172.21.15.83:0/783637077 >> 172.21.15.83:0/783637077 conn(0x7f2abc0c9610 msgr2=0x7f2abc0c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:32:46.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.655+0000 7f2ac152f640 1 -- 172.21.15.83:0/783637077 shutdown_connections 2024-09-19T07:32:46.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:32:46.655+0000 7f2ac152f640 1 -- 172.21.15.83:0/783637077 wait complete. 2024-09-19T07:32:46.670 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-19T07:32:46.670 INFO:tasks.mds_thrash.fs.[a]:waiting for 62 secs before thrashing 2024-09-19T07:33:03.352 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:33:03.355 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:33:12.623 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:12.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.778+0000 7fab0bbcd640 1 Processor -- start 2024-09-19T07:33:12.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.778+0000 7fab0bbcd640 1 -- start start 2024-09-19T07:33:12.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.778+0000 7fab0bbcd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab04153300 0x7fab0406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:12.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.778+0000 7fab0bbcd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fab0406d1d0 0x7fab040c23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:12.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.778+0000 7fab0bbcd640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab041593f0 con 0x7fab040c2910 2024-09-19T07:33:12.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.778+0000 7fab0bbcd640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fab041591a0 con 0x7fab0406d1d0 2024-09-19T07:33:12.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.778+0000 7fab0bbcd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fab041596d0 con 0x7fab04153300 2024-09-19T07:33:12.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.782+0000 7fab09969640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab04153300 0x7fab0406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:12.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.782+0000 7fab09168640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fab0406d1d0 0x7fab040c23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:12.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.782+0000 7fab09969640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab04153300 0x7fab0406cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:53976/0 (socket says 172.21.15.83:53976) 2024-09-19T07:33:12.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.782+0000 7fab09969640 1 -- 172.21.15.83:0/3250914064 learned_addr learned my addr 172.21.15.83:0/3250914064 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:12.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.782+0000 7fab09168640 1 -- 172.21.15.83:0/3250914064 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab04153300 msgr2=0x7fab0406cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:12.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.782+0000 7fab09168640 1 --2- 172.21.15.83:0/3250914064 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab04153300 0x7fab0406cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:12.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.782+0000 7fab09168640 1 -- 172.21.15.83:0/3250914064 >> v1:172.21.15.184:6789/0 conn(0x7fab040c2910 legacy=0x7fab04155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:12.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.782+0000 7fab09168640 1 -- 172.21.15.83:0/3250914064 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab041599b0 con 0x7fab0406d1d0 2024-09-19T07:33:12.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.782+0000 7fab09168640 1 --2- 172.21.15.83:0/3250914064 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab0406d1d0 0x7fab040c23d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7faaf40097c0 tx=0x7faaf401c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4707c64b82e6a79 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:12.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.782+0000 7fab08967640 1 -- 172.21.15.83:0/3250914064 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf40b0070 con 0x7fab0406d1d0 2024-09-19T07:33:12.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.782+0000 7fab08967640 1 -- 172.21.15.83:0/3250914064 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaf40ab600 con 0x7fab0406d1d0 2024-09-19T07:33:12.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.782+0000 7fab08967640 1 -- 172.21.15.83:0/3250914064 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf40ab9d0 con 0x7fab0406d1d0 2024-09-19T07:33:12.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3250914064 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab0406d1d0 msgr2=0x7fab040c23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:12.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 --2- 172.21.15.83:0/3250914064 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab0406d1d0 0x7fab040c23d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7faaf40097c0 tx=0x7faaf401c380 comp rx=0 tx=0).stop 2024-09-19T07:33:12.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3250914064 shutdown_connections 2024-09-19T07:33:12.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 --2- 172.21.15.83:0/3250914064 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab0406d1d0 0x7fab040c23d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:12.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 --2- 172.21.15.83:0/3250914064 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab04153300 0x7fab0406cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:12.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3250914064 >> 172.21.15.83:0/3250914064 conn(0x7fab040c5090 msgr2=0x7fab040c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:12.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3250914064 shutdown_connections 2024-09-19T07:33:12.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3250914064 wait complete. 2024-09-19T07:33:12.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 Processor -- start 2024-09-19T07:33:12.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 -- start start 2024-09-19T07:33:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab0406d1d0 0x7fab0414d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab040c2910 0x7fab0414ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab04153300 0x7fab0414f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fab04153860 con 0x7fab040c2910 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab041536e0 con 0x7fab04153300 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fab04162d50 con 0x7fab0406d1d0 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab09168640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab040c2910 0x7fab0414ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab09168640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab040c2910 0x7fab0414ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57116/0 (socket says 172.21.15.83:57116) 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab09168640 1 -- 172.21.15.83:0/3200328113 learned_addr learned my addr 172.21.15.83:0/3200328113 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0a16a640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab04153300 0x7fab0414f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab09969640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab0406d1d0 0x7fab0414d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab09168640 1 -- 172.21.15.83:0/3200328113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab0406d1d0 msgr2=0x7fab0414d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab09168640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab0406d1d0 0x7fab0414d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab09168640 1 -- 172.21.15.83:0/3200328113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab04153300 msgr2=0x7fab0414f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab09168640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab04153300 0x7fab0414f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab09168640 1 -- 172.21.15.83:0/3200328113 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab041ee490 con 0x7fab040c2910 2024-09-19T07:33:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0a16a640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab04153300 0x7fab0414f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:33:12.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab09168640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab040c2910 0x7fab0414ee30 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7faaf40ae040 tx=0x7faaf4019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:12.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7faafa7fc640 1 -- 172.21.15.83:0/3200328113 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf40b0040 con 0x7fab040c2910 2024-09-19T07:33:12.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7faafa7fc640 1 -- 172.21.15.83:0/3200328113 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaf40b43f0 con 0x7fab040c2910 2024-09-19T07:33:12.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3200328113 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab0414e630 con 0x7fab040c2910 2024-09-19T07:33:12.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7faafa7fc640 1 -- 172.21.15.83:0/3200328113 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf40b4800 con 0x7fab040c2910 2024-09-19T07:33:12.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.786+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3200328113 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab041601f0 con 0x7fab040c2910 2024-09-19T07:33:12.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.790+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3200328113 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab040c23d0 con 0x7fab040c2910 2024-09-19T07:33:12.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.790+0000 7faafa7fc640 1 -- 172.21.15.83:0/3200328113 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faaf40b49a0 con 0x7fab040c2910 2024-09-19T07:33:12.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.790+0000 7faafa7fc640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faae80678c0 0x7faae8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:12.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.790+0000 7faafa7fc640 1 -- 172.21.15.83:0/3200328113 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7faaf4120010 con 0x7fab040c2910 2024-09-19T07:33:12.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.790+0000 7fab09969640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faae80678c0 0x7faae8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:12.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.790+0000 7fab09969640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faae80678c0 0x7faae8069d80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fab00009a40 tx=0x7fab00008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:12.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:12.794+0000 7faafa7fc640 1 -- 172.21.15.83:0/3200328113 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7faaf4016090 con 0x7fab040c2910 2024-09-19T07:33:13.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.026+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3200328113 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab0414e3e0 con 0x7fab040c2910 2024-09-19T07:33:13.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.026+0000 7faafa7fc640 1 -- 172.21.15.83:0/3200328113 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+6542 (secure 0 0 0) 0x7faaf40f0230 con 0x7fab040c2910 2024-09-19T07:33:13.022 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:13.022 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":105,"btime":"2024-09-19T07:32:44:320183+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:33:13.022 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 105 2024-09-19T07:33:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.030+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3200328113 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faae80678c0 msgr2=0x7faae8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.030+0000 7fab0bbcd640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faae80678c0 0x7faae8069d80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fab00009a40 tx=0x7fab00008040 comp rx=0 tx=0).stop 2024-09-19T07:33:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.030+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3200328113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab040c2910 msgr2=0x7fab0414ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.030+0000 7fab0bbcd640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab040c2910 0x7fab0414ee30 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7faaf40ae040 tx=0x7faaf4019120 comp rx=0 tx=0).stop 2024-09-19T07:33:13.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.030+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3200328113 shutdown_connections 2024-09-19T07:33:13.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.030+0000 7fab0bbcd640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faae80678c0 0x7faae8069d80 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.030+0000 7fab0bbcd640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab04153300 0x7fab0414f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.030+0000 7fab0bbcd640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab040c2910 0x7fab0414ee30 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.030+0000 7fab0bbcd640 1 --2- 172.21.15.83:0/3200328113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab0406d1d0 0x7fab0414d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.030+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3200328113 >> 172.21.15.83:0/3200328113 conn(0x7fab040c5090 msgr2=0x7fab040726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:13.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.030+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3200328113 shutdown_connections 2024-09-19T07:33:13.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.030+0000 7fab0bbcd640 1 -- 172.21.15.83:0/3200328113 wait complete. 2024-09-19T07:33:13.043 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-19T07:33:13.043 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:33:13.043 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:33:13.056 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:33:13.056 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:13.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad9f70640 1 Processor -- start 2024-09-19T07:33:13.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad9f70640 1 -- start start 2024-09-19T07:33:13.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad9f70640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8ad41494a0 0x7f8ad406e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:13.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad9f70640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f8ad4153390 0x7f8ad406d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:13.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad9f70640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ad4155fa0 con 0x7f8ad4068700 2024-09-19T07:33:13.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad9f70640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad4156e40 con 0x7f8ad41494a0 2024-09-19T07:33:13.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad9f70640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad40c3850 con 0x7f8ad4153390 2024-09-19T07:33:13.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8ad41494a0 0x7f8ad406e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:13.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8ad41494a0 0x7f8ad406e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57124/0 (socket says 172.21.15.83:57124) 2024-09-19T07:33:13.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad37fe640 1 -- 172.21.15.83:0/702145098 learned_addr learned my addr 172.21.15.83:0/702145098 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:13.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad3fff640 1 --2- 172.21.15.83:0/702145098 >> v2:172.21.15.184:3300/0 conn(0x7f8ad4153390 0x7f8ad406d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:13.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad37fe640 1 -- 172.21.15.83:0/702145098 >> v2:172.21.15.184:3300/0 conn(0x7f8ad4153390 msgr2=0x7f8ad406d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:13.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad37fe640 1 --2- 172.21.15.83:0/702145098 >> v2:172.21.15.184:3300/0 conn(0x7f8ad4153390 0x7f8ad406d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad37fe640 1 -- 172.21.15.83:0/702145098 >> v1:172.21.15.83:6789/0 conn(0x7f8ad4068700 legacy=0x7f8ad4156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:13.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.230+0000 7f8ad37fe640 1 -- 172.21.15.83:0/702145098 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad40c3b10 con 0x7f8ad41494a0 2024-09-19T07:33:13.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad37fe640 1 --2- 172.21.15.83:0/702145098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ad41494a0 0x7f8ad406e010 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4150450 tx=0x7f8ac401ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d100a678bf6370da server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:13.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad27fc640 1 -- 172.21.15.83:0/702145098 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40b4070 con 0x7f8ad41494a0 2024-09-19T07:33:13.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad27fc640 1 -- 172.21.15.83:0/702145098 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac40a7ad0 con 0x7f8ad41494a0 2024-09-19T07:33:13.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad27fc640 1 -- 172.21.15.83:0/702145098 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40af440 con 0x7f8ad41494a0 2024-09-19T07:33:13.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad9f70640 1 -- 172.21.15.83:0/702145098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ad41494a0 msgr2=0x7f8ad406e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:13.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad9f70640 1 --2- 172.21.15.83:0/702145098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ad41494a0 0x7f8ad406e010 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4150450 tx=0x7f8ac401ebf0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad9f70640 1 -- 172.21.15.83:0/702145098 shutdown_connections 2024-09-19T07:33:13.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad9f70640 1 --2- 172.21.15.83:0/702145098 >> v2:172.21.15.184:3300/0 conn(0x7f8ad4153390 0x7f8ad406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad9f70640 1 --2- 172.21.15.83:0/702145098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ad41494a0 0x7f8ad406e010 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad9f70640 1 -- 172.21.15.83:0/702145098 >> 172.21.15.83:0/702145098 conn(0x7f8ad40c4940 msgr2=0x7f8ad40c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:13.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad9f70640 1 -- 172.21.15.83:0/702145098 shutdown_connections 2024-09-19T07:33:13.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad9f70640 1 -- 172.21.15.83:0/702145098 wait complete. 2024-09-19T07:33:13.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad9f70640 1 Processor -- start 2024-09-19T07:33:13.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.234+0000 7f8ad9f70640 1 -- start start 2024-09-19T07:33:13.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad9f70640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8ad4068700 0x7f8ad40ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:13.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad9f70640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ad41494a0 0x7f8ad41eb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:13.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad9f70640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ad4153390 0x7f8ad41eb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:13.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad9f70640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad4071eb0 con 0x7f8ad41494a0 2024-09-19T07:33:13.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad9f70640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad40c40d0 con 0x7f8ad4153390 2024-09-19T07:33:13.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad9f70640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad4072030 con 0x7f8ad4068700 2024-09-19T07:33:13.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ad41494a0 0x7f8ad41eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:13.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ad41494a0 0x7f8ad41eb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57140/0 (socket says 172.21.15.83:57140) 2024-09-19T07:33:13.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad2ffd640 1 -- 172.21.15.83:0/847367345 learned_addr learned my addr 172.21.15.83:0/847367345 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:13.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad37fe640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8ad4068700 0x7f8ad40ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:13.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad3fff640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ad4153390 0x7f8ad41eb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:13.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad2ffd640 1 -- 172.21.15.83:0/847367345 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8ad4068700 msgr2=0x7f8ad40ac6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:13.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad2ffd640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8ad4068700 0x7f8ad40ac6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad2ffd640 1 -- 172.21.15.83:0/847367345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ad4153390 msgr2=0x7f8ad41eb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:13.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad2ffd640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ad4153390 0x7f8ad41eb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad2ffd640 1 -- 172.21.15.83:0/847367345 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad4147850 con 0x7f8ad41494a0 2024-09-19T07:33:13.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad37fe640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8ad4068700 0x7f8ad40ac6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:33:13.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad2ffd640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ad41494a0 0x7f8ad41eb330 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f8ac800c420 tx=0x7f8ac800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:13.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8abbfff640 1 -- 172.21.15.83:0/847367345 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80052d0 con 0x7f8ad41494a0 2024-09-19T07:33:13.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8abbfff640 1 -- 172.21.15.83:0/847367345 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac8005470 con 0x7f8ad41494a0 2024-09-19T07:33:13.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad9f70640 1 -- 172.21.15.83:0/847367345 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad40c7b50 con 0x7f8ad41494a0 2024-09-19T07:33:13.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8abbfff640 1 -- 172.21.15.83:0/847367345 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8007260 con 0x7f8ad41494a0 2024-09-19T07:33:13.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad9f70640 1 -- 172.21.15.83:0/847367345 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad4147570 con 0x7f8ad41494a0 2024-09-19T07:33:13.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad9f70640 1 -- 172.21.15.83:0/847367345 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad4072360 con 0x7f8ad41494a0 2024-09-19T07:33:13.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8abbfff640 1 -- 172.21.15.83:0/847367345 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8ac8007400 con 0x7f8ad41494a0 2024-09-19T07:33:13.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8abbfff640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8ac00678c0 0x7f8ac0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:13.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8abbfff640 1 -- 172.21.15.83:0/847367345 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7f8ac8085b70 con 0x7f8ad41494a0 2024-09-19T07:33:13.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.238+0000 7f8ad37fe640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8ac00678c0 0x7f8ac0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:13.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.242+0000 7f8ad37fe640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8ac00678c0 0x7f8ac0069d80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8ac40b1000 tx=0x7f8ac40a9070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:13.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.246+0000 7f8abbfff640 1 -- 172.21.15.83:0/847367345 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f8ac8055cf0 con 0x7f8ad41494a0 2024-09-19T07:33:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.478+0000 7f8ad9f70640 1 -- 172.21.15.83:0/847367345 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ad40c7490 con 0x7f8ad41494a0 2024-09-19T07:33:13.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.478+0000 7f8abbfff640 1 -- 172.21.15.83:0/847367345 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+6542 (secure 0 0 0) 0x7f8ac80555f0 con 0x7f8ad41494a0 2024-09-19T07:33:13.473 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:13.473 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":105,"btime":"2024-09-19T07:32:44:320183+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:33:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 105 2024-09-19T07:33:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.482+0000 7f8ad9f70640 1 -- 172.21.15.83:0/847367345 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8ac00678c0 msgr2=0x7f8ac0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.482+0000 7f8ad9f70640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8ac00678c0 0x7f8ac0069d80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8ac40b1000 tx=0x7f8ac40a9070 comp rx=0 tx=0).stop 2024-09-19T07:33:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.482+0000 7f8ad9f70640 1 -- 172.21.15.83:0/847367345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ad41494a0 msgr2=0x7f8ad41eb330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.482+0000 7f8ad9f70640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ad41494a0 0x7f8ad41eb330 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f8ac800c420 tx=0x7f8ac800c8e0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.482+0000 7f8ad9f70640 1 -- 172.21.15.83:0/847367345 shutdown_connections 2024-09-19T07:33:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.482+0000 7f8ad9f70640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8ac00678c0 0x7f8ac0069d80 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.482+0000 7f8ad9f70640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ad4153390 0x7f8ad41eb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.482+0000 7f8ad9f70640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ad41494a0 0x7f8ad41eb330 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.482+0000 7f8ad9f70640 1 --2- 172.21.15.83:0/847367345 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8ad4068700 0x7f8ad40ac6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.482+0000 7f8ad9f70640 1 -- 172.21.15.83:0/847367345 >> 172.21.15.83:0/847367345 conn(0x7f8ad40c4940 msgr2=0x7f8ad4154fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.482+0000 7f8ad9f70640 1 -- 172.21.15.83:0/847367345 shutdown_connections 2024-09-19T07:33:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:13.482+0000 7f8ad9f70640 1 -- 172.21.15.83:0/847367345 wait complete. 2024-09-19T07:33:13.497 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:33:15.497 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:15.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.658+0000 7f87a7da3640 1 Processor -- start 2024-09-19T07:33:15.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.658+0000 7f87a7da3640 1 -- start start 2024-09-19T07:33:15.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.658+0000 7f87a7da3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f87a0153300 0x7f87a006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:15.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.658+0000 7f87a7da3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87a0155bb0 0x7f87a0156020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:15.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.658+0000 7f87a7da3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87a01593f0 con 0x7f87a006d1d0 2024-09-19T07:33:15.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.658+0000 7f87a7da3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f87a01591a0 con 0x7f87a0155bb0 2024-09-19T07:33:15.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.658+0000 7f87a7da3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f87a01596d0 con 0x7f87a0153300 2024-09-19T07:33:15.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.658+0000 7f87a6340640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87a0155bb0 0x7f87a0156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:15.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.658+0000 7f87a6340640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87a0155bb0 0x7f87a0156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57144/0 (socket says 172.21.15.83:57144) 2024-09-19T07:33:15.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.658+0000 7f87a6340640 1 -- 172.21.15.83:0/813152793 learned_addr learned my addr 172.21.15.83:0/813152793 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:15.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.658+0000 7f87a5b3f640 1 --2- 172.21.15.83:0/813152793 >> v2:172.21.15.184:3300/0 conn(0x7f87a0153300 0x7f87a006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:15.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a6340640 1 -- 172.21.15.83:0/813152793 >> v2:172.21.15.184:3300/0 conn(0x7f87a0153300 msgr2=0x7f87a006cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:15.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a6340640 1 --2- 172.21.15.83:0/813152793 >> v2:172.21.15.184:3300/0 conn(0x7f87a0153300 0x7f87a006cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:15.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a6340640 1 -- 172.21.15.83:0/813152793 >> v1:172.21.15.83:6789/0 conn(0x7f87a006d1d0 legacy=0x7f87a00c23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:15.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a6340640 1 -- 172.21.15.83:0/813152793 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a01599b0 con 0x7f87a0155bb0 2024-09-19T07:33:15.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a5b3f640 1 --2- 172.21.15.83:0/813152793 >> v2:172.21.15.184:3300/0 conn(0x7f87a0153300 0x7f87a006cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:15.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a6340640 1 --2- 172.21.15.83:0/813152793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a0155bb0 0x7f87a0156020 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f87a00c2de0 tx=0x7f879801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3388a3705122882e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:15.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a4b3d640 1 -- 172.21.15.83:0/813152793 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87980a56a0 con 0x7f87a0155bb0 2024-09-19T07:33:15.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a4b3d640 1 -- 172.21.15.83:0/813152793 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87980a5840 con 0x7f87a0155bb0 2024-09-19T07:33:15.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a4b3d640 1 -- 172.21.15.83:0/813152793 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87980a5c10 con 0x7f87a0155bb0 2024-09-19T07:33:15.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a7da3640 1 -- 172.21.15.83:0/813152793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a0155bb0 msgr2=0x7f87a0156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:15.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a7da3640 1 --2- 172.21.15.83:0/813152793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a0155bb0 0x7f87a0156020 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f87a00c2de0 tx=0x7f879801c8d0 comp rx=0 tx=0).stop 2024-09-19T07:33:15.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a7da3640 1 -- 172.21.15.83:0/813152793 shutdown_connections 2024-09-19T07:33:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a7da3640 1 --2- 172.21.15.83:0/813152793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a0155bb0 0x7f87a0156020 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a7da3640 1 --2- 172.21.15.83:0/813152793 >> v2:172.21.15.184:3300/0 conn(0x7f87a0153300 0x7f87a006cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a7da3640 1 -- 172.21.15.83:0/813152793 >> 172.21.15.83:0/813152793 conn(0x7f87a00c5090 msgr2=0x7f87a00c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a7da3640 1 -- 172.21.15.83:0/813152793 shutdown_connections 2024-09-19T07:33:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.662+0000 7f87a7da3640 1 -- 172.21.15.83:0/813152793 wait complete. 2024-09-19T07:33:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a7da3640 1 Processor -- start 2024-09-19T07:33:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a7da3640 1 -- start start 2024-09-19T07:33:15.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a7da3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87a006d1d0 0x7f87a014b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a7da3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a0153300 0x7f87a0154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a7da3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87a0155bb0 0x7f87a0149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a7da3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f87a0153860 con 0x7f87a0153300 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a7da3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f87a01536e0 con 0x7f87a0155bb0 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a7da3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f87a0162d50 con 0x7f87a006d1d0 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a533e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a0153300 0x7f87a0154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a533e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a0153300 0x7f87a0154960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57150/0 (socket says 172.21.15.83:57150) 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a533e640 1 -- 172.21.15.83:0/699407450 learned_addr learned my addr 172.21.15.83:0/699407450 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a6340640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87a0155bb0 0x7f87a0149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a5b3f640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87a006d1d0 0x7f87a014b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a533e640 1 -- 172.21.15.83:0/699407450 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87a006d1d0 msgr2=0x7f87a014b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a533e640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87a006d1d0 0x7f87a014b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a533e640 1 -- 172.21.15.83:0/699407450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87a0155bb0 msgr2=0x7f87a0149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:15.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a533e640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87a0155bb0 0x7f87a0149ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a533e640 1 -- 172.21.15.83:0/699407450 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a014b8d0 con 0x7f87a0153300 2024-09-19T07:33:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a533e640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a0153300 0x7f87a0154960 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f879000c460 tx=0x7f879000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87967fc640 1 -- 172.21.15.83:0/699407450 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8790016020 con 0x7f87a0153300 2024-09-19T07:33:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87967fc640 1 -- 172.21.15.83:0/699407450 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87900052d0 con 0x7f87a0153300 2024-09-19T07:33:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a7da3640 1 -- 172.21.15.83:0/699407450 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a0157140 con 0x7f87a0153300 2024-09-19T07:33:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87967fc640 1 -- 172.21.15.83:0/699407450 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8790007260 con 0x7f87a0153300 2024-09-19T07:33:15.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a7da3640 1 -- 172.21.15.83:0/699407450 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a0157700 con 0x7f87a0153300 2024-09-19T07:33:15.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.666+0000 7f87a7da3640 1 -- 172.21.15.83:0/699407450 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87a006cc90 con 0x7f87a0153300 2024-09-19T07:33:15.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.670+0000 7f87967fc640 1 -- 172.21.15.83:0/699407450 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8790007400 con 0x7f87a0153300 2024-09-19T07:33:15.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.670+0000 7f87967fc640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87840678c0 0x7f8784069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:15.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.670+0000 7f87967fc640 1 -- 172.21.15.83:0/699407450 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7f8790085fd0 con 0x7f87a0153300 2024-09-19T07:33:15.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.670+0000 7f87a5b3f640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87840678c0 0x7f8784069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:15.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.670+0000 7f87a5b3f640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87840678c0 0x7f8784069d80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f879c006fd0 tx=0x7f879c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:15.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.674+0000 7f87967fc640 1 -- 172.21.15.83:0/699407450 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f8790020020 con 0x7f87a0153300 2024-09-19T07:33:15.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.918+0000 7f87a7da3640 1 -- 172.21.15.83:0/699407450 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87a0156060 con 0x7f87a0153300 2024-09-19T07:33:15.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.918+0000 7f87967fc640 1 -- 172.21.15.83:0/699407450 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+6542 (secure 0 0 0) 0x7f8790005480 con 0x7f87a0153300 2024-09-19T07:33:15.912 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:15.912 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":105,"btime":"2024-09-19T07:32:44:320183+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:33:15.912 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 105 2024-09-19T07:33:15.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.918+0000 7f87a7da3640 1 -- 172.21.15.83:0/699407450 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87840678c0 msgr2=0x7f8784069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:15.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.918+0000 7f87a7da3640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87840678c0 0x7f8784069d80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f879c006fd0 tx=0x7f879c008040 comp rx=0 tx=0).stop 2024-09-19T07:33:15.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.918+0000 7f87a7da3640 1 -- 172.21.15.83:0/699407450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a0153300 msgr2=0x7f87a0154960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:15.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.918+0000 7f87a7da3640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a0153300 0x7f87a0154960 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f879000c460 tx=0x7f879000c920 comp rx=0 tx=0).stop 2024-09-19T07:33:15.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.918+0000 7f87a7da3640 1 -- 172.21.15.83:0/699407450 shutdown_connections 2024-09-19T07:33:15.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.918+0000 7f87a7da3640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87840678c0 0x7f8784069d80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:15.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.918+0000 7f87a7da3640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87a0155bb0 0x7f87a0149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:15.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.918+0000 7f87a7da3640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a0153300 0x7f87a0154960 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:15.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.918+0000 7f87a7da3640 1 --2- 172.21.15.83:0/699407450 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87a006d1d0 0x7f87a014b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:15.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.918+0000 7f87a7da3640 1 -- 172.21.15.83:0/699407450 >> 172.21.15.83:0/699407450 conn(0x7f87a00c5090 msgr2=0x7f87a00c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:15.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.922+0000 7f87a7da3640 1 -- 172.21.15.83:0/699407450 shutdown_connections 2024-09-19T07:33:15.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:15.922+0000 7f87a7da3640 1 -- 172.21.15.83:0/699407450 wait complete. 2024-09-19T07:33:15.936 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:33:17.937 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:18.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.082+0000 7f512fdaf640 1 Processor -- start 2024-09-19T07:33:18.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.082+0000 7f512fdaf640 1 -- start start 2024-09-19T07:33:18.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512fdaf640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5128068700 0x7f512806e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:18.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512fdaf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5128153390 0x7f512806d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:18.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512fdaf640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5128155fa0 con 0x7f51281494a0 2024-09-19T07:33:18.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512fdaf640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5128156e40 con 0x7f5128153390 2024-09-19T07:33:18.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512fdaf640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f51280c3850 con 0x7f5128068700 2024-09-19T07:33:18.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512e34c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5128153390 0x7f512806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:18.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512e34c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5128153390 0x7f512806d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57156/0 (socket says 172.21.15.83:57156) 2024-09-19T07:33:18.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512e34c640 1 -- 172.21.15.83:0/3242231569 learned_addr learned my addr 172.21.15.83:0/3242231569 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:18.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512e34c640 1 -- 172.21.15.83:0/3242231569 >> v2:172.21.15.184:3300/0 conn(0x7f5128068700 msgr2=0x7f512806e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:18.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512e34c640 1 --2- 172.21.15.83:0/3242231569 >> v2:172.21.15.184:3300/0 conn(0x7f5128068700 0x7f512806e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:18.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512e34c640 1 -- 172.21.15.83:0/3242231569 >> v1:172.21.15.184:6789/0 conn(0x7f51281494a0 legacy=0x7f5128156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:18.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512e34c640 1 -- 172.21.15.83:0/3242231569 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51280c3b10 con 0x7f5128153390 2024-09-19T07:33:18.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.086+0000 7f512e34c640 1 --2- 172.21.15.83:0/3242231569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5128153390 0x7f512806d470 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f51280c5630 tx=0x7f512001ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4ba78eb89d5f89 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:18.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512cb49640 1 -- 172.21.15.83:0/3242231569 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512000b2d0 con 0x7f5128153390 2024-09-19T07:33:18.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512cb49640 1 -- 172.21.15.83:0/3242231569 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f512000b470 con 0x7f5128153390 2024-09-19T07:33:18.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512cb49640 1 -- 172.21.15.83:0/3242231569 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51200a7ab0 con 0x7f5128153390 2024-09-19T07:33:18.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 -- 172.21.15.83:0/3242231569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5128153390 msgr2=0x7f512806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:18.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 --2- 172.21.15.83:0/3242231569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5128153390 0x7f512806d470 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f51280c5630 tx=0x7f512001ebb0 comp rx=0 tx=0).stop 2024-09-19T07:33:18.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 -- 172.21.15.83:0/3242231569 shutdown_connections 2024-09-19T07:33:18.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 --2- 172.21.15.83:0/3242231569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5128153390 0x7f512806d470 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:18.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 --2- 172.21.15.83:0/3242231569 >> v2:172.21.15.184:3300/0 conn(0x7f5128068700 0x7f512806e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:18.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 -- 172.21.15.83:0/3242231569 >> 172.21.15.83:0/3242231569 conn(0x7f51280c4940 msgr2=0x7f51280c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:18.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 -- 172.21.15.83:0/3242231569 shutdown_connections 2024-09-19T07:33:18.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 -- 172.21.15.83:0/3242231569 wait complete. 2024-09-19T07:33:18.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 Processor -- start 2024-09-19T07:33:18.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 -- start start 2024-09-19T07:33:18.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5128068700 0x7f5128143530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:18.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51281494a0 0x7f5128143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:18.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5128153390 0x7f51281edb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:18.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5128071eb0 con 0x7f5128153390 2024-09-19T07:33:18.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f51280c40d0 con 0x7f5128068700 2024-09-19T07:33:18.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5128072030 con 0x7f51281494a0 2024-09-19T07:33:18.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512e34c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5128153390 0x7f51281edb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:18.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512e34c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5128153390 0x7f51281edb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57172/0 (socket says 172.21.15.83:57172) 2024-09-19T07:33:18.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512e34c640 1 -- 172.21.15.83:0/3750963727 learned_addr learned my addr 172.21.15.83:0/3750963727 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:18.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512d34a640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51281494a0 0x7f5128143a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:18.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512db4b640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5128068700 0x7f5128143530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:18.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512e34c640 1 -- 172.21.15.83:0/3750963727 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51281494a0 msgr2=0x7f5128143a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:18.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512e34c640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51281494a0 0x7f5128143a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:18.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512e34c640 1 -- 172.21.15.83:0/3750963727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5128068700 msgr2=0x7f5128143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:18.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512e34c640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5128068700 0x7f5128143530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:18.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512e34c640 1 -- 172.21.15.83:0/3750963727 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51281ee9b0 con 0x7f5128153390 2024-09-19T07:33:18.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512e34c640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5128153390 0x7f51281edb50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f512001c7e0 tx=0x7f512001b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:18.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f511e7fc640 1 -- 172.21.15.83:0/3750963727 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51200be020 con 0x7f5128153390 2024-09-19T07:33:18.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f511e7fc640 1 -- 172.21.15.83:0/3750963727 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5120009820 con 0x7f5128153390 2024-09-19T07:33:18.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f511e7fc640 1 -- 172.21.15.83:0/3750963727 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120009c10 con 0x7f5128153390 2024-09-19T07:33:18.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 -- 172.21.15.83:0/3750963727 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51280acb40 con 0x7f5128153390 2024-09-19T07:33:18.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.090+0000 7f512fdaf640 1 -- 172.21.15.83:0/3750963727 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51280ad100 con 0x7f5128153390 2024-09-19T07:33:18.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.094+0000 7f512fdaf640 1 -- 172.21.15.83:0/3750963727 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51280723a0 con 0x7f5128153390 2024-09-19T07:33:18.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.094+0000 7f511e7fc640 1 -- 172.21.15.83:0/3750963727 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f51200c1020 con 0x7f5128153390 2024-09-19T07:33:18.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.094+0000 7f511e7fc640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f510c0678c0 0x7f510c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:18.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.094+0000 7f511e7fc640 1 -- 172.21.15.83:0/3750963727 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7f51201237a0 con 0x7f5128153390 2024-09-19T07:33:18.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.094+0000 7f512db4b640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f510c0678c0 0x7f510c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:18.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.094+0000 7f512db4b640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f510c0678c0 0x7f510c069d80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5124001600 tx=0x7f5124009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:18.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.098+0000 7f511e7fc640 1 -- 172.21.15.83:0/3750963727 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f51200c8030 con 0x7f5128153390 2024-09-19T07:33:18.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 -- 172.21.15.83:0/3750963727 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5128070060 con 0x7f5128153390 2024-09-19T07:33:18.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f511e7fc640 1 -- 172.21.15.83:0/3750963727 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+6542 (secure 0 0 0) 0x7f51200187a0 con 0x7f5128153390 2024-09-19T07:33:18.336 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:18.336 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":105,"btime":"2024-09-19T07:32:44:320183+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:33:18.336 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 105 2024-09-19T07:33:18.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 -- 172.21.15.83:0/3750963727 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f510c0678c0 msgr2=0x7f510c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:18.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f510c0678c0 0x7f510c069d80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5124001600 tx=0x7f5124009040 comp rx=0 tx=0).stop 2024-09-19T07:33:18.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 -- 172.21.15.83:0/3750963727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5128153390 msgr2=0x7f51281edb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:18.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5128153390 0x7f51281edb50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f512001c7e0 tx=0x7f512001b950 comp rx=0 tx=0).stop 2024-09-19T07:33:18.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 -- 172.21.15.83:0/3750963727 shutdown_connections 2024-09-19T07:33:18.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f510c0678c0 0x7f510c069d80 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:18.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5128153390 0x7f51281edb50 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:18.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f51281494a0 0x7f5128143a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:18.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 --2- 172.21.15.83:0/3750963727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5128068700 0x7f5128143530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:18.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 -- 172.21.15.83:0/3750963727 >> 172.21.15.83:0/3750963727 conn(0x7f51280c4940 msgr2=0x7f512806de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:18.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 -- 172.21.15.83:0/3750963727 shutdown_connections 2024-09-19T07:33:18.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:18.342+0000 7f512fdaf640 1 -- 172.21.15.83:0/3750963727 wait complete. 2024-09-19T07:33:18.356 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:33:20.356 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:20.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.498+0000 7fcaba749640 1 Processor -- start 2024-09-19T07:33:20.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.498+0000 7fcaba749640 1 -- start start 2024-09-19T07:33:20.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.498+0000 7fcaba749640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcab40a6f90 0x7fcab40c6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:20.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.498+0000 7fcaba749640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcab40c6b90 0x7fcab4160290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:20.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.498+0000 7fcaba749640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcab4156750 con 0x7fcab4152ee0 2024-09-19T07:33:20.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.498+0000 7fcaba749640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fcab40c5920 con 0x7fcab40a6f90 2024-09-19T07:33:20.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.498+0000 7fcaba749640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcab406d810 con 0x7fcab40c6b90 2024-09-19T07:33:20.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.502+0000 7fcab8f46640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcab40a6f90 0x7fcab40c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:20.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.502+0000 7fcab8f46640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcab40a6f90 0x7fcab40c6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:51186/0 (socket says 172.21.15.83:51186) 2024-09-19T07:33:20.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.502+0000 7fcab8f46640 1 -- 172.21.15.83:0/3703851331 learned_addr learned my addr 172.21.15.83:0/3703851331 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:20.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.502+0000 7fcab9f48640 1 --2- 172.21.15.83:0/3703851331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcab40c6b90 0x7fcab4160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:20.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.502+0000 7fcab8f46640 1 -- 172.21.15.83:0/3703851331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcab40c6b90 msgr2=0x7fcab4160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:20.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.502+0000 7fcab8f46640 1 --2- 172.21.15.83:0/3703851331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcab40c6b90 0x7fcab4160290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:20.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.502+0000 7fcab8f46640 1 -- 172.21.15.83:0/3703851331 >> v1:172.21.15.184:6789/0 conn(0x7fcab4152ee0 legacy=0x7fcab40c0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:20.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.502+0000 7fcab8f46640 1 -- 172.21.15.83:0/3703851331 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab40c5b70 con 0x7fcab40a6f90 2024-09-19T07:33:20.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.502+0000 7fcab8f46640 1 --2- 172.21.15.83:0/3703851331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcab40a6f90 0x7fcab40c6650 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4009870 tx=0x7fcaa401c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c86fa4da11a4931a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:20.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.502+0000 7fcaabfff640 1 -- 172.21.15.83:0/3703851331 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa40b2070 con 0x7fcab40a6f90 2024-09-19T07:33:20.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.502+0000 7fcaabfff640 1 -- 172.21.15.83:0/3703851331 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaa40a9070 con 0x7fcab40a6f90 2024-09-19T07:33:20.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.502+0000 7fcaabfff640 1 -- 172.21.15.83:0/3703851331 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa40ad880 con 0x7fcab40a6f90 2024-09-19T07:33:20.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 -- 172.21.15.83:0/3703851331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcab40a6f90 msgr2=0x7fcab40c6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:20.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 --2- 172.21.15.83:0/3703851331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcab40a6f90 0x7fcab40c6650 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4009870 tx=0x7fcaa401c560 comp rx=0 tx=0).stop 2024-09-19T07:33:20.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 -- 172.21.15.83:0/3703851331 shutdown_connections 2024-09-19T07:33:20.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 --2- 172.21.15.83:0/3703851331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcab40c6b90 0x7fcab4160290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:20.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 --2- 172.21.15.83:0/3703851331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcab40a6f90 0x7fcab40c6650 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:20.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 -- 172.21.15.83:0/3703851331 >> 172.21.15.83:0/3703851331 conn(0x7fcab40c4520 msgr2=0x7fcab40c4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:20.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 -- 172.21.15.83:0/3703851331 shutdown_connections 2024-09-19T07:33:20.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 -- 172.21.15.83:0/3703851331 wait complete. 2024-09-19T07:33:20.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 Processor -- start 2024-09-19T07:33:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 -- start start 2024-09-19T07:33:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcab40a6f90 0x7fcab4151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcab40c6b90 0x7fcab4158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcab4152ee0 0x7fcab41ee690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcab40c16e0 con 0x7fcab40a6f90 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcab40c1560 con 0x7fcab40c6b90 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaba749640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcab406cfc0 con 0x7fcab4152ee0 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcab9747640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcab40a6f90 0x7fcab4151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcab9747640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcab40a6f90 0x7fcab4151210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38212/0 (socket says 172.21.15.83:38212) 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcab9747640 1 -- 172.21.15.83:0/548668758 learned_addr learned my addr 172.21.15.83:0/548668758 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcab8f46640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcab40c6b90 0x7fcab4158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcab9f48640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcab4152ee0 0x7fcab41ee690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcab9747640 1 -- 172.21.15.83:0/548668758 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcab4152ee0 msgr2=0x7fcab41ee690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcab9747640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcab4152ee0 0x7fcab41ee690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcab9747640 1 -- 172.21.15.83:0/548668758 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcab40c6b90 msgr2=0x7fcab4158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcab9747640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcab40c6b90 0x7fcab4158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcab9747640 1 -- 172.21.15.83:0/548668758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab4072fa0 con 0x7fcab40a6f90 2024-09-19T07:33:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcab9747640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcab40a6f90 0x7fcab4151210 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fcab000c3a0 tx=0x7fcab000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.506+0000 7fcaa9ffb640 1 -- 172.21.15.83:0/548668758 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab00052d0 con 0x7fcab40a6f90 2024-09-19T07:33:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.510+0000 7fcaa9ffb640 1 -- 172.21.15.83:0/548668758 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcab0005470 con 0x7fcab40a6f90 2024-09-19T07:33:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.510+0000 7fcaba749640 1 -- 172.21.15.83:0/548668758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab4073770 con 0x7fcab40a6f90 2024-09-19T07:33:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.510+0000 7fcaa9ffb640 1 -- 172.21.15.83:0/548668758 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0007260 con 0x7fcab40a6f90 2024-09-19T07:33:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.510+0000 7fcaba749640 1 -- 172.21.15.83:0/548668758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab40a8fa0 con 0x7fcab40a6f90 2024-09-19T07:33:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.510+0000 7fcaba749640 1 -- 172.21.15.83:0/548668758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcab4156390 con 0x7fcab40a6f90 2024-09-19T07:33:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.510+0000 7fcaa9ffb640 1 -- 172.21.15.83:0/548668758 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcab0007400 con 0x7fcab40a6f90 2024-09-19T07:33:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.510+0000 7fcaa9ffb640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fca980678c0 0x7fca98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.510+0000 7fcaa9ffb640 1 -- 172.21.15.83:0/548668758 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7fcab0085a10 con 0x7fcab40a6f90 2024-09-19T07:33:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.510+0000 7fcab8f46640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fca980678c0 0x7fca98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.510+0000 7fcab8f46640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fca980678c0 0x7fca98069d80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40af000 tx=0x7fcaa4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:20.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.514+0000 7fcaa9ffb640 1 -- 172.21.15.83:0/548668758 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcab0055540 con 0x7fcab40a6f90 2024-09-19T07:33:20.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.746+0000 7fcaba749640 1 -- 172.21.15.83:0/548668758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcab40c4b50 con 0x7fcab40a6f90 2024-09-19T07:33:20.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.746+0000 7fcaa9ffb640 1 -- 172.21.15.83:0/548668758 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+6542 (secure 0 0 0) 0x7fcab0059840 con 0x7fcab40a6f90 2024-09-19T07:33:20.741 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:20.741 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":105,"btime":"2024-09-19T07:32:44:320183+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:33:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 105 2024-09-19T07:33:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.750+0000 7fcaba749640 1 -- 172.21.15.83:0/548668758 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fca980678c0 msgr2=0x7fca98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.750+0000 7fcaba749640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fca980678c0 0x7fca98069d80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40af000 tx=0x7fcaa4008040 comp rx=0 tx=0).stop 2024-09-19T07:33:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.750+0000 7fcaba749640 1 -- 172.21.15.83:0/548668758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcab40a6f90 msgr2=0x7fcab4151210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.750+0000 7fcaba749640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcab40a6f90 0x7fcab4151210 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fcab000c3a0 tx=0x7fcab000c860 comp rx=0 tx=0).stop 2024-09-19T07:33:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.750+0000 7fcaba749640 1 -- 172.21.15.83:0/548668758 shutdown_connections 2024-09-19T07:33:20.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.750+0000 7fcaba749640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fca980678c0 0x7fca98069d80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:20.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.750+0000 7fcaba749640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcab4152ee0 0x7fcab41ee690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:20.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.750+0000 7fcaba749640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcab40c6b90 0x7fcab4158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:20.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.750+0000 7fcaba749640 1 --2- 172.21.15.83:0/548668758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcab40a6f90 0x7fcab4151210 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:20.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.750+0000 7fcaba749640 1 -- 172.21.15.83:0/548668758 >> 172.21.15.83:0/548668758 conn(0x7fcab40c4520 msgr2=0x7fcab40c32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:20.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.750+0000 7fcaba749640 1 -- 172.21.15.83:0/548668758 shutdown_connections 2024-09-19T07:33:20.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:20.750+0000 7fcaba749640 1 -- 172.21.15.83:0/548668758 wait complete. 2024-09-19T07:33:20.761 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:33:22.762 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:22.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.910+0000 7f23a4777640 1 Processor -- start 2024-09-19T07:33:22.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.910+0000 7f23a4777640 1 -- start start 2024-09-19T07:33:22.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a4777640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f239c1513a0 0x7f239c154770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:22.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a4777640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f239c0aa180 con 0x7f239c0c2430 2024-09-19T07:33:22.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a4777640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f239c06e740 con 0x7f239c0c1ff0 2024-09-19T07:33:22.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a4777640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f239c06ed00 con 0x7f239c1513a0 2024-09-19T07:33:22.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a2d14640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f239c0c2430 0x7f239c15e560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38328/0 (socket says 172.21.15.83:38328) 2024-09-19T07:33:22.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a2d14640 1 -- 172.21.15.83:0/109754469 learned_addr learned my addr 172.21.15.83:0/109754469 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:33:22.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a2513640 1 --2- 172.21.15.83:0/109754469 >> v2:172.21.15.184:3300/0 conn(0x7f239c1513a0 0x7f239c154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:22.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4022276803 0 0) 0x7f239c0aa180 con 0x7f239c0c2430 2024-09-19T07:33:22.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2388003530 con 0x7f239c0c2430 2024-09-19T07:33:22.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1140009536 0 0) 0x7f2388003530 con 0x7f239c0c2430 2024-09-19T07:33:22.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 >> v2:172.21.15.184:3300/0 conn(0x7f239c1513a0 msgr2=0x7f239c154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:22.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 --2- 172.21.15.83:0/109754469 >> v2:172.21.15.184:3300/0 conn(0x7f239c1513a0 0x7f239c154770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:22.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 >> v1:172.21.15.184:6789/0 conn(0x7f239c0c1ff0 legacy=0x7f239c153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:22.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239c06ea20 con 0x7f239c0c2430 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f2394001a00 con 0x7f239c0c2430 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 >> v1:172.21.15.83:6789/0 conn(0x7f239c0c2430 legacy=0x7f239c15e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 --2- 172.21.15.83:0/109754469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2388003a10 0x7f2388003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 --2- 172.21.15.83:0/109754469 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2388004810 0x7f2388004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 --2- 172.21.15.83:0/109754469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2388005220 0x7f238800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f239c06ed00 con 0x7f2388003a10 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f238800c170 con 0x7f2388005220 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f238800c370 con 0x7f2388004810 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a2513640 1 --2- 172.21.15.83:0/109754469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2388003a10 0x7f2388003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a2d14640 1 --2- 172.21.15.83:0/109754469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2388005220 0x7f238800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a1d12640 1 --2- 172.21.15.83:0/109754469 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2388004810 0x7f2388004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a2513640 1 -- 172.21.15.83:0/109754469 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2388004810 msgr2=0x7f2388004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a2513640 1 --2- 172.21.15.83:0/109754469 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2388004810 0x7f2388004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:22.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a2513640 1 -- 172.21.15.83:0/109754469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2388005220 msgr2=0x7f238800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:22.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a2513640 1 --2- 172.21.15.83:0/109754469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2388005220 0x7f238800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:22.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a2513640 1 -- 172.21.15.83:0/109754469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2388003530 con 0x7f2388003a10 2024-09-19T07:33:22.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a2d14640 1 --2- 172.21.15.83:0/109754469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2388005220 0x7f238800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:22.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.914+0000 7f23a2d14640 1 -- 172.21.15.83:0/109754469 reap_dead start 2024-09-19T07:33:22.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a2513640 1 --2- 172.21.15.83:0/109754469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2388003a10 0x7f2388003e30 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f2398009ab0 tx=0x7f239801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366b1e4b8b781f71 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:22.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23980096b0 con 0x7f2388003a10 2024-09-19T07:33:22.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23980a6410 con 0x7f2388003a10 2024-09-19T07:33:22.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a1511640 1 -- 172.21.15.83:0/109754469 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23980a6800 con 0x7f2388003a10 2024-09-19T07:33:22.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 -- 172.21.15.83:0/109754469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2388003a10 msgr2=0x7f2388003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:22.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 --2- 172.21.15.83:0/109754469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2388003a10 0x7f2388003e30 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f2398009ab0 tx=0x7f239801ca30 comp rx=0 tx=0).stop 2024-09-19T07:33:22.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 -- 172.21.15.83:0/109754469 shutdown_connections 2024-09-19T07:33:22.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 --2- 172.21.15.83:0/109754469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2388003a10 0x7f2388003e30 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:22.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 -- 172.21.15.83:0/109754469 >> 172.21.15.83:0/109754469 conn(0x7f239c06bc80 msgr2=0x7f239c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:22.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 -- 172.21.15.83:0/109754469 shutdown_connections 2024-09-19T07:33:22.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 -- 172.21.15.83:0/109754469 wait complete. 2024-09-19T07:33:22.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 Processor -- start 2024-09-19T07:33:22.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 -- start start 2024-09-19T07:33:22.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2388003a10 0x7f239c0c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:22.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f239c148690 0x7f239c141260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f239c1509b0 0x7f239c1417a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f239c151900 con 0x7f239c1509b0 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f239c151780 con 0x7f239c148690 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a4777640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f239c072f50 con 0x7f2388003a10 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a2d14640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f239c1509b0 0x7f239c1417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a2d14640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f239c1509b0 0x7f239c1417a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38224/0 (socket says 172.21.15.83:38224) 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a2d14640 1 -- 172.21.15.83:0/2433234643 learned_addr learned my addr 172.21.15.83:0/2433234643 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a1d12640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f239c148690 0x7f239c141260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a2513640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2388003a10 0x7f239c0c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a2d14640 1 -- 172.21.15.83:0/2433234643 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2388003a10 msgr2=0x7f239c0c0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a2d14640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2388003a10 0x7f239c0c0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a2d14640 1 -- 172.21.15.83:0/2433234643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f239c148690 msgr2=0x7f239c141260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:22.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a2d14640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f239c148690 0x7f239c141260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:22.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a2d14640 1 -- 172.21.15.83:0/2433234643 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239c141eb0 con 0x7f239c1509b0 2024-09-19T07:33:22.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.918+0000 7f23a1d12640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f239c148690 0x7f239c141260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:22.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f23a2513640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2388003a10 0x7f239c0c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:22.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f23a2d14640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f239c1509b0 0x7f239c1417a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f239c0ad7b0 tx=0x7f239400df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:22.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f2392ffd640 1 -- 172.21.15.83:0/2433234643 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239400b800 con 0x7f239c1509b0 2024-09-19T07:33:22.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f2392ffd640 1 -- 172.21.15.83:0/2433234643 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f239400b9a0 con 0x7f239c1509b0 2024-09-19T07:33:22.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f23a4777640 1 -- 172.21.15.83:0/2433234643 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f239c142150 con 0x7f239c1509b0 2024-09-19T07:33:22.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f2392ffd640 1 -- 172.21.15.83:0/2433234643 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239400bd50 con 0x7f239c1509b0 2024-09-19T07:33:22.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f23a4777640 1 -- 172.21.15.83:0/2433234643 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f239c1e2760 con 0x7f239c1509b0 2024-09-19T07:33:22.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f23a4777640 1 -- 172.21.15.83:0/2433234643 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f239c0a9d30 con 0x7f239c1509b0 2024-09-19T07:33:22.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f2392ffd640 1 -- 172.21.15.83:0/2433234643 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f239401b020 con 0x7f239c1509b0 2024-09-19T07:33:22.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f2392ffd640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f23800678f0 0x7f2380069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:22.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f2392ffd640 1 -- 172.21.15.83:0/2433234643 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7f2394086720 con 0x7f239c1509b0 2024-09-19T07:33:22.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f23a2513640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f23800678f0 0x7f2380069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:22.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.922+0000 7f23a2513640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f23800678f0 0x7f2380069db0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f23980ad000 tx=0x7f23980a9040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:22.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:22.926+0000 7f2392ffd640 1 -- 172.21.15.83:0/2433234643 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2394056810 con 0x7f239c1509b0 2024-09-19T07:33:23.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.134+0000 7f23a4777640 1 -- 172.21.15.83:0/2433234643 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f239c0c4f10 con 0x7f239c1509b0 2024-09-19T07:33:23.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.134+0000 7f2392ffd640 1 -- 172.21.15.83:0/2433234643 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+6542 (secure 0 0 0) 0x7f2394056110 con 0x7f239c1509b0 2024-09-19T07:33:23.129 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:23.129 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":105,"btime":"2024-09-19T07:32:44:320183+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:33:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 105 2024-09-19T07:33:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.138+0000 7f23a4777640 1 -- 172.21.15.83:0/2433234643 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f23800678f0 msgr2=0x7f2380069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.138+0000 7f23a4777640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f23800678f0 0x7f2380069db0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f23980ad000 tx=0x7f23980a9040 comp rx=0 tx=0).stop 2024-09-19T07:33:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.138+0000 7f23a4777640 1 -- 172.21.15.83:0/2433234643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f239c1509b0 msgr2=0x7f239c1417a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.138+0000 7f23a4777640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f239c1509b0 0x7f239c1417a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f239c0ad7b0 tx=0x7f239400df70 comp rx=0 tx=0).stop 2024-09-19T07:33:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.138+0000 7f23a4777640 1 -- 172.21.15.83:0/2433234643 shutdown_connections 2024-09-19T07:33:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.138+0000 7f23a4777640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f23800678f0 0x7f2380069db0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.138+0000 7f23a4777640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f239c1509b0 0x7f239c1417a0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.138+0000 7f23a4777640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f239c148690 0x7f239c141260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.138+0000 7f23a4777640 1 --2- 172.21.15.83:0/2433234643 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2388003a10 0x7f239c0c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.138+0000 7f23a4777640 1 -- 172.21.15.83:0/2433234643 >> 172.21.15.83:0/2433234643 conn(0x7f239c06bc80 msgr2=0x7f239c073b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.138+0000 7f23a4777640 1 -- 172.21.15.83:0/2433234643 shutdown_connections 2024-09-19T07:33:23.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:23.138+0000 7f23a4777640 1 -- 172.21.15.83:0/2433234643 wait complete. 2024-09-19T07:33:23.152 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:33:25.153 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.298+0000 7f6492656640 1 Processor -- start 2024-09-19T07:33:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.298+0000 7f6492656640 1 -- start start 2024-09-19T07:33:25.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6492656640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f648c155d70 0x7f648c0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:25.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6492656640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f648c1592f0 0x7f648c157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:25.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6492656640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648c156200 con 0x7f648c0c6120 2024-09-19T07:33:25.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6492656640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f648c0c9930 con 0x7f648c1592f0 2024-09-19T07:33:25.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6492656640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f648c1564e0 con 0x7f648c155d70 2024-09-19T07:33:25.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6490bf3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f648c1592f0 0x7f648c157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:25.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6490bf3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f648c1592f0 0x7f648c157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38236/0 (socket says 172.21.15.83:38236) 2024-09-19T07:33:25.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6490bf3640 1 -- 172.21.15.83:0/1975468308 learned_addr learned my addr 172.21.15.83:0/1975468308 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:25.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6483fff640 1 --2- 172.21.15.83:0/1975468308 >> v2:172.21.15.184:3300/0 conn(0x7f648c155d70 0x7f648c0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:25.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6490bf3640 1 -- 172.21.15.83:0/1975468308 >> v2:172.21.15.184:3300/0 conn(0x7f648c155d70 msgr2=0x7f648c0c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:25.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6490bf3640 1 --2- 172.21.15.83:0/1975468308 >> v2:172.21.15.184:3300/0 conn(0x7f648c155d70 0x7f648c0c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:25.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6490bf3640 1 -- 172.21.15.83:0/1975468308 >> v1:172.21.15.184:6789/0 conn(0x7f648c0c6120 legacy=0x7f648c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:25.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6490bf3640 1 -- 172.21.15.83:0/1975468308 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c1567c0 con 0x7f648c1592f0 2024-09-19T07:33:25.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6483fff640 1 --2- 172.21.15.83:0/1975468308 >> v2:172.21.15.184:3300/0 conn(0x7f648c155d70 0x7f648c0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:25.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f6490bf3640 1 --2- 172.21.15.83:0/1975468308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c1592f0 0x7f648c157ce0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f648c06cca0 tx=0x7f648401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f52640c372c376 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:25.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f648b7fe640 1 -- 172.21.15.83:0/1975468308 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64840a5670 con 0x7f648c1592f0 2024-09-19T07:33:25.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f648b7fe640 1 -- 172.21.15.83:0/1975468308 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64840a5810 con 0x7f648c1592f0 2024-09-19T07:33:25.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.302+0000 7f648b7fe640 1 -- 172.21.15.83:0/1975468308 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64840a5be0 con 0x7f648c1592f0 2024-09-19T07:33:25.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 -- 172.21.15.83:0/1975468308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c1592f0 msgr2=0x7f648c157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:25.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 --2- 172.21.15.83:0/1975468308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c1592f0 0x7f648c157ce0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f648c06cca0 tx=0x7f648401c9b0 comp rx=0 tx=0).stop 2024-09-19T07:33:25.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 -- 172.21.15.83:0/1975468308 shutdown_connections 2024-09-19T07:33:25.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 --2- 172.21.15.83:0/1975468308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c1592f0 0x7f648c157ce0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:25.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 --2- 172.21.15.83:0/1975468308 >> v2:172.21.15.184:3300/0 conn(0x7f648c155d70 0x7f648c0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:25.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 -- 172.21.15.83:0/1975468308 >> 172.21.15.83:0/1975468308 conn(0x7f648c06bc80 msgr2=0x7f648c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:25.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 -- 172.21.15.83:0/1975468308 shutdown_connections 2024-09-19T07:33:25.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 -- 172.21.15.83:0/1975468308 wait complete. 2024-09-19T07:33:25.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 Processor -- start 2024-09-19T07:33:25.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 -- start start 2024-09-19T07:33:25.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c0c6120 0x7f648c14b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:25.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f648c1592f0 0x7f648c0c1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:25.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f648c0c1bd0 0x7f648c149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:25.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f648c153860 con 0x7f648c0c6120 2024-09-19T07:33:25.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f648c162ea0 con 0x7f648c1592f0 2024-09-19T07:33:25.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6492656640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f648c0c8650 con 0x7f648c0c1bd0 2024-09-19T07:33:25.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f648bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c0c6120 0x7f648c14b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:25.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f648bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c0c6120 0x7f648c14b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38242/0 (socket says 172.21.15.83:38242) 2024-09-19T07:33:25.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f648bfff640 1 -- 172.21.15.83:0/1281643624 learned_addr learned my addr 172.21.15.83:0/1281643624 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:25.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6483fff640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f648c1592f0 0x7f648c0c1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:25.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f6490bf3640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f648c0c1bd0 0x7f648c149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:25.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f648bfff640 1 -- 172.21.15.83:0/1281643624 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f648c0c1bd0 msgr2=0x7f648c149f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:25.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f648bfff640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f648c0c1bd0 0x7f648c149f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:25.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f648bfff640 1 -- 172.21.15.83:0/1281643624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f648c1592f0 msgr2=0x7f648c0c1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:25.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f648bfff640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f648c1592f0 0x7f648c0c1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:25.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.306+0000 7f648bfff640 1 -- 172.21.15.83:0/1281643624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c14bbc0 con 0x7f648c0c6120 2024-09-19T07:33:25.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.310+0000 7f648bfff640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c0c6120 0x7f648c14b320 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f647c00c450 tx=0x7f647c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:25.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.310+0000 7f64897fa640 1 -- 172.21.15.83:0/1281643624 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c012830 con 0x7f648c0c6120 2024-09-19T07:33:25.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.310+0000 7f64897fa640 1 -- 172.21.15.83:0/1281643624 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f647c017070 con 0x7f648c0c6120 2024-09-19T07:33:25.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.310+0000 7f64897fa640 1 -- 172.21.15.83:0/1281643624 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c012c30 con 0x7f648c0c6120 2024-09-19T07:33:25.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.310+0000 7f6492656640 1 -- 172.21.15.83:0/1281643624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648c1ef690 con 0x7f648c0c6120 2024-09-19T07:33:25.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.310+0000 7f6492656640 1 -- 172.21.15.83:0/1281643624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648c1eebb0 con 0x7f648c0c6120 2024-09-19T07:33:25.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.310+0000 7f6492656640 1 -- 172.21.15.83:0/1281643624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f648c0c6500 con 0x7f648c0c6120 2024-09-19T07:33:25.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.310+0000 7f64897fa640 1 -- 172.21.15.83:0/1281643624 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f647c01c460 con 0x7f648c0c6120 2024-09-19T07:33:25.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.310+0000 7f64897fa640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f646c0678f0 0x7f646c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:25.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.310+0000 7f64897fa640 1 -- 172.21.15.83:0/1281643624 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7f647c086450 con 0x7f648c0c6120 2024-09-19T07:33:25.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.310+0000 7f6483fff640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f646c0678f0 0x7f646c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:25.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.310+0000 7f6483fff640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f646c0678f0 0x7f646c069db0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f648c14fc10 tx=0x7f6474008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:25.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.318+0000 7f64897fa640 1 -- 172.21.15.83:0/1281643624 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f647c0565d0 con 0x7f648c0c6120 2024-09-19T07:33:25.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.570+0000 7f6492656640 1 -- 172.21.15.83:0/1281643624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f648c0c96a0 con 0x7f648c0c6120 2024-09-19T07:33:25.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f64897fa640 1 -- 172.21.15.83:0/1281643624 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+6542 (secure 0 0 0) 0x7f647c055ed0 con 0x7f648c0c6120 2024-09-19T07:33:25.567 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:25.567 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":105,"btime":"2024-09-19T07:32:44:320183+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:33:25.568 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 105 2024-09-19T07:33:25.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f6492656640 1 -- 172.21.15.83:0/1281643624 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f646c0678f0 msgr2=0x7f646c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:25.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f6492656640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f646c0678f0 0x7f646c069db0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f648c14fc10 tx=0x7f6474008040 comp rx=0 tx=0).stop 2024-09-19T07:33:25.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f6492656640 1 -- 172.21.15.83:0/1281643624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c0c6120 msgr2=0x7f648c14b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:25.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f6492656640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c0c6120 0x7f648c14b320 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f647c00c450 tx=0x7f647c00c910 comp rx=0 tx=0).stop 2024-09-19T07:33:25.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f6492656640 1 -- 172.21.15.83:0/1281643624 shutdown_connections 2024-09-19T07:33:25.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f6492656640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f646c0678f0 0x7f646c069db0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:25.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f6492656640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f648c0c1bd0 0x7f648c149f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:25.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f6492656640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f648c1592f0 0x7f648c0c1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:25.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f6492656640 1 --2- 172.21.15.83:0/1281643624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c0c6120 0x7f648c14b320 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:25.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f6492656640 1 -- 172.21.15.83:0/1281643624 >> 172.21.15.83:0/1281643624 conn(0x7f648c06bc80 msgr2=0x7f648c159700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:25.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f6492656640 1 -- 172.21.15.83:0/1281643624 shutdown_connections 2024-09-19T07:33:25.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:25.574+0000 7f6492656640 1 -- 172.21.15.83:0/1281643624 wait complete. 2024-09-19T07:33:25.588 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:33:27.589 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:27.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.738+0000 7fe7f131b640 1 Processor -- start 2024-09-19T07:33:27.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.738+0000 7fe7f131b640 1 -- start start 2024-09-19T07:33:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.738+0000 7fe7f131b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe7ec0c3a20 0x7fe7ec1594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.738+0000 7fe7f131b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec06f3f0 0x7fe7ec14c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.738+0000 7fe7f131b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7ec155ee0 con 0x7fe7ec0c4450 2024-09-19T07:33:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.738+0000 7fe7f131b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe7ec068290 con 0x7fe7ec0c3a20 2024-09-19T07:33:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.738+0000 7fe7f131b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe7ec0684e0 con 0x7fe7ec06f3f0 2024-09-19T07:33:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.738+0000 7fe7e3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe7ec0c4450 0x7fe7ec1566c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38338/0 (socket says 172.21.15.83:38338) 2024-09-19T07:33:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.738+0000 7fe7e3fff640 1 -- 172.21.15.83:0/3131363562 learned_addr learned my addr 172.21.15.83:0/3131363562 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:33:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7eaffd640 1 --2- 172.21.15.83:0/3131363562 >> v2:172.21.15.184:3300/0 conn(0x7fe7ec0c3a20 0x7fe7ec1594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7eb7fe640 1 --2- 172.21.15.83:0/3131363562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec06f3f0 0x7fe7ec14c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3345117736 0 0) 0x7fe7ec155ee0 con 0x7fe7ec0c4450 2024-09-19T07:33:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7d8003530 con 0x7fe7ec0c4450 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2823958820 0 0) 0x7fe7d8003530 con 0x7fe7ec0c4450 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec06f3f0 msgr2=0x7fe7ec14c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 --2- 172.21.15.83:0/3131363562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec06f3f0 0x7fe7ec14c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 >> v2:172.21.15.184:3300/0 conn(0x7fe7ec0c3a20 msgr2=0x7fe7ec1594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 --2- 172.21.15.83:0/3131363562 >> v2:172.21.15.184:3300/0 conn(0x7fe7ec0c3a20 0x7fe7ec1594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ec0687c0 con 0x7fe7ec0c4450 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fe7d4001680 con 0x7fe7ec0c4450 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 >> v1:172.21.15.83:6789/0 conn(0x7fe7ec0c4450 legacy=0x7fe7ec1566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 --2- 172.21.15.83:0/3131363562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7d80039c0 0x7fe7d8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 --2- 172.21.15.83:0/3131363562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d8004810 0x7fe7d8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 --2- 172.21.15.83:0/3131363562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec06f3f0 0x7fe7d80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec068290 con 0x7fe7d80039c0 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec0684e0 con 0x7fe7d8004810 2024-09-19T07:33:27.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe7d800ba00 con 0x7fe7ec06f3f0 2024-09-19T07:33:27.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7eaffd640 1 --2- 172.21.15.83:0/3131363562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7d80039c0 0x7fe7d8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:27.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7eb7fe640 1 --2- 172.21.15.83:0/3131363562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec06f3f0 0x7fe7d80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:27.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7eaffd640 1 -- 172.21.15.83:0/3131363562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec06f3f0 msgr2=0x7fe7d80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:27.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7eaffd640 1 --2- 172.21.15.83:0/3131363562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec06f3f0 0x7fe7d80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:27.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7eaffd640 1 -- 172.21.15.83:0/3131363562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d8004810 msgr2=0x7fe7d8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:27.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7eaffd640 1 --2- 172.21.15.83:0/3131363562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d8004810 0x7fe7d8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:27.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7eaffd640 1 -- 172.21.15.83:0/3131363562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d8003530 con 0x7fe7d80039c0 2024-09-19T07:33:27.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7eaffd640 1 --2- 172.21.15.83:0/3131363562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7d80039c0 0x7fe7d8003de0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc009ae0 tx=0x7fe7dc01ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44aea30f38356e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:27.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc0b2070 con 0x7fe7d80039c0 2024-09-19T07:33:27.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7dc0a5570 con 0x7fe7d80039c0 2024-09-19T07:33:27.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.742+0000 7fe7ea7fc640 1 -- 172.21.15.83:0/3131363562 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc0096b0 con 0x7fe7d80039c0 2024-09-19T07:33:27.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 -- 172.21.15.83:0/3131363562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7d80039c0 msgr2=0x7fe7d8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:27.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 --2- 172.21.15.83:0/3131363562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7d80039c0 0x7fe7d8003de0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc009ae0 tx=0x7fe7dc01ca20 comp rx=0 tx=0).stop 2024-09-19T07:33:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7eb7fe640 1 -- 172.21.15.83:0/3131363562 reap_dead start 2024-09-19T07:33:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 -- 172.21.15.83:0/3131363562 shutdown_connections 2024-09-19T07:33:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 -- 172.21.15.83:0/3131363562 >> 172.21.15.83:0/3131363562 conn(0x7fe7ec0c48b0 msgr2=0x7fe7ec0c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 -- 172.21.15.83:0/3131363562 shutdown_connections 2024-09-19T07:33:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 -- 172.21.15.83:0/3131363562 wait complete. 2024-09-19T07:33:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 Processor -- start 2024-09-19T07:33:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 -- start start 2024-09-19T07:33:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d80039c0 0x7fe7ec1ee4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec158700 0x7fe7ec0ab300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec15a620 0x7fe7ec0ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec0c3f80 con 0x7fe7ec158700 2024-09-19T07:33:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec0c3e00 con 0x7fe7d80039c0 2024-09-19T07:33:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe7ec160860 con 0x7fe7ec15a620 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7e3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec158700 0x7fe7ec0ab300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7e3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec158700 0x7fe7ec0ab300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38252/0 (socket says 172.21.15.83:38252) 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7e3fff640 1 -- 172.21.15.83:0/4049208520 learned_addr learned my addr 172.21.15.83:0/4049208520 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7eaffd640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d80039c0 0x7fe7ec1ee4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7eb7fe640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec15a620 0x7fe7ec0ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7e3fff640 1 -- 172.21.15.83:0/4049208520 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec15a620 msgr2=0x7fe7ec0ab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7e3fff640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec15a620 0x7fe7ec0ab840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7e3fff640 1 -- 172.21.15.83:0/4049208520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d80039c0 msgr2=0x7fe7ec1ee4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7e3fff640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d80039c0 0x7fe7ec1ee4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7e3fff640 1 -- 172.21.15.83:0/4049208520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ec0ad610 con 0x7fe7ec158700 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7eaffd640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d80039c0 0x7fe7ec1ee4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7eb7fe640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec15a620 0x7fe7ec0ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7e3fff640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec158700 0x7fe7ec0ab300 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400c7c0 tx=0x7fe7d400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:27.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7e37fe640 1 -- 172.21.15.83:0/4049208520 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4018070 con 0x7fe7ec158700 2024-09-19T07:33:27.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7e37fe640 1 -- 172.21.15.83:0/4049208520 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d40074e0 con 0x7fe7ec158700 2024-09-19T07:33:27.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 -- 172.21.15.83:0/4049208520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7ec1eedc0 con 0x7fe7ec158700 2024-09-19T07:33:27.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7e37fe640 1 -- 172.21.15.83:0/4049208520 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4004a10 con 0x7fe7ec158700 2024-09-19T07:33:27.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.746+0000 7fe7f131b640 1 -- 172.21.15.83:0/4049208520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7ec14ee50 con 0x7fe7ec158700 2024-09-19T07:33:27.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.750+0000 7fe7f131b640 1 -- 172.21.15.83:0/4049208520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7ec14f180 con 0x7fe7ec158700 2024-09-19T07:33:27.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.750+0000 7fe7e37fe640 1 -- 172.21.15.83:0/4049208520 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe7d401b070 con 0x7fe7ec158700 2024-09-19T07:33:27.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.750+0000 7fe7e37fe640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe7cc0678c0 0x7fe7cc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:27.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.750+0000 7fe7e37fe640 1 -- 172.21.15.83:0/4049208520 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8271+0+0 (secure 0 0 0) 0x7fe7d4085c50 con 0x7fe7ec158700 2024-09-19T07:33:27.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.750+0000 7fe7eaffd640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe7cc0678c0 0x7fe7cc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:27.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.750+0000 7fe7eaffd640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe7cc0678c0 0x7fe7cc069d80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0af000 tx=0x7fe7dc0046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:27.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.754+0000 7fe7e37fe640 1 -- 172.21.15.83:0/4049208520 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe7d408b020 con 0x7fe7ec158700 2024-09-19T07:33:27.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.982+0000 7fe7f131b640 1 -- 172.21.15.83:0/4049208520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7ec1594d0 con 0x7fe7ec158700 2024-09-19T07:33:27.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.982+0000 7fe7e37fe640 1 -- 172.21.15.83:0/4049208520 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+6542 (secure 0 0 0) 0x7fe7d4055d40 con 0x7fe7ec158700 2024-09-19T07:33:27.977 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:27.977 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":105,"btime":"2024-09-19T07:32:44:320183+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":5830,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:31:21.234481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6447},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6447":{"gid":6447,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.83:6835/1072903448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1072903448},{"type":"v1","addr":"172.21.15.83:6835","nonce":1072903448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6447,"qdb_cluster":[6447]},"id":2}]} 2024-09-19T07:33:27.977 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 105 2024-09-19T07:33:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.986+0000 7fe7f131b640 1 -- 172.21.15.83:0/4049208520 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe7cc0678c0 msgr2=0x7fe7cc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.986+0000 7fe7f131b640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe7cc0678c0 0x7fe7cc069d80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0af000 tx=0x7fe7dc0046d0 comp rx=0 tx=0).stop 2024-09-19T07:33:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.986+0000 7fe7f131b640 1 -- 172.21.15.83:0/4049208520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec158700 msgr2=0x7fe7ec0ab300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.986+0000 7fe7f131b640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec158700 0x7fe7ec0ab300 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400c7c0 tx=0x7fe7d400cc80 comp rx=0 tx=0).stop 2024-09-19T07:33:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.986+0000 7fe7f131b640 1 -- 172.21.15.83:0/4049208520 shutdown_connections 2024-09-19T07:33:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.986+0000 7fe7f131b640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe7cc0678c0 0x7fe7cc069d80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.986+0000 7fe7f131b640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7ec15a620 0x7fe7ec0ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.986+0000 7fe7f131b640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7ec158700 0x7fe7ec0ab300 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.986+0000 7fe7f131b640 1 --2- 172.21.15.83:0/4049208520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d80039c0 0x7fe7ec1ee4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.986+0000 7fe7f131b640 1 -- 172.21.15.83:0/4049208520 >> 172.21.15.83:0/4049208520 conn(0x7fe7ec0c48b0 msgr2=0x7fe7ec0c5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.986+0000 7fe7f131b640 1 -- 172.21.15.83:0/4049208520 shutdown_connections 2024-09-19T07:33:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:27.986+0000 7fe7f131b640 1 -- 172.21.15.83:0/4049208520 wait complete. 2024-09-19T07:33:27.998 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:33:29.999 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f3801640 1 Processor -- start 2024-09-19T07:33:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f3801640 1 -- start start 2024-09-19T07:33:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f3801640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47ec0c23d0 0x7f47ec0c27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f3801640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f47ec155bb0 0x7f47ec156040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f3801640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ec1593f0 con 0x7f47ec153300 2024-09-19T07:33:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f3801640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f47ec1591a0 con 0x7f47ec155bb0 2024-09-19T07:33:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f3801640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f47ec1596d0 con 0x7f47ec0c23d0 2024-09-19T07:33:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f1d9e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f47ec155bb0 0x7f47ec156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f159d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f47ec153300 0x7f47ec06cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55432/0 (socket says 172.21.15.83:55432) 2024-09-19T07:33:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f159d640 1 -- 172.21.15.83:0/2270105780 learned_addr learned my addr 172.21.15.83:0/2270105780 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:33:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f0d9c640 1 --2- 172.21.15.83:0/2270105780 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47ec0c23d0 0x7f47ec0c27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f1d9e640 1 -- 172.21.15.83:0/2270105780 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47ec0c23d0 msgr2=0x7f47ec0c27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f1d9e640 1 --2- 172.21.15.83:0/2270105780 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47ec0c23d0 0x7f47ec0c27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f1d9e640 1 -- 172.21.15.83:0/2270105780 >> v1:172.21.15.83:6789/0 conn(0x7f47ec153300 legacy=0x7f47ec06cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.142+0000 7f47f1d9e640 1 -- 172.21.15.83:0/2270105780 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ec1599b0 con 0x7f47ec155bb0 2024-09-19T07:33:30.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.146+0000 7f47f1d9e640 1 --2- 172.21.15.83:0/2270105780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47ec155bb0 0x7f47ec156040 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009d90 tx=0x7f47e401c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65732d4fc37c77c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:30.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.146+0000 7f47e3fff640 1 -- 172.21.15.83:0/2270105780 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e40b2070 con 0x7f47ec155bb0 2024-09-19T07:33:30.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.146+0000 7f47e3fff640 1 -- 172.21.15.83:0/2270105780 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47e40a5820 con 0x7f47ec155bb0 2024-09-19T07:33:30.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.146+0000 7f47e3fff640 1 -- 172.21.15.83:0/2270105780 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e40a5bf0 con 0x7f47ec155bb0 2024-09-19T07:33:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.146+0000 7f47f3801640 1 -- 172.21.15.83:0/2270105780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47ec155bb0 msgr2=0x7f47ec156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.146+0000 7f47f3801640 1 --2- 172.21.15.83:0/2270105780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47ec155bb0 0x7f47ec156040 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009d90 tx=0x7f47e401c850 comp rx=0 tx=0).stop 2024-09-19T07:33:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.146+0000 7f47f3801640 1 -- 172.21.15.83:0/2270105780 shutdown_connections 2024-09-19T07:33:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.146+0000 7f47f3801640 1 --2- 172.21.15.83:0/2270105780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47ec155bb0 0x7f47ec156040 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.146+0000 7f47f3801640 1 --2- 172.21.15.83:0/2270105780 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47ec0c23d0 0x7f47ec0c27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.146+0000 7f47f3801640 1 -- 172.21.15.83:0/2270105780 >> 172.21.15.83:0/2270105780 conn(0x7f47ec0c5090 msgr2=0x7f47ec0c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.146+0000 7f47f3801640 1 -- 172.21.15.83:0/2270105780 shutdown_connections 2024-09-19T07:33:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.146+0000 7f47f3801640 1 -- 172.21.15.83:0/2270105780 wait complete. 2024-09-19T07:33:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f3801640 1 Processor -- start 2024-09-19T07:33:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f3801640 1 -- start start 2024-09-19T07:33:30.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f3801640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47ec0c23d0 0x7f47ec149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:30.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f3801640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47ec153300 0x7f47ec1518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:30.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f3801640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47ec155bb0 0x7f47ec151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:30.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f3801640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f47ec153860 con 0x7f47ec0c23d0 2024-09-19T07:33:30.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f3801640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f47ec1536e0 con 0x7f47ec153300 2024-09-19T07:33:30.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f3801640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f47ec162d50 con 0x7f47ec155bb0 2024-09-19T07:33:30.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f159d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47ec0c23d0 0x7f47ec149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:30.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f159d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47ec0c23d0 0x7f47ec149da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55688/0 (socket says 172.21.15.83:55688) 2024-09-19T07:33:30.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f159d640 1 -- 172.21.15.83:0/2151883144 learned_addr learned my addr 172.21.15.83:0/2151883144 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:30.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f0d9c640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47ec153300 0x7f47ec1518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f1d9e640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47ec155bb0 0x7f47ec151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f159d640 1 -- 172.21.15.83:0/2151883144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47ec155bb0 msgr2=0x7f47ec151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f159d640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47ec155bb0 0x7f47ec151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f159d640 1 -- 172.21.15.83:0/2151883144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47ec153300 msgr2=0x7f47ec1518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f159d640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47ec153300 0x7f47ec1518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f159d640 1 -- 172.21.15.83:0/2151883144 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ec155410 con 0x7f47ec0c23d0 2024-09-19T07:33:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f159d640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47ec0c23d0 0x7f47ec149da0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f47e800c450 tx=0x7f47e800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47e1ffb640 1 -- 172.21.15.83:0/2151883144 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8017070 con 0x7f47ec0c23d0 2024-09-19T07:33:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47e1ffb640 1 -- 172.21.15.83:0/2151883144 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47e8005cf0 con 0x7f47ec0c23d0 2024-09-19T07:33:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f3801640 1 -- 172.21.15.83:0/2151883144 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47ec0ae6e0 con 0x7f47ec0c23d0 2024-09-19T07:33:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47e1ffb640 1 -- 172.21.15.83:0/2151883144 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8007290 con 0x7f47ec0c23d0 2024-09-19T07:33:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f3801640 1 -- 172.21.15.83:0/2151883144 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47ec1495a0 con 0x7f47ec0c23d0 2024-09-19T07:33:30.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47f3801640 1 -- 172.21.15.83:0/2151883144 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47ec1493a0 con 0x7f47ec0c23d0 2024-09-19T07:33:30.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.150+0000 7f47e1ffb640 1 -- 172.21.15.83:0/2151883144 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f47e801a070 con 0x7f47ec0c23d0 2024-09-19T07:33:30.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.154+0000 7f47e1ffb640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f47d00678c0 0x7f47d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:30.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.154+0000 7f47e1ffb640 1 -- 172.21.15.83:0/2151883144 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7f47e80859e0 con 0x7f47ec0c23d0 2024-09-19T07:33:30.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.154+0000 7f47f0d9c640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f47d00678c0 0x7f47d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:30.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.154+0000 7f47f0d9c640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f47d00678c0 0x7f47d0069d80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f47ec06f300 tx=0x7f47dc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:30.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.158+0000 7f47e1ffb640 1 -- 172.21.15.83:0/2151883144 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f47e8055b00 con 0x7f47ec0c23d0 2024-09-19T07:33:30.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 -- 172.21.15.83:0/2151883144 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47ec0c27f0 con 0x7f47ec0c23d0 2024-09-19T07:33:30.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47e1ffb640 1 -- 172.21.15.83:0/2151883144 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+5712 (secure 0 0 0) 0x7f47e8055400 con 0x7f47ec0c23d0 2024-09-19T07:33:30.427 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:30.428 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":108,"btime":"2024-09-19T07:33:30:294133+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:29.300579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:33:30.428 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 108 2024-09-19T07:33:30.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 -- 172.21.15.83:0/2151883144 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f47d00678c0 msgr2=0x7f47d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:30.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f47d00678c0 0x7f47d0069d80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f47ec06f300 tx=0x7f47dc008040 comp rx=0 tx=0).stop 2024-09-19T07:33:30.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 -- 172.21.15.83:0/2151883144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47ec0c23d0 msgr2=0x7f47ec149da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:30.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47ec0c23d0 0x7f47ec149da0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f47e800c450 tx=0x7f47e800c910 comp rx=0 tx=0).stop 2024-09-19T07:33:30.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 -- 172.21.15.83:0/2151883144 shutdown_connections 2024-09-19T07:33:30.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f47d00678c0 0x7f47d0069d80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47ec155bb0 0x7f47ec151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47ec153300 0x7f47ec1518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 --2- 172.21.15.83:0/2151883144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47ec0c23d0 0x7f47ec149da0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 -- 172.21.15.83:0/2151883144 >> 172.21.15.83:0/2151883144 conn(0x7f47ec0c5090 msgr2=0x7f47ec06cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:30.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 -- 172.21.15.83:0/2151883144 shutdown_connections 2024-09-19T07:33:30.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.434+0000 7f47f3801640 1 -- 172.21.15.83:0/2151883144 wait complete. 2024-09-19T07:33:30.449 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-19T07:33:30.449 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:33:30.450 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:30.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f0742a55640 1 Processor -- start 2024-09-19T07:33:30.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f0742a55640 1 -- start start 2024-09-19T07:33:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f0742a55640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f073c152b20 0x7f073c0c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f0742a55640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f073c0c45c0 0x7f073c153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f0742a55640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f073c150670 con 0x7f073c150290 2024-09-19T07:33:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f0742a55640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f073c0716c0 con 0x7f073c152b20 2024-09-19T07:33:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f0742a55640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f073c0c3920 con 0x7f073c0c45c0 2024-09-19T07:33:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f07337fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f073c152b20 0x7f073c0c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f07337fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f073c152b20 0x7f073c0c4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55700/0 (socket says 172.21.15.83:55700) 2024-09-19T07:33:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f07337fe640 1 -- 172.21.15.83:0/3174527123 learned_addr learned my addr 172.21.15.83:0/3174527123 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f07337fe640 1 -- 172.21.15.83:0/3174527123 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f073c0c45c0 msgr2=0x7f073c153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f07337fe640 1 --2- 172.21.15.83:0/3174527123 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f073c0c45c0 0x7f073c153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f07337fe640 1 -- 172.21.15.83:0/3174527123 >> v1:172.21.15.184:6789/0 conn(0x7f073c150290 legacy=0x7f073c0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.574+0000 7f07337fe640 1 -- 172.21.15.83:0/3174527123 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073c0c2880 con 0x7f073c152b20 2024-09-19T07:33:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f07337fe640 1 --2- 172.21.15.83:0/3174527123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f073c152b20 0x7f073c0c4080 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f072400bb40 tx=0x7f072401ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9728ab6782eaac server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f073b7fe640 1 -- 172.21.15.83:0/3174527123 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07240b4070 con 0x7f073c152b20 2024-09-19T07:33:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f073b7fe640 1 -- 172.21.15.83:0/3174527123 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07240a7a50 con 0x7f073c152b20 2024-09-19T07:33:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f073b7fe640 1 -- 172.21.15.83:0/3174527123 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07240a7e20 con 0x7f073c152b20 2024-09-19T07:33:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f0742a55640 1 -- 172.21.15.83:0/3174527123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f073c152b20 msgr2=0x7f073c0c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f0742a55640 1 --2- 172.21.15.83:0/3174527123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f073c152b20 0x7f073c0c4080 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f072400bb40 tx=0x7f072401ebd0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f0742a55640 1 -- 172.21.15.83:0/3174527123 shutdown_connections 2024-09-19T07:33:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f0742a55640 1 --2- 172.21.15.83:0/3174527123 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f073c0c45c0 0x7f073c153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f0742a55640 1 --2- 172.21.15.83:0/3174527123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f073c152b20 0x7f073c0c4080 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f0742a55640 1 -- 172.21.15.83:0/3174527123 >> 172.21.15.83:0/3174527123 conn(0x7f073c06bc80 msgr2=0x7f073c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f0742a55640 1 -- 172.21.15.83:0/3174527123 shutdown_connections 2024-09-19T07:33:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f0742a55640 1 -- 172.21.15.83:0/3174527123 wait complete. 2024-09-19T07:33:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f0742a55640 1 Processor -- start 2024-09-19T07:33:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f0742a55640 1 -- start start 2024-09-19T07:33:30.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f0742a55640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f073c0c45c0 0x7f073c0c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:30.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.578+0000 7f0742a55640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f073c150290 0x7f073c1eafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:30.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f0742a55640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f073c152b20 0x7f073c1eb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:30.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f0742a55640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f073c154980 con 0x7f073c0c45c0 2024-09-19T07:33:30.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f0742a55640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f073c154800 con 0x7f073c152b20 2024-09-19T07:33:30.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f0742a55640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f073c06e6b0 con 0x7f073c150290 2024-09-19T07:33:30.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f073bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f073c0c45c0 0x7f073c0c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:30.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f073bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f073c0c45c0 0x7f073c0c6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55708/0 (socket says 172.21.15.83:55708) 2024-09-19T07:33:30.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f073bfff640 1 -- 172.21.15.83:0/2319446055 learned_addr learned my addr 172.21.15.83:0/2319446055 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:30.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f0740ff2640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f073c152b20 0x7f073c1eb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:30.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f073bfff640 1 -- 172.21.15.83:0/2319446055 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f073c150290 msgr2=0x7f073c1eafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:30.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f07337fe640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f073c150290 0x7f073c1eafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:30.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f073bfff640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f073c150290 0x7f073c1eafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f073bfff640 1 -- 172.21.15.83:0/2319446055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f073c152b20 msgr2=0x7f073c1eb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:30.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f073bfff640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f073c152b20 0x7f073c1eb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f073bfff640 1 -- 172.21.15.83:0/2319446055 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073c14ec40 con 0x7f073c0c45c0 2024-09-19T07:33:30.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f0740ff2640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f073c152b20 0x7f073c1eb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:30.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f073bfff640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f073c0c45c0 0x7f073c0c6bb0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f072c00c450 tx=0x7f072c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:30.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f07397fa640 1 -- 172.21.15.83:0/2319446055 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c012830 con 0x7f073c0c45c0 2024-09-19T07:33:30.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f07397fa640 1 -- 172.21.15.83:0/2319446055 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f072c017070 con 0x7f073c0c45c0 2024-09-19T07:33:30.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f0742a55640 1 -- 172.21.15.83:0/2319446055 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073c1eed90 con 0x7f073c0c45c0 2024-09-19T07:33:30.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f0742a55640 1 -- 172.21.15.83:0/2319446055 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f073c152580 con 0x7f073c0c45c0 2024-09-19T07:33:30.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f07397fa640 1 -- 172.21.15.83:0/2319446055 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c012c30 con 0x7f073c0c45c0 2024-09-19T07:33:30.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f0742a55640 1 -- 172.21.15.83:0/2319446055 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f073c06ea70 con 0x7f073c0c45c0 2024-09-19T07:33:30.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f07397fa640 1 -- 172.21.15.83:0/2319446055 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f072c01c460 con 0x7f073c0c45c0 2024-09-19T07:33:30.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f07397fa640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f071c0678f0 0x7f071c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:30.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f07397fa640 1 -- 172.21.15.83:0/2319446055 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7f072c0864b0 con 0x7f073c0c45c0 2024-09-19T07:33:30.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f07337fe640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f071c0678f0 0x7f071c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:30.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.582+0000 7f07337fe640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f071c0678f0 0x7f071c069db0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f07240b1000 tx=0x7f07240a9070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:30.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.590+0000 7f07397fa640 1 -- 172.21.15.83:0/2319446055 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f072c0565d0 con 0x7f073c0c45c0 2024-09-19T07:33:30.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 -- 172.21.15.83:0/2319446055 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f073c14f7c0 con 0x7f073c0c45c0 2024-09-19T07:33:30.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f07397fa640 1 -- 172.21.15.83:0/2319446055 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+5712 (secure 0 0 0) 0x7f072c055ed0 con 0x7f073c0c45c0 2024-09-19T07:33:30.828 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:30.828 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":108,"btime":"2024-09-19T07:33:30:294133+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:29.300579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:33:30.828 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 108 2024-09-19T07:33:30.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 -- 172.21.15.83:0/2319446055 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f071c0678f0 msgr2=0x7f071c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:30.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f071c0678f0 0x7f071c069db0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f07240b1000 tx=0x7f07240a9070 comp rx=0 tx=0).stop 2024-09-19T07:33:30.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 -- 172.21.15.83:0/2319446055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f073c0c45c0 msgr2=0x7f073c0c6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:30.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f073c0c45c0 0x7f073c0c6bb0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f072c00c450 tx=0x7f072c00c910 comp rx=0 tx=0).stop 2024-09-19T07:33:30.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 -- 172.21.15.83:0/2319446055 shutdown_connections 2024-09-19T07:33:30.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f071c0678f0 0x7f071c069db0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f073c152b20 0x7f073c1eb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f073c150290 0x7f073c1eafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 --2- 172.21.15.83:0/2319446055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f073c0c45c0 0x7f073c0c6bb0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:30.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 -- 172.21.15.83:0/2319446055 >> 172.21.15.83:0/2319446055 conn(0x7f073c06bc80 msgr2=0x7f073c152fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:30.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 -- 172.21.15.83:0/2319446055 shutdown_connections 2024-09-19T07:33:30.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:30.834+0000 7f0742a55640 1 -- 172.21.15.83:0/2319446055 wait complete. 2024-09-19T07:33:30.846 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:33:30:294133+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 104, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6561": { "addr": "172.21.15.83:6837/1203406606", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 1203406606, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 1203406606, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6561, "incarnation": 101, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:32:41.308698+0000", "qdb_cluster": [ 6561 ], "qdb_leader": 6561, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6561 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5830": { "addr": "172.21.15.184:6833/2898687304", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2898687304, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2898687304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5830, "incarnation": 106, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:33:29.300579+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5830 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/2112478738", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 2112478738, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 2112478738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 95, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5722, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:33:32.847 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:32.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.982+0000 7f5abf4d3640 1 Processor -- start 2024-09-19T07:33:32.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.982+0000 7f5abf4d3640 1 -- start start 2024-09-19T07:33:32.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.982+0000 7f5abf4d3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5ab80c6c90 0x7f5ab80c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:32.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.982+0000 7f5abf4d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ab806d610 0x7f5ab8155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:32.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.982+0000 7f5abf4d3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ab80c24c0 con 0x7f5ab806da50 2024-09-19T07:33:32.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.982+0000 7f5abf4d3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5ab80c2ad0 con 0x7f5ab80c6c90 2024-09-19T07:33:32.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.982+0000 7f5abf4d3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab8156940 con 0x7f5ab806d610 2024-09-19T07:33:32.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.982+0000 7f5abe4d1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5ab80c6c90 0x7f5ab80c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:32.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.982+0000 7f5abe4d1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5ab80c6c90 0x7f5ab80c2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55718/0 (socket says 172.21.15.83:55718) 2024-09-19T07:33:32.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.982+0000 7f5abe4d1640 1 -- 172.21.15.83:0/3834767215 learned_addr learned my addr 172.21.15.83:0/3834767215 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:32.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.986+0000 7f5abdcd0640 1 --2- 172.21.15.83:0/3834767215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ab806d610 0x7f5ab8155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:32.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.986+0000 7f5abe4d1640 1 -- 172.21.15.83:0/3834767215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ab806d610 msgr2=0x7f5ab8155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:32.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.986+0000 7f5abe4d1640 1 --2- 172.21.15.83:0/3834767215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ab806d610 0x7f5ab8155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:32.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.986+0000 7f5abe4d1640 1 -- 172.21.15.83:0/3834767215 >> v1:172.21.15.184:6789/0 conn(0x7f5ab806da50 legacy=0x7f5ab814fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:32.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.986+0000 7f5abe4d1640 1 -- 172.21.15.83:0/3834767215 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab80c2760 con 0x7f5ab80c6c90 2024-09-19T07:33:32.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.986+0000 7f5abe4d1640 1 --2- 172.21.15.83:0/3834767215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ab80c6c90 0x7f5ab80c2f60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009870 tx=0x7f5ab401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3729ce365a9817 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:32.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.986+0000 7f5abd4cf640 1 -- 172.21.15.83:0/3834767215 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40aa650 con 0x7f5ab80c6c90 2024-09-19T07:33:32.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.986+0000 7f5abd4cf640 1 -- 172.21.15.83:0/3834767215 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab40af070 con 0x7f5ab80c6c90 2024-09-19T07:33:32.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.986+0000 7f5abd4cf640 1 -- 172.21.15.83:0/3834767215 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40aaa10 con 0x7f5ab80c6c90 2024-09-19T07:33:32.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3834767215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ab80c6c90 msgr2=0x7f5ab80c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:32.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 --2- 172.21.15.83:0/3834767215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ab80c6c90 0x7f5ab80c2f60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009870 tx=0x7f5ab401c4d0 comp rx=0 tx=0).stop 2024-09-19T07:33:32.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3834767215 shutdown_connections 2024-09-19T07:33:32.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 --2- 172.21.15.83:0/3834767215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ab806d610 0x7f5ab8155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:32.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 --2- 172.21.15.83:0/3834767215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ab80c6c90 0x7f5ab80c2f60 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:32.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3834767215 >> 172.21.15.83:0/3834767215 conn(0x7f5ab80c4520 msgr2=0x7f5ab80c4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:32.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3834767215 shutdown_connections 2024-09-19T07:33:32.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3834767215 wait complete. 2024-09-19T07:33:32.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 Processor -- start 2024-09-19T07:33:32.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 -- start start 2024-09-19T07:33:32.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ab806d610 0x7f5ab80c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:32.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ab806da50 0x7f5ab815f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:32.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ab80c6c90 0x7f5ab814bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:32.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab815d8f0 con 0x7f5ab806d610 2024-09-19T07:33:32.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab80c72c0 con 0x7f5ab80c6c90 2024-09-19T07:33:32.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abf4d3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab815da70 con 0x7f5ab806da50 2024-09-19T07:33:32.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abe4d1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ab806d610 0x7f5ab80c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:32.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abe4d1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ab806d610 0x7f5ab80c6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55732/0 (socket says 172.21.15.83:55732) 2024-09-19T07:33:32.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abe4d1640 1 -- 172.21.15.83:0/3274777942 learned_addr learned my addr 172.21.15.83:0/3274777942 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:32.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abecd2640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ab80c6c90 0x7f5ab814bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:32.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.990+0000 7f5abdcd0640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ab806da50 0x7f5ab815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:32.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5abe4d1640 1 -- 172.21.15.83:0/3274777942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ab806da50 msgr2=0x7f5ab815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:32.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5abe4d1640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ab806da50 0x7f5ab815f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:32.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5abe4d1640 1 -- 172.21.15.83:0/3274777942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ab80c6c90 msgr2=0x7f5ab814bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:32.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5abe4d1640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ab80c6c90 0x7f5ab814bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:32.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5abe4d1640 1 -- 172.21.15.83:0/3274777942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab814dc40 con 0x7f5ab806d610 2024-09-19T07:33:32.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5abecd2640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ab80c6c90 0x7f5ab814bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:33:32.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5abe4d1640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ab806d610 0x7f5ab80c6030 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40ac040 tx=0x7f5ab401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:32.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5aaeffd640 1 -- 172.21.15.83:0/3274777942 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40bd5f0 con 0x7f5ab806d610 2024-09-19T07:33:32.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5aaeffd640 1 -- 172.21.15.83:0/3274777942 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab40a6040 con 0x7f5ab806d610 2024-09-19T07:33:32.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3274777942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab815ee40 con 0x7f5ab806d610 2024-09-19T07:33:32.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5aaeffd640 1 -- 172.21.15.83:0/3274777942 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40bd9d0 con 0x7f5ab806d610 2024-09-19T07:33:32.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3274777942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab815f400 con 0x7f5ab806d610 2024-09-19T07:33:32.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3274777942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab815deb0 con 0x7f5ab806d610 2024-09-19T07:33:32.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5aaeffd640 1 -- 172.21.15.83:0/3274777942 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5ab40c6440 con 0x7f5ab806d610 2024-09-19T07:33:32.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5aaeffd640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a9c0678c0 0x7f5a9c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:32.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5aaeffd640 1 -- 172.21.15.83:0/3274777942 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7f5ab4120d70 con 0x7f5ab806d610 2024-09-19T07:33:32.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5abdcd0640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a9c0678c0 0x7f5a9c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:32.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:32.994+0000 7f5abdcd0640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a9c0678c0 0x7f5a9c069d80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80adbd0 tx=0x7f5aa8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:32.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.002+0000 7f5aaeffd640 1 -- 172.21.15.83:0/3274777942 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5ab4016150 con 0x7f5ab806d610 2024-09-19T07:33:33.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3274777942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ab815f640 con 0x7f5ab806d610 2024-09-19T07:33:33.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5aaeffd640 1 -- 172.21.15.83:0/3274777942 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+5719 (secure 0 0 0) 0x7f5ab40f0fb0 con 0x7f5ab806d610 2024-09-19T07:33:33.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:33.236 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":109,"btime":"2024-09-19T07:33:31:301764+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:33:33.236 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 109 2024-09-19T07:33:33.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3274777942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a9c0678c0 msgr2=0x7f5a9c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:33.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a9c0678c0 0x7f5a9c069d80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80adbd0 tx=0x7f5aa8008040 comp rx=0 tx=0).stop 2024-09-19T07:33:33.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3274777942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ab806d610 msgr2=0x7f5ab80c6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:33.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ab806d610 0x7f5ab80c6030 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40ac040 tx=0x7f5ab401ad10 comp rx=0 tx=0).stop 2024-09-19T07:33:33.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3274777942 shutdown_connections 2024-09-19T07:33:33.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a9c0678c0 0x7f5a9c069d80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ab80c6c90 0x7f5ab814bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ab806da50 0x7f5ab815f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 --2- 172.21.15.83:0/3274777942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ab806d610 0x7f5ab80c6030 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3274777942 >> 172.21.15.83:0/3274777942 conn(0x7f5ab80c4520 msgr2=0x7f5ab8155970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:33.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3274777942 shutdown_connections 2024-09-19T07:33:33.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.242+0000 7f5abf4d3640 1 -- 172.21.15.83:0/3274777942 wait complete. 2024-09-19T07:33:33.255 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=6447 2024-09-19T07:33:33.255 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.a 2024-09-19T07:33:33.256 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-19T07:33:33.256 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:33:33.256 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:33:33.258 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:33:33.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:33.321 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:33:33.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd7779640 1 Processor -- start 2024-09-19T07:33:33.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd7779640 1 -- start start 2024-09-19T07:33:33.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd7779640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fdfd00c7d70 0x7fdfd0157c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd7779640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfd01547b0 0x7fdfd00c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd7779640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfd0073690 con 0x7fdfd0072d00 2024-09-19T07:33:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd7779640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fdfd0157160 con 0x7fdfd00c7d70 2024-09-19T07:33:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd7779640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdfd0157440 con 0x7fdfd01547b0 2024-09-19T07:33:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd4d14640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfd01547b0 0x7fdfd00c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd5515640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fdfd00c7d70 0x7fdfd0157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd5d16640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fdfd0072d00 0x7fdfd006cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:38870/0 (socket says 172.21.15.83:38870) 2024-09-19T07:33:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd5d16640 1 -- 172.21.15.83:0/2995967426 learned_addr learned my addr 172.21.15.83:0/2995967426 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:33:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfc7fff640 1 -- 172.21.15.83:0/2995967426 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 922613452 0 0) 0x7fdfd0073690 con 0x7fdfd0072d00 2024-09-19T07:33:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfc7fff640 1 -- 172.21.15.83:0/2995967426 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfbc003530 con 0x7fdfd0072d00 2024-09-19T07:33:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd4d14640 1 -- 172.21.15.83:0/2995967426 >> v2:172.21.15.184:3300/0 conn(0x7fdfd00c7d70 msgr2=0x7fdfd0157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd4d14640 1 --2- 172.21.15.83:0/2995967426 >> v2:172.21.15.184:3300/0 conn(0x7fdfd00c7d70 0x7fdfd0157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd4d14640 1 -- 172.21.15.83:0/2995967426 >> v1:172.21.15.184:6789/0 conn(0x7fdfd0072d00 legacy=0x7fdfd006cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd4d14640 1 -- 172.21.15.83:0/2995967426 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd0157720 con 0x7fdfd01547b0 2024-09-19T07:33:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.390+0000 7fdfd5515640 1 --2- 172.21.15.83:0/2995967426 >> v2:172.21.15.184:3300/0 conn(0x7fdfd00c7d70 0x7fdfd0157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:33:33.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.394+0000 7fdfd4d14640 1 --2- 172.21.15.83:0/2995967426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfd01547b0 0x7fdfd00c6de0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00097c0 tx=0x7fdfc001c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4cd0565ad098822 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:33.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.394+0000 7fdfc7fff640 1 -- 172.21.15.83:0/2995967426 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc00b0070 con 0x7fdfd01547b0 2024-09-19T07:33:33.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.394+0000 7fdfc7fff640 1 -- 172.21.15.83:0/2995967426 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc00ab600 con 0x7fdfd01547b0 2024-09-19T07:33:33.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.394+0000 7fdfc7fff640 1 -- 172.21.15.83:0/2995967426 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc00ab9d0 con 0x7fdfd01547b0 2024-09-19T07:33:33.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.394+0000 7fdfd7779640 1 -- 172.21.15.83:0/2995967426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfd01547b0 msgr2=0x7fdfd00c6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:33.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.394+0000 7fdfd7779640 1 --2- 172.21.15.83:0/2995967426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfd01547b0 0x7fdfd00c6de0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00097c0 tx=0x7fdfc001c350 comp rx=0 tx=0).stop 2024-09-19T07:33:33.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.394+0000 7fdfd7779640 1 -- 172.21.15.83:0/2995967426 shutdown_connections 2024-09-19T07:33:33.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.394+0000 7fdfd7779640 1 --2- 172.21.15.83:0/2995967426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfd01547b0 0x7fdfd00c6de0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.394+0000 7fdfd7779640 1 --2- 172.21.15.83:0/2995967426 >> v2:172.21.15.184:3300/0 conn(0x7fdfd00c7d70 0x7fdfd0157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.394+0000 7fdfd7779640 1 -- 172.21.15.83:0/2995967426 >> 172.21.15.83:0/2995967426 conn(0x7fdfd00c5060 msgr2=0x7fdfd00c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:33.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.394+0000 7fdfd7779640 1 -- 172.21.15.83:0/2995967426 shutdown_connections 2024-09-19T07:33:33.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.394+0000 7fdfd7779640 1 -- 172.21.15.83:0/2995967426 wait complete. 2024-09-19T07:33:33.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd7779640 1 Processor -- start 2024-09-19T07:33:33.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd7779640 1 -- start start 2024-09-19T07:33:33.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd7779640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfd0072d00 0x7fdfd01517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:33.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd7779640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd00c7d70 0x7fdfd014fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:33.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd7779640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfd01547b0 0x7fdfd01ed220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:33.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd7779640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd00c74a0 con 0x7fdfd00c7d70 2024-09-19T07:33:33.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd7779640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd00c7320 con 0x7fdfd01547b0 2024-09-19T07:33:33.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd7779640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdfd0070230 con 0x7fdfd0072d00 2024-09-19T07:33:33.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd4d14640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd00c7d70 0x7fdfd014fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:33.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd4d14640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd00c7d70 0x7fdfd014fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55772/0 (socket says 172.21.15.83:55772) 2024-09-19T07:33:33.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd4d14640 1 -- 172.21.15.83:0/1449384897 learned_addr learned my addr 172.21.15.83:0/1449384897 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:33.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd5515640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfd0072d00 0x7fdfd01517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:33.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd5d16640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfd01547b0 0x7fdfd01ed220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:33.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd4d14640 1 -- 172.21.15.83:0/1449384897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfd0072d00 msgr2=0x7fdfd01517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:33.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd4d14640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfd0072d00 0x7fdfd01517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd4d14640 1 -- 172.21.15.83:0/1449384897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfd01547b0 msgr2=0x7fdfd01ed220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:33.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd4d14640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfd01547b0 0x7fdfd01ed220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd4d14640 1 -- 172.21.15.83:0/1449384897 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd01529f0 con 0x7fdfd00c7d70 2024-09-19T07:33:33.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd4d14640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd00c7d70 0x7fdfd014fcd0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0019f80 tx=0x7fdfc001ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:33.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfc5ffb640 1 -- 172.21.15.83:0/1449384897 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc00b0050 con 0x7fdfd00c7d70 2024-09-19T07:33:33.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfc5ffb640 1 -- 172.21.15.83:0/1449384897 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc00bc070 con 0x7fdfd00c7d70 2024-09-19T07:33:33.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd7779640 1 -- 172.21.15.83:0/1449384897 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd0152cd0 con 0x7fdfd00c7d70 2024-09-19T07:33:33.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfc5ffb640 1 -- 172.21.15.83:0/1449384897 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0006e20 con 0x7fdfd00c7d70 2024-09-19T07:33:33.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.398+0000 7fdfd7779640 1 -- 172.21.15.83:0/1449384897 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd01eca20 con 0x7fdfd00c7d70 2024-09-19T07:33:33.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.402+0000 7fdfd7779640 1 -- 172.21.15.83:0/1449384897 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd0070640 con 0x7fdfd00c7d70 2024-09-19T07:33:33.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.402+0000 7fdfc5ffb640 1 -- 172.21.15.83:0/1449384897 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdfc00b5830 con 0x7fdfd00c7d70 2024-09-19T07:33:33.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.402+0000 7fdfc5ffb640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdfb40678c0 0x7fdfb4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:33.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.402+0000 7fdfc5ffb640 1 -- 172.21.15.83:0/1449384897 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7fdfc0120c10 con 0x7fdfd00c7d70 2024-09-19T07:33:33.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.402+0000 7fdfd5515640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdfb40678c0 0x7fdfb4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:33.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.402+0000 7fdfd5515640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdfb40678c0 0x7fdfb4069d80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdfd015eb30 tx=0x7fdfcc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:33.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.406+0000 7fdfc5ffb640 1 -- 172.21.15.83:0/1449384897 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdfc0016090 con 0x7fdfd00c7d70 2024-09-19T07:33:33.456 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:33:33.459 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:33:33.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.626+0000 7fdfd7779640 1 -- 172.21.15.83:0/1449384897 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfd015e220 con 0x7fdfd00c7d70 2024-09-19T07:33:33.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.626+0000 7fdfc5ffb640 1 -- 172.21.15.83:0/1449384897 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+5719 (secure 0 0 0) 0x7fdfc00f06e0 con 0x7fdfd00c7d70 2024-09-19T07:33:33.622 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:33.623 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":109,"btime":"2024-09-19T07:33:31:301764+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:33:33.623 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 109 2024-09-19T07:33:33.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.630+0000 7fdfd7779640 1 -- 172.21.15.83:0/1449384897 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdfb40678c0 msgr2=0x7fdfb4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:33.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.630+0000 7fdfd7779640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdfb40678c0 0x7fdfb4069d80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdfd015eb30 tx=0x7fdfcc009190 comp rx=0 tx=0).stop 2024-09-19T07:33:33.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.630+0000 7fdfd7779640 1 -- 172.21.15.83:0/1449384897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd00c7d70 msgr2=0x7fdfd014fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:33.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.630+0000 7fdfd7779640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd00c7d70 0x7fdfd014fcd0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0019f80 tx=0x7fdfc001ab90 comp rx=0 tx=0).stop 2024-09-19T07:33:33.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.630+0000 7fdfd7779640 1 -- 172.21.15.83:0/1449384897 shutdown_connections 2024-09-19T07:33:33.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.630+0000 7fdfd7779640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdfb40678c0 0x7fdfb4069d80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.630+0000 7fdfd7779640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfd01547b0 0x7fdfd01ed220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.630+0000 7fdfd7779640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd00c7d70 0x7fdfd014fcd0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.630+0000 7fdfd7779640 1 --2- 172.21.15.83:0/1449384897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfd0072d00 0x7fdfd01517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:33.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.630+0000 7fdfd7779640 1 -- 172.21.15.83:0/1449384897 >> 172.21.15.83:0/1449384897 conn(0x7fdfd00c5060 msgr2=0x7fdfd015ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:33.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.630+0000 7fdfd7779640 1 -- 172.21.15.83:0/1449384897 shutdown_connections 2024-09-19T07:33:33.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:33.630+0000 7fdfd7779640 1 -- 172.21.15.83:0/1449384897 wait complete. 2024-09-19T07:33:33.648 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-19T07:33:34.325 INFO:tasks.ceph.mds.d.smithi184.stderr:2024-09-19T07:33:34.330+0000 7f845a1ef640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:33:35.649 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a774fc640 1 Processor -- start 2024-09-19T07:33:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a774fc640 1 -- start start 2024-09-19T07:33:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a774fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5a700c4010 0x7f5a700c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a774fc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a7014df80 con 0x7f5a7014dba0 2024-09-19T07:33:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a774fc640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a700c9410 con 0x7f5a7006de90 2024-09-19T07:33:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a774fc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5a700c7510 con 0x7f5a700c4010 2024-09-19T07:33:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a75298640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5a7014dba0 0x7f5a70154540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55436/0 (socket says 172.21.15.83:55436) 2024-09-19T07:33:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a75298640 1 -- 172.21.15.83:0/2855243662 learned_addr learned my addr 172.21.15.83:0/2855243662 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:33:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a75a99640 1 --2- 172.21.15.83:0/2855243662 >> v2:172.21.15.83:3300/0 conn(0x7f5a700c4010 0x7f5a700c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a67fff640 1 -- 172.21.15.83:0/2855243662 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231457265 0 0) 0x7f5a7014df80 con 0x7f5a7014dba0 2024-09-19T07:33:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a67fff640 1 -- 172.21.15.83:0/2855243662 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a5c003530 con 0x7f5a7014dba0 2024-09-19T07:33:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a75a99640 1 -- 172.21.15.83:0/2855243662 >> v1:172.21.15.184:6789/0 conn(0x7f5a7006de90 legacy=0x7f5a700c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a75a99640 1 -- 172.21.15.83:0/2855243662 >> v1:172.21.15.83:6789/0 conn(0x7f5a7014dba0 legacy=0x7f5a70154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.794+0000 7f5a75a99640 1 -- 172.21.15.83:0/2855243662 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a700c7740 con 0x7f5a700c4010 2024-09-19T07:33:35.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.798+0000 7f5a75a99640 1 --2- 172.21.15.83:0/2855243662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a700c4010 0x7f5a700c44a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009d90 tx=0x7f5a6801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819a6d127c764f55 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:35.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.798+0000 7f5a67fff640 1 -- 172.21.15.83:0/2855243662 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a680b2070 con 0x7f5a700c4010 2024-09-19T07:33:35.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.798+0000 7f5a67fff640 1 -- 172.21.15.83:0/2855243662 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a680a58b0 con 0x7f5a700c4010 2024-09-19T07:33:35.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.798+0000 7f5a67fff640 1 -- 172.21.15.83:0/2855243662 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a680a5c80 con 0x7f5a700c4010 2024-09-19T07:33:35.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.798+0000 7f5a774fc640 1 -- 172.21.15.83:0/2855243662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a700c4010 msgr2=0x7f5a700c44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:35.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.798+0000 7f5a774fc640 1 --2- 172.21.15.83:0/2855243662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a700c4010 0x7f5a700c44a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009d90 tx=0x7f5a6801c8e0 comp rx=0 tx=0).stop 2024-09-19T07:33:35.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.798+0000 7f5a774fc640 1 -- 172.21.15.83:0/2855243662 shutdown_connections 2024-09-19T07:33:35.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.798+0000 7f5a774fc640 1 --2- 172.21.15.83:0/2855243662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a700c4010 0x7f5a700c44a0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:35.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.798+0000 7f5a774fc640 1 -- 172.21.15.83:0/2855243662 >> 172.21.15.83:0/2855243662 conn(0x7f5a7006ff50 msgr2=0x7f5a70070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:35.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.798+0000 7f5a774fc640 1 -- 172.21.15.83:0/2855243662 shutdown_connections 2024-09-19T07:33:35.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.798+0000 7f5a774fc640 1 -- 172.21.15.83:0/2855243662 wait complete. 2024-09-19T07:33:35.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a774fc640 1 Processor -- start 2024-09-19T07:33:35.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a774fc640 1 -- start start 2024-09-19T07:33:35.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a774fc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a7006de90 0x7f5a701421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:35.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a774fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a700c4010 0x7f5a701505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:35.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a774fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a7014dba0 0x7f5a70150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:35.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a774fc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700c3bb0 con 0x7f5a700c4010 2024-09-19T07:33:35.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a774fc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700c8870 con 0x7f5a7014dba0 2024-09-19T07:33:35.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a774fc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5a700c3d30 con 0x7f5a7006de90 2024-09-19T07:33:35.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a74a97640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a700c4010 0x7f5a701505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:35.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a74a97640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a700c4010 0x7f5a701505c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55784/0 (socket says 172.21.15.83:55784) 2024-09-19T07:33:35.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a74a97640 1 -- 172.21.15.83:0/1796921367 learned_addr learned my addr 172.21.15.83:0/1796921367 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:35.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a75a99640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a7014dba0 0x7f5a70150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:35.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a75298640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a7006de90 0x7f5a701421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:35.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a74a97640 1 -- 172.21.15.83:0/1796921367 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a7006de90 msgr2=0x7f5a701421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:35.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a74a97640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a7006de90 0x7f5a701421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:35.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a74a97640 1 -- 172.21.15.83:0/1796921367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a7014dba0 msgr2=0x7f5a70150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:35.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a74a97640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a7014dba0 0x7f5a70150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:35.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a74a97640 1 -- 172.21.15.83:0/1796921367 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a70151040 con 0x7f5a700c4010 2024-09-19T07:33:35.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a75298640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a7006de90 0x7f5a701421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:33:35.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a74a97640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a700c4010 0x7f5a701505c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000c3c0 tx=0x7f5a6000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:35.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a65ffb640 1 -- 172.21.15.83:0/1796921367 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60017070 con 0x7f5a700c4010 2024-09-19T07:33:35.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a65ffb640 1 -- 172.21.15.83:0/1796921367 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a600052d0 con 0x7f5a700c4010 2024-09-19T07:33:35.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a774fc640 1 -- 172.21.15.83:0/1796921367 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a70145a70 con 0x7f5a700c4010 2024-09-19T07:33:35.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a65ffb640 1 -- 172.21.15.83:0/1796921367 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60003840 con 0x7f5a700c4010 2024-09-19T07:33:35.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a774fc640 1 -- 172.21.15.83:0/1796921367 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a701419b0 con 0x7f5a700c4010 2024-09-19T07:33:35.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.802+0000 7f5a774fc640 1 -- 172.21.15.83:0/1796921367 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a700c44a0 con 0x7f5a700c4010 2024-09-19T07:33:35.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.806+0000 7f5a65ffb640 1 -- 172.21.15.83:0/1796921367 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5a6001a070 con 0x7f5a700c4010 2024-09-19T07:33:35.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.806+0000 7f5a65ffb640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a54067910 0x7f5a54069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:35.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.806+0000 7f5a65ffb640 1 -- 172.21.15.83:0/1796921367 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7f5a600859a0 con 0x7f5a700c4010 2024-09-19T07:33:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.806+0000 7f5a75298640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a54067910 0x7f5a54069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.806+0000 7f5a75298640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a54067910 0x7f5a54069dd0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0011f0 tx=0x7f5a6c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:35.810+0000 7f5a65ffb640 1 -- 172.21.15.83:0/1796921367 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5a70151040 con 0x7f5a700c4010 2024-09-19T07:33:36.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.034+0000 7f5a774fc640 1 -- 172.21.15.83:0/1796921367 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a7006fc00 con 0x7f5a700c4010 2024-09-19T07:33:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a65ffb640 1 -- 172.21.15.83:0/1796921367 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7f5a60005480 con 0x7f5a700c4010 2024-09-19T07:33:36.031 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:36.031 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:33:36.032 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:33:36.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a774fc640 1 -- 172.21.15.83:0/1796921367 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a54067910 msgr2=0x7f5a54069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:36.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a774fc640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a54067910 0x7f5a54069dd0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0011f0 tx=0x7f5a6c009040 comp rx=0 tx=0).stop 2024-09-19T07:33:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a774fc640 1 -- 172.21.15.83:0/1796921367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a700c4010 msgr2=0x7f5a701505c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a774fc640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a700c4010 0x7f5a701505c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000c3c0 tx=0x7f5a6000c880 comp rx=0 tx=0).stop 2024-09-19T07:33:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a774fc640 1 -- 172.21.15.83:0/1796921367 shutdown_connections 2024-09-19T07:33:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a774fc640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a54067910 0x7f5a54069dd0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a774fc640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a7014dba0 0x7f5a70150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a774fc640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a700c4010 0x7f5a701505c0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a774fc640 1 --2- 172.21.15.83:0/1796921367 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a7006de90 0x7f5a701421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a774fc640 1 -- 172.21.15.83:0/1796921367 >> 172.21.15.83:0/1796921367 conn(0x7f5a7006ff50 msgr2=0x7f5a701476a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a774fc640 1 -- 172.21.15.83:0/1796921367 shutdown_connections 2024-09-19T07:33:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:36.038+0000 7f5a774fc640 1 -- 172.21.15.83:0/1796921367 wait complete. 2024-09-19T07:33:36.053 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-19T07:33:36.053 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2024-09-19T07:33:48.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:48.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee2a783640 1 Processor -- start 2024-09-19T07:33:48.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee2a783640 1 -- start start 2024-09-19T07:33:48.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee2a783640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee241546a0 0x7fee240c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:48.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee2a783640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fee241573a0 0x7fee24157850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:48.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee2a783640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee24153a50 con 0x7fee24074000 2024-09-19T07:33:48.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee2a783640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fee240730e0 con 0x7fee241573a0 2024-09-19T07:33:48.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee2a783640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fee240733c0 con 0x7fee241546a0 2024-09-19T07:33:48.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee237fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fee24074000 0x7fee2406cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48256/0 (socket says 172.21.15.83:48256) 2024-09-19T07:33:48.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee237fe640 1 -- 172.21.15.83:0/520929922 learned_addr learned my addr 172.21.15.83:0/520929922 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:33:48.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee28d20640 1 --2- 172.21.15.83:0/520929922 >> v2:172.21.15.184:3300/0 conn(0x7fee241573a0 0x7fee24157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:48.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee23fff640 1 --2- 172.21.15.83:0/520929922 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee241546a0 0x7fee240c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:48.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1391754663 0 0) 0x7fee24153a50 con 0x7fee24074000 2024-09-19T07:33:48.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee0c003530 con 0x7fee24074000 2024-09-19T07:33:48.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1046306983 0 0) 0x7fee0c003530 con 0x7fee24074000 2024-09-19T07:33:48.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee241546a0 msgr2=0x7fee240c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:48.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 --2- 172.21.15.83:0/520929922 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee241546a0 0x7fee240c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:48.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 >> v2:172.21.15.184:3300/0 conn(0x7fee241573a0 msgr2=0x7fee24157850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:48.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 --2- 172.21.15.83:0/520929922 >> v2:172.21.15.184:3300/0 conn(0x7fee241573a0 0x7fee24157850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:48.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee240736a0 con 0x7fee24074000 2024-09-19T07:33:48.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fee18001680 con 0x7fee24074000 2024-09-19T07:33:48.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 >> v1:172.21.15.83:6789/0 conn(0x7fee24074000 legacy=0x7fee2406cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:48.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 --2- 172.21.15.83:0/520929922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee0c0039c0 0x7fee0c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:48.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 --2- 172.21.15.83:0/520929922 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee241546a0 0x7fee0c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:48.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 --2- 172.21.15.83:0/520929922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee0c004d50 0x7fee0c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:48.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee240730e0 con 0x7fee0c0039c0 2024-09-19T07:33:48.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fee240733c0 con 0x7fee0c004d50 2024-09-19T07:33:48.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fee0c00b9d0 con 0x7fee241546a0 2024-09-19T07:33:48.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee23fff640 1 --2- 172.21.15.83:0/520929922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee0c0039c0 0x7fee0c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:48.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee23fff640 1 -- 172.21.15.83:0/520929922 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee241546a0 msgr2=0x7fee0c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:48.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee28d20640 1 --2- 172.21.15.83:0/520929922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee0c004d50 0x7fee0c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:48.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee23fff640 1 --2- 172.21.15.83:0/520929922 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee241546a0 0x7fee0c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:48.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee23fff640 1 -- 172.21.15.83:0/520929922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee0c004d50 msgr2=0x7fee0c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:48.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee23fff640 1 --2- 172.21.15.83:0/520929922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee0c004d50 0x7fee0c0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:48.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.830+0000 7fee23fff640 1 -- 172.21.15.83:0/520929922 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c003530 con 0x7fee0c0039c0 2024-09-19T07:33:48.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee23fff640 1 --2- 172.21.15.83:0/520929922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee0c0039c0 0x7fee0c003de0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fee14009fc0 tx=0x7fee1401ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff3aff9c77ded0ad server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:48.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee140b2070 con 0x7fee0c0039c0 2024-09-19T07:33:48.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee140a5bc0 con 0x7fee0c0039c0 2024-09-19T07:33:48.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee22ffd640 1 -- 172.21.15.83:0/520929922 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14009c10 con 0x7fee0c0039c0 2024-09-19T07:33:48.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee2a783640 1 -- 172.21.15.83:0/520929922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee0c0039c0 msgr2=0x7fee0c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:48.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee2a783640 1 --2- 172.21.15.83:0/520929922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee0c0039c0 0x7fee0c003de0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fee14009fc0 tx=0x7fee1401ccc0 comp rx=0 tx=0).stop 2024-09-19T07:33:48.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee28d20640 1 -- 172.21.15.83:0/520929922 reap_dead start 2024-09-19T07:33:48.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee2a783640 1 -- 172.21.15.83:0/520929922 shutdown_connections 2024-09-19T07:33:48.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee2a783640 1 -- 172.21.15.83:0/520929922 >> 172.21.15.83:0/520929922 conn(0x7fee240c5060 msgr2=0x7fee240c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:48.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee2a783640 1 -- 172.21.15.83:0/520929922 shutdown_connections 2024-09-19T07:33:48.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee2a783640 1 -- 172.21.15.83:0/520929922 wait complete. 2024-09-19T07:33:48.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee2a783640 1 Processor -- start 2024-09-19T07:33:48.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.834+0000 7fee2a783640 1 -- start start 2024-09-19T07:33:48.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee2a783640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee0c0039c0 0x7fee24155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee2a783640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee241eed80 0x7fee240c4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee2a783640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee241ef1e0 0x7fee241f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee2a783640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee24072e80 con 0x7fee241ef1e0 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee2a783640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fee24072d00 con 0x7fee241eed80 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee2a783640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fee24070230 con 0x7fee0c0039c0 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee28d20640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee241ef1e0 0x7fee241f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee28d20640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee241ef1e0 0x7fee241f1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36152/0 (socket says 172.21.15.83:36152) 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee28d20640 1 -- 172.21.15.83:0/2190887726 learned_addr learned my addr 172.21.15.83:0/2190887726 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee23fff640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee0c0039c0 0x7fee24155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee237fe640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee241eed80 0x7fee240c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee28d20640 1 -- 172.21.15.83:0/2190887726 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee0c0039c0 msgr2=0x7fee24155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee28d20640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee0c0039c0 0x7fee24155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:48.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee28d20640 1 -- 172.21.15.83:0/2190887726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee241eed80 msgr2=0x7fee240c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:48.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee28d20640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee241eed80 0x7fee240c4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:48.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee28d20640 1 -- 172.21.15.83:0/2190887726 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee24157050 con 0x7fee241ef1e0 2024-09-19T07:33:48.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee237fe640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee241eed80 0x7fee240c4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:33:48.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee28d20640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee241ef1e0 0x7fee241f1d80 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fee1c010790 tx=0x7fee1c010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:48.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee20ff9640 1 -- 172.21.15.83:0/2190887726 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c00f040 con 0x7fee241ef1e0 2024-09-19T07:33:48.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee20ff9640 1 -- 172.21.15.83:0/2190887726 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee1c00e870 con 0x7fee241ef1e0 2024-09-19T07:33:48.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee2a783640 1 -- 172.21.15.83:0/2190887726 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee241ee4a0 con 0x7fee241ef1e0 2024-09-19T07:33:48.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee20ff9640 1 -- 172.21.15.83:0/2190887726 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c00ec60 con 0x7fee241ef1e0 2024-09-19T07:33:48.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee2a783640 1 -- 172.21.15.83:0/2190887726 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee2414e9e0 con 0x7fee241ef1e0 2024-09-19T07:33:48.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee2a783640 1 -- 172.21.15.83:0/2190887726 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee24070620 con 0x7fee241ef1e0 2024-09-19T07:33:48.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.838+0000 7fee20ff9640 1 -- 172.21.15.83:0/2190887726 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fee1c0143f0 con 0x7fee241ef1e0 2024-09-19T07:33:48.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.842+0000 7fee20ff9640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fee100678c0 0x7fee10069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:48.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.842+0000 7fee20ff9640 1 -- 172.21.15.83:0/2190887726 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7fee1c086820 con 0x7fee241ef1e0 2024-09-19T07:33:48.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.842+0000 7fee23fff640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fee100678c0 0x7fee10069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:48.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.842+0000 7fee23fff640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fee100678c0 0x7fee10069d80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fee140af000 tx=0x7fee14004ac0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:48.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:48.846+0000 7fee20ff9640 1 -- 172.21.15.83:0/2190887726 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fee1c018030 con 0x7fee241ef1e0 2024-09-19T07:33:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.082+0000 7fee2a783640 1 -- 172.21.15.83:0/2190887726 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee24068900 con 0x7fee241ef1e0 2024-09-19T07:33:49.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.082+0000 7fee20ff9640 1 -- 172.21.15.83:0/2190887726 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7fee1c056940 con 0x7fee241ef1e0 2024-09-19T07:33:49.077 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:49.078 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:33:49.078 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:33:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.086+0000 7fee2a783640 1 -- 172.21.15.83:0/2190887726 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fee100678c0 msgr2=0x7fee10069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.086+0000 7fee2a783640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fee100678c0 0x7fee10069d80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fee140af000 tx=0x7fee14004ac0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.086+0000 7fee2a783640 1 -- 172.21.15.83:0/2190887726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee241ef1e0 msgr2=0x7fee241f1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.086+0000 7fee2a783640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee241ef1e0 0x7fee241f1d80 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fee1c010790 tx=0x7fee1c010c50 comp rx=0 tx=0).stop 2024-09-19T07:33:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.086+0000 7fee2a783640 1 -- 172.21.15.83:0/2190887726 shutdown_connections 2024-09-19T07:33:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.086+0000 7fee2a783640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fee100678c0 0x7fee10069d80 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.086+0000 7fee2a783640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee241ef1e0 0x7fee241f1d80 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.086+0000 7fee2a783640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee241eed80 0x7fee240c4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.086+0000 7fee2a783640 1 --2- 172.21.15.83:0/2190887726 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee0c0039c0 0x7fee24155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.086+0000 7fee2a783640 1 -- 172.21.15.83:0/2190887726 >> 172.21.15.83:0/2190887726 conn(0x7fee240c5060 msgr2=0x7fee240c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:49.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.086+0000 7fee2a783640 1 -- 172.21.15.83:0/2190887726 shutdown_connections 2024-09-19T07:33:49.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.086+0000 7fee2a783640 1 -- 172.21.15.83:0/2190887726 wait complete. 2024-09-19T07:33:49.099 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-19T07:33:49.099 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:33:49.099 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:33:49.109 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:33:49.109 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:49.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.286+0000 7fed7be9b640 1 Processor -- start 2024-09-19T07:33:49.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.286+0000 7fed7be9b640 1 -- start start 2024-09-19T07:33:49.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed7be9b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7415b7f0 0x7fed7415dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:49.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed7be9b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed7406e110 con 0x7fed740c3130 2024-09-19T07:33:49.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed7be9b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed740c3c20 con 0x7fed740ac290 2024-09-19T07:33:49.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed7be9b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fed740ac700 con 0x7fed7415b7f0 2024-09-19T07:33:49.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed79c37640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fed740c3130 0x7fed7406da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48260/0 (socket says 172.21.15.83:48260) 2024-09-19T07:33:49.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed79c37640 1 -- 172.21.15.83:0/3618313974 learned_addr learned my addr 172.21.15.83:0/3618313974 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:33:49.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed7a438640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7415b7f0 0x7fed7415dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:49.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 610120416 0 0) 0x7fed7406e110 con 0x7fed740c3130 2024-09-19T07:33:49.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed60003530 con 0x7fed740c3130 2024-09-19T07:33:49.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 185725785 0 0) 0x7fed60003530 con 0x7fed740c3130 2024-09-19T07:33:49.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7415b7f0 msgr2=0x7fed7415dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:49.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7415b7f0 0x7fed7415dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 >> v1:172.21.15.184:6789/0 conn(0x7fed740ac290 legacy=0x7fed740c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:49.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed740c4990 con 0x7fed740c3130 2024-09-19T07:33:49.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fed70001680 con 0x7fed740c3130 2024-09-19T07:33:49.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 >> v1:172.21.15.83:6789/0 conn(0x7fed740c3130 legacy=0x7fed7406da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:49.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed600039f0 0x7fed60003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:49.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7415b7f0 0x7fed60004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed60004d50 0x7fed600051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fed740ac700 con 0x7fed600039f0 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fed6000bb00 con 0x7fed60004d50 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fed6000bd00 con 0x7fed7415b7f0 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed79c37640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed600039f0 0x7fed60003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed79436640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7415b7f0 0x7fed60004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed7a438640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed60004d50 0x7fed600051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed79c37640 1 -- 172.21.15.83:0/3618313974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7415b7f0 msgr2=0x7fed60004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed79c37640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7415b7f0 0x7fed60004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed79c37640 1 -- 172.21.15.83:0/3618313974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed60004d50 msgr2=0x7fed600051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed79c37640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed60004d50 0x7fed600051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed79c37640 1 -- 172.21.15.83:0/3618313974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed60003530 con 0x7fed600039f0 2024-09-19T07:33:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.290+0000 7fed7a438640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed60004d50 0x7fed600051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:49.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed79c37640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed600039f0 0x7fed60003e10 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fed7000c240 tx=0x7fed7001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db042ca27230945 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:49.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed70007d20 con 0x7fed600039f0 2024-09-19T07:33:49.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed700a9070 con 0x7fed600039f0 2024-09-19T07:33:49.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed78c35640 1 -- 172.21.15.83:0/3618313974 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed70005d60 con 0x7fed600039f0 2024-09-19T07:33:49.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed7be9b640 1 -- 172.21.15.83:0/3618313974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed600039f0 msgr2=0x7fed60003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:49.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed7be9b640 1 --2- 172.21.15.83:0/3618313974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed600039f0 0x7fed60003e10 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fed7000c240 tx=0x7fed7001cbe0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed7a438640 1 -- 172.21.15.83:0/3618313974 reap_dead start 2024-09-19T07:33:49.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed7be9b640 1 -- 172.21.15.83:0/3618313974 shutdown_connections 2024-09-19T07:33:49.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed7be9b640 1 -- 172.21.15.83:0/3618313974 >> 172.21.15.83:0/3618313974 conn(0x7fed740c5060 msgr2=0x7fed740c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:49.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed7be9b640 1 -- 172.21.15.83:0/3618313974 shutdown_connections 2024-09-19T07:33:49.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed7be9b640 1 -- 172.21.15.83:0/3618313974 wait complete. 2024-09-19T07:33:49.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed7be9b640 1 Processor -- start 2024-09-19T07:33:49.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.294+0000 7fed7be9b640 1 -- start start 2024-09-19T07:33:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed7be9b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed600039f0 0x7fed741529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed7be9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740bf210 0x7fed740bdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed7be9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed740be040 0x7fed741ed1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed7be9b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fed740c82a0 con 0x7fed740bf210 2024-09-19T07:33:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed7be9b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fed740abd70 con 0x7fed740be040 2024-09-19T07:33:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed7be9b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fed740ad520 con 0x7fed600039f0 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed79436640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740bf210 0x7fed740bdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed79436640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740bf210 0x7fed740bdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36166/0 (socket says 172.21.15.83:36166) 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed79436640 1 -- 172.21.15.83:0/1059894192 learned_addr learned my addr 172.21.15.83:0/1059894192 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed7a438640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed740be040 0x7fed741ed1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed79c37640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed600039f0 0x7fed741529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed79436640 1 -- 172.21.15.83:0/1059894192 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed600039f0 msgr2=0x7fed741529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed79436640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed600039f0 0x7fed741529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed79436640 1 -- 172.21.15.83:0/1059894192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed740be040 msgr2=0x7fed741ed1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed79436640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed740be040 0x7fed741ed1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed79436640 1 -- 172.21.15.83:0/1059894192 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed740bd620 con 0x7fed740bf210 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed79436640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740bf210 0x7fed740bdb00 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fed6400d7a0 tx=0x7fed6400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed6a7fc640 1 -- 172.21.15.83:0/1059894192 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed64005270 con 0x7fed740bf210 2024-09-19T07:33:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed6a7fc640 1 -- 172.21.15.83:0/1059894192 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed64017070 con 0x7fed740bf210 2024-09-19T07:33:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed7be9b640 1 -- 172.21.15.83:0/1059894192 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed7415a2b0 con 0x7fed740bf210 2024-09-19T07:33:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed6a7fc640 1 -- 172.21.15.83:0/1059894192 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6400b630 con 0x7fed740bf210 2024-09-19T07:33:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed7be9b640 1 -- 172.21.15.83:0/1059894192 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed741ec910 con 0x7fed740bf210 2024-09-19T07:33:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed7be9b640 1 -- 172.21.15.83:0/1059894192 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed740c3510 con 0x7fed740bf210 2024-09-19T07:33:49.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed6a7fc640 1 -- 172.21.15.83:0/1059894192 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fed6400b7d0 con 0x7fed740bf210 2024-09-19T07:33:49.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed6a7fc640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed580678c0 0x7fed58069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:49.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed6a7fc640 1 -- 172.21.15.83:0/1059894192 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7fed640864d0 con 0x7fed740bf210 2024-09-19T07:33:49.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed79c37640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed580678c0 0x7fed58069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:49.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.298+0000 7fed79c37640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed580678c0 0x7fed58069d80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fed700ab000 tx=0x7fed700b1040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:49.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.306+0000 7fed6a7fc640 1 -- 172.21.15.83:0/1059894192 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fed640565f0 con 0x7fed740bf210 2024-09-19T07:33:49.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.542+0000 7fed7be9b640 1 -- 172.21.15.83:0/1059894192 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed740c56c0 con 0x7fed740bf210 2024-09-19T07:33:49.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.542+0000 7fed6a7fc640 1 -- 172.21.15.83:0/1059894192 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7fed64055ef0 con 0x7fed740bf210 2024-09-19T07:33:49.538 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:49.539 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:33:49.539 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:33:49.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.546+0000 7fed7be9b640 1 -- 172.21.15.83:0/1059894192 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed580678c0 msgr2=0x7fed58069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:49.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.546+0000 7fed7be9b640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed580678c0 0x7fed58069d80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fed700ab000 tx=0x7fed700b1040 comp rx=0 tx=0).stop 2024-09-19T07:33:49.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.546+0000 7fed7be9b640 1 -- 172.21.15.83:0/1059894192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740bf210 msgr2=0x7fed740bdb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:49.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.546+0000 7fed7be9b640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740bf210 0x7fed740bdb00 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fed6400d7a0 tx=0x7fed6400dc60 comp rx=0 tx=0).stop 2024-09-19T07:33:49.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.546+0000 7fed7be9b640 1 -- 172.21.15.83:0/1059894192 shutdown_connections 2024-09-19T07:33:49.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.546+0000 7fed7be9b640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed580678c0 0x7fed58069d80 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.546+0000 7fed7be9b640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed740be040 0x7fed741ed1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.546+0000 7fed7be9b640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740bf210 0x7fed740bdb00 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.546+0000 7fed7be9b640 1 --2- 172.21.15.83:0/1059894192 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed600039f0 0x7fed741529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:49.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.546+0000 7fed7be9b640 1 -- 172.21.15.83:0/1059894192 >> 172.21.15.83:0/1059894192 conn(0x7fed740c5060 msgr2=0x7fed74074f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:49.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.546+0000 7fed7be9b640 1 -- 172.21.15.83:0/1059894192 shutdown_connections 2024-09-19T07:33:49.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:49.546+0000 7fed7be9b640 1 -- 172.21.15.83:0/1059894192 wait complete. 2024-09-19T07:33:49.560 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:33:51.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:51.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6623508640 1 Processor -- start 2024-09-19T07:33:51.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6623508640 1 -- start start 2024-09-19T07:33:51.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6623508640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f661c06e270 0x7f661c1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:51.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6623508640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f661c0abb30 0x7f661c153060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:51.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6623508640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f661c1592c0 con 0x7f661c0c42e0 2024-09-19T07:33:51.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6623508640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f661c06e6e0 con 0x7f661c06e270 2024-09-19T07:33:51.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6623508640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f661c1595f0 con 0x7f661c0abb30 2024-09-19T07:33:51.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6620aa3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f661c06e270 0x7f661c1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:51.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f66212a4640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f661c0c42e0 0x7f661c159a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37886/0 (socket says 172.21.15.83:37886) 2024-09-19T07:33:51.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6620aa3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f661c06e270 0x7f661c1566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59350/0 (socket says 172.21.15.83:59350) 2024-09-19T07:33:51.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f66212a4640 1 -- 172.21.15.83:0/1830622254 learned_addr learned my addr 172.21.15.83:0/1830622254 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:33:51.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6621aa5640 1 --2- 172.21.15.83:0/1830622254 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f661c0abb30 0x7f661c153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:51.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6620aa3640 1 -- 172.21.15.83:0/1830622254 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f661c0abb30 msgr2=0x7f661c153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:51.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6620aa3640 1 --2- 172.21.15.83:0/1830622254 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f661c0abb30 0x7f661c153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:51.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6620aa3640 1 -- 172.21.15.83:0/1830622254 >> v1:172.21.15.83:6789/0 conn(0x7f661c0c42e0 legacy=0x7f661c159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:51.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6620aa3640 1 -- 172.21.15.83:0/1830622254 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661c06e900 con 0x7f661c06e270 2024-09-19T07:33:51.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6613fff640 1 -- 172.21.15.83:0/1830622254 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2485443560 0 0) 0x7f661c1592c0 con 0x7f661c0c42e0 2024-09-19T07:33:51.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.698+0000 7f6621aa5640 1 --2- 172.21.15.83:0/1830622254 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f661c0abb30 0x7f661c153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:51.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6620aa3640 1 --2- 172.21.15.83:0/1830622254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c06e270 0x7f661c1566b0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f661c159770 tx=0x7f661801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4415d254d92878d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:51.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6613fff640 1 -- 172.21.15.83:0/1830622254 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66180b1070 con 0x7f661c06e270 2024-09-19T07:33:51.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6613fff640 1 -- 172.21.15.83:0/1830622254 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66180ac410 con 0x7f661c06e270 2024-09-19T07:33:51.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6613fff640 1 -- 172.21.15.83:0/1830622254 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66180ac7e0 con 0x7f661c06e270 2024-09-19T07:33:51.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 -- 172.21.15.83:0/1830622254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c06e270 msgr2=0x7f661c1566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:51.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 --2- 172.21.15.83:0/1830622254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c06e270 0x7f661c1566b0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f661c159770 tx=0x7f661801c4a0 comp rx=0 tx=0).stop 2024-09-19T07:33:51.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 -- 172.21.15.83:0/1830622254 shutdown_connections 2024-09-19T07:33:51.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 --2- 172.21.15.83:0/1830622254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c06e270 0x7f661c1566b0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:51.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 --2- 172.21.15.83:0/1830622254 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f661c0abb30 0x7f661c153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:51.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 -- 172.21.15.83:0/1830622254 >> 172.21.15.83:0/1830622254 conn(0x7f661c06bc80 msgr2=0x7f661c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:51.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 -- 172.21.15.83:0/1830622254 shutdown_connections 2024-09-19T07:33:51.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 -- 172.21.15.83:0/1830622254 wait complete. 2024-09-19T07:33:51.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 Processor -- start 2024-09-19T07:33:51.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 -- start start 2024-09-19T07:33:51.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f661c06e270 0x7f661c154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:51.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f661c0abb30 0x7f661c14c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:51.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c0c42e0 0x7f661c14cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:51.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f661c0c4840 con 0x7f661c0c42e0 2024-09-19T07:33:51.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f661c0c46c0 con 0x7f661c0abb30 2024-09-19T07:33:51.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6623508640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f661c162dd0 con 0x7f661c06e270 2024-09-19T07:33:51.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6621aa5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c0c42e0 0x7f661c14cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:51.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6621aa5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c0c42e0 0x7f661c14cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59360/0 (socket says 172.21.15.83:59360) 2024-09-19T07:33:51.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6621aa5640 1 -- 172.21.15.83:0/3258669043 learned_addr learned my addr 172.21.15.83:0/3258669043 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:51.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6620aa3640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f661c0abb30 0x7f661c14c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:51.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6621aa5640 1 -- 172.21.15.83:0/3258669043 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f661c06e270 msgr2=0x7f661c154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:51.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6621aa5640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f661c06e270 0x7f661c154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:51.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6621aa5640 1 -- 172.21.15.83:0/3258669043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f661c0abb30 msgr2=0x7f661c14c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:51.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6621aa5640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f661c0abb30 0x7f661c14c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:51.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6621aa5640 1 -- 172.21.15.83:0/3258669043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661c0c7ba0 con 0x7f661c0c42e0 2024-09-19T07:33:51.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.702+0000 7f6621aa5640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c0c42e0 0x7f661c14cef0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f661400cef0 tx=0x7f661400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:51.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.706+0000 7f6611ffb640 1 -- 172.21.15.83:0/3258669043 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661401a070 con 0x7f661c0c42e0 2024-09-19T07:33:51.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.706+0000 7f6611ffb640 1 -- 172.21.15.83:0/3258669043 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f661400d860 con 0x7f661c0c42e0 2024-09-19T07:33:51.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.706+0000 7f6611ffb640 1 -- 172.21.15.83:0/3258669043 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661400dc50 con 0x7f661c0c42e0 2024-09-19T07:33:51.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.706+0000 7f6623508640 1 -- 172.21.15.83:0/3258669043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f661c0c5490 con 0x7f661c0c42e0 2024-09-19T07:33:51.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.706+0000 7f6623508640 1 -- 172.21.15.83:0/3258669043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f661c0c6960 con 0x7f661c0c42e0 2024-09-19T07:33:51.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.706+0000 7f6623508640 1 -- 172.21.15.83:0/3258669043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f661c1566b0 con 0x7f661c0c42e0 2024-09-19T07:33:51.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.706+0000 7f6611ffb640 1 -- 172.21.15.83:0/3258669043 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f661401d070 con 0x7f661c0c42e0 2024-09-19T07:33:51.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.706+0000 7f6611ffb640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66000678c0 0x7f6600069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:51.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.706+0000 7f6611ffb640 1 -- 172.21.15.83:0/3258669043 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7f6614086f10 con 0x7f661c0c42e0 2024-09-19T07:33:51.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.706+0000 7f66212a4640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66000678c0 0x7f6600069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:51.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.706+0000 7f66212a4640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66000678c0 0x7f6600069d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f660c0015a0 tx=0x7f660c007970 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:51.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.710+0000 7f6611ffb640 1 -- 172.21.15.83:0/3258669043 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f66140570f0 con 0x7f661c0c42e0 2024-09-19T07:33:51.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.902+0000 7f6623508640 1 -- 172.21.15.83:0/3258669043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f661c0c7ea0 con 0x7f661c0c42e0 2024-09-19T07:33:51.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.902+0000 7f6611ffb640 1 -- 172.21.15.83:0/3258669043 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7f66140569f0 con 0x7f661c0c42e0 2024-09-19T07:33:51.896 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:51.896 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:33:51.897 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:33:51.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.906+0000 7f6623508640 1 -- 172.21.15.83:0/3258669043 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66000678c0 msgr2=0x7f6600069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:51.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.906+0000 7f6623508640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66000678c0 0x7f6600069d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f660c0015a0 tx=0x7f660c007970 comp rx=0 tx=0).stop 2024-09-19T07:33:51.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.906+0000 7f6623508640 1 -- 172.21.15.83:0/3258669043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c0c42e0 msgr2=0x7f661c14cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:51.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.906+0000 7f6623508640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c0c42e0 0x7f661c14cef0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f661400cef0 tx=0x7f661400a710 comp rx=0 tx=0).stop 2024-09-19T07:33:51.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.906+0000 7f6623508640 1 -- 172.21.15.83:0/3258669043 shutdown_connections 2024-09-19T07:33:51.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.906+0000 7f6623508640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66000678c0 0x7f6600069d80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:51.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.906+0000 7f6623508640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c0c42e0 0x7f661c14cef0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:51.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.906+0000 7f6623508640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f661c0abb30 0x7f661c14c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:51.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.906+0000 7f6623508640 1 --2- 172.21.15.83:0/3258669043 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f661c06e270 0x7f661c154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:51.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.906+0000 7f6623508640 1 -- 172.21.15.83:0/3258669043 >> 172.21.15.83:0/3258669043 conn(0x7f661c06bc80 msgr2=0x7f661c14ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:51.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.906+0000 7f6623508640 1 -- 172.21.15.83:0/3258669043 shutdown_connections 2024-09-19T07:33:51.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:51.906+0000 7f6623508640 1 -- 172.21.15.83:0/3258669043 wait complete. 2024-09-19T07:33:51.914 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:33:53.915 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:54.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe688d0d640 1 Processor -- start 2024-09-19T07:33:54.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe688d0d640 1 -- start start 2024-09-19T07:33:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe688d0d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe684071100 0x7fe6841545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:54.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe688d0d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6840a9e60 con 0x7fe6840c28e0 2024-09-19T07:33:54.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe688d0d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6840aa140 con 0x7fe68406dea0 2024-09-19T07:33:54.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe688d0d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe684071b50 con 0x7fe684071100 2024-09-19T07:33:54.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe681d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe6840c28e0 0x7fe684150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37894/0 (socket says 172.21.15.83:37894) 2024-09-19T07:33:54.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe681d9b640 1 -- 172.21.15.83:0/4110699018 learned_addr learned my addr 172.21.15.83:0/4110699018 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:33:54.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68259c640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe684071100 0x7fe6841545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:54.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4100834119 0 0) 0x7fe6840a9e60 con 0x7fe6840c28e0 2024-09-19T07:33:54.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe670003530 con 0x7fe6840c28e0 2024-09-19T07:33:54.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1029066077 0 0) 0x7fe670003530 con 0x7fe6840c28e0 2024-09-19T07:33:54.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe684071100 msgr2=0x7fe6841545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:54.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe684071100 0x7fe6841545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:54.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 >> v1:172.21.15.184:6789/0 conn(0x7fe68406dea0 legacy=0x7fe68406e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:54.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6840c2240 con 0x7fe6840c28e0 2024-09-19T07:33:54.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fe66c001680 con 0x7fe6840c28e0 2024-09-19T07:33:54.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 >> v1:172.21.15.83:6789/0 conn(0x7fe6840c28e0 legacy=0x7fe684150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:54.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe684071100 0x7fe6700039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:54.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe670003f70 0x7fe670004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:54.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe670004d50 0x7fe6700051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:54.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe684071b50 con 0x7fe670003f70 2024-09-19T07:33:54.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe67000bb60 con 0x7fe670004d50 2024-09-19T07:33:54.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe67000bd60 con 0x7fe684071100 2024-09-19T07:33:54.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.058+0000 7fe681d9b640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe670003f70 0x7fe670004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:54.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe682d9d640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe670004d50 0x7fe6700051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:54.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe68259c640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe684071100 0x7fe6700039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:54.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe681d9b640 1 -- 172.21.15.83:0/4110699018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe684071100 msgr2=0x7fe6700039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:54.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe681d9b640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe684071100 0x7fe6700039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:54.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe681d9b640 1 -- 172.21.15.83:0/4110699018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe670004d50 msgr2=0x7fe6700051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:54.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe681d9b640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe670004d50 0x7fe6700051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:54.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe681d9b640 1 -- 172.21.15.83:0/4110699018 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe670003530 con 0x7fe670003f70 2024-09-19T07:33:54.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe68259c640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe684071100 0x7fe6700039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:54.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe681d9b640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe670003f70 0x7fe670004810 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c190 tx=0x7fe66c01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6c27778f773617 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:54.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c007d00 con 0x7fe670003f70 2024-09-19T07:33:54.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe66c0a9020 con 0x7fe670003f70 2024-09-19T07:33:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe68159a640 1 -- 172.21.15.83:0/4110699018 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c005b00 con 0x7fe670003f70 2024-09-19T07:33:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe688d0d640 1 -- 172.21.15.83:0/4110699018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe670003f70 msgr2=0x7fe670004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe688d0d640 1 --2- 172.21.15.83:0/4110699018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe670003f70 0x7fe670004810 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c190 tx=0x7fe66c01cc40 comp rx=0 tx=0).stop 2024-09-19T07:33:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe682d9d640 1 -- 172.21.15.83:0/4110699018 reap_dead start 2024-09-19T07:33:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe688d0d640 1 -- 172.21.15.83:0/4110699018 shutdown_connections 2024-09-19T07:33:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe688d0d640 1 -- 172.21.15.83:0/4110699018 >> 172.21.15.83:0/4110699018 conn(0x7fe68406bc80 msgr2=0x7fe68406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe688d0d640 1 -- 172.21.15.83:0/4110699018 shutdown_connections 2024-09-19T07:33:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.062+0000 7fe688d0d640 1 -- 172.21.15.83:0/4110699018 wait complete. 2024-09-19T07:33:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe688d0d640 1 Processor -- start 2024-09-19T07:33:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe688d0d640 1 -- start start 2024-09-19T07:33:54.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe688d0d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe670003f70 0x7fe684157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:54.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe688d0d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6840c6da0 0x7fe68414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:54.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe688d0d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe68414ff60 0x7fe684146a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:54.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe688d0d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe6841517d0 con 0x7fe670003f70 2024-09-19T07:33:54.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe688d0d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe684151650 con 0x7fe6840c6da0 2024-09-19T07:33:54.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe688d0d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe6840714e0 con 0x7fe68414ff60 2024-09-19T07:33:54.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe68259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe670003f70 0x7fe684157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:54.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe68259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe670003f70 0x7fe684157870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59384/0 (socket says 172.21.15.83:59384) 2024-09-19T07:33:54.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe68259c640 1 -- 172.21.15.83:0/3550777104 learned_addr learned my addr 172.21.15.83:0/3550777104 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:54.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe681d9b640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6840c6da0 0x7fe68414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:54.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe682d9d640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe68414ff60 0x7fe684146a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:54.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe68259c640 1 -- 172.21.15.83:0/3550777104 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe68414ff60 msgr2=0x7fe684146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:54.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe68259c640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe68414ff60 0x7fe684146a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:54.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe68259c640 1 -- 172.21.15.83:0/3550777104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6840c6da0 msgr2=0x7fe68414fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:54.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe68259c640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6840c6da0 0x7fe68414fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:54.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe68259c640 1 -- 172.21.15.83:0/3550777104 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6841ed340 con 0x7fe670003f70 2024-09-19T07:33:54.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe68259c640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe670003f70 0x7fe684157870 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00f350 tx=0x7fe67c00f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:54.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe67affd640 1 -- 172.21.15.83:0/3550777104 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67c003ba0 con 0x7fe670003f70 2024-09-19T07:33:54.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe67affd640 1 -- 172.21.15.83:0/3550777104 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe67c003d40 con 0x7fe670003f70 2024-09-19T07:33:54.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe688d0d640 1 -- 172.21.15.83:0/3550777104 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe684147560 con 0x7fe670003f70 2024-09-19T07:33:54.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe67affd640 1 -- 172.21.15.83:0/3550777104 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67c004220 con 0x7fe670003f70 2024-09-19T07:33:54.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe688d0d640 1 -- 172.21.15.83:0/3550777104 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6841ec580 con 0x7fe670003f70 2024-09-19T07:33:54.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe688d0d640 1 -- 172.21.15.83:0/3550777104 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe684150f40 con 0x7fe670003f70 2024-09-19T07:33:54.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe67affd640 1 -- 172.21.15.83:0/3550777104 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe67c01e020 con 0x7fe670003f70 2024-09-19T07:33:54.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe67affd640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6640678f0 0x7fe664069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:54.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe681d9b640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6640678f0 0x7fe664069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:54.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.066+0000 7fe67affd640 1 -- 172.21.15.83:0/3550777104 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7fe67c086dd0 con 0x7fe670003f70 2024-09-19T07:33:54.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.070+0000 7fe681d9b640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6640678f0 0x7fe664069db0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c190 tx=0x7fe684072930 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:54.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.074+0000 7fe67affd640 1 -- 172.21.15.83:0/3550777104 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe67c012070 con 0x7fe670003f70 2024-09-19T07:33:54.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.266+0000 7fe688d0d640 1 -- 172.21.15.83:0/3550777104 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6840c4f30 con 0x7fe670003f70 2024-09-19T07:33:54.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe67affd640 1 -- 172.21.15.83:0/3550777104 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7fe67c056ef0 con 0x7fe670003f70 2024-09-19T07:33:54.262 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:54.263 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:33:54.263 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:33:54.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe688d0d640 1 -- 172.21.15.83:0/3550777104 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6640678f0 msgr2=0x7fe664069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:54.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe688d0d640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6640678f0 0x7fe664069db0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c190 tx=0x7fe684072930 comp rx=0 tx=0).stop 2024-09-19T07:33:54.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe688d0d640 1 -- 172.21.15.83:0/3550777104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe670003f70 msgr2=0x7fe684157870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:54.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe688d0d640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe670003f70 0x7fe684157870 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00f350 tx=0x7fe67c00f810 comp rx=0 tx=0).stop 2024-09-19T07:33:54.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe688d0d640 1 -- 172.21.15.83:0/3550777104 shutdown_connections 2024-09-19T07:33:54.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe688d0d640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6640678f0 0x7fe664069db0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:54.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe688d0d640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe68414ff60 0x7fe684146a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:54.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe688d0d640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6840c6da0 0x7fe68414fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:54.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe688d0d640 1 --2- 172.21.15.83:0/3550777104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe670003f70 0x7fe684157870 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:54.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe688d0d640 1 -- 172.21.15.83:0/3550777104 >> 172.21.15.83:0/3550777104 conn(0x7fe68406bc80 msgr2=0x7fe684152910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:54.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe688d0d640 1 -- 172.21.15.83:0/3550777104 shutdown_connections 2024-09-19T07:33:54.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:54.270+0000 7fe688d0d640 1 -- 172.21.15.83:0/3550777104 wait complete. 2024-09-19T07:33:54.285 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:33:56.286 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:56.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.426+0000 7f917320e640 1 Processor -- start 2024-09-19T07:33:56.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f917320e640 1 -- start start 2024-09-19T07:33:56.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f917320e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f916c150290 0x7f916c0c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:56.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f917320e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f916c152b20 0x7f916c0c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:56.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f917320e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f916c0c45c0 0x7f916c153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:56.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f917320e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f916c0716c0 con 0x7f916c0c45c0 2024-09-19T07:33:56.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f917320e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f916c0c3920 con 0x7f916c152b20 2024-09-19T07:33:56.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f917320e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f916c0c3490 con 0x7f916c150290 2024-09-19T07:33:56.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f91717ab640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f916c0c45c0 0x7f916c153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:56.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f91717ab640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f916c0c45c0 0x7f916c153eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59386/0 (socket says 172.21.15.83:59386) 2024-09-19T07:33:56.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f91717ab640 1 -- 172.21.15.83:0/3733675519 learned_addr learned my addr 172.21.15.83:0/3733675519 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:56.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f9170faa640 1 --2- 172.21.15.83:0/3733675519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f916c150290 0x7f916c0c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:56.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f916bfff640 1 --2- 172.21.15.83:0/3733675519 >> v2:172.21.15.184:3300/0 conn(0x7f916c152b20 0x7f916c0c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:56.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f91717ab640 1 -- 172.21.15.83:0/3733675519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f916c150290 msgr2=0x7f916c0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:56.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f91717ab640 1 --2- 172.21.15.83:0/3733675519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f916c150290 0x7f916c0c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f91717ab640 1 -- 172.21.15.83:0/3733675519 >> v2:172.21.15.184:3300/0 conn(0x7f916c152b20 msgr2=0x7f916c0c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:56.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f91717ab640 1 --2- 172.21.15.83:0/3733675519 >> v2:172.21.15.184:3300/0 conn(0x7f916c152b20 0x7f916c0c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f91717ab640 1 -- 172.21.15.83:0/3733675519 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916c150670 con 0x7f916c0c45c0 2024-09-19T07:33:56.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f91717ab640 1 --2- 172.21.15.83:0/3733675519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f916c0c45c0 0x7f916c153eb0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f916c06cae0 tx=0x7f916401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652ce05935c814c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:56.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f916b7fe640 1 -- 172.21.15.83:0/3733675519 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91640a5840 con 0x7f916c0c45c0 2024-09-19T07:33:56.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f916b7fe640 1 -- 172.21.15.83:0/3733675519 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91640a59e0 con 0x7f916c0c45c0 2024-09-19T07:33:56.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.430+0000 7f916b7fe640 1 -- 172.21.15.83:0/3733675519 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91640a5db0 con 0x7f916c0c45c0 2024-09-19T07:33:56.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 -- 172.21.15.83:0/3733675519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f916c0c45c0 msgr2=0x7f916c153eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:56.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 --2- 172.21.15.83:0/3733675519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f916c0c45c0 0x7f916c153eb0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f916c06cae0 tx=0x7f916401c9c0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 -- 172.21.15.83:0/3733675519 shutdown_connections 2024-09-19T07:33:56.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 --2- 172.21.15.83:0/3733675519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f916c0c45c0 0x7f916c153eb0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 --2- 172.21.15.83:0/3733675519 >> v2:172.21.15.184:3300/0 conn(0x7f916c152b20 0x7f916c0c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 --2- 172.21.15.83:0/3733675519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f916c150290 0x7f916c0c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 -- 172.21.15.83:0/3733675519 >> 172.21.15.83:0/3733675519 conn(0x7f916c06bc80 msgr2=0x7f916c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:56.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 -- 172.21.15.83:0/3733675519 shutdown_connections 2024-09-19T07:33:56.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 -- 172.21.15.83:0/3733675519 wait complete. 2024-09-19T07:33:56.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 Processor -- start 2024-09-19T07:33:56.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 -- start start 2024-09-19T07:33:56.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f916c0c45c0 0x7f916c0c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:56.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f916c150290 0x7f916c0ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:56.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f916c152b20 0x7f916c0ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:56.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f916c0c2530 con 0x7f916c0c45c0 2024-09-19T07:33:56.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f916c0c3120 con 0x7f916c152b20 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f916c154800 con 0x7f916c150290 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f9170faa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f916c0c45c0 0x7f916c0c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f9170faa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f916c0c45c0 0x7f916c0c6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59390/0 (socket says 172.21.15.83:59390) 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f9170faa640 1 -- 172.21.15.83:0/4234990175 learned_addr learned my addr 172.21.15.83:0/4234990175 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f916bfff640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f916c150290 0x7f916c0ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f91717ab640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f916c152b20 0x7f916c0ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f9170faa640 1 -- 172.21.15.83:0/4234990175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f916c150290 msgr2=0x7f916c0ac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f9170faa640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f916c150290 0x7f916c0ac1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f9170faa640 1 -- 172.21.15.83:0/4234990175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f916c152b20 msgr2=0x7f916c0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f9170faa640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f916c152b20 0x7f916c0ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f9170faa640 1 -- 172.21.15.83:0/4234990175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916c1469e0 con 0x7f916c0c45c0 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f91717ab640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f916c152b20 0x7f916c0ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f9170faa640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f916c0c45c0 0x7f916c0c6d30 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f916000d830 tx=0x7f916000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:56.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f91697fa640 1 -- 172.21.15.83:0/4234990175 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160018020 con 0x7f916c0c45c0 2024-09-19T07:33:56.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f91697fa640 1 -- 172.21.15.83:0/4234990175 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91600096a0 con 0x7f916c0c45c0 2024-09-19T07:33:56.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 -- 172.21.15.83:0/4234990175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f916c146250 con 0x7f916c0c45c0 2024-09-19T07:33:56.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f91697fa640 1 -- 172.21.15.83:0/4234990175 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160009a50 con 0x7f916c0c45c0 2024-09-19T07:33:56.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.434+0000 7f917320e640 1 -- 172.21.15.83:0/4234990175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f916c0c7270 con 0x7f916c0c45c0 2024-09-19T07:33:56.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.438+0000 7f917320e640 1 -- 172.21.15.83:0/4234990175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f916c0c1ff0 con 0x7f916c0c45c0 2024-09-19T07:33:56.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.438+0000 7f91697fa640 1 -- 172.21.15.83:0/4234990175 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f916001b020 con 0x7f916c0c45c0 2024-09-19T07:33:56.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.438+0000 7f91697fa640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f91500678c0 0x7f9150069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:56.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.438+0000 7f91697fa640 1 -- 172.21.15.83:0/4234990175 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7f9160086c60 con 0x7f916c0c45c0 2024-09-19T07:33:56.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.438+0000 7f916bfff640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f91500678c0 0x7f9150069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:56.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.438+0000 7f916bfff640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f91500678c0 0x7f9150069d80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f916c0c4e90 tx=0x7f9158008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:56.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.442+0000 7f91697fa640 1 -- 172.21.15.83:0/4234990175 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9160022020 con 0x7f916c0c45c0 2024-09-19T07:33:56.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 -- 172.21.15.83:0/4234990175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f916c0abdf0 con 0x7f916c0c45c0 2024-09-19T07:33:56.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f91697fa640 1 -- 172.21.15.83:0/4234990175 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7f9160056cf0 con 0x7f916c0c45c0 2024-09-19T07:33:56.636 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:56.636 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:33:56.636 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:33:56.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 -- 172.21.15.83:0/4234990175 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f91500678c0 msgr2=0x7f9150069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:56.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f91500678c0 0x7f9150069d80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f916c0c4e90 tx=0x7f9158008040 comp rx=0 tx=0).stop 2024-09-19T07:33:56.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 -- 172.21.15.83:0/4234990175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f916c0c45c0 msgr2=0x7f916c0c6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:56.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f916c0c45c0 0x7f916c0c6d30 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f916000d830 tx=0x7f916000dcf0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 -- 172.21.15.83:0/4234990175 shutdown_connections 2024-09-19T07:33:56.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f91500678c0 0x7f9150069d80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f916c152b20 0x7f916c0ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f916c150290 0x7f916c0ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 --2- 172.21.15.83:0/4234990175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f916c0c45c0 0x7f916c0c6d30 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:56.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 -- 172.21.15.83:0/4234990175 >> 172.21.15.83:0/4234990175 conn(0x7f916c06bc80 msgr2=0x7f916c14d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:56.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 -- 172.21.15.83:0/4234990175 shutdown_connections 2024-09-19T07:33:56.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:56.642+0000 7f917320e640 1 -- 172.21.15.83:0/4234990175 wait complete. 2024-09-19T07:33:56.656 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:33:58.657 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:33:58.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffaa65af640 1 Processor -- start 2024-09-19T07:33:58.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffaa65af640 1 -- start start 2024-09-19T07:33:58.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffaa65af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffaa0071100 0x7ffaa01545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffaa65af640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffaa006dea0 0x7ffaa006e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffaa65af640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaa00a9e60 con 0x7ffaa00c28e0 2024-09-19T07:33:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffaa65af640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ffaa0071b50 con 0x7ffaa0071100 2024-09-19T07:33:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffaa65af640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa00c1ff0 con 0x7ffaa006dea0 2024-09-19T07:33:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffa9ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffaa0071100 0x7ffaa01545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffa977fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ffaa00c28e0 0x7ffaa0150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37896/0 (socket says 172.21.15.83:37896) 2024-09-19T07:33:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffa977fe640 1 -- 172.21.15.83:0/1209035494 learned_addr learned my addr 172.21.15.83:0/1209035494 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:33:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffaa4b4c640 1 --2- 172.21.15.83:0/1209035494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffaa006dea0 0x7ffaa006e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffa9ffff640 1 -- 172.21.15.83:0/1209035494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffaa006dea0 msgr2=0x7ffaa006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:33:58.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffa9ffff640 1 --2- 172.21.15.83:0/1209035494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffaa006dea0 0x7ffaa006e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:58.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffa9ffff640 1 -- 172.21.15.83:0/1209035494 >> v1:172.21.15.83:6789/0 conn(0x7ffaa00c28e0 legacy=0x7ffaa0150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:58.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.797+0000 7ffa9ffff640 1 -- 172.21.15.83:0/1209035494 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa00aa140 con 0x7ffaa0071100 2024-09-19T07:33:58.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffa9ffff640 1 --2- 172.21.15.83:0/1209035494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaa0071100 0x7ffaa01545e0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ffa900097c0 tx=0x7ffa9001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee8d91c966c0e86 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:58.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffa9f7fe640 1 -- 172.21.15.83:0/1209035494 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa900b0070 con 0x7ffaa0071100 2024-09-19T07:33:58.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffa9f7fe640 1 -- 172.21.15.83:0/1209035494 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa900ab600 con 0x7ffaa0071100 2024-09-19T07:33:58.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffa9f7fe640 1 -- 172.21.15.83:0/1209035494 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa900ab9d0 con 0x7ffaa0071100 2024-09-19T07:33:58.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 -- 172.21.15.83:0/1209035494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaa0071100 msgr2=0x7ffaa01545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:58.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 --2- 172.21.15.83:0/1209035494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaa0071100 0x7ffaa01545e0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ffa900097c0 tx=0x7ffa9001c410 comp rx=0 tx=0).stop 2024-09-19T07:33:58.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 -- 172.21.15.83:0/1209035494 shutdown_connections 2024-09-19T07:33:58.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 --2- 172.21.15.83:0/1209035494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffaa006dea0 0x7ffaa006e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:58.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 --2- 172.21.15.83:0/1209035494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaa0071100 0x7ffaa01545e0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:58.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 -- 172.21.15.83:0/1209035494 >> 172.21.15.83:0/1209035494 conn(0x7ffaa006bc80 msgr2=0x7ffaa006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:58.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 -- 172.21.15.83:0/1209035494 shutdown_connections 2024-09-19T07:33:58.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 -- 172.21.15.83:0/1209035494 wait complete. 2024-09-19T07:33:58.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 Processor -- start 2024-09-19T07:33:58.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 -- start start 2024-09-19T07:33:58.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffaa006dea0 0x7ffaa00c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:58.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffaa0071100 0x7ffaa00aab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:58.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaa00c28e0 0x7ffaa015c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:58.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa0071660 con 0x7ffaa00c28e0 2024-09-19T07:33:58.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa00714e0 con 0x7ffaa006dea0 2024-09-19T07:33:58.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa0160c60 con 0x7ffaa0071100 2024-09-19T07:33:58.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa4b4c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaa00c28e0 0x7ffaa015c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:58.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa4b4c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaa00c28e0 0x7ffaa015c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59406/0 (socket says 172.21.15.83:59406) 2024-09-19T07:33:58.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa4b4c640 1 -- 172.21.15.83:0/3173745734 learned_addr learned my addr 172.21.15.83:0/3173745734 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:33:58.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffa977fe640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffaa0071100 0x7ffaa00aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:58.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffa9ffff640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffaa006dea0 0x7ffaa00c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:58.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa4b4c640 1 -- 172.21.15.83:0/3173745734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffaa0071100 msgr2=0x7ffaa00aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:58.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa4b4c640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffaa0071100 0x7ffaa00aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:58.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa4b4c640 1 -- 172.21.15.83:0/3173745734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffaa006dea0 msgr2=0x7ffaa00c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:58.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa4b4c640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffaa006dea0 0x7ffaa00c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:58.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa4b4c640 1 -- 172.21.15.83:0/3173745734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa00c59f0 con 0x7ffaa00c28e0 2024-09-19T07:33:58.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffa977fe640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffaa0071100 0x7ffaa00aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:58.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffa9ffff640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffaa006dea0 0x7ffaa00c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:33:58.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa4b4c640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaa00c28e0 0x7ffaa015c870 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7ffa98007e40 tx=0x7ffa9800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:58.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffa9d7fa640 1 -- 172.21.15.83:0/3173745734 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98014620 con 0x7ffaa00c28e0 2024-09-19T07:33:58.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffa9d7fa640 1 -- 172.21.15.83:0/3173745734 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa98019070 con 0x7ffaa00c28e0 2024-09-19T07:33:58.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 -- 172.21.15.83:0/3173745734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa00ab4c0 con 0x7ffaa00c28e0 2024-09-19T07:33:58.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffa9d7fa640 1 -- 172.21.15.83:0/3173745734 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98014a20 con 0x7ffaa00c28e0 2024-09-19T07:33:58.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.801+0000 7ffaa65af640 1 -- 172.21.15.83:0/3173745734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa00aba80 con 0x7ffaa00c28e0 2024-09-19T07:33:58.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.805+0000 7ffaa65af640 1 -- 172.21.15.83:0/3173745734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaa006e330 con 0x7ffaa00c28e0 2024-09-19T07:33:58.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.805+0000 7ffa9d7fa640 1 -- 172.21.15.83:0/3173745734 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffa98014bc0 con 0x7ffaa00c28e0 2024-09-19T07:33:58.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.805+0000 7ffa9d7fa640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ffa800678c0 0x7ffa80069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:33:58.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.805+0000 7ffa9d7fa640 1 -- 172.21.15.83:0/3173745734 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7ffa98086f50 con 0x7ffaa00c28e0 2024-09-19T07:33:58.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.805+0000 7ffa9ffff640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ffa800678c0 0x7ffa80069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:33:58.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.805+0000 7ffa9ffff640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ffa800678c0 0x7ffa80069d80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ffa900ad000 tx=0x7ffa90008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:33:58.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:58.809+0000 7ffa9d7fa640 1 -- 172.21.15.83:0/3173745734 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ffa98057070 con 0x7ffaa00c28e0 2024-09-19T07:33:58.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.005+0000 7ffaa65af640 1 -- 172.21.15.83:0/3173745734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffaa00c4f30 con 0x7ffaa00c28e0 2024-09-19T07:33:59.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.005+0000 7ffa9d7fa640 1 -- 172.21.15.83:0/3173745734 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7ffa98056970 con 0x7ffaa00c28e0 2024-09-19T07:33:59.001 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:33:59.001 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:33:59.001 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:33:59.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.009+0000 7ffaa65af640 1 -- 172.21.15.83:0/3173745734 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ffa800678c0 msgr2=0x7ffa80069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:59.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.009+0000 7ffaa65af640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ffa800678c0 0x7ffa80069d80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ffa900ad000 tx=0x7ffa90008040 comp rx=0 tx=0).stop 2024-09-19T07:33:59.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.009+0000 7ffaa65af640 1 -- 172.21.15.83:0/3173745734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaa00c28e0 msgr2=0x7ffaa015c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:33:59.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.009+0000 7ffaa65af640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaa00c28e0 0x7ffaa015c870 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7ffa98007e40 tx=0x7ffa9800e4b0 comp rx=0 tx=0).stop 2024-09-19T07:33:59.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.009+0000 7ffaa65af640 1 -- 172.21.15.83:0/3173745734 shutdown_connections 2024-09-19T07:33:59.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.009+0000 7ffaa65af640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ffa800678c0 0x7ffa80069d80 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:59.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.009+0000 7ffaa65af640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaa00c28e0 0x7ffaa015c870 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:59.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.009+0000 7ffaa65af640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffaa0071100 0x7ffaa00aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:59.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.009+0000 7ffaa65af640 1 --2- 172.21.15.83:0/3173745734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffaa006dea0 0x7ffaa00c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:33:59.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.009+0000 7ffaa65af640 1 -- 172.21.15.83:0/3173745734 >> 172.21.15.83:0/3173745734 conn(0x7ffaa006bc80 msgr2=0x7ffaa0150f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:33:59.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.009+0000 7ffaa65af640 1 -- 172.21.15.83:0/3173745734 shutdown_connections 2024-09-19T07:33:59.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:33:59.009+0000 7ffaa65af640 1 -- 172.21.15.83:0/3173745734 wait complete. 2024-09-19T07:33:59.018 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:34:01.018 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:01.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efc00217640 1 Processor -- start 2024-09-19T07:34:01.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efc00217640 1 -- start start 2024-09-19T07:34:01.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efc00217640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7efbf814ca60 0x7efbf8159490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:01.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efc00217640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efbf80bef00 0x7efbf8156840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:01.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efc00217640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efbf80cbb00 0x7efbf80cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:01.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efc00217640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7efbf81599d0 con 0x7efbf80bef00 2024-09-19T07:34:01.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efc00217640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7efbf80ca240 con 0x7efbf814ca60 2024-09-19T07:34:01.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efc00217640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efbf80c9c80 con 0x7efbf80cbb00 2024-09-19T07:34:01.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efbfd7b2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efbf80bef00 0x7efbf8156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:01.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efbfd7b2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efbf80bef00 0x7efbf8156840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49764/0 (socket says 172.21.15.83:49764) 2024-09-19T07:34:01.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efbfd7b2640 1 -- 172.21.15.83:0/504787000 learned_addr learned my addr 172.21.15.83:0/504787000 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:01.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efbfe7b4640 1 --2- 172.21.15.83:0/504787000 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efbf80cbb00 0x7efbf80cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:01.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efbfdfb3640 1 --2- 172.21.15.83:0/504787000 >> v2:172.21.15.184:3300/0 conn(0x7efbf814ca60 0x7efbf8159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:01.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efbfd7b2640 1 -- 172.21.15.83:0/504787000 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efbf80cbb00 msgr2=0x7efbf80cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:01.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efbfd7b2640 1 --2- 172.21.15.83:0/504787000 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efbf80cbb00 0x7efbf80cbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:01.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efbfd7b2640 1 -- 172.21.15.83:0/504787000 >> v2:172.21.15.184:3300/0 conn(0x7efbf814ca60 msgr2=0x7efbf8159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:01.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efbfd7b2640 1 --2- 172.21.15.83:0/504787000 >> v2:172.21.15.184:3300/0 conn(0x7efbf814ca60 0x7efbf8159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:01.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.153+0000 7efbfd7b2640 1 -- 172.21.15.83:0/504787000 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf80c9f60 con 0x7efbf80bef00 2024-09-19T07:34:01.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfd7b2640 1 --2- 172.21.15.83:0/504787000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efbf80bef00 0x7efbf8156840 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7efbf8073250 tx=0x7efbf401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820b36ead45a3038 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:01.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfcfb1640 1 -- 172.21.15.83:0/504787000 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf40b0070 con 0x7efbf80bef00 2024-09-19T07:34:01.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfcfb1640 1 -- 172.21.15.83:0/504787000 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf40ab600 con 0x7efbf80bef00 2024-09-19T07:34:01.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfcfb1640 1 -- 172.21.15.83:0/504787000 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf40ab9d0 con 0x7efbf80bef00 2024-09-19T07:34:01.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 -- 172.21.15.83:0/504787000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efbf80bef00 msgr2=0x7efbf8156840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:01.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 --2- 172.21.15.83:0/504787000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efbf80bef00 0x7efbf8156840 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7efbf8073250 tx=0x7efbf401c520 comp rx=0 tx=0).stop 2024-09-19T07:34:01.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 -- 172.21.15.83:0/504787000 shutdown_connections 2024-09-19T07:34:01.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 --2- 172.21.15.83:0/504787000 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efbf80cbb00 0x7efbf80cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:01.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 --2- 172.21.15.83:0/504787000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efbf80bef00 0x7efbf8156840 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:01.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 --2- 172.21.15.83:0/504787000 >> v2:172.21.15.184:3300/0 conn(0x7efbf814ca60 0x7efbf8159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:01.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 -- 172.21.15.83:0/504787000 >> 172.21.15.83:0/504787000 conn(0x7efbf8072470 msgr2=0x7efbf8072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:01.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 -- 172.21.15.83:0/504787000 shutdown_connections 2024-09-19T07:34:01.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 -- 172.21.15.83:0/504787000 wait complete. 2024-09-19T07:34:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 Processor -- start 2024-09-19T07:34:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 -- start start 2024-09-19T07:34:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efbf80bef00 0x7efbf80c2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efbf80cbb00 0x7efbf815fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efbf814ca60 0x7efbf814b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efbf8153a50 con 0x7efbf814ca60 2024-09-19T07:34:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efbf806ccb0 con 0x7efbf80cbb00 2024-09-19T07:34:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efc00217640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efbf80c98c0 con 0x7efbf80bef00 2024-09-19T07:34:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfe7b4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efbf814ca60 0x7efbf814b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfe7b4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efbf814ca60 0x7efbf814b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49776/0 (socket says 172.21.15.83:49776) 2024-09-19T07:34:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfe7b4640 1 -- 172.21.15.83:0/2140692504 learned_addr learned my addr 172.21.15.83:0/2140692504 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfdfb3640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efbf80bef00 0x7efbf80c2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfd7b2640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efbf80cbb00 0x7efbf815fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfe7b4640 1 -- 172.21.15.83:0/2140692504 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efbf80bef00 msgr2=0x7efbf80c2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfe7b4640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efbf80bef00 0x7efbf80c2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfe7b4640 1 -- 172.21.15.83:0/2140692504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efbf80cbb00 msgr2=0x7efbf815fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfe7b4640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efbf80cbb00 0x7efbf815fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfe7b4640 1 -- 172.21.15.83:0/2140692504 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf80c03e0 con 0x7efbf814ca60 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfdfb3640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efbf80bef00 0x7efbf80c2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.157+0000 7efbfe7b4640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efbf814ca60 0x7efbf814b7f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7efbf0007e90 tx=0x7efbf000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.161+0000 7efbee7fc640 1 -- 172.21.15.83:0/2140692504 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf0014420 con 0x7efbf814ca60 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.161+0000 7efbee7fc640 1 -- 172.21.15.83:0/2140692504 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf0019070 con 0x7efbf814ca60 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.161+0000 7efbee7fc640 1 -- 172.21.15.83:0/2140692504 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf0014820 con 0x7efbf814ca60 2024-09-19T07:34:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.161+0000 7efc00217640 1 -- 172.21.15.83:0/2140692504 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efbf80c06c0 con 0x7efbf814ca60 2024-09-19T07:34:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.161+0000 7efc00217640 1 -- 172.21.15.83:0/2140692504 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efbf81efb00 con 0x7efbf814ca60 2024-09-19T07:34:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.161+0000 7efc00217640 1 -- 172.21.15.83:0/2140692504 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efbf8156840 con 0x7efbf814ca60 2024-09-19T07:34:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.161+0000 7efbee7fc640 1 -- 172.21.15.83:0/2140692504 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efbf00149c0 con 0x7efbf814ca60 2024-09-19T07:34:01.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.161+0000 7efbee7fc640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efbdc0678c0 0x7efbdc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:01.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.161+0000 7efbee7fc640 1 -- 172.21.15.83:0/2140692504 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7efbf0086f70 con 0x7efbf814ca60 2024-09-19T07:34:01.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.161+0000 7efbfdfb3640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efbdc0678c0 0x7efbdc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:01.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.161+0000 7efbfdfb3640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efbdc0678c0 0x7efbdc069d80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7efbe80097a0 tx=0x7efbe8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.165+0000 7efbee7fc640 1 -- 172.21.15.83:0/2140692504 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7efbf0057000 con 0x7efbf814ca60 2024-09-19T07:34:01.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.361+0000 7efc00217640 1 -- 172.21.15.83:0/2140692504 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efbf80aba80 con 0x7efbf814ca60 2024-09-19T07:34:01.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.361+0000 7efbee7fc640 1 -- 172.21.15.83:0/2140692504 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7efbf0056900 con 0x7efbf814ca60 2024-09-19T07:34:01.358 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:01.358 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:01.358 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:34:01.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.365+0000 7efc00217640 1 -- 172.21.15.83:0/2140692504 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efbdc0678c0 msgr2=0x7efbdc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:01.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.365+0000 7efc00217640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efbdc0678c0 0x7efbdc069d80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7efbe80097a0 tx=0x7efbe8008040 comp rx=0 tx=0).stop 2024-09-19T07:34:01.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.365+0000 7efc00217640 1 -- 172.21.15.83:0/2140692504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efbf814ca60 msgr2=0x7efbf814b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:01.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.365+0000 7efc00217640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efbf814ca60 0x7efbf814b7f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7efbf0007e90 tx=0x7efbf000e4b0 comp rx=0 tx=0).stop 2024-09-19T07:34:01.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.365+0000 7efc00217640 1 -- 172.21.15.83:0/2140692504 shutdown_connections 2024-09-19T07:34:01.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.365+0000 7efc00217640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efbdc0678c0 0x7efbdc069d80 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:01.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.365+0000 7efc00217640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efbf814ca60 0x7efbf814b7f0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:01.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.365+0000 7efc00217640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efbf80cbb00 0x7efbf815fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:01.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.365+0000 7efc00217640 1 --2- 172.21.15.83:0/2140692504 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efbf80bef00 0x7efbf80c2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:01.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.365+0000 7efc00217640 1 -- 172.21.15.83:0/2140692504 >> 172.21.15.83:0/2140692504 conn(0x7efbf8072470 msgr2=0x7efbf80ab290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:01.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.365+0000 7efc00217640 1 -- 172.21.15.83:0/2140692504 shutdown_connections 2024-09-19T07:34:01.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:01.365+0000 7efc00217640 1 -- 172.21.15.83:0/2140692504 wait complete. 2024-09-19T07:34:01.375 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:34:03.054 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:03.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.189+0000 7fc1d33e1640 1 Processor -- start 2024-09-19T07:34:03.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.189+0000 7fc1d33e1640 1 -- start start 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d33e1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc1cc071100 0x7fc1cc1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d33e1640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc1cc0c28e0 0x7fc1cc150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d33e1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1cc0a9e60 con 0x7fc1cc151480 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d33e1640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc1cc071b50 con 0x7fc1cc071100 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d33e1640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc1cc0c1ff0 con 0x7fc1cc0c28e0 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d117d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc1cc071100 0x7fc1cc1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d197e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc1cc151480 0x7fc1cc06dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41382/0 (socket says 172.21.15.83:41382) 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d197e640 1 -- 172.21.15.83:0/609008194 learned_addr learned my addr 172.21.15.83:0/609008194 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d117d640 1 -- 172.21.15.83:0/609008194 >> v2:172.21.15.184:3300/0 conn(0x7fc1cc0c28e0 msgr2=0x7fc1cc150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d097c640 1 --2- 172.21.15.83:0/609008194 >> v2:172.21.15.184:3300/0 conn(0x7fc1cc0c28e0 0x7fc1cc150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1c3fff640 1 -- 172.21.15.83:0/609008194 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1220481580 0 0) 0x7fc1cc0a9e60 con 0x7fc1cc151480 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d117d640 1 --2- 172.21.15.83:0/609008194 >> v2:172.21.15.184:3300/0 conn(0x7fc1cc0c28e0 0x7fc1cc150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d117d640 1 -- 172.21.15.83:0/609008194 >> v1:172.21.15.83:6789/0 conn(0x7fc1cc151480 legacy=0x7fc1cc06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d117d640 1 -- 172.21.15.83:0/609008194 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1cc0aa140 con 0x7fc1cc071100 2024-09-19T07:34:03.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1d117d640 1 --2- 172.21.15.83:0/609008194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1cc071100 0x7fc1cc1545e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0c2890 tx=0x7fc1bc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0756fc99a292ef0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:03.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1c3fff640 1 -- 172.21.15.83:0/609008194 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc0b0070 con 0x7fc1cc071100 2024-09-19T07:34:03.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1c3fff640 1 -- 172.21.15.83:0/609008194 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1bc0ab600 con 0x7fc1cc071100 2024-09-19T07:34:03.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.193+0000 7fc1c3fff640 1 -- 172.21.15.83:0/609008194 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc0ab9d0 con 0x7fc1cc071100 2024-09-19T07:34:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 -- 172.21.15.83:0/609008194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1cc071100 msgr2=0x7fc1cc1545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 --2- 172.21.15.83:0/609008194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1cc071100 0x7fc1cc1545e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0c2890 tx=0x7fc1bc01c3f0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 -- 172.21.15.83:0/609008194 shutdown_connections 2024-09-19T07:34:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 --2- 172.21.15.83:0/609008194 >> v2:172.21.15.184:3300/0 conn(0x7fc1cc0c28e0 0x7fc1cc150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 --2- 172.21.15.83:0/609008194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1cc071100 0x7fc1cc1545e0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 -- 172.21.15.83:0/609008194 >> 172.21.15.83:0/609008194 conn(0x7fc1cc06bc80 msgr2=0x7fc1cc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 -- 172.21.15.83:0/609008194 shutdown_connections 2024-09-19T07:34:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 -- 172.21.15.83:0/609008194 wait complete. 2024-09-19T07:34:03.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 Processor -- start 2024-09-19T07:34:03.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 -- start start 2024-09-19T07:34:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1cc071100 0x7fc1cc073540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1cc0c28e0 0x7fc1cc145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1cc151480 0x7fc1cc145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc071660 con 0x7fc1cc151480 2024-09-19T07:34:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc0714e0 con 0x7fc1cc0c28e0 2024-09-19T07:34:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc1cc160c60 con 0x7fc1cc071100 2024-09-19T07:34:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d197e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1cc151480 0x7fc1cc145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d197e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1cc151480 0x7fc1cc145c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49792/0 (socket says 172.21.15.83:49792) 2024-09-19T07:34:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d197e640 1 -- 172.21.15.83:0/242995021 learned_addr learned my addr 172.21.15.83:0/242995021 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d197e640 1 -- 172.21.15.83:0/242995021 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1cc071100 msgr2=0x7fc1cc073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d197e640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1cc071100 0x7fc1cc073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d097c640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1cc0c28e0 0x7fc1cc145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d197e640 1 -- 172.21.15.83:0/242995021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1cc0c28e0 msgr2=0x7fc1cc145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d197e640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1cc0c28e0 0x7fc1cc145710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d197e640 1 -- 172.21.15.83:0/242995021 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1cc1520a0 con 0x7fc1cc151480 2024-09-19T07:34:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d197e640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1cc151480 0x7fc1cc145c50 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400c3f0 tx=0x7fc1c400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1c1ffb640 1 -- 172.21.15.83:0/242995021 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c40074e0 con 0x7fc1cc151480 2024-09-19T07:34:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1c1ffb640 1 -- 172.21.15.83:0/242995021 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1c40047f0 con 0x7fc1cc151480 2024-09-19T07:34:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 -- 172.21.15.83:0/242995021 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1cc14d190 con 0x7fc1cc151480 2024-09-19T07:34:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1c1ffb640 1 -- 172.21.15.83:0/242995021 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4007680 con 0x7fc1cc151480 2024-09-19T07:34:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.197+0000 7fc1d33e1640 1 -- 172.21.15.83:0/242995021 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1cc153970 con 0x7fc1cc151480 2024-09-19T07:34:03.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.201+0000 7fc1d33e1640 1 -- 172.21.15.83:0/242995021 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1cc1545e0 con 0x7fc1cc151480 2024-09-19T07:34:03.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.201+0000 7fc1c1ffb640 1 -- 172.21.15.83:0/242995021 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc1c4011070 con 0x7fc1cc151480 2024-09-19T07:34:03.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.201+0000 7fc1c1ffb640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1b80678c0 0x7fc1b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.201+0000 7fc1c1ffb640 1 -- 172.21.15.83:0/242995021 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7fc1c4086280 con 0x7fc1cc151480 2024-09-19T07:34:03.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.201+0000 7fc1d117d640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1b80678c0 0x7fc1b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.201+0000 7fc1d117d640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1b80678c0 0x7fc1b8069d80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc0ad000 tx=0x7fc1bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:03.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.205+0000 7fc1c1ffb640 1 -- 172.21.15.83:0/242995021 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc1c40563a0 con 0x7fc1cc151480 2024-09-19T07:34:03.376 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:03.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 -- 172.21.15.83:0/242995021 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1cc0c4f30 con 0x7fc1cc151480 2024-09-19T07:34:03.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1c1ffb640 1 -- 172.21.15.83:0/242995021 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7fc1c4055ca0 con 0x7fc1cc151480 2024-09-19T07:34:03.396 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:03.396 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:03.397 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:34:03.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 -- 172.21.15.83:0/242995021 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1b80678c0 msgr2=0x7fc1b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1b80678c0 0x7fc1b8069d80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc0ad000 tx=0x7fc1bc008040 comp rx=0 tx=0).stop 2024-09-19T07:34:03.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 -- 172.21.15.83:0/242995021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1cc151480 msgr2=0x7fc1cc145c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1cc151480 0x7fc1cc145c50 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400c3f0 tx=0x7fc1c400c8b0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 -- 172.21.15.83:0/242995021 shutdown_connections 2024-09-19T07:34:03.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1b80678c0 0x7fc1b8069d80 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1cc151480 0x7fc1cc145c50 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1cc0c28e0 0x7fc1cc145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 --2- 172.21.15.83:0/242995021 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1cc071100 0x7fc1cc073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 -- 172.21.15.83:0/242995021 >> 172.21.15.83:0/242995021 conn(0x7fc1cc06bc80 msgr2=0x7fc1cc14f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:03.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 -- 172.21.15.83:0/242995021 shutdown_connections 2024-09-19T07:34:03.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.401+0000 7fc1d33e1640 1 -- 172.21.15.83:0/242995021 wait complete. 2024-09-19T07:34:03.414 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-19T07:34:03.415 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-19T07:34:03.415 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:34:03.432 INFO:tasks.ceph.mds.d:Stopped 2024-09-19T07:34:03.432 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:03.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdd144b640 1 Processor -- start 2024-09-19T07:34:03.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdd144b640 1 -- start start 2024-09-19T07:34:03.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdd144b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc0c3130 0x7efdcc0ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdd144b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdcc0c3c20 con 0x7efdcc06da00 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdd144b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdcc0c4990 con 0x7efdcc06de40 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdd144b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efdcc0c4c70 con 0x7efdcc0c3130 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdca7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7efdcc06da00 0x7efdcc0c3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41392/0 (socket says 172.21.15.83:41392) 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdca7fc640 1 -- 172.21.15.83:0/2395163412 learned_addr learned my addr 172.21.15.83:0/2395163412 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdcaffd640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc0c3130 0x7efdcc0ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 66583551 0 0) 0x7efdcc0c3c20 con 0x7efdcc06da00 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdb4003530 con 0x7efdcc06da00 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 16421286 0 0) 0x7efdb4003530 con 0x7efdcc06da00 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc0c3130 msgr2=0x7efdcc0ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc0c3130 0x7efdcc0ac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 >> v1:172.21.15.184:6789/0 conn(0x7efdcc06de40 legacy=0x7efdcc1563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc0ab7b0 con 0x7efdcc06da00 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7efdc00016a0 con 0x7efdcc06da00 2024-09-19T07:34:03.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 >> v1:172.21.15.83:6789/0 conn(0x7efdcc06da00 legacy=0x7efdcc0c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc0c3130 0x7efdb40039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb4003f30 0x7efdb4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdb4004d50 0x7efdb40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc0c4c70 con 0x7efdb4003f30 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efdb400dcf0 con 0x7efdb4004d50 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efdb400def0 con 0x7efdcc0c3130 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdca7fc640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb4003f30 0x7efdb4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdcb7fe640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdb4004d50 0x7efdb40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdca7fc640 1 -- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc0c3130 msgr2=0x7efdb40039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdcaffd640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc0c3130 0x7efdb40039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdca7fc640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc0c3130 0x7efdb40039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdca7fc640 1 -- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdb4004d50 msgr2=0x7efdb40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:03.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdca7fc640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdb4004d50 0x7efdb40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdca7fc640 1 -- 172.21.15.83:0/2395163412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb4003530 con 0x7efdb4003f30 2024-09-19T07:34:03.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdcb7fe640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdb4004d50 0x7efdb40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:34:03.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.501+0000 7efdcaffd640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc0c3130 0x7efdb40039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:34:03.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdca7fc640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb4003f30 0x7efdb4004810 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c210 tx=0x7efdc001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ede8e2a6f158b29 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:03.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0007d60 con 0x7efdb4003f30 2024-09-19T07:34:03.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc00a9070 con 0x7efdb4003f30 2024-09-19T07:34:03.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdc9ffb640 1 -- 172.21.15.83:0/2395163412 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc00a5dc0 con 0x7efdb4003f30 2024-09-19T07:34:03.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 -- 172.21.15.83:0/2395163412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb4003f30 msgr2=0x7efdb4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 --2- 172.21.15.83:0/2395163412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb4003f30 0x7efdb4004810 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c210 tx=0x7efdc001cc60 comp rx=0 tx=0).stop 2024-09-19T07:34:03.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdcb7fe640 1 -- 172.21.15.83:0/2395163412 reap_dead start 2024-09-19T07:34:03.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 -- 172.21.15.83:0/2395163412 shutdown_connections 2024-09-19T07:34:03.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 -- 172.21.15.83:0/2395163412 >> 172.21.15.83:0/2395163412 conn(0x7efdcc0c5060 msgr2=0x7efdcc0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:03.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 -- 172.21.15.83:0/2395163412 shutdown_connections 2024-09-19T07:34:03.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 -- 172.21.15.83:0/2395163412 wait complete. 2024-09-19T07:34:03.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 Processor -- start 2024-09-19T07:34:03.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 -- start start 2024-09-19T07:34:03.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdb4003f30 0x7efdcc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdcc0a6470 0x7efdcc0a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc1e6f20 0x7efdcc1e73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc0c82a0 con 0x7efdcc0a6470 2024-09-19T07:34:03.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc0abd70 con 0x7efdb4003f30 2024-09-19T07:34:03.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdd144b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efdcc0ad520 con 0x7efdcc1e6f20 2024-09-19T07:34:03.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdca7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdcc0a6470 0x7efdcc0a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdca7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdcc0a6470 0x7efdcc0a68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49820/0 (socket says 172.21.15.83:49820) 2024-09-19T07:34:03.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.505+0000 7efdca7fc640 1 -- 172.21.15.83:0/2363054900 learned_addr learned my addr 172.21.15.83:0/2363054900 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:03.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdcb7fe640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc1e6f20 0x7efdcc1e73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdca7fc640 1 -- 172.21.15.83:0/2363054900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc1e6f20 msgr2=0x7efdcc1e73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdca7fc640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc1e6f20 0x7efdcc1e73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdca7fc640 1 -- 172.21.15.83:0/2363054900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdb4003f30 msgr2=0x7efdcc06eb30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:34:03.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdca7fc640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdb4003f30 0x7efdcc06eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdca7fc640 1 -- 172.21.15.83:0/2363054900 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc0c0450 con 0x7efdcc0a6470 2024-09-19T07:34:03.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdca7fc640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdcc0a6470 0x7efdcc0a68f0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c210 tx=0x7efdb4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:03.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdb37fe640 1 -- 172.21.15.83:0/2363054900 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc00bd070 con 0x7efdcc0a6470 2024-09-19T07:34:03.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdb37fe640 1 -- 172.21.15.83:0/2363054900 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc00047f0 con 0x7efdcc0a6470 2024-09-19T07:34:03.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdd144b640 1 -- 172.21.15.83:0/2363054900 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdcc0a6e30 con 0x7efdcc0a6470 2024-09-19T07:34:03.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdb37fe640 1 -- 172.21.15.83:0/2363054900 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0007d60 con 0x7efdcc0a6470 2024-09-19T07:34:03.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdd144b640 1 -- 172.21.15.83:0/2363054900 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdcc1ea810 con 0x7efdcc0a6470 2024-09-19T07:34:03.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdd144b640 1 -- 172.21.15.83:0/2363054900 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdcc0ac290 con 0x7efdcc0a6470 2024-09-19T07:34:03.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdb37fe640 1 -- 172.21.15.83:0/2363054900 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efdc00c0070 con 0x7efdcc0a6470 2024-09-19T07:34:03.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdb37fe640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efdb80678f0 0x7efdb8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdcaffd640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efdb80678f0 0x7efdb8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdb37fe640 1 -- 172.21.15.83:0/2363054900 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7efdc011ff20 con 0x7efdcc0a6470 2024-09-19T07:34:03.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.509+0000 7efdcaffd640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efdb80678f0 0x7efdb8069db0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0ac8a0 tx=0x7efdc400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:03.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.513+0000 7efdb37fe640 1 -- 172.21.15.83:0/2363054900 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7efdc0016970 con 0x7efdcc0a6470 2024-09-19T07:34:03.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80c0831640 1 Processor -- start 2024-09-19T07:34:03.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80c0831640 1 -- start start 2024-09-19T07:34:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80c0831640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f80bc06cef0 0x7f80bc0c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80c0831640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f80bc1534f0 0x7f80bc14fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80c0831640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80bc0c6b70 con 0x7f80bc06ddd0 2024-09-19T07:34:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80c0831640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f80bc06f730 con 0x7f80bc06cef0 2024-09-19T07:34:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80c0831640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f80bc06e210 con 0x7f80bc1534f0 2024-09-19T07:34:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80ba59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f80bc06cef0 0x7f80bc0c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80b9d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f80bc06ddd0 0x7f80bc0c6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41408/0 (socket says 172.21.15.83:41408) 2024-09-19T07:34:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80b9d9b640 1 -- 172.21.15.83:0/3999428035 learned_addr learned my addr 172.21.15.83:0/3999428035 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80bad9d640 1 --2- 172.21.15.83:0/3999428035 >> v2:172.21.15.184:3300/0 conn(0x7f80bc1534f0 0x7f80bc14fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80ba59c640 1 -- 172.21.15.83:0/3999428035 >> v2:172.21.15.184:3300/0 conn(0x7f80bc1534f0 msgr2=0x7f80bc14fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:03.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80ba59c640 1 --2- 172.21.15.83:0/3999428035 >> v2:172.21.15.184:3300/0 conn(0x7f80bc1534f0 0x7f80bc14fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80b959a640 1 -- 172.21.15.83:0/3999428035 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 579081586 0 0) 0x7f80bc0c6b70 con 0x7f80bc06ddd0 2024-09-19T07:34:03.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80ba59c640 1 -- 172.21.15.83:0/3999428035 >> v1:172.21.15.83:6789/0 conn(0x7f80bc06ddd0 legacy=0x7f80bc0c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.573+0000 7f80ba59c640 1 -- 172.21.15.83:0/3999428035 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80bc0c7fb0 con 0x7f80bc06cef0 2024-09-19T07:34:03.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80ba59c640 1 --2- 172.21.15.83:0/3999428035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80bc06cef0 0x7f80bc0c79d0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0a8210 tx=0x7f80b401c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44e27aefce9c1e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:03.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80b959a640 1 -- 172.21.15.83:0/3999428035 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b40a66e0 con 0x7f80bc06cef0 2024-09-19T07:34:03.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80b959a640 1 -- 172.21.15.83:0/3999428035 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80b40a6880 con 0x7f80bc06cef0 2024-09-19T07:34:03.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80b959a640 1 -- 172.21.15.83:0/3999428035 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b40a6c50 con 0x7f80bc06cef0 2024-09-19T07:34:03.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 -- 172.21.15.83:0/3999428035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80bc06cef0 msgr2=0x7f80bc0c79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 --2- 172.21.15.83:0/3999428035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80bc06cef0 0x7f80bc0c79d0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0a8210 tx=0x7f80b401c810 comp rx=0 tx=0).stop 2024-09-19T07:34:03.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 -- 172.21.15.83:0/3999428035 shutdown_connections 2024-09-19T07:34:03.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 --2- 172.21.15.83:0/3999428035 >> v2:172.21.15.184:3300/0 conn(0x7f80bc1534f0 0x7f80bc14fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 --2- 172.21.15.83:0/3999428035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80bc06cef0 0x7f80bc0c79d0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 -- 172.21.15.83:0/3999428035 >> 172.21.15.83:0/3999428035 conn(0x7f80bc0aa910 msgr2=0x7f80bc0c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:03.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 -- 172.21.15.83:0/3999428035 shutdown_connections 2024-09-19T07:34:03.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 -- 172.21.15.83:0/3999428035 wait complete. 2024-09-19T07:34:03.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 Processor -- start 2024-09-19T07:34:03.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 -- start start 2024-09-19T07:34:03.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80bc06cef0 0x7f80bc1540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80bc06ddd0 0x7f80bc14c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80bc1534f0 0x7f80bc14ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc06d2d0 con 0x7f80bc06cef0 2024-09-19T07:34:03.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc150420 con 0x7f80bc1534f0 2024-09-19T07:34:03.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f80bc06d450 con 0x7f80bc06ddd0 2024-09-19T07:34:03.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80ba59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80bc06cef0 0x7f80bc1540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80ba59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80bc06cef0 0x7f80bc1540c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49848/0 (socket says 172.21.15.83:49848) 2024-09-19T07:34:03.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80ba59c640 1 -- 172.21.15.83:0/1235427638 learned_addr learned my addr 172.21.15.83:0/1235427638 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:03.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80b9d9b640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80bc06ddd0 0x7f80bc14c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80bad9d640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80bc1534f0 0x7f80bc14ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80ba59c640 1 -- 172.21.15.83:0/1235427638 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80bc06ddd0 msgr2=0x7f80bc14c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80ba59c640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80bc06ddd0 0x7f80bc14c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80ba59c640 1 -- 172.21.15.83:0/1235427638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80bc1534f0 msgr2=0x7f80bc14ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80ba59c640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80bc1534f0 0x7f80bc14ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80ba59c640 1 -- 172.21.15.83:0/1235427638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80bc0ad480 con 0x7f80bc06cef0 2024-09-19T07:34:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80ba59c640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80bc06cef0 0x7f80bc1540c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f80b4019420 tx=0x7f80b40195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80b2ffd640 1 -- 172.21.15.83:0/1235427638 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b40be070 con 0x7f80bc06cef0 2024-09-19T07:34:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80b2ffd640 1 -- 172.21.15.83:0/1235427638 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80b40b8840 con 0x7f80bc06cef0 2024-09-19T07:34:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 -- 172.21.15.83:0/1235427638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80bc155e20 con 0x7f80bc06cef0 2024-09-19T07:34:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80b2ffd640 1 -- 172.21.15.83:0/1235427638 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b40b8c10 con 0x7f80bc06cef0 2024-09-19T07:34:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.577+0000 7f80c0831640 1 -- 172.21.15.83:0/1235427638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80bc0ad720 con 0x7f80bc06cef0 2024-09-19T07:34:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.581+0000 7f80c0831640 1 -- 172.21.15.83:0/1235427638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80bc0c79d0 con 0x7f80bc06cef0 2024-09-19T07:34:03.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.581+0000 7f80b2ffd640 1 -- 172.21.15.83:0/1235427638 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f80b40c1070 con 0x7f80bc06cef0 2024-09-19T07:34:03.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.581+0000 7f80b2ffd640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f809c0678c0 0x7f809c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:03.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.581+0000 7f80b2ffd640 1 -- 172.21.15.83:0/1235427638 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7f80b4121e50 con 0x7f80bc06cef0 2024-09-19T07:34:03.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.581+0000 7f80b9d9b640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f809c0678c0 0x7f809c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:03.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.581+0000 7f80b9d9b640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f809c0678c0 0x7f809c069d80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0a9750 tx=0x7f80a4007c10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:03.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.585+0000 7f80b2ffd640 1 -- 172.21.15.83:0/1235427638 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f80b40c8030 con 0x7f80bc06cef0 2024-09-19T07:34:03.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.713+0000 7efdd144b640 1 -- 172.21.15.83:0/2363054900 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efdcc157380 con 0x7efdcc0a6470 2024-09-19T07:34:03.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.713+0000 7efdb37fe640 1 -- 172.21.15.83:0/2363054900 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7efdc00f0040 con 0x7efdcc0a6470 2024-09-19T07:34:03.710 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:03.711 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:03.711 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:34:03.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.717+0000 7efdd144b640 1 -- 172.21.15.83:0/2363054900 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efdb80678f0 msgr2=0x7efdb8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.717+0000 7efdd144b640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efdb80678f0 0x7efdb8069db0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0ac8a0 tx=0x7efdc400b040 comp rx=0 tx=0).stop 2024-09-19T07:34:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.717+0000 7efdd144b640 1 -- 172.21.15.83:0/2363054900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdcc0a6470 msgr2=0x7efdcc0a68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.717+0000 7efdd144b640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdcc0a6470 0x7efdcc0a68f0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c210 tx=0x7efdb4005840 comp rx=0 tx=0).stop 2024-09-19T07:34:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.717+0000 7efdd144b640 1 -- 172.21.15.83:0/2363054900 shutdown_connections 2024-09-19T07:34:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.717+0000 7efdd144b640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efdb80678f0 0x7efdb8069db0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.717+0000 7efdd144b640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efdcc1e6f20 0x7efdcc1e73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.717+0000 7efdd144b640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdcc0a6470 0x7efdcc0a68f0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.717+0000 7efdd144b640 1 --2- 172.21.15.83:0/2363054900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efdb4003f30 0x7efdcc06eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.717+0000 7efdd144b640 1 -- 172.21.15.83:0/2363054900 >> 172.21.15.83:0/2363054900 conn(0x7efdcc0c5060 msgr2=0x7efdcc154a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.717+0000 7efdd144b640 1 -- 172.21.15.83:0/2363054900 shutdown_connections 2024-09-19T07:34:03.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.717+0000 7efdd144b640 1 -- 172.21.15.83:0/2363054900 wait complete. 2024-09-19T07:34:03.728 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:34:03.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.769+0000 7f80c0831640 1 -- 172.21.15.83:0/1235427638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80bc0ada90 con 0x7f80bc06cef0 2024-09-19T07:34:03.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.769+0000 7f80b2ffd640 1 -- 172.21.15.83:0/1235427638 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7f80b4016520 con 0x7f80bc06cef0 2024-09-19T07:34:03.766 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:03.766 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:03.766 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:34:03.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.773+0000 7f80c0831640 1 -- 172.21.15.83:0/1235427638 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f809c0678c0 msgr2=0x7f809c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.773+0000 7f80c0831640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f809c0678c0 0x7f809c069d80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0a9750 tx=0x7f80a4007c10 comp rx=0 tx=0).stop 2024-09-19T07:34:03.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.773+0000 7f80c0831640 1 -- 172.21.15.83:0/1235427638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80bc06cef0 msgr2=0x7f80bc1540c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:03.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.773+0000 7f80c0831640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80bc06cef0 0x7f80bc1540c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f80b4019420 tx=0x7f80b40195b0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.773+0000 7f80c0831640 1 -- 172.21.15.83:0/1235427638 shutdown_connections 2024-09-19T07:34:03.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.773+0000 7f80c0831640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f809c0678c0 0x7f809c069d80 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.773+0000 7f80c0831640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80bc1534f0 0x7f80bc14ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.773+0000 7f80c0831640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80bc06ddd0 0x7f80bc14c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.773+0000 7f80c0831640 1 --2- 172.21.15.83:0/1235427638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80bc06cef0 0x7f80bc1540c0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:03.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.773+0000 7f80c0831640 1 -- 172.21.15.83:0/1235427638 >> 172.21.15.83:0/1235427638 conn(0x7f80bc0aa910 msgr2=0x7f80bc0a82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:03.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.773+0000 7f80c0831640 1 -- 172.21.15.83:0/1235427638 shutdown_connections 2024-09-19T07:34:03.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:03.773+0000 7f80c0831640 1 -- 172.21.15.83:0/1235427638 wait complete. 2024-09-19T07:34:03.782 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:34:03.865 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:34:03.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:34:05.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:05.783 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.873+0000 7fe35ec42640 1 Processor -- start 2024-09-19T07:34:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.873+0000 7fe35ec42640 1 -- start start 2024-09-19T07:34:05.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35ec42640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe35814d9f0 0x7fe35806de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:05.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35ec42640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe358154360 0x7fe3580c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:05.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35ec42640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe35814ddd0 con 0x7fe3580c7510 2024-09-19T07:34:05.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35ec42640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe3580c8830 con 0x7fe358154360 2024-09-19T07:34:05.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35ec42640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe35806fdf0 con 0x7fe35814d9f0 2024-09-19T07:34:05.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35c9de640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe35814d9f0 0x7fe35806de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:05.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35d1df640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe358154360 0x7fe3580c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:05.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe357fff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fe3580c7510 0x7fe3580c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:34216/0 (socket says 172.21.15.83:34216) 2024-09-19T07:34:05.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe357fff640 1 -- 172.21.15.83:0/3600024144 learned_addr learned my addr 172.21.15.83:0/3600024144 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:05.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35d1df640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe358154360 0x7fe3580c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:55854/0 (socket says 172.21.15.83:55854) 2024-09-19T07:34:05.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35c9de640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe35814d9f0 0x7fe35806de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:46112/0 (socket says 172.21.15.83:46112) 2024-09-19T07:34:05.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35c9de640 1 -- 172.21.15.83:0/3600024144 >> v2:172.21.15.184:3300/0 conn(0x7fe358154360 msgr2=0x7fe3580c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:05.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35c9de640 1 --2- 172.21.15.83:0/3600024144 >> v2:172.21.15.184:3300/0 conn(0x7fe358154360 0x7fe3580c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:05.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35c9de640 1 -- 172.21.15.83:0/3600024144 >> v1:172.21.15.184:6789/0 conn(0x7fe3580c7510 legacy=0x7fe3580c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:05.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35c9de640 1 -- 172.21.15.83:0/3600024144 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3580c7f00 con 0x7fe35814d9f0 2024-09-19T07:34:05.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.877+0000 7fe35c9de640 1 --2- 172.21.15.83:0/3600024144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe35814d9f0 0x7fe35806de90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0097c0 tx=0x7fe34c01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b53c1a3a24df7a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:05.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe3577fe640 1 -- 172.21.15.83:0/3600024144 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0b0070 con 0x7fe35814d9f0 2024-09-19T07:34:05.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe3577fe640 1 -- 172.21.15.83:0/3600024144 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe34c0ab600 con 0x7fe35814d9f0 2024-09-19T07:34:05.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe3577fe640 1 -- 172.21.15.83:0/3600024144 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0ab9d0 con 0x7fe35814d9f0 2024-09-19T07:34:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 -- 172.21.15.83:0/3600024144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe35814d9f0 msgr2=0x7fe35806de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 --2- 172.21.15.83:0/3600024144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe35814d9f0 0x7fe35806de90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0097c0 tx=0x7fe34c01c410 comp rx=0 tx=0).stop 2024-09-19T07:34:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 -- 172.21.15.83:0/3600024144 shutdown_connections 2024-09-19T07:34:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 --2- 172.21.15.83:0/3600024144 >> v2:172.21.15.184:3300/0 conn(0x7fe358154360 0x7fe3580c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 --2- 172.21.15.83:0/3600024144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe35814d9f0 0x7fe35806de90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 -- 172.21.15.83:0/3600024144 >> 172.21.15.83:0/3600024144 conn(0x7fe3580c6110 msgr2=0x7fe3580c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 -- 172.21.15.83:0/3600024144 shutdown_connections 2024-09-19T07:34:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 -- 172.21.15.83:0/3600024144 wait complete. 2024-09-19T07:34:05.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 Processor -- start 2024-09-19T07:34:05.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 -- start start 2024-09-19T07:34:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3580c7510 0x7fe358149840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe35814d9f0 0x7fe3581410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe358154360 0x7fe358146b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe358160c00 con 0x7fe3580c7510 2024-09-19T07:34:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe358160a80 con 0x7fe35814d9f0 2024-09-19T07:34:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35ec42640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe358160d80 con 0x7fe358154360 2024-09-19T07:34:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35c9de640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3580c7510 0x7fe358149840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35c9de640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3580c7510 0x7fe358149840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49856/0 (socket says 172.21.15.83:49856) 2024-09-19T07:34:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35c9de640 1 -- 172.21.15.83:0/4134679445 learned_addr learned my addr 172.21.15.83:0/4134679445 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe357fff640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe35814d9f0 0x7fe3581410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35d1df640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe358154360 0x7fe358146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35c9de640 1 -- 172.21.15.83:0/4134679445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe358154360 msgr2=0x7fe358146b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35c9de640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe358154360 0x7fe358146b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35c9de640 1 -- 172.21.15.83:0/4134679445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe35814d9f0 msgr2=0x7fe3581410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35c9de640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe35814d9f0 0x7fe3581410e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.881+0000 7fe35c9de640 1 -- 172.21.15.83:0/4134679445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe358140290 con 0x7fe3580c7510 2024-09-19T07:34:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.885+0000 7fe35c9de640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3580c7510 0x7fe358149840 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0097c0 tx=0x7fe34c01ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.885+0000 7fe3557fa640 1 -- 172.21.15.83:0/4134679445 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0b0050 con 0x7fe3580c7510 2024-09-19T07:34:05.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.885+0000 7fe3557fa640 1 -- 172.21.15.83:0/4134679445 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe34c0bc070 con 0x7fe3580c7510 2024-09-19T07:34:05.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.885+0000 7fe35ec42640 1 -- 172.21.15.83:0/4134679445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe358140570 con 0x7fe3580c7510 2024-09-19T07:34:05.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.885+0000 7fe3557fa640 1 -- 172.21.15.83:0/4134679445 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c006e20 con 0x7fe3580c7510 2024-09-19T07:34:05.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.885+0000 7fe35ec42640 1 -- 172.21.15.83:0/4134679445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe358140b30 con 0x7fe3580c7510 2024-09-19T07:34:05.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.885+0000 7fe35ec42640 1 -- 172.21.15.83:0/4134679445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe35806de90 con 0x7fe3580c7510 2024-09-19T07:34:05.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.885+0000 7fe3557fa640 1 -- 172.21.15.83:0/4134679445 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe34c0b5810 con 0x7fe3580c7510 2024-09-19T07:34:05.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.885+0000 7fe3557fa640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe33c0678c0 0x7fe33c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:05.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.885+0000 7fe3557fa640 1 -- 172.21.15.83:0/4134679445 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7fe34c120f20 con 0x7fe3580c7510 2024-09-19T07:34:05.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.885+0000 7fe357fff640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe33c0678c0 0x7fe33c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:05.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.885+0000 7fe357fff640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe33c0678c0 0x7fe33c069d80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe3581512f0 tx=0x7fe344009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:05.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.893+0000 7fe3557fa640 1 -- 172.21.15.83:0/4134679445 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe34c016090 con 0x7fe3580c7510 2024-09-19T07:34:05.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.909+0000 7fc8048a8640 1 Processor -- start 2024-09-19T07:34:05.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.909+0000 7fc8048a8640 1 -- start start 2024-09-19T07:34:05.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc8048a8640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc80014d9f0 0x7fc80006de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:05.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc8048a8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc800154360 0x7fc8000c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:05.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc8048a8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc80014ddd0 con 0x7fc8000c7510 2024-09-19T07:34:05.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc8048a8640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc8000c8830 con 0x7fc800154360 2024-09-19T07:34:05.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc8048a8640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc80006fdf0 con 0x7fc80014d9f0 2024-09-19T07:34:05.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fed9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc800154360 0x7fc8000c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:05.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fdd9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc8000c7510 0x7fc8000c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41416/0 (socket says 172.21.15.83:41416) 2024-09-19T07:34:05.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fdd9b640 1 -- 172.21.15.83:0/4263317993 learned_addr learned my addr 172.21.15.83:0/4263317993 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fe59c640 1 --2- 172.21.15.83:0/4263317993 >> v2:172.21.15.184:3300/0 conn(0x7fc80014d9f0 0x7fc80006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fed9d640 1 -- 172.21.15.83:0/4263317993 >> v2:172.21.15.184:3300/0 conn(0x7fc80014d9f0 msgr2=0x7fc80006de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fed9d640 1 --2- 172.21.15.83:0/4263317993 >> v2:172.21.15.184:3300/0 conn(0x7fc80014d9f0 0x7fc80006de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fed9d640 1 -- 172.21.15.83:0/4263317993 >> v1:172.21.15.83:6789/0 conn(0x7fc8000c7510 legacy=0x7fc8000c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fd59a640 1 -- 172.21.15.83:0/4263317993 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 611799656 0 0) 0x7fc80014ddd0 con 0x7fc8000c7510 2024-09-19T07:34:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fed9d640 1 -- 172.21.15.83:0/4263317993 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8000c7f00 con 0x7fc800154360 2024-09-19T07:34:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fed9d640 1 --2- 172.21.15.83:0/4263317993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc800154360 0x7fc8000c4010 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fc800063a30 tx=0x7fc7f001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06bcd4aa93a3fc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fd59a640 1 -- 172.21.15.83:0/4263317993 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00b2070 con 0x7fc800154360 2024-09-19T07:34:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fd59a640 1 -- 172.21.15.83:0/4263317993 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7f00a5760 con 0x7fc800154360 2024-09-19T07:34:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc7fd59a640 1 -- 172.21.15.83:0/4263317993 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00a5b30 con 0x7fc800154360 2024-09-19T07:34:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc8048a8640 1 -- 172.21.15.83:0/4263317993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc800154360 msgr2=0x7fc8000c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc8048a8640 1 --2- 172.21.15.83:0/4263317993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc800154360 0x7fc8000c4010 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fc800063a30 tx=0x7fc7f001c8a0 comp rx=0 tx=0).stop 2024-09-19T07:34:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc8048a8640 1 -- 172.21.15.83:0/4263317993 shutdown_connections 2024-09-19T07:34:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc8048a8640 1 --2- 172.21.15.83:0/4263317993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc800154360 0x7fc8000c4010 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc8048a8640 1 --2- 172.21.15.83:0/4263317993 >> v2:172.21.15.184:3300/0 conn(0x7fc80014d9f0 0x7fc80006de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.913+0000 7fc8048a8640 1 -- 172.21.15.83:0/4263317993 >> 172.21.15.83:0/4263317993 conn(0x7fc8000c6110 msgr2=0x7fc8000c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc8048a8640 1 -- 172.21.15.83:0/4263317993 shutdown_connections 2024-09-19T07:34:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc8048a8640 1 -- 172.21.15.83:0/4263317993 wait complete. 2024-09-19T07:34:05.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc8048a8640 1 Processor -- start 2024-09-19T07:34:05.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc8048a8640 1 -- start start 2024-09-19T07:34:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc8048a8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8000c7510 0x7fc800146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc8048a8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc80014d9f0 0x7fc800142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc8048a8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc800154360 0x7fc8001431c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc8048a8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc800160c00 con 0x7fc80014d9f0 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc8048a8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc800160a80 con 0x7fc8000c7510 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc8048a8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc800160d80 con 0x7fc800154360 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7fdd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc80014d9f0 0x7fc800142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7fdd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc80014d9f0 0x7fc800142c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49872/0 (socket says 172.21.15.83:49872) 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7fdd9b640 1 -- 172.21.15.83:0/3550614139 learned_addr learned my addr 172.21.15.83:0/3550614139 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7fed9d640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc800154360 0x7fc8001431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7fe59c640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8000c7510 0x7fc800146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7fdd9b640 1 -- 172.21.15.83:0/3550614139 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc800154360 msgr2=0x7fc8001431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7fdd9b640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc800154360 0x7fc8001431c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7fdd9b640 1 -- 172.21.15.83:0/3550614139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8000c7510 msgr2=0x7fc800146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7fdd9b640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8000c7510 0x7fc800146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7fdd9b640 1 -- 172.21.15.83:0/3550614139 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc800142230 con 0x7fc80014d9f0 2024-09-19T07:34:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7fdd9b640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc80014d9f0 0x7fc800142c80 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc7e800c480 tx=0x7fc7e800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:05.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7f6ffd640 1 -- 172.21.15.83:0/3550614139 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8018070 con 0x7fc80014d9f0 2024-09-19T07:34:05.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7f6ffd640 1 -- 172.21.15.83:0/3550614139 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7e8007500 con 0x7fc80014d9f0 2024-09-19T07:34:05.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc8048a8640 1 -- 172.21.15.83:0/3550614139 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc800143700 con 0x7fc80014d9f0 2024-09-19T07:34:05.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc7f6ffd640 1 -- 172.21.15.83:0/3550614139 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8004a30 con 0x7fc80014d9f0 2024-09-19T07:34:05.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.917+0000 7fc8048a8640 1 -- 172.21.15.83:0/3550614139 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc800156350 con 0x7fc80014d9f0 2024-09-19T07:34:05.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.921+0000 7fc8048a8640 1 -- 172.21.15.83:0/3550614139 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc80006de90 con 0x7fc80014d9f0 2024-09-19T07:34:05.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.921+0000 7fc7f6ffd640 1 -- 172.21.15.83:0/3550614139 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc7e801b070 con 0x7fc80014d9f0 2024-09-19T07:34:05.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.921+0000 7fc7f6ffd640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc7e00678f0 0x7fc7e0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:05.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.921+0000 7fc7fe59c640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc7e00678f0 0x7fc7e0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:05.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.921+0000 7fc7f6ffd640 1 -- 172.21.15.83:0/3550614139 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8357+0+0 (secure 0 0 0) 0x7fc7e8085920 con 0x7fc80014d9f0 2024-09-19T07:34:05.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.921+0000 7fc7fe59c640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc7e00678f0 0x7fc7e0069db0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc7f800b930 tx=0x7fc7f8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:05.925+0000 7fc7f6ffd640 1 -- 172.21.15.83:0/3550614139 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc7e8055a40 con 0x7fc80014d9f0 2024-09-19T07:34:06.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.109+0000 7fe35ec42640 1 -- 172.21.15.83:0/4134679445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe35806faf0 con 0x7fe3580c7510 2024-09-19T07:34:06.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.109+0000 7fe3557fa640 1 -- 172.21.15.83:0/4134679445 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7fe34c0f0f20 con 0x7fe3580c7510 2024-09-19T07:34:06.105 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:06.105 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:06.105 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:34:06.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.113+0000 7fe35ec42640 1 -- 172.21.15.83:0/4134679445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe33c0678c0 msgr2=0x7fe33c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.113+0000 7fe35ec42640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe33c0678c0 0x7fe33c069d80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe3581512f0 tx=0x7fe344009040 comp rx=0 tx=0).stop 2024-09-19T07:34:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.113+0000 7fe35ec42640 1 -- 172.21.15.83:0/4134679445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3580c7510 msgr2=0x7fe358149840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.113+0000 7fe35ec42640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3580c7510 0x7fe358149840 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0097c0 tx=0x7fe34c01ac50 comp rx=0 tx=0).stop 2024-09-19T07:34:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.113+0000 7fe35ec42640 1 -- 172.21.15.83:0/4134679445 shutdown_connections 2024-09-19T07:34:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.113+0000 7fe35ec42640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe33c0678c0 0x7fe33c069d80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.113+0000 7fe35ec42640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe358154360 0x7fe358146b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.113+0000 7fe35ec42640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe35814d9f0 0x7fe3581410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.113+0000 7fe35ec42640 1 --2- 172.21.15.83:0/4134679445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3580c7510 0x7fe358149840 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.113+0000 7fe35ec42640 1 -- 172.21.15.83:0/4134679445 >> 172.21.15.83:0/4134679445 conn(0x7fe3580c6110 msgr2=0x7fe35814ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.113+0000 7fe35ec42640 1 -- 172.21.15.83:0/4134679445 shutdown_connections 2024-09-19T07:34:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.113+0000 7fe35ec42640 1 -- 172.21.15.83:0/4134679445 wait complete. 2024-09-19T07:34:06.124 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:34:06.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.133+0000 7fc8048a8640 1 -- 172.21.15.83:0/3550614139 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc80006faf0 con 0x7fc80014d9f0 2024-09-19T07:34:06.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.133+0000 7fc7f6ffd640 1 -- 172.21.15.83:0/3550614139 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6545 (secure 0 0 0) 0x7fc7e8055340 con 0x7fc80014d9f0 2024-09-19T07:34:06.129 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:06.129 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"btime":"2024-09-19T07:33:34:310965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:32:41.308698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":22,"addr":"172.21.15.83:6837/1203406606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1203406606},{"type":"v1","addr":"172.21.15.83:6837","nonce":1203406606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6561,"qdb_cluster":[6561]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:06.130 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 110 2024-09-19T07:34:06.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.137+0000 7fc8048a8640 1 -- 172.21.15.83:0/3550614139 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc7e00678f0 msgr2=0x7fc7e0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:06.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.137+0000 7fc8048a8640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc7e00678f0 0x7fc7e0069db0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc7f800b930 tx=0x7fc7f8008040 comp rx=0 tx=0).stop 2024-09-19T07:34:06.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.137+0000 7fc8048a8640 1 -- 172.21.15.83:0/3550614139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc80014d9f0 msgr2=0x7fc800142c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:06.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.137+0000 7fc8048a8640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc80014d9f0 0x7fc800142c80 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc7e800c480 tx=0x7fc7e800c940 comp rx=0 tx=0).stop 2024-09-19T07:34:06.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.137+0000 7fc8048a8640 1 -- 172.21.15.83:0/3550614139 shutdown_connections 2024-09-19T07:34:06.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.137+0000 7fc8048a8640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc7e00678f0 0x7fc7e0069db0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:06.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.137+0000 7fc8048a8640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc800154360 0x7fc8001431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:06.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.137+0000 7fc8048a8640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc80014d9f0 0x7fc800142c80 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:06.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.137+0000 7fc8048a8640 1 --2- 172.21.15.83:0/3550614139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8000c7510 0x7fc800146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:06.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.137+0000 7fc8048a8640 1 -- 172.21.15.83:0/3550614139 >> 172.21.15.83:0/3550614139 conn(0x7fc8000c6110 msgr2=0x7fc800151c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:06.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.137+0000 7fc8048a8640 1 -- 172.21.15.83:0/3550614139 shutdown_connections 2024-09-19T07:34:06.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:06.137+0000 7fc8048a8640 1 -- 172.21.15.83:0/3550614139 wait complete. 2024-09-19T07:34:06.147 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:34:08.125 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:08.148 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.253+0000 7f657c7be640 1 Processor -- start 2024-09-19T07:34:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.253+0000 7f657c7be640 1 -- start start 2024-09-19T07:34:08.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f657c7be640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65740c7510 0x7f657406de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f657c7be640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6574154360 0x7f65740c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f657c7be640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f657414ddd0 con 0x7f657414d9f0 2024-09-19T07:34:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f657c7be640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f65740c8830 con 0x7f6574154360 2024-09-19T07:34:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f657c7be640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f657406fdf0 con 0x7f65740c7510 2024-09-19T07:34:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f657a55a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f657414d9f0 0x7f65740c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41432/0 (socket says 172.21.15.83:41432) 2024-09-19T07:34:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f657a55a640 1 -- 172.21.15.83:0/1393362476 learned_addr learned my addr 172.21.15.83:0/1393362476 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 636294860 0 0) 0x7f657414ddd0 con 0x7f657414d9f0 2024-09-19T07:34:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f657ad5b640 1 --2- 172.21.15.83:0/1393362476 >> v2:172.21.15.184:3300/0 conn(0x7f6574154360 0x7f65740c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6560003530 con 0x7f657414d9f0 2024-09-19T07:34:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579d59640 1 --2- 172.21.15.83:0/1393362476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65740c7510 0x7f657406de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2143554985 0 0) 0x7f6560003530 con 0x7f657414d9f0 2024-09-19T07:34:08.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65740c7510 msgr2=0x7f657406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 --2- 172.21.15.83:0/1393362476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65740c7510 0x7f657406de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 >> v2:172.21.15.184:3300/0 conn(0x7f6574154360 msgr2=0x7f65740c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 --2- 172.21.15.83:0/1393362476 >> v2:172.21.15.184:3300/0 conn(0x7f6574154360 0x7f65740c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65740c7f00 con 0x7f657414d9f0 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f6570001680 con 0x7f657414d9f0 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 >> v1:172.21.15.83:6789/0 conn(0x7f657414d9f0 legacy=0x7f65740c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 --2- 172.21.15.83:0/1393362476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65600039c0 0x7f6560003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 --2- 172.21.15.83:0/1393362476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6560004810 0x7f6560004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 --2- 172.21.15.83:0/1393362476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65740c7510 0x7f65600051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f65740c8830 con 0x7f6560004810 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f657406fdf0 con 0x7f65600039c0 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f656000ba20 con 0x7f65740c7510 2024-09-19T07:34:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579d59640 1 --2- 172.21.15.83:0/1393362476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6560004810 0x7f6560004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f657a55a640 1 --2- 172.21.15.83:0/1393362476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65600039c0 0x7f6560003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f657ad5b640 1 --2- 172.21.15.83:0/1393362476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65740c7510 0x7f65600051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579d59640 1 -- 172.21.15.83:0/1393362476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65740c7510 msgr2=0x7f65600051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:08.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579d59640 1 --2- 172.21.15.83:0/1393362476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65740c7510 0x7f65600051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579d59640 1 -- 172.21.15.83:0/1393362476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65600039c0 msgr2=0x7f6560003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:08.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579d59640 1 --2- 172.21.15.83:0/1393362476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65600039c0 0x7f6560003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.257+0000 7f6579d59640 1 -- 172.21.15.83:0/1393362476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6560003530 con 0x7f6560004810 2024-09-19T07:34:08.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.261+0000 7f6579d59640 1 --2- 172.21.15.83:0/1393362476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6560004810 0x7f6560004c90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6564009b10 tx=0x7f656401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f6ce68a8f759022 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:08.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.261+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65640096b0 con 0x7f6560004810 2024-09-19T07:34:08.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.261+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65640a9470 con 0x7f6560004810 2024-09-19T07:34:08.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.261+0000 7f6579558640 1 -- 172.21.15.83:0/1393362476 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65640a9860 con 0x7f6560004810 2024-09-19T07:34:08.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.261+0000 7f657c7be640 1 -- 172.21.15.83:0/1393362476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6560004810 msgr2=0x7f6560004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.261+0000 7f657c7be640 1 --2- 172.21.15.83:0/1393362476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6560004810 0x7f6560004c90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6564009b10 tx=0x7f656401ca70 comp rx=0 tx=0).stop 2024-09-19T07:34:08.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.261+0000 7f657ad5b640 1 -- 172.21.15.83:0/1393362476 reap_dead start 2024-09-19T07:34:08.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.261+0000 7f657c7be640 1 -- 172.21.15.83:0/1393362476 shutdown_connections 2024-09-19T07:34:08.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.261+0000 7f657c7be640 1 -- 172.21.15.83:0/1393362476 >> 172.21.15.83:0/1393362476 conn(0x7f65740c6110 msgr2=0x7f65740c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:08.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.261+0000 7f657c7be640 1 -- 172.21.15.83:0/1393362476 shutdown_connections 2024-09-19T07:34:08.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.261+0000 7f657c7be640 1 -- 172.21.15.83:0/1393362476 wait complete. 2024-09-19T07:34:08.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.261+0000 7f657c7be640 1 Processor -- start 2024-09-19T07:34:08.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f657c7be640 1 -- start start 2024-09-19T07:34:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f657c7be640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6560004810 0x7f65741f0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f657c7be640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6574157500 0x7f65741529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f657c7be640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65741f08c0 0x7f6574140a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f657c7be640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f657415ebf0 con 0x7f6574157500 2024-09-19T07:34:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f657c7be640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f657415ea70 con 0x7f65741f08c0 2024-09-19T07:34:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f657c7be640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f657415ed70 con 0x7f6560004810 2024-09-19T07:34:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f6579d59640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6574157500 0x7f65741529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f6579d59640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6574157500 0x7f65741529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49898/0 (socket says 172.21.15.83:49898) 2024-09-19T07:34:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f6579d59640 1 -- 172.21.15.83:0/72960204 learned_addr learned my addr 172.21.15.83:0/72960204 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f657a55a640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6560004810 0x7f65741f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f657ad5b640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65741f08c0 0x7f6574140a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f6579d59640 1 -- 172.21.15.83:0/72960204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6560004810 msgr2=0x7f65741f0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f6579d59640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6560004810 0x7f65741f0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f6579d59640 1 -- 172.21.15.83:0/72960204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65741f08c0 msgr2=0x7f6574140a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f6579d59640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65741f08c0 0x7f6574140a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f6579d59640 1 -- 172.21.15.83:0/72960204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6574148f00 con 0x7f6574157500 2024-09-19T07:34:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f6579d59640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6574157500 0x7f65741529c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6564009b10 tx=0x7f6560005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f656affd640 1 -- 172.21.15.83:0/72960204 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656400f000 con 0x7f6574157500 2024-09-19T07:34:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f656affd640 1 -- 172.21.15.83:0/72960204 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65640a9c20 con 0x7f6574157500 2024-09-19T07:34:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f657c7be640 1 -- 172.21.15.83:0/72960204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65741f0d80 con 0x7f6574157500 2024-09-19T07:34:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f656affd640 1 -- 172.21.15.83:0/72960204 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65640ba3f0 con 0x7f6574157500 2024-09-19T07:34:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f657c7be640 1 -- 172.21.15.83:0/72960204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65741513d0 con 0x7f6574157500 2024-09-19T07:34:08.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f657c7be640 1 -- 172.21.15.83:0/72960204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65740c4010 con 0x7f6574157500 2024-09-19T07:34:08.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.265+0000 7f656affd640 1 -- 172.21.15.83:0/72960204 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f65640ba590 con 0x7f6574157500 2024-09-19T07:34:08.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.269+0000 7f656affd640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65580678c0 0x7f6558069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.269+0000 7f656affd640 1 -- 172.21.15.83:0/72960204 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8443+0+0 (secure 0 0 0) 0x7f6564121790 con 0x7f6574157500 2024-09-19T07:34:08.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.269+0000 7f657a55a640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65580678c0 0x7f6558069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.269+0000 7f657a55a640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65580678c0 0x7f6558069d80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6570001870 tx=0x7f6570009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:08.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.269+0000 7f656affd640 1 -- 172.21.15.83:0/72960204 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f65640ad020 con 0x7f6574157500 2024-09-19T07:34:08.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4bd54e640 1 Processor -- start 2024-09-19T07:34:08.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4bd54e640 1 -- start start 2024-09-19T07:34:08.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4bd54e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4b806cef0 0x7fb4b80c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4bd54e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb4b8070140 0x7fb4b81579a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4bd54e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b8070830 con 0x7fb4b8154180 2024-09-19T07:34:08.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4bd54e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb4b8153cf0 con 0x7fb4b806cef0 2024-09-19T07:34:08.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4bd54e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fb4b81539a0 con 0x7fb4b8070140 2024-09-19T07:34:08.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4b6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4b806cef0 0x7fb4b80c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4b6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4b806cef0 0x7fb4b80c2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49906/0 (socket says 172.21.15.83:49906) 2024-09-19T07:34:08.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4b6ffd640 1 -- 172.21.15.83:0/145035904 learned_addr learned my addr 172.21.15.83:0/145035904 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:08.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4ae7fc640 1 --2- 172.21.15.83:0/145035904 >> v2:172.21.15.184:3300/0 conn(0x7fb4b8070140 0x7fb4b81579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4b6ffd640 1 -- 172.21.15.83:0/145035904 >> v2:172.21.15.184:3300/0 conn(0x7fb4b8070140 msgr2=0x7fb4b81579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:08.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4b6ffd640 1 --2- 172.21.15.83:0/145035904 >> v2:172.21.15.184:3300/0 conn(0x7fb4b8070140 0x7fb4b81579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4b6ffd640 1 -- 172.21.15.83:0/145035904 >> v1:172.21.15.184:6789/0 conn(0x7fb4b8154180 legacy=0x7fb4b8150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:08.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.289+0000 7fb4b6ffd640 1 -- 172.21.15.83:0/145035904 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b806f2e0 con 0x7fb4b806cef0 2024-09-19T07:34:08.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.293+0000 7fb4b6ffd640 1 --2- 172.21.15.83:0/145035904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b806cef0 0x7fb4b80c2f90 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80c2920 tx=0x7fb4a801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab25562d93ea43e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:08.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.293+0000 7fb4b67fc640 1 -- 172.21.15.83:0/145035904 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a80b0070 con 0x7fb4b806cef0 2024-09-19T07:34:08.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.293+0000 7fb4b67fc640 1 -- 172.21.15.83:0/145035904 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4a80ab600 con 0x7fb4b806cef0 2024-09-19T07:34:08.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.293+0000 7fb4b67fc640 1 -- 172.21.15.83:0/145035904 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a80ab9d0 con 0x7fb4b806cef0 2024-09-19T07:34:08.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.293+0000 7fb4bd54e640 1 -- 172.21.15.83:0/145035904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b806cef0 msgr2=0x7fb4b80c2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.293+0000 7fb4bd54e640 1 --2- 172.21.15.83:0/145035904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b806cef0 0x7fb4b80c2f90 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80c2920 tx=0x7fb4a801c3c0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.293+0000 7fb4bd54e640 1 -- 172.21.15.83:0/145035904 shutdown_connections 2024-09-19T07:34:08.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.293+0000 7fb4bd54e640 1 --2- 172.21.15.83:0/145035904 >> v2:172.21.15.184:3300/0 conn(0x7fb4b8070140 0x7fb4b81579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.293+0000 7fb4bd54e640 1 --2- 172.21.15.83:0/145035904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b806cef0 0x7fb4b80c2f90 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.293+0000 7fb4bd54e640 1 -- 172.21.15.83:0/145035904 >> 172.21.15.83:0/145035904 conn(0x7fb4b80c5090 msgr2=0x7fb4b80c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:08.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.293+0000 7fb4bd54e640 1 -- 172.21.15.83:0/145035904 shutdown_connections 2024-09-19T07:34:08.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.293+0000 7fb4bd54e640 1 -- 172.21.15.83:0/145035904 wait complete. 2024-09-19T07:34:08.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4bd54e640 1 Processor -- start 2024-09-19T07:34:08.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4bd54e640 1 -- start start 2024-09-19T07:34:08.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4bd54e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b806cef0 0x7fb4b81552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4bd54e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b8070140 0x7fb4b8149e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4bd54e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4b8154180 0x7fb4b81ee7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4bd54e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b806d450 con 0x7fb4b806cef0 2024-09-19T07:34:08.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4bd54e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b806d2d0 con 0x7fb4b8070140 2024-09-19T07:34:08.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4bd54e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb4b8160b40 con 0x7fb4b8154180 2024-09-19T07:34:08.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4b6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b806cef0 0x7fb4b81552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4b6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b806cef0 0x7fb4b81552c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49916/0 (socket says 172.21.15.83:49916) 2024-09-19T07:34:08.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4b6ffd640 1 -- 172.21.15.83:0/319757580 learned_addr learned my addr 172.21.15.83:0/319757580 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:08.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4b77fe640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4b8154180 0x7fb4b81ee7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4b6ffd640 1 -- 172.21.15.83:0/319757580 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4b8154180 msgr2=0x7fb4b81ee7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4b6ffd640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4b8154180 0x7fb4b81ee7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4b6ffd640 1 -- 172.21.15.83:0/319757580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b8070140 msgr2=0x7fb4b8149e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4b6ffd640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b8070140 0x7fb4b8149e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4b6ffd640 1 -- 172.21.15.83:0/319757580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b8152f40 con 0x7fb4b806cef0 2024-09-19T07:34:08.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4b6ffd640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b806cef0 0x7fb4b81552c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80ae040 tx=0x7fb4a801ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:08.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4affff640 1 -- 172.21.15.83:0/319757580 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a80ba020 con 0x7fb4b806cef0 2024-09-19T07:34:08.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4affff640 1 -- 172.21.15.83:0/319757580 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4a80b6620 con 0x7fb4b806cef0 2024-09-19T07:34:08.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4affff640 1 -- 172.21.15.83:0/319757580 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a80b0040 con 0x7fb4b806cef0 2024-09-19T07:34:08.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4bd54e640 1 -- 172.21.15.83:0/319757580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b81559c0 con 0x7fb4b806cef0 2024-09-19T07:34:08.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4bd54e640 1 -- 172.21.15.83:0/319757580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b81edfd0 con 0x7fb4b806cef0 2024-09-19T07:34:08.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.297+0000 7fb4bd54e640 1 -- 172.21.15.83:0/319757580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4b80c2f90 con 0x7fb4b806cef0 2024-09-19T07:34:08.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.301+0000 7fb4affff640 1 -- 172.21.15.83:0/319757580 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb4a80bd020 con 0x7fb4b806cef0 2024-09-19T07:34:08.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.301+0000 7fb4affff640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4980678c0 0x7fb498069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.301+0000 7fb4affff640 1 -- 172.21.15.83:0/319757580 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8443+0+0 (secure 0 0 0) 0x7fb4a8120950 con 0x7fb4b806cef0 2024-09-19T07:34:08.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.301+0000 7fb4ae7fc640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4980678c0 0x7fb498069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.301+0000 7fb4ae7fc640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4980678c0 0x7fb498069d80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb4b81efb40 tx=0x7fb4a0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:08.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.305+0000 7fb4affff640 1 -- 172.21.15.83:0/319757580 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb4a80c4030 con 0x7fb4b806cef0 2024-09-19T07:34:08.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.465+0000 7f657c7be640 1 -- 172.21.15.83:0/72960204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f657406faf0 con 0x7f6574157500 2024-09-19T07:34:08.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.465+0000 7f656affd640 1 -- 172.21.15.83:0/72960204 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5711 (secure 0 0 0) 0x7f65640167c0 con 0x7f6574157500 2024-09-19T07:34:08.460 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:08.460 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":111,"btime":"2024-09-19T07:34:08:128296+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:08.128288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:08.460 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 111 2024-09-19T07:34:08.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.469+0000 7f657c7be640 1 -- 172.21.15.83:0/72960204 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65580678c0 msgr2=0x7f6558069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.469+0000 7f657c7be640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65580678c0 0x7f6558069d80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6570001870 tx=0x7f6570009040 comp rx=0 tx=0).stop 2024-09-19T07:34:08.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.469+0000 7f657c7be640 1 -- 172.21.15.83:0/72960204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6574157500 msgr2=0x7f65741529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.469+0000 7f657c7be640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6574157500 0x7f65741529c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6564009b10 tx=0x7f6560005840 comp rx=0 tx=0).stop 2024-09-19T07:34:08.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.469+0000 7f657c7be640 1 -- 172.21.15.83:0/72960204 shutdown_connections 2024-09-19T07:34:08.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.469+0000 7f657c7be640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65580678c0 0x7f6558069d80 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.469+0000 7f657c7be640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65741f08c0 0x7f6574140a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.469+0000 7f657c7be640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6574157500 0x7f65741529c0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.469+0000 7f657c7be640 1 --2- 172.21.15.83:0/72960204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6560004810 0x7f65741f0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.469+0000 7f657c7be640 1 -- 172.21.15.83:0/72960204 >> 172.21.15.83:0/72960204 conn(0x7f65740c6110 msgr2=0x7f657406e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:08.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.469+0000 7f657c7be640 1 -- 172.21.15.83:0/72960204 shutdown_connections 2024-09-19T07:34:08.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.469+0000 7f657c7be640 1 -- 172.21.15.83:0/72960204 wait complete. 2024-09-19T07:34:08.485 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-19T07:34:08.485 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:34:08.485 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:08.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.497+0000 7fb4bd54e640 1 -- 172.21.15.83:0/319757580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4b81504a0 con 0x7fb4b806cef0 2024-09-19T07:34:08.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.497+0000 7fb4affff640 1 -- 172.21.15.83:0/319757580 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5711 (secure 0 0 0) 0x7fb4a80160d0 con 0x7fb4b806cef0 2024-09-19T07:34:08.494 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:08.494 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":111,"btime":"2024-09-19T07:34:08:128296+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:08.128288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:08.495 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 111 2024-09-19T07:34:08.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.501+0000 7fb4bd54e640 1 -- 172.21.15.83:0/319757580 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4980678c0 msgr2=0x7fb498069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.501+0000 7fb4bd54e640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4980678c0 0x7fb498069d80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb4b81efb40 tx=0x7fb4a0008040 comp rx=0 tx=0).stop 2024-09-19T07:34:08.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.501+0000 7fb4bd54e640 1 -- 172.21.15.83:0/319757580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b806cef0 msgr2=0x7fb4b81552c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.501+0000 7fb4bd54e640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b806cef0 0x7fb4b81552c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80ae040 tx=0x7fb4a801ac00 comp rx=0 tx=0).stop 2024-09-19T07:34:08.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.501+0000 7fb4bd54e640 1 -- 172.21.15.83:0/319757580 shutdown_connections 2024-09-19T07:34:08.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.501+0000 7fb4bd54e640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4980678c0 0x7fb498069d80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.501+0000 7fb4bd54e640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4b8154180 0x7fb4b81ee7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.501+0000 7fb4bd54e640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b8070140 0x7fb4b8149e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.501+0000 7fb4bd54e640 1 --2- 172.21.15.83:0/319757580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b806cef0 0x7fb4b81552c0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.501+0000 7fb4bd54e640 1 -- 172.21.15.83:0/319757580 >> 172.21.15.83:0/319757580 conn(0x7fb4b80c5090 msgr2=0x7fb4b80c31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:08.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.501+0000 7fb4bd54e640 1 -- 172.21.15.83:0/319757580 shutdown_connections 2024-09-19T07:34:08.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.501+0000 7fb4bd54e640 1 -- 172.21.15.83:0/319757580 wait complete. 2024-09-19T07:34:08.513 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:34:08.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.597+0000 7fa308b2f640 1 Processor -- start 2024-09-19T07:34:08.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.597+0000 7fa308b2f640 1 -- start start 2024-09-19T07:34:08.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa308b2f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa30414f6f0 0x7fa304072060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa308b2f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa304073bf0 con 0x7fa3040c29b0 2024-09-19T07:34:08.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa308b2f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3041523c0 con 0x7fa3040c4280 2024-09-19T07:34:08.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa308b2f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa3040725a0 con 0x7fa30414f6f0 2024-09-19T07:34:08.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa302d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa3040c29b0 0x7fa3040c2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41434/0 (socket says 172.21.15.83:41434) 2024-09-19T07:34:08.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa302d9d640 1 -- 172.21.15.83:0/3501576650 learned_addr learned my addr 172.21.15.83:0/3501576650 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:08.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa30259c640 1 --2- 172.21.15.83:0/3501576650 >> v2:172.21.15.83:3300/0 conn(0x7fa30414f6f0 0x7fa304072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa30259c640 1 -- 172.21.15.83:0/3501576650 >> v1:172.21.15.184:6789/0 conn(0x7fa3040c4280 legacy=0x7fa30406d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:08.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa30259c640 1 -- 172.21.15.83:0/3501576650 >> v1:172.21.15.83:6789/0 conn(0x7fa3040c29b0 legacy=0x7fa3040c2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa30259c640 1 -- 172.21.15.83:0/3501576650 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3041526a0 con 0x7fa30414f6f0 2024-09-19T07:34:08.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa30159a640 1 -- 172.21.15.83:0/3501576650 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 945770784 0 0) 0x7fa2f4001120 con 0x7fa3040c29b0 2024-09-19T07:34:08.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa30259c640 1 --2- 172.21.15.83:0/3501576650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa30414f6f0 0x7fa304072060 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009d90 tx=0x7fa2fc01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe7c36b31ff967d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:08.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa30159a640 1 -- 172.21.15.83:0/3501576650 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc0b2070 con 0x7fa30414f6f0 2024-09-19T07:34:08.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa30159a640 1 -- 172.21.15.83:0/3501576650 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2fc0a58b0 con 0x7fa30414f6f0 2024-09-19T07:34:08.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa30159a640 1 -- 172.21.15.83:0/3501576650 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc0a5c80 con 0x7fa30414f6f0 2024-09-19T07:34:08.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa308b2f640 1 -- 172.21.15.83:0/3501576650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa30414f6f0 msgr2=0x7fa304072060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa308b2f640 1 --2- 172.21.15.83:0/3501576650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa30414f6f0 0x7fa304072060 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009d90 tx=0x7fa2fc01c870 comp rx=0 tx=0).stop 2024-09-19T07:34:08.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa308b2f640 1 -- 172.21.15.83:0/3501576650 shutdown_connections 2024-09-19T07:34:08.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa308b2f640 1 --2- 172.21.15.83:0/3501576650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa30414f6f0 0x7fa304072060 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa308b2f640 1 -- 172.21.15.83:0/3501576650 >> 172.21.15.83:0/3501576650 conn(0x7fa3040c7eb0 msgr2=0x7fa3040c82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:08.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa308b2f640 1 -- 172.21.15.83:0/3501576650 shutdown_connections 2024-09-19T07:34:08.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.601+0000 7fa308b2f640 1 -- 172.21.15.83:0/3501576650 wait complete. 2024-09-19T07:34:08.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa308b2f640 1 Processor -- start 2024-09-19T07:34:08.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa308b2f640 1 -- start start 2024-09-19T07:34:08.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa308b2f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3040c29b0 0x7fa3040c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa308b2f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040c4280 0x7fa3041f0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa308b2f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa30414f6f0 0x7fa3041508f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa308b2f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa30406dd20 con 0x7fa3040c4280 2024-09-19T07:34:08.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa308b2f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa30406dba0 con 0x7fa30414f6f0 2024-09-19T07:34:08.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa308b2f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa30414fad0 con 0x7fa3040c29b0 2024-09-19T07:34:08.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa301d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040c4280 0x7fa3041f0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa301d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040c4280 0x7fa3041f0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49934/0 (socket says 172.21.15.83:49934) 2024-09-19T07:34:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa301d9b640 1 -- 172.21.15.83:0/4044103347 learned_addr learned my addr 172.21.15.83:0/4044103347 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa301d9b640 1 -- 172.21.15.83:0/4044103347 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3040c29b0 msgr2=0x7fa3040c5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa301d9b640 1 --2- 172.21.15.83:0/4044103347 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3040c29b0 0x7fa3040c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa301d9b640 1 -- 172.21.15.83:0/4044103347 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa30414f6f0 msgr2=0x7fa3041508f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa301d9b640 1 --2- 172.21.15.83:0/4044103347 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa30414f6f0 0x7fa3041508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa301d9b640 1 -- 172.21.15.83:0/4044103347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa304150130 con 0x7fa3040c4280 2024-09-19T07:34:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa301d9b640 1 --2- 172.21.15.83:0/4044103347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040c4280 0x7fa3041f0500 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c450 tx=0x7fa2ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa2faffd640 1 -- 172.21.15.83:0/4044103347 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec011790 con 0x7fa3040c4280 2024-09-19T07:34:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa2faffd640 1 -- 172.21.15.83:0/4044103347 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ec017070 con 0x7fa3040c4280 2024-09-19T07:34:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa308b2f640 1 -- 172.21.15.83:0/4044103347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa304143000 con 0x7fa3040c4280 2024-09-19T07:34:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa2faffd640 1 -- 172.21.15.83:0/4044103347 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec011b90 con 0x7fa3040c4280 2024-09-19T07:34:08.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa308b2f640 1 -- 172.21.15.83:0/4044103347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3041435c0 con 0x7fa3040c4280 2024-09-19T07:34:08.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa308b2f640 1 -- 172.21.15.83:0/4044103347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa304072060 con 0x7fa3040c4280 2024-09-19T07:34:08.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.605+0000 7fa2faffd640 1 -- 172.21.15.83:0/4044103347 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa2ec011d30 con 0x7fa3040c4280 2024-09-19T07:34:08.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.609+0000 7fa2faffd640 1 --2- 172.21.15.83:0/4044103347 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa2e40678c0 0x7fa2e4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:08.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.609+0000 7fa30259c640 1 --2- 172.21.15.83:0/4044103347 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa2e40678c0 0x7fa2e4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:08.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.609+0000 7fa2faffd640 1 -- 172.21.15.83:0/4044103347 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8443+0+0 (secure 0 0 0) 0x7fa2ec086540 con 0x7fa3040c4280 2024-09-19T07:34:08.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.609+0000 7fa30259c640 1 --2- 172.21.15.83:0/4044103347 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa2e40678c0 0x7fa2e4069d80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc0af000 tx=0x7fa2fc004ac0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:08.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.613+0000 7fa2faffd640 1 -- 172.21.15.83:0/4044103347 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa2ec056610 con 0x7fa3040c4280 2024-09-19T07:34:08.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.793+0000 7fa308b2f640 1 -- 172.21.15.83:0/4044103347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3040c3de0 con 0x7fa3040c4280 2024-09-19T07:34:08.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.793+0000 7fa2faffd640 1 -- 172.21.15.83:0/4044103347 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5711 (secure 0 0 0) 0x7fa2ec055f10 con 0x7fa3040c4280 2024-09-19T07:34:08.789 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:08.789 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":111,"btime":"2024-09-19T07:34:08:128296+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:08.128288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:08.789 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 111 2024-09-19T07:34:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.797+0000 7fa308b2f640 1 -- 172.21.15.83:0/4044103347 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa2e40678c0 msgr2=0x7fa2e4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.797+0000 7fa308b2f640 1 --2- 172.21.15.83:0/4044103347 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa2e40678c0 0x7fa2e4069d80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc0af000 tx=0x7fa2fc004ac0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.797+0000 7fa308b2f640 1 -- 172.21.15.83:0/4044103347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040c4280 msgr2=0x7fa3041f0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.797+0000 7fa308b2f640 1 --2- 172.21.15.83:0/4044103347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040c4280 0x7fa3041f0500 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c450 tx=0x7fa2ec00c910 comp rx=0 tx=0).stop 2024-09-19T07:34:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.797+0000 7fa308b2f640 1 -- 172.21.15.83:0/4044103347 shutdown_connections 2024-09-19T07:34:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.797+0000 7fa308b2f640 1 --2- 172.21.15.83:0/4044103347 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa2e40678c0 0x7fa2e4069d80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.797+0000 7fa308b2f640 1 --2- 172.21.15.83:0/4044103347 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa30414f6f0 0x7fa3041508f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.797+0000 7fa308b2f640 1 --2- 172.21.15.83:0/4044103347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040c4280 0x7fa3041f0500 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.797+0000 7fa308b2f640 1 --2- 172.21.15.83:0/4044103347 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3040c29b0 0x7fa3040c5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:08.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.797+0000 7fa308b2f640 1 -- 172.21.15.83:0/4044103347 >> 172.21.15.83:0/4044103347 conn(0x7fa3040c7eb0 msgr2=0x7fa30406f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:08.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.797+0000 7fa308b2f640 1 -- 172.21.15.83:0/4044103347 shutdown_connections 2024-09-19T07:34:08.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:08.797+0000 7fa308b2f640 1 -- 172.21.15.83:0/4044103347 wait complete. 2024-09-19T07:34:08.807 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:34:08:128296+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 111, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 111, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6738": { "addr": "172.21.15.83:6835/1079027718", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 1079027718, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 1079027718, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6738, "incarnation": 111, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:34:08.128288+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6738 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 109, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5830": { "addr": "172.21.15.184:6833/2898687304", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2898687304, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2898687304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5830, "incarnation": 106, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:33:31.301760+0000", "qdb_cluster": [ 5830 ], "qdb_leader": 5830, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5830 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/2112478738", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 2112478738, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 2112478738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 95, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5722, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:34:10.513 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:10.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.649+0000 7f35674a9640 1 Processor -- start 2024-09-19T07:34:10.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.649+0000 7f35674a9640 1 -- start start 2024-09-19T07:34:10.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.649+0000 7f35674a9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f35600c28e0 0x7f35601545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:10.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.649+0000 7f35674a9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35600a9e60 con 0x7f356006dea0 2024-09-19T07:34:10.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.649+0000 7f35674a9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35600aa140 con 0x7f3560071100 2024-09-19T07:34:10.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.649+0000 7f35674a9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3560071b50 con 0x7f35600c28e0 2024-09-19T07:34:10.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.649+0000 7f3564a44640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f35600c28e0 0x7f35601545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:10.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.649+0000 7f3565a46640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f356006dea0 0x7f356006e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46084/0 (socket says 172.21.15.83:46084) 2024-09-19T07:34:10.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.649+0000 7f3565a46640 1 -- 172.21.15.83:0/3635618158 learned_addr learned my addr 172.21.15.83:0/3635618158 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:10.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.649+0000 7f3564a44640 1 -- 172.21.15.83:0/3635618158 >> v1:172.21.15.184:6789/0 conn(0x7f3560071100 legacy=0x7f3560150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:10.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.649+0000 7f3564a44640 1 -- 172.21.15.83:0/3635618158 >> v1:172.21.15.83:6789/0 conn(0x7f356006dea0 legacy=0x7f356006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:10.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.649+0000 7f3564a44640 1 -- 172.21.15.83:0/3635618158 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35600c2240 con 0x7f35600c28e0 2024-09-19T07:34:10.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3564a44640 1 --2- 172.21.15.83:0/3635618158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35600c28e0 0x7f35601545e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f35500097c0 tx=0x7f355001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea42a2e850a3917d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3557fff640 1 -- 172.21.15.83:0/3635618158 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35500b0070 con 0x7f35600c28e0 2024-09-19T07:34:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3557fff640 1 -- 172.21.15.83:0/3635618158 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35500ab600 con 0x7f35600c28e0 2024-09-19T07:34:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3557fff640 1 -- 172.21.15.83:0/3635618158 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35500a4e60 con 0x7f35600c28e0 2024-09-19T07:34:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 -- 172.21.15.83:0/3635618158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35600c28e0 msgr2=0x7f35601545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 --2- 172.21.15.83:0/3635618158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35600c28e0 0x7f35601545e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f35500097c0 tx=0x7f355001c3d0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 -- 172.21.15.83:0/3635618158 shutdown_connections 2024-09-19T07:34:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 --2- 172.21.15.83:0/3635618158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35600c28e0 0x7f35601545e0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 -- 172.21.15.83:0/3635618158 >> 172.21.15.83:0/3635618158 conn(0x7f356006bc80 msgr2=0x7f356006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 -- 172.21.15.83:0/3635618158 shutdown_connections 2024-09-19T07:34:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 -- 172.21.15.83:0/3635618158 wait complete. 2024-09-19T07:34:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 Processor -- start 2024-09-19T07:34:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 -- start start 2024-09-19T07:34:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f356006dea0 0x7f3560151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3560071100 0x7f3560149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:10.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35600c28e0 0x7f35601f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:10.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3560071660 con 0x7f35600c28e0 2024-09-19T07:34:10.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f35600714e0 con 0x7f356006dea0 2024-09-19T07:34:10.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3560161c70 con 0x7f3560071100 2024-09-19T07:34:10.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3565a46640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35600c28e0 0x7f35601f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:10.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3565a46640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35600c28e0 0x7f35601f1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55308/0 (socket says 172.21.15.83:55308) 2024-09-19T07:34:10.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3565a46640 1 -- 172.21.15.83:0/2833869655 learned_addr learned my addr 172.21.15.83:0/2833869655 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:10.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3565245640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f356006dea0 0x7f3560151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:10.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3564a44640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3560071100 0x7f3560149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:10.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3565a46640 1 -- 172.21.15.83:0/2833869655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3560071100 msgr2=0x7f3560149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:10.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3565a46640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3560071100 0x7f3560149400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3565a46640 1 -- 172.21.15.83:0/2833869655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f356006dea0 msgr2=0x7f3560151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:10.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3565a46640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f356006dea0 0x7f3560151cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3565a46640 1 -- 172.21.15.83:0/2833869655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35600abf10 con 0x7f35600c28e0 2024-09-19T07:34:10.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3565245640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f356006dea0 0x7f3560151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:34:10.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3565a46640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35600c28e0 0x7f35601f1e30 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f355800ca80 tx=0x7f355800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:10.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3555ffb640 1 -- 172.21.15.83:0/2833869655 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3558017070 con 0x7f35600c28e0 2024-09-19T07:34:10.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3555ffb640 1 -- 172.21.15.83:0/2833869655 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f355800ac30 con 0x7f35600c28e0 2024-09-19T07:34:10.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 -- 172.21.15.83:0/2833869655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35600ac1f0 con 0x7f35600c28e0 2024-09-19T07:34:10.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f3555ffb640 1 -- 172.21.15.83:0/2833869655 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3558012400 con 0x7f35600c28e0 2024-09-19T07:34:10.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.653+0000 7f35674a9640 1 -- 172.21.15.83:0/2833869655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3560149940 con 0x7f35600c28e0 2024-09-19T07:34:10.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.657+0000 7f35674a9640 1 -- 172.21.15.83:0/2833869655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35601545e0 con 0x7f35600c28e0 2024-09-19T07:34:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.657+0000 7f3555ffb640 1 -- 172.21.15.83:0/2833869655 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f355801a070 con 0x7f35600c28e0 2024-09-19T07:34:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.657+0000 7f3555ffb640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35440678c0 0x7f3544069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.657+0000 7f3565245640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35440678c0 0x7f3544069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.657+0000 7f3555ffb640 1 -- 172.21.15.83:0/2833869655 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8443+0+0 (secure 0 0 0) 0x7f3558086300 con 0x7f35600c28e0 2024-09-19T07:34:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.657+0000 7f3565245640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35440678c0 0x7f3544069d80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f355c00a850 tx=0x7f355c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:10.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.661+0000 7f3555ffb640 1 -- 172.21.15.83:0/2833869655 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3558055d80 con 0x7f35600c28e0 2024-09-19T07:34:10.807 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:10.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 -- 172.21.15.83:0/2833869655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35600c4f30 con 0x7f35600c28e0 2024-09-19T07:34:10.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f3555ffb640 1 -- 172.21.15.83:0/2833869655 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) ==== 95+0+5712 (secure 0 0 0) 0x7f355805a080 con 0x7f35600c28e0 2024-09-19T07:34:10.847 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:10.848 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":113,"btime":"2024-09-19T07:34:10:244580+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:09.249365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:10.848 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 113 2024-09-19T07:34:10.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 -- 172.21.15.83:0/2833869655 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35440678c0 msgr2=0x7f3544069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:10.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35440678c0 0x7f3544069d80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f355c00a850 tx=0x7f355c008040 comp rx=0 tx=0).stop 2024-09-19T07:34:10.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 -- 172.21.15.83:0/2833869655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35600c28e0 msgr2=0x7f35601f1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:10.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35600c28e0 0x7f35601f1e30 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f355800ca80 tx=0x7f355800cf40 comp rx=0 tx=0).stop 2024-09-19T07:34:10.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 -- 172.21.15.83:0/2833869655 shutdown_connections 2024-09-19T07:34:10.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35440678c0 0x7f3544069d80 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35600c28e0 0x7f35601f1e30 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3560071100 0x7f3560149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 --2- 172.21.15.83:0/2833869655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f356006dea0 0x7f3560151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 -- 172.21.15.83:0/2833869655 >> 172.21.15.83:0/2833869655 conn(0x7f356006bc80 msgr2=0x7f35600c3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:10.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 -- 172.21.15.83:0/2833869655 shutdown_connections 2024-09-19T07:34:10.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.853+0000 7f35674a9640 1 -- 172.21.15.83:0/2833869655 wait complete. 2024-09-19T07:34:10.865 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:34:10.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.925+0000 7fb933bd2640 1 Processor -- start 2024-09-19T07:34:10.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.925+0000 7fb933bd2640 1 -- start start 2024-09-19T07:34:10.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.925+0000 7fb933bd2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb92c14ca60 0x7fb92c159490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.925+0000 7fb933bd2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb92c0bef00 0x7fb92c156840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.925+0000 7fb933bd2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb92c1599d0 con 0x7fb92c0cbb00 2024-09-19T07:34:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.925+0000 7fb933bd2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb92c0ca240 con 0x7fb92c14ca60 2024-09-19T07:34:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.925+0000 7fb933bd2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fb92c0c9c80 con 0x7fb92c0bef00 2024-09-19T07:34:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.925+0000 7fb93196e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb92c14ca60 0x7fb92c159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.925+0000 7fb93216f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb92c0cbb00 0x7fb92c0c8210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46086/0 (socket says 172.21.15.83:46086) 2024-09-19T07:34:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb93216f640 1 -- 172.21.15.83:0/2171479960 learned_addr learned my addr 172.21.15.83:0/2171479960 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb93196e640 1 -- 172.21.15.83:0/2171479960 >> v2:172.21.15.184:3300/0 conn(0x7fb92c0bef00 msgr2=0x7fb92c156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb93196e640 1 --2- 172.21.15.83:0/2171479960 >> v2:172.21.15.184:3300/0 conn(0x7fb92c0bef00 0x7fb92c156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb93196e640 1 -- 172.21.15.83:0/2171479960 >> v1:172.21.15.83:6789/0 conn(0x7fb92c0cbb00 legacy=0x7fb92c0c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb93196e640 1 -- 172.21.15.83:0/2171479960 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c0c9f60 con 0x7fb92c14ca60 2024-09-19T07:34:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb93096c640 1 -- 172.21.15.83:0/2171479960 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3867513472 0 0) 0x7fb92c1599d0 con 0x7fb92c0cbb00 2024-09-19T07:34:10.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb93196e640 1 --2- 172.21.15.83:0/2171479960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c14ca60 0x7fb92c159490 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0ca420 tx=0x7fb92801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7b44ff05f0cc8d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb93096c640 1 -- 172.21.15.83:0/2171479960 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9280b2070 con 0x7fb92c14ca60 2024-09-19T07:34:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb93096c640 1 -- 172.21.15.83:0/2171479960 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9280a57f0 con 0x7fb92c14ca60 2024-09-19T07:34:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb93096c640 1 -- 172.21.15.83:0/2171479960 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9280a5bc0 con 0x7fb92c14ca60 2024-09-19T07:34:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb933bd2640 1 -- 172.21.15.83:0/2171479960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c14ca60 msgr2=0x7fb92c159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb933bd2640 1 --2- 172.21.15.83:0/2171479960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c14ca60 0x7fb92c159490 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0ca420 tx=0x7fb92801c930 comp rx=0 tx=0).stop 2024-09-19T07:34:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb933bd2640 1 -- 172.21.15.83:0/2171479960 shutdown_connections 2024-09-19T07:34:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb933bd2640 1 --2- 172.21.15.83:0/2171479960 >> v2:172.21.15.184:3300/0 conn(0x7fb92c0bef00 0x7fb92c156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb933bd2640 1 --2- 172.21.15.83:0/2171479960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c14ca60 0x7fb92c159490 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb933bd2640 1 -- 172.21.15.83:0/2171479960 >> 172.21.15.83:0/2171479960 conn(0x7fb92c072470 msgr2=0x7fb92c072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb933bd2640 1 -- 172.21.15.83:0/2171479960 shutdown_connections 2024-09-19T07:34:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.929+0000 7fb933bd2640 1 -- 172.21.15.83:0/2171479960 wait complete. 2024-09-19T07:34:10.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb933bd2640 1 Processor -- start 2024-09-19T07:34:10.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb933bd2640 1 -- start start 2024-09-19T07:34:10.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb933bd2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c0bef00 0x7fb92c0b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:10.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb933bd2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb92c0cbb00 0x7fb92c0b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:10.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb933bd2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c14ca60 0x7fb92c151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:10.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb933bd2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c06ccb0 con 0x7fb92c0bef00 2024-09-19T07:34:10.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb933bd2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c156d80 con 0x7fb92c14ca60 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb933bd2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb92c153a50 con 0x7fb92c0cbb00 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb93196e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c0bef00 0x7fb92c0b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb93196e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c0bef00 0x7fb92c0b0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55318/0 (socket says 172.21.15.83:55318) 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb93196e640 1 -- 172.21.15.83:0/3555625315 learned_addr learned my addr 172.21.15.83:0/3555625315 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb93196e640 1 -- 172.21.15.83:0/3555625315 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb92c0cbb00 msgr2=0x7fb92c0b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb93196e640 1 --2- 172.21.15.83:0/3555625315 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb92c0cbb00 0x7fb92c0b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb93196e640 1 -- 172.21.15.83:0/3555625315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c14ca60 msgr2=0x7fb92c151330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb93196e640 1 --2- 172.21.15.83:0/3555625315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c14ca60 0x7fb92c151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb93196e640 1 -- 172.21.15.83:0/3555625315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c1efc60 con 0x7fb92c0bef00 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb93196e640 1 --2- 172.21.15.83:0/3555625315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c0bef00 0x7fb92c0b0a20 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fb9280b0040 tx=0x7fb9280196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb9227fc640 1 -- 172.21.15.83:0/3555625315 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9280bd020 con 0x7fb92c0bef00 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb9227fc640 1 -- 172.21.15.83:0/3555625315 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9280ad850 con 0x7fb92c0bef00 2024-09-19T07:34:10.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb9227fc640 1 -- 172.21.15.83:0/3555625315 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9280b2040 con 0x7fb92c0bef00 2024-09-19T07:34:10.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb933bd2640 1 -- 172.21.15.83:0/3555625315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92c150570 con 0x7fb92c0bef00 2024-09-19T07:34:10.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb933bd2640 1 -- 172.21.15.83:0/3555625315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb92c150b30 con 0x7fb92c0bef00 2024-09-19T07:34:10.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb933bd2640 1 -- 172.21.15.83:0/3555625315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb92c06ca90 con 0x7fb92c0bef00 2024-09-19T07:34:10.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb9227fc640 1 -- 172.21.15.83:0/3555625315 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb9280c0020 con 0x7fb92c0bef00 2024-09-19T07:34:10.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb9227fc640 1 --2- 172.21.15.83:0/3555625315 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb9100678c0 0x7fb910069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:10.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb9227fc640 1 -- 172.21.15.83:0/3555625315 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8443+0+0 (secure 0 0 0) 0x7fb928121900 con 0x7fb92c0bef00 2024-09-19T07:34:10.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb93116d640 1 --2- 172.21.15.83:0/3555625315 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb9100678c0 0x7fb910069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:10.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.933+0000 7fb93116d640 1 --2- 172.21.15.83:0/3555625315 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb9100678c0 0x7fb910069d80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb92c06c5c0 tx=0x7fb91c006ca0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:10.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:10.937+0000 7fb9227fc640 1 -- 172.21.15.83:0/3555625315 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb9280c7030 con 0x7fb92c0bef00 2024-09-19T07:34:11.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.129+0000 7fb933bd2640 1 -- 172.21.15.83:0/3555625315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb92c0c6080 con 0x7fb92c0bef00 2024-09-19T07:34:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.129+0000 7fb9227fc640 1 -- 172.21.15.83:0/3555625315 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) ==== 95+0+5712 (secure 0 0 0) 0x7fb928016520 con 0x7fb92c0bef00 2024-09-19T07:34:11.126 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:11.126 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":113,"btime":"2024-09-19T07:34:10:244580+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:09.249365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:11.126 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 113 2024-09-19T07:34:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.133+0000 7fb933bd2640 1 -- 172.21.15.83:0/3555625315 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb9100678c0 msgr2=0x7fb910069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.133+0000 7fb933bd2640 1 --2- 172.21.15.83:0/3555625315 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb9100678c0 0x7fb910069d80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb92c06c5c0 tx=0x7fb91c006ca0 comp rx=0 tx=0).stop 2024-09-19T07:34:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.133+0000 7fb933bd2640 1 -- 172.21.15.83:0/3555625315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c0bef00 msgr2=0x7fb92c0b0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.133+0000 7fb933bd2640 1 --2- 172.21.15.83:0/3555625315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c0bef00 0x7fb92c0b0a20 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fb9280b0040 tx=0x7fb9280196d0 comp rx=0 tx=0).stop 2024-09-19T07:34:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.133+0000 7fb933bd2640 1 -- 172.21.15.83:0/3555625315 shutdown_connections 2024-09-19T07:34:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.133+0000 7fb933bd2640 1 --2- 172.21.15.83:0/3555625315 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb9100678c0 0x7fb910069d80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.133+0000 7fb933bd2640 1 --2- 172.21.15.83:0/3555625315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb92c14ca60 0x7fb92c151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.133+0000 7fb933bd2640 1 --2- 172.21.15.83:0/3555625315 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb92c0cbb00 0x7fb92c0b0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.133+0000 7fb933bd2640 1 --2- 172.21.15.83:0/3555625315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb92c0bef00 0x7fb92c0b0a20 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.133+0000 7fb933bd2640 1 -- 172.21.15.83:0/3555625315 >> 172.21.15.83:0/3555625315 conn(0x7fb92c072470 msgr2=0x7fb92c161040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.133+0000 7fb933bd2640 1 -- 172.21.15.83:0/3555625315 shutdown_connections 2024-09-19T07:34:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:11.133+0000 7fb933bd2640 1 -- 172.21.15.83:0/3555625315 wait complete. 2024-09-19T07:34:11.143 INFO:tasks.mds_thrash.fs.[a]:no change 2024-09-19T07:34:12.866 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:13.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f451310d640 1 Processor -- start 2024-09-19T07:34:13.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f451310d640 1 -- start start 2024-09-19T07:34:13.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f451310d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f450c154540 0x7f450c157490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:13.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f451310d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450c151750 con 0x7f450c1579d0 2024-09-19T07:34:13.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f451310d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450c0c7e90 con 0x7f450c14a500 2024-09-19T07:34:13.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f451310d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f450c073110 con 0x7f450c154540 2024-09-19T07:34:13.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f450bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f450c154540 0x7f450c157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:13.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f45116aa640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f450c1579d0 0x7f450c15e440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46094/0 (socket says 172.21.15.83:46094) 2024-09-19T07:34:13.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f45116aa640 1 -- 172.21.15.83:0/631873031 learned_addr learned my addr 172.21.15.83:0/631873031 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:13.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f450bfff640 1 -- 172.21.15.83:0/631873031 >> v1:172.21.15.184:6789/0 conn(0x7f450c14a500 legacy=0x7f450c0c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:13.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f450bfff640 1 -- 172.21.15.83:0/631873031 >> v1:172.21.15.83:6789/0 conn(0x7f450c1579d0 legacy=0x7f450c15e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:13.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f450bfff640 1 -- 172.21.15.83:0/631873031 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450c0733f0 con 0x7f450c154540 2024-09-19T07:34:13.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.021+0000 7f450bfff640 1 --2- 172.21.15.83:0/631873031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450c154540 0x7f450c157490 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f44f80097c0 tx=0x7f44f801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7552048ae20b7a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:13.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f450b7fe640 1 -- 172.21.15.83:0/631873031 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f80ab5f0 con 0x7f450c154540 2024-09-19T07:34:13.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f450b7fe640 1 -- 172.21.15.83:0/631873031 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f80b0070 con 0x7f450c154540 2024-09-19T07:34:13.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f450b7fe640 1 -- 172.21.15.83:0/631873031 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f80ab9b0 con 0x7f450c154540 2024-09-19T07:34:13.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 -- 172.21.15.83:0/631873031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450c154540 msgr2=0x7f450c157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:13.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 --2- 172.21.15.83:0/631873031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450c154540 0x7f450c157490 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f44f80097c0 tx=0x7f44f801c430 comp rx=0 tx=0).stop 2024-09-19T07:34:13.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 -- 172.21.15.83:0/631873031 shutdown_connections 2024-09-19T07:34:13.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 --2- 172.21.15.83:0/631873031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450c154540 0x7f450c157490 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 -- 172.21.15.83:0/631873031 >> 172.21.15.83:0/631873031 conn(0x7f450c0c9610 msgr2=0x7f450c0c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:13.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 -- 172.21.15.83:0/631873031 shutdown_connections 2024-09-19T07:34:13.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 -- 172.21.15.83:0/631873031 wait complete. 2024-09-19T07:34:13.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 Processor -- start 2024-09-19T07:34:13.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 -- start start 2024-09-19T07:34:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450c14a500 0x7f450c14fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f450c154540 0x7f450c14ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f450c1579d0 0x7f450c139390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f450c0c7850 con 0x7f450c14a500 2024-09-19T07:34:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f450c0c76d0 con 0x7f450c1579d0 2024-09-19T07:34:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f451310d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f450c0c4720 con 0x7f450c154540 2024-09-19T07:34:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f4510ea9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450c14a500 0x7f450c14fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f4510ea9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450c14a500 0x7f450c14fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55328/0 (socket says 172.21.15.83:55328) 2024-09-19T07:34:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f4510ea9640 1 -- 172.21.15.83:0/3525041339 learned_addr learned my addr 172.21.15.83:0/3525041339 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f45116aa640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f450c1579d0 0x7f450c139390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f4510ea9640 1 -- 172.21.15.83:0/3525041339 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f450c154540 msgr2=0x7f450c14ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f4510ea9640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f450c154540 0x7f450c14ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f4510ea9640 1 -- 172.21.15.83:0/3525041339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f450c1579d0 msgr2=0x7f450c139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f4510ea9640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f450c1579d0 0x7f450c139390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f4510ea9640 1 -- 172.21.15.83:0/3525041339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450c14d740 con 0x7f450c14a500 2024-09-19T07:34:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f45116aa640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f450c1579d0 0x7f450c139390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:34:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.025+0000 7f4510ea9640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450c14a500 0x7f450c14fd70 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f450000c450 tx=0x7f450000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.029+0000 7f45097fa640 1 -- 172.21.15.83:0/3525041339 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4500012830 con 0x7f450c14a500 2024-09-19T07:34:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.029+0000 7f45097fa640 1 -- 172.21.15.83:0/3525041339 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4500017070 con 0x7f450c14a500 2024-09-19T07:34:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.029+0000 7f451310d640 1 -- 172.21.15.83:0/3525041339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f450c14eba0 con 0x7f450c14a500 2024-09-19T07:34:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.029+0000 7f45097fa640 1 -- 172.21.15.83:0/3525041339 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4500012c30 con 0x7f450c14a500 2024-09-19T07:34:13.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.029+0000 7f451310d640 1 -- 172.21.15.83:0/3525041339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f450c14f160 con 0x7f450c14a500 2024-09-19T07:34:13.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.029+0000 7f451310d640 1 -- 172.21.15.83:0/3525041339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f450c14f360 con 0x7f450c14a500 2024-09-19T07:34:13.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.029+0000 7f45097fa640 1 -- 172.21.15.83:0/3525041339 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f450001c460 con 0x7f450c14a500 2024-09-19T07:34:13.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.029+0000 7f45097fa640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44f00678c0 0x7f44f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:13.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.029+0000 7f45097fa640 1 -- 172.21.15.83:0/3525041339 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8443+0+0 (secure 0 0 0) 0x7f4500086500 con 0x7f450c14a500 2024-09-19T07:34:13.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.029+0000 7f450bfff640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44f00678c0 0x7f44f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:13.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.029+0000 7f450bfff640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44f00678c0 0x7f44f0069d80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f44f80af000 tx=0x7f44f8004740 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:13.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.033+0000 7f45097fa640 1 -- 172.21.15.83:0/3525041339 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f450c14f360 con 0x7f450c14a500 2024-09-19T07:34:13.144 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:13.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.265+0000 7f451310d640 1 -- 172.21.15.83:0/3525041339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f450c157de0 con 0x7f450c14a500 2024-09-19T07:34:13.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.265+0000 7f45097fa640 1 -- 172.21.15.83:0/3525041339 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+5719 (secure 0 0 0) 0x7f45000565d0 con 0x7f450c14a500 2024-09-19T07:34:13.260 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:13.260 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":114,"btime":"2024-09-19T07:34:11:258178+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:13.260 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 114 2024-09-19T07:34:13.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.265+0000 7f451310d640 1 -- 172.21.15.83:0/3525041339 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44f00678c0 msgr2=0x7f44f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:13.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.265+0000 7f451310d640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44f00678c0 0x7f44f0069d80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f44f80af000 tx=0x7f44f8004740 comp rx=0 tx=0).stop 2024-09-19T07:34:13.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.265+0000 7f451310d640 1 -- 172.21.15.83:0/3525041339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450c14a500 msgr2=0x7f450c14fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:13.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.265+0000 7f451310d640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450c14a500 0x7f450c14fd70 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f450000c450 tx=0x7f450000c910 comp rx=0 tx=0).stop 2024-09-19T07:34:13.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.265+0000 7f451310d640 1 -- 172.21.15.83:0/3525041339 shutdown_connections 2024-09-19T07:34:13.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.265+0000 7f451310d640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44f00678c0 0x7f44f0069d80 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.265+0000 7f451310d640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f450c1579d0 0x7f450c139390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.265+0000 7f451310d640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f450c154540 0x7f450c14ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.265+0000 7f451310d640 1 --2- 172.21.15.83:0/3525041339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450c14a500 0x7f450c14fd70 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.265+0000 7f451310d640 1 -- 172.21.15.83:0/3525041339 >> 172.21.15.83:0/3525041339 conn(0x7f450c0c9610 msgr2=0x7f450c0c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:13.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.269+0000 7f451310d640 1 -- 172.21.15.83:0/3525041339 shutdown_connections 2024-09-19T07:34:13.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.269+0000 7f451310d640 1 -- 172.21.15.83:0/3525041339 wait complete. 2024-09-19T07:34:13.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28f5b68640 1 Processor -- start 2024-09-19T07:34:13.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28f5b68640 1 -- start start 2024-09-19T07:34:13.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28f5b68640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f28f0071100 0x7f28f01545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:13.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28f5b68640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f006dea0 0x7f28f006e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:13.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28f5b68640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28f00a9e60 con 0x7f28f00c28e0 2024-09-19T07:34:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28f5b68640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f28f0071b50 con 0x7f28f0071100 2024-09-19T07:34:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28f5b68640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f28f00c1ff0 con 0x7f28f006dea0 2024-09-19T07:34:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28ef7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f28f0071100 0x7f28f01545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28ef7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f28f0071100 0x7f28f01545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55340/0 (socket says 172.21.15.83:55340) 2024-09-19T07:34:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28ef7fe640 1 -- 172.21.15.83:0/2692289949 learned_addr learned my addr 172.21.15.83:0/2692289949 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28effff640 1 --2- 172.21.15.83:0/2692289949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f006dea0 0x7f28f006e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28ef7fe640 1 -- 172.21.15.83:0/2692289949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f006dea0 msgr2=0x7f28f006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28ef7fe640 1 --2- 172.21.15.83:0/2692289949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f006dea0 0x7f28f006e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28ef7fe640 1 -- 172.21.15.83:0/2692289949 >> v1:172.21.15.184:6789/0 conn(0x7f28f00c28e0 legacy=0x7f28f0150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.277+0000 7f28ef7fe640 1 -- 172.21.15.83:0/2692289949 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f00aa140 con 0x7f28f0071100 2024-09-19T07:34:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.281+0000 7f28ef7fe640 1 --2- 172.21.15.83:0/2692289949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28f0071100 0x7f28f01545e0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f28e0009870 tx=0x7f28e001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c309b73ed9974c1d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.281+0000 7f28ee7fc640 1 -- 172.21.15.83:0/2692289949 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e00b0070 con 0x7f28f0071100 2024-09-19T07:34:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.281+0000 7f28ee7fc640 1 -- 172.21.15.83:0/2692289949 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e00ab600 con 0x7f28f0071100 2024-09-19T07:34:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.281+0000 7f28ee7fc640 1 -- 172.21.15.83:0/2692289949 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e00ab9d0 con 0x7f28f0071100 2024-09-19T07:34:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.281+0000 7f28f5b68640 1 -- 172.21.15.83:0/2692289949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28f0071100 msgr2=0x7f28f01545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.281+0000 7f28f5b68640 1 --2- 172.21.15.83:0/2692289949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28f0071100 0x7f28f01545e0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f28e0009870 tx=0x7f28e001c490 comp rx=0 tx=0).stop 2024-09-19T07:34:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.281+0000 7f28f5b68640 1 -- 172.21.15.83:0/2692289949 shutdown_connections 2024-09-19T07:34:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.281+0000 7f28f5b68640 1 --2- 172.21.15.83:0/2692289949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f006dea0 0x7f28f006e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.281+0000 7f28f5b68640 1 --2- 172.21.15.83:0/2692289949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28f0071100 0x7f28f01545e0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.281+0000 7f28f5b68640 1 -- 172.21.15.83:0/2692289949 >> 172.21.15.83:0/2692289949 conn(0x7f28f006bc80 msgr2=0x7f28f006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.281+0000 7f28f5b68640 1 -- 172.21.15.83:0/2692289949 shutdown_connections 2024-09-19T07:34:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.281+0000 7f28f5b68640 1 -- 172.21.15.83:0/2692289949 wait complete. 2024-09-19T07:34:13.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28f5b68640 1 Processor -- start 2024-09-19T07:34:13.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28f5b68640 1 -- start start 2024-09-19T07:34:13.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28f5b68640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28f006dea0 0x7f28f01538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:13.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28f5b68640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f0071100 0x7f28f0151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:13.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28f5b68640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28f00c28e0 0x7f28f01456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:13.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28f5b68640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f28f0071660 con 0x7f28f00c28e0 2024-09-19T07:34:13.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28f5b68640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f28f00714e0 con 0x7f28f006dea0 2024-09-19T07:34:13.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28f5b68640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f28f0160c60 con 0x7f28f0071100 2024-09-19T07:34:13.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28effff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28f00c28e0 0x7f28f01456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:13.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28effff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28f00c28e0 0x7f28f01456f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55352/0 (socket says 172.21.15.83:55352) 2024-09-19T07:34:13.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28effff640 1 -- 172.21.15.83:0/144648756 learned_addr learned my addr 172.21.15.83:0/144648756 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:13.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28ef7fe640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28f006dea0 0x7f28f01538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:13.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28eeffd640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f0071100 0x7f28f0151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:13.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28effff640 1 -- 172.21.15.83:0/144648756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f0071100 msgr2=0x7f28f0151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:13.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28effff640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f0071100 0x7f28f0151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28effff640 1 -- 172.21.15.83:0/144648756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28f006dea0 msgr2=0x7f28f01538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:13.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28effff640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28f006dea0 0x7f28f01538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28effff640 1 -- 172.21.15.83:0/144648756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f01ed270 con 0x7f28f00c28e0 2024-09-19T07:34:13.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28ef7fe640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28f006dea0 0x7f28f01538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:34:13.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28eeffd640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f0071100 0x7f28f0151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:34:13.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28effff640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28f00c28e0 0x7f28f01456f0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f28e8007e70 tx=0x7f28e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:13.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28d7fff640 1 -- 172.21.15.83:0/144648756 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8014620 con 0x7f28f00c28e0 2024-09-19T07:34:13.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28d7fff640 1 -- 172.21.15.83:0/144648756 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e8019070 con 0x7f28f00c28e0 2024-09-19T07:34:13.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28f5b68640 1 -- 172.21.15.83:0/144648756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f01efc40 con 0x7f28f00c28e0 2024-09-19T07:34:13.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28d7fff640 1 -- 172.21.15.83:0/144648756 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8014a20 con 0x7f28f00c28e0 2024-09-19T07:34:13.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28f5b68640 1 -- 172.21.15.83:0/144648756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f01f0200 con 0x7f28f00c28e0 2024-09-19T07:34:13.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.285+0000 7f28f5b68640 1 -- 172.21.15.83:0/144648756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f006e330 con 0x7f28f00c28e0 2024-09-19T07:34:13.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.289+0000 7f28d7fff640 1 -- 172.21.15.83:0/144648756 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f28e8014bc0 con 0x7f28f00c28e0 2024-09-19T07:34:13.284 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:34:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.289+0000 7f28d7fff640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f28dc067910 0x7f28dc069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.289+0000 7f28d7fff640 1 -- 172.21.15.83:0/144648756 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8443+0+0 (secure 0 0 0) 0x7f28e8088150 con 0x7f28f00c28e0 2024-09-19T07:34:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.289+0000 7f28ef7fe640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f28dc067910 0x7f28dc069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.289+0000 7f28ef7fe640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f28dc067910 0x7f28dc069dd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f28f00c2890 tx=0x7f28e0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:13.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.293+0000 7f28d7fff640 1 -- 172.21.15.83:0/144648756 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f28e80570b0 con 0x7f28f00c28e0 2024-09-19T07:34:13.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.565+0000 7f28f5b68640 1 -- 172.21.15.83:0/144648756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28f00c4f30 con 0x7f28f00c28e0 2024-09-19T07:34:13.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.565+0000 7f28d7fff640 1 -- 172.21.15.83:0/144648756 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+5719 (secure 0 0 0) 0x7f28e80569b0 con 0x7f28f00c28e0 2024-09-19T07:34:13.560 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:13.560 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":114,"btime":"2024-09-19T07:34:11:258178+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:13.560 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 114 2024-09-19T07:34:13.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.565+0000 7f28f5b68640 1 -- 172.21.15.83:0/144648756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f28dc067910 msgr2=0x7f28dc069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:13.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.565+0000 7f28f5b68640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f28dc067910 0x7f28dc069dd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f28f00c2890 tx=0x7f28e0009040 comp rx=0 tx=0).stop 2024-09-19T07:34:13.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.569+0000 7f28f5b68640 1 -- 172.21.15.83:0/144648756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28f00c28e0 msgr2=0x7f28f01456f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:13.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.569+0000 7f28f5b68640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28f00c28e0 0x7f28f01456f0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f28e8007e70 tx=0x7f28e800e4b0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.569+0000 7f28f5b68640 1 -- 172.21.15.83:0/144648756 shutdown_connections 2024-09-19T07:34:13.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.569+0000 7f28f5b68640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f28dc067910 0x7f28dc069dd0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.569+0000 7f28f5b68640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28f00c28e0 0x7f28f01456f0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.569+0000 7f28f5b68640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f0071100 0x7f28f0151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.569+0000 7f28f5b68640 1 --2- 172.21.15.83:0/144648756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28f006dea0 0x7f28f01538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:13.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.569+0000 7f28f5b68640 1 -- 172.21.15.83:0/144648756 >> 172.21.15.83:0/144648756 conn(0x7f28f006bc80 msgr2=0x7f28f006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:13.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.569+0000 7f28f5b68640 1 -- 172.21.15.83:0/144648756 shutdown_connections 2024-09-19T07:34:13.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:13.569+0000 7f28f5b68640 1 -- 172.21.15.83:0/144648756 wait complete. 2024-09-19T07:34:13.586 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6561 2024-09-19T07:34:13.586 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.c 2024-09-19T07:34:14.316 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:34:14.321+0000 7f8a328d0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:34:15.285 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:15.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f2dc6640 1 Processor -- start 2024-09-19T07:34:15.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f2dc6640 1 -- start start 2024-09-19T07:34:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f2dc6640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd5ec068700 0x7fd5ec06e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f2dc6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5ec153390 0x7fd5ec06d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f2dc6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5ec155fa0 con 0x7fd5ec1494a0 2024-09-19T07:34:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f2dc6640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd5ec156e40 con 0x7fd5ec068700 2024-09-19T07:34:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f2dc6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd5ec0c3850 con 0x7fd5ec153390 2024-09-19T07:34:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f0b62640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd5ec1494a0 0x7fd5ec156730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46098/0 (socket says 172.21.15.83:46098) 2024-09-19T07:34:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f0b62640 1 -- 172.21.15.83:0/620211886 learned_addr learned my addr 172.21.15.83:0/620211886 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5ebfff640 1 --2- 172.21.15.83:0/620211886 >> v2:172.21.15.184:3300/0 conn(0x7fd5ec068700 0x7fd5ec06e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f1363640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5ec153390 0x7fd5ec06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 720157799 0 0) 0x7fd5ec155fa0 con 0x7fd5ec1494a0 2024-09-19T07:34:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5dc003430 con 0x7fd5ec1494a0 2024-09-19T07:34:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1294453055 0 0) 0x7fd5dc003430 con 0x7fd5ec1494a0 2024-09-19T07:34:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5ec153390 msgr2=0x7fd5ec06d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5ec153390 0x7fd5ec06d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 >> v2:172.21.15.184:3300/0 conn(0x7fd5ec068700 msgr2=0x7fd5ec06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 --2- 172.21.15.83:0/620211886 >> v2:172.21.15.184:3300/0 conn(0x7fd5ec068700 0x7fd5ec06e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5ec0c3b10 con 0x7fd5ec1494a0 2024-09-19T07:34:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fd5e0001680 con 0x7fd5ec1494a0 2024-09-19T07:34:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 >> v1:172.21.15.83:6789/0 conn(0x7fd5ec1494a0 legacy=0x7fd5ec156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5dc004aa0 0x7fd5dc004ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5ec153390 0x7fd5dc0036b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5dc003bf0 0x7fd5dc004070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec156e40 con 0x7fd5dc004aa0 2024-09-19T07:34:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec0c3850 con 0x7fd5dc003bf0 2024-09-19T07:34:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd5dc00fd30 con 0x7fd5ec153390 2024-09-19T07:34:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f0b62640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5dc004aa0 0x7fd5dc004ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5ebfff640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5ec153390 0x7fd5dc0036b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f1363640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5dc003bf0 0x7fd5dc004070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f0b62640 1 -- 172.21.15.83:0/620211886 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5ec153390 msgr2=0x7fd5dc0036b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:15.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f0b62640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5ec153390 0x7fd5dc0036b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:15.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f0b62640 1 -- 172.21.15.83:0/620211886 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5dc003bf0 msgr2=0x7fd5dc004070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:15.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f0b62640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5dc003bf0 0x7fd5dc004070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:15.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f0b62640 1 -- 172.21.15.83:0/620211886 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5dc003430 con 0x7fd5dc004aa0 2024-09-19T07:34:15.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.421+0000 7fd5f1363640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5dc003bf0 0x7fd5dc004070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:34:15.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.425+0000 7fd5f0b62640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5dc004aa0 0x7fd5dc004ec0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000c2c0 tx=0x7fd5e001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61f68f157cb0460 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:15.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.425+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0007d60 con 0x7fd5dc004aa0 2024-09-19T07:34:15.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.425+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5e00a9020 con 0x7fd5dc004aa0 2024-09-19T07:34:15.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.425+0000 7fd5eb7fe640 1 -- 172.21.15.83:0/620211886 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e00058e0 con 0x7fd5dc004aa0 2024-09-19T07:34:15.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.425+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/620211886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5dc004aa0 msgr2=0x7fd5dc004ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:15.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.425+0000 7fd5f2dc6640 1 --2- 172.21.15.83:0/620211886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5dc004aa0 0x7fd5dc004ec0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000c2c0 tx=0x7fd5e001cc80 comp rx=0 tx=0).stop 2024-09-19T07:34:15.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.425+0000 7fd5f1363640 1 -- 172.21.15.83:0/620211886 reap_dead start 2024-09-19T07:34:15.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.425+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/620211886 shutdown_connections 2024-09-19T07:34:15.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.425+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/620211886 >> 172.21.15.83:0/620211886 conn(0x7fd5ec0c4940 msgr2=0x7fd5ec0c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:15.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.425+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/620211886 shutdown_connections 2024-09-19T07:34:15.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.425+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/620211886 wait complete. 2024-09-19T07:34:15.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f2dc6640 1 Processor -- start 2024-09-19T07:34:15.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f2dc6640 1 -- start start 2024-09-19T07:34:15.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f2dc6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5dc004aa0 0x7fd5ec1edee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:15.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f2dc6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5ec0c78a0 0x7fd5ec144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:15.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f2dc6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5ec0c5a10 0x7fd5ec150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:15.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f2dc6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec0716a0 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f2dc6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec0c40d0 con 0x7fd5ec0c5a10 2024-09-19T07:34:15.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f2dc6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd5ec071820 con 0x7fd5dc004aa0 2024-09-19T07:34:15.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5ebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5ec0c78a0 0x7fd5ec144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:15.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5ebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5ec0c78a0 0x7fd5ec144180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55372/0 (socket says 172.21.15.83:55372) 2024-09-19T07:34:15.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5ebfff640 1 -- 172.21.15.83:0/417930160 learned_addr learned my addr 172.21.15.83:0/417930160 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:15.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f0b62640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5dc004aa0 0x7fd5ec1edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:15.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f1363640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5ec0c5a10 0x7fd5ec150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:15.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5ebfff640 1 -- 172.21.15.83:0/417930160 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5dc004aa0 msgr2=0x7fd5ec1edee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:15.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5ebfff640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5dc004aa0 0x7fd5ec1edee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:15.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5ebfff640 1 -- 172.21.15.83:0/417930160 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5ec0c5a10 msgr2=0x7fd5ec150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:15.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5ebfff640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5ec0c5a10 0x7fd5ec150910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:15.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5ebfff640 1 -- 172.21.15.83:0/417930160 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5ec155280 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5ebfff640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5ec0c78a0 0x7fd5ec144180 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80115e0 tx=0x7fd5d8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:15.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5e97fa640 1 -- 172.21.15.83:0/417930160 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8015870 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5e97fa640 1 -- 172.21.15.83:0/417930160 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d8015a10 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/417930160 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5ec0ab710 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5e97fa640 1 -- 172.21.15.83:0/417930160 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8015dc0 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/417930160 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5ec152400 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/417930160 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5ec070060 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.429+0000 7fd5e97fa640 1 -- 172.21.15.83:0/417930160 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd5d800f460 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.433+0000 7fd5e97fa640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5d00678c0 0x7fd5d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:15.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.433+0000 7fd5e97fa640 1 -- 172.21.15.83:0/417930160 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8443+0+0 (secure 0 0 0) 0x7fd5d8086fd0 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.433+0000 7fd5f0b62640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5d00678c0 0x7fd5d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:15.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.433+0000 7fd5f0b62640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5d00678c0 0x7fd5d0069d80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00aa000 tx=0x7fd5e0001a50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:15.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.437+0000 7fd5e97fa640 1 -- 172.21.15.83:0/417930160 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd5d8019030 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.697+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/417930160 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5ec068470 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.697+0000 7fd5e97fa640 1 -- 172.21.15.83:0/417930160 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+5719 (secure 0 0 0) 0x7fd5d8057010 con 0x7fd5ec0c78a0 2024-09-19T07:34:15.692 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:15.692 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":114,"btime":"2024-09-19T07:34:11:258178+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:15.693 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 114 2024-09-19T07:34:15.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.701+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/417930160 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5d00678c0 msgr2=0x7fd5d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:15.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.701+0000 7fd5f2dc6640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5d00678c0 0x7fd5d0069d80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00aa000 tx=0x7fd5e0001a50 comp rx=0 tx=0).stop 2024-09-19T07:34:15.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.701+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/417930160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5ec0c78a0 msgr2=0x7fd5ec144180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:15.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.701+0000 7fd5f2dc6640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5ec0c78a0 0x7fd5ec144180 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80115e0 tx=0x7fd5d8011aa0 comp rx=0 tx=0).stop 2024-09-19T07:34:15.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.701+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/417930160 shutdown_connections 2024-09-19T07:34:15.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.701+0000 7fd5f2dc6640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5d00678c0 0x7fd5d0069d80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:15.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.701+0000 7fd5f2dc6640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5ec0c5a10 0x7fd5ec150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:15.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.701+0000 7fd5f2dc6640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5ec0c78a0 0x7fd5ec144180 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:15.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.701+0000 7fd5f2dc6640 1 --2- 172.21.15.83:0/417930160 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd5dc004aa0 0x7fd5ec1edee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:15.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.701+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/417930160 >> 172.21.15.83:0/417930160 conn(0x7fd5ec0c4940 msgr2=0x7fd5ec154520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:15.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.701+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/417930160 shutdown_connections 2024-09-19T07:34:15.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:15.701+0000 7fd5f2dc6640 1 -- 172.21.15.83:0/417930160 wait complete. 2024-09-19T07:34:15.714 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:34:17.715 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:17.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.873+0000 7feb25f0b640 1 Processor -- start 2024-09-19T07:34:17.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.873+0000 7feb25f0b640 1 -- start start 2024-09-19T07:34:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb25f0b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb200c6360 0x7feb20153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb25f0b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feb200bee30 0x7feb201570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb25f0b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb200c5b90 con 0x7feb200c9680 2024-09-19T07:34:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb25f0b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7feb2015a050 con 0x7feb200bee30 2024-09-19T07:34:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb25f0b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feb2006c950 con 0x7feb200c6360 2024-09-19T07:34:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb17fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feb200bee30 0x7feb201570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb17fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feb200bee30 0x7feb201570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55384/0 (socket says 172.21.15.83:55384) 2024-09-19T07:34:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb17fff640 1 -- 172.21.15.83:0/2458898543 learned_addr learned my addr 172.21.15.83:0/2458898543 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb1f7fe640 1 --2- 172.21.15.83:0/2458898543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb200c6360 0x7feb20153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb17fff640 1 -- 172.21.15.83:0/2458898543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb200c6360 msgr2=0x7feb20153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb17fff640 1 --2- 172.21.15.83:0/2458898543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb200c6360 0x7feb20153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb17fff640 1 -- 172.21.15.83:0/2458898543 >> v1:172.21.15.184:6789/0 conn(0x7feb200c9680 legacy=0x7feb20157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb17fff640 1 -- 172.21.15.83:0/2458898543 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2006c670 con 0x7feb200bee30 2024-09-19T07:34:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb1f7fe640 1 --2- 172.21.15.83:0/2458898543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb200c6360 0x7feb20153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:34:17.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb17fff640 1 --2- 172.21.15.83:0/2458898543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb200bee30 0x7feb201570f0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7feb08009870 tx=0x7feb0801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7bedf7d298c4a63 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:17.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb1effd640 1 -- 172.21.15.83:0/2458898543 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb080b0070 con 0x7feb200bee30 2024-09-19T07:34:17.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb1effd640 1 -- 172.21.15.83:0/2458898543 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb080ab600 con 0x7feb200bee30 2024-09-19T07:34:17.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb1effd640 1 -- 172.21.15.83:0/2458898543 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb080ab9d0 con 0x7feb200bee30 2024-09-19T07:34:17.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb25f0b640 1 -- 172.21.15.83:0/2458898543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb200bee30 msgr2=0x7feb201570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:17.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb25f0b640 1 --2- 172.21.15.83:0/2458898543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb200bee30 0x7feb201570f0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7feb08009870 tx=0x7feb0801c490 comp rx=0 tx=0).stop 2024-09-19T07:34:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb25f0b640 1 -- 172.21.15.83:0/2458898543 shutdown_connections 2024-09-19T07:34:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb25f0b640 1 --2- 172.21.15.83:0/2458898543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb200bee30 0x7feb201570f0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.877+0000 7feb25f0b640 1 --2- 172.21.15.83:0/2458898543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb200c6360 0x7feb20153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 -- 172.21.15.83:0/2458898543 >> 172.21.15.83:0/2458898543 conn(0x7feb200722a0 msgr2=0x7feb200726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 -- 172.21.15.83:0/2458898543 shutdown_connections 2024-09-19T07:34:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 -- 172.21.15.83:0/2458898543 wait complete. 2024-09-19T07:34:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 Processor -- start 2024-09-19T07:34:17.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 -- start start 2024-09-19T07:34:17.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb200bee30 0x7feb20151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:17.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb200c6360 0x7feb2014c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:17.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb200c9680 0x7feb2014ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:17.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feb200c68c0 con 0x7feb200c6360 2024-09-19T07:34:17.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feb200c6740 con 0x7feb200bee30 2024-09-19T07:34:17.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feb20162da0 con 0x7feb200c9680 2024-09-19T07:34:17.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb200c6360 0x7feb2014c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:17.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb200c6360 0x7feb2014c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55386/0 (socket says 172.21.15.83:55386) 2024-09-19T07:34:17.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb17fff640 1 -- 172.21.15.83:0/681186374 learned_addr learned my addr 172.21.15.83:0/681186374 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:17.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb1f7fe640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb200bee30 0x7feb20151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb1ffff640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb200c9680 0x7feb2014ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb17fff640 1 -- 172.21.15.83:0/681186374 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb200c9680 msgr2=0x7feb2014ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb17fff640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb200c9680 0x7feb2014ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb17fff640 1 -- 172.21.15.83:0/681186374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb200bee30 msgr2=0x7feb20151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb17fff640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb200bee30 0x7feb20151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb17fff640 1 -- 172.21.15.83:0/681186374 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb201ee5f0 con 0x7feb200c6360 2024-09-19T07:34:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb1f7fe640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb200bee30 0x7feb20151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:34:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb17fff640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb200c6360 0x7feb2014c930 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7feb0801ab40 tx=0x7feb08019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb1cff9640 1 -- 172.21.15.83:0/681186374 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb080b9070 con 0x7feb200c6360 2024-09-19T07:34:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb1cff9640 1 -- 172.21.15.83:0/681186374 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb080b4620 con 0x7feb200c6360 2024-09-19T07:34:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 -- 172.21.15.83:0/681186374 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb201f1ba0 con 0x7feb200c6360 2024-09-19T07:34:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb1cff9640 1 -- 172.21.15.83:0/681186374 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb080b0040 con 0x7feb200c6360 2024-09-19T07:34:17.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.881+0000 7feb25f0b640 1 -- 172.21.15.83:0/681186374 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb201f2160 con 0x7feb200c6360 2024-09-19T07:34:17.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.885+0000 7feb25f0b640 1 -- 172.21.15.83:0/681186374 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb20153050 con 0x7feb200c6360 2024-09-19T07:34:17.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.885+0000 7feb1cff9640 1 -- 172.21.15.83:0/681186374 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feb080bc070 con 0x7feb200c6360 2024-09-19T07:34:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.885+0000 7feb1cff9640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb000678c0 0x7feb00069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.885+0000 7feb1cff9640 1 -- 172.21.15.83:0/681186374 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8443+0+0 (secure 0 0 0) 0x7feb0811fa10 con 0x7feb200c6360 2024-09-19T07:34:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.885+0000 7feb1f7fe640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb000678c0 0x7feb00069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.885+0000 7feb1f7fe640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb000678c0 0x7feb00069d80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7feb20073410 tx=0x7feb10008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:17.889+0000 7feb1cff9640 1 -- 172.21.15.83:0/681186374 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7feb08016110 con 0x7feb200c6360 2024-09-19T07:34:18.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.125+0000 7feb25f0b640 1 -- 172.21.15.83:0/681186374 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb200cbc70 con 0x7feb200c6360 2024-09-19T07:34:18.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.125+0000 7feb1cff9640 1 -- 172.21.15.83:0/681186374 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+5719 (secure 0 0 0) 0x7feb080efae0 con 0x7feb200c6360 2024-09-19T07:34:18.120 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:18.120 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":114,"btime":"2024-09-19T07:34:11:258178+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:33:31.301760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"d","rank":0,"incarnation":106,"state":"up:active","state_seq":15,"addr":"172.21.15.184:6833/2898687304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2898687304},{"type":"v1","addr":"172.21.15.184:6833","nonce":2898687304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5830,"qdb_cluster":[5830]},"id":2}]} 2024-09-19T07:34:18.120 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 114 2024-09-19T07:34:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.129+0000 7feb25f0b640 1 -- 172.21.15.83:0/681186374 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb000678c0 msgr2=0x7feb00069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.129+0000 7feb25f0b640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb000678c0 0x7feb00069d80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7feb20073410 tx=0x7feb10008040 comp rx=0 tx=0).stop 2024-09-19T07:34:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.129+0000 7feb25f0b640 1 -- 172.21.15.83:0/681186374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb200c6360 msgr2=0x7feb2014c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.129+0000 7feb25f0b640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb200c6360 0x7feb2014c930 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7feb0801ab40 tx=0x7feb08019230 comp rx=0 tx=0).stop 2024-09-19T07:34:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.129+0000 7feb25f0b640 1 -- 172.21.15.83:0/681186374 shutdown_connections 2024-09-19T07:34:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.129+0000 7feb25f0b640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb000678c0 0x7feb00069d80 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.129+0000 7feb25f0b640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feb200c9680 0x7feb2014ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.129+0000 7feb25f0b640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb200c6360 0x7feb2014c930 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.129+0000 7feb25f0b640 1 --2- 172.21.15.83:0/681186374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feb200bee30 0x7feb20151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.129+0000 7feb25f0b640 1 -- 172.21.15.83:0/681186374 >> 172.21.15.83:0/681186374 conn(0x7feb200722a0 msgr2=0x7feb2015fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.129+0000 7feb25f0b640 1 -- 172.21.15.83:0/681186374 shutdown_connections 2024-09-19T07:34:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:18.129+0000 7feb25f0b640 1 -- 172.21.15.83:0/681186374 wait complete. 2024-09-19T07:34:18.146 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:34:20.147 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:20.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.281+0000 7f65df688640 1 Processor -- start 2024-09-19T07:34:20.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65df688640 1 -- start start 2024-09-19T07:34:20.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65df688640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f65d814a500 0x7f65d80c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65df688640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f65d80c4550 0x7f65d8157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65df688640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65d8151750 con 0x7f65d8154540 2024-09-19T07:34:20.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65df688640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f65d80c7e90 con 0x7f65d80c4550 2024-09-19T07:34:20.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65df688640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f65d8073110 con 0x7f65d814a500 2024-09-19T07:34:20.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65ddc25640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f65d80c4550 0x7f65d8157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65ddc25640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f65d80c4550 0x7f65d8157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46072/0 (socket says 172.21.15.83:46072) 2024-09-19T07:34:20.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65ddc25640 1 -- 172.21.15.83:0/3255199405 learned_addr learned my addr 172.21.15.83:0/3255199405 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:20.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65dd424640 1 --2- 172.21.15.83:0/3255199405 >> v2:172.21.15.184:3300/0 conn(0x7f65d814a500 0x7f65d80c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65ddc25640 1 -- 172.21.15.83:0/3255199405 >> v2:172.21.15.184:3300/0 conn(0x7f65d814a500 msgr2=0x7f65d80c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:20.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65ddc25640 1 --2- 172.21.15.83:0/3255199405 >> v2:172.21.15.184:3300/0 conn(0x7f65d814a500 0x7f65d80c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65ddc25640 1 -- 172.21.15.83:0/3255199405 >> v1:172.21.15.184:6789/0 conn(0x7f65d8154540 legacy=0x7f65d8157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:20.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65ddc25640 1 -- 172.21.15.83:0/3255199405 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d80733f0 con 0x7f65d80c4550 2024-09-19T07:34:20.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65ddc25640 1 --2- 172.21.15.83:0/3255199405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65d80c4550 0x7f65d8157ba0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f65d8072c20 tx=0x7f65d001c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981af1dbc18fe529 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:20.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65cffff640 1 -- 172.21.15.83:0/3255199405 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d00a5620 con 0x7f65d80c4550 2024-09-19T07:34:20.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65cffff640 1 -- 172.21.15.83:0/3255199405 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65d00a57c0 con 0x7f65d80c4550 2024-09-19T07:34:20.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.285+0000 7f65cffff640 1 -- 172.21.15.83:0/3255199405 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d00a5b90 con 0x7f65d80c4550 2024-09-19T07:34:20.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 -- 172.21.15.83:0/3255199405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65d80c4550 msgr2=0x7f65d8157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 --2- 172.21.15.83:0/3255199405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65d80c4550 0x7f65d8157ba0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f65d8072c20 tx=0x7f65d001c7a0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 -- 172.21.15.83:0/3255199405 shutdown_connections 2024-09-19T07:34:20.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 --2- 172.21.15.83:0/3255199405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65d80c4550 0x7f65d8157ba0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 --2- 172.21.15.83:0/3255199405 >> v2:172.21.15.184:3300/0 conn(0x7f65d814a500 0x7f65d80c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 -- 172.21.15.83:0/3255199405 >> 172.21.15.83:0/3255199405 conn(0x7f65d80c9610 msgr2=0x7f65d80c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:20.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 -- 172.21.15.83:0/3255199405 shutdown_connections 2024-09-19T07:34:20.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 -- 172.21.15.83:0/3255199405 wait complete. 2024-09-19T07:34:20.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 Processor -- start 2024-09-19T07:34:20.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 -- start start 2024-09-19T07:34:20.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65d80c4550 0x7f65d80c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65d814a500 0x7f65d8146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65d8154540 0x7f65d8147460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f65d80c7850 con 0x7f65d8154540 2024-09-19T07:34:20.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f65d80c76d0 con 0x7f65d80c4550 2024-09-19T07:34:20.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f65d8160c60 con 0x7f65d814a500 2024-09-19T07:34:20.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65ddc25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65d8154540 0x7f65d8147460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65ddc25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65d8154540 0x7f65d8147460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46082/0 (socket says 172.21.15.83:46082) 2024-09-19T07:34:20.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65ddc25640 1 -- 172.21.15.83:0/2580597098 learned_addr learned my addr 172.21.15.83:0/2580597098 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:20.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65dd424640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65d80c4550 0x7f65d80c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65ddc25640 1 -- 172.21.15.83:0/2580597098 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65d814a500 msgr2=0x7f65d8146f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65dcc23640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65d814a500 0x7f65d8146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65ddc25640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65d814a500 0x7f65d8146f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65ddc25640 1 -- 172.21.15.83:0/2580597098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65d80c4550 msgr2=0x7f65d80c2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65ddc25640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65d80c4550 0x7f65d80c2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65ddc25640 1 -- 172.21.15.83:0/2580597098 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d814ea60 con 0x7f65d8154540 2024-09-19T07:34:20.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65dd424640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65d80c4550 0x7f65d80c2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:34:20.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65ddc25640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65d8154540 0x7f65d8147460 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f65d00af040 tx=0x7f65d0019540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:20.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65cdffb640 1 -- 172.21.15.83:0/2580597098 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d00bc020 con 0x7f65d8154540 2024-09-19T07:34:20.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65cdffb640 1 -- 172.21.15.83:0/2580597098 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65d00b8670 con 0x7f65d8154540 2024-09-19T07:34:20.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 -- 172.21.15.83:0/2580597098 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65d814ab50 con 0x7f65d8154540 2024-09-19T07:34:20.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65cdffb640 1 -- 172.21.15.83:0/2580597098 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0009510 con 0x7f65d8154540 2024-09-19T07:34:20.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.289+0000 7f65df688640 1 -- 172.21.15.83:0/2580597098 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65d814ed00 con 0x7f65d8154540 2024-09-19T07:34:20.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.293+0000 7f65df688640 1 -- 172.21.15.83:0/2580597098 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65d80c4010 con 0x7f65d8154540 2024-09-19T07:34:20.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.293+0000 7f65cdffb640 1 -- 172.21.15.83:0/2580597098 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f65d00bf020 con 0x7f65d8154540 2024-09-19T07:34:20.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.293+0000 7f65cdffb640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65bc0678c0 0x7f65bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.293+0000 7f65dd424640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65bc0678c0 0x7f65bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.293+0000 7f65cdffb640 1 -- 172.21.15.83:0/2580597098 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7f65d0121630 con 0x7f65d8154540 2024-09-19T07:34:20.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.293+0000 7f65dd424640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65bc0678c0 0x7f65bc069d80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f65d40097a0 tx=0x7f65d4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:20.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.297+0000 7f65cdffb640 1 -- 172.21.15.83:0/2580597098 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f65d00c6030 con 0x7f65d8154540 2024-09-19T07:34:20.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.553+0000 7f65df688640 1 -- 172.21.15.83:0/2580597098 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65d8073fd0 con 0x7f65d8154540 2024-09-19T07:34:20.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.553+0000 7f65cdffb640 1 -- 172.21.15.83:0/2580597098 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+4885 (secure 0 0 0) 0x7f65d8151750 con 0x7f65d8154540 2024-09-19T07:34:20.549 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:20.549 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":117,"btime":"2024-09-19T07:34:20:343453+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:19.350541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:rejoin","state_seq":53,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:34:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 117 2024-09-19T07:34:20.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.557+0000 7f65df688640 1 -- 172.21.15.83:0/2580597098 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65bc0678c0 msgr2=0x7f65bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.557+0000 7f65df688640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65bc0678c0 0x7f65bc069d80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f65d40097a0 tx=0x7f65d4008040 comp rx=0 tx=0).stop 2024-09-19T07:34:20.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.557+0000 7f65df688640 1 -- 172.21.15.83:0/2580597098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65d8154540 msgr2=0x7f65d8147460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.557+0000 7f65df688640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65d8154540 0x7f65d8147460 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f65d00af040 tx=0x7f65d0019540 comp rx=0 tx=0).stop 2024-09-19T07:34:20.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.557+0000 7f65df688640 1 -- 172.21.15.83:0/2580597098 shutdown_connections 2024-09-19T07:34:20.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.557+0000 7f65df688640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f65bc0678c0 0x7f65bc069d80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.557+0000 7f65df688640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65d8154540 0x7f65d8147460 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.557+0000 7f65df688640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65d814a500 0x7f65d8146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.557+0000 7f65df688640 1 --2- 172.21.15.83:0/2580597098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65d80c4550 0x7f65d80c2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.557+0000 7f65df688640 1 -- 172.21.15.83:0/2580597098 >> 172.21.15.83:0/2580597098 conn(0x7f65d80c9610 msgr2=0x7f65d8151b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:20.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.557+0000 7f65df688640 1 -- 172.21.15.83:0/2580597098 shutdown_connections 2024-09-19T07:34:20.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.557+0000 7f65df688640 1 -- 172.21.15.83:0/2580597098 wait complete. 2024-09-19T07:34:20.567 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-19T07:34:20.567 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:34:20.567 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:20.587 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-19T07:34:20.587 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:34:20.588 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:34:20.615 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:34:20.616 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:20.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f87efa640 1 Processor -- start 2024-09-19T07:34:20.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f87efa640 1 -- start start 2024-09-19T07:34:20.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f87efa640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6f80151310 0x7f6f800ac5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f87efa640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6f800685f0 0x7f6f8006e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f87efa640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f80154a60 con 0x7f6f800ad8a0 2024-09-19T07:34:20.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f87efa640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6f80154530 con 0x7f6f800685f0 2024-09-19T07:34:20.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f87efa640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6f8006f030 con 0x7f6f80151310 2024-09-19T07:34:20.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f86497640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6f800685f0 0x7f6f8006e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f86497640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6f800685f0 0x7f6f8006e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46100/0 (socket says 172.21.15.83:46100) 2024-09-19T07:34:20.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f86497640 1 -- 172.21.15.83:0/1803076415 learned_addr learned my addr 172.21.15.83:0/1803076415 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:20.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f86497640 1 -- 172.21.15.83:0/1803076415 >> v2:172.21.15.184:3300/0 conn(0x7f6f80151310 msgr2=0x7f6f800ac5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:20.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f85c96640 1 --2- 172.21.15.83:0/1803076415 >> v2:172.21.15.184:3300/0 conn(0x7f6f80151310 0x7f6f800ac5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f86497640 1 --2- 172.21.15.83:0/1803076415 >> v2:172.21.15.184:3300/0 conn(0x7f6f80151310 0x7f6f800ac5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f86497640 1 -- 172.21.15.83:0/1803076415 >> v1:172.21.15.184:6789/0 conn(0x7f6f800ad8a0 legacy=0x7f6f8006f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:20.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.685+0000 7f6f86497640 1 -- 172.21.15.83:0/1803076415 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f801516f0 con 0x7f6f800685f0 2024-09-19T07:34:20.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f86497640 1 --2- 172.21.15.83:0/1803076415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f800685f0 0x7f6f8006e360 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009e50 tx=0x7f6f7801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b5894febbe5b84 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:20.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f84c94640 1 -- 172.21.15.83:0/1803076415 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f780a56f0 con 0x7f6f800685f0 2024-09-19T07:34:20.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f84c94640 1 -- 172.21.15.83:0/1803076415 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f780a5890 con 0x7f6f800685f0 2024-09-19T07:34:20.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f84c94640 1 -- 172.21.15.83:0/1803076415 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f780a5c60 con 0x7f6f800685f0 2024-09-19T07:34:20.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f87efa640 1 -- 172.21.15.83:0/1803076415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f800685f0 msgr2=0x7f6f8006e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f87efa640 1 --2- 172.21.15.83:0/1803076415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f800685f0 0x7f6f8006e360 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009e50 tx=0x7f6f7801c920 comp rx=0 tx=0).stop 2024-09-19T07:34:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f87efa640 1 -- 172.21.15.83:0/1803076415 shutdown_connections 2024-09-19T07:34:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f87efa640 1 --2- 172.21.15.83:0/1803076415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f800685f0 0x7f6f8006e360 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f87efa640 1 --2- 172.21.15.83:0/1803076415 >> v2:172.21.15.184:3300/0 conn(0x7f6f80151310 0x7f6f800ac5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f87efa640 1 -- 172.21.15.83:0/1803076415 >> 172.21.15.83:0/1803076415 conn(0x7f6f800c4670 msgr2=0x7f6f800c4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f87efa640 1 -- 172.21.15.83:0/1803076415 shutdown_connections 2024-09-19T07:34:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f87efa640 1 -- 172.21.15.83:0/1803076415 wait complete. 2024-09-19T07:34:20.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f87efa640 1 Processor -- start 2024-09-19T07:34:20.686 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:34:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.689+0000 7f6f87efa640 1 -- start start 2024-09-19T07:34:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f87efa640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f800685f0 0x7f6f8014afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f87efa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f80151310 0x7f6f8014f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f87efa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8014b4f0 0x7f6f80142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f87efa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f80160c30 con 0x7f6f8014b4f0 2024-09-19T07:34:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f87efa640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6f800ae000 con 0x7f6f80151310 2024-09-19T07:34:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f87efa640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6f80160db0 con 0x7f6f800685f0 2024-09-19T07:34:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f86497640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8014b4f0 0x7f6f80142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f86497640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8014b4f0 0x7f6f80142fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46114/0 (socket says 172.21.15.83:46114) 2024-09-19T07:34:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f86497640 1 -- 172.21.15.83:0/4219179406 learned_addr learned my addr 172.21.15.83:0/4219179406 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f85495640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f80151310 0x7f6f8014f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f85c96640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f800685f0 0x7f6f8014afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f86497640 1 -- 172.21.15.83:0/4219179406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f800685f0 msgr2=0x7f6f8014afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f86497640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f800685f0 0x7f6f8014afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f86497640 1 -- 172.21.15.83:0/4219179406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f80151310 msgr2=0x7f6f8014f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f86497640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f80151310 0x7f6f8014f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f86497640 1 -- 172.21.15.83:0/4219179406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f80149900 con 0x7f6f8014b4f0 2024-09-19T07:34:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f85c96640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f800685f0 0x7f6f8014afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:34:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f86497640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8014b4f0 0x7f6f80142fd0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f6f780af040 tx=0x7f6f780196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f767fc640 1 -- 172.21.15.83:0/4219179406 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f780bc070 con 0x7f6f8014b4f0 2024-09-19T07:34:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f767fc640 1 -- 172.21.15.83:0/4219179406 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f780b6840 con 0x7f6f8014b4f0 2024-09-19T07:34:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f87efa640 1 -- 172.21.15.83:0/4219179406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f801efe00 con 0x7f6f8014b4f0 2024-09-19T07:34:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f767fc640 1 -- 172.21.15.83:0/4219179406 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f780b6c30 con 0x7f6f8014b4f0 2024-09-19T07:34:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f87efa640 1 -- 172.21.15.83:0/4219179406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f801f03c0 con 0x7f6f8014b4f0 2024-09-19T07:34:20.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f767fc640 1 -- 172.21.15.83:0/4219179406 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6f780039a0 con 0x7f6f8014b4f0 2024-09-19T07:34:20.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f87efa640 1 -- 172.21.15.83:0/4219179406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f800008e0 con 0x7f6f8014b4f0 2024-09-19T07:34:20.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f767fc640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f640678f0 0x7f6f64069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.693+0000 7f6f767fc640 1 -- 172.21.15.83:0/4219179406 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7f6f78124d80 con 0x7f6f8014b4f0 2024-09-19T07:34:20.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.697+0000 7f6f85c96640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f640678f0 0x7f6f64069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.697+0000 7f6f85c96640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f640678f0 0x7f6f64069db0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c006fd0 tx=0x7f6f7c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:20.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.701+0000 7f6f767fc640 1 -- 172.21.15.83:0/4219179406 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6f780165a0 con 0x7f6f8014b4f0 2024-09-19T07:34:20.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.741+0000 7f80e3538640 1 Processor -- start 2024-09-19T07:34:20.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.741+0000 7f80e3538640 1 -- start start 2024-09-19T07:34:20.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e3538640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80dc0c2c30 0x7f80dc152e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e3538640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80dc159c60 con 0x7f80dc0aaa60 2024-09-19T07:34:20.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e3538640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80dc0c4b90 con 0x7f80dc06f070 2024-09-19T07:34:20.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e3538640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f80dc15a160 con 0x7f80dc0c2c30 2024-09-19T07:34:20.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e0ad3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f80dc0aaa60 0x7f80dc1533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53346/0 (socket says 172.21.15.83:53346) 2024-09-19T07:34:20.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e0ad3640 1 -- 172.21.15.83:0/4289450218 learned_addr learned my addr 172.21.15.83:0/4289450218 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:20.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e12d4640 1 --2- 172.21.15.83:0/4289450218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80dc0c2c30 0x7f80dc152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3197872618 0 0) 0x7f80dc159c60 con 0x7f80dc0aaa60 2024-09-19T07:34:20.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80c8003530 con 0x7f80dc0aaa60 2024-09-19T07:34:20.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 476478078 0 0) 0x7f80c8003530 con 0x7f80dc0aaa60 2024-09-19T07:34:20.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80dc0c2c30 msgr2=0x7f80dc152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:20.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 --2- 172.21.15.83:0/4289450218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80dc0c2c30 0x7f80dc152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 >> v1:172.21.15.184:6789/0 conn(0x7f80dc06f070 legacy=0x7f80dc155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80dc0aa4a0 con 0x7f80dc0aaa60 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f80cc001680 con 0x7f80dc0aaa60 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 >> v1:172.21.15.83:6789/0 conn(0x7f80dc0aaa60 legacy=0x7f80dc1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 --2- 172.21.15.83:0/4289450218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80dc0c2c30 0x7f80c80039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 --2- 172.21.15.83:0/4289450218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c8003f70 0x7f80c8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 --2- 172.21.15.83:0/4289450218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80c8004d50 0x7f80c80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f80dc15a160 con 0x7f80c8003f70 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f80c800bb60 con 0x7f80c8004d50 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f80c800bd60 con 0x7f80dc0c2c30 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e0ad3640 1 --2- 172.21.15.83:0/4289450218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c8003f70 0x7f80c8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e1ad5640 1 --2- 172.21.15.83:0/4289450218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80c8004d50 0x7f80c80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e12d4640 1 --2- 172.21.15.83:0/4289450218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80dc0c2c30 0x7f80c80039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e0ad3640 1 -- 172.21.15.83:0/4289450218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80dc0c2c30 msgr2=0x7f80c80039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:20.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e0ad3640 1 --2- 172.21.15.83:0/4289450218 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80dc0c2c30 0x7f80c80039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e0ad3640 1 -- 172.21.15.83:0/4289450218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80c8004d50 msgr2=0x7f80c80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:20.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e0ad3640 1 --2- 172.21.15.83:0/4289450218 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80c8004d50 0x7f80c80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.745+0000 7f80e0ad3640 1 -- 172.21.15.83:0/4289450218 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c8003530 con 0x7f80c8003f70 2024-09-19T07:34:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e0ad3640 1 --2- 172.21.15.83:0/4289450218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c8003f70 0x7f80c8004810 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f80cc00c190 tx=0x7f80cc01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992179ee6514b30f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc007d00 con 0x7f80c8003f70 2024-09-19T07:34:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80cc0a9020 con 0x7f80c8003f70 2024-09-19T07:34:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80d3fff640 1 -- 172.21.15.83:0/4289450218 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc005b20 con 0x7f80c8003f70 2024-09-19T07:34:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 -- 172.21.15.83:0/4289450218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c8003f70 msgr2=0x7f80c8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 --2- 172.21.15.83:0/4289450218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c8003f70 0x7f80c8004810 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f80cc00c190 tx=0x7f80cc01cc40 comp rx=0 tx=0).stop 2024-09-19T07:34:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e1ad5640 1 -- 172.21.15.83:0/4289450218 reap_dead start 2024-09-19T07:34:20.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 -- 172.21.15.83:0/4289450218 shutdown_connections 2024-09-19T07:34:20.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 -- 172.21.15.83:0/4289450218 >> 172.21.15.83:0/4289450218 conn(0x7f80dc0c95a0 msgr2=0x7f80dc0c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:20.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 -- 172.21.15.83:0/4289450218 shutdown_connections 2024-09-19T07:34:20.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 -- 172.21.15.83:0/4289450218 wait complete. 2024-09-19T07:34:20.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 Processor -- start 2024-09-19T07:34:20.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 -- start start 2024-09-19T07:34:20.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80c8003f70 0x7f80dc1ef790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80dc1f2ad0 0x7f80dc06e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80dc15c560 0x7f80dc151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f80dc0c3010 con 0x7f80dc15c560 2024-09-19T07:34:20.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f80dc0c3ca0 con 0x7f80dc1f2ad0 2024-09-19T07:34:20.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.749+0000 7f80e3538640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f80dc0c3190 con 0x7f80c8003f70 2024-09-19T07:34:20.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e1ad5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80dc15c560 0x7f80dc151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e1ad5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80dc15c560 0x7f80dc151680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46154/0 (socket says 172.21.15.83:46154) 2024-09-19T07:34:20.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e1ad5640 1 -- 172.21.15.83:0/3746563429 learned_addr learned my addr 172.21.15.83:0/3746563429 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:20.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e12d4640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80c8003f70 0x7f80dc1ef790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e0ad3640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80dc1f2ad0 0x7f80dc06e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e1ad5640 1 -- 172.21.15.83:0/3746563429 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80c8003f70 msgr2=0x7f80dc1ef790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e1ad5640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80c8003f70 0x7f80dc1ef790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e1ad5640 1 -- 172.21.15.83:0/3746563429 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80dc1f2ad0 msgr2=0x7f80dc06e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e1ad5640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80dc1f2ad0 0x7f80dc06e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e1ad5640 1 -- 172.21.15.83:0/3746563429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80dc1524a0 con 0x7f80dc15c560 2024-09-19T07:34:20.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e1ad5640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80dc15c560 0x7f80dc151680 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f80dc153ae0 tx=0x7f80d400fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:20.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80d1ffb640 1 -- 172.21.15.83:0/3746563429 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d4003b90 con 0x7f80dc15c560 2024-09-19T07:34:20.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80d1ffb640 1 -- 172.21.15.83:0/3746563429 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80d4018070 con 0x7f80dc15c560 2024-09-19T07:34:20.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e3538640 1 -- 172.21.15.83:0/3746563429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80dc1eecb0 con 0x7f80dc15c560 2024-09-19T07:34:20.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80d1ffb640 1 -- 172.21.15.83:0/3746563429 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d40050d0 con 0x7f80dc15c560 2024-09-19T07:34:20.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e3538640 1 -- 172.21.15.83:0/3746563429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80dc1ef270 con 0x7f80dc15c560 2024-09-19T07:34:20.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e3538640 1 -- 172.21.15.83:0/3746563429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80dc1ef470 con 0x7f80dc15c560 2024-09-19T07:34:20.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80d1ffb640 1 -- 172.21.15.83:0/3746563429 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f80d4005270 con 0x7f80dc15c560 2024-09-19T07:34:20.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80d1ffb640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80c00678f0 0x7f80c0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:20.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80d1ffb640 1 -- 172.21.15.83:0/3746563429 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7f80d40866d0 con 0x7f80dc15c560 2024-09-19T07:34:20.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e12d4640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80c00678f0 0x7f80c0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:20.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.753+0000 7f80e12d4640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80c00678f0 0x7f80c0069db0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009a30 tx=0x7f80d8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:20.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.761+0000 7f80d1ffb640 1 -- 172.21.15.83:0/3746563429 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f80dc1ef470 con 0x7f80dc15c560 2024-09-19T07:34:20.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.965+0000 7f6f87efa640 1 -- 172.21.15.83:0/4219179406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f800ad8a0 con 0x7f6f8014b4f0 2024-09-19T07:34:20.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.965+0000 7f6f767fc640 1 -- 172.21.15.83:0/4219179406 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+4885 (secure 0 0 0) 0x7f6f780f4eb0 con 0x7f6f8014b4f0 2024-09-19T07:34:20.960 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:20.961 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":117,"btime":"2024-09-19T07:34:20:343453+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:19.350541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:rejoin","state_seq":53,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:34:20.961 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 117 2024-09-19T07:34:20.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.969+0000 7f6f87efa640 1 -- 172.21.15.83:0/4219179406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f640678f0 msgr2=0x7f6f64069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.969+0000 7f6f87efa640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f640678f0 0x7f6f64069db0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c006fd0 tx=0x7f6f7c008040 comp rx=0 tx=0).stop 2024-09-19T07:34:20.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.969+0000 7f6f87efa640 1 -- 172.21.15.83:0/4219179406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8014b4f0 msgr2=0x7f6f80142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:20.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.969+0000 7f6f87efa640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8014b4f0 0x7f6f80142fd0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f6f780af040 tx=0x7f6f780196c0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.969+0000 7f6f87efa640 1 -- 172.21.15.83:0/4219179406 shutdown_connections 2024-09-19T07:34:20.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.969+0000 7f6f87efa640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f640678f0 0x7f6f64069db0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.969+0000 7f6f87efa640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8014b4f0 0x7f6f80142fd0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.969+0000 7f6f87efa640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f80151310 0x7f6f8014f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.969+0000 7f6f87efa640 1 --2- 172.21.15.83:0/4219179406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f800685f0 0x7f6f8014afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:20.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.969+0000 7f6f87efa640 1 -- 172.21.15.83:0/4219179406 >> 172.21.15.83:0/4219179406 conn(0x7f6f800c4670 msgr2=0x7f6f800aa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:20.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.969+0000 7f6f87efa640 1 -- 172.21.15.83:0/4219179406 shutdown_connections 2024-09-19T07:34:20.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:20.969+0000 7f6f87efa640 1 -- 172.21.15.83:0/4219179406 wait complete. 2024-09-19T07:34:20.981 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:34:20:343453+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 117, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 114, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6738": { "addr": "172.21.15.83:6835/1079027718", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 1079027718, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 1079027718, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6738, "incarnation": 111, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:34:11.258171+0000", "qdb_cluster": [ 6738 ], "qdb_leader": 6738, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6738 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 117, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5722": { "addr": "172.21.15.184:6835/2112478738", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 2112478738, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 2112478738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5722, "incarnation": 115, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 53 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:34:19.350541+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5722 } } } ], "standbys": [] } 2024-09-19T07:34:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 -- 172.21.15.83:0/3746563429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80dc155ce0 con 0x7f80dc15c560 2024-09-19T07:34:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80d1ffb640 1 -- 172.21.15.83:0/3746563429 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+4885 (secure 0 0 0) 0x7f80d40566b0 con 0x7f80dc15c560 2024-09-19T07:34:21.027 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:21.028 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":117,"btime":"2024-09-19T07:34:20:343453+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:19.350541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:rejoin","state_seq":53,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:34:21.028 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 117 2024-09-19T07:34:21.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 -- 172.21.15.83:0/3746563429 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80c00678f0 msgr2=0x7f80c0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:21.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80c00678f0 0x7f80c0069db0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009a30 tx=0x7f80d8008040 comp rx=0 tx=0).stop 2024-09-19T07:34:21.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 -- 172.21.15.83:0/3746563429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80dc15c560 msgr2=0x7f80dc151680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:21.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80dc15c560 0x7f80dc151680 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f80dc153ae0 tx=0x7f80d400fd40 comp rx=0 tx=0).stop 2024-09-19T07:34:21.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 -- 172.21.15.83:0/3746563429 shutdown_connections 2024-09-19T07:34:21.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f80c00678f0 0x7f80c0069db0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:21.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80dc15c560 0x7f80dc151680 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:21.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80dc1f2ad0 0x7f80dc06e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:21.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 --2- 172.21.15.83:0/3746563429 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80c8003f70 0x7f80dc1ef790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:21.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 -- 172.21.15.83:0/3746563429 >> 172.21.15.83:0/3746563429 conn(0x7f80dc0c95a0 msgr2=0x7f80dc0c8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:21.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 -- 172.21.15.83:0/3746563429 shutdown_connections 2024-09-19T07:34:21.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:21.033+0000 7f80e3538640 1 -- 172.21.15.83:0/3746563429 wait complete. 2024-09-19T07:34:21.045 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-19T07:34:22.981 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:23.046 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:23.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccbb805640 1 Processor -- start 2024-09-19T07:34:23.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccbb805640 1 -- start start 2024-09-19T07:34:23.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccbb805640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fccb40c6120 0x7fccb40c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccbb805640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fccb4155d70 0x7fccb41575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccbb805640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fccb41592f0 0x7fccb4157b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccbb805640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fccb40c9930 con 0x7fccb4155d70 2024-09-19T07:34:23.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccbb805640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fccb4156200 con 0x7fccb40c6120 2024-09-19T07:34:23.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccbb805640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fccb41564e0 con 0x7fccb41592f0 2024-09-19T07:34:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccb8da0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fccb4155d70 0x7fccb41575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccb8da0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fccb4155d70 0x7fccb41575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46162/0 (socket says 172.21.15.83:46162) 2024-09-19T07:34:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccb8da0640 1 -- 172.21.15.83:0/1726412357 learned_addr learned my addr 172.21.15.83:0/1726412357 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccb95a1640 1 --2- 172.21.15.83:0/1726412357 >> v2:172.21.15.184:3300/0 conn(0x7fccb40c6120 0x7fccb40c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccb8da0640 1 -- 172.21.15.83:0/1726412357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fccb41592f0 msgr2=0x7fccb4157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccb8da0640 1 --2- 172.21.15.83:0/1726412357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fccb41592f0 0x7fccb4157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccb8da0640 1 -- 172.21.15.83:0/1726412357 >> v2:172.21.15.184:3300/0 conn(0x7fccb40c6120 msgr2=0x7fccb40c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccb8da0640 1 --2- 172.21.15.83:0/1726412357 >> v2:172.21.15.184:3300/0 conn(0x7fccb40c6120 0x7fccb40c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.133+0000 7fccb8da0640 1 -- 172.21.15.83:0/1726412357 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb41567c0 con 0x7fccb4155d70 2024-09-19T07:34:23.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccb8da0640 1 --2- 172.21.15.83:0/1726412357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccb4155d70 0x7fccb41575d0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fccb408d540 tx=0x7fcca401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6824f37a03478256 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:23.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccabfff640 1 -- 172.21.15.83:0/1726412357 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca40b0070 con 0x7fccb4155d70 2024-09-19T07:34:23.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccabfff640 1 -- 172.21.15.83:0/1726412357 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca40ab600 con 0x7fccb4155d70 2024-09-19T07:34:23.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccabfff640 1 -- 172.21.15.83:0/1726412357 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca40ab9d0 con 0x7fccb4155d70 2024-09-19T07:34:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccbb805640 1 -- 172.21.15.83:0/1726412357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccb4155d70 msgr2=0x7fccb41575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccbb805640 1 --2- 172.21.15.83:0/1726412357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccb4155d70 0x7fccb41575d0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fccb408d540 tx=0x7fcca401c520 comp rx=0 tx=0).stop 2024-09-19T07:34:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccbb805640 1 -- 172.21.15.83:0/1726412357 shutdown_connections 2024-09-19T07:34:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccbb805640 1 --2- 172.21.15.83:0/1726412357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fccb41592f0 0x7fccb4157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccbb805640 1 --2- 172.21.15.83:0/1726412357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccb4155d70 0x7fccb41575d0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccbb805640 1 --2- 172.21.15.83:0/1726412357 >> v2:172.21.15.184:3300/0 conn(0x7fccb40c6120 0x7fccb40c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccbb805640 1 -- 172.21.15.83:0/1726412357 >> 172.21.15.83:0/1726412357 conn(0x7fccb406bc80 msgr2=0x7fccb406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccbb805640 1 -- 172.21.15.83:0/1726412357 shutdown_connections 2024-09-19T07:34:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccbb805640 1 -- 172.21.15.83:0/1726412357 wait complete. 2024-09-19T07:34:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccbb805640 1 Processor -- start 2024-09-19T07:34:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.137+0000 7fccbb805640 1 -- start start 2024-09-19T07:34:23.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccbb805640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fccb40c6120 0x7fccb40ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccbb805640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccb41592f0 0x7fccb40caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccbb805640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fccb414cf50 0x7fccb414d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccbb805640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fccb40c8b40 con 0x7fccb41592f0 2024-09-19T07:34:23.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccbb805640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fccb40c89c0 con 0x7fccb40c6120 2024-09-19T07:34:23.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccbb805640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fccb40c8cc0 con 0x7fccb414cf50 2024-09-19T07:34:23.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccb8da0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccb41592f0 0x7fccb40caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:23.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccb8da0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccb41592f0 0x7fccb40caa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46176/0 (socket says 172.21.15.83:46176) 2024-09-19T07:34:23.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccb8da0640 1 -- 172.21.15.83:0/1591846057 learned_addr learned my addr 172.21.15.83:0/1591846057 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:23.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccb8da0640 1 -- 172.21.15.83:0/1591846057 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fccb414cf50 msgr2=0x7fccb414d400 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:34:23.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccb8da0640 1 --2- 172.21.15.83:0/1591846057 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fccb414cf50 0x7fccb414d400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccb8da0640 1 -- 172.21.15.83:0/1591846057 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fccb40c6120 msgr2=0x7fccb40ca550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:34:23.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccb8da0640 1 --2- 172.21.15.83:0/1591846057 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fccb40c6120 0x7fccb40ca550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccb8da0640 1 -- 172.21.15.83:0/1591846057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb4160210 con 0x7fccb41592f0 2024-09-19T07:34:23.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccb8da0640 1 --2- 172.21.15.83:0/1591846057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccb41592f0 0x7fccb40caa90 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fcca40ae040 tx=0x7fcca40192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:23.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fcca9ffb640 1 -- 172.21.15.83:0/1591846057 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca40b0040 con 0x7fccb41592f0 2024-09-19T07:34:23.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fcca9ffb640 1 -- 172.21.15.83:0/1591846057 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca40b9070 con 0x7fccb41592f0 2024-09-19T07:34:23.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fcca9ffb640 1 -- 172.21.15.83:0/1591846057 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca40b4680 con 0x7fccb41592f0 2024-09-19T07:34:23.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccbb805640 1 -- 172.21.15.83:0/1591846057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccb415f420 con 0x7fccb41592f0 2024-09-19T07:34:23.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccbb805640 1 -- 172.21.15.83:0/1591846057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccb40ca0a0 con 0x7fccb41592f0 2024-09-19T07:34:23.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fccbb805640 1 -- 172.21.15.83:0/1591846057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccb40ca2e0 con 0x7fccb41592f0 2024-09-19T07:34:23.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fcca9ffb640 1 -- 172.21.15.83:0/1591846057 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcca40b4820 con 0x7fccb41592f0 2024-09-19T07:34:23.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.141+0000 7fcca9ffb640 1 --2- 172.21.15.83:0/1591846057 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc980678f0 0x7fcc98069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.145+0000 7fcca9ffb640 1 -- 172.21.15.83:0/1591846057 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7fcca4120420 con 0x7fccb41592f0 2024-09-19T07:34:23.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.145+0000 7fccb95a1640 1 --2- 172.21.15.83:0/1591846057 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc980678f0 0x7fcc98069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:23.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.149+0000 7fccb95a1640 1 --2- 172.21.15.83:0/1591846057 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc980678f0 0x7fcc98069db0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009870 tx=0x7fccb0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:23.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.149+0000 7fcca9ffb640 1 -- 172.21.15.83:0/1591846057 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fccb40ca2e0 con 0x7fccb41592f0 2024-09-19T07:34:23.164 INFO:tasks.ceph.mds.b.smithi184.stderr:2024-09-19T07:34:23.165+0000 7f0ebd067640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:34:23.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.193+0000 7f777ce0b640 1 Processor -- start 2024-09-19T07:34:23.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.193+0000 7f777ce0b640 1 -- start start 2024-09-19T07:34:23.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777ce0b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7778150290 0x7f77780c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777ce0b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7778152b20 0x7f77780c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7778150290 0x7f77780c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:23.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777ce0b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7778150670 con 0x7f77780c45c0 2024-09-19T07:34:23.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f7776d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f77780c45c0 0x7f7778153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53348/0 (socket says 172.21.15.83:53348) 2024-09-19T07:34:23.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7778150290 0x7f77780c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46182/0 (socket says 172.21.15.83:46182) 2024-09-19T07:34:23.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f7776d9d640 1 -- 172.21.15.83:0/3840907056 learned_addr learned my addr 172.21.15.83:0/3840907056 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:23.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777ce0b640 1 -- 172.21.15.83:0/3840907056 --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f77780716c0 con 0x7f7778150290 2024-09-19T07:34:23.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777ce0b640 1 -- 172.21.15.83:0/3840907056 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f77780c3920 con 0x7f7778152b20 2024-09-19T07:34:23.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f7775d9b640 1 --2- 172.21.15.83:0/3840907056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7778152b20 0x7f77780c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:23.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777559a640 1 -- 172.21.15.83:0/3840907056 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1703875189 0 0) 0x7f7778150670 con 0x7f77780c45c0 2024-09-19T07:34:23.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777559a640 1 -- 172.21.15.83:0/3840907056 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7760003530 con 0x7f77780c45c0 2024-09-19T07:34:23.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777659c640 1 -- 172.21.15.83:0/3840907056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7778152b20 msgr2=0x7f77780c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:23.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777659c640 1 --2- 172.21.15.83:0/3840907056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7778152b20 0x7f77780c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777659c640 1 -- 172.21.15.83:0/3840907056 >> v1:172.21.15.83:6789/0 conn(0x7f77780c45c0 legacy=0x7f7778153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:23.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777659c640 1 -- 172.21.15.83:0/3840907056 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77780c2880 con 0x7f7778150290 2024-09-19T07:34:23.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777659c640 1 --2- 172.21.15.83:0/3840907056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7778150290 0x7f77780c1ff0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f77680097f0 tx=0x7f776801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62854286abc68f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:23.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777559a640 1 -- 172.21.15.83:0/3840907056 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77680b1070 con 0x7f7778150290 2024-09-19T07:34:23.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777559a640 1 -- 172.21.15.83:0/3840907056 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77680ac410 con 0x7f7778150290 2024-09-19T07:34:23.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.197+0000 7f777559a640 1 -- 172.21.15.83:0/3840907056 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77680ac7c0 con 0x7f7778150290 2024-09-19T07:34:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 -- 172.21.15.83:0/3840907056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7778150290 msgr2=0x7f77780c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 --2- 172.21.15.83:0/3840907056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7778150290 0x7f77780c1ff0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f77680097f0 tx=0x7f776801c4a0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 -- 172.21.15.83:0/3840907056 shutdown_connections 2024-09-19T07:34:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 --2- 172.21.15.83:0/3840907056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7778152b20 0x7f77780c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 --2- 172.21.15.83:0/3840907056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7778150290 0x7f77780c1ff0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 -- 172.21.15.83:0/3840907056 >> 172.21.15.83:0/3840907056 conn(0x7f777806bc80 msgr2=0x7f777806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 -- 172.21.15.83:0/3840907056 shutdown_connections 2024-09-19T07:34:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 -- 172.21.15.83:0/3840907056 wait complete. 2024-09-19T07:34:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 Processor -- start 2024-09-19T07:34:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 -- start start 2024-09-19T07:34:23.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77780c45c0 0x7f7778073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7778150290 0x7f7778072990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7778152b20 0x7f777814ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f77780c2530 con 0x7f77780c45c0 2024-09-19T07:34:23.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f77780c3120 con 0x7f7778152b20 2024-09-19T07:34:23.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777ce0b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f777815fb60 con 0x7f7778150290 2024-09-19T07:34:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77780c45c0 0x7f7778073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77780c45c0 0x7f7778073ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46188/0 (socket says 172.21.15.83:46188) 2024-09-19T07:34:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f777659c640 1 -- 172.21.15.83:0/3323521388 learned_addr learned my addr 172.21.15.83:0/3323521388 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f7775d9b640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7778150290 0x7f7778072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.201+0000 7f7776d9d640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7778152b20 0x7f777814ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f777659c640 1 -- 172.21.15.83:0/3323521388 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7778150290 msgr2=0x7f7778072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f777659c640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7778150290 0x7f7778072990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f777659c640 1 -- 172.21.15.83:0/3323521388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7778152b20 msgr2=0x7f777814ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f777659c640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7778152b20 0x7f777814ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f777659c640 1 -- 172.21.15.83:0/3323521388 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7778074000 con 0x7f77780c45c0 2024-09-19T07:34:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f777659c640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77780c45c0 0x7f7778073ac0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f776801a0d0 tx=0x7f7768019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f7766ffd640 1 -- 172.21.15.83:0/3323521388 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77680b9070 con 0x7f77780c45c0 2024-09-19T07:34:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f7766ffd640 1 -- 172.21.15.83:0/3323521388 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77680accc0 con 0x7f77780c45c0 2024-09-19T07:34:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f7766ffd640 1 -- 172.21.15.83:0/3323521388 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77680b1040 con 0x7f77780c45c0 2024-09-19T07:34:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f777ce0b640 1 -- 172.21.15.83:0/3323521388 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7778072ed0 con 0x7f77780c45c0 2024-09-19T07:34:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f777ce0b640 1 -- 172.21.15.83:0/3323521388 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77780bcf20 con 0x7f77780c45c0 2024-09-19T07:34:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f777ce0b640 1 -- 172.21.15.83:0/3323521388 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77780c3e90 con 0x7f77780c45c0 2024-09-19T07:34:23.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f7766ffd640 1 -- 172.21.15.83:0/3323521388 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f77680bc070 con 0x7f77780c45c0 2024-09-19T07:34:23.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f7766ffd640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77580678f0 0x7f7758069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:23.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f7766ffd640 1 -- 172.21.15.83:0/3323521388 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7f7768120590 con 0x7f77780c45c0 2024-09-19T07:34:23.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f7775d9b640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77580678f0 0x7f7758069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:23.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.205+0000 7f7775d9b640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77580678f0 0x7f7758069db0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7778153d80 tx=0x7f776c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:23.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.209+0000 7f7766ffd640 1 -- 172.21.15.83:0/3323521388 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7768016090 con 0x7f77780c45c0 2024-09-19T07:34:23.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.393+0000 7fccbb805640 1 -- 172.21.15.83:0/1591846057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fccb40c6500 con 0x7fccb41592f0 2024-09-19T07:34:23.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.393+0000 7fcca9ffb640 1 -- 172.21.15.83:0/1591846057 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+5717 (secure 0 0 0) 0x7fcca4016110 con 0x7fccb41592f0 2024-09-19T07:34:23.389 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:23.390 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":118,"btime":"2024-09-19T07:34:21:357390+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6918,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:21.357382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:active","state_seq":54,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5722,"qdb_cluster":[5722]},"id":2}]} 2024-09-19T07:34:23.390 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 118 2024-09-19T07:34:23.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.397+0000 7fccbb805640 1 -- 172.21.15.83:0/1591846057 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc980678f0 msgr2=0x7fcc98069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:23.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.397+0000 7fccbb805640 1 --2- 172.21.15.83:0/1591846057 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc980678f0 0x7fcc98069db0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009870 tx=0x7fccb0008040 comp rx=0 tx=0).stop 2024-09-19T07:34:23.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.397+0000 7fccbb805640 1 -- 172.21.15.83:0/1591846057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccb41592f0 msgr2=0x7fccb40caa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:23.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.397+0000 7fccbb805640 1 --2- 172.21.15.83:0/1591846057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccb41592f0 0x7fccb40caa90 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fcca40ae040 tx=0x7fcca40192c0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.397+0000 7fccbb805640 1 -- 172.21.15.83:0/1591846057 shutdown_connections 2024-09-19T07:34:23.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.397+0000 7fccbb805640 1 --2- 172.21.15.83:0/1591846057 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc980678f0 0x7fcc98069db0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.397+0000 7fccbb805640 1 --2- 172.21.15.83:0/1591846057 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fccb414cf50 0x7fccb414d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.397+0000 7fccbb805640 1 --2- 172.21.15.83:0/1591846057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccb41592f0 0x7fccb40caa90 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.397+0000 7fccbb805640 1 --2- 172.21.15.83:0/1591846057 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fccb40c6120 0x7fccb40ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.397+0000 7fccbb805640 1 -- 172.21.15.83:0/1591846057 >> 172.21.15.83:0/1591846057 conn(0x7fccb406bc80 msgr2=0x7fccb406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:23.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.397+0000 7fccbb805640 1 -- 172.21.15.83:0/1591846057 shutdown_connections 2024-09-19T07:34:23.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.397+0000 7fccbb805640 1 -- 172.21.15.83:0/1591846057 wait complete. 2024-09-19T07:34:23.410 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=5830 2024-09-19T07:34:23.410 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.d 2024-09-19T07:34:23.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.461+0000 7f777ce0b640 1 -- 172.21.15.83:0/3323521388 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77780c1ff0 con 0x7f77780c45c0 2024-09-19T07:34:23.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.461+0000 7f7766ffd640 1 -- 172.21.15.83:0/3323521388 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+5717 (secure 0 0 0) 0x7f77680f0030 con 0x7f77780c45c0 2024-09-19T07:34:23.458 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:23.458 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":118,"btime":"2024-09-19T07:34:21:357390+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6918,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:21.357382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:active","state_seq":54,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5722,"qdb_cluster":[5722]},"id":2}]} 2024-09-19T07:34:23.458 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 118 2024-09-19T07:34:23.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.465+0000 7f777ce0b640 1 -- 172.21.15.83:0/3323521388 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77580678f0 msgr2=0x7f7758069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:23.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.465+0000 7f777ce0b640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77580678f0 0x7f7758069db0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7778153d80 tx=0x7f776c009190 comp rx=0 tx=0).stop 2024-09-19T07:34:23.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.465+0000 7f777ce0b640 1 -- 172.21.15.83:0/3323521388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77780c45c0 msgr2=0x7f7778073ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:23.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.465+0000 7f777ce0b640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77780c45c0 0x7f7778073ac0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f776801a0d0 tx=0x7f7768019240 comp rx=0 tx=0).stop 2024-09-19T07:34:23.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.465+0000 7f777ce0b640 1 -- 172.21.15.83:0/3323521388 shutdown_connections 2024-09-19T07:34:23.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.465+0000 7f777ce0b640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77580678f0 0x7f7758069db0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.465+0000 7f777ce0b640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7778152b20 0x7f777814ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.465+0000 7f777ce0b640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7778150290 0x7f7778072990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.465+0000 7f777ce0b640 1 --2- 172.21.15.83:0/3323521388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77780c45c0 0x7f7778073ac0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:23.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.465+0000 7f777ce0b640 1 -- 172.21.15.83:0/3323521388 >> 172.21.15.83:0/3323521388 conn(0x7f777806bc80 msgr2=0x7f777814b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:23.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.465+0000 7f777ce0b640 1 -- 172.21.15.83:0/3323521388 shutdown_connections 2024-09-19T07:34:23.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:23.465+0000 7f777ce0b640 1 -- 172.21.15.83:0/3323521388 wait complete. 2024-09-19T07:34:23.477 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-19T07:34:23.478 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2024-09-19T07:34:30.411 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-19T07:34:30.411 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-19T07:34:30.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-19T07:34:30.414 INFO:tasks.ceph.mds.d:Started 2024-09-19T07:34:30.415 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:30.493 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-19T07:34:30.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.565+0000 7fe7d5ece640 1 Processor -- start 2024-09-19T07:34:30.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.565+0000 7fe7d5ece640 1 -- start start 2024-09-19T07:34:30.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7d5ece640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe7d00c3ab0 0x7fe7d0157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:30.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7d5ece640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7d00aa4a0 0x7fe7d0157550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:30.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7d5ece640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7d0073d90 con 0x7fe7d014dbe0 2024-09-19T07:34:30.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7d5ece640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe7d0154700 con 0x7fe7d00c3ab0 2024-09-19T07:34:30.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7d5ece640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe7d0153b30 con 0x7fe7d00aa4a0 2024-09-19T07:34:30.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7c6ffd640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe7d00c3ab0 0x7fe7d0157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:30.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7cf7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fe7d014dbe0 0x7fe7d00c3fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:36698/0 (socket says 172.21.15.83:36698) 2024-09-19T07:34:30.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7cf7fe640 1 -- 172.21.15.83:0/721355519 learned_addr learned my addr 172.21.15.83:0/721355519 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:34:30.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7cffff640 1 --2- 172.21.15.83:0/721355519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7d00aa4a0 0x7fe7d0157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:30.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7ceffd640 1 -- 172.21.15.83:0/721355519 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 359408704 0 0) 0x7fe7d0073d90 con 0x7fe7d014dbe0 2024-09-19T07:34:30.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7c6ffd640 1 -- 172.21.15.83:0/721355519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7d00aa4a0 msgr2=0x7fe7d0157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:30.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7c6ffd640 1 --2- 172.21.15.83:0/721355519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7d00aa4a0 0x7fe7d0157550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:30.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7c6ffd640 1 -- 172.21.15.83:0/721355519 >> v1:172.21.15.184:6789/0 conn(0x7fe7d014dbe0 legacy=0x7fe7d00c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:30.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7c6ffd640 1 -- 172.21.15.83:0/721355519 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d0153dd0 con 0x7fe7d00c3ab0 2024-09-19T07:34:30.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7cffff640 1 --2- 172.21.15.83:0/721355519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7d00aa4a0 0x7fe7d0157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:34:30.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7c6ffd640 1 --2- 172.21.15.83:0/721355519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d00c3ab0 0x7fe7d0157010 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0154880 tx=0x7fe7b801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=309b61278fa20113 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:30.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7ceffd640 1 -- 172.21.15.83:0/721355519 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80b0070 con 0x7fe7d00c3ab0 2024-09-19T07:34:30.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7ceffd640 1 -- 172.21.15.83:0/721355519 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7b80ab600 con 0x7fe7d00c3ab0 2024-09-19T07:34:30.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.569+0000 7fe7ceffd640 1 -- 172.21.15.83:0/721355519 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80ab9d0 con 0x7fe7d00c3ab0 2024-09-19T07:34:30.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 -- 172.21.15.83:0/721355519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d00c3ab0 msgr2=0x7fe7d0157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:30.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 --2- 172.21.15.83:0/721355519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d00c3ab0 0x7fe7d0157010 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0154880 tx=0x7fe7b801c410 comp rx=0 tx=0).stop 2024-09-19T07:34:30.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 -- 172.21.15.83:0/721355519 shutdown_connections 2024-09-19T07:34:30.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 --2- 172.21.15.83:0/721355519 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7d00aa4a0 0x7fe7d0157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:30.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 --2- 172.21.15.83:0/721355519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d00c3ab0 0x7fe7d0157010 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:30.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 -- 172.21.15.83:0/721355519 >> 172.21.15.83:0/721355519 conn(0x7fe7d00c95a0 msgr2=0x7fe7d00c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:30.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 -- 172.21.15.83:0/721355519 shutdown_connections 2024-09-19T07:34:30.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 -- 172.21.15.83:0/721355519 wait complete. 2024-09-19T07:34:30.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 Processor -- start 2024-09-19T07:34:30.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 -- start start 2024-09-19T07:34:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d00aa4a0 0x7fe7d0150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7d014dbe0 0x7fe7d00ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7d00aec40 0x7fe7d00c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d00c4830 con 0x7fe7d014dbe0 2024-09-19T07:34:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d00c46b0 con 0x7fe7d00aa4a0 2024-09-19T07:34:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe7d0160bc0 con 0x7fe7d00aec40 2024-09-19T07:34:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7c6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7d014dbe0 0x7fe7d00ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7c6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7d014dbe0 0x7fe7d00ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40780/0 (socket says 172.21.15.83:40780) 2024-09-19T07:34:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7c6ffd640 1 -- 172.21.15.83:0/1134496768 learned_addr learned my addr 172.21.15.83:0/1134496768 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7cf7fe640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d00aa4a0 0x7fe7d0150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7cffff640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7d00aec40 0x7fe7d00c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7c6ffd640 1 -- 172.21.15.83:0/1134496768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7d00aec40 msgr2=0x7fe7d00c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7c6ffd640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7d00aec40 0x7fe7d00c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7c6ffd640 1 -- 172.21.15.83:0/1134496768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d00aa4a0 msgr2=0x7fe7d0150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7c6ffd640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d00aa4a0 0x7fe7d0150740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7c6ffd640 1 -- 172.21.15.83:0/1134496768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d00c7330 con 0x7fe7d014dbe0 2024-09-19T07:34:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7c6ffd640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7d014dbe0 0x7fe7d00ae700 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80ae040 tx=0x7fe7b80191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7ccff9640 1 -- 172.21.15.83:0/1134496768 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80b0040 con 0x7fe7d014dbe0 2024-09-19T07:34:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7ccff9640 1 -- 172.21.15.83:0/1134496768 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7b80ab400 con 0x7fe7d014dbe0 2024-09-19T07:34:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 -- 172.21.15.83:0/1134496768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d00c7610 con 0x7fe7d014dbe0 2024-09-19T07:34:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7ccff9640 1 -- 172.21.15.83:0/1134496768 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8006e20 con 0x7fe7d014dbe0 2024-09-19T07:34:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.573+0000 7fe7d5ece640 1 -- 172.21.15.83:0/1134496768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d00c7bd0 con 0x7fe7d014dbe0 2024-09-19T07:34:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.577+0000 7fe7d5ece640 1 -- 172.21.15.83:0/1134496768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d00c3fa0 con 0x7fe7d014dbe0 2024-09-19T07:34:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.577+0000 7fe7ccff9640 1 -- 172.21.15.83:0/1134496768 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe7b80bb020 con 0x7fe7d014dbe0 2024-09-19T07:34:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.577+0000 7fe7ccff9640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe7b00678c0 0x7fe7b0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.577+0000 7fe7ccff9640 1 -- 172.21.15.83:0/1134496768 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7fe7b8120350 con 0x7fe7d014dbe0 2024-09-19T07:34:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.577+0000 7fe7cf7fe640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe7b00678c0 0x7fe7b0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.577+0000 7fe7cf7fe640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe7b00678c0 0x7fe7b0069d80 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0007fd0 tx=0x7fe7c0007d60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:30.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.581+0000 7fe7ccff9640 1 -- 172.21.15.83:0/1134496768 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe7b80c2020 con 0x7fe7d014dbe0 2024-09-19T07:34:30.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.845+0000 7fe7d5ece640 1 -- 172.21.15.83:0/1134496768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7d00af050 con 0x7fe7d014dbe0 2024-09-19T07:34:30.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.845+0000 7fe7ccff9640 1 -- 172.21.15.83:0/1134496768 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+5717 (secure 0 0 0) 0x7fe7b8016090 con 0x7fe7d014dbe0 2024-09-19T07:34:30.840 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:30.840 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":118,"btime":"2024-09-19T07:34:21:357390+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6918,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:21.357382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:active","state_seq":54,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5722,"qdb_cluster":[5722]},"id":2}]} 2024-09-19T07:34:30.841 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 118 2024-09-19T07:34:30.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.849+0000 7fe7d5ece640 1 -- 172.21.15.83:0/1134496768 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe7b00678c0 msgr2=0x7fe7b0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:30.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.849+0000 7fe7d5ece640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe7b00678c0 0x7fe7b0069d80 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0007fd0 tx=0x7fe7c0007d60 comp rx=0 tx=0).stop 2024-09-19T07:34:30.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.849+0000 7fe7d5ece640 1 -- 172.21.15.83:0/1134496768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7d014dbe0 msgr2=0x7fe7d00ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:30.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.849+0000 7fe7d5ece640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7d014dbe0 0x7fe7d00ae700 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80ae040 tx=0x7fe7b80191b0 comp rx=0 tx=0).stop 2024-09-19T07:34:30.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.849+0000 7fe7d5ece640 1 -- 172.21.15.83:0/1134496768 shutdown_connections 2024-09-19T07:34:30.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.849+0000 7fe7d5ece640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe7b00678c0 0x7fe7b0069d80 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:30.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.849+0000 7fe7d5ece640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7d00aec40 0x7fe7d00c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:30.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.849+0000 7fe7d5ece640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7d014dbe0 0x7fe7d00ae700 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:30.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.849+0000 7fe7d5ece640 1 --2- 172.21.15.83:0/1134496768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7d00aa4a0 0x7fe7d0150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:30.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.849+0000 7fe7d5ece640 1 -- 172.21.15.83:0/1134496768 >> 172.21.15.83:0/1134496768 conn(0x7fe7d00c95a0 msgr2=0x7fe7d00ab810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:30.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.849+0000 7fe7d5ece640 1 -- 172.21.15.83:0/1134496768 shutdown_connections 2024-09-19T07:34:30.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:30.849+0000 7fe7d5ece640 1 -- 172.21.15.83:0/1134496768 wait complete. 2024-09-19T07:34:30.863 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-19T07:34:32.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:34:33.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.013+0000 7f5d19322640 1 Processor -- start 2024-09-19T07:34:33.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.013+0000 7f5d19322640 1 -- start start 2024-09-19T07:34:33.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.013+0000 7f5d19322640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5d140c4880 0x7f5d140c1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:33.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.013+0000 7f5d19322640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5d140c2330 0x7f5d14152f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:33.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.013+0000 7f5d19322640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d141538e0 con 0x7f5d14153470 2024-09-19T07:34:33.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.013+0000 7f5d19322640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5d1415a040 con 0x7f5d140c4880 2024-09-19T07:34:33.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.013+0000 7f5d19322640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5d140c2c10 con 0x7f5d140c2330 2024-09-19T07:34:33.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.013+0000 7f5d12ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5d140c4880 0x7f5d140c1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:33.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.013+0000 7f5d12ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5d140c4880 0x7f5d140c1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40784/0 (socket says 172.21.15.83:40784) 2024-09-19T07:34:33.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.013+0000 7f5d12ffd640 1 -- 172.21.15.83:0/466651180 learned_addr learned my addr 172.21.15.83:0/466651180 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:33.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.013+0000 7f5d0a7fc640 1 --2- 172.21.15.83:0/466651180 >> v2:172.21.15.184:3300/0 conn(0x7f5d140c2330 0x7f5d14152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:33.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d12ffd640 1 -- 172.21.15.83:0/466651180 >> v2:172.21.15.184:3300/0 conn(0x7f5d140c2330 msgr2=0x7f5d14152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:34:33.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d12ffd640 1 --2- 172.21.15.83:0/466651180 >> v2:172.21.15.184:3300/0 conn(0x7f5d140c2330 0x7f5d14152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:33.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d12ffd640 1 -- 172.21.15.83:0/466651180 >> v1:172.21.15.184:6789/0 conn(0x7f5d14153470 legacy=0x7f5d14155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:33.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d12ffd640 1 -- 172.21.15.83:0/466651180 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d14159430 con 0x7f5d140c4880 2024-09-19T07:34:33.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d12ffd640 1 --2- 172.21.15.83:0/466651180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d140c4880 0x7f5d140c1df0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f5d14155ca0 tx=0x7f5d0401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dff8051ea877b50 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:33.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d127fc640 1 -- 172.21.15.83:0/466651180 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d040b0070 con 0x7f5d140c4880 2024-09-19T07:34:33.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d127fc640 1 -- 172.21.15.83:0/466651180 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d040ab600 con 0x7f5d140c4880 2024-09-19T07:34:33.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d127fc640 1 -- 172.21.15.83:0/466651180 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d040ab9d0 con 0x7f5d140c4880 2024-09-19T07:34:33.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d19322640 1 -- 172.21.15.83:0/466651180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d140c4880 msgr2=0x7f5d140c1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:33.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d19322640 1 --2- 172.21.15.83:0/466651180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d140c4880 0x7f5d140c1df0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f5d14155ca0 tx=0x7f5d0401c3d0 comp rx=0 tx=0).stop 2024-09-19T07:34:33.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d19322640 1 -- 172.21.15.83:0/466651180 shutdown_connections 2024-09-19T07:34:33.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d19322640 1 --2- 172.21.15.83:0/466651180 >> v2:172.21.15.184:3300/0 conn(0x7f5d140c2330 0x7f5d14152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:33.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d19322640 1 --2- 172.21.15.83:0/466651180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d140c4880 0x7f5d140c1df0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:33.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d19322640 1 -- 172.21.15.83:0/466651180 >> 172.21.15.83:0/466651180 conn(0x7f5d140aa910 msgr2=0x7f5d140c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:33.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d19322640 1 -- 172.21.15.83:0/466651180 shutdown_connections 2024-09-19T07:34:33.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.017+0000 7f5d19322640 1 -- 172.21.15.83:0/466651180 wait complete. 2024-09-19T07:34:33.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d19322640 1 Processor -- start 2024-09-19T07:34:33.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d19322640 1 -- start start 2024-09-19T07:34:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d19322640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d140c2330 0x7f5d14155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d19322640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d140c4880 0x7f5d140ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d19322640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d14153470 0x7f5d140ac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d19322640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d14156ab0 con 0x7f5d140c4880 2024-09-19T07:34:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d19322640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d14156930 con 0x7f5d140c2330 2024-09-19T07:34:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d19322640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5d140c4c60 con 0x7f5d14153470 2024-09-19T07:34:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d0a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d140c4880 0x7f5d140ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d0a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d140c4880 0x7f5d140ab5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40796/0 (socket says 172.21.15.83:40796) 2024-09-19T07:34:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d0a7fc640 1 -- 172.21.15.83:0/3614130182 learned_addr learned my addr 172.21.15.83:0/3614130182 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:34:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d12ffd640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d140c2330 0x7f5d14155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d0a7fc640 1 -- 172.21.15.83:0/3614130182 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d14153470 msgr2=0x7f5d140ac0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d0a7fc640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d14153470 0x7f5d140ac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d0a7fc640 1 -- 172.21.15.83:0/3614130182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d140c2330 msgr2=0x7f5d14155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d0a7fc640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d140c2330 0x7f5d14155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d0a7fc640 1 -- 172.21.15.83:0/3614130182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d140a7fc0 con 0x7f5d140c4880 2024-09-19T07:34:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d0a7fc640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d140c4880 0x7f5d140ab5e0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00d800 tx=0x7f5cfc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d0bfff640 1 -- 172.21.15.83:0/3614130182 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc00b660 con 0x7f5d140c4880 2024-09-19T07:34:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d0bfff640 1 -- 172.21.15.83:0/3614130182 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cfc019070 con 0x7f5d140c4880 2024-09-19T07:34:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d19322640 1 -- 172.21.15.83:0/3614130182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d140a86d0 con 0x7f5d140c4880 2024-09-19T07:34:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d0bfff640 1 -- 172.21.15.83:0/3614130182 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc00ba00 con 0x7f5d140c4880 2024-09-19T07:34:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.021+0000 7f5d19322640 1 -- 172.21.15.83:0/3614130182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d14156320 con 0x7f5d140c4880 2024-09-19T07:34:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.025+0000 7f5d097fa640 1 -- 172.21.15.83:0/3614130182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cec0052a0 con 0x7f5d140c4880 2024-09-19T07:34:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.025+0000 7f5d0bfff640 1 -- 172.21.15.83:0/3614130182 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5cfc00bba0 con 0x7f5d140c4880 2024-09-19T07:34:33.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.025+0000 7f5d0bfff640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5cf40678c0 0x7f5cf4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:34:33.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.025+0000 7f5d0bfff640 1 -- 172.21.15.83:0/3614130182 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7f5cfc086b70 con 0x7f5d140c4880 2024-09-19T07:34:33.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.025+0000 7f5d12ffd640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5cf40678c0 0x7f5cf4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:34:33.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.025+0000 7f5d12ffd640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5cf40678c0 0x7f5cf4069d80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5d040ad000 tx=0x7f5d040a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:34:33.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.029+0000 7f5d0bfff640 1 -- 172.21.15.83:0/3614130182 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5cfc056b80 con 0x7f5d140c4880 2024-09-19T07:34:33.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 -- 172.21.15.83:0/3614130182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5cec005590 con 0x7f5d140c4880 2024-09-19T07:34:33.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d0bfff640 1 -- 172.21.15.83:0/3614130182 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+6543 (secure 0 0 0) 0x7f5cfc056480 con 0x7f5d140c4880 2024-09-19T07:34:33.247 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:34:33.247 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":119,"btime":"2024-09-19T07:34:31:383334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":6918,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:21.357382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:active","state_seq":54,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5722,"qdb_cluster":[5722]},"id":2}]} 2024-09-19T07:34:33.247 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 119 2024-09-19T07:34:33.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 -- 172.21.15.83:0/3614130182 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5cf40678c0 msgr2=0x7f5cf4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:33.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5cf40678c0 0x7f5cf4069d80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5d040ad000 tx=0x7f5d040a6000 comp rx=0 tx=0).stop 2024-09-19T07:34:33.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 -- 172.21.15.83:0/3614130182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d140c4880 msgr2=0x7f5d140ab5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:34:33.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d140c4880 0x7f5d140ab5e0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00d800 tx=0x7f5cfc00dcc0 comp rx=0 tx=0).stop 2024-09-19T07:34:33.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 -- 172.21.15.83:0/3614130182 shutdown_connections 2024-09-19T07:34:33.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5cf40678c0 0x7f5cf4069d80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:33.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d14153470 0x7f5d140ac0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:33.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d140c4880 0x7f5d140ab5e0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:33.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 --2- 172.21.15.83:0/3614130182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d140c2330 0x7f5d14155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:34:33.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 -- 172.21.15.83:0/3614130182 >> 172.21.15.83:0/3614130182 conn(0x7f5d140aa910 msgr2=0x7f5d14151110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:34:33.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 -- 172.21.15.83:0/3614130182 shutdown_connections 2024-09-19T07:34:33.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:34:33.253+0000 7f5d097fa640 1 -- 172.21.15.83:0/3614130182 wait complete. 2024-09-19T07:34:33.266 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-19T07:34:33.266 INFO:tasks.mds_thrash.fs.[b]:waiting for 42 secs before thrashing 2024-09-19T07:34:33.930 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:34:33.933 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:35:04.106 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:35:04.110 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:35:15.267 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a6d12c640 1 Processor -- start 2024-09-19T07:35:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a6d12c640 1 -- start start 2024-09-19T07:35:15.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a6d12c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5a680c6120 0x7f5a680c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:15.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a6d12c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a68155d70 0x7f5a681575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:15.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a6d12c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a68156200 con 0x7f5a681592f0 2024-09-19T07:35:15.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a6d12c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5a680c9930 con 0x7f5a680c6120 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a6d12c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5a681564e0 con 0x7f5a68155d70 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a6759e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5a681592f0 0x7f5a68157b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54334/0 (socket says 172.21.15.83:54334) 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a6759e640 1 -- 172.21.15.83:0/2286307661 learned_addr learned my addr 172.21.15.83:0/2286307661 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a66d9d640 1 --2- 172.21.15.83:0/2286307661 >> v2:172.21.15.184:3300/0 conn(0x7f5a680c6120 0x7f5a680c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a5ffff640 1 --2- 172.21.15.83:0/2286307661 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a68155d70 0x7f5a681575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a6659c640 1 -- 172.21.15.83:0/2286307661 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1876862468 0 0) 0x7f5a68156200 con 0x7f5a681592f0 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a6659c640 1 -- 172.21.15.83:0/2286307661 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a54003530 con 0x7f5a681592f0 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a6659c640 1 -- 172.21.15.83:0/2286307661 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1308863412 0 0) 0x7f5a54003530 con 0x7f5a681592f0 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a5ffff640 1 -- 172.21.15.83:0/2286307661 >> v2:172.21.15.184:3300/0 conn(0x7f5a680c6120 msgr2=0x7f5a680c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a5ffff640 1 --2- 172.21.15.83:0/2286307661 >> v2:172.21.15.184:3300/0 conn(0x7f5a680c6120 0x7f5a680c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a5ffff640 1 -- 172.21.15.83:0/2286307661 >> v1:172.21.15.83:6789/0 conn(0x7f5a681592f0 legacy=0x7f5a68157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a5ffff640 1 -- 172.21.15.83:0/2286307661 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a681567c0 con 0x7f5a68155d70 2024-09-19T07:35:15.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.415+0000 7f5a66d9d640 1 --2- 172.21.15.83:0/2286307661 >> v2:172.21.15.184:3300/0 conn(0x7f5a680c6120 0x7f5a680c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:35:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a5ffff640 1 --2- 172.21.15.83:0/2286307661 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a68155d70 0x7f5a681575d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5a500097c0 tx=0x7f5a5001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c858a657c6c239c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6659c640 1 -- 172.21.15.83:0/2286307661 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a500b0070 con 0x7f5a68155d70 2024-09-19T07:35:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6659c640 1 -- 172.21.15.83:0/2286307661 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a500ab400 con 0x7f5a68155d70 2024-09-19T07:35:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6659c640 1 -- 172.21.15.83:0/2286307661 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60001a00 con 0x7f5a68155d70 2024-09-19T07:35:15.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2286307661 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a68155d70 msgr2=0x7f5a681575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:15.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6d12c640 1 --2- 172.21.15.83:0/2286307661 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a68155d70 0x7f5a681575d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5a500097c0 tx=0x7f5a5001c3c0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2286307661 shutdown_connections 2024-09-19T07:35:15.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6d12c640 1 --2- 172.21.15.83:0/2286307661 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a68155d70 0x7f5a681575d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6d12c640 1 --2- 172.21.15.83:0/2286307661 >> v2:172.21.15.184:3300/0 conn(0x7f5a680c6120 0x7f5a680c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2286307661 >> 172.21.15.83:0/2286307661 conn(0x7f5a6806bc80 msgr2=0x7f5a6806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:15.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2286307661 shutdown_connections 2024-09-19T07:35:15.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2286307661 wait complete. 2024-09-19T07:35:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6d12c640 1 Processor -- start 2024-09-19T07:35:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.419+0000 7f5a6d12c640 1 -- start start 2024-09-19T07:35:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6d12c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a680c6120 0x7f5a680ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6d12c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a681592f0 0x7f5a680ab340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6d12c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a680ab880 0x7f5a680ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6d12c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a68153860 con 0x7f5a680ab880 2024-09-19T07:35:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6d12c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a68162ea0 con 0x7f5a681592f0 2024-09-19T07:35:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6d12c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5a6814ed20 con 0x7f5a680c6120 2024-09-19T07:35:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a680ab880 0x7f5a680ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a680ab880 0x7f5a680ad750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48760/0 (socket says 172.21.15.83:48760) 2024-09-19T07:35:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6759e640 1 -- 172.21.15.83:0/2729075015 learned_addr learned my addr 172.21.15.83:0/2729075015 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:15.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a5ffff640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a681592f0 0x7f5a680ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6759e640 1 -- 172.21.15.83:0/2729075015 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a680c6120 msgr2=0x7f5a680ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a66d9d640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a680c6120 0x7f5a680ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6759e640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a680c6120 0x7f5a680ca650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6759e640 1 -- 172.21.15.83:0/2729075015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a681592f0 msgr2=0x7f5a680ab340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:15.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6759e640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a681592f0 0x7f5a680ab340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6759e640 1 -- 172.21.15.83:0/2729075015 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a680ad2a0 con 0x7f5a680ab880 2024-09-19T07:35:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6759e640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a680ab880 0x7f5a680ad750 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000cbc0 tx=0x7f5a6000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a5f7fe640 1 -- 172.21.15.83:0/2729075015 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60017070 con 0x7f5a680ab880 2024-09-19T07:35:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a5f7fe640 1 -- 172.21.15.83:0/2729075015 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a6000ace0 con 0x7f5a680ab880 2024-09-19T07:35:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2729075015 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a6814ef70 con 0x7f5a680ab880 2024-09-19T07:35:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a5f7fe640 1 -- 172.21.15.83:0/2729075015 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60012600 con 0x7f5a680ab880 2024-09-19T07:35:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2729075015 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a6814c1b0 con 0x7f5a680ab880 2024-09-19T07:35:15.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2729075015 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a680c6500 con 0x7f5a680ab880 2024-09-19T07:35:15.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.423+0000 7f5a5f7fe640 1 -- 172.21.15.83:0/2729075015 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5a6001a070 con 0x7f5a680ab880 2024-09-19T07:35:15.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.427+0000 7f5a5f7fe640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a480678f0 0x7f5a48069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:15.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.427+0000 7f5a5f7fe640 1 -- 172.21.15.83:0/2729075015 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7f5a60086540 con 0x7f5a680ab880 2024-09-19T07:35:15.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.427+0000 7f5a66d9d640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a480678f0 0x7f5a48069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:15.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.427+0000 7f5a66d9d640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a480678f0 0x7f5a48069db0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5a58009580 tx=0x7f5a58008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:15.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.431+0000 7f5a5f7fe640 1 -- 172.21.15.83:0/2729075015 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5a60055ed0 con 0x7f5a680ab880 2024-09-19T07:35:15.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2729075015 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a680c9ac0 con 0x7f5a680ab880 2024-09-19T07:35:15.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a5f7fe640 1 -- 172.21.15.83:0/2729075015 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+6543 (secure 0 0 0) 0x7f5a6005a1d0 con 0x7f5a680ab880 2024-09-19T07:35:15.657 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:15.657 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":119,"btime":"2024-09-19T07:34:31:383334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":6918,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:21.357382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:active","state_seq":54,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5722,"qdb_cluster":[5722]},"id":2}]} 2024-09-19T07:35:15.658 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 119 2024-09-19T07:35:15.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2729075015 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a480678f0 msgr2=0x7f5a48069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:15.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a480678f0 0x7f5a48069db0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5a58009580 tx=0x7f5a58008040 comp rx=0 tx=0).stop 2024-09-19T07:35:15.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2729075015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a680ab880 msgr2=0x7f5a680ad750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a680ab880 0x7f5a680ad750 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000cbc0 tx=0x7f5a6000a430 comp rx=0 tx=0).stop 2024-09-19T07:35:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2729075015 shutdown_connections 2024-09-19T07:35:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a480678f0 0x7f5a48069db0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a680ab880 0x7f5a680ad750 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a681592f0 0x7f5a680ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 --2- 172.21.15.83:0/2729075015 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a680c6120 0x7f5a680ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2729075015 >> 172.21.15.83:0/2729075015 conn(0x7f5a6806bc80 msgr2=0x7f5a68151f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2729075015 shutdown_connections 2024-09-19T07:35:15.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.663+0000 7f5a6d12c640 1 -- 172.21.15.83:0/2729075015 wait complete. 2024-09-19T07:35:15.675 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-09-19T07:35:15.676 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-19T07:35:15.676 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:35:15.697 INFO:tasks.ceph.mds.b:Stopped 2024-09-19T07:35:15.698 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:15.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.823+0000 7fbd737e9640 1 Processor -- start 2024-09-19T07:35:15.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.823+0000 7fbd737e9640 1 -- start start 2024-09-19T07:35:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd737e9640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbd6c14a500 0x7fbd6c0c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd737e9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd6c151750 con 0x7fbd6c0c4550 2024-09-19T07:35:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd737e9640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd6c0c7e90 con 0x7fbd6c154540 2024-09-19T07:35:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd737e9640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbd6c073110 con 0x7fbd6c14a500 2024-09-19T07:35:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd71d86640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbd6c0c4550 0x7fbd6c15e440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54342/0 (socket says 172.21.15.83:54342) 2024-09-19T07:35:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd71d86640 1 -- 172.21.15.83:0/3161730674 learned_addr learned my addr 172.21.15.83:0/3161730674 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd71585640 1 --2- 172.21.15.83:0/3161730674 >> v2:172.21.15.184:3300/0 conn(0x7fbd6c14a500 0x7fbd6c0c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd63fff640 1 -- 172.21.15.83:0/3161730674 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3541600013 0 0) 0x7fbd640014a0 con 0x7fbd6c0c4550 2024-09-19T07:35:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd63fff640 1 -- 172.21.15.83:0/3161730674 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd58003530 con 0x7fbd6c0c4550 2024-09-19T07:35:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd63fff640 1 -- 172.21.15.83:0/3161730674 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3847465229 0 0) 0x7fbd58003530 con 0x7fbd6c0c4550 2024-09-19T07:35:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd71585640 1 -- 172.21.15.83:0/3161730674 >> v1:172.21.15.184:6789/0 conn(0x7fbd6c154540 legacy=0x7fbd6c157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd71585640 1 -- 172.21.15.83:0/3161730674 >> v1:172.21.15.83:6789/0 conn(0x7fbd6c0c4550 legacy=0x7fbd6c15e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:15.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.827+0000 7fbd71585640 1 -- 172.21.15.83:0/3161730674 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6c0733f0 con 0x7fbd6c14a500 2024-09-19T07:35:15.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd71585640 1 --2- 172.21.15.83:0/3161730674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbd6c14a500 0x7fbd6c0c4010 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbd680097c0 tx=0x7fbd6801c390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2ef7187c05334a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:15.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd63fff640 1 -- 172.21.15.83:0/3161730674 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd680b0070 con 0x7fbd6c14a500 2024-09-19T07:35:15.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd63fff640 1 -- 172.21.15.83:0/3161730674 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd680ab400 con 0x7fbd6c14a500 2024-09-19T07:35:15.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd63fff640 1 -- 172.21.15.83:0/3161730674 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd64001bd0 con 0x7fbd6c14a500 2024-09-19T07:35:15.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 -- 172.21.15.83:0/3161730674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbd6c14a500 msgr2=0x7fbd6c0c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:15.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 --2- 172.21.15.83:0/3161730674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbd6c14a500 0x7fbd6c0c4010 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbd680097c0 tx=0x7fbd6801c390 comp rx=0 tx=0).stop 2024-09-19T07:35:15.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 -- 172.21.15.83:0/3161730674 shutdown_connections 2024-09-19T07:35:15.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 --2- 172.21.15.83:0/3161730674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbd6c14a500 0x7fbd6c0c4010 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 -- 172.21.15.83:0/3161730674 >> 172.21.15.83:0/3161730674 conn(0x7fbd6c0c9610 msgr2=0x7fbd6c0c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:15.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 -- 172.21.15.83:0/3161730674 shutdown_connections 2024-09-19T07:35:15.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 -- 172.21.15.83:0/3161730674 wait complete. 2024-09-19T07:35:15.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 Processor -- start 2024-09-19T07:35:15.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 -- start start 2024-09-19T07:35:15.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbd6c0c4550 0x7fbd6c1ef570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:15.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbd6c14a500 0x7fbd6c0c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:15.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd6c154540 0x7fbd6c156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c0c7850 con 0x7fbd6c154540 2024-09-19T07:35:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c0c76d0 con 0x7fbd6c14a500 2024-09-19T07:35:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd737e9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbd6c162790 con 0x7fbd6c0c4550 2024-09-19T07:35:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd71d86640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd6c154540 0x7fbd6c156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd71d86640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd6c154540 0x7fbd6c156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48764/0 (socket says 172.21.15.83:48764) 2024-09-19T07:35:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd71d86640 1 -- 172.21.15.83:0/617637201 learned_addr learned my addr 172.21.15.83:0/617637201 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd70d84640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbd6c14a500 0x7fbd6c0c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd71585640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbd6c0c4550 0x7fbd6c1ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd71d86640 1 -- 172.21.15.83:0/617637201 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbd6c0c4550 msgr2=0x7fbd6c1ef570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd71d86640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbd6c0c4550 0x7fbd6c1ef570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd71d86640 1 -- 172.21.15.83:0/617637201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbd6c14a500 msgr2=0x7fbd6c0c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd71d86640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbd6c14a500 0x7fbd6c0c2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.831+0000 7fbd71d86640 1 -- 172.21.15.83:0/617637201 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6c149390 con 0x7fbd6c154540 2024-09-19T07:35:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.835+0000 7fbd71d86640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd6c154540 0x7fbd6c156940 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400cc00 tx=0x7fbd6400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.835+0000 7fbd61ffb640 1 -- 172.21.15.83:0/617637201 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6400ce10 con 0x7fbd6c154540 2024-09-19T07:35:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.835+0000 7fbd61ffb640 1 -- 172.21.15.83:0/617637201 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd64012400 con 0x7fbd6c154540 2024-09-19T07:35:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.835+0000 7fbd737e9640 1 -- 172.21.15.83:0/617637201 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd6c144430 con 0x7fbd6c154540 2024-09-19T07:35:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.835+0000 7fbd61ffb640 1 -- 172.21.15.83:0/617637201 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd640127f0 con 0x7fbd6c154540 2024-09-19T07:35:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.835+0000 7fbd737e9640 1 -- 172.21.15.83:0/617637201 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd6c1449f0 con 0x7fbd6c154540 2024-09-19T07:35:15.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.835+0000 7fbd737e9640 1 -- 172.21.15.83:0/617637201 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd6c0c4010 con 0x7fbd6c154540 2024-09-19T07:35:15.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.835+0000 7fbd61ffb640 1 -- 172.21.15.83:0/617637201 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbd64012990 con 0x7fbd6c154540 2024-09-19T07:35:15.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.835+0000 7fbd61ffb640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbd500678c0 0x7fbd50069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:15.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.835+0000 7fbd61ffb640 1 -- 172.21.15.83:0/617637201 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7fbd64087060 con 0x7fbd6c154540 2024-09-19T07:35:15.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.835+0000 7fbd71585640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbd500678c0 0x7fbd50069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:15.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.835+0000 7fbd71585640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbd500678c0 0x7fbd50069d80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbd680ad000 tx=0x7fbd680a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:15.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:15.839+0000 7fbd61ffb640 1 -- 172.21.15.83:0/617637201 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbd6c151750 con 0x7fbd6c154540 2024-09-19T07:35:16.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 -- 172.21.15.83:0/617637201 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd6c153640 con 0x7fbd6c154540 2024-09-19T07:35:16.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd61ffb640 1 -- 172.21.15.83:0/617637201 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+6543 (secure 0 0 0) 0x7fbd64057040 con 0x7fbd6c154540 2024-09-19T07:35:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:16.072 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":119,"btime":"2024-09-19T07:34:31:383334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":6918,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:21.357382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:active","state_seq":54,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5722,"qdb_cluster":[5722]},"id":2}]} 2024-09-19T07:35:16.072 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 119 2024-09-19T07:35:16.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 -- 172.21.15.83:0/617637201 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbd500678c0 msgr2=0x7fbd50069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbd500678c0 0x7fbd50069d80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbd680ad000 tx=0x7fbd680a6000 comp rx=0 tx=0).stop 2024-09-19T07:35:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 -- 172.21.15.83:0/617637201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd6c154540 msgr2=0x7fbd6c156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd6c154540 0x7fbd6c156940 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400cc00 tx=0x7fbd6400a430 comp rx=0 tx=0).stop 2024-09-19T07:35:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 -- 172.21.15.83:0/617637201 shutdown_connections 2024-09-19T07:35:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbd500678c0 0x7fbd50069d80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd6c154540 0x7fbd6c156940 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbd6c14a500 0x7fbd6c0c2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 --2- 172.21.15.83:0/617637201 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbd6c0c4550 0x7fbd6c1ef570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 -- 172.21.15.83:0/617637201 >> 172.21.15.83:0/617637201 conn(0x7fbd6c0c9610 msgr2=0x7fbd6c0c80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 -- 172.21.15.83:0/617637201 shutdown_connections 2024-09-19T07:35:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:16.079+0000 7fbd737e9640 1 -- 172.21.15.83:0/617637201 wait complete. 2024-09-19T07:35:16.090 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:35:18.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:18.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f7787239640 1 Processor -- start 2024-09-19T07:35:18.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f7787239640 1 -- start start 2024-09-19T07:35:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f7787239640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f778006cef0 0x7f77800c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f7787239640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f77801579a0 0x7f7780150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f7787239640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7780070830 con 0x7f7780070140 2024-09-19T07:35:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f7787239640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7780153cf0 con 0x7f77801579a0 2024-09-19T07:35:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f7787239640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7780157e50 con 0x7f778006cef0 2024-09-19T07:35:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f77857d6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f77801579a0 0x7f7780150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f77857d6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f77801579a0 0x7f7780150ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48766/0 (socket says 172.21.15.83:48766) 2024-09-19T07:35:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f777ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7780070140 0x7f7780154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54348/0 (socket says 172.21.15.83:54348) 2024-09-19T07:35:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f77857d6640 1 -- 172.21.15.83:0/1903705943 learned_addr learned my addr 172.21.15.83:0/1903705943 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f7784fd5640 1 --2- 172.21.15.83:0/1903705943 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f778006cef0 0x7f77800c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f77857d6640 1 -- 172.21.15.83:0/1903705943 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f778006cef0 msgr2=0x7f77800c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f77857d6640 1 --2- 172.21.15.83:0/1903705943 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f778006cef0 0x7f77800c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f77857d6640 1 -- 172.21.15.83:0/1903705943 >> v1:172.21.15.83:6789/0 conn(0x7f7780070140 legacy=0x7f7780154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.219+0000 7f77857d6640 1 -- 172.21.15.83:0/1903705943 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77801539a0 con 0x7f77801579a0 2024-09-19T07:35:18.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.223+0000 7f77857d6640 1 --2- 172.21.15.83:0/1903705943 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77801579a0 0x7f7780150ba0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f7778009e50 tx=0x7f777801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd58b34a5f71f0c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:18.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.223+0000 7f777f7fe640 1 -- 172.21.15.83:0/1903705943 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77780a5750 con 0x7f77801579a0 2024-09-19T07:35:18.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.223+0000 7f777f7fe640 1 -- 172.21.15.83:0/1903705943 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77780a58f0 con 0x7f77801579a0 2024-09-19T07:35:18.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.223+0000 7f777f7fe640 1 -- 172.21.15.83:0/1903705943 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77780a5cc0 con 0x7f77801579a0 2024-09-19T07:35:18.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.223+0000 7f7787239640 1 -- 172.21.15.83:0/1903705943 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77801579a0 msgr2=0x7f7780150ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:18.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.223+0000 7f7787239640 1 --2- 172.21.15.83:0/1903705943 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77801579a0 0x7f7780150ba0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f7778009e50 tx=0x7f777801c9a0 comp rx=0 tx=0).stop 2024-09-19T07:35:18.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.223+0000 7f7787239640 1 -- 172.21.15.83:0/1903705943 shutdown_connections 2024-09-19T07:35:18.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.223+0000 7f7787239640 1 --2- 172.21.15.83:0/1903705943 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77801579a0 0x7f7780150ba0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:18.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.223+0000 7f7787239640 1 --2- 172.21.15.83:0/1903705943 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f778006cef0 0x7f77800c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:18.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.223+0000 7f7787239640 1 -- 172.21.15.83:0/1903705943 >> 172.21.15.83:0/1903705943 conn(0x7f77800c5090 msgr2=0x7f77800c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:18.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.223+0000 7f7787239640 1 -- 172.21.15.83:0/1903705943 shutdown_connections 2024-09-19T07:35:18.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.223+0000 7f7787239640 1 -- 172.21.15.83:0/1903705943 wait complete. 2024-09-19T07:35:18.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f7787239640 1 Processor -- start 2024-09-19T07:35:18.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f7787239640 1 -- start start 2024-09-19T07:35:18.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f7787239640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f778006cef0 0x7f77800c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:18.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f7787239640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7780070140 0x7f7780147f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:18.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f7787239640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77801579a0 0x7f77801484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:18.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f7787239640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f778006d2d0 con 0x7f7780070140 2024-09-19T07:35:18.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f7787239640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f778006f8a0 con 0x7f77801579a0 2024-09-19T07:35:18.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f7787239640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f778006d450 con 0x7f778006cef0 2024-09-19T07:35:18.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f777ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7780070140 0x7f7780147f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:18.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f777ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7780070140 0x7f7780147f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48780/0 (socket says 172.21.15.83:48780) 2024-09-19T07:35:18.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f777ffff640 1 -- 172.21.15.83:0/3440795736 learned_addr learned my addr 172.21.15.83:0/3440795736 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:18.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f7784fd5640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f778006cef0 0x7f77800c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:18.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f77857d6640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77801579a0 0x7f77801484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:18.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f777ffff640 1 -- 172.21.15.83:0/3440795736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f778006cef0 msgr2=0x7f77800c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:18.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f777ffff640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f778006cef0 0x7f77800c7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:18.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f777ffff640 1 -- 172.21.15.83:0/3440795736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77801579a0 msgr2=0x7f77801484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:18.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f777ffff640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77801579a0 0x7f77801484c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:18.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f777ffff640 1 -- 172.21.15.83:0/3440795736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f778014cd70 con 0x7f7780070140 2024-09-19T07:35:18.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f7784fd5640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f778006cef0 0x7f77800c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:35:18.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f777ffff640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7780070140 0x7f7780147f80 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f776c00c460 tx=0x7f776c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:18.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f777d7fa640 1 -- 172.21.15.83:0/3440795736 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c018070 con 0x7f7780070140 2024-09-19T07:35:18.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f777d7fa640 1 -- 172.21.15.83:0/3440795736 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f776c007480 con 0x7f7780070140 2024-09-19T07:35:18.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f7787239640 1 -- 172.21.15.83:0/3440795736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f778014d050 con 0x7f7780070140 2024-09-19T07:35:18.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f777d7fa640 1 -- 172.21.15.83:0/3440795736 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c0047f0 con 0x7f7780070140 2024-09-19T07:35:18.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f7787239640 1 -- 172.21.15.83:0/3440795736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f778014d610 con 0x7f7780070140 2024-09-19T07:35:18.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.227+0000 7f776affd640 1 -- 172.21.15.83:0/3440795736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77580052a0 con 0x7f7780070140 2024-09-19T07:35:18.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.231+0000 7f777d7fa640 1 -- 172.21.15.83:0/3440795736 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f776c01b070 con 0x7f7780070140 2024-09-19T07:35:18.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.231+0000 7f777d7fa640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77640678c0 0x7f7764069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:18.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.231+0000 7f777d7fa640 1 -- 172.21.15.83:0/3440795736 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7f776c085a30 con 0x7f7780070140 2024-09-19T07:35:18.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.231+0000 7f7784fd5640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77640678c0 0x7f7764069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:18.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.231+0000 7f7784fd5640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77640678c0 0x7f7764069d80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7774007a70 tx=0x7f7774008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:18.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.235+0000 7f777d7fa640 1 -- 172.21.15.83:0/3440795736 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f776c055aa0 con 0x7f7780070140 2024-09-19T07:35:18.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.439+0000 7f776affd640 1 -- 172.21.15.83:0/3440795736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7758005590 con 0x7f7780070140 2024-09-19T07:35:18.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.439+0000 7f777d7fa640 1 -- 172.21.15.83:0/3440795736 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+6543 (secure 0 0 0) 0x7f776c0553a0 con 0x7f7780070140 2024-09-19T07:35:18.434 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:18.434 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":119,"btime":"2024-09-19T07:34:31:383334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":6918,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:21.357382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:active","state_seq":54,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5722,"qdb_cluster":[5722]},"id":2}]} 2024-09-19T07:35:18.434 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 119 2024-09-19T07:35:18.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.443+0000 7f776affd640 1 -- 172.21.15.83:0/3440795736 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77640678c0 msgr2=0x7f7764069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:18.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.443+0000 7f776affd640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77640678c0 0x7f7764069d80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7774007a70 tx=0x7f7774008040 comp rx=0 tx=0).stop 2024-09-19T07:35:18.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.443+0000 7f776affd640 1 -- 172.21.15.83:0/3440795736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7780070140 msgr2=0x7f7780147f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:18.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.443+0000 7f776affd640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7780070140 0x7f7780147f80 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f776c00c460 tx=0x7f776c00c920 comp rx=0 tx=0).stop 2024-09-19T07:35:18.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.443+0000 7f776affd640 1 -- 172.21.15.83:0/3440795736 shutdown_connections 2024-09-19T07:35:18.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.443+0000 7f776affd640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77640678c0 0x7f7764069d80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:18.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.443+0000 7f776affd640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77801579a0 0x7f77801484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:18.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.443+0000 7f776affd640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7780070140 0x7f7780147f80 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:18.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.443+0000 7f776affd640 1 --2- 172.21.15.83:0/3440795736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f778006cef0 0x7f77800c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:18.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.443+0000 7f776affd640 1 -- 172.21.15.83:0/3440795736 >> 172.21.15.83:0/3440795736 conn(0x7f77800c5090 msgr2=0x7f7780151810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:18.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.443+0000 7f776affd640 1 -- 172.21.15.83:0/3440795736 shutdown_connections 2024-09-19T07:35:18.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:18.443+0000 7f776affd640 1 -- 172.21.15.83:0/3440795736 wait complete. 2024-09-19T07:35:18.454 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:35:20.454 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:20.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.623+0000 7fcc5b2d6640 1 Processor -- start 2024-09-19T07:35:20.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.623+0000 7fcc5b2d6640 1 -- start start 2024-09-19T07:35:20.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.623+0000 7fcc5b2d6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc54150040 0x7fcc5406d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:20.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.623+0000 7fcc5b2d6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcc540ad010 0x7fcc54152930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:20.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.623+0000 7fcc5b2d6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc54153240 con 0x7fcc54071ea0 2024-09-19T07:35:20.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.623+0000 7fcc5b2d6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fcc54153520 con 0x7fcc540ad010 2024-09-19T07:35:20.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.623+0000 7fcc5b2d6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc54153800 con 0x7fcc54150040 2024-09-19T07:35:20.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.623+0000 7fcc58871640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcc540ad010 0x7fcc54152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:20.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.623+0000 7fcc59873640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcc54071ea0 0x7fcc540c3950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60652/0 (socket says 172.21.15.83:60652) 2024-09-19T07:35:20.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.623+0000 7fcc59873640 1 -- 172.21.15.83:0/815175018 learned_addr learned my addr 172.21.15.83:0/815175018 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:20.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc58871640 1 -- 172.21.15.83:0/815175018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc54150040 msgr2=0x7fcc5406d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:20.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc58871640 1 --2- 172.21.15.83:0/815175018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc54150040 0x7fcc5406d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:20.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc58871640 1 -- 172.21.15.83:0/815175018 >> v1:172.21.15.83:6789/0 conn(0x7fcc54071ea0 legacy=0x7fcc540c3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:20.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc58871640 1 -- 172.21.15.83:0/815175018 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc540716a0 con 0x7fcc540ad010 2024-09-19T07:35:20.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc58871640 1 --2- 172.21.15.83:0/815175018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc540ad010 0x7fcc54152930 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fcc440097c0 tx=0x7fcc4401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c48dc17f5b0459a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:20.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc4bfff640 1 -- 172.21.15.83:0/815175018 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc440b0070 con 0x7fcc540ad010 2024-09-19T07:35:20.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc4bfff640 1 -- 172.21.15.83:0/815175018 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc440ab600 con 0x7fcc540ad010 2024-09-19T07:35:20.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc4bfff640 1 -- 172.21.15.83:0/815175018 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc440ab9d0 con 0x7fcc540ad010 2024-09-19T07:35:20.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/815175018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc540ad010 msgr2=0x7fcc54152930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:20.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc5b2d6640 1 --2- 172.21.15.83:0/815175018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc540ad010 0x7fcc54152930 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fcc440097c0 tx=0x7fcc4401c410 comp rx=0 tx=0).stop 2024-09-19T07:35:20.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/815175018 shutdown_connections 2024-09-19T07:35:20.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc5b2d6640 1 --2- 172.21.15.83:0/815175018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc540ad010 0x7fcc54152930 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:20.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc5b2d6640 1 --2- 172.21.15.83:0/815175018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc54150040 0x7fcc5406d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:20.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/815175018 >> 172.21.15.83:0/815175018 conn(0x7fcc540c7e40 msgr2=0x7fcc540c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:20.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/815175018 shutdown_connections 2024-09-19T07:35:20.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.627+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/815175018 wait complete. 2024-09-19T07:35:20.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc5b2d6640 1 Processor -- start 2024-09-19T07:35:20.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc5b2d6640 1 -- start start 2024-09-19T07:35:20.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc5b2d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc54071ea0 0x7fcc541ecce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:20.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc5b2d6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc540ad010 0x7fcc54147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc5b2d6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc54150040 0x7fcc54156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc5b2d6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcc541505a0 con 0x7fcc54071ea0 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc5b2d6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcc54150420 con 0x7fcc54150040 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc5b2d6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5415fad0 con 0x7fcc540ad010 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc59072640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc54071ea0 0x7fcc541ecce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc59072640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc54071ea0 0x7fcc541ecce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60628/0 (socket says 172.21.15.83:60628) 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc59072640 1 -- 172.21.15.83:0/3934178785 learned_addr learned my addr 172.21.15.83:0/3934178785 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc58871640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc540ad010 0x7fcc54147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc59072640 1 -- 172.21.15.83:0/3934178785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc540ad010 msgr2=0x7fcc54147b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc59072640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc540ad010 0x7fcc54147b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc59873640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc54150040 0x7fcc54156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc59072640 1 -- 172.21.15.83:0/3934178785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc54150040 msgr2=0x7fcc54156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:20.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc59072640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc54150040 0x7fcc54156410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:20.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc59072640 1 -- 172.21.15.83:0/3934178785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5414f3d0 con 0x7fcc54071ea0 2024-09-19T07:35:20.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc59072640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc54071ea0 0x7fcc541ecce0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000b550 tx=0x7fcc5000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:20.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc49ffb640 1 -- 172.21.15.83:0/3934178785 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc50014070 con 0x7fcc54071ea0 2024-09-19T07:35:20.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc49ffb640 1 -- 172.21.15.83:0/3934178785 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc50010890 con 0x7fcc54071ea0 2024-09-19T07:35:20.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/3934178785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc5414ee10 con 0x7fcc54071ea0 2024-09-19T07:35:20.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc49ffb640 1 -- 172.21.15.83:0/3934178785 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc50010c80 con 0x7fcc54071ea0 2024-09-19T07:35:20.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/3934178785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc540c74f0 con 0x7fcc54071ea0 2024-09-19T07:35:20.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.631+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/3934178785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc54152930 con 0x7fcc54071ea0 2024-09-19T07:35:20.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.635+0000 7fcc49ffb640 1 -- 172.21.15.83:0/3934178785 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcc50017070 con 0x7fcc54071ea0 2024-09-19T07:35:20.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.635+0000 7fcc49ffb640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc380678c0 0x7fcc38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:20.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.635+0000 7fcc49ffb640 1 -- 172.21.15.83:0/3934178785 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7fcc50086530 con 0x7fcc54071ea0 2024-09-19T07:35:20.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.635+0000 7fcc58871640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc380678c0 0x7fcc38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:20.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.635+0000 7fcc58871640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc380678c0 0x7fcc38069d80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fcc440ad000 tx=0x7fcc440a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:20.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.639+0000 7fcc49ffb640 1 -- 172.21.15.83:0/3934178785 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcc500565a0 con 0x7fcc54071ea0 2024-09-19T07:35:20.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.887+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/3934178785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc5406db10 con 0x7fcc54071ea0 2024-09-19T07:35:20.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.887+0000 7fcc49ffb640 1 -- 172.21.15.83:0/3934178785 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+6543 (secure 0 0 0) 0x7fcc50010e20 con 0x7fcc54071ea0 2024-09-19T07:35:20.882 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:20.882 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":119,"btime":"2024-09-19T07:34:31:383334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":6918,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:21.357382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:active","state_seq":54,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5722,"qdb_cluster":[5722]},"id":2}]} 2024-09-19T07:35:20.882 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 119 2024-09-19T07:35:20.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.891+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/3934178785 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc380678c0 msgr2=0x7fcc38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:20.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.891+0000 7fcc5b2d6640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc380678c0 0x7fcc38069d80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fcc440ad000 tx=0x7fcc440a6000 comp rx=0 tx=0).stop 2024-09-19T07:35:20.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.891+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/3934178785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc54071ea0 msgr2=0x7fcc541ecce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:20.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.891+0000 7fcc5b2d6640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc54071ea0 0x7fcc541ecce0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000b550 tx=0x7fcc5000ba10 comp rx=0 tx=0).stop 2024-09-19T07:35:20.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.891+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/3934178785 shutdown_connections 2024-09-19T07:35:20.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.891+0000 7fcc5b2d6640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc380678c0 0x7fcc38069d80 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:20.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.891+0000 7fcc5b2d6640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc54150040 0x7fcc54156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:20.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.891+0000 7fcc5b2d6640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc540ad010 0x7fcc54147b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:20.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.891+0000 7fcc5b2d6640 1 --2- 172.21.15.83:0/3934178785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc54071ea0 0x7fcc541ecce0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:20.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.891+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/3934178785 >> 172.21.15.83:0/3934178785 conn(0x7fcc540c7e40 msgr2=0x7fcc54151520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:20.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.891+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/3934178785 shutdown_connections 2024-09-19T07:35:20.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:20.891+0000 7fcc5b2d6640 1 -- 172.21.15.83:0/3934178785 wait complete. 2024-09-19T07:35:20.904 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:35:22.905 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:23.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa335397640 1 Processor -- start 2024-09-19T07:35:23.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa335397640 1 -- start start 2024-09-19T07:35:23.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa335397640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3300c4a60 0x7fa330068290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:23.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa335397640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa330073de0 0x7fa3300c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:23.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa335397640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fa33006f0c0 0x7fa3300c0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:23.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa335397640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa3301565d0 con 0x7fa330073de0 2024-09-19T07:35:23.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa335397640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fa33006f550 con 0x7fa33006f0c0 2024-09-19T07:35:23.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa335397640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa3300c1c20 con 0x7fa3300c4a60 2024-09-19T07:35:23.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa32f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa330073de0 0x7fa3300c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:23.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa32f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa330073de0 0x7fa3300c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60640/0 (socket says 172.21.15.83:60640) 2024-09-19T07:35:23.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa32f7fe640 1 -- 172.21.15.83:0/3009330469 learned_addr learned my addr 172.21.15.83:0/3009330469 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:23.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa32ffff640 1 --2- 172.21.15.83:0/3009330469 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3300c4a60 0x7fa330068290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:23.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa334b96640 1 --2- 172.21.15.83:0/3009330469 >> v2:172.21.15.184:3300/0 conn(0x7fa33006f0c0 0x7fa3300c0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:23.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa32f7fe640 1 -- 172.21.15.83:0/3009330469 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3300c4a60 msgr2=0x7fa330068290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:23.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa32f7fe640 1 --2- 172.21.15.83:0/3009330469 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3300c4a60 0x7fa330068290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:23.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa32f7fe640 1 -- 172.21.15.83:0/3009330469 >> v2:172.21.15.184:3300/0 conn(0x7fa33006f0c0 msgr2=0x7fa3300c0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:23.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa32f7fe640 1 --2- 172.21.15.83:0/3009330469 >> v2:172.21.15.184:3300/0 conn(0x7fa33006f0c0 0x7fa3300c0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:23.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.047+0000 7fa32f7fe640 1 -- 172.21.15.83:0/3009330469 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa330156870 con 0x7fa330073de0 2024-09-19T07:35:23.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa32f7fe640 1 --2- 172.21.15.83:0/3009330469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa330073de0 0x7fa3300c3f80 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fa330074430 tx=0x7fa32401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a215ae9c72a1008a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:23.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa32effd640 1 -- 172.21.15.83:0/3009330469 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3240b0070 con 0x7fa330073de0 2024-09-19T07:35:23.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa32effd640 1 -- 172.21.15.83:0/3009330469 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3240ab600 con 0x7fa330073de0 2024-09-19T07:35:23.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa32effd640 1 -- 172.21.15.83:0/3009330469 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3240ab9d0 con 0x7fa330073de0 2024-09-19T07:35:23.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 -- 172.21.15.83:0/3009330469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa330073de0 msgr2=0x7fa3300c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:23.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 --2- 172.21.15.83:0/3009330469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa330073de0 0x7fa3300c3f80 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fa330074430 tx=0x7fa32401c520 comp rx=0 tx=0).stop 2024-09-19T07:35:23.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 -- 172.21.15.83:0/3009330469 shutdown_connections 2024-09-19T07:35:23.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 --2- 172.21.15.83:0/3009330469 >> v2:172.21.15.184:3300/0 conn(0x7fa33006f0c0 0x7fa3300c0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:23.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 --2- 172.21.15.83:0/3009330469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa330073de0 0x7fa3300c3f80 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:23.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 --2- 172.21.15.83:0/3009330469 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3300c4a60 0x7fa330068290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:23.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 -- 172.21.15.83:0/3009330469 >> 172.21.15.83:0/3009330469 conn(0x7fa3300c7b00 msgr2=0x7fa3300c7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:23.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 -- 172.21.15.83:0/3009330469 shutdown_connections 2024-09-19T07:35:23.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 -- 172.21.15.83:0/3009330469 wait complete. 2024-09-19T07:35:23.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 Processor -- start 2024-09-19T07:35:23.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 -- start start 2024-09-19T07:35:23.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa33006f0c0 0x7fa330157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:23.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.051+0000 7fa335397640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa330073de0 0x7fa3301501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:23.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa335397640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3300c4a60 0x7fa3301f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa335397640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa3300c2a40 con 0x7fa33006f0c0 2024-09-19T07:35:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa335397640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa3300c28c0 con 0x7fa3300c4a60 2024-09-19T07:35:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa335397640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa3300a7300 con 0x7fa330073de0 2024-09-19T07:35:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa33006f0c0 0x7fa330157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa33006f0c0 0x7fa330157e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60652/0 (socket says 172.21.15.83:60652) 2024-09-19T07:35:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32ffff640 1 -- 172.21.15.83:0/137486691 learned_addr learned my addr 172.21.15.83:0/137486691 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32f7fe640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa330073de0 0x7fa3301501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa334b96640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3300c4a60 0x7fa3301f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32ffff640 1 -- 172.21.15.83:0/137486691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa330073de0 msgr2=0x7fa3301501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32ffff640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa330073de0 0x7fa3301501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32ffff640 1 -- 172.21.15.83:0/137486691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3300c4a60 msgr2=0x7fa3301f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32ffff640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3300c4a60 0x7fa3301f1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:23.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32ffff640 1 -- 172.21.15.83:0/137486691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3301f2660 con 0x7fa33006f0c0 2024-09-19T07:35:23.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa334b96640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3300c4a60 0x7fa3301f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:35:23.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32ffff640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa33006f0c0 0x7fa330157e40 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fa32000e8a0 tx=0x7fa32000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:23.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32cff9640 1 -- 172.21.15.83:0/137486691 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa320017070 con 0x7fa33006f0c0 2024-09-19T07:35:23.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32cff9640 1 -- 172.21.15.83:0/137486691 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa32000c940 con 0x7fa33006f0c0 2024-09-19T07:35:23.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa335397640 1 -- 172.21.15.83:0/137486691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3301576b0 con 0x7fa33006f0c0 2024-09-19T07:35:23.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32cff9640 1 -- 172.21.15.83:0/137486691 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32000cd10 con 0x7fa33006f0c0 2024-09-19T07:35:23.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa335397640 1 -- 172.21.15.83:0/137486691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa330149c90 con 0x7fa33006f0c0 2024-09-19T07:35:23.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa335397640 1 -- 172.21.15.83:0/137486691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3300c0f30 con 0x7fa33006f0c0 2024-09-19T07:35:23.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32cff9640 1 -- 172.21.15.83:0/137486691 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa32001a070 con 0x7fa33006f0c0 2024-09-19T07:35:23.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32cff9640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa31c0678c0 0x7fa31c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:23.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32cff9640 1 -- 172.21.15.83:0/137486691 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7fa3200869b0 con 0x7fa33006f0c0 2024-09-19T07:35:23.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32f7fe640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa31c0678c0 0x7fa31c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:23.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.055+0000 7fa32f7fe640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa31c0678c0 0x7fa31c069d80 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa3240ad000 tx=0x7fa324008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:23.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.059+0000 7fa32cff9640 1 -- 172.21.15.83:0/137486691 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa3200563d0 con 0x7fa33006f0c0 2024-09-19T07:35:23.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.307+0000 7fa335397640 1 -- 172.21.15.83:0/137486691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa33006cc90 con 0x7fa33006f0c0 2024-09-19T07:35:23.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.307+0000 7fa32cff9640 1 -- 172.21.15.83:0/137486691 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+6543 (secure 0 0 0) 0x7fa32005a6d0 con 0x7fa33006f0c0 2024-09-19T07:35:23.302 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:23.302 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":119,"btime":"2024-09-19T07:34:31:383334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":6918,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:21.357382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:active","state_seq":54,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5722,"qdb_cluster":[5722]},"id":2}]} 2024-09-19T07:35:23.302 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 119 2024-09-19T07:35:23.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.311+0000 7fa335397640 1 -- 172.21.15.83:0/137486691 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa31c0678c0 msgr2=0x7fa31c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:23.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.311+0000 7fa335397640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa31c0678c0 0x7fa31c069d80 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa3240ad000 tx=0x7fa324008040 comp rx=0 tx=0).stop 2024-09-19T07:35:23.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.311+0000 7fa335397640 1 -- 172.21.15.83:0/137486691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa33006f0c0 msgr2=0x7fa330157e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:23.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.311+0000 7fa335397640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa33006f0c0 0x7fa330157e40 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fa32000e8a0 tx=0x7fa32000ed60 comp rx=0 tx=0).stop 2024-09-19T07:35:23.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.311+0000 7fa335397640 1 -- 172.21.15.83:0/137486691 shutdown_connections 2024-09-19T07:35:23.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.311+0000 7fa335397640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa31c0678c0 0x7fa31c069d80 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:23.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.311+0000 7fa335397640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3300c4a60 0x7fa3301f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:23.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.311+0000 7fa335397640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa330073de0 0x7fa3301501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:23.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.311+0000 7fa335397640 1 --2- 172.21.15.83:0/137486691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa33006f0c0 0x7fa330157e40 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:23.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.311+0000 7fa335397640 1 -- 172.21.15.83:0/137486691 >> 172.21.15.83:0/137486691 conn(0x7fa3300c7b00 msgr2=0x7fa3300c62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:23.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.311+0000 7fa335397640 1 -- 172.21.15.83:0/137486691 shutdown_connections 2024-09-19T07:35:23.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:23.311+0000 7fa335397640 1 -- 172.21.15.83:0/137486691 wait complete. 2024-09-19T07:35:23.325 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:35:25.327 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:25.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f010859a640 1 Processor -- start 2024-09-19T07:35:25.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f010859a640 1 -- start start 2024-09-19T07:35:25.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f010859a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0100071e20 0x7f01000716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:25.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f010859a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f01001515e0 0x7f01001575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:25.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f010859a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01000c37c0 con 0x7f010006f950 2024-09-19T07:35:25.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f010859a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f01000c4670 con 0x7f01001515e0 2024-09-19T07:35:25.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f010859a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f01000c3a60 con 0x7f0100071e20 2024-09-19T07:35:25.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f0105b35640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0100071e20 0x7f01000716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:25.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f0106b37640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f01001515e0 0x7f01001575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:25.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f0105b35640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0100071e20 0x7f01000716a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:35270/0 (socket says 172.21.15.83:35270) 2024-09-19T07:35:25.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f0105b35640 1 -- 172.21.15.83:0/2788949674 learned_addr learned my addr 172.21.15.83:0/2788949674 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:25.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f0105b35640 1 -- 172.21.15.83:0/2788949674 >> v2:172.21.15.184:3300/0 conn(0x7f01001515e0 msgr2=0x7f01001575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:25.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f0105b35640 1 --2- 172.21.15.83:0/2788949674 >> v2:172.21.15.184:3300/0 conn(0x7f01001515e0 0x7f01001575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:25.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f0105b35640 1 -- 172.21.15.83:0/2788949674 >> v1:172.21.15.184:6789/0 conn(0x7f010006f950 legacy=0x7f0100150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:25.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f0105b35640 1 -- 172.21.15.83:0/2788949674 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01000c3d40 con 0x7f0100071e20 2024-09-19T07:35:25.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.475+0000 7f0106b37640 1 --2- 172.21.15.83:0/2788949674 >> v2:172.21.15.184:3300/0 conn(0x7f01001515e0 0x7f01001575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:35:25.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.479+0000 7f0105b35640 1 --2- 172.21.15.83:0/2788949674 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0100071e20 0x7f01000716a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f00f0009870 tx=0x7f00f001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32b671b384ec6fb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:25.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.479+0000 7f0105334640 1 -- 172.21.15.83:0/2788949674 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f00b0070 con 0x7f0100071e20 2024-09-19T07:35:25.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.479+0000 7f0105334640 1 -- 172.21.15.83:0/2788949674 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00f00ab600 con 0x7f0100071e20 2024-09-19T07:35:25.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.479+0000 7f0105334640 1 -- 172.21.15.83:0/2788949674 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f00ab9d0 con 0x7f0100071e20 2024-09-19T07:35:25.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.479+0000 7f010859a640 1 -- 172.21.15.83:0/2788949674 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0100071e20 msgr2=0x7f01000716a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:25.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.479+0000 7f010859a640 1 --2- 172.21.15.83:0/2788949674 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0100071e20 0x7f01000716a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f00f0009870 tx=0x7f00f001c4d0 comp rx=0 tx=0).stop 2024-09-19T07:35:25.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.479+0000 7f010859a640 1 -- 172.21.15.83:0/2788949674 shutdown_connections 2024-09-19T07:35:25.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.479+0000 7f010859a640 1 --2- 172.21.15.83:0/2788949674 >> v2:172.21.15.184:3300/0 conn(0x7f01001515e0 0x7f01001575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:25.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.479+0000 7f010859a640 1 --2- 172.21.15.83:0/2788949674 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0100071e20 0x7f01000716a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:25.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.479+0000 7f010859a640 1 -- 172.21.15.83:0/2788949674 >> 172.21.15.83:0/2788949674 conn(0x7f01000c48b0 msgr2=0x7f01000c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:25.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.479+0000 7f010859a640 1 -- 172.21.15.83:0/2788949674 shutdown_connections 2024-09-19T07:35:25.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.479+0000 7f010859a640 1 -- 172.21.15.83:0/2788949674 wait complete. 2024-09-19T07:35:25.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f010859a640 1 Processor -- start 2024-09-19T07:35:25.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f010859a640 1 -- start start 2024-09-19T07:35:25.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f010859a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f010006f950 0x7f0100156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:25.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f010859a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0100071e20 0x7f01001414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:25.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f010859a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01001515e0 0x7f0100141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:25.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f010859a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0100160690 con 0x7f01001515e0 2024-09-19T07:35:25.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f010859a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f010006ff80 con 0x7f010006f950 2024-09-19T07:35:25.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f010859a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0100160810 con 0x7f0100071e20 2024-09-19T07:35:25.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f0106b37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01001515e0 0x7f0100141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:25.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f0106b37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01001515e0 0x7f0100141a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60654/0 (socket says 172.21.15.83:60654) 2024-09-19T07:35:25.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f0106b37640 1 -- 172.21.15.83:0/4168779056 learned_addr learned my addr 172.21.15.83:0/4168779056 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:25.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f0105b35640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0100071e20 0x7f01001414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:25.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f0106336640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f010006f950 0x7f0100156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:25.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f0106b37640 1 -- 172.21.15.83:0/4168779056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0100071e20 msgr2=0x7f01001414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:25.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f0106b37640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0100071e20 0x7f01001414f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:25.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f0106b37640 1 -- 172.21.15.83:0/4168779056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f010006f950 msgr2=0x7f0100156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:25.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f0106b37640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f010006f950 0x7f0100156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:25.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f0106b37640 1 -- 172.21.15.83:0/4168779056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01000acfa0 con 0x7f01001515e0 2024-09-19T07:35:25.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f0106b37640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01001515e0 0x7f0100141a30 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f00f800efc0 tx=0x7f00f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:25.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f00f6ffd640 1 -- 172.21.15.83:0/4168779056 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f8018030 con 0x7f01001515e0 2024-09-19T07:35:25.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f00f6ffd640 1 -- 172.21.15.83:0/4168779056 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00f800cdf0 con 0x7f01001515e0 2024-09-19T07:35:25.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f010859a640 1 -- 172.21.15.83:0/4168779056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f010015d7d0 con 0x7f01001515e0 2024-09-19T07:35:25.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f00f6ffd640 1 -- 172.21.15.83:0/4168779056 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f8003bc0 con 0x7f01001515e0 2024-09-19T07:35:25.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f010859a640 1 -- 172.21.15.83:0/4168779056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0100141f70 con 0x7f01001515e0 2024-09-19T07:35:25.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f010859a640 1 -- 172.21.15.83:0/4168779056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01000716a0 con 0x7f01001515e0 2024-09-19T07:35:25.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.483+0000 7f00f6ffd640 1 -- 172.21.15.83:0/4168779056 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f00f801b070 con 0x7f01001515e0 2024-09-19T07:35:25.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.487+0000 7f00f6ffd640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00e40678c0 0x7f00e4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:25.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.487+0000 7f00f6ffd640 1 -- 172.21.15.83:0/4168779056 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7f00f8087110 con 0x7f01001515e0 2024-09-19T07:35:25.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.487+0000 7f0106336640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00e40678c0 0x7f00e4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:25.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.487+0000 7f0106336640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00e40678c0 0x7f00e4069d80 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f00fc000f70 tx=0x7f00fc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:25.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.491+0000 7f00f6ffd640 1 -- 172.21.15.83:0/4168779056 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f00f8022070 con 0x7f01001515e0 2024-09-19T07:35:25.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.739+0000 7f010859a640 1 -- 172.21.15.83:0/4168779056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f010015d9a0 con 0x7f01001515e0 2024-09-19T07:35:25.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.739+0000 7f00f6ffd640 1 -- 172.21.15.83:0/4168779056 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+6543 (secure 0 0 0) 0x7f00f8010040 con 0x7f01001515e0 2024-09-19T07:35:25.734 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:25.734 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":119,"btime":"2024-09-19T07:34:31:383334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":6918,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:21.357382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:active","state_seq":54,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5722,"qdb_cluster":[5722]},"id":2}]} 2024-09-19T07:35:25.734 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 119 2024-09-19T07:35:25.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.743+0000 7f010859a640 1 -- 172.21.15.83:0/4168779056 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00e40678c0 msgr2=0x7f00e4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:25.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.743+0000 7f010859a640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00e40678c0 0x7f00e4069d80 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f00fc000f70 tx=0x7f00fc009040 comp rx=0 tx=0).stop 2024-09-19T07:35:25.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.743+0000 7f010859a640 1 -- 172.21.15.83:0/4168779056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01001515e0 msgr2=0x7f0100141a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:25.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.743+0000 7f010859a640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01001515e0 0x7f0100141a30 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f00f800efc0 tx=0x7f00f800c490 comp rx=0 tx=0).stop 2024-09-19T07:35:25.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.743+0000 7f010859a640 1 -- 172.21.15.83:0/4168779056 shutdown_connections 2024-09-19T07:35:25.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.743+0000 7f010859a640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00e40678c0 0x7f00e4069d80 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:25.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.743+0000 7f010859a640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01001515e0 0x7f0100141a30 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:25.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.743+0000 7f010859a640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0100071e20 0x7f01001414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:25.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.743+0000 7f010859a640 1 --2- 172.21.15.83:0/4168779056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f010006f950 0x7f0100156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:25.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.743+0000 7f010859a640 1 -- 172.21.15.83:0/4168779056 >> 172.21.15.83:0/4168779056 conn(0x7f01000c48b0 msgr2=0x7f01000aa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:25.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.743+0000 7f010859a640 1 -- 172.21.15.83:0/4168779056 shutdown_connections 2024-09-19T07:35:25.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:25.743+0000 7f010859a640 1 -- 172.21.15.83:0/4168779056 wait complete. 2024-09-19T07:35:25.759 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:35:27.760 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:27.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.903+0000 7f7816ee0640 1 Processor -- start 2024-09-19T07:35:27.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.903+0000 7f7816ee0640 1 -- start start 2024-09-19T07:35:27.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f7816ee0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f781014f820 0x7f7810152b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:27.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f7816ee0640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f78100c2fb0 0x7f78100c3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:27.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f7816ee0640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f781014fcf0 con 0x7f781006d6f0 2024-09-19T07:35:27.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f7816ee0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f781006dff0 con 0x7f781014f820 2024-09-19T07:35:27.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f7816ee0640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f78100c2840 con 0x7f78100c2fb0 2024-09-19T07:35:27.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f780ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f781014f820 0x7f7810152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:27.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f780ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f781014f820 0x7f7810152b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60662/0 (socket says 172.21.15.83:60662) 2024-09-19T07:35:27.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f780ffff640 1 -- 172.21.15.83:0/956269064 learned_addr learned my addr 172.21.15.83:0/956269064 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:27.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f781547d640 1 --2- 172.21.15.83:0/956269064 >> v2:172.21.15.184:3300/0 conn(0x7f78100c2fb0 0x7f78100c3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:27.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f780ffff640 1 -- 172.21.15.83:0/956269064 >> v2:172.21.15.184:3300/0 conn(0x7f78100c2fb0 msgr2=0x7f78100c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:27.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f780ffff640 1 --2- 172.21.15.83:0/956269064 >> v2:172.21.15.184:3300/0 conn(0x7f78100c2fb0 0x7f78100c3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:27.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f780ffff640 1 -- 172.21.15.83:0/956269064 >> v1:172.21.15.184:6789/0 conn(0x7f781006d6f0 legacy=0x7f781014f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:27.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f780ffff640 1 -- 172.21.15.83:0/956269064 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f781006e240 con 0x7f781014f820 2024-09-19T07:35:27.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f781547d640 1 --2- 172.21.15.83:0/956269064 >> v2:172.21.15.184:3300/0 conn(0x7f78100c2fb0 0x7f78100c3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:35:27.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f780ffff640 1 --2- 172.21.15.83:0/956269064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f781014f820 0x7f7810152b50 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f78100c43c0 tx=0x7f77fc01e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca181d1ec216f81 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:27.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f780f7fe640 1 -- 172.21.15.83:0/956269064 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc0ad070 con 0x7f781014f820 2024-09-19T07:35:27.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f780f7fe640 1 -- 172.21.15.83:0/956269064 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77fc004230 con 0x7f781014f820 2024-09-19T07:35:27.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.907+0000 7f780f7fe640 1 -- 172.21.15.83:0/956269064 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc0a76f0 con 0x7f781014f820 2024-09-19T07:35:27.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 -- 172.21.15.83:0/956269064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f781014f820 msgr2=0x7f7810152b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:27.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 --2- 172.21.15.83:0/956269064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f781014f820 0x7f7810152b50 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f78100c43c0 tx=0x7f77fc01e990 comp rx=0 tx=0).stop 2024-09-19T07:35:27.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 -- 172.21.15.83:0/956269064 shutdown_connections 2024-09-19T07:35:27.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 --2- 172.21.15.83:0/956269064 >> v2:172.21.15.184:3300/0 conn(0x7f78100c2fb0 0x7f78100c3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:27.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 --2- 172.21.15.83:0/956269064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f781014f820 0x7f7810152b50 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:27.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 -- 172.21.15.83:0/956269064 >> 172.21.15.83:0/956269064 conn(0x7f78100c7e40 msgr2=0x7f78100c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:27.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 -- 172.21.15.83:0/956269064 shutdown_connections 2024-09-19T07:35:27.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 -- 172.21.15.83:0/956269064 wait complete. 2024-09-19T07:35:27.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 Processor -- start 2024-09-19T07:35:27.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 -- start start 2024-09-19T07:35:27.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f781006d6f0 0x7f781015ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:27.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78100c2fb0 0x7f7810141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:27.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f781014f820 0x7f7810142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:27.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f781006dad0 con 0x7f781014f820 2024-09-19T07:35:27.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f781006fef0 con 0x7f781006d6f0 2024-09-19T07:35:27.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f781006dc50 con 0x7f78100c2fb0 2024-09-19T07:35:27.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f781547d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f781014f820 0x7f7810142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:27.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f781547d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f781014f820 0x7f7810142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60666/0 (socket says 172.21.15.83:60666) 2024-09-19T07:35:27.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f781547d640 1 -- 172.21.15.83:0/3072697426 learned_addr learned my addr 172.21.15.83:0/3072697426 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:27.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f780ffff640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78100c2fb0 0x7f7810141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:27.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7814c7c640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f781006d6f0 0x7f781015ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:27.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f781547d640 1 -- 172.21.15.83:0/3072697426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78100c2fb0 msgr2=0x7f7810141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:27.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f781547d640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78100c2fb0 0x7f7810141cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:27.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f781547d640 1 -- 172.21.15.83:0/3072697426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f781006d6f0 msgr2=0x7f781015ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:27.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f781547d640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f781006d6f0 0x7f781015ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:27.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f781547d640 1 -- 172.21.15.83:0/3072697426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78101f2a30 con 0x7f781014f820 2024-09-19T07:35:27.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7814c7c640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f781006d6f0 0x7f781015ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:35:27.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f781547d640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f781014f820 0x7f7810142210 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f780800cf20 tx=0x7f780800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:27.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f780d7fa640 1 -- 172.21.15.83:0/3072697426 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780801a070 con 0x7f781014f820 2024-09-19T07:35:27.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f780d7fa640 1 -- 172.21.15.83:0/3072697426 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f780800d840 con 0x7f781014f820 2024-09-19T07:35:27.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 -- 172.21.15.83:0/3072697426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78100c64c0 con 0x7f781014f820 2024-09-19T07:35:27.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f780d7fa640 1 -- 172.21.15.83:0/3072697426 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780800dc30 con 0x7f781014f820 2024-09-19T07:35:27.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.911+0000 7f7816ee0640 1 -- 172.21.15.83:0/3072697426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78100c7830 con 0x7f781014f820 2024-09-19T07:35:27.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.915+0000 7f7816ee0640 1 -- 172.21.15.83:0/3072697426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78100c3420 con 0x7f781014f820 2024-09-19T07:35:27.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.915+0000 7f780d7fa640 1 -- 172.21.15.83:0/3072697426 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f780801d070 con 0x7f781014f820 2024-09-19T07:35:27.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.915+0000 7f780d7fa640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77f40678c0 0x7f77f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:27.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.915+0000 7f780d7fa640 1 -- 172.21.15.83:0/3072697426 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8529+0+0 (secure 0 0 0) 0x7f7808087090 con 0x7f781014f820 2024-09-19T07:35:27.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.915+0000 7f7814c7c640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77f40678c0 0x7f77f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:27.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.915+0000 7f7814c7c640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77f40678c0 0x7f77f4069d80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f78101f2c90 tx=0x7f7804009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:27.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:27.919+0000 7f780d7fa640 1 -- 172.21.15.83:0/3072697426 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f78080570a0 con 0x7f781014f820 2024-09-19T07:35:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.131+0000 7f7816ee0640 1 -- 172.21.15.83:0/3072697426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78100c69e0 con 0x7f781014f820 2024-09-19T07:35:28.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.131+0000 7f780d7fa640 1 -- 172.21.15.83:0/3072697426 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+6543 (secure 0 0 0) 0x7f78080569a0 con 0x7f781014f820 2024-09-19T07:35:28.126 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:28.126 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":119,"btime":"2024-09-19T07:34:31:383334+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":6918,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:34:21.357382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5722":{"gid":5722,"name":"b","rank":0,"incarnation":115,"state":"up:active","state_seq":54,"addr":"172.21.15.184:6835/2112478738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":2112478738},{"type":"v1","addr":"172.21.15.184:6835","nonce":2112478738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5722,"qdb_cluster":[5722]},"id":2}]} 2024-09-19T07:35:28.126 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 119 2024-09-19T07:35:28.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.135+0000 7f7816ee0640 1 -- 172.21.15.83:0/3072697426 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77f40678c0 msgr2=0x7f77f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:28.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.135+0000 7f7816ee0640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77f40678c0 0x7f77f4069d80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f78101f2c90 tx=0x7f7804009040 comp rx=0 tx=0).stop 2024-09-19T07:35:28.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.135+0000 7f7816ee0640 1 -- 172.21.15.83:0/3072697426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f781014f820 msgr2=0x7f7810142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:28.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.135+0000 7f7816ee0640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f781014f820 0x7f7810142210 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f780800cf20 tx=0x7f780800a7d0 comp rx=0 tx=0).stop 2024-09-19T07:35:28.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.135+0000 7f7816ee0640 1 -- 172.21.15.83:0/3072697426 shutdown_connections 2024-09-19T07:35:28.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.135+0000 7f7816ee0640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77f40678c0 0x7f77f4069d80 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:28.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.135+0000 7f7816ee0640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f781014f820 0x7f7810142210 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:28.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.135+0000 7f7816ee0640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78100c2fb0 0x7f7810141cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:28.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.135+0000 7f7816ee0640 1 --2- 172.21.15.83:0/3072697426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f781006d6f0 0x7f781015ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:28.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.135+0000 7f7816ee0640 1 -- 172.21.15.83:0/3072697426 >> 172.21.15.83:0/3072697426 conn(0x7f78100c7e40 msgr2=0x7f781014f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:28.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.135+0000 7f7816ee0640 1 -- 172.21.15.83:0/3072697426 shutdown_connections 2024-09-19T07:35:28.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:28.135+0000 7f7816ee0640 1 -- 172.21.15.83:0/3072697426 wait complete. 2024-09-19T07:35:28.153 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:35:29.478 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:29.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.615+0000 7f0e85598640 1 Processor -- start 2024-09-19T07:35:29.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.615+0000 7f0e85598640 1 -- start start 2024-09-19T07:35:29.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e85598640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f0e800c3130 0x7f0e800ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:29.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e85598640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e8006de70 0x7f0e801563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:29.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e85598640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e800c3c20 con 0x7f0e8006da00 2024-09-19T07:35:29.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e85598640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f0e800c4990 con 0x7f0e8006de70 2024-09-19T07:35:29.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e85598640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f0e800c4c70 con 0x7f0e800c3130 2024-09-19T07:35:29.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e7f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e8006de70 0x7f0e801563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:29.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e77fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0e8006da00 0x7f0e800c3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60658/0 (socket says 172.21.15.83:60658) 2024-09-19T07:35:29.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e7f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e8006de70 0x7f0e801563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60676/0 (socket says 172.21.15.83:60676) 2024-09-19T07:35:29.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e77fff640 1 -- 172.21.15.83:0/4281463500 learned_addr learned my addr 172.21.15.83:0/4281463500 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:29.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e7f7fe640 1 -- 172.21.15.83:0/4281463500 >> v2:172.21.15.184:3300/0 conn(0x7f0e800c3130 msgr2=0x7f0e800ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:29.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e7f7fe640 1 --2- 172.21.15.83:0/4281463500 >> v2:172.21.15.184:3300/0 conn(0x7f0e800c3130 0x7f0e800ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:29.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e7f7fe640 1 -- 172.21.15.83:0/4281463500 >> v1:172.21.15.83:6789/0 conn(0x7f0e8006da00 legacy=0x7f0e800c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:29.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e7e7fc640 1 -- 172.21.15.83:0/4281463500 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1565326817 0 0) 0x7f0e800c3c20 con 0x7f0e8006da00 2024-09-19T07:35:29.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e7f7fe640 1 -- 172.21.15.83:0/4281463500 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e800ab7b0 con 0x7f0e8006de70 2024-09-19T07:35:29.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e7f7fe640 1 --2- 172.21.15.83:0/4281463500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e8006de70 0x7f0e801563a0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f0e800c5eb0 tx=0x7f0e7801e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4ec938d5bf0533 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:29.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e7e7fc640 1 -- 172.21.15.83:0/4281463500 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7800b2d0 con 0x7f0e8006de70 2024-09-19T07:35:29.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e7e7fc640 1 -- 172.21.15.83:0/4281463500 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e780a75a0 con 0x7f0e8006de70 2024-09-19T07:35:29.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e7e7fc640 1 -- 172.21.15.83:0/4281463500 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e780a7970 con 0x7f0e8006de70 2024-09-19T07:35:29.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e85598640 1 -- 172.21.15.83:0/4281463500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e8006de70 msgr2=0x7f0e801563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:29.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e85598640 1 --2- 172.21.15.83:0/4281463500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e8006de70 0x7f0e801563a0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f0e800c5eb0 tx=0x7f0e7801e7e0 comp rx=0 tx=0).stop 2024-09-19T07:35:29.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e85598640 1 -- 172.21.15.83:0/4281463500 shutdown_connections 2024-09-19T07:35:29.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e85598640 1 --2- 172.21.15.83:0/4281463500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e8006de70 0x7f0e801563a0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:29.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e85598640 1 --2- 172.21.15.83:0/4281463500 >> v2:172.21.15.184:3300/0 conn(0x7f0e800c3130 0x7f0e800ac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:29.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.619+0000 7f0e85598640 1 -- 172.21.15.83:0/4281463500 >> 172.21.15.83:0/4281463500 conn(0x7f0e800c5060 msgr2=0x7f0e800c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:29.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e85598640 1 -- 172.21.15.83:0/4281463500 shutdown_connections 2024-09-19T07:35:29.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e85598640 1 -- 172.21.15.83:0/4281463500 wait complete. 2024-09-19T07:35:29.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e85598640 1 Processor -- start 2024-09-19T07:35:29.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e85598640 1 -- start start 2024-09-19T07:35:29.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e85598640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e8006da00 0x7f0e80158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:29.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e85598640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e8006de70 0x7f0e8015a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:29.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e85598640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e800c3130 0x7f0e8015a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e85598640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0e800ad6a0 con 0x7f0e800c3130 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e85598640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0e800ad520 con 0x7f0e8006da00 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e85598640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0e80074f90 con 0x7f0e8006de70 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e7f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e800c3130 0x7f0e8015a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e7f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e800c3130 0x7f0e8015a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60690/0 (socket says 172.21.15.83:60690) 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e7f7fe640 1 -- 172.21.15.83:0/631552372 learned_addr learned my addr 172.21.15.83:0/631552372 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e7effd640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e8006da00 0x7f0e80158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e7f7fe640 1 -- 172.21.15.83:0/631552372 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e8006de70 msgr2=0x7f0e8015a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e7f7fe640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e8006de70 0x7f0e8015a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e7f7fe640 1 -- 172.21.15.83:0/631552372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e8006da00 msgr2=0x7f0e80158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e7f7fe640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e8006da00 0x7f0e80158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e7f7fe640 1 -- 172.21.15.83:0/631552372 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e80157af0 con 0x7f0e800c3130 2024-09-19T07:35:29.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e7effd640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e8006da00 0x7f0e80158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:35:29.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e7f7fe640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e800c3130 0x7f0e8015a720 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0e7801b3f0 tx=0x7f0e7801b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:29.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e777fe640 1 -- 172.21.15.83:0/631552372 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e780be020 con 0x7f0e800c3130 2024-09-19T07:35:29.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e777fe640 1 -- 172.21.15.83:0/631552372 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e78009540 con 0x7f0e800c3130 2024-09-19T07:35:29.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e777fe640 1 -- 172.21.15.83:0/631552372 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e78009910 con 0x7f0e800c3130 2024-09-19T07:35:29.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e85598640 1 -- 172.21.15.83:0/631552372 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e80157810 con 0x7f0e800c3130 2024-09-19T07:35:29.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.623+0000 7f0e85598640 1 -- 172.21.15.83:0/631552372 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e801ea0d0 con 0x7f0e800c3130 2024-09-19T07:35:29.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.627+0000 7f0e777fe640 1 -- 172.21.15.83:0/631552372 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0e780c1020 con 0x7f0e800c3130 2024-09-19T07:35:29.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.627+0000 7f0e85598640 1 -- 172.21.15.83:0/631552372 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e801563a0 con 0x7f0e800c3130 2024-09-19T07:35:29.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.627+0000 7f0e777fe640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0e600678c0 0x7f0e60069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:29.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.627+0000 7f0e777fe640 1 -- 172.21.15.83:0/631552372 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8615+0+0 (secure 0 0 0) 0x7f0e78123980 con 0x7f0e800c3130 2024-09-19T07:35:29.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.627+0000 7f0e7effd640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0e600678c0 0x7f0e60069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:29.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.627+0000 7f0e7effd640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0e600678c0 0x7f0e60069d80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000a870 tx=0x7f0e70008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:29.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.631+0000 7f0e777fe640 1 -- 172.21.15.83:0/631552372 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f0e780c8030 con 0x7f0e800c3130 2024-09-19T07:35:29.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.899+0000 7f0e85598640 1 -- 172.21.15.83:0/631552372 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e800c5830 con 0x7f0e800c3130 2024-09-19T07:35:29.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e777fe640 1 -- 172.21.15.83:0/631552372 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+5710 (secure 0 0 0) 0x7f0e78018720 con 0x7f0e800c3130 2024-09-19T07:35:29.895 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:29.895 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":121,"btime":"2024-09-19T07:35:29:297667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:28.555869+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:35:29.896 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 121 2024-09-19T07:35:29.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e85598640 1 -- 172.21.15.83:0/631552372 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0e600678c0 msgr2=0x7f0e60069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:29.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e85598640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0e600678c0 0x7f0e60069d80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000a870 tx=0x7f0e70008040 comp rx=0 tx=0).stop 2024-09-19T07:35:29.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e85598640 1 -- 172.21.15.83:0/631552372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e800c3130 msgr2=0x7f0e8015a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:29.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e85598640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e800c3130 0x7f0e8015a720 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0e7801b3f0 tx=0x7f0e7801b580 comp rx=0 tx=0).stop 2024-09-19T07:35:29.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e85598640 1 -- 172.21.15.83:0/631552372 shutdown_connections 2024-09-19T07:35:29.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e85598640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0e600678c0 0x7f0e60069d80 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:29.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e85598640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e800c3130 0x7f0e8015a720 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:29.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e85598640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e8006de70 0x7f0e8015a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:29.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e85598640 1 --2- 172.21.15.83:0/631552372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e8006da00 0x7f0e80158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:29.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e85598640 1 -- 172.21.15.83:0/631552372 >> 172.21.15.83:0/631552372 conn(0x7f0e800c5060 msgr2=0x7f0e800c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:29.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e85598640 1 -- 172.21.15.83:0/631552372 shutdown_connections 2024-09-19T07:35:29.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:29.903+0000 7f0e85598640 1 -- 172.21.15.83:0/631552372 wait complete. 2024-09-19T07:35:29.915 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-19T07:35:29.915 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:35:29.915 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:35:29.927 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:35:29.927 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:30.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.095+0000 7f0968291640 1 Processor -- start 2024-09-19T07:35:30.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.095+0000 7f0968291640 1 -- start start 2024-09-19T07:35:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.095+0000 7f0968291640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f09600adae0 0x7f096006f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.095+0000 7f0968291640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f096006fcc0 0x7f09601539a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.095+0000 7f0968291640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09600c3aa0 con 0x7f0960150f40 2024-09-19T07:35:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.095+0000 7f0968291640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f09600c3850 con 0x7f09600adae0 2024-09-19T07:35:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.095+0000 7f0968291640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f09600c43d0 con 0x7f096006fcc0 2024-09-19T07:35:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.095+0000 7f096582c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f09600adae0 0x7f096006f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.095+0000 7f096582c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f09600adae0 0x7f096006f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60692/0 (socket says 172.21.15.83:60692) 2024-09-19T07:35:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.095+0000 7f096582c640 1 -- 172.21.15.83:0/3766627195 learned_addr learned my addr 172.21.15.83:0/3766627195 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.095+0000 7f096682e640 1 --2- 172.21.15.83:0/3766627195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f096006fcc0 0x7f09601539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f096582c640 1 -- 172.21.15.83:0/3766627195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f096006fcc0 msgr2=0x7f09601539a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f096582c640 1 --2- 172.21.15.83:0/3766627195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f096006fcc0 0x7f09601539a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f096582c640 1 -- 172.21.15.83:0/3766627195 >> v1:172.21.15.184:6789/0 conn(0x7f0960150f40 legacy=0x7f09600733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f096582c640 1 -- 172.21.15.83:0/3766627195 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09600c3d80 con 0x7f09600adae0 2024-09-19T07:35:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f096682e640 1 --2- 172.21.15.83:0/3766627195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f096006fcc0 0x7f09601539a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:35:30.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f096582c640 1 --2- 172.21.15.83:0/3766627195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09600adae0 0x7f096006f780 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0950009870 tx=0x7f095001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d05629ae50493b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:30.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f096502b640 1 -- 172.21.15.83:0/3766627195 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09500b0070 con 0x7f09600adae0 2024-09-19T07:35:30.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f096502b640 1 -- 172.21.15.83:0/3766627195 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09500ab600 con 0x7f09600adae0 2024-09-19T07:35:30.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f096502b640 1 -- 172.21.15.83:0/3766627195 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09500ab9d0 con 0x7f09600adae0 2024-09-19T07:35:30.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f0968291640 1 -- 172.21.15.83:0/3766627195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09600adae0 msgr2=0x7f096006f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f0968291640 1 --2- 172.21.15.83:0/3766627195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09600adae0 0x7f096006f780 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0950009870 tx=0x7f095001c520 comp rx=0 tx=0).stop 2024-09-19T07:35:30.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f0968291640 1 -- 172.21.15.83:0/3766627195 shutdown_connections 2024-09-19T07:35:30.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f0968291640 1 --2- 172.21.15.83:0/3766627195 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f096006fcc0 0x7f09601539a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f0968291640 1 --2- 172.21.15.83:0/3766627195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09600adae0 0x7f096006f780 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f0968291640 1 -- 172.21.15.83:0/3766627195 >> 172.21.15.83:0/3766627195 conn(0x7f09600c4940 msgr2=0x7f09600c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:30.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f0968291640 1 -- 172.21.15.83:0/3766627195 shutdown_connections 2024-09-19T07:35:30.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.099+0000 7f0968291640 1 -- 172.21.15.83:0/3766627195 wait complete. 2024-09-19T07:35:30.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0968291640 1 Processor -- start 2024-09-19T07:35:30.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0968291640 1 -- start start 2024-09-19T07:35:30.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0968291640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096006fcc0 0x7f09600aa700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0968291640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09600adae0 0x7f096014a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0968291640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0960150f40 0x7f096014ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0968291640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f09601606c0 con 0x7f096006fcc0 2024-09-19T07:35:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0968291640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0960160540 con 0x7f09600adae0 2024-09-19T07:35:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0968291640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0960160840 con 0x7f0960150f40 2024-09-19T07:35:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f096602d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096006fcc0 0x7f09600aa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f096602d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096006fcc0 0x7f09600aa700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60694/0 (socket says 172.21.15.83:60694) 2024-09-19T07:35:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f096602d640 1 -- 172.21.15.83:0/4246245171 learned_addr learned my addr 172.21.15.83:0/4246245171 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f096582c640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09600adae0 0x7f096014a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f096602d640 1 -- 172.21.15.83:0/4246245171 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0960150f40 msgr2=0x7f096014ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f096602d640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0960150f40 0x7f096014ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f096602d640 1 -- 172.21.15.83:0/4246245171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09600adae0 msgr2=0x7f096014a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f096602d640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09600adae0 0x7f096014a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f096602d640 1 -- 172.21.15.83:0/4246245171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0960156710 con 0x7f096006fcc0 2024-09-19T07:35:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f096602d640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096006fcc0 0x7f09600aa700 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f095c00b410 tx=0x7f095c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0956ffd640 1 -- 172.21.15.83:0/4246245171 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c015030 con 0x7f096006fcc0 2024-09-19T07:35:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0956ffd640 1 -- 172.21.15.83:0/4246245171 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f095c004020 con 0x7f096006fcc0 2024-09-19T07:35:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0968291640 1 -- 172.21.15.83:0/4246245171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09601569f0 con 0x7f096006fcc0 2024-09-19T07:35:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0956ffd640 1 -- 172.21.15.83:0/4246245171 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c004410 con 0x7f096006fcc0 2024-09-19T07:35:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0968291640 1 -- 172.21.15.83:0/4246245171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0960156fb0 con 0x7f096006fcc0 2024-09-19T07:35:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.103+0000 7f0968291640 1 -- 172.21.15.83:0/4246245171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09600c50f0 con 0x7f096006fcc0 2024-09-19T07:35:30.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.107+0000 7f0956ffd640 1 -- 172.21.15.83:0/4246245171 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f095c018070 con 0x7f096006fcc0 2024-09-19T07:35:30.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.107+0000 7f0956ffd640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09440678c0 0x7f0944069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.107+0000 7f0956ffd640 1 -- 172.21.15.83:0/4246245171 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8615+0+0 (secure 0 0 0) 0x7f095c0863a0 con 0x7f096006fcc0 2024-09-19T07:35:30.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.107+0000 7f096582c640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09440678c0 0x7f0944069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.107+0000 7f096582c640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09440678c0 0x7f0944069d80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f09500ad000 tx=0x7f0950008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:30.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.111+0000 7f0956ffd640 1 -- 172.21.15.83:0/4246245171 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f095c0563c0 con 0x7f096006fcc0 2024-09-19T07:35:30.153 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:30.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.283+0000 7f44f4a73640 1 Processor -- start 2024-09-19T07:35:30.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.283+0000 7f44f4a73640 1 -- start start 2024-09-19T07:35:30.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.283+0000 7f44f4a73640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44f00c6360 0x7f44f0153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.283+0000 7f44f4a73640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f44f00bee30 0x7f44f01570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.283+0000 7f44f4a73640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f00c5b90 con 0x7f44f00c9680 2024-09-19T07:35:30.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.283+0000 7f44f4a73640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f44f015a050 con 0x7f44f00bee30 2024-09-19T07:35:30.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.283+0000 7f44f4a73640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f44f006c950 con 0x7f44f00c6360 2024-09-19T07:35:30.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44edd9b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f44f00bee30 0x7f44f01570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44ee59c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44f00c6360 0x7f44f0153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44ee59c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44f00c6360 0x7f44f0153050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:39708/0 (socket says 172.21.15.83:39708) 2024-09-19T07:35:30.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44edd9b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f44f00bee30 0x7f44f01570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:52126/0 (socket says 172.21.15.83:52126) 2024-09-19T07:35:30.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44ee59c640 1 -- 172.21.15.83:0/4035985315 learned_addr learned my addr 172.21.15.83:0/4035985315 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:30.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44eed9d640 1 --1- 172.21.15.83:0/4035985315 >> v1:172.21.15.184:6789/0 conn(0x7f44f00c9680 0x7f44f0157630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:46642/0 (socket says 172.21.15.83:46642) 2024-09-19T07:35:30.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44edd9b640 1 -- 172.21.15.83:0/4035985315 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44f00c6360 msgr2=0x7f44f0153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:30.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44edd9b640 1 --2- 172.21.15.83:0/4035985315 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44f00c6360 0x7f44f0153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44edd9b640 1 -- 172.21.15.83:0/4035985315 >> v1:172.21.15.184:6789/0 conn(0x7f44f00c9680 legacy=0x7f44f0157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44edd9b640 1 -- 172.21.15.83:0/4035985315 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f006c670 con 0x7f44f00bee30 2024-09-19T07:35:30.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44edd9b640 1 --2- 172.21.15.83:0/4035985315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44f00bee30 0x7f44f01570f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f44e40097c0 tx=0x7f44e401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f42b05bc987c8f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:30.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44ed59a640 1 -- 172.21.15.83:0/4035985315 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e40b0070 con 0x7f44f00bee30 2024-09-19T07:35:30.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44ed59a640 1 -- 172.21.15.83:0/4035985315 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e40ab600 con 0x7f44f00bee30 2024-09-19T07:35:30.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.287+0000 7f44ed59a640 1 -- 172.21.15.83:0/4035985315 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e40ab9d0 con 0x7f44f00bee30 2024-09-19T07:35:30.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 -- 172.21.15.83:0/4035985315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44f00bee30 msgr2=0x7f44f01570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 --2- 172.21.15.83:0/4035985315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44f00bee30 0x7f44f01570f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f44e40097c0 tx=0x7f44e401c410 comp rx=0 tx=0).stop 2024-09-19T07:35:30.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 -- 172.21.15.83:0/4035985315 shutdown_connections 2024-09-19T07:35:30.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 --2- 172.21.15.83:0/4035985315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44f00bee30 0x7f44f01570f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 --2- 172.21.15.83:0/4035985315 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44f00c6360 0x7f44f0153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 -- 172.21.15.83:0/4035985315 >> 172.21.15.83:0/4035985315 conn(0x7f44f00722a0 msgr2=0x7f44f00726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:30.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 -- 172.21.15.83:0/4035985315 shutdown_connections 2024-09-19T07:35:30.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 -- 172.21.15.83:0/4035985315 wait complete. 2024-09-19T07:35:30.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 Processor -- start 2024-09-19T07:35:30.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 -- start start 2024-09-19T07:35:30.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44f00bee30 0x7f44f01f2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44f00c6360 0x7f44f014f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44f00c9680 0x7f44f014fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f44f00c68c0 con 0x7f44f00bee30 2024-09-19T07:35:30.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f44f00c6740 con 0x7f44f00c9680 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f44f0162da0 con 0x7f44f00c6360 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44ee59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44f00bee30 0x7f44f01f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44ee59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44f00bee30 0x7f44f01f2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33348/0 (socket says 172.21.15.83:33348) 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44ee59c640 1 -- 172.21.15.83:0/3820938025 learned_addr learned my addr 172.21.15.83:0/3820938025 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44edd9b640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44f00c6360 0x7f44f014f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44eed9d640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44f00c9680 0x7f44f014fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44ee59c640 1 -- 172.21.15.83:0/3820938025 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44f00c6360 msgr2=0x7f44f014f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44ee59c640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44f00c6360 0x7f44f014f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44ee59c640 1 -- 172.21.15.83:0/3820938025 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44f00c9680 msgr2=0x7f44f014fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44ee59c640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44f00c9680 0x7f44f014fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44ee59c640 1 -- 172.21.15.83:0/3820938025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f0154fd0 con 0x7f44f00bee30 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44eed9d640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44f00c9680 0x7f44f014fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:35:30.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44ee59c640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44f00bee30 0x7f44f01f2110 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f44e000e7f0 tx=0x7f44e000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:30.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44deffd640 1 -- 172.21.15.83:0/3820938025 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0017070 con 0x7f44f00bee30 2024-09-19T07:35:30.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44deffd640 1 -- 172.21.15.83:0/3820938025 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e000c940 con 0x7f44f00bee30 2024-09-19T07:35:30.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44deffd640 1 -- 172.21.15.83:0/3820938025 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e000cd30 con 0x7f44f00bee30 2024-09-19T07:35:30.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 -- 172.21.15.83:0/3820938025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f0153e10 con 0x7f44f00bee30 2024-09-19T07:35:30.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.291+0000 7f44f4a73640 1 -- 172.21.15.83:0/3820938025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f0150270 con 0x7f44f00bee30 2024-09-19T07:35:30.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.295+0000 7f44f4a73640 1 -- 172.21.15.83:0/3820938025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f0153050 con 0x7f44f00bee30 2024-09-19T07:35:30.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.295+0000 7f44deffd640 1 -- 172.21.15.83:0/3820938025 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f44e001a070 con 0x7f44f00bee30 2024-09-19T07:35:30.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.295+0000 7f44deffd640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44d00678c0 0x7f44d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.295+0000 7f44deffd640 1 -- 172.21.15.83:0/3820938025 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8615+0+0 (secure 0 0 0) 0x7f44e0086770 con 0x7f44f00bee30 2024-09-19T07:35:30.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.295+0000 7f44edd9b640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44d00678c0 0x7f44d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.295+0000 7f44edd9b640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44d00678c0 0x7f44d0069d80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f44e40ad000 tx=0x7f44e4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:30.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.303+0000 7f44deffd640 1 -- 172.21.15.83:0/3820938025 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f44e0056680 con 0x7f44f00bee30 2024-09-19T07:35:30.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.375+0000 7f0968291640 1 -- 172.21.15.83:0/4246245171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0960153960 con 0x7f096006fcc0 2024-09-19T07:35:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.375+0000 7f0956ffd640 1 -- 172.21.15.83:0/4246245171 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5707 (secure 0 0 0) 0x7f095c055cc0 con 0x7f096006fcc0 2024-09-19T07:35:30.370 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:30.370 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":122,"btime":"2024-09-19T07:35:30:306576+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:29.309817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:35:30.371 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 122 2024-09-19T07:35:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.379+0000 7f0968291640 1 -- 172.21.15.83:0/4246245171 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09440678c0 msgr2=0x7f0944069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.379+0000 7f0968291640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09440678c0 0x7f0944069d80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f09500ad000 tx=0x7f0950008040 comp rx=0 tx=0).stop 2024-09-19T07:35:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.379+0000 7f0968291640 1 -- 172.21.15.83:0/4246245171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096006fcc0 msgr2=0x7f09600aa700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.379+0000 7f0968291640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096006fcc0 0x7f09600aa700 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f095c00b410 tx=0x7f095c00b8d0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.379+0000 7f0968291640 1 -- 172.21.15.83:0/4246245171 shutdown_connections 2024-09-19T07:35:30.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.379+0000 7f0968291640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09440678c0 0x7f0944069d80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.379+0000 7f0968291640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0960150f40 0x7f096014ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.379+0000 7f0968291640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09600adae0 0x7f096014a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.379+0000 7f0968291640 1 --2- 172.21.15.83:0/4246245171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096006fcc0 0x7f09600aa700 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.379+0000 7f0968291640 1 -- 172.21.15.83:0/4246245171 >> 172.21.15.83:0/4246245171 conn(0x7f09600c4940 msgr2=0x7f0960150230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:30.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.379+0000 7f0968291640 1 -- 172.21.15.83:0/4246245171 shutdown_connections 2024-09-19T07:35:30.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.379+0000 7f0968291640 1 -- 172.21.15.83:0/4246245171 wait complete. 2024-09-19T07:35:30.392 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:35:30.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.539+0000 7f44f4a73640 1 -- 172.21.15.83:0/3820938025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44f0158d20 con 0x7f44f00bee30 2024-09-19T07:35:30.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.539+0000 7f44deffd640 1 -- 172.21.15.83:0/3820938025 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5707 (secure 0 0 0) 0x7f44e0055f80 con 0x7f44f00bee30 2024-09-19T07:35:30.535 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:30.535 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":122,"btime":"2024-09-19T07:35:30:306576+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:29.309817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:35:30.535 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 122 2024-09-19T07:35:30.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.543+0000 7f44f4a73640 1 -- 172.21.15.83:0/3820938025 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44d00678c0 msgr2=0x7f44d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.543+0000 7f44f4a73640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44d00678c0 0x7f44d0069d80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f44e40ad000 tx=0x7f44e4008040 comp rx=0 tx=0).stop 2024-09-19T07:35:30.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.543+0000 7f44f4a73640 1 -- 172.21.15.83:0/3820938025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44f00bee30 msgr2=0x7f44f01f2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.543+0000 7f44f4a73640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44f00bee30 0x7f44f01f2110 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f44e000e7f0 tx=0x7f44e000ecb0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.543+0000 7f44f4a73640 1 -- 172.21.15.83:0/3820938025 shutdown_connections 2024-09-19T07:35:30.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.543+0000 7f44f4a73640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44d00678c0 0x7f44d0069d80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.543+0000 7f44f4a73640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44f00c9680 0x7f44f014fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.543+0000 7f44f4a73640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44f00c6360 0x7f44f014f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.543+0000 7f44f4a73640 1 --2- 172.21.15.83:0/3820938025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44f00bee30 0x7f44f01f2110 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.543+0000 7f44f4a73640 1 -- 172.21.15.83:0/3820938025 >> 172.21.15.83:0/3820938025 conn(0x7f44f00722a0 msgr2=0x7f44f00ac250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:30.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.543+0000 7f44f4a73640 1 -- 172.21.15.83:0/3820938025 shutdown_connections 2024-09-19T07:35:30.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.543+0000 7f44f4a73640 1 -- 172.21.15.83:0/3820938025 wait complete. 2024-09-19T07:35:30.554 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-09-19T07:35:30.554 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:35:30.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:30.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd04133f640 1 Processor -- start 2024-09-19T07:35:30.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd04133f640 1 -- start start 2024-09-19T07:35:30.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd04133f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd03c0c6120 0x7fd03c0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd04133f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c1592f0 0x7fd03c157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd04133f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03c156200 con 0x7fd03c155d70 2024-09-19T07:35:30.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd04133f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd03c0c9930 con 0x7fd03c0c6120 2024-09-19T07:35:30.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd04133f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd03c1564e0 con 0x7fd03c1592f0 2024-09-19T07:35:30.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd03affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd03c0c6120 0x7fd03c0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd03a7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd03c155d70 0x7fd03c1575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36038/0 (socket says 172.21.15.83:36038) 2024-09-19T07:35:30.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd03a7fc640 1 -- 172.21.15.83:0/87502410 learned_addr learned my addr 172.21.15.83:0/87502410 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:30.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd03b7fe640 1 --2- 172.21.15.83:0/87502410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c1592f0 0x7fd03c157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd03affd640 1 -- 172.21.15.83:0/87502410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c1592f0 msgr2=0x7fd03c157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd03affd640 1 --2- 172.21.15.83:0/87502410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c1592f0 0x7fd03c157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd03affd640 1 -- 172.21.15.83:0/87502410 >> v1:172.21.15.83:6789/0 conn(0x7fd03c155d70 legacy=0x7fd03c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd039ffb640 1 -- 172.21.15.83:0/87502410 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2557795587 0 0) 0x7fd03c156200 con 0x7fd03c155d70 2024-09-19T07:35:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.675+0000 7fd03affd640 1 -- 172.21.15.83:0/87502410 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03c1567c0 con 0x7fd03c0c6120 2024-09-19T07:35:30.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.679+0000 7fd03affd640 1 --2- 172.21.15.83:0/87502410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd03c0c6120 0x7fd03c0c6500 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0097c0 tx=0x7fd02c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7605f4ef2a1430e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:30.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.679+0000 7fd039ffb640 1 -- 172.21.15.83:0/87502410 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c0b0070 con 0x7fd03c0c6120 2024-09-19T07:35:30.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.679+0000 7fd039ffb640 1 -- 172.21.15.83:0/87502410 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd02c0ab600 con 0x7fd03c0c6120 2024-09-19T07:35:30.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.679+0000 7fd039ffb640 1 -- 172.21.15.83:0/87502410 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c0ab9d0 con 0x7fd03c0c6120 2024-09-19T07:35:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.679+0000 7fd04133f640 1 -- 172.21.15.83:0/87502410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd03c0c6120 msgr2=0x7fd03c0c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.679+0000 7fd04133f640 1 --2- 172.21.15.83:0/87502410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd03c0c6120 0x7fd03c0c6500 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0097c0 tx=0x7fd02c01c4a0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.679+0000 7fd04133f640 1 -- 172.21.15.83:0/87502410 shutdown_connections 2024-09-19T07:35:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.679+0000 7fd04133f640 1 --2- 172.21.15.83:0/87502410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c1592f0 0x7fd03c157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.679+0000 7fd04133f640 1 --2- 172.21.15.83:0/87502410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd03c0c6120 0x7fd03c0c6500 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.679+0000 7fd04133f640 1 -- 172.21.15.83:0/87502410 >> 172.21.15.83:0/87502410 conn(0x7fd03c06bc80 msgr2=0x7fd03c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.679+0000 7fd04133f640 1 -- 172.21.15.83:0/87502410 shutdown_connections 2024-09-19T07:35:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.679+0000 7fd04133f640 1 -- 172.21.15.83:0/87502410 wait complete. 2024-09-19T07:35:30.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd04133f640 1 Processor -- start 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd04133f640 1 -- start start 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd04133f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c0c6120 0x7fd03c0ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd04133f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c1592f0 0x7fd03c1ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd04133f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd03c0adac0 0x7fd03c1eeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd04133f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c153860 con 0x7fd03c0adac0 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd04133f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c162ea0 con 0x7fd03c0c6120 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd04133f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd03c0ab340 con 0x7fd03c1592f0 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd03b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd03c0adac0 0x7fd03c1eeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd03b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd03c0adac0 0x7fd03c1eeee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33364/0 (socket says 172.21.15.83:33364) 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd03b7fe640 1 -- 172.21.15.83:0/1524369224 learned_addr learned my addr 172.21.15.83:0/1524369224 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd03a7fc640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c1592f0 0x7fd03c1ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd03affd640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c0c6120 0x7fd03c0ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd03b7fe640 1 -- 172.21.15.83:0/1524369224 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c1592f0 msgr2=0x7fd03c1ee9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd03b7fe640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c1592f0 0x7fd03c1ee9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd03b7fe640 1 -- 172.21.15.83:0/1524369224 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c0c6120 msgr2=0x7fd03c0ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd03b7fe640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c0c6120 0x7fd03c0ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd03b7fe640 1 -- 172.21.15.83:0/1524369224 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03c0abb50 con 0x7fd03c0adac0 2024-09-19T07:35:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd03affd640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c0c6120 0x7fd03c0ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:35:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd03b7fe640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd03c0adac0 0x7fd03c1eeee0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd03400cc30 tx=0x7fd03400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd0337fe640 1 -- 172.21.15.83:0/1524369224 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03400ce40 con 0x7fd03c0adac0 2024-09-19T07:35:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd0337fe640 1 -- 172.21.15.83:0/1524369224 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0340039a0 con 0x7fd03c0adac0 2024-09-19T07:35:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd04133f640 1 -- 172.21.15.83:0/1524369224 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd03c1ee4e0 con 0x7fd03c0adac0 2024-09-19T07:35:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd0337fe640 1 -- 172.21.15.83:0/1524369224 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd034003d50 con 0x7fd03c0adac0 2024-09-19T07:35:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd04133f640 1 -- 172.21.15.83:0/1524369224 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd03c0cac20 con 0x7fd03c0adac0 2024-09-19T07:35:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd04133f640 1 -- 172.21.15.83:0/1524369224 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd03c0c6500 con 0x7fd03c0adac0 2024-09-19T07:35:30.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd0337fe640 1 -- 172.21.15.83:0/1524369224 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd034003f70 con 0x7fd03c0adac0 2024-09-19T07:35:30.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.683+0000 7fd0337fe640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd01c0678c0 0x7fd01c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:30.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.687+0000 7fd03affd640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd01c0678c0 0x7fd01c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:30.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.687+0000 7fd0337fe640 1 -- 172.21.15.83:0/1524369224 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8615+0+0 (secure 0 0 0) 0x7fd034086c50 con 0x7fd03c0adac0 2024-09-19T07:35:30.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.687+0000 7fd03affd640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd01c0678c0 0x7fd01c069d80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0ad000 tx=0x7fd02c0a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:30.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.691+0000 7fd0337fe640 1 -- 172.21.15.83:0/1524369224 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd03408c020 con 0x7fd03c0adac0 2024-09-19T07:35:30.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.871+0000 7fd04133f640 1 -- 172.21.15.83:0/1524369224 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd03c0c96a0 con 0x7fd03c0adac0 2024-09-19T07:35:30.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.871+0000 7fd0337fe640 1 -- 172.21.15.83:0/1524369224 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5707 (secure 0 0 0) 0x7fd03c0abb50 con 0x7fd03c0adac0 2024-09-19T07:35:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:30.866 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":122,"btime":"2024-09-19T07:35:30:306576+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:29.309817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:35:30.866 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 122 2024-09-19T07:35:30.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.875+0000 7fd04133f640 1 -- 172.21.15.83:0/1524369224 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd01c0678c0 msgr2=0x7fd01c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.875+0000 7fd04133f640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd01c0678c0 0x7fd01c069d80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0ad000 tx=0x7fd02c0a6000 comp rx=0 tx=0).stop 2024-09-19T07:35:30.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.875+0000 7fd04133f640 1 -- 172.21.15.83:0/1524369224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd03c0adac0 msgr2=0x7fd03c1eeee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:30.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.875+0000 7fd04133f640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd03c0adac0 0x7fd03c1eeee0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd03400cc30 tx=0x7fd03400a5c0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.875+0000 7fd04133f640 1 -- 172.21.15.83:0/1524369224 shutdown_connections 2024-09-19T07:35:30.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.875+0000 7fd04133f640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd01c0678c0 0x7fd01c069d80 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.875+0000 7fd04133f640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd03c0adac0 0x7fd03c1eeee0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.875+0000 7fd04133f640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c1592f0 0x7fd03c1ee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.875+0000 7fd04133f640 1 --2- 172.21.15.83:0/1524369224 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c0c6120 0x7fd03c0ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:30.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.875+0000 7fd04133f640 1 -- 172.21.15.83:0/1524369224 >> 172.21.15.83:0/1524369224 conn(0x7fd03c06bc80 msgr2=0x7fd03c1575d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:30.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.875+0000 7fd04133f640 1 -- 172.21.15.83:0/1524369224 shutdown_connections 2024-09-19T07:35:30.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:30.875+0000 7fd04133f640 1 -- 172.21.15.83:0/1524369224 wait complete. 2024-09-19T07:35:30.884 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:35:30:306576+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 122, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 114, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6738": { "addr": "172.21.15.83:6835/1079027718", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 1079027718, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 1079027718, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6738, "incarnation": 111, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:34:11.258171+0000", "qdb_cluster": [ 6738 ], "qdb_leader": 6738, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6738 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 122, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6918": { "addr": "172.21.15.83:6837/3707393259", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 3707393259, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 3707393259, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6918, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:35:29.309817+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6918 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/381468017", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 381468017, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 381468017, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 119, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5555, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:35:31.366 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:35:31.371+0000 7f4235345640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:35:32.393 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:32.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.527+0000 7f6169a73640 1 Processor -- start 2024-09-19T07:35:32.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.527+0000 7f6169a73640 1 -- start start 2024-09-19T07:35:32.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f6169a73640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f61640c6120 0x7f61640c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:32.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f6169a73640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6164155d70 0x7f61641575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:32.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f6169a73640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6164156200 con 0x7f61641592f0 2024-09-19T07:35:32.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f6169a73640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f61640c9930 con 0x7f61640c6120 2024-09-19T07:35:32.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f6169a73640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f61641564e0 con 0x7f6164155d70 2024-09-19T07:35:32.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f61637fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f61640c6120 0x7f61640c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:32.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f6163fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f61641592f0 0x7f6164157b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36052/0 (socket says 172.21.15.83:36052) 2024-09-19T07:35:32.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f6163fff640 1 -- 172.21.15.83:0/1545635638 learned_addr learned my addr 172.21.15.83:0/1545635638 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:32.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f6162ffd640 1 --2- 172.21.15.83:0/1545635638 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6164155d70 0x7f61641575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:32.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f61637fe640 1 -- 172.21.15.83:0/1545635638 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6164155d70 msgr2=0x7f61641575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:32.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f61637fe640 1 --2- 172.21.15.83:0/1545635638 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6164155d70 0x7f61641575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:32.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f61637fe640 1 -- 172.21.15.83:0/1545635638 >> v1:172.21.15.83:6789/0 conn(0x7f61641592f0 legacy=0x7f6164157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:32.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f61637fe640 1 -- 172.21.15.83:0/1545635638 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61641567c0 con 0x7f61640c6120 2024-09-19T07:35:32.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f61627fc640 1 -- 172.21.15.83:0/1545635638 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2254402953 0 0) 0x7f6164156200 con 0x7f61641592f0 2024-09-19T07:35:32.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f61637fe640 1 --2- 172.21.15.83:0/1545635638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61640c6120 0x7f61640c6500 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f61540097c0 tx=0x7f615401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e2a88addc63957 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:32.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f61627fc640 1 -- 172.21.15.83:0/1545635638 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61540b0070 con 0x7f61640c6120 2024-09-19T07:35:32.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f61627fc640 1 -- 172.21.15.83:0/1545635638 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61540ab600 con 0x7f61640c6120 2024-09-19T07:35:32.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.531+0000 7f61627fc640 1 -- 172.21.15.83:0/1545635638 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61540ab9d0 con 0x7f61640c6120 2024-09-19T07:35:32.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 -- 172.21.15.83:0/1545635638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61640c6120 msgr2=0x7f61640c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:32.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 --2- 172.21.15.83:0/1545635638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61640c6120 0x7f61640c6500 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f61540097c0 tx=0x7f615401c4a0 comp rx=0 tx=0).stop 2024-09-19T07:35:32.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 -- 172.21.15.83:0/1545635638 shutdown_connections 2024-09-19T07:35:32.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 --2- 172.21.15.83:0/1545635638 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6164155d70 0x7f61641575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:32.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 --2- 172.21.15.83:0/1545635638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61640c6120 0x7f61640c6500 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:32.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 -- 172.21.15.83:0/1545635638 >> 172.21.15.83:0/1545635638 conn(0x7f616406bc80 msgr2=0x7f616406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:32.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 -- 172.21.15.83:0/1545635638 shutdown_connections 2024-09-19T07:35:32.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 -- 172.21.15.83:0/1545635638 wait complete. 2024-09-19T07:35:32.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 Processor -- start 2024-09-19T07:35:32.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 -- start start 2024-09-19T07:35:32.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61640c6120 0x7f61641f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:32.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61641592f0 0x7f616414a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:32.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61640c06d0 0x7f616414aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:32.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6164153860 con 0x7f61641592f0 2024-09-19T07:35:32.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6164162ea0 con 0x7f61640c06d0 2024-09-19T07:35:32.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6164153f80 con 0x7f61640c6120 2024-09-19T07:35:32.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6162ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61641592f0 0x7f616414a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6162ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61641592f0 0x7f616414a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33384/0 (socket says 172.21.15.83:33384) 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6162ffd640 1 -- 172.21.15.83:0/3631529121 learned_addr learned my addr 172.21.15.83:0/3631529121 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f61637fe640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61640c6120 0x7f61641f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6162ffd640 1 -- 172.21.15.83:0/3631529121 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61640c6120 msgr2=0x7f61641f1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6163fff640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61640c06d0 0x7f616414aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6162ffd640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61640c6120 0x7f61641f1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6162ffd640 1 -- 172.21.15.83:0/3631529121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61640c06d0 msgr2=0x7f616414aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6162ffd640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61640c06d0 0x7f616414aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6162ffd640 1 -- 172.21.15.83:0/3631529121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61640cb9b0 con 0x7f61641592f0 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6162ffd640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61641592f0 0x7f616414a990 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f615800d800 tx=0x7f615800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f614bfff640 1 -- 172.21.15.83:0/3631529121 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615800b660 con 0x7f61641592f0 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f614bfff640 1 -- 172.21.15.83:0/3631529121 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6158019070 con 0x7f61641592f0 2024-09-19T07:35:32.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 -- 172.21.15.83:0/3631529121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61640ca030 con 0x7f61641592f0 2024-09-19T07:35:32.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f614bfff640 1 -- 172.21.15.83:0/3631529121 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615800ba00 con 0x7f61641592f0 2024-09-19T07:35:32.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.535+0000 7f6169a73640 1 -- 172.21.15.83:0/3631529121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61641f2280 con 0x7f61641592f0 2024-09-19T07:35:32.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.539+0000 7f6169a73640 1 -- 172.21.15.83:0/3631529121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61640c6500 con 0x7f61641592f0 2024-09-19T07:35:32.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.539+0000 7f614bfff640 1 -- 172.21.15.83:0/3631529121 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f615800bba0 con 0x7f61641592f0 2024-09-19T07:35:32.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.539+0000 7f614bfff640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61500678c0 0x7f6150069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:32.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.539+0000 7f614bfff640 1 -- 172.21.15.83:0/3631529121 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8615+0+0 (secure 0 0 0) 0x7f6158086c20 con 0x7f61641592f0 2024-09-19T07:35:32.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.539+0000 7f61637fe640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61500678c0 0x7f6150069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:32.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.539+0000 7f61637fe640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61500678c0 0x7f6150069d80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f61540ad000 tx=0x7f61540a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:32.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.543+0000 7f614bfff640 1 -- 172.21.15.83:0/3631529121 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6158056be0 con 0x7f61641592f0 2024-09-19T07:35:32.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 -- 172.21.15.83:0/3631529121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61640c96a0 con 0x7f61641592f0 2024-09-19T07:35:32.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f614bfff640 1 -- 172.21.15.83:0/3631529121 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+5714 (secure 0 0 0) 0x7f61580564e0 con 0x7f61641592f0 2024-09-19T07:35:32.740 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:32.740 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":123,"btime":"2024-09-19T07:35:31:315258+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:35:32.741 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 123 2024-09-19T07:35:32.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 -- 172.21.15.83:0/3631529121 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61500678c0 msgr2=0x7f6150069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:32.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61500678c0 0x7f6150069d80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f61540ad000 tx=0x7f61540a6000 comp rx=0 tx=0).stop 2024-09-19T07:35:32.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 -- 172.21.15.83:0/3631529121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61641592f0 msgr2=0x7f616414a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:32.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61641592f0 0x7f616414a990 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f615800d800 tx=0x7f615800dcc0 comp rx=0 tx=0).stop 2024-09-19T07:35:32.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 -- 172.21.15.83:0/3631529121 shutdown_connections 2024-09-19T07:35:32.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f61500678c0 0x7f6150069d80 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:32.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61640c06d0 0x7f616414aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:32.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61641592f0 0x7f616414a990 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:32.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 --2- 172.21.15.83:0/3631529121 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61640c6120 0x7f61641f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:32.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 -- 172.21.15.83:0/3631529121 >> 172.21.15.83:0/3631529121 conn(0x7f616406bc80 msgr2=0x7f61641597a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:32.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 -- 172.21.15.83:0/3631529121 shutdown_connections 2024-09-19T07:35:32.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:32.747+0000 7f6169a73640 1 -- 172.21.15.83:0/3631529121 wait complete. 2024-09-19T07:35:32.759 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:35:32.885 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:33.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.019+0000 7fc07546c640 1 Processor -- start 2024-09-19T07:35:33.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.019+0000 7fc07546c640 1 -- start start 2024-09-19T07:35:33.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc07546c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc0700c28e0 0x7fc0701545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:33.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc07546c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0700a9e60 con 0x7fc070071100 2024-09-19T07:35:33.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc07546c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0700aa140 con 0x7fc07006dea0 2024-09-19T07:35:33.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc07546c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc070071b50 con 0x7fc0700c28e0 2024-09-19T07:35:33.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06effd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc070071100 0x7fc070150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36060/0 (socket says 172.21.15.83:36060) 2024-09-19T07:35:33.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06effd640 1 -- 172.21.15.83:0/2045975923 learned_addr learned my addr 172.21.15.83:0/2045975923 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:33.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06e7fc640 1 --2- 172.21.15.83:0/2045975923 >> v2:172.21.15.184:3300/0 conn(0x7fc0700c28e0 0x7fc0701545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:33.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3676222529 0 0) 0x7fc0700a9e60 con 0x7fc070071100 2024-09-19T07:35:33.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc058003530 con 0x7fc070071100 2024-09-19T07:35:33.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 398181586 0 0) 0x7fc058003530 con 0x7fc070071100 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 >> v2:172.21.15.184:3300/0 conn(0x7fc0700c28e0 msgr2=0x7fc0701545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 --2- 172.21.15.83:0/2045975923 >> v2:172.21.15.184:3300/0 conn(0x7fc0700c28e0 0x7fc0701545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 >> v1:172.21.15.184:6789/0 conn(0x7fc07006dea0 legacy=0x7fc07006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0700c2240 con 0x7fc070071100 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fc060001a00 con 0x7fc070071100 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 >> v1:172.21.15.83:6789/0 conn(0x7fc070071100 legacy=0x7fc070150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 --2- 172.21.15.83:0/2045975923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0580039c0 0x7fc058003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 --2- 172.21.15.83:0/2045975923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 0x7fc058004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 --2- 172.21.15.83:0/2045975923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0580051d0 0x7fc05800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc070071b50 con 0x7fc058004810 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc05800c170 con 0x7fc0580051d0 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc05800c370 con 0x7fc0580039c0 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06e7fc640 1 --2- 172.21.15.83:0/2045975923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 0x7fc058004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:33.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06f7fe640 1 --2- 172.21.15.83:0/2045975923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0580051d0 0x7fc05800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06e7fc640 1 -- 172.21.15.83:0/2045975923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0580039c0 msgr2=0x7fc058003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06effd640 1 --2- 172.21.15.83:0/2045975923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0580039c0 0x7fc058003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06e7fc640 1 --2- 172.21.15.83:0/2045975923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0580039c0 0x7fc058003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06e7fc640 1 -- 172.21.15.83:0/2045975923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0580051d0 msgr2=0x7fc05800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06e7fc640 1 --2- 172.21.15.83:0/2045975923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0580051d0 0x7fc05800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06e7fc640 1 -- 172.21.15.83:0/2045975923 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc058003530 con 0x7fc058004810 2024-09-19T07:35:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06f7fe640 1 --2- 172.21.15.83:0/2045975923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0580051d0 0x7fc05800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:35:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.023+0000 7fc06f7fe640 1 -- 172.21.15.83:0/2045975923 reap_dead start 2024-09-19T07:35:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06e7fc640 1 --2- 172.21.15.83:0/2045975923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 0x7fc058004c90 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fc064009ae0 tx=0x7fc06401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c92b56ab5404c05 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0640096b0 con 0x7fc058004810 2024-09-19T07:35:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0640a64d0 con 0x7fc058004810 2024-09-19T07:35:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06dffb640 1 -- 172.21.15.83:0/2045975923 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0640a68c0 con 0x7fc058004810 2024-09-19T07:35:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 -- 172.21.15.83:0/2045975923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 msgr2=0x7fc058004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 --2- 172.21.15.83:0/2045975923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 0x7fc058004c90 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fc064009ae0 tx=0x7fc06401ca70 comp rx=0 tx=0).stop 2024-09-19T07:35:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 -- 172.21.15.83:0/2045975923 shutdown_connections 2024-09-19T07:35:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 --2- 172.21.15.83:0/2045975923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 0x7fc058004c90 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 -- 172.21.15.83:0/2045975923 >> 172.21.15.83:0/2045975923 conn(0x7fc07006bc80 msgr2=0x7fc07006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 -- 172.21.15.83:0/2045975923 shutdown_connections 2024-09-19T07:35:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 -- 172.21.15.83:0/2045975923 wait complete. 2024-09-19T07:35:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 Processor -- start 2024-09-19T07:35:33.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 -- start start 2024-09-19T07:35:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 0x7fc0700bc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc070156830 0x7fc0700c3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0700c3740 0x7fc0701ed180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc07006e280 con 0x7fc058004810 2024-09-19T07:35:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc0700716f0 con 0x7fc0700c3740 2024-09-19T07:35:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc07546c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc07006e400 con 0x7fc070156830 2024-09-19T07:35:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 0x7fc0700bc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 0x7fc0700bc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33394/0 (socket says 172.21.15.83:33394) 2024-09-19T07:35:33.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06effd640 1 -- 172.21.15.83:0/504399956 learned_addr learned my addr 172.21.15.83:0/504399956 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:33.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06f7fe640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0700c3740 0x7fc0701ed180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:33.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06e7fc640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc070156830 0x7fc0700c3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:33.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06effd640 1 -- 172.21.15.83:0/504399956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc070156830 msgr2=0x7fc0700c3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:33.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06effd640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc070156830 0x7fc0700c3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:33.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06effd640 1 -- 172.21.15.83:0/504399956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0700c3740 msgr2=0x7fc0701ed180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:33.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06effd640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0700c3740 0x7fc0701ed180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:33.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06effd640 1 -- 172.21.15.83:0/504399956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc070155a80 con 0x7fc058004810 2024-09-19T07:35:33.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.027+0000 7fc06effd640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 0x7fc0700bc610 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fc060002150 tx=0x7fc06000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:33.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.031+0000 7fc0577fe640 1 -- 172.21.15.83:0/504399956 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc060019070 con 0x7fc058004810 2024-09-19T07:35:33.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.031+0000 7fc0577fe640 1 -- 172.21.15.83:0/504399956 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc06000be50 con 0x7fc058004810 2024-09-19T07:35:33.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.031+0000 7fc0577fe640 1 -- 172.21.15.83:0/504399956 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0600070d0 con 0x7fc058004810 2024-09-19T07:35:33.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.031+0000 7fc07546c640 1 -- 172.21.15.83:0/504399956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0701f0c90 con 0x7fc058004810 2024-09-19T07:35:33.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.031+0000 7fc07546c640 1 -- 172.21.15.83:0/504399956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc070155490 con 0x7fc058004810 2024-09-19T07:35:33.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.031+0000 7fc07546c640 1 -- 172.21.15.83:0/504399956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc07006e790 con 0x7fc058004810 2024-09-19T07:35:33.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.031+0000 7fc0577fe640 1 -- 172.21.15.83:0/504399956 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc06001c070 con 0x7fc058004810 2024-09-19T07:35:33.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.031+0000 7fc0577fe640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc05c0678f0 0x7fc05c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:33.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.031+0000 7fc06e7fc640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc05c0678f0 0x7fc05c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:33.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.031+0000 7fc0577fe640 1 -- 172.21.15.83:0/504399956 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8615+0+0 (secure 0 0 0) 0x7fc060086930 con 0x7fc058004810 2024-09-19T07:35:33.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.031+0000 7fc06e7fc640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc05c0678f0 0x7fc05c069db0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc0640af000 tx=0x7fc0640046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:33.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.035+0000 7fc0577fe640 1 -- 172.21.15.83:0/504399956 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc060056950 con 0x7fc058004810 2024-09-19T07:35:33.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.235+0000 7fc07546c640 1 -- 172.21.15.83:0/504399956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0700c4f30 con 0x7fc058004810 2024-09-19T07:35:33.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.235+0000 7fc0577fe640 1 -- 172.21.15.83:0/504399956 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+5714 (secure 0 0 0) 0x7fc060056250 con 0x7fc058004810 2024-09-19T07:35:33.229 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:33.230 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":123,"btime":"2024-09-19T07:35:31:315258+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:35:33.230 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 123 2024-09-19T07:35:33.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.239+0000 7fc07546c640 1 -- 172.21.15.83:0/504399956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc05c0678f0 msgr2=0x7fc05c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:33.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.239+0000 7fc07546c640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc05c0678f0 0x7fc05c069db0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc0640af000 tx=0x7fc0640046d0 comp rx=0 tx=0).stop 2024-09-19T07:35:33.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.239+0000 7fc07546c640 1 -- 172.21.15.83:0/504399956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 msgr2=0x7fc0700bc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:33.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.239+0000 7fc07546c640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 0x7fc0700bc610 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fc060002150 tx=0x7fc06000b4a0 comp rx=0 tx=0).stop 2024-09-19T07:35:33.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.239+0000 7fc07546c640 1 -- 172.21.15.83:0/504399956 shutdown_connections 2024-09-19T07:35:33.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.239+0000 7fc07546c640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc05c0678f0 0x7fc05c069db0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:33.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.239+0000 7fc07546c640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0700c3740 0x7fc0701ed180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:33.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.239+0000 7fc07546c640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc070156830 0x7fc0700c3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:33.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.239+0000 7fc07546c640 1 --2- 172.21.15.83:0/504399956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc058004810 0x7fc0700bc610 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:33.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.239+0000 7fc07546c640 1 -- 172.21.15.83:0/504399956 >> 172.21.15.83:0/504399956 conn(0x7fc07006bc80 msgr2=0x7fc0701e2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:33.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.239+0000 7fc07546c640 1 -- 172.21.15.83:0/504399956 shutdown_connections 2024-09-19T07:35:33.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:33.239+0000 7fc07546c640 1 -- 172.21.15.83:0/504399956 wait complete. 2024-09-19T07:35:33.249 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5722 2024-09-19T07:35:33.250 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.b 2024-09-19T07:35:34.254 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:35:34.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:35:34.760 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:34.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc7b89640 1 Processor -- start 2024-09-19T07:35:34.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc7b89640 1 -- start start 2024-09-19T07:35:34.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc7b89640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4cc0150290 0x7f4cc00c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:34.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc7b89640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cc0152b20 0x7f4cc00c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:34.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc7b89640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4cc00c45c0 0x7f4cc0153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:34.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc7b89640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f4cc00716c0 con 0x7f4cc0150290 2024-09-19T07:35:34.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc7b89640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4cc00c3920 con 0x7f4cc00c45c0 2024-09-19T07:35:34.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc7b89640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4cc00c3490 con 0x7f4cc0152b20 2024-09-19T07:35:34.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc5925640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4cc0150290 0x7f4cc00c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:34.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc5925640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4cc0150290 0x7f4cc00c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33410/0 (socket says 172.21.15.83:33410) 2024-09-19T07:35:34.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc5925640 1 -- 172.21.15.83:0/2257052289 learned_addr learned my addr 172.21.15.83:0/2257052289 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:34.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc6126640 1 --2- 172.21.15.83:0/2257052289 >> v2:172.21.15.184:3300/0 conn(0x7f4cc00c45c0 0x7f4cc0153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:34.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc5124640 1 --2- 172.21.15.83:0/2257052289 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cc0152b20 0x7f4cc00c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:34.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc5925640 1 -- 172.21.15.83:0/2257052289 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cc0152b20 msgr2=0x7f4cc00c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:34.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc5925640 1 --2- 172.21.15.83:0/2257052289 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cc0152b20 0x7f4cc00c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:34.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc5925640 1 -- 172.21.15.83:0/2257052289 >> v2:172.21.15.184:3300/0 conn(0x7f4cc00c45c0 msgr2=0x7f4cc0153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:34.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc5925640 1 --2- 172.21.15.83:0/2257052289 >> v2:172.21.15.184:3300/0 conn(0x7f4cc00c45c0 0x7f4cc0153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:34.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc5925640 1 -- 172.21.15.83:0/2257052289 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc0150670 con 0x7f4cc0150290 2024-09-19T07:35:34.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.899+0000 7f4cc5124640 1 --2- 172.21.15.83:0/2257052289 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cc0152b20 0x7f4cc00c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:35:34.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc5925640 1 --2- 172.21.15.83:0/2257052289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc0150290 0x7f4cc00c1ff0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00ad7d0 tx=0x7f4cb801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dfeba1484dbae70 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:34.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc4923640 1 -- 172.21.15.83:0/2257052289 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80a5840 con 0x7f4cc0150290 2024-09-19T07:35:34.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc4923640 1 -- 172.21.15.83:0/2257052289 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cb80a59e0 con 0x7f4cc0150290 2024-09-19T07:35:34.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc4923640 1 -- 172.21.15.83:0/2257052289 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80a5db0 con 0x7f4cc0150290 2024-09-19T07:35:34.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 -- 172.21.15.83:0/2257052289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc0150290 msgr2=0x7f4cc00c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:34.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 --2- 172.21.15.83:0/2257052289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc0150290 0x7f4cc00c1ff0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00ad7d0 tx=0x7f4cb801c9c0 comp rx=0 tx=0).stop 2024-09-19T07:35:34.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 -- 172.21.15.83:0/2257052289 shutdown_connections 2024-09-19T07:35:34.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 --2- 172.21.15.83:0/2257052289 >> v2:172.21.15.184:3300/0 conn(0x7f4cc00c45c0 0x7f4cc0153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:34.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 --2- 172.21.15.83:0/2257052289 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cc0152b20 0x7f4cc00c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:34.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 --2- 172.21.15.83:0/2257052289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc0150290 0x7f4cc00c1ff0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:34.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 -- 172.21.15.83:0/2257052289 >> 172.21.15.83:0/2257052289 conn(0x7f4cc006bc80 msgr2=0x7f4cc006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:34.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 -- 172.21.15.83:0/2257052289 shutdown_connections 2024-09-19T07:35:34.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 -- 172.21.15.83:0/2257052289 wait complete. 2024-09-19T07:35:34.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 Processor -- start 2024-09-19T07:35:34.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 -- start start 2024-09-19T07:35:34.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc00c45c0 0x7f4cc00ab100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:34.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.903+0000 7f4cc7b89640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cc0150290 0x7f4cc0071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc7b89640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cc0152b20 0x7f4cc01f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc7b89640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc00c2530 con 0x7f4cc00c45c0 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc7b89640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc00c3120 con 0x7f4cc0150290 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc7b89640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4cc0154800 con 0x7f4cc0152b20 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5925640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc00c45c0 0x7f4cc00ab100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5925640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc00c45c0 0x7f4cc00ab100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33420/0 (socket says 172.21.15.83:33420) 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5925640 1 -- 172.21.15.83:0/3761628984 learned_addr learned my addr 172.21.15.83:0/3761628984 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc6126640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cc0152b20 0x7f4cc01f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5124640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cc0150290 0x7f4cc0071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5925640 1 -- 172.21.15.83:0/3761628984 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cc0152b20 msgr2=0x7f4cc01f1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5925640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cc0152b20 0x7f4cc01f1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5925640 1 -- 172.21.15.83:0/3761628984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cc0150290 msgr2=0x7f4cc0071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5925640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cc0150290 0x7f4cc0071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:34.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5925640 1 -- 172.21.15.83:0/3761628984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc01f2780 con 0x7f4cc00c45c0 2024-09-19T07:35:34.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5124640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cc0150290 0x7f4cc0071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:35:34.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5925640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc00c45c0 0x7f4cc00ab100 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f4cb801b070 tx=0x7f4cb8019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:34.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cbe7fc640 1 -- 172.21.15.83:0/3761628984 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80bc020 con 0x7f4cc00c45c0 2024-09-19T07:35:34.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cbe7fc640 1 -- 172.21.15.83:0/3761628984 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cb80b8670 con 0x7f4cc00c45c0 2024-09-19T07:35:34.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc7b89640 1 -- 172.21.15.83:0/3761628984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc0149eb0 con 0x7f4cc00c45c0 2024-09-19T07:35:34.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cbe7fc640 1 -- 172.21.15.83:0/3761628984 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80b8a60 con 0x7f4cc00c45c0 2024-09-19T07:35:34.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc7b89640 1 -- 172.21.15.83:0/3761628984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc00aa490 con 0x7f4cc00c45c0 2024-09-19T07:35:34.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc7b89640 1 -- 172.21.15.83:0/3761628984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc00c1ff0 con 0x7f4cc00c45c0 2024-09-19T07:35:34.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cbe7fc640 1 -- 172.21.15.83:0/3761628984 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4cb80bf020 con 0x7f4cc00c45c0 2024-09-19T07:35:34.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cbe7fc640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4cac0678c0 0x7f4cac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:34.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cbe7fc640 1 -- 172.21.15.83:0/3761628984 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8615+0+0 (secure 0 0 0) 0x7f4cb81219a0 con 0x7f4cc00c45c0 2024-09-19T07:35:34.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5124640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4cac0678c0 0x7f4cac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:34.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.907+0000 7f4cc5124640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4cac0678c0 0x7f4cac069d80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00aa090 tx=0x7f4cb4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:34.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:34.911+0000 7f4cbe7fc640 1 -- 172.21.15.83:0/3761628984 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4cb80c6030 con 0x7f4cc00c45c0 2024-09-19T07:35:35.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.107+0000 7f4cc7b89640 1 -- 172.21.15.83:0/3761628984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cc0070fd0 con 0x7f4cc00c45c0 2024-09-19T07:35:35.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cbe7fc640 1 -- 172.21.15.83:0/3761628984 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+5714 (secure 0 0 0) 0x7f4cb80165e0 con 0x7f4cc00c45c0 2024-09-19T07:35:35.104 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:35.104 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":123,"btime":"2024-09-19T07:35:31:315258+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:35:35.104 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 123 2024-09-19T07:35:35.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cc7b89640 1 -- 172.21.15.83:0/3761628984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4cac0678c0 msgr2=0x7f4cac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:35.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cc7b89640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4cac0678c0 0x7f4cac069d80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00aa090 tx=0x7f4cb4008040 comp rx=0 tx=0).stop 2024-09-19T07:35:35.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cc7b89640 1 -- 172.21.15.83:0/3761628984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc00c45c0 msgr2=0x7f4cc00ab100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:35.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cc7b89640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc00c45c0 0x7f4cc00ab100 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f4cb801b070 tx=0x7f4cb8019760 comp rx=0 tx=0).stop 2024-09-19T07:35:35.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cc7b89640 1 -- 172.21.15.83:0/3761628984 shutdown_connections 2024-09-19T07:35:35.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cc7b89640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4cac0678c0 0x7f4cac069d80 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:35.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cc7b89640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cc0152b20 0x7f4cc01f1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:35.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cc7b89640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cc0150290 0x7f4cc0071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:35.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cc7b89640 1 --2- 172.21.15.83:0/3761628984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc00c45c0 0x7f4cc00ab100 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:35.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cc7b89640 1 -- 172.21.15.83:0/3761628984 >> 172.21.15.83:0/3761628984 conn(0x7f4cc006bc80 msgr2=0x7f4cc006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:35.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cc7b89640 1 -- 172.21.15.83:0/3761628984 shutdown_connections 2024-09-19T07:35:35.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:35.111+0000 7f4cc7b89640 1 -- 172.21.15.83:0/3761628984 wait complete. 2024-09-19T07:35:35.123 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:35:36.250 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-09-19T07:35:36.251 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-19T07:35:36.251 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-19T07:35:36.254 INFO:tasks.ceph.mds.b:Started 2024-09-19T07:35:36.254 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:36.332 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-19T07:35:36.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a0e668640 1 Processor -- start 2024-09-19T07:35:36.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a0e668640 1 -- start start 2024-09-19T07:35:36.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a0e668640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a080c42e0 0x7f0a0806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:36.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a0e668640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0a080abb30 0x7f0a08153060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:36.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a0e668640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a081592c0 con 0x7f0a081566b0 2024-09-19T07:35:36.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a0e668640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f0a081595f0 con 0x7f0a080abb30 2024-09-19T07:35:36.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a0e668640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0a0806e900 con 0x7f0a080c42e0 2024-09-19T07:35:36.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a07fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a080c42e0 0x7f0a0806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:36.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a0cc05640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0a080abb30 0x7f0a08153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:36.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a07fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a080c42e0 0x7f0a0806e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:39804/0 (socket says 172.21.15.83:39804) 2024-09-19T07:35:36.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f09ff7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f0a081566b0 0x7f0a08159a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:46666/0 (socket says 172.21.15.83:46666) 2024-09-19T07:35:36.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a07fff640 1 -- 172.21.15.83:0/580690487 learned_addr learned my addr 172.21.15.83:0/580690487 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:36.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a0cc05640 1 -- 172.21.15.83:0/580690487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a080c42e0 msgr2=0x7f0a0806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:36.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a0cc05640 1 --2- 172.21.15.83:0/580690487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a080c42e0 0x7f0a0806e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:36.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a0cc05640 1 -- 172.21.15.83:0/580690487 >> v1:172.21.15.184:6789/0 conn(0x7f0a081566b0 legacy=0x7f0a08159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:36.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a077fe640 1 -- 172.21.15.83:0/580690487 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2031929662 0 0) 0x7f0a081592c0 con 0x7f0a081566b0 2024-09-19T07:35:36.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a0cc05640 1 -- 172.21.15.83:0/580690487 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0806ebe0 con 0x7f0a080abb30 2024-09-19T07:35:36.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.390+0000 7f0a07fff640 1 --2- 172.21.15.83:0/580690487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a080c42e0 0x7f0a0806e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:35:36.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0cc05640 1 --2- 172.21.15.83:0/580690487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a080abb30 0x7f0a08153060 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009d90 tx=0x7f0a0001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a086bf025ab3ee6d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:36.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a077fe640 1 -- 172.21.15.83:0/580690487 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a000b2070 con 0x7f0a080abb30 2024-09-19T07:35:36.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a077fe640 1 -- 172.21.15.83:0/580690487 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a000a5a20 con 0x7f0a080abb30 2024-09-19T07:35:36.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a077fe640 1 -- 172.21.15.83:0/580690487 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a000a5df0 con 0x7f0a080abb30 2024-09-19T07:35:36.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 -- 172.21.15.83:0/580690487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a080abb30 msgr2=0x7f0a08153060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:36.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 --2- 172.21.15.83:0/580690487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a080abb30 0x7f0a08153060 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009d90 tx=0x7f0a0001c940 comp rx=0 tx=0).stop 2024-09-19T07:35:36.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 -- 172.21.15.83:0/580690487 shutdown_connections 2024-09-19T07:35:36.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 --2- 172.21.15.83:0/580690487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a080abb30 0x7f0a08153060 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:36.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 --2- 172.21.15.83:0/580690487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a080c42e0 0x7f0a0806e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:36.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 -- 172.21.15.83:0/580690487 >> 172.21.15.83:0/580690487 conn(0x7f0a0806bc80 msgr2=0x7f0a0806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:36.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 -- 172.21.15.83:0/580690487 shutdown_connections 2024-09-19T07:35:36.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 -- 172.21.15.83:0/580690487 wait complete. 2024-09-19T07:35:36.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 Processor -- start 2024-09-19T07:35:36.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 -- start start 2024-09-19T07:35:36.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a080abb30 0x7f0a08072210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:36.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a080c42e0 0x7f0a0814db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:36.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a081566b0 0x7f0a0814a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:36.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0a080c4840 con 0x7f0a080abb30 2024-09-19T07:35:36.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0a080c46c0 con 0x7f0a081566b0 2024-09-19T07:35:36.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0e668640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0a08162dd0 con 0x7f0a080c42e0 2024-09-19T07:35:36.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a07fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a080abb30 0x7f0a08072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:36.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a07fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a080abb30 0x7f0a08072210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33436/0 (socket says 172.21.15.83:33436) 2024-09-19T07:35:36.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a07fff640 1 -- 172.21.15.83:0/2702414441 learned_addr learned my addr 172.21.15.83:0/2702414441 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:36.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f09ff7fe640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a080c42e0 0x7f0a0814db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:36.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.394+0000 7f0a0cc05640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a081566b0 0x7f0a0814a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:36.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a07fff640 1 -- 172.21.15.83:0/2702414441 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a080c42e0 msgr2=0x7f0a0814db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:36.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a07fff640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a080c42e0 0x7f0a0814db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:36.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a07fff640 1 -- 172.21.15.83:0/2702414441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a081566b0 msgr2=0x7f0a0814a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:36.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a07fff640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a081566b0 0x7f0a0814a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:36.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a07fff640 1 -- 172.21.15.83:0/2702414441 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a08073d40 con 0x7f0a080abb30 2024-09-19T07:35:36.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a07fff640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a080abb30 0x7f0a08072210 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f09f000e9d0 tx=0x7f09f000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:36.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a057fa640 1 -- 172.21.15.83:0/2702414441 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f0017070 con 0x7f0a080abb30 2024-09-19T07:35:36.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a057fa640 1 -- 172.21.15.83:0/2702414441 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09f000ca60 con 0x7f0a080abb30 2024-09-19T07:35:36.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a0e668640 1 -- 172.21.15.83:0/2702414441 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a08073690 con 0x7f0a080abb30 2024-09-19T07:35:36.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a057fa640 1 -- 172.21.15.83:0/2702414441 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f000ce10 con 0x7f0a080abb30 2024-09-19T07:35:36.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a0e668640 1 -- 172.21.15.83:0/2702414441 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a080710b0 con 0x7f0a080abb30 2024-09-19T07:35:36.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a0e668640 1 -- 172.21.15.83:0/2702414441 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a08153060 con 0x7f0a080abb30 2024-09-19T07:35:36.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a057fa640 1 -- 172.21.15.83:0/2702414441 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f09f001a070 con 0x7f0a080abb30 2024-09-19T07:35:36.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a057fa640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09e80678f0 0x7f09e8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:36.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f09ff7fe640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09e80678f0 0x7f09e8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:36.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f0a057fa640 1 -- 172.21.15.83:0/2702414441 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8615+0+0 (secure 0 0 0) 0x7f09f00866d0 con 0x7f0a080abb30 2024-09-19T07:35:36.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.398+0000 7f09ff7fe640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09e80678f0 0x7f09e8069db0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0a0814d6b0 tx=0x7f09f8007df0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:36.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.402+0000 7f0a057fa640 1 -- 172.21.15.83:0/2702414441 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f09f0056040 con 0x7f0a080abb30 2024-09-19T07:35:36.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.602+0000 7f0a0e668640 1 -- 172.21.15.83:0/2702414441 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a0806e050 con 0x7f0a080abb30 2024-09-19T07:35:36.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.602+0000 7f0a057fa640 1 -- 172.21.15.83:0/2702414441 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+5714 (secure 0 0 0) 0x7f09f005a340 con 0x7f0a080abb30 2024-09-19T07:35:36.598 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:36.598 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":123,"btime":"2024-09-19T07:35:31:315258+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:35:36.598 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 123 2024-09-19T07:35:36.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.606+0000 7f0a0e668640 1 -- 172.21.15.83:0/2702414441 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09e80678f0 msgr2=0x7f09e8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:36.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.606+0000 7f0a0e668640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09e80678f0 0x7f09e8069db0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0a0814d6b0 tx=0x7f09f8007df0 comp rx=0 tx=0).stop 2024-09-19T07:35:36.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.606+0000 7f0a0e668640 1 -- 172.21.15.83:0/2702414441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a080abb30 msgr2=0x7f0a08072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:36.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.606+0000 7f0a0e668640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a080abb30 0x7f0a08072210 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f09f000e9d0 tx=0x7f09f000ee90 comp rx=0 tx=0).stop 2024-09-19T07:35:36.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.606+0000 7f0a0e668640 1 -- 172.21.15.83:0/2702414441 shutdown_connections 2024-09-19T07:35:36.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.606+0000 7f0a0e668640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f09e80678f0 0x7f09e8069db0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:36.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.606+0000 7f0a0e668640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0a081566b0 0x7f0a0814a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:36.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.606+0000 7f0a0e668640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0a080c42e0 0x7f0a0814db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:36.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.606+0000 7f0a0e668640 1 --2- 172.21.15.83:0/2702414441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a080abb30 0x7f0a08072210 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:36.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.606+0000 7f0a0e668640 1 -- 172.21.15.83:0/2702414441 >> 172.21.15.83:0/2702414441 conn(0x7f0a0806bc80 msgr2=0x7f0a08150720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:36.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.606+0000 7f0a0e668640 1 -- 172.21.15.83:0/2702414441 shutdown_connections 2024-09-19T07:35:36.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:36.606+0000 7f0a0e668640 1 -- 172.21.15.83:0/2702414441 wait complete. 2024-09-19T07:35:36.617 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-19T07:35:37.124 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:37.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f582db57640 1 Processor -- start 2024-09-19T07:35:37.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f582db57640 1 -- start start 2024-09-19T07:35:37.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f582db57640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58280c9660 0x7f58281563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:37.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f582db57640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58280c6180 con 0x7f5828156910 2024-09-19T07:35:37.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f582db57640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5828155bf0 con 0x7f58280b0d00 2024-09-19T07:35:37.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f582db57640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f58280c69b0 con 0x7f58280c9660 2024-09-19T07:35:37.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f5827fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5828156910 0x7f58280c8210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36068/0 (socket says 172.21.15.83:36068) 2024-09-19T07:35:37.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f5827fff640 1 -- 172.21.15.83:0/1310578866 learned_addr learned my addr 172.21.15.83:0/1310578866 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:37.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f5826ffd640 1 --2- 172.21.15.83:0/1310578866 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58280c9660 0x7f58281563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:37.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2143786272 0 0) 0x7f58280c6180 con 0x7f5828156910 2024-09-19T07:35:37.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5810003530 con 0x7f5828156910 2024-09-19T07:35:37.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 473409219 0 0) 0x7f5810003530 con 0x7f5828156910 2024-09-19T07:35:37.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58280c9660 msgr2=0x7f58281563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:37.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 --2- 172.21.15.83:0/1310578866 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58280c9660 0x7f58281563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:37.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 >> v1:172.21.15.184:6789/0 conn(0x7f58280b0d00 legacy=0x7f5828159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:37.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58280c63f0 con 0x7f5828156910 2024-09-19T07:35:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f5820001a00 con 0x7f5828156910 2024-09-19T07:35:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 >> v1:172.21.15.83:6789/0 conn(0x7f5828156910 legacy=0x7f58280c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 --2- 172.21.15.83:0/1310578866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58100039f0 0x7f5810003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 --2- 172.21.15.83:0/1310578866 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58280c9660 0x7f5810004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 --2- 172.21.15.83:0/1310578866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5810004d50 0x7f58100051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f581000bad0 con 0x7f58100039f0 2024-09-19T07:35:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f581000bc80 con 0x7f5810004d50 2024-09-19T07:35:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f581000be80 con 0x7f58280c9660 2024-09-19T07:35:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58277fe640 1 --2- 172.21.15.83:0/1310578866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58100039f0 0x7f5810003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f5826ffd640 1 --2- 172.21.15.83:0/1310578866 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58280c9660 0x7f5810004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f5827fff640 1 --2- 172.21.15.83:0/1310578866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5810004d50 0x7f58100051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58277fe640 1 -- 172.21.15.83:0/1310578866 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58280c9660 msgr2=0x7f5810004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58277fe640 1 --2- 172.21.15.83:0/1310578866 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58280c9660 0x7f5810004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58277fe640 1 -- 172.21.15.83:0/1310578866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5810004d50 msgr2=0x7f58100051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58277fe640 1 --2- 172.21.15.83:0/1310578866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5810004d50 0x7f58100051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.262+0000 7f58277fe640 1 -- 172.21.15.83:0/1310578866 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5810003530 con 0x7f58100039f0 2024-09-19T07:35:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f58277fe640 1 --2- 172.21.15.83:0/1310578866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58100039f0 0x7f5810003e10 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f5818001780 tx=0x7f581801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb9006976b054278 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58180013a0 con 0x7f58100039f0 2024-09-19T07:35:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58180052f0 con 0x7f58100039f0 2024-09-19T07:35:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f58267fc640 1 -- 172.21.15.83:0/1310578866 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5818003840 con 0x7f58100039f0 2024-09-19T07:35:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f582db57640 1 -- 172.21.15.83:0/1310578866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58100039f0 msgr2=0x7f5810003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:37.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f582db57640 1 --2- 172.21.15.83:0/1310578866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58100039f0 0x7f5810003e10 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f5818001780 tx=0x7f581801c360 comp rx=0 tx=0).stop 2024-09-19T07:35:37.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f5827fff640 1 -- 172.21.15.83:0/1310578866 reap_dead start 2024-09-19T07:35:37.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f582db57640 1 -- 172.21.15.83:0/1310578866 shutdown_connections 2024-09-19T07:35:37.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f582db57640 1 -- 172.21.15.83:0/1310578866 >> 172.21.15.83:0/1310578866 conn(0x7f58280722a0 msgr2=0x7f58280726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:37.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f582db57640 1 -- 172.21.15.83:0/1310578866 shutdown_connections 2024-09-19T07:35:37.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f582db57640 1 -- 172.21.15.83:0/1310578866 wait complete. 2024-09-19T07:35:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f582db57640 1 Processor -- start 2024-09-19T07:35:37.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.266+0000 7f582db57640 1 -- start start 2024-09-19T07:35:37.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f582db57640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58100039f0 0x7f58281eed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:37.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f582db57640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58281592d0 0x7f58280c5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:37.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f582db57640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58280c1a20 0x7f58281f2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:37.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f582db57640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f58280c8aa0 con 0x7f58280c1a20 2024-09-19T07:35:37.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f582db57640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f58280c8920 con 0x7f58281592d0 2024-09-19T07:35:37.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f582db57640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5828161d90 con 0x7f58100039f0 2024-09-19T07:35:37.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f5827fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58280c1a20 0x7f58281f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:37.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f5827fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58280c1a20 0x7f58281f2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33460/0 (socket says 172.21.15.83:33460) 2024-09-19T07:35:37.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f5827fff640 1 -- 172.21.15.83:0/1706649575 learned_addr learned my addr 172.21.15.83:0/1706649575 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:37.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f5826ffd640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58281592d0 0x7f58280c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:37.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f58277fe640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58100039f0 0x7f58281eed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:37.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f5827fff640 1 -- 172.21.15.83:0/1706649575 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58100039f0 msgr2=0x7f58281eed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:37.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f5827fff640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58100039f0 0x7f58281eed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:37.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f5827fff640 1 -- 172.21.15.83:0/1706649575 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58281592d0 msgr2=0x7f58280c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:37.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f5827fff640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58281592d0 0x7f58280c5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:37.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f5827fff640 1 -- 172.21.15.83:0/1706649575 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58280acfa0 con 0x7f58280c1a20 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f5827fff640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58280c1a20 0x7f58281f2280 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f58280c7570 tx=0x7f582000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f580ffff640 1 -- 172.21.15.83:0/1706649575 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582000ebb0 con 0x7f58280c1a20 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f580ffff640 1 -- 172.21.15.83:0/1706649575 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5820018070 con 0x7f58280c1a20 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f582db57640 1 -- 172.21.15.83:0/1706649575 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5828158280 con 0x7f58280c1a20 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f580ffff640 1 -- 172.21.15.83:0/1706649575 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5820013420 con 0x7f58280c1a20 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f582db57640 1 -- 172.21.15.83:0/1706649575 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5828158840 con 0x7f58280c1a20 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f582db57640 1 -- 172.21.15.83:0/1706649575 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5828159bf0 con 0x7f58280c1a20 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f580ffff640 1 -- 172.21.15.83:0/1706649575 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f58200135c0 con 0x7f58280c1a20 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f580ffff640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f58140678f0 0x7f5814069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f580ffff640 1 -- 172.21.15.83:0/1706649575 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8615+0+0 (secure 0 0 0) 0x7f5820086ad0 con 0x7f58280c1a20 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f58277fe640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f58140678f0 0x7f5814069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.270+0000 7f58277fe640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f58140678f0 0x7f5814069db0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5818001750 tx=0x7f58180011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.274+0000 7f580ffff640 1 -- 172.21.15.83:0/1706649575 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5820056af0 con 0x7f58280c1a20 2024-09-19T07:35:37.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.478+0000 7f582db57640 1 -- 172.21.15.83:0/1706649575 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5828160570 con 0x7f58280c1a20 2024-09-19T07:35:37.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f580ffff640 1 -- 172.21.15.83:0/1706649575 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+6543 (secure 0 0 0) 0x7f58200563f0 con 0x7f58280c1a20 2024-09-19T07:35:37.476 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:37.476 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":124,"btime":"2024-09-19T07:35:37:340512+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":5612,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:35:37.476 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 124 2024-09-19T07:35:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f582db57640 1 -- 172.21.15.83:0/1706649575 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f58140678f0 msgr2=0x7f5814069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f582db57640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f58140678f0 0x7f5814069db0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5818001750 tx=0x7f58180011a0 comp rx=0 tx=0).stop 2024-09-19T07:35:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f582db57640 1 -- 172.21.15.83:0/1706649575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58280c1a20 msgr2=0x7f58281f2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f582db57640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58280c1a20 0x7f58281f2280 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f58280c7570 tx=0x7f582000e530 comp rx=0 tx=0).stop 2024-09-19T07:35:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f582db57640 1 -- 172.21.15.83:0/1706649575 shutdown_connections 2024-09-19T07:35:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f582db57640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f58140678f0 0x7f5814069db0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f582db57640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58280c1a20 0x7f58281f2280 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f582db57640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58281592d0 0x7f58280c5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f582db57640 1 --2- 172.21.15.83:0/1706649575 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58100039f0 0x7f58281eed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f582db57640 1 -- 172.21.15.83:0/1706649575 >> 172.21.15.83:0/1706649575 conn(0x7f58280722a0 msgr2=0x7f58280ca9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f582db57640 1 -- 172.21.15.83:0/1706649575 shutdown_connections 2024-09-19T07:35:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:37.482+0000 7f582db57640 1 -- 172.21.15.83:0/1706649575 wait complete. 2024-09-19T07:35:37.495 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:35:38.618 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:38.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.750+0000 7f5bae8b4640 1 Processor -- start 2024-09-19T07:35:38.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5bae8b4640 1 -- start start 2024-09-19T07:35:38.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5bae8b4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5ba8150290 0x7f5ba80c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:38.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5bae8b4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ba8150670 con 0x7f5ba8152b20 2024-09-19T07:35:38.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5bae8b4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ba80c2880 con 0x7f5ba8153eb0 2024-09-19T07:35:38.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5bae8b4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5ba80716c0 con 0x7f5ba8150290 2024-09-19T07:35:38.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5ba7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5ba8150290 0x7f5ba80c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:38.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5ba77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5ba8152b20 0x7f5ba80c4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36070/0 (socket says 172.21.15.83:36070) 2024-09-19T07:35:38.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5ba7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5ba8150290 0x7f5ba80c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33464/0 (socket says 172.21.15.83:33464) 2024-09-19T07:35:38.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5ba77fe640 1 -- 172.21.15.83:0/3169104804 learned_addr learned my addr 172.21.15.83:0/3169104804 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:38.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5ba7fff640 1 -- 172.21.15.83:0/3169104804 >> v1:172.21.15.184:6789/0 conn(0x7f5ba8153eb0 legacy=0x7f5ba8154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:38.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5ba7fff640 1 -- 172.21.15.83:0/3169104804 >> v1:172.21.15.83:6789/0 conn(0x7f5ba8152b20 legacy=0x7f5ba80c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:38.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5ba6ffd640 1 -- 172.21.15.83:0/3169104804 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 41417647 0 0) 0x7f5ba8150670 con 0x7f5ba8152b20 2024-09-19T07:35:38.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5ba7fff640 1 -- 172.21.15.83:0/3169104804 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba80c2b60 con 0x7f5ba8150290 2024-09-19T07:35:38.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5ba7fff640 1 --2- 172.21.15.83:0/3169104804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba8150290 0x7f5ba80c1ff0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0097c0 tx=0x7f5b9c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511fb973af130fb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:38.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.754+0000 7f5ba6ffd640 1 -- 172.21.15.83:0/3169104804 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c0b0070 con 0x7f5ba8150290 2024-09-19T07:35:38.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba6ffd640 1 -- 172.21.15.83:0/3169104804 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b9c0ab600 con 0x7f5ba8150290 2024-09-19T07:35:38.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba6ffd640 1 -- 172.21.15.83:0/3169104804 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c0ab9d0 con 0x7f5ba8150290 2024-09-19T07:35:38.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3169104804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba8150290 msgr2=0x7f5ba80c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:38.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 --2- 172.21.15.83:0/3169104804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba8150290 0x7f5ba80c1ff0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0097c0 tx=0x7f5b9c01c460 comp rx=0 tx=0).stop 2024-09-19T07:35:38.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3169104804 shutdown_connections 2024-09-19T07:35:38.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 --2- 172.21.15.83:0/3169104804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba8150290 0x7f5ba80c1ff0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:38.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3169104804 >> 172.21.15.83:0/3169104804 conn(0x7f5ba806bc80 msgr2=0x7f5ba806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:38.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3169104804 shutdown_connections 2024-09-19T07:35:38.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3169104804 wait complete. 2024-09-19T07:35:38.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 Processor -- start 2024-09-19T07:35:38.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 -- start start 2024-09-19T07:35:38.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba8150290 0x7f5ba80be2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:38.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ba8152b20 0x7f5ba81eff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ba8153eb0 0x7f5ba81499f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba80c4790 con 0x7f5ba8150290 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba80c3490 con 0x7f5ba8153eb0 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5ba80c4910 con 0x7f5ba8152b20 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba8150290 0x7f5ba80be2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba8150290 0x7f5ba80be2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33474/0 (socket says 172.21.15.83:33474) 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba7fff640 1 -- 172.21.15.83:0/3513872508 learned_addr learned my addr 172.21.15.83:0/3513872508 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba7fff640 1 -- 172.21.15.83:0/3513872508 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ba8152b20 msgr2=0x7f5ba81eff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba7fff640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ba8152b20 0x7f5ba81eff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba7fff640 1 -- 172.21.15.83:0/3513872508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ba8153eb0 msgr2=0x7f5ba81499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bace51640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ba8153eb0 0x7f5ba81499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba7fff640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ba8153eb0 0x7f5ba81499f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba7fff640 1 -- 172.21.15.83:0/3513872508 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba80bd8c0 con 0x7f5ba8150290 2024-09-19T07:35:38.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba7fff640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba8150290 0x7f5ba80be2d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c019070 tx=0x7f5b9c019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:38.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba4ff9640 1 -- 172.21.15.83:0/3513872508 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c0b8020 con 0x7f5ba8150290 2024-09-19T07:35:38.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba4ff9640 1 -- 172.21.15.83:0/3513872508 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b9c0b4610 con 0x7f5ba8150290 2024-09-19T07:35:38.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3513872508 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ba81f0990 con 0x7f5ba8150290 2024-09-19T07:35:38.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5ba4ff9640 1 -- 172.21.15.83:0/3513872508 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c0b0040 con 0x7f5ba8150290 2024-09-19T07:35:38.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.758+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3513872508 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ba81527f0 con 0x7f5ba8150290 2024-09-19T07:35:38.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.762+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3513872508 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ba80c1ff0 con 0x7f5ba8150290 2024-09-19T07:35:38.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.762+0000 7f5ba4ff9640 1 -- 172.21.15.83:0/3513872508 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5b9c0bb020 con 0x7f5ba8150290 2024-09-19T07:35:38.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.762+0000 7f5ba4ff9640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b940678c0 0x7f5b94069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:38.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.762+0000 7f5ba4ff9640 1 -- 172.21.15.83:0/3513872508 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8615+0+0 (secure 0 0 0) 0x7f5b9c1205e0 con 0x7f5ba8150290 2024-09-19T07:35:38.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.762+0000 7f5ba77fe640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b940678c0 0x7f5b94069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:38.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.762+0000 7f5ba77fe640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b940678c0 0x7f5b94069d80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5b980011f0 tx=0x7f5b98007950 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:38.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.766+0000 7f5ba4ff9640 1 -- 172.21.15.83:0/3513872508 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5b9c0c2030 con 0x7f5ba8150290 2024-09-19T07:35:38.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.966+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3513872508 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ba8152000 con 0x7f5ba8150290 2024-09-19T07:35:38.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.966+0000 7f5ba4ff9640 1 -- 172.21.15.83:0/3513872508 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+6543 (secure 0 0 0) 0x7f5b9c016050 con 0x7f5ba8150290 2024-09-19T07:35:38.962 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:38.962 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":124,"btime":"2024-09-19T07:35:37:340512+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":5612,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:35:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 124 2024-09-19T07:35:38.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.970+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3513872508 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b940678c0 msgr2=0x7f5b94069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.970+0000 7f5bae8b4640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b940678c0 0x7f5b94069d80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5b980011f0 tx=0x7f5b98007950 comp rx=0 tx=0).stop 2024-09-19T07:35:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.970+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3513872508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba8150290 msgr2=0x7f5ba80be2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.970+0000 7f5bae8b4640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba8150290 0x7f5ba80be2d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c019070 tx=0x7f5b9c019200 comp rx=0 tx=0).stop 2024-09-19T07:35:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.970+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3513872508 shutdown_connections 2024-09-19T07:35:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.970+0000 7f5bae8b4640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b940678c0 0x7f5b94069d80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.970+0000 7f5bae8b4640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ba8153eb0 0x7f5ba81499f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.970+0000 7f5bae8b4640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ba8152b20 0x7f5ba81eff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.970+0000 7f5bae8b4640 1 --2- 172.21.15.83:0/3513872508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba8150290 0x7f5ba80be2d0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.970+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3513872508 >> 172.21.15.83:0/3513872508 conn(0x7f5ba806bc80 msgr2=0x7f5ba8156930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.970+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3513872508 shutdown_connections 2024-09-19T07:35:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:38.970+0000 7f5bae8b4640 1 -- 172.21.15.83:0/3513872508 wait complete. 2024-09-19T07:35:38.981 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-09-19T07:35:38.981 INFO:tasks.mds_thrash.fs.[b]:waiting for 85 secs before thrashing 2024-09-19T07:35:39.496 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:39.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.630+0000 7ff215ad6640 1 Processor -- start 2024-09-19T07:35:39.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.630+0000 7ff215ad6640 1 -- start start 2024-09-19T07:35:39.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff215ad6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff210150290 0x7ff2100c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:39.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff215ad6640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff210152b20 0x7ff2100c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:39.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff215ad6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff210150670 con 0x7ff2100c45c0 2024-09-19T07:35:39.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff215ad6640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff2100716c0 con 0x7ff210152b20 2024-09-19T07:35:39.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff215ad6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff2100c3920 con 0x7ff210150290 2024-09-19T07:35:39.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff2100c45c0 0x7ff210153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36076/0 (socket says 172.21.15.83:36076) 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20ffff640 1 -- 172.21.15.83:0/3080602972 learned_addr learned my addr 172.21.15.83:0/3080602972 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20f7fe640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff210150290 0x7ff2100c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20effd640 1 --2- 172.21.15.83:0/3080602972 >> v2:172.21.15.184:3300/0 conn(0x7ff210152b20 0x7ff2100c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3594759594 0 0) 0x7ff210150670 con 0x7ff2100c45c0 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1fc003530 con 0x7ff2100c45c0 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 721270208 0 0) 0x7ff1fc003530 con 0x7ff2100c45c0 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff210150290 msgr2=0x7ff2100c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff210150290 0x7ff2100c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 >> v2:172.21.15.184:3300/0 conn(0x7ff210152b20 msgr2=0x7ff2100c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 --2- 172.21.15.83:0/3080602972 >> v2:172.21.15.184:3300/0 conn(0x7ff210152b20 0x7ff2100c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2100c2880 con 0x7ff2100c45c0 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7ff200001680 con 0x7ff2100c45c0 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 >> v1:172.21.15.83:6789/0 conn(0x7ff2100c45c0 legacy=0x7ff210153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:39.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1fc0039f0 0x7ff1fc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff210150290 0x7ff1fc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1fc004d50 0x7ff1fc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc00b9d0 con 0x7ff1fc0039f0 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc00bb50 con 0x7ff1fc004d50 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff1fc00bcd0 con 0x7ff210150290 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20f7fe640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1fc0039f0 0x7ff1fc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20ffff640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1fc004d50 0x7ff1fc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20effd640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff210150290 0x7ff1fc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20f7fe640 1 -- 172.21.15.83:0/3080602972 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff210150290 msgr2=0x7ff1fc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20f7fe640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff210150290 0x7ff1fc004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20f7fe640 1 -- 172.21.15.83:0/3080602972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1fc004d50 msgr2=0x7ff1fc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20f7fe640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1fc004d50 0x7ff1fc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:39.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20f7fe640 1 -- 172.21.15.83:0/3080602972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1fc003530 con 0x7ff1fc0039f0 2024-09-19T07:35:39.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.634+0000 7ff20ffff640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1fc004d50 0x7ff1fc0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:35:39.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff20f7fe640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1fc0039f0 0x7ff1fc003e10 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009ae0 tx=0x7ff1f801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aec67af2166dfa1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:39.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80b2070 con 0x7ff1fc0039f0 2024-09-19T07:35:39.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1f80a5550 con 0x7ff1fc0039f0 2024-09-19T07:35:39.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff20e7fc640 1 -- 172.21.15.83:0/3080602972 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80096b0 con 0x7ff1fc0039f0 2024-09-19T07:35:39.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff215ad6640 1 -- 172.21.15.83:0/3080602972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1fc0039f0 msgr2=0x7ff1fc003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:39.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff215ad6640 1 --2- 172.21.15.83:0/3080602972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1fc0039f0 0x7ff1fc003e10 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009ae0 tx=0x7ff1f801ca70 comp rx=0 tx=0).stop 2024-09-19T07:35:39.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff20ffff640 1 -- 172.21.15.83:0/3080602972 reap_dead start 2024-09-19T07:35:39.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff215ad6640 1 -- 172.21.15.83:0/3080602972 shutdown_connections 2024-09-19T07:35:39.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff215ad6640 1 -- 172.21.15.83:0/3080602972 >> 172.21.15.83:0/3080602972 conn(0x7ff21006bc80 msgr2=0x7ff21006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:39.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff215ad6640 1 -- 172.21.15.83:0/3080602972 shutdown_connections 2024-09-19T07:35:39.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff215ad6640 1 -- 172.21.15.83:0/3080602972 wait complete. 2024-09-19T07:35:39.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff215ad6640 1 Processor -- start 2024-09-19T07:35:39.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff215ad6640 1 -- start start 2024-09-19T07:35:39.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.638+0000 7ff215ad6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1fc0039f0 0x7ff2101ec2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:39.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff215ad6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2101f0260 0x7ff21015a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:39.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff215ad6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2101ecc70 0x7ff210151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:39.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff215ad6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2100c2400 con 0x7ff2101ecc70 2024-09-19T07:35:39.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff215ad6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff2100c3120 con 0x7ff2101f0260 2024-09-19T07:35:39.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff215ad6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff2100c2580 con 0x7ff1fc0039f0 2024-09-19T07:35:39.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2101ecc70 0x7ff210151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:39.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2101ecc70 0x7ff210151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33482/0 (socket says 172.21.15.83:33482) 2024-09-19T07:35:39.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20ffff640 1 -- 172.21.15.83:0/4219052742 learned_addr learned my addr 172.21.15.83:0/4219052742 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:39.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20f7fe640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1fc0039f0 0x7ff2101ec2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:39.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20effd640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2101f0260 0x7ff21015a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:39.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20ffff640 1 -- 172.21.15.83:0/4219052742 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1fc0039f0 msgr2=0x7ff2101ec2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:39.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20ffff640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1fc0039f0 0x7ff2101ec2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:39.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20ffff640 1 -- 172.21.15.83:0/4219052742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2101f0260 msgr2=0x7ff21015a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:39.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20ffff640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2101f0260 0x7ff21015a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:39.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20ffff640 1 -- 172.21.15.83:0/4219052742 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21015a740 con 0x7ff2101ecc70 2024-09-19T07:35:39.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20ffff640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2101ecc70 0x7ff210151f40 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ff21006f620 tx=0x7ff20000db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:39.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff207fff640 1 -- 172.21.15.83:0/4219052742 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2000072d0 con 0x7ff2101ecc70 2024-09-19T07:35:39.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff207fff640 1 -- 172.21.15.83:0/4219052742 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff200007470 con 0x7ff2101ecc70 2024-09-19T07:35:39.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff215ad6640 1 -- 172.21.15.83:0/4219052742 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2101f06e0 con 0x7ff2101ecc70 2024-09-19T07:35:39.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff207fff640 1 -- 172.21.15.83:0/4219052742 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2000047f0 con 0x7ff2101ecc70 2024-09-19T07:35:39.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff215ad6640 1 -- 172.21.15.83:0/4219052742 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2100ac810 con 0x7ff2101ecc70 2024-09-19T07:35:39.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff215ad6640 1 -- 172.21.15.83:0/4219052742 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2100aa650 con 0x7ff2101ecc70 2024-09-19T07:35:39.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff207fff640 1 -- 172.21.15.83:0/4219052742 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff20001b020 con 0x7ff2101ecc70 2024-09-19T07:35:39.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff207fff640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1f00678c0 0x7ff1f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:39.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff207fff640 1 -- 172.21.15.83:0/4219052742 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8615+0+0 (secure 0 0 0) 0x7ff200086130 con 0x7ff2101ecc70 2024-09-19T07:35:39.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20f7fe640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1f00678c0 0x7ff1f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:39.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.642+0000 7ff20f7fe640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1f00678c0 0x7ff1f0069d80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80af000 tx=0x7ff1f80046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:39.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.646+0000 7ff207fff640 1 -- 172.21.15.83:0/4219052742 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff21015a740 con 0x7ff2101ecc70 2024-09-19T07:35:39.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.846+0000 7ff215ad6640 1 -- 172.21.15.83:0/4219052742 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff21014d1d0 con 0x7ff2101ecc70 2024-09-19T07:35:39.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.846+0000 7ff207fff640 1 -- 172.21.15.83:0/4219052742 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+6543 (secure 0 0 0) 0x7ff2000561f0 con 0x7ff2101ecc70 2024-09-19T07:35:39.843 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:39.843 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":124,"btime":"2024-09-19T07:35:37:340512+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":5612,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:35:39.843 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 124 2024-09-19T07:35:39.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.850+0000 7ff215ad6640 1 -- 172.21.15.83:0/4219052742 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1f00678c0 msgr2=0x7ff1f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:39.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.850+0000 7ff215ad6640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1f00678c0 0x7ff1f0069d80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80af000 tx=0x7ff1f80046d0 comp rx=0 tx=0).stop 2024-09-19T07:35:39.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.850+0000 7ff215ad6640 1 -- 172.21.15.83:0/4219052742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2101ecc70 msgr2=0x7ff210151f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:39.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.850+0000 7ff215ad6640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2101ecc70 0x7ff210151f40 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ff21006f620 tx=0x7ff20000db80 comp rx=0 tx=0).stop 2024-09-19T07:35:39.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.850+0000 7ff215ad6640 1 -- 172.21.15.83:0/4219052742 shutdown_connections 2024-09-19T07:35:39.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.850+0000 7ff215ad6640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1f00678c0 0x7ff1f0069d80 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:39.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.850+0000 7ff215ad6640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2101ecc70 0x7ff210151f40 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:39.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.850+0000 7ff215ad6640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2101f0260 0x7ff21015a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:39.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.850+0000 7ff215ad6640 1 --2- 172.21.15.83:0/4219052742 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1fc0039f0 0x7ff2101ec2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:39.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.850+0000 7ff215ad6640 1 -- 172.21.15.83:0/4219052742 >> 172.21.15.83:0/4219052742 conn(0x7ff21006bc80 msgr2=0x7ff210152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:39.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.850+0000 7ff215ad6640 1 -- 172.21.15.83:0/4219052742 shutdown_connections 2024-09-19T07:35:39.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:39.850+0000 7ff215ad6640 1 -- 172.21.15.83:0/4219052742 wait complete. 2024-09-19T07:35:39.862 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:35:41.865 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:41.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97ff5b5640 1 Processor -- start 2024-09-19T07:35:41.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97ff5b5640 1 -- start start 2024-09-19T07:35:41.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97ff5b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f97f8150290 0x7f97f80c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:41.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97ff5b5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97f8152b20 0x7f97f80c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:41.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97ff5b5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f8150670 con 0x7f97f80c45c0 2024-09-19T07:35:41.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97ff5b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f97f80716c0 con 0x7f97f8150290 2024-09-19T07:35:41.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97ff5b5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f97f80c3920 con 0x7f97f8152b20 2024-09-19T07:35:41.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97fd351640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f97f8150290 0x7f97f80c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:41.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97fd351640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f97f8150290 0x7f97f80c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57146/0 (socket says 172.21.15.83:57146) 2024-09-19T07:35:41.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97fd351640 1 -- 172.21.15.83:0/2235009542 learned_addr learned my addr 172.21.15.83:0/2235009542 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:41.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97fcb50640 1 --2- 172.21.15.83:0/2235009542 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97f8152b20 0x7f97f80c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:41.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97fd351640 1 -- 172.21.15.83:0/2235009542 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97f8152b20 msgr2=0x7f97f80c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:41.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97fd351640 1 --2- 172.21.15.83:0/2235009542 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97f8152b20 0x7f97f80c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:41.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97fd351640 1 -- 172.21.15.83:0/2235009542 >> v1:172.21.15.184:6789/0 conn(0x7f97f80c45c0 legacy=0x7f97f8153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:41.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.002+0000 7f97fd351640 1 -- 172.21.15.83:0/2235009542 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f80c2880 con 0x7f97f8150290 2024-09-19T07:35:42.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97fd351640 1 --2- 172.21.15.83:0/2235009542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f8150290 0x7f97f80c1ff0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f97f40098a0 tx=0x7f97f401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce5d161b49574b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:42.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97effff640 1 -- 172.21.15.83:0/2235009542 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f40b1070 con 0x7f97f8150290 2024-09-19T07:35:42.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97effff640 1 -- 172.21.15.83:0/2235009542 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f40ac630 con 0x7f97f8150290 2024-09-19T07:35:42.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97effff640 1 -- 172.21.15.83:0/2235009542 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f40a8050 con 0x7f97f8150290 2024-09-19T07:35:42.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 -- 172.21.15.83:0/2235009542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f8150290 msgr2=0x7f97f80c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:42.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 --2- 172.21.15.83:0/2235009542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f8150290 0x7f97f80c1ff0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f97f40098a0 tx=0x7f97f401c500 comp rx=0 tx=0).stop 2024-09-19T07:35:42.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 -- 172.21.15.83:0/2235009542 shutdown_connections 2024-09-19T07:35:42.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 --2- 172.21.15.83:0/2235009542 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97f8152b20 0x7f97f80c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:42.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 --2- 172.21.15.83:0/2235009542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f8150290 0x7f97f80c1ff0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:42.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 -- 172.21.15.83:0/2235009542 >> 172.21.15.83:0/2235009542 conn(0x7f97f806bc80 msgr2=0x7f97f806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:42.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 -- 172.21.15.83:0/2235009542 shutdown_connections 2024-09-19T07:35:42.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 -- 172.21.15.83:0/2235009542 wait complete. 2024-09-19T07:35:42.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 Processor -- start 2024-09-19T07:35:42.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 -- start start 2024-09-19T07:35:42.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f97f80c45c0 0x7f97f814d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:42.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f8150290 0x7f97f8072210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:42.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97f8152b20 0x7f97f81495a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:42.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f97f80c2530 con 0x7f97f8150290 2024-09-19T07:35:42.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f97f80c3120 con 0x7f97f80c45c0 2024-09-19T07:35:42.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.006+0000 7f97ff5b5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f97f815db50 con 0x7f97f8152b20 2024-09-19T07:35:42.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fcb50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f8150290 0x7f97f8072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:42.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fcb50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f8150290 0x7f97f8072210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57162/0 (socket says 172.21.15.83:57162) 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fcb50640 1 -- 172.21.15.83:0/1627371315 learned_addr learned my addr 172.21.15.83:0/1627371315 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fd351640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f97f80c45c0 0x7f97f814d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fdb52640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97f8152b20 0x7f97f81495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fcb50640 1 -- 172.21.15.83:0/1627371315 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97f8152b20 msgr2=0x7f97f81495a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fcb50640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97f8152b20 0x7f97f81495a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fcb50640 1 -- 172.21.15.83:0/1627371315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f97f80c45c0 msgr2=0x7f97f814d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fcb50640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f97f80c45c0 0x7f97f814d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fcb50640 1 -- 172.21.15.83:0/1627371315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f8073fc0 con 0x7f97f8150290 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fcb50640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f8150290 0x7f97f8072210 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f97e800d600 tx=0x7f97e800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97edffb640 1 -- 172.21.15.83:0/1627371315 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8004240 con 0x7f97f8150290 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97edffb640 1 -- 172.21.15.83:0/1627371315 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e80043e0 con 0x7f97f8150290 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97ff5b5640 1 -- 172.21.15.83:0/1627371315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f814a2b0 con 0x7f97f8150290 2024-09-19T07:35:42.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97edffb640 1 -- 172.21.15.83:0/1627371315 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8003840 con 0x7f97f8150290 2024-09-19T07:35:42.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97ff5b5640 1 -- 172.21.15.83:0/1627371315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f8072750 con 0x7f97f8150290 2024-09-19T07:35:42.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97ff5b5640 1 -- 172.21.15.83:0/1627371315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f80741e0 con 0x7f97f8150290 2024-09-19T07:35:42.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97edffb640 1 -- 172.21.15.83:0/1627371315 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f97e80039e0 con 0x7f97f8150290 2024-09-19T07:35:42.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97edffb640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f97dc0678c0 0x7f97dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:42.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fd351640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f97dc0678c0 0x7f97dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:42.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97edffb640 1 -- 172.21.15.83:0/1627371315 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8615+0+0 (secure 0 0 0) 0x7f97e8086b70 con 0x7f97f8150290 2024-09-19T07:35:42.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.010+0000 7f97fd351640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f97dc0678c0 0x7f97dc069d80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f97f40ae000 tx=0x7f97f4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:42.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.014+0000 7f97edffb640 1 -- 172.21.15.83:0/1627371315 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f97f80741e0 con 0x7f97f8150290 2024-09-19T07:35:42.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.210+0000 7f97ff5b5640 1 -- 172.21.15.83:0/1627371315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97f8073db0 con 0x7f97f8150290 2024-09-19T07:35:42.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97edffb640 1 -- 172.21.15.83:0/1627371315 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+6543 (secure 0 0 0) 0x7f97e801c360 con 0x7f97f8150290 2024-09-19T07:35:42.207 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:42.208 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":124,"btime":"2024-09-19T07:35:37:340512+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":5612,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:34:11.258171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6738":{"gid":6738,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":12,"addr":"172.21.15.83:6835/1079027718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1079027718},{"type":"v1","addr":"172.21.15.83:6835","nonce":1079027718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6738,"qdb_cluster":[6738]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:35:42.208 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 124 2024-09-19T07:35:42.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97ff5b5640 1 -- 172.21.15.83:0/1627371315 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f97dc0678c0 msgr2=0x7f97dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:42.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97ff5b5640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f97dc0678c0 0x7f97dc069d80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f97f40ae000 tx=0x7f97f4008040 comp rx=0 tx=0).stop 2024-09-19T07:35:42.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97ff5b5640 1 -- 172.21.15.83:0/1627371315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f8150290 msgr2=0x7f97f8072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:42.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97ff5b5640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f8150290 0x7f97f8072210 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f97e800d600 tx=0x7f97e800dac0 comp rx=0 tx=0).stop 2024-09-19T07:35:42.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97ff5b5640 1 -- 172.21.15.83:0/1627371315 shutdown_connections 2024-09-19T07:35:42.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97ff5b5640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f97dc0678c0 0x7f97dc069d80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:42.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97ff5b5640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97f8152b20 0x7f97f81495a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:42.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97ff5b5640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f8150290 0x7f97f8072210 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:42.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97ff5b5640 1 --2- 172.21.15.83:0/1627371315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f97f80c45c0 0x7f97f814d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:42.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97ff5b5640 1 -- 172.21.15.83:0/1627371315 >> 172.21.15.83:0/1627371315 conn(0x7f97f806bc80 msgr2=0x7f97f806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:42.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97ff5b5640 1 -- 172.21.15.83:0/1627371315 shutdown_connections 2024-09-19T07:35:42.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:42.214+0000 7f97ff5b5640 1 -- 172.21.15.83:0/1627371315 wait complete. 2024-09-19T07:35:42.227 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:35:44.229 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:44.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.366+0000 7fd709668640 1 Processor -- start 2024-09-19T07:35:44.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.366+0000 7fd709668640 1 -- start start 2024-09-19T07:35:44.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd709668640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7040c6360 0x7fd704153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:44.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd709668640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd7040c9680 0x7fd704157800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:44.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd709668640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7040c5b90 con 0x7fd7040bee30 2024-09-19T07:35:44.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd709668640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd70415a050 con 0x7fd7040c9680 2024-09-19T07:35:44.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd709668640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd70406c950 con 0x7fd7040c6360 2024-09-19T07:35:44.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd7027fc640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fd7040bee30 0x7fd7041570f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:58026/0 (socket says 172.21.15.83:58026) 2024-09-19T07:35:44.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd7027fc640 1 -- 172.21.15.83:0/2926870229 learned_addr learned my addr 172.21.15.83:0/2926870229 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:44.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd7037fe640 1 --2- 172.21.15.83:0/2926870229 >> v2:172.21.15.184:3300/0 conn(0x7fd7040c9680 0x7fd704157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:44.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd702ffd640 1 --2- 172.21.15.83:0/2926870229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7040c6360 0x7fd704153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:44.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd702ffd640 1 -- 172.21.15.83:0/2926870229 >> v2:172.21.15.184:3300/0 conn(0x7fd7040c9680 msgr2=0x7fd704157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:44.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd702ffd640 1 --2- 172.21.15.83:0/2926870229 >> v2:172.21.15.184:3300/0 conn(0x7fd7040c9680 0x7fd704157800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd702ffd640 1 -- 172.21.15.83:0/2926870229 >> v1:172.21.15.184:6789/0 conn(0x7fd7040bee30 legacy=0x7fd7041570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:44.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.370+0000 7fd702ffd640 1 -- 172.21.15.83:0/2926870229 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70406c670 con 0x7fd7040c6360 2024-09-19T07:35:44.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.374+0000 7fd702ffd640 1 --2- 172.21.15.83:0/2926870229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7040c6360 0x7fd704153050 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd6f40097c0 tx=0x7fd6f401c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3649c48370e2d37f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:44.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.374+0000 7fd701ffb640 1 -- 172.21.15.83:0/2926870229 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40b0070 con 0x7fd7040c6360 2024-09-19T07:35:44.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.374+0000 7fd701ffb640 1 -- 172.21.15.83:0/2926870229 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6f40ab600 con 0x7fd7040c6360 2024-09-19T07:35:44.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.374+0000 7fd701ffb640 1 -- 172.21.15.83:0/2926870229 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40ab9d0 con 0x7fd7040c6360 2024-09-19T07:35:44.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 -- 172.21.15.83:0/2926870229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7040c6360 msgr2=0x7fd704153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:44.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 --2- 172.21.15.83:0/2926870229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7040c6360 0x7fd704153050 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd6f40097c0 tx=0x7fd6f401c410 comp rx=0 tx=0).stop 2024-09-19T07:35:44.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 -- 172.21.15.83:0/2926870229 shutdown_connections 2024-09-19T07:35:44.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 --2- 172.21.15.83:0/2926870229 >> v2:172.21.15.184:3300/0 conn(0x7fd7040c9680 0x7fd704157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 --2- 172.21.15.83:0/2926870229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7040c6360 0x7fd704153050 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 -- 172.21.15.83:0/2926870229 >> 172.21.15.83:0/2926870229 conn(0x7fd7040722a0 msgr2=0x7fd7040726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:44.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 -- 172.21.15.83:0/2926870229 shutdown_connections 2024-09-19T07:35:44.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 -- 172.21.15.83:0/2926870229 wait complete. 2024-09-19T07:35:44.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 Processor -- start 2024-09-19T07:35:44.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 -- start start 2024-09-19T07:35:44.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7040bee30 0x7fd704154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:44.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7040c6360 0x7fd70414dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:44.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7040c9680 0x7fd7041510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:44.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd7040c68c0 con 0x7fd7040bee30 2024-09-19T07:35:44.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd7040c6740 con 0x7fd7040c9680 2024-09-19T07:35:44.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd709668640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd704162da0 con 0x7fd7040c6360 2024-09-19T07:35:44.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd702ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7040bee30 0x7fd704154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:44.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd702ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7040bee30 0x7fd704154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57172/0 (socket says 172.21.15.83:57172) 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd702ffd640 1 -- 172.21.15.83:0/728709533 learned_addr learned my addr 172.21.15.83:0/728709533 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd7037fe640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7040c9680 0x7fd7041510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd7027fc640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7040c6360 0x7fd70414dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd702ffd640 1 -- 172.21.15.83:0/728709533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7040c6360 msgr2=0x7fd70414dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd702ffd640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7040c6360 0x7fd70414dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd702ffd640 1 -- 172.21.15.83:0/728709533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7040c9680 msgr2=0x7fd7041510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd702ffd640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7040c9680 0x7fd7041510a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd702ffd640 1 -- 172.21.15.83:0/728709533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7041f1db0 con 0x7fd7040bee30 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd702ffd640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7040bee30 0x7fd704154d40 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fd6f401a040 tx=0x7fd6f401ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.378+0000 7fd6eb7fe640 1 -- 172.21.15.83:0/728709533 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40b0050 con 0x7fd7040bee30 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.382+0000 7fd6eb7fe640 1 -- 172.21.15.83:0/728709533 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6f40bc070 con 0x7fd7040bee30 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.382+0000 7fd709668640 1 -- 172.21.15.83:0/728709533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7041f2090 con 0x7fd7040bee30 2024-09-19T07:35:44.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.382+0000 7fd6eb7fe640 1 -- 172.21.15.83:0/728709533 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40b5830 con 0x7fd7040bee30 2024-09-19T07:35:44.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.382+0000 7fd709668640 1 -- 172.21.15.83:0/728709533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd704155280 con 0x7fd7040bee30 2024-09-19T07:35:44.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.382+0000 7fd709668640 1 -- 172.21.15.83:0/728709533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd704153050 con 0x7fd7040bee30 2024-09-19T07:35:44.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.382+0000 7fd6eb7fe640 1 -- 172.21.15.83:0/728709533 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd6f40b59d0 con 0x7fd7040bee30 2024-09-19T07:35:44.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.382+0000 7fd6eb7fe640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd6f00678c0 0x7fd6f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:44.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.382+0000 7fd7027fc640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd6f00678c0 0x7fd6f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:44.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.382+0000 7fd6eb7fe640 1 -- 172.21.15.83:0/728709533 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7fd6f4120dd0 con 0x7fd7040bee30 2024-09-19T07:35:44.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.382+0000 7fd7027fc640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd6f00678c0 0x7fd6f0069d80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd70415f7a0 tx=0x7fd6f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:44.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.386+0000 7fd6eb7fe640 1 -- 172.21.15.83:0/728709533 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd6f4016090 con 0x7fd7040bee30 2024-09-19T07:35:44.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.582+0000 7fd709668640 1 -- 172.21.15.83:0/728709533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7040ab520 con 0x7fd7040bee30 2024-09-19T07:35:44.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd6eb7fe640 1 -- 172.21.15.83:0/728709533 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+5712 (secure 0 0 0) 0x7fd6f4122070 con 0x7fd7040bee30 2024-09-19T07:35:44.580 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:44.580 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":126,"btime":"2024-09-19T07:35:44:369058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:43.555492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:35:44.580 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 126 2024-09-19T07:35:44.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd709668640 1 -- 172.21.15.83:0/728709533 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd6f00678c0 msgr2=0x7fd6f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:44.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd709668640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd6f00678c0 0x7fd6f0069d80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd70415f7a0 tx=0x7fd6f8009040 comp rx=0 tx=0).stop 2024-09-19T07:35:44.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd709668640 1 -- 172.21.15.83:0/728709533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7040bee30 msgr2=0x7fd704154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:44.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd709668640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7040bee30 0x7fd704154d40 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fd6f401a040 tx=0x7fd6f401ac50 comp rx=0 tx=0).stop 2024-09-19T07:35:44.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd709668640 1 -- 172.21.15.83:0/728709533 shutdown_connections 2024-09-19T07:35:44.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd709668640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd6f00678c0 0x7fd6f0069d80 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd709668640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7040c9680 0x7fd7041510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd709668640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7040c6360 0x7fd70414dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd709668640 1 --2- 172.21.15.83:0/728709533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7040bee30 0x7fd704154d40 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd709668640 1 -- 172.21.15.83:0/728709533 >> 172.21.15.83:0/728709533 conn(0x7fd7040722a0 msgr2=0x7fd7040abd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:44.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd709668640 1 -- 172.21.15.83:0/728709533 shutdown_connections 2024-09-19T07:35:44.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.586+0000 7fd709668640 1 -- 172.21.15.83:0/728709533 wait complete. 2024-09-19T07:35:44.600 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-19T07:35:44.600 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:35:44.600 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:44.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.730+0000 7f5a590e7640 1 Processor -- start 2024-09-19T07:35:44.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.730+0000 7f5a590e7640 1 -- start start 2024-09-19T07:35:44.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.730+0000 7f5a590e7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5a540c6120 0x7f5a540c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:44.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.730+0000 7f5a590e7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5a54155d70 0x7f5a541575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:44.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.730+0000 7f5a590e7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a54156200 con 0x7f5a541592f0 2024-09-19T07:35:44.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.730+0000 7f5a590e7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5a540c9930 con 0x7f5a54155d70 2024-09-19T07:35:44.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.730+0000 7f5a590e7640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5a541564e0 con 0x7f5a540c6120 2024-09-19T07:35:44.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.730+0000 7f5a5259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5a54155d70 0x7f5a541575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:44.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.730+0000 7f5a52d9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5a540c6120 0x7f5a540c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:44.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.730+0000 7f5a5259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5a54155d70 0x7f5a541575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57184/0 (socket says 172.21.15.83:57184) 2024-09-19T07:35:44.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.730+0000 7f5a5259c640 1 -- 172.21.15.83:0/3765781975 learned_addr learned my addr 172.21.15.83:0/3765781975 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:44.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a5259c640 1 -- 172.21.15.83:0/3765781975 >> v2:172.21.15.184:3300/0 conn(0x7f5a540c6120 msgr2=0x7f5a540c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:44.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a5259c640 1 --2- 172.21.15.83:0/3765781975 >> v2:172.21.15.184:3300/0 conn(0x7f5a540c6120 0x7f5a540c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a5259c640 1 -- 172.21.15.83:0/3765781975 >> v1:172.21.15.184:6789/0 conn(0x7f5a541592f0 legacy=0x7f5a54157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:44.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a5259c640 1 -- 172.21.15.83:0/3765781975 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a541567c0 con 0x7f5a54155d70 2024-09-19T07:35:44.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a5259c640 1 --2- 172.21.15.83:0/3765781975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a54155d70 0x7f5a541575d0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f5a5408d540 tx=0x7f5a3c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b90326a01b68c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:44.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a51d9b640 1 -- 172.21.15.83:0/3765781975 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0b0070 con 0x7f5a54155d70 2024-09-19T07:35:44.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a51d9b640 1 -- 172.21.15.83:0/3765781975 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a3c0ab600 con 0x7f5a54155d70 2024-09-19T07:35:44.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a51d9b640 1 -- 172.21.15.83:0/3765781975 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0ab9d0 con 0x7f5a54155d70 2024-09-19T07:35:44.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a590e7640 1 -- 172.21.15.83:0/3765781975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a54155d70 msgr2=0x7f5a541575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:44.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a590e7640 1 --2- 172.21.15.83:0/3765781975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a54155d70 0x7f5a541575d0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f5a5408d540 tx=0x7f5a3c01c520 comp rx=0 tx=0).stop 2024-09-19T07:35:44.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a590e7640 1 -- 172.21.15.83:0/3765781975 shutdown_connections 2024-09-19T07:35:44.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a590e7640 1 --2- 172.21.15.83:0/3765781975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a54155d70 0x7f5a541575d0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a590e7640 1 --2- 172.21.15.83:0/3765781975 >> v2:172.21.15.184:3300/0 conn(0x7f5a540c6120 0x7f5a540c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a590e7640 1 -- 172.21.15.83:0/3765781975 >> 172.21.15.83:0/3765781975 conn(0x7f5a5406bc80 msgr2=0x7f5a5406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:44.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a590e7640 1 -- 172.21.15.83:0/3765781975 shutdown_connections 2024-09-19T07:35:44.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.734+0000 7f5a590e7640 1 -- 172.21.15.83:0/3765781975 wait complete. 2024-09-19T07:35:44.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a590e7640 1 Processor -- start 2024-09-19T07:35:44.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a590e7640 1 -- start start 2024-09-19T07:35:44.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a590e7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a540c6120 0x7f5a54154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:44.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a590e7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a541592f0 0x7f5a541ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:44.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a590e7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a541ef630 0x7f5a541f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:44.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a590e7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a54153860 con 0x7f5a541592f0 2024-09-19T07:35:44.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a590e7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a54162ea0 con 0x7f5a540c6120 2024-09-19T07:35:44.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a590e7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5a540cb460 con 0x7f5a541ef630 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a5259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a541592f0 0x7f5a541ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a5259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a541592f0 0x7f5a541ef0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57200/0 (socket says 172.21.15.83:57200) 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a5259c640 1 -- 172.21.15.83:0/482034963 learned_addr learned my addr 172.21.15.83:0/482034963 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a5359e640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a541ef630 0x7f5a541f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a52d9d640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a540c6120 0x7f5a54154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a5259c640 1 -- 172.21.15.83:0/482034963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a541ef630 msgr2=0x7f5a541f2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a5259c640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a541ef630 0x7f5a541f2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a5259c640 1 -- 172.21.15.83:0/482034963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a540c6120 msgr2=0x7f5a54154b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a5259c640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a540c6120 0x7f5a54154b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a5259c640 1 -- 172.21.15.83:0/482034963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a540cc050 con 0x7f5a541592f0 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a52d9d640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a540c6120 0x7f5a54154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a5259c640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a541592f0 0x7f5a541ef0f0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0ae040 tx=0x7f5a3c0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:44.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a4b7fe640 1 -- 172.21.15.83:0/482034963 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0bb070 con 0x7f5a541592f0 2024-09-19T07:35:44.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a4b7fe640 1 -- 172.21.15.83:0/482034963 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a3c0b6430 con 0x7f5a541592f0 2024-09-19T07:35:44.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a590e7640 1 -- 172.21.15.83:0/482034963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a540ca010 con 0x7f5a541592f0 2024-09-19T07:35:44.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a4b7fe640 1 -- 172.21.15.83:0/482034963 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0b0040 con 0x7f5a541592f0 2024-09-19T07:35:44.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a590e7640 1 -- 172.21.15.83:0/482034963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a540cbce0 con 0x7f5a541592f0 2024-09-19T07:35:44.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a590e7640 1 -- 172.21.15.83:0/482034963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a540c6500 con 0x7f5a541592f0 2024-09-19T07:35:44.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a4b7fe640 1 -- 172.21.15.83:0/482034963 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5a3c0be070 con 0x7f5a541592f0 2024-09-19T07:35:44.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a4b7fe640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a340678c0 0x7f5a34069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:44.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a4b7fe640 1 -- 172.21.15.83:0/482034963 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7f5a3c11ff00 con 0x7f5a541592f0 2024-09-19T07:35:44.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.738+0000 7f5a52d9d640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a340678c0 0x7f5a34069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:44.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.742+0000 7f5a52d9d640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a340678c0 0x7f5a34069d80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5a541f33d0 tx=0x7f5a44008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:44.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.746+0000 7f5a4b7fe640 1 -- 172.21.15.83:0/482034963 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5a3c016110 con 0x7f5a541592f0 2024-09-19T07:35:44.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.942+0000 7f5a590e7640 1 -- 172.21.15.83:0/482034963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a541eeda0 con 0x7f5a541592f0 2024-09-19T07:35:44.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.942+0000 7f5a4b7fe640 1 -- 172.21.15.83:0/482034963 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+5712 (secure 0 0 0) 0x7f5a3c0efed0 con 0x7f5a541592f0 2024-09-19T07:35:44.937 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:44.937 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":126,"btime":"2024-09-19T07:35:44:369058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:43.555492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:35:44.938 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 126 2024-09-19T07:35:44.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.946+0000 7f5a590e7640 1 -- 172.21.15.83:0/482034963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a340678c0 msgr2=0x7f5a34069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:44.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.946+0000 7f5a590e7640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a340678c0 0x7f5a34069d80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5a541f33d0 tx=0x7f5a44008040 comp rx=0 tx=0).stop 2024-09-19T07:35:44.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.946+0000 7f5a590e7640 1 -- 172.21.15.83:0/482034963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a541592f0 msgr2=0x7f5a541ef0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:44.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.946+0000 7f5a590e7640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a541592f0 0x7f5a541ef0f0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0ae040 tx=0x7f5a3c0192c0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.946+0000 7f5a590e7640 1 -- 172.21.15.83:0/482034963 shutdown_connections 2024-09-19T07:35:44.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.946+0000 7f5a590e7640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a340678c0 0x7f5a34069d80 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.946+0000 7f5a590e7640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a541ef630 0x7f5a541f2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.946+0000 7f5a590e7640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a541592f0 0x7f5a541ef0f0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.946+0000 7f5a590e7640 1 --2- 172.21.15.83:0/482034963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a540c6120 0x7f5a54154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:44.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.946+0000 7f5a590e7640 1 -- 172.21.15.83:0/482034963 >> 172.21.15.83:0/482034963 conn(0x7f5a5406bc80 msgr2=0x7f5a54159790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:44.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.946+0000 7f5a590e7640 1 -- 172.21.15.83:0/482034963 shutdown_connections 2024-09-19T07:35:44.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:44.946+0000 7f5a590e7640 1 -- 172.21.15.83:0/482034963 wait complete. 2024-09-19T07:35:44.957 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:35:44:369058+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 126, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 126, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5612": { "addr": "172.21.15.184:6835/1519298917", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 1519298917, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 1519298917, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5612, "incarnation": 125, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:35:43.555492+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5612 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 123, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6918": { "addr": "172.21.15.83:6837/3707393259", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 3707393259, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 3707393259, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6918, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:35:31.315245+0000", "qdb_cluster": [ 6918 ], "qdb_leader": 6918, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6918 } } } ], "standbys": [ { "addr": "172.21.15.184:6833/381468017", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 381468017, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 381468017, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 119, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5555, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:35:46.958 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:47.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0447dc8640 1 Processor -- start 2024-09-19T07:35:47.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0447dc8640 1 -- start start 2024-09-19T07:35:47.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0447dc8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0440153300 0x7f044006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:47.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0447dc8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04401593f0 con 0x7f0440155bb0 2024-09-19T07:35:47.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0447dc8640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04401596d0 con 0x7f044006d1d0 2024-09-19T07:35:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0447dc8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f04401591a0 con 0x7f0440153300 2024-09-19T07:35:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0446365640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0440155bb0 0x7f0440156060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46034/0 (socket says 172.21.15.83:46034) 2024-09-19T07:35:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0446365640 1 -- 172.21.15.83:0/3446611321 learned_addr learned my addr 172.21.15.83:0/3446611321 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0445b64640 1 --2- 172.21.15.83:0/3446611321 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0440153300 0x7f044006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0444b62640 1 -- 172.21.15.83:0/3446611321 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3715035621 0 0) 0x7f04401593f0 con 0x7f0440155bb0 2024-09-19T07:35:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0444b62640 1 -- 172.21.15.83:0/3446611321 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f042c003530 con 0x7f0440155bb0 2024-09-19T07:35:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0444b62640 1 -- 172.21.15.83:0/3446611321 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 174761932 0 0) 0x7f04401596d0 con 0x7f044006d1d0 2024-09-19T07:35:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0444b62640 1 -- 172.21.15.83:0/3446611321 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04401593f0 con 0x7f044006d1d0 2024-09-19T07:35:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0445b64640 1 -- 172.21.15.83:0/3446611321 >> v1:172.21.15.184:6789/0 conn(0x7f044006d1d0 legacy=0x7f04400c23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0445b64640 1 -- 172.21.15.83:0/3446611321 >> v1:172.21.15.83:6789/0 conn(0x7f0440155bb0 legacy=0x7f0440156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.098+0000 7f0445b64640 1 -- 172.21.15.83:0/3446611321 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04401599b0 con 0x7f0440153300 2024-09-19T07:35:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0445b64640 1 --2- 172.21.15.83:0/3446611321 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0440153300 0x7f044006cc90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f04300097f0 tx=0x7f043001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d9c53ee8e24f6b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0444b62640 1 -- 172.21.15.83:0/3446611321 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04300b0070 con 0x7f0440153300 2024-09-19T07:35:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0444b62640 1 -- 172.21.15.83:0/3446611321 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04300ab600 con 0x7f0440153300 2024-09-19T07:35:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0444b62640 1 -- 172.21.15.83:0/3446611321 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04300ab9b0 con 0x7f0440153300 2024-09-19T07:35:47.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0447dc8640 1 -- 172.21.15.83:0/3446611321 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0440153300 msgr2=0x7f044006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:47.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0447dc8640 1 --2- 172.21.15.83:0/3446611321 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0440153300 0x7f044006cc90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f04300097f0 tx=0x7f043001c3c0 comp rx=0 tx=0).stop 2024-09-19T07:35:47.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0447dc8640 1 -- 172.21.15.83:0/3446611321 shutdown_connections 2024-09-19T07:35:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0447dc8640 1 --2- 172.21.15.83:0/3446611321 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0440153300 0x7f044006cc90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0447dc8640 1 -- 172.21.15.83:0/3446611321 >> 172.21.15.83:0/3446611321 conn(0x7f04400c5090 msgr2=0x7f04400c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0447dc8640 1 -- 172.21.15.83:0/3446611321 shutdown_connections 2024-09-19T07:35:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0447dc8640 1 -- 172.21.15.83:0/3446611321 wait complete. 2024-09-19T07:35:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0447dc8640 1 Processor -- start 2024-09-19T07:35:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.102+0000 7f0447dc8640 1 -- start start 2024-09-19T07:35:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0447dc8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f044006d1d0 0x7f0440151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0447dc8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0440153300 0x7f0440152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0447dc8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0440155bb0 0x7f04401f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0447dc8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f044006f0a0 con 0x7f044006d1d0 2024-09-19T07:35:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0447dc8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f044006ef20 con 0x7f0440155bb0 2024-09-19T07:35:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0447dc8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f04401536e0 con 0x7f0440153300 2024-09-19T07:35:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0445b64640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f044006d1d0 0x7f0440151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0445b64640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f044006d1d0 0x7f0440151330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57204/0 (socket says 172.21.15.83:57204) 2024-09-19T07:35:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0445b64640 1 -- 172.21.15.83:0/696591020 learned_addr learned my addr 172.21.15.83:0/696591020 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0445363640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0440153300 0x7f0440152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0445b64640 1 -- 172.21.15.83:0/696591020 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0440153300 msgr2=0x7f0440152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0445b64640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0440153300 0x7f0440152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0445b64640 1 -- 172.21.15.83:0/696591020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0440155bb0 msgr2=0x7f04401f1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0445b64640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0440155bb0 0x7f04401f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0445b64640 1 -- 172.21.15.83:0/696591020 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044006f510 con 0x7f044006d1d0 2024-09-19T07:35:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0445b64640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f044006d1d0 0x7f0440151330 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f04300097f0 tx=0x7f043001ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f04367fc640 1 -- 172.21.15.83:0/696591020 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04300b0050 con 0x7f044006d1d0 2024-09-19T07:35:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0447dc8640 1 -- 172.21.15.83:0/696591020 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f044006f7f0 con 0x7f044006d1d0 2024-09-19T07:35:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f04367fc640 1 -- 172.21.15.83:0/696591020 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04300b1040 con 0x7f044006d1d0 2024-09-19T07:35:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f04367fc640 1 -- 172.21.15.83:0/696591020 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0430006e20 con 0x7f044006d1d0 2024-09-19T07:35:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0447dc8640 1 -- 172.21.15.83:0/696591020 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f044006fdb0 con 0x7f044006d1d0 2024-09-19T07:35:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0447dc8640 1 -- 172.21.15.83:0/696591020 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f044006cc90 con 0x7f044006d1d0 2024-09-19T07:35:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f04367fc640 1 -- 172.21.15.83:0/696591020 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f04300be070 con 0x7f044006d1d0 2024-09-19T07:35:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f04367fc640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f04240678f0 0x7f0424069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f04367fc640 1 -- 172.21.15.83:0/696591020 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7f043011ff00 con 0x7f044006d1d0 2024-09-19T07:35:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.106+0000 7f0445363640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f04240678f0 0x7f0424069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.110+0000 7f0445363640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f04240678f0 0x7f0424069db0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0440160a60 tx=0x7f043c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:47.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.114+0000 7f04367fc640 1 -- 172.21.15.83:0/696591020 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f04300160d0 con 0x7f044006d1d0 2024-09-19T07:35:47.348 INFO:tasks.ceph.mds.b.smithi184.stderr:2024-09-19T07:35:47.348+0000 7fc867954640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:35:47.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.370+0000 7f0447dc8640 1 -- 172.21.15.83:0/696591020 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0440154ca0 con 0x7f044006d1d0 2024-09-19T07:35:47.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.370+0000 7f04367fc640 1 -- 172.21.15.83:0/696591020 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) ==== 95+0+5716 (secure 0 0 0) 0x7f04300efed0 con 0x7f044006d1d0 2024-09-19T07:35:47.367 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:47.367 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":128,"btime":"2024-09-19T07:35:46:388836+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:35:47.367 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 128 2024-09-19T07:35:47.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.374+0000 7f0447dc8640 1 -- 172.21.15.83:0/696591020 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f04240678f0 msgr2=0x7f0424069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:47.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.374+0000 7f0447dc8640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f04240678f0 0x7f0424069db0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0440160a60 tx=0x7f043c009040 comp rx=0 tx=0).stop 2024-09-19T07:35:47.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.374+0000 7f0447dc8640 1 -- 172.21.15.83:0/696591020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f044006d1d0 msgr2=0x7f0440151330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:47.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.374+0000 7f0447dc8640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f044006d1d0 0x7f0440151330 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f04300097f0 tx=0x7f043001ac00 comp rx=0 tx=0).stop 2024-09-19T07:35:47.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.374+0000 7f0447dc8640 1 -- 172.21.15.83:0/696591020 shutdown_connections 2024-09-19T07:35:47.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.374+0000 7f0447dc8640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f04240678f0 0x7f0424069db0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:47.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.374+0000 7f0447dc8640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0440155bb0 0x7f04401f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:47.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.374+0000 7f0447dc8640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0440153300 0x7f0440152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:47.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.374+0000 7f0447dc8640 1 --2- 172.21.15.83:0/696591020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f044006d1d0 0x7f0440151330 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:47.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.374+0000 7f0447dc8640 1 -- 172.21.15.83:0/696591020 >> 172.21.15.83:0/696591020 conn(0x7f04400c5090 msgr2=0x7f0440154040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:47.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.374+0000 7f0447dc8640 1 -- 172.21.15.83:0/696591020 shutdown_connections 2024-09-19T07:35:47.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:47.374+0000 7f0447dc8640 1 -- 172.21.15.83:0/696591020 wait complete. 2024-09-19T07:35:47.389 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=6738 2024-09-19T07:35:47.389 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.a 2024-09-19T07:35:50.390 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-19T07:35:50.390 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:35:50.391 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:35:50.394 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:35:50.395 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:35:50.477 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:35:50.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.534+0000 7f909e80d640 1 Processor -- start 2024-09-19T07:35:50.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.534+0000 7f909e80d640 1 -- start start 2024-09-19T07:35:50.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.534+0000 7f909e80d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f90980aa4a0 0x7f9098157720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:50.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f909e80d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9098073d90 con 0x7f90980c3ab0 2024-09-19T07:35:50.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f909e80d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9098153b30 con 0x7f909814dbe0 2024-09-19T07:35:50.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f909e80d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f9098154700 con 0x7f90980aa4a0 2024-09-19T07:35:50.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f90977fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f90980c3ab0 0x7f9098157010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:32990/0 (socket says 172.21.15.83:32990) 2024-09-19T07:35:50.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f90977fe640 1 -- 172.21.15.83:0/2261044004 learned_addr learned my addr 172.21.15.83:0/2261044004 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:35:50.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f909cdaa640 1 --2- 172.21.15.83:0/2261044004 >> v2:172.21.15.184:3300/0 conn(0x7f90980aa4a0 0x7f9098157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:50.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3626665441 0 0) 0x7f9098073d90 con 0x7f90980c3ab0 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9080003530 con 0x7f90980c3ab0 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4243022766 0 0) 0x7f9080003530 con 0x7f90980c3ab0 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 >> v2:172.21.15.184:3300/0 conn(0x7f90980aa4a0 msgr2=0x7f9098157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 --2- 172.21.15.83:0/2261044004 >> v2:172.21.15.184:3300/0 conn(0x7f90980aa4a0 0x7f9098157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 >> v1:172.21.15.184:6789/0 conn(0x7f909814dbe0 legacy=0x7f90980c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9098153dd0 con 0x7f90980c3ab0 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f908c001680 con 0x7f90980c3ab0 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 >> v1:172.21.15.83:6789/0 conn(0x7f90980c3ab0 legacy=0x7f9098157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 --2- 172.21.15.83:0/2261044004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9080003a10 0x7f9080003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 --2- 172.21.15.83:0/2261044004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9080004810 0x7f9080004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 --2- 172.21.15.83:0/2261044004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9080005240 0x7f908000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9098154700 con 0x7f9080005240 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f908000c1a0 con 0x7f9080004810 2024-09-19T07:35:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f908000c3a0 con 0x7f9080003a10 2024-09-19T07:35:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f909cdaa640 1 --2- 172.21.15.83:0/2261044004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9080005240 0x7f908000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f90977fe640 1 --2- 172.21.15.83:0/2261044004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9080004810 0x7f9080004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9097fff640 1 --2- 172.21.15.83:0/2261044004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9080003a10 0x7f9080003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f909cdaa640 1 -- 172.21.15.83:0/2261044004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9080003a10 msgr2=0x7f9080003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f909cdaa640 1 --2- 172.21.15.83:0/2261044004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9080003a10 0x7f9080003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f909cdaa640 1 -- 172.21.15.83:0/2261044004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9080004810 msgr2=0x7f9080004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:35:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f909cdaa640 1 --2- 172.21.15.83:0/2261044004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9080004810 0x7f9080004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f909cdaa640 1 -- 172.21.15.83:0/2261044004 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9080003530 con 0x7f9080005240 2024-09-19T07:35:50.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f9097fff640 1 --2- 172.21.15.83:0/2261044004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9080003a10 0x7f9080003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:35:50.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f909cdaa640 1 -- 172.21.15.83:0/2261044004 reap_dead start 2024-09-19T07:35:50.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.538+0000 7f909cdaa640 1 --2- 172.21.15.83:0/2261044004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9080005240 0x7f908000b9d0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f9090009fc0 tx=0x7f909001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e108bdb08617fe server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:50.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090009c10 con 0x7f9080005240 2024-09-19T07:35:50.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90900a5c50 con 0x7f9080005240 2024-09-19T07:35:50.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f9096ffd640 1 -- 172.21.15.83:0/2261044004 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c007e60 con 0x7f9080005240 2024-09-19T07:35:50.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 -- 172.21.15.83:0/2261044004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9080005240 msgr2=0x7f908000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 --2- 172.21.15.83:0/2261044004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9080005240 0x7f908000b9d0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f9090009fc0 tx=0x7f909001cc00 comp rx=0 tx=0).stop 2024-09-19T07:35:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 -- 172.21.15.83:0/2261044004 shutdown_connections 2024-09-19T07:35:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 --2- 172.21.15.83:0/2261044004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9080005240 0x7f908000b9d0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 -- 172.21.15.83:0/2261044004 >> 172.21.15.83:0/2261044004 conn(0x7f90980c95a0 msgr2=0x7f90980c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 -- 172.21.15.83:0/2261044004 shutdown_connections 2024-09-19T07:35:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 -- 172.21.15.83:0/2261044004 wait complete. 2024-09-19T07:35:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 Processor -- start 2024-09-19T07:35:50.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 -- start start 2024-09-19T07:35:50.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9080005240 0x7f90981dd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:50.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90981ddde0 0x7f9098146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:50.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90981472a0 0x7f90980c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:50.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f90980741a0 con 0x7f90981ddde0 2024-09-19T07:35:50.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9098154390 con 0x7f9080005240 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909e80d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f90980c3fa0 con 0x7f90981472a0 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f90977fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90981ddde0 0x7f9098146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f90977fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90981ddde0 0x7f9098146d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53332/0 (socket says 172.21.15.83:53332) 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f90977fe640 1 -- 172.21.15.83:0/1253088443 learned_addr learned my addr 172.21.15.83:0/1253088443 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f9097fff640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9080005240 0x7f90981dd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f909cdaa640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90981472a0 0x7f90980c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f90977fe640 1 -- 172.21.15.83:0/1253088443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90981472a0 msgr2=0x7f90980c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f90977fe640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90981472a0 0x7f90980c7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f90977fe640 1 -- 172.21.15.83:0/1253088443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9080005240 msgr2=0x7f90981dd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f90977fe640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9080005240 0x7f90981dd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f90977fe640 1 -- 172.21.15.83:0/1253088443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90981de240 con 0x7f90981ddde0 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.542+0000 7f90977fe640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90981ddde0 0x7f9098146d60 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f908c001030 tx=0x7f908c0126f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.546+0000 7f9094ff9640 1 -- 172.21.15.83:0/1253088443 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c01c070 con 0x7f90981ddde0 2024-09-19T07:35:50.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.546+0000 7f9094ff9640 1 -- 172.21.15.83:0/1253088443 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f908c014770 con 0x7f90981ddde0 2024-09-19T07:35:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.546+0000 7f909e80d640 1 -- 172.21.15.83:0/1253088443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90981e9f20 con 0x7f90981ddde0 2024-09-19T07:35:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.546+0000 7f9094ff9640 1 -- 172.21.15.83:0/1253088443 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c014b60 con 0x7f90981ddde0 2024-09-19T07:35:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.546+0000 7f909e80d640 1 -- 172.21.15.83:0/1253088443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90981dfab0 con 0x7f90981ddde0 2024-09-19T07:35:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.546+0000 7f909e80d640 1 -- 172.21.15.83:0/1253088443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90981df8a0 con 0x7f90981ddde0 2024-09-19T07:35:50.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.546+0000 7f9094ff9640 1 -- 172.21.15.83:0/1253088443 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f908c01f070 con 0x7f90981ddde0 2024-09-19T07:35:50.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.546+0000 7f9094ff9640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90840678f0 0x7f9084069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:35:50.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.546+0000 7f9097fff640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90840678f0 0x7f9084069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:35:50.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.546+0000 7f9094ff9640 1 -- 172.21.15.83:0/1253088443 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7f908c0893c0 con 0x7f90981ddde0 2024-09-19T07:35:50.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.546+0000 7f9097fff640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90840678f0 0x7f9084069db0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9088000c10 tx=0x7f9088009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:35:50.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.554+0000 7f9094ff9640 1 -- 172.21.15.83:0/1253088443 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f908c058df0 con 0x7f90981ddde0 2024-09-19T07:35:50.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.790+0000 7f909e80d640 1 -- 172.21.15.83:0/1253088443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90980c4330 con 0x7f90981ddde0 2024-09-19T07:35:50.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f9094ff9640 1 -- 172.21.15.83:0/1253088443 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7f908c088020 con 0x7f90981ddde0 2024-09-19T07:35:50.788 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:35:50.788 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:35:50.788 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:35:50.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f909e80d640 1 -- 172.21.15.83:0/1253088443 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90840678f0 msgr2=0x7f9084069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:50.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f909e80d640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90840678f0 0x7f9084069db0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9088000c10 tx=0x7f9088009040 comp rx=0 tx=0).stop 2024-09-19T07:35:50.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f909e80d640 1 -- 172.21.15.83:0/1253088443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90981ddde0 msgr2=0x7f9098146d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:35:50.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f909e80d640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90981ddde0 0x7f9098146d60 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f908c001030 tx=0x7f908c0126f0 comp rx=0 tx=0).stop 2024-09-19T07:35:50.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f909e80d640 1 -- 172.21.15.83:0/1253088443 shutdown_connections 2024-09-19T07:35:50.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f909e80d640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90840678f0 0x7f9084069db0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:50.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f909e80d640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90981472a0 0x7f90980c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:50.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f909e80d640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90981ddde0 0x7f9098146d60 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:50.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f909e80d640 1 --2- 172.21.15.83:0/1253088443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9080005240 0x7f90981dd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:35:50.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f909e80d640 1 -- 172.21.15.83:0/1253088443 >> 172.21.15.83:0/1253088443 conn(0x7f90980c95a0 msgr2=0x7f909814beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:35:50.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f909e80d640 1 -- 172.21.15.83:0/1253088443 shutdown_connections 2024-09-19T07:35:50.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:35:50.794+0000 7f909e80d640 1 -- 172.21.15.83:0/1253088443 wait complete. 2024-09-19T07:35:50.808 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-19T07:35:50.808 INFO:tasks.mds_thrash.fs.[a]:waiting for 74 secs before thrashing 2024-09-19T07:36:04.370 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:36:04.374 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:36:34.414 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:36:34.417 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:37:03.983 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.124+0000 7efc58d22640 1 Processor -- start 2024-09-19T07:37:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.124+0000 7efc58d22640 1 -- start start 2024-09-19T07:37:04.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc58d22640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc540c23d0 0x7efc540c27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc58d22640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efc54155bb0 0x7efc54156040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc58d22640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc541593f0 con 0x7efc54153300 2024-09-19T07:37:04.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc58d22640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7efc541591a0 con 0x7efc54155bb0 2024-09-19T07:37:04.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc58d22640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efc541596d0 con 0x7efc540c23d0 2024-09-19T07:37:04.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc52d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efc54155bb0 0x7efc54156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc52d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efc54155bb0 0x7efc54156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45136/0 (socket says 172.21.15.83:45136) 2024-09-19T07:37:04.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc52d9d640 1 -- 172.21.15.83:0/3562712342 learned_addr learned my addr 172.21.15.83:0/3562712342 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:04.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc51d9b640 1 --2- 172.21.15.83:0/3562712342 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc540c23d0 0x7efc540c27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc52d9d640 1 -- 172.21.15.83:0/3562712342 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc540c23d0 msgr2=0x7efc540c27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc52d9d640 1 --2- 172.21.15.83:0/3562712342 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc540c23d0 0x7efc540c27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc52d9d640 1 -- 172.21.15.83:0/3562712342 >> v1:172.21.15.184:6789/0 conn(0x7efc54153300 legacy=0x7efc5406cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc52d9d640 1 -- 172.21.15.83:0/3562712342 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc541599b0 con 0x7efc54155bb0 2024-09-19T07:37:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc52d9d640 1 --2- 172.21.15.83:0/3562712342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc54155bb0 0x7efc54156040 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7efc4c009e50 tx=0x7efc4c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a783c122d80383f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc5159a640 1 -- 172.21.15.83:0/3562712342 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4c0a56e0 con 0x7efc54155bb0 2024-09-19T07:37:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc5159a640 1 -- 172.21.15.83:0/3562712342 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc4c0a5880 con 0x7efc54155bb0 2024-09-19T07:37:04.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.128+0000 7efc5159a640 1 -- 172.21.15.83:0/3562712342 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4c0a5c50 con 0x7efc54155bb0 2024-09-19T07:37:04.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 -- 172.21.15.83:0/3562712342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc54155bb0 msgr2=0x7efc54156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 --2- 172.21.15.83:0/3562712342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc54155bb0 0x7efc54156040 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7efc4c009e50 tx=0x7efc4c01c910 comp rx=0 tx=0).stop 2024-09-19T07:37:04.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 -- 172.21.15.83:0/3562712342 shutdown_connections 2024-09-19T07:37:04.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 --2- 172.21.15.83:0/3562712342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc54155bb0 0x7efc54156040 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 --2- 172.21.15.83:0/3562712342 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc540c23d0 0x7efc540c27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 -- 172.21.15.83:0/3562712342 >> 172.21.15.83:0/3562712342 conn(0x7efc540c5090 msgr2=0x7efc540c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:04.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 -- 172.21.15.83:0/3562712342 shutdown_connections 2024-09-19T07:37:04.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 -- 172.21.15.83:0/3562712342 wait complete. 2024-09-19T07:37:04.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 Processor -- start 2024-09-19T07:37:04.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 -- start start 2024-09-19T07:37:04.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc540c23d0 0x7efc5414c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc54153300 0x7efc541eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc54155bb0 0x7efc541ef110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efc54153860 con 0x7efc540c23d0 2024-09-19T07:37:04.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efc541536e0 con 0x7efc54153300 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efc54162d50 con 0x7efc54155bb0 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc5259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc540c23d0 0x7efc5414c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc5259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc540c23d0 0x7efc5414c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45138/0 (socket says 172.21.15.83:45138) 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc5259c640 1 -- 172.21.15.83:0/3717404968 learned_addr learned my addr 172.21.15.83:0/3717404968 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc51d9b640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc54153300 0x7efc541eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc52d9d640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc54155bb0 0x7efc541ef110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc5259c640 1 -- 172.21.15.83:0/3717404968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc54155bb0 msgr2=0x7efc541ef110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc5259c640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc54155bb0 0x7efc541ef110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc5259c640 1 -- 172.21.15.83:0/3717404968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc54153300 msgr2=0x7efc541eebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc5259c640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc54153300 0x7efc541eebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc5259c640 1 -- 172.21.15.83:0/3717404968 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc540c6eb0 con 0x7efc540c23d0 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc5259c640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc540c23d0 0x7efc5414c4d0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7efc4400c500 tx=0x7efc4400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:04.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc42ffd640 1 -- 172.21.15.83:0/3717404968 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc44016020 con 0x7efc540c23d0 2024-09-19T07:37:04.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc42ffd640 1 -- 172.21.15.83:0/3717404968 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc44005220 con 0x7efc540c23d0 2024-09-19T07:37:04.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 -- 172.21.15.83:0/3717404968 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc540c7190 con 0x7efc540c23d0 2024-09-19T07:37:04.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc42ffd640 1 -- 172.21.15.83:0/3717404968 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc44007030 con 0x7efc540c23d0 2024-09-19T07:37:04.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.132+0000 7efc58d22640 1 -- 172.21.15.83:0/3717404968 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc540c7750 con 0x7efc540c23d0 2024-09-19T07:37:04.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.136+0000 7efc58d22640 1 -- 172.21.15.83:0/3717404968 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc540c27f0 con 0x7efc540c23d0 2024-09-19T07:37:04.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.136+0000 7efc42ffd640 1 -- 172.21.15.83:0/3717404968 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efc440071f0 con 0x7efc540c23d0 2024-09-19T07:37:04.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.136+0000 7efc42ffd640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc340678c0 0x7efc34069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.136+0000 7efc42ffd640 1 -- 172.21.15.83:0/3717404968 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7efc440862b0 con 0x7efc540c23d0 2024-09-19T07:37:04.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.136+0000 7efc51d9b640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc340678c0 0x7efc34069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.136+0000 7efc51d9b640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc340678c0 0x7efc34069d80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7efc540c8010 tx=0x7efc48008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:04.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.144+0000 7efc42ffd640 1 -- 172.21.15.83:0/3717404968 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7efc44020020 con 0x7efc540c23d0 2024-09-19T07:37:04.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.392+0000 7efc58d22640 1 -- 172.21.15.83:0/3717404968 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc540c3670 con 0x7efc540c23d0 2024-09-19T07:37:04.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.392+0000 7efc42ffd640 1 -- 172.21.15.83:0/3717404968 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7efc44056200 con 0x7efc540c23d0 2024-09-19T07:37:04.391 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:04.391 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:04.391 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:04.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.396+0000 7efc58d22640 1 -- 172.21.15.83:0/3717404968 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc340678c0 msgr2=0x7efc34069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.396+0000 7efc58d22640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc340678c0 0x7efc34069d80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7efc540c8010 tx=0x7efc48008040 comp rx=0 tx=0).stop 2024-09-19T07:37:04.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.396+0000 7efc58d22640 1 -- 172.21.15.83:0/3717404968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc540c23d0 msgr2=0x7efc5414c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.396+0000 7efc58d22640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc540c23d0 0x7efc5414c4d0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7efc4400c500 tx=0x7efc4400c9c0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.396+0000 7efc58d22640 1 -- 172.21.15.83:0/3717404968 shutdown_connections 2024-09-19T07:37:04.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.396+0000 7efc58d22640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc340678c0 0x7efc34069d80 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.396+0000 7efc58d22640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc54155bb0 0x7efc541ef110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.396+0000 7efc58d22640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc54153300 0x7efc541eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.396+0000 7efc58d22640 1 --2- 172.21.15.83:0/3717404968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc540c23d0 0x7efc5414c4d0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.396+0000 7efc58d22640 1 -- 172.21.15.83:0/3717404968 >> 172.21.15.83:0/3717404968 conn(0x7efc540c5090 msgr2=0x7efc540c2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:04.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.396+0000 7efc58d22640 1 -- 172.21.15.83:0/3717404968 shutdown_connections 2024-09-19T07:37:04.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.396+0000 7efc58d22640 1 -- 172.21.15.83:0/3717404968 wait complete. 2024-09-19T07:37:04.410 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-19T07:37:04.410 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:37:04.410 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:37:04.421 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:37:04.421 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:04.461 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:37:04.472 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:37:04.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f5172606640 1 Processor -- start 2024-09-19T07:37:04.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f5172606640 1 -- start start 2024-09-19T07:37:04.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f5172606640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f516c0c6360 0x7f516c153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f5172606640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f516c0c9680 0x7f516c157800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f5172606640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f516c0c5b90 con 0x7f516c0bee30 2024-09-19T07:37:04.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f5172606640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f516c15a050 con 0x7f516c0c9680 2024-09-19T07:37:04.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f5172606640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f516c06c950 con 0x7f516c0c6360 2024-09-19T07:37:04.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f516bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f516c0c6360 0x7f516c153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f5170ba3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f516c0c9680 0x7f516c157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f516bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f516c0c6360 0x7f516c153050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:53268/0 (socket says 172.21.15.83:53268) 2024-09-19T07:37:04.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f516b7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f516c0bee30 0x7f516c1570f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:39714/0 (socket says 172.21.15.83:39714) 2024-09-19T07:37:04.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f516bfff640 1 -- 172.21.15.83:0/3978248130 learned_addr learned my addr 172.21.15.83:0/3978248130 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:04.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f5170ba3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f516c0c9680 0x7f516c157800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:60078/0 (socket says 172.21.15.83:60078) 2024-09-19T07:37:04.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f516bfff640 1 -- 172.21.15.83:0/3978248130 >> v2:172.21.15.184:3300/0 conn(0x7f516c0c9680 msgr2=0x7f516c157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:04.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f516bfff640 1 --2- 172.21.15.83:0/3978248130 >> v2:172.21.15.184:3300/0 conn(0x7f516c0c9680 0x7f516c157800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f516bfff640 1 -- 172.21.15.83:0/3978248130 >> v1:172.21.15.184:6789/0 conn(0x7f516c0bee30 legacy=0x7f516c1570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f516bfff640 1 -- 172.21.15.83:0/3978248130 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516c06c670 con 0x7f516c0c6360 2024-09-19T07:37:04.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.600+0000 7f5170ba3640 1 --2- 172.21.15.83:0/3978248130 >> v2:172.21.15.184:3300/0 conn(0x7f516c0c9680 0x7f516c157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:37:04.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f516bfff640 1 --2- 172.21.15.83:0/3978248130 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f516c0c6360 0x7f516c153050 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f515c009870 tx=0x7f515c01c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8724c8bce399573 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:04.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f516affd640 1 -- 172.21.15.83:0/3978248130 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515c0b1070 con 0x7f516c0c6360 2024-09-19T07:37:04.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f516affd640 1 -- 172.21.15.83:0/3978248130 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f515c0ac620 con 0x7f516c0c6360 2024-09-19T07:37:04.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f516affd640 1 -- 172.21.15.83:0/3978248130 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515c0ac9f0 con 0x7f516c0c6360 2024-09-19T07:37:04.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f5172606640 1 -- 172.21.15.83:0/3978248130 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f516c0c6360 msgr2=0x7f516c153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f5172606640 1 --2- 172.21.15.83:0/3978248130 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f516c0c6360 0x7f516c153050 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f515c009870 tx=0x7f515c01c4d0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f5172606640 1 -- 172.21.15.83:0/3978248130 shutdown_connections 2024-09-19T07:37:04.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f5172606640 1 --2- 172.21.15.83:0/3978248130 >> v2:172.21.15.184:3300/0 conn(0x7f516c0c9680 0x7f516c157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f5172606640 1 --2- 172.21.15.83:0/3978248130 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f516c0c6360 0x7f516c153050 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f5172606640 1 -- 172.21.15.83:0/3978248130 >> 172.21.15.83:0/3978248130 conn(0x7f516c0722a0 msgr2=0x7f516c0726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:04.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f5172606640 1 -- 172.21.15.83:0/3978248130 shutdown_connections 2024-09-19T07:37:04.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f5172606640 1 -- 172.21.15.83:0/3978248130 wait complete. 2024-09-19T07:37:04.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f5172606640 1 Processor -- start 2024-09-19T07:37:04.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.604+0000 7f5172606640 1 -- start start 2024-09-19T07:37:04.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5172606640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f516c0bee30 0x7f516c0c2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5172606640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f516c0c6360 0x7f516c1f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5172606640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f516c0c9680 0x7f516c1f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5172606640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f516c0c68c0 con 0x7f516c0c6360 2024-09-19T07:37:04.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5172606640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f516c0c6740 con 0x7f516c0c9680 2024-09-19T07:37:04.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5172606640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f516c162da0 con 0x7f516c0bee30 2024-09-19T07:37:04.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f516b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f516c0c6360 0x7f516c1f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f516b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f516c0c6360 0x7f516c1f2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45148/0 (socket says 172.21.15.83:45148) 2024-09-19T07:37:04.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f516b7fe640 1 -- 172.21.15.83:0/4285509466 learned_addr learned my addr 172.21.15.83:0/4285509466 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:04.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f516bfff640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f516c0bee30 0x7f516c0c2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5170ba3640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f516c0c9680 0x7f516c1f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f516b7fe640 1 -- 172.21.15.83:0/4285509466 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f516c0bee30 msgr2=0x7f516c0c2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f516b7fe640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f516c0bee30 0x7f516c0c2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f516b7fe640 1 -- 172.21.15.83:0/4285509466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f516c0c9680 msgr2=0x7f516c1f29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f516b7fe640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f516c0c9680 0x7f516c1f29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f516b7fe640 1 -- 172.21.15.83:0/4285509466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516c14cc30 con 0x7f516c0c6360 2024-09-19T07:37:04.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f516b7fe640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f516c0c6360 0x7f516c1f2470 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f516000c310 tx=0x7f516000c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:04.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5168ff9640 1 -- 172.21.15.83:0/4285509466 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51600074e0 con 0x7f516c0c6360 2024-09-19T07:37:04.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5168ff9640 1 -- 172.21.15.83:0/4285509466 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5160007680 con 0x7f516c0c6360 2024-09-19T07:37:04.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5172606640 1 -- 172.21.15.83:0/4285509466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f516c14cf10 con 0x7f516c0c6360 2024-09-19T07:37:04.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5172606640 1 -- 172.21.15.83:0/4285509466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f516c0c0040 con 0x7f516c0c6360 2024-09-19T07:37:04.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5168ff9640 1 -- 172.21.15.83:0/4285509466 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5160004a30 con 0x7f516c0c6360 2024-09-19T07:37:04.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5172606640 1 -- 172.21.15.83:0/4285509466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f514c0052a0 con 0x7f516c0c6360 2024-09-19T07:37:04.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5168ff9640 1 -- 172.21.15.83:0/4285509466 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f516001a020 con 0x7f516c0c6360 2024-09-19T07:37:04.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5168ff9640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f51580678c0 0x7f5158069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f5168ff9640 1 -- 172.21.15.83:0/4285509466 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7f5160085e30 con 0x7f516c0c6360 2024-09-19T07:37:04.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.608+0000 7f516bfff640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f51580678c0 0x7f5158069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.612+0000 7f516bfff640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f51580678c0 0x7f5158069d80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f515c0ae000 tx=0x7f515c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:04.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.616+0000 7f5168ff9640 1 -- 172.21.15.83:0/4285509466 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5160055730 con 0x7f516c0c6360 2024-09-19T07:37:04.809 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:04.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.912+0000 7f5172606640 1 -- 172.21.15.83:0/4285509466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f514c005590 con 0x7f516c0c6360 2024-09-19T07:37:04.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.912+0000 7f5168ff9640 1 -- 172.21.15.83:0/4285509466 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7f5160059a30 con 0x7f516c0c6360 2024-09-19T07:37:04.910 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:04.910 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:04.911 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:04.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.916+0000 7f5172606640 1 -- 172.21.15.83:0/4285509466 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f51580678c0 msgr2=0x7f5158069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.916+0000 7f5172606640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f51580678c0 0x7f5158069d80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f515c0ae000 tx=0x7f515c008040 comp rx=0 tx=0).stop 2024-09-19T07:37:04.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.916+0000 7f5172606640 1 -- 172.21.15.83:0/4285509466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f516c0c6360 msgr2=0x7f516c1f2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.916+0000 7f5172606640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f516c0c6360 0x7f516c1f2470 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f516000c310 tx=0x7f516000c7d0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.916+0000 7f5172606640 1 -- 172.21.15.83:0/4285509466 shutdown_connections 2024-09-19T07:37:04.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.916+0000 7f5172606640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f51580678c0 0x7f5158069d80 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.916+0000 7f5172606640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f516c0c9680 0x7f516c1f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.916+0000 7f5172606640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f516c0c6360 0x7f516c1f2470 secure :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f516000c310 tx=0x7f516000c7d0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.916+0000 7f5172606640 1 --2- 172.21.15.83:0/4285509466 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f516c0bee30 0x7f516c0c2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.916+0000 7f5172606640 1 -- 172.21.15.83:0/4285509466 >> 172.21.15.83:0/4285509466 conn(0x7f516c0722a0 msgr2=0x7f516c0cb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:04.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.916+0000 7f5172606640 1 -- 172.21.15.83:0/4285509466 shutdown_connections 2024-09-19T07:37:04.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.916+0000 7f5172606640 1 -- 172.21.15.83:0/4285509466 wait complete. 2024-09-19T07:37:04.932 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:37:04.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7f5bb640 1 Processor -- start 2024-09-19T07:37:04.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7f5bb640 1 -- start start 2024-09-19T07:37:04.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7f5bb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5c78153570 0x7f5c780c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7f5bb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5c780c5110 0x7f5c78159270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7f5bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c780c4130 0x7f5c781597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7f5bb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5c780c3c20 con 0x7f5c780c5110 2024-09-19T07:37:04.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7f5bb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5c780c89f0 con 0x7f5c78153570 2024-09-19T07:37:04.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7f5bb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5c780c8cd0 con 0x7f5c780c4130 2024-09-19T07:37:04.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7cb56640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5c780c5110 0x7f5c78159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7cb56640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5c780c5110 0x7f5c78159270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45164/0 (socket says 172.21.15.83:45164) 2024-09-19T07:37:04.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7cb56640 1 -- 172.21.15.83:0/3171753259 learned_addr learned my addr 172.21.15.83:0/3171753259 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:04.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7d357640 1 --2- 172.21.15.83:0/3171753259 >> v2:172.21.15.184:3300/0 conn(0x7f5c78153570 0x7f5c780c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7db58640 1 --2- 172.21.15.83:0/3171753259 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c780c4130 0x7f5c781597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7cb56640 1 -- 172.21.15.83:0/3171753259 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c780c4130 msgr2=0x7f5c781597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:04.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7cb56640 1 --2- 172.21.15.83:0/3171753259 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c780c4130 0x7f5c781597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7cb56640 1 -- 172.21.15.83:0/3171753259 >> v2:172.21.15.184:3300/0 conn(0x7f5c78153570 msgr2=0x7f5c780c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:04.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7cb56640 1 --2- 172.21.15.83:0/3171753259 >> v2:172.21.15.184:3300/0 conn(0x7f5c78153570 0x7f5c780c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.936+0000 7f5c7cb56640 1 -- 172.21.15.83:0/3171753259 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c780c8fb0 con 0x7f5c780c5110 2024-09-19T07:37:04.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c7cb56640 1 --2- 172.21.15.83:0/3171753259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c780c5110 0x7f5c78159270 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f5c780c3ee0 tx=0x7f5c6801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827a90cd6f8913a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:04.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c6ffff640 1 -- 172.21.15.83:0/3171753259 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c680b1070 con 0x7f5c780c5110 2024-09-19T07:37:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c6ffff640 1 -- 172.21.15.83:0/3171753259 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c680a8050 con 0x7f5c780c5110 2024-09-19T07:37:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c6ffff640 1 -- 172.21.15.83:0/3171753259 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c680ac860 con 0x7f5c780c5110 2024-09-19T07:37:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3171753259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c780c5110 msgr2=0x7f5c78159270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c7f5bb640 1 --2- 172.21.15.83:0/3171753259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c780c5110 0x7f5c78159270 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f5c780c3ee0 tx=0x7f5c6801c400 comp rx=0 tx=0).stop 2024-09-19T07:37:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3171753259 shutdown_connections 2024-09-19T07:37:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c7f5bb640 1 --2- 172.21.15.83:0/3171753259 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c780c4130 0x7f5c781597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c7f5bb640 1 --2- 172.21.15.83:0/3171753259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c780c5110 0x7f5c78159270 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c7f5bb640 1 --2- 172.21.15.83:0/3171753259 >> v2:172.21.15.184:3300/0 conn(0x7f5c78153570 0x7f5c780c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3171753259 >> 172.21.15.83:0/3171753259 conn(0x7f5c780c6110 msgr2=0x7f5c780c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3171753259 shutdown_connections 2024-09-19T07:37:04.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.940+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3171753259 wait complete. 2024-09-19T07:37:04.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7f5bb640 1 Processor -- start 2024-09-19T07:37:04.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7f5bb640 1 -- start start 2024-09-19T07:37:04.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7f5bb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c780c4130 0x7f5c78151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7f5bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c780c5110 0x7f5c78146090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7f5bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c78153570 0x7f5c781465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7f5bb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5c78156780 con 0x7f5c780c5110 2024-09-19T07:37:04.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7f5bb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5c78156600 con 0x7f5c780c4130 2024-09-19T07:37:04.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7f5bb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5c78156900 con 0x7f5c78153570 2024-09-19T07:37:04.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7cb56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c780c5110 0x7f5c78146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7cb56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c780c5110 0x7f5c78146090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45168/0 (socket says 172.21.15.83:45168) 2024-09-19T07:37:04.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7cb56640 1 -- 172.21.15.83:0/3556683341 learned_addr learned my addr 172.21.15.83:0/3556683341 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:04.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7d357640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c780c4130 0x7f5c78151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7cb56640 1 -- 172.21.15.83:0/3556683341 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c78153570 msgr2=0x7f5c781465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7cb56640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c78153570 0x7f5c781465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7cb56640 1 -- 172.21.15.83:0/3556683341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c780c4130 msgr2=0x7f5c78151470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:04.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7cb56640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c780c4130 0x7f5c78151470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:04.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7cb56640 1 -- 172.21.15.83:0/3556683341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c78146b10 con 0x7f5c780c5110 2024-09-19T07:37:04.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7cb56640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c780c5110 0x7f5c78146090 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f5c6801aab0 tx=0x7f5c78156bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:04.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c6dffb640 1 -- 172.21.15.83:0/3556683341 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c680b1040 con 0x7f5c780c5110 2024-09-19T07:37:04.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c6dffb640 1 -- 172.21.15.83:0/3556683341 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c680ba070 con 0x7f5c780c5110 2024-09-19T07:37:04.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c6dffb640 1 -- 172.21.15.83:0/3556683341 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c680b5640 con 0x7f5c780c5110 2024-09-19T07:37:04.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3556683341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c781f2510 con 0x7f5c780c5110 2024-09-19T07:37:04.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3556683341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c780af1a0 con 0x7f5c780c5110 2024-09-19T07:37:04.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.944+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3556683341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c780af3e0 con 0x7f5c780c5110 2024-09-19T07:37:04.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.948+0000 7f5c6dffb640 1 -- 172.21.15.83:0/3556683341 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5c680b57e0 con 0x7f5c780c5110 2024-09-19T07:37:04.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.948+0000 7f5c6dffb640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5c5c0678c0 0x7f5c5c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:04.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.948+0000 7f5c6dffb640 1 -- 172.21.15.83:0/3556683341 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7f5c68120510 con 0x7f5c780c5110 2024-09-19T07:37:04.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.948+0000 7f5c7d357640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5c5c0678c0 0x7f5c5c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:04.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.948+0000 7f5c7d357640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5c5c0678c0 0x7f5c5c069d80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5c78160290 tx=0x7f5c74008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:04.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:04.952+0000 7f5c6dffb640 1 -- 172.21.15.83:0/3556683341 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5c78146b10 con 0x7f5c780c5110 2024-09-19T07:37:05.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.156+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3556683341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c780c4bd0 con 0x7f5c780c5110 2024-09-19T07:37:05.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.156+0000 7f5c6dffb640 1 -- 172.21.15.83:0/3556683341 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7f5c68016110 con 0x7f5c780c5110 2024-09-19T07:37:05.154 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:05.154 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:05.155 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:05.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.160+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3556683341 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5c5c0678c0 msgr2=0x7f5c5c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:05.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.160+0000 7f5c7f5bb640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5c5c0678c0 0x7f5c5c069d80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5c78160290 tx=0x7f5c74008040 comp rx=0 tx=0).stop 2024-09-19T07:37:05.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.160+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3556683341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c780c5110 msgr2=0x7f5c78146090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.160+0000 7f5c7f5bb640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c780c5110 0x7f5c78146090 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f5c6801aab0 tx=0x7f5c78156bb0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.160+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3556683341 shutdown_connections 2024-09-19T07:37:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.160+0000 7f5c7f5bb640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5c5c0678c0 0x7f5c5c069d80 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.160+0000 7f5c7f5bb640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c78153570 0x7f5c781465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.160+0000 7f5c7f5bb640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c780c5110 0x7f5c78146090 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.160+0000 7f5c7f5bb640 1 --2- 172.21.15.83:0/3556683341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c780c4130 0x7f5c78151470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.160+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3556683341 >> 172.21.15.83:0/3556683341 conn(0x7f5c780c6110 msgr2=0x7f5c78154500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.160+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3556683341 shutdown_connections 2024-09-19T07:37:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.160+0000 7f5c7f5bb640 1 -- 172.21.15.83:0/3556683341 wait complete. 2024-09-19T07:37:05.173 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-09-19T07:37:05.174 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-19T07:37:05.174 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:37:05.187 INFO:tasks.ceph.mds.b:Stopped 2024-09-19T07:37:05.188 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:05.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.328+0000 7fe50ca03640 1 Processor -- start 2024-09-19T07:37:05.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.328+0000 7fe50ca03640 1 -- start start 2024-09-19T07:37:05.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.328+0000 7fe50ca03640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe5080c4880 0x7fe5080c1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:05.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.328+0000 7fe50ca03640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5080c2330 0x7fe508152f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:05.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.328+0000 7fe50ca03640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5081538e0 con 0x7fe508153470 2024-09-19T07:37:05.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.328+0000 7fe50ca03640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe50815a040 con 0x7fe5080c2330 2024-09-19T07:37:05.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.328+0000 7fe50ca03640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe5080c2c10 con 0x7fe5080c4880 2024-09-19T07:37:05.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe505d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5080c2330 0x7fe508152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:05.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe505d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5080c2330 0x7fe508152f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45184/0 (socket says 172.21.15.83:45184) 2024-09-19T07:37:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe505d9b640 1 -- 172.21.15.83:0/1894436338 learned_addr learned my addr 172.21.15.83:0/1894436338 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe50659c640 1 --2- 172.21.15.83:0/1894436338 >> v2:172.21.15.184:3300/0 conn(0x7fe5080c4880 0x7fe5080c1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe505d9b640 1 -- 172.21.15.83:0/1894436338 >> v2:172.21.15.184:3300/0 conn(0x7fe5080c4880 msgr2=0x7fe5080c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe505d9b640 1 --2- 172.21.15.83:0/1894436338 >> v2:172.21.15.184:3300/0 conn(0x7fe5080c4880 0x7fe5080c1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe505d9b640 1 -- 172.21.15.83:0/1894436338 >> v1:172.21.15.83:6789/0 conn(0x7fe508153470 legacy=0x7fe508155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe505d9b640 1 -- 172.21.15.83:0/1894436338 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe508159430 con 0x7fe5080c2330 2024-09-19T07:37:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe505d9b640 1 --2- 172.21.15.83:0/1894436338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe5080c2330 0x7fe508152f30 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fe5080ab380 tx=0x7fe4fc01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3c9484f3856a46 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe50559a640 1 -- 172.21.15.83:0/1894436338 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc0b1070 con 0x7fe5080c2330 2024-09-19T07:37:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe50559a640 1 -- 172.21.15.83:0/1894436338 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4fc0ac630 con 0x7fe5080c2330 2024-09-19T07:37:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe50559a640 1 -- 172.21.15.83:0/1894436338 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc0a8050 con 0x7fe5080c2330 2024-09-19T07:37:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe50ca03640 1 -- 172.21.15.83:0/1894436338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe5080c2330 msgr2=0x7fe508152f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe50ca03640 1 --2- 172.21.15.83:0/1894436338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe5080c2330 0x7fe508152f30 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fe5080ab380 tx=0x7fe4fc01c3d0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe50ca03640 1 -- 172.21.15.83:0/1894436338 shutdown_connections 2024-09-19T07:37:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe50ca03640 1 --2- 172.21.15.83:0/1894436338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe5080c2330 0x7fe508152f30 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe50ca03640 1 --2- 172.21.15.83:0/1894436338 >> v2:172.21.15.184:3300/0 conn(0x7fe5080c4880 0x7fe5080c1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe50ca03640 1 -- 172.21.15.83:0/1894436338 >> 172.21.15.83:0/1894436338 conn(0x7fe5080aa910 msgr2=0x7fe5080c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe50ca03640 1 -- 172.21.15.83:0/1894436338 shutdown_connections 2024-09-19T07:37:05.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.332+0000 7fe50ca03640 1 -- 172.21.15.83:0/1894436338 wait complete. 2024-09-19T07:37:05.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe50ca03640 1 Processor -- start 2024-09-19T07:37:05.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe50ca03640 1 -- start start 2024-09-19T07:37:05.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe50ca03640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5080c2330 0x7fe50814f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:05.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe50ca03640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe5080c4880 0x7fe50814e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:05.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe50ca03640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe508153470 0x7fe5080ab400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:05.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe50ca03640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe508156ab0 con 0x7fe508153470 2024-09-19T07:37:05.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe50ca03640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe508156930 con 0x7fe5080c4880 2024-09-19T07:37:05.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe50ca03640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe5080c4c60 con 0x7fe5080c2330 2024-09-19T07:37:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe506d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe508153470 0x7fe5080ab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe506d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe508153470 0x7fe5080ab400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45198/0 (socket says 172.21.15.83:45198) 2024-09-19T07:37:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe506d9d640 1 -- 172.21.15.83:0/2187935669 learned_addr learned my addr 172.21.15.83:0/2187935669 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe505d9b640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe5080c4880 0x7fe50814e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe50659c640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5080c2330 0x7fe50814f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe506d9d640 1 -- 172.21.15.83:0/2187935669 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5080c2330 msgr2=0x7fe50814f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe506d9d640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5080c2330 0x7fe50814f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe506d9d640 1 -- 172.21.15.83:0/2187935669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe5080c4880 msgr2=0x7fe50814e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe506d9d640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe5080c4880 0x7fe50814e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe506d9d640 1 -- 172.21.15.83:0/2187935669 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50806db90 con 0x7fe508153470 2024-09-19T07:37:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe506d9d640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe508153470 0x7fe5080ab400 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fe50000c730 tx=0x7fe50000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:05.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/2187935669 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe500003b80 con 0x7fe508153470 2024-09-19T07:37:05.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/2187935669 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe500003d20 con 0x7fe508153470 2024-09-19T07:37:05.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe50ca03640 1 -- 172.21.15.83:0/2187935669 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5080ac130 con 0x7fe508153470 2024-09-19T07:37:05.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/2187935669 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe500005310 con 0x7fe508153470 2024-09-19T07:37:05.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.336+0000 7fe50ca03640 1 -- 172.21.15.83:0/2187935669 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5080ac6f0 con 0x7fe508153470 2024-09-19T07:37:05.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.340+0000 7fe50ca03640 1 -- 172.21.15.83:0/2187935669 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe508152f30 con 0x7fe508153470 2024-09-19T07:37:05.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.340+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/2187935669 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe5000054b0 con 0x7fe508153470 2024-09-19T07:37:05.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.340+0000 7fe4f6ffd640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe4e80678f0 0x7fe4e8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:05.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.340+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/2187935669 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7fe500086550 con 0x7fe508153470 2024-09-19T07:37:05.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.340+0000 7fe50659c640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe4e80678f0 0x7fe4e8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:05.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.340+0000 7fe50659c640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe4e80678f0 0x7fe4e8069db0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8006aa0 tx=0x7fe4f8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:05.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.344+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/2187935669 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe500055ed0 con 0x7fe508153470 2024-09-19T07:37:05.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.576+0000 7fe50ca03640 1 -- 172.21.15.83:0/2187935669 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5080688c0 con 0x7fe508153470 2024-09-19T07:37:05.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/2187935669 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7fe50005a1d0 con 0x7fe508153470 2024-09-19T07:37:05.576 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:05.576 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:05.576 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:05.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe50ca03640 1 -- 172.21.15.83:0/2187935669 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe4e80678f0 msgr2=0x7fe4e8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:05.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe50ca03640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe4e80678f0 0x7fe4e8069db0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8006aa0 tx=0x7fe4f8008040 comp rx=0 tx=0).stop 2024-09-19T07:37:05.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe50ca03640 1 -- 172.21.15.83:0/2187935669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe508153470 msgr2=0x7fe5080ab400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:05.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe50ca03640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe508153470 0x7fe5080ab400 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fe50000c730 tx=0x7fe50000cbf0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe50ca03640 1 -- 172.21.15.83:0/2187935669 shutdown_connections 2024-09-19T07:37:05.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe50ca03640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe4e80678f0 0x7fe4e8069db0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe50ca03640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe508153470 0x7fe5080ab400 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe50ca03640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe5080c4880 0x7fe50814e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe50ca03640 1 --2- 172.21.15.83:0/2187935669 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5080c2330 0x7fe50814f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:05.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe50ca03640 1 -- 172.21.15.83:0/2187935669 >> 172.21.15.83:0/2187935669 conn(0x7fe5080aa910 msgr2=0x7fe508154ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:05.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe50ca03640 1 -- 172.21.15.83:0/2187935669 shutdown_connections 2024-09-19T07:37:05.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:05.580+0000 7fe50ca03640 1 -- 172.21.15.83:0/2187935669 wait complete. 2024-09-19T07:37:05.595 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:37:06.936 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:07.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.072+0000 7ff3655c4640 1 Processor -- start 2024-09-19T07:37:07.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.072+0000 7ff3655c4640 1 -- start start 2024-09-19T07:37:07.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.072+0000 7ff3655c4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff360071100 0x7ff3601545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:07.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.072+0000 7ff3655c4640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff3600c28e0 0x7ff360150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:07.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.072+0000 7ff3655c4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3600a9e60 con 0x7ff360151480 2024-09-19T07:37:07.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.072+0000 7ff3655c4640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff360071b50 con 0x7ff3600c28e0 2024-09-19T07:37:07.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.072+0000 7ff3655c4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff3600c1ff0 con 0x7ff360071100 2024-09-19T07:37:07.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.072+0000 7ff35e7fc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff3600c28e0 0x7ff360150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:07.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.072+0000 7ff35effd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff360071100 0x7ff3601545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:07.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.072+0000 7ff35e7fc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff3600c28e0 0x7ff360150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:60118/0 (socket says 172.21.15.83:60118) 2024-09-19T07:37:07.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.072+0000 7ff35f7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7ff360151480 0x7ff36006dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:39716/0 (socket says 172.21.15.83:39716) 2024-09-19T07:37:07.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.072+0000 7ff35e7fc640 1 -- 172.21.15.83:0/4132433800 learned_addr learned my addr 172.21.15.83:0/4132433800 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:07.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff35effd640 1 -- 172.21.15.83:0/4132433800 >> v2:172.21.15.184:3300/0 conn(0x7ff3600c28e0 msgr2=0x7ff360150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:07.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff35effd640 1 --2- 172.21.15.83:0/4132433800 >> v2:172.21.15.184:3300/0 conn(0x7ff3600c28e0 0x7ff360150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff35effd640 1 -- 172.21.15.83:0/4132433800 >> v1:172.21.15.184:6789/0 conn(0x7ff360151480 legacy=0x7ff36006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff35effd640 1 -- 172.21.15.83:0/4132433800 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3600aa140 con 0x7ff360071100 2024-09-19T07:37:07.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff35e7fc640 1 --2- 172.21.15.83:0/4132433800 >> v2:172.21.15.184:3300/0 conn(0x7ff3600c28e0 0x7ff360150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:37:07.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff35effd640 1 --2- 172.21.15.83:0/4132433800 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff360071100 0x7ff3601545e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff3500097c0 tx=0x7ff35001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf4312d11c077670 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:07.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff35dffb640 1 -- 172.21.15.83:0/4132433800 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3500b0070 con 0x7ff360071100 2024-09-19T07:37:07.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff35dffb640 1 -- 172.21.15.83:0/4132433800 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3500ab600 con 0x7ff360071100 2024-09-19T07:37:07.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff35dffb640 1 -- 172.21.15.83:0/4132433800 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3500ab9d0 con 0x7ff360071100 2024-09-19T07:37:07.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff3655c4640 1 -- 172.21.15.83:0/4132433800 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff360071100 msgr2=0x7ff3601545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff3655c4640 1 --2- 172.21.15.83:0/4132433800 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff360071100 0x7ff3601545e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff3500097c0 tx=0x7ff35001c410 comp rx=0 tx=0).stop 2024-09-19T07:37:07.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff3655c4640 1 -- 172.21.15.83:0/4132433800 shutdown_connections 2024-09-19T07:37:07.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff3655c4640 1 --2- 172.21.15.83:0/4132433800 >> v2:172.21.15.184:3300/0 conn(0x7ff3600c28e0 0x7ff360150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff3655c4640 1 --2- 172.21.15.83:0/4132433800 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff360071100 0x7ff3601545e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff3655c4640 1 -- 172.21.15.83:0/4132433800 >> 172.21.15.83:0/4132433800 conn(0x7ff36006bc80 msgr2=0x7ff36006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:07.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff3655c4640 1 -- 172.21.15.83:0/4132433800 shutdown_connections 2024-09-19T07:37:07.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.076+0000 7ff3655c4640 1 -- 172.21.15.83:0/4132433800 wait complete. 2024-09-19T07:37:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3655c4640 1 Processor -- start 2024-09-19T07:37:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3655c4640 1 -- start start 2024-09-19T07:37:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3655c4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff360071100 0x7ff36014a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3655c4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3600c28e0 0x7ff360148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3655c4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff360151480 0x7ff3601490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3655c4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff360071660 con 0x7ff360151480 2024-09-19T07:37:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3655c4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff3600714e0 con 0x7ff360071100 2024-09-19T07:37:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3655c4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff360160c60 con 0x7ff3600c28e0 2024-09-19T07:37:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff35f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff360151480 0x7ff3601490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff35f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff360151480 0x7ff3601490b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45206/0 (socket says 172.21.15.83:45206) 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff35f7fe640 1 -- 172.21.15.83:0/3408028539 learned_addr learned my addr 172.21.15.83:0/3408028539 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff35effd640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff360071100 0x7ff36014a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff35e7fc640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3600c28e0 0x7ff360148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff35f7fe640 1 -- 172.21.15.83:0/3408028539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3600c28e0 msgr2=0x7ff360148b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff35f7fe640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3600c28e0 0x7ff360148b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff35f7fe640 1 -- 172.21.15.83:0/3408028539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff360071100 msgr2=0x7ff36014a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff35f7fe640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff360071100 0x7ff36014a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff35f7fe640 1 -- 172.21.15.83:0/3408028539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff360153810 con 0x7ff360151480 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff35f7fe640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff360151480 0x7ff3601490b0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7ff35800c740 tx=0x7ff35800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3477fe640 1 -- 172.21.15.83:0/3408028539 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff358017070 con 0x7ff360151480 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3477fe640 1 -- 172.21.15.83:0/3408028539 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff35800ce50 con 0x7ff360151480 2024-09-19T07:37:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3655c4640 1 -- 172.21.15.83:0/3408028539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3601ecd70 con 0x7ff360151480 2024-09-19T07:37:07.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3477fe640 1 -- 172.21.15.83:0/3408028539 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35800a860 con 0x7ff360151480 2024-09-19T07:37:07.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3655c4640 1 -- 172.21.15.83:0/3408028539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3601ed330 con 0x7ff360151480 2024-09-19T07:37:07.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3655c4640 1 -- 172.21.15.83:0/3408028539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3601ed530 con 0x7ff360151480 2024-09-19T07:37:07.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.080+0000 7ff3477fe640 1 -- 172.21.15.83:0/3408028539 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff35801a070 con 0x7ff360151480 2024-09-19T07:37:07.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.084+0000 7ff3477fe640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff34c0678f0 0x7ff34c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:07.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.084+0000 7ff3477fe640 1 -- 172.21.15.83:0/3408028539 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7ff358085f50 con 0x7ff360151480 2024-09-19T07:37:07.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.084+0000 7ff35effd640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff34c0678f0 0x7ff34c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:07.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.084+0000 7ff35effd640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff34c0678f0 0x7ff34c069db0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff3500ad000 tx=0x7ff3500a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:07.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.088+0000 7ff3477fe640 1 -- 172.21.15.83:0/3408028539 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff3601ed530 con 0x7ff360151480 2024-09-19T07:37:07.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 -- 172.21.15.83:0/3408028539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3601545e0 con 0x7ff360151480 2024-09-19T07:37:07.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3477fe640 1 -- 172.21.15.83:0/3408028539 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7ff358055e30 con 0x7ff360151480 2024-09-19T07:37:07.269 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:07.269 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:07.270 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:07.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 -- 172.21.15.83:0/3408028539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff34c0678f0 msgr2=0x7ff34c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff34c0678f0 0x7ff34c069db0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff3500ad000 tx=0x7ff3500a6000 comp rx=0 tx=0).stop 2024-09-19T07:37:07.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 -- 172.21.15.83:0/3408028539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff360151480 msgr2=0x7ff3601490b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff360151480 0x7ff3601490b0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7ff35800c740 tx=0x7ff35800cc00 comp rx=0 tx=0).stop 2024-09-19T07:37:07.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 -- 172.21.15.83:0/3408028539 shutdown_connections 2024-09-19T07:37:07.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff34c0678f0 0x7ff34c069db0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff360151480 0x7ff3601490b0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3600c28e0 0x7ff360148b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 --2- 172.21.15.83:0/3408028539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff360071100 0x7ff36014a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 -- 172.21.15.83:0/3408028539 >> 172.21.15.83:0/3408028539 conn(0x7ff36006bc80 msgr2=0x7ff3600c39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:07.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 -- 172.21.15.83:0/3408028539 shutdown_connections 2024-09-19T07:37:07.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.272+0000 7ff3655c4640 1 -- 172.21.15.83:0/3408028539 wait complete. 2024-09-19T07:37:07.286 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:37:07.597 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:07.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79f859640 1 Processor -- start 2024-09-19T07:37:07.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79f859640 1 -- start start 2024-09-19T07:37:07.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79f859640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff798071100 0x7ff7981545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:07.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79f859640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7980a9e60 con 0x7ff79806dea0 2024-09-19T07:37:07.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79f859640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7980aa140 con 0x7ff7980c28e0 2024-09-19T07:37:07.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79f859640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff798071b50 con 0x7ff798071100 2024-09-19T07:37:07.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79d5f5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff798071100 0x7ff7981545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:07.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79d5f5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff798071100 0x7ff7981545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45214/0 (socket says 172.21.15.83:45214) 2024-09-19T07:37:07.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79ddf6640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff79806dea0 0x7ff79806e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40964/0 (socket says 172.21.15.83:40964) 2024-09-19T07:37:07.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79d5f5640 1 -- 172.21.15.83:0/3298430304 learned_addr learned my addr 172.21.15.83:0/3298430304 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:07.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79d5f5640 1 -- 172.21.15.83:0/3298430304 >> v1:172.21.15.184:6789/0 conn(0x7ff7980c28e0 legacy=0x7ff798150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79d5f5640 1 -- 172.21.15.83:0/3298430304 >> v1:172.21.15.83:6789/0 conn(0x7ff79806dea0 legacy=0x7ff79806e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.732+0000 7ff79d5f5640 1 -- 172.21.15.83:0/3298430304 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7980c2240 con 0x7ff798071100 2024-09-19T07:37:07.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff79d5f5640 1 --2- 172.21.15.83:0/3298430304 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff798071100 0x7ff7981545e0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff794009870 tx=0x7ff79401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42288be4df4fa802 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:07.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff78ffff640 1 -- 172.21.15.83:0/3298430304 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7940b0070 con 0x7ff798071100 2024-09-19T07:37:07.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff78ffff640 1 -- 172.21.15.83:0/3298430304 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7940ab600 con 0x7ff798071100 2024-09-19T07:37:07.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff78ffff640 1 -- 172.21.15.83:0/3298430304 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7940ab9d0 con 0x7ff798071100 2024-09-19T07:37:07.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff79f859640 1 -- 172.21.15.83:0/3298430304 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff798071100 msgr2=0x7ff7981545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff79f859640 1 --2- 172.21.15.83:0/3298430304 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff798071100 0x7ff7981545e0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff794009870 tx=0x7ff79401c4d0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff79f859640 1 -- 172.21.15.83:0/3298430304 shutdown_connections 2024-09-19T07:37:07.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff79f859640 1 --2- 172.21.15.83:0/3298430304 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff798071100 0x7ff7981545e0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff79f859640 1 -- 172.21.15.83:0/3298430304 >> 172.21.15.83:0/3298430304 conn(0x7ff79806bc80 msgr2=0x7ff79806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:07.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff79f859640 1 -- 172.21.15.83:0/3298430304 shutdown_connections 2024-09-19T07:37:07.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff79f859640 1 -- 172.21.15.83:0/3298430304 wait complete. 2024-09-19T07:37:07.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff79f859640 1 Processor -- start 2024-09-19T07:37:07.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.736+0000 7ff79f859640 1 -- start start 2024-09-19T07:37:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79f859640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79806dea0 0x7ff79814bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79f859640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff798071100 0x7ff7981459b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79f859640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff7980c28e0 0x7ff798145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79f859640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff798071660 con 0x7ff79806dea0 2024-09-19T07:37:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79f859640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff7980714e0 con 0x7ff798071100 2024-09-19T07:37:07.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79f859640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff798161c70 con 0x7ff7980c28e0 2024-09-19T07:37:07.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79d5f5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79806dea0 0x7ff79814bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:07.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79d5f5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79806dea0 0x7ff79814bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45230/0 (socket says 172.21.15.83:45230) 2024-09-19T07:37:07.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79d5f5640 1 -- 172.21.15.83:0/109889444 learned_addr learned my addr 172.21.15.83:0/109889444 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:07.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79ddf6640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff7980c28e0 0x7ff798145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:07.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79d5f5640 1 -- 172.21.15.83:0/109889444 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff7980c28e0 msgr2=0x7ff798145ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79d5f5640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff7980c28e0 0x7ff798145ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79d5f5640 1 -- 172.21.15.83:0/109889444 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff798071100 msgr2=0x7ff7981459b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79d5f5640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff798071100 0x7ff7981459b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79d5f5640 1 -- 172.21.15.83:0/109889444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7981ec330 con 0x7ff79806dea0 2024-09-19T07:37:07.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79d5f5640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79806dea0 0x7ff79814bd70 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff7940ae040 tx=0x7ff79401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:07.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff78dffb640 1 -- 172.21.15.83:0/109889444 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7940ba020 con 0x7ff79806dea0 2024-09-19T07:37:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff78dffb640 1 -- 172.21.15.83:0/109889444 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7940b6640 con 0x7ff79806dea0 2024-09-19T07:37:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79f859640 1 -- 172.21.15.83:0/109889444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7981ec5d0 con 0x7ff79806dea0 2024-09-19T07:37:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff78dffb640 1 -- 172.21.15.83:0/109889444 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7940b0040 con 0x7ff79806dea0 2024-09-19T07:37:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79f859640 1 -- 172.21.15.83:0/109889444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7981ecb90 con 0x7ff79806dea0 2024-09-19T07:37:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79f859640 1 -- 172.21.15.83:0/109889444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7981545e0 con 0x7ff79806dea0 2024-09-19T07:37:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff78dffb640 1 -- 172.21.15.83:0/109889444 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff7940bd020 con 0x7ff79806dea0 2024-09-19T07:37:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff78dffb640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff77c0678c0 0x7ff77c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff78dffb640 1 -- 172.21.15.83:0/109889444 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7ff794120a70 con 0x7ff79806dea0 2024-09-19T07:37:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79cdf4640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff77c0678c0 0x7ff77c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.740+0000 7ff79cdf4640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff77c0678c0 0x7ff77c069d80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff788000f70 tx=0x7ff7880079e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:07.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.744+0000 7ff78dffb640 1 -- 172.21.15.83:0/109889444 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff7940c4030 con 0x7ff79806dea0 2024-09-19T07:37:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.928+0000 7ff79f859640 1 -- 172.21.15.83:0/109889444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff798146430 con 0x7ff79806dea0 2024-09-19T07:37:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.928+0000 7ff78dffb640 1 -- 172.21.15.83:0/109889444 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7ff794016150 con 0x7ff79806dea0 2024-09-19T07:37:07.926 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:07.926 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:07.926 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.932+0000 7ff79f859640 1 -- 172.21.15.83:0/109889444 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff77c0678c0 msgr2=0x7ff77c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.932+0000 7ff79f859640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff77c0678c0 0x7ff77c069d80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff788000f70 tx=0x7ff7880079e0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.932+0000 7ff79f859640 1 -- 172.21.15.83:0/109889444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79806dea0 msgr2=0x7ff79814bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.932+0000 7ff79f859640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79806dea0 0x7ff79814bd70 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff7940ae040 tx=0x7ff79401ad10 comp rx=0 tx=0).stop 2024-09-19T07:37:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.932+0000 7ff79f859640 1 -- 172.21.15.83:0/109889444 shutdown_connections 2024-09-19T07:37:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.932+0000 7ff79f859640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff77c0678c0 0x7ff77c069d80 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.932+0000 7ff79f859640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff7980c28e0 0x7ff798145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.932+0000 7ff79f859640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff798071100 0x7ff7981459b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.932+0000 7ff79f859640 1 --2- 172.21.15.83:0/109889444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79806dea0 0x7ff79814bd70 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.932+0000 7ff79f859640 1 -- 172.21.15.83:0/109889444 >> 172.21.15.83:0/109889444 conn(0x7ff79806bc80 msgr2=0x7ff79806e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.932+0000 7ff79f859640 1 -- 172.21.15.83:0/109889444 shutdown_connections 2024-09-19T07:37:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:07.932+0000 7ff79f859640 1 -- 172.21.15.83:0/109889444 wait complete. 2024-09-19T07:37:07.944 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:37:09.287 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:09.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dd0435640 1 Processor -- start 2024-09-19T07:37:09.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dd0435640 1 -- start start 2024-09-19T07:37:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dd0435640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5dc80bee30 0x7f5dc81570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dd0435640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dc80c5b90 con 0x7f5dc80c6360 2024-09-19T07:37:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dd0435640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dc815a050 con 0x7f5dc80c9680 2024-09-19T07:37:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dd0435640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5dc806c950 con 0x7f5dc80bee30 2024-09-19T07:37:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dce1d1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5dc80c6360 0x7f5dc8153050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40966/0 (socket says 172.21.15.83:40966) 2024-09-19T07:37:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dce1d1640 1 -- 172.21.15.83:0/1750640858 learned_addr learned my addr 172.21.15.83:0/1750640858 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:37:09.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd9d0640 1 --2- 172.21.15.83:0/1750640858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5dc80bee30 0x7f5dc81570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:09.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3477630042 0 0) 0x7f5dc80c5b90 con 0x7f5dc80c6360 2024-09-19T07:37:09.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5db4003530 con 0x7f5dc80c6360 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4269609965 0 0) 0x7f5db4003530 con 0x7f5dc80c6360 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5dc80bee30 msgr2=0x7f5dc81570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 --2- 172.21.15.83:0/1750640858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5dc80bee30 0x7f5dc81570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 >> v1:172.21.15.184:6789/0 conn(0x7f5dc80c9680 legacy=0x7f5dc8157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc806c670 con 0x7f5dc80c6360 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f5dc4001680 con 0x7f5dc80c6360 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 >> v1:172.21.15.83:6789/0 conn(0x7f5dc80c6360 legacy=0x7f5dc8153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 --2- 172.21.15.83:0/1750640858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5dc80bee30 0x7f5db40039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 --2- 172.21.15.83:0/1750640858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5db4003f70 0x7f5db4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 --2- 172.21.15.83:0/1750640858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5db4004d50 0x7f5db40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc806c950 con 0x7f5db4003f70 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5db400bb60 con 0x7f5db4004d50 2024-09-19T07:37:09.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5db400bd60 con 0x7f5dc80bee30 2024-09-19T07:37:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd9d0640 1 --2- 172.21.15.83:0/1750640858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5db4003f70 0x7f5db4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dce1d1640 1 --2- 172.21.15.83:0/1750640858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5dc80bee30 0x7f5db40039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dce9d2640 1 --2- 172.21.15.83:0/1750640858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5db4004d50 0x7f5db40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd9d0640 1 -- 172.21.15.83:0/1750640858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5dc80bee30 msgr2=0x7f5db40039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd9d0640 1 --2- 172.21.15.83:0/1750640858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5dc80bee30 0x7f5db40039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd9d0640 1 -- 172.21.15.83:0/1750640858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5db4004d50 msgr2=0x7f5db40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd9d0640 1 --2- 172.21.15.83:0/1750640858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5db4004d50 0x7f5db40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.423+0000 7f5dcd9d0640 1 -- 172.21.15.83:0/1750640858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db4003530 con 0x7f5db4003f70 2024-09-19T07:37:09.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dcd9d0640 1 --2- 172.21.15.83:0/1750640858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5db4003f70 0x7f5db4004810 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80bf280 tx=0x7f5db801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203418d3f7c5d79 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:09.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db8009690 con 0x7f5db4003f70 2024-09-19T07:37:09.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5db80a8430 con 0x7f5db4003f70 2024-09-19T07:37:09.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dcd1cf640 1 -- 172.21.15.83:0/1750640858 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db80a8800 con 0x7f5db4003f70 2024-09-19T07:37:09.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 -- 172.21.15.83:0/1750640858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5db4003f70 msgr2=0x7f5db4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:09.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 --2- 172.21.15.83:0/1750640858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5db4003f70 0x7f5db4004810 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80bf280 tx=0x7f5db801c9f0 comp rx=0 tx=0).stop 2024-09-19T07:37:09.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dce9d2640 1 -- 172.21.15.83:0/1750640858 reap_dead start 2024-09-19T07:37:09.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 -- 172.21.15.83:0/1750640858 shutdown_connections 2024-09-19T07:37:09.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 -- 172.21.15.83:0/1750640858 >> 172.21.15.83:0/1750640858 conn(0x7f5dc80722a0 msgr2=0x7f5dc80726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:09.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 -- 172.21.15.83:0/1750640858 shutdown_connections 2024-09-19T07:37:09.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 -- 172.21.15.83:0/1750640858 wait complete. 2024-09-19T07:37:09.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 Processor -- start 2024-09-19T07:37:09.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 -- start start 2024-09-19T07:37:09.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5db4003f70 0x7f5dc80abc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:09.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5dc8160160 0x7f5dc81454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:09.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc8146c70 0x7f5dc81470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc81538e0 con 0x7f5dc8146c70 2024-09-19T07:37:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc8153760 con 0x7f5dc8160160 2024-09-19T07:37:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.427+0000 7f5dd0435640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5dc80c6740 con 0x7f5db4003f70 2024-09-19T07:37:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dce9d2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc8146c70 0x7f5dc81470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dce9d2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc8146c70 0x7f5dc81470f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45248/0 (socket says 172.21.15.83:45248) 2024-09-19T07:37:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dce9d2640 1 -- 172.21.15.83:0/4248592177 learned_addr learned my addr 172.21.15.83:0/4248592177 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dce1d1640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5db4003f70 0x7f5dc80abc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dcd9d0640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5dc8160160 0x7f5dc81454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dce9d2640 1 -- 172.21.15.83:0/4248592177 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5db4003f70 msgr2=0x7f5dc80abc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dce9d2640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5db4003f70 0x7f5dc80abc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dce9d2640 1 -- 172.21.15.83:0/4248592177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5dc8160160 msgr2=0x7f5dc81454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dce9d2640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5dc8160160 0x7f5dc81454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dce9d2640 1 -- 172.21.15.83:0/4248592177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc8145e90 con 0x7f5dc8146c70 2024-09-19T07:37:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dce9d2640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc8146c70 0x7f5dc81470f0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80cc760 tx=0x7f5dc000e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dbeffd640 1 -- 172.21.15.83:0/4248592177 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0019070 con 0x7f5dc8146c70 2024-09-19T07:37:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dbeffd640 1 -- 172.21.15.83:0/4248592177 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc0003b90 con 0x7f5dc8146c70 2024-09-19T07:37:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dd0435640 1 -- 172.21.15.83:0/4248592177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc815f680 con 0x7f5dc8146c70 2024-09-19T07:37:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dbeffd640 1 -- 172.21.15.83:0/4248592177 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc00050d0 con 0x7f5dc8146c70 2024-09-19T07:37:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dd0435640 1 -- 172.21.15.83:0/4248592177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc815fc40 con 0x7f5dc8146c70 2024-09-19T07:37:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dd0435640 1 -- 172.21.15.83:0/4248592177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc8153050 con 0x7f5dc8146c70 2024-09-19T07:37:09.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dbeffd640 1 -- 172.21.15.83:0/4248592177 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5dc001c070 con 0x7f5dc8146c70 2024-09-19T07:37:09.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dbeffd640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5dac0678f0 0x7f5dac069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:09.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dbeffd640 1 -- 172.21.15.83:0/4248592177 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7f5dc0085e00 con 0x7f5dc8146c70 2024-09-19T07:37:09.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dce1d1640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5dac0678f0 0x7f5dac069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:09.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.431+0000 7f5dce1d1640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5dac0678f0 0x7f5dac069db0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40017f0 tx=0x7f5dc4001600 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:09.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.435+0000 7f5dbeffd640 1 -- 172.21.15.83:0/4248592177 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5dc0055dd0 con 0x7f5dc8146c70 2024-09-19T07:37:09.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.631+0000 7f5dd0435640 1 -- 172.21.15.83:0/4248592177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5dc814bcb0 con 0x7f5dc8146c70 2024-09-19T07:37:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dbeffd640 1 -- 172.21.15.83:0/4248592177 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7f5dc00556d0 con 0x7f5dc8146c70 2024-09-19T07:37:09.632 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:09.632 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:09.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dd0435640 1 -- 172.21.15.83:0/4248592177 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5dac0678f0 msgr2=0x7f5dac069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:09.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dd0435640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5dac0678f0 0x7f5dac069db0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40017f0 tx=0x7f5dc4001600 comp rx=0 tx=0).stop 2024-09-19T07:37:09.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dd0435640 1 -- 172.21.15.83:0/4248592177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc8146c70 msgr2=0x7f5dc81470f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:09.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dd0435640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc8146c70 0x7f5dc81470f0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80cc760 tx=0x7f5dc000e920 comp rx=0 tx=0).stop 2024-09-19T07:37:09.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dd0435640 1 -- 172.21.15.83:0/4248592177 shutdown_connections 2024-09-19T07:37:09.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dd0435640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5dac0678f0 0x7f5dac069db0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:09.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dd0435640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc8146c70 0x7f5dc81470f0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:09.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dd0435640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5dc8160160 0x7f5dc81454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:09.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dd0435640 1 --2- 172.21.15.83:0/4248592177 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5db4003f70 0x7f5dc80abc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:09.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dd0435640 1 -- 172.21.15.83:0/4248592177 >> 172.21.15.83:0/4248592177 conn(0x7f5dc80722a0 msgr2=0x7f5dc8151070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:09.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dd0435640 1 -- 172.21.15.83:0/4248592177 shutdown_connections 2024-09-19T07:37:09.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:09.635+0000 7f5dd0435640 1 -- 172.21.15.83:0/4248592177 wait complete. 2024-09-19T07:37:09.651 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:37:09.944 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:10.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2befae4640 1 Processor -- start 2024-09-19T07:37:10.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2befae4640 1 -- start start 2024-09-19T07:37:10.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2befae4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2be81545e0 0x7f2be806e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:10.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2befae4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be8154a70 con 0x7f2be80c28e0 2024-09-19T07:37:10.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2befae4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be80a9e60 con 0x7f2be8071100 2024-09-19T07:37:10.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2befae4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f2be8071b50 con 0x7f2be81545e0 2024-09-19T07:37:10.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2bed07f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2be80c28e0 0x7f2be806dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40978/0 (socket says 172.21.15.83:40978) 2024-09-19T07:37:10.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2bed07f640 1 -- 172.21.15.83:0/2775300666 learned_addr learned my addr 172.21.15.83:0/2775300666 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:37:10.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2bee081640 1 --2- 172.21.15.83:0/2775300666 >> v2:172.21.15.83:3300/0 conn(0x7f2be81545e0 0x7f2be806e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:10.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2bec87e640 1 -- 172.21.15.83:0/2775300666 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440146485 0 0) 0x7f2be8154a70 con 0x7f2be80c28e0 2024-09-19T07:37:10.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2bec87e640 1 -- 172.21.15.83:0/2775300666 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bd4003530 con 0x7f2be80c28e0 2024-09-19T07:37:10.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2bee081640 1 -- 172.21.15.83:0/2775300666 >> v1:172.21.15.184:6789/0 conn(0x7f2be8071100 legacy=0x7f2be8150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:10.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2bee081640 1 -- 172.21.15.83:0/2775300666 >> v1:172.21.15.83:6789/0 conn(0x7f2be80c28e0 legacy=0x7f2be806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:10.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.079+0000 7f2bee081640 1 -- 172.21.15.83:0/2775300666 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be80aa140 con 0x7f2be81545e0 2024-09-19T07:37:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bee081640 1 --2- 172.21.15.83:0/2775300666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2be81545e0 0x7f2be806e5b0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009d90 tx=0x7f2be001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c08c672b817aba8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bec87e640 1 -- 172.21.15.83:0/2775300666 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be00b2070 con 0x7f2be81545e0 2024-09-19T07:37:10.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bec87e640 1 -- 172.21.15.83:0/2775300666 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2be00a5890 con 0x7f2be81545e0 2024-09-19T07:37:10.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bec87e640 1 -- 172.21.15.83:0/2775300666 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be00a5c60 con 0x7f2be81545e0 2024-09-19T07:37:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 -- 172.21.15.83:0/2775300666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2be81545e0 msgr2=0x7f2be806e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 --2- 172.21.15.83:0/2775300666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2be81545e0 0x7f2be806e5b0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009d90 tx=0x7f2be001c8e0 comp rx=0 tx=0).stop 2024-09-19T07:37:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 -- 172.21.15.83:0/2775300666 shutdown_connections 2024-09-19T07:37:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 --2- 172.21.15.83:0/2775300666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2be81545e0 0x7f2be806e5b0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 -- 172.21.15.83:0/2775300666 >> 172.21.15.83:0/2775300666 conn(0x7f2be806bc80 msgr2=0x7f2be806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 -- 172.21.15.83:0/2775300666 shutdown_connections 2024-09-19T07:37:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 -- 172.21.15.83:0/2775300666 wait complete. 2024-09-19T07:37:10.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 Processor -- start 2024-09-19T07:37:10.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 -- start start 2024-09-19T07:37:10.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2be8071100 0x7f2be8148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:10.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be80c28e0 0x7f2be81f15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:10.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2be81545e0 0x7f2be8149e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:10.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2be81517d0 con 0x7f2be81545e0 2024-09-19T07:37:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2be8151650 con 0x7f2be80c28e0 2024-09-19T07:37:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2befae4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2be80714e0 con 0x7f2be8071100 2024-09-19T07:37:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bee081640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2be81545e0 0x7f2be8149e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bee081640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2be81545e0 0x7f2be8149e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45278/0 (socket says 172.21.15.83:45278) 2024-09-19T07:37:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bee081640 1 -- 172.21.15.83:0/1356743762 learned_addr learned my addr 172.21.15.83:0/1356743762 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bee081640 1 -- 172.21.15.83:0/1356743762 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2be8071100 msgr2=0x7f2be8148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bee081640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2be8071100 0x7f2be8148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bee081640 1 -- 172.21.15.83:0/1356743762 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be80c28e0 msgr2=0x7f2be81f15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bed07f640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be80c28e0 0x7f2be81f15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bee081640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be80c28e0 0x7f2be81f15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.083+0000 7f2bee081640 1 -- 172.21.15.83:0/1356743762 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be80aaae0 con 0x7f2be81545e0 2024-09-19T07:37:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.087+0000 7f2bee081640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2be81545e0 0x7f2be8149e30 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f2be001a510 tx=0x7f2be0019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.087+0000 7f2bde7fc640 1 -- 172.21.15.83:0/1356743762 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be00b2050 con 0x7f2be81545e0 2024-09-19T07:37:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.087+0000 7f2bde7fc640 1 -- 172.21.15.83:0/1356743762 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2be00be070 con 0x7f2be81545e0 2024-09-19T07:37:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.087+0000 7f2befae4640 1 -- 172.21.15.83:0/1356743762 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be8149340 con 0x7f2be81545e0 2024-09-19T07:37:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.087+0000 7f2bde7fc640 1 -- 172.21.15.83:0/1356743762 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be00ada70 con 0x7f2be81545e0 2024-09-19T07:37:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.087+0000 7f2befae4640 1 -- 172.21.15.83:0/1356743762 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be80aad50 con 0x7f2be81545e0 2024-09-19T07:37:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.087+0000 7f2befae4640 1 -- 172.21.15.83:0/1356743762 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be8150f40 con 0x7f2be81545e0 2024-09-19T07:37:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.087+0000 7f2bde7fc640 1 -- 172.21.15.83:0/1356743762 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2be00a5710 con 0x7f2be81545e0 2024-09-19T07:37:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.087+0000 7f2bde7fc640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2bcc0678f0 0x7f2bcc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.087+0000 7f2bde7fc640 1 -- 172.21.15.83:0/1356743762 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7f2be0122050 con 0x7f2be81545e0 2024-09-19T07:37:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.087+0000 7f2bed880640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2bcc0678f0 0x7f2bcc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:10.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.087+0000 7f2bed880640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2bcc0678f0 0x7f2bcc069db0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2be4000f70 tx=0x7f2be40079e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:10.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.091+0000 7f2bde7fc640 1 -- 172.21.15.83:0/1356743762 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2be0016560 con 0x7f2be81545e0 2024-09-19T07:37:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.283+0000 7f2befae4640 1 -- 172.21.15.83:0/1356743762 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2be80c34c0 con 0x7f2be81545e0 2024-09-19T07:37:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2bde7fc640 1 -- 172.21.15.83:0/1356743762 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7f2be00f2010 con 0x7f2be81545e0 2024-09-19T07:37:10.285 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:10.285 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2befae4640 1 -- 172.21.15.83:0/1356743762 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2bcc0678f0 msgr2=0x7f2bcc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2befae4640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2bcc0678f0 0x7f2bcc069db0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2be4000f70 tx=0x7f2be40079e0 comp rx=0 tx=0).stop 2024-09-19T07:37:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2befae4640 1 -- 172.21.15.83:0/1356743762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2be81545e0 msgr2=0x7f2be8149e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2befae4640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2be81545e0 0x7f2be8149e30 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f2be001a510 tx=0x7f2be0019680 comp rx=0 tx=0).stop 2024-09-19T07:37:10.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2befae4640 1 -- 172.21.15.83:0/1356743762 shutdown_connections 2024-09-19T07:37:10.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2befae4640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2bcc0678f0 0x7f2bcc069db0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:10.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2befae4640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2be81545e0 0x7f2be8149e30 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:10.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2befae4640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be80c28e0 0x7f2be81f15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:10.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2befae4640 1 --2- 172.21.15.83:0/1356743762 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2be8071100 0x7f2be8148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:10.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2befae4640 1 -- 172.21.15.83:0/1356743762 >> 172.21.15.83:0/1356743762 conn(0x7f2be806bc80 msgr2=0x7f2be814e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:10.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2befae4640 1 -- 172.21.15.83:0/1356743762 shutdown_connections 2024-09-19T07:37:10.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:10.287+0000 7f2befae4640 1 -- 172.21.15.83:0/1356743762 wait complete. 2024-09-19T07:37:10.303 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:37:11.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21f109640 1 Processor -- start 2024-09-19T07:37:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21f109640 1 -- start start 2024-09-19T07:37:11.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21f109640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc218071100 0x7fc2181545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:11.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21f109640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2180a9e60 con 0x7fc2180c28e0 2024-09-19T07:37:11.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21f109640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2180aa140 con 0x7fc21806dea0 2024-09-19T07:37:11.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21f109640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc218071b50 con 0x7fc218071100 2024-09-19T07:37:11.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc217fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc2180c28e0 0x7fc218150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37892/0 (socket says 172.21.15.83:37892) 2024-09-19T07:37:11.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc217fff640 1 -- 172.21.15.83:0/2382937787 learned_addr learned my addr 172.21.15.83:0/2382937787 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:37:11.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21cea5640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc218071100 0x7fc2181545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3900175353 0 0) 0x7fc2180a9e60 con 0x7fc2180c28e0 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc208003530 con 0x7fc2180c28e0 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2655377998 0 0) 0x7fc208003530 con 0x7fc2180c28e0 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc218071100 msgr2=0x7fc2181545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc218071100 0x7fc2181545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 >> v1:172.21.15.184:6789/0 conn(0x7fc21806dea0 legacy=0x7fc21806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2180c2240 con 0x7fc2180c28e0 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fc204001680 con 0x7fc2180c28e0 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 >> v1:172.21.15.83:6789/0 conn(0x7fc2180c28e0 legacy=0x7fc218150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2080039f0 0x7fc208003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc208004810 0x7fc208004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc218071100 0x7fc208005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc218071b50 con 0x7fc2080039f0 2024-09-19T07:37:11.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc20800bb00 con 0x7fc208004810 2024-09-19T07:37:11.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc20800bd00 con 0x7fc218071100 2024-09-19T07:37:11.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21cea5640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2080039f0 0x7fc208003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:11.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc217fff640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc208004810 0x7fc208004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:11.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21d6a6640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc218071100 0x7fc208005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:11.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21cea5640 1 -- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc218071100 msgr2=0x7fc208005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:11.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21cea5640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc218071100 0x7fc208005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:11.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21cea5640 1 -- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc208004810 msgr2=0x7fc208004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:11.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21cea5640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc208004810 0x7fc208004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:11.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21cea5640 1 -- 172.21.15.83:0/2382937787 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc208003530 con 0x7fc2080039f0 2024-09-19T07:37:11.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc217fff640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc208004810 0x7fc208004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:37:11.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.787+0000 7fc21d6a6640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc218071100 0x7fc208005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:37:11.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21cea5640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2080039f0 0x7fc208003e10 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc20c009ac0 tx=0x7fc20c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30dd84997ae3964b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:11.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c009690 con 0x7fc2080039f0 2024-09-19T07:37:11.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc20c0a6430 con 0x7fc2080039f0 2024-09-19T07:37:11.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc2177fe640 1 -- 172.21.15.83:0/2382937787 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c0a6800 con 0x7fc2080039f0 2024-09-19T07:37:11.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 -- 172.21.15.83:0/2382937787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2080039f0 msgr2=0x7fc208003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:11.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 --2- 172.21.15.83:0/2382937787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2080039f0 0x7fc208003e10 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc20c009ac0 tx=0x7fc20c01ca50 comp rx=0 tx=0).stop 2024-09-19T07:37:11.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21d6a6640 1 -- 172.21.15.83:0/2382937787 reap_dead start 2024-09-19T07:37:11.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 -- 172.21.15.83:0/2382937787 shutdown_connections 2024-09-19T07:37:11.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 -- 172.21.15.83:0/2382937787 >> 172.21.15.83:0/2382937787 conn(0x7fc21806bc80 msgr2=0x7fc21806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:11.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 -- 172.21.15.83:0/2382937787 shutdown_connections 2024-09-19T07:37:11.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 -- 172.21.15.83:0/2382937787 wait complete. 2024-09-19T07:37:11.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 Processor -- start 2024-09-19T07:37:11.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 -- start start 2024-09-19T07:37:11.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2080039f0 0x7fc2180ab870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:11.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2180c1b90 0x7fc218149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:11.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc21814afe0 0x7fc21814b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:11.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181517d0 con 0x7fc2180c1b90 2024-09-19T07:37:11.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc218151650 con 0x7fc2080039f0 2024-09-19T07:37:11.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21f109640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc2180714e0 con 0x7fc21814afe0 2024-09-19T07:37:11.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc217fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2180c1b90 0x7fc218149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:11.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc217fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2180c1b90 0x7fc218149290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32956/0 (socket says 172.21.15.83:32956) 2024-09-19T07:37:11.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc217fff640 1 -- 172.21.15.83:0/3733142518 learned_addr learned my addr 172.21.15.83:0/3733142518 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:11.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21cea5640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2080039f0 0x7fc2180ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:11.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.791+0000 7fc21d6a6640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc21814afe0 0x7fc21814b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:11.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc217fff640 1 -- 172.21.15.83:0/3733142518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc21814afe0 msgr2=0x7fc21814b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:11.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc217fff640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc21814afe0 0x7fc21814b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:11.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc217fff640 1 -- 172.21.15.83:0/3733142518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2080039f0 msgr2=0x7fc2180ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:11.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc217fff640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2080039f0 0x7fc2180ab870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:11.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc217fff640 1 -- 172.21.15.83:0/3733142518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc218145af0 con 0x7fc2180c1b90 2024-09-19T07:37:11.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc217fff640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2180c1b90 0x7fc218149290 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fc20400d7e0 tx=0x7fc20400dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:11.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc2157fa640 1 -- 172.21.15.83:0/3733142518 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc204019070 con 0x7fc2180c1b90 2024-09-19T07:37:11.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc2157fa640 1 -- 172.21.15.83:0/3733142518 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2040074e0 con 0x7fc2180c1b90 2024-09-19T07:37:11.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc21f109640 1 -- 172.21.15.83:0/3733142518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc218145810 con 0x7fc2180c1b90 2024-09-19T07:37:11.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc2157fa640 1 -- 172.21.15.83:0/3733142518 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc204004a10 con 0x7fc2180c1b90 2024-09-19T07:37:11.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc21f109640 1 -- 172.21.15.83:0/3733142518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2180ac380 con 0x7fc2180c1b90 2024-09-19T07:37:11.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc21f109640 1 -- 172.21.15.83:0/3733142518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc218150f40 con 0x7fc2180c1b90 2024-09-19T07:37:11.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc2157fa640 1 -- 172.21.15.83:0/3733142518 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc20401c070 con 0x7fc2180c1b90 2024-09-19T07:37:11.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc2157fa640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1fc0678c0 0x7fc1fc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:11.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc2157fa640 1 -- 172.21.15.83:0/3733142518 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7fc2040866e0 con 0x7fc2180c1b90 2024-09-19T07:37:11.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc21cea5640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1fc0678c0 0x7fc1fc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:11.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.795+0000 7fc21cea5640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1fc0678c0 0x7fc1fc069d80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0ad000 tx=0x7fc20c0b2040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:11.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.799+0000 7fc2157fa640 1 -- 172.21.15.83:0/3733142518 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc204023020 con 0x7fc2180c1b90 2024-09-19T07:37:11.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.983+0000 7fc21f109640 1 -- 172.21.15.83:0/3733142518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2180c4f30 con 0x7fc2180c1b90 2024-09-19T07:37:11.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.983+0000 7fc2157fa640 1 -- 172.21.15.83:0/3733142518 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7fc204056730 con 0x7fc2180c1b90 2024-09-19T07:37:11.983 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:11.983 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:11.983 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:11.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.987+0000 7fc21f109640 1 -- 172.21.15.83:0/3733142518 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1fc0678c0 msgr2=0x7fc1fc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:11.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.987+0000 7fc21f109640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1fc0678c0 0x7fc1fc069d80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0ad000 tx=0x7fc20c0b2040 comp rx=0 tx=0).stop 2024-09-19T07:37:11.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.987+0000 7fc21f109640 1 -- 172.21.15.83:0/3733142518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2180c1b90 msgr2=0x7fc218149290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:11.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.987+0000 7fc21f109640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2180c1b90 0x7fc218149290 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fc20400d7e0 tx=0x7fc20400dca0 comp rx=0 tx=0).stop 2024-09-19T07:37:11.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.987+0000 7fc21f109640 1 -- 172.21.15.83:0/3733142518 shutdown_connections 2024-09-19T07:37:11.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.987+0000 7fc21f109640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc1fc0678c0 0x7fc1fc069d80 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:11.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.987+0000 7fc21f109640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc21814afe0 0x7fc21814b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:11.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.987+0000 7fc21f109640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2180c1b90 0x7fc218149290 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:11.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.987+0000 7fc21f109640 1 --2- 172.21.15.83:0/3733142518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2080039f0 0x7fc2180ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:11.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.987+0000 7fc21f109640 1 -- 172.21.15.83:0/3733142518 >> 172.21.15.83:0/3733142518 conn(0x7fc21806bc80 msgr2=0x7fc218152ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:11.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.987+0000 7fc21f109640 1 -- 172.21.15.83:0/3733142518 shutdown_connections 2024-09-19T07:37:11.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:11.987+0000 7fc21f109640 1 -- 172.21.15.83:0/3733142518 wait complete. 2024-09-19T07:37:12.006 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:37:12.304 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:12.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.431+0000 7f32122f8640 1 Processor -- start 2024-09-19T07:37:12.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.431+0000 7f32122f8640 1 -- start start 2024-09-19T07:37:12.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f32122f8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c071100 0x7f320c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:12.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f32122f8640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f320c0c28e0 0x7f320c150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:12.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f32122f8640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f320c0a9e60 con 0x7f320c151480 2024-09-19T07:37:12.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f32122f8640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f320c071b50 con 0x7f320c0c28e0 2024-09-19T07:37:12.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f32122f8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f320c0c1ff0 con 0x7f320c071100 2024-09-19T07:37:12.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f320b7fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f320c0c28e0 0x7f320c150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:12.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f320bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c071100 0x7f320c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:12.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f320b7fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f320c0c28e0 0x7f320c150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:41646/0 (socket says 172.21.15.83:41646) 2024-09-19T07:37:12.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f320b7fe640 1 -- 172.21.15.83:0/3340944765 learned_addr learned my addr 172.21.15.83:0/3340944765 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:12.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f320bfff640 1 -- 172.21.15.83:0/3340944765 >> v2:172.21.15.184:3300/0 conn(0x7f320c0c28e0 msgr2=0x7f320c150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:12.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f320bfff640 1 --2- 172.21.15.83:0/3340944765 >> v2:172.21.15.184:3300/0 conn(0x7f320c0c28e0 0x7f320c150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:12.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f320bfff640 1 -- 172.21.15.83:0/3340944765 >> v1:172.21.15.184:6789/0 conn(0x7f320c151480 legacy=0x7f320c06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:12.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f320bfff640 1 -- 172.21.15.83:0/3340944765 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f320c0aa140 con 0x7f320c071100 2024-09-19T07:37:12.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.435+0000 7f320bfff640 1 --2- 172.21.15.83:0/3340944765 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c071100 0x7f320c1545e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0097c0 tx=0x7f31fc01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54199b5f69ea24f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:12.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320affd640 1 -- 172.21.15.83:0/3340944765 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0b0070 con 0x7f320c071100 2024-09-19T07:37:12.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320affd640 1 -- 172.21.15.83:0/3340944765 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31fc0ab600 con 0x7f320c071100 2024-09-19T07:37:12.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320affd640 1 -- 172.21.15.83:0/3340944765 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0ab9d0 con 0x7f320c071100 2024-09-19T07:37:12.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 -- 172.21.15.83:0/3340944765 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c071100 msgr2=0x7f320c1545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:12.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 --2- 172.21.15.83:0/3340944765 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c071100 0x7f320c1545e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0097c0 tx=0x7f31fc01c410 comp rx=0 tx=0).stop 2024-09-19T07:37:12.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 -- 172.21.15.83:0/3340944765 shutdown_connections 2024-09-19T07:37:12.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 --2- 172.21.15.83:0/3340944765 >> v2:172.21.15.184:3300/0 conn(0x7f320c0c28e0 0x7f320c150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:12.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 --2- 172.21.15.83:0/3340944765 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c071100 0x7f320c1545e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:12.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 -- 172.21.15.83:0/3340944765 >> 172.21.15.83:0/3340944765 conn(0x7f320c06bc80 msgr2=0x7f320c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:12.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 -- 172.21.15.83:0/3340944765 shutdown_connections 2024-09-19T07:37:12.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 -- 172.21.15.83:0/3340944765 wait complete. 2024-09-19T07:37:12.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 Processor -- start 2024-09-19T07:37:12.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 -- start start 2024-09-19T07:37:12.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c071100 0x7f320c144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:12.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0c28e0 0x7f320c151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:12.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f320c151480 0x7f320c145dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:12.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f320c071660 con 0x7f320c0c28e0 2024-09-19T07:37:12.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f320c0714e0 con 0x7f320c151480 2024-09-19T07:37:12.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f320c160c60 con 0x7f320c071100 2024-09-19T07:37:12.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0c28e0 0x7f320c151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:12.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0c28e0 0x7f320c151c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32972/0 (socket says 172.21.15.83:32972) 2024-09-19T07:37:12.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320b7fe640 1 -- 172.21.15.83:0/1226671186 learned_addr learned my addr 172.21.15.83:0/1226671186 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:12.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f3210895640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f320c151480 0x7f320c145dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:12.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320bfff640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c071100 0x7f320c144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:12.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320b7fe640 1 -- 172.21.15.83:0/1226671186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c071100 msgr2=0x7f320c144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:12.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320b7fe640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c071100 0x7f320c144910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:12.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320b7fe640 1 -- 172.21.15.83:0/1226671186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f320c151480 msgr2=0x7f320c145dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:12.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320b7fe640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f320c151480 0x7f320c145dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:12.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320b7fe640 1 -- 172.21.15.83:0/1226671186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f320c153ce0 con 0x7f320c0c28e0 2024-09-19T07:37:12.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f320b7fe640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0c28e0 0x7f320c151c30 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f31f400e820 tx=0x7f31f400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:12.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f3208ff9640 1 -- 172.21.15.83:0/1226671186 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f4017070 con 0x7f320c0c28e0 2024-09-19T07:37:12.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f3208ff9640 1 -- 172.21.15.83:0/1226671186 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31f400c940 con 0x7f320c0c28e0 2024-09-19T07:37:12.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f3208ff9640 1 -- 172.21.15.83:0/1226671186 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f400ccf0 con 0x7f320c0c28e0 2024-09-19T07:37:12.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 -- 172.21.15.83:0/1226671186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f320c1474b0 con 0x7f320c0c28e0 2024-09-19T07:37:12.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.439+0000 7f32122f8640 1 -- 172.21.15.83:0/1226671186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f320c1458b0 con 0x7f320c0c28e0 2024-09-19T07:37:12.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.443+0000 7f32122f8640 1 -- 172.21.15.83:0/1226671186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f320c153830 con 0x7f320c0c28e0 2024-09-19T07:37:12.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.443+0000 7f3208ff9640 1 -- 172.21.15.83:0/1226671186 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f31f401a070 con 0x7f320c0c28e0 2024-09-19T07:37:12.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.443+0000 7f3208ff9640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f31ec0678f0 0x7f31ec069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:12.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.443+0000 7f3208ff9640 1 -- 172.21.15.83:0/1226671186 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7f31f4086700 con 0x7f320c0c28e0 2024-09-19T07:37:12.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.443+0000 7f320bfff640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f31ec0678f0 0x7f31ec069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:12.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.443+0000 7f320bfff640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f31ec0678f0 0x7f31ec069db0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0ad000 tx=0x7f31fc0a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:12.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.447+0000 7f3208ff9640 1 -- 172.21.15.83:0/1226671186 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f31f40566d0 con 0x7f320c0c28e0 2024-09-19T07:37:12.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.643+0000 7f32122f8640 1 -- 172.21.15.83:0/1226671186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f320c1545e0 con 0x7f320c0c28e0 2024-09-19T07:37:12.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f3208ff9640 1 -- 172.21.15.83:0/1226671186 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7f31f4055fd0 con 0x7f320c0c28e0 2024-09-19T07:37:12.645 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:12.645 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:12.645 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f32122f8640 1 -- 172.21.15.83:0/1226671186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f31ec0678f0 msgr2=0x7f31ec069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f32122f8640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f31ec0678f0 0x7f31ec069db0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0ad000 tx=0x7f31fc0a6000 comp rx=0 tx=0).stop 2024-09-19T07:37:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f32122f8640 1 -- 172.21.15.83:0/1226671186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0c28e0 msgr2=0x7f320c151c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f32122f8640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0c28e0 0x7f320c151c30 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f31f400e820 tx=0x7f31f400ece0 comp rx=0 tx=0).stop 2024-09-19T07:37:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f32122f8640 1 -- 172.21.15.83:0/1226671186 shutdown_connections 2024-09-19T07:37:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f32122f8640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f31ec0678f0 0x7f31ec069db0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f32122f8640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f320c151480 0x7f320c145dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f32122f8640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f320c0c28e0 0x7f320c151c30 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f32122f8640 1 --2- 172.21.15.83:0/1226671186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f320c071100 0x7f320c144910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:12.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f32122f8640 1 -- 172.21.15.83:0/1226671186 >> 172.21.15.83:0/1226671186 conn(0x7f320c06bc80 msgr2=0x7f320c14f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:12.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f32122f8640 1 -- 172.21.15.83:0/1226671186 shutdown_connections 2024-09-19T07:37:12.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:12.647+0000 7f32122f8640 1 -- 172.21.15.83:0/1226671186 wait complete. 2024-09-19T07:37:12.663 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:37:14.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:14.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca080ad640 1 Processor -- start 2024-09-19T07:37:14.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca080ad640 1 -- start start 2024-09-19T07:37:14.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca080ad640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fca000c28e0 0x7fca001545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca080ad640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0006dea0 0x7fca0006e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca080ad640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca000a9e60 con 0x7fca00071100 2024-09-19T07:37:14.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca080ad640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fca00071b50 con 0x7fca000c28e0 2024-09-19T07:37:14.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca080ad640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fca000c1ff0 con 0x7fca0006dea0 2024-09-19T07:37:14.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca05e49640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fca00071100 0x7fca00150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37898/0 (socket says 172.21.15.83:37898) 2024-09-19T07:37:14.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca05e49640 1 -- 172.21.15.83:0/310008265 learned_addr learned my addr 172.21.15.83:0/310008265 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:37:14.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca05648640 1 --2- 172.21.15.83:0/310008265 >> v2:172.21.15.184:3300/0 conn(0x7fca000c28e0 0x7fca001545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca0664a640 1 --2- 172.21.15.83:0/310008265 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0006dea0 0x7fca0006e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1244000235 0 0) 0x7fca000a9e60 con 0x7fca00071100 2024-09-19T07:37:14.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e8003530 con 0x7fca00071100 2024-09-19T07:37:14.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2090347434 0 0) 0x7fc9e8003530 con 0x7fca00071100 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0006dea0 msgr2=0x7fca0006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca04e47640 1 --2- 172.21.15.83:0/310008265 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0006dea0 0x7fca0006e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 >> v2:172.21.15.184:3300/0 conn(0x7fca000c28e0 msgr2=0x7fca001545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca04e47640 1 --2- 172.21.15.83:0/310008265 >> v2:172.21.15.184:3300/0 conn(0x7fca000c28e0 0x7fca001545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca000aa140 con 0x7fca00071100 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.143+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fc9f0001a00 con 0x7fca00071100 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 >> v1:172.21.15.83:6789/0 conn(0x7fca00071100 legacy=0x7fca00150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca04e47640 1 --2- 172.21.15.83:0/310008265 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0006dea0 0x7fc9e80039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca04e47640 1 --2- 172.21.15.83:0/310008265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e8003f40 0x7fc9e8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca04e47640 1 --2- 172.21.15.83:0/310008265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e8004d50 0x7fc9e80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fca00071b50 con 0x7fc9e8003f40 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fca000c1ff0 con 0x7fc9e8004d50 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e800ba20 con 0x7fca0006dea0 2024-09-19T07:37:14.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca05648640 1 --2- 172.21.15.83:0/310008265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e8003f40 0x7fc9e8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca05e49640 1 --2- 172.21.15.83:0/310008265 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0006dea0 0x7fc9e80039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca05648640 1 -- 172.21.15.83:0/310008265 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0006dea0 msgr2=0x7fc9e80039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:14.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca05648640 1 --2- 172.21.15.83:0/310008265 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0006dea0 0x7fc9e80039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca05648640 1 -- 172.21.15.83:0/310008265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e8004d50 msgr2=0x7fc9e80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:14.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca0664a640 1 --2- 172.21.15.83:0/310008265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e8004d50 0x7fc9e80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca05648640 1 --2- 172.21.15.83:0/310008265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e8004d50 0x7fc9e80051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca05648640 1 -- 172.21.15.83:0/310008265 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e8003530 con 0x7fc9e8003f40 2024-09-19T07:37:14.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca05648640 1 --2- 172.21.15.83:0/310008265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e8003f40 0x7fc9e8004810 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009b10 tx=0x7fc9f401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89987add4cf4f390 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:14.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f40096b0 con 0x7fc9e8003f40 2024-09-19T07:37:14.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f40af070 con 0x7fc9e8003f40 2024-09-19T07:37:14.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca04e47640 1 -- 172.21.15.83:0/310008265 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f40aa6a0 con 0x7fc9e8003f40 2024-09-19T07:37:14.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca080ad640 1 -- 172.21.15.83:0/310008265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e8003f40 msgr2=0x7fc9e8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:14.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca080ad640 1 --2- 172.21.15.83:0/310008265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9e8003f40 0x7fc9e8004810 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009b10 tx=0x7fc9f401c7f0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca0664a640 1 -- 172.21.15.83:0/310008265 reap_dead start 2024-09-19T07:37:14.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca080ad640 1 -- 172.21.15.83:0/310008265 shutdown_connections 2024-09-19T07:37:14.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca080ad640 1 -- 172.21.15.83:0/310008265 >> 172.21.15.83:0/310008265 conn(0x7fca0006bc80 msgr2=0x7fca0006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:14.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca080ad640 1 -- 172.21.15.83:0/310008265 shutdown_connections 2024-09-19T07:37:14.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.147+0000 7fca080ad640 1 -- 172.21.15.83:0/310008265 wait complete. 2024-09-19T07:37:14.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca080ad640 1 Processor -- start 2024-09-19T07:37:14.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca080ad640 1 -- start start 2024-09-19T07:37:14.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca080ad640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e8003f40 0x7fca001ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca080ad640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0015b520 0x7fca000abbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca080ad640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca001579a0 0x7fca000c0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca080ad640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fca00071660 con 0x7fca001579a0 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca080ad640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fca000714e0 con 0x7fc9e8003f40 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca080ad640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fca0015e440 con 0x7fca0015b520 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca0664a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca001579a0 0x7fca000c0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca0664a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca001579a0 0x7fca000c0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32998/0 (socket says 172.21.15.83:32998) 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca0664a640 1 -- 172.21.15.83:0/2534936157 learned_addr learned my addr 172.21.15.83:0/2534936157 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca05e49640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e8003f40 0x7fca001ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca05648640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0015b520 0x7fca000abbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca0664a640 1 -- 172.21.15.83:0/2534936157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0015b520 msgr2=0x7fca000abbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca0664a640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0015b520 0x7fca000abbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca0664a640 1 -- 172.21.15.83:0/2534936157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e8003f40 msgr2=0x7fca001ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca0664a640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e8003f40 0x7fca001ee230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca0664a640 1 -- 172.21.15.83:0/2534936157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca001f21b0 con 0x7fca001579a0 2024-09-19T07:37:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca05648640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0015b520 0x7fca000abbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:37:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca0664a640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca001579a0 0x7fca000c0a20 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fca0006cd60 tx=0x7fc9f8010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fc9fe7fc640 1 -- 172.21.15.83:0/2534936157 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f800f040 con 0x7fca001579a0 2024-09-19T07:37:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fc9fe7fc640 1 -- 172.21.15.83:0/2534936157 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f800e840 con 0x7fca001579a0 2024-09-19T07:37:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca080ad640 1 -- 172.21.15.83:0/2534936157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca001f2490 con 0x7fca001579a0 2024-09-19T07:37:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fc9fe7fc640 1 -- 172.21.15.83:0/2534936157 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f800ec30 con 0x7fca001579a0 2024-09-19T07:37:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca080ad640 1 -- 172.21.15.83:0/2534936157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca00156950 con 0x7fca001579a0 2024-09-19T07:37:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.151+0000 7fca080ad640 1 -- 172.21.15.83:0/2534936157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca001545e0 con 0x7fca001579a0 2024-09-19T07:37:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.155+0000 7fc9fe7fc640 1 -- 172.21.15.83:0/2534936157 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc9f801c400 con 0x7fca001579a0 2024-09-19T07:37:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.155+0000 7fc9fe7fc640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9ec0678c0 0x7fc9ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.155+0000 7fc9fe7fc640 1 -- 172.21.15.83:0/2534936157 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7fc9f8086be0 con 0x7fca001579a0 2024-09-19T07:37:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.155+0000 7fca05e49640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9ec0678c0 0x7fc9ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.155+0000 7fca05e49640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9ec0678c0 0x7fc9ec069d80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0001340 tx=0x7fc9f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.159+0000 7fc9fe7fc640 1 -- 172.21.15.83:0/2534936157 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc9f8018030 con 0x7fca001579a0 2024-09-19T07:37:14.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.351+0000 7fca080ad640 1 -- 172.21.15.83:0/2534936157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca000c4f30 con 0x7fca001579a0 2024-09-19T07:37:14.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.351+0000 7fc9fe7fc640 1 -- 172.21.15.83:0/2534936157 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7fc9f8056560 con 0x7fca001579a0 2024-09-19T07:37:14.352 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:14.352 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:14.352 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.355+0000 7fca080ad640 1 -- 172.21.15.83:0/2534936157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9ec0678c0 msgr2=0x7fc9ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.355+0000 7fca080ad640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9ec0678c0 0x7fc9ec069d80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0001340 tx=0x7fc9f0009040 comp rx=0 tx=0).stop 2024-09-19T07:37:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.355+0000 7fca080ad640 1 -- 172.21.15.83:0/2534936157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca001579a0 msgr2=0x7fca000c0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.355+0000 7fca080ad640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca001579a0 0x7fca000c0a20 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fca0006cd60 tx=0x7fc9f8010c20 comp rx=0 tx=0).stop 2024-09-19T07:37:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.355+0000 7fca080ad640 1 -- 172.21.15.83:0/2534936157 shutdown_connections 2024-09-19T07:37:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.355+0000 7fca080ad640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9ec0678c0 0x7fc9ec069d80 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.355+0000 7fca080ad640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca001579a0 0x7fca000c0a20 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.355+0000 7fca080ad640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca0015b520 0x7fca000abbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.355+0000 7fca080ad640 1 --2- 172.21.15.83:0/2534936157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9e8003f40 0x7fca001ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.355+0000 7fca080ad640 1 -- 172.21.15.83:0/2534936157 >> 172.21.15.83:0/2534936157 conn(0x7fca0006bc80 msgr2=0x7fca000c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.355+0000 7fca080ad640 1 -- 172.21.15.83:0/2534936157 shutdown_connections 2024-09-19T07:37:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.355+0000 7fca080ad640 1 -- 172.21.15.83:0/2534936157 wait complete. 2024-09-19T07:37:14.370 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:37:14.665 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:14.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f073828a640 1 Processor -- start 2024-09-19T07:37:14.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f073828a640 1 -- start start 2024-09-19T07:37:14.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f073828a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f07300c6120 0x7f07300c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f073828a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0730156200 con 0x7f07301592f0 2024-09-19T07:37:14.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f073828a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07301564e0 con 0x7f0730155d70 2024-09-19T07:37:14.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f073828a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f07300c9930 con 0x7f07300c6120 2024-09-19T07:37:14.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0736827640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f07301592f0 0x7f0730157ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37904/0 (socket says 172.21.15.83:37904) 2024-09-19T07:37:14.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0736827640 1 -- 172.21.15.83:0/3410870291 learned_addr learned my addr 172.21.15.83:0/3410870291 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:37:14.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0736026640 1 --2- 172.21.15.83:0/3410870291 >> v2:172.21.15.184:3300/0 conn(0x7f07300c6120 0x7f07300c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3211148947 0 0) 0x7f0730156200 con 0x7f07301592f0 2024-09-19T07:37:14.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f071c003530 con 0x7f07301592f0 2024-09-19T07:37:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1396158769 0 0) 0x7f071c003530 con 0x7f07301592f0 2024-09-19T07:37:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 >> v2:172.21.15.184:3300/0 conn(0x7f07300c6120 msgr2=0x7f07300c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 --2- 172.21.15.83:0/3410870291 >> v2:172.21.15.184:3300/0 conn(0x7f07300c6120 0x7f07300c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 >> v1:172.21.15.184:6789/0 conn(0x7f0730155d70 legacy=0x7f07301575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07301567c0 con 0x7f07301592f0 2024-09-19T07:37:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f0728001a00 con 0x7f07301592f0 2024-09-19T07:37:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0736026640 1 --2- 172.21.15.83:0/3410870291 >> v2:172.21.15.184:3300/0 conn(0x7f07300c6120 0x7f07300c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:37:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 >> v1:172.21.15.83:6789/0 conn(0x7f07301592f0 legacy=0x7f0730157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 --2- 172.21.15.83:0/3410870291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f071c0039c0 0x7f071c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 --2- 172.21.15.83:0/3410870291 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f071c004810 0x7f071c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 --2- 172.21.15.83:0/3410870291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f071c0051d0 0x7f071c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f071c00c140 con 0x7f071c0051d0 2024-09-19T07:37:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f071c00c2f0 con 0x7f071c0039c0 2024-09-19T07:37:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f071c00c4f0 con 0x7f071c004810 2024-09-19T07:37:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0736827640 1 --2- 172.21.15.83:0/3410870291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f071c0051d0 0x7f071c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0736026640 1 --2- 172.21.15.83:0/3410870291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f071c0039c0 0x7f071c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0735825640 1 --2- 172.21.15.83:0/3410870291 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f071c004810 0x7f071c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0736827640 1 -- 172.21.15.83:0/3410870291 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f071c004810 msgr2=0x7f071c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0736827640 1 --2- 172.21.15.83:0/3410870291 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f071c004810 0x7f071c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0736827640 1 -- 172.21.15.83:0/3410870291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f071c0039c0 msgr2=0x7f071c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0736827640 1 --2- 172.21.15.83:0/3410870291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f071c0039c0 0x7f071c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0736827640 1 -- 172.21.15.83:0/3410870291 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071c003530 con 0x7f071c0051d0 2024-09-19T07:37:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.795+0000 7f0736026640 1 --2- 172.21.15.83:0/3410870291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f071c0039c0 0x7f071c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:37:14.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f0736827640 1 -- 172.21.15.83:0/3410870291 reap_dead start 2024-09-19T07:37:14.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f0736827640 1 --2- 172.21.15.83:0/3410870291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f071c0051d0 0x7f071c00b9d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f072800a860 tx=0x7f072801fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237aa607846a7c5c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:14.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072800a480 con 0x7f071c0051d0 2024-09-19T07:37:14.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07280aa510 con 0x7f071c0051d0 2024-09-19T07:37:14.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f0735024640 1 -- 172.21.15.83:0/3410870291 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07280aa8e0 con 0x7f071c0051d0 2024-09-19T07:37:14.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 -- 172.21.15.83:0/3410870291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f071c0051d0 msgr2=0x7f071c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:14.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 --2- 172.21.15.83:0/3410870291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f071c0051d0 0x7f071c00b9d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f072800a860 tx=0x7f072801fae0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 -- 172.21.15.83:0/3410870291 shutdown_connections 2024-09-19T07:37:14.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 --2- 172.21.15.83:0/3410870291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f071c0051d0 0x7f071c00b9d0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 -- 172.21.15.83:0/3410870291 >> 172.21.15.83:0/3410870291 conn(0x7f073006bc80 msgr2=0x7f073006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:14.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 -- 172.21.15.83:0/3410870291 shutdown_connections 2024-09-19T07:37:14.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 -- 172.21.15.83:0/3410870291 wait complete. 2024-09-19T07:37:14.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 Processor -- start 2024-09-19T07:37:14.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 -- start start 2024-09-19T07:37:14.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f071c0051d0 0x7f073014b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07300ac450 0x7f07300c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07300c1d00 0x7f07301ef0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0730153860 con 0x7f07300ac450 2024-09-19T07:37:14.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f07300cc630 con 0x7f07300c1d00 2024-09-19T07:37:14.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f073828a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f07300acd10 con 0x7f071c0051d0 2024-09-19T07:37:14.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.799+0000 7f0735825640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07300ac450 0x7f07300c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0735825640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07300ac450 0x7f07300c17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33024/0 (socket says 172.21.15.83:33024) 2024-09-19T07:37:14.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0735825640 1 -- 172.21.15.83:0/3540277963 learned_addr learned my addr 172.21.15.83:0/3540277963 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0736827640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07300c1d00 0x7f07301ef0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0736026640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f071c0051d0 0x7f073014b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0735825640 1 -- 172.21.15.83:0/3540277963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f071c0051d0 msgr2=0x7f073014b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0735825640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f071c0051d0 0x7f073014b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0735825640 1 -- 172.21.15.83:0/3540277963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07300c1d00 msgr2=0x7f07301ef0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0735825640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07300c1d00 0x7f07301ef0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0735825640 1 -- 172.21.15.83:0/3540277963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0730152190 con 0x7f07300ac450 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0735825640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07300ac450 0x7f07300c17c0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f072000e410 tx=0x7f072000e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0726ffd640 1 -- 172.21.15.83:0/3540277963 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720019070 con 0x7f07300ac450 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0726ffd640 1 -- 172.21.15.83:0/3540277963 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07200052e0 con 0x7f07300ac450 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f073828a640 1 -- 172.21.15.83:0/3540277963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073014cbe0 con 0x7f07300ac450 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0726ffd640 1 -- 172.21.15.83:0/3540277963 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720003800 con 0x7f07300ac450 2024-09-19T07:37:14.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f073828a640 1 -- 172.21.15.83:0/3540277963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07300ad220 con 0x7f07300ac450 2024-09-19T07:37:14.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f073828a640 1 -- 172.21.15.83:0/3540277963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07301561c0 con 0x7f07300ac450 2024-09-19T07:37:14.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0726ffd640 1 -- 172.21.15.83:0/3540277963 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f072001c070 con 0x7f07300ac450 2024-09-19T07:37:14.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0726ffd640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07140678f0 0x7f0714069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:14.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0736026640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07140678f0 0x7f0714069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:14.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0726ffd640 1 -- 172.21.15.83:0/3540277963 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7f0720085d80 con 0x7f07300ac450 2024-09-19T07:37:14.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.803+0000 7f0736026640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07140678f0 0x7f0714069db0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f072c009aa0 tx=0x7f072c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:14.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:14.807+0000 7f0726ffd640 1 -- 172.21.15.83:0/3540277963 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f0720055700 con 0x7f07300ac450 2024-09-19T07:37:15.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 -- 172.21.15.83:0/3540277963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07300c96a0 con 0x7f07300ac450 2024-09-19T07:37:15.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f0726ffd640 1 -- 172.21.15.83:0/3540277963 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7f0720059a00 con 0x7f07300ac450 2024-09-19T07:37:15.001 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:15.001 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:15.001 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 -- 172.21.15.83:0/3540277963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07140678f0 msgr2=0x7f0714069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07140678f0 0x7f0714069db0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f072c009aa0 tx=0x7f072c008040 comp rx=0 tx=0).stop 2024-09-19T07:37:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 -- 172.21.15.83:0/3540277963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07300ac450 msgr2=0x7f07300c17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07300ac450 0x7f07300c17c0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f072000e410 tx=0x7f072000e8d0 comp rx=0 tx=0).stop 2024-09-19T07:37:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 -- 172.21.15.83:0/3540277963 shutdown_connections 2024-09-19T07:37:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07140678f0 0x7f0714069db0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07300c1d00 0x7f07301ef0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07300ac450 0x7f07300c17c0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 --2- 172.21.15.83:0/3540277963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f071c0051d0 0x7f073014b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:15.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 -- 172.21.15.83:0/3540277963 >> 172.21.15.83:0/3540277963 conn(0x7f073006bc80 msgr2=0x7f0730158480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:15.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 -- 172.21.15.83:0/3540277963 shutdown_connections 2024-09-19T07:37:15.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:15.003+0000 7f073828a640 1 -- 172.21.15.83:0/3540277963 wait complete. 2024-09-19T07:37:15.018 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:37:16.371 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:16.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbe2e2f640 1 Processor -- start 2024-09-19T07:37:16.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbe2e2f640 1 -- start start 2024-09-19T07:37:16.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbe2e2f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbbdc0c6120 0x7fbbdc0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:16.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbe2e2f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbbdc155d70 0x7fbbdc1575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:16.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbe2e2f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbdc1592f0 0x7fbbdc157b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:16.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbe2e2f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fbbdc0c9930 con 0x7fbbdc155d70 2024-09-19T07:37:16.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbe2e2f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbbdc156200 con 0x7fbbdc0c6120 2024-09-19T07:37:16.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbe2e2f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbbdc1564e0 con 0x7fbbdc1592f0 2024-09-19T07:37:16.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbdbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbbdc155d70 0x7fbbdc1575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:16.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbdbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbbdc155d70 0x7fbbdc1575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33034/0 (socket says 172.21.15.83:33034) 2024-09-19T07:37:16.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbdbfff640 1 -- 172.21.15.83:0/1866823372 learned_addr learned my addr 172.21.15.83:0/1866823372 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:16.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbe13cc640 1 --2- 172.21.15.83:0/1866823372 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbdc1592f0 0x7fbbdc157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:16.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbdbfff640 1 -- 172.21.15.83:0/1866823372 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbdc1592f0 msgr2=0x7fbbdc157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:16.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbe0bcb640 1 --2- 172.21.15.83:0/1866823372 >> v2:172.21.15.184:3300/0 conn(0x7fbbdc0c6120 0x7fbbdc0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:16.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbdbfff640 1 --2- 172.21.15.83:0/1866823372 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbdc1592f0 0x7fbbdc157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:16.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbdbfff640 1 -- 172.21.15.83:0/1866823372 >> v2:172.21.15.184:3300/0 conn(0x7fbbdc0c6120 msgr2=0x7fbbdc0c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:16.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbdbfff640 1 --2- 172.21.15.83:0/1866823372 >> v2:172.21.15.184:3300/0 conn(0x7fbbdc0c6120 0x7fbbdc0c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:16.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbdbfff640 1 -- 172.21.15.83:0/1866823372 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbdc1567c0 con 0x7fbbdc155d70 2024-09-19T07:37:16.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.507+0000 7fbbe13cc640 1 --2- 172.21.15.83:0/1866823372 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbdc1592f0 0x7fbbdc157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:37:16.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbdbfff640 1 --2- 172.21.15.83:0/1866823372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbdc155d70 0x7fbbdc1575d0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc08d540 tx=0x7fbbcc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62da1ca62fee0a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:16.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbdb7fe640 1 -- 172.21.15.83:0/1866823372 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc0b0070 con 0x7fbbdc155d70 2024-09-19T07:37:16.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbdb7fe640 1 -- 172.21.15.83:0/1866823372 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbcc0ab600 con 0x7fbbdc155d70 2024-09-19T07:37:16.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbdb7fe640 1 -- 172.21.15.83:0/1866823372 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc0ab9d0 con 0x7fbbdc155d70 2024-09-19T07:37:16.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/1866823372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbdc155d70 msgr2=0x7fbbdc1575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:16.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 --2- 172.21.15.83:0/1866823372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbdc155d70 0x7fbbdc1575d0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc08d540 tx=0x7fbbcc01c520 comp rx=0 tx=0).stop 2024-09-19T07:37:16.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/1866823372 shutdown_connections 2024-09-19T07:37:16.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 --2- 172.21.15.83:0/1866823372 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbdc1592f0 0x7fbbdc157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:16.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 --2- 172.21.15.83:0/1866823372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbdc155d70 0x7fbbdc1575d0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:16.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 --2- 172.21.15.83:0/1866823372 >> v2:172.21.15.184:3300/0 conn(0x7fbbdc0c6120 0x7fbbdc0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:16.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/1866823372 >> 172.21.15.83:0/1866823372 conn(0x7fbbdc06bc80 msgr2=0x7fbbdc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:16.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/1866823372 shutdown_connections 2024-09-19T07:37:16.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/1866823372 wait complete. 2024-09-19T07:37:16.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 Processor -- start 2024-09-19T07:37:16.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 -- start start 2024-09-19T07:37:16.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbdc0c6120 0x7fbbdc0ca730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:16.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbdc1592f0 0x7fbbdc0cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:16.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbdc0cb9f0 0x7fbbdc149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:16.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbbdc0c8b40 con 0x7fbbdc0c6120 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbbdc0c89c0 con 0x7fbbdc1592f0 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe2e2f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbbdc0c8cc0 con 0x7fbbdc0cb9f0 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe0bcb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbdc0c6120 0x7fbbdc0ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe0bcb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbdc0c6120 0x7fbbdc0ca730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33044/0 (socket says 172.21.15.83:33044) 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.511+0000 7fbbe0bcb640 1 -- 172.21.15.83:0/3657705307 learned_addr learned my addr 172.21.15.83:0/3657705307 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbe13cc640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbdc0cb9f0 0x7fbbdc149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbdbfff640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbdc1592f0 0x7fbbdc0cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbe0bcb640 1 -- 172.21.15.83:0/3657705307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbdc0cb9f0 msgr2=0x7fbbdc149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbe0bcb640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbdc0cb9f0 0x7fbbdc149a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbe0bcb640 1 -- 172.21.15.83:0/3657705307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbdc1592f0 msgr2=0x7fbbdc0cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbe0bcb640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbdc1592f0 0x7fbbdc0cb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbe0bcb640 1 -- 172.21.15.83:0/3657705307 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbdc144f40 con 0x7fbbdc0c6120 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbe13cc640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbdc0cb9f0 0x7fbbdc149a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:37:16.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbe0bcb640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbdc0c6120 0x7fbbdc0ca730 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000c7f0 tx=0x7fbbd000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:16.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbd97fa640 1 -- 172.21.15.83:0/3657705307 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0015070 con 0x7fbbdc0c6120 2024-09-19T07:37:16.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbd97fa640 1 -- 172.21.15.83:0/3657705307 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbd0004220 con 0x7fbbdc0c6120 2024-09-19T07:37:16.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/3657705307 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbdc145220 con 0x7fbbdc0c6120 2024-09-19T07:37:16.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbd97fa640 1 -- 172.21.15.83:0/3657705307 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0003620 con 0x7fbbdc0c6120 2024-09-19T07:37:16.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/3657705307 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbdc149240 con 0x7fbbdc0c6120 2024-09-19T07:37:16.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/3657705307 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbdc0c6500 con 0x7fbbdc0c6120 2024-09-19T07:37:16.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbd97fa640 1 -- 172.21.15.83:0/3657705307 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbbd00039d0 con 0x7fbbdc0c6120 2024-09-19T07:37:16.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbd97fa640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbbc80678c0 0x7fbbc8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:16.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbd97fa640 1 -- 172.21.15.83:0/3657705307 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7fbbd00868d0 con 0x7fbbdc0c6120 2024-09-19T07:37:16.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbdbfff640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbbc80678c0 0x7fbbc8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:16.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.515+0000 7fbbdbfff640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbbc80678c0 0x7fbbc8069d80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc0ad000 tx=0x7fbbcc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:16.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.519+0000 7fbbd97fa640 1 -- 172.21.15.83:0/3657705307 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbbd0056810 con 0x7fbbdc0c6120 2024-09-19T07:37:16.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.715+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/3657705307 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbbdc0c96a0 con 0x7fbbdc0c6120 2024-09-19T07:37:16.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.715+0000 7fbbd97fa640 1 -- 172.21.15.83:0/3657705307 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7fbbd0056110 con 0x7fbbdc0c6120 2024-09-19T07:37:16.710 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:16.710 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:16.710 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:16.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.719+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/3657705307 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbbc80678c0 msgr2=0x7fbbc8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:16.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.719+0000 7fbbe2e2f640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbbc80678c0 0x7fbbc8069d80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc0ad000 tx=0x7fbbcc008040 comp rx=0 tx=0).stop 2024-09-19T07:37:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.719+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/3657705307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbdc0c6120 msgr2=0x7fbbdc0ca730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.719+0000 7fbbe2e2f640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbdc0c6120 0x7fbbdc0ca730 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000c7f0 tx=0x7fbbd000ccb0 comp rx=0 tx=0).stop 2024-09-19T07:37:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.719+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/3657705307 shutdown_connections 2024-09-19T07:37:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.719+0000 7fbbe2e2f640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbbc80678c0 0x7fbbc8069d80 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.719+0000 7fbbe2e2f640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbdc0cb9f0 0x7fbbdc149a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.719+0000 7fbbe2e2f640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbdc1592f0 0x7fbbdc0cb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.719+0000 7fbbe2e2f640 1 --2- 172.21.15.83:0/3657705307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbdc0c6120 0x7fbbdc0ca730 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.719+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/3657705307 >> 172.21.15.83:0/3657705307 conn(0x7fbbdc06bc80 msgr2=0x7fbbdc14f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.719+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/3657705307 shutdown_connections 2024-09-19T07:37:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:16.719+0000 7fbbe2e2f640 1 -- 172.21.15.83:0/3657705307 wait complete. 2024-09-19T07:37:16.730 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:37:17.020 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:17.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc3ca80640 1 Processor -- start 2024-09-19T07:37:17.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc3ca80640 1 -- start start 2024-09-19T07:37:17.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc3ca80640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbc38150290 0x7fbc380c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc3ca80640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc38152b20 0x7fbc380c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc3ca80640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbc380c45c0 0x7fbc38153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc3ca80640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fbc380716c0 con 0x7fbc380c45c0 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc3ca80640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbc380c3920 con 0x7fbc38150290 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc3ca80640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbc380c3490 con 0x7fbc38152b20 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc36d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbc380c45c0 0x7fbc38153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc36d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbc380c45c0 0x7fbc38153eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33052/0 (socket says 172.21.15.83:33052) 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc36d9d640 1 -- 172.21.15.83:0/1424707153 learned_addr learned my addr 172.21.15.83:0/1424707153 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc2dd9b640 1 --2- 172.21.15.83:0/1424707153 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc38152b20 0x7fbc380c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc36d9d640 1 -- 172.21.15.83:0/1424707153 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc38152b20 msgr2=0x7fbc380c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc36d9d640 1 --2- 172.21.15.83:0/1424707153 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc38152b20 0x7fbc380c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc3659c640 1 --2- 172.21.15.83:0/1424707153 >> v2:172.21.15.184:3300/0 conn(0x7fbc38150290 0x7fbc380c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:17.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc36d9d640 1 -- 172.21.15.83:0/1424707153 >> v2:172.21.15.184:3300/0 conn(0x7fbc38150290 msgr2=0x7fbc380c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:17.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc36d9d640 1 --2- 172.21.15.83:0/1424707153 >> v2:172.21.15.184:3300/0 conn(0x7fbc38150290 0x7fbc380c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:17.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc36d9d640 1 -- 172.21.15.83:0/1424707153 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc38150670 con 0x7fbc380c45c0 2024-09-19T07:37:17.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.155+0000 7fbc36d9d640 1 --2- 172.21.15.83:0/1424707153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc380c45c0 0x7fbc38153eb0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fbc3806f590 tx=0x7fbc3001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb8966458544802 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:17.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc35d9b640 1 -- 172.21.15.83:0/1424707153 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc300a5840 con 0x7fbc380c45c0 2024-09-19T07:37:17.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc35d9b640 1 -- 172.21.15.83:0/1424707153 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc300a59e0 con 0x7fbc380c45c0 2024-09-19T07:37:17.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc35d9b640 1 -- 172.21.15.83:0/1424707153 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc300a5db0 con 0x7fbc380c45c0 2024-09-19T07:37:17.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1424707153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc380c45c0 msgr2=0x7fbc38153eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:17.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 --2- 172.21.15.83:0/1424707153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc380c45c0 0x7fbc38153eb0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fbc3806f590 tx=0x7fbc3001c9c0 comp rx=0 tx=0).stop 2024-09-19T07:37:17.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1424707153 shutdown_connections 2024-09-19T07:37:17.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 --2- 172.21.15.83:0/1424707153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc380c45c0 0x7fbc38153eb0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:17.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 --2- 172.21.15.83:0/1424707153 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc38152b20 0x7fbc380c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:17.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 --2- 172.21.15.83:0/1424707153 >> v2:172.21.15.184:3300/0 conn(0x7fbc38150290 0x7fbc380c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:17.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1424707153 >> 172.21.15.83:0/1424707153 conn(0x7fbc3806bc80 msgr2=0x7fbc3806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:17.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1424707153 shutdown_connections 2024-09-19T07:37:17.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1424707153 wait complete. 2024-09-19T07:37:17.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 Processor -- start 2024-09-19T07:37:17.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 -- start start 2024-09-19T07:37:17.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc380c45c0 0x7fbc38148880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:17.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc38150290 0x7fbc3814f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:17.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc38152b20 0x7fbc381508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:17.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbc380c2530 con 0x7fbc38150290 2024-09-19T07:37:17.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc380c3120 con 0x7fbc380c45c0 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbc38154800 con 0x7fbc38152b20 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc2dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc38150290 0x7fbc3814f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc2dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc38150290 0x7fbc3814f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33054/0 (socket says 172.21.15.83:33054) 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc2dd9b640 1 -- 172.21.15.83:0/1332574117 learned_addr learned my addr 172.21.15.83:0/1332574117 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc36d9d640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc38152b20 0x7fbc381508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3659c640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc380c45c0 0x7fbc38148880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc2dd9b640 1 -- 172.21.15.83:0/1332574117 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc38152b20 msgr2=0x7fbc381508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc2dd9b640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc38152b20 0x7fbc381508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc2dd9b640 1 -- 172.21.15.83:0/1332574117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc380c45c0 msgr2=0x7fbc38148880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc2dd9b640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc380c45c0 0x7fbc38148880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc2dd9b640 1 -- 172.21.15.83:0/1332574117 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc38151c00 con 0x7fbc38150290 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3659c640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc380c45c0 0x7fbc38148880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:37:17.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc2dd9b640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc38150290 0x7fbc3814f0b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000d5d0 tx=0x7fbc2000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:17.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc2f7fe640 1 -- 172.21.15.83:0/1332574117 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20004240 con 0x7fbc38150290 2024-09-19T07:37:17.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc2f7fe640 1 -- 172.21.15.83:0/1332574117 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc200043e0 con 0x7fbc38150290 2024-09-19T07:37:17.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1332574117 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc381ef110 con 0x7fbc38150290 2024-09-19T07:37:17.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc2f7fe640 1 -- 172.21.15.83:0/1332574117 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20003840 con 0x7fbc38150290 2024-09-19T07:37:17.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.159+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1332574117 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc3814a0b0 con 0x7fbc38150290 2024-09-19T07:37:17.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.163+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1332574117 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc380c1ff0 con 0x7fbc38150290 2024-09-19T07:37:17.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.163+0000 7fbc2f7fe640 1 -- 172.21.15.83:0/1332574117 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbc200039e0 con 0x7fbc38150290 2024-09-19T07:37:17.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.163+0000 7fbc2f7fe640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc180678c0 0x7fbc18069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:17.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.163+0000 7fbc3659c640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc180678c0 0x7fbc18069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:17.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.163+0000 7fbc2f7fe640 1 -- 172.21.15.83:0/1332574117 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8701+0+0 (secure 0 0 0) 0x7fbc20086b30 con 0x7fbc38150290 2024-09-19T07:37:17.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.163+0000 7fbc3659c640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc180678c0 0x7fbc18069d80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbc3814fb20 tx=0x7fbc28008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:17.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.167+0000 7fbc2f7fe640 1 -- 172.21.15.83:0/1332574117 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbc2001c360 con 0x7fbc38150290 2024-09-19T07:37:17.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.363+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1332574117 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc380c3ab0 con 0x7fbc38150290 2024-09-19T07:37:17.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc2f7fe640 1 -- 172.21.15.83:0/1332574117 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6542 (secure 0 0 0) 0x7fbc20056b00 con 0x7fbc38150290 2024-09-19T07:37:17.360 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:17.360 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":129,"btime":"2024-09-19T07:35:50:558585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119},{"gid":7113,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:35:46.388822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5612},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5612":{"gid":5612,"name":"b","rank":0,"incarnation":125,"state":"up:active","state_seq":5,"addr":"172.21.15.184:6835/1519298917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1519298917},{"type":"v1","addr":"172.21.15.184:6835","nonce":1519298917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5612,"qdb_cluster":[5612]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:35:31.315245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"c","rank":0,"incarnation":120,"state":"up:active","state_seq":20,"addr":"172.21.15.83:6837/3707393259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3707393259},{"type":"v1","addr":"172.21.15.83:6837","nonce":3707393259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6918,"qdb_cluster":[6918]},"id":2}]} 2024-09-19T07:37:17.361 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 129 2024-09-19T07:37:17.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1332574117 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc180678c0 msgr2=0x7fbc18069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:17.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc3ca80640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc180678c0 0x7fbc18069d80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbc3814fb20 tx=0x7fbc28008040 comp rx=0 tx=0).stop 2024-09-19T07:37:17.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1332574117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc38150290 msgr2=0x7fbc3814f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:17.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc3ca80640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc38150290 0x7fbc3814f0b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000d5d0 tx=0x7fbc2000da90 comp rx=0 tx=0).stop 2024-09-19T07:37:17.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1332574117 shutdown_connections 2024-09-19T07:37:17.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc3ca80640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc180678c0 0x7fbc18069d80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:17.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc3ca80640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc38152b20 0x7fbc381508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:17.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc3ca80640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc38150290 0x7fbc3814f0b0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:17.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc3ca80640 1 --2- 172.21.15.83:0/1332574117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc380c45c0 0x7fbc38148880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:17.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1332574117 >> 172.21.15.83:0/1332574117 conn(0x7fbc3806bc80 msgr2=0x7fbc38152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:17.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1332574117 shutdown_connections 2024-09-19T07:37:17.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:17.367+0000 7fbc3ca80640 1 -- 172.21.15.83:0/1332574117 wait complete. 2024-09-19T07:37:17.379 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-19T07:37:18.731 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:18.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.871+0000 7f7c29a87640 1 Processor -- start 2024-09-19T07:37:18.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.871+0000 7f7c29a87640 1 -- start start 2024-09-19T07:37:18.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c29a87640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c240c28e0 0x7f7c241545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:18.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c29a87640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7c2406dea0 0x7f7c2406e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:18.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c29a87640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c240a9e60 con 0x7f7c24071100 2024-09-19T07:37:18.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c29a87640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7c24071b50 con 0x7f7c2406dea0 2024-09-19T07:37:18.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c29a87640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7c240c1ff0 con 0x7f7c240c28e0 2024-09-19T07:37:18.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c23fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7c2406dea0 0x7f7c2406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:18.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c23fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7c2406dea0 0x7f7c2406e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33070/0 (socket says 172.21.15.83:33070) 2024-09-19T07:37:18.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c23fff640 1 -- 172.21.15.83:0/1870052086 learned_addr learned my addr 172.21.15.83:0/1870052086 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:18.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c22ffd640 1 --2- 172.21.15.83:0/1870052086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c240c28e0 0x7f7c241545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:18.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c23fff640 1 -- 172.21.15.83:0/1870052086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c240c28e0 msgr2=0x7f7c241545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:18.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c23fff640 1 --2- 172.21.15.83:0/1870052086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c240c28e0 0x7f7c241545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:18.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c23fff640 1 -- 172.21.15.83:0/1870052086 >> v1:172.21.15.184:6789/0 conn(0x7f7c24071100 legacy=0x7f7c24150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:18.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c23fff640 1 -- 172.21.15.83:0/1870052086 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c240aa140 con 0x7f7c2406dea0 2024-09-19T07:37:18.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c22ffd640 1 --2- 172.21.15.83:0/1870052086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c240c28e0 0x7f7c241545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:37:18.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.875+0000 7f7c23fff640 1 --2- 172.21.15.83:0/1870052086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c2406dea0 0x7f7c2406e350 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c009e50 tx=0x7f7c1c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=762c99f67baf6637 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:18.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c227fc640 1 -- 172.21.15.83:0/1870052086 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c0a55e0 con 0x7f7c2406dea0 2024-09-19T07:37:18.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c227fc640 1 -- 172.21.15.83:0/1870052086 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c1c0a5780 con 0x7f7c2406dea0 2024-09-19T07:37:18.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c227fc640 1 -- 172.21.15.83:0/1870052086 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c0a5b50 con 0x7f7c2406dea0 2024-09-19T07:37:18.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 -- 172.21.15.83:0/1870052086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c2406dea0 msgr2=0x7f7c2406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:18.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 --2- 172.21.15.83:0/1870052086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c2406dea0 0x7f7c2406e350 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c009e50 tx=0x7f7c1c01c960 comp rx=0 tx=0).stop 2024-09-19T07:37:18.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 -- 172.21.15.83:0/1870052086 shutdown_connections 2024-09-19T07:37:18.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 --2- 172.21.15.83:0/1870052086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c2406dea0 0x7f7c2406e350 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:18.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 --2- 172.21.15.83:0/1870052086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c240c28e0 0x7f7c241545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:18.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 -- 172.21.15.83:0/1870052086 >> 172.21.15.83:0/1870052086 conn(0x7f7c2406bc80 msgr2=0x7f7c2406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:18.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 -- 172.21.15.83:0/1870052086 shutdown_connections 2024-09-19T07:37:18.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 -- 172.21.15.83:0/1870052086 wait complete. 2024-09-19T07:37:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 Processor -- start 2024-09-19T07:37:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 -- start start 2024-09-19T07:37:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c2406dea0 0x7f7c24146b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c24071100 0x7f7c241502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c240c28e0 0x7f7c241eff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7c24071660 con 0x7f7c2406dea0 2024-09-19T07:37:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c240714e0 con 0x7f7c240c28e0 2024-09-19T07:37:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7c24160c60 con 0x7f7c24071100 2024-09-19T07:37:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c237fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c2406dea0 0x7f7c24146b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c237fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c2406dea0 0x7f7c24146b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33072/0 (socket says 172.21.15.83:33072) 2024-09-19T07:37:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c237fe640 1 -- 172.21.15.83:0/826887464 learned_addr learned my addr 172.21.15.83:0/826887464 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c23fff640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c240c28e0 0x7f7c241eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:18.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c22ffd640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c24071100 0x7f7c241502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:18.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c237fe640 1 -- 172.21.15.83:0/826887464 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c24071100 msgr2=0x7f7c241502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:18.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c237fe640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c24071100 0x7f7c241502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:18.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c237fe640 1 -- 172.21.15.83:0/826887464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c240c28e0 msgr2=0x7f7c241eff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:18.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c237fe640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c240c28e0 0x7f7c241eff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:18.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c237fe640 1 -- 172.21.15.83:0/826887464 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c241519d0 con 0x7f7c2406dea0 2024-09-19T07:37:18.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c23fff640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c240c28e0 0x7f7c241eff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:37:18.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c22ffd640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c24071100 0x7f7c241502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:37:18.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c237fe640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c2406dea0 0x7f7c24146b60 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400c480 tx=0x7f7c1400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:18.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c0bfff640 1 -- 172.21.15.83:0/826887464 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c14012830 con 0x7f7c2406dea0 2024-09-19T07:37:18.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c0bfff640 1 -- 172.21.15.83:0/826887464 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c14017070 con 0x7f7c2406dea0 2024-09-19T07:37:18.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 -- 172.21.15.83:0/826887464 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c240c6fd0 con 0x7f7c2406dea0 2024-09-19T07:37:18.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c0bfff640 1 -- 172.21.15.83:0/826887464 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c14012c30 con 0x7f7c2406dea0 2024-09-19T07:37:18.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.879+0000 7f7c29a87640 1 -- 172.21.15.83:0/826887464 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c241507e0 con 0x7f7c2406dea0 2024-09-19T07:37:18.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.883+0000 7f7c29a87640 1 -- 172.21.15.83:0/826887464 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c2406e350 con 0x7f7c2406dea0 2024-09-19T07:37:18.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.883+0000 7f7c0bfff640 1 -- 172.21.15.83:0/826887464 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7c1401c460 con 0x7f7c2406dea0 2024-09-19T07:37:18.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.883+0000 7f7c0bfff640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c100678c0 0x7f7c10069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:18.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.883+0000 7f7c0bfff640 1 -- 172.21.15.83:0/826887464 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7f7c14086760 con 0x7f7c2406dea0 2024-09-19T07:37:18.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.883+0000 7f7c22ffd640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c100678c0 0x7f7c10069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:18.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.883+0000 7f7c22ffd640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c100678c0 0x7f7c10069d80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7c240c6e00 tx=0x7f7c18008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:18.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:18.887+0000 7f7c0bfff640 1 -- 172.21.15.83:0/826887464 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7c14056680 con 0x7f7c2406dea0 2024-09-19T07:37:19.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.083+0000 7f7c29a87640 1 -- 172.21.15.83:0/826887464 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c240c4f30 con 0x7f7c2406dea0 2024-09-19T07:37:19.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.083+0000 7f7c0bfff640 1 -- 172.21.15.83:0/826887464 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) ==== 95+0+4873 (secure 0 0 0) 0x7f7c14055f80 con 0x7f7c2406dea0 2024-09-19T07:37:19.078 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:19.079 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":130,"btime":"2024-09-19T07:37:18:180208+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:18.180199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:18.180200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:replay","state_seq":1,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:37:19.079 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 130 2024-09-19T07:37:19.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.087+0000 7f7c29a87640 1 -- 172.21.15.83:0/826887464 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c100678c0 msgr2=0x7f7c10069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.087+0000 7f7c29a87640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c100678c0 0x7f7c10069d80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7c240c6e00 tx=0x7f7c18008040 comp rx=0 tx=0).stop 2024-09-19T07:37:19.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.087+0000 7f7c29a87640 1 -- 172.21.15.83:0/826887464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c2406dea0 msgr2=0x7f7c24146b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.087+0000 7f7c29a87640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c2406dea0 0x7f7c24146b60 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400c480 tx=0x7f7c1400c940 comp rx=0 tx=0).stop 2024-09-19T07:37:19.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.087+0000 7f7c29a87640 1 -- 172.21.15.83:0/826887464 shutdown_connections 2024-09-19T07:37:19.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.087+0000 7f7c29a87640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c100678c0 0x7f7c10069d80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.087+0000 7f7c29a87640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c240c28e0 0x7f7c241eff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.087+0000 7f7c29a87640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c24071100 0x7f7c241502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.087+0000 7f7c29a87640 1 --2- 172.21.15.83:0/826887464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c2406dea0 0x7f7c24146b60 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.087+0000 7f7c29a87640 1 -- 172.21.15.83:0/826887464 >> 172.21.15.83:0/826887464 conn(0x7f7c2406bc80 msgr2=0x7f7c24073030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:19.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.087+0000 7f7c29a87640 1 -- 172.21.15.83:0/826887464 shutdown_connections 2024-09-19T07:37:19.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.087+0000 7f7c29a87640 1 -- 172.21.15.83:0/826887464 wait complete. 2024-09-19T07:37:19.097 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-19T07:37:19.097 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:37:19.097 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:19.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f562688f640 1 Processor -- start 2024-09-19T07:37:19.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f562688f640 1 -- start start 2024-09-19T07:37:19.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f562688f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5620153570 0x7f56200c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f562688f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56200c3c20 con 0x7f56200c4130 2024-09-19T07:37:19.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f562688f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56200c89f0 con 0x7f56200c5110 2024-09-19T07:37:19.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f562688f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f56200c8cd0 con 0x7f5620153570 2024-09-19T07:37:19.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f5624e2c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f56200c4130 0x7f5620159980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37920/0 (socket says 172.21.15.83:37920) 2024-09-19T07:37:19.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f5624e2c640 1 -- 172.21.15.83:0/2623682362 learned_addr learned my addr 172.21.15.83:0/2623682362 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:37:19.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561ffff640 1 --2- 172.21.15.83:0/2623682362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5620153570 0x7f56200c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2634645414 0 0) 0x7f56200c3c20 con 0x7f56200c4130 2024-09-19T07:37:19.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5608003530 con 0x7f56200c4130 2024-09-19T07:37:19.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 449884220 0 0) 0x7f5608003530 con 0x7f56200c4130 2024-09-19T07:37:19.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5620153570 msgr2=0x7f56200c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:19.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 --2- 172.21.15.83:0/2623682362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5620153570 0x7f56200c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 >> v1:172.21.15.184:6789/0 conn(0x7f56200c5110 legacy=0x7f5620159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56200c8fb0 con 0x7f56200c4130 2024-09-19T07:37:19.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f5618001a00 con 0x7f56200c4130 2024-09-19T07:37:19.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 >> v1:172.21.15.83:6789/0 conn(0x7f56200c4130 legacy=0x7f5620159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 --2- 172.21.15.83:0/2623682362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5620153570 0x7f56080039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 --2- 172.21.15.83:0/2623682362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5608003f70 0x7f5608004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 --2- 172.21.15.83:0/2623682362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5608004d50 0x7f56080051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f56200c8cd0 con 0x7f5608003f70 2024-09-19T07:37:19.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f560800bc10 con 0x7f5608004d50 2024-09-19T07:37:19.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f560800be10 con 0x7f5620153570 2024-09-19T07:37:19.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561f7fe640 1 --2- 172.21.15.83:0/2623682362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5608003f70 0x7f5608004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f5624e2c640 1 --2- 172.21.15.83:0/2623682362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5608004d50 0x7f56080051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561f7fe640 1 -- 172.21.15.83:0/2623682362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5620153570 msgr2=0x7f56080039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561f7fe640 1 --2- 172.21.15.83:0/2623682362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5620153570 0x7f56080039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561f7fe640 1 -- 172.21.15.83:0/2623682362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5608004d50 msgr2=0x7f56080051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561f7fe640 1 --2- 172.21.15.83:0/2623682362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5608004d50 0x7f56080051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.223+0000 7f561f7fe640 1 -- 172.21.15.83:0/2623682362 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5608003530 con 0x7f5608003f70 2024-09-19T07:37:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f561f7fe640 1 --2- 172.21.15.83:0/2623682362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5608003f70 0x7f5608004810 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f5614001750 tx=0x7f561401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83f38c97c59471c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:19.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56140013a0 con 0x7f5608003f70 2024-09-19T07:37:19.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56140052f0 con 0x7f5608003f70 2024-09-19T07:37:19.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f561effd640 1 -- 172.21.15.83:0/2623682362 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614003840 con 0x7f5608003f70 2024-09-19T07:37:19.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 -- 172.21.15.83:0/2623682362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5608003f70 msgr2=0x7f5608004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 --2- 172.21.15.83:0/2623682362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5608003f70 0x7f5608004810 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f5614001750 tx=0x7f561401c3e0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f5624e2c640 1 -- 172.21.15.83:0/2623682362 reap_dead start 2024-09-19T07:37:19.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 -- 172.21.15.83:0/2623682362 shutdown_connections 2024-09-19T07:37:19.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 -- 172.21.15.83:0/2623682362 >> 172.21.15.83:0/2623682362 conn(0x7f56200c6110 msgr2=0x7f56200c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:19.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 -- 172.21.15.83:0/2623682362 shutdown_connections 2024-09-19T07:37:19.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 -- 172.21.15.83:0/2623682362 wait complete. 2024-09-19T07:37:19.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 Processor -- start 2024-09-19T07:37:19.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 -- start start 2024-09-19T07:37:19.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5608003f70 0x7f5620156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5620146c80 0x7f56201459f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56200ad320 0x7f5620145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5620162070 con 0x7f5608003f70 2024-09-19T07:37:19.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5620161ef0 con 0x7f56200ad320 2024-09-19T07:37:19.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f562688f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f56201621f0 con 0x7f5620146c80 2024-09-19T07:37:19.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f561ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5608003f70 0x7f5620156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f561ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5608003f70 0x7f5620156f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33102/0 (socket says 172.21.15.83:33102) 2024-09-19T07:37:19.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f561ffff640 1 -- 172.21.15.83:0/402005823 learned_addr learned my addr 172.21.15.83:0/402005823 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:19.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f5624e2c640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56200ad320 0x7f5620145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.227+0000 7f561f7fe640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5620146c80 0x7f56201459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561ffff640 1 -- 172.21.15.83:0/402005823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5620146c80 msgr2=0x7f56201459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561ffff640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5620146c80 0x7f56201459f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561ffff640 1 -- 172.21.15.83:0/402005823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56200ad320 msgr2=0x7f5620145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561ffff640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56200ad320 0x7f5620145f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561ffff640 1 -- 172.21.15.83:0/402005823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56200aaa80 con 0x7f5608003f70 2024-09-19T07:37:19.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f5624e2c640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56200ad320 0x7f5620145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:37:19.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561ffff640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5608003f70 0x7f5620156f30 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f561000e820 tx=0x7f561000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:19.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561cff9640 1 -- 172.21.15.83:0/402005823 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5610017070 con 0x7f5608003f70 2024-09-19T07:37:19.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561cff9640 1 -- 172.21.15.83:0/402005823 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f561000c940 con 0x7f5608003f70 2024-09-19T07:37:19.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f562688f640 1 -- 172.21.15.83:0/402005823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56201517b0 con 0x7f5608003f70 2024-09-19T07:37:19.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561cff9640 1 -- 172.21.15.83:0/402005823 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561000cd30 con 0x7f5608003f70 2024-09-19T07:37:19.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f562688f640 1 -- 172.21.15.83:0/402005823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56201f2160 con 0x7f5608003f70 2024-09-19T07:37:19.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f562688f640 1 -- 172.21.15.83:0/402005823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56200c3c20 con 0x7f5608003f70 2024-09-19T07:37:19.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561cff9640 1 -- 172.21.15.83:0/402005823 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f561001a070 con 0x7f5608003f70 2024-09-19T07:37:19.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561cff9640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f560c0678c0 0x7f560c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561cff9640 1 -- 172.21.15.83:0/402005823 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7f56100867c0 con 0x7f5608003f70 2024-09-19T07:37:19.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561f7fe640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f560c0678c0 0x7f560c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.231+0000 7f561f7fe640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f560c0678c0 0x7f560c069d80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5620072bc0 tx=0x7f56140011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:19.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.235+0000 7f561cff9640 1 -- 172.21.15.83:0/402005823 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f56100566e0 con 0x7f5608003f70 2024-09-19T07:37:19.380 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:19.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.427+0000 7f562688f640 1 -- 172.21.15.83:0/402005823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5620147f80 con 0x7f5608003f70 2024-09-19T07:37:19.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.427+0000 7f561cff9640 1 -- 172.21.15.83:0/402005823 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+4881 (secure 0 0 0) 0x7f5610055fe0 con 0x7f5608003f70 2024-09-19T07:37:19.423 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:19.423 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":131,"btime":"2024-09-19T07:37:19:182840+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:18.553369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:18.640562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:reconnect","state_seq":44,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:37:19.423 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 131 2024-09-19T07:37:19.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.431+0000 7f562688f640 1 -- 172.21.15.83:0/402005823 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f560c0678c0 msgr2=0x7f560c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.431+0000 7f562688f640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f560c0678c0 0x7f560c069d80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5620072bc0 tx=0x7f56140011a0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.431+0000 7f562688f640 1 -- 172.21.15.83:0/402005823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5608003f70 msgr2=0x7f5620156f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.431+0000 7f562688f640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5608003f70 0x7f5620156f30 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f561000e820 tx=0x7f561000ece0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.431+0000 7f562688f640 1 -- 172.21.15.83:0/402005823 shutdown_connections 2024-09-19T07:37:19.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.431+0000 7f562688f640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f560c0678c0 0x7f560c069d80 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.431+0000 7f562688f640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56200ad320 0x7f5620145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.431+0000 7f562688f640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5620146c80 0x7f56201459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.431+0000 7f562688f640 1 --2- 172.21.15.83:0/402005823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5608003f70 0x7f5620156f30 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.431+0000 7f562688f640 1 -- 172.21.15.83:0/402005823 >> 172.21.15.83:0/402005823 conn(0x7f56200c6110 msgr2=0x7f56201470b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:19.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.431+0000 7f562688f640 1 -- 172.21.15.83:0/402005823 shutdown_connections 2024-09-19T07:37:19.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.431+0000 7f562688f640 1 -- 172.21.15.83:0/402005823 wait complete. 2024-09-19T07:37:19.442 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:37:19:182840+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 131, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 131, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7113": { "addr": "172.21.15.83:6835/2168217861", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 2168217861, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 2168217861, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7113, "incarnation": 130, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:37:18.553369+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7113 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 131, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5555": { "addr": "172.21.15.184:6833/381468017", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 381468017, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 381468017, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5555, "incarnation": 130, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 44 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:37:18.640562+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5555 } } } ], "standbys": [] } 2024-09-19T07:37:19.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.499+0000 7eff4bf2e640 1 Processor -- start 2024-09-19T07:37:19.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.499+0000 7eff4bf2e640 1 -- start start 2024-09-19T07:37:19.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.499+0000 7eff4bf2e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7eff440c6120 0x7eff440c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.499+0000 7eff4bf2e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff441592f0 0x7eff44157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.499+0000 7eff4bf2e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff44156200 con 0x7eff44155d70 2024-09-19T07:37:19.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.499+0000 7eff4bf2e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7eff440c9930 con 0x7eff440c6120 2024-09-19T07:37:19.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.499+0000 7eff4bf2e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7eff441564e0 con 0x7eff441592f0 2024-09-19T07:37:19.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.499+0000 7eff49cca640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7eff440c6120 0x7eff440c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.499+0000 7eff49cca640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7eff440c6120 0x7eff440c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33108/0 (socket says 172.21.15.83:33108) 2024-09-19T07:37:19.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.499+0000 7eff49cca640 1 -- 172.21.15.83:0/3798712940 learned_addr learned my addr 172.21.15.83:0/3798712940 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:19.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff49cca640 1 -- 172.21.15.83:0/3798712940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff441592f0 msgr2=0x7eff44157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:19.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff4a4cb640 1 --2- 172.21.15.83:0/3798712940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff441592f0 0x7eff44157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff49cca640 1 --2- 172.21.15.83:0/3798712940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff441592f0 0x7eff44157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff49cca640 1 -- 172.21.15.83:0/3798712940 >> v1:172.21.15.184:6789/0 conn(0x7eff44155d70 legacy=0x7eff441575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:19.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff49cca640 1 -- 172.21.15.83:0/3798712940 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff441567c0 con 0x7eff440c6120 2024-09-19T07:37:19.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff4a4cb640 1 --2- 172.21.15.83:0/3798712940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff441592f0 0x7eff44157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:37:19.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff49cca640 1 --2- 172.21.15.83:0/3798712940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff440c6120 0x7eff440c6500 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7eff3c009870 tx=0x7eff3c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b3c0cceba0f7f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:19.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff48cc8640 1 -- 172.21.15.83:0/3798712940 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3c0b0070 con 0x7eff440c6120 2024-09-19T07:37:19.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff48cc8640 1 -- 172.21.15.83:0/3798712940 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff3c0ab600 con 0x7eff440c6120 2024-09-19T07:37:19.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff48cc8640 1 -- 172.21.15.83:0/3798712940 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3c0ab9d0 con 0x7eff440c6120 2024-09-19T07:37:19.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff4bf2e640 1 -- 172.21.15.83:0/3798712940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff440c6120 msgr2=0x7eff440c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff4bf2e640 1 --2- 172.21.15.83:0/3798712940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff440c6120 0x7eff440c6500 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7eff3c009870 tx=0x7eff3c01c520 comp rx=0 tx=0).stop 2024-09-19T07:37:19.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff4bf2e640 1 -- 172.21.15.83:0/3798712940 shutdown_connections 2024-09-19T07:37:19.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff4bf2e640 1 --2- 172.21.15.83:0/3798712940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff441592f0 0x7eff44157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff4bf2e640 1 --2- 172.21.15.83:0/3798712940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff440c6120 0x7eff440c6500 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff4bf2e640 1 -- 172.21.15.83:0/3798712940 >> 172.21.15.83:0/3798712940 conn(0x7eff4406bc80 msgr2=0x7eff4406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:19.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff4bf2e640 1 -- 172.21.15.83:0/3798712940 shutdown_connections 2024-09-19T07:37:19.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.503+0000 7eff4bf2e640 1 -- 172.21.15.83:0/3798712940 wait complete. 2024-09-19T07:37:19.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff4bf2e640 1 Processor -- start 2024-09-19T07:37:19.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff4bf2e640 1 -- start start 2024-09-19T07:37:19.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff4bf2e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff440c6120 0x7eff4414ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff4bf2e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff441592f0 0x7eff4414b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff4bf2e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff440adff0 0x7eff44149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff4bf2e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7eff44153860 con 0x7eff441592f0 2024-09-19T07:37:19.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff4bf2e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7eff44162ea0 con 0x7eff440adff0 2024-09-19T07:37:19.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff4bf2e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7eff440ab850 con 0x7eff440c6120 2024-09-19T07:37:19.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff494c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff441592f0 0x7eff4414b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff494c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff441592f0 0x7eff4414b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33116/0 (socket says 172.21.15.83:33116) 2024-09-19T07:37:19.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff494c9640 1 -- 172.21.15.83:0/4102336614 learned_addr learned my addr 172.21.15.83:0/4102336614 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:19.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff49cca640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff440c6120 0x7eff4414ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff4a4cb640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff440adff0 0x7eff44149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff494c9640 1 -- 172.21.15.83:0/4102336614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff440c6120 msgr2=0x7eff4414ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff494c9640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff440c6120 0x7eff4414ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff494c9640 1 -- 172.21.15.83:0/4102336614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff440adff0 msgr2=0x7eff44149f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff494c9640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff440adff0 0x7eff44149f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff494c9640 1 -- 172.21.15.83:0/4102336614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff440ad8f0 con 0x7eff441592f0 2024-09-19T07:37:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff494c9640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff441592f0 0x7eff4414b360 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7eff3800c480 tx=0x7eff3800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff427fc640 1 -- 172.21.15.83:0/4102336614 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff38012830 con 0x7eff441592f0 2024-09-19T07:37:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff427fc640 1 -- 172.21.15.83:0/4102336614 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff38017070 con 0x7eff441592f0 2024-09-19T07:37:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff427fc640 1 -- 172.21.15.83:0/4102336614 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff38012c30 con 0x7eff441592f0 2024-09-19T07:37:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff4bf2e640 1 -- 172.21.15.83:0/4102336614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff440adb90 con 0x7eff441592f0 2024-09-19T07:37:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff4bf2e640 1 -- 172.21.15.83:0/4102336614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff440ab4e0 con 0x7eff441592f0 2024-09-19T07:37:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.507+0000 7eff4bf2e640 1 -- 172.21.15.83:0/4102336614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff440c2400 con 0x7eff441592f0 2024-09-19T07:37:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.511+0000 7eff427fc640 1 -- 172.21.15.83:0/4102336614 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7eff3801c460 con 0x7eff441592f0 2024-09-19T07:37:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.511+0000 7eff427fc640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff300678c0 0x7eff30069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.511+0000 7eff427fc640 1 -- 172.21.15.83:0/4102336614 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7eff38086710 con 0x7eff441592f0 2024-09-19T07:37:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.511+0000 7eff49cca640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff300678c0 0x7eff30069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.511+0000 7eff49cca640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff300678c0 0x7eff30069d80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0ad000 tx=0x7eff3c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:19.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.515+0000 7eff427fc640 1 -- 172.21.15.83:0/4102336614 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7eff440c2400 con 0x7eff441592f0 2024-09-19T07:37:19.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.703+0000 7eff4bf2e640 1 -- 172.21.15.83:0/4102336614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff440c6500 con 0x7eff441592f0 2024-09-19T07:37:19.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.703+0000 7eff427fc640 1 -- 172.21.15.83:0/4102336614 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+4881 (secure 0 0 0) 0x7eff38056630 con 0x7eff441592f0 2024-09-19T07:37:19.699 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:19.699 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":131,"btime":"2024-09-19T07:37:19:182840+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:18.553369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:18.640562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:reconnect","state_seq":44,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:37:19.699 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 131 2024-09-19T07:37:19.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.707+0000 7eff4bf2e640 1 -- 172.21.15.83:0/4102336614 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff300678c0 msgr2=0x7eff30069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.707+0000 7eff4bf2e640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff300678c0 0x7eff30069d80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0ad000 tx=0x7eff3c008040 comp rx=0 tx=0).stop 2024-09-19T07:37:19.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.707+0000 7eff4bf2e640 1 -- 172.21.15.83:0/4102336614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff441592f0 msgr2=0x7eff4414b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.707+0000 7eff4bf2e640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff441592f0 0x7eff4414b360 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7eff3800c480 tx=0x7eff3800c940 comp rx=0 tx=0).stop 2024-09-19T07:37:19.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.707+0000 7eff4bf2e640 1 -- 172.21.15.83:0/4102336614 shutdown_connections 2024-09-19T07:37:19.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.707+0000 7eff4bf2e640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff300678c0 0x7eff30069d80 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.707+0000 7eff4bf2e640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff440adff0 0x7eff44149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.707+0000 7eff4bf2e640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff441592f0 0x7eff4414b360 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.707+0000 7eff4bf2e640 1 --2- 172.21.15.83:0/4102336614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff440c6120 0x7eff4414ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.707+0000 7eff4bf2e640 1 -- 172.21.15.83:0/4102336614 >> 172.21.15.83:0/4102336614 conn(0x7eff4406bc80 msgr2=0x7eff441596d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:19.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.707+0000 7eff4bf2e640 1 -- 172.21.15.83:0/4102336614 shutdown_connections 2024-09-19T07:37:19.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.707+0000 7eff4bf2e640 1 -- 172.21.15.83:0/4102336614 wait complete. 2024-09-19T07:37:19.716 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-09-19T07:37:19.716 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:37:19.716 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:19.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30f108c640 1 Processor -- start 2024-09-19T07:37:19.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30f108c640 1 -- start start 2024-09-19T07:37:19.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30f108c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f30ec071100 0x7f30ec1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30f108c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30ec0c28e0 0x7f30ec150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30f108c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f30ec151480 0x7f30ec06dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30f108c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f30ec071b50 con 0x7f30ec151480 2024-09-19T07:37:19.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30f108c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f30ec0c1ff0 con 0x7f30ec071100 2024-09-19T07:37:19.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30f108c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f30ec0a9e60 con 0x7f30ec0c28e0 2024-09-19T07:37:19.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30eb59e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f30ec151480 0x7f30ec06dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30eb59e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f30ec151480 0x7f30ec06dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33128/0 (socket says 172.21.15.83:33128) 2024-09-19T07:37:19.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30eb59e640 1 -- 172.21.15.83:0/1394991176 learned_addr learned my addr 172.21.15.83:0/1394991176 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:19.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30ea59c640 1 --2- 172.21.15.83:0/1394991176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30ec0c28e0 0x7f30ec150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30ead9d640 1 --2- 172.21.15.83:0/1394991176 >> v2:172.21.15.184:3300/0 conn(0x7f30ec071100 0x7f30ec1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30eb59e640 1 -- 172.21.15.83:0/1394991176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30ec0c28e0 msgr2=0x7f30ec150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:19.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30eb59e640 1 --2- 172.21.15.83:0/1394991176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30ec0c28e0 0x7f30ec150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30eb59e640 1 -- 172.21.15.83:0/1394991176 >> v2:172.21.15.184:3300/0 conn(0x7f30ec071100 msgr2=0x7f30ec1545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:19.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30eb59e640 1 --2- 172.21.15.83:0/1394991176 >> v2:172.21.15.184:3300/0 conn(0x7f30ec071100 0x7f30ec1545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30eb59e640 1 -- 172.21.15.83:0/1394991176 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ec0aa140 con 0x7f30ec151480 2024-09-19T07:37:19.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.843+0000 7f30eb59e640 1 --2- 172.21.15.83:0/1394991176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ec151480 0x7f30ec06dea0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f30ec06f620 tx=0x7f30e401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ddbaab069d8fefb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:19.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30e9d9b640 1 -- 172.21.15.83:0/1394991176 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e40a55e0 con 0x7f30ec151480 2024-09-19T07:37:19.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30e9d9b640 1 -- 172.21.15.83:0/1394991176 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30e40a5780 con 0x7f30ec151480 2024-09-19T07:37:19.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30e9d9b640 1 -- 172.21.15.83:0/1394991176 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e40a5b50 con 0x7f30ec151480 2024-09-19T07:37:19.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 -- 172.21.15.83:0/1394991176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ec151480 msgr2=0x7f30ec06dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 --2- 172.21.15.83:0/1394991176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ec151480 0x7f30ec06dea0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f30ec06f620 tx=0x7f30e401c960 comp rx=0 tx=0).stop 2024-09-19T07:37:19.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 -- 172.21.15.83:0/1394991176 shutdown_connections 2024-09-19T07:37:19.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 --2- 172.21.15.83:0/1394991176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ec151480 0x7f30ec06dea0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 --2- 172.21.15.83:0/1394991176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30ec0c28e0 0x7f30ec150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 --2- 172.21.15.83:0/1394991176 >> v2:172.21.15.184:3300/0 conn(0x7f30ec071100 0x7f30ec1545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 -- 172.21.15.83:0/1394991176 >> 172.21.15.83:0/1394991176 conn(0x7f30ec06bc80 msgr2=0x7f30ec06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:19.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 -- 172.21.15.83:0/1394991176 shutdown_connections 2024-09-19T07:37:19.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 -- 172.21.15.83:0/1394991176 wait complete. 2024-09-19T07:37:19.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 Processor -- start 2024-09-19T07:37:19.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 -- start start 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ec071100 0x7f30ec0c1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30ec0c28e0 0x7f30ec1ec1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30ec151480 0x7f30ec0bd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1570d0 con 0x7f30ec071100 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec071730 con 0x7f30ec0c28e0 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f30ec157250 con 0x7f30ec151480 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30ead9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ec071100 0x7f30ec0c1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30ead9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ec071100 0x7f30ec0c1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33132/0 (socket says 172.21.15.83:33132) 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30ead9d640 1 -- 172.21.15.83:0/1700475106 learned_addr learned my addr 172.21.15.83:0/1700475106 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30eb59e640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30ec151480 0x7f30ec0bd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30ea59c640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30ec0c28e0 0x7f30ec1ec1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30ead9d640 1 -- 172.21.15.83:0/1700475106 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30ec151480 msgr2=0x7f30ec0bd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30ead9d640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30ec151480 0x7f30ec0bd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30ead9d640 1 -- 172.21.15.83:0/1700475106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30ec0c28e0 msgr2=0x7f30ec1ec1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:19.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30ead9d640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30ec0c28e0 0x7f30ec1ec1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:19.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30ead9d640 1 -- 172.21.15.83:0/1700475106 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ec1ef870 con 0x7f30ec071100 2024-09-19T07:37:19.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30ead9d640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ec071100 0x7f30ec0c1660 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00d800 tx=0x7f30dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:19.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30d37fe640 1 -- 172.21.15.83:0/1700475106 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc00b6c0 con 0x7f30ec071100 2024-09-19T07:37:19.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30d37fe640 1 -- 172.21.15.83:0/1700475106 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30dc019070 con 0x7f30ec071100 2024-09-19T07:37:19.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 -- 172.21.15.83:0/1700475106 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ec0bc610 con 0x7f30ec071100 2024-09-19T07:37:19.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30d37fe640 1 -- 172.21.15.83:0/1700475106 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc00ba60 con 0x7f30ec071100 2024-09-19T07:37:19.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.847+0000 7f30f108c640 1 -- 172.21.15.83:0/1700475106 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ec1efb10 con 0x7f30ec071100 2024-09-19T07:37:19.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.851+0000 7f30f108c640 1 -- 172.21.15.83:0/1700475106 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30ec150f40 con 0x7f30ec071100 2024-09-19T07:37:19.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.851+0000 7f30d37fe640 1 -- 172.21.15.83:0/1700475106 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f30dc00bc00 con 0x7f30ec071100 2024-09-19T07:37:19.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.851+0000 7f30d37fe640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f30d80678f0 0x7f30d8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:19.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.851+0000 7f30ea59c640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f30d80678f0 0x7f30d8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:19.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.851+0000 7f30d37fe640 1 -- 172.21.15.83:0/1700475106 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7f30dc086d80 con 0x7f30ec071100 2024-09-19T07:37:19.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.851+0000 7f30ea59c640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f30d80678f0 0x7f30d8069db0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f30ec1544f0 tx=0x7f30e0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:19.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:19.855+0000 7f30d37fe640 1 -- 172.21.15.83:0/1700475106 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f30dc056c40 con 0x7f30ec071100 2024-09-19T07:37:20.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.055+0000 7f30f108c640 1 -- 172.21.15.83:0/1700475106 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30ec0c4f30 con 0x7f30ec071100 2024-09-19T07:37:20.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.055+0000 7f30d37fe640 1 -- 172.21.15.83:0/1700475106 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+4881 (secure 0 0 0) 0x7f30dc056540 con 0x7f30ec071100 2024-09-19T07:37:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:20.049 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":131,"btime":"2024-09-19T07:37:19:182840+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:18.553369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:18.640562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:reconnect","state_seq":44,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:37:20.050 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 131 2024-09-19T07:37:20.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.055+0000 7f30f108c640 1 -- 172.21.15.83:0/1700475106 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f30d80678f0 msgr2=0x7f30d8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:20.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.055+0000 7f30f108c640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f30d80678f0 0x7f30d8069db0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f30ec1544f0 tx=0x7f30e0008040 comp rx=0 tx=0).stop 2024-09-19T07:37:20.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.055+0000 7f30f108c640 1 -- 172.21.15.83:0/1700475106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ec071100 msgr2=0x7f30ec0c1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:20.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.055+0000 7f30f108c640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ec071100 0x7f30ec0c1660 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00d800 tx=0x7f30dc00dcc0 comp rx=0 tx=0).stop 2024-09-19T07:37:20.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.055+0000 7f30f108c640 1 -- 172.21.15.83:0/1700475106 shutdown_connections 2024-09-19T07:37:20.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.055+0000 7f30f108c640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f30d80678f0 0x7f30d8069db0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:20.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.055+0000 7f30f108c640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30ec151480 0x7f30ec0bd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:20.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.055+0000 7f30f108c640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30ec0c28e0 0x7f30ec1ec1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:20.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.055+0000 7f30f108c640 1 --2- 172.21.15.83:0/1700475106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ec071100 0x7f30ec0c1660 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:20.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.055+0000 7f30f108c640 1 -- 172.21.15.83:0/1700475106 >> 172.21.15.83:0/1700475106 conn(0x7f30ec06bc80 msgr2=0x7f30ec14a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:20.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.059+0000 7f30f108c640 1 -- 172.21.15.83:0/1700475106 shutdown_connections 2024-09-19T07:37:20.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:20.059+0000 7f30f108c640 1 -- 172.21.15.83:0/1700475106 wait complete. 2024-09-19T07:37:20.070 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:37:19:182840+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 131, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 131, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7113": { "addr": "172.21.15.83:6835/2168217861", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 2168217861, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 2168217861, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7113, "incarnation": 130, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:37:18.553369+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7113 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 131, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5555": { "addr": "172.21.15.184:6833/381468017", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 381468017, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 381468017, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5555, "incarnation": 130, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 44 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:37:18.640562+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5555 } } } ], "standbys": [] } 2024-09-19T07:37:21.393 INFO:tasks.ceph.mds.d.smithi184.stderr:2024-09-19T07:37:21.398+0000 7fbda63b8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:37:21.443 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:21.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.607+0000 7f5732f18640 1 Processor -- start 2024-09-19T07:37:21.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.607+0000 7f5732f18640 1 -- start start 2024-09-19T07:37:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.607+0000 7f5732f18640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f572c0aa4a0 0x7f572c157720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.607+0000 7f5732f18640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f572c073d90 con 0x7f572c0c3ab0 2024-09-19T07:37:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.607+0000 7f5732f18640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f572c153b30 con 0x7f572c14dbe0 2024-09-19T07:37:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.607+0000 7f5732f18640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f572c154700 con 0x7f572c0aa4a0 2024-09-19T07:37:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.607+0000 7f57314b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f572c0aa4a0 0x7f572c157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f57314b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f572c0aa4a0 0x7f572c157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60274/0 (socket says 172.21.15.83:60274) 2024-09-19T07:37:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f57314b5640 1 -- 172.21.15.83:0/3341454891 learned_addr learned my addr 172.21.15.83:0/3341454891 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f57314b5640 1 -- 172.21.15.83:0/3341454891 >> v1:172.21.15.184:6789/0 conn(0x7f572c14dbe0 legacy=0x7f572c0c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:21.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f57314b5640 1 -- 172.21.15.83:0/3341454891 >> v1:172.21.15.83:6789/0 conn(0x7f572c0c3ab0 legacy=0x7f572c157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:21.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f57314b5640 1 -- 172.21.15.83:0/3341454891 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c153dd0 con 0x7f572c0aa4a0 2024-09-19T07:37:21.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f57314b5640 1 --2- 172.21.15.83:0/3341454891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f572c0aa4a0 0x7f572c157720 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f572c154880 tx=0x7f572401c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675d87dbbd904a38 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:21.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f572b7fe640 1 -- 172.21.15.83:0/3341454891 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57240a5750 con 0x7f572c0aa4a0 2024-09-19T07:37:21.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f572b7fe640 1 -- 172.21.15.83:0/3341454891 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57240a58f0 con 0x7f572c0aa4a0 2024-09-19T07:37:21.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f572b7fe640 1 -- 172.21.15.83:0/3341454891 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57240a5cc0 con 0x7f572c0aa4a0 2024-09-19T07:37:21.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f5732f18640 1 -- 172.21.15.83:0/3341454891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f572c0aa4a0 msgr2=0x7f572c157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:21.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f5732f18640 1 --2- 172.21.15.83:0/3341454891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f572c0aa4a0 0x7f572c157720 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f572c154880 tx=0x7f572401c980 comp rx=0 tx=0).stop 2024-09-19T07:37:21.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f5732f18640 1 -- 172.21.15.83:0/3341454891 shutdown_connections 2024-09-19T07:37:21.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f5732f18640 1 --2- 172.21.15.83:0/3341454891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f572c0aa4a0 0x7f572c157720 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:21.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f5732f18640 1 -- 172.21.15.83:0/3341454891 >> 172.21.15.83:0/3341454891 conn(0x7f572c0c95a0 msgr2=0x7f572c0c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:21.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f5732f18640 1 -- 172.21.15.83:0/3341454891 shutdown_connections 2024-09-19T07:37:21.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.611+0000 7f5732f18640 1 -- 172.21.15.83:0/3341454891 wait complete. 2024-09-19T07:37:21.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5732f18640 1 Processor -- start 2024-09-19T07:37:21.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5732f18640 1 -- start start 2024-09-19T07:37:21.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5732f18640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f572c0aa4a0 0x7f572c0af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5732f18640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f572c14dbe0 0x7f572c1efdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5732f18640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f572c144060 0x7f572c1444f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5732f18640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f572c0c46b0 con 0x7f572c0aa4a0 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5732f18640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f572c0741a0 con 0x7f572c144060 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5732f18640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f572c0c4830 con 0x7f572c14dbe0 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5730cb4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f572c0aa4a0 0x7f572c0af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5730cb4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f572c0aa4a0 0x7f572c0af1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60284/0 (socket says 172.21.15.83:60284) 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5730cb4640 1 -- 172.21.15.83:0/918310809 learned_addr learned my addr 172.21.15.83:0/918310809 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f57314b5640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f572c144060 0x7f572c1444f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f572bfff640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f572c14dbe0 0x7f572c1efdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5730cb4640 1 -- 172.21.15.83:0/918310809 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f572c14dbe0 msgr2=0x7f572c1efdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5730cb4640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f572c14dbe0 0x7f572c1efdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:21.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5730cb4640 1 -- 172.21.15.83:0/918310809 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f572c144060 msgr2=0x7f572c1444f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:21.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5730cb4640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f572c144060 0x7f572c1444f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:21.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5730cb4640 1 -- 172.21.15.83:0/918310809 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c14ab90 con 0x7f572c0aa4a0 2024-09-19T07:37:21.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5730cb4640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f572c0aa4a0 0x7f572c0af1c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f572000c370 tx=0x7f572000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:21.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f57297fa640 1 -- 172.21.15.83:0/918310809 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57200052d0 con 0x7f572c0aa4a0 2024-09-19T07:37:21.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f57297fa640 1 -- 172.21.15.83:0/918310809 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5720005470 con 0x7f572c0aa4a0 2024-09-19T07:37:21.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5732f18640 1 -- 172.21.15.83:0/918310809 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f572c15ed20 con 0x7f572c0aa4a0 2024-09-19T07:37:21.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f57297fa640 1 -- 172.21.15.83:0/918310809 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720007260 con 0x7f572c0aa4a0 2024-09-19T07:37:21.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5732f18640 1 -- 172.21.15.83:0/918310809 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f572c1f0600 con 0x7f572c0aa4a0 2024-09-19T07:37:21.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.615+0000 7f5732f18640 1 -- 172.21.15.83:0/918310809 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f572c152a40 con 0x7f572c0aa4a0 2024-09-19T07:37:21.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.619+0000 7f57297fa640 1 -- 172.21.15.83:0/918310809 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5720007400 con 0x7f572c0aa4a0 2024-09-19T07:37:21.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.619+0000 7f57297fa640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f57100678c0 0x7f5710069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:21.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.619+0000 7f57297fa640 1 -- 172.21.15.83:0/918310809 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7f5720085c80 con 0x7f572c0aa4a0 2024-09-19T07:37:21.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.619+0000 7f572bfff640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f57100678c0 0x7f5710069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:21.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.619+0000 7f572bfff640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f57100678c0 0x7f5710069d80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f572c14d900 tx=0x7f5718009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:21.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.623+0000 7f57297fa640 1 -- 172.21.15.83:0/918310809 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5720055550 con 0x7f572c0aa4a0 2024-09-19T07:37:21.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.891+0000 7f5732f18640 1 -- 172.21.15.83:0/918310809 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f572c0c9b20 con 0x7f572c0aa4a0 2024-09-19T07:37:21.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.891+0000 7f57297fa640 1 -- 172.21.15.83:0/918310809 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) ==== 95+0+4889 (secure 0 0 0) 0x7f5720059850 con 0x7f572c0aa4a0 2024-09-19T07:37:21.887 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:21.888 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":133,"btime":"2024-09-19T07:37:21:194300+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:37:21.888 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 133 2024-09-19T07:37:21.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.895+0000 7f5732f18640 1 -- 172.21.15.83:0/918310809 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f57100678c0 msgr2=0x7f5710069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.895+0000 7f5732f18640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f57100678c0 0x7f5710069d80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f572c14d900 tx=0x7f5718009040 comp rx=0 tx=0).stop 2024-09-19T07:37:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.895+0000 7f5732f18640 1 -- 172.21.15.83:0/918310809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f572c0aa4a0 msgr2=0x7f572c0af1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.895+0000 7f5732f18640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f572c0aa4a0 0x7f572c0af1c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f572000c370 tx=0x7f572000c830 comp rx=0 tx=0).stop 2024-09-19T07:37:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.895+0000 7f5732f18640 1 -- 172.21.15.83:0/918310809 shutdown_connections 2024-09-19T07:37:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.895+0000 7f5732f18640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f57100678c0 0x7f5710069d80 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.895+0000 7f5732f18640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f572c144060 0x7f572c1444f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.895+0000 7f5732f18640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f572c14dbe0 0x7f572c1efdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.895+0000 7f5732f18640 1 --2- 172.21.15.83:0/918310809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f572c0aa4a0 0x7f572c0af1c0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.895+0000 7f5732f18640 1 -- 172.21.15.83:0/918310809 >> 172.21.15.83:0/918310809 conn(0x7f572c0c95a0 msgr2=0x7f572c0c6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.895+0000 7f5732f18640 1 -- 172.21.15.83:0/918310809 shutdown_connections 2024-09-19T07:37:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:21.895+0000 7f5732f18640 1 -- 172.21.15.83:0/918310809 wait complete. 2024-09-19T07:37:21.908 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=6918 2024-09-19T07:37:21.909 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.c 2024-09-19T07:37:22.071 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:22.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e7034f640 1 Processor -- start 2024-09-19T07:37:22.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e7034f640 1 -- start start 2024-09-19T07:37:22.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e7034f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5e680c3ab0 0x7f5e68157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:22.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e7034f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5e680aa4a0 0x7f5e68157550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:22.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e7034f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e68073d90 con 0x7f5e6814dbe0 2024-09-19T07:37:22.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e7034f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5e68154700 con 0x7f5e680aa4a0 2024-09-19T07:37:22.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e7034f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5e68153b30 con 0x7f5e680c3ab0 2024-09-19T07:37:22.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e6e8ec640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5e680aa4a0 0x7f5e68157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:22.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e6e8ec640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5e680aa4a0 0x7f5e68157550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60300/0 (socket says 172.21.15.83:60300) 2024-09-19T07:37:22.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e6e8ec640 1 -- 172.21.15.83:0/2689818211 learned_addr learned my addr 172.21.15.83:0/2689818211 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:22.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e6d8ea640 1 --2- 172.21.15.83:0/2689818211 >> v2:172.21.15.184:3300/0 conn(0x7f5e680c3ab0 0x7f5e68157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:22.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e6e8ec640 1 -- 172.21.15.83:0/2689818211 >> v2:172.21.15.184:3300/0 conn(0x7f5e680c3ab0 msgr2=0x7f5e68157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:22.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e6e8ec640 1 --2- 172.21.15.83:0/2689818211 >> v2:172.21.15.184:3300/0 conn(0x7f5e680c3ab0 0x7f5e68157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:22.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e6e8ec640 1 -- 172.21.15.83:0/2689818211 >> v1:172.21.15.83:6789/0 conn(0x7f5e6814dbe0 legacy=0x7f5e680c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:22.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.211+0000 7f5e6e8ec640 1 -- 172.21.15.83:0/2689818211 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e68153dd0 con 0x7f5e680aa4a0 2024-09-19T07:37:22.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6e8ec640 1 --2- 172.21.15.83:0/2689818211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e680aa4a0 0x7f5e68157550 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f5e680c3830 tx=0x7f5e6001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f428120014cf0094 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:22.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6d0e9640 1 -- 172.21.15.83:0/2689818211 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e600a5730 con 0x7f5e680aa4a0 2024-09-19T07:37:22.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6d0e9640 1 -- 172.21.15.83:0/2689818211 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e600a58d0 con 0x7f5e680aa4a0 2024-09-19T07:37:22.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6d0e9640 1 -- 172.21.15.83:0/2689818211 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e600a5ca0 con 0x7f5e680aa4a0 2024-09-19T07:37:22.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 -- 172.21.15.83:0/2689818211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e680aa4a0 msgr2=0x7f5e68157550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:22.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 --2- 172.21.15.83:0/2689818211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e680aa4a0 0x7f5e68157550 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f5e680c3830 tx=0x7f5e6001c960 comp rx=0 tx=0).stop 2024-09-19T07:37:22.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 -- 172.21.15.83:0/2689818211 shutdown_connections 2024-09-19T07:37:22.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 --2- 172.21.15.83:0/2689818211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e680aa4a0 0x7f5e68157550 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:22.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 --2- 172.21.15.83:0/2689818211 >> v2:172.21.15.184:3300/0 conn(0x7f5e680c3ab0 0x7f5e68157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:22.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 -- 172.21.15.83:0/2689818211 >> 172.21.15.83:0/2689818211 conn(0x7f5e680c95a0 msgr2=0x7f5e680c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:22.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 -- 172.21.15.83:0/2689818211 shutdown_connections 2024-09-19T07:37:22.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 -- 172.21.15.83:0/2689818211 wait complete. 2024-09-19T07:37:22.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 Processor -- start 2024-09-19T07:37:22.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 -- start start 2024-09-19T07:37:22.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e680aa4a0 0x7f5e6814e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:22.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e6814dbe0 0x7f5e68149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:22.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e68147e90 0x7f5e68144980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:22.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5e680c4830 con 0x7f5e6814dbe0 2024-09-19T07:37:22.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5e680c46b0 con 0x7f5e68147e90 2024-09-19T07:37:22.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e7034f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5e68160bc0 con 0x7f5e680aa4a0 2024-09-19T07:37:22.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6d8ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e6814dbe0 0x7f5e68149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:22.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6d8ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e6814dbe0 0x7f5e68149e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60314/0 (socket says 172.21.15.83:60314) 2024-09-19T07:37:22.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6d8ea640 1 -- 172.21.15.83:0/3606667892 learned_addr learned my addr 172.21.15.83:0/3606667892 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:22.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6e0eb640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e680aa4a0 0x7f5e6814e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:22.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6e8ec640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e68147e90 0x7f5e68144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:22.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6d8ea640 1 -- 172.21.15.83:0/3606667892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e680aa4a0 msgr2=0x7f5e6814e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:22.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6d8ea640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e680aa4a0 0x7f5e6814e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:22.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6d8ea640 1 -- 172.21.15.83:0/3606667892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e68147e90 msgr2=0x7f5e68144980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:22.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6d8ea640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e68147e90 0x7f5e68144980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:22.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.215+0000 7f5e6d8ea640 1 -- 172.21.15.83:0/3606667892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e680c41b0 con 0x7f5e6814dbe0 2024-09-19T07:37:22.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.219+0000 7f5e6d8ea640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e6814dbe0 0x7f5e68149e80 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800d800 tx=0x7f5e5800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:22.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.219+0000 7f5e5effd640 1 -- 172.21.15.83:0/3606667892 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5800b6c0 con 0x7f5e6814dbe0 2024-09-19T07:37:22.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.219+0000 7f5e5effd640 1 -- 172.21.15.83:0/3606667892 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e58019070 con 0x7f5e6814dbe0 2024-09-19T07:37:22.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.219+0000 7f5e7034f640 1 -- 172.21.15.83:0/3606667892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e681edab0 con 0x7f5e6814dbe0 2024-09-19T07:37:22.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.219+0000 7f5e5effd640 1 -- 172.21.15.83:0/3606667892 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5800ba60 con 0x7f5e6814dbe0 2024-09-19T07:37:22.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.219+0000 7f5e7034f640 1 -- 172.21.15.83:0/3606667892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e6814ff50 con 0x7f5e6814dbe0 2024-09-19T07:37:22.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.219+0000 7f5e7034f640 1 -- 172.21.15.83:0/3606667892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e680c3fa0 con 0x7f5e6814dbe0 2024-09-19T07:37:22.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.219+0000 7f5e5effd640 1 -- 172.21.15.83:0/3606667892 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5e5800bc00 con 0x7f5e6814dbe0 2024-09-19T07:37:22.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.219+0000 7f5e5effd640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5e4c0678c0 0x7f5e4c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:22.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.219+0000 7f5e5effd640 1 -- 172.21.15.83:0/3606667892 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7f5e58086c20 con 0x7f5e6814dbe0 2024-09-19T07:37:22.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.219+0000 7f5e6e0eb640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5e4c0678c0 0x7f5e4c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:22.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.219+0000 7f5e6e0eb640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5e4c0678c0 0x7f5e4c069d80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5e6814d930 tx=0x7f5e64009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:22.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.223+0000 7f5e5effd640 1 -- 172.21.15.83:0/3606667892 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5e580564f0 con 0x7f5e6814dbe0 2024-09-19T07:37:22.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.499+0000 7f5e7034f640 1 -- 172.21.15.83:0/3606667892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e6814cd70 con 0x7f5e6814dbe0 2024-09-19T07:37:22.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.499+0000 7f5e5effd640 1 -- 172.21.15.83:0/3606667892 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) ==== 95+0+4889 (secure 0 0 0) 0x7f5e5805a7f0 con 0x7f5e6814dbe0 2024-09-19T07:37:22.494 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:22.494 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":133,"btime":"2024-09-19T07:37:21:194300+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:37:22.494 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 133 2024-09-19T07:37:22.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.503+0000 7f5e7034f640 1 -- 172.21.15.83:0/3606667892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5e4c0678c0 msgr2=0x7f5e4c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:22.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.503+0000 7f5e7034f640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5e4c0678c0 0x7f5e4c069d80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5e6814d930 tx=0x7f5e64009040 comp rx=0 tx=0).stop 2024-09-19T07:37:22.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.503+0000 7f5e7034f640 1 -- 172.21.15.83:0/3606667892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e6814dbe0 msgr2=0x7f5e68149e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:22.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.503+0000 7f5e7034f640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e6814dbe0 0x7f5e68149e80 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800d800 tx=0x7f5e5800dcc0 comp rx=0 tx=0).stop 2024-09-19T07:37:22.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.503+0000 7f5e7034f640 1 -- 172.21.15.83:0/3606667892 shutdown_connections 2024-09-19T07:37:22.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.503+0000 7f5e7034f640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5e4c0678c0 0x7f5e4c069d80 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:22.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.503+0000 7f5e7034f640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e68147e90 0x7f5e68144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:22.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.503+0000 7f5e7034f640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e6814dbe0 0x7f5e68149e80 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:22.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.503+0000 7f5e7034f640 1 --2- 172.21.15.83:0/3606667892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e680aa4a0 0x7f5e6814e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:22.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.503+0000 7f5e7034f640 1 -- 172.21.15.83:0/3606667892 >> 172.21.15.83:0/3606667892 conn(0x7f5e680c95a0 msgr2=0x7f5e680c6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:22.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.503+0000 7f5e7034f640 1 -- 172.21.15.83:0/3606667892 shutdown_connections 2024-09-19T07:37:22.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:22.503+0000 7f5e7034f640 1 -- 172.21.15.83:0/3606667892 wait complete. 2024-09-19T07:37:22.513 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=5612 2024-09-19T07:37:22.514 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.b 2024-09-19T07:37:25.562 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:37:25.567+0000 7f932f5dd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:37:26.515 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-09-19T07:37:26.515 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-19T07:37:26.515 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-19T07:37:26.518 INFO:tasks.ceph.mds.b:Started 2024-09-19T07:37:26.518 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:26.591 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-19T07:37:26.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf68940640 1 Processor -- start 2024-09-19T07:37:26.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf68940640 1 -- start start 2024-09-19T07:37:26.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf68940640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf6414d9f0 0x7fdf6406de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:26.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf68940640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdf64154360 0x7fdf640c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:26.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf68940640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf6414ddd0 con 0x7fdf640c7510 2024-09-19T07:37:26.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf68940640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fdf640c8830 con 0x7fdf64154360 2024-09-19T07:37:26.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf68940640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdf6406fdf0 con 0x7fdf6414d9f0 2024-09-19T07:37:26.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf62d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdf64154360 0x7fdf640c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:26.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf62d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdf64154360 0x7fdf640c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60318/0 (socket says 172.21.15.83:60318) 2024-09-19T07:37:26.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf62d9d640 1 -- 172.21.15.83:0/62261479 learned_addr learned my addr 172.21.15.83:0/62261479 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:26.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf6259c640 1 --2- 172.21.15.83:0/62261479 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf6414d9f0 0x7fdf6406de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:26.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf62d9d640 1 -- 172.21.15.83:0/62261479 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf6414d9f0 msgr2=0x7fdf6406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:26.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf62d9d640 1 --2- 172.21.15.83:0/62261479 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf6414d9f0 0x7fdf6406de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:26.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf62d9d640 1 -- 172.21.15.83:0/62261479 >> v1:172.21.15.184:6789/0 conn(0x7fdf640c7510 legacy=0x7fdf640c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:26.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.655+0000 7fdf62d9d640 1 -- 172.21.15.83:0/62261479 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf640c7f00 con 0x7fdf64154360 2024-09-19T07:37:26.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.659+0000 7fdf62d9d640 1 --2- 172.21.15.83:0/62261479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf64154360 0x7fdf640c4010 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009e50 tx=0x7fdf5c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d92aba89a3f8707 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:26.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.659+0000 7fdf6159a640 1 -- 172.21.15.83:0/62261479 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0a5660 con 0x7fdf64154360 2024-09-19T07:37:26.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.659+0000 7fdf6159a640 1 -- 172.21.15.83:0/62261479 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf5c0a5800 con 0x7fdf64154360 2024-09-19T07:37:26.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.659+0000 7fdf6159a640 1 -- 172.21.15.83:0/62261479 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0a5bd0 con 0x7fdf64154360 2024-09-19T07:37:26.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.659+0000 7fdf68940640 1 -- 172.21.15.83:0/62261479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf64154360 msgr2=0x7fdf640c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:26.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.659+0000 7fdf68940640 1 --2- 172.21.15.83:0/62261479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf64154360 0x7fdf640c4010 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009e50 tx=0x7fdf5c01c9a0 comp rx=0 tx=0).stop 2024-09-19T07:37:26.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.659+0000 7fdf68940640 1 -- 172.21.15.83:0/62261479 shutdown_connections 2024-09-19T07:37:26.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.659+0000 7fdf68940640 1 --2- 172.21.15.83:0/62261479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf64154360 0x7fdf640c4010 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:26.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.659+0000 7fdf68940640 1 --2- 172.21.15.83:0/62261479 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf6414d9f0 0x7fdf6406de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:26.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.659+0000 7fdf68940640 1 -- 172.21.15.83:0/62261479 >> 172.21.15.83:0/62261479 conn(0x7fdf640c6110 msgr2=0x7fdf640c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:26.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.659+0000 7fdf68940640 1 -- 172.21.15.83:0/62261479 shutdown_connections 2024-09-19T07:37:26.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.659+0000 7fdf68940640 1 -- 172.21.15.83:0/62261479 wait complete. 2024-09-19T07:37:26.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf68940640 1 Processor -- start 2024-09-19T07:37:26.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf68940640 1 -- start start 2024-09-19T07:37:26.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf68940640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf640c7510 0x7fdf640c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf68940640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf6414d9f0 0x7fdf6414c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf68940640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf64154360 0x7fdf6414caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf68940640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf64160c00 con 0x7fdf64154360 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf68940640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdf64160a80 con 0x7fdf6414d9f0 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf68940640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdf64160d80 con 0x7fdf640c7510 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf62d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf64154360 0x7fdf6414caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf62d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf64154360 0x7fdf6414caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60334/0 (socket says 172.21.15.83:60334) 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf62d9d640 1 -- 172.21.15.83:0/2077523717 learned_addr learned my addr 172.21.15.83:0/2077523717 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf6259c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf640c7510 0x7fdf640c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf61d9b640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf6414d9f0 0x7fdf6414c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf62d9d640 1 -- 172.21.15.83:0/2077523717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf640c7510 msgr2=0x7fdf640c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf62d9d640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf640c7510 0x7fdf640c2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:26.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf62d9d640 1 -- 172.21.15.83:0/2077523717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf6414d9f0 msgr2=0x7fdf6414c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:26.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf62d9d640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf6414d9f0 0x7fdf6414c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:26.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf62d9d640 1 -- 172.21.15.83:0/2077523717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf64141120 con 0x7fdf64154360 2024-09-19T07:37:26.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf6259c640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf640c7510 0x7fdf640c2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:37:26.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf62d9d640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf64154360 0x7fdf6414caa0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c01a5d0 tx=0x7fdf5c019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:26.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf52ffd640 1 -- 172.21.15.83:0/2077523717 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0bc020 con 0x7fdf64154360 2024-09-19T07:37:26.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf52ffd640 1 -- 172.21.15.83:0/2077523717 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf5c0b8670 con 0x7fdf64154360 2024-09-19T07:37:26.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.663+0000 7fdf52ffd640 1 -- 172.21.15.83:0/2077523717 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c009510 con 0x7fdf64154360 2024-09-19T07:37:26.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.667+0000 7fdf68940640 1 -- 172.21.15.83:0/2077523717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf64149870 con 0x7fdf64154360 2024-09-19T07:37:26.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.667+0000 7fdf68940640 1 -- 172.21.15.83:0/2077523717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf641415b0 con 0x7fdf64154360 2024-09-19T07:37:26.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.667+0000 7fdf68940640 1 -- 172.21.15.83:0/2077523717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf6406de90 con 0x7fdf64154360 2024-09-19T07:37:26.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.667+0000 7fdf52ffd640 1 -- 172.21.15.83:0/2077523717 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdf5c0bf020 con 0x7fdf64154360 2024-09-19T07:37:26.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.667+0000 7fdf52ffd640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf440678c0 0x7fdf44069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:26.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.667+0000 7fdf52ffd640 1 -- 172.21.15.83:0/2077523717 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7fdf5c121570 con 0x7fdf64154360 2024-09-19T07:37:26.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.667+0000 7fdf6259c640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf440678c0 0x7fdf44069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:26.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.667+0000 7fdf6259c640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf440678c0 0x7fdf44069d80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fdf54009eb0 tx=0x7fdf54009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:26.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.671+0000 7fdf52ffd640 1 -- 172.21.15.83:0/2077523717 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdf5c0c6030 con 0x7fdf64154360 2024-09-19T07:37:26.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.891+0000 7fdf68940640 1 -- 172.21.15.83:0/2077523717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf6406faf0 con 0x7fdf64154360 2024-09-19T07:37:26.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.891+0000 7fdf52ffd640 1 -- 172.21.15.83:0/2077523717 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) ==== 95+0+4889 (secure 0 0 0) 0x7fdf64141120 con 0x7fdf64154360 2024-09-19T07:37:26.886 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:26.887 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":133,"btime":"2024-09-19T07:37:21:194300+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:37:26.887 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 133 2024-09-19T07:37:26.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.895+0000 7fdf68940640 1 -- 172.21.15.83:0/2077523717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf440678c0 msgr2=0x7fdf44069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:26.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.895+0000 7fdf68940640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf440678c0 0x7fdf44069d80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fdf54009eb0 tx=0x7fdf54009190 comp rx=0 tx=0).stop 2024-09-19T07:37:26.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.895+0000 7fdf68940640 1 -- 172.21.15.83:0/2077523717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf64154360 msgr2=0x7fdf6414caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:26.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.895+0000 7fdf68940640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf64154360 0x7fdf6414caa0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c01a5d0 tx=0x7fdf5c019740 comp rx=0 tx=0).stop 2024-09-19T07:37:26.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.895+0000 7fdf68940640 1 -- 172.21.15.83:0/2077523717 shutdown_connections 2024-09-19T07:37:26.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.895+0000 7fdf68940640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf440678c0 0x7fdf44069d80 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:26.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.895+0000 7fdf68940640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf64154360 0x7fdf6414caa0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:26.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.895+0000 7fdf68940640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf6414d9f0 0x7fdf6414c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:26.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.895+0000 7fdf68940640 1 --2- 172.21.15.83:0/2077523717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf640c7510 0x7fdf640c2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:26.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.895+0000 7fdf68940640 1 -- 172.21.15.83:0/2077523717 >> 172.21.15.83:0/2077523717 conn(0x7fdf640c6110 msgr2=0x7fdf640c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:26.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.895+0000 7fdf68940640 1 -- 172.21.15.83:0/2077523717 shutdown_connections 2024-09-19T07:37:26.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:26.895+0000 7fdf68940640 1 -- 172.21.15.83:0/2077523717 wait complete. 2024-09-19T07:37:26.907 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-19T07:37:28.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:29.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.051+0000 7f4b58f19640 1 Processor -- start 2024-09-19T07:37:29.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.051+0000 7f4b58f19640 1 -- start start 2024-09-19T07:37:29.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.051+0000 7f4b58f19640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b540c6120 0x7f4b540c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:29.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.051+0000 7f4b58f19640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4b54155d70 0x7f4b541575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:29.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.051+0000 7f4b58f19640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b54156200 con 0x7f4b541592f0 2024-09-19T07:37:29.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.051+0000 7f4b58f19640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f4b540c9930 con 0x7f4b54155d70 2024-09-19T07:37:29.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.051+0000 7f4b58f19640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4b541564e0 con 0x7f4b540c6120 2024-09-19T07:37:29.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.051+0000 7f4b51d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4b54155d70 0x7f4b541575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:29.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.055+0000 7f4b51d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4b54155d70 0x7f4b541575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60344/0 (socket says 172.21.15.83:60344) 2024-09-19T07:37:29.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.055+0000 7f4b51d9b640 1 -- 172.21.15.83:0/580115959 learned_addr learned my addr 172.21.15.83:0/580115959 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:29.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.055+0000 7f4b5259c640 1 --2- 172.21.15.83:0/580115959 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b540c6120 0x7f4b540c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:29.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.055+0000 7f4b51d9b640 1 -- 172.21.15.83:0/580115959 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b540c6120 msgr2=0x7f4b540c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:29.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.055+0000 7f4b51d9b640 1 --2- 172.21.15.83:0/580115959 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b540c6120 0x7f4b540c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:29.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.055+0000 7f4b51d9b640 1 -- 172.21.15.83:0/580115959 >> v1:172.21.15.83:6789/0 conn(0x7f4b541592f0 legacy=0x7f4b54157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:29.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.055+0000 7f4b5159a640 1 -- 172.21.15.83:0/580115959 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3376605236 0 0) 0x7f4b54156200 con 0x7f4b541592f0 2024-09-19T07:37:29.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.055+0000 7f4b51d9b640 1 -- 172.21.15.83:0/580115959 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b541567c0 con 0x7f4b54155d70 2024-09-19T07:37:29.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.055+0000 7f4b51d9b640 1 --2- 172.21.15.83:0/580115959 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b54155d70 0x7f4b541575d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009870 tx=0x7f4b4801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75671caf337dd9e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:29.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.055+0000 7f4b5159a640 1 -- 172.21.15.83:0/580115959 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b480b2070 con 0x7f4b54155d70 2024-09-19T07:37:29.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.055+0000 7f4b5159a640 1 -- 172.21.15.83:0/580115959 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b480a9070 con 0x7f4b54155d70 2024-09-19T07:37:29.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b5159a640 1 -- 172.21.15.83:0/580115959 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b480ad880 con 0x7f4b54155d70 2024-09-19T07:37:29.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 -- 172.21.15.83:0/580115959 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b54155d70 msgr2=0x7f4b541575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:29.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 --2- 172.21.15.83:0/580115959 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b54155d70 0x7f4b541575d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009870 tx=0x7f4b4801c560 comp rx=0 tx=0).stop 2024-09-19T07:37:29.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 -- 172.21.15.83:0/580115959 shutdown_connections 2024-09-19T07:37:29.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 --2- 172.21.15.83:0/580115959 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b54155d70 0x7f4b541575d0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:29.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 --2- 172.21.15.83:0/580115959 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b540c6120 0x7f4b540c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:29.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 -- 172.21.15.83:0/580115959 >> 172.21.15.83:0/580115959 conn(0x7f4b5406bc80 msgr2=0x7f4b5406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:29.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 -- 172.21.15.83:0/580115959 shutdown_connections 2024-09-19T07:37:29.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 -- 172.21.15.83:0/580115959 wait complete. 2024-09-19T07:37:29.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 Processor -- start 2024-09-19T07:37:29.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 -- start start 2024-09-19T07:37:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b540c6120 0x7f4b541f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b541592f0 0x7f4b540c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b540c0ff0 0x7f4b540c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4b54153860 con 0x7f4b540c0ff0 2024-09-19T07:37:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b54162ea0 con 0x7f4b540c6120 2024-09-19T07:37:29.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b58f19640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4b541ef380 con 0x7f4b541592f0 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b52d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b540c0ff0 0x7f4b540c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b52d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b540c0ff0 0x7f4b540c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60356/0 (socket says 172.21.15.83:60356) 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b52d9d640 1 -- 172.21.15.83:0/3561682817 learned_addr learned my addr 172.21.15.83:0/3561682817 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b51d9b640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b541592f0 0x7f4b540c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b5259c640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b540c6120 0x7f4b541f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b52d9d640 1 -- 172.21.15.83:0/3561682817 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b541592f0 msgr2=0x7f4b540c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b52d9d640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b541592f0 0x7f4b540c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b52d9d640 1 -- 172.21.15.83:0/3561682817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b540c6120 msgr2=0x7f4b541f2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b52d9d640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b540c6120 0x7f4b541f2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b52d9d640 1 -- 172.21.15.83:0/3561682817 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b541ee5c0 con 0x7f4b540c0ff0 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b5259c640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b540c6120 0x7f4b541f2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b51d9b640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b541592f0 0x7f4b540c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:37:29.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.059+0000 7f4b52d9d640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b540c0ff0 0x7f4b540c9e10 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00c780 tx=0x7f4b4c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:29.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.063+0000 7f4b42ffd640 1 -- 172.21.15.83:0/3561682817 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c003db0 con 0x7f4b540c0ff0 2024-09-19T07:37:29.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.063+0000 7f4b42ffd640 1 -- 172.21.15.83:0/3561682817 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b4c00ce30 con 0x7f4b540c0ff0 2024-09-19T07:37:29.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.063+0000 7f4b58f19640 1 -- 172.21.15.83:0/3561682817 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b541ee8a0 con 0x7f4b540c0ff0 2024-09-19T07:37:29.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.063+0000 7f4b42ffd640 1 -- 172.21.15.83:0/3561682817 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c005290 con 0x7f4b540c0ff0 2024-09-19T07:37:29.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.063+0000 7f4b58f19640 1 -- 172.21.15.83:0/3561682817 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b54144cc0 con 0x7f4b540c0ff0 2024-09-19T07:37:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.063+0000 7f4b58f19640 1 -- 172.21.15.83:0/3561682817 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b540c6500 con 0x7f4b540c0ff0 2024-09-19T07:37:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.063+0000 7f4b42ffd640 1 -- 172.21.15.83:0/3561682817 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4b4c005430 con 0x7f4b540c0ff0 2024-09-19T07:37:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.063+0000 7f4b42ffd640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4b340678c0 0x7f4b34069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.063+0000 7f4b42ffd640 1 -- 172.21.15.83:0/3561682817 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7f4b4c086810 con 0x7f4b540c0ff0 2024-09-19T07:37:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.063+0000 7f4b5259c640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4b340678c0 0x7f4b34069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:29.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.063+0000 7f4b5259c640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4b340678c0 0x7f4b34069d80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4b44007a70 tx=0x7f4b44008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:29.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.071+0000 7f4b42ffd640 1 -- 172.21.15.83:0/3561682817 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4b4c0566a0 con 0x7f4b540c0ff0 2024-09-19T07:37:29.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.311+0000 7f4b58f19640 1 -- 172.21.15.83:0/3561682817 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b540c96a0 con 0x7f4b540c0ff0 2024-09-19T07:37:29.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.311+0000 7f4b42ffd640 1 -- 172.21.15.83:0/3561682817 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5717 (secure 0 0 0) 0x7f4b4c055fa0 con 0x7f4b540c0ff0 2024-09-19T07:37:29.306 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:29.306 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":134,"btime":"2024-09-19T07:37:27:316638+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:37:29.307 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 134 2024-09-19T07:37:29.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.315+0000 7f4b58f19640 1 -- 172.21.15.83:0/3561682817 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4b340678c0 msgr2=0x7f4b34069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:29.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.315+0000 7f4b58f19640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4b340678c0 0x7f4b34069d80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4b44007a70 tx=0x7f4b44008040 comp rx=0 tx=0).stop 2024-09-19T07:37:29.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.315+0000 7f4b58f19640 1 -- 172.21.15.83:0/3561682817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b540c0ff0 msgr2=0x7f4b540c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:29.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.315+0000 7f4b58f19640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b540c0ff0 0x7f4b540c9e10 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00c780 tx=0x7f4b4c00cc40 comp rx=0 tx=0).stop 2024-09-19T07:37:29.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.315+0000 7f4b58f19640 1 -- 172.21.15.83:0/3561682817 shutdown_connections 2024-09-19T07:37:29.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.315+0000 7f4b58f19640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4b340678c0 0x7f4b34069d80 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:29.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.315+0000 7f4b58f19640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b540c0ff0 0x7f4b540c9e10 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:29.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.315+0000 7f4b58f19640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b541592f0 0x7f4b540c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:29.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.315+0000 7f4b58f19640 1 --2- 172.21.15.83:0/3561682817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b540c6120 0x7f4b541f2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:29.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.315+0000 7f4b58f19640 1 -- 172.21.15.83:0/3561682817 >> 172.21.15.83:0/3561682817 conn(0x7f4b5406bc80 msgr2=0x7f4b5406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:29.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.315+0000 7f4b58f19640 1 -- 172.21.15.83:0/3561682817 shutdown_connections 2024-09-19T07:37:29.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:29.315+0000 7f4b58f19640 1 -- 172.21.15.83:0/3561682817 wait complete. 2024-09-19T07:37:29.329 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-09-19T07:37:29.330 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2024-09-19T07:37:29.909 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-19T07:37:29.909 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:37:29.910 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:37:29.912 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:37:29.912 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:29.988 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:37:30.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.079+0000 7f7c8dfb2640 1 Processor -- start 2024-09-19T07:37:30.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.079+0000 7f7c8dfb2640 1 -- start start 2024-09-19T07:37:30.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c8dfb2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c8814fdd0 0x7f7c880c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:30.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c8dfb2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7c8806ed40 0x7f7c880c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:30.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c8dfb2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c88153820 con 0x7f7c8814f9f0 2024-09-19T07:37:30.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c8dfb2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f7c880c2890 con 0x7f7c8806ed40 2024-09-19T07:37:30.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c8dfb2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7c880c2b70 con 0x7f7c8814fdd0 2024-09-19T07:37:30.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c877fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7c8814f9f0 0x7f7c88155b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53476/0 (socket says 172.21.15.83:53476) 2024-09-19T07:37:30.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c877fe640 1 -- 172.21.15.83:0/3887976826 learned_addr learned my addr 172.21.15.83:0/3887976826 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:37:30.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c7effd640 1 --2- 172.21.15.83:0/3887976826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c8814fdd0 0x7f7c880c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:30.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c87fff640 1 --2- 172.21.15.83:0/3887976826 >> v2:172.21.15.184:3300/0 conn(0x7f7c8806ed40 0x7f7c880c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:30.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3508796244 0 0) 0x7f7c78001120 con 0x7f7c8814f9f0 2024-09-19T07:37:30.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c74003530 con 0x7f7c8814f9f0 2024-09-19T07:37:30.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1815190752 0 0) 0x7f7c74003530 con 0x7f7c8814f9f0 2024-09-19T07:37:30.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c8814fdd0 msgr2=0x7f7c880c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:30.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 --2- 172.21.15.83:0/3887976826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c8814fdd0 0x7f7c880c2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:30.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 >> v2:172.21.15.184:3300/0 conn(0x7f7c8806ed40 msgr2=0x7f7c880c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:30.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 --2- 172.21.15.83:0/3887976826 >> v2:172.21.15.184:3300/0 conn(0x7f7c8806ed40 0x7f7c880c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:30.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c88159220 con 0x7f7c8814f9f0 2024-09-19T07:37:30.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f7c78001850 con 0x7f7c8814f9f0 2024-09-19T07:37:30.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 >> v1:172.21.15.83:6789/0 conn(0x7f7c8814f9f0 legacy=0x7f7c88155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:30.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 --2- 172.21.15.83:0/3887976826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c740039c0 0x7f7c74003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:30.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 --2- 172.21.15.83:0/3887976826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c74004810 0x7f7c74004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:30.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 --2- 172.21.15.83:0/3887976826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c74005190 0x7f7c7400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:30.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7c880c2890 con 0x7f7c740039c0 2024-09-19T07:37:30.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c880c2b70 con 0x7f7c74005190 2024-09-19T07:37:30.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7c74004640 con 0x7f7c74004810 2024-09-19T07:37:30.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c877fe640 1 --2- 172.21.15.83:0/3887976826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c740039c0 0x7f7c74003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:30.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c7effd640 1 --2- 172.21.15.83:0/3887976826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c74004810 0x7f7c74004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:30.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c87fff640 1 --2- 172.21.15.83:0/3887976826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c74005190 0x7f7c7400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:30.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c877fe640 1 -- 172.21.15.83:0/3887976826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c74004810 msgr2=0x7f7c74004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:30.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c877fe640 1 --2- 172.21.15.83:0/3887976826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c74004810 0x7f7c74004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:30.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c877fe640 1 -- 172.21.15.83:0/3887976826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c74005190 msgr2=0x7f7c7400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:30.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c877fe640 1 --2- 172.21.15.83:0/3887976826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c74005190 0x7f7c7400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:30.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.083+0000 7f7c877fe640 1 -- 172.21.15.83:0/3887976826 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c74003530 con 0x7f7c740039c0 2024-09-19T07:37:30.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c877fe640 1 --2- 172.21.15.83:0/3887976826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c740039c0 0x7f7c74003de0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800c490 tx=0x7f7c7801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=296ced39b6e1ace9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:30.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7800c0e0 con 0x7f7c740039c0 2024-09-19T07:37:30.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c780a9020 con 0x7f7c740039c0 2024-09-19T07:37:30.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c86ffd640 1 -- 172.21.15.83:0/3887976826 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c78005b20 con 0x7f7c740039c0 2024-09-19T07:37:30.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/3887976826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c740039c0 msgr2=0x7f7c74003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:30.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 --2- 172.21.15.83:0/3887976826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c740039c0 0x7f7c74003de0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800c490 tx=0x7f7c7801cca0 comp rx=0 tx=0).stop 2024-09-19T07:37:30.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c87fff640 1 -- 172.21.15.83:0/3887976826 reap_dead start 2024-09-19T07:37:30.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/3887976826 shutdown_connections 2024-09-19T07:37:30.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/3887976826 >> 172.21.15.83:0/3887976826 conn(0x7f7c880c5090 msgr2=0x7f7c880c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:30.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/3887976826 shutdown_connections 2024-09-19T07:37:30.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/3887976826 wait complete. 2024-09-19T07:37:30.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 Processor -- start 2024-09-19T07:37:30.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 -- start start 2024-09-19T07:37:30.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c740039c0 0x7f7c880ae210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:30.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c88155a00 0x7f7c881ee840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:30.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c881f26e0 0x7f7c881520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:30.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7c880c2310 con 0x7f7c740039c0 2024-09-19T07:37:30.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c880c2190 con 0x7f7c88155a00 2024-09-19T07:37:30.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c8dfb2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7c880c2490 con 0x7f7c881f26e0 2024-09-19T07:37:30.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c877fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c740039c0 0x7f7c880ae210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:30.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c877fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c740039c0 0x7f7c880ae210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60384/0 (socket says 172.21.15.83:60384) 2024-09-19T07:37:30.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c877fe640 1 -- 172.21.15.83:0/2115043967 learned_addr learned my addr 172.21.15.83:0/2115043967 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:30.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c7effd640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c88155a00 0x7f7c881ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:30.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c87fff640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c881f26e0 0x7f7c881520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c877fe640 1 -- 172.21.15.83:0/2115043967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c881f26e0 msgr2=0x7f7c881520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c877fe640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c881f26e0 0x7f7c881520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c877fe640 1 -- 172.21.15.83:0/2115043967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c88155a00 msgr2=0x7f7c881ee840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c877fe640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c88155a00 0x7f7c881ee840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c877fe640 1 -- 172.21.15.83:0/2115043967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c881eed80 con 0x7f7c740039c0 2024-09-19T07:37:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c87fff640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c881f26e0 0x7f7c881520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:37:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.087+0000 7f7c7effd640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c88155a00 0x7f7c881ee840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:37:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.091+0000 7f7c877fe640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c740039c0 0x7f7c880ae210 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800c490 tx=0x7f7c74005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.091+0000 7f7c84ff9640 1 -- 172.21.15.83:0/2115043967 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c78001810 con 0x7f7c740039c0 2024-09-19T07:37:30.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.091+0000 7f7c84ff9640 1 -- 172.21.15.83:0/2115043967 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c780019b0 con 0x7f7c740039c0 2024-09-19T07:37:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.091+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/2115043967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c88150230 con 0x7f7c740039c0 2024-09-19T07:37:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.091+0000 7f7c84ff9640 1 -- 172.21.15.83:0/2115043967 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c780a5a70 con 0x7f7c740039c0 2024-09-19T07:37:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.091+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/2115043967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c881f21c0 con 0x7f7c740039c0 2024-09-19T07:37:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.091+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/2115043967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c88070790 con 0x7f7c740039c0 2024-09-19T07:37:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.091+0000 7f7c84ff9640 1 -- 172.21.15.83:0/2115043967 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7c780a5c10 con 0x7f7c740039c0 2024-09-19T07:37:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.091+0000 7f7c84ff9640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c680678f0 0x7f7c68069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.091+0000 7f7c84ff9640 1 -- 172.21.15.83:0/2115043967 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7f7c78120090 con 0x7f7c740039c0 2024-09-19T07:37:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.091+0000 7f7c7effd640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c680678f0 0x7f7c68069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:30.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.091+0000 7f7c7effd640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c680678f0 0x7f7c68069db0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7c8815b740 tx=0x7f7c70008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:30.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.095+0000 7f7c84ff9640 1 -- 172.21.15.83:0/2115043967 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7c88153820 con 0x7f7c740039c0 2024-09-19T07:37:30.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.331+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/2115043967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c88068900 con 0x7f7c740039c0 2024-09-19T07:37:30.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.331+0000 7f7c84ff9640 1 -- 172.21.15.83:0/2115043967 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5717 (secure 0 0 0) 0x7f7c780effb0 con 0x7f7c740039c0 2024-09-19T07:37:30.325 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:30.326 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":134,"btime":"2024-09-19T07:37:27:316638+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:37:30.326 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 134 2024-09-19T07:37:30.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.335+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/2115043967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c680678f0 msgr2=0x7f7c68069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:30.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.335+0000 7f7c8dfb2640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c680678f0 0x7f7c68069db0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7c8815b740 tx=0x7f7c70008040 comp rx=0 tx=0).stop 2024-09-19T07:37:30.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.335+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/2115043967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c740039c0 msgr2=0x7f7c880ae210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:30.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.335+0000 7f7c8dfb2640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c740039c0 0x7f7c880ae210 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800c490 tx=0x7f7c74005840 comp rx=0 tx=0).stop 2024-09-19T07:37:30.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.335+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/2115043967 shutdown_connections 2024-09-19T07:37:30.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.335+0000 7f7c8dfb2640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7c680678f0 0x7f7c68069db0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:30.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.335+0000 7f7c8dfb2640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c881f26e0 0x7f7c881520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:30.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.335+0000 7f7c8dfb2640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c88155a00 0x7f7c881ee840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:30.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.335+0000 7f7c8dfb2640 1 --2- 172.21.15.83:0/2115043967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c740039c0 0x7f7c880ae210 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:30.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.335+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/2115043967 >> 172.21.15.83:0/2115043967 conn(0x7f7c880c5090 msgr2=0x7f7c8815e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:30.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.335+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/2115043967 shutdown_connections 2024-09-19T07:37:30.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:30.335+0000 7f7c8dfb2640 1 -- 172.21.15.83:0/2115043967 wait complete. 2024-09-19T07:37:30.354 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-19T07:37:32.355 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:32.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.503+0000 7f2e47020640 1 Processor -- start 2024-09-19T07:37:32.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.503+0000 7f2e47020640 1 -- start start 2024-09-19T07:37:32.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.503+0000 7f2e47020640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2e40153300 0x7f2e4006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:32.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.503+0000 7f2e47020640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2e4006d1d0 0x7f2e400c23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:32.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.503+0000 7f2e47020640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e401593f0 con 0x7f2e400c2910 2024-09-19T07:37:32.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.503+0000 7f2e47020640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f2e401591a0 con 0x7f2e40153300 2024-09-19T07:37:32.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.503+0000 7f2e47020640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f2e401596d0 con 0x7f2e4006d1d0 2024-09-19T07:37:32.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.503+0000 7f2e44dbc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2e40153300 0x7f2e4006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:32.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.503+0000 7f2e44dbc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2e40153300 0x7f2e4006cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39884/0 (socket says 172.21.15.83:39884) 2024-09-19T07:37:32.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.503+0000 7f2e44dbc640 1 -- 172.21.15.83:0/2149211526 learned_addr learned my addr 172.21.15.83:0/2149211526 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:32.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.503+0000 7f2e3ffff640 1 --2- 172.21.15.83:0/2149211526 >> v2:172.21.15.184:3300/0 conn(0x7f2e4006d1d0 0x7f2e400c23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:32.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e44dbc640 1 -- 172.21.15.83:0/2149211526 >> v2:172.21.15.184:3300/0 conn(0x7f2e4006d1d0 msgr2=0x7f2e400c23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:32.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e44dbc640 1 --2- 172.21.15.83:0/2149211526 >> v2:172.21.15.184:3300/0 conn(0x7f2e4006d1d0 0x7f2e400c23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:32.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e44dbc640 1 -- 172.21.15.83:0/2149211526 >> v1:172.21.15.184:6789/0 conn(0x7f2e400c2910 legacy=0x7f2e40155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:32.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e44dbc640 1 -- 172.21.15.83:0/2149211526 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e401599b0 con 0x7f2e40153300 2024-09-19T07:37:32.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e44dbc640 1 --2- 172.21.15.83:0/2149211526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e40153300 0x7f2e4006cc90 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f2e400c2120 tx=0x7f2e3801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271c9d7472d8cc98 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:32.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e3f7fe640 1 -- 172.21.15.83:0/2149211526 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e380a5660 con 0x7f2e40153300 2024-09-19T07:37:32.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e3f7fe640 1 -- 172.21.15.83:0/2149211526 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e380a5800 con 0x7f2e40153300 2024-09-19T07:37:32.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e3f7fe640 1 -- 172.21.15.83:0/2149211526 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e380a5bd0 con 0x7f2e40153300 2024-09-19T07:37:32.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e47020640 1 -- 172.21.15.83:0/2149211526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e40153300 msgr2=0x7f2e4006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:32.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e47020640 1 --2- 172.21.15.83:0/2149211526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e40153300 0x7f2e4006cc90 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f2e400c2120 tx=0x7f2e3801c890 comp rx=0 tx=0).stop 2024-09-19T07:37:32.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e47020640 1 -- 172.21.15.83:0/2149211526 shutdown_connections 2024-09-19T07:37:32.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e47020640 1 --2- 172.21.15.83:0/2149211526 >> v2:172.21.15.184:3300/0 conn(0x7f2e4006d1d0 0x7f2e400c23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:32.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e47020640 1 --2- 172.21.15.83:0/2149211526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e40153300 0x7f2e4006cc90 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:32.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e47020640 1 -- 172.21.15.83:0/2149211526 >> 172.21.15.83:0/2149211526 conn(0x7f2e400c5090 msgr2=0x7f2e400c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:32.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e47020640 1 -- 172.21.15.83:0/2149211526 shutdown_connections 2024-09-19T07:37:32.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.507+0000 7f2e47020640 1 -- 172.21.15.83:0/2149211526 wait complete. 2024-09-19T07:37:32.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e47020640 1 Processor -- start 2024-09-19T07:37:32.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e47020640 1 -- start start 2024-09-19T07:37:32.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e47020640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e4006d1d0 0x7f2e40150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:32.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e47020640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e400c2910 0x7f2e4014c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:32.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e47020640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e40153300 0x7f2e4014aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:32.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e47020640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2e40153860 con 0x7f2e40153300 2024-09-19T07:37:32.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e47020640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2e401536e0 con 0x7f2e400c2910 2024-09-19T07:37:32.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e47020640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2e40162d50 con 0x7f2e4006d1d0 2024-09-19T07:37:32.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e455bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e40153300 0x7f2e4014aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:32.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e455bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e40153300 0x7f2e4014aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39896/0 (socket says 172.21.15.83:39896) 2024-09-19T07:37:32.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e455bd640 1 -- 172.21.15.83:0/2126082958 learned_addr learned my addr 172.21.15.83:0/2126082958 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:32.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e44dbc640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e4006d1d0 0x7f2e40150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:32.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e3ffff640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e400c2910 0x7f2e4014c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:32.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e455bd640 1 -- 172.21.15.83:0/2126082958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e4006d1d0 msgr2=0x7f2e40150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:32.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e455bd640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e4006d1d0 0x7f2e40150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e455bd640 1 -- 172.21.15.83:0/2126082958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e400c2910 msgr2=0x7f2e4014c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e455bd640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e400c2910 0x7f2e4014c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e455bd640 1 -- 172.21.15.83:0/2126082958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e40153da0 con 0x7f2e40153300 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e455bd640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e40153300 0x7f2e4014aef0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c410 tx=0x7f2e3400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e3d7fa640 1 -- 172.21.15.83:0/2126082958 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e34017070 con 0x7f2e40153300 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e3d7fa640 1 -- 172.21.15.83:0/2126082958 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e34005230 con 0x7f2e40153300 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e47020640 1 -- 172.21.15.83:0/2126082958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e400c6460 con 0x7f2e40153300 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e3d7fa640 1 -- 172.21.15.83:0/2126082958 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e34004590 con 0x7f2e40153300 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e47020640 1 -- 172.21.15.83:0/2126082958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e40153ac0 con 0x7f2e40153300 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.511+0000 7f2e47020640 1 -- 172.21.15.83:0/2126082958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e400c23d0 con 0x7f2e40153300 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.515+0000 7f2e3d7fa640 1 -- 172.21.15.83:0/2126082958 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2e3401a070 con 0x7f2e40153300 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.515+0000 7f2e3d7fa640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2e2c0678f0 0x7f2e2c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:32.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.515+0000 7f2e44dbc640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2e2c0678f0 0x7f2e2c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:32.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.515+0000 7f2e3d7fa640 1 -- 172.21.15.83:0/2126082958 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7f2e34085b00 con 0x7f2e40153300 2024-09-19T07:37:32.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.515+0000 7f2e44dbc640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2e2c0678f0 0x7f2e2c069db0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2e380ae000 tx=0x7f2e380a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:32.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.519+0000 7f2e3d7fa640 1 -- 172.21.15.83:0/2126082958 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2e34055a20 con 0x7f2e40153300 2024-09-19T07:37:32.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.735+0000 7f2e47020640 1 -- 172.21.15.83:0/2126082958 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e4006c090 con 0x7f2e40153300 2024-09-19T07:37:32.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.735+0000 7f2e3d7fa640 1 -- 172.21.15.83:0/2126082958 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6543 (secure 0 0 0) 0x7f2e34055320 con 0x7f2e40153300 2024-09-19T07:37:32.731 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:32.731 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":135,"btime":"2024-09-19T07:37:30:334185+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:37:32.731 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 135 2024-09-19T07:37:32.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.739+0000 7f2e47020640 1 -- 172.21.15.83:0/2126082958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2e2c0678f0 msgr2=0x7f2e2c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:32.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.739+0000 7f2e47020640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2e2c0678f0 0x7f2e2c069db0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2e380ae000 tx=0x7f2e380a7070 comp rx=0 tx=0).stop 2024-09-19T07:37:32.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.739+0000 7f2e47020640 1 -- 172.21.15.83:0/2126082958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e40153300 msgr2=0x7f2e4014aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:32.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.739+0000 7f2e47020640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e40153300 0x7f2e4014aef0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c410 tx=0x7f2e3400c8d0 comp rx=0 tx=0).stop 2024-09-19T07:37:32.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.739+0000 7f2e47020640 1 -- 172.21.15.83:0/2126082958 shutdown_connections 2024-09-19T07:37:32.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.739+0000 7f2e47020640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2e2c0678f0 0x7f2e2c069db0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:32.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.739+0000 7f2e47020640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e40153300 0x7f2e4014aef0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:32.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.739+0000 7f2e47020640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e400c2910 0x7f2e4014c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:32.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.739+0000 7f2e47020640 1 --2- 172.21.15.83:0/2126082958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e4006d1d0 0x7f2e40150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:32.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.739+0000 7f2e47020640 1 -- 172.21.15.83:0/2126082958 >> 172.21.15.83:0/2126082958 conn(0x7f2e400c5090 msgr2=0x7f2e4006c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:32.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.739+0000 7f2e47020640 1 -- 172.21.15.83:0/2126082958 shutdown_connections 2024-09-19T07:37:32.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:32.739+0000 7f2e47020640 1 -- 172.21.15.83:0/2126082958 wait complete. 2024-09-19T07:37:32.751 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-19T07:37:32.751 INFO:tasks.mds_thrash.fs.[b]:waiting for 74 secs before thrashing 2024-09-19T07:37:34.800 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:37:34.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:37:55.331 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:55.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc96c097640 1 Processor -- start 2024-09-19T07:37:55.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc96c097640 1 -- start start 2024-09-19T07:37:55.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc96c097640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc9640c6360 0x7fc964153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:55.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc96c097640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc9640bee30 0x7fc9641570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:55.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc96c097640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9640c5b90 con 0x7fc9640c9680 2024-09-19T07:37:55.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc96c097640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc96415a050 con 0x7fc9640c6360 2024-09-19T07:37:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc96c097640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc96406c950 con 0x7fc9640bee30 2024-09-19T07:37:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc969e33640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc9640c6360 0x7fc964153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc969e33640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc9640c6360 0x7fc964153050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38640/0 (socket says 172.21.15.83:38640) 2024-09-19T07:37:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc969e33640 1 -- 172.21.15.83:0/1426849326 learned_addr learned my addr 172.21.15.83:0/1426849326 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc969632640 1 --2- 172.21.15.83:0/1426849326 >> v2:172.21.15.184:3300/0 conn(0x7fc9640bee30 0x7fc9641570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc969e33640 1 -- 172.21.15.83:0/1426849326 >> v2:172.21.15.184:3300/0 conn(0x7fc9640bee30 msgr2=0x7fc9641570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc969e33640 1 --2- 172.21.15.83:0/1426849326 >> v2:172.21.15.184:3300/0 conn(0x7fc9640bee30 0x7fc9641570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc969e33640 1 -- 172.21.15.83:0/1426849326 >> v1:172.21.15.184:6789/0 conn(0x7fc9640c9680 legacy=0x7fc964157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:55.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.486+0000 7fc969e33640 1 -- 172.21.15.83:0/1426849326 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96406c670 con 0x7fc9640c6360 2024-09-19T07:37:55.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc969e33640 1 --2- 172.21.15.83:0/1426849326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9640c6360 0x7fc964153050 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fc9640c2650 tx=0x7fc96001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b7d32d1a34bf83 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:55.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc968e31640 1 -- 172.21.15.83:0/1426849326 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9600b0070 con 0x7fc9640c6360 2024-09-19T07:37:55.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc968e31640 1 -- 172.21.15.83:0/1426849326 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9600ab600 con 0x7fc9640c6360 2024-09-19T07:37:55.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc968e31640 1 -- 172.21.15.83:0/1426849326 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9600ab9d0 con 0x7fc9640c6360 2024-09-19T07:37:55.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 -- 172.21.15.83:0/1426849326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9640c6360 msgr2=0x7fc964153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:55.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 --2- 172.21.15.83:0/1426849326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9640c6360 0x7fc964153050 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fc9640c2650 tx=0x7fc96001c400 comp rx=0 tx=0).stop 2024-09-19T07:37:55.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 -- 172.21.15.83:0/1426849326 shutdown_connections 2024-09-19T07:37:55.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 --2- 172.21.15.83:0/1426849326 >> v2:172.21.15.184:3300/0 conn(0x7fc9640bee30 0x7fc9641570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 --2- 172.21.15.83:0/1426849326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9640c6360 0x7fc964153050 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 -- 172.21.15.83:0/1426849326 >> 172.21.15.83:0/1426849326 conn(0x7fc9640722a0 msgr2=0x7fc9640726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:55.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 -- 172.21.15.83:0/1426849326 shutdown_connections 2024-09-19T07:37:55.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 -- 172.21.15.83:0/1426849326 wait complete. 2024-09-19T07:37:55.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 Processor -- start 2024-09-19T07:37:55.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 -- start start 2024-09-19T07:37:55.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9640bee30 0x7fc96414c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:55.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9640c9680 0x7fc9641444d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:55.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc96414a280 0x7fc964144a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:55.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc9640c68c0 con 0x7fc96414a280 2024-09-19T07:37:55.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc9640c6740 con 0x7fc9640bee30 2024-09-19T07:37:55.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96c097640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc964162da0 con 0x7fc9640c9680 2024-09-19T07:37:55.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96a634640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc96414a280 0x7fc964144a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:55.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96a634640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc96414a280 0x7fc964144a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38648/0 (socket says 172.21.15.83:38648) 2024-09-19T07:37:55.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96a634640 1 -- 172.21.15.83:0/2019473614 learned_addr learned my addr 172.21.15.83:0/2019473614 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:55.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc969632640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9640c9680 0x7fc9641444d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:55.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc969e33640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9640bee30 0x7fc96414c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:55.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96a634640 1 -- 172.21.15.83:0/2019473614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9640c9680 msgr2=0x7fc9641444d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:55.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96a634640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9640c9680 0x7fc9641444d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96a634640 1 -- 172.21.15.83:0/2019473614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9640bee30 msgr2=0x7fc96414c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:55.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96a634640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9640bee30 0x7fc96414c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96a634640 1 -- 172.21.15.83:0/2019473614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9641f2fd0 con 0x7fc96414a280 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.490+0000 7fc96a634640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc96414a280 0x7fc964144a10 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00c730 tx=0x7fc95c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.494+0000 7fc95a7fc640 1 -- 172.21.15.83:0/2019473614 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c016020 con 0x7fc96414a280 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.494+0000 7fc95a7fc640 1 -- 172.21.15.83:0/2019473614 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc95c006d60 con 0x7fc96414a280 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.494+0000 7fc96c097640 1 -- 172.21.15.83:0/2019473614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc96414d130 con 0x7fc96414a280 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.494+0000 7fc95a7fc640 1 -- 172.21.15.83:0/2019473614 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c005260 con 0x7fc96414a280 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.494+0000 7fc96c097640 1 -- 172.21.15.83:0/2019473614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9641f0440 con 0x7fc96414a280 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.494+0000 7fc96c097640 1 -- 172.21.15.83:0/2019473614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc964153050 con 0x7fc96414a280 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.494+0000 7fc95a7fc640 1 -- 172.21.15.83:0/2019473614 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc95c005400 con 0x7fc96414a280 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.494+0000 7fc95a7fc640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9480678c0 0x7fc948069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.494+0000 7fc95a7fc640 1 -- 172.21.15.83:0/2019473614 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7fc95c0863e0 con 0x7fc96414a280 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.494+0000 7fc969e33640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9480678c0 0x7fc948069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:55.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.494+0000 7fc969e33640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9480678c0 0x7fc948069d80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc9600ad000 tx=0x7fc960008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:55.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.498+0000 7fc95a7fc640 1 -- 172.21.15.83:0/2019473614 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc95c020020 con 0x7fc96414a280 2024-09-19T07:37:55.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 -- 172.21.15.83:0/2019473614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc96415ffe0 con 0x7fc96414a280 2024-09-19T07:37:55.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc95a7fc640 1 -- 172.21.15.83:0/2019473614 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6543 (secure 0 0 0) 0x7fc9641f2fd0 con 0x7fc96414a280 2024-09-19T07:37:55.735 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:55.735 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":135,"btime":"2024-09-19T07:37:30:334185+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:37:55.735 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 135 2024-09-19T07:37:55.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 -- 172.21.15.83:0/2019473614 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9480678c0 msgr2=0x7fc948069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:55.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9480678c0 0x7fc948069d80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc9600ad000 tx=0x7fc960008040 comp rx=0 tx=0).stop 2024-09-19T07:37:55.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 -- 172.21.15.83:0/2019473614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc96414a280 msgr2=0x7fc964144a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:55.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc96414a280 0x7fc964144a10 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00c730 tx=0x7fc95c00cbf0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 -- 172.21.15.83:0/2019473614 shutdown_connections 2024-09-19T07:37:55.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9480678c0 0x7fc948069d80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc96414a280 0x7fc964144a10 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9640c9680 0x7fc9641444d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 --2- 172.21.15.83:0/2019473614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9640bee30 0x7fc96414c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 -- 172.21.15.83:0/2019473614 >> 172.21.15.83:0/2019473614 conn(0x7fc9640722a0 msgr2=0x7fc9640c8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:55.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 -- 172.21.15.83:0/2019473614 shutdown_connections 2024-09-19T07:37:55.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.738+0000 7fc96c097640 1 -- 172.21.15.83:0/2019473614 wait complete. 2024-09-19T07:37:55.754 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-19T07:37:55.754 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:37:55.754 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:37:55.766 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:37:55.766 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:55.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9abd65640 1 Processor -- start 2024-09-19T07:37:55.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9abd65640 1 -- start start 2024-09-19T07:37:55.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9abd65640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9a40c3130 0x7fd9a40ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:55.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9abd65640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd9a406de40 0x7fd9a41563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:55.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9abd65640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9a40c3c20 con 0x7fd9a406da00 2024-09-19T07:37:55.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9abd65640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd9a40c4990 con 0x7fd9a406de40 2024-09-19T07:37:55.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9abd65640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd9a40c4c70 con 0x7fd9a40c3130 2024-09-19T07:37:55.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9a9b01640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9a40c3130 0x7fd9a40ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:55.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9a9300640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fd9a406da00 0x7fd9a40c3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:41954/0 (socket says 172.21.15.83:41954) 2024-09-19T07:37:55.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9a9b01640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9a40c3130 0x7fd9a40ac290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:36600/0 (socket says 172.21.15.83:36600) 2024-09-19T07:37:55.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9a9300640 1 -- 172.21.15.83:0/2031283253 learned_addr learned my addr 172.21.15.83:0/2031283253 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:37:55.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9aa302640 1 --2- 172.21.15.83:0/2031283253 >> v2:172.21.15.184:3300/0 conn(0x7fd9a406de40 0x7fd9a41563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:55.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9a8aff640 1 -- 172.21.15.83:0/2031283253 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 698405413 0 0) 0x7fd9a40c3c20 con 0x7fd9a406da00 2024-09-19T07:37:55.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9a9b01640 1 -- 172.21.15.83:0/2031283253 >> v2:172.21.15.184:3300/0 conn(0x7fd9a406de40 msgr2=0x7fd9a41563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:55.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9a9b01640 1 --2- 172.21.15.83:0/2031283253 >> v2:172.21.15.184:3300/0 conn(0x7fd9a406de40 0x7fd9a41563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9a9b01640 1 -- 172.21.15.83:0/2031283253 >> v1:172.21.15.184:6789/0 conn(0x7fd9a406da00 legacy=0x7fd9a40c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:55.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.934+0000 7fd9a9b01640 1 -- 172.21.15.83:0/2031283253 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a40ab7b0 con 0x7fd9a40c3130 2024-09-19T07:37:55.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9a9b01640 1 --2- 172.21.15.83:0/2031283253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9a40c3130 0x7fd9a40ac290 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00097c0 tx=0x7fd9a001c150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaf5fcbc4b71fee5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:55.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9a8aff640 1 -- 172.21.15.83:0/2031283253 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a00b0070 con 0x7fd9a40c3130 2024-09-19T07:37:55.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9a8aff640 1 -- 172.21.15.83:0/2031283253 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9a00ab600 con 0x7fd9a40c3130 2024-09-19T07:37:55.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9a8aff640 1 -- 172.21.15.83:0/2031283253 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a00ab9d0 con 0x7fd9a40c3130 2024-09-19T07:37:55.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 -- 172.21.15.83:0/2031283253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9a40c3130 msgr2=0x7fd9a40ac290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:55.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 --2- 172.21.15.83:0/2031283253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9a40c3130 0x7fd9a40ac290 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00097c0 tx=0x7fd9a001c150 comp rx=0 tx=0).stop 2024-09-19T07:37:55.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 -- 172.21.15.83:0/2031283253 shutdown_connections 2024-09-19T07:37:55.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 --2- 172.21.15.83:0/2031283253 >> v2:172.21.15.184:3300/0 conn(0x7fd9a406de40 0x7fd9a41563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 --2- 172.21.15.83:0/2031283253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9a40c3130 0x7fd9a40ac290 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 -- 172.21.15.83:0/2031283253 >> 172.21.15.83:0/2031283253 conn(0x7fd9a40c5060 msgr2=0x7fd9a40c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:55.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 -- 172.21.15.83:0/2031283253 shutdown_connections 2024-09-19T07:37:55.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 -- 172.21.15.83:0/2031283253 wait complete. 2024-09-19T07:37:55.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 Processor -- start 2024-09-19T07:37:55.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 -- start start 2024-09-19T07:37:55.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9a406da00 0x7fd9a414c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:55.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9a406de40 0x7fd9a4154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:55.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9a40c3130 0x7fd9a41f1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:55.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.938+0000 7fd9abd65640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a40ad6a0 con 0x7fd9a406da00 2024-09-19T07:37:55.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9abd65640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a40ad520 con 0x7fd9a406de40 2024-09-19T07:37:55.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9abd65640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd9a415f810 con 0x7fd9a40c3130 2024-09-19T07:37:55.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9a9b01640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9a406da00 0x7fd9a414c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9a9b01640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9a406da00 0x7fd9a414c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38660/0 (socket says 172.21.15.83:38660) 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9a9b01640 1 -- 172.21.15.83:0/1113170401 learned_addr learned my addr 172.21.15.83:0/1113170401 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9a9300640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9a406de40 0x7fd9a4154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9aa302640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9a40c3130 0x7fd9a41f1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9a9b01640 1 -- 172.21.15.83:0/1113170401 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9a40c3130 msgr2=0x7fd9a41f1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9a9b01640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9a40c3130 0x7fd9a41f1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9a9b01640 1 -- 172.21.15.83:0/1113170401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9a406de40 msgr2=0x7fd9a4154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9a9b01640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9a406de40 0x7fd9a4154fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9a9b01640 1 -- 172.21.15.83:0/1113170401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a41ee490 con 0x7fd9a406da00 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9a9b01640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9a406da00 0x7fd9a414c300 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00097c0 tx=0x7fd9a0018ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd99a7fc640 1 -- 172.21.15.83:0/1113170401 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a00b0040 con 0x7fd9a406da00 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd99a7fc640 1 -- 172.21.15.83:0/1113170401 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9a0004030 con 0x7fd9a406da00 2024-09-19T07:37:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9abd65640 1 -- 172.21.15.83:0/1113170401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a41ee770 con 0x7fd9a406da00 2024-09-19T07:37:55.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd99a7fc640 1 -- 172.21.15.83:0/1113170401 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0006e20 con 0x7fd9a406da00 2024-09-19T07:37:55.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9abd65640 1 -- 172.21.15.83:0/1113170401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a41eed30 con 0x7fd9a406da00 2024-09-19T07:37:55.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9abd65640 1 -- 172.21.15.83:0/1113170401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a41563a0 con 0x7fd9a406da00 2024-09-19T07:37:55.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd99a7fc640 1 -- 172.21.15.83:0/1113170401 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd9a00b3040 con 0x7fd9a406da00 2024-09-19T07:37:55.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd99a7fc640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd9880678c0 0x7fd988069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:55.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9a9300640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd9880678c0 0x7fd988069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:55.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd99a7fc640 1 -- 172.21.15.83:0/1113170401 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7fd9a0120ac0 con 0x7fd9a406da00 2024-09-19T07:37:55.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.942+0000 7fd9a9300640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd9880678c0 0x7fd988069d80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4075f10 tx=0x7fd994009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:55.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:55.946+0000 7fd99a7fc640 1 -- 172.21.15.83:0/1113170401 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd9a00bb020 con 0x7fd9a406da00 2024-09-19T07:37:56.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.178+0000 7fd9abd65640 1 -- 172.21.15.83:0/1113170401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9a40ab540 con 0x7fd9a406da00 2024-09-19T07:37:56.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.178+0000 7fd99a7fc640 1 -- 172.21.15.83:0/1113170401 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6543 (secure 0 0 0) 0x7fd9a0016090 con 0x7fd9a406da00 2024-09-19T07:37:56.174 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:56.174 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":135,"btime":"2024-09-19T07:37:30:334185+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:37:56.174 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 135 2024-09-19T07:37:56.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.182+0000 7fd9abd65640 1 -- 172.21.15.83:0/1113170401 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd9880678c0 msgr2=0x7fd988069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:56.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.182+0000 7fd9abd65640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd9880678c0 0x7fd988069d80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4075f10 tx=0x7fd994009040 comp rx=0 tx=0).stop 2024-09-19T07:37:56.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.182+0000 7fd9abd65640 1 -- 172.21.15.83:0/1113170401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9a406da00 msgr2=0x7fd9a414c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:56.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.182+0000 7fd9abd65640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9a406da00 0x7fd9a414c300 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00097c0 tx=0x7fd9a0018ef0 comp rx=0 tx=0).stop 2024-09-19T07:37:56.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.182+0000 7fd9abd65640 1 -- 172.21.15.83:0/1113170401 shutdown_connections 2024-09-19T07:37:56.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.182+0000 7fd9abd65640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd9880678c0 0x7fd988069d80 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:56.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.182+0000 7fd9abd65640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9a40c3130 0x7fd9a41f1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:56.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.182+0000 7fd9abd65640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9a406de40 0x7fd9a4154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:56.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.182+0000 7fd9abd65640 1 --2- 172.21.15.83:0/1113170401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9a406da00 0x7fd9a414c300 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:56.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.182+0000 7fd9abd65640 1 -- 172.21.15.83:0/1113170401 >> 172.21.15.83:0/1113170401 conn(0x7fd9a40c5060 msgr2=0x7fd9a40c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:56.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.182+0000 7fd9abd65640 1 -- 172.21.15.83:0/1113170401 shutdown_connections 2024-09-19T07:37:56.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:56.182+0000 7fd9abd65640 1 -- 172.21.15.83:0/1113170401 wait complete. 2024-09-19T07:37:56.193 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:37:58.194 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:37:58.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f6687b10640 1 Processor -- start 2024-09-19T07:37:58.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f6687b10640 1 -- start start 2024-09-19T07:37:58.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f6687b10640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6680071100 0x7f66801545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:58.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f6687b10640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f668006dea0 0x7f668006e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:58.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f6687b10640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66800a9e60 con 0x7f66800c28e0 2024-09-19T07:37:58.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f6687b10640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6680071b50 con 0x7f6680071100 2024-09-19T07:37:58.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f6687b10640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f66800c1ff0 con 0x7f668006dea0 2024-09-19T07:37:58.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66850ab640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f66800c28e0 0x7f6680150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34216/0 (socket says 172.21.15.83:34216) 2024-09-19T07:37:58.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66850ab640 1 -- 172.21.15.83:0/3231718636 learned_addr learned my addr 172.21.15.83:0/3231718636 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66860ad640 1 --2- 172.21.15.83:0/3231718636 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f668006dea0 0x7f668006e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66858ac640 1 --2- 172.21.15.83:0/3231718636 >> v2:172.21.15.184:3300/0 conn(0x7f6680071100 0x7f66801545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3759672654 0 0) 0x7f66800a9e60 con 0x7f66800c28e0 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f666c003530 con 0x7f66800c28e0 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3819004931 0 0) 0x7f666c003530 con 0x7f66800c28e0 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f668006dea0 msgr2=0x7f668006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 --2- 172.21.15.83:0/3231718636 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f668006dea0 0x7f668006e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 >> v2:172.21.15.184:3300/0 conn(0x7f6680071100 msgr2=0x7f66801545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 --2- 172.21.15.83:0/3231718636 >> v2:172.21.15.184:3300/0 conn(0x7f6680071100 0x7f66801545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66800aa140 con 0x7f66800c28e0 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f6670001680 con 0x7f66800c28e0 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 >> v1:172.21.15.83:6789/0 conn(0x7f66800c28e0 legacy=0x7f6680150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 --2- 172.21.15.83:0/3231718636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f666c0039c0 0x7f666c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 --2- 172.21.15.83:0/3231718636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f666c004810 0x7f666c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 --2- 172.21.15.83:0/3231718636 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f668006dea0 0x7f666c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6680071b50 con 0x7f666c004810 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f66800c1ff0 con 0x7f666c0039c0 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f666c00ba00 con 0x7f668006dea0 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66850ab640 1 --2- 172.21.15.83:0/3231718636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f666c004810 0x7f666c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66858ac640 1 --2- 172.21.15.83:0/3231718636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f666c0039c0 0x7f666c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66860ad640 1 --2- 172.21.15.83:0/3231718636 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f668006dea0 0x7f666c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66850ab640 1 -- 172.21.15.83:0/3231718636 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f668006dea0 msgr2=0x7f666c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66850ab640 1 --2- 172.21.15.83:0/3231718636 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f668006dea0 0x7f666c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66850ab640 1 -- 172.21.15.83:0/3231718636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f666c0039c0 msgr2=0x7f666c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66850ab640 1 --2- 172.21.15.83:0/3231718636 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f666c0039c0 0x7f666c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:58.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.326+0000 7f66850ab640 1 -- 172.21.15.83:0/3231718636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f666c003530 con 0x7f666c004810 2024-09-19T07:37:58.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f66850ab640 1 --2- 172.21.15.83:0/3231718636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f666c004810 0x7f666c004c90 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f667000c240 tx=0x7f667001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c67bee14cced75 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670007d40 con 0x7f666c004810 2024-09-19T07:37:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66700a9020 con 0x7f666c004810 2024-09-19T07:37:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f66848aa640 1 -- 172.21.15.83:0/3231718636 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670005b00 con 0x7f666c004810 2024-09-19T07:37:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 -- 172.21.15.83:0/3231718636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f666c004810 msgr2=0x7f666c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 --2- 172.21.15.83:0/3231718636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f666c004810 0x7f666c004c90 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f667000c240 tx=0x7f667001cc40 comp rx=0 tx=0).stop 2024-09-19T07:37:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f66860ad640 1 -- 172.21.15.83:0/3231718636 reap_dead start 2024-09-19T07:37:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 -- 172.21.15.83:0/3231718636 shutdown_connections 2024-09-19T07:37:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 -- 172.21.15.83:0/3231718636 >> 172.21.15.83:0/3231718636 conn(0x7f668006bc80 msgr2=0x7f668006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 -- 172.21.15.83:0/3231718636 shutdown_connections 2024-09-19T07:37:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 -- 172.21.15.83:0/3231718636 wait complete. 2024-09-19T07:37:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 Processor -- start 2024-09-19T07:37:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 -- start start 2024-09-19T07:37:58.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f666c004810 0x7f66800c0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:58.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6680153c40 0x7f6680158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:58.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6680158f90 0x7f6680159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:58.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6680071660 con 0x7f6680158f90 2024-09-19T07:37:58.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f66800714e0 con 0x7f6680153c40 2024-09-19T07:37:58.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f6687b10640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f668015e440 con 0x7f666c004810 2024-09-19T07:37:58.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f66860ad640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6680158f90 0x7f6680159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:58.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f66860ad640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6680158f90 0x7f6680159890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38684/0 (socket says 172.21.15.83:38684) 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f66860ad640 1 -- 172.21.15.83:0/3573573288 learned_addr learned my addr 172.21.15.83:0/3573573288 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f66850ab640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6680153c40 0x7f6680158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.330+0000 7f66858ac640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f666c004810 0x7f66800c0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66860ad640 1 -- 172.21.15.83:0/3573573288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f666c004810 msgr2=0x7f66800c0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66860ad640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f666c004810 0x7f66800c0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66860ad640 1 -- 172.21.15.83:0/3573573288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6680153c40 msgr2=0x7f6680158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66860ad640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6680153c40 0x7f6680158a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66860ad640 1 -- 172.21.15.83:0/3573573288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66801ef320 con 0x7f6680158f90 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66858ac640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f666c004810 0x7f66800c0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66860ad640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6680158f90 0x7f6680159890 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6678007d10 tx=0x7f66780115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66767fc640 1 -- 172.21.15.83:0/3573573288 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678011e40 con 0x7f6680158f90 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66767fc640 1 -- 172.21.15.83:0/3573573288 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f667801b070 con 0x7f6680158f90 2024-09-19T07:37:58.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f6687b10640 1 -- 172.21.15.83:0/3573573288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66801ef5c0 con 0x7f6680158f90 2024-09-19T07:37:58.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66767fc640 1 -- 172.21.15.83:0/3573573288 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66780039a0 con 0x7f6680158f90 2024-09-19T07:37:58.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f6687b10640 1 -- 172.21.15.83:0/3573573288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66801efb80 con 0x7f6680158f90 2024-09-19T07:37:58.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f6687b10640 1 -- 172.21.15.83:0/3573573288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66801545e0 con 0x7f6680158f90 2024-09-19T07:37:58.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66767fc640 1 -- 172.21.15.83:0/3573573288 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6678003b40 con 0x7f6680158f90 2024-09-19T07:37:58.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66767fc640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66640678c0 0x7f6664069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:37:58.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66767fc640 1 -- 172.21.15.83:0/3573573288 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7f6678087070 con 0x7f6680158f90 2024-09-19T07:37:58.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66858ac640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66640678c0 0x7f6664069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:37:58.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.334+0000 7f66858ac640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66640678c0 0x7f6664069d80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f667c009af0 tx=0x7f667c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:37:58.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.338+0000 7f66767fc640 1 -- 172.21.15.83:0/3573573288 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6678056f00 con 0x7f6680158f90 2024-09-19T07:37:58.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 -- 172.21.15.83:0/3573573288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66800c4f30 con 0x7f6680158f90 2024-09-19T07:37:58.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f66767fc640 1 -- 172.21.15.83:0/3573573288 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6543 (secure 0 0 0) 0x7f6678056800 con 0x7f6680158f90 2024-09-19T07:37:58.533 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:37:58.533 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":135,"btime":"2024-09-19T07:37:30:334185+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:37:58.533 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 135 2024-09-19T07:37:58.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 -- 172.21.15.83:0/3573573288 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66640678c0 msgr2=0x7f6664069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:58.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66640678c0 0x7f6664069d80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f667c009af0 tx=0x7f667c008040 comp rx=0 tx=0).stop 2024-09-19T07:37:58.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 -- 172.21.15.83:0/3573573288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6680158f90 msgr2=0x7f6680159890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:37:58.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6680158f90 0x7f6680159890 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6678007d10 tx=0x7f66780115a0 comp rx=0 tx=0).stop 2024-09-19T07:37:58.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 -- 172.21.15.83:0/3573573288 shutdown_connections 2024-09-19T07:37:58.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66640678c0 0x7f6664069d80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:58.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6680158f90 0x7f6680159890 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:58.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6680153c40 0x7f6680158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:58.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 --2- 172.21.15.83:0/3573573288 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f666c004810 0x7f66800c0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:37:58.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 -- 172.21.15.83:0/3573573288 >> 172.21.15.83:0/3573573288 conn(0x7f668006bc80 msgr2=0x7f6680150390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:37:58.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 -- 172.21.15.83:0/3573573288 shutdown_connections 2024-09-19T07:37:58.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:37:58.538+0000 7f6687b10640 1 -- 172.21.15.83:0/3573573288 wait complete. 2024-09-19T07:37:58.553 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:38:00.554 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:00.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f125e189640 1 Processor -- start 2024-09-19T07:38:00.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f125e189640 1 -- start start 2024-09-19T07:38:00.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f125e189640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12580c42e0 0x7f125806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:00.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f125e189640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12581592c0 con 0x7f12580abb30 2024-09-19T07:38:00.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f125e189640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f125806e900 con 0x7f12581566b0 2024-09-19T07:38:00.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f125e189640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f12581595f0 con 0x7f12580c42e0 2024-09-19T07:38:00.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f1257fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f12580abb30 0x7f1258153060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57892/0 (socket says 172.21.15.83:57892) 2024-09-19T07:38:00.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f1257fff640 1 -- 172.21.15.83:0/3728730689 learned_addr learned my addr 172.21.15.83:0/3728730689 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:38:00.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1311058530 0 0) 0x7f12581592c0 con 0x7f12580abb30 2024-09-19T07:38:00.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1240003530 con 0x7f12580abb30 2024-09-19T07:38:00.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f12577fe640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12580c42e0 0x7f125806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:00.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1733410513 0 0) 0x7f1240003530 con 0x7f12580abb30 2024-09-19T07:38:00.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12580c42e0 msgr2=0x7f125806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.690+0000 7f12567fc640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12580c42e0 0x7f125806e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 >> v1:172.21.15.184:6789/0 conn(0x7f12581566b0 legacy=0x7f1258159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125806ebe0 con 0x7f12580abb30 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f1250001a00 con 0x7f12580abb30 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 >> v1:172.21.15.83:6789/0 conn(0x7f12580abb30 legacy=0x7f1258153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12580c42e0 0x7f1240003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1240004040 0x7f1240004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1240004d50 0x7f12400051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f12581595f0 con 0x7f1240004040 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f124000bb90 con 0x7f1240004d50 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f124000bd90 con 0x7f12580c42e0 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f1256ffd640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1240004040 0x7f1240004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f1257fff640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1240004d50 0x7f12400051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:00.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12577fe640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12580c42e0 0x7f1240003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:00.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f1256ffd640 1 -- 172.21.15.83:0/3728730689 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12580c42e0 msgr2=0x7f1240003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:00.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f1256ffd640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12580c42e0 0x7f1240003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:00.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f1256ffd640 1 -- 172.21.15.83:0/3728730689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1240004d50 msgr2=0x7f12400051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:00.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f1256ffd640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1240004d50 0x7f12400051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:00.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f1256ffd640 1 -- 172.21.15.83:0/3728730689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1240003530 con 0x7f1240004040 2024-09-19T07:38:00.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12577fe640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12580c42e0 0x7f1240003a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:38:00.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f1256ffd640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1240004040 0x7f1240004810 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f1248001750 tx=0x7f124801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef55afd207c7799 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:00.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12480013a0 con 0x7f1240004040 2024-09-19T07:38:00.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12480052f0 con 0x7f1240004040 2024-09-19T07:38:00.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f12567fc640 1 -- 172.21.15.83:0/3728730689 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248003840 con 0x7f1240004040 2024-09-19T07:38:00.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f125e189640 1 -- 172.21.15.83:0/3728730689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1240004040 msgr2=0x7f1240004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:00.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f125e189640 1 --2- 172.21.15.83:0/3728730689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1240004040 0x7f1240004810 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f1248001750 tx=0x7f124801c3e0 comp rx=0 tx=0).stop 2024-09-19T07:38:00.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f1257fff640 1 -- 172.21.15.83:0/3728730689 reap_dead start 2024-09-19T07:38:00.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f125e189640 1 -- 172.21.15.83:0/3728730689 shutdown_connections 2024-09-19T07:38:00.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f125e189640 1 -- 172.21.15.83:0/3728730689 >> 172.21.15.83:0/3728730689 conn(0x7f125806bc80 msgr2=0x7f125806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:00.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f125e189640 1 -- 172.21.15.83:0/3728730689 shutdown_connections 2024-09-19T07:38:00.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.694+0000 7f125e189640 1 -- 172.21.15.83:0/3728730689 wait complete. 2024-09-19T07:38:00.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f125e189640 1 Processor -- start 2024-09-19T07:38:00.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f125e189640 1 -- start start 2024-09-19T07:38:00.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f125e189640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1240004040 0x7f12581df4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:00.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f125e189640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f12580a9d30 0x7f12580c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:00.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f125e189640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12580c6f80 0x7f125814bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:00.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f125e189640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1258156af0 con 0x7f12580c6f80 2024-09-19T07:38:00.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f125e189640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f125806eec0 con 0x7f12580a9d30 2024-09-19T07:38:00.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f125e189640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1258153770 con 0x7f1240004040 2024-09-19T07:38:00.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f1257fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12580c6f80 0x7f125814bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:00.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f1257fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12580c6f80 0x7f125814bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57554/0 (socket says 172.21.15.83:57554) 2024-09-19T07:38:00.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f1257fff640 1 -- 172.21.15.83:0/2805200781 learned_addr learned my addr 172.21.15.83:0/2805200781 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:00.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f12577fe640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1240004040 0x7f12581df4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:00.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f1256ffd640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f12580a9d30 0x7f12580c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:00.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f1257fff640 1 -- 172.21.15.83:0/2805200781 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1240004040 msgr2=0x7f12581df4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:00.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f1257fff640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1240004040 0x7f12581df4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:00.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f1257fff640 1 -- 172.21.15.83:0/2805200781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f12580a9d30 msgr2=0x7f12580c6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:00.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f1257fff640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f12580a9d30 0x7f12580c6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:00.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f1257fff640 1 -- 172.21.15.83:0/2805200781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125814c9b0 con 0x7f12580c6f80 2024-09-19T07:38:00.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f1257fff640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12580c6f80 0x7f125814bca0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f12580c1b90 tx=0x7f1250011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:00.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f123ffff640 1 -- 172.21.15.83:0/2805200781 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250010040 con 0x7f12580c6f80 2024-09-19T07:38:00.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f123ffff640 1 -- 172.21.15.83:0/2805200781 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f125001a070 con 0x7f12580c6f80 2024-09-19T07:38:00.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f125e189640 1 -- 172.21.15.83:0/2805200781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12581516a0 con 0x7f12580c6f80 2024-09-19T07:38:00.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f123ffff640 1 -- 172.21.15.83:0/2805200781 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125000fa40 con 0x7f12580c6f80 2024-09-19T07:38:00.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f125e189640 1 -- 172.21.15.83:0/2805200781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12580c7410 con 0x7f12580c6f80 2024-09-19T07:38:00.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f125e189640 1 -- 172.21.15.83:0/2805200781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12580c46c0 con 0x7f12580c6f80 2024-09-19T07:38:00.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.698+0000 7f123ffff640 1 -- 172.21.15.83:0/2805200781 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f125000fbe0 con 0x7f12580c6f80 2024-09-19T07:38:00.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.702+0000 7f123ffff640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f12440678c0 0x7f1244069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:00.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.702+0000 7f123ffff640 1 -- 172.21.15.83:0/2805200781 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7f12500897d0 con 0x7f12580c6f80 2024-09-19T07:38:00.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.702+0000 7f12577fe640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f12440678c0 0x7f1244069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:00.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.702+0000 7f12577fe640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f12440678c0 0x7f1244069d80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1258152ce0 tx=0x7f124c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:00.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.706+0000 7f123ffff640 1 -- 172.21.15.83:0/2805200781 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1250059670 con 0x7f12580c6f80 2024-09-19T07:38:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.902+0000 7f125e189640 1 -- 172.21.15.83:0/2805200781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f125806e050 con 0x7f12580c6f80 2024-09-19T07:38:00.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f123ffff640 1 -- 172.21.15.83:0/2805200781 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6543 (secure 0 0 0) 0x7f1250058f70 con 0x7f12580c6f80 2024-09-19T07:38:00.900 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:00.900 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":135,"btime":"2024-09-19T07:37:30:334185+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:00.901 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 135 2024-09-19T07:38:00.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f125e189640 1 -- 172.21.15.83:0/2805200781 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f12440678c0 msgr2=0x7f1244069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:00.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f125e189640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f12440678c0 0x7f1244069d80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1258152ce0 tx=0x7f124c008040 comp rx=0 tx=0).stop 2024-09-19T07:38:00.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f125e189640 1 -- 172.21.15.83:0/2805200781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12580c6f80 msgr2=0x7f125814bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:00.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f125e189640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12580c6f80 0x7f125814bca0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f12580c1b90 tx=0x7f1250011a70 comp rx=0 tx=0).stop 2024-09-19T07:38:00.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f125e189640 1 -- 172.21.15.83:0/2805200781 shutdown_connections 2024-09-19T07:38:00.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f125e189640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f12440678c0 0x7f1244069d80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:00.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f125e189640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12580c6f80 0x7f125814bca0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:00.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f125e189640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f12580a9d30 0x7f12580c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:00.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f125e189640 1 --2- 172.21.15.83:0/2805200781 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1240004040 0x7f12581df4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:00.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f125e189640 1 -- 172.21.15.83:0/2805200781 >> 172.21.15.83:0/2805200781 conn(0x7f125806bc80 msgr2=0x7f1258153040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:00.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f125e189640 1 -- 172.21.15.83:0/2805200781 shutdown_connections 2024-09-19T07:38:00.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:00.906+0000 7f125e189640 1 -- 172.21.15.83:0/2805200781 wait complete. 2024-09-19T07:38:00.920 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:38:02.921 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:03.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.054+0000 7fa7453de640 1 Processor -- start 2024-09-19T07:38:03.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.054+0000 7fa7453de640 1 -- start start 2024-09-19T07:38:03.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa7453de640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa74014ca60 0x7fa740159490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:03.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa7453de640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fa7400cbb00 0x7fa7400cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:03.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa7453de640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7401599d0 con 0x7fa7400bef00 2024-09-19T07:38:03.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa7453de640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa7400ca240 con 0x7fa74014ca60 2024-09-19T07:38:03.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa7453de640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fa7400c9c80 con 0x7fa7400cbb00 2024-09-19T07:38:03.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa73e7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa7400bef00 0x7fa740156840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57906/0 (socket says 172.21.15.83:57906) 2024-09-19T07:38:03.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa73effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa74014ca60 0x7fa740159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:03.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa73e7fc640 1 -- 172.21.15.83:0/1373424013 learned_addr learned my addr 172.21.15.83:0/1373424013 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:38:03.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa73f7fe640 1 --2- 172.21.15.83:0/1373424013 >> v2:172.21.15.184:3300/0 conn(0x7fa7400cbb00 0x7fa7400cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:03.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa73dffb640 1 -- 172.21.15.83:0/1373424013 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2190193838 0 0) 0x7fa7401599d0 con 0x7fa7400bef00 2024-09-19T07:38:03.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa73dffb640 1 -- 172.21.15.83:0/1373424013 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa72c003530 con 0x7fa7400bef00 2024-09-19T07:38:03.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa73effd640 1 -- 172.21.15.83:0/1373424013 >> v2:172.21.15.184:3300/0 conn(0x7fa7400cbb00 msgr2=0x7fa7400cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:03.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa73effd640 1 --2- 172.21.15.83:0/1373424013 >> v2:172.21.15.184:3300/0 conn(0x7fa7400cbb00 0x7fa7400cbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:03.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa73effd640 1 -- 172.21.15.83:0/1373424013 >> v1:172.21.15.83:6789/0 conn(0x7fa7400bef00 legacy=0x7fa740156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:03.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.058+0000 7fa73effd640 1 -- 172.21.15.83:0/1373424013 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7400c9f60 con 0x7fa74014ca60 2024-09-19T07:38:03.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73effd640 1 --2- 172.21.15.83:0/1373424013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa74014ca60 0x7fa740159490 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fa7400cb6b0 tx=0x7fa73001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3cdbedb825e0a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:03.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73dffb640 1 -- 172.21.15.83:0/1373424013 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7300b0070 con 0x7fa74014ca60 2024-09-19T07:38:03.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73dffb640 1 -- 172.21.15.83:0/1373424013 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7300ab600 con 0x7fa74014ca60 2024-09-19T07:38:03.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73dffb640 1 -- 172.21.15.83:0/1373424013 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7300ab9d0 con 0x7fa74014ca60 2024-09-19T07:38:03.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 -- 172.21.15.83:0/1373424013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa74014ca60 msgr2=0x7fa740159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:03.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 --2- 172.21.15.83:0/1373424013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa74014ca60 0x7fa740159490 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fa7400cb6b0 tx=0x7fa73001c480 comp rx=0 tx=0).stop 2024-09-19T07:38:03.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 -- 172.21.15.83:0/1373424013 shutdown_connections 2024-09-19T07:38:03.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 --2- 172.21.15.83:0/1373424013 >> v2:172.21.15.184:3300/0 conn(0x7fa7400cbb00 0x7fa7400cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:03.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 --2- 172.21.15.83:0/1373424013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa74014ca60 0x7fa740159490 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:03.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 -- 172.21.15.83:0/1373424013 >> 172.21.15.83:0/1373424013 conn(0x7fa740072470 msgr2=0x7fa740072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:03.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 -- 172.21.15.83:0/1373424013 shutdown_connections 2024-09-19T07:38:03.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 -- 172.21.15.83:0/1373424013 wait complete. 2024-09-19T07:38:03.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 Processor -- start 2024-09-19T07:38:03.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 -- start start 2024-09-19T07:38:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7400bef00 0x7fa7400b07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa7400cbb00 0x7fa7401504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa74014ca60 0x7fa740150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa7400c98c0 con 0x7fa7400bef00 2024-09-19T07:38:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa740153a50 con 0x7fa74014ca60 2024-09-19T07:38:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa7400c9a40 con 0x7fa7400cbb00 2024-09-19T07:38:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7400bef00 0x7fa7400b07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7400bef00 0x7fa7400b07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57570/0 (socket says 172.21.15.83:57570) 2024-09-19T07:38:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73effd640 1 -- 172.21.15.83:0/1114932001 learned_addr learned my addr 172.21.15.83:0/1114932001 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73e7fc640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa7400cbb00 0x7fa7401504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73f7fe640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa74014ca60 0x7fa740150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73effd640 1 -- 172.21.15.83:0/1114932001 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa7400cbb00 msgr2=0x7fa7401504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73effd640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa7400cbb00 0x7fa7401504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73effd640 1 -- 172.21.15.83:0/1114932001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa74014ca60 msgr2=0x7fa740150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73effd640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa74014ca60 0x7fa740150a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73effd640 1 -- 172.21.15.83:0/1114932001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa740153180 con 0x7fa7400bef00 2024-09-19T07:38:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa73effd640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7400bef00 0x7fa7400b07a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fa7300ae040 tx=0x7fa730019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7377fe640 1 -- 172.21.15.83:0/1114932001 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7300b0040 con 0x7fa7400bef00 2024-09-19T07:38:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7377fe640 1 -- 172.21.15.83:0/1114932001 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7300b9070 con 0x7fa7400bef00 2024-09-19T07:38:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.062+0000 7fa7453de640 1 -- 172.21.15.83:0/1114932001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa740153460 con 0x7fa7400bef00 2024-09-19T07:38:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.066+0000 7fa7377fe640 1 -- 172.21.15.83:0/1114932001 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa730006e20 con 0x7fa7400bef00 2024-09-19T07:38:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.066+0000 7fa7453de640 1 -- 172.21.15.83:0/1114932001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7400b11e0 con 0x7fa7400bef00 2024-09-19T07:38:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.066+0000 7fa7453de640 1 -- 172.21.15.83:0/1114932001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7400cbf20 con 0x7fa7400bef00 2024-09-19T07:38:03.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.066+0000 7fa7377fe640 1 -- 172.21.15.83:0/1114932001 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa7300b4650 con 0x7fa7400bef00 2024-09-19T07:38:03.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.066+0000 7fa7377fe640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa7200678c0 0x7fa720069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:03.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.066+0000 7fa7377fe640 1 -- 172.21.15.83:0/1114932001 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7fa730120590 con 0x7fa7400bef00 2024-09-19T07:38:03.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.066+0000 7fa73e7fc640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa7200678c0 0x7fa720069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:03.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.066+0000 7fa73e7fc640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa7200678c0 0x7fa720069d80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa7400ab8a0 tx=0x7fa728009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:03.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.070+0000 7fa7377fe640 1 -- 172.21.15.83:0/1114932001 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa730016070 con 0x7fa7400bef00 2024-09-19T07:38:03.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.258+0000 7fa7453de640 1 -- 172.21.15.83:0/1114932001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa74006ca90 con 0x7fa7400bef00 2024-09-19T07:38:03.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.258+0000 7fa7377fe640 1 -- 172.21.15.83:0/1114932001 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6543 (secure 0 0 0) 0x7fa7300f0420 con 0x7fa7400bef00 2024-09-19T07:38:03.254 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:03.254 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":135,"btime":"2024-09-19T07:37:30:334185+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:03.255 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 135 2024-09-19T07:38:03.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.262+0000 7fa7453de640 1 -- 172.21.15.83:0/1114932001 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa7200678c0 msgr2=0x7fa720069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:03.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.262+0000 7fa7453de640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa7200678c0 0x7fa720069d80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa7400ab8a0 tx=0x7fa728009040 comp rx=0 tx=0).stop 2024-09-19T07:38:03.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.262+0000 7fa7453de640 1 -- 172.21.15.83:0/1114932001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7400bef00 msgr2=0x7fa7400b07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:03.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.262+0000 7fa7453de640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7400bef00 0x7fa7400b07a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fa7300ae040 tx=0x7fa730019220 comp rx=0 tx=0).stop 2024-09-19T07:38:03.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.262+0000 7fa7453de640 1 -- 172.21.15.83:0/1114932001 shutdown_connections 2024-09-19T07:38:03.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.262+0000 7fa7453de640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa7200678c0 0x7fa720069d80 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:03.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.262+0000 7fa7453de640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa74014ca60 0x7fa740150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:03.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.262+0000 7fa7453de640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa7400cbb00 0x7fa7401504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:03.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.262+0000 7fa7453de640 1 --2- 172.21.15.83:0/1114932001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7400bef00 0x7fa7400b07a0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:03.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.262+0000 7fa7453de640 1 -- 172.21.15.83:0/1114932001 >> 172.21.15.83:0/1114932001 conn(0x7fa740072470 msgr2=0x7fa740155910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:03.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.262+0000 7fa7453de640 1 -- 172.21.15.83:0/1114932001 shutdown_connections 2024-09-19T07:38:03.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:03.262+0000 7fa7453de640 1 -- 172.21.15.83:0/1114932001 wait complete. 2024-09-19T07:38:03.272 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:38:04.905 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:38:04.909 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:38:05.274 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:05.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff65e45c640 1 Processor -- start 2024-09-19T07:38:05.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff65e45c640 1 -- start start 2024-09-19T07:38:05.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff65e45c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff658150290 0x7ff6580c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:05.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff65e45c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff658152b20 0x7ff6580c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:05.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff65e45c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff658150670 con 0x7ff6580c45c0 2024-09-19T07:38:05.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff65e45c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff6580716c0 con 0x7ff658152b20 2024-09-19T07:38:05.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff65e45c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff6580c3920 con 0x7ff658150290 2024-09-19T07:38:05.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff6577fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff658152b20 0x7ff6580c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:05.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff6577fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff658152b20 0x7ff6580c4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57586/0 (socket says 172.21.15.83:57586) 2024-09-19T07:38:05.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff65c9f9640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff6580c45c0 0x7ff658153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57910/0 (socket says 172.21.15.83:57910) 2024-09-19T07:38:05.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff6577fe640 1 -- 172.21.15.83:0/3996689628 learned_addr learned my addr 172.21.15.83:0/3996689628 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:05.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff657fff640 1 --2- 172.21.15.83:0/3996689628 >> v2:172.21.15.184:3300/0 conn(0x7ff658150290 0x7ff6580c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:05.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff6577fe640 1 -- 172.21.15.83:0/3996689628 >> v2:172.21.15.184:3300/0 conn(0x7ff658150290 msgr2=0x7ff6580c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:05.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff6577fe640 1 --2- 172.21.15.83:0/3996689628 >> v2:172.21.15.184:3300/0 conn(0x7ff658150290 0x7ff6580c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:05.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff656ffd640 1 -- 172.21.15.83:0/3996689628 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1494938395 0 0) 0x7ff658150670 con 0x7ff6580c45c0 2024-09-19T07:38:05.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff6577fe640 1 -- 172.21.15.83:0/3996689628 >> v1:172.21.15.83:6789/0 conn(0x7ff6580c45c0 legacy=0x7ff658153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:05.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff6577fe640 1 -- 172.21.15.83:0/3996689628 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6580c2880 con 0x7ff658152b20 2024-09-19T07:38:05.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff657fff640 1 --2- 172.21.15.83:0/3996689628 >> v2:172.21.15.184:3300/0 conn(0x7ff658150290 0x7ff6580c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:38:05.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.410+0000 7ff6577fe640 1 --2- 172.21.15.83:0/3996689628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff658152b20 0x7ff6580c4080 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7ff65806f590 tx=0x7ff64001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1078e5e5a0973763 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:05.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff656ffd640 1 -- 172.21.15.83:0/3996689628 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6400ab430 con 0x7ff658152b20 2024-09-19T07:38:05.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff656ffd640 1 -- 172.21.15.83:0/3996689628 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6400b0070 con 0x7ff658152b20 2024-09-19T07:38:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff656ffd640 1 -- 172.21.15.83:0/3996689628 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6400ab7f0 con 0x7ff658152b20 2024-09-19T07:38:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 -- 172.21.15.83:0/3996689628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff658152b20 msgr2=0x7ff6580c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 --2- 172.21.15.83:0/3996689628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff658152b20 0x7ff6580c4080 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7ff65806f590 tx=0x7ff64001c520 comp rx=0 tx=0).stop 2024-09-19T07:38:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 -- 172.21.15.83:0/3996689628 shutdown_connections 2024-09-19T07:38:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 --2- 172.21.15.83:0/3996689628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff658152b20 0x7ff6580c4080 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 --2- 172.21.15.83:0/3996689628 >> v2:172.21.15.184:3300/0 conn(0x7ff658150290 0x7ff6580c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 -- 172.21.15.83:0/3996689628 >> 172.21.15.83:0/3996689628 conn(0x7ff65806bc80 msgr2=0x7ff65806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 -- 172.21.15.83:0/3996689628 shutdown_connections 2024-09-19T07:38:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 -- 172.21.15.83:0/3996689628 wait complete. 2024-09-19T07:38:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 Processor -- start 2024-09-19T07:38:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 -- start start 2024-09-19T07:38:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6580c45c0 0x7ff658073240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff658150290 0x7ff65814e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff658152b20 0x7ff65814eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff6580c2530 con 0x7ff6580c45c0 2024-09-19T07:38:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff6580c3120 con 0x7ff658152b20 2024-09-19T07:38:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff65815db50 con 0x7ff658150290 2024-09-19T07:38:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff657fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6580c45c0 0x7ff658073240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff657fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6580c45c0 0x7ff658073240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57588/0 (socket says 172.21.15.83:57588) 2024-09-19T07:38:05.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff657fff640 1 -- 172.21.15.83:0/2123436855 learned_addr learned my addr 172.21.15.83:0/2123436855 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:05.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff657fff640 1 -- 172.21.15.83:0/2123436855 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff658150290 msgr2=0x7ff65814e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:05.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff657fff640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff658150290 0x7ff65814e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:05.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff657fff640 1 -- 172.21.15.83:0/2123436855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff658152b20 msgr2=0x7ff65814eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:05.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65c9f9640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff658152b20 0x7ff65814eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:05.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff657fff640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff658152b20 0x7ff65814eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:05.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff657fff640 1 -- 172.21.15.83:0/2123436855 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6580aca40 con 0x7ff6580c45c0 2024-09-19T07:38:05.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff657fff640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6580c45c0 0x7ff658073240 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7ff65000c7c0 tx=0x7ff65000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:05.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff654ff9640 1 -- 172.21.15.83:0/2123436855 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650007a30 con 0x7ff6580c45c0 2024-09-19T07:38:05.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff654ff9640 1 -- 172.21.15.83:0/2123436855 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff650007bd0 con 0x7ff6580c45c0 2024-09-19T07:38:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 -- 172.21.15.83:0/2123436855 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6580ac760 con 0x7ff6580c45c0 2024-09-19T07:38:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff654ff9640 1 -- 172.21.15.83:0/2123436855 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6500113f0 con 0x7ff6580c45c0 2024-09-19T07:38:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.414+0000 7ff65e45c640 1 -- 172.21.15.83:0/2123436855 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff658151b70 con 0x7ff6580c45c0 2024-09-19T07:38:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.418+0000 7ff65e45c640 1 -- 172.21.15.83:0/2123436855 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff65815df10 con 0x7ff6580c45c0 2024-09-19T07:38:05.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.418+0000 7ff654ff9640 1 -- 172.21.15.83:0/2123436855 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff650011590 con 0x7ff6580c45c0 2024-09-19T07:38:05.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.418+0000 7ff654ff9640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6380678c0 0x7ff638069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:05.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.418+0000 7ff654ff9640 1 -- 172.21.15.83:0/2123436855 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7ff650086a70 con 0x7ff6580c45c0 2024-09-19T07:38:05.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.418+0000 7ff6577fe640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6380678c0 0x7ff638069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:05.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.418+0000 7ff6577fe640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6380678c0 0x7ff638069d80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff6400af000 tx=0x7ff640004740 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:05.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.422+0000 7ff654ff9640 1 -- 172.21.15.83:0/2123436855 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff650015020 con 0x7ff6580c45c0 2024-09-19T07:38:05.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.618+0000 7ff65e45c640 1 -- 172.21.15.83:0/2123436855 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff658068290 con 0x7ff6580c45c0 2024-09-19T07:38:05.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.618+0000 7ff654ff9640 1 -- 172.21.15.83:0/2123436855 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6543 (secure 0 0 0) 0x7ff650056990 con 0x7ff6580c45c0 2024-09-19T07:38:05.616 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:05.616 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":135,"btime":"2024-09-19T07:37:30:334185+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:05.616 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 135 2024-09-19T07:38:05.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.622+0000 7ff65e45c640 1 -- 172.21.15.83:0/2123436855 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6380678c0 msgr2=0x7ff638069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:05.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.622+0000 7ff65e45c640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6380678c0 0x7ff638069d80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff6400af000 tx=0x7ff640004740 comp rx=0 tx=0).stop 2024-09-19T07:38:05.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.622+0000 7ff65e45c640 1 -- 172.21.15.83:0/2123436855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6580c45c0 msgr2=0x7ff658073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:05.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.622+0000 7ff65e45c640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6580c45c0 0x7ff658073240 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7ff65000c7c0 tx=0x7ff65000cc80 comp rx=0 tx=0).stop 2024-09-19T07:38:05.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.622+0000 7ff65e45c640 1 -- 172.21.15.83:0/2123436855 shutdown_connections 2024-09-19T07:38:05.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.622+0000 7ff65e45c640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff6380678c0 0x7ff638069d80 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:05.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.622+0000 7ff65e45c640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff658152b20 0x7ff65814eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:05.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.622+0000 7ff65e45c640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff658150290 0x7ff65814e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:05.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.622+0000 7ff65e45c640 1 --2- 172.21.15.83:0/2123436855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6580c45c0 0x7ff658073240 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:05.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.622+0000 7ff65e45c640 1 -- 172.21.15.83:0/2123436855 >> 172.21.15.83:0/2123436855 conn(0x7ff65806bc80 msgr2=0x7ff65815c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.622+0000 7ff65e45c640 1 -- 172.21.15.83:0/2123436855 shutdown_connections 2024-09-19T07:38:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:05.622+0000 7ff65e45c640 1 -- 172.21.15.83:0/2123436855 wait complete. 2024-09-19T07:38:05.634 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:38:07.636 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:07.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cde81e640 1 Processor -- start 2024-09-19T07:38:07.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cde81e640 1 -- start start 2024-09-19T07:38:07.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cde81e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4cd80c6120 0x7f4cd80c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:07.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cde81e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd8155d70 0x7f4cd81575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:07.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cde81e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cd8156200 con 0x7f4cd81592f0 2024-09-19T07:38:07.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cde81e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f4cd80c9930 con 0x7f4cd80c6120 2024-09-19T07:38:07.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cde81e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4cd81564e0 con 0x7f4cd8155d70 2024-09-19T07:38:07.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cd7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4cd80c6120 0x7f4cd80c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:07.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cd7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4cd80c6120 0x7f4cd80c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57596/0 (socket says 172.21.15.83:57596) 2024-09-19T07:38:07.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cd7fff640 1 -- 172.21.15.83:0/1926150904 learned_addr learned my addr 172.21.15.83:0/1926150904 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:07.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cd77fe640 1 --2- 172.21.15.83:0/1926150904 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd8155d70 0x7f4cd81575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:07.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cd7fff640 1 -- 172.21.15.83:0/1926150904 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd8155d70 msgr2=0x7f4cd81575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:07.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cd7fff640 1 --2- 172.21.15.83:0/1926150904 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd8155d70 0x7f4cd81575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:07.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cd7fff640 1 -- 172.21.15.83:0/1926150904 >> v1:172.21.15.184:6789/0 conn(0x7f4cd81592f0 legacy=0x7f4cd8157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:07.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.766+0000 7f4cd7fff640 1 -- 172.21.15.83:0/1926150904 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd81567c0 con 0x7f4cd80c6120 2024-09-19T07:38:07.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd7fff640 1 --2- 172.21.15.83:0/1926150904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cd80c6120 0x7f4cd80c6500 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009870 tx=0x7f4ccc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0d608eb44c335 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:07.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd6ffd640 1 -- 172.21.15.83:0/1926150904 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc0b0070 con 0x7f4cd80c6120 2024-09-19T07:38:07.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd6ffd640 1 -- 172.21.15.83:0/1926150904 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ccc0ab600 con 0x7f4cd80c6120 2024-09-19T07:38:07.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd6ffd640 1 -- 172.21.15.83:0/1926150904 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc0ab9d0 con 0x7f4cd80c6120 2024-09-19T07:38:07.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 -- 172.21.15.83:0/1926150904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cd80c6120 msgr2=0x7f4cd80c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:07.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 --2- 172.21.15.83:0/1926150904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cd80c6120 0x7f4cd80c6500 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009870 tx=0x7f4ccc01c520 comp rx=0 tx=0).stop 2024-09-19T07:38:07.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 -- 172.21.15.83:0/1926150904 shutdown_connections 2024-09-19T07:38:07.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 --2- 172.21.15.83:0/1926150904 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd8155d70 0x7f4cd81575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:07.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 --2- 172.21.15.83:0/1926150904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cd80c6120 0x7f4cd80c6500 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:07.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 -- 172.21.15.83:0/1926150904 >> 172.21.15.83:0/1926150904 conn(0x7f4cd806bc80 msgr2=0x7f4cd806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:07.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 -- 172.21.15.83:0/1926150904 shutdown_connections 2024-09-19T07:38:07.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 -- 172.21.15.83:0/1926150904 wait complete. 2024-09-19T07:38:07.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 Processor -- start 2024-09-19T07:38:07.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 -- start start 2024-09-19T07:38:07.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cd80c6120 0x7f4cd80ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:07.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cd81592f0 0x7f4cd80cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:07.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd814a690 0x7f4cd81495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:07.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd8153860 con 0x7f4cd81592f0 2024-09-19T07:38:07.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd8162ea0 con 0x7f4cd80c6120 2024-09-19T07:38:07.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4cd81482a0 con 0x7f4cd814a690 2024-09-19T07:38:07.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cd81592f0 0x7f4cd80cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:07.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cd81592f0 0x7f4cd80cb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57604/0 (socket says 172.21.15.83:57604) 2024-09-19T07:38:07.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd77fe640 1 -- 172.21.15.83:0/314302598 learned_addr learned my addr 172.21.15.83:0/314302598 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:07.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd7fff640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cd80c6120 0x7f4cd80ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:07.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cdcdbb640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd814a690 0x7f4cd81495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:07.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd77fe640 1 -- 172.21.15.83:0/314302598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd814a690 msgr2=0x7f4cd81495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:07.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd77fe640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd814a690 0x7f4cd81495d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:07.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd77fe640 1 -- 172.21.15.83:0/314302598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cd80c6120 msgr2=0x7f4cd80ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:07.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd77fe640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cd80c6120 0x7f4cd80ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:07.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd77fe640 1 -- 172.21.15.83:0/314302598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd8148f90 con 0x7f4cd81592f0 2024-09-19T07:38:07.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd7fff640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cd80c6120 0x7f4cd80ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:38:07.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd77fe640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cd81592f0 0x7f4cd80cb830 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800d800 tx=0x7f4cc800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:07.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd4ff9640 1 -- 172.21.15.83:0/314302598 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc800b6c0 con 0x7f4cd81592f0 2024-09-19T07:38:07.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cd4ff9640 1 -- 172.21.15.83:0/314302598 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cc8019070 con 0x7f4cd81592f0 2024-09-19T07:38:07.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.770+0000 7f4cde81e640 1 -- 172.21.15.83:0/314302598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd80cbd70 con 0x7f4cd81592f0 2024-09-19T07:38:07.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.774+0000 7f4cd4ff9640 1 -- 172.21.15.83:0/314302598 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc800ba60 con 0x7f4cd81592f0 2024-09-19T07:38:07.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.774+0000 7f4cde81e640 1 -- 172.21.15.83:0/314302598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd814f440 con 0x7f4cd81592f0 2024-09-19T07:38:07.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.774+0000 7f4cde81e640 1 -- 172.21.15.83:0/314302598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cd80c6500 con 0x7f4cd81592f0 2024-09-19T07:38:07.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.774+0000 7f4cd4ff9640 1 -- 172.21.15.83:0/314302598 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4cc800bc00 con 0x7f4cd81592f0 2024-09-19T07:38:07.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.774+0000 7f4cd4ff9640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4cc40678c0 0x7f4cc4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:07.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.774+0000 7f4cd4ff9640 1 -- 172.21.15.83:0/314302598 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8873+0+0 (secure 0 0 0) 0x7f4cc8086de0 con 0x7f4cd81592f0 2024-09-19T07:38:07.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.774+0000 7f4cd7fff640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4cc40678c0 0x7f4cc4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:07.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.774+0000 7f4cd7fff640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4cc40678c0 0x7f4cc4069d80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0ad000 tx=0x7f4ccc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:07.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.778+0000 7f4cd4ff9640 1 -- 172.21.15.83:0/314302598 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4cc8056c70 con 0x7f4cd81592f0 2024-09-19T07:38:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.974+0000 7f4cde81e640 1 -- 172.21.15.83:0/314302598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cd80c96a0 con 0x7f4cd81592f0 2024-09-19T07:38:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.974+0000 7f4cd4ff9640 1 -- 172.21.15.83:0/314302598 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6543 (secure 0 0 0) 0x7f4cc8056570 con 0x7f4cd81592f0 2024-09-19T07:38:07.972 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:07.972 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":135,"btime":"2024-09-19T07:37:30:334185+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:37:21.194289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":26,"addr":"172.21.15.83:6835/2168217861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2168217861},{"type":"v1","addr":"172.21.15.83:6835","nonce":2168217861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7113,"qdb_cluster":[7113]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:07.972 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 135 2024-09-19T07:38:07.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.978+0000 7f4cde81e640 1 -- 172.21.15.83:0/314302598 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4cc40678c0 msgr2=0x7f4cc4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:07.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.978+0000 7f4cde81e640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4cc40678c0 0x7f4cc4069d80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0ad000 tx=0x7f4ccc008040 comp rx=0 tx=0).stop 2024-09-19T07:38:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.978+0000 7f4cde81e640 1 -- 172.21.15.83:0/314302598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cd81592f0 msgr2=0x7f4cd80cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.978+0000 7f4cde81e640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cd81592f0 0x7f4cd80cb830 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800d800 tx=0x7f4cc800dcc0 comp rx=0 tx=0).stop 2024-09-19T07:38:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.978+0000 7f4cde81e640 1 -- 172.21.15.83:0/314302598 shutdown_connections 2024-09-19T07:38:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.978+0000 7f4cde81e640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4cc40678c0 0x7f4cc4069d80 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.978+0000 7f4cde81e640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd814a690 0x7f4cd81495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.978+0000 7f4cde81e640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cd81592f0 0x7f4cd80cb830 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.978+0000 7f4cde81e640 1 --2- 172.21.15.83:0/314302598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cd80c6120 0x7f4cd80ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.978+0000 7f4cde81e640 1 -- 172.21.15.83:0/314302598 >> 172.21.15.83:0/314302598 conn(0x7f4cd806bc80 msgr2=0x7f4cd81574d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.978+0000 7f4cde81e640 1 -- 172.21.15.83:0/314302598 shutdown_connections 2024-09-19T07:38:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:07.978+0000 7f4cde81e640 1 -- 172.21.15.83:0/314302598 wait complete. 2024-09-19T07:38:07.991 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:38:09.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:10.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa173bc7640 1 Processor -- start 2024-09-19T07:38:10.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa173bc7640 1 -- start start 2024-09-19T07:38:10.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa173bc7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fa16c0c42e0 0x7fa16c06e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa173bc7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16c1592c0 con 0x7fa16c1566b0 2024-09-19T07:38:10.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa173bc7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16c06e900 con 0x7fa16c0abb30 2024-09-19T07:38:10.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa173bc7640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fa16c1595f0 con 0x7fa16c0c42e0 2024-09-19T07:38:10.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa171162640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa16c1566b0 0x7fa16c159a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43892/0 (socket says 172.21.15.83:43892) 2024-09-19T07:38:10.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa171162640 1 -- 172.21.15.83:0/550205370 learned_addr learned my addr 172.21.15.83:0/550205370 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:38:10.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa171963640 1 --2- 172.21.15.83:0/550205370 >> v2:172.21.15.184:3300/0 conn(0x7fa16c0c42e0 0x7fa16c06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 655272070 0 0) 0x7fa16c1592c0 con 0x7fa16c1566b0 2024-09-19T07:38:10.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa158003530 con 0x7fa16c1566b0 2024-09-19T07:38:10.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1451409340 0 0) 0x7fa158003530 con 0x7fa16c1566b0 2024-09-19T07:38:10.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 >> v2:172.21.15.184:3300/0 conn(0x7fa16c0c42e0 msgr2=0x7fa16c06e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:10.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 --2- 172.21.15.83:0/550205370 >> v2:172.21.15.184:3300/0 conn(0x7fa16c0c42e0 0x7fa16c06e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 >> v1:172.21.15.184:6789/0 conn(0x7fa16c0abb30 legacy=0x7fa16c153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16c06ebe0 con 0x7fa16c1566b0 2024-09-19T07:38:10.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fa1680016a0 con 0x7fa16c1566b0 2024-09-19T07:38:10.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 >> v1:172.21.15.83:6789/0 conn(0x7fa16c1566b0 legacy=0x7fa16c159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 --2- 172.21.15.83:0/550205370 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1580039c0 0x7fa158003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 --2- 172.21.15.83:0/550205370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 0x7fa158004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 --2- 172.21.15.83:0/550205370 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1580051d0 0x7fa15800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c1595f0 con 0x7fa158004810 2024-09-19T07:38:10.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa15800c070 con 0x7fa1580039c0 2024-09-19T07:38:10.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa15800c270 con 0x7fa1580051d0 2024-09-19T07:38:10.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa171162640 1 --2- 172.21.15.83:0/550205370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 0x7fa158004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa171963640 1 --2- 172.21.15.83:0/550205370 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1580039c0 0x7fa158003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa171162640 1 -- 172.21.15.83:0/550205370 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1580051d0 msgr2=0x7fa15800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:10.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa171162640 1 --2- 172.21.15.83:0/550205370 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa1580051d0 0x7fa15800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa171162640 1 -- 172.21.15.83:0/550205370 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1580039c0 msgr2=0x7fa158003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:10.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa171162640 1 --2- 172.21.15.83:0/550205370 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1580039c0 0x7fa158003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa171162640 1 -- 172.21.15.83:0/550205370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa158003530 con 0x7fa158004810 2024-09-19T07:38:10.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa171963640 1 --2- 172.21.15.83:0/550205370 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1580039c0 0x7fa158003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:38:10.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.126+0000 7fa172164640 1 -- 172.21.15.83:0/550205370 reap_dead start 2024-09-19T07:38:10.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa171162640 1 --2- 172.21.15.83:0/550205370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 0x7fa158004c90 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c210 tx=0x7fa16801f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f8cf2f6bf1ece4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:10.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa168007d20 con 0x7fa158004810 2024-09-19T07:38:10.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa168005b00 con 0x7fa158004810 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa170961640 1 -- 172.21.15.83:0/550205370 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa168007070 con 0x7fa158004810 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 -- 172.21.15.83:0/550205370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 msgr2=0x7fa158004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 --2- 172.21.15.83:0/550205370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 0x7fa158004c90 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c210 tx=0x7fa16801f390 comp rx=0 tx=0).stop 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 -- 172.21.15.83:0/550205370 shutdown_connections 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 --2- 172.21.15.83:0/550205370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 0x7fa158004c90 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 -- 172.21.15.83:0/550205370 >> 172.21.15.83:0/550205370 conn(0x7fa16c06bc80 msgr2=0x7fa16c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 -- 172.21.15.83:0/550205370 shutdown_connections 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 -- 172.21.15.83:0/550205370 wait complete. 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 Processor -- start 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 -- start start 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 0x7fa16c0c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa16c0c8230 0x7fa16c14c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa16c0c7780 0x7fa16c14c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c0abf60 con 0x7fa158004810 2024-09-19T07:38:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c06eec0 con 0x7fa16c0c7780 2024-09-19T07:38:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa173bc7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa16c0c46c0 con 0x7fa16c0c8230 2024-09-19T07:38:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa171963640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 0x7fa16c0c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa171963640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 0x7fa16c0c7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42624/0 (socket says 172.21.15.83:42624) 2024-09-19T07:38:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa171963640 1 -- 172.21.15.83:0/2778442176 learned_addr learned my addr 172.21.15.83:0/2778442176 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa171162640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa16c0c8230 0x7fa16c14c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.130+0000 7fa172164640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa16c0c7780 0x7fa16c14c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa171963640 1 -- 172.21.15.83:0/2778442176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa16c0c8230 msgr2=0x7fa16c14c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa171963640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa16c0c8230 0x7fa16c14c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa171963640 1 -- 172.21.15.83:0/2778442176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa16c0c7780 msgr2=0x7fa16c14c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa171963640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa16c0c7780 0x7fa16c14c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa171963640 1 -- 172.21.15.83:0/2778442176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16c0c7cd0 con 0x7fa158004810 2024-09-19T07:38:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa171963640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 0x7fa16c0c7240 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fa160011930 tx=0x7fa160011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa15e7fc640 1 -- 172.21.15.83:0/2778442176 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16000fb70 con 0x7fa158004810 2024-09-19T07:38:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa15e7fc640 1 -- 172.21.15.83:0/2778442176 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa16000fd10 con 0x7fa158004810 2024-09-19T07:38:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa15e7fc640 1 -- 172.21.15.83:0/2778442176 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa160015680 con 0x7fa158004810 2024-09-19T07:38:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa173bc7640 1 -- 172.21.15.83:0/2778442176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa16c1ec860 con 0x7fa158004810 2024-09-19T07:38:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa173bc7640 1 -- 172.21.15.83:0/2778442176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa16c0c5640 con 0x7fa158004810 2024-09-19T07:38:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa173bc7640 1 -- 172.21.15.83:0/2778442176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa16c06c530 con 0x7fa158004810 2024-09-19T07:38:10.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa15e7fc640 1 -- 172.21.15.83:0/2778442176 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa160015820 con 0x7fa158004810 2024-09-19T07:38:10.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa15e7fc640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1500678f0 0x7fa150069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa15e7fc640 1 -- 172.21.15.83:0/2778442176 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8959+0+0 (secure 0 0 0) 0x7fa1600872b0 con 0x7fa158004810 2024-09-19T07:38:10.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa171162640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1500678f0 0x7fa150069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.134+0000 7fa171162640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1500678f0 0x7fa150069db0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa1680ae000 tx=0x7fa168007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:10.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.138+0000 7fa15e7fc640 1 -- 172.21.15.83:0/2778442176 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa160019030 con 0x7fa158004810 2024-09-19T07:38:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.322+0000 7fa173bc7640 1 -- 172.21.15.83:0/2778442176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa16c0c2480 con 0x7fa158004810 2024-09-19T07:38:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.322+0000 7fa15e7fc640 1 -- 172.21.15.83:0/2778442176 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+5713 (secure 0 0 0) 0x7fa160057220 con 0x7fa158004810 2024-09-19T07:38:10.319 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:10.319 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":137,"btime":"2024-09-19T07:38:09:338642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:08.552334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:10.319 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 137 2024-09-19T07:38:10.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.326+0000 7fa173bc7640 1 -- 172.21.15.83:0/2778442176 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1500678f0 msgr2=0x7fa150069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.326+0000 7fa173bc7640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1500678f0 0x7fa150069db0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa1680ae000 tx=0x7fa168007040 comp rx=0 tx=0).stop 2024-09-19T07:38:10.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.326+0000 7fa173bc7640 1 -- 172.21.15.83:0/2778442176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 msgr2=0x7fa16c0c7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.326+0000 7fa173bc7640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 0x7fa16c0c7240 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fa160011930 tx=0x7fa160011df0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.326+0000 7fa173bc7640 1 -- 172.21.15.83:0/2778442176 shutdown_connections 2024-09-19T07:38:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.326+0000 7fa173bc7640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1500678f0 0x7fa150069db0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.326+0000 7fa173bc7640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa16c0c7780 0x7fa16c14c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.326+0000 7fa173bc7640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa16c0c8230 0x7fa16c14c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.326+0000 7fa173bc7640 1 --2- 172.21.15.83:0/2778442176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa158004810 0x7fa16c0c7240 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.326+0000 7fa173bc7640 1 -- 172.21.15.83:0/2778442176 >> 172.21.15.83:0/2778442176 conn(0x7fa16c06bc80 msgr2=0x7fa16c155ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.326+0000 7fa173bc7640 1 -- 172.21.15.83:0/2778442176 shutdown_connections 2024-09-19T07:38:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.326+0000 7fa173bc7640 1 -- 172.21.15.83:0/2778442176 wait complete. 2024-09-19T07:38:10.337 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-19T07:38:10.337 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:38:10.337 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:10.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b65b3d640 1 Processor -- start 2024-09-19T07:38:10.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b65b3d640 1 -- start start 2024-09-19T07:38:10.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b65b3d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 0x7f5b600c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b65b3d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5b60155d70 0x7f5b601575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b65b3d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b60156200 con 0x7f5b601592f0 2024-09-19T07:38:10.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b65b3d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5b600c9930 con 0x7f5b60155d70 2024-09-19T07:38:10.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b65b3d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5b601564e0 con 0x7f5b600c6120 2024-09-19T07:38:10.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b5f7fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 0x7f5b600c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b57fff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5b60155d70 0x7f5b601575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b5f7fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 0x7f5b600c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:46230/0 (socket says 172.21.15.83:46230) 2024-09-19T07:38:10.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b5f7fe640 1 -- 172.21.15.83:0/614863273 learned_addr learned my addr 172.21.15.83:0/614863273 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:10.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b57fff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5b60155d70 0x7f5b601575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:58730/0 (socket says 172.21.15.83:58730) 2024-09-19T07:38:10.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b5ffff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f5b601592f0 0x7f5b60157b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:37816/0 (socket says 172.21.15.83:37816) 2024-09-19T07:38:10.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b5f7fe640 1 -- 172.21.15.83:0/614863273 >> v2:172.21.15.184:3300/0 conn(0x7f5b60155d70 msgr2=0x7f5b601575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:10.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b5f7fe640 1 --2- 172.21.15.83:0/614863273 >> v2:172.21.15.184:3300/0 conn(0x7f5b60155d70 0x7f5b601575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b5f7fe640 1 -- 172.21.15.83:0/614863273 >> v1:172.21.15.184:6789/0 conn(0x7f5b601592f0 legacy=0x7f5b60157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.462+0000 7f5b5f7fe640 1 -- 172.21.15.83:0/614863273 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b601567c0 con 0x7f5b600c6120 2024-09-19T07:38:10.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b5f7fe640 1 --2- 172.21.15.83:0/614863273 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 0x7f5b600c6500 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009870 tx=0x7f5b5001c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=225c37689f032440 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:10.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b5effd640 1 -- 172.21.15.83:0/614863273 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b500b2070 con 0x7f5b600c6120 2024-09-19T07:38:10.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b5effd640 1 -- 172.21.15.83:0/614863273 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b500a9070 con 0x7f5b600c6120 2024-09-19T07:38:10.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b5effd640 1 -- 172.21.15.83:0/614863273 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b500ad880 con 0x7f5b600c6120 2024-09-19T07:38:10.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 -- 172.21.15.83:0/614863273 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 msgr2=0x7f5b600c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 --2- 172.21.15.83:0/614863273 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 0x7f5b600c6500 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009870 tx=0x7f5b5001c560 comp rx=0 tx=0).stop 2024-09-19T07:38:10.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 -- 172.21.15.83:0/614863273 shutdown_connections 2024-09-19T07:38:10.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 --2- 172.21.15.83:0/614863273 >> v2:172.21.15.184:3300/0 conn(0x7f5b60155d70 0x7f5b601575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 --2- 172.21.15.83:0/614863273 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 0x7f5b600c6500 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 -- 172.21.15.83:0/614863273 >> 172.21.15.83:0/614863273 conn(0x7f5b6006bc80 msgr2=0x7f5b6006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:10.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 -- 172.21.15.83:0/614863273 shutdown_connections 2024-09-19T07:38:10.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 -- 172.21.15.83:0/614863273 wait complete. 2024-09-19T07:38:10.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 Processor -- start 2024-09-19T07:38:10.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 -- start start 2024-09-19T07:38:10.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 0x7f5b600ab340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b601592f0 0x7f5b60145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b600c18f0 0x7f5b6014b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5b60153860 con 0x7f5b601592f0 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5b60162ea0 con 0x7f5b600c18f0 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b65b3d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5b600ac790 con 0x7f5b600c6120 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b57fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b601592f0 0x7f5b60145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b57fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b601592f0 0x7f5b60145d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42628/0 (socket says 172.21.15.83:42628) 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b57fff640 1 -- 172.21.15.83:0/2747203383 learned_addr learned my addr 172.21.15.83:0/2747203383 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b5ffff640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b600c18f0 0x7f5b6014b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.466+0000 7f5b5f7fe640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 0x7f5b600ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b57fff640 1 -- 172.21.15.83:0/2747203383 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 msgr2=0x7f5b600ab340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b57fff640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 0x7f5b600ab340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b57fff640 1 -- 172.21.15.83:0/2747203383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b600c18f0 msgr2=0x7f5b6014b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b57fff640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b600c18f0 0x7f5b6014b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b57fff640 1 -- 172.21.15.83:0/2747203383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b600caa90 con 0x7f5b601592f0 2024-09-19T07:38:10.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b5f7fe640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 0x7f5b600ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:38:10.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b57fff640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b601592f0 0x7f5b60145d40 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f5b48006d50 tx=0x7f5b4800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:10.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b5cff9640 1 -- 172.21.15.83:0/2747203383 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b48017070 con 0x7f5b601592f0 2024-09-19T07:38:10.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b5cff9640 1 -- 172.21.15.83:0/2747203383 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b4800c8e0 con 0x7f5b601592f0 2024-09-19T07:38:10.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b65b3d640 1 -- 172.21.15.83:0/2747203383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b6014c3c0 con 0x7f5b601592f0 2024-09-19T07:38:10.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b5cff9640 1 -- 172.21.15.83:0/2747203383 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4800cc90 con 0x7f5b601592f0 2024-09-19T07:38:10.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b65b3d640 1 -- 172.21.15.83:0/2747203383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b600ca570 con 0x7f5b601592f0 2024-09-19T07:38:10.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b65b3d640 1 -- 172.21.15.83:0/2747203383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b600c6500 con 0x7f5b601592f0 2024-09-19T07:38:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b5cff9640 1 -- 172.21.15.83:0/2747203383 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5b4801a070 con 0x7f5b601592f0 2024-09-19T07:38:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b5cff9640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b400678c0 0x7f5b40069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b5cff9640 1 -- 172.21.15.83:0/2747203383 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8959+0+0 (secure 0 0 0) 0x7f5b48086c10 con 0x7f5b601592f0 2024-09-19T07:38:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b5f7fe640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b400678c0 0x7f5b40069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.470+0000 7f5b5f7fe640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b400678c0 0x7f5b40069d80 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5b500af000 tx=0x7f5b500a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.474+0000 7f5b5cff9640 1 -- 172.21.15.83:0/2747203383 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5b48056ae0 con 0x7f5b601592f0 2024-09-19T07:38:10.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.678+0000 7f5b65b3d640 1 -- 172.21.15.83:0/2747203383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b600c96a0 con 0x7f5b601592f0 2024-09-19T07:38:10.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.678+0000 7f5b5cff9640 1 -- 172.21.15.83:0/2747203383 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) ==== 95+0+5710 (secure 0 0 0) 0x7f5b480563e0 con 0x7f5b601592f0 2024-09-19T07:38:10.674 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:10.674 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":138,"btime":"2024-09-19T07:38:10:350493+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:09.355702+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:10.674 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 138 2024-09-19T07:38:10.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.678+0000 7f5b65b3d640 1 -- 172.21.15.83:0/2747203383 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b400678c0 msgr2=0x7f5b40069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.678+0000 7f5b65b3d640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b400678c0 0x7f5b40069d80 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5b500af000 tx=0x7f5b500a7000 comp rx=0 tx=0).stop 2024-09-19T07:38:10.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.678+0000 7f5b65b3d640 1 -- 172.21.15.83:0/2747203383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b601592f0 msgr2=0x7f5b60145d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:10.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.678+0000 7f5b65b3d640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b601592f0 0x7f5b60145d40 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f5b48006d50 tx=0x7f5b4800ed60 comp rx=0 tx=0).stop 2024-09-19T07:38:10.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.678+0000 7f5b65b3d640 1 -- 172.21.15.83:0/2747203383 shutdown_connections 2024-09-19T07:38:10.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.678+0000 7f5b65b3d640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5b400678c0 0x7f5b40069d80 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.678+0000 7f5b65b3d640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b600c18f0 0x7f5b6014b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.678+0000 7f5b65b3d640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b601592f0 0x7f5b60145d40 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.678+0000 7f5b65b3d640 1 --2- 172.21.15.83:0/2747203383 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b600c6120 0x7f5b600ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:10.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.678+0000 7f5b65b3d640 1 -- 172.21.15.83:0/2747203383 >> 172.21.15.83:0/2747203383 conn(0x7f5b6006bc80 msgr2=0x7f5b60152990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:10.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.682+0000 7f5b65b3d640 1 -- 172.21.15.83:0/2747203383 shutdown_connections 2024-09-19T07:38:10.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:10.682+0000 7f5b65b3d640 1 -- 172.21.15.83:0/2747203383 wait complete. 2024-09-19T07:38:10.694 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:38:10:350493+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 138, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 138, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7266": { "addr": "172.21.15.83:6837/2326757290", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 2326757290, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 2326757290, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7266, "incarnation": 136, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:38:09.355702+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7266 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 133, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5555": { "addr": "172.21.15.184:6833/381468017", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 381468017, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 381468017, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5555, "incarnation": 130, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 46 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:37:21.194290+0000", "qdb_cluster": [ 5555 ], "qdb_leader": 5555, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5555 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:38:12.696 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:12.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.830+0000 7f5476f19640 1 Processor -- start 2024-09-19T07:38:12.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.830+0000 7f5476f19640 1 -- start start 2024-09-19T07:38:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.830+0000 7f5476f19640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f54701494a0 0x7f547006e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.830+0000 7f5476f19640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5470153390 0x7f547006d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.830+0000 7f5476f19640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5470155fa0 con 0x7f5470068700 2024-09-19T07:38:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.830+0000 7f5476f19640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5470156e40 con 0x7f5470153390 2024-09-19T07:38:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f5476f19640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f54700c3850 con 0x7f54701494a0 2024-09-19T07:38:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f54754b6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5470153390 0x7f547006d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f54754b6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5470153390 0x7f547006d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42644/0 (socket says 172.21.15.83:42644) 2024-09-19T07:38:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f54754b6640 1 -- 172.21.15.83:0/3368124603 learned_addr learned my addr 172.21.15.83:0/3368124603 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f5474cb5640 1 --2- 172.21.15.83:0/3368124603 >> v2:172.21.15.184:3300/0 conn(0x7f54701494a0 0x7f547006e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f54754b6640 1 -- 172.21.15.83:0/3368124603 >> v2:172.21.15.184:3300/0 conn(0x7f54701494a0 msgr2=0x7f547006e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f54754b6640 1 --2- 172.21.15.83:0/3368124603 >> v2:172.21.15.184:3300/0 conn(0x7f54701494a0 0x7f547006e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f54754b6640 1 -- 172.21.15.83:0/3368124603 >> v1:172.21.15.83:6789/0 conn(0x7f5470068700 legacy=0x7f5470156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:12.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f54754b6640 1 -- 172.21.15.83:0/3368124603 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54700c3b10 con 0x7f5470153390 2024-09-19T07:38:12.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f546f7fe640 1 -- 172.21.15.83:0/3368124603 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1258292867 0 0) 0x7f5470155fa0 con 0x7f5470068700 2024-09-19T07:38:12.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f54754b6640 1 --2- 172.21.15.83:0/3368124603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5470153390 0x7f547006d470 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f5470070dd0 tx=0x7f546801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3562378cb16cd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:12.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f546f7fe640 1 -- 172.21.15.83:0/3368124603 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54680a57d0 con 0x7f5470153390 2024-09-19T07:38:12.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f546f7fe640 1 -- 172.21.15.83:0/3368124603 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54680a5970 con 0x7f5470153390 2024-09-19T07:38:12.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f546f7fe640 1 -- 172.21.15.83:0/3368124603 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54680a5d40 con 0x7f5470153390 2024-09-19T07:38:12.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f5476f19640 1 -- 172.21.15.83:0/3368124603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5470153390 msgr2=0x7f547006d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:12.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f5476f19640 1 --2- 172.21.15.83:0/3368124603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5470153390 0x7f547006d470 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f5470070dd0 tx=0x7f546801c9f0 comp rx=0 tx=0).stop 2024-09-19T07:38:12.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f5476f19640 1 -- 172.21.15.83:0/3368124603 shutdown_connections 2024-09-19T07:38:12.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f5476f19640 1 --2- 172.21.15.83:0/3368124603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5470153390 0x7f547006d470 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:12.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f5476f19640 1 --2- 172.21.15.83:0/3368124603 >> v2:172.21.15.184:3300/0 conn(0x7f54701494a0 0x7f547006e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:12.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f5476f19640 1 -- 172.21.15.83:0/3368124603 >> 172.21.15.83:0/3368124603 conn(0x7f54700c4940 msgr2=0x7f54700c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:12.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f5476f19640 1 -- 172.21.15.83:0/3368124603 shutdown_connections 2024-09-19T07:38:12.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.834+0000 7f5476f19640 1 -- 172.21.15.83:0/3368124603 wait complete. 2024-09-19T07:38:12.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f5476f19640 1 Processor -- start 2024-09-19T07:38:12.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f5476f19640 1 -- start start 2024-09-19T07:38:12.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f5476f19640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5470068700 0x7f5470072ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:12.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f5476f19640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54701494a0 0x7f54700c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:12.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f5476f19640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5470153390 0x7f54700c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f5476f19640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f547015fab0 con 0x7f54701494a0 2024-09-19T07:38:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f5476f19640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f54700c40d0 con 0x7f5470068700 2024-09-19T07:38:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f5476f19640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f547015fc30 con 0x7f5470153390 2024-09-19T07:38:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f546ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54701494a0 0x7f54700c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f546ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54701494a0 0x7f54700c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42652/0 (socket says 172.21.15.83:42652) 2024-09-19T07:38:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f546ffff640 1 -- 172.21.15.83:0/2763339342 learned_addr learned my addr 172.21.15.83:0/2763339342 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f5474cb5640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5470068700 0x7f5470072ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f54754b6640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5470153390 0x7f54700c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f546ffff640 1 -- 172.21.15.83:0/2763339342 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5470153390 msgr2=0x7f54700c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f546ffff640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5470153390 0x7f54700c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f546ffff640 1 -- 172.21.15.83:0/2763339342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5470068700 msgr2=0x7f5470072ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f546ffff640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5470068700 0x7f5470072ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:12.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f546ffff640 1 -- 172.21.15.83:0/2763339342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54701ec210 con 0x7f54701494a0 2024-09-19T07:38:12.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f546ffff640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54701494a0 0x7f54700c7060 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f545c00c4d0 tx=0x7f545c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:12.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f546d7fa640 1 -- 172.21.15.83:0/2763339342 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c017070 con 0x7f54701494a0 2024-09-19T07:38:12.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f546d7fa640 1 -- 172.21.15.83:0/2763339342 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f545c005480 con 0x7f54701494a0 2024-09-19T07:38:12.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f5476f19640 1 -- 172.21.15.83:0/2763339342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54701ec4f0 con 0x7f54701494a0 2024-09-19T07:38:12.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f546d7fa640 1 -- 172.21.15.83:0/2763339342 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c003840 con 0x7f54701494a0 2024-09-19T07:38:12.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.838+0000 7f5476f19640 1 -- 172.21.15.83:0/2763339342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54701ecab0 con 0x7f54701494a0 2024-09-19T07:38:12.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.842+0000 7f5476f19640 1 -- 172.21.15.83:0/2763339342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f547006d470 con 0x7f54701494a0 2024-09-19T07:38:12.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.842+0000 7f546d7fa640 1 -- 172.21.15.83:0/2763339342 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f545c01a070 con 0x7f54701494a0 2024-09-19T07:38:12.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.842+0000 7f546d7fa640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f54540678c0 0x7f5454069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:12.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.842+0000 7f5474cb5640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f54540678c0 0x7f5454069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:12.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.842+0000 7f546d7fa640 1 -- 172.21.15.83:0/2763339342 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8959+0+0 (secure 0 0 0) 0x7f545c085d00 con 0x7f54701494a0 2024-09-19T07:38:12.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.842+0000 7f5474cb5640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f54540678c0 0x7f5454069d80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f54700738f0 tx=0x7f5464008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:12.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:12.846+0000 7f546d7fa640 1 -- 172.21.15.83:0/2763339342 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f545c055c90 con 0x7f54701494a0 2024-09-19T07:38:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.082+0000 7f5476f19640 1 -- 172.21.15.83:0/2763339342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5470070060 con 0x7f54701494a0 2024-09-19T07:38:13.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.082+0000 7f546d7fa640 1 -- 172.21.15.83:0/2763339342 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+5717 (secure 0 0 0) 0x7f545c055590 con 0x7f54701494a0 2024-09-19T07:38:13.079 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:13.079 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":139,"btime":"2024-09-19T07:38:11:364479+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:13.079 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 139 2024-09-19T07:38:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.086+0000 7f5476f19640 1 -- 172.21.15.83:0/2763339342 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f54540678c0 msgr2=0x7f5454069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.086+0000 7f5476f19640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f54540678c0 0x7f5454069d80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f54700738f0 tx=0x7f5464008040 comp rx=0 tx=0).stop 2024-09-19T07:38:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.086+0000 7f5476f19640 1 -- 172.21.15.83:0/2763339342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54701494a0 msgr2=0x7f54700c7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.086+0000 7f5476f19640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54701494a0 0x7f54700c7060 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f545c00c4d0 tx=0x7f545c00c990 comp rx=0 tx=0).stop 2024-09-19T07:38:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.086+0000 7f5476f19640 1 -- 172.21.15.83:0/2763339342 shutdown_connections 2024-09-19T07:38:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.086+0000 7f5476f19640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f54540678c0 0x7f5454069d80 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.086+0000 7f5476f19640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5470153390 0x7f54700c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.086+0000 7f5476f19640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54701494a0 0x7f54700c7060 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.086+0000 7f5476f19640 1 --2- 172.21.15.83:0/2763339342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5470068700 0x7f5470072ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.086+0000 7f5476f19640 1 -- 172.21.15.83:0/2763339342 >> 172.21.15.83:0/2763339342 conn(0x7f54700c4940 msgr2=0x7f547006f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.086+0000 7f5476f19640 1 -- 172.21.15.83:0/2763339342 shutdown_connections 2024-09-19T07:38:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:13.086+0000 7f5476f19640 1 -- 172.21.15.83:0/2763339342 wait complete. 2024-09-19T07:38:13.099 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=7113 2024-09-19T07:38:13.100 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.a 2024-09-19T07:38:15.101 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-19T07:38:15.101 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:38:15.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:38:15.104 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:38:15.104 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:15.178 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:38:15.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.242+0000 7f9427b7d640 1 Processor -- start 2024-09-19T07:38:15.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.242+0000 7f9427b7d640 1 -- start start 2024-09-19T07:38:15.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.242+0000 7f9427b7d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f94200c8000 0x7f94200c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:15.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9427b7d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94200c69b0 con 0x7f9420153460 2024-09-19T07:38:15.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9427b7d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9420155d90 con 0x7f94200bee70 2024-09-19T07:38:15.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9427b7d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f94201566c0 con 0x7f94200c8000 2024-09-19T07:38:15.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9425919640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9420153460 0x7f94200c60f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43924/0 (socket says 172.21.15.83:43924) 2024-09-19T07:38:15.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9425919640 1 -- 172.21.15.83:0/1352374832 learned_addr learned my addr 172.21.15.83:0/1352374832 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:38:15.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f942611a640 1 --2- 172.21.15.83:0/1352374832 >> v2:172.21.15.184:3300/0 conn(0x7f94200c8000 0x7f94200c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:15.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3751961871 0 0) 0x7f94200c69b0 con 0x7f9420153460 2024-09-19T07:38:15.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f940c003530 con 0x7f9420153460 2024-09-19T07:38:15.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3097558676 0 0) 0x7f940c003530 con 0x7f9420153460 2024-09-19T07:38:15.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 >> v2:172.21.15.184:3300/0 conn(0x7f94200c8000 msgr2=0x7f94200c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:15.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 --2- 172.21.15.83:0/1352374832 >> v2:172.21.15.184:3300/0 conn(0x7f94200c8000 0x7f94200c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:15.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 >> v1:172.21.15.184:6789/0 conn(0x7f94200bee70 legacy=0x7f94200cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:15.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9420156070 con 0x7f9420153460 2024-09-19T07:38:15.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f941c001680 con 0x7f9420153460 2024-09-19T07:38:15.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f942611a640 1 --2- 172.21.15.83:0/1352374832 >> v2:172.21.15.184:3300/0 conn(0x7f94200c8000 0x7f94200c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:38:15.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 >> v1:172.21.15.83:6789/0 conn(0x7f9420153460 legacy=0x7f94200c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:15.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 --2- 172.21.15.83:0/1352374832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0039c0 0x7f940c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:15.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 --2- 172.21.15.83:0/1352374832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f940c004810 0x7f940c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:15.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 --2- 172.21.15.83:0/1352374832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f940c0051d0 0x7f940c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:15.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f94201566c0 con 0x7f940c0039c0 2024-09-19T07:38:15.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f940c00c070 con 0x7f940c0051d0 2024-09-19T07:38:15.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f940c00c270 con 0x7f940c004810 2024-09-19T07:38:15.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9425919640 1 --2- 172.21.15.83:0/1352374832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0039c0 0x7f940c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:15.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f942611a640 1 --2- 172.21.15.83:0/1352374832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f940c0051d0 0x7f940c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:15.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9425919640 1 -- 172.21.15.83:0/1352374832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f940c004810 msgr2=0x7f940c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:15.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9425919640 1 --2- 172.21.15.83:0/1352374832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f940c004810 0x7f940c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:15.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9425919640 1 -- 172.21.15.83:0/1352374832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f940c0051d0 msgr2=0x7f940c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:15.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9425919640 1 --2- 172.21.15.83:0/1352374832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f940c0051d0 0x7f940c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:15.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f942611a640 1 -- 172.21.15.83:0/1352374832 reap_dead start 2024-09-19T07:38:15.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9425919640 1 -- 172.21.15.83:0/1352374832 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f940c003530 con 0x7f940c0039c0 2024-09-19T07:38:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9425919640 1 --2- 172.21.15.83:0/1352374832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0039c0 0x7f940c003de0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f941c00c1c0 tx=0x7f941c01cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3882bd8a3cc966cd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c007d00 con 0x7f940c0039c0 2024-09-19T07:38:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f941c0a9020 con 0x7f940c0039c0 2024-09-19T07:38:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.246+0000 7f9424917640 1 -- 172.21.15.83:0/1352374832 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c005ae0 con 0x7f940c0039c0 2024-09-19T07:38:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 -- 172.21.15.83:0/1352374832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0039c0 msgr2=0x7f940c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 --2- 172.21.15.83:0/1352374832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0039c0 0x7f940c003de0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f941c00c1c0 tx=0x7f941c01cc20 comp rx=0 tx=0).stop 2024-09-19T07:38:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 -- 172.21.15.83:0/1352374832 shutdown_connections 2024-09-19T07:38:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 --2- 172.21.15.83:0/1352374832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0039c0 0x7f940c003de0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 -- 172.21.15.83:0/1352374832 >> 172.21.15.83:0/1352374832 conn(0x7f94200726a0 msgr2=0x7f9420072ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 -- 172.21.15.83:0/1352374832 shutdown_connections 2024-09-19T07:38:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 -- 172.21.15.83:0/1352374832 wait complete. 2024-09-19T07:38:15.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 Processor -- start 2024-09-19T07:38:15.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 -- start start 2024-09-19T07:38:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f940c0039c0 0x7f94201f1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94201f2230 0x7f94201efa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94200c1fd0 0x7f94201eff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f94200c6270 con 0x7f94200c1fd0 2024-09-19T07:38:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f94200c60f0 con 0x7f940c0039c0 2024-09-19T07:38:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f94200c63f0 con 0x7f94201f2230 2024-09-19T07:38:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f942611a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94200c1fd0 0x7f94201eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f942611a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94200c1fd0 0x7f94201eff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42692/0 (socket says 172.21.15.83:42692) 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f942611a640 1 -- 172.21.15.83:0/275268756 learned_addr learned my addr 172.21.15.83:0/275268756 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9425919640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f940c0039c0 0x7f94201f1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9425118640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94201f2230 0x7f94201efa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f942611a640 1 -- 172.21.15.83:0/275268756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94201f2230 msgr2=0x7f94201efa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f942611a640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94201f2230 0x7f94201efa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f942611a640 1 -- 172.21.15.83:0/275268756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f940c0039c0 msgr2=0x7f94201f1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f942611a640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f940c0039c0 0x7f94201f1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f942611a640 1 -- 172.21.15.83:0/275268756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94201f2780 con 0x7f94200c1fd0 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f942611a640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94200c1fd0 0x7f94201eff40 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f94200c7740 tx=0x7f9418009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f94167fc640 1 -- 172.21.15.83:0/275268756 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418011da0 con 0x7f94200c1fd0 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f94167fc640 1 -- 172.21.15.83:0/275268756 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f941801b070 con 0x7f94200c1fd0 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 -- 172.21.15.83:0/275268756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94200c5120 con 0x7f94200c1fd0 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f94167fc640 1 -- 172.21.15.83:0/275268756 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94180039a0 con 0x7f94200c1fd0 2024-09-19T07:38:15.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.250+0000 7f9427b7d640 1 -- 172.21.15.83:0/275268756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94200be870 con 0x7f94200c1fd0 2024-09-19T07:38:15.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.254+0000 7f9427b7d640 1 -- 172.21.15.83:0/275268756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94200bea70 con 0x7f94200c1fd0 2024-09-19T07:38:15.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.254+0000 7f94167fc640 1 -- 172.21.15.83:0/275268756 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9418003b40 con 0x7f94200c1fd0 2024-09-19T07:38:15.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.254+0000 7f94167fc640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94040678f0 0x7f9404069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:15.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.254+0000 7f94167fc640 1 -- 172.21.15.83:0/275268756 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8959+0+0 (secure 0 0 0) 0x7f9418087020 con 0x7f94200c1fd0 2024-09-19T07:38:15.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.254+0000 7f9425919640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94040678f0 0x7f9404069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:15.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.254+0000 7f9425919640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94040678f0 0x7f9404069db0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f941c0aa000 tx=0x7f9420155f60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:15.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.258+0000 7f94167fc640 1 -- 172.21.15.83:0/275268756 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f94200bea70 con 0x7f94200c1fd0 2024-09-19T07:38:15.339 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:38:15.342+0000 7fe1ed51e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:38:15.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.498+0000 7f9427b7d640 1 -- 172.21.15.83:0/275268756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94200be660 con 0x7f94200c1fd0 2024-09-19T07:38:15.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.498+0000 7f94167fc640 1 -- 172.21.15.83:0/275268756 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+6540 (secure 0 0 0) 0x7f9418056810 con 0x7f94200c1fd0 2024-09-19T07:38:15.495 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:15.495 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":140,"btime":"2024-09-19T07:38:15:379040+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:15.495 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 140 2024-09-19T07:38:15.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.502+0000 7f9427b7d640 1 -- 172.21.15.83:0/275268756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94040678f0 msgr2=0x7f9404069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:15.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.502+0000 7f9427b7d640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94040678f0 0x7f9404069db0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f941c0aa000 tx=0x7f9420155f60 comp rx=0 tx=0).stop 2024-09-19T07:38:15.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.502+0000 7f9427b7d640 1 -- 172.21.15.83:0/275268756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94200c1fd0 msgr2=0x7f94201eff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:15.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.502+0000 7f9427b7d640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94200c1fd0 0x7f94201eff40 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f94200c7740 tx=0x7f9418009f90 comp rx=0 tx=0).stop 2024-09-19T07:38:15.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.502+0000 7f9427b7d640 1 -- 172.21.15.83:0/275268756 shutdown_connections 2024-09-19T07:38:15.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.502+0000 7f9427b7d640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94040678f0 0x7f9404069db0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:15.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.502+0000 7f9427b7d640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94200c1fd0 0x7f94201eff40 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:15.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.502+0000 7f9427b7d640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94201f2230 0x7f94201efa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:15.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.502+0000 7f9427b7d640 1 --2- 172.21.15.83:0/275268756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f940c0039c0 0x7f94201f1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:15.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.502+0000 7f9427b7d640 1 -- 172.21.15.83:0/275268756 >> 172.21.15.83:0/275268756 conn(0x7f94200726a0 msgr2=0x7f94200c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:15.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.502+0000 7f9427b7d640 1 -- 172.21.15.83:0/275268756 shutdown_connections 2024-09-19T07:38:15.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:15.502+0000 7f9427b7d640 1 -- 172.21.15.83:0/275268756 wait complete. 2024-09-19T07:38:15.518 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-19T07:38:15.518 INFO:tasks.mds_thrash.fs.[a]:waiting for 113 secs before thrashing 2024-09-19T07:38:35.070 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:38:35.073 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:38:46.753 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:46.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbcf622a640 1 Processor -- start 2024-09-19T07:38:46.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbcf622a640 1 -- start start 2024-09-19T07:38:46.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbcf622a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbcf00bee30 0x7fbcf01570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:46.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbcf622a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbcf00c9680 0x7fbcf0157630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:46.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbcf622a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcf0157b70 con 0x7fbcf00c6360 2024-09-19T07:38:46.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbcf622a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fbcf00c5b90 con 0x7fbcf00bee30 2024-09-19T07:38:46.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbcf622a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbcf015a050 con 0x7fbcf00c9680 2024-09-19T07:38:46.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbceeffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbcf00bee30 0x7fbcf01570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:46.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbceeffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbcf00bee30 0x7fbcf01570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56648/0 (socket says 172.21.15.83:56648) 2024-09-19T07:38:46.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbceeffd640 1 -- 172.21.15.83:0/63415146 learned_addr learned my addr 172.21.15.83:0/63415146 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:46.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbceffff640 1 --2- 172.21.15.83:0/63415146 >> v2:172.21.15.184:3300/0 conn(0x7fbcf00c9680 0x7fbcf0157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:46.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbceeffd640 1 -- 172.21.15.83:0/63415146 >> v2:172.21.15.184:3300/0 conn(0x7fbcf00c9680 msgr2=0x7fbcf0157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:46.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbceeffd640 1 --2- 172.21.15.83:0/63415146 >> v2:172.21.15.184:3300/0 conn(0x7fbcf00c9680 0x7fbcf0157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:46.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbceeffd640 1 -- 172.21.15.83:0/63415146 >> v1:172.21.15.184:6789/0 conn(0x7fbcf00c6360 legacy=0x7fbcf0153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:46.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.901+0000 7fbceeffd640 1 -- 172.21.15.83:0/63415146 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf006c950 con 0x7fbcf00bee30 2024-09-19T07:38:46.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.905+0000 7fbceeffd640 1 --2- 172.21.15.83:0/63415146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf00bee30 0x7fbcf01570f0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fbcf006cdf0 tx=0x7fbce001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154e2b7028d335e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:46.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.905+0000 7fbcee7fc640 1 -- 172.21.15.83:0/63415146 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce00b0070 con 0x7fbcf00bee30 2024-09-19T07:38:46.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.905+0000 7fbcee7fc640 1 -- 172.21.15.83:0/63415146 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbce00ab600 con 0x7fbcf00bee30 2024-09-19T07:38:46.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.905+0000 7fbcee7fc640 1 -- 172.21.15.83:0/63415146 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce00ab9d0 con 0x7fbcf00bee30 2024-09-19T07:38:46.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.905+0000 7fbcf622a640 1 -- 172.21.15.83:0/63415146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf00bee30 msgr2=0x7fbcf01570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:46.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.905+0000 7fbcf622a640 1 --2- 172.21.15.83:0/63415146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf00bee30 0x7fbcf01570f0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fbcf006cdf0 tx=0x7fbce001c3c0 comp rx=0 tx=0).stop 2024-09-19T07:38:46.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.905+0000 7fbcf622a640 1 -- 172.21.15.83:0/63415146 shutdown_connections 2024-09-19T07:38:46.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.905+0000 7fbcf622a640 1 --2- 172.21.15.83:0/63415146 >> v2:172.21.15.184:3300/0 conn(0x7fbcf00c9680 0x7fbcf0157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:46.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.905+0000 7fbcf622a640 1 --2- 172.21.15.83:0/63415146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf00bee30 0x7fbcf01570f0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:46.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.905+0000 7fbcf622a640 1 -- 172.21.15.83:0/63415146 >> 172.21.15.83:0/63415146 conn(0x7fbcf00722a0 msgr2=0x7fbcf00726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:46.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.905+0000 7fbcf622a640 1 -- 172.21.15.83:0/63415146 shutdown_connections 2024-09-19T07:38:46.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.905+0000 7fbcf622a640 1 -- 172.21.15.83:0/63415146 wait complete. 2024-09-19T07:38:46.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcf622a640 1 Processor -- start 2024-09-19T07:38:46.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcf622a640 1 -- start start 2024-09-19T07:38:46.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcf622a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcf00bee30 0x7fbcf0149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:46.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcf622a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbcf00c6360 0x7fbcf0147a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:46.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcf622a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf00c9680 0x7fbcf0147f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:46.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcf622a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf0153760 con 0x7fbcf00c9680 2024-09-19T07:38:46.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcf622a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf0159850 con 0x7fbcf00bee30 2024-09-19T07:38:46.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcf622a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbcf01538e0 con 0x7fbcf00c6360 2024-09-19T07:38:46.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbceffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf00c9680 0x7fbcf0147f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:46.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbceffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf00c9680 0x7fbcf0147f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56654/0 (socket says 172.21.15.83:56654) 2024-09-19T07:38:46.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbceffff640 1 -- 172.21.15.83:0/2297313059 learned_addr learned my addr 172.21.15.83:0/2297313059 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:46.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcef7fe640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcf00bee30 0x7fbcf0149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:46.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbceeffd640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbcf00c6360 0x7fbcf0147a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:46.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbceffff640 1 -- 172.21.15.83:0/2297313059 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbcf00c6360 msgr2=0x7fbcf0147a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:46.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbceffff640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbcf00c6360 0x7fbcf0147a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:46.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbceffff640 1 -- 172.21.15.83:0/2297313059 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcf00bee30 msgr2=0x7fbcf0149cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:46.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbceffff640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcf00bee30 0x7fbcf0149cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:46.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbceffff640 1 -- 172.21.15.83:0/2297313059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf0150350 con 0x7fbcf00c9680 2024-09-19T07:38:46.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbceffff640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf00c9680 0x7fbcf0147f90 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fbce8007cf0 tx=0x7fbce800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:46.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcd7fff640 1 -- 172.21.15.83:0/2297313059 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8014420 con 0x7fbcf00c9680 2024-09-19T07:38:46.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcd7fff640 1 -- 172.21.15.83:0/2297313059 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbce8019070 con 0x7fbcf00c9680 2024-09-19T07:38:46.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcf622a640 1 -- 172.21.15.83:0/2297313059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf014e230 con 0x7fbcf00c9680 2024-09-19T07:38:46.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcd7fff640 1 -- 172.21.15.83:0/2297313059 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8014820 con 0x7fbcf00c9680 2024-09-19T07:38:46.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcf622a640 1 -- 172.21.15.83:0/2297313059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf014e7f0 con 0x7fbcf00c9680 2024-09-19T07:38:46.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.909+0000 7fbcf622a640 1 -- 172.21.15.83:0/2297313059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcf00c6740 con 0x7fbcf00c9680 2024-09-19T07:38:46.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.913+0000 7fbcd7fff640 1 -- 172.21.15.83:0/2297313059 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbce80149c0 con 0x7fbcf00c9680 2024-09-19T07:38:46.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.913+0000 7fbcd7fff640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbcdc0678c0 0x7fbcdc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:46.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.913+0000 7fbcd7fff640 1 -- 172.21.15.83:0/2297313059 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8959+0+0 (secure 0 0 0) 0x7fbce8087150 con 0x7fbcf00c9680 2024-09-19T07:38:46.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.913+0000 7fbcef7fe640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbcdc0678c0 0x7fbcdc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:46.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.913+0000 7fbcef7fe640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbcdc0678c0 0x7fbcdc069d80 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbce400b810 tx=0x7fbce4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:46.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:46.917+0000 7fbcd7fff640 1 -- 172.21.15.83:0/2297313059 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbce8056f90 con 0x7fbcf00c9680 2024-09-19T07:38:47.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 -- 172.21.15.83:0/2297313059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbcf014ce30 con 0x7fbcf00c9680 2024-09-19T07:38:47.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcd7fff640 1 -- 172.21.15.83:0/2297313059 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+6540 (secure 0 0 0) 0x7fbce8056890 con 0x7fbcf00c9680 2024-09-19T07:38:47.141 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:47.142 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":140,"btime":"2024-09-19T07:38:15:379040+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:47.142 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 140 2024-09-19T07:38:47.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 -- 172.21.15.83:0/2297313059 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbcdc0678c0 msgr2=0x7fbcdc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:47.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbcdc0678c0 0x7fbcdc069d80 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbce400b810 tx=0x7fbce4008040 comp rx=0 tx=0).stop 2024-09-19T07:38:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 -- 172.21.15.83:0/2297313059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf00c9680 msgr2=0x7fbcf0147f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf00c9680 0x7fbcf0147f90 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fbce8007cf0 tx=0x7fbce800e4b0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 -- 172.21.15.83:0/2297313059 shutdown_connections 2024-09-19T07:38:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbcdc0678c0 0x7fbcdc069d80 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcf00c9680 0x7fbcf0147f90 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbcf00c6360 0x7fbcf0147a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 --2- 172.21.15.83:0/2297313059 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbcf00bee30 0x7fbcf0149cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 -- 172.21.15.83:0/2297313059 >> 172.21.15.83:0/2297313059 conn(0x7fbcf00722a0 msgr2=0x7fbcf00acbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 -- 172.21.15.83:0/2297313059 shutdown_connections 2024-09-19T07:38:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.145+0000 7fbcf622a640 1 -- 172.21.15.83:0/2297313059 wait complete. 2024-09-19T07:38:47.160 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-19T07:38:47.160 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-19T07:38:47.160 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:38:47.178 INFO:tasks.ceph.mds.d:Stopped 2024-09-19T07:38:47.178 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:47.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.297+0000 7efc35df6640 1 Processor -- start 2024-09-19T07:38:47.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.297+0000 7efc35df6640 1 -- start start 2024-09-19T07:38:47.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc35df6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc300c21e0 0x7efc30152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:47.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc35df6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc30071740 con 0x7efc300721b0 2024-09-19T07:38:47.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc35df6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc3014f980 con 0x7efc3014f5a0 2024-09-19T07:38:47.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc35df6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efc300730c0 con 0x7efc300c21e0 2024-09-19T07:38:47.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7efc300721b0 0x7efc300c3850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43506/0 (socket says 172.21.15.83:43506) 2024-09-19T07:38:47.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2ffff640 1 -- 172.21.15.83:0/3635247234 learned_addr learned my addr 172.21.15.83:0/3635247234 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2effd640 1 --2- 172.21.15.83:0/3635247234 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc300c21e0 0x7efc30152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622675310 0 0) 0x7efc30071740 con 0x7efc300721b0 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc18003530 con 0x7efc300721b0 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 486188720 0 0) 0x7efc18003530 con 0x7efc300721b0 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc300c21e0 msgr2=0x7efc30152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 --2- 172.21.15.83:0/3635247234 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc300c21e0 0x7efc30152ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 >> v1:172.21.15.184:6789/0 conn(0x7efc3014f5a0 legacy=0x7efc30153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc300719b0 con 0x7efc300721b0 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7efc28001a00 con 0x7efc300721b0 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 >> v1:172.21.15.83:6789/0 conn(0x7efc300721b0 legacy=0x7efc300c3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 --2- 172.21.15.83:0/3635247234 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc300c21e0 0x7efc180039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 --2- 172.21.15.83:0/3635247234 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc18003f70 0x7efc18004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 --2- 172.21.15.83:0/3635247234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc18004d50 0x7efc180051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:47.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efc300730c0 con 0x7efc18004d50 2024-09-19T07:38:47.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efc1800bab0 con 0x7efc18003f70 2024-09-19T07:38:47.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efc1800bcb0 con 0x7efc300c21e0 2024-09-19T07:38:47.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2ffff640 1 --2- 172.21.15.83:0/3635247234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc18004d50 0x7efc180051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:47.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2effd640 1 --2- 172.21.15.83:0/3635247234 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc18003f70 0x7efc18004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:47.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2f7fe640 1 --2- 172.21.15.83:0/3635247234 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc300c21e0 0x7efc180039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:47.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2ffff640 1 -- 172.21.15.83:0/3635247234 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc300c21e0 msgr2=0x7efc180039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:47.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2ffff640 1 --2- 172.21.15.83:0/3635247234 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc300c21e0 0x7efc180039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2ffff640 1 -- 172.21.15.83:0/3635247234 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc18003f70 msgr2=0x7efc18004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:47.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2ffff640 1 --2- 172.21.15.83:0/3635247234 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc18003f70 0x7efc18004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.301+0000 7efc2ffff640 1 -- 172.21.15.83:0/3635247234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc18003530 con 0x7efc18004d50 2024-09-19T07:38:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc2ffff640 1 --2- 172.21.15.83:0/3635247234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc18004d50 0x7efc180051d0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7efc2800a850 tx=0x7efc2801d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ca3ff25330a43d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2800a4a0 con 0x7efc18004d50 2024-09-19T07:38:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc280ad650 con 0x7efc18004d50 2024-09-19T07:38:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc2e7fc640 1 -- 172.21.15.83:0/3635247234 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc280ada80 con 0x7efc18004d50 2024-09-19T07:38:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc35df6640 1 -- 172.21.15.83:0/3635247234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc18004d50 msgr2=0x7efc180051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc35df6640 1 --2- 172.21.15.83:0/3635247234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc18004d50 0x7efc180051d0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7efc2800a850 tx=0x7efc2801d3a0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc2ffff640 1 -- 172.21.15.83:0/3635247234 reap_dead start 2024-09-19T07:38:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc35df6640 1 -- 172.21.15.83:0/3635247234 shutdown_connections 2024-09-19T07:38:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc35df6640 1 -- 172.21.15.83:0/3635247234 >> 172.21.15.83:0/3635247234 conn(0x7efc300c4940 msgr2=0x7efc300c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:47.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc35df6640 1 -- 172.21.15.83:0/3635247234 shutdown_connections 2024-09-19T07:38:47.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc35df6640 1 -- 172.21.15.83:0/3635247234 wait complete. 2024-09-19T07:38:47.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc35df6640 1 Processor -- start 2024-09-19T07:38:47.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.305+0000 7efc35df6640 1 -- start start 2024-09-19T07:38:47.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc35df6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc18004d50 0x7efc300ad700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:47.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc35df6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc300acd60 0x7efc301e9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:47.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc35df6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc301e9c70 0x7efc301455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:47.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc35df6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efc300733a0 con 0x7efc301e9c70 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc35df6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efc30072600 con 0x7efc300acd60 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc35df6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efc300c3f60 con 0x7efc18004d50 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc301e9c70 0x7efc301455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc301e9c70 0x7efc301455e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56668/0 (socket says 172.21.15.83:56668) 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2ffff640 1 -- 172.21.15.83:0/3184185807 learned_addr learned my addr 172.21.15.83:0/3184185807 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2effd640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc300acd60 0x7efc301e9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2f7fe640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc18004d50 0x7efc300ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2ffff640 1 -- 172.21.15.83:0/3184185807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc18004d50 msgr2=0x7efc300ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2ffff640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc18004d50 0x7efc300ad700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2ffff640 1 -- 172.21.15.83:0/3184185807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc300acd60 msgr2=0x7efc301e9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2ffff640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc300acd60 0x7efc301e9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2ffff640 1 -- 172.21.15.83:0/3184185807 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc30154100 con 0x7efc301e9c70 2024-09-19T07:38:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2f7fe640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc18004d50 0x7efc300ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:38:47.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2ffff640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc301e9c70 0x7efc301455e0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7efc2801ce10 tx=0x7efc18005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:47.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc17fff640 1 -- 172.21.15.83:0/3184185807 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc280bc070 con 0x7efc301e9c70 2024-09-19T07:38:47.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc17fff640 1 -- 172.21.15.83:0/3184185807 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc2800a480 con 0x7efc301e9c70 2024-09-19T07:38:47.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc35df6640 1 -- 172.21.15.83:0/3184185807 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc30153860 con 0x7efc301e9c70 2024-09-19T07:38:47.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc17fff640 1 -- 172.21.15.83:0/3184185807 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc280b2040 con 0x7efc301e9c70 2024-09-19T07:38:47.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc35df6640 1 -- 172.21.15.83:0/3184185807 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc30153e20 con 0x7efc301e9c70 2024-09-19T07:38:47.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc35df6640 1 -- 172.21.15.83:0/3184185807 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc30152ab0 con 0x7efc301e9c70 2024-09-19T07:38:47.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc17fff640 1 -- 172.21.15.83:0/3184185807 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efc280bf070 con 0x7efc301e9c70 2024-09-19T07:38:47.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc17fff640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc1c0678f0 0x7efc1c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:47.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc17fff640 1 -- 172.21.15.83:0/3184185807 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8959+0+0 (secure 0 0 0) 0x7efc281206f0 con 0x7efc301e9c70 2024-09-19T07:38:47.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2f7fe640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc1c0678f0 0x7efc1c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:47.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.309+0000 7efc2f7fe640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc1c0678f0 0x7efc1c069db0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7efc24000c10 tx=0x7efc24009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:47.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.313+0000 7efc17fff640 1 -- 172.21.15.83:0/3184185807 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7efc280170b0 con 0x7efc301e9c70 2024-09-19T07:38:47.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.557+0000 7efc35df6640 1 -- 172.21.15.83:0/3184185807 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc300c4770 con 0x7efc301e9c70 2024-09-19T07:38:47.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc17fff640 1 -- 172.21.15.83:0/3184185807 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+6540 (secure 0 0 0) 0x7efc280f05c0 con 0x7efc301e9c70 2024-09-19T07:38:47.556 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:47.557 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":140,"btime":"2024-09-19T07:38:15:379040+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:47.557 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 140 2024-09-19T07:38:47.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc35df6640 1 -- 172.21.15.83:0/3184185807 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc1c0678f0 msgr2=0x7efc1c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:47.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc35df6640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc1c0678f0 0x7efc1c069db0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7efc24000c10 tx=0x7efc24009040 comp rx=0 tx=0).stop 2024-09-19T07:38:47.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc35df6640 1 -- 172.21.15.83:0/3184185807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc301e9c70 msgr2=0x7efc301455e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:47.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc35df6640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc301e9c70 0x7efc301455e0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7efc2801ce10 tx=0x7efc18005840 comp rx=0 tx=0).stop 2024-09-19T07:38:47.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc35df6640 1 -- 172.21.15.83:0/3184185807 shutdown_connections 2024-09-19T07:38:47.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc35df6640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efc1c0678f0 0x7efc1c069db0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc35df6640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc301e9c70 0x7efc301455e0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc35df6640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc300acd60 0x7efc301e9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc35df6640 1 --2- 172.21.15.83:0/3184185807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc18004d50 0x7efc300ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:47.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc35df6640 1 -- 172.21.15.83:0/3184185807 >> 172.21.15.83:0/3184185807 conn(0x7efc300c4940 msgr2=0x7efc300c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:47.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc35df6640 1 -- 172.21.15.83:0/3184185807 shutdown_connections 2024-09-19T07:38:47.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:47.561+0000 7efc35df6640 1 -- 172.21.15.83:0/3184185807 wait complete. 2024-09-19T07:38:47.575 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:38:49.576 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f732693d640 1 Processor -- start 2024-09-19T07:38:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f732693d640 1 -- start start 2024-09-19T07:38:49.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f732693d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f73200ad010 0x7f732006d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:49.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f732693d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7320071ea0 0x7f73200c3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:49.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f732693d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7320153240 con 0x7f7320150040 2024-09-19T07:38:49.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f732693d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f7320153520 con 0x7f73200ad010 2024-09-19T07:38:49.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f732693d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7320153800 con 0x7f7320071ea0 2024-09-19T07:38:49.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f7324eda640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7320071ea0 0x7f73200c3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:49.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f731ffff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f7320150040 0x7f7320152930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:42608/0 (socket says 172.21.15.83:42608) 2024-09-19T07:38:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f731f7fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f73200ad010 0x7f732006d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f731ffff640 1 -- 172.21.15.83:0/3941696035 learned_addr learned my addr 172.21.15.83:0/3941696035 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:38:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f731effd640 1 -- 172.21.15.83:0/3941696035 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1183778281 0 0) 0x7f7320153240 con 0x7f7320150040 2024-09-19T07:38:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f731effd640 1 -- 172.21.15.83:0/3941696035 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7308003530 con 0x7f7320150040 2024-09-19T07:38:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f7324eda640 1 -- 172.21.15.83:0/3941696035 >> v2:172.21.15.184:3300/0 conn(0x7f73200ad010 msgr2=0x7f732006d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f7324eda640 1 --2- 172.21.15.83:0/3941696035 >> v2:172.21.15.184:3300/0 conn(0x7f73200ad010 0x7f732006d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f7324eda640 1 -- 172.21.15.83:0/3941696035 >> v1:172.21.15.184:6789/0 conn(0x7f7320150040 legacy=0x7f7320152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f7324eda640 1 -- 172.21.15.83:0/3941696035 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73200716a0 con 0x7f7320071ea0 2024-09-19T07:38:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.713+0000 7f731f7fe640 1 --2- 172.21.15.83:0/3941696035 >> v2:172.21.15.184:3300/0 conn(0x7f73200ad010 0x7f732006d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:38:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.717+0000 7f7324eda640 1 --2- 172.21.15.83:0/3941696035 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7320071ea0 0x7f73200c3950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7318009d90 tx=0x7f731801c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9d9f3b320b8ccc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.717+0000 7f731effd640 1 -- 172.21.15.83:0/3941696035 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73180b2070 con 0x7f7320071ea0 2024-09-19T07:38:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.717+0000 7f731effd640 1 -- 172.21.15.83:0/3941696035 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73180a57a0 con 0x7f7320071ea0 2024-09-19T07:38:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.717+0000 7f731effd640 1 -- 172.21.15.83:0/3941696035 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73180a5b50 con 0x7f7320071ea0 2024-09-19T07:38:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.717+0000 7f732693d640 1 -- 172.21.15.83:0/3941696035 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7320071ea0 msgr2=0x7f73200c3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.717+0000 7f732693d640 1 --2- 172.21.15.83:0/3941696035 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7320071ea0 0x7f73200c3950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7318009d90 tx=0x7f731801c8e0 comp rx=0 tx=0).stop 2024-09-19T07:38:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.717+0000 7f732693d640 1 -- 172.21.15.83:0/3941696035 shutdown_connections 2024-09-19T07:38:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.717+0000 7f732693d640 1 --2- 172.21.15.83:0/3941696035 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7320071ea0 0x7f73200c3950 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.717+0000 7f732693d640 1 --2- 172.21.15.83:0/3941696035 >> v2:172.21.15.184:3300/0 conn(0x7f73200ad010 0x7f732006d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.717+0000 7f732693d640 1 -- 172.21.15.83:0/3941696035 >> 172.21.15.83:0/3941696035 conn(0x7f73200c7e40 msgr2=0x7f73200c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.717+0000 7f732693d640 1 -- 172.21.15.83:0/3941696035 shutdown_connections 2024-09-19T07:38:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.717+0000 7f732693d640 1 -- 172.21.15.83:0/3941696035 wait complete. 2024-09-19T07:38:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f732693d640 1 Processor -- start 2024-09-19T07:38:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f732693d640 1 -- start start 2024-09-19T07:38:49.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f732693d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7320071ea0 0x7f73200c2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:49.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f732693d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73200ad010 0x7f73201466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:49.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f732693d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7320150040 0x7f7320146bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:49.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f732693d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f732015fe60 con 0x7f73200ad010 2024-09-19T07:38:49.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f732693d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f732015fce0 con 0x7f7320071ea0 2024-09-19T07:38:49.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f732693d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f732015ffe0 con 0x7f7320150040 2024-09-19T07:38:49.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73200ad010 0x7f73201466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:49.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73200ad010 0x7f73201466b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56674/0 (socket says 172.21.15.83:56674) 2024-09-19T07:38:49.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731f7fe640 1 -- 172.21.15.83:0/926370368 learned_addr learned my addr 172.21.15.83:0/926370368 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:49.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731ffff640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7320071ea0 0x7f73200c2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:49.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f7324eda640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7320150040 0x7f7320146bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731f7fe640 1 -- 172.21.15.83:0/926370368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7320150040 msgr2=0x7f7320146bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731f7fe640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7320150040 0x7f7320146bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731f7fe640 1 -- 172.21.15.83:0/926370368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7320071ea0 msgr2=0x7f73200c2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731f7fe640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7320071ea0 0x7f73200c2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731f7fe640 1 -- 172.21.15.83:0/926370368 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73200c7c50 con 0x7f73200ad010 2024-09-19T07:38:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731f7fe640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73200ad010 0x7f73201466b0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f731400ebe0 tx=0x7f731400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731cff9640 1 -- 172.21.15.83:0/926370368 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7314017070 con 0x7f73200ad010 2024-09-19T07:38:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731cff9640 1 -- 172.21.15.83:0/926370368 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f731400cc30 con 0x7f73200ad010 2024-09-19T07:38:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f731cff9640 1 -- 172.21.15.83:0/926370368 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7314013470 con 0x7f73200ad010 2024-09-19T07:38:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f732693d640 1 -- 172.21.15.83:0/926370368 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73201516c0 con 0x7f73200ad010 2024-09-19T07:38:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.721+0000 7f732693d640 1 -- 172.21.15.83:0/926370368 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7320150f80 con 0x7f73200ad010 2024-09-19T07:38:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.725+0000 7f732693d640 1 -- 172.21.15.83:0/926370368 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7320150a90 con 0x7f73200ad010 2024-09-19T07:38:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.725+0000 7f731cff9640 1 -- 172.21.15.83:0/926370368 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f731401a070 con 0x7f73200ad010 2024-09-19T07:38:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.725+0000 7f731cff9640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f730c0678f0 0x7f730c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.725+0000 7f731cff9640 1 -- 172.21.15.83:0/926370368 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8959+0+0 (secure 0 0 0) 0x7f7314086bd0 con 0x7f73200ad010 2024-09-19T07:38:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.725+0000 7f731ffff640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f730c0678f0 0x7f730c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.725+0000 7f731ffff640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f730c0678f0 0x7f730c069db0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7310007fd0 tx=0x7f7310009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:49.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.729+0000 7f731cff9640 1 -- 172.21.15.83:0/926370368 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f73140563c0 con 0x7f73200ad010 2024-09-19T07:38:49.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.953+0000 7f732693d640 1 -- 172.21.15.83:0/926370368 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73200c8350 con 0x7f73200ad010 2024-09-19T07:38:49.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.953+0000 7f731cff9640 1 -- 172.21.15.83:0/926370368 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+6540 (secure 0 0 0) 0x7f731405a6c0 con 0x7f73200ad010 2024-09-19T07:38:49.951 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:49.951 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":140,"btime":"2024-09-19T07:38:15:379040+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:49.951 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 140 2024-09-19T07:38:49.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.957+0000 7f732693d640 1 -- 172.21.15.83:0/926370368 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f730c0678f0 msgr2=0x7f730c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:49.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.957+0000 7f732693d640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f730c0678f0 0x7f730c069db0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7310007fd0 tx=0x7f7310009040 comp rx=0 tx=0).stop 2024-09-19T07:38:49.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.957+0000 7f732693d640 1 -- 172.21.15.83:0/926370368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73200ad010 msgr2=0x7f73201466b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:49.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.957+0000 7f732693d640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73200ad010 0x7f73201466b0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f731400ebe0 tx=0x7f731400c520 comp rx=0 tx=0).stop 2024-09-19T07:38:49.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.957+0000 7f732693d640 1 -- 172.21.15.83:0/926370368 shutdown_connections 2024-09-19T07:38:49.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.957+0000 7f732693d640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f730c0678f0 0x7f730c069db0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:49.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.957+0000 7f732693d640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7320150040 0x7f7320146bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:49.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.957+0000 7f732693d640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73200ad010 0x7f73201466b0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:49.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.957+0000 7f732693d640 1 --2- 172.21.15.83:0/926370368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7320071ea0 0x7f73200c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:49.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.957+0000 7f732693d640 1 -- 172.21.15.83:0/926370368 >> 172.21.15.83:0/926370368 conn(0x7f73200c7e40 msgr2=0x7f73200c4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:49.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.957+0000 7f732693d640 1 -- 172.21.15.83:0/926370368 shutdown_connections 2024-09-19T07:38:49.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:49.957+0000 7f732693d640 1 -- 172.21.15.83:0/926370368 wait complete. 2024-09-19T07:38:49.972 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:38:51.973 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:52.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.113+0000 7fed64789640 1 Processor -- start 2024-09-19T07:38:52.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.113+0000 7fed64789640 1 -- start start 2024-09-19T07:38:52.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed64789640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fed5c150600 0x7fed5c156220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:52.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed64789640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fed5c06d3e0 0x7fed5c153560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:52.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed64789640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed5c068290 con 0x7fed5c150220 2024-09-19T07:38:52.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed64789640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fed5c068570 con 0x7fed5c06d3e0 2024-09-19T07:38:52.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed64789640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fed5c152d60 con 0x7fed5c150600 2024-09-19T07:38:52.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed62d26640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fed5c06d3e0 0x7fed5c153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:52.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed62525640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fed5c150220 0x7fed5c0ad2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56864/0 (socket says 172.21.15.83:56864) 2024-09-19T07:38:52.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed62525640 1 -- 172.21.15.83:0/1527798475 learned_addr learned my addr 172.21.15.83:0/1527798475 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:38:52.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed61d24640 1 --2- 172.21.15.83:0/1527798475 >> v2:172.21.15.184:3300/0 conn(0x7fed5c150600 0x7fed5c156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:52.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed62d26640 1 -- 172.21.15.83:0/1527798475 >> v2:172.21.15.184:3300/0 conn(0x7fed5c150600 msgr2=0x7fed5c156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:52.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed62d26640 1 --2- 172.21.15.83:0/1527798475 >> v2:172.21.15.184:3300/0 conn(0x7fed5c150600 0x7fed5c156220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:52.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed61523640 1 -- 172.21.15.83:0/1527798475 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4255389811 0 0) 0x7fed5c068290 con 0x7fed5c150220 2024-09-19T07:38:52.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed62d26640 1 -- 172.21.15.83:0/1527798475 >> v1:172.21.15.83:6789/0 conn(0x7fed5c150220 legacy=0x7fed5c0ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:52.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed62d26640 1 -- 172.21.15.83:0/1527798475 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5c153040 con 0x7fed5c06d3e0 2024-09-19T07:38:52.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.117+0000 7fed62d26640 1 --2- 172.21.15.83:0/1527798475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed5c06d3e0 0x7fed5c153560 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0aadb0 tx=0x7fed5401c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979d73d965cfba6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:52.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed61523640 1 -- 172.21.15.83:0/1527798475 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed540b2070 con 0x7fed5c06d3e0 2024-09-19T07:38:52.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed61523640 1 -- 172.21.15.83:0/1527798475 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed540a5770 con 0x7fed5c06d3e0 2024-09-19T07:38:52.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed61523640 1 -- 172.21.15.83:0/1527798475 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed540a5b40 con 0x7fed5c06d3e0 2024-09-19T07:38:52.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 -- 172.21.15.83:0/1527798475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed5c06d3e0 msgr2=0x7fed5c153560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:52.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 --2- 172.21.15.83:0/1527798475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed5c06d3e0 0x7fed5c153560 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0aadb0 tx=0x7fed5401c750 comp rx=0 tx=0).stop 2024-09-19T07:38:52.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 -- 172.21.15.83:0/1527798475 shutdown_connections 2024-09-19T07:38:52.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 --2- 172.21.15.83:0/1527798475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed5c06d3e0 0x7fed5c153560 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:52.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 --2- 172.21.15.83:0/1527798475 >> v2:172.21.15.184:3300/0 conn(0x7fed5c150600 0x7fed5c156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:52.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 -- 172.21.15.83:0/1527798475 >> 172.21.15.83:0/1527798475 conn(0x7fed5c0c48b0 msgr2=0x7fed5c0c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:52.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 -- 172.21.15.83:0/1527798475 shutdown_connections 2024-09-19T07:38:52.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 -- 172.21.15.83:0/1527798475 wait complete. 2024-09-19T07:38:52.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 Processor -- start 2024-09-19T07:38:52.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 -- start start 2024-09-19T07:38:52.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed5c06d3e0 0x7fed5c15aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:52.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed5c150220 0x7fed5c157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:52.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed5c150600 0x7fed5c1ed760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:52.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fed5c15d980 con 0x7fed5c06d3e0 2024-09-19T07:38:52.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fed5c15d800 con 0x7fed5c150220 2024-09-19T07:38:52.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed64789640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fed5c15db00 con 0x7fed5c150600 2024-09-19T07:38:52.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed62525640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed5c06d3e0 0x7fed5c15aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:52.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed62525640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed5c06d3e0 0x7fed5c15aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41800/0 (socket says 172.21.15.83:41800) 2024-09-19T07:38:52.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed62525640 1 -- 172.21.15.83:0/3044752761 learned_addr learned my addr 172.21.15.83:0/3044752761 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:52.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed61d24640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed5c150220 0x7fed5c157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:52.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed62525640 1 -- 172.21.15.83:0/3044752761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed5c150600 msgr2=0x7fed5c1ed760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:52.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed62d26640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed5c150600 0x7fed5c1ed760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:52.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed62525640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed5c150600 0x7fed5c1ed760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:52.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed62525640 1 -- 172.21.15.83:0/3044752761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed5c150220 msgr2=0x7fed5c157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:52.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed62525640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed5c150220 0x7fed5c157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:52.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed62525640 1 -- 172.21.15.83:0/3044752761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5c15b4d0 con 0x7fed5c06d3e0 2024-09-19T07:38:52.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed61d24640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed5c150220 0x7fed5c157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:38:52.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.121+0000 7fed62525640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed5c06d3e0 0x7fed5c15aaf0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fed5800c520 tx=0x7fed5800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:52.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.125+0000 7fed52ffd640 1 -- 172.21.15.83:0/3044752761 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed58018070 con 0x7fed5c06d3e0 2024-09-19T07:38:52.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.125+0000 7fed52ffd640 1 -- 172.21.15.83:0/3044752761 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed580074e0 con 0x7fed5c06d3e0 2024-09-19T07:38:52.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.125+0000 7fed64789640 1 -- 172.21.15.83:0/3044752761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed5c15b790 con 0x7fed5c06d3e0 2024-09-19T07:38:52.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.125+0000 7fed52ffd640 1 -- 172.21.15.83:0/3044752761 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed580049f0 con 0x7fed5c06d3e0 2024-09-19T07:38:52.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.125+0000 7fed64789640 1 -- 172.21.15.83:0/3044752761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed5c1efc70 con 0x7fed5c06d3e0 2024-09-19T07:38:52.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.125+0000 7fed64789640 1 -- 172.21.15.83:0/3044752761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed5c153560 con 0x7fed5c06d3e0 2024-09-19T07:38:52.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.125+0000 7fed52ffd640 1 -- 172.21.15.83:0/3044752761 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fed5801b070 con 0x7fed5c06d3e0 2024-09-19T07:38:52.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.125+0000 7fed52ffd640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed400678c0 0x7fed40069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:52.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.125+0000 7fed52ffd640 1 -- 172.21.15.83:0/3044752761 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8959+0+0 (secure 0 0 0) 0x7fed58085cd0 con 0x7fed5c06d3e0 2024-09-19T07:38:52.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.125+0000 7fed61d24640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed400678c0 0x7fed40069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:52.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.125+0000 7fed61d24640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed400678c0 0x7fed40069d80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fed4c006fd0 tx=0x7fed4c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.129+0000 7fed52ffd640 1 -- 172.21.15.83:0/3044752761 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fed58055ba0 con 0x7fed5c06d3e0 2024-09-19T07:38:52.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.345+0000 7fed64789640 1 -- 172.21.15.83:0/3044752761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed5c0c54d0 con 0x7fed5c06d3e0 2024-09-19T07:38:52.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.345+0000 7fed52ffd640 1 -- 172.21.15.83:0/3044752761 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+6540 (secure 0 0 0) 0x7fed580554a0 con 0x7fed5c06d3e0 2024-09-19T07:38:52.344 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":140,"btime":"2024-09-19T07:38:15:379040+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:52.344 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 140 2024-09-19T07:38:52.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.349+0000 7fed64789640 1 -- 172.21.15.83:0/3044752761 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed400678c0 msgr2=0x7fed40069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:52.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.349+0000 7fed64789640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed400678c0 0x7fed40069d80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fed4c006fd0 tx=0x7fed4c008040 comp rx=0 tx=0).stop 2024-09-19T07:38:52.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.349+0000 7fed64789640 1 -- 172.21.15.83:0/3044752761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed5c06d3e0 msgr2=0x7fed5c15aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:52.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.349+0000 7fed64789640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed5c06d3e0 0x7fed5c15aaf0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fed5800c520 tx=0x7fed5800c9e0 comp rx=0 tx=0).stop 2024-09-19T07:38:52.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.349+0000 7fed64789640 1 -- 172.21.15.83:0/3044752761 shutdown_connections 2024-09-19T07:38:52.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.349+0000 7fed64789640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed400678c0 0x7fed40069d80 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:52.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.349+0000 7fed64789640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed5c150600 0x7fed5c1ed760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:52.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.349+0000 7fed64789640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed5c150220 0x7fed5c157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:52.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.349+0000 7fed64789640 1 --2- 172.21.15.83:0/3044752761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed5c06d3e0 0x7fed5c15aaf0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:52.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.349+0000 7fed64789640 1 -- 172.21.15.83:0/3044752761 >> 172.21.15.83:0/3044752761 conn(0x7fed5c0c48b0 msgr2=0x7fed5c06dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:52.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.349+0000 7fed64789640 1 -- 172.21.15.83:0/3044752761 shutdown_connections 2024-09-19T07:38:52.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:52.349+0000 7fed64789640 1 -- 172.21.15.83:0/3044752761 wait complete. 2024-09-19T07:38:52.364 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:38:54.365 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:54.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.505+0000 7f27ef862640 1 Processor -- start 2024-09-19T07:38:54.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.505+0000 7f27ef862640 1 -- start start 2024-09-19T07:38:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.505+0000 7f27ef862640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27e80bee30 0x7f27e81570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.505+0000 7f27ef862640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f27e80c9680 0x7f27e8157630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.505+0000 7f27ef862640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e8157b70 con 0x7f27e80c6360 2024-09-19T07:38:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.505+0000 7f27ef862640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f27e80c5b90 con 0x7f27e80c9680 2024-09-19T07:38:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.505+0000 7f27ef862640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f27e815a050 con 0x7f27e80bee30 2024-09-19T07:38:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.509+0000 7f27eddff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f27e80c9680 0x7f27e8157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.509+0000 7f27ecdfd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27e80bee30 0x7f27e81570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:54.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.509+0000 7f27eddff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f27e80c9680 0x7f27e8157630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:44396/0 (socket says 172.21.15.83:44396) 2024-09-19T07:38:54.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.509+0000 7f27ed5fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f27e80c6360 0x7f27e8153050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:41062/0 (socket says 172.21.15.83:41062) 2024-09-19T07:38:54.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.509+0000 7f27eddff640 1 -- 172.21.15.83:0/1080307562 learned_addr learned my addr 172.21.15.83:0/1080307562 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:54.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.509+0000 7f27ecdfd640 1 -- 172.21.15.83:0/1080307562 >> v2:172.21.15.184:3300/0 conn(0x7f27e80c9680 msgr2=0x7f27e8157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:54.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.509+0000 7f27ecdfd640 1 --2- 172.21.15.83:0/1080307562 >> v2:172.21.15.184:3300/0 conn(0x7f27e80c9680 0x7f27e8157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:54.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.509+0000 7f27ecdfd640 1 -- 172.21.15.83:0/1080307562 >> v1:172.21.15.184:6789/0 conn(0x7f27e80c6360 legacy=0x7f27e8153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.509+0000 7f27ecdfd640 1 -- 172.21.15.83:0/1080307562 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e806c950 con 0x7f27e80bee30 2024-09-19T07:38:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.509+0000 7f27eddff640 1 --2- 172.21.15.83:0/1080307562 >> v2:172.21.15.184:3300/0 conn(0x7f27e80c9680 0x7f27e8157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:38:54.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ecdfd640 1 --2- 172.21.15.83:0/1080307562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27e80bee30 0x7f27e81570f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f27d80097c0 tx=0x7f27d801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4155ba39c1148abc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:54.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27dffff640 1 -- 172.21.15.83:0/1080307562 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d80b0070 con 0x7f27e80bee30 2024-09-19T07:38:54.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27dffff640 1 -- 172.21.15.83:0/1080307562 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27d80ab600 con 0x7f27e80bee30 2024-09-19T07:38:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27dffff640 1 -- 172.21.15.83:0/1080307562 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d80ab9d0 con 0x7f27e80bee30 2024-09-19T07:38:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 -- 172.21.15.83:0/1080307562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27e80bee30 msgr2=0x7f27e81570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 --2- 172.21.15.83:0/1080307562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27e80bee30 0x7f27e81570f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f27d80097c0 tx=0x7f27d801c410 comp rx=0 tx=0).stop 2024-09-19T07:38:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 -- 172.21.15.83:0/1080307562 shutdown_connections 2024-09-19T07:38:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 --2- 172.21.15.83:0/1080307562 >> v2:172.21.15.184:3300/0 conn(0x7f27e80c9680 0x7f27e8157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 --2- 172.21.15.83:0/1080307562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27e80bee30 0x7f27e81570f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 -- 172.21.15.83:0/1080307562 >> 172.21.15.83:0/1080307562 conn(0x7f27e80722a0 msgr2=0x7f27e80726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 -- 172.21.15.83:0/1080307562 shutdown_connections 2024-09-19T07:38:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 -- 172.21.15.83:0/1080307562 wait complete. 2024-09-19T07:38:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 Processor -- start 2024-09-19T07:38:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 -- start start 2024-09-19T07:38:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27e80bee30 0x7f27e81492c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27e80c6360 0x7f27e814e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27e80c9680 0x7f27e814ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f27e8153760 con 0x7f27e80bee30 2024-09-19T07:38:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f27e8159850 con 0x7f27e80c9680 2024-09-19T07:38:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ef862640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f27e81538e0 con 0x7f27e80c6360 2024-09-19T07:38:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ed5fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27e80bee30 0x7f27e81492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.513+0000 7f27ed5fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27e80bee30 0x7f27e81492c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41812/0 (socket says 172.21.15.83:41812) 2024-09-19T07:38:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ed5fe640 1 -- 172.21.15.83:0/3506508067 learned_addr learned my addr 172.21.15.83:0/3506508067 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ecdfd640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27e80c6360 0x7f27e814e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27eddff640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27e80c9680 0x7f27e814ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ed5fe640 1 -- 172.21.15.83:0/3506508067 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27e80c6360 msgr2=0x7f27e814e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ed5fe640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27e80c6360 0x7f27e814e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ed5fe640 1 -- 172.21.15.83:0/3506508067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27e80c9680 msgr2=0x7f27e814ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ed5fe640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27e80c9680 0x7f27e814ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ed5fe640 1 -- 172.21.15.83:0/3506508067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e81477d0 con 0x7f27e80bee30 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ecdfd640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27e80c6360 0x7f27e814e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ed5fe640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27e80bee30 0x7f27e81492c0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f27e400c370 tx=0x7f27e400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ddffb640 1 -- 172.21.15.83:0/3506508067 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e40074e0 con 0x7f27e80bee30 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ddffb640 1 -- 172.21.15.83:0/3506508067 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e4007680 con 0x7f27e80bee30 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ef862640 1 -- 172.21.15.83:0/3506508067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e81eed00 con 0x7f27e80bee30 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ddffb640 1 -- 172.21.15.83:0/3506508067 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e4004a30 con 0x7f27e80bee30 2024-09-19T07:38:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ef862640 1 -- 172.21.15.83:0/3506508067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e814f850 con 0x7f27e80bee30 2024-09-19T07:38:54.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ef862640 1 -- 172.21.15.83:0/3506508067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27e80c6740 con 0x7f27e80bee30 2024-09-19T07:38:54.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ddffb640 1 -- 172.21.15.83:0/3506508067 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f27e401a020 con 0x7f27e80bee30 2024-09-19T07:38:54.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ddffb640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f27cc0678c0 0x7f27cc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:54.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ddffb640 1 -- 172.21.15.83:0/3506508067 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8959+0+0 (secure 0 0 0) 0x7f27e4086070 con 0x7f27e80bee30 2024-09-19T07:38:54.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ecdfd640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f27cc0678c0 0x7f27cc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:54.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.517+0000 7f27ecdfd640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f27cc0678c0 0x7f27cc069d80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f27d80ad000 tx=0x7f27d8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:54.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.525+0000 7f27ddffb640 1 -- 172.21.15.83:0/3506508067 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f27e4055f40 con 0x7f27e80bee30 2024-09-19T07:38:54.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.761+0000 7f27ef862640 1 -- 172.21.15.83:0/3506508067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27e80b16a0 con 0x7f27e80bee30 2024-09-19T07:38:54.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ddffb640 1 -- 172.21.15.83:0/3506508067 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+6540 (secure 0 0 0) 0x7f27e4055840 con 0x7f27e80bee30 2024-09-19T07:38:54.761 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:54.761 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":140,"btime":"2024-09-19T07:38:15:379040+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:54.761 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 140 2024-09-19T07:38:54.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ef862640 1 -- 172.21.15.83:0/3506508067 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f27cc0678c0 msgr2=0x7f27cc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:54.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ef862640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f27cc0678c0 0x7f27cc069d80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f27d80ad000 tx=0x7f27d8008040 comp rx=0 tx=0).stop 2024-09-19T07:38:54.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ef862640 1 -- 172.21.15.83:0/3506508067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27e80bee30 msgr2=0x7f27e81492c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:54.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ef862640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27e80bee30 0x7f27e81492c0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f27e400c370 tx=0x7f27e400c830 comp rx=0 tx=0).stop 2024-09-19T07:38:54.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ef862640 1 -- 172.21.15.83:0/3506508067 shutdown_connections 2024-09-19T07:38:54.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ef862640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f27cc0678c0 0x7f27cc069d80 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:54.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ef862640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27e80c9680 0x7f27e814ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:54.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ef862640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27e80c6360 0x7f27e814e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:54.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ef862640 1 --2- 172.21.15.83:0/3506508067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27e80bee30 0x7f27e81492c0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:54.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ef862640 1 -- 172.21.15.83:0/3506508067 >> 172.21.15.83:0/3506508067 conn(0x7f27e80722a0 msgr2=0x7f27e80ab4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:54.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ef862640 1 -- 172.21.15.83:0/3506508067 shutdown_connections 2024-09-19T07:38:54.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:54.765+0000 7f27ef862640 1 -- 172.21.15.83:0/3506508067 wait complete. 2024-09-19T07:38:54.780 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:38:56.781 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:56.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.917+0000 7fd90ce98640 1 Processor -- start 2024-09-19T07:38:56.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd90ce98640 1 -- start start 2024-09-19T07:38:56.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd90ce98640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9080c2c30 0x7fd908152e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:56.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd90ce98640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd9080aaa60 0x7fd9081533d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:56.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd90ce98640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd908159c60 con 0x7fd90806f070 2024-09-19T07:38:56.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd90ce98640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd90815a160 con 0x7fd9080aaa60 2024-09-19T07:38:56.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd90ce98640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd9080c4940 con 0x7fd9080c2c30 2024-09-19T07:38:56.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd905d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd9080aaa60 0x7fd9081533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:56.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd905d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd9080aaa60 0x7fd9081533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41824/0 (socket says 172.21.15.83:41824) 2024-09-19T07:38:56.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd905d9b640 1 -- 172.21.15.83:0/1188670594 learned_addr learned my addr 172.21.15.83:0/1188670594 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:56.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd90659c640 1 --2- 172.21.15.83:0/1188670594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9080c2c30 0x7fd908152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:56.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd905d9b640 1 -- 172.21.15.83:0/1188670594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9080c2c30 msgr2=0x7fd908152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:56.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd905d9b640 1 --2- 172.21.15.83:0/1188670594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9080c2c30 0x7fd908152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:56.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd905d9b640 1 -- 172.21.15.83:0/1188670594 >> v1:172.21.15.184:6789/0 conn(0x7fd90806f070 legacy=0x7fd908155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:56.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd905d9b640 1 -- 172.21.15.83:0/1188670594 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9080c4b90 con 0x7fd9080aaa60 2024-09-19T07:38:56.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.921+0000 7fd905d9b640 1 --2- 172.21.15.83:0/1188670594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9080aaa60 0x7fd9081533d0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fd9080c4ac0 tx=0x7fd8f801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4f0e05c3099c06 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:56.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90559a640 1 -- 172.21.15.83:0/1188670594 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f80ab620 con 0x7fd9080aaa60 2024-09-19T07:38:56.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90559a640 1 -- 172.21.15.83:0/1188670594 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8f80b0070 con 0x7fd9080aaa60 2024-09-19T07:38:56.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90559a640 1 -- 172.21.15.83:0/1188670594 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f80aba20 con 0x7fd9080aaa60 2024-09-19T07:38:56.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 -- 172.21.15.83:0/1188670594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9080aaa60 msgr2=0x7fd9081533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:56.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 --2- 172.21.15.83:0/1188670594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9080aaa60 0x7fd9081533d0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fd9080c4ac0 tx=0x7fd8f801c460 comp rx=0 tx=0).stop 2024-09-19T07:38:56.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 -- 172.21.15.83:0/1188670594 shutdown_connections 2024-09-19T07:38:56.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 --2- 172.21.15.83:0/1188670594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9080aaa60 0x7fd9081533d0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:56.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 --2- 172.21.15.83:0/1188670594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9080c2c30 0x7fd908152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:56.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 -- 172.21.15.83:0/1188670594 >> 172.21.15.83:0/1188670594 conn(0x7fd9080c95a0 msgr2=0x7fd9080c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:56.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 -- 172.21.15.83:0/1188670594 shutdown_connections 2024-09-19T07:38:56.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 -- 172.21.15.83:0/1188670594 wait complete. 2024-09-19T07:38:56.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 Processor -- start 2024-09-19T07:38:56.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 -- start start 2024-09-19T07:38:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd90806f070 0x7fd9081f1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9080aaa60 0x7fd9081f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9080c2c30 0x7fd90814e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd9080c3a90 con 0x7fd9080c2c30 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd908156800 con 0x7fd9080aaa60 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd9080c3c10 con 0x7fd90806f070 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd906d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9080c2c30 0x7fd90814e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd906d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9080c2c30 0x7fd90814e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41830/0 (socket says 172.21.15.83:41830) 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd906d9d640 1 -- 172.21.15.83:0/875155877 learned_addr learned my addr 172.21.15.83:0/875155877 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90659c640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd90806f070 0x7fd9081f1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd905d9b640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9080aaa60 0x7fd9081f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd906d9d640 1 -- 172.21.15.83:0/875155877 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd90806f070 msgr2=0x7fd9081f1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd906d9d640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd90806f070 0x7fd9081f1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd906d9d640 1 -- 172.21.15.83:0/875155877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9080aaa60 msgr2=0x7fd9081f1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd906d9d640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9080aaa60 0x7fd9081f1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd906d9d640 1 -- 172.21.15.83:0/875155877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9080adde0 con 0x7fd9080c2c30 2024-09-19T07:38:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd906d9d640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9080c2c30 0x7fd90814e840 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00c390 tx=0x7fd8fc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd8f6ffd640 1 -- 172.21.15.83:0/875155877 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc011790 con 0x7fd9080c2c30 2024-09-19T07:38:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd8f6ffd640 1 -- 172.21.15.83:0/875155877 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8fc017070 con 0x7fd9080c2c30 2024-09-19T07:38:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.925+0000 7fd90ce98640 1 -- 172.21.15.83:0/875155877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9081f22c0 con 0x7fd9080c2c30 2024-09-19T07:38:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.929+0000 7fd8f6ffd640 1 -- 172.21.15.83:0/875155877 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc011b70 con 0x7fd9080c2c30 2024-09-19T07:38:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.929+0000 7fd90ce98640 1 -- 172.21.15.83:0/875155877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd908157a30 con 0x7fd9080c2c30 2024-09-19T07:38:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.929+0000 7fd90ce98640 1 -- 172.21.15.83:0/875155877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9080c30f0 con 0x7fd9080c2c30 2024-09-19T07:38:56.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.929+0000 7fd8f6ffd640 1 -- 172.21.15.83:0/875155877 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd8fc011d10 con 0x7fd9080c2c30 2024-09-19T07:38:56.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.929+0000 7fd8f6ffd640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8e80678f0 0x7fd8e8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:56.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.929+0000 7fd90659c640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8e80678f0 0x7fd8e8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:56.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.929+0000 7fd8f6ffd640 1 -- 172.21.15.83:0/875155877 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8959+0+0 (secure 0 0 0) 0x7fd8fc086510 con 0x7fd9080c2c30 2024-09-19T07:38:56.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.929+0000 7fd90659c640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8e80678f0 0x7fd8e8069db0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd90000f360 tx=0x7fd900007710 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:56.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:56.933+0000 7fd8f6ffd640 1 -- 172.21.15.83:0/875155877 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd8fc021020 con 0x7fd9080c2c30 2024-09-19T07:38:57.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.153+0000 7fd90ce98640 1 -- 172.21.15.83:0/875155877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9080c6810 con 0x7fd9080c2c30 2024-09-19T07:38:57.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.153+0000 7fd8f6ffd640 1 -- 172.21.15.83:0/875155877 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+6540 (secure 0 0 0) 0x7fd8fc0562c0 con 0x7fd9080c2c30 2024-09-19T07:38:57.149 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:57.149 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":140,"btime":"2024-09-19T07:38:15:379040+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:57.150 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 140 2024-09-19T07:38:57.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.153+0000 7fd90ce98640 1 -- 172.21.15.83:0/875155877 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8e80678f0 msgr2=0x7fd8e8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:57.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.153+0000 7fd90ce98640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8e80678f0 0x7fd8e8069db0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd90000f360 tx=0x7fd900007710 comp rx=0 tx=0).stop 2024-09-19T07:38:57.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.153+0000 7fd90ce98640 1 -- 172.21.15.83:0/875155877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9080c2c30 msgr2=0x7fd90814e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:57.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.153+0000 7fd90ce98640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9080c2c30 0x7fd90814e840 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00c390 tx=0x7fd8fc00c850 comp rx=0 tx=0).stop 2024-09-19T07:38:57.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.157+0000 7fd90ce98640 1 -- 172.21.15.83:0/875155877 shutdown_connections 2024-09-19T07:38:57.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.157+0000 7fd90ce98640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8e80678f0 0x7fd8e8069db0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:57.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.157+0000 7fd90ce98640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9080c2c30 0x7fd90814e840 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:57.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.157+0000 7fd90ce98640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9080aaa60 0x7fd9081f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:57.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.157+0000 7fd90ce98640 1 --2- 172.21.15.83:0/875155877 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd90806f070 0x7fd9081f1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:57.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.157+0000 7fd90ce98640 1 -- 172.21.15.83:0/875155877 >> 172.21.15.83:0/875155877 conn(0x7fd9080c95a0 msgr2=0x7fd9080c6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:57.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.157+0000 7fd90ce98640 1 -- 172.21.15.83:0/875155877 shutdown_connections 2024-09-19T07:38:57.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:57.157+0000 7fd90ce98640 1 -- 172.21.15.83:0/875155877 wait complete. 2024-09-19T07:38:57.172 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:38:59.173 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:38:59.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd4240f0640 1 Processor -- start 2024-09-19T07:38:59.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd4240f0640 1 -- start start 2024-09-19T07:38:59.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd4240f0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd41c14a500 0x7fd41c0c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd4240f0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd41c154540 0x7fd41c157490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd4240f0640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd41c0c4550 0x7fd41c1579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd4240f0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd41c151750 con 0x7fd41c154540 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd4240f0640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd41c157f10 con 0x7fd41c0c4550 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd4240f0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd41c0c7e90 con 0x7fd41c14a500 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd42168b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd41c154540 0x7fd41c157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd42168b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd41c154540 0x7fd41c157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41846/0 (socket says 172.21.15.83:41846) 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd42168b640 1 -- 172.21.15.83:0/3031208052 learned_addr learned my addr 172.21.15.83:0/3031208052 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd421e8c640 1 --2- 172.21.15.83:0/3031208052 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd41c14a500 0x7fd41c0c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd42268d640 1 --2- 172.21.15.83:0/3031208052 >> v2:172.21.15.184:3300/0 conn(0x7fd41c0c4550 0x7fd41c1579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd42168b640 1 -- 172.21.15.83:0/3031208052 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd41c14a500 msgr2=0x7fd41c0c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd42168b640 1 --2- 172.21.15.83:0/3031208052 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd41c14a500 0x7fd41c0c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd42168b640 1 -- 172.21.15.83:0/3031208052 >> v2:172.21.15.184:3300/0 conn(0x7fd41c0c4550 msgr2=0x7fd41c1579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:38:59.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd42168b640 1 --2- 172.21.15.83:0/3031208052 >> v2:172.21.15.184:3300/0 conn(0x7fd41c0c4550 0x7fd41c1579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.325+0000 7fd42168b640 1 -- 172.21.15.83:0/3031208052 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41c073110 con 0x7fd41c154540 2024-09-19T07:38:59.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd42168b640 1 --2- 172.21.15.83:0/3031208052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd41c154540 0x7fd41c157490 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fd41c0c3c10 tx=0x7fd40c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba27d76e32df076 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:59.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd420e8a640 1 -- 172.21.15.83:0/3031208052 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40c0aa610 con 0x7fd41c154540 2024-09-19T07:38:59.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd420e8a640 1 -- 172.21.15.83:0/3031208052 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd40c0af070 con 0x7fd41c154540 2024-09-19T07:38:59.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd420e8a640 1 -- 172.21.15.83:0/3031208052 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40c0aa9d0 con 0x7fd41c154540 2024-09-19T07:38:59.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 -- 172.21.15.83:0/3031208052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd41c154540 msgr2=0x7fd41c157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:59.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 --2- 172.21.15.83:0/3031208052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd41c154540 0x7fd41c157490 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fd41c0c3c10 tx=0x7fd40c01c4f0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 -- 172.21.15.83:0/3031208052 shutdown_connections 2024-09-19T07:38:59.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 --2- 172.21.15.83:0/3031208052 >> v2:172.21.15.184:3300/0 conn(0x7fd41c0c4550 0x7fd41c1579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 --2- 172.21.15.83:0/3031208052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd41c154540 0x7fd41c157490 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 --2- 172.21.15.83:0/3031208052 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd41c14a500 0x7fd41c0c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 -- 172.21.15.83:0/3031208052 >> 172.21.15.83:0/3031208052 conn(0x7fd41c0c9610 msgr2=0x7fd41c0c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:59.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 -- 172.21.15.83:0/3031208052 shutdown_connections 2024-09-19T07:38:59.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 -- 172.21.15.83:0/3031208052 wait complete. 2024-09-19T07:38:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 Processor -- start 2024-09-19T07:38:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 -- start start 2024-09-19T07:38:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd41c0c4550 0x7fd41c1f03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd41c14a500 0x7fd41c14f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd41c154540 0x7fd41c143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c0c76d0 con 0x7fd41c0c4550 2024-09-19T07:38:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c0741f0 con 0x7fd41c154540 2024-09-19T07:38:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd4240f0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd41c0c7850 con 0x7fd41c14a500 2024-09-19T07:38:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd421e8c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd41c0c4550 0x7fd41c1f03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd421e8c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd41c0c4550 0x7fd41c1f03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41852/0 (socket says 172.21.15.83:41852) 2024-09-19T07:38:59.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.329+0000 7fd421e8c640 1 -- 172.21.15.83:0/1646971492 learned_addr learned my addr 172.21.15.83:0/1646971492 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:38:59.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd42168b640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd41c14a500 0x7fd41c14f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:59.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd42268d640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd41c154540 0x7fd41c143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:59.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd421e8c640 1 -- 172.21.15.83:0/1646971492 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd41c14a500 msgr2=0x7fd41c14f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:59.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd421e8c640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd41c14a500 0x7fd41c14f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd421e8c640 1 -- 172.21.15.83:0/1646971492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd41c154540 msgr2=0x7fd41c143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:59.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd421e8c640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd41c154540 0x7fd41c143d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd421e8c640 1 -- 172.21.15.83:0/1646971492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41c14e0f0 con 0x7fd41c0c4550 2024-09-19T07:38:59.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd421e8c640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd41c0c4550 0x7fd41c1f03d0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fd41800d800 tx=0x7fd41800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:59.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd4127fc640 1 -- 172.21.15.83:0/1646971492 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41800b660 con 0x7fd41c0c4550 2024-09-19T07:38:59.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd4127fc640 1 -- 172.21.15.83:0/1646971492 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd418019070 con 0x7fd41c0c4550 2024-09-19T07:38:59.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd4240f0640 1 -- 172.21.15.83:0/1646971492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd41c1473d0 con 0x7fd41c0c4550 2024-09-19T07:38:59.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd4127fc640 1 -- 172.21.15.83:0/1646971492 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41800ba00 con 0x7fd41c0c4550 2024-09-19T07:38:59.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd4240f0640 1 -- 172.21.15.83:0/1646971492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd41c147990 con 0x7fd41c0c4550 2024-09-19T07:38:59.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd4240f0640 1 -- 172.21.15.83:0/1646971492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd41c0c4010 con 0x7fd41c0c4550 2024-09-19T07:38:59.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd4127fc640 1 -- 172.21.15.83:0/1646971492 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd41800bba0 con 0x7fd41c0c4550 2024-09-19T07:38:59.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd4127fc640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4000678c0 0x7fd400069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:38:59.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd42168b640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4000678c0 0x7fd400069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:38:59.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd4127fc640 1 -- 172.21.15.83:0/1646971492 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8959+0+0 (secure 0 0 0) 0x7fd418086dc0 con 0x7fd41c0c4550 2024-09-19T07:38:59.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.333+0000 7fd42168b640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4000678c0 0x7fd400069d80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd40c0ae000 tx=0x7fd40c0a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:38:59.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.341+0000 7fd4127fc640 1 -- 172.21.15.83:0/1646971492 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd418056c90 con 0x7fd41c0c4550 2024-09-19T07:38:59.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.597+0000 7fd4240f0640 1 -- 172.21.15.83:0/1646971492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd41c0c9090 con 0x7fd41c0c4550 2024-09-19T07:38:59.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.597+0000 7fd4127fc640 1 -- 172.21.15.83:0/1646971492 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+6540 (secure 0 0 0) 0x7fd418056590 con 0x7fd41c0c4550 2024-09-19T07:38:59.594 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:38:59.594 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":140,"btime":"2024-09-19T07:38:15:379040+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:38:59.595 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 140 2024-09-19T07:38:59.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.601+0000 7fd4240f0640 1 -- 172.21.15.83:0/1646971492 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4000678c0 msgr2=0x7fd400069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:59.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.601+0000 7fd4240f0640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4000678c0 0x7fd400069d80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd40c0ae000 tx=0x7fd40c0a7000 comp rx=0 tx=0).stop 2024-09-19T07:38:59.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.601+0000 7fd4240f0640 1 -- 172.21.15.83:0/1646971492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd41c0c4550 msgr2=0x7fd41c1f03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:38:59.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.601+0000 7fd4240f0640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd41c0c4550 0x7fd41c1f03d0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fd41800d800 tx=0x7fd41800dcc0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.601+0000 7fd4240f0640 1 -- 172.21.15.83:0/1646971492 shutdown_connections 2024-09-19T07:38:59.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.601+0000 7fd4240f0640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4000678c0 0x7fd400069d80 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.601+0000 7fd4240f0640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd41c154540 0x7fd41c143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.601+0000 7fd4240f0640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd41c14a500 0x7fd41c14f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.601+0000 7fd4240f0640 1 --2- 172.21.15.83:0/1646971492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd41c0c4550 0x7fd41c1f03d0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:38:59.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.601+0000 7fd4240f0640 1 -- 172.21.15.83:0/1646971492 >> 172.21.15.83:0/1646971492 conn(0x7fd41c0c9610 msgr2=0x7fd41c0c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:38:59.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.601+0000 7fd4240f0640 1 -- 172.21.15.83:0/1646971492 shutdown_connections 2024-09-19T07:38:59.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:38:59.601+0000 7fd4240f0640 1 -- 172.21.15.83:0/1646971492 wait complete. 2024-09-19T07:38:59.615 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:39:01.616 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:39:01.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.749+0000 7fcc22e29640 1 Processor -- start 2024-09-19T07:39:01.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.749+0000 7fcc22e29640 1 -- start start 2024-09-19T07:39:01.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc22e29640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcc1c0c4450 0x7fcc1c1594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:01.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc22e29640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c06f3f0 0x7fcc1c14c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:01.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc22e29640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc1c155ee0 con 0x7fcc1c0c3a20 2024-09-19T07:39:01.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc22e29640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fcc1c068290 con 0x7fcc1c0c4450 2024-09-19T07:39:01.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc22e29640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc1c0684e0 con 0x7fcc1c06f3f0 2024-09-19T07:39:01.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc20bc5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcc1c0c3a20 0x7fcc1c1566c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39184/0 (socket says 172.21.15.83:39184) 2024-09-19T07:39:01.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc20bc5640 1 -- 172.21.15.83:0/2393174640 learned_addr learned my addr 172.21.15.83:0/2393174640 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:39:01.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc213c6640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c06f3f0 0x7fcc1c14c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:01.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1bfff640 1 --2- 172.21.15.83:0/2393174640 >> v2:172.21.15.184:3300/0 conn(0x7fcc1c0c4450 0x7fcc1c1594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:01.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1025312400 0 0) 0x7fcc1c155ee0 con 0x7fcc1c0c3a20 2024-09-19T07:39:01.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc0c003530 con 0x7fcc1c0c3a20 2024-09-19T07:39:01.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1120431159 0 0) 0x7fcc0c003530 con 0x7fcc1c0c3a20 2024-09-19T07:39:01.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c06f3f0 msgr2=0x7fcc1c14c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:39:01.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c06f3f0 0x7fcc1c14c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:01.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 >> v2:172.21.15.184:3300/0 conn(0x7fcc1c0c4450 msgr2=0x7fcc1c1594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:39:01.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 --2- 172.21.15.83:0/2393174640 >> v2:172.21.15.184:3300/0 conn(0x7fcc1c0c4450 0x7fcc1c1594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:01.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1c0687c0 con 0x7fcc1c0c3a20 2024-09-19T07:39:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fcc10001680 con 0x7fcc1c0c3a20 2024-09-19T07:39:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 >> v1:172.21.15.83:6789/0 conn(0x7fcc1c0c3a20 legacy=0x7fcc1c1566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0039c0 0x7fcc0c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c06f3f0 0x7fcc0c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc0c004d50 0x7fcc0c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c068290 con 0x7fcc0c0039c0 2024-09-19T07:39:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c0684e0 con 0x7fcc0c004d50 2024-09-19T07:39:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc0c00ba00 con 0x7fcc1c06f3f0 2024-09-19T07:39:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc20bc5640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0039c0 0x7fcc0c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc213c6640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc0c004d50 0x7fcc0c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1bfff640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c06f3f0 0x7fcc0c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc20bc5640 1 -- 172.21.15.83:0/2393174640 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c06f3f0 msgr2=0x7fcc0c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:39:01.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc20bc5640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c06f3f0 0x7fcc0c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:01.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc20bc5640 1 -- 172.21.15.83:0/2393174640 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc0c004d50 msgr2=0x7fcc0c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:39:01.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc20bc5640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc0c004d50 0x7fcc0c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:01.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc20bc5640 1 -- 172.21.15.83:0/2393174640 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0c003530 con 0x7fcc0c0039c0 2024-09-19T07:39:01.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.753+0000 7fcc1bfff640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c06f3f0 0x7fcc0c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:39:01.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc20bc5640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0039c0 0x7fcc0c003de0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000c240 tx=0x7fcc1001cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc29e7ef160a424 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:01.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10007d40 con 0x7fcc0c0039c0 2024-09-19T07:39:01.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc100a9020 con 0x7fcc0c0039c0 2024-09-19T07:39:01.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc1b7fe640 1 -- 172.21.15.83:0/2393174640 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100a5b70 con 0x7fcc0c0039c0 2024-09-19T07:39:01.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 -- 172.21.15.83:0/2393174640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0039c0 msgr2=0x7fcc0c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:01.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 --2- 172.21.15.83:0/2393174640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0039c0 0x7fcc0c003de0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000c240 tx=0x7fcc1001cc70 comp rx=0 tx=0).stop 2024-09-19T07:39:01.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc213c6640 1 -- 172.21.15.83:0/2393174640 reap_dead start 2024-09-19T07:39:01.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 -- 172.21.15.83:0/2393174640 shutdown_connections 2024-09-19T07:39:01.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 -- 172.21.15.83:0/2393174640 >> 172.21.15.83:0/2393174640 conn(0x7fcc1c0c48b0 msgr2=0x7fcc1c0c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:39:01.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 -- 172.21.15.83:0/2393174640 shutdown_connections 2024-09-19T07:39:01.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 -- 172.21.15.83:0/2393174640 wait complete. 2024-09-19T07:39:01.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 Processor -- start 2024-09-19T07:39:01.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 -- start start 2024-09-19T07:39:01.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc1c1586b0 0x7fcc1c0aa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:01.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c072430 0x7fcc1c149680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:01.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc1c1f1fd0 0x7fcc1c142850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:01.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c0c3e00 con 0x7fcc1c1f1fd0 2024-09-19T07:39:01.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c06fcd0 con 0x7fcc1c1586b0 2024-09-19T07:39:01.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc22e29640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc1c0c3f80 con 0x7fcc1c072430 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc213c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc1c1f1fd0 0x7fcc1c142850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc213c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc1c1f1fd0 0x7fcc1c142850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55634/0 (socket says 172.21.15.83:55634) 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc213c6640 1 -- 172.21.15.83:0/2894181204 learned_addr learned my addr 172.21.15.83:0/2894181204 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc1bfff640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c072430 0x7fcc1c149680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc20bc5640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc1c1586b0 0x7fcc1c0aa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc213c6640 1 -- 172.21.15.83:0/2894181204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c072430 msgr2=0x7fcc1c149680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc213c6640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c072430 0x7fcc1c149680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc213c6640 1 -- 172.21.15.83:0/2894181204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc1c1586b0 msgr2=0x7fcc1c0aa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc213c6640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc1c1586b0 0x7fcc1c0aa4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.757+0000 7fcc213c6640 1 -- 172.21.15.83:0/2894181204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1c149bc0 con 0x7fcc1c1f1fd0 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.761+0000 7fcc213c6640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc1c1f1fd0 0x7fcc1c142850 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c06db10 tx=0x7fcc140115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.761+0000 7fcc197fa640 1 -- 172.21.15.83:0/2894181204 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14007e60 con 0x7fcc1c1f1fd0 2024-09-19T07:39:01.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.761+0000 7fcc197fa640 1 -- 172.21.15.83:0/2894181204 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc1401b070 con 0x7fcc1c1f1fd0 2024-09-19T07:39:01.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.761+0000 7fcc22e29640 1 -- 172.21.15.83:0/2894181204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1c148ec0 con 0x7fcc1c1f1fd0 2024-09-19T07:39:01.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.761+0000 7fcc197fa640 1 -- 172.21.15.83:0/2894181204 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14011e40 con 0x7fcc1c1f1fd0 2024-09-19T07:39:01.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.761+0000 7fcc22e29640 1 -- 172.21.15.83:0/2894181204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc1c0aab10 con 0x7fcc1c1f1fd0 2024-09-19T07:39:01.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.761+0000 7fcc22e29640 1 -- 172.21.15.83:0/2894181204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc1c1594d0 con 0x7fcc1c1f1fd0 2024-09-19T07:39:01.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.761+0000 7fcc197fa640 1 -- 172.21.15.83:0/2894181204 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcc140039a0 con 0x7fcc1c1f1fd0 2024-09-19T07:39:01.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.761+0000 7fcc197fa640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc000678f0 0x7fcc00069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:01.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.761+0000 7fcc197fa640 1 -- 172.21.15.83:0/2894181204 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8959+0+0 (secure 0 0 0) 0x7fcc14086f20 con 0x7fcc1c1f1fd0 2024-09-19T07:39:01.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.761+0000 7fcc20bc5640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc000678f0 0x7fcc00069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:01.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.761+0000 7fcc20bc5640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc000678f0 0x7fcc00069db0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcc100aa000 tx=0x7fcc1c06e970 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:01.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:01.765+0000 7fcc197fa640 1 -- 172.21.15.83:0/2894181204 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcc14056d60 con 0x7fcc1c1f1fd0 2024-09-19T07:39:02.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.005+0000 7fcc22e29640 1 -- 172.21.15.83:0/2894181204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc1c153900 con 0x7fcc1c1f1fd0 2024-09-19T07:39:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.005+0000 7fcc197fa640 1 -- 172.21.15.83:0/2894181204 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+6540 (secure 0 0 0) 0x7fcc14056660 con 0x7fcc1c1f1fd0 2024-09-19T07:39:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:39:02.002 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":140,"btime":"2024-09-19T07:38:15:379040+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:37:21.194290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":46,"addr":"172.21.15.184:6833/381468017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":381468017},{"type":"v1","addr":"172.21.15.184:6833","nonce":381468017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5555,"qdb_cluster":[5555]},"id":2}]} 2024-09-19T07:39:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 140 2024-09-19T07:39:02.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.009+0000 7fcc22e29640 1 -- 172.21.15.83:0/2894181204 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc000678f0 msgr2=0x7fcc00069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:02.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.009+0000 7fcc22e29640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc000678f0 0x7fcc00069db0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcc100aa000 tx=0x7fcc1c06e970 comp rx=0 tx=0).stop 2024-09-19T07:39:02.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.009+0000 7fcc22e29640 1 -- 172.21.15.83:0/2894181204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc1c1f1fd0 msgr2=0x7fcc1c142850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.009+0000 7fcc22e29640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc1c1f1fd0 0x7fcc1c142850 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c06db10 tx=0x7fcc140115a0 comp rx=0 tx=0).stop 2024-09-19T07:39:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.009+0000 7fcc22e29640 1 -- 172.21.15.83:0/2894181204 shutdown_connections 2024-09-19T07:39:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.009+0000 7fcc22e29640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc000678f0 0x7fcc00069db0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.009+0000 7fcc22e29640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc1c1f1fd0 0x7fcc1c142850 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.009+0000 7fcc22e29640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc1c072430 0x7fcc1c149680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.009+0000 7fcc22e29640 1 --2- 172.21.15.83:0/2894181204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc1c1586b0 0x7fcc1c0aa4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.009+0000 7fcc22e29640 1 -- 172.21.15.83:0/2894181204 >> 172.21.15.83:0/2894181204 conn(0x7fcc1c0c48b0 msgr2=0x7fcc1c153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:39:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.009+0000 7fcc22e29640 1 -- 172.21.15.83:0/2894181204 shutdown_connections 2024-09-19T07:39:02.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:02.009+0000 7fcc22e29640 1 -- 172.21.15.83:0/2894181204 wait complete. 2024-09-19T07:39:02.024 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:39:04.025 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:39:04.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.148+0000 7fbffae95640 1 Processor -- start 2024-09-19T07:39:04.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.148+0000 7fbffae95640 1 -- start start 2024-09-19T07:39:04.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.148+0000 7fbffae95640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbff41501e0 0x7fbff406ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.148+0000 7fbffae95640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff40c6b70 con 0x7fbff40c79d0 2024-09-19T07:39:04.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.148+0000 7fbffae95640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff406f730 con 0x7fbff406cef0 2024-09-19T07:39:04.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.148+0000 7fbffae95640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbff40c7fb0 con 0x7fbff41501e0 2024-09-19T07:39:04.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.148+0000 7fbff3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbff40c79d0 0x7fbff414fad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39196/0 (socket says 172.21.15.83:39196) 2024-09-19T07:39:04.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.148+0000 7fbff3fff640 1 -- 172.21.15.83:0/2643011853 learned_addr learned my addr 172.21.15.83:0/2643011853 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:39:04.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1592764487 0 0) 0x7fbff40c6b70 con 0x7fbff40c79d0 2024-09-19T07:39:04.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfe4003530 con 0x7fbff40c79d0 2024-09-19T07:39:04.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff9432640 1 --2- 172.21.15.83:0/2643011853 >> v2:172.21.15.184:3300/0 conn(0x7fbff41501e0 0x7fbff406ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2045473138 0 0) 0x7fbfe4003530 con 0x7fbff40c79d0 2024-09-19T07:39:04.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 >> v2:172.21.15.184:3300/0 conn(0x7fbff41501e0 msgr2=0x7fbff406ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:39:04.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 --2- 172.21.15.83:0/2643011853 >> v2:172.21.15.184:3300/0 conn(0x7fbff41501e0 0x7fbff406ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 >> v1:172.21.15.184:6789/0 conn(0x7fbff406cef0 legacy=0x7fbff40c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff406f9f0 con 0x7fbff40c79d0 2024-09-19T07:39:04.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fbfe0001680 con 0x7fbff40c79d0 2024-09-19T07:39:04.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 >> v1:172.21.15.83:6789/0 conn(0x7fbff40c79d0 legacy=0x7fbff414fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 --2- 172.21.15.83:0/2643011853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfe4003a10 0x7fbfe4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 --2- 172.21.15.83:0/2643011853 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfe4004810 0x7fbfe4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 --2- 172.21.15.83:0/2643011853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 0x7fbfe400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbff40c7fb0 con 0x7fbfe40052f0 2024-09-19T07:39:04.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe400e400 con 0x7fbfe4003a10 2024-09-19T07:39:04.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe40059a0 con 0x7fbfe4004810 2024-09-19T07:39:04.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff9432640 1 --2- 172.21.15.83:0/2643011853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 0x7fbfe400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff8c31640 1 --2- 172.21.15.83:0/2643011853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfe4003a10 0x7fbfe4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff3fff640 1 --2- 172.21.15.83:0/2643011853 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfe4004810 0x7fbfe4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff9432640 1 -- 172.21.15.83:0/2643011853 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfe4004810 msgr2=0x7fbfe4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:39:04.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff9432640 1 --2- 172.21.15.83:0/2643011853 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfe4004810 0x7fbfe4004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff9432640 1 -- 172.21.15.83:0/2643011853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfe4003a10 msgr2=0x7fbfe4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:39:04.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff9432640 1 --2- 172.21.15.83:0/2643011853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfe4003a10 0x7fbfe4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff9432640 1 -- 172.21.15.83:0/2643011853 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe4003530 con 0x7fbfe40052f0 2024-09-19T07:39:04.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff9432640 1 -- 172.21.15.83:0/2643011853 reap_dead start 2024-09-19T07:39:04.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff9432640 1 --2- 172.21.15.83:0/2643011853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 0x7fbfe400dbe0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009f90 tx=0x7fbfec01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c3fd4e2949d201 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:04.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec009c10 con 0x7fbfe40052f0 2024-09-19T07:39:04.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfec003ba0 con 0x7fbfe40052f0 2024-09-19T07:39:04.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.152+0000 7fbff37fe640 1 -- 172.21.15.83:0/2643011853 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec0a5770 con 0x7fbfe40052f0 2024-09-19T07:39:04.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 -- 172.21.15.83:0/2643011853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 msgr2=0x7fbfe400dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 --2- 172.21.15.83:0/2643011853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 0x7fbfe400dbe0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009f90 tx=0x7fbfec01cc80 comp rx=0 tx=0).stop 2024-09-19T07:39:04.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 -- 172.21.15.83:0/2643011853 shutdown_connections 2024-09-19T07:39:04.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 --2- 172.21.15.83:0/2643011853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 0x7fbfe400dbe0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 -- 172.21.15.83:0/2643011853 >> 172.21.15.83:0/2643011853 conn(0x7fbff40aa910 msgr2=0x7fbff40c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:39:04.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 -- 172.21.15.83:0/2643011853 shutdown_connections 2024-09-19T07:39:04.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 -- 172.21.15.83:0/2643011853 wait complete. 2024-09-19T07:39:04.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 Processor -- start 2024-09-19T07:39:04.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 -- start start 2024-09-19T07:39:04.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 0x7fbff41498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbff414bd90 0x7fbff414c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbff40c3ad0 0x7fbff40c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbff40c7db0 con 0x7fbfe40052f0 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbff406d4e0 con 0x7fbff40c3ad0 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbffae95640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbff406ddb0 con 0x7fbff414bd90 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbff8c31640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 0x7fbff41498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbff8c31640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 0x7fbff41498f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55664/0 (socket says 172.21.15.83:55664) 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbff8c31640 1 -- 172.21.15.83:0/1095192363 learned_addr learned my addr 172.21.15.83:0/1095192363 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbff8c31640 1 -- 172.21.15.83:0/1095192363 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbff414bd90 msgr2=0x7fbff414c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbff8c31640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbff414bd90 0x7fbff414c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbff8c31640 1 -- 172.21.15.83:0/1095192363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbff40c3ad0 msgr2=0x7fbff40c3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbff9432640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbff40c3ad0 0x7fbff40c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbff8c31640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbff40c3ad0 0x7fbff40c3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbff8c31640 1 -- 172.21.15.83:0/1095192363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff40abf90 con 0x7fbfe40052f0 2024-09-19T07:39:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.156+0000 7fbff8c31640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 0x7fbff41498f0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800e470 tx=0x7fbfe800e870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:04.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.160+0000 7fbff17fa640 1 -- 172.21.15.83:0/1095192363 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8019070 con 0x7fbfe40052f0 2024-09-19T07:39:04.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.160+0000 7fbff17fa640 1 -- 172.21.15.83:0/1095192363 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe80052e0 con 0x7fbfe40052f0 2024-09-19T07:39:04.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.160+0000 7fbffae95640 1 -- 172.21.15.83:0/1095192363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff40ac270 con 0x7fbfe40052f0 2024-09-19T07:39:04.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.160+0000 7fbff17fa640 1 -- 172.21.15.83:0/1095192363 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8003820 con 0x7fbfe40052f0 2024-09-19T07:39:04.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.160+0000 7fbffae95640 1 -- 172.21.15.83:0/1095192363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff40acac0 con 0x7fbfe40052f0 2024-09-19T07:39:04.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.160+0000 7fbffae95640 1 -- 172.21.15.83:0/1095192363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff414fad0 con 0x7fbfe40052f0 2024-09-19T07:39:04.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.160+0000 7fbff17fa640 1 -- 172.21.15.83:0/1095192363 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbfe801c070 con 0x7fbfe40052f0 2024-09-19T07:39:04.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.160+0000 7fbff17fa640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbfd80678c0 0x7fbfd8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.160+0000 7fbff17fa640 1 -- 172.21.15.83:0/1095192363 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 9045+0+0 (secure 0 0 0) 0x7fbfe8085ed0 con 0x7fbfe40052f0 2024-09-19T07:39:04.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.160+0000 7fbff3fff640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbfd80678c0 0x7fbfd8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.160+0000 7fbff3fff640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbfd80678c0 0x7fbfd8069d80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0000fc0 tx=0x7fbfe0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:04.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.164+0000 7fbff17fa640 1 -- 172.21.15.83:0/1095192363 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbfe8055720 con 0x7fbfe40052f0 2024-09-19T07:39:04.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.400+0000 7fbffae95640 1 -- 172.21.15.83:0/1095192363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbff4152830 con 0x7fbfe40052f0 2024-09-19T07:39:04.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.400+0000 7fbff17fa640 1 -- 172.21.15.83:0/1095192363 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+5706 (secure 0 0 0) 0x7fbfe8059a20 con 0x7fbfe40052f0 2024-09-19T07:39:04.399 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:39:04.399 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":141,"btime":"2024-09-19T07:39:03:211182+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:03.211171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:39:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 141 2024-09-19T07:39:04.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.404+0000 7fbffae95640 1 -- 172.21.15.83:0/1095192363 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbfd80678c0 msgr2=0x7fbfd8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.404+0000 7fbffae95640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbfd80678c0 0x7fbfd8069d80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0000fc0 tx=0x7fbfe0009040 comp rx=0 tx=0).stop 2024-09-19T07:39:04.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.404+0000 7fbffae95640 1 -- 172.21.15.83:0/1095192363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 msgr2=0x7fbff41498f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.404+0000 7fbffae95640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 0x7fbff41498f0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800e470 tx=0x7fbfe800e870 comp rx=0 tx=0).stop 2024-09-19T07:39:04.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.404+0000 7fbffae95640 1 -- 172.21.15.83:0/1095192363 shutdown_connections 2024-09-19T07:39:04.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.404+0000 7fbffae95640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbfd80678c0 0x7fbfd8069d80 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.404+0000 7fbffae95640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbff40c3ad0 0x7fbff40c3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.404+0000 7fbffae95640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbff414bd90 0x7fbff414c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.404+0000 7fbffae95640 1 --2- 172.21.15.83:0/1095192363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfe40052f0 0x7fbff41498f0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.404+0000 7fbffae95640 1 -- 172.21.15.83:0/1095192363 >> 172.21.15.83:0/1095192363 conn(0x7fbff40aa910 msgr2=0x7fbff4152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:39:04.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.404+0000 7fbffae95640 1 -- 172.21.15.83:0/1095192363 shutdown_connections 2024-09-19T07:39:04.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.404+0000 7fbffae95640 1 -- 172.21.15.83:0/1095192363 wait complete. 2024-09-19T07:39:04.422 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-19T07:39:04.422 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:39:04.423 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:39:04.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.544+0000 7f94242a4640 1 Processor -- start 2024-09-19T07:39:04.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.544+0000 7f94242a4640 1 -- start start 2024-09-19T07:39:04.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.544+0000 7f94242a4640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f941c06ffe0 0x7f941c0c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.544+0000 7f94242a4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f941c159af0 0x7f941c0c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.544+0000 7f94242a4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f941c0c26c0 con 0x7f941c1535d0 2024-09-19T07:39:04.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.544+0000 7f94242a4640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f941c0707e0 con 0x7f941c06ffe0 2024-09-19T07:39:04.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.544+0000 7f94242a4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f941c0c2c80 con 0x7f941c159af0 2024-09-19T07:39:04.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.544+0000 7f9422841640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f941c159af0 0x7f941c0c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.544+0000 7f942183f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f941c06ffe0 0x7f941c0c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.544+0000 7f9422841640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f941c159af0 0x7f941c0c2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:36938/0 (socket says 172.21.15.83:36938) 2024-09-19T07:39:04.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.544+0000 7f9422040640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f941c1535d0 0x7f941c155c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:44992/0 (socket says 172.21.15.83:44992) 2024-09-19T07:39:04.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.544+0000 7f9422841640 1 -- 172.21.15.83:0/391199307 learned_addr learned my addr 172.21.15.83:0/391199307 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:39:04.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.548+0000 7f9422841640 1 -- 172.21.15.83:0/391199307 >> v2:172.21.15.184:3300/0 conn(0x7f941c06ffe0 msgr2=0x7f941c0c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:39:04.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.548+0000 7f9422841640 1 --2- 172.21.15.83:0/391199307 >> v2:172.21.15.184:3300/0 conn(0x7f941c06ffe0 0x7f941c0c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.548+0000 7f9422841640 1 -- 172.21.15.83:0/391199307 >> v1:172.21.15.184:6789/0 conn(0x7f941c1535d0 legacy=0x7f941c155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.548+0000 7f9422841640 1 -- 172.21.15.83:0/391199307 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941c0c29a0 con 0x7f941c159af0 2024-09-19T07:39:04.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.548+0000 7f942183f640 1 --2- 172.21.15.83:0/391199307 >> v2:172.21.15.184:3300/0 conn(0x7f941c06ffe0 0x7f941c0c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:39:04.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.548+0000 7f9422841640 1 --2- 172.21.15.83:0/391199307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f941c159af0 0x7f941c0c2f60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9414009e50 tx=0x7f941401c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69472fa25bf5a0b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:04.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.548+0000 7f942103e640 1 -- 172.21.15.83:0/391199307 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94140a56e0 con 0x7f941c159af0 2024-09-19T07:39:04.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.548+0000 7f942103e640 1 -- 172.21.15.83:0/391199307 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94140a5880 con 0x7f941c159af0 2024-09-19T07:39:04.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.548+0000 7f942103e640 1 -- 172.21.15.83:0/391199307 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94140a5c50 con 0x7f941c159af0 2024-09-19T07:39:04.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 -- 172.21.15.83:0/391199307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f941c159af0 msgr2=0x7f941c0c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 --2- 172.21.15.83:0/391199307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f941c159af0 0x7f941c0c2f60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9414009e50 tx=0x7f941401c910 comp rx=0 tx=0).stop 2024-09-19T07:39:04.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 -- 172.21.15.83:0/391199307 shutdown_connections 2024-09-19T07:39:04.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 --2- 172.21.15.83:0/391199307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f941c159af0 0x7f941c0c2f60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 --2- 172.21.15.83:0/391199307 >> v2:172.21.15.184:3300/0 conn(0x7f941c06ffe0 0x7f941c0c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 -- 172.21.15.83:0/391199307 >> 172.21.15.83:0/391199307 conn(0x7f941c0c5060 msgr2=0x7f941c0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:39:04.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 -- 172.21.15.83:0/391199307 shutdown_connections 2024-09-19T07:39:04.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 -- 172.21.15.83:0/391199307 wait complete. 2024-09-19T07:39:04.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 Processor -- start 2024-09-19T07:39:04.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 -- start start 2024-09-19T07:39:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f941c06ffe0 0x7f941c15a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f941c1535d0 0x7f941c0ac190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f941c159af0 0x7f941c0acd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f941c162e90 con 0x7f941c159af0 2024-09-19T07:39:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f941c162d10 con 0x7f941c1535d0 2024-09-19T07:39:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f941c163010 con 0x7f941c06ffe0 2024-09-19T07:39:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9422841640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f941c159af0 0x7f941c0acd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9422841640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f941c159af0 0x7f941c0acd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55680/0 (socket says 172.21.15.83:55680) 2024-09-19T07:39:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9422841640 1 -- 172.21.15.83:0/5176721 learned_addr learned my addr 172.21.15.83:0/5176721 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:39:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f942183f640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f941c1535d0 0x7f941c0ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9422040640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f941c06ffe0 0x7f941c15a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9422841640 1 -- 172.21.15.83:0/5176721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f941c06ffe0 msgr2=0x7f941c15a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9422841640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f941c06ffe0 0x7f941c15a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9422841640 1 -- 172.21.15.83:0/5176721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f941c1535d0 msgr2=0x7f941c0ac190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9422841640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f941c1535d0 0x7f941c0ac190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9422841640 1 -- 172.21.15.83:0/5176721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941c0ab810 con 0x7f941c159af0 2024-09-19T07:39:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9422841640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f941c159af0 0x7f941c0acd30 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f941401a540 tx=0x7f94140196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9412ffd640 1 -- 172.21.15.83:0/5176721 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94140bc020 con 0x7f941c159af0 2024-09-19T07:39:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9412ffd640 1 -- 172.21.15.83:0/5176721 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94140b8670 con 0x7f941c159af0 2024-09-19T07:39:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 -- 172.21.15.83:0/5176721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f941c0ad570 con 0x7f941c159af0 2024-09-19T07:39:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f9412ffd640 1 -- 172.21.15.83:0/5176721 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414009510 con 0x7f941c159af0 2024-09-19T07:39:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.552+0000 7f94242a4640 1 -- 172.21.15.83:0/5176721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f941c0ad270 con 0x7f941c159af0 2024-09-19T07:39:04.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.556+0000 7f94242a4640 1 -- 172.21.15.83:0/5176721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f941c0c6460 con 0x7f941c159af0 2024-09-19T07:39:04.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.556+0000 7f9412ffd640 1 -- 172.21.15.83:0/5176721 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f94140bf020 con 0x7f941c159af0 2024-09-19T07:39:04.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.556+0000 7f9412ffd640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94000678c0 0x7f9400069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:04.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.556+0000 7f9412ffd640 1 -- 172.21.15.83:0/5176721 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 9045+0+0 (secure 0 0 0) 0x7f9414121790 con 0x7f941c159af0 2024-09-19T07:39:04.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.556+0000 7f9422040640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94000678c0 0x7f9400069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:04.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.556+0000 7f9422040640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94000678c0 0x7f9400069d80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9418000f70 tx=0x7f9418007c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:04.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.564+0000 7f9412ffd640 1 -- 172.21.15.83:0/5176721 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f94140c6030 con 0x7f941c159af0 2024-09-19T07:39:04.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.820+0000 7f94242a4640 1 -- 172.21.15.83:0/5176721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f941c0adaf0 con 0x7f941c159af0 2024-09-19T07:39:04.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.820+0000 7f9412ffd640 1 -- 172.21.15.83:0/5176721 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+5710 (secure 0 0 0) 0x7f9414016620 con 0x7f941c159af0 2024-09-19T07:39:04.818 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:39:04.819 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":142,"btime":"2024-09-19T07:39:04:410372+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:03.647156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:39:04.819 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 142 2024-09-19T07:39:04.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.824+0000 7f94242a4640 1 -- 172.21.15.83:0/5176721 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94000678c0 msgr2=0x7f9400069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.824+0000 7f94242a4640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94000678c0 0x7f9400069d80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9418000f70 tx=0x7f9418007c90 comp rx=0 tx=0).stop 2024-09-19T07:39:04.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.824+0000 7f94242a4640 1 -- 172.21.15.83:0/5176721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f941c159af0 msgr2=0x7f941c0acd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:04.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.824+0000 7f94242a4640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f941c159af0 0x7f941c0acd30 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f941401a540 tx=0x7f94140196b0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.824+0000 7f94242a4640 1 -- 172.21.15.83:0/5176721 shutdown_connections 2024-09-19T07:39:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.824+0000 7f94242a4640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94000678c0 0x7f9400069d80 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.824+0000 7f94242a4640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f941c159af0 0x7f941c0acd30 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.824+0000 7f94242a4640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f941c1535d0 0x7f941c0ac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.824+0000 7f94242a4640 1 --2- 172.21.15.83:0/5176721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f941c06ffe0 0x7f941c15a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.824+0000 7f94242a4640 1 -- 172.21.15.83:0/5176721 >> 172.21.15.83:0/5176721 conn(0x7f941c0c5060 msgr2=0x7f941c0c5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:39:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.824+0000 7f94242a4640 1 -- 172.21.15.83:0/5176721 shutdown_connections 2024-09-19T07:39:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:04.824+0000 7f94242a4640 1 -- 172.21.15.83:0/5176721 wait complete. 2024-09-19T07:39:04.838 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:39:04:410372+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 142, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 139, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7266": { "addr": "172.21.15.83:6837/2326757290", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 2326757290, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 2326757290, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7266, "incarnation": 136, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:38:11.364472+0000", "qdb_cluster": [ 7266 ], "qdb_leader": 7266, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7266 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 142, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7350": { "addr": "172.21.15.83:6835/365493751", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 365493751, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 365493751, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7350, "incarnation": 141, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:39:03.647156+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7350 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:39:05.473 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:39:05.476 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:39:06.839 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:39:06.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa3817640 1 Processor -- start 2024-09-19T07:39:06.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa3817640 1 -- start start 2024-09-19T07:39:06.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa3817640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fef9c06ffe0 0x7fef9c0c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:06.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa3817640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef9c159af0 0x7fef9c0c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:06.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa3817640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef9c0c26c0 con 0x7fef9c1535d0 2024-09-19T07:39:06.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa3817640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fef9c0707e0 con 0x7fef9c06ffe0 2024-09-19T07:39:06.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa3817640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef9c0c2c80 con 0x7fef9c159af0 2024-09-19T07:39:06.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa0db2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fef9c06ffe0 0x7fef9c0c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:06.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa1db4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef9c159af0 0x7fef9c0c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:06.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa0db2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fef9c06ffe0 0x7fef9c0c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:40674/0 (socket says 172.21.15.83:40674) 2024-09-19T07:39:06.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa1db4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef9c159af0 0x7fef9c0c2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:36952/0 (socket says 172.21.15.83:36952) 2024-09-19T07:39:06.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa15b3640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fef9c1535d0 0x7fef9c155c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:44996/0 (socket says 172.21.15.83:44996) 2024-09-19T07:39:06.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa0db2640 1 -- 172.21.15.83:0/3103941296 learned_addr learned my addr 172.21.15.83:0/3103941296 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:39:06.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa0db2640 1 -- 172.21.15.83:0/3103941296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef9c159af0 msgr2=0x7fef9c0c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:39:06.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa0db2640 1 --2- 172.21.15.83:0/3103941296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef9c159af0 0x7fef9c0c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:06.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa0db2640 1 -- 172.21.15.83:0/3103941296 >> v1:172.21.15.184:6789/0 conn(0x7fef9c1535d0 legacy=0x7fef9c155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:06.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa0db2640 1 -- 172.21.15.83:0/3103941296 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9c0c29a0 con 0x7fef9c06ffe0 2024-09-19T07:39:06.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.992+0000 7fefa1db4640 1 --2- 172.21.15.83:0/3103941296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef9c159af0 0x7fef9c0c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:39:06.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fefa0db2640 1 --2- 172.21.15.83:0/3103941296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef9c06ffe0 0x7fef9c0c6460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fef8c009870 tx=0x7fef8c01c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d59eff271c8a21e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:06.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fef93fff640 1 -- 172.21.15.83:0/3103941296 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef8c0b0070 con 0x7fef9c06ffe0 2024-09-19T07:39:06.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fef93fff640 1 -- 172.21.15.83:0/3103941296 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef8c0ab600 con 0x7fef9c06ffe0 2024-09-19T07:39:06.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fef93fff640 1 -- 172.21.15.83:0/3103941296 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef8c0ab9d0 con 0x7fef9c06ffe0 2024-09-19T07:39:06.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fefa3817640 1 -- 172.21.15.83:0/3103941296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef9c06ffe0 msgr2=0x7fef9c0c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fefa3817640 1 --2- 172.21.15.83:0/3103941296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef9c06ffe0 0x7fef9c0c6460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fef8c009870 tx=0x7fef8c01c440 comp rx=0 tx=0).stop 2024-09-19T07:39:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fefa3817640 1 -- 172.21.15.83:0/3103941296 shutdown_connections 2024-09-19T07:39:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fefa3817640 1 --2- 172.21.15.83:0/3103941296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef9c159af0 0x7fef9c0c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fefa3817640 1 --2- 172.21.15.83:0/3103941296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef9c06ffe0 0x7fef9c0c6460 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fefa3817640 1 -- 172.21.15.83:0/3103941296 >> 172.21.15.83:0/3103941296 conn(0x7fef9c0c5060 msgr2=0x7fef9c0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:39:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fefa3817640 1 -- 172.21.15.83:0/3103941296 shutdown_connections 2024-09-19T07:39:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fefa3817640 1 -- 172.21.15.83:0/3103941296 wait complete. 2024-09-19T07:39:06.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:06.996+0000 7fefa3817640 1 Processor -- start 2024-09-19T07:39:06.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa3817640 1 -- start start 2024-09-19T07:39:06.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa3817640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef9c06ffe0 0x7fef9c0ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:06.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa3817640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef9c1535d0 0x7fef9c154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:06.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa3817640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef9c159af0 0x7fef9c152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:06.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa3817640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fef9c162e90 con 0x7fef9c06ffe0 2024-09-19T07:39:06.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa3817640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fef9c162d10 con 0x7fef9c159af0 2024-09-19T07:39:06.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa3817640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef9c163010 con 0x7fef9c1535d0 2024-09-19T07:39:06.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa15b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef9c06ffe0 0x7fef9c0ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa15b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef9c06ffe0 0x7fef9c0ac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55688/0 (socket says 172.21.15.83:55688) 2024-09-19T07:39:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa15b3640 1 -- 172.21.15.83:0/3756618379 learned_addr learned my addr 172.21.15.83:0/3756618379 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:39:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa0db2640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef9c1535d0 0x7fef9c154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa1db4640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef9c159af0 0x7fef9c152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa15b3640 1 -- 172.21.15.83:0/3756618379 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef9c1535d0 msgr2=0x7fef9c154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa15b3640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef9c1535d0 0x7fef9c154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa15b3640 1 -- 172.21.15.83:0/3756618379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef9c159af0 msgr2=0x7fef9c152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa15b3640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef9c159af0 0x7fef9c152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa15b3640 1 -- 172.21.15.83:0/3756618379 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9c15a720 con 0x7fef9c06ffe0 2024-09-19T07:39:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa15b3640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef9c06ffe0 0x7fef9c0ac7e0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fef9800c420 tx=0x7fef9800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fef91ffb640 1 -- 172.21.15.83:0/3756618379 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef980074e0 con 0x7fef9c06ffe0 2024-09-19T07:39:07.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa3817640 1 -- 172.21.15.83:0/3756618379 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef9c1f2460 con 0x7fef9c06ffe0 2024-09-19T07:39:07.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fef91ffb640 1 -- 172.21.15.83:0/3756618379 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef98007680 con 0x7fef9c06ffe0 2024-09-19T07:39:07.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fef91ffb640 1 -- 172.21.15.83:0/3756618379 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef98004a30 con 0x7fef9c06ffe0 2024-09-19T07:39:07.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa3817640 1 -- 172.21.15.83:0/3756618379 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef9c1f2a20 con 0x7fef9c06ffe0 2024-09-19T07:39:07.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.000+0000 7fefa3817640 1 -- 172.21.15.83:0/3756618379 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef9c0c6460 con 0x7fef9c06ffe0 2024-09-19T07:39:07.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.004+0000 7fef91ffb640 1 -- 172.21.15.83:0/3756618379 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fef9801a020 con 0x7fef9c06ffe0 2024-09-19T07:39:07.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.004+0000 7fef91ffb640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef800678c0 0x7fef80069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:07.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.004+0000 7fef91ffb640 1 -- 172.21.15.83:0/3756618379 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 9045+0+0 (secure 0 0 0) 0x7fef98086860 con 0x7fef9c06ffe0 2024-09-19T07:39:07.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.004+0000 7fefa0db2640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef800678c0 0x7fef80069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:07.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.004+0000 7fefa0db2640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef800678c0 0x7fef80069d80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0ad000 tx=0x7fef8c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:07.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.008+0000 7fef91ffb640 1 -- 172.21.15.83:0/3756618379 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fef98021070 con 0x7fef9c06ffe0 2024-09-19T07:39:07.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 -- 172.21.15.83:0/3756618379 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef9c0c1ba0 con 0x7fef9c06ffe0 2024-09-19T07:39:07.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fef91ffb640 1 -- 172.21.15.83:0/3756618379 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5714 (secure 0 0 0) 0x7fef98056670 con 0x7fef9c06ffe0 2024-09-19T07:39:07.229 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:39:07.229 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":144,"btime":"2024-09-19T07:39:06:430806+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:39:07.230 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 144 2024-09-19T07:39:07.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 -- 172.21.15.83:0/3756618379 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef800678c0 msgr2=0x7fef80069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:07.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef800678c0 0x7fef80069d80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0ad000 tx=0x7fef8c008040 comp rx=0 tx=0).stop 2024-09-19T07:39:07.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 -- 172.21.15.83:0/3756618379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef9c06ffe0 msgr2=0x7fef9c0ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:07.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef9c06ffe0 0x7fef9c0ac7e0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fef9800c420 tx=0x7fef9800c820 comp rx=0 tx=0).stop 2024-09-19T07:39:07.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 -- 172.21.15.83:0/3756618379 shutdown_connections 2024-09-19T07:39:07.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef800678c0 0x7fef80069d80 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:07.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef9c159af0 0x7fef9c152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:07.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef9c1535d0 0x7fef9c154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:07.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 --2- 172.21.15.83:0/3756618379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef9c06ffe0 0x7fef9c0ac7e0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:07.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 -- 172.21.15.83:0/3756618379 >> 172.21.15.83:0/3756618379 conn(0x7fef9c0c5060 msgr2=0x7fef9c0c7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:39:07.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 -- 172.21.15.83:0/3756618379 shutdown_connections 2024-09-19T07:39:07.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:07.232+0000 7fefa3817640 1 -- 172.21.15.83:0/3756618379 wait complete. 2024-09-19T07:39:07.249 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5555 2024-09-19T07:39:07.250 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before reviving mds.d 2024-09-19T07:39:10.387 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:39:10.388+0000 7f5e9821f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:39:11.250 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-19T07:39:11.251 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-19T07:39:11.251 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-19T07:39:11.253 INFO:tasks.ceph.mds.d:Started 2024-09-19T07:39:11.254 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:39:11.335 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-19T07:39:11.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.400+0000 7f991cde9640 1 Processor -- start 2024-09-19T07:39:11.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.400+0000 7f991cde9640 1 -- start start 2024-09-19T07:39:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991cde9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f991806cef0 0x7f99180abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991cde9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180ab610 0x7f99180c3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991cde9640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f99180c6fc0 0x7f9918157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991cde9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f99180c6a90 con 0x7f991806cef0 2024-09-19T07:39:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991cde9640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f99180c8360 con 0x7f99180c6fc0 2024-09-19T07:39:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991cde9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f99180c7de0 con 0x7f99180ab610 2024-09-19T07:39:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f991806cef0 0x7f99180abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f991806cef0 0x7f99180abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46016/0 (socket says 172.21.15.83:46016) 2024-09-19T07:39:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991659c640 1 -- 172.21.15.83:0/1087380262 learned_addr learned my addr 172.21.15.83:0/1087380262 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:39:11.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f9916d9d640 1 --2- 172.21.15.83:0/1087380262 >> v2:172.21.15.184:3300/0 conn(0x7f99180c6fc0 0x7f9918157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:11.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f990ffff640 1 --2- 172.21.15.83:0/1087380262 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180ab610 0x7f99180c3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:11.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991659c640 1 -- 172.21.15.83:0/1087380262 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180ab610 msgr2=0x7f99180c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:39:11.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991659c640 1 --2- 172.21.15.83:0/1087380262 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180ab610 0x7f99180c3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:11.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991659c640 1 -- 172.21.15.83:0/1087380262 >> v2:172.21.15.184:3300/0 conn(0x7f99180c6fc0 msgr2=0x7f9918157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:39:11.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991659c640 1 --2- 172.21.15.83:0/1087380262 >> v2:172.21.15.184:3300/0 conn(0x7f99180c6fc0 0x7f9918157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:11.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.404+0000 7f991659c640 1 -- 172.21.15.83:0/1087380262 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99180c8080 con 0x7f991806cef0 2024-09-19T07:39:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991659c640 1 --2- 172.21.15.83:0/1087380262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f991806cef0 0x7f99180abe80 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f991806fe00 tx=0x7f990801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd10395a940d467e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f9915d9b640 1 -- 172.21.15.83:0/1087380262 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99080b1070 con 0x7f991806cef0 2024-09-19T07:39:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f9915d9b640 1 -- 172.21.15.83:0/1087380262 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99080ac410 con 0x7f991806cef0 2024-09-19T07:39:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f9915d9b640 1 -- 172.21.15.83:0/1087380262 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99080ac7e0 con 0x7f991806cef0 2024-09-19T07:39:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 -- 172.21.15.83:0/1087380262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f991806cef0 msgr2=0x7f99180abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 --2- 172.21.15.83:0/1087380262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f991806cef0 0x7f99180abe80 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f991806fe00 tx=0x7f990801c3d0 comp rx=0 tx=0).stop 2024-09-19T07:39:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 -- 172.21.15.83:0/1087380262 shutdown_connections 2024-09-19T07:39:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 --2- 172.21.15.83:0/1087380262 >> v2:172.21.15.184:3300/0 conn(0x7f99180c6fc0 0x7f9918157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 --2- 172.21.15.83:0/1087380262 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180ab610 0x7f99180c3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 --2- 172.21.15.83:0/1087380262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f991806cef0 0x7f99180abe80 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 -- 172.21.15.83:0/1087380262 >> 172.21.15.83:0/1087380262 conn(0x7f99180aa910 msgr2=0x7f99180c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:39:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 -- 172.21.15.83:0/1087380262 shutdown_connections 2024-09-19T07:39:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 -- 172.21.15.83:0/1087380262 wait complete. 2024-09-19T07:39:11.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 Processor -- start 2024-09-19T07:39:11.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 -- start start 2024-09-19T07:39:11.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f991806cef0 0x7f99181ef510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:11.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180ab610 0x7f9918073c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:11.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99180c6fc0 0x7f991814b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:11.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f991806d2d0 con 0x7f99180c6fc0 2024-09-19T07:39:11.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f99181581e0 con 0x7f991806cef0 2024-09-19T07:39:11.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991cde9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f991806d450 con 0x7f99180ab610 2024-09-19T07:39:11.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f9916d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99180c6fc0 0x7f991814b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:11.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f9916d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99180c6fc0 0x7f991814b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46028/0 (socket says 172.21.15.83:46028) 2024-09-19T07:39:11.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f9916d9d640 1 -- 172.21.15.83:0/3090235489 learned_addr learned my addr 172.21.15.83:0/3090235489 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:39:11.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f991659c640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f991806cef0 0x7f99181ef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:11.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f990ffff640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180ab610 0x7f9918073c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:11.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f9916d9d640 1 -- 172.21.15.83:0/3090235489 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180ab610 msgr2=0x7f9918073c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:11.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f9916d9d640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180ab610 0x7f9918073c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:11.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f9916d9d640 1 -- 172.21.15.83:0/3090235489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f991806cef0 msgr2=0x7f99181ef510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:11.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f9916d9d640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f991806cef0 0x7f99181ef510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:11.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.408+0000 7f9916d9d640 1 -- 172.21.15.83:0/3090235489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9918154ca0 con 0x7f99180c6fc0 2024-09-19T07:39:11.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.412+0000 7f9916d9d640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99180c6fc0 0x7f991814b2f0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f991000efc0 tx=0x7f991000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:11.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.412+0000 7f990effd640 1 -- 172.21.15.83:0/3090235489 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9910018020 con 0x7f99180c6fc0 2024-09-19T07:39:11.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.412+0000 7f990effd640 1 -- 172.21.15.83:0/3090235489 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9910014480 con 0x7f99180c6fc0 2024-09-19T07:39:11.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.412+0000 7f991cde9640 1 -- 172.21.15.83:0/3090235489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99180a9140 con 0x7f99180c6fc0 2024-09-19T07:39:11.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.412+0000 7f990effd640 1 -- 172.21.15.83:0/3090235489 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9910014870 con 0x7f99180c6fc0 2024-09-19T07:39:11.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.412+0000 7f991cde9640 1 -- 172.21.15.83:0/3090235489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f991806f650 con 0x7f99180c6fc0 2024-09-19T07:39:11.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.412+0000 7f991cde9640 1 -- 172.21.15.83:0/3090235489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99180c3970 con 0x7f99180c6fc0 2024-09-19T07:39:11.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.412+0000 7f990effd640 1 -- 172.21.15.83:0/3090235489 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f991001b020 con 0x7f99180c6fc0 2024-09-19T07:39:11.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.412+0000 7f990effd640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98f80678c0 0x7f98f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:39:11.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.412+0000 7f990effd640 1 -- 172.21.15.83:0/3090235489 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 9045+0+0 (secure 0 0 0) 0x7f99100872f0 con 0x7f99180c6fc0 2024-09-19T07:39:11.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.412+0000 7f991659c640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98f80678c0 0x7f98f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:39:11.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.412+0000 7f991659c640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98f80678c0 0x7f98f8069d80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f99080ae000 tx=0x7f9908009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:39:11.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.420+0000 7f990effd640 1 -- 172.21.15.83:0/3090235489 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9910022070 con 0x7f99180c6fc0 2024-09-19T07:39:11.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.684+0000 7f991cde9640 1 -- 172.21.15.83:0/3090235489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f991815e3e0 con 0x7f99180c6fc0 2024-09-19T07:39:11.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.684+0000 7f990effd640 1 -- 172.21.15.83:0/3090235489 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+6543 (secure 0 0 0) 0x7f9910010040 con 0x7f99180c6fc0 2024-09-19T07:39:11.684 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:39:11.684 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":145,"btime":"2024-09-19T07:39:11:439402+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5852,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:39:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 145 2024-09-19T07:39:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.688+0000 7f991cde9640 1 -- 172.21.15.83:0/3090235489 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98f80678c0 msgr2=0x7f98f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.688+0000 7f991cde9640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98f80678c0 0x7f98f8069d80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f99080ae000 tx=0x7f9908009040 comp rx=0 tx=0).stop 2024-09-19T07:39:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.688+0000 7f991cde9640 1 -- 172.21.15.83:0/3090235489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99180c6fc0 msgr2=0x7f991814b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:39:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.688+0000 7f991cde9640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99180c6fc0 0x7f991814b2f0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f991000efc0 tx=0x7f991000c490 comp rx=0 tx=0).stop 2024-09-19T07:39:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.688+0000 7f991cde9640 1 -- 172.21.15.83:0/3090235489 shutdown_connections 2024-09-19T07:39:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.688+0000 7f991cde9640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f98f80678c0 0x7f98f8069d80 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.688+0000 7f991cde9640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99180c6fc0 0x7f991814b2f0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.688+0000 7f991cde9640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180ab610 0x7f9918073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.688+0000 7f991cde9640 1 --2- 172.21.15.83:0/3090235489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f991806cef0 0x7f99181ef510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:39:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.688+0000 7f991cde9640 1 -- 172.21.15.83:0/3090235489 >> 172.21.15.83:0/3090235489 conn(0x7f99180aa910 msgr2=0x7f991814f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:39:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.688+0000 7f991cde9640 1 -- 172.21.15.83:0/3090235489 shutdown_connections 2024-09-19T07:39:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:39:11.688+0000 7f991cde9640 1 -- 172.21.15.83:0/3090235489 wait complete. 2024-09-19T07:39:11.704 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-19T07:39:11.705 INFO:tasks.mds_thrash.fs.[b]:waiting for 100 secs before thrashing 2024-09-19T07:39:35.521 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:39:35.524 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:40:04.578 INFO:tasks.workunit.client.0.smithi083.stdout:fs setup took 1117 secs 2024-09-19T07:40:05.568 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:40:05.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:40:06.473 INFO:tasks.workunit.client.0.smithi083.stdout:Syncing()...1 sec 2024-09-19T07:40:06.474 INFO:tasks.workunit.client.0.smithi083.stdout:Starting Actual Benchmark At: Thu Sep 19 07:40:06 2024 2024-09-19T07:40:06.474 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:40:08.520 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:08.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.694+0000 7f1d4a1cf640 1 Processor -- start 2024-09-19T07:40:08.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.694+0000 7f1d4a1cf640 1 -- start start 2024-09-19T07:40:08.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.694+0000 7f1d4a1cf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d440c6350 0x7f1d441544f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:08.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.694+0000 7f1d4a1cf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d440c22e0 con 0x7f1d440c5740 2024-09-19T07:40:08.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.694+0000 7f1d4a1cf640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d440c3b20 con 0x7f1d44157630 2024-09-19T07:40:08.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.694+0000 7f1d4a1cf640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f1d44154a30 con 0x7f1d440c6350 2024-09-19T07:40:08.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.694+0000 7f1d42ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1d440c5740 0x7f1d440c4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54250/0 (socket says 172.21.15.83:54250) 2024-09-19T07:40:08.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.694+0000 7f1d42ffd640 1 -- 172.21.15.83:0/1717602909 learned_addr learned my addr 172.21.15.83:0/1717602909 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:40:08.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.698+0000 7f1d437fe640 1 --2- 172.21.15.83:0/1717602909 >> v2:172.21.15.83:3300/0 conn(0x7f1d440c6350 0x7f1d441544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:08.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.698+0000 7f1d427fc640 1 -- 172.21.15.83:0/1717602909 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1054588202 0 0) 0x7f1d440c3b20 con 0x7f1d44157630 2024-09-19T07:40:08.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.698+0000 7f1d427fc640 1 -- 172.21.15.83:0/1717602909 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d2c003530 con 0x7f1d44157630 2024-09-19T07:40:08.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.698+0000 7f1d437fe640 1 -- 172.21.15.83:0/1717602909 >> v1:172.21.15.184:6789/0 conn(0x7f1d44157630 legacy=0x7f1d4415e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:08.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.698+0000 7f1d437fe640 1 -- 172.21.15.83:0/1717602909 >> v1:172.21.15.83:6789/0 conn(0x7f1d440c5740 legacy=0x7f1d440c4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:08.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.698+0000 7f1d437fe640 1 -- 172.21.15.83:0/1717602909 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d440c2550 con 0x7f1d440c6350 2024-09-19T07:40:08.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.698+0000 7f1d437fe640 1 --2- 172.21.15.83:0/1717602909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d440c6350 0x7f1d441544f0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f1d340097f0 tx=0x7f1d3401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f255fc2f0ceabd4c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:08.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.698+0000 7f1d427fc640 1 -- 172.21.15.83:0/1717602909 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d340b0070 con 0x7f1d440c6350 2024-09-19T07:40:08.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.698+0000 7f1d427fc640 1 -- 172.21.15.83:0/1717602909 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d340ab600 con 0x7f1d440c6350 2024-09-19T07:40:08.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.698+0000 7f1d427fc640 1 -- 172.21.15.83:0/1717602909 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d340ab9d0 con 0x7f1d440c6350 2024-09-19T07:40:08.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.702+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1717602909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d440c6350 msgr2=0x7f1d441544f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:08.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.702+0000 7f1d4a1cf640 1 --2- 172.21.15.83:0/1717602909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d440c6350 0x7f1d441544f0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f1d340097f0 tx=0x7f1d3401c380 comp rx=0 tx=0).stop 2024-09-19T07:40:08.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.702+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1717602909 shutdown_connections 2024-09-19T07:40:08.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.702+0000 7f1d4a1cf640 1 --2- 172.21.15.83:0/1717602909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d440c6350 0x7f1d441544f0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:08.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.702+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1717602909 >> 172.21.15.83:0/1717602909 conn(0x7f1d4406c080 msgr2=0x7f1d4406c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:08.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.702+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1717602909 shutdown_connections 2024-09-19T07:40:08.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.702+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1717602909 wait complete. 2024-09-19T07:40:08.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.702+0000 7f1d4a1cf640 1 Processor -- start 2024-09-19T07:40:08.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.702+0000 7f1d4a1cf640 1 -- start start 2024-09-19T07:40:08.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d4a1cf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d440c5740 0x7f1d44071220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:08.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d4a1cf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d440c6350 0x7f1d44146540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:08.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d4a1cf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d44157630 0x7f1d44155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:08.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d4a1cf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1d440c4e00 con 0x7f1d440c6350 2024-09-19T07:40:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d42ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d440c6350 0x7f1d44146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d437fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d440c5740 0x7f1d44071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d42ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d440c6350 0x7f1d44146540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55168/0 (socket says 172.21.15.83:55168) 2024-09-19T07:40:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d42ffd640 1 -- 172.21.15.83:0/1888919810 learned_addr learned my addr 172.21.15.83:0/1888919810 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1888919810 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1d440c4c80 con 0x7f1d440c5740 2024-09-19T07:40:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1888919810 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1d440c6910 con 0x7f1d44157630 2024-09-19T07:40:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d43fff640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d44157630 0x7f1d44155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d42ffd640 1 -- 172.21.15.83:0/1888919810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d44157630 msgr2=0x7f1d44155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d42ffd640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d44157630 0x7f1d44155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d42ffd640 1 -- 172.21.15.83:0/1888919810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d440c5740 msgr2=0x7f1d44071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d42ffd640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d440c5740 0x7f1d44071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d42ffd640 1 -- 172.21.15.83:0/1888919810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d44147be0 con 0x7f1d440c6350 2024-09-19T07:40:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d42ffd640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d440c6350 0x7f1d44146540 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800c500 tx=0x7f1d3800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d2bfff640 1 -- 172.21.15.83:0/1888919810 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38017070 con 0x7f1d440c6350 2024-09-19T07:40:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1888919810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d44147ec0 con 0x7f1d440c6350 2024-09-19T07:40:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.706+0000 7f1d2bfff640 1 -- 172.21.15.83:0/1888919810 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d38005cf0 con 0x7f1d440c6350 2024-09-19T07:40:08.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.710+0000 7f1d2bfff640 1 -- 172.21.15.83:0/1888919810 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d380072f0 con 0x7f1d440c6350 2024-09-19T07:40:08.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.710+0000 7f1d2bfff640 1 -- 172.21.15.83:0/1888919810 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1d3801a070 con 0x7f1d440c6350 2024-09-19T07:40:08.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.710+0000 7f1d2bfff640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1d30067870 0x7f1d30069d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:08.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.710+0000 7f1d437fe640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1d30067870 0x7f1d30069d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:08.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.710+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1888919810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d44146a80 con 0x7f1d440c6350 2024-09-19T07:40:08.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.710+0000 7f1d2bfff640 1 -- 172.21.15.83:0/1888919810 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 9045+0+0 (secure 0 0 0) 0x7f1d380534c0 con 0x7f1d440c6350 2024-09-19T07:40:08.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.710+0000 7f1d437fe640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1d30067870 0x7f1d30069d30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1d340ad000 tx=0x7f1d340a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:08.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.710+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1888919810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d441544f0 con 0x7f1d440c6350 2024-09-19T07:40:08.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:08.722+0000 7f1d2bfff640 1 -- 172.21.15.83:0/1888919810 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1d38055420 con 0x7f1d440c6350 2024-09-19T07:40:09.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.102+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1888919810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d4406c950 con 0x7f1d440c6350 2024-09-19T07:40:09.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.102+0000 7f1d2bfff640 1 -- 172.21.15.83:0/1888919810 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+6543 (secure 0 0 0) 0x7f1d38059720 con 0x7f1d440c6350 2024-09-19T07:40:09.099 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:09.099 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":145,"btime":"2024-09-19T07:39:11:439402+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5852,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:09.100 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 145 2024-09-19T07:40:09.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.106+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1888919810 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1d30067870 msgr2=0x7f1d30069d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:09.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.106+0000 7f1d4a1cf640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1d30067870 0x7f1d30069d30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1d340ad000 tx=0x7f1d340a6000 comp rx=0 tx=0).stop 2024-09-19T07:40:09.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.106+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1888919810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d440c6350 msgr2=0x7f1d44146540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:09.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.106+0000 7f1d4a1cf640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d440c6350 0x7f1d44146540 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800c500 tx=0x7f1d3800c9c0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.106+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1888919810 shutdown_connections 2024-09-19T07:40:09.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.106+0000 7f1d4a1cf640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1d30067870 0x7f1d30069d30 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.106+0000 7f1d4a1cf640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d44157630 0x7f1d44155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.106+0000 7f1d4a1cf640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d440c6350 0x7f1d44146540 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.106+0000 7f1d4a1cf640 1 --2- 172.21.15.83:0/1888919810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d440c5740 0x7f1d44071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.106+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1888919810 >> 172.21.15.83:0/1888919810 conn(0x7f1d4406c080 msgr2=0x7f1d440c2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:09.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.106+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1888919810 shutdown_connections 2024-09-19T07:40:09.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.106+0000 7f1d4a1cf640 1 -- 172.21.15.83:0/1888919810 wait complete. 2024-09-19T07:40:09.127 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-19T07:40:09.127 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:40:09.127 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:40:09.142 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:40:09.143 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:09.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b41c25640 1 Processor -- start 2024-09-19T07:40:09.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b41c25640 1 -- start start 2024-09-19T07:40:09.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b41c25640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b3c0c6120 0x7f1b3c0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:09.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b41c25640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1b3c1592f0 0x7f1b3c157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:09.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b41c25640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b3c156200 con 0x7f1b3c155d70 2024-09-19T07:40:09.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b41c25640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f1b3c0c9930 con 0x7f1b3c1592f0 2024-09-19T07:40:09.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b41c25640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1b3c1564e0 con 0x7f1b3c0c6120 2024-09-19T07:40:09.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b3bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1b3c1592f0 0x7f1b3c157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:09.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b3bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1b3c1592f0 0x7f1b3c157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55174/0 (socket says 172.21.15.83:55174) 2024-09-19T07:40:09.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b3bfff640 1 -- 172.21.15.83:0/341504675 learned_addr learned my addr 172.21.15.83:0/341504675 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:09.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b3b7fe640 1 --2- 172.21.15.83:0/341504675 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b3c0c6120 0x7f1b3c0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:09.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b3bfff640 1 -- 172.21.15.83:0/341504675 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b3c0c6120 msgr2=0x7f1b3c0c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:09.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b3bfff640 1 --2- 172.21.15.83:0/341504675 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b3c0c6120 0x7f1b3c0c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b3bfff640 1 -- 172.21.15.83:0/341504675 >> v1:172.21.15.184:6789/0 conn(0x7f1b3c155d70 legacy=0x7f1b3c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:09.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.350+0000 7f1b3bfff640 1 -- 172.21.15.83:0/341504675 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3c1567c0 con 0x7f1b3c1592f0 2024-09-19T07:40:09.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.354+0000 7f1b3bfff640 1 --2- 172.21.15.83:0/341504675 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c1592f0 0x7f1b3c157ce0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f1b34009e50 tx=0x7f1b3401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50468c6b4e73775 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:09.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.354+0000 7f1b3a7fc640 1 -- 172.21.15.83:0/341504675 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b340a5700 con 0x7f1b3c1592f0 2024-09-19T07:40:09.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.354+0000 7f1b3a7fc640 1 -- 172.21.15.83:0/341504675 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b340a58a0 con 0x7f1b3c1592f0 2024-09-19T07:40:09.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.354+0000 7f1b3a7fc640 1 -- 172.21.15.83:0/341504675 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b340a5c70 con 0x7f1b3c1592f0 2024-09-19T07:40:09.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 -- 172.21.15.83:0/341504675 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c1592f0 msgr2=0x7f1b3c157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:09.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 --2- 172.21.15.83:0/341504675 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c1592f0 0x7f1b3c157ce0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f1b34009e50 tx=0x7f1b3401ca30 comp rx=0 tx=0).stop 2024-09-19T07:40:09.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 -- 172.21.15.83:0/341504675 shutdown_connections 2024-09-19T07:40:09.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 --2- 172.21.15.83:0/341504675 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c1592f0 0x7f1b3c157ce0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 --2- 172.21.15.83:0/341504675 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b3c0c6120 0x7f1b3c0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 -- 172.21.15.83:0/341504675 >> 172.21.15.83:0/341504675 conn(0x7f1b3c06bc80 msgr2=0x7f1b3c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:09.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 -- 172.21.15.83:0/341504675 shutdown_connections 2024-09-19T07:40:09.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 -- 172.21.15.83:0/341504675 wait complete. 2024-09-19T07:40:09.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 Processor -- start 2024-09-19T07:40:09.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 -- start start 2024-09-19T07:40:09.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b3c0c6120 0x7f1b3c0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:09.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b3c1592f0 0x7f1b3c0cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:09.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0ade20 0x7f1b3c1ea140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:09.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3c153860 con 0x7f1b3c0ade20 2024-09-19T07:40:09.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3c162ea0 con 0x7f1b3c1592f0 2024-09-19T07:40:09.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b41c25640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1b3c1e7e10 con 0x7f1b3c0c6120 2024-09-19T07:40:09.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b3affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b3c1592f0 0x7f1b3c0cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b3b7fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b3c0c6120 0x7f1b3c0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b3affd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b3c1592f0 0x7f1b3c0cb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:46840/0 (socket says 172.21.15.83:46840) 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b3b7fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b3c0c6120 0x7f1b3c0cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:41422/0 (socket says 172.21.15.83:41422) 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b3affd640 1 -- 172.21.15.83:0/2290263967 learned_addr learned my addr 172.21.15.83:0/2290263967 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b3bfff640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0ade20 0x7f1b3c1ea140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b3affd640 1 -- 172.21.15.83:0/2290263967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b3c0c6120 msgr2=0x7f1b3c0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b3affd640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b3c0c6120 0x7f1b3c0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b3affd640 1 -- 172.21.15.83:0/2290263967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0ade20 msgr2=0x7f1b3c1ea140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b3affd640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0ade20 0x7f1b3c1ea140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b3affd640 1 -- 172.21.15.83:0/2290263967 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3c1ee1f0 con 0x7f1b3c1592f0 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.358+0000 7f1b3affd640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b3c1592f0 0x7f1b3c0cb2b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000c3c0 tx=0x7f1b3000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.362+0000 7f1b23fff640 1 -- 172.21.15.83:0/2290263967 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b300052d0 con 0x7f1b3c1592f0 2024-09-19T07:40:09.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.362+0000 7f1b23fff640 1 -- 172.21.15.83:0/2290263967 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b30005470 con 0x7f1b3c1592f0 2024-09-19T07:40:09.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.362+0000 7f1b23fff640 1 -- 172.21.15.83:0/2290263967 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b30007260 con 0x7f1b3c1592f0 2024-09-19T07:40:09.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.362+0000 7f1b41c25640 1 -- 172.21.15.83:0/2290263967 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b3c1ee4d0 con 0x7f1b3c1592f0 2024-09-19T07:40:09.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.362+0000 7f1b41c25640 1 -- 172.21.15.83:0/2290263967 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b3c0c5180 con 0x7f1b3c1592f0 2024-09-19T07:40:09.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.362+0000 7f1b41c25640 1 -- 172.21.15.83:0/2290263967 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b3c0c6500 con 0x7f1b3c1592f0 2024-09-19T07:40:09.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.366+0000 7f1b23fff640 1 -- 172.21.15.83:0/2290263967 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1b30007400 con 0x7f1b3c1592f0 2024-09-19T07:40:09.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.366+0000 7f1b23fff640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1b280678c0 0x7f1b28069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:09.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.366+0000 7f1b23fff640 1 -- 172.21.15.83:0/2290263967 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 9045+0+0 (secure 0 0 0) 0x7f1b30085f30 con 0x7f1b3c1592f0 2024-09-19T07:40:09.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.370+0000 7f1b3b7fe640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1b280678c0 0x7f1b28069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:09.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.370+0000 7f1b3b7fe640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1b280678c0 0x7f1b28069d80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c0cbfe0 tx=0x7f1b2c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:09.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.374+0000 7f1b23fff640 1 -- 172.21.15.83:0/2290263967 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1b30055770 con 0x7f1b3c1592f0 2024-09-19T07:40:09.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.734+0000 7f1b41c25640 1 -- 172.21.15.83:0/2290263967 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b3c160460 con 0x7f1b3c1592f0 2024-09-19T07:40:09.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.734+0000 7f1b23fff640 1 -- 172.21.15.83:0/2290263967 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+6543 (secure 0 0 0) 0x7f1b30059a70 con 0x7f1b3c1592f0 2024-09-19T07:40:09.732 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:09.732 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":145,"btime":"2024-09-19T07:39:11:439402+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5852,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:09.732 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 145 2024-09-19T07:40:09.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.738+0000 7f1b41c25640 1 -- 172.21.15.83:0/2290263967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1b280678c0 msgr2=0x7f1b28069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:09.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.738+0000 7f1b41c25640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1b280678c0 0x7f1b28069d80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c0cbfe0 tx=0x7f1b2c008040 comp rx=0 tx=0).stop 2024-09-19T07:40:09.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.738+0000 7f1b41c25640 1 -- 172.21.15.83:0/2290263967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b3c1592f0 msgr2=0x7f1b3c0cb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:09.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.738+0000 7f1b41c25640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b3c1592f0 0x7f1b3c0cb2b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000c3c0 tx=0x7f1b3000c880 comp rx=0 tx=0).stop 2024-09-19T07:40:09.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.742+0000 7f1b41c25640 1 -- 172.21.15.83:0/2290263967 shutdown_connections 2024-09-19T07:40:09.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.742+0000 7f1b41c25640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1b280678c0 0x7f1b28069d80 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.742+0000 7f1b41c25640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0ade20 0x7f1b3c1ea140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.742+0000 7f1b41c25640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b3c1592f0 0x7f1b3c0cb2b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.742+0000 7f1b41c25640 1 --2- 172.21.15.83:0/2290263967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b3c0c6120 0x7f1b3c0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:09.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.742+0000 7f1b41c25640 1 -- 172.21.15.83:0/2290263967 >> 172.21.15.83:0/2290263967 conn(0x7f1b3c06bc80 msgr2=0x7f1b3c06c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:09.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.742+0000 7f1b41c25640 1 -- 172.21.15.83:0/2290263967 shutdown_connections 2024-09-19T07:40:09.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:09.742+0000 7f1b41c25640 1 -- 172.21.15.83:0/2290263967 wait complete. 2024-09-19T07:40:09.765 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:40:11.766 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:11.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818fcb4640 1 Processor -- start 2024-09-19T07:40:11.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818fcb4640 1 -- start start 2024-09-19T07:40:11.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818fcb4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f81880716a0 0x7f8188155c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:11.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818fcb4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8188071b80 con 0x7f818814fe10 2024-09-19T07:40:11.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818fcb4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f818806eef0 con 0x7f818806d550 2024-09-19T07:40:11.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818fcb4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f818806ecc0 con 0x7f81880716a0 2024-09-19T07:40:11.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818da50640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f818814fe10 0x7f81880c3850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48938/0 (socket says 172.21.15.83:48938) 2024-09-19T07:40:11.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818da50640 1 -- 172.21.15.83:0/3489034979 learned_addr learned my addr 172.21.15.83:0/3489034979 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:40:11.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818d24f640 1 --2- 172.21.15.83:0/3489034979 >> v2:172.21.15.83:3300/0 conn(0x7f81880716a0 0x7f8188155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:11.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818ca4e640 1 -- 172.21.15.83:0/3489034979 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1338865977 0 0) 0x7f8188071b80 con 0x7f818814fe10 2024-09-19T07:40:11.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818ca4e640 1 -- 172.21.15.83:0/3489034979 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8174003530 con 0x7f818814fe10 2024-09-19T07:40:11.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818ca4e640 1 -- 172.21.15.83:0/3489034979 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 822142272 0 0) 0x7f8174003530 con 0x7f818814fe10 2024-09-19T07:40:11.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818d24f640 1 -- 172.21.15.83:0/3489034979 >> v1:172.21.15.184:6789/0 conn(0x7f818806d550 legacy=0x7f81881605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:11.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818e251640 1 --1- 172.21.15.83:0/3489034979 >> v1:172.21.15.184:6789/0 conn(0x7f818806d550 0x7f81881605a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:36314/0 (socket says 172.21.15.83:36314) 2024-09-19T07:40:11.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818d24f640 1 -- 172.21.15.83:0/3489034979 >> v1:172.21.15.83:6789/0 conn(0x7f818814fe10 legacy=0x7f81880c3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:11.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.926+0000 7f818d24f640 1 -- 172.21.15.83:0/3489034979 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818806f1d0 con 0x7f81880716a0 2024-09-19T07:40:11.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.930+0000 7f818d24f640 1 --2- 172.21.15.83:0/3489034979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81880716a0 0x7f8188155c20 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f81780097c0 tx=0x7f817801c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2fd2ba7ea8a790 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:11.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.930+0000 7f818ca4e640 1 -- 172.21.15.83:0/3489034979 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81780b0070 con 0x7f81880716a0 2024-09-19T07:40:11.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.930+0000 7f818ca4e640 1 -- 172.21.15.83:0/3489034979 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81780ab400 con 0x7f81880716a0 2024-09-19T07:40:11.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.930+0000 7f818ca4e640 1 -- 172.21.15.83:0/3489034979 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8184001680 con 0x7f81880716a0 2024-09-19T07:40:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 -- 172.21.15.83:0/3489034979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81880716a0 msgr2=0x7f8188155c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 --2- 172.21.15.83:0/3489034979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81880716a0 0x7f8188155c20 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f81780097c0 tx=0x7f817801c390 comp rx=0 tx=0).stop 2024-09-19T07:40:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 -- 172.21.15.83:0/3489034979 shutdown_connections 2024-09-19T07:40:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 --2- 172.21.15.83:0/3489034979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81880716a0 0x7f8188155c20 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 -- 172.21.15.83:0/3489034979 >> 172.21.15.83:0/3489034979 conn(0x7f81880c4940 msgr2=0x7f81880c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 -- 172.21.15.83:0/3489034979 shutdown_connections 2024-09-19T07:40:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 -- 172.21.15.83:0/3489034979 wait complete. 2024-09-19T07:40:11.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 Processor -- start 2024-09-19T07:40:11.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 -- start start 2024-09-19T07:40:11.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f818806d550 0x7f8188159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:11.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81880716a0 0x7f81881f8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:11.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f818814fe10 0x7f8188147980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:11.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818d24f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81880716a0 0x7f81881f8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:11.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818e251640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f818814fe10 0x7f8188147980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:11.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818e251640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f818814fe10 0x7f8188147980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37118/0 (socket says 172.21.15.83:37118) 2024-09-19T07:40:11.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818e251640 1 -- 172.21.15.83:0/2936363561 learned_addr learned my addr 172.21.15.83:0/2936363561 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:11.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818da50640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f818806d550 0x7f8188159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:11.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f818806fb50 con 0x7f818814fe10 2024-09-19T07:40:11.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 -- 172.21.15.83:0/2936363561 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f818806f9d0 con 0x7f818806d550 2024-09-19T07:40:11.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.934+0000 7f818fcb4640 1 -- 172.21.15.83:0/2936363561 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f81881644e0 con 0x7f81880716a0 2024-09-19T07:40:11.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f818e251640 1 -- 172.21.15.83:0/2936363561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81880716a0 msgr2=0x7f81881f8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:11.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f818e251640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81880716a0 0x7f81881f8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:11.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f818e251640 1 -- 172.21.15.83:0/2936363561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f818806d550 msgr2=0x7f8188159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:11.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f818e251640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f818806d550 0x7f8188159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:11.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f818e251640 1 -- 172.21.15.83:0/2936363561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818806cf20 con 0x7f818814fe10 2024-09-19T07:40:11.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f818e251640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f818814fe10 0x7f8188147980 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f818000c6b0 tx=0x7f818000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:11.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f817e7fc640 1 -- 172.21.15.83:0/2936363561 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8180018070 con 0x7f818814fe10 2024-09-19T07:40:11.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f817e7fc640 1 -- 172.21.15.83:0/2936363561 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8180004d70 con 0x7f818814fe10 2024-09-19T07:40:11.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f817e7fc640 1 -- 172.21.15.83:0/2936363561 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818000a840 con 0x7f818814fe10 2024-09-19T07:40:11.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f818fcb4640 1 -- 172.21.15.83:0/2936363561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81881ffa80 con 0x7f818814fe10 2024-09-19T07:40:11.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f818fcb4640 1 -- 172.21.15.83:0/2936363561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81880aa820 con 0x7f818814fe10 2024-09-19T07:40:11.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f818fcb4640 1 -- 172.21.15.83:0/2936363561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8188155c20 con 0x7f818814fe10 2024-09-19T07:40:11.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.938+0000 7f817e7fc640 1 -- 172.21.15.83:0/2936363561 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f818000a9e0 con 0x7f818814fe10 2024-09-19T07:40:11.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.942+0000 7f817e7fc640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f816c0678c0 0x7f816c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:11.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.942+0000 7f817e7fc640 1 -- 172.21.15.83:0/2936363561 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 9045+0+0 (secure 0 0 0) 0x7f8180085de0 con 0x7f818814fe10 2024-09-19T07:40:11.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.946+0000 7f818da50640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f816c0678c0 0x7f816c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:11.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.946+0000 7f817e7fc640 1 -- 172.21.15.83:0/2936363561 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f81800555a0 con 0x7f818814fe10 2024-09-19T07:40:11.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:11.946+0000 7f818da50640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f816c0678c0 0x7f816c069d80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f81840011f0 tx=0x7f8184001700 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:12.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.290+0000 7f818fcb4640 1 -- 172.21.15.83:0/2936363561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f818815a070 con 0x7f818814fe10 2024-09-19T07:40:12.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.290+0000 7f817e7fc640 1 -- 172.21.15.83:0/2936363561 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+6543 (secure 0 0 0) 0x7f81800598a0 con 0x7f818814fe10 2024-09-19T07:40:12.289 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:12.289 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":145,"btime":"2024-09-19T07:39:11:439402+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5852,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:12.289 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 145 2024-09-19T07:40:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.294+0000 7f818fcb4640 1 -- 172.21.15.83:0/2936363561 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f816c0678c0 msgr2=0x7f816c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.294+0000 7f818fcb4640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f816c0678c0 0x7f816c069d80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f81840011f0 tx=0x7f8184001700 comp rx=0 tx=0).stop 2024-09-19T07:40:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.294+0000 7f818fcb4640 1 -- 172.21.15.83:0/2936363561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f818814fe10 msgr2=0x7f8188147980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.294+0000 7f818fcb4640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f818814fe10 0x7f8188147980 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f818000c6b0 tx=0x7f818000cb70 comp rx=0 tx=0).stop 2024-09-19T07:40:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.294+0000 7f818fcb4640 1 -- 172.21.15.83:0/2936363561 shutdown_connections 2024-09-19T07:40:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.294+0000 7f818fcb4640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f816c0678c0 0x7f816c069d80 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.294+0000 7f818fcb4640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f818814fe10 0x7f8188147980 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.294+0000 7f818fcb4640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81880716a0 0x7f81881f8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.294+0000 7f818fcb4640 1 --2- 172.21.15.83:0/2936363561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f818806d550 0x7f8188159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:12.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.294+0000 7f818fcb4640 1 -- 172.21.15.83:0/2936363561 >> 172.21.15.83:0/2936363561 conn(0x7f81880c4940 msgr2=0x7f8188072980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:12.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.294+0000 7f818fcb4640 1 -- 172.21.15.83:0/2936363561 shutdown_connections 2024-09-19T07:40:12.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:12.294+0000 7f818fcb4640 1 -- 172.21.15.83:0/2936363561 wait complete. 2024-09-19T07:40:12.319 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:40:14.319 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:14.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d28426640 1 Processor -- start 2024-09-19T07:40:14.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d28426640 1 -- start start 2024-09-19T07:40:14.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d28426640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2d20155740 0x7f2d200c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:14.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d28426640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2d200c2f80 0x7f2d20155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:14.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d28426640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d20156770 con 0x7f2d200c7510 2024-09-19T07:40:14.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d28426640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f2d2015a010 con 0x7f2d200c2f80 2024-09-19T07:40:14.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d28426640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f2d200c4a30 con 0x7f2d20155740 2024-09-19T07:40:14.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d259c1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2d200c2f80 0x7f2d20155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:14.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d259c1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2d200c2f80 0x7f2d20155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37130/0 (socket says 172.21.15.83:37130) 2024-09-19T07:40:14.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d259c1640 1 -- 172.21.15.83:0/3278370874 learned_addr learned my addr 172.21.15.83:0/3278370874 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:14.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d259c1640 1 -- 172.21.15.83:0/3278370874 >> v2:172.21.15.184:3300/0 conn(0x7f2d20155740 msgr2=0x7f2d200c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:14.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d259c1640 1 --2- 172.21.15.83:0/3278370874 >> v2:172.21.15.184:3300/0 conn(0x7f2d20155740 0x7f2d200c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:14.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d259c1640 1 -- 172.21.15.83:0/3278370874 >> v1:172.21.15.184:6789/0 conn(0x7f2d200c7510 legacy=0x7f2d20156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:14.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.454+0000 7f2d259c1640 1 -- 172.21.15.83:0/3278370874 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d200c4d10 con 0x7f2d200c2f80 2024-09-19T07:40:14.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d259c1640 1 --2- 172.21.15.83:0/3278370874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d200c2f80 0x7f2d20155b20 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f2d200af550 tx=0x7f2d1401c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26191cd8dd05ca75 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:14.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d251c0640 1 -- 172.21.15.83:0/3278370874 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d140b0070 con 0x7f2d200c2f80 2024-09-19T07:40:14.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d251c0640 1 -- 172.21.15.83:0/3278370874 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d140ab600 con 0x7f2d200c2f80 2024-09-19T07:40:14.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d251c0640 1 -- 172.21.15.83:0/3278370874 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d140ab9d0 con 0x7f2d200c2f80 2024-09-19T07:40:14.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d28426640 1 -- 172.21.15.83:0/3278370874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d200c2f80 msgr2=0x7f2d20155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:14.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d28426640 1 --2- 172.21.15.83:0/3278370874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d200c2f80 0x7f2d20155b20 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f2d200af550 tx=0x7f2d1401c270 comp rx=0 tx=0).stop 2024-09-19T07:40:14.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d28426640 1 -- 172.21.15.83:0/3278370874 shutdown_connections 2024-09-19T07:40:14.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d28426640 1 --2- 172.21.15.83:0/3278370874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d200c2f80 0x7f2d20155b20 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:14.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d28426640 1 --2- 172.21.15.83:0/3278370874 >> v2:172.21.15.184:3300/0 conn(0x7f2d20155740 0x7f2d200c2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:14.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d28426640 1 -- 172.21.15.83:0/3278370874 >> 172.21.15.83:0/3278370874 conn(0x7f2d200c9610 msgr2=0x7f2d200c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:14.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d28426640 1 -- 172.21.15.83:0/3278370874 shutdown_connections 2024-09-19T07:40:14.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d28426640 1 -- 172.21.15.83:0/3278370874 wait complete. 2024-09-19T07:40:14.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d28426640 1 Processor -- start 2024-09-19T07:40:14.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.458+0000 7f2d28426640 1 -- start start 2024-09-19T07:40:14.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d28426640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d200c2f80 0x7f2d2014c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:14.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d28426640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d200c7510 0x7f2d20148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:14.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d28426640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d20155740 0x7f2d20148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:14.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d28426640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2006f3e0 con 0x7f2d20155740 2024-09-19T07:40:14.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d28426640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2d200c3c40 con 0x7f2d200c2f80 2024-09-19T07:40:14.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d28426640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2d200aa6b0 con 0x7f2d200c7510 2024-09-19T07:40:14.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d269c3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d20155740 0x7f2d20148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:14.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d269c3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d20155740 0x7f2d20148950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37134/0 (socket says 172.21.15.83:37134) 2024-09-19T07:40:14.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d269c3640 1 -- 172.21.15.83:0/1076235892 learned_addr learned my addr 172.21.15.83:0/1076235892 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d261c2640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d200c2f80 0x7f2d2014c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d259c1640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d200c7510 0x7f2d20148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d269c3640 1 -- 172.21.15.83:0/1076235892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d200c7510 msgr2=0x7f2d20148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d269c3640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d200c7510 0x7f2d20148410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d269c3640 1 -- 172.21.15.83:0/1076235892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d200c2f80 msgr2=0x7f2d2014c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d269c3640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d200c2f80 0x7f2d2014c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d269c3640 1 -- 172.21.15.83:0/1076235892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d201547f0 con 0x7f2d20155740 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d269c3640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d20155740 0x7f2d20148950 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f2d1800b9e0 tx=0x7f2d1800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d1effd640 1 -- 172.21.15.83:0/1076235892 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d18019070 con 0x7f2d20155740 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d1effd640 1 -- 172.21.15.83:0/1076235892 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d1800caa0 con 0x7f2d20155740 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d28426640 1 -- 172.21.15.83:0/1076235892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d20154ad0 con 0x7f2d20155740 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d1effd640 1 -- 172.21.15.83:0/1076235892 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d18014440 con 0x7f2d20155740 2024-09-19T07:40:14.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d28426640 1 -- 172.21.15.83:0/1076235892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d2014cda0 con 0x7f2d20155740 2024-09-19T07:40:14.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d28426640 1 -- 172.21.15.83:0/1076235892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d20147120 con 0x7f2d20155740 2024-09-19T07:40:14.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d1effd640 1 -- 172.21.15.83:0/1076235892 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2d1801c070 con 0x7f2d20155740 2024-09-19T07:40:14.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d1effd640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d0c0678c0 0x7f2d0c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:14.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d261c2640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d0c0678c0 0x7f2d0c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:14.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.462+0000 7f2d1effd640 1 -- 172.21.15.83:0/1076235892 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 9045+0+0 (secure 0 0 0) 0x7f2d180866c0 con 0x7f2d20155740 2024-09-19T07:40:14.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.466+0000 7f2d261c2640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d0c0678c0 0x7f2d0c069d80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2d10006fd0 tx=0x7f2d10008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:14.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.470+0000 7f2d1effd640 1 -- 172.21.15.83:0/1076235892 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2d20147120 con 0x7f2d20155740 2024-09-19T07:40:14.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.650+0000 7f2d28426640 1 -- 172.21.15.83:0/1076235892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d200c2a40 con 0x7f2d20155740 2024-09-19T07:40:14.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.650+0000 7f2d1effd640 1 -- 172.21.15.83:0/1076235892 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+6543 (secure 0 0 0) 0x7f2d180564a0 con 0x7f2d20155740 2024-09-19T07:40:14.649 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:14.650 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":145,"btime":"2024-09-19T07:39:11:439402+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5852,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:14.651 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 145 2024-09-19T07:40:14.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.654+0000 7f2d28426640 1 -- 172.21.15.83:0/1076235892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d0c0678c0 msgr2=0x7f2d0c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:14.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.654+0000 7f2d28426640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d0c0678c0 0x7f2d0c069d80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2d10006fd0 tx=0x7f2d10008040 comp rx=0 tx=0).stop 2024-09-19T07:40:14.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.654+0000 7f2d28426640 1 -- 172.21.15.83:0/1076235892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d20155740 msgr2=0x7f2d20148950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:14.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.654+0000 7f2d28426640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d20155740 0x7f2d20148950 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f2d1800b9e0 tx=0x7f2d1800bea0 comp rx=0 tx=0).stop 2024-09-19T07:40:14.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.654+0000 7f2d28426640 1 -- 172.21.15.83:0/1076235892 shutdown_connections 2024-09-19T07:40:14.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.654+0000 7f2d28426640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d0c0678c0 0x7f2d0c069d80 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:14.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.654+0000 7f2d28426640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d20155740 0x7f2d20148950 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:14.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.654+0000 7f2d28426640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d200c7510 0x7f2d20148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:14.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.654+0000 7f2d28426640 1 --2- 172.21.15.83:0/1076235892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d200c2f80 0x7f2d2014c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:14.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.654+0000 7f2d28426640 1 -- 172.21.15.83:0/1076235892 >> 172.21.15.83:0/1076235892 conn(0x7f2d200c9610 msgr2=0x7f2d2014d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:14.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.654+0000 7f2d28426640 1 -- 172.21.15.83:0/1076235892 shutdown_connections 2024-09-19T07:40:14.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:14.654+0000 7f2d28426640 1 -- 172.21.15.83:0/1076235892 wait complete. 2024-09-19T07:40:14.666 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:40:16.667 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:16.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.802+0000 7fdf44168640 1 Processor -- start 2024-09-19T07:40:16.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.802+0000 7fdf44168640 1 -- start start 2024-09-19T07:40:16.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf44168640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fdf3c1545e0 0x7fdf3c06e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:16.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf44168640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf3c154a70 con 0x7fdf3c071100 2024-09-19T07:40:16.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf44168640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf3c0a9e60 con 0x7fdf3c0c28e0 2024-09-19T07:40:16.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf44168640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fdf3c071b50 con 0x7fdf3c1545e0 2024-09-19T07:40:16.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf41f04640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdf3c071100 0x7fdf3c150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48952/0 (socket says 172.21.15.83:48952) 2024-09-19T07:40:16.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf41f04640 1 -- 172.21.15.83:0/2923812813 learned_addr learned my addr 172.21.15.83:0/2923812813 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:40:16.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf42705640 1 --2- 172.21.15.83:0/2923812813 >> v2:172.21.15.184:3300/0 conn(0x7fdf3c1545e0 0x7fdf3c06e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:16.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 769593657 0 0) 0x7fdf3c154a70 con 0x7fdf3c071100 2024-09-19T07:40:16.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf28003530 con 0x7fdf3c071100 2024-09-19T07:40:16.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4181787251 0 0) 0x7fdf28003530 con 0x7fdf3c071100 2024-09-19T07:40:16.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 >> v2:172.21.15.184:3300/0 conn(0x7fdf3c1545e0 msgr2=0x7fdf3c06e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:16.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 --2- 172.21.15.83:0/2923812813 >> v2:172.21.15.184:3300/0 conn(0x7fdf3c1545e0 0x7fdf3c06e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:16.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 >> v1:172.21.15.184:6789/0 conn(0x7fdf3c0c28e0 legacy=0x7fdf3c06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:16.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf3c0aa140 con 0x7fdf3c071100 2024-09-19T07:40:16.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fdf38001680 con 0x7fdf3c071100 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 >> v1:172.21.15.83:6789/0 conn(0x7fdf3c071100 legacy=0x7fdf3c150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 --2- 172.21.15.83:0/2923812813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf280039c0 0x7fdf28003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 --2- 172.21.15.83:0/2923812813 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf28004810 0x7fdf28004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 --2- 172.21.15.83:0/2923812813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf280052f0 0x7fdf2800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c071b50 con 0x7fdf280052f0 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2800c1c0 con 0x7fdf280039c0 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdf2800c3c0 con 0x7fdf28004810 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf42705640 1 --2- 172.21.15.83:0/2923812813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf280052f0 0x7fdf2800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf41f04640 1 --2- 172.21.15.83:0/2923812813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf280039c0 0x7fdf28003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf42705640 1 -- 172.21.15.83:0/2923812813 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf28004810 msgr2=0x7fdf28004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf42705640 1 --2- 172.21.15.83:0/2923812813 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf28004810 0x7fdf28004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf42705640 1 -- 172.21.15.83:0/2923812813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf280039c0 msgr2=0x7fdf28003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf42705640 1 --2- 172.21.15.83:0/2923812813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf280039c0 0x7fdf28003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:16.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf42705640 1 -- 172.21.15.83:0/2923812813 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf28003530 con 0x7fdf280052f0 2024-09-19T07:40:16.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf42705640 1 -- 172.21.15.83:0/2923812813 reap_dead start 2024-09-19T07:40:16.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.806+0000 7fdf42705640 1 --2- 172.21.15.83:0/2923812813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf280052f0 0x7fdf2800b9d0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fdf34007330 tx=0x7fdf3401ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6138becd240bba3a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:16.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf34009be0 con 0x7fdf280052f0 2024-09-19T07:40:16.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf340a5c50 con 0x7fdf280052f0 2024-09-19T07:40:16.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf40f02640 1 -- 172.21.15.83:0/2923812813 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf340039a0 con 0x7fdf280052f0 2024-09-19T07:40:16.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 -- 172.21.15.83:0/2923812813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf280052f0 msgr2=0x7fdf2800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:16.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 --2- 172.21.15.83:0/2923812813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf280052f0 0x7fdf2800b9d0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fdf34007330 tx=0x7fdf3401ce60 comp rx=0 tx=0).stop 2024-09-19T07:40:16.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 -- 172.21.15.83:0/2923812813 shutdown_connections 2024-09-19T07:40:16.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 --2- 172.21.15.83:0/2923812813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf280052f0 0x7fdf2800b9d0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:16.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 -- 172.21.15.83:0/2923812813 >> 172.21.15.83:0/2923812813 conn(0x7fdf3c06bc80 msgr2=0x7fdf3c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:16.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 -- 172.21.15.83:0/2923812813 shutdown_connections 2024-09-19T07:40:16.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 -- 172.21.15.83:0/2923812813 wait complete. 2024-09-19T07:40:16.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 Processor -- start 2024-09-19T07:40:16.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 -- start start 2024-09-19T07:40:16.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf280052f0 0x7fdf3c1dd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:16.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf3c14f1e0 0x7fdf3c1e9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:16.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf3c14e980 0x7fdf3c152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:16.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c0c2520 con 0x7fdf3c14e980 2024-09-19T07:40:16.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c0c1ff0 con 0x7fdf280052f0 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdf3c0714e0 con 0x7fdf3c14f1e0 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf42705640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf3c14e980 0x7fdf3c152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf42705640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf3c14e980 0x7fdf3c152fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37148/0 (socket says 172.21.15.83:37148) 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf42705640 1 -- 172.21.15.83:0/677733346 learned_addr learned my addr 172.21.15.83:0/677733346 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf41f04640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf280052f0 0x7fdf3c1dd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf41703640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf3c14f1e0 0x7fdf3c1e9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf42705640 1 -- 172.21.15.83:0/677733346 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf3c14f1e0 msgr2=0x7fdf3c1e9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf42705640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf3c14f1e0 0x7fdf3c1e9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf42705640 1 -- 172.21.15.83:0/677733346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf280052f0 msgr2=0x7fdf3c1dd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf42705640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf280052f0 0x7fdf3c1dd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf42705640 1 -- 172.21.15.83:0/677733346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf3c1dd820 con 0x7fdf3c14e980 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf41703640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf3c14f1e0 0x7fdf3c1e9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf42705640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf3c14e980 0x7fdf3c152fb0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fdf3401d700 tx=0x7fdf34019c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:16.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf327fc640 1 -- 172.21.15.83:0/677733346 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf340ba070 con 0x7fdf3c14e980 2024-09-19T07:40:16.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf327fc640 1 -- 172.21.15.83:0/677733346 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf34004290 con 0x7fdf3c14e980 2024-09-19T07:40:16.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 -- 172.21.15.83:0/677733346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf3c14f670 con 0x7fdf3c14e980 2024-09-19T07:40:16.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf327fc640 1 -- 172.21.15.83:0/677733346 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf34004680 con 0x7fdf3c14e980 2024-09-19T07:40:16.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.810+0000 7fdf44168640 1 -- 172.21.15.83:0/677733346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf3c14ef00 con 0x7fdf3c14e980 2024-09-19T07:40:16.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.814+0000 7fdf44168640 1 -- 172.21.15.83:0/677733346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf3c150f40 con 0x7fdf3c14e980 2024-09-19T07:40:16.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.814+0000 7fdf327fc640 1 -- 172.21.15.83:0/677733346 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdf340bd070 con 0x7fdf3c14e980 2024-09-19T07:40:16.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.814+0000 7fdf327fc640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf200678c0 0x7fdf20069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:16.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.814+0000 7fdf327fc640 1 -- 172.21.15.83:0/677733346 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 9045+0+0 (secure 0 0 0) 0x7fdf34120570 con 0x7fdf3c14e980 2024-09-19T07:40:16.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.814+0000 7fdf41f04640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf200678c0 0x7fdf20069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:16.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.814+0000 7fdf41f04640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf200678c0 0x7fdf20069d80 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fdf38004880 tx=0x7fdf380047f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:16.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:16.818+0000 7fdf327fc640 1 -- 172.21.15.83:0/677733346 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdf34016b70 con 0x7fdf3c14e980 2024-09-19T07:40:17.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.014+0000 7fdf44168640 1 -- 172.21.15.83:0/677733346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf3c0c4f30 con 0x7fdf3c14e980 2024-09-19T07:40:17.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf327fc640 1 -- 172.21.15.83:0/677733346 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+6543 (secure 0 0 0) 0x7fdf340f03e0 con 0x7fdf3c14e980 2024-09-19T07:40:17.014 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:17.014 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":145,"btime":"2024-09-19T07:39:11:439402+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5852,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:17.014 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 145 2024-09-19T07:40:17.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf44168640 1 -- 172.21.15.83:0/677733346 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf200678c0 msgr2=0x7fdf20069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:17.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf44168640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf200678c0 0x7fdf20069d80 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fdf38004880 tx=0x7fdf380047f0 comp rx=0 tx=0).stop 2024-09-19T07:40:17.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf44168640 1 -- 172.21.15.83:0/677733346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf3c14e980 msgr2=0x7fdf3c152fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:17.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf44168640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf3c14e980 0x7fdf3c152fb0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fdf3401d700 tx=0x7fdf34019c00 comp rx=0 tx=0).stop 2024-09-19T07:40:17.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf44168640 1 -- 172.21.15.83:0/677733346 shutdown_connections 2024-09-19T07:40:17.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf44168640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf200678c0 0x7fdf20069d80 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:17.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf44168640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf3c14e980 0x7fdf3c152fb0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:17.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf44168640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf3c14f1e0 0x7fdf3c1e9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:17.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf44168640 1 --2- 172.21.15.83:0/677733346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf280052f0 0x7fdf3c1dd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:17.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf44168640 1 -- 172.21.15.83:0/677733346 >> 172.21.15.83:0/677733346 conn(0x7fdf3c06bc80 msgr2=0x7fdf3c06dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:17.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf44168640 1 -- 172.21.15.83:0/677733346 shutdown_connections 2024-09-19T07:40:17.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:17.018+0000 7fdf44168640 1 -- 172.21.15.83:0/677733346 wait complete. 2024-09-19T07:40:17.035 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:40:19.036 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:19.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.174+0000 7f540daa4640 1 Processor -- start 2024-09-19T07:40:19.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.174+0000 7f540daa4640 1 -- start start 2024-09-19T07:40:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f540daa4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f54080c28e0 0x7f54081545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f540daa4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f540806dea0 0x7f540806e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f540daa4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54080a9e60 con 0x7f5408071100 2024-09-19T07:40:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f540daa4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5408071b50 con 0x7f54080c28e0 2024-09-19T07:40:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f540daa4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f54080c1ff0 con 0x7f540806dea0 2024-09-19T07:40:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f5406ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f54080c28e0 0x7f54081545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f5406ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f54080c28e0 0x7f54081545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37154/0 (socket says 172.21.15.83:37154) 2024-09-19T07:40:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f5406ffd640 1 -- 172.21.15.83:0/409649159 learned_addr learned my addr 172.21.15.83:0/409649159 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f54077fe640 1 --1- 172.21.15.83:0/409649159 >> v1:172.21.15.83:6789/0 conn(0x7f5408071100 0x7f5408150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48966/0 (socket says 172.21.15.83:48966) 2024-09-19T07:40:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f5407fff640 1 --2- 172.21.15.83:0/409649159 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f540806dea0 0x7f540806e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:19.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f5406ffd640 1 -- 172.21.15.83:0/409649159 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f540806dea0 msgr2=0x7f540806e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:19.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f5406ffd640 1 --2- 172.21.15.83:0/409649159 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f540806dea0 0x7f540806e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:19.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f5406ffd640 1 -- 172.21.15.83:0/409649159 >> v1:172.21.15.83:6789/0 conn(0x7f5408071100 legacy=0x7f5408150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:19.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f5406ffd640 1 -- 172.21.15.83:0/409649159 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54080aa140 con 0x7f54080c28e0 2024-09-19T07:40:19.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f5406ffd640 1 --2- 172.21.15.83:0/409649159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080c28e0 0x7f54081545e0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009870 tx=0x7f53f801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539c7b783e6bb6b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:19.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f54067fc640 1 -- 172.21.15.83:0/409649159 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f80b1070 con 0x7f54080c28e0 2024-09-19T07:40:19.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f54067fc640 1 -- 172.21.15.83:0/409649159 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53f80ac620 con 0x7f54080c28e0 2024-09-19T07:40:19.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.178+0000 7f54067fc640 1 -- 172.21.15.83:0/409649159 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f80ac9f0 con 0x7f54080c28e0 2024-09-19T07:40:19.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 -- 172.21.15.83:0/409649159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080c28e0 msgr2=0x7f54081545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:19.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 --2- 172.21.15.83:0/409649159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080c28e0 0x7f54081545e0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009870 tx=0x7f53f801c4d0 comp rx=0 tx=0).stop 2024-09-19T07:40:19.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 -- 172.21.15.83:0/409649159 shutdown_connections 2024-09-19T07:40:19.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 --2- 172.21.15.83:0/409649159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080c28e0 0x7f54081545e0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:19.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 --2- 172.21.15.83:0/409649159 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f540806dea0 0x7f540806e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:19.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 -- 172.21.15.83:0/409649159 >> 172.21.15.83:0/409649159 conn(0x7f540806bc80 msgr2=0x7f540806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:19.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 -- 172.21.15.83:0/409649159 shutdown_connections 2024-09-19T07:40:19.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 -- 172.21.15.83:0/409649159 wait complete. 2024-09-19T07:40:19.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 Processor -- start 2024-09-19T07:40:19.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 -- start start 2024-09-19T07:40:19.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f540806dea0 0x7f540814c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:19.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5408071100 0x7f5408153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:19.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080c28e0 0x7f54081537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:19.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5408071660 con 0x7f54080c28e0 2024-09-19T07:40:19.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f54080714e0 con 0x7f5408071100 2024-09-19T07:40:19.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5408160c60 con 0x7f540806dea0 2024-09-19T07:40:19.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f5407fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080c28e0 0x7f54081537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:19.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f5407fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080c28e0 0x7f54081537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37164/0 (socket says 172.21.15.83:37164) 2024-09-19T07:40:19.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f5407fff640 1 -- 172.21.15.83:0/2040646795 learned_addr learned my addr 172.21.15.83:0/2040646795 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:19.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f54077fe640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f540806dea0 0x7f540814c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:19.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f5406ffd640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5408071100 0x7f5408153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:19.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f5407fff640 1 -- 172.21.15.83:0/2040646795 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f540806dea0 msgr2=0x7f540814c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:19.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f5407fff640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f540806dea0 0x7f540814c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:19.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f5407fff640 1 -- 172.21.15.83:0/2040646795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5408071100 msgr2=0x7f5408153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:19.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f5407fff640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5408071100 0x7f5408153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:19.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f5407fff640 1 -- 172.21.15.83:0/2040646795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54081efb00 con 0x7f54080c28e0 2024-09-19T07:40:19.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f5406ffd640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5408071100 0x7f5408153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:40:19.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f5407fff640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080c28e0 0x7f54081537a0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f5400007e70 tx=0x7f540000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:19.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f53effff640 1 -- 172.21.15.83:0/2040646795 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5400014620 con 0x7f54080c28e0 2024-09-19T07:40:19.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f53effff640 1 -- 172.21.15.83:0/2040646795 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5400019070 con 0x7f54080c28e0 2024-09-19T07:40:19.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 -- 172.21.15.83:0/2040646795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54081efde0 con 0x7f54080c28e0 2024-09-19T07:40:19.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f53effff640 1 -- 172.21.15.83:0/2040646795 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5400014a20 con 0x7f54080c28e0 2024-09-19T07:40:19.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.182+0000 7f540daa4640 1 -- 172.21.15.83:0/2040646795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54081f03a0 con 0x7f54080c28e0 2024-09-19T07:40:19.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.186+0000 7f540daa4640 1 -- 172.21.15.83:0/2040646795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f540806e330 con 0x7f54080c28e0 2024-09-19T07:40:19.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.186+0000 7f53effff640 1 -- 172.21.15.83:0/2040646795 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5400014bc0 con 0x7f54080c28e0 2024-09-19T07:40:19.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.186+0000 7f53effff640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53f40678f0 0x7f53f4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:19.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.186+0000 7f53effff640 1 -- 172.21.15.83:0/2040646795 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 9045+0+0 (secure 0 0 0) 0x7f5400087350 con 0x7f54080c28e0 2024-09-19T07:40:19.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.186+0000 7f54077fe640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53f40678f0 0x7f53f4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:19.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.186+0000 7f54077fe640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53f40678f0 0x7f53f4069db0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f53fc000f70 tx=0x7f53fc007cd0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:19.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.190+0000 7f53effff640 1 -- 172.21.15.83:0/2040646795 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f54000570a0 con 0x7f54080c28e0 2024-09-19T07:40:19.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.394+0000 7f540daa4640 1 -- 172.21.15.83:0/2040646795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54080c4f30 con 0x7f54080c28e0 2024-09-19T07:40:19.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.394+0000 7f53effff640 1 -- 172.21.15.83:0/2040646795 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+6543 (secure 0 0 0) 0x7f54000569a0 con 0x7f54080c28e0 2024-09-19T07:40:19.392 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:19.393 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":145,"btime":"2024-09-19T07:39:11:439402+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5852,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:19.393 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 145 2024-09-19T07:40:19.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.398+0000 7f540daa4640 1 -- 172.21.15.83:0/2040646795 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53f40678f0 msgr2=0x7f53f4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:19.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.398+0000 7f540daa4640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53f40678f0 0x7f53f4069db0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f53fc000f70 tx=0x7f53fc007cd0 comp rx=0 tx=0).stop 2024-09-19T07:40:19.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.398+0000 7f540daa4640 1 -- 172.21.15.83:0/2040646795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080c28e0 msgr2=0x7f54081537a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:19.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.398+0000 7f540daa4640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080c28e0 0x7f54081537a0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f5400007e70 tx=0x7f540000e4b0 comp rx=0 tx=0).stop 2024-09-19T07:40:19.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.398+0000 7f540daa4640 1 -- 172.21.15.83:0/2040646795 shutdown_connections 2024-09-19T07:40:19.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.398+0000 7f540daa4640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53f40678f0 0x7f53f4069db0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:19.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.398+0000 7f540daa4640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080c28e0 0x7f54081537a0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:19.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.398+0000 7f540daa4640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5408071100 0x7f5408153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:19.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.398+0000 7f540daa4640 1 --2- 172.21.15.83:0/2040646795 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f540806dea0 0x7f540814c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:19.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.398+0000 7f540daa4640 1 -- 172.21.15.83:0/2040646795 >> 172.21.15.83:0/2040646795 conn(0x7f540806bc80 msgr2=0x7f540814eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:19.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.398+0000 7f540daa4640 1 -- 172.21.15.83:0/2040646795 shutdown_connections 2024-09-19T07:40:19.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:19.398+0000 7f540daa4640 1 -- 172.21.15.83:0/2040646795 wait complete. 2024-09-19T07:40:19.412 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:40:21.413 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:21.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e6503b640 1 Processor -- start 2024-09-19T07:40:21.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e6503b640 1 -- start start 2024-09-19T07:40:21.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e6503b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5e601545e0 0x7f5e6006e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:21.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e6503b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e60154a70 con 0x7f5e600c28e0 2024-09-19T07:40:21.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e6503b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e600a9e60 con 0x7f5e60071100 2024-09-19T07:40:21.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e6503b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5e60071b50 con 0x7f5e601545e0 2024-09-19T07:40:21.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5e600c28e0 0x7f5e6006dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38666/0 (socket says 172.21.15.83:38666) 2024-09-19T07:40:21.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5659c640 1 -- 172.21.15.83:0/1572943172 learned_addr learned my addr 172.21.15.83:0/1572943172 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:40:21.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5f59e640 1 --2- 172.21.15.83:0/1572943172 >> v2:172.21.15.184:3300/0 conn(0x7f5e601545e0 0x7f5e6006e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:21.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298834756 0 0) 0x7f5e60154a70 con 0x7f5e600c28e0 2024-09-19T07:40:21.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e4c003530 con 0x7f5e600c28e0 2024-09-19T07:40:21.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1033846158 0 0) 0x7f5e4c003530 con 0x7f5e600c28e0 2024-09-19T07:40:21.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 >> v2:172.21.15.184:3300/0 conn(0x7f5e601545e0 msgr2=0x7f5e6006e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:21.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 --2- 172.21.15.83:0/1572943172 >> v2:172.21.15.184:3300/0 conn(0x7f5e601545e0 0x7f5e6006e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:21.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 >> v1:172.21.15.184:6789/0 conn(0x7f5e60071100 legacy=0x7f5e60150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:21.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e600aa140 con 0x7f5e600c28e0 2024-09-19T07:40:21.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f5e48001680 con 0x7f5e600c28e0 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 >> v1:172.21.15.83:6789/0 conn(0x7f5e600c28e0 legacy=0x7f5e6006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e4c0039c0 0x7f5e4c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e4c004810 0x7f5e4c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e4c0051d0 0x7f5e4c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5e60071b50 con 0x7f5e4c0039c0 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c00c170 con 0x7f5e4c0051d0 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5e4c00c370 con 0x7f5e4c004810 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5ed9d640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e4c0039c0 0x7f5e4c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5659c640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e4c004810 0x7f5e4c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5f59e640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e4c0051d0 0x7f5e4c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5ed9d640 1 -- 172.21.15.83:0/1572943172 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e4c004810 msgr2=0x7f5e4c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5ed9d640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e4c004810 0x7f5e4c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:21.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5ed9d640 1 -- 172.21.15.83:0/1572943172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e4c0051d0 msgr2=0x7f5e4c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:21.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5ed9d640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e4c0051d0 0x7f5e4c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:21.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5ed9d640 1 -- 172.21.15.83:0/1572943172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4c003530 con 0x7f5e4c0039c0 2024-09-19T07:40:21.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5f59e640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e4c0051d0 0x7f5e4c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:40:21.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5659c640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e4c004810 0x7f5e4c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:40:21.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.549+0000 7f5e5f59e640 1 -- 172.21.15.83:0/1572943172 reap_dead start 2024-09-19T07:40:21.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e5ed9d640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e4c0039c0 0x7f5e4c003de0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f5e50001780 tx=0x7f5e5001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7313443d9f7ef08f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:21.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e500013a0 con 0x7f5e4c0039c0 2024-09-19T07:40:21.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e500052f0 con 0x7f5e4c0039c0 2024-09-19T07:40:21.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e5e59c640 1 -- 172.21.15.83:0/1572943172 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50003840 con 0x7f5e4c0039c0 2024-09-19T07:40:21.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e6503b640 1 -- 172.21.15.83:0/1572943172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e4c0039c0 msgr2=0x7f5e4c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:21.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e6503b640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e4c0039c0 0x7f5e4c003de0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f5e50001780 tx=0x7f5e5001c400 comp rx=0 tx=0).stop 2024-09-19T07:40:21.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e6503b640 1 -- 172.21.15.83:0/1572943172 shutdown_connections 2024-09-19T07:40:21.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e6503b640 1 --2- 172.21.15.83:0/1572943172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e4c0039c0 0x7f5e4c003de0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:21.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e6503b640 1 -- 172.21.15.83:0/1572943172 >> 172.21.15.83:0/1572943172 conn(0x7f5e6006bc80 msgr2=0x7f5e6006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:21.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e6503b640 1 -- 172.21.15.83:0/1572943172 shutdown_connections 2024-09-19T07:40:21.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e6503b640 1 -- 172.21.15.83:0/1572943172 wait complete. 2024-09-19T07:40:21.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e6503b640 1 Processor -- start 2024-09-19T07:40:21.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.553+0000 7f5e6503b640 1 -- start start 2024-09-19T07:40:21.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e6503b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e4c0039c0 0x7f5e600aab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:21.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e6503b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e600c0c80 0x7f5e60142c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:21.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e6503b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e60143180 0x7f5e600c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:21.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e6503b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6006dea0 con 0x7f5e60143180 2024-09-19T07:40:21.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e6503b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5e600716f0 con 0x7f5e4c0039c0 2024-09-19T07:40:21.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e6503b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5e6006e020 con 0x7f5e600c0c80 2024-09-19T07:40:21.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e60143180 0x7f5e600c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:21.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e60143180 0x7f5e600c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57218/0 (socket says 172.21.15.83:57218) 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5f59e640 1 -- 172.21.15.83:0/1588935250 learned_addr learned my addr 172.21.15.83:0/1588935250 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5ed9d640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e4c0039c0 0x7f5e600aab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5659c640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e600c0c80 0x7f5e60142c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5f59e640 1 -- 172.21.15.83:0/1588935250 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e600c0c80 msgr2=0x7f5e60142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5f59e640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e600c0c80 0x7f5e60142c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5f59e640 1 -- 172.21.15.83:0/1588935250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e4c0039c0 msgr2=0x7f5e600aab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5f59e640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e4c0039c0 0x7f5e600aab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5f59e640 1 -- 172.21.15.83:0/1588935250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e6006e7c0 con 0x7f5e60143180 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5f59e640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e60143180 0x7f5e600c53f0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f5e601545b0 tx=0x7f5e5800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e57fff640 1 -- 172.21.15.83:0/1588935250 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e58016420 con 0x7f5e60143180 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e57fff640 1 -- 172.21.15.83:0/1588935250 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e5801b070 con 0x7f5e60143180 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e57fff640 1 -- 172.21.15.83:0/1588935250 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e580167c0 con 0x7f5e60143180 2024-09-19T07:40:21.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e6503b640 1 -- 172.21.15.83:0/1588935250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e60145e80 con 0x7f5e60143180 2024-09-19T07:40:21.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e6503b640 1 -- 172.21.15.83:0/1588935250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e600c67f0 con 0x7f5e60143180 2024-09-19T07:40:21.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e6503b640 1 -- 172.21.15.83:0/1588935250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e6006e3b0 con 0x7f5e60143180 2024-09-19T07:40:21.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e57fff640 1 -- 172.21.15.83:0/1588935250 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5e58016960 con 0x7f5e60143180 2024-09-19T07:40:21.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e57fff640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5e400678f0 0x7f5e40069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:21.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e57fff640 1 -- 172.21.15.83:0/1588935250 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 9045+0+0 (secure 0 0 0) 0x7f5e58087730 con 0x7f5e60143180 2024-09-19T07:40:21.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5ed9d640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5e400678f0 0x7f5e40069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:21.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.557+0000 7f5e5ed9d640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5e400678f0 0x7f5e40069db0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5e50001750 tx=0x7f5e500011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:21.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.561+0000 7f5e57fff640 1 -- 172.21.15.83:0/1588935250 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5e580575a0 con 0x7f5e60143180 2024-09-19T07:40:21.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.761+0000 7f5e6503b640 1 -- 172.21.15.83:0/1588935250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e600c4f30 con 0x7f5e60143180 2024-09-19T07:40:21.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.761+0000 7f5e57fff640 1 -- 172.21.15.83:0/1588935250 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+6543 (secure 0 0 0) 0x7f5e58056ea0 con 0x7f5e60143180 2024-09-19T07:40:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:21.761 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":145,"btime":"2024-09-19T07:39:11:439402+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5852,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:38:11.364472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7266":{"gid":7266,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/2326757290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2326757290},{"type":"v1","addr":"172.21.15.83:6837","nonce":2326757290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7266,"qdb_cluster":[7266]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:21.761 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 145 2024-09-19T07:40:21.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.765+0000 7f5e6503b640 1 -- 172.21.15.83:0/1588935250 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5e400678f0 msgr2=0x7f5e40069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:21.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.765+0000 7f5e6503b640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5e400678f0 0x7f5e40069db0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5e50001750 tx=0x7f5e500011a0 comp rx=0 tx=0).stop 2024-09-19T07:40:21.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.765+0000 7f5e6503b640 1 -- 172.21.15.83:0/1588935250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e60143180 msgr2=0x7f5e600c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:21.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.765+0000 7f5e6503b640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e60143180 0x7f5e600c53f0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f5e601545b0 tx=0x7f5e5800f520 comp rx=0 tx=0).stop 2024-09-19T07:40:21.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.765+0000 7f5e6503b640 1 -- 172.21.15.83:0/1588935250 shutdown_connections 2024-09-19T07:40:21.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.765+0000 7f5e6503b640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5e400678f0 0x7f5e40069db0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:21.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.765+0000 7f5e6503b640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e60143180 0x7f5e600c53f0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:21.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.765+0000 7f5e6503b640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e600c0c80 0x7f5e60142c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:21.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.765+0000 7f5e6503b640 1 --2- 172.21.15.83:0/1588935250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e4c0039c0 0x7f5e600aab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:21.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.765+0000 7f5e6503b640 1 -- 172.21.15.83:0/1588935250 >> 172.21.15.83:0/1588935250 conn(0x7f5e6006bc80 msgr2=0x7f5e600c3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:21.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.765+0000 7f5e6503b640 1 -- 172.21.15.83:0/1588935250 shutdown_connections 2024-09-19T07:40:21.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:21.765+0000 7f5e6503b640 1 -- 172.21.15.83:0/1588935250 wait complete. 2024-09-19T07:40:21.781 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:40:23.782 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b400d640 1 Processor -- start 2024-09-19T07:40:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b400d640 1 -- start start 2024-09-19T07:40:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b400d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f79ac0c28e0 0x7f79ac1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b400d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f79ac06dea0 0x7f79ac06e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b400d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79ac0a9e60 con 0x7f79ac071100 2024-09-19T07:40:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b400d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f79ac071b50 con 0x7f79ac0c28e0 2024-09-19T07:40:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b400d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f79ac0c1ff0 con 0x7f79ac06dea0 2024-09-19T07:40:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b15a8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f79ac0c28e0 0x7f79ac1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b15a8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f79ac0c28e0 0x7f79ac1545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57224/0 (socket says 172.21.15.83:57224) 2024-09-19T07:40:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b15a8640 1 -- 172.21.15.83:0/3205559564 learned_addr learned my addr 172.21.15.83:0/3205559564 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b1da9640 1 --1- 172.21.15.83:0/3205559564 >> v1:172.21.15.83:6789/0 conn(0x7f79ac071100 0x7f79ac150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38670/0 (socket says 172.21.15.83:38670) 2024-09-19T07:40:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b25aa640 1 --2- 172.21.15.83:0/3205559564 >> v2:172.21.15.184:3300/0 conn(0x7f79ac06dea0 0x7f79ac06e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b15a8640 1 -- 172.21.15.83:0/3205559564 >> v2:172.21.15.184:3300/0 conn(0x7f79ac06dea0 msgr2=0x7f79ac06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b15a8640 1 --2- 172.21.15.83:0/3205559564 >> v2:172.21.15.184:3300/0 conn(0x7f79ac06dea0 0x7f79ac06e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b15a8640 1 -- 172.21.15.83:0/3205559564 >> v1:172.21.15.83:6789/0 conn(0x7f79ac071100 legacy=0x7f79ac150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.917+0000 7f79b15a8640 1 -- 172.21.15.83:0/3205559564 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ac0aa140 con 0x7f79ac0c28e0 2024-09-19T07:40:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b15a8640 1 --2- 172.21.15.83:0/3205559564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79ac0c28e0 0x7f79ac1545e0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0c4f00 tx=0x7f799c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59bad26289aa9a9c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b0da7640 1 -- 172.21.15.83:0/3205559564 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c0b2070 con 0x7f79ac0c28e0 2024-09-19T07:40:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b0da7640 1 -- 172.21.15.83:0/3205559564 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f799c0ad680 con 0x7f79ac0c28e0 2024-09-19T07:40:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b0da7640 1 -- 172.21.15.83:0/3205559564 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c0ada50 con 0x7f79ac0c28e0 2024-09-19T07:40:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 -- 172.21.15.83:0/3205559564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79ac0c28e0 msgr2=0x7f79ac1545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 --2- 172.21.15.83:0/3205559564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79ac0c28e0 0x7f79ac1545e0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0c4f00 tx=0x7f799c01c490 comp rx=0 tx=0).stop 2024-09-19T07:40:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 -- 172.21.15.83:0/3205559564 shutdown_connections 2024-09-19T07:40:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 --2- 172.21.15.83:0/3205559564 >> v2:172.21.15.184:3300/0 conn(0x7f79ac06dea0 0x7f79ac06e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 --2- 172.21.15.83:0/3205559564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79ac0c28e0 0x7f79ac1545e0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 -- 172.21.15.83:0/3205559564 >> 172.21.15.83:0/3205559564 conn(0x7f79ac06bc80 msgr2=0x7f79ac06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 -- 172.21.15.83:0/3205559564 shutdown_connections 2024-09-19T07:40:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 -- 172.21.15.83:0/3205559564 wait complete. 2024-09-19T07:40:23.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 Processor -- start 2024-09-19T07:40:23.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 -- start start 2024-09-19T07:40:23.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79ac06dea0 0x7f79ac151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:23.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79ac071100 0x7f79ac144970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:23.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f79ac0c28e0 0x7f79ac14c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:23.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac071660 con 0x7f79ac071100 2024-09-19T07:40:23.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac0714e0 con 0x7f79ac06dea0 2024-09-19T07:40:23.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b400d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f79ac160c60 con 0x7f79ac0c28e0 2024-09-19T07:40:23.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b15a8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79ac071100 0x7f79ac144970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:23.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b15a8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79ac071100 0x7f79ac144970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57228/0 (socket says 172.21.15.83:57228) 2024-09-19T07:40:23.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b15a8640 1 -- 172.21.15.83:0/1370005022 learned_addr learned my addr 172.21.15.83:0/1370005022 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:23.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b25aa640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f79ac0c28e0 0x7f79ac14c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b1da9640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79ac06dea0 0x7f79ac151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b15a8640 1 -- 172.21.15.83:0/1370005022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f79ac0c28e0 msgr2=0x7f79ac14c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b15a8640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f79ac0c28e0 0x7f79ac14c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b15a8640 1 -- 172.21.15.83:0/1370005022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79ac06dea0 msgr2=0x7f79ac151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b15a8640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79ac06dea0 0x7f79ac151ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b15a8640 1 -- 172.21.15.83:0/1370005022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ac1476d0 con 0x7f79ac071100 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.921+0000 7f79b15a8640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79ac071100 0x7f79ac144970 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f799c01a0c0 tx=0x7f799c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.925+0000 7f79a27fc640 1 -- 172.21.15.83:0/1370005022 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c0b2040 con 0x7f79ac071100 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.925+0000 7f79a27fc640 1 -- 172.21.15.83:0/1370005022 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f799c0ada50 con 0x7f79ac071100 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.925+0000 7f79b400d640 1 -- 172.21.15.83:0/1370005022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79ac1f0790 con 0x7f79ac071100 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.925+0000 7f79a27fc640 1 -- 172.21.15.83:0/1370005022 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c0ade20 con 0x7f79ac071100 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.925+0000 7f79b400d640 1 -- 172.21.15.83:0/1370005022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79ac0c1870 con 0x7f79ac071100 2024-09-19T07:40:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.925+0000 7f79b400d640 1 -- 172.21.15.83:0/1370005022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79ac06c210 con 0x7f79ac071100 2024-09-19T07:40:23.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.925+0000 7f79a27fc640 1 -- 172.21.15.83:0/1370005022 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f799c0bf030 con 0x7f79ac071100 2024-09-19T07:40:23.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.925+0000 7f79a27fc640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f79900678c0 0x7f7990069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:23.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.925+0000 7f79a27fc640 1 -- 172.21.15.83:0/1370005022 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 9131+0+0 (secure 0 0 0) 0x7f799c120030 con 0x7f79ac071100 2024-09-19T07:40:23.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.925+0000 7f79b1da9640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f79900678c0 0x7f7990069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:23.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.925+0000 7f79b1da9640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f79900678c0 0x7f7990069d80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f79a8000f70 tx=0x7f79a8007cd0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:23.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:23.929+0000 7f79a27fc640 1 -- 172.21.15.83:0/1370005022 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f799c016110 con 0x7f79ac071100 2024-09-19T07:40:24.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.129+0000 7f79b400d640 1 -- 172.21.15.83:0/1370005022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79ac0c4f30 con 0x7f79ac071100 2024-09-19T07:40:24.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.129+0000 7f79a27fc640 1 -- 172.21.15.83:0/1370005022 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+5709 (secure 0 0 0) 0x7f799c0efe50 con 0x7f79ac071100 2024-09-19T07:40:24.128 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:24.128 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":146,"btime":"2024-09-19T07:40:23:231327+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:23.231317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:replay","state_seq":1,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:24.128 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 146 2024-09-19T07:40:24.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.133+0000 7f79b400d640 1 -- 172.21.15.83:0/1370005022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f79900678c0 msgr2=0x7f7990069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:24.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.133+0000 7f79b400d640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f79900678c0 0x7f7990069d80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f79a8000f70 tx=0x7f79a8007cd0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.133+0000 7f79b400d640 1 -- 172.21.15.83:0/1370005022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79ac071100 msgr2=0x7f79ac144970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:24.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.133+0000 7f79b400d640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79ac071100 0x7f79ac144970 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f799c01a0c0 tx=0x7f799c01acd0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.133+0000 7f79b400d640 1 -- 172.21.15.83:0/1370005022 shutdown_connections 2024-09-19T07:40:24.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.133+0000 7f79b400d640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f79900678c0 0x7f7990069d80 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.133+0000 7f79b400d640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f79ac0c28e0 0x7f79ac14c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.133+0000 7f79b400d640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79ac071100 0x7f79ac144970 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.133+0000 7f79b400d640 1 --2- 172.21.15.83:0/1370005022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79ac06dea0 0x7f79ac151ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.133+0000 7f79b400d640 1 -- 172.21.15.83:0/1370005022 >> 172.21.15.83:0/1370005022 conn(0x7f79ac06bc80 msgr2=0x7f79ac150d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:24.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.133+0000 7f79b400d640 1 -- 172.21.15.83:0/1370005022 shutdown_connections 2024-09-19T07:40:24.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.133+0000 7f79b400d640 1 -- 172.21.15.83:0/1370005022 wait complete. 2024-09-19T07:40:24.148 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-19T07:40:24.148 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:40:24.148 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.265+0000 7fe50cd6d640 1 Processor -- start 2024-09-19T07:40:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.265+0000 7fe50cd6d640 1 -- start start 2024-09-19T07:40:24.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50cd6d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe508150290 0x7fe5080c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:24.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50cd6d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe508152b20 0x7fe5080c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:24.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50cd6d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe5080c45c0 0x7fe508153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50cd6d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe5080716c0 con 0x7fe508150290 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50cd6d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe5080c3920 con 0x7fe5080c45c0 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50cd6d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe5080c3490 con 0x7fe508152b20 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe508150290 0x7fe5080c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe508150290 0x7fe5080c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57236/0 (socket says 172.21.15.83:57236) 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50659c640 1 -- 172.21.15.83:0/1117787126 learned_addr learned my addr 172.21.15.83:0/1117787126 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe505d9b640 1 --2- 172.21.15.83:0/1117787126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe508152b20 0x7fe5080c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50659c640 1 -- 172.21.15.83:0/1117787126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe508152b20 msgr2=0x7fe5080c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50659c640 1 --2- 172.21.15.83:0/1117787126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe508152b20 0x7fe5080c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50659c640 1 -- 172.21.15.83:0/1117787126 >> v2:172.21.15.184:3300/0 conn(0x7fe5080c45c0 msgr2=0x7fe508153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe506d9d640 1 --2- 172.21.15.83:0/1117787126 >> v2:172.21.15.184:3300/0 conn(0x7fe5080c45c0 0x7fe508153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50659c640 1 --2- 172.21.15.83:0/1117787126 >> v2:172.21.15.184:3300/0 conn(0x7fe5080c45c0 0x7fe508153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50659c640 1 -- 172.21.15.83:0/1117787126 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe508150670 con 0x7fe508150290 2024-09-19T07:40:24.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe506d9d640 1 --2- 172.21.15.83:0/1117787126 >> v2:172.21.15.184:3300/0 conn(0x7fe5080c45c0 0x7fe508153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:40:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50659c640 1 --2- 172.21.15.83:0/1117787126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe508150290 0x7fe5080c1ff0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fe5080ad7d0 tx=0x7fe4f801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cceac81c6047d06c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50559a640 1 -- 172.21.15.83:0/1117787126 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f80aa610 con 0x7fe508150290 2024-09-19T07:40:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50559a640 1 -- 172.21.15.83:0/1117787126 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f80af070 con 0x7fe508150290 2024-09-19T07:40:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.269+0000 7fe50559a640 1 -- 172.21.15.83:0/1117787126 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f80aa9d0 con 0x7fe508150290 2024-09-19T07:40:24.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 -- 172.21.15.83:0/1117787126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe508150290 msgr2=0x7fe5080c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 --2- 172.21.15.83:0/1117787126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe508150290 0x7fe5080c1ff0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fe5080ad7d0 tx=0x7fe4f801c4f0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 -- 172.21.15.83:0/1117787126 shutdown_connections 2024-09-19T07:40:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 --2- 172.21.15.83:0/1117787126 >> v2:172.21.15.184:3300/0 conn(0x7fe5080c45c0 0x7fe508153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 --2- 172.21.15.83:0/1117787126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe508152b20 0x7fe5080c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 --2- 172.21.15.83:0/1117787126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe508150290 0x7fe5080c1ff0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 -- 172.21.15.83:0/1117787126 >> 172.21.15.83:0/1117787126 conn(0x7fe50806bc80 msgr2=0x7fe50806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 -- 172.21.15.83:0/1117787126 shutdown_connections 2024-09-19T07:40:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 -- 172.21.15.83:0/1117787126 wait complete. 2024-09-19T07:40:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 Processor -- start 2024-09-19T07:40:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 -- start start 2024-09-19T07:40:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe5080c45c0 0x7fe5080ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe508150290 0x7fe5080aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe508152b20 0x7fe50815b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe5080c2530 con 0x7fe5080c45c0 2024-09-19T07:40:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe5080c3120 con 0x7fe508152b20 2024-09-19T07:40:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe508154800 con 0x7fe508150290 2024-09-19T07:40:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe5080c45c0 0x7fe5080ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe5080c45c0 0x7fe5080ab900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57248/0 (socket says 172.21.15.83:57248) 2024-09-19T07:40:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50659c640 1 -- 172.21.15.83:0/375877637 learned_addr learned my addr 172.21.15.83:0/375877637 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe505d9b640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe508150290 0x7fe5080aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:24.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe506d9d640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe508152b20 0x7fe50815b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:24.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50659c640 1 -- 172.21.15.83:0/375877637 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe508150290 msgr2=0x7fe5080aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:24.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50659c640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe508150290 0x7fe5080aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50659c640 1 -- 172.21.15.83:0/375877637 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe508152b20 msgr2=0x7fe50815b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:24.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50659c640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe508152b20 0x7fe50815b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50659c640 1 -- 172.21.15.83:0/375877637 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5080aca80 con 0x7fe5080c45c0 2024-09-19T07:40:24.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50659c640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe5080c45c0 0x7fe5080ab900 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fe4f801cd90 tx=0x7fe4f801ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:24.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/375877637 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f80bd5f0 con 0x7fe5080c45c0 2024-09-19T07:40:24.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/375877637 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f80bd790 con 0x7fe5080c45c0 2024-09-19T07:40:24.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 -- 172.21.15.83:0/375877637 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5080aa120 con 0x7fe5080c45c0 2024-09-19T07:40:24.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/375877637 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f80bdb80 con 0x7fe5080c45c0 2024-09-19T07:40:24.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.273+0000 7fe50cd6d640 1 -- 172.21.15.83:0/375877637 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe50815bcb0 con 0x7fe5080c45c0 2024-09-19T07:40:24.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.277+0000 7fe50cd6d640 1 -- 172.21.15.83:0/375877637 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5080c1ff0 con 0x7fe5080c45c0 2024-09-19T07:40:24.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.277+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/375877637 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe4f80b33f0 con 0x7fe5080c45c0 2024-09-19T07:40:24.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.277+0000 7fe4f6ffd640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe4e80678f0 0x7fe4e8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:24.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.277+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/375877637 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 9131+0+0 (secure 0 0 0) 0x7fe4f81216d0 con 0x7fe5080c45c0 2024-09-19T07:40:24.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.277+0000 7fe505d9b640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe4e80678f0 0x7fe4e8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:24.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.277+0000 7fe505d9b640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe4e80678f0 0x7fe4e8069db0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe50806e120 tx=0x7fe4fc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:24.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.281+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/375877637 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe4f80c3060 con 0x7fe5080c45c0 2024-09-19T07:40:24.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.481+0000 7fe50cd6d640 1 -- 172.21.15.83:0/375877637 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5080c3e90 con 0x7fe5080c45c0 2024-09-19T07:40:24.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.481+0000 7fe4f6ffd640 1 -- 172.21.15.83:0/375877637 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) ==== 95+0+5713 (secure 0 0 0) 0x7fe4f80a7070 con 0x7fe5080c45c0 2024-09-19T07:40:24.480 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:24.480 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":147,"btime":"2024-09-19T07:40:24:449268+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:23.675491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:24.481 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 147 2024-09-19T07:40:24.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.485+0000 7fe50cd6d640 1 -- 172.21.15.83:0/375877637 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe4e80678f0 msgr2=0x7fe4e8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:24.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.485+0000 7fe50cd6d640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe4e80678f0 0x7fe4e8069db0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe50806e120 tx=0x7fe4fc008040 comp rx=0 tx=0).stop 2024-09-19T07:40:24.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.485+0000 7fe50cd6d640 1 -- 172.21.15.83:0/375877637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe5080c45c0 msgr2=0x7fe5080ab900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:24.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.485+0000 7fe50cd6d640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe5080c45c0 0x7fe5080ab900 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fe4f801cd90 tx=0x7fe4f801ad30 comp rx=0 tx=0).stop 2024-09-19T07:40:24.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.485+0000 7fe50cd6d640 1 -- 172.21.15.83:0/375877637 shutdown_connections 2024-09-19T07:40:24.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.485+0000 7fe50cd6d640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe4e80678f0 0x7fe4e8069db0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.485+0000 7fe50cd6d640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe508152b20 0x7fe50815b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.485+0000 7fe50cd6d640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe508150290 0x7fe5080aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.485+0000 7fe50cd6d640 1 --2- 172.21.15.83:0/375877637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe5080c45c0 0x7fe5080ab900 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:24.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.485+0000 7fe50cd6d640 1 -- 172.21.15.83:0/375877637 >> 172.21.15.83:0/375877637 conn(0x7fe50806bc80 msgr2=0x7fe508152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:24.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.485+0000 7fe50cd6d640 1 -- 172.21.15.83:0/375877637 shutdown_connections 2024-09-19T07:40:24.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:24.485+0000 7fe50cd6d640 1 -- 172.21.15.83:0/375877637 wait complete. 2024-09-19T07:40:24.500 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:40:24:449268+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 147, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 147, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5852": { "addr": "172.21.15.184:6833/1335198090", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 1335198090, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 1335198090, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5852, "incarnation": 146, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:40:23.675491+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5852 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 144, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7350": { "addr": "172.21.15.83:6835/365493751", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 365493751, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 365493751, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7350, "incarnation": 141, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:39:06.430793+0000", "qdb_cluster": [ 7350 ], "qdb_leader": 7350, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7350 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:40:26.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:26.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.665+0000 7f02caf65640 1 Processor -- start 2024-09-19T07:40:26.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.665+0000 7f02caf65640 1 -- start start 2024-09-19T07:40:26.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02caf65640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f02c406e840 0x7f02c40c2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:26.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02caf65640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02c406ddd0 con 0x7f02c4073cc0 2024-09-19T07:40:26.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02caf65640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02c4073620 con 0x7f02c406c800 2024-09-19T07:40:26.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02caf65640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f02c406ecd0 con 0x7f02c406e840 2024-09-19T07:40:26.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02c8d01640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f02c4073cc0 0x7f02c41544a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38682/0 (socket says 172.21.15.83:38682) 2024-09-19T07:40:26.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02c8d01640 1 -- 172.21.15.83:0/737340131 learned_addr learned my addr 172.21.15.83:0/737340131 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:40:26.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02c9502640 1 --2- 172.21.15.83:0/737340131 >> v2:172.21.15.184:3300/0 conn(0x7f02c406e840 0x7f02c40c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:26.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02c37fe640 1 -- 172.21.15.83:0/737340131 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3975771336 0 0) 0x7f02c406ddd0 con 0x7f02c4073cc0 2024-09-19T07:40:26.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02c37fe640 1 -- 172.21.15.83:0/737340131 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02b4003530 con 0x7f02c4073cc0 2024-09-19T07:40:26.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02c37fe640 1 -- 172.21.15.83:0/737340131 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4280449256 0 0) 0x7f02c4073620 con 0x7f02c406c800 2024-09-19T07:40:26.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02c37fe640 1 -- 172.21.15.83:0/737340131 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02c406ddd0 con 0x7f02c406c800 2024-09-19T07:40:26.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02c9502640 1 -- 172.21.15.83:0/737340131 >> v1:172.21.15.184:6789/0 conn(0x7f02c406c800 legacy=0x7f02c4072f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:26.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02c9502640 1 -- 172.21.15.83:0/737340131 >> v1:172.21.15.83:6789/0 conn(0x7f02c4073cc0 legacy=0x7f02c41544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:26.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02c9502640 1 -- 172.21.15.83:0/737340131 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c40ac7c0 con 0x7f02c406e840 2024-09-19T07:40:26.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02c9502640 1 --2- 172.21.15.83:0/737340131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02c406e840 0x7f02c40c2210 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009d90 tx=0x7f02bc01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27572ab1fa0c191b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:26.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.669+0000 7f02c37fe640 1 -- 172.21.15.83:0/737340131 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc0b2070 con 0x7f02c406e840 2024-09-19T07:40:26.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c37fe640 1 -- 172.21.15.83:0/737340131 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02bc0a57a0 con 0x7f02c406e840 2024-09-19T07:40:26.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c37fe640 1 -- 172.21.15.83:0/737340131 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc0a5b50 con 0x7f02c406e840 2024-09-19T07:40:26.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 -- 172.21.15.83:0/737340131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02c406e840 msgr2=0x7f02c40c2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:26.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 --2- 172.21.15.83:0/737340131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02c406e840 0x7f02c40c2210 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009d90 tx=0x7f02bc01c8e0 comp rx=0 tx=0).stop 2024-09-19T07:40:26.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 -- 172.21.15.83:0/737340131 shutdown_connections 2024-09-19T07:40:26.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 --2- 172.21.15.83:0/737340131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02c406e840 0x7f02c40c2210 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:26.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 -- 172.21.15.83:0/737340131 >> 172.21.15.83:0/737340131 conn(0x7f02c4075390 msgr2=0x7f02c406bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:26.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 -- 172.21.15.83:0/737340131 shutdown_connections 2024-09-19T07:40:26.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 -- 172.21.15.83:0/737340131 wait complete. 2024-09-19T07:40:26.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 Processor -- start 2024-09-19T07:40:26.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 -- start start 2024-09-19T07:40:26.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02c406c800 0x7f02c41f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:26.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02c406e840 0x7f02c41555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:26.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02c4073cc0 0x7f02c414ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:26.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f02c40713d0 con 0x7f02c406e840 2024-09-19T07:40:26.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f02c4070ea0 con 0x7f02c406c800 2024-09-19T07:40:26.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02caf65640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f02c4164410 con 0x7f02c4073cc0 2024-09-19T07:40:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c8d01640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02c406c800 0x7f02c41f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c8d01640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02c406c800 0x7f02c41f2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:45456/0 (socket says 172.21.15.83:45456) 2024-09-19T07:40:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02c406e840 0x7f02c41555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c8d01640 1 -- 172.21.15.83:0/1970323065 learned_addr learned my addr 172.21.15.83:0/1970323065 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c9502640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02c4073cc0 0x7f02c414ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c3fff640 1 -- 172.21.15.83:0/1970323065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02c4073cc0 msgr2=0x7f02c414ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c3fff640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02c4073cc0 0x7f02c414ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c3fff640 1 -- 172.21.15.83:0/1970323065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02c406c800 msgr2=0x7f02c41f2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c3fff640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02c406c800 0x7f02c41f2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c3fff640 1 -- 172.21.15.83:0/1970323065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c414a410 con 0x7f02c406e840 2024-09-19T07:40:26.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.673+0000 7f02c8d01640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02c406c800 0x7f02c41f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:40:26.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.677+0000 7f02c3fff640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02c406e840 0x7f02c41555d0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f02b000c3c0 tx=0x7f02b000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:26.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.677+0000 7f02c17fa640 1 -- 172.21.15.83:0/1970323065 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b0017070 con 0x7f02c406e840 2024-09-19T07:40:26.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.677+0000 7f02c17fa640 1 -- 172.21.15.83:0/1970323065 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02b00052f0 con 0x7f02c406e840 2024-09-19T07:40:26.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.677+0000 7f02caf65640 1 -- 172.21.15.83:0/1970323065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c40bcad0 con 0x7f02c406e840 2024-09-19T07:40:26.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.677+0000 7f02c17fa640 1 -- 172.21.15.83:0/1970323065 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b0003840 con 0x7f02c406e840 2024-09-19T07:40:26.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.677+0000 7f02caf65640 1 -- 172.21.15.83:0/1970323065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c40bc7f0 con 0x7f02c406e840 2024-09-19T07:40:26.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.677+0000 7f02caf65640 1 -- 172.21.15.83:0/1970323065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02c40bc610 con 0x7f02c406e840 2024-09-19T07:40:26.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.677+0000 7f02c17fa640 1 -- 172.21.15.83:0/1970323065 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f02b001a070 con 0x7f02c406e840 2024-09-19T07:40:26.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.677+0000 7f02c17fa640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f02a8067910 0x7f02a8069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:26.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.677+0000 7f02c17fa640 1 -- 172.21.15.83:0/1970323065 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 9131+0+0 (secure 0 0 0) 0x7f02b0085c50 con 0x7f02c406e840 2024-09-19T07:40:26.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.681+0000 7f02c8d01640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f02a8067910 0x7f02a8069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:26.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.681+0000 7f02c8d01640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f02a8067910 0x7f02a8069dd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f02b8000f70 tx=0x7f02b8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:26.685+0000 7f02c17fa640 1 -- 172.21.15.83:0/1970323065 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f02c40bc610 con 0x7f02c406e840 2024-09-19T07:40:27.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.025+0000 7f02caf65640 1 -- 172.21.15.83:0/1970323065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02c40c1d10 con 0x7f02c406e840 2024-09-19T07:40:27.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.025+0000 7f02c17fa640 1 -- 172.21.15.83:0/1970323065 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) ==== 95+0+5717 (secure 0 0 0) 0x7f02c40bc610 con 0x7f02c406e840 2024-09-19T07:40:27.022 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:27.022 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":149,"btime":"2024-09-19T07:40:26:470359+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:27.023 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 149 2024-09-19T07:40:27.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.025+0000 7f02caf65640 1 -- 172.21.15.83:0/1970323065 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f02a8067910 msgr2=0x7f02a8069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:27.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.025+0000 7f02caf65640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f02a8067910 0x7f02a8069dd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f02b8000f70 tx=0x7f02b8009040 comp rx=0 tx=0).stop 2024-09-19T07:40:27.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.025+0000 7f02caf65640 1 -- 172.21.15.83:0/1970323065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02c406e840 msgr2=0x7f02c41555d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:27.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.025+0000 7f02caf65640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02c406e840 0x7f02c41555d0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f02b000c3c0 tx=0x7f02b000c880 comp rx=0 tx=0).stop 2024-09-19T07:40:27.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.029+0000 7f02caf65640 1 -- 172.21.15.83:0/1970323065 shutdown_connections 2024-09-19T07:40:27.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.029+0000 7f02caf65640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f02a8067910 0x7f02a8069dd0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:27.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.029+0000 7f02caf65640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02c4073cc0 0x7f02c414ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:27.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.029+0000 7f02caf65640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02c406e840 0x7f02c41555d0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:27.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.029+0000 7f02caf65640 1 --2- 172.21.15.83:0/1970323065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02c406c800 0x7f02c41f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:27.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.029+0000 7f02caf65640 1 -- 172.21.15.83:0/1970323065 >> 172.21.15.83:0/1970323065 conn(0x7f02c4075390 msgr2=0x7f02c4073080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:27.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.029+0000 7f02caf65640 1 -- 172.21.15.83:0/1970323065 shutdown_connections 2024-09-19T07:40:27.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:27.029+0000 7f02caf65640 1 -- 172.21.15.83:0/1970323065 wait complete. 2024-09-19T07:40:27.053 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=7266 2024-09-19T07:40:27.053 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.c 2024-09-19T07:40:31.054 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-19T07:40:31.054 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:40:31.055 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:40:31.057 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:40:31.057 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:31.160 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:40:31.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.297+0000 7fbb38fef640 1 Processor -- start 2024-09-19T07:40:31.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.297+0000 7fbb38fef640 1 -- start start 2024-09-19T07:40:31.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.297+0000 7fbb38fef640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbb2c0d3800 0x7fbb2c0076a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:31.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.297+0000 7fbb38fef640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb2c0da270 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.297+0000 7fbb38fef640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb2c046550 con 0x7fbb2c049a30 2024-09-19T07:40:31.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.297+0000 7fbb38fef640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fbb2c046830 con 0x7fbb2c0d3800 2024-09-19T07:40:31.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.301+0000 7fbb3259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbb2c0d3800 0x7fbb2c0076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:31.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.301+0000 7fbb3259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbb2c0d3800 0x7fbb2c0076a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38994/0 (socket says 172.21.15.83:38994) 2024-09-19T07:40:31.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.301+0000 7fbb3259c640 1 -- 172.21.15.83:0/2904812018 learned_addr learned my addr 172.21.15.83:0/2904812018 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:31.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.301+0000 7fbb3159a640 1 -- 172.21.15.83:0/2904812018 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1964161986 0 0) 0x7fbb2c0da270 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.301+0000 7fbb3259c640 1 -- 172.21.15.83:0/2904812018 >> v1:172.21.15.184:6789/0 conn(0x7fbb2c049a30 legacy=0x7fbb2c0d61d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:40:31.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.301+0000 7fbb3259c640 1 -- 172.21.15.83:0/2904812018 >> v1:172.21.15.83:6789/0 conn(0x7fbb2c03f1c0 legacy=0x7fbb2c0d68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:31.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.301+0000 7fbb3259c640 1 -- 172.21.15.83:0/2904812018 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2c0da550 con 0x7fbb2c0d3800 2024-09-19T07:40:31.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.305+0000 7fbb3259c640 1 --2- 172.21.15.83:0/2904812018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb2c0d3800 0x7fbb2c0076a0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fbb24006fd0 tx=0x7fbb2401d630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d1d3dd44118b0fa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:31.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.305+0000 7fbb3159a640 1 -- 172.21.15.83:0/2904812018 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240ab430 con 0x7fbb2c0d3800 2024-09-19T07:40:31.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.305+0000 7fbb3159a640 1 -- 172.21.15.83:0/2904812018 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb240b0070 con 0x7fbb2c0d3800 2024-09-19T07:40:31.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.305+0000 7fbb3159a640 1 -- 172.21.15.83:0/2904812018 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240ab7f0 con 0x7fbb2c0d3800 2024-09-19T07:40:31.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.305+0000 7fbb38fef640 1 -- 172.21.15.83:0/2904812018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb2c0d3800 msgr2=0x7fbb2c0076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:31.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.305+0000 7fbb38fef640 1 --2- 172.21.15.83:0/2904812018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb2c0d3800 0x7fbb2c0076a0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fbb24006fd0 tx=0x7fbb2401d630 comp rx=0 tx=0).stop 2024-09-19T07:40:31.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.305+0000 7fbb38fef640 1 -- 172.21.15.83:0/2904812018 shutdown_connections 2024-09-19T07:40:31.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.305+0000 7fbb38fef640 1 --2- 172.21.15.83:0/2904812018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb2c0d3800 0x7fbb2c0076a0 secure :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fbb24006fd0 tx=0x7fbb2401d630 comp rx=0 tx=0).stop 2024-09-19T07:40:31.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.305+0000 7fbb38fef640 1 -- 172.21.15.83:0/2904812018 >> 172.21.15.83:0/2904812018 conn(0x7fbb2c0485b0 msgr2=0x7fbb2c0489c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:31.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.305+0000 7fbb38fef640 1 -- 172.21.15.83:0/2904812018 shutdown_connections 2024-09-19T07:40:31.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.305+0000 7fbb38fef640 1 -- 172.21.15.83:0/2904812018 wait complete. 2024-09-19T07:40:31.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.309+0000 7fbb38fef640 1 Processor -- start 2024-09-19T07:40:31.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.309+0000 7fbb38fef640 1 -- start start 2024-09-19T07:40:31.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb38fef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb2c03f1c0 0x7fbb2c00a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:31.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb38fef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb2c049a30 0x7fbb2c172760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:31.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb38fef640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb2c0d1b00 0x7fbb2c040ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:31.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb38fef640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c0e3e00 con 0x7fbb2c049a30 2024-09-19T07:40:31.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb38fef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c0e3c80 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb38fef640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbb2c0e3f80 con 0x7fbb2c0d1b00 2024-09-19T07:40:31.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb3259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb2c03f1c0 0x7fbb2c00a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:31.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb3259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb2c03f1c0 0x7fbb2c00a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:57826/0 (socket says 172.21.15.83:57826) 2024-09-19T07:40:31.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb3259c640 1 -- 172.21.15.83:0/522402725 learned_addr learned my addr 172.21.15.83:0/522402725 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:31.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb32d9d640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb2c0d1b00 0x7fbb2c040ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:31.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb31d9b640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb2c049a30 0x7fbb2c172760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:31.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb3259c640 1 -- 172.21.15.83:0/522402725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb2c0d1b00 msgr2=0x7fbb2c040ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:31.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb3259c640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb2c0d1b00 0x7fbb2c040ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:31.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb3259c640 1 -- 172.21.15.83:0/522402725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb2c049a30 msgr2=0x7fbb2c172760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:31.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb3259c640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb2c049a30 0x7fbb2c172760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:31.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb3259c640 1 -- 172.21.15.83:0/522402725 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2c176bf0 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb31d9b640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb2c049a30 0x7fbb2c172760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:40:31.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb3259c640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb2c03f1c0 0x7fbb2c00a780 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbb2401d0a0 tx=0x7fbb2401a3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:31.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb22ffd640 1 -- 172.21.15.83:0/522402725 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240b8070 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb22ffd640 1 -- 172.21.15.83:0/522402725 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb24004440 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb22ffd640 1 -- 172.21.15.83:0/522402725 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24006380 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb38fef640 1 -- 172.21.15.83:0/522402725 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb2c176910 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.313+0000 7fbb38fef640 1 -- 172.21.15.83:0/522402725 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb2c0d2340 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.317+0000 7fbb38fef640 1 -- 172.21.15.83:0/522402725 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb2c0da270 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.317+0000 7fbb22ffd640 1 -- 172.21.15.83:0/522402725 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbb240bb070 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.317+0000 7fbb22ffd640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbb14089900 0x7fbb1408bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:31.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.317+0000 7fbb22ffd640 1 -- 172.21.15.83:0/522402725 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 9131+0+0 (secure 0 0 0) 0x7fbb24120fd0 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.317+0000 7fbb31d9b640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbb14089900 0x7fbb1408bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:31.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.317+0000 7fbb31d9b640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbb14089900 0x7fbb1408bdc0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c007900 tx=0x7fbb28008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:31.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.321+0000 7fbb22ffd640 1 -- 172.21.15.83:0/522402725 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbb24017340 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.452 INFO:tasks.ceph.mds.d.smithi184.stderr:2024-09-19T07:40:31.453+0000 7f0b8fcf2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:40:31.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.681+0000 7fbb38fef640 1 -- 172.21.15.83:0/522402725 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb2c0d6ca0 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.681+0000 7fbb22ffd640 1 -- 172.21.15.83:0/522402725 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+6543 (secure 0 0 0) 0x7fbb240f0820 con 0x7fbb2c03f1c0 2024-09-19T07:40:31.681 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:31.681 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":150,"btime":"2024-09-19T07:40:31:394320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:31.681 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 150 2024-09-19T07:40:31.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.685+0000 7fbb38fef640 1 -- 172.21.15.83:0/522402725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbb14089900 msgr2=0x7fbb1408bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:31.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.685+0000 7fbb38fef640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbb14089900 0x7fbb1408bdc0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c007900 tx=0x7fbb28008040 comp rx=0 tx=0).stop 2024-09-19T07:40:31.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.685+0000 7fbb38fef640 1 -- 172.21.15.83:0/522402725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb2c03f1c0 msgr2=0x7fbb2c00a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:31.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.685+0000 7fbb38fef640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb2c03f1c0 0x7fbb2c00a780 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbb2401d0a0 tx=0x7fbb2401a3d0 comp rx=0 tx=0).stop 2024-09-19T07:40:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.689+0000 7fbb38fef640 1 -- 172.21.15.83:0/522402725 shutdown_connections 2024-09-19T07:40:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.689+0000 7fbb38fef640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbb14089900 0x7fbb1408bdc0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.689+0000 7fbb38fef640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb2c0d1b00 0x7fbb2c040ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.689+0000 7fbb38fef640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb2c049a30 0x7fbb2c172760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.689+0000 7fbb38fef640 1 --2- 172.21.15.83:0/522402725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb2c03f1c0 0x7fbb2c00a780 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.689+0000 7fbb38fef640 1 -- 172.21.15.83:0/522402725 >> 172.21.15.83:0/522402725 conn(0x7fbb2c0485b0 msgr2=0x7fbb2c0d58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.689+0000 7fbb38fef640 1 -- 172.21.15.83:0/522402725 shutdown_connections 2024-09-19T07:40:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:31.689+0000 7fbb38fef640 1 -- 172.21.15.83:0/522402725 wait complete. 2024-09-19T07:40:31.711 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-19T07:40:31.711 INFO:tasks.mds_thrash.fs.[a]:waiting for 46 secs before thrashing 2024-09-19T07:40:36.003 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:40:36.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:40:51.705 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:51.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.876+0000 7ff0da7df640 1 Processor -- start 2024-09-19T07:40:51.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.876+0000 7ff0da7df640 1 -- start start 2024-09-19T07:40:51.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.880+0000 7ff0da7df640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff0d40c0c80 0x7ff0d4156790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:51.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.880+0000 7ff0da7df640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d40acb90 con 0x7ff0d4153480 2024-09-19T07:40:51.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.880+0000 7ff0da7df640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d40abf80 con 0x7ff0d4153040 2024-09-19T07:40:51.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.880+0000 7ff0da7df640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff0d40ac260 con 0x7ff0d40c0c80 2024-09-19T07:40:51.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.880+0000 7ff0d3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff0d40c0c80 0x7ff0d4156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:51.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.880+0000 7ff0d3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff0d40c0c80 0x7ff0d4156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60686/0 (socket says 172.21.15.83:60686) 2024-09-19T07:40:51.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.880+0000 7ff0d3fff640 1 -- 172.21.15.83:0/3565512885 learned_addr learned my addr 172.21.15.83:0/3565512885 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:51.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.880+0000 7ff0d3fff640 1 -- 172.21.15.83:0/3565512885 >> v1:172.21.15.184:6789/0 conn(0x7ff0d4153040 legacy=0x7ff0d406e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:51.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.880+0000 7ff0d3fff640 1 -- 172.21.15.83:0/3565512885 >> v1:172.21.15.83:6789/0 conn(0x7ff0d4153480 legacy=0x7ff0d406e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:51.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.880+0000 7ff0d3fff640 1 -- 172.21.15.83:0/3565512885 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d40ac540 con 0x7ff0d40c0c80 2024-09-19T07:40:51.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.884+0000 7ff0d3fff640 1 --2- 172.21.15.83:0/3565512885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d40c0c80 0x7ff0d4156790 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009e50 tx=0x7ff0cc01d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df002060869e7446 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.884+0000 7ff0d2ffd640 1 -- 172.21.15.83:0/3565512885 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc0a5730 con 0x7ff0d40c0c80 2024-09-19T07:40:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.884+0000 7ff0d2ffd640 1 -- 172.21.15.83:0/3565512885 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0cc0a58d0 con 0x7ff0d40c0c80 2024-09-19T07:40:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.884+0000 7ff0da7df640 1 -- 172.21.15.83:0/3565512885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d40c0c80 msgr2=0x7ff0d4156790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.884+0000 7ff0da7df640 1 --2- 172.21.15.83:0/3565512885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d40c0c80 0x7ff0d4156790 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009e50 tx=0x7ff0cc01d3c0 comp rx=0 tx=0).stop 2024-09-19T07:40:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.884+0000 7ff0da7df640 1 -- 172.21.15.83:0/3565512885 shutdown_connections 2024-09-19T07:40:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.884+0000 7ff0da7df640 1 --2- 172.21.15.83:0/3565512885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d40c0c80 0x7ff0d4156790 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.884+0000 7ff0da7df640 1 -- 172.21.15.83:0/3565512885 >> 172.21.15.83:0/3565512885 conn(0x7ff0d4075390 msgr2=0x7ff0d406bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.884+0000 7ff0da7df640 1 -- 172.21.15.83:0/3565512885 shutdown_connections 2024-09-19T07:40:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.884+0000 7ff0da7df640 1 -- 172.21.15.83:0/3565512885 wait complete. 2024-09-19T07:40:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0da7df640 1 Processor -- start 2024-09-19T07:40:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0da7df640 1 -- start start 2024-09-19T07:40:51.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0da7df640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d4153040 0x7ff0d41491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0da7df640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0d4153480 0x7ff0d41496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0da7df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d408cb30 0x7ff0d40c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0da7df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d40abcb0 con 0x7ff0d408cb30 2024-09-19T07:40:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0da7df640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d40abb30 con 0x7ff0d4153040 2024-09-19T07:40:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0da7df640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d40c1060 con 0x7ff0d4153480 2024-09-19T07:40:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d8d7c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d408cb30 0x7ff0d40c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d8d7c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d408cb30 0x7ff0d40c6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60698/0 (socket says 172.21.15.83:60698) 2024-09-19T07:40:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d8d7c640 1 -- 172.21.15.83:0/1990476024 learned_addr learned my addr 172.21.15.83:0/1990476024 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d3fff640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d4153040 0x7ff0d41491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d37fe640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0d4153480 0x7ff0d41496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d8d7c640 1 -- 172.21.15.83:0/1990476024 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0d4153480 msgr2=0x7ff0d41496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d8d7c640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0d4153480 0x7ff0d41496e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d8d7c640 1 -- 172.21.15.83:0/1990476024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d4153040 msgr2=0x7ff0d41491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d8d7c640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d4153040 0x7ff0d41491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d8d7c640 1 -- 172.21.15.83:0/1990476024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d4160670 con 0x7ff0d408cb30 2024-09-19T07:40:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d37fe640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0d4153480 0x7ff0d41496e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:40:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d8d7c640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d408cb30 0x7ff0d40c6200 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400c780 tx=0x7ff0c400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d0ff9640 1 -- 172.21.15.83:0/1990476024 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4016020 con 0x7ff0d408cb30 2024-09-19T07:40:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d0ff9640 1 -- 172.21.15.83:0/1990476024 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c40050d0 con 0x7ff0d408cb30 2024-09-19T07:40:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0d0ff9640 1 -- 172.21.15.83:0/1990476024 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4005480 con 0x7ff0d408cb30 2024-09-19T07:40:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0da7df640 1 -- 172.21.15.83:0/1990476024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d40c8240 con 0x7ff0d408cb30 2024-09-19T07:40:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.888+0000 7ff0da7df640 1 -- 172.21.15.83:0/1990476024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d4073590 con 0x7ff0d408cb30 2024-09-19T07:40:51.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.892+0000 7ff0ba7fc640 1 -- 172.21.15.83:0/1990476024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0ac0052a0 con 0x7ff0d408cb30 2024-09-19T07:40:51.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.892+0000 7ff0d0ff9640 1 -- 172.21.15.83:0/1990476024 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff0c4007020 con 0x7ff0d408cb30 2024-09-19T07:40:51.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.892+0000 7ff0d0ff9640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff0c0067940 0x7ff0c0069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:51.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.892+0000 7ff0d3fff640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff0c0067940 0x7ff0c0069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:51.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.896+0000 7ff0d3fff640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff0c0067940 0x7ff0c0069e00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0ae000 tx=0x7ff0cc004ac0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:51.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.896+0000 7ff0d0ff9640 1 -- 172.21.15.83:0/1990476024 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 9131+0+0 (secure 0 0 0) 0x7ff0c40539a0 con 0x7ff0d408cb30 2024-09-19T07:40:51.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:51.904+0000 7ff0d0ff9640 1 -- 172.21.15.83:0/1990476024 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff0c4020020 con 0x7ff0d408cb30 2024-09-19T07:40:52.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.304+0000 7ff0ba7fc640 1 -- 172.21.15.83:0/1990476024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0ac0055c0 con 0x7ff0d408cb30 2024-09-19T07:40:52.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.304+0000 7ff0d0ff9640 1 -- 172.21.15.83:0/1990476024 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+6543 (secure 0 0 0) 0x7ff0c4020020 con 0x7ff0d408cb30 2024-09-19T07:40:52.304 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:52.304 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":150,"btime":"2024-09-19T07:40:31:394320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:52.304 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 150 2024-09-19T07:40:52.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.308+0000 7ff0ba7fc640 1 -- 172.21.15.83:0/1990476024 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff0c0067940 msgr2=0x7ff0c0069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:52.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.308+0000 7ff0ba7fc640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff0c0067940 0x7ff0c0069e00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0ae000 tx=0x7ff0cc004ac0 comp rx=0 tx=0).stop 2024-09-19T07:40:52.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.308+0000 7ff0ba7fc640 1 -- 172.21.15.83:0/1990476024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d408cb30 msgr2=0x7ff0d40c6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:52.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.308+0000 7ff0ba7fc640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d408cb30 0x7ff0d40c6200 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400c780 tx=0x7ff0c400cc40 comp rx=0 tx=0).stop 2024-09-19T07:40:52.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.308+0000 7ff0ba7fc640 1 -- 172.21.15.83:0/1990476024 shutdown_connections 2024-09-19T07:40:52.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.308+0000 7ff0ba7fc640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff0c0067940 0x7ff0c0069e00 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:52.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.308+0000 7ff0ba7fc640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d408cb30 0x7ff0d40c6200 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:52.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.308+0000 7ff0ba7fc640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0d4153480 0x7ff0d41496e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:52.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.308+0000 7ff0ba7fc640 1 --2- 172.21.15.83:0/1990476024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d4153040 0x7ff0d41491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:52.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.308+0000 7ff0ba7fc640 1 -- 172.21.15.83:0/1990476024 >> 172.21.15.83:0/1990476024 conn(0x7ff0d4075390 msgr2=0x7ff0d4151330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:52.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.308+0000 7ff0ba7fc640 1 -- 172.21.15.83:0/1990476024 shutdown_connections 2024-09-19T07:40:52.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.308+0000 7ff0ba7fc640 1 -- 172.21.15.83:0/1990476024 wait complete. 2024-09-19T07:40:52.347 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-19T07:40:52.347 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:40:52.348 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:40:52.367 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:40:52.367 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:52.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.612+0000 7f26c1bb6640 1 Processor -- start 2024-09-19T07:40:52.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.612+0000 7f26c1bb6640 1 -- start start 2024-09-19T07:40:52.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.612+0000 7f26c1bb6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f26bc150310 0x7f26bc0728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:52.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.612+0000 7f26c1bb6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26bc0c4990 0x7f26bc156070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:52.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.612+0000 7f26c1bb6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26bc0c2f90 con 0x7f26bc06cef0 2024-09-19T07:40:52.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.612+0000 7f26c1bb6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f26bc0c3e20 con 0x7f26bc150310 2024-09-19T07:40:52.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.612+0000 7f26c1bb6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f26bc0c3210 con 0x7f26bc0c4990 2024-09-19T07:40:52.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.612+0000 7f26baffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f26bc150310 0x7f26bc0728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:52.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.612+0000 7f26baffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f26bc150310 0x7f26bc0728d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60704/0 (socket says 172.21.15.83:60704) 2024-09-19T07:40:52.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.616+0000 7f26baffd640 1 -- 172.21.15.83:0/461948268 learned_addr learned my addr 172.21.15.83:0/461948268 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:52.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.612+0000 7f26bb7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f26bc06cef0 0x7f26bc14fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37992/0 (socket says 172.21.15.83:37992) 2024-09-19T07:40:52.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.616+0000 7f26bbfff640 1 --2- 172.21.15.83:0/461948268 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26bc0c4990 0x7f26bc156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:52.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.616+0000 7f26baffd640 1 -- 172.21.15.83:0/461948268 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26bc0c4990 msgr2=0x7f26bc156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:52.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.616+0000 7f26baffd640 1 --2- 172.21.15.83:0/461948268 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26bc0c4990 0x7f26bc156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:52.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.616+0000 7f26baffd640 1 -- 172.21.15.83:0/461948268 >> v1:172.21.15.83:6789/0 conn(0x7f26bc06cef0 legacy=0x7f26bc14fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:52.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.616+0000 7f26ba7fc640 1 -- 172.21.15.83:0/461948268 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2794370691 0 0) 0x7f26b40014a0 con 0x7f26bc06cef0 2024-09-19T07:40:52.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.616+0000 7f26baffd640 1 -- 172.21.15.83:0/461948268 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26bc0c34f0 con 0x7f26bc150310 2024-09-19T07:40:52.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.616+0000 7f26baffd640 1 --2- 172.21.15.83:0/461948268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26bc150310 0x7f26bc0728d0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00bb10 tx=0x7f26ac01eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7db842a9a195147 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:52.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26ba7fc640 1 -- 172.21.15.83:0/461948268 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac0b4070 con 0x7f26bc150310 2024-09-19T07:40:52.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26ba7fc640 1 -- 172.21.15.83:0/461948268 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ac0a7b60 con 0x7f26bc150310 2024-09-19T07:40:52.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26ba7fc640 1 -- 172.21.15.83:0/461948268 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac0af440 con 0x7f26bc150310 2024-09-19T07:40:52.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26c1bb6640 1 -- 172.21.15.83:0/461948268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26bc150310 msgr2=0x7f26bc0728d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:52.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26c1bb6640 1 --2- 172.21.15.83:0/461948268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26bc150310 0x7f26bc0728d0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00bb10 tx=0x7f26ac01eba0 comp rx=0 tx=0).stop 2024-09-19T07:40:52.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26c1bb6640 1 -- 172.21.15.83:0/461948268 shutdown_connections 2024-09-19T07:40:52.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26c1bb6640 1 --2- 172.21.15.83:0/461948268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26bc150310 0x7f26bc0728d0 secure :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00bb10 tx=0x7f26ac01eba0 comp rx=0 tx=0).stop 2024-09-19T07:40:52.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26c1bb6640 1 --2- 172.21.15.83:0/461948268 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26bc0c4990 0x7f26bc156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:52.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26c1bb6640 1 -- 172.21.15.83:0/461948268 >> 172.21.15.83:0/461948268 conn(0x7f26bc0c5090 msgr2=0x7f26bc0c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:52.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26c1bb6640 1 -- 172.21.15.83:0/461948268 shutdown_connections 2024-09-19T07:40:52.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26c1bb6640 1 -- 172.21.15.83:0/461948268 wait complete. 2024-09-19T07:40:52.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26c1bb6640 1 Processor -- start 2024-09-19T07:40:52.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.620+0000 7f26c1bb6640 1 -- start start 2024-09-19T07:40:52.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26c1bb6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26bc06cef0 0x7f26bc1497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:52.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26c1bb6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26bc0c4990 0x7f26bc14a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:52.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26c1bb6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26bc14aea0 0x7f26bc140d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:52.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26c1bb6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f26bc074f90 con 0x7f26bc14aea0 2024-09-19T07:40:52.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26c1bb6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26bc06d520 con 0x7f26bc0c4990 2024-09-19T07:40:52.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26c1bb6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f26bc075110 con 0x7f26bc06cef0 2024-09-19T07:40:52.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26baffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26bc0c4990 0x7f26bc14a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:52.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26bc14aea0 0x7f26bc140d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:52.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26baffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26bc0c4990 0x7f26bc14a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:50700/0 (socket says 172.21.15.83:50700) 2024-09-19T07:40:52.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26baffd640 1 -- 172.21.15.83:0/3463379158 learned_addr learned my addr 172.21.15.83:0/3463379158 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:52.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26bb7fe640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26bc06cef0 0x7f26bc1497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:52.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26baffd640 1 -- 172.21.15.83:0/3463379158 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26bc06cef0 msgr2=0x7f26bc1497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:52.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26baffd640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26bc06cef0 0x7f26bc1497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:52.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26baffd640 1 -- 172.21.15.83:0/3463379158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26bc14aea0 msgr2=0x7f26bc140d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:52.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26baffd640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26bc14aea0 0x7f26bc140d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:52.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26baffd640 1 -- 172.21.15.83:0/3463379158 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26bc14bd00 con 0x7f26bc0c4990 2024-09-19T07:40:52.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26bbfff640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26bc14aea0 0x7f26bc140d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:40:52.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26baffd640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26bc0c4990 0x7f26bc14a960 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00bb10 tx=0x7f26ac01d3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:52.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26a3fff640 1 -- 172.21.15.83:0/3463379158 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac0b4070 con 0x7f26bc0c4990 2024-09-19T07:40:52.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26a3fff640 1 -- 172.21.15.83:0/3463379158 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ac0afdc0 con 0x7f26bc0c4990 2024-09-19T07:40:52.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26a3fff640 1 -- 172.21.15.83:0/3463379158 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac004260 con 0x7f26bc0c4990 2024-09-19T07:40:52.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26c1bb6640 1 -- 172.21.15.83:0/3463379158 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26bc14a160 con 0x7f26bc0c4990 2024-09-19T07:40:52.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.624+0000 7f26c1bb6640 1 -- 172.21.15.83:0/3463379158 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26bc0c2810 con 0x7f26bc0c4990 2024-09-19T07:40:52.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.628+0000 7f26c1bb6640 1 -- 172.21.15.83:0/3463379158 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f269c0052a0 con 0x7f26bc0c4990 2024-09-19T07:40:52.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.628+0000 7f26a3fff640 1 -- 172.21.15.83:0/3463379158 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f26ac0c1060 con 0x7f26bc0c4990 2024-09-19T07:40:52.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.628+0000 7f26a3fff640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26a80678c0 0x7f26a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:52.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.628+0000 7f26a3fff640 1 -- 172.21.15.83:0/3463379158 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 9131+0+0 (secure 0 0 0) 0x7f26ac120ce0 con 0x7f26bc0c4990 2024-09-19T07:40:52.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.628+0000 7f26bb7fe640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26a80678c0 0x7f26a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:52.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.628+0000 7f26bb7fe640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26a80678c0 0x7f26a8069d80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f26b4004fd0 tx=0x7f26b4001260 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:52.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:52.636+0000 7f26a3fff640 1 -- 172.21.15.83:0/3463379158 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f26ac018820 con 0x7f26bc0c4990 2024-09-19T07:40:53.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.004+0000 7f26c1bb6640 1 -- 172.21.15.83:0/3463379158 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f269c005590 con 0x7f26bc0c4990 2024-09-19T07:40:53.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.008+0000 7f26a3fff640 1 -- 172.21.15.83:0/3463379158 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+6543 (secure 0 0 0) 0x7f26ac0f04f0 con 0x7f26bc0c4990 2024-09-19T07:40:53.009 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:53.009 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":150,"btime":"2024-09-19T07:40:31:394320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:53.009 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 150 2024-09-19T07:40:53.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.012+0000 7f26a1ffb640 1 -- 172.21.15.83:0/3463379158 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26a80678c0 msgr2=0x7f26a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:53.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.012+0000 7f26a1ffb640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26a80678c0 0x7f26a8069d80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f26b4004fd0 tx=0x7f26b4001260 comp rx=0 tx=0).stop 2024-09-19T07:40:53.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.012+0000 7f26a1ffb640 1 -- 172.21.15.83:0/3463379158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26bc0c4990 msgr2=0x7f26bc14a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:53.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.012+0000 7f26a1ffb640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26bc0c4990 0x7f26bc14a960 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00bb10 tx=0x7f26ac01d3e0 comp rx=0 tx=0).stop 2024-09-19T07:40:53.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.012+0000 7f26a1ffb640 1 -- 172.21.15.83:0/3463379158 shutdown_connections 2024-09-19T07:40:53.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.012+0000 7f26a1ffb640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f26a80678c0 0x7f26a8069d80 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:53.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.012+0000 7f26a1ffb640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26bc14aea0 0x7f26bc140d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:53.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.012+0000 7f26a1ffb640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26bc0c4990 0x7f26bc14a960 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:53.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.012+0000 7f26a1ffb640 1 --2- 172.21.15.83:0/3463379158 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26bc06cef0 0x7f26bc1497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:53.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.012+0000 7f26a1ffb640 1 -- 172.21.15.83:0/3463379158 >> 172.21.15.83:0/3463379158 conn(0x7f26bc0c5090 msgr2=0x7f26bc06edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:53.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.016+0000 7f26a1ffb640 1 -- 172.21.15.83:0/3463379158 shutdown_connections 2024-09-19T07:40:53.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:53.016+0000 7f26a1ffb640 1 -- 172.21.15.83:0/3463379158 wait complete. 2024-09-19T07:40:53.046 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:40:55.048 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:55.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.232+0000 7f002166a640 1 Processor -- start 2024-09-19T07:40:55.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.232+0000 7f002166a640 1 -- start start 2024-09-19T07:40:55.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.232+0000 7f002166a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f001c0c3fa0 0x7f001c070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:55.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.232+0000 7f002166a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f001c0ab3c0 0x7f001c0c4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:55.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.232+0000 7f002166a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f001c0aa230 con 0x7f001c0c3360 2024-09-19T07:40:55.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.232+0000 7f002166a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f001c074190 con 0x7f001c0c3fa0 2024-09-19T07:40:55.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.232+0000 7f002166a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f001c0c1ff0 con 0x7f001c0ab3c0 2024-09-19T07:40:55.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.232+0000 7f001b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f001c0c3360 0x7f001c06dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38006/0 (socket says 172.21.15.83:38006) 2024-09-19T07:40:55.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.232+0000 7f001b7fe640 1 -- 172.21.15.83:0/1281196896 learned_addr learned my addr 172.21.15.83:0/1281196896 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:40:55.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.232+0000 7f0013fff640 1 --2- 172.21.15.83:0/1281196896 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f001c0ab3c0 0x7f001c0c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:55.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.232+0000 7f001affd640 1 --2- 172.21.15.83:0/1281196896 >> v2:172.21.15.184:3300/0 conn(0x7f001c0c3fa0 0x7f001c070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:55.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.232+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1565330706 0 0) 0x7f001c0aa230 con 0x7f001c0c3360 2024-09-19T07:40:55.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0008007940 con 0x7f001c0c3360 2024-09-19T07:40:55.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2515924004 0 0) 0x7f0008007940 con 0x7f001c0c3360 2024-09-19T07:40:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f001c0ab3c0 msgr2=0x7f001c0c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 --2- 172.21.15.83:0/1281196896 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f001c0ab3c0 0x7f001c0c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 >> v2:172.21.15.184:3300/0 conn(0x7f001c0c3fa0 msgr2=0x7f001c070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 --2- 172.21.15.83:0/1281196896 >> v2:172.21.15.184:3300/0 conn(0x7f001c0c3fa0 0x7f001c070ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001c0c28a0 con 0x7f001c0c3360 2024-09-19T07:40:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f0014001a00 con 0x7f001c0c3360 2024-09-19T07:40:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 >> v1:172.21.15.83:6789/0 conn(0x7f001c0c3360 legacy=0x7f001c06dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 --2- 172.21.15.83:0/1281196896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0008007dd0 0x7f00080081f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 --2- 172.21.15.83:0/1281196896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0008008c20 0x7f00080090a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 --2- 172.21.15.83:0/1281196896 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f001c0ab3c0 0x7f00080095e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001affd640 1 --2- 172.21.15.83:0/1281196896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0008007dd0 0x7f00080081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f001c0c1ff0 con 0x7f0008008c20 2024-09-19T07:40:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0008007450 con 0x7f0008007dd0 2024-09-19T07:40:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0008007620 con 0x7f001c0ab3c0 2024-09-19T07:40:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001b7fe640 1 --2- 172.21.15.83:0/1281196896 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f001c0ab3c0 0x7f00080095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f0013fff640 1 --2- 172.21.15.83:0/1281196896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0008008c20 0x7f00080090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001affd640 1 -- 172.21.15.83:0/1281196896 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f001c0ab3c0 msgr2=0x7f00080095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:55.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001affd640 1 --2- 172.21.15.83:0/1281196896 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f001c0ab3c0 0x7f00080095e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:55.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001affd640 1 -- 172.21.15.83:0/1281196896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0008008c20 msgr2=0x7f00080090a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:55.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001affd640 1 --2- 172.21.15.83:0/1281196896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0008008c20 0x7f00080090a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:55.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.236+0000 7f001affd640 1 -- 172.21.15.83:0/1281196896 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0008007940 con 0x7f0008007dd0 2024-09-19T07:40:55.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.240+0000 7f001affd640 1 --2- 172.21.15.83:0/1281196896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0008007dd0 0x7f00080081f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f000c00cae0 tx=0x7f000c01cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85e819e7fa5b89f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:55.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.240+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c00c790 con 0x7f0008007dd0 2024-09-19T07:40:55.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.240+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f000c00c390 con 0x7f0008007dd0 2024-09-19T07:40:55.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.240+0000 7f001a7fc640 1 -- 172.21.15.83:0/1281196896 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c0a5a30 con 0x7f0008007dd0 2024-09-19T07:40:55.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.240+0000 7f002166a640 1 -- 172.21.15.83:0/1281196896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0008007dd0 msgr2=0x7f00080081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:55.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.240+0000 7f002166a640 1 --2- 172.21.15.83:0/1281196896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0008007dd0 0x7f00080081f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f000c00cae0 tx=0x7f000c01cd20 comp rx=0 tx=0).stop 2024-09-19T07:40:55.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.240+0000 7f001b7fe640 1 -- 172.21.15.83:0/1281196896 reap_dead start 2024-09-19T07:40:55.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.240+0000 7f002166a640 1 -- 172.21.15.83:0/1281196896 shutdown_connections 2024-09-19T07:40:55.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.240+0000 7f002166a640 1 -- 172.21.15.83:0/1281196896 >> 172.21.15.83:0/1281196896 conn(0x7f001c075390 msgr2=0x7f001c06bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:55.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.240+0000 7f002166a640 1 -- 172.21.15.83:0/1281196896 shutdown_connections 2024-09-19T07:40:55.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.240+0000 7f002166a640 1 -- 172.21.15.83:0/1281196896 wait complete. 2024-09-19T07:40:55.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.244+0000 7f002166a640 1 Processor -- start 2024-09-19T07:40:55.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.244+0000 7f002166a640 1 -- start start 2024-09-19T07:40:55.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.244+0000 7f002166a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0008007dd0 0x7f001c160590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:55.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.244+0000 7f002166a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f001c161840 0x7f001c1fb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:55.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.244+0000 7f002166a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f001c08b920 0x7f001c1fbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:55.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.244+0000 7f002166a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f001c0c4790 con 0x7f001c161840 2024-09-19T07:40:55.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.244+0000 7f002166a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f001c0c2e40 con 0x7f001c08b920 2024-09-19T07:40:55.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.244+0000 7f002166a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f001c0c4910 con 0x7f0008007dd0 2024-09-19T07:40:55.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.244+0000 7f0013fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f001c161840 0x7f001c1fb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:55.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.244+0000 7f0013fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f001c161840 0x7f001c1fb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60720/0 (socket says 172.21.15.83:60720) 2024-09-19T07:40:55.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.244+0000 7f0013fff640 1 -- 172.21.15.83:0/5425977 learned_addr learned my addr 172.21.15.83:0/5425977 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:55.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.244+0000 7f001affd640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0008007dd0 0x7f001c160590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:55.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f001affd640 1 -- 172.21.15.83:0/5425977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f001c08b920 msgr2=0x7f001c1fbb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:40:55.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f001affd640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f001c08b920 0x7f001c1fbb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:55.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f001affd640 1 -- 172.21.15.83:0/5425977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f001c161840 msgr2=0x7f001c1fb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:55.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f001affd640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f001c161840 0x7f001c1fb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:55.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f001affd640 1 -- 172.21.15.83:0/5425977 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001c1fc0c0 con 0x7f0008007dd0 2024-09-19T07:40:55.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f0013fff640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f001c161840 0x7f001c1fb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:40:55.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f001affd640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0008007dd0 0x7f001c160590 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0008008ba0 tx=0x7f000c00a3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:55.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f00137fe640 1 -- 172.21.15.83:0/5425977 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c0ba070 con 0x7f0008007dd0 2024-09-19T07:40:55.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f00137fe640 1 -- 172.21.15.83:0/5425977 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f000c00a540 con 0x7f0008007dd0 2024-09-19T07:40:55.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f00137fe640 1 -- 172.21.15.83:0/5425977 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c00c790 con 0x7f0008007dd0 2024-09-19T07:40:55.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f002166a640 1 -- 172.21.15.83:0/5425977 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f001c1ffa10 con 0x7f0008007dd0 2024-09-19T07:40:55.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f002166a640 1 -- 172.21.15.83:0/5425977 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f001c1ff210 con 0x7f0008007dd0 2024-09-19T07:40:55.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.248+0000 7f002166a640 1 -- 172.21.15.83:0/5425977 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f001c070e80 con 0x7f0008007dd0 2024-09-19T07:40:55.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.252+0000 7f00137fe640 1 -- 172.21.15.83:0/5425977 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f000c0bd070 con 0x7f0008007dd0 2024-09-19T07:40:55.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.252+0000 7f00137fe640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efffc067990 0x7efffc069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:55.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.252+0000 7f00137fe640 1 -- 172.21.15.83:0/5425977 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(54..54 src has 1..54) ==== 9131+0+0 (secure 0 0 0) 0x7f000c121330 con 0x7f0008007dd0 2024-09-19T07:40:55.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.252+0000 7f0013fff640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efffc067990 0x7efffc069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:55.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.252+0000 7f0013fff640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efffc067990 0x7efffc069e50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f001c161c40 tx=0x7f0004008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:55.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.260+0000 7f00137fe640 1 -- 172.21.15.83:0/5425977 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f000c0c4070 con 0x7f0008007dd0 2024-09-19T07:40:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.640+0000 7f002166a640 1 -- 172.21.15.83:0/5425977 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f001c06e010 con 0x7f0008007dd0 2024-09-19T07:40:55.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f00137fe640 1 -- 172.21.15.83:0/5425977 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+6543 (secure 0 0 0) 0x7f000c016a60 con 0x7f0008007dd0 2024-09-19T07:40:55.642 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:55.642 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":150,"btime":"2024-09-19T07:40:31:394320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:55.643 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 150 2024-09-19T07:40:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f002166a640 1 -- 172.21.15.83:0/5425977 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efffc067990 msgr2=0x7efffc069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f002166a640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efffc067990 0x7efffc069e50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f001c161c40 tx=0x7f0004008040 comp rx=0 tx=0).stop 2024-09-19T07:40:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f002166a640 1 -- 172.21.15.83:0/5425977 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0008007dd0 msgr2=0x7f001c160590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f002166a640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0008007dd0 0x7f001c160590 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0008008ba0 tx=0x7f000c00a3d0 comp rx=0 tx=0).stop 2024-09-19T07:40:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f002166a640 1 -- 172.21.15.83:0/5425977 shutdown_connections 2024-09-19T07:40:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f002166a640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7efffc067990 0x7efffc069e50 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f002166a640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f001c08b920 0x7f001c1fbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f002166a640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f001c161840 0x7f001c1fb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f002166a640 1 --2- 172.21.15.83:0/5425977 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0008007dd0 0x7f001c160590 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f002166a640 1 -- 172.21.15.83:0/5425977 >> 172.21.15.83:0/5425977 conn(0x7f001c075390 msgr2=0x7f001c168ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f002166a640 1 -- 172.21.15.83:0/5425977 shutdown_connections 2024-09-19T07:40:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:55.644+0000 7f002166a640 1 -- 172.21.15.83:0/5425977 wait complete. 2024-09-19T07:40:55.674 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:40:57.675 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:40:57.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f6164e9e640 1 Processor -- start 2024-09-19T07:40:57.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f6164e9e640 1 -- start start 2024-09-19T07:40:57.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f6164e9e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f616014f5a0 0x7f6160152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:57.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f6164e9e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f61600c21e0 0x7f61600721b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:57.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f6164e9e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6160153520 0x7f61600c3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:57.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f6164e9e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6160071740 con 0x7f61600c21e0 2024-09-19T07:40:57.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f615e59c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f616014f5a0 0x7f6160152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:57.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f615e59c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f616014f5a0 0x7f6160152ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:50742/0 (socket says 172.21.15.83:50742) 2024-09-19T07:40:57.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f615e59c640 1 -- 172.21.15.83:0/830975126 learned_addr learned my addr 172.21.15.83:0/830975126 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:57.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f615dd9b640 1 --2- 172.21.15.83:0/830975126 >> v2:172.21.15.83:3300/0 conn(0x7f61600c21e0 0x7f61600721b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:57.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f615ed9d640 1 --2- 172.21.15.83:0/830975126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6160153520 0x7f61600c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:57.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f6164e9e640 1 -- 172.21.15.83:0/830975126 --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f616014f980 con 0x7f616014f5a0 2024-09-19T07:40:57.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.880+0000 7f6164e9e640 1 -- 172.21.15.83:0/830975126 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f61600730c0 con 0x7f6160153520 2024-09-19T07:40:57.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.884+0000 7f615e59c640 1 -- 172.21.15.83:0/830975126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6160153520 msgr2=0x7f61600c3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:57.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.884+0000 7f615e59c640 1 --2- 172.21.15.83:0/830975126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6160153520 0x7f61600c3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:57.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.884+0000 7f615e59c640 1 -- 172.21.15.83:0/830975126 >> v2:172.21.15.83:3300/0 conn(0x7f61600c21e0 msgr2=0x7f61600721b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:40:57.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.884+0000 7f615e59c640 1 --2- 172.21.15.83:0/830975126 >> v2:172.21.15.83:3300/0 conn(0x7f61600c21e0 0x7f61600721b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:57.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.884+0000 7f615e59c640 1 -- 172.21.15.83:0/830975126 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61600719b0 con 0x7f616014f5a0 2024-09-19T07:40:57.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.884+0000 7f615dd9b640 1 --2- 172.21.15.83:0/830975126 >> v2:172.21.15.83:3300/0 conn(0x7f61600c21e0 0x7f61600721b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:40:57.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.884+0000 7f615e59c640 1 --2- 172.21.15.83:0/830975126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f616014f5a0 0x7f6160152ab0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f61600c2710 tx=0x7f615401c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5daf44f7c5204cc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:57.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.884+0000 7f615d59a640 1 -- 172.21.15.83:0/830975126 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61540b2070 con 0x7f616014f5a0 2024-09-19T07:40:57.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.884+0000 7f615d59a640 1 -- 172.21.15.83:0/830975126 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61540a5780 con 0x7f616014f5a0 2024-09-19T07:40:57.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.884+0000 7f615d59a640 1 -- 172.21.15.83:0/830975126 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615400b040 con 0x7f616014f5a0 2024-09-19T07:40:57.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.888+0000 7f6164e9e640 1 -- 172.21.15.83:0/830975126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f616014f5a0 msgr2=0x7f6160152ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:57.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.888+0000 7f6164e9e640 1 --2- 172.21.15.83:0/830975126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f616014f5a0 0x7f6160152ab0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f61600c2710 tx=0x7f615401c7f0 comp rx=0 tx=0).stop 2024-09-19T07:40:57.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.888+0000 7f6164e9e640 1 -- 172.21.15.83:0/830975126 shutdown_connections 2024-09-19T07:40:57.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.888+0000 7f6164e9e640 1 --2- 172.21.15.83:0/830975126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6160153520 0x7f61600c3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:57.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.888+0000 7f6164e9e640 1 --2- 172.21.15.83:0/830975126 >> v2:172.21.15.83:3300/0 conn(0x7f61600c21e0 0x7f61600721b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:57.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.888+0000 7f6164e9e640 1 --2- 172.21.15.83:0/830975126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f616014f5a0 0x7f6160152ab0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:57.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.888+0000 7f6164e9e640 1 -- 172.21.15.83:0/830975126 >> 172.21.15.83:0/830975126 conn(0x7f61600c4940 msgr2=0x7f61600c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:57.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.888+0000 7f6164e9e640 1 -- 172.21.15.83:0/830975126 shutdown_connections 2024-09-19T07:40:57.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.888+0000 7f6164e9e640 1 -- 172.21.15.83:0/830975126 wait complete. 2024-09-19T07:40:57.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.888+0000 7f6164e9e640 1 Processor -- start 2024-09-19T07:40:57.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.888+0000 7f6164e9e640 1 -- start start 2024-09-19T07:40:57.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f6164e9e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61600c21e0 0x7f61600c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:57.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f6164e9e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f616014f5a0 0x7f6160150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:57.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61600c21e0 0x7f61600c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:57.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61600c21e0 0x7f61600c5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60738/0 (socket says 172.21.15.83:60738) 2024-09-19T07:40:57.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615e59c640 1 -- 172.21.15.83:0/1372129363 learned_addr learned my addr 172.21.15.83:0/1372129363 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:40:57.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f6164e9e640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6160153520 0x7f6160151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:57.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f61601559b0 con 0x7f61600c21e0 2024-09-19T07:40:57.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615dd9b640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f616014f5a0 0x7f6160150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:57.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6160155830 con 0x7f6160153520 2024-09-19T07:40:57.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6160155b30 con 0x7f616014f5a0 2024-09-19T07:40:57.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615ed9d640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6160153520 0x7f6160151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:57.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615e59c640 1 -- 172.21.15.83:0/1372129363 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f616014f5a0 msgr2=0x7f6160150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:57.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615e59c640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f616014f5a0 0x7f6160150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:57.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615e59c640 1 -- 172.21.15.83:0/1372129363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6160153520 msgr2=0x7f6160151260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:57.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615e59c640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6160153520 0x7f6160151260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:57.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615e59c640 1 -- 172.21.15.83:0/1372129363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61600c7a50 con 0x7f61600c21e0 2024-09-19T07:40:57.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615ed9d640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6160153520 0x7f6160151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:40:57.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615dd9b640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f616014f5a0 0x7f6160150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:40:57.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f615e59c640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61600c21e0 0x7f61600c5a10 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f615400a1c0 tx=0x7f615401b030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:57.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f614effd640 1 -- 172.21.15.83:0/1372129363 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61540bd070 con 0x7f61600c21e0 2024-09-19T07:40:57.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f614effd640 1 -- 172.21.15.83:0/1372129363 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61540a6e40 con 0x7f61600c21e0 2024-09-19T07:40:57.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f614effd640 1 -- 172.21.15.83:0/1372129363 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61540b2040 con 0x7f61600c21e0 2024-09-19T07:40:57.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.892+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f616014b340 con 0x7f61600c21e0 2024-09-19T07:40:57.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.896+0000 7f614effd640 1 -- 172.21.15.83:0/1372129363 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f61540c0070 con 0x7f61600c21e0 2024-09-19T07:40:57.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.896+0000 7f614effd640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6140067910 0x7f6140069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:40:57.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.896+0000 7f615dd9b640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6140067910 0x7f6140069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:40:57.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.896+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f616014aa10 con 0x7f61600c21e0 2024-09-19T07:40:57.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.896+0000 7f614effd640 1 -- 172.21.15.83:0/1372129363 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 9131+0+0 (secure 0 0 0) 0x7f61540ee180 con 0x7f61600c21e0 2024-09-19T07:40:57.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.900+0000 7f615dd9b640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6140067910 0x7f6140069dd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f61600c21b0 tx=0x7f6150009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:40:57.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.900+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6160068e20 con 0x7f61600c21e0 2024-09-19T07:40:57.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:57.908+0000 7f614effd640 1 -- 172.21.15.83:0/1372129363 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f61540163e0 con 0x7f61600c21e0 2024-09-19T07:40:58.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.264+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6160072d00 con 0x7f61600c21e0 2024-09-19T07:40:58.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.264+0000 7f614effd640 1 -- 172.21.15.83:0/1372129363 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+6543 (secure 0 0 0) 0x7f61540f00e0 con 0x7f61600c21e0 2024-09-19T07:40:58.265 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:40:58.266 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":150,"btime":"2024-09-19T07:40:31:394320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:40:58.266 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 150 2024-09-19T07:40:58.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.268+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6140067910 msgr2=0x7f6140069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:58.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.268+0000 7f6164e9e640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6140067910 0x7f6140069dd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f61600c21b0 tx=0x7f6150009190 comp rx=0 tx=0).stop 2024-09-19T07:40:58.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.268+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61600c21e0 msgr2=0x7f61600c5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:40:58.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.268+0000 7f6164e9e640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61600c21e0 0x7f61600c5a10 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f615400a1c0 tx=0x7f615401b030 comp rx=0 tx=0).stop 2024-09-19T07:40:58.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.268+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 shutdown_connections 2024-09-19T07:40:58.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.268+0000 7f6164e9e640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6140067910 0x7f6140069dd0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:58.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.268+0000 7f6164e9e640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6160153520 0x7f6160151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:58.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.268+0000 7f6164e9e640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f616014f5a0 0x7f6160150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:40:58.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.268+0000 7f6164e9e640 1 --2- 172.21.15.83:0/1372129363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61600c21e0 0x7f61600c5a10 secure :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f615400a1c0 tx=0x7f615401b030 comp rx=0 tx=0).stop 2024-09-19T07:40:58.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.268+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 >> 172.21.15.83:0/1372129363 conn(0x7f61600c4940 msgr2=0x7f61600c2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:40:58.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.268+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 shutdown_connections 2024-09-19T07:40:58.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:40:58.268+0000 7f6164e9e640 1 -- 172.21.15.83:0/1372129363 wait complete. 2024-09-19T07:40:58.297 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:41:00.298 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:00.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.452+0000 7f4a3946c640 1 Processor -- start 2024-09-19T07:41:00.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a3946c640 1 -- start start 2024-09-19T07:41:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a3946c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a340c33f0 0x7f4a340c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a3946c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4a340c1680 0x7f4a340ab8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a3946c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a3406e4f0 con 0x7f4a340abde0 2024-09-19T07:41:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a3946c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4a34165bb0 con 0x7f4a340c1680 2024-09-19T07:41:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a3946c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4a34165e00 con 0x7f4a340c33f0 2024-09-19T07:41:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a38c6b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4a340abde0 0x7f4a340aa420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57746/0 (socket says 172.21.15.83:57746) 2024-09-19T07:41:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a38c6b640 1 -- 172.21.15.83:0/3304941047 learned_addr learned my addr 172.21.15.83:0/3304941047 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:41:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a337fe640 1 --2- 172.21.15.83:0/3304941047 >> v2:172.21.15.184:3300/0 conn(0x7f4a340c1680 0x7f4a340ab8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a33fff640 1 --2- 172.21.15.83:0/3304941047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a340c33f0 0x7f4a340c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a337fe640 1 -- 172.21.15.83:0/3304941047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a340c33f0 msgr2=0x7f4a340c37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a337fe640 1 --2- 172.21.15.83:0/3304941047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a340c33f0 0x7f4a340c37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:00.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a337fe640 1 -- 172.21.15.83:0/3304941047 >> v1:172.21.15.83:6789/0 conn(0x7f4a340abde0 legacy=0x7f4a340aa420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:00.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a32ffd640 1 -- 172.21.15.83:0/3304941047 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3253501816 0 0) 0x7f4a24001120 con 0x7f4a340abde0 2024-09-19T07:41:00.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.456+0000 7f4a337fe640 1 -- 172.21.15.83:0/3304941047 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a341660e0 con 0x7f4a340c1680 2024-09-19T07:41:00.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.460+0000 7f4a337fe640 1 --2- 172.21.15.83:0/3304941047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a340c1680 0x7f4a340ab8a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4a280097c0 tx=0x7f4a2801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c84c3f06ea67cfd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:00.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.460+0000 7f4a32ffd640 1 -- 172.21.15.83:0/3304941047 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a280b0070 con 0x7f4a340c1680 2024-09-19T07:41:00.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.460+0000 7f4a32ffd640 1 -- 172.21.15.83:0/3304941047 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a280ab600 con 0x7f4a340c1680 2024-09-19T07:41:00.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.460+0000 7f4a32ffd640 1 -- 172.21.15.83:0/3304941047 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a280ab9d0 con 0x7f4a340c1680 2024-09-19T07:41:00.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.460+0000 7f4a3946c640 1 -- 172.21.15.83:0/3304941047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a340c1680 msgr2=0x7f4a340ab8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.460+0000 7f4a3946c640 1 --2- 172.21.15.83:0/3304941047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a340c1680 0x7f4a340ab8a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4a280097c0 tx=0x7f4a2801c4a0 comp rx=0 tx=0).stop 2024-09-19T07:41:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.460+0000 7f4a3946c640 1 -- 172.21.15.83:0/3304941047 shutdown_connections 2024-09-19T07:41:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.460+0000 7f4a3946c640 1 --2- 172.21.15.83:0/3304941047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a340c1680 0x7f4a340ab8a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:00.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.460+0000 7f4a3946c640 1 --2- 172.21.15.83:0/3304941047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a340c33f0 0x7f4a340c37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:00.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.460+0000 7f4a3946c640 1 -- 172.21.15.83:0/3304941047 >> 172.21.15.83:0/3304941047 conn(0x7f4a340c7370 msgr2=0x7f4a340c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:00.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.464+0000 7f4a3946c640 1 -- 172.21.15.83:0/3304941047 shutdown_connections 2024-09-19T07:41:00.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.464+0000 7f4a3946c640 1 -- 172.21.15.83:0/3304941047 wait complete. 2024-09-19T07:41:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.464+0000 7f4a3946c640 1 Processor -- start 2024-09-19T07:41:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.464+0000 7f4a3946c640 1 -- start start 2024-09-19T07:41:00.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a3946c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a340abde0 0x7f4a34167db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:00.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a3946c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a340c1680 0x7f4a34073510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:00.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a3946c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a340c33f0 0x7f4a342011d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:00.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a337fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a340c1680 0x7f4a34073510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:00.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a337fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a340c1680 0x7f4a34073510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60706/0 (socket says 172.21.15.83:60706) 2024-09-19T07:41:00.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a3946c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3408c910 con 0x7f4a340c1680 2024-09-19T07:41:00.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a3946c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4a34170040 con 0x7f4a340abde0 2024-09-19T07:41:00.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a3946c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4a3408ca90 con 0x7f4a340c33f0 2024-09-19T07:41:00.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a337fe640 1 -- 172.21.15.83:0/2930860342 learned_addr learned my addr 172.21.15.83:0/2930860342 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:00.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a38c6b640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a340c33f0 0x7f4a342011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:00.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a337fe640 1 -- 172.21.15.83:0/2930860342 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a340c33f0 msgr2=0x7f4a342011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:00.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a337fe640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a340c33f0 0x7f4a342011d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:00.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a337fe640 1 -- 172.21.15.83:0/2930860342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a340abde0 msgr2=0x7f4a34167db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:00.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a337fe640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a340abde0 0x7f4a34167db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:00.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a337fe640 1 -- 172.21.15.83:0/2930860342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a34169b00 con 0x7f4a340c1680 2024-09-19T07:41:00.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a337fe640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a340c1680 0x7f4a34073510 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f4a2801a0d0 tx=0x7f4a28019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:00.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a30ff9640 1 -- 172.21.15.83:0/2930860342 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a280b0040 con 0x7f4a340c1680 2024-09-19T07:41:00.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a30ff9640 1 -- 172.21.15.83:0/2930860342 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a280b43f0 con 0x7f4a340c1680 2024-09-19T07:41:00.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a30ff9640 1 -- 172.21.15.83:0/2930860342 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a280b47c0 con 0x7f4a340c1680 2024-09-19T07:41:00.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a3946c640 1 -- 172.21.15.83:0/2930860342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a34166e80 con 0x7f4a340c1680 2024-09-19T07:41:00.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.468+0000 7f4a3946c640 1 -- 172.21.15.83:0/2930860342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a34164e80 con 0x7f4a340c1680 2024-09-19T07:41:00.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.472+0000 7f4a3946c640 1 -- 172.21.15.83:0/2930860342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a340ab8a0 con 0x7f4a340c1680 2024-09-19T07:41:00.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.480+0000 7f4a30ff9640 1 -- 172.21.15.83:0/2930860342 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4a280b4960 con 0x7f4a340c1680 2024-09-19T07:41:00.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.480+0000 7f4a30ff9640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4a200678c0 0x7f4a20069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:00.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.480+0000 7f4a33fff640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4a200678c0 0x7f4a20069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:00.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.480+0000 7f4a33fff640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4a200678c0 0x7f4a20069d80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00e190 tx=0x7f4a2c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:00.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.480+0000 7f4a30ff9640 1 -- 172.21.15.83:0/2930860342 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 9131+0+0 (secure 0 0 0) 0x7f4a281205c0 con 0x7f4a340c1680 2024-09-19T07:41:00.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.480+0000 7f4a30ff9640 1 -- 172.21.15.83:0/2930860342 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4a28016090 con 0x7f4a340c1680 2024-09-19T07:41:00.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.908+0000 7f4a3946c640 1 -- 172.21.15.83:0/2930860342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a340c48f0 con 0x7f4a340c1680 2024-09-19T07:41:00.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.912+0000 7f4a30ff9640 1 -- 172.21.15.83:0/2930860342 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+6543 (secure 0 0 0) 0x7f4a280f0380 con 0x7f4a340c1680 2024-09-19T07:41:00.914 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:00.914 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":150,"btime":"2024-09-19T07:40:31:394320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:41:00.915 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 150 2024-09-19T07:41:00.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.916+0000 7f4a1a7fc640 1 -- 172.21.15.83:0/2930860342 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4a200678c0 msgr2=0x7f4a20069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:00.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.916+0000 7f4a1a7fc640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4a200678c0 0x7f4a20069d80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00e190 tx=0x7f4a2c00b040 comp rx=0 tx=0).stop 2024-09-19T07:41:00.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.916+0000 7f4a1a7fc640 1 -- 172.21.15.83:0/2930860342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a340c1680 msgr2=0x7f4a34073510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:00.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.916+0000 7f4a1a7fc640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a340c1680 0x7f4a34073510 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f4a2801a0d0 tx=0x7f4a28019240 comp rx=0 tx=0).stop 2024-09-19T07:41:00.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.916+0000 7f4a1a7fc640 1 -- 172.21.15.83:0/2930860342 shutdown_connections 2024-09-19T07:41:00.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.916+0000 7f4a1a7fc640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4a200678c0 0x7f4a20069d80 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:00.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.916+0000 7f4a1a7fc640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a340c33f0 0x7f4a342011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:00.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.916+0000 7f4a1a7fc640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a340c1680 0x7f4a34073510 secure :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f4a2801a0d0 tx=0x7f4a28019240 comp rx=0 tx=0).stop 2024-09-19T07:41:00.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.916+0000 7f4a1a7fc640 1 --2- 172.21.15.83:0/2930860342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a340abde0 0x7f4a34167db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:00.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.916+0000 7f4a1a7fc640 1 -- 172.21.15.83:0/2930860342 >> 172.21.15.83:0/2930860342 conn(0x7f4a340c7370 msgr2=0x7f4a3416c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:00.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.916+0000 7f4a1a7fc640 1 -- 172.21.15.83:0/2930860342 shutdown_connections 2024-09-19T07:41:00.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:00.916+0000 7f4a1a7fc640 1 -- 172.21.15.83:0/2930860342 wait complete. 2024-09-19T07:41:00.938 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:41:02.940 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:03.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedf3ba640 1 Processor -- start 2024-09-19T07:41:03.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedf3ba640 1 -- start start 2024-09-19T07:41:03.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedf3ba640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feed80c6360 0x7feed8153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:03.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedf3ba640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feed80c5b90 con 0x7feed80bee30 2024-09-19T07:41:03.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedf3ba640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feed815a050 con 0x7feed80c9680 2024-09-19T07:41:03.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedf3ba640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7feed806c950 con 0x7feed80c6360 2024-09-19T07:41:03.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedd156640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feed80c6360 0x7feed8153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:03.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedc955640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7feed80bee30 0x7feed81570f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57754/0 (socket says 172.21.15.83:57754) 2024-09-19T07:41:03.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedd156640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feed80c6360 0x7feed8153050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60718/0 (socket says 172.21.15.83:60718) 2024-09-19T07:41:03.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedc955640 1 -- 172.21.15.83:0/2110951859 learned_addr learned my addr 172.21.15.83:0/2110951859 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:41:03.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedd156640 1 -- 172.21.15.83:0/2110951859 >> v1:172.21.15.184:6789/0 conn(0x7feed80c9680 legacy=0x7feed8160580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:03.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedd156640 1 -- 172.21.15.83:0/2110951859 >> v1:172.21.15.83:6789/0 conn(0x7feed80bee30 legacy=0x7feed81570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:03.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feedd156640 1 -- 172.21.15.83:0/2110951859 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed806c670 con 0x7feed80c6360 2024-09-19T07:41:03.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.160+0000 7feecffff640 1 -- 172.21.15.83:0/2110951859 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3917054401 0 0) 0x7feed80c5b90 con 0x7feed80bee30 2024-09-19T07:41:03.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.164+0000 7feedd156640 1 --2- 172.21.15.83:0/2110951859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed80c6360 0x7feed8153050 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7feed0009d90 tx=0x7feed001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5459411f8e214eb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:03.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.164+0000 7feecffff640 1 -- 172.21.15.83:0/2110951859 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed00b2070 con 0x7feed80c6360 2024-09-19T07:41:03.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.164+0000 7feecffff640 1 -- 172.21.15.83:0/2110951859 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed00a5800 con 0x7feed80c6360 2024-09-19T07:41:03.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.164+0000 7feecffff640 1 -- 172.21.15.83:0/2110951859 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed00a5bd0 con 0x7feed80c6360 2024-09-19T07:41:03.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.164+0000 7feedf3ba640 1 -- 172.21.15.83:0/2110951859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed80c6360 msgr2=0x7feed8153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:03.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.164+0000 7feedf3ba640 1 --2- 172.21.15.83:0/2110951859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed80c6360 0x7feed8153050 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7feed0009d90 tx=0x7feed001c850 comp rx=0 tx=0).stop 2024-09-19T07:41:03.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.164+0000 7feedf3ba640 1 -- 172.21.15.83:0/2110951859 shutdown_connections 2024-09-19T07:41:03.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.164+0000 7feedf3ba640 1 --2- 172.21.15.83:0/2110951859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed80c6360 0x7feed8153050 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:03.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.164+0000 7feedf3ba640 1 -- 172.21.15.83:0/2110951859 >> 172.21.15.83:0/2110951859 conn(0x7feed80722a0 msgr2=0x7feed80726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:03.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.164+0000 7feedf3ba640 1 -- 172.21.15.83:0/2110951859 shutdown_connections 2024-09-19T07:41:03.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.164+0000 7feedf3ba640 1 -- 172.21.15.83:0/2110951859 wait complete. 2024-09-19T07:41:03.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.168+0000 7feedf3ba640 1 Processor -- start 2024-09-19T07:41:03.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.168+0000 7feedf3ba640 1 -- start start 2024-09-19T07:41:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedf3ba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed80bee30 0x7feed8146c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedf3ba640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed80c6360 0x7feed81471a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedf3ba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed80c9680 0x7feed81f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedf3ba640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feed80c68c0 con 0x7feed80c9680 2024-09-19T07:41:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedf3ba640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feed80c6740 con 0x7feed80bee30 2024-09-19T07:41:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedf3ba640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feed81644c0 con 0x7feed80c6360 2024-09-19T07:41:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedc955640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed80c6360 0x7feed81471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedc955640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed80c6360 0x7feed81471a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:42208/0 (socket says 172.21.15.83:42208) 2024-09-19T07:41:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedc955640 1 -- 172.21.15.83:0/2103298832 learned_addr learned my addr 172.21.15.83:0/2103298832 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedd957640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed80c9680 0x7feed81f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:03.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedc955640 1 -- 172.21.15.83:0/2103298832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed80bee30 msgr2=0x7feed8146c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:03.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedc955640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed80bee30 0x7feed8146c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:03.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedc955640 1 -- 172.21.15.83:0/2103298832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed80c9680 msgr2=0x7feed81f6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:03.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedc955640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed80c9680 0x7feed81f6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:03.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedc955640 1 -- 172.21.15.83:0/2103298832 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed8149260 con 0x7feed80c6360 2024-09-19T07:41:03.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feedc955640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed80c6360 0x7feed81471a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feed400c410 tx=0x7feed400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:03.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feecdffb640 1 -- 172.21.15.83:0/2103298832 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed40074e0 con 0x7feed80c6360 2024-09-19T07:41:03.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.180+0000 7feecdffb640 1 -- 172.21.15.83:0/2103298832 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed40047f0 con 0x7feed80c6360 2024-09-19T07:41:03.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.184+0000 7feecdffb640 1 -- 172.21.15.83:0/2103298832 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed4007680 con 0x7feed80c6360 2024-09-19T07:41:03.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.184+0000 7feedf3ba640 1 -- 172.21.15.83:0/2103298832 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feed80c51b0 con 0x7feed80c6360 2024-09-19T07:41:03.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.184+0000 7feedf3ba640 1 -- 172.21.15.83:0/2103298832 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feed81f6eb0 con 0x7feed80c6360 2024-09-19T07:41:03.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.184+0000 7feedf3ba640 1 -- 172.21.15.83:0/2103298832 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feed80c54a0 con 0x7feed80c6360 2024-09-19T07:41:03.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.188+0000 7feecdffb640 1 -- 172.21.15.83:0/2103298832 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feed4011070 con 0x7feed80c6360 2024-09-19T07:41:03.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.188+0000 7feecdffb640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feebc0678c0 0x7feebc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:03.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.188+0000 7feecdffb640 1 -- 172.21.15.83:0/2103298832 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(54..54 src has 1..54) ==== 9131+0+0 (secure 0 0 0) 0x7feed4086b80 con 0x7feed80c6360 2024-09-19T07:41:03.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.188+0000 7feedd156640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feebc0678c0 0x7feebc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:03.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.188+0000 7feedd156640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feebc0678c0 0x7feebc069d80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7feed00af000 tx=0x7feed0004ac0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.192+0000 7feecdffb640 1 -- 172.21.15.83:0/2103298832 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7feed4056a20 con 0x7feed80c6360 2024-09-19T07:41:03.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.560+0000 7feedf3ba640 1 -- 172.21.15.83:0/2103298832 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feed8153050 con 0x7feed80c6360 2024-09-19T07:41:03.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.564+0000 7feecdffb640 1 -- 172.21.15.83:0/2103298832 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+6543 (secure 0 0 0) 0x7feed4056320 con 0x7feed80c6360 2024-09-19T07:41:03.562 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:03.562 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":150,"btime":"2024-09-19T07:40:31:394320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:41:03.563 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 150 2024-09-19T07:41:03.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.568+0000 7feedf3ba640 1 -- 172.21.15.83:0/2103298832 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feebc0678c0 msgr2=0x7feebc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:03.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.568+0000 7feedf3ba640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feebc0678c0 0x7feebc069d80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7feed00af000 tx=0x7feed0004ac0 comp rx=0 tx=0).stop 2024-09-19T07:41:03.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.568+0000 7feedf3ba640 1 -- 172.21.15.83:0/2103298832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed80c6360 msgr2=0x7feed81471a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:03.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.568+0000 7feedf3ba640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed80c6360 0x7feed81471a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feed400c410 tx=0x7feed400c8d0 comp rx=0 tx=0).stop 2024-09-19T07:41:03.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.568+0000 7feedf3ba640 1 -- 172.21.15.83:0/2103298832 shutdown_connections 2024-09-19T07:41:03.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.568+0000 7feedf3ba640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feebc0678c0 0x7feebc069d80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:03.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.568+0000 7feedf3ba640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed80c9680 0x7feed81f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:03.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.568+0000 7feedf3ba640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed80c6360 0x7feed81471a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:03.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.568+0000 7feedf3ba640 1 --2- 172.21.15.83:0/2103298832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed80bee30 0x7feed8146c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:03.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.568+0000 7feedf3ba640 1 -- 172.21.15.83:0/2103298832 >> 172.21.15.83:0/2103298832 conn(0x7feed80722a0 msgr2=0x7feed8150fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.568+0000 7feedf3ba640 1 -- 172.21.15.83:0/2103298832 shutdown_connections 2024-09-19T07:41:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:03.568+0000 7feedf3ba640 1 -- 172.21.15.83:0/2103298832 wait complete. 2024-09-19T07:41:03.589 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:41:05.590 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:05.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.768+0000 7f5a222bd640 1 Processor -- start 2024-09-19T07:41:05.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.768+0000 7f5a222bd640 1 -- start start 2024-09-19T07:41:05.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.768+0000 7f5a222bd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a1c071100 0x7f5a1c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:05.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.768+0000 7f5a222bd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5a1c0c28e0 0x7f5a1c150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:05.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.768+0000 7f5a222bd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a1c0a9e60 con 0x7f5a1c151480 2024-09-19T07:41:05.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.768+0000 7f5a222bd640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5a1c071b50 con 0x7f5a1c0c28e0 2024-09-19T07:41:05.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.768+0000 7f5a222bd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5a1c0c1ff0 con 0x7f5a1c071100 2024-09-19T07:41:05.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a2085a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5a1c151480 0x7f5a1c06dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57764/0 (socket says 172.21.15.83:57764) 2024-09-19T07:41:05.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a2085a640 1 -- 172.21.15.83:0/535055463 learned_addr learned my addr 172.21.15.83:0/535055463 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:41:05.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a1bfff640 1 --2- 172.21.15.83:0/535055463 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a1c071100 0x7f5a1c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:05.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a13fff640 1 --2- 172.21.15.83:0/535055463 >> v2:172.21.15.83:3300/0 conn(0x7f5a1c0c28e0 0x7f5a1c150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:05.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a1b7fe640 1 -- 172.21.15.83:0/535055463 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3322149313 0 0) 0x7f5a1c0a9e60 con 0x7f5a1c151480 2024-09-19T07:41:05.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a1b7fe640 1 -- 172.21.15.83:0/535055463 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a08003530 con 0x7f5a1c151480 2024-09-19T07:41:05.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a13fff640 1 -- 172.21.15.83:0/535055463 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a1c071100 msgr2=0x7f5a1c1545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:05.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a13fff640 1 --2- 172.21.15.83:0/535055463 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a1c071100 0x7f5a1c1545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:05.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a13fff640 1 -- 172.21.15.83:0/535055463 >> v1:172.21.15.83:6789/0 conn(0x7f5a1c151480 legacy=0x7f5a1c06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:05.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a13fff640 1 -- 172.21.15.83:0/535055463 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1c0aa140 con 0x7f5a1c0c28e0 2024-09-19T07:41:05.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a1bfff640 1 --2- 172.21.15.83:0/535055463 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a1c071100 0x7f5a1c1545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:41:05.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a13fff640 1 --2- 172.21.15.83:0/535055463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a1c0c28e0 0x7f5a1c150f40 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009a20 tx=0x7f5a0401c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196426f696e3c273 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:05.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a1b7fe640 1 -- 172.21.15.83:0/535055463 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a040a9640 con 0x7f5a1c0c28e0 2024-09-19T07:41:05.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a1b7fe640 1 -- 172.21.15.83:0/535055463 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a040a97e0 con 0x7f5a1c0c28e0 2024-09-19T07:41:05.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.772+0000 7f5a1b7fe640 1 -- 172.21.15.83:0/535055463 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a040a9bb0 con 0x7f5a1c0c28e0 2024-09-19T07:41:05.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.776+0000 7f5a222bd640 1 -- 172.21.15.83:0/535055463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a1c0c28e0 msgr2=0x7f5a1c150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:05.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.776+0000 7f5a222bd640 1 --2- 172.21.15.83:0/535055463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a1c0c28e0 0x7f5a1c150f40 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009a20 tx=0x7f5a0401c670 comp rx=0 tx=0).stop 2024-09-19T07:41:05.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.776+0000 7f5a222bd640 1 -- 172.21.15.83:0/535055463 shutdown_connections 2024-09-19T07:41:05.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.776+0000 7f5a222bd640 1 --2- 172.21.15.83:0/535055463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a1c0c28e0 0x7f5a1c150f40 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:05.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.776+0000 7f5a222bd640 1 --2- 172.21.15.83:0/535055463 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a1c071100 0x7f5a1c1545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:05.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.776+0000 7f5a222bd640 1 -- 172.21.15.83:0/535055463 >> 172.21.15.83:0/535055463 conn(0x7f5a1c06bc80 msgr2=0x7f5a1c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:05.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.776+0000 7f5a222bd640 1 -- 172.21.15.83:0/535055463 shutdown_connections 2024-09-19T07:41:05.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.776+0000 7f5a222bd640 1 -- 172.21.15.83:0/535055463 wait complete. 2024-09-19T07:41:05.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.776+0000 7f5a222bd640 1 Processor -- start 2024-09-19T07:41:05.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.776+0000 7f5a222bd640 1 -- start start 2024-09-19T07:41:05.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a222bd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a1c071100 0x7f5a1c0c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:05.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a222bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a1c0c28e0 0x7f5a1c14cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:05.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a1bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a1c071100 0x7f5a1c0c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:05.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a1bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a1c071100 0x7f5a1c0c6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:42228/0 (socket says 172.21.15.83:42228) 2024-09-19T07:41:05.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a1bfff640 1 -- 172.21.15.83:0/2578417348 learned_addr learned my addr 172.21.15.83:0/2578417348 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:05.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a222bd640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a1c151480 0x7f5a1c14d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:05.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a13fff640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a1c0c28e0 0x7f5a1c14cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:05.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1c0714e0 con 0x7f5a1c0c28e0 2024-09-19T07:41:05.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1c0ac7b0 con 0x7f5a1c151480 2024-09-19T07:41:05.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5a1c071660 con 0x7f5a1c071100 2024-09-19T07:41:05.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a2085a640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a1c151480 0x7f5a1c14d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:05.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a2085a640 1 -- 172.21.15.83:0/2578417348 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a1c071100 msgr2=0x7f5a1c0c6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:05.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a2085a640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a1c071100 0x7f5a1c0c6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:05.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a2085a640 1 -- 172.21.15.83:0/2578417348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a1c0c28e0 msgr2=0x7f5a1c14cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:05.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a2085a640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a1c0c28e0 0x7f5a1c14cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:05.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a2085a640 1 -- 172.21.15.83:0/2578417348 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1c152cb0 con 0x7f5a1c151480 2024-09-19T07:41:05.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a13fff640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a1c0c28e0 0x7f5a1c14cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:41:05.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a2085a640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a1c151480 0x7f5a1c14d2e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400ca70 tx=0x7f5a1400cf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:05.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a197fa640 1 -- 172.21.15.83:0/2578417348 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a14017070 con 0x7f5a1c151480 2024-09-19T07:41:05.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a197fa640 1 -- 172.21.15.83:0/2578417348 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a1400ab70 con 0x7f5a1c151480 2024-09-19T07:41:05.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a197fa640 1 -- 172.21.15.83:0/2578417348 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a14012400 con 0x7f5a1c151480 2024-09-19T07:41:05.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a1c152f90 con 0x7f5a1c151480 2024-09-19T07:41:05.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.780+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a1c1472d0 con 0x7f5a1c151480 2024-09-19T07:41:05.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.784+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a1c1545e0 con 0x7f5a1c151480 2024-09-19T07:41:05.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.784+0000 7f5a197fa640 1 -- 172.21.15.83:0/2578417348 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5a1401a070 con 0x7f5a1c151480 2024-09-19T07:41:05.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.784+0000 7f5a197fa640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59fc0678c0 0x7f59fc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:05.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.784+0000 7f5a197fa640 1 -- 172.21.15.83:0/2578417348 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 9131+0+0 (secure 0 0 0) 0x7f5a140866b0 con 0x7f5a1c151480 2024-09-19T07:41:05.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.784+0000 7f5a1bfff640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59fc0678c0 0x7f59fc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:05.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.784+0000 7f5a1bfff640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59fc0678c0 0x7f59fc069d80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009600 tx=0x7f5a0c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:05.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:05.788+0000 7f5a197fa640 1 -- 172.21.15.83:0/2578417348 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5a14055e20 con 0x7f5a1c151480 2024-09-19T07:41:06.052 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:41:06.055 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:41:06.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.296+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a1c1529a0 con 0x7f5a1c151480 2024-09-19T07:41:06.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.296+0000 7f5a197fa640 1 -- 172.21.15.83:0/2578417348 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+6543 (secure 0 0 0) 0x7f5a1405a120 con 0x7f5a1c151480 2024-09-19T07:41:06.297 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:06.297 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":150,"btime":"2024-09-19T07:40:31:394320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:39:06.430793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":16,"addr":"172.21.15.83:6835/365493751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":365493751},{"type":"v1","addr":"172.21.15.83:6835","nonce":365493751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":2}]} 2024-09-19T07:41:06.297 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 150 2024-09-19T07:41:06.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.300+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59fc0678c0 msgr2=0x7f59fc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:06.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.300+0000 7f5a222bd640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59fc0678c0 0x7f59fc069d80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009600 tx=0x7f5a0c009190 comp rx=0 tx=0).stop 2024-09-19T07:41:06.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.300+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a1c151480 msgr2=0x7f5a1c14d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:06.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.300+0000 7f5a222bd640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a1c151480 0x7f5a1c14d2e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400ca70 tx=0x7f5a1400cf30 comp rx=0 tx=0).stop 2024-09-19T07:41:06.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.300+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 shutdown_connections 2024-09-19T07:41:06.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.300+0000 7f5a222bd640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59fc0678c0 0x7f59fc069d80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:06.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.300+0000 7f5a222bd640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a1c151480 0x7f5a1c14d2e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:06.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.300+0000 7f5a222bd640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a1c0c28e0 0x7f5a1c14cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:06.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.300+0000 7f5a222bd640 1 --2- 172.21.15.83:0/2578417348 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a1c071100 0x7f5a1c0c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:06.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.300+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 >> 172.21.15.83:0/2578417348 conn(0x7f5a1c06bc80 msgr2=0x7f5a1c06dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:06.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.300+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 shutdown_connections 2024-09-19T07:41:06.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:06.300+0000 7f5a222bd640 1 -- 172.21.15.83:0/2578417348 wait complete. 2024-09-19T07:41:06.326 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:41:08.327 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:08.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.632+0000 7f1f3499e640 1 Processor -- start 2024-09-19T07:41:08.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.632+0000 7f1f3499e640 1 -- start start 2024-09-19T07:41:08.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.636+0000 7f1f3499e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1f280f5010 0x7f1f280491a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.636+0000 7f1f2e59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1f2800aca0 0x7f1f28005b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57780/0 (socket says 172.21.15.83:57780) 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.636+0000 7f1f2e59c640 1 -- 172.21.15.83:0/2714275997 learned_addr learned my addr 172.21.15.83:0/2714275997 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.636+0000 7f1f3499e640 1 -- 172.21.15.83:0/2714275997 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f2800b4f0 con 0x7f1f2800aca0 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.636+0000 7f1f3499e640 1 -- 172.21.15.83:0/2714275997 --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f280f7a50 con 0x7f1f2800b0d0 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.636+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 117353294 0 0) 0x7f1f2800b4f0 con 0x7f1f2800aca0 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.636+0000 7f1f3499e640 1 -- 172.21.15.83:0/2714275997 --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f1f280f7d30 con 0x7f1f280f5010 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.636+0000 7f1f2ed9d640 1 --2- 172.21.15.83:0/2714275997 >> v2:172.21.15.83:3300/0 conn(0x7f1f280f5010 0x7f1f280491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f18003450 con 0x7f1f2800aca0 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3347096861 0 0) 0x7f1f280f7a50 con 0x7f1f2800b0d0 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f2800b4f0 con 0x7f1f2800b0d0 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1433500030 0 0) 0x7f1f18003450 con 0x7f1f2800aca0 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 >> v2:172.21.15.83:3300/0 conn(0x7f1f280f5010 msgr2=0x7f1f280491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:08.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 --2- 172.21.15.83:0/2714275997 >> v2:172.21.15.83:3300/0 conn(0x7f1f280f5010 0x7f1f280491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:08.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 >> v1:172.21.15.184:6789/0 conn(0x7f1f2800b0d0 legacy=0x7f1f28006220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:08.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f280f8010 con 0x7f1f2800aca0 2024-09-19T07:41:08.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f1f24001680 con 0x7f1f2800aca0 2024-09-19T07:41:08.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 >> v1:172.21.15.83:6789/0 conn(0x7f1f2800aca0 legacy=0x7f1f28005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:08.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 --2- 172.21.15.83:0/2714275997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f18003d20 0x7f1f18006130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:08.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 --2- 172.21.15.83:0/2714275997 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18006670 0x7f1f18006af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:08.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 --2- 172.21.15.83:0/2714275997 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f18007140 0x7f1f180075c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:08.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1f280f7d30 con 0x7f1f18006670 2024-09-19T07:41:08.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f18007ea0 con 0x7f1f18003d20 2024-09-19T07:41:08.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f180080a0 con 0x7f1f18007140 2024-09-19T07:41:08.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.640+0000 7f1f2e59c640 1 --2- 172.21.15.83:0/2714275997 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18006670 0x7f1f18006af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:08.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.644+0000 7f1f2f59e640 1 --2- 172.21.15.83:0/2714275997 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f18007140 0x7f1f180075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:08.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.644+0000 7f1f2ed9d640 1 --2- 172.21.15.83:0/2714275997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f18003d20 0x7f1f18006130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:08.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.644+0000 7f1f2e59c640 1 -- 172.21.15.83:0/2714275997 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f18007140 msgr2=0x7f1f180075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:08.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.644+0000 7f1f2e59c640 1 --2- 172.21.15.83:0/2714275997 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f18007140 0x7f1f180075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:08.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.644+0000 7f1f2e59c640 1 -- 172.21.15.83:0/2714275997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f18003d20 msgr2=0x7f1f18006130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:08.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.644+0000 7f1f2e59c640 1 --2- 172.21.15.83:0/2714275997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f18003d20 0x7f1f18006130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:08.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.644+0000 7f1f2e59c640 1 -- 172.21.15.83:0/2714275997 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f280f8010 con 0x7f1f18006670 2024-09-19T07:41:08.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.644+0000 7f1f2f59e640 1 -- 172.21.15.83:0/2714275997 reap_dead start 2024-09-19T07:41:08.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.644+0000 7f1f2e59c640 1 --2- 172.21.15.83:0/2714275997 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18006670 0x7f1f18006af0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400c340 tx=0x7f1f2401cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6f9e572c4c0098 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:08.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.644+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f24007dc0 con 0x7f1f18006670 2024-09-19T07:41:08.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.644+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f240a9020 con 0x7f1f18006670 2024-09-19T07:41:08.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.644+0000 7f1f2dd9b640 1 -- 172.21.15.83:0/2714275997 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f24005ae0 con 0x7f1f18006670 2024-09-19T07:41:08.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.648+0000 7f1f3499e640 1 -- 172.21.15.83:0/2714275997 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18006670 msgr2=0x7f1f18006af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:08.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.648+0000 7f1f3499e640 1 --2- 172.21.15.83:0/2714275997 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18006670 0x7f1f18006af0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400c340 tx=0x7f1f2401cce0 comp rx=0 tx=0).stop 2024-09-19T07:41:08.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.648+0000 7f1f3499e640 1 -- 172.21.15.83:0/2714275997 shutdown_connections 2024-09-19T07:41:08.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.648+0000 7f1f3499e640 1 --2- 172.21.15.83:0/2714275997 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18006670 0x7f1f18006af0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:08.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.648+0000 7f1f3499e640 1 -- 172.21.15.83:0/2714275997 >> 172.21.15.83:0/2714275997 conn(0x7f1f28047d20 msgr2=0x7f1f28048130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:08.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.648+0000 7f1f3499e640 1 -- 172.21.15.83:0/2714275997 shutdown_connections 2024-09-19T07:41:08.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.648+0000 7f1f3499e640 1 -- 172.21.15.83:0/2714275997 wait complete. 2024-09-19T07:41:08.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.664+0000 7f1f3499e640 1 Processor -- start 2024-09-19T07:41:08.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.664+0000 7f1f3499e640 1 -- start start 2024-09-19T07:41:08.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.668+0000 7f1f3499e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f18006670 0x7f1f280f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:08.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.668+0000 7f1f3499e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f280fba50 0x7f1f28195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:08.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.668+0000 7f1f3499e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f281962e0 0x7f1f280e9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:08.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.668+0000 7f1f3499e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1f280f53f0 con 0x7f1f281962e0 2024-09-19T07:41:08.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.668+0000 7f1f3499e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f2802aa90 con 0x7f1f18006670 2024-09-19T07:41:08.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.668+0000 7f1f3499e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f28199d00 con 0x7f1f280fba50 2024-09-19T07:41:08.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.668+0000 7f1f2f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f281962e0 0x7f1f280e9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:08.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.668+0000 7f1f2f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f281962e0 0x7f1f280e9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60768/0 (socket says 172.21.15.83:60768) 2024-09-19T07:41:08.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.668+0000 7f1f2f59e640 1 -- 172.21.15.83:0/3711322905 learned_addr learned my addr 172.21.15.83:0/3711322905 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:08.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.668+0000 7f1f2ed9d640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f18006670 0x7f1f280f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:08.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.668+0000 7f1f2e59c640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f280fba50 0x7f1f28195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:08.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f2f59e640 1 -- 172.21.15.83:0/3711322905 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f280fba50 msgr2=0x7f1f28195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:08.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f2f59e640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f280fba50 0x7f1f28195da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:08.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f2f59e640 1 -- 172.21.15.83:0/3711322905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f18006670 msgr2=0x7f1f280f6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:08.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f2f59e640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f18006670 0x7f1f280f6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:08.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f2f59e640 1 -- 172.21.15.83:0/3711322905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f28199fa0 con 0x7f1f281962e0 2024-09-19T07:41:08.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f2e59c640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f280fba50 0x7f1f28195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:41:08.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f2f59e640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f281962e0 0x7f1f280e9c90 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f1f30074560 tx=0x7f1f300734c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:08.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f177fe640 1 -- 172.21.15.83:0/3711322905 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3006caf0 con 0x7f1f281962e0 2024-09-19T07:41:08.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f177fe640 1 -- 172.21.15.83:0/3711322905 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f30073d50 con 0x7f1f281962e0 2024-09-19T07:41:08.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f177fe640 1 -- 172.21.15.83:0/3711322905 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30067a70 con 0x7f1f281962e0 2024-09-19T07:41:08.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f3499e640 1 -- 172.21.15.83:0/3711322905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f280f9fa0 con 0x7f1f281962e0 2024-09-19T07:41:08.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f3499e640 1 -- 172.21.15.83:0/3711322905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f280f7020 con 0x7f1f281962e0 2024-09-19T07:41:08.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f3499e640 1 -- 172.21.15.83:0/3711322905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f280491a0 con 0x7f1f281962e0 2024-09-19T07:41:08.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f177fe640 1 -- 172.21.15.83:0/3711322905 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1f30067c10 con 0x7f1f281962e0 2024-09-19T07:41:08.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f177fe640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f1c0678f0 0x7f1f1c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:08.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.672+0000 7f1f2ed9d640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f1c0678f0 0x7f1f1c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:08.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.676+0000 7f1f2ed9d640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f1c0678f0 0x7f1f1c069db0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1f28047730 tx=0x7f1f20008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:08.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.680+0000 7f1f177fe640 1 -- 172.21.15.83:0/3711322905 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9217+0+0 (secure 0 0 0) 0x7f1f300f1be0 con 0x7f1f281962e0 2024-09-19T07:41:08.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:08.684+0000 7f1f177fe640 1 -- 172.21.15.83:0/3711322905 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1f30081070 con 0x7f1f281962e0 2024-09-19T07:41:09.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.116+0000 7f1f3499e640 1 -- 172.21.15.83:0/3711322905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f280f2620 con 0x7f1f281962e0 2024-09-19T07:41:09.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.120+0000 7f1f177fe640 1 -- 172.21.15.83:0/3711322905 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5712 (secure 0 0 0) 0x7f1f300c4170 con 0x7f1f281962e0 2024-09-19T07:41:09.118 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:09.118 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":151,"btime":"2024-09-19T07:41:08:240079+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:08.240075+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:41:09.118 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 151 2024-09-19T07:41:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.124+0000 7f1f3499e640 1 -- 172.21.15.83:0/3711322905 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f1c0678f0 msgr2=0x7f1f1c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.124+0000 7f1f3499e640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f1c0678f0 0x7f1f1c069db0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1f28047730 tx=0x7f1f20008040 comp rx=0 tx=0).stop 2024-09-19T07:41:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.124+0000 7f1f3499e640 1 -- 172.21.15.83:0/3711322905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f281962e0 msgr2=0x7f1f280e9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.124+0000 7f1f3499e640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f281962e0 0x7f1f280e9c90 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f1f30074560 tx=0x7f1f300734c0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.124+0000 7f1f3499e640 1 -- 172.21.15.83:0/3711322905 shutdown_connections 2024-09-19T07:41:09.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.124+0000 7f1f3499e640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f1c0678f0 0x7f1f1c069db0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.124+0000 7f1f3499e640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f281962e0 0x7f1f280e9c90 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.124+0000 7f1f3499e640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f280fba50 0x7f1f28195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.124+0000 7f1f3499e640 1 --2- 172.21.15.83:0/3711322905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f18006670 0x7f1f280f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.124+0000 7f1f3499e640 1 -- 172.21.15.83:0/3711322905 >> 172.21.15.83:0/3711322905 conn(0x7f1f28047d20 msgr2=0x7f1f280059f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:09.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.124+0000 7f1f3499e640 1 -- 172.21.15.83:0/3711322905 shutdown_connections 2024-09-19T07:41:09.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.124+0000 7f1f3499e640 1 -- 172.21.15.83:0/3711322905 wait complete. 2024-09-19T07:41:09.168 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-19T07:41:09.168 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:41:09.168 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:09.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4648028640 1 Processor -- start 2024-09-19T07:41:09.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4648028640 1 -- start start 2024-09-19T07:41:09.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4648028640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f46400c2520 0x7f46400c3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:09.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4648028640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f464014f670 0x7f464006e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:09.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4648028640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4640152d60 con 0x7f464014f200 2024-09-19T07:41:09.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4648028640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f464006ecf0 con 0x7f46400c2520 2024-09-19T07:41:09.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4648028640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f46400c37d0 con 0x7f464014f670 2024-09-19T07:41:09.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4645dc4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f46400c2520 0x7f46400c3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:09.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4645dc4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f46400c2520 0x7f46400c3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60778/0 (socket says 172.21.15.83:60778) 2024-09-19T07:41:09.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4645dc4640 1 -- 172.21.15.83:0/2601174886 learned_addr learned my addr 172.21.15.83:0/2601174886 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:09.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4645dc4640 1 -- 172.21.15.83:0/2601174886 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f464014f670 msgr2=0x7f464006e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:09.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f46465c5640 1 --2- 172.21.15.83:0/2601174886 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f464014f670 0x7f464006e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:09.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4645dc4640 1 --2- 172.21.15.83:0/2601174886 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f464014f670 0x7f464006e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4645dc4640 1 -- 172.21.15.83:0/2601174886 >> v1:172.21.15.184:6789/0 conn(0x7f464014f200 legacy=0x7f464006e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:09.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f4645dc4640 1 -- 172.21.15.83:0/2601174886 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46401521e0 con 0x7f46400c2520 2024-09-19T07:41:09.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.412+0000 7f46465c5640 1 --2- 172.21.15.83:0/2601174886 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f464014f670 0x7f464006e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:41:09.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.416+0000 7f4645dc4640 1 --2- 172.21.15.83:0/2601174886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46400c2520 0x7f46400c3290 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f463c007360 tx=0x7f463c01d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409843a816a3cf6b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:09.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.420+0000 7f4644dc2640 1 -- 172.21.15.83:0/2601174886 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c0b1070 con 0x7f46400c2520 2024-09-19T07:41:09.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.420+0000 7f4644dc2640 1 -- 172.21.15.83:0/2601174886 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f463c008be0 con 0x7f46400c2520 2024-09-19T07:41:09.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.420+0000 7f4644dc2640 1 -- 172.21.15.83:0/2601174886 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c00a0d0 con 0x7f46400c2520 2024-09-19T07:41:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.420+0000 7f4648028640 1 -- 172.21.15.83:0/2601174886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46400c2520 msgr2=0x7f46400c3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.420+0000 7f4648028640 1 --2- 172.21.15.83:0/2601174886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46400c2520 0x7f46400c3290 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f463c007360 tx=0x7f463c01d4a0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.420+0000 7f4648028640 1 -- 172.21.15.83:0/2601174886 shutdown_connections 2024-09-19T07:41:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.420+0000 7f4648028640 1 --2- 172.21.15.83:0/2601174886 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f464014f670 0x7f464006e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.420+0000 7f4648028640 1 --2- 172.21.15.83:0/2601174886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46400c2520 0x7f46400c3290 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.420+0000 7f4648028640 1 -- 172.21.15.83:0/2601174886 >> 172.21.15.83:0/2601174886 conn(0x7f4640075390 msgr2=0x7f464006bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:09.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.420+0000 7f4648028640 1 -- 172.21.15.83:0/2601174886 shutdown_connections 2024-09-19T07:41:09.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.420+0000 7f4648028640 1 -- 172.21.15.83:0/2601174886 wait complete. 2024-09-19T07:41:09.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f4648028640 1 Processor -- start 2024-09-19T07:41:09.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f4648028640 1 -- start start 2024-09-19T07:41:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f4648028640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46400c2520 0x7f46401ee9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f4648028640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464014f200 0x7f46401eef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f4648028640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f464014f670 0x7f464015afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f4648028640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4640151f80 con 0x7f464014f200 2024-09-19T07:41:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f4648028640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4640151e00 con 0x7f464014f670 2024-09-19T07:41:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f4648028640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f46400aa000 con 0x7f46400c2520 2024-09-19T07:41:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f4645dc4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46400c2520 0x7f46401ee9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f4645dc4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46400c2520 0x7f46401ee9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:42274/0 (socket says 172.21.15.83:42274) 2024-09-19T07:41:09.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f4645dc4640 1 -- 172.21.15.83:0/3290850263 learned_addr learned my addr 172.21.15.83:0/3290850263 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:09.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f46455c3640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464014f200 0x7f46401eef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:09.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.424+0000 7f4645dc4640 1 -- 172.21.15.83:0/3290850263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f464014f670 msgr2=0x7f464015afe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:41:09.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.428+0000 7f4645dc4640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f464014f670 0x7f464015afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.428+0000 7f4645dc4640 1 -- 172.21.15.83:0/3290850263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464014f200 msgr2=0x7f46401eef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:09.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.428+0000 7f4645dc4640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464014f200 0x7f46401eef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.428+0000 7f4645dc4640 1 -- 172.21.15.83:0/3290850263 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46400c5fa0 con 0x7f46400c2520 2024-09-19T07:41:09.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.428+0000 7f4645dc4640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46400c2520 0x7f46401ee9f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f463c007360 tx=0x7f463c01a240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:09.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.428+0000 7f46367fc640 1 -- 172.21.15.83:0/3290850263 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c0bb030 con 0x7f46400c2520 2024-09-19T07:41:09.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.428+0000 7f46367fc640 1 -- 172.21.15.83:0/3290850263 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f463c00a210 con 0x7f46400c2520 2024-09-19T07:41:09.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.428+0000 7f46367fc640 1 -- 172.21.15.83:0/3290850263 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c0b1040 con 0x7f46400c2520 2024-09-19T07:41:09.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.428+0000 7f4648028640 1 -- 172.21.15.83:0/3290850263 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46400c6280 con 0x7f46400c2520 2024-09-19T07:41:09.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.428+0000 7f4648028640 1 -- 172.21.15.83:0/3290850263 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f46400c6ea0 con 0x7f46400c2520 2024-09-19T07:41:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.432+0000 7f46367fc640 1 -- 172.21.15.83:0/3290850263 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f463c0be070 con 0x7f46400c2520 2024-09-19T07:41:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.432+0000 7f4648028640 1 -- 172.21.15.83:0/3290850263 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46400c2900 con 0x7f46400c2520 2024-09-19T07:41:09.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.432+0000 7f46367fc640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4624067910 0x7f4624069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.432+0000 7f46367fc640 1 -- 172.21.15.83:0/3290850263 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 9217+0+0 (secure 0 0 0) 0x7f463c120c50 con 0x7f46400c2520 2024-09-19T07:41:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.432+0000 7f46455c3640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4624067910 0x7f4624069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:09.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.432+0000 7f46455c3640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4624067910 0x7f4624069dd0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f464014b420 tx=0x7f4630009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:09.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.440+0000 7f46367fc640 1 -- 172.21.15.83:0/3290850263 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f463c0c5070 con 0x7f46400c2520 2024-09-19T07:41:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.856+0000 7f4648028640 1 -- 172.21.15.83:0/3290850263 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46400c4cc0 con 0x7f46400c2520 2024-09-19T07:41:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.856+0000 7f46367fc640 1 -- 172.21.15.83:0/3290850263 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+5716 (secure 0 0 0) 0x7f463c017120 con 0x7f46400c2520 2024-09-19T07:41:09.855 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:09.855 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":152,"btime":"2024-09-19T07:41:09:419370+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:09.409318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:41:09.855 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 152 2024-09-19T07:41:09.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.860+0000 7f4648028640 1 -- 172.21.15.83:0/3290850263 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4624067910 msgr2=0x7f4624069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:09.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.860+0000 7f4648028640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4624067910 0x7f4624069dd0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f464014b420 tx=0x7f4630009040 comp rx=0 tx=0).stop 2024-09-19T07:41:09.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.860+0000 7f4648028640 1 -- 172.21.15.83:0/3290850263 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46400c2520 msgr2=0x7f46401ee9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:09.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.860+0000 7f4648028640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46400c2520 0x7f46401ee9f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f463c007360 tx=0x7f463c01a240 comp rx=0 tx=0).stop 2024-09-19T07:41:09.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.864+0000 7f4648028640 1 -- 172.21.15.83:0/3290850263 shutdown_connections 2024-09-19T07:41:09.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.864+0000 7f4648028640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4624067910 0x7f4624069dd0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.864+0000 7f4648028640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f464014f670 0x7f464015afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.864+0000 7f4648028640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464014f200 0x7f46401eef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.864+0000 7f4648028640 1 --2- 172.21.15.83:0/3290850263 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46400c2520 0x7f46401ee9f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:09.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.864+0000 7f4648028640 1 -- 172.21.15.83:0/3290850263 >> 172.21.15.83:0/3290850263 conn(0x7f4640075390 msgr2=0x7f464014dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:09.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.864+0000 7f4648028640 1 -- 172.21.15.83:0/3290850263 shutdown_connections 2024-09-19T07:41:09.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:09.864+0000 7f4648028640 1 -- 172.21.15.83:0/3290850263 wait complete. 2024-09-19T07:41:09.886 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:41:09:419370+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 152, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 149, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5852": { "addr": "172.21.15.184:6833/1335198090", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 1335198090, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 1335198090, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5852, "incarnation": 146, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:40:26.470349+0000", "qdb_cluster": [ 5852 ], "qdb_leader": 5852, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5852 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 152, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7503": { "addr": "172.21.15.83:6837/1990119490", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 1990119490, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 1990119490, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7503, "incarnation": 151, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:41:09.409318+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7503 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:41:11.887 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:12.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.096+0000 7febbb7b6640 1 Processor -- start 2024-09-19T07:41:12.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.096+0000 7febbb7b6640 1 -- start start 2024-09-19T07:41:12.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.096+0000 7febbb7b6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febb40b0d00 0x7febb4159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:12.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.096+0000 7febbb7b6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7febb40c9660 0x7febb41563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:12.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.096+0000 7febbb7b6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febb40c6180 con 0x7febb4156910 2024-09-19T07:41:12.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.096+0000 7febbb7b6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7febb40c69b0 con 0x7febb40c9660 2024-09-19T07:41:12.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.096+0000 7febbb7b6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7febb415a130 con 0x7febb40b0d00 2024-09-19T07:41:12.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.096+0000 7febb9552640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febb40b0d00 0x7febb4159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:12.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.096+0000 7febb9552640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febb40b0d00 0x7febb4159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:42778/0 (socket says 172.21.15.83:42778) 2024-09-19T07:41:12.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.096+0000 7febb9552640 1 -- 172.21.15.83:0/1160535527 learned_addr learned my addr 172.21.15.83:0/1160535527 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:12.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.096+0000 7febb9d53640 1 --1- 172.21.15.83:0/1160535527 >> v1:172.21.15.184:6789/0 conn(0x7febb4156910 0x7febb40c8210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:48430/0 (socket says 172.21.15.83:48430) 2024-09-19T07:41:12.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.100+0000 7febb8d51640 1 --2- 172.21.15.83:0/1160535527 >> v2:172.21.15.83:3300/0 conn(0x7febb40c9660 0x7febb41563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:12.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.100+0000 7febabfff640 1 -- 172.21.15.83:0/1160535527 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3317368354 0 0) 0x7febb40c6180 con 0x7febb4156910 2024-09-19T07:41:12.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.100+0000 7febabfff640 1 -- 172.21.15.83:0/1160535527 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feba0003530 con 0x7febb4156910 2024-09-19T07:41:12.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.100+0000 7febb9552640 1 -- 172.21.15.83:0/1160535527 >> v2:172.21.15.83:3300/0 conn(0x7febb40c9660 msgr2=0x7febb41563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:12.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.100+0000 7febb9552640 1 --2- 172.21.15.83:0/1160535527 >> v2:172.21.15.83:3300/0 conn(0x7febb40c9660 0x7febb41563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:12.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.100+0000 7febb9552640 1 -- 172.21.15.83:0/1160535527 >> v1:172.21.15.184:6789/0 conn(0x7febb4156910 legacy=0x7febb40c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:12.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.100+0000 7febb9552640 1 -- 172.21.15.83:0/1160535527 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb4155bf0 con 0x7febb40b0d00 2024-09-19T07:41:12.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.100+0000 7febb8d51640 1 --2- 172.21.15.83:0/1160535527 >> v2:172.21.15.83:3300/0 conn(0x7febb40c9660 0x7febb41563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:41:12.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.100+0000 7febb9552640 1 --2- 172.21.15.83:0/1160535527 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febb40b0d00 0x7febb4159bf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7febb0009870 tx=0x7febb001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c1f6fd31750a0ac server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:12.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.100+0000 7febabfff640 1 -- 172.21.15.83:0/1160535527 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb00b0070 con 0x7febb40b0d00 2024-09-19T07:41:12.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.104+0000 7febabfff640 1 -- 172.21.15.83:0/1160535527 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb00ab600 con 0x7febb40b0d00 2024-09-19T07:41:12.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.104+0000 7febabfff640 1 -- 172.21.15.83:0/1160535527 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb00ab9d0 con 0x7febb40b0d00 2024-09-19T07:41:12.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.104+0000 7febbb7b6640 1 -- 172.21.15.83:0/1160535527 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febb40b0d00 msgr2=0x7febb4159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:12.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.104+0000 7febbb7b6640 1 --2- 172.21.15.83:0/1160535527 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febb40b0d00 0x7febb4159bf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7febb0009870 tx=0x7febb001c4d0 comp rx=0 tx=0).stop 2024-09-19T07:41:12.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.104+0000 7febbb7b6640 1 -- 172.21.15.83:0/1160535527 shutdown_connections 2024-09-19T07:41:12.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.104+0000 7febbb7b6640 1 --2- 172.21.15.83:0/1160535527 >> v2:172.21.15.83:3300/0 conn(0x7febb40c9660 0x7febb41563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:12.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.104+0000 7febbb7b6640 1 --2- 172.21.15.83:0/1160535527 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febb40b0d00 0x7febb4159bf0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:12.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.104+0000 7febbb7b6640 1 -- 172.21.15.83:0/1160535527 >> 172.21.15.83:0/1160535527 conn(0x7febb40722a0 msgr2=0x7febb40726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:12.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.104+0000 7febbb7b6640 1 -- 172.21.15.83:0/1160535527 shutdown_connections 2024-09-19T07:41:12.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.104+0000 7febbb7b6640 1 -- 172.21.15.83:0/1160535527 wait complete. 2024-09-19T07:41:12.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febbb7b6640 1 Processor -- start 2024-09-19T07:41:12.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febbb7b6640 1 -- start start 2024-09-19T07:41:12.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febbb7b6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7febb40b0d00 0x7febb41523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:12.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febbb7b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febb40c9660 0x7febb41fb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:12.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febbb7b6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febb40c5830 0x7febb41fbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:12.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febb9552640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7febb40b0d00 0x7febb41523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:12.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febb9552640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7febb40b0d00 0x7febb41523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:52024/0 (socket says 172.21.15.83:52024) 2024-09-19T07:41:12.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febb8d51640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febb40c9660 0x7febb41fb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:12.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febb8d51640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febb40c9660 0x7febb41fb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56854/0 (socket says 172.21.15.83:56854) 2024-09-19T07:41:12.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febb9552640 1 -- 172.21.15.83:0/566905086 learned_addr learned my addr 172.21.15.83:0/566905086 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:12.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febbb7b6640 1 -- 172.21.15.83:0/566905086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7febb40c8920 con 0x7febb40c9660 2024-09-19T07:41:12.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febbb7b6640 1 -- 172.21.15.83:0/566905086 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7febb40b10e0 con 0x7febb40b0d00 2024-09-19T07:41:12.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febbb7b6640 1 -- 172.21.15.83:0/566905086 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7febb40c8aa0 con 0x7febb40c5830 2024-09-19T07:41:12.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febb9d53640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febb40c5830 0x7febb41fbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:12.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febb9552640 1 -- 172.21.15.83:0/566905086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febb40c5830 msgr2=0x7febb41fbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:12.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febb9552640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febb40c5830 0x7febb41fbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:12.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febb9552640 1 -- 172.21.15.83:0/566905086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febb40c9660 msgr2=0x7febb41fb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:12.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febb9552640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febb40c9660 0x7febb41fb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:12.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febb9552640 1 -- 172.21.15.83:0/566905086 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb414f7f0 con 0x7febb40b0d00 2024-09-19T07:41:12.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febb9552640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7febb40b0d00 0x7febb41523e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7febb00ae040 tx=0x7febb0019270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:12.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7feba9ffb640 1 -- 172.21.15.83:0/566905086 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb00b0040 con 0x7febb40b0d00 2024-09-19T07:41:12.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7feba9ffb640 1 -- 172.21.15.83:0/566905086 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb00b9070 con 0x7febb40b0d00 2024-09-19T07:41:12.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febbb7b6640 1 -- 172.21.15.83:0/566905086 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febb414fab0 con 0x7febb40b0d00 2024-09-19T07:41:12.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7febbb7b6640 1 -- 172.21.15.83:0/566905086 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febb4152920 con 0x7febb40b0d00 2024-09-19T07:41:12.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.108+0000 7feba9ffb640 1 -- 172.21.15.83:0/566905086 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb00b4660 con 0x7febb40b0d00 2024-09-19T07:41:12.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.112+0000 7feba9ffb640 1 -- 172.21.15.83:0/566905086 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7febb00b4800 con 0x7febb40b0d00 2024-09-19T07:41:12.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.112+0000 7feba9ffb640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb980678c0 0x7feb98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:12.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.112+0000 7feba9ffb640 1 -- 172.21.15.83:0/566905086 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9217+0+0 (secure 0 0 0) 0x7febb0120630 con 0x7febb40b0d00 2024-09-19T07:41:12.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.112+0000 7febb8d51640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb980678c0 0x7feb98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:12.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.112+0000 7febb8d51640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb980678c0 0x7feb98069d80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7febb4050170 tx=0x7feba4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:12.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.116+0000 7febbb7b6640 1 -- 172.21.15.83:0/566905086 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb940052a0 con 0x7febb40b0d00 2024-09-19T07:41:12.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.124+0000 7feba9ffb640 1 -- 172.21.15.83:0/566905086 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7febb010ce20 con 0x7febb40b0d00 2024-09-19T07:41:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.640+0000 7febbb7b6640 1 -- 172.21.15.83:0/566905086 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb94005590 con 0x7febb40b0d00 2024-09-19T07:41:12.642 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:12.642 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":154,"btime":"2024-09-19T07:41:11:437315+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:12.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.644+0000 7feba9ffb640 1 -- 172.21.15.83:0/566905086 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5720 (secure 0 0 0) 0x7febb011f020 con 0x7febb40b0d00 2024-09-19T07:41:12.642 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 154 2024-09-19T07:41:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.648+0000 7feb9f7fe640 1 -- 172.21.15.83:0/566905086 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb980678c0 msgr2=0x7feb98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.648+0000 7feb9f7fe640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb980678c0 0x7feb98069d80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7febb4050170 tx=0x7feba4008040 comp rx=0 tx=0).stop 2024-09-19T07:41:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.648+0000 7feb9f7fe640 1 -- 172.21.15.83:0/566905086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7febb40b0d00 msgr2=0x7febb41523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.648+0000 7feb9f7fe640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7febb40b0d00 0x7febb41523e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7febb00ae040 tx=0x7febb0019270 comp rx=0 tx=0).stop 2024-09-19T07:41:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.648+0000 7feb9f7fe640 1 -- 172.21.15.83:0/566905086 shutdown_connections 2024-09-19T07:41:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.648+0000 7feb9f7fe640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feb980678c0 0x7feb98069d80 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.648+0000 7feb9f7fe640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febb40c5830 0x7febb41fbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.648+0000 7feb9f7fe640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febb40c9660 0x7febb41fb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.648+0000 7feb9f7fe640 1 --2- 172.21.15.83:0/566905086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7febb40b0d00 0x7febb41523e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:12.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.648+0000 7feb9f7fe640 1 -- 172.21.15.83:0/566905086 >> 172.21.15.83:0/566905086 conn(0x7febb40722a0 msgr2=0x7febb415f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:12.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.648+0000 7feb9f7fe640 1 -- 172.21.15.83:0/566905086 shutdown_connections 2024-09-19T07:41:12.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:12.648+0000 7feb9f7fe640 1 -- 172.21.15.83:0/566905086 wait complete. 2024-09-19T07:41:12.677 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=7350 2024-09-19T07:41:12.677 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.a 2024-09-19T07:41:14.679 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-19T07:41:14.679 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:41:14.679 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:41:14.681 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:41:14.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:14.810 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:41:14.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.860+0000 7f7f74d92640 1 Processor -- start 2024-09-19T07:41:14.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.860+0000 7f7f74d92640 1 -- start start 2024-09-19T07:41:14.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.860+0000 7f7f74d92640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f700a7740 0x7f7f7015e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.860+0000 7f7f74d92640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f700ace10 con 0x7f7f7006cef0 2024-09-19T07:41:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.860+0000 7f7f74d92640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f700ad0f0 con 0x7f7f7015afb0 2024-09-19T07:41:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.860+0000 7f7f74d92640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7f700ada20 con 0x7f7f700a7740 2024-09-19T07:41:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.860+0000 7f7f6e59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7f7006cef0 0x7f7f7015a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45428/0 (socket says 172.21.15.83:45428) 2024-09-19T07:41:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.860+0000 7f7f6e59c640 1 -- 172.21.15.83:0/1932562977 learned_addr learned my addr 172.21.15.83:0/1932562977 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:41:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.860+0000 7f7f6dd9b640 1 --2- 172.21.15.83:0/1932562977 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f700a7740 0x7f7f7015e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.860+0000 7f7f6dd9b640 1 -- 172.21.15.83:0/1932562977 >> v1:172.21.15.184:6789/0 conn(0x7f7f7015afb0 legacy=0x7f7f700c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.860+0000 7f7f6dd9b640 1 -- 172.21.15.83:0/1932562977 >> v1:172.21.15.83:6789/0 conn(0x7f7f7006cef0 legacy=0x7f7f7015a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:14.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.860+0000 7f7f6dd9b640 1 -- 172.21.15.83:0/1932562977 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f700ad3d0 con 0x7f7f700a7740 2024-09-19T07:41:14.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.864+0000 7f7f6dd9b640 1 --2- 172.21.15.83:0/1932562977 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f700a7740 0x7f7f7015e150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7f600097f0 tx=0x7f7f6001c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27b12cf3233a108 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:14.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.864+0000 7f7f6d59a640 1 -- 172.21.15.83:0/1932562977 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f600b0070 con 0x7f7f700a7740 2024-09-19T07:41:14.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.864+0000 7f7f6d59a640 1 -- 172.21.15.83:0/1932562977 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f600ab600 con 0x7f7f700a7740 2024-09-19T07:41:14.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.864+0000 7f7f6d59a640 1 -- 172.21.15.83:0/1932562977 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f600ab9d0 con 0x7f7f700a7740 2024-09-19T07:41:14.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.864+0000 7f7f74d92640 1 -- 172.21.15.83:0/1932562977 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f700a7740 msgr2=0x7f7f7015e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:14.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.864+0000 7f7f74d92640 1 --2- 172.21.15.83:0/1932562977 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f700a7740 0x7f7f7015e150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7f600097f0 tx=0x7f7f6001c300 comp rx=0 tx=0).stop 2024-09-19T07:41:14.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.864+0000 7f7f74d92640 1 -- 172.21.15.83:0/1932562977 shutdown_connections 2024-09-19T07:41:14.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.864+0000 7f7f74d92640 1 --2- 172.21.15.83:0/1932562977 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f700a7740 0x7f7f7015e150 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:14.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.864+0000 7f7f74d92640 1 -- 172.21.15.83:0/1932562977 >> 172.21.15.83:0/1932562977 conn(0x7f7f700c5060 msgr2=0x7f7f700c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:14.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.868+0000 7f7f74d92640 1 -- 172.21.15.83:0/1932562977 shutdown_connections 2024-09-19T07:41:14.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.868+0000 7f7f74d92640 1 -- 172.21.15.83:0/1932562977 wait complete. 2024-09-19T07:41:14.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.868+0000 7f7f74d92640 1 Processor -- start 2024-09-19T07:41:14.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.868+0000 7f7f74d92640 1 -- start start 2024-09-19T07:41:14.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f74d92640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f7006cef0 0x7f7f7015d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:14.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f74d92640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f700a7740 0x7f7f700751e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:14.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f74d92640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7015afb0 0x7f7f700a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:14.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f74d92640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7006d450 con 0x7f7f7015afb0 2024-09-19T07:41:14.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f74d92640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7006d2d0 con 0x7f7f700a7740 2024-09-19T07:41:14.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f74d92640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7f70168700 con 0x7f7f7006cef0 2024-09-19T07:41:14.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f6ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7015afb0 0x7f7f700a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:14.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f6ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7015afb0 0x7f7f700a55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56874/0 (socket says 172.21.15.83:56874) 2024-09-19T07:41:14.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f6e59c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f7006cef0 0x7f7f7015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:14.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f6ed9d640 1 -- 172.21.15.83:0/3455387806 learned_addr learned my addr 172.21.15.83:0/3455387806 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:14.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f6e59c640 1 -- 172.21.15.83:0/3455387806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f700a7740 msgr2=0x7f7f700751e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:14.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f6e59c640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f700a7740 0x7f7f700751e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:14.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f6e59c640 1 -- 172.21.15.83:0/3455387806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7015afb0 msgr2=0x7f7f700a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:14.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f6e59c640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7015afb0 0x7f7f700a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:14.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f6e59c640 1 -- 172.21.15.83:0/3455387806 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f70201d80 con 0x7f7f7006cef0 2024-09-19T07:41:14.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f6ed9d640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7015afb0 0x7f7f700a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:41:14.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f6e59c640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f7006cef0 0x7f7f7015d340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400c780 tx=0x7f7f6400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:14.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f5effd640 1 -- 172.21.15.83:0/3455387806 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f64018070 con 0x7f7f7006cef0 2024-09-19T07:41:14.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f5effd640 1 -- 172.21.15.83:0/3455387806 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f640072d0 con 0x7f7f7006cef0 2024-09-19T07:41:14.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f5effd640 1 -- 172.21.15.83:0/3455387806 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f640047f0 con 0x7f7f7006cef0 2024-09-19T07:41:14.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f74d92640 1 -- 172.21.15.83:0/3455387806 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f7015d880 con 0x7f7f7006cef0 2024-09-19T07:41:14.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.872+0000 7f7f74d92640 1 -- 172.21.15.83:0/3455387806 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f70207b70 con 0x7f7f7006cef0 2024-09-19T07:41:14.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.876+0000 7f7f5effd640 1 -- 172.21.15.83:0/3455387806 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7f64004990 con 0x7f7f7006cef0 2024-09-19T07:41:14.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.876+0000 7f7f5effd640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f500678c0 0x7f7f50069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:14.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.876+0000 7f7f5effd640 1 -- 172.21.15.83:0/3455387806 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 9217+0+0 (secure 0 0 0) 0x7f7f64085d60 con 0x7f7f7006cef0 2024-09-19T07:41:14.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.876+0000 7f7f74d92640 1 -- 172.21.15.83:0/3455387806 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f7015e150 con 0x7f7f7006cef0 2024-09-19T07:41:14.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.876+0000 7f7f6dd9b640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f500678c0 0x7f7f50069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:14.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.876+0000 7f7f6dd9b640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f500678c0 0x7f7f50069d80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7f600ad000 tx=0x7f7f60004740 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:14.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:14.884+0000 7f7f5effd640 1 -- 172.21.15.83:0/3455387806 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7f640554d0 con 0x7f7f7006cef0 2024-09-19T07:41:15.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.228+0000 7f7f74d92640 1 -- 172.21.15.83:0/3455387806 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f7006ff30 con 0x7f7f7006cef0 2024-09-19T07:41:15.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.228+0000 7f7f5effd640 1 -- 172.21.15.83:0/3455387806 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5720 (secure 0 0 0) 0x7f7f640597d0 con 0x7f7f7006cef0 2024-09-19T07:41:15.230 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:15.230 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":154,"btime":"2024-09-19T07:41:11:437315+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:15.230 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 154 2024-09-19T07:41:15.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.232+0000 7f7f5cff9640 1 -- 172.21.15.83:0/3455387806 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f500678c0 msgr2=0x7f7f50069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:15.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.232+0000 7f7f5cff9640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f500678c0 0x7f7f50069d80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7f600ad000 tx=0x7f7f60004740 comp rx=0 tx=0).stop 2024-09-19T07:41:15.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.232+0000 7f7f5cff9640 1 -- 172.21.15.83:0/3455387806 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f7006cef0 msgr2=0x7f7f7015d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:15.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.232+0000 7f7f5cff9640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f7006cef0 0x7f7f7015d340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400c780 tx=0x7f7f6400cc40 comp rx=0 tx=0).stop 2024-09-19T07:41:15.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.232+0000 7f7f5cff9640 1 -- 172.21.15.83:0/3455387806 shutdown_connections 2024-09-19T07:41:15.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.232+0000 7f7f5cff9640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7f500678c0 0x7f7f50069d80 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:15.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.232+0000 7f7f5cff9640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f7015afb0 0x7f7f700a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:15.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.232+0000 7f7f5cff9640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f700a7740 0x7f7f700751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:15.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.232+0000 7f7f5cff9640 1 --2- 172.21.15.83:0/3455387806 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f7006cef0 0x7f7f7015d340 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:15.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.232+0000 7f7f5cff9640 1 -- 172.21.15.83:0/3455387806 >> 172.21.15.83:0/3455387806 conn(0x7f7f700c5060 msgr2=0x7f7f700c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:15.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.232+0000 7f7f5cff9640 1 -- 172.21.15.83:0/3455387806 shutdown_connections 2024-09-19T07:41:15.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:15.232+0000 7f7f5cff9640 1 -- 172.21.15.83:0/3455387806 wait complete. 2024-09-19T07:41:15.260 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-19T07:41:16.406 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:41:16.408+0000 7f70d67c4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:41:17.261 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:17.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.448+0000 7fbbb8243640 1 Processor -- start 2024-09-19T07:41:17.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.448+0000 7fbbb8243640 1 -- start start 2024-09-19T07:41:17.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb8243640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbbb006c700 0x7fbbb01529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:17.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb8243640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbb00c2a10 0x7fbbb0155470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:17.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb8243640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbbb01559b0 0x7fbbb006f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:17.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb8243640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fbbb0152fc0 con 0x7fbbb006c700 2024-09-19T07:41:17.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb8243640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbbb01532a0 con 0x7fbbb01559b0 2024-09-19T07:41:17.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb8243640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb0153580 con 0x7fbbb00c2a10 2024-09-19T07:41:17.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb5fdf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbbb006c700 0x7fbbb01529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:17.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb57de640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbb00c2a10 0x7fbbb0155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:17.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb57de640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbb00c2a10 0x7fbbb0155470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:42846/0 (socket says 172.21.15.83:42846) 2024-09-19T07:41:17.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb5fdf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbbb006c700 0x7fbbb01529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56886/0 (socket says 172.21.15.83:56886) 2024-09-19T07:41:17.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb57de640 1 -- 172.21.15.83:0/1200494851 learned_addr learned my addr 172.21.15.83:0/1200494851 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:17.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb67e0640 1 --2- 172.21.15.83:0/1200494851 >> v2:172.21.15.184:3300/0 conn(0x7fbbb01559b0 0x7fbbb006f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:17.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb5fdf640 1 -- 172.21.15.83:0/1200494851 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbb00c2a10 msgr2=0x7fbbb0155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:17.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb5fdf640 1 --2- 172.21.15.83:0/1200494851 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbb00c2a10 0x7fbbb0155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:17.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb5fdf640 1 -- 172.21.15.83:0/1200494851 >> v2:172.21.15.184:3300/0 conn(0x7fbbb01559b0 msgr2=0x7fbbb006f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:17.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb5fdf640 1 --2- 172.21.15.83:0/1200494851 >> v2:172.21.15.184:3300/0 conn(0x7fbbb01559b0 0x7fbbb006f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:17.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb5fdf640 1 -- 172.21.15.83:0/1200494851 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb0153860 con 0x7fbbb006c700 2024-09-19T07:41:17.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb67e0640 1 --2- 172.21.15.83:0/1200494851 >> v2:172.21.15.184:3300/0 conn(0x7fbbb01559b0 0x7fbbb006f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:41:17.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.452+0000 7fbbb57de640 1 --2- 172.21.15.83:0/1200494851 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbb00c2a10 0x7fbbb0155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:41:17.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.456+0000 7fbbb5fdf640 1 --2- 172.21.15.83:0/1200494851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbb006c700 0x7fbbb01529e0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0070020 tx=0x7fbbac01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fcb4ff33ab12657 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.456+0000 7fbbb4fdd640 1 -- 172.21.15.83:0/1200494851 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac0b0070 con 0x7fbbb006c700 2024-09-19T07:41:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.456+0000 7fbbb4fdd640 1 -- 172.21.15.83:0/1200494851 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbac0ab600 con 0x7fbbb006c700 2024-09-19T07:41:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.456+0000 7fbbb4fdd640 1 -- 172.21.15.83:0/1200494851 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac0ab9d0 con 0x7fbbb006c700 2024-09-19T07:41:17.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.456+0000 7fbbb8243640 1 -- 172.21.15.83:0/1200494851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbb006c700 msgr2=0x7fbbb01529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:17.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.456+0000 7fbbb8243640 1 --2- 172.21.15.83:0/1200494851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbb006c700 0x7fbbb01529e0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0070020 tx=0x7fbbac01c3f0 comp rx=0 tx=0).stop 2024-09-19T07:41:17.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 -- 172.21.15.83:0/1200494851 shutdown_connections 2024-09-19T07:41:17.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 --2- 172.21.15.83:0/1200494851 >> v2:172.21.15.184:3300/0 conn(0x7fbbb01559b0 0x7fbbb006f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:17.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 --2- 172.21.15.83:0/1200494851 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbb00c2a10 0x7fbbb0155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:17.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 --2- 172.21.15.83:0/1200494851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbb006c700 0x7fbbb01529e0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:17.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 -- 172.21.15.83:0/1200494851 >> 172.21.15.83:0/1200494851 conn(0x7fbbb00c8840 msgr2=0x7fbbb00c8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:17.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 -- 172.21.15.83:0/1200494851 shutdown_connections 2024-09-19T07:41:17.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 -- 172.21.15.83:0/1200494851 wait complete. 2024-09-19T07:41:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 Processor -- start 2024-09-19T07:41:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 -- start start 2024-09-19T07:41:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbb006c700 0x7fbbb00c11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbb00c2a10 0x7fbbb014c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbb01559b0 0x7fbbb014cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb006cae0 con 0x7fbbb006c700 2024-09-19T07:41:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb006fd30 con 0x7fbbb01559b0 2024-09-19T07:41:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb8243640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb006cc60 con 0x7fbbb00c2a10 2024-09-19T07:41:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb5fdf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbb006c700 0x7fbbb00c11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:17.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb5fdf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbb006c700 0x7fbbb00c11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56900/0 (socket says 172.21.15.83:56900) 2024-09-19T07:41:17.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb5fdf640 1 -- 172.21.15.83:0/4224382400 learned_addr learned my addr 172.21.15.83:0/4224382400 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:17.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb57de640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbb00c2a10 0x7fbbb014c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:17.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb67e0640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbb01559b0 0x7fbbb014cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:17.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb5fdf640 1 -- 172.21.15.83:0/4224382400 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbb00c2a10 msgr2=0x7fbbb014c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:17.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb5fdf640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbb00c2a10 0x7fbbb014c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:17.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb5fdf640 1 -- 172.21.15.83:0/4224382400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbb01559b0 msgr2=0x7fbbb014cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:17.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb5fdf640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbb01559b0 0x7fbbb014cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:17.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb5fdf640 1 -- 172.21.15.83:0/4224382400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb01f61e0 con 0x7fbbb006c700 2024-09-19T07:41:17.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.460+0000 7fbbb5fdf640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbb006c700 0x7fbbb00c11e0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0ae040 tx=0x7fbbac019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.464+0000 7fbba67fc640 1 -- 172.21.15.83:0/4224382400 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac0b8020 con 0x7fbbb006c700 2024-09-19T07:41:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.464+0000 7fbbb8243640 1 -- 172.21.15.83:0/4224382400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb014d8d0 con 0x7fbbb006c700 2024-09-19T07:41:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.464+0000 7fbbb8243640 1 -- 172.21.15.83:0/4224382400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb014f7a0 con 0x7fbbb006c700 2024-09-19T07:41:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.464+0000 7fbba67fc640 1 -- 172.21.15.83:0/4224382400 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbac0b4410 con 0x7fbbb006c700 2024-09-19T07:41:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.464+0000 7fbba67fc640 1 -- 172.21.15.83:0/4224382400 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac0b0040 con 0x7fbbb006c700 2024-09-19T07:41:17.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.468+0000 7fbba67fc640 1 -- 172.21.15.83:0/4224382400 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbbac0bb020 con 0x7fbbb006c700 2024-09-19T07:41:17.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.468+0000 7fbba67fc640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbb940678c0 0x7fbb94069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:17.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.468+0000 7fbba67fc640 1 -- 172.21.15.83:0/4224382400 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9217+0+0 (secure 0 0 0) 0x7fbbac1203d0 con 0x7fbbb006c700 2024-09-19T07:41:17.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.468+0000 7fbbb57de640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbb940678c0 0x7fbb94069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:17.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.468+0000 7fbbb8243640 1 -- 172.21.15.83:0/4224382400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb980052a0 con 0x7fbbb006c700 2024-09-19T07:41:17.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.468+0000 7fbbb57de640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbb940678c0 0x7fbb94069d80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbba000dad0 tx=0x7fbba000f040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:17.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.480+0000 7fbba67fc640 1 -- 172.21.15.83:0/4224382400 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbbac0c2030 con 0x7fbbb006c700 2024-09-19T07:41:17.713 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:17.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.968+0000 7fbbb8243640 1 -- 172.21.15.83:0/4224382400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb98005860 con 0x7fbbb006c700 2024-09-19T07:41:17.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.968+0000 7fbba67fc640 1 -- 172.21.15.83:0/4224382400 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) ==== 95+0+6546 (secure 0 0 0) 0x7fbbac016070 con 0x7fbbb006c700 2024-09-19T07:41:17.969 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:17.970 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":155,"btime":"2024-09-19T07:41:15:459617+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:17.970 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 155 2024-09-19T07:41:17.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.972+0000 7fbbb8243640 1 -- 172.21.15.83:0/4224382400 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbb940678c0 msgr2=0x7fbb94069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:17.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.972+0000 7fbbb8243640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbb940678c0 0x7fbb94069d80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbba000dad0 tx=0x7fbba000f040 comp rx=0 tx=0).stop 2024-09-19T07:41:17.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.972+0000 7fbbb8243640 1 -- 172.21.15.83:0/4224382400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbb006c700 msgr2=0x7fbbb00c11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:17.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.972+0000 7fbbb8243640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbb006c700 0x7fbbb00c11e0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0ae040 tx=0x7fbbac019190 comp rx=0 tx=0).stop 2024-09-19T07:41:17.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.972+0000 7fbbb8243640 1 -- 172.21.15.83:0/4224382400 shutdown_connections 2024-09-19T07:41:17.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.972+0000 7fbbb8243640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbb940678c0 0x7fbb94069d80 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:17.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.972+0000 7fbbb8243640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbb01559b0 0x7fbbb014cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:17.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.972+0000 7fbbb8243640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbb00c2a10 0x7fbbb014c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:17.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.972+0000 7fbbb8243640 1 --2- 172.21.15.83:0/4224382400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbb006c700 0x7fbbb00c11e0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:17.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.972+0000 7fbbb8243640 1 -- 172.21.15.83:0/4224382400 >> 172.21.15.83:0/4224382400 conn(0x7fbbb00c8840 msgr2=0x7fbbb00ad520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:17.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.972+0000 7fbbb8243640 1 -- 172.21.15.83:0/4224382400 shutdown_connections 2024-09-19T07:41:17.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:17.972+0000 7fbbb8243640 1 -- 172.21.15.83:0/4224382400 wait complete. 2024-09-19T07:41:18.015 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-19T07:41:18.015 INFO:tasks.mds_thrash.fs.[b]:waiting for 66 secs before thrashing 2024-09-19T07:41:18.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.032+0000 7f1f1d922640 1 Processor -- start 2024-09-19T07:41:18.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.032+0000 7f1f1d922640 1 -- start start 2024-09-19T07:41:18.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f1d922640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1f18150310 0x7f1f180728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f1d922640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f180c4990 0x7f1f18156070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f1d922640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f180c2f90 con 0x7f1f1806cef0 2024-09-19T07:41:18.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f1d922640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f1f180c3e20 con 0x7f1f18150310 2024-09-19T07:41:18.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f1d922640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f180c3210 con 0x7f1f180c4990 2024-09-19T07:41:18.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f167fc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1f18150310 0x7f1f180728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f16ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1f1806cef0 0x7f1f1814fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45442/0 (socket says 172.21.15.83:45442) 2024-09-19T07:41:18.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f167fc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1f18150310 0x7f1f180728d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:52100/0 (socket says 172.21.15.83:52100) 2024-09-19T07:41:18.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f167fc640 1 -- 172.21.15.83:0/1114527968 learned_addr learned my addr 172.21.15.83:0/1114527968 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:18.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f177fe640 1 --2- 172.21.15.83:0/1114527968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f180c4990 0x7f1f18156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f177fe640 1 -- 172.21.15.83:0/1114527968 >> v2:172.21.15.184:3300/0 conn(0x7f1f18150310 msgr2=0x7f1f180728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:18.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f177fe640 1 --2- 172.21.15.83:0/1114527968 >> v2:172.21.15.184:3300/0 conn(0x7f1f18150310 0x7f1f180728d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f177fe640 1 -- 172.21.15.83:0/1114527968 >> v1:172.21.15.83:6789/0 conn(0x7f1f1806cef0 legacy=0x7f1f1814fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:18.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.036+0000 7f1f177fe640 1 -- 172.21.15.83:0/1114527968 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f180c34f0 con 0x7f1f180c4990 2024-09-19T07:41:18.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.040+0000 7f1f177fe640 1 --2- 172.21.15.83:0/1114527968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f180c4990 0x7f1f18156070 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009d90 tx=0x7f1f1001c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f3f66177991eaff server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:18.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.040+0000 7f1f15ffb640 1 -- 172.21.15.83:0/1114527968 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f100ac070 con 0x7f1f180c4990 2024-09-19T07:41:18.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.040+0000 7f1f15ffb640 1 -- 172.21.15.83:0/1114527968 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f100a6690 con 0x7f1f180c4990 2024-09-19T07:41:18.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.040+0000 7f1f15ffb640 1 -- 172.21.15.83:0/1114527968 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f100a6aa0 con 0x7f1f180c4990 2024-09-19T07:41:18.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.044+0000 7f1f1d922640 1 -- 172.21.15.83:0/1114527968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f180c4990 msgr2=0x7f1f18156070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:18.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.044+0000 7f1f1d922640 1 --2- 172.21.15.83:0/1114527968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f180c4990 0x7f1f18156070 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009d90 tx=0x7f1f1001c790 comp rx=0 tx=0).stop 2024-09-19T07:41:18.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.044+0000 7f1f1d922640 1 -- 172.21.15.83:0/1114527968 shutdown_connections 2024-09-19T07:41:18.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.044+0000 7f1f1d922640 1 --2- 172.21.15.83:0/1114527968 >> v2:172.21.15.184:3300/0 conn(0x7f1f18150310 0x7f1f180728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.044+0000 7f1f1d922640 1 --2- 172.21.15.83:0/1114527968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f180c4990 0x7f1f18156070 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.044+0000 7f1f1d922640 1 -- 172.21.15.83:0/1114527968 >> 172.21.15.83:0/1114527968 conn(0x7f1f180c5090 msgr2=0x7f1f180c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:18.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.044+0000 7f1f1d922640 1 -- 172.21.15.83:0/1114527968 shutdown_connections 2024-09-19T07:41:18.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.044+0000 7f1f1d922640 1 -- 172.21.15.83:0/1114527968 wait complete. 2024-09-19T07:41:18.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f1d922640 1 Processor -- start 2024-09-19T07:41:18.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f1d922640 1 -- start start 2024-09-19T07:41:18.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f1d922640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f1806cef0 0x7f1f1815eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f1d922640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f180c4990 0x7f1f18143230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f1d922640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18150310 0x7f1f1814e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f1d922640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1815d1c0 con 0x7f1f18150310 2024-09-19T07:41:18.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f1d922640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1806d4e0 con 0x7f1f180c4990 2024-09-19T07:41:18.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f1d922640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f1815d340 con 0x7f1f1806cef0 2024-09-19T07:41:18.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f177fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18150310 0x7f1f1814e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f177fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18150310 0x7f1f1814e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56910/0 (socket says 172.21.15.83:56910) 2024-09-19T07:41:18.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f177fe640 1 -- 172.21.15.83:0/3913539303 learned_addr learned my addr 172.21.15.83:0/3913539303 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:18.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f16ffd640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f1806cef0 0x7f1f1815eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f167fc640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f180c4990 0x7f1f18143230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f177fe640 1 -- 172.21.15.83:0/3913539303 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f1806cef0 msgr2=0x7f1f1815eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:18.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f177fe640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f1806cef0 0x7f1f1815eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f177fe640 1 -- 172.21.15.83:0/3913539303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f180c4990 msgr2=0x7f1f18143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:18.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f177fe640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f180c4990 0x7f1f18143230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f177fe640 1 -- 172.21.15.83:0/3913539303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f18072f40 con 0x7f1f18150310 2024-09-19T07:41:18.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f177fe640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18150310 0x7f1f1814e6e0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f1f1001d030 tx=0x7f1f10019530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:18.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1efffff640 1 -- 172.21.15.83:0/3913539303 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f100c0070 con 0x7f1f18150310 2024-09-19T07:41:18.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1efffff640 1 -- 172.21.15.83:0/3913539303 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f100a6490 con 0x7f1f18150310 2024-09-19T07:41:18.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1efffff640 1 -- 172.21.15.83:0/3913539303 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f100ac050 con 0x7f1f18150310 2024-09-19T07:41:18.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f1d922640 1 -- 172.21.15.83:0/3913539303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f181ee640 con 0x7f1f18150310 2024-09-19T07:41:18.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.048+0000 7f1f1d922640 1 -- 172.21.15.83:0/3913539303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f1814cf80 con 0x7f1f18150310 2024-09-19T07:41:18.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.052+0000 7f1f1d922640 1 -- 172.21.15.83:0/3913539303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f181f1c70 con 0x7f1f18150310 2024-09-19T07:41:18.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.052+0000 7f1efffff640 1 -- 172.21.15.83:0/3913539303 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1f100c3070 con 0x7f1f18150310 2024-09-19T07:41:18.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.052+0000 7f1efffff640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f040678c0 0x7f1f04069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.052+0000 7f1efffff640 1 -- 172.21.15.83:0/3913539303 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9217+0+0 (secure 0 0 0) 0x7f1f101211b0 con 0x7f1f18150310 2024-09-19T07:41:18.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.052+0000 7f1f16ffd640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f040678c0 0x7f1f04069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.060+0000 7f1f16ffd640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f040678c0 0x7f1f04069d80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1f08000c10 tx=0x7f1f08007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:18.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.068+0000 7f1efffff640 1 -- 172.21.15.83:0/3913539303 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1f181f1c70 con 0x7f1f18150310 2024-09-19T07:41:18.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.452+0000 7f1f1d922640 1 -- 172.21.15.83:0/3913539303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f180c2a50 con 0x7f1f18150310 2024-09-19T07:41:18.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.452+0000 7f1efffff640 1 -- 172.21.15.83:0/3913539303 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) ==== 95+0+6546 (secure 0 0 0) 0x7f1f100164a0 con 0x7f1f18150310 2024-09-19T07:41:18.455 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:18.455 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":155,"btime":"2024-09-19T07:41:15:459617+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:18.456 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 155 2024-09-19T07:41:18.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.456+0000 7f1efdffb640 1 -- 172.21.15.83:0/3913539303 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f040678c0 msgr2=0x7f1f04069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:18.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.456+0000 7f1efdffb640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f040678c0 0x7f1f04069d80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1f08000c10 tx=0x7f1f08007e30 comp rx=0 tx=0).stop 2024-09-19T07:41:18.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.456+0000 7f1efdffb640 1 -- 172.21.15.83:0/3913539303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18150310 msgr2=0x7f1f1814e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:18.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.456+0000 7f1efdffb640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18150310 0x7f1f1814e6e0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f1f1001d030 tx=0x7f1f10019530 comp rx=0 tx=0).stop 2024-09-19T07:41:18.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.456+0000 7f1efdffb640 1 -- 172.21.15.83:0/3913539303 shutdown_connections 2024-09-19T07:41:18.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.460+0000 7f1efdffb640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f040678c0 0x7f1f04069d80 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.460+0000 7f1efdffb640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f18150310 0x7f1f1814e6e0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.460+0000 7f1efdffb640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f180c4990 0x7f1f18143230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.460+0000 7f1efdffb640 1 --2- 172.21.15.83:0/3913539303 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f1806cef0 0x7f1f1815eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.460+0000 7f1efdffb640 1 -- 172.21.15.83:0/3913539303 >> 172.21.15.83:0/3913539303 conn(0x7f1f180c5090 msgr2=0x7f1f18074f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:18.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.460+0000 7f1efdffb640 1 -- 172.21.15.83:0/3913539303 shutdown_connections 2024-09-19T07:41:18.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.460+0000 7f1efdffb640 1 -- 172.21.15.83:0/3913539303 wait complete. 2024-09-19T07:41:18.487 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-19T07:41:18.487 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-19T07:41:18.487 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:41:18.503 INFO:tasks.ceph.mds.d:Stopped 2024-09-19T07:41:18.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:18.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f458131c640 1 Processor -- start 2024-09-19T07:41:18.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f458131c640 1 -- start start 2024-09-19T07:41:18.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f458131c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 0x7f4574047d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f458131c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f457404bef0 0x7f45740d4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f458131c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f45740d5520 0x7f45740d7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f457bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 0x7f4574047d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f457bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 0x7f4574047d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:42876/0 (socket says 172.21.15.83:42876) 2024-09-19T07:41:18.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f457b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f457404bef0 0x7f45740d4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f457bfff640 1 -- 172.21.15.83:0/1023458096 learned_addr learned my addr 172.21.15.83:0/1023458096 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:18.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f458131c640 1 -- 172.21.15.83:0/1023458096 --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f457404bcf0 con 0x7f457404bef0 2024-09-19T07:41:18.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f458131c640 1 -- 172.21.15.83:0/1023458096 --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4574047140 con 0x7f45740d5520 2024-09-19T07:41:18.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f458131c640 1 -- 172.21.15.83:0/1023458096 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4574046e60 con 0x7f45740317b0 2024-09-19T07:41:18.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f4580b1b640 1 --2- 172.21.15.83:0/1023458096 >> v2:172.21.15.184:3300/0 conn(0x7f45740d5520 0x7f45740d7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f457bfff640 1 -- 172.21.15.83:0/1023458096 >> v2:172.21.15.184:3300/0 conn(0x7f45740d5520 msgr2=0x7f45740d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:18.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f457bfff640 1 --2- 172.21.15.83:0/1023458096 >> v2:172.21.15.184:3300/0 conn(0x7f45740d5520 0x7f45740d7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f457bfff640 1 -- 172.21.15.83:0/1023458096 >> v2:172.21.15.83:3300/0 conn(0x7f457404bef0 msgr2=0x7f45740d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:18.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f457bfff640 1 --2- 172.21.15.83:0/1023458096 >> v2:172.21.15.83:3300/0 conn(0x7f457404bef0 0x7f45740d4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.732+0000 7f457bfff640 1 -- 172.21.15.83:0/1023458096 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45740499c0 con 0x7f45740317b0 2024-09-19T07:41:18.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.736+0000 7f457bfff640 1 --2- 172.21.15.83:0/1023458096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 0x7f4574047d90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4574043410 tx=0x7f457001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdf3fb61622cf26e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:18.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.736+0000 7f457b7fe640 1 --2- 172.21.15.83:0/1023458096 >> v2:172.21.15.83:3300/0 conn(0x7f457404bef0 0x7f45740d4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:41:18.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.736+0000 7f457affd640 1 -- 172.21.15.83:0/1023458096 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45700b0070 con 0x7f45740317b0 2024-09-19T07:41:18.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.736+0000 7f457affd640 1 -- 172.21.15.83:0/1023458096 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45700ab600 con 0x7f45740317b0 2024-09-19T07:41:18.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.736+0000 7f457affd640 1 -- 172.21.15.83:0/1023458096 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45700ab9b0 con 0x7f45740317b0 2024-09-19T07:41:18.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 -- 172.21.15.83:0/1023458096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 msgr2=0x7f4574047d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:18.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 --2- 172.21.15.83:0/1023458096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 0x7f4574047d90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4574043410 tx=0x7f457001c420 comp rx=0 tx=0).stop 2024-09-19T07:41:18.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 -- 172.21.15.83:0/1023458096 shutdown_connections 2024-09-19T07:41:18.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 --2- 172.21.15.83:0/1023458096 >> v2:172.21.15.184:3300/0 conn(0x7f45740d5520 0x7f45740d7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 --2- 172.21.15.83:0/1023458096 >> v2:172.21.15.83:3300/0 conn(0x7f457404bef0 0x7f45740d4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 --2- 172.21.15.83:0/1023458096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 0x7f4574047d90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 -- 172.21.15.83:0/1023458096 >> 172.21.15.83:0/1023458096 conn(0x7f4574048e30 msgr2=0x7f4574049210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:18.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 -- 172.21.15.83:0/1023458096 shutdown_connections 2024-09-19T07:41:18.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 -- 172.21.15.83:0/1023458096 wait complete. 2024-09-19T07:41:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 Processor -- start 2024-09-19T07:41:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 -- start start 2024-09-19T07:41:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 0x7f45740d6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f457404bef0 0x7f457417c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f45740d5520 0x7f457417c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f45740db130 con 0x7f457404bef0 2024-09-19T07:41:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4574031da0 con 0x7f45740d5520 2024-09-19T07:41:18.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f458131c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f45740db2b0 con 0x7f45740317b0 2024-09-19T07:41:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f457b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f457404bef0 0x7f457417c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f457bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 0x7f45740d6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f457b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f457404bef0 0x7f457417c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56928/0 (socket says 172.21.15.83:56928) 2024-09-19T07:41:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f457bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 0x7f45740d6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:42886/0 (socket says 172.21.15.83:42886) 2024-09-19T07:41:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f457b7fe640 1 -- 172.21.15.83:0/1723829657 learned_addr learned my addr 172.21.15.83:0/1723829657 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:18.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f4580b1b640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f45740d5520 0x7f457417c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f457b7fe640 1 -- 172.21.15.83:0/1723829657 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 msgr2=0x7f45740d6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f457b7fe640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 0x7f45740d6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f457b7fe640 1 -- 172.21.15.83:0/1723829657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f45740d5520 msgr2=0x7f457417c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f457b7fe640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f45740d5520 0x7f457417c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.740+0000 7f457b7fe640 1 -- 172.21.15.83:0/1723829657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4574030480 con 0x7f457404bef0 2024-09-19T07:41:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.744+0000 7f457b7fe640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f457404bef0 0x7f457417c2b0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f456800ec10 tx=0x7f456800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.744+0000 7f4578ff9640 1 -- 172.21.15.83:0/1723829657 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4568017070 con 0x7f457404bef0 2024-09-19T07:41:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.744+0000 7f4578ff9640 1 -- 172.21.15.83:0/1723829657 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f456800cc30 con 0x7f457404bef0 2024-09-19T07:41:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.744+0000 7f458131c640 1 -- 172.21.15.83:0/1723829657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4574030760 con 0x7f457404bef0 2024-09-19T07:41:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.744+0000 7f4578ff9640 1 -- 172.21.15.83:0/1723829657 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4568013470 con 0x7f457404bef0 2024-09-19T07:41:18.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.744+0000 7f458131c640 1 -- 172.21.15.83:0/1723829657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45740da890 con 0x7f457404bef0 2024-09-19T07:41:18.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.744+0000 7f45667fc640 1 -- 172.21.15.83:0/1723829657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45540052a0 con 0x7f457404bef0 2024-09-19T07:41:18.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.748+0000 7f4578ff9640 1 -- 172.21.15.83:0/1723829657 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f456801a070 con 0x7f457404bef0 2024-09-19T07:41:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.748+0000 7f4578ff9640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f45600678c0 0x7f4560069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.748+0000 7f4578ff9640 1 -- 172.21.15.83:0/1723829657 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9217+0+0 (secure 0 0 0) 0x7f4568086cf0 con 0x7f457404bef0 2024-09-19T07:41:18.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.752+0000 7f457bfff640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f45600678c0 0x7f4560069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:18.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.752+0000 7f457bfff640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f45600678c0 0x7f4560069d80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f45700ad000 tx=0x7f45700a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:18.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:18.752+0000 7f4578ff9640 1 -- 172.21.15.83:0/1723829657 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4568056400 con 0x7f457404bef0 2024-09-19T07:41:19.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.208+0000 7f45667fc640 1 -- 172.21.15.83:0/1723829657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4554005590 con 0x7f457404bef0 2024-09-19T07:41:19.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.208+0000 7f4578ff9640 1 -- 172.21.15.83:0/1723829657 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) ==== 95+0+6546 (secure 0 0 0) 0x7f456805a700 con 0x7f457404bef0 2024-09-19T07:41:19.212 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:19.212 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":155,"btime":"2024-09-19T07:41:15:459617+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:19.213 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 155 2024-09-19T07:41:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.224+0000 7f45667fc640 1 -- 172.21.15.83:0/1723829657 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f45600678c0 msgr2=0x7f4560069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.224+0000 7f45667fc640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f45600678c0 0x7f4560069d80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f45700ad000 tx=0x7f45700a6000 comp rx=0 tx=0).stop 2024-09-19T07:41:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.224+0000 7f45667fc640 1 -- 172.21.15.83:0/1723829657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f457404bef0 msgr2=0x7f457417c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.224+0000 7f45667fc640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f457404bef0 0x7f457417c2b0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f456800ec10 tx=0x7f456800c520 comp rx=0 tx=0).stop 2024-09-19T07:41:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.224+0000 7f45667fc640 1 -- 172.21.15.83:0/1723829657 shutdown_connections 2024-09-19T07:41:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.224+0000 7f45667fc640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f45600678c0 0x7f4560069d80 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.224+0000 7f45667fc640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f45740d5520 0x7f457417c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.224+0000 7f45667fc640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f457404bef0 0x7f457417c2b0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:19.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.224+0000 7f45667fc640 1 --2- 172.21.15.83:0/1723829657 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f45740317b0 0x7f45740d6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:19.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.224+0000 7f45667fc640 1 -- 172.21.15.83:0/1723829657 >> 172.21.15.83:0/1723829657 conn(0x7f4574048e30 msgr2=0x7f45740cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:19.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.224+0000 7f45667fc640 1 -- 172.21.15.83:0/1723829657 shutdown_connections 2024-09-19T07:41:19.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:19.224+0000 7f45667fc640 1 -- 172.21.15.83:0/1723829657 wait complete. 2024-09-19T07:41:19.250 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:41:21.252 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:21.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbdc96a640 1 Processor -- start 2024-09-19T07:41:21.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbdc96a640 1 -- start start 2024-09-19T07:41:21.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbdc96a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fdbd8156790 0x7fdbd806e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:21.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbdc96a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdbd806e550 0x7fdbd806e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:21.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbdc96a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbd80acb90 con 0x7fdbd80c0c80 2024-09-19T07:41:21.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbdc96a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fdbd80abf80 con 0x7fdbd806e550 2024-09-19T07:41:21.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbdc96a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fdbd80ac260 con 0x7fdbd8156790 2024-09-19T07:41:21.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbd6d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdbd806e550 0x7fdbd806e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:21.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbd6d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdbd806e550 0x7fdbd806e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43988/0 (socket says 172.21.15.83:43988) 2024-09-19T07:41:21.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbd6d9d640 1 -- 172.21.15.83:0/2732288623 learned_addr learned my addr 172.21.15.83:0/2732288623 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:21.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbd5d9b640 1 --2- 172.21.15.83:0/2732288623 >> v2:172.21.15.184:3300/0 conn(0x7fdbd8156790 0x7fdbd806e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:21.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbd6d9d640 1 -- 172.21.15.83:0/2732288623 >> v2:172.21.15.184:3300/0 conn(0x7fdbd8156790 msgr2=0x7fdbd806e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:21.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbd6d9d640 1 --2- 172.21.15.83:0/2732288623 >> v2:172.21.15.184:3300/0 conn(0x7fdbd8156790 0x7fdbd806e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:21.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbd6d9d640 1 -- 172.21.15.83:0/2732288623 >> v1:172.21.15.83:6789/0 conn(0x7fdbd80c0c80 legacy=0x7fdbd8153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:21.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbd6d9d640 1 -- 172.21.15.83:0/2732288623 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd80ac540 con 0x7fdbd806e550 2024-09-19T07:41:21.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.435+0000 7fdbd5d9b640 1 --2- 172.21.15.83:0/2732288623 >> v2:172.21.15.184:3300/0 conn(0x7fdbd8156790 0x7fdbd806e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:41:21.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.443+0000 7fdbd6d9d640 1 --2- 172.21.15.83:0/2732288623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd806e550 0x7fdbd806e960 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fdbd80c4ad0 tx=0x7fdbc801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4772169a088ca666 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:21.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.443+0000 7fdbd559a640 1 -- 172.21.15.83:0/2732288623 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc80ab650 con 0x7fdbd806e550 2024-09-19T07:41:21.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.447+0000 7fdbd559a640 1 -- 172.21.15.83:0/2732288623 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbc80b0070 con 0x7fdbd806e550 2024-09-19T07:41:21.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.447+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2732288623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd806e550 msgr2=0x7fdbd806e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:21.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.447+0000 7fdbd559a640 1 -- 172.21.15.83:0/2732288623 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc80ab650 con 0x7fdbd806e550 2024-09-19T07:41:21.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.447+0000 7fdbdc96a640 1 --2- 172.21.15.83:0/2732288623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd806e550 0x7fdbd806e960 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fdbd80c4ad0 tx=0x7fdbc801c4f0 comp rx=0 tx=0).stop 2024-09-19T07:41:21.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.451+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2732288623 shutdown_connections 2024-09-19T07:41:21.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.451+0000 7fdbdc96a640 1 --2- 172.21.15.83:0/2732288623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd806e550 0x7fdbd806e960 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:21.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.451+0000 7fdbdc96a640 1 --2- 172.21.15.83:0/2732288623 >> v2:172.21.15.184:3300/0 conn(0x7fdbd8156790 0x7fdbd806e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:21.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.451+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2732288623 >> 172.21.15.83:0/2732288623 conn(0x7fdbd8075390 msgr2=0x7fdbd806bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:21.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.455+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2732288623 shutdown_connections 2024-09-19T07:41:21.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.455+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2732288623 wait complete. 2024-09-19T07:41:21.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.455+0000 7fdbdc96a640 1 Processor -- start 2024-09-19T07:41:21.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbdc96a640 1 -- start start 2024-09-19T07:41:21.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbdc96a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd806e550 0x7fdbd8070f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:21.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbdc96a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbd80c0c80 0x7fdbd81f8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:21.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbdc96a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbd8156790 0x7fdbd80c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:21.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbdc96a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd80abcb0 con 0x7fdbd806e550 2024-09-19T07:41:21.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbdc96a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd80abb30 con 0x7fdbd8156790 2024-09-19T07:41:21.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbdc96a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdbd8153750 con 0x7fdbd80c0c80 2024-09-19T07:41:21.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbd659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd806e550 0x7fdbd8070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:21.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbd659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd806e550 0x7fdbd8070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43994/0 (socket says 172.21.15.83:43994) 2024-09-19T07:41:21.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbd659c640 1 -- 172.21.15.83:0/2070462268 learned_addr learned my addr 172.21.15.83:0/2070462268 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbd5d9b640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbd80c0c80 0x7fdbd81f8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbd6d9d640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbd8156790 0x7fdbd80c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbd659c640 1 -- 172.21.15.83:0/2070462268 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbd80c0c80 msgr2=0x7fdbd81f8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbd659c640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbd80c0c80 0x7fdbd81f8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbd659c640 1 -- 172.21.15.83:0/2070462268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbd8156790 msgr2=0x7fdbd80c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbd659c640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbd8156790 0x7fdbd80c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbd659c640 1 -- 172.21.15.83:0/2070462268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd80c6af0 con 0x7fdbd806e550 2024-09-19T07:41:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbd6d9d640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbd8156790 0x7fdbd80c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:41:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbd659c640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd806e550 0x7fdbd8070f50 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000c790 tx=0x7fdbd000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbc6ffd640 1 -- 172.21.15.83:0/2070462268 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0016020 con 0x7fdbd806e550 2024-09-19T07:41:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbc6ffd640 1 -- 172.21.15.83:0/2070462268 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbd0006d60 con 0x7fdbd806e550 2024-09-19T07:41:21.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbc6ffd640 1 -- 172.21.15.83:0/2070462268 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0003bc0 con 0x7fdbd806e550 2024-09-19T07:41:21.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2070462268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd81fbd50 con 0x7fdbd806e550 2024-09-19T07:41:21.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2070462268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbd80c7070 con 0x7fdbd806e550 2024-09-19T07:41:21.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.459+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2070462268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbd80c7270 con 0x7fdbd806e550 2024-09-19T07:41:21.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.463+0000 7fdbc6ffd640 1 -- 172.21.15.83:0/2070462268 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdbd000a850 con 0x7fdbd806e550 2024-09-19T07:41:21.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.463+0000 7fdbc6ffd640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdbb8067990 0x7fdbb8069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:21.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.463+0000 7fdbc6ffd640 1 -- 172.21.15.83:0/2070462268 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9217+0+0 (secure 0 0 0) 0x7fdbd0087480 con 0x7fdbd806e550 2024-09-19T07:41:21.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.467+0000 7fdbd5d9b640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdbb8067990 0x7fdbb8069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:21.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.467+0000 7fdbd5d9b640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdbb8067990 0x7fdbb8069e50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009580 tx=0x7fdbcc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:21.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.467+0000 7fdbc6ffd640 1 -- 172.21.15.83:0/2070462268 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdbd80c7270 con 0x7fdbd806e550 2024-09-19T07:41:21.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.839+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2070462268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdbd80722a0 con 0x7fdbd806e550 2024-09-19T07:41:21.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.839+0000 7fdbc6ffd640 1 -- 172.21.15.83:0/2070462268 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) ==== 95+0+6546 (secure 0 0 0) 0x7fdbd0020020 con 0x7fdbd806e550 2024-09-19T07:41:21.840 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:21.840 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":155,"btime":"2024-09-19T07:41:15:459617+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:21.841 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 155 2024-09-19T07:41:21.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.843+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2070462268 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdbb8067990 msgr2=0x7fdbb8069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:21.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.843+0000 7fdbdc96a640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdbb8067990 0x7fdbb8069e50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009580 tx=0x7fdbcc009190 comp rx=0 tx=0).stop 2024-09-19T07:41:21.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.843+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2070462268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd806e550 msgr2=0x7fdbd8070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:21.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.843+0000 7fdbdc96a640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd806e550 0x7fdbd8070f50 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000c790 tx=0x7fdbd000cc50 comp rx=0 tx=0).stop 2024-09-19T07:41:21.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.843+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2070462268 shutdown_connections 2024-09-19T07:41:21.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.843+0000 7fdbdc96a640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdbb8067990 0x7fdbb8069e50 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:21.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.843+0000 7fdbdc96a640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbd8156790 0x7fdbd80c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:21.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.843+0000 7fdbdc96a640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbd80c0c80 0x7fdbd81f8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:21.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.843+0000 7fdbdc96a640 1 --2- 172.21.15.83:0/2070462268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbd806e550 0x7fdbd8070f50 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:21.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.843+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2070462268 >> 172.21.15.83:0/2070462268 conn(0x7fdbd8075390 msgr2=0x7fdbd815fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:21.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.843+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2070462268 shutdown_connections 2024-09-19T07:41:21.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:21.843+0000 7fdbdc96a640 1 -- 172.21.15.83:0/2070462268 wait complete. 2024-09-19T07:41:21.863 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:41:23.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:23.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.987+0000 7f9c1945a640 1 Processor -- start 2024-09-19T07:41:23.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.987+0000 7f9c1945a640 1 -- start start 2024-09-19T07:41:23.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c1945a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c141512c0 0x7f9c14157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:23.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c1945a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9c140c4010 0x7f9c140c4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:23.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c1945a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c14158000 con 0x7f9c141540f0 2024-09-19T07:41:23.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c1945a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f9c14153c60 con 0x7f9c140c4010 2024-09-19T07:41:23.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c1945a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9c140730f0 con 0x7f9c141512c0 2024-09-19T07:41:23.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c137fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9c141540f0 0x7f9c1406de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38028/0 (socket says 172.21.15.83:38028) 2024-09-19T07:41:23.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c137fe640 1 -- 172.21.15.83:0/1042914246 learned_addr learned my addr 172.21.15.83:0/1042914246 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:41:23.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c127fc640 1 --2- 172.21.15.83:0/1042914246 >> v2:172.21.15.184:3300/0 conn(0x7f9c140c4010 0x7f9c140c4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:23.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c12ffd640 1 --2- 172.21.15.83:0/1042914246 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c141512c0 0x7f9c14157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:23.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c11ffb640 1 -- 172.21.15.83:0/1042914246 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3111220807 0 0) 0x7f9c0c0014a0 con 0x7f9c141540f0 2024-09-19T07:41:23.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c11ffb640 1 -- 172.21.15.83:0/1042914246 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bfc003530 con 0x7f9c141540f0 2024-09-19T07:41:23.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c12ffd640 1 -- 172.21.15.83:0/1042914246 >> v2:172.21.15.184:3300/0 conn(0x7f9c140c4010 msgr2=0x7f9c140c4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:23.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c12ffd640 1 --2- 172.21.15.83:0/1042914246 >> v2:172.21.15.184:3300/0 conn(0x7f9c140c4010 0x7f9c140c4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:23.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c12ffd640 1 -- 172.21.15.83:0/1042914246 >> v1:172.21.15.83:6789/0 conn(0x7f9c141540f0 legacy=0x7f9c1406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:23.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c12ffd640 1 -- 172.21.15.83:0/1042914246 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c140733b0 con 0x7f9c141512c0 2024-09-19T07:41:23.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c127fc640 1 --2- 172.21.15.83:0/1042914246 >> v2:172.21.15.184:3300/0 conn(0x7f9c140c4010 0x7f9c140c4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:41:23.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c12ffd640 1 --2- 172.21.15.83:0/1042914246 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c141512c0 0x7f9c14157ac0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9c040097c0 tx=0x7f9c0401cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8310ec138bec718 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:23.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c11ffb640 1 -- 172.21.15.83:0/1042914246 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c040b0070 con 0x7f9c141512c0 2024-09-19T07:41:23.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c11ffb640 1 -- 172.21.15.83:0/1042914246 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c040ab600 con 0x7f9c141512c0 2024-09-19T07:41:23.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c11ffb640 1 -- 172.21.15.83:0/1042914246 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c040ab9d0 con 0x7f9c141512c0 2024-09-19T07:41:23.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c1945a640 1 -- 172.21.15.83:0/1042914246 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c141512c0 msgr2=0x7f9c14157ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:23.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c1945a640 1 --2- 172.21.15.83:0/1042914246 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c141512c0 0x7f9c14157ac0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9c040097c0 tx=0x7f9c0401cde0 comp rx=0 tx=0).stop 2024-09-19T07:41:23.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c1945a640 1 -- 172.21.15.83:0/1042914246 shutdown_connections 2024-09-19T07:41:23.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c1945a640 1 --2- 172.21.15.83:0/1042914246 >> v2:172.21.15.184:3300/0 conn(0x7f9c140c4010 0x7f9c140c4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:23.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c1945a640 1 --2- 172.21.15.83:0/1042914246 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c141512c0 0x7f9c14157ac0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:23.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.991+0000 7f9c1945a640 1 -- 172.21.15.83:0/1042914246 >> 172.21.15.83:0/1042914246 conn(0x7f9c140c9610 msgr2=0x7f9c140c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:23.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 -- 172.21.15.83:0/1042914246 shutdown_connections 2024-09-19T07:41:23.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 -- 172.21.15.83:0/1042914246 wait complete. 2024-09-19T07:41:23.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 Processor -- start 2024-09-19T07:41:23.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 -- start start 2024-09-19T07:41:23.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c140c4010 0x7f9c141482c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:23.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c141512c0 0x7f9c140ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:23.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c141540f0 0x7f9c1406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:23.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9c140c76d0 con 0x7f9c141512c0 2024-09-19T07:41:23.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9c14154930 con 0x7f9c141540f0 2024-09-19T07:41:23.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9c140c7850 con 0x7f9c140c4010 2024-09-19T07:41:23.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c127fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c141512c0 0x7f9c140ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:23.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c127fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c141512c0 0x7f9c140ae8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44002/0 (socket says 172.21.15.83:44002) 2024-09-19T07:41:23.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c127fc640 1 -- 172.21.15.83:0/723676672 learned_addr learned my addr 172.21.15.83:0/723676672 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:23.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c12ffd640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c140c4010 0x7f9c141482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:23.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c137fe640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c141540f0 0x7f9c1406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:23.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c12ffd640 1 -- 172.21.15.83:0/723676672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c141540f0 msgr2=0x7f9c1406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:23.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c12ffd640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c141540f0 0x7f9c1406c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:23.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c12ffd640 1 -- 172.21.15.83:0/723676672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c141512c0 msgr2=0x7f9c140ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:23.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c12ffd640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c141512c0 0x7f9c140ae8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:23.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c12ffd640 1 -- 172.21.15.83:0/723676672 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1406bd60 con 0x7f9c140c4010 2024-09-19T07:41:23.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c12ffd640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c140c4010 0x7f9c141482c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9c040ab000 tx=0x7f9c0401abc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:23.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9bfb7fe640 1 -- 172.21.15.83:0/723676672 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c040b0050 con 0x7f9c140c4010 2024-09-19T07:41:23.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9bfb7fe640 1 -- 172.21.15.83:0/723676672 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c040bc070 con 0x7f9c140c4010 2024-09-19T07:41:23.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 -- 172.21.15.83:0/723676672 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c1406c0d0 con 0x7f9c140c4010 2024-09-19T07:41:24.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9bfb7fe640 1 -- 172.21.15.83:0/723676672 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c04006e20 con 0x7f9c140c4010 2024-09-19T07:41:24.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 -- 172.21.15.83:0/723676672 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c141f1dc0 con 0x7f9c140c4010 2024-09-19T07:41:24.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.995+0000 7f9c1945a640 1 -- 172.21.15.83:0/723676672 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c140c7fa0 con 0x7f9c140c4010 2024-09-19T07:41:24.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.999+0000 7f9bfb7fe640 1 -- 172.21.15.83:0/723676672 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9c040b5850 con 0x7f9c140c4010 2024-09-19T07:41:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.999+0000 7f9bfb7fe640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9c000678f0 0x7f9c00069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.999+0000 7f9bfb7fe640 1 -- 172.21.15.83:0/723676672 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 9217+0+0 (secure 0 0 0) 0x7f9c041219e0 con 0x7f9c140c4010 2024-09-19T07:41:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.999+0000 7f9c127fc640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9c000678f0 0x7f9c00069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:23.999+0000 7f9c127fc640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9c000678f0 0x7f9c00069db0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009580 tx=0x7f9c08009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:24.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.003+0000 7f9bfb7fe640 1 -- 172.21.15.83:0/723676672 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9c04016090 con 0x7f9c140c4010 2024-09-19T07:41:24.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.187+0000 7f9c1945a640 1 -- 172.21.15.83:0/723676672 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c140c3990 con 0x7f9c140c4010 2024-09-19T07:41:24.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.187+0000 7f9bfb7fe640 1 -- 172.21.15.83:0/723676672 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) ==== 95+0+6546 (secure 0 0 0) 0x7f9c040f11d0 con 0x7f9c140c4010 2024-09-19T07:41:24.188 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:24.188 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":155,"btime":"2024-09-19T07:41:15:459617+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:24.188 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 155 2024-09-19T07:41:24.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.187+0000 7f9c1945a640 1 -- 172.21.15.83:0/723676672 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9c000678f0 msgr2=0x7f9c00069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:24.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.187+0000 7f9c1945a640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9c000678f0 0x7f9c00069db0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009580 tx=0x7f9c08009190 comp rx=0 tx=0).stop 2024-09-19T07:41:24.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.187+0000 7f9c1945a640 1 -- 172.21.15.83:0/723676672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c140c4010 msgr2=0x7f9c141482c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:24.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.187+0000 7f9c1945a640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c140c4010 0x7f9c141482c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9c040ab000 tx=0x7f9c0401abc0 comp rx=0 tx=0).stop 2024-09-19T07:41:24.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.191+0000 7f9c1945a640 1 -- 172.21.15.83:0/723676672 shutdown_connections 2024-09-19T07:41:24.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.191+0000 7f9c1945a640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9c000678f0 0x7f9c00069db0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:24.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.191+0000 7f9c1945a640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c141540f0 0x7f9c1406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:24.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.191+0000 7f9c1945a640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c141512c0 0x7f9c140ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:24.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.191+0000 7f9c1945a640 1 --2- 172.21.15.83:0/723676672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c140c4010 0x7f9c141482c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:24.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.191+0000 7f9c1945a640 1 -- 172.21.15.83:0/723676672 >> 172.21.15.83:0/723676672 conn(0x7f9c140c9610 msgr2=0x7f9c1414f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:24.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.191+0000 7f9c1945a640 1 -- 172.21.15.83:0/723676672 shutdown_connections 2024-09-19T07:41:24.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:24.191+0000 7f9c1945a640 1 -- 172.21.15.83:0/723676672 wait complete. 2024-09-19T07:41:24.205 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:41:26.206 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:26.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.343+0000 7f7fe70b2640 1 Processor -- start 2024-09-19T07:41:26.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.343+0000 7f7fe70b2640 1 -- start start 2024-09-19T07:41:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe00c28e0 0x7f7fe01545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7fe006dea0 0x7f7fe006e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fe00a9e60 con 0x7f7fe0071100 2024-09-19T07:41:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7fe0071b50 con 0x7f7fe006dea0 2024-09-19T07:41:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe00c1ff0 con 0x7f7fe00c28e0 2024-09-19T07:41:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe564f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7fe006dea0 0x7f7fe006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe564f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7fe006dea0 0x7f7fe006e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44004/0 (socket says 172.21.15.83:44004) 2024-09-19T07:41:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe564f640 1 -- 172.21.15.83:0/1432001633 learned_addr learned my addr 172.21.15.83:0/1432001633 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe564f640 1 -- 172.21.15.83:0/1432001633 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe00c28e0 msgr2=0x7f7fe01545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fdffff640 1 --2- 172.21.15.83:0/1432001633 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe00c28e0 0x7f7fe01545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe564f640 1 --2- 172.21.15.83:0/1432001633 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe00c28e0 0x7f7fe01545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:26.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe564f640 1 -- 172.21.15.83:0/1432001633 >> v1:172.21.15.184:6789/0 conn(0x7f7fe0071100 legacy=0x7f7fe0150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:26.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe564f640 1 -- 172.21.15.83:0/1432001633 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe00aa140 con 0x7f7fe006dea0 2024-09-19T07:41:26.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe564f640 1 --2- 172.21.15.83:0/1432001633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe006dea0 0x7f7fe006e350 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009e50 tx=0x7f7fd801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20240a843bcd3a6c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:26.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fdf7fe640 1 -- 172.21.15.83:0/1432001633 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80a55e0 con 0x7f7fe006dea0 2024-09-19T07:41:26.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fdf7fe640 1 -- 172.21.15.83:0/1432001633 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fd80a5780 con 0x7f7fe006dea0 2024-09-19T07:41:26.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fdf7fe640 1 -- 172.21.15.83:0/1432001633 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80a5b50 con 0x7f7fe006dea0 2024-09-19T07:41:26.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 -- 172.21.15.83:0/1432001633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe006dea0 msgr2=0x7f7fe006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:26.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 --2- 172.21.15.83:0/1432001633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe006dea0 0x7f7fe006e350 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009e50 tx=0x7f7fd801c960 comp rx=0 tx=0).stop 2024-09-19T07:41:26.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 -- 172.21.15.83:0/1432001633 shutdown_connections 2024-09-19T07:41:26.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 --2- 172.21.15.83:0/1432001633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe006dea0 0x7f7fe006e350 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:26.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 --2- 172.21.15.83:0/1432001633 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe00c28e0 0x7f7fe01545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:26.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 -- 172.21.15.83:0/1432001633 >> 172.21.15.83:0/1432001633 conn(0x7f7fe006bc80 msgr2=0x7f7fe006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:26.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 -- 172.21.15.83:0/1432001633 shutdown_connections 2024-09-19T07:41:26.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.347+0000 7f7fe70b2640 1 -- 172.21.15.83:0/1432001633 wait complete. 2024-09-19T07:41:26.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe70b2640 1 Processor -- start 2024-09-19T07:41:26.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe70b2640 1 -- start start 2024-09-19T07:41:26.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe70b2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe006dea0 0x7f7fe0157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:26.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe70b2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe0071100 0x7f7fe0147900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:26.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe70b2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe00c28e0 0x7f7fe0152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:26.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe70b2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe0071660 con 0x7f7fe006dea0 2024-09-19T07:41:26.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe70b2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe00714e0 con 0x7f7fe00c28e0 2024-09-19T07:41:26.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe70b2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe0160c60 con 0x7f7fe0071100 2024-09-19T07:41:26.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe4e4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe006dea0 0x7f7fe0157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:26.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe4e4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe006dea0 0x7f7fe0157b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44008/0 (socket says 172.21.15.83:44008) 2024-09-19T07:41:26.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe4e4e640 1 -- 172.21.15.83:0/122730704 learned_addr learned my addr 172.21.15.83:0/122730704 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:26.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe564f640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe00c28e0 0x7f7fe0152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:26.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fdffff640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe0071100 0x7f7fe0147900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:26.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe4e4e640 1 -- 172.21.15.83:0/122730704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe0071100 msgr2=0x7f7fe0147900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:26.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe4e4e640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe0071100 0x7f7fe0147900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:26.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe4e4e640 1 -- 172.21.15.83:0/122730704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe00c28e0 msgr2=0x7f7fe0152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:26.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe4e4e640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe00c28e0 0x7f7fe0152390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:26.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe4e4e640 1 -- 172.21.15.83:0/122730704 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe0147e40 con 0x7f7fe006dea0 2024-09-19T07:41:26.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe564f640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe00c28e0 0x7f7fe0152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:41:26.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe4e4e640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe006dea0 0x7f7fe0157b40 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000c450 tx=0x7f7fd000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:26.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fdd7fa640 1 -- 172.21.15.83:0/122730704 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0012830 con 0x7f7fe006dea0 2024-09-19T07:41:26.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fdd7fa640 1 -- 172.21.15.83:0/122730704 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fd0017070 con 0x7f7fe006dea0 2024-09-19T07:41:26.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe70b2640 1 -- 172.21.15.83:0/122730704 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe01528d0 con 0x7f7fe006dea0 2024-09-19T07:41:26.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fdd7fa640 1 -- 172.21.15.83:0/122730704 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0012c30 con 0x7f7fe006dea0 2024-09-19T07:41:26.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe70b2640 1 -- 172.21.15.83:0/122730704 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe01f0ba0 con 0x7f7fe006dea0 2024-09-19T07:41:26.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.351+0000 7f7fe70b2640 1 -- 172.21.15.83:0/122730704 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe006e350 con 0x7f7fe006dea0 2024-09-19T07:41:26.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.355+0000 7f7fdd7fa640 1 -- 172.21.15.83:0/122730704 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7fd001c460 con 0x7f7fe006dea0 2024-09-19T07:41:26.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.355+0000 7f7fdd7fa640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7fc40678f0 0x7f7fc4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:26.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.355+0000 7f7fdd7fa640 1 -- 172.21.15.83:0/122730704 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9217+0+0 (secure 0 0 0) 0x7f7fd0086820 con 0x7f7fe006dea0 2024-09-19T07:41:26.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.355+0000 7f7fdffff640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7fc40678f0 0x7f7fc4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:26.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.355+0000 7f7fdffff640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7fc40678f0 0x7f7fc4069db0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7fe01544f0 tx=0x7f7fc8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:26.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.359+0000 7f7fdd7fa640 1 -- 172.21.15.83:0/122730704 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7fd0056550 con 0x7f7fe006dea0 2024-09-19T07:41:26.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 -- 172.21.15.83:0/122730704 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7fe00c4f30 con 0x7f7fe006dea0 2024-09-19T07:41:26.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fdd7fa640 1 -- 172.21.15.83:0/122730704 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) ==== 95+0+6546 (secure 0 0 0) 0x7f7fd0055e50 con 0x7f7fe006dea0 2024-09-19T07:41:26.555 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:26.556 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":155,"btime":"2024-09-19T07:41:15:459617+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:26.556 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 155 2024-09-19T07:41:26.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 -- 172.21.15.83:0/122730704 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7fc40678f0 msgr2=0x7f7fc4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:26.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7fc40678f0 0x7f7fc4069db0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7fe01544f0 tx=0x7f7fc8008040 comp rx=0 tx=0).stop 2024-09-19T07:41:26.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 -- 172.21.15.83:0/122730704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe006dea0 msgr2=0x7f7fe0157b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:26.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe006dea0 0x7f7fe0157b40 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000c450 tx=0x7f7fd000c910 comp rx=0 tx=0).stop 2024-09-19T07:41:26.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 -- 172.21.15.83:0/122730704 shutdown_connections 2024-09-19T07:41:26.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7fc40678f0 0x7f7fc4069db0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:26.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fe00c28e0 0x7f7fe0152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:26.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fe0071100 0x7f7fe0147900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:26.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 --2- 172.21.15.83:0/122730704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fe006dea0 0x7f7fe0157b40 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:26.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 -- 172.21.15.83:0/122730704 >> 172.21.15.83:0/122730704 conn(0x7f7fe006bc80 msgr2=0x7f7fe0150390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:26.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 -- 172.21.15.83:0/122730704 shutdown_connections 2024-09-19T07:41:26.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:26.559+0000 7f7fe70b2640 1 -- 172.21.15.83:0/122730704 wait complete. 2024-09-19T07:41:26.574 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:41:28.575 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:28.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc66e21d640 1 Processor -- start 2024-09-19T07:41:28.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc66e21d640 1 -- start start 2024-09-19T07:41:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc66e21d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6680ab610 0x7fc6680abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc66e21d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6680c7de0 con 0x7fc6680c6fc0 2024-09-19T07:41:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc66e21d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6680c8080 con 0x7fc66806cef0 2024-09-19T07:41:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc66e21d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc6680c6a90 con 0x7fc6680ab610 2024-09-19T07:41:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc667fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc6680c6fc0 0x7fc668157890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38040/0 (socket says 172.21.15.83:38040) 2024-09-19T07:41:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc667fff640 1 -- 172.21.15.83:0/1115519058 learned_addr learned my addr 172.21.15.83:0/1115519058 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:41:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc666ffd640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6680ab610 0x7fc6680abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1538466345 0 0) 0x7fc6680c7de0 con 0x7fc6680c6fc0 2024-09-19T07:41:28.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc650003530 con 0x7fc6680c6fc0 2024-09-19T07:41:28.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4106036251 0 0) 0x7fc650003530 con 0x7fc6680c6fc0 2024-09-19T07:41:28.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6680ab610 msgr2=0x7fc6680abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:28.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6680ab610 0x7fc6680abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:28.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 >> v1:172.21.15.184:6789/0 conn(0x7fc66806cef0 legacy=0x7fc6680c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:28.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6680c6720 con 0x7fc6680c6fc0 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc666ffd640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6680ab610 0x7fc6680abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fc660001a00 con 0x7fc6680c6fc0 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 >> v1:172.21.15.83:6789/0 conn(0x7fc6680c6fc0 legacy=0x7fc668157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6680ab610 0x7fc6500039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc650003fe0 0x7fc650004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc650004d50 0x7fc6500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc65000bc30 con 0x7fc650003fe0 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc65000bde0 con 0x7fc650004d50 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc65000bfe0 con 0x7fc6680ab610 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc666ffd640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc650003fe0 0x7fc650004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc6677fe640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6680ab610 0x7fc6500039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc667fff640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc650004d50 0x7fc6500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:28.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc666ffd640 1 -- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6680ab610 msgr2=0x7fc6500039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:28.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc666ffd640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6680ab610 0x7fc6500039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:28.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc666ffd640 1 -- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc650004d50 msgr2=0x7fc6500051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:28.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc666ffd640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc650004d50 0x7fc6500051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:28.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc666ffd640 1 -- 172.21.15.83:0/1115519058 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc650003530 con 0x7fc650003fe0 2024-09-19T07:41:28.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.707+0000 7fc667fff640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc650004d50 0x7fc6500051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:41:28.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.711+0000 7fc666ffd640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc650003fe0 0x7fc650004810 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fc6680a8250 tx=0x7fc65801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf69e29c83ca85 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:28.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.711+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6580b2070 con 0x7fc650003fe0 2024-09-19T07:41:28.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.711+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6580a9070 con 0x7fc650003fe0 2024-09-19T07:41:28.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.711+0000 7fc6667fc640 1 -- 172.21.15.83:0/1115519058 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65800f040 con 0x7fc650003fe0 2024-09-19T07:41:28.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.711+0000 7fc66e21d640 1 -- 172.21.15.83:0/1115519058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc650003fe0 msgr2=0x7fc650004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:28.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.711+0000 7fc66e21d640 1 --2- 172.21.15.83:0/1115519058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc650003fe0 0x7fc650004810 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fc6680a8250 tx=0x7fc65801c530 comp rx=0 tx=0).stop 2024-09-19T07:41:28.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.711+0000 7fc667fff640 1 -- 172.21.15.83:0/1115519058 reap_dead start 2024-09-19T07:41:28.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.711+0000 7fc66e21d640 1 -- 172.21.15.83:0/1115519058 shutdown_connections 2024-09-19T07:41:28.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.711+0000 7fc66e21d640 1 -- 172.21.15.83:0/1115519058 >> 172.21.15.83:0/1115519058 conn(0x7fc6680aa910 msgr2=0x7fc6680c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:28.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.711+0000 7fc66e21d640 1 -- 172.21.15.83:0/1115519058 shutdown_connections 2024-09-19T07:41:28.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.711+0000 7fc66e21d640 1 -- 172.21.15.83:0/1115519058 wait complete. 2024-09-19T07:41:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.711+0000 7fc66e21d640 1 Processor -- start 2024-09-19T07:41:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc66e21d640 1 -- start start 2024-09-19T07:41:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc66e21d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc650003fe0 0x7fc6680c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc66e21d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668153040 0x7fc668149620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc66e21d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc668152740 0x7fc668149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc66e21d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc66806d4e0 con 0x7fc668153040 2024-09-19T07:41:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc66e21d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681581b0 con 0x7fc668152740 2024-09-19T07:41:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc66e21d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc6680c73f0 con 0x7fc650003fe0 2024-09-19T07:41:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc666ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668153040 0x7fc668149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc666ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668153040 0x7fc668149620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44032/0 (socket says 172.21.15.83:44032) 2024-09-19T07:41:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc666ffd640 1 -- 172.21.15.83:0/457378928 learned_addr learned my addr 172.21.15.83:0/457378928 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc6677fe640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc650003fe0 0x7fc6680c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc667fff640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc668152740 0x7fc668149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc666ffd640 1 -- 172.21.15.83:0/457378928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc650003fe0 msgr2=0x7fc6680c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc666ffd640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc650003fe0 0x7fc6680c1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc666ffd640 1 -- 172.21.15.83:0/457378928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc668152740 msgr2=0x7fc668149b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc666ffd640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc668152740 0x7fc668149b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc666ffd640 1 -- 172.21.15.83:0/457378928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc668153f40 con 0x7fc668153040 2024-09-19T07:41:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc666ffd640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668153040 0x7fc668149620 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fc6580b0040 tx=0x7fc650005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc64ffff640 1 -- 172.21.15.83:0/457378928 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6580bd020 con 0x7fc668153040 2024-09-19T07:41:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc64ffff640 1 -- 172.21.15.83:0/457378928 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6580add00 con 0x7fc668153040 2024-09-19T07:41:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc66e21d640 1 -- 172.21.15.83:0/457378928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6681512d0 con 0x7fc668153040 2024-09-19T07:41:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc64ffff640 1 -- 172.21.15.83:0/457378928 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6580b2040 con 0x7fc668153040 2024-09-19T07:41:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc66e21d640 1 -- 172.21.15.83:0/457378928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6680c2130 con 0x7fc668153040 2024-09-19T07:41:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc66e21d640 1 -- 172.21.15.83:0/457378928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6680abe80 con 0x7fc668153040 2024-09-19T07:41:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc64ffff640 1 -- 172.21.15.83:0/457378928 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc6580c0020 con 0x7fc668153040 2024-09-19T07:41:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc64ffff640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6540678f0 0x7fc654069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc64ffff640 1 -- 172.21.15.83:0/457378928 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9217+0+0 (secure 0 0 0) 0x7fc658120a10 con 0x7fc668153040 2024-09-19T07:41:28.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.715+0000 7fc6677fe640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6540678f0 0x7fc654069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:28.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.719+0000 7fc6677fe640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6540678f0 0x7fc654069db0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc65c000c10 tx=0x7fc65c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:28.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.723+0000 7fc64ffff640 1 -- 172.21.15.83:0/457378928 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc6580c7070 con 0x7fc668153040 2024-09-19T07:41:28.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.919+0000 7fc66e21d640 1 -- 172.21.15.83:0/457378928 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6680c4340 con 0x7fc668153040 2024-09-19T07:41:28.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.919+0000 7fc64ffff640 1 -- 172.21.15.83:0/457378928 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) ==== 95+0+6546 (secure 0 0 0) 0x7fc658016240 con 0x7fc668153040 2024-09-19T07:41:28.917 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:28.917 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":155,"btime":"2024-09-19T07:41:15:459617+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:28.918 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 155 2024-09-19T07:41:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.923+0000 7fc66e21d640 1 -- 172.21.15.83:0/457378928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6540678f0 msgr2=0x7fc654069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.923+0000 7fc66e21d640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6540678f0 0x7fc654069db0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc65c000c10 tx=0x7fc65c009040 comp rx=0 tx=0).stop 2024-09-19T07:41:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.923+0000 7fc66e21d640 1 -- 172.21.15.83:0/457378928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668153040 msgr2=0x7fc668149620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.923+0000 7fc66e21d640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668153040 0x7fc668149620 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fc6580b0040 tx=0x7fc650005840 comp rx=0 tx=0).stop 2024-09-19T07:41:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.923+0000 7fc66e21d640 1 -- 172.21.15.83:0/457378928 shutdown_connections 2024-09-19T07:41:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.923+0000 7fc66e21d640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc6540678f0 0x7fc654069db0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.923+0000 7fc66e21d640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc668152740 0x7fc668149b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.923+0000 7fc66e21d640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc668153040 0x7fc668149620 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.923+0000 7fc66e21d640 1 --2- 172.21.15.83:0/457378928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc650003fe0 0x7fc6680c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.923+0000 7fc66e21d640 1 -- 172.21.15.83:0/457378928 >> 172.21.15.83:0/457378928 conn(0x7fc6680aa910 msgr2=0x7fc6680ac090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.923+0000 7fc66e21d640 1 -- 172.21.15.83:0/457378928 shutdown_connections 2024-09-19T07:41:28.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:28.923+0000 7fc66e21d640 1 -- 172.21.15.83:0/457378928 wait complete. 2024-09-19T07:41:28.936 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:41:30.938 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:31.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.067+0000 7f5f6dc88640 1 Processor -- start 2024-09-19T07:41:31.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.067+0000 7f5f6dc88640 1 -- start start 2024-09-19T07:41:31.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f6dc88640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f680ab750 0x7f5f680c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:31.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f6dc88640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f5f680c2530 0x7f5f681528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:31.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f6dc88640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f680ab060 0x7f5f68152e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:31.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f6dc88640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5f6806de90 con 0x7f5f680ab750 2024-09-19T07:41:31.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f6dc88640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f5f6806e170 con 0x7f5f680c2530 2024-09-19T07:41:31.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f6dc88640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6806e450 con 0x7f5f680ab060 2024-09-19T07:41:31.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f677fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f680ab750 0x7f5f680c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:31.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f677fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f680ab750 0x7f5f680c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59638/0 (socket says 172.21.15.83:59638) 2024-09-19T07:41:31.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f677fe640 1 -- 172.21.15.83:0/1068803476 learned_addr learned my addr 172.21.15.83:0/1068803476 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:31.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f67fff640 1 --2- 172.21.15.83:0/1068803476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f680ab060 0x7f5f68152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:31.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f66ffd640 1 --2- 172.21.15.83:0/1068803476 >> v2:172.21.15.184:3300/0 conn(0x7f5f680c2530 0x7f5f681528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:31.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f677fe640 1 -- 172.21.15.83:0/1068803476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f680ab060 msgr2=0x7f5f68152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:31.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f677fe640 1 --2- 172.21.15.83:0/1068803476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f680ab060 0x7f5f68152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:31.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f677fe640 1 -- 172.21.15.83:0/1068803476 >> v2:172.21.15.184:3300/0 conn(0x7f5f680c2530 msgr2=0x7f5f681528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:31.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f677fe640 1 --2- 172.21.15.83:0/1068803476 >> v2:172.21.15.184:3300/0 conn(0x7f5f680c2530 0x7f5f681528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:31.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f677fe640 1 -- 172.21.15.83:0/1068803476 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6806e730 con 0x7f5f680ab750 2024-09-19T07:41:31.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f677fe640 1 --2- 172.21.15.83:0/1068803476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f680ab750 0x7f5f680c1ff0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f5f6806cd40 tx=0x7f5f5c01eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10cd9a699172f1e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:31.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f667fc640 1 -- 172.21.15.83:0/1068803476 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0b4070 con 0x7f5f680ab750 2024-09-19T07:41:31.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f667fc640 1 -- 172.21.15.83:0/1068803476 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f5c0a79f0 con 0x7f5f680ab750 2024-09-19T07:41:31.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.071+0000 7f5f667fc640 1 -- 172.21.15.83:0/1068803476 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0a7dc0 con 0x7f5f680ab750 2024-09-19T07:41:31.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1068803476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f680ab750 msgr2=0x7f5f680c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:31.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 --2- 172.21.15.83:0/1068803476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f680ab750 0x7f5f680c1ff0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f5f6806cd40 tx=0x7f5f5c01eb60 comp rx=0 tx=0).stop 2024-09-19T07:41:31.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1068803476 shutdown_connections 2024-09-19T07:41:31.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 --2- 172.21.15.83:0/1068803476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f680ab060 0x7f5f68152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:31.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 --2- 172.21.15.83:0/1068803476 >> v2:172.21.15.184:3300/0 conn(0x7f5f680c2530 0x7f5f681528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:31.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 --2- 172.21.15.83:0/1068803476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f680ab750 0x7f5f680c1ff0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:31.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1068803476 >> 172.21.15.83:0/1068803476 conn(0x7f5f6806bc80 msgr2=0x7f5f6806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:31.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1068803476 shutdown_connections 2024-09-19T07:41:31.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1068803476 wait complete. 2024-09-19T07:41:31.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 Processor -- start 2024-09-19T07:41:31.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 -- start start 2024-09-19T07:41:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f680ab060 0x7f5f6814f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f680ab750 0x7f5f680c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f680c2530 0x7f5f6814bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5f68155c20 con 0x7f5f680ab060 2024-09-19T07:41:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5f68155aa0 con 0x7f5f680c2530 2024-09-19T07:41:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5f68155da0 con 0x7f5f680ab750 2024-09-19T07:41:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f677fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f680ab060 0x7f5f6814f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f677fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f680ab060 0x7f5f6814f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59642/0 (socket says 172.21.15.83:59642) 2024-09-19T07:41:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f677fe640 1 -- 172.21.15.83:0/1248133647 learned_addr learned my addr 172.21.15.83:0/1248133647 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f66ffd640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f680ab750 0x7f5f680c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f67fff640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f680c2530 0x7f5f6814bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f677fe640 1 -- 172.21.15.83:0/1248133647 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f680ab750 msgr2=0x7f5f680c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f677fe640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f680ab750 0x7f5f680c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f677fe640 1 -- 172.21.15.83:0/1248133647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f680c2530 msgr2=0x7f5f6814bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f677fe640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f680c2530 0x7f5f6814bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f677fe640 1 -- 172.21.15.83:0/1248133647 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f680c6ff0 con 0x7f5f680ab060 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f677fe640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f680ab060 0x7f5f6814f9e0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c01d210 tx=0x7f5f5c01b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f4ffff640 1 -- 172.21.15.83:0/1248133647 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0b4040 con 0x7f5f680ab060 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f4ffff640 1 -- 172.21.15.83:0/1248133647 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f5c0af860 con 0x7f5f680ab060 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f4ffff640 1 -- 172.21.15.83:0/1248133647 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0afc50 con 0x7f5f680ab060 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1248133647 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6814e040 con 0x7f5f680ab060 2024-09-19T07:41:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.075+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1248133647 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f6814b3e0 con 0x7f5f680ab060 2024-09-19T07:41:31.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.079+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1248133647 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f680c1ff0 con 0x7f5f680ab060 2024-09-19T07:41:31.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.079+0000 7f5f4ffff640 1 -- 172.21.15.83:0/1248133647 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5f5c0b8430 con 0x7f5f680ab060 2024-09-19T07:41:31.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.079+0000 7f5f4ffff640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5f540678c0 0x7f5f54069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.079+0000 7f5f4ffff640 1 -- 172.21.15.83:0/1248133647 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9217+0+0 (secure 0 0 0) 0x7f5f5c1238e0 con 0x7f5f680ab060 2024-09-19T07:41:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.079+0000 7f5f66ffd640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5f540678c0 0x7f5f54069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.079+0000 7f5f66ffd640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5f540678c0 0x7f5f54069d80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5f580070b0 tx=0x7f5f58007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:31.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.083+0000 7f5f4ffff640 1 -- 172.21.15.83:0/1248133647 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5f5c0187e0 con 0x7f5f680ab060 2024-09-19T07:41:31.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.279+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1248133647 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f68073840 con 0x7f5f680ab060 2024-09-19T07:41:31.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f4ffff640 1 -- 172.21.15.83:0/1248133647 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) ==== 95+0+6546 (secure 0 0 0) 0x7f5f5c0f35b0 con 0x7f5f680ab060 2024-09-19T07:41:31.278 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:31.278 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":155,"btime":"2024-09-19T07:41:15:459617+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":5951,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:40:26.470349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5852":{"gid":5852,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":22,"addr":"172.21.15.184:6833/1335198090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1335198090},{"type":"v1","addr":"172.21.15.184:6833","nonce":1335198090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5852,"qdb_cluster":[5852]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:31.278 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 155 2024-09-19T07:41:31.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1248133647 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5f540678c0 msgr2=0x7f5f54069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:31.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f6dc88640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5f540678c0 0x7f5f54069d80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5f580070b0 tx=0x7f5f58007040 comp rx=0 tx=0).stop 2024-09-19T07:41:31.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1248133647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f680ab060 msgr2=0x7f5f6814f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:31.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f6dc88640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f680ab060 0x7f5f6814f9e0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c01d210 tx=0x7f5f5c01b900 comp rx=0 tx=0).stop 2024-09-19T07:41:31.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1248133647 shutdown_connections 2024-09-19T07:41:31.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f6dc88640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5f540678c0 0x7f5f54069d80 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:31.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f6dc88640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f680c2530 0x7f5f6814bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:31.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f6dc88640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f680ab750 0x7f5f680c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:31.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f6dc88640 1 --2- 172.21.15.83:0/1248133647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f680ab060 0x7f5f6814f9e0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:31.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1248133647 >> 172.21.15.83:0/1248133647 conn(0x7f5f6806bc80 msgr2=0x7f5f6806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:31.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1248133647 shutdown_connections 2024-09-19T07:41:31.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:31.283+0000 7f5f6dc88640 1 -- 172.21.15.83:0/1248133647 wait complete. 2024-09-19T07:41:31.297 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:41:33.298 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:33.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.415+0000 7f95f63d6640 1 Processor -- start 2024-09-19T07:41:33.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.415+0000 7f95f63d6640 1 -- start start 2024-09-19T07:41:33.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.415+0000 7f95f63d6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95f00ac290 0x7f95f00c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.415+0000 7f95f63d6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f95f00c3a50 0x7f95f01563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.415+0000 7f95f63d6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95f00ac6d0 con 0x7f95f00c3130 2024-09-19T07:41:33.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.415+0000 7f95f63d6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f95f006e110 con 0x7f95f00c3a50 2024-09-19T07:41:33.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.415+0000 7f95f63d6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f95f00c4990 con 0x7f95f00ac290 2024-09-19T07:41:33.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.419+0000 7f95f4973640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f95f00c3a50 0x7f95f01563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.419+0000 7f95f4973640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f95f00c3a50 0x7f95f01563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59650/0 (socket says 172.21.15.83:59650) 2024-09-19T07:41:33.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.419+0000 7f95f4973640 1 -- 172.21.15.83:0/2937268963 learned_addr learned my addr 172.21.15.83:0/2937268963 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:33.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.419+0000 7f95ef7fe640 1 --2- 172.21.15.83:0/2937268963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95f00ac290 0x7f95f00c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.419+0000 7f95f4973640 1 -- 172.21.15.83:0/2937268963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95f00ac290 msgr2=0x7f95f00c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:33.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.419+0000 7f95f4973640 1 --2- 172.21.15.83:0/2937268963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95f00ac290 0x7f95f00c3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.419+0000 7f95f4973640 1 -- 172.21.15.83:0/2937268963 >> v1:172.21.15.184:6789/0 conn(0x7f95f00c3130 legacy=0x7f95f006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.419+0000 7f95f4973640 1 -- 172.21.15.83:0/2937268963 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f00c4c70 con 0x7f95f00c3a50 2024-09-19T07:41:33.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.419+0000 7f95f4973640 1 --2- 172.21.15.83:0/2937268963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95f00c3a50 0x7f95f01563a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f95e800bf60 tx=0x7f95e8020af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d530e7b9a6ba534a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:33.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.419+0000 7f95eeffd640 1 -- 172.21.15.83:0/2937268963 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e800b2d0 con 0x7f95f00c3a50 2024-09-19T07:41:33.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.419+0000 7f95eeffd640 1 -- 172.21.15.83:0/2937268963 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e800b470 con 0x7f95f00c3a50 2024-09-19T07:41:33.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.419+0000 7f95eeffd640 1 -- 172.21.15.83:0/2937268963 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e80a9ad0 con 0x7f95f00c3a50 2024-09-19T07:41:33.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 -- 172.21.15.83:0/2937268963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95f00c3a50 msgr2=0x7f95f01563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 --2- 172.21.15.83:0/2937268963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95f00c3a50 0x7f95f01563a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f95e800bf60 tx=0x7f95e8020af0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 -- 172.21.15.83:0/2937268963 shutdown_connections 2024-09-19T07:41:33.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 --2- 172.21.15.83:0/2937268963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95f00c3a50 0x7f95f01563a0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 --2- 172.21.15.83:0/2937268963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95f00ac290 0x7f95f00c3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 -- 172.21.15.83:0/2937268963 >> 172.21.15.83:0/2937268963 conn(0x7f95f00c5060 msgr2=0x7f95f00c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:33.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 -- 172.21.15.83:0/2937268963 shutdown_connections 2024-09-19T07:41:33.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 -- 172.21.15.83:0/2937268963 wait complete. 2024-09-19T07:41:33.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 Processor -- start 2024-09-19T07:41:33.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 -- start start 2024-09-19T07:41:33.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95f00ac290 0x7f95f00735b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95f00c3130 0x7f95f014d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95f00c3a50 0x7f95f014f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f95f00ad520 con 0x7f95f00c3a50 2024-09-19T07:41:33.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f95f00c82a0 con 0x7f95f00ac290 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f95f00ad6a0 con 0x7f95f00c3130 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f4973640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95f00c3a50 0x7f95f014f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f4973640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95f00c3a50 0x7f95f014f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59664/0 (socket says 172.21.15.83:59664) 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f4973640 1 -- 172.21.15.83:0/1920430563 learned_addr learned my addr 172.21.15.83:0/1920430563 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95effff640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95f00ac290 0x7f95f00735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95ef7fe640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95f00c3130 0x7f95f014d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f4973640 1 -- 172.21.15.83:0/1920430563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95f00c3130 msgr2=0x7f95f014d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f4973640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95f00c3130 0x7f95f014d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f4973640 1 -- 172.21.15.83:0/1920430563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95f00ac290 msgr2=0x7f95f00735b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f4973640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95f00ac290 0x7f95f00735b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f4973640 1 -- 172.21.15.83:0/1920430563 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f0073b90 con 0x7f95f00c3a50 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95ef7fe640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95f00c3130 0x7f95f014d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95effff640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95f00ac290 0x7f95f00735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:41:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f4973640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95f00c3a50 0x7f95f014f190 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f95e80b3040 tx=0x7f95e801d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:33.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95ecff9640 1 -- 172.21.15.83:0/1920430563 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e80c0020 con 0x7f95f00c3a50 2024-09-19T07:41:33.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95ecff9640 1 -- 172.21.15.83:0/1920430563 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e8009800 con 0x7f95f00c3a50 2024-09-19T07:41:33.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95ecff9640 1 -- 172.21.15.83:0/1920430563 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e8009c10 con 0x7f95f00c3a50 2024-09-19T07:41:33.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 -- 172.21.15.83:0/1920430563 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95f01e9f40 con 0x7f95f00c3a50 2024-09-19T07:41:33.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.423+0000 7f95f63d6640 1 -- 172.21.15.83:0/1920430563 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95f01ea500 con 0x7f95f00c3a50 2024-09-19T07:41:33.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.427+0000 7f95f63d6640 1 -- 172.21.15.83:0/1920430563 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95f00c3510 con 0x7f95f00c3a50 2024-09-19T07:41:33.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.427+0000 7f95ecff9640 1 -- 172.21.15.83:0/1920430563 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f95e80c3020 con 0x7f95f00c3a50 2024-09-19T07:41:33.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.427+0000 7f95ecff9640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f95d00678c0 0x7f95d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.427+0000 7f95ecff9640 1 -- 172.21.15.83:0/1920430563 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9303+0+0 (secure 0 0 0) 0x7f95e8125c40 con 0x7f95f00c3a50 2024-09-19T07:41:33.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.427+0000 7f95effff640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f95d00678c0 0x7f95d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.427+0000 7f95effff640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f95d00678c0 0x7f95d0069d80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f95e0001670 tx=0x7f95e0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:33.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.431+0000 7f95ecff9640 1 -- 172.21.15.83:0/1920430563 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f95e80ca030 con 0x7f95f00c3a50 2024-09-19T07:41:33.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 -- 172.21.15.83:0/1920430563 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95f014d850 con 0x7f95f00c3a50 2024-09-19T07:41:33.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95ecff9640 1 -- 172.21.15.83:0/1920430563 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+5709 (secure 0 0 0) 0x7f95e801aa30 con 0x7f95f00c3a50 2024-09-19T07:41:33.651 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:33.652 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":156,"btime":"2024-09-19T07:41:33:246048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:41:33.246038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5951":{"gid":5951,"name":"a","rank":0,"incarnation":156,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:33.652 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 156 2024-09-19T07:41:33.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 -- 172.21.15.83:0/1920430563 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f95d00678c0 msgr2=0x7f95d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f95d00678c0 0x7f95d0069d80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f95e0001670 tx=0x7f95e0009040 comp rx=0 tx=0).stop 2024-09-19T07:41:33.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 -- 172.21.15.83:0/1920430563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95f00c3a50 msgr2=0x7f95f014f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95f00c3a50 0x7f95f014f190 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f95e80b3040 tx=0x7f95e801d890 comp rx=0 tx=0).stop 2024-09-19T07:41:33.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 -- 172.21.15.83:0/1920430563 shutdown_connections 2024-09-19T07:41:33.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f95d00678c0 0x7f95d0069d80 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95f00c3a50 0x7f95f014f190 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95f00c3130 0x7f95f014d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 --2- 172.21.15.83:0/1920430563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95f00ac290 0x7f95f00735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 -- 172.21.15.83:0/1920430563 >> 172.21.15.83:0/1920430563 conn(0x7f95f00c5060 msgr2=0x7f95f00c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:33.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 -- 172.21.15.83:0/1920430563 shutdown_connections 2024-09-19T07:41:33.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.655+0000 7f95f63d6640 1 -- 172.21.15.83:0/1920430563 wait complete. 2024-09-19T07:41:33.670 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-19T07:41:33.670 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:41:33.670 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:33.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.787+0000 7f3424c29640 1 Processor -- start 2024-09-19T07:41:33.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.787+0000 7f3424c29640 1 -- start start 2024-09-19T07:41:33.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f3424c29640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f34201545e0 0x7f342006e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f3424c29640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3420154a70 con 0x7f34200c28e0 2024-09-19T07:41:33.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f3424c29640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34200a9e60 con 0x7f3420071100 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f3424c29640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f3420071b50 con 0x7f34201545e0 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341dd9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f34200c28e0 0x7f342006dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51190/0 (socket says 172.21.15.83:51190) 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341dd9b640 1 -- 172.21.15.83:0/438661178 learned_addr learned my addr 172.21.15.83:0/438661178 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341ed9d640 1 --2- 172.21.15.83:0/438661178 >> v2:172.21.15.184:3300/0 conn(0x7f34201545e0 0x7f342006e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 529559744 0 0) 0x7f3420154a70 con 0x7f34200c28e0 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3408003530 con 0x7f34200c28e0 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3707926953 0 0) 0x7f3408003530 con 0x7f34200c28e0 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 >> v2:172.21.15.184:3300/0 conn(0x7f34201545e0 msgr2=0x7f342006e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 --2- 172.21.15.83:0/438661178 >> v2:172.21.15.184:3300/0 conn(0x7f34201545e0 0x7f342006e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 >> v1:172.21.15.184:6789/0 conn(0x7f3420071100 legacy=0x7f3420150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34200aa140 con 0x7f34200c28e0 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f3410001680 con 0x7f34200c28e0 2024-09-19T07:41:33.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 >> v1:172.21.15.83:6789/0 conn(0x7f34200c28e0 legacy=0x7f342006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 --2- 172.21.15.83:0/438661178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34080039c0 0x7f3408003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 --2- 172.21.15.83:0/438661178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3408004810 0x7f3408004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 --2- 172.21.15.83:0/438661178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 0x7f340800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3420071b50 con 0x7f34080051d0 2024-09-19T07:41:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f340800c170 con 0x7f3408004810 2024-09-19T07:41:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f340800c370 con 0x7f34080039c0 2024-09-19T07:41:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341ed9d640 1 --2- 172.21.15.83:0/438661178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 0x7f340800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341dd9b640 1 --2- 172.21.15.83:0/438661178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3408004810 0x7f3408004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341e59c640 1 --2- 172.21.15.83:0/438661178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34080039c0 0x7f3408003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341ed9d640 1 -- 172.21.15.83:0/438661178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34080039c0 msgr2=0x7f3408003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341ed9d640 1 --2- 172.21.15.83:0/438661178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34080039c0 0x7f3408003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341ed9d640 1 -- 172.21.15.83:0/438661178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3408004810 msgr2=0x7f3408004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:33.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341ed9d640 1 --2- 172.21.15.83:0/438661178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3408004810 0x7f3408004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341ed9d640 1 -- 172.21.15.83:0/438661178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3408003530 con 0x7f34080051d0 2024-09-19T07:41:33.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.791+0000 7f341dd9b640 1 --2- 172.21.15.83:0/438661178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3408004810 0x7f3408004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:41:33.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341ed9d640 1 -- 172.21.15.83:0/438661178 reap_dead start 2024-09-19T07:41:33.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341ed9d640 1 --2- 172.21.15.83:0/438661178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 0x7f340800b9d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f3418009fa0 tx=0x7f341801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55798c4467eaf793 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:33.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34180b2070 con 0x7f34080051d0 2024-09-19T07:41:33.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34180a5d50 con 0x7f34080051d0 2024-09-19T07:41:33.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341d59a640 1 -- 172.21.15.83:0/438661178 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3418009bf0 con 0x7f34080051d0 2024-09-19T07:41:33.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 -- 172.21.15.83:0/438661178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 msgr2=0x7f340800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 --2- 172.21.15.83:0/438661178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 0x7f340800b9d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f3418009fa0 tx=0x7f341801cc80 comp rx=0 tx=0).stop 2024-09-19T07:41:33.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 -- 172.21.15.83:0/438661178 shutdown_connections 2024-09-19T07:41:33.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 --2- 172.21.15.83:0/438661178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 0x7f340800b9d0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 -- 172.21.15.83:0/438661178 >> 172.21.15.83:0/438661178 conn(0x7f342006bc80 msgr2=0x7f342006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:33.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 -- 172.21.15.83:0/438661178 shutdown_connections 2024-09-19T07:41:33.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 -- 172.21.15.83:0/438661178 wait complete. 2024-09-19T07:41:33.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 Processor -- start 2024-09-19T07:41:33.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 -- start start 2024-09-19T07:41:33.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 0x7f34201ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3420157cc0 0x7f3420150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3420157770 0x7f34200aaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f34200c2520 con 0x7f34080051d0 2024-09-19T07:41:33.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f34200c1ff0 con 0x7f3420157770 2024-09-19T07:41:33.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f34200714e0 con 0x7f3420157cc0 2024-09-19T07:41:33.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 0x7f34201ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 0x7f34201ee7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59694/0 (socket says 172.21.15.83:59694) 2024-09-19T07:41:33.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341e59c640 1 -- 172.21.15.83:0/2488221256 learned_addr learned my addr 172.21.15.83:0/2488221256 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:33.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341ed9d640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3420157770 0x7f34200aaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341dd9b640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3420157cc0 0x7f3420150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341e59c640 1 -- 172.21.15.83:0/2488221256 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3420157cc0 msgr2=0x7f3420150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341e59c640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3420157cc0 0x7f3420150450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341e59c640 1 -- 172.21.15.83:0/2488221256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3420157770 msgr2=0x7f34200aaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341e59c640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3420157770 0x7f34200aaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341e59c640 1 -- 172.21.15.83:0/2488221256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3420153240 con 0x7f34080051d0 2024-09-19T07:41:33.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f341e59c640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 0x7f34201ee7c0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f341400d4d0 tx=0x7f341400d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:33.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f340effd640 1 -- 172.21.15.83:0/2488221256 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414005480 con 0x7f34080051d0 2024-09-19T07:41:33.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f340effd640 1 -- 172.21.15.83:0/2488221256 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3414017070 con 0x7f34080051d0 2024-09-19T07:41:33.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 -- 172.21.15.83:0/2488221256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3420153520 con 0x7f34080051d0 2024-09-19T07:41:33.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f340effd640 1 -- 172.21.15.83:0/2488221256 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414003820 con 0x7f34080051d0 2024-09-19T07:41:33.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.795+0000 7f3424c29640 1 -- 172.21.15.83:0/2488221256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3420150990 con 0x7f34080051d0 2024-09-19T07:41:33.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.799+0000 7f3424c29640 1 -- 172.21.15.83:0/2488221256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3420150f40 con 0x7f34080051d0 2024-09-19T07:41:33.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.799+0000 7f340effd640 1 -- 172.21.15.83:0/2488221256 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f34140039c0 con 0x7f34080051d0 2024-09-19T07:41:33.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.799+0000 7f340effd640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f34000678c0 0x7f3400069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:33.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.799+0000 7f340effd640 1 -- 172.21.15.83:0/2488221256 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9303+0+0 (secure 0 0 0) 0x7f3414086740 con 0x7f34080051d0 2024-09-19T07:41:33.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.799+0000 7f341dd9b640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f34000678c0 0x7f3400069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:33.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.799+0000 7f341dd9b640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f34000678c0 0x7f3400069d80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f34100017f0 tx=0x7f3410001160 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:33.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.803+0000 7f340effd640 1 -- 172.21.15.83:0/2488221256 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f34140564b0 con 0x7f34080051d0 2024-09-19T07:41:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 -- 172.21.15.83:0/2488221256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34200c4f30 con 0x7f34080051d0 2024-09-19T07:41:33.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f340effd640 1 -- 172.21.15.83:0/2488221256 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+5709 (secure 0 0 0) 0x7f3414055db0 con 0x7f34080051d0 2024-09-19T07:41:33.995 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:33.996 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":156,"btime":"2024-09-19T07:41:33:246048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:41:33.246038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5951":{"gid":5951,"name":"a","rank":0,"incarnation":156,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:33.996 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 156 2024-09-19T07:41:33.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 -- 172.21.15.83:0/2488221256 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f34000678c0 msgr2=0x7f3400069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f34000678c0 0x7f3400069d80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f34100017f0 tx=0x7f3410001160 comp rx=0 tx=0).stop 2024-09-19T07:41:33.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 -- 172.21.15.83:0/2488221256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 msgr2=0x7f34201ee7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:33.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 0x7f34201ee7c0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f341400d4d0 tx=0x7f341400d990 comp rx=0 tx=0).stop 2024-09-19T07:41:33.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 -- 172.21.15.83:0/2488221256 shutdown_connections 2024-09-19T07:41:33.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f34000678c0 0x7f3400069d80 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3420157770 0x7f34200aaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3420157cc0 0x7f3420150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 --2- 172.21.15.83:0/2488221256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34080051d0 0x7f34201ee7c0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:33.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 -- 172.21.15.83:0/2488221256 >> 172.21.15.83:0/2488221256 conn(0x7f342006bc80 msgr2=0x7f342014dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:33.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 -- 172.21.15.83:0/2488221256 shutdown_connections 2024-09-19T07:41:33.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:33.999+0000 7f3424c29640 1 -- 172.21.15.83:0/2488221256 wait complete. 2024-09-19T07:41:34.015 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:41:33:246048+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 156, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 156, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5951": { "addr": "172.21.15.83:6835/2776269095", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 2776269095, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 2776269095, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5951, "incarnation": 156, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:41:33.246038+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5951 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7503": { "addr": "172.21.15.83:6837/1990119490", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 1990119490, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 1990119490, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7503, "incarnation": 151, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:41:11.437309+0000", "qdb_cluster": [ 7503 ], "qdb_leader": 7503, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7503 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:41:36.016 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:36.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb98bd640 1 Processor -- start 2024-09-19T07:41:36.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb98bd640 1 -- start start 2024-09-19T07:41:36.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb98bd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb414dbe0 0x7fdeb40c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:36.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb98bd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeb4073d90 con 0x7fdeb40c3ab0 2024-09-19T07:41:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb98bd640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeb4153b30 con 0x7fdeb40c44e0 2024-09-19T07:41:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb98bd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb4154700 con 0x7fdeb414dbe0 2024-09-19T07:41:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb27fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdeb40c3ab0 0x7fdeb4157010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51200/0 (socket says 172.21.15.83:51200) 2024-09-19T07:41:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb27fc640 1 -- 172.21.15.83:0/212396084 learned_addr learned my addr 172.21.15.83:0/212396084 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:41:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb2ffd640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb414dbe0 0x7fdeb40c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3777645296 0 0) 0x7fdeb4073d90 con 0x7fdeb40c3ab0 2024-09-19T07:41:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde9c003530 con 0x7fdeb40c3ab0 2024-09-19T07:41:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2442739224 0 0) 0x7fde9c003530 con 0x7fdeb40c3ab0 2024-09-19T07:41:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb414dbe0 msgr2=0x7fdeb40c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb414dbe0 0x7fdeb40c3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 >> v1:172.21.15.184:6789/0 conn(0x7fdeb40c44e0 legacy=0x7fdeb4157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb4153dd0 con 0x7fdeb40c3ab0 2024-09-19T07:41:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fdea8001680 con 0x7fdeb40c3ab0 2024-09-19T07:41:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 >> v1:172.21.15.83:6789/0 conn(0x7fdeb40c3ab0 legacy=0x7fdeb4157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde9c0039f0 0x7fde9c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb414dbe0 0x7fde9c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde9c004da0 0x7fde9c005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb4154700 con 0x7fde9c004da0 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c00bc50 con 0x7fde9c0039f0 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fde9c00be50 con 0x7fdeb414dbe0 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb37fe640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde9c004da0 0x7fde9c005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb2ffd640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde9c0039f0 0x7fde9c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb27fc640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb414dbe0 0x7fde9c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb37fe640 1 -- 172.21.15.83:0/212396084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb414dbe0 msgr2=0x7fde9c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb37fe640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb414dbe0 0x7fde9c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb37fe640 1 -- 172.21.15.83:0/212396084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde9c0039f0 msgr2=0x7fde9c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb37fe640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde9c0039f0 0x7fde9c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:36.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb37fe640 1 -- 172.21.15.83:0/212396084 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9c003530 con 0x7fde9c004da0 2024-09-19T07:41:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.147+0000 7fdeb2ffd640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde9c0039f0 0x7fde9c003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:41:36.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb37fe640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde9c004da0 0x7fde9c005220 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fdeac001e40 tx=0x7fdeac01c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438f1863c0d94328 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:36.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac001a90 con 0x7fde9c004da0 2024-09-19T07:41:36.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeac003b80 con 0x7fde9c004da0 2024-09-19T07:41:36.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb1ffb640 1 -- 172.21.15.83:0/212396084 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac005080 con 0x7fde9c004da0 2024-09-19T07:41:36.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb98bd640 1 -- 172.21.15.83:0/212396084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde9c004da0 msgr2=0x7fde9c005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:36.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb98bd640 1 --2- 172.21.15.83:0/212396084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde9c004da0 0x7fde9c005220 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fdeac001e40 tx=0x7fdeac01c740 comp rx=0 tx=0).stop 2024-09-19T07:41:36.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb37fe640 1 -- 172.21.15.83:0/212396084 reap_dead start 2024-09-19T07:41:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb98bd640 1 -- 172.21.15.83:0/212396084 shutdown_connections 2024-09-19T07:41:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb98bd640 1 -- 172.21.15.83:0/212396084 >> 172.21.15.83:0/212396084 conn(0x7fdeb40c95a0 msgr2=0x7fdeb40c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb98bd640 1 -- 172.21.15.83:0/212396084 shutdown_connections 2024-09-19T07:41:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb98bd640 1 -- 172.21.15.83:0/212396084 wait complete. 2024-09-19T07:41:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb98bd640 1 Processor -- start 2024-09-19T07:41:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.151+0000 7fdeb98bd640 1 -- start start 2024-09-19T07:41:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb98bd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb40c8950 0x7fdeb41edb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb98bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb414e5a0 0x7fdeb4148600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb98bd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb4148b40 0x7fdeb41468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb98bd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb415fd40 con 0x7fdeb414e5a0 2024-09-19T07:41:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb98bd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb415fbc0 con 0x7fdeb40c8950 2024-09-19T07:41:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb98bd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb415fec0 con 0x7fdeb4148b40 2024-09-19T07:41:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb414e5a0 0x7fdeb4148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb414e5a0 0x7fdeb4148600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59722/0 (socket says 172.21.15.83:59722) 2024-09-19T07:41:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb27fc640 1 -- 172.21.15.83:0/1419302351 learned_addr learned my addr 172.21.15.83:0/1419302351 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb2ffd640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb40c8950 0x7fdeb41edb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb27fc640 1 -- 172.21.15.83:0/1419302351 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb4148b40 msgr2=0x7fdeb41468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb27fc640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb4148b40 0x7fdeb41468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb27fc640 1 -- 172.21.15.83:0/1419302351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb40c8950 msgr2=0x7fdeb41edb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb27fc640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb40c8950 0x7fdeb41edb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb27fc640 1 -- 172.21.15.83:0/1419302351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb4149be0 con 0x7fdeb414e5a0 2024-09-19T07:41:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb27fc640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb414e5a0 0x7fdeb4148600 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fdea800e700 tx=0x7fdea800ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fde9bfff640 1 -- 172.21.15.83:0/1419302351 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea800c6c0 con 0x7fdeb414e5a0 2024-09-19T07:41:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fde9bfff640 1 -- 172.21.15.83:0/1419302351 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdea8018070 con 0x7fdeb414e5a0 2024-09-19T07:41:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fde9bfff640 1 -- 172.21.15.83:0/1419302351 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea800caa0 con 0x7fdeb414e5a0 2024-09-19T07:41:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb98bd640 1 -- 172.21.15.83:0/1419302351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb4149ec0 con 0x7fdeb414e5a0 2024-09-19T07:41:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb98bd640 1 -- 172.21.15.83:0/1419302351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb41480e0 con 0x7fdeb414e5a0 2024-09-19T07:41:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb98bd640 1 -- 172.21.15.83:0/1419302351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeb41482e0 con 0x7fdeb414e5a0 2024-09-19T07:41:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fde9bfff640 1 -- 172.21.15.83:0/1419302351 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdea800cc40 con 0x7fdeb414e5a0 2024-09-19T07:41:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fde9bfff640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea00678f0 0x7fdea0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fdeb2ffd640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea00678f0 0x7fdea0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.155+0000 7fde9bfff640 1 -- 172.21.15.83:0/1419302351 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9303+0+0 (secure 0 0 0) 0x7fdea80864e0 con 0x7fdeb414e5a0 2024-09-19T07:41:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.159+0000 7fdeb2ffd640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea00678f0 0x7fdea0069db0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4151540 tx=0x7fdea4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:36.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.163+0000 7fde9bfff640 1 -- 172.21.15.83:0/1419302351 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdeb41482e0 con 0x7fdeb414e5a0 2024-09-19T07:41:36.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.363+0000 7fdeb98bd640 1 -- 172.21.15.83:0/1419302351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdeb4157010 con 0x7fdeb414e5a0 2024-09-19T07:41:36.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.363+0000 7fde9bfff640 1 -- 172.21.15.83:0/1419302351 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) ==== 95+0+5709 (secure 0 0 0) 0x7fdea8056250 con 0x7fdeb414e5a0 2024-09-19T07:41:36.362 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:36.362 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":158,"btime":"2024-09-19T07:41:35:466842+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:41:34.475587+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5951":{"gid":5951,"name":"a","rank":0,"incarnation":156,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:36.362 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 158 2024-09-19T07:41:36.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.367+0000 7fdeb98bd640 1 -- 172.21.15.83:0/1419302351 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea00678f0 msgr2=0x7fdea0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:36.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.367+0000 7fdeb98bd640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea00678f0 0x7fdea0069db0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4151540 tx=0x7fdea4008040 comp rx=0 tx=0).stop 2024-09-19T07:41:36.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.367+0000 7fdeb98bd640 1 -- 172.21.15.83:0/1419302351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb414e5a0 msgr2=0x7fdeb4148600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:36.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.367+0000 7fdeb98bd640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb414e5a0 0x7fdeb4148600 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fdea800e700 tx=0x7fdea800ebc0 comp rx=0 tx=0).stop 2024-09-19T07:41:36.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.367+0000 7fdeb98bd640 1 -- 172.21.15.83:0/1419302351 shutdown_connections 2024-09-19T07:41:36.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.367+0000 7fdeb98bd640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdea00678f0 0x7fdea0069db0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:36.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.367+0000 7fdeb98bd640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdeb4148b40 0x7fdeb41468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:36.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.367+0000 7fdeb98bd640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb414e5a0 0x7fdeb4148600 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:36.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.367+0000 7fdeb98bd640 1 --2- 172.21.15.83:0/1419302351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb40c8950 0x7fdeb41edb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:36.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.367+0000 7fdeb98bd640 1 -- 172.21.15.83:0/1419302351 >> 172.21.15.83:0/1419302351 conn(0x7fdeb40c95a0 msgr2=0x7fdeb4157220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:36.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.367+0000 7fdeb98bd640 1 -- 172.21.15.83:0/1419302351 shutdown_connections 2024-09-19T07:41:36.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:36.367+0000 7fdeb98bd640 1 -- 172.21.15.83:0/1419302351 wait complete. 2024-09-19T07:41:36.382 INFO:tasks.mds_thrash.fs.[a]:no change 2024-09-19T07:41:36.464 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:41:36.467 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:41:38.383 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:38.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af952a640 1 Processor -- start 2024-09-19T07:41:38.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af952a640 1 -- start start 2024-09-19T07:41:38.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af952a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9aec03f1c0 0x7f9aec0d68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:38.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af952a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aec0da270 con 0x7f9aec049a30 2024-09-19T07:41:38.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af952a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aec046550 con 0x7f9aec0d3800 2024-09-19T07:41:38.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af8d29640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9aec03f1c0 0x7f9aec0d68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:38.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af3fff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f9aec0d3800 0x7f9aec0076a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:56692/0 (socket says 172.21.15.83:56692) 2024-09-19T07:41:38.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af3fff640 1 -- 172.21.15.83:0/3812385305 learned_addr learned my addr 172.21.15.83:0/3812385305 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:41:38.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af952a640 1 -- 172.21.15.83:0/3812385305 --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f9aec046830 con 0x7f9aec03f1c0 2024-09-19T07:41:38.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af2ffd640 1 -- 172.21.15.83:0/3812385305 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3015233438 0 0) 0x7f9aec0da270 con 0x7f9aec049a30 2024-09-19T07:41:38.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af2ffd640 1 -- 172.21.15.83:0/3812385305 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ae4003530 con 0x7f9aec049a30 2024-09-19T07:41:38.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af2ffd640 1 -- 172.21.15.83:0/3812385305 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2824370803 0 0) 0x7f9aec046550 con 0x7f9aec0d3800 2024-09-19T07:41:38.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af2ffd640 1 -- 172.21.15.83:0/3812385305 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9aec0da270 con 0x7f9aec0d3800 2024-09-19T07:41:38.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af2ffd640 1 -- 172.21.15.83:0/3812385305 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 555380413 0 0) 0x7f9ae4003530 con 0x7f9aec049a30 2024-09-19T07:41:38.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af8d29640 1 -- 172.21.15.83:0/3812385305 >> v1:172.21.15.184:6789/0 conn(0x7f9aec0d3800 legacy=0x7f9aec0076a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:38.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af8d29640 1 -- 172.21.15.83:0/3812385305 >> v1:172.21.15.83:6789/0 conn(0x7f9aec049a30 legacy=0x7f9aec0d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:38.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.571+0000 7f9af8d29640 1 -- 172.21.15.83:0/3812385305 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aec0d6e20 con 0x7f9aec03f1c0 2024-09-19T07:41:38.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.579+0000 7f9af8d29640 1 --2- 172.21.15.83:0/3812385305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec03f1c0 0x7f9aec0d68e0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f9af406a530 tx=0x7f9af4098e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a13d41d0f4779f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:38.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.579+0000 7f9af2ffd640 1 -- 172.21.15.83:0/3812385305 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af412e070 con 0x7f9aec03f1c0 2024-09-19T07:41:38.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.579+0000 7f9af2ffd640 1 -- 172.21.15.83:0/3812385305 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9af4121870 con 0x7f9aec03f1c0 2024-09-19T07:41:38.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.579+0000 7f9af2ffd640 1 -- 172.21.15.83:0/3812385305 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae8001680 con 0x7f9aec03f1c0 2024-09-19T07:41:38.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.579+0000 7f9af952a640 1 -- 172.21.15.83:0/3812385305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec03f1c0 msgr2=0x7f9aec0d68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:38.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.579+0000 7f9af952a640 1 --2- 172.21.15.83:0/3812385305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec03f1c0 0x7f9aec0d68e0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f9af406a530 tx=0x7f9af4098e00 comp rx=0 tx=0).stop 2024-09-19T07:41:38.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.579+0000 7f9af952a640 1 -- 172.21.15.83:0/3812385305 shutdown_connections 2024-09-19T07:41:38.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.579+0000 7f9af952a640 1 --2- 172.21.15.83:0/3812385305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec03f1c0 0x7f9aec0d68e0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:38.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.579+0000 7f9af952a640 1 -- 172.21.15.83:0/3812385305 >> 172.21.15.83:0/3812385305 conn(0x7f9aec0485b0 msgr2=0x7f9aec0489c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:38.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.579+0000 7f9af952a640 1 -- 172.21.15.83:0/3812385305 shutdown_connections 2024-09-19T07:41:38.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.579+0000 7f9af952a640 1 -- 172.21.15.83:0/3812385305 wait complete. 2024-09-19T07:41:38.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af952a640 1 Processor -- start 2024-09-19T07:41:38.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af952a640 1 -- start start 2024-09-19T07:41:38.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af952a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9aec03f1c0 0x7f9aec044fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:38.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af952a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec049a30 0x7f9aec0d1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:38.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af952a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9aec0d3800 0x7f9aec0c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:38.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af952a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9aec00bcd0 con 0x7f9aec049a30 2024-09-19T07:41:38.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af952a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9aec00bb50 con 0x7f9aec03f1c0 2024-09-19T07:41:38.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af952a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9aec0e3a70 con 0x7f9aec0d3800 2024-09-19T07:41:38.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9aec03f1c0 0x7f9aec044fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:38.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9aec03f1c0 0x7f9aec044fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:60604/0 (socket says 172.21.15.83:60604) 2024-09-19T07:41:38.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af3fff640 1 -- 172.21.15.83:0/1393440199 learned_addr learned my addr 172.21.15.83:0/1393440199 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:38.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af3fff640 1 -- 172.21.15.83:0/1393440199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9aec0d3800 msgr2=0x7f9aec0c98d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:41:38.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af3fff640 1 --2- 172.21.15.83:0/1393440199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9aec0d3800 0x7f9aec0c98d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:38.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af3fff640 1 -- 172.21.15.83:0/1393440199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec049a30 msgr2=0x7f9aec0d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:38.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af3fff640 1 --2- 172.21.15.83:0/1393440199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec049a30 0x7f9aec0d1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:38.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af3fff640 1 -- 172.21.15.83:0/1393440199 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aec0454f0 con 0x7f9aec03f1c0 2024-09-19T07:41:38.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af3fff640 1 --2- 172.21.15.83:0/1393440199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9aec03f1c0 0x7f9aec044fb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0001270 tx=0x7f9ae000ca50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:38.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af0ff9640 1 -- 172.21.15.83:0/1393440199 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae0028070 con 0x7f9aec03f1c0 2024-09-19T07:41:38.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af0ff9640 1 -- 172.21.15.83:0/1393440199 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ae0007280 con 0x7f9aec03f1c0 2024-09-19T07:41:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af0ff9640 1 -- 172.21.15.83:0/1393440199 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae00057b0 con 0x7f9aec03f1c0 2024-09-19T07:41:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af952a640 1 -- 172.21.15.83:0/1393440199 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aec0d74c0 con 0x7f9aec03f1c0 2024-09-19T07:41:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.583+0000 7f9af952a640 1 -- 172.21.15.83:0/1393440199 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aec0d7a80 con 0x7f9aec03f1c0 2024-09-19T07:41:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.587+0000 7f9af952a640 1 -- 172.21.15.83:0/1393440199 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ad40052a0 con 0x7f9aec03f1c0 2024-09-19T07:41:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.587+0000 7f9af0ff9640 1 -- 172.21.15.83:0/1393440199 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9ae002b070 con 0x7f9aec03f1c0 2024-09-19T07:41:38.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.587+0000 7f9af0ff9640 1 --2- 172.21.15.83:0/1393440199 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9ad80679c0 0x7f9ad8069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:38.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.587+0000 7f9af37fe640 1 --2- 172.21.15.83:0/1393440199 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9ad80679c0 0x7f9ad8069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:38.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.587+0000 7f9af37fe640 1 --2- 172.21.15.83:0/1393440199 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9ad80679c0 0x7f9ad8069e80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9ae80011f0 tx=0x7f9ae80010b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:38.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.587+0000 7f9af0ff9640 1 -- 172.21.15.83:0/1393440199 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9303+0+0 (secure 0 0 0) 0x7f9ae0096bb0 con 0x7f9aec03f1c0 2024-09-19T07:41:38.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:38.595+0000 7f9af0ff9640 1 -- 172.21.15.83:0/1393440199 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9aec0454f0 con 0x7f9aec03f1c0 2024-09-19T07:41:39.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.047+0000 7f9af952a640 1 -- 172.21.15.83:0/1393440199 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ad4005590 con 0x7f9aec03f1c0 2024-09-19T07:41:39.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.047+0000 7f9af0ff9640 1 -- 172.21.15.83:0/1393440199 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) ==== 95+0+5716 (secure 0 0 0) 0x7f9ae00662f0 con 0x7f9aec03f1c0 2024-09-19T07:41:39.046 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:39.046 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":159,"btime":"2024-09-19T07:41:36:478581+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:41:36.478574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5951":{"gid":5951,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5951,"qdb_cluster":[5951]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:39.046 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 159 2024-09-19T07:41:39.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.051+0000 7f9af952a640 1 -- 172.21.15.83:0/1393440199 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9ad80679c0 msgr2=0x7f9ad8069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:39.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.051+0000 7f9af952a640 1 --2- 172.21.15.83:0/1393440199 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9ad80679c0 0x7f9ad8069e80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9ae80011f0 tx=0x7f9ae80010b0 comp rx=0 tx=0).stop 2024-09-19T07:41:39.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.051+0000 7f9af952a640 1 -- 172.21.15.83:0/1393440199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9aec03f1c0 msgr2=0x7f9aec044fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:39.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.051+0000 7f9af952a640 1 --2- 172.21.15.83:0/1393440199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9aec03f1c0 0x7f9aec044fb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0001270 tx=0x7f9ae000ca50 comp rx=0 tx=0).stop 2024-09-19T07:41:39.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.051+0000 7f9af952a640 1 -- 172.21.15.83:0/1393440199 shutdown_connections 2024-09-19T07:41:39.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.051+0000 7f9af952a640 1 --2- 172.21.15.83:0/1393440199 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9ad80679c0 0x7f9ad8069e80 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:39.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.051+0000 7f9af952a640 1 --2- 172.21.15.83:0/1393440199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9aec0d3800 0x7f9aec0c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:39.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.051+0000 7f9af952a640 1 --2- 172.21.15.83:0/1393440199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9aec049a30 0x7f9aec0d1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:39.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.051+0000 7f9af952a640 1 --2- 172.21.15.83:0/1393440199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9aec03f1c0 0x7f9aec044fb0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:39.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.051+0000 7f9af952a640 1 -- 172.21.15.83:0/1393440199 >> 172.21.15.83:0/1393440199 conn(0x7f9aec0485b0 msgr2=0x7f9aec00a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:39.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.051+0000 7f9af952a640 1 -- 172.21.15.83:0/1393440199 shutdown_connections 2024-09-19T07:41:39.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:39.051+0000 7f9af952a640 1 -- 172.21.15.83:0/1393440199 wait complete. 2024-09-19T07:41:39.085 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=5852 2024-09-19T07:41:39.085 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.d 2024-09-19T07:41:40.492 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:41:40.495+0000 7f2b96f87640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:41:46.086 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-19T07:41:46.087 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-19T07:41:46.087 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-19T07:41:46.089 INFO:tasks.ceph.mds.d:Started 2024-09-19T07:41:46.089 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:46.171 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-19T07:41:46.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4eceb640 1 Processor -- start 2024-09-19T07:41:46.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4eceb640 1 -- start start 2024-09-19T07:41:46.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4eceb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e480c2b00 0x7f3e48153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:46.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4eceb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3e480c5b70 0x7f3e48153590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:46.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4eceb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e4806f630 0x7f3e481592a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:46.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4eceb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3e4806f100 con 0x7f3e4806f630 2024-09-19T07:41:46.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4eceb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f3e4806f350 con 0x7f3e480c5b70 2024-09-19T07:41:46.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4eceb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3e480abb10 con 0x7f3e480c2b00 2024-09-19T07:41:46.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4dce9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e480c2b00 0x7f3e48153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:46.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4d4e8640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3e480c5b70 0x7f3e48153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:46.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4dce9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e480c2b00 0x7f3e48153050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:41300/0 (socket says 172.21.15.83:41300) 2024-09-19T07:41:46.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4dce9640 1 -- 172.21.15.83:0/4184312028 learned_addr learned my addr 172.21.15.83:0/4184312028 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:46.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4dce9640 1 -- 172.21.15.83:0/4184312028 >> v2:172.21.15.184:3300/0 conn(0x7f3e480c5b70 msgr2=0x7f3e48153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:46.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4dce9640 1 --2- 172.21.15.83:0/4184312028 >> v2:172.21.15.184:3300/0 conn(0x7f3e480c5b70 0x7f3e48153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:46.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4dce9640 1 -- 172.21.15.83:0/4184312028 >> v2:172.21.15.83:3300/0 conn(0x7f3e4806f630 msgr2=0x7f3e481592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:46.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4dce9640 1 --2- 172.21.15.83:0/4184312028 >> v2:172.21.15.83:3300/0 conn(0x7f3e4806f630 0x7f3e481592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:46.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.347+0000 7f3e4dce9640 1 -- 172.21.15.83:0/4184312028 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e480ab7c0 con 0x7f3e480c2b00 2024-09-19T07:41:46.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.351+0000 7f3e4dce9640 1 --2- 172.21.15.83:0/4184312028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e480c2b00 0x7f3e48153050 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3e480ae870 tx=0x7f3e4401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a573fc642dc6625 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:46.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.355+0000 7f3e4cce7640 1 -- 172.21.15.83:0/4184312028 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e440b0070 con 0x7f3e480c2b00 2024-09-19T07:41:46.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.355+0000 7f3e4cce7640 1 -- 172.21.15.83:0/4184312028 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e440ab600 con 0x7f3e480c2b00 2024-09-19T07:41:46.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.355+0000 7f3e4cce7640 1 -- 172.21.15.83:0/4184312028 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e440ab9d0 con 0x7f3e480c2b00 2024-09-19T07:41:46.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.355+0000 7f3e4eceb640 1 -- 172.21.15.83:0/4184312028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e480c2b00 msgr2=0x7f3e48153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:46.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.355+0000 7f3e4eceb640 1 --2- 172.21.15.83:0/4184312028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e480c2b00 0x7f3e48153050 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3e480ae870 tx=0x7f3e4401c540 comp rx=0 tx=0).stop 2024-09-19T07:41:46.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.359+0000 7f3e4eceb640 1 -- 172.21.15.83:0/4184312028 shutdown_connections 2024-09-19T07:41:46.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.359+0000 7f3e4eceb640 1 --2- 172.21.15.83:0/4184312028 >> v2:172.21.15.83:3300/0 conn(0x7f3e4806f630 0x7f3e481592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:46.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.359+0000 7f3e4eceb640 1 --2- 172.21.15.83:0/4184312028 >> v2:172.21.15.184:3300/0 conn(0x7f3e480c5b70 0x7f3e48153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:46.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.359+0000 7f3e4eceb640 1 --2- 172.21.15.83:0/4184312028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e480c2b00 0x7f3e48153050 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:46.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.359+0000 7f3e4eceb640 1 -- 172.21.15.83:0/4184312028 >> 172.21.15.83:0/4184312028 conn(0x7f3e480c9470 msgr2=0x7f3e480c9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:46.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4eceb640 1 -- 172.21.15.83:0/4184312028 shutdown_connections 2024-09-19T07:41:46.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4eceb640 1 -- 172.21.15.83:0/4184312028 wait complete. 2024-09-19T07:41:46.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4eceb640 1 Processor -- start 2024-09-19T07:41:46.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4eceb640 1 -- start start 2024-09-19T07:41:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4eceb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4806f630 0x7f3e4806dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4dce9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4806f630 0x7f3e4806dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4eceb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e480c2b00 0x7f3e480c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4dce9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4806f630 0x7f3e4806dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40538/0 (socket says 172.21.15.83:40538) 2024-09-19T07:41:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4dce9640 1 -- 172.21.15.83:0/3800690329 learned_addr learned my addr 172.21.15.83:0/3800690329 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4eceb640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e480c5b70 0x7f3e481f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4eceb640 1 -- 172.21.15.83:0/3800690329 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3e480c30f0 con 0x7f3e4806f630 2024-09-19T07:41:46.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4eceb640 1 -- 172.21.15.83:0/3800690329 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3e48073cb0 con 0x7f3e480c5b70 2024-09-19T07:41:46.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4eceb640 1 -- 172.21.15.83:0/3800690329 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3e480ad5c0 con 0x7f3e480c2b00 2024-09-19T07:41:46.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4d4e8640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e480c2b00 0x7f3e480c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:46.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4dce9640 1 -- 172.21.15.83:0/3800690329 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e480c2b00 msgr2=0x7f3e480c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:46.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4dce9640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e480c2b00 0x7f3e480c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:46.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4dce9640 1 -- 172.21.15.83:0/3800690329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e480c5b70 msgr2=0x7f3e481f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:46.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4dce9640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e480c5b70 0x7f3e481f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:46.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.363+0000 7f3e4dce9640 1 -- 172.21.15.83:0/3800690329 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e481f6560 con 0x7f3e4806f630 2024-09-19T07:41:46.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.367+0000 7f3e4dce9640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4806f630 0x7f3e4806dcb0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f3e4401a190 tx=0x7f3e44019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:46.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.367+0000 7f3e3e7fc640 1 -- 172.21.15.83:0/3800690329 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e440b0050 con 0x7f3e4806f630 2024-09-19T07:41:46.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.367+0000 7f3e3e7fc640 1 -- 172.21.15.83:0/3800690329 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e440ba070 con 0x7f3e4806f630 2024-09-19T07:41:46.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.367+0000 7f3e3e7fc640 1 -- 172.21.15.83:0/3800690329 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e440b49f0 con 0x7f3e4806f630 2024-09-19T07:41:46.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.367+0000 7f3e4eceb640 1 -- 172.21.15.83:0/3800690329 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e481f6840 con 0x7f3e4806f630 2024-09-19T07:41:46.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.367+0000 7f3e4eceb640 1 -- 172.21.15.83:0/3800690329 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e481e9fd0 con 0x7f3e4806f630 2024-09-19T07:41:46.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.367+0000 7f3e2bfff640 1 -- 172.21.15.83:0/3800690329 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e4806e960 con 0x7f3e4806f630 2024-09-19T07:41:46.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.371+0000 7f3e3e7fc640 1 -- 172.21.15.83:0/3800690329 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3e440b4b90 con 0x7f3e4806f630 2024-09-19T07:41:46.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.371+0000 7f3e3e7fc640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e2c0678c0 0x7f3e2c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:46.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.375+0000 7f3e3e7fc640 1 -- 172.21.15.83:0/3800690329 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9303+0+0 (secure 0 0 0) 0x7f3e44121520 con 0x7f3e4806f630 2024-09-19T07:41:46.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.375+0000 7f3e4d4e8640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e2c0678c0 0x7f3e2c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:46.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.379+0000 7f3e4d4e8640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e2c0678c0 0x7f3e2c069d80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3e380079c0 tx=0x7f3e38008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:46.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.379+0000 7f3e3e7fc640 1 -- 172.21.15.83:0/3800690329 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3e44016150 con 0x7f3e4806f630 2024-09-19T07:41:46.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.883+0000 7f3e2bfff640 1 -- 172.21.15.83:0/3800690329 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e480707e0 con 0x7f3e4806f630 2024-09-19T07:41:46.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.883+0000 7f3e3e7fc640 1 -- 172.21.15.83:0/3800690329 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+6545 (secure 0 0 0) 0x7f3e440f1310 con 0x7f3e4806f630 2024-09-19T07:41:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:46.881 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":160,"btime":"2024-09-19T07:41:46:425682+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:41:36.478574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5951":{"gid":5951,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5951,"qdb_cluster":[5951]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:46.881 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 160 2024-09-19T07:41:46.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.887+0000 7f3e2bfff640 1 -- 172.21.15.83:0/3800690329 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e2c0678c0 msgr2=0x7f3e2c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:46.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.887+0000 7f3e2bfff640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e2c0678c0 0x7f3e2c069d80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3e380079c0 tx=0x7f3e38008040 comp rx=0 tx=0).stop 2024-09-19T07:41:46.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.887+0000 7f3e2bfff640 1 -- 172.21.15.83:0/3800690329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4806f630 msgr2=0x7f3e4806dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:46.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.887+0000 7f3e2bfff640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4806f630 0x7f3e4806dcb0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f3e4401a190 tx=0x7f3e44019300 comp rx=0 tx=0).stop 2024-09-19T07:41:46.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.887+0000 7f3e2bfff640 1 -- 172.21.15.83:0/3800690329 shutdown_connections 2024-09-19T07:41:46.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.887+0000 7f3e2bfff640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3e2c0678c0 0x7f3e2c069d80 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:46.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.887+0000 7f3e2bfff640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e480c5b70 0x7f3e481f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:46.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.887+0000 7f3e2bfff640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3e480c2b00 0x7f3e480c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:46.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.887+0000 7f3e2bfff640 1 --2- 172.21.15.83:0/3800690329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e4806f630 0x7f3e4806dcb0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:46.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.887+0000 7f3e2bfff640 1 -- 172.21.15.83:0/3800690329 >> 172.21.15.83:0/3800690329 conn(0x7f3e480c9470 msgr2=0x7f3e48070a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:46.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.887+0000 7f3e2bfff640 1 -- 172.21.15.83:0/3800690329 shutdown_connections 2024-09-19T07:41:46.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:46.887+0000 7f3e2bfff640 1 -- 172.21.15.83:0/3800690329 wait complete. 2024-09-19T07:41:46.925 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-19T07:41:46.925 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before thrashing 2024-09-19T07:41:53.926 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:54.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bfd175640 1 Processor -- start 2024-09-19T07:41:54.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bfd175640 1 -- start start 2024-09-19T07:41:54.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bfd175640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9bf80c5740 0x7f9bf81544f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:54.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bfd175640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf8157630 0x7f9bf8157a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bfd175640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bf80c22e0 con 0x7f9bf80c6350 2024-09-19T07:41:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bfd175640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f9bf8154a30 con 0x7f9bf80c5740 2024-09-19T07:41:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bfd175640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9bf80c2550 con 0x7f9bf8157630 2024-09-19T07:41:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bf6d9d640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f9bf80c6350 0x7f9bf80c4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:49736/0 (socket says 172.21.15.83:49736) 2024-09-19T07:41:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bf6d9d640 1 -- 172.21.15.83:0/2197743374 learned_addr learned my addr 172.21.15.83:0/2197743374 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:41:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bf5d9b640 1 -- 172.21.15.83:0/2197743374 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4074760447 0 0) 0x7f9bf80c22e0 con 0x7f9bf80c6350 2024-09-19T07:41:54.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bf5d9b640 1 -- 172.21.15.83:0/2197743374 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9be0003530 con 0x7f9bf80c6350 2024-09-19T07:41:54.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bf759e640 1 --2- 172.21.15.83:0/2197743374 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf8157630 0x7f9bf8157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bf759e640 1 -- 172.21.15.83:0/2197743374 >> v2:172.21.15.184:3300/0 conn(0x7f9bf80c5740 msgr2=0x7f9bf81544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bf759e640 1 --2- 172.21.15.83:0/2197743374 >> v2:172.21.15.184:3300/0 conn(0x7f9bf80c5740 0x7f9bf81544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bf759e640 1 -- 172.21.15.83:0/2197743374 >> v1:172.21.15.184:6789/0 conn(0x7f9bf80c6350 legacy=0x7f9bf80c4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.182+0000 7f9bf759e640 1 -- 172.21.15.83:0/2197743374 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf80c3b20 con 0x7f9bf8157630 2024-09-19T07:41:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.186+0000 7f9bf759e640 1 --2- 172.21.15.83:0/2197743374 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf8157630 0x7f9bf8157a80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009d90 tx=0x7f9bf001c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a5b2086dffd2ac5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.186+0000 7f9bf5d9b640 1 -- 172.21.15.83:0/2197743374 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf00b2070 con 0x7f9bf8157630 2024-09-19T07:41:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.186+0000 7f9bf5d9b640 1 -- 172.21.15.83:0/2197743374 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf00a5820 con 0x7f9bf8157630 2024-09-19T07:41:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.186+0000 7f9bf5d9b640 1 -- 172.21.15.83:0/2197743374 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf00a5bf0 con 0x7f9bf8157630 2024-09-19T07:41:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.190+0000 7f9bfd175640 1 -- 172.21.15.83:0/2197743374 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf8157630 msgr2=0x7f9bf8157a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.190+0000 7f9bfd175640 1 --2- 172.21.15.83:0/2197743374 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf8157630 0x7f9bf8157a80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009d90 tx=0x7f9bf001c870 comp rx=0 tx=0).stop 2024-09-19T07:41:54.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.190+0000 7f9bfd175640 1 -- 172.21.15.83:0/2197743374 shutdown_connections 2024-09-19T07:41:54.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.190+0000 7f9bfd175640 1 --2- 172.21.15.83:0/2197743374 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf8157630 0x7f9bf8157a80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:54.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.190+0000 7f9bfd175640 1 --2- 172.21.15.83:0/2197743374 >> v2:172.21.15.184:3300/0 conn(0x7f9bf80c5740 0x7f9bf81544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:54.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.190+0000 7f9bfd175640 1 -- 172.21.15.83:0/2197743374 >> 172.21.15.83:0/2197743374 conn(0x7f9bf806c080 msgr2=0x7f9bf806c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:54.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bfd175640 1 -- 172.21.15.83:0/2197743374 shutdown_connections 2024-09-19T07:41:54.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bfd175640 1 -- 172.21.15.83:0/2197743374 wait complete. 2024-09-19T07:41:54.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bfd175640 1 Processor -- start 2024-09-19T07:41:54.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bfd175640 1 -- start start 2024-09-19T07:41:54.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bfd175640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bf80c5740 0x7f9bf80ac5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:54.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bfd175640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bf80c6350 0x7f9bf814ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:54.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bfd175640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf8157630 0x7f9bf80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:54.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bfd175640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf80c4e00 con 0x7f9bf80c5740 2024-09-19T07:41:54.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bfd175640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf80c4c80 con 0x7f9bf80c6350 2024-09-19T07:41:54.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bfd175640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9bf80c6910 con 0x7f9bf8157630 2024-09-19T07:41:54.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bf759e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf8157630 0x7f9bf80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:54.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bf659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bf80c6350 0x7f9bf814ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:54.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bf759e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf8157630 0x7f9bf80a6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:46598/0 (socket says 172.21.15.83:46598) 2024-09-19T07:41:54.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bf759e640 1 -- 172.21.15.83:0/4131827601 learned_addr learned my addr 172.21.15.83:0/4131827601 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:54.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.194+0000 7f9bf6d9d640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bf80c5740 0x7f9bf80ac5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:54.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bf659c640 1 -- 172.21.15.83:0/4131827601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf8157630 msgr2=0x7f9bf80a6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:54.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bf659c640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf8157630 0x7f9bf80a6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:54.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bf659c640 1 -- 172.21.15.83:0/4131827601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bf80c5740 msgr2=0x7f9bf80ac5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:54.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bf659c640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bf80c5740 0x7f9bf80ac5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:54.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bf659c640 1 -- 172.21.15.83:0/4131827601 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf814d810 con 0x7f9bf80c6350 2024-09-19T07:41:54.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bf6d9d640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bf80c5740 0x7f9bf80ac5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:41:54.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bf659c640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bf80c6350 0x7f9bf814ce30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00b440 tx=0x7f9bec00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:54.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bdf7fe640 1 -- 172.21.15.83:0/4131827601 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bec015020 con 0x7f9bf80c6350 2024-09-19T07:41:54.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bdf7fe640 1 -- 172.21.15.83:0/4131827601 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bec011830 con 0x7f9bf80c6350 2024-09-19T07:41:54.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bdf7fe640 1 -- 172.21.15.83:0/4131827601 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bec011c20 con 0x7f9bf80c6350 2024-09-19T07:41:54.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bfd175640 1 -- 172.21.15.83:0/4131827601 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf8154e20 con 0x7f9bf80c6350 2024-09-19T07:41:54.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bfd175640 1 -- 172.21.15.83:0/4131827601 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf81f35b0 con 0x7f9bf80c6350 2024-09-19T07:41:54.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.198+0000 7f9bfd175640 1 -- 172.21.15.83:0/4131827601 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bf81544f0 con 0x7f9bf80c6350 2024-09-19T07:41:54.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.202+0000 7f9bdf7fe640 1 -- 172.21.15.83:0/4131827601 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9bec018020 con 0x7f9bf80c6350 2024-09-19T07:41:54.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.202+0000 7f9bdf7fe640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9be40678c0 0x7f9be4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:54.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.202+0000 7f9bdf7fe640 1 -- 172.21.15.83:0/4131827601 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9303+0+0 (secure 0 0 0) 0x7f9bec086d80 con 0x7f9bf80c6350 2024-09-19T07:41:54.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.202+0000 7f9bf6d9d640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9be40678c0 0x7f9be4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:54.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.202+0000 7f9bf6d9d640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9be40678c0 0x7f9be4069d80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9be8000f70 tx=0x7f9be8007d40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:54.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.206+0000 7f9bdf7fe640 1 -- 172.21.15.83:0/4131827601 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9bec01f020 con 0x7f9bf80c6350 2024-09-19T07:41:54.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.702+0000 7f9bfd175640 1 -- 172.21.15.83:0/4131827601 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9bf80ac320 con 0x7f9bf80c6350 2024-09-19T07:41:54.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.702+0000 7f9bdf7fe640 1 -- 172.21.15.83:0/4131827601 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+6545 (secure 0 0 0) 0x7f9bec0564c0 con 0x7f9bf80c6350 2024-09-19T07:41:54.702 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:54.702 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":160,"btime":"2024-09-19T07:41:46:425682+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:41:36.478574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5951":{"gid":5951,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5951,"qdb_cluster":[5951]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:54.702 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 160 2024-09-19T07:41:54.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.706+0000 7f9bfd175640 1 -- 172.21.15.83:0/4131827601 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9be40678c0 msgr2=0x7f9be4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:54.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.706+0000 7f9bfd175640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9be40678c0 0x7f9be4069d80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9be8000f70 tx=0x7f9be8007d40 comp rx=0 tx=0).stop 2024-09-19T07:41:54.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.706+0000 7f9bfd175640 1 -- 172.21.15.83:0/4131827601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bf80c6350 msgr2=0x7f9bf814ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:54.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.706+0000 7f9bfd175640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bf80c6350 0x7f9bf814ce30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00b440 tx=0x7f9bec00b840 comp rx=0 tx=0).stop 2024-09-19T07:41:54.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.706+0000 7f9bfd175640 1 -- 172.21.15.83:0/4131827601 shutdown_connections 2024-09-19T07:41:54.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.706+0000 7f9bfd175640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9be40678c0 0x7f9be4069d80 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:54.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.706+0000 7f9bfd175640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf8157630 0x7f9bf80a6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:54.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.706+0000 7f9bfd175640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bf80c6350 0x7f9bf814ce30 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:54.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.706+0000 7f9bfd175640 1 --2- 172.21.15.83:0/4131827601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bf80c5740 0x7f9bf80ac5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:54.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.706+0000 7f9bfd175640 1 -- 172.21.15.83:0/4131827601 >> 172.21.15.83:0/4131827601 conn(0x7f9bf806c080 msgr2=0x7f9bf8153ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:54.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.706+0000 7f9bfd175640 1 -- 172.21.15.83:0/4131827601 shutdown_connections 2024-09-19T07:41:54.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.706+0000 7f9bfd175640 1 -- 172.21.15.83:0/4131827601 wait complete. 2024-09-19T07:41:54.745 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-19T07:41:54.746 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:41:54.746 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:41:54.762 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:41:54.762 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:54.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.994+0000 7f020d9df640 1 Processor -- start 2024-09-19T07:41:54.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.994+0000 7f020d9df640 1 -- start start 2024-09-19T07:41:54.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f020d9df640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f020806de90 0x7f02080c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:54.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f020d9df640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f02080c4010 0x7f02080c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:54.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f020d9df640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f020814df80 con 0x7f020814dba0 2024-09-19T07:41:54.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f020d9df640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f02080c7510 con 0x7f02080c4010 2024-09-19T07:41:54.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f020d9df640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f02080c8870 con 0x7f020806de90 2024-09-19T07:41:54.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f0206ffd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f020814dba0 0x7f0208154540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:49750/0 (socket says 172.21.15.83:49750) 2024-09-19T07:41:54.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f0206ffd640 1 -- 172.21.15.83:0/2416889242 learned_addr learned my addr 172.21.15.83:0/2416889242 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:41:54.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f02077fe640 1 --2- 172.21.15.83:0/2416889242 >> v2:172.21.15.83:3300/0 conn(0x7f02080c4010 0x7f02080c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:54.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f02067fc640 1 --2- 172.21.15.83:0/2416889242 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f020806de90 0x7f02080c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:54.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f02077fe640 1 -- 172.21.15.83:0/2416889242 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f020806de90 msgr2=0x7f02080c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:41:54.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f02077fe640 1 --2- 172.21.15.83:0/2416889242 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f020806de90 0x7f02080c8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:54.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f02077fe640 1 -- 172.21.15.83:0/2416889242 >> v1:172.21.15.184:6789/0 conn(0x7f020814dba0 legacy=0x7f0208154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:54.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f02077fe640 1 -- 172.21.15.83:0/2416889242 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02080c7740 con 0x7f02080c4010 2024-09-19T07:41:54.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f02067fc640 1 --2- 172.21.15.83:0/2416889242 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f020806de90 0x7f02080c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:41:54.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:54.998+0000 7f02077fe640 1 --2- 172.21.15.83:0/2416889242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02080c4010 0x7f02080c44c0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f02080c89f0 tx=0x7f020001c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b19be8101ed71bd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:54.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.002+0000 7f0205ffb640 1 -- 172.21.15.83:0/2416889242 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02000b2070 con 0x7f02080c4010 2024-09-19T07:41:54.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.002+0000 7f0205ffb640 1 -- 172.21.15.83:0/2416889242 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02000a5890 con 0x7f02080c4010 2024-09-19T07:41:54.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.002+0000 7f0205ffb640 1 -- 172.21.15.83:0/2416889242 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02000a5c60 con 0x7f02080c4010 2024-09-19T07:41:54.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.002+0000 7f020d9df640 1 -- 172.21.15.83:0/2416889242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02080c4010 msgr2=0x7f02080c44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:54.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.002+0000 7f020d9df640 1 --2- 172.21.15.83:0/2416889242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02080c4010 0x7f02080c44c0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f02080c89f0 tx=0x7f020001c8c0 comp rx=0 tx=0).stop 2024-09-19T07:41:55.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.002+0000 7f020d9df640 1 -- 172.21.15.83:0/2416889242 shutdown_connections 2024-09-19T07:41:55.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.002+0000 7f020d9df640 1 --2- 172.21.15.83:0/2416889242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02080c4010 0x7f02080c44c0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:55.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.002+0000 7f020d9df640 1 --2- 172.21.15.83:0/2416889242 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f020806de90 0x7f02080c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:55.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.002+0000 7f020d9df640 1 -- 172.21.15.83:0/2416889242 >> 172.21.15.83:0/2416889242 conn(0x7f020806ff50 msgr2=0x7f0208070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:55.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.002+0000 7f020d9df640 1 -- 172.21.15.83:0/2416889242 shutdown_connections 2024-09-19T07:41:55.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.002+0000 7f020d9df640 1 -- 172.21.15.83:0/2416889242 wait complete. 2024-09-19T07:41:55.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f020d9df640 1 Processor -- start 2024-09-19T07:41:55.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f020d9df640 1 -- start start 2024-09-19T07:41:55.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f020d9df640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f020806de90 0x7f0208153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:55.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f020d9df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02080c4010 0x7f020814ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:55.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f020d9df640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f020814dba0 0x7f0208156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:55.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f02067fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02080c4010 0x7f020814ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:55.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f02067fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02080c4010 0x7f020814ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38462/0 (socket says 172.21.15.83:38462) 2024-09-19T07:41:55.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f02067fc640 1 -- 172.21.15.83:0/3935575446 learned_addr learned my addr 172.21.15.83:0/3935575446 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:55.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f0206ffd640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f020806de90 0x7f0208153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:55.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f02080c3bb0 con 0x7f02080c4010 2024-09-19T07:41:55.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f020806dc90 con 0x7f020806de90 2024-09-19T07:41:55.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f02080c3d30 con 0x7f020814dba0 2024-09-19T07:41:55.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f02077fe640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f020814dba0 0x7f0208156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:55.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f0206ffd640 1 -- 172.21.15.83:0/3935575446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f020814dba0 msgr2=0x7f0208156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:55.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f0206ffd640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f020814dba0 0x7f0208156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:55.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f0206ffd640 1 -- 172.21.15.83:0/3935575446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02080c4010 msgr2=0x7f020814ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:55.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f0206ffd640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02080c4010 0x7f020814ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:55.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f0206ffd640 1 -- 172.21.15.83:0/3935575446 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020815ca50 con 0x7f020806de90 2024-09-19T07:41:55.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f02067fc640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02080c4010 0x7f020814ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:41:55.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.006+0000 7f0206ffd640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f020806de90 0x7f0208153d60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f01f800c3c0 tx=0x7f01f800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:55.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.010+0000 7f01effff640 1 -- 172.21.15.83:0/3935575446 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f8017070 con 0x7f020806de90 2024-09-19T07:41:55.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.010+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02081f1b60 con 0x7f020806de90 2024-09-19T07:41:55.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.010+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0208154750 con 0x7f020806de90 2024-09-19T07:41:55.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.010+0000 7f01effff640 1 -- 172.21.15.83:0/3935575446 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01f8005cf0 con 0x7f020806de90 2024-09-19T07:41:55.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.010+0000 7f01effff640 1 -- 172.21.15.83:0/3935575446 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f80072f0 con 0x7f020806de90 2024-09-19T07:41:55.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.010+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e80052a0 con 0x7f020806de90 2024-09-19T07:41:55.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.010+0000 7f01effff640 1 -- 172.21.15.83:0/3935575446 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f01f801a070 con 0x7f020806de90 2024-09-19T07:41:55.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.010+0000 7f01effff640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01f4067910 0x7f01f4069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:55.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.010+0000 7f01effff640 1 -- 172.21.15.83:0/3935575446 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9303+0+0 (secure 0 0 0) 0x7f01f8085c30 con 0x7f020806de90 2024-09-19T07:41:55.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.010+0000 7f02067fc640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01f4067910 0x7f01f4069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:55.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.010+0000 7f02067fc640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01f4067910 0x7f01f4069dd0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009600 tx=0x7f01fc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:55.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.018+0000 7f01effff640 1 -- 172.21.15.83:0/3935575446 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f01f80553d0 con 0x7f020806de90 2024-09-19T07:41:55.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.402+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01e8005860 con 0x7f020806de90 2024-09-19T07:41:55.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.402+0000 7f01effff640 1 -- 172.21.15.83:0/3935575446 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+6545 (secure 0 0 0) 0x7f01f80596d0 con 0x7f020806de90 2024-09-19T07:41:55.401 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:55.401 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":160,"btime":"2024-09-19T07:41:46:425682+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:41:36.478574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5951":{"gid":5951,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5951,"qdb_cluster":[5951]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:55.402 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 160 2024-09-19T07:41:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.406+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01f4067910 msgr2=0x7f01f4069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.406+0000 7f020d9df640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01f4067910 0x7f01f4069dd0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009600 tx=0x7f01fc009190 comp rx=0 tx=0).stop 2024-09-19T07:41:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.406+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f020806de90 msgr2=0x7f0208153d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.406+0000 7f020d9df640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f020806de90 0x7f0208153d60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f01f800c3c0 tx=0x7f01f800c880 comp rx=0 tx=0).stop 2024-09-19T07:41:55.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.410+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 shutdown_connections 2024-09-19T07:41:55.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.410+0000 7f020d9df640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01f4067910 0x7f01f4069dd0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:55.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.410+0000 7f020d9df640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f020814dba0 0x7f0208156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:55.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.410+0000 7f020d9df640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02080c4010 0x7f020814ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:55.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.410+0000 7f020d9df640 1 --2- 172.21.15.83:0/3935575446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f020806de90 0x7f0208153d60 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:55.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.410+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 >> 172.21.15.83:0/3935575446 conn(0x7f020806ff50 msgr2=0x7f020806f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:55.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.410+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 shutdown_connections 2024-09-19T07:41:55.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:55.410+0000 7f020d9df640 1 -- 172.21.15.83:0/3935575446 wait complete. 2024-09-19T07:41:55.437 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:41:57.438 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:41:57.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.618+0000 7f15ebf47640 1 Processor -- start 2024-09-19T07:41:57.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.618+0000 7f15ebf47640 1 -- start start 2024-09-19T07:41:57.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.622+0000 7f15ebf47640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f15e40c6120 0x7f15e40c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:57.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.622+0000 7f15ebf47640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15e41592f0 0x7f15e4157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:57.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.622+0000 7f15ebf47640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15e4156200 con 0x7f15e4155d70 2024-09-19T07:41:57.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.622+0000 7f15ebf47640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f15e40c9930 con 0x7f15e40c6120 2024-09-19T07:41:57.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.622+0000 7f15ebf47640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f15e41564e0 con 0x7f15e41592f0 2024-09-19T07:41:57.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.622+0000 7f15ea4e4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15e41592f0 0x7f15e4157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:57.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.622+0000 7f15ea4e4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15e41592f0 0x7f15e4157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:46622/0 (socket says 172.21.15.83:46622) 2024-09-19T07:41:57.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.622+0000 7f15ea4e4640 1 -- 172.21.15.83:0/157133178 learned_addr learned my addr 172.21.15.83:0/157133178 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:57.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.622+0000 7f15ea4e4640 1 -- 172.21.15.83:0/157133178 >> v2:172.21.15.184:3300/0 conn(0x7f15e40c6120 msgr2=0x7f15e40c6500 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:41:57.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.622+0000 7f15ea4e4640 1 --2- 172.21.15.83:0/157133178 >> v2:172.21.15.184:3300/0 conn(0x7f15e40c6120 0x7f15e40c6500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:57.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.622+0000 7f15ea4e4640 1 -- 172.21.15.83:0/157133178 >> v1:172.21.15.83:6789/0 conn(0x7f15e4155d70 legacy=0x7f15e41575d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:41:57.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.622+0000 7f15ea4e4640 1 -- 172.21.15.83:0/157133178 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e41567c0 con 0x7f15e41592f0 2024-09-19T07:41:57.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.634+0000 7f15ea4e4640 1 --2- 172.21.15.83:0/157133178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15e41592f0 0x7f15e4157ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009e50 tx=0x7f15dc01ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69ebedf258f91e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:57.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.634+0000 7f15e8ce1640 1 -- 172.21.15.83:0/157133178 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc0a5690 con 0x7f15e41592f0 2024-09-19T07:41:57.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.634+0000 7f15e8ce1640 1 -- 172.21.15.83:0/157133178 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15dc0a5830 con 0x7f15e41592f0 2024-09-19T07:41:57.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.634+0000 7f15e8ce1640 1 -- 172.21.15.83:0/157133178 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc0a5c00 con 0x7f15e41592f0 2024-09-19T07:41:57.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.634+0000 7f15ebf47640 1 -- 172.21.15.83:0/157133178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15e41592f0 msgr2=0x7f15e4157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:57.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.634+0000 7f15ebf47640 1 --2- 172.21.15.83:0/157133178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15e41592f0 0x7f15e4157ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009e50 tx=0x7f15dc01ca10 comp rx=0 tx=0).stop 2024-09-19T07:41:57.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.634+0000 7f15ebf47640 1 -- 172.21.15.83:0/157133178 shutdown_connections 2024-09-19T07:41:57.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.634+0000 7f15ebf47640 1 --2- 172.21.15.83:0/157133178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15e41592f0 0x7f15e4157ce0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:57.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.634+0000 7f15ebf47640 1 --2- 172.21.15.83:0/157133178 >> v2:172.21.15.184:3300/0 conn(0x7f15e40c6120 0x7f15e40c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:57.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.634+0000 7f15ebf47640 1 -- 172.21.15.83:0/157133178 >> 172.21.15.83:0/157133178 conn(0x7f15e406bc80 msgr2=0x7f15e406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:57.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.634+0000 7f15ebf47640 1 -- 172.21.15.83:0/157133178 shutdown_connections 2024-09-19T07:41:57.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.634+0000 7f15ebf47640 1 -- 172.21.15.83:0/157133178 wait complete. 2024-09-19T07:41:57.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ebf47640 1 Processor -- start 2024-09-19T07:41:57.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ebf47640 1 -- start start 2024-09-19T07:41:57.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ebf47640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15e40c6120 0x7f15e40ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:57.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ebf47640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e41592f0 0x7f15e4155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:57.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ebf47640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e4155660 0x7f15e40c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:57.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ebf47640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f15e4153860 con 0x7f15e4155660 2024-09-19T07:41:57.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ebf47640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f15e4162ea0 con 0x7f15e41592f0 2024-09-19T07:41:57.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ebf47640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f15e40cb410 con 0x7f15e40c6120 2024-09-19T07:41:57.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ea4e4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e4155660 0x7f15e40c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:57.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ea4e4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e4155660 0x7f15e40c1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38468/0 (socket says 172.21.15.83:38468) 2024-09-19T07:41:57.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ea4e4640 1 -- 172.21.15.83:0/178928717 learned_addr learned my addr 172.21.15.83:0/178928717 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:41:57.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15e9ce3640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15e40c6120 0x7f15e40ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:57.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ea4e4640 1 -- 172.21.15.83:0/178928717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15e40c6120 msgr2=0x7f15e40ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:57.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ea4e4640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15e40c6120 0x7f15e40ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:57.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ea4e4640 1 -- 172.21.15.83:0/178928717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e41592f0 msgr2=0x7f15e4155120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:41:57.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ea4e4640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e41592f0 0x7f15e4155120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:57.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ea4e4640 1 -- 172.21.15.83:0/178928717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e40cb6f0 con 0x7f15e4155660 2024-09-19T07:41:57.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ea4e4640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e4155660 0x7f15e40c1bf0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f15dc0af040 tx=0x7f15dc0197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:57.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15da7fc640 1 -- 172.21.15.83:0/178928717 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc0b8670 con 0x7f15e4155660 2024-09-19T07:41:57.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ebf47640 1 -- 172.21.15.83:0/178928717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e40c13f0 con 0x7f15e4155660 2024-09-19T07:41:57.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.638+0000 7f15ebf47640 1 -- 172.21.15.83:0/178928717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e40c0730 con 0x7f15e4155660 2024-09-19T07:41:57.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.642+0000 7f15da7fc640 1 -- 172.21.15.83:0/178928717 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15dc0be070 con 0x7f15e4155660 2024-09-19T07:41:57.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.642+0000 7f15da7fc640 1 -- 172.21.15.83:0/178928717 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc0b8a50 con 0x7f15e4155660 2024-09-19T07:41:57.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.642+0000 7f15ebf47640 1 -- 172.21.15.83:0/178928717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e40c6500 con 0x7f15e4155660 2024-09-19T07:41:57.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.642+0000 7f15da7fc640 1 -- 172.21.15.83:0/178928717 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f15dc0b8bf0 con 0x7f15e4155660 2024-09-19T07:41:57.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.642+0000 7f15da7fc640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15c80678c0 0x7f15c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:41:57.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.642+0000 7f15da7fc640 1 -- 172.21.15.83:0/178928717 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9303+0+0 (secure 0 0 0) 0x7f15dc121c70 con 0x7f15e4155660 2024-09-19T07:41:57.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.642+0000 7f15e9ce3640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15c80678c0 0x7f15c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:41:57.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.642+0000 7f15e9ce3640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15c80678c0 0x7f15c8069d80 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009890 tx=0x7f15d4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:41:57.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:57.650+0000 7f15da7fc640 1 -- 172.21.15.83:0/178928717 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f15dc016600 con 0x7f15e4155660 2024-09-19T07:41:58.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.030+0000 7f15ebf47640 1 -- 172.21.15.83:0/178928717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15e40c96a0 con 0x7f15e4155660 2024-09-19T07:41:58.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.030+0000 7f15da7fc640 1 -- 172.21.15.83:0/178928717 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+6545 (secure 0 0 0) 0x7f15dc0f1950 con 0x7f15e4155660 2024-09-19T07:41:58.029 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:41:58.029 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":160,"btime":"2024-09-19T07:41:46:425682+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:41:36.478574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5951":{"gid":5951,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5951,"qdb_cluster":[5951]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:41:58.030 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 160 2024-09-19T07:41:58.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.034+0000 7f15ebf47640 1 -- 172.21.15.83:0/178928717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15c80678c0 msgr2=0x7f15c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:58.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.034+0000 7f15ebf47640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15c80678c0 0x7f15c8069d80 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009890 tx=0x7f15d4008040 comp rx=0 tx=0).stop 2024-09-19T07:41:58.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.034+0000 7f15ebf47640 1 -- 172.21.15.83:0/178928717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e4155660 msgr2=0x7f15e40c1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:41:58.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.034+0000 7f15ebf47640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e4155660 0x7f15e40c1bf0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f15dc0af040 tx=0x7f15dc0197b0 comp rx=0 tx=0).stop 2024-09-19T07:41:58.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.034+0000 7f15ebf47640 1 -- 172.21.15.83:0/178928717 shutdown_connections 2024-09-19T07:41:58.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.034+0000 7f15ebf47640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f15c80678c0 0x7f15c8069d80 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:58.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.034+0000 7f15ebf47640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e4155660 0x7f15e40c1bf0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:58.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.034+0000 7f15ebf47640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e41592f0 0x7f15e4155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:58.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.034+0000 7f15ebf47640 1 --2- 172.21.15.83:0/178928717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15e40c6120 0x7f15e40ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:41:58.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.034+0000 7f15ebf47640 1 -- 172.21.15.83:0/178928717 >> 172.21.15.83:0/178928717 conn(0x7f15e406bc80 msgr2=0x7f15e41575d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:41:58.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.038+0000 7f15ebf47640 1 -- 172.21.15.83:0/178928717 shutdown_connections 2024-09-19T07:41:58.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:41:58.038+0000 7f15ebf47640 1 -- 172.21.15.83:0/178928717 wait complete. 2024-09-19T07:41:58.082 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:42:00.083 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:00.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f5d2f2640 1 Processor -- start 2024-09-19T07:42:00.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f5d2f2640 1 -- start start 2024-09-19T07:42:00.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f5d2f2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4f580c6120 0x7f4f580c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:00.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f5d2f2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f58155d70 0x7f4f581575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:00.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f5d2f2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f581592f0 0x7f4f58157b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:00.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f5d2f2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f4f580c9930 con 0x7f4f581592f0 2024-09-19T07:42:00.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f5d2f2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4f58156200 con 0x7f4f580c6120 2024-09-19T07:42:00.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f5d2f2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4f581564e0 con 0x7f4f58155d70 2024-09-19T07:42:00.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f577fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f581592f0 0x7f4f58157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:00.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f577fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f581592f0 0x7f4f58157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60020/0 (socket says 172.21.15.83:60020) 2024-09-19T07:42:00.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f577fe640 1 -- 172.21.15.83:0/2044042574 learned_addr learned my addr 172.21.15.83:0/2044042574 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:00.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f567fc640 1 --2- 172.21.15.83:0/2044042574 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f58155d70 0x7f4f581575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:00.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f56ffd640 1 --2- 172.21.15.83:0/2044042574 >> v2:172.21.15.184:3300/0 conn(0x7f4f580c6120 0x7f4f580c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:00.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f577fe640 1 -- 172.21.15.83:0/2044042574 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f58155d70 msgr2=0x7f4f581575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:00.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f577fe640 1 --2- 172.21.15.83:0/2044042574 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f58155d70 0x7f4f581575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f577fe640 1 -- 172.21.15.83:0/2044042574 >> v2:172.21.15.184:3300/0 conn(0x7f4f580c6120 msgr2=0x7f4f580c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:00.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f577fe640 1 --2- 172.21.15.83:0/2044042574 >> v2:172.21.15.184:3300/0 conn(0x7f4f580c6120 0x7f4f580c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.218+0000 7f4f577fe640 1 -- 172.21.15.83:0/2044042574 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f581567c0 con 0x7f4f581592f0 2024-09-19T07:42:00.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f577fe640 1 --2- 172.21.15.83:0/2044042574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f581592f0 0x7f4f58157b10 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f4f5808d540 tx=0x7f4f5001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbca5fbad7a6a8c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:00.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f55ffb640 1 -- 172.21.15.83:0/2044042574 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f500a5670 con 0x7f4f581592f0 2024-09-19T07:42:00.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f55ffb640 1 -- 172.21.15.83:0/2044042574 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f500a5810 con 0x7f4f581592f0 2024-09-19T07:42:00.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f55ffb640 1 -- 172.21.15.83:0/2044042574 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f500a5be0 con 0x7f4f581592f0 2024-09-19T07:42:00.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2044042574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f581592f0 msgr2=0x7f4f58157b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:00.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 --2- 172.21.15.83:0/2044042574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f581592f0 0x7f4f58157b10 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f4f5808d540 tx=0x7f4f5001c9f0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2044042574 shutdown_connections 2024-09-19T07:42:00.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 --2- 172.21.15.83:0/2044042574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f581592f0 0x7f4f58157b10 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 --2- 172.21.15.83:0/2044042574 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f58155d70 0x7f4f581575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 --2- 172.21.15.83:0/2044042574 >> v2:172.21.15.184:3300/0 conn(0x7f4f580c6120 0x7f4f580c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2044042574 >> 172.21.15.83:0/2044042574 conn(0x7f4f5806bc80 msgr2=0x7f4f5806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:00.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2044042574 shutdown_connections 2024-09-19T07:42:00.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2044042574 wait complete. 2024-09-19T07:42:00.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 Processor -- start 2024-09-19T07:42:00.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 -- start start 2024-09-19T07:42:00.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f580c6120 0x7f4f580caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:00.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f581592f0 0x7f4f581ef460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:00.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f58159a10 0x7f4f581efe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:00.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4f580c8b40 con 0x7f4f581592f0 2024-09-19T07:42:00.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4f580c89c0 con 0x7f4f58159a10 2024-09-19T07:42:00.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f5d2f2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4f580c8cc0 con 0x7f4f580c6120 2024-09-19T07:42:00.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f567fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f581592f0 0x7f4f581ef460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:00.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f567fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f581592f0 0x7f4f581ef460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60032/0 (socket says 172.21.15.83:60032) 2024-09-19T07:42:00.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f567fc640 1 -- 172.21.15.83:0/2816061691 learned_addr learned my addr 172.21.15.83:0/2816061691 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:00.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f56ffd640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f580c6120 0x7f4f580caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:00.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.222+0000 7f4f577fe640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f58159a10 0x7f4f581efe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f567fc640 1 -- 172.21.15.83:0/2816061691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f580c6120 msgr2=0x7f4f580caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f567fc640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f580c6120 0x7f4f580caaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f567fc640 1 -- 172.21.15.83:0/2816061691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f58159a10 msgr2=0x7f4f581efe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f567fc640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f58159a10 0x7f4f581efe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f567fc640 1 -- 172.21.15.83:0/2816061691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f580cbc70 con 0x7f4f581592f0 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f577fe640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f58159a10 0x7f4f581efe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f567fc640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f581592f0 0x7f4f581ef460 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800e9e0 tx=0x7f4f4800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f3f7fe640 1 -- 172.21.15.83:0/2816061691 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f48017070 con 0x7f4f581592f0 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f3f7fe640 1 -- 172.21.15.83:0/2816061691 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f4800c9d0 con 0x7f4f581592f0 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2816061691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f580ca010 con 0x7f4f581592f0 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f3f7fe640 1 -- 172.21.15.83:0/2816061691 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f4800cda0 con 0x7f4f581592f0 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2816061691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f580cb940 con 0x7f4f581592f0 2024-09-19T07:42:00.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2816061691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f580c8f60 con 0x7f4f581592f0 2024-09-19T07:42:00.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f3f7fe640 1 -- 172.21.15.83:0/2816061691 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4f4801a070 con 0x7f4f581592f0 2024-09-19T07:42:00.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f3f7fe640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f440678f0 0x7f4f44069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:00.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f56ffd640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f440678f0 0x7f4f44069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:00.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f3f7fe640 1 -- 172.21.15.83:0/2816061691 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9303+0+0 (secure 0 0 0) 0x7f4f48086d90 con 0x7f4f581592f0 2024-09-19T07:42:00.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.226+0000 7f4f56ffd640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f440678f0 0x7f4f44069db0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009ae0 tx=0x7f4f4c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:00.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.230+0000 7f4f3f7fe640 1 -- 172.21.15.83:0/2816061691 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4f480564b0 con 0x7f4f581592f0 2024-09-19T07:42:00.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.414+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2816061691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f580c96a0 con 0x7f4f581592f0 2024-09-19T07:42:00.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f3f7fe640 1 -- 172.21.15.83:0/2816061691 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+6545 (secure 0 0 0) 0x7f4f4805a7b0 con 0x7f4f581592f0 2024-09-19T07:42:00.415 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:00.415 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":160,"btime":"2024-09-19T07:41:46:425682+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:41:36.478574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5951":{"gid":5951,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5951,"qdb_cluster":[5951]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:00.415 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 160 2024-09-19T07:42:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2816061691 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f440678f0 msgr2=0x7f4f44069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f5d2f2640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f440678f0 0x7f4f44069db0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009ae0 tx=0x7f4f4c009190 comp rx=0 tx=0).stop 2024-09-19T07:42:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2816061691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f581592f0 msgr2=0x7f4f581ef460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f5d2f2640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f581592f0 0x7f4f581ef460 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800e9e0 tx=0x7f4f4800eea0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2816061691 shutdown_connections 2024-09-19T07:42:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f5d2f2640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4f440678f0 0x7f4f44069db0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f5d2f2640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f58159a10 0x7f4f581efe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f5d2f2640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f581592f0 0x7f4f581ef460 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f5d2f2640 1 --2- 172.21.15.83:0/2816061691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f580c6120 0x7f4f580caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2816061691 >> 172.21.15.83:0/2816061691 conn(0x7f4f5806bc80 msgr2=0x7f4f5815fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2816061691 shutdown_connections 2024-09-19T07:42:00.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:00.418+0000 7f4f5d2f2640 1 -- 172.21.15.83:0/2816061691 wait complete. 2024-09-19T07:42:00.432 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:42:02.433 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:02.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f253cf67640 1 Processor -- start 2024-09-19T07:42:02.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f253cf67640 1 -- start start 2024-09-19T07:42:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f253cf67640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2538071100 0x7f25381545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f253cf67640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f25380c28e0 0x7f2538150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f253cf67640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25380a9e60 con 0x7f2538151480 2024-09-19T07:42:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f253cf67640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f2538071b50 con 0x7f25380c28e0 2024-09-19T07:42:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f253cf67640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f25380c1ff0 con 0x7f2538071100 2024-09-19T07:42:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f2535d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f25380c28e0 0x7f2538150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f2536d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2538151480 0x7f253806dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56922/0 (socket says 172.21.15.83:56922) 2024-09-19T07:42:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f2536d9d640 1 -- 172.21.15.83:0/676970424 learned_addr learned my addr 172.21.15.83:0/676970424 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:42:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f253659c640 1 --2- 172.21.15.83:0/676970424 >> v2:172.21.15.184:3300/0 conn(0x7f2538071100 0x7f25381545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f2535d9b640 1 -- 172.21.15.83:0/676970424 >> v2:172.21.15.184:3300/0 conn(0x7f2538071100 msgr2=0x7f25381545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f2535d9b640 1 --2- 172.21.15.83:0/676970424 >> v2:172.21.15.184:3300/0 conn(0x7f2538071100 0x7f25381545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:02.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f2535d9b640 1 -- 172.21.15.83:0/676970424 >> v1:172.21.15.83:6789/0 conn(0x7f2538151480 legacy=0x7f253806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:02.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f2535d9b640 1 -- 172.21.15.83:0/676970424 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25380aa140 con 0x7f25380c28e0 2024-09-19T07:42:02.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f253559a640 1 -- 172.21.15.83:0/676970424 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2059228745 0 0) 0x7f25380a9e60 con 0x7f2538151480 2024-09-19T07:42:02.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f2535d9b640 1 --2- 172.21.15.83:0/676970424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25380c28e0 0x7f2538150f40 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f253806f620 tx=0x7f252801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90bcd797f8eb79f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:02.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.570+0000 7f253559a640 1 -- 172.21.15.83:0/676970424 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25280b0070 con 0x7f25380c28e0 2024-09-19T07:42:02.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253559a640 1 -- 172.21.15.83:0/676970424 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25280ab600 con 0x7f25380c28e0 2024-09-19T07:42:02.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253559a640 1 -- 172.21.15.83:0/676970424 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25280ab9d0 con 0x7f25380c28e0 2024-09-19T07:42:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 -- 172.21.15.83:0/676970424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25380c28e0 msgr2=0x7f2538150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 --2- 172.21.15.83:0/676970424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25380c28e0 0x7f2538150f40 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f253806f620 tx=0x7f252801c3f0 comp rx=0 tx=0).stop 2024-09-19T07:42:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 -- 172.21.15.83:0/676970424 shutdown_connections 2024-09-19T07:42:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 --2- 172.21.15.83:0/676970424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25380c28e0 0x7f2538150f40 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 --2- 172.21.15.83:0/676970424 >> v2:172.21.15.184:3300/0 conn(0x7f2538071100 0x7f25381545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 -- 172.21.15.83:0/676970424 >> 172.21.15.83:0/676970424 conn(0x7f253806bc80 msgr2=0x7f253806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 -- 172.21.15.83:0/676970424 shutdown_connections 2024-09-19T07:42:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 -- 172.21.15.83:0/676970424 wait complete. 2024-09-19T07:42:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 Processor -- start 2024-09-19T07:42:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 -- start start 2024-09-19T07:42:02.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2538071100 0x7f2538153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:02.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25380c28e0 0x7f2538148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:02.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2538151480 0x7f25380c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:02.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2538071660 con 0x7f2538151480 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f25380714e0 con 0x7f2538071100 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253cf67640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2538160c60 con 0x7f25380c28e0 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f2536d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2538151480 0x7f25380c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f2536d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2538151480 0x7f25380c5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60042/0 (socket says 172.21.15.83:60042) 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f2536d9d640 1 -- 172.21.15.83:0/875360370 learned_addr learned my addr 172.21.15.83:0/875360370 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f2535d9b640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25380c28e0 0x7f2538148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f253659c640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2538071100 0x7f2538153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f2536d9d640 1 -- 172.21.15.83:0/875360370 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25380c28e0 msgr2=0x7f2538148cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f2536d9d640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25380c28e0 0x7f2538148cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f2536d9d640 1 -- 172.21.15.83:0/875360370 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2538071100 msgr2=0x7f2538153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f2536d9d640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2538071100 0x7f2538153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f2536d9d640 1 -- 172.21.15.83:0/875360370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2538154250 con 0x7f2538151480 2024-09-19T07:42:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.574+0000 7f2536d9d640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2538151480 0x7f25380c5730 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f253000c730 tx=0x7f253000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:02.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.578+0000 7f2526ffd640 1 -- 172.21.15.83:0/875360370 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2530018070 con 0x7f2538151480 2024-09-19T07:42:02.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.578+0000 7f2526ffd640 1 -- 172.21.15.83:0/875360370 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f253000ce40 con 0x7f2538151480 2024-09-19T07:42:02.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.578+0000 7f2526ffd640 1 -- 172.21.15.83:0/875360370 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253000a880 con 0x7f2538151480 2024-09-19T07:42:02.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.578+0000 7f253cf67640 1 -- 172.21.15.83:0/875360370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2538153940 con 0x7f2538151480 2024-09-19T07:42:02.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.578+0000 7f253cf67640 1 -- 172.21.15.83:0/875360370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25381525f0 con 0x7f2538151480 2024-09-19T07:42:02.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.578+0000 7f253cf67640 1 -- 172.21.15.83:0/875360370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25381545e0 con 0x7f2538151480 2024-09-19T07:42:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.578+0000 7f2526ffd640 1 -- 172.21.15.83:0/875360370 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f253000aa20 con 0x7f2538151480 2024-09-19T07:42:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.578+0000 7f2526ffd640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f25180678c0 0x7f2518069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.578+0000 7f253659c640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f25180678c0 0x7f2518069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.578+0000 7f2526ffd640 1 -- 172.21.15.83:0/875360370 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9303+0+0 (secure 0 0 0) 0x7f25300860e0 con 0x7f2538151480 2024-09-19T07:42:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.578+0000 7f253659c640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f25180678c0 0x7f2518069d80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f252c00b9d0 tx=0x7f252c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:02.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.582+0000 7f2526ffd640 1 -- 172.21.15.83:0/875360370 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2530055dc0 con 0x7f2538151480 2024-09-19T07:42:02.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.782+0000 7f253cf67640 1 -- 172.21.15.83:0/875360370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25380c4f30 con 0x7f2538151480 2024-09-19T07:42:02.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.782+0000 7f2526ffd640 1 -- 172.21.15.83:0/875360370 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+6545 (secure 0 0 0) 0x7f25300556c0 con 0x7f2538151480 2024-09-19T07:42:02.781 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:02.781 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":160,"btime":"2024-09-19T07:41:46:425682+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:41:36.478574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5951":{"gid":5951,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5951,"qdb_cluster":[5951]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:02.781 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 160 2024-09-19T07:42:02.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.786+0000 7f253cf67640 1 -- 172.21.15.83:0/875360370 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f25180678c0 msgr2=0x7f2518069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:02.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.786+0000 7f253cf67640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f25180678c0 0x7f2518069d80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f252c00b9d0 tx=0x7f252c008040 comp rx=0 tx=0).stop 2024-09-19T07:42:02.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.786+0000 7f253cf67640 1 -- 172.21.15.83:0/875360370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2538151480 msgr2=0x7f25380c5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:02.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.786+0000 7f253cf67640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2538151480 0x7f25380c5730 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f253000c730 tx=0x7f253000cbf0 comp rx=0 tx=0).stop 2024-09-19T07:42:02.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.786+0000 7f253cf67640 1 -- 172.21.15.83:0/875360370 shutdown_connections 2024-09-19T07:42:02.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.786+0000 7f253cf67640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f25180678c0 0x7f2518069d80 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:02.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.786+0000 7f253cf67640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2538151480 0x7f25380c5730 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:02.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.786+0000 7f253cf67640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25380c28e0 0x7f2538148cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:02.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.786+0000 7f253cf67640 1 --2- 172.21.15.83:0/875360370 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2538071100 0x7f2538153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:02.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.786+0000 7f253cf67640 1 -- 172.21.15.83:0/875360370 >> 172.21.15.83:0/875360370 conn(0x7f253806bc80 msgr2=0x7f253814f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:02.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.786+0000 7f253cf67640 1 -- 172.21.15.83:0/875360370 shutdown_connections 2024-09-19T07:42:02.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:02.786+0000 7f253cf67640 1 -- 172.21.15.83:0/875360370 wait complete. 2024-09-19T07:42:02.800 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:42:04.801 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:04.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff245569640 1 Processor -- start 2024-09-19T07:42:04.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff245569640 1 -- start start 2024-09-19T07:42:04.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff245569640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2400c42e0 0x7ff24006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:04.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff245569640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2401566b0 0x7ff2400abb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:04.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff245569640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2401592c0 con 0x7ff240159a80 2024-09-19T07:42:04.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff245569640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff2400ac070 con 0x7ff2400c42e0 2024-09-19T07:42:04.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff245569640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff2401595f0 con 0x7ff2401566b0 2024-09-19T07:42:04.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff23f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff240159a80 0x7ff240153060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56926/0 (socket says 172.21.15.83:56926) 2024-09-19T07:42:04.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff23effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2400c42e0 0x7ff24006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:04.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff23f7fe640 1 -- 172.21.15.83:0/2902069867 learned_addr learned my addr 172.21.15.83:0/2902069867 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:42:04.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff23e7fc640 1 --2- 172.21.15.83:0/2902069867 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2401566b0 0x7ff2400abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:04.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff23dffb640 1 -- 172.21.15.83:0/2902069867 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2507034704 0 0) 0x7ff2401592c0 con 0x7ff240159a80 2024-09-19T07:42:04.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff23dffb640 1 -- 172.21.15.83:0/2902069867 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff22c003530 con 0x7ff240159a80 2024-09-19T07:42:04.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff23effd640 1 -- 172.21.15.83:0/2902069867 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2401566b0 msgr2=0x7ff2400abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:04.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff23effd640 1 --2- 172.21.15.83:0/2902069867 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2401566b0 0x7ff2400abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:04.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff23effd640 1 -- 172.21.15.83:0/2902069867 >> v1:172.21.15.83:6789/0 conn(0x7ff240159a80 legacy=0x7ff240153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:04.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.934+0000 7ff23effd640 1 -- 172.21.15.83:0/2902069867 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24006e900 con 0x7ff2400c42e0 2024-09-19T07:42:04.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.938+0000 7ff23effd640 1 --2- 172.21.15.83:0/2902069867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2400c42e0 0x7ff24006e270 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7ff2280097c0 tx=0x7ff22801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1141fed2c279f559 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:04.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.938+0000 7ff23dffb640 1 -- 172.21.15.83:0/2902069867 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2280aa5f0 con 0x7ff2400c42e0 2024-09-19T07:42:04.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.938+0000 7ff23dffb640 1 -- 172.21.15.83:0/2902069867 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2280af070 con 0x7ff2400c42e0 2024-09-19T07:42:04.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.938+0000 7ff23dffb640 1 -- 172.21.15.83:0/2902069867 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2280aa9b0 con 0x7ff2400c42e0 2024-09-19T07:42:04.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 -- 172.21.15.83:0/2902069867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2400c42e0 msgr2=0x7ff24006e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 --2- 172.21.15.83:0/2902069867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2400c42e0 0x7ff24006e270 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7ff2280097c0 tx=0x7ff22801c470 comp rx=0 tx=0).stop 2024-09-19T07:42:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 -- 172.21.15.83:0/2902069867 shutdown_connections 2024-09-19T07:42:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 --2- 172.21.15.83:0/2902069867 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2401566b0 0x7ff2400abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 --2- 172.21.15.83:0/2902069867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2400c42e0 0x7ff24006e270 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:04.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 -- 172.21.15.83:0/2902069867 >> 172.21.15.83:0/2902069867 conn(0x7ff24006bc80 msgr2=0x7ff24006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:04.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 -- 172.21.15.83:0/2902069867 shutdown_connections 2024-09-19T07:42:04.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 -- 172.21.15.83:0/2902069867 wait complete. 2024-09-19T07:42:04.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 Processor -- start 2024-09-19T07:42:04.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 -- start start 2024-09-19T07:42:04.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2400c42e0 0x7ff2400c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:04.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2401566b0 0x7ff240072b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:04.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff240159a80 0x7ff24014d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:04.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401538f0 con 0x7ff240159a80 2024-09-19T07:42:04.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff240153770 con 0x7ff2400c42e0 2024-09-19T07:42:04.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff2400c46c0 con 0x7ff2401566b0 2024-09-19T07:42:04.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff23f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff240159a80 0x7ff24014d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:04.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff23f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff240159a80 0x7ff24014d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60070/0 (socket says 172.21.15.83:60070) 2024-09-19T07:42:04.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff23f7fe640 1 -- 172.21.15.83:0/758921359 learned_addr learned my addr 172.21.15.83:0/758921359 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:04.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff23e7fc640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2401566b0 0x7ff240072b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:04.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff23effd640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2400c42e0 0x7ff2400c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:04.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff23f7fe640 1 -- 172.21.15.83:0/758921359 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2401566b0 msgr2=0x7ff240072b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff23f7fe640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2401566b0 0x7ff240072b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff23f7fe640 1 -- 172.21.15.83:0/758921359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2400c42e0 msgr2=0x7ff2400c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff23f7fe640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2400c42e0 0x7ff2400c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff23f7fe640 1 -- 172.21.15.83:0/758921359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24014e4f0 con 0x7ff240159a80 2024-09-19T07:42:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff23effd640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2400c42e0 0x7ff2400c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:42:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff23f7fe640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff240159a80 0x7ff24014d820 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7ff23800c450 tx=0x7ff23800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff2377fe640 1 -- 172.21.15.83:0/758921359 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238017070 con 0x7ff240159a80 2024-09-19T07:42:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff2377fe640 1 -- 172.21.15.83:0/758921359 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff238005cf0 con 0x7ff240159a80 2024-09-19T07:42:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 -- 172.21.15.83:0/758921359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24014e7d0 con 0x7ff240159a80 2024-09-19T07:42:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff2377fe640 1 -- 172.21.15.83:0/758921359 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2380072b0 con 0x7ff240159a80 2024-09-19T07:42:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 -- 172.21.15.83:0/758921359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2400c7d60 con 0x7ff240159a80 2024-09-19T07:42:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.942+0000 7ff245569640 1 -- 172.21.15.83:0/758921359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff240159e90 con 0x7ff240159a80 2024-09-19T07:42:04.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.946+0000 7ff2377fe640 1 -- 172.21.15.83:0/758921359 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff23801a070 con 0x7ff240159a80 2024-09-19T07:42:04.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.946+0000 7ff2377fe640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2200678c0 0x7ff220069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:04.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.946+0000 7ff2377fe640 1 -- 172.21.15.83:0/758921359 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9303+0+0 (secure 0 0 0) 0x7ff238085d80 con 0x7ff240159a80 2024-09-19T07:42:04.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.946+0000 7ff23effd640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2200678c0 0x7ff220069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:04.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.946+0000 7ff23effd640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2200678c0 0x7ff220069d80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff2280ae000 tx=0x7ff2280a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:04.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:04.950+0000 7ff2377fe640 1 -- 172.21.15.83:0/758921359 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff238055af0 con 0x7ff240159a80 2024-09-19T07:42:05.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.150+0000 7ff245569640 1 -- 172.21.15.83:0/758921359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff24006e050 con 0x7ff240159a80 2024-09-19T07:42:05.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff2377fe640 1 -- 172.21.15.83:0/758921359 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+6545 (secure 0 0 0) 0x7ff2380553f0 con 0x7ff240159a80 2024-09-19T07:42:05.151 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:05.151 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":160,"btime":"2024-09-19T07:41:46:425682+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:41:36.478574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5951":{"gid":5951,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5951,"qdb_cluster":[5951]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:05.151 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 160 2024-09-19T07:42:05.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff245569640 1 -- 172.21.15.83:0/758921359 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2200678c0 msgr2=0x7ff220069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:05.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff245569640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2200678c0 0x7ff220069d80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff2280ae000 tx=0x7ff2280a7000 comp rx=0 tx=0).stop 2024-09-19T07:42:05.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff245569640 1 -- 172.21.15.83:0/758921359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff240159a80 msgr2=0x7ff24014d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:05.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff245569640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff240159a80 0x7ff24014d820 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7ff23800c450 tx=0x7ff23800c910 comp rx=0 tx=0).stop 2024-09-19T07:42:05.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff245569640 1 -- 172.21.15.83:0/758921359 shutdown_connections 2024-09-19T07:42:05.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff245569640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2200678c0 0x7ff220069d80 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:05.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff245569640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff240159a80 0x7ff24014d820 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:05.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff245569640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2401566b0 0x7ff240072b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:05.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff245569640 1 --2- 172.21.15.83:0/758921359 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2400c42e0 0x7ff2400c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:05.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff245569640 1 -- 172.21.15.83:0/758921359 >> 172.21.15.83:0/758921359 conn(0x7ff24006bc80 msgr2=0x7ff240155ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:05.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff245569640 1 -- 172.21.15.83:0/758921359 shutdown_connections 2024-09-19T07:42:05.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:05.154+0000 7ff245569640 1 -- 172.21.15.83:0/758921359 wait complete. 2024-09-19T07:42:05.171 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:42:06.515 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:42:06.518 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:42:07.171 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:07.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.302+0000 7f6a20216640 1 Processor -- start 2024-09-19T07:42:07.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.302+0000 7f6a20216640 1 -- start start 2024-09-19T07:42:07.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a20216640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6a181545e0 0x7f6a1806e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:07.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a20216640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a18154a70 con 0x7f6a180c28e0 2024-09-19T07:42:07.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a20216640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a180a9e60 con 0x7f6a18071100 2024-09-19T07:42:07.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a20216640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6a18071b50 con 0x7f6a181545e0 2024-09-19T07:42:07.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a1e7b3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6a181545e0 0x7f6a1806e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:07.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a1e7b3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6a181545e0 0x7f6a1806e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60084/0 (socket says 172.21.15.83:60084) 2024-09-19T07:42:07.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a1e7b3640 1 -- 172.21.15.83:0/175432528 learned_addr learned my addr 172.21.15.83:0/175432528 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:07.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a1d7b1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6a180c28e0 0x7f6a1806dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56930/0 (socket says 172.21.15.83:56930) 2024-09-19T07:42:07.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a1e7b3640 1 -- 172.21.15.83:0/175432528 >> v1:172.21.15.184:6789/0 conn(0x7f6a18071100 legacy=0x7f6a18150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:07.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a1e7b3640 1 -- 172.21.15.83:0/175432528 >> v1:172.21.15.83:6789/0 conn(0x7f6a180c28e0 legacy=0x7f6a1806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:07.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a1e7b3640 1 -- 172.21.15.83:0/175432528 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a180aa140 con 0x7f6a181545e0 2024-09-19T07:42:07.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a1cfb0640 1 -- 172.21.15.83:0/175432528 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4251112147 0 0) 0x7f6a18154a70 con 0x7f6a180c28e0 2024-09-19T07:42:07.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a1e7b3640 1 --2- 172.21.15.83:0/175432528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a181545e0 0x7f6a1806e5b0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6a10009e50 tx=0x7f6a1001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a432865f5163c13 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:07.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a1cfb0640 1 -- 172.21.15.83:0/175432528 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a100a5750 con 0x7f6a181545e0 2024-09-19T07:42:07.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a1cfb0640 1 -- 172.21.15.83:0/175432528 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a100a58f0 con 0x7f6a181545e0 2024-09-19T07:42:07.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.306+0000 7f6a1cfb0640 1 -- 172.21.15.83:0/175432528 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a100a5cc0 con 0x7f6a181545e0 2024-09-19T07:42:07.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 -- 172.21.15.83:0/175432528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a181545e0 msgr2=0x7f6a1806e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:07.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 --2- 172.21.15.83:0/175432528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a181545e0 0x7f6a1806e5b0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6a10009e50 tx=0x7f6a1001c9a0 comp rx=0 tx=0).stop 2024-09-19T07:42:07.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 -- 172.21.15.83:0/175432528 shutdown_connections 2024-09-19T07:42:07.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 --2- 172.21.15.83:0/175432528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a181545e0 0x7f6a1806e5b0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:07.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 -- 172.21.15.83:0/175432528 >> 172.21.15.83:0/175432528 conn(0x7f6a1806bc80 msgr2=0x7f6a1806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:07.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 -- 172.21.15.83:0/175432528 shutdown_connections 2024-09-19T07:42:07.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 -- 172.21.15.83:0/175432528 wait complete. 2024-09-19T07:42:07.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 Processor -- start 2024-09-19T07:42:07.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 -- start start 2024-09-19T07:42:07.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a18071100 0x7f6a180c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:07.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a180c28e0 0x7f6a181409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:07.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a181545e0 0x7f6a18140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:07.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181517d0 con 0x7f6a181545e0 2024-09-19T07:42:07.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6a18151650 con 0x7f6a18071100 2024-09-19T07:42:07.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6a180714e0 con 0x7f6a180c28e0 2024-09-19T07:42:07.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a1e7b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a181545e0 0x7f6a18140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:07.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a1e7b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a181545e0 0x7f6a18140f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60092/0 (socket says 172.21.15.83:60092) 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a1e7b3640 1 -- 172.21.15.83:0/2238711534 learned_addr learned my addr 172.21.15.83:0/2238711534 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a1dfb2640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a18071100 0x7f6a180c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a1d7b1640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a180c28e0 0x7f6a181409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a1e7b3640 1 -- 172.21.15.83:0/2238711534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a180c28e0 msgr2=0x7f6a181409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a1e7b3640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a180c28e0 0x7f6a181409d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a1e7b3640 1 -- 172.21.15.83:0/2238711534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a18071100 msgr2=0x7f6a180c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a1e7b3640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a18071100 0x7f6a180c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a1e7b3640 1 -- 172.21.15.83:0/2238711534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a18141450 con 0x7f6a181545e0 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a1e7b3640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a181545e0 0x7f6a18140f10 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f6a100af040 tx=0x7f6a10019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a0e7fc640 1 -- 172.21.15.83:0/2238711534 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a100bc070 con 0x7f6a181545e0 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a0e7fc640 1 -- 172.21.15.83:0/2238711534 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a100b8680 con 0x7f6a181545e0 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 -- 172.21.15.83:0/2238711534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a180c1df0 con 0x7f6a181545e0 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a0e7fc640 1 -- 172.21.15.83:0/2238711534 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a100b8a70 con 0x7f6a181545e0 2024-09-19T07:42:07.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.310+0000 7f6a20216640 1 -- 172.21.15.83:0/2238711534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a181efe00 con 0x7f6a181545e0 2024-09-19T07:42:07.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.314+0000 7f6a20216640 1 -- 172.21.15.83:0/2238711534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a181efc10 con 0x7f6a181545e0 2024-09-19T07:42:07.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.314+0000 7f6a0e7fc640 1 -- 172.21.15.83:0/2238711534 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6a100b8c10 con 0x7f6a181545e0 2024-09-19T07:42:07.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.314+0000 7f6a0e7fc640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69fc0678c0 0x7f69fc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:07.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.314+0000 7f6a1dfb2640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69fc0678c0 0x7f69fc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:07.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.314+0000 7f6a0e7fc640 1 -- 172.21.15.83:0/2238711534 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9303+0+0 (secure 0 0 0) 0x7f6a10121cb0 con 0x7f6a181545e0 2024-09-19T07:42:07.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.314+0000 7f6a1dfb2640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69fc0678c0 0x7f69fc069d80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6a08000f70 tx=0x7f6a08009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:07.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.318+0000 7f6a0e7fc640 1 -- 172.21.15.83:0/2238711534 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6a100c6020 con 0x7f6a181545e0 2024-09-19T07:42:07.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.522+0000 7f6a20216640 1 -- 172.21.15.83:0/2238711534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a180c4f30 con 0x7f6a181545e0 2024-09-19T07:42:07.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.522+0000 7f6a0e7fc640 1 -- 172.21.15.83:0/2238711534 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+6545 (secure 0 0 0) 0x7f6a10016620 con 0x7f6a181545e0 2024-09-19T07:42:07.521 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:07.521 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":160,"btime":"2024-09-19T07:41:46:425682+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:41:36.478574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5951":{"gid":5951,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.83:6835/2776269095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2776269095},{"type":"v1","addr":"172.21.15.83:6835","nonce":2776269095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5951,"qdb_cluster":[5951]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:07.521 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 160 2024-09-19T07:42:07.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.526+0000 7f6a20216640 1 -- 172.21.15.83:0/2238711534 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69fc0678c0 msgr2=0x7f69fc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:07.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.526+0000 7f6a20216640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69fc0678c0 0x7f69fc069d80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6a08000f70 tx=0x7f6a08009040 comp rx=0 tx=0).stop 2024-09-19T07:42:07.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.526+0000 7f6a20216640 1 -- 172.21.15.83:0/2238711534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a181545e0 msgr2=0x7f6a18140f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:07.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.526+0000 7f6a20216640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a181545e0 0x7f6a18140f10 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f6a100af040 tx=0x7f6a10019740 comp rx=0 tx=0).stop 2024-09-19T07:42:07.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.526+0000 7f6a20216640 1 -- 172.21.15.83:0/2238711534 shutdown_connections 2024-09-19T07:42:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.526+0000 7f6a20216640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f69fc0678c0 0x7f69fc069d80 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.526+0000 7f6a20216640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a181545e0 0x7f6a18140f10 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.526+0000 7f6a20216640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a180c28e0 0x7f6a181409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.526+0000 7f6a20216640 1 --2- 172.21.15.83:0/2238711534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a18071100 0x7f6a180c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.526+0000 7f6a20216640 1 -- 172.21.15.83:0/2238711534 >> 172.21.15.83:0/2238711534 conn(0x7f6a1806bc80 msgr2=0x7f6a1814f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.526+0000 7f6a20216640 1 -- 172.21.15.83:0/2238711534 shutdown_connections 2024-09-19T07:42:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:07.526+0000 7f6a20216640 1 -- 172.21.15.83:0/2238711534 wait complete. 2024-09-19T07:42:07.539 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:42:09.541 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:09.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.678+0000 7f8ea1ed8640 1 Processor -- start 2024-09-19T07:42:09.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.678+0000 7f8ea1ed8640 1 -- start start 2024-09-19T07:42:09.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8ea1ed8640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f8e9c155d70 0x7f8e9c0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:09.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8ea1ed8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8e9c1592f0 0x7f8e9c157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:09.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8ea1ed8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e9c156200 con 0x7f8e9c0c6120 2024-09-19T07:42:09.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8ea1ed8640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f8e9c0c9930 con 0x7f8e9c1592f0 2024-09-19T07:42:09.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8ea1ed8640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f8e9c1564e0 con 0x7f8e9c155d70 2024-09-19T07:42:09.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8e9b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8e9c0c6120 0x7f8e9c1575d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56936/0 (socket says 172.21.15.83:56936) 2024-09-19T07:42:09.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8e9b7fe640 1 -- 172.21.15.83:0/816882860 learned_addr learned my addr 172.21.15.83:0/816882860 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:42:09.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8e9bfff640 1 --2- 172.21.15.83:0/816882860 >> v2:172.21.15.83:3300/0 conn(0x7f8e9c1592f0 0x7f8e9c157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:09.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8e92ffd640 1 --2- 172.21.15.83:0/816882860 >> v2:172.21.15.184:3300/0 conn(0x7f8e9c155d70 0x7f8e9c0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:09.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8e9bfff640 1 -- 172.21.15.83:0/816882860 >> v2:172.21.15.184:3300/0 conn(0x7f8e9c155d70 msgr2=0x7f8e9c0c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:09.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8e9bfff640 1 --2- 172.21.15.83:0/816882860 >> v2:172.21.15.184:3300/0 conn(0x7f8e9c155d70 0x7f8e9c0c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:09.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8e9bfff640 1 -- 172.21.15.83:0/816882860 >> v1:172.21.15.83:6789/0 conn(0x7f8e9c0c6120 legacy=0x7f8e9c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:09.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8e9affd640 1 -- 172.21.15.83:0/816882860 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3251332189 0 0) 0x7f8e9c156200 con 0x7f8e9c0c6120 2024-09-19T07:42:09.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8e9bfff640 1 -- 172.21.15.83:0/816882860 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9c1567c0 con 0x7f8e9c1592f0 2024-09-19T07:42:09.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.682+0000 7f8e9bfff640 1 --2- 172.21.15.83:0/816882860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c1592f0 0x7f8e9c157ce0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c06cca0 tx=0x7f8e9401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde287df4f057512 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:09.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e9affd640 1 -- 172.21.15.83:0/816882860 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e940b2070 con 0x7f8e9c1592f0 2024-09-19T07:42:09.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e9affd640 1 -- 172.21.15.83:0/816882860 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e940a57f0 con 0x7f8e9c1592f0 2024-09-19T07:42:09.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e9affd640 1 -- 172.21.15.83:0/816882860 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e940a5bc0 con 0x7f8e9c1592f0 2024-09-19T07:42:09.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/816882860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c1592f0 msgr2=0x7f8e9c157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:09.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 --2- 172.21.15.83:0/816882860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c1592f0 0x7f8e9c157ce0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c06cca0 tx=0x7f8e9401c930 comp rx=0 tx=0).stop 2024-09-19T07:42:09.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/816882860 shutdown_connections 2024-09-19T07:42:09.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 --2- 172.21.15.83:0/816882860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c1592f0 0x7f8e9c157ce0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:09.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 --2- 172.21.15.83:0/816882860 >> v2:172.21.15.184:3300/0 conn(0x7f8e9c155d70 0x7f8e9c0c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:09.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/816882860 >> 172.21.15.83:0/816882860 conn(0x7f8e9c06bc80 msgr2=0x7f8e9c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:09.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/816882860 shutdown_connections 2024-09-19T07:42:09.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/816882860 wait complete. 2024-09-19T07:42:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 Processor -- start 2024-09-19T07:42:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 -- start start 2024-09-19T07:42:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e9c0c6120 0x7f8e9c0ac5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c1592f0 0x7f8e9c1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e9c1f1910 0x7f8e9c1f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9c153860 con 0x7f8e9c1592f0 2024-09-19T07:42:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9c162ea0 con 0x7f8e9c0c6120 2024-09-19T07:42:09.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8ea1ed8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8e9c14cd90 con 0x7f8e9c1f1910 2024-09-19T07:42:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e92ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c1592f0 0x7f8e9c1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e92ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c1592f0 0x7f8e9c1549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60110/0 (socket says 172.21.15.83:60110) 2024-09-19T07:42:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e92ffd640 1 -- 172.21.15.83:0/4016427721 learned_addr learned my addr 172.21.15.83:0/4016427721 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e9b7fe640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e9c0c6120 0x7f8e9c0ac5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e9bfff640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e9c1f1910 0x7f8e9c1f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e92ffd640 1 -- 172.21.15.83:0/4016427721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e9c1f1910 msgr2=0x7f8e9c1f1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e92ffd640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e9c1f1910 0x7f8e9c1f1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e92ffd640 1 -- 172.21.15.83:0/4016427721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e9c0c6120 msgr2=0x7f8e9c0ac5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e92ffd640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e9c0c6120 0x7f8e9c0ac5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:09.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e92ffd640 1 -- 172.21.15.83:0/4016427721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9c14cad0 con 0x7f8e9c1592f0 2024-09-19T07:42:09.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e9b7fe640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e9c0c6120 0x7f8e9c0ac5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:42:09.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e92ffd640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c1592f0 0x7f8e9c1549b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f8e8400d800 tx=0x7f8e8400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:09.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e98ff9640 1 -- 172.21.15.83:0/4016427721 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8400b6c0 con 0x7f8e9c1592f0 2024-09-19T07:42:09.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e98ff9640 1 -- 172.21.15.83:0/4016427721 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e84019070 con 0x7f8e9c1592f0 2024-09-19T07:42:09.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.686+0000 7f8e98ff9640 1 -- 172.21.15.83:0/4016427721 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8400ba60 con 0x7f8e9c1592f0 2024-09-19T07:42:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.690+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/4016427721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e9c0ada70 con 0x7f8e9c1592f0 2024-09-19T07:42:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.690+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/4016427721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e9c155430 con 0x7f8e9c1592f0 2024-09-19T07:42:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.690+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/4016427721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e9c0c6500 con 0x7f8e9c1592f0 2024-09-19T07:42:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.690+0000 7f8e98ff9640 1 -- 172.21.15.83:0/4016427721 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8e8400bc00 con 0x7f8e9c1592f0 2024-09-19T07:42:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.690+0000 7f8e98ff9640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8e7c0678c0 0x7f8e7c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.690+0000 7f8e98ff9640 1 -- 172.21.15.83:0/4016427721 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9389+0+0 (secure 0 0 0) 0x7f8e84087000 con 0x7f8e9c1592f0 2024-09-19T07:42:09.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.690+0000 7f8e9b7fe640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8e7c0678c0 0x7f8e7c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:09.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.690+0000 7f8e9b7fe640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8e7c0678c0 0x7f8e7c069d80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c007fd0 tx=0x7f8e8c007c20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:09.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.694+0000 7f8e98ff9640 1 -- 172.21.15.83:0/4016427721 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f8e84056cc0 con 0x7f8e9c1592f0 2024-09-19T07:42:09.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.898+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/4016427721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e9c0c96a0 con 0x7f8e9c1592f0 2024-09-19T07:42:09.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.898+0000 7f8e98ff9640 1 -- 172.21.15.83:0/4016427721 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+5715 (secure 0 0 0) 0x7f8e840565c0 con 0x7f8e9c1592f0 2024-09-19T07:42:09.897 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:09.898 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":162,"btime":"2024-09-19T07:42:09:429246+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:08.803139+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:09.898 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 162 2024-09-19T07:42:09.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.902+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/4016427721 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8e7c0678c0 msgr2=0x7f8e7c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:09.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.902+0000 7f8ea1ed8640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8e7c0678c0 0x7f8e7c069d80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c007fd0 tx=0x7f8e8c007c20 comp rx=0 tx=0).stop 2024-09-19T07:42:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.902+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/4016427721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c1592f0 msgr2=0x7f8e9c1549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.902+0000 7f8ea1ed8640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c1592f0 0x7f8e9c1549b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f8e8400d800 tx=0x7f8e8400dcc0 comp rx=0 tx=0).stop 2024-09-19T07:42:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.902+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/4016427721 shutdown_connections 2024-09-19T07:42:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.902+0000 7f8ea1ed8640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8e7c0678c0 0x7f8e7c069d80 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.902+0000 7f8ea1ed8640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e9c1f1910 0x7f8e9c1f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.902+0000 7f8ea1ed8640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e9c1592f0 0x7f8e9c1549b0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.902+0000 7f8ea1ed8640 1 --2- 172.21.15.83:0/4016427721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e9c0c6120 0x7f8e9c0ac5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:09.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.902+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/4016427721 >> 172.21.15.83:0/4016427721 conn(0x7f8e9c06bc80 msgr2=0x7f8e9c153b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:09.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.902+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/4016427721 shutdown_connections 2024-09-19T07:42:09.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:09.902+0000 7f8ea1ed8640 1 -- 172.21.15.83:0/4016427721 wait complete. 2024-09-19T07:42:09.917 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-19T07:42:09.917 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:42:09.917 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:10.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2f80d640 1 Processor -- start 2024-09-19T07:42:10.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2f80d640 1 -- start start 2024-09-19T07:42:10.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2f80d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7e28071100 0x7f7e281545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:10.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2f80d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e2806dea0 0x7f7e2806e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:10.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2f80d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e280a9e60 con 0x7f7e280c28e0 2024-09-19T07:42:10.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2f80d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f7e28071b50 con 0x7f7e28071100 2024-09-19T07:42:10.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2f80d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7e280c1ff0 con 0x7f7e2806dea0 2024-09-19T07:42:10.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2d5a9640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7e28071100 0x7f7e281545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:10.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2ddaa640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e2806dea0 0x7f7e2806e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:10.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2cda8640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f7e280c28e0 0x7f7e28150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:41648/0 (socket says 172.21.15.83:41648) 2024-09-19T07:42:10.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2ddaa640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e2806dea0 0x7f7e2806e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:58890/0 (socket says 172.21.15.83:58890) 2024-09-19T07:42:10.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2d5a9640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7e28071100 0x7f7e281545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:51746/0 (socket says 172.21.15.83:51746) 2024-09-19T07:42:10.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2ddaa640 1 -- 172.21.15.83:0/1200138672 learned_addr learned my addr 172.21.15.83:0/1200138672 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:10.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2ddaa640 1 -- 172.21.15.83:0/1200138672 >> v2:172.21.15.184:3300/0 conn(0x7f7e28071100 msgr2=0x7f7e281545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:10.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2ddaa640 1 --2- 172.21.15.83:0/1200138672 >> v2:172.21.15.184:3300/0 conn(0x7f7e28071100 0x7f7e281545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:10.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2ddaa640 1 -- 172.21.15.83:0/1200138672 >> v1:172.21.15.184:6789/0 conn(0x7f7e280c28e0 legacy=0x7f7e28150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:10.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2ddaa640 1 -- 172.21.15.83:0/1200138672 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e280aa140 con 0x7f7e2806dea0 2024-09-19T07:42:10.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e1ffff640 1 -- 172.21.15.83:0/1200138672 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 391247961 0 0) 0x7f7e280a9e60 con 0x7f7e280c28e0 2024-09-19T07:42:10.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.042+0000 7f7e2d5a9640 1 --2- 172.21.15.83:0/1200138672 >> v2:172.21.15.184:3300/0 conn(0x7f7e28071100 0x7f7e281545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:42:10.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2ddaa640 1 --2- 172.21.15.83:0/1200138672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e2806dea0 0x7f7e2806e330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009e50 tx=0x7f7e2001c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b24c40bc8f2df011 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:10.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e1ffff640 1 -- 172.21.15.83:0/1200138672 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e200a5750 con 0x7f7e2806dea0 2024-09-19T07:42:10.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e1ffff640 1 -- 172.21.15.83:0/1200138672 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e200a58f0 con 0x7f7e2806dea0 2024-09-19T07:42:10.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e1ffff640 1 -- 172.21.15.83:0/1200138672 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e200a5cc0 con 0x7f7e2806dea0 2024-09-19T07:42:10.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 -- 172.21.15.83:0/1200138672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e2806dea0 msgr2=0x7f7e2806e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:10.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 --2- 172.21.15.83:0/1200138672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e2806dea0 0x7f7e2806e330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009e50 tx=0x7f7e2001c9a0 comp rx=0 tx=0).stop 2024-09-19T07:42:10.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 -- 172.21.15.83:0/1200138672 shutdown_connections 2024-09-19T07:42:10.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 --2- 172.21.15.83:0/1200138672 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e2806dea0 0x7f7e2806e330 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:10.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 --2- 172.21.15.83:0/1200138672 >> v2:172.21.15.184:3300/0 conn(0x7f7e28071100 0x7f7e281545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:10.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 -- 172.21.15.83:0/1200138672 >> 172.21.15.83:0/1200138672 conn(0x7f7e2806bc80 msgr2=0x7f7e2806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:10.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 -- 172.21.15.83:0/1200138672 shutdown_connections 2024-09-19T07:42:10.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 -- 172.21.15.83:0/1200138672 wait complete. 2024-09-19T07:42:10.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 Processor -- start 2024-09-19T07:42:10.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 -- start start 2024-09-19T07:42:10.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e2806dea0 0x7f7e280ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:10.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e28071100 0x7f7e2815c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:10.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e280c28e0 0x7f7e281f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:10.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28071660 con 0x7f7e28071100 2024-09-19T07:42:10.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7e280714e0 con 0x7f7e2806dea0 2024-09-19T07:42:10.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2f80d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7e28160c60 con 0x7f7e280c28e0 2024-09-19T07:42:10.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2cda8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e28071100 0x7f7e2815c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:10.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2cda8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e28071100 0x7f7e2815c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60120/0 (socket says 172.21.15.83:60120) 2024-09-19T07:42:10.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2cda8640 1 -- 172.21.15.83:0/766232470 learned_addr learned my addr 172.21.15.83:0/766232470 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:10.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2d5a9640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e2806dea0 0x7f7e280ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:10.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.046+0000 7f7e2ddaa640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e280c28e0 0x7f7e281f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:10.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e2cda8640 1 -- 172.21.15.83:0/766232470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e280c28e0 msgr2=0x7f7e281f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:10.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e2cda8640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e280c28e0 0x7f7e281f2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:10.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e2cda8640 1 -- 172.21.15.83:0/766232470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e2806dea0 msgr2=0x7f7e280ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:10.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e2cda8640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e2806dea0 0x7f7e280ab490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:10.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e2cda8640 1 -- 172.21.15.83:0/766232470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e280ab9d0 con 0x7f7e28071100 2024-09-19T07:42:10.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e2cda8640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e28071100 0x7f7e2815c870 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800c4b0 tx=0x7f7e1800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:10.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e1dffb640 1 -- 172.21.15.83:0/766232470 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18018070 con 0x7f7e28071100 2024-09-19T07:42:10.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e1dffb640 1 -- 172.21.15.83:0/766232470 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e18007500 con 0x7f7e28071100 2024-09-19T07:42:10.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e2f80d640 1 -- 172.21.15.83:0/766232470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e280c79e0 con 0x7f7e28071100 2024-09-19T07:42:10.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e1dffb640 1 -- 172.21.15.83:0/766232470 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18004a30 con 0x7f7e28071100 2024-09-19T07:42:10.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e2f80d640 1 -- 172.21.15.83:0/766232470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e280c4250 con 0x7f7e28071100 2024-09-19T07:42:10.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e2f80d640 1 -- 172.21.15.83:0/766232470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e280c4f30 con 0x7f7e28071100 2024-09-19T07:42:10.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e1dffb640 1 -- 172.21.15.83:0/766232470 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7e1801b070 con 0x7f7e28071100 2024-09-19T07:42:10.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e1dffb640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7e0c0678f0 0x7f7e0c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:10.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e1dffb640 1 -- 172.21.15.83:0/766232470 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9389+0+0 (secure 0 0 0) 0x7f7e18085cc0 con 0x7f7e28071100 2024-09-19T07:42:10.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e2d5a9640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7e0c0678f0 0x7f7e0c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:10.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.050+0000 7f7e2d5a9640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7e0c0678f0 0x7f7e0c069db0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009580 tx=0x7f7e24009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:10.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.054+0000 7f7e1dffb640 1 -- 172.21.15.83:0/766232470 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7e18055300 con 0x7f7e28071100 2024-09-19T07:42:10.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.254+0000 7f7e2f80d640 1 -- 172.21.15.83:0/766232470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e2814ef00 con 0x7f7e28071100 2024-09-19T07:42:10.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.254+0000 7f7e1dffb640 1 -- 172.21.15.83:0/766232470 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+5715 (secure 0 0 0) 0x7f7e18059600 con 0x7f7e28071100 2024-09-19T07:42:10.254 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:10.254 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":162,"btime":"2024-09-19T07:42:09:429246+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:08.803139+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:10.254 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 162 2024-09-19T07:42:10.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.258+0000 7f7e2f80d640 1 -- 172.21.15.83:0/766232470 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7e0c0678f0 msgr2=0x7f7e0c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.258+0000 7f7e2f80d640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7e0c0678f0 0x7f7e0c069db0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009580 tx=0x7f7e24009190 comp rx=0 tx=0).stop 2024-09-19T07:42:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.258+0000 7f7e2f80d640 1 -- 172.21.15.83:0/766232470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e28071100 msgr2=0x7f7e2815c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.258+0000 7f7e2f80d640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e28071100 0x7f7e2815c870 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800c4b0 tx=0x7f7e1800c970 comp rx=0 tx=0).stop 2024-09-19T07:42:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.258+0000 7f7e2f80d640 1 -- 172.21.15.83:0/766232470 shutdown_connections 2024-09-19T07:42:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.258+0000 7f7e2f80d640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7e0c0678f0 0x7f7e0c069db0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.258+0000 7f7e2f80d640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e280c28e0 0x7f7e281f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.258+0000 7f7e2f80d640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e28071100 0x7f7e2815c870 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.258+0000 7f7e2f80d640 1 --2- 172.21.15.83:0/766232470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e2806dea0 0x7f7e280ab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.258+0000 7f7e2f80d640 1 -- 172.21.15.83:0/766232470 >> 172.21.15.83:0/766232470 conn(0x7f7e2806bc80 msgr2=0x7f7e2814ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.258+0000 7f7e2f80d640 1 -- 172.21.15.83:0/766232470 shutdown_connections 2024-09-19T07:42:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:10.258+0000 7f7e2f80d640 1 -- 172.21.15.83:0/766232470 wait complete. 2024-09-19T07:42:10.273 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:42:09:429246+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 162, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 162, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6565": { "addr": "172.21.15.184:6833/3311721080", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 3311721080, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 3311721080, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6565, "incarnation": 161, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:42:08.803139+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6565 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7503": { "addr": "172.21.15.83:6837/1990119490", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 1990119490, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 1990119490, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7503, "incarnation": 151, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:41:11.437309+0000", "qdb_cluster": [ 7503 ], "qdb_leader": 7503, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7503 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:42:12.275 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:12.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.518+0000 7f2b06efc640 1 Processor -- start 2024-09-19T07:42:12.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.522+0000 7f2b06efc640 1 -- start start 2024-09-19T07:42:12.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.522+0000 7f2b06efc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b000c3290 0x7f2b0006e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:12.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.522+0000 7f2b06efc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b00152d60 con 0x7f2b0006e5e0 2024-09-19T07:42:12.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.522+0000 7f2b06efc640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b001521e0 con 0x7f2b000c2520 2024-09-19T07:42:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.522+0000 7f2b06efc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2b001524c0 con 0x7f2b000c3290 2024-09-19T07:42:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.522+0000 7f2b05499640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2b0006e5e0 0x7f2b0015c6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39774/0 (socket says 172.21.15.83:39774) 2024-09-19T07:42:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.522+0000 7f2b05499640 1 -- 172.21.15.83:0/2330457356 learned_addr learned my addr 172.21.15.83:0/2330457356 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:42:12.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.522+0000 7f2afffff640 1 --2- 172.21.15.83:0/2330457356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b000c3290 0x7f2b0006e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:12.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.522+0000 7f2afffff640 1 -- 172.21.15.83:0/2330457356 >> v1:172.21.15.184:6789/0 conn(0x7f2b000c2520 legacy=0x7f2b0014f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:12.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.522+0000 7f2afffff640 1 -- 172.21.15.83:0/2330457356 >> v1:172.21.15.83:6789/0 conn(0x7f2b0006e5e0 legacy=0x7f2b0015c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:12.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.522+0000 7f2afffff640 1 -- 172.21.15.83:0/2330457356 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b001527a0 con 0x7f2b000c3290 2024-09-19T07:42:12.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.526+0000 7f2afffff640 1 --2- 172.21.15.83:0/2330457356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b000c3290 0x7f2b0006e0a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2af8009dc0 tx=0x7f2af801c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c40f83069c8ee27 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:12.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2aff7fe640 1 -- 172.21.15.83:0/2330457356 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af80b1070 con 0x7f2b000c3290 2024-09-19T07:42:12.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2aff7fe640 1 -- 172.21.15.83:0/2330457356 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2af80a4be0 con 0x7f2b000c3290 2024-09-19T07:42:12.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 -- 172.21.15.83:0/2330457356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b000c3290 msgr2=0x7f2b0006e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:12.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 --2- 172.21.15.83:0/2330457356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b000c3290 0x7f2b0006e0a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2af8009dc0 tx=0x7f2af801c380 comp rx=0 tx=0).stop 2024-09-19T07:42:12.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 -- 172.21.15.83:0/2330457356 shutdown_connections 2024-09-19T07:42:12.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 --2- 172.21.15.83:0/2330457356 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b000c3290 0x7f2b0006e0a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:12.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 -- 172.21.15.83:0/2330457356 >> 172.21.15.83:0/2330457356 conn(0x7f2b00075390 msgr2=0x7f2b0006bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:12.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 -- 172.21.15.83:0/2330457356 shutdown_connections 2024-09-19T07:42:12.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 -- 172.21.15.83:0/2330457356 wait complete. 2024-09-19T07:42:12.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 Processor -- start 2024-09-19T07:42:12.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 -- start start 2024-09-19T07:42:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b0006e5e0 0x7f2b000c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b000c2520 0x7f2b001ed050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b000c3290 0x7f2b001efd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0014fa90 con 0x7f2b000c3290 2024-09-19T07:42:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0014f910 con 0x7f2b0006e5e0 2024-09-19T07:42:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b06efc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2b000aa000 con 0x7f2b000c2520 2024-09-19T07:42:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b05499640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b000c3290 0x7f2b001efd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2afffff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b000c2520 0x7f2b001ed050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:12.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b05499640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b000c3290 0x7f2b001efd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59098/0 (socket says 172.21.15.83:59098) 2024-09-19T07:42:12.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b05499640 1 -- 172.21.15.83:0/3931945092 learned_addr learned my addr 172.21.15.83:0/3931945092 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:12.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.530+0000 7f2b04c98640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b0006e5e0 0x7f2b000c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:12.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.534+0000 7f2b05499640 1 -- 172.21.15.83:0/3931945092 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b000c2520 msgr2=0x7f2b001ed050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:12.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.534+0000 7f2b05499640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b000c2520 0x7f2b001ed050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:12.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.534+0000 7f2b05499640 1 -- 172.21.15.83:0/3931945092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b0006e5e0 msgr2=0x7f2b000c0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:12.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.534+0000 7f2b05499640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b0006e5e0 0x7f2b000c0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:12.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.534+0000 7f2b05499640 1 -- 172.21.15.83:0/3931945092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b00163730 con 0x7f2b000c3290 2024-09-19T07:42:12.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.534+0000 7f2b05499640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b000c3290 0x7f2b001efd50 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009e40 tx=0x7f2af000c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:12.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.534+0000 7f2afd7fa640 1 -- 172.21.15.83:0/3931945092 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af00078f0 con 0x7f2b000c3290 2024-09-19T07:42:12.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.534+0000 7f2afd7fa640 1 -- 172.21.15.83:0/3931945092 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2af0007ab0 con 0x7f2b000c3290 2024-09-19T07:42:12.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.534+0000 7f2afd7fa640 1 -- 172.21.15.83:0/3931945092 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af0009070 con 0x7f2b000c3290 2024-09-19T07:42:12.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.534+0000 7f2b06efc640 1 -- 172.21.15.83:0/3931945092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b000c7350 con 0x7f2b000c3290 2024-09-19T07:42:12.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.534+0000 7f2b06efc640 1 -- 172.21.15.83:0/3931945092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b000abb20 con 0x7f2b000c3290 2024-09-19T07:42:12.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.538+0000 7f2afd7fa640 1 -- 172.21.15.83:0/3931945092 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2af0009210 con 0x7f2b000c3290 2024-09-19T07:42:12.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.538+0000 7f2afd7fa640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2aec067990 0x7f2aec069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:12.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.538+0000 7f2afd7fa640 1 -- 172.21.15.83:0/3931945092 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9389+0+0 (secure 0 0 0) 0x7f2af0088ee0 con 0x7f2b000c3290 2024-09-19T07:42:12.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.542+0000 7f2b04c98640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2aec067990 0x7f2aec069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:12.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.542+0000 7f2b06efc640 1 -- 172.21.15.83:0/3931945092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b000c2900 con 0x7f2b000c3290 2024-09-19T07:42:12.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.546+0000 7f2b04c98640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2aec067990 0x7f2aec069e50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f2af4007fd0 tx=0x7f2af4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:12.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:12.546+0000 7f2afd7fa640 1 -- 172.21.15.83:0/3931945092 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2af00585d0 con 0x7f2b000c3290 2024-09-19T07:42:13.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.018+0000 7f2b06efc640 1 -- 172.21.15.83:0/3931945092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b0014c120 con 0x7f2b000c3290 2024-09-19T07:42:13.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.018+0000 7f2afd7fa640 1 -- 172.21.15.83:0/3931945092 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) ==== 95+0+5719 (secure 0 0 0) 0x7f2af005c8d0 con 0x7f2b000c3290 2024-09-19T07:42:13.017 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:13.017 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":164,"btime":"2024-09-19T07:42:11:447689+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:13.017 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 164 2024-09-19T07:42:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.022+0000 7f2b06efc640 1 -- 172.21.15.83:0/3931945092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2aec067990 msgr2=0x7f2aec069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.026+0000 7f2b06efc640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2aec067990 0x7f2aec069e50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f2af4007fd0 tx=0x7f2af4009040 comp rx=0 tx=0).stop 2024-09-19T07:42:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.026+0000 7f2b06efc640 1 -- 172.21.15.83:0/3931945092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b000c3290 msgr2=0x7f2b001efd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.026+0000 7f2b06efc640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b000c3290 0x7f2b001efd50 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009e40 tx=0x7f2af000c430 comp rx=0 tx=0).stop 2024-09-19T07:42:13.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.026+0000 7f2b06efc640 1 -- 172.21.15.83:0/3931945092 shutdown_connections 2024-09-19T07:42:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.026+0000 7f2b06efc640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2aec067990 0x7f2aec069e50 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.026+0000 7f2b06efc640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b000c3290 0x7f2b001efd50 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.026+0000 7f2b06efc640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b000c2520 0x7f2b001ed050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.026+0000 7f2b06efc640 1 --2- 172.21.15.83:0/3931945092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b0006e5e0 0x7f2b000c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.026+0000 7f2b06efc640 1 -- 172.21.15.83:0/3931945092 >> 172.21.15.83:0/3931945092 conn(0x7f2b00075390 msgr2=0x7f2b0014fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.026+0000 7f2b06efc640 1 -- 172.21.15.83:0/3931945092 shutdown_connections 2024-09-19T07:42:13.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.026+0000 7f2b06efc640 1 -- 172.21.15.83:0/3931945092 wait complete. 2024-09-19T07:42:13.083 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=5951 2024-09-19T07:42:13.083 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.a 2024-09-19T07:42:13.084 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-19T07:42:13.084 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:42:13.084 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:42:13.086 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:42:13.086 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:13.301 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:42:13.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.382+0000 7f7667304640 1 Processor -- start 2024-09-19T07:42:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.382+0000 7f7667304640 1 -- start start 2024-09-19T07:42:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.382+0000 7f7667304640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f76600c3290 0x7f766006e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.382+0000 7f7667304640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7660152d60 con 0x7f76600c2520 2024-09-19T07:42:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.382+0000 7f7667304640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76601521e0 con 0x7f766006e5e0 2024-09-19T07:42:13.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.382+0000 7f7667304640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f76601524c0 con 0x7f76600c3290 2024-09-19T07:42:13.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.382+0000 7f76650a0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f76600c2520 0x7f766014f200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39786/0 (socket says 172.21.15.83:39786) 2024-09-19T07:42:13.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.382+0000 7f76650a0640 1 -- 172.21.15.83:0/4122282692 learned_addr learned my addr 172.21.15.83:0/4122282692 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:42:13.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.382+0000 7f766489f640 1 --2- 172.21.15.83:0/4122282692 >> v2:172.21.15.83:3300/0 conn(0x7f76600c3290 0x7f766006e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:13.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.382+0000 7f766489f640 1 -- 172.21.15.83:0/4122282692 >> v1:172.21.15.184:6789/0 conn(0x7f766006e5e0 legacy=0x7f766015c6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:42:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.382+0000 7f766489f640 1 -- 172.21.15.83:0/4122282692 >> v1:172.21.15.83:6789/0 conn(0x7f76600c2520 legacy=0x7f766014f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.382+0000 7f766489f640 1 -- 172.21.15.83:0/4122282692 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76601527a0 con 0x7f76600c3290 2024-09-19T07:42:13.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.390+0000 7f766489f640 1 --2- 172.21.15.83:0/4122282692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76600c3290 0x7f766006e0a0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f765c00ba50 tx=0x7f765c01ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a0fa3dd8981ec9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.390+0000 7f7657fff640 1 -- 172.21.15.83:0/4122282692 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c0ad070 con 0x7f76600c3290 2024-09-19T07:42:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.390+0000 7f7657fff640 1 -- 172.21.15.83:0/4122282692 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f765c0a7ad0 con 0x7f76600c3290 2024-09-19T07:42:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.390+0000 7f7657fff640 1 -- 172.21.15.83:0/4122282692 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c0b1400 con 0x7f76600c3290 2024-09-19T07:42:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.390+0000 7f7667304640 1 -- 172.21.15.83:0/4122282692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76600c3290 msgr2=0x7f766006e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.390+0000 7f7667304640 1 --2- 172.21.15.83:0/4122282692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76600c3290 0x7f766006e0a0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f765c00ba50 tx=0x7f765c01ea40 comp rx=0 tx=0).stop 2024-09-19T07:42:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.394+0000 7f7667304640 1 -- 172.21.15.83:0/4122282692 shutdown_connections 2024-09-19T07:42:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.394+0000 7f7667304640 1 --2- 172.21.15.83:0/4122282692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76600c3290 0x7f766006e0a0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.394+0000 7f7667304640 1 -- 172.21.15.83:0/4122282692 >> 172.21.15.83:0/4122282692 conn(0x7f7660075390 msgr2=0x7f766006bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:13.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.394+0000 7f7667304640 1 -- 172.21.15.83:0/4122282692 shutdown_connections 2024-09-19T07:42:13.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.394+0000 7f7667304640 1 -- 172.21.15.83:0/4122282692 wait complete. 2024-09-19T07:42:13.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f7667304640 1 Processor -- start 2024-09-19T07:42:13.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f7667304640 1 -- start start 2024-09-19T07:42:13.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f7667304640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766006e5e0 0x7f7660141580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:13.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f7667304640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f76600c2520 0x7f76600712f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:13.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f7667304640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76600c3290 0x7f76600c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:13.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f7667304640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f766014fa90 con 0x7f766006e5e0 2024-09-19T07:42:13.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f7667304640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f766014f910 con 0x7f76600c3290 2024-09-19T07:42:13.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f7667304640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f76600aa000 con 0x7f76600c2520 2024-09-19T07:42:13.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f76650a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766006e5e0 0x7f7660141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:13.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f76650a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766006e5e0 0x7f7660141580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59110/0 (socket says 172.21.15.83:59110) 2024-09-19T07:42:13.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f76650a0640 1 -- 172.21.15.83:0/2483475244 learned_addr learned my addr 172.21.15.83:0/2483475244 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:13.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f76650a0640 1 -- 172.21.15.83:0/2483475244 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f76600c2520 msgr2=0x7f76600712f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:42:13.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f76650a0640 1 --2- 172.21.15.83:0/2483475244 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f76600c2520 0x7f76600712f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:13.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f76650a0640 1 -- 172.21.15.83:0/2483475244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76600c3290 msgr2=0x7f76600c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:13.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f76650a0640 1 --2- 172.21.15.83:0/2483475244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76600c3290 0x7f76600c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:13.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f76650a0640 1 -- 172.21.15.83:0/2483475244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76600c72d0 con 0x7f766006e5e0 2024-09-19T07:42:13.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.398+0000 7f76650a0640 1 --2- 172.21.15.83:0/2483475244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766006e5e0 0x7f7660141580 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f765800c870 tx=0x7f765800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:13.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.402+0000 7f7655ffb640 1 -- 172.21.15.83:0/2483475244 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7658016020 con 0x7f766006e5e0 2024-09-19T07:42:13.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.402+0000 7f7667304640 1 -- 172.21.15.83:0/2483475244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76600c75b0 con 0x7f766006e5e0 2024-09-19T07:42:13.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.402+0000 7f7667304640 1 -- 172.21.15.83:0/2483475244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7660073d20 con 0x7f766006e5e0 2024-09-19T07:42:13.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.402+0000 7f7655ffb640 1 -- 172.21.15.83:0/2483475244 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f765800a9f0 con 0x7f766006e5e0 2024-09-19T07:42:13.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.402+0000 7f7655ffb640 1 -- 172.21.15.83:0/2483475244 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765800ada0 con 0x7f766006e5e0 2024-09-19T07:42:13.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.402+0000 7f7655ffb640 1 -- 172.21.15.83:0/2483475244 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7658019020 con 0x7f766006e5e0 2024-09-19T07:42:13.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.402+0000 7f7655ffb640 1 --2- 172.21.15.83:0/2483475244 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7644067910 0x7f7644069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:13.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.402+0000 7f7655ffb640 1 -- 172.21.15.83:0/2483475244 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9389+0+0 (secure 0 0 0) 0x7f7658086df0 con 0x7f766006e5e0 2024-09-19T07:42:13.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.406+0000 7f766489f640 1 --2- 172.21.15.83:0/2483475244 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7644067910 0x7f7644069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:13.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.406+0000 7f764b7fe640 1 -- 172.21.15.83:0/2483475244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76400052a0 con 0x7f766006e5e0 2024-09-19T07:42:13.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.406+0000 7f766489f640 1 --2- 172.21.15.83:0/2483475244 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7644067910 0x7f7644069dd0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f765c0b2000 tx=0x7f765c0a9070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:13.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.418+0000 7f7655ffb640 1 -- 172.21.15.83:0/2483475244 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7658020020 con 0x7f766006e5e0 2024-09-19T07:42:13.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.986+0000 7f764b7fe640 1 -- 172.21.15.83:0/2483475244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7640005860 con 0x7f766006e5e0 2024-09-19T07:42:13.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.986+0000 7f7655ffb640 1 -- 172.21.15.83:0/2483475244 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) ==== 95+0+6545 (secure 0 0 0) 0x7f7658056b90 con 0x7f766006e5e0 2024-09-19T07:42:13.987 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:13.987 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":165,"btime":"2024-09-19T07:42:13:462305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:13.987 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 165 2024-09-19T07:42:13.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.998+0000 7f764b7fe640 1 -- 172.21.15.83:0/2483475244 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7644067910 msgr2=0x7f7644069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:13.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.998+0000 7f764b7fe640 1 --2- 172.21.15.83:0/2483475244 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7644067910 0x7f7644069dd0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f765c0b2000 tx=0x7f765c0a9070 comp rx=0 tx=0).stop 2024-09-19T07:42:13.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.998+0000 7f764b7fe640 1 -- 172.21.15.83:0/2483475244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766006e5e0 msgr2=0x7f7660141580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:13.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.998+0000 7f764b7fe640 1 --2- 172.21.15.83:0/2483475244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766006e5e0 0x7f7660141580 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f765800c870 tx=0x7f765800cd30 comp rx=0 tx=0).stop 2024-09-19T07:42:13.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.998+0000 7f764b7fe640 1 -- 172.21.15.83:0/2483475244 shutdown_connections 2024-09-19T07:42:13.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:13.998+0000 7f764b7fe640 1 --2- 172.21.15.83:0/2483475244 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7644067910 0x7f7644069dd0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:13.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:14.002+0000 7f764b7fe640 1 --2- 172.21.15.83:0/2483475244 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76600c3290 0x7f76600c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:13.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:14.002+0000 7f764b7fe640 1 --2- 172.21.15.83:0/2483475244 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f76600c2520 0x7f76600712f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:13.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:14.002+0000 7f764b7fe640 1 --2- 172.21.15.83:0/2483475244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766006e5e0 0x7f7660141580 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:13.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:14.002+0000 7f764b7fe640 1 -- 172.21.15.83:0/2483475244 >> 172.21.15.83:0/2483475244 conn(0x7f7660075390 msgr2=0x7f766014ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:14.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:14.002+0000 7f764b7fe640 1 -- 172.21.15.83:0/2483475244 shutdown_connections 2024-09-19T07:42:14.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:14.002+0000 7f764b7fe640 1 -- 172.21.15.83:0/2483475244 wait complete. 2024-09-19T07:42:14.052 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-19T07:42:14.053 INFO:tasks.mds_thrash.fs.[a]:waiting for 111 secs before thrashing 2024-09-19T07:42:16.438 INFO:tasks.ceph.mds.d.smithi184.stderr:2024-09-19T07:42:16.443+0000 7f4f30f78640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:42:24.016 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2e0c90640 1 Processor -- start 2024-09-19T07:42:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2e0c90640 1 -- start start 2024-09-19T07:42:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2e0c90640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc2d400aca0 0x7fc2d400b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2e0c90640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc2d4005b10 0x7fc2d4005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:24.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2e0c90640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d40f7a50 con 0x7fc2d40f5010 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2e0c90640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc2d40f7d30 con 0x7fc2d4005b10 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2e0c90640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc2d40f8010 con 0x7fc2d400aca0 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2db7fe640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fc2d40f5010 0x7fc2d40491a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:36328/0 (socket says 172.21.15.83:36328) 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2db7fe640 1 -- 172.21.15.83:0/81241605 learned_addr learned my addr 172.21.15.83:0/81241605 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2dbfff640 1 --2- 172.21.15.83:0/81241605 >> v2:172.21.15.83:3300/0 conn(0x7fc2d4005b10 0x7fc2d4005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2daffd640 1 --2- 172.21.15.83:0/81241605 >> v2:172.21.15.184:3300/0 conn(0x7fc2d400aca0 0x7fc2d400b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2da7fc640 1 -- 172.21.15.83:0/81241605 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2379960004 0 0) 0x7fc2d40f7a50 con 0x7fc2d40f5010 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2da7fc640 1 -- 172.21.15.83:0/81241605 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2cc00bd40 con 0x7fc2d40f5010 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2daffd640 1 -- 172.21.15.83:0/81241605 >> v2:172.21.15.83:3300/0 conn(0x7fc2d4005b10 msgr2=0x7fc2d4005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2daffd640 1 --2- 172.21.15.83:0/81241605 >> v2:172.21.15.83:3300/0 conn(0x7fc2d4005b10 0x7fc2d4005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2daffd640 1 -- 172.21.15.83:0/81241605 >> v1:172.21.15.184:6789/0 conn(0x7fc2d40f5010 legacy=0x7fc2d40491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2daffd640 1 -- 172.21.15.83:0/81241605 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d40f82f0 con 0x7fc2d400aca0 2024-09-19T07:42:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.269+0000 7fc2dbfff640 1 --2- 172.21.15.83:0/81241605 >> v2:172.21.15.83:3300/0 conn(0x7fc2d4005b10 0x7fc2d4005f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:42:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.277+0000 7fc2daffd640 1 --2- 172.21.15.83:0/81241605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d400aca0 0x7fc2d400b0d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400cea0 tx=0x7fc2c801eb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a107962b6e17e7a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.277+0000 7fc2da7fc640 1 -- 172.21.15.83:0/81241605 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c800d2d0 con 0x7fc2d400aca0 2024-09-19T07:42:24.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.277+0000 7fc2da7fc640 1 -- 172.21.15.83:0/81241605 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c80a8670 con 0x7fc2d400aca0 2024-09-19T07:42:24.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.277+0000 7fc2da7fc640 1 -- 172.21.15.83:0/81241605 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c80a8a20 con 0x7fc2d400aca0 2024-09-19T07:42:24.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.277+0000 7fc2e0c90640 1 -- 172.21.15.83:0/81241605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d400aca0 msgr2=0x7fc2d400b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:24.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.277+0000 7fc2e0c90640 1 --2- 172.21.15.83:0/81241605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d400aca0 0x7fc2d400b0d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400cea0 tx=0x7fc2c801eb60 comp rx=0 tx=0).stop 2024-09-19T07:42:24.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.277+0000 7fc2e0c90640 1 -- 172.21.15.83:0/81241605 shutdown_connections 2024-09-19T07:42:24.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.277+0000 7fc2e0c90640 1 --2- 172.21.15.83:0/81241605 >> v2:172.21.15.83:3300/0 conn(0x7fc2d4005b10 0x7fc2d4005f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:24.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.277+0000 7fc2e0c90640 1 --2- 172.21.15.83:0/81241605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d400aca0 0x7fc2d400b0d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:24.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.277+0000 7fc2e0c90640 1 -- 172.21.15.83:0/81241605 >> 172.21.15.83:0/81241605 conn(0x7fc2d4047d20 msgr2=0x7fc2d4048130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:24.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.277+0000 7fc2e0c90640 1 -- 172.21.15.83:0/81241605 shutdown_connections 2024-09-19T07:42:24.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.277+0000 7fc2e0c90640 1 -- 172.21.15.83:0/81241605 wait complete. 2024-09-19T07:42:24.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.281+0000 7fc2e0c90640 1 Processor -- start 2024-09-19T07:42:24.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2e0c90640 1 -- start start 2024-09-19T07:42:24.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2e0c90640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2d4005b10 0x7fc2d40309d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:24.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2e0c90640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2d400aca0 0x7fc2d40f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:24.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2e0c90640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d40f5010 0x7fc2d40ea500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:24.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2daffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2d400aca0 0x7fc2d40f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:24.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2daffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2d400aca0 0x7fc2d40f69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50308/0 (socket says 172.21.15.83:50308) 2024-09-19T07:42:24.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2daffd640 1 -- 172.21.15.83:0/844489791 learned_addr learned my addr 172.21.15.83:0/844489791 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:24.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2e0c90640 1 -- 172.21.15.83:0/844489791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d402aa90 con 0x7fc2d400aca0 2024-09-19T07:42:24.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2e0c90640 1 -- 172.21.15.83:0/844489791 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d4006470 con 0x7fc2d40f5010 2024-09-19T07:42:24.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2e0c90640 1 -- 172.21.15.83:0/844489791 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc2d40452a0 con 0x7fc2d4005b10 2024-09-19T07:42:24.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2dbfff640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d40f5010 0x7fc2d40ea500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:24.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2db7fe640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2d4005b10 0x7fc2d40309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:24.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2daffd640 1 -- 172.21.15.83:0/844489791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2d4005b10 msgr2=0x7fc2d40309d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:24.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2daffd640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2d4005b10 0x7fc2d40309d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:24.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2daffd640 1 -- 172.21.15.83:0/844489791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d40f5010 msgr2=0x7fc2d40ea500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:24.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2daffd640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d40f5010 0x7fc2d40ea500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:24.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.285+0000 7fc2daffd640 1 -- 172.21.15.83:0/844489791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d4042d80 con 0x7fc2d400aca0 2024-09-19T07:42:24.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.289+0000 7fc2daffd640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2d400aca0 0x7fc2d40f69b0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80b2040 tx=0x7fc2c801b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:24.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.289+0000 7fc2c7fff640 1 -- 172.21.15.83:0/844489791 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8006e20 con 0x7fc2d400aca0 2024-09-19T07:42:24.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.289+0000 7fc2c7fff640 1 -- 172.21.15.83:0/844489791 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c80bf070 con 0x7fc2d400aca0 2024-09-19T07:42:24.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.289+0000 7fc2c7fff640 1 -- 172.21.15.83:0/844489791 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c80b9880 con 0x7fc2d400aca0 2024-09-19T07:42:24.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.289+0000 7fc2e0c90640 1 -- 172.21.15.83:0/844489791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d4043320 con 0x7fc2d400aca0 2024-09-19T07:42:24.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.293+0000 7fc2c7fff640 1 -- 172.21.15.83:0/844489791 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc2c80b9a20 con 0x7fc2d400aca0 2024-09-19T07:42:24.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.293+0000 7fc2c7fff640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2c0067940 0x7fc2c0069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:24.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.293+0000 7fc2db7fe640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2c0067940 0x7fc2c0069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:24.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.293+0000 7fc2e0c90640 1 -- 172.21.15.83:0/844489791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d4043bc0 con 0x7fc2d400aca0 2024-09-19T07:42:24.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.293+0000 7fc2c7fff640 1 -- 172.21.15.83:0/844489791 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9389+0+0 (secure 0 0 0) 0x7fc2c80f1850 con 0x7fc2d400aca0 2024-09-19T07:42:24.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.293+0000 7fc2db7fe640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2c0067940 0x7fc2c0069e00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0005890 tx=0x7fc2d0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:24.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.293+0000 7fc2c5ffb640 1 -- 172.21.15.83:0/844489791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b40052a0 con 0x7fc2d400aca0 2024-09-19T07:42:24.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.305+0000 7fc2c7fff640 1 -- 172.21.15.83:0/844489791 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc2c80186c0 con 0x7fc2d400aca0 2024-09-19T07:42:24.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.781+0000 7fc2c5ffb640 1 -- 172.21.15.83:0/844489791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2b4005860 con 0x7fc2d400aca0 2024-09-19T07:42:24.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.781+0000 7fc2c7fff640 1 -- 172.21.15.83:0/844489791 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) ==== 95+0+6545 (secure 0 0 0) 0x7fc2c80f3eb0 con 0x7fc2d400aca0 2024-09-19T07:42:24.782 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:24.782 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":165,"btime":"2024-09-19T07:42:13:462305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:24.782 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 165 2024-09-19T07:42:24.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.785+0000 7fc2c5ffb640 1 -- 172.21.15.83:0/844489791 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2c0067940 msgr2=0x7fc2c0069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:24.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.785+0000 7fc2c5ffb640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2c0067940 0x7fc2c0069e00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0005890 tx=0x7fc2d0009040 comp rx=0 tx=0).stop 2024-09-19T07:42:24.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.785+0000 7fc2c5ffb640 1 -- 172.21.15.83:0/844489791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2d400aca0 msgr2=0x7fc2d40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:24.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.785+0000 7fc2c5ffb640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2d400aca0 0x7fc2d40f69b0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80b2040 tx=0x7fc2c801b900 comp rx=0 tx=0).stop 2024-09-19T07:42:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.789+0000 7fc2c5ffb640 1 -- 172.21.15.83:0/844489791 shutdown_connections 2024-09-19T07:42:24.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.789+0000 7fc2c5ffb640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2c0067940 0x7fc2c0069e00 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:24.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.789+0000 7fc2c5ffb640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2d40f5010 0x7fc2d40ea500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:24.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.789+0000 7fc2c5ffb640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2d400aca0 0x7fc2d40f69b0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:24.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.789+0000 7fc2c5ffb640 1 --2- 172.21.15.83:0/844489791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2d4005b10 0x7fc2d40309d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:24.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.789+0000 7fc2c5ffb640 1 -- 172.21.15.83:0/844489791 >> 172.21.15.83:0/844489791 conn(0x7fc2d4047d20 msgr2=0x7fc2d40f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:24.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.789+0000 7fc2c5ffb640 1 -- 172.21.15.83:0/844489791 shutdown_connections 2024-09-19T07:42:24.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:24.789+0000 7fc2c5ffb640 1 -- 172.21.15.83:0/844489791 wait complete. 2024-09-19T07:42:24.839 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-19T07:42:24.839 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:42:24.839 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:42:24.858 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:42:24.859 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:25.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6999e62640 1 Processor -- start 2024-09-19T07:42:25.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6999e62640 1 -- start start 2024-09-19T07:42:25.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6999e62640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f698c0db810 0x7f698c00aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:25.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6999e62640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f698c00b1e0 0x7f698c0de540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:25.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6999e62640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c0def30 con 0x7f698c0dea80 2024-09-19T07:42:25.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6999e62640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f698c03e270 con 0x7f698c00b1e0 2024-09-19T07:42:25.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6999e62640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f698c030780 con 0x7f698c0db810 2024-09-19T07:42:25.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6999661640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f698c0dea80 0x7f698c0e15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35454/0 (socket says 172.21.15.83:35454) 2024-09-19T07:42:25.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6999661640 1 -- 172.21.15.83:0/3530965494 learned_addr learned my addr 172.21.15.83:0/3530965494 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:42:25.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6998e60640 1 --2- 172.21.15.83:0/3530965494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f698c0db810 0x7f698c00aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:25.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6998e60640 1 -- 172.21.15.83:0/3530965494 >> v2:172.21.15.184:3300/0 conn(0x7f698c00b1e0 msgr2=0x7f698c0de540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:42:25.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6998e60640 1 --2- 172.21.15.83:0/3530965494 >> v2:172.21.15.184:3300/0 conn(0x7f698c00b1e0 0x7f698c0de540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:25.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6998e60640 1 -- 172.21.15.83:0/3530965494 >> v1:172.21.15.83:6789/0 conn(0x7f698c0dea80 legacy=0x7f698c0e15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:25.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.149+0000 7f6998e60640 1 -- 172.21.15.83:0/3530965494 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c030a60 con 0x7f698c0db810 2024-09-19T07:42:25.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.153+0000 7f6998e60640 1 --2- 172.21.15.83:0/3530965494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f698c0db810 0x7f698c00aca0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f69880097c0 tx=0x7f698801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff9ced2c30be2cb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:25.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.153+0000 7f69937fe640 1 -- 172.21.15.83:0/3530965494 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69880b0070 con 0x7f698c0db810 2024-09-19T07:42:25.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.153+0000 7f69937fe640 1 -- 172.21.15.83:0/3530965494 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69880ab400 con 0x7f698c0db810 2024-09-19T07:42:25.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.153+0000 7f69937fe640 1 -- 172.21.15.83:0/3530965494 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69880ab7d0 con 0x7f698c0db810 2024-09-19T07:42:25.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.153+0000 7f6999e62640 1 -- 172.21.15.83:0/3530965494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f698c0db810 msgr2=0x7f698c00aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:25.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.153+0000 7f6999e62640 1 --2- 172.21.15.83:0/3530965494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f698c0db810 0x7f698c00aca0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f69880097c0 tx=0x7f698801c450 comp rx=0 tx=0).stop 2024-09-19T07:42:25.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.157+0000 7f6999e62640 1 -- 172.21.15.83:0/3530965494 shutdown_connections 2024-09-19T07:42:25.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.157+0000 7f6999e62640 1 --2- 172.21.15.83:0/3530965494 >> v2:172.21.15.184:3300/0 conn(0x7f698c00b1e0 0x7f698c0de540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:25.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.157+0000 7f6999e62640 1 --2- 172.21.15.83:0/3530965494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f698c0db810 0x7f698c00aca0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:25.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.157+0000 7f6999e62640 1 -- 172.21.15.83:0/3530965494 >> 172.21.15.83:0/3530965494 conn(0x7f698c047d20 msgr2=0x7f698c048130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:25.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.157+0000 7f6999e62640 1 -- 172.21.15.83:0/3530965494 shutdown_connections 2024-09-19T07:42:25.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.157+0000 7f6999e62640 1 -- 172.21.15.83:0/3530965494 wait complete. 2024-09-19T07:42:25.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.157+0000 7f6999e62640 1 Processor -- start 2024-09-19T07:42:25.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.157+0000 7f6999e62640 1 -- start start 2024-09-19T07:42:25.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6999e62640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f698c00b1e0 0x7f698c0e0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:25.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6999e62640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f698c0db810 0x7f698c042370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:25.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6999e62640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f698c0dea80 0x7f698c0428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:25.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6999e62640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f698c0455b0 con 0x7f698c00b1e0 2024-09-19T07:42:25.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6999e62640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f698c005a10 con 0x7f698c0dea80 2024-09-19T07:42:25.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6999e62640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f698c045730 con 0x7f698c0db810 2024-09-19T07:42:25.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6998e60640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f698c00b1e0 0x7f698c0e0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:25.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6998e60640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f698c00b1e0 0x7f698c0e0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50324/0 (socket says 172.21.15.83:50324) 2024-09-19T07:42:25.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6998e60640 1 -- 172.21.15.83:0/4094617199 learned_addr learned my addr 172.21.15.83:0/4094617199 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:25.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6993fff640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f698c0db810 0x7f698c042370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:25.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6999661640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f698c0dea80 0x7f698c0428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:25.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6998e60640 1 -- 172.21.15.83:0/4094617199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f698c0db810 msgr2=0x7f698c042370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:25.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6998e60640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f698c0db810 0x7f698c042370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:25.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6998e60640 1 -- 172.21.15.83:0/4094617199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f698c0dea80 msgr2=0x7f698c0428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:25.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6998e60640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f698c0dea80 0x7f698c0428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:25.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6998e60640 1 -- 172.21.15.83:0/4094617199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c041f40 con 0x7f698c00b1e0 2024-09-19T07:42:25.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6993fff640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f698c0db810 0x7f698c042370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:42:25.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6999661640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f698c0dea80 0x7f698c0428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:42:25.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6998e60640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f698c00b1e0 0x7f698c0e0720 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f69880ae040 tx=0x7f698801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:25.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f69917fa640 1 -- 172.21.15.83:0/4094617199 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69880b0050 con 0x7f698c00b1e0 2024-09-19T07:42:25.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f69917fa640 1 -- 172.21.15.83:0/4094617199 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69880bc070 con 0x7f698c00b1e0 2024-09-19T07:42:25.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f69917fa640 1 -- 172.21.15.83:0/4094617199 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69880b7890 con 0x7f698c00b1e0 2024-09-19T07:42:25.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6999e62640 1 -- 172.21.15.83:0/4094617199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698c03ef00 con 0x7f698c00b1e0 2024-09-19T07:42:25.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.161+0000 7f6999e62640 1 -- 172.21.15.83:0/4094617199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f698c041c60 con 0x7f698c00b1e0 2024-09-19T07:42:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.169+0000 7f697effd640 1 -- 172.21.15.83:0/4094617199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f696c0052a0 con 0x7f698c00b1e0 2024-09-19T07:42:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.169+0000 7f69917fa640 1 -- 172.21.15.83:0/4094617199 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f69880b7a30 con 0x7f698c00b1e0 2024-09-19T07:42:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.173+0000 7f69917fa640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6978067940 0x7f6978069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.173+0000 7f69917fa640 1 -- 172.21.15.83:0/4094617199 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9389+0+0 (secure 0 0 0) 0x7f6988121410 con 0x7f698c00b1e0 2024-09-19T07:42:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.173+0000 7f6993fff640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6978067940 0x7f6978069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.173+0000 7f6993fff640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6978067940 0x7f6978069e00 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f698c02cc20 tx=0x7f6980009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:25.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.181+0000 7f69917fa640 1 -- 172.21.15.83:0/4094617199 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f69880160d0 con 0x7f698c00b1e0 2024-09-19T07:42:25.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.665+0000 7f697effd640 1 -- 172.21.15.83:0/4094617199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f696c005590 con 0x7f698c00b1e0 2024-09-19T07:42:25.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.669+0000 7f69917fa640 1 -- 172.21.15.83:0/4094617199 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) ==== 95+0+6545 (secure 0 0 0) 0x7f69880f0c80 con 0x7f698c00b1e0 2024-09-19T07:42:25.669 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:25.669 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":165,"btime":"2024-09-19T07:42:13:462305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 165 2024-09-19T07:42:25.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.673+0000 7f697effd640 1 -- 172.21.15.83:0/4094617199 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6978067940 msgr2=0x7f6978069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:25.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.673+0000 7f697effd640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6978067940 0x7f6978069e00 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f698c02cc20 tx=0x7f6980009060 comp rx=0 tx=0).stop 2024-09-19T07:42:25.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.673+0000 7f697effd640 1 -- 172.21.15.83:0/4094617199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f698c00b1e0 msgr2=0x7f698c0e0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:25.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.673+0000 7f697effd640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f698c00b1e0 0x7f698c0e0720 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f69880ae040 tx=0x7f698801ac90 comp rx=0 tx=0).stop 2024-09-19T07:42:25.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.673+0000 7f697effd640 1 -- 172.21.15.83:0/4094617199 shutdown_connections 2024-09-19T07:42:25.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.673+0000 7f697effd640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6978067940 0x7f6978069e00 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:25.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.673+0000 7f697effd640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f698c0dea80 0x7f698c0428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:25.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.673+0000 7f697effd640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f698c0db810 0x7f698c042370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.673+0000 7f697effd640 1 --2- 172.21.15.83:0/4094617199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f698c00b1e0 0x7f698c0e0720 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.673+0000 7f697effd640 1 -- 172.21.15.83:0/4094617199 >> 172.21.15.83:0/4094617199 conn(0x7f698c047d20 msgr2=0x7f698c047000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.673+0000 7f697effd640 1 -- 172.21.15.83:0/4094617199 shutdown_connections 2024-09-19T07:42:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:25.673+0000 7f697effd640 1 -- 172.21.15.83:0/4094617199 wait complete. 2024-09-19T07:42:25.710 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:42:27.711 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:27.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf2a36e640 1 Processor -- start 2024-09-19T07:42:27.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf2a36e640 1 -- start start 2024-09-19T07:42:27.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf2a36e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf2406ef00 0x7fbf2406e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:27.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf2a36e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbf240aa3b0 0x7fbf240aa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:27.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf2a36e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf2406ce30 con 0x7fbf241577a0 2024-09-19T07:42:27.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf2a36e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbf2406d110 con 0x7fbf240aa3b0 2024-09-19T07:42:27.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf2a36e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbf2406d3f0 con 0x7fbf2406ef00 2024-09-19T07:42:27.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf28b6b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf2406ef00 0x7fbf2406e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:27.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf28b6b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf2406ef00 0x7fbf2406e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:54626/0 (socket says 172.21.15.83:54626) 2024-09-19T07:42:27.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf28b6b640 1 -- 172.21.15.83:0/1441163981 learned_addr learned my addr 172.21.15.83:0/1441163981 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:27.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf28b6b640 1 -- 172.21.15.83:0/1441163981 >> v2:172.21.15.184:3300/0 conn(0x7fbf240aa3b0 msgr2=0x7fbf240aa840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:42:27.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf28b6b640 1 --2- 172.21.15.83:0/1441163981 >> v2:172.21.15.184:3300/0 conn(0x7fbf240aa3b0 0x7fbf240aa840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:27.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf28b6b640 1 -- 172.21.15.83:0/1441163981 >> v1:172.21.15.83:6789/0 conn(0x7fbf241577a0 legacy=0x7fbf240c2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:42:27.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:27.993+0000 7fbf28b6b640 1 -- 172.21.15.83:0/1441163981 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf240aad80 con 0x7fbf2406ef00 2024-09-19T07:42:27.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.001+0000 7fbf28b6b640 1 --2- 172.21.15.83:0/1441163981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf2406ef00 0x7fbf2406e5f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009e50 tx=0x7fbf1c01ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d0bc0f8c9274d8d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:28.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.001+0000 7fbf1bfff640 1 -- 172.21.15.83:0/1441163981 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c0a5690 con 0x7fbf2406ef00 2024-09-19T07:42:28.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.001+0000 7fbf1bfff640 1 -- 172.21.15.83:0/1441163981 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf1c0a5830 con 0x7fbf2406ef00 2024-09-19T07:42:28.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.001+0000 7fbf1bfff640 1 -- 172.21.15.83:0/1441163981 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c0a5c00 con 0x7fbf2406ef00 2024-09-19T07:42:28.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.005+0000 7fbf2a36e640 1 -- 172.21.15.83:0/1441163981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf2406ef00 msgr2=0x7fbf2406e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:28.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.005+0000 7fbf2a36e640 1 --2- 172.21.15.83:0/1441163981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf2406ef00 0x7fbf2406e5f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009e50 tx=0x7fbf1c01ca10 comp rx=0 tx=0).stop 2024-09-19T07:42:28.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.009+0000 7fbf2a36e640 1 -- 172.21.15.83:0/1441163981 shutdown_connections 2024-09-19T07:42:28.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.009+0000 7fbf2a36e640 1 --2- 172.21.15.83:0/1441163981 >> v2:172.21.15.184:3300/0 conn(0x7fbf240aa3b0 0x7fbf240aa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:28.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.009+0000 7fbf2a36e640 1 --2- 172.21.15.83:0/1441163981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf2406ef00 0x7fbf2406e5f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:28.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.009+0000 7fbf2a36e640 1 -- 172.21.15.83:0/1441163981 >> 172.21.15.83:0/1441163981 conn(0x7fbf240c3d90 msgr2=0x7fbf240c41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2a36e640 1 -- 172.21.15.83:0/1441163981 shutdown_connections 2024-09-19T07:42:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2a36e640 1 -- 172.21.15.83:0/1441163981 wait complete. 2024-09-19T07:42:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2a36e640 1 Processor -- start 2024-09-19T07:42:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2a36e640 1 -- start start 2024-09-19T07:42:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2a36e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf2406ef00 0x7fbf2415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2a36e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf240aa3b0 0x7fbf2415d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2a36e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf241577a0 0x7fbf241564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2a36e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2415a130 con 0x7fbf240aa3b0 2024-09-19T07:42:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2a36e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbf24159fb0 con 0x7fbf2406ef00 2024-09-19T07:42:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2a36e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbf24165140 con 0x7fbf241577a0 2024-09-19T07:42:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf28b6b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf240aa3b0 0x7fbf2415d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2936c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf2406ef00 0x7fbf2415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf29b6d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf241577a0 0x7fbf241564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2936c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf2406ef00 0x7fbf2415c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:35666/0 (socket says 172.21.15.83:35666) 2024-09-19T07:42:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf2936c640 1 -- 172.21.15.83:0/4102004069 learned_addr learned my addr 172.21.15.83:0/4102004069 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:28.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf29b6d640 1 -- 172.21.15.83:0/4102004069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf2406ef00 msgr2=0x7fbf2415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:28.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf29b6d640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf2406ef00 0x7fbf2415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:28.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf29b6d640 1 -- 172.21.15.83:0/4102004069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf240aa3b0 msgr2=0x7fbf2415d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:28.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf29b6d640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf240aa3b0 0x7fbf2415d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:28.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf29b6d640 1 -- 172.21.15.83:0/4102004069 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf241ff390 con 0x7fbf241577a0 2024-09-19T07:42:28.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf28b6b640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf240aa3b0 0x7fbf2415d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:42:28.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf29b6d640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf241577a0 0x7fbf241564a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf1400b410 tx=0x7fbf1400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:28.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf19ffb640 1 -- 172.21.15.83:0/4102004069 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf14015020 con 0x7fbf241577a0 2024-09-19T07:42:28.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf19ffb640 1 -- 172.21.15.83:0/4102004069 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf14011830 con 0x7fbf241577a0 2024-09-19T07:42:28.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.017+0000 7fbf19ffb640 1 -- 172.21.15.83:0/4102004069 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf14011c20 con 0x7fbf241577a0 2024-09-19T07:42:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.021+0000 7fbf2a36e640 1 -- 172.21.15.83:0/4102004069 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf241ff630 con 0x7fbf241577a0 2024-09-19T07:42:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.021+0000 7fbf2a36e640 1 -- 172.21.15.83:0/4102004069 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf2415b820 con 0x7fbf241577a0 2024-09-19T07:42:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.021+0000 7fbf2a36e640 1 -- 172.21.15.83:0/4102004069 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf241654e0 con 0x7fbf241577a0 2024-09-19T07:42:28.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.029+0000 7fbf19ffb640 1 -- 172.21.15.83:0/4102004069 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbf14018020 con 0x7fbf241577a0 2024-09-19T07:42:28.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.029+0000 7fbf19ffb640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf08067910 0x7fbf08069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:28.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.029+0000 7fbf19ffb640 1 -- 172.21.15.83:0/4102004069 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 9389+0+0 (secure 0 0 0) 0x7fbf14086cd0 con 0x7fbf241577a0 2024-09-19T07:42:28.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.029+0000 7fbf2936c640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf08067910 0x7fbf08069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:28.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.029+0000 7fbf19ffb640 1 -- 172.21.15.83:0/4102004069 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbf1401f020 con 0x7fbf241577a0 2024-09-19T07:42:28.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.029+0000 7fbf2936c640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf08067910 0x7fbf08069dd0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbf240abf90 tx=0x7fbf20009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:28.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.417+0000 7fbf2a36e640 1 -- 172.21.15.83:0/4102004069 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf2406e3d0 con 0x7fbf241577a0 2024-09-19T07:42:28.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.421+0000 7fbf19ffb640 1 -- 172.21.15.83:0/4102004069 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) ==== 95+0+6545 (secure 0 0 0) 0x7fbf14056420 con 0x7fbf241577a0 2024-09-19T07:42:28.419 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:28.419 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":165,"btime":"2024-09-19T07:42:13:462305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:28.420 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 165 2024-09-19T07:42:28.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.425+0000 7fbf2a36e640 1 -- 172.21.15.83:0/4102004069 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf08067910 msgr2=0x7fbf08069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:28.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.425+0000 7fbf2a36e640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf08067910 0x7fbf08069dd0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbf240abf90 tx=0x7fbf20009110 comp rx=0 tx=0).stop 2024-09-19T07:42:28.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.425+0000 7fbf2a36e640 1 -- 172.21.15.83:0/4102004069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf241577a0 msgr2=0x7fbf241564a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:28.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.429+0000 7fbf2a36e640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf241577a0 0x7fbf241564a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf1400b410 tx=0x7fbf1400b8d0 comp rx=0 tx=0).stop 2024-09-19T07:42:28.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.429+0000 7fbf2a36e640 1 -- 172.21.15.83:0/4102004069 shutdown_connections 2024-09-19T07:42:28.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.429+0000 7fbf2a36e640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbf08067910 0x7fbf08069dd0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:28.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.429+0000 7fbf2a36e640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf241577a0 0x7fbf241564a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:28.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.429+0000 7fbf2a36e640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf240aa3b0 0x7fbf2415d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:28.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.429+0000 7fbf2a36e640 1 --2- 172.21.15.83:0/4102004069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf2406ef00 0x7fbf2415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:28.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.429+0000 7fbf2a36e640 1 -- 172.21.15.83:0/4102004069 >> 172.21.15.83:0/4102004069 conn(0x7fbf240c3d90 msgr2=0x7fbf240c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:28.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.429+0000 7fbf2a36e640 1 -- 172.21.15.83:0/4102004069 shutdown_connections 2024-09-19T07:42:28.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:28.429+0000 7fbf2a36e640 1 -- 172.21.15.83:0/4102004069 wait complete. 2024-09-19T07:42:28.498 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:42:30.499 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:30.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f9108e03640 1 Processor -- start 2024-09-19T07:42:30.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f9108e03640 1 -- start start 2024-09-19T07:42:30.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f9108e03640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f90fc0db840 0x7f90fc00acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:30.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f9108e03640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f90fc0dec80 0x7f90fc0e89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:30.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f9108e03640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90fc03e2a0 con 0x7f90fc00b210 2024-09-19T07:42:30.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f9108e03640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f90fc0307b0 con 0x7f90fc0db840 2024-09-19T07:42:30.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f9108e03640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f90fc030a90 con 0x7f90fc0dec80 2024-09-19T07:42:30.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f910259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f90fc0db840 0x7f90fc00acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:30.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f910259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f90fc0db840 0x7f90fc00acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54872/0 (socket says 172.21.15.83:54872) 2024-09-19T07:42:30.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f910259c640 1 -- 172.21.15.83:0/672049563 learned_addr learned my addr 172.21.15.83:0/672049563 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:30.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f9102d9d640 1 --2- 172.21.15.83:0/672049563 >> v2:172.21.15.184:3300/0 conn(0x7f90fc0dec80 0x7f90fc0e89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:30.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f910259c640 1 -- 172.21.15.83:0/672049563 >> v2:172.21.15.184:3300/0 conn(0x7f90fc0dec80 msgr2=0x7f90fc0e89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:30.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f910259c640 1 --2- 172.21.15.83:0/672049563 >> v2:172.21.15.184:3300/0 conn(0x7f90fc0dec80 0x7f90fc0e89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:30.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f910259c640 1 -- 172.21.15.83:0/672049563 >> v1:172.21.15.184:6789/0 conn(0x7f90fc00b210 legacy=0x7f90fc0de570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:30.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.693+0000 7f910259c640 1 -- 172.21.15.83:0/672049563 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90fc030d70 con 0x7f90fc0db840 2024-09-19T07:42:30.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.697+0000 7f910259c640 1 --2- 172.21.15.83:0/672049563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90fc0db840 0x7f90fc00acd0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f910407bc70 tx=0x7f9104098b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d26d9e5eec82bf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:30.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.697+0000 7f910159a640 1 -- 172.21.15.83:0/672049563 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9104121750 con 0x7f90fc0db840 2024-09-19T07:42:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.697+0000 7f910159a640 1 -- 172.21.15.83:0/672049563 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9104121950 con 0x7f90fc0db840 2024-09-19T07:42:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.697+0000 7f9108e03640 1 -- 172.21.15.83:0/672049563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90fc0db840 msgr2=0x7f90fc00acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.697+0000 7f9108e03640 1 --2- 172.21.15.83:0/672049563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90fc0db840 0x7f90fc00acd0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f910407bc70 tx=0x7f9104098b70 comp rx=0 tx=0).stop 2024-09-19T07:42:30.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.701+0000 7f9108e03640 1 -- 172.21.15.83:0/672049563 shutdown_connections 2024-09-19T07:42:30.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.701+0000 7f9108e03640 1 --2- 172.21.15.83:0/672049563 >> v2:172.21.15.184:3300/0 conn(0x7f90fc0dec80 0x7f90fc0e89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:30.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.701+0000 7f9108e03640 1 --2- 172.21.15.83:0/672049563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90fc0db840 0x7f90fc00acd0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:30.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.701+0000 7f9108e03640 1 -- 172.21.15.83:0/672049563 >> 172.21.15.83:0/672049563 conn(0x7f90fc047d50 msgr2=0x7f90fc048160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:30.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.701+0000 7f9108e03640 1 -- 172.21.15.83:0/672049563 shutdown_connections 2024-09-19T07:42:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.701+0000 7f9108e03640 1 -- 172.21.15.83:0/672049563 wait complete. 2024-09-19T07:42:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.701+0000 7f9108e03640 1 Processor -- start 2024-09-19T07:42:30.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.701+0000 7f9108e03640 1 -- start start 2024-09-19T07:42:30.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9108e03640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90fc00b210 0x7f90fc0daaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:30.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9108e03640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90fc0db840 0x7f90fc0447a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:30.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9108e03640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90fc0dec80 0x7f90fc17fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:30.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9108e03640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f90fc045760 con 0x7f90fc0dec80 2024-09-19T07:42:30.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9108e03640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90fc0455e0 con 0x7f90fc00b210 2024-09-19T07:42:30.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9108e03640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f90fc03e9c0 con 0x7f90fc0db840 2024-09-19T07:42:30.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f910259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90fc00b210 0x7f90fc0daaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:30.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f910259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90fc00b210 0x7f90fc0daaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:55358/0 (socket says 172.21.15.83:55358) 2024-09-19T07:42:30.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f910259c640 1 -- 172.21.15.83:0/531644913 learned_addr learned my addr 172.21.15.83:0/531644913 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:30.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9102d9d640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90fc0dec80 0x7f90fc17fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:30.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9102d9d640 1 -- 172.21.15.83:0/531644913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90fc0db840 msgr2=0x7f90fc0447a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:42:30.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9102d9d640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90fc0db840 0x7f90fc0447a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9102d9d640 1 -- 172.21.15.83:0/531644913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90fc00b210 msgr2=0x7f90fc0daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9102d9d640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90fc00b210 0x7f90fc0daaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9102d9d640 1 -- 172.21.15.83:0/531644913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90fc0dfdb0 con 0x7f90fc0dec80 2024-09-19T07:42:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9102d9d640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90fc0dec80 0x7f90fc17fa00 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f90f400e7f0 tx=0x7f90f400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f90f2ffd640 1 -- 172.21.15.83:0/531644913 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f4017070 con 0x7f90fc0dec80 2024-09-19T07:42:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9108e03640 1 -- 172.21.15.83:0/531644913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90fc044160 con 0x7f90fc0dec80 2024-09-19T07:42:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.705+0000 7f9108e03640 1 -- 172.21.15.83:0/531644913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90fc045300 con 0x7f90fc0dec80 2024-09-19T07:42:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.709+0000 7f90f2ffd640 1 -- 172.21.15.83:0/531644913 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90f400c730 con 0x7f90fc0dec80 2024-09-19T07:42:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.709+0000 7f90f2ffd640 1 -- 172.21.15.83:0/531644913 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f400cb00 con 0x7f90fc0dec80 2024-09-19T07:42:30.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.709+0000 7f9108e03640 1 -- 172.21.15.83:0/531644913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90fc0310b0 con 0x7f90fc0dec80 2024-09-19T07:42:30.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.709+0000 7f90f2ffd640 1 -- 172.21.15.83:0/531644913 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f90f401a070 con 0x7f90fc0dec80 2024-09-19T07:42:30.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.709+0000 7f90f2ffd640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90e40678c0 0x7f90e4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:30.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.709+0000 7f90f2ffd640 1 -- 172.21.15.83:0/531644913 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9389+0+0 (secure 0 0 0) 0x7f90f4086fa0 con 0x7f90fc0dec80 2024-09-19T07:42:30.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.713+0000 7f910259c640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90e40678c0 0x7f90e4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:30.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.717+0000 7f910259c640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90e40678c0 0x7f90e4069d80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9104051960 tx=0x7f910406bbb0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:30.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:30.717+0000 7f90f2ffd640 1 -- 172.21.15.83:0/531644913 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f90f4056690 con 0x7f90fc0dec80 2024-09-19T07:42:31.183 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:31.183 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":165,"btime":"2024-09-19T07:42:13:462305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:31.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.181+0000 7f9108e03640 1 -- 172.21.15.83:0/531644913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90fc007030 con 0x7f90fc0dec80 2024-09-19T07:42:31.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.181+0000 7f90f2ffd640 1 -- 172.21.15.83:0/531644913 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) ==== 95+0+6545 (secure 0 0 0) 0x7f90f405a990 con 0x7f90fc0dec80 2024-09-19T07:42:31.183 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 165 2024-09-19T07:42:31.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.185+0000 7f90f0ff9640 1 -- 172.21.15.83:0/531644913 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90e40678c0 msgr2=0x7f90e4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:31.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.185+0000 7f90f0ff9640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90e40678c0 0x7f90e4069d80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9104051960 tx=0x7f910406bbb0 comp rx=0 tx=0).stop 2024-09-19T07:42:31.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.185+0000 7f90f0ff9640 1 -- 172.21.15.83:0/531644913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90fc0dec80 msgr2=0x7f90fc17fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:31.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.185+0000 7f90f0ff9640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90fc0dec80 0x7f90fc17fa00 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f90f400e7f0 tx=0x7f90f400ecb0 comp rx=0 tx=0).stop 2024-09-19T07:42:31.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.185+0000 7f90f0ff9640 1 -- 172.21.15.83:0/531644913 shutdown_connections 2024-09-19T07:42:31.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.185+0000 7f90f0ff9640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90e40678c0 0x7f90e4069d80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:31.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.185+0000 7f90f0ff9640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90fc0dec80 0x7f90fc17fa00 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:31.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.185+0000 7f90f0ff9640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90fc0db840 0x7f90fc0447a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:31.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.185+0000 7f90f0ff9640 1 --2- 172.21.15.83:0/531644913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90fc00b210 0x7f90fc0daaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:31.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.185+0000 7f90f0ff9640 1 -- 172.21.15.83:0/531644913 >> 172.21.15.83:0/531644913 conn(0x7f90fc047d50 msgr2=0x7f90fc02feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:31.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.185+0000 7f90f0ff9640 1 -- 172.21.15.83:0/531644913 shutdown_connections 2024-09-19T07:42:31.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:31.185+0000 7f90f0ff9640 1 -- 172.21.15.83:0/531644913 wait complete. 2024-09-19T07:42:31.217 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:42:33.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:33.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93d637e640 1 Processor -- start 2024-09-19T07:42:33.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93d637e640 1 -- start start 2024-09-19T07:42:33.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93d637e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c80d5240 0x7f93c8049ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:33.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93d637e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c800d730 con 0x7f93c804b3c0 2024-09-19T07:42:33.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93d637e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c8047fe0 con 0x7f93c8040b50 2024-09-19T07:42:33.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93d637e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f93c80d8a20 con 0x7f93c80d5240 2024-09-19T07:42:33.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93d537c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c80d5240 0x7f93c8049ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:33.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93d4b7b640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f93c8040b50 0x7f93c800c050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:59854/0 (socket says 172.21.15.83:59854) 2024-09-19T07:42:33.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93d4b7b640 1 -- 172.21.15.83:0/945658307 learned_addr learned my addr 172.21.15.83:0/945658307 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:42:33.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93d537c640 1 -- 172.21.15.83:0/945658307 >> v1:172.21.15.184:6789/0 conn(0x7f93c8040b50 legacy=0x7f93c800c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:33.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93c7fff640 1 -- 172.21.15.83:0/945658307 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 800078702 0 0) 0x7f93c8047fe0 con 0x7f93c8040b50 2024-09-19T07:42:33.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93d537c640 1 -- 172.21.15.83:0/945658307 >> v1:172.21.15.83:6789/0 conn(0x7f93c804b3c0 legacy=0x7f93c800c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:33.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.481+0000 7f93d537c640 1 -- 172.21.15.83:0/945658307 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c80d5620 con 0x7f93c80d5240 2024-09-19T07:42:33.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.485+0000 7f93d537c640 1 --2- 172.21.15.83:0/945658307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c80d5240 0x7f93c8049ed0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f93cc0097f0 tx=0x7f93cc01c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ac15f468fa7af9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:33.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.485+0000 7f93c7fff640 1 -- 172.21.15.83:0/945658307 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc0b0070 con 0x7f93c80d5240 2024-09-19T07:42:33.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.485+0000 7f93c7fff640 1 -- 172.21.15.83:0/945658307 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93cc0ab600 con 0x7f93c80d5240 2024-09-19T07:42:33.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.485+0000 7f93c7fff640 1 -- 172.21.15.83:0/945658307 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc0ab9d0 con 0x7f93c80d5240 2024-09-19T07:42:33.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.485+0000 7f93d637e640 1 -- 172.21.15.83:0/945658307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c80d5240 msgr2=0x7f93c8049ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:33.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.485+0000 7f93d637e640 1 --2- 172.21.15.83:0/945658307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c80d5240 0x7f93c8049ed0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f93cc0097f0 tx=0x7f93cc01c380 comp rx=0 tx=0).stop 2024-09-19T07:42:33.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.485+0000 7f93d637e640 1 -- 172.21.15.83:0/945658307 shutdown_connections 2024-09-19T07:42:33.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.485+0000 7f93d637e640 1 --2- 172.21.15.83:0/945658307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c80d5240 0x7f93c8049ed0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:33.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.485+0000 7f93d637e640 1 -- 172.21.15.83:0/945658307 >> 172.21.15.83:0/945658307 conn(0x7f93c802f070 msgr2=0x7f93c802f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:33.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.485+0000 7f93d637e640 1 -- 172.21.15.83:0/945658307 shutdown_connections 2024-09-19T07:42:33.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.485+0000 7f93d637e640 1 -- 172.21.15.83:0/945658307 wait complete. 2024-09-19T07:42:33.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d637e640 1 Processor -- start 2024-09-19T07:42:33.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d637e640 1 -- start start 2024-09-19T07:42:33.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d637e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93c8040b50 0x7f93c80cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:33.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d637e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c804b3c0 0x7f93c80cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:33.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d637e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c80d5240 0x7f93c804c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:33.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d637e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f93c80e5d10 con 0x7f93c80d5240 2024-09-19T07:42:33.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d637e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f93c80e5b90 con 0x7f93c8040b50 2024-09-19T07:42:33.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d637e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f93c80e5e90 con 0x7f93c804b3c0 2024-09-19T07:42:33.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d5b7d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c80d5240 0x7f93c804c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:33.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d5b7d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c80d5240 0x7f93c804c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54878/0 (socket says 172.21.15.83:54878) 2024-09-19T07:42:33.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d5b7d640 1 -- 172.21.15.83:0/2654148060 learned_addr learned my addr 172.21.15.83:0/2654148060 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:33.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d5b7d640 1 -- 172.21.15.83:0/2654148060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c804b3c0 msgr2=0x7f93c80cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:33.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d5b7d640 1 --2- 172.21.15.83:0/2654148060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c804b3c0 0x7f93c80cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:33.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d5b7d640 1 -- 172.21.15.83:0/2654148060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93c8040b50 msgr2=0x7f93c80cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:33.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d5b7d640 1 --2- 172.21.15.83:0/2654148060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93c8040b50 0x7f93c80cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:33.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d5b7d640 1 -- 172.21.15.83:0/2654148060 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c80d0060 con 0x7f93c80d5240 2024-09-19T07:42:33.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d5b7d640 1 --2- 172.21.15.83:0/2654148060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c80d5240 0x7f93c804c450 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f93d00657c0 tx=0x7f93d0069900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:33.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93c5ffb640 1 -- 172.21.15.83:0/2654148060 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d00a1070 con 0x7f93c80d5240 2024-09-19T07:42:33.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93c5ffb640 1 -- 172.21.15.83:0/2654148060 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93d0067a70 con 0x7f93c80d5240 2024-09-19T07:42:33.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d637e640 1 -- 172.21.15.83:0/2654148060 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c80d0340 con 0x7f93c80d5240 2024-09-19T07:42:33.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93c5ffb640 1 -- 172.21.15.83:0/2654148060 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d0067e60 con 0x7f93c80d5240 2024-09-19T07:42:33.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.489+0000 7f93d637e640 1 -- 172.21.15.83:0/2654148060 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c80d0990 con 0x7f93c80d5240 2024-09-19T07:42:33.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.493+0000 7f93d637e640 1 -- 172.21.15.83:0/2654148060 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b00052a0 con 0x7f93c80d5240 2024-09-19T07:42:33.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.493+0000 7f93c5ffb640 1 -- 172.21.15.83:0/2654148060 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f93d0071630 con 0x7f93c80d5240 2024-09-19T07:42:33.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.493+0000 7f93c5ffb640 1 --2- 172.21.15.83:0/2654148060 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f93b40678f0 0x7f93b4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:33.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.493+0000 7f93d537c640 1 --2- 172.21.15.83:0/2654148060 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f93b40678f0 0x7f93b4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:33.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.493+0000 7f93d537c640 1 --2- 172.21.15.83:0/2654148060 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f93b40678f0 0x7f93b4069db0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f93cc0ad000 tx=0x7f93cc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:33.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.497+0000 7f93c5ffb640 1 -- 172.21.15.83:0/2654148060 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9389+0+0 (secure 0 0 0) 0x7f93d00e1670 con 0x7f93c80d5240 2024-09-19T07:42:33.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.501+0000 7f93c5ffb640 1 -- 172.21.15.83:0/2654148060 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f93d00e3cd0 con 0x7f93c80d5240 2024-09-19T07:42:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.973+0000 7f93d637e640 1 -- 172.21.15.83:0/2654148060 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93b0005590 con 0x7f93c80d5240 2024-09-19T07:42:33.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.973+0000 7f93c5ffb640 1 -- 172.21.15.83:0/2654148060 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) ==== 95+0+6545 (secure 0 0 0) 0x7f93d00e35d0 con 0x7f93c80d5240 2024-09-19T07:42:33.973 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:33.974 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":165,"btime":"2024-09-19T07:42:13:462305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:33.974 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 165 2024-09-19T07:42:33.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.977+0000 7f93d637e640 1 -- 172.21.15.83:0/2654148060 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f93b40678f0 msgr2=0x7f93b4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:33.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.977+0000 7f93d637e640 1 --2- 172.21.15.83:0/2654148060 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f93b40678f0 0x7f93b4069db0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f93cc0ad000 tx=0x7f93cc008040 comp rx=0 tx=0).stop 2024-09-19T07:42:33.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.977+0000 7f93d637e640 1 -- 172.21.15.83:0/2654148060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c80d5240 msgr2=0x7f93c804c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:33.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.977+0000 7f93d637e640 1 --2- 172.21.15.83:0/2654148060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c80d5240 0x7f93c804c450 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f93d00657c0 tx=0x7f93d0069900 comp rx=0 tx=0).stop 2024-09-19T07:42:33.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.977+0000 7f93d637e640 1 -- 172.21.15.83:0/2654148060 shutdown_connections 2024-09-19T07:42:33.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.977+0000 7f93d637e640 1 --2- 172.21.15.83:0/2654148060 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f93b40678f0 0x7f93b4069db0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:33.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.977+0000 7f93d637e640 1 --2- 172.21.15.83:0/2654148060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93c80d5240 0x7f93c804c450 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:33.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.977+0000 7f93d637e640 1 --2- 172.21.15.83:0/2654148060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93c804b3c0 0x7f93c80cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:33.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.981+0000 7f93d637e640 1 --2- 172.21.15.83:0/2654148060 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93c8040b50 0x7f93c80cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:33.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.981+0000 7f93d637e640 1 -- 172.21.15.83:0/2654148060 >> 172.21.15.83:0/2654148060 conn(0x7f93c802f070 msgr2=0x7f93c80ceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:33.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.981+0000 7f93d637e640 1 -- 172.21.15.83:0/2654148060 shutdown_connections 2024-09-19T07:42:33.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:33.981+0000 7f93d637e640 1 -- 172.21.15.83:0/2654148060 wait complete. 2024-09-19T07:42:34.022 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:42:36.023 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feed565b640 1 Processor -- start 2024-09-19T07:42:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feed565b640 1 -- start start 2024-09-19T07:42:36.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feed565b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00731d0 0x7feed0157490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:36.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feed565b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feed00c7640 con 0x7feed00ac6c0 2024-09-19T07:42:36.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feed565b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feed00c7920 con 0x7feed00c9140 2024-09-19T07:42:36.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feed565b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feed00c8400 con 0x7feed00731d0 2024-09-19T07:42:36.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feeceffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00731d0 0x7feed0157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:36.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feeceffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00731d0 0x7feed0157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:51466/0 (socket says 172.21.15.83:51466) 2024-09-19T07:42:36.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feeceffd640 1 -- 172.21.15.83:0/3251075989 learned_addr learned my addr 172.21.15.83:0/3251075989 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:36.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feecdffb640 1 -- 172.21.15.83:0/3251075989 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3347317890 0 0) 0x7feed00c7920 con 0x7feed00c9140 2024-09-19T07:42:36.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feecdffb640 1 -- 172.21.15.83:0/3251075989 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feeb8003530 con 0x7feed00c9140 2024-09-19T07:42:36.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feeceffd640 1 -- 172.21.15.83:0/3251075989 >> v1:172.21.15.184:6789/0 conn(0x7feed00c9140 legacy=0x7feed015d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:36.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feeceffd640 1 -- 172.21.15.83:0/3251075989 >> v1:172.21.15.83:6789/0 conn(0x7feed00ac6c0 legacy=0x7feed0164a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:36.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.333+0000 7feeceffd640 1 -- 172.21.15.83:0/3251075989 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed00c7c00 con 0x7feed00731d0 2024-09-19T07:42:36.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.341+0000 7feeceffd640 1 --2- 172.21.15.83:0/3251075989 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00731d0 0x7feed0157490 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7feec00098a0 tx=0x7feec002d470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da0a3945c246d061 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:36.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.341+0000 7feecdffb640 1 -- 172.21.15.83:0/3251075989 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec00c1070 con 0x7feed00731d0 2024-09-19T07:42:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.341+0000 7feecdffb640 1 -- 172.21.15.83:0/3251075989 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feec00bc600 con 0x7feed00731d0 2024-09-19T07:42:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.341+0000 7feecdffb640 1 -- 172.21.15.83:0/3251075989 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec00bc9d0 con 0x7feed00731d0 2024-09-19T07:42:36.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.341+0000 7feed565b640 1 -- 172.21.15.83:0/3251075989 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00731d0 msgr2=0x7feed0157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:36.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.341+0000 7feed565b640 1 --2- 172.21.15.83:0/3251075989 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00731d0 0x7feed0157490 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7feec00098a0 tx=0x7feec002d470 comp rx=0 tx=0).stop 2024-09-19T07:42:36.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.349+0000 7feed565b640 1 -- 172.21.15.83:0/3251075989 shutdown_connections 2024-09-19T07:42:36.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.349+0000 7feed565b640 1 --2- 172.21.15.83:0/3251075989 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00731d0 0x7feed0157490 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7feec00098a0 tx=0x7feec002d470 comp rx=0 tx=0).stop 2024-09-19T07:42:36.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.349+0000 7feed565b640 1 -- 172.21.15.83:0/3251075989 >> 172.21.15.83:0/3251075989 conn(0x7feed00c9610 msgr2=0x7feed00c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:36.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.357+0000 7feed565b640 1 -- 172.21.15.83:0/3251075989 shutdown_connections 2024-09-19T07:42:36.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.357+0000 7feed565b640 1 -- 172.21.15.83:0/3251075989 wait complete. 2024-09-19T07:42:36.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feed565b640 1 Processor -- start 2024-09-19T07:42:36.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feed565b640 1 -- start start 2024-09-19T07:42:36.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feed565b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00ac6c0 0x7feed014fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:36.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feed565b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed00c9140 0x7feed015bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:36.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feed565b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed0156fd0 0x7feed0154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:36.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feed565b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feed0157de0 con 0x7feed00c9140 2024-09-19T07:42:36.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feed565b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feed00af3a0 con 0x7feed0156fd0 2024-09-19T07:42:36.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feed565b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feed014b830 con 0x7feed00ac6c0 2024-09-19T07:42:36.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feeceffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00ac6c0 0x7feed014fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:36.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feeceffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00ac6c0 0x7feed014fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:51478/0 (socket says 172.21.15.83:51478) 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feeceffd640 1 -- 172.21.15.83:0/271570807 learned_addr learned my addr 172.21.15.83:0/271570807 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feecf7fe640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed0156fd0 0x7feed0154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feeceffd640 1 -- 172.21.15.83:0/271570807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed0156fd0 msgr2=0x7feed0154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feeceffd640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed0156fd0 0x7feed0154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feeceffd640 1 -- 172.21.15.83:0/271570807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed00c9140 msgr2=0x7feed015bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feece7fc640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed00c9140 0x7feed015bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feeceffd640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed00c9140 0x7feed015bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feeceffd640 1 -- 172.21.15.83:0/271570807 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed014bab0 con 0x7feed00ac6c0 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feece7fc640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed00c9140 0x7feed015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feeceffd640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00ac6c0 0x7feed014fd90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feec00097f0 tx=0x7feec002bcb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feeb77fe640 1 -- 172.21.15.83:0/271570807 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec00c1040 con 0x7feed00ac6c0 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feeb77fe640 1 -- 172.21.15.83:0/271570807 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feec0004030 con 0x7feed00ac6c0 2024-09-19T07:42:36.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feeb77fe640 1 -- 172.21.15.83:0/271570807 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec0004420 con 0x7feed00ac6c0 2024-09-19T07:42:36.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feed565b640 1 -- 172.21.15.83:0/271570807 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feed014edc0 con 0x7feed00ac6c0 2024-09-19T07:42:36.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.365+0000 7feed565b640 1 -- 172.21.15.83:0/271570807 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feed014e800 con 0x7feed00ac6c0 2024-09-19T07:42:36.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.369+0000 7feed565b640 1 -- 172.21.15.83:0/271570807 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeb00052a0 con 0x7feed00ac6c0 2024-09-19T07:42:36.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.373+0000 7feeb77fe640 1 -- 172.21.15.83:0/271570807 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feec00c4070 con 0x7feed00ac6c0 2024-09-19T07:42:36.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.373+0000 7feeb77fe640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feebc0678f0 0x7feebc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:36.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.373+0000 7feeb77fe640 1 -- 172.21.15.83:0/271570807 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 9389+0+0 (secure 0 0 0) 0x7feec0131c30 con 0x7feed00ac6c0 2024-09-19T07:42:36.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.373+0000 7feece7fc640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feebc0678f0 0x7feebc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:36.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.373+0000 7feece7fc640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feebc0678f0 0x7feebc069db0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7feed00c3940 tx=0x7feec4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:36.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.381+0000 7feeb77fe640 1 -- 172.21.15.83:0/271570807 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7feec00d4030 con 0x7feed00ac6c0 2024-09-19T07:42:36.628 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:42:36.632 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:42:36.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.873+0000 7feed565b640 1 -- 172.21.15.83:0/271570807 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feeb0005590 con 0x7feed00ac6c0 2024-09-19T07:42:36.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.873+0000 7feeb77fe640 1 -- 172.21.15.83:0/271570807 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) ==== 95+0+6545 (secure 0 0 0) 0x7feec00cb020 con 0x7feed00ac6c0 2024-09-19T07:42:36.876 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:36.876 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":165,"btime":"2024-09-19T07:42:13:462305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:41:11.437309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":13,"addr":"172.21.15.83:6837/1990119490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1990119490},{"type":"v1","addr":"172.21.15.83:6837","nonce":1990119490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":2}]} 2024-09-19T07:42:36.876 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 165 2024-09-19T07:42:36.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.877+0000 7feeb57fa640 1 -- 172.21.15.83:0/271570807 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feebc0678f0 msgr2=0x7feebc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:36.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.877+0000 7feeb57fa640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feebc0678f0 0x7feebc069db0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7feed00c3940 tx=0x7feec4009040 comp rx=0 tx=0).stop 2024-09-19T07:42:36.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.877+0000 7feeb57fa640 1 -- 172.21.15.83:0/271570807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00ac6c0 msgr2=0x7feed014fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:36.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.877+0000 7feeb57fa640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00ac6c0 0x7feed014fd90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feec00097f0 tx=0x7feec002bcb0 comp rx=0 tx=0).stop 2024-09-19T07:42:36.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.877+0000 7feeb57fa640 1 -- 172.21.15.83:0/271570807 shutdown_connections 2024-09-19T07:42:36.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.877+0000 7feeb57fa640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feebc0678f0 0x7feebc069db0 secure :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7feed00c3940 tx=0x7feec4009040 comp rx=0 tx=0).stop 2024-09-19T07:42:36.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.877+0000 7feeb57fa640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7feed0156fd0 0x7feed0154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:36.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.877+0000 7feeb57fa640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feed00c9140 0x7feed015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:36.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.877+0000 7feeb57fa640 1 --2- 172.21.15.83:0/271570807 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feed00ac6c0 0x7feed014fd90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:36.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.877+0000 7feeb57fa640 1 -- 172.21.15.83:0/271570807 >> 172.21.15.83:0/271570807 conn(0x7feed00c9610 msgr2=0x7feed00c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:36.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.877+0000 7feeb57fa640 1 -- 172.21.15.83:0/271570807 shutdown_connections 2024-09-19T07:42:36.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:36.881+0000 7feeb57fa640 1 -- 172.21.15.83:0/271570807 wait complete. 2024-09-19T07:42:36.910 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:42:38.912 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:39.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.233+0000 7fa207efc640 1 Processor -- start 2024-09-19T07:42:39.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.233+0000 7fa207efc640 1 -- start start 2024-09-19T07:42:39.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.233+0000 7fa207efc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa20006df90 0x7fa2000c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:39.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.233+0000 7fa207efc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2000ac0d0 con 0x7fa2000abc60 2024-09-19T07:42:39.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.233+0000 7fa207efc640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2000c2700 con 0x7fa2000739c0 2024-09-19T07:42:39.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.233+0000 7fa207efc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa20006c530 con 0x7fa20006df90 2024-09-19T07:42:39.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.233+0000 7fa205c98640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa20006df90 0x7fa2000c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:39.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.233+0000 7fa205c98640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa20006df90 0x7fa2000c3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:51494/0 (socket says 172.21.15.83:51494) 2024-09-19T07:42:39.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.233+0000 7fa205c98640 1 -- 172.21.15.83:0/532111737 learned_addr learned my addr 172.21.15.83:0/532111737 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:39.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.233+0000 7fa205c98640 1 -- 172.21.15.83:0/532111737 >> v1:172.21.15.184:6789/0 conn(0x7fa2000739c0 legacy=0x7fa2000c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:39.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.233+0000 7fa205c98640 1 -- 172.21.15.83:0/532111737 >> v1:172.21.15.83:6789/0 conn(0x7fa2000abc60 legacy=0x7fa2001529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:39.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.233+0000 7fa205c98640 1 -- 172.21.15.83:0/532111737 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20006c7d0 con 0x7fa20006df90 2024-09-19T07:42:39.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.241+0000 7fa205c98640 1 --2- 172.21.15.83:0/532111737 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa20006df90 0x7fa2000c3ae0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00098a0 tx=0x7fa1f001c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec47602352d7b943 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:39.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.241+0000 7fa204c96640 1 -- 172.21.15.83:0/532111737 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f00b1070 con 0x7fa20006df90 2024-09-19T07:42:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.241+0000 7fa204c96640 1 -- 172.21.15.83:0/532111737 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f00ac600 con 0x7fa20006df90 2024-09-19T07:42:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.241+0000 7fa204c96640 1 -- 172.21.15.83:0/532111737 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f00ac9d0 con 0x7fa20006df90 2024-09-19T07:42:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.245+0000 7fa207efc640 1 -- 172.21.15.83:0/532111737 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa20006df90 msgr2=0x7fa2000c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.245+0000 7fa207efc640 1 --2- 172.21.15.83:0/532111737 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa20006df90 0x7fa2000c3ae0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00098a0 tx=0x7fa1f001c4b0 comp rx=0 tx=0).stop 2024-09-19T07:42:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.245+0000 7fa207efc640 1 -- 172.21.15.83:0/532111737 shutdown_connections 2024-09-19T07:42:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.245+0000 7fa207efc640 1 --2- 172.21.15.83:0/532111737 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa20006df90 0x7fa2000c3ae0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00098a0 tx=0x7fa1f001c4b0 comp rx=0 tx=0).stop 2024-09-19T07:42:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.245+0000 7fa207efc640 1 -- 172.21.15.83:0/532111737 >> 172.21.15.83:0/532111737 conn(0x7fa200075390 msgr2=0x7fa20006bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:39.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.245+0000 7fa207efc640 1 -- 172.21.15.83:0/532111737 shutdown_connections 2024-09-19T07:42:39.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.245+0000 7fa207efc640 1 -- 172.21.15.83:0/532111737 wait complete. 2024-09-19T07:42:39.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.253+0000 7fa207efc640 1 Processor -- start 2024-09-19T07:42:39.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa207efc640 1 -- start start 2024-09-19T07:42:39.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa207efc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2000739c0 0x7fa2000bd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:39.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa207efc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2000abc60 0x7fa2001606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:39.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa207efc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa200160c30 0x7fa2000c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:39.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa207efc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa200153100 con 0x7fa2000739c0 2024-09-19T07:42:39.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa207efc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa20006e370 con 0x7fa2000abc60 2024-09-19T07:42:39.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa207efc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa2000bc610 con 0x7fa200160c30 2024-09-19T07:42:39.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa206499640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa200160c30 0x7fa2000c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:39.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa206499640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa200160c30 0x7fa2000c6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:51498/0 (socket says 172.21.15.83:51498) 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa206499640 1 -- 172.21.15.83:0/1252656229 learned_addr learned my addr 172.21.15.83:0/1252656229 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa205497640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2000abc60 0x7fa2001606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa205c98640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2000739c0 0x7fa2000bd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa205497640 1 -- 172.21.15.83:0/1252656229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa200160c30 msgr2=0x7fa2000c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa205497640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa200160c30 0x7fa2000c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.261+0000 7fa205497640 1 -- 172.21.15.83:0/1252656229 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2000739c0 msgr2=0x7fa2000bd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.265+0000 7fa205497640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2000739c0 0x7fa2000bd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.265+0000 7fa205497640 1 -- 172.21.15.83:0/1252656229 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20008c640 con 0x7fa2000abc60 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.265+0000 7fa206499640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa200160c30 0x7fa2000c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.265+0000 7fa205c98640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2000739c0 0x7fa2000bd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.265+0000 7fa205497640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2000abc60 0x7fa2001606f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00b3e0 tx=0x7fa1fc00b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.265+0000 7fa1f67fc640 1 -- 172.21.15.83:0/1252656229 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc015020 con 0x7fa2000abc60 2024-09-19T07:42:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.265+0000 7fa207efc640 1 -- 172.21.15.83:0/1252656229 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2000bc860 con 0x7fa2000abc60 2024-09-19T07:42:39.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.265+0000 7fa207efc640 1 -- 172.21.15.83:0/1252656229 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2000a9ec0 con 0x7fa2000abc60 2024-09-19T07:42:39.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.265+0000 7fa1f67fc640 1 -- 172.21.15.83:0/1252656229 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1fc011830 con 0x7fa2000abc60 2024-09-19T07:42:39.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.265+0000 7fa1f67fc640 1 -- 172.21.15.83:0/1252656229 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc011c60 con 0x7fa2000abc60 2024-09-19T07:42:39.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.265+0000 7fa207efc640 1 -- 172.21.15.83:0/1252656229 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2000aa260 con 0x7fa2000abc60 2024-09-19T07:42:39.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.285+0000 7fa1f67fc640 1 -- 172.21.15.83:0/1252656229 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa1fc018020 con 0x7fa2000abc60 2024-09-19T07:42:39.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.285+0000 7fa1f67fc640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1e40ab8d0 0x7fa1e40add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:39.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.285+0000 7fa205c98640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1e40ab8d0 0x7fa1e40add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:39.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.285+0000 7fa1f67fc640 1 -- 172.21.15.83:0/1252656229 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9475+0+0 (secure 0 0 0) 0x7fa1fc086eb0 con 0x7fa2000abc60 2024-09-19T07:42:39.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.285+0000 7fa1f67fc640 1 -- 172.21.15.83:0/1252656229 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa1fc01f020 con 0x7fa2000abc60 2024-09-19T07:42:39.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.285+0000 7fa205c98640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1e40ab8d0 0x7fa1e40add90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00ae000 tx=0x7fa1f00a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:39.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.853+0000 7fa207efc640 1 -- 172.21.15.83:0/1252656229 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2001563f0 con 0x7fa2000abc60 2024-09-19T07:42:39.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.853+0000 7fa1f67fc640 1 -- 172.21.15.83:0/1252656229 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+5711 (secure 0 0 0) 0x7fa1fc0565a0 con 0x7fa2000abc60 2024-09-19T07:42:39.854 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:39.854 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":166,"btime":"2024-09-19T07:42:38:261182+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:38.261178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:42:39.854 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 166 2024-09-19T07:42:39.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.861+0000 7fa207efc640 1 -- 172.21.15.83:0/1252656229 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1e40ab8d0 msgr2=0x7fa1e40add90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:39.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.861+0000 7fa207efc640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1e40ab8d0 0x7fa1e40add90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00ae000 tx=0x7fa1f00a7000 comp rx=0 tx=0).stop 2024-09-19T07:42:39.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.861+0000 7fa207efc640 1 -- 172.21.15.83:0/1252656229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2000abc60 msgr2=0x7fa2001606f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:39.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.865+0000 7fa207efc640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2000abc60 0x7fa2001606f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00b3e0 tx=0x7fa1fc00b8a0 comp rx=0 tx=0).stop 2024-09-19T07:42:39.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.865+0000 7fa207efc640 1 -- 172.21.15.83:0/1252656229 shutdown_connections 2024-09-19T07:42:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.869+0000 7fa207efc640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1e40ab8d0 0x7fa1e40add90 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.869+0000 7fa207efc640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa200160c30 0x7fa2000c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.869+0000 7fa207efc640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2000abc60 0x7fa2001606f0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.869+0000 7fa207efc640 1 --2- 172.21.15.83:0/1252656229 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2000739c0 0x7fa2000bd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.869+0000 7fa207efc640 1 -- 172.21.15.83:0/1252656229 >> 172.21.15.83:0/1252656229 conn(0x7fa200075390 msgr2=0x7fa2001546e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.869+0000 7fa207efc640 1 -- 172.21.15.83:0/1252656229 shutdown_connections 2024-09-19T07:42:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:39.869+0000 7fa207efc640 1 -- 172.21.15.83:0/1252656229 wait complete. 2024-09-19T07:42:39.896 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-19T07:42:39.896 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:42:39.896 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:40.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f2266e640 1 Processor -- start 2024-09-19T07:42:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f2266e640 1 -- start start 2024-09-19T07:42:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f2266e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c15d9e0 0x7f6f1c06e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f2266e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6f1c15e220 0x7f6f1c1574f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f2266e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6f1c157a30 0x7f6f1c1582b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f2266e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6f1c0ab020 con 0x7f6f1c15e220 2024-09-19T07:42:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f2266e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f6f1c0ab5e0 con 0x7f6f1c157a30 2024-09-19T07:42:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f2266e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6f1c0c2170 con 0x7f6f1c15d9e0 2024-09-19T07:42:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f20c0b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6f1c157a30 0x7f6f1c1582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f20c0b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f6f1c157a30 0x7f6f1c1582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:55406/0 (socket says 172.21.15.83:55406) 2024-09-19T07:42:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f20c0b640 1 -- 172.21.15.83:0/1570537588 learned_addr learned my addr 172.21.15.83:0/1570537588 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f1bfff640 1 --2- 172.21.15.83:0/1570537588 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c15d9e0 0x7f6f1c06e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:40.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f1b7fe640 1 --2- 172.21.15.83:0/1570537588 >> v2:172.21.15.83:3300/0 conn(0x7f6f1c15e220 0x7f6f1c1574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:40.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f1bfff640 1 -- 172.21.15.83:0/1570537588 >> v2:172.21.15.184:3300/0 conn(0x7f6f1c157a30 msgr2=0x7f6f1c1582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:40.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f1bfff640 1 --2- 172.21.15.83:0/1570537588 >> v2:172.21.15.184:3300/0 conn(0x7f6f1c157a30 0x7f6f1c1582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:40.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f1bfff640 1 -- 172.21.15.83:0/1570537588 >> v2:172.21.15.83:3300/0 conn(0x7f6f1c15e220 msgr2=0x7f6f1c1574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:40.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f1bfff640 1 --2- 172.21.15.83:0/1570537588 >> v2:172.21.15.83:3300/0 conn(0x7f6f1c15e220 0x7f6f1c1574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:40.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.089+0000 7f6f1bfff640 1 -- 172.21.15.83:0/1570537588 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1c0c2450 con 0x7f6f1c15d9e0 2024-09-19T07:42:40.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.093+0000 7f6f1b7fe640 1 --2- 172.21.15.83:0/1570537588 >> v2:172.21.15.83:3300/0 conn(0x7f6f1c15e220 0x7f6f1c1574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:42:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.097+0000 7f6f1bfff640 1 --2- 172.21.15.83:0/1570537588 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c15d9e0 0x7f6f1c06e710 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c070e30 tx=0x7f6f1401c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b73cf831aaa41d9a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:40.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.097+0000 7f6f1affd640 1 -- 172.21.15.83:0/1570537588 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f140b2070 con 0x7f6f1c15d9e0 2024-09-19T07:42:40.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.097+0000 7f6f1affd640 1 -- 172.21.15.83:0/1570537588 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f140a5830 con 0x7f6f1c15d9e0 2024-09-19T07:42:40.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.097+0000 7f6f1affd640 1 -- 172.21.15.83:0/1570537588 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f140a5c00 con 0x7f6f1c15d9e0 2024-09-19T07:42:40.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.101+0000 7f6f2266e640 1 -- 172.21.15.83:0/1570537588 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c15d9e0 msgr2=0x7f6f1c06e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:40.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.101+0000 7f6f2266e640 1 --2- 172.21.15.83:0/1570537588 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c15d9e0 0x7f6f1c06e710 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c070e30 tx=0x7f6f1401c8c0 comp rx=0 tx=0).stop 2024-09-19T07:42:40.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.101+0000 7f6f2266e640 1 -- 172.21.15.83:0/1570537588 shutdown_connections 2024-09-19T07:42:40.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.101+0000 7f6f2266e640 1 --2- 172.21.15.83:0/1570537588 >> v2:172.21.15.184:3300/0 conn(0x7f6f1c157a30 0x7f6f1c1582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:40.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.101+0000 7f6f2266e640 1 --2- 172.21.15.83:0/1570537588 >> v2:172.21.15.83:3300/0 conn(0x7f6f1c15e220 0x7f6f1c1574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:40.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.101+0000 7f6f2266e640 1 --2- 172.21.15.83:0/1570537588 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c15d9e0 0x7f6f1c06e710 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c070e30 tx=0x7f6f1401c8c0 comp rx=0 tx=0).stop 2024-09-19T07:42:40.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.101+0000 7f6f2266e640 1 -- 172.21.15.83:0/1570537588 >> 172.21.15.83:0/1570537588 conn(0x7f6f1c075390 msgr2=0x7f6f1c06bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:40.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.101+0000 7f6f2266e640 1 -- 172.21.15.83:0/1570537588 shutdown_connections 2024-09-19T07:42:40.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.101+0000 7f6f2266e640 1 -- 172.21.15.83:0/1570537588 wait complete. 2024-09-19T07:42:40.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.101+0000 7f6f2266e640 1 Processor -- start 2024-09-19T07:42:40.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.101+0000 7f6f2266e640 1 -- start start 2024-09-19T07:42:40.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f2266e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f1c157a30 0x7f6f1c148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:40.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f2266e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f1c15e220 0x7f6f1c163040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:40.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f2266e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c0c7d70 0x7f6f1c163580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:40.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f2266e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c15a190 con 0x7f6f1c15e220 2024-09-19T07:42:40.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f2266e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c15e010 con 0x7f6f1c157a30 2024-09-19T07:42:40.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f2266e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6f1c15a310 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f20c0b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c0c7d70 0x7f6f1c163580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:40.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f20c0b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c0c7d70 0x7f6f1c163580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:51522/0 (socket says 172.21.15.83:51522) 2024-09-19T07:42:40.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f20c0b640 1 -- 172.21.15.83:0/1550081170 learned_addr learned my addr 172.21.15.83:0/1550081170 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:40.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f1bfff640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f1c157a30 0x7f6f1c148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:40.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f20c0b640 1 -- 172.21.15.83:0/1550081170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f1c157a30 msgr2=0x7f6f1c148f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:40.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f20c0b640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f1c157a30 0x7f6f1c148f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:40.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f20c0b640 1 -- 172.21.15.83:0/1550081170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f1c15e220 msgr2=0x7f6f1c163040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:40.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f1b7fe640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f1c15e220 0x7f6f1c163040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:40.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f20c0b640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f1c15e220 0x7f6f1c163040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:40.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f20c0b640 1 -- 172.21.15.83:0/1550081170 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1c0c71c0 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.105+0000 7f6f1b7fe640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f1c15e220 0x7f6f1c163040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:42:40.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.109+0000 7f6f20c0b640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c0c7d70 0x7f6f1c163580 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e890 tx=0x7f6f0c00ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:40.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.109+0000 7f6f18ff9640 1 -- 172.21.15.83:0/1550081170 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c017070 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.109+0000 7f6f18ff9640 1 -- 172.21.15.83:0/1550081170 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f0c00c8e0 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.109+0000 7f6f2266e640 1 -- 172.21.15.83:0/1550081170 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f1c1498b0 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.109+0000 7f6f18ff9640 1 -- 172.21.15.83:0/1550081170 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c00ccd0 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.109+0000 7f6f2266e640 1 -- 172.21.15.83:0/1550081170 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f1c071a10 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.109+0000 7f6f18ff9640 1 -- 172.21.15.83:0/1550081170 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6f0c01a070 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.109+0000 7f6f027fc640 1 -- 172.21.15.83:0/1550081170 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef40052a0 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.109+0000 7f6f18ff9640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f08067910 0x7f6f08069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:40.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.113+0000 7f6f1bfff640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f08067910 0x7f6f08069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:40.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.113+0000 7f6f1bfff640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f08067910 0x7f6f08069dd0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6f140af000 tx=0x7f6f140a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:40.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.113+0000 7f6f18ff9640 1 -- 172.21.15.83:0/1550081170 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 9475+0+0 (secure 0 0 0) 0x7f6f0c08a060 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.121+0000 7f6f18ff9640 1 -- 172.21.15.83:0/1550081170 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6f0c055820 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.611 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:40.611 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":167,"btime":"2024-09-19T07:42:40:513527+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:39.624511+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:42:40.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.609+0000 7f6f027fc640 1 -- 172.21.15.83:0/1550081170 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ef4005860 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.611 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 167 2024-09-19T07:42:40.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.609+0000 7f6f18ff9640 1 -- 172.21.15.83:0/1550081170 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+5714 (secure 0 0 0) 0x7f6f0c059b20 con 0x7f6f1c0c7d70 2024-09-19T07:42:40.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.613+0000 7f6f027fc640 1 -- 172.21.15.83:0/1550081170 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f08067910 msgr2=0x7f6f08069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:40.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.613+0000 7f6f027fc640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f08067910 0x7f6f08069dd0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6f140af000 tx=0x7f6f140a7070 comp rx=0 tx=0).stop 2024-09-19T07:42:40.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.613+0000 7f6f027fc640 1 -- 172.21.15.83:0/1550081170 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c0c7d70 msgr2=0x7f6f1c163580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:40.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.613+0000 7f6f027fc640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c0c7d70 0x7f6f1c163580 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e890 tx=0x7f6f0c00ed50 comp rx=0 tx=0).stop 2024-09-19T07:42:40.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.613+0000 7f6f027fc640 1 -- 172.21.15.83:0/1550081170 shutdown_connections 2024-09-19T07:42:40.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.613+0000 7f6f027fc640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f6f08067910 0x7f6f08069dd0 secure :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6f140af000 tx=0x7f6f140a7070 comp rx=0 tx=0).stop 2024-09-19T07:42:40.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.613+0000 7f6f027fc640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6f1c0c7d70 0x7f6f1c163580 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:40.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.613+0000 7f6f027fc640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f1c15e220 0x7f6f1c163040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:40.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.613+0000 7f6f027fc640 1 --2- 172.21.15.83:0/1550081170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f1c157a30 0x7f6f1c148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:40.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.613+0000 7f6f027fc640 1 -- 172.21.15.83:0/1550081170 >> 172.21.15.83:0/1550081170 conn(0x7f6f1c075390 msgr2=0x7f6f1c153a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:40.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.613+0000 7f6f027fc640 1 -- 172.21.15.83:0/1550081170 shutdown_connections 2024-09-19T07:42:40.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:40.613+0000 7f6f027fc640 1 -- 172.21.15.83:0/1550081170 wait complete. 2024-09-19T07:42:40.653 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:42:40:513527+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 167, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 164, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6565": { "addr": "172.21.15.184:6833/3311721080", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 3311721080, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 3311721080, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6565, "incarnation": 161, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:42:11.447678+0000", "qdb_cluster": [ 6565 ], "qdb_leader": 6565, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6565 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 167, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6628": { "addr": "172.21.15.83:6835/1429520945", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 1429520945, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 1429520945, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6628, "incarnation": 166, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:42:39.624511+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6628 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:42:42.654 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:42.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f9928d94640 1 Processor -- start 2024-09-19T07:42:42.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f9928d94640 1 -- start start 2024-09-19T07:42:42.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f9928d94640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f99240751f0 0x7f9924152f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:42.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f9928d94640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f99240c3c50 0x7f9924153440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:42.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f9928d94640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9924155f10 con 0x7f99240c19c0 2024-09-19T07:42:42.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f9928d94640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f99241561f0 con 0x7f99240c3c50 2024-09-19T07:42:42.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f9928d94640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f99241564d0 con 0x7f99240751f0 2024-09-19T07:42:42.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f992259c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f99240751f0 0x7f9924152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:42.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f992259c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f99240751f0 0x7f9924152f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:52926/0 (socket says 172.21.15.83:52926) 2024-09-19T07:42:42.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f992259c640 1 -- 172.21.15.83:0/2901741953 learned_addr learned my addr 172.21.15.83:0/2901741953 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:42.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f9922d9d640 1 --1- 172.21.15.83:0/2901741953 >> v1:172.21.15.83:6789/0 conn(0x7f99240c19c0 0x7f992406ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50312/0 (socket says 172.21.15.83:50312) 2024-09-19T07:42:42.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f992259c640 1 -- 172.21.15.83:0/2901741953 >> v2:172.21.15.83:3300/0 conn(0x7f99240c3c50 msgr2=0x7f9924153440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:42:42.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f992259c640 1 --2- 172.21.15.83:0/2901741953 >> v2:172.21.15.83:3300/0 conn(0x7f99240c3c50 0x7f9924153440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:42.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f992259c640 1 -- 172.21.15.83:0/2901741953 >> v1:172.21.15.83:6789/0 conn(0x7f99240c19c0 legacy=0x7f992406ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:42.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.873+0000 7f992259c640 1 -- 172.21.15.83:0/2901741953 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99241567b0 con 0x7f99240751f0 2024-09-19T07:42:42.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.877+0000 7f992259c640 1 --2- 172.21.15.83:0/2901741953 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99240751f0 0x7f9924152f00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f99240c9510 tx=0x7f991801c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c53221092768825d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:42.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.877+0000 7f992159a640 1 -- 172.21.15.83:0/2901741953 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99180b0070 con 0x7f99240751f0 2024-09-19T07:42:42.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.881+0000 7f992159a640 1 -- 172.21.15.83:0/2901741953 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99180ab600 con 0x7f99240751f0 2024-09-19T07:42:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.881+0000 7f992159a640 1 -- 172.21.15.83:0/2901741953 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99180ab9d0 con 0x7f99240751f0 2024-09-19T07:42:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.881+0000 7f9928d94640 1 -- 172.21.15.83:0/2901741953 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99240751f0 msgr2=0x7f9924152f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.881+0000 7f9928d94640 1 --2- 172.21.15.83:0/2901741953 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99240751f0 0x7f9924152f00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f99240c9510 tx=0x7f991801c4b0 comp rx=0 tx=0).stop 2024-09-19T07:42:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.881+0000 7f9928d94640 1 -- 172.21.15.83:0/2901741953 shutdown_connections 2024-09-19T07:42:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.881+0000 7f9928d94640 1 --2- 172.21.15.83:0/2901741953 >> v2:172.21.15.83:3300/0 conn(0x7f99240c3c50 0x7f9924153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:42.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.881+0000 7f9928d94640 1 --2- 172.21.15.83:0/2901741953 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99240751f0 0x7f9924152f00 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:42.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.881+0000 7f9928d94640 1 -- 172.21.15.83:0/2901741953 >> 172.21.15.83:0/2901741953 conn(0x7f99240aaaa0 msgr2=0x7f99240c8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:42.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.881+0000 7f9928d94640 1 -- 172.21.15.83:0/2901741953 shutdown_connections 2024-09-19T07:42:42.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.881+0000 7f9928d94640 1 -- 172.21.15.83:0/2901741953 wait complete. 2024-09-19T07:42:42.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9928d94640 1 Processor -- start 2024-09-19T07:42:42.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9928d94640 1 -- start start 2024-09-19T07:42:42.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9928d94640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99240c19c0 0x7f992414d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:42.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9928d94640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99240c3c50 0x7f99240c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:42.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9928d94640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f992414bf90 0x7f992414c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:42.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9928d94640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9924163120 con 0x7f992414bf90 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9928d94640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9924162fa0 con 0x7f99240c3c50 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9928d94640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f99241632a0 con 0x7f99240c19c0 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9922d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f992414bf90 0x7f992414c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9922d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f992414bf90 0x7f992414c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53948/0 (socket says 172.21.15.83:53948) 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9922d9d640 1 -- 172.21.15.83:0/2691724562 learned_addr learned my addr 172.21.15.83:0/2691724562 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f992259c640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99240c19c0 0x7f992414d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9922d9d640 1 -- 172.21.15.83:0/2691724562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99240c19c0 msgr2=0x7f992414d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9921d9b640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99240c3c50 0x7f99240c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9922d9d640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99240c19c0 0x7f992414d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9922d9d640 1 -- 172.21.15.83:0/2691724562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99240c3c50 msgr2=0x7f99240c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9922d9d640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99240c3c50 0x7f99240c7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.885+0000 7f9922d9d640 1 -- 172.21.15.83:0/2691724562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99240733e0 con 0x7f992414bf90 2024-09-19T07:42:42.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.889+0000 7f9922d9d640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f992414bf90 0x7f992414c420 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f991402e430 tx=0x7f991402e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:42.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.889+0000 7f9912ffd640 1 -- 172.21.15.83:0/2691724562 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99140072e0 con 0x7f992414bf90 2024-09-19T07:42:42.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.889+0000 7f9912ffd640 1 -- 172.21.15.83:0/2691724562 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9914007480 con 0x7f992414bf90 2024-09-19T07:42:42.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.889+0000 7f9912ffd640 1 -- 172.21.15.83:0/2691724562 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99140047f0 con 0x7f992414bf90 2024-09-19T07:42:42.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.889+0000 7f9928d94640 1 -- 172.21.15.83:0/2691724562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9924073680 con 0x7f992414bf90 2024-09-19T07:42:42.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.889+0000 7f9928d94640 1 -- 172.21.15.83:0/2691724562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9924150340 con 0x7f992414bf90 2024-09-19T07:42:42.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.889+0000 7f9928d94640 1 -- 172.21.15.83:0/2691724562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f992406ecc0 con 0x7f992414bf90 2024-09-19T07:42:42.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.893+0000 7f9912ffd640 1 -- 172.21.15.83:0/2691724562 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f991403c020 con 0x7f992414bf90 2024-09-19T07:42:42.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.893+0000 7f9912ffd640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f99040678c0 0x7f9904069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:42.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.893+0000 7f9912ffd640 1 -- 172.21.15.83:0/2691724562 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9475+0+0 (secure 0 0 0) 0x7f99140a8090 con 0x7f992414bf90 2024-09-19T07:42:42.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.897+0000 7f992259c640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f99040678c0 0x7f9904069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:42.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.897+0000 7f992259c640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f99040678c0 0x7f9904069d80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f99180ad000 tx=0x7f99180a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:42.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:42.897+0000 7f9912ffd640 1 -- 172.21.15.83:0/2691724562 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f991406f090 con 0x7f992414bf90 2024-09-19T07:42:43.475 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:42:43.477+0000 7f54029a4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:42:43.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.481+0000 7f9928d94640 1 -- 172.21.15.83:0/2691724562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99240742b0 con 0x7f992414bf90 2024-09-19T07:42:43.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.481+0000 7f9912ffd640 1 -- 172.21.15.83:0/2691724562 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) ==== 95+0+5719 (secure 0 0 0) 0x7f9914077c60 con 0x7f992414bf90 2024-09-19T07:42:43.484 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:43.484 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":169,"btime":"2024-09-19T07:42:42:535415+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:42:43.485 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 169 2024-09-19T07:42:43.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.485+0000 7f9928d94640 1 -- 172.21.15.83:0/2691724562 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f99040678c0 msgr2=0x7f9904069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:43.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.485+0000 7f9928d94640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f99040678c0 0x7f9904069d80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f99180ad000 tx=0x7f99180a6000 comp rx=0 tx=0).stop 2024-09-19T07:42:43.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.485+0000 7f9928d94640 1 -- 172.21.15.83:0/2691724562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f992414bf90 msgr2=0x7f992414c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:43.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.485+0000 7f9928d94640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f992414bf90 0x7f992414c420 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f991402e430 tx=0x7f991402e8f0 comp rx=0 tx=0).stop 2024-09-19T07:42:43.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.489+0000 7f9928d94640 1 -- 172.21.15.83:0/2691724562 shutdown_connections 2024-09-19T07:42:43.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.489+0000 7f9928d94640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f99040678c0 0x7f9904069d80 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:43.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.489+0000 7f9928d94640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f992414bf90 0x7f992414c420 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:43.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.489+0000 7f9928d94640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99240c3c50 0x7f99240c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:43.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.489+0000 7f9928d94640 1 --2- 172.21.15.83:0/2691724562 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99240c19c0 0x7f992414d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:43.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.489+0000 7f9928d94640 1 -- 172.21.15.83:0/2691724562 >> 172.21.15.83:0/2691724562 conn(0x7f99240aaaa0 msgr2=0x7f992415fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:43.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.489+0000 7f9928d94640 1 -- 172.21.15.83:0/2691724562 shutdown_connections 2024-09-19T07:42:43.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:43.489+0000 7f9928d94640 1 -- 172.21.15.83:0/2691724562 wait complete. 2024-09-19T07:42:43.520 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=7503 2024-09-19T07:42:43.520 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before reviving mds.c 2024-09-19T07:42:52.521 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-19T07:42:52.521 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:42:52.522 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:42:52.524 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:42:52.525 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:52.793 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:42:52.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.845+0000 7f2d3551a640 1 Processor -- start 2024-09-19T07:42:52.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.845+0000 7f2d3551a640 1 -- start start 2024-09-19T07:42:52.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.849+0000 7f2d3551a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d280dee00 0x7f2d28046da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:52.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.849+0000 7f2d3551a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2d28047310 0x7f2d2800bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:52.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.849+0000 7f2d3551a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2d2800c2f0 0x7f2d280e1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:52.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.849+0000 7f2d3551a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f2d2803fd10 con 0x7f2d2800c2f0 2024-09-19T07:42:52.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.849+0000 7f2d3551a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f2d2803f750 con 0x7f2d28047310 2024-09-19T07:42:52.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.849+0000 7f2d3551a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2d2803fa30 con 0x7f2d280dee00 2024-09-19T07:42:52.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.853+0000 7f2d2ffff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d280dee00 0x7f2d28046da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:52.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.853+0000 7f2d34d19640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2d2800c2f0 0x7f2d280e1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:52.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.853+0000 7f2d34d19640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2d2800c2f0 0x7f2d280e1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56094/0 (socket says 172.21.15.83:56094) 2024-09-19T07:42:52.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.853+0000 7f2d34d19640 1 -- 172.21.15.83:0/2553643183 learned_addr learned my addr 172.21.15.83:0/2553643183 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:52.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.853+0000 7f2d34d19640 1 -- 172.21.15.83:0/2553643183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d280dee00 msgr2=0x7f2d28046da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:52.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.853+0000 7f2d34d19640 1 --2- 172.21.15.83:0/2553643183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d280dee00 0x7f2d28046da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:52.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.853+0000 7f2d34d19640 1 -- 172.21.15.83:0/2553643183 >> v2:172.21.15.184:3300/0 conn(0x7f2d28047310 msgr2=0x7f2d2800bdb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:42:52.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.853+0000 7f2d34d19640 1 --2- 172.21.15.83:0/2553643183 >> v2:172.21.15.184:3300/0 conn(0x7f2d28047310 0x7f2d2800bdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:52.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.853+0000 7f2d34d19640 1 -- 172.21.15.83:0/2553643183 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d2804b5f0 con 0x7f2d2800c2f0 2024-09-19T07:42:52.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d34d19640 1 --2- 172.21.15.83:0/2553643183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d2800c2f0 0x7f2d280e1720 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f2d3007bc70 tx=0x7f2d30098bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408842bae18a0233 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:52.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d2effd640 1 -- 172.21.15.83:0/2553643183 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30121790 con 0x7f2d2800c2f0 2024-09-19T07:42:52.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d2effd640 1 -- 172.21.15.83:0/2553643183 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d30121970 con 0x7f2d2800c2f0 2024-09-19T07:42:52.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d2effd640 1 -- 172.21.15.83:0/2553643183 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30121e20 con 0x7f2d2800c2f0 2024-09-19T07:42:52.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d3551a640 1 -- 172.21.15.83:0/2553643183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d2800c2f0 msgr2=0x7f2d280e1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:52.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d3551a640 1 --2- 172.21.15.83:0/2553643183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d2800c2f0 0x7f2d280e1720 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f2d3007bc70 tx=0x7f2d30098bb0 comp rx=0 tx=0).stop 2024-09-19T07:42:52.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d3551a640 1 -- 172.21.15.83:0/2553643183 shutdown_connections 2024-09-19T07:42:52.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d3551a640 1 --2- 172.21.15.83:0/2553643183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d2800c2f0 0x7f2d280e1720 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:52.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d3551a640 1 --2- 172.21.15.83:0/2553643183 >> v2:172.21.15.184:3300/0 conn(0x7f2d28047310 0x7f2d2800bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:52.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d3551a640 1 --2- 172.21.15.83:0/2553643183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d280dee00 0x7f2d28046da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:52.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d3551a640 1 -- 172.21.15.83:0/2553643183 >> 172.21.15.83:0/2553643183 conn(0x7f2d28048e30 msgr2=0x7f2d28049240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:52.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d3551a640 1 -- 172.21.15.83:0/2553643183 shutdown_connections 2024-09-19T07:42:52.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.857+0000 7f2d3551a640 1 -- 172.21.15.83:0/2553643183 wait complete. 2024-09-19T07:42:52.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.861+0000 7f2d3551a640 1 Processor -- start 2024-09-19T07:42:52.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.861+0000 7f2d3551a640 1 -- start start 2024-09-19T07:42:52.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d3551a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d2800c2f0 0x7f2d2800b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:52.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d3551a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d28047310 0x7f2d2817a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:52.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d3551a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d280dee00 0x7f2d2817a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:52.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d3551a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2804aac0 con 0x7f2d28047310 2024-09-19T07:42:52.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d3551a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2804c0d0 con 0x7f2d280dee00 2024-09-19T07:42:52.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d3551a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2d2804ac40 con 0x7f2d2800c2f0 2024-09-19T07:42:52.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d2ffff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d2800c2f0 0x7f2d2800b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:52.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d2ffff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d2800c2f0 0x7f2d2800b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:37276/0 (socket says 172.21.15.83:37276) 2024-09-19T07:42:52.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d2ffff640 1 -- 172.21.15.83:0/2184039446 learned_addr learned my addr 172.21.15.83:0/2184039446 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:52.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d2ffff640 1 -- 172.21.15.83:0/2184039446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d280dee00 msgr2=0x7f2d2817a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:42:52.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d2ffff640 1 --2- 172.21.15.83:0/2184039446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d280dee00 0x7f2d2817a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:52.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d2ffff640 1 -- 172.21.15.83:0/2184039446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d28047310 msgr2=0x7f2d2817a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:52.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d2ffff640 1 --2- 172.21.15.83:0/2184039446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d28047310 0x7f2d2817a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:52.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d2ffff640 1 -- 172.21.15.83:0/2184039446 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d280d8c60 con 0x7f2d2800c2f0 2024-09-19T07:42:52.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d2ffff640 1 --2- 172.21.15.83:0/2184039446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d2800c2f0 0x7f2d2800b000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400d620 tx=0x7f2d2400dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:52.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d2cff9640 1 -- 172.21.15.83:0/2184039446 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2400b620 con 0x7f2d2800c2f0 2024-09-19T07:42:52.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d2cff9640 1 -- 172.21.15.83:0/2184039446 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d24019070 con 0x7f2d2800c2f0 2024-09-19T07:42:52.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d2cff9640 1 -- 172.21.15.83:0/2184039446 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2400b9c0 con 0x7f2d2800c2f0 2024-09-19T07:42:52.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d3551a640 1 -- 172.21.15.83:0/2184039446 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d28040970 con 0x7f2d2800c2f0 2024-09-19T07:42:52.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.865+0000 7f2d3551a640 1 -- 172.21.15.83:0/2184039446 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d28179d80 con 0x7f2d2800c2f0 2024-09-19T07:42:52.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.869+0000 7f2d2cff9640 1 -- 172.21.15.83:0/2184039446 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2d2400bb60 con 0x7f2d2800c2f0 2024-09-19T07:42:52.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.869+0000 7f2d3551a640 1 -- 172.21.15.83:0/2184039446 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d28179f80 con 0x7f2d2800c2f0 2024-09-19T07:42:52.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.869+0000 7f2d2cff9640 1 --2- 172.21.15.83:0/2184039446 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d140678c0 0x7f2d14069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:52.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.869+0000 7f2d2cff9640 1 -- 172.21.15.83:0/2184039446 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 9475+0+0 (secure 0 0 0) 0x7f2d24086fb0 con 0x7f2d2800c2f0 2024-09-19T07:42:52.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.877+0000 7f2d2f7fe640 1 --2- 172.21.15.83:0/2184039446 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d140678c0 0x7f2d14069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:52.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.881+0000 7f2d2cff9640 1 -- 172.21.15.83:0/2184039446 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2d28179f80 con 0x7f2d2800c2f0 2024-09-19T07:42:52.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:52.881+0000 7f2d2f7fe640 1 --2- 172.21.15.83:0/2184039446 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d140678c0 0x7f2d14069d80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2d280e1ad0 tx=0x7f2d1c006c50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:53.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.377+0000 7f2d3551a640 1 -- 172.21.15.83:0/2184039446 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d2804b2c0 con 0x7f2d2800c2f0 2024-09-19T07:42:53.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.381+0000 7f2d2cff9640 1 -- 172.21.15.83:0/2184039446 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) ==== 95+0+5719 (secure 0 0 0) 0x7f2d24056c10 con 0x7f2d2800c2f0 2024-09-19T07:42:53.380 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:53.381 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":169,"btime":"2024-09-19T07:42:42:535415+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:42:53.381 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 169 2024-09-19T07:42:53.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.385+0000 7f2d1a7fc640 1 -- 172.21.15.83:0/2184039446 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d140678c0 msgr2=0x7f2d14069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:53.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.385+0000 7f2d1a7fc640 1 --2- 172.21.15.83:0/2184039446 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d140678c0 0x7f2d14069d80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2d280e1ad0 tx=0x7f2d1c006c50 comp rx=0 tx=0).stop 2024-09-19T07:42:53.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.385+0000 7f2d1a7fc640 1 -- 172.21.15.83:0/2184039446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d2800c2f0 msgr2=0x7f2d2800b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:53.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.385+0000 7f2d1a7fc640 1 --2- 172.21.15.83:0/2184039446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d2800c2f0 0x7f2d2800b000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400d620 tx=0x7f2d2400dae0 comp rx=0 tx=0).stop 2024-09-19T07:42:53.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.385+0000 7f2d1a7fc640 1 -- 172.21.15.83:0/2184039446 shutdown_connections 2024-09-19T07:42:53.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.385+0000 7f2d1a7fc640 1 --2- 172.21.15.83:0/2184039446 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d140678c0 0x7f2d14069d80 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:53.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.385+0000 7f2d1a7fc640 1 --2- 172.21.15.83:0/2184039446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d280dee00 0x7f2d2817a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:53.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.385+0000 7f2d1a7fc640 1 --2- 172.21.15.83:0/2184039446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d28047310 0x7f2d2817a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:53.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.385+0000 7f2d1a7fc640 1 --2- 172.21.15.83:0/2184039446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d2800c2f0 0x7f2d2800b000 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:53.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.385+0000 7f2d1a7fc640 1 -- 172.21.15.83:0/2184039446 >> 172.21.15.83:0/2184039446 conn(0x7f2d28048e30 msgr2=0x7f2d28007cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:53.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.385+0000 7f2d1a7fc640 1 -- 172.21.15.83:0/2184039446 shutdown_connections 2024-09-19T07:42:53.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:53.385+0000 7f2d1a7fc640 1 -- 172.21.15.83:0/2184039446 wait complete. 2024-09-19T07:42:53.479 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-19T07:42:55.481 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:42:58.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.740+0000 7f2d261fb640 1 Processor -- start 2024-09-19T07:42:58.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.740+0000 7f2d261fb640 1 -- start start 2024-09-19T07:42:58.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d261fb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1803f5e0 0x7f2d180de8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:58.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d261fb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2d18008880 0x7f2d180dede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:58.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d261fb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d18049b70 con 0x7f2d18008330 2024-09-19T07:42:58.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d261fb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f2d18049920 con 0x7f2d18008880 2024-09-19T07:42:58.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d261fb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2d18046d30 con 0x7f2d1803f5e0 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d251f9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1803f5e0 0x7f2d180de8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d251f9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1803f5e0 0x7f2d180de8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:37282/0 (socket says 172.21.15.83:37282) 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d251f9640 1 -- 172.21.15.83:0/2026060332 learned_addr learned my addr 172.21.15.83:0/2026060332 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d249f8640 1 --2- 172.21.15.83:0/2026060332 >> v2:172.21.15.184:3300/0 conn(0x7f2d18008880 0x7f2d180dede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d251f9640 1 -- 172.21.15.83:0/2026060332 >> v2:172.21.15.184:3300/0 conn(0x7f2d18008880 msgr2=0x7f2d180dede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d251f9640 1 --2- 172.21.15.83:0/2026060332 >> v2:172.21.15.184:3300/0 conn(0x7f2d18008880 0x7f2d180dede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d251f9640 1 -- 172.21.15.83:0/2026060332 >> v1:172.21.15.184:6789/0 conn(0x7f2d18008330 legacy=0x7f2d18007360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d251f9640 1 -- 172.21.15.83:0/2026060332 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d18046f80 con 0x7f2d1803f5e0 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.744+0000 7f2d249f8640 1 --2- 172.21.15.83:0/2026060332 >> v2:172.21.15.184:3300/0 conn(0x7f2d18008880 0x7f2d180dede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.748+0000 7f2d251f9640 1 --2- 172.21.15.83:0/2026060332 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1803f5e0 0x7f2d180de8a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d2007bc70 tx=0x7f2d20098c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29bf08c0d2b70cc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.748+0000 7f2d17fff640 1 -- 172.21.15.83:0/2026060332 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d20121820 con 0x7f2d1803f5e0 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.748+0000 7f2d17fff640 1 -- 172.21.15.83:0/2026060332 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d20121a20 con 0x7f2d1803f5e0 2024-09-19T07:42:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.748+0000 7f2d17fff640 1 -- 172.21.15.83:0/2026060332 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d20129440 con 0x7f2d1803f5e0 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.748+0000 7f2d261fb640 1 -- 172.21.15.83:0/2026060332 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1803f5e0 msgr2=0x7f2d180de8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.748+0000 7f2d261fb640 1 --2- 172.21.15.83:0/2026060332 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1803f5e0 0x7f2d180de8a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d2007bc70 tx=0x7f2d20098c40 comp rx=0 tx=0).stop 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.748+0000 7f2d261fb640 1 -- 172.21.15.83:0/2026060332 shutdown_connections 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.748+0000 7f2d261fb640 1 --2- 172.21.15.83:0/2026060332 >> v2:172.21.15.184:3300/0 conn(0x7f2d18008880 0x7f2d180dede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.748+0000 7f2d261fb640 1 --2- 172.21.15.83:0/2026060332 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d1803f5e0 0x7f2d180de8a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.748+0000 7f2d261fb640 1 -- 172.21.15.83:0/2026060332 >> 172.21.15.83:0/2026060332 conn(0x7f2d18048e30 msgr2=0x7f2d18049210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.748+0000 7f2d261fb640 1 -- 172.21.15.83:0/2026060332 shutdown_connections 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.748+0000 7f2d261fb640 1 -- 172.21.15.83:0/2026060332 wait complete. 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.752+0000 7f2d261fb640 1 Processor -- start 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.752+0000 7f2d261fb640 1 -- start start 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.752+0000 7f2d261fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d18008330 0x7f2d1817a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.752+0000 7f2d261fb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d18008880 0x7f2d180d4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.752+0000 7f2d261fb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d1803f5e0 0x7f2d180d5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.752+0000 7f2d261fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1803fb40 con 0x7f2d18008330 2024-09-19T07:42:58.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.752+0000 7f2d261fb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1803f9c0 con 0x7f2d1803f5e0 2024-09-19T07:42:58.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.752+0000 7f2d261fb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2d180eb3c0 con 0x7f2d18008880 2024-09-19T07:42:58.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.752+0000 7f2d251f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d18008330 0x7f2d1817a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:58.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.752+0000 7f2d251f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d18008330 0x7f2d1817a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56108/0 (socket says 172.21.15.83:56108) 2024-09-19T07:42:58.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.752+0000 7f2d251f9640 1 -- 172.21.15.83:0/1461620378 learned_addr learned my addr 172.21.15.83:0/1461620378 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:42:58.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.752+0000 7f2d259fa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d1803f5e0 0x7f2d180d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:58.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.756+0000 7f2d251f9640 1 -- 172.21.15.83:0/1461620378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d18008880 msgr2=0x7f2d180d4e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:42:58.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.756+0000 7f2d251f9640 1 --2- 172.21.15.83:0/1461620378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d18008880 0x7f2d180d4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:58.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.756+0000 7f2d251f9640 1 -- 172.21.15.83:0/1461620378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d1803f5e0 msgr2=0x7f2d180d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:58.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.756+0000 7f2d251f9640 1 --2- 172.21.15.83:0/1461620378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d1803f5e0 0x7f2d180d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:58.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.756+0000 7f2d251f9640 1 -- 172.21.15.83:0/1461620378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d18000db0 con 0x7f2d18008330 2024-09-19T07:42:58.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.756+0000 7f2d251f9640 1 --2- 172.21.15.83:0/1461620378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d18008330 0x7f2d1817a0f0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f2d2002dd60 tx=0x7f2d20074b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:58.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.756+0000 7f2d15ffb640 1 -- 172.21.15.83:0/1461620378 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d20138070 con 0x7f2d18008330 2024-09-19T07:42:58.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.756+0000 7f2d15ffb640 1 -- 172.21.15.83:0/1461620378 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d20129a10 con 0x7f2d18008330 2024-09-19T07:42:58.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.756+0000 7f2d15ffb640 1 -- 172.21.15.83:0/1461620378 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d20129e00 con 0x7f2d18008330 2024-09-19T07:42:58.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.756+0000 7f2d261fb640 1 -- 172.21.15.83:0/1461620378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1817a940 con 0x7f2d18008330 2024-09-19T07:42:58.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.756+0000 7f2d261fb640 1 -- 172.21.15.83:0/1461620378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d180db210 con 0x7f2d18008330 2024-09-19T07:42:58.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.760+0000 7f2d037fe640 1 -- 172.21.15.83:0/1461620378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d18007360 con 0x7f2d18008330 2024-09-19T07:42:58.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.760+0000 7f2d15ffb640 1 -- 172.21.15.83:0/1461620378 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2d20133430 con 0x7f2d18008330 2024-09-19T07:42:58.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.760+0000 7f2d15ffb640 1 --2- 172.21.15.83:0/1461620378 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d040678c0 0x7f2d04069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:42:58.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.760+0000 7f2d15ffb640 1 -- 172.21.15.83:0/1461620378 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9475+0+0 (secure 0 0 0) 0x7f2d2019c9d0 con 0x7f2d18008330 2024-09-19T07:42:58.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.764+0000 7f2d249f8640 1 --2- 172.21.15.83:0/1461620378 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d040678c0 0x7f2d04069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:42:58.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.764+0000 7f2d249f8640 1 --2- 172.21.15.83:0/1461620378 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d040678c0 0x7f2d04069d80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f2d10006fd0 tx=0x7f2d10008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:42:58.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:57.772+0000 7f2d15ffb640 1 -- 172.21.15.83:0/1461620378 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2d200929d0 con 0x7f2d18008330 2024-09-19T07:42:58.246 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:42:58.246 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":170,"btime":"2024-09-19T07:42:53:467120+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7803,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:42:58.246 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 170 2024-09-19T07:42:58.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.244+0000 7f2d037fe640 1 -- 172.21.15.83:0/1461620378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d180db3e0 con 0x7f2d18008330 2024-09-19T07:42:58.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.244+0000 7f2d15ffb640 1 -- 172.21.15.83:0/1461620378 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) ==== 95+0+6545 (secure 0 0 0) 0x7f2d2016bfe0 con 0x7f2d18008330 2024-09-19T07:42:58.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.248+0000 7f2d037fe640 1 -- 172.21.15.83:0/1461620378 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d040678c0 msgr2=0x7f2d04069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:58.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.248+0000 7f2d037fe640 1 --2- 172.21.15.83:0/1461620378 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d040678c0 0x7f2d04069d80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f2d10006fd0 tx=0x7f2d10008040 comp rx=0 tx=0).stop 2024-09-19T07:42:58.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.248+0000 7f2d037fe640 1 -- 172.21.15.83:0/1461620378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d18008330 msgr2=0x7f2d1817a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:42:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.248+0000 7f2d037fe640 1 --2- 172.21.15.83:0/1461620378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d18008330 0x7f2d1817a0f0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f2d2002dd60 tx=0x7f2d20074b50 comp rx=0 tx=0).stop 2024-09-19T07:42:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.252+0000 7f2d037fe640 1 -- 172.21.15.83:0/1461620378 shutdown_connections 2024-09-19T07:42:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.252+0000 7f2d037fe640 1 --2- 172.21.15.83:0/1461620378 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d040678c0 0x7f2d04069d80 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.252+0000 7f2d037fe640 1 --2- 172.21.15.83:0/1461620378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d1803f5e0 0x7f2d180d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.252+0000 7f2d037fe640 1 --2- 172.21.15.83:0/1461620378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d18008880 0x7f2d180d4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.252+0000 7f2d037fe640 1 --2- 172.21.15.83:0/1461620378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d18008330 0x7f2d1817a0f0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:42:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.252+0000 7f2d037fe640 1 -- 172.21.15.83:0/1461620378 >> 172.21.15.83:0/1461620378 conn(0x7f2d18048e30 msgr2=0x7f2d180ddbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:42:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.252+0000 7f2d037fe640 1 -- 172.21.15.83:0/1461620378 shutdown_connections 2024-09-19T07:42:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:42:58.252+0000 7f2d037fe640 1 -- 172.21.15.83:0/1461620378 wait complete. 2024-09-19T07:42:58.627 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-19T07:42:58.627 INFO:tasks.mds_thrash.fs.[b]:waiting for 115 secs before thrashing 2024-09-19T07:43:06.749 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:43:06.751 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:43:36.887 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:43:36.890 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:43:39.155 INFO:tasks.workunit.client.1.smithi184.stdout:fs setup took 1339 secs 2024-09-19T07:43:43.142 INFO:tasks.workunit.client.1.smithi184.stdout:Syncing()...3 sec 2024-09-19T07:43:43.142 INFO:tasks.workunit.client.1.smithi184.stdout:Starting Actual Benchmark At: Thu Sep 19 07:43:43 2024 2024-09-19T07:43:43.143 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:44:05.054 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:44:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.419+0000 7f4496057640 1 Processor -- start 2024-09-19T07:44:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.419+0000 7f4496057640 1 -- start start 2024-09-19T07:44:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4496057640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f448800c100 0x7f4488008770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4496057640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448803f860 con 0x7f4488008cb0 2024-09-19T07:44:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4496057640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448803fb40 con 0x7f44880dec70 2024-09-19T07:44:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4496057640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f448803f620 con 0x7f448800c100 2024-09-19T07:44:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4495856640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4488008cb0 0x7f44880e1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60678/0 (socket says 172.21.15.83:60678) 2024-09-19T07:44:05.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4495856640 1 -- 172.21.15.83:0/3415915722 learned_addr learned my addr 172.21.15.83:0/3415915722 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:44:05.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 527309919 0 0) 0x7f448803f860 con 0x7f4488008cb0 2024-09-19T07:44:05.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f447c003530 con 0x7f4488008cb0 2024-09-19T07:44:05.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4172314994 0 0) 0x7f448803fb40 con 0x7f44880dec70 2024-09-19T07:44:05.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f448803f860 con 0x7f44880dec70 2024-09-19T07:44:05.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4494854640 1 --2- 172.21.15.83:0/3415915722 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f448800c100 0x7f4488008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:05.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1797300454 0 0) 0x7f448803f860 con 0x7f44880dec70 2024-09-19T07:44:05.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f448800c100 msgr2=0x7f4488008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:44:05.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 --2- 172.21.15.83:0/3415915722 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f448800c100 0x7f4488008770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:05.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 >> v1:172.21.15.83:6789/0 conn(0x7f4488008cb0 legacy=0x7f44880e1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f448804a500 con 0x7f44880dec70 2024-09-19T07:44:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f448c001680 con 0x7f44880dec70 2024-09-19T07:44:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 >> v1:172.21.15.184:6789/0 conn(0x7f44880dec70 legacy=0x7f448804aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 --2- 172.21.15.83:0/3415915722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f447c004bb0 0x7f447c004fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 --2- 172.21.15.83:0/3415915722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f447c0037b0 0x7f447c003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 --2- 172.21.15.83:0/3415915722 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f448800c100 0x7f447c0041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f448803f620 con 0x7f447c004bb0 2024-09-19T07:44:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f447c00bad0 con 0x7f447c0037b0 2024-09-19T07:44:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f447c00bcd0 con 0x7f448800c100 2024-09-19T07:44:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4495055640 1 --2- 172.21.15.83:0/3415915722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f447c004bb0 0x7f447c004fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4494854640 1 --2- 172.21.15.83:0/3415915722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f447c0037b0 0x7f447c003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:05.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4495856640 1 --2- 172.21.15.83:0/3415915722 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f448800c100 0x7f447c0041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:05.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4495055640 1 -- 172.21.15.83:0/3415915722 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f448800c100 msgr2=0x7f447c0041d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:44:05.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4495055640 1 --2- 172.21.15.83:0/3415915722 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f448800c100 0x7f447c0041d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:05.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4495055640 1 -- 172.21.15.83:0/3415915722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f447c0037b0 msgr2=0x7f447c003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:44:05.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4495055640 1 --2- 172.21.15.83:0/3415915722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f447c0037b0 0x7f447c003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:05.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.423+0000 7f4495055640 1 -- 172.21.15.83:0/3415915722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f448803f860 con 0x7f447c004bb0 2024-09-19T07:44:05.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.427+0000 7f4495055640 1 --2- 172.21.15.83:0/3415915722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f447c004bb0 0x7f447c004fb0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f448c007f90 tx=0x7f448c01cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c4e499a3e4dc209 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:05.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.427+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c0b2070 con 0x7f447c004bb0 2024-09-19T07:44:05.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.427+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f448c0a5910 con 0x7f447c004bb0 2024-09-19T07:44:05.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.427+0000 7f4487fff640 1 -- 172.21.15.83:0/3415915722 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c007bc0 con 0x7f447c004bb0 2024-09-19T07:44:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.435+0000 7f4496057640 1 -- 172.21.15.83:0/3415915722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f447c004bb0 msgr2=0x7f447c004fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.435+0000 7f4496057640 1 --2- 172.21.15.83:0/3415915722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f447c004bb0 0x7f447c004fb0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f448c007f90 tx=0x7f448c01cb20 comp rx=0 tx=0).stop 2024-09-19T07:44:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.435+0000 7f4495856640 1 -- 172.21.15.83:0/3415915722 reap_dead start 2024-09-19T07:44:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.439+0000 7f4496057640 1 -- 172.21.15.83:0/3415915722 shutdown_connections 2024-09-19T07:44:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.439+0000 7f4496057640 1 -- 172.21.15.83:0/3415915722 >> 172.21.15.83:0/3415915722 conn(0x7f4488048e30 msgr2=0x7f4488049240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:05.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4496057640 1 -- 172.21.15.83:0/3415915722 shutdown_connections 2024-09-19T07:44:05.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4496057640 1 -- 172.21.15.83:0/3415915722 wait complete. 2024-09-19T07:44:05.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4496057640 1 Processor -- start 2024-09-19T07:44:05.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4496057640 1 -- start start 2024-09-19T07:44:05.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4496057640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f447c004bb0 0x7f4488045c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:05.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4495055640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f447c004bb0 0x7f4488045c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:05.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4495055640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f447c004bb0 0x7f4488045c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57410/0 (socket says 172.21.15.83:57410) 2024-09-19T07:44:05.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4495055640 1 -- 172.21.15.83:0/957935742 learned_addr learned my addr 172.21.15.83:0/957935742 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:05.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4496057640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44880e0bd0 0x7f4488171cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:05.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4496057640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4488172200 0x7f448803feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:05.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4496057640 1 -- 172.21.15.83:0/957935742 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f44880376d0 con 0x7f447c004bb0 2024-09-19T07:44:05.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4496057640 1 -- 172.21.15.83:0/957935742 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4488037550 con 0x7f4488172200 2024-09-19T07:44:05.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4495856640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4488172200 0x7f448803feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:05.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.443+0000 7f4496057640 1 -- 172.21.15.83:0/957935742 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4488037850 con 0x7f44880e0bd0 2024-09-19T07:44:05.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.447+0000 7f4495856640 1 -- 172.21.15.83:0/957935742 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44880e0bd0 msgr2=0x7f4488171cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:44:05.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.447+0000 7f4495856640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44880e0bd0 0x7f4488171cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:05.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.447+0000 7f4495856640 1 -- 172.21.15.83:0/957935742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f447c004bb0 msgr2=0x7f4488045c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:05.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.447+0000 7f4495856640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f447c004bb0 0x7f4488045c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:05.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.447+0000 7f4495856640 1 -- 172.21.15.83:0/957935742 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4488045130 con 0x7f4488172200 2024-09-19T07:44:05.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.447+0000 7f4495856640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4488172200 0x7f448803feb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f44900515b0 tx=0x7f4490069900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:05.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.447+0000 7f4485ffb640 1 -- 172.21.15.83:0/957935742 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490069e00 con 0x7f4488172200 2024-09-19T07:44:05.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.447+0000 7f4485ffb640 1 -- 172.21.15.83:0/957935742 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4490069fa0 con 0x7f4488172200 2024-09-19T07:44:05.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.447+0000 7f4485ffb640 1 -- 172.21.15.83:0/957935742 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44900738d0 con 0x7f4488172200 2024-09-19T07:44:05.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.451+0000 7f4496057640 1 -- 172.21.15.83:0/957935742 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4488045410 con 0x7f4488172200 2024-09-19T07:44:05.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.451+0000 7f4496057640 1 -- 172.21.15.83:0/957935742 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4488042f60 con 0x7f4488172200 2024-09-19T07:44:05.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.455+0000 7f4496057640 1 -- 172.21.15.83:0/957935742 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4488047b50 con 0x7f4488172200 2024-09-19T07:44:05.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.463+0000 7f4485ffb640 1 -- 172.21.15.83:0/957935742 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4490073a70 con 0x7f4488172200 2024-09-19T07:44:05.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.463+0000 7f4485ffb640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44740679c0 0x7f4474069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:05.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.463+0000 7f4485ffb640 1 -- 172.21.15.83:0/957935742 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9475+0+0 (secure 0 0 0) 0x7f4490102960 con 0x7f4488172200 2024-09-19T07:44:05.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.463+0000 7f4495055640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44740679c0 0x7f4474069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:05.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.483+0000 7f4485ffb640 1 -- 172.21.15.83:0/957935742 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f449009b060 con 0x7f4488172200 2024-09-19T07:44:05.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:05.483+0000 7f4495055640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44740679c0 0x7f4474069e80 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f448c000c10 tx=0x7f448c005040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:06.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.375+0000 7f4496057640 1 -- 172.21.15.83:0/957935742 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4488171900 con 0x7f4488172200 2024-09-19T07:44:06.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.375+0000 7f4485ffb640 1 -- 172.21.15.83:0/957935742 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) ==== 95+0+6545 (secure 0 0 0) 0x7f44900d2640 con 0x7f4488172200 2024-09-19T07:44:06.380 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:44:06.380 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":170,"btime":"2024-09-19T07:42:53:467120+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7803,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:44:06.381 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 170 2024-09-19T07:44:06.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.383+0000 7f44737fe640 1 -- 172.21.15.83:0/957935742 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44740679c0 msgr2=0x7f4474069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:06.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.383+0000 7f44737fe640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44740679c0 0x7f4474069e80 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f448c000c10 tx=0x7f448c005040 comp rx=0 tx=0).stop 2024-09-19T07:44:06.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.383+0000 7f44737fe640 1 -- 172.21.15.83:0/957935742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4488172200 msgr2=0x7f448803feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:06.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.383+0000 7f44737fe640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4488172200 0x7f448803feb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f44900515b0 tx=0x7f4490069900 comp rx=0 tx=0).stop 2024-09-19T07:44:06.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.383+0000 7f44737fe640 1 -- 172.21.15.83:0/957935742 shutdown_connections 2024-09-19T07:44:06.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.383+0000 7f44737fe640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44740679c0 0x7f4474069e80 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:06.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.383+0000 7f44737fe640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4488172200 0x7f448803feb0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:06.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.383+0000 7f44737fe640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44880e0bd0 0x7f4488171cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:06.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.383+0000 7f44737fe640 1 --2- 172.21.15.83:0/957935742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f447c004bb0 0x7f4488045c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:06.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.383+0000 7f44737fe640 1 -- 172.21.15.83:0/957935742 >> 172.21.15.83:0/957935742 conn(0x7f4488048e30 msgr2=0x7f4488007570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:06.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.383+0000 7f44737fe640 1 -- 172.21.15.83:0/957935742 shutdown_connections 2024-09-19T07:44:06.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:06.383+0000 7f44737fe640 1 -- 172.21.15.83:0/957935742 wait complete. 2024-09-19T07:44:06.442 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-19T07:44:06.443 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-19T07:44:06.443 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:44:07.212 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:44:07.214 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:44:12.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-09-19T07:44:12.545 INFO:tasks.ceph.mds.d:Stopped 2024-09-19T07:44:12.545 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:44:12.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.758+0000 7f07a34bb640 1 Processor -- start 2024-09-19T07:44:12.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.758+0000 7f07a34bb640 1 -- start start 2024-09-19T07:44:12.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.758+0000 7f07a34bb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f079c0c7510 0x7f079c06de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:12.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.758+0000 7f07a34bb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f079c154360 0x7f079c0c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:12.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.758+0000 7f07a34bb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f079c14ddd0 con 0x7f079c14d9f0 2024-09-19T07:44:12.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.758+0000 7f07a34bb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f079c0c8830 con 0x7f079c0c7510 2024-09-19T07:44:12.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.758+0000 7f07a34bb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f079c06fdf0 con 0x7f079c154360 2024-09-19T07:44:12.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.758+0000 7f07a1257640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f079c14d9f0 0x7f079c0c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:36710/0 (socket says 172.21.15.83:36710) 2024-09-19T07:44:12.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.758+0000 7f07a1257640 1 -- 172.21.15.83:0/2938867476 learned_addr learned my addr 172.21.15.83:0/2938867476 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:44:12.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.758+0000 7f07a1a58640 1 --2- 172.21.15.83:0/2938867476 >> v2:172.21.15.184:3300/0 conn(0x7f079c154360 0x7f079c0c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:12.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.758+0000 7f07a0a56640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f079c0c7510 0x7f079c06de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:12.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.758+0000 7f07a1a58640 1 -- 172.21.15.83:0/2938867476 >> v2:172.21.15.83:3300/0 conn(0x7f079c0c7510 msgr2=0x7f079c06de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:44:12.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.762+0000 7f07a1a58640 1 --2- 172.21.15.83:0/2938867476 >> v2:172.21.15.83:3300/0 conn(0x7f079c0c7510 0x7f079c06de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:12.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.762+0000 7f0793fff640 1 -- 172.21.15.83:0/2938867476 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3785634757 0 0) 0x7f079c14ddd0 con 0x7f079c14d9f0 2024-09-19T07:44:12.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.762+0000 7f07a1a58640 1 -- 172.21.15.83:0/2938867476 >> v1:172.21.15.184:6789/0 conn(0x7f079c14d9f0 legacy=0x7f079c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:12.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.762+0000 7f07a1a58640 1 -- 172.21.15.83:0/2938867476 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079c0c7f00 con 0x7f079c154360 2024-09-19T07:44:12.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.762+0000 7f07a0a56640 1 --2- 172.21.15.83:0/2938867476 >> v2:172.21.15.83:3300/0 conn(0x7f079c0c7510 0x7f079c06de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:44:12.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.766+0000 7f07a1a58640 1 --2- 172.21.15.83:0/2938867476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c154360 0x7f079c0c4010 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f079c0743c0 tx=0x7f079401c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f85c3330f8dcb00e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:12.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.766+0000 7f0793fff640 1 -- 172.21.15.83:0/2938867476 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07940a6700 con 0x7f079c154360 2024-09-19T07:44:12.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.766+0000 7f0793fff640 1 -- 172.21.15.83:0/2938867476 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07940a68a0 con 0x7f079c154360 2024-09-19T07:44:12.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.766+0000 7f0793fff640 1 -- 172.21.15.83:0/2938867476 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07940a6cb0 con 0x7f079c154360 2024-09-19T07:44:12.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.770+0000 7f07a34bb640 1 -- 172.21.15.83:0/2938867476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c154360 msgr2=0x7f079c0c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:12.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.770+0000 7f07a34bb640 1 --2- 172.21.15.83:0/2938867476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c154360 0x7f079c0c4010 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f079c0743c0 tx=0x7f079401c830 comp rx=0 tx=0).stop 2024-09-19T07:44:12.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.770+0000 7f07a34bb640 1 -- 172.21.15.83:0/2938867476 shutdown_connections 2024-09-19T07:44:12.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.770+0000 7f07a34bb640 1 --2- 172.21.15.83:0/2938867476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c154360 0x7f079c0c4010 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:12.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.770+0000 7f07a34bb640 1 --2- 172.21.15.83:0/2938867476 >> v2:172.21.15.83:3300/0 conn(0x7f079c0c7510 0x7f079c06de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:12.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.770+0000 7f07a34bb640 1 -- 172.21.15.83:0/2938867476 >> 172.21.15.83:0/2938867476 conn(0x7f079c0c6110 msgr2=0x7f079c0c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:12.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.782+0000 7f07a34bb640 1 -- 172.21.15.83:0/2938867476 shutdown_connections 2024-09-19T07:44:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.782+0000 7f07a34bb640 1 -- 172.21.15.83:0/2938867476 wait complete. 2024-09-19T07:44:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a34bb640 1 Processor -- start 2024-09-19T07:44:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a34bb640 1 -- start start 2024-09-19T07:44:12.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a34bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f079c0c7510 0x7f079c142f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:12.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a1257640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f079c0c7510 0x7f079c142f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:12.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a1257640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f079c0c7510 0x7f079c142f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51088/0 (socket says 172.21.15.83:51088) 2024-09-19T07:44:12.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a1257640 1 -- 172.21.15.83:0/2247645526 learned_addr learned my addr 172.21.15.83:0/2247645526 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:12.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a34bb640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c14d9f0 0x7f079c1ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:12.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a34bb640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f079c154360 0x7f079c1ec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a1a58640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f079c154360 0x7f079c1ec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f079c160c00 con 0x7f079c0c7510 2024-09-19T07:44:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f079c160a80 con 0x7f079c14d9f0 2024-09-19T07:44:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f079c160d80 con 0x7f079c154360 2024-09-19T07:44:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a1a58640 1 -- 172.21.15.83:0/2247645526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c14d9f0 msgr2=0x7f079c1ebf80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:44:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a1a58640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c14d9f0 0x7f079c1ebf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a1a58640 1 -- 172.21.15.83:0/2247645526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f079c0c7510 msgr2=0x7f079c142f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a1a58640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f079c0c7510 0x7f079c142f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.786+0000 7f07a1a58640 1 -- 172.21.15.83:0/2247645526 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079c143a00 con 0x7f079c154360 2024-09-19T07:44:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.790+0000 7f07a1a58640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f079c154360 0x7f079c1ec4c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f07940ac000 tx=0x7f07940195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.790+0000 7f0791ffb640 1 -- 172.21.15.83:0/2247645526 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07940bab40 con 0x7f079c154360 2024-09-19T07:44:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.790+0000 7f0791ffb640 1 -- 172.21.15.83:0/2247645526 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07940bace0 con 0x7f079c154360 2024-09-19T07:44:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.790+0000 7f0791ffb640 1 -- 172.21.15.83:0/2247645526 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07940ae870 con 0x7f079c154360 2024-09-19T07:44:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.790+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f079c154d70 con 0x7f079c154360 2024-09-19T07:44:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.790+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f079c1434a0 con 0x7f079c154360 2024-09-19T07:44:12.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.790+0000 7f0791ffb640 1 -- 172.21.15.83:0/2247645526 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f07940aea10 con 0x7f079c154360 2024-09-19T07:44:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.794+0000 7f0791ffb640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07800678f0 0x7f0780069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.794+0000 7f0791ffb640 1 -- 172.21.15.83:0/2247645526 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 9475+0+0 (secure 0 0 0) 0x7f07941261c0 con 0x7f079c154360 2024-09-19T07:44:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.794+0000 7f07a1257640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07800678f0 0x7f0780069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.794+0000 7f07a1257640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07800678f0 0x7f0780069db0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0798000f70 tx=0x7f0798007ee0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:12.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.798+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f079c06faf0 con 0x7f079c154360 2024-09-19T07:44:12.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:12.814+0000 7f0791ffb640 1 -- 172.21.15.83:0/2247645526 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f07940cc030 con 0x7f079c154360 2024-09-19T07:44:13.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.214+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f079c1f0360 con 0x7f079c154360 2024-09-19T07:44:13.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.214+0000 7f0791ffb640 1 -- 172.21.15.83:0/2247645526 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) ==== 95+0+6545 (secure 0 0 0) 0x7f079c143a00 con 0x7f079c154360 2024-09-19T07:44:13.212 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:44:13.212 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":170,"btime":"2024-09-19T07:42:53:467120+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7803,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:44:13.212 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 170 2024-09-19T07:44:13.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.218+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07800678f0 msgr2=0x7f0780069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:13.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.218+0000 7f07a34bb640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07800678f0 0x7f0780069db0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0798000f70 tx=0x7f0798007ee0 comp rx=0 tx=0).stop 2024-09-19T07:44:13.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.218+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f079c154360 msgr2=0x7f079c1ec4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:13.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.218+0000 7f07a34bb640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f079c154360 0x7f079c1ec4c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f07940ac000 tx=0x7f07940195d0 comp rx=0 tx=0).stop 2024-09-19T07:44:13.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.218+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 shutdown_connections 2024-09-19T07:44:13.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.218+0000 7f07a34bb640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07800678f0 0x7f0780069db0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:13.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.218+0000 7f07a34bb640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f079c154360 0x7f079c1ec4c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:13.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.218+0000 7f07a34bb640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c14d9f0 0x7f079c1ebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:13.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.218+0000 7f07a34bb640 1 --2- 172.21.15.83:0/2247645526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f079c0c7510 0x7f079c142f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:13.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.218+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 >> 172.21.15.83:0/2247645526 conn(0x7f079c0c6110 msgr2=0x7f079c1473f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:13.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.218+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 shutdown_connections 2024-09-19T07:44:13.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:13.218+0000 7f07a34bb640 1 -- 172.21.15.83:0/2247645526 wait complete. 2024-09-19T07:44:13.234 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:44:15.235 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:44:15.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f91827f4640 1 Processor -- start 2024-09-19T07:44:15.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f91827f4640 1 -- start start 2024-09-19T07:44:15.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f91827f4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f917c0c3130 0x7f917c0ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f91827f4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f917c06de40 0x7f917c1563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f91827f4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917c0c3c20 con 0x7f917c06da00 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f91827f4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f917c0c4990 con 0x7f917c06de40 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f91827f4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f917c0c4c70 con 0x7f917c0c3130 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f9180d91640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f917c06de40 0x7f917c1563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f9180d91640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f917c06de40 0x7f917c1563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51098/0 (socket says 172.21.15.83:51098) 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f9180d91640 1 -- 172.21.15.83:0/2403679916 learned_addr learned my addr 172.21.15.83:0/2403679916 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f917bfff640 1 --2- 172.21.15.83:0/2403679916 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f917c0c3130 0x7f917c0ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f9180d91640 1 -- 172.21.15.83:0/2403679916 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f917c0c3130 msgr2=0x7f917c0ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f9180d91640 1 --2- 172.21.15.83:0/2403679916 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f917c0c3130 0x7f917c0ac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f9180d91640 1 -- 172.21.15.83:0/2403679916 >> v1:172.21.15.184:6789/0 conn(0x7f917c06da00 legacy=0x7f917c0c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f917b7fe640 1 -- 172.21.15.83:0/2403679916 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4276003575 0 0) 0x7f917c0c3c20 con 0x7f917c06da00 2024-09-19T07:44:15.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f9180d91640 1 -- 172.21.15.83:0/2403679916 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c0ab7b0 con 0x7f917c06de40 2024-09-19T07:44:15.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f9180d91640 1 --2- 172.21.15.83:0/2403679916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f917c06de40 0x7f917c1563a0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f917400d150 tx=0x7f917401e8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d1eef063f68c58 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:15.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f917b7fe640 1 -- 172.21.15.83:0/2403679916 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917400b2d0 con 0x7f917c06de40 2024-09-19T07:44:15.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f917b7fe640 1 -- 172.21.15.83:0/2403679916 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f917400b470 con 0x7f917c06de40 2024-09-19T07:44:15.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.378+0000 7f917b7fe640 1 -- 172.21.15.83:0/2403679916 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91740a78c0 con 0x7f917c06de40 2024-09-19T07:44:15.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.382+0000 7f91827f4640 1 -- 172.21.15.83:0/2403679916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f917c06de40 msgr2=0x7f917c1563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:15.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.382+0000 7f91827f4640 1 --2- 172.21.15.83:0/2403679916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f917c06de40 0x7f917c1563a0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f917400d150 tx=0x7f917401e8e0 comp rx=0 tx=0).stop 2024-09-19T07:44:15.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.382+0000 7f91827f4640 1 -- 172.21.15.83:0/2403679916 shutdown_connections 2024-09-19T07:44:15.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.382+0000 7f91827f4640 1 --2- 172.21.15.83:0/2403679916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f917c06de40 0x7f917c1563a0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:15.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.382+0000 7f91827f4640 1 --2- 172.21.15.83:0/2403679916 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f917c0c3130 0x7f917c0ac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:15.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.382+0000 7f91827f4640 1 -- 172.21.15.83:0/2403679916 >> 172.21.15.83:0/2403679916 conn(0x7f917c0c5060 msgr2=0x7f917c0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:15.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.382+0000 7f91827f4640 1 -- 172.21.15.83:0/2403679916 shutdown_connections 2024-09-19T07:44:15.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.382+0000 7f91827f4640 1 -- 172.21.15.83:0/2403679916 wait complete. 2024-09-19T07:44:15.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.382+0000 7f91827f4640 1 Processor -- start 2024-09-19T07:44:15.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91827f4640 1 -- start start 2024-09-19T07:44:15.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91827f4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f917c06da00 0x7f917c157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:15.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91827f4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f917c06de40 0x7f917c158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:15.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91827f4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f917c0c3130 0x7f917c14f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:15.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91827f4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f917c0ad6a0 con 0x7f917c0c3130 2024-09-19T07:44:15.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91827f4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f917c0ad520 con 0x7f917c06da00 2024-09-19T07:44:15.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91827f4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f917c074f90 con 0x7f917c06de40 2024-09-19T07:44:15.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91737fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f917c06de40 0x7f917c158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:15.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91737fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f917c06de40 0x7f917c158180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:40708/0 (socket says 172.21.15.83:40708) 2024-09-19T07:44:15.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91737fe640 1 -- 172.21.15.83:0/2396794566 learned_addr learned my addr 172.21.15.83:0/2396794566 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:15.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91737fe640 1 -- 172.21.15.83:0/2396794566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f917c06da00 msgr2=0x7f917c157c40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:44:15.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91737fe640 1 --2- 172.21.15.83:0/2396794566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f917c06da00 0x7f917c157c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:15.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91737fe640 1 -- 172.21.15.83:0/2396794566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f917c0c3130 msgr2=0x7f917c14f310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:44:15.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91737fe640 1 --2- 172.21.15.83:0/2396794566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f917c0c3130 0x7f917c14f310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:15.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91737fe640 1 -- 172.21.15.83:0/2396794566 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c0c3c20 con 0x7f917c06de40 2024-09-19T07:44:15.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91737fe640 1 --2- 172.21.15.83:0/2396794566 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f917c06de40 0x7f917c158180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f916400c520 tx=0x7f916400c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:15.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91797fa640 1 -- 172.21.15.83:0/2396794566 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164017070 con 0x7f917c06de40 2024-09-19T07:44:15.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91797fa640 1 -- 172.21.15.83:0/2396794566 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91640052d0 con 0x7f917c06de40 2024-09-19T07:44:15.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.386+0000 7f91827f4640 1 -- 172.21.15.83:0/2396794566 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f917c155c70 con 0x7f917c06de40 2024-09-19T07:44:15.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.390+0000 7f91797fa640 1 -- 172.21.15.83:0/2396794566 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91640054c0 con 0x7f917c06de40 2024-09-19T07:44:15.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.390+0000 7f91827f4640 1 -- 172.21.15.83:0/2396794566 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f917c155f10 con 0x7f917c06de40 2024-09-19T07:44:15.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.390+0000 7f91797fa640 1 -- 172.21.15.83:0/2396794566 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f916401a070 con 0x7f917c06de40 2024-09-19T07:44:15.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.390+0000 7f91797fa640 1 --2- 172.21.15.83:0/2396794566 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f915c067910 0x7f915c069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:15.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.390+0000 7f917bfff640 1 --2- 172.21.15.83:0/2396794566 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f915c067910 0x7f915c069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:15.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.390+0000 7f91797fa640 1 -- 172.21.15.83:0/2396794566 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 9475+0+0 (secure 0 0 0) 0x7f9164053250 con 0x7f917c06de40 2024-09-19T07:44:15.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.390+0000 7f91827f4640 1 -- 172.21.15.83:0/2396794566 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f917c075e60 con 0x7f917c06de40 2024-09-19T07:44:15.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.390+0000 7f917bfff640 1 --2- 172.21.15.83:0/2396794566 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f915c067910 0x7f915c069dd0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f916c009950 tx=0x7f916c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:15.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.398+0000 7f91797fa640 1 -- 172.21.15.83:0/2396794566 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f91640558b0 con 0x7f917c06de40 2024-09-19T07:44:15.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.686+0000 7f91827f4640 1 -- 172.21.15.83:0/2396794566 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f917c0c54d0 con 0x7f917c06de40 2024-09-19T07:44:15.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.686+0000 7f91797fa640 1 -- 172.21.15.83:0/2396794566 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) ==== 95+0+6545 (secure 0 0 0) 0x7f91640551b0 con 0x7f917c06de40 2024-09-19T07:44:15.686 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:44:15.686 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":170,"btime":"2024-09-19T07:42:53:467120+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7803,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:44:15.686 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 170 2024-09-19T07:44:15.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.690+0000 7f91827f4640 1 -- 172.21.15.83:0/2396794566 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f915c067910 msgr2=0x7f915c069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:15.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.690+0000 7f91827f4640 1 --2- 172.21.15.83:0/2396794566 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f915c067910 0x7f915c069dd0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f916c009950 tx=0x7f916c009190 comp rx=0 tx=0).stop 2024-09-19T07:44:15.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.690+0000 7f91827f4640 1 -- 172.21.15.83:0/2396794566 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f917c06de40 msgr2=0x7f917c158180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:15.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.690+0000 7f91827f4640 1 --2- 172.21.15.83:0/2396794566 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f917c06de40 0x7f917c158180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f916400c520 tx=0x7f916400c9e0 comp rx=0 tx=0).stop 2024-09-19T07:44:15.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.690+0000 7f91827f4640 1 -- 172.21.15.83:0/2396794566 shutdown_connections 2024-09-19T07:44:15.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.690+0000 7f91827f4640 1 --2- 172.21.15.83:0/2396794566 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f915c067910 0x7f915c069dd0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:15.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.690+0000 7f91827f4640 1 --2- 172.21.15.83:0/2396794566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f917c0c3130 0x7f917c14f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:15.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.690+0000 7f91827f4640 1 --2- 172.21.15.83:0/2396794566 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f917c06de40 0x7f917c158180 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:15.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.690+0000 7f91827f4640 1 --2- 172.21.15.83:0/2396794566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f917c06da00 0x7f917c157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:15.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.690+0000 7f91827f4640 1 -- 172.21.15.83:0/2396794566 >> 172.21.15.83:0/2396794566 conn(0x7f917c0c5060 msgr2=0x7f917c0c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:15.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.690+0000 7f91827f4640 1 -- 172.21.15.83:0/2396794566 shutdown_connections 2024-09-19T07:44:15.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:15.690+0000 7f91827f4640 1 -- 172.21.15.83:0/2396794566 wait complete. 2024-09-19T07:44:15.709 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:44:17.710 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:44:17.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.862+0000 7fd5664a7640 1 Processor -- start 2024-09-19T07:44:17.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.862+0000 7fd5664a7640 1 -- start start 2024-09-19T07:44:17.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd5664a7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd56006c800 0x7fd56006e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:17.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd5664a7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd560072f10 0x7fd5600c2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:17.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd5664a7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd56006ddd0 con 0x7fd560073cc0 2024-09-19T07:44:17.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd5664a7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd5600740a0 con 0x7fd560072f10 2024-09-19T07:44:17.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd5664a7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd560070ea0 con 0x7fd56006c800 2024-09-19T07:44:17.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd564a44640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd560072f10 0x7fd5600c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:17.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd564a44640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd560072f10 0x7fd5600c2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51114/0 (socket says 172.21.15.83:51114) 2024-09-19T07:44:17.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd564a44640 1 -- 172.21.15.83:0/1879065773 learned_addr learned my addr 172.21.15.83:0/1879065773 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:17.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd55ffff640 1 --1- 172.21.15.83:0/1879065773 >> v1:172.21.15.83:6789/0 conn(0x7fd560073cc0 0x7fd5601544a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38712/0 (socket says 172.21.15.83:38712) 2024-09-19T07:44:17.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd564a44640 1 -- 172.21.15.83:0/1879065773 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd56006c800 msgr2=0x7fd56006e840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:44:17.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd564a44640 1 --2- 172.21.15.83:0/1879065773 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd56006c800 0x7fd56006e840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:17.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd564a44640 1 -- 172.21.15.83:0/1879065773 >> v1:172.21.15.83:6789/0 conn(0x7fd560073cc0 legacy=0x7fd5601544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:17.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd55effd640 1 -- 172.21.15.83:0/1879065773 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1315759557 0 0) 0x7fd56006ddd0 con 0x7fd560073cc0 2024-09-19T07:44:17.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.866+0000 7fd564a44640 1 -- 172.21.15.83:0/1879065773 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5600710f0 con 0x7fd560072f10 2024-09-19T07:44:17.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd564a44640 1 --2- 172.21.15.83:0/1879065773 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560072f10 0x7fd5600c2210 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fd558009e50 tx=0x7fd55801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659c11fdbc539c3f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:17.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd55effd640 1 -- 172.21.15.83:0/1879065773 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5580a58f0 con 0x7fd560072f10 2024-09-19T07:44:17.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd55effd640 1 -- 172.21.15.83:0/1879065773 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5580a5a90 con 0x7fd560072f10 2024-09-19T07:44:17.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd55effd640 1 -- 172.21.15.83:0/1879065773 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5580a5e60 con 0x7fd560072f10 2024-09-19T07:44:17.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 -- 172.21.15.83:0/1879065773 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560072f10 msgr2=0x7fd5600c2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:17.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 --2- 172.21.15.83:0/1879065773 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560072f10 0x7fd5600c2210 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fd558009e50 tx=0x7fd55801ca20 comp rx=0 tx=0).stop 2024-09-19T07:44:17.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 -- 172.21.15.83:0/1879065773 shutdown_connections 2024-09-19T07:44:17.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 --2- 172.21.15.83:0/1879065773 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560072f10 0x7fd5600c2210 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:17.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 --2- 172.21.15.83:0/1879065773 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd56006c800 0x7fd56006e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:17.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 -- 172.21.15.83:0/1879065773 >> 172.21.15.83:0/1879065773 conn(0x7fd560075390 msgr2=0x7fd56006bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:17.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 -- 172.21.15.83:0/1879065773 shutdown_connections 2024-09-19T07:44:17.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 -- 172.21.15.83:0/1879065773 wait complete. 2024-09-19T07:44:17.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 Processor -- start 2024-09-19T07:44:17.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 -- start start 2024-09-19T07:44:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd56006c800 0x7fd56014c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd560072f10 0x7fd5600abf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.870+0000 7fd5664a7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560073cc0 0x7fd5600c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd5664a7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd560160d70 con 0x7fd560073cc0 2024-09-19T07:44:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd5664a7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd560160bf0 con 0x7fd560072f10 2024-09-19T07:44:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd5664a7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd560160ef0 con 0x7fd56006c800 2024-09-19T07:44:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd564a44640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560073cc0 0x7fd5600c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd564a44640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560073cc0 0x7fd5600c0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51128/0 (socket says 172.21.15.83:51128) 2024-09-19T07:44:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd564a44640 1 -- 172.21.15.83:0/1120268405 learned_addr learned my addr 172.21.15.83:0/1120268405 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd55f7fe640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd560072f10 0x7fd5600abf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd55ffff640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd56006c800 0x7fd56014c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd564a44640 1 -- 172.21.15.83:0/1120268405 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd56006c800 msgr2=0x7fd56014c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd564a44640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd56006c800 0x7fd56014c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd564a44640 1 -- 172.21.15.83:0/1120268405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd560072f10 msgr2=0x7fd5600abf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd564a44640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd560072f10 0x7fd5600abf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd564a44640 1 -- 172.21.15.83:0/1120268405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5600c4930 con 0x7fd560073cc0 2024-09-19T07:44:17.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd564a44640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560073cc0 0x7fd5600c0c80 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fd5580af040 tx=0x7fd5580197c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:17.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd55cff9640 1 -- 172.21.15.83:0/1120268405 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558009510 con 0x7fd560073cc0 2024-09-19T07:44:17.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd55cff9640 1 -- 172.21.15.83:0/1120268405 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd558007c60 con 0x7fd560073cc0 2024-09-19T07:44:17.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd5664a7640 1 -- 172.21.15.83:0/1120268405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5600ac630 con 0x7fd560073cc0 2024-09-19T07:44:17.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd55cff9640 1 -- 172.21.15.83:0/1120268405 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5580b86b0 con 0x7fd560073cc0 2024-09-19T07:44:17.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd5664a7640 1 -- 172.21.15.83:0/1120268405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5600aacd0 con 0x7fd560073cc0 2024-09-19T07:44:17.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd55cff9640 1 -- 172.21.15.83:0/1120268405 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd5580c0030 con 0x7fd560073cc0 2024-09-19T07:44:17.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.874+0000 7fd5664a7640 1 -- 172.21.15.83:0/1120268405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd56006e840 con 0x7fd560073cc0 2024-09-19T07:44:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.878+0000 7fd55cff9640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd54c0678f0 0x7fd54c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.878+0000 7fd55cff9640 1 -- 172.21.15.83:0/1120268405 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9475+0+0 (secure 0 0 0) 0x7fd5581212b0 con 0x7fd560073cc0 2024-09-19T07:44:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.878+0000 7fd55ffff640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd54c0678f0 0x7fd54c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:17.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.878+0000 7fd55ffff640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd54c0678f0 0x7fd54c069db0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd550007fd0 tx=0x7fd550007d60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:17.882+0000 7fd55cff9640 1 -- 172.21.15.83:0/1120268405 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd558016640 con 0x7fd560073cc0 2024-09-19T07:44:18.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.170+0000 7fd5664a7640 1 -- 172.21.15.83:0/1120268405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd56015e1b0 con 0x7fd560073cc0 2024-09-19T07:44:18.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.170+0000 7fd55cff9640 1 -- 172.21.15.83:0/1120268405 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) ==== 95+0+6545 (secure 0 0 0) 0x7fd5580f0c00 con 0x7fd560073cc0 2024-09-19T07:44:18.170 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:44:18.170 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":170,"btime":"2024-09-19T07:42:53:467120+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7803,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:42:11.447678+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6565":{"gid":6565,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":9,"addr":"172.21.15.184:6833/3311721080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3311721080},{"type":"v1","addr":"172.21.15.184:6833","nonce":3311721080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6565,"qdb_cluster":[6565]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:44:18.170 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 170 2024-09-19T07:44:18.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.174+0000 7fd5664a7640 1 -- 172.21.15.83:0/1120268405 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd54c0678f0 msgr2=0x7fd54c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:18.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.174+0000 7fd5664a7640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd54c0678f0 0x7fd54c069db0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd550007fd0 tx=0x7fd550007d60 comp rx=0 tx=0).stop 2024-09-19T07:44:18.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.174+0000 7fd5664a7640 1 -- 172.21.15.83:0/1120268405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560073cc0 msgr2=0x7fd5600c0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:18.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.174+0000 7fd5664a7640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560073cc0 0x7fd5600c0c80 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fd5580af040 tx=0x7fd5580197c0 comp rx=0 tx=0).stop 2024-09-19T07:44:18.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.174+0000 7fd5664a7640 1 -- 172.21.15.83:0/1120268405 shutdown_connections 2024-09-19T07:44:18.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.174+0000 7fd5664a7640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd54c0678f0 0x7fd54c069db0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:18.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.174+0000 7fd5664a7640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560073cc0 0x7fd5600c0c80 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:18.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.174+0000 7fd5664a7640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd560072f10 0x7fd5600abf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:18.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.174+0000 7fd5664a7640 1 --2- 172.21.15.83:0/1120268405 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd56006c800 0x7fd56014c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:18.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.174+0000 7fd5664a7640 1 -- 172.21.15.83:0/1120268405 >> 172.21.15.83:0/1120268405 conn(0x7fd560075390 msgr2=0x7fd560152850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:18.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.174+0000 7fd5664a7640 1 -- 172.21.15.83:0/1120268405 shutdown_connections 2024-09-19T07:44:18.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:18.174+0000 7fd5664a7640 1 -- 172.21.15.83:0/1120268405 wait complete. 2024-09-19T07:44:18.196 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:44:18.999 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-09-19T07:44:20.197 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:44:20.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.342+0000 7fc5c6ff4640 1 Processor -- start 2024-09-19T07:44:20.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.342+0000 7fc5c6ff4640 1 -- start start 2024-09-19T07:44:20.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.342+0000 7fc5c6ff4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc5c00ac290 0x7fc5c00c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:20.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.342+0000 7fc5c6ff4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c006e110 con 0x7fc5c00c3a50 2024-09-19T07:44:20.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.342+0000 7fc5c6ff4640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c00c4990 con 0x7fc5c00c3130 2024-09-19T07:44:20.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.342+0000 7fc5c6ff4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc5c00ac700 con 0x7fc5c00ac290 2024-09-19T07:44:20.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.342+0000 7fc5c4d90640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fc5c00c3130 0x7fc5c006da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:54044/0 (socket says 172.21.15.83:54044) 2024-09-19T07:44:20.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.342+0000 7fc5c4d90640 1 -- 172.21.15.83:0/1648632793 learned_addr learned my addr 172.21.15.83:0/1648632793 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:44:20.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.342+0000 7fc5bffff640 1 --2- 172.21.15.83:0/1648632793 >> v2:172.21.15.83:3300/0 conn(0x7fc5c00ac290 0x7fc5c00c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:20.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.346+0000 7fc5bffff640 1 -- 172.21.15.83:0/1648632793 >> v1:172.21.15.184:6789/0 conn(0x7fc5c00c3130 legacy=0x7fc5c006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:20.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.346+0000 7fc5bffff640 1 -- 172.21.15.83:0/1648632793 >> v1:172.21.15.83:6789/0 conn(0x7fc5c00c3a50 legacy=0x7fc5c01563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:20.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.346+0000 7fc5bffff640 1 -- 172.21.15.83:0/1648632793 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c00c4c70 con 0x7fc5c00ac290 2024-09-19T07:44:20.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.346+0000 7fc5bffff640 1 --2- 172.21.15.83:0/1648632793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5c00ac290 0x7fc5c00c3510 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac009740 tx=0x7fc5ac01c240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f856c7d0322565 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:20.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.346+0000 7fc5bf7fe640 1 -- 172.21.15.83:0/1648632793 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0b0070 con 0x7fc5c00ac290 2024-09-19T07:44:20.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.346+0000 7fc5bf7fe640 1 -- 172.21.15.83:0/1648632793 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ac0ab600 con 0x7fc5c00ac290 2024-09-19T07:44:20.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.346+0000 7fc5bf7fe640 1 -- 172.21.15.83:0/1648632793 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0ab9d0 con 0x7fc5c00ac290 2024-09-19T07:44:20.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/1648632793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5c00ac290 msgr2=0x7fc5c00c3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:20.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 --2- 172.21.15.83:0/1648632793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5c00ac290 0x7fc5c00c3510 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac009740 tx=0x7fc5ac01c240 comp rx=0 tx=0).stop 2024-09-19T07:44:20.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/1648632793 shutdown_connections 2024-09-19T07:44:20.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 --2- 172.21.15.83:0/1648632793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5c00ac290 0x7fc5c00c3510 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/1648632793 >> 172.21.15.83:0/1648632793 conn(0x7fc5c00c5060 msgr2=0x7fc5c00c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:20.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/1648632793 shutdown_connections 2024-09-19T07:44:20.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/1648632793 wait complete. 2024-09-19T07:44:20.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 Processor -- start 2024-09-19T07:44:20.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 -- start start 2024-09-19T07:44:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5c00ac290 0x7fc5c0160a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5c00c3130 0x7fc5c0162fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5c00c3a50 0x7fc5c0163510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c00ad520 con 0x7fc5c00c3a50 2024-09-19T07:44:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c00c82a0 con 0x7fc5c00ac290 2024-09-19T07:44:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.350+0000 7fc5c6ff4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c00ad6a0 con 0x7fc5c00c3130 2024-09-19T07:44:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5c5591640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5c00c3a50 0x7fc5c0163510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5c5591640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5c00c3a50 0x7fc5c0163510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60258/0 (socket says 172.21.15.83:60258) 2024-09-19T07:44:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5c5591640 1 -- 172.21.15.83:0/338417682 learned_addr learned my addr 172.21.15.83:0/338417682 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5bffff640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5c00c3130 0x7fc5c0162fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5c5591640 1 -- 172.21.15.83:0/338417682 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5c00c3130 msgr2=0x7fc5c0162fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5c5591640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5c00c3130 0x7fc5c0162fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5c5591640 1 -- 172.21.15.83:0/338417682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5c00ac290 msgr2=0x7fc5c0160a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5c5591640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5c00ac290 0x7fc5c0160a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5c5591640 1 -- 172.21.15.83:0/338417682 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c0162090 con 0x7fc5c00c3a50 2024-09-19T07:44:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5c5591640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5c00c3a50 0x7fc5c0163510 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c730 tx=0x7fc5b800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5bd7fa640 1 -- 172.21.15.83:0/338417682 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8016020 con 0x7fc5c00c3a50 2024-09-19T07:44:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5bd7fa640 1 -- 172.21.15.83:0/338417682 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b8006d80 con 0x7fc5c00c3a50 2024-09-19T07:44:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5bd7fa640 1 -- 172.21.15.83:0/338417682 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8003bc0 con 0x7fc5c00c3a50 2024-09-19T07:44:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/338417682 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c00a7380 con 0x7fc5c00c3a50 2024-09-19T07:44:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/338417682 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c0161db0 con 0x7fc5c00c3a50 2024-09-19T07:44:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.354+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/338417682 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c0069030 con 0x7fc5c00c3a50 2024-09-19T07:44:20.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.358+0000 7fc5bd7fa640 1 -- 172.21.15.83:0/338417682 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc5b80050d0 con 0x7fc5c00c3a50 2024-09-19T07:44:20.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.358+0000 7fc5bd7fa640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5a40678f0 0x7fc5a4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:20.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.358+0000 7fc5bd7fa640 1 -- 172.21.15.83:0/338417682 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7fc5b80868a0 con 0x7fc5c00c3a50 2024-09-19T07:44:20.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.358+0000 7fc5c4d90640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5a40678f0 0x7fc5a4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:20.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.358+0000 7fc5c4d90640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5a40678f0 0x7fc5a4069db0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc5b40017d0 tx=0x7fc5b4007970 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:20.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.366+0000 7fc5bd7fa640 1 -- 172.21.15.83:0/338417682 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc5b8020020 con 0x7fc5c00c3a50 2024-09-19T07:44:20.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.658+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/338417682 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5c006c2e0 con 0x7fc5c00c3a50 2024-09-19T07:44:20.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.658+0000 7fc5bd7fa640 1 -- 172.21.15.83:0/338417682 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) ==== 95+0+5710 (secure 0 0 0) 0x7fc5b8056510 con 0x7fc5c00c3a50 2024-09-19T07:44:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:44:20.657 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":173,"btime":"2024-09-19T07:44:20:635635+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:19.641476+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:44:20.658 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 173 2024-09-19T07:44:20.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.662+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/338417682 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5a40678f0 msgr2=0x7fc5a4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:20.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.662+0000 7fc5c6ff4640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5a40678f0 0x7fc5a4069db0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc5b40017d0 tx=0x7fc5b4007970 comp rx=0 tx=0).stop 2024-09-19T07:44:20.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.662+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/338417682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5c00c3a50 msgr2=0x7fc5c0163510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:20.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.662+0000 7fc5c6ff4640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5c00c3a50 0x7fc5c0163510 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c730 tx=0x7fc5b800cbf0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.662+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/338417682 shutdown_connections 2024-09-19T07:44:20.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.662+0000 7fc5c6ff4640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc5a40678f0 0x7fc5a4069db0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.662+0000 7fc5c6ff4640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5c00c3a50 0x7fc5c0163510 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.662+0000 7fc5c6ff4640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc5c00c3130 0x7fc5c0162fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.662+0000 7fc5c6ff4640 1 --2- 172.21.15.83:0/338417682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5c00ac290 0x7fc5c0160a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.662+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/338417682 >> 172.21.15.83:0/338417682 conn(0x7fc5c00c5060 msgr2=0x7fc5c006da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:20.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.662+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/338417682 shutdown_connections 2024-09-19T07:44:20.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.662+0000 7fc5c6ff4640 1 -- 172.21.15.83:0/338417682 wait complete. 2024-09-19T07:44:20.686 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-19T07:44:20.686 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:44:20.686 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:44:20.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed7a4c9640 1 Processor -- start 2024-09-19T07:44:20.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed7a4c9640 1 -- start start 2024-09-19T07:44:20.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed7a4c9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed74152f30 0x7fed74153370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:20.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed7a4c9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fed74155e10 0x7fed74156240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:20.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed7a4c9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed74159430 con 0x7fed740c4880 2024-09-19T07:44:20.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed7a4c9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fed7415a040 con 0x7fed74155e10 2024-09-19T07:44:20.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed7a4c9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fed740c2c10 con 0x7fed74152f30 2024-09-19T07:44:20.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed78a66640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fed74155e10 0x7fed74156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:20.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed73fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fed740c4880 0x7fed740c1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36516/0 (socket says 172.21.15.83:36516) 2024-09-19T07:44:20.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed78a66640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fed74155e10 0x7fed74156240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60260/0 (socket says 172.21.15.83:60260) 2024-09-19T07:44:20.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed73fff640 1 -- 172.21.15.83:0/2113888454 learned_addr learned my addr 172.21.15.83:0/2113888454 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:44:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed737fe640 1 --2- 172.21.15.83:0/2113888454 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed74152f30 0x7fed74153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed78a66640 1 -- 172.21.15.83:0/2113888454 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed74152f30 msgr2=0x7fed74153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:44:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed78a66640 1 --2- 172.21.15.83:0/2113888454 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed74152f30 0x7fed74153370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed78a66640 1 -- 172.21.15.83:0/2113888454 >> v1:172.21.15.83:6789/0 conn(0x7fed740c4880 legacy=0x7fed740c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed78a66640 1 -- 172.21.15.83:0/2113888454 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed74159710 con 0x7fed74155e10 2024-09-19T07:44:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.834+0000 7fed72ffd640 1 -- 172.21.15.83:0/2113888454 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2656618149 0 0) 0x7fed74159430 con 0x7fed740c4880 2024-09-19T07:44:20.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed78a66640 1 --2- 172.21.15.83:0/2113888454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed74155e10 0x7fed74156240 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0097c0 tx=0x7fed6c01c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7254f2cf04cfda5b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:20.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed72ffd640 1 -- 172.21.15.83:0/2113888454 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c0b0070 con 0x7fed74155e10 2024-09-19T07:44:20.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed72ffd640 1 -- 172.21.15.83:0/2113888454 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed6c0ab600 con 0x7fed74155e10 2024-09-19T07:44:20.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed72ffd640 1 -- 172.21.15.83:0/2113888454 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c0ab9d0 con 0x7fed74155e10 2024-09-19T07:44:20.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed7a4c9640 1 -- 172.21.15.83:0/2113888454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed74155e10 msgr2=0x7fed74156240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:20.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed7a4c9640 1 --2- 172.21.15.83:0/2113888454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed74155e10 0x7fed74156240 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0097c0 tx=0x7fed6c01c350 comp rx=0 tx=0).stop 2024-09-19T07:44:20.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed7a4c9640 1 -- 172.21.15.83:0/2113888454 shutdown_connections 2024-09-19T07:44:20.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed7a4c9640 1 --2- 172.21.15.83:0/2113888454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed74155e10 0x7fed74156240 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed7a4c9640 1 --2- 172.21.15.83:0/2113888454 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed74152f30 0x7fed74153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed7a4c9640 1 -- 172.21.15.83:0/2113888454 >> 172.21.15.83:0/2113888454 conn(0x7fed740aa910 msgr2=0x7fed740c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:20.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed7a4c9640 1 -- 172.21.15.83:0/2113888454 shutdown_connections 2024-09-19T07:44:20.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed7a4c9640 1 -- 172.21.15.83:0/2113888454 wait complete. 2024-09-19T07:44:20.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed7a4c9640 1 Processor -- start 2024-09-19T07:44:20.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed7a4c9640 1 -- start start 2024-09-19T07:44:20.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.838+0000 7fed7a4c9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed740c4880 0x7fed740ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:20.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed7a4c9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed74152f30 0x7fed74157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:20.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed7a4c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed74155e10 0x7fed741576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:20.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed7a4c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fed740c2680 con 0x7fed74155e10 2024-09-19T07:44:20.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed7a4c9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fed740c2500 con 0x7fed740c4880 2024-09-19T07:44:20.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed7a4c9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fed740c4c60 con 0x7fed74152f30 2024-09-19T07:44:20.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed78a66640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed74155e10 0x7fed741576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:20.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed78a66640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed74155e10 0x7fed741576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60262/0 (socket says 172.21.15.83:60262) 2024-09-19T07:44:20.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed78a66640 1 -- 172.21.15.83:0/1697706445 learned_addr learned my addr 172.21.15.83:0/1697706445 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:20.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed73fff640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed740c4880 0x7fed740ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:20.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed737fe640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed74152f30 0x7fed74157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:20.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed78a66640 1 -- 172.21.15.83:0/1697706445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed74152f30 msgr2=0x7fed74157190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:20.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed78a66640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed74152f30 0x7fed74157190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed78a66640 1 -- 172.21.15.83:0/1697706445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed740c4880 msgr2=0x7fed740ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:20.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed78a66640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed740c4880 0x7fed740ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:20.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed78a66640 1 -- 172.21.15.83:0/1697706445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed74158360 con 0x7fed74155e10 2024-09-19T07:44:20.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed78a66640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed74155e10 0x7fed741576d0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0ae040 tx=0x7fed6c01ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:20.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed70ff9640 1 -- 172.21.15.83:0/1697706445 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c0b0050 con 0x7fed74155e10 2024-09-19T07:44:20.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed70ff9640 1 -- 172.21.15.83:0/1697706445 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed6c0bc070 con 0x7fed74155e10 2024-09-19T07:44:20.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed70ff9640 1 -- 172.21.15.83:0/1697706445 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c006e20 con 0x7fed74155e10 2024-09-19T07:44:20.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed7a4c9640 1 -- 172.21.15.83:0/1697706445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed74158640 con 0x7fed74155e10 2024-09-19T07:44:20.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed7a4c9640 1 -- 172.21.15.83:0/1697706445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed74158c00 con 0x7fed74155e10 2024-09-19T07:44:20.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.842+0000 7fed7a4c9640 1 -- 172.21.15.83:0/1697706445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed74153370 con 0x7fed74155e10 2024-09-19T07:44:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.846+0000 7fed70ff9640 1 -- 172.21.15.83:0/1697706445 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fed6c0b5830 con 0x7fed74155e10 2024-09-19T07:44:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.846+0000 7fed70ff9640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed600678c0 0x7fed60069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.846+0000 7fed73fff640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed600678c0 0x7fed60069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.846+0000 7fed73fff640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed600678c0 0x7fed60069d80 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fed640012f0 tx=0x7fed64001770 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:20.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.850+0000 7fed70ff9640 1 -- 172.21.15.83:0/1697706445 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7fed6c11e900 con 0x7fed74155e10 2024-09-19T07:44:20.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:20.858+0000 7fed70ff9640 1 -- 172.21.15.83:0/1697706445 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fed6c016090 con 0x7fed74155e10 2024-09-19T07:44:21.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.138+0000 7fed7a4c9640 1 -- 172.21.15.83:0/1697706445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed740688c0 con 0x7fed74155e10 2024-09-19T07:44:21.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.138+0000 7fed70ff9640 1 -- 172.21.15.83:0/1697706445 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) ==== 95+0+5710 (secure 0 0 0) 0x7fed6c0f0e00 con 0x7fed74155e10 2024-09-19T07:44:21.138 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:44:21.139 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":173,"btime":"2024-09-19T07:44:20:635635+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:19.641476+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:44:21.139 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 173 2024-09-19T07:44:21.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.142+0000 7fed7a4c9640 1 -- 172.21.15.83:0/1697706445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed600678c0 msgr2=0x7fed60069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:21.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.142+0000 7fed7a4c9640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed600678c0 0x7fed60069d80 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fed640012f0 tx=0x7fed64001770 comp rx=0 tx=0).stop 2024-09-19T07:44:21.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.142+0000 7fed7a4c9640 1 -- 172.21.15.83:0/1697706445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed74155e10 msgr2=0x7fed741576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:21.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.142+0000 7fed7a4c9640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed74155e10 0x7fed741576d0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0ae040 tx=0x7fed6c01ab90 comp rx=0 tx=0).stop 2024-09-19T07:44:21.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.142+0000 7fed7a4c9640 1 -- 172.21.15.83:0/1697706445 shutdown_connections 2024-09-19T07:44:21.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.142+0000 7fed7a4c9640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed600678c0 0x7fed60069d80 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:21.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.142+0000 7fed7a4c9640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed74155e10 0x7fed741576d0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:21.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.142+0000 7fed7a4c9640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed74152f30 0x7fed74157190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:21.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.142+0000 7fed7a4c9640 1 --2- 172.21.15.83:0/1697706445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed740c4880 0x7fed740ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:21.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.142+0000 7fed7a4c9640 1 -- 172.21.15.83:0/1697706445 >> 172.21.15.83:0/1697706445 conn(0x7fed740aa910 msgr2=0x7fed74150da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:21.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.142+0000 7fed7a4c9640 1 -- 172.21.15.83:0/1697706445 shutdown_connections 2024-09-19T07:44:21.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:21.142+0000 7fed7a4c9640 1 -- 172.21.15.83:0/1697706445 wait complete. 2024-09-19T07:44:21.162 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:44:20:635635+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 173, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 173, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7803": { "addr": "172.21.15.83:6837/3490578864", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 3490578864, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 3490578864, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7803, "incarnation": 171, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:44:19.641476+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7803 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 169, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6628": { "addr": "172.21.15.83:6835/1429520945", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 1429520945, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 1429520945, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6628, "incarnation": 166, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:42:42.535410+0000", "qdb_cluster": [ 6628 ], "qdb_leader": 6628, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6628 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:44:23.163 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:44:23.633 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:44:23.634+0000 7f390aa5c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:44:23.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.638+0000 7fd8ccbcb640 1 Processor -- start 2024-09-19T07:44:23.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.638+0000 7fd8ccbcb640 1 -- start start 2024-09-19T07:44:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.638+0000 7fd8ccbcb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd8c80c14d0 0x7fd8c8155c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.638+0000 7fd8ccbcb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8c80c2840 0x7fd8c806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.638+0000 7fd8ccbcb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd8c806e830 0x7fd8c813b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.638+0000 7fd8ccbcb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd8c80c3980 con 0x7fd8c80c14d0 2024-09-19T07:44:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.638+0000 7fd8ccbcb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd8c80c2d70 con 0x7fd8c806e830 2024-09-19T07:44:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.638+0000 7fd8ccbcb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c80c3050 con 0x7fd8c80c2840 2024-09-19T07:44:23.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.638+0000 7fd8c659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd8c80c14d0 0x7fd8c8155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:23.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.638+0000 7fd8c659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd8c80c14d0 0x7fd8c8155c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60270/0 (socket says 172.21.15.83:60270) 2024-09-19T07:44:23.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.638+0000 7fd8c659c640 1 -- 172.21.15.83:0/3134549520 learned_addr learned my addr 172.21.15.83:0/3134549520 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:23.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.642+0000 7fd8c659c640 1 -- 172.21.15.83:0/3134549520 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8c80c2840 msgr2=0x7fd8c806e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:44:23.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.642+0000 7fd8c659c640 1 --2- 172.21.15.83:0/3134549520 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8c80c2840 0x7fd8c806e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:23.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.642+0000 7fd8c659c640 1 -- 172.21.15.83:0/3134549520 >> v2:172.21.15.184:3300/0 conn(0x7fd8c806e830 msgr2=0x7fd8c813b6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:44:23.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.642+0000 7fd8c659c640 1 --2- 172.21.15.83:0/3134549520 >> v2:172.21.15.184:3300/0 conn(0x7fd8c806e830 0x7fd8c813b6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:23.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.642+0000 7fd8c659c640 1 -- 172.21.15.83:0/3134549520 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c80c3330 con 0x7fd8c80c14d0 2024-09-19T07:44:23.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.646+0000 7fd8c659c640 1 --2- 172.21.15.83:0/3134549520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8c80c14d0 0x7fd8c8155c80 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8070230 tx=0x7fd8b801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cfb75eb2e090ef6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:23.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8c5d9b640 1 -- 172.21.15.83:0/3134549520 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b80ab680 con 0x7fd8c80c14d0 2024-09-19T07:44:23.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8c5d9b640 1 -- 172.21.15.83:0/3134549520 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b80b0070 con 0x7fd8c80c14d0 2024-09-19T07:44:23.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8c5d9b640 1 -- 172.21.15.83:0/3134549520 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b80aba40 con 0x7fd8c80c14d0 2024-09-19T07:44:23.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8ccbcb640 1 -- 172.21.15.83:0/3134549520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8c80c14d0 msgr2=0x7fd8c8155c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:23.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8ccbcb640 1 --2- 172.21.15.83:0/3134549520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8c80c14d0 0x7fd8c8155c80 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8070230 tx=0x7fd8b801c4a0 comp rx=0 tx=0).stop 2024-09-19T07:44:23.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8ccbcb640 1 -- 172.21.15.83:0/3134549520 shutdown_connections 2024-09-19T07:44:23.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8ccbcb640 1 --2- 172.21.15.83:0/3134549520 >> v2:172.21.15.184:3300/0 conn(0x7fd8c806e830 0x7fd8c813b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8ccbcb640 1 --2- 172.21.15.83:0/3134549520 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8c80c2840 0x7fd8c806e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8ccbcb640 1 --2- 172.21.15.83:0/3134549520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8c80c14d0 0x7fd8c8155c80 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8ccbcb640 1 -- 172.21.15.83:0/3134549520 >> 172.21.15.83:0/3134549520 conn(0x7fd8c80c7e40 msgr2=0x7fd8c80c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8ccbcb640 1 -- 172.21.15.83:0/3134549520 shutdown_connections 2024-09-19T07:44:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8ccbcb640 1 -- 172.21.15.83:0/3134549520 wait complete. 2024-09-19T07:44:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8ccbcb640 1 Processor -- start 2024-09-19T07:44:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.650+0000 7fd8ccbcb640 1 -- start start 2024-09-19T07:44:23.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8ccbcb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8c806e830 0x7fd8c81ee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:23.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8ccbcb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8c80c14d0 0x7fd8c80c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:23.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8ccbcb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8c80c2840 0x7fd8c81f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:23.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8ccbcb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c8159a80 con 0x7fd8c80c14d0 2024-09-19T07:44:23.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8ccbcb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c80c1ac0 con 0x7fd8c806e830 2024-09-19T07:44:23.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8ccbcb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c8159c00 con 0x7fd8c80c2840 2024-09-19T07:44:23.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8bdd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8c80c14d0 0x7fd8c80c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:23.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8bdd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8c80c14d0 0x7fd8c80c6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60284/0 (socket says 172.21.15.83:60284) 2024-09-19T07:44:23.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8bdd9b640 1 -- 172.21.15.83:0/1517518232 learned_addr learned my addr 172.21.15.83:0/1517518232 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:23.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8bdd9b640 1 -- 172.21.15.83:0/1517518232 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8c80c2840 msgr2=0x7fd8c81f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:23.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8bdd9b640 1 --2- 172.21.15.83:0/1517518232 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8c80c2840 0x7fd8c81f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:23.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8bdd9b640 1 -- 172.21.15.83:0/1517518232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8c806e830 msgr2=0x7fd8c81ee7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:44:23.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8bdd9b640 1 --2- 172.21.15.83:0/1517518232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8c806e830 0x7fd8c81ee7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:23.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8bdd9b640 1 -- 172.21.15.83:0/1517518232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c8155a30 con 0x7fd8c80c14d0 2024-09-19T07:44:23.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8bdd9b640 1 --2- 172.21.15.83:0/1517518232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8c80c14d0 0x7fd8c80c6ce0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fd8b000b410 tx=0x7fd8b000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:23.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8bf7fe640 1 -- 172.21.15.83:0/1517518232 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0015020 con 0x7fd8c80c14d0 2024-09-19T07:44:23.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8bf7fe640 1 -- 172.21.15.83:0/1517518232 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b0011850 con 0x7fd8c80c14d0 2024-09-19T07:44:23.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8ccbcb640 1 -- 172.21.15.83:0/1517518232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c813ea70 con 0x7fd8c80c14d0 2024-09-19T07:44:23.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.654+0000 7fd8bf7fe640 1 -- 172.21.15.83:0/1517518232 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0011c40 con 0x7fd8c80c14d0 2024-09-19T07:44:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.658+0000 7fd8ccbcb640 1 -- 172.21.15.83:0/1517518232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c80aadc0 con 0x7fd8c80c14d0 2024-09-19T07:44:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.658+0000 7fd8bf7fe640 1 -- 172.21.15.83:0/1517518232 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd8b0018020 con 0x7fd8c80c14d0 2024-09-19T07:44:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.662+0000 7fd8bf7fe640 1 --2- 172.21.15.83:0/1517518232 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8a80678c0 0x7fd8a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:23.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.662+0000 7fd8bf7fe640 1 -- 172.21.15.83:0/1517518232 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7fd8b0086ae0 con 0x7fd8c80c14d0 2024-09-19T07:44:23.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.662+0000 7fd8c659c640 1 --2- 172.21.15.83:0/1517518232 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8a80678c0 0x7fd8a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.662+0000 7fd8ccbcb640 1 -- 172.21.15.83:0/1517518232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c813b6b0 con 0x7fd8c80c14d0 2024-09-19T07:44:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.662+0000 7fd8c659c640 1 --2- 172.21.15.83:0/1517518232 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8a80678c0 0x7fd8a8069d80 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80af000 tx=0x7fd8b80a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:23.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:23.678+0000 7fd8bf7fe640 1 -- 172.21.15.83:0/1517518232 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd8b001f020 con 0x7fd8c80c14d0 2024-09-19T07:44:24.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.890+0000 7fd8ccbcb640 1 -- 172.21.15.83:0/1517518232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8c80c7220 con 0x7fd8c80c14d0 2024-09-19T07:44:24.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.890+0000 7fd8bf7fe640 1 -- 172.21.15.83:0/1517518232 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) ==== 95+0+5717 (secure 0 0 0) 0x7fd8b00172c0 con 0x7fd8c80c14d0 2024-09-19T07:44:24.903 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:44:24.903 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":174,"btime":"2024-09-19T07:44:21:648187+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:44:24.903 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 174 2024-09-19T07:44:24.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.914+0000 7fd8bcd99640 1 -- 172.21.15.83:0/1517518232 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8a80678c0 msgr2=0x7fd8a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:24.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.914+0000 7fd8bcd99640 1 --2- 172.21.15.83:0/1517518232 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8a80678c0 0x7fd8a8069d80 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80af000 tx=0x7fd8b80a7000 comp rx=0 tx=0).stop 2024-09-19T07:44:24.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.914+0000 7fd8bcd99640 1 -- 172.21.15.83:0/1517518232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8c80c14d0 msgr2=0x7fd8c80c6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:24.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.914+0000 7fd8bcd99640 1 --2- 172.21.15.83:0/1517518232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8c80c14d0 0x7fd8c80c6ce0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fd8b000b410 tx=0x7fd8b000b8d0 comp rx=0 tx=0).stop 2024-09-19T07:44:24.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.914+0000 7fd8bcd99640 1 -- 172.21.15.83:0/1517518232 shutdown_connections 2024-09-19T07:44:24.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.914+0000 7fd8bcd99640 1 --2- 172.21.15.83:0/1517518232 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8a80678c0 0x7fd8a8069d80 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:24.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.914+0000 7fd8bcd99640 1 --2- 172.21.15.83:0/1517518232 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8c80c2840 0x7fd8c81f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:24.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.914+0000 7fd8bcd99640 1 --2- 172.21.15.83:0/1517518232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8c80c14d0 0x7fd8c80c6ce0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:24.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.914+0000 7fd8bcd99640 1 --2- 172.21.15.83:0/1517518232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8c806e830 0x7fd8c81ee7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:24.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.914+0000 7fd8bcd99640 1 -- 172.21.15.83:0/1517518232 >> 172.21.15.83:0/1517518232 conn(0x7fd8c80c7e40 msgr2=0x7fd8c814fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:24.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.914+0000 7fd8bcd99640 1 -- 172.21.15.83:0/1517518232 shutdown_connections 2024-09-19T07:44:24.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:24.918+0000 7fd8bcd99640 1 -- 172.21.15.83:0/1517518232 wait complete. 2024-09-19T07:44:24.965 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=6565 2024-09-19T07:44:24.965 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before reviving mds.d 2024-09-19T07:44:32.967 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-19T07:44:32.967 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-19T07:44:32.967 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-19T07:44:32.970 INFO:tasks.ceph.mds.d:Started 2024-09-19T07:44:32.970 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:44:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379703f640 1 Processor -- start 2024-09-19T07:44:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379703f640 1 -- start start 2024-09-19T07:44:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379703f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f378804a990 0x7f378804b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379703f640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f37880d42a0 0x7f378804b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379703f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f378804a500 con 0x7f378803fa40 2024-09-19T07:44:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379703f640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f378804a2b0 con 0x7f37880d42a0 2024-09-19T07:44:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379703f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f378804b010 con 0x7f378804a990 2024-09-19T07:44:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379683e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f37880d42a0 0x7f378804b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379683e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f37880d42a0 0x7f378804b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:44126/0 (socket says 172.21.15.83:44126) 2024-09-19T07:44:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379683e640 1 -- 172.21.15.83:0/1533947119 learned_addr learned my addr 172.21.15.83:0/1533947119 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379683e640 1 -- 172.21.15.83:0/1533947119 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f378804a990 msgr2=0x7f378804b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:44:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379683e640 1 --2- 172.21.15.83:0/1533947119 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f378804a990 0x7f378804b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379683e640 1 -- 172.21.15.83:0/1533947119 >> v1:172.21.15.83:6789/0 conn(0x7f378803fa40 legacy=0x7f3788046d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:44:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.358+0000 7f379683e640 1 -- 172.21.15.83:0/1533947119 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37880d7550 con 0x7f37880d42a0 2024-09-19T07:44:33.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.366+0000 7f379683e640 1 --2- 172.21.15.83:0/1533947119 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37880d42a0 0x7f378804b9e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f37900694d0 tx=0x7f379009b500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1907dc2e41f1cc45 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:33.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.374+0000 7f379503b640 1 -- 172.21.15.83:0/1533947119 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379012b440 con 0x7f37880d42a0 2024-09-19T07:44:33.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.374+0000 7f379503b640 1 -- 172.21.15.83:0/1533947119 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f379012b620 con 0x7f37880d42a0 2024-09-19T07:44:33.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.374+0000 7f379503b640 1 -- 172.21.15.83:0/1533947119 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379012ba90 con 0x7f37880d42a0 2024-09-19T07:44:33.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.374+0000 7f379703f640 1 -- 172.21.15.83:0/1533947119 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37880d42a0 msgr2=0x7f378804b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:33.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.374+0000 7f379703f640 1 --2- 172.21.15.83:0/1533947119 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37880d42a0 0x7f378804b9e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f37900694d0 tx=0x7f379009b500 comp rx=0 tx=0).stop 2024-09-19T07:44:33.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.398+0000 7f379703f640 1 -- 172.21.15.83:0/1533947119 shutdown_connections 2024-09-19T07:44:33.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.398+0000 7f379703f640 1 --2- 172.21.15.83:0/1533947119 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37880d42a0 0x7f378804b9e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:33.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.398+0000 7f379703f640 1 --2- 172.21.15.83:0/1533947119 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f378804a990 0x7f378804b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:33.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.402+0000 7f379703f640 1 -- 172.21.15.83:0/1533947119 >> 172.21.15.83:0/1533947119 conn(0x7f3788048e30 msgr2=0x7f3788049210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:33.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.402+0000 7f379703f640 1 -- 172.21.15.83:0/1533947119 shutdown_connections 2024-09-19T07:44:33.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.402+0000 7f379703f640 1 -- 172.21.15.83:0/1533947119 wait complete. 2024-09-19T07:44:33.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.402+0000 7f379703f640 1 Processor -- start 2024-09-19T07:44:33.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.402+0000 7f379703f640 1 -- start start 2024-09-19T07:44:33.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.402+0000 7f379703f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f378803fa40 0x7f37880d4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:33.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.402+0000 7f379703f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f378804a990 0x7f37880d7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:33.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.402+0000 7f379703f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37880d42a0 0x7f37880d8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:33.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.402+0000 7f379703f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3788047440 con 0x7f37880d42a0 2024-09-19T07:44:33.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.406+0000 7f379703f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3788049c80 con 0x7f378804a990 2024-09-19T07:44:33.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.406+0000 7f379703f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f37880475c0 con 0x7f378803fa40 2024-09-19T07:44:33.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.406+0000 7f379683e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37880d42a0 0x7f37880d8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:33.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.406+0000 7f379683e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37880d42a0 0x7f37880d8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:38784/0 (socket says 172.21.15.83:38784) 2024-09-19T07:44:33.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.406+0000 7f379683e640 1 -- 172.21.15.83:0/3850224984 learned_addr learned my addr 172.21.15.83:0/3850224984 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:33.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.406+0000 7f379603d640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f378803fa40 0x7f37880d4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:33.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.410+0000 7f379683e640 1 -- 172.21.15.83:0/3850224984 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f378803fa40 msgr2=0x7f37880d4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:33.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.410+0000 7f379683e640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f378803fa40 0x7f37880d4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:33.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.410+0000 7f379683e640 1 -- 172.21.15.83:0/3850224984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f378804a990 msgr2=0x7f37880d7d20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:44:33.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.410+0000 7f379683e640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f378804a990 0x7f37880d7d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:33.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.410+0000 7f379683e640 1 -- 172.21.15.83:0/3850224984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37880da110 con 0x7f37880d42a0 2024-09-19T07:44:33.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.410+0000 7f379683e640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37880d42a0 0x7f37880d8260 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f379006a530 tx=0x7f37900748d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:33.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.410+0000 7f3786ffd640 1 -- 172.21.15.83:0/3850224984 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379013a070 con 0x7f37880d42a0 2024-09-19T07:44:33.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.410+0000 7f3786ffd640 1 -- 172.21.15.83:0/3850224984 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3790136640 con 0x7f37880d42a0 2024-09-19T07:44:33.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.410+0000 7f3786ffd640 1 -- 172.21.15.83:0/3850224984 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3790136a30 con 0x7f37880d42a0 2024-09-19T07:44:33.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.410+0000 7f379703f640 1 -- 172.21.15.83:0/3850224984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3788045ea0 con 0x7f37880d42a0 2024-09-19T07:44:33.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.410+0000 7f379703f640 1 -- 172.21.15.83:0/3850224984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37880da530 con 0x7f37880d42a0 2024-09-19T07:44:33.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.414+0000 7f379703f640 1 -- 172.21.15.83:0/3850224984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37880da730 con 0x7f37880d42a0 2024-09-19T07:44:33.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.418+0000 7f3786ffd640 1 -- 172.21.15.83:0/3850224984 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f379013d070 con 0x7f37880d42a0 2024-09-19T07:44:33.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.418+0000 7f3786ffd640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37740679c0 0x7f3774069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:33.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.418+0000 7f3786ffd640 1 -- 172.21.15.83:0/3850224984 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7f37901a0ba0 con 0x7f37880d42a0 2024-09-19T07:44:33.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.422+0000 7f379603d640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37740679c0 0x7f3774069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:33.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.422+0000 7f379603d640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37740679c0 0x7f3774069e80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f378c006fd0 tx=0x7f378c009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:33.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:33.430+0000 7f3786ffd640 1 -- 172.21.15.83:0/3850224984 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f37880da730 con 0x7f37880d42a0 2024-09-19T07:44:33.641 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-19T07:44:34.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.318+0000 7f379703f640 1 -- 172.21.15.83:0/3850224984 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37880d6350 con 0x7f37880d42a0 2024-09-19T07:44:34.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.330+0000 7f3786ffd640 1 -- 172.21.15.83:0/3850224984 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+6546 (secure 0 0 0) 0x7f3790144030 con 0x7f37880d42a0 2024-09-19T07:44:34.334 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:44:34.334 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":175,"btime":"2024-09-19T07:44:34:305274+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:44:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 175 2024-09-19T07:44:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.338+0000 7f3784ef9640 1 -- 172.21.15.83:0/3850224984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37740679c0 msgr2=0x7f3774069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.338+0000 7f3784ef9640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37740679c0 0x7f3774069e80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f378c006fd0 tx=0x7f378c009060 comp rx=0 tx=0).stop 2024-09-19T07:44:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.338+0000 7f3784ef9640 1 -- 172.21.15.83:0/3850224984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37880d42a0 msgr2=0x7f37880d8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.338+0000 7f3784ef9640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37880d42a0 0x7f37880d8260 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f379006a530 tx=0x7f37900748d0 comp rx=0 tx=0).stop 2024-09-19T07:44:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.338+0000 7f3784ef9640 1 -- 172.21.15.83:0/3850224984 shutdown_connections 2024-09-19T07:44:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.338+0000 7f3784ef9640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f37740679c0 0x7f3774069e80 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.338+0000 7f3784ef9640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37880d42a0 0x7f37880d8260 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.338+0000 7f3784ef9640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f378804a990 0x7f37880d7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.338+0000 7f3784ef9640 1 --2- 172.21.15.83:0/3850224984 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f378803fa40 0x7f37880d4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.338+0000 7f3784ef9640 1 -- 172.21.15.83:0/3850224984 >> 172.21.15.83:0/3850224984 conn(0x7f3788048e30 msgr2=0x7f37880cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.338+0000 7f3784ef9640 1 -- 172.21.15.83:0/3850224984 shutdown_connections 2024-09-19T07:44:34.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:34.342+0000 7f3784ef9640 1 -- 172.21.15.83:0/3850224984 wait complete. 2024-09-19T07:44:34.415 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-19T07:44:34.415 INFO:tasks.mds_thrash.fs.[a]:waiting for 36 secs before thrashing 2024-09-19T07:44:40.178 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:44:40.181 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:44:53.628 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:44:54.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.057+0000 7f29ec803640 1 Processor -- start 2024-09-19T07:44:54.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.057+0000 7f29ec803640 1 -- start start 2024-09-19T07:44:54.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.061+0000 7f29ec803640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29e808c850 0x7f29e8165bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:54.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.061+0000 7f29ec803640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f29e80c1680 0x7f29e80c2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:54.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.061+0000 7f29ec803640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e806e4e0 con 0x7f29e80c3650 2024-09-19T07:44:54.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.061+0000 7f29ec803640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f29e81662b0 con 0x7f29e80c1680 2024-09-19T07:44:54.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.061+0000 7f29ec803640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f29e8166590 con 0x7f29e808c850 2024-09-19T07:44:54.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.061+0000 7f29e759e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f29e80c1680 0x7f29e80c2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:54.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.061+0000 7f29e759e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f29e80c1680 0x7f29e80c2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:36696/0 (socket says 172.21.15.83:36696) 2024-09-19T07:44:54.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.061+0000 7f29e759e640 1 -- 172.21.15.83:0/2047814976 learned_addr learned my addr 172.21.15.83:0/2047814976 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:54.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.065+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2226503604 0 0) 0x7f29e806e4e0 con 0x7f29e80c3650 2024-09-19T07:44:54.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.065+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29d0003510 con 0x7f29e80c3650 2024-09-19T07:44:54.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.065+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 469126550 0 0) 0x7f29d0003510 con 0x7f29e80c3650 2024-09-19T07:44:54.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.065+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29e808c850 msgr2=0x7f29e8165bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:44:54.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.065+0000 7f29e5d9b640 1 --2- 172.21.15.83:0/2047814976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29e808c850 0x7f29e8165bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:54.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.065+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 >> v2:172.21.15.184:3300/0 conn(0x7f29e80c1680 msgr2=0x7f29e80c2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:44:54.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.065+0000 7f29e5d9b640 1 --2- 172.21.15.83:0/2047814976 >> v2:172.21.15.184:3300/0 conn(0x7f29e80c1680 0x7f29e80c2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:54.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.065+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e8166870 con 0x7f29e80c3650 2024-09-19T07:44:54.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.065+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f29d8001680 con 0x7f29e80c3650 2024-09-19T07:44:54.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.065+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 >> v1:172.21.15.83:6789/0 conn(0x7f29e80c3650 legacy=0x7f29e808c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:54.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.069+0000 7f29e5d9b640 1 --2- 172.21.15.83:0/2047814976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29d0003b20 0x7f29d0005f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:54.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.069+0000 7f29e5d9b640 1 --2- 172.21.15.83:0/2047814976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29d00065c0 0x7f29d0006a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:54.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.069+0000 7f29e5d9b640 1 --2- 172.21.15.83:0/2047814976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d00071f0 0x7f29d00147e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:54.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.069+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29e81662b0 con 0x7f29d00071f0 2024-09-19T07:44:54.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.069+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f29e8166590 con 0x7f29d0003b20 2024-09-19T07:44:54.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.069+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f29d00076a0 con 0x7f29d00065c0 2024-09-19T07:44:54.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.069+0000 7f29e759e640 1 --2- 172.21.15.83:0/2047814976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d00071f0 0x7f29d00147e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:54.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.069+0000 7f29e759e640 1 -- 172.21.15.83:0/2047814976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29d00065c0 msgr2=0x7f29d0006a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:44:54.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.069+0000 7f29e759e640 1 --2- 172.21.15.83:0/2047814976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29d00065c0 0x7f29d0006a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:54.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.069+0000 7f29e759e640 1 -- 172.21.15.83:0/2047814976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29d0003b20 msgr2=0x7f29d0005f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:44:54.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.069+0000 7f29e759e640 1 --2- 172.21.15.83:0/2047814976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29d0003b20 0x7f29d0005f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:54.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.069+0000 7f29e759e640 1 -- 172.21.15.83:0/2047814976 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d0003510 con 0x7f29d00071f0 2024-09-19T07:44:54.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.077+0000 7f29e759e640 1 --2- 172.21.15.83:0/2047814976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d00071f0 0x7f29d00147e0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f29e000a150 tx=0x7f29e001d0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0163b7626e19f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:54.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.077+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e00b3070 con 0x7f29d00071f0 2024-09-19T07:44:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.093+0000 7f29e5d9b640 1 -- 172.21.15.83:0/2047814976 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e00ae410 con 0x7f29d00071f0 2024-09-19T07:44:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.093+0000 7f29ec803640 1 -- 172.21.15.83:0/2047814976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d00071f0 msgr2=0x7f29d00147e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.093+0000 7f29ec803640 1 --2- 172.21.15.83:0/2047814976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d00071f0 0x7f29d00147e0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f29e000a150 tx=0x7f29e001d0b0 comp rx=0 tx=0).stop 2024-09-19T07:44:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.093+0000 7f29e759e640 1 -- 172.21.15.83:0/2047814976 reap_dead start 2024-09-19T07:44:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.093+0000 7f29ec803640 1 -- 172.21.15.83:0/2047814976 shutdown_connections 2024-09-19T07:44:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.093+0000 7f29ec803640 1 -- 172.21.15.83:0/2047814976 >> 172.21.15.83:0/2047814976 conn(0x7f29e80c7370 msgr2=0x7f29e80c7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.097+0000 7f29ec803640 1 -- 172.21.15.83:0/2047814976 shutdown_connections 2024-09-19T07:44:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.097+0000 7f29ec803640 1 -- 172.21.15.83:0/2047814976 wait complete. 2024-09-19T07:44:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.097+0000 7f29ec803640 1 Processor -- start 2024-09-19T07:44:54.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.097+0000 7f29ec803640 1 -- start start 2024-09-19T07:44:54.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.097+0000 7f29ec803640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29e80c67b0 0x7f29e80a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:54.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.097+0000 7f29ec803640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29e8164af0 0x7f29e80a4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:54.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.097+0000 7f29ec803640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29e81637a0 0x7f29e8163c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:54.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.097+0000 7f29ec803640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29e80c3bb0 con 0x7f29e81637a0 2024-09-19T07:44:54.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.097+0000 7f29ec803640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f29e80c3a30 con 0x7f29e8164af0 2024-09-19T07:44:54.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.097+0000 7f29ec803640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f29e8165bb0 con 0x7f29e80c67b0 2024-09-19T07:44:54.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.101+0000 7f29e6d9d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29e80c67b0 0x7f29e80a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:54.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.101+0000 7f29e759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29e81637a0 0x7f29e8163c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:54.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.101+0000 7f29e759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29e81637a0 0x7f29e8163c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54326/0 (socket says 172.21.15.83:54326) 2024-09-19T07:44:54.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.101+0000 7f29e759e640 1 -- 172.21.15.83:0/3107531205 learned_addr learned my addr 172.21.15.83:0/3107531205 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:54.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.101+0000 7f29e759e640 1 -- 172.21.15.83:0/3107531205 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29e80c67b0 msgr2=0x7f29e80a41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:54.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.101+0000 7f29e759e640 1 --2- 172.21.15.83:0/3107531205 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29e80c67b0 0x7f29e80a41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:54.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.101+0000 7f29e759e640 1 -- 172.21.15.83:0/3107531205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29e8164af0 msgr2=0x7f29e80a4730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:44:54.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.101+0000 7f29e759e640 1 --2- 172.21.15.83:0/3107531205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29e8164af0 0x7f29e80a4730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:54.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.101+0000 7f29e759e640 1 -- 172.21.15.83:0/3107531205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e81fb9f0 con 0x7f29e81637a0 2024-09-19T07:44:54.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.101+0000 7f29e759e640 1 --2- 172.21.15.83:0/3107531205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29e81637a0 0x7f29e8163c20 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f29e0098be0 tx=0x7f29e0008800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:54.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.121+0000 7f29cf7fe640 1 -- 172.21.15.83:0/3107531205 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e00bb020 con 0x7f29e81637a0 2024-09-19T07:44:54.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.121+0000 7f29cf7fe640 1 -- 172.21.15.83:0/3107531205 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e00ae410 con 0x7f29e81637a0 2024-09-19T07:44:54.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.121+0000 7f29ec803640 1 -- 172.21.15.83:0/3107531205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29e81fbcd0 con 0x7f29e81637a0 2024-09-19T07:44:54.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.121+0000 7f29ec803640 1 -- 172.21.15.83:0/3107531205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29e8164160 con 0x7f29e81637a0 2024-09-19T07:44:54.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.121+0000 7f29ec803640 1 -- 172.21.15.83:0/3107531205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29e8165f90 con 0x7f29e81637a0 2024-09-19T07:44:54.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.125+0000 7f29cf7fe640 1 -- 172.21.15.83:0/3107531205 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e00b3040 con 0x7f29e81637a0 2024-09-19T07:44:54.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.125+0000 7f29cf7fe640 1 -- 172.21.15.83:0/3107531205 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f29e00be070 con 0x7f29e81637a0 2024-09-19T07:44:54.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.125+0000 7f29cf7fe640 1 --2- 172.21.15.83:0/3107531205 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29d40679f0 0x7f29d4069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:54.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.129+0000 7f29e6d9d640 1 --2- 172.21.15.83:0/3107531205 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29d40679f0 0x7f29d4069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:54.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.129+0000 7f29e6d9d640 1 --2- 172.21.15.83:0/3107531205 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29d40679f0 0x7f29d4069eb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f29d8001ec0 tx=0x7f29d8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:54.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.129+0000 7f29cf7fe640 1 -- 172.21.15.83:0/3107531205 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7f29e0120ef0 con 0x7f29e81637a0 2024-09-19T07:44:54.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:54.137+0000 7f29cf7fe640 1 -- 172.21.15.83:0/3107531205 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f29e0016dc0 con 0x7f29e81637a0 2024-09-19T07:44:55.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.133+0000 7f29ec803640 1 -- 172.21.15.83:0/3107531205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29e8068290 con 0x7f29e81637a0 2024-09-19T07:44:55.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.133+0000 7f29cf7fe640 1 -- 172.21.15.83:0/3107531205 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+6546 (secure 0 0 0) 0x7f29e00f04a0 con 0x7f29e81637a0 2024-09-19T07:44:55.136 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:44:55.136 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":175,"btime":"2024-09-19T07:44:34:305274+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:44:55.137 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 175 2024-09-19T07:44:55.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.141+0000 7f29ec803640 1 -- 172.21.15.83:0/3107531205 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29d40679f0 msgr2=0x7f29d4069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:55.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.141+0000 7f29ec803640 1 --2- 172.21.15.83:0/3107531205 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29d40679f0 0x7f29d4069eb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f29d8001ec0 tx=0x7f29d8009040 comp rx=0 tx=0).stop 2024-09-19T07:44:55.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.141+0000 7f29ec803640 1 -- 172.21.15.83:0/3107531205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29e81637a0 msgr2=0x7f29e8163c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:55.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.141+0000 7f29ec803640 1 --2- 172.21.15.83:0/3107531205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29e81637a0 0x7f29e8163c20 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f29e0098be0 tx=0x7f29e0008800 comp rx=0 tx=0).stop 2024-09-19T07:44:55.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.145+0000 7f29ec803640 1 -- 172.21.15.83:0/3107531205 shutdown_connections 2024-09-19T07:44:55.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.145+0000 7f29ec803640 1 --2- 172.21.15.83:0/3107531205 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29d40679f0 0x7f29d4069eb0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:55.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.145+0000 7f29ec803640 1 --2- 172.21.15.83:0/3107531205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29e81637a0 0x7f29e8163c20 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:55.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.145+0000 7f29ec803640 1 --2- 172.21.15.83:0/3107531205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29e8164af0 0x7f29e80a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:55.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.145+0000 7f29ec803640 1 --2- 172.21.15.83:0/3107531205 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29e80c67b0 0x7f29e80a41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:55.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.145+0000 7f29ec803640 1 -- 172.21.15.83:0/3107531205 >> 172.21.15.83:0/3107531205 conn(0x7f29e80c7370 msgr2=0x7f29e8086890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:55.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.145+0000 7f29ec803640 1 -- 172.21.15.83:0/3107531205 shutdown_connections 2024-09-19T07:44:55.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.145+0000 7f29ec803640 1 -- 172.21.15.83:0/3107531205 wait complete. 2024-09-19T07:44:55.180 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-19T07:44:55.181 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:44:55.181 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:44:55.204 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:44:55.204 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:44:55.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.801+0000 7fabf9c43640 1 Processor -- start 2024-09-19T07:44:55.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf9c43640 1 -- start start 2024-09-19T07:44:55.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf9c43640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabec00b880 0x7fabec049480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:55.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf9c43640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fabec04ac30 0x7fabec0499c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:55.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fabec04ac30 0x7fabec0499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:55.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fabec04ac30 0x7fabec0499c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54338/0 (socket says 172.21.15.83:54338) 2024-09-19T07:44:55.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf2ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabec00b880 0x7fabec049480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:55.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf2ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabec00b880 0x7fabec049480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:47136/0 (socket says 172.21.15.83:47136) 2024-09-19T07:44:55.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf3fff640 1 -- 172.21.15.83:0/2491673863 learned_addr learned my addr 172.21.15.83:0/2491673863 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:55.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf9c43640 1 -- 172.21.15.83:0/2491673863 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabec0ddfa0 con 0x7fabec006c80 2024-09-19T07:44:55.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf9c43640 1 -- 172.21.15.83:0/2491673863 --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fabec0dee70 con 0x7fabec04ac30 2024-09-19T07:44:55.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf9c43640 1 -- 172.21.15.83:0/2491673863 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fabec0de260 con 0x7fabec00b880 2024-09-19T07:44:55.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf3fff640 1 -- 172.21.15.83:0/2491673863 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabec00b880 msgr2=0x7fabec049480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:44:55.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf3fff640 1 --2- 172.21.15.83:0/2491673863 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabec00b880 0x7fabec049480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:55.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf3fff640 1 -- 172.21.15.83:0/2491673863 >> v1:172.21.15.83:6789/0 conn(0x7fabec006c80 legacy=0x7fabec0d45b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:44:55.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.805+0000 7fabf3fff640 1 -- 172.21.15.83:0/2491673863 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec0de540 con 0x7fabec04ac30 2024-09-19T07:44:55.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.809+0000 7fabf2ffd640 1 --2- 172.21.15.83:0/2491673863 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabec00b880 0x7fabec049480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:44:55.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.809+0000 7fabf3fff640 1 --2- 172.21.15.83:0/2491673863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabec04ac30 0x7fabec0499c0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fabf4063ae0 tx=0x7fabf4098ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70785ed31b531d9a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:55.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.821+0000 7fabf27fc640 1 -- 172.21.15.83:0/2491673863 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf4121b10 con 0x7fabec04ac30 2024-09-19T07:44:55.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.821+0000 7fabf27fc640 1 -- 172.21.15.83:0/2491673863 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabf4121cb0 con 0x7fabec04ac30 2024-09-19T07:44:55.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.821+0000 7fabf27fc640 1 -- 172.21.15.83:0/2491673863 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf4129640 con 0x7fabec04ac30 2024-09-19T07:44:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.821+0000 7fabf9c43640 1 -- 172.21.15.83:0/2491673863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabec04ac30 msgr2=0x7fabec0499c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.821+0000 7fabf9c43640 1 --2- 172.21.15.83:0/2491673863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabec04ac30 0x7fabec0499c0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fabf4063ae0 tx=0x7fabf4098ca0 comp rx=0 tx=0).stop 2024-09-19T07:44:55.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.821+0000 7fabf9c43640 1 -- 172.21.15.83:0/2491673863 shutdown_connections 2024-09-19T07:44:55.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.821+0000 7fabf9c43640 1 --2- 172.21.15.83:0/2491673863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabec04ac30 0x7fabec0499c0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:55.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.821+0000 7fabf9c43640 1 --2- 172.21.15.83:0/2491673863 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabec00b880 0x7fabec049480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:55.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.821+0000 7fabf9c43640 1 -- 172.21.15.83:0/2491673863 >> 172.21.15.83:0/2491673863 conn(0x7fabec047d20 msgr2=0x7fabec048130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:55.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.833+0000 7fabf9c43640 1 -- 172.21.15.83:0/2491673863 shutdown_connections 2024-09-19T07:44:55.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.833+0000 7fabf9c43640 1 -- 172.21.15.83:0/2491673863 wait complete. 2024-09-19T07:44:55.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.833+0000 7fabf9c43640 1 Processor -- start 2024-09-19T07:44:55.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.833+0000 7fabf9c43640 1 -- start start 2024-09-19T07:44:55.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.837+0000 7fabf9c43640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabec006c80 0x7fabec041950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:55.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.837+0000 7fabf9c43640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabec00b880 0x7fabec02fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:55.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.837+0000 7fabf9c43640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabec04ac30 0x7fabec042370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:55.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.837+0000 7fabf9c43640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fabec0d4e40 con 0x7fabec04ac30 2024-09-19T07:44:55.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.837+0000 7fabf9c43640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fabec0d4cc0 con 0x7fabec006c80 2024-09-19T07:44:55.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.837+0000 7fabf9c43640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fabec007060 con 0x7fabec00b880 2024-09-19T07:44:55.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.841+0000 7fabf3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabec04ac30 0x7fabec042370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:55.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.841+0000 7fabf3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabec04ac30 0x7fabec042370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54346/0 (socket says 172.21.15.83:54346) 2024-09-19T07:44:55.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.841+0000 7fabf3fff640 1 -- 172.21.15.83:0/3700067615 learned_addr learned my addr 172.21.15.83:0/3700067615 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:55.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.841+0000 7fabf3fff640 1 -- 172.21.15.83:0/3700067615 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabec00b880 msgr2=0x7fabec02fd20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:44:55.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.841+0000 7fabf3fff640 1 --2- 172.21.15.83:0/3700067615 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabec00b880 0x7fabec02fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:55.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.841+0000 7fabf3fff640 1 -- 172.21.15.83:0/3700067615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabec006c80 msgr2=0x7fabec041950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:44:55.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.841+0000 7fabf3fff640 1 --2- 172.21.15.83:0/3700067615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabec006c80 0x7fabec041950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:55.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.841+0000 7fabf3fff640 1 -- 172.21.15.83:0/3700067615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec03f2d0 con 0x7fabec04ac30 2024-09-19T07:44:55.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.841+0000 7fabf3fff640 1 --2- 172.21.15.83:0/3700067615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabec04ac30 0x7fabec042370 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fabf4000b70 tx=0x7fabf40748d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:55.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.857+0000 7fabdffff640 1 -- 172.21.15.83:0/3700067615 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf4121e60 con 0x7fabec04ac30 2024-09-19T07:44:55.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.857+0000 7fabf9c43640 1 -- 172.21.15.83:0/3700067615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabec0d7820 con 0x7fabec04ac30 2024-09-19T07:44:55.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.861+0000 7fabf9c43640 1 -- 172.21.15.83:0/3700067615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabec030a90 con 0x7fabec04ac30 2024-09-19T07:44:55.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.861+0000 7fabdffff640 1 -- 172.21.15.83:0/3700067615 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabf4131400 con 0x7fabec04ac30 2024-09-19T07:44:55.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.861+0000 7fabdffff640 1 -- 172.21.15.83:0/3700067615 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf41315f0 con 0x7fabec04ac30 2024-09-19T07:44:55.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.861+0000 7fabf9c43640 1 -- 172.21.15.83:0/3700067615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabec03ed50 con 0x7fabec04ac30 2024-09-19T07:44:55.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.865+0000 7fabdffff640 1 -- 172.21.15.83:0/3700067615 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fabf4139020 con 0x7fabec04ac30 2024-09-19T07:44:55.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.869+0000 7fabdffff640 1 --2- 172.21.15.83:0/3700067615 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fabd80679f0 0x7fabd8069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:55.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.869+0000 7fabdffff640 1 -- 172.21.15.83:0/3700067615 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7fabf4050810 con 0x7fabec04ac30 2024-09-19T07:44:55.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.869+0000 7fabf37fe640 1 --2- 172.21.15.83:0/3700067615 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fabd80679f0 0x7fabd8069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:55.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.869+0000 7fabf37fe640 1 --2- 172.21.15.83:0/3700067615 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fabd80679f0 0x7fabd8069eb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fabe80095d0 tx=0x7fabe8006b30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:55.885+0000 7fabdffff640 1 -- 172.21.15.83:0/3700067615 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fabf40929f0 con 0x7fabec04ac30 2024-09-19T07:44:56.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.709+0000 7fabf9c43640 1 -- 172.21.15.83:0/3700067615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabec000e30 con 0x7fabec04ac30 2024-09-19T07:44:56.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.717+0000 7fabdffff640 1 -- 172.21.15.83:0/3700067615 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+6546 (secure 0 0 0) 0x7fabf416c630 con 0x7fabec04ac30 2024-09-19T07:44:56.716 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:44:56.716 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":175,"btime":"2024-09-19T07:44:34:305274+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:44:56.716 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 175 2024-09-19T07:44:56.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.717+0000 7fabf9c43640 1 -- 172.21.15.83:0/3700067615 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fabd80679f0 msgr2=0x7fabd8069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:56.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.717+0000 7fabf9c43640 1 --2- 172.21.15.83:0/3700067615 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fabd80679f0 0x7fabd8069eb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fabe80095d0 tx=0x7fabe8006b30 comp rx=0 tx=0).stop 2024-09-19T07:44:56.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.717+0000 7fabf9c43640 1 -- 172.21.15.83:0/3700067615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabec04ac30 msgr2=0x7fabec042370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:56.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.717+0000 7fabf9c43640 1 --2- 172.21.15.83:0/3700067615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabec04ac30 0x7fabec042370 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fabf4000b70 tx=0x7fabf40748d0 comp rx=0 tx=0).stop 2024-09-19T07:44:56.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.721+0000 7fabf9c43640 1 -- 172.21.15.83:0/3700067615 shutdown_connections 2024-09-19T07:44:56.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.721+0000 7fabf9c43640 1 --2- 172.21.15.83:0/3700067615 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fabd80679f0 0x7fabd8069eb0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:56.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.721+0000 7fabf9c43640 1 --2- 172.21.15.83:0/3700067615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabec04ac30 0x7fabec042370 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:56.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.721+0000 7fabf9c43640 1 --2- 172.21.15.83:0/3700067615 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabec00b880 0x7fabec02fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:56.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.721+0000 7fabf9c43640 1 --2- 172.21.15.83:0/3700067615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabec006c80 0x7fabec041950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:56.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.721+0000 7fabf9c43640 1 -- 172.21.15.83:0/3700067615 >> 172.21.15.83:0/3700067615 conn(0x7fabec047d20 msgr2=0x7fabec049480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:56.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.721+0000 7fabf9c43640 1 -- 172.21.15.83:0/3700067615 shutdown_connections 2024-09-19T07:44:56.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:56.721+0000 7fabf9c43640 1 -- 172.21.15.83:0/3700067615 wait complete. 2024-09-19T07:44:56.783 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:44:58.783 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:44:59.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.169+0000 7fcc25042640 1 Processor -- start 2024-09-19T07:44:59.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.169+0000 7fcc25042640 1 -- start start 2024-09-19T07:44:59.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.173+0000 7fcc25042640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcc200c3fa0 0x7fcc20070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:59.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.173+0000 7fcc25042640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc200ab3c0 0x7fcc200c4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:59.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.173+0000 7fcc25042640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc200aa230 con 0x7fcc200c3360 2024-09-19T07:44:59.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.173+0000 7fcc25042640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fcc20074190 con 0x7fcc200c3fa0 2024-09-19T07:44:59.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.173+0000 7fcc25042640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc200c1ff0 con 0x7fcc200ab3c0 2024-09-19T07:44:59.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.173+0000 7fcc1ed9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcc200c3fa0 0x7fcc20070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:59.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.173+0000 7fcc1ed9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcc200c3fa0 0x7fcc20070ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:36710/0 (socket says 172.21.15.83:36710) 2024-09-19T07:44:59.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.173+0000 7fcc1ed9d640 1 -- 172.21.15.83:0/1951946538 learned_addr learned my addr 172.21.15.83:0/1951946538 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:59.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.173+0000 7fcc17fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc200ab3c0 0x7fcc200c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:59.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.177+0000 7fcc1ed9d640 1 -- 172.21.15.83:0/1951946538 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc200ab3c0 msgr2=0x7fcc200c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:44:59.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.177+0000 7fcc1ed9d640 1 --2- 172.21.15.83:0/1951946538 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc200ab3c0 0x7fcc200c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:59.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.177+0000 7fcc1ed9d640 1 -- 172.21.15.83:0/1951946538 >> v1:172.21.15.83:6789/0 conn(0x7fcc200c3360 legacy=0x7fcc2006dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:59.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.177+0000 7fcc1ed9d640 1 -- 172.21.15.83:0/1951946538 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc200c28a0 con 0x7fcc200c3fa0 2024-09-19T07:44:59.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.181+0000 7fcc1ed9d640 1 --2- 172.21.15.83:0/1951946538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc200c3fa0 0x7fcc20070ea0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcc100091e0 tx=0x7fcc1001d6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=868975ff3b604d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:59.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.185+0000 7fcc1e59c640 1 -- 172.21.15.83:0/1951946538 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100a9610 con 0x7fcc200c3fa0 2024-09-19T07:44:59.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.185+0000 7fcc1e59c640 1 -- 172.21.15.83:0/1951946538 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc100a97b0 con 0x7fcc200c3fa0 2024-09-19T07:44:59.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.185+0000 7fcc1e59c640 1 -- 172.21.15.83:0/1951946538 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100a9b80 con 0x7fcc200c3fa0 2024-09-19T07:44:59.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.185+0000 7fcc25042640 1 -- 172.21.15.83:0/1951946538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc200c3fa0 msgr2=0x7fcc20070ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:59.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.185+0000 7fcc25042640 1 --2- 172.21.15.83:0/1951946538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc200c3fa0 0x7fcc20070ea0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcc100091e0 tx=0x7fcc1001d6d0 comp rx=0 tx=0).stop 2024-09-19T07:44:59.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.201+0000 7fcc25042640 1 -- 172.21.15.83:0/1951946538 shutdown_connections 2024-09-19T07:44:59.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.201+0000 7fcc25042640 1 --2- 172.21.15.83:0/1951946538 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc200ab3c0 0x7fcc200c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:59.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.201+0000 7fcc25042640 1 --2- 172.21.15.83:0/1951946538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc200c3fa0 0x7fcc20070ea0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:59.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.201+0000 7fcc25042640 1 -- 172.21.15.83:0/1951946538 >> 172.21.15.83:0/1951946538 conn(0x7fcc20075390 msgr2=0x7fcc2006bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:59.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.201+0000 7fcc25042640 1 -- 172.21.15.83:0/1951946538 shutdown_connections 2024-09-19T07:44:59.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.201+0000 7fcc25042640 1 -- 172.21.15.83:0/1951946538 wait complete. 2024-09-19T07:44:59.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.201+0000 7fcc25042640 1 Processor -- start 2024-09-19T07:44:59.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.201+0000 7fcc25042640 1 -- start start 2024-09-19T07:44:59.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc25042640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc200ab3c0 0x7fcc2009d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:59.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc25042640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc200c3360 0x7fcc2009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:59.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc25042640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc200c3fa0 0x7fcc20202c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:59.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc25042640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcc200c48c0 con 0x7fcc200c3360 2024-09-19T07:44:59.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc25042640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcc200c2e40 con 0x7fcc200c3fa0 2024-09-19T07:44:59.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc25042640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc2016f7a0 con 0x7fcc200ab3c0 2024-09-19T07:44:59.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc1f59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc200c3fa0 0x7fcc20202c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:59.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc200c3360 0x7fcc2009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:59.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc200c3360 0x7fcc2009cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54362/0 (socket says 172.21.15.83:54362) 2024-09-19T07:44:59.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc1f59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc200c3fa0 0x7fcc20202c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:36718/0 (socket says 172.21.15.83:36718) 2024-09-19T07:44:59.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc17fff640 1 -- 172.21.15.83:0/519173564 learned_addr learned my addr 172.21.15.83:0/519173564 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:44:59.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc1f59e640 1 -- 172.21.15.83:0/519173564 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc200ab3c0 msgr2=0x7fcc2009d980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:44:59.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc1f59e640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc200ab3c0 0x7fcc2009d980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:59.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc1f59e640 1 -- 172.21.15.83:0/519173564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc200c3360 msgr2=0x7fcc2009cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:59.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc1f59e640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc200c3360 0x7fcc2009cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:59.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc1f59e640 1 -- 172.21.15.83:0/519173564 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2009dec0 con 0x7fcc200c3fa0 2024-09-19T07:44:59.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc17fff640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc200c3360 0x7fcc2009cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:44:59.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.205+0000 7fcc1f59e640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc200c3fa0 0x7fcc20202c40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fcc180021b0 tx=0x7fcc1800a4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:59.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.209+0000 7fcc177fe640 1 -- 172.21.15.83:0/519173564 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18003b80 con 0x7fcc200c3fa0 2024-09-19T07:44:59.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.209+0000 7fcc25042640 1 -- 172.21.15.83:0/519173564 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc20166b90 con 0x7fcc200c3fa0 2024-09-19T07:44:59.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.209+0000 7fcc25042640 1 -- 172.21.15.83:0/519173564 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc20167150 con 0x7fcc200c3fa0 2024-09-19T07:44:59.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.209+0000 7fcc177fe640 1 -- 172.21.15.83:0/519173564 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc18003d20 con 0x7fcc200c3fa0 2024-09-19T07:44:59.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.209+0000 7fcc177fe640 1 -- 172.21.15.83:0/519173564 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18013b20 con 0x7fcc200c3fa0 2024-09-19T07:44:59.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.209+0000 7fcc25042640 1 -- 172.21.15.83:0/519173564 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc200c4380 con 0x7fcc200c3fa0 2024-09-19T07:44:59.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.209+0000 7fcc177fe640 1 -- 172.21.15.83:0/519173564 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcc180050d0 con 0x7fcc200c3fa0 2024-09-19T07:44:59.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.217+0000 7fcc177fe640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc00067910 0x7fcc00069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:44:59.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.217+0000 7fcc177fe640 1 -- 172.21.15.83:0/519173564 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7fcc1808ec60 con 0x7fcc200c3fa0 2024-09-19T07:44:59.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.217+0000 7fcc1ed9d640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc00067910 0x7fcc00069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:44:59.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.217+0000 7fcc1ed9d640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc00067910 0x7fcc00069dd0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcc100b0000 tx=0x7fcc100a8000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:44:59.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.229+0000 7fcc177fe640 1 -- 172.21.15.83:0/519173564 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcc1805e950 con 0x7fcc200c3fa0 2024-09-19T07:44:59.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.957+0000 7fcc25042640 1 -- 172.21.15.83:0/519173564 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc20071170 con 0x7fcc200c3fa0 2024-09-19T07:44:59.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.965+0000 7fcc177fe640 1 -- 172.21.15.83:0/519173564 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+6546 (secure 0 0 0) 0x7fcc1805e250 con 0x7fcc200c3fa0 2024-09-19T07:44:59.973 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:44:59.974 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":175,"btime":"2024-09-19T07:44:34:305274+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:44:59.974 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 175 2024-09-19T07:44:59.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.977+0000 7fcc25042640 1 -- 172.21.15.83:0/519173564 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc00067910 msgr2=0x7fcc00069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:59.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.977+0000 7fcc25042640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc00067910 0x7fcc00069dd0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcc100b0000 tx=0x7fcc100a8000 comp rx=0 tx=0).stop 2024-09-19T07:44:59.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.977+0000 7fcc25042640 1 -- 172.21.15.83:0/519173564 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc200c3fa0 msgr2=0x7fcc20202c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:44:59.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.977+0000 7fcc25042640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc200c3fa0 0x7fcc20202c40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fcc180021b0 tx=0x7fcc1800a4f0 comp rx=0 tx=0).stop 2024-09-19T07:44:59.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.981+0000 7fcc25042640 1 -- 172.21.15.83:0/519173564 shutdown_connections 2024-09-19T07:44:59.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.981+0000 7fcc25042640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc00067910 0x7fcc00069dd0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:59.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.981+0000 7fcc25042640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc200c3fa0 0x7fcc20202c40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:59.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.981+0000 7fcc25042640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc200c3360 0x7fcc2009cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:59.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.981+0000 7fcc25042640 1 --2- 172.21.15.83:0/519173564 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc200ab3c0 0x7fcc2009d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:44:59.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.981+0000 7fcc25042640 1 -- 172.21.15.83:0/519173564 >> 172.21.15.83:0/519173564 conn(0x7fcc20075390 msgr2=0x7fcc200c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:44:59.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.985+0000 7fcc25042640 1 -- 172.21.15.83:0/519173564 shutdown_connections 2024-09-19T07:44:59.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:44:59.985+0000 7fcc25042640 1 -- 172.21.15.83:0/519173564 wait complete. 2024-09-19T07:45:00.027 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:45:02.028 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:02.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.461+0000 7fd60eca6640 1 Processor -- start 2024-09-19T07:45:02.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.461+0000 7fd60eca6640 1 -- start start 2024-09-19T07:45:02.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.465+0000 7fd60eca6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd600006c80 0x7fd6000d45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:02.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.465+0000 7fd60eca6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd60004ac30 0x7fd6000d4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:02.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.465+0000 7fd60eca6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6000ddfa0 con 0x7fd60000b880 2024-09-19T07:45:02.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.465+0000 7fd60eca6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd6000dee70 con 0x7fd60004ac30 2024-09-19T07:45:02.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.465+0000 7fd60eca6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd6000de260 con 0x7fd600006c80 2024-09-19T07:45:02.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.465+0000 7fd60e4a5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd60004ac30 0x7fd6000d4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:02.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.465+0000 7fd60e4a5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd60004ac30 0x7fd6000d4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59936/0 (socket says 172.21.15.83:59936) 2024-09-19T07:45:02.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.465+0000 7fd60e4a5640 1 -- 172.21.15.83:0/3196119736 learned_addr learned my addr 172.21.15.83:0/3196119736 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:02.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.469+0000 7fd60dca4640 1 --2- 172.21.15.83:0/3196119736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd600006c80 0x7fd6000d45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:02.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.469+0000 7fd60dca4640 1 -- 172.21.15.83:0/3196119736 >> v2:172.21.15.83:3300/0 conn(0x7fd60004ac30 msgr2=0x7fd6000d4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.469+0000 7fd60dca4640 1 --2- 172.21.15.83:0/3196119736 >> v2:172.21.15.83:3300/0 conn(0x7fd60004ac30 0x7fd6000d4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.469+0000 7fd60dca4640 1 -- 172.21.15.83:0/3196119736 >> v1:172.21.15.83:6789/0 conn(0x7fd60000b880 legacy=0x7fd600049480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:45:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.469+0000 7fd60dca4640 1 -- 172.21.15.83:0/3196119736 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6000de540 con 0x7fd600006c80 2024-09-19T07:45:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.469+0000 7fd60dca4640 1 --2- 172.21.15.83:0/3196119736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd600006c80 0x7fd6000d45b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd6040097c0 tx=0x7fd60401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f864afb5d308bd89 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:02.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.473+0000 7fd60cca2640 1 -- 172.21.15.83:0/3196119736 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6040b0070 con 0x7fd600006c80 2024-09-19T07:45:02.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.473+0000 7fd60cca2640 1 -- 172.21.15.83:0/3196119736 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6040ab600 con 0x7fd600006c80 2024-09-19T07:45:02.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.473+0000 7fd60cca2640 1 -- 172.21.15.83:0/3196119736 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6040ab9d0 con 0x7fd600006c80 2024-09-19T07:45:02.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.473+0000 7fd60eca6640 1 -- 172.21.15.83:0/3196119736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd600006c80 msgr2=0x7fd6000d45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:02.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.473+0000 7fd60eca6640 1 --2- 172.21.15.83:0/3196119736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd600006c80 0x7fd6000d45b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd6040097c0 tx=0x7fd60401c450 comp rx=0 tx=0).stop 2024-09-19T07:45:02.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.477+0000 7fd60eca6640 1 -- 172.21.15.83:0/3196119736 shutdown_connections 2024-09-19T07:45:02.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.477+0000 7fd60eca6640 1 --2- 172.21.15.83:0/3196119736 >> v2:172.21.15.83:3300/0 conn(0x7fd60004ac30 0x7fd6000d4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:02.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.477+0000 7fd60eca6640 1 --2- 172.21.15.83:0/3196119736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd600006c80 0x7fd6000d45b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:02.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.477+0000 7fd60eca6640 1 -- 172.21.15.83:0/3196119736 >> 172.21.15.83:0/3196119736 conn(0x7fd600047d20 msgr2=0x7fd600048130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:02.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.481+0000 7fd60eca6640 1 -- 172.21.15.83:0/3196119736 shutdown_connections 2024-09-19T07:45:02.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60eca6640 1 -- 172.21.15.83:0/3196119736 wait complete. 2024-09-19T07:45:02.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60eca6640 1 Processor -- start 2024-09-19T07:45:02.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60eca6640 1 -- start start 2024-09-19T07:45:02.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60eca6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd600006c80 0x7fd6000dc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:02.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60eca6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd60000b880 0x7fd6000d5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:02.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60eca6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd60004ac30 0x7fd6000d58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:02.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60e4a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd60004ac30 0x7fd6000d58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:02.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60d4a3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd60000b880 0x7fd6000d5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:02.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60e4a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd60004ac30 0x7fd6000d58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59938/0 (socket says 172.21.15.83:59938) 2024-09-19T07:45:02.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60d4a3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd60000b880 0x7fd6000d5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:59618/0 (socket says 172.21.15.83:59618) 2024-09-19T07:45:02.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60e4a5640 1 -- 172.21.15.83:0/726131487 learned_addr learned my addr 172.21.15.83:0/726131487 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:02.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60eca6640 1 -- 172.21.15.83:0/726131487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd60004a120 con 0x7fd60004ac30 2024-09-19T07:45:02.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60eca6640 1 -- 172.21.15.83:0/726131487 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd600049fa0 con 0x7fd60000b880 2024-09-19T07:45:02.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60eca6640 1 -- 172.21.15.83:0/726131487 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd600007060 con 0x7fd600006c80 2024-09-19T07:45:02.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60e4a5640 1 -- 172.21.15.83:0/726131487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd600006c80 msgr2=0x7fd6000dc010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:45:02.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60e4a5640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd600006c80 0x7fd6000dc010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:02.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60e4a5640 1 -- 172.21.15.83:0/726131487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd60000b880 msgr2=0x7fd6000d5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:02.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60e4a5640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd60000b880 0x7fd6000d5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:02.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60e4a5640 1 -- 172.21.15.83:0/726131487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd600179f30 con 0x7fd60004ac30 2024-09-19T07:45:02.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60d4a3640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd60000b880 0x7fd6000d5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:45:02.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd60e4a5640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd60004ac30 0x7fd6000d58d0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fd6080657c0 tx=0x7fd608074460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:02.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd5fe7fc640 1 -- 172.21.15.83:0/726131487 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd608067a70 con 0x7fd60004ac30 2024-09-19T07:45:02.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd5fe7fc640 1 -- 172.21.15.83:0/726131487 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd608090070 con 0x7fd60004ac30 2024-09-19T07:45:02.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.485+0000 7fd5fe7fc640 1 -- 172.21.15.83:0/726131487 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd608067c50 con 0x7fd60004ac30 2024-09-19T07:45:02.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.489+0000 7fd60eca6640 1 -- 172.21.15.83:0/726131487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60017a210 con 0x7fd60004ac30 2024-09-19T07:45:02.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.489+0000 7fd5fe7fc640 1 -- 172.21.15.83:0/726131487 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd608073710 con 0x7fd60004ac30 2024-09-19T07:45:02.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.489+0000 7fd5fe7fc640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5ec067910 0x7fd5ec069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:02.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.489+0000 7fd60eca6640 1 -- 172.21.15.83:0/726131487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd60017a7d0 con 0x7fd60004ac30 2024-09-19T07:45:02.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.489+0000 7fd5fe7fc640 1 -- 172.21.15.83:0/726131487 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7fd608067df0 con 0x7fd60004ac30 2024-09-19T07:45:02.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.493+0000 7fd60dca4640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5ec067910 0x7fd5ec069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:02.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.493+0000 7fd5ebfff640 1 -- 172.21.15.83:0/726131487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6000d8050 con 0x7fd60004ac30 2024-09-19T07:45:02.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.501+0000 7fd5fe7fc640 1 -- 172.21.15.83:0/726131487 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd6080d0ff0 con 0x7fd60004ac30 2024-09-19T07:45:02.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:02.505+0000 7fd60dca4640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5ec067910 0x7fd5ec069dd0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd6040ad000 tx=0x7fd604008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:03.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.449+0000 7fd5ebfff640 1 -- 172.21.15.83:0/726131487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd600049480 con 0x7fd60004ac30 2024-09-19T07:45:03.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.449+0000 7fd5fe7fc640 1 -- 172.21.15.83:0/726131487 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+6546 (secure 0 0 0) 0x7fd6080d52f0 con 0x7fd60004ac30 2024-09-19T07:45:03.452 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:03.452 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":175,"btime":"2024-09-19T07:44:34:305274+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:45:03.452 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 175 2024-09-19T07:45:03.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.453+0000 7fd5ebfff640 1 -- 172.21.15.83:0/726131487 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5ec067910 msgr2=0x7fd5ec069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:03.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.453+0000 7fd5ebfff640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5ec067910 0x7fd5ec069dd0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd6040ad000 tx=0x7fd604008040 comp rx=0 tx=0).stop 2024-09-19T07:45:03.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.453+0000 7fd5ebfff640 1 -- 172.21.15.83:0/726131487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd60004ac30 msgr2=0x7fd6000d58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:03.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.453+0000 7fd5ebfff640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd60004ac30 0x7fd6000d58d0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fd6080657c0 tx=0x7fd608074460 comp rx=0 tx=0).stop 2024-09-19T07:45:03.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.465+0000 7fd5ebfff640 1 -- 172.21.15.83:0/726131487 shutdown_connections 2024-09-19T07:45:03.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.469+0000 7fd5ebfff640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd5ec067910 0x7fd5ec069dd0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:03.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.469+0000 7fd5ebfff640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd60004ac30 0x7fd6000d58d0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:03.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.469+0000 7fd5ebfff640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd60000b880 0x7fd6000d5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:03.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.469+0000 7fd5ebfff640 1 --2- 172.21.15.83:0/726131487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd600006c80 0x7fd6000dc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:03.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.469+0000 7fd5ebfff640 1 -- 172.21.15.83:0/726131487 >> 172.21.15.83:0/726131487 conn(0x7fd600047d20 msgr2=0x7fd600046d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:03.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.469+0000 7fd5ebfff640 1 -- 172.21.15.83:0/726131487 shutdown_connections 2024-09-19T07:45:03.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:03.477+0000 7fd5ebfff640 1 -- 172.21.15.83:0/726131487 wait complete. 2024-09-19T07:45:03.591 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:45:05.592 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:05.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d6f79640 1 Processor -- start 2024-09-19T07:45:05.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d6f79640 1 -- start start 2024-09-19T07:45:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d6f79640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f22d00c9140 0x7f22d015d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d6f79640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f22d00ac6c0 0x7f22d015dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d6f79640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22d00c7640 con 0x7f22d00731d0 2024-09-19T07:45:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d6f79640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f22d00c8400 con 0x7f22d00c9140 2024-09-19T07:45:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d6f79640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f22d00c7f70 con 0x7f22d00ac6c0 2024-09-19T07:45:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d4d15640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f22d00731d0 0x7f22d0157490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48150/0 (socket says 172.21.15.83:48150) 2024-09-19T07:45:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d5516640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f22d00ac6c0 0x7f22d015dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d5516640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f22d00ac6c0 0x7f22d015dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:59624/0 (socket says 172.21.15.83:59624) 2024-09-19T07:45:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d5516640 1 -- 172.21.15.83:0/3480780106 learned_addr learned my addr 172.21.15.83:0/3480780106 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22cffff640 1 --2- 172.21.15.83:0/3480780106 >> v2:172.21.15.83:3300/0 conn(0x7f22d00c9140 0x7f22d015d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:05.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d5516640 1 -- 172.21.15.83:0/3480780106 >> v2:172.21.15.83:3300/0 conn(0x7f22d00c9140 msgr2=0x7f22d015d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:05.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d5516640 1 --2- 172.21.15.83:0/3480780106 >> v2:172.21.15.83:3300/0 conn(0x7f22d00c9140 0x7f22d015d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:05.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d5516640 1 -- 172.21.15.83:0/3480780106 >> v1:172.21.15.83:6789/0 conn(0x7f22d00731d0 legacy=0x7f22d0157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:05.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.869+0000 7f22d5516640 1 -- 172.21.15.83:0/3480780106 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d00c7920 con 0x7f22d00ac6c0 2024-09-19T07:45:05.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.873+0000 7f22d5516640 1 --2- 172.21.15.83:0/3480780106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d00ac6c0 0x7f22d015dc20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f22d0072d30 tx=0x7f22c801c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=225553939671154 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:05.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.873+0000 7f22cf7fe640 1 -- 172.21.15.83:0/3480780106 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c80a5590 con 0x7f22d00ac6c0 2024-09-19T07:45:05.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.873+0000 7f22cf7fe640 1 -- 172.21.15.83:0/3480780106 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22c80a5730 con 0x7f22d00ac6c0 2024-09-19T07:45:05.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.873+0000 7f22cf7fe640 1 -- 172.21.15.83:0/3480780106 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c80a5b00 con 0x7f22d00ac6c0 2024-09-19T07:45:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.873+0000 7f22d6f79640 1 -- 172.21.15.83:0/3480780106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d00ac6c0 msgr2=0x7f22d015dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.873+0000 7f22d6f79640 1 --2- 172.21.15.83:0/3480780106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d00ac6c0 0x7f22d015dc20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f22d0072d30 tx=0x7f22c801c8d0 comp rx=0 tx=0).stop 2024-09-19T07:45:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.873+0000 7f22d6f79640 1 -- 172.21.15.83:0/3480780106 shutdown_connections 2024-09-19T07:45:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.873+0000 7f22d6f79640 1 --2- 172.21.15.83:0/3480780106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d00ac6c0 0x7f22d015dc20 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.873+0000 7f22d6f79640 1 --2- 172.21.15.83:0/3480780106 >> v2:172.21.15.83:3300/0 conn(0x7f22d00c9140 0x7f22d015d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.873+0000 7f22d6f79640 1 -- 172.21.15.83:0/3480780106 >> 172.21.15.83:0/3480780106 conn(0x7f22d00c9610 msgr2=0x7f22d00c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d6f79640 1 -- 172.21.15.83:0/3480780106 shutdown_connections 2024-09-19T07:45:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d6f79640 1 -- 172.21.15.83:0/3480780106 wait complete. 2024-09-19T07:45:05.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d6f79640 1 Processor -- start 2024-09-19T07:45:05.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d6f79640 1 -- start start 2024-09-19T07:45:05.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d6f79640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d00731d0 0x7f22d0154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:05.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d6f79640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22d00ac6c0 0x7f22d01f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:05.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d6f79640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22d00c9140 0x7f22d01f34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:05.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d6f79640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f22d0157d20 con 0x7f22d00ac6c0 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d6f79640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f22d0157ba0 con 0x7f22d00731d0 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d6f79640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f22d00735b0 con 0x7f22d00c9140 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22cffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22d00ac6c0 0x7f22d01f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22cffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22d00ac6c0 0x7f22d01f2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59958/0 (socket says 172.21.15.83:59958) 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22cffff640 1 -- 172.21.15.83:0/2566145586 learned_addr learned my addr 172.21.15.83:0/2566145586 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d5516640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22d00c9140 0x7f22d01f34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d4d15640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d00731d0 0x7f22d0154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d4d15640 1 -- 172.21.15.83:0/2566145586 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22d00c9140 msgr2=0x7f22d01f34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d4d15640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22d00c9140 0x7f22d01f34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d4d15640 1 -- 172.21.15.83:0/2566145586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22d00ac6c0 msgr2=0x7f22d01f2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d4d15640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22d00ac6c0 0x7f22d01f2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d4d15640 1 -- 172.21.15.83:0/2566145586 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d0157040 con 0x7f22d00731d0 2024-09-19T07:45:05.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d5516640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22d00c9140 0x7f22d01f34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:45:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d4d15640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d00731d0 0x7f22d0154c60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f22c000c390 tx=0x7f22c000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22cd7fa640 1 -- 172.21.15.83:0/2566145586 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c00074e0 con 0x7f22d00731d0 2024-09-19T07:45:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22cd7fa640 1 -- 172.21.15.83:0/2566145586 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22c0007680 con 0x7f22d00731d0 2024-09-19T07:45:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d6f79640 1 -- 172.21.15.83:0/2566145586 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d01f2ad0 con 0x7f22d00731d0 2024-09-19T07:45:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22cd7fa640 1 -- 172.21.15.83:0/2566145586 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c00049f0 con 0x7f22d00731d0 2024-09-19T07:45:05.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.877+0000 7f22d6f79640 1 -- 172.21.15.83:0/2566145586 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d0156d60 con 0x7f22d00731d0 2024-09-19T07:45:05.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.881+0000 7f22d6f79640 1 -- 172.21.15.83:0/2566145586 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b00052a0 con 0x7f22d00731d0 2024-09-19T07:45:05.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.889+0000 7f22cd7fa640 1 -- 172.21.15.83:0/2566145586 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f22c001a020 con 0x7f22d00731d0 2024-09-19T07:45:05.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.889+0000 7f22cd7fa640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f22bc0678c0 0x7f22bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:05.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.889+0000 7f22cffff640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f22bc0678c0 0x7f22bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:05.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.889+0000 7f22cd7fa640 1 -- 172.21.15.83:0/2566145586 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7f22c00868f0 con 0x7f22d00731d0 2024-09-19T07:45:05.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.889+0000 7f22cffff640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f22bc0678c0 0x7f22bc069d80 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f22d00c6f60 tx=0x7f22c4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:05.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:05.897+0000 7f22cd7fa640 1 -- 172.21.15.83:0/2566145586 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f22c0021070 con 0x7f22d00731d0 2024-09-19T07:45:06.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.349+0000 7f22d6f79640 1 -- 172.21.15.83:0/2566145586 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22b0005590 con 0x7f22d00731d0 2024-09-19T07:45:06.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.353+0000 7f22cd7fa640 1 -- 172.21.15.83:0/2566145586 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+6546 (secure 0 0 0) 0x7f22c0055f30 con 0x7f22d00731d0 2024-09-19T07:45:06.361 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:06.363 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":175,"btime":"2024-09-19T07:44:34:305274+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:45:06.363 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 175 2024-09-19T07:45:06.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.373+0000 7f22d6f79640 1 -- 172.21.15.83:0/2566145586 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f22bc0678c0 msgr2=0x7f22bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:06.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.373+0000 7f22d6f79640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f22bc0678c0 0x7f22bc069d80 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f22d00c6f60 tx=0x7f22c4009190 comp rx=0 tx=0).stop 2024-09-19T07:45:06.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.373+0000 7f22d6f79640 1 -- 172.21.15.83:0/2566145586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d00731d0 msgr2=0x7f22d0154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:06.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.373+0000 7f22d6f79640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d00731d0 0x7f22d0154c60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f22c000c390 tx=0x7f22c000c850 comp rx=0 tx=0).stop 2024-09-19T07:45:06.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.373+0000 7f22d6f79640 1 -- 172.21.15.83:0/2566145586 shutdown_connections 2024-09-19T07:45:06.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.373+0000 7f22d6f79640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f22bc0678c0 0x7f22bc069d80 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:06.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.373+0000 7f22d6f79640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22d00c9140 0x7f22d01f34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.373+0000 7f22d6f79640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22d00ac6c0 0x7f22d01f2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.373+0000 7f22d6f79640 1 --2- 172.21.15.83:0/2566145586 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22d00731d0 0x7f22d0154c60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.373+0000 7f22d6f79640 1 -- 172.21.15.83:0/2566145586 >> 172.21.15.83:0/2566145586 conn(0x7f22d00c9610 msgr2=0x7f22d015b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.377+0000 7f22d6f79640 1 -- 172.21.15.83:0/2566145586 shutdown_connections 2024-09-19T07:45:06.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:06.377+0000 7f22d6f79640 1 -- 172.21.15.83:0/2566145586 wait complete. 2024-09-19T07:45:06.418 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:45:08.419 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:08.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.673+0000 7f010e0c7640 1 Processor -- start 2024-09-19T07:45:08.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.673+0000 7f010e0c7640 1 -- start start 2024-09-19T07:45:08.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.677+0000 7f010e0c7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108152f30 0x7f0108153370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:08.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.677+0000 7f010e0c7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0108155e10 0x7f0108156240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:08.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.677+0000 7f010e0c7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0108159430 con 0x7f01080c4880 2024-09-19T07:45:08.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.677+0000 7f010e0c7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f010815a040 con 0x7f0108155e10 2024-09-19T07:45:08.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.677+0000 7f010e0c7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f01080c2c10 con 0x7f0108152f30 2024-09-19T07:45:08.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.677+0000 7f0106ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108152f30 0x7f0108153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:08.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.677+0000 7f0106ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108152f30 0x7f0108153370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:42292/0 (socket says 172.21.15.83:42292) 2024-09-19T07:45:08.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.677+0000 7f0106ffd640 1 -- 172.21.15.83:0/845907972 learned_addr learned my addr 172.21.15.83:0/845907972 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:08.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.677+0000 7f0106ffd640 1 -- 172.21.15.83:0/845907972 >> v2:172.21.15.83:3300/0 conn(0x7f0108155e10 msgr2=0x7f0108156240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:45:08.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.677+0000 7f0106ffd640 1 --2- 172.21.15.83:0/845907972 >> v2:172.21.15.83:3300/0 conn(0x7f0108155e10 0x7f0108156240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:08.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.677+0000 7f0106ffd640 1 -- 172.21.15.83:0/845907972 >> v1:172.21.15.184:6789/0 conn(0x7f01080c4880 legacy=0x7f01080c1df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:45:08.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.677+0000 7f0106ffd640 1 -- 172.21.15.83:0/845907972 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0108159710 con 0x7f0108152f30 2024-09-19T07:45:08.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.681+0000 7f0106ffd640 1 --2- 172.21.15.83:0/845907972 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108152f30 0x7f0108153370 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f00fc009870 tx=0x7f00fc01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6fc3b9cf76e61dd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:08.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.681+0000 7f01067fc640 1 -- 172.21.15.83:0/845907972 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc0b0070 con 0x7f0108152f30 2024-09-19T07:45:08.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.681+0000 7f01067fc640 1 -- 172.21.15.83:0/845907972 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00fc0ab600 con 0x7f0108152f30 2024-09-19T07:45:08.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.681+0000 7f01067fc640 1 -- 172.21.15.83:0/845907972 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc0ab9d0 con 0x7f0108152f30 2024-09-19T07:45:08.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.681+0000 7f010e0c7640 1 -- 172.21.15.83:0/845907972 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108152f30 msgr2=0x7f0108153370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:08.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.681+0000 7f010e0c7640 1 --2- 172.21.15.83:0/845907972 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108152f30 0x7f0108153370 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f00fc009870 tx=0x7f00fc01c3f0 comp rx=0 tx=0).stop 2024-09-19T07:45:08.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.689+0000 7f010e0c7640 1 -- 172.21.15.83:0/845907972 shutdown_connections 2024-09-19T07:45:08.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.689+0000 7f010e0c7640 1 --2- 172.21.15.83:0/845907972 >> v2:172.21.15.83:3300/0 conn(0x7f0108155e10 0x7f0108156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:08.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.689+0000 7f010e0c7640 1 --2- 172.21.15.83:0/845907972 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108152f30 0x7f0108153370 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:08.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.689+0000 7f010e0c7640 1 -- 172.21.15.83:0/845907972 >> 172.21.15.83:0/845907972 conn(0x7f01080aa910 msgr2=0x7f01080c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:08.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.693+0000 7f010e0c7640 1 -- 172.21.15.83:0/845907972 shutdown_connections 2024-09-19T07:45:08.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.693+0000 7f010e0c7640 1 -- 172.21.15.83:0/845907972 wait complete. 2024-09-19T07:45:08.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f010e0c7640 1 Processor -- start 2024-09-19T07:45:08.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f010e0c7640 1 -- start start 2024-09-19T07:45:08.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f010e0c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01080c4880 0x7f01081588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f010e0c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0108152f30 0x7f0108071e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f010e0c7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108155e10 0x7f0108072340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f010e0c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f01080c2680 con 0x7f01080c4880 2024-09-19T07:45:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f010e0c7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01080c2500 con 0x7f0108152f30 2024-09-19T07:45:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f010e0c7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f01080c4c60 con 0x7f0108155e10 2024-09-19T07:45:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f0107fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108155e10 0x7f0108072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f0107fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108155e10 0x7f0108072340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:42302/0 (socket says 172.21.15.83:42302) 2024-09-19T07:45:08.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f0107fff640 1 -- 172.21.15.83:0/2473756498 learned_addr learned my addr 172.21.15.83:0/2473756498 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:08.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f0107fff640 1 -- 172.21.15.83:0/2473756498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0108152f30 msgr2=0x7f0108071e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:45:08.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f0107fff640 1 --2- 172.21.15.83:0/2473756498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0108152f30 0x7f0108071e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:08.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f0107fff640 1 -- 172.21.15.83:0/2473756498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01080c4880 msgr2=0x7f01081588e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:45:08.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f0107fff640 1 --2- 172.21.15.83:0/2473756498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01080c4880 0x7f01081588e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:08.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f0107fff640 1 -- 172.21.15.83:0/2473756498 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01080731e0 con 0x7f0108155e10 2024-09-19T07:45:08.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f0107fff640 1 --2- 172.21.15.83:0/2473756498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108155e10 0x7f0108072340 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f010000b990 tx=0x7f010000be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:08.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f00effff640 1 -- 172.21.15.83:0/2473756498 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0100019070 con 0x7f0108155e10 2024-09-19T07:45:08.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f00effff640 1 -- 172.21.15.83:0/2473756498 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f010000cba0 con 0x7f0108155e10 2024-09-19T07:45:08.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.701+0000 7f00effff640 1 -- 172.21.15.83:0/2473756498 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0100014440 con 0x7f0108155e10 2024-09-19T07:45:08.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.705+0000 7f010e0c7640 1 -- 172.21.15.83:0/2473756498 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01080734c0 con 0x7f0108155e10 2024-09-19T07:45:08.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.705+0000 7f010e0c7640 1 -- 172.21.15.83:0/2473756498 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0108158e20 con 0x7f0108155e10 2024-09-19T07:45:08.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.705+0000 7f010e0c7640 1 -- 172.21.15.83:0/2473756498 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00e80052a0 con 0x7f0108155e10 2024-09-19T07:45:08.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.705+0000 7f00effff640 1 -- 172.21.15.83:0/2473756498 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f010001c070 con 0x7f0108155e10 2024-09-19T07:45:08.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.709+0000 7f00effff640 1 --2- 172.21.15.83:0/2473756498 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00f40678c0 0x7f00f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:08.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.709+0000 7f00effff640 1 -- 172.21.15.83:0/2473756498 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7f0100086910 con 0x7f0108155e10 2024-09-19T07:45:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.709+0000 7f01077fe640 1 --2- 172.21.15.83:0/2473756498 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00f40678c0 0x7f00f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:08.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.709+0000 7f01077fe640 1 --2- 172.21.15.83:0/2473756498 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00f40678c0 0x7f00f4069d80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f00f8009690 tx=0x7f00f8006be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:08.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:08.721+0000 7f00effff640 1 -- 172.21.15.83:0/2473756498 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f01000564f0 con 0x7f0108155e10 2024-09-19T07:45:09.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.197+0000 7f010e0c7640 1 -- 172.21.15.83:0/2473756498 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00e8005590 con 0x7f0108155e10 2024-09-19T07:45:09.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.201+0000 7f00effff640 1 -- 172.21.15.83:0/2473756498 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+6546 (secure 0 0 0) 0x7f0100055df0 con 0x7f0108155e10 2024-09-19T07:45:09.202 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:09.202 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":175,"btime":"2024-09-19T07:44:34:305274+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:45:09.202 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 175 2024-09-19T07:45:09.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.205+0000 7f010e0c7640 1 -- 172.21.15.83:0/2473756498 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00f40678c0 msgr2=0x7f00f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:09.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.205+0000 7f010e0c7640 1 --2- 172.21.15.83:0/2473756498 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00f40678c0 0x7f00f4069d80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f00f8009690 tx=0x7f00f8006be0 comp rx=0 tx=0).stop 2024-09-19T07:45:09.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.205+0000 7f010e0c7640 1 -- 172.21.15.83:0/2473756498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108155e10 msgr2=0x7f0108072340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:09.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.205+0000 7f010e0c7640 1 --2- 172.21.15.83:0/2473756498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108155e10 0x7f0108072340 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f010000b990 tx=0x7f010000be50 comp rx=0 tx=0).stop 2024-09-19T07:45:09.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.205+0000 7f010e0c7640 1 -- 172.21.15.83:0/2473756498 shutdown_connections 2024-09-19T07:45:09.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.205+0000 7f010e0c7640 1 --2- 172.21.15.83:0/2473756498 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f00f40678c0 0x7f00f4069d80 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:09.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.205+0000 7f010e0c7640 1 --2- 172.21.15.83:0/2473756498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0108155e10 0x7f0108072340 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:09.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.205+0000 7f010e0c7640 1 --2- 172.21.15.83:0/2473756498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0108152f30 0x7f0108071e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:09.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.205+0000 7f010e0c7640 1 --2- 172.21.15.83:0/2473756498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01080c4880 0x7f01081588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:09.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.205+0000 7f010e0c7640 1 -- 172.21.15.83:0/2473756498 >> 172.21.15.83:0/2473756498 conn(0x7f01080aa910 msgr2=0x7f010815fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:09.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.205+0000 7f010e0c7640 1 -- 172.21.15.83:0/2473756498 shutdown_connections 2024-09-19T07:45:09.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:09.205+0000 7f010e0c7640 1 -- 172.21.15.83:0/2473756498 wait complete. 2024-09-19T07:45:09.251 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:45:10.416 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:10.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.541+0000 7f74857ef640 1 Processor -- start 2024-09-19T07:45:10.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.541+0000 7f74857ef640 1 -- start start 2024-09-19T07:45:10.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f74857ef640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f7480071100 0x7f74801545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:10.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f74857ef640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74800c28e0 0x7f7480150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:10.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f74857ef640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74800a9e60 con 0x7f7480151480 2024-09-19T07:45:10.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f74857ef640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7480071b50 con 0x7f74800c28e0 2024-09-19T07:45:10.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f74857ef640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f74800c1ff0 con 0x7f7480071100 2024-09-19T07:45:10.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747e7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74800c28e0 0x7f7480150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:10.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747e7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74800c28e0 0x7f7480150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46252/0 (socket says 172.21.15.83:46252) 2024-09-19T07:45:10.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747e7fc640 1 -- 172.21.15.83:0/1284434262 learned_addr learned my addr 172.21.15.83:0/1284434262 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:10.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747f7fe640 1 --1- 172.21.15.83:0/1284434262 >> v1:172.21.15.83:6789/0 conn(0x7f7480151480 0x7f748006dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59744/0 (socket says 172.21.15.83:59744) 2024-09-19T07:45:10.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747effd640 1 --2- 172.21.15.83:0/1284434262 >> v2:172.21.15.184:3300/0 conn(0x7f7480071100 0x7f74801545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:10.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747dffb640 1 -- 172.21.15.83:0/1284434262 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2885621373 0 0) 0x7f74800a9e60 con 0x7f7480151480 2024-09-19T07:45:10.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747dffb640 1 -- 172.21.15.83:0/1284434262 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f746c003530 con 0x7f7480151480 2024-09-19T07:45:10.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747e7fc640 1 -- 172.21.15.83:0/1284434262 >> v2:172.21.15.184:3300/0 conn(0x7f7480071100 msgr2=0x7f74801545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:10.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747e7fc640 1 --2- 172.21.15.83:0/1284434262 >> v2:172.21.15.184:3300/0 conn(0x7f7480071100 0x7f74801545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747e7fc640 1 -- 172.21.15.83:0/1284434262 >> v1:172.21.15.83:6789/0 conn(0x7f7480151480 legacy=0x7f748006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:10.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747e7fc640 1 -- 172.21.15.83:0/1284434262 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74800aa140 con 0x7f74800c28e0 2024-09-19T07:45:10.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747effd640 1 --2- 172.21.15.83:0/1284434262 >> v2:172.21.15.184:3300/0 conn(0x7f7480071100 0x7f74801545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:45:10.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747e7fc640 1 --2- 172.21.15.83:0/1284434262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74800c28e0 0x7f7480150f40 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f748006f620 tx=0x7f746801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5017d78b5d8cdb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:10.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747dffb640 1 -- 172.21.15.83:0/1284434262 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74680b0070 con 0x7f74800c28e0 2024-09-19T07:45:10.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747dffb640 1 -- 172.21.15.83:0/1284434262 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74680ab600 con 0x7f74800c28e0 2024-09-19T07:45:10.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f747dffb640 1 -- 172.21.15.83:0/1284434262 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74680ab9d0 con 0x7f74800c28e0 2024-09-19T07:45:10.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f74857ef640 1 -- 172.21.15.83:0/1284434262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74800c28e0 msgr2=0x7f7480150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:10.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f74857ef640 1 --2- 172.21.15.83:0/1284434262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74800c28e0 0x7f7480150f40 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f748006f620 tx=0x7f746801c490 comp rx=0 tx=0).stop 2024-09-19T07:45:10.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f74857ef640 1 -- 172.21.15.83:0/1284434262 shutdown_connections 2024-09-19T07:45:10.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f74857ef640 1 --2- 172.21.15.83:0/1284434262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74800c28e0 0x7f7480150f40 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.545+0000 7f74857ef640 1 --2- 172.21.15.83:0/1284434262 >> v2:172.21.15.184:3300/0 conn(0x7f7480071100 0x7f74801545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 -- 172.21.15.83:0/1284434262 >> 172.21.15.83:0/1284434262 conn(0x7f748006bc80 msgr2=0x7f748006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:10.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 -- 172.21.15.83:0/1284434262 shutdown_connections 2024-09-19T07:45:10.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 -- 172.21.15.83:0/1284434262 wait complete. 2024-09-19T07:45:10.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 Processor -- start 2024-09-19T07:45:10.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 -- start start 2024-09-19T07:45:10.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7480071100 0x7f748015c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:10.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74800c28e0 0x7f7480152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:10.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7480151480 0x7f7480149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:10.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7480071660 con 0x7f74800c28e0 2024-09-19T07:45:10.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f74800714e0 con 0x7f7480071100 2024-09-19T07:45:10.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7480160c60 con 0x7f7480151480 2024-09-19T07:45:10.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f747e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74800c28e0 0x7f7480152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:10.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f747e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74800c28e0 0x7f7480152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46264/0 (socket says 172.21.15.83:46264) 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f747e7fc640 1 -- 172.21.15.83:0/3910130722 learned_addr learned my addr 172.21.15.83:0/3910130722 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f747effd640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7480071100 0x7f748015c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f747e7fc640 1 -- 172.21.15.83:0/3910130722 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7480151480 msgr2=0x7f7480149db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f747e7fc640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7480151480 0x7f7480149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f747e7fc640 1 -- 172.21.15.83:0/3910130722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7480071100 msgr2=0x7f748015c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f747e7fc640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7480071100 0x7f748015c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f747e7fc640 1 -- 172.21.15.83:0/3910130722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7480148250 con 0x7f74800c28e0 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f747e7fc640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74800c28e0 0x7f7480152210 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f74680ae040 tx=0x7f7468019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74777fe640 1 -- 172.21.15.83:0/3910130722 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74680b0040 con 0x7f74800c28e0 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74777fe640 1 -- 172.21.15.83:0/3910130722 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74680b9070 con 0x7f74800c28e0 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 -- 172.21.15.83:0/3910130722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7480153250 con 0x7f74800c28e0 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74777fe640 1 -- 172.21.15.83:0/3910130722 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74680b4680 con 0x7f74800c28e0 2024-09-19T07:45:10.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 -- 172.21.15.83:0/3910130722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f748015cdb0 con 0x7f74800c28e0 2024-09-19T07:45:10.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.549+0000 7f74857ef640 1 -- 172.21.15.83:0/3910130722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74800c4f30 con 0x7f74800c28e0 2024-09-19T07:45:10.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.553+0000 7f74777fe640 1 -- 172.21.15.83:0/3910130722 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f74680b4820 con 0x7f74800c28e0 2024-09-19T07:45:10.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.553+0000 7f74777fe640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f74600678f0 0x7f7460069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:10.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.553+0000 7f74777fe640 1 -- 172.21.15.83:0/3910130722 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7f7468121240 con 0x7f74800c28e0 2024-09-19T07:45:10.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.553+0000 7f747effd640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f74600678f0 0x7f7460069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:10.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.553+0000 7f747effd640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f74600678f0 0x7f7460069db0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7470006fd0 tx=0x7f7470008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:10.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.557+0000 7f74777fe640 1 -- 172.21.15.83:0/3910130722 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7468016110 con 0x7f74800c28e0 2024-09-19T07:45:10.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.737+0000 7f74857ef640 1 -- 172.21.15.83:0/3910130722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7480153020 con 0x7f74800c28e0 2024-09-19T07:45:10.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.737+0000 7f74777fe640 1 -- 172.21.15.83:0/3910130722 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+6546 (secure 0 0 0) 0x7f74680f0f30 con 0x7f74800c28e0 2024-09-19T07:45:10.739 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:10.739 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":175,"btime":"2024-09-19T07:44:34:305274+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:45:10.739 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 175 2024-09-19T07:45:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.741+0000 7f74857ef640 1 -- 172.21.15.83:0/3910130722 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f74600678f0 msgr2=0x7f7460069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.741+0000 7f74857ef640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f74600678f0 0x7f7460069db0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7470006fd0 tx=0x7f7470008040 comp rx=0 tx=0).stop 2024-09-19T07:45:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.741+0000 7f74857ef640 1 -- 172.21.15.83:0/3910130722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74800c28e0 msgr2=0x7f7480152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.741+0000 7f74857ef640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74800c28e0 0x7f7480152210 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f74680ae040 tx=0x7f7468019230 comp rx=0 tx=0).stop 2024-09-19T07:45:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.741+0000 7f74857ef640 1 -- 172.21.15.83:0/3910130722 shutdown_connections 2024-09-19T07:45:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.741+0000 7f74857ef640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f74600678f0 0x7f7460069db0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.741+0000 7f74857ef640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7480151480 0x7f7480149db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.741+0000 7f74857ef640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74800c28e0 0x7f7480152210 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.741+0000 7f74857ef640 1 --2- 172.21.15.83:0/3910130722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7480071100 0x7f748015c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.741+0000 7f74857ef640 1 -- 172.21.15.83:0/3910130722 >> 172.21.15.83:0/3910130722 conn(0x7f748006bc80 msgr2=0x7f748006dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:10.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.741+0000 7f74857ef640 1 -- 172.21.15.83:0/3910130722 shutdown_connections 2024-09-19T07:45:10.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.741+0000 7f74857ef640 1 -- 172.21.15.83:0/3910130722 wait complete. 2024-09-19T07:45:10.756 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-19T07:45:10.756 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:45:10.756 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:45:10.767 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:45:10.767 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:10.809 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:45:10.814 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:45:10.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.913+0000 7fbc8507d640 1 Processor -- start 2024-09-19T07:45:10.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc8507d640 1 -- start start 2024-09-19T07:45:10.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc8507d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbc8014dcc0 0x7fbc801570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:10.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc8507d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 0x7fbc800c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:10.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc8507d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc800c57d0 con 0x7fbc8006f2f0 2024-09-19T07:45:10.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc8507d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbc80154b20 con 0x7fbc8014dcc0 2024-09-19T07:45:10.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc8507d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbc80153f80 con 0x7fbc8006e460 2024-09-19T07:45:10.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc7ed9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbc8014dcc0 0x7fbc801570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:10.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc7f59e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 0x7fbc800c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:10.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc7ed9d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbc8014dcc0 0x7fbc801570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:41186/0 (socket says 172.21.15.83:41186) 2024-09-19T07:45:10.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc7f59e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 0x7fbc800c60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:34160/0 (socket says 172.21.15.83:34160) 2024-09-19T07:45:10.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc77fff640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fbc8006f2f0 0x7fbc80154410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:35426/0 (socket says 172.21.15.83:35426) 2024-09-19T07:45:10.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc7ed9d640 1 -- 172.21.15.83:0/1316768846 learned_addr learned my addr 172.21.15.83:0/1316768846 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:10.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc7f59e640 1 -- 172.21.15.83:0/1316768846 >> v2:172.21.15.184:3300/0 conn(0x7fbc8014dcc0 msgr2=0x7fbc801570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:10.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc7f59e640 1 --2- 172.21.15.83:0/1316768846 >> v2:172.21.15.184:3300/0 conn(0x7fbc8014dcc0 0x7fbc801570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc7f59e640 1 -- 172.21.15.83:0/1316768846 >> v1:172.21.15.184:6789/0 conn(0x7fbc8006f2f0 legacy=0x7fbc80154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:10.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc7f59e640 1 -- 172.21.15.83:0/1316768846 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc800c5ab0 con 0x7fbc8006e460 2024-09-19T07:45:10.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.917+0000 7fbc7ed9d640 1 --2- 172.21.15.83:0/1316768846 >> v2:172.21.15.184:3300/0 conn(0x7fbc8014dcc0 0x7fbc801570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:45:10.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7f59e640 1 --2- 172.21.15.83:0/1316768846 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 0x7fbc800c60a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800f370 tx=0x7fbc78020d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=140d827eb5af0bef server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:10.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7e59c640 1 -- 172.21.15.83:0/1316768846 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc780a9c60 con 0x7fbc8006e460 2024-09-19T07:45:10.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7e59c640 1 -- 172.21.15.83:0/1316768846 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc780a9e00 con 0x7fbc8006e460 2024-09-19T07:45:10.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7e59c640 1 -- 172.21.15.83:0/1316768846 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7800d2f0 con 0x7fbc8006e460 2024-09-19T07:45:10.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 -- 172.21.15.83:0/1316768846 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 msgr2=0x7fbc800c60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:10.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 --2- 172.21.15.83:0/1316768846 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 0x7fbc800c60a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800f370 tx=0x7fbc78020d70 comp rx=0 tx=0).stop 2024-09-19T07:45:10.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 -- 172.21.15.83:0/1316768846 shutdown_connections 2024-09-19T07:45:10.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 --2- 172.21.15.83:0/1316768846 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 0x7fbc800c60a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 --2- 172.21.15.83:0/1316768846 >> v2:172.21.15.184:3300/0 conn(0x7fbc8014dcc0 0x7fbc801570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 -- 172.21.15.83:0/1316768846 >> 172.21.15.83:0/1316768846 conn(0x7fbc8006ffc0 msgr2=0x7fbc800703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:10.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 -- 172.21.15.83:0/1316768846 shutdown_connections 2024-09-19T07:45:10.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 -- 172.21.15.83:0/1316768846 wait complete. 2024-09-19T07:45:10.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 Processor -- start 2024-09-19T07:45:10.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 -- start start 2024-09-19T07:45:10.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 0x7fbc8014cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:10.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc8006f2f0 0x7fbc8014d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:10.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc8014dcc0 0x7fbc800bf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:10.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8015ee10 con 0x7fbc8014dcc0 2024-09-19T07:45:10.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8015ec90 con 0x7fbc8006f2f0 2024-09-19T07:45:10.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbc8015ef90 con 0x7fbc8006e460 2024-09-19T07:45:10.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc8014dcc0 0x7fbc800bf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:10.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc8014dcc0 0x7fbc800bf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46278/0 (socket says 172.21.15.83:46278) 2024-09-19T07:45:10.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7f59e640 1 -- 172.21.15.83:0/1346475364 learned_addr learned my addr 172.21.15.83:0/1346475364 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc77fff640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc8006f2f0 0x7fbc8014d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7ed9d640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 0x7fbc8014cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7f59e640 1 -- 172.21.15.83:0/1346475364 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 msgr2=0x7fbc8014cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7f59e640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 0x7fbc8014cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7f59e640 1 -- 172.21.15.83:0/1346475364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc8006f2f0 msgr2=0x7fbc8014d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7f59e640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc8006f2f0 0x7fbc8014d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7f59e640 1 -- 172.21.15.83:0/1346475364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc800c0a70 con 0x7fbc8014dcc0 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7ed9d640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 0x7fbc8014cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc7f59e640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc8014dcc0 0x7fbc800bf0d0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fbc7801f420 tx=0x7fbc7801db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc777fe640 1 -- 172.21.15.83:0/1346475364 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7800b950 con 0x7fbc8014dcc0 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc777fe640 1 -- 172.21.15.83:0/1346475364 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc7800baf0 con 0x7fbc8014dcc0 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc777fe640 1 -- 172.21.15.83:0/1346475364 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc780bc440 con 0x7fbc8014dcc0 2024-09-19T07:45:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 -- 172.21.15.83:0/1346475364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc800be6d0 con 0x7fbc8014dcc0 2024-09-19T07:45:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.921+0000 7fbc8507d640 1 -- 172.21.15.83:0/1346475364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc800be970 con 0x7fbc8014dcc0 2024-09-19T07:45:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.925+0000 7fbc8507d640 1 -- 172.21.15.83:0/1346475364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc800c93f0 con 0x7fbc8014dcc0 2024-09-19T07:45:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.925+0000 7fbc777fe640 1 -- 172.21.15.83:0/1346475364 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbc780c5020 con 0x7fbc8014dcc0 2024-09-19T07:45:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.925+0000 7fbc777fe640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc600678c0 0x7fbc60069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:10.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.925+0000 7fbc777fe640 1 -- 172.21.15.83:0/1346475364 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7fbc781253e0 con 0x7fbc8014dcc0 2024-09-19T07:45:10.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.925+0000 7fbc7ed9d640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc600678c0 0x7fbc60069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:10.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.925+0000 7fbc7ed9d640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc600678c0 0x7fbc60069d80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbc70009a30 tx=0x7fbc70008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:10.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:10.929+0000 7fbc777fe640 1 -- 172.21.15.83:0/1346475364 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbc780cc020 con 0x7fbc8014dcc0 2024-09-19T07:45:11.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.113+0000 7fbc8507d640 1 -- 172.21.15.83:0/1346475364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc8006e900 con 0x7fbc8014dcc0 2024-09-19T07:45:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.113+0000 7fbc777fe640 1 -- 172.21.15.83:0/1346475364 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+6546 (secure 0 0 0) 0x7fbc800c0a70 con 0x7fbc8014dcc0 2024-09-19T07:45:11.113 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:11.113 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":175,"btime":"2024-09-19T07:44:34:305274+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:45:11.113 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 175 2024-09-19T07:45:11.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.113+0000 7fbc8507d640 1 -- 172.21.15.83:0/1346475364 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc600678c0 msgr2=0x7fbc60069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:11.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.113+0000 7fbc8507d640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc600678c0 0x7fbc60069d80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbc70009a30 tx=0x7fbc70008040 comp rx=0 tx=0).stop 2024-09-19T07:45:11.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.113+0000 7fbc8507d640 1 -- 172.21.15.83:0/1346475364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc8014dcc0 msgr2=0x7fbc800bf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:11.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.113+0000 7fbc8507d640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc8014dcc0 0x7fbc800bf0d0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fbc7801f420 tx=0x7fbc7801db10 comp rx=0 tx=0).stop 2024-09-19T07:45:11.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.113+0000 7fbc8507d640 1 -- 172.21.15.83:0/1346475364 shutdown_connections 2024-09-19T07:45:11.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.113+0000 7fbc8507d640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbc600678c0 0x7fbc60069d80 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.113+0000 7fbc8507d640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc8014dcc0 0x7fbc800bf0d0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.113+0000 7fbc8507d640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc8006f2f0 0x7fbc8014d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.113+0000 7fbc8507d640 1 --2- 172.21.15.83:0/1346475364 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc8006e460 0x7fbc8014cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.113+0000 7fbc8507d640 1 -- 172.21.15.83:0/1346475364 >> 172.21.15.83:0/1346475364 conn(0x7fbc8006ffc0 msgr2=0x7fbc800703a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:11.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.117+0000 7fbc8507d640 1 -- 172.21.15.83:0/1346475364 shutdown_connections 2024-09-19T07:45:11.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.117+0000 7fbc8507d640 1 -- 172.21.15.83:0/1346475364 wait complete. 2024-09-19T07:45:11.130 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:45:11.251 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:11.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.393+0000 7f01355dc640 1 Processor -- start 2024-09-19T07:45:11.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.393+0000 7f01355dc640 1 -- start start 2024-09-19T07:45:11.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f01355dc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f01301536b0 0x7f01300c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:11.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f01355dc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01300c9200 con 0x7f01300af350 2024-09-19T07:45:11.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f01355dc640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0130156d20 con 0x7f0130156820 2024-09-19T07:45:11.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f01355dc640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f0130153a90 con 0x7f01301536b0 2024-09-19T07:45:11.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012e7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f01300af350 0x7f01300c7ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59760/0 (socket says 172.21.15.83:59760) 2024-09-19T07:45:11.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012e7fc640 1 -- 172.21.15.83:0/3074449572 learned_addr learned my addr 172.21.15.83:0/3074449572 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:45:11.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012effd640 1 --2- 172.21.15.83:0/3074449572 >> v2:172.21.15.184:3300/0 conn(0x7f01301536b0 0x7f01300c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:11.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2808652838 0 0) 0x7f01300c9200 con 0x7f01300af350 2024-09-19T07:45:11.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f011c003530 con 0x7f01300af350 2024-09-19T07:45:11.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 672986138 0 0) 0x7f011c003530 con 0x7f01300af350 2024-09-19T07:45:11.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 >> v2:172.21.15.184:3300/0 conn(0x7f01301536b0 msgr2=0x7f01300c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:11.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 --2- 172.21.15.83:0/3074449572 >> v2:172.21.15.184:3300/0 conn(0x7f01301536b0 0x7f01300c2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 >> v1:172.21.15.184:6789/0 conn(0x7f0130156820 legacy=0x7f0130160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:11.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01300c8c40 con 0x7f01300af350 2024-09-19T07:45:11.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f0118001680 con 0x7f01300af350 2024-09-19T07:45:11.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 >> v1:172.21.15.83:6789/0 conn(0x7f01300af350 legacy=0x7f01300c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:11.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 --2- 172.21.15.83:0/3074449572 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f011c0039c0 0x7f011c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:11.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 --2- 172.21.15.83:0/3074449572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f011c004810 0x7f011c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:11.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 --2- 172.21.15.83:0/3074449572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f011c0051d0 0x7f011c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:11.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0130153a90 con 0x7f011c0051d0 2024-09-19T07:45:11.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f011c00c170 con 0x7f011c004810 2024-09-19T07:45:11.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f011c00c370 con 0x7f011c0039c0 2024-09-19T07:45:11.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012f7fe640 1 --2- 172.21.15.83:0/3074449572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f011c0051d0 0x7f011c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:11.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012effd640 1 --2- 172.21.15.83:0/3074449572 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f011c0039c0 0x7f011c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:11.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012e7fc640 1 --2- 172.21.15.83:0/3074449572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f011c004810 0x7f011c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:11.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012f7fe640 1 -- 172.21.15.83:0/3074449572 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f011c0039c0 msgr2=0x7f011c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:11.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012f7fe640 1 --2- 172.21.15.83:0/3074449572 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f011c0039c0 0x7f011c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012f7fe640 1 -- 172.21.15.83:0/3074449572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f011c004810 msgr2=0x7f011c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:11.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012f7fe640 1 --2- 172.21.15.83:0/3074449572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f011c004810 0x7f011c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012f7fe640 1 -- 172.21.15.83:0/3074449572 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011c003530 con 0x7f011c0051d0 2024-09-19T07:45:11.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012f7fe640 1 -- 172.21.15.83:0/3074449572 reap_dead start 2024-09-19T07:45:11.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012f7fe640 1 --2- 172.21.15.83:0/3074449572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f011c0051d0 0x7f011c00b9d0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f01300688c0 tx=0x7f012001c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4c7a2b3aa109c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:11.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.397+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0120001a90 con 0x7f011c0051d0 2024-09-19T07:45:11.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0120003b80 con 0x7f011c0051d0 2024-09-19T07:45:11.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012dffb640 1 -- 172.21.15.83:0/3074449572 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01200050d0 con 0x7f011c0051d0 2024-09-19T07:45:11.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 -- 172.21.15.83:0/3074449572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f011c0051d0 msgr2=0x7f011c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:11.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 --2- 172.21.15.83:0/3074449572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f011c0051d0 0x7f011c00b9d0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f01300688c0 tx=0x7f012001c720 comp rx=0 tx=0).stop 2024-09-19T07:45:11.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 -- 172.21.15.83:0/3074449572 shutdown_connections 2024-09-19T07:45:11.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 --2- 172.21.15.83:0/3074449572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f011c0051d0 0x7f011c00b9d0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 -- 172.21.15.83:0/3074449572 >> 172.21.15.83:0/3074449572 conn(0x7f013006ffa0 msgr2=0x7f01300703b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 -- 172.21.15.83:0/3074449572 shutdown_connections 2024-09-19T07:45:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 -- 172.21.15.83:0/3074449572 wait complete. 2024-09-19T07:45:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 Processor -- start 2024-09-19T07:45:11.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 -- start start 2024-09-19T07:45:11.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f011c0051d0 0x7f0130159580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:11.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0130157190 0x7f0130151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:11.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0130159fd0 0x7f013014d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:11.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f01300c7640 con 0x7f0130159fd0 2024-09-19T07:45:11.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01300c3670 con 0x7f011c0051d0 2024-09-19T07:45:11.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f01300c37f0 con 0x7f0130157190 2024-09-19T07:45:11.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0130159fd0 0x7f013014d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:11.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0130159fd0 0x7f013014d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46294/0 (socket says 172.21.15.83:46294) 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012f7fe640 1 -- 172.21.15.83:0/2339387455 learned_addr learned my addr 172.21.15.83:0/2339387455 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012effd640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f011c0051d0 0x7f0130159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012e7fc640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0130157190 0x7f0130151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012f7fe640 1 -- 172.21.15.83:0/2339387455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0130157190 msgr2=0x7f0130151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012f7fe640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0130157190 0x7f0130151a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012f7fe640 1 -- 172.21.15.83:0/2339387455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f011c0051d0 msgr2=0x7f0130159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012f7fe640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f011c0051d0 0x7f0130159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012f7fe640 1 -- 172.21.15.83:0/2339387455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013014af80 con 0x7f0130159fd0 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012e7fc640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0130157190 0x7f0130151a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f012f7fe640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0130159fd0 0x7f013014d280 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f012001cfc0 tx=0x7f011c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01277fe640 1 -- 172.21.15.83:0/2339387455 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0120001a90 con 0x7f0130159fd0 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01277fe640 1 -- 172.21.15.83:0/2339387455 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0120003980 con 0x7f0130159fd0 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01277fe640 1 -- 172.21.15.83:0/2339387455 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01200b3640 con 0x7f0130159fd0 2024-09-19T07:45:11.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 -- 172.21.15.83:0/2339387455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013014b220 con 0x7f0130159fd0 2024-09-19T07:45:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.401+0000 7f01355dc640 1 -- 172.21.15.83:0/2339387455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01301528e0 con 0x7f0130159fd0 2024-09-19T07:45:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.405+0000 7f01355dc640 1 -- 172.21.15.83:0/2339387455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01301efa70 con 0x7f0130159fd0 2024-09-19T07:45:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.405+0000 7f01277fe640 1 -- 172.21.15.83:0/2339387455 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f01200a9410 con 0x7f0130159fd0 2024-09-19T07:45:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.405+0000 7f01277fe640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01100678c0 0x7f0110069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.405+0000 7f012effd640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01100678c0 0x7f0110069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.405+0000 7f01277fe640 1 -- 172.21.15.83:0/2339387455 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7f0120120040 con 0x7f0130159fd0 2024-09-19T07:45:11.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.405+0000 7f012effd640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01100678c0 0x7f0110069d80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f01280099b0 tx=0x7f0128008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:11.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.409+0000 7f01277fe640 1 -- 172.21.15.83:0/2339387455 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f01301efa70 con 0x7f0130159fd0 2024-09-19T07:45:11.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.589+0000 7f01355dc640 1 -- 172.21.15.83:0/2339387455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01300c2a40 con 0x7f0130159fd0 2024-09-19T07:45:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01277fe640 1 -- 172.21.15.83:0/2339387455 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+6546 (secure 0 0 0) 0x7f01200b9060 con 0x7f0130159fd0 2024-09-19T07:45:11.592 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:11.592 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":175,"btime":"2024-09-19T07:44:34:305274+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:42:42.535410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6628":{"gid":6628,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":10,"addr":"172.21.15.83:6835/1429520945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1429520945},{"type":"v1","addr":"172.21.15.83:6835","nonce":1429520945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6628,"qdb_cluster":[6628]},"id":2}]} 2024-09-19T07:45:11.593 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 175 2024-09-19T07:45:11.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01355dc640 1 -- 172.21.15.83:0/2339387455 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01100678c0 msgr2=0x7f0110069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:11.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01355dc640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01100678c0 0x7f0110069d80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f01280099b0 tx=0x7f0128008040 comp rx=0 tx=0).stop 2024-09-19T07:45:11.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01355dc640 1 -- 172.21.15.83:0/2339387455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0130159fd0 msgr2=0x7f013014d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:11.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01355dc640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0130159fd0 0x7f013014d280 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f012001cfc0 tx=0x7f011c005840 comp rx=0 tx=0).stop 2024-09-19T07:45:11.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01355dc640 1 -- 172.21.15.83:0/2339387455 shutdown_connections 2024-09-19T07:45:11.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01355dc640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f01100678c0 0x7f0110069d80 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01355dc640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0130159fd0 0x7f013014d280 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01355dc640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0130157190 0x7f0130151a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01355dc640 1 --2- 172.21.15.83:0/2339387455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f011c0051d0 0x7f0130159580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:11.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01355dc640 1 -- 172.21.15.83:0/2339387455 >> 172.21.15.83:0/2339387455 conn(0x7f013006ffa0 msgr2=0x7f01300c50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:11.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01355dc640 1 -- 172.21.15.83:0/2339387455 shutdown_connections 2024-09-19T07:45:11.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:11.593+0000 7f01355dc640 1 -- 172.21.15.83:0/2339387455 wait complete. 2024-09-19T07:45:11.609 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:45:13.131 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:13.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9eb05640 1 Processor -- start 2024-09-19T07:45:13.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9eb05640 1 -- start start 2024-09-19T07:45:13.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9eb05640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fee98071100 0x7fee981545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:13.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9eb05640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee9806dea0 0x7fee9806e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:13.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9eb05640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee980a9e60 con 0x7fee980c28e0 2024-09-19T07:45:13.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9eb05640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fee98071b50 con 0x7fee98071100 2024-09-19T07:45:13.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9eb05640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fee980c1ff0 con 0x7fee9806dea0 2024-09-19T07:45:13.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9c8a1640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fee98071100 0x7fee981545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:13.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9d0a2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee9806dea0 0x7fee9806e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:13.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9d0a2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee9806dea0 0x7fee9806e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:34196/0 (socket says 172.21.15.83:34196) 2024-09-19T07:45:13.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9d0a2640 1 -- 172.21.15.83:0/2903942833 learned_addr learned my addr 172.21.15.83:0/2903942833 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:13.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9d0a2640 1 -- 172.21.15.83:0/2903942833 >> v2:172.21.15.184:3300/0 conn(0x7fee98071100 msgr2=0x7fee981545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:13.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9d0a2640 1 --2- 172.21.15.83:0/2903942833 >> v2:172.21.15.184:3300/0 conn(0x7fee98071100 0x7fee981545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9d0a2640 1 -- 172.21.15.83:0/2903942833 >> v1:172.21.15.184:6789/0 conn(0x7fee980c28e0 legacy=0x7fee98150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.261+0000 7fee9d0a2640 1 -- 172.21.15.83:0/2903942833 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee980aa140 con 0x7fee9806dea0 2024-09-19T07:45:13.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9d0a2640 1 --2- 172.21.15.83:0/2903942833 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee9806dea0 0x7fee9806e330 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fee90009e50 tx=0x7fee9001c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13f7da621a659c76 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:13.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee977fe640 1 -- 172.21.15.83:0/2903942833 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee900a5750 con 0x7fee9806dea0 2024-09-19T07:45:13.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee977fe640 1 -- 172.21.15.83:0/2903942833 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee900a58f0 con 0x7fee9806dea0 2024-09-19T07:45:13.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee977fe640 1 -- 172.21.15.83:0/2903942833 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee900a5cc0 con 0x7fee9806dea0 2024-09-19T07:45:13.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 -- 172.21.15.83:0/2903942833 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee9806dea0 msgr2=0x7fee9806e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 --2- 172.21.15.83:0/2903942833 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee9806dea0 0x7fee9806e330 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fee90009e50 tx=0x7fee9001c9a0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 -- 172.21.15.83:0/2903942833 shutdown_connections 2024-09-19T07:45:13.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 --2- 172.21.15.83:0/2903942833 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee9806dea0 0x7fee9806e330 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 --2- 172.21.15.83:0/2903942833 >> v2:172.21.15.184:3300/0 conn(0x7fee98071100 0x7fee981545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 -- 172.21.15.83:0/2903942833 >> 172.21.15.83:0/2903942833 conn(0x7fee9806bc80 msgr2=0x7fee9806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:13.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 -- 172.21.15.83:0/2903942833 shutdown_connections 2024-09-19T07:45:13.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 -- 172.21.15.83:0/2903942833 wait complete. 2024-09-19T07:45:13.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 Processor -- start 2024-09-19T07:45:13.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 -- start start 2024-09-19T07:45:13.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee9806dea0 0x7fee98151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:13.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee98071100 0x7fee98144900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:13.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee980c28e0 0x7fee98146ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:13.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee98071660 con 0x7fee980c28e0 2024-09-19T07:45:13.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fee980714e0 con 0x7fee9806dea0 2024-09-19T07:45:13.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9eb05640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fee98160c60 con 0x7fee98071100 2024-09-19T07:45:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9d0a2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee980c28e0 0x7fee98146ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9d0a2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee980c28e0 0x7fee98146ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46308/0 (socket says 172.21.15.83:46308) 2024-09-19T07:45:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.265+0000 7fee9d0a2640 1 -- 172.21.15.83:0/2977374539 learned_addr learned my addr 172.21.15.83:0/2977374539 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9c8a1640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee9806dea0 0x7fee98151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee97fff640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee98071100 0x7fee98144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9d0a2640 1 -- 172.21.15.83:0/2977374539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee98071100 msgr2=0x7fee98144900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9d0a2640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee98071100 0x7fee98144900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9d0a2640 1 -- 172.21.15.83:0/2977374539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee9806dea0 msgr2=0x7fee98151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9d0a2640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee9806dea0 0x7fee98151ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9d0a2640 1 -- 172.21.15.83:0/2977374539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee980c6050 con 0x7fee980c28e0 2024-09-19T07:45:13.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9c8a1640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee9806dea0 0x7fee98151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:45:13.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9d0a2640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee980c28e0 0x7fee98146ac0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fee90009e50 tx=0x7fee90019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee957fa640 1 -- 172.21.15.83:0/2977374539 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee900bc020 con 0x7fee980c28e0 2024-09-19T07:45:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee957fa640 1 -- 172.21.15.83:0/2977374539 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee900b8670 con 0x7fee980c28e0 2024-09-19T07:45:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9eb05640 1 -- 172.21.15.83:0/2977374539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee981462c0 con 0x7fee980c28e0 2024-09-19T07:45:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee957fa640 1 -- 172.21.15.83:0/2977374539 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee900b8a20 con 0x7fee980c28e0 2024-09-19T07:45:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9eb05640 1 -- 172.21.15.83:0/2977374539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee98150d10 con 0x7fee980c28e0 2024-09-19T07:45:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9eb05640 1 -- 172.21.15.83:0/2977374539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee9806e330 con 0x7fee980c28e0 2024-09-19T07:45:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee957fa640 1 -- 172.21.15.83:0/2977374539 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fee900bf020 con 0x7fee980c28e0 2024-09-19T07:45:13.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee957fa640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fee840678c0 0x7fee84069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:13.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee957fa640 1 -- 172.21.15.83:0/2977374539 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9561+0+0 (secure 0 0 0) 0x7fee90121b20 con 0x7fee980c28e0 2024-09-19T07:45:13.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9c8a1640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fee840678c0 0x7fee84069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:13.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.269+0000 7fee9c8a1640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fee840678c0 0x7fee84069d80 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fee98148520 tx=0x7fee88008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.273+0000 7fee957fa640 1 -- 172.21.15.83:0/2977374539 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fee900c6030 con 0x7fee980c28e0 2024-09-19T07:45:13.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.469+0000 7fee9eb05640 1 -- 172.21.15.83:0/2977374539 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee980c4f30 con 0x7fee980c28e0 2024-09-19T07:45:13.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.469+0000 7fee957fa640 1 -- 172.21.15.83:0/2977374539 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+5712 (secure 0 0 0) 0x7fee90016620 con 0x7fee980c28e0 2024-09-19T07:45:13.469 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:13.469 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":176,"btime":"2024-09-19T07:45:13:308212+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:13.308208+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:replay","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:45:13.469 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 176 2024-09-19T07:45:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.469+0000 7fee9eb05640 1 -- 172.21.15.83:0/2977374539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fee840678c0 msgr2=0x7fee84069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.469+0000 7fee9eb05640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fee840678c0 0x7fee84069d80 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fee98148520 tx=0x7fee88008040 comp rx=0 tx=0).stop 2024-09-19T07:45:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.469+0000 7fee9eb05640 1 -- 172.21.15.83:0/2977374539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee980c28e0 msgr2=0x7fee98146ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.469+0000 7fee9eb05640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee980c28e0 0x7fee98146ac0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fee90009e50 tx=0x7fee90019740 comp rx=0 tx=0).stop 2024-09-19T07:45:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.469+0000 7fee9eb05640 1 -- 172.21.15.83:0/2977374539 shutdown_connections 2024-09-19T07:45:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.469+0000 7fee9eb05640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fee840678c0 0x7fee84069d80 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.469+0000 7fee9eb05640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee980c28e0 0x7fee98146ac0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.469+0000 7fee9eb05640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee98071100 0x7fee98144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.469+0000 7fee9eb05640 1 --2- 172.21.15.83:0/2977374539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee9806dea0 0x7fee98151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.473+0000 7fee9eb05640 1 -- 172.21.15.83:0/2977374539 >> 172.21.15.83:0/2977374539 conn(0x7fee9806bc80 msgr2=0x7fee9806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.473+0000 7fee9eb05640 1 -- 172.21.15.83:0/2977374539 shutdown_connections 2024-09-19T07:45:13.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.473+0000 7fee9eb05640 1 -- 172.21.15.83:0/2977374539 wait complete. 2024-09-19T07:45:13.487 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:45:13.610 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:13.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef78665640 1 Processor -- start 2024-09-19T07:45:13.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef78665640 1 -- start start 2024-09-19T07:45:13.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef78665640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fef70150290 0x7fef700c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:13.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef78665640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef70150670 con 0x7fef70153eb0 2024-09-19T07:45:13.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef78665640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef700c2880 con 0x7fef70152b20 2024-09-19T07:45:13.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef78665640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fef700716c0 con 0x7fef70150290 2024-09-19T07:45:13.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef76401640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fef70150290 0x7fef700c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:13.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef76c02640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fef70153eb0 0x7fef70154340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59772/0 (socket says 172.21.15.83:59772) 2024-09-19T07:45:13.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef76c02640 1 -- 172.21.15.83:0/3935309817 learned_addr learned my addr 172.21.15.83:0/3935309817 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:45:13.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef76401640 1 -- 172.21.15.83:0/3935309817 >> v1:172.21.15.184:6789/0 conn(0x7fef70152b20 legacy=0x7fef700c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef76401640 1 -- 172.21.15.83:0/3935309817 >> v1:172.21.15.83:6789/0 conn(0x7fef70153eb0 legacy=0x7fef70154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef76401640 1 -- 172.21.15.83:0/3935309817 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef700c2b60 con 0x7fef70150290 2024-09-19T07:45:13.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.737+0000 7fef753ff640 1 -- 172.21.15.83:0/3935309817 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1870751351 0 0) 0x7fef70150670 con 0x7fef70153eb0 2024-09-19T07:45:13.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef76401640 1 --2- 172.21.15.83:0/3935309817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef70150290 0x7fef700c1ff0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fef6c0097c0 tx=0x7fef6c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ae8a56fd425465 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:13.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef753ff640 1 -- 172.21.15.83:0/3935309817 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c0b0070 con 0x7fef70150290 2024-09-19T07:45:13.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef753ff640 1 -- 172.21.15.83:0/3935309817 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef6c0ab600 con 0x7fef70150290 2024-09-19T07:45:13.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef753ff640 1 -- 172.21.15.83:0/3935309817 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c0ab9d0 con 0x7fef70150290 2024-09-19T07:45:13.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 -- 172.21.15.83:0/3935309817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef70150290 msgr2=0x7fef700c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 --2- 172.21.15.83:0/3935309817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef70150290 0x7fef700c1ff0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fef6c0097c0 tx=0x7fef6c01c4a0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 -- 172.21.15.83:0/3935309817 shutdown_connections 2024-09-19T07:45:13.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 --2- 172.21.15.83:0/3935309817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef70150290 0x7fef700c1ff0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 -- 172.21.15.83:0/3935309817 >> 172.21.15.83:0/3935309817 conn(0x7fef7006bc80 msgr2=0x7fef7006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:13.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 -- 172.21.15.83:0/3935309817 shutdown_connections 2024-09-19T07:45:13.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 -- 172.21.15.83:0/3935309817 wait complete. 2024-09-19T07:45:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 Processor -- start 2024-09-19T07:45:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 -- start start 2024-09-19T07:45:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef70150290 0x7fef700c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef70152b20 0x7fef700c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef70153eb0 0x7fef701484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:13.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fef700c4790 con 0x7fef70152b20 2024-09-19T07:45:13.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fef700c3490 con 0x7fef70150290 2024-09-19T07:45:13.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef78665640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef700c4910 con 0x7fef70153eb0 2024-09-19T07:45:13.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef75c00640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef70152b20 0x7fef700c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:13.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef75c00640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef70152b20 0x7fef700c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46338/0 (socket says 172.21.15.83:46338) 2024-09-19T07:45:13.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.741+0000 7fef75c00640 1 -- 172.21.15.83:0/1980677840 learned_addr learned my addr 172.21.15.83:0/1980677840 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:13.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef76401640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef70150290 0x7fef700c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:13.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef76c02640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef70153eb0 0x7fef701484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:13.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef75c00640 1 -- 172.21.15.83:0/1980677840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef70153eb0 msgr2=0x7fef701484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef75c00640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef70153eb0 0x7fef701484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef75c00640 1 -- 172.21.15.83:0/1980677840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef70150290 msgr2=0x7fef700c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef75c00640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef70150290 0x7fef700c5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef75c00640 1 -- 172.21.15.83:0/1980677840 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef70151720 con 0x7fef70152b20 2024-09-19T07:45:13.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef76c02640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef70153eb0 0x7fef701484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:45:13.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef75c00640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef70152b20 0x7fef700c6c00 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fef6000c500 tx=0x7fef6000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:13.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef66ffd640 1 -- 172.21.15.83:0/1980677840 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60016020 con 0x7fef70152b20 2024-09-19T07:45:13.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef66ffd640 1 -- 172.21.15.83:0/1980677840 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef60005220 con 0x7fef70152b20 2024-09-19T07:45:13.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef78665640 1 -- 172.21.15.83:0/1980677840 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef70151a00 con 0x7fef70152b20 2024-09-19T07:45:13.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef66ffd640 1 -- 172.21.15.83:0/1980677840 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60007030 con 0x7fef70152b20 2024-09-19T07:45:13.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef78665640 1 -- 172.21.15.83:0/1980677840 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef700c59c0 con 0x7fef70152b20 2024-09-19T07:45:13.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef78665640 1 -- 172.21.15.83:0/1980677840 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef700c5bc0 con 0x7fef70152b20 2024-09-19T07:45:13.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef66ffd640 1 -- 172.21.15.83:0/1980677840 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fef600071f0 con 0x7fef70152b20 2024-09-19T07:45:13.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef66ffd640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef540678c0 0x7fef54069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:13.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef66ffd640 1 -- 172.21.15.83:0/1980677840 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9647+0+0 (secure 0 0 0) 0x7fef60086590 con 0x7fef70152b20 2024-09-19T07:45:13.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef76401640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef540678c0 0x7fef54069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:13.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.745+0000 7fef76401640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef540678c0 0x7fef54069d80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fef6c0ad000 tx=0x7fef6c0a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:13.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.749+0000 7fef66ffd640 1 -- 172.21.15.83:0/1980677840 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fef700c5bc0 con 0x7fef70152b20 2024-09-19T07:45:13.764 INFO:tasks.workunit.client.0.smithi083.stdout:Syncing()...3 sec 2024-09-19T07:45:13.764 INFO:tasks.workunit.client.0.smithi083.stdout:FFSB benchmark finished at: Thu Sep 19 07:45:13 2024 2024-09-19T07:45:13.764 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:45:13.764 INFO:tasks.workunit.client.0.smithi083.stdout:Results: 2024-09-19T07:45:13.764 INFO:tasks.workunit.client.0.smithi083.stdout:Benchmark took 307.29 sec 2024-09-19T07:45:13.764 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:45:13.764 INFO:tasks.workunit.client.0.smithi083.stdout:Total Results 2024-09-19T07:45:13.764 INFO:tasks.workunit.client.0.smithi083.stdout:=============== 2024-09-19T07:45:13.765 INFO:tasks.workunit.client.0.smithi083.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-09-19T07:45:13.765 INFO:tasks.workunit.client.0.smithi083.stdout: ======= ============ ========= ======= =========== ========== 2024-09-19T07:45:13.765 INFO:tasks.workunit.client.0.smithi083.stdout: write : 1662720 5410.93 100.000% 100.000% 21.1MB/sec 2024-09-19T07:45:13.765 INFO:tasks.workunit.client.0.smithi083.stdout:- 2024-09-19T07:45:13.765 INFO:tasks.workunit.client.0.smithi083.stdout:5410.93 Transactions per Second 2024-09-19T07:45:13.765 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:45:13.765 INFO:tasks.workunit.client.0.smithi083.stdout:Throughput Results 2024-09-19T07:45:13.765 INFO:tasks.workunit.client.0.smithi083.stdout:=================== 2024-09-19T07:45:13.765 INFO:tasks.workunit.client.0.smithi083.stdout:Write Throughput: 21.1MB/sec 2024-09-19T07:45:13.771 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:45:13.772 INFO:tasks.workunit.client.0.smithi083.stdout:System Call Latency statistics in millisecs 2024-09-19T07:45:13.772 INFO:tasks.workunit.client.0.smithi083.stdout:===== 2024-09-19T07:45:13.772 INFO:tasks.workunit.client.0.smithi083.stdout: Min Avg Max Total Calls 2024-09-19T07:45:13.772 INFO:tasks.workunit.client.0.smithi083.stdout: ======== ======== ======== ============ 2024-09-19T07:45:13.772 INFO:tasks.workunit.client.0.smithi083.stdout:[ open] 0.417000 1420.525881 17499.775391 1299 2024-09-19T07:45:13.772 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[0] 5000.000000 - 10000.000000 : 6 2024-09-19T07:45:13.772 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-19T07:45:13.772 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-19T07:45:13.772 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-19T07:45:13.772 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-19T07:45:13.773 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-19T07:45:13.773 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[6] 50.000000 - 100.000000 : 25 2024-09-19T07:45:13.773 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[7] 20.000000 - 50.000000 : 285 2024-09-19T07:45:13.773 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[8] 10.000000 - 20.000000 : 526 2024-09-19T07:45:13.773 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[9] 5.000000 - 10.000000 : 218 2024-09-19T07:45:13.773 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[10] 2.000000 - 5.000000 : 79 2024-09-19T07:45:13.773 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[11] 1.000000 - 2.000000 : 27 2024-09-19T07:45:13.773 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[12] 0.500000 - 1.000000 : 10 2024-09-19T07:45:13.773 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[13] 0.200000 - 0.500000 : 1 2024-09-19T07:45:13.773 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-09-19T07:45:13.774 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-09-19T07:45:13.774 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-09-19T07:45:13.774 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-19T07:45:13.774 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-19T07:45:13.774 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:45:13.774 INFO:tasks.workunit.client.0.smithi083.stdout:[ write] 0.083000 4.699255 202.048996 1662720 2024-09-19T07:45:13.774 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-19T07:45:13.774 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-19T07:45:13.774 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-19T07:45:13.774 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-19T07:45:13.774 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[4] 200.000000 - 500.000000 : 1 2024-09-19T07:45:13.775 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[5] 100.000000 - 200.000000 : 93 2024-09-19T07:45:13.775 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[6] 50.000000 - 100.000000 : 3801 2024-09-19T07:45:13.775 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[7] 20.000000 - 50.000000 : 50972 2024-09-19T07:45:13.775 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[8] 10.000000 - 20.000000 : 138454 2024-09-19T07:45:13.775 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[9] 5.000000 - 10.000000 : 311536 2024-09-19T07:45:13.775 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[10] 2.000000 - 5.000000 : 478798 2024-09-19T07:45:13.775 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[11] 1.000000 - 2.000000 : 207105 2024-09-19T07:45:13.775 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[12] 0.500000 - 1.000000 : 74179 2024-09-19T07:45:13.775 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[13] 0.200000 - 0.500000 : 181957 2024-09-19T07:45:13.775 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[14] 0.100000 - 0.200000 : 208493 2024-09-19T07:45:13.775 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[15] 0.050000 - 0.100000 : 7331 2024-09-19T07:45:13.775 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-09-19T07:45:13.776 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-19T07:45:13.776 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-19T07:45:13.776 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:45:13.776 INFO:tasks.workunit.client.0.smithi083.stdout:[ lseek] 0.000000 0.000994 3.455000 1662720 2024-09-19T07:45:13.776 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-19T07:45:13.776 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-19T07:45:13.776 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-19T07:45:13.776 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-19T07:45:13.776 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-19T07:45:13.776 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-19T07:45:13.776 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-09-19T07:45:13.776 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-09-19T07:45:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-09-19T07:45:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-09-19T07:45:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[10] 2.000000 - 5.000000 : 5 2024-09-19T07:45:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[11] 1.000000 - 2.000000 : 1 2024-09-19T07:45:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[12] 0.500000 - 1.000000 : 7 2024-09-19T07:45:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[13] 0.200000 - 0.500000 : 17 2024-09-19T07:45:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[14] 0.100000 - 0.200000 : 29 2024-09-19T07:45:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[15] 0.050000 - 0.100000 : 62 2024-09-19T07:45:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[16] 0.020000 - 0.050000 : 422 2024-09-19T07:45:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[17] 0.010000 - 0.020000 : 1126 2024-09-19T07:45:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[18] 0.000000 - 0.010000 : 1661051 2024-09-19T07:45:13.777 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout:[ close] 0.019000 2.348793 50.880001 1299 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[6] 50.000000 - 100.000000 : 1 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[7] 20.000000 - 50.000000 : 28 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[8] 10.000000 - 20.000000 : 41 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[9] 5.000000 - 10.000000 : 83 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[10] 2.000000 - 5.000000 : 180 2024-09-19T07:45:13.778 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[11] 1.000000 - 2.000000 : 183 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[12] 0.500000 - 1.000000 : 111 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[13] 0.200000 - 0.500000 : 69 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[14] 0.100000 - 0.200000 : 93 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[15] 0.050000 - 0.100000 : 117 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[16] 0.020000 - 0.050000 : 392 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[17] 0.010000 - 0.020000 : 1 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout: 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout:0.6% User Time 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout:43.0% System Time 2024-09-19T07:45:13.779 INFO:tasks.workunit.client.0.smithi083.stdout:43.6% CPU Utilization 2024-09-19T07:45:13.780 INFO:tasks.workunit.client.0.smithi083.stderr:+ cd .. 2024-09-19T07:45:13.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.945+0000 7fef78665640 1 -- 172.21.15.83:0/1980677840 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef700c1ff0 con 0x7fef70152b20 2024-09-19T07:45:13.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef66ffd640 1 -- 172.21.15.83:0/1980677840 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+5712 (secure 0 0 0) 0x7fef60020020 con 0x7fef70152b20 2024-09-19T07:45:13.948 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:13.948 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":176,"btime":"2024-09-19T07:45:13:308212+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:13.308208+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:replay","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:45:13.948 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 176 2024-09-19T07:45:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef78665640 1 -- 172.21.15.83:0/1980677840 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef540678c0 msgr2=0x7fef54069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef78665640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef540678c0 0x7fef54069d80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fef6c0ad000 tx=0x7fef6c0a6000 comp rx=0 tx=0).stop 2024-09-19T07:45:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef78665640 1 -- 172.21.15.83:0/1980677840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef70152b20 msgr2=0x7fef700c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef78665640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef70152b20 0x7fef700c6c00 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fef6000c500 tx=0x7fef6000c9c0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef78665640 1 -- 172.21.15.83:0/1980677840 shutdown_connections 2024-09-19T07:45:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef78665640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef540678c0 0x7fef54069d80 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef78665640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef70153eb0 0x7fef701484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef78665640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef70152b20 0x7fef700c6c00 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef78665640 1 --2- 172.21.15.83:0/1980677840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef70150290 0x7fef700c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef78665640 1 -- 172.21.15.83:0/1980677840 >> 172.21.15.83:0/1980677840 conn(0x7fef7006bc80 msgr2=0x7fef7006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef78665640 1 -- 172.21.15.83:0/1980677840 shutdown_connections 2024-09-19T07:45:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:13.949+0000 7fef78665640 1 -- 172.21.15.83:0/1980677840 wait complete. 2024-09-19T07:45:13.967 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-19T07:45:13.967 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:45:13.967 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:14.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.089+0000 7f5f0c50b640 1 Processor -- start 2024-09-19T07:45:14.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.089+0000 7f5f0c50b640 1 -- start start 2024-09-19T07:45:14.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f0c50b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f04071100 0x7f5f041545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:14.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f0c50b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f040a9e60 con 0x7f5f040c28e0 2024-09-19T07:45:14.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f0c50b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f040aa140 con 0x7f5f0406dea0 2024-09-19T07:45:14.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f0c50b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5f04071b50 con 0x7f5f04071100 2024-09-19T07:45:14.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f0a2a7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f04071100 0x7f5f041545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:14.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f09aa6640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5f040c28e0 0x7f5f04150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59780/0 (socket says 172.21.15.83:59780) 2024-09-19T07:45:14.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f09aa6640 1 -- 172.21.15.83:0/2066539304 learned_addr learned my addr 172.21.15.83:0/2066539304 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:45:14.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f0a2a7640 1 -- 172.21.15.83:0/2066539304 >> v1:172.21.15.184:6789/0 conn(0x7f5f0406dea0 legacy=0x7f5f0406e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:14.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f0a2a7640 1 -- 172.21.15.83:0/2066539304 >> v1:172.21.15.83:6789/0 conn(0x7f5f040c28e0 legacy=0x7f5f04150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:14.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f0a2a7640 1 -- 172.21.15.83:0/2066539304 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f040c2240 con 0x7f5f04071100 2024-09-19T07:45:14.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f092a5640 1 -- 172.21.15.83:0/2066539304 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 585344671 0 0) 0x7f5f040a9e60 con 0x7f5f040c28e0 2024-09-19T07:45:14.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f0a2a7640 1 --2- 172.21.15.83:0/2066539304 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f04071100 0x7f5f041545e0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40097c0 tx=0x7f5ef401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346517c96065b5ca server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:14.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.093+0000 7f5f092a5640 1 -- 172.21.15.83:0/2066539304 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef40b0070 con 0x7f5f04071100 2024-09-19T07:45:14.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f092a5640 1 -- 172.21.15.83:0/2066539304 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ef40ab600 con 0x7f5f04071100 2024-09-19T07:45:14.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f092a5640 1 -- 172.21.15.83:0/2066539304 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef40a4e60 con 0x7f5f04071100 2024-09-19T07:45:14.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2066539304 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f04071100 msgr2=0x7f5f041545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:14.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 --2- 172.21.15.83:0/2066539304 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f04071100 0x7f5f041545e0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40097c0 tx=0x7f5ef401c3d0 comp rx=0 tx=0).stop 2024-09-19T07:45:14.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2066539304 shutdown_connections 2024-09-19T07:45:14.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 --2- 172.21.15.83:0/2066539304 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f04071100 0x7f5f041545e0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:14.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2066539304 >> 172.21.15.83:0/2066539304 conn(0x7f5f0406bc80 msgr2=0x7f5f0406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:14.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2066539304 shutdown_connections 2024-09-19T07:45:14.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2066539304 wait complete. 2024-09-19T07:45:14.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 Processor -- start 2024-09-19T07:45:14.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 -- start start 2024-09-19T07:45:14.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0406dea0 0x7f5f04152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:14.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f04071100 0x7f5f04146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:14.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f040c28e0 0x7f5f04147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:14.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5f04071660 con 0x7f5f04071100 2024-09-19T07:45:14.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5f040714e0 con 0x7f5f040c28e0 2024-09-19T07:45:14.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5f04161c70 con 0x7f5f0406dea0 2024-09-19T07:45:14.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f09aa6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f04071100 0x7f5f04146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:14.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f09aa6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f04071100 0x7f5f04146fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46362/0 (socket says 172.21.15.83:46362) 2024-09-19T07:45:14.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f09aa6640 1 -- 172.21.15.83:0/2349320892 learned_addr learned my addr 172.21.15.83:0/2349320892 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:14.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0aaa8640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f040c28e0 0x7f5f04147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:14.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0a2a7640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0406dea0 0x7f5f04152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:14.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f09aa6640 1 -- 172.21.15.83:0/2349320892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0406dea0 msgr2=0x7f5f04152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f09aa6640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0406dea0 0x7f5f04152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f09aa6640 1 -- 172.21.15.83:0/2349320892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f040c28e0 msgr2=0x7f5f04147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f09aa6640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f040c28e0 0x7f5f04147510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f09aa6640 1 -- 172.21.15.83:0/2349320892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0414c070 con 0x7f5f04071100 2024-09-19T07:45:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0aaa8640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f040c28e0 0x7f5f04147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:45:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f09aa6640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f04071100 0x7f5f04146fd0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f5efc001570 tx=0x7f5efc00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5efaffd640 1 -- 172.21.15.83:0/2349320892 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc018070 con 0x7f5f04071100 2024-09-19T07:45:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5efaffd640 1 -- 172.21.15.83:0/2349320892 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5efc00a640 con 0x7f5f04071100 2024-09-19T07:45:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5efaffd640 1 -- 172.21.15.83:0/2349320892 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc00aa30 con 0x7f5f04071100 2024-09-19T07:45:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2349320892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f041ec9c0 con 0x7f5f04071100 2024-09-19T07:45:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.097+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2349320892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f040c6df0 con 0x7f5f04071100 2024-09-19T07:45:14.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.101+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2349320892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f041ecec0 con 0x7f5f04071100 2024-09-19T07:45:14.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.101+0000 7f5efaffd640 1 -- 172.21.15.83:0/2349320892 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5efc00abd0 con 0x7f5f04071100 2024-09-19T07:45:14.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.101+0000 7f5efaffd640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5ee80678c0 0x7f5ee8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:14.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.101+0000 7f5efaffd640 1 -- 172.21.15.83:0/2349320892 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9647+0+0 (secure 0 0 0) 0x7f5efc086150 con 0x7f5f04071100 2024-09-19T07:45:14.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.101+0000 7f5f0a2a7640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5ee80678c0 0x7f5ee8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:14.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.101+0000 7f5f0a2a7640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5ee80678c0 0x7f5ee8069d80 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40ad000 tx=0x7f5ef4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:14.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.105+0000 7f5efaffd640 1 -- 172.21.15.83:0/2349320892 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5f041ecec0 con 0x7f5f04071100 2024-09-19T07:45:14.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.297+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2349320892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f041545e0 con 0x7f5f04071100 2024-09-19T07:45:14.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5efaffd640 1 -- 172.21.15.83:0/2349320892 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+5712 (secure 0 0 0) 0x7f5efc055d70 con 0x7f5f04071100 2024-09-19T07:45:14.300 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:14.300 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":176,"btime":"2024-09-19T07:45:13:308212+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:13.308208+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:replay","state_seq":1,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:45:14.300 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 176 2024-09-19T07:45:14.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2349320892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5ee80678c0 msgr2=0x7f5ee8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:14.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5f0c50b640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5ee80678c0 0x7f5ee8069d80 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40ad000 tx=0x7f5ef4008040 comp rx=0 tx=0).stop 2024-09-19T07:45:14.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2349320892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f04071100 msgr2=0x7f5f04146fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:14.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5f0c50b640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f04071100 0x7f5f04146fd0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f5efc001570 tx=0x7f5efc00cd20 comp rx=0 tx=0).stop 2024-09-19T07:45:14.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2349320892 shutdown_connections 2024-09-19T07:45:14.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5f0c50b640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5ee80678c0 0x7f5ee8069d80 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:14.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5f0c50b640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f040c28e0 0x7f5f04147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:14.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5f0c50b640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f04071100 0x7f5f04146fd0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:14.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5f0c50b640 1 --2- 172.21.15.83:0/2349320892 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f0406dea0 0x7f5f04152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:14.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2349320892 >> 172.21.15.83:0/2349320892 conn(0x7f5f0406bc80 msgr2=0x7f5f0415ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2349320892 shutdown_connections 2024-09-19T07:45:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:14.301+0000 7f5f0c50b640 1 -- 172.21.15.83:0/2349320892 wait complete. 2024-09-19T07:45:14.319 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:45:13:308212+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 176, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 174, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7803": { "addr": "172.21.15.83:6837/3490578864", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 3490578864, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 3490578864, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7803, "incarnation": 171, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:44:21.648179+0000", "qdb_cluster": [ 7803 ], "qdb_leader": 7803, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7803 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 176, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6152": { "addr": "172.21.15.184:6833/1207928072", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 1207928072, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 1207928072, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6152, "incarnation": 176, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:45:13.308208+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6152 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:45:15.488 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:15.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.617+0000 7f52f9a4e640 1 Processor -- start 2024-09-19T07:45:15.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.617+0000 7f52f9a4e640 1 -- start start 2024-09-19T07:45:15.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.617+0000 7f52f9a4e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52f40c28e0 0x7f52f41545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:15.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.617+0000 7f52f9a4e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f52f406dea0 0x7f52f406e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:15.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.617+0000 7f52f9a4e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52f40a9e60 con 0x7f52f4071100 2024-09-19T07:45:15.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.617+0000 7f52f9a4e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f52f4071b50 con 0x7f52f406dea0 2024-09-19T07:45:15.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.617+0000 7f52f9a4e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f52f40c1ff0 con 0x7f52f40c28e0 2024-09-19T07:45:15.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f52f406dea0 0x7f52f406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:15.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f2ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f52f4071100 0x7f52f4150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59782/0 (socket says 172.21.15.83:59782) 2024-09-19T07:45:15.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f2ffd640 1 -- 172.21.15.83:0/239317233 learned_addr learned my addr 172.21.15.83:0/239317233 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:45:15.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f27fc640 1 --2- 172.21.15.83:0/239317233 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52f40c28e0 0x7f52f41545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:15.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f37fe640 1 -- 172.21.15.83:0/239317233 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52f40c28e0 msgr2=0x7f52f41545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:15.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f37fe640 1 --2- 172.21.15.83:0/239317233 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52f40c28e0 0x7f52f41545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:15.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f37fe640 1 -- 172.21.15.83:0/239317233 >> v1:172.21.15.83:6789/0 conn(0x7f52f4071100 legacy=0x7f52f4150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:15.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f37fe640 1 -- 172.21.15.83:0/239317233 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f40aa140 con 0x7f52f406dea0 2024-09-19T07:45:15.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f1ffb640 1 -- 172.21.15.83:0/239317233 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2586645726 0 0) 0x7f52f40a9e60 con 0x7f52f4071100 2024-09-19T07:45:15.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f37fe640 1 --2- 172.21.15.83:0/239317233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f406dea0 0x7f52f406e350 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009d90 tx=0x7f52e401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4d3911bc8a8b2a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:15.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f1ffb640 1 -- 172.21.15.83:0/239317233 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e40b2070 con 0x7f52f406dea0 2024-09-19T07:45:15.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f1ffb640 1 -- 172.21.15.83:0/239317233 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e40a57a0 con 0x7f52f406dea0 2024-09-19T07:45:15.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f1ffb640 1 -- 172.21.15.83:0/239317233 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e40a5b70 con 0x7f52f406dea0 2024-09-19T07:45:15.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f9a4e640 1 -- 172.21.15.83:0/239317233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f406dea0 msgr2=0x7f52f406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:15.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f9a4e640 1 --2- 172.21.15.83:0/239317233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f406dea0 0x7f52f406e350 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009d90 tx=0x7f52e401c8e0 comp rx=0 tx=0).stop 2024-09-19T07:45:15.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f9a4e640 1 -- 172.21.15.83:0/239317233 shutdown_connections 2024-09-19T07:45:15.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f9a4e640 1 --2- 172.21.15.83:0/239317233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f406dea0 0x7f52f406e350 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:15.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f9a4e640 1 --2- 172.21.15.83:0/239317233 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52f40c28e0 0x7f52f41545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:15.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f9a4e640 1 -- 172.21.15.83:0/239317233 >> 172.21.15.83:0/239317233 conn(0x7f52f406bc80 msgr2=0x7f52f406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:15.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f9a4e640 1 -- 172.21.15.83:0/239317233 shutdown_connections 2024-09-19T07:45:15.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.621+0000 7f52f9a4e640 1 -- 172.21.15.83:0/239317233 wait complete. 2024-09-19T07:45:15.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f9a4e640 1 Processor -- start 2024-09-19T07:45:15.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f9a4e640 1 -- start start 2024-09-19T07:45:15.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f9a4e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52f406dea0 0x7f52f4152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:15.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f9a4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f4071100 0x7f52f4156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:15.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f9a4e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52f40c28e0 0x7f52f4154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:15.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f9a4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f52f4071660 con 0x7f52f4071100 2024-09-19T07:45:15.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f9a4e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f52f40714e0 con 0x7f52f406dea0 2024-09-19T07:45:15.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f9a4e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f52f4160c60 con 0x7f52f40c28e0 2024-09-19T07:45:15.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f4071100 0x7f52f4156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:15.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f4071100 0x7f52f4156990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46388/0 (socket says 172.21.15.83:46388) 2024-09-19T07:45:15.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f27fc640 1 -- 172.21.15.83:0/889395194 learned_addr learned my addr 172.21.15.83:0/889395194 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:15.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f27fc640 1 -- 172.21.15.83:0/889395194 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52f40c28e0 msgr2=0x7f52f4154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:15.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f27fc640 1 --2- 172.21.15.83:0/889395194 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52f40c28e0 0x7f52f4154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:15.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f27fc640 1 -- 172.21.15.83:0/889395194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52f406dea0 msgr2=0x7f52f4152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:15.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f27fc640 1 --2- 172.21.15.83:0/889395194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52f406dea0 0x7f52f4152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:15.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f27fc640 1 -- 172.21.15.83:0/889395194 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f4156ed0 con 0x7f52f4071100 2024-09-19T07:45:15.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f27fc640 1 --2- 172.21.15.83:0/889395194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f4071100 0x7f52f4156990 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00d5d0 tx=0x7f52dc00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:15.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52ebfff640 1 -- 172.21.15.83:0/889395194 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc004240 con 0x7f52f4071100 2024-09-19T07:45:15.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52ebfff640 1 -- 172.21.15.83:0/889395194 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52dc0043e0 con 0x7f52f4071100 2024-09-19T07:45:15.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f9a4e640 1 -- 172.21.15.83:0/889395194 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f41ec2a0 con 0x7f52f4071100 2024-09-19T07:45:15.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52ebfff640 1 -- 172.21.15.83:0/889395194 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc003840 con 0x7f52f4071100 2024-09-19T07:45:15.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f9a4e640 1 -- 172.21.15.83:0/889395194 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f4151f80 con 0x7f52f4071100 2024-09-19T07:45:15.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52f9a4e640 1 -- 172.21.15.83:0/889395194 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52f406e350 con 0x7f52f4071100 2024-09-19T07:45:15.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.625+0000 7f52ebfff640 1 -- 172.21.15.83:0/889395194 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f52dc0039e0 con 0x7f52f4071100 2024-09-19T07:45:15.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.629+0000 7f52ebfff640 1 --2- 172.21.15.83:0/889395194 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52d40678c0 0x7f52d4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:15.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.629+0000 7f52ebfff640 1 -- 172.21.15.83:0/889395194 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9647+0+0 (secure 0 0 0) 0x7f52dc086e10 con 0x7f52f4071100 2024-09-19T07:45:15.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.629+0000 7f52f2ffd640 1 --2- 172.21.15.83:0/889395194 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52d40678c0 0x7f52d4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:15.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.629+0000 7f52f2ffd640 1 --2- 172.21.15.83:0/889395194 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52d40678c0 0x7f52d4069d80 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0011f0 tx=0x7f52ec007930 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:15.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.633+0000 7f52ebfff640 1 -- 172.21.15.83:0/889395194 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f52f4156ed0 con 0x7f52f4071100 2024-09-19T07:45:15.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.825+0000 7f52f9a4e640 1 -- 172.21.15.83:0/889395194 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52f41530b0 con 0x7f52f4071100 2024-09-19T07:45:15.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.825+0000 7f52ebfff640 1 -- 172.21.15.83:0/889395194 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) ==== 95+0+5713 (secure 0 0 0) 0x7f52dc01c360 con 0x7f52f4071100 2024-09-19T07:45:15.826 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:15.826 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":178,"btime":"2024-09-19T07:45:15:734288+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:14.739219+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:45:15.826 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 178 2024-09-19T07:45:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.829+0000 7f52f9a4e640 1 -- 172.21.15.83:0/889395194 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52d40678c0 msgr2=0x7f52d4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.829+0000 7f52f9a4e640 1 --2- 172.21.15.83:0/889395194 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52d40678c0 0x7f52d4069d80 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0011f0 tx=0x7f52ec007930 comp rx=0 tx=0).stop 2024-09-19T07:45:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.829+0000 7f52f9a4e640 1 -- 172.21.15.83:0/889395194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f4071100 msgr2=0x7f52f4156990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.829+0000 7f52f9a4e640 1 --2- 172.21.15.83:0/889395194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f4071100 0x7f52f4156990 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00d5d0 tx=0x7f52dc00da90 comp rx=0 tx=0).stop 2024-09-19T07:45:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.829+0000 7f52f9a4e640 1 -- 172.21.15.83:0/889395194 shutdown_connections 2024-09-19T07:45:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.829+0000 7f52f9a4e640 1 --2- 172.21.15.83:0/889395194 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f52d40678c0 0x7f52d4069d80 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.829+0000 7f52f9a4e640 1 --2- 172.21.15.83:0/889395194 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52f40c28e0 0x7f52f4154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.829+0000 7f52f9a4e640 1 --2- 172.21.15.83:0/889395194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f4071100 0x7f52f4156990 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:15.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.829+0000 7f52f9a4e640 1 --2- 172.21.15.83:0/889395194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52f406dea0 0x7f52f4152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.829+0000 7f52f9a4e640 1 -- 172.21.15.83:0/889395194 >> 172.21.15.83:0/889395194 conn(0x7f52f406bc80 msgr2=0x7f52f406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.829+0000 7f52f9a4e640 1 -- 172.21.15.83:0/889395194 shutdown_connections 2024-09-19T07:45:15.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:15.829+0000 7f52f9a4e640 1 -- 172.21.15.83:0/889395194 wait complete. 2024-09-19T07:45:15.844 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:45:16.320 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:16.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.453+0000 7fcf90935640 1 Processor -- start 2024-09-19T07:45:16.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.453+0000 7fcf90935640 1 -- start start 2024-09-19T07:45:16.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf90935640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcf8c0c28e0 0x7fcf8c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:16.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf90935640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf8c06dea0 0x7fcf8c06e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:16.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf90935640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf8c0a9e60 con 0x7fcf8c071100 2024-09-19T07:45:16.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf90935640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fcf8c071b50 con 0x7fcf8c06dea0 2024-09-19T07:45:16.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf90935640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fcf8c0c1ff0 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf8ad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf8c06dea0 0x7fcf8c06e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:16.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf8ad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf8c06dea0 0x7fcf8c06e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46398/0 (socket says 172.21.15.83:46398) 2024-09-19T07:45:16.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf8ad9d640 1 -- 172.21.15.83:0/1808859506 learned_addr learned my addr 172.21.15.83:0/1808859506 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:16.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf81d9b640 1 --2- 172.21.15.83:0/1808859506 >> v2:172.21.15.184:3300/0 conn(0x7fcf8c0c28e0 0x7fcf8c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:16.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf8ad9d640 1 -- 172.21.15.83:0/1808859506 >> v2:172.21.15.184:3300/0 conn(0x7fcf8c0c28e0 msgr2=0x7fcf8c1545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:16.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf8ad9d640 1 --2- 172.21.15.83:0/1808859506 >> v2:172.21.15.184:3300/0 conn(0x7fcf8c0c28e0 0x7fcf8c1545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:16.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf8ad9d640 1 -- 172.21.15.83:0/1808859506 >> v1:172.21.15.184:6789/0 conn(0x7fcf8c071100 legacy=0x7fcf8c150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:16.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf8ad9d640 1 -- 172.21.15.83:0/1808859506 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8c0aa140 con 0x7fcf8c06dea0 2024-09-19T07:45:16.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf81d9b640 1 --2- 172.21.15.83:0/1808859506 >> v2:172.21.15.184:3300/0 conn(0x7fcf8c0c28e0 0x7fcf8c1545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:45:16.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf8ad9d640 1 --2- 172.21.15.83:0/1808859506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8c06dea0 0x7fcf8c06e330 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c0c2890 tx=0x7fcf8401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb607e2b2aa3f61 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:16.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf89d9b640 1 -- 172.21.15.83:0/1808859506 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf840a5710 con 0x7fcf8c06dea0 2024-09-19T07:45:16.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf89d9b640 1 -- 172.21.15.83:0/1808859506 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf840a58b0 con 0x7fcf8c06dea0 2024-09-19T07:45:16.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.457+0000 7fcf89d9b640 1 -- 172.21.15.83:0/1808859506 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf840a5c80 con 0x7fcf8c06dea0 2024-09-19T07:45:16.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 -- 172.21.15.83:0/1808859506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8c06dea0 msgr2=0x7fcf8c06e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:16.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 --2- 172.21.15.83:0/1808859506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8c06dea0 0x7fcf8c06e330 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c0c2890 tx=0x7fcf8401c960 comp rx=0 tx=0).stop 2024-09-19T07:45:16.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 -- 172.21.15.83:0/1808859506 shutdown_connections 2024-09-19T07:45:16.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 --2- 172.21.15.83:0/1808859506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8c06dea0 0x7fcf8c06e330 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:16.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 --2- 172.21.15.83:0/1808859506 >> v2:172.21.15.184:3300/0 conn(0x7fcf8c0c28e0 0x7fcf8c1545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:16.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 -- 172.21.15.83:0/1808859506 >> 172.21.15.83:0/1808859506 conn(0x7fcf8c06bc80 msgr2=0x7fcf8c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:16.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 -- 172.21.15.83:0/1808859506 shutdown_connections 2024-09-19T07:45:16.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 -- 172.21.15.83:0/1808859506 wait complete. 2024-09-19T07:45:16.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 Processor -- start 2024-09-19T07:45:16.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 -- start start 2024-09-19T07:45:16.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf8c06dea0 0x7fcf8c1469a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:16.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf8c071100 0x7fcf8c152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:16.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8c0c28e0 0x7fcf8c153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:16.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c071660 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c0714e0 con 0x7fcf8c071100 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcf8c160c60 con 0x7fcf8c06dea0 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf8ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8c0c28e0 0x7fcf8c153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf8ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8c0c28e0 0x7fcf8c153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46404/0 (socket says 172.21.15.83:46404) 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf8ad9d640 1 -- 172.21.15.83:0/2750122144 learned_addr learned my addr 172.21.15.83:0/2750122144 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf81d9b640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf8c071100 0x7fcf8c152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf8a59c640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf8c06dea0 0x7fcf8c1469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf8ad9d640 1 -- 172.21.15.83:0/2750122144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf8c06dea0 msgr2=0x7fcf8c1469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf8ad9d640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf8c06dea0 0x7fcf8c1469a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf8ad9d640 1 -- 172.21.15.83:0/2750122144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf8c071100 msgr2=0x7fcf8c152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf8ad9d640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf8c071100 0x7fcf8c152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf8ad9d640 1 -- 172.21.15.83:0/2750122144 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8c14eea0 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf8ad9d640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8c0c28e0 0x7fcf8c153450 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fcf840af040 tx=0x7fcf84019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:16.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf837fe640 1 -- 172.21.15.83:0/2750122144 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf840bc020 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf837fe640 1 -- 172.21.15.83:0/2750122144 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf840b8690 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 -- 172.21.15.83:0/2750122144 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf8c13f980 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf837fe640 1 -- 172.21.15.83:0/2750122144 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf840b8a80 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.461+0000 7fcf90935640 1 -- 172.21.15.83:0/2750122144 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf8c1ed2f0 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.465+0000 7fcf90935640 1 -- 172.21.15.83:0/2750122144 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf8c06e330 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.465+0000 7fcf837fe640 1 -- 172.21.15.83:0/2750122144 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcf840bf020 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.465+0000 7fcf837fe640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf6c0678c0 0x7fcf6c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:16.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.465+0000 7fcf837fe640 1 -- 172.21.15.83:0/2750122144 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9647+0+0 (secure 0 0 0) 0x7fcf84121c60 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.465+0000 7fcf8a59c640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf6c0678c0 0x7fcf6c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:16.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.465+0000 7fcf8a59c640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf6c0678c0 0x7fcf6c069d80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c001650 tx=0x7fcf7c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:16.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.469+0000 7fcf837fe640 1 -- 172.21.15.83:0/2750122144 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcf840c6030 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.661+0000 7fcf90935640 1 -- 172.21.15.83:0/2750122144 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf8c0c4f30 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.661+0000 7fcf837fe640 1 -- 172.21.15.83:0/2750122144 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) ==== 95+0+5713 (secure 0 0 0) 0x7fcf840165e0 con 0x7fcf8c0c28e0 2024-09-19T07:45:16.662 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:16.662 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":178,"btime":"2024-09-19T07:45:15:734288+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:14.739219+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:45:16.662 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 178 2024-09-19T07:45:16.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.665+0000 7fcf90935640 1 -- 172.21.15.83:0/2750122144 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf6c0678c0 msgr2=0x7fcf6c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:16.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.665+0000 7fcf90935640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf6c0678c0 0x7fcf6c069d80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c001650 tx=0x7fcf7c009040 comp rx=0 tx=0).stop 2024-09-19T07:45:16.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.665+0000 7fcf90935640 1 -- 172.21.15.83:0/2750122144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8c0c28e0 msgr2=0x7fcf8c153450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:16.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.665+0000 7fcf90935640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8c0c28e0 0x7fcf8c153450 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fcf840af040 tx=0x7fcf84019700 comp rx=0 tx=0).stop 2024-09-19T07:45:16.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.665+0000 7fcf90935640 1 -- 172.21.15.83:0/2750122144 shutdown_connections 2024-09-19T07:45:16.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.665+0000 7fcf90935640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcf6c0678c0 0x7fcf6c069d80 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:16.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.665+0000 7fcf90935640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8c0c28e0 0x7fcf8c153450 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:16.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.665+0000 7fcf90935640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf8c071100 0x7fcf8c152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:16.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.665+0000 7fcf90935640 1 --2- 172.21.15.83:0/2750122144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf8c06dea0 0x7fcf8c1469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:16.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.665+0000 7fcf90935640 1 -- 172.21.15.83:0/2750122144 >> 172.21.15.83:0/2750122144 conn(0x7fcf8c06bc80 msgr2=0x7fcf8c14f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:16.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.665+0000 7fcf90935640 1 -- 172.21.15.83:0/2750122144 shutdown_connections 2024-09-19T07:45:16.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:16.665+0000 7fcf90935640 1 -- 172.21.15.83:0/2750122144 wait complete. 2024-09-19T07:45:16.680 INFO:tasks.mds_thrash.fs.[b]:no change 2024-09-19T07:45:17.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:17.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc782ed640 1 Processor -- start 2024-09-19T07:45:17.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc782ed640 1 -- start start 2024-09-19T07:45:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc782ed640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc7014db20 0x7fcc700abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc782ed640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc700ab7b0 con 0x7fcc7006d490 2024-09-19T07:45:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc782ed640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc7006e970 con 0x7fcc70153b60 2024-09-19T07:45:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc782ed640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc7006e720 con 0x7fcc7014db20 2024-09-19T07:45:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc7688a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcc7006d490 0x7fcc7015e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59792/0 (socket says 172.21.15.83:59792) 2024-09-19T07:45:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc7688a640 1 -- 172.21.15.83:0/3018412238 learned_addr learned my addr 172.21.15.83:0/3018412238 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:45:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2669791783 0 0) 0x7fcc700ab7b0 con 0x7fcc7006d490 2024-09-19T07:45:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc5c003530 con 0x7fcc7006d490 2024-09-19T07:45:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc76089640 1 --2- 172.21.15.83:0/3018412238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc7014db20 0x7fcc700abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3680476750 0 0) 0x7fcc5c003530 con 0x7fcc7006d490 2024-09-19T07:45:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc7014db20 msgr2=0x7fcc700abbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 --2- 172.21.15.83:0/3018412238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc7014db20 0x7fcc700abbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 >> v1:172.21.15.184:6789/0 conn(0x7fcc70153b60 legacy=0x7fcc7006f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7006ec50 con 0x7fcc7006d490 2024-09-19T07:45:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fcc68001a00 con 0x7fcc7006d490 2024-09-19T07:45:18.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 >> v1:172.21.15.83:6789/0 conn(0x7fcc7006d490 legacy=0x7fcc7015e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:18.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 --2- 172.21.15.83:0/3018412238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc5c0039f0 0x7fcc5c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 --2- 172.21.15.83:0/3018412238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c004810 0x7fcc5c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 --2- 172.21.15.83:0/3018412238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc5c0051d0 0x7fcc5c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7006e720 con 0x7fcc5c004810 2024-09-19T07:45:18.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c00c020 con 0x7fcc5c0051d0 2024-09-19T07:45:18.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5c00c220 con 0x7fcc5c0039f0 2024-09-19T07:45:18.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75888640 1 --2- 172.21.15.83:0/3018412238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c004810 0x7fcc5c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:18.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75888640 1 -- 172.21.15.83:0/3018412238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc5c0039f0 msgr2=0x7fcc5c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:18.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75888640 1 --2- 172.21.15.83:0/3018412238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc5c0039f0 0x7fcc5c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75888640 1 -- 172.21.15.83:0/3018412238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc5c0051d0 msgr2=0x7fcc5c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:18.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75888640 1 --2- 172.21.15.83:0/3018412238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc5c0051d0 0x7fcc5c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:17.997+0000 7fcc75888640 1 -- 172.21.15.83:0/3018412238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7006ec50 con 0x7fcc5c004810 2024-09-19T07:45:18.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.001+0000 7fcc75888640 1 --2- 172.21.15.83:0/3018412238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c004810 0x7fcc5c004c90 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fcc700c5a70 tx=0x7fcc6001c2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8ea96b17ef69e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:18.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.001+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc60001290 con 0x7fcc5c004810 2024-09-19T07:45:18.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.001+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc60005270 con 0x7fcc5c004810 2024-09-19T07:45:18.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.001+0000 7fcc75087640 1 -- 172.21.15.83:0/3018412238 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc60003840 con 0x7fcc5c004810 2024-09-19T07:45:18.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.001+0000 7fcc782ed640 1 -- 172.21.15.83:0/3018412238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c004810 msgr2=0x7fcc5c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:18.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.001+0000 7fcc782ed640 1 --2- 172.21.15.83:0/3018412238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c004810 0x7fcc5c004c90 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fcc700c5a70 tx=0x7fcc6001c2c0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.001+0000 7fcc7688a640 1 -- 172.21.15.83:0/3018412238 reap_dead start 2024-09-19T07:45:18.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.001+0000 7fcc782ed640 1 -- 172.21.15.83:0/3018412238 shutdown_connections 2024-09-19T07:45:18.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.001+0000 7fcc782ed640 1 -- 172.21.15.83:0/3018412238 >> 172.21.15.83:0/3018412238 conn(0x7fcc700c7eb0 msgr2=0x7fcc700c82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:18.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.001+0000 7fcc782ed640 1 -- 172.21.15.83:0/3018412238 shutdown_connections 2024-09-19T07:45:18.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.001+0000 7fcc782ed640 1 -- 172.21.15.83:0/3018412238 wait complete. 2024-09-19T07:45:18.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc782ed640 1 Processor -- start 2024-09-19T07:45:18.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc782ed640 1 -- start start 2024-09-19T07:45:18.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc782ed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c004810 0x7fcc700c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc782ed640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc70157b50 0x7fcc700c7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc782ed640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc701567d0 0x7fcc70156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc782ed640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7006fe40 con 0x7fcc5c004810 2024-09-19T07:45:18.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc782ed640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7006f2a0 con 0x7fcc701567d0 2024-09-19T07:45:18.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc782ed640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc7014df00 con 0x7fcc70157b50 2024-09-19T07:45:18.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc76089640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c004810 0x7fcc700c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:18.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc76089640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c004810 0x7fcc700c6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46416/0 (socket says 172.21.15.83:46416) 2024-09-19T07:45:18.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc76089640 1 -- 172.21.15.83:0/1264867933 learned_addr learned my addr 172.21.15.83:0/1264867933 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:18.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc76089640 1 -- 172.21.15.83:0/1264867933 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc70157b50 msgr2=0x7fcc700c7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:18.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc76089640 1 --2- 172.21.15.83:0/1264867933 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc70157b50 0x7fcc700c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc76089640 1 -- 172.21.15.83:0/1264867933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc701567d0 msgr2=0x7fcc70156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:18.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc76089640 1 --2- 172.21.15.83:0/1264867933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc701567d0 0x7fcc70156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc76089640 1 -- 172.21.15.83:0/1264867933 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc700c7760 con 0x7fcc5c004810 2024-09-19T07:45:18.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc76089640 1 --2- 172.21.15.83:0/1264867933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c004810 0x7fcc700c6ce0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00d5e0 tx=0x7fcc6c00daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:18.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc66ffd640 1 -- 172.21.15.83:0/1264867933 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c0125f0 con 0x7fcc5c004810 2024-09-19T07:45:18.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc66ffd640 1 -- 172.21.15.83:0/1264867933 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc6c019070 con 0x7fcc5c004810 2024-09-19T07:45:18.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc782ed640 1 -- 172.21.15.83:0/1264867933 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc700c34d0 con 0x7fcc5c004810 2024-09-19T07:45:18.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc66ffd640 1 -- 172.21.15.83:0/1264867933 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c0129f0 con 0x7fcc5c004810 2024-09-19T07:45:18.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.005+0000 7fcc782ed640 1 -- 172.21.15.83:0/1264867933 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc700c3a70 con 0x7fcc5c004810 2024-09-19T07:45:18.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.009+0000 7fcc782ed640 1 -- 172.21.15.83:0/1264867933 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc70151950 con 0x7fcc5c004810 2024-09-19T07:45:18.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.009+0000 7fcc66ffd640 1 -- 172.21.15.83:0/1264867933 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcc6c012b90 con 0x7fcc5c004810 2024-09-19T07:45:18.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.009+0000 7fcc66ffd640 1 --2- 172.21.15.83:0/1264867933 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc540678c0 0x7fcc54069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.009+0000 7fcc66ffd640 1 -- 172.21.15.83:0/1264867933 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9647+0+0 (secure 0 0 0) 0x7fcc6c086f70 con 0x7fcc5c004810 2024-09-19T07:45:18.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.009+0000 7fcc75888640 1 --2- 172.21.15.83:0/1264867933 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc540678c0 0x7fcc54069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:18.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.009+0000 7fcc75888640 1 --2- 172.21.15.83:0/1264867933 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc540678c0 0x7fcc54069d80 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fcc700742e0 tx=0x7fcc60000fc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:18.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.013+0000 7fcc66ffd640 1 -- 172.21.15.83:0/1264867933 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcc6c056b90 con 0x7fcc5c004810 2024-09-19T07:45:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.297+0000 7fcc782ed640 1 -- 172.21.15.83:0/1264867933 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc700aa4a0 con 0x7fcc5c004810 2024-09-19T07:45:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc66ffd640 1 -- 172.21.15.83:0/1264867933 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) ==== 95+0+5720 (secure 0 0 0) 0x7fcc6c056490 con 0x7fcc5c004810 2024-09-19T07:45:18.300 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:18.300 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":179,"btime":"2024-09-19T07:45:16:742879+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 179 2024-09-19T07:45:18.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc782ed640 1 -- 172.21.15.83:0/1264867933 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc540678c0 msgr2=0x7fcc54069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:18.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc782ed640 1 --2- 172.21.15.83:0/1264867933 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc540678c0 0x7fcc54069d80 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fcc700742e0 tx=0x7fcc60000fc0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc782ed640 1 -- 172.21.15.83:0/1264867933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c004810 msgr2=0x7fcc700c6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:18.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc782ed640 1 --2- 172.21.15.83:0/1264867933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c004810 0x7fcc700c6ce0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00d5e0 tx=0x7fcc6c00daa0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc782ed640 1 -- 172.21.15.83:0/1264867933 shutdown_connections 2024-09-19T07:45:18.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc782ed640 1 --2- 172.21.15.83:0/1264867933 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcc540678c0 0x7fcc54069d80 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc782ed640 1 --2- 172.21.15.83:0/1264867933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc701567d0 0x7fcc70156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc782ed640 1 --2- 172.21.15.83:0/1264867933 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc70157b50 0x7fcc700c7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc782ed640 1 --2- 172.21.15.83:0/1264867933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c004810 0x7fcc700c6ce0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc782ed640 1 -- 172.21.15.83:0/1264867933 >> 172.21.15.83:0/1264867933 conn(0x7fcc700c7eb0 msgr2=0x7fcc7014ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:18.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc782ed640 1 -- 172.21.15.83:0/1264867933 shutdown_connections 2024-09-19T07:45:18.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.301+0000 7fcc782ed640 1 -- 172.21.15.83:0/1264867933 wait complete. 2024-09-19T07:45:18.331 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:45:18.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:18.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.829+0000 7f4955dcd640 1 Processor -- start 2024-09-19T07:45:18.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.829+0000 7f4955dcd640 1 -- start start 2024-09-19T07:45:18.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.829+0000 7f4955dcd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49500c7510 0x7f495006de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.829+0000 7f4955dcd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f495014ddd0 con 0x7f4950154360 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.829+0000 7f4955dcd640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f495006fdf0 con 0x7f495014d9f0 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.829+0000 7f4955dcd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f49500c8830 con 0x7f49500c7510 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.829+0000 7f494effd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49500c7510 0x7f495006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.829+0000 7f494effd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49500c7510 0x7f495006de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:34294/0 (socket says 172.21.15.83:34294) 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.829+0000 7f494effd640 1 -- 172.21.15.83:0/2705102054 learned_addr learned my addr 172.21.15.83:0/2705102054 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 855257000 0 0) 0x7f495006fdf0 con 0x7f495014d9f0 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4938003530 con 0x7f495014d9f0 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2839624070 0 0) 0x7f4938003530 con 0x7f495014d9f0 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49500c7510 msgr2=0x7f495006de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 --2- 172.21.15.83:0/2705102054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49500c7510 0x7f495006de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 >> v1:172.21.15.83:6789/0 conn(0x7f4950154360 legacy=0x7f49500c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49500c7f00 con 0x7f495014d9f0 2024-09-19T07:45:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f4940001680 con 0x7f495014d9f0 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 >> v1:172.21.15.184:6789/0 conn(0x7f495014d9f0 legacy=0x7f49500c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 --2- 172.21.15.83:0/2705102054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49380039c0 0x7f4938003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 --2- 172.21.15.83:0/2705102054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4938004810 0x7f4938004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 --2- 172.21.15.83:0/2705102054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4938005220 0x7f493801c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f49500c8830 con 0x7f49380039c0 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f493801d0e0 con 0x7f4938004810 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f493801d2e0 con 0x7f4938005220 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494f7fe640 1 --2- 172.21.15.83:0/2705102054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49380039c0 0x7f4938003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494effd640 1 --2- 172.21.15.83:0/2705102054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4938004810 0x7f4938004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494ffff640 1 --2- 172.21.15.83:0/2705102054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4938005220 0x7f493801c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494f7fe640 1 -- 172.21.15.83:0/2705102054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4938005220 msgr2=0x7f493801c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494f7fe640 1 --2- 172.21.15.83:0/2705102054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4938005220 0x7f493801c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494f7fe640 1 -- 172.21.15.83:0/2705102054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4938004810 msgr2=0x7f4938004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:18.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494f7fe640 1 --2- 172.21.15.83:0/2705102054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4938004810 0x7f4938004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494f7fe640 1 -- 172.21.15.83:0/2705102054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49500c7f00 con 0x7f49380039c0 2024-09-19T07:45:18.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.833+0000 7f494ffff640 1 --2- 172.21.15.83:0/2705102054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4938005220 0x7f493801c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:45:18.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.837+0000 7f494f7fe640 1 --2- 172.21.15.83:0/2705102054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49380039c0 0x7f4938003dc0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f494000c260 tx=0x7f494001cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e2c9bda8cbfe22 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:18.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.837+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49400b2070 con 0x7f49380039c0 2024-09-19T07:45:18.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.837+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49400a5a00 con 0x7f49380039c0 2024-09-19T07:45:18.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.837+0000 7f494e7fc640 1 -- 172.21.15.83:0/2705102054 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940007d50 con 0x7f49380039c0 2024-09-19T07:45:18.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.837+0000 7f4955dcd640 1 -- 172.21.15.83:0/2705102054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49380039c0 msgr2=0x7f4938003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:18.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.837+0000 7f4955dcd640 1 --2- 172.21.15.83:0/2705102054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49380039c0 0x7f4938003dc0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f494000c260 tx=0x7f494001cd00 comp rx=0 tx=0).stop 2024-09-19T07:45:18.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.837+0000 7f494ffff640 1 -- 172.21.15.83:0/2705102054 reap_dead start 2024-09-19T07:45:18.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.837+0000 7f4955dcd640 1 -- 172.21.15.83:0/2705102054 shutdown_connections 2024-09-19T07:45:18.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.837+0000 7f4955dcd640 1 -- 172.21.15.83:0/2705102054 >> 172.21.15.83:0/2705102054 conn(0x7f49500c6110 msgr2=0x7f49500c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:18.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.837+0000 7f4955dcd640 1 -- 172.21.15.83:0/2705102054 shutdown_connections 2024-09-19T07:45:18.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.837+0000 7f4955dcd640 1 -- 172.21.15.83:0/2705102054 wait complete. 2024-09-19T07:45:18.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4955dcd640 1 Processor -- start 2024-09-19T07:45:18.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4955dcd640 1 -- start start 2024-09-19T07:45:18.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4955dcd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4950152fa0 0x7f49501ed870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4955dcd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f495014ddd0 0x7f49501eddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4955dcd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f495014b520 0x7f4950154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4955dcd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f49500c48a0 con 0x7f495014b520 2024-09-19T07:45:18.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4955dcd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49500c4720 con 0x7f4950152fa0 2024-09-19T07:45:18.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4955dcd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f495006c490 con 0x7f495014ddd0 2024-09-19T07:45:18.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f494ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f495014b520 0x7f4950154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:18.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f494ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f495014b520 0x7f4950154cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46430/0 (socket says 172.21.15.83:46430) 2024-09-19T07:45:18.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f494ffff640 1 -- 172.21.15.83:0/3375641428 learned_addr learned my addr 172.21.15.83:0/3375641428 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f494ffff640 1 -- 172.21.15.83:0/3375641428 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f495014ddd0 msgr2=0x7f49501eddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f494ffff640 1 --2- 172.21.15.83:0/3375641428 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f495014ddd0 0x7f49501eddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f494ffff640 1 -- 172.21.15.83:0/3375641428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4950152fa0 msgr2=0x7f49501ed870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f494ffff640 1 --2- 172.21.15.83:0/3375641428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4950152fa0 0x7f49501ed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f494ffff640 1 -- 172.21.15.83:0/3375641428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4950151000 con 0x7f495014b520 2024-09-19T07:45:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f494ffff640 1 --2- 172.21.15.83:0/3375641428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f495014b520 0x7f4950154cd0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f4950154740 tx=0x7f494800db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4937fff640 1 -- 172.21.15.83:0/3375641428 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948018070 con 0x7f495014b520 2024-09-19T07:45:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4955dcd640 1 -- 172.21.15.83:0/3375641428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4950150d20 con 0x7f495014b520 2024-09-19T07:45:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4955dcd640 1 -- 172.21.15.83:0/3375641428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4950157730 con 0x7f495014b520 2024-09-19T07:45:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4937fff640 1 -- 172.21.15.83:0/3375641428 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4948003ba0 con 0x7f495014b520 2024-09-19T07:45:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.841+0000 7f4937fff640 1 -- 172.21.15.83:0/3375641428 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494800de80 con 0x7f495014b520 2024-09-19T07:45:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.845+0000 7f4955dcd640 1 -- 172.21.15.83:0/3375641428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49300052a0 con 0x7f495014b520 2024-09-19T07:45:18.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.845+0000 7f4937fff640 1 -- 172.21.15.83:0/3375641428 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f494801b070 con 0x7f495014b520 2024-09-19T07:45:18.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.845+0000 7f4937fff640 1 --2- 172.21.15.83:0/3375641428 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f493c0678c0 0x7f493c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:18.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.845+0000 7f4937fff640 1 -- 172.21.15.83:0/3375641428 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9647+0+0 (secure 0 0 0) 0x7f4948086210 con 0x7f495014b520 2024-09-19T07:45:18.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.845+0000 7f494f7fe640 1 --2- 172.21.15.83:0/3375641428 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f493c0678c0 0x7f493c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:18.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.845+0000 7f494f7fe640 1 --2- 172.21.15.83:0/3375641428 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f493c0678c0 0x7f493c069d80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f49400af000 tx=0x7f4940001a50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:18.853+0000 7f4937fff640 1 -- 172.21.15.83:0/3375641428 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4948055dd0 con 0x7f495014b520 2024-09-19T07:45:19.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.133+0000 7f4955dcd640 1 -- 172.21.15.83:0/3375641428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4930005590 con 0x7f495014b520 2024-09-19T07:45:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.133+0000 7f4937fff640 1 -- 172.21.15.83:0/3375641428 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) ==== 95+0+5720 (secure 0 0 0) 0x7f49480556d0 con 0x7f495014b520 2024-09-19T07:45:19.133 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:19.133 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":179,"btime":"2024-09-19T07:45:16:742879+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 179 2024-09-19T07:45:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.137+0000 7f4955dcd640 1 -- 172.21.15.83:0/3375641428 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f493c0678c0 msgr2=0x7f493c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.137+0000 7f4955dcd640 1 --2- 172.21.15.83:0/3375641428 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f493c0678c0 0x7f493c069d80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f49400af000 tx=0x7f4940001a50 comp rx=0 tx=0).stop 2024-09-19T07:45:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.137+0000 7f4955dcd640 1 -- 172.21.15.83:0/3375641428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f495014b520 msgr2=0x7f4950154cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.137+0000 7f4955dcd640 1 --2- 172.21.15.83:0/3375641428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f495014b520 0x7f4950154cd0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f4950154740 tx=0x7f494800db40 comp rx=0 tx=0).stop 2024-09-19T07:45:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.137+0000 7f4955dcd640 1 -- 172.21.15.83:0/3375641428 shutdown_connections 2024-09-19T07:45:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.137+0000 7f4955dcd640 1 --2- 172.21.15.83:0/3375641428 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f493c0678c0 0x7f493c069d80 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.137+0000 7f4955dcd640 1 --2- 172.21.15.83:0/3375641428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f495014b520 0x7f4950154cd0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.137+0000 7f4955dcd640 1 --2- 172.21.15.83:0/3375641428 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f495014ddd0 0x7f49501eddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.137+0000 7f4955dcd640 1 --2- 172.21.15.83:0/3375641428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4950152fa0 0x7f49501ed870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:19.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.137+0000 7f4955dcd640 1 -- 172.21.15.83:0/3375641428 >> 172.21.15.83:0/3375641428 conn(0x7f49500c6110 msgr2=0x7f49500c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:19.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.137+0000 7f4955dcd640 1 -- 172.21.15.83:0/3375641428 shutdown_connections 2024-09-19T07:45:19.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:19.137+0000 7f4955dcd640 1 -- 172.21.15.83:0/3375641428 wait complete. 2024-09-19T07:45:19.159 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=6628 2024-09-19T07:45:19.160 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.a 2024-09-19T07:45:19.337 INFO:tasks.ceph.mds.d.smithi184.stderr:2024-09-19T07:45:19.340+0000 7f99f2dcc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:45:20.332 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:20.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe40beef640 1 Processor -- start 2024-09-19T07:45:20.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe40beef640 1 -- start start 2024-09-19T07:45:20.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe40beef640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe40415afb0 0x7fe40415e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:20.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe40beef640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4040ace10 con 0x7fe4040a7740 2024-09-19T07:45:20.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe40beef640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4040ad0f0 con 0x7fe40406cef0 2024-09-19T07:45:20.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe40beef640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe4040ada20 con 0x7fe40415afb0 2024-09-19T07:45:20.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe40948a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe4040a7740 0x7fe4040c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47088/0 (socket says 172.21.15.83:47088) 2024-09-19T07:45:20.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe40948a640 1 -- 172.21.15.83:0/3542879047 learned_addr learned my addr 172.21.15.83:0/3542879047 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:45:20.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2484667816 0 0) 0x7fe4040ace10 con 0x7fe4040a7740 2024-09-19T07:45:20.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3f0003530 con 0x7fe4040a7740 2024-09-19T07:45:20.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1503219392 0 0) 0x7fe3f0003530 con 0x7fe4040a7740 2024-09-19T07:45:20.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 >> v2:172.21.15.184:3300/0 conn(0x7fe40415afb0 msgr2=0x7fe40415e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe408c89640 1 --2- 172.21.15.83:0/3542879047 >> v2:172.21.15.184:3300/0 conn(0x7fe40415afb0 0x7fe40415e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 >> v1:172.21.15.184:6789/0 conn(0x7fe40406cef0 legacy=0x7fe40415a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4040ad3d0 con 0x7fe4040a7740 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fe3f4001680 con 0x7fe4040a7740 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 >> v1:172.21.15.83:6789/0 conn(0x7fe4040a7740 legacy=0x7fe4040c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.561+0000 7fe408c89640 1 --2- 172.21.15.83:0/3542879047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f0003a10 0x7fe3f0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe408c89640 1 --2- 172.21.15.83:0/3542879047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3f0004810 0x7fe3f0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe408c89640 1 --2- 172.21.15.83:0/3542879047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3f0005380 0x7fe3f000b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f000c2a0 con 0x7fe3f0003a10 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f000c450 con 0x7fe3f0004810 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f000c650 con 0x7fe3f0005380 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe40948a640 1 --2- 172.21.15.83:0/3542879047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3f0004810 0x7fe3f0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe409c8b640 1 --2- 172.21.15.83:0/3542879047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f0003a10 0x7fe3f0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe409c8b640 1 -- 172.21.15.83:0/3542879047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3f0005380 msgr2=0x7fe3f000b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe409c8b640 1 --2- 172.21.15.83:0/3542879047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3f0005380 0x7fe3f000b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:20.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe409c8b640 1 -- 172.21.15.83:0/3542879047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3f0004810 msgr2=0x7fe3f0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:20.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe409c8b640 1 --2- 172.21.15.83:0/3542879047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3f0004810 0x7fe3f0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:20.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe409c8b640 1 -- 172.21.15.83:0/3542879047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f0003530 con 0x7fe3f0003a10 2024-09-19T07:45:20.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe40a48c640 1 -- 172.21.15.83:0/3542879047 reap_dead start 2024-09-19T07:45:20.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe409c8b640 1 --2- 172.21.15.83:0/3542879047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f0003a10 0x7fe3f0003e10 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fe400001670 tx=0x7fe40001c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a5c5cbf424b78 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:20.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe400001270 con 0x7fe3f0003a10 2024-09-19T07:45:20.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe400005270 con 0x7fe3f0003a10 2024-09-19T07:45:20.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.565+0000 7fe408c89640 1 -- 172.21.15.83:0/3542879047 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe400003620 con 0x7fe3f0003a10 2024-09-19T07:45:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 -- 172.21.15.83:0/3542879047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f0003a10 msgr2=0x7fe3f0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 --2- 172.21.15.83:0/3542879047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f0003a10 0x7fe3f0003e10 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fe400001670 tx=0x7fe40001c2d0 comp rx=0 tx=0).stop 2024-09-19T07:45:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 -- 172.21.15.83:0/3542879047 shutdown_connections 2024-09-19T07:45:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 --2- 172.21.15.83:0/3542879047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f0003a10 0x7fe3f0003e10 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 -- 172.21.15.83:0/3542879047 >> 172.21.15.83:0/3542879047 conn(0x7fe4040c5060 msgr2=0x7fe4040c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 -- 172.21.15.83:0/3542879047 shutdown_connections 2024-09-19T07:45:20.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 -- 172.21.15.83:0/3542879047 wait complete. 2024-09-19T07:45:20.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 Processor -- start 2024-09-19T07:45:20.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 -- start start 2024-09-19T07:45:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3f0003a10 0x7fe4040a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe40406e860 0x7fe40406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe4041585f0 0x7fe404158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe40415ae30 con 0x7fe4041585f0 2024-09-19T07:45:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe40415acb0 con 0x7fe3f0003a10 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe40406d2d0 con 0x7fe40406e860 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40a48c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe4041585f0 0x7fe404158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40a48c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe4041585f0 0x7fe404158a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55252/0 (socket says 172.21.15.83:55252) 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40a48c640 1 -- 172.21.15.83:0/3734910871 learned_addr learned my addr 172.21.15.83:0/3734910871 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe409c8b640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3f0003a10 0x7fe4040a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40948a640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe40406e860 0x7fe40406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40a48c640 1 -- 172.21.15.83:0/3734910871 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe40406e860 msgr2=0x7fe40406ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40a48c640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe40406e860 0x7fe40406ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40a48c640 1 -- 172.21.15.83:0/3734910871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3f0003a10 msgr2=0x7fe4040a44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40a48c640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3f0003a10 0x7fe4040a44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40a48c640 1 -- 172.21.15.83:0/3734910871 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40406f220 con 0x7fe4041585f0 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40a48c640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe4041585f0 0x7fe404158a70 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fe3f00057f0 tx=0x7fe3fc00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe3fa7fc640 1 -- 172.21.15.83:0/3734910871 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc017070 con 0x7fe4041585f0 2024-09-19T07:45:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe3fa7fc640 1 -- 172.21.15.83:0/3734910871 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3fc00a830 con 0x7fe4041585f0 2024-09-19T07:45:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 -- 172.21.15.83:0/3734910871 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4040701d0 con 0x7fe4041585f0 2024-09-19T07:45:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe3fa7fc640 1 -- 172.21.15.83:0/3734910871 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc00ac20 con 0x7fe4041585f0 2024-09-19T07:45:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.569+0000 7fe40beef640 1 -- 172.21.15.83:0/3734910871 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe40406daa0 con 0x7fe4041585f0 2024-09-19T07:45:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.573+0000 7fe40beef640 1 -- 172.21.15.83:0/3734910871 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4040c7b60 con 0x7fe4041585f0 2024-09-19T07:45:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.573+0000 7fe3fa7fc640 1 -- 172.21.15.83:0/3734910871 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe3fc01a070 con 0x7fe4041585f0 2024-09-19T07:45:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.573+0000 7fe3fa7fc640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe3e80678c0 0x7fe3e8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.573+0000 7fe3fa7fc640 1 -- 172.21.15.83:0/3734910871 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9647+0+0 (secure 0 0 0) 0x7fe3fc086270 con 0x7fe4041585f0 2024-09-19T07:45:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.573+0000 7fe409c8b640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe3e80678c0 0x7fe3e8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.573+0000 7fe409c8b640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe3e80678c0 0x7fe3e8069d80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe4000aa000 tx=0x7fe400005040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:20.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.577+0000 7fe3fa7fc640 1 -- 172.21.15.83:0/3734910871 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe3fc0557b0 con 0x7fe4041585f0 2024-09-19T07:45:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.861+0000 7fe40beef640 1 -- 172.21.15.83:0/3734910871 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe40415a130 con 0x7fe4041585f0 2024-09-19T07:45:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.861+0000 7fe3fa7fc640 1 -- 172.21.15.83:0/3734910871 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) ==== 95+0+5720 (secure 0 0 0) 0x7fe3fc059ab0 con 0x7fe4041585f0 2024-09-19T07:45:20.861 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:20.861 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":179,"btime":"2024-09-19T07:45:16:742879+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 179 2024-09-19T07:45:20.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.865+0000 7fe40beef640 1 -- 172.21.15.83:0/3734910871 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe3e80678c0 msgr2=0x7fe3e8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:20.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.865+0000 7fe40beef640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe3e80678c0 0x7fe3e8069d80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe4000aa000 tx=0x7fe400005040 comp rx=0 tx=0).stop 2024-09-19T07:45:20.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.865+0000 7fe40beef640 1 -- 172.21.15.83:0/3734910871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe4041585f0 msgr2=0x7fe404158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:20.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.865+0000 7fe40beef640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe4041585f0 0x7fe404158a70 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fe3f00057f0 tx=0x7fe3fc00cb60 comp rx=0 tx=0).stop 2024-09-19T07:45:20.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.865+0000 7fe40beef640 1 -- 172.21.15.83:0/3734910871 shutdown_connections 2024-09-19T07:45:20.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.865+0000 7fe40beef640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe3e80678c0 0x7fe3e8069d80 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:20.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.865+0000 7fe40beef640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe4041585f0 0x7fe404158a70 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:20.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.865+0000 7fe40beef640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe40406e860 0x7fe40406ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:20.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.865+0000 7fe40beef640 1 --2- 172.21.15.83:0/3734910871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3f0003a10 0x7fe4040a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:20.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.865+0000 7fe40beef640 1 -- 172.21.15.83:0/3734910871 >> 172.21.15.83:0/3734910871 conn(0x7fe4040c5060 msgr2=0x7fe4040a8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:20.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.865+0000 7fe40beef640 1 -- 172.21.15.83:0/3734910871 shutdown_connections 2024-09-19T07:45:20.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:20.865+0000 7fe40beef640 1 -- 172.21.15.83:0/3734910871 wait complete. 2024-09-19T07:45:20.896 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:45:22.161 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-19T07:45:22.161 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:45:22.161 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:45:22.164 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:45:22.164 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:22.302 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:45:22.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf81088640 1 Processor -- start 2024-09-19T07:45:22.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf81088640 1 -- start start 2024-09-19T07:45:22.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf81088640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf7c155d70 0x7faf7c0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:22.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf81088640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf7c156200 con 0x7faf7c0c6120 2024-09-19T07:45:22.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf81088640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf7c1564e0 con 0x7faf7c1592f0 2024-09-19T07:45:22.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf81088640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faf7c0c9930 con 0x7faf7c155d70 2024-09-19T07:45:22.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf7c155d70 0x7faf7c0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:22.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf7b59e640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7faf7c1592f0 0x7faf7c157ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:40210/0 (socket says 172.21.15.83:40210) 2024-09-19T07:45:22.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf7b59e640 1 -- 172.21.15.83:0/227230137 learned_addr learned my addr 172.21.15.83:0/227230137 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:45:22.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf7a59c640 1 -- 172.21.15.83:0/227230137 >> v1:172.21.15.184:6789/0 conn(0x7faf7c1592f0 legacy=0x7faf7c157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:22.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf7a59c640 1 -- 172.21.15.83:0/227230137 >> v1:172.21.15.83:6789/0 conn(0x7faf7c0c6120 legacy=0x7faf7c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:22.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf7a59c640 1 -- 172.21.15.83:0/227230137 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf7c1567c0 con 0x7faf7c155d70 2024-09-19T07:45:22.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.321+0000 7faf79d9b640 1 -- 172.21.15.83:0/227230137 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2685433291 0 0) 0x7faf7c1564e0 con 0x7faf7c1592f0 2024-09-19T07:45:22.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.329+0000 7faf7a59c640 1 --2- 172.21.15.83:0/227230137 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf7c155d70 0x7faf7c0c6500 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faf700097f0 tx=0x7faf7001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60f5b32441b2e01b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:22.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.329+0000 7faf79d9b640 1 -- 172.21.15.83:0/227230137 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf700b0070 con 0x7faf7c155d70 2024-09-19T07:45:22.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.329+0000 7faf79d9b640 1 -- 172.21.15.83:0/227230137 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf700ab600 con 0x7faf7c155d70 2024-09-19T07:45:22.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.329+0000 7faf79d9b640 1 -- 172.21.15.83:0/227230137 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf700ab9d0 con 0x7faf7c155d70 2024-09-19T07:45:22.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.329+0000 7faf81088640 1 -- 172.21.15.83:0/227230137 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf7c155d70 msgr2=0x7faf7c0c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:22.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.329+0000 7faf81088640 1 --2- 172.21.15.83:0/227230137 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf7c155d70 0x7faf7c0c6500 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faf700097f0 tx=0x7faf7001c470 comp rx=0 tx=0).stop 2024-09-19T07:45:22.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.329+0000 7faf81088640 1 -- 172.21.15.83:0/227230137 shutdown_connections 2024-09-19T07:45:22.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.329+0000 7faf81088640 1 --2- 172.21.15.83:0/227230137 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf7c155d70 0x7faf7c0c6500 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:22.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.329+0000 7faf81088640 1 -- 172.21.15.83:0/227230137 >> 172.21.15.83:0/227230137 conn(0x7faf7c06bc80 msgr2=0x7faf7c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:22.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.329+0000 7faf81088640 1 -- 172.21.15.83:0/227230137 shutdown_connections 2024-09-19T07:45:22.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.329+0000 7faf81088640 1 -- 172.21.15.83:0/227230137 wait complete. 2024-09-19T07:45:22.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.333+0000 7faf81088640 1 Processor -- start 2024-09-19T07:45:22.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.333+0000 7faf81088640 1 -- start start 2024-09-19T07:45:22.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.333+0000 7faf81088640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf7c0c6120 0x7faf7c152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:22.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.333+0000 7faf81088640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf7c1592f0 0x7faf7c149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:22.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.333+0000 7faf7ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf7c0c6120 0x7faf7c152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:22.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.333+0000 7faf7ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf7c0c6120 0x7faf7c152810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55276/0 (socket says 172.21.15.83:55276) 2024-09-19T07:45:22.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.333+0000 7faf7ad9d640 1 -- 172.21.15.83:0/2318978788 learned_addr learned my addr 172.21.15.83:0/2318978788 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:22.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.333+0000 7faf81088640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf7c0c21d0 0x7faf7c149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:22.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.333+0000 7faf81088640 1 -- 172.21.15.83:0/2318978788 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c164030 con 0x7faf7c0c6120 2024-09-19T07:45:22.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.333+0000 7faf81088640 1 -- 172.21.15.83:0/2318978788 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c163eb0 con 0x7faf7c0c21d0 2024-09-19T07:45:22.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.333+0000 7faf81088640 1 -- 172.21.15.83:0/2318978788 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faf7c1641b0 con 0x7faf7c1592f0 2024-09-19T07:45:22.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf7b59e640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf7c0c21d0 0x7faf7c149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:22.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf7ad9d640 1 -- 172.21.15.83:0/2318978788 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf7c1592f0 msgr2=0x7faf7c149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:22.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf7ad9d640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf7c1592f0 0x7faf7c149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:22.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf7ad9d640 1 -- 172.21.15.83:0/2318978788 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf7c0c21d0 msgr2=0x7faf7c149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:22.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf7ad9d640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf7c0c21d0 0x7faf7c149e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:22.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf7ad9d640 1 -- 172.21.15.83:0/2318978788 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf7c0caca0 con 0x7faf7c0c6120 2024-09-19T07:45:22.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf7ad9d640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf7c0c6120 0x7faf7c152810 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00c750 tx=0x7faf6c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:22.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf637fe640 1 -- 172.21.15.83:0/2318978788 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c016020 con 0x7faf7c0c6120 2024-09-19T07:45:22.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf637fe640 1 -- 172.21.15.83:0/2318978788 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf6c0050d0 con 0x7faf7c0c6120 2024-09-19T07:45:22.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf81088640 1 -- 172.21.15.83:0/2318978788 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf7c0ca9c0 con 0x7faf7c0c6120 2024-09-19T07:45:22.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf637fe640 1 -- 172.21.15.83:0/2318978788 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c0054c0 con 0x7faf7c0c6120 2024-09-19T07:45:22.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf81088640 1 -- 172.21.15.83:0/2318978788 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf7c0cb7c0 con 0x7faf7c0c6120 2024-09-19T07:45:22.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf637fe640 1 -- 172.21.15.83:0/2318978788 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf6c003600 con 0x7faf7c0c6120 2024-09-19T07:45:22.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf617fa640 1 -- 172.21.15.83:0/2318978788 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf7c069080 con 0x7faf7c0c6120 2024-09-19T07:45:22.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.337+0000 7faf637fe640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf680678f0 0x7faf68069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:22.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.341+0000 7faf7a59c640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf680678f0 0x7faf68069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:22.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.341+0000 7faf637fe640 1 -- 172.21.15.83:0/2318978788 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9647+0+0 (secure 0 0 0) 0x7faf6c0534c0 con 0x7faf7c0c6120 2024-09-19T07:45:22.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.341+0000 7faf7a59c640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf680678f0 0x7faf68069db0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7faf700ad000 tx=0x7faf70004740 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:22.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.345+0000 7faf637fe640 1 -- 172.21.15.83:0/2318978788 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7faf6c055b20 con 0x7faf7c0c6120 2024-09-19T07:45:22.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.629+0000 7faf617fa640 1 -- 172.21.15.83:0/2318978788 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf7c0c96a0 con 0x7faf7c0c6120 2024-09-19T07:45:22.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.629+0000 7faf637fe640 1 -- 172.21.15.83:0/2318978788 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) ==== 95+0+5720 (secure 0 0 0) 0x7faf6c055420 con 0x7faf7c0c6120 2024-09-19T07:45:22.630 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:22.630 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":179,"btime":"2024-09-19T07:45:16:742879+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:22.630 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 179 2024-09-19T07:45:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.633+0000 7faf617fa640 1 -- 172.21.15.83:0/2318978788 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf680678f0 msgr2=0x7faf68069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.633+0000 7faf617fa640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf680678f0 0x7faf68069db0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7faf700ad000 tx=0x7faf70004740 comp rx=0 tx=0).stop 2024-09-19T07:45:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.633+0000 7faf617fa640 1 -- 172.21.15.83:0/2318978788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf7c0c6120 msgr2=0x7faf7c152810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.633+0000 7faf617fa640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf7c0c6120 0x7faf7c152810 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00c750 tx=0x7faf6c00cc10 comp rx=0 tx=0).stop 2024-09-19T07:45:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.633+0000 7faf617fa640 1 -- 172.21.15.83:0/2318978788 shutdown_connections 2024-09-19T07:45:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.633+0000 7faf617fa640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7faf680678f0 0x7faf68069db0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:22.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.633+0000 7faf617fa640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf7c0c21d0 0x7faf7c149e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:22.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.633+0000 7faf617fa640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf7c1592f0 0x7faf7c149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:22.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.633+0000 7faf617fa640 1 --2- 172.21.15.83:0/2318978788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf7c0c6120 0x7faf7c152810 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:22.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.633+0000 7faf617fa640 1 -- 172.21.15.83:0/2318978788 >> 172.21.15.83:0/2318978788 conn(0x7faf7c06bc80 msgr2=0x7faf7c151ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:22.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.633+0000 7faf617fa640 1 -- 172.21.15.83:0/2318978788 shutdown_connections 2024-09-19T07:45:22.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:22.633+0000 7faf617fa640 1 -- 172.21.15.83:0/2318978788 wait complete. 2024-09-19T07:45:22.655 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-19T07:45:22.897 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:23.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.041+0000 7f1d16de6640 1 Processor -- start 2024-09-19T07:45:23.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.041+0000 7f1d16de6640 1 -- start start 2024-09-19T07:45:23.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.045+0000 7f1d16de6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d1014dbe0 0x7f1d100c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:23.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.045+0000 7f1d16de6640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1d100c3ab0 0x7f1d10157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:23.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.045+0000 7f1d16de6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d10073d90 con 0x7f1d100c44e0 2024-09-19T07:45:23.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.045+0000 7f1d16de6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f1d10154700 con 0x7f1d1014dbe0 2024-09-19T07:45:23.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.045+0000 7f1d16de6640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f1d10153b30 con 0x7f1d100c3ab0 2024-09-19T07:45:23.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.045+0000 7f1d14b82640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d1014dbe0 0x7f1d100c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:23.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.045+0000 7f1d14b82640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d1014dbe0 0x7f1d100c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55288/0 (socket says 172.21.15.83:55288) 2024-09-19T07:45:23.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.045+0000 7f1d14b82640 1 -- 172.21.15.83:0/2984578257 learned_addr learned my addr 172.21.15.83:0/2984578257 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:23.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.045+0000 7f1d14b82640 1 -- 172.21.15.83:0/2984578257 >> v2:172.21.15.184:3300/0 conn(0x7f1d100c3ab0 msgr2=0x7f1d10157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:23.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.045+0000 7f1d14b82640 1 --2- 172.21.15.83:0/2984578257 >> v2:172.21.15.184:3300/0 conn(0x7f1d100c3ab0 0x7f1d10157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:23.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.045+0000 7f1d14b82640 1 -- 172.21.15.83:0/2984578257 >> v1:172.21.15.184:6789/0 conn(0x7f1d100c44e0 legacy=0x7f1d10157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:23.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.045+0000 7f1d14b82640 1 -- 172.21.15.83:0/2984578257 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d10153dd0 con 0x7f1d1014dbe0 2024-09-19T07:45:23.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d14b82640 1 --2- 172.21.15.83:0/2984578257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d1014dbe0 0x7f1d100c3fa0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f1d100c3830 tx=0x7f1d0401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d77cb33119de055 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:23.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d0f7fe640 1 -- 172.21.15.83:0/2984578257 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d040b1070 con 0x7f1d1014dbe0 2024-09-19T07:45:23.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d0f7fe640 1 -- 172.21.15.83:0/2984578257 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d040ac600 con 0x7f1d1014dbe0 2024-09-19T07:45:23.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d0f7fe640 1 -- 172.21.15.83:0/2984578257 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d040ac9d0 con 0x7f1d1014dbe0 2024-09-19T07:45:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d16de6640 1 -- 172.21.15.83:0/2984578257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d1014dbe0 msgr2=0x7f1d100c3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d16de6640 1 --2- 172.21.15.83:0/2984578257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d1014dbe0 0x7f1d100c3fa0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f1d100c3830 tx=0x7f1d0401c490 comp rx=0 tx=0).stop 2024-09-19T07:45:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d16de6640 1 -- 172.21.15.83:0/2984578257 shutdown_connections 2024-09-19T07:45:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d16de6640 1 --2- 172.21.15.83:0/2984578257 >> v2:172.21.15.184:3300/0 conn(0x7f1d100c3ab0 0x7f1d10157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d16de6640 1 --2- 172.21.15.83:0/2984578257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d1014dbe0 0x7f1d100c3fa0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d16de6640 1 -- 172.21.15.83:0/2984578257 >> 172.21.15.83:0/2984578257 conn(0x7f1d100c95a0 msgr2=0x7f1d100c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d16de6640 1 -- 172.21.15.83:0/2984578257 shutdown_connections 2024-09-19T07:45:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d16de6640 1 -- 172.21.15.83:0/2984578257 wait complete. 2024-09-19T07:45:23.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.049+0000 7f1d16de6640 1 Processor -- start 2024-09-19T07:45:23.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d16de6640 1 -- start start 2024-09-19T07:45:23.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d16de6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d100c44e0 0x7f1d100c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:23.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d16de6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d1014dbe0 0x7f1d10142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:23.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d16de6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d100c8de0 0x7f1d101431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:23.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d16de6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1d10160bc0 con 0x7f1d100c8de0 2024-09-19T07:45:23.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d16de6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1d100741a0 con 0x7f1d100c44e0 2024-09-19T07:45:23.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d16de6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1d10160d40 con 0x7f1d1014dbe0 2024-09-19T07:45:23.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d15383640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d100c8de0 0x7f1d101431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:23.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d15383640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d100c8de0 0x7f1d101431d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55296/0 (socket says 172.21.15.83:55296) 2024-09-19T07:45:23.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d15383640 1 -- 172.21.15.83:0/2496870339 learned_addr learned my addr 172.21.15.83:0/2496870339 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:23.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d14b82640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d100c44e0 0x7f1d100c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:23.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d0ffff640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d1014dbe0 0x7f1d10142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:23.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d15383640 1 -- 172.21.15.83:0/2496870339 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d1014dbe0 msgr2=0x7f1d10142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:23.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d15383640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d1014dbe0 0x7f1d10142c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:23.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d15383640 1 -- 172.21.15.83:0/2496870339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d100c44e0 msgr2=0x7f1d100c2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:23.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d15383640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d100c44e0 0x7f1d100c2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:23.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d15383640 1 -- 172.21.15.83:0/2496870339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d1006d2b0 con 0x7f1d100c8de0 2024-09-19T07:45:23.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d0ffff640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d1014dbe0 0x7f1d10142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:45:23.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d15383640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d100c8de0 0x7f1d101431d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800c770 tx=0x7f1d0800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:23.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d0d7fa640 1 -- 172.21.15.83:0/2496870339 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d08003b80 con 0x7f1d100c8de0 2024-09-19T07:45:23.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d0d7fa640 1 -- 172.21.15.83:0/2496870339 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d08003d20 con 0x7f1d100c8de0 2024-09-19T07:45:23.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d0d7fa640 1 -- 172.21.15.83:0/2496870339 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0800aa90 con 0x7f1d100c8de0 2024-09-19T07:45:23.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d16de6640 1 -- 172.21.15.83:0/2496870339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d1006d590 con 0x7f1d100c8de0 2024-09-19T07:45:23.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.053+0000 7f1d16de6640 1 -- 172.21.15.83:0/2496870339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d1006be00 con 0x7f1d100c8de0 2024-09-19T07:45:23.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.057+0000 7f1d0d7fa640 1 -- 172.21.15.83:0/2496870339 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1d0800ac30 con 0x7f1d100c8de0 2024-09-19T07:45:23.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.057+0000 7f1d16de6640 1 -- 172.21.15.83:0/2496870339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf00052a0 con 0x7f1d100c8de0 2024-09-19T07:45:23.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.057+0000 7f1d0d7fa640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1cf4067940 0x7f1cf4069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:23.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.057+0000 7f1d0d7fa640 1 -- 172.21.15.83:0/2496870339 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9647+0+0 (secure 0 0 0) 0x7f1d08086aa0 con 0x7f1d100c8de0 2024-09-19T07:45:23.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.057+0000 7f1d14b82640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1cf4067940 0x7f1cf4069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:23.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.057+0000 7f1d14b82640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1cf4067940 0x7f1cf4069e00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1d040ae000 tx=0x7f1d040a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:23.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.061+0000 7f1d0d7fa640 1 -- 172.21.15.83:0/2496870339 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1d080566b0 con 0x7f1d100c8de0 2024-09-19T07:45:23.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.345+0000 7f1d16de6640 1 -- 172.21.15.83:0/2496870339 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cf0005860 con 0x7f1d100c8de0 2024-09-19T07:45:23.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.345+0000 7f1d0d7fa640 1 -- 172.21.15.83:0/2496870339 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+6543 (secure 0 0 0) 0x7f1d08055fb0 con 0x7f1d100c8de0 2024-09-19T07:45:23.345 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:23.345 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":180,"btime":"2024-09-19T07:45:22:927687+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7968,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:23.345 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 180 2024-09-19T07:45:23.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.349+0000 7f1d16de6640 1 -- 172.21.15.83:0/2496870339 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1cf4067940 msgr2=0x7f1cf4069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:23.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.349+0000 7f1d16de6640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1cf4067940 0x7f1cf4069e00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1d040ae000 tx=0x7f1d040a7000 comp rx=0 tx=0).stop 2024-09-19T07:45:23.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.349+0000 7f1d16de6640 1 -- 172.21.15.83:0/2496870339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d100c8de0 msgr2=0x7f1d101431d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:23.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.349+0000 7f1d16de6640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d100c8de0 0x7f1d101431d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800c770 tx=0x7f1d0800cc30 comp rx=0 tx=0).stop 2024-09-19T07:45:23.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.349+0000 7f1d16de6640 1 -- 172.21.15.83:0/2496870339 shutdown_connections 2024-09-19T07:45:23.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.349+0000 7f1d16de6640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1cf4067940 0x7f1cf4069e00 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:23.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.349+0000 7f1d16de6640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d100c8de0 0x7f1d101431d0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:23.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.349+0000 7f1d16de6640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d1014dbe0 0x7f1d10142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:23.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.349+0000 7f1d16de6640 1 --2- 172.21.15.83:0/2496870339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d100c44e0 0x7f1d100c2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:23.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.349+0000 7f1d16de6640 1 -- 172.21.15.83:0/2496870339 >> 172.21.15.83:0/2496870339 conn(0x7f1d100c95a0 msgr2=0x7f1d100abb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:23.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.349+0000 7f1d16de6640 1 -- 172.21.15.83:0/2496870339 shutdown_connections 2024-09-19T07:45:23.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:23.349+0000 7f1d16de6640 1 -- 172.21.15.83:0/2496870339 wait complete. 2024-09-19T07:45:23.373 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:45:24.656 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:24.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f25456e1640 1 Processor -- start 2024-09-19T07:45:24.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f25456e1640 1 -- start start 2024-09-19T07:45:24.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f25456e1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f25400ac290 0x7f25400c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:24.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f25456e1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f254006e110 con 0x7f25400c3a50 2024-09-19T07:45:24.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f25456e1640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25400c4990 con 0x7f25400c3130 2024-09-19T07:45:24.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f25456e1640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f25400ac700 con 0x7f25400ac290 2024-09-19T07:45:24.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f253f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f25400c3a50 0x7f25401563a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47112/0 (socket says 172.21.15.83:47112) 2024-09-19T07:45:24.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f253f7fe640 1 -- 172.21.15.83:0/1618818383 learned_addr learned my addr 172.21.15.83:0/1618818383 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:45:24.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f25367fc640 1 --2- 172.21.15.83:0/1618818383 >> v2:172.21.15.83:3300/0 conn(0x7f25400ac290 0x7f25400c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:24.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f253e7fc640 1 -- 172.21.15.83:0/1618818383 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1530938522 0 0) 0x7f254006e110 con 0x7f25400c3a50 2024-09-19T07:45:24.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f253e7fc640 1 -- 172.21.15.83:0/1618818383 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f252c003530 con 0x7f25400c3a50 2024-09-19T07:45:24.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f25367fc640 1 -- 172.21.15.83:0/1618818383 >> v1:172.21.15.184:6789/0 conn(0x7f25400c3130 legacy=0x7f254006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:24.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f25367fc640 1 -- 172.21.15.83:0/1618818383 >> v1:172.21.15.83:6789/0 conn(0x7f25400c3a50 legacy=0x7f25401563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:24.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.801+0000 7f25367fc640 1 -- 172.21.15.83:0/1618818383 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25400c4c70 con 0x7f25400ac290 2024-09-19T07:45:24.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f25367fc640 1 --2- 172.21.15.83:0/1618818383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25400ac290 0x7f25400c3510 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f25280097c0 tx=0x7f252801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cec490585abade1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:24.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f253e7fc640 1 -- 172.21.15.83:0/1618818383 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25280b0070 con 0x7f25400ac290 2024-09-19T07:45:24.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f253e7fc640 1 -- 172.21.15.83:0/1618818383 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25280ab600 con 0x7f25400ac290 2024-09-19T07:45:24.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f253e7fc640 1 -- 172.21.15.83:0/1618818383 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25280ab9d0 con 0x7f25400ac290 2024-09-19T07:45:24.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f25456e1640 1 -- 172.21.15.83:0/1618818383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25400ac290 msgr2=0x7f25400c3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:24.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f25456e1640 1 --2- 172.21.15.83:0/1618818383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25400ac290 0x7f25400c3510 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f25280097c0 tx=0x7f252801c3d0 comp rx=0 tx=0).stop 2024-09-19T07:45:24.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f25456e1640 1 -- 172.21.15.83:0/1618818383 shutdown_connections 2024-09-19T07:45:24.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f25456e1640 1 --2- 172.21.15.83:0/1618818383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25400ac290 0x7f25400c3510 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:24.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f25456e1640 1 -- 172.21.15.83:0/1618818383 >> 172.21.15.83:0/1618818383 conn(0x7f25400c5060 msgr2=0x7f25400c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:24.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f25456e1640 1 -- 172.21.15.83:0/1618818383 shutdown_connections 2024-09-19T07:45:24.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f25456e1640 1 -- 172.21.15.83:0/1618818383 wait complete. 2024-09-19T07:45:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f25456e1640 1 Processor -- start 2024-09-19T07:45:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.805+0000 7f25456e1640 1 -- start start 2024-09-19T07:45:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f25456e1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25400ac290 0x7f25401ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f25456e1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25400c3130 0x7f254014dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:24.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f25456e1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25400c3a50 0x7f25401482c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:24.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f25456e1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f25400ad520 con 0x7f25400ac290 2024-09-19T07:45:24.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f25456e1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f25400c82a0 con 0x7f25400c3130 2024-09-19T07:45:24.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f25456e1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f25400ad6a0 con 0x7f25400c3a50 2024-09-19T07:45:24.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f253effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25400ac290 0x7f25401ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:24.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f253effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25400ac290 0x7f25401ef0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55316/0 (socket says 172.21.15.83:55316) 2024-09-19T07:45:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f253effd640 1 -- 172.21.15.83:0/3425540789 learned_addr learned my addr 172.21.15.83:0/3425540789 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f253effd640 1 -- 172.21.15.83:0/3425540789 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25400c3a50 msgr2=0x7f25401482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f253effd640 1 --2- 172.21.15.83:0/3425540789 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25400c3a50 0x7f25401482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f253effd640 1 -- 172.21.15.83:0/3425540789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25400c3130 msgr2=0x7f254014dee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:45:24.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f253effd640 1 --2- 172.21.15.83:0/3425540789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25400c3130 0x7f254014dee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:24.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f253effd640 1 -- 172.21.15.83:0/3425540789 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25401ee630 con 0x7f25400ac290 2024-09-19T07:45:24.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f253effd640 1 --2- 172.21.15.83:0/3425540789 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25400ac290 0x7f25401ef0f0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f253000c4c0 tx=0x7f2530001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f2537fff640 1 -- 172.21.15.83:0/3425540789 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2530017070 con 0x7f25400ac290 2024-09-19T07:45:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f2537fff640 1 -- 172.21.15.83:0/3425540789 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2530003600 con 0x7f25400ac290 2024-09-19T07:45:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f2537fff640 1 -- 172.21.15.83:0/3425540789 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25300039f0 con 0x7f25400ac290 2024-09-19T07:45:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f25456e1640 1 -- 172.21.15.83:0/3425540789 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2540148c60 con 0x7f25400ac290 2024-09-19T07:45:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f25456e1640 1 -- 172.21.15.83:0/3425540789 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25401ee8f0 con 0x7f25400ac290 2024-09-19T07:45:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.809+0000 7f25456e1640 1 -- 172.21.15.83:0/3425540789 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25400a8100 con 0x7f25400ac290 2024-09-19T07:45:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.813+0000 7f2537fff640 1 -- 172.21.15.83:0/3425540789 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f253001a070 con 0x7f25400ac290 2024-09-19T07:45:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.813+0000 7f2537fff640 1 --2- 172.21.15.83:0/3425540789 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2520067910 0x7f2520069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.813+0000 7f2537fff640 1 -- 172.21.15.83:0/3425540789 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9647+0+0 (secure 0 0 0) 0x7f2530086360 con 0x7f25400ac290 2024-09-19T07:45:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.813+0000 7f25367fc640 1 --2- 172.21.15.83:0/3425540789 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2520067910 0x7f2520069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.813+0000 7f25367fc640 1 --2- 172.21.15.83:0/3425540789 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2520067910 0x7f2520069dd0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f25280ad000 tx=0x7f2528008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:24.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:24.817+0000 7f2537fff640 1 -- 172.21.15.83:0/3425540789 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2530021020 con 0x7f25400ac290 2024-09-19T07:45:25.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.105+0000 7f25456e1640 1 -- 172.21.15.83:0/3425540789 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25400c3510 con 0x7f25400ac290 2024-09-19T07:45:25.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.109+0000 7f2537fff640 1 -- 172.21.15.83:0/3425540789 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+6543 (secure 0 0 0) 0x7f2530056000 con 0x7f25400ac290 2024-09-19T07:45:25.107 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:25.107 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":180,"btime":"2024-09-19T07:45:22:927687+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7968,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:25.108 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 180 2024-09-19T07:45:25.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.109+0000 7f25456e1640 1 -- 172.21.15.83:0/3425540789 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2520067910 msgr2=0x7f2520069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:25.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.109+0000 7f25456e1640 1 --2- 172.21.15.83:0/3425540789 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2520067910 0x7f2520069dd0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f25280ad000 tx=0x7f2528008040 comp rx=0 tx=0).stop 2024-09-19T07:45:25.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.109+0000 7f25456e1640 1 -- 172.21.15.83:0/3425540789 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25400ac290 msgr2=0x7f25401ef0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:25.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.109+0000 7f25456e1640 1 --2- 172.21.15.83:0/3425540789 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25400ac290 0x7f25401ef0f0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f253000c4c0 tx=0x7f2530001e80 comp rx=0 tx=0).stop 2024-09-19T07:45:25.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.109+0000 7f25456e1640 1 -- 172.21.15.83:0/3425540789 shutdown_connections 2024-09-19T07:45:25.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.109+0000 7f25456e1640 1 --2- 172.21.15.83:0/3425540789 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2520067910 0x7f2520069dd0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.109+0000 7f25456e1640 1 --2- 172.21.15.83:0/3425540789 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25400c3a50 0x7f25401482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.109+0000 7f25456e1640 1 --2- 172.21.15.83:0/3425540789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25400c3130 0x7f254014dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.109+0000 7f25456e1640 1 --2- 172.21.15.83:0/3425540789 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25400ac290 0x7f25401ef0f0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.109+0000 7f25456e1640 1 -- 172.21.15.83:0/3425540789 >> 172.21.15.83:0/3425540789 conn(0x7f25400c5060 msgr2=0x7f254006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:25.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.113+0000 7f25456e1640 1 -- 172.21.15.83:0/3425540789 shutdown_connections 2024-09-19T07:45:25.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.113+0000 7f25456e1640 1 -- 172.21.15.83:0/3425540789 wait complete. 2024-09-19T07:45:25.133 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-19T07:45:25.133 INFO:tasks.mds_thrash.fs.[b]:waiting for 86 secs before thrashing 2024-09-19T07:45:25.374 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:25.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652ee49640 1 Processor -- start 2024-09-19T07:45:25.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652ee49640 1 -- start start 2024-09-19T07:45:25.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652ee49640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65280c3130 0x7f65280ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:25.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652ee49640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f652806da00 0x7f65280c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:25.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652ee49640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f65280c3a50 0x7f65280c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:25.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652ee49640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f65280c4990 con 0x7f65280c3a50 2024-09-19T07:45:25.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652ee49640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f65280c4c70 con 0x7f652806da00 2024-09-19T07:45:25.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652ee49640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f65280ab7b0 con 0x7f65280c3130 2024-09-19T07:45:25.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652d3e6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f65280c3a50 0x7f65280c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:25.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652d3e6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f65280c3a50 0x7f65280c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55330/0 (socket says 172.21.15.83:55330) 2024-09-19T07:45:25.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652d3e6640 1 -- 172.21.15.83:0/887627757 learned_addr learned my addr 172.21.15.83:0/887627757 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:25.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652cbe5640 1 --2- 172.21.15.83:0/887627757 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65280c3130 0x7f65280ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:25.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652d3e6640 1 -- 172.21.15.83:0/887627757 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65280c3130 msgr2=0x7f65280ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:25.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652d3e6640 1 --2- 172.21.15.83:0/887627757 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65280c3130 0x7f65280ac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652d3e6640 1 -- 172.21.15.83:0/887627757 >> v2:172.21.15.184:3300/0 conn(0x7f652806da00 msgr2=0x7f65280c3510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:45:25.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652d3e6640 1 --2- 172.21.15.83:0/887627757 >> v2:172.21.15.184:3300/0 conn(0x7f652806da00 0x7f65280c3510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.517+0000 7f652d3e6640 1 -- 172.21.15.83:0/887627757 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65280aba90 con 0x7f65280c3a50 2024-09-19T07:45:25.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f652d3e6640 1 --2- 172.21.15.83:0/887627757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65280c3a50 0x7f65280c6460 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f65280ab3d0 tx=0x7f652001eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359399460419da7d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:25.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f65277fe640 1 -- 172.21.15.83:0/887627757 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652000b2d0 con 0x7f65280c3a50 2024-09-19T07:45:25.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f65277fe640 1 -- 172.21.15.83:0/887627757 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f652000b470 con 0x7f65280c3a50 2024-09-19T07:45:25.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f65277fe640 1 -- 172.21.15.83:0/887627757 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65200a7b40 con 0x7f65280c3a50 2024-09-19T07:45:25.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f652ee49640 1 -- 172.21.15.83:0/887627757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65280c3a50 msgr2=0x7f65280c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:25.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f652ee49640 1 --2- 172.21.15.83:0/887627757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65280c3a50 0x7f65280c6460 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f65280ab3d0 tx=0x7f652001eba0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f652ee49640 1 -- 172.21.15.83:0/887627757 shutdown_connections 2024-09-19T07:45:25.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f652ee49640 1 --2- 172.21.15.83:0/887627757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65280c3a50 0x7f65280c6460 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f652ee49640 1 --2- 172.21.15.83:0/887627757 >> v2:172.21.15.184:3300/0 conn(0x7f652806da00 0x7f65280c3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f652ee49640 1 --2- 172.21.15.83:0/887627757 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65280c3130 0x7f65280ac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f652ee49640 1 -- 172.21.15.83:0/887627757 >> 172.21.15.83:0/887627757 conn(0x7f65280c5060 msgr2=0x7f65280c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:25.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f652ee49640 1 -- 172.21.15.83:0/887627757 shutdown_connections 2024-09-19T07:45:25.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f652ee49640 1 -- 172.21.15.83:0/887627757 wait complete. 2024-09-19T07:45:25.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f652ee49640 1 Processor -- start 2024-09-19T07:45:25.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.521+0000 7f652ee49640 1 -- start start 2024-09-19T07:45:25.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f652ee49640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f652806da00 0x7f652815be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:25.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f652ee49640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65280c3130 0x7f65281f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:25.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f652ee49640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280c3a50 0x7f65281f2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:25.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f652ee49640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f65281563a0 con 0x7f65280c3130 2024-09-19T07:45:25.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f652ee49640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f65280ad730 con 0x7f65280c3a50 2024-09-19T07:45:25.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f652ee49640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6528156520 con 0x7f652806da00 2024-09-19T07:45:25.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f6527fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65280c3130 0x7f65281f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:25.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f6527fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65280c3130 0x7f65281f2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55344/0 (socket says 172.21.15.83:55344) 2024-09-19T07:45:25.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f6527fff640 1 -- 172.21.15.83:0/11985357 learned_addr learned my addr 172.21.15.83:0/11985357 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:25.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f652d3e6640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280c3a50 0x7f65281f2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:25.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f652cbe5640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f652806da00 0x7f652815be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:25.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f6527fff640 1 -- 172.21.15.83:0/11985357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f652806da00 msgr2=0x7f652815be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:25.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f6527fff640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f652806da00 0x7f652815be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f6527fff640 1 -- 172.21.15.83:0/11985357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280c3a50 msgr2=0x7f65281f2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:25.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f6527fff640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280c3a50 0x7f65281f2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f6527fff640 1 -- 172.21.15.83:0/11985357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65281f1f50 con 0x7f65280c3130 2024-09-19T07:45:25.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f6527fff640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65280c3130 0x7f65281f2400 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f651400b4a0 tx=0x7f651400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:25.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f65257fa640 1 -- 172.21.15.83:0/11985357 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514014070 con 0x7f65280c3130 2024-09-19T07:45:25.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f65257fa640 1 -- 172.21.15.83:0/11985357 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6514010610 con 0x7f65280c3130 2024-09-19T07:45:25.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f65257fa640 1 -- 172.21.15.83:0/11985357 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514010a00 con 0x7f65280c3130 2024-09-19T07:45:25.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f652ee49640 1 -- 172.21.15.83:0/11985357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6528072270 con 0x7f65280c3130 2024-09-19T07:45:25.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.525+0000 7f652ee49640 1 -- 172.21.15.83:0/11985357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f652815c380 con 0x7f65280c3130 2024-09-19T07:45:25.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.529+0000 7f65257fa640 1 -- 172.21.15.83:0/11985357 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6514017070 con 0x7f65280c3130 2024-09-19T07:45:25.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.529+0000 7f6512ffd640 1 -- 172.21.15.83:0/11985357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65280c3510 con 0x7f65280c3130 2024-09-19T07:45:25.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.529+0000 7f65257fa640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f650c067990 0x7f650c069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:25.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.529+0000 7f65257fa640 1 -- 172.21.15.83:0/11985357 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9647+0+0 (secure 0 0 0) 0x7f6514086560 con 0x7f65280c3130 2024-09-19T07:45:25.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.529+0000 7f652cbe5640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f650c067990 0x7f650c069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:25.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.529+0000 7f652cbe5640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f650c067990 0x7f650c069e50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f651c007a60 tx=0x7f651c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:25.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.533+0000 7f65257fa640 1 -- 172.21.15.83:0/11985357 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f6514055b50 con 0x7f65280c3130 2024-09-19T07:45:25.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.801+0000 7f6512ffd640 1 -- 172.21.15.83:0/11985357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65280c55e0 con 0x7f65280c3130 2024-09-19T07:45:25.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.801+0000 7f65257fa640 1 -- 172.21.15.83:0/11985357 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+6543 (secure 0 0 0) 0x7f6514059e50 con 0x7f65280c3130 2024-09-19T07:45:25.803 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:25.803 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":180,"btime":"2024-09-19T07:45:22:927687+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7968,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:25.803 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 180 2024-09-19T07:45:25.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.805+0000 7f6512ffd640 1 -- 172.21.15.83:0/11985357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f650c067990 msgr2=0x7f650c069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:25.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.805+0000 7f6512ffd640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f650c067990 0x7f650c069e50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f651c007a60 tx=0x7f651c008040 comp rx=0 tx=0).stop 2024-09-19T07:45:25.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.805+0000 7f6512ffd640 1 -- 172.21.15.83:0/11985357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65280c3130 msgr2=0x7f65281f2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:25.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.805+0000 7f6512ffd640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65280c3130 0x7f65281f2400 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f651400b4a0 tx=0x7f651400b960 comp rx=0 tx=0).stop 2024-09-19T07:45:25.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.805+0000 7f6512ffd640 1 -- 172.21.15.83:0/11985357 shutdown_connections 2024-09-19T07:45:25.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.805+0000 7f6512ffd640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f650c067990 0x7f650c069e50 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.805+0000 7f6512ffd640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65280c3a50 0x7f65281f2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.805+0000 7f6512ffd640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65280c3130 0x7f65281f2400 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.805+0000 7f6512ffd640 1 --2- 172.21.15.83:0/11985357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f652806da00 0x7f652815be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:25.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.805+0000 7f6512ffd640 1 -- 172.21.15.83:0/11985357 >> 172.21.15.83:0/11985357 conn(0x7f65280c5060 msgr2=0x7f6528156900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:25.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.805+0000 7f6512ffd640 1 -- 172.21.15.83:0/11985357 shutdown_connections 2024-09-19T07:45:25.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:25.805+0000 7f6512ffd640 1 -- 172.21.15.83:0/11985357 wait complete. 2024-09-19T07:45:25.825 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:45:27.826 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:27.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa11c877640 1 Processor -- start 2024-09-19T07:45:27.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa11c877640 1 -- start start 2024-09-19T07:45:27.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa11c877640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fa118153300 0x7fa11806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:27.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa11c877640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa118155bb0 0x7fa118156020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:27.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa11c877640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1181593f0 con 0x7fa11806d1d0 2024-09-19T07:45:27.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa11c877640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa1181591a0 con 0x7fa118155bb0 2024-09-19T07:45:27.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa11c877640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fa1181596d0 con 0x7fa118153300 2024-09-19T07:45:27.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa10ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa11806d1d0 0x7fa1180c23d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47118/0 (socket says 172.21.15.83:47118) 2024-09-19T07:45:27.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa10ffff640 1 -- 172.21.15.83:0/2758633721 learned_addr learned my addr 172.21.15.83:0/2758633721 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:45:27.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa116d9d640 1 --2- 172.21.15.83:0/2758633721 >> v2:172.21.15.83:3300/0 conn(0x7fa118155bb0 0x7fa118156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:27.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa116d9d640 1 -- 172.21.15.83:0/2758633721 >> v2:172.21.15.184:3300/0 conn(0x7fa118153300 msgr2=0x7fa11806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:27.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa116d9d640 1 --2- 172.21.15.83:0/2758633721 >> v2:172.21.15.184:3300/0 conn(0x7fa118153300 0x7fa11806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:27.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa115d9b640 1 -- 172.21.15.83:0/2758633721 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3702600647 0 0) 0x7fa1181593f0 con 0x7fa11806d1d0 2024-09-19T07:45:27.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa116d9d640 1 -- 172.21.15.83:0/2758633721 >> v1:172.21.15.83:6789/0 conn(0x7fa11806d1d0 legacy=0x7fa1180c23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:27.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.977+0000 7fa116d9d640 1 -- 172.21.15.83:0/2758633721 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1181599b0 con 0x7fa118155bb0 2024-09-19T07:45:27.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.981+0000 7fa116d9d640 1 --2- 172.21.15.83:0/2758633721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa118155bb0 0x7fa118156020 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fa1180c2de0 tx=0x7fa11002d840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f7cb7b48c3667f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:27.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.981+0000 7fa115d9b640 1 -- 172.21.15.83:0/2758633721 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1100c3070 con 0x7fa118155bb0 2024-09-19T07:45:27.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.981+0000 7fa115d9b640 1 -- 172.21.15.83:0/2758633721 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1100b6800 con 0x7fa118155bb0 2024-09-19T07:45:27.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.981+0000 7fa115d9b640 1 -- 172.21.15.83:0/2758633721 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1100b6bd0 con 0x7fa118155bb0 2024-09-19T07:45:27.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.981+0000 7fa11c877640 1 -- 172.21.15.83:0/2758633721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa118155bb0 msgr2=0x7fa118156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:27.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.981+0000 7fa11c877640 1 --2- 172.21.15.83:0/2758633721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa118155bb0 0x7fa118156020 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fa1180c2de0 tx=0x7fa11002d840 comp rx=0 tx=0).stop 2024-09-19T07:45:27.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.981+0000 7fa11c877640 1 -- 172.21.15.83:0/2758633721 shutdown_connections 2024-09-19T07:45:27.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.981+0000 7fa11c877640 1 --2- 172.21.15.83:0/2758633721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa118155bb0 0x7fa118156020 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:27.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.981+0000 7fa11c877640 1 --2- 172.21.15.83:0/2758633721 >> v2:172.21.15.184:3300/0 conn(0x7fa118153300 0x7fa11806cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:27.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.981+0000 7fa11c877640 1 -- 172.21.15.83:0/2758633721 >> 172.21.15.83:0/2758633721 conn(0x7fa1180c5090 msgr2=0x7fa1180c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:27.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11c877640 1 -- 172.21.15.83:0/2758633721 shutdown_connections 2024-09-19T07:45:27.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11c877640 1 -- 172.21.15.83:0/2758633721 wait complete. 2024-09-19T07:45:27.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11c877640 1 Processor -- start 2024-09-19T07:45:27.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11c877640 1 -- start start 2024-09-19T07:45:27.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11c877640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa11806d1d0 0x7fa1180c63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:27.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11c877640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa118153300 0x7fa118156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:27.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11c877640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa118155bb0 0x7fa1181574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:27.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11c877640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa118153860 con 0x7fa11806d1d0 2024-09-19T07:45:27.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11c877640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181536e0 con 0x7fa118155bb0 2024-09-19T07:45:27.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11c877640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa1180bfdd0 con 0x7fa118153300 2024-09-19T07:45:27.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa11806d1d0 0x7fa1180c63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:27.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa11806d1d0 0x7fa1180c63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55360/0 (socket says 172.21.15.83:55360) 2024-09-19T07:45:27.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11659c640 1 -- 172.21.15.83:0/1136976250 learned_addr learned my addr 172.21.15.83:0/1136976250 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:27.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa10ffff640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa118153300 0x7fa118156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:27.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa116d9d640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa118155bb0 0x7fa1181574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:27.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11659c640 1 -- 172.21.15.83:0/1136976250 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa118153300 msgr2=0x7fa118156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:27.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11659c640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa118153300 0x7fa118156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:27.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11659c640 1 -- 172.21.15.83:0/1136976250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa118155bb0 msgr2=0x7fa1181574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:27.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11659c640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa118155bb0 0x7fa1181574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:27.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.985+0000 7fa11659c640 1 -- 172.21.15.83:0/1136976250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa118154190 con 0x7fa11806d1d0 2024-09-19T07:45:27.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.989+0000 7fa11659c640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa11806d1d0 0x7fa1180c63e0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fa10800b550 tx=0x7fa10800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:27.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.989+0000 7fa10effd640 1 -- 172.21.15.83:0/1136976250 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa108014070 con 0x7fa11806d1d0 2024-09-19T07:45:27.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.989+0000 7fa10effd640 1 -- 172.21.15.83:0/1136976250 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa108010890 con 0x7fa11806d1d0 2024-09-19T07:45:27.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.989+0000 7fa11c877640 1 -- 172.21.15.83:0/1136976250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa118154430 con 0x7fa11806d1d0 2024-09-19T07:45:27.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.989+0000 7fa10effd640 1 -- 172.21.15.83:0/1136976250 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa108010c80 con 0x7fa11806d1d0 2024-09-19T07:45:27.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.989+0000 7fa11c877640 1 -- 172.21.15.83:0/1136976250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1181549f0 con 0x7fa11806d1d0 2024-09-19T07:45:27.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.989+0000 7fa11c877640 1 -- 172.21.15.83:0/1136976250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa11806cc90 con 0x7fa11806d1d0 2024-09-19T07:45:27.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.993+0000 7fa10effd640 1 -- 172.21.15.83:0/1136976250 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa108017070 con 0x7fa11806d1d0 2024-09-19T07:45:27.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.993+0000 7fa10effd640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa0f80678c0 0x7fa0f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:27.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.993+0000 7fa10ffff640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa0f80678c0 0x7fa0f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:27.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.993+0000 7fa10effd640 1 -- 172.21.15.83:0/1136976250 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9647+0+0 (secure 0 0 0) 0x7fa108086910 con 0x7fa11806d1d0 2024-09-19T07:45:27.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:27.993+0000 7fa10ffff640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa0f80678c0 0x7fa0f8069d80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa11814dcf0 tx=0x7fa100007240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:28.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.001+0000 7fa10effd640 1 -- 172.21.15.83:0/1136976250 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa108055ee0 con 0x7fa11806d1d0 2024-09-19T07:45:28.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.289+0000 7fa11c877640 1 -- 172.21.15.83:0/1136976250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1180c57f0 con 0x7fa11806d1d0 2024-09-19T07:45:28.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.289+0000 7fa10effd640 1 -- 172.21.15.83:0/1136976250 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+6543 (secure 0 0 0) 0x7fa108010e20 con 0x7fa11806d1d0 2024-09-19T07:45:28.291 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:28.291 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":180,"btime":"2024-09-19T07:45:22:927687+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":7968,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:44:21.648179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7803":{"gid":7803,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":25,"addr":"172.21.15.83:6837/3490578864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3490578864},{"type":"v1","addr":"172.21.15.83:6837","nonce":3490578864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7803,"qdb_cluster":[7803]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:28.291 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 180 2024-09-19T07:45:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.293+0000 7fa11c877640 1 -- 172.21.15.83:0/1136976250 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa0f80678c0 msgr2=0x7fa0f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.293+0000 7fa11c877640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa0f80678c0 0x7fa0f8069d80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa11814dcf0 tx=0x7fa100007240 comp rx=0 tx=0).stop 2024-09-19T07:45:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.293+0000 7fa11c877640 1 -- 172.21.15.83:0/1136976250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa11806d1d0 msgr2=0x7fa1180c63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.293+0000 7fa11c877640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa11806d1d0 0x7fa1180c63e0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fa10800b550 tx=0x7fa10800ba10 comp rx=0 tx=0).stop 2024-09-19T07:45:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.293+0000 7fa11c877640 1 -- 172.21.15.83:0/1136976250 shutdown_connections 2024-09-19T07:45:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.293+0000 7fa11c877640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa0f80678c0 0x7fa0f8069d80 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.293+0000 7fa11c877640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa118155bb0 0x7fa1181574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.293+0000 7fa11c877640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa118153300 0x7fa118156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.293+0000 7fa11c877640 1 --2- 172.21.15.83:0/1136976250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa11806d1d0 0x7fa1180c63e0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.293+0000 7fa11c877640 1 -- 172.21.15.83:0/1136976250 >> 172.21.15.83:0/1136976250 conn(0x7fa1180c5090 msgr2=0x7fa118151730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.293+0000 7fa11c877640 1 -- 172.21.15.83:0/1136976250 shutdown_connections 2024-09-19T07:45:28.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:28.293+0000 7fa11c877640 1 -- 172.21.15.83:0/1136976250 wait complete. 2024-09-19T07:45:28.318 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:45:30.319 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:30.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.469+0000 7f25d39e3640 1 Processor -- start 2024-09-19T07:45:30.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.469+0000 7f25d39e3640 1 -- start start 2024-09-19T07:45:30.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.469+0000 7f25d39e3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25cc153310 0x7f25cc06cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:30.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.469+0000 7f25d39e3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f25cc06d100 0x7f25cc0c4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:30.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.469+0000 7f25d39e3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25cc06ef10 con 0x7f25cc0c46d0 2024-09-19T07:45:30.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.469+0000 7f25d39e3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f25cc06c3a0 con 0x7f25cc06d100 2024-09-19T07:45:30.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.469+0000 7f25d39e3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f25cc06c680 con 0x7f25cc153310 2024-09-19T07:45:30.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.469+0000 7f25d1f80640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f25cc0c46d0 0x7f25cc074f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55910/0 (socket says 172.21.15.83:55910) 2024-09-19T07:45:30.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.469+0000 7f25d1f80640 1 -- 172.21.15.83:0/644055013 learned_addr learned my addr 172.21.15.83:0/644055013 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:45:30.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.469+0000 7f25d177f640 1 --2- 172.21.15.83:0/644055013 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25cc153310 0x7f25cc06cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:30.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.473+0000 7f25d0f7e640 1 --2- 172.21.15.83:0/644055013 >> v2:172.21.15.184:3300/0 conn(0x7f25cc06d100 0x7f25cc0c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.473+0000 7f25d177f640 1 -- 172.21.15.83:0/644055013 >> v2:172.21.15.184:3300/0 conn(0x7f25cc06d100 msgr2=0x7f25cc0c4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.473+0000 7f25d177f640 1 --2- 172.21.15.83:0/644055013 >> v2:172.21.15.184:3300/0 conn(0x7f25cc06d100 0x7f25cc0c4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.473+0000 7f25d177f640 1 -- 172.21.15.83:0/644055013 >> v1:172.21.15.83:6789/0 conn(0x7f25cc0c46d0 legacy=0x7f25cc074f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.473+0000 7f25d177f640 1 -- 172.21.15.83:0/644055013 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25cc06f1f0 con 0x7f25cc153310 2024-09-19T07:45:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.473+0000 7f25c3fff640 1 -- 172.21.15.83:0/644055013 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1959707272 0 0) 0x7f25cc06ef10 con 0x7f25cc0c46d0 2024-09-19T07:45:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.473+0000 7f25d0f7e640 1 --2- 172.21.15.83:0/644055013 >> v2:172.21.15.184:3300/0 conn(0x7f25cc06d100 0x7f25cc0c4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:45:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.477+0000 7f25d177f640 1 --2- 172.21.15.83:0/644055013 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25cc153310 0x7f25cc06cb10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f25c80097c0 tx=0x7f25c801c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdc4dfa1395b6b18 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.477+0000 7f25c3fff640 1 -- 172.21.15.83:0/644055013 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c80b0070 con 0x7f25cc153310 2024-09-19T07:45:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.477+0000 7f25c3fff640 1 -- 172.21.15.83:0/644055013 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25c80ab600 con 0x7f25cc153310 2024-09-19T07:45:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.477+0000 7f25c3fff640 1 -- 172.21.15.83:0/644055013 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c80ab9d0 con 0x7f25cc153310 2024-09-19T07:45:30.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.477+0000 7f25d39e3640 1 -- 172.21.15.83:0/644055013 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25cc153310 msgr2=0x7f25cc06cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:30.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.477+0000 7f25d39e3640 1 --2- 172.21.15.83:0/644055013 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25cc153310 0x7f25cc06cb10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f25c80097c0 tx=0x7f25c801c380 comp rx=0 tx=0).stop 2024-09-19T07:45:30.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.477+0000 7f25d39e3640 1 -- 172.21.15.83:0/644055013 shutdown_connections 2024-09-19T07:45:30.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.477+0000 7f25d39e3640 1 --2- 172.21.15.83:0/644055013 >> v2:172.21.15.184:3300/0 conn(0x7f25cc06d100 0x7f25cc0c4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.477+0000 7f25d39e3640 1 --2- 172.21.15.83:0/644055013 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25cc153310 0x7f25cc06cb10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.477+0000 7f25d39e3640 1 -- 172.21.15.83:0/644055013 >> 172.21.15.83:0/644055013 conn(0x7f25cc0c90e0 msgr2=0x7f25cc0aa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:30.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.477+0000 7f25d39e3640 1 -- 172.21.15.83:0/644055013 shutdown_connections 2024-09-19T07:45:30.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.477+0000 7f25d39e3640 1 -- 172.21.15.83:0/644055013 wait complete. 2024-09-19T07:45:30.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d39e3640 1 Processor -- start 2024-09-19T07:45:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d39e3640 1 -- start start 2024-09-19T07:45:30.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d39e3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25cc06d100 0x7f25cc0c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:30.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d39e3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc0c46d0 0x7f25cc0c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:30.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d39e3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25cc153310 0x7f25cc0730e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d39e3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc160e80 con 0x7f25cc0c46d0 2024-09-19T07:45:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d39e3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc160d00 con 0x7f25cc06d100 2024-09-19T07:45:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d39e3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f25cc161000 con 0x7f25cc153310 2024-09-19T07:45:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d0f7e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc0c46d0 0x7f25cc0c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d0f7e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc0c46d0 0x7f25cc0c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42838/0 (socket says 172.21.15.83:42838) 2024-09-19T07:45:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d0f7e640 1 -- 172.21.15.83:0/3894486594 learned_addr learned my addr 172.21.15.83:0/3894486594 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d1f80640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25cc153310 0x7f25cc0730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d177f640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25cc06d100 0x7f25cc0c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d0f7e640 1 -- 172.21.15.83:0/3894486594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25cc153310 msgr2=0x7f25cc0730e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d0f7e640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25cc153310 0x7f25cc0730e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d0f7e640 1 -- 172.21.15.83:0/3894486594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25cc06d100 msgr2=0x7f25cc0c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d0f7e640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25cc06d100 0x7f25cc0c4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d0f7e640 1 -- 172.21.15.83:0/3894486594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25cc0c5040 con 0x7f25cc0c46d0 2024-09-19T07:45:30.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d0f7e640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc0c46d0 0x7f25cc0c73c0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00e760 tx=0x7f25bc00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:30.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25c1ffb640 1 -- 172.21.15.83:0/3894486594 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25bc017070 con 0x7f25cc0c46d0 2024-09-19T07:45:30.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d39e3640 1 -- 172.21.15.83:0/3894486594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25cc0c7900 con 0x7f25cc0c46d0 2024-09-19T07:45:30.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25c1ffb640 1 -- 172.21.15.83:0/3894486594 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25bc00c8d0 con 0x7f25cc0c46d0 2024-09-19T07:45:30.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25c1ffb640 1 -- 172.21.15.83:0/3894486594 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25bc00f040 con 0x7f25cc0c46d0 2024-09-19T07:45:30.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d39e3640 1 -- 172.21.15.83:0/3894486594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25cc0c6ad0 con 0x7f25cc0c46d0 2024-09-19T07:45:30.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.481+0000 7f25d39e3640 1 -- 172.21.15.83:0/3894486594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25cc0c4190 con 0x7f25cc0c46d0 2024-09-19T07:45:30.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.485+0000 7f25c1ffb640 1 -- 172.21.15.83:0/3894486594 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f25bc01a070 con 0x7f25cc0c46d0 2024-09-19T07:45:30.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.485+0000 7f25c1ffb640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f25b00678c0 0x7f25b0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:30.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.485+0000 7f25c1ffb640 1 -- 172.21.15.83:0/3894486594 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9733+0+0 (secure 0 0 0) 0x7f25bc086810 con 0x7f25cc0c46d0 2024-09-19T07:45:30.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.485+0000 7f25d177f640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f25b00678c0 0x7f25b0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:30.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.485+0000 7f25d177f640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f25b00678c0 0x7f25b0069d80 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f25c80ad000 tx=0x7f25c8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:30.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.493+0000 7f25c1ffb640 1 -- 172.21.15.83:0/3894486594 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f25bc055d00 con 0x7f25cc0c46d0 2024-09-19T07:45:30.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.781+0000 7f25d39e3640 1 -- 172.21.15.83:0/3894486594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25cc154850 con 0x7f25cc0c46d0 2024-09-19T07:45:30.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.781+0000 7f25c1ffb640 1 -- 172.21.15.83:0/3894486594 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) ==== 95+0+5709 (secure 0 0 0) 0x7f25bc05a000 con 0x7f25cc0c46d0 2024-09-19T07:45:30.780 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:30.780 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":183,"btime":"2024-09-19T07:45:30:600551+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:29.605751+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:30.780 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 183 2024-09-19T07:45:30.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.781+0000 7f25d39e3640 1 -- 172.21.15.83:0/3894486594 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f25b00678c0 msgr2=0x7f25b0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.781+0000 7f25d39e3640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f25b00678c0 0x7f25b0069d80 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f25c80ad000 tx=0x7f25c8008040 comp rx=0 tx=0).stop 2024-09-19T07:45:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.781+0000 7f25d39e3640 1 -- 172.21.15.83:0/3894486594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc0c46d0 msgr2=0x7f25cc0c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.781+0000 7f25d39e3640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc0c46d0 0x7f25cc0c73c0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00e760 tx=0x7f25bc00ec20 comp rx=0 tx=0).stop 2024-09-19T07:45:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.781+0000 7f25d39e3640 1 -- 172.21.15.83:0/3894486594 shutdown_connections 2024-09-19T07:45:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.785+0000 7f25d39e3640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f25b00678c0 0x7f25b0069d80 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.785+0000 7f25d39e3640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25cc153310 0x7f25cc0730e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.785+0000 7f25d39e3640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc0c46d0 0x7f25cc0c73c0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.785+0000 7f25d39e3640 1 --2- 172.21.15.83:0/3894486594 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25cc06d100 0x7f25cc0c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.785+0000 7f25d39e3640 1 -- 172.21.15.83:0/3894486594 >> 172.21.15.83:0/3894486594 conn(0x7f25cc0c90e0 msgr2=0x7f25cc154d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.785+0000 7f25d39e3640 1 -- 172.21.15.83:0/3894486594 shutdown_connections 2024-09-19T07:45:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.785+0000 7f25d39e3640 1 -- 172.21.15.83:0/3894486594 wait complete. 2024-09-19T07:45:30.807 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-19T07:45:30.807 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:45:30.807 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:30.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b8dc45640 1 Processor -- start 2024-09-19T07:45:30.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b8dc45640 1 -- start start 2024-09-19T07:45:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b8dc45640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9b880c3130 0x7f9b880ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b8dc45640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b8806da00 0x7f9b880c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b8dc45640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b880c4990 con 0x7f9b880c3a50 2024-09-19T07:45:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b8dc45640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f9b880c4c70 con 0x7f9b880c3130 2024-09-19T07:45:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b8dc45640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9b880ab7b0 con 0x7f9b8806da00 2024-09-19T07:45:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b86ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b8806da00 0x7f9b880c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b86ffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b8806da00 0x7f9b880c3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:59822/0 (socket says 172.21.15.83:59822) 2024-09-19T07:45:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b86ffd640 1 -- 172.21.15.83:0/2500340761 learned_addr learned my addr 172.21.15.83:0/2500340761 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b877fe640 1 --2- 172.21.15.83:0/2500340761 >> v2:172.21.15.83:3300/0 conn(0x7f9b880c3130 0x7f9b880ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b877fe640 1 -- 172.21.15.83:0/2500340761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b8806da00 msgr2=0x7f9b880c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b877fe640 1 --2- 172.21.15.83:0/2500340761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b8806da00 0x7f9b880c3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b877fe640 1 -- 172.21.15.83:0/2500340761 >> v1:172.21.15.184:6789/0 conn(0x7f9b880c3a50 legacy=0x7f9b880c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.957+0000 7f9b877fe640 1 -- 172.21.15.83:0/2500340761 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b880aba90 con 0x7f9b880c3130 2024-09-19T07:45:30.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.961+0000 7f9b877fe640 1 --2- 172.21.15.83:0/2500340761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b880c3130 0x7f9b880ac290 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f9b780097c0 tx=0x7f9b7801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624afcd6c9108692 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:30.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.961+0000 7f9b867fc640 1 -- 172.21.15.83:0/2500340761 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b780b0070 con 0x7f9b880c3130 2024-09-19T07:45:30.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.961+0000 7f9b867fc640 1 -- 172.21.15.83:0/2500340761 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b780ab600 con 0x7f9b880c3130 2024-09-19T07:45:30.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.961+0000 7f9b867fc640 1 -- 172.21.15.83:0/2500340761 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b780ab9d0 con 0x7f9b880c3130 2024-09-19T07:45:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.961+0000 7f9b8dc45640 1 -- 172.21.15.83:0/2500340761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b880c3130 msgr2=0x7f9b880ac290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.961+0000 7f9b8dc45640 1 --2- 172.21.15.83:0/2500340761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b880c3130 0x7f9b880ac290 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f9b780097c0 tx=0x7f9b7801c3d0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 -- 172.21.15.83:0/2500340761 shutdown_connections 2024-09-19T07:45:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 --2- 172.21.15.83:0/2500340761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b8806da00 0x7f9b880c3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 --2- 172.21.15.83:0/2500340761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b880c3130 0x7f9b880ac290 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 -- 172.21.15.83:0/2500340761 >> 172.21.15.83:0/2500340761 conn(0x7f9b880c5060 msgr2=0x7f9b880c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 -- 172.21.15.83:0/2500340761 shutdown_connections 2024-09-19T07:45:30.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 -- 172.21.15.83:0/2500340761 wait complete. 2024-09-19T07:45:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 Processor -- start 2024-09-19T07:45:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 -- start start 2024-09-19T07:45:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b8806da00 0x7f9b88073c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b880c3130 0x7f9b8814a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b880c3a50 0x7f9b8814abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8815d800 con 0x7f9b880c3a50 2024-09-19T07:45:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9b880ad730 con 0x7f9b880c3130 2024-09-19T07:45:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9b8815d980 con 0x7f9b8806da00 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b87fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b880c3a50 0x7f9b8814abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b87fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b880c3a50 0x7f9b8814abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42850/0 (socket says 172.21.15.83:42850) 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b87fff640 1 -- 172.21.15.83:0/1673206811 learned_addr learned my addr 172.21.15.83:0/1673206811 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b87fff640 1 -- 172.21.15.83:0/1673206811 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b8806da00 msgr2=0x7f9b88073c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b87fff640 1 --2- 172.21.15.83:0/1673206811 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b8806da00 0x7f9b88073c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b87fff640 1 -- 172.21.15.83:0/1673206811 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b880c3130 msgr2=0x7f9b8814a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b87fff640 1 --2- 172.21.15.83:0/1673206811 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b880c3130 0x7f9b8814a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b87fff640 1 -- 172.21.15.83:0/1673206811 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8814c6f0 con 0x7f9b880c3a50 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b87fff640 1 --2- 172.21.15.83:0/1673206811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b880c3a50 0x7f9b8814abe0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000b9c0 tx=0x7f9b8000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b6ffff640 1 -- 172.21.15.83:0/1673206811 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80019070 con 0x7f9b880c3a50 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b6ffff640 1 -- 172.21.15.83:0/1673206811 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b8000cba0 con 0x7f9b880c3a50 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 -- 172.21.15.83:0/1673206811 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b8814c960 con 0x7f9b880c3a50 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b6ffff640 1 -- 172.21.15.83:0/1673206811 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80014440 con 0x7f9b880c3a50 2024-09-19T07:45:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.965+0000 7f9b8dc45640 1 -- 172.21.15.83:0/1673206811 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b8814f060 con 0x7f9b880c3a50 2024-09-19T07:45:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.969+0000 7f9b8dc45640 1 -- 172.21.15.83:0/1673206811 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b88068e20 con 0x7f9b880c3a50 2024-09-19T07:45:30.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.969+0000 7f9b6ffff640 1 -- 172.21.15.83:0/1673206811 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9b8001c070 con 0x7f9b880c3a50 2024-09-19T07:45:30.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.969+0000 7f9b6ffff640 1 --2- 172.21.15.83:0/1673206811 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9b740678c0 0x7f9b74069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:30.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.969+0000 7f9b6ffff640 1 -- 172.21.15.83:0/1673206811 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9733+0+0 (secure 0 0 0) 0x7f9b80086e90 con 0x7f9b880c3a50 2024-09-19T07:45:30.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.969+0000 7f9b877fe640 1 --2- 172.21.15.83:0/1673206811 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9b740678c0 0x7f9b74069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:30.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.969+0000 7f9b877fe640 1 --2- 172.21.15.83:0/1673206811 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9b740678c0 0x7f9b74069d80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9b780ad000 tx=0x7f9b78008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:30.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:30.977+0000 7f9b6ffff640 1 -- 172.21.15.83:0/1673206811 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9b80023020 con 0x7f9b880c3a50 2024-09-19T07:45:31.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.237+0000 7f9b8dc45640 1 -- 172.21.15.83:0/1673206811 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b8806f900 con 0x7f9b880c3a50 2024-09-19T07:45:31.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.237+0000 7f9b6ffff640 1 -- 172.21.15.83:0/1673206811 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) ==== 95+0+5709 (secure 0 0 0) 0x7f9b80056ad0 con 0x7f9b880c3a50 2024-09-19T07:45:31.239 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:31.239 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":183,"btime":"2024-09-19T07:45:30:600551+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:29.605751+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:31.239 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 183 2024-09-19T07:45:31.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.241+0000 7f9b8dc45640 1 -- 172.21.15.83:0/1673206811 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9b740678c0 msgr2=0x7f9b74069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:31.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.241+0000 7f9b8dc45640 1 --2- 172.21.15.83:0/1673206811 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9b740678c0 0x7f9b74069d80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9b780ad000 tx=0x7f9b78008040 comp rx=0 tx=0).stop 2024-09-19T07:45:31.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.241+0000 7f9b8dc45640 1 -- 172.21.15.83:0/1673206811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b880c3a50 msgr2=0x7f9b8814abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.241+0000 7f9b8dc45640 1 --2- 172.21.15.83:0/1673206811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b880c3a50 0x7f9b8814abe0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000b9c0 tx=0x7f9b8000be80 comp rx=0 tx=0).stop 2024-09-19T07:45:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.241+0000 7f9b8dc45640 1 -- 172.21.15.83:0/1673206811 shutdown_connections 2024-09-19T07:45:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.241+0000 7f9b8dc45640 1 --2- 172.21.15.83:0/1673206811 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9b740678c0 0x7f9b74069d80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.241+0000 7f9b8dc45640 1 --2- 172.21.15.83:0/1673206811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b880c3a50 0x7f9b8814abe0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.241+0000 7f9b8dc45640 1 --2- 172.21.15.83:0/1673206811 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b880c3130 0x7f9b8814a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.241+0000 7f9b8dc45640 1 --2- 172.21.15.83:0/1673206811 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b8806da00 0x7f9b88073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.241+0000 7f9b8dc45640 1 -- 172.21.15.83:0/1673206811 >> 172.21.15.83:0/1673206811 conn(0x7f9b880c5060 msgr2=0x7f9b880c5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:31.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.241+0000 7f9b8dc45640 1 -- 172.21.15.83:0/1673206811 shutdown_connections 2024-09-19T07:45:31.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:31.241+0000 7f9b8dc45640 1 -- 172.21.15.83:0/1673206811 wait complete. 2024-09-19T07:45:31.266 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:45:30:600551+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 183, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 183, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7968": { "addr": "172.21.15.83:6835/119695014", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 119695014, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 119695014, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7968, "incarnation": 181, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:45:29.605751+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7968 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 179, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6152": { "addr": "172.21.15.184:6833/1207928072", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 1207928072, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 1207928072, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6152, "incarnation": 176, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:45:16.742872+0000", "qdb_cluster": [ 6152 ], "qdb_leader": 6152, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6152 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:45:31.651 INFO:tasks.workunit.client.0.smithi083.stderr:+ rm -r tmp ffsb 2024-09-19T07:45:32.934 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:45:32.933+0000 7fc0e68e9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:45:33.267 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4074022640 1 Processor -- start 2024-09-19T07:45:33.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4074022640 1 -- start start 2024-09-19T07:45:33.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4074022640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f406c14dae0 0x7f406c0c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:33.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4074022640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f406c0c57f0 0x7f406c0c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:33.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4074022640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f406c153b20 con 0x7f406c0c7510 2024-09-19T07:45:33.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4074022640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f406c06fb30 con 0x7f406c14dae0 2024-09-19T07:45:33.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4074022640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f406c153d90 con 0x7f406c0c57f0 2024-09-19T07:45:33.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4071dbe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f406c14dae0 0x7f406c0c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:33.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4071dbe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f406c14dae0 0x7f406c0c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:48654/0 (socket says 172.21.15.83:48654) 2024-09-19T07:45:33.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f40725bf640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f406c0c7510 0x7f406c157000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55918/0 (socket says 172.21.15.83:55918) 2024-09-19T07:45:33.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4071dbe640 1 -- 172.21.15.83:0/3521062142 learned_addr learned my addr 172.21.15.83:0/3521062142 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:33.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4070dbc640 1 -- 172.21.15.83:0/3521062142 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1968720732 0 0) 0x7f406c153b20 con 0x7f406c0c7510 2024-09-19T07:45:33.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4070dbc640 1 -- 172.21.15.83:0/3521062142 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4058003530 con 0x7f406c0c7510 2024-09-19T07:45:33.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4071dbe640 1 -- 172.21.15.83:0/3521062142 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f406c0c57f0 msgr2=0x7f406c0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:33.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4071dbe640 1 --2- 172.21.15.83:0/3521062142 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f406c0c57f0 0x7f406c0c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:33.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4071dbe640 1 -- 172.21.15.83:0/3521062142 >> v1:172.21.15.83:6789/0 conn(0x7f406c0c7510 legacy=0x7f406c157000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:33.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.425+0000 7f4071dbe640 1 -- 172.21.15.83:0/3521062142 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c14dec0 con 0x7f406c14dae0 2024-09-19T07:45:33.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.429+0000 7f4071dbe640 1 --2- 172.21.15.83:0/3521062142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f406c14dae0 0x7f406c0c4010 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f406c06fcb0 tx=0x7f405c01c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6584591412006bb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:33.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.429+0000 7f4070dbc640 1 -- 172.21.15.83:0/3521062142 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c0b1070 con 0x7f406c14dae0 2024-09-19T07:45:33.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.429+0000 7f4070dbc640 1 -- 172.21.15.83:0/3521062142 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405c0ac640 con 0x7f406c14dae0 2024-09-19T07:45:33.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.429+0000 7f4070dbc640 1 -- 172.21.15.83:0/3521062142 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c0aca10 con 0x7f406c14dae0 2024-09-19T07:45:33.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.429+0000 7f4074022640 1 -- 172.21.15.83:0/3521062142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f406c14dae0 msgr2=0x7f406c0c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:33.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.429+0000 7f4074022640 1 --2- 172.21.15.83:0/3521062142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f406c14dae0 0x7f406c0c4010 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f406c06fcb0 tx=0x7f405c01c420 comp rx=0 tx=0).stop 2024-09-19T07:45:33.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.429+0000 7f4074022640 1 -- 172.21.15.83:0/3521062142 shutdown_connections 2024-09-19T07:45:33.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.429+0000 7f4074022640 1 --2- 172.21.15.83:0/3521062142 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f406c0c57f0 0x7f406c0c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:33.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.429+0000 7f4074022640 1 --2- 172.21.15.83:0/3521062142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f406c14dae0 0x7f406c0c4010 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:33.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.429+0000 7f4074022640 1 -- 172.21.15.83:0/3521062142 >> 172.21.15.83:0/3521062142 conn(0x7f406c0c6110 msgr2=0x7f406c0c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:33.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.429+0000 7f4074022640 1 -- 172.21.15.83:0/3521062142 shutdown_connections 2024-09-19T07:45:33.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4074022640 1 -- 172.21.15.83:0/3521062142 wait complete. 2024-09-19T07:45:33.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4074022640 1 Processor -- start 2024-09-19T07:45:33.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4074022640 1 -- start start 2024-09-19T07:45:33.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4074022640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0c57f0 0x7f406c0bf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:33.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4074022640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f406c0c7510 0x7f406c1449b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:33.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4074022640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f406c14dae0 0x7f406c144ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:33.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4074022640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f406c0c3b20 con 0x7f406c0c57f0 2024-09-19T07:45:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4074022640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f406c0c9370 con 0x7f406c0c7510 2024-09-19T07:45:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4074022640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f406c0c3ca0 con 0x7f406c14dae0 2024-09-19T07:45:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4071dbe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0c57f0 0x7f406c0bf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4071dbe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0c57f0 0x7f406c0bf8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42866/0 (socket says 172.21.15.83:42866) 2024-09-19T07:45:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4071dbe640 1 -- 172.21.15.83:0/4263628181 learned_addr learned my addr 172.21.15.83:0/4263628181 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f40715bd640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f406c0c7510 0x7f406c1449b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f40725bf640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f406c14dae0 0x7f406c144ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4071dbe640 1 -- 172.21.15.83:0/4263628181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f406c14dae0 msgr2=0x7f406c144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4071dbe640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f406c14dae0 0x7f406c144ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4071dbe640 1 -- 172.21.15.83:0/4263628181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f406c0c7510 msgr2=0x7f406c1449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4071dbe640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f406c0c7510 0x7f406c1449b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4071dbe640 1 -- 172.21.15.83:0/4263628181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c0c12c0 con 0x7f406c0c57f0 2024-09-19T07:45:33.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4071dbe640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0c57f0 0x7f406c0bf8c0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f405c0af040 tx=0x7f405c01ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:33.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f40627fc640 1 -- 172.21.15.83:0/4263628181 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c0bb020 con 0x7f406c0c57f0 2024-09-19T07:45:33.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f40627fc640 1 -- 172.21.15.83:0/4263628181 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405c0b7420 con 0x7f406c0c57f0 2024-09-19T07:45:33.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4074022640 1 -- 172.21.15.83:0/4263628181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406c1ed550 con 0x7f406c0c57f0 2024-09-19T07:45:33.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f40627fc640 1 -- 172.21.15.83:0/4263628181 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c0b1040 con 0x7f406c0c57f0 2024-09-19T07:45:33.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.433+0000 7f4074022640 1 -- 172.21.15.83:0/4263628181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f406c1568a0 con 0x7f406c0c57f0 2024-09-19T07:45:33.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.437+0000 7f4074022640 1 -- 172.21.15.83:0/4263628181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f406c06dfb0 con 0x7f406c0c57f0 2024-09-19T07:45:33.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.437+0000 7f40627fc640 1 -- 172.21.15.83:0/4263628181 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f405c0be020 con 0x7f406c0c57f0 2024-09-19T07:45:33.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.437+0000 7f40627fc640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f40500678c0 0x7f4050069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:33.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.437+0000 7f40627fc640 1 -- 172.21.15.83:0/4263628181 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9733+0+0 (secure 0 0 0) 0x7f405c120dc0 con 0x7f406c0c57f0 2024-09-19T07:45:33.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.437+0000 7f40715bd640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f40500678c0 0x7f4050069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:33.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.437+0000 7f40715bd640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f40500678c0 0x7f4050069d80 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f406c15f0c0 tx=0x7f4064006f90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:33.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.445+0000 7f40627fc640 1 -- 172.21.15.83:0/4263628181 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f405c0c5030 con 0x7f406c0c57f0 2024-09-19T07:45:33.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.741+0000 7f4074022640 1 -- 172.21.15.83:0/4263628181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f406c06f910 con 0x7f406c0c57f0 2024-09-19T07:45:33.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.741+0000 7f40627fc640 1 -- 172.21.15.83:0/4263628181 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+5716 (secure 0 0 0) 0x7f405c016130 con 0x7f406c0c57f0 2024-09-19T07:45:33.741 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:33.741 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":184,"btime":"2024-09-19T07:45:31:612734+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:33.742 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 184 2024-09-19T07:45:33.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.745+0000 7f4074022640 1 -- 172.21.15.83:0/4263628181 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f40500678c0 msgr2=0x7f4050069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:33.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.745+0000 7f4074022640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f40500678c0 0x7f4050069d80 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f406c15f0c0 tx=0x7f4064006f90 comp rx=0 tx=0).stop 2024-09-19T07:45:33.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.745+0000 7f4074022640 1 -- 172.21.15.83:0/4263628181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0c57f0 msgr2=0x7f406c0bf8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:33.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.745+0000 7f4074022640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0c57f0 0x7f406c0bf8c0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f405c0af040 tx=0x7f405c01ac60 comp rx=0 tx=0).stop 2024-09-19T07:45:33.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.745+0000 7f4074022640 1 -- 172.21.15.83:0/4263628181 shutdown_connections 2024-09-19T07:45:33.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.745+0000 7f4074022640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f40500678c0 0x7f4050069d80 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:33.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.745+0000 7f4074022640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f406c14dae0 0x7f406c144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:33.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.745+0000 7f4074022640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f406c0c7510 0x7f406c1449b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:33.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.745+0000 7f4074022640 1 --2- 172.21.15.83:0/4263628181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0c57f0 0x7f406c0bf8c0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:33.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.745+0000 7f4074022640 1 -- 172.21.15.83:0/4263628181 >> 172.21.15.83:0/4263628181 conn(0x7f406c0c6110 msgr2=0x7f406c06ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:33.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.745+0000 7f4074022640 1 -- 172.21.15.83:0/4263628181 shutdown_connections 2024-09-19T07:45:33.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:33.745+0000 7f4074022640 1 -- 172.21.15.83:0/4263628181 wait complete. 2024-09-19T07:45:33.777 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=7803 2024-09-19T07:45:33.777 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.c 2024-09-19T07:45:33.965 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-19T07:45:33.966 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-19T07:45:33.995 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-09-19T07:45:33.995 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-19T07:45:34.351 DEBUG:teuthology.parallel:result is None 2024-09-19T07:45:40.778 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-19T07:45:40.778 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:45:40.779 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:45:40.781 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:45:40.782 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:40.922 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:45:40.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55d040b640 1 Processor -- start 2024-09-19T07:45:40.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55d040b640 1 -- start start 2024-09-19T07:45:40.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55d040b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f55c8155d70 0x7f55c80c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:40.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55d040b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f55c81592f0 0x7f55c8157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:40.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55d040b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55c8156200 con 0x7f55c80c6120 2024-09-19T07:45:40.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55d040b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f55c80c9930 con 0x7f55c8155d70 2024-09-19T07:45:40.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55d040b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f55c81564e0 con 0x7f55c81592f0 2024-09-19T07:45:40.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55cd9a6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f55c8155d70 0x7f55c80c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:40.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55cd9a6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f55c8155d70 0x7f55c80c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48880/0 (socket says 172.21.15.83:48880) 2024-09-19T07:45:40.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55cd9a6640 1 -- 172.21.15.83:0/2494949883 learned_addr learned my addr 172.21.15.83:0/2494949883 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:40.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55ce9a8640 1 --2- 172.21.15.83:0/2494949883 >> v2:172.21.15.184:3300/0 conn(0x7f55c81592f0 0x7f55c8157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:40.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55cd1a5640 1 -- 172.21.15.83:0/2494949883 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 813637431 0 0) 0x7f55c8156200 con 0x7f55c80c6120 2024-09-19T07:45:40.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55cd1a5640 1 -- 172.21.15.83:0/2494949883 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55b4003530 con 0x7f55c80c6120 2024-09-19T07:45:40.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55ce9a8640 1 -- 172.21.15.83:0/2494949883 >> v2:172.21.15.83:3300/0 conn(0x7f55c8155d70 msgr2=0x7f55c80c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:40.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55ce9a8640 1 --2- 172.21.15.83:0/2494949883 >> v2:172.21.15.83:3300/0 conn(0x7f55c8155d70 0x7f55c80c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:40.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55ce9a8640 1 -- 172.21.15.83:0/2494949883 >> v1:172.21.15.184:6789/0 conn(0x7f55c80c6120 legacy=0x7f55c81575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:40.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.992+0000 7f55ce9a8640 1 -- 172.21.15.83:0/2494949883 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c81567c0 con 0x7f55c81592f0 2024-09-19T07:45:40.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.996+0000 7f55ce9a8640 1 --2- 172.21.15.83:0/2494949883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c81592f0 0x7f55c8157ce0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f55c80b1990 tx=0x7f55c001c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2a4311c889a7bc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:40.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.996+0000 7f55cd1a5640 1 -- 172.21.15.83:0/2494949883 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c00b2070 con 0x7f55c81592f0 2024-09-19T07:45:40.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.996+0000 7f55cd1a5640 1 -- 172.21.15.83:0/2494949883 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55c00a57f0 con 0x7f55c81592f0 2024-09-19T07:45:40.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.996+0000 7f55cd1a5640 1 -- 172.21.15.83:0/2494949883 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c00a5bc0 con 0x7f55c81592f0 2024-09-19T07:45:40.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.996+0000 7f55d040b640 1 -- 172.21.15.83:0/2494949883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c81592f0 msgr2=0x7f55c8157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:40.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.996+0000 7f55d040b640 1 --2- 172.21.15.83:0/2494949883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c81592f0 0x7f55c8157ce0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f55c80b1990 tx=0x7f55c001c930 comp rx=0 tx=0).stop 2024-09-19T07:45:40.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.996+0000 7f55d040b640 1 -- 172.21.15.83:0/2494949883 shutdown_connections 2024-09-19T07:45:40.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.996+0000 7f55d040b640 1 --2- 172.21.15.83:0/2494949883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c81592f0 0x7f55c8157ce0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:40.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.996+0000 7f55d040b640 1 --2- 172.21.15.83:0/2494949883 >> v2:172.21.15.83:3300/0 conn(0x7f55c8155d70 0x7f55c80c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:40.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:40.996+0000 7f55d040b640 1 -- 172.21.15.83:0/2494949883 >> 172.21.15.83:0/2494949883 conn(0x7f55c806bc80 msgr2=0x7f55c806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:40.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.000+0000 7f55d040b640 1 -- 172.21.15.83:0/2494949883 shutdown_connections 2024-09-19T07:45:41.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.000+0000 7f55d040b640 1 -- 172.21.15.83:0/2494949883 wait complete. 2024-09-19T07:45:41.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55d040b640 1 Processor -- start 2024-09-19T07:45:41.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55d040b640 1 -- start start 2024-09-19T07:45:41.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55d040b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c80c6120 0x7f55c80c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:41.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55d040b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55c81592f0 0x7f55c80c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:41.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55d040b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55c81f7e30 0x7f55c81f82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:41.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55d040b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f55c8153860 con 0x7f55c81592f0 2024-09-19T07:45:41.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55cd9a6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55c81592f0 0x7f55c80c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:41.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55cd9a6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55c81592f0 0x7f55c80c43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48892/0 (socket says 172.21.15.83:48892) 2024-09-19T07:45:41.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55cd9a6640 1 -- 172.21.15.83:0/2014037925 learned_addr learned my addr 172.21.15.83:0/2014037925 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:41.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55d040b640 1 -- 172.21.15.83:0/2014037925 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f55c8162ea0 con 0x7f55c80c6120 2024-09-19T07:45:41.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55ce9a8640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55c81f7e30 0x7f55c81f82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:41.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55d040b640 1 -- 172.21.15.83:0/2014037925 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f55c80c3c50 con 0x7f55c81f7e30 2024-09-19T07:45:41.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55ce1a7640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c80c6120 0x7f55c80c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:41.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55ce9a8640 1 -- 172.21.15.83:0/2014037925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c80c6120 msgr2=0x7f55c80c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:41.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55ce9a8640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c80c6120 0x7f55c80c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:41.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55ce9a8640 1 -- 172.21.15.83:0/2014037925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55c81592f0 msgr2=0x7f55c80c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:41.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55ce9a8640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55c81592f0 0x7f55c80c43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:41.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55ce9a8640 1 -- 172.21.15.83:0/2014037925 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c806c5d0 con 0x7f55c81f7e30 2024-09-19T07:45:41.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55ce9a8640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55c81f7e30 0x7f55c81f82a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f55c8156660 tx=0x7f55c00196d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:41.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55beffd640 1 -- 172.21.15.83:0/2014037925 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c00b2050 con 0x7f55c81f7e30 2024-09-19T07:45:41.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55beffd640 1 -- 172.21.15.83:0/2014037925 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55c00be070 con 0x7f55c81f7e30 2024-09-19T07:45:41.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55beffd640 1 -- 172.21.15.83:0/2014037925 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c00ada50 con 0x7f55c81f7e30 2024-09-19T07:45:41.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.004+0000 7f55d040b640 1 -- 172.21.15.83:0/2014037925 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55c80ad8a0 con 0x7f55c81f7e30 2024-09-19T07:45:41.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.008+0000 7f55d040b640 1 -- 172.21.15.83:0/2014037925 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55c80c3320 con 0x7f55c81f7e30 2024-09-19T07:45:41.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.008+0000 7f55d040b640 1 -- 172.21.15.83:0/2014037925 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55c80c6500 con 0x7f55c81f7e30 2024-09-19T07:45:41.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.012+0000 7f55beffd640 1 -- 172.21.15.83:0/2014037925 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f55c00a56b0 con 0x7f55c81f7e30 2024-09-19T07:45:41.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.012+0000 7f55beffd640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f55ac0678c0 0x7f55ac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:41.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.012+0000 7f55ce1a7640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f55ac0678c0 0x7f55ac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:41.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.012+0000 7f55ce1a7640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f55ac0678c0 0x7f55ac069d80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f55c40011f0 tx=0x7f55c4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:41.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.012+0000 7f55beffd640 1 -- 172.21.15.83:0/2014037925 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(61..61 src has 1..61) ==== 9733+0+0 (secure 0 0 0) 0x7f55c00efb90 con 0x7f55c81f7e30 2024-09-19T07:45:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.032+0000 7f55beffd640 1 -- 172.21.15.83:0/2014037925 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f55c0016520 con 0x7f55c81f7e30 2024-09-19T07:45:41.083 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:45:41.085 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:45:41.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.352+0000 7f55d040b640 1 -- 172.21.15.83:0/2014037925 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55c80c3dd0 con 0x7f55c81f7e30 2024-09-19T07:45:41.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55beffd640 1 -- 172.21.15.83:0/2014037925 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+5716 (secure 0 0 0) 0x7f55c00f2e20 con 0x7f55c81f7e30 2024-09-19T07:45:41.355 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:41.356 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":184,"btime":"2024-09-19T07:45:31:612734+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:41.356 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 184 2024-09-19T07:45:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55d040b640 1 -- 172.21.15.83:0/2014037925 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f55ac0678c0 msgr2=0x7f55ac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55d040b640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f55ac0678c0 0x7f55ac069d80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f55c40011f0 tx=0x7f55c4009040 comp rx=0 tx=0).stop 2024-09-19T07:45:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55d040b640 1 -- 172.21.15.83:0/2014037925 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55c81f7e30 msgr2=0x7f55c81f82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55d040b640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55c81f7e30 0x7f55c81f82a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f55c8156660 tx=0x7f55c00196d0 comp rx=0 tx=0).stop 2024-09-19T07:45:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55d040b640 1 -- 172.21.15.83:0/2014037925 shutdown_connections 2024-09-19T07:45:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55d040b640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f55ac0678c0 0x7f55ac069d80 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:41.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55d040b640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55c81f7e30 0x7f55c81f82a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:41.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55d040b640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55c81592f0 0x7f55c80c43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:41.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55d040b640 1 --2- 172.21.15.83:0/2014037925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c80c6120 0x7f55c80c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:41.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55d040b640 1 -- 172.21.15.83:0/2014037925 >> 172.21.15.83:0/2014037925 conn(0x7f55c806bc80 msgr2=0x7f55c815fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:41.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55d040b640 1 -- 172.21.15.83:0/2014037925 shutdown_connections 2024-09-19T07:45:41.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:41.356+0000 7f55d040b640 1 -- 172.21.15.83:0/2014037925 wait complete. 2024-09-19T07:45:41.378 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-19T07:45:43.380 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:45:43.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.572+0000 7ff56d714640 1 Processor -- start 2024-09-19T07:45:43.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.572+0000 7ff56d714640 1 -- start start 2024-09-19T07:45:43.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff56d714640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff568149030 0x7ff568152e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:43.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff56d714640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff568073980 con 0x7ff5680c3a20 2024-09-19T07:45:43.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff56d714640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff568072d70 con 0x7ff568072890 2024-09-19T07:45:43.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff56d714640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff5680716a0 con 0x7ff568149030 2024-09-19T07:45:43.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff566ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff5680c3a20 0x7ff56806d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58950/0 (socket says 172.21.15.83:58950) 2024-09-19T07:45:43.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff566ffd640 1 -- 172.21.15.83:0/2285264656 learned_addr learned my addr 172.21.15.83:0/2285264656 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:45:43.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff5677fe640 1 --2- 172.21.15.83:0/2285264656 >> v2:172.21.15.83:3300/0 conn(0x7ff568149030 0x7ff568152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:43.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3461342425 0 0) 0x7ff568073980 con 0x7ff5680c3a20 2024-09-19T07:45:43.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff554003530 con 0x7ff5680c3a20 2024-09-19T07:45:43.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2389484606 0 0) 0x7ff554003530 con 0x7ff5680c3a20 2024-09-19T07:45:43.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 >> v2:172.21.15.83:3300/0 conn(0x7ff568149030 msgr2=0x7ff568152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:43.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 --2- 172.21.15.83:0/2285264656 >> v2:172.21.15.83:3300/0 conn(0x7ff568149030 0x7ff568152e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:43.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 >> v1:172.21.15.184:6789/0 conn(0x7ff568072890 legacy=0x7ff568148920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:43.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5680718f0 con 0x7ff5680c3a20 2024-09-19T07:45:43.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7ff5600016a0 con 0x7ff5680c3a20 2024-09-19T07:45:43.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 >> v1:172.21.15.83:6789/0 conn(0x7ff5680c3a20 legacy=0x7ff56806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:43.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 --2- 172.21.15.83:0/2285264656 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5540039c0 0x7ff554003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:43.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 --2- 172.21.15.83:0/2285264656 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff554004810 0x7ff554004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:43.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 --2- 172.21.15.83:0/2285264656 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5540051d0 0x7ff55400cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:43.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff5680716a0 con 0x7ff554004810 2024-09-19T07:45:43.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff554005bb0 con 0x7ff5540039c0 2024-09-19T07:45:43.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff554005db0 con 0x7ff5540051d0 2024-09-19T07:45:43.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff5667fc640 1 --2- 172.21.15.83:0/2285264656 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff554004810 0x7ff554004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:43.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff5677fe640 1 --2- 172.21.15.83:0/2285264656 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5540051d0 0x7ff55400cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:43.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff5667fc640 1 -- 172.21.15.83:0/2285264656 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5540051d0 msgr2=0x7ff55400cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:45:43.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff5667fc640 1 --2- 172.21.15.83:0/2285264656 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5540051d0 0x7ff55400cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:43.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff5667fc640 1 -- 172.21.15.83:0/2285264656 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5540039c0 msgr2=0x7ff554003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:45:43.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff5667fc640 1 --2- 172.21.15.83:0/2285264656 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5540039c0 0x7ff554003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:43.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff5667fc640 1 -- 172.21.15.83:0/2285264656 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff554003530 con 0x7ff554004810 2024-09-19T07:45:43.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff5677fe640 1 --2- 172.21.15.83:0/2285264656 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5540051d0 0x7ff55400cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:45:43.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.576+0000 7ff5677fe640 1 -- 172.21.15.83:0/2285264656 reap_dead start 2024-09-19T07:45:43.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff5667fc640 1 --2- 172.21.15.83:0/2285264656 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff554004810 0x7ff554004c90 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7ff568072ca0 tx=0x7ff55801c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84257b8ce110ef34 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:43.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5580ae070 con 0x7ff554004810 2024-09-19T07:45:43.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff558003ba0 con 0x7ff554004810 2024-09-19T07:45:43.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff565ffb640 1 -- 172.21.15.83:0/2285264656 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558001970 con 0x7ff554004810 2024-09-19T07:45:43.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff56d714640 1 -- 172.21.15.83:0/2285264656 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff554004810 msgr2=0x7ff554004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:43.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff56d714640 1 --2- 172.21.15.83:0/2285264656 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff554004810 0x7ff554004c90 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7ff568072ca0 tx=0x7ff55801c660 comp rx=0 tx=0).stop 2024-09-19T07:45:43.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff56d714640 1 -- 172.21.15.83:0/2285264656 shutdown_connections 2024-09-19T07:45:43.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff56d714640 1 --2- 172.21.15.83:0/2285264656 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff554004810 0x7ff554004c90 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:43.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff56d714640 1 -- 172.21.15.83:0/2285264656 >> 172.21.15.83:0/2285264656 conn(0x7ff5680c48b0 msgr2=0x7ff5680c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff56d714640 1 -- 172.21.15.83:0/2285264656 shutdown_connections 2024-09-19T07:45:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff56d714640 1 -- 172.21.15.83:0/2285264656 wait complete. 2024-09-19T07:45:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff56d714640 1 Processor -- start 2024-09-19T07:45:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.580+0000 7ff56d714640 1 -- start start 2024-09-19T07:45:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff56d714640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff554004810 0x7ff5681556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff56d714640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5681530e0 0x7ff568154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff56d714640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff56814bce0 0x7ff568154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff56d714640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff5680c3e00 con 0x7ff5681530e0 2024-09-19T07:45:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff56d714640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff568071eb0 con 0x7ff554004810 2024-09-19T07:45:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff56d714640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff5680c3f80 con 0x7ff56814bce0 2024-09-19T07:45:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff5667fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5681530e0 0x7ff568154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff5667fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5681530e0 0x7ff568154150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48924/0 (socket says 172.21.15.83:48924) 2024-09-19T07:45:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff5667fc640 1 -- 172.21.15.83:0/2500035696 learned_addr learned my addr 172.21.15.83:0/2500035696 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:45:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff566ffd640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff554004810 0x7ff5681556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff5677fe640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff56814bce0 0x7ff568154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff5667fc640 1 -- 172.21.15.83:0/2500035696 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff56814bce0 msgr2=0x7ff568154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff5667fc640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff56814bce0 0x7ff568154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff5667fc640 1 -- 172.21.15.83:0/2500035696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff554004810 msgr2=0x7ff5681556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:43.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff5667fc640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff554004810 0x7ff5681556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:43.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff5667fc640 1 -- 172.21.15.83:0/2500035696 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56814c360 con 0x7ff5681530e0 2024-09-19T07:45:43.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff5667fc640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5681530e0 0x7ff568154150 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7ff55801c490 tx=0x7ff554005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:43.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff55f7fe640 1 -- 172.21.15.83:0/2500035696 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5580ae040 con 0x7ff5681530e0 2024-09-19T07:45:43.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff55f7fe640 1 -- 172.21.15.83:0/2500035696 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff558001950 con 0x7ff5681530e0 2024-09-19T07:45:43.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff55f7fe640 1 -- 172.21.15.83:0/2500035696 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5580a9dc0 con 0x7ff5681530e0 2024-09-19T07:45:43.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff56d714640 1 -- 172.21.15.83:0/2500035696 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5681e7a80 con 0x7ff5681530e0 2024-09-19T07:45:43.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff56d714640 1 -- 172.21.15.83:0/2500035696 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff568148760 con 0x7ff5681530e0 2024-09-19T07:45:43.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff55f7fe640 1 -- 172.21.15.83:0/2500035696 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff558006b40 con 0x7ff5681530e0 2024-09-19T07:45:43.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff55f7fe640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5480678c0 0x7ff548069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:45:43.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.584+0000 7ff55f7fe640 1 -- 172.21.15.83:0/2500035696 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9733+0+0 (secure 0 0 0) 0x7ff558121310 con 0x7ff5681530e0 2024-09-19T07:45:43.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.588+0000 7ff56d714640 1 -- 172.21.15.83:0/2500035696 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5680723d0 con 0x7ff5681530e0 2024-09-19T07:45:43.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.588+0000 7ff566ffd640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5480678c0 0x7ff548069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:45:43.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.588+0000 7ff566ffd640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5480678c0 0x7ff548069d80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff560001890 tx=0x7ff560001740 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:45:43.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.596+0000 7ff55f7fe640 1 -- 172.21.15.83:0/2500035696 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff558016370 con 0x7ff5681530e0 2024-09-19T07:45:43.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 -- 172.21.15.83:0/2500035696 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5680c3010 con 0x7ff5681530e0 2024-09-19T07:45:43.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff55f7fe640 1 -- 172.21.15.83:0/2500035696 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+6542 (secure 0 0 0) 0x7ff5580f0880 con 0x7ff5681530e0 2024-09-19T07:45:43.860 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:45:43.860 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":185,"btime":"2024-09-19T07:45:41:636892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8028,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":185}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:45:43.860 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 185 2024-09-19T07:45:43.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 -- 172.21.15.83:0/2500035696 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5480678c0 msgr2=0x7ff548069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:43.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5480678c0 0x7ff548069d80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff560001890 tx=0x7ff560001740 comp rx=0 tx=0).stop 2024-09-19T07:45:43.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 -- 172.21.15.83:0/2500035696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5681530e0 msgr2=0x7ff568154150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:45:43.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5681530e0 0x7ff568154150 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7ff55801c490 tx=0x7ff554005840 comp rx=0 tx=0).stop 2024-09-19T07:45:43.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 -- 172.21.15.83:0/2500035696 shutdown_connections 2024-09-19T07:45:43.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff5480678c0 0x7ff548069d80 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:43.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff56814bce0 0x7ff568154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:43.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5681530e0 0x7ff568154150 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:43.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 --2- 172.21.15.83:0/2500035696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff554004810 0x7ff5681556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:45:43.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 -- 172.21.15.83:0/2500035696 >> 172.21.15.83:0/2500035696 conn(0x7ff5680c48b0 msgr2=0x7ff56806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:45:43.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 -- 172.21.15.83:0/2500035696 shutdown_connections 2024-09-19T07:45:43.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:45:43.860+0000 7ff56d714640 1 -- 172.21.15.83:0/2500035696 wait complete. 2024-09-19T07:45:43.883 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-19T07:45:43.884 INFO:tasks.mds_thrash.fs.[a]:waiting for 115 secs before thrashing 2024-09-19T07:46:11.166 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:46:11.169 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:46:41.553 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:46:41.556 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:46:51.135 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:46:51.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.307+0000 7fc9f4e4e640 1 Processor -- start 2024-09-19T07:46:51.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.307+0000 7fc9f4e4e640 1 -- start start 2024-09-19T07:46:51.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.307+0000 7fc9f4e4e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc9f0152f30 0x7fc9f01533a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:51.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.307+0000 7fc9f4e4e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9f0155e10 0x7fc9f0156280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:51.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.307+0000 7fc9f4e4e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9f01538e0 con 0x7fc9f00c4880 2024-09-19T07:46:51.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.307+0000 7fc9f4e4e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc9f015a040 con 0x7fc9f0152f30 2024-09-19T07:46:51.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.307+0000 7fc9f4e4e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f00c2c10 con 0x7fc9f0155e10 2024-09-19T07:46:51.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.307+0000 7fc9eed9d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9f0155e10 0x7fc9f0156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:51.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.307+0000 7fc9eed9d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9f0155e10 0x7fc9f0156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:56278/0 (socket says 172.21.15.83:56278) 2024-09-19T07:46:51.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.307+0000 7fc9eed9d640 1 -- 172.21.15.83:0/2688415854 learned_addr learned my addr 172.21.15.83:0/2688415854 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:46:51.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.311+0000 7fc9eed9d640 1 -- 172.21.15.83:0/2688415854 >> v2:172.21.15.184:3300/0 conn(0x7fc9f0152f30 msgr2=0x7fc9f01533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:46:51.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.311+0000 7fc9eed9d640 1 --2- 172.21.15.83:0/2688415854 >> v2:172.21.15.184:3300/0 conn(0x7fc9f0152f30 0x7fc9f01533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.311+0000 7fc9eed9d640 1 -- 172.21.15.83:0/2688415854 >> v1:172.21.15.184:6789/0 conn(0x7fc9f00c4880 legacy=0x7fc9f00c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:46:51.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.311+0000 7fc9eed9d640 1 -- 172.21.15.83:0/2688415854 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f0159430 con 0x7fc9f0155e10 2024-09-19T07:46:51.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.311+0000 7fc9eed9d640 1 --2- 172.21.15.83:0/2688415854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9f0155e10 0x7fc9f0156280 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009e50 tx=0x7fc9e801c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd36cbd7bc2943b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:46:51.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.311+0000 7fc9ed59a640 1 -- 172.21.15.83:0/2688415854 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80a5720 con 0x7fc9f0155e10 2024-09-19T07:46:51.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.311+0000 7fc9ed59a640 1 -- 172.21.15.83:0/2688415854 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e80a58c0 con 0x7fc9f0155e10 2024-09-19T07:46:51.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.311+0000 7fc9ed59a640 1 -- 172.21.15.83:0/2688415854 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80a5c90 con 0x7fc9f0155e10 2024-09-19T07:46:51.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.315+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2688415854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9f0155e10 msgr2=0x7fc9f0156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:51.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.315+0000 7fc9f4e4e640 1 --2- 172.21.15.83:0/2688415854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9f0155e10 0x7fc9f0156280 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009e50 tx=0x7fc9e801c8a0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.315+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2688415854 shutdown_connections 2024-09-19T07:46:51.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.315+0000 7fc9f4e4e640 1 --2- 172.21.15.83:0/2688415854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9f0155e10 0x7fc9f0156280 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.315+0000 7fc9f4e4e640 1 --2- 172.21.15.83:0/2688415854 >> v2:172.21.15.184:3300/0 conn(0x7fc9f0152f30 0x7fc9f01533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.315+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2688415854 >> 172.21.15.83:0/2688415854 conn(0x7fc9f00aa910 msgr2=0x7fc9f00c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:46:51.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.315+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2688415854 shutdown_connections 2024-09-19T07:46:51.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.315+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2688415854 wait complete. 2024-09-19T07:46:51.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.315+0000 7fc9f4e4e640 1 Processor -- start 2024-09-19T07:46:51.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.315+0000 7fc9f4e4e640 1 -- start start 2024-09-19T07:46:51.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9f4e4e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9f00c4880 0x7fc9f006eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:51.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9f4e4e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9f0152f30 0x7fc9f014fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9f4e4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f0155e10 0x7fc9f01503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9f4e4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f00c2680 con 0x7fc9f0155e10 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9f4e4e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f00c2500 con 0x7fc9f0152f30 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9f4e4e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f00c4c60 con 0x7fc9f00c4880 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9eed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f0155e10 0x7fc9f01503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9eed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f0155e10 0x7fc9f01503d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52744/0 (socket says 172.21.15.83:52744) 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9eed9d640 1 -- 172.21.15.83:0/2099016875 learned_addr learned my addr 172.21.15.83:0/2099016875 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9eed9d640 1 -- 172.21.15.83:0/2099016875 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9f00c4880 msgr2=0x7fc9f006eca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9eed9d640 1 --2- 172.21.15.83:0/2099016875 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9f00c4880 0x7fc9f006eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9eed9d640 1 -- 172.21.15.83:0/2099016875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9f0152f30 msgr2=0x7fc9f014fe90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9eed9d640 1 --2- 172.21.15.83:0/2099016875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9f0152f30 0x7fc9f014fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9eed9d640 1 -- 172.21.15.83:0/2099016875 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f01ee400 con 0x7fc9f0155e10 2024-09-19T07:46:51.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9eed9d640 1 --2- 172.21.15.83:0/2099016875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f0155e10 0x7fc9f01503d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009e50 tx=0x7fc9e8019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:46:51.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9deffd640 1 -- 172.21.15.83:0/2099016875 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80cd020 con 0x7fc9f0155e10 2024-09-19T07:46:51.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2099016875 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f00c80a0 con 0x7fc9f0155e10 2024-09-19T07:46:51.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2099016875 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f0158de0 con 0x7fc9f0155e10 2024-09-19T07:46:51.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9deffd640 1 -- 172.21.15.83:0/2099016875 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e80c9680 con 0x7fc9f0155e10 2024-09-19T07:46:51.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.319+0000 7fc9deffd640 1 -- 172.21.15.83:0/2099016875 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80c9a30 con 0x7fc9f0155e10 2024-09-19T07:46:51.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.323+0000 7fc9deffd640 1 -- 172.21.15.83:0/2099016875 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc9e80d0020 con 0x7fc9f0155e10 2024-09-19T07:46:51.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.323+0000 7fc9deffd640 1 --2- 172.21.15.83:0/2099016875 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9d00678f0 0x7fc9d0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:51.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.323+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2099016875 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9d40052a0 con 0x7fc9f0155e10 2024-09-19T07:46:51.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.323+0000 7fc9deffd640 1 -- 172.21.15.83:0/2099016875 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9733+0+0 (secure 0 0 0) 0x7fc9e81002c0 con 0x7fc9f0155e10 2024-09-19T07:46:51.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.323+0000 7fc9ee59c640 1 --2- 172.21.15.83:0/2099016875 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9d00678f0 0x7fc9d0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:51.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.327+0000 7fc9ee59c640 1 --2- 172.21.15.83:0/2099016875 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9d00678f0 0x7fc9d0069db0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4007fd0 tx=0x7fc9e40079a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:46:51.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.335+0000 7fc9deffd640 1 -- 172.21.15.83:0/2099016875 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc9e80d7030 con 0x7fc9f0155e10 2024-09-19T07:46:51.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.623+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2099016875 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9d4005860 con 0x7fc9f0155e10 2024-09-19T07:46:51.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.623+0000 7fc9deffd640 1 -- 172.21.15.83:0/2099016875 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+6542 (secure 0 0 0) 0x7fc9e80165e0 con 0x7fc9f0155e10 2024-09-19T07:46:51.621 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:46:51.621 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":185,"btime":"2024-09-19T07:45:41:636892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8028,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":185}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:46:51.622 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 185 2024-09-19T07:46:51.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.627+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2099016875 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9d00678f0 msgr2=0x7fc9d0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:51.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.627+0000 7fc9f4e4e640 1 --2- 172.21.15.83:0/2099016875 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9d00678f0 0x7fc9d0069db0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4007fd0 tx=0x7fc9e40079a0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.627+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2099016875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f0155e10 msgr2=0x7fc9f01503d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:51.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.627+0000 7fc9f4e4e640 1 --2- 172.21.15.83:0/2099016875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f0155e10 0x7fc9f01503d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009e50 tx=0x7fc9e8019640 comp rx=0 tx=0).stop 2024-09-19T07:46:51.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.627+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2099016875 shutdown_connections 2024-09-19T07:46:51.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.627+0000 7fc9f4e4e640 1 --2- 172.21.15.83:0/2099016875 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc9d00678f0 0x7fc9d0069db0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.627+0000 7fc9f4e4e640 1 --2- 172.21.15.83:0/2099016875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9f0155e10 0x7fc9f01503d0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.627+0000 7fc9f4e4e640 1 --2- 172.21.15.83:0/2099016875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9f0152f30 0x7fc9f014fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.627+0000 7fc9f4e4e640 1 --2- 172.21.15.83:0/2099016875 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9f00c4880 0x7fc9f006eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.627+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2099016875 >> 172.21.15.83:0/2099016875 conn(0x7fc9f00aa910 msgr2=0x7fc9f0151280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:46:51.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.627+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2099016875 shutdown_connections 2024-09-19T07:46:51.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.627+0000 7fc9f4e4e640 1 -- 172.21.15.83:0/2099016875 wait complete. 2024-09-19T07:46:51.642 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-19T07:46:51.642 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-19T07:46:51.642 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:46:51.656 INFO:tasks.ceph.mds.d:Stopped 2024-09-19T07:46:51.656 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:46:51.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f3510a38640 1 Processor -- start 2024-09-19T07:46:51.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f3510a38640 1 -- start start 2024-09-19T07:46:51.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f3510a38640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f350c151150 0x7f350c153a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:51.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f3510a38640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f350c074f90 0x7f350c0753c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:51.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f3510a38640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f350c157b70 con 0x7f350c0c3220 2024-09-19T07:46:51.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f3510a38640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f350c06f150 con 0x7f350c151150 2024-09-19T07:46:51.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f3510a38640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f350c153f90 con 0x7f350c074f90 2024-09-19T07:46:51.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f3509d9b640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7f350c0c3220 0x7f350c06cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:59570/0 (socket says 172.21.15.83:59570) 2024-09-19T07:46:51.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f3509d9b640 1 -- 172.21.15.83:0/2998368524 learned_addr learned my addr 172.21.15.83:0/2998368524 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:46:51.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f350ad9d640 1 --2- 172.21.15.83:0/2998368524 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f350c074f90 0x7f350c0753c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:51.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f350959a640 1 -- 172.21.15.83:0/2998368524 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 826135451 0 0) 0x7f350c157b70 con 0x7f350c0c3220 2024-09-19T07:46:51.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f350959a640 1 -- 172.21.15.83:0/2998368524 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34f4003530 con 0x7f350c0c3220 2024-09-19T07:46:51.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f350ad9d640 1 -- 172.21.15.83:0/2998368524 >> v2:172.21.15.83:3300/0 conn(0x7f350c151150 msgr2=0x7f350c153a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:46:51.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f350ad9d640 1 --2- 172.21.15.83:0/2998368524 >> v2:172.21.15.83:3300/0 conn(0x7f350c151150 0x7f350c153a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f350ad9d640 1 -- 172.21.15.83:0/2998368524 >> v1:172.21.15.184:6789/0 conn(0x7f350c0c3220 legacy=0x7f350c06cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:51.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.795+0000 7f350ad9d640 1 -- 172.21.15.83:0/2998368524 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350c157e10 con 0x7f350c074f90 2024-09-19T07:46:51.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f350ad9d640 1 --2- 172.21.15.83:0/2998368524 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f350c074f90 0x7f350c0753c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3504009d90 tx=0x7f350401c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76a049894ff8602c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:46:51.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f350959a640 1 -- 172.21.15.83:0/2998368524 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35040b2070 con 0x7f350c074f90 2024-09-19T07:46:51.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f350959a640 1 -- 172.21.15.83:0/2998368524 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35040a5a20 con 0x7f350c074f90 2024-09-19T07:46:51.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f350959a640 1 -- 172.21.15.83:0/2998368524 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35040a5df0 con 0x7f350c074f90 2024-09-19T07:46:51.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f3510a38640 1 -- 172.21.15.83:0/2998368524 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f350c074f90 msgr2=0x7f350c0753c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:51.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f3510a38640 1 --2- 172.21.15.83:0/2998368524 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f350c074f90 0x7f350c0753c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3504009d90 tx=0x7f350401c960 comp rx=0 tx=0).stop 2024-09-19T07:46:51.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f3510a38640 1 -- 172.21.15.83:0/2998368524 shutdown_connections 2024-09-19T07:46:51.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f3510a38640 1 --2- 172.21.15.83:0/2998368524 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f350c074f90 0x7f350c0753c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f3510a38640 1 --2- 172.21.15.83:0/2998368524 >> v2:172.21.15.83:3300/0 conn(0x7f350c151150 0x7f350c153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f3510a38640 1 -- 172.21.15.83:0/2998368524 >> 172.21.15.83:0/2998368524 conn(0x7f350c0c90e0 msgr2=0x7f350c0aa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:46:51.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f3510a38640 1 -- 172.21.15.83:0/2998368524 shutdown_connections 2024-09-19T07:46:51.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f3510a38640 1 -- 172.21.15.83:0/2998368524 wait complete. 2024-09-19T07:46:51.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f3510a38640 1 Processor -- start 2024-09-19T07:46:51.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.799+0000 7f3510a38640 1 -- start start 2024-09-19T07:46:51.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f3510a38640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f350c074f90 0x7f350c0c6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:51.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f3510a38640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f350c0c3220 0x7f350c152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:51.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f3510a38640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f350c151150 0x7f350c155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:51.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f3510a38640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f350c1516b0 con 0x7f350c151150 2024-09-19T07:46:51.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f3510a38640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f350c151530 con 0x7f350c0c3220 2024-09-19T07:46:51.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f3510a38640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f350c0c2b80 con 0x7f350c074f90 2024-09-19T07:46:51.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f350ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f350c151150 0x7f350c155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:51.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f350ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f350c151150 0x7f350c155d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52750/0 (socket says 172.21.15.83:52750) 2024-09-19T07:46:51.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f350ad9d640 1 -- 172.21.15.83:0/2416467945 learned_addr learned my addr 172.21.15.83:0/2416467945 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:46:51.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f3509d9b640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f350c0c3220 0x7f350c152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:51.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f350a59c640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f350c074f90 0x7f350c0c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:51.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f350ad9d640 1 -- 172.21.15.83:0/2416467945 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f350c074f90 msgr2=0x7f350c0c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f350ad9d640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f350c074f90 0x7f350c0c6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f350ad9d640 1 -- 172.21.15.83:0/2416467945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f350c0c3220 msgr2=0x7f350c152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f350ad9d640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f350c0c3220 0x7f350c152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f350ad9d640 1 -- 172.21.15.83:0/2416467945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350c156940 con 0x7f350c151150 2024-09-19T07:46:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f350ad9d640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f350c151150 0x7f350c155d30 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f35040b0040 tx=0x7f3504019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:46:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f34faffd640 1 -- 172.21.15.83:0/2416467945 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35040bd020 con 0x7f350c151150 2024-09-19T07:46:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f34faffd640 1 -- 172.21.15.83:0/2416467945 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35040ad6c0 con 0x7f350c151150 2024-09-19T07:46:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f3510a38640 1 -- 172.21.15.83:0/2416467945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f350c156c20 con 0x7f350c151150 2024-09-19T07:46:51.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f34faffd640 1 -- 172.21.15.83:0/2416467945 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35040b2040 con 0x7f350c151150 2024-09-19T07:46:51.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f3510a38640 1 -- 172.21.15.83:0/2416467945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f350c1530b0 con 0x7f350c151150 2024-09-19T07:46:51.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.803+0000 7f3510a38640 1 -- 172.21.15.83:0/2416467945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f350c156df0 con 0x7f350c151150 2024-09-19T07:46:51.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.807+0000 7f34faffd640 1 -- 172.21.15.83:0/2416467945 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f35040c0020 con 0x7f350c151150 2024-09-19T07:46:51.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.807+0000 7f34faffd640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f34ec0678c0 0x7f34ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:51.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.807+0000 7f34faffd640 1 -- 172.21.15.83:0/2416467945 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9733+0+0 (secure 0 0 0) 0x7f3504121c90 con 0x7f350c151150 2024-09-19T07:46:51.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.807+0000 7f350a59c640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f34ec0678c0 0x7f34ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:51.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.807+0000 7f350a59c640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f34ec0678c0 0x7f34ec069d80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f350c14add0 tx=0x7f34fc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:46:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:51.811+0000 7f34faffd640 1 -- 172.21.15.83:0/2416467945 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f35040c7030 con 0x7f350c151150 2024-09-19T07:46:52.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.079+0000 7f3510a38640 1 -- 172.21.15.83:0/2416467945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f350c0753c0 con 0x7f350c151150 2024-09-19T07:46:52.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.079+0000 7f34faffd640 1 -- 172.21.15.83:0/2416467945 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+6542 (secure 0 0 0) 0x7f3504016580 con 0x7f350c151150 2024-09-19T07:46:52.077 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:46:52.077 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":185,"btime":"2024-09-19T07:45:41:636892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8028,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":185}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:46:52.078 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 185 2024-09-19T07:46:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.083+0000 7f3510a38640 1 -- 172.21.15.83:0/2416467945 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f34ec0678c0 msgr2=0x7f34ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.083+0000 7f3510a38640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f34ec0678c0 0x7f34ec069d80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f350c14add0 tx=0x7f34fc008040 comp rx=0 tx=0).stop 2024-09-19T07:46:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.083+0000 7f3510a38640 1 -- 172.21.15.83:0/2416467945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f350c151150 msgr2=0x7f350c155d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.083+0000 7f3510a38640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f350c151150 0x7f350c155d30 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f35040b0040 tx=0x7f3504019700 comp rx=0 tx=0).stop 2024-09-19T07:46:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.083+0000 7f3510a38640 1 -- 172.21.15.83:0/2416467945 shutdown_connections 2024-09-19T07:46:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.083+0000 7f3510a38640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f34ec0678c0 0x7f34ec069d80 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:52.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.083+0000 7f3510a38640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f350c151150 0x7f350c155d30 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:52.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.083+0000 7f3510a38640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f350c0c3220 0x7f350c152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:52.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.083+0000 7f3510a38640 1 --2- 172.21.15.83:0/2416467945 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f350c074f90 0x7f350c0c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:52.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.083+0000 7f3510a38640 1 -- 172.21.15.83:0/2416467945 >> 172.21.15.83:0/2416467945 conn(0x7f350c0c90e0 msgr2=0x7f350c0aa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:46:52.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.083+0000 7f3510a38640 1 -- 172.21.15.83:0/2416467945 shutdown_connections 2024-09-19T07:46:52.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:52.083+0000 7f3510a38640 1 -- 172.21.15.83:0/2416467945 wait complete. 2024-09-19T07:46:52.100 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:46:54.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:46:54.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.243+0000 7f5a9a74b640 1 Processor -- start 2024-09-19T07:46:54.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.243+0000 7f5a9a74b640 1 -- start start 2024-09-19T07:46:54.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a9a74b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a94153570 0x7f5a940c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:54.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a9a74b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5a940c5110 0x7f5a94159270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:54.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a9a74b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a940c3c20 con 0x7f5a940c4130 2024-09-19T07:46:54.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a9a74b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f5a940c89f0 con 0x7f5a940c5110 2024-09-19T07:46:54.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a9a74b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5a940c8cd0 con 0x7f5a94153570 2024-09-19T07:46:54.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a937fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5a940c5110 0x7f5a94159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:54.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a937fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5a940c5110 0x7f5a94159270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52754/0 (socket says 172.21.15.83:52754) 2024-09-19T07:46:54.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a937fe640 1 -- 172.21.15.83:0/210665666 learned_addr learned my addr 172.21.15.83:0/210665666 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:46:54.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a93fff640 1 --2- 172.21.15.83:0/210665666 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a94153570 0x7f5a940c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:54.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a937fe640 1 -- 172.21.15.83:0/210665666 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a94153570 msgr2=0x7f5a940c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:46:54.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a937fe640 1 --2- 172.21.15.83:0/210665666 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a94153570 0x7f5a940c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:54.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a937fe640 1 -- 172.21.15.83:0/210665666 >> v1:172.21.15.184:6789/0 conn(0x7f5a940c4130 legacy=0x7f5a941597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:54.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a937fe640 1 -- 172.21.15.83:0/210665666 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a940c8fb0 con 0x7f5a940c5110 2024-09-19T07:46:54.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.247+0000 7f5a93fff640 1 --2- 172.21.15.83:0/210665666 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a94153570 0x7f5a940c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:46:54.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.251+0000 7f5a937fe640 1 --2- 172.21.15.83:0/210665666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a940c5110 0x7f5a94159270 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f5a880098a0 tx=0x7f5a8801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f31de5ce93a986 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:46:54.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.251+0000 7f5a92ffd640 1 -- 172.21.15.83:0/210665666 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a880ab640 con 0x7f5a940c5110 2024-09-19T07:46:54.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.251+0000 7f5a92ffd640 1 -- 172.21.15.83:0/210665666 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a880b0070 con 0x7f5a940c5110 2024-09-19T07:46:54.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.251+0000 7f5a92ffd640 1 -- 172.21.15.83:0/210665666 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a880aba00 con 0x7f5a940c5110 2024-09-19T07:46:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.255+0000 7f5a9a74b640 1 -- 172.21.15.83:0/210665666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a940c5110 msgr2=0x7f5a94159270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.255+0000 7f5a9a74b640 1 --2- 172.21.15.83:0/210665666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a940c5110 0x7f5a94159270 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f5a880098a0 tx=0x7f5a8801c420 comp rx=0 tx=0).stop 2024-09-19T07:46:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.255+0000 7f5a9a74b640 1 -- 172.21.15.83:0/210665666 shutdown_connections 2024-09-19T07:46:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.255+0000 7f5a9a74b640 1 --2- 172.21.15.83:0/210665666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a940c5110 0x7f5a94159270 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.255+0000 7f5a9a74b640 1 --2- 172.21.15.83:0/210665666 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a94153570 0x7f5a940c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.255+0000 7f5a9a74b640 1 -- 172.21.15.83:0/210665666 >> 172.21.15.83:0/210665666 conn(0x7f5a940c6110 msgr2=0x7f5a940c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:46:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.255+0000 7f5a9a74b640 1 -- 172.21.15.83:0/210665666 shutdown_connections 2024-09-19T07:46:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.255+0000 7f5a9a74b640 1 -- 172.21.15.83:0/210665666 wait complete. 2024-09-19T07:46:54.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.255+0000 7f5a9a74b640 1 Processor -- start 2024-09-19T07:46:54.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a9a74b640 1 -- start start 2024-09-19T07:46:54.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a9a74b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a940c4130 0x7f5a9414f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:54.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a9a74b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a940c5110 0x7f5a9414cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:54.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a9a74b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a94153570 0x7f5a9414d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a9a74b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a94163070 con 0x7f5a94153570 2024-09-19T07:46:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a9a74b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a94162ef0 con 0x7f5a940c5110 2024-09-19T07:46:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a9a74b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5a941631f0 con 0x7f5a940c4130 2024-09-19T07:46:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a98ce8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a94153570 0x7f5a9414d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a98ce8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a94153570 0x7f5a9414d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52760/0 (socket says 172.21.15.83:52760) 2024-09-19T07:46:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a937fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a940c5110 0x7f5a9414cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a98ce8640 1 -- 172.21.15.83:0/4031488309 learned_addr learned my addr 172.21.15.83:0/4031488309 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:46:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a93fff640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a940c4130 0x7f5a9414f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a937fe640 1 -- 172.21.15.83:0/4031488309 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a940c4130 msgr2=0x7f5a9414f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a937fe640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a940c4130 0x7f5a9414f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a937fe640 1 -- 172.21.15.83:0/4031488309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a94153570 msgr2=0x7f5a9414d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a937fe640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a94153570 0x7f5a9414d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a937fe640 1 -- 172.21.15.83:0/4031488309 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a941f1f90 con 0x7f5a940c5110 2024-09-19T07:46:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a937fe640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a940c5110 0x7f5a9414cce0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5a880ac000 tx=0x7f5a8801ac60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:46:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a90ff9640 1 -- 172.21.15.83:0/4031488309 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a880be5f0 con 0x7f5a940c5110 2024-09-19T07:46:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a90ff9640 1 -- 172.21.15.83:0/4031488309 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a880be7d0 con 0x7f5a940c5110 2024-09-19T07:46:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a9a74b640 1 -- 172.21.15.83:0/4031488309 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a941f2270 con 0x7f5a940c5110 2024-09-19T07:46:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a90ff9640 1 -- 172.21.15.83:0/4031488309 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a880bebc0 con 0x7f5a940c5110 2024-09-19T07:46:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.259+0000 7f5a9a74b640 1 -- 172.21.15.83:0/4031488309 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a941f2830 con 0x7f5a940c5110 2024-09-19T07:46:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.263+0000 7f5a9a74b640 1 -- 172.21.15.83:0/4031488309 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a94069650 con 0x7f5a940c5110 2024-09-19T07:46:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.263+0000 7f5a90ff9640 1 -- 172.21.15.83:0/4031488309 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5a880b43f0 con 0x7f5a940c5110 2024-09-19T07:46:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.263+0000 7f5a90ff9640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a800678c0 0x7f5a80069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.263+0000 7f5a90ff9640 1 -- 172.21.15.83:0/4031488309 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9733+0+0 (secure 0 0 0) 0x7f5a88121510 con 0x7f5a940c5110 2024-09-19T07:46:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.263+0000 7f5a93fff640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a800678c0 0x7f5a80069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.263+0000 7f5a93fff640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a800678c0 0x7f5a80069d80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400c900 tx=0x7f5a84008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:46:54.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.267+0000 7f5a90ff9640 1 -- 172.21.15.83:0/4031488309 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5a88016130 con 0x7f5a940c5110 2024-09-19T07:46:54.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.555+0000 7f5a9a74b640 1 -- 172.21.15.83:0/4031488309 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a94071950 con 0x7f5a940c5110 2024-09-19T07:46:54.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.555+0000 7f5a90ff9640 1 -- 172.21.15.83:0/4031488309 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+6542 (secure 0 0 0) 0x7f5a880f1e20 con 0x7f5a940c5110 2024-09-19T07:46:54.554 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:46:54.554 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":185,"btime":"2024-09-19T07:45:41:636892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8028,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":185}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:46:54.554 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 185 2024-09-19T07:46:54.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.559+0000 7f5a9a74b640 1 -- 172.21.15.83:0/4031488309 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a800678c0 msgr2=0x7f5a80069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:54.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.559+0000 7f5a9a74b640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a800678c0 0x7f5a80069d80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400c900 tx=0x7f5a84008040 comp rx=0 tx=0).stop 2024-09-19T07:46:54.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.559+0000 7f5a9a74b640 1 -- 172.21.15.83:0/4031488309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a940c5110 msgr2=0x7f5a9414cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:54.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.559+0000 7f5a9a74b640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a940c5110 0x7f5a9414cce0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5a880ac000 tx=0x7f5a8801ac60 comp rx=0 tx=0).stop 2024-09-19T07:46:54.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.559+0000 7f5a9a74b640 1 -- 172.21.15.83:0/4031488309 shutdown_connections 2024-09-19T07:46:54.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.559+0000 7f5a9a74b640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f5a800678c0 0x7f5a80069d80 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:54.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.559+0000 7f5a9a74b640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a94153570 0x7f5a9414d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:54.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.559+0000 7f5a9a74b640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a940c5110 0x7f5a9414cce0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:54.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.559+0000 7f5a9a74b640 1 --2- 172.21.15.83:0/4031488309 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a940c4130 0x7f5a9414f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:54.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.559+0000 7f5a9a74b640 1 -- 172.21.15.83:0/4031488309 >> 172.21.15.83:0/4031488309 conn(0x7f5a940c6110 msgr2=0x7f5a941593e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:46:54.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.559+0000 7f5a9a74b640 1 -- 172.21.15.83:0/4031488309 shutdown_connections 2024-09-19T07:46:54.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:54.559+0000 7f5a9a74b640 1 -- 172.21.15.83:0/4031488309 wait complete. 2024-09-19T07:46:54.582 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:46:56.583 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:46:56.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.707+0000 7f06af7c7640 1 Processor -- start 2024-09-19T07:46:56.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.707+0000 7f06af7c7640 1 -- start start 2024-09-19T07:46:56.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.707+0000 7f06af7c7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f06a80c28e0 0x7f06a81545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:56.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06af7c7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a806dea0 0x7f06a806e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:56.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06af7c7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06a80a9e60 con 0x7f06a8071100 2024-09-19T07:46:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06af7c7640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f06a8071b50 con 0x7f06a80c28e0 2024-09-19T07:46:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06af7c7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f06a80c1ff0 con 0x7f06a806dea0 2024-09-19T07:46:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06add64640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a806dea0 0x7f06a806e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06acd62640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f06a80c28e0 0x7f06a81545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06add64640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a806dea0 0x7f06a806e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:56334/0 (socket says 172.21.15.83:56334) 2024-09-19T07:46:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06add64640 1 -- 172.21.15.83:0/1074162540 learned_addr learned my addr 172.21.15.83:0/1074162540 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:46:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06add64640 1 -- 172.21.15.83:0/1074162540 >> v2:172.21.15.184:3300/0 conn(0x7f06a80c28e0 msgr2=0x7f06a81545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:46:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06add64640 1 --2- 172.21.15.83:0/1074162540 >> v2:172.21.15.184:3300/0 conn(0x7f06a80c28e0 0x7f06a81545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06add64640 1 -- 172.21.15.83:0/1074162540 >> v1:172.21.15.184:6789/0 conn(0x7f06a8071100 legacy=0x7f06a8150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06add64640 1 -- 172.21.15.83:0/1074162540 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a80aa140 con 0x7f06a806dea0 2024-09-19T07:46:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f06add64640 1 --2- 172.21.15.83:0/1074162540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a806dea0 0x7f06a806e330 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009870 tx=0x7f06a001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c01663437b474fe3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:46:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f069ffff640 1 -- 172.21.15.83:0/1074162540 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a00b1070 con 0x7f06a806dea0 2024-09-19T07:46:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f069ffff640 1 -- 172.21.15.83:0/1074162540 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a00ac620 con 0x7f06a806dea0 2024-09-19T07:46:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.711+0000 7f069ffff640 1 -- 172.21.15.83:0/1074162540 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a00ac9f0 con 0x7f06a806dea0 2024-09-19T07:46:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 -- 172.21.15.83:0/1074162540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a806dea0 msgr2=0x7f06a806e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 --2- 172.21.15.83:0/1074162540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a806dea0 0x7f06a806e330 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009870 tx=0x7f06a001c4d0 comp rx=0 tx=0).stop 2024-09-19T07:46:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 -- 172.21.15.83:0/1074162540 shutdown_connections 2024-09-19T07:46:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 --2- 172.21.15.83:0/1074162540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a806dea0 0x7f06a806e330 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 --2- 172.21.15.83:0/1074162540 >> v2:172.21.15.184:3300/0 conn(0x7f06a80c28e0 0x7f06a81545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 -- 172.21.15.83:0/1074162540 >> 172.21.15.83:0/1074162540 conn(0x7f06a806bc80 msgr2=0x7f06a806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:46:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 -- 172.21.15.83:0/1074162540 shutdown_connections 2024-09-19T07:46:56.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 -- 172.21.15.83:0/1074162540 wait complete. 2024-09-19T07:46:56.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 Processor -- start 2024-09-19T07:46:56.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 -- start start 2024-09-19T07:46:56.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06a806dea0 0x7f06a8156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:56.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06a8071100 0x7f06a8157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:56.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a80c28e0 0x7f06a8152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:56.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f06a8071660 con 0x7f06a806dea0 2024-09-19T07:46:56.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f06a80714e0 con 0x7f06a8071100 2024-09-19T07:46:56.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f06a8160c60 con 0x7f06a80c28e0 2024-09-19T07:46:56.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06ad563640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06a806dea0 0x7f06a8156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:56.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06ad563640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06a806dea0 0x7f06a8156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52766/0 (socket says 172.21.15.83:52766) 2024-09-19T07:46:56.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06ad563640 1 -- 172.21.15.83:0/3420733872 learned_addr learned my addr 172.21.15.83:0/3420733872 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:46:56.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06add64640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a80c28e0 0x7f06a8152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:56.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06acd62640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06a8071100 0x7f06a8157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:56.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06ad563640 1 -- 172.21.15.83:0/3420733872 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a80c28e0 msgr2=0x7f06a8152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:56.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06ad563640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a80c28e0 0x7f06a8152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:56.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06ad563640 1 -- 172.21.15.83:0/3420733872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06a8071100 msgr2=0x7f06a8157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:56.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06ad563640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06a8071100 0x7f06a8157220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:56.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06ad563640 1 -- 172.21.15.83:0/3420733872 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a8151ba0 con 0x7f06a806dea0 2024-09-19T07:46:56.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06ad563640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06a806dea0 0x7f06a8156ce0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c790 tx=0x7f06a400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:46:56.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f069dffb640 1 -- 172.21.15.83:0/3420733872 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4016020 con 0x7f06a806dea0 2024-09-19T07:46:56.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f069dffb640 1 -- 172.21.15.83:0/3420733872 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a4006d60 con 0x7f06a806dea0 2024-09-19T07:46:56.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 -- 172.21.15.83:0/3420733872 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a8153b20 con 0x7f06a806dea0 2024-09-19T07:46:56.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f069dffb640 1 -- 172.21.15.83:0/3420733872 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4003bc0 con 0x7f06a806dea0 2024-09-19T07:46:56.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.715+0000 7f06af7c7640 1 -- 172.21.15.83:0/3420733872 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a80c7020 con 0x7f06a806dea0 2024-09-19T07:46:56.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.719+0000 7f06af7c7640 1 -- 172.21.15.83:0/3420733872 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a81ece90 con 0x7f06a806dea0 2024-09-19T07:46:56.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.719+0000 7f069dffb640 1 -- 172.21.15.83:0/3420733872 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f06a400a850 con 0x7f06a806dea0 2024-09-19T07:46:56.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.719+0000 7f069dffb640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f068c0678c0 0x7f068c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:46:56.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.719+0000 7f069dffb640 1 -- 172.21.15.83:0/3420733872 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9733+0+0 (secure 0 0 0) 0x7f06a4086860 con 0x7f06a806dea0 2024-09-19T07:46:56.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.719+0000 7f06acd62640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f068c0678c0 0x7f068c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:46:56.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.719+0000 7f06acd62640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f068c0678c0 0x7f068c069d80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f0698009ac0 tx=0x7f0698009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:46:56.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.723+0000 7f069dffb640 1 -- 172.21.15.83:0/3420733872 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f06a4020020 con 0x7f06a806dea0 2024-09-19T07:46:56.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.923+0000 7f06af7c7640 1 -- 172.21.15.83:0/3420733872 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06a806e330 con 0x7f06a806dea0 2024-09-19T07:46:56.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.923+0000 7f069dffb640 1 -- 172.21.15.83:0/3420733872 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+6542 (secure 0 0 0) 0x7f06a4055d70 con 0x7f06a806dea0 2024-09-19T07:46:56.922 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:46:56.922 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":185,"btime":"2024-09-19T07:45:41:636892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8028,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":185}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:46:56.922 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 185 2024-09-19T07:46:56.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.927+0000 7f06af7c7640 1 -- 172.21.15.83:0/3420733872 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f068c0678c0 msgr2=0x7f068c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.927+0000 7f06af7c7640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f068c0678c0 0x7f068c069d80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f0698009ac0 tx=0x7f0698009190 comp rx=0 tx=0).stop 2024-09-19T07:46:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.927+0000 7f06af7c7640 1 -- 172.21.15.83:0/3420733872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06a806dea0 msgr2=0x7f06a8156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:46:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.927+0000 7f06af7c7640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06a806dea0 0x7f06a8156ce0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c790 tx=0x7f06a400cc50 comp rx=0 tx=0).stop 2024-09-19T07:46:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.927+0000 7f06af7c7640 1 -- 172.21.15.83:0/3420733872 shutdown_connections 2024-09-19T07:46:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.927+0000 7f06af7c7640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f068c0678c0 0x7f068c069d80 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.927+0000 7f06af7c7640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a80c28e0 0x7f06a8152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.927+0000 7f06af7c7640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06a8071100 0x7f06a8157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.927+0000 7f06af7c7640 1 --2- 172.21.15.83:0/3420733872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06a806dea0 0x7f06a8156ce0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:46:56.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.927+0000 7f06af7c7640 1 -- 172.21.15.83:0/3420733872 >> 172.21.15.83:0/3420733872 conn(0x7f06a806bc80 msgr2=0x7f06a8150f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:46:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.927+0000 7f06af7c7640 1 -- 172.21.15.83:0/3420733872 shutdown_connections 2024-09-19T07:46:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:46:56.927+0000 7f06af7c7640 1 -- 172.21.15.83:0/3420733872 wait complete. 2024-09-19T07:46:56.941 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:46:58.943 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:00.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b3e277640 1 Processor -- start 2024-09-19T07:47:00.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b3e277640 1 -- start start 2024-09-19T07:47:00.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b3e277640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f3b38150290 0x7f3b380c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:00.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b3e277640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3b38152b20 0x7f3b380c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:00.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b3e277640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b380c45c0 0x7f3b38153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:00.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b3e277640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3b380716c0 con 0x7f3b38152b20 2024-09-19T07:47:00.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b3e277640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f3b380c3920 con 0x7f3b38150290 2024-09-19T07:47:00.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b3e277640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3b380c3490 con 0x7f3b380c45c0 2024-09-19T07:47:00.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b2f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3b38152b20 0x7f3b380c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:00.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b2f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3b38152b20 0x7f3b380c4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43566/0 (socket says 172.21.15.83:43566) 2024-09-19T07:47:00.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b2f7fe640 1 -- 172.21.15.83:0/3998932060 learned_addr learned my addr 172.21.15.83:0/3998932060 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:00.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b37fff640 1 --2- 172.21.15.83:0/3998932060 >> v2:172.21.15.184:3300/0 conn(0x7f3b38150290 0x7f3b380c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:00.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b2f7fe640 1 -- 172.21.15.83:0/3998932060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b380c45c0 msgr2=0x7f3b38153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:00.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b2f7fe640 1 --2- 172.21.15.83:0/3998932060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b380c45c0 0x7f3b38153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b2f7fe640 1 -- 172.21.15.83:0/3998932060 >> v2:172.21.15.184:3300/0 conn(0x7f3b38150290 msgr2=0x7f3b380c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:00.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b2f7fe640 1 --2- 172.21.15.83:0/3998932060 >> v2:172.21.15.184:3300/0 conn(0x7f3b38150290 0x7f3b380c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b2f7fe640 1 -- 172.21.15.83:0/3998932060 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b38150670 con 0x7f3b38152b20 2024-09-19T07:47:00.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.479+0000 7f3b37fff640 1 --2- 172.21.15.83:0/3998932060 >> v2:172.21.15.184:3300/0 conn(0x7f3b38150290 0x7f3b380c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:47:00.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b2f7fe640 1 --2- 172.21.15.83:0/3998932060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b38152b20 0x7f3b380c4080 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f3b3806f590 tx=0x7f3b2001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570b38ae67410dd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:00.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b377fe640 1 -- 172.21.15.83:0/3998932060 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b200aa610 con 0x7f3b38152b20 2024-09-19T07:47:00.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b377fe640 1 -- 172.21.15.83:0/3998932060 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b200af070 con 0x7f3b38152b20 2024-09-19T07:47:00.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b377fe640 1 -- 172.21.15.83:0/3998932060 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b200aa9d0 con 0x7f3b38152b20 2024-09-19T07:47:00.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 -- 172.21.15.83:0/3998932060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b38152b20 msgr2=0x7f3b380c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:00.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 --2- 172.21.15.83:0/3998932060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b38152b20 0x7f3b380c4080 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f3b3806f590 tx=0x7f3b2001c4f0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 -- 172.21.15.83:0/3998932060 shutdown_connections 2024-09-19T07:47:00.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 --2- 172.21.15.83:0/3998932060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b380c45c0 0x7f3b38153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 --2- 172.21.15.83:0/3998932060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b38152b20 0x7f3b380c4080 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 --2- 172.21.15.83:0/3998932060 >> v2:172.21.15.184:3300/0 conn(0x7f3b38150290 0x7f3b380c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 -- 172.21.15.83:0/3998932060 >> 172.21.15.83:0/3998932060 conn(0x7f3b3806bc80 msgr2=0x7f3b3806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:00.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 -- 172.21.15.83:0/3998932060 shutdown_connections 2024-09-19T07:47:00.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 -- 172.21.15.83:0/3998932060 wait complete. 2024-09-19T07:47:00.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 Processor -- start 2024-09-19T07:47:00.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 -- start start 2024-09-19T07:47:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b380c45c0 0x7f3b381eebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b38150290 0x7f3b38147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b38152b20 0x7f3b381508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3b380c2530 con 0x7f3b38152b20 2024-09-19T07:47:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3b380c3120 con 0x7f3b380c45c0 2024-09-19T07:47:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.483+0000 7f3b3e277640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3b38154800 con 0x7f3b38150290 2024-09-19T07:47:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b3c814640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b38152b20 0x7f3b381508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b3c814640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b38152b20 0x7f3b381508b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43578/0 (socket says 172.21.15.83:43578) 2024-09-19T07:47:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b3c814640 1 -- 172.21.15.83:0/1213508823 learned_addr learned my addr 172.21.15.83:0/1213508823 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b2f7fe640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b38150290 0x7f3b38147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:00.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b37fff640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b380c45c0 0x7f3b381eebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:00.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b3c814640 1 -- 172.21.15.83:0/1213508823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b38150290 msgr2=0x7f3b38147ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:00.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b3c814640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b38150290 0x7f3b38147ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b3c814640 1 -- 172.21.15.83:0/1213508823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b380c45c0 msgr2=0x7f3b381eebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:00.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b3c814640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b380c45c0 0x7f3b381eebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b3c814640 1 -- 172.21.15.83:0/1213508823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b38151240 con 0x7f3b38152b20 2024-09-19T07:47:00.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b3c814640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b38152b20 0x7f3b381508b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000dbc0 tx=0x7f3b3000dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:00.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b357fa640 1 -- 172.21.15.83:0/1213508823 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000e980 con 0x7f3b38152b20 2024-09-19T07:47:00.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b357fa640 1 -- 172.21.15.83:0/1213508823 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b3000eb40 con 0x7f3b38152b20 2024-09-19T07:47:00.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b3e277640 1 -- 172.21.15.83:0/1213508823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b38151890 con 0x7f3b38152b20 2024-09-19T07:47:00.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b357fa640 1 -- 172.21.15.83:0/1213508823 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b30009580 con 0x7f3b38152b20 2024-09-19T07:47:00.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b3e277640 1 -- 172.21.15.83:0/1213508823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b380c6720 con 0x7f3b38152b20 2024-09-19T07:47:00.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b3e277640 1 -- 172.21.15.83:0/1213508823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b380c1ff0 con 0x7f3b38152b20 2024-09-19T07:47:00.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b357fa640 1 -- 172.21.15.83:0/1213508823 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3b30009720 con 0x7f3b38152b20 2024-09-19T07:47:00.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b357fa640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3b180678c0 0x7f3b18069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:00.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b37fff640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3b180678c0 0x7f3b18069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:00.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b357fa640 1 -- 172.21.15.83:0/1213508823 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9733+0+0 (secure 0 0 0) 0x7f3b30088990 con 0x7f3b38152b20 2024-09-19T07:47:00.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.487+0000 7f3b37fff640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3b180678c0 0x7f3b18069d80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800ab60 tx=0x7f3b28008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:00.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.491+0000 7f3b357fa640 1 -- 172.21.15.83:0/1213508823 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3b30057de0 con 0x7f3b38152b20 2024-09-19T07:47:00.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.691+0000 7f3b3e277640 1 -- 172.21.15.83:0/1213508823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b38151a60 con 0x7f3b38152b20 2024-09-19T07:47:00.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.691+0000 7f3b357fa640 1 -- 172.21.15.83:0/1213508823 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+6542 (secure 0 0 0) 0x7f3b3005c0e0 con 0x7f3b38152b20 2024-09-19T07:47:00.689 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:00.689 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":185,"btime":"2024-09-19T07:45:41:636892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8028,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":185}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:47:00.690 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 185 2024-09-19T07:47:00.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.695+0000 7f3b3e277640 1 -- 172.21.15.83:0/1213508823 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3b180678c0 msgr2=0x7f3b18069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:00.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.695+0000 7f3b3e277640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3b180678c0 0x7f3b18069d80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800ab60 tx=0x7f3b28008040 comp rx=0 tx=0).stop 2024-09-19T07:47:00.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.695+0000 7f3b3e277640 1 -- 172.21.15.83:0/1213508823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b38152b20 msgr2=0x7f3b381508b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:00.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.695+0000 7f3b3e277640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b38152b20 0x7f3b381508b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000dbc0 tx=0x7f3b3000dcb0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.695+0000 7f3b3e277640 1 -- 172.21.15.83:0/1213508823 shutdown_connections 2024-09-19T07:47:00.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.695+0000 7f3b3e277640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3b180678c0 0x7f3b18069d80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.695+0000 7f3b3e277640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b38152b20 0x7f3b381508b0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.695+0000 7f3b3e277640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b38150290 0x7f3b38147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.695+0000 7f3b3e277640 1 --2- 172.21.15.83:0/1213508823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b380c45c0 0x7f3b381eebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:00.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.695+0000 7f3b3e277640 1 -- 172.21.15.83:0/1213508823 >> 172.21.15.83:0/1213508823 conn(0x7f3b3806bc80 msgr2=0x7f3b3815c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:00.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.695+0000 7f3b3e277640 1 -- 172.21.15.83:0/1213508823 shutdown_connections 2024-09-19T07:47:00.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:00.695+0000 7f3b3e277640 1 -- 172.21.15.83:0/1213508823 wait complete. 2024-09-19T07:47:00.708 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:47:02.709 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:02.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.839+0000 7f4e303bc640 1 Processor -- start 2024-09-19T07:47:02.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e303bc640 1 -- start start 2024-09-19T07:47:02.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e303bc640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4e280c28e0 0x7f4e281545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:02.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e303bc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e280a9e60 con 0x7f4e28071100 2024-09-19T07:47:02.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e303bc640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e280aa140 con 0x7f4e2806dea0 2024-09-19T07:47:02.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e303bc640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4e28071b50 con 0x7f4e280c28e0 2024-09-19T07:47:02.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2e158640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4e28071100 0x7f4e28150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60774/0 (socket says 172.21.15.83:60774) 2024-09-19T07:47:02.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2e158640 1 -- 172.21.15.83:0/3451470978 learned_addr learned my addr 172.21.15.83:0/3451470978 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:47:02.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1813729210 0 0) 0x7f4e280a9e60 con 0x7f4e28071100 2024-09-19T07:47:02.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d957640 1 --2- 172.21.15.83:0/3451470978 >> v2:172.21.15.184:3300/0 conn(0x7f4e280c28e0 0x7f4e281545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:02.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e14003530 con 0x7f4e28071100 2024-09-19T07:47:02.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 401527684 0 0) 0x7f4e14003530 con 0x7f4e28071100 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 >> v2:172.21.15.184:3300/0 conn(0x7f4e280c28e0 msgr2=0x7f4e281545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 --2- 172.21.15.83:0/3451470978 >> v2:172.21.15.184:3300/0 conn(0x7f4e280c28e0 0x7f4e281545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 >> v1:172.21.15.184:6789/0 conn(0x7f4e2806dea0 legacy=0x7f4e2806e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e280c2240 con 0x7f4e28071100 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f4e18001680 con 0x7f4e28071100 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 >> v1:172.21.15.83:6789/0 conn(0x7f4e28071100 legacy=0x7f4e28150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 --2- 172.21.15.83:0/3451470978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e140039c0 0x7f4e14003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 --2- 172.21.15.83:0/3451470978 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e14004810 0x7f4e14004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 --2- 172.21.15.83:0/3451470978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 0x7f4e1400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4e28071b50 con 0x7f4e140052f0 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1400c1c0 con 0x7f4e140039c0 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4e1400c3c0 con 0x7f4e14004810 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2d957640 1 --2- 172.21.15.83:0/3451470978 >> v2:172.21.15.184:3300/0 conn(0x7f4e280c28e0 0x7f4e281545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:47:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2e959640 1 --2- 172.21.15.83:0/3451470978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 0x7f4e1400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:02.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2e158640 1 --2- 172.21.15.83:0/3451470978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e140039c0 0x7f4e14003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:02.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2e959640 1 -- 172.21.15.83:0/3451470978 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e14004810 msgr2=0x7f4e14004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:02.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2e959640 1 --2- 172.21.15.83:0/3451470978 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e14004810 0x7f4e14004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:02.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2e959640 1 -- 172.21.15.83:0/3451470978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e140039c0 msgr2=0x7f4e14003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:02.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2e959640 1 --2- 172.21.15.83:0/3451470978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e140039c0 0x7f4e14003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:02.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2e959640 1 -- 172.21.15.83:0/3451470978 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e14003530 con 0x7f4e140052f0 2024-09-19T07:47:02.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.843+0000 7f4e2e158640 1 --2- 172.21.15.83:0/3451470978 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e140039c0 0x7f4e14003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:47:02.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2e959640 1 -- 172.21.15.83:0/3451470978 reap_dead start 2024-09-19T07:47:02.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2e959640 1 --2- 172.21.15.83:0/3451470978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 0x7f4e1400b9d0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f4e20001e40 tx=0x7f4e2001c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7fb7cfbdfaae559 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:02.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e20001a90 con 0x7f4e140052f0 2024-09-19T07:47:02.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e20003b80 con 0x7f4e140052f0 2024-09-19T07:47:02.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2d156640 1 -- 172.21.15.83:0/3451470978 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e20005020 con 0x7f4e140052f0 2024-09-19T07:47:02.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 -- 172.21.15.83:0/3451470978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 msgr2=0x7f4e1400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 --2- 172.21.15.83:0/3451470978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 0x7f4e1400b9d0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f4e20001e40 tx=0x7f4e2001c760 comp rx=0 tx=0).stop 2024-09-19T07:47:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 -- 172.21.15.83:0/3451470978 shutdown_connections 2024-09-19T07:47:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 --2- 172.21.15.83:0/3451470978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 0x7f4e1400b9d0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 -- 172.21.15.83:0/3451470978 >> 172.21.15.83:0/3451470978 conn(0x7f4e2806bc80 msgr2=0x7f4e2806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 -- 172.21.15.83:0/3451470978 shutdown_connections 2024-09-19T07:47:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 -- 172.21.15.83:0/3451470978 wait complete. 2024-09-19T07:47:02.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 Processor -- start 2024-09-19T07:47:02.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 -- start start 2024-09-19T07:47:02.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 0x7f4e281ed1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:02.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e280c6440 0x7f4e281dd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:02.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e28142490 0x7f4e281500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:02.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4e280714e0 con 0x7f4e140052f0 2024-09-19T07:47:02.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4e280c1ff0 con 0x7f4e280c6440 2024-09-19T07:47:02.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4e28071660 con 0x7f4e28142490 2024-09-19T07:47:02.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2e158640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 0x7f4e281ed1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:02.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2e158640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 0x7f4e281ed1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43590/0 (socket says 172.21.15.83:43590) 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2e158640 1 -- 172.21.15.83:0/143533293 learned_addr learned my addr 172.21.15.83:0/143533293 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2e158640 1 -- 172.21.15.83:0/143533293 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e28142490 msgr2=0x7f4e281500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2e959640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e28142490 0x7f4e281500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2e158640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e28142490 0x7f4e281500a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2e158640 1 -- 172.21.15.83:0/143533293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e280c6440 msgr2=0x7f4e281dd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2e158640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e280c6440 0x7f4e281dd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2e158640 1 -- 172.21.15.83:0/143533293 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e281ed6f0 con 0x7f4e140052f0 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e2e158640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 0x7f4e281ed1b0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f4e18000c10 tx=0x7f4e1800e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e1effd640 1 -- 172.21.15.83:0/143533293 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1801b070 con 0x7f4e140052f0 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e1effd640 1 -- 172.21.15.83:0/143533293 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e18011040 con 0x7f4e140052f0 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 -- 172.21.15.83:0/143533293 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e280c68e0 con 0x7f4e140052f0 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e1effd640 1 -- 172.21.15.83:0/143533293 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e180143a0 con 0x7f4e140052f0 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.847+0000 7f4e303bc640 1 -- 172.21.15.83:0/143533293 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e281ec9b0 con 0x7f4e140052f0 2024-09-19T07:47:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.851+0000 7f4e303bc640 1 -- 172.21.15.83:0/143533293 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e280c6e00 con 0x7f4e140052f0 2024-09-19T07:47:02.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.851+0000 7f4e1effd640 1 -- 172.21.15.83:0/143533293 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4e1801e070 con 0x7f4e140052f0 2024-09-19T07:47:02.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.851+0000 7f4e1effd640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4e0c0678f0 0x7f4e0c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:02.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.851+0000 7f4e1effd640 1 -- 172.21.15.83:0/143533293 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9733+0+0 (secure 0 0 0) 0x7f4e18089770 con 0x7f4e140052f0 2024-09-19T07:47:02.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.851+0000 7f4e2d957640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4e0c0678f0 0x7f4e0c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:02.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.851+0000 7f4e2d957640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4e0c0678f0 0x7f4e0c069db0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4e2814e610 tx=0x7f4e24008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:02.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:02.855+0000 7f4e1effd640 1 -- 172.21.15.83:0/143533293 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4e280c6e00 con 0x7f4e140052f0 2024-09-19T07:47:03.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.047+0000 7f4e303bc640 1 -- 172.21.15.83:0/143533293 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e28154670 con 0x7f4e140052f0 2024-09-19T07:47:03.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e1effd640 1 -- 172.21.15.83:0/143533293 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+6542 (secure 0 0 0) 0x7f4e18059330 con 0x7f4e140052f0 2024-09-19T07:47:03.047 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:03.047 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":185,"btime":"2024-09-19T07:45:41:636892+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8028,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":185}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:45:16.742872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6152":{"gid":6152,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6833/1207928072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1207928072},{"type":"v1","addr":"172.21.15.184:6833","nonce":1207928072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6152,"qdb_cluster":[6152]},"id":2}]} 2024-09-19T07:47:03.048 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 185 2024-09-19T07:47:03.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e303bc640 1 -- 172.21.15.83:0/143533293 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4e0c0678f0 msgr2=0x7f4e0c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:03.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e303bc640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4e0c0678f0 0x7f4e0c069db0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4e2814e610 tx=0x7f4e24008040 comp rx=0 tx=0).stop 2024-09-19T07:47:03.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e303bc640 1 -- 172.21.15.83:0/143533293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 msgr2=0x7f4e281ed1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:03.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e303bc640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 0x7f4e281ed1b0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f4e18000c10 tx=0x7f4e1800e4d0 comp rx=0 tx=0).stop 2024-09-19T07:47:03.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e303bc640 1 -- 172.21.15.83:0/143533293 shutdown_connections 2024-09-19T07:47:03.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e303bc640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4e0c0678f0 0x7f4e0c069db0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:03.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e303bc640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e28142490 0x7f4e281500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:03.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e303bc640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e280c6440 0x7f4e281dd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:03.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e303bc640 1 --2- 172.21.15.83:0/143533293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e140052f0 0x7f4e281ed1b0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:03.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e303bc640 1 -- 172.21.15.83:0/143533293 >> 172.21.15.83:0/143533293 conn(0x7f4e2806bc80 msgr2=0x7f4e2806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:03.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e303bc640 1 -- 172.21.15.83:0/143533293 shutdown_connections 2024-09-19T07:47:03.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:03.051+0000 7f4e303bc640 1 -- 172.21.15.83:0/143533293 wait complete. 2024-09-19T07:47:03.065 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:47:05.067 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:05.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.203+0000 7fbc048f3640 1 Processor -- start 2024-09-19T07:47:05.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.203+0000 7fbc048f3640 1 -- start start 2024-09-19T07:47:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbc048f3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc00071100 0x7fbc001545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbc048f3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fbc0006dea0 0x7fbc0006e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbc048f3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc000a9e60 con 0x7fbc000c28e0 2024-09-19T07:47:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbc048f3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fbc00071b50 con 0x7fbc0006dea0 2024-09-19T07:47:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbc048f3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbc000c1ff0 con 0x7fbc00071100 2024-09-19T07:47:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfdd9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbc000c28e0 0x7fbc00150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60784/0 (socket says 172.21.15.83:60784) 2024-09-19T07:47:05.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfdd9b640 1 -- 172.21.15.83:0/3316751213 learned_addr learned my addr 172.21.15.83:0/3316751213 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:47:05.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfed9d640 1 --2- 172.21.15.83:0/3316751213 >> v2:172.21.15.184:3300/0 conn(0x7fbc0006dea0 0x7fbc0006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2604611078 0 0) 0x7fbc000a9e60 con 0x7fbc000c28e0 2024-09-19T07:47:05.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfe59c640 1 --2- 172.21.15.83:0/3316751213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc00071100 0x7fbc001545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbe8003530 con 0x7fbc000c28e0 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3843174191 0 0) 0x7fbbe8003530 con 0x7fbc000c28e0 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc00071100 msgr2=0x7fbc001545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 --2- 172.21.15.83:0/3316751213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc00071100 0x7fbc001545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 >> v2:172.21.15.184:3300/0 conn(0x7fbc0006dea0 msgr2=0x7fbc0006e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 --2- 172.21.15.83:0/3316751213 >> v2:172.21.15.184:3300/0 conn(0x7fbc0006dea0 0x7fbc0006e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc000aa140 con 0x7fbc000c28e0 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fbbf0001680 con 0x7fbc000c28e0 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 >> v1:172.21.15.83:6789/0 conn(0x7fbc000c28e0 legacy=0x7fbc00150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 --2- 172.21.15.83:0/3316751213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbe80039c0 0x7fbbe8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 --2- 172.21.15.83:0/3316751213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbe8004810 0x7fbbe8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 --2- 172.21.15.83:0/3316751213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbe80051d0 0x7fbbe800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbc00071b50 con 0x7fbbe80039c0 2024-09-19T07:47:05.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc000c1ff0 con 0x7fbbe80051d0 2024-09-19T07:47:05.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbbe800bff0 con 0x7fbbe8004810 2024-09-19T07:47:05.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfe59c640 1 --2- 172.21.15.83:0/3316751213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbe80039c0 0x7fbbe8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfed9d640 1 --2- 172.21.15.83:0/3316751213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbe80051d0 0x7fbbe800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfdd9b640 1 --2- 172.21.15.83:0/3316751213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbe8004810 0x7fbbe8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfe59c640 1 -- 172.21.15.83:0/3316751213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbe8004810 msgr2=0x7fbbe8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:05.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfe59c640 1 --2- 172.21.15.83:0/3316751213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbe8004810 0x7fbbe8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfe59c640 1 -- 172.21.15.83:0/3316751213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbe80051d0 msgr2=0x7fbbe800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:05.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfe59c640 1 --2- 172.21.15.83:0/3316751213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbe80051d0 0x7fbbe800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.207+0000 7fbbfe59c640 1 -- 172.21.15.83:0/3316751213 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe8003530 con 0x7fbbe80039c0 2024-09-19T07:47:05.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfe59c640 1 --2- 172.21.15.83:0/3316751213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbe80039c0 0x7fbbe8003de0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fbc00070820 tx=0x7fbbf401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7115b92cce4810f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:05.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf400b530 con 0x7fbbe80039c0 2024-09-19T07:47:05.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf40af620 con 0x7fbbe80039c0 2024-09-19T07:47:05.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfd59a640 1 -- 172.21.15.83:0/3316751213 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf40afa10 con 0x7fbbe80039c0 2024-09-19T07:47:05.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 -- 172.21.15.83:0/3316751213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbe80039c0 msgr2=0x7fbbe8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:05.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 --2- 172.21.15.83:0/3316751213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbe80039c0 0x7fbbe8003de0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fbc00070820 tx=0x7fbbf401c530 comp rx=0 tx=0).stop 2024-09-19T07:47:05.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfed9d640 1 -- 172.21.15.83:0/3316751213 reap_dead start 2024-09-19T07:47:05.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 -- 172.21.15.83:0/3316751213 shutdown_connections 2024-09-19T07:47:05.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 -- 172.21.15.83:0/3316751213 >> 172.21.15.83:0/3316751213 conn(0x7fbc0006bc80 msgr2=0x7fbc0006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:05.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 -- 172.21.15.83:0/3316751213 shutdown_connections 2024-09-19T07:47:05.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 -- 172.21.15.83:0/3316751213 wait complete. 2024-09-19T07:47:05.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 Processor -- start 2024-09-19T07:47:05.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 -- start start 2024-09-19T07:47:05.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbe80039c0 0x7fbc001eced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc000c5690 0x7fbc001573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc00157920 0x7fbc000c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001517d0 con 0x7fbc00157920 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc00151650 con 0x7fbc000c5690 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbc000714e0 con 0x7fbbe80039c0 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc00157920 0x7fbc000c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc00157920 0x7fbc000c68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43596/0 (socket says 172.21.15.83:43596) 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfed9d640 1 -- 172.21.15.83:0/3019362764 learned_addr learned my addr 172.21.15.83:0/3019362764 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfe59c640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbe80039c0 0x7fbc001eced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfdd9b640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc000c5690 0x7fbc001573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfed9d640 1 -- 172.21.15.83:0/3019362764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbe80039c0 msgr2=0x7fbc001eced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfed9d640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbe80039c0 0x7fbc001eced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfed9d640 1 -- 172.21.15.83:0/3019362764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc000c5690 msgr2=0x7fbc001573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfed9d640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc000c5690 0x7fbc001573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfed9d640 1 -- 172.21.15.83:0/3019362764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc001ed410 con 0x7fbc00157920 2024-09-19T07:47:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfdd9b640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc000c5690 0x7fbc001573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:47:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbfed9d640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc00157920 0x7fbc000c68f0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fbc001545b0 tx=0x7fbbf80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbeeffd640 1 -- 172.21.15.83:0/3019362764 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8011de0 con 0x7fbc00157920 2024-09-19T07:47:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbeeffd640 1 -- 172.21.15.83:0/3019362764 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf801b070 con 0x7fbc00157920 2024-09-19T07:47:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 -- 172.21.15.83:0/3019362764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc000c6130 con 0x7fbc00157920 2024-09-19T07:47:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbbeeffd640 1 -- 172.21.15.83:0/3019362764 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf80039a0 con 0x7fbc00157920 2024-09-19T07:47:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.211+0000 7fbc048f3640 1 -- 172.21.15.83:0/3019362764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc000c7f00 con 0x7fbc00157920 2024-09-19T07:47:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.215+0000 7fbc048f3640 1 -- 172.21.15.83:0/3019362764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc0006e2b0 con 0x7fbc00157920 2024-09-19T07:47:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.215+0000 7fbbeeffd640 1 -- 172.21.15.83:0/3019362764 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbbf8003b40 con 0x7fbc00157920 2024-09-19T07:47:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.215+0000 7fbbeeffd640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbbe00678c0 0x7fbbe0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.215+0000 7fbbeeffd640 1 -- 172.21.15.83:0/3019362764 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7fbbf8088280 con 0x7fbc00157920 2024-09-19T07:47:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.215+0000 7fbbfe59c640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbbe00678c0 0x7fbbe0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.215+0000 7fbbfe59c640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbbe00678c0 0x7fbbe0069d80 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40b0000 tx=0x7fbbf40046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:05.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.219+0000 7fbbeeffd640 1 -- 172.21.15.83:0/3019362764 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fbbf8025020 con 0x7fbc00157920 2024-09-19T07:47:05.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.415+0000 7fbc048f3640 1 -- 172.21.15.83:0/3019362764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc000c4f30 con 0x7fbc00157920 2024-09-19T07:47:05.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbbeeffd640 1 -- 172.21.15.83:0/3019362764 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+5709 (secure 0 0 0) 0x7fbbf8057e70 con 0x7fbc00157920 2024-09-19T07:47:05.415 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:05.415 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":187,"btime":"2024-09-19T07:47:04:695786+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:03.841766+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:47:05.415 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 187 2024-09-19T07:47:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbc048f3640 1 -- 172.21.15.83:0/3019362764 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbbe00678c0 msgr2=0x7fbbe0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbc048f3640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbbe00678c0 0x7fbbe0069d80 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40b0000 tx=0x7fbbf40046d0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbc048f3640 1 -- 172.21.15.83:0/3019362764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc00157920 msgr2=0x7fbc000c68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbc048f3640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc00157920 0x7fbc000c68f0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fbc001545b0 tx=0x7fbbf80115a0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbc048f3640 1 -- 172.21.15.83:0/3019362764 shutdown_connections 2024-09-19T07:47:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbc048f3640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fbbe00678c0 0x7fbbe0069d80 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbc048f3640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc00157920 0x7fbc000c68f0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbc048f3640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc000c5690 0x7fbc001573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbc048f3640 1 --2- 172.21.15.83:0/3019362764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbbe80039c0 0x7fbc001eced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbc048f3640 1 -- 172.21.15.83:0/3019362764 >> 172.21.15.83:0/3019362764 conn(0x7fbc0006bc80 msgr2=0x7fbc00073820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbc048f3640 1 -- 172.21.15.83:0/3019362764 shutdown_connections 2024-09-19T07:47:05.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.419+0000 7fbc048f3640 1 -- 172.21.15.83:0/3019362764 wait complete. 2024-09-19T07:47:05.434 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-19T07:47:05.434 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:47:05.434 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:05.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb625082640 1 Processor -- start 2024-09-19T07:47:05.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb625082640 1 -- start start 2024-09-19T07:47:05.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb625082640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb6200c28e0 0x7fb6201545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb625082640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6200a9e60 con 0x7fb62006dea0 2024-09-19T07:47:05.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb625082640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6200aa140 con 0x7fb620071100 2024-09-19T07:47:05.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb625082640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb620071b50 con 0x7fb6200c28e0 2024-09-19T07:47:05.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb61e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb6200c28e0 0x7fb6201545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb61f59e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb62006dea0 0x7fb62006e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60790/0 (socket says 172.21.15.83:60790) 2024-09-19T07:47:05.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb61f59e640 1 -- 172.21.15.83:0/2670646671 learned_addr learned my addr 172.21.15.83:0/2670646671 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:47:05.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb61e59c640 1 -- 172.21.15.83:0/2670646671 >> v1:172.21.15.184:6789/0 conn(0x7fb620071100 legacy=0x7fb620150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:05.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb61dd9b640 1 -- 172.21.15.83:0/2670646671 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1474339255 0 0) 0x7fb6200a9e60 con 0x7fb62006dea0 2024-09-19T07:47:05.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb61e59c640 1 -- 172.21.15.83:0/2670646671 >> v1:172.21.15.83:6789/0 conn(0x7fb62006dea0 legacy=0x7fb62006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:05.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.559+0000 7fb61e59c640 1 -- 172.21.15.83:0/2670646671 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6200c2240 con 0x7fb6200c28e0 2024-09-19T07:47:05.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61e59c640 1 --2- 172.21.15.83:0/2670646671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6200c28e0 0x7fb6201545e0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fb6140097c0 tx=0x7fb61401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=662138cb85ac3e54 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:05.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61dd9b640 1 -- 172.21.15.83:0/2670646671 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6140b0070 con 0x7fb6200c28e0 2024-09-19T07:47:05.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61dd9b640 1 -- 172.21.15.83:0/2670646671 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6140ab600 con 0x7fb6200c28e0 2024-09-19T07:47:05.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61dd9b640 1 -- 172.21.15.83:0/2670646671 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6140ab9d0 con 0x7fb6200c28e0 2024-09-19T07:47:05.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 -- 172.21.15.83:0/2670646671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6200c28e0 msgr2=0x7fb6201545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:05.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 --2- 172.21.15.83:0/2670646671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6200c28e0 0x7fb6201545e0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fb6140097c0 tx=0x7fb61401c410 comp rx=0 tx=0).stop 2024-09-19T07:47:05.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 -- 172.21.15.83:0/2670646671 shutdown_connections 2024-09-19T07:47:05.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 --2- 172.21.15.83:0/2670646671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6200c28e0 0x7fb6201545e0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 -- 172.21.15.83:0/2670646671 >> 172.21.15.83:0/2670646671 conn(0x7fb62006bc80 msgr2=0x7fb62006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:05.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 -- 172.21.15.83:0/2670646671 shutdown_connections 2024-09-19T07:47:05.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 -- 172.21.15.83:0/2670646671 wait complete. 2024-09-19T07:47:05.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 Processor -- start 2024-09-19T07:47:05.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 -- start start 2024-09-19T07:47:05.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb62006dea0 0x7fb6201efb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620071100 0x7fb620147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb6200c28e0 0x7fb620147e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb620071660 con 0x7fb620071100 2024-09-19T07:47:05.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb6200714e0 con 0x7fb62006dea0 2024-09-19T07:47:05.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb625082640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb620161c70 con 0x7fb6200c28e0 2024-09-19T07:47:05.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620071100 0x7fb620147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620071100 0x7fb620147910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43620/0 (socket says 172.21.15.83:43620) 2024-09-19T07:47:05.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61e59c640 1 -- 172.21.15.83:0/4142444709 learned_addr learned my addr 172.21.15.83:0/4142444709 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:05.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61f59e640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb6200c28e0 0x7fb620147e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61ed9d640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb62006dea0 0x7fb6201efb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61e59c640 1 -- 172.21.15.83:0/4142444709 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb6200c28e0 msgr2=0x7fb620147e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:05.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61e59c640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb6200c28e0 0x7fb620147e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61e59c640 1 -- 172.21.15.83:0/4142444709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb62006dea0 msgr2=0x7fb6201efb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61e59c640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb62006dea0 0x7fb6201efb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61e59c640 1 -- 172.21.15.83:0/4142444709 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb620148390 con 0x7fb620071100 2024-09-19T07:47:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.563+0000 7fb61e59c640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620071100 0x7fb620147910 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fb6140097c0 tx=0x7fb6140191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.567+0000 7fb6077fe640 1 -- 172.21.15.83:0/4142444709 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6140b9020 con 0x7fb620071100 2024-09-19T07:47:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.567+0000 7fb6077fe640 1 -- 172.21.15.83:0/4142444709 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6140b56a0 con 0x7fb620071100 2024-09-19T07:47:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.567+0000 7fb625082640 1 -- 172.21.15.83:0/4142444709 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62014aff0 con 0x7fb620071100 2024-09-19T07:47:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.567+0000 7fb6077fe640 1 -- 172.21.15.83:0/4142444709 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6140b0040 con 0x7fb620071100 2024-09-19T07:47:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.567+0000 7fb625082640 1 -- 172.21.15.83:0/4142444709 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb620152580 con 0x7fb620071100 2024-09-19T07:47:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.567+0000 7fb625082640 1 -- 172.21.15.83:0/4142444709 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6201545e0 con 0x7fb620071100 2024-09-19T07:47:05.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.567+0000 7fb6077fe640 1 -- 172.21.15.83:0/4142444709 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb6140bc020 con 0x7fb620071100 2024-09-19T07:47:05.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.567+0000 7fb6077fe640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb60c0678c0 0x7fb60c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:05.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.567+0000 7fb6077fe640 1 -- 172.21.15.83:0/4142444709 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7fb614120e30 con 0x7fb620071100 2024-09-19T07:47:05.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.567+0000 7fb61ed9d640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb60c0678c0 0x7fb60c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:05.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.567+0000 7fb61ed9d640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb60c0678c0 0x7fb60c069d80 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb610000f70 tx=0x7fb610009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:05.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.571+0000 7fb6077fe640 1 -- 172.21.15.83:0/4142444709 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb6140c3030 con 0x7fb620071100 2024-09-19T07:47:05.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.763+0000 7fb625082640 1 -- 172.21.15.83:0/4142444709 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb62004ffc0 con 0x7fb620071100 2024-09-19T07:47:05.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.763+0000 7fb6077fe640 1 -- 172.21.15.83:0/4142444709 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) ==== 95+0+5706 (secure 0 0 0) 0x7fb620148390 con 0x7fb620071100 2024-09-19T07:47:05.762 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:05.762 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":188,"btime":"2024-09-19T07:47:05:708136+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:04.717801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:47:05.762 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 188 2024-09-19T07:47:05.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.767+0000 7fb625082640 1 -- 172.21.15.83:0/4142444709 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb60c0678c0 msgr2=0x7fb60c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:05.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.767+0000 7fb625082640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb60c0678c0 0x7fb60c069d80 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb610000f70 tx=0x7fb610009040 comp rx=0 tx=0).stop 2024-09-19T07:47:05.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.767+0000 7fb625082640 1 -- 172.21.15.83:0/4142444709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620071100 msgr2=0x7fb620147910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:05.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.767+0000 7fb625082640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620071100 0x7fb620147910 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fb6140097c0 tx=0x7fb6140191b0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.767+0000 7fb625082640 1 -- 172.21.15.83:0/4142444709 shutdown_connections 2024-09-19T07:47:05.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.767+0000 7fb625082640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb60c0678c0 0x7fb60c069d80 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.767+0000 7fb625082640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb6200c28e0 0x7fb620147e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.767+0000 7fb625082640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620071100 0x7fb620147910 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.767+0000 7fb625082640 1 --2- 172.21.15.83:0/4142444709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb62006dea0 0x7fb6201efb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:05.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.767+0000 7fb625082640 1 -- 172.21.15.83:0/4142444709 >> 172.21.15.83:0/4142444709 conn(0x7fb62006bc80 msgr2=0x7fb62015ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:05.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.767+0000 7fb625082640 1 -- 172.21.15.83:0/4142444709 shutdown_connections 2024-09-19T07:47:05.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:05.767+0000 7fb625082640 1 -- 172.21.15.83:0/4142444709 wait complete. 2024-09-19T07:47:05.781 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:47:05:708136+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 188, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 184, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7968": { "addr": "172.21.15.83:6835/119695014", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 119695014, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 119695014, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7968, "incarnation": 181, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:45:31.612728+0000", "qdb_cluster": [ 7968 ], "qdb_leader": 7968, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7968 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 188, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8028": { "addr": "172.21.15.83:6837/1602509240", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 1602509240, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 1602509240, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8028, "incarnation": 186, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:47:04.717801+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8028 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:47:07.782 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:07.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.939+0000 7f1d397ee640 1 Processor -- start 2024-09-19T07:47:07.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.939+0000 7f1d397ee640 1 -- start start 2024-09-19T07:47:07.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d397ee640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d340ab3c0 0x7f1d34070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:07.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d397ee640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1d340c3360 0x7f1d34166330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:07.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d397ee640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d340aa230 con 0x7f1d340c3fa0 2024-09-19T07:47:07.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d397ee640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f1d34074190 con 0x7f1d340ab3c0 2024-09-19T07:47:07.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d397ee640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f1d340c1ff0 con 0x7f1d340c3360 2024-09-19T07:47:07.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d2bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d340ab3c0 0x7f1d34070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:07.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d2bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d340ab3c0 0x7f1d34070ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43622/0 (socket says 172.21.15.83:43622) 2024-09-19T07:47:07.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d2bfff640 1 -- 172.21.15.83:0/406391274 learned_addr learned my addr 172.21.15.83:0/406391274 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:07.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d337fe640 1 --2- 172.21.15.83:0/406391274 >> v2:172.21.15.184:3300/0 conn(0x7f1d340c3360 0x7f1d34166330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d327fc640 1 -- 172.21.15.83:0/406391274 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2665376374 0 0) 0x7f1d340aa230 con 0x7f1d340c3fa0 2024-09-19T07:47:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d327fc640 1 -- 172.21.15.83:0/406391274 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d20003530 con 0x7f1d340c3fa0 2024-09-19T07:47:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d337fe640 1 -- 172.21.15.83:0/406391274 >> v2:172.21.15.83:3300/0 conn(0x7f1d340ab3c0 msgr2=0x7f1d34070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d337fe640 1 --2- 172.21.15.83:0/406391274 >> v2:172.21.15.83:3300/0 conn(0x7f1d340ab3c0 0x7f1d34070ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d337fe640 1 -- 172.21.15.83:0/406391274 >> v1:172.21.15.184:6789/0 conn(0x7f1d340c3fa0 legacy=0x7f1d340c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.943+0000 7f1d337fe640 1 -- 172.21.15.83:0/406391274 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d340c28a0 con 0x7f1d340c3360 2024-09-19T07:47:07.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d337fe640 1 --2- 172.21.15.83:0/406391274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d340c3360 0x7f1d34166330 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1d3408c8b0 tx=0x7f1d2c01ea90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55ef883d69f63906 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:07.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d327fc640 1 -- 172.21.15.83:0/406391274 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c00c2d0 con 0x7f1d340c3360 2024-09-19T07:47:07.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d327fc640 1 -- 172.21.15.83:0/406391274 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d2c0a7850 con 0x7f1d340c3360 2024-09-19T07:47:07.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d327fc640 1 -- 172.21.15.83:0/406391274 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0a7c20 con 0x7f1d340c3360 2024-09-19T07:47:07.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d397ee640 1 -- 172.21.15.83:0/406391274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d340c3360 msgr2=0x7f1d34166330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:07.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d397ee640 1 --2- 172.21.15.83:0/406391274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d340c3360 0x7f1d34166330 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1d3408c8b0 tx=0x7f1d2c01ea90 comp rx=0 tx=0).stop 2024-09-19T07:47:07.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d397ee640 1 -- 172.21.15.83:0/406391274 shutdown_connections 2024-09-19T07:47:07.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d397ee640 1 --2- 172.21.15.83:0/406391274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d340c3360 0x7f1d34166330 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:07.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d397ee640 1 --2- 172.21.15.83:0/406391274 >> v2:172.21.15.83:3300/0 conn(0x7f1d340ab3c0 0x7f1d34070ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:07.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d397ee640 1 -- 172.21.15.83:0/406391274 >> 172.21.15.83:0/406391274 conn(0x7f1d34075390 msgr2=0x7f1d3406bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:07.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d397ee640 1 -- 172.21.15.83:0/406391274 shutdown_connections 2024-09-19T07:47:07.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d397ee640 1 -- 172.21.15.83:0/406391274 wait complete. 2024-09-19T07:47:07.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d397ee640 1 Processor -- start 2024-09-19T07:47:07.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.947+0000 7f1d397ee640 1 -- start start 2024-09-19T07:47:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d397ee640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d340ab3c0 0x7f1d342036b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d397ee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d340c3360 0x7f1d3416b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d397ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d340c3fa0 0x7f1d341fcb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d397ee640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3416f7a0 con 0x7f1d340ab3c0 2024-09-19T07:47:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d397ee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1d340c2e40 con 0x7f1d340c3fa0 2024-09-19T07:47:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d397ee640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1d3416f920 con 0x7f1d340c3360 2024-09-19T07:47:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d32ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d340ab3c0 0x7f1d342036b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d32ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d340ab3c0 0x7f1d342036b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43630/0 (socket says 172.21.15.83:43630) 2024-09-19T07:47:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d32ffd640 1 -- 172.21.15.83:0/2087282827 learned_addr learned my addr 172.21.15.83:0/2087282827 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d2bfff640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d340c3360 0x7f1d3416b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d32ffd640 1 -- 172.21.15.83:0/2087282827 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d340c3360 msgr2=0x7f1d3416b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d32ffd640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d340c3360 0x7f1d3416b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d32ffd640 1 -- 172.21.15.83:0/2087282827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d340c3fa0 msgr2=0x7f1d341fcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d337fe640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d340c3fa0 0x7f1d341fcb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d32ffd640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d340c3fa0 0x7f1d341fcb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d32ffd640 1 -- 172.21.15.83:0/2087282827 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d34203bf0 con 0x7f1d340ab3c0 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d337fe640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d340c3fa0 0x7f1d341fcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d32ffd640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d340ab3c0 0x7f1d342036b0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400c430 tx=0x7f1d2400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d2b7fe640 1 -- 172.21.15.83:0/2087282827 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d24017070 con 0x7f1d340ab3c0 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d2b7fe640 1 -- 172.21.15.83:0/2087282827 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d240052f0 con 0x7f1d340ab3c0 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d397ee640 1 -- 172.21.15.83:0/2087282827 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d3409cfe0 con 0x7f1d340ab3c0 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d397ee640 1 -- 172.21.15.83:0/2087282827 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d3409f6f0 con 0x7f1d340ab3c0 2024-09-19T07:47:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d2b7fe640 1 -- 172.21.15.83:0/2087282827 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d24003840 con 0x7f1d340ab3c0 2024-09-19T07:47:07.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d2b7fe640 1 -- 172.21.15.83:0/2087282827 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1d2401a070 con 0x7f1d340ab3c0 2024-09-19T07:47:07.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d297fa640 1 -- 172.21.15.83:0/2087282827 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d0c0052a0 con 0x7f1d340ab3c0 2024-09-19T07:47:07.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d2b7fe640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1d14067910 0x7f1d14069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:07.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d2bfff640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1d14067910 0x7f1d14069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:07.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.951+0000 7f1d2b7fe640 1 -- 172.21.15.83:0/2087282827 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7f1d24089170 con 0x7f1d340ab3c0 2024-09-19T07:47:07.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.955+0000 7f1d2bfff640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1d14067910 0x7f1d14069dd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1d340c62c0 tx=0x7f1d1c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:07.959+0000 7f1d2b7fe640 1 -- 172.21.15.83:0/2087282827 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1d24055840 con 0x7f1d340ab3c0 2024-09-19T07:47:08.270 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:08.270 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":189,"btime":"2024-09-19T07:47:06:716128+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:47:08.270 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 189 2024-09-19T07:47:08.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.271+0000 7f1d297fa640 1 -- 172.21.15.83:0/2087282827 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d0c005860 con 0x7f1d340ab3c0 2024-09-19T07:47:08.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.271+0000 7f1d2b7fe640 1 -- 172.21.15.83:0/2087282827 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+5713 (secure 0 0 0) 0x7f1d24055140 con 0x7f1d340ab3c0 2024-09-19T07:47:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.275+0000 7f1d297fa640 1 -- 172.21.15.83:0/2087282827 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1d14067910 msgr2=0x7f1d14069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.275+0000 7f1d297fa640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1d14067910 0x7f1d14069dd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1d340c62c0 tx=0x7f1d1c008040 comp rx=0 tx=0).stop 2024-09-19T07:47:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.275+0000 7f1d297fa640 1 -- 172.21.15.83:0/2087282827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d340ab3c0 msgr2=0x7f1d342036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.275+0000 7f1d297fa640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d340ab3c0 0x7f1d342036b0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400c430 tx=0x7f1d2400c830 comp rx=0 tx=0).stop 2024-09-19T07:47:08.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.275+0000 7f1d297fa640 1 -- 172.21.15.83:0/2087282827 shutdown_connections 2024-09-19T07:47:08.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.275+0000 7f1d297fa640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1d14067910 0x7f1d14069dd0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:08.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.275+0000 7f1d297fa640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d340c3fa0 0x7f1d341fcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:08.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.275+0000 7f1d297fa640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d340c3360 0x7f1d3416b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:08.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.275+0000 7f1d297fa640 1 --2- 172.21.15.83:0/2087282827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d340ab3c0 0x7f1d342036b0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:08.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.275+0000 7f1d297fa640 1 -- 172.21.15.83:0/2087282827 >> 172.21.15.83:0/2087282827 conn(0x7f1d34075390 msgr2=0x7f1d34072af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:08.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.275+0000 7f1d297fa640 1 -- 172.21.15.83:0/2087282827 shutdown_connections 2024-09-19T07:47:08.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:08.275+0000 7f1d297fa640 1 -- 172.21.15.83:0/2087282827 wait complete. 2024-09-19T07:47:08.300 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=6152 2024-09-19T07:47:08.300 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.d 2024-09-19T07:47:10.301 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-19T07:47:10.301 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-19T07:47:10.301 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-19T07:47:10.303 INFO:tasks.ceph.mds.d:Started 2024-09-19T07:47:10.304 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:10.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.462+0000 7fa9b917f640 1 Processor -- start 2024-09-19T07:47:10.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.462+0000 7fa9b917f640 1 -- start start 2024-09-19T07:47:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.462+0000 7fa9b917f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b41546a0 0x7fa9b40c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.462+0000 7fa9b917f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9b4153a50 con 0x7fa9b41573a0 2024-09-19T07:47:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.462+0000 7fa9b917f640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9b40730e0 con 0x7fa9b4074000 2024-09-19T07:47:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.462+0000 7fa9b917f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b40733c0 con 0x7fa9b41546a0 2024-09-19T07:47:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b359e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa9b41573a0 0x7fa9b4157850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52932/0 (socket says 172.21.15.83:52932) 2024-09-19T07:47:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b359e640 1 -- 172.21.15.83:0/245932083 learned_addr learned my addr 172.21.15.83:0/245932083 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:47:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2381015873 0 0) 0x7fa9b40730e0 con 0x7fa9b4074000 2024-09-19T07:47:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9a0003530 con 0x7fa9b4074000 2024-09-19T07:47:10.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 <== mon.1 v1:172.21.15.184:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1146495821 0 0) 0x7fa9a0003530 con 0x7fa9b4074000 2024-09-19T07:47:10.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b41546a0 msgr2=0x7fa9b40c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:10.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 --2- 172.21.15.83:0/245932083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b41546a0 0x7fa9b40c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:10.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 >> v1:172.21.15.83:6789/0 conn(0x7fa9b41573a0 legacy=0x7fa9b4157850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:10.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 --> v1:172.21.15.184:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b40736a0 con 0x7fa9b4074000 2024-09-19T07:47:10.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 <== mon.1 v1:172.21.15.184:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fa99c001680 con 0x7fa9b4074000 2024-09-19T07:47:10.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 >> v1:172.21.15.184:6789/0 conn(0x7fa9b4074000 legacy=0x7fa9b406cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 --2- 172.21.15.83:0/245932083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9a00039f0 0x7fa9a0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 --2- 172.21.15.83:0/245932083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b41546a0 0x7fa9a0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 --2- 172.21.15.83:0/245932083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9a0004e20 0x7fa9a00052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b40733c0 con 0x7fa9a0004e20 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a000bc30 con 0x7fa9a00039f0 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa9a000be30 con 0x7fa9b41546a0 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b359e640 1 --2- 172.21.15.83:0/245932083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9a0004e20 0x7fa9a00052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b2d9d640 1 --2- 172.21.15.83:0/245932083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9a00039f0 0x7fa9a0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9abfff640 1 --2- 172.21.15.83:0/245932083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b41546a0 0x7fa9a0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b359e640 1 -- 172.21.15.83:0/245932083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b41546a0 msgr2=0x7fa9a0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b359e640 1 --2- 172.21.15.83:0/245932083 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b41546a0 0x7fa9a0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b359e640 1 -- 172.21.15.83:0/245932083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9a00039f0 msgr2=0x7fa9a0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:10.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b359e640 1 --2- 172.21.15.83:0/245932083 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9a00039f0 0x7fa9a0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:10.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.466+0000 7fa9b359e640 1 -- 172.21.15.83:0/245932083 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a0003530 con 0x7fa9a0004e20 2024-09-19T07:47:10.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.470+0000 7fa9b359e640 1 --2- 172.21.15.83:0/245932083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9a0004e20 0x7fa9a00052a0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00c190 tx=0x7fa9ac01cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225e337d23947faf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:10.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.470+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac001ed0 con 0x7fa9a0004e20 2024-09-19T07:47:10.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.470+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9ac0a9020 con 0x7fa9a0004e20 2024-09-19T07:47:10.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.470+0000 7fa9b259c640 1 -- 172.21.15.83:0/245932083 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac0a5b00 con 0x7fa9a0004e20 2024-09-19T07:47:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.470+0000 7fa9b917f640 1 -- 172.21.15.83:0/245932083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9a0004e20 msgr2=0x7fa9a00052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.470+0000 7fa9b917f640 1 --2- 172.21.15.83:0/245932083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9a0004e20 0x7fa9a00052a0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00c190 tx=0x7fa9ac01cbe0 comp rx=0 tx=0).stop 2024-09-19T07:47:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.470+0000 7fa9b359e640 1 -- 172.21.15.83:0/245932083 reap_dead start 2024-09-19T07:47:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.470+0000 7fa9b917f640 1 -- 172.21.15.83:0/245932083 shutdown_connections 2024-09-19T07:47:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.470+0000 7fa9b917f640 1 -- 172.21.15.83:0/245932083 >> 172.21.15.83:0/245932083 conn(0x7fa9b40c5060 msgr2=0x7fa9b40c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:10.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b917f640 1 -- 172.21.15.83:0/245932083 shutdown_connections 2024-09-19T07:47:10.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b917f640 1 -- 172.21.15.83:0/245932083 wait complete. 2024-09-19T07:47:10.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b917f640 1 Processor -- start 2024-09-19T07:47:10.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b917f640 1 -- start start 2024-09-19T07:47:10.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b917f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b40bdf40 0x7fa9b40bef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:10.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b917f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9b40bf490 0x7fa9b4156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:10.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b917f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b4156780 0x7fa9b406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:10.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b917f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b406d520 con 0x7fa9b40bdf40 2024-09-19T07:47:10.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b917f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b406d3a0 con 0x7fa9b40bf490 2024-09-19T07:47:10.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b917f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b4072d00 con 0x7fa9b4156780 2024-09-19T07:47:10.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b40bdf40 0x7fa9b40bef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:10.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b40bdf40 0x7fa9b40bef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45288/0 (socket says 172.21.15.83:45288) 2024-09-19T07:47:10.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b2d9d640 1 -- 172.21.15.83:0/2126956865 learned_addr learned my addr 172.21.15.83:0/2126956865 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:10.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b359e640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b4156780 0x7fa9b406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:10.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9abfff640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9b40bf490 0x7fa9b4156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:10.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b2d9d640 1 -- 172.21.15.83:0/2126956865 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b4156780 msgr2=0x7fa9b406ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:10.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b2d9d640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b4156780 0x7fa9b406ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:10.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b2d9d640 1 -- 172.21.15.83:0/2126956865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9b40bf490 msgr2=0x7fa9b4156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:10.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b2d9d640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9b40bf490 0x7fa9b4156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:10.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.474+0000 7fa9b2d9d640 1 -- 172.21.15.83:0/2126956865 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b41510a0 con 0x7fa9b40bdf40 2024-09-19T07:47:10.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.478+0000 7fa9b2d9d640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b40bdf40 0x7fa9b40bef50 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400d5a0 tx=0x7fa9a400da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:10.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.478+0000 7fa9ab7fe640 1 -- 172.21.15.83:0/2126956865 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4018020 con 0x7fa9b40bdf40 2024-09-19T07:47:10.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.478+0000 7fa9ab7fe640 1 -- 172.21.15.83:0/2126956865 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a4014620 con 0x7fa9b40bdf40 2024-09-19T07:47:10.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.478+0000 7fa9b917f640 1 -- 172.21.15.83:0/2126956865 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b4155270 con 0x7fa9b40bdf40 2024-09-19T07:47:10.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.478+0000 7fa9ab7fe640 1 -- 172.21.15.83:0/2126956865 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a40149d0 con 0x7fa9b40bdf40 2024-09-19T07:47:10.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.478+0000 7fa9b917f640 1 -- 172.21.15.83:0/2126956865 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b4155830 con 0x7fa9b40bdf40 2024-09-19T07:47:10.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.478+0000 7fa9b917f640 1 -- 172.21.15.83:0/2126956865 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b4155ea0 con 0x7fa9b40bdf40 2024-09-19T07:47:10.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.478+0000 7fa9ab7fe640 1 -- 172.21.15.83:0/2126956865 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa9a401b020 con 0x7fa9b40bdf40 2024-09-19T07:47:10.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.478+0000 7fa9ab7fe640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9940678c0 0x7fa994069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:10.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.478+0000 7fa9ab7fe640 1 -- 172.21.15.83:0/2126956865 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7fa9a4086d50 con 0x7fa9b40bdf40 2024-09-19T07:47:10.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.478+0000 7fa9abfff640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9940678c0 0x7fa994069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:10.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.478+0000 7fa9abfff640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9940678c0 0x7fa994069d80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa99c001ec0 tx=0x7fa99c001d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:10.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.494+0000 7fa9ab7fe640 1 -- 172.21.15.83:0/2126956865 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa9a4022030 con 0x7fa9b40bdf40 2024-09-19T07:47:10.552 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-19T07:47:10.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.778+0000 7fa9b917f640 1 -- 172.21.15.83:0/2126956865 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9b40c70f0 con 0x7fa9b40bdf40 2024-09-19T07:47:10.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.778+0000 7fa9ab7fe640 1 -- 172.21.15.83:0/2126956865 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+6542 (secure 0 0 0) 0x7fa9a4056860 con 0x7fa9b40bdf40 2024-09-19T07:47:10.778 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:10.778 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":190,"btime":"2024-09-19T07:47:10:732800+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:47:10.779 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 190 2024-09-19T07:47:10.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.782+0000 7fa9b917f640 1 -- 172.21.15.83:0/2126956865 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9940678c0 msgr2=0x7fa994069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.782+0000 7fa9b917f640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9940678c0 0x7fa994069d80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa99c001ec0 tx=0x7fa99c001d50 comp rx=0 tx=0).stop 2024-09-19T07:47:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.782+0000 7fa9b917f640 1 -- 172.21.15.83:0/2126956865 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b40bdf40 msgr2=0x7fa9b40bef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.782+0000 7fa9b917f640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b40bdf40 0x7fa9b40bef50 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400d5a0 tx=0x7fa9a400da60 comp rx=0 tx=0).stop 2024-09-19T07:47:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.782+0000 7fa9b917f640 1 -- 172.21.15.83:0/2126956865 shutdown_connections 2024-09-19T07:47:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.782+0000 7fa9b917f640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9940678c0 0x7fa994069d80 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.782+0000 7fa9b917f640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9b4156780 0x7fa9b406ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.782+0000 7fa9b917f640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9b40bf490 0x7fa9b4156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.782+0000 7fa9b917f640 1 --2- 172.21.15.83:0/2126956865 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9b40bdf40 0x7fa9b40bef50 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.782+0000 7fa9b917f640 1 -- 172.21.15.83:0/2126956865 >> 172.21.15.83:0/2126956865 conn(0x7fa9b40c5060 msgr2=0x7fa9b41529d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.782+0000 7fa9b917f640 1 -- 172.21.15.83:0/2126956865 shutdown_connections 2024-09-19T07:47:10.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:10.782+0000 7fa9b917f640 1 -- 172.21.15.83:0/2126956865 wait complete. 2024-09-19T07:47:10.803 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-19T07:47:10.803 INFO:tasks.mds_thrash.fs.[b]:waiting for 57 secs before thrashing 2024-09-19T07:47:11.602 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:47:11.604 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:47:11.655 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:47:11.658+0000 7fecd5e9a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:47:38.885 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:39.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd889e30640 1 Processor -- start 2024-09-19T07:47:39.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd889e30640 1 -- start start 2024-09-19T07:47:39.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd889e30640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd884153300 0x7fd88406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd889e30640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd88406d1d0 0x7fd8840c23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd889e30640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8841593f0 con 0x7fd8840c2910 2024-09-19T07:47:39.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd889e30640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd8841591a0 con 0x7fd88406d1d0 2024-09-19T07:47:39.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd889e30640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd8841596d0 con 0x7fd884153300 2024-09-19T07:47:39.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd882ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd88406d1d0 0x7fd8840c23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:39.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd883fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd8840c2910 0x7fd884155bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42532/0 (socket says 172.21.15.83:42532) 2024-09-19T07:47:39.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd883fff640 1 -- 172.21.15.83:0/3493225076 learned_addr learned my addr 172.21.15.83:0/3493225076 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:47:39.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd8837fe640 1 --2- 172.21.15.83:0/3493225076 >> v2:172.21.15.184:3300/0 conn(0x7fd884153300 0x7fd88406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:39.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd882ffd640 1 -- 172.21.15.83:0/3493225076 >> v2:172.21.15.184:3300/0 conn(0x7fd884153300 msgr2=0x7fd88406cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:39.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd882ffd640 1 --2- 172.21.15.83:0/3493225076 >> v2:172.21.15.184:3300/0 conn(0x7fd884153300 0x7fd88406cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd882ffd640 1 -- 172.21.15.83:0/3493225076 >> v1:172.21.15.83:6789/0 conn(0x7fd8840c2910 legacy=0x7fd884155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:39.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.026+0000 7fd882ffd640 1 -- 172.21.15.83:0/3493225076 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8841599b0 con 0x7fd88406d1d0 2024-09-19T07:47:39.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.030+0000 7fd882ffd640 1 --2- 172.21.15.83:0/3493225076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88406d1d0 0x7fd8840c23d0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fd8840c2de0 tx=0x7fd87801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c711aab3975e83 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:39.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.030+0000 7fd8827fc640 1 -- 172.21.15.83:0/3493225076 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8780b0070 con 0x7fd88406d1d0 2024-09-19T07:47:39.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.030+0000 7fd8827fc640 1 -- 172.21.15.83:0/3493225076 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8780ab600 con 0x7fd88406d1d0 2024-09-19T07:47:39.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.030+0000 7fd8827fc640 1 -- 172.21.15.83:0/3493225076 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8780ab9d0 con 0x7fd88406d1d0 2024-09-19T07:47:39.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.030+0000 7fd889e30640 1 -- 172.21.15.83:0/3493225076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88406d1d0 msgr2=0x7fd8840c23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:39.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.030+0000 7fd889e30640 1 --2- 172.21.15.83:0/3493225076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88406d1d0 0x7fd8840c23d0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fd8840c2de0 tx=0x7fd87801c360 comp rx=0 tx=0).stop 2024-09-19T07:47:39.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.030+0000 7fd889e30640 1 -- 172.21.15.83:0/3493225076 shutdown_connections 2024-09-19T07:47:39.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.030+0000 7fd889e30640 1 --2- 172.21.15.83:0/3493225076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88406d1d0 0x7fd8840c23d0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.030+0000 7fd889e30640 1 --2- 172.21.15.83:0/3493225076 >> v2:172.21.15.184:3300/0 conn(0x7fd884153300 0x7fd88406cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.030+0000 7fd889e30640 1 -- 172.21.15.83:0/3493225076 >> 172.21.15.83:0/3493225076 conn(0x7fd8840c5090 msgr2=0x7fd8840c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:39.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.030+0000 7fd889e30640 1 -- 172.21.15.83:0/3493225076 shutdown_connections 2024-09-19T07:47:39.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.030+0000 7fd889e30640 1 -- 172.21.15.83:0/3493225076 wait complete. 2024-09-19T07:47:39.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd889e30640 1 Processor -- start 2024-09-19T07:47:39.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd889e30640 1 -- start start 2024-09-19T07:47:39.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd889e30640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88406d1d0 0x7fd8841eef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd889e30640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8840c2910 0x7fd884151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd889e30640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd884153300 0x7fd884154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd889e30640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd884153860 con 0x7fd88406d1d0 2024-09-19T07:47:39.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd889e30640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd8841536e0 con 0x7fd8840c2910 2024-09-19T07:47:39.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd889e30640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd884162d50 con 0x7fd884153300 2024-09-19T07:47:39.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd883fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd884153300 0x7fd884154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:39.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd883fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd884153300 0x7fd884154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:42388/0 (socket says 172.21.15.83:42388) 2024-09-19T07:47:39.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd883fff640 1 -- 172.21.15.83:0/1587185601 learned_addr learned my addr 172.21.15.83:0/1587185601 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:39.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd883fff640 1 -- 172.21.15.83:0/1587185601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8840c2910 msgr2=0x7fd884151930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:39.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd883fff640 1 --2- 172.21.15.83:0/1587185601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8840c2910 0x7fd884151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd883fff640 1 -- 172.21.15.83:0/1587185601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88406d1d0 msgr2=0x7fd8841eef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd883fff640 1 --2- 172.21.15.83:0/1587185601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88406d1d0 0x7fd8841eef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd883fff640 1 -- 172.21.15.83:0/1587185601 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88414caf0 con 0x7fd884153300 2024-09-19T07:47:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd883fff640 1 --2- 172.21.15.83:0/1587185601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd884153300 0x7fd884154f90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd87c00caa0 tx=0x7fd87c00cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.034+0000 7fd86bfff640 1 -- 172.21.15.83:0/1587185601 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c017070 con 0x7fd884153300 2024-09-19T07:47:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.038+0000 7fd86bfff640 1 -- 172.21.15.83:0/1587185601 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd87c00ac00 con 0x7fd884153300 2024-09-19T07:47:39.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.038+0000 7fd86bfff640 1 -- 172.21.15.83:0/1587185601 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c012400 con 0x7fd884153300 2024-09-19T07:47:39.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.038+0000 7fd889e30640 1 -- 172.21.15.83:0/1587185601 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8841f23d0 con 0x7fd884153300 2024-09-19T07:47:39.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.038+0000 7fd889e30640 1 -- 172.21.15.83:0/1587185601 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8841ee710 con 0x7fd884153300 2024-09-19T07:47:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.038+0000 7fd889e30640 1 -- 172.21.15.83:0/1587185601 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8840c23d0 con 0x7fd884153300 2024-09-19T07:47:39.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.038+0000 7fd86bfff640 1 -- 172.21.15.83:0/1587185601 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd87c01a070 con 0x7fd884153300 2024-09-19T07:47:39.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.038+0000 7fd86bfff640 1 --2- 172.21.15.83:0/1587185601 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd870067910 0x7fd870069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.038+0000 7fd86bfff640 1 -- 172.21.15.83:0/1587185601 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7fd87c086850 con 0x7fd884153300 2024-09-19T07:47:39.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.042+0000 7fd8837fe640 1 --2- 172.21.15.83:0/1587185601 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd870067910 0x7fd870069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:39.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.042+0000 7fd8837fe640 1 --2- 172.21.15.83:0/1587185601 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd870067910 0x7fd870069dd0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd874006aa0 tx=0x7fd874006a30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:39.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.050+0000 7fd86bfff640 1 -- 172.21.15.83:0/1587185601 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd87c055d90 con 0x7fd884153300 2024-09-19T07:47:39.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.342+0000 7fd889e30640 1 -- 172.21.15.83:0/1587185601 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd884151e70 con 0x7fd884153300 2024-09-19T07:47:39.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.342+0000 7fd86bfff640 1 -- 172.21.15.83:0/1587185601 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+6542 (secure 0 0 0) 0x7fd87c05a090 con 0x7fd884153300 2024-09-19T07:47:39.342 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:39.342 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":190,"btime":"2024-09-19T07:47:10:732800+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:47:39.342 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 190 2024-09-19T07:47:39.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.346+0000 7fd889e30640 1 -- 172.21.15.83:0/1587185601 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd870067910 msgr2=0x7fd870069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:39.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.346+0000 7fd889e30640 1 --2- 172.21.15.83:0/1587185601 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd870067910 0x7fd870069dd0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd874006aa0 tx=0x7fd874006a30 comp rx=0 tx=0).stop 2024-09-19T07:47:39.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.346+0000 7fd889e30640 1 -- 172.21.15.83:0/1587185601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd884153300 msgr2=0x7fd884154f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:39.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.346+0000 7fd889e30640 1 --2- 172.21.15.83:0/1587185601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd884153300 0x7fd884154f90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd87c00caa0 tx=0x7fd87c00cf60 comp rx=0 tx=0).stop 2024-09-19T07:47:39.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.346+0000 7fd889e30640 1 -- 172.21.15.83:0/1587185601 shutdown_connections 2024-09-19T07:47:39.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.346+0000 7fd889e30640 1 --2- 172.21.15.83:0/1587185601 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd870067910 0x7fd870069dd0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.346+0000 7fd889e30640 1 --2- 172.21.15.83:0/1587185601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd884153300 0x7fd884154f90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.346+0000 7fd889e30640 1 --2- 172.21.15.83:0/1587185601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8840c2910 0x7fd884151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.346+0000 7fd889e30640 1 --2- 172.21.15.83:0/1587185601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88406d1d0 0x7fd8841eef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.346+0000 7fd889e30640 1 -- 172.21.15.83:0/1587185601 >> 172.21.15.83:0/1587185601 conn(0x7fd8840c5090 msgr2=0x7fd8840c2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:39.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.346+0000 7fd889e30640 1 -- 172.21.15.83:0/1587185601 shutdown_connections 2024-09-19T07:47:39.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.346+0000 7fd889e30640 1 -- 172.21.15.83:0/1587185601 wait complete. 2024-09-19T07:47:39.373 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-19T07:47:39.374 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:47:39.374 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:47:39.387 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:47:39.387 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:39.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.566+0000 7f53ae7d8640 1 Processor -- start 2024-09-19T07:47:39.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.566+0000 7f53ae7d8640 1 -- start start 2024-09-19T07:47:39.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53ae7d8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53a806f930 0x7f53a815aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53ae7d8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f53a806db30 0x7f53a81573d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53ae7d8640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f53a8157910 0x7f53a80720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53ae7d8640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f53a8071c60 con 0x7f53a806db30 2024-09-19T07:47:39.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53ae7d8640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f53a8073480 con 0x7f53a8157910 2024-09-19T07:47:39.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53ae7d8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f53a8073760 con 0x7f53a806f930 2024-09-19T07:47:39.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53acd75640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f53a8157910 0x7f53a80720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:39.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53acd75640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f53a8157910 0x7f53a80720f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:45146/0 (socket says 172.21.15.83:45146) 2024-09-19T07:47:39.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53acd75640 1 -- 172.21.15.83:0/3150159420 learned_addr learned my addr 172.21.15.83:0/3150159420 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:39.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53acd75640 1 -- 172.21.15.83:0/3150159420 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53a806f930 msgr2=0x7f53a815aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:39.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f539ffff640 1 --2- 172.21.15.83:0/3150159420 >> v2:172.21.15.83:3300/0 conn(0x7f53a806db30 0x7f53a81573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:39.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53a7fff640 1 --2- 172.21.15.83:0/3150159420 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53a806f930 0x7f53a815aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:39.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53acd75640 1 --2- 172.21.15.83:0/3150159420 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53a806f930 0x7f53a815aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53acd75640 1 -- 172.21.15.83:0/3150159420 >> v2:172.21.15.83:3300/0 conn(0x7f53a806db30 msgr2=0x7f53a81573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:39.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53acd75640 1 --2- 172.21.15.83:0/3150159420 >> v2:172.21.15.83:3300/0 conn(0x7f53a806db30 0x7f53a81573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53acd75640 1 -- 172.21.15.83:0/3150159420 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a8073a40 con 0x7f53a8157910 2024-09-19T07:47:39.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f53a7fff640 1 --2- 172.21.15.83:0/3150159420 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53a806f930 0x7f53a815aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:47:39.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.570+0000 7f539ffff640 1 --2- 172.21.15.83:0/3150159420 >> v2:172.21.15.83:3300/0 conn(0x7f53a806db30 0x7f53a81573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:47:39.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53acd75640 1 --2- 172.21.15.83:0/3150159420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a8157910 0x7f53a80720f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f53a8076130 tx=0x7f53a001c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92f421bde7d85d34 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:39.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53a77fe640 1 -- 172.21.15.83:0/3150159420 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a00a5610 con 0x7f53a8157910 2024-09-19T07:47:39.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53a77fe640 1 -- 172.21.15.83:0/3150159420 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a00a57b0 con 0x7f53a8157910 2024-09-19T07:47:39.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53a77fe640 1 -- 172.21.15.83:0/3150159420 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a00a5b80 con 0x7f53a8157910 2024-09-19T07:47:39.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53ae7d8640 1 -- 172.21.15.83:0/3150159420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a8157910 msgr2=0x7f53a80720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:39.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53ae7d8640 1 --2- 172.21.15.83:0/3150159420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a8157910 0x7f53a80720f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f53a8076130 tx=0x7f53a001c960 comp rx=0 tx=0).stop 2024-09-19T07:47:39.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53ae7d8640 1 -- 172.21.15.83:0/3150159420 shutdown_connections 2024-09-19T07:47:39.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53ae7d8640 1 --2- 172.21.15.83:0/3150159420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a8157910 0x7f53a80720f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53ae7d8640 1 --2- 172.21.15.83:0/3150159420 >> v2:172.21.15.83:3300/0 conn(0x7f53a806db30 0x7f53a81573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53ae7d8640 1 --2- 172.21.15.83:0/3150159420 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53a806f930 0x7f53a815aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53ae7d8640 1 -- 172.21.15.83:0/3150159420 >> 172.21.15.83:0/3150159420 conn(0x7f53a80c7e40 msgr2=0x7f53a80c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:39.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53ae7d8640 1 -- 172.21.15.83:0/3150159420 shutdown_connections 2024-09-19T07:47:39.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.578+0000 7f53ae7d8640 1 -- 172.21.15.83:0/3150159420 wait complete. 2024-09-19T07:47:39.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53ae7d8640 1 Processor -- start 2024-09-19T07:47:39.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53ae7d8640 1 -- start start 2024-09-19T07:47:39.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53ae7d8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53a806db30 0x7f53a81561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53ae7d8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53a806f930 0x7f53a8147df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53ae7d8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a8157910 0x7f53a8154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53ae7d8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f53a815d790 con 0x7f53a806db30 2024-09-19T07:47:39.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53ae7d8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f53a806ff20 con 0x7f53a8157910 2024-09-19T07:47:39.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53ae7d8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f53a815d910 con 0x7f53a806f930 2024-09-19T07:47:39.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53a7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53a806db30 0x7f53a81561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:39.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53a7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53a806db30 0x7f53a81561a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48072/0 (socket says 172.21.15.83:48072) 2024-09-19T07:47:39.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53a7fff640 1 -- 172.21.15.83:0/532582455 learned_addr learned my addr 172.21.15.83:0/532582455 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:39.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f539ffff640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53a806f930 0x7f53a8147df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:39.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53acd75640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a8157910 0x7f53a8154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:39.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53a7fff640 1 -- 172.21.15.83:0/532582455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53a806f930 msgr2=0x7f53a8147df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:39.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53a7fff640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53a806f930 0x7f53a8147df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53a7fff640 1 -- 172.21.15.83:0/532582455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a8157910 msgr2=0x7f53a8154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:39.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53a7fff640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a8157910 0x7f53a8154e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53a7fff640 1 -- 172.21.15.83:0/532582455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a8146460 con 0x7f53a806db30 2024-09-19T07:47:39.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f539ffff640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53a806f930 0x7f53a8147df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:47:39.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53acd75640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a8157910 0x7f53a8154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:47:39.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53a7fff640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53a806db30 0x7f53a81561a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f539800c7c0 tx=0x7f539800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:39.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53a57fa640 1 -- 172.21.15.83:0/532582455 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398007a30 con 0x7f53a806db30 2024-09-19T07:47:39.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53a57fa640 1 -- 172.21.15.83:0/532582455 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5398007bd0 con 0x7f53a806db30 2024-09-19T07:47:39.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53ae7d8640 1 -- 172.21.15.83:0/532582455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a8156880 con 0x7f53a806db30 2024-09-19T07:47:39.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.582+0000 7f53a57fa640 1 -- 172.21.15.83:0/532582455 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53980113f0 con 0x7f53a806db30 2024-09-19T07:47:39.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.586+0000 7f53ae7d8640 1 -- 172.21.15.83:0/532582455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a80c72e0 con 0x7f53a806db30 2024-09-19T07:47:39.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.586+0000 7f53ae7d8640 1 -- 172.21.15.83:0/532582455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a815aca0 con 0x7f53a806db30 2024-09-19T07:47:39.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.586+0000 7f53a57fa640 1 -- 172.21.15.83:0/532582455 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5398011590 con 0x7f53a806db30 2024-09-19T07:47:39.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.586+0000 7f53a57fa640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53880678c0 0x7f5388069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:39.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.586+0000 7f53a57fa640 1 -- 172.21.15.83:0/532582455 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7f5398086ea0 con 0x7f53a806db30 2024-09-19T07:47:39.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.590+0000 7f539ffff640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53880678c0 0x7f5388069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:39.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.590+0000 7f539ffff640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53880678c0 0x7f5388069d80 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f53a80c31e0 tx=0x7f5390008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:39.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.590+0000 7f53a57fa640 1 -- 172.21.15.83:0/532582455 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f5398015020 con 0x7f53a806db30 2024-09-19T07:47:39.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.862+0000 7f53ae7d8640 1 -- 172.21.15.83:0/532582455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53a80c2f30 con 0x7f53a806db30 2024-09-19T07:47:39.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.866+0000 7f53a57fa640 1 -- 172.21.15.83:0/532582455 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+6542 (secure 0 0 0) 0x7f5398056990 con 0x7f53a806db30 2024-09-19T07:47:39.863 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:39.863 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":190,"btime":"2024-09-19T07:47:10:732800+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:47:39.863 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 190 2024-09-19T07:47:39.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.870+0000 7f53ae7d8640 1 -- 172.21.15.83:0/532582455 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53880678c0 msgr2=0x7f5388069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:39.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.870+0000 7f53ae7d8640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53880678c0 0x7f5388069d80 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f53a80c31e0 tx=0x7f5390008040 comp rx=0 tx=0).stop 2024-09-19T07:47:39.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.870+0000 7f53ae7d8640 1 -- 172.21.15.83:0/532582455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53a806db30 msgr2=0x7f53a81561a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:39.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.870+0000 7f53ae7d8640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53a806db30 0x7f53a81561a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f539800c7c0 tx=0x7f539800cc80 comp rx=0 tx=0).stop 2024-09-19T07:47:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.870+0000 7f53ae7d8640 1 -- 172.21.15.83:0/532582455 shutdown_connections 2024-09-19T07:47:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.870+0000 7f53ae7d8640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f53880678c0 0x7f5388069d80 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.870+0000 7f53ae7d8640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53a8157910 0x7f53a8154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.870+0000 7f53ae7d8640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f53a806f930 0x7f53a8147df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.870+0000 7f53ae7d8640 1 --2- 172.21.15.83:0/532582455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53a806db30 0x7f53a81561a0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.870+0000 7f53ae7d8640 1 -- 172.21.15.83:0/532582455 >> 172.21.15.83:0/532582455 conn(0x7f53a80c7e40 msgr2=0x7f53a80c8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.870+0000 7f53ae7d8640 1 -- 172.21.15.83:0/532582455 shutdown_connections 2024-09-19T07:47:39.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:39.870+0000 7f53ae7d8640 1 -- 172.21.15.83:0/532582455 wait complete. 2024-09-19T07:47:39.890 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:47:41.728 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:47:41.731 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:47:41.891 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:42.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f86465f5640 1 Processor -- start 2024-09-19T07:47:42.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f86465f5640 1 -- start start 2024-09-19T07:47:42.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f86465f5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86400c4130 0x7f86400716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:42.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f86465f5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f86401592a0 0x7f8640071be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:42.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f86465f5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86400c3090 con 0x7f86400c14d0 2024-09-19T07:47:42.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f86465f5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f86400c3370 con 0x7f86401592a0 2024-09-19T07:47:42.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f86465f5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f86400c3650 con 0x7f86400c4130 2024-09-19T07:47:42.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f8644b92640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f86401592a0 0x7f8640071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:42.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f8644b92640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f86401592a0 0x7f8640071be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58290/0 (socket says 172.21.15.83:58290) 2024-09-19T07:47:42.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f8644b92640 1 -- 172.21.15.83:0/1732941252 learned_addr learned my addr 172.21.15.83:0/1732941252 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:42.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f863f7fe640 1 --2- 172.21.15.83:0/1732941252 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86400c4130 0x7f86400716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:42.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f8644b92640 1 -- 172.21.15.83:0/1732941252 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86400c4130 msgr2=0x7f86400716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:42.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f8644b92640 1 --2- 172.21.15.83:0/1732941252 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86400c4130 0x7f86400716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:42.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f8644b92640 1 -- 172.21.15.83:0/1732941252 >> v1:172.21.15.83:6789/0 conn(0x7f86400c14d0 legacy=0x7f86400c2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:42.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.038+0000 7f8644b92640 1 -- 172.21.15.83:0/1732941252 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86400c3930 con 0x7f86401592a0 2024-09-19T07:47:42.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.042+0000 7f8644b92640 1 --2- 172.21.15.83:0/1732941252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86401592a0 0x7f8640071be0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f8638009e50 tx=0x7f863801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fdfdf11c5dd138d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:42.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f863effd640 1 -- 172.21.15.83:0/1732941252 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86380a55d0 con 0x7f86401592a0 2024-09-19T07:47:42.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f863effd640 1 -- 172.21.15.83:0/1732941252 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86380a5770 con 0x7f86401592a0 2024-09-19T07:47:42.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f863effd640 1 -- 172.21.15.83:0/1732941252 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86380a5b40 con 0x7f86401592a0 2024-09-19T07:47:42.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f86465f5640 1 -- 172.21.15.83:0/1732941252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86401592a0 msgr2=0x7f8640071be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:42.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f86465f5640 1 --2- 172.21.15.83:0/1732941252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86401592a0 0x7f8640071be0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f8638009e50 tx=0x7f863801c910 comp rx=0 tx=0).stop 2024-09-19T07:47:42.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f86465f5640 1 -- 172.21.15.83:0/1732941252 shutdown_connections 2024-09-19T07:47:42.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f86465f5640 1 --2- 172.21.15.83:0/1732941252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86401592a0 0x7f8640071be0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:42.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f86465f5640 1 --2- 172.21.15.83:0/1732941252 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86400c4130 0x7f86400716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:42.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f86465f5640 1 -- 172.21.15.83:0/1732941252 >> 172.21.15.83:0/1732941252 conn(0x7f86400c7e40 msgr2=0x7f86400c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:42.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f86465f5640 1 -- 172.21.15.83:0/1732941252 shutdown_connections 2024-09-19T07:47:42.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f86465f5640 1 -- 172.21.15.83:0/1732941252 wait complete. 2024-09-19T07:47:42.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f86465f5640 1 Processor -- start 2024-09-19T07:47:42.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.046+0000 7f86465f5640 1 -- start start 2024-09-19T07:47:42.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f86465f5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86400c14d0 0x7f86400aa700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:42.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f86465f5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86400c4130 0x7f8640156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:42.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f86465f5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86401592a0 0x7f8640156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:42.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f86465f5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8640162640 con 0x7f86401592a0 2024-09-19T07:47:42.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f86465f5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f864013bf50 con 0x7f86400c14d0 2024-09-19T07:47:42.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f86465f5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f86401627c0 con 0x7f86400c4130 2024-09-19T07:47:42.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f863f7fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86400c4130 0x7f8640156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:42.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f863f7fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86400c4130 0x7f8640156140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:53272/0 (socket says 172.21.15.83:53272) 2024-09-19T07:47:42.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f863f7fe640 1 -- 172.21.15.83:0/1039488950 learned_addr learned my addr 172.21.15.83:0/1039488950 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:42.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f8644b92640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86401592a0 0x7f8640156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:42.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f8644b92640 1 -- 172.21.15.83:0/1039488950 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86400c4130 msgr2=0x7f8640156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:42.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f8644b92640 1 --2- 172.21.15.83:0/1039488950 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86400c4130 0x7f8640156140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:42.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f8644b92640 1 -- 172.21.15.83:0/1039488950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86400c14d0 msgr2=0x7f86400aa700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:42.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f8644b92640 1 --2- 172.21.15.83:0/1039488950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86400c14d0 0x7f86400aa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:42.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f8644b92640 1 -- 172.21.15.83:0/1039488950 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86400ac540 con 0x7f86401592a0 2024-09-19T07:47:42.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f8644b92640 1 --2- 172.21.15.83:0/1039488950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86401592a0 0x7f8640156680 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f86380ab000 tx=0x7f86380196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:42.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f863cff9640 1 -- 172.21.15.83:0/1039488950 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86380bc020 con 0x7f86401592a0 2024-09-19T07:47:42.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f863cff9640 1 -- 172.21.15.83:0/1039488950 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86380b8880 con 0x7f86401592a0 2024-09-19T07:47:42.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f86465f5640 1 -- 172.21.15.83:0/1039488950 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86400ac7e0 con 0x7f86401592a0 2024-09-19T07:47:42.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f863cff9640 1 -- 172.21.15.83:0/1039488950 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86380a55d0 con 0x7f86401592a0 2024-09-19T07:47:42.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f86465f5640 1 -- 172.21.15.83:0/1039488950 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86400acda0 con 0x7f86401592a0 2024-09-19T07:47:42.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f86465f5640 1 -- 172.21.15.83:0/1039488950 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86401edf00 con 0x7f86401592a0 2024-09-19T07:47:42.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f863cff9640 1 -- 172.21.15.83:0/1039488950 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f86380bf020 con 0x7f86401592a0 2024-09-19T07:47:42.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f863cff9640 1 --2- 172.21.15.83:0/1039488950 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f862c0678c0 0x7f862c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:42.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f863cff9640 1 -- 172.21.15.83:0/1039488950 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7f8638121c70 con 0x7f86401592a0 2024-09-19T07:47:42.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.050+0000 7f863ffff640 1 --2- 172.21.15.83:0/1039488950 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f862c0678c0 0x7f862c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:42.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.054+0000 7f863ffff640 1 --2- 172.21.15.83:0/1039488950 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f862c0678c0 0x7f862c069d80 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8630000f70 tx=0x7f86300079e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:42.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.058+0000 7f863cff9640 1 -- 172.21.15.83:0/1039488950 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f86380c6030 con 0x7f86401592a0 2024-09-19T07:47:42.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.346+0000 7f86465f5640 1 -- 172.21.15.83:0/1039488950 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86400c2840 con 0x7f86401592a0 2024-09-19T07:47:42.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.346+0000 7f863cff9640 1 -- 172.21.15.83:0/1039488950 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+6542 (secure 0 0 0) 0x7f8638016620 con 0x7f86401592a0 2024-09-19T07:47:42.345 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:42.345 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":190,"btime":"2024-09-19T07:47:10:732800+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:47:42.346 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 190 2024-09-19T07:47:42.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.350+0000 7f86465f5640 1 -- 172.21.15.83:0/1039488950 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f862c0678c0 msgr2=0x7f862c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:42.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.350+0000 7f86465f5640 1 --2- 172.21.15.83:0/1039488950 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f862c0678c0 0x7f862c069d80 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8630000f70 tx=0x7f86300079e0 comp rx=0 tx=0).stop 2024-09-19T07:47:42.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.350+0000 7f86465f5640 1 -- 172.21.15.83:0/1039488950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86401592a0 msgr2=0x7f8640156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:42.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.350+0000 7f86465f5640 1 --2- 172.21.15.83:0/1039488950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86401592a0 0x7f8640156680 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f86380ab000 tx=0x7f86380196b0 comp rx=0 tx=0).stop 2024-09-19T07:47:42.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.350+0000 7f86465f5640 1 -- 172.21.15.83:0/1039488950 shutdown_connections 2024-09-19T07:47:42.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.350+0000 7f86465f5640 1 --2- 172.21.15.83:0/1039488950 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f862c0678c0 0x7f862c069d80 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:42.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.350+0000 7f86465f5640 1 --2- 172.21.15.83:0/1039488950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86401592a0 0x7f8640156680 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:42.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.350+0000 7f86465f5640 1 --2- 172.21.15.83:0/1039488950 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86400c4130 0x7f8640156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:42.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.350+0000 7f86465f5640 1 --2- 172.21.15.83:0/1039488950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86400c14d0 0x7f86400aa700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:42.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.350+0000 7f86465f5640 1 -- 172.21.15.83:0/1039488950 >> 172.21.15.83:0/1039488950 conn(0x7f86400c7e40 msgr2=0x7f86400716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:42.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.350+0000 7f86465f5640 1 -- 172.21.15.83:0/1039488950 shutdown_connections 2024-09-19T07:47:42.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:42.350+0000 7f86465f5640 1 -- 172.21.15.83:0/1039488950 wait complete. 2024-09-19T07:47:42.369 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:47:44.370 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:44.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373fd04640 1 Processor -- start 2024-09-19T07:47:44.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373fd04640 1 -- start start 2024-09-19T07:47:44.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373fd04640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3738150600 0x7f3738156220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:44.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373fd04640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f373806d3e0 0x7f3738153560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:44.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373fd04640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3738068290 con 0x7f3738150220 2024-09-19T07:47:44.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373fd04640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3738068570 con 0x7f373806d3e0 2024-09-19T07:47:44.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373fd04640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3738152d60 con 0x7f3738150600 2024-09-19T07:47:44.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373d29f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3738150600 0x7f3738156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:44.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373e2a1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f373806d3e0 0x7f3738153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:44.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373e2a1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f373806d3e0 0x7f3738153560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58296/0 (socket says 172.21.15.83:58296) 2024-09-19T07:47:44.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373d29f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3738150600 0x7f3738156220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:53286/0 (socket says 172.21.15.83:53286) 2024-09-19T07:47:44.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373e2a1640 1 -- 172.21.15.83:0/117519667 learned_addr learned my addr 172.21.15.83:0/117519667 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:44.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373e2a1640 1 -- 172.21.15.83:0/117519667 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3738150600 msgr2=0x7f3738156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:44.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373e2a1640 1 --2- 172.21.15.83:0/117519667 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3738150600 0x7f3738156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:44.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373e2a1640 1 -- 172.21.15.83:0/117519667 >> v1:172.21.15.184:6789/0 conn(0x7f3738150220 legacy=0x7f37380ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:44.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.510+0000 7f373e2a1640 1 -- 172.21.15.83:0/117519667 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3738153040 con 0x7f373806d3e0 2024-09-19T07:47:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.514+0000 7f373e2a1640 1 --2- 172.21.15.83:0/117519667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f373806d3e0 0x7f3738153560 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f3730009e50 tx=0x7f373001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=266857d1ff1a8366 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.514+0000 7f373ca9e640 1 -- 172.21.15.83:0/117519667 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37300a5670 con 0x7f373806d3e0 2024-09-19T07:47:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.514+0000 7f373ca9e640 1 -- 172.21.15.83:0/117519667 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37300a5810 con 0x7f373806d3e0 2024-09-19T07:47:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.514+0000 7f373ca9e640 1 -- 172.21.15.83:0/117519667 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37300a5be0 con 0x7f373806d3e0 2024-09-19T07:47:44.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.514+0000 7f373fd04640 1 -- 172.21.15.83:0/117519667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f373806d3e0 msgr2=0x7f3738153560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:44.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.514+0000 7f373fd04640 1 --2- 172.21.15.83:0/117519667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f373806d3e0 0x7f3738153560 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f3730009e50 tx=0x7f373001c850 comp rx=0 tx=0).stop 2024-09-19T07:47:44.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.514+0000 7f373fd04640 1 -- 172.21.15.83:0/117519667 shutdown_connections 2024-09-19T07:47:44.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.514+0000 7f373fd04640 1 --2- 172.21.15.83:0/117519667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f373806d3e0 0x7f3738153560 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:44.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.514+0000 7f373fd04640 1 --2- 172.21.15.83:0/117519667 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3738150600 0x7f3738156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:44.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.514+0000 7f373fd04640 1 -- 172.21.15.83:0/117519667 >> 172.21.15.83:0/117519667 conn(0x7f37380c48b0 msgr2=0x7f37380c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:44.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.514+0000 7f373fd04640 1 -- 172.21.15.83:0/117519667 shutdown_connections 2024-09-19T07:47:44.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.514+0000 7f373fd04640 1 -- 172.21.15.83:0/117519667 wait complete. 2024-09-19T07:47:44.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373fd04640 1 Processor -- start 2024-09-19T07:47:44.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373fd04640 1 -- start start 2024-09-19T07:47:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373fd04640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f373806d3e0 0x7f373814aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373fd04640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3738150220 0x7f373815b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373fd04640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3738150600 0x7f373815b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373fd04640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f373815f950 con 0x7f373806d3e0 2024-09-19T07:47:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373fd04640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f373815f7d0 con 0x7f3738150220 2024-09-19T07:47:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373fd04640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f373815fad0 con 0x7f3738150600 2024-09-19T07:47:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373daa0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f373806d3e0 0x7f373814aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373daa0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f373806d3e0 0x7f373814aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58308/0 (socket says 172.21.15.83:58308) 2024-09-19T07:47:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373daa0640 1 -- 172.21.15.83:0/2249145771 learned_addr learned my addr 172.21.15.83:0/2249145771 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373d29f640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3738150220 0x7f373815b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373e2a1640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3738150600 0x7f373815b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373daa0640 1 -- 172.21.15.83:0/2249145771 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3738150600 msgr2=0x7f373815b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373daa0640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3738150600 0x7f373815b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373daa0640 1 -- 172.21.15.83:0/2249145771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3738150220 msgr2=0x7f373815b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373daa0640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3738150220 0x7f373815b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373daa0640 1 -- 172.21.15.83:0/2249145771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373815be60 con 0x7f373806d3e0 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373d29f640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3738150220 0x7f373815b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373daa0640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f373806d3e0 0x7f373814aad0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f373400c3c0 tx=0x7f373400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f372e7fc640 1 -- 172.21.15.83:0/2249145771 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3734017070 con 0x7f373806d3e0 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f372e7fc640 1 -- 172.21.15.83:0/2249145771 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37340052d0 con 0x7f373806d3e0 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373fd04640 1 -- 172.21.15.83:0/2249145771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37381ed110 con 0x7f373806d3e0 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f372e7fc640 1 -- 172.21.15.83:0/2249145771 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3734003840 con 0x7f373806d3e0 2024-09-19T07:47:44.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373fd04640 1 -- 172.21.15.83:0/2249145771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3738071a60 con 0x7f373806d3e0 2024-09-19T07:47:44.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.518+0000 7f373fd04640 1 -- 172.21.15.83:0/2249145771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3738153560 con 0x7f373806d3e0 2024-09-19T07:47:44.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.522+0000 7f372e7fc640 1 -- 172.21.15.83:0/2249145771 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f373401a070 con 0x7f373806d3e0 2024-09-19T07:47:44.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.522+0000 7f372e7fc640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f371c0678c0 0x7f371c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:44.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.522+0000 7f373d29f640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f371c0678c0 0x7f371c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:44.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.522+0000 7f372e7fc640 1 -- 172.21.15.83:0/2249145771 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7f3734086050 con 0x7f373806d3e0 2024-09-19T07:47:44.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.522+0000 7f373d29f640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f371c0678c0 0x7f371c069d80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f373806dff0 tx=0x7f3728008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:44.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.526+0000 7f372e7fc640 1 -- 172.21.15.83:0/2249145771 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3734005480 con 0x7f373806d3e0 2024-09-19T07:47:44.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.810+0000 7f373fd04640 1 -- 172.21.15.83:0/2249145771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37381ed2e0 con 0x7f373806d3e0 2024-09-19T07:47:44.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.810+0000 7f372e7fc640 1 -- 172.21.15.83:0/2249145771 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+6542 (secure 0 0 0) 0x7f3734055b30 con 0x7f373806d3e0 2024-09-19T07:47:44.810 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:44.811 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":190,"btime":"2024-09-19T07:47:10:732800+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:47:44.811 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 190 2024-09-19T07:47:44.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.814+0000 7f373fd04640 1 -- 172.21.15.83:0/2249145771 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f371c0678c0 msgr2=0x7f371c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:44.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.814+0000 7f373fd04640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f371c0678c0 0x7f371c069d80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f373806dff0 tx=0x7f3728008040 comp rx=0 tx=0).stop 2024-09-19T07:47:44.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.814+0000 7f373fd04640 1 -- 172.21.15.83:0/2249145771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f373806d3e0 msgr2=0x7f373814aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:44.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.814+0000 7f373fd04640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f373806d3e0 0x7f373814aad0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f373400c3c0 tx=0x7f373400c880 comp rx=0 tx=0).stop 2024-09-19T07:47:44.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.814+0000 7f373fd04640 1 -- 172.21.15.83:0/2249145771 shutdown_connections 2024-09-19T07:47:44.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.814+0000 7f373fd04640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f371c0678c0 0x7f371c069d80 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:44.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.814+0000 7f373fd04640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3738150600 0x7f373815b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:44.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.814+0000 7f373fd04640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3738150220 0x7f373815b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:44.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.814+0000 7f373fd04640 1 --2- 172.21.15.83:0/2249145771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f373806d3e0 0x7f373814aad0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:44.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.814+0000 7f373fd04640 1 -- 172.21.15.83:0/2249145771 >> 172.21.15.83:0/2249145771 conn(0x7f37380c48b0 msgr2=0x7f37381567f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:44.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.814+0000 7f373fd04640 1 -- 172.21.15.83:0/2249145771 shutdown_connections 2024-09-19T07:47:44.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:44.814+0000 7f373fd04640 1 -- 172.21.15.83:0/2249145771 wait complete. 2024-09-19T07:47:44.835 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:47:46.836 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:46.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78b142b640 1 Processor -- start 2024-09-19T07:47:46.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78b142b640 1 -- start start 2024-09-19T07:47:46.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78b142b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78ac06cef0 0x7f78ac0728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:46.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78b142b640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f78ac0c4b00 0x7f78ac14fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:46.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78b142b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78ac1504f0 con 0x7f78ac150040 2024-09-19T07:47:46.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78b142b640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f78ac152cd0 con 0x7f78ac0c4b00 2024-09-19T07:47:46.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78b142b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f78ac152fb0 con 0x7f78ac06cef0 2024-09-19T07:47:46.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78a3fff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f78ac0c4b00 0x7f78ac14fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:46.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78aaffd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78ac06cef0 0x7f78ac0728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:46.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78a3fff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f78ac0c4b00 0x7f78ac14fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:49692/0 (socket says 172.21.15.83:49692) 2024-09-19T07:47:46.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78a3fff640 1 -- 172.21.15.83:0/2773059399 learned_addr learned my addr 172.21.15.83:0/2773059399 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:46.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78aaffd640 1 -- 172.21.15.83:0/2773059399 >> v2:172.21.15.184:3300/0 conn(0x7f78ac0c4b00 msgr2=0x7f78ac14fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:46.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.986+0000 7f78aaffd640 1 --2- 172.21.15.83:0/2773059399 >> v2:172.21.15.184:3300/0 conn(0x7f78ac0c4b00 0x7f78ac14fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:46.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.990+0000 7f78aaffd640 1 -- 172.21.15.83:0/2773059399 >> v1:172.21.15.184:6789/0 conn(0x7f78ac150040 legacy=0x7f78ac0c3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:46.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.990+0000 7f78aaffd640 1 -- 172.21.15.83:0/2773059399 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac153290 con 0x7f78ac06cef0 2024-09-19T07:47:46.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.994+0000 7f78aaffd640 1 --2- 172.21.15.83:0/2773059399 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78ac06cef0 0x7f78ac0728b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f789c0097c0 tx=0x7f789c01e5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77c176539d7ab1b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:46.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.994+0000 7f78aa7fc640 1 -- 172.21.15.83:0/2773059399 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c0b2070 con 0x7f78ac06cef0 2024-09-19T07:47:46.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.994+0000 7f78aa7fc640 1 -- 172.21.15.83:0/2773059399 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f789c0ad600 con 0x7f78ac06cef0 2024-09-19T07:47:46.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.994+0000 7f78aa7fc640 1 -- 172.21.15.83:0/2773059399 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c0ad9d0 con 0x7f78ac06cef0 2024-09-19T07:47:46.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.998+0000 7f78b142b640 1 -- 172.21.15.83:0/2773059399 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78ac06cef0 msgr2=0x7f78ac0728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:46.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.998+0000 7f78b142b640 1 --2- 172.21.15.83:0/2773059399 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78ac06cef0 0x7f78ac0728b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f789c0097c0 tx=0x7f789c01e5e0 comp rx=0 tx=0).stop 2024-09-19T07:47:46.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.998+0000 7f78b142b640 1 -- 172.21.15.83:0/2773059399 shutdown_connections 2024-09-19T07:47:46.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.998+0000 7f78b142b640 1 --2- 172.21.15.83:0/2773059399 >> v2:172.21.15.184:3300/0 conn(0x7f78ac0c4b00 0x7f78ac14fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:46.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.998+0000 7f78b142b640 1 --2- 172.21.15.83:0/2773059399 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78ac06cef0 0x7f78ac0728b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:46.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.998+0000 7f78b142b640 1 -- 172.21.15.83:0/2773059399 >> 172.21.15.83:0/2773059399 conn(0x7f78ac0c5090 msgr2=0x7f78ac0c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:46.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.998+0000 7f78b142b640 1 -- 172.21.15.83:0/2773059399 shutdown_connections 2024-09-19T07:47:46.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.998+0000 7f78b142b640 1 -- 172.21.15.83:0/2773059399 wait complete. 2024-09-19T07:47:46.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.998+0000 7f78b142b640 1 Processor -- start 2024-09-19T07:47:46.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:46.998+0000 7f78b142b640 1 -- start start 2024-09-19T07:47:47.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78b142b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78ac06cef0 0x7f78ac14cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:47.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78b142b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78ac0c4b00 0x7f78ac0c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:47.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78b142b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78ac150040 0x7f78ac0c7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:47.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78b142b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac152a90 con 0x7f78ac0c4b00 2024-09-19T07:47:47.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78b142b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac06d4e0 con 0x7f78ac150040 2024-09-19T07:47:47.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78b142b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f78ac0757a0 con 0x7f78ac06cef0 2024-09-19T07:47:47.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78ac0c4b00 0x7f78ac0c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:47.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78ac0c4b00 0x7f78ac0c6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58324/0 (socket says 172.21.15.83:58324) 2024-09-19T07:47:47.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a3fff640 1 -- 172.21.15.83:0/244229610 learned_addr learned my addr 172.21.15.83:0/244229610 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78aaffd640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78ac06cef0 0x7f78ac14cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78ab7fe640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78ac150040 0x7f78ac0c7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a3fff640 1 -- 172.21.15.83:0/244229610 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78ac06cef0 msgr2=0x7f78ac14cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a3fff640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78ac06cef0 0x7f78ac14cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a3fff640 1 -- 172.21.15.83:0/244229610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78ac150040 msgr2=0x7f78ac0c7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a3fff640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78ac150040 0x7f78ac0c7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a3fff640 1 -- 172.21.15.83:0/244229610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac0ac060 con 0x7f78ac0c4b00 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a3fff640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78ac0c4b00 0x7f78ac0c6d30 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f789400e8c0 tx=0x7f789400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a37fe640 1 -- 172.21.15.83:0/244229610 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7894017070 con 0x7f78ac0c4b00 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a37fe640 1 -- 172.21.15.83:0/244229610 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f789400c940 con 0x7f78ac0c4b00 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78b142b640 1 -- 172.21.15.83:0/244229610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ac1eafc0 con 0x7f78ac0c4b00 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a37fe640 1 -- 172.21.15.83:0/244229610 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789400cd10 con 0x7f78ac0c4b00 2024-09-19T07:47:47.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78b142b640 1 -- 172.21.15.83:0/244229610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ac0c6530 con 0x7f78ac0c4b00 2024-09-19T07:47:47.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78b142b640 1 -- 172.21.15.83:0/244229610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78ac075ba0 con 0x7f78ac0c4b00 2024-09-19T07:47:47.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a37fe640 1 -- 172.21.15.83:0/244229610 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f789401a070 con 0x7f78ac0c4b00 2024-09-19T07:47:47.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.002+0000 7f78a37fe640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f788c0678c0 0x7f788c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:47.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.006+0000 7f78a37fe640 1 -- 172.21.15.83:0/244229610 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7f7894086b70 con 0x7f78ac0c4b00 2024-09-19T07:47:47.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.006+0000 7f78aaffd640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f788c0678c0 0x7f788c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:47.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.006+0000 7f78aaffd640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f788c0678c0 0x7f788c069d80 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f789c0af000 tx=0x7f789c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:47.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.014+0000 7f78a37fe640 1 -- 172.21.15.83:0/244229610 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7894056090 con 0x7f78ac0c4b00 2024-09-19T07:47:47.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.298+0000 7f78b142b640 1 -- 172.21.15.83:0/244229610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78ac06ff20 con 0x7f78ac0c4b00 2024-09-19T07:47:47.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.302+0000 7f78a37fe640 1 -- 172.21.15.83:0/244229610 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+6542 (secure 0 0 0) 0x7f789405a390 con 0x7f78ac0c4b00 2024-09-19T07:47:47.299 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:47.299 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":190,"btime":"2024-09-19T07:47:10:732800+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:47:47.299 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 190 2024-09-19T07:47:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.306+0000 7f78b142b640 1 -- 172.21.15.83:0/244229610 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f788c0678c0 msgr2=0x7f788c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.306+0000 7f78b142b640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f788c0678c0 0x7f788c069d80 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f789c0af000 tx=0x7f789c008040 comp rx=0 tx=0).stop 2024-09-19T07:47:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.306+0000 7f78b142b640 1 -- 172.21.15.83:0/244229610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78ac0c4b00 msgr2=0x7f78ac0c6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:47.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.306+0000 7f78b142b640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78ac0c4b00 0x7f78ac0c6d30 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f789400e8c0 tx=0x7f789400ed80 comp rx=0 tx=0).stop 2024-09-19T07:47:47.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.306+0000 7f78b142b640 1 -- 172.21.15.83:0/244229610 shutdown_connections 2024-09-19T07:47:47.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.306+0000 7f78b142b640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f788c0678c0 0x7f788c069d80 secure :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f789c0af000 tx=0x7f789c008040 comp rx=0 tx=0).stop 2024-09-19T07:47:47.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.306+0000 7f78b142b640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78ac150040 0x7f78ac0c7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:47.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.306+0000 7f78b142b640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f78ac0c4b00 0x7f78ac0c6d30 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:47.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.306+0000 7f78b142b640 1 --2- 172.21.15.83:0/244229610 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78ac06cef0 0x7f78ac14cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:47.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.306+0000 7f78b142b640 1 -- 172.21.15.83:0/244229610 >> 172.21.15.83:0/244229610 conn(0x7f78ac0c5090 msgr2=0x7f78ac1511d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:47.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.306+0000 7f78b142b640 1 -- 172.21.15.83:0/244229610 shutdown_connections 2024-09-19T07:47:47.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:47.306+0000 7f78b142b640 1 -- 172.21.15.83:0/244229610 wait complete. 2024-09-19T07:47:47.330 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:47:49.331 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:49.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f460c20a640 1 Processor -- start 2024-09-19T07:47:49.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f460c20a640 1 -- start start 2024-09-19T07:47:49.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f460c20a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46040c0c80 0x7f4604156790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:49.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f460c20a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4604153040 0x7f4604153480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f460c20a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f460406e010 0x7f460406e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f460c20a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f46040acb90 con 0x7f4604153040 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f460c20a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f46040abf80 con 0x7f460406e010 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f460c20a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f46040ac260 con 0x7f46040c0c80 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f46097a5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4604153040 0x7f4604153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f46097a5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4604153040 0x7f4604153480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58326/0 (socket says 172.21.15.83:58326) 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f46097a5640 1 -- 172.21.15.83:0/352678009 learned_addr learned my addr 172.21.15.83:0/352678009 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f4609fa6640 1 --2- 172.21.15.83:0/352678009 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46040c0c80 0x7f4604156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f460a7a7640 1 --2- 172.21.15.83:0/352678009 >> v2:172.21.15.184:3300/0 conn(0x7f460406e010 0x7f460406e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f46097a5640 1 -- 172.21.15.83:0/352678009 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46040c0c80 msgr2=0x7f4604156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f46097a5640 1 --2- 172.21.15.83:0/352678009 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46040c0c80 0x7f4604156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f46097a5640 1 -- 172.21.15.83:0/352678009 >> v2:172.21.15.184:3300/0 conn(0x7f460406e010 msgr2=0x7f460406e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:49.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f46097a5640 1 --2- 172.21.15.83:0/352678009 >> v2:172.21.15.184:3300/0 conn(0x7f460406e010 0x7f460406e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:49.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.482+0000 7f46097a5640 1 -- 172.21.15.83:0/352678009 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46040ac540 con 0x7f4604153040 2024-09-19T07:47:49.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.486+0000 7f46097a5640 1 --2- 172.21.15.83:0/352678009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4604153040 0x7f4604153480 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f46040c1bf0 tx=0x7f45f401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db1d26c041452664 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:49.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.486+0000 7f4608fa4640 1 -- 172.21.15.83:0/352678009 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f40b0070 con 0x7f4604153040 2024-09-19T07:47:49.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.486+0000 7f4608fa4640 1 -- 172.21.15.83:0/352678009 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45f40ab600 con 0x7f4604153040 2024-09-19T07:47:49.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.490+0000 7f460c20a640 1 -- 172.21.15.83:0/352678009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4604153040 msgr2=0x7f4604153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:49.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.490+0000 7f460c20a640 1 --2- 172.21.15.83:0/352678009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4604153040 0x7f4604153480 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f46040c1bf0 tx=0x7f45f401c520 comp rx=0 tx=0).stop 2024-09-19T07:47:49.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.490+0000 7f460c20a640 1 -- 172.21.15.83:0/352678009 shutdown_connections 2024-09-19T07:47:49.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.490+0000 7f460c20a640 1 --2- 172.21.15.83:0/352678009 >> v2:172.21.15.184:3300/0 conn(0x7f460406e010 0x7f460406e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:49.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.490+0000 7f460c20a640 1 --2- 172.21.15.83:0/352678009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4604153040 0x7f4604153480 secure :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f46040c1bf0 tx=0x7f45f401c520 comp rx=0 tx=0).stop 2024-09-19T07:47:49.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.490+0000 7f460c20a640 1 --2- 172.21.15.83:0/352678009 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46040c0c80 0x7f4604156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:49.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.490+0000 7f460c20a640 1 -- 172.21.15.83:0/352678009 >> 172.21.15.83:0/352678009 conn(0x7f4604075390 msgr2=0x7f460406bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:49.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.490+0000 7f460c20a640 1 -- 172.21.15.83:0/352678009 shutdown_connections 2024-09-19T07:47:49.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.490+0000 7f460c20a640 1 -- 172.21.15.83:0/352678009 wait complete. 2024-09-19T07:47:49.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f460c20a640 1 Processor -- start 2024-09-19T07:47:49.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f460c20a640 1 -- start start 2024-09-19T07:47:49.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f460c20a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f460406e010 0x7f4604150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:49.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f460c20a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46040c0c80 0x7f460413fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:49.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f460c20a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4604151010 0x7f4604140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f460c20a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f46040c1060 con 0x7f460406e010 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f460c20a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f46040abd40 con 0x7f4604151010 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f460c20a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f46040c11e0 con 0x7f46040c0c80 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f4609fa6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f460406e010 0x7f4604150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f4609fa6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f460406e010 0x7f4604150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58334/0 (socket says 172.21.15.83:58334) 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f4609fa6640 1 -- 172.21.15.83:0/2908446056 learned_addr learned my addr 172.21.15.83:0/2908446056 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f4609fa6640 1 -- 172.21.15.83:0/2908446056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46040c0c80 msgr2=0x7f460413fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f4609fa6640 1 --2- 172.21.15.83:0/2908446056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46040c0c80 0x7f460413fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f4609fa6640 1 -- 172.21.15.83:0/2908446056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4604151010 msgr2=0x7f4604140020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f4609fa6640 1 --2- 172.21.15.83:0/2908446056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4604151010 0x7f4604140020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f4609fa6640 1 -- 172.21.15.83:0/2908446056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4604143790 con 0x7f460406e010 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f4609fa6640 1 --2- 172.21.15.83:0/2908446056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f460406e010 0x7f4604150ad0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f45fc00ef00 tx=0x7f45fc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:49.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f45fa7fc640 1 -- 172.21.15.83:0/2908446056 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45fc018020 con 0x7f460406e010 2024-09-19T07:47:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f45fa7fc640 1 -- 172.21.15.83:0/2908446056 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45fc00bd90 con 0x7f460406e010 2024-09-19T07:47:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f460c20a640 1 -- 172.21.15.83:0/2908446056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4604143a70 con 0x7f460406e010 2024-09-19T07:47:49.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.494+0000 7f45fa7fc640 1 -- 172.21.15.83:0/2908446056 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45fc014620 con 0x7f460406e010 2024-09-19T07:47:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.498+0000 7f460c20a640 1 -- 172.21.15.83:0/2908446056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46040c3bd0 con 0x7f460406e010 2024-09-19T07:47:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.498+0000 7f45fa7fc640 1 -- 172.21.15.83:0/2908446056 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f45fc01b020 con 0x7f460406e010 2024-09-19T07:47:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.498+0000 7f460c20a640 1 -- 172.21.15.83:0/2908446056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46040c36f0 con 0x7f460406e010 2024-09-19T07:47:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.498+0000 7f45fa7fc640 1 --2- 172.21.15.83:0/2908446056 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f45e80678f0 0x7f45e8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.498+0000 7f45fa7fc640 1 -- 172.21.15.83:0/2908446056 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7f45fc08a170 con 0x7f460406e010 2024-09-19T07:47:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.498+0000 7f46097a5640 1 --2- 172.21.15.83:0/2908446056 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f45e80678f0 0x7f45e8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:49.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.498+0000 7f46097a5640 1 --2- 172.21.15.83:0/2908446056 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f45e80678f0 0x7f45e8069db0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f45f40ad000 tx=0x7f45f4004740 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:49.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.502+0000 7f45fa7fc640 1 -- 172.21.15.83:0/2908446056 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f45fc056360 con 0x7f460406e010 2024-09-19T07:47:49.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.806+0000 7f460c20a640 1 -- 172.21.15.83:0/2908446056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f460406e4a0 con 0x7f460406e010 2024-09-19T07:47:49.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.806+0000 7f45fa7fc640 1 -- 172.21.15.83:0/2908446056 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+6542 (secure 0 0 0) 0x7f45fc055c60 con 0x7f460406e010 2024-09-19T07:47:49.806 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:49.807 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":190,"btime":"2024-09-19T07:47:10:732800+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:47:49.807 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 190 2024-09-19T07:47:49.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.810+0000 7f460c20a640 1 -- 172.21.15.83:0/2908446056 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f45e80678f0 msgr2=0x7f45e8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:49.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.810+0000 7f460c20a640 1 --2- 172.21.15.83:0/2908446056 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f45e80678f0 0x7f45e8069db0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f45f40ad000 tx=0x7f45f4004740 comp rx=0 tx=0).stop 2024-09-19T07:47:49.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.810+0000 7f460c20a640 1 -- 172.21.15.83:0/2908446056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f460406e010 msgr2=0x7f4604150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:49.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.810+0000 7f460c20a640 1 --2- 172.21.15.83:0/2908446056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f460406e010 0x7f4604150ad0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f45fc00ef00 tx=0x7f45fc00b480 comp rx=0 tx=0).stop 2024-09-19T07:47:49.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.810+0000 7f460c20a640 1 -- 172.21.15.83:0/2908446056 shutdown_connections 2024-09-19T07:47:49.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.810+0000 7f460c20a640 1 --2- 172.21.15.83:0/2908446056 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f45e80678f0 0x7f45e8069db0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:49.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.810+0000 7f460c20a640 1 --2- 172.21.15.83:0/2908446056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4604151010 0x7f4604140020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:49.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.810+0000 7f460c20a640 1 --2- 172.21.15.83:0/2908446056 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46040c0c80 0x7f460413fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:49.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.810+0000 7f460c20a640 1 --2- 172.21.15.83:0/2908446056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f460406e010 0x7f4604150ad0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:49.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.810+0000 7f460c20a640 1 -- 172.21.15.83:0/2908446056 >> 172.21.15.83:0/2908446056 conn(0x7f4604075390 msgr2=0x7f46041554c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:49.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.810+0000 7f460c20a640 1 -- 172.21.15.83:0/2908446056 shutdown_connections 2024-09-19T07:47:49.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:49.810+0000 7f460c20a640 1 -- 172.21.15.83:0/2908446056 wait complete. 2024-09-19T07:47:49.830 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:47:51.831 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:51.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.978+0000 7fed46661640 1 Processor -- start 2024-09-19T07:47:51.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.978+0000 7fed46661640 1 -- start start 2024-09-19T07:47:51.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.982+0000 7fed46661640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fed4015ae80 0x7fed40157460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:51.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.982+0000 7fed46661640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed4015a100 con 0x7fed400abf70 2024-09-19T07:47:51.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.982+0000 7fed46661640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed4006f040 con 0x7fed40073110 2024-09-19T07:47:51.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.982+0000 7fed46661640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fed4015a3a0 con 0x7fed4015ae80 2024-09-19T07:47:51.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.982+0000 7fed44bfe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fed400abf70 0x7fed40164a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46058/0 (socket says 172.21.15.83:46058) 2024-09-19T07:47:51.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.982+0000 7fed44bfe640 1 -- 172.21.15.83:0/551412965 learned_addr learned my addr 172.21.15.83:0/551412965 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:47:51.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.982+0000 7fed37fff640 1 --2- 172.21.15.83:0/551412965 >> v2:172.21.15.83:3300/0 conn(0x7fed4015ae80 0x7fed40157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:51.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.982+0000 7fed37fff640 1 -- 172.21.15.83:0/551412965 >> v1:172.21.15.184:6789/0 conn(0x7fed40073110 legacy=0x7fed4015d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:51.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.982+0000 7fed37fff640 1 -- 172.21.15.83:0/551412965 >> v1:172.21.15.83:6789/0 conn(0x7fed400abf70 legacy=0x7fed40164a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:51.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.982+0000 7fed37fff640 1 -- 172.21.15.83:0/551412965 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4015a680 con 0x7fed4015ae80 2024-09-19T07:47:51.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.994+0000 7fed37fff640 1 --2- 172.21.15.83:0/551412965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed4015ae80 0x7fed40157460 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fed280097c0 tx=0x7fed2801bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad48a323427cd00d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:51.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.994+0000 7fed3f7fe640 1 -- 172.21.15.83:0/551412965 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed280aa660 con 0x7fed4015ae80 2024-09-19T07:47:51.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.994+0000 7fed3f7fe640 1 -- 172.21.15.83:0/551412965 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed280af070 con 0x7fed4015ae80 2024-09-19T07:47:51.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.994+0000 7fed3f7fe640 1 -- 172.21.15.83:0/551412965 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed280aaa20 con 0x7fed4015ae80 2024-09-19T07:47:51.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.994+0000 7fed46661640 1 -- 172.21.15.83:0/551412965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed4015ae80 msgr2=0x7fed40157460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:51.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.994+0000 7fed46661640 1 --2- 172.21.15.83:0/551412965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed4015ae80 0x7fed40157460 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fed280097c0 tx=0x7fed2801bec0 comp rx=0 tx=0).stop 2024-09-19T07:47:51.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 -- 172.21.15.83:0/551412965 shutdown_connections 2024-09-19T07:47:51.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 --2- 172.21.15.83:0/551412965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed4015ae80 0x7fed40157460 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:51.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 -- 172.21.15.83:0/551412965 >> 172.21.15.83:0/551412965 conn(0x7fed400c95a0 msgr2=0x7fed400c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:51.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 -- 172.21.15.83:0/551412965 shutdown_connections 2024-09-19T07:47:51.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 -- 172.21.15.83:0/551412965 wait complete. 2024-09-19T07:47:51.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 Processor -- start 2024-09-19T07:47:51.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 -- start start 2024-09-19T07:47:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed40073110 0x7fed40152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed400abf70 0x7fed4014cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed4015ae80 0x7fed4014d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fed400734f0 con 0x7fed400abf70 2024-09-19T07:47:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fed40157db0 con 0x7fed4015ae80 2024-09-19T07:47:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed46661640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fed40073670 con 0x7fed40073110 2024-09-19T07:47:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed37fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed400abf70 0x7fed4014cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed37fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed400abf70 0x7fed4014cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46620/0 (socket says 172.21.15.83:46620) 2024-09-19T07:47:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed37fff640 1 -- 172.21.15.83:0/3950725102 learned_addr learned my addr 172.21.15.83:0/3950725102 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed3ffff640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed40073110 0x7fed40152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed37fff640 1 -- 172.21.15.83:0/3950725102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed40073110 msgr2=0x7fed40152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed37fff640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed40073110 0x7fed40152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:52.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed37fff640 1 -- 172.21.15.83:0/3950725102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed4015ae80 msgr2=0x7fed4014d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:52.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed37fff640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed4015ae80 0x7fed4014d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:52.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed37fff640 1 -- 172.21.15.83:0/3950725102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4015cd00 con 0x7fed400abf70 2024-09-19T07:47:52.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed3ffff640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed40073110 0x7fed40152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:47:52.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed37fff640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed400abf70 0x7fed4014cc90 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fed28019d00 tx=0x7fed2801a700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:52.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed3d7fa640 1 -- 172.21.15.83:0/3950725102 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed280b4810 con 0x7fed400abf70 2024-09-19T07:47:52.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed3d7fa640 1 -- 172.21.15.83:0/3950725102 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed280b49d0 con 0x7fed400abf70 2024-09-19T07:47:52.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:51.998+0000 7fed3d7fa640 1 -- 172.21.15.83:0/3950725102 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed280b4dc0 con 0x7fed400abf70 2024-09-19T07:47:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.002+0000 7fed46661640 1 -- 172.21.15.83:0/3950725102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4015cfe0 con 0x7fed400abf70 2024-09-19T07:47:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.002+0000 7fed46661640 1 -- 172.21.15.83:0/3950725102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed40153820 con 0x7fed400abf70 2024-09-19T07:47:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.002+0000 7fed46661640 1 -- 172.21.15.83:0/3950725102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed40157460 con 0x7fed400abf70 2024-09-19T07:47:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.002+0000 7fed3d7fa640 1 -- 172.21.15.83:0/3950725102 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fed280bc020 con 0x7fed400abf70 2024-09-19T07:47:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.002+0000 7fed3d7fa640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed200678c0 0x7fed20069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:52.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.002+0000 7fed3d7fa640 1 -- 172.21.15.83:0/3950725102 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7fed28120910 con 0x7fed400abf70 2024-09-19T07:47:52.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.006+0000 7fed3ffff640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed200678c0 0x7fed20069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.010+0000 7fed3d7fa640 1 -- 172.21.15.83:0/3950725102 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fed28015fb0 con 0x7fed400abf70 2024-09-19T07:47:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.010+0000 7fed3ffff640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed200678c0 0x7fed20069d80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fed30001ec0 tx=0x7fed30009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:52.278 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:52.279 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":190,"btime":"2024-09-19T07:47:10:732800+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:47:52.279 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 190 2024-09-19T07:47:52.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.278+0000 7fed46661640 1 -- 172.21.15.83:0/3950725102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed40151650 con 0x7fed400abf70 2024-09-19T07:47:52.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.278+0000 7fed3d7fa640 1 -- 172.21.15.83:0/3950725102 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+6542 (secure 0 0 0) 0x7fed280f0390 con 0x7fed400abf70 2024-09-19T07:47:52.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.286+0000 7fed46661640 1 -- 172.21.15.83:0/3950725102 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed200678c0 msgr2=0x7fed20069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:52.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.286+0000 7fed46661640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed200678c0 0x7fed20069d80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fed30001ec0 tx=0x7fed30009040 comp rx=0 tx=0).stop 2024-09-19T07:47:52.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.286+0000 7fed46661640 1 -- 172.21.15.83:0/3950725102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed400abf70 msgr2=0x7fed4014cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:52.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.286+0000 7fed46661640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed400abf70 0x7fed4014cc90 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fed28019d00 tx=0x7fed2801a700 comp rx=0 tx=0).stop 2024-09-19T07:47:52.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.286+0000 7fed46661640 1 -- 172.21.15.83:0/3950725102 shutdown_connections 2024-09-19T07:47:52.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.286+0000 7fed46661640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fed200678c0 0x7fed20069d80 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:52.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.286+0000 7fed46661640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed4015ae80 0x7fed4014d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:52.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.286+0000 7fed46661640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed400abf70 0x7fed4014cc90 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:52.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.286+0000 7fed46661640 1 --2- 172.21.15.83:0/3950725102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed40073110 0x7fed40152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:52.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.286+0000 7fed46661640 1 -- 172.21.15.83:0/3950725102 >> 172.21.15.83:0/3950725102 conn(0x7fed400c95a0 msgr2=0x7fed40150780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:52.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.286+0000 7fed46661640 1 -- 172.21.15.83:0/3950725102 shutdown_connections 2024-09-19T07:47:52.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:52.286+0000 7fed46661640 1 -- 172.21.15.83:0/3950725102 wait complete. 2024-09-19T07:47:52.306 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:47:54.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:54.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.454+0000 7f3b995b9640 1 Processor -- start 2024-09-19T07:47:54.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.454+0000 7f3b995b9640 1 -- start start 2024-09-19T07:47:54.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.454+0000 7f3b995b9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3b940ac290 0x7f3b940c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:54.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.454+0000 7f3b995b9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940c3a50 0x7f3b941563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:54.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.454+0000 7f3b995b9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b9406e110 con 0x7f3b940c3130 2024-09-19T07:47:54.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.454+0000 7f3b995b9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3b940ac700 con 0x7f3b940ac290 2024-09-19T07:47:54.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.454+0000 7f3b995b9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3b940c4990 con 0x7f3b940c3a50 2024-09-19T07:47:54.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.454+0000 7f3b927fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3b940ac290 0x7f3b940c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b927fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3b940ac290 0x7f3b940c3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46628/0 (socket says 172.21.15.83:46628) 2024-09-19T07:47:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b927fc640 1 -- 172.21.15.83:0/4159369668 learned_addr learned my addr 172.21.15.83:0/4159369668 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b937fe640 1 --2- 172.21.15.83:0/4159369668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940c3a50 0x7f3b941563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b927fc640 1 -- 172.21.15.83:0/4159369668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940c3a50 msgr2=0x7f3b941563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b927fc640 1 --2- 172.21.15.83:0/4159369668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940c3a50 0x7f3b941563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b91ffb640 1 -- 172.21.15.83:0/4159369668 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1657918506 0 0) 0x7f3b9406e110 con 0x7f3b940c3130 2024-09-19T07:47:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b927fc640 1 -- 172.21.15.83:0/4159369668 >> v1:172.21.15.83:6789/0 conn(0x7f3b940c3130 legacy=0x7f3b9406da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b927fc640 1 -- 172.21.15.83:0/4159369668 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b940c4c70 con 0x7f3b940ac290 2024-09-19T07:47:54.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b927fc640 1 --2- 172.21.15.83:0/4159369668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b940ac290 0x7f3b940c3510 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f3b88009870 tx=0x7f3b8801c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82df40aa12ca43c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:54.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b91ffb640 1 -- 172.21.15.83:0/4159369668 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b880b0070 con 0x7f3b940ac290 2024-09-19T07:47:54.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b91ffb640 1 -- 172.21.15.83:0/4159369668 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b880ab600 con 0x7f3b940ac290 2024-09-19T07:47:54.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b91ffb640 1 -- 172.21.15.83:0/4159369668 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b880ab9d0 con 0x7f3b940ac290 2024-09-19T07:47:54.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b995b9640 1 -- 172.21.15.83:0/4159369668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b940ac290 msgr2=0x7f3b940c3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:54.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b995b9640 1 --2- 172.21.15.83:0/4159369668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b940ac290 0x7f3b940c3510 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f3b88009870 tx=0x7f3b8801c210 comp rx=0 tx=0).stop 2024-09-19T07:47:54.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b995b9640 1 -- 172.21.15.83:0/4159369668 shutdown_connections 2024-09-19T07:47:54.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b995b9640 1 --2- 172.21.15.83:0/4159369668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b940ac290 0x7f3b940c3510 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:54.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b995b9640 1 --2- 172.21.15.83:0/4159369668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940c3a50 0x7f3b941563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:54.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b995b9640 1 -- 172.21.15.83:0/4159369668 >> 172.21.15.83:0/4159369668 conn(0x7f3b940c5060 msgr2=0x7f3b940c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:54.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b995b9640 1 -- 172.21.15.83:0/4159369668 shutdown_connections 2024-09-19T07:47:54.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.458+0000 7f3b995b9640 1 -- 172.21.15.83:0/4159369668 wait complete. 2024-09-19T07:47:54.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b995b9640 1 Processor -- start 2024-09-19T07:47:54.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b995b9640 1 -- start start 2024-09-19T07:47:54.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b995b9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940ac290 0x7f3b94074f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:54.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b995b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b940c3130 0x7f3b9414d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:54.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b995b9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b940c3a50 0x7f3b9414d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:54.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b995b9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3b940ad520 con 0x7f3b940c3130 2024-09-19T07:47:54.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b995b9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3b940c82a0 con 0x7f3b940c3a50 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b995b9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3b940ad6a0 con 0x7f3b940ac290 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b927fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b940c3130 0x7f3b9414d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b927fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b940c3130 0x7f3b9414d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46644/0 (socket says 172.21.15.83:46644) 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b927fc640 1 -- 172.21.15.83:0/2593610054 learned_addr learned my addr 172.21.15.83:0/2593610054 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b92ffd640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940ac290 0x7f3b94074f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b927fc640 1 -- 172.21.15.83:0/2593610054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940ac290 msgr2=0x7f3b94074f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b927fc640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940ac290 0x7f3b94074f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b927fc640 1 -- 172.21.15.83:0/2593610054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b940c3a50 msgr2=0x7f3b9414d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b927fc640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b940c3a50 0x7f3b9414d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b927fc640 1 -- 172.21.15.83:0/2593610054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b9415a930 con 0x7f3b940c3130 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b927fc640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b940c3130 0x7f3b9414d310 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f3b880ae040 tx=0x7f3b88018fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b7b7fe640 1 -- 172.21.15.83:0/2593610054 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b880abe60 con 0x7f3b940c3130 2024-09-19T07:47:54.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b7b7fe640 1 -- 172.21.15.83:0/2593610054 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b880b4420 con 0x7f3b940c3130 2024-09-19T07:47:54.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b995b9640 1 -- 172.21.15.83:0/2593610054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b941f0360 con 0x7f3b940c3130 2024-09-19T07:47:54.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b7b7fe640 1 -- 172.21.15.83:0/2593610054 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b880b0040 con 0x7f3b940c3130 2024-09-19T07:47:54.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b995b9640 1 -- 172.21.15.83:0/2593610054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b94153170 con 0x7f3b940c3130 2024-09-19T07:47:54.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.462+0000 7f3b995b9640 1 -- 172.21.15.83:0/2593610054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b940c3510 con 0x7f3b940c3130 2024-09-19T07:47:54.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.466+0000 7f3b7b7fe640 1 -- 172.21.15.83:0/2593610054 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3b880b4810 con 0x7f3b940c3130 2024-09-19T07:47:54.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.466+0000 7f3b7b7fe640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3b800678c0 0x7f3b80069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:54.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.466+0000 7f3b7b7fe640 1 -- 172.21.15.83:0/2593610054 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7f3b881206e0 con 0x7f3b940c3130 2024-09-19T07:47:54.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.470+0000 7f3b92ffd640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3b800678c0 0x7f3b80069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:54.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.470+0000 7f3b92ffd640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3b800678c0 0x7f3b80069d80 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3b84000f70 tx=0x7f3b84009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:54.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.474+0000 7f3b7b7fe640 1 -- 172.21.15.83:0/2593610054 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f3b88016150 con 0x7f3b940c3130 2024-09-19T07:47:54.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.766+0000 7f3b995b9640 1 -- 172.21.15.83:0/2593610054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b940a7cb0 con 0x7f3b940c3130 2024-09-19T07:47:54.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.766+0000 7f3b7b7fe640 1 -- 172.21.15.83:0/2593610054 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+6542 (secure 0 0 0) 0x7f3b880f0250 con 0x7f3b940c3130 2024-09-19T07:47:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:54.766 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":190,"btime":"2024-09-19T07:47:10:732800+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:47:54.766 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 190 2024-09-19T07:47:54.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.770+0000 7f3b995b9640 1 -- 172.21.15.83:0/2593610054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3b800678c0 msgr2=0x7f3b80069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:54.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.770+0000 7f3b995b9640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3b800678c0 0x7f3b80069d80 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3b84000f70 tx=0x7f3b84009040 comp rx=0 tx=0).stop 2024-09-19T07:47:54.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.770+0000 7f3b995b9640 1 -- 172.21.15.83:0/2593610054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b940c3130 msgr2=0x7f3b9414d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:54.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.770+0000 7f3b995b9640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b940c3130 0x7f3b9414d310 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f3b880ae040 tx=0x7f3b88018fb0 comp rx=0 tx=0).stop 2024-09-19T07:47:54.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.770+0000 7f3b995b9640 1 -- 172.21.15.83:0/2593610054 shutdown_connections 2024-09-19T07:47:54.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.770+0000 7f3b995b9640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f3b800678c0 0x7f3b80069d80 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:54.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.770+0000 7f3b995b9640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b940c3a50 0x7f3b9414d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:54.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.770+0000 7f3b995b9640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b940c3130 0x7f3b9414d310 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:54.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.770+0000 7f3b995b9640 1 --2- 172.21.15.83:0/2593610054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940ac290 0x7f3b94074f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:54.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.770+0000 7f3b995b9640 1 -- 172.21.15.83:0/2593610054 >> 172.21.15.83:0/2593610054 conn(0x7f3b940c5060 msgr2=0x7f3b94153f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:54.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.770+0000 7f3b995b9640 1 -- 172.21.15.83:0/2593610054 shutdown_connections 2024-09-19T07:47:54.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:54.770+0000 7f3b995b9640 1 -- 172.21.15.83:0/2593610054 wait complete. 2024-09-19T07:47:54.790 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:47:56.792 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:56.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.930+0000 7f3923d34640 1 Processor -- start 2024-09-19T07:47:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.930+0000 7f3923d34640 1 -- start start 2024-09-19T07:47:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f3923d34640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f391c0c3ab0 0x7f391c157010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f3923d34640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f391c0aa4a0 0x7f391c157550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f3923d34640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f391c073d90 con 0x7f391c14dbe0 2024-09-19T07:47:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f3923d34640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f391c154700 con 0x7f391c0aa4a0 2024-09-19T07:47:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f3923d34640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f391c153b30 con 0x7f391c0c3ab0 2024-09-19T07:47:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f39222d1640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f391c0aa4a0 0x7f391c157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f39212cf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f391c0c3ab0 0x7f391c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:56.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f39222d1640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f391c0aa4a0 0x7f391c157550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:54654/0 (socket says 172.21.15.83:54654) 2024-09-19T07:47:56.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f39222d1640 1 -- 172.21.15.83:0/2768669778 learned_addr learned my addr 172.21.15.83:0/2768669778 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:56.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f39222d1640 1 -- 172.21.15.83:0/2768669778 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f391c0c3ab0 msgr2=0x7f391c157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:56.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f39222d1640 1 --2- 172.21.15.83:0/2768669778 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f391c0c3ab0 0x7f391c157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:56.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f39222d1640 1 -- 172.21.15.83:0/2768669778 >> v1:172.21.15.184:6789/0 conn(0x7f391c14dbe0 legacy=0x7f391c0c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:56.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.934+0000 7f39222d1640 1 -- 172.21.15.83:0/2768669778 --> v2:172.21.15.184:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391c153dd0 con 0x7f391c0aa4a0 2024-09-19T07:47:56.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f39222d1640 1 --2- 172.21.15.83:0/2768669778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f391c0aa4a0 0x7f391c157550 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3914009e50 tx=0x7f391401c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=875c86d879c21a79 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:56.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3920ace640 1 -- 172.21.15.83:0/2768669778 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39140a56f0 con 0x7f391c0aa4a0 2024-09-19T07:47:56.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3920ace640 1 -- 172.21.15.83:0/2768669778 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39140a5890 con 0x7f391c0aa4a0 2024-09-19T07:47:56.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3920ace640 1 -- 172.21.15.83:0/2768669778 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39140a5c60 con 0x7f391c0aa4a0 2024-09-19T07:47:56.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3923d34640 1 -- 172.21.15.83:0/2768669778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f391c0aa4a0 msgr2=0x7f391c157550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:56.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3923d34640 1 --2- 172.21.15.83:0/2768669778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f391c0aa4a0 0x7f391c157550 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3914009e50 tx=0x7f391401c960 comp rx=0 tx=0).stop 2024-09-19T07:47:56.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3923d34640 1 -- 172.21.15.83:0/2768669778 shutdown_connections 2024-09-19T07:47:56.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3923d34640 1 --2- 172.21.15.83:0/2768669778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f391c0aa4a0 0x7f391c157550 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:56.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3923d34640 1 --2- 172.21.15.83:0/2768669778 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f391c0c3ab0 0x7f391c157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:56.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3923d34640 1 -- 172.21.15.83:0/2768669778 >> 172.21.15.83:0/2768669778 conn(0x7f391c0c95a0 msgr2=0x7f391c0c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:56.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3923d34640 1 -- 172.21.15.83:0/2768669778 shutdown_connections 2024-09-19T07:47:56.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3923d34640 1 -- 172.21.15.83:0/2768669778 wait complete. 2024-09-19T07:47:56.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3923d34640 1 Processor -- start 2024-09-19T07:47:56.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.938+0000 7f3923d34640 1 -- start start 2024-09-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f3923d34640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f391c0aa4a0 0x7f391c15d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f3923d34640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f391c14dbe0 0x7f391c145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f3923d34640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f391c14e0e0 0x7f391c146190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39212cf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f391c14dbe0 0x7f391c145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39212cf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f391c14dbe0 0x7f391c145c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46660/0 (socket says 172.21.15.83:46660) 2024-09-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39212cf640 1 -- 172.21.15.83:0/694198336 learned_addr learned my addr 172.21.15.83:0/694198336 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f391c0c4830 con 0x7f391c14dbe0 2024-09-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f391c0c46b0 con 0x7f391c0aa4a0 2024-09-19T07:47:56.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f391c160bc0 con 0x7f391c14e0e0 2024-09-19T07:47:56.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39212cf640 1 -- 172.21.15.83:0/694198336 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f391c14e0e0 msgr2=0x7f391c146190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:56.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39212cf640 1 --2- 172.21.15.83:0/694198336 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f391c14e0e0 0x7f391c146190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:56.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39212cf640 1 -- 172.21.15.83:0/694198336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f391c0aa4a0 msgr2=0x7f391c15d830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:47:56.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39212cf640 1 --2- 172.21.15.83:0/694198336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f391c0aa4a0 0x7f391c15d830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:56.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39212cf640 1 -- 172.21.15.83:0/694198336 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391c1ef3a0 con 0x7f391c14dbe0 2024-09-19T07:47:56.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39212cf640 1 --2- 172.21.15.83:0/694198336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f391c14dbe0 0x7f391c145c50 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f390c00d7d0 tx=0x7f390c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:56.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39127fc640 1 -- 172.21.15.83:0/694198336 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c00b6c0 con 0x7f391c14dbe0 2024-09-19T07:47:56.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39127fc640 1 -- 172.21.15.83:0/694198336 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f390c019070 con 0x7f391c14dbe0 2024-09-19T07:47:56.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39127fc640 1 -- 172.21.15.83:0/694198336 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c00ba60 con 0x7f391c14dbe0 2024-09-19T07:47:56.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f391c1ef680 con 0x7f391c14dbe0 2024-09-19T07:47:56.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f391c14e5e0 con 0x7f391c14dbe0 2024-09-19T07:47:56.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39127fc640 1 -- 172.21.15.83:0/694198336 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f390c00bc00 con 0x7f391c14dbe0 2024-09-19T07:47:56.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39040052a0 con 0x7f391c14dbe0 2024-09-19T07:47:56.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39127fc640 1 --2- 172.21.15.83:0/694198336 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f39000678f0 0x7f3900069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:56.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f39127fc640 1 -- 172.21.15.83:0/694198336 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9819+0+0 (secure 0 0 0) 0x7f390c087020 con 0x7f391c14dbe0 2024-09-19T07:47:56.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.942+0000 7f3921ad0640 1 --2- 172.21.15.83:0/694198336 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f39000678f0 0x7f3900069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:56.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.946+0000 7f3921ad0640 1 --2- 172.21.15.83:0/694198336 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f39000678f0 0x7f3900069db0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3918007fd0 tx=0x7f39180079a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:56.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:56.950+0000 7f39127fc640 1 -- 172.21.15.83:0/694198336 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f390c056bd0 con 0x7f391c14dbe0 2024-09-19T07:47:57.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.226+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3904005590 con 0x7f391c14dbe0 2024-09-19T07:47:57.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.226+0000 7f39127fc640 1 -- 172.21.15.83:0/694198336 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+6542 (secure 0 0 0) 0x7f390c0564d0 con 0x7f391c14dbe0 2024-09-19T07:47:57.225 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":190,"btime":"2024-09-19T07:47:10:732800+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:45:31.612728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":181,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6835/119695014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":119695014},{"type":"v1","addr":"172.21.15.83:6835","nonce":119695014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:47:57.226 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 190 2024-09-19T07:47:57.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.230+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f39000678f0 msgr2=0x7f3900069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:57.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.230+0000 7f3923d34640 1 --2- 172.21.15.83:0/694198336 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f39000678f0 0x7f3900069db0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3918007fd0 tx=0x7f39180079a0 comp rx=0 tx=0).stop 2024-09-19T07:47:57.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.230+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f391c14dbe0 msgr2=0x7f391c145c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:57.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.230+0000 7f3923d34640 1 --2- 172.21.15.83:0/694198336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f391c14dbe0 0x7f391c145c50 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f390c00d7d0 tx=0x7f390c00dc90 comp rx=0 tx=0).stop 2024-09-19T07:47:57.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.230+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 shutdown_connections 2024-09-19T07:47:57.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.230+0000 7f3923d34640 1 --2- 172.21.15.83:0/694198336 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f39000678f0 0x7f3900069db0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:57.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.230+0000 7f3923d34640 1 --2- 172.21.15.83:0/694198336 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f391c14e0e0 0x7f391c146190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:57.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.230+0000 7f3923d34640 1 --2- 172.21.15.83:0/694198336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f391c14dbe0 0x7f391c145c50 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:57.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.230+0000 7f3923d34640 1 --2- 172.21.15.83:0/694198336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f391c0aa4a0 0x7f391c15d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:57.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.230+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 >> 172.21.15.83:0/694198336 conn(0x7f391c0c95a0 msgr2=0x7f391c14b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:57.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.230+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 shutdown_connections 2024-09-19T07:47:57.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:57.230+0000 7f3923d34640 1 -- 172.21.15.83:0/694198336 wait complete. 2024-09-19T07:47:57.247 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-19T07:47:59.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:59.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f506ea36640 1 Processor -- start 2024-09-19T07:47:59.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f506ea36640 1 -- start start 2024-09-19T07:47:59.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f506ea36640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5068074000 0x7f50680c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f506ea36640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f50681573a0 0x7f5068157830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f506ea36640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5068153a50 con 0x7f50681546a0 2024-09-19T07:47:59.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f506ea36640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f50680730e0 con 0x7f5068074000 2024-09-19T07:47:59.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f506ea36640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f50680733c0 con 0x7f50681573a0 2024-09-19T07:47:59.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f50677fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5068074000 0x7f50680c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:59.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f50677fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5068074000 0x7f50680c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46676/0 (socket says 172.21.15.83:46676) 2024-09-19T07:47:59.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f50677fe640 1 -- 172.21.15.83:0/1313090379 learned_addr learned my addr 172.21.15.83:0/1313090379 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:59.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f50677fe640 1 -- 172.21.15.83:0/1313090379 >> v2:172.21.15.184:3300/0 conn(0x7f50681573a0 msgr2=0x7f5068157830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:47:59.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f50677fe640 1 --2- 172.21.15.83:0/1313090379 >> v2:172.21.15.184:3300/0 conn(0x7f50681573a0 0x7f5068157830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f50677fe640 1 -- 172.21.15.83:0/1313090379 >> v1:172.21.15.184:6789/0 conn(0x7f50681546a0 legacy=0x7f506806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:59.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.394+0000 7f50677fe640 1 -- 172.21.15.83:0/1313090379 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50680736a0 con 0x7f5068074000 2024-09-19T07:47:59.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.398+0000 7f50677fe640 1 --2- 172.21.15.83:0/1313090379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068074000 0x7f50680c70f0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f50680ab6a0 tx=0x7f505c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e239029492760d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:59.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.398+0000 7f5066ffd640 1 -- 172.21.15.83:0/1313090379 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505c0a7070 con 0x7f5068074000 2024-09-19T07:47:59.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.398+0000 7f5066ffd640 1 -- 172.21.15.83:0/1313090379 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f505c0b0070 con 0x7f5068074000 2024-09-19T07:47:59.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.398+0000 7f5066ffd640 1 -- 172.21.15.83:0/1313090379 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505c0ab8d0 con 0x7f5068074000 2024-09-19T07:47:59.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.398+0000 7f506ea36640 1 -- 172.21.15.83:0/1313090379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068074000 msgr2=0x7f50680c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:59.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.398+0000 7f506ea36640 1 --2- 172.21.15.83:0/1313090379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068074000 0x7f50680c70f0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f50680ab6a0 tx=0x7f505c01c560 comp rx=0 tx=0).stop 2024-09-19T07:47:59.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.398+0000 7f506ea36640 1 -- 172.21.15.83:0/1313090379 shutdown_connections 2024-09-19T07:47:59.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.398+0000 7f506ea36640 1 --2- 172.21.15.83:0/1313090379 >> v2:172.21.15.184:3300/0 conn(0x7f50681573a0 0x7f5068157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.398+0000 7f506ea36640 1 --2- 172.21.15.83:0/1313090379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068074000 0x7f50680c70f0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.398+0000 7f506ea36640 1 -- 172.21.15.83:0/1313090379 >> 172.21.15.83:0/1313090379 conn(0x7f50680c5060 msgr2=0x7f50680c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:59.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.398+0000 7f506ea36640 1 -- 172.21.15.83:0/1313090379 shutdown_connections 2024-09-19T07:47:59.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.398+0000 7f506ea36640 1 -- 172.21.15.83:0/1313090379 wait complete. 2024-09-19T07:47:59.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f506ea36640 1 Processor -- start 2024-09-19T07:47:59.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f506ea36640 1 -- start start 2024-09-19T07:47:59.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f506ea36640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068074000 0x7f50680c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f506ea36640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50681546a0 0x7f506814eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f506ea36640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50681573a0 0x7f506814f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f506ea36640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5068157ef0 con 0x7f5068074000 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f506ea36640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5068157d70 con 0x7f50681546a0 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f506ea36640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5068070230 con 0x7f50681573a0 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f5067fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068074000 0x7f50680c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f5067fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068074000 0x7f50680c1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46690/0 (socket says 172.21.15.83:46690) 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f5067fff640 1 -- 172.21.15.83:0/2924575013 learned_addr learned my addr 172.21.15.83:0/2924575013 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f50677fe640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50681546a0 0x7f506814eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f506cfd3640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50681573a0 0x7f506814f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f5067fff640 1 -- 172.21.15.83:0/2924575013 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50681573a0 msgr2=0x7f506814f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f5067fff640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50681573a0 0x7f506814f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f5067fff640 1 -- 172.21.15.83:0/2924575013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50681546a0 msgr2=0x7f506814eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:59.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f5067fff640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50681546a0 0x7f506814eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f5067fff640 1 -- 172.21.15.83:0/2924575013 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50680ad6d0 con 0x7f5068074000 2024-09-19T07:47:59.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f5067fff640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068074000 0x7f50680c1bf0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f505800c420 tx=0x7f505800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:59.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f5064ff9640 1 -- 172.21.15.83:0/2924575013 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50580052d0 con 0x7f5068074000 2024-09-19T07:47:59.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f5064ff9640 1 -- 172.21.15.83:0/2924575013 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5058005470 con 0x7f5068074000 2024-09-19T07:47:59.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f5064ff9640 1 -- 172.21.15.83:0/2924575013 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5058007260 con 0x7f5068074000 2024-09-19T07:47:59.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f506ea36640 1 -- 172.21.15.83:0/2924575013 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50680ab500 con 0x7f5068074000 2024-09-19T07:47:59.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.402+0000 7f506ea36640 1 -- 172.21.15.83:0/2924575013 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50680ad3f0 con 0x7f5068074000 2024-09-19T07:47:59.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.406+0000 7f506ea36640 1 -- 172.21.15.83:0/2924575013 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50680705f0 con 0x7f5068074000 2024-09-19T07:47:59.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.406+0000 7f5064ff9640 1 -- 172.21.15.83:0/2924575013 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5058007400 con 0x7f5068074000 2024-09-19T07:47:59.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.406+0000 7f5064ff9640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50540678f0 0x7f5054069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.406+0000 7f5064ff9640 1 -- 172.21.15.83:0/2924575013 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7f5058086140 con 0x7f5068074000 2024-09-19T07:47:59.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.406+0000 7f50677fe640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50540678f0 0x7f5054069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:59.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.406+0000 7f50677fe640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50540678f0 0x7f5054069db0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f505c0af000 tx=0x7f505c0a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:59.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.414+0000 7f5064ff9640 1 -- 172.21.15.83:0/2924575013 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f50580555a0 con 0x7f5068074000 2024-09-19T07:47:59.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.702+0000 7f506ea36640 1 -- 172.21.15.83:0/2924575013 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5068157a40 con 0x7f5068074000 2024-09-19T07:47:59.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.702+0000 7f5064ff9640 1 -- 172.21.15.83:0/2924575013 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) ==== 95+0+5712 (secure 0 0 0) 0x7f50580598a0 con 0x7f5068074000 2024-09-19T07:47:59.701 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:47:59.701 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":191,"btime":"2024-09-19T07:47:58:342324+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:47:58.342320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:replay","state_seq":1,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:47:59.702 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 191 2024-09-19T07:47:59.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.710+0000 7f506ea36640 1 -- 172.21.15.83:0/2924575013 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50540678f0 msgr2=0x7f5054069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:59.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.710+0000 7f506ea36640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50540678f0 0x7f5054069db0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f505c0af000 tx=0x7f505c0a7000 comp rx=0 tx=0).stop 2024-09-19T07:47:59.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.710+0000 7f506ea36640 1 -- 172.21.15.83:0/2924575013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068074000 msgr2=0x7f50680c1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:59.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.710+0000 7f506ea36640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068074000 0x7f50680c1bf0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f505800c420 tx=0x7f505800c8e0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.710+0000 7f506ea36640 1 -- 172.21.15.83:0/2924575013 shutdown_connections 2024-09-19T07:47:59.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.710+0000 7f506ea36640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f50540678f0 0x7f5054069db0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.710+0000 7f506ea36640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50681573a0 0x7f506814f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.710+0000 7f506ea36640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50681546a0 0x7f506814eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.710+0000 7f506ea36640 1 --2- 172.21.15.83:0/2924575013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5068074000 0x7f50680c1bf0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.710+0000 7f506ea36640 1 -- 172.21.15.83:0/2924575013 >> 172.21.15.83:0/2924575013 conn(0x7f50680c5060 msgr2=0x7f506806e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:59.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.710+0000 7f506ea36640 1 -- 172.21.15.83:0/2924575013 shutdown_connections 2024-09-19T07:47:59.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.710+0000 7f506ea36640 1 -- 172.21.15.83:0/2924575013 wait complete. 2024-09-19T07:47:59.739 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-19T07:47:59.739 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:47:59.739 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:47:59.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13794eb640 1 Processor -- start 2024-09-19T07:47:59.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13794eb640 1 -- start start 2024-09-19T07:47:59.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13794eb640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1374154360 0x7f137406de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13794eb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f137414ddd0 con 0x7f13740c7510 2024-09-19T07:47:59.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13794eb640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f137406fdf0 con 0x7f137414d9f0 2024-09-19T07:47:59.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13794eb640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f13740c8830 con 0x7f1374154360 2024-09-19T07:47:59.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f136bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f13740c7510 0x7f13740c4010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46074/0 (socket says 172.21.15.83:46074) 2024-09-19T07:47:59.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f136bfff640 1 -- 172.21.15.83:0/2720929159 learned_addr learned my addr 172.21.15.83:0/2720929159 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:47:59.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13737fe640 1 --2- 172.21.15.83:0/2720929159 >> v2:172.21.15.184:3300/0 conn(0x7f1374154360 0x7f137406de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:59.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 443149664 0 0) 0x7f137414ddd0 con 0x7f13740c7510 2024-09-19T07:47:59.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1360003530 con 0x7f13740c7510 2024-09-19T07:47:59.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2433112317 0 0) 0x7f1360003530 con 0x7f13740c7510 2024-09-19T07:47:59.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 >> v2:172.21.15.184:3300/0 conn(0x7f1374154360 msgr2=0x7f137406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:59.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 --2- 172.21.15.83:0/2720929159 >> v2:172.21.15.184:3300/0 conn(0x7f1374154360 0x7f137406de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 >> v1:172.21.15.184:6789/0 conn(0x7f137414d9f0 legacy=0x7f13740c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:59.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13740c7f00 con 0x7f13740c7510 2024-09-19T07:47:59.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f135c001680 con 0x7f13740c7510 2024-09-19T07:47:59.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 >> v1:172.21.15.83:6789/0 conn(0x7f13740c7510 legacy=0x7f13740c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:59.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 --2- 172.21.15.83:0/2720929159 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13600039c0 0x7f1360003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 --2- 172.21.15.83:0/2720929159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1360004810 0x7f1360004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 --2- 172.21.15.83:0/2720929159 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13600051d0 0x7f136000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13740c8830 con 0x7f1360004810 2024-09-19T07:47:59.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f136000c170 con 0x7f13600039c0 2024-09-19T07:47:59.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f136000c370 con 0x7f13600051d0 2024-09-19T07:47:59.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f1372ffd640 1 --2- 172.21.15.83:0/2720929159 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13600039c0 0x7f1360003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:59.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f1372ffd640 1 -- 172.21.15.83:0/2720929159 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13600051d0 msgr2=0x7f136000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:59.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f1372ffd640 1 --2- 172.21.15.83:0/2720929159 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13600051d0 0x7f136000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f1372ffd640 1 -- 172.21.15.83:0/2720929159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1360004810 msgr2=0x7f1360004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:47:59.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f1372ffd640 1 --2- 172.21.15.83:0/2720929159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1360004810 0x7f1360004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f1372ffd640 1 -- 172.21.15.83:0/2720929159 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1360003530 con 0x7f13600039c0 2024-09-19T07:47:59.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.886+0000 7f13737fe640 1 -- 172.21.15.83:0/2720929159 reap_dead start 2024-09-19T07:47:59.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f1372ffd640 1 --2- 172.21.15.83:0/2720929159 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13600039c0 0x7f1360003de0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1364001780 tx=0x7f136401c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=413dcd107bce01ca server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:59.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13640013a0 con 0x7f13600039c0 2024-09-19T07:47:59.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13640052f0 con 0x7f13600039c0 2024-09-19T07:47:59.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f13727fc640 1 -- 172.21.15.83:0/2720929159 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364003840 con 0x7f13600039c0 2024-09-19T07:47:59.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f13794eb640 1 -- 172.21.15.83:0/2720929159 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13600039c0 msgr2=0x7f1360003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:59.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f13794eb640 1 --2- 172.21.15.83:0/2720929159 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13600039c0 0x7f1360003de0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1364001780 tx=0x7f136401c400 comp rx=0 tx=0).stop 2024-09-19T07:47:59.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f13794eb640 1 -- 172.21.15.83:0/2720929159 shutdown_connections 2024-09-19T07:47:59.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f13794eb640 1 --2- 172.21.15.83:0/2720929159 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13600039c0 0x7f1360003de0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f13794eb640 1 -- 172.21.15.83:0/2720929159 >> 172.21.15.83:0/2720929159 conn(0x7f13740c6110 msgr2=0x7f13740c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:47:59.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f13794eb640 1 -- 172.21.15.83:0/2720929159 shutdown_connections 2024-09-19T07:47:59.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f13794eb640 1 -- 172.21.15.83:0/2720929159 wait complete. 2024-09-19T07:47:59.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f13794eb640 1 Processor -- start 2024-09-19T07:47:59.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.894+0000 7f13794eb640 1 -- start start 2024-09-19T07:47:59.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13794eb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13600039c0 0x7f137413a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13794eb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f137413dc90 0x7f1374073230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13794eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137413b5e0 0x7f137413ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13794eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13740c43a0 con 0x7f137413b5e0 2024-09-19T07:47:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13794eb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13740c4220 con 0x7f13600039c0 2024-09-19T07:47:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13794eb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f13740c4520 con 0x7f137413dc90 2024-09-19T07:47:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13737fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137413b5e0 0x7f137413ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13737fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137413b5e0 0x7f137413ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46714/0 (socket says 172.21.15.83:46714) 2024-09-19T07:47:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13737fe640 1 -- 172.21.15.83:0/1085072546 learned_addr learned my addr 172.21.15.83:0/1085072546 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:47:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f136bfff640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f137413dc90 0x7f1374073230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13737fe640 1 -- 172.21.15.83:0/1085072546 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f137413dc90 msgr2=0x7f1374073230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13737fe640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f137413dc90 0x7f1374073230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f1372ffd640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13600039c0 0x7f137413a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:59.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13737fe640 1 -- 172.21.15.83:0/1085072546 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13600039c0 msgr2=0x7f137413a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:47:59.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13737fe640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13600039c0 0x7f137413a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:47:59.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13737fe640 1 -- 172.21.15.83:0/1085072546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137413b0c0 con 0x7f137413b5e0 2024-09-19T07:47:59.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f1372ffd640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13600039c0 0x7f137413a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:47:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13737fe640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137413b5e0 0x7f137413ba70 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f13741514b0 tx=0x7f136c02e4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f136b7fe640 1 -- 172.21.15.83:0/1085072546 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c02eaa0 con 0x7f137413b5e0 2024-09-19T07:47:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f136b7fe640 1 -- 172.21.15.83:0/1085072546 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f136c02ec60 con 0x7f137413b5e0 2024-09-19T07:47:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13794eb640 1 -- 172.21.15.83:0/1085072546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f137413e510 con 0x7f137413b5e0 2024-09-19T07:47:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13794eb640 1 -- 172.21.15.83:0/1085072546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f137413ead0 con 0x7f137413b5e0 2024-09-19T07:47:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f136b7fe640 1 -- 172.21.15.83:0/1085072546 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c0039a0 con 0x7f137413b5e0 2024-09-19T07:47:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f13794eb640 1 -- 172.21.15.83:0/1085072546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13580052a0 con 0x7f137413b5e0 2024-09-19T07:47:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.898+0000 7f136b7fe640 1 -- 172.21.15.83:0/1085072546 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f136c03e020 con 0x7f137413b5e0 2024-09-19T07:47:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.902+0000 7f136b7fe640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13540678f0 0x7f1354069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:47:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.902+0000 7f136b7fe640 1 -- 172.21.15.83:0/1085072546 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7f136c0a89f0 con 0x7f137413b5e0 2024-09-19T07:47:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.902+0000 7f1372ffd640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13540678f0 0x7f1354069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:47:59.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.902+0000 7f1372ffd640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13540678f0 0x7f1354069db0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1364001750 tx=0x7f13640011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:47:59.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:47:59.906+0000 7f136b7fe640 1 -- 172.21.15.83:0/1085072546 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f136c078470 con 0x7f137413b5e0 2024-09-19T07:48:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.162+0000 7f13794eb640 1 -- 172.21.15.83:0/1085072546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1358005590 con 0x7f137413b5e0 2024-09-19T07:48:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.162+0000 7f136b7fe640 1 -- 172.21.15.83:0/1085072546 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) ==== 95+0+5712 (secure 0 0 0) 0x7f136c077d70 con 0x7f137413b5e0 2024-09-19T07:48:00.161 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:00.161 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":191,"btime":"2024-09-19T07:47:58:342324+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:47:58.342320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:replay","state_seq":1,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:48:00.161 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 191 2024-09-19T07:48:00.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.162+0000 7f13794eb640 1 -- 172.21.15.83:0/1085072546 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13540678f0 msgr2=0x7f1354069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:00.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.162+0000 7f13794eb640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13540678f0 0x7f1354069db0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1364001750 tx=0x7f13640011a0 comp rx=0 tx=0).stop 2024-09-19T07:48:00.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.162+0000 7f13794eb640 1 -- 172.21.15.83:0/1085072546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137413b5e0 msgr2=0x7f137413ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:00.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.162+0000 7f13794eb640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137413b5e0 0x7f137413ba70 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f13741514b0 tx=0x7f136c02e4a0 comp rx=0 tx=0).stop 2024-09-19T07:48:00.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.166+0000 7f13794eb640 1 -- 172.21.15.83:0/1085072546 shutdown_connections 2024-09-19T07:48:00.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.166+0000 7f13794eb640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13540678f0 0x7f1354069db0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:00.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.166+0000 7f13794eb640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137413b5e0 0x7f137413ba70 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:00.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.166+0000 7f13794eb640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f137413dc90 0x7f1374073230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:00.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.166+0000 7f13794eb640 1 --2- 172.21.15.83:0/1085072546 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13600039c0 0x7f137413a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:00.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.166+0000 7f13794eb640 1 -- 172.21.15.83:0/1085072546 >> 172.21.15.83:0/1085072546 conn(0x7f13740c6110 msgr2=0x7f137414a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:00.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.166+0000 7f13794eb640 1 -- 172.21.15.83:0/1085072546 shutdown_connections 2024-09-19T07:48:00.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:00.166+0000 7f13794eb640 1 -- 172.21.15.83:0/1085072546 wait complete. 2024-09-19T07:48:00.393 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:47:58:342324+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 191, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 191, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6269": { "addr": "172.21.15.184:6833/2502231963", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2502231963, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2502231963, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6269, "incarnation": 191, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:47:58.342320+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6269 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 189, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8028": { "addr": "172.21.15.83:6837/1602509240", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 1602509240, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 1602509240, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8028, "incarnation": 186, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:47:06.716118+0000", "qdb_cluster": [ 8028 ], "qdb_leader": 8028, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8028 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:48:02.393 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:02.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab6bfe2640 1 Processor -- start 2024-09-19T07:48:02.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab6bfe2640 1 -- start start 2024-09-19T07:48:02.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab6bfe2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab64070140 0x7fab640c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:02.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab6bfe2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab64070830 con 0x7fab6406cef0 2024-09-19T07:48:02.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab6bfe2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab64157df0 con 0x7fab641579a0 2024-09-19T07:48:02.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab6bfe2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fab64153cf0 con 0x7fab64070140 2024-09-19T07:48:02.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab6957d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab64070140 0x7fab640c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:02.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab69d7e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fab6406cef0 0x7fab64154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56694/0 (socket says 172.21.15.83:56694) 2024-09-19T07:48:02.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab6957d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab64070140 0x7fab640c2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36304/0 (socket says 172.21.15.83:36304) 2024-09-19T07:48:02.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab69d7e640 1 -- 172.21.15.83:0/777471408 learned_addr learned my addr 172.21.15.83:0/777471408 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:02.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab6957d640 1 -- 172.21.15.83:0/777471408 >> v1:172.21.15.184:6789/0 conn(0x7fab641579a0 legacy=0x7fab64150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:02.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab6957d640 1 -- 172.21.15.83:0/777471408 >> v1:172.21.15.83:6789/0 conn(0x7fab6406cef0 legacy=0x7fab64154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:02.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.541+0000 7fab6957d640 1 -- 172.21.15.83:0/777471408 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab641539a0 con 0x7fab64070140 2024-09-19T07:48:02.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.545+0000 7fab6957d640 1 --2- 172.21.15.83:0/777471408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab64070140 0x7fab640c2f90 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fab540097c0 tx=0x7fab5401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56fb5bce44a49503 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:02.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.545+0000 7fab68d7c640 1 -- 172.21.15.83:0/777471408 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab540b0070 con 0x7fab64070140 2024-09-19T07:48:02.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.545+0000 7fab68d7c640 1 -- 172.21.15.83:0/777471408 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab540ab600 con 0x7fab64070140 2024-09-19T07:48:02.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.545+0000 7fab68d7c640 1 -- 172.21.15.83:0/777471408 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab540ab9d0 con 0x7fab64070140 2024-09-19T07:48:02.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.545+0000 7fab6bfe2640 1 -- 172.21.15.83:0/777471408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab64070140 msgr2=0x7fab640c2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:02.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.545+0000 7fab6bfe2640 1 --2- 172.21.15.83:0/777471408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab64070140 0x7fab640c2f90 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fab540097c0 tx=0x7fab5401c380 comp rx=0 tx=0).stop 2024-09-19T07:48:02.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.545+0000 7fab6bfe2640 1 -- 172.21.15.83:0/777471408 shutdown_connections 2024-09-19T07:48:02.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.545+0000 7fab6bfe2640 1 --2- 172.21.15.83:0/777471408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab64070140 0x7fab640c2f90 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:02.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.545+0000 7fab6bfe2640 1 -- 172.21.15.83:0/777471408 >> 172.21.15.83:0/777471408 conn(0x7fab640c5090 msgr2=0x7fab640c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:02.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.549+0000 7fab6bfe2640 1 -- 172.21.15.83:0/777471408 shutdown_connections 2024-09-19T07:48:02.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.549+0000 7fab6bfe2640 1 -- 172.21.15.83:0/777471408 wait complete. 2024-09-19T07:48:02.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6bfe2640 1 Processor -- start 2024-09-19T07:48:02.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6bfe2640 1 -- start start 2024-09-19T07:48:02.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6bfe2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab6406cef0 0x7fab6414c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:02.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6bfe2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab64070140 0x7fab64073740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:02.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6bfe2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab641579a0 0x7fab64154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:02.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6bfe2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fab641512b0 con 0x7fab64070140 2024-09-19T07:48:02.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6bfe2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab6406f8a0 con 0x7fab6406cef0 2024-09-19T07:48:02.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6bfe2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fab6414d9a0 con 0x7fab641579a0 2024-09-19T07:48:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6957d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab64070140 0x7fab64073740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab69d7e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab6406cef0 0x7fab6414c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6957d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab64070140 0x7fab64073740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36314/0 (socket says 172.21.15.83:36314) 2024-09-19T07:48:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6957d640 1 -- 172.21.15.83:0/918280320 learned_addr learned my addr 172.21.15.83:0/918280320 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6957d640 1 -- 172.21.15.83:0/918280320 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab641579a0 msgr2=0x7fab64154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6957d640 1 --2- 172.21.15.83:0/918280320 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab641579a0 0x7fab64154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6957d640 1 -- 172.21.15.83:0/918280320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab6406cef0 msgr2=0x7fab6414c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6957d640 1 --2- 172.21.15.83:0/918280320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab6406cef0 0x7fab6414c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6957d640 1 -- 172.21.15.83:0/918280320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab6414dbf0 con 0x7fab64070140 2024-09-19T07:48:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.553+0000 7fab6957d640 1 --2- 172.21.15.83:0/918280320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab64070140 0x7fab64073740 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fab54018f90 tx=0x7fab54019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:02.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.557+0000 7fab5a7fc640 1 -- 172.21.15.83:0/918280320 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab540b0040 con 0x7fab64070140 2024-09-19T07:48:02.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.557+0000 7fab5a7fc640 1 -- 172.21.15.83:0/918280320 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab540b9070 con 0x7fab64070140 2024-09-19T07:48:02.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.557+0000 7fab6bfe2640 1 -- 172.21.15.83:0/918280320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab6414c850 con 0x7fab64070140 2024-09-19T07:48:02.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.557+0000 7fab5a7fc640 1 -- 172.21.15.83:0/918280320 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab540aa9f0 con 0x7fab64070140 2024-09-19T07:48:02.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.557+0000 7fab6bfe2640 1 -- 172.21.15.83:0/918280320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab6414b140 con 0x7fab64070140 2024-09-19T07:48:02.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.557+0000 7fab6bfe2640 1 -- 172.21.15.83:0/918280320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab4c0052a0 con 0x7fab64070140 2024-09-19T07:48:02.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.557+0000 7fab5a7fc640 1 -- 172.21.15.83:0/918280320 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fab540aab90 con 0x7fab64070140 2024-09-19T07:48:02.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.557+0000 7fab5a7fc640 1 --2- 172.21.15.83:0/918280320 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab48067940 0x7fab48069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:02.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.557+0000 7fab5a7fc640 1 -- 172.21.15.83:0/918280320 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7fab54120ce0 con 0x7fab64070140 2024-09-19T07:48:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.557+0000 7fab69d7e640 1 --2- 172.21.15.83:0/918280320 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab48067940 0x7fab48069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.561+0000 7fab69d7e640 1 --2- 172.21.15.83:0/918280320 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab48067940 0x7fab48069e00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fab60005890 tx=0x7fab60009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:02.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.565+0000 7fab5a7fc640 1 -- 172.21.15.83:0/918280320 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fab54016090 con 0x7fab64070140 2024-09-19T07:48:02.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.841+0000 7fab6bfe2640 1 -- 172.21.15.83:0/918280320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab4c005860 con 0x7fab64070140 2024-09-19T07:48:02.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.841+0000 7fab5a7fc640 1 -- 172.21.15.83:0/918280320 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) ==== 95+0+5720 (secure 0 0 0) 0x7fab540f0870 con 0x7fab64070140 2024-09-19T07:48:02.842 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:02.842 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":194,"btime":"2024-09-19T07:48:02:799460+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:48:02.843 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 194 2024-09-19T07:48:02.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.845+0000 7fab6bfe2640 1 -- 172.21.15.83:0/918280320 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab48067940 msgr2=0x7fab48069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.845+0000 7fab6bfe2640 1 --2- 172.21.15.83:0/918280320 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab48067940 0x7fab48069e00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fab60005890 tx=0x7fab60009040 comp rx=0 tx=0).stop 2024-09-19T07:48:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.845+0000 7fab6bfe2640 1 -- 172.21.15.83:0/918280320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab64070140 msgr2=0x7fab64073740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.845+0000 7fab6bfe2640 1 --2- 172.21.15.83:0/918280320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab64070140 0x7fab64073740 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fab54018f90 tx=0x7fab54019120 comp rx=0 tx=0).stop 2024-09-19T07:48:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.845+0000 7fab6bfe2640 1 -- 172.21.15.83:0/918280320 shutdown_connections 2024-09-19T07:48:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.845+0000 7fab6bfe2640 1 --2- 172.21.15.83:0/918280320 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab48067940 0x7fab48069e00 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.845+0000 7fab6bfe2640 1 --2- 172.21.15.83:0/918280320 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab641579a0 0x7fab64154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.845+0000 7fab6bfe2640 1 --2- 172.21.15.83:0/918280320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab64070140 0x7fab64073740 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.845+0000 7fab6bfe2640 1 --2- 172.21.15.83:0/918280320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab6406cef0 0x7fab6414c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.845+0000 7fab6bfe2640 1 -- 172.21.15.83:0/918280320 >> 172.21.15.83:0/918280320 conn(0x7fab640c5090 msgr2=0x7fab64074f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.845+0000 7fab6bfe2640 1 -- 172.21.15.83:0/918280320 shutdown_connections 2024-09-19T07:48:02.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:02.845+0000 7fab6bfe2640 1 -- 172.21.15.83:0/918280320 wait complete. 2024-09-19T07:48:02.862 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=7968 2024-09-19T07:48:02.862 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before reviving mds.a 2024-09-19T07:48:07.804 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:07.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.945+0000 7f2a13372640 1 Processor -- start 2024-09-19T07:48:07.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.945+0000 7f2a13372640 1 -- start start 2024-09-19T07:48:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a13372640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2a0c0abda0 0x7f2a0c0ac180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a13372640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2a0c0ad150 0x7f2a0c0c2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a13372640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a0c06d630 con 0x7f2a0c0c2d80 2024-09-19T07:48:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a13372640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f2a0c154ae0 con 0x7f2a0c0ad150 2024-09-19T07:48:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a13372640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2a0c0acc00 con 0x7f2a0c0abda0 2024-09-19T07:48:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a1190f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2a0c0c2d80 0x7f2a0c150c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56702/0 (socket says 172.21.15.83:56702) 2024-09-19T07:48:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a1190f640 1 -- 172.21.15.83:0/3711580521 learned_addr learned my addr 172.21.15.83:0/3711580521 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:07.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a1090d640 1 --2- 172.21.15.83:0/3711580521 >> v2:172.21.15.184:3300/0 conn(0x7f2a0c0ad150 0x7f2a0c0c2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:07.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a1110e640 1 --2- 172.21.15.83:0/3711580521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2a0c0abda0 0x7f2a0c0ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3895562243 0 0) 0x7f2a0c06d630 con 0x7f2a0c0c2d80 2024-09-19T07:48:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29f8003530 con 0x7f2a0c0c2d80 2024-09-19T07:48:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1061714824 0 0) 0x7f29f8003530 con 0x7f2a0c0c2d80 2024-09-19T07:48:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2a0c0abda0 msgr2=0x7f2a0c0ac180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 --2- 172.21.15.83:0/3711580521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2a0c0abda0 0x7f2a0c0ac180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 >> v2:172.21.15.184:3300/0 conn(0x7f2a0c0ad150 msgr2=0x7f2a0c0c2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 --2- 172.21.15.83:0/3711580521 >> v2:172.21.15.184:3300/0 conn(0x7f2a0c0ad150 0x7f2a0c0c2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0c06e450 con 0x7f2a0c0c2d80 2024-09-19T07:48:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f2a04001a00 con 0x7f2a0c0c2d80 2024-09-19T07:48:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 >> v1:172.21.15.83:6789/0 conn(0x7f2a0c0c2d80 legacy=0x7f2a0c150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 --2- 172.21.15.83:0/3711580521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29f80039c0 0x7f29f8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 --2- 172.21.15.83:0/3711580521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29f80043d0 0x7f29f80048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 --2- 172.21.15.83:0/3711580521 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29f8005070 0x7f29f800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c154ae0 con 0x7f29f80043d0 2024-09-19T07:48:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c0acc00 con 0x7f29f8005070 2024-09-19T07:48:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f29f8010420 con 0x7f29f80039c0 2024-09-19T07:48:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a1090d640 1 --2- 172.21.15.83:0/3711580521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29f80043d0 0x7f29f80048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a1090d640 1 -- 172.21.15.83:0/3711580521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29f80039c0 msgr2=0x7f29f8003e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:48:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a1090d640 1 --2- 172.21.15.83:0/3711580521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29f80039c0 0x7f29f8003e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a1090d640 1 -- 172.21.15.83:0/3711580521 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29f8005070 msgr2=0x7f29f800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a1090d640 1 --2- 172.21.15.83:0/3711580521 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29f8005070 0x7f29f800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.949+0000 7f2a1090d640 1 -- 172.21.15.83:0/3711580521 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f8003530 con 0x7f29f80043d0 2024-09-19T07:48:07.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.953+0000 7f2a1090d640 1 --2- 172.21.15.83:0/3711580521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29f80043d0 0x7f29f80048b0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009ae0 tx=0x7f29fc01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=776cbee0ab1e12c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:07.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.953+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0096b0 con 0x7f29f80043d0 2024-09-19T07:48:07.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.953+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29fc0a8430 con 0x7f29f80043d0 2024-09-19T07:48:07.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.953+0000 7f2a03fff640 1 -- 172.21.15.83:0/3711580521 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0a8660 con 0x7f29f80043d0 2024-09-19T07:48:07.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 -- 172.21.15.83:0/3711580521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29f80043d0 msgr2=0x7f29f80048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 --2- 172.21.15.83:0/3711580521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29f80043d0 0x7f29f80048b0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009ae0 tx=0x7f29fc01ca10 comp rx=0 tx=0).stop 2024-09-19T07:48:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a1190f640 1 -- 172.21.15.83:0/3711580521 reap_dead start 2024-09-19T07:48:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 -- 172.21.15.83:0/3711580521 shutdown_connections 2024-09-19T07:48:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 -- 172.21.15.83:0/3711580521 >> 172.21.15.83:0/3711580521 conn(0x7f2a0c0c7e40 msgr2=0x7f2a0c0c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 -- 172.21.15.83:0/3711580521 shutdown_connections 2024-09-19T07:48:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 -- 172.21.15.83:0/3711580521 wait complete. 2024-09-19T07:48:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 Processor -- start 2024-09-19T07:48:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 -- start start 2024-09-19T07:48:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2a0c0c3930 0x7f2a0c156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a0c147570 0x7f2a0c1479f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a0c0c4810 0x7f2a0c0c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c0c31b0 con 0x7f2a0c0c4810 2024-09-19T07:48:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c06db10 con 0x7f2a0c147570 2024-09-19T07:48:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a13372640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2a0c0ac180 con 0x7f2a0c0c3930 2024-09-19T07:48:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a1190f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a0c0c4810 0x7f2a0c0c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a1190f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a0c0c4810 0x7f2a0c0c4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36328/0 (socket says 172.21.15.83:36328) 2024-09-19T07:48:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a1190f640 1 -- 172.21.15.83:0/2482917005 learned_addr learned my addr 172.21.15.83:0/2482917005 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a1110e640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2a0c0c3930 0x7f2a0c156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.957+0000 7f2a1090d640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a0c147570 0x7f2a0c1479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a1190f640 1 -- 172.21.15.83:0/2482917005 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2a0c0c3930 msgr2=0x7f2a0c156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a1190f640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2a0c0c3930 0x7f2a0c156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a1190f640 1 -- 172.21.15.83:0/2482917005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a0c147570 msgr2=0x7f2a0c1479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a1190f640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a0c147570 0x7f2a0c1479f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a1190f640 1 -- 172.21.15.83:0/2482917005 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0c1ea390 con 0x7f2a0c0c4810 2024-09-19T07:48:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a1090d640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a0c147570 0x7f2a0c1479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:48:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a1190f640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a0c0c4810 0x7f2a0c0c4ca0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c0aaff0 tx=0x7f2a0400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a01ffb640 1 -- 172.21.15.83:0/2482917005 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0400a670 con 0x7f2a0c0c4810 2024-09-19T07:48:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a13372640 1 -- 172.21.15.83:0/2482917005 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a0c1ea670 con 0x7f2a0c0c4810 2024-09-19T07:48:07.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a01ffb640 1 -- 172.21.15.83:0/2482917005 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a0400a810 con 0x7f2a0c0c4810 2024-09-19T07:48:07.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a01ffb640 1 -- 172.21.15.83:0/2482917005 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0400abe0 con 0x7f2a0c0c4810 2024-09-19T07:48:07.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a13372640 1 -- 172.21.15.83:0/2482917005 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a0c155670 con 0x7f2a0c0c4810 2024-09-19T07:48:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a13372640 1 -- 172.21.15.83:0/2482917005 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a0c069030 con 0x7f2a0c0c4810 2024-09-19T07:48:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a01ffb640 1 -- 172.21.15.83:0/2482917005 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2a04011070 con 0x7f2a0c0c4810 2024-09-19T07:48:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a01ffb640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29f00678c0 0x7f29f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a01ffb640 1 -- 172.21.15.83:0/2482917005 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7f2a04086950 con 0x7f2a0c0c4810 2024-09-19T07:48:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a1110e640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29f00678c0 0x7f29f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.961+0000 7f2a1110e640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29f00678c0 0x7f29f0069d80 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c150720 tx=0x7f2a08008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:07.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:07.969+0000 7f2a01ffb640 1 -- 172.21.15.83:0/2482917005 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2a04056460 con 0x7f2a0c0c4810 2024-09-19T07:48:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.245+0000 7f2a13372640 1 -- 172.21.15.83:0/2482917005 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a0c1529a0 con 0x7f2a0c0c4810 2024-09-19T07:48:08.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.245+0000 7f2a01ffb640 1 -- 172.21.15.83:0/2482917005 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) ==== 95+0+5720 (secure 0 0 0) 0x7f2a04055d60 con 0x7f2a0c0c4810 2024-09-19T07:48:08.245 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:08.245 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":194,"btime":"2024-09-19T07:48:02:799460+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:48:08.245 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 194 2024-09-19T07:48:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.249+0000 7f2a13372640 1 -- 172.21.15.83:0/2482917005 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29f00678c0 msgr2=0x7f29f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.249+0000 7f2a13372640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29f00678c0 0x7f29f0069d80 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c150720 tx=0x7f2a08008040 comp rx=0 tx=0).stop 2024-09-19T07:48:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.249+0000 7f2a13372640 1 -- 172.21.15.83:0/2482917005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a0c0c4810 msgr2=0x7f2a0c0c4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.249+0000 7f2a13372640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a0c0c4810 0x7f2a0c0c4ca0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c0aaff0 tx=0x7f2a0400cf00 comp rx=0 tx=0).stop 2024-09-19T07:48:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.249+0000 7f2a13372640 1 -- 172.21.15.83:0/2482917005 shutdown_connections 2024-09-19T07:48:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.249+0000 7f2a13372640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29f00678c0 0x7f29f0069d80 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.249+0000 7f2a13372640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a0c0c4810 0x7f2a0c0c4ca0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.249+0000 7f2a13372640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a0c147570 0x7f2a0c1479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.249+0000 7f2a13372640 1 --2- 172.21.15.83:0/2482917005 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2a0c0c3930 0x7f2a0c156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.249+0000 7f2a13372640 1 -- 172.21.15.83:0/2482917005 >> 172.21.15.83:0/2482917005 conn(0x7f2a0c0c7e40 msgr2=0x7f2a0c151910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:08.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.249+0000 7f2a13372640 1 -- 172.21.15.83:0/2482917005 shutdown_connections 2024-09-19T07:48:08.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.249+0000 7f2a13372640 1 -- 172.21.15.83:0/2482917005 wait complete. 2024-09-19T07:48:08.269 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-19T07:48:08.270 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:48:08.270 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:48:08.283 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:48:08.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:08.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a88d8640 1 Processor -- start 2024-09-19T07:48:08.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a88d8640 1 -- start start 2024-09-19T07:48:08.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a88d8640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f49a414a500 0x7f49a40c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:08.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a88d8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f49a40c4550 0x7f49a4157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:08.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a88d8640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49a4151750 con 0x7f49a4154540 2024-09-19T07:48:08.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a88d8640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f49a40c7e90 con 0x7f49a40c4550 2024-09-19T07:48:08.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a88d8640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f49a4073110 con 0x7f49a414a500 2024-09-19T07:48:08.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a2d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f49a40c4550 0x7f49a4157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:08.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a2d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f49a40c4550 0x7f49a4157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36332/0 (socket says 172.21.15.83:36332) 2024-09-19T07:48:08.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a2d9d640 1 -- 172.21.15.83:0/4164315961 learned_addr learned my addr 172.21.15.83:0/4164315961 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:08.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a2d9d640 1 -- 172.21.15.83:0/4164315961 >> v2:172.21.15.184:3300/0 conn(0x7f49a414a500 msgr2=0x7f49a40c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:08.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a2d9d640 1 --2- 172.21.15.83:0/4164315961 >> v2:172.21.15.184:3300/0 conn(0x7f49a414a500 0x7f49a40c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a2d9d640 1 -- 172.21.15.83:0/4164315961 >> v1:172.21.15.184:6789/0 conn(0x7f49a4154540 legacy=0x7f49a4157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:08.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.469+0000 7f49a2d9d640 1 -- 172.21.15.83:0/4164315961 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a40733f0 con 0x7f49a40c4550 2024-09-19T07:48:08.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.473+0000 7f49a2d9d640 1 --2- 172.21.15.83:0/4164315961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a40c4550 0x7f49a4157ba0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f49a4072c20 tx=0x7f499c01c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8350a8ad3ba39b3f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:08.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.473+0000 7f49a159a640 1 -- 172.21.15.83:0/4164315961 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c0a5620 con 0x7f49a40c4550 2024-09-19T07:48:08.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.473+0000 7f49a159a640 1 -- 172.21.15.83:0/4164315961 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f499c0a57c0 con 0x7f49a40c4550 2024-09-19T07:48:08.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.473+0000 7f49a159a640 1 -- 172.21.15.83:0/4164315961 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c0a5b90 con 0x7f49a40c4550 2024-09-19T07:48:08.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.473+0000 7f49a88d8640 1 -- 172.21.15.83:0/4164315961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a40c4550 msgr2=0x7f49a4157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:08.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.473+0000 7f49a88d8640 1 --2- 172.21.15.83:0/4164315961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a40c4550 0x7f49a4157ba0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f49a4072c20 tx=0x7f499c01c7a0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.473+0000 7f49a88d8640 1 -- 172.21.15.83:0/4164315961 shutdown_connections 2024-09-19T07:48:08.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.473+0000 7f49a88d8640 1 --2- 172.21.15.83:0/4164315961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a40c4550 0x7f49a4157ba0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.473+0000 7f49a88d8640 1 --2- 172.21.15.83:0/4164315961 >> v2:172.21.15.184:3300/0 conn(0x7f49a414a500 0x7f49a40c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.473+0000 7f49a88d8640 1 -- 172.21.15.83:0/4164315961 >> 172.21.15.83:0/4164315961 conn(0x7f49a40c9610 msgr2=0x7f49a40c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:08.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a88d8640 1 -- 172.21.15.83:0/4164315961 shutdown_connections 2024-09-19T07:48:08.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a88d8640 1 -- 172.21.15.83:0/4164315961 wait complete. 2024-09-19T07:48:08.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a88d8640 1 Processor -- start 2024-09-19T07:48:08.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a88d8640 1 -- start start 2024-09-19T07:48:08.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a88d8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a40c4550 0x7f49a4147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:08.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a88d8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a414a500 0x7f49a414cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:08.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a88d8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a4154540 0x7f49a41f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:08.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a88d8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f49a40c7850 con 0x7f49a4154540 2024-09-19T07:48:08.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a88d8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49a40c76d0 con 0x7f49a414a500 2024-09-19T07:48:08.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a88d8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f49a4160c60 con 0x7f49a40c4550 2024-09-19T07:48:08.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a4154540 0x7f49a41f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:08.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a4154540 0x7f49a41f2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36342/0 (socket says 172.21.15.83:36342) 2024-09-19T07:48:08.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a2d9d640 1 -- 172.21.15.83:0/1020988323 learned_addr learned my addr 172.21.15.83:0/1020988323 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:08.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a259c640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a40c4550 0x7f49a4147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:08.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a1d9b640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a414a500 0x7f49a414cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:08.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a2d9d640 1 -- 172.21.15.83:0/1020988323 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a40c4550 msgr2=0x7f49a4147160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:08.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a2d9d640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a40c4550 0x7f49a4147160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a2d9d640 1 -- 172.21.15.83:0/1020988323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a414a500 msgr2=0x7f49a414cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:08.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a2d9d640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a414a500 0x7f49a414cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a2d9d640 1 -- 172.21.15.83:0/1020988323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a41f27d0 con 0x7f49a4154540 2024-09-19T07:48:08.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a2d9d640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a4154540 0x7f49a41f2cd0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f499c01a3d0 tx=0x7f499c019540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:08.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f4992ffd640 1 -- 172.21.15.83:0/1020988323 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c0bc020 con 0x7f49a4154540 2024-09-19T07:48:08.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a88d8640 1 -- 172.21.15.83:0/1020988323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a41f61d0 con 0x7f49a4154540 2024-09-19T07:48:08.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f4992ffd640 1 -- 172.21.15.83:0/1020988323 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f499c0b8880 con 0x7f49a4154540 2024-09-19T07:48:08.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f4992ffd640 1 -- 172.21.15.83:0/1020988323 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c009510 con 0x7f49a4154540 2024-09-19T07:48:08.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.477+0000 7f49a88d8640 1 -- 172.21.15.83:0/1020988323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a41f6790 con 0x7f49a4154540 2024-09-19T07:48:08.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.481+0000 7f4990ff9640 1 -- 172.21.15.83:0/1020988323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f497c0052a0 con 0x7f49a4154540 2024-09-19T07:48:08.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.481+0000 7f4992ffd640 1 -- 172.21.15.83:0/1020988323 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f499c0bf020 con 0x7f49a4154540 2024-09-19T07:48:08.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.481+0000 7f4992ffd640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f49840678c0 0x7f4984069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:08.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.481+0000 7f4992ffd640 1 -- 172.21.15.83:0/1020988323 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7f499c121cf0 con 0x7f49a4154540 2024-09-19T07:48:08.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.481+0000 7f49a259c640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f49840678c0 0x7f4984069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:08.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.481+0000 7f49a259c640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f49840678c0 0x7f4984069d80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4994006fd0 tx=0x7f4994008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:08.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.485+0000 7f4992ffd640 1 -- 172.21.15.83:0/1020988323 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f499c126070 con 0x7f49a4154540 2024-09-19T07:48:08.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.773+0000 7f4990ff9640 1 -- 172.21.15.83:0/1020988323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f497c005590 con 0x7f49a4154540 2024-09-19T07:48:08.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.773+0000 7f4992ffd640 1 -- 172.21.15.83:0/1020988323 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) ==== 95+0+5720 (secure 0 0 0) 0x7f49a4151750 con 0x7f49a4154540 2024-09-19T07:48:08.773 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:08.773 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":194,"btime":"2024-09-19T07:48:02:799460+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:48:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 194 2024-09-19T07:48:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.773+0000 7f4990ff9640 1 -- 172.21.15.83:0/1020988323 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f49840678c0 msgr2=0x7f4984069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.773+0000 7f4990ff9640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f49840678c0 0x7f4984069d80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4994006fd0 tx=0x7f4994008040 comp rx=0 tx=0).stop 2024-09-19T07:48:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.773+0000 7f4990ff9640 1 -- 172.21.15.83:0/1020988323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a4154540 msgr2=0x7f49a41f2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.773+0000 7f4990ff9640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a4154540 0x7f49a41f2cd0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f499c01a3d0 tx=0x7f499c019540 comp rx=0 tx=0).stop 2024-09-19T07:48:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.773+0000 7f4990ff9640 1 -- 172.21.15.83:0/1020988323 shutdown_connections 2024-09-19T07:48:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.777+0000 7f4990ff9640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f49840678c0 0x7f4984069d80 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.777+0000 7f4990ff9640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49a4154540 0x7f49a41f2cd0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.777+0000 7f4990ff9640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a414a500 0x7f49a414cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.777+0000 7f4990ff9640 1 --2- 172.21.15.83:0/1020988323 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a40c4550 0x7f49a4147160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.777+0000 7f4990ff9640 1 -- 172.21.15.83:0/1020988323 >> 172.21.15.83:0/1020988323 conn(0x7f49a40c9610 msgr2=0x7f49a40c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.781+0000 7f4990ff9640 1 -- 172.21.15.83:0/1020988323 shutdown_connections 2024-09-19T07:48:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:08.781+0000 7f4990ff9640 1 -- 172.21.15.83:0/1020988323 wait complete. 2024-09-19T07:48:08.796 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:48:10.797 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:10.863 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-19T07:48:10.863 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-19T07:48:10.863 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-19T07:48:10.865 INFO:tasks.ceph.mds.a:Started 2024-09-19T07:48:10.866 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:10.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e6590a640 1 Processor -- start 2024-09-19T07:48:10.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e6590a640 1 -- start start 2024-09-19T07:48:10.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e6590a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9e600c7d70 0x7f9e60157c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e6590a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e601547b0 0x7f9e600c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e6590a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e60073690 con 0x7f9e60072d00 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e6590a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f9e60157160 con 0x7f9e601547b0 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e6590a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f9e60157440 con 0x7f9e600c7d70 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e5f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9e60072d00 0x7f9e6006cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46846/0 (socket says 172.21.15.83:46846) 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e5e7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e601547b0 0x7f9e600c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e5f7fe640 1 -- 172.21.15.83:0/4110337911 learned_addr learned my addr 172.21.15.83:0/4110337911 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e5effd640 1 --2- 172.21.15.83:0/4110337911 >> v2:172.21.15.184:3300/0 conn(0x7f9e600c7d70 0x7f9e60157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e5e7fc640 1 -- 172.21.15.83:0/4110337911 >> v2:172.21.15.184:3300/0 conn(0x7f9e600c7d70 msgr2=0x7f9e60157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e5e7fc640 1 --2- 172.21.15.83:0/4110337911 >> v2:172.21.15.184:3300/0 conn(0x7f9e600c7d70 0x7f9e60157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e5e7fc640 1 -- 172.21.15.83:0/4110337911 >> v1:172.21.15.83:6789/0 conn(0x7f9e60072d00 legacy=0x7f9e6006cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e5dffb640 1 -- 172.21.15.83:0/4110337911 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4157628101 0 0) 0x7f9e60073690 con 0x7f9e60072d00 2024-09-19T07:48:10.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.965+0000 7f9e5e7fc640 1 -- 172.21.15.83:0/4110337911 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e60157720 con 0x7f9e601547b0 2024-09-19T07:48:10.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.969+0000 7f9e5e7fc640 1 --2- 172.21.15.83:0/4110337911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e601547b0 0x7f9e600c6de0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f9e60076000 tx=0x7f9e5401c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2476ef4169bce2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:10.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.969+0000 7f9e5dffb640 1 -- 172.21.15.83:0/4110337911 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e540b0070 con 0x7f9e601547b0 2024-09-19T07:48:10.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.969+0000 7f9e5dffb640 1 -- 172.21.15.83:0/4110337911 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e540ab600 con 0x7f9e601547b0 2024-09-19T07:48:10.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.969+0000 7f9e5dffb640 1 -- 172.21.15.83:0/4110337911 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e540ab9d0 con 0x7f9e601547b0 2024-09-19T07:48:10.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.973+0000 7f9e6590a640 1 -- 172.21.15.83:0/4110337911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e601547b0 msgr2=0x7f9e600c6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:10.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.973+0000 7f9e6590a640 1 --2- 172.21.15.83:0/4110337911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e601547b0 0x7f9e600c6de0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f9e60076000 tx=0x7f9e5401c330 comp rx=0 tx=0).stop 2024-09-19T07:48:10.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.981+0000 7f9e6590a640 1 -- 172.21.15.83:0/4110337911 shutdown_connections 2024-09-19T07:48:10.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.981+0000 7f9e6590a640 1 --2- 172.21.15.83:0/4110337911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e601547b0 0x7f9e600c6de0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:10.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.981+0000 7f9e6590a640 1 --2- 172.21.15.83:0/4110337911 >> v2:172.21.15.184:3300/0 conn(0x7f9e600c7d70 0x7f9e60157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:10.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.981+0000 7f9e6590a640 1 -- 172.21.15.83:0/4110337911 >> 172.21.15.83:0/4110337911 conn(0x7f9e600c5060 msgr2=0x7f9e600c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.989+0000 7f9e6590a640 1 -- 172.21.15.83:0/4110337911 shutdown_connections 2024-09-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.989+0000 7f9e6590a640 1 -- 172.21.15.83:0/4110337911 wait complete. 2024-09-19T07:48:10.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.989+0000 7f9e6590a640 1 Processor -- start 2024-09-19T07:48:10.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.989+0000 7f9e6590a640 1 -- start start 2024-09-19T07:48:10.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e6590a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e60072d00 0x7f9e600beb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:10.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e6590a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e600c7d70 0x7f9e600c0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:10.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e5effd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e60072d00 0x7f9e600beb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:10.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e5effd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e60072d00 0x7f9e600beb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:44992/0 (socket says 172.21.15.83:44992) 2024-09-19T07:48:10.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e5e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e600c7d70 0x7f9e600c0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:10.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e5e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e600c7d70 0x7f9e600c0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40096/0 (socket says 172.21.15.83:40096) 2024-09-19T07:48:10.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e5effd640 1 -- 172.21.15.83:0/654550532 learned_addr learned my addr 172.21.15.83:0/654550532 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:10.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e6590a640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e601547b0 0x7f9e601efcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:10.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e600c74a0 con 0x7f9e600c7d70 2024-09-19T07:48:10.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9e600c7320 con 0x7f9e601547b0 2024-09-19T07:48:10.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9e60070230 con 0x7f9e60072d00 2024-09-19T07:48:10.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e5effd640 1 -- 172.21.15.83:0/654550532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e601547b0 msgr2=0x7f9e601efcf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T07:48:10.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e5effd640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e601547b0 0x7f9e601efcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:10.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e5effd640 1 -- 172.21.15.83:0/654550532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e600c7d70 msgr2=0x7f9e600c0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:10.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e5effd640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e600c7d70 0x7f9e600c0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:10.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e5effd640 1 -- 172.21.15.83:0/654550532 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e600bfe30 con 0x7f9e60072d00 2024-09-19T07:48:10.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e5effd640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e60072d00 0x7f9e600beb60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000e960 tx=0x7f9e5000ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:10.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e47fff640 1 -- 172.21.15.83:0/654550532 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50017070 con 0x7f9e60072d00 2024-09-19T07:48:10.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e47fff640 1 -- 172.21.15.83:0/654550532 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e5000ca00 con 0x7f9e60072d00 2024-09-19T07:48:10.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e47fff640 1 -- 172.21.15.83:0/654550532 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5000cdb0 con 0x7f9e60072d00 2024-09-19T07:48:10.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e600be0c0 con 0x7f9e60072d00 2024-09-19T07:48:10.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.993+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e600be360 con 0x7f9e60072d00 2024-09-19T07:48:10.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.997+0000 7f9e47fff640 1 -- 172.21.15.83:0/654550532 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9e5001a070 con 0x7f9e60072d00 2024-09-19T07:48:10.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.997+0000 7f9e47fff640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e4c067990 0x7f9e4c069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:10.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:10.997+0000 7f9e47fff640 1 -- 172.21.15.83:0/654550532 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7f9e50086cd0 con 0x7f9e60072d00 2024-09-19T07:48:10.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.001+0000 7f9e5e7fc640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e4c067990 0x7f9e4c069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:11.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.001+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e60069030 con 0x7f9e60072d00 2024-09-19T07:48:11.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.005+0000 7f9e5e7fc640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e4c067990 0x7f9e4c069e50 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9e540ad000 tx=0x7f9e540a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:11.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.009+0000 7f9e47fff640 1 -- 172.21.15.83:0/654550532 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9e500561b0 con 0x7f9e60072d00 2024-09-19T07:48:11.024 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:48:11.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9df621c640 1 Processor -- start 2024-09-19T07:48:11.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9df621c640 1 -- start start 2024-09-19T07:48:11.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9df621c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9df0150220 0x7f9df0156220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:11.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9df621c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9df0150600 0x7f9df00ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:11.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9df621c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df0068290 con 0x7f9df006d3e0 2024-09-19T07:48:11.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9df621c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f9df0068570 con 0x7f9df0150600 2024-09-19T07:48:11.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9df621c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9df0152d60 con 0x7f9df0150220 2024-09-19T07:48:11.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9deffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9df006d3e0 0x7f9df0153560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46860/0 (socket says 172.21.15.83:46860) 2024-09-19T07:48:11.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9deffff640 1 -- 172.21.15.83:0/4032662987 learned_addr learned my addr 172.21.15.83:0/4032662987 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:11.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9def7fe640 1 --2- 172.21.15.83:0/4032662987 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9df0150220 0x7f9df0156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:11.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9deeffd640 1 --2- 172.21.15.83:0/4032662987 >> v2:172.21.15.184:3300/0 conn(0x7f9df0150600 0x7f9df00ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:11.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9dee7fc640 1 -- 172.21.15.83:0/4032662987 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 702594459 0 0) 0x7f9df0068290 con 0x7f9df006d3e0 2024-09-19T07:48:11.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9dee7fc640 1 -- 172.21.15.83:0/4032662987 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dd8003530 con 0x7f9df006d3e0 2024-09-19T07:48:11.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9def7fe640 1 -- 172.21.15.83:0/4032662987 >> v2:172.21.15.184:3300/0 conn(0x7f9df0150600 msgr2=0x7f9df00ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:11.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9def7fe640 1 --2- 172.21.15.83:0/4032662987 >> v2:172.21.15.184:3300/0 conn(0x7f9df0150600 0x7f9df00ad2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9def7fe640 1 -- 172.21.15.83:0/4032662987 >> v1:172.21.15.83:6789/0 conn(0x7f9df006d3e0 legacy=0x7f9df0153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:11.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.073+0000 7f9def7fe640 1 -- 172.21.15.83:0/4032662987 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df0153040 con 0x7f9df0150220 2024-09-19T07:48:11.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.077+0000 7f9def7fe640 1 --2- 172.21.15.83:0/4032662987 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9df0150220 0x7f9df0156220 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9de400ba60 tx=0x7f9de401ea30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32cbf13d0b66aa09 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:11.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.077+0000 7f9dee7fc640 1 -- 172.21.15.83:0/4032662987 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de40b4070 con 0x7f9df0150220 2024-09-19T07:48:11.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.077+0000 7f9dee7fc640 1 -- 172.21.15.83:0/4032662987 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de40a7a80 con 0x7f9df0150220 2024-09-19T07:48:11.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.077+0000 7f9dee7fc640 1 -- 172.21.15.83:0/4032662987 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de40a7e50 con 0x7f9df0150220 2024-09-19T07:48:11.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.077+0000 7f9df621c640 1 -- 172.21.15.83:0/4032662987 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9df0150220 msgr2=0x7f9df0156220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:11.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.077+0000 7f9df621c640 1 --2- 172.21.15.83:0/4032662987 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9df0150220 0x7f9df0156220 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9de400ba60 tx=0x7f9de401ea30 comp rx=0 tx=0).stop 2024-09-19T07:48:11.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.077+0000 7f9df621c640 1 -- 172.21.15.83:0/4032662987 shutdown_connections 2024-09-19T07:48:11.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.077+0000 7f9df621c640 1 --2- 172.21.15.83:0/4032662987 >> v2:172.21.15.184:3300/0 conn(0x7f9df0150600 0x7f9df00ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.077+0000 7f9df621c640 1 --2- 172.21.15.83:0/4032662987 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9df0150220 0x7f9df0156220 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.077+0000 7f9df621c640 1 -- 172.21.15.83:0/4032662987 >> 172.21.15.83:0/4032662987 conn(0x7f9df00c48b0 msgr2=0x7f9df00c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9df621c640 1 -- 172.21.15.83:0/4032662987 shutdown_connections 2024-09-19T07:48:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9df621c640 1 -- 172.21.15.83:0/4032662987 wait complete. 2024-09-19T07:48:11.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9df621c640 1 Processor -- start 2024-09-19T07:48:11.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9df621c640 1 -- start start 2024-09-19T07:48:11.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9df621c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9df006d3e0 0x7f9df015c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:11.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9df621c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9df0150600 0x7f9df01f2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:11.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9df621c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df0157b40 0x7f9df01567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9df621c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9df0071a30 con 0x7f9df0157b40 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9df621c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9df00718b0 con 0x7f9df0150600 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9df621c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9df0071bb0 con 0x7f9df006d3e0 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9deffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df0157b40 0x7f9df01567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9deffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df0157b40 0x7f9df01567f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40116/0 (socket says 172.21.15.83:40116) 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9deffff640 1 -- 172.21.15.83:0/2806509524 learned_addr learned my addr 172.21.15.83:0/2806509524 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9deeffd640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9df0150600 0x7f9df01f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9def7fe640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9df006d3e0 0x7f9df015c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9deffff640 1 -- 172.21.15.83:0/2806509524 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9df006d3e0 msgr2=0x7f9df015c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9deffff640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9df006d3e0 0x7f9df015c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9deffff640 1 -- 172.21.15.83:0/2806509524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9df0150600 msgr2=0x7f9df01f2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9deffff640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9df0150600 0x7f9df01f2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.085+0000 7f9deffff640 1 -- 172.21.15.83:0/2806509524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df0158220 con 0x7f9df0157b40 2024-09-19T07:48:11.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.089+0000 7f9deffff640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df0157b40 0x7f9df01567f0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f9de800cb10 tx=0x7f9de800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:11.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.089+0000 7f9dd7fff640 1 -- 172.21.15.83:0/2806509524 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8017070 con 0x7f9df0157b40 2024-09-19T07:48:11.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.089+0000 7f9df621c640 1 -- 172.21.15.83:0/2806509524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df01f1820 con 0x7f9df0157b40 2024-09-19T07:48:11.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.089+0000 7f9dd7fff640 1 -- 172.21.15.83:0/2806509524 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de800ac70 con 0x7f9df0157b40 2024-09-19T07:48:11.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.089+0000 7f9dd7fff640 1 -- 172.21.15.83:0/2806509524 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de800ae60 con 0x7f9df0157b40 2024-09-19T07:48:11.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.089+0000 7f9df621c640 1 -- 172.21.15.83:0/2806509524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df01f1de0 con 0x7f9df0157b40 2024-09-19T07:48:11.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.089+0000 7f9dd7fff640 1 -- 172.21.15.83:0/2806509524 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9de801a070 con 0x7f9df0157b40 2024-09-19T07:48:11.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.089+0000 7f9dd7fff640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9ddc0678f0 0x7f9ddc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:11.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.089+0000 7f9dd7fff640 1 -- 172.21.15.83:0/2806509524 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7f9de8086a50 con 0x7f9df0157b40 2024-09-19T07:48:11.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.093+0000 7f9df621c640 1 -- 172.21.15.83:0/2806509524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dd00052a0 con 0x7f9df0157b40 2024-09-19T07:48:11.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.093+0000 7f9def7fe640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9ddc0678f0 0x7f9ddc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:11.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.093+0000 7f9def7fe640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9ddc0678f0 0x7f9ddc069db0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9de40b1000 tx=0x7f9de4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:11.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.101+0000 7f9dd7fff640 1 -- 172.21.15.83:0/2806509524 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9de8055e90 con 0x7f9df0157b40 2024-09-19T07:48:11.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.325+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e600c3400 con 0x7f9e60072d00 2024-09-19T07:48:11.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.329+0000 7f9e47fff640 1 -- 172.21.15.83:0/654550532 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) ==== 95+0+5720 (secure 0 0 0) 0x7f9e5005a4b0 con 0x7f9e60072d00 2024-09-19T07:48:11.327 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:11.327 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":194,"btime":"2024-09-19T07:48:02:799460+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:48:11.327 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 194 2024-09-19T07:48:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.329+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e4c067990 msgr2=0x7f9e4c069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.329+0000 7f9e6590a640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e4c067990 0x7f9e4c069e50 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9e540ad000 tx=0x7f9e540a6000 comp rx=0 tx=0).stop 2024-09-19T07:48:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.329+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e60072d00 msgr2=0x7f9e600beb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.329+0000 7f9e6590a640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e60072d00 0x7f9e600beb60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000e960 tx=0x7f9e5000ee20 comp rx=0 tx=0).stop 2024-09-19T07:48:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.329+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 shutdown_connections 2024-09-19T07:48:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.329+0000 7f9e6590a640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e4c067990 0x7f9e4c069e50 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.329+0000 7f9e6590a640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e601547b0 0x7f9e601efcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.329+0000 7f9e6590a640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e600c7d70 0x7f9e600c0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.329+0000 7f9e6590a640 1 --2- 172.21.15.83:0/654550532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e60072d00 0x7f9e600beb60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.329+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 >> 172.21.15.83:0/654550532 conn(0x7f9e600c5060 msgr2=0x7f9e6006da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:11.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.333+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 shutdown_connections 2024-09-19T07:48:11.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.333+0000 7f9e6590a640 1 -- 172.21.15.83:0/654550532 wait complete. 2024-09-19T07:48:11.357 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:48:11.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.433+0000 7f9df621c640 1 -- 172.21.15.83:0/2806509524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9dd0005590 con 0x7f9df0157b40 2024-09-19T07:48:11.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.433+0000 7f9dd7fff640 1 -- 172.21.15.83:0/2806509524 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) ==== 95+0+5720 (secure 0 0 0) 0x7f9de805a190 con 0x7f9df0157b40 2024-09-19T07:48:11.431 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:11.432 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":194,"btime":"2024-09-19T07:48:02:799460+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:48:11.432 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 194 2024-09-19T07:48:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.437+0000 7f9df621c640 1 -- 172.21.15.83:0/2806509524 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9ddc0678f0 msgr2=0x7f9ddc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.437+0000 7f9df621c640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9ddc0678f0 0x7f9ddc069db0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9de40b1000 tx=0x7f9de4008040 comp rx=0 tx=0).stop 2024-09-19T07:48:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.437+0000 7f9df621c640 1 -- 172.21.15.83:0/2806509524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df0157b40 msgr2=0x7f9df01567f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.437+0000 7f9df621c640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df0157b40 0x7f9df01567f0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f9de800cb10 tx=0x7f9de800a060 comp rx=0 tx=0).stop 2024-09-19T07:48:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.437+0000 7f9df621c640 1 -- 172.21.15.83:0/2806509524 shutdown_connections 2024-09-19T07:48:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.437+0000 7f9df621c640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9ddc0678f0 0x7f9ddc069db0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.437+0000 7f9df621c640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9df0157b40 0x7f9df01567f0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.437+0000 7f9df621c640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9df0150600 0x7f9df01f2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.437+0000 7f9df621c640 1 --2- 172.21.15.83:0/2806509524 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9df006d3e0 0x7f9df015c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.437+0000 7f9df621c640 1 -- 172.21.15.83:0/2806509524 >> 172.21.15.83:0/2806509524 conn(0x7f9df00c48b0 msgr2=0x7f9df0067a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.437+0000 7f9df621c640 1 -- 172.21.15.83:0/2806509524 shutdown_connections 2024-09-19T07:48:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:11.437+0000 7f9df621c640 1 -- 172.21.15.83:0/2806509524 wait complete. 2024-09-19T07:48:11.451 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-19T07:48:11.963 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:48:11.966 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:48:13.358 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:13.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:13.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.473+0000 7fc4d2d56640 1 Processor -- start 2024-09-19T07:48:13.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.473+0000 7fc4d2d56640 1 -- start start 2024-09-19T07:48:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4d2d56640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc4cc0c3a20 0x7fc4cc1594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4d2d56640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4cc155ee0 con 0x7fc4cc06f3f0 2024-09-19T07:48:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4d2d56640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4cc0684e0 con 0x7fc4cc0c4450 2024-09-19T07:48:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4d2d56640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc4cc068290 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4d0af2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc4cc0c3a20 0x7fc4cc1594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4d12f3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc4cc06f3f0 0x7fc4cc14c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46870/0 (socket says 172.21.15.83:46870) 2024-09-19T07:48:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4d12f3640 1 -- 172.21.15.83:0/824016766 learned_addr learned my addr 172.21.15.83:0/824016766 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4d0af2640 1 -- 172.21.15.83:0/824016766 >> v1:172.21.15.184:6789/0 conn(0x7fc4cc0c4450 legacy=0x7fc4cc1566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4d0af2640 1 -- 172.21.15.83:0/824016766 >> v1:172.21.15.83:6789/0 conn(0x7fc4cc06f3f0 legacy=0x7fc4cc14c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4d0af2640 1 -- 172.21.15.83:0/824016766 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4cc0687c0 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4d0af2640 1 --2- 172.21.15.83:0/824016766 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc0c3a20 0x7fc4cc1594d0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc068410 tx=0x7fc4c001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f46f93ebc4bddf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:13.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4cb7fe640 1 -- 172.21.15.83:0/824016766 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c00b1070 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4cb7fe640 1 -- 172.21.15.83:0/824016766 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4c00ac600 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.477+0000 7fc4cb7fe640 1 -- 172.21.15.83:0/824016766 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c00ac9d0 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 -- 172.21.15.83:0/824016766 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc0c3a20 msgr2=0x7fc4cc1594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:13.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 --2- 172.21.15.83:0/824016766 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc0c3a20 0x7fc4cc1594d0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc068410 tx=0x7fc4c001c3a0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 -- 172.21.15.83:0/824016766 shutdown_connections 2024-09-19T07:48:13.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 --2- 172.21.15.83:0/824016766 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc0c3a20 0x7fc4cc1594d0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 -- 172.21.15.83:0/824016766 >> 172.21.15.83:0/824016766 conn(0x7fc4cc0c48b0 msgr2=0x7fc4cc0c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:13.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 -- 172.21.15.83:0/824016766 shutdown_connections 2024-09-19T07:48:13.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 -- 172.21.15.83:0/824016766 wait complete. 2024-09-19T07:48:13.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 Processor -- start 2024-09-19T07:48:13.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 -- start start 2024-09-19T07:48:13.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4cc06f3f0 0x7fc4cc0c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:13.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc0c3a20 0x7fc4cc147120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:13.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4cc0c4450 0x7fc4cc1f1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:13.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc0c3f80 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc0c3e00 con 0x7fc4cc06f3f0 2024-09-19T07:48:13.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc4cc1644a0 con 0x7fc4cc0c4450 2024-09-19T07:48:13.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4cbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc0c3a20 0x7fc4cc147120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:13.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4cbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc0c3a20 0x7fc4cc147120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40132/0 (socket says 172.21.15.83:40132) 2024-09-19T07:48:13.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4cbfff640 1 -- 172.21.15.83:0/3879486045 learned_addr learned my addr 172.21.15.83:0/3879486045 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:13.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d0af2640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4cc06f3f0 0x7fc4cc0c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:13.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d12f3640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4cc0c4450 0x7fc4cc1f1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:13.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4cbfff640 1 -- 172.21.15.83:0/3879486045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4cc0c4450 msgr2=0x7fc4cc1f1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:13.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4cbfff640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4cc0c4450 0x7fc4cc1f1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4cbfff640 1 -- 172.21.15.83:0/3879486045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4cc06f3f0 msgr2=0x7fc4cc0c6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:13.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4cbfff640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4cc06f3f0 0x7fc4cc0c6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4cbfff640 1 -- 172.21.15.83:0/3879486045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4cc150300 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4cbfff640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc0c3a20 0x7fc4cc147120 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800c480 tx=0x7fc4b800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:13.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4c97fa640 1 -- 172.21.15.83:0/3879486045 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8012830 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4c97fa640 1 -- 172.21.15.83:0/3879486045 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4b8017070 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 -- 172.21.15.83:0/3879486045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4cc0c6000 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4c97fa640 1 -- 172.21.15.83:0/3879486045 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8012c30 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.481+0000 7fc4d2d56640 1 -- 172.21.15.83:0/3879486045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4cc073500 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.485+0000 7fc4d2d56640 1 -- 172.21.15.83:0/3879486045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4cc1594d0 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.485+0000 7fc4c97fa640 1 -- 172.21.15.83:0/3879486045 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc4b801c460 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.485+0000 7fc4c97fa640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc4b00678c0 0x7fc4b0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:13.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.485+0000 7fc4d0af2640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc4b00678c0 0x7fc4b0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:13.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.485+0000 7fc4c97fa640 1 -- 172.21.15.83:0/3879486045 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7fc4b8086aa0 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.485+0000 7fc4d0af2640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc4b00678c0 0x7fc4b0069d80 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc4c00ae000 tx=0x7fc4c00a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:13.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.489+0000 7fc4c97fa640 1 -- 172.21.15.83:0/3879486045 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc4b80565b0 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.573+0000 7f43682f5640 1 Processor -- start 2024-09-19T07:48:13.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.573+0000 7f43682f5640 1 -- start start 2024-09-19T07:48:13.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f43682f5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4360153570 0x7f43600c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:13.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f43682f5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43600c5110 0x7f4360159270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:13.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f43682f5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43600c3c20 con 0x7f43600c4130 2024-09-19T07:48:13.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f43682f5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f43600c89f0 con 0x7f4360153570 2024-09-19T07:48:13.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f43682f5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f43600c8cd0 con 0x7f43600c5110 2024-09-19T07:48:13.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f4366091640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4360153570 0x7f43600c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:13.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f4366091640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4360153570 0x7f43600c4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40140/0 (socket says 172.21.15.83:40140) 2024-09-19T07:48:13.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f4366091640 1 -- 172.21.15.83:0/3533281337 learned_addr learned my addr 172.21.15.83:0/3533281337 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:13.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f4365890640 1 --2- 172.21.15.83:0/3533281337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43600c5110 0x7f4360159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:13.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f4366091640 1 -- 172.21.15.83:0/3533281337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43600c5110 msgr2=0x7f4360159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:13.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f4366091640 1 --2- 172.21.15.83:0/3533281337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43600c5110 0x7f4360159270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f4366091640 1 -- 172.21.15.83:0/3533281337 >> v1:172.21.15.184:6789/0 conn(0x7f43600c4130 legacy=0x7f43601597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:13.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f4366091640 1 -- 172.21.15.83:0/3533281337 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43600c8fb0 con 0x7f4360153570 2024-09-19T07:48:13.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f4366091640 1 --2- 172.21.15.83:0/3533281337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4360153570 0x7f43600c4bd0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f43500098a0 tx=0x7f435001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796ac3d93b16ae29 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:13.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f436508f640 1 -- 172.21.15.83:0/3533281337 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43500b1070 con 0x7f4360153570 2024-09-19T07:48:13.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f436508f640 1 -- 172.21.15.83:0/3533281337 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43500ac620 con 0x7f4360153570 2024-09-19T07:48:13.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.577+0000 7f436508f640 1 -- 172.21.15.83:0/3533281337 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43500ac9f0 con 0x7f4360153570 2024-09-19T07:48:13.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 -- 172.21.15.83:0/3533281337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4360153570 msgr2=0x7f43600c4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:13.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 --2- 172.21.15.83:0/3533281337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4360153570 0x7f43600c4bd0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f43500098a0 tx=0x7f435001c3e0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 -- 172.21.15.83:0/3533281337 shutdown_connections 2024-09-19T07:48:13.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 --2- 172.21.15.83:0/3533281337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43600c5110 0x7f4360159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 --2- 172.21.15.83:0/3533281337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4360153570 0x7f43600c4bd0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 -- 172.21.15.83:0/3533281337 >> 172.21.15.83:0/3533281337 conn(0x7f43600c6110 msgr2=0x7f43600c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:13.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 -- 172.21.15.83:0/3533281337 shutdown_connections 2024-09-19T07:48:13.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 -- 172.21.15.83:0/3533281337 wait complete. 2024-09-19T07:48:13.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 Processor -- start 2024-09-19T07:48:13.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 -- start start 2024-09-19T07:48:13.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43600c4130 0x7f4360149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:13.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43600c5110 0x7f4360158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:13.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4360153570 0x7f43601f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:13.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4360163070 con 0x7f43600c4130 2024-09-19T07:48:13.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4360162ef0 con 0x7f43600c5110 2024-09-19T07:48:13.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f43601631f0 con 0x7f4360153570 2024-09-19T07:48:13.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4366091640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43600c4130 0x7f4360149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:13.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4366091640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43600c4130 0x7f4360149430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40142/0 (socket says 172.21.15.83:40142) 2024-09-19T07:48:13.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4366091640 1 -- 172.21.15.83:0/4256997444 learned_addr learned my addr 172.21.15.83:0/4256997444 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:13.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4365890640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43600c5110 0x7f4360158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:13.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4366892640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4360153570 0x7f43601f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:13.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4366091640 1 -- 172.21.15.83:0/4256997444 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4360153570 msgr2=0x7f43601f27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:13.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4366091640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4360153570 0x7f43601f27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4366091640 1 -- 172.21.15.83:0/4256997444 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43600c5110 msgr2=0x7f4360158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:13.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4366091640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43600c5110 0x7f4360158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4366091640 1 -- 172.21.15.83:0/4256997444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436014fd80 con 0x7f43600c4130 2024-09-19T07:48:13.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4366892640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4360153570 0x7f43601f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:48:13.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4366091640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43600c4130 0x7f4360149430 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f43500af040 tx=0x7f435001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:13.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4356ffd640 1 -- 172.21.15.83:0/4256997444 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43500bb020 con 0x7f43600c4130 2024-09-19T07:48:13.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4356ffd640 1 -- 172.21.15.83:0/4256997444 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43500b7640 con 0x7f43600c4130 2024-09-19T07:48:13.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 -- 172.21.15.83:0/4256997444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4360146ec0 con 0x7f43600c4130 2024-09-19T07:48:13.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f4356ffd640 1 -- 172.21.15.83:0/4256997444 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43500b1040 con 0x7f43600c4130 2024-09-19T07:48:13.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.581+0000 7f43682f5640 1 -- 172.21.15.83:0/4256997444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4360148760 con 0x7f43600c4130 2024-09-19T07:48:13.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.585+0000 7f43682f5640 1 -- 172.21.15.83:0/4256997444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43601489a0 con 0x7f43600c4130 2024-09-19T07:48:13.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.585+0000 7f4356ffd640 1 -- 172.21.15.83:0/4256997444 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f43500be020 con 0x7f43600c4130 2024-09-19T07:48:13.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.585+0000 7f4356ffd640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f43440678f0 0x7f4344069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:13.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.585+0000 7f4356ffd640 1 -- 172.21.15.83:0/4256997444 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7f4350121450 con 0x7f43600c4130 2024-09-19T07:48:13.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.585+0000 7f4365890640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f43440678f0 0x7f4344069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:13.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.585+0000 7f4365890640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f43440678f0 0x7f4344069db0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4360147570 tx=0x7f435c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.589+0000 7f4356ffd640 1 -- 172.21.15.83:0/4256997444 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f43601489a0 con 0x7f43600c4130 2024-09-19T07:48:13.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.701+0000 7fc4d2d56640 1 -- 172.21.15.83:0/3879486045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4cc06cc90 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4c97fa640 1 -- 172.21.15.83:0/3879486045 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) ==== 95+0+6546 (secure 0 0 0) 0x7fc4b8055eb0 con 0x7fc4cc0c3a20 2024-09-19T07:48:13.703 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:13.703 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":195,"btime":"2024-09-19T07:48:11:656818+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:48:13.703 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 195 2024-09-19T07:48:13.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4d2d56640 1 -- 172.21.15.83:0/3879486045 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc4b00678c0 msgr2=0x7fc4b0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:13.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4d2d56640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc4b00678c0 0x7fc4b0069d80 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc4c00ae000 tx=0x7fc4c00a7000 comp rx=0 tx=0).stop 2024-09-19T07:48:13.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4d2d56640 1 -- 172.21.15.83:0/3879486045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc0c3a20 msgr2=0x7fc4cc147120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:13.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4d2d56640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc0c3a20 0x7fc4cc147120 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800c480 tx=0x7fc4b800c940 comp rx=0 tx=0).stop 2024-09-19T07:48:13.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4d2d56640 1 -- 172.21.15.83:0/3879486045 shutdown_connections 2024-09-19T07:48:13.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4d2d56640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc4b00678c0 0x7fc4b0069d80 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4d2d56640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4cc0c4450 0x7fc4cc1f1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4d2d56640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc0c3a20 0x7fc4cc147120 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4d2d56640 1 --2- 172.21.15.83:0/3879486045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4cc06f3f0 0x7fc4cc0c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4d2d56640 1 -- 172.21.15.83:0/3879486045 >> 172.21.15.83:0/3879486045 conn(0x7fc4cc0c48b0 msgr2=0x7fc4cc1586c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:13.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4d2d56640 1 -- 172.21.15.83:0/3879486045 shutdown_connections 2024-09-19T07:48:13.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.705+0000 7fc4d2d56640 1 -- 172.21.15.83:0/3879486045 wait complete. 2024-09-19T07:48:13.723 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:48:13.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.797+0000 7f43682f5640 1 -- 172.21.15.83:0/4256997444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43600c4bd0 con 0x7f43600c4130 2024-09-19T07:48:13.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.797+0000 7f4356ffd640 1 -- 172.21.15.83:0/4256997444 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) ==== 95+0+6546 (secure 0 0 0) 0x7f43500c5030 con 0x7f43600c4130 2024-09-19T07:48:13.798 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:13.798 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":195,"btime":"2024-09-19T07:48:11:656818+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:48:13.798 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 195 2024-09-19T07:48:13.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.801+0000 7f43682f5640 1 -- 172.21.15.83:0/4256997444 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f43440678f0 msgr2=0x7f4344069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:13.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.801+0000 7f43682f5640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f43440678f0 0x7f4344069db0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4360147570 tx=0x7f435c008040 comp rx=0 tx=0).stop 2024-09-19T07:48:13.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.801+0000 7f43682f5640 1 -- 172.21.15.83:0/4256997444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43600c4130 msgr2=0x7f4360149430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:13.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.801+0000 7f43682f5640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43600c4130 0x7f4360149430 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f43500af040 tx=0x7f435001ac20 comp rx=0 tx=0).stop 2024-09-19T07:48:13.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.801+0000 7f43682f5640 1 -- 172.21.15.83:0/4256997444 shutdown_connections 2024-09-19T07:48:13.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.801+0000 7f43682f5640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f43440678f0 0x7f4344069db0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.801+0000 7f43682f5640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4360153570 0x7f43601f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.801+0000 7f43682f5640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43600c5110 0x7f4360158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.801+0000 7f43682f5640 1 --2- 172.21.15.83:0/4256997444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43600c4130 0x7f4360149430 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:13.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.801+0000 7f43682f5640 1 -- 172.21.15.83:0/4256997444 >> 172.21.15.83:0/4256997444 conn(0x7f43600c6110 msgr2=0x7f43601518b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:13.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.801+0000 7f43682f5640 1 -- 172.21.15.83:0/4256997444 shutdown_connections 2024-09-19T07:48:13.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:13.801+0000 7f43682f5640 1 -- 172.21.15.83:0/4256997444 wait complete. 2024-09-19T07:48:13.817 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-19T07:48:13.817 INFO:tasks.mds_thrash.fs.[a]:waiting for 30 secs before thrashing 2024-09-19T07:48:15.724 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:15.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63e1a5a640 1 Processor -- start 2024-09-19T07:48:15.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63e1a5a640 1 -- start start 2024-09-19T07:48:15.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63e1a5a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63dc150290 0x7f63dc0c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:15.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63e1a5a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f63dc153eb0 0x7f63dc154360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:15.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63e1a5a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63dc150670 con 0x7f63dc152b20 2024-09-19T07:48:15.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63e1a5a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f63dc0716c0 con 0x7f63dc153eb0 2024-09-19T07:48:15.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63e1a5a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f63dc0c3920 con 0x7f63dc150290 2024-09-19T07:48:15.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63da7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f63dc152b20 0x7f63dc0c4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46878/0 (socket says 172.21.15.83:46878) 2024-09-19T07:48:15.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63db7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f63dc153eb0 0x7f63dc154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:15.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63da7fc640 1 -- 172.21.15.83:0/1635052460 learned_addr learned my addr 172.21.15.83:0/1635052460 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:15.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63daffd640 1 --2- 172.21.15.83:0/1635052460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63dc150290 0x7f63dc0c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:15.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63d9ffb640 1 -- 172.21.15.83:0/1635052460 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2448722752 0 0) 0x7f63dc150670 con 0x7f63dc152b20 2024-09-19T07:48:15.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63d9ffb640 1 -- 172.21.15.83:0/1635052460 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63c4003530 con 0x7f63dc152b20 2024-09-19T07:48:15.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63db7fe640 1 -- 172.21.15.83:0/1635052460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63dc150290 msgr2=0x7f63dc0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:15.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63db7fe640 1 --2- 172.21.15.83:0/1635052460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63dc150290 0x7f63dc0c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:15.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63db7fe640 1 -- 172.21.15.83:0/1635052460 >> v1:172.21.15.83:6789/0 conn(0x7f63dc152b20 legacy=0x7f63dc0c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:15.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63db7fe640 1 -- 172.21.15.83:0/1635052460 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc0c2880 con 0x7f63dc153eb0 2024-09-19T07:48:15.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.857+0000 7f63daffd640 1 --2- 172.21.15.83:0/1635052460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63dc150290 0x7f63dc0c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:48:15.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63db7fe640 1 --2- 172.21.15.83:0/1635052460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63dc153eb0 0x7f63dc154360 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009d90 tx=0x7f63d401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5747bbf0f14439 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:15.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63d9ffb640 1 -- 172.21.15.83:0/1635052460 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d40b2070 con 0x7f63dc153eb0 2024-09-19T07:48:15.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63d9ffb640 1 -- 172.21.15.83:0/1635052460 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63d40a5920 con 0x7f63dc153eb0 2024-09-19T07:48:15.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63d9ffb640 1 -- 172.21.15.83:0/1635052460 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d40a5cd0 con 0x7f63dc153eb0 2024-09-19T07:48:15.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 -- 172.21.15.83:0/1635052460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63dc153eb0 msgr2=0x7f63dc154360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:15.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 --2- 172.21.15.83:0/1635052460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63dc153eb0 0x7f63dc154360 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009d90 tx=0x7f63d401c970 comp rx=0 tx=0).stop 2024-09-19T07:48:15.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 -- 172.21.15.83:0/1635052460 shutdown_connections 2024-09-19T07:48:15.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 --2- 172.21.15.83:0/1635052460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63dc153eb0 0x7f63dc154360 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:15.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 --2- 172.21.15.83:0/1635052460 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63dc150290 0x7f63dc0c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:15.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 -- 172.21.15.83:0/1635052460 >> 172.21.15.83:0/1635052460 conn(0x7f63dc06bc80 msgr2=0x7f63dc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:15.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 -- 172.21.15.83:0/1635052460 shutdown_connections 2024-09-19T07:48:15.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 -- 172.21.15.83:0/1635052460 wait complete. 2024-09-19T07:48:15.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 Processor -- start 2024-09-19T07:48:15.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 -- start start 2024-09-19T07:48:15.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63dc150290 0x7f63dc14ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:15.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63dc152b20 0x7f63dc1ea4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:15.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63dc153eb0 0x7f63dc1eaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:15.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc0c4790 con 0x7f63dc150290 2024-09-19T07:48:15.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc0c2530 con 0x7f63dc152b20 2024-09-19T07:48:15.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63e1a5a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f63dc0c4910 con 0x7f63dc153eb0 2024-09-19T07:48:15.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63daffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63dc150290 0x7f63dc14ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:15.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63daffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63dc150290 0x7f63dc14ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40152/0 (socket says 172.21.15.83:40152) 2024-09-19T07:48:15.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63daffd640 1 -- 172.21.15.83:0/570387057 learned_addr learned my addr 172.21.15.83:0/570387057 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:15.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63da7fc640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63dc152b20 0x7f63dc1ea4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:15.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63daffd640 1 -- 172.21.15.83:0/570387057 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63dc153eb0 msgr2=0x7f63dc1eaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:15.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63db7fe640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63dc153eb0 0x7f63dc1eaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:15.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63daffd640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63dc153eb0 0x7f63dc1eaa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:15.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63daffd640 1 -- 172.21.15.83:0/570387057 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63dc152b20 msgr2=0x7f63dc1ea4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:15.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63daffd640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63dc152b20 0x7f63dc1ea4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:15.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63daffd640 1 -- 172.21.15.83:0/570387057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc073a20 con 0x7f63dc150290 2024-09-19T07:48:15.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.861+0000 7f63da7fc640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63dc152b20 0x7f63dc1ea4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:48:15.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.865+0000 7f63daffd640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63dc150290 0x7f63dc14ea50 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f63d000c7c0 tx=0x7f63d000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:15.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.865+0000 7f63c3fff640 1 -- 172.21.15.83:0/570387057 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0007a30 con 0x7f63dc150290 2024-09-19T07:48:15.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.865+0000 7f63c3fff640 1 -- 172.21.15.83:0/570387057 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63d0007bd0 con 0x7f63dc150290 2024-09-19T07:48:15.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.865+0000 7f63c3fff640 1 -- 172.21.15.83:0/570387057 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d00113f0 con 0x7f63dc150290 2024-09-19T07:48:15.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.865+0000 7f63e1a5a640 1 -- 172.21.15.83:0/570387057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63dc14dc90 con 0x7f63dc150290 2024-09-19T07:48:15.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.865+0000 7f63e1a5a640 1 -- 172.21.15.83:0/570387057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63dc073460 con 0x7f63dc150290 2024-09-19T07:48:15.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.865+0000 7f63e1a5a640 1 -- 172.21.15.83:0/570387057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63dc0c1ff0 con 0x7f63dc150290 2024-09-19T07:48:15.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.865+0000 7f63c3fff640 1 -- 172.21.15.83:0/570387057 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f63d0011590 con 0x7f63dc150290 2024-09-19T07:48:15.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.865+0000 7f63c3fff640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f63c80678c0 0x7f63c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:15.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.865+0000 7f63c3fff640 1 -- 172.21.15.83:0/570387057 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7f63d0086f10 con 0x7f63dc150290 2024-09-19T07:48:15.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.865+0000 7f63da7fc640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f63c80678c0 0x7f63c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:15.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.865+0000 7f63da7fc640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f63c80678c0 0x7f63c8069d80 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f63dc1e7cd0 tx=0x7f63cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:15.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:15.869+0000 7f63c3fff640 1 -- 172.21.15.83:0/570387057 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f63d0015020 con 0x7f63dc150290 2024-09-19T07:48:16.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.061+0000 7f63e1a5a640 1 -- 172.21.15.83:0/570387057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63dc14c4b0 con 0x7f63dc150290 2024-09-19T07:48:16.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.061+0000 7f63c3fff640 1 -- 172.21.15.83:0/570387057 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) ==== 95+0+6546 (secure 0 0 0) 0x7f63d0056990 con 0x7f63dc150290 2024-09-19T07:48:16.060 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:16.061 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":195,"btime":"2024-09-19T07:48:11:656818+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:48:16.061 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 195 2024-09-19T07:48:16.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.065+0000 7f63e1a5a640 1 -- 172.21.15.83:0/570387057 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f63c80678c0 msgr2=0x7f63c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:16.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.065+0000 7f63e1a5a640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f63c80678c0 0x7f63c8069d80 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f63dc1e7cd0 tx=0x7f63cc009040 comp rx=0 tx=0).stop 2024-09-19T07:48:16.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.065+0000 7f63e1a5a640 1 -- 172.21.15.83:0/570387057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63dc150290 msgr2=0x7f63dc14ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:16.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.065+0000 7f63e1a5a640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63dc150290 0x7f63dc14ea50 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f63d000c7c0 tx=0x7f63d000cc80 comp rx=0 tx=0).stop 2024-09-19T07:48:16.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.065+0000 7f63e1a5a640 1 -- 172.21.15.83:0/570387057 shutdown_connections 2024-09-19T07:48:16.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.065+0000 7f63e1a5a640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f63c80678c0 0x7f63c8069d80 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:16.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.065+0000 7f63e1a5a640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63dc153eb0 0x7f63dc1eaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:16.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.065+0000 7f63e1a5a640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63dc152b20 0x7f63dc1ea4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:16.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.065+0000 7f63e1a5a640 1 --2- 172.21.15.83:0/570387057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63dc150290 0x7f63dc14ea50 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:16.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.065+0000 7f63e1a5a640 1 -- 172.21.15.83:0/570387057 >> 172.21.15.83:0/570387057 conn(0x7f63dc06bc80 msgr2=0x7f63dc14c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:16.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.065+0000 7f63e1a5a640 1 -- 172.21.15.83:0/570387057 shutdown_connections 2024-09-19T07:48:16.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:16.065+0000 7f63e1a5a640 1 -- 172.21.15.83:0/570387057 wait complete. 2024-09-19T07:48:16.078 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:48:18.079 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:18.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd62da26640 1 Processor -- start 2024-09-19T07:48:18.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd62da26640 1 -- start start 2024-09-19T07:48:18.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd62da26640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd628152b20 0x7fd6280c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:18.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd62da26640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd6280c45c0 0x7fd628153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:18.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd62da26640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd628150670 con 0x7fd628150290 2024-09-19T07:48:18.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd62da26640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd6280716c0 con 0x7fd6280c45c0 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd62da26640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd6280c3920 con 0x7fd628152b20 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd626ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd628150290 0x7fd6280c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46886/0 (socket says 172.21.15.83:46886) 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd626ffd640 1 -- 172.21.15.83:0/997877592 learned_addr learned my addr 172.21.15.83:0/997877592 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 390857381 0 0) 0x7fd628150670 con 0x7fd628150290 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd608003530 con 0x7fd628150290 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd61e7fc640 1 --2- 172.21.15.83:0/997877592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd628152b20 0x7fd6280c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6277fe640 1 --2- 172.21.15.83:0/997877592 >> v2:172.21.15.184:3300/0 conn(0x7fd6280c45c0 0x7fd628153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2732614026 0 0) 0x7fd608003530 con 0x7fd628150290 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd628152b20 msgr2=0x7fd6280c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 --2- 172.21.15.83:0/997877592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd628152b20 0x7fd6280c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 >> v2:172.21.15.184:3300/0 conn(0x7fd6280c45c0 msgr2=0x7fd628153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 --2- 172.21.15.83:0/997877592 >> v2:172.21.15.184:3300/0 conn(0x7fd6280c45c0 0x7fd628153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6280c2880 con 0x7fd628150290 2024-09-19T07:48:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fd618001680 con 0x7fd628150290 2024-09-19T07:48:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 >> v1:172.21.15.83:6789/0 conn(0x7fd628150290 legacy=0x7fd6280c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 --2- 172.21.15.83:0/997877592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6080039f0 0x7fd608003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 --2- 172.21.15.83:0/997877592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd628152b20 0x7fd608004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 --2- 172.21.15.83:0/997877592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd608004d50 0x7fd6080051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd60800b9d0 con 0x7fd608004d50 2024-09-19T07:48:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd60800bb50 con 0x7fd6080039f0 2024-09-19T07:48:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd60800bcd0 con 0x7fd628152b20 2024-09-19T07:48:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6277fe640 1 --2- 172.21.15.83:0/997877592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd608004d50 0x7fd6080051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd626ffd640 1 --2- 172.21.15.83:0/997877592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6080039f0 0x7fd608003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd61e7fc640 1 --2- 172.21.15.83:0/997877592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd628152b20 0x7fd608004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6277fe640 1 -- 172.21.15.83:0/997877592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd628152b20 msgr2=0x7fd608004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6277fe640 1 --2- 172.21.15.83:0/997877592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd628152b20 0x7fd608004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6277fe640 1 -- 172.21.15.83:0/997877592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6080039f0 msgr2=0x7fd608003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6277fe640 1 --2- 172.21.15.83:0/997877592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6080039f0 0x7fd608003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.213+0000 7fd6277fe640 1 -- 172.21.15.83:0/997877592 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6280c2880 con 0x7fd608004d50 2024-09-19T07:48:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd6277fe640 1 --2- 172.21.15.83:0/997877592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd608004d50 0x7fd6080051d0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fd620009fc0 tx=0x7fd62001ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55148a669ac3d2ff server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:18.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6200b2070 con 0x7fd608004d50 2024-09-19T07:48:18.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6200a5cd0 con 0x7fd608004d50 2024-09-19T07:48:18.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd6267fc640 1 -- 172.21.15.83:0/997877592 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd620009c10 con 0x7fd608004d50 2024-09-19T07:48:18.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd62da26640 1 -- 172.21.15.83:0/997877592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd608004d50 msgr2=0x7fd6080051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:18.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd62da26640 1 --2- 172.21.15.83:0/997877592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd608004d50 0x7fd6080051d0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fd620009fc0 tx=0x7fd62001ccc0 comp rx=0 tx=0).stop 2024-09-19T07:48:18.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd6277fe640 1 -- 172.21.15.83:0/997877592 reap_dead start 2024-09-19T07:48:18.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd62da26640 1 -- 172.21.15.83:0/997877592 shutdown_connections 2024-09-19T07:48:18.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd62da26640 1 -- 172.21.15.83:0/997877592 >> 172.21.15.83:0/997877592 conn(0x7fd62806bc80 msgr2=0x7fd62806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:18.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd62da26640 1 -- 172.21.15.83:0/997877592 shutdown_connections 2024-09-19T07:48:18.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd62da26640 1 -- 172.21.15.83:0/997877592 wait complete. 2024-09-19T07:48:18.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd62da26640 1 Processor -- start 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.217+0000 7fd62da26640 1 -- start start 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd62da26640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd6280c0c80 0x7fd6280c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd62da26640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6280ac2c0 0x7fd6280c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd62da26640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6280a9d30 0x7fd628072180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd62da26640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd6280c2e40 con 0x7fd6280ac2c0 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd62da26640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd6280c3490 con 0x7fd6280a9d30 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd62da26640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd6280c3120 con 0x7fd6280c0c80 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd61e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6280ac2c0 0x7fd6280c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd61e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6280ac2c0 0x7fd6280c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40170/0 (socket says 172.21.15.83:40170) 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd61e7fc640 1 -- 172.21.15.83:0/172408750 learned_addr learned my addr 172.21.15.83:0/172408750 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd6277fe640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6280a9d30 0x7fd628072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd626ffd640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd6280c0c80 0x7fd6280c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:18.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd61e7fc640 1 -- 172.21.15.83:0/172408750 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd6280c0c80 msgr2=0x7fd6280c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:18.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd61e7fc640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd6280c0c80 0x7fd6280c6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:18.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd61e7fc640 1 -- 172.21.15.83:0/172408750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6280a9d30 msgr2=0x7fd628072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:18.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd61e7fc640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6280a9d30 0x7fd628072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:18.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd61e7fc640 1 -- 172.21.15.83:0/172408750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6281eb1a0 con 0x7fd6280ac2c0 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd6277fe640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6280a9d30 0x7fd628072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd626ffd640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd6280c0c80 0x7fd6280c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd61e7fc640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6280ac2c0 0x7fd6280c72f0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fd610011590 tx=0x7fd610011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd624ff9640 1 -- 172.21.15.83:0/172408750 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610004220 con 0x7fd6280ac2c0 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd624ff9640 1 -- 172.21.15.83:0/172408750 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6100043c0 con 0x7fd6280ac2c0 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd62da26640 1 -- 172.21.15.83:0/172408750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6281eb480 con 0x7fd6280ac2c0 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd624ff9640 1 -- 172.21.15.83:0/172408750 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610015870 con 0x7fd6280ac2c0 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd62da26640 1 -- 172.21.15.83:0/172408750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6281eba40 con 0x7fd6280ac2c0 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd62da26640 1 -- 172.21.15.83:0/172408750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd62814bdb0 con 0x7fd6280ac2c0 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd624ff9640 1 -- 172.21.15.83:0/172408750 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd610015a10 con 0x7fd6280ac2c0 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd624ff9640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd60c0678f0 0x7fd60c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd624ff9640 1 -- 172.21.15.83:0/172408750 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7fd610086f90 con 0x7fd6280ac2c0 2024-09-19T07:48:18.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd626ffd640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd60c0678f0 0x7fd60c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:18.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.221+0000 7fd626ffd640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd60c0678f0 0x7fd60c069db0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd618004850 tx=0x7fd618000f60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:18.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.225+0000 7fd624ff9640 1 -- 172.21.15.83:0/172408750 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd610019030 con 0x7fd6280ac2c0 2024-09-19T07:48:18.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.409+0000 7fd62da26640 1 -- 172.21.15.83:0/172408750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd62814bad0 con 0x7fd6280ac2c0 2024-09-19T07:48:18.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.409+0000 7fd624ff9640 1 -- 172.21.15.83:0/172408750 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) ==== 95+0+6546 (secure 0 0 0) 0x7fd610056aa0 con 0x7fd6280ac2c0 2024-09-19T07:48:18.410 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:18.410 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":195,"btime":"2024-09-19T07:48:11:656818+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:48:18.410 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 195 2024-09-19T07:48:18.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.413+0000 7fd62da26640 1 -- 172.21.15.83:0/172408750 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd60c0678f0 msgr2=0x7fd60c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:18.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.413+0000 7fd62da26640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd60c0678f0 0x7fd60c069db0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd618004850 tx=0x7fd618000f60 comp rx=0 tx=0).stop 2024-09-19T07:48:18.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.413+0000 7fd62da26640 1 -- 172.21.15.83:0/172408750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6280ac2c0 msgr2=0x7fd6280c72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:18.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.413+0000 7fd62da26640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6280ac2c0 0x7fd6280c72f0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fd610011590 tx=0x7fd610011a50 comp rx=0 tx=0).stop 2024-09-19T07:48:18.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.413+0000 7fd62da26640 1 -- 172.21.15.83:0/172408750 shutdown_connections 2024-09-19T07:48:18.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.413+0000 7fd62da26640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd60c0678f0 0x7fd60c069db0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:18.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.413+0000 7fd62da26640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6280a9d30 0x7fd628072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:18.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.413+0000 7fd62da26640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6280ac2c0 0x7fd6280c72f0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:18.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.413+0000 7fd62da26640 1 --2- 172.21.15.83:0/172408750 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd6280c0c80 0x7fd6280c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:18.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.413+0000 7fd62da26640 1 -- 172.21.15.83:0/172408750 >> 172.21.15.83:0/172408750 conn(0x7fd62806bc80 msgr2=0x7fd6280c1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:18.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.413+0000 7fd62da26640 1 -- 172.21.15.83:0/172408750 shutdown_connections 2024-09-19T07:48:18.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:18.413+0000 7fd62da26640 1 -- 172.21.15.83:0/172408750 wait complete. 2024-09-19T07:48:18.428 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:48:20.429 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:20.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.561+0000 7f9f7977c640 1 Processor -- start 2024-09-19T07:48:20.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.561+0000 7f9f7977c640 1 -- start start 2024-09-19T07:48:20.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f7977c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9f74152b20 0x7f9f740c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:20.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f7977c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f74150670 con 0x7f9f74150290 2024-09-19T07:48:20.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f7977c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f740c2880 con 0x7f9f740c45c0 2024-09-19T07:48:20.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f7977c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f9f740716c0 con 0x7f9f74152b20 2024-09-19T07:48:20.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f727fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9f74152b20 0x7f9f740c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:20.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f72ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9f74150290 0x7f9f740c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54928/0 (socket says 172.21.15.83:54928) 2024-09-19T07:48:20.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f72ffd640 1 -- 172.21.15.83:0/3321319916 learned_addr learned my addr 172.21.15.83:0/3321319916 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f727fc640 1 -- 172.21.15.83:0/3321319916 >> v1:172.21.15.184:6789/0 conn(0x7f9f740c45c0 legacy=0x7f9f74153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f727fc640 1 -- 172.21.15.83:0/3321319916 >> v1:172.21.15.83:6789/0 conn(0x7f9f74150290 legacy=0x7f9f740c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f71ffb640 1 -- 172.21.15.83:0/3321319916 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 60360269 0 0) 0x7f9f74150670 con 0x7f9f74150290 2024-09-19T07:48:20.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f727fc640 1 -- 172.21.15.83:0/3321319916 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f740c2b60 con 0x7f9f74152b20 2024-09-19T07:48:20.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f727fc640 1 --2- 172.21.15.83:0/3321319916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f74152b20 0x7f9f740c4080 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f9f640097c0 tx=0x7f9f6401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60423f5db994d49 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:20.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f71ffb640 1 -- 172.21.15.83:0/3321319916 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f640b0070 con 0x7f9f74152b20 2024-09-19T07:48:20.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f71ffb640 1 -- 172.21.15.83:0/3321319916 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f640ab600 con 0x7f9f74152b20 2024-09-19T07:48:20.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.565+0000 7f9f71ffb640 1 -- 172.21.15.83:0/3321319916 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f640ab9d0 con 0x7f9f74152b20 2024-09-19T07:48:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 -- 172.21.15.83:0/3321319916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f74152b20 msgr2=0x7f9f740c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 --2- 172.21.15.83:0/3321319916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f74152b20 0x7f9f740c4080 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f9f640097c0 tx=0x7f9f6401c460 comp rx=0 tx=0).stop 2024-09-19T07:48:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 -- 172.21.15.83:0/3321319916 shutdown_connections 2024-09-19T07:48:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 --2- 172.21.15.83:0/3321319916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f74152b20 0x7f9f740c4080 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 -- 172.21.15.83:0/3321319916 >> 172.21.15.83:0/3321319916 conn(0x7f9f7406bc80 msgr2=0x7f9f7406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 -- 172.21.15.83:0/3321319916 shutdown_connections 2024-09-19T07:48:20.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 -- 172.21.15.83:0/3321319916 wait complete. 2024-09-19T07:48:20.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 Processor -- start 2024-09-19T07:48:20.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 -- start start 2024-09-19T07:48:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f740c45c0 0x7f9f7414f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f74150290 0x7f9f740c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f74152b20 0x7f9f74072780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7406e020 con 0x7f9f74152b20 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7406dea0 con 0x7f9f74150290 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f7977c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9f7406e1a0 con 0x7f9f740c45c0 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f737fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f74152b20 0x7f9f74072780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f737fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f74152b20 0x7f9f74072780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60768/0 (socket says 172.21.15.83:60768) 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f737fe640 1 -- 172.21.15.83:0/2570791593 learned_addr learned my addr 172.21.15.83:0/2570791593 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f727fc640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f74150290 0x7f9f740c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f72ffd640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f740c45c0 0x7f9f7414f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f737fe640 1 -- 172.21.15.83:0/2570791593 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f740c45c0 msgr2=0x7f9f7414f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f737fe640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f740c45c0 0x7f9f7414f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f737fe640 1 -- 172.21.15.83:0/2570791593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f74150290 msgr2=0x7f9f740c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f737fe640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f74150290 0x7f9f740c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:20.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f737fe640 1 -- 172.21.15.83:0/2570791593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7414cc10 con 0x7f9f74152b20 2024-09-19T07:48:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f72ffd640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f740c45c0 0x7f9f7414f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:48:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f737fe640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f74152b20 0x7f9f74072780 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00c780 tx=0x7f9f6c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f5b7fe640 1 -- 172.21.15.83:0/2570791593 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c004b70 con 0x7f9f74152b20 2024-09-19T07:48:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.569+0000 7f9f5b7fe640 1 -- 172.21.15.83:0/2570791593 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f6c004d10 con 0x7f9f74152b20 2024-09-19T07:48:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.573+0000 7f9f7977c640 1 -- 172.21.15.83:0/2570791593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f741497a0 con 0x7f9f74152b20 2024-09-19T07:48:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.573+0000 7f9f5b7fe640 1 -- 172.21.15.83:0/2570791593 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c00aa90 con 0x7f9f74152b20 2024-09-19T07:48:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.573+0000 7f9f7977c640 1 -- 172.21.15.83:0/2570791593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f74073ec0 con 0x7f9f74152b20 2024-09-19T07:48:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.573+0000 7f9f5b7fe640 1 -- 172.21.15.83:0/2570791593 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9f6c00ac30 con 0x7f9f74152b20 2024-09-19T07:48:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.573+0000 7f9f7977c640 1 -- 172.21.15.83:0/2570791593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f741f2620 con 0x7f9f74152b20 2024-09-19T07:48:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.573+0000 7f9f5b7fe640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9f600678c0 0x7f9f60069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.573+0000 7f9f5b7fe640 1 -- 172.21.15.83:0/2570791593 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7f9f6c086f60 con 0x7f9f74152b20 2024-09-19T07:48:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.573+0000 7f9f72ffd640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9f600678c0 0x7f9f60069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.573+0000 7f9f72ffd640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9f600678c0 0x7f9f60069d80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9f68000f70 tx=0x7f9f68009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:20.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.577+0000 7f9f5b7fe640 1 -- 172.21.15.83:0/2570791593 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9f6c020070 con 0x7f9f74152b20 2024-09-19T07:48:20.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.777+0000 7f9f7977c640 1 -- 172.21.15.83:0/2570791593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f740724e0 con 0x7f9f74152b20 2024-09-19T07:48:20.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.777+0000 7f9f5b7fe640 1 -- 172.21.15.83:0/2570791593 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) ==== 95+0+6546 (secure 0 0 0) 0x7f9f6c056a70 con 0x7f9f74152b20 2024-09-19T07:48:20.776 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:20.776 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":195,"btime":"2024-09-19T07:48:11:656818+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:48:20.777 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 195 2024-09-19T07:48:20.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.777+0000 7f9f7977c640 1 -- 172.21.15.83:0/2570791593 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9f600678c0 msgr2=0x7f9f60069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:20.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.777+0000 7f9f7977c640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9f600678c0 0x7f9f60069d80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9f68000f70 tx=0x7f9f68009040 comp rx=0 tx=0).stop 2024-09-19T07:48:20.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.777+0000 7f9f7977c640 1 -- 172.21.15.83:0/2570791593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f74152b20 msgr2=0x7f9f74072780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:20.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.777+0000 7f9f7977c640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f74152b20 0x7f9f74072780 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00c780 tx=0x7f9f6c00cc40 comp rx=0 tx=0).stop 2024-09-19T07:48:20.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.777+0000 7f9f7977c640 1 -- 172.21.15.83:0/2570791593 shutdown_connections 2024-09-19T07:48:20.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.777+0000 7f9f7977c640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9f600678c0 0x7f9f60069d80 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:20.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.781+0000 7f9f7977c640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f74152b20 0x7f9f74072780 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:20.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.781+0000 7f9f7977c640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f74150290 0x7f9f740c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:20.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.781+0000 7f9f7977c640 1 --2- 172.21.15.83:0/2570791593 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f740c45c0 0x7f9f7414f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:20.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.781+0000 7f9f7977c640 1 -- 172.21.15.83:0/2570791593 >> 172.21.15.83:0/2570791593 conn(0x7f9f7406bc80 msgr2=0x7f9f7414ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:20.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.781+0000 7f9f7977c640 1 -- 172.21.15.83:0/2570791593 shutdown_connections 2024-09-19T07:48:20.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:20.781+0000 7f9f7977c640 1 -- 172.21.15.83:0/2570791593 wait complete. 2024-09-19T07:48:20.795 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:48:22.797 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:22.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.929+0000 7ff833552640 1 Processor -- start 2024-09-19T07:48:22.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.929+0000 7ff833552640 1 -- start start 2024-09-19T07:48:22.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.929+0000 7ff833552640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff82c071100 0x7ff82c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:22.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.929+0000 7ff833552640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff82c0c28e0 0x7ff82c150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:22.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.929+0000 7ff833552640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82c0a9e60 con 0x7ff82c151480 2024-09-19T07:48:22.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.929+0000 7ff833552640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff82c071b50 con 0x7ff82c071100 2024-09-19T07:48:22.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.929+0000 7ff833552640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff82c0c1ff0 con 0x7ff82c0c28e0 2024-09-19T07:48:22.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.929+0000 7ff8312ee640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff82c071100 0x7ff82c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:22.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff8312ee640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff82c071100 0x7ff82c1545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60774/0 (socket says 172.21.15.83:60774) 2024-09-19T07:48:22.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff8312ee640 1 -- 172.21.15.83:0/3495570135 learned_addr learned my addr 172.21.15.83:0/3495570135 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:22.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff830aed640 1 --2- 172.21.15.83:0/3495570135 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff82c0c28e0 0x7ff82c150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:22.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff8312ee640 1 -- 172.21.15.83:0/3495570135 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff82c0c28e0 msgr2=0x7ff82c150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:22.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff8312ee640 1 --2- 172.21.15.83:0/3495570135 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff82c0c28e0 0x7ff82c150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:22.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff8312ee640 1 -- 172.21.15.83:0/3495570135 >> v1:172.21.15.184:6789/0 conn(0x7ff82c151480 legacy=0x7ff82c06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:22.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff8312ee640 1 -- 172.21.15.83:0/3495570135 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82c0aa140 con 0x7ff82c071100 2024-09-19T07:48:22.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff8312ee640 1 --2- 172.21.15.83:0/3495570135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff82c071100 0x7ff82c1545e0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7ff828009e50 tx=0x7ff82801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3499a613d466f709 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:22.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff823fff640 1 -- 172.21.15.83:0/3495570135 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8280a55e0 con 0x7ff82c071100 2024-09-19T07:48:22.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff823fff640 1 -- 172.21.15.83:0/3495570135 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8280a5780 con 0x7ff82c071100 2024-09-19T07:48:22.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff823fff640 1 -- 172.21.15.83:0/3495570135 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8280a5b50 con 0x7ff82c071100 2024-09-19T07:48:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff833552640 1 -- 172.21.15.83:0/3495570135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff82c071100 msgr2=0x7ff82c1545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff833552640 1 --2- 172.21.15.83:0/3495570135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff82c071100 0x7ff82c1545e0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7ff828009e50 tx=0x7ff82801c960 comp rx=0 tx=0).stop 2024-09-19T07:48:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff833552640 1 -- 172.21.15.83:0/3495570135 shutdown_connections 2024-09-19T07:48:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff833552640 1 --2- 172.21.15.83:0/3495570135 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff82c0c28e0 0x7ff82c150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff833552640 1 --2- 172.21.15.83:0/3495570135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff82c071100 0x7ff82c1545e0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff833552640 1 -- 172.21.15.83:0/3495570135 >> 172.21.15.83:0/3495570135 conn(0x7ff82c06bc80 msgr2=0x7ff82c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff833552640 1 -- 172.21.15.83:0/3495570135 shutdown_connections 2024-09-19T07:48:22.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.933+0000 7ff833552640 1 -- 172.21.15.83:0/3495570135 wait complete. 2024-09-19T07:48:22.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff833552640 1 Processor -- start 2024-09-19T07:48:22.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff833552640 1 -- start start 2024-09-19T07:48:22.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff833552640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff82c071100 0x7ff82c0c3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:22.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff833552640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff82c0c28e0 0x7ff82c0ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:22.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff833552640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff82c151480 0x7ff82c0ab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:22.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff833552640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c071660 con 0x7ff82c0c28e0 2024-09-19T07:48:22.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff833552640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c0714e0 con 0x7ff82c071100 2024-09-19T07:48:22.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff833552640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff82c160c60 con 0x7ff82c151480 2024-09-19T07:48:22.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff830aed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff82c0c28e0 0x7ff82c0ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:22.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff830aed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff82c0c28e0 0x7ff82c0ab490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60780/0 (socket says 172.21.15.83:60780) 2024-09-19T07:48:22.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff830aed640 1 -- 172.21.15.83:0/1815042077 learned_addr learned my addr 172.21.15.83:0/1815042077 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:22.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff8312ee640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff82c071100 0x7ff82c0c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:22.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff831aef640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff82c151480 0x7ff82c0ab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:22.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff830aed640 1 -- 172.21.15.83:0/1815042077 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff82c151480 msgr2=0x7ff82c0ab9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff830aed640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff82c151480 0x7ff82c0ab9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff830aed640 1 -- 172.21.15.83:0/1815042077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff82c071100 msgr2=0x7ff82c0c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff830aed640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff82c071100 0x7ff82c0c3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff830aed640 1 -- 172.21.15.83:0/1815042077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82c146df0 con 0x7ff82c0c28e0 2024-09-19T07:48:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff830aed640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff82c0c28e0 0x7ff82c0ab490 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00d5f0 tx=0x7ff81c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff821ffb640 1 -- 172.21.15.83:0/1815042077 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c0125f0 con 0x7ff82c0c28e0 2024-09-19T07:48:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff821ffb640 1 -- 172.21.15.83:0/1815042077 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff81c019070 con 0x7ff82c0c28e0 2024-09-19T07:48:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff833552640 1 -- 172.21.15.83:0/1815042077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff82c0c7170 con 0x7ff82c0c28e0 2024-09-19T07:48:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff821ffb640 1 -- 172.21.15.83:0/1815042077 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c0129d0 con 0x7ff82c0c28e0 2024-09-19T07:48:22.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff833552640 1 -- 172.21.15.83:0/1815042077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff82c147060 con 0x7ff82c0c28e0 2024-09-19T07:48:22.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff833552640 1 -- 172.21.15.83:0/1815042077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff82c0aad60 con 0x7ff82c0c28e0 2024-09-19T07:48:22.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.937+0000 7ff821ffb640 1 -- 172.21.15.83:0/1815042077 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff81c012b70 con 0x7ff82c0c28e0 2024-09-19T07:48:22.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.941+0000 7ff821ffb640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff8100678c0 0x7ff810069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:22.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.941+0000 7ff821ffb640 1 -- 172.21.15.83:0/1815042077 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9905+0+0 (secure 0 0 0) 0x7ff81c086f00 con 0x7ff82c0c28e0 2024-09-19T07:48:22.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.941+0000 7ff8312ee640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff8100678c0 0x7ff810069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:22.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.941+0000 7ff8312ee640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff8100678c0 0x7ff810069d80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff8280ae000 tx=0x7ff8280a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:22.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:22.945+0000 7ff821ffb640 1 -- 172.21.15.83:0/1815042077 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff81c023020 con 0x7ff82c0c28e0 2024-09-19T07:48:23.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.137+0000 7ff833552640 1 -- 172.21.15.83:0/1815042077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff82c1545e0 con 0x7ff82c0c28e0 2024-09-19T07:48:23.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.137+0000 7ff821ffb640 1 -- 172.21.15.83:0/1815042077 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) ==== 95+0+6546 (secure 0 0 0) 0x7ff81c056a10 con 0x7ff82c0c28e0 2024-09-19T07:48:23.136 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:23.136 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":195,"btime":"2024-09-19T07:48:11:656818+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8196,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:47:06.716118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8028":{"gid":8028,"name":"c","rank":0,"incarnation":186,"state":"up:active","state_seq":24,"addr":"172.21.15.83:6837/1602509240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":1602509240},{"type":"v1","addr":"172.21.15.83:6837","nonce":1602509240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8028,"qdb_cluster":[8028]},"id":2}]} 2024-09-19T07:48:23.137 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 195 2024-09-19T07:48:23.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.137+0000 7ff833552640 1 -- 172.21.15.83:0/1815042077 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff8100678c0 msgr2=0x7ff810069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:23.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.137+0000 7ff833552640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff8100678c0 0x7ff810069d80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff8280ae000 tx=0x7ff8280a7070 comp rx=0 tx=0).stop 2024-09-19T07:48:23.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.137+0000 7ff833552640 1 -- 172.21.15.83:0/1815042077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff82c0c28e0 msgr2=0x7ff82c0ab490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:23.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.137+0000 7ff833552640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff82c0c28e0 0x7ff82c0ab490 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00d5f0 tx=0x7ff81c00dab0 comp rx=0 tx=0).stop 2024-09-19T07:48:23.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.141+0000 7ff833552640 1 -- 172.21.15.83:0/1815042077 shutdown_connections 2024-09-19T07:48:23.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.141+0000 7ff833552640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff8100678c0 0x7ff810069d80 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:23.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.141+0000 7ff833552640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff82c151480 0x7ff82c0ab9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:23.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.141+0000 7ff833552640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff82c0c28e0 0x7ff82c0ab490 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:23.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.141+0000 7ff833552640 1 --2- 172.21.15.83:0/1815042077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff82c071100 0x7ff82c0c3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:23.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.141+0000 7ff833552640 1 -- 172.21.15.83:0/1815042077 >> 172.21.15.83:0/1815042077 conn(0x7ff82c06bc80 msgr2=0x7ff82c06dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:23.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.141+0000 7ff833552640 1 -- 172.21.15.83:0/1815042077 shutdown_connections 2024-09-19T07:48:23.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:23.141+0000 7ff833552640 1 -- 172.21.15.83:0/1815042077 wait complete. 2024-09-19T07:48:23.154 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:48:25.155 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:25.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.285+0000 7f8a694d1640 1 Processor -- start 2024-09-19T07:48:25.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.285+0000 7f8a694d1640 1 -- start start 2024-09-19T07:48:25.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.285+0000 7f8a694d1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64150290 0x7f8a640c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.285+0000 7f8a694d1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a64150670 con 0x7f8a64152b20 2024-09-19T07:48:25.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.285+0000 7f8a694d1640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a640c2880 con 0x7f8a64153eb0 2024-09-19T07:48:25.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.285+0000 7f8a694d1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8a640716c0 con 0x7f8a64150290 2024-09-19T07:48:25.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a627fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8a64152b20 0x7f8a640c4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54944/0 (socket says 172.21.15.83:54944) 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a627fc640 1 -- 172.21.15.83:0/2900821462 learned_addr learned my addr 172.21.15.83:0/2900821462 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a62ffd640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64150290 0x7f8a640c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3437266484 0 0) 0x7f8a64150670 con 0x7f8a64152b20 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a4c005740 con 0x7f8a64152b20 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 344557247 0 0) 0x7f8a4c005740 con 0x7f8a64152b20 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64150290 msgr2=0x7f8a640c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64150290 0x7f8a640c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 >> v1:172.21.15.184:6789/0 conn(0x7f8a64153eb0 legacy=0x7f8a64154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a640c2b60 con 0x7f8a64152b20 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a62ffd640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64150290 0x7f8a640c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f8a58001680 con 0x7f8a64152b20 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 >> v1:172.21.15.83:6789/0 conn(0x7f8a64152b20 legacy=0x7f8a640c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c005c00 0x7f8a4c006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64150290 0x7f8a4c006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4c006f60 0x7f8a4c0073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c00dcc0 con 0x7f8a4c006f60 2024-09-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c00de70 con 0x7f8a4c005c00 2024-09-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8a4c00e070 con 0x7f8a64150290 2024-09-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a637fe640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4c006f60 0x7f8a4c0073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a627fc640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64150290 0x7f8a4c006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a62ffd640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c005c00 0x7f8a4c006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a637fe640 1 -- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64150290 msgr2=0x7f8a4c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a637fe640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64150290 0x7f8a4c006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a637fe640 1 -- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c005c00 msgr2=0x7f8a4c006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a637fe640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c005c00 0x7f8a4c006020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a637fe640 1 -- 172.21.15.83:0/2900821462 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c005740 con 0x7f8a4c006f60 2024-09-19T07:48:25.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a62ffd640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c005c00 0x7f8a4c006020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:48:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a637fe640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4c006f60 0x7f8a4c0073e0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c001e40 tx=0x7f8a5c01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c5496fb5a09fe server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c001a90 con 0x7f8a4c006f60 2024-09-19T07:48:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a5c003b80 con 0x7f8a4c006f60 2024-09-19T07:48:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.289+0000 7f8a61ffb640 1 -- 172.21.15.83:0/2900821462 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c005080 con 0x7f8a4c006f60 2024-09-19T07:48:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 -- 172.21.15.83:0/2900821462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4c006f60 msgr2=0x7f8a4c0073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 --2- 172.21.15.83:0/2900821462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4c006f60 0x7f8a4c0073e0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c001e40 tx=0x7f8a5c01c760 comp rx=0 tx=0).stop 2024-09-19T07:48:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a637fe640 1 -- 172.21.15.83:0/2900821462 reap_dead start 2024-09-19T07:48:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 -- 172.21.15.83:0/2900821462 shutdown_connections 2024-09-19T07:48:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 -- 172.21.15.83:0/2900821462 >> 172.21.15.83:0/2900821462 conn(0x7f8a6406bc80 msgr2=0x7f8a6406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 -- 172.21.15.83:0/2900821462 shutdown_connections 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 -- 172.21.15.83:0/2900821462 wait complete. 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 Processor -- start 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 -- start start 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4c006f60 0x7f8a64149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64143c40 0x7f8a641ea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a641eac20 0x7f8a641de250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8a640c3490 con 0x7f8a4c006f60 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a640c3120 con 0x7f8a641eac20 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8a640c4790 con 0x7f8a64143c40 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a62ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4c006f60 0x7f8a64149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a62ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4c006f60 0x7f8a64149470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60786/0 (socket says 172.21.15.83:60786) 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a62ffd640 1 -- 172.21.15.83:0/339219466 learned_addr learned my addr 172.21.15.83:0/339219466 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a637fe640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a641eac20 0x7f8a641de250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a627fc640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64143c40 0x7f8a641ea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a62ffd640 1 -- 172.21.15.83:0/339219466 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64143c40 msgr2=0x7f8a641ea6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a62ffd640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64143c40 0x7f8a641ea6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a62ffd640 1 -- 172.21.15.83:0/339219466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a641eac20 msgr2=0x7f8a641de250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a62ffd640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a641eac20 0x7f8a641de250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a62ffd640 1 -- 172.21.15.83:0/339219466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6414dc60 con 0x7f8a4c006f60 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a62ffd640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4c006f60 0x7f8a64149470 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400f650 tx=0x7f8a5400fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a4b7fe640 1 -- 172.21.15.83:0/339219466 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a54018070 con 0x7f8a4c006f60 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a4b7fe640 1 -- 172.21.15.83:0/339219466 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a54009880 con 0x7f8a4c006f60 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 -- 172.21.15.83:0/339219466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6414df00 con 0x7f8a4c006f60 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a4b7fe640 1 -- 172.21.15.83:0/339219466 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a54009c50 con 0x7f8a4c006f60 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 -- 172.21.15.83:0/339219466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a6414e4c0 con 0x7f8a4c006f60 2024-09-19T07:48:25.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.293+0000 7f8a694d1640 1 -- 172.21.15.83:0/339219466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a6414c150 con 0x7f8a4c006f60 2024-09-19T07:48:25.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.297+0000 7f8a4b7fe640 1 -- 172.21.15.83:0/339219466 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8a5401b070 con 0x7f8a4c006f60 2024-09-19T07:48:25.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.297+0000 7f8a4b7fe640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8a500678f0 0x7f8a50069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.297+0000 7f8a627fc640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8a500678f0 0x7f8a50069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.297+0000 7f8a4b7fe640 1 -- 172.21.15.83:0/339219466 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9991+0+0 (secure 0 0 0) 0x7f8a54086fe0 con 0x7f8a4c006f60 2024-09-19T07:48:25.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.297+0000 7f8a627fc640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8a500678f0 0x7f8a50069db0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8a6414abf0 tx=0x7f8a58009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:25.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.301+0000 7f8a4b7fe640 1 -- 172.21.15.83:0/339219466 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f8a54056a60 con 0x7f8a4c006f60 2024-09-19T07:48:25.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.497+0000 7f8a694d1640 1 -- 172.21.15.83:0/339219466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a641499b0 con 0x7f8a4c006f60 2024-09-19T07:48:25.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.497+0000 7f8a4b7fe640 1 -- 172.21.15.83:0/339219466 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+5715 (secure 0 0 0) 0x7f8a54056360 con 0x7f8a4c006f60 2024-09-19T07:48:25.497 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:25.497 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":197,"btime":"2024-09-19T07:48:24:698802+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:23.846369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:reconnect","state_seq":5,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:48:25.497 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 197 2024-09-19T07:48:25.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.501+0000 7f8a694d1640 1 -- 172.21.15.83:0/339219466 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8a500678f0 msgr2=0x7f8a50069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:25.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.501+0000 7f8a694d1640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8a500678f0 0x7f8a50069db0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8a6414abf0 tx=0x7f8a58009040 comp rx=0 tx=0).stop 2024-09-19T07:48:25.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.501+0000 7f8a694d1640 1 -- 172.21.15.83:0/339219466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4c006f60 msgr2=0x7f8a64149470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:25.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.501+0000 7f8a694d1640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4c006f60 0x7f8a64149470 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400f650 tx=0x7f8a5400fb10 comp rx=0 tx=0).stop 2024-09-19T07:48:25.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.501+0000 7f8a694d1640 1 -- 172.21.15.83:0/339219466 shutdown_connections 2024-09-19T07:48:25.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.501+0000 7f8a694d1640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8a500678f0 0x7f8a50069db0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.501+0000 7f8a694d1640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a641eac20 0x7f8a641de250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.501+0000 7f8a694d1640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a64143c40 0x7f8a641ea6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.501+0000 7f8a694d1640 1 --2- 172.21.15.83:0/339219466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4c006f60 0x7f8a64149470 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.501+0000 7f8a694d1640 1 -- 172.21.15.83:0/339219466 >> 172.21.15.83:0/339219466 conn(0x7f8a6406bc80 msgr2=0x7f8a64152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:25.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.501+0000 7f8a694d1640 1 -- 172.21.15.83:0/339219466 shutdown_connections 2024-09-19T07:48:25.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.501+0000 7f8a694d1640 1 -- 172.21.15.83:0/339219466 wait complete. 2024-09-19T07:48:25.516 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-19T07:48:25.516 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-19T07:48:25.517 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:25.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff015663640 1 Processor -- start 2024-09-19T07:48:25.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff015663640 1 -- start start 2024-09-19T07:48:25.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff015663640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff0100c28e0 0x7ff0101545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff015663640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff01006dea0 0x7ff01006e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff015663640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0100a9e60 con 0x7ff010071100 2024-09-19T07:48:25.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff015663640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff010071b50 con 0x7ff01006dea0 2024-09-19T07:48:25.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff015663640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff0100c1ff0 con 0x7ff0100c28e0 2024-09-19T07:48:25.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff00f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff01006dea0 0x7ff01006e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff00f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff01006dea0 0x7ff01006e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60802/0 (socket says 172.21.15.83:60802) 2024-09-19T07:48:25.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff00f7fe640 1 -- 172.21.15.83:0/3560310736 learned_addr learned my addr 172.21.15.83:0/3560310736 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:25.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff007fff640 1 --2- 172.21.15.83:0/3560310736 >> v2:172.21.15.184:3300/0 conn(0x7ff0100c28e0 0x7ff0101545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff00f7fe640 1 -- 172.21.15.83:0/3560310736 >> v2:172.21.15.184:3300/0 conn(0x7ff0100c28e0 msgr2=0x7ff0101545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:25.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff00f7fe640 1 --2- 172.21.15.83:0/3560310736 >> v2:172.21.15.184:3300/0 conn(0x7ff0100c28e0 0x7ff0101545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff00f7fe640 1 -- 172.21.15.83:0/3560310736 >> v1:172.21.15.184:6789/0 conn(0x7ff010071100 legacy=0x7ff010150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:25.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.629+0000 7ff00f7fe640 1 -- 172.21.15.83:0/3560310736 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0100aa140 con 0x7ff01006dea0 2024-09-19T07:48:25.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff00f7fe640 1 --2- 172.21.15.83:0/3560310736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01006dea0 0x7ff01006e330 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7ff0100c2890 tx=0x7ff00801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6f597ebeff5cac server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:25.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff00e7fc640 1 -- 172.21.15.83:0/3560310736 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0080a56d0 con 0x7ff01006dea0 2024-09-19T07:48:25.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff00e7fc640 1 -- 172.21.15.83:0/3560310736 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0080a5870 con 0x7ff01006dea0 2024-09-19T07:48:25.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff00e7fc640 1 -- 172.21.15.83:0/3560310736 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0080a5c40 con 0x7ff01006dea0 2024-09-19T07:48:25.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 -- 172.21.15.83:0/3560310736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01006dea0 msgr2=0x7ff01006e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:25.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 --2- 172.21.15.83:0/3560310736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01006dea0 0x7ff01006e330 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7ff0100c2890 tx=0x7ff00801c920 comp rx=0 tx=0).stop 2024-09-19T07:48:25.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 -- 172.21.15.83:0/3560310736 shutdown_connections 2024-09-19T07:48:25.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 --2- 172.21.15.83:0/3560310736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01006dea0 0x7ff01006e330 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 --2- 172.21.15.83:0/3560310736 >> v2:172.21.15.184:3300/0 conn(0x7ff0100c28e0 0x7ff0101545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 -- 172.21.15.83:0/3560310736 >> 172.21.15.83:0/3560310736 conn(0x7ff01006bc80 msgr2=0x7ff01006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:25.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 -- 172.21.15.83:0/3560310736 shutdown_connections 2024-09-19T07:48:25.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 -- 172.21.15.83:0/3560310736 wait complete. 2024-09-19T07:48:25.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 Processor -- start 2024-09-19T07:48:25.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 -- start start 2024-09-19T07:48:25.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01006dea0 0x7ff010148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff010071100 0x7ff010152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0100c28e0 0x7ff0100c6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff010071660 con 0x7ff01006dea0 2024-09-19T07:48:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff0100714e0 con 0x7ff0100c28e0 2024-09-19T07:48:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff015663640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff010160c60 con 0x7ff010071100 2024-09-19T07:48:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.633+0000 7ff00effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01006dea0 0x7ff010148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff00effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01006dea0 0x7ff010148210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60818/0 (socket says 172.21.15.83:60818) 2024-09-19T07:48:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff00effd640 1 -- 172.21.15.83:0/3747086904 learned_addr learned my addr 172.21.15.83:0/3747086904 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff00f7fe640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0100c28e0 0x7ff0100c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff007fff640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff010071100 0x7ff010152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff00effd640 1 -- 172.21.15.83:0/3747086904 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff010071100 msgr2=0x7ff010152aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff00effd640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff010071100 0x7ff010152aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff00effd640 1 -- 172.21.15.83:0/3747086904 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0100c28e0 msgr2=0x7ff0100c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:25.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff00effd640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0100c28e0 0x7ff0100c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff00effd640 1 -- 172.21.15.83:0/3747086904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0101475c0 con 0x7ff01006dea0 2024-09-19T07:48:25.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff00f7fe640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0100c28e0 0x7ff0100c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:48:25.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff00effd640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01006dea0 0x7ff010148210 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7ff00000c480 tx=0x7ff00000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:25.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff0077fe640 1 -- 172.21.15.83:0/3747086904 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff000012830 con 0x7ff01006dea0 2024-09-19T07:48:25.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff0077fe640 1 -- 172.21.15.83:0/3747086904 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff000017070 con 0x7ff01006dea0 2024-09-19T07:48:25.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff015663640 1 -- 172.21.15.83:0/3747086904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0100c5640 con 0x7ff01006dea0 2024-09-19T07:48:25.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff0077fe640 1 -- 172.21.15.83:0/3747086904 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff000012c30 con 0x7ff01006dea0 2024-09-19T07:48:25.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff015663640 1 -- 172.21.15.83:0/3747086904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff010148750 con 0x7ff01006dea0 2024-09-19T07:48:25.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff015663640 1 -- 172.21.15.83:0/3747086904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff01006e330 con 0x7ff01006dea0 2024-09-19T07:48:25.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff0077fe640 1 -- 172.21.15.83:0/3747086904 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff00001c460 con 0x7ff01006dea0 2024-09-19T07:48:25.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff0077fe640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feff00678f0 0x7feff0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:25.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff0077fe640 1 -- 172.21.15.83:0/3747086904 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9991+0+0 (secure 0 0 0) 0x7ff000086bb0 con 0x7ff01006dea0 2024-09-19T07:48:25.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff007fff640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feff00678f0 0x7feff0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:25.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.637+0000 7ff007fff640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feff00678f0 0x7feff0069db0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff010158140 tx=0x7feff8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:25.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.645+0000 7ff0077fe640 1 -- 172.21.15.83:0/3747086904 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff000056610 con 0x7ff01006dea0 2024-09-19T07:48:25.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.833+0000 7ff015663640 1 -- 172.21.15.83:0/3747086904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0100c4f30 con 0x7ff01006dea0 2024-09-19T07:48:25.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.833+0000 7ff0077fe640 1 -- 172.21.15.83:0/3747086904 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+5712 (secure 0 0 0) 0x7ff000055f10 con 0x7ff01006dea0 2024-09-19T07:48:25.832 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:25.832 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":198,"btime":"2024-09-19T07:48:25:712952+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:24.716163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-19T07:48:25.833 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 198 2024-09-19T07:48:25.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.837+0000 7ff015663640 1 -- 172.21.15.83:0/3747086904 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feff00678f0 msgr2=0x7feff0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:25.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.837+0000 7ff015663640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feff00678f0 0x7feff0069db0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff010158140 tx=0x7feff8008040 comp rx=0 tx=0).stop 2024-09-19T07:48:25.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.837+0000 7ff015663640 1 -- 172.21.15.83:0/3747086904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01006dea0 msgr2=0x7ff010148210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:25.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.837+0000 7ff015663640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01006dea0 0x7ff010148210 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7ff00000c480 tx=0x7ff00000c940 comp rx=0 tx=0).stop 2024-09-19T07:48:25.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.837+0000 7ff015663640 1 -- 172.21.15.83:0/3747086904 shutdown_connections 2024-09-19T07:48:25.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.837+0000 7ff015663640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7feff00678f0 0x7feff0069db0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.837+0000 7ff015663640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0100c28e0 0x7ff0100c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.837+0000 7ff015663640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff010071100 0x7ff010152aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.837+0000 7ff015663640 1 --2- 172.21.15.83:0/3747086904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01006dea0 0x7ff010148210 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:25.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.837+0000 7ff015663640 1 -- 172.21.15.83:0/3747086904 >> 172.21.15.83:0/3747086904 conn(0x7ff01006bc80 msgr2=0x7ff01006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:25.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.837+0000 7ff015663640 1 -- 172.21.15.83:0/3747086904 shutdown_connections 2024-09-19T07:48:25.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:25.837+0000 7ff015663640 1 -- 172.21.15.83:0/3747086904 wait complete. 2024-09-19T07:48:25.851 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-19T07:48:25:712952+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 198, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 194, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6269": { "addr": "172.21.15.184:6833/2502231963", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6832", "nonce": 2502231963, "type": "v2" }, { "addr": "172.21.15.184:6833", "nonce": 2502231963, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6269, "incarnation": 191, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:48:02.799452+0000", "qdb_cluster": [ 6269 ], "qdb_leader": 6269, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6269 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 198, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8196": { "addr": "172.21.15.83:6835/3580797659", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 3580797659, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 3580797659, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8196, "incarnation": 196, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:48:24.716163+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8196 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:48:27.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:28.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.013+0000 7f57041d4640 1 Processor -- start 2024-09-19T07:48:28.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.013+0000 7f57041d4640 1 -- start start 2024-09-19T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f57041d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f56fc157630 0x7f56fc073270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f57041d4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56fc0737b0 0x7f56fc0c3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f57041d4640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f56fc0c3d90 0x7f56fc15a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f57041d4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f56fc157a10 con 0x7f56fc157630 2024-09-19T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f57041d4640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f56fc06d610 con 0x7f56fc0c3d90 2024-09-19T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f57041d4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f56fc06d8f0 con 0x7f56fc0737b0 2024-09-19T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f570176f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56fc0737b0 0x7f56fc0c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f570176f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56fc0737b0 0x7f56fc0c3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:58248/0 (socket says 172.21.15.83:58248) 2024-09-19T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f570176f640 1 -- 172.21.15.83:0/1852855292 learned_addr learned my addr 172.21.15.83:0/1852855292 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f570176f640 1 -- 172.21.15.83:0/1852855292 >> v2:172.21.15.184:3300/0 conn(0x7f56fc0c3d90 msgr2=0x7f56fc15a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f570176f640 1 --2- 172.21.15.83:0/1852855292 >> v2:172.21.15.184:3300/0 conn(0x7f56fc0c3d90 0x7f56fc15a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f570176f640 1 -- 172.21.15.83:0/1852855292 >> v2:172.21.15.83:3300/0 conn(0x7f56fc157630 msgr2=0x7f56fc073270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f570176f640 1 --2- 172.21.15.83:0/1852855292 >> v2:172.21.15.83:3300/0 conn(0x7f56fc157630 0x7f56fc073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.017+0000 7f570176f640 1 -- 172.21.15.83:0/1852855292 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56fc06dbd0 con 0x7f56fc0737b0 2024-09-19T07:48:28.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.021+0000 7f570176f640 1 --2- 172.21.15.83:0/1852855292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56fc0737b0 0x7f56fc0c3850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f56fc070210 tx=0x7f56ec01c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f99881db30b21ed server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:28.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.021+0000 7f5700f6e640 1 -- 172.21.15.83:0/1852855292 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec0ab5f0 con 0x7f56fc0737b0 2024-09-19T07:48:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.021+0000 7f5700f6e640 1 -- 172.21.15.83:0/1852855292 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec0b0070 con 0x7f56fc0737b0 2024-09-19T07:48:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.021+0000 7f5700f6e640 1 -- 172.21.15.83:0/1852855292 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec0ab9b0 con 0x7f56fc0737b0 2024-09-19T07:48:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.021+0000 7f57041d4640 1 -- 172.21.15.83:0/1852855292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56fc0737b0 msgr2=0x7f56fc0c3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.021+0000 7f57041d4640 1 --2- 172.21.15.83:0/1852855292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56fc0737b0 0x7f56fc0c3850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f56fc070210 tx=0x7f56ec01c520 comp rx=0 tx=0).stop 2024-09-19T07:48:28.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.025+0000 7f57041d4640 1 -- 172.21.15.83:0/1852855292 shutdown_connections 2024-09-19T07:48:28.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.025+0000 7f57041d4640 1 --2- 172.21.15.83:0/1852855292 >> v2:172.21.15.184:3300/0 conn(0x7f56fc0c3d90 0x7f56fc15a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:28.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.025+0000 7f57041d4640 1 --2- 172.21.15.83:0/1852855292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56fc0737b0 0x7f56fc0c3850 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:28.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.025+0000 7f57041d4640 1 --2- 172.21.15.83:0/1852855292 >> v2:172.21.15.83:3300/0 conn(0x7f56fc157630 0x7f56fc073270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:28.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.025+0000 7f57041d4640 1 -- 172.21.15.83:0/1852855292 >> 172.21.15.83:0/1852855292 conn(0x7f56fc0c4940 msgr2=0x7f56fc0c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:28.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.029+0000 7f57041d4640 1 -- 172.21.15.83:0/1852855292 shutdown_connections 2024-09-19T07:48:28.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.029+0000 7f57041d4640 1 -- 172.21.15.83:0/1852855292 wait complete. 2024-09-19T07:48:28.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.029+0000 7f57041d4640 1 Processor -- start 2024-09-19T07:48:28.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f57041d4640 1 -- start start 2024-09-19T07:48:28.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f57041d4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56fc0737b0 0x7f56fc0aad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:28.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f57041d4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56fc0c3d90 0x7f56fc1f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:28.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f57041d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56fc157630 0x7f56fc1f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:28.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f57041d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc15e1b0 con 0x7f56fc157630 2024-09-19T07:48:28.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f57041d4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc15e030 con 0x7f56fc0c3d90 2024-09-19T07:48:28.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f57041d4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f56fc15e330 con 0x7f56fc0737b0 2024-09-19T07:48:28.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f570176f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56fc0c3d90 0x7f56fc1f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:28.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f570176f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56fc0c3d90 0x7f56fc1f3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:40676/0 (socket says 172.21.15.83:40676) 2024-09-19T07:48:28.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f570176f640 1 -- 172.21.15.83:0/3391997535 learned_addr learned my addr 172.21.15.83:0/3391997535 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:28.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f5702771640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56fc157630 0x7f56fc1f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:28.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f5701f70640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56fc0737b0 0x7f56fc0aad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:28.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f5702771640 1 -- 172.21.15.83:0/3391997535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56fc0737b0 msgr2=0x7f56fc0aad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:28.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f5702771640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56fc0737b0 0x7f56fc0aad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:28.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f5702771640 1 -- 172.21.15.83:0/3391997535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56fc0c3d90 msgr2=0x7f56fc1f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:28.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f5702771640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56fc0c3d90 0x7f56fc1f3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:28.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f5702771640 1 -- 172.21.15.83:0/3391997535 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56fc1f3db0 con 0x7f56fc157630 2024-09-19T07:48:28.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f5702771640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56fc157630 0x7f56fc1f3870 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f56f400be20 tx=0x7f56f400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:28.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f56f27fc640 1 -- 172.21.15.83:0/3391997535 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4019070 con 0x7f56fc157630 2024-09-19T07:48:28.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f56f27fc640 1 -- 172.21.15.83:0/3391997535 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f400ce30 con 0x7f56fc157630 2024-09-19T07:48:28.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f56f27fc640 1 -- 172.21.15.83:0/3391997535 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4014660 con 0x7f56fc157630 2024-09-19T07:48:28.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f57041d4640 1 -- 172.21.15.83:0/3391997535 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56fc14dfd0 con 0x7f56fc157630 2024-09-19T07:48:28.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.033+0000 7f57041d4640 1 -- 172.21.15.83:0/3391997535 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56fc14e590 con 0x7f56fc157630 2024-09-19T07:48:28.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.037+0000 7f57041d4640 1 -- 172.21.15.83:0/3391997535 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56fc0688e0 con 0x7f56fc157630 2024-09-19T07:48:28.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.041+0000 7f56f27fc640 1 -- 172.21.15.83:0/3391997535 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f56f401c070 con 0x7f56fc157630 2024-09-19T07:48:28.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.041+0000 7f56f27fc640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f56e00678c0 0x7f56e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:28.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.041+0000 7f56f27fc640 1 -- 172.21.15.83:0/3391997535 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9991+0+0 (secure 0 0 0) 0x7f56f4087050 con 0x7f56fc157630 2024-09-19T07:48:28.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.041+0000 7f5701f70640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f56e00678c0 0x7f56e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:28.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.041+0000 7f5701f70640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f56e00678c0 0x7f56e0069d80 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009990 tx=0x7f56f8009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:28.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.045+0000 7f56f27fc640 1 -- 172.21.15.83:0/3391997535 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f56f4023020 con 0x7f56fc157630 2024-09-19T07:48:28.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.349+0000 7f57041d4640 1 -- 172.21.15.83:0/3391997535 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56fc15a220 con 0x7f56fc157630 2024-09-19T07:48:28.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.349+0000 7f56f27fc640 1 -- 172.21.15.83:0/3391997535 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) ==== 95+0+5719 (secure 0 0 0) 0x7f56f40564e0 con 0x7f56fc157630 2024-09-19T07:48:28.350 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:28.350 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":199,"btime":"2024-09-19T07:48:26:723719+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:48:28.350 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 199 2024-09-19T07:48:28.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.353+0000 7f57041d4640 1 -- 172.21.15.83:0/3391997535 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f56e00678c0 msgr2=0x7f56e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:28.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.353+0000 7f57041d4640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f56e00678c0 0x7f56e0069d80 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009990 tx=0x7f56f8009110 comp rx=0 tx=0).stop 2024-09-19T07:48:28.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.353+0000 7f57041d4640 1 -- 172.21.15.83:0/3391997535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56fc157630 msgr2=0x7f56fc1f3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:28.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.353+0000 7f57041d4640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56fc157630 0x7f56fc1f3870 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f56f400be20 tx=0x7f56f400bf10 comp rx=0 tx=0).stop 2024-09-19T07:48:28.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.353+0000 7f57041d4640 1 -- 172.21.15.83:0/3391997535 shutdown_connections 2024-09-19T07:48:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.353+0000 7f57041d4640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f56e00678c0 0x7f56e0069d80 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.353+0000 7f57041d4640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56fc157630 0x7f56fc1f3870 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.353+0000 7f57041d4640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56fc0c3d90 0x7f56fc1f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.353+0000 7f57041d4640 1 --2- 172.21.15.83:0/3391997535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56fc0737b0 0x7f56fc0aad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.353+0000 7f57041d4640 1 -- 172.21.15.83:0/3391997535 >> 172.21.15.83:0/3391997535 conn(0x7f56fc0c4940 msgr2=0x7f56fc163cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.353+0000 7f57041d4640 1 -- 172.21.15.83:0/3391997535 shutdown_connections 2024-09-19T07:48:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:28.353+0000 7f57041d4640 1 -- 172.21.15.83:0/3391997535 wait complete. 2024-09-19T07:48:28.381 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=8028 2024-09-19T07:48:28.381 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.c 2024-09-19T07:48:29.382 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-19T07:48:29.383 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-19T07:48:29.383 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-19T07:48:29.385 INFO:tasks.ceph.mds.c:Started 2024-09-19T07:48:29.385 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:29.479 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-09-19T07:48:29.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.549+0000 7fa5c7624640 1 Processor -- start 2024-09-19T07:48:29.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.549+0000 7fa5c7624640 1 -- start start 2024-09-19T07:48:29.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.553+0000 7fa5c7624640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5c0072120 0x7fa5c00c2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:29.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.553+0000 7fa5c7624640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa5c0159300 0x7fa5c01597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:29.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.553+0000 7fa5c7624640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5c0159eb0 con 0x7fa5c014ff50 2024-09-19T07:48:29.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.553+0000 7fa5c7624640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa5c00bf0b0 con 0x7fa5c0159300 2024-09-19T07:48:29.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.553+0000 7fa5c7624640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c00beaf0 con 0x7fa5c0072120 2024-09-19T07:48:29.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.553+0000 7fa5c5bc1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa5c0159300 0x7fa5c01597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:29.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.553+0000 7fa5c53c0640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fa5c014ff50 0x7fa5c00bdf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:40328/0 (socket says 172.21.15.83:40328) 2024-09-19T07:48:29.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.553+0000 7fa5c53c0640 1 -- 172.21.15.83:0/1126706964 learned_addr learned my addr 172.21.15.83:0/1126706964 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:29.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.553+0000 7fa5c5bc1640 1 -- 172.21.15.83:0/1126706964 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5c0072120 msgr2=0x7fa5c00c2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:29.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.553+0000 7fa5c5bc1640 1 --2- 172.21.15.83:0/1126706964 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5c0072120 0x7fa5c00c2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:29.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.553+0000 7fa5c5bc1640 1 -- 172.21.15.83:0/1126706964 >> v1:172.21.15.184:6789/0 conn(0x7fa5c014ff50 legacy=0x7fa5c00bdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:29.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.553+0000 7fa5c5bc1640 1 -- 172.21.15.83:0/1126706964 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c00bed40 con 0x7fa5c0159300 2024-09-19T07:48:29.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.557+0000 7fa5c5bc1640 1 --2- 172.21.15.83:0/1126706964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c0159300 0x7fa5c01597b0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009d90 tx=0x7fa5b801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5f4cc123d600cc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:29.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.557+0000 7fa5b7fff640 1 -- 172.21.15.83:0/1126706964 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b80b2070 con 0x7fa5c0159300 2024-09-19T07:48:29.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.557+0000 7fa5b7fff640 1 -- 172.21.15.83:0/1126706964 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b80a58d0 con 0x7fa5c0159300 2024-09-19T07:48:29.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.557+0000 7fa5b7fff640 1 -- 172.21.15.83:0/1126706964 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b80a5ca0 con 0x7fa5c0159300 2024-09-19T07:48:29.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 -- 172.21.15.83:0/1126706964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c0159300 msgr2=0x7fa5c01597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:29.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 --2- 172.21.15.83:0/1126706964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c0159300 0x7fa5c01597b0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009d90 tx=0x7fa5b801c900 comp rx=0 tx=0).stop 2024-09-19T07:48:29.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 -- 172.21.15.83:0/1126706964 shutdown_connections 2024-09-19T07:48:29.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 --2- 172.21.15.83:0/1126706964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c0159300 0x7fa5c01597b0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:29.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 --2- 172.21.15.83:0/1126706964 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5c0072120 0x7fa5c00c2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:29.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 -- 172.21.15.83:0/1126706964 >> 172.21.15.83:0/1126706964 conn(0x7fa5c00c7e40 msgr2=0x7fa5c00c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:29.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 -- 172.21.15.83:0/1126706964 shutdown_connections 2024-09-19T07:48:29.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 -- 172.21.15.83:0/1126706964 wait complete. 2024-09-19T07:48:29.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 Processor -- start 2024-09-19T07:48:29.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 -- start start 2024-09-19T07:48:29.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5c0072120 0x7fa5c014e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:29.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5c014ff50 0x7fa5c0148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:29.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c0159300 0x7fa5c01492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:29.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0163030 con 0x7fa5c0159300 2024-09-19T07:48:29.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0162eb0 con 0x7fa5c014ff50 2024-09-19T07:48:29.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c01631b0 con 0x7fa5c0072120 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c5bc1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c0159300 0x7fa5c01492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c5bc1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c0159300 0x7fa5c01492d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60874/0 (socket says 172.21.15.83:60874) 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c5bc1640 1 -- 172.21.15.83:0/2140084467 learned_addr learned my addr 172.21.15.83:0/2140084467 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c53c0640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5c0072120 0x7fa5c014e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c5bc1640 1 -- 172.21.15.83:0/2140084467 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5c0072120 msgr2=0x7fa5c014e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c5bc1640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5c0072120 0x7fa5c014e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c5bc1640 1 -- 172.21.15.83:0/2140084467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5c014ff50 msgr2=0x7fa5c0148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c5bc1640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5c014ff50 0x7fa5c0148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c5bc1640 1 -- 172.21.15.83:0/2140084467 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c00aa4a0 con 0x7fa5c0159300 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c5bc1640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c0159300 0x7fa5c01492d0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009d90 tx=0x7fa5b80196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5b5ffb640 1 -- 172.21.15.83:0/2140084467 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b80bb070 con 0x7fa5c0159300 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5b5ffb640 1 -- 172.21.15.83:0/2140084467 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b80ad700 con 0x7fa5c0159300 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5c7624640 1 -- 172.21.15.83:0/2140084467 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c00ab950 con 0x7fa5c0159300 2024-09-19T07:48:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.561+0000 7fa5b5ffb640 1 -- 172.21.15.83:0/2140084467 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b80b2040 con 0x7fa5c0159300 2024-09-19T07:48:29.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.565+0000 7fa5c7624640 1 -- 172.21.15.83:0/2140084467 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c00abf10 con 0x7fa5c0159300 2024-09-19T07:48:29.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.565+0000 7fa5b5ffb640 1 -- 172.21.15.83:0/2140084467 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa5b80b7410 con 0x7fa5c0159300 2024-09-19T07:48:29.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.565+0000 7fa5b5ffb640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa5a40678c0 0x7fa5a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:29.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.565+0000 7fa5b5ffb640 1 -- 172.21.15.83:0/2140084467 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9991+0+0 (secure 0 0 0) 0x7fa5b8121150 con 0x7fa5c0159300 2024-09-19T07:48:29.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.565+0000 7fa5c7624640 1 -- 172.21.15.83:0/2140084467 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5c00c2840 con 0x7fa5c0159300 2024-09-19T07:48:29.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.569+0000 7fa5c53c0640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa5a40678c0 0x7fa5a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:29.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.569+0000 7fa5c53c0640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa5a40678c0 0x7fa5a4069d80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc005890 tx=0x7fa5bc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:29.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.577+0000 7fa5b5ffb640 1 -- 172.21.15.83:0/2140084467 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa5b80c4020 con 0x7fa5c0159300 2024-09-19T07:48:29.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.873+0000 7fa5c7624640 1 -- 172.21.15.83:0/2140084467 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5c00bd010 con 0x7fa5c0159300 2024-09-19T07:48:29.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.873+0000 7fa5b5ffb640 1 -- 172.21.15.83:0/2140084467 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) ==== 95+0+6545 (secure 0 0 0) 0x7fa5b8016580 con 0x7fa5c0159300 2024-09-19T07:48:29.872 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:29.873 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":200,"btime":"2024-09-19T07:48:29:736412+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:48:29.873 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 200 2024-09-19T07:48:29.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.877+0000 7fa5c7624640 1 -- 172.21.15.83:0/2140084467 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa5a40678c0 msgr2=0x7fa5a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:29.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.877+0000 7fa5c7624640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa5a40678c0 0x7fa5a4069d80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc005890 tx=0x7fa5bc009040 comp rx=0 tx=0).stop 2024-09-19T07:48:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.877+0000 7fa5c7624640 1 -- 172.21.15.83:0/2140084467 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c0159300 msgr2=0x7fa5c01492d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.877+0000 7fa5c7624640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c0159300 0x7fa5c01492d0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009d90 tx=0x7fa5b80196a0 comp rx=0 tx=0).stop 2024-09-19T07:48:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.877+0000 7fa5c7624640 1 -- 172.21.15.83:0/2140084467 shutdown_connections 2024-09-19T07:48:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.877+0000 7fa5c7624640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa5a40678c0 0x7fa5a4069d80 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.877+0000 7fa5c7624640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c0159300 0x7fa5c01492d0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.877+0000 7fa5c7624640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5c014ff50 0x7fa5c0148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.877+0000 7fa5c7624640 1 --2- 172.21.15.83:0/2140084467 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5c0072120 0x7fa5c014e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.877+0000 7fa5c7624640 1 -- 172.21.15.83:0/2140084467 >> 172.21.15.83:0/2140084467 conn(0x7fa5c00c7e40 msgr2=0x7fa5c00bdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.877+0000 7fa5c7624640 1 -- 172.21.15.83:0/2140084467 shutdown_connections 2024-09-19T07:48:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:29.877+0000 7fa5c7624640 1 -- 172.21.15.83:0/2140084467 wait complete. 2024-09-19T07:48:29.896 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-19T07:48:29.896 INFO:tasks.mds_thrash.fs.[b]:waiting for 37 secs before thrashing 2024-09-19T07:48:31.676 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-09-19T07:48:31.677+0000 7f14173d9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:48:42.035 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:48:42.038 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:48:43.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:43.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e43ee640 1 Processor -- start 2024-09-19T07:48:43.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e43ee640 1 -- start start 2024-09-19T07:48:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e43ee640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f44dc150290 0x7f44dc0c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e43ee640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f44dc152b20 0x7f44dc0c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e43ee640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44dc150670 con 0x7f44dc0c45c0 2024-09-19T07:48:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e43ee640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f44dc0716c0 con 0x7f44dc150290 2024-09-19T07:48:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e43ee640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f44dc0c3920 con 0x7f44dc152b20 2024-09-19T07:48:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e298b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f44dc0c45c0 0x7f44dc153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57576/0 (socket says 172.21.15.83:57576) 2024-09-19T07:48:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e298b640 1 -- 172.21.15.83:0/1342423541 learned_addr learned my addr 172.21.15.83:0/1342423541 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e218a640 1 --2- 172.21.15.83:0/1342423541 >> v2:172.21.15.83:3300/0 conn(0x7f44dc150290 0x7f44dc0c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e1188640 1 -- 172.21.15.83:0/1342423541 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 873992404 0 0) 0x7f44dc150670 con 0x7f44dc0c45c0 2024-09-19T07:48:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e1188640 1 -- 172.21.15.83:0/1342423541 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44c8005740 con 0x7f44dc0c45c0 2024-09-19T07:48:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e218a640 1 -- 172.21.15.83:0/1342423541 >> v2:172.21.15.184:3300/0 conn(0x7f44dc152b20 msgr2=0x7f44dc0c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e218a640 1 --2- 172.21.15.83:0/1342423541 >> v2:172.21.15.184:3300/0 conn(0x7f44dc152b20 0x7f44dc0c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:43.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e218a640 1 -- 172.21.15.83:0/1342423541 >> v1:172.21.15.83:6789/0 conn(0x7f44dc0c45c0 legacy=0x7f44dc153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:43.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.981+0000 7f44e218a640 1 -- 172.21.15.83:0/1342423541 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44dc0c2880 con 0x7f44dc150290 2024-09-19T07:48:43.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.985+0000 7f44e218a640 1 --2- 172.21.15.83:0/1342423541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44dc150290 0x7f44dc0c1ff0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f44dc070820 tx=0x7f44d801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69a37b7d31a5dbf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:43.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.985+0000 7f44e1188640 1 -- 172.21.15.83:0/1342423541 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d80b0070 con 0x7f44dc150290 2024-09-19T07:48:43.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.985+0000 7f44e1188640 1 -- 172.21.15.83:0/1342423541 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44d80ab600 con 0x7f44dc150290 2024-09-19T07:48:43.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.985+0000 7f44e1188640 1 -- 172.21.15.83:0/1342423541 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d80ab9d0 con 0x7f44dc150290 2024-09-19T07:48:43.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.985+0000 7f44e43ee640 1 -- 172.21.15.83:0/1342423541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44dc150290 msgr2=0x7f44dc0c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:43.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.985+0000 7f44e43ee640 1 --2- 172.21.15.83:0/1342423541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44dc150290 0x7f44dc0c1ff0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f44dc070820 tx=0x7f44d801c460 comp rx=0 tx=0).stop 2024-09-19T07:48:43.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.985+0000 7f44e43ee640 1 -- 172.21.15.83:0/1342423541 shutdown_connections 2024-09-19T07:48:43.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.985+0000 7f44e43ee640 1 --2- 172.21.15.83:0/1342423541 >> v2:172.21.15.184:3300/0 conn(0x7f44dc152b20 0x7f44dc0c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:43.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.985+0000 7f44e43ee640 1 --2- 172.21.15.83:0/1342423541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44dc150290 0x7f44dc0c1ff0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.985+0000 7f44e43ee640 1 -- 172.21.15.83:0/1342423541 >> 172.21.15.83:0/1342423541 conn(0x7f44dc06bc80 msgr2=0x7f44dc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.985+0000 7f44e43ee640 1 -- 172.21.15.83:0/1342423541 shutdown_connections 2024-09-19T07:48:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.985+0000 7f44e43ee640 1 -- 172.21.15.83:0/1342423541 wait complete. 2024-09-19T07:48:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.989+0000 7f44e43ee640 1 Processor -- start 2024-09-19T07:48:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.989+0000 7f44e43ee640 1 -- start start 2024-09-19T07:48:43.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.989+0000 7f44e43ee640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44dc0c45c0 0x7f44dc0ab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:43.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.989+0000 7f44e43ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44dc150290 0x7f44dc14f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:43.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.989+0000 7f44e43ee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44dc152b20 0x7f44dc1f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:43.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.989+0000 7f44e43ee640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc0c2530 con 0x7f44dc0c45c0 2024-09-19T07:48:43.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.989+0000 7f44e43ee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc0c3120 con 0x7f44dc150290 2024-09-19T07:48:43.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.989+0000 7f44e43ee640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f44dc06e6b0 con 0x7f44dc152b20 2024-09-19T07:48:43.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.989+0000 7f44e1989640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44dc150290 0x7f44dc14f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:43.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.989+0000 7f44e1989640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44dc150290 0x7f44dc14f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.83:48170/0 (socket says 172.21.15.83:48170) 2024-09-19T07:48:43.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.989+0000 7f44e1989640 1 -- 172.21.15.83:0/3620379269 learned_addr learned my addr 172.21.15.83:0/3620379269 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:43.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.989+0000 7f44e298b640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44dc152b20 0x7f44dc1f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:43.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44e298b640 1 -- 172.21.15.83:0/3620379269 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44dc150290 msgr2=0x7f44dc14f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:43.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44e298b640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44dc150290 0x7f44dc14f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:43.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44e298b640 1 -- 172.21.15.83:0/3620379269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44dc0c45c0 msgr2=0x7f44dc0ab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:43.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44e298b640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44dc0c45c0 0x7f44dc0ab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:43.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44e298b640 1 -- 172.21.15.83:0/3620379269 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44dc072320 con 0x7f44dc152b20 2024-09-19T07:48:43.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44e298b640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44dc152b20 0x7f44dc1f2100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f44d400c700 tx=0x7f44d400cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:43.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44d2ffd640 1 -- 172.21.15.83:0/3620379269 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4018070 con 0x7f44dc152b20 2024-09-19T07:48:43.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44d2ffd640 1 -- 172.21.15.83:0/3620379269 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44d400ce10 con 0x7f44dc152b20 2024-09-19T07:48:43.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44d2ffd640 1 -- 172.21.15.83:0/3620379269 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d400a820 con 0x7f44dc152b20 2024-09-19T07:48:43.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44e43ee640 1 -- 172.21.15.83:0/3620379269 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44dc0ac5c0 con 0x7f44dc152b20 2024-09-19T07:48:43.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44e43ee640 1 -- 172.21.15.83:0/3620379269 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f44dc0ac2c0 con 0x7f44dc152b20 2024-09-19T07:48:43.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44e43ee640 1 -- 172.21.15.83:0/3620379269 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44dc06ea70 con 0x7f44dc152b20 2024-09-19T07:48:43.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44d2ffd640 1 -- 172.21.15.83:0/3620379269 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f44d400a9c0 con 0x7f44dc152b20 2024-09-19T07:48:43.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.993+0000 7f44d2ffd640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44c00678c0 0x7f44c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:43.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.997+0000 7f44d2ffd640 1 -- 172.21.15.83:0/3620379269 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 9991+0+0 (secure 0 0 0) 0x7f44d40861c0 con 0x7f44dc152b20 2024-09-19T07:48:43.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.997+0000 7f44e218a640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44c00678c0 0x7f44c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:43.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:43.997+0000 7f44e218a640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44c00678c0 0x7f44c0069d80 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f44d80ad000 tx=0x7f44d8004740 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:43.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.001+0000 7f44d2ffd640 1 -- 172.21.15.83:0/3620379269 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f44d4055d00 con 0x7f44dc152b20 2024-09-19T07:48:44.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.265+0000 7f44e43ee640 1 -- 172.21.15.83:0/3620379269 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44dc1536d0 con 0x7f44dc152b20 2024-09-19T07:48:44.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.265+0000 7f44d2ffd640 1 -- 172.21.15.83:0/3620379269 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) ==== 95+0+6545 (secure 0 0 0) 0x7f44d4055600 con 0x7f44dc152b20 2024-09-19T07:48:44.265 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:44.266 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":200,"btime":"2024-09-19T07:48:29:736412+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:48:44.266 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 200 2024-09-19T07:48:44.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.269+0000 7f44e43ee640 1 -- 172.21.15.83:0/3620379269 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44c00678c0 msgr2=0x7f44c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:44.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.269+0000 7f44e43ee640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44c00678c0 0x7f44c0069d80 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f44d80ad000 tx=0x7f44d8004740 comp rx=0 tx=0).stop 2024-09-19T07:48:44.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.269+0000 7f44e43ee640 1 -- 172.21.15.83:0/3620379269 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44dc152b20 msgr2=0x7f44dc1f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:44.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.269+0000 7f44e43ee640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44dc152b20 0x7f44dc1f2100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f44d400c700 tx=0x7f44d400cbc0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.273+0000 7f44e43ee640 1 -- 172.21.15.83:0/3620379269 shutdown_connections 2024-09-19T07:48:44.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.273+0000 7f44e43ee640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f44c00678c0 0x7f44c0069d80 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.273+0000 7f44e43ee640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44dc152b20 0x7f44dc1f2100 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.273+0000 7f44e43ee640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44dc150290 0x7f44dc14f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.273+0000 7f44e43ee640 1 --2- 172.21.15.83:0/3620379269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44dc0c45c0 0x7f44dc0ab6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.273+0000 7f44e43ee640 1 -- 172.21.15.83:0/3620379269 >> 172.21.15.83:0/3620379269 conn(0x7f44dc06bc80 msgr2=0x7f44dc14ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:44.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.273+0000 7f44e43ee640 1 -- 172.21.15.83:0/3620379269 shutdown_connections 2024-09-19T07:48:44.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.273+0000 7f44e43ee640 1 -- 172.21.15.83:0/3620379269 wait complete. 2024-09-19T07:48:44.292 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-19T07:48:44.292 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-19T07:48:44.292 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:48:44.316 INFO:tasks.ceph.mds.d:Stopped 2024-09-19T07:48:44.317 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:44.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.461+0000 7f8a4ed63640 1 Processor -- start 2024-09-19T07:48:44.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.461+0000 7f8a4ed63640 1 -- start start 2024-09-19T07:48:44.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.461+0000 7f8a4ed63640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f8a480a7ad0 0x7f8a480c2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.461+0000 7f8a4ed63640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a480c7160 0x7f8a480c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.461+0000 7f8a4ed63640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8a4815dcd0 0x7f8a4815e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.461+0000 7f8a4ed63640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f8a480c3f70 con 0x7f8a4815dcd0 2024-09-19T07:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.461+0000 7f8a4ed63640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f8a480c4250 con 0x7f8a480a7ad0 2024-09-19T07:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.461+0000 7f8a4ed63640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8a480c6960 con 0x7f8a480c7160 2024-09-19T07:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.461+0000 7f8a4d300640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8a4815dcd0 0x7f8a4815e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.461+0000 7f8a4d300640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8a4815dcd0 0x7f8a4815e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48174/0 (socket says 172.21.15.83:48174) 2024-09-19T07:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.461+0000 7f8a47fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a480c7160 0x7f8a480c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.461+0000 7f8a4d300640 1 -- 172.21.15.83:0/3310609768 learned_addr learned my addr 172.21.15.83:0/3310609768 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:44.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.465+0000 7f8a4d300640 1 -- 172.21.15.83:0/3310609768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a480c7160 msgr2=0x7f8a480c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:44.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.465+0000 7f8a4d300640 1 --2- 172.21.15.83:0/3310609768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a480c7160 0x7f8a480c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.465+0000 7f8a4d300640 1 -- 172.21.15.83:0/3310609768 >> v2:172.21.15.184:3300/0 conn(0x7f8a480a7ad0 msgr2=0x7f8a480c2850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-19T07:48:44.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.465+0000 7f8a4d300640 1 --2- 172.21.15.83:0/3310609768 >> v2:172.21.15.184:3300/0 conn(0x7f8a480a7ad0 0x7f8a480c2850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.465+0000 7f8a4d300640 1 -- 172.21.15.83:0/3310609768 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a480c6c40 con 0x7f8a4815dcd0 2024-09-19T07:48:44.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.469+0000 7f8a4d300640 1 --2- 172.21.15.83:0/3310609768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4815dcd0 0x7f8a4815e100 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f8a480c57e0 tx=0x7f8a4001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeec8487183224a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:44.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.469+0000 7f8a477fe640 1 -- 172.21.15.83:0/3310609768 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a400a56e0 con 0x7f8a4815dcd0 2024-09-19T07:48:44.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.469+0000 7f8a477fe640 1 -- 172.21.15.83:0/3310609768 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a400a5880 con 0x7f8a4815dcd0 2024-09-19T07:48:44.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.469+0000 7f8a477fe640 1 -- 172.21.15.83:0/3310609768 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a400a5c50 con 0x7f8a4815dcd0 2024-09-19T07:48:44.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.469+0000 7f8a4ed63640 1 -- 172.21.15.83:0/3310609768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4815dcd0 msgr2=0x7f8a4815e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:44.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.469+0000 7f8a4ed63640 1 --2- 172.21.15.83:0/3310609768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4815dcd0 0x7f8a4815e100 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f8a480c57e0 tx=0x7f8a4001ca30 comp rx=0 tx=0).stop 2024-09-19T07:48:44.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.469+0000 7f8a4ed63640 1 -- 172.21.15.83:0/3310609768 shutdown_connections 2024-09-19T07:48:44.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.469+0000 7f8a4ed63640 1 --2- 172.21.15.83:0/3310609768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a4815dcd0 0x7f8a4815e100 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.469+0000 7f8a4ed63640 1 --2- 172.21.15.83:0/3310609768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a480c7160 0x7f8a480c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.469+0000 7f8a4ed63640 1 --2- 172.21.15.83:0/3310609768 >> v2:172.21.15.184:3300/0 conn(0x7f8a480a7ad0 0x7f8a480c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.469+0000 7f8a4ed63640 1 -- 172.21.15.83:0/3310609768 >> 172.21.15.83:0/3310609768 conn(0x7f8a480aa910 msgr2=0x7f8a480c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:44.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.473+0000 7f8a4ed63640 1 -- 172.21.15.83:0/3310609768 shutdown_connections 2024-09-19T07:48:44.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.473+0000 7f8a4ed63640 1 -- 172.21.15.83:0/3310609768 wait complete. 2024-09-19T07:48:44.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4ed63640 1 Processor -- start 2024-09-19T07:48:44.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4ed63640 1 -- start start 2024-09-19T07:48:44.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4ed63640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a480a7ad0 0x7f8a480acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:44.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4ed63640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a480c7160 0x7f8a480c2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:44.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4ed63640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4815dcd0 0x7f8a480c1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:44.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4ed63640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8a480c3120 con 0x7f8a480a7ad0 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4ed63640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a480c4530 con 0x7f8a4815dcd0 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4ed63640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8a480c32a0 con 0x7f8a480c7160 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4caff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a480a7ad0 0x7f8a480acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4caff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a480a7ad0 0x7f8a480acdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48180/0 (socket says 172.21.15.83:48180) 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4caff640 1 -- 172.21.15.83:0/2838928389 learned_addr learned my addr 172.21.15.83:0/2838928389 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4d300640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4815dcd0 0x7f8a480c1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a47fff640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a480c7160 0x7f8a480c2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4caff640 1 -- 172.21.15.83:0/2838928389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a480c7160 msgr2=0x7f8a480c2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4caff640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a480c7160 0x7f8a480c2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4caff640 1 -- 172.21.15.83:0/2838928389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4815dcd0 msgr2=0x7f8a480c1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4caff640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4815dcd0 0x7f8a480c1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4caff640 1 -- 172.21.15.83:0/2838928389 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a481577e0 con 0x7f8a480a7ad0 2024-09-19T07:48:44.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4d300640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4815dcd0 0x7f8a480c1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:48:44.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4caff640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a480a7ad0 0x7f8a480acdb0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400b3e0 tx=0x7f8a3400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:44.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a457fa640 1 -- 172.21.15.83:0/2838928389 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34015020 con 0x7f8a480a7ad0 2024-09-19T07:48:44.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a457fa640 1 -- 172.21.15.83:0/2838928389 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a34011850 con 0x7f8a480a7ad0 2024-09-19T07:48:44.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4ed63640 1 -- 172.21.15.83:0/2838928389 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a48073f20 con 0x7f8a480a7ad0 2024-09-19T07:48:44.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a457fa640 1 -- 172.21.15.83:0/2838928389 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34011c40 con 0x7f8a480a7ad0 2024-09-19T07:48:44.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.477+0000 7f8a4ed63640 1 -- 172.21.15.83:0/2838928389 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a480ad2f0 con 0x7f8a480a7ad0 2024-09-19T07:48:44.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.481+0000 7f8a4ed63640 1 -- 172.21.15.83:0/2838928389 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a280052a0 con 0x7f8a480a7ad0 2024-09-19T07:48:44.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.481+0000 7f8a457fa640 1 -- 172.21.15.83:0/2838928389 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8a34018020 con 0x7f8a480a7ad0 2024-09-19T07:48:44.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.481+0000 7f8a457fa640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8a2c0678c0 0x7f8a2c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:44.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.481+0000 7f8a457fa640 1 -- 172.21.15.83:0/2838928389 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9991+0+0 (secure 0 0 0) 0x7f8a34086c50 con 0x7f8a480a7ad0 2024-09-19T07:48:44.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.481+0000 7f8a47fff640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8a2c0678c0 0x7f8a2c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:44.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.481+0000 7f8a47fff640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8a2c0678c0 0x7f8a2c069d80 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8a4815d990 tx=0x7f8a3c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:44.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.489+0000 7f8a457fa640 1 -- 172.21.15.83:0/2838928389 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f8a3408c020 con 0x7f8a480a7ad0 2024-09-19T07:48:44.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 -- 172.21.15.83:0/2838928389 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a28005590 con 0x7f8a480a7ad0 2024-09-19T07:48:44.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a457fa640 1 -- 172.21.15.83:0/2838928389 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) ==== 95+0+6545 (secure 0 0 0) 0x7f8a3401f020 con 0x7f8a480a7ad0 2024-09-19T07:48:44.759 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:44.759 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":200,"btime":"2024-09-19T07:48:29:736412+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:48:44.760 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 200 2024-09-19T07:48:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 -- 172.21.15.83:0/2838928389 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8a2c0678c0 msgr2=0x7f8a2c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8a2c0678c0 0x7f8a2c069d80 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8a4815d990 tx=0x7f8a3c008040 comp rx=0 tx=0).stop 2024-09-19T07:48:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 -- 172.21.15.83:0/2838928389 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a480a7ad0 msgr2=0x7f8a480acdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a480a7ad0 0x7f8a480acdb0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400b3e0 tx=0x7f8a3400b8a0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 -- 172.21.15.83:0/2838928389 shutdown_connections 2024-09-19T07:48:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f8a2c0678c0 0x7f8a2c069d80 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4815dcd0 0x7f8a480c1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a480c7160 0x7f8a480c2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 --2- 172.21.15.83:0/2838928389 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a480a7ad0 0x7f8a480acdb0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 -- 172.21.15.83:0/2838928389 >> 172.21.15.83:0/2838928389 conn(0x7f8a480aa910 msgr2=0x7f8a48158ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 -- 172.21.15.83:0/2838928389 shutdown_connections 2024-09-19T07:48:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:44.761+0000 7f8a4ed63640 1 -- 172.21.15.83:0/2838928389 wait complete. 2024-09-19T07:48:44.785 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:48:46.786 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:46.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.941+0000 7f8581a45640 1 Processor -- start 2024-09-19T07:48:46.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.941+0000 7f8581a45640 1 -- start start 2024-09-19T07:48:46.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.941+0000 7f8581a45640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f857c14dcc0 0x7f857c1570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:46.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.941+0000 7f8581a45640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f857c06f2f0 0x7f857c154410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:46.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.941+0000 7f8581a45640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f857c0c57d0 con 0x7f857c06e460 2024-09-19T07:48:46.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.941+0000 7f8581a45640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f857c153f80 con 0x7f857c14dcc0 2024-09-19T07:48:46.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.941+0000 7f8581a45640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f857c0c5ab0 con 0x7f857c06f2f0 2024-09-19T07:48:46.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.941+0000 7f857affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f857c14dcc0 0x7f857c1570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:46.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.941+0000 7f857b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f857c06e460 0x7f857c0c60a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57578/0 (socket says 172.21.15.83:57578) 2024-09-19T07:48:46.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.941+0000 7f857b7fe640 1 -- 172.21.15.83:0/245037828 learned_addr learned my addr 172.21.15.83:0/245037828 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:46.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.941+0000 7f857a7fc640 1 --2- 172.21.15.83:0/245037828 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f857c06f2f0 0x7f857c154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:46.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f8579ffb640 1 -- 172.21.15.83:0/245037828 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1824561381 0 0) 0x7f857c0c57d0 con 0x7f857c06e460 2024-09-19T07:48:46.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f857affd640 1 -- 172.21.15.83:0/245037828 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f857c06f2f0 msgr2=0x7f857c154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:46.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f857affd640 1 --2- 172.21.15.83:0/245037828 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f857c06f2f0 0x7f857c154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:46.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f857affd640 1 -- 172.21.15.83:0/245037828 >> v1:172.21.15.83:6789/0 conn(0x7f857c06e460 legacy=0x7f857c0c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:46.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f857affd640 1 -- 172.21.15.83:0/245037828 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857c0c5d90 con 0x7f857c14dcc0 2024-09-19T07:48:46.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f857affd640 1 --2- 172.21.15.83:0/245037828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f857c14dcc0 0x7f857c1570f0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f85740097c0 tx=0x7f857401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd97570e7f7331f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:46.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f8579ffb640 1 -- 172.21.15.83:0/245037828 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85740aa620 con 0x7f857c14dcc0 2024-09-19T07:48:46.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f8579ffb640 1 -- 172.21.15.83:0/245037828 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85740af070 con 0x7f857c14dcc0 2024-09-19T07:48:46.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f8579ffb640 1 -- 172.21.15.83:0/245037828 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85740aa9e0 con 0x7f857c14dcc0 2024-09-19T07:48:46.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f8581a45640 1 -- 172.21.15.83:0/245037828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f857c14dcc0 msgr2=0x7f857c1570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:46.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f8581a45640 1 --2- 172.21.15.83:0/245037828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f857c14dcc0 0x7f857c1570f0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f85740097c0 tx=0x7f857401c4a0 comp rx=0 tx=0).stop 2024-09-19T07:48:46.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f8581a45640 1 -- 172.21.15.83:0/245037828 shutdown_connections 2024-09-19T07:48:46.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f8581a45640 1 --2- 172.21.15.83:0/245037828 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f857c06f2f0 0x7f857c154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:46.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f8581a45640 1 --2- 172.21.15.83:0/245037828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f857c14dcc0 0x7f857c1570f0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:46.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f8581a45640 1 -- 172.21.15.83:0/245037828 >> 172.21.15.83:0/245037828 conn(0x7f857c06ffc0 msgr2=0x7f857c0703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:46.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f8581a45640 1 -- 172.21.15.83:0/245037828 shutdown_connections 2024-09-19T07:48:46.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.945+0000 7f8581a45640 1 -- 172.21.15.83:0/245037828 wait complete. 2024-09-19T07:48:46.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8581a45640 1 Processor -- start 2024-09-19T07:48:46.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8581a45640 1 -- start start 2024-09-19T07:48:46.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8581a45640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f857c06e460 0x7f857c14cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:46.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8581a45640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f857c06f2f0 0x7f857c14b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:46.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8581a45640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f857c14dcc0 0x7f857c14aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8581a45640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f857c0c5520 con 0x7f857c06e460 2024-09-19T07:48:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8581a45640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f857c0c53a0 con 0x7f857c06f2f0 2024-09-19T07:48:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8581a45640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f857c160ca0 con 0x7f857c14dcc0 2024-09-19T07:48:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f857affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f857c06e460 0x7f857c14cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f857affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f857c06e460 0x7f857c14cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48204/0 (socket says 172.21.15.83:48204) 2024-09-19T07:48:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f857affd640 1 -- 172.21.15.83:0/424953780 learned_addr learned my addr 172.21.15.83:0/424953780 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f857b7fe640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f857c14dcc0 0x7f857c14aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f857a7fc640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f857c06f2f0 0x7f857c14b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f857affd640 1 -- 172.21.15.83:0/424953780 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f857c14dcc0 msgr2=0x7f857c14aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f857affd640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f857c14dcc0 0x7f857c14aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f857affd640 1 -- 172.21.15.83:0/424953780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f857c06f2f0 msgr2=0x7f857c14b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f857affd640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f857c06f2f0 0x7f857c14b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f857affd640 1 -- 172.21.15.83:0/424953780 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857c0ae3e0 con 0x7f857c06e460 2024-09-19T07:48:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f857affd640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f857c06e460 0x7f857c14cbc0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f857401ab50 tx=0x7f857401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8563fff640 1 -- 172.21.15.83:0/424953780 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85740aabe0 con 0x7f857c06e460 2024-09-19T07:48:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8563fff640 1 -- 172.21.15.83:0/424953780 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85740aad80 con 0x7f857c06e460 2024-09-19T07:48:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8581a45640 1 -- 172.21.15.83:0/424953780 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f857c1580f0 con 0x7f857c06e460 2024-09-19T07:48:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8563fff640 1 -- 172.21.15.83:0/424953780 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85740be650 con 0x7f857c06e460 2024-09-19T07:48:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8581a45640 1 -- 172.21.15.83:0/424953780 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f857c0ae6a0 con 0x7f857c06e460 2024-09-19T07:48:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8581a45640 1 -- 172.21.15.83:0/424953780 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f857c154410 con 0x7f857c06e460 2024-09-19T07:48:46.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8563fff640 1 -- 172.21.15.83:0/424953780 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f85740bc020 con 0x7f857c06e460 2024-09-19T07:48:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8563fff640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f85680678f0 0x7f8568069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f857a7fc640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f85680678f0 0x7f8568069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.949+0000 7f8563fff640 1 -- 172.21.15.83:0/424953780 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9991+0+0 (secure 0 0 0) 0x7f8574120ca0 con 0x7f857c06e460 2024-09-19T07:48:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.953+0000 7f857a7fc640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f85680678f0 0x7f8568069db0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f857c152fe0 tx=0x7f8570009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:46.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:46.957+0000 7f8563fff640 1 -- 172.21.15.83:0/424953780 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f8574016090 con 0x7f857c06e460 2024-09-19T07:48:47.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.217+0000 7f8581a45640 1 -- 172.21.15.83:0/424953780 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f857c0c6c80 con 0x7f857c06e460 2024-09-19T07:48:47.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.221+0000 7f8563fff640 1 -- 172.21.15.83:0/424953780 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) ==== 95+0+6545 (secure 0 0 0) 0x7f85740f0230 con 0x7f857c06e460 2024-09-19T07:48:47.225 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:47.225 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":200,"btime":"2024-09-19T07:48:29:736412+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:48:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 200 2024-09-19T07:48:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.233+0000 7f8581a45640 1 -- 172.21.15.83:0/424953780 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f85680678f0 msgr2=0x7f8568069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.233+0000 7f8581a45640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f85680678f0 0x7f8568069db0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f857c152fe0 tx=0x7f8570009190 comp rx=0 tx=0).stop 2024-09-19T07:48:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.233+0000 7f8581a45640 1 -- 172.21.15.83:0/424953780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f857c06e460 msgr2=0x7f857c14cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.233+0000 7f8581a45640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f857c06e460 0x7f857c14cbc0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f857401ab50 tx=0x7f857401ace0 comp rx=0 tx=0).stop 2024-09-19T07:48:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.233+0000 7f8581a45640 1 -- 172.21.15.83:0/424953780 shutdown_connections 2024-09-19T07:48:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.233+0000 7f8581a45640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f85680678f0 0x7f8568069db0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.233+0000 7f8581a45640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f857c14dcc0 0x7f857c14aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.233+0000 7f8581a45640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f857c06f2f0 0x7f857c14b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.233+0000 7f8581a45640 1 --2- 172.21.15.83:0/424953780 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f857c06e460 0x7f857c14cbc0 secure :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f857401ab50 tx=0x7f857401ace0 comp rx=0 tx=0).stop 2024-09-19T07:48:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.233+0000 7f8581a45640 1 -- 172.21.15.83:0/424953780 >> 172.21.15.83:0/424953780 conn(0x7f857c06ffc0 msgr2=0x7f857c0c7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.233+0000 7f8581a45640 1 -- 172.21.15.83:0/424953780 shutdown_connections 2024-09-19T07:48:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:47.233+0000 7f8581a45640 1 -- 172.21.15.83:0/424953780 wait complete. 2024-09-19T07:48:47.264 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:48:49.197 INFO:tasks.workunit.client.1.smithi184.stdout:Syncing()...1 sec 2024-09-19T07:48:49.197 INFO:tasks.workunit.client.1.smithi184.stdout:FFSB benchmark finished at: Thu Sep 19 07:48:49 2024 2024-09-19T07:48:49.197 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:48:49.197 INFO:tasks.workunit.client.1.smithi184.stdout:Results: 2024-09-19T07:48:49.197 INFO:tasks.workunit.client.1.smithi184.stdout:Benchmark took 306.05 sec 2024-09-19T07:48:49.197 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:48:49.197 INFO:tasks.workunit.client.1.smithi184.stdout:Total Results 2024-09-19T07:48:49.197 INFO:tasks.workunit.client.1.smithi184.stdout:=============== 2024-09-19T07:48:49.198 INFO:tasks.workunit.client.1.smithi184.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-09-19T07:48:49.198 INFO:tasks.workunit.client.1.smithi184.stdout: ======= ============ ========= ======= =========== ========== 2024-09-19T07:48:49.198 INFO:tasks.workunit.client.1.smithi184.stdout: write : 1532160 5006.19 100.000% 100.000% 19.6MB/sec 2024-09-19T07:48:49.198 INFO:tasks.workunit.client.1.smithi184.stdout:- 2024-09-19T07:48:49.198 INFO:tasks.workunit.client.1.smithi184.stdout:5006.19 Transactions per Second 2024-09-19T07:48:49.198 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:48:49.198 INFO:tasks.workunit.client.1.smithi184.stdout:Throughput Results 2024-09-19T07:48:49.198 INFO:tasks.workunit.client.1.smithi184.stdout:=================== 2024-09-19T07:48:49.198 INFO:tasks.workunit.client.1.smithi184.stdout:Write Throughput: 19.6MB/sec 2024-09-19T07:48:49.204 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:48:49.204 INFO:tasks.workunit.client.1.smithi184.stdout:System Call Latency statistics in millisecs 2024-09-19T07:48:49.205 INFO:tasks.workunit.client.1.smithi184.stdout:===== 2024-09-19T07:48:49.205 INFO:tasks.workunit.client.1.smithi184.stdout: Min Avg Max Total Calls 2024-09-19T07:48:49.205 INFO:tasks.workunit.client.1.smithi184.stdout: ======== ======== ======== ============ 2024-09-19T07:48:49.205 INFO:tasks.workunit.client.1.smithi184.stdout:[ open] 0.316000 1214.549206 21634.951172 1197 2024-09-19T07:48:49.205 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-19T07:48:49.205 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-19T07:48:49.205 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-19T07:48:49.205 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-19T07:48:49.205 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[4] 200.000000 - 500.000000 : 8 2024-09-19T07:48:49.205 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[5] 100.000000 - 200.000000 : 13 2024-09-19T07:48:49.205 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[6] 50.000000 - 100.000000 : 28 2024-09-19T07:48:49.206 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[7] 20.000000 - 50.000000 : 375 2024-09-19T07:48:49.206 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[8] 10.000000 - 20.000000 : 420 2024-09-19T07:48:49.206 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[9] 5.000000 - 10.000000 : 147 2024-09-19T07:48:49.206 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[10] 2.000000 - 5.000000 : 62 2024-09-19T07:48:49.206 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[11] 1.000000 - 2.000000 : 34 2024-09-19T07:48:49.206 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[12] 0.500000 - 1.000000 : 14 2024-09-19T07:48:49.206 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[13] 0.200000 - 0.500000 : 1 2024-09-19T07:48:49.206 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-09-19T07:48:49.206 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-09-19T07:48:49.206 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-09-19T07:48:49.206 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-19T07:48:49.206 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-19T07:48:49.207 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:48:49.207 INFO:tasks.workunit.client.1.smithi184.stdout:[ write] 0.088000 5.358568 18504.568359 1532160 2024-09-19T07:48:49.207 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-19T07:48:49.207 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-19T07:48:49.207 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-19T07:48:49.207 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-19T07:48:49.207 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[4] 200.000000 - 500.000000 : 4 2024-09-19T07:48:49.208 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[5] 100.000000 - 200.000000 : 179 2024-09-19T07:48:49.208 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[6] 50.000000 - 100.000000 : 6506 2024-09-19T07:48:49.208 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[7] 20.000000 - 50.000000 : 75215 2024-09-19T07:48:49.208 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[8] 10.000000 - 20.000000 : 152522 2024-09-19T07:48:49.208 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[9] 5.000000 - 10.000000 : 272595 2024-09-19T07:48:49.208 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[10] 2.000000 - 5.000000 : 341933 2024-09-19T07:48:49.208 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[11] 1.000000 - 2.000000 : 150869 2024-09-19T07:48:49.208 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[12] 0.500000 - 1.000000 : 74014 2024-09-19T07:48:49.208 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[13] 0.200000 - 0.500000 : 223833 2024-09-19T07:48:49.208 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[14] 0.100000 - 0.200000 : 233753 2024-09-19T07:48:49.208 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[15] 0.050000 - 0.100000 : 736 2024-09-19T07:48:49.208 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-09-19T07:48:49.209 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-19T07:48:49.209 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-19T07:48:49.209 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:48:49.209 INFO:tasks.workunit.client.1.smithi184.stdout:[ lseek] 0.000000 0.001007 6.496000 1532160 2024-09-19T07:48:49.209 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-19T07:48:49.209 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-19T07:48:49.209 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-19T07:48:49.210 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-19T07:48:49.210 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-19T07:48:49.210 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-19T07:48:49.210 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-09-19T07:48:49.210 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-09-19T07:48:49.210 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-09-19T07:48:49.210 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[9] 5.000000 - 10.000000 : 2 2024-09-19T07:48:49.210 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[10] 2.000000 - 5.000000 : 4 2024-09-19T07:48:49.210 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[11] 1.000000 - 2.000000 : 2 2024-09-19T07:48:49.210 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[12] 0.500000 - 1.000000 : 7 2024-09-19T07:48:49.210 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[13] 0.200000 - 0.500000 : 19 2024-09-19T07:48:49.210 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[14] 0.100000 - 0.200000 : 26 2024-09-19T07:48:49.211 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[15] 0.050000 - 0.100000 : 44 2024-09-19T07:48:49.211 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[16] 0.020000 - 0.050000 : 282 2024-09-19T07:48:49.211 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[17] 0.010000 - 0.020000 : 840 2024-09-19T07:48:49.211 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[18] 0.000000 - 0.010000 : 1530934 2024-09-19T07:48:49.211 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:48:49.211 INFO:tasks.workunit.client.1.smithi184.stdout:[ close] 0.021000 2.800028 77.719002 1197 2024-09-19T07:48:49.211 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-19T07:48:49.211 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-19T07:48:49.211 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-19T07:48:49.211 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-19T07:48:49.211 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-19T07:48:49.211 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-19T07:48:49.212 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[6] 50.000000 - 100.000000 : 5 2024-09-19T07:48:49.212 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[7] 20.000000 - 50.000000 : 44 2024-09-19T07:48:49.212 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[8] 10.000000 - 20.000000 : 43 2024-09-19T07:48:49.212 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[9] 5.000000 - 10.000000 : 70 2024-09-19T07:48:49.212 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[10] 2.000000 - 5.000000 : 127 2024-09-19T07:48:49.212 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[11] 1.000000 - 2.000000 : 119 2024-09-19T07:48:49.212 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[12] 0.500000 - 1.000000 : 71 2024-09-19T07:48:49.212 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[13] 0.200000 - 0.500000 : 72 2024-09-19T07:48:49.212 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[14] 0.100000 - 0.200000 : 141 2024-09-19T07:48:49.213 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[15] 0.050000 - 0.100000 : 99 2024-09-19T07:48:49.213 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[16] 0.020000 - 0.050000 : 406 2024-09-19T07:48:49.213 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-19T07:48:49.213 INFO:tasks.workunit.client.1.smithi184.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-19T07:48:49.213 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:48:49.213 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:48:49.213 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:48:49.213 INFO:tasks.workunit.client.1.smithi184.stdout: 2024-09-19T07:48:49.213 INFO:tasks.workunit.client.1.smithi184.stdout:0.5% User Time 2024-09-19T07:48:49.213 INFO:tasks.workunit.client.1.smithi184.stdout:58.1% System Time 2024-09-19T07:48:49.213 INFO:tasks.workunit.client.1.smithi184.stdout:58.6% CPU Utilization 2024-09-19T07:48:49.214 INFO:tasks.workunit.client.1.smithi184.stderr:+ cd .. 2024-09-19T07:48:49.218 INFO:tasks.workunit.client.1.smithi184.stderr:+ rm -r tmp ffsb 2024-09-19T07:48:49.266 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.409+0000 7f781ae33640 1 Processor -- start 2024-09-19T07:48:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.409+0000 7f781ae33640 1 -- start start 2024-09-19T07:48:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.409+0000 7f781ae33640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7814150290 0x7f78140c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.409+0000 7f781ae33640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7814152b20 0x7f78140c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.409+0000 7f781ae33640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7814150670 con 0x7f78140c45c0 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.409+0000 7f781ae33640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f78140716c0 con 0x7f7814152b20 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.409+0000 7f781ae33640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f78140c3920 con 0x7f7814150290 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f7813fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7814152b20 0x7f78140c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f78193d0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f78140c45c0 0x7f7814153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57594/0 (socket says 172.21.15.83:57594) 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f78193d0640 1 -- 172.21.15.83:0/3971930709 learned_addr learned my addr 172.21.15.83:0/3971930709 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f7818bcf640 1 --2- 172.21.15.83:0/3971930709 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7814150290 0x7f78140c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f78137fe640 1 -- 172.21.15.83:0/3971930709 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317139537 0 0) 0x7f7814150670 con 0x7f78140c45c0 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f78137fe640 1 -- 172.21.15.83:0/3971930709 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77fc003530 con 0x7f78140c45c0 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f7813fff640 1 -- 172.21.15.83:0/3971930709 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7814150290 msgr2=0x7f78140c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f7813fff640 1 --2- 172.21.15.83:0/3971930709 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7814150290 0x7f78140c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f7813fff640 1 -- 172.21.15.83:0/3971930709 >> v1:172.21.15.83:6789/0 conn(0x7f78140c45c0 legacy=0x7f7814153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f7813fff640 1 -- 172.21.15.83:0/3971930709 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78140c2880 con 0x7f7814152b20 2024-09-19T07:48:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f7813fff640 1 --2- 172.21.15.83:0/3971930709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7814152b20 0x7f78140c4080 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f780800ba90 tx=0x7f780801eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4fca554d9caa1f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:49.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f78137fe640 1 -- 172.21.15.83:0/3971930709 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78080b4070 con 0x7f7814152b20 2024-09-19T07:48:49.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f78137fe640 1 -- 172.21.15.83:0/3971930709 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78080a7a50 con 0x7f7814152b20 2024-09-19T07:48:49.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.413+0000 7f78137fe640 1 -- 172.21.15.83:0/3971930709 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78080a7e20 con 0x7f7814152b20 2024-09-19T07:48:49.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 -- 172.21.15.83:0/3971930709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7814152b20 msgr2=0x7f78140c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 --2- 172.21.15.83:0/3971930709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7814152b20 0x7f78140c4080 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f780800ba90 tx=0x7f780801eb70 comp rx=0 tx=0).stop 2024-09-19T07:48:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 -- 172.21.15.83:0/3971930709 shutdown_connections 2024-09-19T07:48:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 --2- 172.21.15.83:0/3971930709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7814152b20 0x7f78140c4080 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 --2- 172.21.15.83:0/3971930709 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7814150290 0x7f78140c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 -- 172.21.15.83:0/3971930709 >> 172.21.15.83:0/3971930709 conn(0x7f781406bc80 msgr2=0x7f781406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 -- 172.21.15.83:0/3971930709 shutdown_connections 2024-09-19T07:48:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 -- 172.21.15.83:0/3971930709 wait complete. 2024-09-19T07:48:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 Processor -- start 2024-09-19T07:48:49.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 -- start start 2024-09-19T07:48:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78140c45c0 0x7f7814149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7814150290 0x7f78141469a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7814152b20 0x7f781414e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f78140c2530 con 0x7f7814150290 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f78140c3120 con 0x7f7814152b20 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f781406e6b0 con 0x7f78140c45c0 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f7813fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7814150290 0x7f78141469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f7813fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7814150290 0x7f78141469a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48220/0 (socket says 172.21.15.83:48220) 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f7813fff640 1 -- 172.21.15.83:0/92401717 learned_addr learned my addr 172.21.15.83:0/92401717 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f7818bcf640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78140c45c0 0x7f7814149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f7813fff640 1 -- 172.21.15.83:0/92401717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78140c45c0 msgr2=0x7f7814149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f7813fff640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78140c45c0 0x7f7814149b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f7813fff640 1 -- 172.21.15.83:0/92401717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7814152b20 msgr2=0x7f781414e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f7813fff640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7814152b20 0x7f781414e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f7813fff640 1 -- 172.21.15.83:0/92401717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78140c1680 con 0x7f7814150290 2024-09-19T07:48:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f7813fff640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7814150290 0x7f78141469a0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f780800ba90 tx=0x7f780801b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:49.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f78117fa640 1 -- 172.21.15.83:0/92401717 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78080b4040 con 0x7f7814150290 2024-09-19T07:48:49.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f78117fa640 1 -- 172.21.15.83:0/92401717 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78080b8420 con 0x7f7814150290 2024-09-19T07:48:49.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 -- 172.21.15.83:0/92401717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7814072b50 con 0x7f7814150290 2024-09-19T07:48:49.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f78117fa640 1 -- 172.21.15.83:0/92401717 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78080b8810 con 0x7f7814150290 2024-09-19T07:48:49.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.417+0000 7f781ae33640 1 -- 172.21.15.83:0/92401717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78140730d0 con 0x7f7814150290 2024-09-19T07:48:49.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.421+0000 7f781ae33640 1 -- 172.21.15.83:0/92401717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f781406eaa0 con 0x7f7814150290 2024-09-19T07:48:49.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.421+0000 7f78117fa640 1 -- 172.21.15.83:0/92401717 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f78080b89b0 con 0x7f7814150290 2024-09-19T07:48:49.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.421+0000 7f78117fa640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78000678f0 0x7f7800069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:49.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.421+0000 7f78117fa640 1 -- 172.21.15.83:0/92401717 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9991+0+0 (secure 0 0 0) 0x7f7808122350 con 0x7f7814150290 2024-09-19T07:48:49.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.421+0000 7f7818bcf640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78000678f0 0x7f7800069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:49.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.421+0000 7f7818bcf640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78000678f0 0x7f7800069db0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7814153d20 tx=0x7f7804008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:49.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.425+0000 7f78117fa640 1 -- 172.21.15.83:0/92401717 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7808018760 con 0x7f7814150290 2024-09-19T07:48:49.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.625+0000 7f781ae33640 1 -- 172.21.15.83:0/92401717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78140c3e90 con 0x7f7814150290 2024-09-19T07:48:49.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f78117fa640 1 -- 172.21.15.83:0/92401717 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) ==== 95+0+6545 (secure 0 0 0) 0x7f78080f1e90 con 0x7f7814150290 2024-09-19T07:48:49.627 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:49.628 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":200,"btime":"2024-09-19T07:48:29:736412+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:48:49.628 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 200 2024-09-19T07:48:49.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f781ae33640 1 -- 172.21.15.83:0/92401717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78000678f0 msgr2=0x7f7800069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:49.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f781ae33640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78000678f0 0x7f7800069db0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7814153d20 tx=0x7f7804008040 comp rx=0 tx=0).stop 2024-09-19T07:48:49.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f781ae33640 1 -- 172.21.15.83:0/92401717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7814150290 msgr2=0x7f78141469a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:49.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f781ae33640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7814150290 0x7f78141469a0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f780800ba90 tx=0x7f780801b910 comp rx=0 tx=0).stop 2024-09-19T07:48:49.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f781ae33640 1 -- 172.21.15.83:0/92401717 shutdown_connections 2024-09-19T07:48:49.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f781ae33640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f78000678f0 0x7f7800069db0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:49.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f781ae33640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7814152b20 0x7f781414e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:49.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f781ae33640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7814150290 0x7f78141469a0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:49.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f781ae33640 1 --2- 172.21.15.83:0/92401717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78140c45c0 0x7f7814149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:49.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f781ae33640 1 -- 172.21.15.83:0/92401717 >> 172.21.15.83:0/92401717 conn(0x7f781406bc80 msgr2=0x7f781414d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:49.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f781ae33640 1 -- 172.21.15.83:0/92401717 shutdown_connections 2024-09-19T07:48:49.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:49.629+0000 7f781ae33640 1 -- 172.21.15.83:0/92401717 wait complete. 2024-09-19T07:48:49.647 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:48:51.649 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:51.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f298827c640 1 Processor -- start 2024-09-19T07:48:51.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f298827c640 1 -- start start 2024-09-19T07:48:51.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f298827c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f29800c7510 0x7f298006de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:51.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f298827c640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2980154360 0x7f29800c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:51.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f298827c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f298014ddd0 con 0x7f298014d9f0 2024-09-19T07:48:51.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f298827c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f29800c8830 con 0x7f29800c7510 2024-09-19T07:48:51.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f298827c640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f298006fdf0 con 0x7f2980154360 2024-09-19T07:48:51.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f2985817640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f29800c7510 0x7f298006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:51.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f2986018640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f298014d9f0 0x7f29800c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47930/0 (socket says 172.21.15.83:47930) 2024-09-19T07:48:51.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f2986018640 1 -- 172.21.15.83:0/2609330938 learned_addr learned my addr 172.21.15.83:0/2609330938 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f2986819640 1 --2- 172.21.15.83:0/2609330938 >> v2:172.21.15.184:3300/0 conn(0x7f2980154360 0x7f29800c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f2985016640 1 -- 172.21.15.83:0/2609330938 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 517498256 0 0) 0x7f298014ddd0 con 0x7f298014d9f0 2024-09-19T07:48:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f2985016640 1 -- 172.21.15.83:0/2609330938 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f296c003530 con 0x7f298014d9f0 2024-09-19T07:48:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f2985817640 1 -- 172.21.15.83:0/2609330938 >> v2:172.21.15.184:3300/0 conn(0x7f2980154360 msgr2=0x7f29800c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f2985817640 1 --2- 172.21.15.83:0/2609330938 >> v2:172.21.15.184:3300/0 conn(0x7f2980154360 0x7f29800c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f2985817640 1 -- 172.21.15.83:0/2609330938 >> v1:172.21.15.83:6789/0 conn(0x7f298014d9f0 legacy=0x7f29800c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:51.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.801+0000 7f2985817640 1 -- 172.21.15.83:0/2609330938 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29800c7f00 con 0x7f29800c7510 2024-09-19T07:48:51.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.805+0000 7f2985817640 1 --2- 172.21.15.83:0/2609330938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29800c7510 0x7f298006de90 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f29800743c0 tx=0x7f297001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b21c90101a1f78c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:51.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.805+0000 7f2985016640 1 -- 172.21.15.83:0/2609330938 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29700b0070 con 0x7f29800c7510 2024-09-19T07:48:51.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.805+0000 7f2985016640 1 -- 172.21.15.83:0/2609330938 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29700ab600 con 0x7f29800c7510 2024-09-19T07:48:51.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.805+0000 7f2985016640 1 -- 172.21.15.83:0/2609330938 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29700ab9b0 con 0x7f29800c7510 2024-09-19T07:48:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.805+0000 7f298827c640 1 -- 172.21.15.83:0/2609330938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29800c7510 msgr2=0x7f298006de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.805+0000 7f298827c640 1 --2- 172.21.15.83:0/2609330938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29800c7510 0x7f298006de90 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f29800743c0 tx=0x7f297001c360 comp rx=0 tx=0).stop 2024-09-19T07:48:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.805+0000 7f298827c640 1 -- 172.21.15.83:0/2609330938 shutdown_connections 2024-09-19T07:48:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.805+0000 7f298827c640 1 --2- 172.21.15.83:0/2609330938 >> v2:172.21.15.184:3300/0 conn(0x7f2980154360 0x7f29800c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.805+0000 7f298827c640 1 --2- 172.21.15.83:0/2609330938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29800c7510 0x7f298006de90 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.805+0000 7f298827c640 1 -- 172.21.15.83:0/2609330938 >> 172.21.15.83:0/2609330938 conn(0x7f29800c6110 msgr2=0x7f29800c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.805+0000 7f298827c640 1 -- 172.21.15.83:0/2609330938 shutdown_connections 2024-09-19T07:48:51.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.805+0000 7f298827c640 1 -- 172.21.15.83:0/2609330938 wait complete. 2024-09-19T07:48:51.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f298827c640 1 Processor -- start 2024-09-19T07:48:51.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f298827c640 1 -- start start 2024-09-19T07:48:51.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f298827c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29800c7510 0x7f29800aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:51.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f298827c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f298014d9f0 0x7f2980158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:51.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f298827c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2980154360 0x7f298006c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:51.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f298827c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2980160c00 con 0x7f2980154360 2024-09-19T07:48:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f298827c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2980160a80 con 0x7f298014d9f0 2024-09-19T07:48:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f298827c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2980160d80 con 0x7f29800c7510 2024-09-19T07:48:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2986819640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2980154360 0x7f298006c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2986819640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2980154360 0x7f298006c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55512/0 (socket says 172.21.15.83:55512) 2024-09-19T07:48:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2986819640 1 -- 172.21.15.83:0/3981740279 learned_addr learned my addr 172.21.15.83:0/3981740279 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2985817640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f298014d9f0 0x7f2980158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2986018640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29800c7510 0x7f29800aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2986819640 1 -- 172.21.15.83:0/3981740279 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29800c7510 msgr2=0x7f29800aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2986819640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29800c7510 0x7f29800aeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2986819640 1 -- 172.21.15.83:0/3981740279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f298014d9f0 msgr2=0x7f2980158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2986819640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f298014d9f0 0x7f2980158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:51.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2986819640 1 -- 172.21.15.83:0/3981740279 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29801ef540 con 0x7f2980154360 2024-09-19T07:48:51.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2985817640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f298014d9f0 0x7f2980158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:48:51.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2986819640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2980154360 0x7f298006c480 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f2978007e60 tx=0x7f297800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:51.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2976ffd640 1 -- 172.21.15.83:0/3981740279 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2978014420 con 0x7f2980154360 2024-09-19T07:48:51.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2976ffd640 1 -- 172.21.15.83:0/3981740279 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2978019070 con 0x7f2980154360 2024-09-19T07:48:51.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f2976ffd640 1 -- 172.21.15.83:0/3981740279 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2978014820 con 0x7f2980154360 2024-09-19T07:48:51.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f298827c640 1 -- 172.21.15.83:0/3981740279 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f298006d900 con 0x7f2980154360 2024-09-19T07:48:51.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.809+0000 7f298827c640 1 -- 172.21.15.83:0/3981740279 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29800af050 con 0x7f2980154360 2024-09-19T07:48:51.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.813+0000 7f298827c640 1 -- 172.21.15.83:0/3981740279 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f298006f590 con 0x7f2980154360 2024-09-19T07:48:51.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.813+0000 7f2976ffd640 1 -- 172.21.15.83:0/3981740279 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f29780149c0 con 0x7f2980154360 2024-09-19T07:48:51.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.813+0000 7f2976ffd640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29640678c0 0x7f2964069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:51.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.813+0000 7f2976ffd640 1 -- 172.21.15.83:0/3981740279 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9991+0+0 (secure 0 0 0) 0x7f2978087560 con 0x7f2980154360 2024-09-19T07:48:51.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.813+0000 7f2986018640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29640678c0 0x7f2964069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:51.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.813+0000 7f2986018640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29640678c0 0x7f2964069d80 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f297c007fd0 tx=0x7f297c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:51.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:51.817+0000 7f2976ffd640 1 -- 172.21.15.83:0/3981740279 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2978057020 con 0x7f2980154360 2024-09-19T07:48:52.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.033+0000 7f298827c640 1 -- 172.21.15.83:0/3981740279 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f298015ddc0 con 0x7f2980154360 2024-09-19T07:48:52.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.037+0000 7f2976ffd640 1 -- 172.21.15.83:0/3981740279 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) ==== 95+0+6545 (secure 0 0 0) 0x7f2978056920 con 0x7f2980154360 2024-09-19T07:48:52.035 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:52.035 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":200,"btime":"2024-09-19T07:48:29:736412+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:48:52.035 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 200 2024-09-19T07:48:52.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.041+0000 7f298827c640 1 -- 172.21.15.83:0/3981740279 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29640678c0 msgr2=0x7f2964069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.041+0000 7f298827c640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29640678c0 0x7f2964069d80 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f297c007fd0 tx=0x7f297c009040 comp rx=0 tx=0).stop 2024-09-19T07:48:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.041+0000 7f298827c640 1 -- 172.21.15.83:0/3981740279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2980154360 msgr2=0x7f298006c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.041+0000 7f298827c640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2980154360 0x7f298006c480 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f2978007e60 tx=0x7f297800e4b0 comp rx=0 tx=0).stop 2024-09-19T07:48:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.041+0000 7f298827c640 1 -- 172.21.15.83:0/3981740279 shutdown_connections 2024-09-19T07:48:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.041+0000 7f298827c640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29640678c0 0x7f2964069d80 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.041+0000 7f298827c640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2980154360 0x7f298006c480 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.041+0000 7f298827c640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f298014d9f0 0x7f2980158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.041+0000 7f298827c640 1 --2- 172.21.15.83:0/3981740279 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29800c7510 0x7f29800aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.041+0000 7f298827c640 1 -- 172.21.15.83:0/3981740279 >> 172.21.15.83:0/3981740279 conn(0x7f29800c6110 msgr2=0x7f298014aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.041+0000 7f298827c640 1 -- 172.21.15.83:0/3981740279 shutdown_connections 2024-09-19T07:48:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:52.041+0000 7f298827c640 1 -- 172.21.15.83:0/3981740279 wait complete. 2024-09-19T07:48:52.057 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:48:53.785 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-19T07:48:53.785 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-09-19T07:48:53.840 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.1... 2024-09-19T07:48:53.840 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-09-19T07:48:54.058 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:54.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.244+0000 7fc22c820640 1 Processor -- start 2024-09-19T07:48:54.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc22c820640 1 -- start start 2024-09-19T07:48:54.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc22c820640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc220046d30 0x7fc220008230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:54.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc22c820640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22002bc30 con 0x7fc2200ded00 2024-09-19T07:48:54.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc22c820640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc220047240 con 0x7fc220008770 2024-09-19T07:48:54.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc22c820640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc220007da0 con 0x7fc220046d30 2024-09-19T07:48:54.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc226d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc2200ded00 0x7fc22000bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47934/0 (socket says 172.21.15.83:47934) 2024-09-19T07:48:54.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc226d9d640 1 -- 172.21.15.83:0/195960753 learned_addr learned my addr 172.21.15.83:0/195960753 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 96965321 0 0) 0x7fc22002bc30 con 0x7fc2200ded00 2024-09-19T07:48:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc210003530 con 0x7fc2200ded00 2024-09-19T07:48:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 453711851 0 0) 0x7fc220047240 con 0x7fc220008770 2024-09-19T07:48:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2100048f0 con 0x7fc220008770 2024-09-19T07:48:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3032275351 0 0) 0x7fc210003530 con 0x7fc2200ded00 2024-09-19T07:48:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 >> v2:172.21.15.83:3300/0 conn(0x7fc220046d30 msgr2=0x7fc220008230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T07:48:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc22659c640 1 --2- 172.21.15.83:0/195960753 >> v2:172.21.15.83:3300/0 conn(0x7fc220046d30 0x7fc220008230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 --2- 172.21.15.83:0/195960753 >> v2:172.21.15.83:3300/0 conn(0x7fc220046d30 0x7fc220008230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 >> v1:172.21.15.184:6789/0 conn(0x7fc220008770 legacy=0x7fc2200e8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2200df0e0 con 0x7fc2200ded00 2024-09-19T07:48:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fc228050810 con 0x7fc2200ded00 2024-09-19T07:48:54.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 >> v1:172.21.15.83:6789/0 conn(0x7fc2200ded00 legacy=0x7fc22000bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 --2- 172.21.15.83:0/195960753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc210004da0 0x7fc2100051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 --2- 172.21.15.83:0/195960753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100037b0 0x7fc210003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 --2- 172.21.15.83:0/195960753 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2100043c0 0x7fc21000b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.248+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc220007da0 con 0x7fc2100037b0 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc21000c2d0 con 0x7fc210004da0 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc21000c4d0 con 0x7fc2100043c0 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc22659c640 1 --2- 172.21.15.83:0/195960753 >> v2:172.21.15.83:3300/0 conn(0x7fc220046d30 0x7fc220008230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc22759e640 1 --2- 172.21.15.83:0/195960753 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2100043c0 0x7fc21000b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc226d9d640 1 --2- 172.21.15.83:0/195960753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc210004da0 0x7fc2100051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc22659c640 1 --2- 172.21.15.83:0/195960753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100037b0 0x7fc210003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc22759e640 1 -- 172.21.15.83:0/195960753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc210004da0 msgr2=0x7fc2100051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc22759e640 1 --2- 172.21.15.83:0/195960753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc210004da0 0x7fc2100051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:54.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc22759e640 1 -- 172.21.15.83:0/195960753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100037b0 msgr2=0x7fc210003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:54.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc22759e640 1 --2- 172.21.15.83:0/195960753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100037b0 0x7fc210003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:54.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc22759e640 1 -- 172.21.15.83:0/195960753 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2200df0e0 con 0x7fc2100043c0 2024-09-19T07:48:54.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc22759e640 1 -- 172.21.15.83:0/195960753 reap_dead start 2024-09-19T07:48:54.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc22759e640 1 --2- 172.21.15.83:0/195960753 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2100043c0 0x7fc21000b9f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc2180018c0 tx=0x7fc21801c580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1a22315b0d5de4b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:54.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc218001510 con 0x7fc2100043c0 2024-09-19T07:48:54.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2180052d0 con 0x7fc2100043c0 2024-09-19T07:48:54.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc225d9b640 1 -- 172.21.15.83:0/195960753 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc218003800 con 0x7fc2100043c0 2024-09-19T07:48:54.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc22c820640 1 -- 172.21.15.83:0/195960753 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2100043c0 msgr2=0x7fc21000b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:54.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.252+0000 7fc22c820640 1 --2- 172.21.15.83:0/195960753 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2100043c0 0x7fc21000b9f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc2180018c0 tx=0x7fc21801c580 comp rx=0 tx=0).stop 2024-09-19T07:48:54.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 -- 172.21.15.83:0/195960753 shutdown_connections 2024-09-19T07:48:54.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 --2- 172.21.15.83:0/195960753 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2100043c0 0x7fc21000b9f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:54.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 -- 172.21.15.83:0/195960753 >> 172.21.15.83:0/195960753 conn(0x7fc220048e30 msgr2=0x7fc220049240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:54.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 -- 172.21.15.83:0/195960753 shutdown_connections 2024-09-19T07:48:54.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 -- 172.21.15.83:0/195960753 wait complete. 2024-09-19T07:48:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 Processor -- start 2024-09-19T07:48:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 -- start start 2024-09-19T07:48:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100043c0 0x7fc2200d53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2200458b0 0x7fc220041390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc22003fee0 0x7fc220040390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc22003fb60 con 0x7fc2100043c0 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc220007a30 con 0x7fc2200458b0 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22c820640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc22003fce0 con 0x7fc22003fee0 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc226d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100043c0 0x7fc2200d53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc226d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100043c0 0x7fc2200d53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55538/0 (socket says 172.21.15.83:55538) 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc226d9d640 1 -- 172.21.15.83:0/4190779932 learned_addr learned my addr 172.21.15.83:0/4190779932 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc22759e640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc22003fee0 0x7fc220040390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.256+0000 7fc226d9d640 1 -- 172.21.15.83:0/4190779932 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc22003fee0 msgr2=0x7fc220040390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc226d9d640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc22003fee0 0x7fc220040390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc226d9d640 1 -- 172.21.15.83:0/4190779932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2200458b0 msgr2=0x7fc220041390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc22659c640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2200458b0 0x7fc220041390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc226d9d640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2200458b0 0x7fc220041390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc226d9d640 1 -- 172.21.15.83:0/4190779932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc220036310 con 0x7fc2100043c0 2024-09-19T07:48:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc226d9d640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100043c0 0x7fc2200d53c0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fc2280504e0 tx=0x7fc2280506c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc20f7fe640 1 -- 172.21.15.83:0/4190779932 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc228099070 con 0x7fc2100043c0 2024-09-19T07:48:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc20f7fe640 1 -- 172.21.15.83:0/4190779932 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc228071890 con 0x7fc2100043c0 2024-09-19T07:48:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc20f7fe640 1 -- 172.21.15.83:0/4190779932 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc228071c80 con 0x7fc2100043c0 2024-09-19T07:48:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc22c820640 1 -- 172.21.15.83:0/4190779932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2200d7be0 con 0x7fc2100043c0 2024-09-19T07:48:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc22c820640 1 -- 172.21.15.83:0/4190779932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc220046250 con 0x7fc2100043c0 2024-09-19T07:48:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc22c820640 1 -- 172.21.15.83:0/4190779932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc220047240 con 0x7fc2100043c0 2024-09-19T07:48:54.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc20f7fe640 1 -- 172.21.15.83:0/4190779932 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc22809c360 con 0x7fc2100043c0 2024-09-19T07:48:54.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc20f7fe640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2140678c0 0x7fc214069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:54.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc22659c640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2140678c0 0x7fc214069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:54.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.260+0000 7fc22659c640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2140678c0 0x7fc214069d80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc2200de2e0 tx=0x7fc21c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:54.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.264+0000 7fc20f7fe640 1 -- 172.21.15.83:0/4190779932 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9991+0+0 (secure 0 0 0) 0x7fc228069300 con 0x7fc2100043c0 2024-09-19T07:48:54.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.268+0000 7fc20f7fe640 1 -- 172.21.15.83:0/4190779932 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc228071e20 con 0x7fc2100043c0 2024-09-19T07:48:54.321 DEBUG:teuthology.parallel:result is None 2024-09-19T07:48:54.321 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-19T07:48:54.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.748+0000 7fc22c820640 1 -- 172.21.15.83:0/4190779932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2200d4fe0 con 0x7fc2100043c0 2024-09-19T07:48:54.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.748+0000 7fc20f7fe640 1 -- 172.21.15.83:0/4190779932 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) ==== 95+0+6545 (secure 0 0 0) 0x7fc2280d2210 con 0x7fc2100043c0 2024-09-19T07:48:54.751 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:54.751 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":200,"btime":"2024-09-19T07:48:29:736412+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:48:54.751 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 200 2024-09-19T07:48:54.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.756+0000 7fc20d7fa640 1 -- 172.21.15.83:0/4190779932 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2140678c0 msgr2=0x7fc214069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:54.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.756+0000 7fc20d7fa640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2140678c0 0x7fc214069d80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc2200de2e0 tx=0x7fc21c008040 comp rx=0 tx=0).stop 2024-09-19T07:48:54.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.756+0000 7fc20d7fa640 1 -- 172.21.15.83:0/4190779932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100043c0 msgr2=0x7fc2200d53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:54.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.756+0000 7fc20d7fa640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100043c0 0x7fc2200d53c0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fc2280504e0 tx=0x7fc2280506c0 comp rx=0 tx=0).stop 2024-09-19T07:48:54.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.756+0000 7fc20d7fa640 1 -- 172.21.15.83:0/4190779932 shutdown_connections 2024-09-19T07:48:54.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.756+0000 7fc20d7fa640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2140678c0 0x7fc214069d80 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:54.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.756+0000 7fc20d7fa640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc22003fee0 0x7fc220040390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:54.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.756+0000 7fc20d7fa640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2200458b0 0x7fc220041390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:54.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.756+0000 7fc20d7fa640 1 --2- 172.21.15.83:0/4190779932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2100043c0 0x7fc2200d53c0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:54.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.756+0000 7fc20d7fa640 1 -- 172.21.15.83:0/4190779932 >> 172.21.15.83:0/4190779932 conn(0x7fc220048e30 msgr2=0x7fc22002c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:54.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.756+0000 7fc20d7fa640 1 -- 172.21.15.83:0/4190779932 shutdown_connections 2024-09-19T07:48:54.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:54.756+0000 7fc20d7fa640 1 -- 172.21.15.83:0/4190779932 wait complete. 2024-09-19T07:48:54.795 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:48:56.796 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:56.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59fe40d640 1 Processor -- start 2024-09-19T07:48:56.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59fe40d640 1 -- start start 2024-09-19T07:48:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59fe40d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59f8152b20 0x7f59f80c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59fe40d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f8150670 con 0x7f59f80c45c0 2024-09-19T07:48:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59fe40d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f80c2880 con 0x7f59f8150290 2024-09-19T07:48:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59fe40d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f59f80716c0 con 0x7f59f8152b20 2024-09-19T07:48:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59fc9aa640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f59f80c45c0 0x7f59f8153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47950/0 (socket says 172.21.15.83:47950) 2024-09-19T07:48:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59fc9aa640 1 -- 172.21.15.83:0/314473854 learned_addr learned my addr 172.21.15.83:0/314473854 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59effff640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59f8152b20 0x7f59f80c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2041016170 0 0) 0x7f59f8150670 con 0x7f59f80c45c0 2024-09-19T07:48:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59e4003530 con 0x7f59f80c45c0 2024-09-19T07:48:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1076901482 0 0) 0x7f59e4003530 con 0x7f59f80c45c0 2024-09-19T07:48:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59f8152b20 msgr2=0x7f59f80c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:56.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59f8152b20 0x7f59f80c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 >> v1:172.21.15.184:6789/0 conn(0x7f59f8150290 legacy=0x7f59f80c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f80c2b60 con 0x7f59f80c45c0 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f59f0001a00 con 0x7f59f80c45c0 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 >> v1:172.21.15.83:6789/0 conn(0x7f59f80c45c0 legacy=0x7f59f8153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59e40039f0 0x7f59e4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59f8152b20 0x7f59e4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59e4004d50 0x7f59e40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f59e400ba80 con 0x7f59e40039f0 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f59e400bc00 con 0x7f59e4004d50 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f59e400e010 con 0x7f59f8152b20 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f7fff640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59e40039f0 0x7f59e4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59fc9aa640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59e4004d50 0x7f59e40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59effff640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59f8152b20 0x7f59e4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:56.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f7fff640 1 -- 172.21.15.83:0/314473854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59f8152b20 msgr2=0x7f59e4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:56.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f7fff640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59f8152b20 0x7f59e4004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:56.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f7fff640 1 -- 172.21.15.83:0/314473854 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59e4004d50 msgr2=0x7f59e40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:56.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f7fff640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59e4004d50 0x7f59e40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:56.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59f7fff640 1 -- 172.21.15.83:0/314473854 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e4003530 con 0x7f59e40039f0 2024-09-19T07:48:56.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.924+0000 7f59fc9aa640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59e4004d50 0x7f59e40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:48:56.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59f7fff640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59e40039f0 0x7f59e4003e10 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f59e8001750 tx=0x7f59e801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aca37bd58fac478 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:56.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e80013a0 con 0x7f59e40039f0 2024-09-19T07:48:56.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e80052f0 con 0x7f59e40039f0 2024-09-19T07:48:56.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59f77fe640 1 -- 172.21.15.83:0/314473854 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e8003840 con 0x7f59e40039f0 2024-09-19T07:48:56.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 -- 172.21.15.83:0/314473854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59e40039f0 msgr2=0x7f59e4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:56.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 --2- 172.21.15.83:0/314473854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59e40039f0 0x7f59e4003e10 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f59e8001750 tx=0x7f59e801c3e0 comp rx=0 tx=0).stop 2024-09-19T07:48:56.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fc9aa640 1 -- 172.21.15.83:0/314473854 reap_dead start 2024-09-19T07:48:56.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 -- 172.21.15.83:0/314473854 shutdown_connections 2024-09-19T07:48:56.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 -- 172.21.15.83:0/314473854 >> 172.21.15.83:0/314473854 conn(0x7f59f806bc80 msgr2=0x7f59f806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:56.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 -- 172.21.15.83:0/314473854 shutdown_connections 2024-09-19T07:48:56.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 -- 172.21.15.83:0/314473854 wait complete. 2024-09-19T07:48:56.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 Processor -- start 2024-09-19T07:48:56.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 -- start start 2024-09-19T07:48:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59e40039f0 0x7f59f81eb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59f80aae20 0x7f59f814ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59f80c53f0 0x7f59f8150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f59f806e6b0 con 0x7f59f80c53f0 2024-09-19T07:48:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f59f80c3490 con 0x7f59f80aae20 2024-09-19T07:48:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fe40d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f59f806e830 con 0x7f59e40039f0 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fc9aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59f80c53f0 0x7f59f8150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fc9aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59f80c53f0 0x7f59f8150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55550/0 (socket says 172.21.15.83:55550) 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fc9aa640 1 -- 172.21.15.83:0/418994105 learned_addr learned my addr 172.21.15.83:0/418994105 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59f7fff640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59e40039f0 0x7f59f81eb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59effff640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59f80aae20 0x7f59f814ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fc9aa640 1 -- 172.21.15.83:0/418994105 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59e40039f0 msgr2=0x7f59f81eb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fc9aa640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59e40039f0 0x7f59f81eb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fc9aa640 1 -- 172.21.15.83:0/418994105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59f80aae20 msgr2=0x7f59f814ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fc9aa640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59f80aae20 0x7f59f814ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fc9aa640 1 -- 172.21.15.83:0/418994105 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f806f290 con 0x7f59f80c53f0 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59fc9aa640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59f80c53f0 0x7f59f8150f70 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f59f0001590 tx=0x7f59f000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.928+0000 7f59f57fa640 1 -- 172.21.15.83:0/418994105 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f000bb80 con 0x7f59f80c53f0 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.932+0000 7f59f57fa640 1 -- 172.21.15.83:0/418994105 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f000bd20 con 0x7f59f80c53f0 2024-09-19T07:48:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.932+0000 7f59fe40d640 1 -- 172.21.15.83:0/418994105 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f8152120 con 0x7f59f80c53f0 2024-09-19T07:48:56.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.932+0000 7f59f57fa640 1 -- 172.21.15.83:0/418994105 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f0007350 con 0x7f59f80c53f0 2024-09-19T07:48:56.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.932+0000 7f59fe40d640 1 -- 172.21.15.83:0/418994105 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f81526e0 con 0x7f59f80c53f0 2024-09-19T07:48:56.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.932+0000 7f59fe40d640 1 -- 172.21.15.83:0/418994105 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59f806ec80 con 0x7f59f80c53f0 2024-09-19T07:48:56.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.932+0000 7f59f57fa640 1 -- 172.21.15.83:0/418994105 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f59f001b060 con 0x7f59f80c53f0 2024-09-19T07:48:56.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.932+0000 7f59f57fa640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59d80678f0 0x7f59d8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:56.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.932+0000 7f59f7fff640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59d80678f0 0x7f59d8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:56.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.932+0000 7f59f57fa640 1 -- 172.21.15.83:0/418994105 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9991+0+0 (secure 0 0 0) 0x7f59f0086db0 con 0x7f59f80c53f0 2024-09-19T07:48:56.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.932+0000 7f59f7fff640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59d80678f0 0x7f59d8069db0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f59e40057f0 tx=0x7f59e80011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:56.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:56.936+0000 7f59f57fa640 1 -- 172.21.15.83:0/418994105 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f59f0056870 con 0x7f59f80c53f0 2024-09-19T07:48:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.116+0000 7f59fe40d640 1 -- 172.21.15.83:0/418994105 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59f80c3e90 con 0x7f59f80c53f0 2024-09-19T07:48:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.116+0000 7f59f57fa640 1 -- 172.21.15.83:0/418994105 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) ==== 95+0+6545 (secure 0 0 0) 0x7f59f0056170 con 0x7f59f80c53f0 2024-09-19T07:48:57.117 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:57.117 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":200,"btime":"2024-09-19T07:48:29:736412+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:02.799452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"d","rank":0,"incarnation":191,"state":"up:active","state_seq":16,"addr":"172.21.15.184:6833/2502231963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2502231963},{"type":"v1","addr":"172.21.15.184:6833","nonce":2502231963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:48:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 200 2024-09-19T07:48:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.120+0000 7f59fe40d640 1 -- 172.21.15.83:0/418994105 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59d80678f0 msgr2=0x7f59d8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.120+0000 7f59fe40d640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59d80678f0 0x7f59d8069db0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f59e40057f0 tx=0x7f59e80011a0 comp rx=0 tx=0).stop 2024-09-19T07:48:57.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.120+0000 7f59fe40d640 1 -- 172.21.15.83:0/418994105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59f80c53f0 msgr2=0x7f59f8150f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:57.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.120+0000 7f59fe40d640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59f80c53f0 0x7f59f8150f70 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f59f0001590 tx=0x7f59f000b4a0 comp rx=0 tx=0).stop 2024-09-19T07:48:57.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.120+0000 7f59fe40d640 1 -- 172.21.15.83:0/418994105 shutdown_connections 2024-09-19T07:48:57.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.120+0000 7f59fe40d640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f59d80678f0 0x7f59d8069db0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:57.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.120+0000 7f59fe40d640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59f80c53f0 0x7f59f8150f70 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:57.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.120+0000 7f59fe40d640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59f80aae20 0x7f59f814ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:57.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.120+0000 7f59fe40d640 1 --2- 172.21.15.83:0/418994105 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59e40039f0 0x7f59f81eb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:57.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.120+0000 7f59fe40d640 1 -- 172.21.15.83:0/418994105 >> 172.21.15.83:0/418994105 conn(0x7f59f806bc80 msgr2=0x7f59f81583a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:57.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.120+0000 7f59fe40d640 1 -- 172.21.15.83:0/418994105 shutdown_connections 2024-09-19T07:48:57.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:57.120+0000 7f59fe40d640 1 -- 172.21.15.83:0/418994105 wait complete. 2024-09-19T07:48:57.135 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-19T07:48:59.136 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:59.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.260+0000 7f66636f5640 1 Processor -- start 2024-09-19T07:48:59.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.260+0000 7f66636f5640 1 -- start start 2024-09-19T07:48:59.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f66636f5640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f665c0c28e0 0x7f665c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f66636f5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f665c0a9e60 con 0x7f665c06dea0 2024-09-19T07:48:59.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f66636f5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f665c0aa140 con 0x7f665c071100 2024-09-19T07:48:59.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f66636f5640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f665c071b50 con 0x7f665c0c28e0 2024-09-19T07:48:59.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6661c92640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f665c06dea0 0x7f665c06e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47962/0 (socket says 172.21.15.83:47962) 2024-09-19T07:48:59.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6661c92640 1 -- 172.21.15.83:0/2173548175 learned_addr learned my addr 172.21.15.83:0/2173548175 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6660c90640 1 --2- 172.21.15.83:0/2173548175 >> v2:172.21.15.184:3300/0 conn(0x7f665c0c28e0 0x7f665c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2095030302 0 0) 0x7f665c0a9e60 con 0x7f665c06dea0 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6648003530 con 0x7f665c06dea0 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 523613560 0 0) 0x7f6648003530 con 0x7f665c06dea0 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 >> v2:172.21.15.184:3300/0 conn(0x7f665c0c28e0 msgr2=0x7f665c1545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 --2- 172.21.15.83:0/2173548175 >> v2:172.21.15.184:3300/0 conn(0x7f665c0c28e0 0x7f665c1545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 >> v1:172.21.15.184:6789/0 conn(0x7f665c071100 legacy=0x7f665c150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665c0c2240 con 0x7f665c06dea0 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f6654001a00 con 0x7f665c06dea0 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 >> v1:172.21.15.83:6789/0 conn(0x7f665c06dea0 legacy=0x7f665c06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 --2- 172.21.15.83:0/2173548175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66480039c0 0x7f6648003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 --2- 172.21.15.83:0/2173548175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6648004810 0x7f6648004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 --2- 172.21.15.83:0/2173548175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66480051d0 0x7f664800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f665c071b50 con 0x7f66480039c0 2024-09-19T07:48:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f664800c170 con 0x7f66480051d0 2024-09-19T07:48:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f664800c370 con 0x7f6648004810 2024-09-19T07:48:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6661491640 1 --2- 172.21.15.83:0/2173548175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66480039c0 0x7f6648003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6661c92640 1 --2- 172.21.15.83:0/2173548175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66480051d0 0x7f664800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6661491640 1 -- 172.21.15.83:0/2173548175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6648004810 msgr2=0x7f6648004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6661491640 1 --2- 172.21.15.83:0/2173548175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6648004810 0x7f6648004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6661491640 1 -- 172.21.15.83:0/2173548175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66480051d0 msgr2=0x7f664800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6661491640 1 --2- 172.21.15.83:0/2173548175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66480051d0 0x7f664800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6661491640 1 -- 172.21.15.83:0/2173548175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6648003530 con 0x7f66480039c0 2024-09-19T07:48:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6661c92640 1 --2- 172.21.15.83:0/2173548175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66480051d0 0x7f664800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:48:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6661c92640 1 -- 172.21.15.83:0/2173548175 reap_dead start 2024-09-19T07:48:59.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6661491640 1 --2- 172.21.15.83:0/2173548175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66480039c0 0x7f6648003de0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f6658001780 tx=0x7f665801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7ed45b0157690b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:59.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66580013a0 con 0x7f66480039c0 2024-09-19T07:48:59.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66580052f0 con 0x7f66480039c0 2024-09-19T07:48:59.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.264+0000 7f6653fff640 1 -- 172.21.15.83:0/2173548175 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6658003840 con 0x7f66480039c0 2024-09-19T07:48:59.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 -- 172.21.15.83:0/2173548175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66480039c0 msgr2=0x7f6648003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:59.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 --2- 172.21.15.83:0/2173548175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66480039c0 0x7f6648003de0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f6658001780 tx=0x7f665801c400 comp rx=0 tx=0).stop 2024-09-19T07:48:59.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 -- 172.21.15.83:0/2173548175 shutdown_connections 2024-09-19T07:48:59.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 --2- 172.21.15.83:0/2173548175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66480039c0 0x7f6648003de0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 -- 172.21.15.83:0/2173548175 >> 172.21.15.83:0/2173548175 conn(0x7f665c06bc80 msgr2=0x7f665c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:59.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 -- 172.21.15.83:0/2173548175 shutdown_connections 2024-09-19T07:48:59.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 -- 172.21.15.83:0/2173548175 wait complete. 2024-09-19T07:48:59.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 Processor -- start 2024-09-19T07:48:59.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 -- start start 2024-09-19T07:48:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66480039c0 0x7f665c1eeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0c3ad0 0x7f665c14caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f665c0aa570 0x7f665c14fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f665c06e280 con 0x7f665c0c3ad0 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f665c0716f0 con 0x7f665c0aa570 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f665c06e400 con 0x7f66480039c0 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6660c90640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0c3ad0 0x7f665c14caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6660c90640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0c3ad0 0x7f665c14caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55576/0 (socket says 172.21.15.83:55576) 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6660c90640 1 -- 172.21.15.83:0/3791689314 learned_addr learned my addr 172.21.15.83:0/3791689314 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6661491640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66480039c0 0x7f665c1eeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6660c90640 1 -- 172.21.15.83:0/3791689314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66480039c0 msgr2=0x7f665c1eeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6661c92640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f665c0aa570 0x7f665c14fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6660c90640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66480039c0 0x7f665c1eeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6660c90640 1 -- 172.21.15.83:0/3791689314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f665c0aa570 msgr2=0x7f665c14fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6660c90640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f665c0aa570 0x7f665c14fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6660c90640 1 -- 172.21.15.83:0/3791689314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665c1f2c30 con 0x7f665c0c3ad0 2024-09-19T07:48:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6660c90640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0c3ad0 0x7f665c14caa0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f664c00e840 tx=0x7f664c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6651ffb640 1 -- 172.21.15.83:0/3791689314 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c017070 con 0x7f665c0c3ad0 2024-09-19T07:48:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6651ffb640 1 -- 172.21.15.83:0/3791689314 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f664c00c940 con 0x7f665c0c3ad0 2024-09-19T07:48:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 -- 172.21.15.83:0/3791689314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f665c1f2f10 con 0x7f665c0c3ad0 2024-09-19T07:48:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f6651ffb640 1 -- 172.21.15.83:0/3791689314 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c00cd30 con 0x7f665c0c3ad0 2024-09-19T07:48:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.268+0000 7f66636f5640 1 -- 172.21.15.83:0/3791689314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f665c0ac580 con 0x7f665c0c3ad0 2024-09-19T07:48:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.272+0000 7f66636f5640 1 -- 172.21.15.83:0/3791689314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f665c06e790 con 0x7f665c0c3ad0 2024-09-19T07:48:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.272+0000 7f6651ffb640 1 -- 172.21.15.83:0/3791689314 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f664c01a070 con 0x7f665c0c3ad0 2024-09-19T07:48:59.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.272+0000 7f6651ffb640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66400678c0 0x7f6640069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.272+0000 7f6651ffb640 1 -- 172.21.15.83:0/3791689314 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f664c086cb0 con 0x7f665c0c3ad0 2024-09-19T07:48:59.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.272+0000 7f6661491640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66400678c0 0x7f6640069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.272+0000 7f6661491640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66400678c0 0x7f6640069d80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6658001750 tx=0x7f66580011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:59.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.276+0000 7f6651ffb640 1 -- 172.21.15.83:0/3791689314 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f664c056720 con 0x7f665c0c3ad0 2024-09-19T07:48:59.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.460+0000 7f66636f5640 1 -- 172.21.15.83:0/3791689314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f665c150960 con 0x7f665c0c3ad0 2024-09-19T07:48:59.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.460+0000 7f6651ffb640 1 -- 172.21.15.83:0/3791689314 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) ==== 95+0+5708 (secure 0 0 0) 0x7f664c056020 con 0x7f665c0c3ad0 2024-09-19T07:48:59.462 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:59.462 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":201,"btime":"2024-09-19T07:48:58:354521+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:58.354512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:replay","state_seq":1,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:48:59.462 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 201 2024-09-19T07:48:59.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.464+0000 7f66636f5640 1 -- 172.21.15.83:0/3791689314 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66400678c0 msgr2=0x7f6640069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:59.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.464+0000 7f66636f5640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66400678c0 0x7f6640069d80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6658001750 tx=0x7f66580011a0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.464+0000 7f66636f5640 1 -- 172.21.15.83:0/3791689314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0c3ad0 msgr2=0x7f665c14caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:59.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.464+0000 7f66636f5640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0c3ad0 0x7f665c14caa0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f664c00e840 tx=0x7f664c00ed00 comp rx=0 tx=0).stop 2024-09-19T07:48:59.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.464+0000 7f66636f5640 1 -- 172.21.15.83:0/3791689314 shutdown_connections 2024-09-19T07:48:59.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.464+0000 7f66636f5640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f66400678c0 0x7f6640069d80 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.464+0000 7f66636f5640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f665c0aa570 0x7f665c14fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.464+0000 7f66636f5640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0c3ad0 0x7f665c14caa0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.464+0000 7f66636f5640 1 --2- 172.21.15.83:0/3791689314 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f66480039c0 0x7f665c1eeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.464+0000 7f66636f5640 1 -- 172.21.15.83:0/3791689314 >> 172.21.15.83:0/3791689314 conn(0x7f665c06bc80 msgr2=0x7f665c0c3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:59.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.464+0000 7f66636f5640 1 -- 172.21.15.83:0/3791689314 shutdown_connections 2024-09-19T07:48:59.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.464+0000 7f66636f5640 1 -- 172.21.15.83:0/3791689314 wait complete. 2024-09-19T07:48:59.479 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-19T07:48:59.479 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:48:59.479 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:48:59.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.592+0000 7f299baf5640 1 Processor -- start 2024-09-19T07:48:59.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299baf5640 1 -- start start 2024-09-19T07:48:59.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299baf5640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f29940b0d00 0x7f2994159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299baf5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f299406c600 0x7f29940c8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299baf5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29940c6180 con 0x7f29940c9660 2024-09-19T07:48:59.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299baf5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f29940c69b0 con 0x7f299406c600 2024-09-19T07:48:59.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299baf5640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f299415a130 con 0x7f29940b0d00 2024-09-19T07:48:59.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299a092640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f299406c600 0x7f29940c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299a092640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f299406c600 0x7f29940c8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55584/0 (socket says 172.21.15.83:55584) 2024-09-19T07:48:59.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299a092640 1 -- 172.21.15.83:0/2667217630 learned_addr learned my addr 172.21.15.83:0/2667217630 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:59.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299a092640 1 -- 172.21.15.83:0/2667217630 >> v2:172.21.15.184:3300/0 conn(0x7f29940b0d00 msgr2=0x7f2994159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:59.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f2999891640 1 --2- 172.21.15.83:0/2667217630 >> v2:172.21.15.184:3300/0 conn(0x7f29940b0d00 0x7f2994159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299a092640 1 --2- 172.21.15.83:0/2667217630 >> v2:172.21.15.184:3300/0 conn(0x7f29940b0d00 0x7f2994159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299a092640 1 -- 172.21.15.83:0/2667217630 >> v1:172.21.15.184:6789/0 conn(0x7f29940c9660 legacy=0x7f29941563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:48:59.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299a092640 1 -- 172.21.15.83:0/2667217630 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2994155bf0 con 0x7f299406c600 2024-09-19T07:48:59.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299a092640 1 --2- 172.21.15.83:0/2667217630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299406c600 0x7f29940c8210 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f29940c80c0 tx=0x7f298c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66741df808e84306 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:59.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299888f640 1 -- 172.21.15.83:0/2667217630 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c0a55e0 con 0x7f299406c600 2024-09-19T07:48:59.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299888f640 1 -- 172.21.15.83:0/2667217630 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f298c0a5780 con 0x7f299406c600 2024-09-19T07:48:59.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.596+0000 7f299888f640 1 -- 172.21.15.83:0/2667217630 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c0a5b50 con 0x7f299406c600 2024-09-19T07:48:59.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 -- 172.21.15.83:0/2667217630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299406c600 msgr2=0x7f29940c8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:59.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 --2- 172.21.15.83:0/2667217630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299406c600 0x7f29940c8210 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f29940c80c0 tx=0x7f298c01c920 comp rx=0 tx=0).stop 2024-09-19T07:48:59.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 -- 172.21.15.83:0/2667217630 shutdown_connections 2024-09-19T07:48:59.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 --2- 172.21.15.83:0/2667217630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299406c600 0x7f29940c8210 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 --2- 172.21.15.83:0/2667217630 >> v2:172.21.15.184:3300/0 conn(0x7f29940b0d00 0x7f2994159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 -- 172.21.15.83:0/2667217630 >> 172.21.15.83:0/2667217630 conn(0x7f29940722a0 msgr2=0x7f29940726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:59.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 -- 172.21.15.83:0/2667217630 shutdown_connections 2024-09-19T07:48:59.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 -- 172.21.15.83:0/2667217630 wait complete. 2024-09-19T07:48:59.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 Processor -- start 2024-09-19T07:48:59.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 -- start start 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299406c600 0x7f29940c4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29940b0d00 0x7f29941f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29940c9660 0x7f29941f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29940b10e0 con 0x7f299406c600 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2994155eb0 con 0x7f29940c9660 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299baf5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2994162d90 con 0x7f29940b0d00 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f2999891640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299406c600 0x7f29940c4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f2999891640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299406c600 0x7f29940c4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55592/0 (socket says 172.21.15.83:55592) 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f2999891640 1 -- 172.21.15.83:0/840064825 learned_addr learned my addr 172.21.15.83:0/840064825 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f2999090640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29940b0d00 0x7f29941f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299a092640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29940c9660 0x7f29941f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f2999891640 1 -- 172.21.15.83:0/840064825 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29940b0d00 msgr2=0x7f29941f1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f2999891640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29940b0d00 0x7f29941f1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f2999891640 1 -- 172.21.15.83:0/840064825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29940c9660 msgr2=0x7f29941f1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f2999891640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29940c9660 0x7f29941f1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f2999891640 1 -- 172.21.15.83:0/840064825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29940c52a0 con 0x7f299406c600 2024-09-19T07:48:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.600+0000 7f299a092640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29940c9660 0x7f29941f1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:48:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.604+0000 7f2999891640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299406c600 0x7f29940c4d60 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f299000c7f0 tx=0x7f299000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.604+0000 7f298a7fc640 1 -- 172.21.15.83:0/840064825 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990015070 con 0x7f299406c600 2024-09-19T07:48:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.604+0000 7f298a7fc640 1 -- 172.21.15.83:0/840064825 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2990007820 con 0x7f299406c600 2024-09-19T07:48:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.604+0000 7f299baf5640 1 -- 172.21.15.83:0/840064825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29940c1cb0 con 0x7f299406c600 2024-09-19T07:48:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.604+0000 7f298a7fc640 1 -- 172.21.15.83:0/840064825 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990007c50 con 0x7f299406c600 2024-09-19T07:48:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.604+0000 7f299baf5640 1 -- 172.21.15.83:0/840064825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29940bf3a0 con 0x7f299406c600 2024-09-19T07:48:59.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.604+0000 7f299baf5640 1 -- 172.21.15.83:0/840064825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29940c8210 con 0x7f299406c600 2024-09-19T07:48:59.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.604+0000 7f298a7fc640 1 -- 172.21.15.83:0/840064825 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2990018070 con 0x7f299406c600 2024-09-19T07:48:59.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.608+0000 7f298a7fc640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29780678f0 0x7f2978069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:48:59.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.608+0000 7f2999090640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29780678f0 0x7f2978069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:48:59.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.608+0000 7f298a7fc640 1 -- 172.21.15.83:0/840064825 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f2990086c70 con 0x7f299406c600 2024-09-19T07:48:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.608+0000 7f2999090640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29780678f0 0x7f2978069db0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2984007fd0 tx=0x7f2984009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:48:59.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.612+0000 7f298a7fc640 1 -- 172.21.15.83:0/840064825 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f29900566e0 con 0x7f299406c600 2024-09-19T07:48:59.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.800+0000 7f299baf5640 1 -- 172.21.15.83:0/840064825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2994069e00 con 0x7f299406c600 2024-09-19T07:48:59.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.800+0000 7f298a7fc640 1 -- 172.21.15.83:0/840064825 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) ==== 95+0+5711 (secure 0 0 0) 0x7f2990055fe0 con 0x7f299406c600 2024-09-19T07:48:59.800 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:48:59.801 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":202,"btime":"2024-09-19T07:48:59:753942+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:48:58.894629+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:48:59.801 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 202 2024-09-19T07:48:59.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.804+0000 7f299baf5640 1 -- 172.21.15.83:0/840064825 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29780678f0 msgr2=0x7f2978069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:59.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.804+0000 7f299baf5640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29780678f0 0x7f2978069db0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2984007fd0 tx=0x7f2984009040 comp rx=0 tx=0).stop 2024-09-19T07:48:59.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.804+0000 7f299baf5640 1 -- 172.21.15.83:0/840064825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299406c600 msgr2=0x7f29940c4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:48:59.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.804+0000 7f299baf5640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299406c600 0x7f29940c4d60 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f299000c7f0 tx=0x7f299000ccb0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.804+0000 7f299baf5640 1 -- 172.21.15.83:0/840064825 shutdown_connections 2024-09-19T07:48:59.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.804+0000 7f299baf5640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29780678f0 0x7f2978069db0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.804+0000 7f299baf5640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29940c9660 0x7f29941f1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.804+0000 7f299baf5640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29940b0d00 0x7f29941f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.804+0000 7f299baf5640 1 --2- 172.21.15.83:0/840064825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f299406c600 0x7f29940c4d60 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:48:59.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.804+0000 7f299baf5640 1 -- 172.21.15.83:0/840064825 >> 172.21.15.83:0/840064825 conn(0x7f29940722a0 msgr2=0x7f2994154ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:48:59.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.804+0000 7f299baf5640 1 -- 172.21.15.83:0/840064825 shutdown_connections 2024-09-19T07:48:59.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:48:59.804+0000 7f299baf5640 1 -- 172.21.15.83:0/840064825 wait complete. 2024-09-19T07:48:59.818 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:48:59:753942+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 202, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 202, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8265": { "addr": "172.21.15.83:6837/3426675376", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6836", "nonce": 3426675376, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 3426675376, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8265, "incarnation": 201, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:48:58.894629+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8265 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 199, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8196": { "addr": "172.21.15.83:6835/3580797659", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 3580797659, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 3580797659, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8196, "incarnation": 196, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:48:26.723707+0000", "qdb_cluster": [ 8196 ], "qdb_leader": 8196, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8196 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:49:01.801 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-19T07:49:01.801 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-09-19T07:49:01.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:49:01.828 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-09-19T07:49:01.828 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-09-19T07:49:01.845 INFO:tasks.mds_thrash:joining mds_thrasher 2024-09-19T07:49:01.845 ERROR:tasks.mds_thrash.fs.[b]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa/tasks/mds_thrash.py", line 121, in _run self.do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa/tasks/mds_thrash.py", line 237, in do_thrash self.sleep_unless_stopped(delay) File "/home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-09-19T07:49:01.846 INFO:tasks.mds_thrash:done joining 2024-09-19T07:49:01.846 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-09-19T07:49:01.859 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-09-19T07:49:01.860 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:49:01.860 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-19T07:49:01.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.944+0000 7f0763cd3640 1 Processor -- start 2024-09-19T07:49:01.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0763cd3640 1 -- start start 2024-09-19T07:49:01.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0763cd3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f075c150290 0x7f075c0c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:01.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0763cd3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f075c152b20 0x7f075c0c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:01.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0763cd3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f075c0c45c0 0x7f075c153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:01.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0763cd3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f075c0716c0 con 0x7f075c0c45c0 2024-09-19T07:49:01.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0763cd3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f075c0c3920 con 0x7f075c150290 2024-09-19T07:49:01.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0763cd3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f075c0c3490 con 0x7f075c152b20 2024-09-19T07:49:01.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0762270640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f075c0c45c0 0x7f075c153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:01.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0762270640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f075c0c45c0 0x7f075c153eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36588/0 (socket says 172.21.15.83:36588) 2024-09-19T07:49:01.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0762270640 1 -- 172.21.15.83:0/2962683543 learned_addr learned my addr 172.21.15.83:0/2962683543 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:01.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0761a6f640 1 --2- 172.21.15.83:0/2962683543 >> v2:172.21.15.184:3300/0 conn(0x7f075c150290 0x7f075c0c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:01.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f076126e640 1 --2- 172.21.15.83:0/2962683543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f075c152b20 0x7f075c0c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:01.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0762270640 1 -- 172.21.15.83:0/2962683543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f075c152b20 msgr2=0x7f075c0c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:01.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0762270640 1 --2- 172.21.15.83:0/2962683543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f075c152b20 0x7f075c0c4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:01.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0762270640 1 -- 172.21.15.83:0/2962683543 >> v2:172.21.15.184:3300/0 conn(0x7f075c150290 msgr2=0x7f075c0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:01.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0762270640 1 --2- 172.21.15.83:0/2962683543 >> v2:172.21.15.184:3300/0 conn(0x7f075c150290 0x7f075c0c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:01.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0762270640 1 -- 172.21.15.83:0/2962683543 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075c150670 con 0x7f075c0c45c0 2024-09-19T07:49:01.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0761a6f640 1 --2- 172.21.15.83:0/2962683543 >> v2:172.21.15.184:3300/0 conn(0x7f075c150290 0x7f075c0c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:01.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0762270640 1 --2- 172.21.15.83:0/2962683543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c0c45c0 0x7f075c153eb0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f075c06f590 tx=0x7f075401ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e0662f9f2b920f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:01.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0760a6d640 1 -- 172.21.15.83:0/2962683543 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075400b2d0 con 0x7f075c0c45c0 2024-09-19T07:49:01.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0760a6d640 1 -- 172.21.15.83:0/2962683543 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f075400b470 con 0x7f075c0c45c0 2024-09-19T07:49:01.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.948+0000 7f0760a6d640 1 -- 172.21.15.83:0/2962683543 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540a7c70 con 0x7f075c0c45c0 2024-09-19T07:49:01.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 -- 172.21.15.83:0/2962683543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c0c45c0 msgr2=0x7f075c153eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:01.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 --2- 172.21.15.83:0/2962683543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c0c45c0 0x7f075c153eb0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f075c06f590 tx=0x7f075401ebc0 comp rx=0 tx=0).stop 2024-09-19T07:49:01.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 -- 172.21.15.83:0/2962683543 shutdown_connections 2024-09-19T07:49:01.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 --2- 172.21.15.83:0/2962683543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c0c45c0 0x7f075c153eb0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:01.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 --2- 172.21.15.83:0/2962683543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f075c152b20 0x7f075c0c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:01.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 --2- 172.21.15.83:0/2962683543 >> v2:172.21.15.184:3300/0 conn(0x7f075c150290 0x7f075c0c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:01.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 -- 172.21.15.83:0/2962683543 >> 172.21.15.83:0/2962683543 conn(0x7f075c06bc80 msgr2=0x7f075c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:01.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 -- 172.21.15.83:0/2962683543 shutdown_connections 2024-09-19T07:49:01.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 -- 172.21.15.83:0/2962683543 wait complete. 2024-09-19T07:49:01.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 Processor -- start 2024-09-19T07:49:01.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 -- start start 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f075c0c45c0 0x7f075c15c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f075c150290 0x7f075c146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c152b20 0x7f075c146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f075c0c2530 con 0x7f075c152b20 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f075c0c3120 con 0x7f075c150290 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f075c154800 con 0x7f075c0c45c0 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0762270640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c152b20 0x7f075c146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0762270640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c152b20 0x7f075c146660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36590/0 (socket says 172.21.15.83:36590) 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0762270640 1 -- 172.21.15.83:0/1152781350 learned_addr learned my addr 172.21.15.83:0/1152781350 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f076126e640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f075c150290 0x7f075c146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0761a6f640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f075c0c45c0 0x7f075c15c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0762270640 1 -- 172.21.15.83:0/1152781350 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f075c0c45c0 msgr2=0x7f075c15c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:01.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0762270640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f075c0c45c0 0x7f075c15c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:01.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0762270640 1 -- 172.21.15.83:0/1152781350 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f075c150290 msgr2=0x7f075c146120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:01.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0762270640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f075c150290 0x7f075c146120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:01.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0762270640 1 -- 172.21.15.83:0/1152781350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075c1528c0 con 0x7f075c152b20 2024-09-19T07:49:01.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0762270640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c152b20 0x7f075c146660 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f075401d270 tx=0x7f075401b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:01.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f07527fc640 1 -- 172.21.15.83:0/1152781350 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540be020 con 0x7f075c152b20 2024-09-19T07:49:01.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f07527fc640 1 -- 172.21.15.83:0/1152781350 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0754009800 con 0x7f075c152b20 2024-09-19T07:49:01.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 -- 172.21.15.83:0/1152781350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f075c0737b0 con 0x7f075c152b20 2024-09-19T07:49:01.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f07527fc640 1 -- 172.21.15.83:0/1152781350 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0754009bf0 con 0x7f075c152b20 2024-09-19T07:49:01.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.952+0000 7f0763cd3640 1 -- 172.21.15.83:0/1152781350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f075c149d60 con 0x7f075c152b20 2024-09-19T07:49:01.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.956+0000 7f0763cd3640 1 -- 172.21.15.83:0/1152781350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f075c0c3e90 con 0x7f075c152b20 2024-09-19T07:49:01.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.956+0000 7f07527fc640 1 -- 172.21.15.83:0/1152781350 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f07540c1020 con 0x7f075c152b20 2024-09-19T07:49:01.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.956+0000 7f07527fc640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07400678c0 0x7f0740069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:01.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.956+0000 7f07527fc640 1 -- 172.21.15.83:0/1152781350 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f0754124200 con 0x7f075c152b20 2024-09-19T07:49:01.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.956+0000 7f0761a6f640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07400678c0 0x7f0740069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:01.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.956+0000 7f0761a6f640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07400678c0 0x7f0740069d80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f075800b820 tx=0x7f0758008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:01.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:01.960+0000 7f07527fc640 1 -- 172.21.15.83:0/1152781350 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f07540c8030 con 0x7f075c152b20 2024-09-19T07:49:01.970 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:49:01.970 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-19T07:49:02.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-19T07:49:02.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.160+0000 7f0763cd3640 1 -- 172.21.15.83:0/1152781350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f075c0c1ff0 con 0x7f075c152b20 2024-09-19T07:49:02.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.160+0000 7f07527fc640 1 -- 172.21.15.83:0/1152781350 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+5716 (secure 0 0 0) 0x7f07540187e0 con 0x7f075c152b20 2024-09-19T07:49:02.161 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:02.162 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":204,"btime":"2024-09-19T07:49:01:773318+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:49:01.773306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8265,"qdb_cluster":[8265]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:49:02.162 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 204 2024-09-19T07:49:02.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.164+0000 7f0763cd3640 1 -- 172.21.15.83:0/1152781350 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07400678c0 msgr2=0x7f0740069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.164+0000 7f0763cd3640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07400678c0 0x7f0740069d80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f075800b820 tx=0x7f0758008040 comp rx=0 tx=0).stop 2024-09-19T07:49:02.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.164+0000 7f0763cd3640 1 -- 172.21.15.83:0/1152781350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c152b20 msgr2=0x7f075c146660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.164+0000 7f0763cd3640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c152b20 0x7f075c146660 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f075401d270 tx=0x7f075401b960 comp rx=0 tx=0).stop 2024-09-19T07:49:02.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.164+0000 7f0763cd3640 1 -- 172.21.15.83:0/1152781350 shutdown_connections 2024-09-19T07:49:02.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.164+0000 7f0763cd3640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07400678c0 0x7f0740069d80 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.164+0000 7f0763cd3640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c152b20 0x7f075c146660 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.164+0000 7f0763cd3640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f075c150290 0x7f075c146120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.164+0000 7f0763cd3640 1 --2- 172.21.15.83:0/1152781350 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f075c0c45c0 0x7f075c15c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.164+0000 7f0763cd3640 1 -- 172.21.15.83:0/1152781350 >> 172.21.15.83:0/1152781350 conn(0x7f075c06bc80 msgr2=0x7f075c14abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:02.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.164+0000 7f0763cd3640 1 -- 172.21.15.83:0/1152781350 shutdown_connections 2024-09-19T07:49:02.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.164+0000 7f0763cd3640 1 -- 172.21.15.83:0/1152781350 wait complete. 2024-09-19T07:49:02.185 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=6269 2024-09-19T07:49:02.185 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before reviving mds.d 2024-09-19T07:49:02.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.180+0000 7fdeeed45640 1 Processor -- start 2024-09-19T07:49:02.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.180+0000 7fdeeed45640 1 -- start start 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdeeed45640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fdee80c7660 0x7fdee8156530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdeeed45640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c7a40 0x7fdee8159110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdeeed45640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee8156a70 con 0x7fdee80c5b30 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdeeed45640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fdee81560a0 con 0x7fdee80c7660 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdeeed45640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdee80c4680 con 0x7fdee80c7a40 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdeecae1640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fdee80c7660 0x7fdee8156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdee7fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c7a40 0x7fdee8159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdeed2e2640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7fdee80c5b30 0x7fdee806cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:58968/0 (socket says 172.21.15.83:58968) 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdee7fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c7a40 0x7fdee8159110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:59846/0 (socket says 172.21.15.83:59846) 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdeed2e2640 1 -- 172.21.15.83:0/3270688732 learned_addr learned my addr 172.21.15.83:0/3270688732 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdee7fff640 1 -- 172.21.15.83:0/3270688732 >> v2:172.21.15.184:3300/0 conn(0x7fdee80c7660 msgr2=0x7fdee8156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdee7fff640 1 --2- 172.21.15.83:0/3270688732 >> v2:172.21.15.184:3300/0 conn(0x7fdee80c7660 0x7fdee8156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdee7fff640 1 -- 172.21.15.83:0/3270688732 >> v1:172.21.15.184:6789/0 conn(0x7fdee80c5b30 legacy=0x7fdee806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.184+0000 7fdee7fff640 1 -- 172.21.15.83:0/3270688732 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee80c2ca0 con 0x7fdee80c7a40 2024-09-19T07:49:02.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdee7fff640 1 --2- 172.21.15.83:0/3270688732 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c7a40 0x7fdee8159110 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fded00097c0 tx=0x7fded001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1765e707e8e34f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:02.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdee77fe640 1 -- 172.21.15.83:0/3270688732 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded00b0070 con 0x7fdee80c7a40 2024-09-19T07:49:02.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdee77fe640 1 -- 172.21.15.83:0/3270688732 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded00ab600 con 0x7fdee80c7a40 2024-09-19T07:49:02.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdee77fe640 1 -- 172.21.15.83:0/3270688732 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded00ab9d0 con 0x7fdee80c7a40 2024-09-19T07:49:02.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 -- 172.21.15.83:0/3270688732 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c7a40 msgr2=0x7fdee8159110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 --2- 172.21.15.83:0/3270688732 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c7a40 0x7fdee8159110 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fded00097c0 tx=0x7fded001c410 comp rx=0 tx=0).stop 2024-09-19T07:49:02.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 -- 172.21.15.83:0/3270688732 shutdown_connections 2024-09-19T07:49:02.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 --2- 172.21.15.83:0/3270688732 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c7a40 0x7fdee8159110 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 --2- 172.21.15.83:0/3270688732 >> v2:172.21.15.184:3300/0 conn(0x7fdee80c7660 0x7fdee8156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 -- 172.21.15.83:0/3270688732 >> 172.21.15.83:0/3270688732 conn(0x7fdee80c67f0 msgr2=0x7fdee80c9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 -- 172.21.15.83:0/3270688732 shutdown_connections 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 -- 172.21.15.83:0/3270688732 wait complete. 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 Processor -- start 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 -- start start 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c5b30 0x7fdee814d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdee80c7660 0x7fdee81f1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdee814ddf0 0x7fdee81f2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdee806d3a0 con 0x7fdee814ddf0 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdee80c2930 con 0x7fdee80c7660 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeeed45640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdee806d520 con 0x7fdee80c5b30 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeed2e2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdee814ddf0 0x7fdee81f2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:02.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeed2e2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdee814ddf0 0x7fdee81f2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36592/0 (socket says 172.21.15.83:36592) 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeed2e2640 1 -- 172.21.15.83:0/1211288523 learned_addr learned my addr 172.21.15.83:0/1211288523 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdee7fff640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdee80c7660 0x7fdee81f1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeecae1640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c5b30 0x7fdee814d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeed2e2640 1 -- 172.21.15.83:0/1211288523 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c5b30 msgr2=0x7fdee814d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeed2e2640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c5b30 0x7fdee814d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeed2e2640 1 -- 172.21.15.83:0/1211288523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdee80c7660 msgr2=0x7fdee81f1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeed2e2640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdee80c7660 0x7fdee81f1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeed2e2640 1 -- 172.21.15.83:0/1211288523 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee8070550 con 0x7fdee814ddf0 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeecae1640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c5b30 0x7fdee814d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdee7fff640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdee80c7660 0x7fdee81f1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.188+0000 7fdeed2e2640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdee814ddf0 0x7fdee81f2230 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fdee000c430 tx=0x7fdee000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.192+0000 7fdee57fa640 1 -- 172.21.15.83:0/1211288523 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee0017070 con 0x7fdee814ddf0 2024-09-19T07:49:02.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.192+0000 7fdee57fa640 1 -- 172.21.15.83:0/1211288523 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdee0005cf0 con 0x7fdee814ddf0 2024-09-19T07:49:02.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.192+0000 7fdeeed45640 1 -- 172.21.15.83:0/1211288523 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee8070830 con 0x7fdee814ddf0 2024-09-19T07:49:02.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.192+0000 7fdee57fa640 1 -- 172.21.15.83:0/1211288523 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee00072d0 con 0x7fdee814ddf0 2024-09-19T07:49:02.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.192+0000 7fdeeed45640 1 -- 172.21.15.83:0/1211288523 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee80ac700 con 0x7fdee814ddf0 2024-09-19T07:49:02.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.192+0000 7fdeeed45640 1 -- 172.21.15.83:0/1211288523 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdee8156530 con 0x7fdee814ddf0 2024-09-19T07:49:02.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.192+0000 7fdee57fa640 1 -- 172.21.15.83:0/1211288523 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdee001a070 con 0x7fdee814ddf0 2024-09-19T07:49:02.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.192+0000 7fdee57fa640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdec80678c0 0x7fdec8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:02.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.192+0000 7fdee57fa640 1 -- 172.21.15.83:0/1211288523 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fdee0086470 con 0x7fdee814ddf0 2024-09-19T07:49:02.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.192+0000 7fdeecae1640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdec80678c0 0x7fdec8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:02.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.192+0000 7fdeecae1640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdec80678c0 0x7fdec8069d80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fdee81589c0 tx=0x7fded800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:02.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.196+0000 7fdee57fa640 1 -- 172.21.15.83:0/1211288523 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdee8156a70 con 0x7fdee814ddf0 2024-09-19T07:49:02.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.372+0000 7fdeeed45640 1 -- 172.21.15.83:0/1211288523 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdee80c9ee0 con 0x7fdee814ddf0 2024-09-19T07:49:02.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.372+0000 7fdee57fa640 1 -- 172.21.15.83:0/1211288523 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 80 entries v65) ==== 81+0+4890 (secure 0 0 0) 0x7fdee0055910 con 0x7fdee814ddf0 2024-09-19T07:49:02.373 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/2502231963 2024-09-20T07:48:58.353983+0000 2024-09-19T07:49:02.373 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1602509240 2024-09-20T07:48:23.346687+0000 2024-09-19T07:49:02.373 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1602509240 2024-09-20T07:48:23.346687+0000 2024-09-19T07:49:02.373 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/119695014 2024-09-20T07:47:58.342001+0000 2024-09-19T07:49:02.374 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/1207928072 2024-09-20T07:47:03.329199+0000 2024-09-19T07:49:02.374 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/3490578864 2024-09-20T07:45:28.310975+0000 2024-09-19T07:49:02.374 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1429520945 2024-09-20T07:45:13.307981+0000 2024-09-19T07:49:02.374 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1429520945 2024-09-20T07:45:13.307981+0000 2024-09-19T07:49:02.374 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/3311721080 2024-09-20T07:44:18.290353+0000 2024-09-19T07:49:02.374 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/3311721080 2024-09-20T07:44:18.290353+0000 2024-09-19T07:49:02.374 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1990119490 2024-09-20T07:42:38.260871+0000 2024-09-19T07:49:02.375 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1990119490 2024-09-20T07:42:38.260871+0000 2024-09-19T07:49:02.375 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/2776269095 2024-09-20T07:42:08.252858+0000 2024-09-19T07:49:02.375 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/1335198090 2024-09-20T07:41:33.245545+0000 2024-09-19T07:49:02.375 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/1335198090 2024-09-20T07:41:33.245545+0000 2024-09-19T07:49:02.375 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/365493751 2024-09-20T07:41:08.239813+0000 2024-09-19T07:49:02.376 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/119695014 2024-09-20T07:47:58.342001+0000 2024-09-19T07:49:02.376 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/104517500 2024-09-20T07:28:38.045210+0000 2024-09-19T07:49:02.376 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/2326757290 2024-09-20T07:40:23.230841+0000 2024-09-19T07:49:02.376 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/872719174 2024-09-20T07:28:08.038207+0000 2024-09-19T07:49:02.376 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/3763485069 2024-09-20T07:27:18.018593+0000 2024-09-19T07:49:02.376 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/872719174 2024-09-20T07:28:08.038207+0000 2024-09-19T07:49:02.377 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/1207928072 2024-09-20T07:47:03.329199+0000 2024-09-19T07:49:02.377 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/455754420 2024-09-20T07:27:38.022965+0000 2024-09-19T07:49:02.377 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/3763485069 2024-09-20T07:27:18.018593+0000 2024-09-19T07:49:02.377 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/1773885233 2024-09-20T07:22:22.945705+0000 2024-09-19T07:49:02.377 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/144714632 2024-09-20T07:25:36.204686+0000 2024-09-19T07:49:02.378 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/26497755 2024-09-20T07:25:27.991655+0000 2024-09-19T07:49:02.378 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/2993984999 2024-09-20T07:32:38.105001+0000 2024-09-19T07:49:02.378 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/4265456507 2024-09-20T07:22:32.948957+0000 2024-09-19T07:49:02.378 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1072903448 2024-09-20T07:33:28.117348+0000 2024-09-19T07:49:02.378 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/3442755673 2024-09-20T07:24:47.982274+0000 2024-09-19T07:49:02.379 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/3490578864 2024-09-20T07:45:28.310975+0000 2024-09-19T07:49:02.379 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/149004692 2024-09-20T07:20:57.928172+0000 2024-09-19T07:49:02.379 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/3841282254 2024-09-20T07:30:48.076144+0000 2024-09-19T07:49:02.379 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/2502231963 2024-09-20T07:48:58.353983+0000 2024-09-19T07:49:02.379 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1229829432 2024-09-20T07:29:48.061079+0000 2024-09-19T07:49:02.379 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1819616410 2024-09-20T07:21:07.930616+0000 2024-09-19T07:49:02.379 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1819616410 2024-09-20T07:21:07.930616+0000 2024-09-19T07:49:02.380 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/149004692 2024-09-20T07:20:57.928172+0000 2024-09-19T07:49:02.380 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/1773885233 2024-09-20T07:22:22.945705+0000 2024-09-19T07:49:02.380 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/2112478738 2024-09-20T07:35:28.148108+0000 2024-09-19T07:49:02.380 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/2993984999 2024-09-20T07:32:38.105001+0000 2024-09-19T07:49:02.380 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/26497755 2024-09-20T07:25:27.991655+0000 2024-09-19T07:49:02.380 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/2898687304 2024-09-20T07:34:18.130590+0000 2024-09-19T07:49:02.380 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2776269095 2024-09-20T07:42:08.252858+0000 2024-09-19T07:49:02.380 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1049139637 2024-09-20T07:23:37.966033+0000 2024-09-19T07:49:02.381 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/3442755673 2024-09-20T07:24:47.982274+0000 2024-09-19T07:49:02.381 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/4265456507 2024-09-20T07:22:32.948957+0000 2024-09-19T07:49:02.381 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/2898687304 2024-09-20T07:34:18.130590+0000 2024-09-19T07:49:02.381 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/4115191178 2024-09-20T07:23:52.164421+0000 2024-09-19T07:49:02.381 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/4115191178 2024-09-20T07:23:52.164421+0000 2024-09-19T07:49:02.381 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1774585869 2024-09-20T07:23:07.959477+0000 2024-09-19T07:49:02.381 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/144714632 2024-09-20T07:25:36.204686+0000 2024-09-19T07:49:02.381 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/104517500 2024-09-20T07:28:38.045210+0000 2024-09-19T07:49:02.382 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1049139637 2024-09-20T07:23:37.966033+0000 2024-09-19T07:49:02.382 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/2004814399 2024-09-20T07:30:48.076245+0000 2024-09-19T07:49:02.382 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1229829432 2024-09-20T07:29:48.061079+0000 2024-09-19T07:49:02.382 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2004814399 2024-09-20T07:30:48.076245+0000 2024-09-19T07:49:02.382 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/3841282254 2024-09-20T07:30:48.076144+0000 2024-09-19T07:49:02.382 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/2581473241 2024-09-20T07:31:18.082843+0000 2024-09-19T07:49:02.382 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/455754420 2024-09-20T07:27:38.022965+0000 2024-09-19T07:49:02.382 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1079027718 2024-09-20T07:35:43.151965+0000 2024-09-19T07:49:02.383 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/2581473241 2024-09-20T07:31:18.082843+0000 2024-09-19T07:49:02.383 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/2326757290 2024-09-20T07:40:23.230841+0000 2024-09-19T07:49:02.383 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1072903448 2024-09-20T07:33:28.117348+0000 2024-09-19T07:49:02.383 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1774585869 2024-09-20T07:23:07.959477+0000 2024-09-19T07:49:02.383 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/1519298917 2024-09-20T07:37:18.179638+0000 2024-09-19T07:49:02.383 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1203406606 2024-09-20T07:34:08.127781+0000 2024-09-19T07:49:02.384 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/365493751 2024-09-20T07:41:08.239813+0000 2024-09-19T07:49:02.384 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1203406606 2024-09-20T07:34:08.127781+0000 2024-09-19T07:49:02.384 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1079027718 2024-09-20T07:35:43.151965+0000 2024-09-19T07:49:02.384 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/2112478738 2024-09-20T07:35:28.148108+0000 2024-09-19T07:49:02.384 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/1519298917 2024-09-20T07:37:18.179638+0000 2024-09-19T07:49:02.384 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/3707393259 2024-09-20T07:37:18.179736+0000 2024-09-19T07:49:02.384 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/3707393259 2024-09-20T07:37:18.179736+0000 2024-09-19T07:49:02.385 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/2168217861 2024-09-20T07:38:08.197381+0000 2024-09-19T07:49:02.385 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2168217861 2024-09-20T07:38:08.197381+0000 2024-09-19T07:49:02.385 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/381468017 2024-09-20T07:39:03.210538+0000 2024-09-19T07:49:02.385 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/381468017 2024-09-20T07:39:03.210538+0000 2024-09-19T07:49:02.388 INFO:teuthology.orchestra.run.smithi083.stderr:listed 80 entries 2024-09-19T07:49:02.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.376+0000 7fdeeed45640 1 -- 172.21.15.83:0/1211288523 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdec80678c0 msgr2=0x7fdec8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.376+0000 7fdeeed45640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdec80678c0 0x7fdec8069d80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fdee81589c0 tx=0x7fded800b040 comp rx=0 tx=0).stop 2024-09-19T07:49:02.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.376+0000 7fdeeed45640 1 -- 172.21.15.83:0/1211288523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdee814ddf0 msgr2=0x7fdee81f2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.376+0000 7fdeeed45640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdee814ddf0 0x7fdee81f2230 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fdee000c430 tx=0x7fdee000c8f0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.376+0000 7fdeeed45640 1 -- 172.21.15.83:0/1211288523 shutdown_connections 2024-09-19T07:49:02.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.376+0000 7fdeeed45640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdec80678c0 0x7fdec8069d80 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.376+0000 7fdeeed45640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdee814ddf0 0x7fdee81f2230 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.376+0000 7fdeeed45640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdee80c7660 0x7fdee81f1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.376+0000 7fdeeed45640 1 --2- 172.21.15.83:0/1211288523 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdee80c5b30 0x7fdee814d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.376+0000 7fdeeed45640 1 -- 172.21.15.83:0/1211288523 >> 172.21.15.83:0/1211288523 conn(0x7fdee80c67f0 msgr2=0x7fdee806ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:02.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.376+0000 7fdeeed45640 1 -- 172.21.15.83:0/1211288523 shutdown_connections 2024-09-19T07:49:02.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.376+0000 7fdeeed45640 1 -- 172.21.15.83:0/1211288523 wait complete. 2024-09-19T07:49:02.392 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-19T07:49:02.392 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-19T07:49:02.440 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-19T07:49:02.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856fa52640 1 Processor -- start 2024-09-19T07:49:02.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856fa52640 1 -- start start 2024-09-19T07:49:02.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856fa52640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f85681534b0 0x7f85680c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:02.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856fa52640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f85680c2b10 0x7f856815d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:02.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856fa52640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85680c3810 con 0x7f8568150060 2024-09-19T07:49:02.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856fa52640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f8568150440 con 0x7f85680c2b10 2024-09-19T07:49:02.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856fa52640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f85681506c0 con 0x7f85681534b0 2024-09-19T07:49:02.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856dfef640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f85680c2b10 0x7f856815d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:02.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856dfef640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f85680c2b10 0x7f856815d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36604/0 (socket says 172.21.15.83:36604) 2024-09-19T07:49:02.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856dfef640 1 -- 172.21.15.83:0/367944419 learned_addr learned my addr 172.21.15.83:0/367944419 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:02.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856cfed640 1 --2- 172.21.15.83:0/367944419 >> v2:172.21.15.184:3300/0 conn(0x7f85681534b0 0x7f85680c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:02.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856dfef640 1 -- 172.21.15.83:0/367944419 >> v2:172.21.15.184:3300/0 conn(0x7f85681534b0 msgr2=0x7f85680c25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:02.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856dfef640 1 --2- 172.21.15.83:0/367944419 >> v2:172.21.15.184:3300/0 conn(0x7f85681534b0 0x7f85680c25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856dfef640 1 -- 172.21.15.83:0/367944419 >> v1:172.21.15.184:6789/0 conn(0x7f8568150060 legacy=0x7f85680c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:02.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.612+0000 7f856dfef640 1 -- 172.21.15.83:0/367944419 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85680c3ab0 con 0x7f85680c2b10 2024-09-19T07:49:02.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856dfef640 1 --2- 172.21.15.83:0/367944419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85680c2b10 0x7f856815d3b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f85680c1850 tx=0x7f856001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e27fb856d44c825 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:02.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f855ffff640 1 -- 172.21.15.83:0/367944419 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85600a55c0 con 0x7f85680c2b10 2024-09-19T07:49:02.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f855ffff640 1 -- 172.21.15.83:0/367944419 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85600a5760 con 0x7f85680c2b10 2024-09-19T07:49:02.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f855ffff640 1 -- 172.21.15.83:0/367944419 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85600a5b30 con 0x7f85680c2b10 2024-09-19T07:49:02.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 -- 172.21.15.83:0/367944419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85680c2b10 msgr2=0x7f856815d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 --2- 172.21.15.83:0/367944419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85680c2b10 0x7f856815d3b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f85680c1850 tx=0x7f856001c900 comp rx=0 tx=0).stop 2024-09-19T07:49:02.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 -- 172.21.15.83:0/367944419 shutdown_connections 2024-09-19T07:49:02.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 --2- 172.21.15.83:0/367944419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85680c2b10 0x7f856815d3b0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 --2- 172.21.15.83:0/367944419 >> v2:172.21.15.184:3300/0 conn(0x7f85681534b0 0x7f85680c25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 -- 172.21.15.83:0/367944419 >> 172.21.15.83:0/367944419 conn(0x7f856806bc80 msgr2=0x7f856806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:02.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 -- 172.21.15.83:0/367944419 shutdown_connections 2024-09-19T07:49:02.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 -- 172.21.15.83:0/367944419 wait complete. 2024-09-19T07:49:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 Processor -- start 2024-09-19T07:49:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 -- start start 2024-09-19T07:49:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85680c2b10 0x7f85680c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8568150060 0x7f85680c0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85681534b0 0x7f85681f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f85680aae10 con 0x7f85681534b0 2024-09-19T07:49:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f85680c3d90 con 0x7f85680c2b10 2024-09-19T07:49:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f85680aaf90 con 0x7f8568150060 2024-09-19T07:49:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856dfef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85681534b0 0x7f85681f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856dfef640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85681534b0 0x7f85681f1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36618/0 (socket says 172.21.15.83:36618) 2024-09-19T07:49:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856dfef640 1 -- 172.21.15.83:0/956682748 learned_addr learned my addr 172.21.15.83:0/956682748 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856d7ee640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85680c2b10 0x7f85680c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:02.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856dfef640 1 -- 172.21.15.83:0/956682748 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8568150060 msgr2=0x7f85680c0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856dfef640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8568150060 0x7f85680c0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856dfef640 1 -- 172.21.15.83:0/956682748 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85680c2b10 msgr2=0x7f85680c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856dfef640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85680c2b10 0x7f85680c5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856dfef640 1 -- 172.21.15.83:0/956682748 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85680c65a0 con 0x7f85681534b0 2024-09-19T07:49:02.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856dfef640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85681534b0 0x7f85681f1f30 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f856001a530 tx=0x7f85600196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:02.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f855dffb640 1 -- 172.21.15.83:0/956682748 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85600bc020 con 0x7f85681534b0 2024-09-19T07:49:02.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f855dffb640 1 -- 172.21.15.83:0/956682748 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85600b8690 con 0x7f85681534b0 2024-09-19T07:49:02.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 -- 172.21.15.83:0/956682748 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8568152670 con 0x7f85681534b0 2024-09-19T07:49:02.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f855dffb640 1 -- 172.21.15.83:0/956682748 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8560009510 con 0x7f85681534b0 2024-09-19T07:49:02.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.616+0000 7f856fa52640 1 -- 172.21.15.83:0/956682748 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8568147de0 con 0x7f85681534b0 2024-09-19T07:49:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.620+0000 7f856fa52640 1 -- 172.21.15.83:0/956682748 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85680c1ab0 con 0x7f85681534b0 2024-09-19T07:49:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.620+0000 7f855dffb640 1 -- 172.21.15.83:0/956682748 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f85600bf020 con 0x7f85681534b0 2024-09-19T07:49:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.620+0000 7f855dffb640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f854c0678f0 0x7f854c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:02.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.620+0000 7f855dffb640 1 -- 172.21.15.83:0/956682748 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f8560121ec0 con 0x7f85681534b0 2024-09-19T07:49:02.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.620+0000 7f856d7ee640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f854c0678f0 0x7f854c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:02.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.620+0000 7f856d7ee640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f854c0678f0 0x7f854c069db0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f856400a750 tx=0x7f8564008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.624+0000 7f855dffb640 1 -- 172.21.15.83:0/956682748 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f8560016580 con 0x7f85681534b0 2024-09-19T07:49:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.800+0000 7f856fa52640 1 -- 172.21.15.83:0/956682748 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f85680c3450 con 0x7f85681534b0 2024-09-19T07:49:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.800+0000 7f855dffb640 1 -- 172.21.15.83:0/956682748 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 80 entries v65) ==== 81+0+4890 (secure 0 0 0) 0x7f85600f19d0 con 0x7f85681534b0 2024-09-19T07:49:02.801 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/2502231963 2024-09-20T07:48:58.353983+0000 2024-09-19T07:49:02.802 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1602509240 2024-09-20T07:48:23.346687+0000 2024-09-19T07:49:02.802 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1602509240 2024-09-20T07:48:23.346687+0000 2024-09-19T07:49:02.802 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/119695014 2024-09-20T07:47:58.342001+0000 2024-09-19T07:49:02.802 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/1207928072 2024-09-20T07:47:03.329199+0000 2024-09-19T07:49:02.802 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/3490578864 2024-09-20T07:45:28.310975+0000 2024-09-19T07:49:02.802 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1429520945 2024-09-20T07:45:13.307981+0000 2024-09-19T07:49:02.802 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1429520945 2024-09-20T07:45:13.307981+0000 2024-09-19T07:49:02.802 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/3311721080 2024-09-20T07:44:18.290353+0000 2024-09-19T07:49:02.802 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/3311721080 2024-09-20T07:44:18.290353+0000 2024-09-19T07:49:02.802 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1990119490 2024-09-20T07:42:38.260871+0000 2024-09-19T07:49:02.802 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1990119490 2024-09-20T07:42:38.260871+0000 2024-09-19T07:49:02.802 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/2776269095 2024-09-20T07:42:08.252858+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/1335198090 2024-09-20T07:41:33.245545+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/1335198090 2024-09-20T07:41:33.245545+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/365493751 2024-09-20T07:41:08.239813+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/119695014 2024-09-20T07:47:58.342001+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/104517500 2024-09-20T07:28:38.045210+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/2326757290 2024-09-20T07:40:23.230841+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/872719174 2024-09-20T07:28:08.038207+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/3763485069 2024-09-20T07:27:18.018593+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/872719174 2024-09-20T07:28:08.038207+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/1207928072 2024-09-20T07:47:03.329199+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/455754420 2024-09-20T07:27:38.022965+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/3763485069 2024-09-20T07:27:18.018593+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/1773885233 2024-09-20T07:22:22.945705+0000 2024-09-19T07:49:02.803 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/144714632 2024-09-20T07:25:36.204686+0000 2024-09-19T07:49:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/26497755 2024-09-20T07:25:27.991655+0000 2024-09-19T07:49:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/2993984999 2024-09-20T07:32:38.105001+0000 2024-09-19T07:49:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/4265456507 2024-09-20T07:22:32.948957+0000 2024-09-19T07:49:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1072903448 2024-09-20T07:33:28.117348+0000 2024-09-19T07:49:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/3442755673 2024-09-20T07:24:47.982274+0000 2024-09-19T07:49:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/3490578864 2024-09-20T07:45:28.310975+0000 2024-09-19T07:49:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/149004692 2024-09-20T07:20:57.928172+0000 2024-09-19T07:49:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/3841282254 2024-09-20T07:30:48.076144+0000 2024-09-19T07:49:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/2502231963 2024-09-20T07:48:58.353983+0000 2024-09-19T07:49:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1229829432 2024-09-20T07:29:48.061079+0000 2024-09-19T07:49:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1819616410 2024-09-20T07:21:07.930616+0000 2024-09-19T07:49:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1819616410 2024-09-20T07:21:07.930616+0000 2024-09-19T07:49:02.805 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/149004692 2024-09-20T07:20:57.928172+0000 2024-09-19T07:49:02.805 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/1773885233 2024-09-20T07:22:22.945705+0000 2024-09-19T07:49:02.805 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/2112478738 2024-09-20T07:35:28.148108+0000 2024-09-19T07:49:02.805 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/2993984999 2024-09-20T07:32:38.105001+0000 2024-09-19T07:49:02.805 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/26497755 2024-09-20T07:25:27.991655+0000 2024-09-19T07:49:02.805 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/2898687304 2024-09-20T07:34:18.130590+0000 2024-09-19T07:49:02.805 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2776269095 2024-09-20T07:42:08.252858+0000 2024-09-19T07:49:02.805 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1049139637 2024-09-20T07:23:37.966033+0000 2024-09-19T07:49:02.805 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/3442755673 2024-09-20T07:24:47.982274+0000 2024-09-19T07:49:02.805 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/4265456507 2024-09-20T07:22:32.948957+0000 2024-09-19T07:49:02.805 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/2898687304 2024-09-20T07:34:18.130590+0000 2024-09-19T07:49:02.805 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/4115191178 2024-09-20T07:23:52.164421+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/4115191178 2024-09-20T07:23:52.164421+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1774585869 2024-09-20T07:23:07.959477+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/144714632 2024-09-20T07:25:36.204686+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/104517500 2024-09-20T07:28:38.045210+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1049139637 2024-09-20T07:23:37.966033+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/2004814399 2024-09-20T07:30:48.076245+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1229829432 2024-09-20T07:29:48.061079+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2004814399 2024-09-20T07:30:48.076245+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/3841282254 2024-09-20T07:30:48.076144+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/2581473241 2024-09-20T07:31:18.082843+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/455754420 2024-09-20T07:27:38.022965+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1079027718 2024-09-20T07:35:43.151965+0000 2024-09-19T07:49:02.806 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/2581473241 2024-09-20T07:31:18.082843+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/2326757290 2024-09-20T07:40:23.230841+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1072903448 2024-09-20T07:33:28.117348+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1774585869 2024-09-20T07:23:07.959477+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/1519298917 2024-09-20T07:37:18.179638+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1203406606 2024-09-20T07:34:08.127781+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/365493751 2024-09-20T07:41:08.239813+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1203406606 2024-09-20T07:34:08.127781+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1079027718 2024-09-20T07:35:43.151965+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/2112478738 2024-09-20T07:35:28.148108+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/1519298917 2024-09-20T07:37:18.179638+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/3707393259 2024-09-20T07:37:18.179736+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/3707393259 2024-09-20T07:37:18.179736+0000 2024-09-19T07:49:02.807 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/2168217861 2024-09-20T07:38:08.197381+0000 2024-09-19T07:49:02.808 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2168217861 2024-09-20T07:38:08.197381+0000 2024-09-19T07:49:02.808 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/381468017 2024-09-20T07:39:03.210538+0000 2024-09-19T07:49:02.808 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/381468017 2024-09-20T07:39:03.210538+0000 2024-09-19T07:49:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:listed 80 entries 2024-09-19T07:49:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.804+0000 7f856fa52640 1 -- 172.21.15.83:0/956682748 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f854c0678f0 msgr2=0x7f854c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.804+0000 7f856fa52640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f854c0678f0 0x7f854c069db0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f856400a750 tx=0x7f8564008040 comp rx=0 tx=0).stop 2024-09-19T07:49:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.804+0000 7f856fa52640 1 -- 172.21.15.83:0/956682748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85681534b0 msgr2=0x7f85681f1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.804+0000 7f856fa52640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85681534b0 0x7f85681f1f30 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f856001a530 tx=0x7f85600196a0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.804+0000 7f856fa52640 1 -- 172.21.15.83:0/956682748 shutdown_connections 2024-09-19T07:49:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.804+0000 7f856fa52640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f854c0678f0 0x7f854c069db0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.804+0000 7f856fa52640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85681534b0 0x7f85681f1f30 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.804+0000 7f856fa52640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8568150060 0x7f85680c0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.804+0000 7f856fa52640 1 --2- 172.21.15.83:0/956682748 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85680c2b10 0x7f85680c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.804+0000 7f856fa52640 1 -- 172.21.15.83:0/956682748 >> 172.21.15.83:0/956682748 conn(0x7f856806bc80 msgr2=0x7f856814ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.804+0000 7f856fa52640 1 -- 172.21.15.83:0/956682748 shutdown_connections 2024-09-19T07:49:02.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:02.804+0000 7f856fa52640 1 -- 172.21.15.83:0/956682748 wait complete. 2024-09-19T07:49:02.821 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-09-19T07:49:02.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:49:02.822 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-19T07:49:02.840 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:49:02.843 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[36570]: fuse finished with error 0 and tester_r 0 2024-09-19T07:49:04.752 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-09-19T07:49:04.752+0000 7f3e7faa5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:49:08.186 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-19T07:49:08.186 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-19T07:49:08.186 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-19T07:49:08.189 INFO:tasks.ceph.mds.d:Started 2024-09-19T07:49:08.189 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:49:08.268 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-19T07:49:08.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd469d0e640 1 Processor -- start 2024-09-19T07:49:08.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd469d0e640 1 -- start start 2024-09-19T07:49:08.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd469d0e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd46414f9f0 0x7fd4640c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:08.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd469d0e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd46406ed40 0x7fd4640c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:08.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd469d0e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd464153820 con 0x7fd46414fdd0 2024-09-19T07:49:08.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd469d0e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd4640c2890 con 0x7fd46406ed40 2024-09-19T07:49:08.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd469d0e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd4640c2b70 con 0x7fd46414f9f0 2024-09-19T07:49:08.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd463fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd46406ed40 0x7fd4640c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:08.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd463fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd46406ed40 0x7fd4640c2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36622/0 (socket says 172.21.15.83:36622) 2024-09-19T07:49:08.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd463fff640 1 -- 172.21.15.83:0/158892624 learned_addr learned my addr 172.21.15.83:0/158892624 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:08.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd4637fe640 1 --2- 172.21.15.83:0/158892624 >> v2:172.21.15.184:3300/0 conn(0x7fd46414f9f0 0x7fd4640c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:08.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd463fff640 1 -- 172.21.15.83:0/158892624 >> v2:172.21.15.184:3300/0 conn(0x7fd46414f9f0 msgr2=0x7fd4640c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:08.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd463fff640 1 --2- 172.21.15.83:0/158892624 >> v2:172.21.15.184:3300/0 conn(0x7fd46414f9f0 0x7fd4640c2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:08.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd463fff640 1 -- 172.21.15.83:0/158892624 >> v1:172.21.15.184:6789/0 conn(0x7fd46414fdd0 legacy=0x7fd464155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:08.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.320+0000 7fd463fff640 1 -- 172.21.15.83:0/158892624 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd464159220 con 0x7fd46406ed40 2024-09-19T07:49:08.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd463fff640 1 --2- 172.21.15.83:0/158892624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd46406ed40 0x7fd4640c2f90 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fd464050930 tx=0x7fd45c01c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b479f9ec1628c888 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:08.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd4627fc640 1 -- 172.21.15.83:0/158892624 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c0a5510 con 0x7fd46406ed40 2024-09-19T07:49:08.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd4627fc640 1 -- 172.21.15.83:0/158892624 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd45c0a56b0 con 0x7fd46406ed40 2024-09-19T07:49:08.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd4627fc640 1 -- 172.21.15.83:0/158892624 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c0a5a80 con 0x7fd46406ed40 2024-09-19T07:49:08.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 -- 172.21.15.83:0/158892624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd46406ed40 msgr2=0x7fd4640c2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:08.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 --2- 172.21.15.83:0/158892624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd46406ed40 0x7fd4640c2f90 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fd464050930 tx=0x7fd45c01c740 comp rx=0 tx=0).stop 2024-09-19T07:49:08.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 -- 172.21.15.83:0/158892624 shutdown_connections 2024-09-19T07:49:08.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 --2- 172.21.15.83:0/158892624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd46406ed40 0x7fd4640c2f90 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:08.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 --2- 172.21.15.83:0/158892624 >> v2:172.21.15.184:3300/0 conn(0x7fd46414f9f0 0x7fd4640c2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:08.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 -- 172.21.15.83:0/158892624 >> 172.21.15.83:0/158892624 conn(0x7fd4640c5090 msgr2=0x7fd4640c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:08.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 -- 172.21.15.83:0/158892624 shutdown_connections 2024-09-19T07:49:08.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 -- 172.21.15.83:0/158892624 wait complete. 2024-09-19T07:49:08.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 Processor -- start 2024-09-19T07:49:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 -- start start 2024-09-19T07:49:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd46406ed40 0x7fd464158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd46414f9f0 0x7fd46414da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4640ae420 0x7fd46414dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd464162eb0 con 0x7fd46406ed40 2024-09-19T07:49:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd464162d30 con 0x7fd4640ae420 2024-09-19T07:49:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd469d0e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd464163030 con 0x7fd46414f9f0 2024-09-19T07:49:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd4637fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd46406ed40 0x7fd464158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd4637fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd46406ed40 0x7fd464158930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36638/0 (socket says 172.21.15.83:36638) 2024-09-19T07:49:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.324+0000 7fd4637fe640 1 -- 172.21.15.83:0/1805440238 learned_addr learned my addr 172.21.15.83:0/1805440238 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd463fff640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4640ae420 0x7fd46414dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd462ffd640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd46414f9f0 0x7fd46414da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd4637fe640 1 -- 172.21.15.83:0/1805440238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd46414f9f0 msgr2=0x7fd46414da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd4637fe640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd46414f9f0 0x7fd46414da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd4637fe640 1 -- 172.21.15.83:0/1805440238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4640ae420 msgr2=0x7fd46414dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd4637fe640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4640ae420 0x7fd46414dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd4637fe640 1 -- 172.21.15.83:0/1805440238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd464150260 con 0x7fd46406ed40 2024-09-19T07:49:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd4637fe640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd46406ed40 0x7fd464158930 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fd45400d830 tx=0x7fd45400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd44bfff640 1 -- 172.21.15.83:0/1805440238 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45400b6c0 con 0x7fd46406ed40 2024-09-19T07:49:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd44bfff640 1 -- 172.21.15.83:0/1805440238 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd454019070 con 0x7fd46406ed40 2024-09-19T07:49:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd44bfff640 1 -- 172.21.15.83:0/1805440238 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45400ba60 con 0x7fd46406ed40 2024-09-19T07:49:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd469d0e640 1 -- 172.21.15.83:0/1805440238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4641f2f00 con 0x7fd46406ed40 2024-09-19T07:49:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd469d0e640 1 -- 172.21.15.83:0/1805440238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4641f31e0 con 0x7fd46406ed40 2024-09-19T07:49:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd469d0e640 1 -- 172.21.15.83:0/1805440238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4640c2f90 con 0x7fd46406ed40 2024-09-19T07:49:08.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd44bfff640 1 -- 172.21.15.83:0/1805440238 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd45401e460 con 0x7fd46406ed40 2024-09-19T07:49:08.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd44bfff640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4500678c0 0x7fd450069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:08.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd44bfff640 1 -- 172.21.15.83:0/1805440238 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fd4540874f0 con 0x7fd46406ed40 2024-09-19T07:49:08.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd462ffd640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4500678c0 0x7fd450069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:08.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.328+0000 7fd462ffd640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4500678c0 0x7fd450069d80 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd46415fcd0 tx=0x7fd458008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:08.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.332+0000 7fd44bfff640 1 -- 172.21.15.83:0/1805440238 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd454056ed0 con 0x7fd46406ed40 2024-09-19T07:49:08.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.532+0000 7fd469d0e640 1 -- 172.21.15.83:0/1805440238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd464072ae0 con 0x7fd46406ed40 2024-09-19T07:49:08.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd44bfff640 1 -- 172.21.15.83:0/1805440238 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+5716 (secure 0 0 0) 0x7fd4540567d0 con 0x7fd46406ed40 2024-09-19T07:49:08.535 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:08.535 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":204,"btime":"2024-09-19T07:49:01:773318+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:49:01.773306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8265,"qdb_cluster":[8265]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:49:08.535 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 204 2024-09-19T07:49:08.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd469d0e640 1 -- 172.21.15.83:0/1805440238 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4500678c0 msgr2=0x7fd450069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:08.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd469d0e640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4500678c0 0x7fd450069d80 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd46415fcd0 tx=0x7fd458008040 comp rx=0 tx=0).stop 2024-09-19T07:49:08.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd469d0e640 1 -- 172.21.15.83:0/1805440238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd46406ed40 msgr2=0x7fd464158930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:08.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd469d0e640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd46406ed40 0x7fd464158930 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fd45400d830 tx=0x7fd45400dcf0 comp rx=0 tx=0).stop 2024-09-19T07:49:08.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd469d0e640 1 -- 172.21.15.83:0/1805440238 shutdown_connections 2024-09-19T07:49:08.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd469d0e640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd4500678c0 0x7fd450069d80 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:08.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd469d0e640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4640ae420 0x7fd46414dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:08.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd469d0e640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd46414f9f0 0x7fd46414da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:08.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd469d0e640 1 --2- 172.21.15.83:0/1805440238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd46406ed40 0x7fd464158930 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:08.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd469d0e640 1 -- 172.21.15.83:0/1805440238 >> 172.21.15.83:0/1805440238 conn(0x7fd4640c5090 msgr2=0x7fd464155a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:08.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd469d0e640 1 -- 172.21.15.83:0/1805440238 shutdown_connections 2024-09-19T07:49:08.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:08.536+0000 7fd469d0e640 1 -- 172.21.15.83:0/1805440238 wait complete. 2024-09-19T07:49:08.553 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-19T07:49:08.943 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-09-19T07:49:08.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:49:08.943 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-19T07:49:09.211 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:49:09.211 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-19T07:49:09.292 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:49:09.292 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-19T07:49:09.339 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-19T07:49:09.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcec2340640 1 Processor -- start 2024-09-19T07:49:09.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcec2340640 1 -- start start 2024-09-19T07:49:09.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcec2340640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fcebc0c21e0 0x7fcebc0718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcec2340640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc1515c0 0x7fcebc1571b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcec2340640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcebc153cd0 con 0x7fcebc0c2bb0 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcec2340640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fcebc153f40 con 0x7fcebc0c21e0 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcec2340640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcebc154220 con 0x7fcebc1515c0 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebb7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcebc0c2bb0 0x7fcebc150eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60700/0 (socket says 172.21.15.83:60700) 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebb7fe640 1 -- 172.21.15.83:0/47272930 learned_addr learned my addr 172.21.15.83:0/47272930 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebbfff640 1 --2- 172.21.15.83:0/47272930 >> v2:172.21.15.184:3300/0 conn(0x7fcebc0c21e0 0x7fcebc0718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3823179774 0 0) 0x7fcebc153cd0 con 0x7fcebc0c2bb0 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcea4003530 con 0x7fcebc0c2bb0 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcec08dd640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc1515c0 0x7fcebc1571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 390206893 0 0) 0x7fcea4003530 con 0x7fcebc0c2bb0 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc1515c0 msgr2=0x7fcebc1571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:09.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc1515c0 0x7fcebc1571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 >> v2:172.21.15.184:3300/0 conn(0x7fcebc0c21e0 msgr2=0x7fcebc0718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 --2- 172.21.15.83:0/47272930 >> v2:172.21.15.184:3300/0 conn(0x7fcebc0c21e0 0x7fcebc0718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc154500 con 0x7fcebc0c2bb0 2024-09-19T07:49:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fceb0001680 con 0x7fcebc0c2bb0 2024-09-19T07:49:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcec08dd640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc1515c0 0x7fcebc1571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:49:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebbfff640 1 --2- 172.21.15.83:0/47272930 >> v2:172.21.15.184:3300/0 conn(0x7fcebc0c21e0 0x7fcebc0718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:49:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 >> v1:172.21.15.83:6789/0 conn(0x7fcebc0c2bb0 legacy=0x7fcebc150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcea4003a30 0x7fcea4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc1515c0 0x7fcea4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcea4004d50 0x7fcea40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc153f40 con 0x7fcea4003a30 2024-09-19T07:49:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc154220 con 0x7fcea4004d50 2024-09-19T07:49:09.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcea400bab0 con 0x7fcebc1515c0 2024-09-19T07:49:09.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebbfff640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcea4003a30 0x7fcea4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebb7fe640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc1515c0 0x7fcea4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcec08dd640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcea4004d50 0x7fcea40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebbfff640 1 -- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc1515c0 msgr2=0x7fcea4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:09.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebbfff640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc1515c0 0x7fcea4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebbfff640 1 -- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcea4004d50 msgr2=0x7fcea40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:09.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebbfff640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcea4004d50 0x7fcea40051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebbfff640 1 -- 172.21.15.83:0/47272930 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea4003530 con 0x7fcea4003a30 2024-09-19T07:49:09.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.468+0000 7fcebb7fe640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc1515c0 0x7fcea4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.472+0000 7fcebbfff640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcea4003a30 0x7fcea4003e50 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fceac0073e0 tx=0x7fceac01cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f455aadc7750b17 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.472+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac0b2070 con 0x7fcea4003a30 2024-09-19T07:49:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.472+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceac0a5d80 con 0x7fcea4003a30 2024-09-19T07:49:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.472+0000 7fcebaffd640 1 -- 172.21.15.83:0/47272930 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac009c80 con 0x7fcea4003a30 2024-09-19T07:49:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.472+0000 7fcec2340640 1 -- 172.21.15.83:0/47272930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcea4003a30 msgr2=0x7fcea4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.472+0000 7fcec2340640 1 --2- 172.21.15.83:0/47272930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcea4003a30 0x7fcea4003e50 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fceac0073e0 tx=0x7fceac01cee0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.472+0000 7fcec08dd640 1 -- 172.21.15.83:0/47272930 reap_dead start 2024-09-19T07:49:09.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.472+0000 7fcec2340640 1 -- 172.21.15.83:0/47272930 shutdown_connections 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.472+0000 7fcec2340640 1 -- 172.21.15.83:0/47272930 >> 172.21.15.83:0/47272930 conn(0x7fcebc06bc80 msgr2=0x7fcebc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.472+0000 7fcec2340640 1 -- 172.21.15.83:0/47272930 shutdown_connections 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.472+0000 7fcec2340640 1 -- 172.21.15.83:0/47272930 wait complete. 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.472+0000 7fcec2340640 1 Processor -- start 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcec2340640 1 -- start start 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcec2340640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcea4003a30 0x7fcebc0c0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcec2340640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcebc0abbe0 0x7fcebc14c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcec2340640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc14cdc0 0x7fcebc1f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcec2340640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc0c2740 con 0x7fcea4003a30 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcec2340640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc0c25c0 con 0x7fcebc0abbe0 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcec2340640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcebc15e540 con 0x7fcebc14cdc0 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcea4003a30 0x7fcebc0c0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcea4003a30 0x7fcebc0c0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36662/0 (socket says 172.21.15.83:36662) 2024-09-19T07:49:09.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcebbfff640 1 -- 172.21.15.83:0/2694972549 learned_addr learned my addr 172.21.15.83:0/2694972549 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcec08dd640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc14cdc0 0x7fcebc1f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcebb7fe640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcebc0abbe0 0x7fcebc14c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcebbfff640 1 -- 172.21.15.83:0/2694972549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc14cdc0 msgr2=0x7fcebc1f1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcebbfff640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc14cdc0 0x7fcebc1f1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcebbfff640 1 -- 172.21.15.83:0/2694972549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcebc0abbe0 msgr2=0x7fcebc14c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcebbfff640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcebc0abbe0 0x7fcebc14c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcebbfff640 1 -- 172.21.15.83:0/2694972549 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc0ab0c0 con 0x7fcea4003a30 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcebbfff640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcea4003a30 0x7fcebc0c0c10 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fceac01d780 tx=0x7fcea4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fceb8ff9640 1 -- 172.21.15.83:0/2694972549 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac0bc030 con 0x7fcea4003a30 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fceb8ff9640 1 -- 172.21.15.83:0/2694972549 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceac0a5b80 con 0x7fcea4003a30 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcec2340640 1 -- 172.21.15.83:0/2694972549 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcebc0ab3a0 con 0x7fcea4003a30 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fceb8ff9640 1 -- 172.21.15.83:0/2694972549 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac0b2040 con 0x7fcea4003a30 2024-09-19T07:49:09.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcec2340640 1 -- 172.21.15.83:0/2694972549 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcebc0c66f0 con 0x7fcea4003a30 2024-09-19T07:49:09.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcec2340640 1 -- 172.21.15.83:0/2694972549 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcebc0718b0 con 0x7fcea4003a30 2024-09-19T07:49:09.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fceb8ff9640 1 -- 172.21.15.83:0/2694972549 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fceac0bf070 con 0x7fcea4003a30 2024-09-19T07:49:09.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fceb8ff9640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcea80678f0 0x7fcea8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fcebb7fe640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcea80678f0 0x7fcea8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.476+0000 7fceb8ff9640 1 -- 172.21.15.83:0/2694972549 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fceac121280 con 0x7fcea4003a30 2024-09-19T07:49:09.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.480+0000 7fcebb7fe640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcea80678f0 0x7fcea8069db0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fceb00018b0 tx=0x7fceb0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:09.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.484+0000 7fceb8ff9640 1 -- 172.21.15.83:0/2694972549 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fceac0c7020 con 0x7fcea4003a30 2024-09-19T07:49:09.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.652+0000 7fcec2340640 1 -- 172.21.15.83:0/2694972549 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcebc1580e0 con 0x7fcea4003a30 2024-09-19T07:49:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fceb8ff9640 1 -- 172.21.15.83:0/2694972549 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 80 entries v65) ==== 81+0+4890 (secure 0 0 0) 0x7fceac016c50 con 0x7fcea4003a30 2024-09-19T07:49:09.654 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/2502231963 2024-09-20T07:48:58.353983+0000 2024-09-19T07:49:09.654 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1602509240 2024-09-20T07:48:23.346687+0000 2024-09-19T07:49:09.654 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1602509240 2024-09-20T07:48:23.346687+0000 2024-09-19T07:49:09.655 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/119695014 2024-09-20T07:47:58.342001+0000 2024-09-19T07:49:09.655 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/1207928072 2024-09-20T07:47:03.329199+0000 2024-09-19T07:49:09.655 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/3490578864 2024-09-20T07:45:28.310975+0000 2024-09-19T07:49:09.655 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1429520945 2024-09-20T07:45:13.307981+0000 2024-09-19T07:49:09.655 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1429520945 2024-09-20T07:45:13.307981+0000 2024-09-19T07:49:09.655 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/3311721080 2024-09-20T07:44:18.290353+0000 2024-09-19T07:49:09.655 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/3311721080 2024-09-20T07:44:18.290353+0000 2024-09-19T07:49:09.655 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1990119490 2024-09-20T07:42:38.260871+0000 2024-09-19T07:49:09.655 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1990119490 2024-09-20T07:42:38.260871+0000 2024-09-19T07:49:09.655 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/2776269095 2024-09-20T07:42:08.252858+0000 2024-09-19T07:49:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/1335198090 2024-09-20T07:41:33.245545+0000 2024-09-19T07:49:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/1335198090 2024-09-20T07:41:33.245545+0000 2024-09-19T07:49:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/365493751 2024-09-20T07:41:08.239813+0000 2024-09-19T07:49:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/119695014 2024-09-20T07:47:58.342001+0000 2024-09-19T07:49:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/104517500 2024-09-20T07:28:38.045210+0000 2024-09-19T07:49:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/2326757290 2024-09-20T07:40:23.230841+0000 2024-09-19T07:49:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/872719174 2024-09-20T07:28:08.038207+0000 2024-09-19T07:49:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/3763485069 2024-09-20T07:27:18.018593+0000 2024-09-19T07:49:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/872719174 2024-09-20T07:28:08.038207+0000 2024-09-19T07:49:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/1207928072 2024-09-20T07:47:03.329199+0000 2024-09-19T07:49:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/455754420 2024-09-20T07:27:38.022965+0000 2024-09-19T07:49:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/3763485069 2024-09-20T07:27:18.018593+0000 2024-09-19T07:49:09.657 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/1773885233 2024-09-20T07:22:22.945705+0000 2024-09-19T07:49:09.657 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/144714632 2024-09-20T07:25:36.204686+0000 2024-09-19T07:49:09.657 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/26497755 2024-09-20T07:25:27.991655+0000 2024-09-19T07:49:09.657 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/2993984999 2024-09-20T07:32:38.105001+0000 2024-09-19T07:49:09.657 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/4265456507 2024-09-20T07:22:32.948957+0000 2024-09-19T07:49:09.657 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1072903448 2024-09-20T07:33:28.117348+0000 2024-09-19T07:49:09.657 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/3442755673 2024-09-20T07:24:47.982274+0000 2024-09-19T07:49:09.657 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/3490578864 2024-09-20T07:45:28.310975+0000 2024-09-19T07:49:09.657 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/149004692 2024-09-20T07:20:57.928172+0000 2024-09-19T07:49:09.657 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/3841282254 2024-09-20T07:30:48.076144+0000 2024-09-19T07:49:09.657 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/2502231963 2024-09-20T07:48:58.353983+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1229829432 2024-09-20T07:29:48.061079+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1819616410 2024-09-20T07:21:07.930616+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1819616410 2024-09-20T07:21:07.930616+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/149004692 2024-09-20T07:20:57.928172+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/1773885233 2024-09-20T07:22:22.945705+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/2112478738 2024-09-20T07:35:28.148108+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/2993984999 2024-09-20T07:32:38.105001+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/26497755 2024-09-20T07:25:27.991655+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/2898687304 2024-09-20T07:34:18.130590+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2776269095 2024-09-20T07:42:08.252858+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1049139637 2024-09-20T07:23:37.966033+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/3442755673 2024-09-20T07:24:47.982274+0000 2024-09-19T07:49:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/4265456507 2024-09-20T07:22:32.948957+0000 2024-09-19T07:49:09.659 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/2898687304 2024-09-20T07:34:18.130590+0000 2024-09-19T07:49:09.659 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/4115191178 2024-09-20T07:23:52.164421+0000 2024-09-19T07:49:09.659 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/4115191178 2024-09-20T07:23:52.164421+0000 2024-09-19T07:49:09.659 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1774585869 2024-09-20T07:23:07.959477+0000 2024-09-19T07:49:09.659 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/144714632 2024-09-20T07:25:36.204686+0000 2024-09-19T07:49:09.659 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/104517500 2024-09-20T07:28:38.045210+0000 2024-09-19T07:49:09.659 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1049139637 2024-09-20T07:23:37.966033+0000 2024-09-19T07:49:09.659 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/2004814399 2024-09-20T07:30:48.076245+0000 2024-09-19T07:49:09.659 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1229829432 2024-09-20T07:29:48.061079+0000 2024-09-19T07:49:09.659 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2004814399 2024-09-20T07:30:48.076245+0000 2024-09-19T07:49:09.659 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/3841282254 2024-09-20T07:30:48.076144+0000 2024-09-19T07:49:09.659 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/2581473241 2024-09-20T07:31:18.082843+0000 2024-09-19T07:49:09.660 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/455754420 2024-09-20T07:27:38.022965+0000 2024-09-19T07:49:09.660 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1079027718 2024-09-20T07:35:43.151965+0000 2024-09-19T07:49:09.660 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/2581473241 2024-09-20T07:31:18.082843+0000 2024-09-19T07:49:09.660 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/2326757290 2024-09-20T07:40:23.230841+0000 2024-09-19T07:49:09.660 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1072903448 2024-09-20T07:33:28.117348+0000 2024-09-19T07:49:09.660 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1774585869 2024-09-20T07:23:07.959477+0000 2024-09-19T07:49:09.660 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/1519298917 2024-09-20T07:37:18.179638+0000 2024-09-19T07:49:09.660 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1203406606 2024-09-20T07:34:08.127781+0000 2024-09-19T07:49:09.660 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/365493751 2024-09-20T07:41:08.239813+0000 2024-09-19T07:49:09.660 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1203406606 2024-09-20T07:34:08.127781+0000 2024-09-19T07:49:09.660 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1079027718 2024-09-20T07:35:43.151965+0000 2024-09-19T07:49:09.661 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/2112478738 2024-09-20T07:35:28.148108+0000 2024-09-19T07:49:09.661 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/1519298917 2024-09-20T07:37:18.179638+0000 2024-09-19T07:49:09.661 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/3707393259 2024-09-20T07:37:18.179736+0000 2024-09-19T07:49:09.661 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/3707393259 2024-09-20T07:37:18.179736+0000 2024-09-19T07:49:09.661 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/2168217861 2024-09-20T07:38:08.197381+0000 2024-09-19T07:49:09.661 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2168217861 2024-09-20T07:38:08.197381+0000 2024-09-19T07:49:09.661 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/381468017 2024-09-20T07:39:03.210538+0000 2024-09-19T07:49:09.661 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/381468017 2024-09-20T07:39:03.210538+0000 2024-09-19T07:49:09.662 INFO:teuthology.orchestra.run.smithi083.stderr:listed 80 entries 2024-09-19T07:49:09.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fcec2340640 1 -- 172.21.15.83:0/2694972549 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcea80678f0 msgr2=0x7fcea8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:09.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fcec2340640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcea80678f0 0x7fcea8069db0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fceb00018b0 tx=0x7fceb0009040 comp rx=0 tx=0).stop 2024-09-19T07:49:09.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fcec2340640 1 -- 172.21.15.83:0/2694972549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcea4003a30 msgr2=0x7fcebc0c0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:09.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fcec2340640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcea4003a30 0x7fcebc0c0c10 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fceac01d780 tx=0x7fcea4005840 comp rx=0 tx=0).stop 2024-09-19T07:49:09.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fcec2340640 1 -- 172.21.15.83:0/2694972549 shutdown_connections 2024-09-19T07:49:09.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fcec2340640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fcea80678f0 0x7fcea8069db0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fcec2340640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcebc14cdc0 0x7fcebc1f1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fcec2340640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcebc0abbe0 0x7fcebc14c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fcec2340640 1 --2- 172.21.15.83:0/2694972549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcea4003a30 0x7fcebc0c0c10 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fcec2340640 1 -- 172.21.15.83:0/2694972549 >> 172.21.15.83:0/2694972549 conn(0x7fcebc06bc80 msgr2=0x7fcebc0c3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:09.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fcec2340640 1 -- 172.21.15.83:0/2694972549 shutdown_connections 2024-09-19T07:49:09.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.656+0000 7fcec2340640 1 -- 172.21.15.83:0/2694972549 wait complete. 2024-09-19T07:49:09.673 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-19T07:49:09.673 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-19T07:49:09.680 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-19T07:49:09.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd892852640 1 Processor -- start 2024-09-19T07:49:09.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd892852640 1 -- start start 2024-09-19T07:49:09.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd892852640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fd88c0c21e0 0x7fd88c0718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd892852640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd88c0c2bb0 0x7fd88c150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd892852640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd88c153cd0 con 0x7fd88c1513f0 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd892852640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fd88c1518a0 con 0x7fd88c0c21e0 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd892852640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd88c153f40 con 0x7fd88c0c2bb0 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd890def640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd88c1513f0 0x7fd88c1571b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60702/0 (socket says 172.21.15.83:60702) 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd890def640 1 -- 172.21.15.83:0/2974243668 learned_addr learned my addr 172.21.15.83:0/2974243668 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88b7fe640 1 --2- 172.21.15.83:0/2974243668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd88c0c2bb0 0x7fd88c150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3241925663 0 0) 0x7fd88c153cd0 con 0x7fd88c1513f0 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88bfff640 1 --2- 172.21.15.83:0/2974243668 >> v2:172.21.15.184:3300/0 conn(0x7fd88c0c21e0 0x7fd88c0718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd874003530 con 0x7fd88c1513f0 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2418990229 0 0) 0x7fd874003530 con 0x7fd88c1513f0 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd88c0c2bb0 msgr2=0x7fd88c150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 --2- 172.21.15.83:0/2974243668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd88c0c2bb0 0x7fd88c150eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 >> v2:172.21.15.184:3300/0 conn(0x7fd88c0c21e0 msgr2=0x7fd88c0718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 --2- 172.21.15.83:0/2974243668 >> v2:172.21.15.184:3300/0 conn(0x7fd88c0c21e0 0x7fd88c0718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88c154220 con 0x7fd88c1513f0 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fd884001680 con 0x7fd88c1513f0 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 >> v1:172.21.15.83:6789/0 conn(0x7fd88c1513f0 legacy=0x7fd88c1571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 --2- 172.21.15.83:0/2974243668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd874003a30 0x7fd874003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 --2- 172.21.15.83:0/2974243668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd88c0c2bb0 0x7fd874004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 --2- 172.21.15.83:0/2974243668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd874004d50 0x7fd8740051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c153f40 con 0x7fd874003a30 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd87400b9d0 con 0x7fd874004d50 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd87400bb80 con 0x7fd88c0c2bb0 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88bfff640 1 --2- 172.21.15.83:0/2974243668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd874003a30 0x7fd874003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88bfff640 1 -- 172.21.15.83:0/2974243668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd88c0c2bb0 msgr2=0x7fd874004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88b7fe640 1 --2- 172.21.15.83:0/2974243668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd88c0c2bb0 0x7fd874004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88bfff640 1 --2- 172.21.15.83:0/2974243668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd88c0c2bb0 0x7fd874004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88bfff640 1 -- 172.21.15.83:0/2974243668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd874004d50 msgr2=0x7fd8740051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:09.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd890def640 1 --2- 172.21.15.83:0/2974243668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd874004d50 0x7fd8740051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88bfff640 1 --2- 172.21.15.83:0/2974243668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd874004d50 0x7fd8740051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.800+0000 7fd88bfff640 1 -- 172.21.15.83:0/2974243668 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd874003530 con 0x7fd874003a30 2024-09-19T07:49:09.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88bfff640 1 --2- 172.21.15.83:0/2974243668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd874003a30 0x7fd874003e50 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009fc0 tx=0x7fd87c01cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d38f95ffa3930c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:09.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c0b2070 con 0x7fd874003a30 2024-09-19T07:49:09.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd87c0a5ca0 con 0x7fd874003a30 2024-09-19T07:49:09.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88affd640 1 -- 172.21.15.83:0/2974243668 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c009c10 con 0x7fd874003a30 2024-09-19T07:49:09.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 -- 172.21.15.83:0/2974243668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd874003a30 msgr2=0x7fd874003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:09.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 --2- 172.21.15.83:0/2974243668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd874003a30 0x7fd874003e50 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009fc0 tx=0x7fd87c01cc90 comp rx=0 tx=0).stop 2024-09-19T07:49:09.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd890def640 1 -- 172.21.15.83:0/2974243668 reap_dead start 2024-09-19T07:49:09.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 -- 172.21.15.83:0/2974243668 shutdown_connections 2024-09-19T07:49:09.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 -- 172.21.15.83:0/2974243668 >> 172.21.15.83:0/2974243668 conn(0x7fd88c06bc80 msgr2=0x7fd88c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:09.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 -- 172.21.15.83:0/2974243668 shutdown_connections 2024-09-19T07:49:09.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 -- 172.21.15.83:0/2974243668 wait complete. 2024-09-19T07:49:09.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 Processor -- start 2024-09-19T07:49:09.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 -- start start 2024-09-19T07:49:09.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd874003a30 0x7fd88c156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88c14c670 0x7fd88c0c3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd88c1efac0 0x7fd88c1498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c0a9eb0 con 0x7fd88c14c670 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c0a9d30 con 0x7fd88c1efac0 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd892852640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd88c0c25c0 con 0x7fd874003a30 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88c14c670 0x7fd88c0c3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88c14c670 0x7fd88c0c3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36674/0 (socket says 172.21.15.83:36674) 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88b7fe640 1 -- 172.21.15.83:0/1099609357 learned_addr learned my addr 172.21.15.83:0/1099609357 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd890def640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd88c1efac0 0x7fd88c1498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88bfff640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd874003a30 0x7fd88c156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88b7fe640 1 -- 172.21.15.83:0/1099609357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd874003a30 msgr2=0x7fd88c156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88b7fe640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd874003a30 0x7fd88c156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88b7fe640 1 -- 172.21.15.83:0/1099609357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd88c1efac0 msgr2=0x7fd88c1498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88b7fe640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd88c1efac0 0x7fd88c1498f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88b7fe640 1 -- 172.21.15.83:0/1099609357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88c1589a0 con 0x7fd88c14c670 2024-09-19T07:49:09.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.804+0000 7fd88bfff640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd874003a30 0x7fd88c156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:09.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.808+0000 7fd88b7fe640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88c14c670 0x7fd88c0c3ce0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fd8800108c0 tx=0x7fd880010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:09.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.808+0000 7fd888ff9640 1 -- 172.21.15.83:0/1099609357 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88000f040 con 0x7fd88c14c670 2024-09-19T07:49:09.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.808+0000 7fd888ff9640 1 -- 172.21.15.83:0/1099609357 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd88000eb60 con 0x7fd88c14c670 2024-09-19T07:49:09.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.808+0000 7fd892852640 1 -- 172.21.15.83:0/1099609357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88c155b00 con 0x7fd88c14c670 2024-09-19T07:49:09.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.808+0000 7fd888ff9640 1 -- 172.21.15.83:0/1099609357 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88001c400 con 0x7fd88c14c670 2024-09-19T07:49:09.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.808+0000 7fd892852640 1 -- 172.21.15.83:0/1099609357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd88c1560c0 con 0x7fd88c14c670 2024-09-19T07:49:09.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.808+0000 7fd892852640 1 -- 172.21.15.83:0/1099609357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd88c0718b0 con 0x7fd88c14c670 2024-09-19T07:49:09.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.808+0000 7fd888ff9640 1 -- 172.21.15.83:0/1099609357 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd88001c5a0 con 0x7fd88c14c670 2024-09-19T07:49:09.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.808+0000 7fd888ff9640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8780678f0 0x7fd878069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:09.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.808+0000 7fd888ff9640 1 -- 172.21.15.83:0/1099609357 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fd8800874f0 con 0x7fd88c14c670 2024-09-19T07:49:09.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.808+0000 7fd88bfff640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8780678f0 0x7fd878069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:09.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.808+0000 7fd88bfff640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8780678f0 0x7fd878069db0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0af000 tx=0x7fd87c0a7040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:09.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.812+0000 7fd888ff9640 1 -- 172.21.15.83:0/1099609357 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd88c1589a0 con 0x7fd88c14c670 2024-09-19T07:49:09.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.980+0000 7fd892852640 1 -- 172.21.15.83:0/1099609357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd88c073eb0 con 0x7fd88c14c670 2024-09-19T07:49:09.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.980+0000 7fd888ff9640 1 -- 172.21.15.83:0/1099609357 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 80 entries v65) ==== 81+0+4890 (secure 0 0 0) 0x7fd880018030 con 0x7fd88c14c670 2024-09-19T07:49:09.981 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/2502231963 2024-09-20T07:48:58.353983+0000 2024-09-19T07:49:09.981 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1602509240 2024-09-20T07:48:23.346687+0000 2024-09-19T07:49:09.981 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1602509240 2024-09-20T07:48:23.346687+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/119695014 2024-09-20T07:47:58.342001+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/1207928072 2024-09-20T07:47:03.329199+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/3490578864 2024-09-20T07:45:28.310975+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1429520945 2024-09-20T07:45:13.307981+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1429520945 2024-09-20T07:45:13.307981+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/3311721080 2024-09-20T07:44:18.290353+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/3311721080 2024-09-20T07:44:18.290353+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1990119490 2024-09-20T07:42:38.260871+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1990119490 2024-09-20T07:42:38.260871+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/2776269095 2024-09-20T07:42:08.252858+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/1335198090 2024-09-20T07:41:33.245545+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/1335198090 2024-09-20T07:41:33.245545+0000 2024-09-19T07:49:09.982 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/365493751 2024-09-20T07:41:08.239813+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/119695014 2024-09-20T07:47:58.342001+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/104517500 2024-09-20T07:28:38.045210+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/2326757290 2024-09-20T07:40:23.230841+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/872719174 2024-09-20T07:28:08.038207+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/3763485069 2024-09-20T07:27:18.018593+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/872719174 2024-09-20T07:28:08.038207+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/1207928072 2024-09-20T07:47:03.329199+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/455754420 2024-09-20T07:27:38.022965+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/3763485069 2024-09-20T07:27:18.018593+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/1773885233 2024-09-20T07:22:22.945705+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/144714632 2024-09-20T07:25:36.204686+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/26497755 2024-09-20T07:25:27.991655+0000 2024-09-19T07:49:09.983 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/2993984999 2024-09-20T07:32:38.105001+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/4265456507 2024-09-20T07:22:32.948957+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1072903448 2024-09-20T07:33:28.117348+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/3442755673 2024-09-20T07:24:47.982274+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/3490578864 2024-09-20T07:45:28.310975+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/149004692 2024-09-20T07:20:57.928172+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/3841282254 2024-09-20T07:30:48.076144+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/2502231963 2024-09-20T07:48:58.353983+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1229829432 2024-09-20T07:29:48.061079+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1819616410 2024-09-20T07:21:07.930616+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1819616410 2024-09-20T07:21:07.930616+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/149004692 2024-09-20T07:20:57.928172+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/1773885233 2024-09-20T07:22:22.945705+0000 2024-09-19T07:49:09.984 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/2112478738 2024-09-20T07:35:28.148108+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/2993984999 2024-09-20T07:32:38.105001+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/26497755 2024-09-20T07:25:27.991655+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/2898687304 2024-09-20T07:34:18.130590+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2776269095 2024-09-20T07:42:08.252858+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1049139637 2024-09-20T07:23:37.966033+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/3442755673 2024-09-20T07:24:47.982274+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/4265456507 2024-09-20T07:22:32.948957+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/2898687304 2024-09-20T07:34:18.130590+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/4115191178 2024-09-20T07:23:52.164421+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/4115191178 2024-09-20T07:23:52.164421+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1774585869 2024-09-20T07:23:07.959477+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/144714632 2024-09-20T07:25:36.204686+0000 2024-09-19T07:49:09.985 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/104517500 2024-09-20T07:28:38.045210+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1049139637 2024-09-20T07:23:37.966033+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/2004814399 2024-09-20T07:30:48.076245+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1229829432 2024-09-20T07:29:48.061079+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2004814399 2024-09-20T07:30:48.076245+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/3841282254 2024-09-20T07:30:48.076144+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/2581473241 2024-09-20T07:31:18.082843+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/455754420 2024-09-20T07:27:38.022965+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/1079027718 2024-09-20T07:35:43.151965+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/2581473241 2024-09-20T07:31:18.082843+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/2326757290 2024-09-20T07:40:23.230841+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1072903448 2024-09-20T07:33:28.117348+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1774585869 2024-09-20T07:23:07.959477+0000 2024-09-19T07:49:09.986 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6835/1519298917 2024-09-20T07:37:18.179638+0000 2024-09-19T07:49:09.987 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/1203406606 2024-09-20T07:34:08.127781+0000 2024-09-19T07:49:09.987 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/365493751 2024-09-20T07:41:08.239813+0000 2024-09-19T07:49:09.987 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/1203406606 2024-09-20T07:34:08.127781+0000 2024-09-19T07:49:09.987 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/1079027718 2024-09-20T07:35:43.151965+0000 2024-09-19T07:49:09.987 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/2112478738 2024-09-20T07:35:28.148108+0000 2024-09-19T07:49:09.987 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6834/1519298917 2024-09-20T07:37:18.179638+0000 2024-09-19T07:49:09.987 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/3707393259 2024-09-20T07:37:18.179736+0000 2024-09-19T07:49:09.987 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/3707393259 2024-09-20T07:37:18.179736+0000 2024-09-19T07:49:09.987 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/2168217861 2024-09-20T07:38:08.197381+0000 2024-09-19T07:49:09.987 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2168217861 2024-09-20T07:38:08.197381+0000 2024-09-19T07:49:09.987 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6832/381468017 2024-09-20T07:39:03.210538+0000 2024-09-19T07:49:09.987 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.184:6833/381468017 2024-09-20T07:39:03.210538+0000 2024-09-19T07:49:09.988 INFO:teuthology.orchestra.run.smithi083.stderr:listed 80 entries 2024-09-19T07:49:09.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.984+0000 7fd892852640 1 -- 172.21.15.83:0/1099609357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8780678f0 msgr2=0x7fd878069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:09.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.984+0000 7fd892852640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8780678f0 0x7fd878069db0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0af000 tx=0x7fd87c0a7040 comp rx=0 tx=0).stop 2024-09-19T07:49:09.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.984+0000 7fd892852640 1 -- 172.21.15.83:0/1099609357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88c14c670 msgr2=0x7fd88c0c3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:09.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.984+0000 7fd892852640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88c14c670 0x7fd88c0c3ce0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fd8800108c0 tx=0x7fd880010d80 comp rx=0 tx=0).stop 2024-09-19T07:49:09.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.984+0000 7fd892852640 1 -- 172.21.15.83:0/1099609357 shutdown_connections 2024-09-19T07:49:09.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.984+0000 7fd892852640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd8780678f0 0x7fd878069db0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.984+0000 7fd892852640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd88c1efac0 0x7fd88c1498f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.984+0000 7fd892852640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd88c14c670 0x7fd88c0c3ce0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.984+0000 7fd892852640 1 --2- 172.21.15.83:0/1099609357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd874003a30 0x7fd88c156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:09.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.984+0000 7fd892852640 1 -- 172.21.15.83:0/1099609357 >> 172.21.15.83:0/1099609357 conn(0x7fd88c06bc80 msgr2=0x7fd88c158b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:09.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.984+0000 7fd892852640 1 -- 172.21.15.83:0/1099609357 shutdown_connections 2024-09-19T07:49:09.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:09.984+0000 7fd892852640 1 -- 172.21.15.83:0/1099609357 wait complete. 2024-09-19T07:49:09.999 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-19T07:49:09.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:49:09.999 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-19T07:49:10.028 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:49:10.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[34414]: fuse finished with error 0 and tester_r 0 2024-09-19T07:49:10.554 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:49:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.684+0000 7f94abc83640 1 Processor -- start 2024-09-19T07:49:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94abc83640 1 -- start start 2024-09-19T07:49:10.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94abc83640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94a4152b20 0x7f94a40c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:10.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94abc83640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a4150670 con 0x7f94a40c45c0 2024-09-19T07:49:10.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94abc83640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a40c2880 con 0x7f94a4150290 2024-09-19T07:49:10.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94abc83640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f94a40716c0 con 0x7f94a4152b20 2024-09-19T07:49:10.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94a921e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94a4152b20 0x7f94a40c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:10.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94aa220640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f94a40c45c0 0x7f94a4153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60296/0 (socket says 172.21.15.83:60296) 2024-09-19T07:49:10.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94aa220640 1 -- 172.21.15.83:0/1897608225 learned_addr learned my addr 172.21.15.83:0/1897608225 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:10.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94a921e640 1 -- 172.21.15.83:0/1897608225 >> v1:172.21.15.184:6789/0 conn(0x7f94a4150290 legacy=0x7f94a40c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:10.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94a921e640 1 -- 172.21.15.83:0/1897608225 >> v1:172.21.15.83:6789/0 conn(0x7f94a40c45c0 legacy=0x7f94a4153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:10.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94a921e640 1 -- 172.21.15.83:0/1897608225 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a40c2b60 con 0x7f94a4152b20 2024-09-19T07:49:10.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94a921e640 1 --2- 172.21.15.83:0/1897608225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94a4152b20 0x7f94a40c4080 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f94a00097c0 tx=0x7f94a001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341aa9b0b48e6cc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:10.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94a8a1d640 1 -- 172.21.15.83:0/1897608225 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a00b0070 con 0x7f94a4152b20 2024-09-19T07:49:10.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94a8a1d640 1 -- 172.21.15.83:0/1897608225 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a00ab600 con 0x7f94a4152b20 2024-09-19T07:49:10.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.688+0000 7f94a8a1d640 1 -- 172.21.15.83:0/1897608225 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a00ab9d0 con 0x7f94a4152b20 2024-09-19T07:49:10.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 -- 172.21.15.83:0/1897608225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94a4152b20 msgr2=0x7f94a40c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:10.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 --2- 172.21.15.83:0/1897608225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94a4152b20 0x7f94a40c4080 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f94a00097c0 tx=0x7f94a001c4a0 comp rx=0 tx=0).stop 2024-09-19T07:49:10.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 -- 172.21.15.83:0/1897608225 shutdown_connections 2024-09-19T07:49:10.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 --2- 172.21.15.83:0/1897608225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94a4152b20 0x7f94a40c4080 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:10.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 -- 172.21.15.83:0/1897608225 >> 172.21.15.83:0/1897608225 conn(0x7f94a406bc80 msgr2=0x7f94a406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:10.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 -- 172.21.15.83:0/1897608225 shutdown_connections 2024-09-19T07:49:10.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 -- 172.21.15.83:0/1897608225 wait complete. 2024-09-19T07:49:10.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 Processor -- start 2024-09-19T07:49:10.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 -- start start 2024-09-19T07:49:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94a40c45c0 0x7f94a40bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94a4150290 0x7f94a41f01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94a4152b20 0x7f94a41f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4160cf0 con 0x7f94a4152b20 2024-09-19T07:49:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4160b70 con 0x7f94a4150290 2024-09-19T07:49:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f94a4160e70 con 0x7f94a40c45c0 2024-09-19T07:49:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94aa220640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94a4152b20 0x7f94a41f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94aa220640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94a4152b20 0x7f94a41f0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41960/0 (socket says 172.21.15.83:41960) 2024-09-19T07:49:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94aa220640 1 -- 172.21.15.83:0/967703826 learned_addr learned my addr 172.21.15.83:0/967703826 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94a921e640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94a4150290 0x7f94a41f01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94a9a1f640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94a40c45c0 0x7f94a40bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:10.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94aa220640 1 -- 172.21.15.83:0/967703826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94a40c45c0 msgr2=0x7f94a40bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94aa220640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94a40c45c0 0x7f94a40bdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94aa220640 1 -- 172.21.15.83:0/967703826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94a4150290 msgr2=0x7f94a41f01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94aa220640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94a4150290 0x7f94a41f01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94aa220640 1 -- 172.21.15.83:0/967703826 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a41f0e00 con 0x7f94a4152b20 2024-09-19T07:49:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94a9a1f640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94a40c45c0 0x7f94a40bdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94aa220640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94a4152b20 0x7f94a41f0720 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f949c00cbc0 tx=0x7f949c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f949a7fc640 1 -- 172.21.15.83:0/967703826 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c017070 con 0x7f94a4152b20 2024-09-19T07:49:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f949a7fc640 1 -- 172.21.15.83:0/967703826 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f949c00ace0 con 0x7f94a4152b20 2024-09-19T07:49:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f949a7fc640 1 -- 172.21.15.83:0/967703826 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c012600 con 0x7f94a4152b20 2024-09-19T07:49:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 -- 172.21.15.83:0/967703826 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a40bcf30 con 0x7f94a4152b20 2024-09-19T07:49:10.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.692+0000 7f94abc83640 1 -- 172.21.15.83:0/967703826 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a40aac40 con 0x7f94a4152b20 2024-09-19T07:49:10.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.696+0000 7f94abc83640 1 -- 172.21.15.83:0/967703826 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94a40c4ba0 con 0x7f94a4152b20 2024-09-19T07:49:10.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.696+0000 7f949a7fc640 1 -- 172.21.15.83:0/967703826 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f949c01a070 con 0x7f94a4152b20 2024-09-19T07:49:10.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.696+0000 7f949a7fc640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94880678c0 0x7f9488069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:10.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.696+0000 7f949a7fc640 1 -- 172.21.15.83:0/967703826 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f949c0869d0 con 0x7f94a4152b20 2024-09-19T07:49:10.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.696+0000 7f94a9a1f640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94880678c0 0x7f9488069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:10.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.696+0000 7f94a9a1f640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94880678c0 0x7f9488069d80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f9494000f70 tx=0x7f9494009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:10.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.700+0000 7f949a7fc640 1 -- 172.21.15.83:0/967703826 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f949c055df0 con 0x7f94a4152b20 2024-09-19T07:49:10.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.896+0000 7f94abc83640 1 -- 172.21.15.83:0/967703826 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94a4156430 con 0x7f94a4152b20 2024-09-19T07:49:10.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.896+0000 7f949a7fc640 1 -- 172.21.15.83:0/967703826 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) ==== 95+0+6542 (secure 0 0 0) 0x7f949c05a0f0 con 0x7f94a4152b20 2024-09-19T07:49:10.898 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:10.898 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":205,"btime":"2024-09-19T07:49:08:666589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6407,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:49:01.773306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8265,"qdb_cluster":[8265]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:49:10.898 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 205 2024-09-19T07:49:10.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.900+0000 7f94abc83640 1 -- 172.21.15.83:0/967703826 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94880678c0 msgr2=0x7f9488069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:10.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.900+0000 7f94abc83640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94880678c0 0x7f9488069d80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f9494000f70 tx=0x7f9494009040 comp rx=0 tx=0).stop 2024-09-19T07:49:10.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.900+0000 7f94abc83640 1 -- 172.21.15.83:0/967703826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94a4152b20 msgr2=0x7f94a41f0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:10.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.900+0000 7f94abc83640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94a4152b20 0x7f94a41f0720 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f949c00cbc0 tx=0x7f949c00a430 comp rx=0 tx=0).stop 2024-09-19T07:49:10.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.900+0000 7f94abc83640 1 -- 172.21.15.83:0/967703826 shutdown_connections 2024-09-19T07:49:10.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.900+0000 7f94abc83640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f94880678c0 0x7f9488069d80 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:10.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.900+0000 7f94abc83640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94a4152b20 0x7f94a41f0720 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:10.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.900+0000 7f94abc83640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94a4150290 0x7f94a41f01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:10.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.900+0000 7f94abc83640 1 --2- 172.21.15.83:0/967703826 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f94a40c45c0 0x7f94a40bdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:10.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.900+0000 7f94abc83640 1 -- 172.21.15.83:0/967703826 >> 172.21.15.83:0/967703826 conn(0x7f94a406bc80 msgr2=0x7f94a40c1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:10.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.900+0000 7f94abc83640 1 -- 172.21.15.83:0/967703826 shutdown_connections 2024-09-19T07:49:10.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:10.900+0000 7f94abc83640 1 -- 172.21.15.83:0/967703826 wait complete. 2024-09-19T07:49:10.917 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-19T07:49:10.918 INFO:tasks.mds_thrash.fs.[a]:waiting for 35 secs before thrashing 2024-09-19T07:49:12.079 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:49:12.082 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:49:16.130 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-19T07:49:16.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:49:16.131 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-19T07:49:16.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:49:16.138 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-09-19T07:49:16.272 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-19T07:49:16.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:49:16.273 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-19T07:49:16.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:49:16.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link delete ceph-brx 2024-09-19T07:49:16.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:49:16.508 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-19T07:49:16.564 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-19T07:49:16.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:49:16.565 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-19T07:49:16.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T07:49:16.579 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link delete ceph-brx 2024-09-19T07:49:16.760 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-19T07:49:16.769 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-19T07:49:16.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:49:16.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c1636d640 1 Processor -- start 2024-09-19T07:49:16.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c1636d640 1 -- start start 2024-09-19T07:49:16.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c1636d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f0c100ab750 0x7f0c100c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:16.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c1636d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c100c2530 0x7f0c101528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:16.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c1636d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c100ab060 0x7f0c10152e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:16.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c1636d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f0c1006de90 con 0x7f0c100c2530 2024-09-19T07:49:16.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c1636d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f0c1006e170 con 0x7f0c100ab750 2024-09-19T07:49:16.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c1636d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0c1006e450 con 0x7f0c100ab060 2024-09-19T07:49:16.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c077fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c100c2530 0x7f0c101528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:16.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c077fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c100c2530 0x7f0c101528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41968/0 (socket says 172.21.15.83:41968) 2024-09-19T07:49:16.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c077fe640 1 -- 172.21.15.83:0/3639738440 learned_addr learned my addr 172.21.15.83:0/3639738440 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:16.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c1490a640 1 --2- 172.21.15.83:0/3639738440 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c100ab060 0x7f0c10152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:16.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c077fe640 1 -- 172.21.15.83:0/3639738440 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c100ab060 msgr2=0x7f0c10152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:16.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c0ffff640 1 --2- 172.21.15.83:0/3639738440 >> v2:172.21.15.184:3300/0 conn(0x7f0c100ab750 0x7f0c100c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:16.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c077fe640 1 --2- 172.21.15.83:0/3639738440 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c100ab060 0x7f0c10152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:16.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c077fe640 1 -- 172.21.15.83:0/3639738440 >> v2:172.21.15.184:3300/0 conn(0x7f0c100ab750 msgr2=0x7f0c100c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:16.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c077fe640 1 --2- 172.21.15.83:0/3639738440 >> v2:172.21.15.184:3300/0 conn(0x7f0c100ab750 0x7f0c100c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:16.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c077fe640 1 -- 172.21.15.83:0/3639738440 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1006e730 con 0x7f0c100c2530 2024-09-19T07:49:16.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.900+0000 7f0c0ffff640 1 --2- 172.21.15.83:0/3639738440 >> v2:172.21.15.184:3300/0 conn(0x7f0c100ab750 0x7f0c100c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:16.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c077fe640 1 --2- 172.21.15.83:0/3639738440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c100c2530 0x7f0c101528c0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f0c100ad7d0 tx=0x7f0bf801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba641afb1ad81e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:16.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c0f7fe640 1 -- 172.21.15.83:0/3639738440 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80b0070 con 0x7f0c100c2530 2024-09-19T07:49:16.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c0f7fe640 1 -- 172.21.15.83:0/3639738440 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf80ab600 con 0x7f0c100c2530 2024-09-19T07:49:16.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c0f7fe640 1 -- 172.21.15.83:0/3639738440 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80ab9d0 con 0x7f0c100c2530 2024-09-19T07:49:16.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 -- 172.21.15.83:0/3639738440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c100c2530 msgr2=0x7f0c101528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:16.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 --2- 172.21.15.83:0/3639738440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c100c2530 0x7f0c101528c0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f0c100ad7d0 tx=0x7f0bf801c490 comp rx=0 tx=0).stop 2024-09-19T07:49:16.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 -- 172.21.15.83:0/3639738440 shutdown_connections 2024-09-19T07:49:16.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 --2- 172.21.15.83:0/3639738440 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c100ab060 0x7f0c10152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:16.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 --2- 172.21.15.83:0/3639738440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c100c2530 0x7f0c101528c0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:16.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 --2- 172.21.15.83:0/3639738440 >> v2:172.21.15.184:3300/0 conn(0x7f0c100ab750 0x7f0c100c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:16.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 -- 172.21.15.83:0/3639738440 >> 172.21.15.83:0/3639738440 conn(0x7f0c1006bc80 msgr2=0x7f0c1006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:16.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 -- 172.21.15.83:0/3639738440 shutdown_connections 2024-09-19T07:49:16.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 -- 172.21.15.83:0/3639738440 wait complete. 2024-09-19T07:49:16.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 Processor -- start 2024-09-19T07:49:16.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 -- start start 2024-09-19T07:49:16.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c100ab060 0x7f0c1015d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c100ab750 0x7f0c101ef120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c100c2530 0x7f0c100c6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0c10155c20 con 0x7f0c100ab750 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0c10155aa0 con 0x7f0c100c2530 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1636d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0c10155da0 con 0x7f0c100ab060 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c100ab750 0x7f0c101ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c100ab750 0x7f0c101ef120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41980/0 (socket says 172.21.15.83:41980) 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c077fe640 1 -- 172.21.15.83:0/2851359604 learned_addr learned my addr 172.21.15.83:0/2851359604 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c0ffff640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c100ab060 0x7f0c1015d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1490a640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c100c2530 0x7f0c100c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c077fe640 1 -- 172.21.15.83:0/2851359604 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c100ab060 msgr2=0x7f0c1015d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c077fe640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c100ab060 0x7f0c1015d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:16.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c077fe640 1 -- 172.21.15.83:0/2851359604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c100c2530 msgr2=0x7f0c100c6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:16.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c077fe640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c100c2530 0x7f0c100c6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:16.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c077fe640 1 -- 172.21.15.83:0/2851359604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c10151460 con 0x7f0c100ab750 2024-09-19T07:49:16.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c1490a640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c100c2530 0x7f0c100c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:16.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.904+0000 7f0c0ffff640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c100ab060 0x7f0c1015d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:16.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.908+0000 7f0c077fe640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c100ab750 0x7f0c101ef120 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f0bf801ab40 tx=0x7f0bf8019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:16.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.908+0000 7f0c0d7fa640 1 -- 172.21.15.83:0/2851359604 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80b9070 con 0x7f0c100ab750 2024-09-19T07:49:16.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.908+0000 7f0c0d7fa640 1 -- 172.21.15.83:0/2851359604 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf80b4600 con 0x7f0c100ab750 2024-09-19T07:49:16.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.908+0000 7f0c1636d640 1 -- 172.21.15.83:0/2851359604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c10151740 con 0x7f0c100ab750 2024-09-19T07:49:16.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.908+0000 7f0c0d7fa640 1 -- 172.21.15.83:0/2851359604 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80b0040 con 0x7f0c100ab750 2024-09-19T07:49:16.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.908+0000 7f0c1636d640 1 -- 172.21.15.83:0/2851359604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c101459e0 con 0x7f0c100ab750 2024-09-19T07:49:16.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.908+0000 7f0c1636d640 1 -- 172.21.15.83:0/2851359604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c100c1ff0 con 0x7f0c100ab750 2024-09-19T07:49:16.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.908+0000 7f0c0d7fa640 1 -- 172.21.15.83:0/2851359604 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0bf80bc070 con 0x7f0c100ab750 2024-09-19T07:49:16.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.908+0000 7f0c0d7fa640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0bf00678c0 0x7f0bf0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:16.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.908+0000 7f0c0ffff640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0bf00678c0 0x7f0bf0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:16.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.908+0000 7f0c0d7fa640 1 -- 172.21.15.83:0/2851359604 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f0bf811fee0 con 0x7f0c100ab750 2024-09-19T07:49:16.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.908+0000 7f0c0ffff640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0bf00678c0 0x7f0bf0069d80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0c00008930 tx=0x7f0c00008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:16.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:16.912+0000 7f0c0d7fa640 1 -- 172.21.15.83:0/2851359604 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f0bf8016110 con 0x7f0c100ab750 2024-09-19T07:49:17.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.072+0000 7f0c1636d640 1 -- 172.21.15.83:0/2851359604 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0c10073880 con 0x7f0bf00678c0 2024-09-19T07:49:17.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.104+0000 7f0c0d7fa640 1 -- 172.21.15.83:0/2851359604 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493875 (secure 0 0 0) 0x7f0c10073880 con 0x7f0bf00678c0 2024-09-19T07:49:17.105 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:17.113 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:49:17.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.112+0000 7f0c1636d640 1 -- 172.21.15.83:0/2851359604 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0bf00678c0 msgr2=0x7f0bf0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.112+0000 7f0c1636d640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0bf00678c0 0x7f0bf0069d80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0c00008930 tx=0x7f0c00008040 comp rx=0 tx=0).stop 2024-09-19T07:49:17.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.112+0000 7f0c1636d640 1 -- 172.21.15.83:0/2851359604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c100ab750 msgr2=0x7f0c101ef120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.112+0000 7f0c1636d640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c100ab750 0x7f0c101ef120 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f0bf801ab40 tx=0x7f0bf8019230 comp rx=0 tx=0).stop 2024-09-19T07:49:17.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.112+0000 7f0c1636d640 1 -- 172.21.15.83:0/2851359604 shutdown_connections 2024-09-19T07:49:17.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.112+0000 7f0c1636d640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f0bf00678c0 0x7f0bf0069d80 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.112+0000 7f0c1636d640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c100c2530 0x7f0c100c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.112+0000 7f0c1636d640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c100ab750 0x7f0c101ef120 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.112+0000 7f0c1636d640 1 --2- 172.21.15.83:0/2851359604 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0c100ab060 0x7f0c1015d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.112+0000 7f0c1636d640 1 -- 172.21.15.83:0/2851359604 >> 172.21.15.83:0/2851359604 conn(0x7f0c1006bc80 msgr2=0x7f0c10155130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:17.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.112+0000 7f0c1636d640 1 -- 172.21.15.83:0/2851359604 shutdown_connections 2024-09-19T07:49:17.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.112+0000 7f0c1636d640 1 -- 172.21.15.83:0/2851359604 wait complete. 2024-09-19T07:49:17.130 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":955,"stamp":"2024-09-19T07:49:15.612510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39797278,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3062,"num_read_kb":412170,"num_write":3183053,"num_write_kb":38996435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":596828,"ondisk_log_size":596828,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7748160,"kb_used_data":84416,"kb_used_omap":12,"kb_used_meta":7663603,"kb_avail":747226560,"statfs":{"total":773094113280,"available":765159997440,"internally_reserved":0,"allocated":86441984,"data_stored":83166972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":7847530033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26509,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":48,"num_write":32,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005583"},"pg_stats":[{"pgid":"5.18","version":"64'4995","reported_seq":5099,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363394+0000","last_change":"2024-09-19T07:19:40.358457+0000","last_active":"2024-09-19T07:48:58.363394+0000","last_peered":"2024-09-19T07:48:58.363394+0000","last_clean":"2024-09-19T07:48:58.363394+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:48:58.363394+0000","last_undegraded":"2024-09-19T07:48:58.363394+0000","last_fullsized":"2024-09-19T07:48:58.363394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4995,"log_dups_size":0,"ondisk_log_size":4995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27070,"num_write_kb":342022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"61'953","reported_seq":1118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.393736+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:48:58.393736+0000","last_peered":"2024-09-19T07:48:58.393736+0000","last_clean":"2024-09-19T07:48:58.393736+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:48:58.393736+0000","last_undegraded":"2024-09-19T07:48:58.393736+0000","last_fullsized":"2024-09-19T07:48:58.393736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:57.799442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":18964,"num_write":931,"num_write_kb":4658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"61'4894","reported_seq":5008,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364803+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:48:58.364803+0000","last_peered":"2024-09-19T07:48:58.364803+0000","last_clean":"2024-09-19T07:48:58.364803+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:48:58.364803+0000","last_undegraded":"2024-09-19T07:48:58.364803+0000","last_fullsized":"2024-09-19T07:48:58.364803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26245,"num_write_kb":306037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"64'3","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362486+0000","last_change":"2024-09-19T07:19:39.401621+0000","last_active":"2024-09-19T07:48:58.362486+0000","last_peered":"2024-09-19T07:48:58.362486+0000","last_clean":"2024-09-19T07:48:58.362486+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:48:58.362486+0000","last_undegraded":"2024-09-19T07:48:58.362486+0000","last_fullsized":"2024-09-19T07:48:58.362486+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:32.095381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"64'5242","reported_seq":5346,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371808+0000","last_change":"2024-09-19T07:19:40.366073+0000","last_active":"2024-09-19T07:48:58.371808+0000","last_peered":"2024-09-19T07:48:58.371808+0000","last_clean":"2024-09-19T07:48:58.371808+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:48:58.371808+0000","last_undegraded":"2024-09-19T07:48:58.371808+0000","last_fullsized":"2024-09-19T07:48:58.371808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5242,"log_dups_size":0,"ondisk_log_size":5242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27567,"num_write_kb":356452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"65'25","reported_seq":141,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365251+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:48:58.365251+0000","last_peered":"2024-09-19T07:48:58.365251+0000","last_clean":"2024-09-19T07:48:58.365251+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:48:58.365251+0000","last_undegraded":"2024-09-19T07:48:58.365251+0000","last_fullsized":"2024-09-19T07:48:58.365251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"61'5997","reported_seq":6111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369375+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:48:58.369375+0000","last_peered":"2024-09-19T07:48:58.369375+0000","last_clean":"2024-09-19T07:48:58.369375+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:48:58.369375+0000","last_undegraded":"2024-09-19T07:48:58.369375+0000","last_fullsized":"2024-09-19T07:48:58.369375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:15.036396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32297,"num_write_kb":399913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"64'2","reported_seq":108,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363459+0000","last_change":"2024-09-19T07:19:39.403810+0000","last_active":"2024-09-19T07:48:58.363459+0000","last_peered":"2024-09-19T07:48:58.363459+0000","last_clean":"2024-09-19T07:48:58.363459+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:48:58.363459+0000","last_undegraded":"2024-09-19T07:48:58.363459+0000","last_fullsized":"2024-09-19T07:48:58.363459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:55.714964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"64'3801","reported_seq":3905,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373585+0000","last_change":"2024-09-19T07:19:40.390455+0000","last_active":"2024-09-19T07:48:58.373585+0000","last_peered":"2024-09-19T07:48:58.373585+0000","last_clean":"2024-09-19T07:48:58.373585+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:48:58.373585+0000","last_undegraded":"2024-09-19T07:48:58.373585+0000","last_fullsized":"2024-09-19T07:48:58.373585+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20272,"num_write_kb":245240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"65'44","reported_seq":197,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.801541+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:49:01.801541+0000","last_peered":"2024-09-19T07:49:01.801541+0000","last_clean":"2024-09-19T07:49:01.801541+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:49:01.801541+0000","last_undegraded":"2024-09-19T07:49:01.801541+0000","last_fullsized":"2024-09-19T07:49:01.801541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"61'4645","reported_seq":4758,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368108+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:48:58.368108+0000","last_peered":"2024-09-19T07:48:58.368108+0000","last_clean":"2024-09-19T07:48:58.368108+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:48:58.368108+0000","last_undegraded":"2024-09-19T07:48:58.368108+0000","last_fullsized":"2024-09-19T07:48:58.368108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24199,"num_write_kb":281402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"64'764","reported_seq":895,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369315+0000","last_change":"2024-09-19T07:19:39.408309+0000","last_active":"2024-09-19T07:48:58.369315+0000","last_peered":"2024-09-19T07:48:58.369315+0000","last_clean":"2024-09-19T07:48:58.369315+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:48:58.369315+0000","last_undegraded":"2024-09-19T07:48:58.369315+0000","last_fullsized":"2024-09-19T07:48:58.369315+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:19.396261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":71635,"num_write":751,"num_write_kb":4331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"64'3560","reported_seq":3663,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365772+0000","last_change":"2024-09-19T07:19:40.372543+0000","last_active":"2024-09-19T07:48:58.365772+0000","last_peered":"2024-09-19T07:48:58.365772+0000","last_clean":"2024-09-19T07:48:58.365772+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:48:58.365772+0000","last_undegraded":"2024-09-19T07:48:58.365772+0000","last_fullsized":"2024-09-19T07:48:58.365772+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3560,"log_dups_size":0,"ondisk_log_size":3560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:53:15.682896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18641,"num_write_kb":242587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"61'5","reported_seq":158,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.793364+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:49:01.793364+0000","last_peered":"2024-09-19T07:49:01.793364+0000","last_clean":"2024-09-19T07:49:01.793364+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:49:01.793364+0000","last_undegraded":"2024-09-19T07:49:01.793364+0000","last_fullsized":"2024-09-19T07:49:01.793364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"61'5989","reported_seq":6100,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368025+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:48:58.368025+0000","last_peered":"2024-09-19T07:48:58.368025+0000","last_clean":"2024-09-19T07:48:58.368025+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:48:58.368025+0000","last_undegraded":"2024-09-19T07:48:58.368025+0000","last_fullsized":"2024-09-19T07:48:58.368025+0000","mapping_epoch":19,"log_start":"53'200","ondisk_log_start":"53'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32376,"num_write_kb":375806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'385","reported_seq":492,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:06.042052+0000","last_change":"2024-09-19T07:19:39.407045+0000","last_active":"2024-09-19T07:49:06.042052+0000","last_peered":"2024-09-19T07:49:06.042052+0000","last_clean":"2024-09-19T07:49:06.042052+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:49:06.042052+0000","last_undegraded":"2024-09-19T07:49:06.042052+0000","last_fullsized":"2024-09-19T07:49:06.042052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:24.846974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2764415,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":637,"num_write":359,"num_write_kb":2908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"64'4768","reported_seq":4873,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372391+0000","last_change":"2024-09-19T07:19:40.396520+0000","last_active":"2024-09-19T07:48:58.372391+0000","last_peered":"2024-09-19T07:48:58.372391+0000","last_clean":"2024-09-19T07:48:58.372391+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:48:58.372391+0000","last_undegraded":"2024-09-19T07:48:58.372391+0000","last_fullsized":"2024-09-19T07:48:58.372391+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25484,"num_write_kb":302986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"61'5596","reported_seq":5709,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380964+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:48:58.380964+0000","last_peered":"2024-09-19T07:48:58.380964+0000","last_clean":"2024-09-19T07:48:58.380964+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:48:58.380964+0000","last_undegraded":"2024-09-19T07:48:58.380964+0000","last_fullsized":"2024-09-19T07:48:58.380964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30081,"num_write_kb":342050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'1456","reported_seq":1615,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.401109+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:48:58.401109+0000","last_peered":"2024-09-19T07:48:58.401109+0000","last_clean":"2024-09-19T07:48:58.401109+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:48:58.401109+0000","last_undegraded":"2024-09-19T07:48:58.401109+0000","last_fullsized":"2024-09-19T07:48:58.401109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1456,"log_dups_size":0,"ondisk_log_size":1456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":42401,"num_write":1433,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"64'6","reported_seq":133,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364879+0000","last_change":"2024-09-19T07:19:39.410500+0000","last_active":"2024-09-19T07:48:58.364879+0000","last_peered":"2024-09-19T07:48:58.364879+0000","last_clean":"2024-09-19T07:48:58.364879+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:48:58.364879+0000","last_undegraded":"2024-09-19T07:48:58.364879+0000","last_fullsized":"2024-09-19T07:48:58.364879+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:04.182606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"64'4389","reported_seq":4494,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378148+0000","last_change":"2024-09-19T07:19:40.381259+0000","last_active":"2024-09-19T07:48:58.378148+0000","last_peered":"2024-09-19T07:48:58.378148+0000","last_clean":"2024-09-19T07:48:58.378148+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:48:58.378148+0000","last_undegraded":"2024-09-19T07:48:58.378148+0000","last_fullsized":"2024-09-19T07:48:58.378148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4389,"log_dups_size":0,"ondisk_log_size":4389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:22.017682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23791,"num_write_kb":304343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"61'3782","reported_seq":3895,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364288+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:48:58.364288+0000","last_peered":"2024-09-19T07:48:58.364288+0000","last_clean":"2024-09-19T07:48:58.364288+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:48:58.364288+0000","last_undegraded":"2024-09-19T07:48:58.364288+0000","last_fullsized":"2024-09-19T07:48:58.364288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19997,"num_write_kb":252226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"61'3","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371722+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:48:58.371722+0000","last_peered":"2024-09-19T07:48:58.371722+0000","last_clean":"2024-09-19T07:48:58.371722+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:48:58.371722+0000","last_undegraded":"2024-09-19T07:48:58.371722+0000","last_fullsized":"2024-09-19T07:48:58.371722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"64'5","reported_seq":112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368513+0000","last_change":"2024-09-19T07:19:39.412123+0000","last_active":"2024-09-19T07:48:58.368513+0000","last_peered":"2024-09-19T07:48:58.368513+0000","last_clean":"2024-09-19T07:48:58.368513+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:48:58.368513+0000","last_undegraded":"2024-09-19T07:48:58.368513+0000","last_fullsized":"2024-09-19T07:48:58.368513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"64'3306","reported_seq":3411,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364068+0000","last_change":"2024-09-19T07:19:40.383214+0000","last_active":"2024-09-19T07:48:58.364068+0000","last_peered":"2024-09-19T07:48:58.364068+0000","last_clean":"2024-09-19T07:48:58.364068+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:48:58.364068+0000","last_undegraded":"2024-09-19T07:48:58.364068+0000","last_fullsized":"2024-09-19T07:48:58.364068+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17836,"num_write_kb":239496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"61'4850","reported_seq":4963,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363649+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:48:58.363649+0000","last_peered":"2024-09-19T07:48:58.363649+0000","last_clean":"2024-09-19T07:48:58.363649+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:48:58.363649+0000","last_undegraded":"2024-09-19T07:48:58.363649+0000","last_fullsized":"2024-09-19T07:48:58.363649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26728,"num_write_kb":308134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"61'2","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372455+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:48:58.372455+0000","last_peered":"2024-09-19T07:48:58.372455+0000","last_clean":"2024-09-19T07:48:58.372455+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:48:58.372455+0000","last_undegraded":"2024-09-19T07:48:58.372455+0000","last_fullsized":"2024-09-19T07:48:58.372455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"64'4","reported_seq":131,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374187+0000","last_change":"2024-09-19T07:19:39.408228+0000","last_active":"2024-09-19T07:48:58.374187+0000","last_peered":"2024-09-19T07:48:58.374187+0000","last_clean":"2024-09-19T07:48:58.374187+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:48:58.374187+0000","last_undegraded":"2024-09-19T07:48:58.374187+0000","last_fullsized":"2024-09-19T07:48:58.374187+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"64'5157","reported_seq":5262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371211+0000","last_change":"2024-09-19T07:19:40.395566+0000","last_active":"2024-09-19T07:48:58.371211+0000","last_peered":"2024-09-19T07:48:58.371211+0000","last_clean":"2024-09-19T07:48:58.371211+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:48:58.371211+0000","last_undegraded":"2024-09-19T07:48:58.371211+0000","last_fullsized":"2024-09-19T07:48:58.371211+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:54:02.766021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27729,"num_write_kb":365262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"61'4508","reported_seq":4621,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365068+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:48:58.365068+0000","last_peered":"2024-09-19T07:48:58.365068+0000","last_clean":"2024-09-19T07:48:58.365068+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:48:58.365068+0000","last_undegraded":"2024-09-19T07:48:58.365068+0000","last_fullsized":"2024-09-19T07:48:58.365068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24068,"num_write_kb":289158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"61'5","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374924+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:48:58.374924+0000","last_peered":"2024-09-19T07:48:58.374924+0000","last_clean":"2024-09-19T07:48:58.374924+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:48:58.374924+0000","last_undegraded":"2024-09-19T07:48:58.374924+0000","last_fullsized":"2024-09-19T07:48:58.374924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"64'7","reported_seq":154,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364522+0000","last_change":"2024-09-19T07:19:39.424661+0000","last_active":"2024-09-19T07:48:58.364522+0000","last_peered":"2024-09-19T07:48:58.364522+0000","last_clean":"2024-09-19T07:48:58.364522+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:48:58.364522+0000","last_undegraded":"2024-09-19T07:48:58.364522+0000","last_fullsized":"2024-09-19T07:48:58.364522+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"64'3458","reported_seq":3564,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362033+0000","last_change":"2024-09-19T07:19:40.388972+0000","last_active":"2024-09-19T07:48:58.362033+0000","last_peered":"2024-09-19T07:48:58.362033+0000","last_clean":"2024-09-19T07:48:58.362033+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:48:58.362033+0000","last_undegraded":"2024-09-19T07:48:58.362033+0000","last_fullsized":"2024-09-19T07:48:58.362033+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3458,"log_dups_size":0,"ondisk_log_size":3458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18862,"num_write_kb":239540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"61'5647","reported_seq":5761,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369864+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:48:58.369864+0000","last_peered":"2024-09-19T07:48:58.369864+0000","last_clean":"2024-09-19T07:48:58.369864+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:48:58.369864+0000","last_undegraded":"2024-09-19T07:48:58.369864+0000","last_fullsized":"2024-09-19T07:48:58.369864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:23.791880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29735,"num_write_kb":364939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"63'9","reported_seq":164,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.795335+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:49:01.795335+0000","last_peered":"2024-09-19T07:49:01.795335+0000","last_clean":"2024-09-19T07:49:01.795335+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:49:01.795335+0000","last_undegraded":"2024-09-19T07:49:01.795335+0000","last_fullsized":"2024-09-19T07:49:01.795335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'45","reported_seq":213,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:08.425477+0000","last_change":"2024-09-19T07:19:39.412192+0000","last_active":"2024-09-19T07:49:08.425477+0000","last_peered":"2024-09-19T07:49:08.425477+0000","last_clean":"2024-09-19T07:49:08.425477+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:49:08.425477+0000","last_undegraded":"2024-09-19T07:49:08.425477+0000","last_fullsized":"2024-09-19T07:49:08.425477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":108,"num_write":41,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"64'4007","reported_seq":4113,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363551+0000","last_change":"2024-09-19T07:19:40.390217+0000","last_active":"2024-09-19T07:48:58.363551+0000","last_peered":"2024-09-19T07:48:58.363551+0000","last_clean":"2024-09-19T07:48:58.363551+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:48:58.363551+0000","last_undegraded":"2024-09-19T07:48:58.363551+0000","last_fullsized":"2024-09-19T07:48:58.363551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21606,"num_write_kb":266928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"61'4152","reported_seq":4265,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364929+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:48:58.364929+0000","last_peered":"2024-09-19T07:48:58.364929+0000","last_clean":"2024-09-19T07:48:58.364929+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:48:58.364929+0000","last_undegraded":"2024-09-19T07:48:58.364929+0000","last_fullsized":"2024-09-19T07:48:58.364929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22349,"num_write_kb":269915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"65'474","reported_seq":609,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.366499+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:49:03.366499+0000","last_peered":"2024-09-19T07:49:03.366499+0000","last_clean":"2024-09-19T07:49:03.366499+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:49:03.366499+0000","last_undegraded":"2024-09-19T07:49:03.366499+0000","last_fullsized":"2024-09-19T07:49:03.366499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:50.700962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":59,"num_write":1141,"num_write_kb":380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"64'10","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362143+0000","last_change":"2024-09-19T07:19:39.434150+0000","last_active":"2024-09-19T07:48:58.362143+0000","last_peered":"2024-09-19T07:48:58.362143+0000","last_clean":"2024-09-19T07:48:58.362143+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:48:58.362143+0000","last_undegraded":"2024-09-19T07:48:58.362143+0000","last_fullsized":"2024-09-19T07:48:58.362143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:13.654208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"64'4199","reported_seq":4305,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380465+0000","last_change":"2024-09-19T07:19:40.380140+0000","last_active":"2024-09-19T07:48:58.380465+0000","last_peered":"2024-09-19T07:48:58.380465+0000","last_clean":"2024-09-19T07:48:58.380465+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:48:58.380465+0000","last_undegraded":"2024-09-19T07:48:58.380465+0000","last_fullsized":"2024-09-19T07:48:58.380465+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22884,"num_write_kb":280362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"61'5900","reported_seq":6014,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372730+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:48:58.372730+0000","last_peered":"2024-09-19T07:48:58.372730+0000","last_clean":"2024-09-19T07:48:58.372730+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:48:58.372730+0000","last_undegraded":"2024-09-19T07:48:58.372730+0000","last_fullsized":"2024-09-19T07:48:58.372730+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31464,"num_write_kb":364012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"61'2","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379634+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:48:58.379634+0000","last_peered":"2024-09-19T07:48:58.379634+0000","last_clean":"2024-09-19T07:48:58.379634+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:48:58.379634+0000","last_undegraded":"2024-09-19T07:48:58.379634+0000","last_fullsized":"2024-09-19T07:48:58.379634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"64'3","reported_seq":111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370135+0000","last_change":"2024-09-19T07:19:39.411605+0000","last_active":"2024-09-19T07:48:58.370135+0000","last_peered":"2024-09-19T07:48:58.370135+0000","last_clean":"2024-09-19T07:48:58.370135+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:48:58.370135+0000","last_undegraded":"2024-09-19T07:48:58.370135+0000","last_fullsized":"2024-09-19T07:48:58.370135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:55.654483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"64'4404","reported_seq":4510,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370924+0000","last_change":"2024-09-19T07:19:40.372083+0000","last_active":"2024-09-19T07:48:58.370924+0000","last_peered":"2024-09-19T07:48:58.370924+0000","last_clean":"2024-09-19T07:48:58.370924+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:48:58.370924+0000","last_undegraded":"2024-09-19T07:48:58.370924+0000","last_fullsized":"2024-09-19T07:48:58.370924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:39.665475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24188,"num_write_kb":314198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"61'3777","reported_seq":3891,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368386+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:48:58.368386+0000","last_peered":"2024-09-19T07:48:58.368386+0000","last_clean":"2024-09-19T07:48:58.368386+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:48:58.368386+0000","last_undegraded":"2024-09-19T07:48:58.368386+0000","last_fullsized":"2024-09-19T07:48:58.368386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20312,"num_write_kb":245406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"61'2","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372774+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:48:58.372774+0000","last_peered":"2024-09-19T07:48:58.372774+0000","last_clean":"2024-09-19T07:48:58.372774+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:48:58.372774+0000","last_undegraded":"2024-09-19T07:48:58.372774+0000","last_fullsized":"2024-09-19T07:48:58.372774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'4","reported_seq":112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371516+0000","last_change":"2024-09-19T07:19:39.420340+0000","last_active":"2024-09-19T07:48:58.371516+0000","last_peered":"2024-09-19T07:48:58.371516+0000","last_clean":"2024-09-19T07:48:58.371516+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:48:58.371516+0000","last_undegraded":"2024-09-19T07:48:58.371516+0000","last_fullsized":"2024-09-19T07:48:58.371516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:29.338848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"64'4365","reported_seq":4471,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369175+0000","last_change":"2024-09-19T07:19:40.383082+0000","last_active":"2024-09-19T07:48:58.369175+0000","last_peered":"2024-09-19T07:48:58.369175+0000","last_clean":"2024-09-19T07:48:58.369175+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:48:58.369175+0000","last_undegraded":"2024-09-19T07:48:58.369175+0000","last_fullsized":"2024-09-19T07:48:58.369175+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:54.436493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24012,"num_write_kb":288918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'5396","reported_seq":5510,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370199+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:48:58.370199+0000","last_peered":"2024-09-19T07:48:58.370199+0000","last_clean":"2024-09-19T07:48:58.370199+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:48:58.370199+0000","last_undegraded":"2024-09-19T07:48:58.370199+0000","last_fullsized":"2024-09-19T07:48:58.370199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28800,"num_write_kb":336838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'4","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368215+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:48:58.368215+0000","last_peered":"2024-09-19T07:48:58.368215+0000","last_clean":"2024-09-19T07:48:58.368215+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:48:58.368215+0000","last_undegraded":"2024-09-19T07:48:58.368215+0000","last_fullsized":"2024-09-19T07:48:58.368215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"64'2","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372603+0000","last_change":"2024-09-19T07:19:39.407135+0000","last_active":"2024-09-19T07:48:58.372603+0000","last_peered":"2024-09-19T07:48:58.372603+0000","last_clean":"2024-09-19T07:48:58.372603+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:48:58.372603+0000","last_undegraded":"2024-09-19T07:48:58.372603+0000","last_fullsized":"2024-09-19T07:48:58.372603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:27.890595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"64'4926","reported_seq":5032,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378101+0000","last_change":"2024-09-19T07:19:40.368866+0000","last_active":"2024-09-19T07:48:58.378101+0000","last_peered":"2024-09-19T07:48:58.378101+0000","last_clean":"2024-09-19T07:48:58.378101+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:48:58.378101+0000","last_undegraded":"2024-09-19T07:48:58.378101+0000","last_fullsized":"2024-09-19T07:48:58.378101+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:28.893439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26066,"num_write_kb":337871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'3534","reported_seq":3648,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371281+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:48:58.371281+0000","last_peered":"2024-09-19T07:48:58.371281+0000","last_clean":"2024-09-19T07:48:58.371281+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:48:58.371281+0000","last_undegraded":"2024-09-19T07:48:58.371281+0000","last_fullsized":"2024-09-19T07:48:58.371281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19019,"num_write_kb":227884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"65'46","reported_seq":221,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.373338+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:49:03.373338+0000","last_peered":"2024-09-19T07:49:03.373338+0000","last_clean":"2024-09-19T07:49:03.373338+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:49:03.373338+0000","last_undegraded":"2024-09-19T07:49:03.373338+0000","last_fullsized":"2024-09-19T07:49:03.373338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":109,"num_write":40,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'9","reported_seq":156,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374298+0000","last_change":"2024-09-19T07:19:39.420497+0000","last_active":"2024-09-19T07:48:58.374298+0000","last_peered":"2024-09-19T07:48:58.374298+0000","last_clean":"2024-09-19T07:48:58.374298+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:48:58.374298+0000","last_undegraded":"2024-09-19T07:48:58.374298+0000","last_fullsized":"2024-09-19T07:48:58.374298+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:19.547270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"64'3425","reported_seq":3530,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371701+0000","last_change":"2024-09-19T07:19:40.387573+0000","last_active":"2024-09-19T07:48:58.371701+0000","last_peered":"2024-09-19T07:48:58.371701+0000","last_clean":"2024-09-19T07:48:58.371701+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:48:58.371701+0000","last_undegraded":"2024-09-19T07:48:58.371701+0000","last_fullsized":"2024-09-19T07:48:58.371701+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:20:43.234793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19036,"num_write_kb":252575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"61'4467","reported_seq":4581,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368938+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:48:58.368938+0000","last_peered":"2024-09-19T07:48:58.368938+0000","last_clean":"2024-09-19T07:48:58.368938+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:48:58.368938+0000","last_undegraded":"2024-09-19T07:48:58.368938+0000","last_fullsized":"2024-09-19T07:48:58.368938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:20.824916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23642,"num_write_kb":275072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'13","reported_seq":129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375436+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:48:58.375436+0000","last_peered":"2024-09-19T07:48:58.375436+0000","last_clean":"2024-09-19T07:48:58.375436+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:48:58.375436+0000","last_undegraded":"2024-09-19T07:48:58.375436+0000","last_fullsized":"2024-09-19T07:48:58.375436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'515","reported_seq":642,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:06.371206+0000","last_change":"2024-09-19T07:19:39.410752+0000","last_active":"2024-09-19T07:49:06.371206+0000","last_peered":"2024-09-19T07:49:06.371206+0000","last_clean":"2024-09-19T07:49:06.371206+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:49:06.371206+0000","last_undegraded":"2024-09-19T07:49:06.371206+0000","last_fullsized":"2024-09-19T07:49:06.371206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:47:09.376497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":53,"num_write":1231,"num_write_kb":406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"64'4906","reported_seq":5012,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372981+0000","last_change":"2024-09-19T07:19:40.392973+0000","last_active":"2024-09-19T07:48:58.372981+0000","last_peered":"2024-09-19T07:48:58.372981+0000","last_clean":"2024-09-19T07:48:58.372981+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:48:58.372981+0000","last_undegraded":"2024-09-19T07:48:58.372981+0000","last_fullsized":"2024-09-19T07:48:58.372981+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27102,"num_write_kb":350472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"61'4338","reported_seq":4452,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363793+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:48:58.363793+0000","last_peered":"2024-09-19T07:48:58.363793+0000","last_clean":"2024-09-19T07:48:58.363793+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:48:58.363793+0000","last_undegraded":"2024-09-19T07:48:58.363793+0000","last_fullsized":"2024-09-19T07:48:58.363793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22767,"num_write_kb":271598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"61'3","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365893+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:48:58.365893+0000","last_peered":"2024-09-19T07:48:58.365893+0000","last_clean":"2024-09-19T07:48:58.365893+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:48:58.365893+0000","last_undegraded":"2024-09-19T07:48:58.365893+0000","last_fullsized":"2024-09-19T07:48:58.365893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":272,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.376591+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:48:58.376591+0000","last_peered":"2024-09-19T07:48:58.376591+0000","last_clean":"2024-09-19T07:48:58.376591+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:48:58.376591+0000","last_undegraded":"2024-09-19T07:48:58.376591+0000","last_fullsized":"2024-09-19T07:48:58.376591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'2","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380743+0000","last_change":"2024-09-19T07:19:39.429085+0000","last_active":"2024-09-19T07:48:58.380743+0000","last_peered":"2024-09-19T07:48:58.380743+0000","last_clean":"2024-09-19T07:48:58.380743+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:48:58.380743+0000","last_undegraded":"2024-09-19T07:48:58.380743+0000","last_fullsized":"2024-09-19T07:48:58.380743+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"64'3599","reported_seq":3704,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378157+0000","last_change":"2024-09-19T07:19:40.382352+0000","last_active":"2024-09-19T07:48:58.378157+0000","last_peered":"2024-09-19T07:48:58.378157+0000","last_clean":"2024-09-19T07:48:58.378157+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:48:58.378157+0000","last_undegraded":"2024-09-19T07:48:58.378157+0000","last_fullsized":"2024-09-19T07:48:58.378157+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:31.024388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19570,"num_write_kb":246554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"61'5018","reported_seq":5131,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379025+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:48:58.379025+0000","last_peered":"2024-09-19T07:48:58.379025+0000","last_clean":"2024-09-19T07:48:58.379025+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:48:58.379025+0000","last_undegraded":"2024-09-19T07:48:58.379025+0000","last_fullsized":"2024-09-19T07:48:58.379025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26753,"num_write_kb":316367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"61'5","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378898+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:48:58.378898+0000","last_peered":"2024-09-19T07:48:58.378898+0000","last_clean":"2024-09-19T07:48:58.378898+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:48:58.378898+0000","last_undegraded":"2024-09-19T07:48:58.378898+0000","last_fullsized":"2024-09-19T07:48:58.378898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"64'3","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380502+0000","last_change":"2024-09-19T07:19:39.427644+0000","last_active":"2024-09-19T07:48:58.380502+0000","last_peered":"2024-09-19T07:48:58.380502+0000","last_clean":"2024-09-19T07:48:58.380502+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:48:58.380502+0000","last_undegraded":"2024-09-19T07:48:58.380502+0000","last_fullsized":"2024-09-19T07:48:58.380502+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"64'4954","reported_seq":5059,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377576+0000","last_change":"2024-09-19T07:19:40.390380+0000","last_active":"2024-09-19T07:48:58.377576+0000","last_peered":"2024-09-19T07:48:58.377576+0000","last_clean":"2024-09-19T07:48:58.377576+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:48:58.377576+0000","last_undegraded":"2024-09-19T07:48:58.377576+0000","last_fullsized":"2024-09-19T07:48:58.377576+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:26.385490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27293,"num_write_kb":322594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"61'5220","reported_seq":5318,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377825+0000","last_change":"2024-09-19T07:20:13.952747+0000","last_active":"2024-09-19T07:48:58.377825+0000","last_peered":"2024-09-19T07:48:58.377825+0000","last_clean":"2024-09-19T07:48:58.377825+0000","last_became_active":"2024-09-19T07:20:13.951449+0000","last_became_peered":"2024-09-19T07:20:13.951449+0000","last_unstale":"2024-09-19T07:48:58.377825+0000","last_undegraded":"2024-09-19T07:48:58.377825+0000","last_fullsized":"2024-09-19T07:48:58.377825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:56.805700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27694,"num_write_kb":340542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"61'6","reported_seq":141,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374333+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:48:58.374333+0000","last_peered":"2024-09-19T07:48:58.374333+0000","last_clean":"2024-09-19T07:48:58.374333+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:48:58.374333+0000","last_undegraded":"2024-09-19T07:48:58.374333+0000","last_fullsized":"2024-09-19T07:48:58.374333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"64'1403","reported_seq":1553,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369177+0000","last_change":"2024-09-19T07:19:39.410638+0000","last_active":"2024-09-19T07:48:58.369177+0000","last_peered":"2024-09-19T07:48:58.369177+0000","last_clean":"2024-09-19T07:48:58.369177+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:48:58.369177+0000","last_undegraded":"2024-09-19T07:48:58.369177+0000","last_fullsized":"2024-09-19T07:48:58.369177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1403,"log_dups_size":0,"ondisk_log_size":1403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:32.073136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":63,"num_read_kb":30148,"num_write":1379,"num_write_kb":4466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"64'3869","reported_seq":3974,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372595+0000","last_change":"2024-09-19T07:19:40.378481+0000","last_active":"2024-09-19T07:48:58.372595+0000","last_peered":"2024-09-19T07:48:58.372595+0000","last_clean":"2024-09-19T07:48:58.372595+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:48:58.372595+0000","last_undegraded":"2024-09-19T07:48:58.372595+0000","last_fullsized":"2024-09-19T07:48:58.372595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3869,"log_dups_size":0,"ondisk_log_size":3869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:30.585686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20920,"num_write_kb":264190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"61'6113","reported_seq":6211,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373547+0000","last_change":"2024-09-19T07:20:14.209767+0000","last_active":"2024-09-19T07:48:58.373547+0000","last_peered":"2024-09-19T07:48:58.373547+0000","last_clean":"2024-09-19T07:48:58.373547+0000","last_became_active":"2024-09-19T07:20:14.209141+0000","last_became_peered":"2024-09-19T07:20:14.209141+0000","last_unstale":"2024-09-19T07:48:58.373547+0000","last_undegraded":"2024-09-19T07:48:58.373547+0000","last_fullsized":"2024-09-19T07:48:58.373547+0000","mapping_epoch":26,"log_start":"54'300","ondisk_log_start":"54'300","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:34:10.251087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32041,"num_write_kb":407182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"61'7","reported_seq":162,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.794320+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:49:01.794320+0000","last_peered":"2024-09-19T07:49:01.794320+0000","last_clean":"2024-09-19T07:49:01.794320+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:49:01.794320+0000","last_undegraded":"2024-09-19T07:49:01.794320+0000","last_fullsized":"2024-09-19T07:49:01.794320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"64'5","reported_seq":112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365126+0000","last_change":"2024-09-19T07:19:39.437656+0000","last_active":"2024-09-19T07:48:58.365126+0000","last_peered":"2024-09-19T07:48:58.365126+0000","last_clean":"2024-09-19T07:48:58.365126+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:48:58.365126+0000","last_undegraded":"2024-09-19T07:48:58.365126+0000","last_fullsized":"2024-09-19T07:48:58.365126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:41.468096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"64'4634","reported_seq":4739,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377494+0000","last_change":"2024-09-19T07:19:40.379289+0000","last_active":"2024-09-19T07:48:58.377494+0000","last_peered":"2024-09-19T07:48:58.377494+0000","last_clean":"2024-09-19T07:48:58.377494+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:48:58.377494+0000","last_undegraded":"2024-09-19T07:48:58.377494+0000","last_fullsized":"2024-09-19T07:48:58.377494+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:20.399106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25462,"num_write_kb":298798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"61'4007","reported_seq":4120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375692+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:48:58.375692+0000","last_peered":"2024-09-19T07:48:58.375692+0000","last_clean":"2024-09-19T07:48:58.375692+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:48:58.375692+0000","last_undegraded":"2024-09-19T07:48:58.375692+0000","last_fullsized":"2024-09-19T07:48:58.375692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21937,"num_write_kb":251948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"61'4","reported_seq":139,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369549+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:48:58.369549+0000","last_peered":"2024-09-19T07:48:58.369549+0000","last_clean":"2024-09-19T07:48:58.369549+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:48:58.369549+0000","last_undegraded":"2024-09-19T07:48:58.369549+0000","last_fullsized":"2024-09-19T07:48:58.369549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"64'2","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363818+0000","last_change":"2024-09-19T07:19:39.413589+0000","last_active":"2024-09-19T07:48:58.363818+0000","last_peered":"2024-09-19T07:48:58.363818+0000","last_clean":"2024-09-19T07:48:58.363818+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:48:58.363818+0000","last_undegraded":"2024-09-19T07:48:58.363818+0000","last_fullsized":"2024-09-19T07:48:58.363818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"64'3939","reported_seq":4043,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364105+0000","last_change":"2024-09-19T07:19:40.387159+0000","last_active":"2024-09-19T07:48:58.364105+0000","last_peered":"2024-09-19T07:48:58.364105+0000","last_clean":"2024-09-19T07:48:58.364105+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:48:58.364105+0000","last_undegraded":"2024-09-19T07:48:58.364105+0000","last_fullsized":"2024-09-19T07:48:58.364105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20741,"num_write_kb":275861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'4660","reported_seq":4774,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373923+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:48:58.373923+0000","last_peered":"2024-09-19T07:48:58.373923+0000","last_clean":"2024-09-19T07:48:58.373923+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:48:58.373923+0000","last_undegraded":"2024-09-19T07:48:58.373923+0000","last_fullsized":"2024-09-19T07:48:58.373923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25264,"num_write_kb":302007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"65'15","reported_seq":213,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.799096+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:49:01.799096+0000","last_peered":"2024-09-19T07:49:01.799096+0000","last_clean":"2024-09-19T07:49:01.799096+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:49:01.799096+0000","last_undegraded":"2024-09-19T07:49:01.799096+0000","last_fullsized":"2024-09-19T07:49:01.799096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"64'3","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369815+0000","last_change":"2024-09-19T07:19:39.406110+0000","last_active":"2024-09-19T07:48:58.369815+0000","last_peered":"2024-09-19T07:48:58.369815+0000","last_clean":"2024-09-19T07:48:58.369815+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:48:58.369815+0000","last_undegraded":"2024-09-19T07:48:58.369815+0000","last_fullsized":"2024-09-19T07:48:58.369815+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:10:05.224225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"64'4590","reported_seq":4694,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366527+0000","last_change":"2024-09-19T07:19:40.365764+0000","last_active":"2024-09-19T07:48:58.366527+0000","last_peered":"2024-09-19T07:48:58.366527+0000","last_clean":"2024-09-19T07:48:58.366527+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:48:58.366527+0000","last_undegraded":"2024-09-19T07:48:58.366527+0000","last_fullsized":"2024-09-19T07:48:58.366527+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:04:03.242292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24790,"num_write_kb":304361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"61'3873","reported_seq":3985,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380282+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:48:58.380282+0000","last_peered":"2024-09-19T07:48:58.380282+0000","last_clean":"2024-09-19T07:48:58.380282+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:48:58.380282+0000","last_undegraded":"2024-09-19T07:48:58.380282+0000","last_fullsized":"2024-09-19T07:48:58.380282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20508,"num_write_kb":250225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"61'5","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381953+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:48:58.381953+0000","last_peered":"2024-09-19T07:48:58.381953+0000","last_clean":"2024-09-19T07:48:58.381953+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:48:58.381953+0000","last_undegraded":"2024-09-19T07:48:58.381953+0000","last_fullsized":"2024-09-19T07:48:58.381953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"64'4","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364247+0000","last_change":"2024-09-19T07:19:39.429768+0000","last_active":"2024-09-19T07:48:58.364247+0000","last_peered":"2024-09-19T07:48:58.364247+0000","last_clean":"2024-09-19T07:48:58.364247+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:48:58.364247+0000","last_undegraded":"2024-09-19T07:48:58.364247+0000","last_fullsized":"2024-09-19T07:48:58.364247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:47.503631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"64'3485","reported_seq":3589,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366523+0000","last_change":"2024-09-19T07:19:40.373542+0000","last_active":"2024-09-19T07:48:58.366523+0000","last_peered":"2024-09-19T07:48:58.366523+0000","last_clean":"2024-09-19T07:48:58.366523+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:48:58.366523+0000","last_undegraded":"2024-09-19T07:48:58.366523+0000","last_fullsized":"2024-09-19T07:48:58.366523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:14.326662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19669,"num_write_kb":242834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"61'5231","reported_seq":5343,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373420+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:48:58.373420+0000","last_peered":"2024-09-19T07:48:58.373420+0000","last_clean":"2024-09-19T07:48:58.373420+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:48:58.373420+0000","last_undegraded":"2024-09-19T07:48:58.373420+0000","last_fullsized":"2024-09-19T07:48:58.373420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28071,"num_write_kb":333869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"61'3","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367894+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:48:58.367894+0000","last_peered":"2024-09-19T07:48:58.367894+0000","last_clean":"2024-09-19T07:48:58.367894+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:48:58.367894+0000","last_undegraded":"2024-09-19T07:48:58.367894+0000","last_fullsized":"2024-09-19T07:48:58.367894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"64'328","reported_seq":535,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363630+0000","last_change":"2024-09-19T07:19:39.411074+0000","last_active":"2024-09-19T07:48:58.363630+0000","last_peered":"2024-09-19T07:48:58.363630+0000","last_clean":"2024-09-19T07:48:58.363630+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:48:58.363630+0000","last_undegraded":"2024-09-19T07:48:58.363630+0000","last_fullsized":"2024-09-19T07:48:58.363630+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:53.992787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":181,"num_read_kb":77990,"num_write":325,"num_write_kb":4282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"64'5112","reported_seq":5216,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363063+0000","last_change":"2024-09-19T07:19:40.364814+0000","last_active":"2024-09-19T07:48:58.363063+0000","last_peered":"2024-09-19T07:48:58.363063+0000","last_clean":"2024-09-19T07:48:58.363063+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:48:58.363063+0000","last_undegraded":"2024-09-19T07:48:58.363063+0000","last_fullsized":"2024-09-19T07:48:58.363063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5112,"log_dups_size":0,"ondisk_log_size":5112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27577,"num_write_kb":360586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"61'4085","reported_seq":4197,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369111+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:48:58.369111+0000","last_peered":"2024-09-19T07:48:58.369111+0000","last_clean":"2024-09-19T07:48:58.369111+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:48:58.369111+0000","last_undegraded":"2024-09-19T07:48:58.369111+0000","last_fullsized":"2024-09-19T07:48:58.369111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:57.131912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22122,"num_write_kb":273218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"65'10","reported_seq":164,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.897451+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:48:58.897451+0000","last_peered":"2024-09-19T07:48:58.897451+0000","last_clean":"2024-09-19T07:48:58.897451+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:48:58.897451+0000","last_undegraded":"2024-09-19T07:48:58.897451+0000","last_fullsized":"2024-09-19T07:48:58.897451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:23.367365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'29","reported_seq":195,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.423376+0000","last_change":"2024-09-19T07:19:39.431616+0000","last_active":"2024-09-19T07:48:58.423376+0000","last_peered":"2024-09-19T07:48:58.423376+0000","last_clean":"2024-09-19T07:48:58.423376+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:48:58.423376+0000","last_undegraded":"2024-09-19T07:48:58.423376+0000","last_fullsized":"2024-09-19T07:48:58.423376+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:18.149581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"64'4543","reported_seq":4647,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363327+0000","last_change":"2024-09-19T07:19:40.381443+0000","last_active":"2024-09-19T07:48:58.363327+0000","last_peered":"2024-09-19T07:48:58.363327+0000","last_clean":"2024-09-19T07:48:58.363327+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:48:58.363327+0000","last_undegraded":"2024-09-19T07:48:58.363327+0000","last_fullsized":"2024-09-19T07:48:58.363327+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24993,"num_write_kb":317373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"61'3981","reported_seq":4093,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375912+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:48:58.375912+0000","last_peered":"2024-09-19T07:48:58.375912+0000","last_clean":"2024-09-19T07:48:58.375912+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:48:58.375912+0000","last_undegraded":"2024-09-19T07:48:58.375912+0000","last_fullsized":"2024-09-19T07:48:58.375912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3981,"log_dups_size":0,"ondisk_log_size":3981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21890,"num_write_kb":263866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"65'352","reported_seq":567,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.802657+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:49:01.802657+0000","last_peered":"2024-09-19T07:49:01.802657+0000","last_clean":"2024-09-19T07:49:01.802657+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:49:01.802657+0000","last_undegraded":"2024-09-19T07:49:01.802657+0000","last_fullsized":"2024-09-19T07:49:01.802657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":181,"num_read_kb":77990,"num_write":347,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"64'3","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371323+0000","last_change":"2024-09-19T07:19:39.423972+0000","last_active":"2024-09-19T07:48:58.371323+0000","last_peered":"2024-09-19T07:48:58.371323+0000","last_clean":"2024-09-19T07:48:58.371323+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:48:58.371323+0000","last_undegraded":"2024-09-19T07:48:58.371323+0000","last_fullsized":"2024-09-19T07:48:58.371323+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:45.387040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"64'3743","reported_seq":3847,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371573+0000","last_change":"2024-09-19T07:19:40.382092+0000","last_active":"2024-09-19T07:48:58.371573+0000","last_peered":"2024-09-19T07:48:58.371573+0000","last_clean":"2024-09-19T07:48:58.371573+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:48:58.371573+0000","last_undegraded":"2024-09-19T07:48:58.371573+0000","last_fullsized":"2024-09-19T07:48:58.371573+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20626,"num_write_kb":263086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"61'6160","reported_seq":6272,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369147+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:48:58.369147+0000","last_peered":"2024-09-19T07:48:58.369147+0000","last_clean":"2024-09-19T07:48:58.369147+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:48:58.369147+0000","last_undegraded":"2024-09-19T07:48:58.369147+0000","last_fullsized":"2024-09-19T07:48:58.369147+0000","mapping_epoch":19,"log_start":"54'700","ondisk_log_start":"54'700","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32461,"num_write_kb":376158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"65'29","reported_seq":203,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.368449+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:49:03.368449+0000","last_peered":"2024-09-19T07:49:03.368449+0000","last_clean":"2024-09-19T07:49:03.368449+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:49:03.368449+0000","last_undegraded":"2024-09-19T07:49:03.368449+0000","last_fullsized":"2024-09-19T07:49:03.368449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"64'8","reported_seq":154,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369556+0000","last_change":"2024-09-19T07:19:39.424879+0000","last_active":"2024-09-19T07:48:58.369556+0000","last_peered":"2024-09-19T07:48:58.369556+0000","last_clean":"2024-09-19T07:48:58.369556+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:48:58.369556+0000","last_undegraded":"2024-09-19T07:48:58.369556+0000","last_fullsized":"2024-09-19T07:48:58.369556+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:30.021543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"64'4564","reported_seq":4670,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382249+0000","last_change":"2024-09-19T07:19:40.384136+0000","last_active":"2024-09-19T07:48:58.382249+0000","last_peered":"2024-09-19T07:48:58.382249+0000","last_clean":"2024-09-19T07:48:58.382249+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:48:58.382249+0000","last_undegraded":"2024-09-19T07:48:58.382249+0000","last_fullsized":"2024-09-19T07:48:58.382249+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25533,"num_write_kb":303195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"61'5068","reported_seq":5180,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369896+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:48:58.369896+0000","last_peered":"2024-09-19T07:48:58.369896+0000","last_clean":"2024-09-19T07:48:58.369896+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:48:58.369896+0000","last_undegraded":"2024-09-19T07:48:58.369896+0000","last_fullsized":"2024-09-19T07:48:58.369896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:48.308137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26828,"num_write_kb":316657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"61'3","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368011+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:48:58.368011+0000","last_peered":"2024-09-19T07:48:58.368011+0000","last_clean":"2024-09-19T07:48:58.368011+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:48:58.368011+0000","last_undegraded":"2024-09-19T07:48:58.368011+0000","last_fullsized":"2024-09-19T07:48:58.368011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'15","reported_seq":205,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362420+0000","last_change":"2024-09-19T07:19:39.410445+0000","last_active":"2024-09-19T07:48:58.362420+0000","last_peered":"2024-09-19T07:48:58.362420+0000","last_clean":"2024-09-19T07:48:58.362420+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:48:58.362420+0000","last_undegraded":"2024-09-19T07:48:58.362420+0000","last_fullsized":"2024-09-19T07:48:58.362420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:27.020327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"64'3771","reported_seq":3875,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373168+0000","last_change":"2024-09-19T07:19:40.389499+0000","last_active":"2024-09-19T07:48:58.373168+0000","last_peered":"2024-09-19T07:48:58.373168+0000","last_clean":"2024-09-19T07:48:58.373168+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:48:58.373168+0000","last_undegraded":"2024-09-19T07:48:58.373168+0000","last_fullsized":"2024-09-19T07:48:58.373168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3771,"log_dups_size":0,"ondisk_log_size":3771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.520872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19869,"num_write_kb":255725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"61'4973","reported_seq":5085,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367096+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:48:58.367096+0000","last_peered":"2024-09-19T07:48:58.367096+0000","last_clean":"2024-09-19T07:48:58.367096+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:48:58.367096+0000","last_undegraded":"2024-09-19T07:48:58.367096+0000","last_fullsized":"2024-09-19T07:48:58.367096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26723,"num_write_kb":320185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"65'6","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382937+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:48:58.382937+0000","last_peered":"2024-09-19T07:48:58.382937+0000","last_clean":"2024-09-19T07:48:58.382937+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:48:58.382937+0000","last_undegraded":"2024-09-19T07:48:58.382937+0000","last_fullsized":"2024-09-19T07:48:58.382937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"64'5","reported_seq":111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362191+0000","last_change":"2024-09-19T07:19:39.416943+0000","last_active":"2024-09-19T07:48:58.362191+0000","last_peered":"2024-09-19T07:48:58.362191+0000","last_clean":"2024-09-19T07:48:58.362191+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:48:58.362191+0000","last_undegraded":"2024-09-19T07:48:58.362191+0000","last_fullsized":"2024-09-19T07:48:58.362191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:35.555826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"64'5394","reported_seq":5500,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382132+0000","last_change":"2024-09-19T07:19:40.394993+0000","last_active":"2024-09-19T07:48:58.382132+0000","last_peered":"2024-09-19T07:48:58.382132+0000","last_clean":"2024-09-19T07:48:58.382132+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:48:58.382132+0000","last_undegraded":"2024-09-19T07:48:58.382132+0000","last_fullsized":"2024-09-19T07:48:58.382132+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5394,"log_dups_size":0,"ondisk_log_size":5394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30095,"num_write_kb":374705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"64'25","reported_seq":133,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369509+0000","last_change":"2024-09-19T07:19:39.418885+0000","last_active":"2024-09-19T07:48:58.369509+0000","last_peered":"2024-09-19T07:48:58.369509+0000","last_clean":"2024-09-19T07:48:58.369509+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:48:58.369509+0000","last_undegraded":"2024-09-19T07:48:58.369509+0000","last_fullsized":"2024-09-19T07:48:58.369509+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:35:45.790411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"61'4822","reported_seq":4934,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371995+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:48:58.371995+0000","last_peered":"2024-09-19T07:48:58.371995+0000","last_clean":"2024-09-19T07:48:58.371995+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:48:58.371995+0000","last_undegraded":"2024-09-19T07:48:58.371995+0000","last_fullsized":"2024-09-19T07:48:58.371995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25208,"num_write_kb":326524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"61'2","reported_seq":116,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367640+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:48:58.367640+0000","last_peered":"2024-09-19T07:48:58.367640+0000","last_clean":"2024-09-19T07:48:58.367640+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:48:58.367640+0000","last_undegraded":"2024-09-19T07:48:58.367640+0000","last_fullsized":"2024-09-19T07:48:58.367640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"64'4690","reported_seq":4796,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369407+0000","last_change":"2024-09-19T07:19:40.374048+0000","last_active":"2024-09-19T07:48:58.369407+0000","last_peered":"2024-09-19T07:48:58.369407+0000","last_clean":"2024-09-19T07:48:58.369407+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:48:58.369407+0000","last_undegraded":"2024-09-19T07:48:58.369407+0000","last_fullsized":"2024-09-19T07:48:58.369407+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:40.084827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25924,"num_write_kb":313037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"64'964","reported_seq":1117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382679+0000","last_change":"2024-09-19T07:19:39.432426+0000","last_active":"2024-09-19T07:48:58.382679+0000","last_peered":"2024-09-19T07:48:58.382679+0000","last_clean":"2024-09-19T07:48:58.382679+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:48:58.382679+0000","last_undegraded":"2024-09-19T07:48:58.382679+0000","last_fullsized":"2024-09-19T07:48:58.382679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":964,"log_dups_size":0,"ondisk_log_size":964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":8896,"num_write":940,"num_write_kb":4641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"61'5311","reported_seq":5423,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362632+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:48:58.362632+0000","last_peered":"2024-09-19T07:48:58.362632+0000","last_clean":"2024-09-19T07:48:58.362632+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:48:58.362632+0000","last_undegraded":"2024-09-19T07:48:58.362632+0000","last_fullsized":"2024-09-19T07:48:58.362632+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28108,"num_write_kb":333978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"61'3","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363067+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:48:58.363067+0000","last_peered":"2024-09-19T07:48:58.363067+0000","last_clean":"2024-09-19T07:48:58.363067+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:48:58.363067+0000","last_undegraded":"2024-09-19T07:48:58.363067+0000","last_fullsized":"2024-09-19T07:48:58.363067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"64'5270","reported_seq":5373,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373581+0000","last_change":"2024-09-19T07:19:40.384934+0000","last_active":"2024-09-19T07:48:58.373581+0000","last_peered":"2024-09-19T07:48:58.373581+0000","last_clean":"2024-09-19T07:48:58.373581+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:48:58.373581+0000","last_undegraded":"2024-09-19T07:48:58.373581+0000","last_fullsized":"2024-09-19T07:48:58.373581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29270,"num_write_kb":363254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"65'416","reported_seq":535,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:04.513350+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:49:04.513350+0000","last_peered":"2024-09-19T07:49:04.513350+0000","last_clean":"2024-09-19T07:49:04.513350+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:49:04.513350+0000","last_undegraded":"2024-09-19T07:49:04.513350+0000","last_fullsized":"2024-09-19T07:49:04.513350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2715831,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6193,"num_write":390,"num_write_kb":2885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"61'3569","reported_seq":3680,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371694+0000","last_change":"2024-09-19T07:19:32.289374+0000","last_active":"2024-09-19T07:48:58.371694+0000","last_peered":"2024-09-19T07:48:58.371694+0000","last_clean":"2024-09-19T07:48:58.371694+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:48:58.371694+0000","last_undegraded":"2024-09-19T07:48:58.371694+0000","last_fullsized":"2024-09-19T07:48:58.371694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18120,"num_write_kb":244659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"64'5","reported_seq":150,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378720+0000","last_change":"2024-09-19T07:19:39.410582+0000","last_active":"2024-09-19T07:48:58.378720+0000","last_peered":"2024-09-19T07:48:58.378720+0000","last_clean":"2024-09-19T07:48:58.378720+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:48:58.378720+0000","last_undegraded":"2024-09-19T07:48:58.378720+0000","last_fullsized":"2024-09-19T07:48:58.378720+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:41:40.592907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"64'3415","reported_seq":3520,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364298+0000","last_change":"2024-09-19T07:19:40.393950+0000","last_active":"2024-09-19T07:48:58.364298+0000","last_peered":"2024-09-19T07:48:58.364298+0000","last_clean":"2024-09-19T07:48:58.364298+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:48:58.364298+0000","last_undegraded":"2024-09-19T07:48:58.364298+0000","last_fullsized":"2024-09-19T07:48:58.364298+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18072,"num_write_kb":240550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"61'712","reported_seq":848,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364844+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:48:58.364844+0000","last_peered":"2024-09-19T07:48:58.364844+0000","last_clean":"2024-09-19T07:48:58.364844+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:48:58.364844+0000","last_undegraded":"2024-09-19T07:48:58.364844+0000","last_fullsized":"2024-09-19T07:48:58.364844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":74743,"num_write":701,"num_write_kb":4298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"61'4395","reported_seq":4507,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368988+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:48:58.368988+0000","last_peered":"2024-09-19T07:48:58.368988+0000","last_clean":"2024-09-19T07:48:58.368988+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:48:58.368988+0000","last_undegraded":"2024-09-19T07:48:58.368988+0000","last_fullsized":"2024-09-19T07:48:58.368988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22455,"num_write_kb":278564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"64'40","reported_seq":185,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363443+0000","last_change":"2024-09-19T07:19:39.410662+0000","last_active":"2024-09-19T07:48:58.363443+0000","last_peered":"2024-09-19T07:48:58.363443+0000","last_clean":"2024-09-19T07:48:58.363443+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:48:58.363443+0000","last_undegraded":"2024-09-19T07:48:58.363443+0000","last_fullsized":"2024-09-19T07:48:58.363443+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:19.196074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"64'4798","reported_seq":4903,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364274+0000","last_change":"2024-09-19T07:19:40.385581+0000","last_active":"2024-09-19T07:48:58.364274+0000","last_peered":"2024-09-19T07:48:58.364274+0000","last_clean":"2024-09-19T07:48:58.364274+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:48:58.364274+0000","last_undegraded":"2024-09-19T07:48:58.364274+0000","last_fullsized":"2024-09-19T07:48:58.364274+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:14:13.749729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27082,"num_write_kb":321689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"61'4997","reported_seq":5111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380242+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:48:58.380242+0000","last_peered":"2024-09-19T07:48:58.380242+0000","last_clean":"2024-09-19T07:48:58.380242+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:48:58.380242+0000","last_undegraded":"2024-09-19T07:48:58.380242+0000","last_fullsized":"2024-09-19T07:48:58.380242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4997,"log_dups_size":0,"ondisk_log_size":4997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26206,"num_write_kb":309941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"64'5490","reported_seq":5596,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373817+0000","last_change":"2024-09-19T07:19:40.377608+0000","last_active":"2024-09-19T07:48:58.373817+0000","last_peered":"2024-09-19T07:48:58.373817+0000","last_clean":"2024-09-19T07:48:58.373817+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:48:58.373817+0000","last_undegraded":"2024-09-19T07:48:58.373817+0000","last_fullsized":"2024-09-19T07:48:58.373817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5490,"log_dups_size":0,"ondisk_log_size":5490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29613,"num_write_kb":364653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"61'4890","reported_seq":5005,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377509+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:48:58.377509+0000","last_peered":"2024-09-19T07:48:58.377509+0000","last_clean":"2024-09-19T07:48:58.377509+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:48:58.377509+0000","last_undegraded":"2024-09-19T07:48:58.377509+0000","last_fullsized":"2024-09-19T07:48:58.377509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4890,"log_dups_size":0,"ondisk_log_size":4890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25338,"num_write_kb":314559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"64'4569","reported_seq":4675,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366516+0000","last_change":"2024-09-19T07:19:40.381262+0000","last_active":"2024-09-19T07:48:58.366516+0000","last_peered":"2024-09-19T07:48:58.366516+0000","last_clean":"2024-09-19T07:48:58.366516+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:48:58.366516+0000","last_undegraded":"2024-09-19T07:48:58.366516+0000","last_fullsized":"2024-09-19T07:48:58.366516+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:34.425472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25275,"num_write_kb":314704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"61'5227","reported_seq":5343,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367085+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:48:58.367085+0000","last_peered":"2024-09-19T07:48:58.367085+0000","last_clean":"2024-09-19T07:48:58.367085+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:48:58.367085+0000","last_undegraded":"2024-09-19T07:48:58.367085+0000","last_fullsized":"2024-09-19T07:48:58.367085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27444,"num_write_kb":331433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"64'4939","reported_seq":5044,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373561+0000","last_change":"2024-09-19T07:19:40.386432+0000","last_active":"2024-09-19T07:48:58.373561+0000","last_peered":"2024-09-19T07:48:58.373561+0000","last_clean":"2024-09-19T07:48:58.373561+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:48:58.373561+0000","last_undegraded":"2024-09-19T07:48:58.373561+0000","last_fullsized":"2024-09-19T07:48:58.373561+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27263,"num_write_kb":334786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"61'4385","reported_seq":4500,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371314+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:48:58.371314+0000","last_peered":"2024-09-19T07:48:58.371314+0000","last_clean":"2024-09-19T07:48:58.371314+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:48:58.371314+0000","last_undegraded":"2024-09-19T07:48:58.371314+0000","last_fullsized":"2024-09-19T07:48:58.371314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22838,"num_write_kb":271812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"64'4859","reported_seq":4967,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372828+0000","last_change":"2024-09-19T07:19:40.378630+0000","last_active":"2024-09-19T07:48:58.372828+0000","last_peered":"2024-09-19T07:48:58.372828+0000","last_clean":"2024-09-19T07:48:58.372828+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:48:58.372828+0000","last_undegraded":"2024-09-19T07:48:58.372828+0000","last_fullsized":"2024-09-19T07:48:58.372828+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26164,"num_write_kb":338417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"61'4483","reported_seq":4597,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364173+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:48:58.364173+0000","last_peered":"2024-09-19T07:48:58.364173+0000","last_clean":"2024-09-19T07:48:58.364173+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:48:58.364173+0000","last_undegraded":"2024-09-19T07:48:58.364173+0000","last_fullsized":"2024-09-19T07:48:58.364173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23870,"num_write_kb":292652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"64'4105","reported_seq":4212,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368603+0000","last_change":"2024-09-19T07:19:40.381365+0000","last_active":"2024-09-19T07:48:58.368603+0000","last_peered":"2024-09-19T07:48:58.368603+0000","last_clean":"2024-09-19T07:48:58.368603+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:48:58.368603+0000","last_undegraded":"2024-09-19T07:48:58.368603+0000","last_fullsized":"2024-09-19T07:48:58.368603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21923,"num_write_kb":284540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"61'5078","reported_seq":5191,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374056+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:48:58.374056+0000","last_peered":"2024-09-19T07:48:58.374056+0000","last_clean":"2024-09-19T07:48:58.374056+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:48:58.374056+0000","last_undegraded":"2024-09-19T07:48:58.374056+0000","last_fullsized":"2024-09-19T07:48:58.374056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28059,"num_write_kb":313378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"64'3824","reported_seq":3930,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372461+0000","last_change":"2024-09-19T07:19:40.387116+0000","last_active":"2024-09-19T07:48:58.372461+0000","last_peered":"2024-09-19T07:48:58.372461+0000","last_clean":"2024-09-19T07:48:58.372461+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:48:58.372461+0000","last_undegraded":"2024-09-19T07:48:58.372461+0000","last_fullsized":"2024-09-19T07:48:58.372461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20622,"num_write_kb":295649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"61'4725","reported_seq":4838,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382564+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:48:58.382564+0000","last_peered":"2024-09-19T07:48:58.382564+0000","last_clean":"2024-09-19T07:48:58.382564+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:48:58.382564+0000","last_undegraded":"2024-09-19T07:48:58.382564+0000","last_fullsized":"2024-09-19T07:48:58.382564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25170,"num_write_kb":293477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"64'4764","reported_seq":4871,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380511+0000","last_change":"2024-09-19T07:19:40.357629+0000","last_active":"2024-09-19T07:48:58.380511+0000","last_peered":"2024-09-19T07:48:58.380511+0000","last_clean":"2024-09-19T07:48:58.380511+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:48:58.380511+0000","last_undegraded":"2024-09-19T07:48:58.380511+0000","last_fullsized":"2024-09-19T07:48:58.380511+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25208,"num_write_kb":309995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"61'5411","reported_seq":5525,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364434+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:48:58.364434+0000","last_peered":"2024-09-19T07:48:58.364434+0000","last_clean":"2024-09-19T07:48:58.364434+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:48:58.364434+0000","last_undegraded":"2024-09-19T07:48:58.364434+0000","last_fullsized":"2024-09-19T07:48:58.364434+0000","mapping_epoch":19,"log_start":"54'200","ondisk_log_start":"54'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28760,"num_write_kb":348949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"64'4221","reported_seq":4326,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363476+0000","last_change":"2024-09-19T07:19:40.388597+0000","last_active":"2024-09-19T07:48:58.363476+0000","last_peered":"2024-09-19T07:48:58.363476+0000","last_clean":"2024-09-19T07:48:58.363476+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:48:58.363476+0000","last_undegraded":"2024-09-19T07:48:58.363476+0000","last_fullsized":"2024-09-19T07:48:58.363476+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:34:18.402016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21805,"num_write_kb":288249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"61'4974","reported_seq":5093,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365664+0000","last_change":"2024-09-19T07:20:13.954029+0000","last_active":"2024-09-19T07:48:58.365664+0000","last_peered":"2024-09-19T07:48:58.365664+0000","last_clean":"2024-09-19T07:48:58.365664+0000","last_became_active":"2024-09-19T07:20:13.953118+0000","last_became_peered":"2024-09-19T07:20:13.953118+0000","last_unstale":"2024-09-19T07:48:58.365664+0000","last_undegraded":"2024-09-19T07:48:58.365664+0000","last_fullsized":"2024-09-19T07:48:58.365664+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:27:21.388787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25995,"num_write_kb":309048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"64'5715","reported_seq":5819,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375222+0000","last_change":"2024-09-19T07:19:40.383779+0000","last_active":"2024-09-19T07:48:58.375222+0000","last_peered":"2024-09-19T07:48:58.375222+0000","last_clean":"2024-09-19T07:48:58.375222+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:48:58.375222+0000","last_undegraded":"2024-09-19T07:48:58.375222+0000","last_fullsized":"2024-09-19T07:48:58.375222+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5715,"log_dups_size":0,"ondisk_log_size":5715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30021,"num_write_kb":374301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"61'4405","reported_seq":4518,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362645+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:48:58.362645+0000","last_peered":"2024-09-19T07:48:58.362645+0000","last_clean":"2024-09-19T07:48:58.362645+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:48:58.362645+0000","last_undegraded":"2024-09-19T07:48:58.362645+0000","last_fullsized":"2024-09-19T07:48:58.362645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4405,"log_dups_size":0,"ondisk_log_size":4405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23742,"num_write_kb":275461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"64'3931","reported_seq":4036,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378138+0000","last_change":"2024-09-19T07:19:40.376342+0000","last_active":"2024-09-19T07:48:58.378138+0000","last_peered":"2024-09-19T07:48:58.378138+0000","last_clean":"2024-09-19T07:48:58.378138+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:48:58.378138+0000","last_undegraded":"2024-09-19T07:48:58.378138+0000","last_fullsized":"2024-09-19T07:48:58.378138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3931,"log_dups_size":0,"ondisk_log_size":3931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:22.997744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22002,"num_write_kb":260466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"61'4975","reported_seq":5088,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365882+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:48:58.365882+0000","last_peered":"2024-09-19T07:48:58.365882+0000","last_clean":"2024-09-19T07:48:58.365882+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:48:58.365882+0000","last_undegraded":"2024-09-19T07:48:58.365882+0000","last_fullsized":"2024-09-19T07:48:58.365882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4975,"log_dups_size":0,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26500,"num_write_kb":311194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"64'5003","reported_seq":5108,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362879+0000","last_change":"2024-09-19T07:19:40.361504+0000","last_active":"2024-09-19T07:48:58.362879+0000","last_peered":"2024-09-19T07:48:58.362879+0000","last_clean":"2024-09-19T07:48:58.362879+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:48:58.362879+0000","last_undegraded":"2024-09-19T07:48:58.362879+0000","last_fullsized":"2024-09-19T07:48:58.362879+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:30:31.110669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26890,"num_write_kb":325027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"61'5877","reported_seq":5992,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370513+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:48:58.370513+0000","last_peered":"2024-09-19T07:48:58.370513+0000","last_clean":"2024-09-19T07:48:58.370513+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:48:58.370513+0000","last_undegraded":"2024-09-19T07:48:58.370513+0000","last_fullsized":"2024-09-19T07:48:58.370513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:19.033192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":30970,"num_write_kb":370397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"64'4684","reported_seq":4789,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364347+0000","last_change":"2024-09-19T07:19:40.390205+0000","last_active":"2024-09-19T07:48:58.364347+0000","last_peered":"2024-09-19T07:48:58.364347+0000","last_clean":"2024-09-19T07:48:58.364347+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:48:58.364347+0000","last_undegraded":"2024-09-19T07:48:58.364347+0000","last_fullsized":"2024-09-19T07:48:58.364347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25566,"num_write_kb":331994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"61'4261","reported_seq":4374,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381790+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:48:58.381790+0000","last_peered":"2024-09-19T07:48:58.381790+0000","last_clean":"2024-09-19T07:48:58.381790+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:48:58.381790+0000","last_undegraded":"2024-09-19T07:48:58.381790+0000","last_fullsized":"2024-09-19T07:48:58.381790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22573,"num_write_kb":274878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"64'5240","reported_seq":5347,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363651+0000","last_change":"2024-09-19T07:19:40.390559+0000","last_active":"2024-09-19T07:48:58.363651+0000","last_peered":"2024-09-19T07:48:58.363651+0000","last_clean":"2024-09-19T07:48:58.363651+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:48:58.363651+0000","last_undegraded":"2024-09-19T07:48:58.363651+0000","last_fullsized":"2024-09-19T07:48:58.363651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":28154,"num_write_kb":354961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"61'5095","reported_seq":5208,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379268+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:48:58.379268+0000","last_peered":"2024-09-19T07:48:58.379268+0000","last_clean":"2024-09-19T07:48:58.379268+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:48:58.379268+0000","last_undegraded":"2024-09-19T07:48:58.379268+0000","last_fullsized":"2024-09-19T07:48:58.379268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26748,"num_write_kb":324403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"64'4078","reported_seq":4184,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363850+0000","last_change":"2024-09-19T07:19:40.358433+0000","last_active":"2024-09-19T07:48:58.363850+0000","last_peered":"2024-09-19T07:48:58.363850+0000","last_clean":"2024-09-19T07:48:58.363850+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:48:58.363850+0000","last_undegraded":"2024-09-19T07:48:58.363850+0000","last_fullsized":"2024-09-19T07:48:58.363850+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:10:25.226677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22110,"num_write_kb":293452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"61'4908","reported_seq":5021,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378171+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:48:58.378171+0000","last_peered":"2024-09-19T07:48:58.378171+0000","last_clean":"2024-09-19T07:48:58.378171+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:48:58.378171+0000","last_undegraded":"2024-09-19T07:48:58.378171+0000","last_fullsized":"2024-09-19T07:48:58.378171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:51.710997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25574,"num_write_kb":315713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"64'3983","reported_seq":4088,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368849+0000","last_change":"2024-09-19T07:19:40.374374+0000","last_active":"2024-09-19T07:48:58.368849+0000","last_peered":"2024-09-19T07:48:58.368849+0000","last_clean":"2024-09-19T07:48:58.368849+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:48:58.368849+0000","last_undegraded":"2024-09-19T07:48:58.368849+0000","last_fullsized":"2024-09-19T07:48:58.368849+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:34:05.875388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21503,"num_write_kb":274721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"61'5975","reported_seq":6087,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371924+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:48:58.371924+0000","last_peered":"2024-09-19T07:48:58.371924+0000","last_clean":"2024-09-19T07:48:58.371924+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:48:58.371924+0000","last_undegraded":"2024-09-19T07:48:58.371924+0000","last_fullsized":"2024-09-19T07:48:58.371924+0000","mapping_epoch":19,"log_start":"53'200","ondisk_log_start":"53'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31472,"num_write_kb":384213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"64'4930","reported_seq":5034,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373470+0000","last_change":"2024-09-19T07:19:40.387341+0000","last_active":"2024-09-19T07:48:58.373470+0000","last_peered":"2024-09-19T07:48:58.373470+0000","last_clean":"2024-09-19T07:48:58.373470+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:48:58.373470+0000","last_undegraded":"2024-09-19T07:48:58.373470+0000","last_fullsized":"2024-09-19T07:48:58.373470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:56:40.928844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26807,"num_write_kb":336825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"61'5543","reported_seq":5655,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365417+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:48:58.365417+0000","last_peered":"2024-09-19T07:48:58.365417+0000","last_clean":"2024-09-19T07:48:58.365417+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:48:58.365417+0000","last_undegraded":"2024-09-19T07:48:58.365417+0000","last_fullsized":"2024-09-19T07:48:58.365417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5543,"log_dups_size":0,"ondisk_log_size":5543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29751,"num_write_kb":336461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"64'5766","reported_seq":5872,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369027+0000","last_change":"2024-09-19T07:19:40.388484+0000","last_active":"2024-09-19T07:48:58.369027+0000","last_peered":"2024-09-19T07:48:58.369027+0000","last_clean":"2024-09-19T07:48:58.369027+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:48:58.369027+0000","last_undegraded":"2024-09-19T07:48:58.369027+0000","last_fullsized":"2024-09-19T07:48:58.369027+0000","mapping_epoch":23,"log_start":"58'300","ondisk_log_start":"58'300","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5466,"log_dups_size":0,"ondisk_log_size":5466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32274,"num_write_kb":395900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"61'4235","reported_seq":4347,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372319+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:48:58.372319+0000","last_peered":"2024-09-19T07:48:58.372319+0000","last_clean":"2024-09-19T07:48:58.372319+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:48:58.372319+0000","last_undegraded":"2024-09-19T07:48:58.372319+0000","last_fullsized":"2024-09-19T07:48:58.372319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22758,"num_write_kb":275574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"64'3417","reported_seq":3521,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365853+0000","last_change":"2024-09-19T07:19:40.367578+0000","last_active":"2024-09-19T07:48:58.365853+0000","last_peered":"2024-09-19T07:48:58.365853+0000","last_clean":"2024-09-19T07:48:58.365853+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:48:58.365853+0000","last_undegraded":"2024-09-19T07:48:58.365853+0000","last_fullsized":"2024-09-19T07:48:58.365853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:00:25.357854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18157,"num_write_kb":232641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"61'4616","reported_seq":4728,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368993+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:48:58.368993+0000","last_peered":"2024-09-19T07:48:58.368993+0000","last_clean":"2024-09-19T07:48:58.368993+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:48:58.368993+0000","last_undegraded":"2024-09-19T07:48:58.368993+0000","last_fullsized":"2024-09-19T07:48:58.368993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:43.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24409,"num_write_kb":298784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"64'4320","reported_seq":4424,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374872+0000","last_change":"2024-09-19T07:19:40.385641+0000","last_active":"2024-09-19T07:48:58.374872+0000","last_peered":"2024-09-19T07:48:58.374872+0000","last_clean":"2024-09-19T07:48:58.374872+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:48:58.374872+0000","last_undegraded":"2024-09-19T07:48:58.374872+0000","last_fullsized":"2024-09-19T07:48:58.374872+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22755,"num_write_kb":287920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"61'4140","reported_seq":4253,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363901+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:48:58.363901+0000","last_peered":"2024-09-19T07:48:58.363901+0000","last_clean":"2024-09-19T07:48:58.363901+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:48:58.363901+0000","last_undegraded":"2024-09-19T07:48:58.363901+0000","last_fullsized":"2024-09-19T07:48:58.363901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22395,"num_write_kb":253944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"64'3972","reported_seq":4076,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364583+0000","last_change":"2024-09-19T07:19:40.367101+0000","last_active":"2024-09-19T07:48:58.364583+0000","last_peered":"2024-09-19T07:48:58.364583+0000","last_clean":"2024-09-19T07:48:58.364583+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:48:58.364583+0000","last_undegraded":"2024-09-19T07:48:58.364583+0000","last_fullsized":"2024-09-19T07:48:58.364583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21876,"num_write_kb":260064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"61'3526","reported_seq":3638,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368962+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:48:58.368962+0000","last_peered":"2024-09-19T07:48:58.368962+0000","last_clean":"2024-09-19T07:48:58.368962+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:48:58.368962+0000","last_undegraded":"2024-09-19T07:48:58.368962+0000","last_fullsized":"2024-09-19T07:48:58.368962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18762,"num_write_kb":235013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"64'3609","reported_seq":3713,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377616+0000","last_change":"2024-09-19T07:19:40.384212+0000","last_active":"2024-09-19T07:48:58.377616+0000","last_peered":"2024-09-19T07:48:58.377616+0000","last_clean":"2024-09-19T07:48:58.377616+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:48:58.377616+0000","last_undegraded":"2024-09-19T07:48:58.377616+0000","last_fullsized":"2024-09-19T07:48:58.377616+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:57.724404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20332,"num_write_kb":258088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"61'4660","reported_seq":4777,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381227+0000","last_change":"2024-09-19T07:20:13.952931+0000","last_active":"2024-09-19T07:48:58.381227+0000","last_peered":"2024-09-19T07:48:58.381227+0000","last_clean":"2024-09-19T07:48:58.381227+0000","last_became_active":"2024-09-19T07:20:13.952103+0000","last_became_peered":"2024-09-19T07:20:13.952103+0000","last_unstale":"2024-09-19T07:48:58.381227+0000","last_undegraded":"2024-09-19T07:48:58.381227+0000","last_fullsized":"2024-09-19T07:48:58.381227+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:23.862116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24375,"num_write_kb":290368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"64'3991","reported_seq":4095,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375705+0000","last_change":"2024-09-19T07:19:40.387401+0000","last_active":"2024-09-19T07:48:58.375705+0000","last_peered":"2024-09-19T07:48:58.375705+0000","last_clean":"2024-09-19T07:48:58.375705+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:48:58.375705+0000","last_undegraded":"2024-09-19T07:48:58.375705+0000","last_fullsized":"2024-09-19T07:48:58.375705+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3991,"log_dups_size":0,"ondisk_log_size":3991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21657,"num_write_kb":287665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"61'5448","reported_seq":5560,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381563+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:48:58.381563+0000","last_peered":"2024-09-19T07:48:58.381563+0000","last_clean":"2024-09-19T07:48:58.381563+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:48:58.381563+0000","last_undegraded":"2024-09-19T07:48:58.381563+0000","last_fullsized":"2024-09-19T07:48:58.381563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29399,"num_write_kb":347457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"64'4539","reported_seq":4643,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365336+0000","last_change":"2024-09-19T07:19:40.361430+0000","last_active":"2024-09-19T07:48:58.365336+0000","last_peered":"2024-09-19T07:48:58.365336+0000","last_clean":"2024-09-19T07:48:58.365336+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:48:58.365336+0000","last_undegraded":"2024-09-19T07:48:58.365336+0000","last_fullsized":"2024-09-19T07:48:58.365336+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4539,"log_dups_size":0,"ondisk_log_size":4539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:23:05.185943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24944,"num_write_kb":313178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"61'6280","reported_seq":6394,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.376418+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:48:58.376418+0000","last_peered":"2024-09-19T07:48:58.376418+0000","last_clean":"2024-09-19T07:48:58.376418+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:48:58.376418+0000","last_undegraded":"2024-09-19T07:48:58.376418+0000","last_fullsized":"2024-09-19T07:48:58.376418+0000","mapping_epoch":19,"log_start":"54'500","ondisk_log_start":"54'500","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":34141,"num_write_kb":395176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"64'5507","reported_seq":5613,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363380+0000","last_change":"2024-09-19T07:19:40.378172+0000","last_active":"2024-09-19T07:48:58.363380+0000","last_peered":"2024-09-19T07:48:58.363380+0000","last_clean":"2024-09-19T07:48:58.363380+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:48:58.363380+0000","last_undegraded":"2024-09-19T07:48:58.363380+0000","last_fullsized":"2024-09-19T07:48:58.363380+0000","mapping_epoch":23,"log_start":"58'300","ondisk_log_start":"58'300","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:52.321790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30086,"num_write_kb":370527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"61'4537","reported_seq":4648,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362562+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:48:58.362562+0000","last_peered":"2024-09-19T07:48:58.362562+0000","last_clean":"2024-09-19T07:48:58.362562+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:48:58.362562+0000","last_undegraded":"2024-09-19T07:48:58.362562+0000","last_fullsized":"2024-09-19T07:48:58.362562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24234,"num_write_kb":310307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"64'4532","reported_seq":4639,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381234+0000","last_change":"2024-09-19T07:19:40.373909+0000","last_active":"2024-09-19T07:48:58.381234+0000","last_peered":"2024-09-19T07:48:58.381234+0000","last_clean":"2024-09-19T07:48:58.381234+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:48:58.381234+0000","last_undegraded":"2024-09-19T07:48:58.381234+0000","last_fullsized":"2024-09-19T07:48:58.381234+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25325,"num_write_kb":306661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"61'4832","reported_seq":4943,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373789+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:48:58.373789+0000","last_peered":"2024-09-19T07:48:58.373789+0000","last_clean":"2024-09-19T07:48:58.373789+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:48:58.373789+0000","last_undegraded":"2024-09-19T07:48:58.373789+0000","last_fullsized":"2024-09-19T07:48:58.373789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26222,"num_write_kb":306099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"64'3448","reported_seq":3555,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371996+0000","last_change":"2024-09-19T07:19:40.396146+0000","last_active":"2024-09-19T07:48:58.371996+0000","last_peered":"2024-09-19T07:48:58.371996+0000","last_clean":"2024-09-19T07:48:58.371996+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:48:58.371996+0000","last_undegraded":"2024-09-19T07:48:58.371996+0000","last_fullsized":"2024-09-19T07:48:58.371996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3448,"log_dups_size":0,"ondisk_log_size":3448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18660,"num_write_kb":230569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"61'4317","reported_seq":4428,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365728+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:48:58.365728+0000","last_peered":"2024-09-19T07:48:58.365728+0000","last_clean":"2024-09-19T07:48:58.365728+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:48:58.365728+0000","last_undegraded":"2024-09-19T07:48:58.365728+0000","last_fullsized":"2024-09-19T07:48:58.365728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22609,"num_write_kb":267014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"64'3912","reported_seq":4019,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367632+0000","last_change":"2024-09-19T07:19:40.373552+0000","last_active":"2024-09-19T07:48:58.367632+0000","last_peered":"2024-09-19T07:48:58.367632+0000","last_clean":"2024-09-19T07:48:58.367632+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:48:58.367632+0000","last_undegraded":"2024-09-19T07:48:58.367632+0000","last_fullsized":"2024-09-19T07:48:58.367632+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:23:33.629831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21554,"num_write_kb":270790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"61'4720","reported_seq":4831,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373254+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:48:58.373254+0000","last_peered":"2024-09-19T07:48:58.373254+0000","last_clean":"2024-09-19T07:48:58.373254+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:48:58.373254+0000","last_undegraded":"2024-09-19T07:48:58.373254+0000","last_fullsized":"2024-09-19T07:48:58.373254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24228,"num_write_kb":289919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"64'3227","reported_seq":3330,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362718+0000","last_change":"2024-09-19T07:19:40.356989+0000","last_active":"2024-09-19T07:48:58.362718+0000","last_peered":"2024-09-19T07:48:58.362718+0000","last_clean":"2024-09-19T07:48:58.362718+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:48:58.362718+0000","last_undegraded":"2024-09-19T07:48:58.362718+0000","last_fullsized":"2024-09-19T07:48:58.362718+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17253,"num_write_kb":237102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"61'4326","reported_seq":4441,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366880+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:48:58.366880+0000","last_peered":"2024-09-19T07:48:58.366880+0000","last_clean":"2024-09-19T07:48:58.366880+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:48:58.366880+0000","last_undegraded":"2024-09-19T07:48:58.366880+0000","last_fullsized":"2024-09-19T07:48:58.366880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23399,"num_write_kb":274225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"64'3524","reported_seq":3627,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377515+0000","last_change":"2024-09-19T07:19:40.376487+0000","last_active":"2024-09-19T07:48:58.377515+0000","last_peered":"2024-09-19T07:48:58.377515+0000","last_clean":"2024-09-19T07:48:58.377515+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:48:58.377515+0000","last_undegraded":"2024-09-19T07:48:58.377515+0000","last_fullsized":"2024-09-19T07:48:58.377515+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:05.185451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18216,"num_write_kb":261619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"61'4656","reported_seq":4771,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369817+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:48:58.369817+0000","last_peered":"2024-09-19T07:48:58.369817+0000","last_clean":"2024-09-19T07:48:58.369817+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:48:58.369817+0000","last_undegraded":"2024-09-19T07:48:58.369817+0000","last_fullsized":"2024-09-19T07:48:58.369817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:25.667168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24790,"num_write_kb":287946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"64'4503","reported_seq":4606,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372440+0000","last_change":"2024-09-19T07:19:40.377604+0000","last_active":"2024-09-19T07:48:58.372440+0000","last_peered":"2024-09-19T07:48:58.372440+0000","last_clean":"2024-09-19T07:48:58.372440+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:48:58.372440+0000","last_undegraded":"2024-09-19T07:48:58.372440+0000","last_fullsized":"2024-09-19T07:48:58.372440+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24213,"num_write_kb":310059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"61'5949","reported_seq":6063,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373642+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:48:58.373642+0000","last_peered":"2024-09-19T07:48:58.373642+0000","last_clean":"2024-09-19T07:48:58.373642+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:48:58.373642+0000","last_undegraded":"2024-09-19T07:48:58.373642+0000","last_fullsized":"2024-09-19T07:48:58.373642+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5849,"log_dups_size":0,"ondisk_log_size":5849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31740,"num_write_kb":364971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"64'4360","reported_seq":4463,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372581+0000","last_change":"2024-09-19T07:19:40.377580+0000","last_active":"2024-09-19T07:48:58.372581+0000","last_peered":"2024-09-19T07:48:58.372581+0000","last_clean":"2024-09-19T07:48:58.372581+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:48:58.372581+0000","last_undegraded":"2024-09-19T07:48:58.372581+0000","last_fullsized":"2024-09-19T07:48:58.372581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4360,"log_dups_size":0,"ondisk_log_size":4360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23533,"num_write_kb":311499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"61'4830","reported_seq":4945,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375882+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:48:58.375882+0000","last_peered":"2024-09-19T07:48:58.375882+0000","last_clean":"2024-09-19T07:48:58.375882+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:48:58.375882+0000","last_undegraded":"2024-09-19T07:48:58.375882+0000","last_fullsized":"2024-09-19T07:48:58.375882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26589,"num_write_kb":315697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1521393,"num_write_kb":19214666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279640,"ondisk_log_size":279640,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19627747,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1449,"num_read_kb":190539,"num_write":5111,"num_write_kb":21283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39313408,"data_stored":39255494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4608,"ondisk_log_size":4608,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":0,"num_write":1651322,"num_write_kb":19736802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307839,"ondisk_log_size":307839,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":19579163,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1454,"num_read_kb":221531,"num_write":5068,"num_write_kb":21268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39223296,"data_stored":39158326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4624,"ondisk_log_size":4624,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542506,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1114512,"kb_used_data":13696,"kb_used_omap":1,"kb_used_meta":1100798,"kb_avail":93257328,"statfs":{"total":96636764160,"available":95495503872,"internally_reserved":0,"allocated":14024704,"data_stored":13620504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1127217610},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542507,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766708,"kb_used_data":11684,"kb_used_omap":1,"kb_used_meta":755006,"kb_avail":93605132,"statfs":{"total":96636764160,"available":95851655168,"internally_reserved":0,"allocated":11964416,"data_stored":11552279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":773126598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542505,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1069796,"kb_used_data":5012,"kb_used_omap":1,"kb_used_meta":1064766,"kb_avail":93302044,"statfs":{"total":96636764160,"available":95541293056,"internally_reserved":0,"allocated":5132288,"data_stored":4726264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1090320837},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542507,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1229016,"kb_used_data":14472,"kb_used_omap":1,"kb_used_meta":1214526,"kb_avail":93142824,"statfs":{"total":96636764160,"available":95378251776,"internally_reserved":0,"allocated":14819328,"data_stored":14400282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1243675077},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542505,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1080696,"kb_used_data":20008,"kb_used_omap":1,"kb_used_meta":1060670,"kb_avail":93291144,"statfs":{"total":96636764160,"available":95530131456,"internally_reserved":0,"allocated":20488192,"data_stored":20074195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1086126535},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542507,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688028,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":687102,"kb_avail":93683812,"statfs":{"total":96636764160,"available":95932223488,"internally_reserved":0,"allocated":929792,"data_stored":531268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":703592902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542506,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":891928,"kb_used_data":9032,"kb_used_omap":1,"kb_used_meta":882878,"kb_avail":93479912,"statfs":{"total":96636764160,"available":95723429888,"internally_reserved":0,"allocated":9248768,"data_stored":8836014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904067525},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542505,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":907476,"kb_used_data":9604,"kb_used_omap":1,"kb_used_meta":897854,"kb_avail":93464364,"statfs":{"total":96636764160,"available":95707508736,"internally_reserved":0,"allocated":9834496,"data_stored":9426166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919402949},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":4194942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":84646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8482816,"data_stored":8473926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6926336,"data_stored":6910261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6922240,"data_stored":6910807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11165696,"data_stored":11153717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7053312,"data_stored":7043469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4279622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:49:17.186 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:49:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.312+0000 7fa14daa7640 1 Processor -- start 2024-09-19T07:49:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.312+0000 7fa14daa7640 1 -- start start 2024-09-19T07:49:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa14daa7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa148071100 0x7fa1481545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa14daa7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1480a9e60 con 0x7fa14806dea0 2024-09-19T07:49:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa14daa7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1480aa140 con 0x7fa1480c28e0 2024-09-19T07:49:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa14daa7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa148071b50 con 0x7fa148071100 2024-09-19T07:49:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa147fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa14806dea0 0x7fa14806e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60302/0 (socket says 172.21.15.83:60302) 2024-09-19T07:49:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa147fff640 1 -- 172.21.15.83:0/2967711769 learned_addr learned my addr 172.21.15.83:0/2967711769 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1477fe640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa148071100 0x7fa1481545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036625301 0 0) 0x7fa1480a9e60 con 0x7fa14806dea0 2024-09-19T07:49:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa134003530 con 0x7fa14806dea0 2024-09-19T07:49:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1315511281 0 0) 0x7fa134003530 con 0x7fa14806dea0 2024-09-19T07:49:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa148071100 msgr2=0x7fa1481545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa148071100 0x7fa1481545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 >> v1:172.21.15.184:6789/0 conn(0x7fa1480c28e0 legacy=0x7fa148150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1480c2240 con 0x7fa14806dea0 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fa138001680 con 0x7fa14806dea0 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 >> v1:172.21.15.83:6789/0 conn(0x7fa14806dea0 legacy=0x7fa14806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa148071100 0x7fa1340039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa134003f70 0x7fa134004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa134004d50 0x7fa1340051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa148071b50 con 0x7fa134003f70 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa13400ff70 con 0x7fa134004d50 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa134010170 con 0x7fa148071100 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa146ffd640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa134003f70 0x7fa134004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa1477fe640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa148071100 0x7fa1340039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa147fff640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa134004d50 0x7fa1340051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa146ffd640 1 -- 172.21.15.83:0/2967711769 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa148071100 msgr2=0x7fa1340039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa146ffd640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa148071100 0x7fa1340039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa146ffd640 1 -- 172.21.15.83:0/2967711769 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa134004d50 msgr2=0x7fa1340051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa146ffd640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa134004d50 0x7fa1340051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa146ffd640 1 -- 172.21.15.83:0/2967711769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa134003530 con 0x7fa134003f70 2024-09-19T07:49:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.316+0000 7fa147fff640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa134004d50 0x7fa1340051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa146ffd640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa134003f70 0x7fa134004810 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fa130001750 tx=0x7fa13001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=676e6d0967b91396 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300013a0 con 0x7fa134003f70 2024-09-19T07:49:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1300052f0 con 0x7fa134003f70 2024-09-19T07:49:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa1467fc640 1 -- 172.21.15.83:0/2967711769 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa130003840 con 0x7fa134003f70 2024-09-19T07:49:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 -- 172.21.15.83:0/2967711769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa134003f70 msgr2=0x7fa134004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 --2- 172.21.15.83:0/2967711769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa134003f70 0x7fa134004810 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fa130001750 tx=0x7fa13001c3e0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa147fff640 1 -- 172.21.15.83:0/2967711769 reap_dead start 2024-09-19T07:49:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 -- 172.21.15.83:0/2967711769 shutdown_connections 2024-09-19T07:49:17.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 -- 172.21.15.83:0/2967711769 >> 172.21.15.83:0/2967711769 conn(0x7fa14806bc80 msgr2=0x7fa14806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:17.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 -- 172.21.15.83:0/2967711769 shutdown_connections 2024-09-19T07:49:17.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 -- 172.21.15.83:0/2967711769 wait complete. 2024-09-19T07:49:17.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 Processor -- start 2024-09-19T07:49:17.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 -- start start 2024-09-19T07:49:17.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa134003f70 0x7fa1481df740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1481dfc80 0x7fa1481478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1480c1a90 0x7fa148147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa1481517d0 con 0x7fa1481dfc80 2024-09-19T07:49:17.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa148151650 con 0x7fa1480c1a90 2024-09-19T07:49:17.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa14daa7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa1480714e0 con 0x7fa134003f70 2024-09-19T07:49:17.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1481dfc80 0x7fa1481478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1481dfc80 0x7fa1481478d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41988/0 (socket says 172.21.15.83:41988) 2024-09-19T07:49:17.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa146ffd640 1 -- 172.21.15.83:0/231927743 learned_addr learned my addr 172.21.15.83:0/231927743 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:17.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa147fff640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1480c1a90 0x7fa148147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa1477fe640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa134003f70 0x7fa1481df740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa146ffd640 1 -- 172.21.15.83:0/231927743 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa134003f70 msgr2=0x7fa1481df740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa146ffd640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa134003f70 0x7fa1481df740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa146ffd640 1 -- 172.21.15.83:0/231927743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1480c1a90 msgr2=0x7fa148147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa146ffd640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1480c1a90 0x7fa148147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa146ffd640 1 -- 172.21.15.83:0/231927743 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1480c5ad0 con 0x7fa1481dfc80 2024-09-19T07:49:17.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa146ffd640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1481dfc80 0x7fa1481478d0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fa130005040 tx=0x7fa134005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:17.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa13ffff640 1 -- 172.21.15.83:0/231927743 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300b9020 con 0x7fa1481dfc80 2024-09-19T07:49:17.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa13ffff640 1 -- 172.21.15.83:0/231927743 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1300054a0 con 0x7fa1481dfc80 2024-09-19T07:49:17.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.320+0000 7fa13ffff640 1 -- 172.21.15.83:0/231927743 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa130007a40 con 0x7fa1481dfc80 2024-09-19T07:49:17.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.324+0000 7fa14daa7640 1 -- 172.21.15.83:0/231927743 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1481e0620 con 0x7fa1481dfc80 2024-09-19T07:49:17.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.324+0000 7fa14daa7640 1 -- 172.21.15.83:0/231927743 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1481e0140 con 0x7fa1481dfc80 2024-09-19T07:49:17.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.324+0000 7fa14daa7640 1 -- 172.21.15.83:0/231927743 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa148146620 con 0x7fa1481dfc80 2024-09-19T07:49:17.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.324+0000 7fa13ffff640 1 -- 172.21.15.83:0/231927743 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa1300bc020 con 0x7fa1481dfc80 2024-09-19T07:49:17.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.324+0000 7fa13ffff640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1280678c0 0x7fa128069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.324+0000 7fa13ffff640 1 -- 172.21.15.83:0/231927743 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fa130120df0 con 0x7fa1481dfc80 2024-09-19T07:49:17.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.324+0000 7fa1477fe640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1280678c0 0x7fa128069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.324+0000 7fa1477fe640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1280678c0 0x7fa128069d80 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa140007520 tx=0x7fa14000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:17.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.328+0000 7fa13ffff640 1 -- 172.21.15.83:0/231927743 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa148146620 con 0x7fa1481dfc80 2024-09-19T07:49:17.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.488+0000 7fa14daa7640 1 -- 172.21.15.83:0/231927743 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa14806e350 con 0x7fa1280678c0 2024-09-19T07:49:17.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.520+0000 7fa13ffff640 1 -- 172.21.15.83:0/231927743 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493875 (secure 0 0 0) 0x7fa14806e350 con 0x7fa1280678c0 2024-09-19T07:49:17.526 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:17.534 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:49:17.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.524+0000 7fa14daa7640 1 -- 172.21.15.83:0/231927743 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1280678c0 msgr2=0x7fa128069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.524+0000 7fa14daa7640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1280678c0 0x7fa128069d80 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa140007520 tx=0x7fa14000b040 comp rx=0 tx=0).stop 2024-09-19T07:49:17.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.524+0000 7fa14daa7640 1 -- 172.21.15.83:0/231927743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1481dfc80 msgr2=0x7fa1481478d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.524+0000 7fa14daa7640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1481dfc80 0x7fa1481478d0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fa130005040 tx=0x7fa134005840 comp rx=0 tx=0).stop 2024-09-19T07:49:17.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.528+0000 7fa14daa7640 1 -- 172.21.15.83:0/231927743 shutdown_connections 2024-09-19T07:49:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.528+0000 7fa14daa7640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa1280678c0 0x7fa128069d80 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.528+0000 7fa14daa7640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1480c1a90 0x7fa148147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.528+0000 7fa14daa7640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1481dfc80 0x7fa1481478d0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.528+0000 7fa14daa7640 1 --2- 172.21.15.83:0/231927743 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa134003f70 0x7fa1481df740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.528+0000 7fa14daa7640 1 -- 172.21.15.83:0/231927743 >> 172.21.15.83:0/231927743 conn(0x7fa14806bc80 msgr2=0x7fa148151f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.528+0000 7fa14daa7640 1 -- 172.21.15.83:0/231927743 shutdown_connections 2024-09-19T07:49:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.528+0000 7fa14daa7640 1 -- 172.21.15.83:0/231927743 wait complete. 2024-09-19T07:49:17.546 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":955,"stamp":"2024-09-19T07:49:15.612510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39797278,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3062,"num_read_kb":412170,"num_write":3183053,"num_write_kb":38996435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":596828,"ondisk_log_size":596828,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7748160,"kb_used_data":84416,"kb_used_omap":12,"kb_used_meta":7663603,"kb_avail":747226560,"statfs":{"total":773094113280,"available":765159997440,"internally_reserved":0,"allocated":86441984,"data_stored":83166972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":7847530033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26509,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":48,"num_write":32,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005583"},"pg_stats":[{"pgid":"5.18","version":"64'4995","reported_seq":5099,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363394+0000","last_change":"2024-09-19T07:19:40.358457+0000","last_active":"2024-09-19T07:48:58.363394+0000","last_peered":"2024-09-19T07:48:58.363394+0000","last_clean":"2024-09-19T07:48:58.363394+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:48:58.363394+0000","last_undegraded":"2024-09-19T07:48:58.363394+0000","last_fullsized":"2024-09-19T07:48:58.363394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4995,"log_dups_size":0,"ondisk_log_size":4995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27070,"num_write_kb":342022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"61'953","reported_seq":1118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.393736+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:48:58.393736+0000","last_peered":"2024-09-19T07:48:58.393736+0000","last_clean":"2024-09-19T07:48:58.393736+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:48:58.393736+0000","last_undegraded":"2024-09-19T07:48:58.393736+0000","last_fullsized":"2024-09-19T07:48:58.393736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:57.799442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":18964,"num_write":931,"num_write_kb":4658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"61'4894","reported_seq":5008,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364803+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:48:58.364803+0000","last_peered":"2024-09-19T07:48:58.364803+0000","last_clean":"2024-09-19T07:48:58.364803+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:48:58.364803+0000","last_undegraded":"2024-09-19T07:48:58.364803+0000","last_fullsized":"2024-09-19T07:48:58.364803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26245,"num_write_kb":306037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"64'3","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362486+0000","last_change":"2024-09-19T07:19:39.401621+0000","last_active":"2024-09-19T07:48:58.362486+0000","last_peered":"2024-09-19T07:48:58.362486+0000","last_clean":"2024-09-19T07:48:58.362486+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:48:58.362486+0000","last_undegraded":"2024-09-19T07:48:58.362486+0000","last_fullsized":"2024-09-19T07:48:58.362486+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:32.095381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"64'5242","reported_seq":5346,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371808+0000","last_change":"2024-09-19T07:19:40.366073+0000","last_active":"2024-09-19T07:48:58.371808+0000","last_peered":"2024-09-19T07:48:58.371808+0000","last_clean":"2024-09-19T07:48:58.371808+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:48:58.371808+0000","last_undegraded":"2024-09-19T07:48:58.371808+0000","last_fullsized":"2024-09-19T07:48:58.371808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5242,"log_dups_size":0,"ondisk_log_size":5242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27567,"num_write_kb":356452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"65'25","reported_seq":141,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365251+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:48:58.365251+0000","last_peered":"2024-09-19T07:48:58.365251+0000","last_clean":"2024-09-19T07:48:58.365251+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:48:58.365251+0000","last_undegraded":"2024-09-19T07:48:58.365251+0000","last_fullsized":"2024-09-19T07:48:58.365251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"61'5997","reported_seq":6111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369375+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:48:58.369375+0000","last_peered":"2024-09-19T07:48:58.369375+0000","last_clean":"2024-09-19T07:48:58.369375+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:48:58.369375+0000","last_undegraded":"2024-09-19T07:48:58.369375+0000","last_fullsized":"2024-09-19T07:48:58.369375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:15.036396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32297,"num_write_kb":399913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"64'2","reported_seq":108,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363459+0000","last_change":"2024-09-19T07:19:39.403810+0000","last_active":"2024-09-19T07:48:58.363459+0000","last_peered":"2024-09-19T07:48:58.363459+0000","last_clean":"2024-09-19T07:48:58.363459+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:48:58.363459+0000","last_undegraded":"2024-09-19T07:48:58.363459+0000","last_fullsized":"2024-09-19T07:48:58.363459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:55.714964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"64'3801","reported_seq":3905,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373585+0000","last_change":"2024-09-19T07:19:40.390455+0000","last_active":"2024-09-19T07:48:58.373585+0000","last_peered":"2024-09-19T07:48:58.373585+0000","last_clean":"2024-09-19T07:48:58.373585+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:48:58.373585+0000","last_undegraded":"2024-09-19T07:48:58.373585+0000","last_fullsized":"2024-09-19T07:48:58.373585+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20272,"num_write_kb":245240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"65'44","reported_seq":197,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.801541+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:49:01.801541+0000","last_peered":"2024-09-19T07:49:01.801541+0000","last_clean":"2024-09-19T07:49:01.801541+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:49:01.801541+0000","last_undegraded":"2024-09-19T07:49:01.801541+0000","last_fullsized":"2024-09-19T07:49:01.801541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"61'4645","reported_seq":4758,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368108+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:48:58.368108+0000","last_peered":"2024-09-19T07:48:58.368108+0000","last_clean":"2024-09-19T07:48:58.368108+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:48:58.368108+0000","last_undegraded":"2024-09-19T07:48:58.368108+0000","last_fullsized":"2024-09-19T07:48:58.368108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24199,"num_write_kb":281402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"64'764","reported_seq":895,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369315+0000","last_change":"2024-09-19T07:19:39.408309+0000","last_active":"2024-09-19T07:48:58.369315+0000","last_peered":"2024-09-19T07:48:58.369315+0000","last_clean":"2024-09-19T07:48:58.369315+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:48:58.369315+0000","last_undegraded":"2024-09-19T07:48:58.369315+0000","last_fullsized":"2024-09-19T07:48:58.369315+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:19.396261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":71635,"num_write":751,"num_write_kb":4331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"64'3560","reported_seq":3663,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365772+0000","last_change":"2024-09-19T07:19:40.372543+0000","last_active":"2024-09-19T07:48:58.365772+0000","last_peered":"2024-09-19T07:48:58.365772+0000","last_clean":"2024-09-19T07:48:58.365772+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:48:58.365772+0000","last_undegraded":"2024-09-19T07:48:58.365772+0000","last_fullsized":"2024-09-19T07:48:58.365772+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3560,"log_dups_size":0,"ondisk_log_size":3560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:53:15.682896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18641,"num_write_kb":242587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"61'5","reported_seq":158,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.793364+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:49:01.793364+0000","last_peered":"2024-09-19T07:49:01.793364+0000","last_clean":"2024-09-19T07:49:01.793364+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:49:01.793364+0000","last_undegraded":"2024-09-19T07:49:01.793364+0000","last_fullsized":"2024-09-19T07:49:01.793364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"61'5989","reported_seq":6100,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368025+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:48:58.368025+0000","last_peered":"2024-09-19T07:48:58.368025+0000","last_clean":"2024-09-19T07:48:58.368025+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:48:58.368025+0000","last_undegraded":"2024-09-19T07:48:58.368025+0000","last_fullsized":"2024-09-19T07:48:58.368025+0000","mapping_epoch":19,"log_start":"53'200","ondisk_log_start":"53'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32376,"num_write_kb":375806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'385","reported_seq":492,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:06.042052+0000","last_change":"2024-09-19T07:19:39.407045+0000","last_active":"2024-09-19T07:49:06.042052+0000","last_peered":"2024-09-19T07:49:06.042052+0000","last_clean":"2024-09-19T07:49:06.042052+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:49:06.042052+0000","last_undegraded":"2024-09-19T07:49:06.042052+0000","last_fullsized":"2024-09-19T07:49:06.042052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:24.846974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2764415,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":637,"num_write":359,"num_write_kb":2908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"64'4768","reported_seq":4873,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372391+0000","last_change":"2024-09-19T07:19:40.396520+0000","last_active":"2024-09-19T07:48:58.372391+0000","last_peered":"2024-09-19T07:48:58.372391+0000","last_clean":"2024-09-19T07:48:58.372391+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:48:58.372391+0000","last_undegraded":"2024-09-19T07:48:58.372391+0000","last_fullsized":"2024-09-19T07:48:58.372391+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25484,"num_write_kb":302986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"61'5596","reported_seq":5709,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380964+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:48:58.380964+0000","last_peered":"2024-09-19T07:48:58.380964+0000","last_clean":"2024-09-19T07:48:58.380964+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:48:58.380964+0000","last_undegraded":"2024-09-19T07:48:58.380964+0000","last_fullsized":"2024-09-19T07:48:58.380964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30081,"num_write_kb":342050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'1456","reported_seq":1615,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.401109+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:48:58.401109+0000","last_peered":"2024-09-19T07:48:58.401109+0000","last_clean":"2024-09-19T07:48:58.401109+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:48:58.401109+0000","last_undegraded":"2024-09-19T07:48:58.401109+0000","last_fullsized":"2024-09-19T07:48:58.401109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1456,"log_dups_size":0,"ondisk_log_size":1456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":42401,"num_write":1433,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"64'6","reported_seq":133,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364879+0000","last_change":"2024-09-19T07:19:39.410500+0000","last_active":"2024-09-19T07:48:58.364879+0000","last_peered":"2024-09-19T07:48:58.364879+0000","last_clean":"2024-09-19T07:48:58.364879+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:48:58.364879+0000","last_undegraded":"2024-09-19T07:48:58.364879+0000","last_fullsized":"2024-09-19T07:48:58.364879+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:04.182606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"64'4389","reported_seq":4494,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378148+0000","last_change":"2024-09-19T07:19:40.381259+0000","last_active":"2024-09-19T07:48:58.378148+0000","last_peered":"2024-09-19T07:48:58.378148+0000","last_clean":"2024-09-19T07:48:58.378148+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:48:58.378148+0000","last_undegraded":"2024-09-19T07:48:58.378148+0000","last_fullsized":"2024-09-19T07:48:58.378148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4389,"log_dups_size":0,"ondisk_log_size":4389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:22.017682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23791,"num_write_kb":304343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"61'3782","reported_seq":3895,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364288+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:48:58.364288+0000","last_peered":"2024-09-19T07:48:58.364288+0000","last_clean":"2024-09-19T07:48:58.364288+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:48:58.364288+0000","last_undegraded":"2024-09-19T07:48:58.364288+0000","last_fullsized":"2024-09-19T07:48:58.364288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19997,"num_write_kb":252226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"61'3","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371722+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:48:58.371722+0000","last_peered":"2024-09-19T07:48:58.371722+0000","last_clean":"2024-09-19T07:48:58.371722+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:48:58.371722+0000","last_undegraded":"2024-09-19T07:48:58.371722+0000","last_fullsized":"2024-09-19T07:48:58.371722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"64'5","reported_seq":112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368513+0000","last_change":"2024-09-19T07:19:39.412123+0000","last_active":"2024-09-19T07:48:58.368513+0000","last_peered":"2024-09-19T07:48:58.368513+0000","last_clean":"2024-09-19T07:48:58.368513+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:48:58.368513+0000","last_undegraded":"2024-09-19T07:48:58.368513+0000","last_fullsized":"2024-09-19T07:48:58.368513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"64'3306","reported_seq":3411,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364068+0000","last_change":"2024-09-19T07:19:40.383214+0000","last_active":"2024-09-19T07:48:58.364068+0000","last_peered":"2024-09-19T07:48:58.364068+0000","last_clean":"2024-09-19T07:48:58.364068+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:48:58.364068+0000","last_undegraded":"2024-09-19T07:48:58.364068+0000","last_fullsized":"2024-09-19T07:48:58.364068+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17836,"num_write_kb":239496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"61'4850","reported_seq":4963,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363649+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:48:58.363649+0000","last_peered":"2024-09-19T07:48:58.363649+0000","last_clean":"2024-09-19T07:48:58.363649+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:48:58.363649+0000","last_undegraded":"2024-09-19T07:48:58.363649+0000","last_fullsized":"2024-09-19T07:48:58.363649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26728,"num_write_kb":308134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"61'2","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372455+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:48:58.372455+0000","last_peered":"2024-09-19T07:48:58.372455+0000","last_clean":"2024-09-19T07:48:58.372455+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:48:58.372455+0000","last_undegraded":"2024-09-19T07:48:58.372455+0000","last_fullsized":"2024-09-19T07:48:58.372455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"64'4","reported_seq":131,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374187+0000","last_change":"2024-09-19T07:19:39.408228+0000","last_active":"2024-09-19T07:48:58.374187+0000","last_peered":"2024-09-19T07:48:58.374187+0000","last_clean":"2024-09-19T07:48:58.374187+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:48:58.374187+0000","last_undegraded":"2024-09-19T07:48:58.374187+0000","last_fullsized":"2024-09-19T07:48:58.374187+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"64'5157","reported_seq":5262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371211+0000","last_change":"2024-09-19T07:19:40.395566+0000","last_active":"2024-09-19T07:48:58.371211+0000","last_peered":"2024-09-19T07:48:58.371211+0000","last_clean":"2024-09-19T07:48:58.371211+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:48:58.371211+0000","last_undegraded":"2024-09-19T07:48:58.371211+0000","last_fullsized":"2024-09-19T07:48:58.371211+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:54:02.766021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27729,"num_write_kb":365262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"61'4508","reported_seq":4621,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365068+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:48:58.365068+0000","last_peered":"2024-09-19T07:48:58.365068+0000","last_clean":"2024-09-19T07:48:58.365068+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:48:58.365068+0000","last_undegraded":"2024-09-19T07:48:58.365068+0000","last_fullsized":"2024-09-19T07:48:58.365068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24068,"num_write_kb":289158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"61'5","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374924+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:48:58.374924+0000","last_peered":"2024-09-19T07:48:58.374924+0000","last_clean":"2024-09-19T07:48:58.374924+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:48:58.374924+0000","last_undegraded":"2024-09-19T07:48:58.374924+0000","last_fullsized":"2024-09-19T07:48:58.374924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"64'7","reported_seq":154,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364522+0000","last_change":"2024-09-19T07:19:39.424661+0000","last_active":"2024-09-19T07:48:58.364522+0000","last_peered":"2024-09-19T07:48:58.364522+0000","last_clean":"2024-09-19T07:48:58.364522+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:48:58.364522+0000","last_undegraded":"2024-09-19T07:48:58.364522+0000","last_fullsized":"2024-09-19T07:48:58.364522+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"64'3458","reported_seq":3564,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362033+0000","last_change":"2024-09-19T07:19:40.388972+0000","last_active":"2024-09-19T07:48:58.362033+0000","last_peered":"2024-09-19T07:48:58.362033+0000","last_clean":"2024-09-19T07:48:58.362033+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:48:58.362033+0000","last_undegraded":"2024-09-19T07:48:58.362033+0000","last_fullsized":"2024-09-19T07:48:58.362033+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3458,"log_dups_size":0,"ondisk_log_size":3458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18862,"num_write_kb":239540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"61'5647","reported_seq":5761,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369864+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:48:58.369864+0000","last_peered":"2024-09-19T07:48:58.369864+0000","last_clean":"2024-09-19T07:48:58.369864+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:48:58.369864+0000","last_undegraded":"2024-09-19T07:48:58.369864+0000","last_fullsized":"2024-09-19T07:48:58.369864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:23.791880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29735,"num_write_kb":364939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"63'9","reported_seq":164,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.795335+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:49:01.795335+0000","last_peered":"2024-09-19T07:49:01.795335+0000","last_clean":"2024-09-19T07:49:01.795335+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:49:01.795335+0000","last_undegraded":"2024-09-19T07:49:01.795335+0000","last_fullsized":"2024-09-19T07:49:01.795335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'45","reported_seq":213,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:08.425477+0000","last_change":"2024-09-19T07:19:39.412192+0000","last_active":"2024-09-19T07:49:08.425477+0000","last_peered":"2024-09-19T07:49:08.425477+0000","last_clean":"2024-09-19T07:49:08.425477+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:49:08.425477+0000","last_undegraded":"2024-09-19T07:49:08.425477+0000","last_fullsized":"2024-09-19T07:49:08.425477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":108,"num_write":41,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"64'4007","reported_seq":4113,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363551+0000","last_change":"2024-09-19T07:19:40.390217+0000","last_active":"2024-09-19T07:48:58.363551+0000","last_peered":"2024-09-19T07:48:58.363551+0000","last_clean":"2024-09-19T07:48:58.363551+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:48:58.363551+0000","last_undegraded":"2024-09-19T07:48:58.363551+0000","last_fullsized":"2024-09-19T07:48:58.363551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21606,"num_write_kb":266928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"61'4152","reported_seq":4265,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364929+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:48:58.364929+0000","last_peered":"2024-09-19T07:48:58.364929+0000","last_clean":"2024-09-19T07:48:58.364929+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:48:58.364929+0000","last_undegraded":"2024-09-19T07:48:58.364929+0000","last_fullsized":"2024-09-19T07:48:58.364929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22349,"num_write_kb":269915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"65'474","reported_seq":609,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.366499+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:49:03.366499+0000","last_peered":"2024-09-19T07:49:03.366499+0000","last_clean":"2024-09-19T07:49:03.366499+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:49:03.366499+0000","last_undegraded":"2024-09-19T07:49:03.366499+0000","last_fullsized":"2024-09-19T07:49:03.366499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:50.700962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":59,"num_write":1141,"num_write_kb":380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"64'10","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362143+0000","last_change":"2024-09-19T07:19:39.434150+0000","last_active":"2024-09-19T07:48:58.362143+0000","last_peered":"2024-09-19T07:48:58.362143+0000","last_clean":"2024-09-19T07:48:58.362143+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:48:58.362143+0000","last_undegraded":"2024-09-19T07:48:58.362143+0000","last_fullsized":"2024-09-19T07:48:58.362143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:13.654208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"64'4199","reported_seq":4305,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380465+0000","last_change":"2024-09-19T07:19:40.380140+0000","last_active":"2024-09-19T07:48:58.380465+0000","last_peered":"2024-09-19T07:48:58.380465+0000","last_clean":"2024-09-19T07:48:58.380465+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:48:58.380465+0000","last_undegraded":"2024-09-19T07:48:58.380465+0000","last_fullsized":"2024-09-19T07:48:58.380465+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22884,"num_write_kb":280362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"61'5900","reported_seq":6014,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372730+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:48:58.372730+0000","last_peered":"2024-09-19T07:48:58.372730+0000","last_clean":"2024-09-19T07:48:58.372730+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:48:58.372730+0000","last_undegraded":"2024-09-19T07:48:58.372730+0000","last_fullsized":"2024-09-19T07:48:58.372730+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31464,"num_write_kb":364012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"61'2","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379634+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:48:58.379634+0000","last_peered":"2024-09-19T07:48:58.379634+0000","last_clean":"2024-09-19T07:48:58.379634+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:48:58.379634+0000","last_undegraded":"2024-09-19T07:48:58.379634+0000","last_fullsized":"2024-09-19T07:48:58.379634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"64'3","reported_seq":111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370135+0000","last_change":"2024-09-19T07:19:39.411605+0000","last_active":"2024-09-19T07:48:58.370135+0000","last_peered":"2024-09-19T07:48:58.370135+0000","last_clean":"2024-09-19T07:48:58.370135+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:48:58.370135+0000","last_undegraded":"2024-09-19T07:48:58.370135+0000","last_fullsized":"2024-09-19T07:48:58.370135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:55.654483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"64'4404","reported_seq":4510,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370924+0000","last_change":"2024-09-19T07:19:40.372083+0000","last_active":"2024-09-19T07:48:58.370924+0000","last_peered":"2024-09-19T07:48:58.370924+0000","last_clean":"2024-09-19T07:48:58.370924+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:48:58.370924+0000","last_undegraded":"2024-09-19T07:48:58.370924+0000","last_fullsized":"2024-09-19T07:48:58.370924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:39.665475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24188,"num_write_kb":314198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"61'3777","reported_seq":3891,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368386+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:48:58.368386+0000","last_peered":"2024-09-19T07:48:58.368386+0000","last_clean":"2024-09-19T07:48:58.368386+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:48:58.368386+0000","last_undegraded":"2024-09-19T07:48:58.368386+0000","last_fullsized":"2024-09-19T07:48:58.368386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20312,"num_write_kb":245406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"61'2","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372774+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:48:58.372774+0000","last_peered":"2024-09-19T07:48:58.372774+0000","last_clean":"2024-09-19T07:48:58.372774+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:48:58.372774+0000","last_undegraded":"2024-09-19T07:48:58.372774+0000","last_fullsized":"2024-09-19T07:48:58.372774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'4","reported_seq":112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371516+0000","last_change":"2024-09-19T07:19:39.420340+0000","last_active":"2024-09-19T07:48:58.371516+0000","last_peered":"2024-09-19T07:48:58.371516+0000","last_clean":"2024-09-19T07:48:58.371516+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:48:58.371516+0000","last_undegraded":"2024-09-19T07:48:58.371516+0000","last_fullsized":"2024-09-19T07:48:58.371516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:29.338848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"64'4365","reported_seq":4471,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369175+0000","last_change":"2024-09-19T07:19:40.383082+0000","last_active":"2024-09-19T07:48:58.369175+0000","last_peered":"2024-09-19T07:48:58.369175+0000","last_clean":"2024-09-19T07:48:58.369175+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:48:58.369175+0000","last_undegraded":"2024-09-19T07:48:58.369175+0000","last_fullsized":"2024-09-19T07:48:58.369175+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:54.436493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24012,"num_write_kb":288918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'5396","reported_seq":5510,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370199+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:48:58.370199+0000","last_peered":"2024-09-19T07:48:58.370199+0000","last_clean":"2024-09-19T07:48:58.370199+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:48:58.370199+0000","last_undegraded":"2024-09-19T07:48:58.370199+0000","last_fullsized":"2024-09-19T07:48:58.370199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28800,"num_write_kb":336838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'4","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368215+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:48:58.368215+0000","last_peered":"2024-09-19T07:48:58.368215+0000","last_clean":"2024-09-19T07:48:58.368215+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:48:58.368215+0000","last_undegraded":"2024-09-19T07:48:58.368215+0000","last_fullsized":"2024-09-19T07:48:58.368215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"64'2","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372603+0000","last_change":"2024-09-19T07:19:39.407135+0000","last_active":"2024-09-19T07:48:58.372603+0000","last_peered":"2024-09-19T07:48:58.372603+0000","last_clean":"2024-09-19T07:48:58.372603+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:48:58.372603+0000","last_undegraded":"2024-09-19T07:48:58.372603+0000","last_fullsized":"2024-09-19T07:48:58.372603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:27.890595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"64'4926","reported_seq":5032,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378101+0000","last_change":"2024-09-19T07:19:40.368866+0000","last_active":"2024-09-19T07:48:58.378101+0000","last_peered":"2024-09-19T07:48:58.378101+0000","last_clean":"2024-09-19T07:48:58.378101+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:48:58.378101+0000","last_undegraded":"2024-09-19T07:48:58.378101+0000","last_fullsized":"2024-09-19T07:48:58.378101+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:28.893439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26066,"num_write_kb":337871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'3534","reported_seq":3648,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371281+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:48:58.371281+0000","last_peered":"2024-09-19T07:48:58.371281+0000","last_clean":"2024-09-19T07:48:58.371281+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:48:58.371281+0000","last_undegraded":"2024-09-19T07:48:58.371281+0000","last_fullsized":"2024-09-19T07:48:58.371281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19019,"num_write_kb":227884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"65'46","reported_seq":221,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.373338+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:49:03.373338+0000","last_peered":"2024-09-19T07:49:03.373338+0000","last_clean":"2024-09-19T07:49:03.373338+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:49:03.373338+0000","last_undegraded":"2024-09-19T07:49:03.373338+0000","last_fullsized":"2024-09-19T07:49:03.373338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":109,"num_write":40,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'9","reported_seq":156,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374298+0000","last_change":"2024-09-19T07:19:39.420497+0000","last_active":"2024-09-19T07:48:58.374298+0000","last_peered":"2024-09-19T07:48:58.374298+0000","last_clean":"2024-09-19T07:48:58.374298+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:48:58.374298+0000","last_undegraded":"2024-09-19T07:48:58.374298+0000","last_fullsized":"2024-09-19T07:48:58.374298+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:19.547270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"64'3425","reported_seq":3530,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371701+0000","last_change":"2024-09-19T07:19:40.387573+0000","last_active":"2024-09-19T07:48:58.371701+0000","last_peered":"2024-09-19T07:48:58.371701+0000","last_clean":"2024-09-19T07:48:58.371701+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:48:58.371701+0000","last_undegraded":"2024-09-19T07:48:58.371701+0000","last_fullsized":"2024-09-19T07:48:58.371701+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:20:43.234793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19036,"num_write_kb":252575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"61'4467","reported_seq":4581,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368938+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:48:58.368938+0000","last_peered":"2024-09-19T07:48:58.368938+0000","last_clean":"2024-09-19T07:48:58.368938+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:48:58.368938+0000","last_undegraded":"2024-09-19T07:48:58.368938+0000","last_fullsized":"2024-09-19T07:48:58.368938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:20.824916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23642,"num_write_kb":275072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'13","reported_seq":129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375436+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:48:58.375436+0000","last_peered":"2024-09-19T07:48:58.375436+0000","last_clean":"2024-09-19T07:48:58.375436+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:48:58.375436+0000","last_undegraded":"2024-09-19T07:48:58.375436+0000","last_fullsized":"2024-09-19T07:48:58.375436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'515","reported_seq":642,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:06.371206+0000","last_change":"2024-09-19T07:19:39.410752+0000","last_active":"2024-09-19T07:49:06.371206+0000","last_peered":"2024-09-19T07:49:06.371206+0000","last_clean":"2024-09-19T07:49:06.371206+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:49:06.371206+0000","last_undegraded":"2024-09-19T07:49:06.371206+0000","last_fullsized":"2024-09-19T07:49:06.371206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:47:09.376497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":53,"num_write":1231,"num_write_kb":406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"64'4906","reported_seq":5012,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372981+0000","last_change":"2024-09-19T07:19:40.392973+0000","last_active":"2024-09-19T07:48:58.372981+0000","last_peered":"2024-09-19T07:48:58.372981+0000","last_clean":"2024-09-19T07:48:58.372981+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:48:58.372981+0000","last_undegraded":"2024-09-19T07:48:58.372981+0000","last_fullsized":"2024-09-19T07:48:58.372981+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27102,"num_write_kb":350472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"61'4338","reported_seq":4452,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363793+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:48:58.363793+0000","last_peered":"2024-09-19T07:48:58.363793+0000","last_clean":"2024-09-19T07:48:58.363793+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:48:58.363793+0000","last_undegraded":"2024-09-19T07:48:58.363793+0000","last_fullsized":"2024-09-19T07:48:58.363793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22767,"num_write_kb":271598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"61'3","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365893+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:48:58.365893+0000","last_peered":"2024-09-19T07:48:58.365893+0000","last_clean":"2024-09-19T07:48:58.365893+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:48:58.365893+0000","last_undegraded":"2024-09-19T07:48:58.365893+0000","last_fullsized":"2024-09-19T07:48:58.365893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":272,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.376591+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:48:58.376591+0000","last_peered":"2024-09-19T07:48:58.376591+0000","last_clean":"2024-09-19T07:48:58.376591+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:48:58.376591+0000","last_undegraded":"2024-09-19T07:48:58.376591+0000","last_fullsized":"2024-09-19T07:48:58.376591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'2","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380743+0000","last_change":"2024-09-19T07:19:39.429085+0000","last_active":"2024-09-19T07:48:58.380743+0000","last_peered":"2024-09-19T07:48:58.380743+0000","last_clean":"2024-09-19T07:48:58.380743+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:48:58.380743+0000","last_undegraded":"2024-09-19T07:48:58.380743+0000","last_fullsized":"2024-09-19T07:48:58.380743+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"64'3599","reported_seq":3704,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378157+0000","last_change":"2024-09-19T07:19:40.382352+0000","last_active":"2024-09-19T07:48:58.378157+0000","last_peered":"2024-09-19T07:48:58.378157+0000","last_clean":"2024-09-19T07:48:58.378157+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:48:58.378157+0000","last_undegraded":"2024-09-19T07:48:58.378157+0000","last_fullsized":"2024-09-19T07:48:58.378157+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:31.024388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19570,"num_write_kb":246554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"61'5018","reported_seq":5131,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379025+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:48:58.379025+0000","last_peered":"2024-09-19T07:48:58.379025+0000","last_clean":"2024-09-19T07:48:58.379025+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:48:58.379025+0000","last_undegraded":"2024-09-19T07:48:58.379025+0000","last_fullsized":"2024-09-19T07:48:58.379025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26753,"num_write_kb":316367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"61'5","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378898+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:48:58.378898+0000","last_peered":"2024-09-19T07:48:58.378898+0000","last_clean":"2024-09-19T07:48:58.378898+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:48:58.378898+0000","last_undegraded":"2024-09-19T07:48:58.378898+0000","last_fullsized":"2024-09-19T07:48:58.378898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"64'3","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380502+0000","last_change":"2024-09-19T07:19:39.427644+0000","last_active":"2024-09-19T07:48:58.380502+0000","last_peered":"2024-09-19T07:48:58.380502+0000","last_clean":"2024-09-19T07:48:58.380502+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:48:58.380502+0000","last_undegraded":"2024-09-19T07:48:58.380502+0000","last_fullsized":"2024-09-19T07:48:58.380502+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"64'4954","reported_seq":5059,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377576+0000","last_change":"2024-09-19T07:19:40.390380+0000","last_active":"2024-09-19T07:48:58.377576+0000","last_peered":"2024-09-19T07:48:58.377576+0000","last_clean":"2024-09-19T07:48:58.377576+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:48:58.377576+0000","last_undegraded":"2024-09-19T07:48:58.377576+0000","last_fullsized":"2024-09-19T07:48:58.377576+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:26.385490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27293,"num_write_kb":322594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"61'5220","reported_seq":5318,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377825+0000","last_change":"2024-09-19T07:20:13.952747+0000","last_active":"2024-09-19T07:48:58.377825+0000","last_peered":"2024-09-19T07:48:58.377825+0000","last_clean":"2024-09-19T07:48:58.377825+0000","last_became_active":"2024-09-19T07:20:13.951449+0000","last_became_peered":"2024-09-19T07:20:13.951449+0000","last_unstale":"2024-09-19T07:48:58.377825+0000","last_undegraded":"2024-09-19T07:48:58.377825+0000","last_fullsized":"2024-09-19T07:48:58.377825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:56.805700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27694,"num_write_kb":340542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"61'6","reported_seq":141,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374333+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:48:58.374333+0000","last_peered":"2024-09-19T07:48:58.374333+0000","last_clean":"2024-09-19T07:48:58.374333+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:48:58.374333+0000","last_undegraded":"2024-09-19T07:48:58.374333+0000","last_fullsized":"2024-09-19T07:48:58.374333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"64'1403","reported_seq":1553,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369177+0000","last_change":"2024-09-19T07:19:39.410638+0000","last_active":"2024-09-19T07:48:58.369177+0000","last_peered":"2024-09-19T07:48:58.369177+0000","last_clean":"2024-09-19T07:48:58.369177+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:48:58.369177+0000","last_undegraded":"2024-09-19T07:48:58.369177+0000","last_fullsized":"2024-09-19T07:48:58.369177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1403,"log_dups_size":0,"ondisk_log_size":1403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:32.073136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":63,"num_read_kb":30148,"num_write":1379,"num_write_kb":4466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"64'3869","reported_seq":3974,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372595+0000","last_change":"2024-09-19T07:19:40.378481+0000","last_active":"2024-09-19T07:48:58.372595+0000","last_peered":"2024-09-19T07:48:58.372595+0000","last_clean":"2024-09-19T07:48:58.372595+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:48:58.372595+0000","last_undegraded":"2024-09-19T07:48:58.372595+0000","last_fullsized":"2024-09-19T07:48:58.372595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3869,"log_dups_size":0,"ondisk_log_size":3869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:30.585686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20920,"num_write_kb":264190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"61'6113","reported_seq":6211,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373547+0000","last_change":"2024-09-19T07:20:14.209767+0000","last_active":"2024-09-19T07:48:58.373547+0000","last_peered":"2024-09-19T07:48:58.373547+0000","last_clean":"2024-09-19T07:48:58.373547+0000","last_became_active":"2024-09-19T07:20:14.209141+0000","last_became_peered":"2024-09-19T07:20:14.209141+0000","last_unstale":"2024-09-19T07:48:58.373547+0000","last_undegraded":"2024-09-19T07:48:58.373547+0000","last_fullsized":"2024-09-19T07:48:58.373547+0000","mapping_epoch":26,"log_start":"54'300","ondisk_log_start":"54'300","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:34:10.251087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32041,"num_write_kb":407182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"61'7","reported_seq":162,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.794320+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:49:01.794320+0000","last_peered":"2024-09-19T07:49:01.794320+0000","last_clean":"2024-09-19T07:49:01.794320+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:49:01.794320+0000","last_undegraded":"2024-09-19T07:49:01.794320+0000","last_fullsized":"2024-09-19T07:49:01.794320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"64'5","reported_seq":112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365126+0000","last_change":"2024-09-19T07:19:39.437656+0000","last_active":"2024-09-19T07:48:58.365126+0000","last_peered":"2024-09-19T07:48:58.365126+0000","last_clean":"2024-09-19T07:48:58.365126+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:48:58.365126+0000","last_undegraded":"2024-09-19T07:48:58.365126+0000","last_fullsized":"2024-09-19T07:48:58.365126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:41.468096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"64'4634","reported_seq":4739,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377494+0000","last_change":"2024-09-19T07:19:40.379289+0000","last_active":"2024-09-19T07:48:58.377494+0000","last_peered":"2024-09-19T07:48:58.377494+0000","last_clean":"2024-09-19T07:48:58.377494+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:48:58.377494+0000","last_undegraded":"2024-09-19T07:48:58.377494+0000","last_fullsized":"2024-09-19T07:48:58.377494+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:20.399106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25462,"num_write_kb":298798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"61'4007","reported_seq":4120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375692+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:48:58.375692+0000","last_peered":"2024-09-19T07:48:58.375692+0000","last_clean":"2024-09-19T07:48:58.375692+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:48:58.375692+0000","last_undegraded":"2024-09-19T07:48:58.375692+0000","last_fullsized":"2024-09-19T07:48:58.375692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21937,"num_write_kb":251948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"61'4","reported_seq":139,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369549+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:48:58.369549+0000","last_peered":"2024-09-19T07:48:58.369549+0000","last_clean":"2024-09-19T07:48:58.369549+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:48:58.369549+0000","last_undegraded":"2024-09-19T07:48:58.369549+0000","last_fullsized":"2024-09-19T07:48:58.369549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"64'2","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363818+0000","last_change":"2024-09-19T07:19:39.413589+0000","last_active":"2024-09-19T07:48:58.363818+0000","last_peered":"2024-09-19T07:48:58.363818+0000","last_clean":"2024-09-19T07:48:58.363818+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:48:58.363818+0000","last_undegraded":"2024-09-19T07:48:58.363818+0000","last_fullsized":"2024-09-19T07:48:58.363818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"64'3939","reported_seq":4043,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364105+0000","last_change":"2024-09-19T07:19:40.387159+0000","last_active":"2024-09-19T07:48:58.364105+0000","last_peered":"2024-09-19T07:48:58.364105+0000","last_clean":"2024-09-19T07:48:58.364105+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:48:58.364105+0000","last_undegraded":"2024-09-19T07:48:58.364105+0000","last_fullsized":"2024-09-19T07:48:58.364105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20741,"num_write_kb":275861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'4660","reported_seq":4774,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373923+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:48:58.373923+0000","last_peered":"2024-09-19T07:48:58.373923+0000","last_clean":"2024-09-19T07:48:58.373923+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:48:58.373923+0000","last_undegraded":"2024-09-19T07:48:58.373923+0000","last_fullsized":"2024-09-19T07:48:58.373923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25264,"num_write_kb":302007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"65'15","reported_seq":213,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.799096+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:49:01.799096+0000","last_peered":"2024-09-19T07:49:01.799096+0000","last_clean":"2024-09-19T07:49:01.799096+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:49:01.799096+0000","last_undegraded":"2024-09-19T07:49:01.799096+0000","last_fullsized":"2024-09-19T07:49:01.799096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"64'3","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369815+0000","last_change":"2024-09-19T07:19:39.406110+0000","last_active":"2024-09-19T07:48:58.369815+0000","last_peered":"2024-09-19T07:48:58.369815+0000","last_clean":"2024-09-19T07:48:58.369815+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:48:58.369815+0000","last_undegraded":"2024-09-19T07:48:58.369815+0000","last_fullsized":"2024-09-19T07:48:58.369815+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:10:05.224225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"64'4590","reported_seq":4694,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366527+0000","last_change":"2024-09-19T07:19:40.365764+0000","last_active":"2024-09-19T07:48:58.366527+0000","last_peered":"2024-09-19T07:48:58.366527+0000","last_clean":"2024-09-19T07:48:58.366527+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:48:58.366527+0000","last_undegraded":"2024-09-19T07:48:58.366527+0000","last_fullsized":"2024-09-19T07:48:58.366527+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:04:03.242292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24790,"num_write_kb":304361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"61'3873","reported_seq":3985,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380282+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:48:58.380282+0000","last_peered":"2024-09-19T07:48:58.380282+0000","last_clean":"2024-09-19T07:48:58.380282+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:48:58.380282+0000","last_undegraded":"2024-09-19T07:48:58.380282+0000","last_fullsized":"2024-09-19T07:48:58.380282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20508,"num_write_kb":250225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"61'5","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381953+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:48:58.381953+0000","last_peered":"2024-09-19T07:48:58.381953+0000","last_clean":"2024-09-19T07:48:58.381953+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:48:58.381953+0000","last_undegraded":"2024-09-19T07:48:58.381953+0000","last_fullsized":"2024-09-19T07:48:58.381953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"64'4","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364247+0000","last_change":"2024-09-19T07:19:39.429768+0000","last_active":"2024-09-19T07:48:58.364247+0000","last_peered":"2024-09-19T07:48:58.364247+0000","last_clean":"2024-09-19T07:48:58.364247+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:48:58.364247+0000","last_undegraded":"2024-09-19T07:48:58.364247+0000","last_fullsized":"2024-09-19T07:48:58.364247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:47.503631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"64'3485","reported_seq":3589,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366523+0000","last_change":"2024-09-19T07:19:40.373542+0000","last_active":"2024-09-19T07:48:58.366523+0000","last_peered":"2024-09-19T07:48:58.366523+0000","last_clean":"2024-09-19T07:48:58.366523+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:48:58.366523+0000","last_undegraded":"2024-09-19T07:48:58.366523+0000","last_fullsized":"2024-09-19T07:48:58.366523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:14.326662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19669,"num_write_kb":242834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"61'5231","reported_seq":5343,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373420+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:48:58.373420+0000","last_peered":"2024-09-19T07:48:58.373420+0000","last_clean":"2024-09-19T07:48:58.373420+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:48:58.373420+0000","last_undegraded":"2024-09-19T07:48:58.373420+0000","last_fullsized":"2024-09-19T07:48:58.373420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28071,"num_write_kb":333869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"61'3","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367894+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:48:58.367894+0000","last_peered":"2024-09-19T07:48:58.367894+0000","last_clean":"2024-09-19T07:48:58.367894+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:48:58.367894+0000","last_undegraded":"2024-09-19T07:48:58.367894+0000","last_fullsized":"2024-09-19T07:48:58.367894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"64'328","reported_seq":535,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363630+0000","last_change":"2024-09-19T07:19:39.411074+0000","last_active":"2024-09-19T07:48:58.363630+0000","last_peered":"2024-09-19T07:48:58.363630+0000","last_clean":"2024-09-19T07:48:58.363630+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:48:58.363630+0000","last_undegraded":"2024-09-19T07:48:58.363630+0000","last_fullsized":"2024-09-19T07:48:58.363630+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:53.992787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":181,"num_read_kb":77990,"num_write":325,"num_write_kb":4282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"64'5112","reported_seq":5216,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363063+0000","last_change":"2024-09-19T07:19:40.364814+0000","last_active":"2024-09-19T07:48:58.363063+0000","last_peered":"2024-09-19T07:48:58.363063+0000","last_clean":"2024-09-19T07:48:58.363063+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:48:58.363063+0000","last_undegraded":"2024-09-19T07:48:58.363063+0000","last_fullsized":"2024-09-19T07:48:58.363063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5112,"log_dups_size":0,"ondisk_log_size":5112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27577,"num_write_kb":360586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"61'4085","reported_seq":4197,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369111+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:48:58.369111+0000","last_peered":"2024-09-19T07:48:58.369111+0000","last_clean":"2024-09-19T07:48:58.369111+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:48:58.369111+0000","last_undegraded":"2024-09-19T07:48:58.369111+0000","last_fullsized":"2024-09-19T07:48:58.369111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:57.131912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22122,"num_write_kb":273218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"65'10","reported_seq":164,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.897451+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:48:58.897451+0000","last_peered":"2024-09-19T07:48:58.897451+0000","last_clean":"2024-09-19T07:48:58.897451+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:48:58.897451+0000","last_undegraded":"2024-09-19T07:48:58.897451+0000","last_fullsized":"2024-09-19T07:48:58.897451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:23.367365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'29","reported_seq":195,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.423376+0000","last_change":"2024-09-19T07:19:39.431616+0000","last_active":"2024-09-19T07:48:58.423376+0000","last_peered":"2024-09-19T07:48:58.423376+0000","last_clean":"2024-09-19T07:48:58.423376+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:48:58.423376+0000","last_undegraded":"2024-09-19T07:48:58.423376+0000","last_fullsized":"2024-09-19T07:48:58.423376+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:18.149581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"64'4543","reported_seq":4647,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363327+0000","last_change":"2024-09-19T07:19:40.381443+0000","last_active":"2024-09-19T07:48:58.363327+0000","last_peered":"2024-09-19T07:48:58.363327+0000","last_clean":"2024-09-19T07:48:58.363327+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:48:58.363327+0000","last_undegraded":"2024-09-19T07:48:58.363327+0000","last_fullsized":"2024-09-19T07:48:58.363327+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24993,"num_write_kb":317373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"61'3981","reported_seq":4093,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375912+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:48:58.375912+0000","last_peered":"2024-09-19T07:48:58.375912+0000","last_clean":"2024-09-19T07:48:58.375912+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:48:58.375912+0000","last_undegraded":"2024-09-19T07:48:58.375912+0000","last_fullsized":"2024-09-19T07:48:58.375912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3981,"log_dups_size":0,"ondisk_log_size":3981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21890,"num_write_kb":263866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"65'352","reported_seq":567,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.802657+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:49:01.802657+0000","last_peered":"2024-09-19T07:49:01.802657+0000","last_clean":"2024-09-19T07:49:01.802657+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:49:01.802657+0000","last_undegraded":"2024-09-19T07:49:01.802657+0000","last_fullsized":"2024-09-19T07:49:01.802657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":181,"num_read_kb":77990,"num_write":347,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"64'3","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371323+0000","last_change":"2024-09-19T07:19:39.423972+0000","last_active":"2024-09-19T07:48:58.371323+0000","last_peered":"2024-09-19T07:48:58.371323+0000","last_clean":"2024-09-19T07:48:58.371323+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:48:58.371323+0000","last_undegraded":"2024-09-19T07:48:58.371323+0000","last_fullsized":"2024-09-19T07:48:58.371323+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:45.387040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"64'3743","reported_seq":3847,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371573+0000","last_change":"2024-09-19T07:19:40.382092+0000","last_active":"2024-09-19T07:48:58.371573+0000","last_peered":"2024-09-19T07:48:58.371573+0000","last_clean":"2024-09-19T07:48:58.371573+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:48:58.371573+0000","last_undegraded":"2024-09-19T07:48:58.371573+0000","last_fullsized":"2024-09-19T07:48:58.371573+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20626,"num_write_kb":263086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"61'6160","reported_seq":6272,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369147+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:48:58.369147+0000","last_peered":"2024-09-19T07:48:58.369147+0000","last_clean":"2024-09-19T07:48:58.369147+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:48:58.369147+0000","last_undegraded":"2024-09-19T07:48:58.369147+0000","last_fullsized":"2024-09-19T07:48:58.369147+0000","mapping_epoch":19,"log_start":"54'700","ondisk_log_start":"54'700","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32461,"num_write_kb":376158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"65'29","reported_seq":203,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.368449+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:49:03.368449+0000","last_peered":"2024-09-19T07:49:03.368449+0000","last_clean":"2024-09-19T07:49:03.368449+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:49:03.368449+0000","last_undegraded":"2024-09-19T07:49:03.368449+0000","last_fullsized":"2024-09-19T07:49:03.368449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"64'8","reported_seq":154,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369556+0000","last_change":"2024-09-19T07:19:39.424879+0000","last_active":"2024-09-19T07:48:58.369556+0000","last_peered":"2024-09-19T07:48:58.369556+0000","last_clean":"2024-09-19T07:48:58.369556+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:48:58.369556+0000","last_undegraded":"2024-09-19T07:48:58.369556+0000","last_fullsized":"2024-09-19T07:48:58.369556+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:30.021543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"64'4564","reported_seq":4670,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382249+0000","last_change":"2024-09-19T07:19:40.384136+0000","last_active":"2024-09-19T07:48:58.382249+0000","last_peered":"2024-09-19T07:48:58.382249+0000","last_clean":"2024-09-19T07:48:58.382249+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:48:58.382249+0000","last_undegraded":"2024-09-19T07:48:58.382249+0000","last_fullsized":"2024-09-19T07:48:58.382249+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25533,"num_write_kb":303195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"61'5068","reported_seq":5180,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369896+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:48:58.369896+0000","last_peered":"2024-09-19T07:48:58.369896+0000","last_clean":"2024-09-19T07:48:58.369896+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:48:58.369896+0000","last_undegraded":"2024-09-19T07:48:58.369896+0000","last_fullsized":"2024-09-19T07:48:58.369896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:48.308137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26828,"num_write_kb":316657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"61'3","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368011+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:48:58.368011+0000","last_peered":"2024-09-19T07:48:58.368011+0000","last_clean":"2024-09-19T07:48:58.368011+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:48:58.368011+0000","last_undegraded":"2024-09-19T07:48:58.368011+0000","last_fullsized":"2024-09-19T07:48:58.368011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'15","reported_seq":205,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362420+0000","last_change":"2024-09-19T07:19:39.410445+0000","last_active":"2024-09-19T07:48:58.362420+0000","last_peered":"2024-09-19T07:48:58.362420+0000","last_clean":"2024-09-19T07:48:58.362420+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:48:58.362420+0000","last_undegraded":"2024-09-19T07:48:58.362420+0000","last_fullsized":"2024-09-19T07:48:58.362420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:27.020327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"64'3771","reported_seq":3875,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373168+0000","last_change":"2024-09-19T07:19:40.389499+0000","last_active":"2024-09-19T07:48:58.373168+0000","last_peered":"2024-09-19T07:48:58.373168+0000","last_clean":"2024-09-19T07:48:58.373168+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:48:58.373168+0000","last_undegraded":"2024-09-19T07:48:58.373168+0000","last_fullsized":"2024-09-19T07:48:58.373168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3771,"log_dups_size":0,"ondisk_log_size":3771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.520872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19869,"num_write_kb":255725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"61'4973","reported_seq":5085,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367096+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:48:58.367096+0000","last_peered":"2024-09-19T07:48:58.367096+0000","last_clean":"2024-09-19T07:48:58.367096+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:48:58.367096+0000","last_undegraded":"2024-09-19T07:48:58.367096+0000","last_fullsized":"2024-09-19T07:48:58.367096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26723,"num_write_kb":320185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"65'6","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382937+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:48:58.382937+0000","last_peered":"2024-09-19T07:48:58.382937+0000","last_clean":"2024-09-19T07:48:58.382937+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:48:58.382937+0000","last_undegraded":"2024-09-19T07:48:58.382937+0000","last_fullsized":"2024-09-19T07:48:58.382937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"64'5","reported_seq":111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362191+0000","last_change":"2024-09-19T07:19:39.416943+0000","last_active":"2024-09-19T07:48:58.362191+0000","last_peered":"2024-09-19T07:48:58.362191+0000","last_clean":"2024-09-19T07:48:58.362191+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:48:58.362191+0000","last_undegraded":"2024-09-19T07:48:58.362191+0000","last_fullsized":"2024-09-19T07:48:58.362191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:35.555826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"64'5394","reported_seq":5500,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382132+0000","last_change":"2024-09-19T07:19:40.394993+0000","last_active":"2024-09-19T07:48:58.382132+0000","last_peered":"2024-09-19T07:48:58.382132+0000","last_clean":"2024-09-19T07:48:58.382132+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:48:58.382132+0000","last_undegraded":"2024-09-19T07:48:58.382132+0000","last_fullsized":"2024-09-19T07:48:58.382132+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5394,"log_dups_size":0,"ondisk_log_size":5394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30095,"num_write_kb":374705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"64'25","reported_seq":133,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369509+0000","last_change":"2024-09-19T07:19:39.418885+0000","last_active":"2024-09-19T07:48:58.369509+0000","last_peered":"2024-09-19T07:48:58.369509+0000","last_clean":"2024-09-19T07:48:58.369509+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:48:58.369509+0000","last_undegraded":"2024-09-19T07:48:58.369509+0000","last_fullsized":"2024-09-19T07:48:58.369509+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:35:45.790411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"61'4822","reported_seq":4934,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371995+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:48:58.371995+0000","last_peered":"2024-09-19T07:48:58.371995+0000","last_clean":"2024-09-19T07:48:58.371995+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:48:58.371995+0000","last_undegraded":"2024-09-19T07:48:58.371995+0000","last_fullsized":"2024-09-19T07:48:58.371995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25208,"num_write_kb":326524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"61'2","reported_seq":116,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367640+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:48:58.367640+0000","last_peered":"2024-09-19T07:48:58.367640+0000","last_clean":"2024-09-19T07:48:58.367640+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:48:58.367640+0000","last_undegraded":"2024-09-19T07:48:58.367640+0000","last_fullsized":"2024-09-19T07:48:58.367640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"64'4690","reported_seq":4796,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369407+0000","last_change":"2024-09-19T07:19:40.374048+0000","last_active":"2024-09-19T07:48:58.369407+0000","last_peered":"2024-09-19T07:48:58.369407+0000","last_clean":"2024-09-19T07:48:58.369407+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:48:58.369407+0000","last_undegraded":"2024-09-19T07:48:58.369407+0000","last_fullsized":"2024-09-19T07:48:58.369407+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:40.084827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25924,"num_write_kb":313037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"64'964","reported_seq":1117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382679+0000","last_change":"2024-09-19T07:19:39.432426+0000","last_active":"2024-09-19T07:48:58.382679+0000","last_peered":"2024-09-19T07:48:58.382679+0000","last_clean":"2024-09-19T07:48:58.382679+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:48:58.382679+0000","last_undegraded":"2024-09-19T07:48:58.382679+0000","last_fullsized":"2024-09-19T07:48:58.382679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":964,"log_dups_size":0,"ondisk_log_size":964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":8896,"num_write":940,"num_write_kb":4641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"61'5311","reported_seq":5423,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362632+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:48:58.362632+0000","last_peered":"2024-09-19T07:48:58.362632+0000","last_clean":"2024-09-19T07:48:58.362632+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:48:58.362632+0000","last_undegraded":"2024-09-19T07:48:58.362632+0000","last_fullsized":"2024-09-19T07:48:58.362632+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28108,"num_write_kb":333978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"61'3","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363067+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:48:58.363067+0000","last_peered":"2024-09-19T07:48:58.363067+0000","last_clean":"2024-09-19T07:48:58.363067+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:48:58.363067+0000","last_undegraded":"2024-09-19T07:48:58.363067+0000","last_fullsized":"2024-09-19T07:48:58.363067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"64'5270","reported_seq":5373,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373581+0000","last_change":"2024-09-19T07:19:40.384934+0000","last_active":"2024-09-19T07:48:58.373581+0000","last_peered":"2024-09-19T07:48:58.373581+0000","last_clean":"2024-09-19T07:48:58.373581+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:48:58.373581+0000","last_undegraded":"2024-09-19T07:48:58.373581+0000","last_fullsized":"2024-09-19T07:48:58.373581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29270,"num_write_kb":363254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"65'416","reported_seq":535,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:04.513350+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:49:04.513350+0000","last_peered":"2024-09-19T07:49:04.513350+0000","last_clean":"2024-09-19T07:49:04.513350+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:49:04.513350+0000","last_undegraded":"2024-09-19T07:49:04.513350+0000","last_fullsized":"2024-09-19T07:49:04.513350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2715831,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6193,"num_write":390,"num_write_kb":2885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"61'3569","reported_seq":3680,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371694+0000","last_change":"2024-09-19T07:19:32.289374+0000","last_active":"2024-09-19T07:48:58.371694+0000","last_peered":"2024-09-19T07:48:58.371694+0000","last_clean":"2024-09-19T07:48:58.371694+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:48:58.371694+0000","last_undegraded":"2024-09-19T07:48:58.371694+0000","last_fullsized":"2024-09-19T07:48:58.371694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18120,"num_write_kb":244659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"64'5","reported_seq":150,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378720+0000","last_change":"2024-09-19T07:19:39.410582+0000","last_active":"2024-09-19T07:48:58.378720+0000","last_peered":"2024-09-19T07:48:58.378720+0000","last_clean":"2024-09-19T07:48:58.378720+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:48:58.378720+0000","last_undegraded":"2024-09-19T07:48:58.378720+0000","last_fullsized":"2024-09-19T07:48:58.378720+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:41:40.592907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"64'3415","reported_seq":3520,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364298+0000","last_change":"2024-09-19T07:19:40.393950+0000","last_active":"2024-09-19T07:48:58.364298+0000","last_peered":"2024-09-19T07:48:58.364298+0000","last_clean":"2024-09-19T07:48:58.364298+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:48:58.364298+0000","last_undegraded":"2024-09-19T07:48:58.364298+0000","last_fullsized":"2024-09-19T07:48:58.364298+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18072,"num_write_kb":240550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"61'712","reported_seq":848,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364844+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:48:58.364844+0000","last_peered":"2024-09-19T07:48:58.364844+0000","last_clean":"2024-09-19T07:48:58.364844+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:48:58.364844+0000","last_undegraded":"2024-09-19T07:48:58.364844+0000","last_fullsized":"2024-09-19T07:48:58.364844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":74743,"num_write":701,"num_write_kb":4298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"61'4395","reported_seq":4507,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368988+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:48:58.368988+0000","last_peered":"2024-09-19T07:48:58.368988+0000","last_clean":"2024-09-19T07:48:58.368988+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:48:58.368988+0000","last_undegraded":"2024-09-19T07:48:58.368988+0000","last_fullsized":"2024-09-19T07:48:58.368988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22455,"num_write_kb":278564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"64'40","reported_seq":185,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363443+0000","last_change":"2024-09-19T07:19:39.410662+0000","last_active":"2024-09-19T07:48:58.363443+0000","last_peered":"2024-09-19T07:48:58.363443+0000","last_clean":"2024-09-19T07:48:58.363443+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:48:58.363443+0000","last_undegraded":"2024-09-19T07:48:58.363443+0000","last_fullsized":"2024-09-19T07:48:58.363443+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:19.196074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"64'4798","reported_seq":4903,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364274+0000","last_change":"2024-09-19T07:19:40.385581+0000","last_active":"2024-09-19T07:48:58.364274+0000","last_peered":"2024-09-19T07:48:58.364274+0000","last_clean":"2024-09-19T07:48:58.364274+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:48:58.364274+0000","last_undegraded":"2024-09-19T07:48:58.364274+0000","last_fullsized":"2024-09-19T07:48:58.364274+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:14:13.749729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27082,"num_write_kb":321689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"61'4997","reported_seq":5111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380242+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:48:58.380242+0000","last_peered":"2024-09-19T07:48:58.380242+0000","last_clean":"2024-09-19T07:48:58.380242+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:48:58.380242+0000","last_undegraded":"2024-09-19T07:48:58.380242+0000","last_fullsized":"2024-09-19T07:48:58.380242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4997,"log_dups_size":0,"ondisk_log_size":4997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26206,"num_write_kb":309941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"64'5490","reported_seq":5596,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373817+0000","last_change":"2024-09-19T07:19:40.377608+0000","last_active":"2024-09-19T07:48:58.373817+0000","last_peered":"2024-09-19T07:48:58.373817+0000","last_clean":"2024-09-19T07:48:58.373817+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:48:58.373817+0000","last_undegraded":"2024-09-19T07:48:58.373817+0000","last_fullsized":"2024-09-19T07:48:58.373817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5490,"log_dups_size":0,"ondisk_log_size":5490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29613,"num_write_kb":364653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"61'4890","reported_seq":5005,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377509+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:48:58.377509+0000","last_peered":"2024-09-19T07:48:58.377509+0000","last_clean":"2024-09-19T07:48:58.377509+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:48:58.377509+0000","last_undegraded":"2024-09-19T07:48:58.377509+0000","last_fullsized":"2024-09-19T07:48:58.377509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4890,"log_dups_size":0,"ondisk_log_size":4890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25338,"num_write_kb":314559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"64'4569","reported_seq":4675,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366516+0000","last_change":"2024-09-19T07:19:40.381262+0000","last_active":"2024-09-19T07:48:58.366516+0000","last_peered":"2024-09-19T07:48:58.366516+0000","last_clean":"2024-09-19T07:48:58.366516+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:48:58.366516+0000","last_undegraded":"2024-09-19T07:48:58.366516+0000","last_fullsized":"2024-09-19T07:48:58.366516+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:34.425472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25275,"num_write_kb":314704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"61'5227","reported_seq":5343,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367085+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:48:58.367085+0000","last_peered":"2024-09-19T07:48:58.367085+0000","last_clean":"2024-09-19T07:48:58.367085+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:48:58.367085+0000","last_undegraded":"2024-09-19T07:48:58.367085+0000","last_fullsized":"2024-09-19T07:48:58.367085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27444,"num_write_kb":331433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"64'4939","reported_seq":5044,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373561+0000","last_change":"2024-09-19T07:19:40.386432+0000","last_active":"2024-09-19T07:48:58.373561+0000","last_peered":"2024-09-19T07:48:58.373561+0000","last_clean":"2024-09-19T07:48:58.373561+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:48:58.373561+0000","last_undegraded":"2024-09-19T07:48:58.373561+0000","last_fullsized":"2024-09-19T07:48:58.373561+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27263,"num_write_kb":334786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"61'4385","reported_seq":4500,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371314+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:48:58.371314+0000","last_peered":"2024-09-19T07:48:58.371314+0000","last_clean":"2024-09-19T07:48:58.371314+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:48:58.371314+0000","last_undegraded":"2024-09-19T07:48:58.371314+0000","last_fullsized":"2024-09-19T07:48:58.371314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22838,"num_write_kb":271812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"64'4859","reported_seq":4967,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372828+0000","last_change":"2024-09-19T07:19:40.378630+0000","last_active":"2024-09-19T07:48:58.372828+0000","last_peered":"2024-09-19T07:48:58.372828+0000","last_clean":"2024-09-19T07:48:58.372828+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:48:58.372828+0000","last_undegraded":"2024-09-19T07:48:58.372828+0000","last_fullsized":"2024-09-19T07:48:58.372828+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26164,"num_write_kb":338417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"61'4483","reported_seq":4597,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364173+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:48:58.364173+0000","last_peered":"2024-09-19T07:48:58.364173+0000","last_clean":"2024-09-19T07:48:58.364173+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:48:58.364173+0000","last_undegraded":"2024-09-19T07:48:58.364173+0000","last_fullsized":"2024-09-19T07:48:58.364173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23870,"num_write_kb":292652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"64'4105","reported_seq":4212,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368603+0000","last_change":"2024-09-19T07:19:40.381365+0000","last_active":"2024-09-19T07:48:58.368603+0000","last_peered":"2024-09-19T07:48:58.368603+0000","last_clean":"2024-09-19T07:48:58.368603+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:48:58.368603+0000","last_undegraded":"2024-09-19T07:48:58.368603+0000","last_fullsized":"2024-09-19T07:48:58.368603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21923,"num_write_kb":284540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"61'5078","reported_seq":5191,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374056+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:48:58.374056+0000","last_peered":"2024-09-19T07:48:58.374056+0000","last_clean":"2024-09-19T07:48:58.374056+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:48:58.374056+0000","last_undegraded":"2024-09-19T07:48:58.374056+0000","last_fullsized":"2024-09-19T07:48:58.374056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28059,"num_write_kb":313378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"64'3824","reported_seq":3930,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372461+0000","last_change":"2024-09-19T07:19:40.387116+0000","last_active":"2024-09-19T07:48:58.372461+0000","last_peered":"2024-09-19T07:48:58.372461+0000","last_clean":"2024-09-19T07:48:58.372461+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:48:58.372461+0000","last_undegraded":"2024-09-19T07:48:58.372461+0000","last_fullsized":"2024-09-19T07:48:58.372461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20622,"num_write_kb":295649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"61'4725","reported_seq":4838,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382564+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:48:58.382564+0000","last_peered":"2024-09-19T07:48:58.382564+0000","last_clean":"2024-09-19T07:48:58.382564+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:48:58.382564+0000","last_undegraded":"2024-09-19T07:48:58.382564+0000","last_fullsized":"2024-09-19T07:48:58.382564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25170,"num_write_kb":293477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"64'4764","reported_seq":4871,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380511+0000","last_change":"2024-09-19T07:19:40.357629+0000","last_active":"2024-09-19T07:48:58.380511+0000","last_peered":"2024-09-19T07:48:58.380511+0000","last_clean":"2024-09-19T07:48:58.380511+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:48:58.380511+0000","last_undegraded":"2024-09-19T07:48:58.380511+0000","last_fullsized":"2024-09-19T07:48:58.380511+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25208,"num_write_kb":309995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"61'5411","reported_seq":5525,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364434+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:48:58.364434+0000","last_peered":"2024-09-19T07:48:58.364434+0000","last_clean":"2024-09-19T07:48:58.364434+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:48:58.364434+0000","last_undegraded":"2024-09-19T07:48:58.364434+0000","last_fullsized":"2024-09-19T07:48:58.364434+0000","mapping_epoch":19,"log_start":"54'200","ondisk_log_start":"54'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28760,"num_write_kb":348949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"64'4221","reported_seq":4326,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363476+0000","last_change":"2024-09-19T07:19:40.388597+0000","last_active":"2024-09-19T07:48:58.363476+0000","last_peered":"2024-09-19T07:48:58.363476+0000","last_clean":"2024-09-19T07:48:58.363476+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:48:58.363476+0000","last_undegraded":"2024-09-19T07:48:58.363476+0000","last_fullsized":"2024-09-19T07:48:58.363476+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:34:18.402016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21805,"num_write_kb":288249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"61'4974","reported_seq":5093,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365664+0000","last_change":"2024-09-19T07:20:13.954029+0000","last_active":"2024-09-19T07:48:58.365664+0000","last_peered":"2024-09-19T07:48:58.365664+0000","last_clean":"2024-09-19T07:48:58.365664+0000","last_became_active":"2024-09-19T07:20:13.953118+0000","last_became_peered":"2024-09-19T07:20:13.953118+0000","last_unstale":"2024-09-19T07:48:58.365664+0000","last_undegraded":"2024-09-19T07:48:58.365664+0000","last_fullsized":"2024-09-19T07:48:58.365664+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:27:21.388787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25995,"num_write_kb":309048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"64'5715","reported_seq":5819,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375222+0000","last_change":"2024-09-19T07:19:40.383779+0000","last_active":"2024-09-19T07:48:58.375222+0000","last_peered":"2024-09-19T07:48:58.375222+0000","last_clean":"2024-09-19T07:48:58.375222+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:48:58.375222+0000","last_undegraded":"2024-09-19T07:48:58.375222+0000","last_fullsized":"2024-09-19T07:48:58.375222+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5715,"log_dups_size":0,"ondisk_log_size":5715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30021,"num_write_kb":374301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"61'4405","reported_seq":4518,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362645+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:48:58.362645+0000","last_peered":"2024-09-19T07:48:58.362645+0000","last_clean":"2024-09-19T07:48:58.362645+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:48:58.362645+0000","last_undegraded":"2024-09-19T07:48:58.362645+0000","last_fullsized":"2024-09-19T07:48:58.362645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4405,"log_dups_size":0,"ondisk_log_size":4405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23742,"num_write_kb":275461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"64'3931","reported_seq":4036,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378138+0000","last_change":"2024-09-19T07:19:40.376342+0000","last_active":"2024-09-19T07:48:58.378138+0000","last_peered":"2024-09-19T07:48:58.378138+0000","last_clean":"2024-09-19T07:48:58.378138+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:48:58.378138+0000","last_undegraded":"2024-09-19T07:48:58.378138+0000","last_fullsized":"2024-09-19T07:48:58.378138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3931,"log_dups_size":0,"ondisk_log_size":3931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:22.997744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22002,"num_write_kb":260466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"61'4975","reported_seq":5088,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365882+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:48:58.365882+0000","last_peered":"2024-09-19T07:48:58.365882+0000","last_clean":"2024-09-19T07:48:58.365882+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:48:58.365882+0000","last_undegraded":"2024-09-19T07:48:58.365882+0000","last_fullsized":"2024-09-19T07:48:58.365882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4975,"log_dups_size":0,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26500,"num_write_kb":311194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"64'5003","reported_seq":5108,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362879+0000","last_change":"2024-09-19T07:19:40.361504+0000","last_active":"2024-09-19T07:48:58.362879+0000","last_peered":"2024-09-19T07:48:58.362879+0000","last_clean":"2024-09-19T07:48:58.362879+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:48:58.362879+0000","last_undegraded":"2024-09-19T07:48:58.362879+0000","last_fullsized":"2024-09-19T07:48:58.362879+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:30:31.110669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26890,"num_write_kb":325027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"61'5877","reported_seq":5992,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370513+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:48:58.370513+0000","last_peered":"2024-09-19T07:48:58.370513+0000","last_clean":"2024-09-19T07:48:58.370513+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:48:58.370513+0000","last_undegraded":"2024-09-19T07:48:58.370513+0000","last_fullsized":"2024-09-19T07:48:58.370513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:19.033192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":30970,"num_write_kb":370397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"64'4684","reported_seq":4789,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364347+0000","last_change":"2024-09-19T07:19:40.390205+0000","last_active":"2024-09-19T07:48:58.364347+0000","last_peered":"2024-09-19T07:48:58.364347+0000","last_clean":"2024-09-19T07:48:58.364347+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:48:58.364347+0000","last_undegraded":"2024-09-19T07:48:58.364347+0000","last_fullsized":"2024-09-19T07:48:58.364347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25566,"num_write_kb":331994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"61'4261","reported_seq":4374,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381790+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:48:58.381790+0000","last_peered":"2024-09-19T07:48:58.381790+0000","last_clean":"2024-09-19T07:48:58.381790+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:48:58.381790+0000","last_undegraded":"2024-09-19T07:48:58.381790+0000","last_fullsized":"2024-09-19T07:48:58.381790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22573,"num_write_kb":274878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"64'5240","reported_seq":5347,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363651+0000","last_change":"2024-09-19T07:19:40.390559+0000","last_active":"2024-09-19T07:48:58.363651+0000","last_peered":"2024-09-19T07:48:58.363651+0000","last_clean":"2024-09-19T07:48:58.363651+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:48:58.363651+0000","last_undegraded":"2024-09-19T07:48:58.363651+0000","last_fullsized":"2024-09-19T07:48:58.363651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":28154,"num_write_kb":354961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"61'5095","reported_seq":5208,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379268+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:48:58.379268+0000","last_peered":"2024-09-19T07:48:58.379268+0000","last_clean":"2024-09-19T07:48:58.379268+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:48:58.379268+0000","last_undegraded":"2024-09-19T07:48:58.379268+0000","last_fullsized":"2024-09-19T07:48:58.379268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26748,"num_write_kb":324403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"64'4078","reported_seq":4184,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363850+0000","last_change":"2024-09-19T07:19:40.358433+0000","last_active":"2024-09-19T07:48:58.363850+0000","last_peered":"2024-09-19T07:48:58.363850+0000","last_clean":"2024-09-19T07:48:58.363850+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:48:58.363850+0000","last_undegraded":"2024-09-19T07:48:58.363850+0000","last_fullsized":"2024-09-19T07:48:58.363850+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:10:25.226677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22110,"num_write_kb":293452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"61'4908","reported_seq":5021,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378171+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:48:58.378171+0000","last_peered":"2024-09-19T07:48:58.378171+0000","last_clean":"2024-09-19T07:48:58.378171+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:48:58.378171+0000","last_undegraded":"2024-09-19T07:48:58.378171+0000","last_fullsized":"2024-09-19T07:48:58.378171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:51.710997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25574,"num_write_kb":315713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"64'3983","reported_seq":4088,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368849+0000","last_change":"2024-09-19T07:19:40.374374+0000","last_active":"2024-09-19T07:48:58.368849+0000","last_peered":"2024-09-19T07:48:58.368849+0000","last_clean":"2024-09-19T07:48:58.368849+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:48:58.368849+0000","last_undegraded":"2024-09-19T07:48:58.368849+0000","last_fullsized":"2024-09-19T07:48:58.368849+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:34:05.875388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21503,"num_write_kb":274721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"61'5975","reported_seq":6087,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371924+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:48:58.371924+0000","last_peered":"2024-09-19T07:48:58.371924+0000","last_clean":"2024-09-19T07:48:58.371924+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:48:58.371924+0000","last_undegraded":"2024-09-19T07:48:58.371924+0000","last_fullsized":"2024-09-19T07:48:58.371924+0000","mapping_epoch":19,"log_start":"53'200","ondisk_log_start":"53'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31472,"num_write_kb":384213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"64'4930","reported_seq":5034,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373470+0000","last_change":"2024-09-19T07:19:40.387341+0000","last_active":"2024-09-19T07:48:58.373470+0000","last_peered":"2024-09-19T07:48:58.373470+0000","last_clean":"2024-09-19T07:48:58.373470+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:48:58.373470+0000","last_undegraded":"2024-09-19T07:48:58.373470+0000","last_fullsized":"2024-09-19T07:48:58.373470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:56:40.928844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26807,"num_write_kb":336825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"61'5543","reported_seq":5655,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365417+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:48:58.365417+0000","last_peered":"2024-09-19T07:48:58.365417+0000","last_clean":"2024-09-19T07:48:58.365417+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:48:58.365417+0000","last_undegraded":"2024-09-19T07:48:58.365417+0000","last_fullsized":"2024-09-19T07:48:58.365417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5543,"log_dups_size":0,"ondisk_log_size":5543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29751,"num_write_kb":336461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"64'5766","reported_seq":5872,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369027+0000","last_change":"2024-09-19T07:19:40.388484+0000","last_active":"2024-09-19T07:48:58.369027+0000","last_peered":"2024-09-19T07:48:58.369027+0000","last_clean":"2024-09-19T07:48:58.369027+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:48:58.369027+0000","last_undegraded":"2024-09-19T07:48:58.369027+0000","last_fullsized":"2024-09-19T07:48:58.369027+0000","mapping_epoch":23,"log_start":"58'300","ondisk_log_start":"58'300","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5466,"log_dups_size":0,"ondisk_log_size":5466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32274,"num_write_kb":395900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"61'4235","reported_seq":4347,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372319+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:48:58.372319+0000","last_peered":"2024-09-19T07:48:58.372319+0000","last_clean":"2024-09-19T07:48:58.372319+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:48:58.372319+0000","last_undegraded":"2024-09-19T07:48:58.372319+0000","last_fullsized":"2024-09-19T07:48:58.372319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22758,"num_write_kb":275574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"64'3417","reported_seq":3521,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365853+0000","last_change":"2024-09-19T07:19:40.367578+0000","last_active":"2024-09-19T07:48:58.365853+0000","last_peered":"2024-09-19T07:48:58.365853+0000","last_clean":"2024-09-19T07:48:58.365853+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:48:58.365853+0000","last_undegraded":"2024-09-19T07:48:58.365853+0000","last_fullsized":"2024-09-19T07:48:58.365853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:00:25.357854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18157,"num_write_kb":232641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"61'4616","reported_seq":4728,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368993+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:48:58.368993+0000","last_peered":"2024-09-19T07:48:58.368993+0000","last_clean":"2024-09-19T07:48:58.368993+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:48:58.368993+0000","last_undegraded":"2024-09-19T07:48:58.368993+0000","last_fullsized":"2024-09-19T07:48:58.368993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:43.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24409,"num_write_kb":298784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"64'4320","reported_seq":4424,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374872+0000","last_change":"2024-09-19T07:19:40.385641+0000","last_active":"2024-09-19T07:48:58.374872+0000","last_peered":"2024-09-19T07:48:58.374872+0000","last_clean":"2024-09-19T07:48:58.374872+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:48:58.374872+0000","last_undegraded":"2024-09-19T07:48:58.374872+0000","last_fullsized":"2024-09-19T07:48:58.374872+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22755,"num_write_kb":287920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"61'4140","reported_seq":4253,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363901+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:48:58.363901+0000","last_peered":"2024-09-19T07:48:58.363901+0000","last_clean":"2024-09-19T07:48:58.363901+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:48:58.363901+0000","last_undegraded":"2024-09-19T07:48:58.363901+0000","last_fullsized":"2024-09-19T07:48:58.363901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22395,"num_write_kb":253944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"64'3972","reported_seq":4076,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364583+0000","last_change":"2024-09-19T07:19:40.367101+0000","last_active":"2024-09-19T07:48:58.364583+0000","last_peered":"2024-09-19T07:48:58.364583+0000","last_clean":"2024-09-19T07:48:58.364583+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:48:58.364583+0000","last_undegraded":"2024-09-19T07:48:58.364583+0000","last_fullsized":"2024-09-19T07:48:58.364583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21876,"num_write_kb":260064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"61'3526","reported_seq":3638,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368962+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:48:58.368962+0000","last_peered":"2024-09-19T07:48:58.368962+0000","last_clean":"2024-09-19T07:48:58.368962+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:48:58.368962+0000","last_undegraded":"2024-09-19T07:48:58.368962+0000","last_fullsized":"2024-09-19T07:48:58.368962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18762,"num_write_kb":235013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"64'3609","reported_seq":3713,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377616+0000","last_change":"2024-09-19T07:19:40.384212+0000","last_active":"2024-09-19T07:48:58.377616+0000","last_peered":"2024-09-19T07:48:58.377616+0000","last_clean":"2024-09-19T07:48:58.377616+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:48:58.377616+0000","last_undegraded":"2024-09-19T07:48:58.377616+0000","last_fullsized":"2024-09-19T07:48:58.377616+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:57.724404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20332,"num_write_kb":258088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"61'4660","reported_seq":4777,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381227+0000","last_change":"2024-09-19T07:20:13.952931+0000","last_active":"2024-09-19T07:48:58.381227+0000","last_peered":"2024-09-19T07:48:58.381227+0000","last_clean":"2024-09-19T07:48:58.381227+0000","last_became_active":"2024-09-19T07:20:13.952103+0000","last_became_peered":"2024-09-19T07:20:13.952103+0000","last_unstale":"2024-09-19T07:48:58.381227+0000","last_undegraded":"2024-09-19T07:48:58.381227+0000","last_fullsized":"2024-09-19T07:48:58.381227+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:23.862116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24375,"num_write_kb":290368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"64'3991","reported_seq":4095,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375705+0000","last_change":"2024-09-19T07:19:40.387401+0000","last_active":"2024-09-19T07:48:58.375705+0000","last_peered":"2024-09-19T07:48:58.375705+0000","last_clean":"2024-09-19T07:48:58.375705+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:48:58.375705+0000","last_undegraded":"2024-09-19T07:48:58.375705+0000","last_fullsized":"2024-09-19T07:48:58.375705+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3991,"log_dups_size":0,"ondisk_log_size":3991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21657,"num_write_kb":287665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"61'5448","reported_seq":5560,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381563+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:48:58.381563+0000","last_peered":"2024-09-19T07:48:58.381563+0000","last_clean":"2024-09-19T07:48:58.381563+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:48:58.381563+0000","last_undegraded":"2024-09-19T07:48:58.381563+0000","last_fullsized":"2024-09-19T07:48:58.381563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29399,"num_write_kb":347457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"64'4539","reported_seq":4643,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365336+0000","last_change":"2024-09-19T07:19:40.361430+0000","last_active":"2024-09-19T07:48:58.365336+0000","last_peered":"2024-09-19T07:48:58.365336+0000","last_clean":"2024-09-19T07:48:58.365336+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:48:58.365336+0000","last_undegraded":"2024-09-19T07:48:58.365336+0000","last_fullsized":"2024-09-19T07:48:58.365336+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4539,"log_dups_size":0,"ondisk_log_size":4539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:23:05.185943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24944,"num_write_kb":313178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"61'6280","reported_seq":6394,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.376418+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:48:58.376418+0000","last_peered":"2024-09-19T07:48:58.376418+0000","last_clean":"2024-09-19T07:48:58.376418+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:48:58.376418+0000","last_undegraded":"2024-09-19T07:48:58.376418+0000","last_fullsized":"2024-09-19T07:48:58.376418+0000","mapping_epoch":19,"log_start":"54'500","ondisk_log_start":"54'500","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":34141,"num_write_kb":395176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"64'5507","reported_seq":5613,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363380+0000","last_change":"2024-09-19T07:19:40.378172+0000","last_active":"2024-09-19T07:48:58.363380+0000","last_peered":"2024-09-19T07:48:58.363380+0000","last_clean":"2024-09-19T07:48:58.363380+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:48:58.363380+0000","last_undegraded":"2024-09-19T07:48:58.363380+0000","last_fullsized":"2024-09-19T07:48:58.363380+0000","mapping_epoch":23,"log_start":"58'300","ondisk_log_start":"58'300","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:52.321790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30086,"num_write_kb":370527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"61'4537","reported_seq":4648,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362562+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:48:58.362562+0000","last_peered":"2024-09-19T07:48:58.362562+0000","last_clean":"2024-09-19T07:48:58.362562+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:48:58.362562+0000","last_undegraded":"2024-09-19T07:48:58.362562+0000","last_fullsized":"2024-09-19T07:48:58.362562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24234,"num_write_kb":310307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"64'4532","reported_seq":4639,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381234+0000","last_change":"2024-09-19T07:19:40.373909+0000","last_active":"2024-09-19T07:48:58.381234+0000","last_peered":"2024-09-19T07:48:58.381234+0000","last_clean":"2024-09-19T07:48:58.381234+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:48:58.381234+0000","last_undegraded":"2024-09-19T07:48:58.381234+0000","last_fullsized":"2024-09-19T07:48:58.381234+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25325,"num_write_kb":306661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"61'4832","reported_seq":4943,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373789+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:48:58.373789+0000","last_peered":"2024-09-19T07:48:58.373789+0000","last_clean":"2024-09-19T07:48:58.373789+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:48:58.373789+0000","last_undegraded":"2024-09-19T07:48:58.373789+0000","last_fullsized":"2024-09-19T07:48:58.373789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26222,"num_write_kb":306099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"64'3448","reported_seq":3555,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371996+0000","last_change":"2024-09-19T07:19:40.396146+0000","last_active":"2024-09-19T07:48:58.371996+0000","last_peered":"2024-09-19T07:48:58.371996+0000","last_clean":"2024-09-19T07:48:58.371996+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:48:58.371996+0000","last_undegraded":"2024-09-19T07:48:58.371996+0000","last_fullsized":"2024-09-19T07:48:58.371996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3448,"log_dups_size":0,"ondisk_log_size":3448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18660,"num_write_kb":230569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"61'4317","reported_seq":4428,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365728+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:48:58.365728+0000","last_peered":"2024-09-19T07:48:58.365728+0000","last_clean":"2024-09-19T07:48:58.365728+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:48:58.365728+0000","last_undegraded":"2024-09-19T07:48:58.365728+0000","last_fullsized":"2024-09-19T07:48:58.365728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22609,"num_write_kb":267014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"64'3912","reported_seq":4019,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367632+0000","last_change":"2024-09-19T07:19:40.373552+0000","last_active":"2024-09-19T07:48:58.367632+0000","last_peered":"2024-09-19T07:48:58.367632+0000","last_clean":"2024-09-19T07:48:58.367632+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:48:58.367632+0000","last_undegraded":"2024-09-19T07:48:58.367632+0000","last_fullsized":"2024-09-19T07:48:58.367632+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:23:33.629831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21554,"num_write_kb":270790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"61'4720","reported_seq":4831,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373254+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:48:58.373254+0000","last_peered":"2024-09-19T07:48:58.373254+0000","last_clean":"2024-09-19T07:48:58.373254+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:48:58.373254+0000","last_undegraded":"2024-09-19T07:48:58.373254+0000","last_fullsized":"2024-09-19T07:48:58.373254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24228,"num_write_kb":289919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"64'3227","reported_seq":3330,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362718+0000","last_change":"2024-09-19T07:19:40.356989+0000","last_active":"2024-09-19T07:48:58.362718+0000","last_peered":"2024-09-19T07:48:58.362718+0000","last_clean":"2024-09-19T07:48:58.362718+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:48:58.362718+0000","last_undegraded":"2024-09-19T07:48:58.362718+0000","last_fullsized":"2024-09-19T07:48:58.362718+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17253,"num_write_kb":237102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"61'4326","reported_seq":4441,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366880+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:48:58.366880+0000","last_peered":"2024-09-19T07:48:58.366880+0000","last_clean":"2024-09-19T07:48:58.366880+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:48:58.366880+0000","last_undegraded":"2024-09-19T07:48:58.366880+0000","last_fullsized":"2024-09-19T07:48:58.366880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23399,"num_write_kb":274225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"64'3524","reported_seq":3627,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377515+0000","last_change":"2024-09-19T07:19:40.376487+0000","last_active":"2024-09-19T07:48:58.377515+0000","last_peered":"2024-09-19T07:48:58.377515+0000","last_clean":"2024-09-19T07:48:58.377515+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:48:58.377515+0000","last_undegraded":"2024-09-19T07:48:58.377515+0000","last_fullsized":"2024-09-19T07:48:58.377515+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:05.185451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18216,"num_write_kb":261619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"61'4656","reported_seq":4771,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369817+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:48:58.369817+0000","last_peered":"2024-09-19T07:48:58.369817+0000","last_clean":"2024-09-19T07:48:58.369817+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:48:58.369817+0000","last_undegraded":"2024-09-19T07:48:58.369817+0000","last_fullsized":"2024-09-19T07:48:58.369817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:25.667168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24790,"num_write_kb":287946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"64'4503","reported_seq":4606,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372440+0000","last_change":"2024-09-19T07:19:40.377604+0000","last_active":"2024-09-19T07:48:58.372440+0000","last_peered":"2024-09-19T07:48:58.372440+0000","last_clean":"2024-09-19T07:48:58.372440+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:48:58.372440+0000","last_undegraded":"2024-09-19T07:48:58.372440+0000","last_fullsized":"2024-09-19T07:48:58.372440+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24213,"num_write_kb":310059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"61'5949","reported_seq":6063,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373642+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:48:58.373642+0000","last_peered":"2024-09-19T07:48:58.373642+0000","last_clean":"2024-09-19T07:48:58.373642+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:48:58.373642+0000","last_undegraded":"2024-09-19T07:48:58.373642+0000","last_fullsized":"2024-09-19T07:48:58.373642+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5849,"log_dups_size":0,"ondisk_log_size":5849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31740,"num_write_kb":364971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"64'4360","reported_seq":4463,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372581+0000","last_change":"2024-09-19T07:19:40.377580+0000","last_active":"2024-09-19T07:48:58.372581+0000","last_peered":"2024-09-19T07:48:58.372581+0000","last_clean":"2024-09-19T07:48:58.372581+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:48:58.372581+0000","last_undegraded":"2024-09-19T07:48:58.372581+0000","last_fullsized":"2024-09-19T07:48:58.372581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4360,"log_dups_size":0,"ondisk_log_size":4360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23533,"num_write_kb":311499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"61'4830","reported_seq":4945,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375882+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:48:58.375882+0000","last_peered":"2024-09-19T07:48:58.375882+0000","last_clean":"2024-09-19T07:48:58.375882+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:48:58.375882+0000","last_undegraded":"2024-09-19T07:48:58.375882+0000","last_fullsized":"2024-09-19T07:48:58.375882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26589,"num_write_kb":315697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1521393,"num_write_kb":19214666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279640,"ondisk_log_size":279640,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19627747,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1449,"num_read_kb":190539,"num_write":5111,"num_write_kb":21283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39313408,"data_stored":39255494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4608,"ondisk_log_size":4608,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":0,"num_write":1651322,"num_write_kb":19736802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307839,"ondisk_log_size":307839,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":19579163,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1454,"num_read_kb":221531,"num_write":5068,"num_write_kb":21268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39223296,"data_stored":39158326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4624,"ondisk_log_size":4624,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542506,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1114512,"kb_used_data":13696,"kb_used_omap":1,"kb_used_meta":1100798,"kb_avail":93257328,"statfs":{"total":96636764160,"available":95495503872,"internally_reserved":0,"allocated":14024704,"data_stored":13620504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1127217610},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542507,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766708,"kb_used_data":11684,"kb_used_omap":1,"kb_used_meta":755006,"kb_avail":93605132,"statfs":{"total":96636764160,"available":95851655168,"internally_reserved":0,"allocated":11964416,"data_stored":11552279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":773126598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542505,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1069796,"kb_used_data":5012,"kb_used_omap":1,"kb_used_meta":1064766,"kb_avail":93302044,"statfs":{"total":96636764160,"available":95541293056,"internally_reserved":0,"allocated":5132288,"data_stored":4726264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1090320837},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542507,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1229016,"kb_used_data":14472,"kb_used_omap":1,"kb_used_meta":1214526,"kb_avail":93142824,"statfs":{"total":96636764160,"available":95378251776,"internally_reserved":0,"allocated":14819328,"data_stored":14400282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1243675077},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542505,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1080696,"kb_used_data":20008,"kb_used_omap":1,"kb_used_meta":1060670,"kb_avail":93291144,"statfs":{"total":96636764160,"available":95530131456,"internally_reserved":0,"allocated":20488192,"data_stored":20074195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1086126535},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542507,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688028,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":687102,"kb_avail":93683812,"statfs":{"total":96636764160,"available":95932223488,"internally_reserved":0,"allocated":929792,"data_stored":531268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":703592902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542506,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":891928,"kb_used_data":9032,"kb_used_omap":1,"kb_used_meta":882878,"kb_avail":93479912,"statfs":{"total":96636764160,"available":95723429888,"internally_reserved":0,"allocated":9248768,"data_stored":8836014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904067525},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542505,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":907476,"kb_used_data":9604,"kb_used_omap":1,"kb_used_meta":897854,"kb_avail":93464364,"statfs":{"total":96636764160,"available":95707508736,"internally_reserved":0,"allocated":9834496,"data_stored":9426166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919402949},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":4194942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":84646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8482816,"data_stored":8473926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6926336,"data_stored":6910261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6922240,"data_stored":6910807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11165696,"data_stored":11153717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7053312,"data_stored":7043469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4279622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:49:17.555 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-19T07:49:17.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:49:17.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13b2ff3640 1 Processor -- start 2024-09-19T07:49:17.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13b2ff3640 1 -- start start 2024-09-19T07:49:17.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13b2ff3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f13ac0c6120 0x7f13ac0c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13b2ff3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13ac156200 con 0x7f13ac1592f0 2024-09-19T07:49:17.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13b2ff3640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13ac1564e0 con 0x7f13ac155d70 2024-09-19T07:49:17.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13b2ff3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f13ac0c9930 con 0x7f13ac0c6120 2024-09-19T07:49:17.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13b0d8f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f13ac0c6120 0x7f13ac0c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13b1590640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f13ac1592f0 0x7f13ac157ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60304/0 (socket says 172.21.15.83:60304) 2024-09-19T07:49:17.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13b1590640 1 -- 172.21.15.83:0/3440411968 learned_addr learned my addr 172.21.15.83:0/3440411968 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:17.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13b0d8f640 1 -- 172.21.15.83:0/3440411968 >> v1:172.21.15.184:6789/0 conn(0x7f13ac155d70 legacy=0x7f13ac1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:17.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13b0d8f640 1 -- 172.21.15.83:0/3440411968 >> v1:172.21.15.83:6789/0 conn(0x7f13ac1592f0 legacy=0x7f13ac157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13ab7fe640 1 -- 172.21.15.83:0/3440411968 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 977969178 0 0) 0x7f13ac156200 con 0x7f13ac1592f0 2024-09-19T07:49:17.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.716+0000 7f13b0d8f640 1 -- 172.21.15.83:0/3440411968 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ac1567c0 con 0x7f13ac0c6120 2024-09-19T07:49:17.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b0d8f640 1 --2- 172.21.15.83:0/3440411968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13ac0c6120 0x7f13ac0c6500 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f139c0097c0 tx=0x7f139c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f357ccfe455ec62 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:17.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13ab7fe640 1 -- 172.21.15.83:0/3440411968 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c0b0070 con 0x7f13ac0c6120 2024-09-19T07:49:17.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13ab7fe640 1 -- 172.21.15.83:0/3440411968 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f139c0ab600 con 0x7f13ac0c6120 2024-09-19T07:49:17.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13ab7fe640 1 -- 172.21.15.83:0/3440411968 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c0a4e60 con 0x7f13ac0c6120 2024-09-19T07:49:17.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 -- 172.21.15.83:0/3440411968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13ac0c6120 msgr2=0x7f13ac0c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 --2- 172.21.15.83:0/3440411968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13ac0c6120 0x7f13ac0c6500 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f139c0097c0 tx=0x7f139c01c3d0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 -- 172.21.15.83:0/3440411968 shutdown_connections 2024-09-19T07:49:17.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 --2- 172.21.15.83:0/3440411968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13ac0c6120 0x7f13ac0c6500 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 -- 172.21.15.83:0/3440411968 >> 172.21.15.83:0/3440411968 conn(0x7f13ac06bc80 msgr2=0x7f13ac06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:17.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 -- 172.21.15.83:0/3440411968 shutdown_connections 2024-09-19T07:49:17.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 -- 172.21.15.83:0/3440411968 wait complete. 2024-09-19T07:49:17.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 Processor -- start 2024-09-19T07:49:17.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 -- start start 2024-09-19T07:49:17.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ac0c6120 0x7f13ac1efbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13ac1592f0 0x7f13ac0c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ac0c1d00 0x7f13ac0ad260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac164030 con 0x7f13ac1592f0 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac163eb0 con 0x7f13ac0c6120 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b2ff3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f13ac1641b0 con 0x7f13ac0c1d00 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13abfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13ac1592f0 0x7f13ac0c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13abfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13ac1592f0 0x7f13ac0c17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42004/0 (socket says 172.21.15.83:42004) 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13abfff640 1 -- 172.21.15.83:0/113983496 learned_addr learned my addr 172.21.15.83:0/113983496 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b0d8f640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ac0c6120 0x7f13ac1efbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13b1590640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ac0c1d00 0x7f13ac0ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13abfff640 1 -- 172.21.15.83:0/113983496 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ac0c1d00 msgr2=0x7f13ac0ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13abfff640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ac0c1d00 0x7f13ac0ad260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13abfff640 1 -- 172.21.15.83:0/113983496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ac0c6120 msgr2=0x7f13ac1efbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13abfff640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ac0c6120 0x7f13ac1efbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13abfff640 1 -- 172.21.15.83:0/113983496 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ac0ab590 con 0x7f13ac1592f0 2024-09-19T07:49:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13abfff640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13ac1592f0 0x7f13ac0c17c0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f13a000c450 tx=0x7f13a000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:17.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.720+0000 7f13a97fa640 1 -- 172.21.15.83:0/113983496 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0011790 con 0x7f13ac1592f0 2024-09-19T07:49:17.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.724+0000 7f13a97fa640 1 -- 172.21.15.83:0/113983496 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a0017070 con 0x7f13ac1592f0 2024-09-19T07:49:17.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.724+0000 7f13b2ff3640 1 -- 172.21.15.83:0/113983496 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ac0ab870 con 0x7f13ac1592f0 2024-09-19T07:49:17.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.724+0000 7f13a97fa640 1 -- 172.21.15.83:0/113983496 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0011b90 con 0x7f13ac1592f0 2024-09-19T07:49:17.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.724+0000 7f13b2ff3640 1 -- 172.21.15.83:0/113983496 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ac152c40 con 0x7f13ac1592f0 2024-09-19T07:49:17.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.724+0000 7f13b2ff3640 1 -- 172.21.15.83:0/113983496 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13ac156200 con 0x7f13ac1592f0 2024-09-19T07:49:17.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.724+0000 7f13a97fa640 1 -- 172.21.15.83:0/113983496 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f13a0011d30 con 0x7f13ac1592f0 2024-09-19T07:49:17.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.724+0000 7f13a97fa640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13980678c0 0x7f1398069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:17.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.724+0000 7f13a97fa640 1 -- 172.21.15.83:0/113983496 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f13a0086c90 con 0x7f13ac1592f0 2024-09-19T07:49:17.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.724+0000 7f13b0d8f640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13980678c0 0x7f1398069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:17.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.724+0000 7f13b0d8f640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13980678c0 0x7f1398069d80 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f139c0ad000 tx=0x7f139c0a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:17.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.728+0000 7f13a97fa640 1 -- 172.21.15.83:0/113983496 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f13a0056670 con 0x7f13ac1592f0 2024-09-19T07:49:17.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.892+0000 7f13b2ff3640 1 -- 172.21.15.83:0/113983496 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f13ac0cc410 con 0x7f13980678c0 2024-09-19T07:49:17.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.924+0000 7f13a97fa640 1 -- 172.21.15.83:0/113983496 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493839 (secure 0 0 0) 0x7f13ac0cc410 con 0x7f13980678c0 2024-09-19T07:49:17.926 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:17.932 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:49:17.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.932+0000 7f13b2ff3640 1 -- 172.21.15.83:0/113983496 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13980678c0 msgr2=0x7f1398069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.932+0000 7f13b2ff3640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13980678c0 0x7f1398069d80 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f139c0ad000 tx=0x7f139c0a6000 comp rx=0 tx=0).stop 2024-09-19T07:49:17.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.932+0000 7f13b2ff3640 1 -- 172.21.15.83:0/113983496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13ac1592f0 msgr2=0x7f13ac0c17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:17.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.932+0000 7f13b2ff3640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13ac1592f0 0x7f13ac0c17c0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f13a000c450 tx=0x7f13a000c910 comp rx=0 tx=0).stop 2024-09-19T07:49:17.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.932+0000 7f13b2ff3640 1 -- 172.21.15.83:0/113983496 shutdown_connections 2024-09-19T07:49:17.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.932+0000 7f13b2ff3640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f13980678c0 0x7f1398069d80 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.932+0000 7f13b2ff3640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13ac0c1d00 0x7f13ac0ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.932+0000 7f13b2ff3640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13ac1592f0 0x7f13ac0c17c0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.932+0000 7f13b2ff3640 1 --2- 172.21.15.83:0/113983496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ac0c6120 0x7f13ac1efbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:17.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.932+0000 7f13b2ff3640 1 -- 172.21.15.83:0/113983496 >> 172.21.15.83:0/113983496 conn(0x7f13ac06bc80 msgr2=0x7f13ac160be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:17.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.932+0000 7f13b2ff3640 1 -- 172.21.15.83:0/113983496 shutdown_connections 2024-09-19T07:49:17.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:17.932+0000 7f13b2ff3640 1 -- 172.21.15.83:0/113983496 wait complete. 2024-09-19T07:49:17.950 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":956,"stamp":"2024-09-19T07:49:17.613081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39797278,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3062,"num_read_kb":412170,"num_write":3183053,"num_write_kb":38996435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":596828,"ondisk_log_size":596828,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7748160,"kb_used_data":84416,"kb_used_omap":12,"kb_used_meta":7663603,"kb_avail":747226560,"statfs":{"total":773094113280,"available":765159997440,"internally_reserved":0,"allocated":86441984,"data_stored":83166972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":7847530033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26509,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":48,"num_write":32,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005174"},"pg_stats":[{"pgid":"5.18","version":"64'4995","reported_seq":5099,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363394+0000","last_change":"2024-09-19T07:19:40.358457+0000","last_active":"2024-09-19T07:48:58.363394+0000","last_peered":"2024-09-19T07:48:58.363394+0000","last_clean":"2024-09-19T07:48:58.363394+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:48:58.363394+0000","last_undegraded":"2024-09-19T07:48:58.363394+0000","last_fullsized":"2024-09-19T07:48:58.363394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4995,"log_dups_size":0,"ondisk_log_size":4995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27070,"num_write_kb":342022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"61'953","reported_seq":1118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.393736+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:48:58.393736+0000","last_peered":"2024-09-19T07:48:58.393736+0000","last_clean":"2024-09-19T07:48:58.393736+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:48:58.393736+0000","last_undegraded":"2024-09-19T07:48:58.393736+0000","last_fullsized":"2024-09-19T07:48:58.393736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:57.799442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":18964,"num_write":931,"num_write_kb":4658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"61'4894","reported_seq":5008,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364803+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:48:58.364803+0000","last_peered":"2024-09-19T07:48:58.364803+0000","last_clean":"2024-09-19T07:48:58.364803+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:48:58.364803+0000","last_undegraded":"2024-09-19T07:48:58.364803+0000","last_fullsized":"2024-09-19T07:48:58.364803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26245,"num_write_kb":306037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"64'3","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362486+0000","last_change":"2024-09-19T07:19:39.401621+0000","last_active":"2024-09-19T07:48:58.362486+0000","last_peered":"2024-09-19T07:48:58.362486+0000","last_clean":"2024-09-19T07:48:58.362486+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:48:58.362486+0000","last_undegraded":"2024-09-19T07:48:58.362486+0000","last_fullsized":"2024-09-19T07:48:58.362486+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:32.095381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"64'5242","reported_seq":5346,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371808+0000","last_change":"2024-09-19T07:19:40.366073+0000","last_active":"2024-09-19T07:48:58.371808+0000","last_peered":"2024-09-19T07:48:58.371808+0000","last_clean":"2024-09-19T07:48:58.371808+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:48:58.371808+0000","last_undegraded":"2024-09-19T07:48:58.371808+0000","last_fullsized":"2024-09-19T07:48:58.371808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5242,"log_dups_size":0,"ondisk_log_size":5242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27567,"num_write_kb":356452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"65'25","reported_seq":141,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365251+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:48:58.365251+0000","last_peered":"2024-09-19T07:48:58.365251+0000","last_clean":"2024-09-19T07:48:58.365251+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:48:58.365251+0000","last_undegraded":"2024-09-19T07:48:58.365251+0000","last_fullsized":"2024-09-19T07:48:58.365251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"61'5997","reported_seq":6111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369375+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:48:58.369375+0000","last_peered":"2024-09-19T07:48:58.369375+0000","last_clean":"2024-09-19T07:48:58.369375+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:48:58.369375+0000","last_undegraded":"2024-09-19T07:48:58.369375+0000","last_fullsized":"2024-09-19T07:48:58.369375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:15.036396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32297,"num_write_kb":399913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"64'2","reported_seq":108,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363459+0000","last_change":"2024-09-19T07:19:39.403810+0000","last_active":"2024-09-19T07:48:58.363459+0000","last_peered":"2024-09-19T07:48:58.363459+0000","last_clean":"2024-09-19T07:48:58.363459+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:48:58.363459+0000","last_undegraded":"2024-09-19T07:48:58.363459+0000","last_fullsized":"2024-09-19T07:48:58.363459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:55.714964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"64'3801","reported_seq":3905,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373585+0000","last_change":"2024-09-19T07:19:40.390455+0000","last_active":"2024-09-19T07:48:58.373585+0000","last_peered":"2024-09-19T07:48:58.373585+0000","last_clean":"2024-09-19T07:48:58.373585+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:48:58.373585+0000","last_undegraded":"2024-09-19T07:48:58.373585+0000","last_fullsized":"2024-09-19T07:48:58.373585+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20272,"num_write_kb":245240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"65'44","reported_seq":197,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.801541+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:49:01.801541+0000","last_peered":"2024-09-19T07:49:01.801541+0000","last_clean":"2024-09-19T07:49:01.801541+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:49:01.801541+0000","last_undegraded":"2024-09-19T07:49:01.801541+0000","last_fullsized":"2024-09-19T07:49:01.801541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"61'4645","reported_seq":4758,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368108+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:48:58.368108+0000","last_peered":"2024-09-19T07:48:58.368108+0000","last_clean":"2024-09-19T07:48:58.368108+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:48:58.368108+0000","last_undegraded":"2024-09-19T07:48:58.368108+0000","last_fullsized":"2024-09-19T07:48:58.368108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24199,"num_write_kb":281402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"64'764","reported_seq":895,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369315+0000","last_change":"2024-09-19T07:19:39.408309+0000","last_active":"2024-09-19T07:48:58.369315+0000","last_peered":"2024-09-19T07:48:58.369315+0000","last_clean":"2024-09-19T07:48:58.369315+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:48:58.369315+0000","last_undegraded":"2024-09-19T07:48:58.369315+0000","last_fullsized":"2024-09-19T07:48:58.369315+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:19.396261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":71635,"num_write":751,"num_write_kb":4331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"64'3560","reported_seq":3663,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365772+0000","last_change":"2024-09-19T07:19:40.372543+0000","last_active":"2024-09-19T07:48:58.365772+0000","last_peered":"2024-09-19T07:48:58.365772+0000","last_clean":"2024-09-19T07:48:58.365772+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:48:58.365772+0000","last_undegraded":"2024-09-19T07:48:58.365772+0000","last_fullsized":"2024-09-19T07:48:58.365772+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3560,"log_dups_size":0,"ondisk_log_size":3560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:53:15.682896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18641,"num_write_kb":242587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"61'5","reported_seq":158,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.793364+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:49:01.793364+0000","last_peered":"2024-09-19T07:49:01.793364+0000","last_clean":"2024-09-19T07:49:01.793364+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:49:01.793364+0000","last_undegraded":"2024-09-19T07:49:01.793364+0000","last_fullsized":"2024-09-19T07:49:01.793364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"61'5989","reported_seq":6100,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368025+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:48:58.368025+0000","last_peered":"2024-09-19T07:48:58.368025+0000","last_clean":"2024-09-19T07:48:58.368025+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:48:58.368025+0000","last_undegraded":"2024-09-19T07:48:58.368025+0000","last_fullsized":"2024-09-19T07:48:58.368025+0000","mapping_epoch":19,"log_start":"53'200","ondisk_log_start":"53'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32376,"num_write_kb":375806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'385","reported_seq":492,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:06.042052+0000","last_change":"2024-09-19T07:19:39.407045+0000","last_active":"2024-09-19T07:49:06.042052+0000","last_peered":"2024-09-19T07:49:06.042052+0000","last_clean":"2024-09-19T07:49:06.042052+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:49:06.042052+0000","last_undegraded":"2024-09-19T07:49:06.042052+0000","last_fullsized":"2024-09-19T07:49:06.042052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:24.846974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2764415,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":637,"num_write":359,"num_write_kb":2908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"64'4768","reported_seq":4873,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372391+0000","last_change":"2024-09-19T07:19:40.396520+0000","last_active":"2024-09-19T07:48:58.372391+0000","last_peered":"2024-09-19T07:48:58.372391+0000","last_clean":"2024-09-19T07:48:58.372391+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:48:58.372391+0000","last_undegraded":"2024-09-19T07:48:58.372391+0000","last_fullsized":"2024-09-19T07:48:58.372391+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25484,"num_write_kb":302986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"61'5596","reported_seq":5709,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380964+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:48:58.380964+0000","last_peered":"2024-09-19T07:48:58.380964+0000","last_clean":"2024-09-19T07:48:58.380964+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:48:58.380964+0000","last_undegraded":"2024-09-19T07:48:58.380964+0000","last_fullsized":"2024-09-19T07:48:58.380964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30081,"num_write_kb":342050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'1456","reported_seq":1615,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.401109+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:48:58.401109+0000","last_peered":"2024-09-19T07:48:58.401109+0000","last_clean":"2024-09-19T07:48:58.401109+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:48:58.401109+0000","last_undegraded":"2024-09-19T07:48:58.401109+0000","last_fullsized":"2024-09-19T07:48:58.401109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1456,"log_dups_size":0,"ondisk_log_size":1456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":42401,"num_write":1433,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"64'6","reported_seq":133,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364879+0000","last_change":"2024-09-19T07:19:39.410500+0000","last_active":"2024-09-19T07:48:58.364879+0000","last_peered":"2024-09-19T07:48:58.364879+0000","last_clean":"2024-09-19T07:48:58.364879+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:48:58.364879+0000","last_undegraded":"2024-09-19T07:48:58.364879+0000","last_fullsized":"2024-09-19T07:48:58.364879+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:04.182606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"64'4389","reported_seq":4494,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378148+0000","last_change":"2024-09-19T07:19:40.381259+0000","last_active":"2024-09-19T07:48:58.378148+0000","last_peered":"2024-09-19T07:48:58.378148+0000","last_clean":"2024-09-19T07:48:58.378148+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:48:58.378148+0000","last_undegraded":"2024-09-19T07:48:58.378148+0000","last_fullsized":"2024-09-19T07:48:58.378148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4389,"log_dups_size":0,"ondisk_log_size":4389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:22.017682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23791,"num_write_kb":304343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"61'3782","reported_seq":3895,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364288+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:48:58.364288+0000","last_peered":"2024-09-19T07:48:58.364288+0000","last_clean":"2024-09-19T07:48:58.364288+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:48:58.364288+0000","last_undegraded":"2024-09-19T07:48:58.364288+0000","last_fullsized":"2024-09-19T07:48:58.364288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19997,"num_write_kb":252226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"61'3","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371722+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:48:58.371722+0000","last_peered":"2024-09-19T07:48:58.371722+0000","last_clean":"2024-09-19T07:48:58.371722+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:48:58.371722+0000","last_undegraded":"2024-09-19T07:48:58.371722+0000","last_fullsized":"2024-09-19T07:48:58.371722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"64'5","reported_seq":112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368513+0000","last_change":"2024-09-19T07:19:39.412123+0000","last_active":"2024-09-19T07:48:58.368513+0000","last_peered":"2024-09-19T07:48:58.368513+0000","last_clean":"2024-09-19T07:48:58.368513+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:48:58.368513+0000","last_undegraded":"2024-09-19T07:48:58.368513+0000","last_fullsized":"2024-09-19T07:48:58.368513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"64'3306","reported_seq":3411,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364068+0000","last_change":"2024-09-19T07:19:40.383214+0000","last_active":"2024-09-19T07:48:58.364068+0000","last_peered":"2024-09-19T07:48:58.364068+0000","last_clean":"2024-09-19T07:48:58.364068+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:48:58.364068+0000","last_undegraded":"2024-09-19T07:48:58.364068+0000","last_fullsized":"2024-09-19T07:48:58.364068+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17836,"num_write_kb":239496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"61'4850","reported_seq":4963,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363649+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:48:58.363649+0000","last_peered":"2024-09-19T07:48:58.363649+0000","last_clean":"2024-09-19T07:48:58.363649+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:48:58.363649+0000","last_undegraded":"2024-09-19T07:48:58.363649+0000","last_fullsized":"2024-09-19T07:48:58.363649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26728,"num_write_kb":308134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"61'2","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372455+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:48:58.372455+0000","last_peered":"2024-09-19T07:48:58.372455+0000","last_clean":"2024-09-19T07:48:58.372455+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:48:58.372455+0000","last_undegraded":"2024-09-19T07:48:58.372455+0000","last_fullsized":"2024-09-19T07:48:58.372455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"64'4","reported_seq":131,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374187+0000","last_change":"2024-09-19T07:19:39.408228+0000","last_active":"2024-09-19T07:48:58.374187+0000","last_peered":"2024-09-19T07:48:58.374187+0000","last_clean":"2024-09-19T07:48:58.374187+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:48:58.374187+0000","last_undegraded":"2024-09-19T07:48:58.374187+0000","last_fullsized":"2024-09-19T07:48:58.374187+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"64'5157","reported_seq":5262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371211+0000","last_change":"2024-09-19T07:19:40.395566+0000","last_active":"2024-09-19T07:48:58.371211+0000","last_peered":"2024-09-19T07:48:58.371211+0000","last_clean":"2024-09-19T07:48:58.371211+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:48:58.371211+0000","last_undegraded":"2024-09-19T07:48:58.371211+0000","last_fullsized":"2024-09-19T07:48:58.371211+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:54:02.766021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27729,"num_write_kb":365262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"61'4508","reported_seq":4621,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365068+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:48:58.365068+0000","last_peered":"2024-09-19T07:48:58.365068+0000","last_clean":"2024-09-19T07:48:58.365068+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:48:58.365068+0000","last_undegraded":"2024-09-19T07:48:58.365068+0000","last_fullsized":"2024-09-19T07:48:58.365068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24068,"num_write_kb":289158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"61'5","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374924+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:48:58.374924+0000","last_peered":"2024-09-19T07:48:58.374924+0000","last_clean":"2024-09-19T07:48:58.374924+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:48:58.374924+0000","last_undegraded":"2024-09-19T07:48:58.374924+0000","last_fullsized":"2024-09-19T07:48:58.374924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"64'7","reported_seq":154,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364522+0000","last_change":"2024-09-19T07:19:39.424661+0000","last_active":"2024-09-19T07:48:58.364522+0000","last_peered":"2024-09-19T07:48:58.364522+0000","last_clean":"2024-09-19T07:48:58.364522+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:48:58.364522+0000","last_undegraded":"2024-09-19T07:48:58.364522+0000","last_fullsized":"2024-09-19T07:48:58.364522+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"64'3458","reported_seq":3564,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362033+0000","last_change":"2024-09-19T07:19:40.388972+0000","last_active":"2024-09-19T07:48:58.362033+0000","last_peered":"2024-09-19T07:48:58.362033+0000","last_clean":"2024-09-19T07:48:58.362033+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:48:58.362033+0000","last_undegraded":"2024-09-19T07:48:58.362033+0000","last_fullsized":"2024-09-19T07:48:58.362033+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3458,"log_dups_size":0,"ondisk_log_size":3458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18862,"num_write_kb":239540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"61'5647","reported_seq":5761,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369864+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:48:58.369864+0000","last_peered":"2024-09-19T07:48:58.369864+0000","last_clean":"2024-09-19T07:48:58.369864+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:48:58.369864+0000","last_undegraded":"2024-09-19T07:48:58.369864+0000","last_fullsized":"2024-09-19T07:48:58.369864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:23.791880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29735,"num_write_kb":364939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"63'9","reported_seq":164,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.795335+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:49:01.795335+0000","last_peered":"2024-09-19T07:49:01.795335+0000","last_clean":"2024-09-19T07:49:01.795335+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:49:01.795335+0000","last_undegraded":"2024-09-19T07:49:01.795335+0000","last_fullsized":"2024-09-19T07:49:01.795335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'45","reported_seq":213,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:08.425477+0000","last_change":"2024-09-19T07:19:39.412192+0000","last_active":"2024-09-19T07:49:08.425477+0000","last_peered":"2024-09-19T07:49:08.425477+0000","last_clean":"2024-09-19T07:49:08.425477+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:49:08.425477+0000","last_undegraded":"2024-09-19T07:49:08.425477+0000","last_fullsized":"2024-09-19T07:49:08.425477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":108,"num_write":41,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"64'4007","reported_seq":4113,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363551+0000","last_change":"2024-09-19T07:19:40.390217+0000","last_active":"2024-09-19T07:48:58.363551+0000","last_peered":"2024-09-19T07:48:58.363551+0000","last_clean":"2024-09-19T07:48:58.363551+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:48:58.363551+0000","last_undegraded":"2024-09-19T07:48:58.363551+0000","last_fullsized":"2024-09-19T07:48:58.363551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21606,"num_write_kb":266928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"61'4152","reported_seq":4265,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364929+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:48:58.364929+0000","last_peered":"2024-09-19T07:48:58.364929+0000","last_clean":"2024-09-19T07:48:58.364929+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:48:58.364929+0000","last_undegraded":"2024-09-19T07:48:58.364929+0000","last_fullsized":"2024-09-19T07:48:58.364929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22349,"num_write_kb":269915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"65'474","reported_seq":609,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.366499+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:49:03.366499+0000","last_peered":"2024-09-19T07:49:03.366499+0000","last_clean":"2024-09-19T07:49:03.366499+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:49:03.366499+0000","last_undegraded":"2024-09-19T07:49:03.366499+0000","last_fullsized":"2024-09-19T07:49:03.366499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:50.700962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":59,"num_write":1141,"num_write_kb":380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"64'10","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362143+0000","last_change":"2024-09-19T07:19:39.434150+0000","last_active":"2024-09-19T07:48:58.362143+0000","last_peered":"2024-09-19T07:48:58.362143+0000","last_clean":"2024-09-19T07:48:58.362143+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:48:58.362143+0000","last_undegraded":"2024-09-19T07:48:58.362143+0000","last_fullsized":"2024-09-19T07:48:58.362143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:13.654208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"64'4199","reported_seq":4305,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380465+0000","last_change":"2024-09-19T07:19:40.380140+0000","last_active":"2024-09-19T07:48:58.380465+0000","last_peered":"2024-09-19T07:48:58.380465+0000","last_clean":"2024-09-19T07:48:58.380465+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:48:58.380465+0000","last_undegraded":"2024-09-19T07:48:58.380465+0000","last_fullsized":"2024-09-19T07:48:58.380465+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22884,"num_write_kb":280362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"61'5900","reported_seq":6014,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372730+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:48:58.372730+0000","last_peered":"2024-09-19T07:48:58.372730+0000","last_clean":"2024-09-19T07:48:58.372730+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:48:58.372730+0000","last_undegraded":"2024-09-19T07:48:58.372730+0000","last_fullsized":"2024-09-19T07:48:58.372730+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31464,"num_write_kb":364012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"61'2","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379634+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:48:58.379634+0000","last_peered":"2024-09-19T07:48:58.379634+0000","last_clean":"2024-09-19T07:48:58.379634+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:48:58.379634+0000","last_undegraded":"2024-09-19T07:48:58.379634+0000","last_fullsized":"2024-09-19T07:48:58.379634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"64'3","reported_seq":111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370135+0000","last_change":"2024-09-19T07:19:39.411605+0000","last_active":"2024-09-19T07:48:58.370135+0000","last_peered":"2024-09-19T07:48:58.370135+0000","last_clean":"2024-09-19T07:48:58.370135+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:48:58.370135+0000","last_undegraded":"2024-09-19T07:48:58.370135+0000","last_fullsized":"2024-09-19T07:48:58.370135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:56:55.654483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"64'4404","reported_seq":4510,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370924+0000","last_change":"2024-09-19T07:19:40.372083+0000","last_active":"2024-09-19T07:48:58.370924+0000","last_peered":"2024-09-19T07:48:58.370924+0000","last_clean":"2024-09-19T07:48:58.370924+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:48:58.370924+0000","last_undegraded":"2024-09-19T07:48:58.370924+0000","last_fullsized":"2024-09-19T07:48:58.370924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:39.665475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24188,"num_write_kb":314198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"61'3777","reported_seq":3891,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368386+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:48:58.368386+0000","last_peered":"2024-09-19T07:48:58.368386+0000","last_clean":"2024-09-19T07:48:58.368386+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:48:58.368386+0000","last_undegraded":"2024-09-19T07:48:58.368386+0000","last_fullsized":"2024-09-19T07:48:58.368386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20312,"num_write_kb":245406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"61'2","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372774+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:48:58.372774+0000","last_peered":"2024-09-19T07:48:58.372774+0000","last_clean":"2024-09-19T07:48:58.372774+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:48:58.372774+0000","last_undegraded":"2024-09-19T07:48:58.372774+0000","last_fullsized":"2024-09-19T07:48:58.372774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'4","reported_seq":112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371516+0000","last_change":"2024-09-19T07:19:39.420340+0000","last_active":"2024-09-19T07:48:58.371516+0000","last_peered":"2024-09-19T07:48:58.371516+0000","last_clean":"2024-09-19T07:48:58.371516+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:48:58.371516+0000","last_undegraded":"2024-09-19T07:48:58.371516+0000","last_fullsized":"2024-09-19T07:48:58.371516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:47:29.338848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"64'4365","reported_seq":4471,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369175+0000","last_change":"2024-09-19T07:19:40.383082+0000","last_active":"2024-09-19T07:48:58.369175+0000","last_peered":"2024-09-19T07:48:58.369175+0000","last_clean":"2024-09-19T07:48:58.369175+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:48:58.369175+0000","last_undegraded":"2024-09-19T07:48:58.369175+0000","last_fullsized":"2024-09-19T07:48:58.369175+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:54.436493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24012,"num_write_kb":288918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'5396","reported_seq":5510,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370199+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:48:58.370199+0000","last_peered":"2024-09-19T07:48:58.370199+0000","last_clean":"2024-09-19T07:48:58.370199+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:48:58.370199+0000","last_undegraded":"2024-09-19T07:48:58.370199+0000","last_fullsized":"2024-09-19T07:48:58.370199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28800,"num_write_kb":336838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'4","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368215+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:48:58.368215+0000","last_peered":"2024-09-19T07:48:58.368215+0000","last_clean":"2024-09-19T07:48:58.368215+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:48:58.368215+0000","last_undegraded":"2024-09-19T07:48:58.368215+0000","last_fullsized":"2024-09-19T07:48:58.368215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"64'2","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372603+0000","last_change":"2024-09-19T07:19:39.407135+0000","last_active":"2024-09-19T07:48:58.372603+0000","last_peered":"2024-09-19T07:48:58.372603+0000","last_clean":"2024-09-19T07:48:58.372603+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:48:58.372603+0000","last_undegraded":"2024-09-19T07:48:58.372603+0000","last_fullsized":"2024-09-19T07:48:58.372603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:27.890595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"64'4926","reported_seq":5032,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378101+0000","last_change":"2024-09-19T07:19:40.368866+0000","last_active":"2024-09-19T07:48:58.378101+0000","last_peered":"2024-09-19T07:48:58.378101+0000","last_clean":"2024-09-19T07:48:58.378101+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:48:58.378101+0000","last_undegraded":"2024-09-19T07:48:58.378101+0000","last_fullsized":"2024-09-19T07:48:58.378101+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:28.893439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26066,"num_write_kb":337871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'3534","reported_seq":3648,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371281+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:48:58.371281+0000","last_peered":"2024-09-19T07:48:58.371281+0000","last_clean":"2024-09-19T07:48:58.371281+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:48:58.371281+0000","last_undegraded":"2024-09-19T07:48:58.371281+0000","last_fullsized":"2024-09-19T07:48:58.371281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19019,"num_write_kb":227884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"65'46","reported_seq":221,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.373338+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:49:03.373338+0000","last_peered":"2024-09-19T07:49:03.373338+0000","last_clean":"2024-09-19T07:49:03.373338+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:49:03.373338+0000","last_undegraded":"2024-09-19T07:49:03.373338+0000","last_fullsized":"2024-09-19T07:49:03.373338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":109,"num_write":40,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'9","reported_seq":156,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374298+0000","last_change":"2024-09-19T07:19:39.420497+0000","last_active":"2024-09-19T07:48:58.374298+0000","last_peered":"2024-09-19T07:48:58.374298+0000","last_clean":"2024-09-19T07:48:58.374298+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:48:58.374298+0000","last_undegraded":"2024-09-19T07:48:58.374298+0000","last_fullsized":"2024-09-19T07:48:58.374298+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:19.547270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"64'3425","reported_seq":3530,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371701+0000","last_change":"2024-09-19T07:19:40.387573+0000","last_active":"2024-09-19T07:48:58.371701+0000","last_peered":"2024-09-19T07:48:58.371701+0000","last_clean":"2024-09-19T07:48:58.371701+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:48:58.371701+0000","last_undegraded":"2024-09-19T07:48:58.371701+0000","last_fullsized":"2024-09-19T07:48:58.371701+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:20:43.234793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19036,"num_write_kb":252575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"61'4467","reported_seq":4581,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368938+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:48:58.368938+0000","last_peered":"2024-09-19T07:48:58.368938+0000","last_clean":"2024-09-19T07:48:58.368938+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:48:58.368938+0000","last_undegraded":"2024-09-19T07:48:58.368938+0000","last_fullsized":"2024-09-19T07:48:58.368938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:20.824916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23642,"num_write_kb":275072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'13","reported_seq":129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375436+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:48:58.375436+0000","last_peered":"2024-09-19T07:48:58.375436+0000","last_clean":"2024-09-19T07:48:58.375436+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:48:58.375436+0000","last_undegraded":"2024-09-19T07:48:58.375436+0000","last_fullsized":"2024-09-19T07:48:58.375436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'515","reported_seq":642,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:06.371206+0000","last_change":"2024-09-19T07:19:39.410752+0000","last_active":"2024-09-19T07:49:06.371206+0000","last_peered":"2024-09-19T07:49:06.371206+0000","last_clean":"2024-09-19T07:49:06.371206+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:49:06.371206+0000","last_undegraded":"2024-09-19T07:49:06.371206+0000","last_fullsized":"2024-09-19T07:49:06.371206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:47:09.376497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":53,"num_write":1231,"num_write_kb":406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"64'4906","reported_seq":5012,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372981+0000","last_change":"2024-09-19T07:19:40.392973+0000","last_active":"2024-09-19T07:48:58.372981+0000","last_peered":"2024-09-19T07:48:58.372981+0000","last_clean":"2024-09-19T07:48:58.372981+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:48:58.372981+0000","last_undegraded":"2024-09-19T07:48:58.372981+0000","last_fullsized":"2024-09-19T07:48:58.372981+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27102,"num_write_kb":350472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"61'4338","reported_seq":4452,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363793+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:48:58.363793+0000","last_peered":"2024-09-19T07:48:58.363793+0000","last_clean":"2024-09-19T07:48:58.363793+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:48:58.363793+0000","last_undegraded":"2024-09-19T07:48:58.363793+0000","last_fullsized":"2024-09-19T07:48:58.363793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22767,"num_write_kb":271598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"61'3","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365893+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:48:58.365893+0000","last_peered":"2024-09-19T07:48:58.365893+0000","last_clean":"2024-09-19T07:48:58.365893+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:48:58.365893+0000","last_undegraded":"2024-09-19T07:48:58.365893+0000","last_fullsized":"2024-09-19T07:48:58.365893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":272,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.376591+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:48:58.376591+0000","last_peered":"2024-09-19T07:48:58.376591+0000","last_clean":"2024-09-19T07:48:58.376591+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:48:58.376591+0000","last_undegraded":"2024-09-19T07:48:58.376591+0000","last_fullsized":"2024-09-19T07:48:58.376591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'2","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380743+0000","last_change":"2024-09-19T07:19:39.429085+0000","last_active":"2024-09-19T07:48:58.380743+0000","last_peered":"2024-09-19T07:48:58.380743+0000","last_clean":"2024-09-19T07:48:58.380743+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:48:58.380743+0000","last_undegraded":"2024-09-19T07:48:58.380743+0000","last_fullsized":"2024-09-19T07:48:58.380743+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"64'3599","reported_seq":3704,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378157+0000","last_change":"2024-09-19T07:19:40.382352+0000","last_active":"2024-09-19T07:48:58.378157+0000","last_peered":"2024-09-19T07:48:58.378157+0000","last_clean":"2024-09-19T07:48:58.378157+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:48:58.378157+0000","last_undegraded":"2024-09-19T07:48:58.378157+0000","last_fullsized":"2024-09-19T07:48:58.378157+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:31.024388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19570,"num_write_kb":246554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"61'5018","reported_seq":5131,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379025+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:48:58.379025+0000","last_peered":"2024-09-19T07:48:58.379025+0000","last_clean":"2024-09-19T07:48:58.379025+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:48:58.379025+0000","last_undegraded":"2024-09-19T07:48:58.379025+0000","last_fullsized":"2024-09-19T07:48:58.379025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26753,"num_write_kb":316367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"61'5","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378898+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:48:58.378898+0000","last_peered":"2024-09-19T07:48:58.378898+0000","last_clean":"2024-09-19T07:48:58.378898+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:48:58.378898+0000","last_undegraded":"2024-09-19T07:48:58.378898+0000","last_fullsized":"2024-09-19T07:48:58.378898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"64'3","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380502+0000","last_change":"2024-09-19T07:19:39.427644+0000","last_active":"2024-09-19T07:48:58.380502+0000","last_peered":"2024-09-19T07:48:58.380502+0000","last_clean":"2024-09-19T07:48:58.380502+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:48:58.380502+0000","last_undegraded":"2024-09-19T07:48:58.380502+0000","last_fullsized":"2024-09-19T07:48:58.380502+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"64'4954","reported_seq":5059,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377576+0000","last_change":"2024-09-19T07:19:40.390380+0000","last_active":"2024-09-19T07:48:58.377576+0000","last_peered":"2024-09-19T07:48:58.377576+0000","last_clean":"2024-09-19T07:48:58.377576+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:48:58.377576+0000","last_undegraded":"2024-09-19T07:48:58.377576+0000","last_fullsized":"2024-09-19T07:48:58.377576+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:26.385490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27293,"num_write_kb":322594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"61'5220","reported_seq":5318,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377825+0000","last_change":"2024-09-19T07:20:13.952747+0000","last_active":"2024-09-19T07:48:58.377825+0000","last_peered":"2024-09-19T07:48:58.377825+0000","last_clean":"2024-09-19T07:48:58.377825+0000","last_became_active":"2024-09-19T07:20:13.951449+0000","last_became_peered":"2024-09-19T07:20:13.951449+0000","last_unstale":"2024-09-19T07:48:58.377825+0000","last_undegraded":"2024-09-19T07:48:58.377825+0000","last_fullsized":"2024-09-19T07:48:58.377825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:56.805700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27694,"num_write_kb":340542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"61'6","reported_seq":141,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374333+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:48:58.374333+0000","last_peered":"2024-09-19T07:48:58.374333+0000","last_clean":"2024-09-19T07:48:58.374333+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:48:58.374333+0000","last_undegraded":"2024-09-19T07:48:58.374333+0000","last_fullsized":"2024-09-19T07:48:58.374333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"64'1403","reported_seq":1553,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369177+0000","last_change":"2024-09-19T07:19:39.410638+0000","last_active":"2024-09-19T07:48:58.369177+0000","last_peered":"2024-09-19T07:48:58.369177+0000","last_clean":"2024-09-19T07:48:58.369177+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:48:58.369177+0000","last_undegraded":"2024-09-19T07:48:58.369177+0000","last_fullsized":"2024-09-19T07:48:58.369177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1403,"log_dups_size":0,"ondisk_log_size":1403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:32.073136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":63,"num_read_kb":30148,"num_write":1379,"num_write_kb":4466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"64'3869","reported_seq":3974,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372595+0000","last_change":"2024-09-19T07:19:40.378481+0000","last_active":"2024-09-19T07:48:58.372595+0000","last_peered":"2024-09-19T07:48:58.372595+0000","last_clean":"2024-09-19T07:48:58.372595+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:48:58.372595+0000","last_undegraded":"2024-09-19T07:48:58.372595+0000","last_fullsized":"2024-09-19T07:48:58.372595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3869,"log_dups_size":0,"ondisk_log_size":3869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:30.585686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20920,"num_write_kb":264190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"61'6113","reported_seq":6211,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373547+0000","last_change":"2024-09-19T07:20:14.209767+0000","last_active":"2024-09-19T07:48:58.373547+0000","last_peered":"2024-09-19T07:48:58.373547+0000","last_clean":"2024-09-19T07:48:58.373547+0000","last_became_active":"2024-09-19T07:20:14.209141+0000","last_became_peered":"2024-09-19T07:20:14.209141+0000","last_unstale":"2024-09-19T07:48:58.373547+0000","last_undegraded":"2024-09-19T07:48:58.373547+0000","last_fullsized":"2024-09-19T07:48:58.373547+0000","mapping_epoch":26,"log_start":"54'300","ondisk_log_start":"54'300","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:34:10.251087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32041,"num_write_kb":407182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"61'7","reported_seq":162,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.794320+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:49:01.794320+0000","last_peered":"2024-09-19T07:49:01.794320+0000","last_clean":"2024-09-19T07:49:01.794320+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:49:01.794320+0000","last_undegraded":"2024-09-19T07:49:01.794320+0000","last_fullsized":"2024-09-19T07:49:01.794320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"64'5","reported_seq":112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365126+0000","last_change":"2024-09-19T07:19:39.437656+0000","last_active":"2024-09-19T07:48:58.365126+0000","last_peered":"2024-09-19T07:48:58.365126+0000","last_clean":"2024-09-19T07:48:58.365126+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:48:58.365126+0000","last_undegraded":"2024-09-19T07:48:58.365126+0000","last_fullsized":"2024-09-19T07:48:58.365126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:41.468096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"64'4634","reported_seq":4739,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377494+0000","last_change":"2024-09-19T07:19:40.379289+0000","last_active":"2024-09-19T07:48:58.377494+0000","last_peered":"2024-09-19T07:48:58.377494+0000","last_clean":"2024-09-19T07:48:58.377494+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:48:58.377494+0000","last_undegraded":"2024-09-19T07:48:58.377494+0000","last_fullsized":"2024-09-19T07:48:58.377494+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:20.399106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25462,"num_write_kb":298798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"61'4007","reported_seq":4120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375692+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:48:58.375692+0000","last_peered":"2024-09-19T07:48:58.375692+0000","last_clean":"2024-09-19T07:48:58.375692+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:48:58.375692+0000","last_undegraded":"2024-09-19T07:48:58.375692+0000","last_fullsized":"2024-09-19T07:48:58.375692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21937,"num_write_kb":251948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"61'4","reported_seq":139,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369549+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:48:58.369549+0000","last_peered":"2024-09-19T07:48:58.369549+0000","last_clean":"2024-09-19T07:48:58.369549+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:48:58.369549+0000","last_undegraded":"2024-09-19T07:48:58.369549+0000","last_fullsized":"2024-09-19T07:48:58.369549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"64'2","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363818+0000","last_change":"2024-09-19T07:19:39.413589+0000","last_active":"2024-09-19T07:48:58.363818+0000","last_peered":"2024-09-19T07:48:58.363818+0000","last_clean":"2024-09-19T07:48:58.363818+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:48:58.363818+0000","last_undegraded":"2024-09-19T07:48:58.363818+0000","last_fullsized":"2024-09-19T07:48:58.363818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"64'3939","reported_seq":4043,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364105+0000","last_change":"2024-09-19T07:19:40.387159+0000","last_active":"2024-09-19T07:48:58.364105+0000","last_peered":"2024-09-19T07:48:58.364105+0000","last_clean":"2024-09-19T07:48:58.364105+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:48:58.364105+0000","last_undegraded":"2024-09-19T07:48:58.364105+0000","last_fullsized":"2024-09-19T07:48:58.364105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20741,"num_write_kb":275861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'4660","reported_seq":4774,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373923+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:48:58.373923+0000","last_peered":"2024-09-19T07:48:58.373923+0000","last_clean":"2024-09-19T07:48:58.373923+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:48:58.373923+0000","last_undegraded":"2024-09-19T07:48:58.373923+0000","last_fullsized":"2024-09-19T07:48:58.373923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25264,"num_write_kb":302007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"65'15","reported_seq":213,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.799096+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:49:01.799096+0000","last_peered":"2024-09-19T07:49:01.799096+0000","last_clean":"2024-09-19T07:49:01.799096+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:49:01.799096+0000","last_undegraded":"2024-09-19T07:49:01.799096+0000","last_fullsized":"2024-09-19T07:49:01.799096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"64'3","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369815+0000","last_change":"2024-09-19T07:19:39.406110+0000","last_active":"2024-09-19T07:48:58.369815+0000","last_peered":"2024-09-19T07:48:58.369815+0000","last_clean":"2024-09-19T07:48:58.369815+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:48:58.369815+0000","last_undegraded":"2024-09-19T07:48:58.369815+0000","last_fullsized":"2024-09-19T07:48:58.369815+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:10:05.224225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"64'4590","reported_seq":4694,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366527+0000","last_change":"2024-09-19T07:19:40.365764+0000","last_active":"2024-09-19T07:48:58.366527+0000","last_peered":"2024-09-19T07:48:58.366527+0000","last_clean":"2024-09-19T07:48:58.366527+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:48:58.366527+0000","last_undegraded":"2024-09-19T07:48:58.366527+0000","last_fullsized":"2024-09-19T07:48:58.366527+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:04:03.242292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24790,"num_write_kb":304361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"61'3873","reported_seq":3985,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380282+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:48:58.380282+0000","last_peered":"2024-09-19T07:48:58.380282+0000","last_clean":"2024-09-19T07:48:58.380282+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:48:58.380282+0000","last_undegraded":"2024-09-19T07:48:58.380282+0000","last_fullsized":"2024-09-19T07:48:58.380282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20508,"num_write_kb":250225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"61'5","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381953+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:48:58.381953+0000","last_peered":"2024-09-19T07:48:58.381953+0000","last_clean":"2024-09-19T07:48:58.381953+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:48:58.381953+0000","last_undegraded":"2024-09-19T07:48:58.381953+0000","last_fullsized":"2024-09-19T07:48:58.381953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"64'4","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364247+0000","last_change":"2024-09-19T07:19:39.429768+0000","last_active":"2024-09-19T07:48:58.364247+0000","last_peered":"2024-09-19T07:48:58.364247+0000","last_clean":"2024-09-19T07:48:58.364247+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:48:58.364247+0000","last_undegraded":"2024-09-19T07:48:58.364247+0000","last_fullsized":"2024-09-19T07:48:58.364247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:47.503631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"64'3485","reported_seq":3589,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366523+0000","last_change":"2024-09-19T07:19:40.373542+0000","last_active":"2024-09-19T07:48:58.366523+0000","last_peered":"2024-09-19T07:48:58.366523+0000","last_clean":"2024-09-19T07:48:58.366523+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:48:58.366523+0000","last_undegraded":"2024-09-19T07:48:58.366523+0000","last_fullsized":"2024-09-19T07:48:58.366523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:14.326662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19669,"num_write_kb":242834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"61'5231","reported_seq":5343,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373420+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:48:58.373420+0000","last_peered":"2024-09-19T07:48:58.373420+0000","last_clean":"2024-09-19T07:48:58.373420+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:48:58.373420+0000","last_undegraded":"2024-09-19T07:48:58.373420+0000","last_fullsized":"2024-09-19T07:48:58.373420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28071,"num_write_kb":333869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"61'3","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367894+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:48:58.367894+0000","last_peered":"2024-09-19T07:48:58.367894+0000","last_clean":"2024-09-19T07:48:58.367894+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:48:58.367894+0000","last_undegraded":"2024-09-19T07:48:58.367894+0000","last_fullsized":"2024-09-19T07:48:58.367894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"64'328","reported_seq":535,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363630+0000","last_change":"2024-09-19T07:19:39.411074+0000","last_active":"2024-09-19T07:48:58.363630+0000","last_peered":"2024-09-19T07:48:58.363630+0000","last_clean":"2024-09-19T07:48:58.363630+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:48:58.363630+0000","last_undegraded":"2024-09-19T07:48:58.363630+0000","last_fullsized":"2024-09-19T07:48:58.363630+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:53.992787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":181,"num_read_kb":77990,"num_write":325,"num_write_kb":4282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"64'5112","reported_seq":5216,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363063+0000","last_change":"2024-09-19T07:19:40.364814+0000","last_active":"2024-09-19T07:48:58.363063+0000","last_peered":"2024-09-19T07:48:58.363063+0000","last_clean":"2024-09-19T07:48:58.363063+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:48:58.363063+0000","last_undegraded":"2024-09-19T07:48:58.363063+0000","last_fullsized":"2024-09-19T07:48:58.363063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5112,"log_dups_size":0,"ondisk_log_size":5112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27577,"num_write_kb":360586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"61'4085","reported_seq":4197,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369111+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:48:58.369111+0000","last_peered":"2024-09-19T07:48:58.369111+0000","last_clean":"2024-09-19T07:48:58.369111+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:48:58.369111+0000","last_undegraded":"2024-09-19T07:48:58.369111+0000","last_fullsized":"2024-09-19T07:48:58.369111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:57.131912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22122,"num_write_kb":273218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"65'10","reported_seq":164,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.897451+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:48:58.897451+0000","last_peered":"2024-09-19T07:48:58.897451+0000","last_clean":"2024-09-19T07:48:58.897451+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:48:58.897451+0000","last_undegraded":"2024-09-19T07:48:58.897451+0000","last_fullsized":"2024-09-19T07:48:58.897451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:23.367365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'29","reported_seq":195,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.423376+0000","last_change":"2024-09-19T07:19:39.431616+0000","last_active":"2024-09-19T07:48:58.423376+0000","last_peered":"2024-09-19T07:48:58.423376+0000","last_clean":"2024-09-19T07:48:58.423376+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:48:58.423376+0000","last_undegraded":"2024-09-19T07:48:58.423376+0000","last_fullsized":"2024-09-19T07:48:58.423376+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:18.149581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"64'4543","reported_seq":4647,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363327+0000","last_change":"2024-09-19T07:19:40.381443+0000","last_active":"2024-09-19T07:48:58.363327+0000","last_peered":"2024-09-19T07:48:58.363327+0000","last_clean":"2024-09-19T07:48:58.363327+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:48:58.363327+0000","last_undegraded":"2024-09-19T07:48:58.363327+0000","last_fullsized":"2024-09-19T07:48:58.363327+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24993,"num_write_kb":317373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"61'3981","reported_seq":4093,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375912+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:48:58.375912+0000","last_peered":"2024-09-19T07:48:58.375912+0000","last_clean":"2024-09-19T07:48:58.375912+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:48:58.375912+0000","last_undegraded":"2024-09-19T07:48:58.375912+0000","last_fullsized":"2024-09-19T07:48:58.375912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3981,"log_dups_size":0,"ondisk_log_size":3981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21890,"num_write_kb":263866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"65'352","reported_seq":567,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.802657+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:49:01.802657+0000","last_peered":"2024-09-19T07:49:01.802657+0000","last_clean":"2024-09-19T07:49:01.802657+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:49:01.802657+0000","last_undegraded":"2024-09-19T07:49:01.802657+0000","last_fullsized":"2024-09-19T07:49:01.802657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":181,"num_read_kb":77990,"num_write":347,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"64'3","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371323+0000","last_change":"2024-09-19T07:19:39.423972+0000","last_active":"2024-09-19T07:48:58.371323+0000","last_peered":"2024-09-19T07:48:58.371323+0000","last_clean":"2024-09-19T07:48:58.371323+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:48:58.371323+0000","last_undegraded":"2024-09-19T07:48:58.371323+0000","last_fullsized":"2024-09-19T07:48:58.371323+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:45.387040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"64'3743","reported_seq":3847,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371573+0000","last_change":"2024-09-19T07:19:40.382092+0000","last_active":"2024-09-19T07:48:58.371573+0000","last_peered":"2024-09-19T07:48:58.371573+0000","last_clean":"2024-09-19T07:48:58.371573+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:48:58.371573+0000","last_undegraded":"2024-09-19T07:48:58.371573+0000","last_fullsized":"2024-09-19T07:48:58.371573+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20626,"num_write_kb":263086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"61'6160","reported_seq":6272,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369147+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:48:58.369147+0000","last_peered":"2024-09-19T07:48:58.369147+0000","last_clean":"2024-09-19T07:48:58.369147+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:48:58.369147+0000","last_undegraded":"2024-09-19T07:48:58.369147+0000","last_fullsized":"2024-09-19T07:48:58.369147+0000","mapping_epoch":19,"log_start":"54'700","ondisk_log_start":"54'700","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32461,"num_write_kb":376158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"65'29","reported_seq":203,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.368449+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:49:03.368449+0000","last_peered":"2024-09-19T07:49:03.368449+0000","last_clean":"2024-09-19T07:49:03.368449+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:49:03.368449+0000","last_undegraded":"2024-09-19T07:49:03.368449+0000","last_fullsized":"2024-09-19T07:49:03.368449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"64'8","reported_seq":154,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369556+0000","last_change":"2024-09-19T07:19:39.424879+0000","last_active":"2024-09-19T07:48:58.369556+0000","last_peered":"2024-09-19T07:48:58.369556+0000","last_clean":"2024-09-19T07:48:58.369556+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:48:58.369556+0000","last_undegraded":"2024-09-19T07:48:58.369556+0000","last_fullsized":"2024-09-19T07:48:58.369556+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:30.021543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"64'4564","reported_seq":4670,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382249+0000","last_change":"2024-09-19T07:19:40.384136+0000","last_active":"2024-09-19T07:48:58.382249+0000","last_peered":"2024-09-19T07:48:58.382249+0000","last_clean":"2024-09-19T07:48:58.382249+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:48:58.382249+0000","last_undegraded":"2024-09-19T07:48:58.382249+0000","last_fullsized":"2024-09-19T07:48:58.382249+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25533,"num_write_kb":303195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"61'5068","reported_seq":5180,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369896+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:48:58.369896+0000","last_peered":"2024-09-19T07:48:58.369896+0000","last_clean":"2024-09-19T07:48:58.369896+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:48:58.369896+0000","last_undegraded":"2024-09-19T07:48:58.369896+0000","last_fullsized":"2024-09-19T07:48:58.369896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:48.308137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26828,"num_write_kb":316657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"61'3","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368011+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:48:58.368011+0000","last_peered":"2024-09-19T07:48:58.368011+0000","last_clean":"2024-09-19T07:48:58.368011+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:48:58.368011+0000","last_undegraded":"2024-09-19T07:48:58.368011+0000","last_fullsized":"2024-09-19T07:48:58.368011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'15","reported_seq":205,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362420+0000","last_change":"2024-09-19T07:19:39.410445+0000","last_active":"2024-09-19T07:48:58.362420+0000","last_peered":"2024-09-19T07:48:58.362420+0000","last_clean":"2024-09-19T07:48:58.362420+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:48:58.362420+0000","last_undegraded":"2024-09-19T07:48:58.362420+0000","last_fullsized":"2024-09-19T07:48:58.362420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:27.020327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"64'3771","reported_seq":3875,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373168+0000","last_change":"2024-09-19T07:19:40.389499+0000","last_active":"2024-09-19T07:48:58.373168+0000","last_peered":"2024-09-19T07:48:58.373168+0000","last_clean":"2024-09-19T07:48:58.373168+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:48:58.373168+0000","last_undegraded":"2024-09-19T07:48:58.373168+0000","last_fullsized":"2024-09-19T07:48:58.373168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3771,"log_dups_size":0,"ondisk_log_size":3771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.520872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19869,"num_write_kb":255725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"61'4973","reported_seq":5085,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367096+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:48:58.367096+0000","last_peered":"2024-09-19T07:48:58.367096+0000","last_clean":"2024-09-19T07:48:58.367096+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:48:58.367096+0000","last_undegraded":"2024-09-19T07:48:58.367096+0000","last_fullsized":"2024-09-19T07:48:58.367096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26723,"num_write_kb":320185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"65'6","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382937+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:48:58.382937+0000","last_peered":"2024-09-19T07:48:58.382937+0000","last_clean":"2024-09-19T07:48:58.382937+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:48:58.382937+0000","last_undegraded":"2024-09-19T07:48:58.382937+0000","last_fullsized":"2024-09-19T07:48:58.382937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"64'5","reported_seq":111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362191+0000","last_change":"2024-09-19T07:19:39.416943+0000","last_active":"2024-09-19T07:48:58.362191+0000","last_peered":"2024-09-19T07:48:58.362191+0000","last_clean":"2024-09-19T07:48:58.362191+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:48:58.362191+0000","last_undegraded":"2024-09-19T07:48:58.362191+0000","last_fullsized":"2024-09-19T07:48:58.362191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:35.555826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"64'5394","reported_seq":5500,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382132+0000","last_change":"2024-09-19T07:19:40.394993+0000","last_active":"2024-09-19T07:48:58.382132+0000","last_peered":"2024-09-19T07:48:58.382132+0000","last_clean":"2024-09-19T07:48:58.382132+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:48:58.382132+0000","last_undegraded":"2024-09-19T07:48:58.382132+0000","last_fullsized":"2024-09-19T07:48:58.382132+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5394,"log_dups_size":0,"ondisk_log_size":5394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30095,"num_write_kb":374705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"64'25","reported_seq":133,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369509+0000","last_change":"2024-09-19T07:19:39.418885+0000","last_active":"2024-09-19T07:48:58.369509+0000","last_peered":"2024-09-19T07:48:58.369509+0000","last_clean":"2024-09-19T07:48:58.369509+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:48:58.369509+0000","last_undegraded":"2024-09-19T07:48:58.369509+0000","last_fullsized":"2024-09-19T07:48:58.369509+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:35:45.790411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"61'4822","reported_seq":4934,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371995+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:48:58.371995+0000","last_peered":"2024-09-19T07:48:58.371995+0000","last_clean":"2024-09-19T07:48:58.371995+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:48:58.371995+0000","last_undegraded":"2024-09-19T07:48:58.371995+0000","last_fullsized":"2024-09-19T07:48:58.371995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25208,"num_write_kb":326524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"61'2","reported_seq":116,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367640+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:48:58.367640+0000","last_peered":"2024-09-19T07:48:58.367640+0000","last_clean":"2024-09-19T07:48:58.367640+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:48:58.367640+0000","last_undegraded":"2024-09-19T07:48:58.367640+0000","last_fullsized":"2024-09-19T07:48:58.367640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"64'4690","reported_seq":4796,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369407+0000","last_change":"2024-09-19T07:19:40.374048+0000","last_active":"2024-09-19T07:48:58.369407+0000","last_peered":"2024-09-19T07:48:58.369407+0000","last_clean":"2024-09-19T07:48:58.369407+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:48:58.369407+0000","last_undegraded":"2024-09-19T07:48:58.369407+0000","last_fullsized":"2024-09-19T07:48:58.369407+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:48:40.084827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25924,"num_write_kb":313037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"64'964","reported_seq":1117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382679+0000","last_change":"2024-09-19T07:19:39.432426+0000","last_active":"2024-09-19T07:48:58.382679+0000","last_peered":"2024-09-19T07:48:58.382679+0000","last_clean":"2024-09-19T07:48:58.382679+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:48:58.382679+0000","last_undegraded":"2024-09-19T07:48:58.382679+0000","last_fullsized":"2024-09-19T07:48:58.382679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":964,"log_dups_size":0,"ondisk_log_size":964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":8896,"num_write":940,"num_write_kb":4641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"61'5311","reported_seq":5423,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362632+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:48:58.362632+0000","last_peered":"2024-09-19T07:48:58.362632+0000","last_clean":"2024-09-19T07:48:58.362632+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:48:58.362632+0000","last_undegraded":"2024-09-19T07:48:58.362632+0000","last_fullsized":"2024-09-19T07:48:58.362632+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28108,"num_write_kb":333978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"61'3","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363067+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:48:58.363067+0000","last_peered":"2024-09-19T07:48:58.363067+0000","last_clean":"2024-09-19T07:48:58.363067+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:48:58.363067+0000","last_undegraded":"2024-09-19T07:48:58.363067+0000","last_fullsized":"2024-09-19T07:48:58.363067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"64'5270","reported_seq":5373,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373581+0000","last_change":"2024-09-19T07:19:40.384934+0000","last_active":"2024-09-19T07:48:58.373581+0000","last_peered":"2024-09-19T07:48:58.373581+0000","last_clean":"2024-09-19T07:48:58.373581+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:48:58.373581+0000","last_undegraded":"2024-09-19T07:48:58.373581+0000","last_fullsized":"2024-09-19T07:48:58.373581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29270,"num_write_kb":363254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"65'416","reported_seq":535,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:04.513350+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:49:04.513350+0000","last_peered":"2024-09-19T07:49:04.513350+0000","last_clean":"2024-09-19T07:49:04.513350+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:49:04.513350+0000","last_undegraded":"2024-09-19T07:49:04.513350+0000","last_fullsized":"2024-09-19T07:49:04.513350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2715831,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6193,"num_write":390,"num_write_kb":2885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"61'3569","reported_seq":3680,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371694+0000","last_change":"2024-09-19T07:19:32.289374+0000","last_active":"2024-09-19T07:48:58.371694+0000","last_peered":"2024-09-19T07:48:58.371694+0000","last_clean":"2024-09-19T07:48:58.371694+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:48:58.371694+0000","last_undegraded":"2024-09-19T07:48:58.371694+0000","last_fullsized":"2024-09-19T07:48:58.371694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18120,"num_write_kb":244659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"64'5","reported_seq":150,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378720+0000","last_change":"2024-09-19T07:19:39.410582+0000","last_active":"2024-09-19T07:48:58.378720+0000","last_peered":"2024-09-19T07:48:58.378720+0000","last_clean":"2024-09-19T07:48:58.378720+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:48:58.378720+0000","last_undegraded":"2024-09-19T07:48:58.378720+0000","last_fullsized":"2024-09-19T07:48:58.378720+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:41:40.592907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"64'3415","reported_seq":3520,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364298+0000","last_change":"2024-09-19T07:19:40.393950+0000","last_active":"2024-09-19T07:48:58.364298+0000","last_peered":"2024-09-19T07:48:58.364298+0000","last_clean":"2024-09-19T07:48:58.364298+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:48:58.364298+0000","last_undegraded":"2024-09-19T07:48:58.364298+0000","last_fullsized":"2024-09-19T07:48:58.364298+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18072,"num_write_kb":240550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"61'712","reported_seq":848,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364844+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:48:58.364844+0000","last_peered":"2024-09-19T07:48:58.364844+0000","last_clean":"2024-09-19T07:48:58.364844+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:48:58.364844+0000","last_undegraded":"2024-09-19T07:48:58.364844+0000","last_fullsized":"2024-09-19T07:48:58.364844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":74743,"num_write":701,"num_write_kb":4298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"61'4395","reported_seq":4507,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368988+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:48:58.368988+0000","last_peered":"2024-09-19T07:48:58.368988+0000","last_clean":"2024-09-19T07:48:58.368988+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:48:58.368988+0000","last_undegraded":"2024-09-19T07:48:58.368988+0000","last_fullsized":"2024-09-19T07:48:58.368988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22455,"num_write_kb":278564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"64'40","reported_seq":185,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363443+0000","last_change":"2024-09-19T07:19:39.410662+0000","last_active":"2024-09-19T07:48:58.363443+0000","last_peered":"2024-09-19T07:48:58.363443+0000","last_clean":"2024-09-19T07:48:58.363443+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:48:58.363443+0000","last_undegraded":"2024-09-19T07:48:58.363443+0000","last_fullsized":"2024-09-19T07:48:58.363443+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:19.196074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"64'4798","reported_seq":4903,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364274+0000","last_change":"2024-09-19T07:19:40.385581+0000","last_active":"2024-09-19T07:48:58.364274+0000","last_peered":"2024-09-19T07:48:58.364274+0000","last_clean":"2024-09-19T07:48:58.364274+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:48:58.364274+0000","last_undegraded":"2024-09-19T07:48:58.364274+0000","last_fullsized":"2024-09-19T07:48:58.364274+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:14:13.749729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27082,"num_write_kb":321689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"61'4997","reported_seq":5111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380242+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:48:58.380242+0000","last_peered":"2024-09-19T07:48:58.380242+0000","last_clean":"2024-09-19T07:48:58.380242+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:48:58.380242+0000","last_undegraded":"2024-09-19T07:48:58.380242+0000","last_fullsized":"2024-09-19T07:48:58.380242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4997,"log_dups_size":0,"ondisk_log_size":4997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26206,"num_write_kb":309941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"64'5490","reported_seq":5596,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373817+0000","last_change":"2024-09-19T07:19:40.377608+0000","last_active":"2024-09-19T07:48:58.373817+0000","last_peered":"2024-09-19T07:48:58.373817+0000","last_clean":"2024-09-19T07:48:58.373817+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:48:58.373817+0000","last_undegraded":"2024-09-19T07:48:58.373817+0000","last_fullsized":"2024-09-19T07:48:58.373817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5490,"log_dups_size":0,"ondisk_log_size":5490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29613,"num_write_kb":364653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"61'4890","reported_seq":5005,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377509+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:48:58.377509+0000","last_peered":"2024-09-19T07:48:58.377509+0000","last_clean":"2024-09-19T07:48:58.377509+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:48:58.377509+0000","last_undegraded":"2024-09-19T07:48:58.377509+0000","last_fullsized":"2024-09-19T07:48:58.377509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4890,"log_dups_size":0,"ondisk_log_size":4890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25338,"num_write_kb":314559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"64'4569","reported_seq":4675,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366516+0000","last_change":"2024-09-19T07:19:40.381262+0000","last_active":"2024-09-19T07:48:58.366516+0000","last_peered":"2024-09-19T07:48:58.366516+0000","last_clean":"2024-09-19T07:48:58.366516+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:48:58.366516+0000","last_undegraded":"2024-09-19T07:48:58.366516+0000","last_fullsized":"2024-09-19T07:48:58.366516+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:34.425472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25275,"num_write_kb":314704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"61'5227","reported_seq":5343,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367085+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:48:58.367085+0000","last_peered":"2024-09-19T07:48:58.367085+0000","last_clean":"2024-09-19T07:48:58.367085+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:48:58.367085+0000","last_undegraded":"2024-09-19T07:48:58.367085+0000","last_fullsized":"2024-09-19T07:48:58.367085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27444,"num_write_kb":331433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"64'4939","reported_seq":5044,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373561+0000","last_change":"2024-09-19T07:19:40.386432+0000","last_active":"2024-09-19T07:48:58.373561+0000","last_peered":"2024-09-19T07:48:58.373561+0000","last_clean":"2024-09-19T07:48:58.373561+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:48:58.373561+0000","last_undegraded":"2024-09-19T07:48:58.373561+0000","last_fullsized":"2024-09-19T07:48:58.373561+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27263,"num_write_kb":334786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"61'4385","reported_seq":4500,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371314+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:48:58.371314+0000","last_peered":"2024-09-19T07:48:58.371314+0000","last_clean":"2024-09-19T07:48:58.371314+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:48:58.371314+0000","last_undegraded":"2024-09-19T07:48:58.371314+0000","last_fullsized":"2024-09-19T07:48:58.371314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22838,"num_write_kb":271812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"64'4859","reported_seq":4967,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372828+0000","last_change":"2024-09-19T07:19:40.378630+0000","last_active":"2024-09-19T07:48:58.372828+0000","last_peered":"2024-09-19T07:48:58.372828+0000","last_clean":"2024-09-19T07:48:58.372828+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:48:58.372828+0000","last_undegraded":"2024-09-19T07:48:58.372828+0000","last_fullsized":"2024-09-19T07:48:58.372828+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26164,"num_write_kb":338417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"61'4483","reported_seq":4597,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364173+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:48:58.364173+0000","last_peered":"2024-09-19T07:48:58.364173+0000","last_clean":"2024-09-19T07:48:58.364173+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:48:58.364173+0000","last_undegraded":"2024-09-19T07:48:58.364173+0000","last_fullsized":"2024-09-19T07:48:58.364173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23870,"num_write_kb":292652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"64'4105","reported_seq":4212,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368603+0000","last_change":"2024-09-19T07:19:40.381365+0000","last_active":"2024-09-19T07:48:58.368603+0000","last_peered":"2024-09-19T07:48:58.368603+0000","last_clean":"2024-09-19T07:48:58.368603+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:48:58.368603+0000","last_undegraded":"2024-09-19T07:48:58.368603+0000","last_fullsized":"2024-09-19T07:48:58.368603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21923,"num_write_kb":284540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"61'5078","reported_seq":5191,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374056+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:48:58.374056+0000","last_peered":"2024-09-19T07:48:58.374056+0000","last_clean":"2024-09-19T07:48:58.374056+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:48:58.374056+0000","last_undegraded":"2024-09-19T07:48:58.374056+0000","last_fullsized":"2024-09-19T07:48:58.374056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28059,"num_write_kb":313378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"64'3824","reported_seq":3930,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372461+0000","last_change":"2024-09-19T07:19:40.387116+0000","last_active":"2024-09-19T07:48:58.372461+0000","last_peered":"2024-09-19T07:48:58.372461+0000","last_clean":"2024-09-19T07:48:58.372461+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:48:58.372461+0000","last_undegraded":"2024-09-19T07:48:58.372461+0000","last_fullsized":"2024-09-19T07:48:58.372461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20622,"num_write_kb":295649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"61'4725","reported_seq":4838,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382564+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:48:58.382564+0000","last_peered":"2024-09-19T07:48:58.382564+0000","last_clean":"2024-09-19T07:48:58.382564+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:48:58.382564+0000","last_undegraded":"2024-09-19T07:48:58.382564+0000","last_fullsized":"2024-09-19T07:48:58.382564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25170,"num_write_kb":293477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"64'4764","reported_seq":4871,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380511+0000","last_change":"2024-09-19T07:19:40.357629+0000","last_active":"2024-09-19T07:48:58.380511+0000","last_peered":"2024-09-19T07:48:58.380511+0000","last_clean":"2024-09-19T07:48:58.380511+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:48:58.380511+0000","last_undegraded":"2024-09-19T07:48:58.380511+0000","last_fullsized":"2024-09-19T07:48:58.380511+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25208,"num_write_kb":309995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"61'5411","reported_seq":5525,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364434+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:48:58.364434+0000","last_peered":"2024-09-19T07:48:58.364434+0000","last_clean":"2024-09-19T07:48:58.364434+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:48:58.364434+0000","last_undegraded":"2024-09-19T07:48:58.364434+0000","last_fullsized":"2024-09-19T07:48:58.364434+0000","mapping_epoch":19,"log_start":"54'200","ondisk_log_start":"54'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28760,"num_write_kb":348949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"64'4221","reported_seq":4326,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363476+0000","last_change":"2024-09-19T07:19:40.388597+0000","last_active":"2024-09-19T07:48:58.363476+0000","last_peered":"2024-09-19T07:48:58.363476+0000","last_clean":"2024-09-19T07:48:58.363476+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:48:58.363476+0000","last_undegraded":"2024-09-19T07:48:58.363476+0000","last_fullsized":"2024-09-19T07:48:58.363476+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:34:18.402016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21805,"num_write_kb":288249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"61'4974","reported_seq":5093,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365664+0000","last_change":"2024-09-19T07:20:13.954029+0000","last_active":"2024-09-19T07:48:58.365664+0000","last_peered":"2024-09-19T07:48:58.365664+0000","last_clean":"2024-09-19T07:48:58.365664+0000","last_became_active":"2024-09-19T07:20:13.953118+0000","last_became_peered":"2024-09-19T07:20:13.953118+0000","last_unstale":"2024-09-19T07:48:58.365664+0000","last_undegraded":"2024-09-19T07:48:58.365664+0000","last_fullsized":"2024-09-19T07:48:58.365664+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:27:21.388787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25995,"num_write_kb":309048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"64'5715","reported_seq":5819,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375222+0000","last_change":"2024-09-19T07:19:40.383779+0000","last_active":"2024-09-19T07:48:58.375222+0000","last_peered":"2024-09-19T07:48:58.375222+0000","last_clean":"2024-09-19T07:48:58.375222+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:48:58.375222+0000","last_undegraded":"2024-09-19T07:48:58.375222+0000","last_fullsized":"2024-09-19T07:48:58.375222+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5715,"log_dups_size":0,"ondisk_log_size":5715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30021,"num_write_kb":374301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"61'4405","reported_seq":4518,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362645+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:48:58.362645+0000","last_peered":"2024-09-19T07:48:58.362645+0000","last_clean":"2024-09-19T07:48:58.362645+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:48:58.362645+0000","last_undegraded":"2024-09-19T07:48:58.362645+0000","last_fullsized":"2024-09-19T07:48:58.362645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4405,"log_dups_size":0,"ondisk_log_size":4405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23742,"num_write_kb":275461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"64'3931","reported_seq":4036,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378138+0000","last_change":"2024-09-19T07:19:40.376342+0000","last_active":"2024-09-19T07:48:58.378138+0000","last_peered":"2024-09-19T07:48:58.378138+0000","last_clean":"2024-09-19T07:48:58.378138+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:48:58.378138+0000","last_undegraded":"2024-09-19T07:48:58.378138+0000","last_fullsized":"2024-09-19T07:48:58.378138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3931,"log_dups_size":0,"ondisk_log_size":3931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:22.997744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22002,"num_write_kb":260466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"61'4975","reported_seq":5088,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365882+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:48:58.365882+0000","last_peered":"2024-09-19T07:48:58.365882+0000","last_clean":"2024-09-19T07:48:58.365882+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:48:58.365882+0000","last_undegraded":"2024-09-19T07:48:58.365882+0000","last_fullsized":"2024-09-19T07:48:58.365882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4975,"log_dups_size":0,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26500,"num_write_kb":311194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"64'5003","reported_seq":5108,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362879+0000","last_change":"2024-09-19T07:19:40.361504+0000","last_active":"2024-09-19T07:48:58.362879+0000","last_peered":"2024-09-19T07:48:58.362879+0000","last_clean":"2024-09-19T07:48:58.362879+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:48:58.362879+0000","last_undegraded":"2024-09-19T07:48:58.362879+0000","last_fullsized":"2024-09-19T07:48:58.362879+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:30:31.110669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26890,"num_write_kb":325027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"61'5877","reported_seq":5992,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370513+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:48:58.370513+0000","last_peered":"2024-09-19T07:48:58.370513+0000","last_clean":"2024-09-19T07:48:58.370513+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:48:58.370513+0000","last_undegraded":"2024-09-19T07:48:58.370513+0000","last_fullsized":"2024-09-19T07:48:58.370513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:19.033192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":30970,"num_write_kb":370397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"64'4684","reported_seq":4789,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364347+0000","last_change":"2024-09-19T07:19:40.390205+0000","last_active":"2024-09-19T07:48:58.364347+0000","last_peered":"2024-09-19T07:48:58.364347+0000","last_clean":"2024-09-19T07:48:58.364347+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:48:58.364347+0000","last_undegraded":"2024-09-19T07:48:58.364347+0000","last_fullsized":"2024-09-19T07:48:58.364347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25566,"num_write_kb":331994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"61'4261","reported_seq":4374,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381790+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:48:58.381790+0000","last_peered":"2024-09-19T07:48:58.381790+0000","last_clean":"2024-09-19T07:48:58.381790+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:48:58.381790+0000","last_undegraded":"2024-09-19T07:48:58.381790+0000","last_fullsized":"2024-09-19T07:48:58.381790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22573,"num_write_kb":274878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"64'5240","reported_seq":5347,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363651+0000","last_change":"2024-09-19T07:19:40.390559+0000","last_active":"2024-09-19T07:48:58.363651+0000","last_peered":"2024-09-19T07:48:58.363651+0000","last_clean":"2024-09-19T07:48:58.363651+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:48:58.363651+0000","last_undegraded":"2024-09-19T07:48:58.363651+0000","last_fullsized":"2024-09-19T07:48:58.363651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":28154,"num_write_kb":354961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"61'5095","reported_seq":5208,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379268+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:48:58.379268+0000","last_peered":"2024-09-19T07:48:58.379268+0000","last_clean":"2024-09-19T07:48:58.379268+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:48:58.379268+0000","last_undegraded":"2024-09-19T07:48:58.379268+0000","last_fullsized":"2024-09-19T07:48:58.379268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26748,"num_write_kb":324403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"64'4078","reported_seq":4184,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363850+0000","last_change":"2024-09-19T07:19:40.358433+0000","last_active":"2024-09-19T07:48:58.363850+0000","last_peered":"2024-09-19T07:48:58.363850+0000","last_clean":"2024-09-19T07:48:58.363850+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:48:58.363850+0000","last_undegraded":"2024-09-19T07:48:58.363850+0000","last_fullsized":"2024-09-19T07:48:58.363850+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:10:25.226677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22110,"num_write_kb":293452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"61'4908","reported_seq":5021,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378171+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:48:58.378171+0000","last_peered":"2024-09-19T07:48:58.378171+0000","last_clean":"2024-09-19T07:48:58.378171+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:48:58.378171+0000","last_undegraded":"2024-09-19T07:48:58.378171+0000","last_fullsized":"2024-09-19T07:48:58.378171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:54:51.710997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25574,"num_write_kb":315713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"64'3983","reported_seq":4088,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368849+0000","last_change":"2024-09-19T07:19:40.374374+0000","last_active":"2024-09-19T07:48:58.368849+0000","last_peered":"2024-09-19T07:48:58.368849+0000","last_clean":"2024-09-19T07:48:58.368849+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:48:58.368849+0000","last_undegraded":"2024-09-19T07:48:58.368849+0000","last_fullsized":"2024-09-19T07:48:58.368849+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:34:05.875388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21503,"num_write_kb":274721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"61'5975","reported_seq":6087,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371924+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:48:58.371924+0000","last_peered":"2024-09-19T07:48:58.371924+0000","last_clean":"2024-09-19T07:48:58.371924+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:48:58.371924+0000","last_undegraded":"2024-09-19T07:48:58.371924+0000","last_fullsized":"2024-09-19T07:48:58.371924+0000","mapping_epoch":19,"log_start":"53'200","ondisk_log_start":"53'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31472,"num_write_kb":384213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"64'4930","reported_seq":5034,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373470+0000","last_change":"2024-09-19T07:19:40.387341+0000","last_active":"2024-09-19T07:48:58.373470+0000","last_peered":"2024-09-19T07:48:58.373470+0000","last_clean":"2024-09-19T07:48:58.373470+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:48:58.373470+0000","last_undegraded":"2024-09-19T07:48:58.373470+0000","last_fullsized":"2024-09-19T07:48:58.373470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:56:40.928844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26807,"num_write_kb":336825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"61'5543","reported_seq":5655,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365417+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:48:58.365417+0000","last_peered":"2024-09-19T07:48:58.365417+0000","last_clean":"2024-09-19T07:48:58.365417+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:48:58.365417+0000","last_undegraded":"2024-09-19T07:48:58.365417+0000","last_fullsized":"2024-09-19T07:48:58.365417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5543,"log_dups_size":0,"ondisk_log_size":5543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29751,"num_write_kb":336461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"64'5766","reported_seq":5872,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369027+0000","last_change":"2024-09-19T07:19:40.388484+0000","last_active":"2024-09-19T07:48:58.369027+0000","last_peered":"2024-09-19T07:48:58.369027+0000","last_clean":"2024-09-19T07:48:58.369027+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:48:58.369027+0000","last_undegraded":"2024-09-19T07:48:58.369027+0000","last_fullsized":"2024-09-19T07:48:58.369027+0000","mapping_epoch":23,"log_start":"58'300","ondisk_log_start":"58'300","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5466,"log_dups_size":0,"ondisk_log_size":5466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32274,"num_write_kb":395900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"61'4235","reported_seq":4347,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372319+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:48:58.372319+0000","last_peered":"2024-09-19T07:48:58.372319+0000","last_clean":"2024-09-19T07:48:58.372319+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:48:58.372319+0000","last_undegraded":"2024-09-19T07:48:58.372319+0000","last_fullsized":"2024-09-19T07:48:58.372319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22758,"num_write_kb":275574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"64'3417","reported_seq":3521,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365853+0000","last_change":"2024-09-19T07:19:40.367578+0000","last_active":"2024-09-19T07:48:58.365853+0000","last_peered":"2024-09-19T07:48:58.365853+0000","last_clean":"2024-09-19T07:48:58.365853+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:48:58.365853+0000","last_undegraded":"2024-09-19T07:48:58.365853+0000","last_fullsized":"2024-09-19T07:48:58.365853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:00:25.357854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18157,"num_write_kb":232641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"61'4616","reported_seq":4728,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368993+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:48:58.368993+0000","last_peered":"2024-09-19T07:48:58.368993+0000","last_clean":"2024-09-19T07:48:58.368993+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:48:58.368993+0000","last_undegraded":"2024-09-19T07:48:58.368993+0000","last_fullsized":"2024-09-19T07:48:58.368993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:10:43.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24409,"num_write_kb":298784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"64'4320","reported_seq":4424,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374872+0000","last_change":"2024-09-19T07:19:40.385641+0000","last_active":"2024-09-19T07:48:58.374872+0000","last_peered":"2024-09-19T07:48:58.374872+0000","last_clean":"2024-09-19T07:48:58.374872+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:48:58.374872+0000","last_undegraded":"2024-09-19T07:48:58.374872+0000","last_fullsized":"2024-09-19T07:48:58.374872+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22755,"num_write_kb":287920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"61'4140","reported_seq":4253,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363901+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:48:58.363901+0000","last_peered":"2024-09-19T07:48:58.363901+0000","last_clean":"2024-09-19T07:48:58.363901+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:48:58.363901+0000","last_undegraded":"2024-09-19T07:48:58.363901+0000","last_fullsized":"2024-09-19T07:48:58.363901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22395,"num_write_kb":253944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"64'3972","reported_seq":4076,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364583+0000","last_change":"2024-09-19T07:19:40.367101+0000","last_active":"2024-09-19T07:48:58.364583+0000","last_peered":"2024-09-19T07:48:58.364583+0000","last_clean":"2024-09-19T07:48:58.364583+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:48:58.364583+0000","last_undegraded":"2024-09-19T07:48:58.364583+0000","last_fullsized":"2024-09-19T07:48:58.364583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21876,"num_write_kb":260064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"61'3526","reported_seq":3638,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368962+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:48:58.368962+0000","last_peered":"2024-09-19T07:48:58.368962+0000","last_clean":"2024-09-19T07:48:58.368962+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:48:58.368962+0000","last_undegraded":"2024-09-19T07:48:58.368962+0000","last_fullsized":"2024-09-19T07:48:58.368962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18762,"num_write_kb":235013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"64'3609","reported_seq":3713,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377616+0000","last_change":"2024-09-19T07:19:40.384212+0000","last_active":"2024-09-19T07:48:58.377616+0000","last_peered":"2024-09-19T07:48:58.377616+0000","last_clean":"2024-09-19T07:48:58.377616+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:48:58.377616+0000","last_undegraded":"2024-09-19T07:48:58.377616+0000","last_fullsized":"2024-09-19T07:48:58.377616+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:57.724404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20332,"num_write_kb":258088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"61'4660","reported_seq":4777,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381227+0000","last_change":"2024-09-19T07:20:13.952931+0000","last_active":"2024-09-19T07:48:58.381227+0000","last_peered":"2024-09-19T07:48:58.381227+0000","last_clean":"2024-09-19T07:48:58.381227+0000","last_became_active":"2024-09-19T07:20:13.952103+0000","last_became_peered":"2024-09-19T07:20:13.952103+0000","last_unstale":"2024-09-19T07:48:58.381227+0000","last_undegraded":"2024-09-19T07:48:58.381227+0000","last_fullsized":"2024-09-19T07:48:58.381227+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:23.862116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24375,"num_write_kb":290368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"64'3991","reported_seq":4095,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375705+0000","last_change":"2024-09-19T07:19:40.387401+0000","last_active":"2024-09-19T07:48:58.375705+0000","last_peered":"2024-09-19T07:48:58.375705+0000","last_clean":"2024-09-19T07:48:58.375705+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:48:58.375705+0000","last_undegraded":"2024-09-19T07:48:58.375705+0000","last_fullsized":"2024-09-19T07:48:58.375705+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3991,"log_dups_size":0,"ondisk_log_size":3991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21657,"num_write_kb":287665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"61'5448","reported_seq":5560,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381563+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:48:58.381563+0000","last_peered":"2024-09-19T07:48:58.381563+0000","last_clean":"2024-09-19T07:48:58.381563+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:48:58.381563+0000","last_undegraded":"2024-09-19T07:48:58.381563+0000","last_fullsized":"2024-09-19T07:48:58.381563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29399,"num_write_kb":347457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"64'4539","reported_seq":4643,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365336+0000","last_change":"2024-09-19T07:19:40.361430+0000","last_active":"2024-09-19T07:48:58.365336+0000","last_peered":"2024-09-19T07:48:58.365336+0000","last_clean":"2024-09-19T07:48:58.365336+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:48:58.365336+0000","last_undegraded":"2024-09-19T07:48:58.365336+0000","last_fullsized":"2024-09-19T07:48:58.365336+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4539,"log_dups_size":0,"ondisk_log_size":4539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:23:05.185943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24944,"num_write_kb":313178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"61'6280","reported_seq":6394,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.376418+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:48:58.376418+0000","last_peered":"2024-09-19T07:48:58.376418+0000","last_clean":"2024-09-19T07:48:58.376418+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:48:58.376418+0000","last_undegraded":"2024-09-19T07:48:58.376418+0000","last_fullsized":"2024-09-19T07:48:58.376418+0000","mapping_epoch":19,"log_start":"54'500","ondisk_log_start":"54'500","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":34141,"num_write_kb":395176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"64'5507","reported_seq":5613,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363380+0000","last_change":"2024-09-19T07:19:40.378172+0000","last_active":"2024-09-19T07:48:58.363380+0000","last_peered":"2024-09-19T07:48:58.363380+0000","last_clean":"2024-09-19T07:48:58.363380+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:48:58.363380+0000","last_undegraded":"2024-09-19T07:48:58.363380+0000","last_fullsized":"2024-09-19T07:48:58.363380+0000","mapping_epoch":23,"log_start":"58'300","ondisk_log_start":"58'300","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:52.321790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30086,"num_write_kb":370527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"61'4537","reported_seq":4648,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362562+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:48:58.362562+0000","last_peered":"2024-09-19T07:48:58.362562+0000","last_clean":"2024-09-19T07:48:58.362562+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:48:58.362562+0000","last_undegraded":"2024-09-19T07:48:58.362562+0000","last_fullsized":"2024-09-19T07:48:58.362562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24234,"num_write_kb":310307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"64'4532","reported_seq":4639,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381234+0000","last_change":"2024-09-19T07:19:40.373909+0000","last_active":"2024-09-19T07:48:58.381234+0000","last_peered":"2024-09-19T07:48:58.381234+0000","last_clean":"2024-09-19T07:48:58.381234+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:48:58.381234+0000","last_undegraded":"2024-09-19T07:48:58.381234+0000","last_fullsized":"2024-09-19T07:48:58.381234+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25325,"num_write_kb":306661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"61'4832","reported_seq":4943,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373789+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:48:58.373789+0000","last_peered":"2024-09-19T07:48:58.373789+0000","last_clean":"2024-09-19T07:48:58.373789+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:48:58.373789+0000","last_undegraded":"2024-09-19T07:48:58.373789+0000","last_fullsized":"2024-09-19T07:48:58.373789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26222,"num_write_kb":306099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"64'3448","reported_seq":3555,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371996+0000","last_change":"2024-09-19T07:19:40.396146+0000","last_active":"2024-09-19T07:48:58.371996+0000","last_peered":"2024-09-19T07:48:58.371996+0000","last_clean":"2024-09-19T07:48:58.371996+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:48:58.371996+0000","last_undegraded":"2024-09-19T07:48:58.371996+0000","last_fullsized":"2024-09-19T07:48:58.371996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3448,"log_dups_size":0,"ondisk_log_size":3448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18660,"num_write_kb":230569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"61'4317","reported_seq":4428,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365728+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:48:58.365728+0000","last_peered":"2024-09-19T07:48:58.365728+0000","last_clean":"2024-09-19T07:48:58.365728+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:48:58.365728+0000","last_undegraded":"2024-09-19T07:48:58.365728+0000","last_fullsized":"2024-09-19T07:48:58.365728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22609,"num_write_kb":267014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"64'3912","reported_seq":4019,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367632+0000","last_change":"2024-09-19T07:19:40.373552+0000","last_active":"2024-09-19T07:48:58.367632+0000","last_peered":"2024-09-19T07:48:58.367632+0000","last_clean":"2024-09-19T07:48:58.367632+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:48:58.367632+0000","last_undegraded":"2024-09-19T07:48:58.367632+0000","last_fullsized":"2024-09-19T07:48:58.367632+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:23:33.629831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21554,"num_write_kb":270790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"61'4720","reported_seq":4831,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373254+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:48:58.373254+0000","last_peered":"2024-09-19T07:48:58.373254+0000","last_clean":"2024-09-19T07:48:58.373254+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:48:58.373254+0000","last_undegraded":"2024-09-19T07:48:58.373254+0000","last_fullsized":"2024-09-19T07:48:58.373254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24228,"num_write_kb":289919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"64'3227","reported_seq":3330,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362718+0000","last_change":"2024-09-19T07:19:40.356989+0000","last_active":"2024-09-19T07:48:58.362718+0000","last_peered":"2024-09-19T07:48:58.362718+0000","last_clean":"2024-09-19T07:48:58.362718+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:48:58.362718+0000","last_undegraded":"2024-09-19T07:48:58.362718+0000","last_fullsized":"2024-09-19T07:48:58.362718+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17253,"num_write_kb":237102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"61'4326","reported_seq":4441,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366880+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:48:58.366880+0000","last_peered":"2024-09-19T07:48:58.366880+0000","last_clean":"2024-09-19T07:48:58.366880+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:48:58.366880+0000","last_undegraded":"2024-09-19T07:48:58.366880+0000","last_fullsized":"2024-09-19T07:48:58.366880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23399,"num_write_kb":274225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"64'3524","reported_seq":3627,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377515+0000","last_change":"2024-09-19T07:19:40.376487+0000","last_active":"2024-09-19T07:48:58.377515+0000","last_peered":"2024-09-19T07:48:58.377515+0000","last_clean":"2024-09-19T07:48:58.377515+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:48:58.377515+0000","last_undegraded":"2024-09-19T07:48:58.377515+0000","last_fullsized":"2024-09-19T07:48:58.377515+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:05.185451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18216,"num_write_kb":261619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"61'4656","reported_seq":4771,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369817+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:48:58.369817+0000","last_peered":"2024-09-19T07:48:58.369817+0000","last_clean":"2024-09-19T07:48:58.369817+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:48:58.369817+0000","last_undegraded":"2024-09-19T07:48:58.369817+0000","last_fullsized":"2024-09-19T07:48:58.369817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:25.667168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24790,"num_write_kb":287946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"64'4503","reported_seq":4606,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372440+0000","last_change":"2024-09-19T07:19:40.377604+0000","last_active":"2024-09-19T07:48:58.372440+0000","last_peered":"2024-09-19T07:48:58.372440+0000","last_clean":"2024-09-19T07:48:58.372440+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:48:58.372440+0000","last_undegraded":"2024-09-19T07:48:58.372440+0000","last_fullsized":"2024-09-19T07:48:58.372440+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24213,"num_write_kb":310059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"61'5949","reported_seq":6063,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373642+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:48:58.373642+0000","last_peered":"2024-09-19T07:48:58.373642+0000","last_clean":"2024-09-19T07:48:58.373642+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:48:58.373642+0000","last_undegraded":"2024-09-19T07:48:58.373642+0000","last_fullsized":"2024-09-19T07:48:58.373642+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5849,"log_dups_size":0,"ondisk_log_size":5849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31740,"num_write_kb":364971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"64'4360","reported_seq":4463,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372581+0000","last_change":"2024-09-19T07:19:40.377580+0000","last_active":"2024-09-19T07:48:58.372581+0000","last_peered":"2024-09-19T07:48:58.372581+0000","last_clean":"2024-09-19T07:48:58.372581+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:48:58.372581+0000","last_undegraded":"2024-09-19T07:48:58.372581+0000","last_fullsized":"2024-09-19T07:48:58.372581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4360,"log_dups_size":0,"ondisk_log_size":4360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23533,"num_write_kb":311499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"61'4830","reported_seq":4945,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375882+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:48:58.375882+0000","last_peered":"2024-09-19T07:48:58.375882+0000","last_clean":"2024-09-19T07:48:58.375882+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:48:58.375882+0000","last_undegraded":"2024-09-19T07:48:58.375882+0000","last_fullsized":"2024-09-19T07:48:58.375882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26589,"num_write_kb":315697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1521393,"num_write_kb":19214666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279640,"ondisk_log_size":279640,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19627747,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1449,"num_read_kb":190539,"num_write":5111,"num_write_kb":21283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39313408,"data_stored":39255494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4608,"ondisk_log_size":4608,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":0,"num_write":1651322,"num_write_kb":19736802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307839,"ondisk_log_size":307839,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":19579163,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1454,"num_read_kb":221531,"num_write":5068,"num_write_kb":21268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39223296,"data_stored":39158326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4624,"ondisk_log_size":4624,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542507,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1114512,"kb_used_data":13696,"kb_used_omap":1,"kb_used_meta":1100798,"kb_avail":93257328,"statfs":{"total":96636764160,"available":95495503872,"internally_reserved":0,"allocated":14024704,"data_stored":13620504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1127217610},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542508,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766708,"kb_used_data":11684,"kb_used_omap":1,"kb_used_meta":755006,"kb_avail":93605132,"statfs":{"total":96636764160,"available":95851655168,"internally_reserved":0,"allocated":11964416,"data_stored":11552279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":773126598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542505,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1069796,"kb_used_data":5012,"kb_used_omap":1,"kb_used_meta":1064766,"kb_avail":93302044,"statfs":{"total":96636764160,"available":95541293056,"internally_reserved":0,"allocated":5132288,"data_stored":4726264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1090320837},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542508,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1229016,"kb_used_data":14472,"kb_used_omap":1,"kb_used_meta":1214526,"kb_avail":93142824,"statfs":{"total":96636764160,"available":95378251776,"internally_reserved":0,"allocated":14819328,"data_stored":14400282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1243675077},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542506,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1080696,"kb_used_data":20008,"kb_used_omap":1,"kb_used_meta":1060670,"kb_avail":93291144,"statfs":{"total":96636764160,"available":95530131456,"internally_reserved":0,"allocated":20488192,"data_stored":20074195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1086126535},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542508,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688028,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":687102,"kb_avail":93683812,"statfs":{"total":96636764160,"available":95932223488,"internally_reserved":0,"allocated":929792,"data_stored":531268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":703592902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542507,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":891928,"kb_used_data":9032,"kb_used_omap":1,"kb_used_meta":882878,"kb_avail":93479912,"statfs":{"total":96636764160,"available":95723429888,"internally_reserved":0,"allocated":9248768,"data_stored":8836014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904067525},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542506,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":907476,"kb_used_data":9604,"kb_used_omap":1,"kb_used_meta":897854,"kb_avail":93464364,"statfs":{"total":96636764160,"available":95707508736,"internally_reserved":0,"allocated":9834496,"data_stored":9426166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919402949},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":4194942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":84646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8482816,"data_stored":8473926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6926336,"data_stored":6910261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6922240,"data_stored":6910807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11165696,"data_stored":11153717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7053312,"data_stored":7043469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4279622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:49:17.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:49:18.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2fd8b6640 1 Processor -- start 2024-09-19T07:49:18.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2fd8b6640 1 -- start start 2024-09-19T07:49:18.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2fd8b6640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff2f80c4990 0x7ff2f80728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:18.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2fd8b6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2f80c2f90 con 0x7ff2f8150310 2024-09-19T07:49:18.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2fd8b6640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2f80c3210 con 0x7ff2f806cef0 2024-09-19T07:49:18.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2fd8b6640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff2f80c3e20 con 0x7ff2f80c4990 2024-09-19T07:49:18.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f67fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff2f8150310 0x7ff2f8156070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60306/0 (socket says 172.21.15.83:60306) 2024-09-19T07:49:18.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f67fc640 1 -- 172.21.15.83:0/947262721 learned_addr learned my addr 172.21.15.83:0/947262721 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:18.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f77fe640 1 --2- 172.21.15.83:0/947262721 >> v2:172.21.15.184:3300/0 conn(0x7ff2f80c4990 0x7ff2f80728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545849941 0 0) 0x7ff2e0001120 con 0x7ff2f8150310 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2e4003530 con 0x7ff2f8150310 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 412127449 0 0) 0x7ff2e4003530 con 0x7ff2f8150310 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 >> v2:172.21.15.184:3300/0 conn(0x7ff2f80c4990 msgr2=0x7ff2f80728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 --2- 172.21.15.83:0/947262721 >> v2:172.21.15.184:3300/0 conn(0x7ff2f80c4990 0x7ff2f80728d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 >> v1:172.21.15.184:6789/0 conn(0x7ff2f806cef0 legacy=0x7ff2f814fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f80c34f0 con 0x7ff2f8150310 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7ff2e0001850 con 0x7ff2f8150310 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 >> v1:172.21.15.83:6789/0 conn(0x7ff2f8150310 legacy=0x7ff2f8156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 --2- 172.21.15.83:0/947262721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2e40039c0 0x7ff2e4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 --2- 172.21.15.83:0/947262721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2e4004810 0x7ff2e4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 --2- 172.21.15.83:0/947262721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2e4005220 0x7ff2e400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f80c3e20 con 0x7ff2e40039c0 2024-09-19T07:49:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e400c150 con 0x7ff2e4005220 2024-09-19T07:49:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff2e400c350 con 0x7ff2e4004810 2024-09-19T07:49:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f6ffd640 1 --2- 172.21.15.83:0/947262721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2e40039c0 0x7ff2e4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f67fc640 1 --2- 172.21.15.83:0/947262721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2e4004810 0x7ff2e4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f77fe640 1 --2- 172.21.15.83:0/947262721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2e4005220 0x7ff2e400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f6ffd640 1 -- 172.21.15.83:0/947262721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2e4004810 msgr2=0x7ff2e4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f6ffd640 1 --2- 172.21.15.83:0/947262721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2e4004810 0x7ff2e4004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f6ffd640 1 -- 172.21.15.83:0/947262721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2e4005220 msgr2=0x7ff2e400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f6ffd640 1 --2- 172.21.15.83:0/947262721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2e4005220 0x7ff2e400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f6ffd640 1 -- 172.21.15.83:0/947262721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e4003530 con 0x7ff2e40039c0 2024-09-19T07:49:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f77fe640 1 --2- 172.21.15.83:0/947262721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2e4005220 0x7ff2e400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.120+0000 7ff2f77fe640 1 -- 172.21.15.83:0/947262721 reap_dead start 2024-09-19T07:49:18.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f6ffd640 1 --2- 172.21.15.83:0/947262721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2e40039c0 0x7ff2e4003e00 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8001e70 tx=0x7ff2e801c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469b4ff762d1d16b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:18.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8001a90 con 0x7ff2e40039c0 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2e8003b80 con 0x7ff2e40039c0 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f5ffb640 1 -- 172.21.15.83:0/947262721 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8005080 con 0x7ff2e40039c0 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/947262721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2e40039c0 msgr2=0x7ff2e4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 --2- 172.21.15.83:0/947262721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2e40039c0 0x7ff2e4003e00 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8001e70 tx=0x7ff2e801c760 comp rx=0 tx=0).stop 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/947262721 shutdown_connections 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 --2- 172.21.15.83:0/947262721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2e40039c0 0x7ff2e4003e00 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/947262721 >> 172.21.15.83:0/947262721 conn(0x7ff2f80c5090 msgr2=0x7ff2f80c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/947262721 shutdown_connections 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/947262721 wait complete. 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 Processor -- start 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 -- start start 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2e40039c0 0x7ff2f80adc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2f8154a90 0x7ff2f814be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2f814ca10 0x7ff2f8154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f8150190 con 0x7ff2f8154a90 2024-09-19T07:49:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f8150010 con 0x7ff2e40039c0 2024-09-19T07:49:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2fd8b6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff2f806d2d0 con 0x7ff2f814ca10 2024-09-19T07:49:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2f8154a90 0x7ff2f814be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2f8154a90 0x7ff2f814be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42030/0 (socket says 172.21.15.83:42030) 2024-09-19T07:49:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f67fc640 1 -- 172.21.15.83:0/4232511902 learned_addr learned my addr 172.21.15.83:0/4232511902 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f6ffd640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2e40039c0 0x7ff2f80adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f77fe640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2f814ca10 0x7ff2f8154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f67fc640 1 -- 172.21.15.83:0/4232511902 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2f814ca10 msgr2=0x7ff2f8154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f67fc640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2f814ca10 0x7ff2f8154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f67fc640 1 -- 172.21.15.83:0/4232511902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2e40039c0 msgr2=0x7ff2f80adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:18.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f67fc640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2e40039c0 0x7ff2f80adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:18.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.124+0000 7ff2f67fc640 1 -- 172.21.15.83:0/4232511902 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f8154f10 con 0x7ff2f8154a90 2024-09-19T07:49:18.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.128+0000 7ff2f67fc640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2f8154a90 0x7ff2f814be60 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7ff2e000d7e0 tx=0x7ff2e000dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:18.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.128+0000 7ff2effff640 1 -- 172.21.15.83:0/4232511902 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e0019070 con 0x7ff2f8154a90 2024-09-19T07:49:18.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.128+0000 7ff2effff640 1 -- 172.21.15.83:0/4232511902 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2e00074e0 con 0x7ff2f8154a90 2024-09-19T07:49:18.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.128+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/4232511902 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2f81eb2e0 con 0x7ff2f8154a90 2024-09-19T07:49:18.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.128+0000 7ff2effff640 1 -- 172.21.15.83:0/4232511902 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e0004a10 con 0x7ff2f8154a90 2024-09-19T07:49:18.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.128+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/4232511902 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2f814ace0 con 0x7ff2f8154a90 2024-09-19T07:49:18.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.128+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/4232511902 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2f8156860 con 0x7ff2f8154a90 2024-09-19T07:49:18.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.128+0000 7ff2effff640 1 -- 172.21.15.83:0/4232511902 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff2e0004bb0 con 0x7ff2f8154a90 2024-09-19T07:49:18.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.128+0000 7ff2effff640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2d80678f0 0x7ff2d8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:18.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.128+0000 7ff2effff640 1 -- 172.21.15.83:0/4232511902 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7ff2e00866f0 con 0x7ff2f8154a90 2024-09-19T07:49:18.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.128+0000 7ff2f6ffd640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2d80678f0 0x7ff2d8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:18.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.128+0000 7ff2f6ffd640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2d80678f0 0x7ff2d8069db0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8005000 tx=0x7ff2e8001520 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.132+0000 7ff2effff640 1 -- 172.21.15.83:0/4232511902 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff2f8154f10 con 0x7ff2f8154a90 2024-09-19T07:49:18.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.296+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/4232511902 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff2f80c56a0 con 0x7ff2f8154a90 2024-09-19T07:49:18.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.296+0000 7ff2effff640 1 -- 172.21.15.83:0/4232511902 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) ==== 74+0+25908 (secure 0 0 0) 0x7ff2f80c2f90 con 0x7ff2f8154a90 2024-09-19T07:49:18.297 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:18.297 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":65,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:48:58.354104+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.f","mappings":[{"from":7,"to":1}]},{"pgid":"3.28","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:6833/2502231963":"2024-09-20T07:48:58.353983+0000","172.21.15.83:6837/1602509240":"2024-09-20T07:48:23.346687+0000","172.21.15.83:6836/1602509240":"2024-09-20T07:48:23.346687+0000","172.21.15.83:6835/119695014":"2024-09-20T07:47:58.342001+0000","172.21.15.184:6833/1207928072":"2024-09-20T07:47:03.329199+0000","172.21.15.83:6836/3490578864":"2024-09-20T07:45:28.310975+0000","172.21.15.83:6835/1429520945":"2024-09-20T07:45:13.307981+0000","172.21.15.83:6834/1429520945":"2024-09-20T07:45:13.307981+0000","172.21.15.184:6833/3311721080":"2024-09-20T07:44:18.290353+0000","172.21.15.184:6832/3311721080":"2024-09-20T07:44:18.290353+0000","172.21.15.83:6837/1990119490":"2024-09-20T07:42:38.260871+0000","172.21.15.83:6836/1990119490":"2024-09-20T07:42:38.260871+0000","172.21.15.83:6834/2776269095":"2024-09-20T07:42:08.252858+0000","172.21.15.184:6833/1335198090":"2024-09-20T07:41:33.245545+0000","172.21.15.184:6832/1335198090":"2024-09-20T07:41:33.245545+0000","172.21.15.83:6835/365493751":"2024-09-20T07:41:08.239813+0000","172.21.15.83:6834/119695014":"2024-09-20T07:47:58.342001+0000","172.21.15.83:6835/104517500":"2024-09-20T07:28:38.045210+0000","172.21.15.83:6837/2326757290":"2024-09-20T07:40:23.230841+0000","172.21.15.184:6835/872719174":"2024-09-20T07:28:08.038207+0000","172.21.15.83:6834/3763485069":"2024-09-20T07:27:18.018593+0000","172.21.15.184:6834/872719174":"2024-09-20T07:28:08.038207+0000","172.21.15.184:6832/1207928072":"2024-09-20T07:47:03.329199+0000","172.21.15.83:6836/455754420":"2024-09-20T07:27:38.022965+0000","172.21.15.83:6835/3763485069":"2024-09-20T07:27:18.018593+0000","172.21.15.184:6833/1773885233":"2024-09-20T07:22:22.945705+0000","172.21.15.184:6835/144714632":"2024-09-20T07:25:36.204686+0000","172.21.15.83:6836/26497755":"2024-09-20T07:25:27.991655+0000","172.21.15.184:6833/2993984999":"2024-09-20T07:32:38.105001+0000","172.21.15.83:6836/4265456507":"2024-09-20T07:22:32.948957+0000","172.21.15.83:6834/1072903448":"2024-09-20T07:33:28.117348+0000","172.21.15.83:6835/3442755673":"2024-09-20T07:24:47.982274+0000","172.21.15.83:6837/3490578864":"2024-09-20T07:45:28.310975+0000","172.21.15.184:6835/149004692":"2024-09-20T07:20:57.928172+0000","172.21.15.184:6834/3841282254":"2024-09-20T07:30:48.076144+0000","172.21.15.184:6832/2502231963":"2024-09-20T07:48:58.353983+0000","172.21.15.83:6836/1229829432":"2024-09-20T07:29:48.061079+0000","172.21.15.83:6834/1819616410":"2024-09-20T07:21:07.930616+0000","172.21.15.83:6835/1819616410":"2024-09-20T07:21:07.930616+0000","172.21.15.184:6834/149004692":"2024-09-20T07:20:57.928172+0000","172.21.15.184:6832/1773885233":"2024-09-20T07:22:22.945705+0000","172.21.15.184:6835/2112478738":"2024-09-20T07:35:28.148108+0000","172.21.15.184:6832/2993984999":"2024-09-20T07:32:38.105001+0000","172.21.15.83:6837/26497755":"2024-09-20T07:25:27.991655+0000","172.21.15.184:6832/2898687304":"2024-09-20T07:34:18.130590+0000","172.21.15.83:6835/2776269095":"2024-09-20T07:42:08.252858+0000","172.21.15.83:6836/1049139637":"2024-09-20T07:23:37.966033+0000","172.21.15.83:6834/3442755673":"2024-09-20T07:24:47.982274+0000","172.21.15.83:6837/4265456507":"2024-09-20T07:22:32.948957+0000","172.21.15.184:6833/2898687304":"2024-09-20T07:34:18.130590+0000","172.21.15.184:6835/4115191178":"2024-09-20T07:23:52.164421+0000","172.21.15.184:6834/4115191178":"2024-09-20T07:23:52.164421+0000","172.21.15.83:6834/1774585869":"2024-09-20T07:23:07.959477+0000","172.21.15.184:6834/144714632":"2024-09-20T07:25:36.204686+0000","172.21.15.83:6834/104517500":"2024-09-20T07:28:38.045210+0000","172.21.15.83:6837/1049139637":"2024-09-20T07:23:37.966033+0000","172.21.15.83:6834/2004814399":"2024-09-20T07:30:48.076245+0000","172.21.15.83:6837/1229829432":"2024-09-20T07:29:48.061079+0000","172.21.15.83:6835/2004814399":"2024-09-20T07:30:48.076245+0000","172.21.15.184:6835/3841282254":"2024-09-20T07:30:48.076144+0000","172.21.15.83:6836/2581473241":"2024-09-20T07:31:18.082843+0000","172.21.15.83:6837/455754420":"2024-09-20T07:27:38.022965+0000","172.21.15.83:6834/1079027718":"2024-09-20T07:35:43.151965+0000","172.21.15.83:6837/2581473241":"2024-09-20T07:31:18.082843+0000","172.21.15.83:6836/2326757290":"2024-09-20T07:40:23.230841+0000","172.21.15.83:6835/1072903448":"2024-09-20T07:33:28.117348+0000","172.21.15.83:6835/1774585869":"2024-09-20T07:23:07.959477+0000","172.21.15.184:6835/1519298917":"2024-09-20T07:37:18.179638+0000","172.21.15.83:6836/1203406606":"2024-09-20T07:34:08.127781+0000","172.21.15.83:6834/365493751":"2024-09-20T07:41:08.239813+0000","172.21.15.83:6837/1203406606":"2024-09-20T07:34:08.127781+0000","172.21.15.83:6835/1079027718":"2024-09-20T07:35:43.151965+0000","172.21.15.184:6834/2112478738":"2024-09-20T07:35:28.148108+0000","172.21.15.184:6834/1519298917":"2024-09-20T07:37:18.179638+0000","172.21.15.83:6836/3707393259":"2024-09-20T07:37:18.179736+0000","172.21.15.83:6837/3707393259":"2024-09-20T07:37:18.179736+0000","172.21.15.83:6834/2168217861":"2024-09-20T07:38:08.197381+0000","172.21.15.83:6835/2168217861":"2024-09-20T07:38:08.197381+0000","172.21.15.184:6832/381468017":"2024-09-20T07:39:03.210538+0000","172.21.15.184:6833/381468017":"2024-09-20T07:39:03.210538+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:49:18.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.300+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/4232511902 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2d80678f0 msgr2=0x7ff2d8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:18.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.300+0000 7ff2fd8b6640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2d80678f0 0x7ff2d8069db0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8005000 tx=0x7ff2e8001520 comp rx=0 tx=0).stop 2024-09-19T07:49:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.300+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/4232511902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2f8154a90 msgr2=0x7ff2f814be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.300+0000 7ff2fd8b6640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2f8154a90 0x7ff2f814be60 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7ff2e000d7e0 tx=0x7ff2e000dca0 comp rx=0 tx=0).stop 2024-09-19T07:49:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.300+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/4232511902 shutdown_connections 2024-09-19T07:49:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.300+0000 7ff2fd8b6640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff2d80678f0 0x7ff2d8069db0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.300+0000 7ff2fd8b6640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2f814ca10 0x7ff2f8154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.300+0000 7ff2fd8b6640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2f8154a90 0x7ff2f814be60 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.300+0000 7ff2fd8b6640 1 --2- 172.21.15.83:0/4232511902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2e40039c0 0x7ff2f80adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.300+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/4232511902 >> 172.21.15.83:0/4232511902 conn(0x7ff2f80c5090 msgr2=0x7ff2f814e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.300+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/4232511902 shutdown_connections 2024-09-19T07:49:18.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:18.300+0000 7ff2fd8b6640 1 -- 172.21.15.83:0/4232511902 wait complete. 2024-09-19T07:49:19.316 INFO:tasks.ceph:Scrubbing osd.0 2024-09-19T07:49:19.316 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-19T07:49:19.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.452+0000 7fb4cbb1a640 1 Processor -- start 2024-09-19T07:49:19.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.452+0000 7fb4cbb1a640 1 -- start start 2024-09-19T07:49:19.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.452+0000 7fb4cbb1a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fb4c40c3e80 0x7fb4c40aa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.452+0000 7fb4cbb1a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4c4153970 con 0x7fb4c40c2ee0 2024-09-19T07:49:19.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.452+0000 7fb4cbb1a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4c40c4970 con 0x7fb4c41533d0 2024-09-19T07:49:19.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.452+0000 7fb4cbb1a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fb4c40aadb0 con 0x7fb4c40c3e80 2024-09-19T07:49:19.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.452+0000 7fb4ca0b7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb4c40c2ee0 0x7fb4c414f9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60308/0 (socket says 172.21.15.83:60308) 2024-09-19T07:49:19.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.452+0000 7fb4ca0b7640 1 -- 172.21.15.83:0/760429662 learned_addr learned my addr 172.21.15.83:0/760429662 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:19.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c98b6640 1 --2- 172.21.15.83:0/760429662 >> v2:172.21.15.184:3300/0 conn(0x7fb4c40c3e80 0x7fb4c40aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4064696477 0 0) 0x7fb4c4153970 con 0x7fb4c40c2ee0 2024-09-19T07:49:19.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4b0003530 con 0x7fb4c40c2ee0 2024-09-19T07:49:19.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 647906375 0 0) 0x7fb4b0003530 con 0x7fb4c40c2ee0 2024-09-19T07:49:19.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 >> v2:172.21.15.184:3300/0 conn(0x7fb4c40c3e80 msgr2=0x7fb4c40aa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:19.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 --2- 172.21.15.83:0/760429662 >> v2:172.21.15.184:3300/0 conn(0x7fb4c40c3e80 0x7fb4c40aa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 >> v1:172.21.15.184:6789/0 conn(0x7fb4c41533d0 legacy=0x7fb4c40c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c40c2120 con 0x7fb4c40c2ee0 2024-09-19T07:49:19.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fb4bc001a00 con 0x7fb4c40c2ee0 2024-09-19T07:49:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 >> v1:172.21.15.83:6789/0 conn(0x7fb4c40c2ee0 legacy=0x7fb4c414f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 --2- 172.21.15.83:0/760429662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b00039c0 0x7fb4b0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 --2- 172.21.15.83:0/760429662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4b0004810 0x7fb4b0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 --2- 172.21.15.83:0/760429662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b00051d0 0x7fb4b000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c40aadb0 con 0x7fb4b00051d0 2024-09-19T07:49:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b000c170 con 0x7fb4b00039c0 2024-09-19T07:49:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb4b000c370 con 0x7fb4b0004810 2024-09-19T07:49:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4ca0b7640 1 --2- 172.21.15.83:0/760429662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b00051d0 0x7fb4b000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c98b6640 1 --2- 172.21.15.83:0/760429662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b00039c0 0x7fb4b0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c90b5640 1 --2- 172.21.15.83:0/760429662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4b0004810 0x7fb4b0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4ca0b7640 1 -- 172.21.15.83:0/760429662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4b0004810 msgr2=0x7fb4b0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4ca0b7640 1 --2- 172.21.15.83:0/760429662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4b0004810 0x7fb4b0004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4ca0b7640 1 -- 172.21.15.83:0/760429662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b00039c0 msgr2=0x7fb4b0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:19.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4ca0b7640 1 --2- 172.21.15.83:0/760429662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b00039c0 0x7fb4b0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4ca0b7640 1 -- 172.21.15.83:0/760429662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b0003530 con 0x7fb4b00051d0 2024-09-19T07:49:19.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c98b6640 1 --2- 172.21.15.83:0/760429662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b00039c0 0x7fb4b0003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:19.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4ca0b7640 1 -- 172.21.15.83:0/760429662 reap_dead start 2024-09-19T07:49:19.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4ca0b7640 1 --2- 172.21.15.83:0/760429662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b00051d0 0x7fb4b000b9d0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00a860 tx=0x7fb4bc01fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6637e616271d915 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:19.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc00a480 con 0x7fb4b00051d0 2024-09-19T07:49:19.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4bc0aa510 con 0x7fb4b00051d0 2024-09-19T07:49:19.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4c88b4640 1 -- 172.21.15.83:0/760429662 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc0aa900 con 0x7fb4b00051d0 2024-09-19T07:49:19.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/760429662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b00051d0 msgr2=0x7fb4b000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4cbb1a640 1 --2- 172.21.15.83:0/760429662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b00051d0 0x7fb4b000b9d0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00a860 tx=0x7fb4bc01fae0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/760429662 shutdown_connections 2024-09-19T07:49:19.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4cbb1a640 1 --2- 172.21.15.83:0/760429662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b00051d0 0x7fb4b000b9d0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.456+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/760429662 >> 172.21.15.83:0/760429662 conn(0x7fb4c406bc80 msgr2=0x7fb4c406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:19.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/760429662 shutdown_connections 2024-09-19T07:49:19.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/760429662 wait complete. 2024-09-19T07:49:19.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 Processor -- start 2024-09-19T07:49:19.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 -- start start 2024-09-19T07:49:19.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b00051d0 0x7fb4c41e8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c40c17e0 0x7fb4c41e7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c40bdc70 0x7fb4c40bceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c40c29c0 con 0x7fb4b00051d0 2024-09-19T07:49:19.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c40c26e0 con 0x7fb4c40c17e0 2024-09-19T07:49:19.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb4c40aa870 con 0x7fb4c40bdc70 2024-09-19T07:49:19.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4c98b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b00051d0 0x7fb4c41e8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4c98b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b00051d0 0x7fb4c41e8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42058/0 (socket says 172.21.15.83:42058) 2024-09-19T07:49:19.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4c98b6640 1 -- 172.21.15.83:0/2590116534 learned_addr learned my addr 172.21.15.83:0/2590116534 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:19.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4ca0b7640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c40bdc70 0x7fb4c40bceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4c90b5640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c40c17e0 0x7fb4c41e7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4c98b6640 1 -- 172.21.15.83:0/2590116534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c40bdc70 msgr2=0x7fb4c40bceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4c98b6640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c40bdc70 0x7fb4c40bceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4c98b6640 1 -- 172.21.15.83:0/2590116534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c40c17e0 msgr2=0x7fb4c41e7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4c98b6640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c40c17e0 0x7fb4c41e7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4c98b6640 1 -- 172.21.15.83:0/2590116534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c4155f40 con 0x7fb4b00051d0 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4c90b5640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c40c17e0 0x7fb4c41e7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4c98b6640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b00051d0 0x7fb4c41e8000 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0010930 tx=0x7fb4c0010df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4ba7fc640 1 -- 172.21.15.83:0/2590116534 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c000f040 con 0x7fb4b00051d0 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4ba7fc640 1 -- 172.21.15.83:0/2590116534 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4c0004220 con 0x7fb4b00051d0 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/2590116534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c4156220 con 0x7fb4b00051d0 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4ba7fc640 1 -- 172.21.15.83:0/2590116534 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c000e900 con 0x7fb4b00051d0 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/2590116534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c40bc6b0 con 0x7fb4b00051d0 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.460+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/2590116534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb4c4153830 con 0x7fb4b00051d0 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.464+0000 7fb4ba7fc640 1 -- 172.21.15.83:0/2590116534 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb4c000eaa0 con 0x7fb4b00051d0 2024-09-19T07:49:19.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.464+0000 7fb4ba7fc640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4a80678f0 0x7fb4a8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.464+0000 7fb4ba7fc640 1 -- 172.21.15.83:0/2590116534 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fb4c0087550 con 0x7fb4b00051d0 2024-09-19T07:49:19.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.464+0000 7fb4c90b5640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4a80678f0 0x7fb4a8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.464+0000 7fb4ba7fc640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] conn(0x7fb4a80742c0 0x7fb4a8076740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.464+0000 7fb4ba7fc640 1 -- 172.21.15.83:0/2590116534 --> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb4c001ee50 con 0x7fb4a80742c0 2024-09-19T07:49:19.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.464+0000 7fb4ba7fc640 1 -- 172.21.15.83:0/2590116534 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fb4c00879a0 con 0x7fb4b00051d0 2024-09-19T07:49:19.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.464+0000 7fb4ca0b7640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] conn(0x7fb4a80742c0 0x7fb4a8076740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.464+0000 7fb4c90b5640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4a80678f0 0x7fb4a8069db0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4000c10 tx=0x7fb4b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:19.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.464+0000 7fb4ca0b7640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] conn(0x7fb4a80742c0 0x7fb4a8076740 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:19.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.464+0000 7fb4ba7fc640 1 -- 172.21.15.83:0/2590116534 <== osd.0 v2:172.21.15.83:6808/1414865007 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb4c001ee50 con 0x7fb4a80742c0 2024-09-19T07:49:19.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.476+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/2590116534 --> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb4c40ab0d0 con 0x7fb4a80742c0 2024-09-19T07:49:19.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.476+0000 7fb4ba7fc640 1 -- 172.21.15.83:0/2590116534 <== osd.0 v2:172.21.15.83:6808/1414865007 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb4c40ab0d0 con 0x7fb4a80742c0 2024-09-19T07:49:19.478 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-19T07:49:19.478 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-19T07:49:19.478 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-19T07:49:19.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.476+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] conn(0x7fb4a80742c0 msgr2=0x7fb4a8076740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.476+0000 7fb4cbb1a640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:6808/1414865007,v1:172.21.15.83:6809/1414865007] conn(0x7fb4a80742c0 0x7fb4a8076740 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.480+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4a80678f0 msgr2=0x7fb4a8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.480+0000 7fb4cbb1a640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb4a80678f0 0x7fb4a8069db0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4000c10 tx=0x7fb4b4009040 comp rx=0 tx=0).stop 2024-09-19T07:49:19.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.480+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b00051d0 msgr2=0x7fb4c41e8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.480+0000 7fb4cbb1a640 1 --2- 172.21.15.83:0/2590116534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4b00051d0 0x7fb4c41e8000 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0010930 tx=0x7fb4c0010df0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.480+0000 7fb4ca0b7640 1 -- 172.21.15.83:0/2590116534 reap_dead start 2024-09-19T07:49:19.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.480+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/2590116534 shutdown_connections 2024-09-19T07:49:19.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.480+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/2590116534 >> 172.21.15.83:0/2590116534 conn(0x7fb4c406bc80 msgr2=0x7fb4c4152c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:19.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.480+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/2590116534 shutdown_connections 2024-09-19T07:49:19.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.480+0000 7fb4cbb1a640 1 -- 172.21.15.83:0/2590116534 wait complete. 2024-09-19T07:49:19.494 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-19T07:49:19.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f743df34640 1 Processor -- start 2024-09-19T07:49:19.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f743df34640 1 -- start start 2024-09-19T07:49:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f743df34640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74380c21e0 0x7f74380718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f743df34640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74380c2bb0 0x7f7438150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f743df34640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f74381513f0 0x7f74381571b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f743df34640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7438153cd0 con 0x7f74380c2bb0 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f743df34640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f7438151860 con 0x7f74381513f0 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f743df34640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7438153f40 con 0x7f74380c21e0 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f7436ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74380c2bb0 0x7f7438150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f7436ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74380c2bb0 0x7f7438150eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42060/0 (socket says 172.21.15.83:42060) 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f7436ffd640 1 -- 172.21.15.83:0/3367785561 learned_addr learned my addr 172.21.15.83:0/3367785561 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f7436ffd640 1 -- 172.21.15.83:0/3367785561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74380c21e0 msgr2=0x7f74380718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f7437fff640 1 --2- 172.21.15.83:0/3367785561 >> v2:172.21.15.184:3300/0 conn(0x7f74381513f0 0x7f74381571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f7436ffd640 1 --2- 172.21.15.83:0/3367785561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74380c21e0 0x7f74380718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f7436ffd640 1 -- 172.21.15.83:0/3367785561 >> v2:172.21.15.184:3300/0 conn(0x7f74381513f0 msgr2=0x7f74381571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f7436ffd640 1 --2- 172.21.15.83:0/3367785561 >> v2:172.21.15.184:3300/0 conn(0x7f74381513f0 0x7f74381571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f7436ffd640 1 -- 172.21.15.83:0/3367785561 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7438154220 con 0x7f74380c2bb0 2024-09-19T07:49:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.608+0000 7f7437fff640 1 --2- 172.21.15.83:0/3367785561 >> v2:172.21.15.184:3300/0 conn(0x7f74381513f0 0x7f74381571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f7436ffd640 1 --2- 172.21.15.83:0/3367785561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380c2bb0 0x7f7438150eb0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f74380c4dd0 tx=0x7f742801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44326992029986c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f74367fc640 1 -- 172.21.15.83:0/3367785561 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74280a5750 con 0x7f74380c2bb0 2024-09-19T07:49:19.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f74367fc640 1 -- 172.21.15.83:0/3367785561 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74280a58f0 con 0x7f74380c2bb0 2024-09-19T07:49:19.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f74367fc640 1 -- 172.21.15.83:0/3367785561 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74280a5cc0 con 0x7f74380c2bb0 2024-09-19T07:49:19.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f743df34640 1 -- 172.21.15.83:0/3367785561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380c2bb0 msgr2=0x7f7438150eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f743df34640 1 --2- 172.21.15.83:0/3367785561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380c2bb0 0x7f7438150eb0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f74380c4dd0 tx=0x7f742801c9c0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f743df34640 1 -- 172.21.15.83:0/3367785561 shutdown_connections 2024-09-19T07:49:19.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f743df34640 1 --2- 172.21.15.83:0/3367785561 >> v2:172.21.15.184:3300/0 conn(0x7f74381513f0 0x7f74381571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f743df34640 1 --2- 172.21.15.83:0/3367785561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380c2bb0 0x7f7438150eb0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f743df34640 1 --2- 172.21.15.83:0/3367785561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74380c21e0 0x7f74380718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f743df34640 1 -- 172.21.15.83:0/3367785561 >> 172.21.15.83:0/3367785561 conn(0x7f743806bc80 msgr2=0x7f743806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:19.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f743df34640 1 -- 172.21.15.83:0/3367785561 shutdown_connections 2024-09-19T07:49:19.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f743df34640 1 -- 172.21.15.83:0/3367785561 wait complete. 2024-09-19T07:49:19.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f743df34640 1 Processor -- start 2024-09-19T07:49:19.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.612+0000 7f743df34640 1 -- start start 2024-09-19T07:49:19.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f743df34640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380c21e0 0x7f7438156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f743df34640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74380c2bb0 0x7f74380c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f743df34640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74381513f0 0x7f743814b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f743df34640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f74380a9d30 con 0x7f74380c21e0 2024-09-19T07:49:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f743df34640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7438154a20 con 0x7f74381513f0 2024-09-19T07:49:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f743df34640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f74380a9eb0 con 0x7f74380c2bb0 2024-09-19T07:49:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f74377fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380c21e0 0x7f7438156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f74377fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380c21e0 0x7f7438156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42076/0 (socket says 172.21.15.83:42076) 2024-09-19T07:49:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f74377fe640 1 -- 172.21.15.83:0/3465927967 learned_addr learned my addr 172.21.15.83:0/3465927967 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f7436ffd640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74380c2bb0 0x7f74380c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f7437fff640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74381513f0 0x7f743814b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f74377fe640 1 -- 172.21.15.83:0/3465927967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74380c2bb0 msgr2=0x7f74380c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f74377fe640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74380c2bb0 0x7f74380c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f74377fe640 1 -- 172.21.15.83:0/3465927967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74381513f0 msgr2=0x7f743814b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f74377fe640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74381513f0 0x7f743814b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f74377fe640 1 -- 172.21.15.83:0/3465927967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7438155bd0 con 0x7f74380c21e0 2024-09-19T07:49:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f7437fff640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74381513f0 0x7f743814b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f74377fe640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380c21e0 0x7f7438156460 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f742000b440 tx=0x7f742000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f742ffff640 1 -- 172.21.15.83:0/3465927967 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7420015020 con 0x7f74380c21e0 2024-09-19T07:49:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f742ffff640 1 -- 172.21.15.83:0/3465927967 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74200118b0 con 0x7f74380c21e0 2024-09-19T07:49:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f742ffff640 1 -- 172.21.15.83:0/3465927967 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7420011ca0 con 0x7f74380c21e0 2024-09-19T07:49:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f743df34640 1 -- 172.21.15.83:0/3465927967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7438155eb0 con 0x7f74380c21e0 2024-09-19T07:49:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f743df34640 1 -- 172.21.15.83:0/3465927967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74381572e0 con 0x7f74380c21e0 2024-09-19T07:49:19.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f743df34640 1 -- 172.21.15.83:0/3465927967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74380c25c0 con 0x7f74380c21e0 2024-09-19T07:49:19.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f742ffff640 1 -- 172.21.15.83:0/3465927967 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7420018020 con 0x7f74380c21e0 2024-09-19T07:49:19.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f742ffff640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f74180678c0 0x7f7418069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f742ffff640 1 -- 172.21.15.83:0/3465927967 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f7420086ed0 con 0x7f74380c21e0 2024-09-19T07:49:19.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.616+0000 7f7436ffd640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f74180678c0 0x7f7418069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.620+0000 7f7436ffd640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f74180678c0 0x7f7418069d80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f74280ae000 tx=0x7f74280a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:19.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.620+0000 7f742ffff640 1 -- 172.21.15.83:0/3465927967 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f742001f020 con 0x7f74380c21e0 2024-09-19T07:49:19.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.780+0000 7f743df34640 1 -- 172.21.15.83:0/3465927967 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f7438156080 con 0x7f74180678c0 2024-09-19T07:49:19.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f742ffff640 1 -- 172.21.15.83:0/3465927967 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7438156080 con 0x7f74180678c0 2024-09-19T07:49:19.782 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 0 to deep-scrub 2024-09-19T07:49:19.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f743df34640 1 -- 172.21.15.83:0/3465927967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f74180678c0 msgr2=0x7f7418069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f743df34640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f74180678c0 0x7f7418069d80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f74280ae000 tx=0x7f74280a7070 comp rx=0 tx=0).stop 2024-09-19T07:49:19.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f743df34640 1 -- 172.21.15.83:0/3465927967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380c21e0 msgr2=0x7f7438156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f743df34640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380c21e0 0x7f7438156460 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f742000b440 tx=0x7f742000b900 comp rx=0 tx=0).stop 2024-09-19T07:49:19.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f743df34640 1 -- 172.21.15.83:0/3465927967 shutdown_connections 2024-09-19T07:49:19.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f743df34640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f74180678c0 0x7f7418069d80 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f743df34640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74381513f0 0x7f743814b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f743df34640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74380c2bb0 0x7f74380c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f743df34640 1 --2- 172.21.15.83:0/3465927967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380c21e0 0x7f7438156460 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f743df34640 1 -- 172.21.15.83:0/3465927967 >> 172.21.15.83:0/3465927967 conn(0x7f743806bc80 msgr2=0x7f743806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:19.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f743df34640 1 -- 172.21.15.83:0/3465927967 shutdown_connections 2024-09-19T07:49:19.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.784+0000 7f743df34640 1 -- 172.21.15.83:0/3465927967 wait complete. 2024-09-19T07:49:19.800 INFO:tasks.ceph:Scrubbing osd.1 2024-09-19T07:49:19.800 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-19T07:49:19.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f637023a640 1 Processor -- start 2024-09-19T07:49:19.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f637023a640 1 -- start start 2024-09-19T07:49:19.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f637023a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63680c3ab0 0x7f6368153300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f637023a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6368068410 con 0x7f6368153840 2024-09-19T07:49:19.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f637023a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63680686d0 con 0x7f636806fcb0 2024-09-19T07:49:19.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f637023a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6368073080 con 0x7f63680c3ab0 2024-09-19T07:49:19.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636d7d5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6368153840 0x7f63680bced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60312/0 (socket says 172.21.15.83:60312) 2024-09-19T07:49:19.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636d7d5640 1 -- 172.21.15.83:0/1803434417 learned_addr learned my addr 172.21.15.83:0/1803434417 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:19.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3656807623 0 0) 0x7f6368068410 con 0x7f6368153840 2024-09-19T07:49:19.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636dfd6640 1 --2- 172.21.15.83:0/1803434417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63680c3ab0 0x7f6368153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6354003530 con 0x7f6368153840 2024-09-19T07:49:19.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2169448727 0 0) 0x7f6354003530 con 0x7f6368153840 2024-09-19T07:49:19.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63680c3ab0 msgr2=0x7f6368153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:19.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 --2- 172.21.15.83:0/1803434417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63680c3ab0 0x7f6368153300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 >> v1:172.21.15.184:6789/0 conn(0x7f636806fcb0 legacy=0x7f63680bd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63680beb20 con 0x7f6368153840 2024-09-19T07:49:19.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f6358001680 con 0x7f6368153840 2024-09-19T07:49:19.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 >> v1:172.21.15.83:6789/0 conn(0x7f6368153840 legacy=0x7f63680bced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 --2- 172.21.15.83:0/1803434417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63540039f0 0x7f6354003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 --2- 172.21.15.83:0/1803434417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63680c3ab0 0x7f6354004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 --2- 172.21.15.83:0/1803434417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6354004d50 0x7f63540051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6368073080 con 0x7f6354004d50 2024-09-19T07:49:19.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f635400bbb0 con 0x7f63540039f0 2024-09-19T07:49:19.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f635400bdb0 con 0x7f63680c3ab0 2024-09-19T07:49:19.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636e7d7640 1 --2- 172.21.15.83:0/1803434417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6354004d50 0x7f63540051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636dfd6640 1 --2- 172.21.15.83:0/1803434417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63540039f0 0x7f6354003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636d7d5640 1 --2- 172.21.15.83:0/1803434417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63680c3ab0 0x7f6354004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636e7d7640 1 -- 172.21.15.83:0/1803434417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63680c3ab0 msgr2=0x7f6354004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:19.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636e7d7640 1 --2- 172.21.15.83:0/1803434417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63680c3ab0 0x7f6354004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636e7d7640 1 -- 172.21.15.83:0/1803434417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63540039f0 msgr2=0x7f6354003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:19.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636e7d7640 1 --2- 172.21.15.83:0/1803434417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63540039f0 0x7f6354003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.916+0000 7f636e7d7640 1 -- 172.21.15.83:0/1803434417 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6354003530 con 0x7f6354004d50 2024-09-19T07:49:19.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f636e7d7640 1 --2- 172.21.15.83:0/1803434417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6354004d50 0x7f63540051d0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f6364001750 tx=0x7f636401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ab5ea782fa6c43 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:19.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63640013a0 con 0x7f6354004d50 2024-09-19T07:49:19.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63640052f0 con 0x7f6354004d50 2024-09-19T07:49:19.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f636cfd4640 1 -- 172.21.15.83:0/1803434417 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6364003840 con 0x7f6354004d50 2024-09-19T07:49:19.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 -- 172.21.15.83:0/1803434417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6354004d50 msgr2=0x7f63540051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 --2- 172.21.15.83:0/1803434417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6354004d50 0x7f63540051d0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f6364001750 tx=0x7f636401c3e0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f636e7d7640 1 -- 172.21.15.83:0/1803434417 reap_dead start 2024-09-19T07:49:19.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 -- 172.21.15.83:0/1803434417 shutdown_connections 2024-09-19T07:49:19.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 -- 172.21.15.83:0/1803434417 >> 172.21.15.83:0/1803434417 conn(0x7f63680c4940 msgr2=0x7f63680c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:19.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 -- 172.21.15.83:0/1803434417 shutdown_connections 2024-09-19T07:49:19.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 -- 172.21.15.83:0/1803434417 wait complete. 2024-09-19T07:49:19.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 Processor -- start 2024-09-19T07:49:19.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 -- start start 2024-09-19T07:49:19.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6354004d50 0x7f63681e02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63680ad770 0x7f63681e0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63681e17d0 0x7f6368155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f63680c3e90 con 0x7f63681e17d0 2024-09-19T07:49:19.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f63680c43b0 con 0x7f63680ad770 2024-09-19T07:49:19.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f637023a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f63680c4010 con 0x7f6354004d50 2024-09-19T07:49:19.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f636e7d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63681e17d0 0x7f6368155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f636e7d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63681e17d0 0x7f6368155f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42088/0 (socket says 172.21.15.83:42088) 2024-09-19T07:49:19.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f636e7d7640 1 -- 172.21.15.83:0/1070026153 learned_addr learned my addr 172.21.15.83:0/1070026153 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:19.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f636dfd6640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6354004d50 0x7f63681e02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.920+0000 7f636d7d5640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63680ad770 0x7f63681e0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f636e7d7640 1 -- 172.21.15.83:0/1070026153 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6354004d50 msgr2=0x7f63681e02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f636e7d7640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6354004d50 0x7f63681e02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f636e7d7640 1 -- 172.21.15.83:0/1070026153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63680ad770 msgr2=0x7f63681e0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f636e7d7640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63680ad770 0x7f63681e0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f636e7d7640 1 -- 172.21.15.83:0/1070026153 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63681dfe70 con 0x7f63681e17d0 2024-09-19T07:49:19.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f636d7d5640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63680ad770 0x7f63681e0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:19.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f636dfd6640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6354004d50 0x7f63681e02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:19.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f636e7d7640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63681e17d0 0x7f6368155f70 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f6364001750 tx=0x7f6354005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:19.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f635e7fc640 1 -- 172.21.15.83:0/1070026153 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63640013a0 con 0x7f63681e17d0 2024-09-19T07:49:19.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f635e7fc640 1 -- 172.21.15.83:0/1070026153 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63640054a0 con 0x7f63681e17d0 2024-09-19T07:49:19.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f637023a640 1 -- 172.21.15.83:0/1070026153 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63681dfbd0 con 0x7f63681e17d0 2024-09-19T07:49:19.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f635e7fc640 1 -- 172.21.15.83:0/1070026153 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6364007a40 con 0x7f63681e17d0 2024-09-19T07:49:19.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f637023a640 1 -- 172.21.15.83:0/1070026153 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63681513b0 con 0x7f63681e17d0 2024-09-19T07:49:19.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f637023a640 1 -- 172.21.15.83:0/1070026153 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f63680c7580 con 0x7f63681e17d0 2024-09-19T07:49:19.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f635e7fc640 1 -- 172.21.15.83:0/1070026153 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f63640bd020 con 0x7f63681e17d0 2024-09-19T07:49:19.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f635e7fc640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f634c0678c0 0x7f634c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f635e7fc640 1 -- 172.21.15.83:0/1070026153 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f6364121550 con 0x7f63681e17d0 2024-09-19T07:49:19.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f636dfd6640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f634c0678c0 0x7f634c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f635e7fc640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] conn(0x7f634c0742a0 0x7f634c076720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:19.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f635e7fc640 1 -- 172.21.15.83:0/1070026153 --> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6364004e70 con 0x7f634c0742a0 2024-09-19T07:49:19.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f635e7fc640 1 -- 172.21.15.83:0/1070026153 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f63641219a0 con 0x7f63681e17d0 2024-09-19T07:49:19.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f636d7d5640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] conn(0x7f634c0742a0 0x7f634c076720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:19.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f636dfd6640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f634c0678c0 0x7f634c069d80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6360004330 tx=0x7f636000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:19.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f636d7d5640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] conn(0x7f634c0742a0 0x7f634c076720 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:19.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.924+0000 7f635e7fc640 1 -- 172.21.15.83:0/1070026153 <== osd.1 v2:172.21.15.83:6800/2685389983 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6364004e70 con 0x7f634c0742a0 2024-09-19T07:49:19.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f637023a640 1 -- 172.21.15.83:0/1070026153 --> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f63680c5a30 con 0x7f634c0742a0 2024-09-19T07:49:19.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f635e7fc640 1 -- 172.21.15.83:0/1070026153 <== osd.1 v2:172.21.15.83:6800/2685389983 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f63680c5a30 con 0x7f634c0742a0 2024-09-19T07:49:19.939 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-19T07:49:19.939 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-19T07:49:19.939 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-19T07:49:19.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f637023a640 1 -- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] conn(0x7f634c0742a0 msgr2=0x7f634c076720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f637023a640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:6800/2685389983,v1:172.21.15.83:6801/2685389983] conn(0x7f634c0742a0 0x7f634c076720 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:19.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f637023a640 1 -- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f634c0678c0 msgr2=0x7f634c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f637023a640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f634c0678c0 0x7f634c069d80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6360004330 tx=0x7f636000b040 comp rx=0 tx=0).stop 2024-09-19T07:49:19.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f637023a640 1 -- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63681e17d0 msgr2=0x7f6368155f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:19.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f637023a640 1 --2- 172.21.15.83:0/1070026153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63681e17d0 0x7f6368155f70 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f6364001750 tx=0x7f6354005840 comp rx=0 tx=0).stop 2024-09-19T07:49:19.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f636e7d7640 1 -- 172.21.15.83:0/1070026153 reap_dead start 2024-09-19T07:49:19.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f637023a640 1 -- 172.21.15.83:0/1070026153 shutdown_connections 2024-09-19T07:49:19.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f637023a640 1 -- 172.21.15.83:0/1070026153 >> 172.21.15.83:0/1070026153 conn(0x7f63680c4940 msgr2=0x7f63680716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:19.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f637023a640 1 -- 172.21.15.83:0/1070026153 shutdown_connections 2024-09-19T07:49:19.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:19.940+0000 7f637023a640 1 -- 172.21.15.83:0/1070026153 wait complete. 2024-09-19T07:49:19.954 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-19T07:49:20.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb9a0e640 1 Processor -- start 2024-09-19T07:49:20.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb9a0e640 1 -- start start 2024-09-19T07:49:20.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb9a0e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1cb40c21e0 0x7f1cb40718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb9a0e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1cb40c2bb0 0x7f1cb4150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb9a0e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cb4153cd0 con 0x7f1cb41513f0 2024-09-19T07:49:20.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb9a0e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f1cb4151820 con 0x7f1cb40c21e0 2024-09-19T07:49:20.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb9a0e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f1cb4153f40 con 0x7f1cb40c2bb0 2024-09-19T07:49:20.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1cb40c21e0 0x7f1cb40718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1cb40c21e0 0x7f1cb40718b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42098/0 (socket says 172.21.15.83:42098) 2024-09-19T07:49:20.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb2ffd640 1 -- 172.21.15.83:0/167534516 learned_addr learned my addr 172.21.15.83:0/167534516 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:20.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb2ffd640 1 -- 172.21.15.83:0/167534516 >> v2:172.21.15.184:3300/0 conn(0x7f1cb40c2bb0 msgr2=0x7f1cb4150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:20.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb2ffd640 1 --2- 172.21.15.83:0/167534516 >> v2:172.21.15.184:3300/0 conn(0x7f1cb40c2bb0 0x7f1cb4150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb2ffd640 1 -- 172.21.15.83:0/167534516 >> v1:172.21.15.184:6789/0 conn(0x7f1cb41513f0 legacy=0x7f1cb41571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:20.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.064+0000 7f1cb2ffd640 1 -- 172.21.15.83:0/167534516 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb4154220 con 0x7f1cb40c21e0 2024-09-19T07:49:20.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb2ffd640 1 --2- 172.21.15.83:0/167534516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb40c21e0 0x7f1cb40718b0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40c2a20 tx=0x7f1ca801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad2242b09423ba8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb1ffb640 1 -- 172.21.15.83:0/167534516 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca80aa430 con 0x7f1cb40c21e0 2024-09-19T07:49:20.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb1ffb640 1 -- 172.21.15.83:0/167534516 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ca80af070 con 0x7f1cb40c21e0 2024-09-19T07:49:20.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb1ffb640 1 -- 172.21.15.83:0/167534516 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca80aa7f0 con 0x7f1cb40c21e0 2024-09-19T07:49:20.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/167534516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb40c21e0 msgr2=0x7f1cb40718b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 --2- 172.21.15.83:0/167534516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb40c21e0 0x7f1cb40718b0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40c2a20 tx=0x7f1ca801c4b0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/167534516 shutdown_connections 2024-09-19T07:49:20.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 --2- 172.21.15.83:0/167534516 >> v2:172.21.15.184:3300/0 conn(0x7f1cb40c2bb0 0x7f1cb4150eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 --2- 172.21.15.83:0/167534516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb40c21e0 0x7f1cb40718b0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/167534516 >> 172.21.15.83:0/167534516 conn(0x7f1cb406bc80 msgr2=0x7f1cb406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:20.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/167534516 shutdown_connections 2024-09-19T07:49:20.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/167534516 wait complete. 2024-09-19T07:49:20.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 Processor -- start 2024-09-19T07:49:20.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 -- start start 2024-09-19T07:49:20.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1cb40c21e0 0x7f1cb415c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb40c2bb0 0x7f1cb4154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cb41513f0 0x7f1cb4156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb40a9eb0 con 0x7f1cb40c2bb0 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb40a9d30 con 0x7f1cb41513f0 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb9a0e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1cb40c25c0 con 0x7f1cb40c21e0 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb40c2bb0 0x7f1cb4154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb40c2bb0 0x7f1cb4154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42106/0 (socket says 172.21.15.83:42106) 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb27fc640 1 -- 172.21.15.83:0/418695725 learned_addr learned my addr 172.21.15.83:0/418695725 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb27fc640 1 -- 172.21.15.83:0/418695725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1cb40c21e0 msgr2=0x7f1cb415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb27fc640 1 --2- 172.21.15.83:0/418695725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1cb40c21e0 0x7f1cb415c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb27fc640 1 -- 172.21.15.83:0/418695725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cb41513f0 msgr2=0x7f1cb4156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb27fc640 1 --2- 172.21.15.83:0/418695725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cb41513f0 0x7f1cb4156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb27fc640 1 -- 172.21.15.83:0/418695725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb40abfd0 con 0x7f1cb40c2bb0 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1cb27fc640 1 --2- 172.21.15.83:0/418695725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb40c2bb0 0x7f1cb4154fb0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400b440 tx=0x7f1ca400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.068+0000 7f1c9bfff640 1 -- 172.21.15.83:0/418695725 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4015020 con 0x7f1cb40c2bb0 2024-09-19T07:49:20.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.072+0000 7f1c9bfff640 1 -- 172.21.15.83:0/418695725 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ca4011850 con 0x7f1cb40c2bb0 2024-09-19T07:49:20.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.072+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/418695725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb40ab740 con 0x7f1cb40c2bb0 2024-09-19T07:49:20.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.072+0000 7f1c9bfff640 1 -- 172.21.15.83:0/418695725 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4011c40 con 0x7f1cb40c2bb0 2024-09-19T07:49:20.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.072+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/418695725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb41f1db0 con 0x7f1cb40c2bb0 2024-09-19T07:49:20.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.072+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/418695725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cb4155b60 con 0x7f1cb40c2bb0 2024-09-19T07:49:20.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.072+0000 7f1c9bfff640 1 -- 172.21.15.83:0/418695725 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1ca4018020 con 0x7f1cb40c2bb0 2024-09-19T07:49:20.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.072+0000 7f1c9bfff640 1 --2- 172.21.15.83:0/418695725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1ca00678c0 0x7f1ca0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.072+0000 7f1c9bfff640 1 -- 172.21.15.83:0/418695725 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f1ca4086c60 con 0x7f1cb40c2bb0 2024-09-19T07:49:20.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.072+0000 7f1cb2ffd640 1 --2- 172.21.15.83:0/418695725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1ca00678c0 0x7f1ca0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.072+0000 7f1cb2ffd640 1 --2- 172.21.15.83:0/418695725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1ca00678c0 0x7f1ca0069d80 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80ae000 tx=0x7f1ca8004740 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.076+0000 7f1c9bfff640 1 -- 172.21.15.83:0/418695725 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1cb4155b60 con 0x7f1cb40c2bb0 2024-09-19T07:49:20.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/418695725 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f1cb41558b0 con 0x7f1ca00678c0 2024-09-19T07:49:20.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1c9bfff640 1 -- 172.21.15.83:0/418695725 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1cb41558b0 con 0x7f1ca00678c0 2024-09-19T07:49:20.236 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 1 to deep-scrub 2024-09-19T07:49:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/418695725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1ca00678c0 msgr2=0x7f1ca0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1cb9a0e640 1 --2- 172.21.15.83:0/418695725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1ca00678c0 0x7f1ca0069d80 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80ae000 tx=0x7f1ca8004740 comp rx=0 tx=0).stop 2024-09-19T07:49:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/418695725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb40c2bb0 msgr2=0x7f1cb4154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1cb9a0e640 1 --2- 172.21.15.83:0/418695725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb40c2bb0 0x7f1cb4154fb0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400b440 tx=0x7f1ca400b900 comp rx=0 tx=0).stop 2024-09-19T07:49:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/418695725 shutdown_connections 2024-09-19T07:49:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1cb9a0e640 1 --2- 172.21.15.83:0/418695725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1ca00678c0 0x7f1ca0069d80 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1cb9a0e640 1 --2- 172.21.15.83:0/418695725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cb41513f0 0x7f1cb4156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1cb9a0e640 1 --2- 172.21.15.83:0/418695725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb40c2bb0 0x7f1cb4154fb0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1cb9a0e640 1 --2- 172.21.15.83:0/418695725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1cb40c21e0 0x7f1cb415c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/418695725 >> 172.21.15.83:0/418695725 conn(0x7f1cb406bc80 msgr2=0x7f1cb406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:20.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.236+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/418695725 shutdown_connections 2024-09-19T07:49:20.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.240+0000 7f1cb9a0e640 1 -- 172.21.15.83:0/418695725 wait complete. 2024-09-19T07:49:20.254 INFO:tasks.ceph:Scrubbing osd.2 2024-09-19T07:49:20.254 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-19T07:49:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.368+0000 7f35b32e5640 1 Processor -- start 2024-09-19T07:49:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.368+0000 7f35b32e5640 1 -- start start 2024-09-19T07:49:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.368+0000 7f35b32e5640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f35ac0b0d00 0x7f35ac159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.368+0000 7f35b32e5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35ac0c9660 0x7f35ac1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.368+0000 7f35b32e5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35ac0c6180 con 0x7f35ac156910 2024-09-19T07:49:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.368+0000 7f35b32e5640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f35ac0c69b0 con 0x7f35ac0b0d00 2024-09-19T07:49:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.368+0000 7f35b32e5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f35ac15a130 con 0x7f35ac0c9660 2024-09-19T07:49:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.368+0000 7f35b1882640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f35ac156910 0x7f35ac0c8210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35884/0 (socket says 172.21.15.83:35884) 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.368+0000 7f35b1882640 1 -- 172.21.15.83:0/2988360856 learned_addr learned my addr 172.21.15.83:0/2988360856 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b0880640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35ac0c9660 0x7f35ac1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b1081640 1 --2- 172.21.15.83:0/2988360856 >> v2:172.21.15.184:3300/0 conn(0x7f35ac0b0d00 0x7f35ac159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 403103989 0 0) 0x7f35ac0c6180 con 0x7f35ac156910 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3598003530 con 0x7f35ac156910 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 649388214 0 0) 0x7f3598003530 con 0x7f35ac156910 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35ac0c9660 msgr2=0x7f35ac1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35ac0c9660 0x7f35ac1563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 >> v2:172.21.15.184:3300/0 conn(0x7f35ac0b0d00 msgr2=0x7f35ac159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 --2- 172.21.15.83:0/2988360856 >> v2:172.21.15.184:3300/0 conn(0x7f35ac0b0d00 0x7f35ac159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35ac155bf0 con 0x7f35ac156910 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f35a4001a00 con 0x7f35ac156910 2024-09-19T07:49:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 >> v1:172.21.15.83:6789/0 conn(0x7f35ac156910 legacy=0x7f35ac0c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35ac0c9660 0x7f35980039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3598003f40 0x7f3598004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3598004d50 0x7f35980051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f35ac15a130 con 0x7f3598004d50 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f359800b9d0 con 0x7f3598003f40 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f359800bba0 con 0x7f35ac0c9660 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b1882640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3598004d50 0x7f35980051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b0880640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3598003f40 0x7f3598004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b1081640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35ac0c9660 0x7f35980039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b1882640 1 -- 172.21.15.83:0/2988360856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35ac0c9660 msgr2=0x7f35980039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b1882640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35ac0c9660 0x7f35980039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b1882640 1 -- 172.21.15.83:0/2988360856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3598003f40 msgr2=0x7f3598004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b1882640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3598003f40 0x7f3598004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b1882640 1 -- 172.21.15.83:0/2988360856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3598003530 con 0x7f3598004d50 2024-09-19T07:49:20.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b0880640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3598003f40 0x7f3598004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b1882640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3598004d50 0x7f35980051d0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0b1650 tx=0x7f35a401fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fac845c6f9b00f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a400a4e0 con 0x7f3598004d50 2024-09-19T07:49:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35a40aa710 con 0x7f3598004d50 2024-09-19T07:49:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35a3fff640 1 -- 172.21.15.83:0/2988360856 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a40aab00 con 0x7f3598004d50 2024-09-19T07:49:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b32e5640 1 -- 172.21.15.83:0/2988360856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3598004d50 msgr2=0x7f35980051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b32e5640 1 --2- 172.21.15.83:0/2988360856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3598004d50 0x7f35980051d0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0b1650 tx=0x7f35a401fac0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b1882640 1 -- 172.21.15.83:0/2988360856 reap_dead start 2024-09-19T07:49:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b32e5640 1 -- 172.21.15.83:0/2988360856 shutdown_connections 2024-09-19T07:49:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b32e5640 1 -- 172.21.15.83:0/2988360856 >> 172.21.15.83:0/2988360856 conn(0x7f35ac0722a0 msgr2=0x7f35ac0726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b32e5640 1 -- 172.21.15.83:0/2988360856 shutdown_connections 2024-09-19T07:49:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.372+0000 7f35b32e5640 1 -- 172.21.15.83:0/2988360856 wait complete. 2024-09-19T07:49:20.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b32e5640 1 Processor -- start 2024-09-19T07:49:20.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b32e5640 1 -- start start 2024-09-19T07:49:20.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b32e5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35ac0af650 0x7f35ac0c2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b32e5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35ac0c09a0 0x7f35ac1ed7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b32e5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35ac1edd30 0x7f35ac1ee900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b32e5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f35ac155eb0 con 0x7f35ac0c09a0 2024-09-19T07:49:20.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b32e5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f35ac0c66d0 con 0x7f35ac0af650 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b32e5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f35ac0b10e0 con 0x7f35ac1edd30 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b0880640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35ac0c09a0 0x7f35ac1ed7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b0880640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35ac0c09a0 0x7f35ac1ed7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50670/0 (socket says 172.21.15.83:50670) 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b0880640 1 -- 172.21.15.83:0/2407859073 learned_addr learned my addr 172.21.15.83:0/2407859073 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b0880640 1 -- 172.21.15.83:0/2407859073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35ac1edd30 msgr2=0x7f35ac1ee900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b0880640 1 --2- 172.21.15.83:0/2407859073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f35ac1edd30 0x7f35ac1ee900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b0880640 1 -- 172.21.15.83:0/2407859073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35ac0af650 msgr2=0x7f35ac0c2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b0880640 1 --2- 172.21.15.83:0/2407859073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f35ac0af650 0x7f35ac0c2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b0880640 1 -- 172.21.15.83:0/2407859073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35ac0bfa80 con 0x7f35ac0c09a0 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b0880640 1 --2- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35ac0c09a0 0x7f35ac1ed7f0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f359c0108f0 tx=0x7f359c010db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35a1ffb640 1 -- 172.21.15.83:0/2407859073 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359c00f040 con 0x7f35ac0c09a0 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35a1ffb640 1 -- 172.21.15.83:0/2407859073 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f359c004220 con 0x7f35ac0c09a0 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b32e5640 1 -- 172.21.15.83:0/2407859073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35ac0bfd20 con 0x7f35ac0c09a0 2024-09-19T07:49:20.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35a1ffb640 1 -- 172.21.15.83:0/2407859073 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359c00e880 con 0x7f35ac0c09a0 2024-09-19T07:49:20.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b32e5640 1 -- 172.21.15.83:0/2407859073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35ac159c70 con 0x7f35ac0c09a0 2024-09-19T07:49:20.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35b32e5640 1 -- 172.21.15.83:0/2407859073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f35ac0c0000 con 0x7f35ac0c09a0 2024-09-19T07:49:20.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.376+0000 7f35a1ffb640 1 -- 172.21.15.83:0/2407859073 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f359c00ea20 con 0x7f35ac0c09a0 2024-09-19T07:49:20.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.380+0000 7f35a1ffb640 1 --2- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35900678f0 0x7f3590069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.380+0000 7f35a1ffb640 1 -- 172.21.15.83:0/2407859073 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f359c0875f0 con 0x7f35ac0c09a0 2024-09-19T07:49:20.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.380+0000 7f35b1081640 1 --2- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35900678f0 0x7f3590069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.380+0000 7f35a1ffb640 1 --2- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] conn(0x7f35900742d0 0x7f3590076750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.380+0000 7f35a1ffb640 1 -- 172.21.15.83:0/2407859073 --> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3590076e20 con 0x7f35900742d0 2024-09-19T07:49:20.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.380+0000 7f35a1ffb640 1 -- 172.21.15.83:0/2407859073 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f359c087a40 con 0x7f35ac0c09a0 2024-09-19T07:49:20.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.380+0000 7f35b1882640 1 --2- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] conn(0x7f35900742d0 0x7f3590076750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.380+0000 7f35b1081640 1 --2- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35900678f0 0x7f3590069db0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f35a8009950 tx=0x7f35a8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.380+0000 7f35b1882640 1 --2- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] conn(0x7f35900742d0 0x7f3590076750 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.380+0000 7f35a1ffb640 1 -- 172.21.15.83:0/2407859073 <== osd.2 v2:172.21.15.83:6816/229839114 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3590076e20 con 0x7f35900742d0 2024-09-19T07:49:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35b32e5640 1 -- 172.21.15.83:0/2407859073 --> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f35ac0c60e0 con 0x7f35900742d0 2024-09-19T07:49:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35a1ffb640 1 -- 172.21.15.83:0/2407859073 <== osd.2 v2:172.21.15.83:6816/229839114 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f35ac0c60e0 con 0x7f35900742d0 2024-09-19T07:49:20.392 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-19T07:49:20.392 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-19T07:49:20.392 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-19T07:49:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35b32e5640 1 -- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] conn(0x7f35900742d0 msgr2=0x7f3590076750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35b32e5640 1 --2- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:6816/229839114,v1:172.21.15.83:6817/229839114] conn(0x7f35900742d0 0x7f3590076750 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35b32e5640 1 -- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35900678f0 msgr2=0x7f3590069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35b32e5640 1 --2- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f35900678f0 0x7f3590069db0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f35a8009950 tx=0x7f35a8008040 comp rx=0 tx=0).stop 2024-09-19T07:49:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35b32e5640 1 -- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35ac0c09a0 msgr2=0x7f35ac1ed7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35b32e5640 1 --2- 172.21.15.83:0/2407859073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35ac0c09a0 0x7f35ac1ed7f0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f359c0108f0 tx=0x7f359c010db0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35b1882640 1 -- 172.21.15.83:0/2407859073 reap_dead start 2024-09-19T07:49:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35b32e5640 1 -- 172.21.15.83:0/2407859073 shutdown_connections 2024-09-19T07:49:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35b32e5640 1 -- 172.21.15.83:0/2407859073 >> 172.21.15.83:0/2407859073 conn(0x7f35ac0722a0 msgr2=0x7f35ac0cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:20.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35b32e5640 1 -- 172.21.15.83:0/2407859073 shutdown_connections 2024-09-19T07:49:20.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.392+0000 7f35b32e5640 1 -- 172.21.15.83:0/2407859073 wait complete. 2024-09-19T07:49:20.407 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-19T07:49:20.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe958a0c640 1 Processor -- start 2024-09-19T07:49:20.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe958a0c640 1 -- start start 2024-09-19T07:49:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe958a0c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe9541534b0 0x7fe9540c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe958a0c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9540c3810 con 0x7fe954150060 2024-09-19T07:49:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe958a0c640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9540c3ab0 con 0x7fe9540c2b10 2024-09-19T07:49:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe958a0c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe954150440 con 0x7fe9541534b0 2024-09-19T07:49:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe95259c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe954150060 0x7fe9540c4010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35898/0 (socket says 172.21.15.83:35898) 2024-09-19T07:49:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe95259c640 1 -- 172.21.15.83:0/311356633 learned_addr learned my addr 172.21.15.83:0/311356633 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe951d9b640 1 --2- 172.21.15.83:0/311356633 >> v2:172.21.15.83:3300/0 conn(0x7fe9541534b0 0x7fe9540c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe951d9b640 1 -- 172.21.15.83:0/311356633 >> v1:172.21.15.184:6789/0 conn(0x7fe9540c2b10 legacy=0x7fe95415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe951d9b640 1 -- 172.21.15.83:0/311356633 >> v1:172.21.15.83:6789/0 conn(0x7fe954150060 legacy=0x7fe9540c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe951d9b640 1 -- 172.21.15.83:0/311356633 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9540c2f40 con 0x7fe9541534b0 2024-09-19T07:49:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.520+0000 7fe95159a640 1 -- 172.21.15.83:0/311356633 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2163508516 0 0) 0x7fe9540c3810 con 0x7fe954150060 2024-09-19T07:49:20.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.524+0000 7fe951d9b640 1 --2- 172.21.15.83:0/311356633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9541534b0 0x7fe9540c25d0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fe9480097c0 tx=0x7fe94801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edab095db93f5373 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.524+0000 7fe95159a640 1 -- 172.21.15.83:0/311356633 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9480b0070 con 0x7fe9541534b0 2024-09-19T07:49:20.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.524+0000 7fe95159a640 1 -- 172.21.15.83:0/311356633 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9480ab600 con 0x7fe9541534b0 2024-09-19T07:49:20.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.524+0000 7fe95159a640 1 -- 172.21.15.83:0/311356633 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9480a4e60 con 0x7fe9541534b0 2024-09-19T07:49:20.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.524+0000 7fe958a0c640 1 -- 172.21.15.83:0/311356633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9541534b0 msgr2=0x7fe9540c25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.524+0000 7fe958a0c640 1 --2- 172.21.15.83:0/311356633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9541534b0 0x7fe9540c25d0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fe9480097c0 tx=0x7fe94801c3d0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.524+0000 7fe958a0c640 1 -- 172.21.15.83:0/311356633 shutdown_connections 2024-09-19T07:49:20.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.524+0000 7fe958a0c640 1 --2- 172.21.15.83:0/311356633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9541534b0 0x7fe9540c25d0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.524+0000 7fe958a0c640 1 -- 172.21.15.83:0/311356633 >> 172.21.15.83:0/311356633 conn(0x7fe95406bc80 msgr2=0x7fe95406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:20.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.524+0000 7fe958a0c640 1 -- 172.21.15.83:0/311356633 shutdown_connections 2024-09-19T07:49:20.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.524+0000 7fe958a0c640 1 -- 172.21.15.83:0/311356633 wait complete. 2024-09-19T07:49:20.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe958a0c640 1 Processor -- start 2024-09-19T07:49:20.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe958a0c640 1 -- start start 2024-09-19T07:49:20.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe958a0c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9540c2b10 0x7fe9541ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe958a0c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe954150060 0x7fe9540c0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe958a0c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9541534b0 0x7fe9541f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe958a0c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9540c3d90 con 0x7fe9541534b0 2024-09-19T07:49:20.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe958a0c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9540c2140 con 0x7fe954150060 2024-09-19T07:49:20.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe958a0c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe9540c1ab0 con 0x7fe9540c2b10 2024-09-19T07:49:20.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe952d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9541534b0 0x7fe9541f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe952d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9541534b0 0x7fe9541f2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50690/0 (socket says 172.21.15.83:50690) 2024-09-19T07:49:20.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe952d9d640 1 -- 172.21.15.83:0/2662708778 learned_addr learned my addr 172.21.15.83:0/2662708778 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:20.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe951d9b640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe954150060 0x7fe9540c0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe95259c640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9540c2b10 0x7fe9541ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe952d9d640 1 -- 172.21.15.83:0/2662708778 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9540c2b10 msgr2=0x7fe9541ee9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe952d9d640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9540c2b10 0x7fe9541ee9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe952d9d640 1 -- 172.21.15.83:0/2662708778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe954150060 msgr2=0x7fe9540c0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe952d9d640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe954150060 0x7fe9540c0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe952d9d640 1 -- 172.21.15.83:0/2662708778 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9541eef10 con 0x7fe9541534b0 2024-09-19T07:49:20.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe952d9d640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9541534b0 0x7fe9541f2870 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00c3c0 tx=0x7fe94c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe942ffd640 1 -- 172.21.15.83:0/2662708778 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c011790 con 0x7fe9541534b0 2024-09-19T07:49:20.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe942ffd640 1 -- 172.21.15.83:0/2662708778 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe94c017070 con 0x7fe9541534b0 2024-09-19T07:49:20.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe958a0c640 1 -- 172.21.15.83:0/2662708778 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe954156540 con 0x7fe9541534b0 2024-09-19T07:49:20.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe942ffd640 1 -- 172.21.15.83:0/2662708778 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c011b70 con 0x7fe9541534b0 2024-09-19T07:49:20.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe958a0c640 1 -- 172.21.15.83:0/2662708778 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9541f2070 con 0x7fe9541534b0 2024-09-19T07:49:20.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe958a0c640 1 -- 172.21.15.83:0/2662708778 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9540c3450 con 0x7fe9541534b0 2024-09-19T07:49:20.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe942ffd640 1 -- 172.21.15.83:0/2662708778 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe94c011d10 con 0x7fe9541534b0 2024-09-19T07:49:20.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe942ffd640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe9340678c0 0x7fe934069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe942ffd640 1 -- 172.21.15.83:0/2662708778 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fe94c086ae0 con 0x7fe9541534b0 2024-09-19T07:49:20.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.528+0000 7fe95259c640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe9340678c0 0x7fe934069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.532+0000 7fe95259c640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe9340678c0 0x7fe934069d80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe9440012f0 tx=0x7fe944009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.536+0000 7fe942ffd640 1 -- 172.21.15.83:0/2662708778 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe94c056550 con 0x7fe9541534b0 2024-09-19T07:49:20.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.700+0000 7fe958a0c640 1 -- 172.21.15.83:0/2662708778 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fe9540c25b0 con 0x7fe9340678c0 2024-09-19T07:49:20.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.700+0000 7fe942ffd640 1 -- 172.21.15.83:0/2662708778 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe9540c25b0 con 0x7fe9340678c0 2024-09-19T07:49:20.700 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 2 to deep-scrub 2024-09-19T07:49:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.700+0000 7fe958a0c640 1 -- 172.21.15.83:0/2662708778 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe9340678c0 msgr2=0x7fe934069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.700+0000 7fe958a0c640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe9340678c0 0x7fe934069d80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe9440012f0 tx=0x7fe944009040 comp rx=0 tx=0).stop 2024-09-19T07:49:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.700+0000 7fe958a0c640 1 -- 172.21.15.83:0/2662708778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9541534b0 msgr2=0x7fe9541f2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.700+0000 7fe958a0c640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9541534b0 0x7fe9541f2870 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00c3c0 tx=0x7fe94c00c880 comp rx=0 tx=0).stop 2024-09-19T07:49:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.704+0000 7fe958a0c640 1 -- 172.21.15.83:0/2662708778 shutdown_connections 2024-09-19T07:49:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.704+0000 7fe958a0c640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe9340678c0 0x7fe934069d80 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.704+0000 7fe958a0c640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9541534b0 0x7fe9541f2870 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.704+0000 7fe958a0c640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe954150060 0x7fe9540c0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.704+0000 7fe958a0c640 1 --2- 172.21.15.83:0/2662708778 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9540c2b10 0x7fe9541ee9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.704+0000 7fe958a0c640 1 -- 172.21.15.83:0/2662708778 >> 172.21.15.83:0/2662708778 conn(0x7fe95406bc80 msgr2=0x7fe95414dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.704+0000 7fe958a0c640 1 -- 172.21.15.83:0/2662708778 shutdown_connections 2024-09-19T07:49:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.704+0000 7fe958a0c640 1 -- 172.21.15.83:0/2662708778 wait complete. 2024-09-19T07:49:20.719 INFO:tasks.ceph:Scrubbing osd.3 2024-09-19T07:49:20.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-19T07:49:20.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff22390640 1 Processor -- start 2024-09-19T07:49:20.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff22390640 1 -- start start 2024-09-19T07:49:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff22390640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff1c151330 0x7eff1c070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff22390640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7eff1c0713e0 0x7eff1c06ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff22390640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff1c06ea10 con 0x7eff1c0a9d30 2024-09-19T07:49:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff22390640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7eff1c154910 con 0x7eff1c0713e0 2024-09-19T07:49:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff22390640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7eff1c153d00 con 0x7eff1c151330 2024-09-19T07:49:20.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff1bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff1c151330 0x7eff1c070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff1b7fe640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7eff1c0713e0 0x7eff1c06ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff1bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff1c151330 0x7eff1c070ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:43034/0 (socket says 172.21.15.83:43034) 2024-09-19T07:49:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff1bfff640 1 -- 172.21.15.83:0/1628829351 learned_addr learned my addr 172.21.15.83:0/1628829351 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff1affd640 1 -- 172.21.15.83:0/1628829351 <== mon.1 v1:172.21.15.184:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3701208460 0 0) 0x7eff1c06ea10 con 0x7eff1c0a9d30 2024-09-19T07:49:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff1affd640 1 -- 172.21.15.83:0/1628829351 --> v1:172.21.15.184:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efefc003530 con 0x7eff1c0a9d30 2024-09-19T07:49:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff1bfff640 1 -- 172.21.15.83:0/1628829351 >> v2:172.21.15.184:3300/0 conn(0x7eff1c0713e0 msgr2=0x7eff1c06ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff1bfff640 1 --2- 172.21.15.83:0/1628829351 >> v2:172.21.15.184:3300/0 conn(0x7eff1c0713e0 0x7eff1c06ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff1bfff640 1 -- 172.21.15.83:0/1628829351 >> v1:172.21.15.184:6789/0 conn(0x7eff1c0a9d30 legacy=0x7eff1c06e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.832+0000 7eff1bfff640 1 -- 172.21.15.83:0/1628829351 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1c153fe0 con 0x7eff1c151330 2024-09-19T07:49:20.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff1bfff640 1 --2- 172.21.15.83:0/1628829351 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff1c151330 0x7eff1c070ea0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7eff14009e50 tx=0x7eff1401c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32dd631c7db6f336 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff1affd640 1 -- 172.21.15.83:0/1628829351 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff140a57c0 con 0x7eff1c151330 2024-09-19T07:49:20.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff1affd640 1 -- 172.21.15.83:0/1628829351 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff140a5960 con 0x7eff1c151330 2024-09-19T07:49:20.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff1affd640 1 -- 172.21.15.83:0/1628829351 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff140a5d10 con 0x7eff1c151330 2024-09-19T07:49:20.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff22390640 1 -- 172.21.15.83:0/1628829351 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff1c151330 msgr2=0x7eff1c070ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff22390640 1 --2- 172.21.15.83:0/1628829351 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff1c151330 0x7eff1c070ea0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7eff14009e50 tx=0x7eff1401c8e0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff22390640 1 -- 172.21.15.83:0/1628829351 shutdown_connections 2024-09-19T07:49:20.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff22390640 1 --2- 172.21.15.83:0/1628829351 >> v2:172.21.15.184:3300/0 conn(0x7eff1c0713e0 0x7eff1c06ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff22390640 1 --2- 172.21.15.83:0/1628829351 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff1c151330 0x7eff1c070ea0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff22390640 1 -- 172.21.15.83:0/1628829351 >> 172.21.15.83:0/1628829351 conn(0x7eff1c06bc80 msgr2=0x7eff1c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:20.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff22390640 1 -- 172.21.15.83:0/1628829351 shutdown_connections 2024-09-19T07:49:20.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff22390640 1 -- 172.21.15.83:0/1628829351 wait complete. 2024-09-19T07:49:20.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff22390640 1 Processor -- start 2024-09-19T07:49:20.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.836+0000 7eff22390640 1 -- start start 2024-09-19T07:49:20.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff22390640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff1c0713e0 0x7eff1c0c1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff22390640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff1c0a9d30 0x7eff1c145810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff22390640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff1c151330 0x7eff1c145d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff22390640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c151890 con 0x7eff1c0a9d30 2024-09-19T07:49:20.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff22390640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c151710 con 0x7eff1c0713e0 2024-09-19T07:49:20.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff22390640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7eff1c160cb0 con 0x7eff1c151330 2024-09-19T07:49:20.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff1c0a9d30 0x7eff1c145810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff1c0a9d30 0x7eff1c145810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50706/0 (socket says 172.21.15.83:50706) 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1b7fe640 1 -- 172.21.15.83:0/15093906 learned_addr learned my addr 172.21.15.83:0/15093906 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1bfff640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff1c0713e0 0x7eff1c0c1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff2092d640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff1c151330 0x7eff1c145d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1b7fe640 1 -- 172.21.15.83:0/15093906 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff1c151330 msgr2=0x7eff1c145d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1b7fe640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff1c151330 0x7eff1c145d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1b7fe640 1 -- 172.21.15.83:0/15093906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff1c0713e0 msgr2=0x7eff1c0c1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1b7fe640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff1c0713e0 0x7eff1c0c1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1b7fe640 1 -- 172.21.15.83:0/15093906 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1c1ef750 con 0x7eff1c0a9d30 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1bfff640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff1c0713e0 0x7eff1c0c1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1b7fe640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff1c0a9d30 0x7eff1c145810 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7eff0400e900 tx=0x7eff0400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff18ff9640 1 -- 172.21.15.83:0/15093906 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff04017070 con 0x7eff1c0a9d30 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff18ff9640 1 -- 172.21.15.83:0/15093906 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff0400ca00 con 0x7eff1c0a9d30 2024-09-19T07:49:20.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff22390640 1 -- 172.21.15.83:0/15093906 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff1c1efa10 con 0x7eff1c0a9d30 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff18ff9640 1 -- 172.21.15.83:0/15093906 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0400cdd0 con 0x7eff1c0a9d30 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff22390640 1 -- 172.21.15.83:0/15093906 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff1c1effd0 con 0x7eff1c0a9d30 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff22390640 1 -- 172.21.15.83:0/15093906 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7eff1c070ea0 con 0x7eff1c0a9d30 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff18ff9640 1 -- 172.21.15.83:0/15093906 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7eff0401a070 con 0x7eff1c0a9d30 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff18ff9640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff000678c0 0x7eff00069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff18ff9640 1 -- 172.21.15.83:0/15093906 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7eff04087660 con 0x7eff1c0a9d30 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1bfff640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff000678c0 0x7eff00069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff18ff9640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] conn(0x7eff00074290 0x7eff00076710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff18ff9640 1 -- 172.21.15.83:0/15093906 --> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff00076de0 con 0x7eff00074290 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff18ff9640 1 -- 172.21.15.83:0/15093906 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7eff04087a10 con 0x7eff1c0a9d30 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff2092d640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] conn(0x7eff00074290 0x7eff00076710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff1bfff640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff000678c0 0x7eff00069d80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7eff140ae000 tx=0x7eff140a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.840+0000 7eff2092d640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] conn(0x7eff00074290 0x7eff00076710 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.844+0000 7eff18ff9640 1 -- 172.21.15.83:0/15093906 <== osd.3 v2:172.21.15.83:6824/2952931730 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff00076de0 con 0x7eff00074290 2024-09-19T07:49:20.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff22390640 1 -- 172.21.15.83:0/15093906 --> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7eff1c073f50 con 0x7eff00074290 2024-09-19T07:49:20.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff18ff9640 1 -- 172.21.15.83:0/15093906 <== osd.3 v2:172.21.15.83:6824/2952931730 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7eff1c073f50 con 0x7eff00074290 2024-09-19T07:49:20.856 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-19T07:49:20.856 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-19T07:49:20.856 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-19T07:49:20.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff22390640 1 -- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] conn(0x7eff00074290 msgr2=0x7eff00076710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff22390640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:6824/2952931730,v1:172.21.15.83:6825/2952931730] conn(0x7eff00074290 0x7eff00076710 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff22390640 1 -- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff000678c0 msgr2=0x7eff00069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff22390640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7eff000678c0 0x7eff00069d80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7eff140ae000 tx=0x7eff140a7070 comp rx=0 tx=0).stop 2024-09-19T07:49:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff22390640 1 -- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff1c0a9d30 msgr2=0x7eff1c145810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff22390640 1 --2- 172.21.15.83:0/15093906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff1c0a9d30 0x7eff1c145810 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7eff0400e900 tx=0x7eff0400edc0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff2092d640 1 -- 172.21.15.83:0/15093906 reap_dead start 2024-09-19T07:49:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff22390640 1 -- 172.21.15.83:0/15093906 shutdown_connections 2024-09-19T07:49:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff22390640 1 -- 172.21.15.83:0/15093906 >> 172.21.15.83:0/15093906 conn(0x7eff1c06bc80 msgr2=0x7eff1c14df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff22390640 1 -- 172.21.15.83:0/15093906 shutdown_connections 2024-09-19T07:49:20.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.856+0000 7eff22390640 1 -- 172.21.15.83:0/15093906 wait complete. 2024-09-19T07:49:20.871 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-19T07:49:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc288878640 1 Processor -- start 2024-09-19T07:49:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc288878640 1 -- start start 2024-09-19T07:49:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc288878640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2841534b0 0x7fc2840c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc288878640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2840c2fc0 con 0x7fc2840c2b10 2024-09-19T07:49:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc288878640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2840c3810 con 0x7fc284150060 2024-09-19T07:49:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc288878640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc284150440 con 0x7fc2841534b0 2024-09-19T07:49:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc282d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc2840c2b10 0x7fc28415d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35900/0 (socket says 172.21.15.83:35900) 2024-09-19T07:49:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc282d9d640 1 -- 172.21.15.83:0/2319430040 learned_addr learned my addr 172.21.15.83:0/2319430040 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc281d9b640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2841534b0 0x7fc2840c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2751357500 0 0) 0x7fc2840c2fc0 con 0x7fc2840c2b10 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc26c005740 con 0x7fc2840c2b10 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2919799436 0 0) 0x7fc26c005740 con 0x7fc2840c2b10 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2841534b0 msgr2=0x7fc2840c25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2841534b0 0x7fc2840c25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 >> v1:172.21.15.184:6789/0 conn(0x7fc284150060 legacy=0x7fc2840c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2840c3ab0 con 0x7fc2840c2b10 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fc27c001a00 con 0x7fc2840c2b10 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 >> v1:172.21.15.83:6789/0 conn(0x7fc2840c2b10 legacy=0x7fc28415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc26c005c70 0x7fc26c006090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2841534b0 0x7fc26c006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c006f60 0x7fc26c0073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc284150440 con 0x7fc26c006f60 2024-09-19T07:49:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc26c00dd70 con 0x7fc26c005c70 2024-09-19T07:49:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc26c00df70 con 0x7fc2841534b0 2024-09-19T07:49:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc282d9d640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c006f60 0x7fc26c0073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc281d9b640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2841534b0 0x7fc26c006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28259c640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc26c005c70 0x7fc26c006090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc282d9d640 1 -- 172.21.15.83:0/2319430040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2841534b0 msgr2=0x7fc26c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc282d9d640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc2841534b0 0x7fc26c006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc282d9d640 1 -- 172.21.15.83:0/2319430040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc26c005c70 msgr2=0x7fc26c006090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc282d9d640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc26c005c70 0x7fc26c006090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc282d9d640 1 -- 172.21.15.83:0/2319430040 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc26c005740 con 0x7fc26c006f60 2024-09-19T07:49:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.976+0000 7fc28259c640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc26c005c70 0x7fc26c006090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:20.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc282d9d640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c006f60 0x7fc26c0073e0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00a860 tx=0x7fc27c01fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5833711dd1f8cb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c00a480 con 0x7fc26c006f60 2024-09-19T07:49:20.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc27c0aa710 con 0x7fc26c006f60 2024-09-19T07:49:20.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc28159a640 1 -- 172.21.15.83:0/2319430040 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c0aab40 con 0x7fc26c006f60 2024-09-19T07:49:20.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 -- 172.21.15.83:0/2319430040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c006f60 msgr2=0x7fc26c0073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 --2- 172.21.15.83:0/2319430040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c006f60 0x7fc26c0073e0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00a860 tx=0x7fc27c01fae0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc282d9d640 1 -- 172.21.15.83:0/2319430040 reap_dead start 2024-09-19T07:49:20.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 -- 172.21.15.83:0/2319430040 shutdown_connections 2024-09-19T07:49:20.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 -- 172.21.15.83:0/2319430040 >> 172.21.15.83:0/2319430040 conn(0x7fc28406bc80 msgr2=0x7fc28406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:20.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 -- 172.21.15.83:0/2319430040 shutdown_connections 2024-09-19T07:49:20.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 -- 172.21.15.83:0/2319430040 wait complete. 2024-09-19T07:49:20.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 Processor -- start 2024-09-19T07:49:20.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 -- start start 2024-09-19T07:49:20.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c006f60 0x7fc284152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc28414d2a0 0x7fc2840c0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc284149a80 0x7fc284149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2841539c0 con 0x7fc26c006f60 2024-09-19T07:49:20.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc2840c2140 con 0x7fc284149a80 2024-09-19T07:49:20.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc288878640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc2840c3d90 con 0x7fc28414d2a0 2024-09-19T07:49:20.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc28259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c006f60 0x7fc284152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc28259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c006f60 0x7fc284152ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50732/0 (socket says 172.21.15.83:50732) 2024-09-19T07:49:20.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc28259c640 1 -- 172.21.15.83:0/2896867337 learned_addr learned my addr 172.21.15.83:0/2896867337 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc281d9b640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc28414d2a0 0x7fc2840c0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc282d9d640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc284149a80 0x7fc284149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc28259c640 1 -- 172.21.15.83:0/2896867337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc28414d2a0 msgr2=0x7fc2840c0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc28259c640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc28414d2a0 0x7fc2840c0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc28259c640 1 -- 172.21.15.83:0/2896867337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc284149a80 msgr2=0x7fc284149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc28259c640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc284149a80 0x7fc284149f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc28259c640 1 -- 172.21.15.83:0/2896867337 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2841eedd0 con 0x7fc26c006f60 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc281d9b640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc28414d2a0 0x7fc2840c0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.980+0000 7fc282d9d640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc284149a80 0x7fc284149f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.984+0000 7fc28259c640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c006f60 0x7fc284152ba0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fc27400d770 tx=0x7fc27400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.984+0000 7fc272ffd640 1 -- 172.21.15.83:0/2896867337 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc274005270 con 0x7fc26c006f60 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.984+0000 7fc272ffd640 1 -- 172.21.15.83:0/2896867337 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc274017070 con 0x7fc26c006f60 2024-09-19T07:49:20.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.984+0000 7fc272ffd640 1 -- 172.21.15.83:0/2896867337 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27400b650 con 0x7fc26c006f60 2024-09-19T07:49:20.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.984+0000 7fc288878640 1 -- 172.21.15.83:0/2896867337 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2841ef0b0 con 0x7fc26c006f60 2024-09-19T07:49:20.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.984+0000 7fc288878640 1 -- 172.21.15.83:0/2896867337 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2841ef670 con 0x7fc26c006f60 2024-09-19T07:49:20.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.984+0000 7fc288878640 1 -- 172.21.15.83:0/2896867337 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2840c3450 con 0x7fc26c006f60 2024-09-19T07:49:20.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.984+0000 7fc272ffd640 1 -- 172.21.15.83:0/2896867337 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc27400b7f0 con 0x7fc26c006f60 2024-09-19T07:49:20.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.984+0000 7fc272ffd640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2640678f0 0x7fc264069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:20.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.984+0000 7fc272ffd640 1 -- 172.21.15.83:0/2896867337 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fc274086cf0 con 0x7fc26c006f60 2024-09-19T07:49:20.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.984+0000 7fc281d9b640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2640678f0 0x7fc264069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:20.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.984+0000 7fc281d9b640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2640678f0 0x7fc264069db0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc28414d930 tx=0x7fc278008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:20.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:20.988+0000 7fc272ffd640 1 -- 172.21.15.83:0/2896867337 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc27408b070 con 0x7fc26c006f60 2024-09-19T07:49:21.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 -- 172.21.15.83:0/2896867337 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fc2841dcd10 con 0x7fc2640678f0 2024-09-19T07:49:21.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc272ffd640 1 -- 172.21.15.83:0/2896867337 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc2841dcd10 con 0x7fc2640678f0 2024-09-19T07:49:21.151 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 3 to deep-scrub 2024-09-19T07:49:21.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 -- 172.21.15.83:0/2896867337 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2640678f0 msgr2=0x7fc264069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2640678f0 0x7fc264069db0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc28414d930 tx=0x7fc278008040 comp rx=0 tx=0).stop 2024-09-19T07:49:21.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 -- 172.21.15.83:0/2896867337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c006f60 msgr2=0x7fc284152ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c006f60 0x7fc284152ba0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fc27400d770 tx=0x7fc27400dc30 comp rx=0 tx=0).stop 2024-09-19T07:49:21.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 -- 172.21.15.83:0/2896867337 shutdown_connections 2024-09-19T07:49:21.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc2640678f0 0x7fc264069db0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc284149a80 0x7fc284149f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc28414d2a0 0x7fc2840c0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 --2- 172.21.15.83:0/2896867337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c006f60 0x7fc284152ba0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 -- 172.21.15.83:0/2896867337 >> 172.21.15.83:0/2896867337 conn(0x7fc28406bc80 msgr2=0x7fc28414f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:21.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 -- 172.21.15.83:0/2896867337 shutdown_connections 2024-09-19T07:49:21.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.152+0000 7fc288878640 1 -- 172.21.15.83:0/2896867337 wait complete. 2024-09-19T07:49:21.170 INFO:tasks.ceph:Scrubbing osd.4 2024-09-19T07:49:21.170 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-19T07:49:21.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27b5af0640 1 Processor -- start 2024-09-19T07:49:21.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27b5af0640 1 -- start start 2024-09-19T07:49:21.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27b5af0640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f27b0155af0 0x7f27b00a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27b5af0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f27b01527c0 0x7f27b0152c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27b5af0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27b00aa270 con 0x7f27b00aa730 2024-09-19T07:49:21.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27b5af0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f27b00c3b20 con 0x7f27b01527c0 2024-09-19T07:49:21.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27b5af0640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f27b00c3e00 con 0x7f27b0155af0 2024-09-19T07:49:21.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27aeffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f27b01527c0 0x7f27b0152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27aeffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f27b01527c0 0x7f27b0152c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50748/0 (socket says 172.21.15.83:50748) 2024-09-19T07:49:21.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27affff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f27b00aa730 0x7f27b00aaba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35902/0 (socket says 172.21.15.83:35902) 2024-09-19T07:49:21.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27aeffd640 1 -- 172.21.15.83:0/262589615 learned_addr learned my addr 172.21.15.83:0/262589615 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:21.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27af7fe640 1 --2- 172.21.15.83:0/262589615 >> v2:172.21.15.184:3300/0 conn(0x7f27b0155af0 0x7f27b00a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27aeffd640 1 -- 172.21.15.83:0/262589615 >> v2:172.21.15.184:3300/0 conn(0x7f27b0155af0 msgr2=0x7f27b00a9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:21.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27aeffd640 1 --2- 172.21.15.83:0/262589615 >> v2:172.21.15.184:3300/0 conn(0x7f27b0155af0 0x7f27b00a9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27aeffd640 1 -- 172.21.15.83:0/262589615 >> v1:172.21.15.83:6789/0 conn(0x7f27b00aa730 legacy=0x7f27b00aaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27aeffd640 1 -- 172.21.15.83:0/262589615 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b00c2a00 con 0x7f27b01527c0 2024-09-19T07:49:21.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27ae7fc640 1 -- 172.21.15.83:0/262589615 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1346037402 0 0) 0x7f27b00aa270 con 0x7f27b00aa730 2024-09-19T07:49:21.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.288+0000 7f27af7fe640 1 --2- 172.21.15.83:0/262589615 >> v2:172.21.15.184:3300/0 conn(0x7f27b0155af0 0x7f27b00a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:21.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27aeffd640 1 --2- 172.21.15.83:0/262589615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27b01527c0 0x7f27b0152c30 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f27b006f370 tx=0x7f27a401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae424350479c66b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27ae7fc640 1 -- 172.21.15.83:0/262589615 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a40ab660 con 0x7f27b01527c0 2024-09-19T07:49:21.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27ae7fc640 1 -- 172.21.15.83:0/262589615 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a40b0070 con 0x7f27b01527c0 2024-09-19T07:49:21.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27ae7fc640 1 -- 172.21.15.83:0/262589615 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a40aba20 con 0x7f27b01527c0 2024-09-19T07:49:21.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 -- 172.21.15.83:0/262589615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27b01527c0 msgr2=0x7f27b0152c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 --2- 172.21.15.83:0/262589615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27b01527c0 0x7f27b0152c30 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f27b006f370 tx=0x7f27a401c460 comp rx=0 tx=0).stop 2024-09-19T07:49:21.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 -- 172.21.15.83:0/262589615 shutdown_connections 2024-09-19T07:49:21.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 --2- 172.21.15.83:0/262589615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27b01527c0 0x7f27b0152c30 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 --2- 172.21.15.83:0/262589615 >> v2:172.21.15.184:3300/0 conn(0x7f27b0155af0 0x7f27b00a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 -- 172.21.15.83:0/262589615 >> 172.21.15.83:0/262589615 conn(0x7f27b006bc80 msgr2=0x7f27b006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:21.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 -- 172.21.15.83:0/262589615 shutdown_connections 2024-09-19T07:49:21.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 -- 172.21.15.83:0/262589615 wait complete. 2024-09-19T07:49:21.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 Processor -- start 2024-09-19T07:49:21.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 -- start start 2024-09-19T07:49:21.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27b00aa730 0x7f27b0156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27b01527c0 0x7f27b015aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27b0155af0 0x7f27b0159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f27b0153170 con 0x7f27b0155af0 2024-09-19T07:49:21.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f27b01564c0 con 0x7f27b00aa730 2024-09-19T07:49:21.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.292+0000 7f27b5af0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f27b00c2590 con 0x7f27b01527c0 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27affff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27b0155af0 0x7f27b0159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27affff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27b0155af0 0x7f27b0159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50762/0 (socket says 172.21.15.83:50762) 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27affff640 1 -- 172.21.15.83:0/1770118249 learned_addr learned my addr 172.21.15.83:0/1770118249 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27af7fe640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27b00aa730 0x7f27b0156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27aeffd640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27b01527c0 0x7f27b015aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27affff640 1 -- 172.21.15.83:0/1770118249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27b01527c0 msgr2=0x7f27b015aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27affff640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27b01527c0 0x7f27b015aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27affff640 1 -- 172.21.15.83:0/1770118249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27b00aa730 msgr2=0x7f27b0156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27affff640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27b00aa730 0x7f27b0156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27affff640 1 -- 172.21.15.83:0/1770118249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b015b290 con 0x7f27b0155af0 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27af7fe640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27b00aa730 0x7f27b0156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27affff640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27b0155af0 0x7f27b0159ab0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f27a800c480 tx=0x7f27a800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f2797fff640 1 -- 172.21.15.83:0/1770118249 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a8018070 con 0x7f27b0155af0 2024-09-19T07:49:21.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f2797fff640 1 -- 172.21.15.83:0/1770118249 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a80074e0 con 0x7f27b0155af0 2024-09-19T07:49:21.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27b5af0640 1 -- 172.21.15.83:0/1770118249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b015af80 con 0x7f27b0155af0 2024-09-19T07:49:21.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f2797fff640 1 -- 172.21.15.83:0/1770118249 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a8004a30 con 0x7f27b0155af0 2024-09-19T07:49:21.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27b5af0640 1 -- 172.21.15.83:0/1770118249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b00c22c0 con 0x7f27b0155af0 2024-09-19T07:49:21.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27b5af0640 1 -- 172.21.15.83:0/1770118249 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f27b015b460 con 0x7f27b0155af0 2024-09-19T07:49:21.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f2797fff640 1 -- 172.21.15.83:0/1770118249 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f27a801b070 con 0x7f27b0155af0 2024-09-19T07:49:21.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f2797fff640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f279c0678c0 0x7f279c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f2797fff640 1 -- 172.21.15.83:0/1770118249 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f27a8086640 con 0x7f27b0155af0 2024-09-19T07:49:21.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27af7fe640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f279c0678c0 0x7f279c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f2797fff640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] conn(0x7f279c0742a0 0x7f279c076720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f2797fff640 1 -- 172.21.15.83:0/1770118249 --> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f279c076df0 con 0x7f279c0742a0 2024-09-19T07:49:21.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f2797fff640 1 -- 172.21.15.83:0/1770118249 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f27a8086a90 con 0x7f27b0155af0 2024-09-19T07:49:21.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27af7fe640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f279c0678c0 0x7f279c069d80 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f27a0007f40 tx=0x7f27a0007490 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27aeffd640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] conn(0x7f279c0742a0 0x7f279c076720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.296+0000 7f27aeffd640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] conn(0x7f279c0742a0 0x7f279c076720 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.300+0000 7f2797fff640 1 -- 172.21.15.83:0/1770118249 <== osd.4 v2:172.21.15.184:6808/3839042106 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f279c076df0 con 0x7f279c0742a0 2024-09-19T07:49:21.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f27b5af0640 1 -- 172.21.15.83:0/1770118249 --> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f27b0155ed0 con 0x7f279c0742a0 2024-09-19T07:49:21.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f2797fff640 1 -- 172.21.15.83:0/1770118249 <== osd.4 v2:172.21.15.184:6808/3839042106 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f27b0155ed0 con 0x7f279c0742a0 2024-09-19T07:49:21.313 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-19T07:49:21.313 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-19T07:49:21.313 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-19T07:49:21.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f27b5af0640 1 -- 172.21.15.83:0/1770118249 >> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] conn(0x7f279c0742a0 msgr2=0x7f279c076720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f27b5af0640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.184:6808/3839042106,v1:172.21.15.184:6809/3839042106] conn(0x7f279c0742a0 0x7f279c076720 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f27b5af0640 1 -- 172.21.15.83:0/1770118249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f279c0678c0 msgr2=0x7f279c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f27b5af0640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f279c0678c0 0x7f279c069d80 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f27a0007f40 tx=0x7f27a0007490 comp rx=0 tx=0).stop 2024-09-19T07:49:21.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f27b5af0640 1 -- 172.21.15.83:0/1770118249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27b0155af0 msgr2=0x7f27b0159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f27b5af0640 1 --2- 172.21.15.83:0/1770118249 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27b0155af0 0x7f27b0159ab0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f27a800c480 tx=0x7f27a800c940 comp rx=0 tx=0).stop 2024-09-19T07:49:21.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f27affff640 1 -- 172.21.15.83:0/1770118249 reap_dead start 2024-09-19T07:49:21.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f27b5af0640 1 -- 172.21.15.83:0/1770118249 shutdown_connections 2024-09-19T07:49:21.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f27b5af0640 1 -- 172.21.15.83:0/1770118249 >> 172.21.15.83:0/1770118249 conn(0x7f27b006bc80 msgr2=0x7f27b00aab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:21.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f27b5af0640 1 -- 172.21.15.83:0/1770118249 shutdown_connections 2024-09-19T07:49:21.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.312+0000 7f27b5af0640 1 -- 172.21.15.83:0/1770118249 wait complete. 2024-09-19T07:49:21.327 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-19T07:49:21.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f08114fd640 1 Processor -- start 2024-09-19T07:49:21.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f08114fd640 1 -- start start 2024-09-19T07:49:21.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f08114fd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f080c0c8d40 0x7f080c1532b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f08114fd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f080c1537f0 0x7f080c0ca140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f08114fd640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f080c155f90 con 0x7f080c0c6370 2024-09-19T07:49:21.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f08114fd640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f080c156b30 con 0x7f080c1537f0 2024-09-19T07:49:21.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f08114fd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f080c156200 con 0x7f080c0c8d40 2024-09-19T07:49:21.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f080b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f080c1537f0 0x7f080c0ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f080b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f080c1537f0 0x7f080c0ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50770/0 (socket says 172.21.15.83:50770) 2024-09-19T07:49:21.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f080b7fe640 1 -- 172.21.15.83:0/1673877239 learned_addr learned my addr 172.21.15.83:0/1673877239 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:21.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f080b7fe640 1 -- 172.21.15.83:0/1673877239 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f080c0c8d40 msgr2=0x7f080c1532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:21.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f080b7fe640 1 --2- 172.21.15.83:0/1673877239 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f080c0c8d40 0x7f080c1532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f080b7fe640 1 -- 172.21.15.83:0/1673877239 >> v1:172.21.15.184:6789/0 conn(0x7f080c0c6370 legacy=0x7f080c159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:21.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f080b7fe640 1 -- 172.21.15.83:0/1673877239 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f080c1564e0 con 0x7f080c1537f0 2024-09-19T07:49:21.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f080b7fe640 1 --2- 172.21.15.83:0/1673877239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f080c1537f0 0x7f080c0ca140 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f0804009e50 tx=0x7f080401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf28c3ab1df1860 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f0809ffb640 1 -- 172.21.15.83:0/1673877239 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08040a56f0 con 0x7f080c1537f0 2024-09-19T07:49:21.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f0809ffb640 1 -- 172.21.15.83:0/1673877239 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08040a5890 con 0x7f080c1537f0 2024-09-19T07:49:21.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.432+0000 7f0809ffb640 1 -- 172.21.15.83:0/1673877239 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08040a5c60 con 0x7f080c1537f0 2024-09-19T07:49:21.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 -- 172.21.15.83:0/1673877239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f080c1537f0 msgr2=0x7f080c0ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 --2- 172.21.15.83:0/1673877239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f080c1537f0 0x7f080c0ca140 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f0804009e50 tx=0x7f080401c960 comp rx=0 tx=0).stop 2024-09-19T07:49:21.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 -- 172.21.15.83:0/1673877239 shutdown_connections 2024-09-19T07:49:21.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 --2- 172.21.15.83:0/1673877239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f080c1537f0 0x7f080c0ca140 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 --2- 172.21.15.83:0/1673877239 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f080c0c8d40 0x7f080c1532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 -- 172.21.15.83:0/1673877239 >> 172.21.15.83:0/1673877239 conn(0x7f080c06bc80 msgr2=0x7f080c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:21.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 -- 172.21.15.83:0/1673877239 shutdown_connections 2024-09-19T07:49:21.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 -- 172.21.15.83:0/1673877239 wait complete. 2024-09-19T07:49:21.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 Processor -- start 2024-09-19T07:49:21.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 -- start start 2024-09-19T07:49:21.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f080c0c6370 0x7f080c145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f080c0c8d40 0x7f080c1460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f080c1537f0 0x7f080c0abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f080c162ba0 con 0x7f080c0c6370 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f080c162a20 con 0x7f080c1537f0 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f080c162d20 con 0x7f080c0c8d40 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f080affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f080c0c6370 0x7f080c145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f080affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f080c0c6370 0x7f080c145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50784/0 (socket says 172.21.15.83:50784) 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f080affd640 1 -- 172.21.15.83:0/4200254053 learned_addr learned my addr 172.21.15.83:0/4200254053 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f080a7fc640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f080c0c8d40 0x7f080c1460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f080b7fe640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f080c1537f0 0x7f080c0abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f080affd640 1 -- 172.21.15.83:0/4200254053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f080c0c8d40 msgr2=0x7f080c1460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f080affd640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f080c0c8d40 0x7f080c1460c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f080affd640 1 -- 172.21.15.83:0/4200254053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f080c1537f0 msgr2=0x7f080c0abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f080affd640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f080c1537f0 0x7f080c0abc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f080affd640 1 -- 172.21.15.83:0/4200254053 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f080c0c20f0 con 0x7f080c0c6370 2024-09-19T07:49:21.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f080affd640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f080c0c6370 0x7f080c145b80 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f07fc00b3e0 tx=0x7f07fc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f07f37fe640 1 -- 172.21.15.83:0/4200254053 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc015020 con 0x7f080c0c6370 2024-09-19T07:49:21.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f07f37fe640 1 -- 172.21.15.83:0/4200254053 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07fc011850 con 0x7f080c0c6370 2024-09-19T07:49:21.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 -- 172.21.15.83:0/4200254053 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f080c0c23d0 con 0x7f080c0c6370 2024-09-19T07:49:21.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f07f37fe640 1 -- 172.21.15.83:0/4200254053 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc011c40 con 0x7f080c0c6370 2024-09-19T07:49:21.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 -- 172.21.15.83:0/4200254053 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f080c0ab720 con 0x7f080c0c6370 2024-09-19T07:49:21.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f08114fd640 1 -- 172.21.15.83:0/4200254053 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f080c1532b0 con 0x7f080c0c6370 2024-09-19T07:49:21.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.436+0000 7f07f37fe640 1 -- 172.21.15.83:0/4200254053 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f07fc018020 con 0x7f080c0c6370 2024-09-19T07:49:21.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.440+0000 7f07f37fe640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07f80678c0 0x7f07f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.440+0000 7f07f37fe640 1 -- 172.21.15.83:0/4200254053 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f07fc086d40 con 0x7f080c0c6370 2024-09-19T07:49:21.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.440+0000 7f080a7fc640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07f80678c0 0x7f07f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.440+0000 7f080a7fc640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07f80678c0 0x7f07f8069d80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f080c153260 tx=0x7f0800008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.444+0000 7f07f37fe640 1 -- 172.21.15.83:0/4200254053 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f07fc01f020 con 0x7f080c0c6370 2024-09-19T07:49:21.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.600+0000 7f08114fd640 1 -- 172.21.15.83:0/4200254053 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f080c0ab290 con 0x7f07f80678c0 2024-09-19T07:49:21.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.600+0000 7f07f37fe640 1 -- 172.21.15.83:0/4200254053 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f080c0ab290 con 0x7f07f80678c0 2024-09-19T07:49:21.601 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 4 to deep-scrub 2024-09-19T07:49:21.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.604+0000 7f08114fd640 1 -- 172.21.15.83:0/4200254053 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07f80678c0 msgr2=0x7f07f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.604+0000 7f08114fd640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07f80678c0 0x7f07f8069d80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f080c153260 tx=0x7f0800008040 comp rx=0 tx=0).stop 2024-09-19T07:49:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.604+0000 7f08114fd640 1 -- 172.21.15.83:0/4200254053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f080c0c6370 msgr2=0x7f080c145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.604+0000 7f08114fd640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f080c0c6370 0x7f080c145b80 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f07fc00b3e0 tx=0x7f07fc00b8a0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.604+0000 7f08114fd640 1 -- 172.21.15.83:0/4200254053 shutdown_connections 2024-09-19T07:49:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.604+0000 7f08114fd640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f07f80678c0 0x7f07f8069d80 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.604+0000 7f08114fd640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f080c1537f0 0x7f080c0abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.604+0000 7f08114fd640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f080c0c8d40 0x7f080c1460c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.604+0000 7f08114fd640 1 --2- 172.21.15.83:0/4200254053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f080c0c6370 0x7f080c145b80 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.604+0000 7f08114fd640 1 -- 172.21.15.83:0/4200254053 >> 172.21.15.83:0/4200254053 conn(0x7f080c06bc80 msgr2=0x7f080c154790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.604+0000 7f08114fd640 1 -- 172.21.15.83:0/4200254053 shutdown_connections 2024-09-19T07:49:21.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.604+0000 7f08114fd640 1 -- 172.21.15.83:0/4200254053 wait complete. 2024-09-19T07:49:21.620 INFO:tasks.ceph:Scrubbing osd.5 2024-09-19T07:49:21.620 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-19T07:49:21.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8967ee7640 1 Processor -- start 2024-09-19T07:49:21.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8967ee7640 1 -- start start 2024-09-19T07:49:21.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8967ee7640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f8960153500 0x7f8960156660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8967ee7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f89600c7c80 0x7f89600c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8967ee7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89600c8d40 con 0x7f89600c3740 2024-09-19T07:49:21.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8967ee7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f89600c7510 con 0x7f89600c7c80 2024-09-19T07:49:21.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8967ee7640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f89601538e0 con 0x7f8960153500 2024-09-19T07:49:21.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8966484640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f89600c7c80 0x7f89600c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8966484640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f89600c7c80 0x7f89600c80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50796/0 (socket says 172.21.15.83:50796) 2024-09-19T07:49:21.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8966484640 1 -- 172.21.15.83:0/2439653420 learned_addr learned my addr 172.21.15.83:0/2439653420 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:21.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8965c83640 1 --2- 172.21.15.83:0/2439653420 >> v2:172.21.15.184:3300/0 conn(0x7f8960153500 0x7f8960156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8966484640 1 -- 172.21.15.83:0/2439653420 >> v2:172.21.15.184:3300/0 conn(0x7f8960153500 msgr2=0x7f8960156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:21.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8966484640 1 --2- 172.21.15.83:0/2439653420 >> v2:172.21.15.184:3300/0 conn(0x7f8960153500 0x7f8960156660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8966484640 1 -- 172.21.15.83:0/2439653420 >> v1:172.21.15.184:6789/0 conn(0x7f89600c3740 legacy=0x7f89600c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:21.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.744+0000 7f8966484640 1 -- 172.21.15.83:0/2439653420 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89600c9020 con 0x7f89600c7c80 2024-09-19T07:49:21.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8966484640 1 --2- 172.21.15.83:0/2439653420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89600c7c80 0x7f89600c80b0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f8960071bd0 tx=0x7f895801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b2217fe4a29337 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8964c81640 1 -- 172.21.15.83:0/2439653420 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89580b0070 con 0x7f89600c7c80 2024-09-19T07:49:21.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8964c81640 1 -- 172.21.15.83:0/2439653420 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89580ab600 con 0x7f89600c7c80 2024-09-19T07:49:21.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8964c81640 1 -- 172.21.15.83:0/2439653420 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89580ab9d0 con 0x7f89600c7c80 2024-09-19T07:49:21.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 -- 172.21.15.83:0/2439653420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89600c7c80 msgr2=0x7f89600c80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 --2- 172.21.15.83:0/2439653420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89600c7c80 0x7f89600c80b0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f8960071bd0 tx=0x7f895801c450 comp rx=0 tx=0).stop 2024-09-19T07:49:21.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 -- 172.21.15.83:0/2439653420 shutdown_connections 2024-09-19T07:49:21.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 --2- 172.21.15.83:0/2439653420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89600c7c80 0x7f89600c80b0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 --2- 172.21.15.83:0/2439653420 >> v2:172.21.15.184:3300/0 conn(0x7f8960153500 0x7f8960156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 -- 172.21.15.83:0/2439653420 >> 172.21.15.83:0/2439653420 conn(0x7f89600c6110 msgr2=0x7f89600c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:21.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 -- 172.21.15.83:0/2439653420 shutdown_connections 2024-09-19T07:49:21.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 -- 172.21.15.83:0/2439653420 wait complete. 2024-09-19T07:49:21.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 Processor -- start 2024-09-19T07:49:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 -- start start 2024-09-19T07:49:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89600c3740 0x7f8960148090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89600c7c80 0x7f8960155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8960153500 0x7f8960155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89600c3150 con 0x7f89600c7c80 2024-09-19T07:49:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f89600c4010 con 0x7f8960153500 2024-09-19T07:49:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8967ee7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f89600c32d0 con 0x7f89600c3740 2024-09-19T07:49:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8965482640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89600c7c80 0x7f8960155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8965482640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89600c7c80 0x7f8960155710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50800/0 (socket says 172.21.15.83:50800) 2024-09-19T07:49:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8965482640 1 -- 172.21.15.83:0/2425013456 learned_addr learned my addr 172.21.15.83:0/2425013456 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8966484640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8960153500 0x7f8960155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8965c83640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89600c3740 0x7f8960148090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8965482640 1 -- 172.21.15.83:0/2425013456 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89600c3740 msgr2=0x7f8960148090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8965482640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89600c3740 0x7f8960148090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8965482640 1 -- 172.21.15.83:0/2425013456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8960153500 msgr2=0x7f8960155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8965482640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8960153500 0x7f8960155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8965482640 1 -- 172.21.15.83:0/2425013456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89601485d0 con 0x7f89600c7c80 2024-09-19T07:49:21.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.748+0000 7f8966484640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8960153500 0x7f8960155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:21.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f8965482640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89600c7c80 0x7f8960155710 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f895000c480 tx=0x7f895000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f89567fc640 1 -- 172.21.15.83:0/2425013456 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950012830 con 0x7f89600c7c80 2024-09-19T07:49:21.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f89567fc640 1 -- 172.21.15.83:0/2425013456 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8950017070 con 0x7f89600c7c80 2024-09-19T07:49:21.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f8967ee7640 1 -- 172.21.15.83:0/2425013456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89601f2610 con 0x7f89600c7c80 2024-09-19T07:49:21.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f89567fc640 1 -- 172.21.15.83:0/2425013456 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950012c30 con 0x7f89600c7c80 2024-09-19T07:49:21.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f8967ee7640 1 -- 172.21.15.83:0/2425013456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8960153bd0 con 0x7f89600c7c80 2024-09-19T07:49:21.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f8967ee7640 1 -- 172.21.15.83:0/2425013456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f89600c80b0 con 0x7f89600c7c80 2024-09-19T07:49:21.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f89567fc640 1 -- 172.21.15.83:0/2425013456 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f895001c460 con 0x7f89600c7c80 2024-09-19T07:49:21.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f89567fc640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f89440678c0 0x7f8944069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f89567fc640 1 -- 172.21.15.83:0/2425013456 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f8950086b40 con 0x7f89600c7c80 2024-09-19T07:49:21.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f8965c83640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f89440678c0 0x7f8944069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f89567fc640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] conn(0x7f8944074290 0x7f8944076710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f89567fc640 1 -- 172.21.15.83:0/2425013456 --> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8944076de0 con 0x7f8944074290 2024-09-19T07:49:21.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f89567fc640 1 -- 172.21.15.83:0/2425013456 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f895008d050 con 0x7f89600c7c80 2024-09-19T07:49:21.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f8965c83640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f89440678c0 0x7f8944069d80 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f89601561e0 tx=0x7f895c0083c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f8966484640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] conn(0x7f8944074290 0x7f8944076710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.752+0000 7f8966484640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] conn(0x7f8944074290 0x7f8944076710 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.756+0000 7f89567fc640 1 -- 172.21.15.83:0/2425013456 <== osd.5 v2:172.21.15.184:6818/767241772 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8944076de0 con 0x7f8944074290 2024-09-19T07:49:21.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f8967ee7640 1 -- 172.21.15.83:0/2425013456 --> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f89600c3520 con 0x7f8944074290 2024-09-19T07:49:21.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f89567fc640 1 -- 172.21.15.83:0/2425013456 <== osd.5 v2:172.21.15.184:6818/767241772 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f89600c3520 con 0x7f8944074290 2024-09-19T07:49:21.769 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-19T07:49:21.769 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-19T07:49:21.769 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-19T07:49:21.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f8967ee7640 1 -- 172.21.15.83:0/2425013456 >> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] conn(0x7f8944074290 msgr2=0x7f8944076710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f8967ee7640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.184:6818/767241772,v1:172.21.15.184:6819/767241772] conn(0x7f8944074290 0x7f8944076710 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f8967ee7640 1 -- 172.21.15.83:0/2425013456 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f89440678c0 msgr2=0x7f8944069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f8967ee7640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f89440678c0 0x7f8944069d80 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f89601561e0 tx=0x7f895c0083c0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f8967ee7640 1 -- 172.21.15.83:0/2425013456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89600c7c80 msgr2=0x7f8960155710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f8967ee7640 1 --2- 172.21.15.83:0/2425013456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89600c7c80 0x7f8960155710 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f895000c480 tx=0x7f895000c940 comp rx=0 tx=0).stop 2024-09-19T07:49:21.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f8966484640 1 -- 172.21.15.83:0/2425013456 reap_dead start 2024-09-19T07:49:21.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f8967ee7640 1 -- 172.21.15.83:0/2425013456 shutdown_connections 2024-09-19T07:49:21.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f8967ee7640 1 -- 172.21.15.83:0/2425013456 >> 172.21.15.83:0/2425013456 conn(0x7f89600c6110 msgr2=0x7f89601519d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:21.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f8967ee7640 1 -- 172.21.15.83:0/2425013456 shutdown_connections 2024-09-19T07:49:21.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.768+0000 7f8967ee7640 1 -- 172.21.15.83:0/2425013456 wait complete. 2024-09-19T07:49:21.784 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-19T07:49:21.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f8236202640 1 Processor -- start 2024-09-19T07:49:21.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f8236202640 1 -- start start 2024-09-19T07:49:21.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f8236202640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f82301534b0 0x7f82300c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f8236202640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f82300c2b10 0x7f823015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f8236202640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82300c3810 con 0x7f8230150060 2024-09-19T07:49:21.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f8236202640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f8230150440 con 0x7f82300c2b10 2024-09-19T07:49:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f8236202640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f82301506c0 con 0x7f82301534b0 2024-09-19T07:49:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f822ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f82300c2b10 0x7f823015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f822ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f82300c2b10 0x7f823015d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50804/0 (socket says 172.21.15.83:50804) 2024-09-19T07:49:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f822ffff640 1 -- 172.21.15.83:0/2263343538 learned_addr learned my addr 172.21.15.83:0/2263343538 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f822ffff640 1 -- 172.21.15.83:0/2263343538 >> v2:172.21.15.184:3300/0 conn(0x7f82301534b0 msgr2=0x7f82300c25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f822effd640 1 --2- 172.21.15.83:0/2263343538 >> v2:172.21.15.184:3300/0 conn(0x7f82301534b0 0x7f82300c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.888+0000 7f822ffff640 1 --2- 172.21.15.83:0/2263343538 >> v2:172.21.15.184:3300/0 conn(0x7f82301534b0 0x7f82300c25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f822ffff640 1 -- 172.21.15.83:0/2263343538 >> v1:172.21.15.184:6789/0 conn(0x7f8230150060 legacy=0x7f82300c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:21.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f822ffff640 1 -- 172.21.15.83:0/2263343538 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82300c3ab0 con 0x7f82300c2b10 2024-09-19T07:49:21.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f822ffff640 1 --2- 172.21.15.83:0/2263343538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82300c2b10 0x7f823015d3b0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f82300c1850 tx=0x7f822801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934e1f038e76a78 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f822e7fc640 1 -- 172.21.15.83:0/2263343538 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82280a55e0 con 0x7f82300c2b10 2024-09-19T07:49:21.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f822e7fc640 1 -- 172.21.15.83:0/2263343538 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82280a5780 con 0x7f82300c2b10 2024-09-19T07:49:21.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f822e7fc640 1 -- 172.21.15.83:0/2263343538 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82280a5b50 con 0x7f82300c2b10 2024-09-19T07:49:21.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f8236202640 1 -- 172.21.15.83:0/2263343538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82300c2b10 msgr2=0x7f823015d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f8236202640 1 --2- 172.21.15.83:0/2263343538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82300c2b10 0x7f823015d3b0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f82300c1850 tx=0x7f822801c920 comp rx=0 tx=0).stop 2024-09-19T07:49:21.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f8236202640 1 -- 172.21.15.83:0/2263343538 shutdown_connections 2024-09-19T07:49:21.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f8236202640 1 --2- 172.21.15.83:0/2263343538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82300c2b10 0x7f823015d3b0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f8236202640 1 --2- 172.21.15.83:0/2263343538 >> v2:172.21.15.184:3300/0 conn(0x7f82301534b0 0x7f82300c25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f8236202640 1 -- 172.21.15.83:0/2263343538 >> 172.21.15.83:0/2263343538 conn(0x7f823006bc80 msgr2=0x7f823006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:21.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f8236202640 1 -- 172.21.15.83:0/2263343538 shutdown_connections 2024-09-19T07:49:21.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f8236202640 1 -- 172.21.15.83:0/2263343538 wait complete. 2024-09-19T07:49:21.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f8236202640 1 Processor -- start 2024-09-19T07:49:21.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.892+0000 7f8236202640 1 -- start start 2024-09-19T07:49:21.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8236202640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82300c2b10 0x7f82300c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8236202640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8230150060 0x7f8230149300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8236202640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82301534b0 0x7f8230149840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8236202640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f82300aae10 con 0x7f8230150060 2024-09-19T07:49:21.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8236202640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f82300c3d90 con 0x7f82301534b0 2024-09-19T07:49:21.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8236202640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f82300aaf90 con 0x7f82300c2b10 2024-09-19T07:49:21.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8230150060 0x7f8230149300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8230150060 0x7f8230149300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50810/0 (socket says 172.21.15.83:50810) 2024-09-19T07:49:21.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822effd640 1 -- 172.21.15.83:0/1530046049 learned_addr learned my addr 172.21.15.83:0/1530046049 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:21.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822ffff640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82301534b0 0x7f8230149840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822f7fe640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82300c2b10 0x7f82300c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822effd640 1 -- 172.21.15.83:0/1530046049 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82300c2b10 msgr2=0x7f82300c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822effd640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82300c2b10 0x7f82300c6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822effd640 1 -- 172.21.15.83:0/1530046049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82301534b0 msgr2=0x7f8230149840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:21.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822effd640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82301534b0 0x7f8230149840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:21.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822effd640 1 -- 172.21.15.83:0/1530046049 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8230154ee0 con 0x7f8230150060 2024-09-19T07:49:21.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822effd640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8230150060 0x7f8230149300 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f822000c820 tx=0x7f822000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8217fff640 1 -- 172.21.15.83:0/1530046049 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8220015070 con 0x7f8230150060 2024-09-19T07:49:21.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8217fff640 1 -- 172.21.15.83:0/1530046049 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8220004020 con 0x7f8230150060 2024-09-19T07:49:21.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8217fff640 1 -- 172.21.15.83:0/1530046049 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8220004450 con 0x7f8230150060 2024-09-19T07:49:21.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8236202640 1 -- 172.21.15.83:0/1530046049 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8230156620 con 0x7f8230150060 2024-09-19T07:49:21.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8236202640 1 -- 172.21.15.83:0/1530046049 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8230155460 con 0x7f8230150060 2024-09-19T07:49:21.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8236202640 1 -- 172.21.15.83:0/1530046049 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82300c1ab0 con 0x7f8230150060 2024-09-19T07:49:21.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8217fff640 1 -- 172.21.15.83:0/1530046049 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8220007830 con 0x7f8230150060 2024-09-19T07:49:21.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8217fff640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f821c0678c0 0x7f821c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:21.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f8217fff640 1 -- 172.21.15.83:0/1530046049 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f8220086c10 con 0x7f8230150060 2024-09-19T07:49:21.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822f7fe640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f821c0678c0 0x7f821c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:21.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.896+0000 7f822f7fe640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f821c0678c0 0x7f821c069d80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f822400a790 tx=0x7f8224008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:21.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:21.900+0000 7f8217fff640 1 -- 172.21.15.83:0/1530046049 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f82200565f0 con 0x7f8230150060 2024-09-19T07:49:22.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.048+0000 7f8236202640 1 -- 172.21.15.83:0/1530046049 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f8230156ec0 con 0x7f821c0678c0 2024-09-19T07:49:22.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8217fff640 1 -- 172.21.15.83:0/1530046049 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8230156ec0 con 0x7f821c0678c0 2024-09-19T07:49:22.050 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 5 to deep-scrub 2024-09-19T07:49:22.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8236202640 1 -- 172.21.15.83:0/1530046049 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f821c0678c0 msgr2=0x7f821c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8236202640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f821c0678c0 0x7f821c069d80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f822400a790 tx=0x7f8224008040 comp rx=0 tx=0).stop 2024-09-19T07:49:22.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8236202640 1 -- 172.21.15.83:0/1530046049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8230150060 msgr2=0x7f8230149300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8236202640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8230150060 0x7f8230149300 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f822000c820 tx=0x7f822000cce0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8236202640 1 -- 172.21.15.83:0/1530046049 shutdown_connections 2024-09-19T07:49:22.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8236202640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f821c0678c0 0x7f821c069d80 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8236202640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82301534b0 0x7f8230149840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8236202640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8230150060 0x7f8230149300 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8236202640 1 --2- 172.21.15.83:0/1530046049 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82300c2b10 0x7f82300c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8236202640 1 -- 172.21.15.83:0/1530046049 >> 172.21.15.83:0/1530046049 conn(0x7f823006bc80 msgr2=0x7f82300c09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:22.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8236202640 1 -- 172.21.15.83:0/1530046049 shutdown_connections 2024-09-19T07:49:22.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.052+0000 7f8236202640 1 -- 172.21.15.83:0/1530046049 wait complete. 2024-09-19T07:49:22.069 INFO:tasks.ceph:Scrubbing osd.6 2024-09-19T07:49:22.069 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-19T07:49:22.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.192+0000 7f77ca78b640 1 Processor -- start 2024-09-19T07:49:22.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.192+0000 7f77ca78b640 1 -- start start 2024-09-19T07:49:22.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.192+0000 7f77ca78b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f77c40c9670 0x7f77c40c8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.192+0000 7f77ca78b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77c4155f00 con 0x7f77c40bee90 2024-09-19T07:49:22.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.192+0000 7f77ca78b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77c41561e0 con 0x7f77c414fde0 2024-09-19T07:49:22.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.192+0000 7f77ca78b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f77c41564c0 con 0x7f77c40c9670 2024-09-19T07:49:22.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.192+0000 7f77c8d28640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f77c40c9670 0x7f77c40c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.192+0000 7f77bb7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f77c40bee90 0x7f77c40c8180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35904/0 (socket says 172.21.15.83:35904) 2024-09-19T07:49:22.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.192+0000 7f77bb7fe640 1 -- 172.21.15.83:0/1650690979 learned_addr learned my addr 172.21.15.83:0/1650690979 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:22.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77c8d28640 1 -- 172.21.15.83:0/1650690979 >> v1:172.21.15.184:6789/0 conn(0x7f77c414fde0 legacy=0x7f77c40cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:22.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77c8d28640 1 -- 172.21.15.83:0/1650690979 >> v1:172.21.15.83:6789/0 conn(0x7f77c40bee90 legacy=0x7f77c40c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77c37fe640 1 -- 172.21.15.83:0/1650690979 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 861717263 0 0) 0x7f77c4155f00 con 0x7f77c40bee90 2024-09-19T07:49:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77c8d28640 1 -- 172.21.15.83:0/1650690979 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c41567a0 con 0x7f77c40c9670 2024-09-19T07:49:22.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77c8d28640 1 --2- 172.21.15.83:0/1650690979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c40c9670 0x7f77c40c8890 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009d90 tx=0x7f77bc01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdfaea85d08e3364 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77c37fe640 1 -- 172.21.15.83:0/1650690979 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc0a66f0 con 0x7f77c40c9670 2024-09-19T07:49:22.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77c37fe640 1 -- 172.21.15.83:0/1650690979 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77bc0a6890 con 0x7f77c40c9670 2024-09-19T07:49:22.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77c37fe640 1 -- 172.21.15.83:0/1650690979 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc0a6c60 con 0x7f77c40c9670 2024-09-19T07:49:22.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77ca78b640 1 -- 172.21.15.83:0/1650690979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c40c9670 msgr2=0x7f77c40c8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77ca78b640 1 --2- 172.21.15.83:0/1650690979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c40c9670 0x7f77c40c8890 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009d90 tx=0x7f77bc01c930 comp rx=0 tx=0).stop 2024-09-19T07:49:22.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77ca78b640 1 -- 172.21.15.83:0/1650690979 shutdown_connections 2024-09-19T07:49:22.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77ca78b640 1 --2- 172.21.15.83:0/1650690979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c40c9670 0x7f77c40c8890 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77ca78b640 1 -- 172.21.15.83:0/1650690979 >> 172.21.15.83:0/1650690979 conn(0x7f77c40726a0 msgr2=0x7f77c4072ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:22.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77ca78b640 1 -- 172.21.15.83:0/1650690979 shutdown_connections 2024-09-19T07:49:22.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.196+0000 7f77ca78b640 1 -- 172.21.15.83:0/1650690979 wait complete. 2024-09-19T07:49:22.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77ca78b640 1 Processor -- start 2024-09-19T07:49:22.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77ca78b640 1 -- start start 2024-09-19T07:49:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77ca78b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77c40bee90 0x7f77c40af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77ca78b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77c40c9670 0x7f77c41ef460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77ca78b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c414fde0 0x7f77c40c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77ca78b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f77c4164230 con 0x7f77c414fde0 2024-09-19T07:49:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77ca78b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f77c41640b0 con 0x7f77c40c9670 2024-09-19T07:49:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77ca78b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f77c41643b0 con 0x7f77c40bee90 2024-09-19T07:49:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c8d28640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c414fde0 0x7f77c40c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c8d28640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c414fde0 0x7f77c40c4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50828/0 (socket says 172.21.15.83:50828) 2024-09-19T07:49:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c8d28640 1 -- 172.21.15.83:0/3645675608 learned_addr learned my addr 172.21.15.83:0/3645675608 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77bb7fe640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77c40c9670 0x7f77c41ef460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c8d28640 1 -- 172.21.15.83:0/3645675608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77c40bee90 msgr2=0x7f77c40af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c3fff640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77c40bee90 0x7f77c40af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c8d28640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77c40bee90 0x7f77c40af010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c8d28640 1 -- 172.21.15.83:0/3645675608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77c40c9670 msgr2=0x7f77c41ef460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c8d28640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77c40c9670 0x7f77c41ef460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c8d28640 1 -- 172.21.15.83:0/3645675608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c40c7580 con 0x7f77c414fde0 2024-09-19T07:49:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77bb7fe640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77c40c9670 0x7f77c41ef460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c8d28640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c414fde0 0x7f77c40c4c10 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009d90 tx=0x7f77bc0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c17fa640 1 -- 172.21.15.83:0/3645675608 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc0b8840 con 0x7f77c414fde0 2024-09-19T07:49:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c17fa640 1 -- 172.21.15.83:0/3645675608 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77bc0c1070 con 0x7f77c414fde0 2024-09-19T07:49:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77ca78b640 1 -- 172.21.15.83:0/3645675608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c40c23b0 con 0x7f77c414fde0 2024-09-19T07:49:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c17fa640 1 -- 172.21.15.83:0/3645675608 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc0b8c40 con 0x7f77c414fde0 2024-09-19T07:49:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77ca78b640 1 -- 172.21.15.83:0/3645675608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c40ae810 con 0x7f77c414fde0 2024-09-19T07:49:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77ca78b640 1 -- 172.21.15.83:0/3645675608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f77c4154070 con 0x7f77c414fde0 2024-09-19T07:49:22.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c17fa640 1 -- 172.21.15.83:0/3645675608 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f77bc0c6460 con 0x7f77c414fde0 2024-09-19T07:49:22.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.200+0000 7f77c17fa640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77a40678c0 0x7f77a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.204+0000 7f77c17fa640 1 -- 172.21.15.83:0/3645675608 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f77bc122c20 con 0x7f77c414fde0 2024-09-19T07:49:22.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.204+0000 7f77c3fff640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77a40678c0 0x7f77a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.204+0000 7f77c17fa640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] conn(0x7f77a4074290 0x7f77a4076710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.204+0000 7f77c17fa640 1 -- 172.21.15.83:0/3645675608 --> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77bc0c7e50 con 0x7f77a4074290 2024-09-19T07:49:22.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.204+0000 7f77c17fa640 1 -- 172.21.15.83:0/3645675608 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f77bc123070 con 0x7f77c414fde0 2024-09-19T07:49:22.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.204+0000 7f77c3fff640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77a40678c0 0x7f77a4069d80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f77b400a750 tx=0x7f77b4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.204+0000 7f77bb7fe640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] conn(0x7f77a4074290 0x7f77a4076710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.204+0000 7f77bb7fe640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] conn(0x7f77a4074290 0x7f77a4076710 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.204+0000 7f77c17fa640 1 -- 172.21.15.83:0/3645675608 <== osd.6 v2:172.21.15.184:6800/1304502305 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77bc0c7e50 con 0x7f77a4074290 2024-09-19T07:49:22.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.216+0000 7f77ca78b640 1 -- 172.21.15.83:0/3645675608 --> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f77c40cbcc0 con 0x7f77a4074290 2024-09-19T07:49:22.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.220+0000 7f77c17fa640 1 -- 172.21.15.83:0/3645675608 <== osd.6 v2:172.21.15.184:6800/1304502305 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f77c40cbcc0 con 0x7f77a4074290 2024-09-19T07:49:22.219 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-19T07:49:22.219 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-19T07:49:22.219 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-19T07:49:22.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.220+0000 7f77ca78b640 1 -- 172.21.15.83:0/3645675608 >> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] conn(0x7f77a4074290 msgr2=0x7f77a4076710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.220+0000 7f77ca78b640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.184:6800/1304502305,v1:172.21.15.184:6801/1304502305] conn(0x7f77a4074290 0x7f77a4076710 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.220+0000 7f77ca78b640 1 -- 172.21.15.83:0/3645675608 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77a40678c0 msgr2=0x7f77a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.220+0000 7f77ca78b640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f77a40678c0 0x7f77a4069d80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f77b400a750 tx=0x7f77b4008040 comp rx=0 tx=0).stop 2024-09-19T07:49:22.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.220+0000 7f77ca78b640 1 -- 172.21.15.83:0/3645675608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c414fde0 msgr2=0x7f77c40c4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.220+0000 7f77ca78b640 1 --2- 172.21.15.83:0/3645675608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c414fde0 0x7f77c40c4c10 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009d90 tx=0x7f77bc0196d0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.220+0000 7f77c8d28640 1 -- 172.21.15.83:0/3645675608 reap_dead start 2024-09-19T07:49:22.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.220+0000 7f77ca78b640 1 -- 172.21.15.83:0/3645675608 shutdown_connections 2024-09-19T07:49:22.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.220+0000 7f77ca78b640 1 -- 172.21.15.83:0/3645675608 >> 172.21.15.83:0/3645675608 conn(0x7f77c40726a0 msgr2=0x7f77c4155730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:22.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.220+0000 7f77ca78b640 1 -- 172.21.15.83:0/3645675608 shutdown_connections 2024-09-19T07:49:22.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.220+0000 7f77ca78b640 1 -- 172.21.15.83:0/3645675608 wait complete. 2024-09-19T07:49:22.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-19T07:49:22.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab190b5640 1 Processor -- start 2024-09-19T07:49:22.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab190b5640 1 -- start start 2024-09-19T07:49:22.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab190b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab140c4450 0x7fab141562d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab190b5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab14156810 0x7fab14160620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab190b5640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab14153930 con 0x7fab14153550 2024-09-19T07:49:22.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab190b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fab14155e40 con 0x7fab140c4450 2024-09-19T07:49:22.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab190b5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fab14156ca0 con 0x7fab14156810 2024-09-19T07:49:22.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab1259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab140c4450 0x7fab141562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab1259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab140c4450 0x7fab141562d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50830/0 (socket says 172.21.15.83:50830) 2024-09-19T07:49:22.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab1259c640 1 -- 172.21.15.83:0/638197009 learned_addr learned my addr 172.21.15.83:0/638197009 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:22.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab1359e640 1 --2- 172.21.15.83:0/638197009 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab14156810 0x7fab14160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab1259c640 1 -- 172.21.15.83:0/638197009 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab14156810 msgr2=0x7fab14160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:22.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab1259c640 1 --2- 172.21.15.83:0/638197009 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab14156810 0x7fab14160620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab1259c640 1 -- 172.21.15.83:0/638197009 >> v1:172.21.15.184:6789/0 conn(0x7fab14153550 legacy=0x7fab140c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.360+0000 7fab1259c640 1 -- 172.21.15.83:0/638197009 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab140adb10 con 0x7fab140c4450 2024-09-19T07:49:22.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.364+0000 7fab1259c640 1 --2- 172.21.15.83:0/638197009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab140c4450 0x7fab141562d0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fab08009870 tx=0x7fab0801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c687e4a8ed6fc9a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.364+0000 7fab11d9b640 1 -- 172.21.15.83:0/638197009 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab080ab650 con 0x7fab140c4450 2024-09-19T07:49:22.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.364+0000 7fab11d9b640 1 -- 172.21.15.83:0/638197009 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab080b0070 con 0x7fab140c4450 2024-09-19T07:49:22.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.364+0000 7fab11d9b640 1 -- 172.21.15.83:0/638197009 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab080aba10 con 0x7fab140c4450 2024-09-19T07:49:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.364+0000 7fab190b5640 1 -- 172.21.15.83:0/638197009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab140c4450 msgr2=0x7fab141562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.364+0000 7fab190b5640 1 --2- 172.21.15.83:0/638197009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab140c4450 0x7fab141562d0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fab08009870 tx=0x7fab0801c560 comp rx=0 tx=0).stop 2024-09-19T07:49:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.364+0000 7fab190b5640 1 -- 172.21.15.83:0/638197009 shutdown_connections 2024-09-19T07:49:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.364+0000 7fab190b5640 1 --2- 172.21.15.83:0/638197009 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab14156810 0x7fab14160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.364+0000 7fab190b5640 1 --2- 172.21.15.83:0/638197009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab140c4450 0x7fab141562d0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.364+0000 7fab190b5640 1 -- 172.21.15.83:0/638197009 >> 172.21.15.83:0/638197009 conn(0x7fab140c48b0 msgr2=0x7fab140c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.364+0000 7fab190b5640 1 -- 172.21.15.83:0/638197009 shutdown_connections 2024-09-19T07:49:22.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.364+0000 7fab190b5640 1 -- 172.21.15.83:0/638197009 wait complete. 2024-09-19T07:49:22.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab190b5640 1 Processor -- start 2024-09-19T07:49:22.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab190b5640 1 -- start start 2024-09-19T07:49:22.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab190b5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab140c4450 0x7fab140c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab190b5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab14153550 0x7fab14073440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab190b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab14156810 0x7fab14143810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab190b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fab140c3ed0 con 0x7fab14156810 2024-09-19T07:49:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab190b5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab14068290 con 0x7fab14153550 2024-09-19T07:49:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab190b5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fab140c4050 con 0x7fab140c4450 2024-09-19T07:49:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab1359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab14156810 0x7fab14143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab1359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab14156810 0x7fab14143810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50838/0 (socket says 172.21.15.83:50838) 2024-09-19T07:49:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab1359e640 1 -- 172.21.15.83:0/162606589 learned_addr learned my addr 172.21.15.83:0/162606589 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab1259c640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab14153550 0x7fab14073440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab1359e640 1 -- 172.21.15.83:0/162606589 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab140c4450 msgr2=0x7fab140c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab1359e640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab140c4450 0x7fab140c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab1359e640 1 -- 172.21.15.83:0/162606589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab14153550 msgr2=0x7fab14073440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab1359e640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab14153550 0x7fab14073440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab1359e640 1 -- 172.21.15.83:0/162606589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab14073980 con 0x7fab14156810 2024-09-19T07:49:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab1359e640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab14156810 0x7fab14143810 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00cd70 tx=0x7fab0c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7faafb7fe640 1 -- 172.21.15.83:0/162606589 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c003ba0 con 0x7fab14156810 2024-09-19T07:49:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7faafb7fe640 1 -- 172.21.15.83:0/162606589 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab0c003d40 con 0x7fab14156810 2024-09-19T07:49:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab190b5640 1 -- 172.21.15.83:0/162606589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab140726f0 con 0x7fab14156810 2024-09-19T07:49:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7faafb7fe640 1 -- 172.21.15.83:0/162606589 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c004240 con 0x7fab14156810 2024-09-19T07:49:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab190b5640 1 -- 172.21.15.83:0/162606589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab14143d50 con 0x7fab14156810 2024-09-19T07:49:22.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7fab190b5640 1 -- 172.21.15.83:0/162606589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab14068560 con 0x7fab14156810 2024-09-19T07:49:22.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.368+0000 7faafb7fe640 1 -- 172.21.15.83:0/162606589 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fab0c0043e0 con 0x7fab14156810 2024-09-19T07:49:22.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.372+0000 7faafb7fe640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab000678c0 0x7fab00069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.372+0000 7faafb7fe640 1 -- 172.21.15.83:0/162606589 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fab0c0878c0 con 0x7fab14156810 2024-09-19T07:49:22.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.372+0000 7fab12d9d640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab000678c0 0x7fab00069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.372+0000 7fab12d9d640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab000678c0 0x7fab00069d80 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fab04000f70 tx=0x7fab040079e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.376+0000 7faafb7fe640 1 -- 172.21.15.83:0/162606589 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fab0c0572a0 con 0x7fab14156810 2024-09-19T07:49:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.536+0000 7fab190b5640 1 -- 172.21.15.83:0/162606589 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fab140c6540 con 0x7fab000678c0 2024-09-19T07:49:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.536+0000 7faafb7fe640 1 -- 172.21.15.83:0/162606589 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fab140c6540 con 0x7fab000678c0 2024-09-19T07:49:22.537 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 6 to deep-scrub 2024-09-19T07:49:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.540+0000 7fab190b5640 1 -- 172.21.15.83:0/162606589 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab000678c0 msgr2=0x7fab00069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.540+0000 7fab190b5640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab000678c0 0x7fab00069d80 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fab04000f70 tx=0x7fab040079e0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.540+0000 7fab190b5640 1 -- 172.21.15.83:0/162606589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab14156810 msgr2=0x7fab14143810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.540+0000 7fab190b5640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab14156810 0x7fab14143810 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00cd70 tx=0x7fab0c00a660 comp rx=0 tx=0).stop 2024-09-19T07:49:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.540+0000 7fab190b5640 1 -- 172.21.15.83:0/162606589 shutdown_connections 2024-09-19T07:49:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.540+0000 7fab190b5640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fab000678c0 0x7fab00069d80 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.540+0000 7fab190b5640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab14156810 0x7fab14143810 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.540+0000 7fab190b5640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab14153550 0x7fab14073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.540+0000 7fab190b5640 1 --2- 172.21.15.83:0/162606589 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab140c4450 0x7fab140c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.540+0000 7fab190b5640 1 -- 172.21.15.83:0/162606589 >> 172.21.15.83:0/162606589 conn(0x7fab140c48b0 msgr2=0x7fab1414dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.540+0000 7fab190b5640 1 -- 172.21.15.83:0/162606589 shutdown_connections 2024-09-19T07:49:22.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.540+0000 7fab190b5640 1 -- 172.21.15.83:0/162606589 wait complete. 2024-09-19T07:49:22.556 INFO:tasks.ceph:Scrubbing osd.7 2024-09-19T07:49:22.557 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-19T07:49:22.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.680+0000 7f9e94b41640 1 Processor -- start 2024-09-19T07:49:22.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.680+0000 7f9e94b41640 1 -- start start 2024-09-19T07:49:22.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.680+0000 7f9e94b41640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9e900c8f80 0x7f9e900c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.680+0000 7f9e94b41640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e900a8d20 0x7f9e901572b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.680+0000 7f9e94b41640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e900c7ce0 con 0x7f9e900c4cb0 2024-09-19T07:49:22.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.680+0000 7f9e94b41640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f9e90151a00 con 0x7f9e900a8d20 2024-09-19T07:49:22.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.680+0000 7f9e94b41640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f9e900c4260 con 0x7f9e900c8f80 2024-09-19T07:49:22.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.680+0000 7f9e8dd9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e900a8d20 0x7f9e901572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.680+0000 7f9e8dd9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e900a8d20 0x7f9e901572b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50852/0 (socket says 172.21.15.83:50852) 2024-09-19T07:49:22.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.680+0000 7f9e8dd9b640 1 -- 172.21.15.83:0/2832571704 learned_addr learned my addr 172.21.15.83:0/2832571704 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:22.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e8dd9b640 1 -- 172.21.15.83:0/2832571704 >> v2:172.21.15.184:3300/0 conn(0x7f9e900c8f80 msgr2=0x7f9e900c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:22.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e8e59c640 1 --2- 172.21.15.83:0/2832571704 >> v2:172.21.15.184:3300/0 conn(0x7f9e900c8f80 0x7f9e900c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e8dd9b640 1 --2- 172.21.15.83:0/2832571704 >> v2:172.21.15.184:3300/0 conn(0x7f9e900c8f80 0x7f9e900c77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e8dd9b640 1 -- 172.21.15.83:0/2832571704 >> v1:172.21.15.184:6789/0 conn(0x7f9e900c4cb0 legacy=0x7f9e900710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:22.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e8dd9b640 1 -- 172.21.15.83:0/2832571704 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e900c7f80 con 0x7f9e900a8d20 2024-09-19T07:49:22.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e8dd9b640 1 --2- 172.21.15.83:0/2832571704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e900a8d20 0x7f9e901572b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f9e90069290 tx=0x7f9e8001c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6f228696cc7570 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e8d59a640 1 -- 172.21.15.83:0/2832571704 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e800b2070 con 0x7f9e900a8d20 2024-09-19T07:49:22.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e8d59a640 1 -- 172.21.15.83:0/2832571704 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e800ad640 con 0x7f9e900a8d20 2024-09-19T07:49:22.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e8d59a640 1 -- 172.21.15.83:0/2832571704 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e800ada10 con 0x7f9e900a8d20 2024-09-19T07:49:22.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e94b41640 1 -- 172.21.15.83:0/2832571704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e900a8d20 msgr2=0x7f9e901572b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e94b41640 1 --2- 172.21.15.83:0/2832571704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e900a8d20 0x7f9e901572b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f9e90069290 tx=0x7f9e8001c390 comp rx=0 tx=0).stop 2024-09-19T07:49:22.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e94b41640 1 -- 172.21.15.83:0/2832571704 shutdown_connections 2024-09-19T07:49:22.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e94b41640 1 --2- 172.21.15.83:0/2832571704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e900a8d20 0x7f9e901572b0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e94b41640 1 --2- 172.21.15.83:0/2832571704 >> v2:172.21.15.184:3300/0 conn(0x7f9e900c8f80 0x7f9e900c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e94b41640 1 -- 172.21.15.83:0/2832571704 >> 172.21.15.83:0/2832571704 conn(0x7f9e900c9860 msgr2=0x7f9e900c9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:22.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e94b41640 1 -- 172.21.15.83:0/2832571704 shutdown_connections 2024-09-19T07:49:22.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.684+0000 7f9e94b41640 1 -- 172.21.15.83:0/2832571704 wait complete. 2024-09-19T07:49:22.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e94b41640 1 Processor -- start 2024-09-19T07:49:22.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e94b41640 1 -- start start 2024-09-19T07:49:22.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e94b41640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e900a8d20 0x7f9e9015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e94b41640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e900c4cb0 0x7f9e901f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e94b41640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e900c8f80 0x7f9e901f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e94b41640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e900c94e0 con 0x7f9e900c8f80 2024-09-19T07:49:22.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e94b41640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9e900c9360 con 0x7f9e900c4cb0 2024-09-19T07:49:22.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e94b41640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9e90071800 con 0x7f9e900a8d20 2024-09-19T07:49:22.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e8ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e900c8f80 0x7f9e901f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e8ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e900c8f80 0x7f9e901f2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50858/0 (socket says 172.21.15.83:50858) 2024-09-19T07:49:22.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e8ed9d640 1 -- 172.21.15.83:0/3714708089 learned_addr learned my addr 172.21.15.83:0/3714708089 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:22.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e8ed9d640 1 -- 172.21.15.83:0/3714708089 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e900a8d20 msgr2=0x7f9e9015ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e8ed9d640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e900a8d20 0x7f9e9015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e8ed9d640 1 -- 172.21.15.83:0/3714708089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e900c4cb0 msgr2=0x7f9e901f21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e8dd9b640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e900c4cb0 0x7f9e901f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e8ed9d640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e900c4cb0 0x7f9e901f21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e8ed9d640 1 -- 172.21.15.83:0/3714708089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9014b390 con 0x7f9e900c8f80 2024-09-19T07:49:22.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e8ed9d640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e900c8f80 0x7f9e901f2710 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f9e8800c780 tx=0x7f9e8800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e86ffd640 1 -- 172.21.15.83:0/3714708089 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e88017070 con 0x7f9e900c8f80 2024-09-19T07:49:22.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e86ffd640 1 -- 172.21.15.83:0/3714708089 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e88003db0 con 0x7f9e900c8f80 2024-09-19T07:49:22.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e94b41640 1 -- 172.21.15.83:0/3714708089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e90154450 con 0x7f9e900c8f80 2024-09-19T07:49:22.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e86ffd640 1 -- 172.21.15.83:0/3714708089 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e880052b0 con 0x7f9e900c8f80 2024-09-19T07:49:22.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e94b41640 1 -- 172.21.15.83:0/3714708089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e901f1cb0 con 0x7f9e900c8f80 2024-09-19T07:49:22.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e94b41640 1 -- 172.21.15.83:0/3714708089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9e901572b0 con 0x7f9e900c8f80 2024-09-19T07:49:22.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e86ffd640 1 -- 172.21.15.83:0/3714708089 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9e88005450 con 0x7f9e900c8f80 2024-09-19T07:49:22.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e86ffd640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e700678c0 0x7f9e70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e86ffd640 1 -- 172.21.15.83:0/3714708089 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f9e880867e0 con 0x7f9e900c8f80 2024-09-19T07:49:22.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e8e59c640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e700678c0 0x7f9e70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e86ffd640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] conn(0x7f9e70074210 0x7f9e70076690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e86ffd640 1 -- 172.21.15.83:0/3714708089 --> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e70076d60 con 0x7f9e70074210 2024-09-19T07:49:22.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.688+0000 7f9e86ffd640 1 -- 172.21.15.83:0/3714708089 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f9e88086c30 con 0x7f9e900c8f80 2024-09-19T07:49:22.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.692+0000 7f9e8e59c640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e700678c0 0x7f9e70069d80 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800a810 tx=0x7f9e78008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.692+0000 7f9e8dd9b640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] conn(0x7f9e70074210 0x7f9e70076690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.692+0000 7f9e8dd9b640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] conn(0x7f9e70074210 0x7f9e70076690 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.692+0000 7f9e86ffd640 1 -- 172.21.15.83:0/3714708089 <== osd.7 v2:172.21.15.184:6816/2447598614 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9e70076d60 con 0x7f9e70074210 2024-09-19T07:49:22.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e94b41640 1 -- 172.21.15.83:0/3714708089 --> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9e900aedf0 con 0x7f9e70074210 2024-09-19T07:49:22.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e86ffd640 1 -- 172.21.15.83:0/3714708089 <== osd.7 v2:172.21.15.184:6816/2447598614 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9e900aedf0 con 0x7f9e70074210 2024-09-19T07:49:22.707 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-19T07:49:22.707 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-19T07:49:22.707 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-19T07:49:22.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e94b41640 1 -- 172.21.15.83:0/3714708089 >> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] conn(0x7f9e70074210 msgr2=0x7f9e70076690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e94b41640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.184:6816/2447598614,v1:172.21.15.184:6817/2447598614] conn(0x7f9e70074210 0x7f9e70076690 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e94b41640 1 -- 172.21.15.83:0/3714708089 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e700678c0 msgr2=0x7f9e70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e94b41640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e700678c0 0x7f9e70069d80 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800a810 tx=0x7f9e78008040 comp rx=0 tx=0).stop 2024-09-19T07:49:22.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e94b41640 1 -- 172.21.15.83:0/3714708089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e900c8f80 msgr2=0x7f9e901f2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e94b41640 1 --2- 172.21.15.83:0/3714708089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e900c8f80 0x7f9e901f2710 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f9e8800c780 tx=0x7f9e8800cc40 comp rx=0 tx=0).stop 2024-09-19T07:49:22.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e8ed9d640 1 -- 172.21.15.83:0/3714708089 reap_dead start 2024-09-19T07:49:22.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e94b41640 1 -- 172.21.15.83:0/3714708089 shutdown_connections 2024-09-19T07:49:22.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e94b41640 1 -- 172.21.15.83:0/3714708089 >> 172.21.15.83:0/3714708089 conn(0x7f9e900c9860 msgr2=0x7f9e90151bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:22.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e94b41640 1 -- 172.21.15.83:0/3714708089 shutdown_connections 2024-09-19T07:49:22.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.708+0000 7f9e94b41640 1 -- 172.21.15.83:0/3714708089 wait complete. 2024-09-19T07:49:22.723 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-19T07:49:22.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c9dec3640 1 Processor -- start 2024-09-19T07:49:22.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c9dec3640 1 -- start start 2024-09-19T07:49:22.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c9dec3640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1c980c2bb0 0x7f1c980718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c9dec3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1c981515c0 0x7f1c981571b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c9dec3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c98153cd0 con 0x7f1c980c21e0 2024-09-19T07:49:22.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c9dec3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f1c98153f40 con 0x7f1c981515c0 2024-09-19T07:49:22.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c9dec3640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f1c98154220 con 0x7f1c980c2bb0 2024-09-19T07:49:22.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c97fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1c981515c0 0x7f1c981571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c977fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1c980c21e0 0x7f1c98150eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35906/0 (socket says 172.21.15.83:35906) 2024-09-19T07:49:22.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c977fe640 1 -- 172.21.15.83:0/2288169666 learned_addr learned my addr 172.21.15.83:0/2288169666 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:22.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c97fff640 1 -- 172.21.15.83:0/2288169666 >> v2:172.21.15.184:3300/0 conn(0x7f1c980c2bb0 msgr2=0x7f1c980718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:22.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c97fff640 1 --2- 172.21.15.83:0/2288169666 >> v2:172.21.15.184:3300/0 conn(0x7f1c980c2bb0 0x7f1c980718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c97fff640 1 -- 172.21.15.83:0/2288169666 >> v1:172.21.15.83:6789/0 conn(0x7f1c980c21e0 legacy=0x7f1c98150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.840+0000 7f1c97fff640 1 -- 172.21.15.83:0/2288169666 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c98154500 con 0x7f1c981515c0 2024-09-19T07:49:22.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c97fff640 1 --2- 172.21.15.83:0/2288169666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c981515c0 0x7f1c981571b0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f1c980c2a20 tx=0x7f1c9001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1560946a19dd7c38 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c967fc640 1 -- 172.21.15.83:0/2288169666 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c900b2070 con 0x7f1c981515c0 2024-09-19T07:49:22.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c967fc640 1 -- 172.21.15.83:0/2288169666 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c900a5a10 con 0x7f1c981515c0 2024-09-19T07:49:22.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c967fc640 1 -- 172.21.15.83:0/2288169666 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c900a5de0 con 0x7f1c981515c0 2024-09-19T07:49:22.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2288169666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c981515c0 msgr2=0x7f1c981571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 --2- 172.21.15.83:0/2288169666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c981515c0 0x7f1c981571b0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f1c980c2a20 tx=0x7f1c9001c930 comp rx=0 tx=0).stop 2024-09-19T07:49:22.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2288169666 shutdown_connections 2024-09-19T07:49:22.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 --2- 172.21.15.83:0/2288169666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c981515c0 0x7f1c981571b0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 --2- 172.21.15.83:0/2288169666 >> v2:172.21.15.184:3300/0 conn(0x7f1c980c2bb0 0x7f1c980718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2288169666 >> 172.21.15.83:0/2288169666 conn(0x7f1c9806bc80 msgr2=0x7f1c9806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2288169666 shutdown_connections 2024-09-19T07:49:22.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2288169666 wait complete. 2024-09-19T07:49:22.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 Processor -- start 2024-09-19T07:49:22.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 -- start start 2024-09-19T07:49:22.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c980c21e0 0x7f1c9814ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c980c2bb0 0x7f1c981504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c981515c0 0x7f1c98147e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c980a9eb0 con 0x7f1c981515c0 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1c980a9d30 con 0x7f1c980c21e0 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c9dec3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1c980c25c0 con 0x7f1c980c2bb0 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c97fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c981515c0 0x7f1c98147e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c97fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c981515c0 0x7f1c98147e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50872/0 (socket says 172.21.15.83:50872) 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c97fff640 1 -- 172.21.15.83:0/2915152319 learned_addr learned my addr 172.21.15.83:0/2915152319 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.844+0000 7f1c977fe640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c980c21e0 0x7f1c9814ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c97fff640 1 -- 172.21.15.83:0/2915152319 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c980c2bb0 msgr2=0x7f1c981504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c96ffd640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c980c2bb0 0x7f1c981504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c97fff640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c980c2bb0 0x7f1c981504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c97fff640 1 -- 172.21.15.83:0/2915152319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c980c21e0 msgr2=0x7f1c9814ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:22.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c97fff640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c980c21e0 0x7f1c9814ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:22.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c97fff640 1 -- 172.21.15.83:0/2915152319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9814bad0 con 0x7f1c981515c0 2024-09-19T07:49:22.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c96ffd640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c980c2bb0 0x7f1c981504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:22.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c97fff640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c981515c0 0x7f1c98147e70 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f1c90019540 tx=0x7f1c900196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c7ffff640 1 -- 172.21.15.83:0/2915152319 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c900b2050 con 0x7f1c981515c0 2024-09-19T07:49:22.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c7ffff640 1 -- 172.21.15.83:0/2915152319 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c900be070 con 0x7f1c981515c0 2024-09-19T07:49:22.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2915152319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9814a210 con 0x7f1c981515c0 2024-09-19T07:49:22.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c7ffff640 1 -- 172.21.15.83:0/2915152319 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c900adb90 con 0x7f1c981515c0 2024-09-19T07:49:22.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2915152319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c980c1800 con 0x7f1c981515c0 2024-09-19T07:49:22.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2915152319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c9814bca0 con 0x7f1c981515c0 2024-09-19T07:49:22.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c7ffff640 1 -- 172.21.15.83:0/2915152319 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1c900c3460 con 0x7f1c981515c0 2024-09-19T07:49:22.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c7ffff640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c840678c0 0x7f1c84069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:22.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c7ffff640 1 -- 172.21.15.83:0/2915152319 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f1c90122cf0 con 0x7f1c981515c0 2024-09-19T07:49:22.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c977fe640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c840678c0 0x7f1c84069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:22.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.848+0000 7f1c977fe640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c840678c0 0x7f1c84069d80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1c88000f70 tx=0x7f1c88009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:22.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:22.852+0000 7f1c7ffff640 1 -- 172.21.15.83:0/2915152319 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1c90016520 con 0x7f1c981515c0 2024-09-19T07:49:23.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.012+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2915152319 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f1c9814ae10 con 0x7f1c840678c0 2024-09-19T07:49:23.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.012+0000 7f1c7ffff640 1 -- 172.21.15.83:0/2915152319 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1c9814ae10 con 0x7f1c840678c0 2024-09-19T07:49:23.014 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 7 to deep-scrub 2024-09-19T07:49:23.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.016+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2915152319 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c840678c0 msgr2=0x7f1c84069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:23.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.016+0000 7f1c9dec3640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c840678c0 0x7f1c84069d80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1c88000f70 tx=0x7f1c88009040 comp rx=0 tx=0).stop 2024-09-19T07:49:23.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.016+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2915152319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c981515c0 msgr2=0x7f1c98147e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:23.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.016+0000 7f1c9dec3640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c981515c0 0x7f1c98147e70 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f1c90019540 tx=0x7f1c900196d0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.016+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2915152319 shutdown_connections 2024-09-19T07:49:23.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.016+0000 7f1c9dec3640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1c840678c0 0x7f1c84069d80 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.016+0000 7f1c9dec3640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c981515c0 0x7f1c98147e70 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.016+0000 7f1c9dec3640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c980c2bb0 0x7f1c981504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.016+0000 7f1c9dec3640 1 --2- 172.21.15.83:0/2915152319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c980c21e0 0x7f1c9814ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.016+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2915152319 >> 172.21.15.83:0/2915152319 conn(0x7f1c9806bc80 msgr2=0x7f1c98073990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:23.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.016+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2915152319 shutdown_connections 2024-09-19T07:49:23.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.016+0000 7f1c9dec3640 1 -- 172.21.15.83:0/2915152319 wait complete. 2024-09-19T07:49:23.031 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:49:23.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.148+0000 7f9088d01640 1 Processor -- start 2024-09-19T07:49:23.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.148+0000 7f9088d01640 1 -- start start 2024-09-19T07:49:23.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9088d01640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f90840c6460 0x7f90840c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:23.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9088d01640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90840c26c0 con 0x7f90841535d0 2024-09-19T07:49:23.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9088d01640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90840c29a0 con 0x7f908406ffe0 2024-09-19T07:49:23.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9088d01640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f90840707e0 con 0x7f90840c6460 2024-09-19T07:49:23.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9082d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f90840c6460 0x7f90840c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:23.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9082d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f90840c6460 0x7f90840c2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50876/0 (socket says 172.21.15.83:50876) 2024-09-19T07:49:23.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9082d9d640 1 -- 172.21.15.83:0/2390109205 learned_addr learned my addr 172.21.15.83:0/2390109205 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:23.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f908259c640 1 --1- 172.21.15.83:0/2390109205 >> v1:172.21.15.83:6789/0 conn(0x7f90841535d0 0x7f9084155c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35922/0 (socket says 172.21.15.83:35922) 2024-09-19T07:49:23.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9082d9d640 1 -- 172.21.15.83:0/2390109205 >> v1:172.21.15.184:6789/0 conn(0x7f908406ffe0 legacy=0x7f9084159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:23.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f908159a640 1 -- 172.21.15.83:0/2390109205 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 674672147 0 0) 0x7f90840c26c0 con 0x7f90841535d0 2024-09-19T07:49:23.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9082d9d640 1 -- 172.21.15.83:0/2390109205 >> v1:172.21.15.83:6789/0 conn(0x7f90841535d0 legacy=0x7f9084155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:23.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9082d9d640 1 -- 172.21.15.83:0/2390109205 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90841592f0 con 0x7f90840c6460 2024-09-19T07:49:23.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9082d9d640 1 --2- 172.21.15.83:0/2390109205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90840c6460 0x7f90840c2f60 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f907c009e50 tx=0x7f907c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ee5d8106c37531 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:23.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f908159a640 1 -- 172.21.15.83:0/2390109205 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c0a5770 con 0x7f90840c6460 2024-09-19T07:49:23.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f908159a640 1 -- 172.21.15.83:0/2390109205 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f907c0a5910 con 0x7f90840c6460 2024-09-19T07:49:23.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f908159a640 1 -- 172.21.15.83:0/2390109205 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c0a5ce0 con 0x7f90840c6460 2024-09-19T07:49:23.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9088d01640 1 -- 172.21.15.83:0/2390109205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90840c6460 msgr2=0x7f90840c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:23.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9088d01640 1 --2- 172.21.15.83:0/2390109205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90840c6460 0x7f90840c2f60 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f907c009e50 tx=0x7f907c01c9a0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9088d01640 1 -- 172.21.15.83:0/2390109205 shutdown_connections 2024-09-19T07:49:23.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9088d01640 1 --2- 172.21.15.83:0/2390109205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90840c6460 0x7f90840c2f60 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.152+0000 7f9088d01640 1 -- 172.21.15.83:0/2390109205 >> 172.21.15.83:0/2390109205 conn(0x7f90840c5060 msgr2=0x7f90840c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:23.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 -- 172.21.15.83:0/2390109205 shutdown_connections 2024-09-19T07:49:23.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 -- 172.21.15.83:0/2390109205 wait complete. 2024-09-19T07:49:23.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 Processor -- start 2024-09-19T07:49:23.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 -- start start 2024-09-19T07:49:23.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f908406ffe0 0x7f90840738a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:23.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90840c6460 0x7f90841eea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:23.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90841535d0 0x7f90841eef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:23.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f90840c2c80 con 0x7f90841535d0 2024-09-19T07:49:23.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90840c6890 con 0x7f90840c6460 2024-09-19T07:49:23.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9084163f30 con 0x7f908406ffe0 2024-09-19T07:49:23.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9082d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90841535d0 0x7f90841eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:23.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9082d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90841535d0 0x7f90841eef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50890/0 (socket says 172.21.15.83:50890) 2024-09-19T07:49:23.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9082d9d640 1 -- 172.21.15.83:0/3036819078 learned_addr learned my addr 172.21.15.83:0/3036819078 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:23.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9082d9d640 1 -- 172.21.15.83:0/3036819078 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f908406ffe0 msgr2=0x7f90840738a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:23.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9082d9d640 1 --2- 172.21.15.83:0/3036819078 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f908406ffe0 0x7f90840738a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9082d9d640 1 -- 172.21.15.83:0/3036819078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90840c6460 msgr2=0x7f90841eea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:23.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9082d9d640 1 --2- 172.21.15.83:0/3036819078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90840c6460 0x7f90841eea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9082d9d640 1 -- 172.21.15.83:0/3036819078 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90841f2570 con 0x7f90841535d0 2024-09-19T07:49:23.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9082d9d640 1 --2- 172.21.15.83:0/3036819078 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90841535d0 0x7f90841eef80 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f907c0af040 tx=0x7f907c019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:23.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9072ffd640 1 -- 172.21.15.83:0/3036819078 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c0bc020 con 0x7f90841535d0 2024-09-19T07:49:23.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9072ffd640 1 -- 172.21.15.83:0/3036819078 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f907c0b8880 con 0x7f90841535d0 2024-09-19T07:49:23.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 -- 172.21.15.83:0/3036819078 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90841f2850 con 0x7f90841535d0 2024-09-19T07:49:23.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9072ffd640 1 -- 172.21.15.83:0/3036819078 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c0b8c70 con 0x7f90841535d0 2024-09-19T07:49:23.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 -- 172.21.15.83:0/3036819078 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9084155e40 con 0x7f90841535d0 2024-09-19T07:49:23.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.156+0000 7f9088d01640 1 -- 172.21.15.83:0/3036819078 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90840c2f60 con 0x7f90841535d0 2024-09-19T07:49:23.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.160+0000 7f9072ffd640 1 -- 172.21.15.83:0/3036819078 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f907c0bf020 con 0x7f90841535d0 2024-09-19T07:49:23.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.160+0000 7f9072ffd640 1 --2- 172.21.15.83:0/3036819078 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90640678c0 0x7f9064069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:23.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.160+0000 7f9072ffd640 1 -- 172.21.15.83:0/3036819078 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f907c122090 con 0x7f90841535d0 2024-09-19T07:49:23.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.160+0000 7f908259c640 1 --2- 172.21.15.83:0/3036819078 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90640678c0 0x7f9064069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:23.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.160+0000 7f908259c640 1 --2- 172.21.15.83:0/3036819078 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90640678c0 0x7f9064069d80 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f9078000c10 tx=0x7f90780079d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:23.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.164+0000 7f9072ffd640 1 -- 172.21.15.83:0/3036819078 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f907c0c6030 con 0x7f90841535d0 2024-09-19T07:49:23.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.308+0000 7f9088d01640 1 -- 172.21.15.83:0/3036819078 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f90840adaf0 con 0x7f90640678c0 2024-09-19T07:49:23.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.328+0000 7f9072ffd640 1 -- 172.21.15.83:0/3036819078 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491588 (secure 0 0 0) 0x7f90840adaf0 con 0x7f90640678c0 2024-09-19T07:49:23.330 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:23.337 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:49:23.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.336+0000 7f9088d01640 1 -- 172.21.15.83:0/3036819078 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90640678c0 msgr2=0x7f9064069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:23.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.336+0000 7f9088d01640 1 --2- 172.21.15.83:0/3036819078 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90640678c0 0x7f9064069d80 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f9078000c10 tx=0x7f90780079d0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.336+0000 7f9088d01640 1 -- 172.21.15.83:0/3036819078 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90841535d0 msgr2=0x7f90841eef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:23.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.336+0000 7f9088d01640 1 --2- 172.21.15.83:0/3036819078 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90841535d0 0x7f90841eef80 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f907c0af040 tx=0x7f907c019740 comp rx=0 tx=0).stop 2024-09-19T07:49:23.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.336+0000 7f9088d01640 1 -- 172.21.15.83:0/3036819078 shutdown_connections 2024-09-19T07:49:23.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.336+0000 7f9088d01640 1 --2- 172.21.15.83:0/3036819078 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f90640678c0 0x7f9064069d80 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.336+0000 7f9088d01640 1 --2- 172.21.15.83:0/3036819078 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90841535d0 0x7f90841eef80 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.336+0000 7f9088d01640 1 --2- 172.21.15.83:0/3036819078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90840c6460 0x7f90841eea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.336+0000 7f9088d01640 1 --2- 172.21.15.83:0/3036819078 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f908406ffe0 0x7f90840738a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:23.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.336+0000 7f9088d01640 1 -- 172.21.15.83:0/3036819078 >> 172.21.15.83:0/3036819078 conn(0x7f90840c5060 msgr2=0x7f9084160da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:23.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.336+0000 7f9088d01640 1 -- 172.21.15.83:0/3036819078 shutdown_connections 2024-09-19T07:49:23.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:23.336+0000 7f9088d01640 1 -- 172.21.15.83:0/3036819078 wait complete. 2024-09-19T07:49:23.353 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":958,"stamp":"2024-09-19T07:49:21.614343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39797278,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3062,"num_read_kb":412170,"num_write":3183053,"num_write_kb":38996435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":596829,"ondisk_log_size":596829,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7748160,"kb_used_data":84416,"kb_used_omap":12,"kb_used_meta":7663603,"kb_avail":747226560,"statfs":{"total":773094113280,"available":765159997440,"internally_reserved":0,"allocated":86441984,"data_stored":83166972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":7847530033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1113,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004636"},"pg_stats":[{"pgid":"5.18","version":"64'4995","reported_seq":5099,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363394+0000","last_change":"2024-09-19T07:19:40.358457+0000","last_active":"2024-09-19T07:48:58.363394+0000","last_peered":"2024-09-19T07:48:58.363394+0000","last_clean":"2024-09-19T07:48:58.363394+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:48:58.363394+0000","last_undegraded":"2024-09-19T07:48:58.363394+0000","last_fullsized":"2024-09-19T07:48:58.363394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4995,"log_dups_size":0,"ondisk_log_size":4995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27070,"num_write_kb":342022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"61'953","reported_seq":1119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.788398+0000","last_change":"2024-09-19T07:19:31.273906+0000","last_active":"2024-09-19T07:49:01.788398+0000","last_peered":"2024-09-19T07:49:01.788398+0000","last_clean":"2024-09-19T07:49:01.788398+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:49:01.788398+0000","last_undegraded":"2024-09-19T07:49:01.788398+0000","last_fullsized":"2024-09-19T07:49:01.788398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":18964,"num_write":931,"num_write_kb":4658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"61'4894","reported_seq":5008,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364803+0000","last_change":"2024-09-19T07:19:32.288362+0000","last_active":"2024-09-19T07:48:58.364803+0000","last_peered":"2024-09-19T07:48:58.364803+0000","last_clean":"2024-09-19T07:48:58.364803+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:48:58.364803+0000","last_undegraded":"2024-09-19T07:48:58.364803+0000","last_fullsized":"2024-09-19T07:48:58.364803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26245,"num_write_kb":306037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"64'3","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362486+0000","last_change":"2024-09-19T07:19:39.401621+0000","last_active":"2024-09-19T07:48:58.362486+0000","last_peered":"2024-09-19T07:48:58.362486+0000","last_clean":"2024-09-19T07:48:58.362486+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:48:58.362486+0000","last_undegraded":"2024-09-19T07:48:58.362486+0000","last_fullsized":"2024-09-19T07:48:58.362486+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:32.095381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"64'5242","reported_seq":5347,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371889+0000","last_change":"2024-09-19T07:19:40.366073+0000","last_active":"2024-09-19T07:48:58.371889+0000","last_peered":"2024-09-19T07:48:58.371889+0000","last_clean":"2024-09-19T07:48:58.371889+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:48:58.371889+0000","last_undegraded":"2024-09-19T07:48:58.371889+0000","last_fullsized":"2024-09-19T07:48:58.371889+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5242,"log_dups_size":0,"ondisk_log_size":5242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27567,"num_write_kb":356452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"65'25","reported_seq":141,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365251+0000","last_change":"2024-09-19T07:19:31.263247+0000","last_active":"2024-09-19T07:48:58.365251+0000","last_peered":"2024-09-19T07:48:58.365251+0000","last_clean":"2024-09-19T07:48:58.365251+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:48:58.365251+0000","last_undegraded":"2024-09-19T07:48:58.365251+0000","last_fullsized":"2024-09-19T07:48:58.365251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"61'5997","reported_seq":6112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369438+0000","last_change":"2024-09-19T07:19:32.271715+0000","last_active":"2024-09-19T07:48:58.369438+0000","last_peered":"2024-09-19T07:48:58.369438+0000","last_clean":"2024-09-19T07:48:58.369438+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:48:58.369438+0000","last_undegraded":"2024-09-19T07:48:58.369438+0000","last_fullsized":"2024-09-19T07:48:58.369438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32297,"num_write_kb":399913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"64'2","reported_seq":108,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363459+0000","last_change":"2024-09-19T07:19:39.403810+0000","last_active":"2024-09-19T07:48:58.363459+0000","last_peered":"2024-09-19T07:48:58.363459+0000","last_clean":"2024-09-19T07:48:58.363459+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:48:58.363459+0000","last_undegraded":"2024-09-19T07:48:58.363459+0000","last_fullsized":"2024-09-19T07:48:58.363459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:55.714964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"64'3801","reported_seq":3905,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373585+0000","last_change":"2024-09-19T07:19:40.390455+0000","last_active":"2024-09-19T07:48:58.373585+0000","last_peered":"2024-09-19T07:48:58.373585+0000","last_clean":"2024-09-19T07:48:58.373585+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:48:58.373585+0000","last_undegraded":"2024-09-19T07:48:58.373585+0000","last_fullsized":"2024-09-19T07:48:58.373585+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20272,"num_write_kb":245240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"65'44","reported_seq":197,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.801541+0000","last_change":"2024-09-19T07:19:31.282140+0000","last_active":"2024-09-19T07:49:01.801541+0000","last_peered":"2024-09-19T07:49:01.801541+0000","last_clean":"2024-09-19T07:49:01.801541+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:49:01.801541+0000","last_undegraded":"2024-09-19T07:49:01.801541+0000","last_fullsized":"2024-09-19T07:49:01.801541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"61'4645","reported_seq":4758,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368108+0000","last_change":"2024-09-19T07:19:32.294544+0000","last_active":"2024-09-19T07:48:58.368108+0000","last_peered":"2024-09-19T07:48:58.368108+0000","last_clean":"2024-09-19T07:48:58.368108+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:48:58.368108+0000","last_undegraded":"2024-09-19T07:48:58.368108+0000","last_fullsized":"2024-09-19T07:48:58.368108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:20:18.310797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24199,"num_write_kb":281402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"64'764","reported_seq":895,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369315+0000","last_change":"2024-09-19T07:19:39.408309+0000","last_active":"2024-09-19T07:48:58.369315+0000","last_peered":"2024-09-19T07:48:58.369315+0000","last_clean":"2024-09-19T07:48:58.369315+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:48:58.369315+0000","last_undegraded":"2024-09-19T07:48:58.369315+0000","last_fullsized":"2024-09-19T07:48:58.369315+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:19.396261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":71635,"num_write":751,"num_write_kb":4331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"64'3560","reported_seq":3663,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365772+0000","last_change":"2024-09-19T07:19:40.372543+0000","last_active":"2024-09-19T07:48:58.365772+0000","last_peered":"2024-09-19T07:48:58.365772+0000","last_clean":"2024-09-19T07:48:58.365772+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:48:58.365772+0000","last_undegraded":"2024-09-19T07:48:58.365772+0000","last_fullsized":"2024-09-19T07:48:58.365772+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3560,"log_dups_size":0,"ondisk_log_size":3560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:53:15.682896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18641,"num_write_kb":242587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"61'5","reported_seq":158,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.793364+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:49:01.793364+0000","last_peered":"2024-09-19T07:49:01.793364+0000","last_clean":"2024-09-19T07:49:01.793364+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:49:01.793364+0000","last_undegraded":"2024-09-19T07:49:01.793364+0000","last_fullsized":"2024-09-19T07:49:01.793364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"61'5989","reported_seq":6100,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368025+0000","last_change":"2024-09-19T07:19:32.294176+0000","last_active":"2024-09-19T07:48:58.368025+0000","last_peered":"2024-09-19T07:48:58.368025+0000","last_clean":"2024-09-19T07:48:58.368025+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:48:58.368025+0000","last_undegraded":"2024-09-19T07:48:58.368025+0000","last_fullsized":"2024-09-19T07:48:58.368025+0000","mapping_epoch":19,"log_start":"53'200","ondisk_log_start":"53'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:57:55.356672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32376,"num_write_kb":375806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'385","reported_seq":492,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:06.042052+0000","last_change":"2024-09-19T07:19:39.407045+0000","last_active":"2024-09-19T07:49:06.042052+0000","last_peered":"2024-09-19T07:49:06.042052+0000","last_clean":"2024-09-19T07:49:06.042052+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:49:06.042052+0000","last_undegraded":"2024-09-19T07:49:06.042052+0000","last_fullsized":"2024-09-19T07:49:06.042052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:24.846974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2764415,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":637,"num_write":359,"num_write_kb":2908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"64'4768","reported_seq":4874,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372445+0000","last_change":"2024-09-19T07:19:40.396520+0000","last_active":"2024-09-19T07:48:58.372445+0000","last_peered":"2024-09-19T07:48:58.372445+0000","last_clean":"2024-09-19T07:48:58.372445+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:48:58.372445+0000","last_undegraded":"2024-09-19T07:48:58.372445+0000","last_fullsized":"2024-09-19T07:48:58.372445+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25484,"num_write_kb":302986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"61'5596","reported_seq":5709,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380964+0000","last_change":"2024-09-19T07:19:32.316471+0000","last_active":"2024-09-19T07:48:58.380964+0000","last_peered":"2024-09-19T07:48:58.380964+0000","last_clean":"2024-09-19T07:48:58.380964+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:48:58.380964+0000","last_undegraded":"2024-09-19T07:48:58.380964+0000","last_fullsized":"2024-09-19T07:48:58.380964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:48.205717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30081,"num_write_kb":342050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'1456","reported_seq":1615,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.401109+0000","last_change":"2024-09-19T07:19:31.278077+0000","last_active":"2024-09-19T07:48:58.401109+0000","last_peered":"2024-09-19T07:48:58.401109+0000","last_clean":"2024-09-19T07:48:58.401109+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:48:58.401109+0000","last_undegraded":"2024-09-19T07:48:58.401109+0000","last_fullsized":"2024-09-19T07:48:58.401109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":1456,"log_dups_size":0,"ondisk_log_size":1456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:32:40.178655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":42401,"num_write":1433,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"64'6","reported_seq":133,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364879+0000","last_change":"2024-09-19T07:19:39.410500+0000","last_active":"2024-09-19T07:48:58.364879+0000","last_peered":"2024-09-19T07:48:58.364879+0000","last_clean":"2024-09-19T07:48:58.364879+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:48:58.364879+0000","last_undegraded":"2024-09-19T07:48:58.364879+0000","last_fullsized":"2024-09-19T07:48:58.364879+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:04.182606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"64'4389","reported_seq":4495,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378223+0000","last_change":"2024-09-19T07:19:40.381259+0000","last_active":"2024-09-19T07:48:58.378223+0000","last_peered":"2024-09-19T07:48:58.378223+0000","last_clean":"2024-09-19T07:48:58.378223+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:48:58.378223+0000","last_undegraded":"2024-09-19T07:48:58.378223+0000","last_fullsized":"2024-09-19T07:48:58.378223+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4389,"log_dups_size":0,"ondisk_log_size":4389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23791,"num_write_kb":304343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"61'3782","reported_seq":3895,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364288+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:48:58.364288+0000","last_peered":"2024-09-19T07:48:58.364288+0000","last_clean":"2024-09-19T07:48:58.364288+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:48:58.364288+0000","last_undegraded":"2024-09-19T07:48:58.364288+0000","last_fullsized":"2024-09-19T07:48:58.364288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:17.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19997,"num_write_kb":252226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"61'3","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371752+0000","last_change":"2024-09-19T07:19:31.263048+0000","last_active":"2024-09-19T07:48:58.371752+0000","last_peered":"2024-09-19T07:48:58.371752+0000","last_clean":"2024-09-19T07:48:58.371752+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:48:58.371752+0000","last_undegraded":"2024-09-19T07:48:58.371752+0000","last_fullsized":"2024-09-19T07:48:58.371752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"64'5","reported_seq":112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368513+0000","last_change":"2024-09-19T07:19:39.412123+0000","last_active":"2024-09-19T07:48:58.368513+0000","last_peered":"2024-09-19T07:48:58.368513+0000","last_clean":"2024-09-19T07:48:58.368513+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:48:58.368513+0000","last_undegraded":"2024-09-19T07:48:58.368513+0000","last_fullsized":"2024-09-19T07:48:58.368513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:21.014837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"64'3306","reported_seq":3411,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364068+0000","last_change":"2024-09-19T07:19:40.383214+0000","last_active":"2024-09-19T07:48:58.364068+0000","last_peered":"2024-09-19T07:48:58.364068+0000","last_clean":"2024-09-19T07:48:58.364068+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:48:58.364068+0000","last_undegraded":"2024-09-19T07:48:58.364068+0000","last_fullsized":"2024-09-19T07:48:58.364068+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:04:07.173799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17836,"num_write_kb":239496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"61'4850","reported_seq":4963,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363649+0000","last_change":"2024-09-19T07:19:32.266283+0000","last_active":"2024-09-19T07:48:58.363649+0000","last_peered":"2024-09-19T07:48:58.363649+0000","last_clean":"2024-09-19T07:48:58.363649+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:48:58.363649+0000","last_undegraded":"2024-09-19T07:48:58.363649+0000","last_fullsized":"2024-09-19T07:48:58.363649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:50:51.256613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26728,"num_write_kb":308134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"61'2","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372501+0000","last_change":"2024-09-19T07:19:31.273964+0000","last_active":"2024-09-19T07:48:58.372501+0000","last_peered":"2024-09-19T07:48:58.372501+0000","last_clean":"2024-09-19T07:48:58.372501+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:48:58.372501+0000","last_undegraded":"2024-09-19T07:48:58.372501+0000","last_fullsized":"2024-09-19T07:48:58.372501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'5","reported_seq":141,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:21.158021+0000","last_change":"2024-09-19T07:49:21.158156+0000","last_active":"2024-09-19T07:49:21.158021+0000","last_peered":"2024-09-19T07:49:21.158021+0000","last_clean":"2024-09-19T07:49:21.158021+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:49:21.158021+0000","last_undegraded":"2024-09-19T07:49:21.158021+0000","last_fullsized":"2024-09-19T07:49:21.158021+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-19T07:49:21.157739+0000","last_deep_scrub":"65'5","last_deep_scrub_stamp":"2024-09-19T07:49:21.157739+0000","last_clean_scrub_stamp":"2024-09-19T07:49:21.157739+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:09:21.564579+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"64'5157","reported_seq":5262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371211+0000","last_change":"2024-09-19T07:19:40.395566+0000","last_active":"2024-09-19T07:48:58.371211+0000","last_peered":"2024-09-19T07:48:58.371211+0000","last_clean":"2024-09-19T07:48:58.371211+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:48:58.371211+0000","last_undegraded":"2024-09-19T07:48:58.371211+0000","last_fullsized":"2024-09-19T07:48:58.371211+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:54:02.766021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27729,"num_write_kb":365262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"61'4508","reported_seq":4621,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365068+0000","last_change":"2024-09-19T07:19:32.292312+0000","last_active":"2024-09-19T07:48:58.365068+0000","last_peered":"2024-09-19T07:48:58.365068+0000","last_clean":"2024-09-19T07:48:58.365068+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:48:58.365068+0000","last_undegraded":"2024-09-19T07:48:58.365068+0000","last_fullsized":"2024-09-19T07:48:58.365068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:33.984225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24068,"num_write_kb":289158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"61'5","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374924+0000","last_change":"2024-09-19T07:19:31.279023+0000","last_active":"2024-09-19T07:48:58.374924+0000","last_peered":"2024-09-19T07:48:58.374924+0000","last_clean":"2024-09-19T07:48:58.374924+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:48:58.374924+0000","last_undegraded":"2024-09-19T07:48:58.374924+0000","last_fullsized":"2024-09-19T07:48:58.374924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"64'7","reported_seq":154,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364522+0000","last_change":"2024-09-19T07:19:39.424661+0000","last_active":"2024-09-19T07:48:58.364522+0000","last_peered":"2024-09-19T07:48:58.364522+0000","last_clean":"2024-09-19T07:48:58.364522+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:48:58.364522+0000","last_undegraded":"2024-09-19T07:48:58.364522+0000","last_fullsized":"2024-09-19T07:48:58.364522+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"64'3458","reported_seq":3564,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362033+0000","last_change":"2024-09-19T07:19:40.388972+0000","last_active":"2024-09-19T07:48:58.362033+0000","last_peered":"2024-09-19T07:48:58.362033+0000","last_clean":"2024-09-19T07:48:58.362033+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:48:58.362033+0000","last_undegraded":"2024-09-19T07:48:58.362033+0000","last_fullsized":"2024-09-19T07:48:58.362033+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3458,"log_dups_size":0,"ondisk_log_size":3458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:13:18.296951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18862,"num_write_kb":239540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"61'5647","reported_seq":5762,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369923+0000","last_change":"2024-09-19T07:19:32.284799+0000","last_active":"2024-09-19T07:48:58.369923+0000","last_peered":"2024-09-19T07:48:58.369923+0000","last_clean":"2024-09-19T07:48:58.369923+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:48:58.369923+0000","last_undegraded":"2024-09-19T07:48:58.369923+0000","last_fullsized":"2024-09-19T07:48:58.369923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29735,"num_write_kb":364939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"63'9","reported_seq":164,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.795335+0000","last_change":"2024-09-19T07:19:31.281674+0000","last_active":"2024-09-19T07:49:01.795335+0000","last_peered":"2024-09-19T07:49:01.795335+0000","last_clean":"2024-09-19T07:49:01.795335+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:49:01.795335+0000","last_undegraded":"2024-09-19T07:49:01.795335+0000","last_fullsized":"2024-09-19T07:49:01.795335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:48.654598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'45","reported_seq":213,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:08.425477+0000","last_change":"2024-09-19T07:19:39.412192+0000","last_active":"2024-09-19T07:49:08.425477+0000","last_peered":"2024-09-19T07:49:08.425477+0000","last_clean":"2024-09-19T07:49:08.425477+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:49:08.425477+0000","last_undegraded":"2024-09-19T07:49:08.425477+0000","last_fullsized":"2024-09-19T07:49:08.425477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":108,"num_write":41,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"64'4007","reported_seq":4113,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363551+0000","last_change":"2024-09-19T07:19:40.390217+0000","last_active":"2024-09-19T07:48:58.363551+0000","last_peered":"2024-09-19T07:48:58.363551+0000","last_clean":"2024-09-19T07:48:58.363551+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:48:58.363551+0000","last_undegraded":"2024-09-19T07:48:58.363551+0000","last_fullsized":"2024-09-19T07:48:58.363551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:20.198919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21606,"num_write_kb":266928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"61'4152","reported_seq":4265,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364929+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:48:58.364929+0000","last_peered":"2024-09-19T07:48:58.364929+0000","last_clean":"2024-09-19T07:48:58.364929+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:48:58.364929+0000","last_undegraded":"2024-09-19T07:48:58.364929+0000","last_fullsized":"2024-09-19T07:48:58.364929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22349,"num_write_kb":269915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"65'474","reported_seq":610,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.368749+0000","last_change":"2024-09-19T07:19:31.272584+0000","last_active":"2024-09-19T07:49:03.368749+0000","last_peered":"2024-09-19T07:49:03.368749+0000","last_clean":"2024-09-19T07:49:03.368749+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:49:03.368749+0000","last_undegraded":"2024-09-19T07:49:03.368749+0000","last_fullsized":"2024-09-19T07:49:03.368749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":59,"num_write":1141,"num_write_kb":380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"64'10","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362143+0000","last_change":"2024-09-19T07:19:39.434150+0000","last_active":"2024-09-19T07:48:58.362143+0000","last_peered":"2024-09-19T07:48:58.362143+0000","last_clean":"2024-09-19T07:48:58.362143+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:48:58.362143+0000","last_undegraded":"2024-09-19T07:48:58.362143+0000","last_fullsized":"2024-09-19T07:48:58.362143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:13.654208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"64'4199","reported_seq":4305,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380465+0000","last_change":"2024-09-19T07:19:40.380140+0000","last_active":"2024-09-19T07:48:58.380465+0000","last_peered":"2024-09-19T07:48:58.380465+0000","last_clean":"2024-09-19T07:48:58.380465+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:48:58.380465+0000","last_undegraded":"2024-09-19T07:48:58.380465+0000","last_fullsized":"2024-09-19T07:48:58.380465+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:05:14.657053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22884,"num_write_kb":280362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"61'5900","reported_seq":6015,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372762+0000","last_change":"2024-09-19T07:19:32.278073+0000","last_active":"2024-09-19T07:48:58.372762+0000","last_peered":"2024-09-19T07:48:58.372762+0000","last_clean":"2024-09-19T07:48:58.372762+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:48:58.372762+0000","last_undegraded":"2024-09-19T07:48:58.372762+0000","last_fullsized":"2024-09-19T07:48:58.372762+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31464,"num_write_kb":364012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"61'2","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379634+0000","last_change":"2024-09-19T07:19:31.282844+0000","last_active":"2024-09-19T07:48:58.379634+0000","last_peered":"2024-09-19T07:48:58.379634+0000","last_clean":"2024-09-19T07:48:58.379634+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:48:58.379634+0000","last_undegraded":"2024-09-19T07:48:58.379634+0000","last_fullsized":"2024-09-19T07:48:58.379634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:17.318351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"64'3","reported_seq":112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370198+0000","last_change":"2024-09-19T07:19:39.411605+0000","last_active":"2024-09-19T07:48:58.370198+0000","last_peered":"2024-09-19T07:48:58.370198+0000","last_clean":"2024-09-19T07:48:58.370198+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:48:58.370198+0000","last_undegraded":"2024-09-19T07:48:58.370198+0000","last_fullsized":"2024-09-19T07:48:58.370198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"64'4404","reported_seq":4511,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370971+0000","last_change":"2024-09-19T07:19:40.372083+0000","last_active":"2024-09-19T07:48:58.370971+0000","last_peered":"2024-09-19T07:48:58.370971+0000","last_clean":"2024-09-19T07:48:58.370971+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:48:58.370971+0000","last_undegraded":"2024-09-19T07:48:58.370971+0000","last_fullsized":"2024-09-19T07:48:58.370971+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24188,"num_write_kb":314198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"61'3777","reported_seq":3891,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368386+0000","last_change":"2024-09-19T07:19:32.273875+0000","last_active":"2024-09-19T07:48:58.368386+0000","last_peered":"2024-09-19T07:48:58.368386+0000","last_clean":"2024-09-19T07:48:58.368386+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:48:58.368386+0000","last_undegraded":"2024-09-19T07:48:58.368386+0000","last_fullsized":"2024-09-19T07:48:58.368386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:08:24.925052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20312,"num_write_kb":245406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"61'2","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372847+0000","last_change":"2024-09-19T07:19:31.263050+0000","last_active":"2024-09-19T07:48:58.372847+0000","last_peered":"2024-09-19T07:48:58.372847+0000","last_clean":"2024-09-19T07:48:58.372847+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:48:58.372847+0000","last_undegraded":"2024-09-19T07:48:58.372847+0000","last_fullsized":"2024-09-19T07:48:58.372847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'4","reported_seq":113,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371572+0000","last_change":"2024-09-19T07:19:39.420340+0000","last_active":"2024-09-19T07:48:58.371572+0000","last_peered":"2024-09-19T07:48:58.371572+0000","last_clean":"2024-09-19T07:48:58.371572+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:48:58.371572+0000","last_undegraded":"2024-09-19T07:48:58.371572+0000","last_fullsized":"2024-09-19T07:48:58.371572+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"64'4365","reported_seq":4472,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369312+0000","last_change":"2024-09-19T07:19:40.383082+0000","last_active":"2024-09-19T07:48:58.369312+0000","last_peered":"2024-09-19T07:48:58.369312+0000","last_clean":"2024-09-19T07:48:58.369312+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:48:58.369312+0000","last_undegraded":"2024-09-19T07:48:58.369312+0000","last_fullsized":"2024-09-19T07:48:58.369312+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24012,"num_write_kb":288918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'5396","reported_seq":5510,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370199+0000","last_change":"2024-09-19T07:19:32.278272+0000","last_active":"2024-09-19T07:48:58.370199+0000","last_peered":"2024-09-19T07:48:58.370199+0000","last_clean":"2024-09-19T07:48:58.370199+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:48:58.370199+0000","last_undegraded":"2024-09-19T07:48:58.370199+0000","last_fullsized":"2024-09-19T07:48:58.370199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:44.593771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28800,"num_write_kb":336838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'4","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368215+0000","last_change":"2024-09-19T07:19:31.281694+0000","last_active":"2024-09-19T07:48:58.368215+0000","last_peered":"2024-09-19T07:48:58.368215+0000","last_clean":"2024-09-19T07:48:58.368215+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:48:58.368215+0000","last_undegraded":"2024-09-19T07:48:58.368215+0000","last_fullsized":"2024-09-19T07:48:58.368215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:56.876104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"64'2","reported_seq":111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372675+0000","last_change":"2024-09-19T07:19:39.407135+0000","last_active":"2024-09-19T07:48:58.372675+0000","last_peered":"2024-09-19T07:48:58.372675+0000","last_clean":"2024-09-19T07:48:58.372675+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:48:58.372675+0000","last_undegraded":"2024-09-19T07:48:58.372675+0000","last_fullsized":"2024-09-19T07:48:58.372675+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"64'4926","reported_seq":5033,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378172+0000","last_change":"2024-09-19T07:19:40.368866+0000","last_active":"2024-09-19T07:48:58.378172+0000","last_peered":"2024-09-19T07:48:58.378172+0000","last_clean":"2024-09-19T07:48:58.378172+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:48:58.378172+0000","last_undegraded":"2024-09-19T07:48:58.378172+0000","last_fullsized":"2024-09-19T07:48:58.378172+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26066,"num_write_kb":337871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'3534","reported_seq":3649,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371369+0000","last_change":"2024-09-19T07:19:32.277674+0000","last_active":"2024-09-19T07:48:58.371369+0000","last_peered":"2024-09-19T07:48:58.371369+0000","last_clean":"2024-09-19T07:48:58.371369+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:48:58.371369+0000","last_undegraded":"2024-09-19T07:48:58.371369+0000","last_fullsized":"2024-09-19T07:48:58.371369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19019,"num_write_kb":227884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"65'46","reported_seq":222,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.373941+0000","last_change":"2024-09-19T07:19:31.260815+0000","last_active":"2024-09-19T07:49:03.373941+0000","last_peered":"2024-09-19T07:49:03.373941+0000","last_clean":"2024-09-19T07:49:03.373941+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:49:03.373941+0000","last_undegraded":"2024-09-19T07:49:03.373941+0000","last_fullsized":"2024-09-19T07:49:03.373941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":109,"num_write":40,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'9","reported_seq":157,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.420212+0000","last_change":"2024-09-19T07:19:39.420497+0000","last_active":"2024-09-19T07:48:58.420212+0000","last_peered":"2024-09-19T07:48:58.420212+0000","last_clean":"2024-09-19T07:48:58.420212+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:48:58.420212+0000","last_undegraded":"2024-09-19T07:48:58.420212+0000","last_fullsized":"2024-09-19T07:48:58.420212+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"64'3425","reported_seq":3530,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371701+0000","last_change":"2024-09-19T07:19:40.387573+0000","last_active":"2024-09-19T07:48:58.371701+0000","last_peered":"2024-09-19T07:48:58.371701+0000","last_clean":"2024-09-19T07:48:58.371701+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:48:58.371701+0000","last_undegraded":"2024-09-19T07:48:58.371701+0000","last_fullsized":"2024-09-19T07:48:58.371701+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:20:43.234793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19036,"num_write_kb":252575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"61'4467","reported_seq":4582,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369004+0000","last_change":"2024-09-19T07:19:32.312907+0000","last_active":"2024-09-19T07:48:58.369004+0000","last_peered":"2024-09-19T07:48:58.369004+0000","last_clean":"2024-09-19T07:48:58.369004+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:48:58.369004+0000","last_undegraded":"2024-09-19T07:48:58.369004+0000","last_fullsized":"2024-09-19T07:48:58.369004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23642,"num_write_kb":275072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'13","reported_seq":129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375436+0000","last_change":"2024-09-19T07:19:31.278461+0000","last_active":"2024-09-19T07:48:58.375436+0000","last_peered":"2024-09-19T07:48:58.375436+0000","last_clean":"2024-09-19T07:48:58.375436+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:48:58.375436+0000","last_undegraded":"2024-09-19T07:48:58.375436+0000","last_fullsized":"2024-09-19T07:48:58.375436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'515","reported_seq":643,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:08.369451+0000","last_change":"2024-09-19T07:19:39.410752+0000","last_active":"2024-09-19T07:49:08.369451+0000","last_peered":"2024-09-19T07:49:08.369451+0000","last_clean":"2024-09-19T07:49:08.369451+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:49:08.369451+0000","last_undegraded":"2024-09-19T07:49:08.369451+0000","last_fullsized":"2024-09-19T07:49:08.369451+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":53,"num_write":1231,"num_write_kb":406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"64'4906","reported_seq":5012,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372981+0000","last_change":"2024-09-19T07:19:40.392973+0000","last_active":"2024-09-19T07:48:58.372981+0000","last_peered":"2024-09-19T07:48:58.372981+0000","last_clean":"2024-09-19T07:48:58.372981+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:48:58.372981+0000","last_undegraded":"2024-09-19T07:48:58.372981+0000","last_fullsized":"2024-09-19T07:48:58.372981+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27102,"num_write_kb":350472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"61'4338","reported_seq":4452,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363793+0000","last_change":"2024-09-19T07:19:32.291247+0000","last_active":"2024-09-19T07:48:58.363793+0000","last_peered":"2024-09-19T07:48:58.363793+0000","last_clean":"2024-09-19T07:48:58.363793+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:48:58.363793+0000","last_undegraded":"2024-09-19T07:48:58.363793+0000","last_fullsized":"2024-09-19T07:48:58.363793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22767,"num_write_kb":271598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"61'3","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365893+0000","last_change":"2024-09-19T07:19:31.284759+0000","last_active":"2024-09-19T07:48:58.365893+0000","last_peered":"2024-09-19T07:48:58.365893+0000","last_clean":"2024-09-19T07:48:58.365893+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:48:58.365893+0000","last_undegraded":"2024-09-19T07:48:58.365893+0000","last_fullsized":"2024-09-19T07:48:58.365893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:09:43.939652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":272,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.376591+0000","last_change":"2024-09-19T07:19:22.629648+0000","last_active":"2024-09-19T07:48:58.376591+0000","last_peered":"2024-09-19T07:48:58.376591+0000","last_clean":"2024-09-19T07:48:58.376591+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:48:58.376591+0000","last_undegraded":"2024-09-19T07:48:58.376591+0000","last_fullsized":"2024-09-19T07:48:58.376591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:21.529843+0000","last_clean_scrub_stamp":"2024-09-19T07:19:21.529843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:21:58.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'2","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380743+0000","last_change":"2024-09-19T07:19:39.429085+0000","last_active":"2024-09-19T07:48:58.380743+0000","last_peered":"2024-09-19T07:48:58.380743+0000","last_clean":"2024-09-19T07:48:58.380743+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:48:58.380743+0000","last_undegraded":"2024-09-19T07:48:58.380743+0000","last_fullsized":"2024-09-19T07:48:58.380743+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:56.721560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"64'3599","reported_seq":3705,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378214+0000","last_change":"2024-09-19T07:19:40.382352+0000","last_active":"2024-09-19T07:48:58.378214+0000","last_peered":"2024-09-19T07:48:58.378214+0000","last_clean":"2024-09-19T07:48:58.378214+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:48:58.378214+0000","last_undegraded":"2024-09-19T07:48:58.378214+0000","last_fullsized":"2024-09-19T07:48:58.378214+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19570,"num_write_kb":246554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"61'5018","reported_seq":5131,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379025+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:48:58.379025+0000","last_peered":"2024-09-19T07:48:58.379025+0000","last_clean":"2024-09-19T07:48:58.379025+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:48:58.379025+0000","last_undegraded":"2024-09-19T07:48:58.379025+0000","last_fullsized":"2024-09-19T07:48:58.379025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:33:38.238956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26753,"num_write_kb":316367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"61'5","reported_seq":121,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378938+0000","last_change":"2024-09-19T07:19:31.286113+0000","last_active":"2024-09-19T07:48:58.378938+0000","last_peered":"2024-09-19T07:48:58.378938+0000","last_clean":"2024-09-19T07:48:58.378938+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:48:58.378938+0000","last_undegraded":"2024-09-19T07:48:58.378938+0000","last_fullsized":"2024-09-19T07:48:58.378938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"64'3","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380502+0000","last_change":"2024-09-19T07:19:39.427644+0000","last_active":"2024-09-19T07:48:58.380502+0000","last_peered":"2024-09-19T07:48:58.380502+0000","last_clean":"2024-09-19T07:48:58.380502+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:48:58.380502+0000","last_undegraded":"2024-09-19T07:48:58.380502+0000","last_fullsized":"2024-09-19T07:48:58.380502+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"64'4954","reported_seq":5060,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377982+0000","last_change":"2024-09-19T07:19:40.390380+0000","last_active":"2024-09-19T07:48:58.377982+0000","last_peered":"2024-09-19T07:48:58.377982+0000","last_clean":"2024-09-19T07:48:58.377982+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:48:58.377982+0000","last_undegraded":"2024-09-19T07:48:58.377982+0000","last_fullsized":"2024-09-19T07:48:58.377982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27293,"num_write_kb":322594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"61'5220","reported_seq":5319,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377909+0000","last_change":"2024-09-19T07:20:13.952747+0000","last_active":"2024-09-19T07:48:58.377909+0000","last_peered":"2024-09-19T07:48:58.377909+0000","last_clean":"2024-09-19T07:48:58.377909+0000","last_became_active":"2024-09-19T07:20:13.951449+0000","last_became_peered":"2024-09-19T07:20:13.951449+0000","last_unstale":"2024-09-19T07:48:58.377909+0000","last_undegraded":"2024-09-19T07:48:58.377909+0000","last_fullsized":"2024-09-19T07:48:58.377909+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27694,"num_write_kb":340542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"61'6","reported_seq":142,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374537+0000","last_change":"2024-09-19T07:19:31.268357+0000","last_active":"2024-09-19T07:48:58.374537+0000","last_peered":"2024-09-19T07:48:58.374537+0000","last_clean":"2024-09-19T07:48:58.374537+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:48:58.374537+0000","last_undegraded":"2024-09-19T07:48:58.374537+0000","last_fullsized":"2024-09-19T07:48:58.374537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"64'1403","reported_seq":1554,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369902+0000","last_change":"2024-09-19T07:19:39.410638+0000","last_active":"2024-09-19T07:48:58.369902+0000","last_peered":"2024-09-19T07:48:58.369902+0000","last_clean":"2024-09-19T07:48:58.369902+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:48:58.369902+0000","last_undegraded":"2024-09-19T07:48:58.369902+0000","last_fullsized":"2024-09-19T07:48:58.369902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":1403,"log_dups_size":0,"ondisk_log_size":1403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":63,"num_read_kb":30148,"num_write":1379,"num_write_kb":4466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"64'3869","reported_seq":3974,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372595+0000","last_change":"2024-09-19T07:19:40.378481+0000","last_active":"2024-09-19T07:48:58.372595+0000","last_peered":"2024-09-19T07:48:58.372595+0000","last_clean":"2024-09-19T07:48:58.372595+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:48:58.372595+0000","last_undegraded":"2024-09-19T07:48:58.372595+0000","last_fullsized":"2024-09-19T07:48:58.372595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3869,"log_dups_size":0,"ondisk_log_size":3869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:30.585686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20920,"num_write_kb":264190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"61'6113","reported_seq":6212,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373612+0000","last_change":"2024-09-19T07:20:14.209767+0000","last_active":"2024-09-19T07:48:58.373612+0000","last_peered":"2024-09-19T07:48:58.373612+0000","last_clean":"2024-09-19T07:48:58.373612+0000","last_became_active":"2024-09-19T07:20:14.209141+0000","last_became_peered":"2024-09-19T07:20:14.209141+0000","last_unstale":"2024-09-19T07:48:58.373612+0000","last_undegraded":"2024-09-19T07:48:58.373612+0000","last_fullsized":"2024-09-19T07:48:58.373612+0000","mapping_epoch":26,"log_start":"54'300","ondisk_log_start":"54'300","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32041,"num_write_kb":407182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"61'7","reported_seq":163,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.805494+0000","last_change":"2024-09-19T07:19:31.258204+0000","last_active":"2024-09-19T07:49:01.805494+0000","last_peered":"2024-09-19T07:49:01.805494+0000","last_clean":"2024-09-19T07:49:01.805494+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:49:01.805494+0000","last_undegraded":"2024-09-19T07:49:01.805494+0000","last_fullsized":"2024-09-19T07:49:01.805494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"64'5","reported_seq":112,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365126+0000","last_change":"2024-09-19T07:19:39.437656+0000","last_active":"2024-09-19T07:48:58.365126+0000","last_peered":"2024-09-19T07:48:58.365126+0000","last_clean":"2024-09-19T07:48:58.365126+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:48:58.365126+0000","last_undegraded":"2024-09-19T07:48:58.365126+0000","last_fullsized":"2024-09-19T07:48:58.365126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:26:41.468096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"64'4634","reported_seq":4740,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377579+0000","last_change":"2024-09-19T07:19:40.379289+0000","last_active":"2024-09-19T07:48:58.377579+0000","last_peered":"2024-09-19T07:48:58.377579+0000","last_clean":"2024-09-19T07:48:58.377579+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:48:58.377579+0000","last_undegraded":"2024-09-19T07:48:58.377579+0000","last_fullsized":"2024-09-19T07:48:58.377579+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25462,"num_write_kb":298798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"61'4007","reported_seq":4120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375692+0000","last_change":"2024-09-19T07:19:32.289946+0000","last_active":"2024-09-19T07:48:58.375692+0000","last_peered":"2024-09-19T07:48:58.375692+0000","last_clean":"2024-09-19T07:48:58.375692+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:48:58.375692+0000","last_undegraded":"2024-09-19T07:48:58.375692+0000","last_fullsized":"2024-09-19T07:48:58.375692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:14.846815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21937,"num_write_kb":251948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"61'4","reported_seq":139,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369549+0000","last_change":"2024-09-19T07:19:31.253645+0000","last_active":"2024-09-19T07:48:58.369549+0000","last_peered":"2024-09-19T07:48:58.369549+0000","last_clean":"2024-09-19T07:48:58.369549+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:48:58.369549+0000","last_undegraded":"2024-09-19T07:48:58.369549+0000","last_fullsized":"2024-09-19T07:48:58.369549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:01.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"64'2","reported_seq":109,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363818+0000","last_change":"2024-09-19T07:19:39.413589+0000","last_active":"2024-09-19T07:48:58.363818+0000","last_peered":"2024-09-19T07:48:58.363818+0000","last_clean":"2024-09-19T07:48:58.363818+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:48:58.363818+0000","last_undegraded":"2024-09-19T07:48:58.363818+0000","last_fullsized":"2024-09-19T07:48:58.363818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:25.382646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"64'3939","reported_seq":4043,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364105+0000","last_change":"2024-09-19T07:19:40.387159+0000","last_active":"2024-09-19T07:48:58.364105+0000","last_peered":"2024-09-19T07:48:58.364105+0000","last_clean":"2024-09-19T07:48:58.364105+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:48:58.364105+0000","last_undegraded":"2024-09-19T07:48:58.364105+0000","last_fullsized":"2024-09-19T07:48:58.364105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20741,"num_write_kb":275861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'4660","reported_seq":4774,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373923+0000","last_change":"2024-09-19T07:19:32.314007+0000","last_active":"2024-09-19T07:48:58.373923+0000","last_peered":"2024-09-19T07:48:58.373923+0000","last_clean":"2024-09-19T07:48:58.373923+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:48:58.373923+0000","last_undegraded":"2024-09-19T07:48:58.373923+0000","last_fullsized":"2024-09-19T07:48:58.373923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25264,"num_write_kb":302007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"65'15","reported_seq":214,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.376639+0000","last_change":"2024-09-19T07:19:31.257183+0000","last_active":"2024-09-19T07:49:03.376639+0000","last_peered":"2024-09-19T07:49:03.376639+0000","last_clean":"2024-09-19T07:49:03.376639+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:49:03.376639+0000","last_undegraded":"2024-09-19T07:49:03.376639+0000","last_fullsized":"2024-09-19T07:49:03.376639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"64'3","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369883+0000","last_change":"2024-09-19T07:19:39.406110+0000","last_active":"2024-09-19T07:48:58.369883+0000","last_peered":"2024-09-19T07:48:58.369883+0000","last_clean":"2024-09-19T07:48:58.369883+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:48:58.369883+0000","last_undegraded":"2024-09-19T07:48:58.369883+0000","last_fullsized":"2024-09-19T07:48:58.369883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"64'4590","reported_seq":4694,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366527+0000","last_change":"2024-09-19T07:19:40.365764+0000","last_active":"2024-09-19T07:48:58.366527+0000","last_peered":"2024-09-19T07:48:58.366527+0000","last_clean":"2024-09-19T07:48:58.366527+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:48:58.366527+0000","last_undegraded":"2024-09-19T07:48:58.366527+0000","last_fullsized":"2024-09-19T07:48:58.366527+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:04:03.242292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24790,"num_write_kb":304361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"61'3873","reported_seq":3985,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380282+0000","last_change":"2024-09-19T07:19:32.307030+0000","last_active":"2024-09-19T07:48:58.380282+0000","last_peered":"2024-09-19T07:48:58.380282+0000","last_clean":"2024-09-19T07:48:58.380282+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:48:58.380282+0000","last_undegraded":"2024-09-19T07:48:58.380282+0000","last_fullsized":"2024-09-19T07:48:58.380282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20508,"num_write_kb":250225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"61'5","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381953+0000","last_change":"2024-09-19T07:19:31.268924+0000","last_active":"2024-09-19T07:48:58.381953+0000","last_peered":"2024-09-19T07:48:58.381953+0000","last_clean":"2024-09-19T07:48:58.381953+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:48:58.381953+0000","last_undegraded":"2024-09-19T07:48:58.381953+0000","last_fullsized":"2024-09-19T07:48:58.381953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"64'4","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364247+0000","last_change":"2024-09-19T07:19:39.429768+0000","last_active":"2024-09-19T07:48:58.364247+0000","last_peered":"2024-09-19T07:48:58.364247+0000","last_clean":"2024-09-19T07:48:58.364247+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:48:58.364247+0000","last_undegraded":"2024-09-19T07:48:58.364247+0000","last_fullsized":"2024-09-19T07:48:58.364247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:47.503631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"64'3485","reported_seq":3589,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366523+0000","last_change":"2024-09-19T07:19:40.373542+0000","last_active":"2024-09-19T07:48:58.366523+0000","last_peered":"2024-09-19T07:48:58.366523+0000","last_clean":"2024-09-19T07:48:58.366523+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:48:58.366523+0000","last_undegraded":"2024-09-19T07:48:58.366523+0000","last_fullsized":"2024-09-19T07:48:58.366523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:14.326662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19669,"num_write_kb":242834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"61'5231","reported_seq":5343,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373420+0000","last_change":"2024-09-19T07:19:32.314033+0000","last_active":"2024-09-19T07:48:58.373420+0000","last_peered":"2024-09-19T07:48:58.373420+0000","last_clean":"2024-09-19T07:48:58.373420+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:48:58.373420+0000","last_undegraded":"2024-09-19T07:48:58.373420+0000","last_fullsized":"2024-09-19T07:48:58.373420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:01:18.234561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28071,"num_write_kb":333869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"61'3","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367894+0000","last_change":"2024-09-19T07:19:31.286110+0000","last_active":"2024-09-19T07:48:58.367894+0000","last_peered":"2024-09-19T07:48:58.367894+0000","last_clean":"2024-09-19T07:48:58.367894+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:48:58.367894+0000","last_undegraded":"2024-09-19T07:48:58.367894+0000","last_fullsized":"2024-09-19T07:48:58.367894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:06:59.236877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"64'328","reported_seq":535,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363630+0000","last_change":"2024-09-19T07:19:39.411074+0000","last_active":"2024-09-19T07:48:58.363630+0000","last_peered":"2024-09-19T07:48:58.363630+0000","last_clean":"2024-09-19T07:48:58.363630+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:48:58.363630+0000","last_undegraded":"2024-09-19T07:48:58.363630+0000","last_fullsized":"2024-09-19T07:48:58.363630+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:53.992787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":181,"num_read_kb":77990,"num_write":325,"num_write_kb":4282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"64'5112","reported_seq":5216,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363063+0000","last_change":"2024-09-19T07:19:40.364814+0000","last_active":"2024-09-19T07:48:58.363063+0000","last_peered":"2024-09-19T07:48:58.363063+0000","last_clean":"2024-09-19T07:48:58.363063+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:48:58.363063+0000","last_undegraded":"2024-09-19T07:48:58.363063+0000","last_fullsized":"2024-09-19T07:48:58.363063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5112,"log_dups_size":0,"ondisk_log_size":5112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27577,"num_write_kb":360586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"61'4085","reported_seq":4198,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369168+0000","last_change":"2024-09-19T07:19:32.280701+0000","last_active":"2024-09-19T07:48:58.369168+0000","last_peered":"2024-09-19T07:48:58.369168+0000","last_clean":"2024-09-19T07:48:58.369168+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:48:58.369168+0000","last_undegraded":"2024-09-19T07:48:58.369168+0000","last_fullsized":"2024-09-19T07:48:58.369168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22122,"num_write_kb":273218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"65'10","reported_seq":165,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.786850+0000","last_change":"2024-09-19T07:19:31.263004+0000","last_active":"2024-09-19T07:49:01.786850+0000","last_peered":"2024-09-19T07:49:01.786850+0000","last_clean":"2024-09-19T07:49:01.786850+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:49:01.786850+0000","last_undegraded":"2024-09-19T07:49:01.786850+0000","last_fullsized":"2024-09-19T07:49:01.786850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'29","reported_seq":195,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.423376+0000","last_change":"2024-09-19T07:19:39.431616+0000","last_active":"2024-09-19T07:48:58.423376+0000","last_peered":"2024-09-19T07:48:58.423376+0000","last_clean":"2024-09-19T07:48:58.423376+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:48:58.423376+0000","last_undegraded":"2024-09-19T07:48:58.423376+0000","last_fullsized":"2024-09-19T07:48:58.423376+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:12:18.149581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"64'4543","reported_seq":4647,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363327+0000","last_change":"2024-09-19T07:19:40.381443+0000","last_active":"2024-09-19T07:48:58.363327+0000","last_peered":"2024-09-19T07:48:58.363327+0000","last_clean":"2024-09-19T07:48:58.363327+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:48:58.363327+0000","last_undegraded":"2024-09-19T07:48:58.363327+0000","last_fullsized":"2024-09-19T07:48:58.363327+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:31:08.702668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24993,"num_write_kb":317373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"61'3981","reported_seq":4093,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375912+0000","last_change":"2024-09-19T07:19:32.284139+0000","last_active":"2024-09-19T07:48:58.375912+0000","last_peered":"2024-09-19T07:48:58.375912+0000","last_clean":"2024-09-19T07:48:58.375912+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:48:58.375912+0000","last_undegraded":"2024-09-19T07:48:58.375912+0000","last_fullsized":"2024-09-19T07:48:58.375912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3981,"log_dups_size":0,"ondisk_log_size":3981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:49.573475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21890,"num_write_kb":263866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"65'352","reported_seq":567,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:01.802657+0000","last_change":"2024-09-19T07:19:31.275976+0000","last_active":"2024-09-19T07:49:01.802657+0000","last_peered":"2024-09-19T07:49:01.802657+0000","last_clean":"2024-09-19T07:49:01.802657+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:49:01.802657+0000","last_undegraded":"2024-09-19T07:49:01.802657+0000","last_fullsized":"2024-09-19T07:49:01.802657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:46.226194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":181,"num_read_kb":77990,"num_write":347,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"64'3","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371378+0000","last_change":"2024-09-19T07:19:39.423972+0000","last_active":"2024-09-19T07:48:58.371378+0000","last_peered":"2024-09-19T07:48:58.371378+0000","last_clean":"2024-09-19T07:48:58.371378+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:48:58.371378+0000","last_undegraded":"2024-09-19T07:48:58.371378+0000","last_fullsized":"2024-09-19T07:48:58.371378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"64'3743","reported_seq":3848,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371627+0000","last_change":"2024-09-19T07:19:40.382092+0000","last_active":"2024-09-19T07:48:58.371627+0000","last_peered":"2024-09-19T07:48:58.371627+0000","last_clean":"2024-09-19T07:48:58.371627+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:48:58.371627+0000","last_undegraded":"2024-09-19T07:48:58.371627+0000","last_fullsized":"2024-09-19T07:48:58.371627+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20626,"num_write_kb":263086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"61'6160","reported_seq":6272,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369147+0000","last_change":"2024-09-19T07:19:32.292251+0000","last_active":"2024-09-19T07:48:58.369147+0000","last_peered":"2024-09-19T07:48:58.369147+0000","last_clean":"2024-09-19T07:48:58.369147+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:48:58.369147+0000","last_undegraded":"2024-09-19T07:48:58.369147+0000","last_fullsized":"2024-09-19T07:48:58.369147+0000","mapping_epoch":19,"log_start":"54'700","ondisk_log_start":"54'700","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:18.328386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32461,"num_write_kb":376158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"65'29","reported_seq":204,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:03.379891+0000","last_change":"2024-09-19T07:19:31.290676+0000","last_active":"2024-09-19T07:49:03.379891+0000","last_peered":"2024-09-19T07:49:03.379891+0000","last_clean":"2024-09-19T07:49:03.379891+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:49:03.379891+0000","last_undegraded":"2024-09-19T07:49:03.379891+0000","last_fullsized":"2024-09-19T07:49:03.379891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"64'8","reported_seq":154,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369556+0000","last_change":"2024-09-19T07:19:39.424879+0000","last_active":"2024-09-19T07:48:58.369556+0000","last_peered":"2024-09-19T07:48:58.369556+0000","last_clean":"2024-09-19T07:48:58.369556+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:48:58.369556+0000","last_undegraded":"2024-09-19T07:48:58.369556+0000","last_fullsized":"2024-09-19T07:48:58.369556+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:30.021543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"64'4564","reported_seq":4670,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382249+0000","last_change":"2024-09-19T07:19:40.384136+0000","last_active":"2024-09-19T07:48:58.382249+0000","last_peered":"2024-09-19T07:48:58.382249+0000","last_clean":"2024-09-19T07:48:58.382249+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:48:58.382249+0000","last_undegraded":"2024-09-19T07:48:58.382249+0000","last_fullsized":"2024-09-19T07:48:58.382249+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:06:33.098225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25533,"num_write_kb":303195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"61'5068","reported_seq":5181,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369959+0000","last_change":"2024-09-19T07:19:32.271355+0000","last_active":"2024-09-19T07:48:58.369959+0000","last_peered":"2024-09-19T07:48:58.369959+0000","last_clean":"2024-09-19T07:48:58.369959+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:48:58.369959+0000","last_undegraded":"2024-09-19T07:48:58.369959+0000","last_fullsized":"2024-09-19T07:48:58.369959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26828,"num_write_kb":316657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"61'3","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368011+0000","last_change":"2024-09-19T07:19:31.287158+0000","last_active":"2024-09-19T07:48:58.368011+0000","last_peered":"2024-09-19T07:48:58.368011+0000","last_clean":"2024-09-19T07:48:58.368011+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:48:58.368011+0000","last_undegraded":"2024-09-19T07:48:58.368011+0000","last_fullsized":"2024-09-19T07:48:58.368011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:44:38.990721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'15","reported_seq":205,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362420+0000","last_change":"2024-09-19T07:19:39.410445+0000","last_active":"2024-09-19T07:48:58.362420+0000","last_peered":"2024-09-19T07:48:58.362420+0000","last_clean":"2024-09-19T07:48:58.362420+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:48:58.362420+0000","last_undegraded":"2024-09-19T07:48:58.362420+0000","last_fullsized":"2024-09-19T07:48:58.362420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:27.020327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"64'3771","reported_seq":3875,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373168+0000","last_change":"2024-09-19T07:19:40.389499+0000","last_active":"2024-09-19T07:48:58.373168+0000","last_peered":"2024-09-19T07:48:58.373168+0000","last_clean":"2024-09-19T07:48:58.373168+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:48:58.373168+0000","last_undegraded":"2024-09-19T07:48:58.373168+0000","last_fullsized":"2024-09-19T07:48:58.373168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3771,"log_dups_size":0,"ondisk_log_size":3771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.520872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19869,"num_write_kb":255725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"61'4973","reported_seq":5085,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367096+0000","last_change":"2024-09-19T07:19:32.272259+0000","last_active":"2024-09-19T07:48:58.367096+0000","last_peered":"2024-09-19T07:48:58.367096+0000","last_clean":"2024-09-19T07:48:58.367096+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:48:58.367096+0000","last_undegraded":"2024-09-19T07:48:58.367096+0000","last_fullsized":"2024-09-19T07:48:58.367096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:16:40.832635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26723,"num_write_kb":320185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"65'6","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382937+0000","last_change":"2024-09-19T07:19:31.293834+0000","last_active":"2024-09-19T07:48:58.382937+0000","last_peered":"2024-09-19T07:48:58.382937+0000","last_clean":"2024-09-19T07:48:58.382937+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:48:58.382937+0000","last_undegraded":"2024-09-19T07:48:58.382937+0000","last_fullsized":"2024-09-19T07:48:58.382937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:32.974191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"64'5","reported_seq":111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362191+0000","last_change":"2024-09-19T07:19:39.416943+0000","last_active":"2024-09-19T07:48:58.362191+0000","last_peered":"2024-09-19T07:48:58.362191+0000","last_clean":"2024-09-19T07:48:58.362191+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:48:58.362191+0000","last_undegraded":"2024-09-19T07:48:58.362191+0000","last_fullsized":"2024-09-19T07:48:58.362191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:35.555826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"64'5394","reported_seq":5500,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382132+0000","last_change":"2024-09-19T07:19:40.394993+0000","last_active":"2024-09-19T07:48:58.382132+0000","last_peered":"2024-09-19T07:48:58.382132+0000","last_clean":"2024-09-19T07:48:58.382132+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:48:58.382132+0000","last_undegraded":"2024-09-19T07:48:58.382132+0000","last_fullsized":"2024-09-19T07:48:58.382132+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5394,"log_dups_size":0,"ondisk_log_size":5394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:28.023172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30095,"num_write_kb":374705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"64'25","reported_seq":134,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369672+0000","last_change":"2024-09-19T07:19:39.418885+0000","last_active":"2024-09-19T07:48:58.369672+0000","last_peered":"2024-09-19T07:48:58.369672+0000","last_clean":"2024-09-19T07:48:58.369672+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:48:58.369672+0000","last_undegraded":"2024-09-19T07:48:58.369672+0000","last_fullsized":"2024-09-19T07:48:58.369672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"61'4822","reported_seq":4935,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372065+0000","last_change":"2024-09-19T07:19:32.275229+0000","last_active":"2024-09-19T07:48:58.372065+0000","last_peered":"2024-09-19T07:48:58.372065+0000","last_clean":"2024-09-19T07:48:58.372065+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:48:58.372065+0000","last_undegraded":"2024-09-19T07:48:58.372065+0000","last_fullsized":"2024-09-19T07:48:58.372065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25208,"num_write_kb":326524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"61'2","reported_seq":116,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367640+0000","last_change":"2024-09-19T07:19:31.286558+0000","last_active":"2024-09-19T07:48:58.367640+0000","last_peered":"2024-09-19T07:48:58.367640+0000","last_clean":"2024-09-19T07:48:58.367640+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:48:58.367640+0000","last_undegraded":"2024-09-19T07:48:58.367640+0000","last_fullsized":"2024-09-19T07:48:58.367640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:44:42.566687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"64'4690","reported_seq":4797,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369471+0000","last_change":"2024-09-19T07:19:40.374048+0000","last_active":"2024-09-19T07:48:58.369471+0000","last_peered":"2024-09-19T07:48:58.369471+0000","last_clean":"2024-09-19T07:48:58.369471+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:48:58.369471+0000","last_undegraded":"2024-09-19T07:48:58.369471+0000","last_fullsized":"2024-09-19T07:48:58.369471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25924,"num_write_kb":313037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"64'964","reported_seq":1117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382679+0000","last_change":"2024-09-19T07:19:39.432426+0000","last_active":"2024-09-19T07:48:58.382679+0000","last_peered":"2024-09-19T07:48:58.382679+0000","last_clean":"2024-09-19T07:48:58.382679+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:48:58.382679+0000","last_undegraded":"2024-09-19T07:48:58.382679+0000","last_fullsized":"2024-09-19T07:48:58.382679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":964,"log_dups_size":0,"ondisk_log_size":964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:12:21.994900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":8896,"num_write":940,"num_write_kb":4641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"61'5311","reported_seq":5423,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362632+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:48:58.362632+0000","last_peered":"2024-09-19T07:48:58.362632+0000","last_clean":"2024-09-19T07:48:58.362632+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:48:58.362632+0000","last_undegraded":"2024-09-19T07:48:58.362632+0000","last_fullsized":"2024-09-19T07:48:58.362632+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:56:37.661555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28108,"num_write_kb":333978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"61'3","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363067+0000","last_change":"2024-09-19T07:19:31.271698+0000","last_active":"2024-09-19T07:48:58.363067+0000","last_peered":"2024-09-19T07:48:58.363067+0000","last_clean":"2024-09-19T07:48:58.363067+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:48:58.363067+0000","last_undegraded":"2024-09-19T07:48:58.363067+0000","last_fullsized":"2024-09-19T07:48:58.363067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:03:38.541649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"64'5270","reported_seq":5374,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373635+0000","last_change":"2024-09-19T07:19:40.384934+0000","last_active":"2024-09-19T07:48:58.373635+0000","last_peered":"2024-09-19T07:48:58.373635+0000","last_clean":"2024-09-19T07:48:58.373635+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:48:58.373635+0000","last_undegraded":"2024-09-19T07:48:58.373635+0000","last_fullsized":"2024-09-19T07:48:58.373635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29270,"num_write_kb":363254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"65'416","reported_seq":535,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:04.513350+0000","last_change":"2024-09-19T07:19:31.269628+0000","last_active":"2024-09-19T07:49:04.513350+0000","last_peered":"2024-09-19T07:49:04.513350+0000","last_clean":"2024-09-19T07:49:04.513350+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:49:04.513350+0000","last_undegraded":"2024-09-19T07:49:04.513350+0000","last_fullsized":"2024-09-19T07:49:04.513350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:41:43.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2715831,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6193,"num_write":390,"num_write_kb":2885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"61'3569","reported_seq":3689,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:20.599524+0000","last_change":"2024-09-19T07:49:20.599569+0000","last_active":"2024-09-19T07:49:20.599524+0000","last_peered":"2024-09-19T07:49:20.599524+0000","last_clean":"2024-09-19T07:49:20.599524+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:49:20.599524+0000","last_undegraded":"2024-09-19T07:49:20.599524+0000","last_fullsized":"2024-09-19T07:49:20.599524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3569","last_scrub_stamp":"2024-09-19T07:49:20.599393+0000","last_deep_scrub":"61'3569","last_deep_scrub_stamp":"2024-09-19T07:49:20.599393+0000","last_clean_scrub_stamp":"2024-09-19T07:49:20.599393+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:56.986977+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18120,"num_write_kb":244659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"64'5","reported_seq":151,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378792+0000","last_change":"2024-09-19T07:19:39.410582+0000","last_active":"2024-09-19T07:48:58.378792+0000","last_peered":"2024-09-19T07:48:58.378792+0000","last_clean":"2024-09-19T07:48:58.378792+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:48:58.378792+0000","last_undegraded":"2024-09-19T07:48:58.378792+0000","last_fullsized":"2024-09-19T07:48:58.378792+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"64'3415","reported_seq":3520,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364298+0000","last_change":"2024-09-19T07:19:40.393950+0000","last_active":"2024-09-19T07:48:58.364298+0000","last_peered":"2024-09-19T07:48:58.364298+0000","last_clean":"2024-09-19T07:48:58.364298+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:48:58.364298+0000","last_undegraded":"2024-09-19T07:48:58.364298+0000","last_fullsized":"2024-09-19T07:48:58.364298+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:42.774552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18072,"num_write_kb":240550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"61'712","reported_seq":848,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364844+0000","last_change":"2024-09-19T07:19:31.266417+0000","last_active":"2024-09-19T07:48:58.364844+0000","last_peered":"2024-09-19T07:48:58.364844+0000","last_clean":"2024-09-19T07:48:58.364844+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:48:58.364844+0000","last_undegraded":"2024-09-19T07:48:58.364844+0000","last_fullsized":"2024-09-19T07:48:58.364844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:58.484806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":74743,"num_write":701,"num_write_kb":4298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"61'4395","reported_seq":4507,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368988+0000","last_change":"2024-09-19T07:19:32.306418+0000","last_active":"2024-09-19T07:48:58.368988+0000","last_peered":"2024-09-19T07:48:58.368988+0000","last_clean":"2024-09-19T07:48:58.368988+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:48:58.368988+0000","last_undegraded":"2024-09-19T07:48:58.368988+0000","last_fullsized":"2024-09-19T07:48:58.368988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:47.236228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22455,"num_write_kb":278564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"64'40","reported_seq":185,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363443+0000","last_change":"2024-09-19T07:19:39.410662+0000","last_active":"2024-09-19T07:48:58.363443+0000","last_peered":"2024-09-19T07:48:58.363443+0000","last_clean":"2024-09-19T07:48:58.363443+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:48:58.363443+0000","last_undegraded":"2024-09-19T07:48:58.363443+0000","last_fullsized":"2024-09-19T07:48:58.363443+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:38.375757+0000","last_clean_scrub_stamp":"2024-09-19T07:19:38.375757+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:50:19.196074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"64'4798","reported_seq":4903,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364274+0000","last_change":"2024-09-19T07:19:40.385581+0000","last_active":"2024-09-19T07:48:58.364274+0000","last_peered":"2024-09-19T07:48:58.364274+0000","last_clean":"2024-09-19T07:48:58.364274+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:48:58.364274+0000","last_undegraded":"2024-09-19T07:48:58.364274+0000","last_fullsized":"2024-09-19T07:48:58.364274+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:14:13.749729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27082,"num_write_kb":321689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"61'4997","reported_seq":5111,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380242+0000","last_change":"2024-09-19T07:19:32.307594+0000","last_active":"2024-09-19T07:48:58.380242+0000","last_peered":"2024-09-19T07:48:58.380242+0000","last_clean":"2024-09-19T07:48:58.380242+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:48:58.380242+0000","last_undegraded":"2024-09-19T07:48:58.380242+0000","last_fullsized":"2024-09-19T07:48:58.380242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4997,"log_dups_size":0,"ondisk_log_size":4997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26206,"num_write_kb":309941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"64'5490","reported_seq":5597,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373867+0000","last_change":"2024-09-19T07:19:40.377608+0000","last_active":"2024-09-19T07:48:58.373867+0000","last_peered":"2024-09-19T07:48:58.373867+0000","last_clean":"2024-09-19T07:48:58.373867+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:48:58.373867+0000","last_undegraded":"2024-09-19T07:48:58.373867+0000","last_fullsized":"2024-09-19T07:48:58.373867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5490,"log_dups_size":0,"ondisk_log_size":5490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29613,"num_write_kb":364653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"61'4890","reported_seq":5006,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377558+0000","last_change":"2024-09-19T07:19:32.297364+0000","last_active":"2024-09-19T07:48:58.377558+0000","last_peered":"2024-09-19T07:48:58.377558+0000","last_clean":"2024-09-19T07:48:58.377558+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:48:58.377558+0000","last_undegraded":"2024-09-19T07:48:58.377558+0000","last_fullsized":"2024-09-19T07:48:58.377558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4890,"log_dups_size":0,"ondisk_log_size":4890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25338,"num_write_kb":314559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"64'4569","reported_seq":4675,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366516+0000","last_change":"2024-09-19T07:19:40.381262+0000","last_active":"2024-09-19T07:48:58.366516+0000","last_peered":"2024-09-19T07:48:58.366516+0000","last_clean":"2024-09-19T07:48:58.366516+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:48:58.366516+0000","last_undegraded":"2024-09-19T07:48:58.366516+0000","last_fullsized":"2024-09-19T07:48:58.366516+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:34.425472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25275,"num_write_kb":314704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"61'5227","reported_seq":5343,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367085+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:48:58.367085+0000","last_peered":"2024-09-19T07:48:58.367085+0000","last_clean":"2024-09-19T07:48:58.367085+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:48:58.367085+0000","last_undegraded":"2024-09-19T07:48:58.367085+0000","last_fullsized":"2024-09-19T07:48:58.367085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:30:50.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27444,"num_write_kb":331433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"64'4939","reported_seq":5045,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373610+0000","last_change":"2024-09-19T07:19:40.386432+0000","last_active":"2024-09-19T07:48:58.373610+0000","last_peered":"2024-09-19T07:48:58.373610+0000","last_clean":"2024-09-19T07:48:58.373610+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:48:58.373610+0000","last_undegraded":"2024-09-19T07:48:58.373610+0000","last_fullsized":"2024-09-19T07:48:58.373610+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27263,"num_write_kb":334786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"61'4385","reported_seq":4501,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371370+0000","last_change":"2024-09-19T07:19:32.268592+0000","last_active":"2024-09-19T07:48:58.371370+0000","last_peered":"2024-09-19T07:48:58.371370+0000","last_clean":"2024-09-19T07:48:58.371370+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:48:58.371370+0000","last_undegraded":"2024-09-19T07:48:58.371370+0000","last_fullsized":"2024-09-19T07:48:58.371370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22838,"num_write_kb":271812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"64'4859","reported_seq":4967,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372828+0000","last_change":"2024-09-19T07:19:40.378630+0000","last_active":"2024-09-19T07:48:58.372828+0000","last_peered":"2024-09-19T07:48:58.372828+0000","last_clean":"2024-09-19T07:48:58.372828+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:48:58.372828+0000","last_undegraded":"2024-09-19T07:48:58.372828+0000","last_fullsized":"2024-09-19T07:48:58.372828+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:28:00.957765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26164,"num_write_kb":338417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"61'4483","reported_seq":4597,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364173+0000","last_change":"2024-09-19T07:19:32.312759+0000","last_active":"2024-09-19T07:48:58.364173+0000","last_peered":"2024-09-19T07:48:58.364173+0000","last_clean":"2024-09-19T07:48:58.364173+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:48:58.364173+0000","last_undegraded":"2024-09-19T07:48:58.364173+0000","last_fullsized":"2024-09-19T07:48:58.364173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:54:57.034521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23870,"num_write_kb":292652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"64'4105","reported_seq":4212,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368603+0000","last_change":"2024-09-19T07:19:40.381365+0000","last_active":"2024-09-19T07:48:58.368603+0000","last_peered":"2024-09-19T07:48:58.368603+0000","last_clean":"2024-09-19T07:48:58.368603+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:48:58.368603+0000","last_undegraded":"2024-09-19T07:48:58.368603+0000","last_fullsized":"2024-09-19T07:48:58.368603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:42:04.956629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21923,"num_write_kb":284540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"61'5078","reported_seq":5192,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374126+0000","last_change":"2024-09-19T07:19:32.277273+0000","last_active":"2024-09-19T07:48:58.374126+0000","last_peered":"2024-09-19T07:48:58.374126+0000","last_clean":"2024-09-19T07:48:58.374126+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:48:58.374126+0000","last_undegraded":"2024-09-19T07:48:58.374126+0000","last_fullsized":"2024-09-19T07:48:58.374126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28059,"num_write_kb":313378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"64'3824","reported_seq":3930,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372461+0000","last_change":"2024-09-19T07:19:40.387116+0000","last_active":"2024-09-19T07:48:58.372461+0000","last_peered":"2024-09-19T07:48:58.372461+0000","last_clean":"2024-09-19T07:48:58.372461+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:48:58.372461+0000","last_undegraded":"2024-09-19T07:48:58.372461+0000","last_fullsized":"2024-09-19T07:48:58.372461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:32.013045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20622,"num_write_kb":295649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"61'4725","reported_seq":4838,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382564+0000","last_change":"2024-09-19T07:19:32.303343+0000","last_active":"2024-09-19T07:48:58.382564+0000","last_peered":"2024-09-19T07:48:58.382564+0000","last_clean":"2024-09-19T07:48:58.382564+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:48:58.382564+0000","last_undegraded":"2024-09-19T07:48:58.382564+0000","last_fullsized":"2024-09-19T07:48:58.382564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25170,"num_write_kb":293477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"64'4764","reported_seq":4871,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.380511+0000","last_change":"2024-09-19T07:19:40.357629+0000","last_active":"2024-09-19T07:48:58.380511+0000","last_peered":"2024-09-19T07:48:58.380511+0000","last_clean":"2024-09-19T07:48:58.380511+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:48:58.380511+0000","last_undegraded":"2024-09-19T07:48:58.380511+0000","last_fullsized":"2024-09-19T07:48:58.380511+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:48:36.558671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25208,"num_write_kb":309995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"61'5411","reported_seq":5525,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364434+0000","last_change":"2024-09-19T07:19:32.289099+0000","last_active":"2024-09-19T07:48:58.364434+0000","last_peered":"2024-09-19T07:48:58.364434+0000","last_clean":"2024-09-19T07:48:58.364434+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:48:58.364434+0000","last_undegraded":"2024-09-19T07:48:58.364434+0000","last_fullsized":"2024-09-19T07:48:58.364434+0000","mapping_epoch":19,"log_start":"54'200","ondisk_log_start":"54'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28760,"num_write_kb":348949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"64'4221","reported_seq":4326,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363476+0000","last_change":"2024-09-19T07:19:40.388597+0000","last_active":"2024-09-19T07:48:58.363476+0000","last_peered":"2024-09-19T07:48:58.363476+0000","last_clean":"2024-09-19T07:48:58.363476+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:48:58.363476+0000","last_undegraded":"2024-09-19T07:48:58.363476+0000","last_fullsized":"2024-09-19T07:48:58.363476+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:34:18.402016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21805,"num_write_kb":288249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"61'4974","reported_seq":5093,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365664+0000","last_change":"2024-09-19T07:20:13.954029+0000","last_active":"2024-09-19T07:48:58.365664+0000","last_peered":"2024-09-19T07:48:58.365664+0000","last_clean":"2024-09-19T07:48:58.365664+0000","last_became_active":"2024-09-19T07:20:13.953118+0000","last_became_peered":"2024-09-19T07:20:13.953118+0000","last_unstale":"2024-09-19T07:48:58.365664+0000","last_undegraded":"2024-09-19T07:48:58.365664+0000","last_fullsized":"2024-09-19T07:48:58.365664+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:27:21.388787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25995,"num_write_kb":309048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"64'5715","reported_seq":5819,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375222+0000","last_change":"2024-09-19T07:19:40.383779+0000","last_active":"2024-09-19T07:48:58.375222+0000","last_peered":"2024-09-19T07:48:58.375222+0000","last_clean":"2024-09-19T07:48:58.375222+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:48:58.375222+0000","last_undegraded":"2024-09-19T07:48:58.375222+0000","last_fullsized":"2024-09-19T07:48:58.375222+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5715,"log_dups_size":0,"ondisk_log_size":5715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:16:28.887199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30021,"num_write_kb":374301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"61'4405","reported_seq":4518,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362645+0000","last_change":"2024-09-19T07:19:32.312703+0000","last_active":"2024-09-19T07:48:58.362645+0000","last_peered":"2024-09-19T07:48:58.362645+0000","last_clean":"2024-09-19T07:48:58.362645+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:48:58.362645+0000","last_undegraded":"2024-09-19T07:48:58.362645+0000","last_fullsized":"2024-09-19T07:48:58.362645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4405,"log_dups_size":0,"ondisk_log_size":4405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23742,"num_write_kb":275461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"64'3931","reported_seq":4037,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378216+0000","last_change":"2024-09-19T07:19:40.376342+0000","last_active":"2024-09-19T07:48:58.378216+0000","last_peered":"2024-09-19T07:48:58.378216+0000","last_clean":"2024-09-19T07:48:58.378216+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:48:58.378216+0000","last_undegraded":"2024-09-19T07:48:58.378216+0000","last_fullsized":"2024-09-19T07:48:58.378216+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3931,"log_dups_size":0,"ondisk_log_size":3931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22002,"num_write_kb":260466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"61'4975","reported_seq":5088,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365882+0000","last_change":"2024-09-19T07:19:32.277968+0000","last_active":"2024-09-19T07:48:58.365882+0000","last_peered":"2024-09-19T07:48:58.365882+0000","last_clean":"2024-09-19T07:48:58.365882+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:48:58.365882+0000","last_undegraded":"2024-09-19T07:48:58.365882+0000","last_fullsized":"2024-09-19T07:48:58.365882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4975,"log_dups_size":0,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:05:40.426447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26500,"num_write_kb":311194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"64'5003","reported_seq":5108,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362879+0000","last_change":"2024-09-19T07:19:40.361504+0000","last_active":"2024-09-19T07:48:58.362879+0000","last_peered":"2024-09-19T07:48:58.362879+0000","last_clean":"2024-09-19T07:48:58.362879+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:48:58.362879+0000","last_undegraded":"2024-09-19T07:48:58.362879+0000","last_fullsized":"2024-09-19T07:48:58.362879+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:30:31.110669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26890,"num_write_kb":325027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"61'5877","reported_seq":5993,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.370562+0000","last_change":"2024-09-19T07:19:32.285510+0000","last_active":"2024-09-19T07:48:58.370562+0000","last_peered":"2024-09-19T07:48:58.370562+0000","last_clean":"2024-09-19T07:48:58.370562+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:48:58.370562+0000","last_undegraded":"2024-09-19T07:48:58.370562+0000","last_fullsized":"2024-09-19T07:48:58.370562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":30970,"num_write_kb":370397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"64'4684","reported_seq":4789,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364347+0000","last_change":"2024-09-19T07:19:40.390205+0000","last_active":"2024-09-19T07:48:58.364347+0000","last_peered":"2024-09-19T07:48:58.364347+0000","last_clean":"2024-09-19T07:48:58.364347+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:48:58.364347+0000","last_undegraded":"2024-09-19T07:48:58.364347+0000","last_fullsized":"2024-09-19T07:48:58.364347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:53:53.730095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25566,"num_write_kb":331994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"61'4261","reported_seq":4374,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381790+0000","last_change":"2024-09-19T07:19:32.316746+0000","last_active":"2024-09-19T07:48:58.381790+0000","last_peered":"2024-09-19T07:48:58.381790+0000","last_clean":"2024-09-19T07:48:58.381790+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:48:58.381790+0000","last_undegraded":"2024-09-19T07:48:58.381790+0000","last_fullsized":"2024-09-19T07:48:58.381790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:32:00.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22573,"num_write_kb":274878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"64'5240","reported_seq":5347,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363651+0000","last_change":"2024-09-19T07:19:40.390559+0000","last_active":"2024-09-19T07:48:58.363651+0000","last_peered":"2024-09-19T07:48:58.363651+0000","last_clean":"2024-09-19T07:48:58.363651+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:48:58.363651+0000","last_undegraded":"2024-09-19T07:48:58.363651+0000","last_fullsized":"2024-09-19T07:48:58.363651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:02:48.506476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":28154,"num_write_kb":354961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"61'5095","reported_seq":5208,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379268+0000","last_change":"2024-09-19T07:19:32.275235+0000","last_active":"2024-09-19T07:48:58.379268+0000","last_peered":"2024-09-19T07:48:58.379268+0000","last_clean":"2024-09-19T07:48:58.379268+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:48:58.379268+0000","last_undegraded":"2024-09-19T07:48:58.379268+0000","last_fullsized":"2024-09-19T07:48:58.379268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:44:12.399186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26748,"num_write_kb":324403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"64'4078","reported_seq":4184,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363850+0000","last_change":"2024-09-19T07:19:40.358433+0000","last_active":"2024-09-19T07:48:58.363850+0000","last_peered":"2024-09-19T07:48:58.363850+0000","last_clean":"2024-09-19T07:48:58.363850+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:48:58.363850+0000","last_undegraded":"2024-09-19T07:48:58.363850+0000","last_fullsized":"2024-09-19T07:48:58.363850+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:10:25.226677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22110,"num_write_kb":293452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"61'4908","reported_seq":5022,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.378245+0000","last_change":"2024-09-19T07:19:32.302287+0000","last_active":"2024-09-19T07:48:58.378245+0000","last_peered":"2024-09-19T07:48:58.378245+0000","last_clean":"2024-09-19T07:48:58.378245+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:48:58.378245+0000","last_undegraded":"2024-09-19T07:48:58.378245+0000","last_fullsized":"2024-09-19T07:48:58.378245+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25574,"num_write_kb":315713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"64'3983","reported_seq":4089,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368947+0000","last_change":"2024-09-19T07:19:40.374374+0000","last_active":"2024-09-19T07:48:58.368947+0000","last_peered":"2024-09-19T07:48:58.368947+0000","last_clean":"2024-09-19T07:48:58.368947+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:48:58.368947+0000","last_undegraded":"2024-09-19T07:48:58.368947+0000","last_fullsized":"2024-09-19T07:48:58.368947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21503,"num_write_kb":274721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"61'5975","reported_seq":6087,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371924+0000","last_change":"2024-09-19T07:19:32.278891+0000","last_active":"2024-09-19T07:48:58.371924+0000","last_peered":"2024-09-19T07:48:58.371924+0000","last_clean":"2024-09-19T07:48:58.371924+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:48:58.371924+0000","last_undegraded":"2024-09-19T07:48:58.371924+0000","last_fullsized":"2024-09-19T07:48:58.371924+0000","mapping_epoch":19,"log_start":"53'200","ondisk_log_start":"53'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:11.060522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31472,"num_write_kb":384213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"64'4930","reported_seq":5035,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373529+0000","last_change":"2024-09-19T07:19:40.387341+0000","last_active":"2024-09-19T07:48:58.373529+0000","last_peered":"2024-09-19T07:48:58.373529+0000","last_clean":"2024-09-19T07:48:58.373529+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:48:58.373529+0000","last_undegraded":"2024-09-19T07:48:58.373529+0000","last_fullsized":"2024-09-19T07:48:58.373529+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26807,"num_write_kb":336825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"61'5543","reported_seq":5655,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365417+0000","last_change":"2024-09-19T07:19:32.276311+0000","last_active":"2024-09-19T07:48:58.365417+0000","last_peered":"2024-09-19T07:48:58.365417+0000","last_clean":"2024-09-19T07:48:58.365417+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:48:58.365417+0000","last_undegraded":"2024-09-19T07:48:58.365417+0000","last_fullsized":"2024-09-19T07:48:58.365417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5543,"log_dups_size":0,"ondisk_log_size":5543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:09.325105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29751,"num_write_kb":336461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"64'5766","reported_seq":5872,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369027+0000","last_change":"2024-09-19T07:19:40.388484+0000","last_active":"2024-09-19T07:48:58.369027+0000","last_peered":"2024-09-19T07:48:58.369027+0000","last_clean":"2024-09-19T07:48:58.369027+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:48:58.369027+0000","last_undegraded":"2024-09-19T07:48:58.369027+0000","last_fullsized":"2024-09-19T07:48:58.369027+0000","mapping_epoch":23,"log_start":"58'300","ondisk_log_start":"58'300","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5466,"log_dups_size":0,"ondisk_log_size":5466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:02.295867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32274,"num_write_kb":395900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"61'4235","reported_seq":4348,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372369+0000","last_change":"2024-09-19T07:19:32.290586+0000","last_active":"2024-09-19T07:48:58.372369+0000","last_peered":"2024-09-19T07:48:58.372369+0000","last_clean":"2024-09-19T07:48:58.372369+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:48:58.372369+0000","last_undegraded":"2024-09-19T07:48:58.372369+0000","last_fullsized":"2024-09-19T07:48:58.372369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22758,"num_write_kb":275574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"64'3417","reported_seq":3521,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365853+0000","last_change":"2024-09-19T07:19:40.367578+0000","last_active":"2024-09-19T07:48:58.365853+0000","last_peered":"2024-09-19T07:48:58.365853+0000","last_clean":"2024-09-19T07:48:58.365853+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:48:58.365853+0000","last_undegraded":"2024-09-19T07:48:58.365853+0000","last_fullsized":"2024-09-19T07:48:58.365853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:00:25.357854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18157,"num_write_kb":232641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"61'4616","reported_seq":4729,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369453+0000","last_change":"2024-09-19T07:19:32.279942+0000","last_active":"2024-09-19T07:48:58.369453+0000","last_peered":"2024-09-19T07:48:58.369453+0000","last_clean":"2024-09-19T07:48:58.369453+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:48:58.369453+0000","last_undegraded":"2024-09-19T07:48:58.369453+0000","last_fullsized":"2024-09-19T07:48:58.369453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24409,"num_write_kb":298784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"64'4320","reported_seq":4424,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374872+0000","last_change":"2024-09-19T07:19:40.385641+0000","last_active":"2024-09-19T07:48:58.374872+0000","last_peered":"2024-09-19T07:48:58.374872+0000","last_clean":"2024-09-19T07:48:58.374872+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:48:58.374872+0000","last_undegraded":"2024-09-19T07:48:58.374872+0000","last_fullsized":"2024-09-19T07:48:58.374872+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22755,"num_write_kb":287920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"61'4140","reported_seq":4253,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363901+0000","last_change":"2024-09-19T07:19:32.284976+0000","last_active":"2024-09-19T07:48:58.363901+0000","last_peered":"2024-09-19T07:48:58.363901+0000","last_clean":"2024-09-19T07:48:58.363901+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:48:58.363901+0000","last_undegraded":"2024-09-19T07:48:58.363901+0000","last_fullsized":"2024-09-19T07:48:58.363901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:04:48.566879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22395,"num_write_kb":253944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"64'3972","reported_seq":4076,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364583+0000","last_change":"2024-09-19T07:19:40.367101+0000","last_active":"2024-09-19T07:48:58.364583+0000","last_peered":"2024-09-19T07:48:58.364583+0000","last_clean":"2024-09-19T07:48:58.364583+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:48:58.364583+0000","last_undegraded":"2024-09-19T07:48:58.364583+0000","last_fullsized":"2024-09-19T07:48:58.364583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:10.541396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21876,"num_write_kb":260064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"61'3526","reported_seq":3638,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.368962+0000","last_change":"2024-09-19T07:19:32.308507+0000","last_active":"2024-09-19T07:48:58.368962+0000","last_peered":"2024-09-19T07:48:58.368962+0000","last_clean":"2024-09-19T07:48:58.368962+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:48:58.368962+0000","last_undegraded":"2024-09-19T07:48:58.368962+0000","last_fullsized":"2024-09-19T07:48:58.368962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:49.664633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18762,"num_write_kb":235013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"64'3609","reported_seq":3714,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377691+0000","last_change":"2024-09-19T07:19:40.384212+0000","last_active":"2024-09-19T07:48:58.377691+0000","last_peered":"2024-09-19T07:48:58.377691+0000","last_clean":"2024-09-19T07:48:58.377691+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:48:58.377691+0000","last_undegraded":"2024-09-19T07:48:58.377691+0000","last_fullsized":"2024-09-19T07:48:58.377691+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20332,"num_write_kb":258088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"61'4660","reported_seq":4777,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381227+0000","last_change":"2024-09-19T07:20:13.952931+0000","last_active":"2024-09-19T07:48:58.381227+0000","last_peered":"2024-09-19T07:48:58.381227+0000","last_clean":"2024-09-19T07:48:58.381227+0000","last_became_active":"2024-09-19T07:20:13.952103+0000","last_became_peered":"2024-09-19T07:20:13.952103+0000","last_unstale":"2024-09-19T07:48:58.381227+0000","last_undegraded":"2024-09-19T07:48:58.381227+0000","last_fullsized":"2024-09-19T07:48:58.381227+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:23.862116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24375,"num_write_kb":290368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"64'3991","reported_seq":4095,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375705+0000","last_change":"2024-09-19T07:19:40.387401+0000","last_active":"2024-09-19T07:48:58.375705+0000","last_peered":"2024-09-19T07:48:58.375705+0000","last_clean":"2024-09-19T07:48:58.375705+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:48:58.375705+0000","last_undegraded":"2024-09-19T07:48:58.375705+0000","last_fullsized":"2024-09-19T07:48:58.375705+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3991,"log_dups_size":0,"ondisk_log_size":3991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:48:44.103780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21657,"num_write_kb":287665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"61'5448","reported_seq":5560,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381563+0000","last_change":"2024-09-19T07:19:32.300583+0000","last_active":"2024-09-19T07:48:58.381563+0000","last_peered":"2024-09-19T07:48:58.381563+0000","last_clean":"2024-09-19T07:48:58.381563+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:48:58.381563+0000","last_undegraded":"2024-09-19T07:48:58.381563+0000","last_fullsized":"2024-09-19T07:48:58.381563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:28.735519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29399,"num_write_kb":347457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"64'4539","reported_seq":4643,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365336+0000","last_change":"2024-09-19T07:19:40.361430+0000","last_active":"2024-09-19T07:48:58.365336+0000","last_peered":"2024-09-19T07:48:58.365336+0000","last_clean":"2024-09-19T07:48:58.365336+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:48:58.365336+0000","last_undegraded":"2024-09-19T07:48:58.365336+0000","last_fullsized":"2024-09-19T07:48:58.365336+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4539,"log_dups_size":0,"ondisk_log_size":4539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:23:05.185943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24944,"num_write_kb":313178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"61'6280","reported_seq":6394,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.376418+0000","last_change":"2024-09-19T07:19:32.283272+0000","last_active":"2024-09-19T07:48:58.376418+0000","last_peered":"2024-09-19T07:48:58.376418+0000","last_clean":"2024-09-19T07:48:58.376418+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:48:58.376418+0000","last_undegraded":"2024-09-19T07:48:58.376418+0000","last_fullsized":"2024-09-19T07:48:58.376418+0000","mapping_epoch":19,"log_start":"54'500","ondisk_log_start":"54'500","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:03:13.866753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":34141,"num_write_kb":395176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"64'5507","reported_seq":5613,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363380+0000","last_change":"2024-09-19T07:19:40.378172+0000","last_active":"2024-09-19T07:48:58.363380+0000","last_peered":"2024-09-19T07:48:58.363380+0000","last_clean":"2024-09-19T07:48:58.363380+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:48:58.363380+0000","last_undegraded":"2024-09-19T07:48:58.363380+0000","last_fullsized":"2024-09-19T07:48:58.363380+0000","mapping_epoch":23,"log_start":"58'300","ondisk_log_start":"58'300","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:52.321790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30086,"num_write_kb":370527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"61'4537","reported_seq":4648,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362562+0000","last_change":"2024-09-19T07:19:32.272279+0000","last_active":"2024-09-19T07:48:58.362562+0000","last_peered":"2024-09-19T07:48:58.362562+0000","last_clean":"2024-09-19T07:48:58.362562+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:48:58.362562+0000","last_undegraded":"2024-09-19T07:48:58.362562+0000","last_fullsized":"2024-09-19T07:48:58.362562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:29.381965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24234,"num_write_kb":310307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"64'4532","reported_seq":4639,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381234+0000","last_change":"2024-09-19T07:19:40.373909+0000","last_active":"2024-09-19T07:48:58.381234+0000","last_peered":"2024-09-19T07:48:58.381234+0000","last_clean":"2024-09-19T07:48:58.381234+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:48:58.381234+0000","last_undegraded":"2024-09-19T07:48:58.381234+0000","last_fullsized":"2024-09-19T07:48:58.381234+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:54.995631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25325,"num_write_kb":306661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"61'4832","reported_seq":4943,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373789+0000","last_change":"2024-09-19T07:19:32.289174+0000","last_active":"2024-09-19T07:48:58.373789+0000","last_peered":"2024-09-19T07:48:58.373789+0000","last_clean":"2024-09-19T07:48:58.373789+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:48:58.373789+0000","last_undegraded":"2024-09-19T07:48:58.373789+0000","last_fullsized":"2024-09-19T07:48:58.373789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:30:12.248177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26222,"num_write_kb":306099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"64'3448","reported_seq":3556,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372093+0000","last_change":"2024-09-19T07:19:40.396146+0000","last_active":"2024-09-19T07:48:58.372093+0000","last_peered":"2024-09-19T07:48:58.372093+0000","last_clean":"2024-09-19T07:48:58.372093+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:48:58.372093+0000","last_undegraded":"2024-09-19T07:48:58.372093+0000","last_fullsized":"2024-09-19T07:48:58.372093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3448,"log_dups_size":0,"ondisk_log_size":3448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18660,"num_write_kb":230569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"61'4317","reported_seq":4428,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365728+0000","last_change":"2024-09-19T07:19:32.285442+0000","last_active":"2024-09-19T07:48:58.365728+0000","last_peered":"2024-09-19T07:48:58.365728+0000","last_clean":"2024-09-19T07:48:58.365728+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:48:58.365728+0000","last_undegraded":"2024-09-19T07:48:58.365728+0000","last_fullsized":"2024-09-19T07:48:58.365728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:31:20.196501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22609,"num_write_kb":267014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"64'3912","reported_seq":4019,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367632+0000","last_change":"2024-09-19T07:19:40.373552+0000","last_active":"2024-09-19T07:48:58.367632+0000","last_peered":"2024-09-19T07:48:58.367632+0000","last_clean":"2024-09-19T07:48:58.367632+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:48:58.367632+0000","last_undegraded":"2024-09-19T07:48:58.367632+0000","last_fullsized":"2024-09-19T07:48:58.367632+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:23:33.629831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21554,"num_write_kb":270790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"61'4720","reported_seq":4831,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373254+0000","last_change":"2024-09-19T07:19:32.281593+0000","last_active":"2024-09-19T07:48:58.373254+0000","last_peered":"2024-09-19T07:48:58.373254+0000","last_clean":"2024-09-19T07:48:58.373254+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:48:58.373254+0000","last_undegraded":"2024-09-19T07:48:58.373254+0000","last_fullsized":"2024-09-19T07:48:58.373254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:53:22.873458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24228,"num_write_kb":289919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"64'3227","reported_seq":3330,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362718+0000","last_change":"2024-09-19T07:19:40.356989+0000","last_active":"2024-09-19T07:48:58.362718+0000","last_peered":"2024-09-19T07:48:58.362718+0000","last_clean":"2024-09-19T07:48:58.362718+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:48:58.362718+0000","last_undegraded":"2024-09-19T07:48:58.362718+0000","last_fullsized":"2024-09-19T07:48:58.362718+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17253,"num_write_kb":237102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"61'4326","reported_seq":4441,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366880+0000","last_change":"2024-09-19T07:19:32.285390+0000","last_active":"2024-09-19T07:48:58.366880+0000","last_peered":"2024-09-19T07:48:58.366880+0000","last_clean":"2024-09-19T07:48:58.366880+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:48:58.366880+0000","last_undegraded":"2024-09-19T07:48:58.366880+0000","last_fullsized":"2024-09-19T07:48:58.366880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:00.044513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23399,"num_write_kb":274225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"64'3524","reported_seq":3628,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.377595+0000","last_change":"2024-09-19T07:19:40.376487+0000","last_active":"2024-09-19T07:48:58.377595+0000","last_peered":"2024-09-19T07:48:58.377595+0000","last_clean":"2024-09-19T07:48:58.377595+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:48:58.377595+0000","last_undegraded":"2024-09-19T07:48:58.377595+0000","last_fullsized":"2024-09-19T07:48:58.377595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18216,"num_write_kb":261619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"61'4656","reported_seq":4772,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.369885+0000","last_change":"2024-09-19T07:19:32.271588+0000","last_active":"2024-09-19T07:48:58.369885+0000","last_peered":"2024-09-19T07:48:58.369885+0000","last_clean":"2024-09-19T07:48:58.369885+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:48:58.369885+0000","last_undegraded":"2024-09-19T07:48:58.369885+0000","last_fullsized":"2024-09-19T07:48:58.369885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24790,"num_write_kb":287946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"64'4503","reported_seq":4606,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372440+0000","last_change":"2024-09-19T07:19:40.377604+0000","last_active":"2024-09-19T07:48:58.372440+0000","last_peered":"2024-09-19T07:48:58.372440+0000","last_clean":"2024-09-19T07:48:58.372440+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:48:58.372440+0000","last_undegraded":"2024-09-19T07:48:58.372440+0000","last_fullsized":"2024-09-19T07:48:58.372440+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:13.607662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24213,"num_write_kb":310059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"61'5949","reported_seq":6064,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373696+0000","last_change":"2024-09-19T07:19:32.286185+0000","last_active":"2024-09-19T07:48:58.373696+0000","last_peered":"2024-09-19T07:48:58.373696+0000","last_clean":"2024-09-19T07:48:58.373696+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:48:58.373696+0000","last_undegraded":"2024-09-19T07:48:58.373696+0000","last_fullsized":"2024-09-19T07:48:58.373696+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5849,"log_dups_size":0,"ondisk_log_size":5849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31740,"num_write_kb":364971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"64'4360","reported_seq":4463,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372581+0000","last_change":"2024-09-19T07:19:40.377580+0000","last_active":"2024-09-19T07:48:58.372581+0000","last_peered":"2024-09-19T07:48:58.372581+0000","last_clean":"2024-09-19T07:48:58.372581+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:48:58.372581+0000","last_undegraded":"2024-09-19T07:48:58.372581+0000","last_fullsized":"2024-09-19T07:48:58.372581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4360,"log_dups_size":0,"ondisk_log_size":4360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:57:17.056882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23533,"num_write_kb":311499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"61'4830","reported_seq":4945,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375882+0000","last_change":"2024-09-19T07:19:32.287022+0000","last_active":"2024-09-19T07:48:58.375882+0000","last_peered":"2024-09-19T07:48:58.375882+0000","last_clean":"2024-09-19T07:48:58.375882+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:48:58.375882+0000","last_undegraded":"2024-09-19T07:48:58.375882+0000","last_fullsized":"2024-09-19T07:48:58.375882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:37:20.742510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26589,"num_write_kb":315697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1521393,"num_write_kb":19214666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279640,"ondisk_log_size":279640,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19627747,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1449,"num_read_kb":190539,"num_write":5111,"num_write_kb":21283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39313408,"data_stored":39255494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4609,"ondisk_log_size":4609,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":0,"num_write":1651322,"num_write_kb":19736802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307839,"ondisk_log_size":307839,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":19579163,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1454,"num_read_kb":221531,"num_write":5068,"num_write_kb":21268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39223296,"data_stored":39158326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4624,"ondisk_log_size":4624,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542508,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1114512,"kb_used_data":13696,"kb_used_omap":1,"kb_used_meta":1100798,"kb_avail":93257328,"statfs":{"total":96636764160,"available":95495503872,"internally_reserved":0,"allocated":14024704,"data_stored":13620504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1127217610},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542509,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766708,"kb_used_data":11684,"kb_used_omap":1,"kb_used_meta":755006,"kb_avail":93605132,"statfs":{"total":96636764160,"available":95851655168,"internally_reserved":0,"allocated":11964416,"data_stored":11552279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":773126598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542507,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1069796,"kb_used_data":5012,"kb_used_omap":1,"kb_used_meta":1064766,"kb_avail":93302044,"statfs":{"total":96636764160,"available":95541293056,"internally_reserved":0,"allocated":5132288,"data_stored":4726264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1090320837},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542509,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1229016,"kb_used_data":14472,"kb_used_omap":1,"kb_used_meta":1214526,"kb_avail":93142824,"statfs":{"total":96636764160,"available":95378251776,"internally_reserved":0,"allocated":14819328,"data_stored":14400282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1243675077},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542507,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1080696,"kb_used_data":20008,"kb_used_omap":1,"kb_used_meta":1060670,"kb_avail":93291144,"statfs":{"total":96636764160,"available":95530131456,"internally_reserved":0,"allocated":20488192,"data_stored":20074195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1086126535},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542509,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688028,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":687102,"kb_avail":93683812,"statfs":{"total":96636764160,"available":95932223488,"internally_reserved":0,"allocated":929792,"data_stored":531268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":703592902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542508,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":891928,"kb_used_data":9032,"kb_used_omap":1,"kb_used_meta":882878,"kb_avail":93479912,"statfs":{"total":96636764160,"available":95723429888,"internally_reserved":0,"allocated":9248768,"data_stored":8836014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904067525},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542507,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":907476,"kb_used_data":9604,"kb_used_omap":1,"kb_used_meta":897854,"kb_avail":93464364,"statfs":{"total":96636764160,"available":95707508736,"internally_reserved":0,"allocated":9834496,"data_stored":9426166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919402949},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":4194942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":84646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8482816,"data_stored":8473926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6926336,"data_stored":6910261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6922240,"data_stored":6910807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11165696,"data_stored":11153717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7053312,"data_stored":7043469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4279622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:49:23.363 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.363 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.364 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.364 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.364 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.364 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.364 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.364 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.364 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.364 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.365 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.365 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.365 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.365 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.365 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.365 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.365 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.365 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.365 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.366 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.366 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.366 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.366 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.366 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.366 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.366 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.366 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.366 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.367 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.367 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.367 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.367 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.367 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.367 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.367 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.367 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.368 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.368 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.368 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.368 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.368 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.368 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.368 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.368 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.368 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.369 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.369 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.369 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.369 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.369 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.369 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.369 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.369 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.370 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.370 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.370 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.370 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.370 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.370 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.370 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.370 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.370 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.371 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-19T07:19:21.529843+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=21, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.371 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.371 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.371 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.371 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.371 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.371 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.371 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.371 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.371 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.372 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.372 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.372 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.372 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.372 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.372 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.372 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.372 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.372 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.373 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.373 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.373 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.373 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.373 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.373 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.373 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.373 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.373 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.374 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.374 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.374 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.374 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.374 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.374 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.374 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.374 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.374 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.374 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.375 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.375 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.375 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.375 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.375 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.375 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.375 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.375 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.376 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.376 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.376 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.376 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.376 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.376 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.376 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.376 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.376 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.376 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.377 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.377 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.377 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.377 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.377 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.377 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.377 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.377 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.377 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-09-19T07:19:38.375757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=38, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.378 INFO:tasks.ceph:pgid 5.26 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.378 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.378 INFO:tasks.ceph:pgid 5.27 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.378 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.378 INFO:tasks.ceph:pgid 5.24 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.378 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.378 INFO:tasks.ceph:pgid 5.25 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.378 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.378 INFO:tasks.ceph:pgid 5.22 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.379 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.379 INFO:tasks.ceph:pgid 5.23 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.379 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.379 INFO:tasks.ceph:pgid 5.20 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.379 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.379 INFO:tasks.ceph:pgid 5.21 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.379 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.379 INFO:tasks.ceph:pgid 5.2e last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.379 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.380 INFO:tasks.ceph:pgid 5.2f last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.380 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.380 INFO:tasks.ceph:pgid 5.2c last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.380 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.380 INFO:tasks.ceph:pgid 5.2d last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.380 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.380 INFO:tasks.ceph:pgid 5.2a last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.380 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.380 INFO:tasks.ceph:pgid 5.2b last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.381 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.381 INFO:tasks.ceph:pgid 5.28 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.381 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.381 INFO:tasks.ceph:pgid 5.29 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.381 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.381 INFO:tasks.ceph:pgid 5.36 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.381 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.381 INFO:tasks.ceph:pgid 5.37 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.381 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.382 INFO:tasks.ceph:pgid 5.34 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.382 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.382 INFO:tasks.ceph:pgid 5.35 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.382 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.382 INFO:tasks.ceph:pgid 5.32 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.382 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.382 INFO:tasks.ceph:pgid 5.33 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.382 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.382 INFO:tasks.ceph:pgid 5.30 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.383 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.383 INFO:tasks.ceph:pgid 5.31 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.383 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.383 INFO:tasks.ceph:pgid 5.3e last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.383 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.383 INFO:tasks.ceph:pgid 5.3f last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.383 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.383 INFO:tasks.ceph:pgid 5.3c last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.383 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.384 INFO:tasks.ceph:pgid 5.3d last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.384 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.384 INFO:tasks.ceph:pgid 5.3a last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.384 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.384 INFO:tasks.ceph:pgid 5.3b last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.384 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.384 INFO:tasks.ceph:pgid 5.38 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.384 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.384 INFO:tasks.ceph:pgid 5.39 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.385 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:23.385 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-19T07:49:42.208 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:49:42.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:49:43.386 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:49:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.516+0000 7f9e42b5a640 1 Processor -- start 2024-09-19T07:49:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.516+0000 7f9e42b5a640 1 -- start start 2024-09-19T07:49:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e42b5a640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9e3c06e270 0x7f9e3c1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e42b5a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e3c1592c0 con 0x7f9e3c0abb30 2024-09-19T07:49:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e42b5a640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e3c06e900 con 0x7f9e3c0c42e0 2024-09-19T07:49:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e42b5a640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f9e3c06e6e0 con 0x7f9e3c06e270 2024-09-19T07:49:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e410f7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9e3c0abb30 0x7f9e3c153060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34044/0 (socket says 172.21.15.83:34044) 2024-09-19T07:49:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e410f7640 1 -- 172.21.15.83:0/3754539482 learned_addr learned my addr 172.21.15.83:0/3754539482 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3bfff640 1 --2- 172.21.15.83:0/3754539482 >> v2:172.21.15.184:3300/0 conn(0x7f9e3c06e270 0x7f9e3c1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2604422599 0 0) 0x7f9e3c1592c0 con 0x7f9e3c0abb30 2024-09-19T07:49:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e24003530 con 0x7f9e3c0abb30 2024-09-19T07:49:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 728504603 0 0) 0x7f9e24003530 con 0x7f9e3c0abb30 2024-09-19T07:49:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 >> v2:172.21.15.184:3300/0 conn(0x7f9e3c06e270 msgr2=0x7f9e3c1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 --2- 172.21.15.83:0/3754539482 >> v2:172.21.15.184:3300/0 conn(0x7f9e3c06e270 0x7f9e3c1566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 >> v1:172.21.15.184:6789/0 conn(0x7f9e3c0c42e0 legacy=0x7f9e3c159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3c06ebe0 con 0x7f9e3c0abb30 2024-09-19T07:49:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f9e340016a0 con 0x7f9e3c0abb30 2024-09-19T07:49:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 >> v1:172.21.15.83:6789/0 conn(0x7f9e3c0abb30 legacy=0x7f9e3c153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 --2- 172.21.15.83:0/3754539482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e240039c0 0x7f9e24003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 --2- 172.21.15.83:0/3754539482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e24004810 0x7f9e24004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 --2- 172.21.15.83:0/3754539482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 0x7f9e2400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3c06e6e0 con 0x7f9e240051d0 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2400c070 con 0x7f9e240039c0 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9e2400c270 con 0x7f9e24004810 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e410f7640 1 --2- 172.21.15.83:0/3754539482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 0x7f9e2400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3bfff640 1 --2- 172.21.15.83:0/3754539482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e24004810 0x7f9e24004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e408f6640 1 --2- 172.21.15.83:0/3754539482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e240039c0 0x7f9e24003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e410f7640 1 -- 172.21.15.83:0/3754539482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e24004810 msgr2=0x7f9e24004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e410f7640 1 --2- 172.21.15.83:0/3754539482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e24004810 0x7f9e24004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e410f7640 1 -- 172.21.15.83:0/3754539482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e240039c0 msgr2=0x7f9e24003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e410f7640 1 --2- 172.21.15.83:0/3754539482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e240039c0 0x7f9e24003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e410f7640 1 -- 172.21.15.83:0/3754539482 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e24003530 con 0x7f9e240051d0 2024-09-19T07:49:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.520+0000 7f9e3bfff640 1 --2- 172.21.15.83:0/3754539482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e24004810 0x7f9e24004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e410f7640 1 -- 172.21.15.83:0/3754539482 reap_dead start 2024-09-19T07:49:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e410f7640 1 --2- 172.21.15.83:0/3754539482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 0x7f9e2400b9d0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400c210 tx=0x7f9e3401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0aa2bc405af776 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e34007d20 con 0x7f9e240051d0 2024-09-19T07:49:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e340a9070 con 0x7f9e240051d0 2024-09-19T07:49:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e3b7fe640 1 -- 172.21.15.83:0/3754539482 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e340a5d80 con 0x7f9e240051d0 2024-09-19T07:49:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 -- 172.21.15.83:0/3754539482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 msgr2=0x7f9e2400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 --2- 172.21.15.83:0/3754539482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 0x7f9e2400b9d0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400c210 tx=0x7f9e3401cc40 comp rx=0 tx=0).stop 2024-09-19T07:49:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 -- 172.21.15.83:0/3754539482 shutdown_connections 2024-09-19T07:49:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 --2- 172.21.15.83:0/3754539482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 0x7f9e2400b9d0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 -- 172.21.15.83:0/3754539482 >> 172.21.15.83:0/3754539482 conn(0x7f9e3c06bc80 msgr2=0x7f9e3c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 -- 172.21.15.83:0/3754539482 shutdown_connections 2024-09-19T07:49:43.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 -- 172.21.15.83:0/3754539482 wait complete. 2024-09-19T07:49:43.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 Processor -- start 2024-09-19T07:49:43.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 -- start start 2024-09-19T07:49:43.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 0x7f9e3c1e0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:43.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e3c146bc0 0x7f9e3c147040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:43.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e3c1e0c70 0x7f9e3c1e10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:43.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3c0abf60 con 0x7f9e240051d0 2024-09-19T07:49:43.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3c1595f0 con 0x7f9e3c146bc0 2024-09-19T07:49:43.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.524+0000 7f9e42b5a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9e3c0c46c0 con 0x7f9e3c1e0c70 2024-09-19T07:49:43.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e408f6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 0x7f9e3c1e0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:43.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e408f6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 0x7f9e3c1e0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50942/0 (socket says 172.21.15.83:50942) 2024-09-19T07:49:43.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e408f6640 1 -- 172.21.15.83:0/4130296725 learned_addr learned my addr 172.21.15.83:0/4130296725 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:43.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e3bfff640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e3c146bc0 0x7f9e3c147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e408f6640 1 -- 172.21.15.83:0/4130296725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e3c1e0c70 msgr2=0x7f9e3c1e10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e408f6640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e3c1e0c70 0x7f9e3c1e10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e408f6640 1 -- 172.21.15.83:0/4130296725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e3c146bc0 msgr2=0x7f9e3c147040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e408f6640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e3c146bc0 0x7f9e3c147040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e408f6640 1 -- 172.21.15.83:0/4130296725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3c1e1d90 con 0x7f9e240051d0 2024-09-19T07:49:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e408f6640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 0x7f9e3c1e0020 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c00d8e0 tx=0x7f9e2c00dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e397fa640 1 -- 172.21.15.83:0/4130296725 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c003db0 con 0x7f9e240051d0 2024-09-19T07:49:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e397fa640 1 -- 172.21.15.83:0/4130296725 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e2c017070 con 0x7f9e240051d0 2024-09-19T07:49:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e42b5a640 1 -- 172.21.15.83:0/4130296725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e3c154b40 con 0x7f9e240051d0 2024-09-19T07:49:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e397fa640 1 -- 172.21.15.83:0/4130296725 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c0052b0 con 0x7f9e240051d0 2024-09-19T07:49:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e42b5a640 1 -- 172.21.15.83:0/4130296725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e3c1df420 con 0x7f9e240051d0 2024-09-19T07:49:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e42b5a640 1 -- 172.21.15.83:0/4130296725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e3c1566b0 con 0x7f9e240051d0 2024-09-19T07:49:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e397fa640 1 -- 172.21.15.83:0/4130296725 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9e2c005450 con 0x7f9e240051d0 2024-09-19T07:49:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e397fa640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e280678f0 0x7f9e28069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e397fa640 1 -- 172.21.15.83:0/4130296725 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f9e2c086fe0 con 0x7f9e240051d0 2024-09-19T07:49:43.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e3bfff640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e280678f0 0x7f9e28069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:43.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.528+0000 7f9e3bfff640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e280678f0 0x7f9e28069db0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c0aa980 tx=0x7f9e30008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:43.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.532+0000 7f9e397fa640 1 -- 172.21.15.83:0/4130296725 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9e2c056a10 con 0x7f9e240051d0 2024-09-19T07:49:43.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.688+0000 7f9e42b5a640 1 -- 172.21.15.83:0/4130296725 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9e3c06e050 con 0x7f9e280678f0 2024-09-19T07:49:43.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.724+0000 7f9e397fa640 1 -- 172.21.15.83:0/4130296725 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493418 (secure 0 0 0) 0x7f9e3c06e050 con 0x7f9e280678f0 2024-09-19T07:49:43.729 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:43.733 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:49:43.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.732+0000 7f9e42b5a640 1 -- 172.21.15.83:0/4130296725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e280678f0 msgr2=0x7f9e28069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:43.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.732+0000 7f9e42b5a640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e280678f0 0x7f9e28069db0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c0aa980 tx=0x7f9e30008040 comp rx=0 tx=0).stop 2024-09-19T07:49:43.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.732+0000 7f9e42b5a640 1 -- 172.21.15.83:0/4130296725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 msgr2=0x7f9e3c1e0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:43.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.732+0000 7f9e42b5a640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 0x7f9e3c1e0020 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c00d8e0 tx=0x7f9e2c00dda0 comp rx=0 tx=0).stop 2024-09-19T07:49:43.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.732+0000 7f9e42b5a640 1 -- 172.21.15.83:0/4130296725 shutdown_connections 2024-09-19T07:49:43.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.732+0000 7f9e42b5a640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f9e280678f0 0x7f9e28069db0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:43.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.732+0000 7f9e42b5a640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e3c1e0c70 0x7f9e3c1e10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:43.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.732+0000 7f9e42b5a640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e3c146bc0 0x7f9e3c147040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:43.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.732+0000 7f9e42b5a640 1 --2- 172.21.15.83:0/4130296725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e240051d0 0x7f9e3c1e0020 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:43.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.732+0000 7f9e42b5a640 1 -- 172.21.15.83:0/4130296725 >> 172.21.15.83:0/4130296725 conn(0x7f9e3c06bc80 msgr2=0x7f9e3c153040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:43.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.732+0000 7f9e42b5a640 1 -- 172.21.15.83:0/4130296725 shutdown_connections 2024-09-19T07:49:43.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:43.732+0000 7f9e42b5a640 1 -- 172.21.15.83:0/4130296725 wait complete. 2024-09-19T07:49:43.748 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":969,"stamp":"2024-09-19T07:49:43.625383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39797278,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3062,"num_read_kb":412170,"num_write":3183053,"num_write_kb":38996435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":596875,"ondisk_log_size":596875,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7748160,"kb_used_data":84416,"kb_used_omap":12,"kb_used_meta":7663603,"kb_avail":747226560,"statfs":{"total":773094113280,"available":765159997440,"internally_reserved":0,"allocated":86441984,"data_stored":83166972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":7847530033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006464"},"pg_stats":[{"pgid":"5.18","version":"64'4995","reported_seq":5108,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.421051+0000","last_change":"2024-09-19T07:49:37.421108+0000","last_active":"2024-09-19T07:49:37.421051+0000","last_peered":"2024-09-19T07:49:37.421051+0000","last_clean":"2024-09-19T07:49:37.421051+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:49:37.421051+0000","last_undegraded":"2024-09-19T07:49:37.421051+0000","last_fullsized":"2024-09-19T07:49:37.421051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4995","last_scrub_stamp":"2024-09-19T07:49:37.420845+0000","last_deep_scrub":"64'4995","last_deep_scrub_stamp":"2024-09-19T07:49:37.420845+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.420845+0000","objects_scrubbed":0,"log_size":4995,"log_dups_size":0,"ondisk_log_size":4995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:02:56.503206+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27070,"num_write_kb":342022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"65'956","reported_seq":1130,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.647691+0000","last_change":"2024-09-19T07:49:26.647769+0000","last_active":"2024-09-19T07:49:26.647691+0000","last_peered":"2024-09-19T07:49:26.647691+0000","last_clean":"2024-09-19T07:49:26.647691+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:49:26.647691+0000","last_undegraded":"2024-09-19T07:49:26.647691+0000","last_fullsized":"2024-09-19T07:49:26.647691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'956","last_scrub_stamp":"2024-09-19T07:49:26.647475+0000","last_deep_scrub":"65'956","last_deep_scrub_stamp":"2024-09-19T07:49:26.647475+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.647475+0000","objects_scrubbed":3,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:10:02.782343+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":18964,"num_write":931,"num_write_kb":4658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"61'4894","reported_seq":5017,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.462628+0000","last_change":"2024-09-19T07:49:33.462691+0000","last_active":"2024-09-19T07:49:33.462628+0000","last_peered":"2024-09-19T07:49:33.462628+0000","last_clean":"2024-09-19T07:49:33.462628+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:49:33.462628+0000","last_undegraded":"2024-09-19T07:49:33.462628+0000","last_fullsized":"2024-09-19T07:49:33.462628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4894","last_scrub_stamp":"2024-09-19T07:49:33.462422+0000","last_deep_scrub":"61'4894","last_deep_scrub_stamp":"2024-09-19T07:49:33.462422+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.462422+0000","objects_scrubbed":0,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:31.966862+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26245,"num_write_kb":306037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"64'3","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.613843+0000","last_change":"2024-09-19T07:49:23.613909+0000","last_active":"2024-09-19T07:49:23.613843+0000","last_peered":"2024-09-19T07:49:23.613843+0000","last_clean":"2024-09-19T07:49:23.613843+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:49:23.613843+0000","last_undegraded":"2024-09-19T07:49:23.613843+0000","last_fullsized":"2024-09-19T07:49:23.613843+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-09-19T07:49:23.613617+0000","last_deep_scrub":"64'3","last_deep_scrub_stamp":"2024-09-19T07:49:23.613617+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.613617+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:24:32.580406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"64'5242","reported_seq":5355,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.333595+0000","last_change":"2024-09-19T07:49:29.333657+0000","last_active":"2024-09-19T07:49:29.333595+0000","last_peered":"2024-09-19T07:49:29.333595+0000","last_clean":"2024-09-19T07:49:29.333595+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:49:29.333595+0000","last_undegraded":"2024-09-19T07:49:29.333595+0000","last_fullsized":"2024-09-19T07:49:29.333595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5242","last_scrub_stamp":"2024-09-19T07:49:29.333381+0000","last_deep_scrub":"64'5242","last_deep_scrub_stamp":"2024-09-19T07:49:29.333381+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.333381+0000","objects_scrubbed":0,"log_size":5242,"log_dups_size":0,"ondisk_log_size":5242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:16.081844+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27567,"num_write_kb":356452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"65'25","reported_seq":151,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.501749+0000","last_change":"2024-09-19T07:49:36.501749+0000","last_active":"2024-09-19T07:49:36.501749+0000","last_peered":"2024-09-19T07:49:36.501749+0000","last_clean":"2024-09-19T07:49:36.501749+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:49:36.501749+0000","last_undegraded":"2024-09-19T07:49:36.501749+0000","last_fullsized":"2024-09-19T07:49:36.501749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'25","last_scrub_stamp":"2024-09-19T07:49:36.501391+0000","last_deep_scrub":"65'25","last_deep_scrub_stamp":"2024-09-19T07:49:36.501391+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.501391+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:47.830192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"61'5997","reported_seq":6120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.631536+0000","last_change":"2024-09-19T07:49:30.631597+0000","last_active":"2024-09-19T07:49:30.631536+0000","last_peered":"2024-09-19T07:49:30.631536+0000","last_clean":"2024-09-19T07:49:30.631536+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:49:30.631536+0000","last_undegraded":"2024-09-19T07:49:30.631536+0000","last_fullsized":"2024-09-19T07:49:30.631536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5997","last_scrub_stamp":"2024-09-19T07:49:30.631337+0000","last_deep_scrub":"61'5997","last_deep_scrub_stamp":"2024-09-19T07:49:30.631337+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.631337+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:11:11.393948+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32297,"num_write_kb":399913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"64'2","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.333780+0000","last_change":"2024-09-19T07:49:35.333920+0000","last_active":"2024-09-19T07:49:35.333780+0000","last_peered":"2024-09-19T07:49:35.333780+0000","last_clean":"2024-09-19T07:49:35.333780+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:49:35.333780+0000","last_undegraded":"2024-09-19T07:49:35.333780+0000","last_fullsized":"2024-09-19T07:49:35.333780+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-19T07:49:35.333511+0000","last_deep_scrub":"64'2","last_deep_scrub_stamp":"2024-09-19T07:49:35.333511+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.333511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:55.298008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"64'3801","reported_seq":3907,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.020250+0000","last_change":"2024-09-19T07:19:40.390455+0000","last_active":"2024-09-19T07:49:23.020250+0000","last_peered":"2024-09-19T07:49:23.020250+0000","last_clean":"2024-09-19T07:49:23.020250+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:49:23.020250+0000","last_undegraded":"2024-09-19T07:49:23.020250+0000","last_fullsized":"2024-09-19T07:49:23.020250+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20272,"num_write_kb":245240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"65'45","reported_seq":208,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.341882+0000","last_change":"2024-09-19T07:49:39.342050+0000","last_active":"2024-09-19T07:49:39.341882+0000","last_peered":"2024-09-19T07:49:39.341882+0000","last_clean":"2024-09-19T07:49:39.341882+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:49:39.341882+0000","last_undegraded":"2024-09-19T07:49:39.341882+0000","last_fullsized":"2024-09-19T07:49:39.341882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'45","last_scrub_stamp":"2024-09-19T07:49:39.341627+0000","last_deep_scrub":"65'45","last_deep_scrub_stamp":"2024-09-19T07:49:39.341627+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.341627+0000","objects_scrubbed":1,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:50.670429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"61'4645","reported_seq":4768,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.558336+0000","last_change":"2024-09-19T07:49:22.558336+0000","last_active":"2024-09-19T07:49:22.558336+0000","last_peered":"2024-09-19T07:49:22.558336+0000","last_clean":"2024-09-19T07:49:22.558336+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:49:22.558336+0000","last_undegraded":"2024-09-19T07:49:22.558336+0000","last_fullsized":"2024-09-19T07:49:22.558336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4645","last_scrub_stamp":"2024-09-19T07:49:22.557969+0000","last_deep_scrub":"61'4645","last_deep_scrub_stamp":"2024-09-19T07:49:22.557969+0000","last_clean_scrub_stamp":"2024-09-19T07:49:22.557969+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:19:09.650471+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24199,"num_write_kb":281402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'765","reported_seq":906,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.307628+0000","last_change":"2024-09-19T07:49:28.307628+0000","last_active":"2024-09-19T07:49:28.307628+0000","last_peered":"2024-09-19T07:49:28.307628+0000","last_clean":"2024-09-19T07:49:28.307628+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:49:28.307628+0000","last_undegraded":"2024-09-19T07:49:28.307628+0000","last_fullsized":"2024-09-19T07:49:28.307628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'765","last_scrub_stamp":"2024-09-19T07:49:28.307365+0000","last_deep_scrub":"65'765","last_deep_scrub_stamp":"2024-09-19T07:49:28.307365+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.307365+0000","objects_scrubbed":1,"log_size":765,"log_dups_size":0,"ondisk_log_size":765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:38:18.776897+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":71635,"num_write":751,"num_write_kb":4331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"64'3560","reported_seq":3665,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.057004+0000","last_change":"2024-09-19T07:19:40.372543+0000","last_active":"2024-09-19T07:49:22.057004+0000","last_peered":"2024-09-19T07:49:22.057004+0000","last_clean":"2024-09-19T07:49:22.057004+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:49:22.057004+0000","last_undegraded":"2024-09-19T07:49:22.057004+0000","last_fullsized":"2024-09-19T07:49:22.057004+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3560,"log_dups_size":0,"ondisk_log_size":3560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18641,"num_write_kb":242587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"61'5","reported_seq":160,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.019809+0000","last_change":"2024-09-19T07:19:31.291844+0000","last_active":"2024-09-19T07:49:23.019809+0000","last_peered":"2024-09-19T07:49:23.019809+0000","last_clean":"2024-09-19T07:49:23.019809+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:49:23.019809+0000","last_undegraded":"2024-09-19T07:49:23.019809+0000","last_fullsized":"2024-09-19T07:49:23.019809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:30.217638+0000","last_clean_scrub_stamp":"2024-09-19T07:19:30.217638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"61'5989","reported_seq":6110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.572806+0000","last_change":"2024-09-19T07:49:23.572806+0000","last_active":"2024-09-19T07:49:23.572806+0000","last_peered":"2024-09-19T07:49:23.572806+0000","last_clean":"2024-09-19T07:49:23.572806+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:49:23.572806+0000","last_undegraded":"2024-09-19T07:49:23.572806+0000","last_fullsized":"2024-09-19T07:49:23.572806+0000","mapping_epoch":19,"log_start":"53'200","ondisk_log_start":"53'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5989","last_scrub_stamp":"2024-09-19T07:49:23.572533+0000","last_deep_scrub":"61'5989","last_deep_scrub_stamp":"2024-09-19T07:49:23.572533+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.572533+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:39.369315+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32376,"num_write_kb":375806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'385","reported_seq":502,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.371994+0000","last_change":"2024-09-19T07:49:36.371994+0000","last_active":"2024-09-19T07:49:36.371994+0000","last_peered":"2024-09-19T07:49:36.371994+0000","last_clean":"2024-09-19T07:49:36.371994+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:49:36.371994+0000","last_undegraded":"2024-09-19T07:49:36.371994+0000","last_fullsized":"2024-09-19T07:49:36.371994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'385","last_scrub_stamp":"2024-09-19T07:49:36.371687+0000","last_deep_scrub":"65'385","last_deep_scrub_stamp":"2024-09-19T07:49:36.371687+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.371687+0000","objects_scrubbed":1,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:50:43.795428+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2764415,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":637,"num_write":359,"num_write_kb":2908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"64'4768","reported_seq":4874,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372445+0000","last_change":"2024-09-19T07:19:40.396520+0000","last_active":"2024-09-19T07:48:58.372445+0000","last_peered":"2024-09-19T07:48:58.372445+0000","last_clean":"2024-09-19T07:48:58.372445+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:48:58.372445+0000","last_undegraded":"2024-09-19T07:48:58.372445+0000","last_fullsized":"2024-09-19T07:48:58.372445+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25484,"num_write_kb":302986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"61'5596","reported_seq":5718,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.478487+0000","last_change":"2024-09-19T07:49:24.478547+0000","last_active":"2024-09-19T07:49:24.478487+0000","last_peered":"2024-09-19T07:49:24.478487+0000","last_clean":"2024-09-19T07:49:24.478487+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:49:24.478487+0000","last_undegraded":"2024-09-19T07:49:24.478487+0000","last_fullsized":"2024-09-19T07:49:24.478487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5596","last_scrub_stamp":"2024-09-19T07:49:24.478287+0000","last_deep_scrub":"61'5596","last_deep_scrub_stamp":"2024-09-19T07:49:24.478287+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.478287+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:10:51.268354+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30081,"num_write_kb":342050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'1458","reported_seq":1626,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.320165+0000","last_change":"2024-09-19T07:49:36.320239+0000","last_active":"2024-09-19T07:49:36.320165+0000","last_peered":"2024-09-19T07:49:36.320165+0000","last_clean":"2024-09-19T07:49:36.320165+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:49:36.320165+0000","last_undegraded":"2024-09-19T07:49:36.320165+0000","last_fullsized":"2024-09-19T07:49:36.320165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1458","last_scrub_stamp":"2024-09-19T07:49:36.319951+0000","last_deep_scrub":"65'1458","last_deep_scrub_stamp":"2024-09-19T07:49:36.319951+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.319951+0000","objects_scrubbed":2,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:34.824390+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":42401,"num_write":1433,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'7","reported_seq":143,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.269934+0000","last_change":"2024-09-19T07:49:26.270002+0000","last_active":"2024-09-19T07:49:26.269934+0000","last_peered":"2024-09-19T07:49:26.269934+0000","last_clean":"2024-09-19T07:49:26.269934+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:49:26.269934+0000","last_undegraded":"2024-09-19T07:49:26.269934+0000","last_fullsized":"2024-09-19T07:49:26.269934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'7","last_scrub_stamp":"2024-09-19T07:49:26.269740+0000","last_deep_scrub":"65'7","last_deep_scrub_stamp":"2024-09-19T07:49:26.269740+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.269740+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:31.988649+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"64'4389","reported_seq":4503,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.653867+0000","last_change":"2024-09-19T07:49:32.653925+0000","last_active":"2024-09-19T07:49:32.653867+0000","last_peered":"2024-09-19T07:49:32.653867+0000","last_clean":"2024-09-19T07:49:32.653867+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:49:32.653867+0000","last_undegraded":"2024-09-19T07:49:32.653867+0000","last_fullsized":"2024-09-19T07:49:32.653867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4389","last_scrub_stamp":"2024-09-19T07:49:32.653662+0000","last_deep_scrub":"64'4389","last_deep_scrub_stamp":"2024-09-19T07:49:32.653662+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.653662+0000","objects_scrubbed":0,"log_size":4389,"log_dups_size":0,"ondisk_log_size":4389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:38:16.691610+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23791,"num_write_kb":304343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"61'3782","reported_seq":3896,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364343+0000","last_change":"2024-09-19T07:19:32.279593+0000","last_active":"2024-09-19T07:48:58.364343+0000","last_peered":"2024-09-19T07:48:58.364343+0000","last_clean":"2024-09-19T07:48:58.364343+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:48:58.364343+0000","last_undegraded":"2024-09-19T07:48:58.364343+0000","last_fullsized":"2024-09-19T07:48:58.364343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19997,"num_write_kb":252226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"61'3","reported_seq":127,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.213798+0000","last_change":"2024-09-19T07:49:32.213860+0000","last_active":"2024-09-19T07:49:32.213798+0000","last_peered":"2024-09-19T07:49:32.213798+0000","last_clean":"2024-09-19T07:49:32.213798+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:49:32.213798+0000","last_undegraded":"2024-09-19T07:49:32.213798+0000","last_fullsized":"2024-09-19T07:49:32.213798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-19T07:49:32.213558+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-19T07:49:32.213558+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.213558+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:48.490205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"64'5","reported_seq":121,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.302563+0000","last_change":"2024-09-19T07:49:24.302627+0000","last_active":"2024-09-19T07:49:24.302563+0000","last_peered":"2024-09-19T07:49:24.302563+0000","last_clean":"2024-09-19T07:49:24.302563+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:49:24.302563+0000","last_undegraded":"2024-09-19T07:49:24.302563+0000","last_fullsized":"2024-09-19T07:49:24.302563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5","last_scrub_stamp":"2024-09-19T07:49:24.302364+0000","last_deep_scrub":"64'5","last_deep_scrub_stamp":"2024-09-19T07:49:24.302364+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.302364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:09:24.709205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"64'3306","reported_seq":3420,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.479880+0000","last_change":"2024-09-19T07:49:39.479952+0000","last_active":"2024-09-19T07:49:39.479880+0000","last_peered":"2024-09-19T07:49:39.479880+0000","last_clean":"2024-09-19T07:49:39.479880+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:49:39.479880+0000","last_undegraded":"2024-09-19T07:49:39.479880+0000","last_fullsized":"2024-09-19T07:49:39.479880+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3306","last_scrub_stamp":"2024-09-19T07:49:39.479617+0000","last_deep_scrub":"64'3306","last_deep_scrub_stamp":"2024-09-19T07:49:39.479617+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.479617+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:23:47.514505+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17836,"num_write_kb":239496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"61'4850","reported_seq":4972,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.460572+0000","last_change":"2024-09-19T07:49:39.460644+0000","last_active":"2024-09-19T07:49:39.460572+0000","last_peered":"2024-09-19T07:49:39.460572+0000","last_clean":"2024-09-19T07:49:39.460572+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:49:39.460572+0000","last_undegraded":"2024-09-19T07:49:39.460572+0000","last_fullsized":"2024-09-19T07:49:39.460572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4850","last_scrub_stamp":"2024-09-19T07:49:39.460360+0000","last_deep_scrub":"61'4850","last_deep_scrub_stamp":"2024-09-19T07:49:39.460360+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.460360+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:35:55.967049+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26728,"num_write_kb":308134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"61'2","reported_seq":126,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.209461+0000","last_change":"2024-09-19T07:49:35.209522+0000","last_active":"2024-09-19T07:49:35.209461+0000","last_peered":"2024-09-19T07:49:35.209461+0000","last_clean":"2024-09-19T07:49:35.209461+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:49:35.209461+0000","last_undegraded":"2024-09-19T07:49:35.209461+0000","last_fullsized":"2024-09-19T07:49:35.209461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-19T07:49:35.209266+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-19T07:49:35.209266+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.209266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:12:48.875188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'5","reported_seq":141,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:21.158021+0000","last_change":"2024-09-19T07:49:21.158156+0000","last_active":"2024-09-19T07:49:21.158021+0000","last_peered":"2024-09-19T07:49:21.158021+0000","last_clean":"2024-09-19T07:49:21.158021+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:49:21.158021+0000","last_undegraded":"2024-09-19T07:49:21.158021+0000","last_fullsized":"2024-09-19T07:49:21.158021+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-19T07:49:21.157739+0000","last_deep_scrub":"65'5","last_deep_scrub_stamp":"2024-09-19T07:49:21.157739+0000","last_clean_scrub_stamp":"2024-09-19T07:49:21.157739+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:09:21.564579+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"64'5157","reported_seq":5263,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:02.920332+0000","last_change":"2024-09-19T07:19:40.395566+0000","last_active":"2024-09-19T07:49:02.920332+0000","last_peered":"2024-09-19T07:49:02.920332+0000","last_clean":"2024-09-19T07:49:02.920332+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:49:02.920332+0000","last_undegraded":"2024-09-19T07:49:02.920332+0000","last_fullsized":"2024-09-19T07:49:02.920332+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27729,"num_write_kb":365262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"61'4508","reported_seq":4630,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.388392+0000","last_change":"2024-09-19T07:49:32.388460+0000","last_active":"2024-09-19T07:49:32.388392+0000","last_peered":"2024-09-19T07:49:32.388392+0000","last_clean":"2024-09-19T07:49:32.388392+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:49:32.388392+0000","last_undegraded":"2024-09-19T07:49:32.388392+0000","last_fullsized":"2024-09-19T07:49:32.388392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4508","last_scrub_stamp":"2024-09-19T07:49:32.388033+0000","last_deep_scrub":"61'4508","last_deep_scrub_stamp":"2024-09-19T07:49:32.388033+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.388033+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:19.136496+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24068,"num_write_kb":289158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"61'5","reported_seq":129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.322376+0000","last_change":"2024-09-19T07:49:38.322517+0000","last_active":"2024-09-19T07:49:38.322376+0000","last_peered":"2024-09-19T07:49:38.322376+0000","last_clean":"2024-09-19T07:49:38.322376+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:49:38.322376+0000","last_undegraded":"2024-09-19T07:49:38.322376+0000","last_fullsized":"2024-09-19T07:49:38.322376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5","last_scrub_stamp":"2024-09-19T07:49:38.322115+0000","last_deep_scrub":"61'5","last_deep_scrub_stamp":"2024-09-19T07:49:38.322115+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.322115+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:57:57.049656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'8","reported_seq":164,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.317551+0000","last_change":"2024-09-19T07:49:25.317688+0000","last_active":"2024-09-19T07:49:25.317551+0000","last_peered":"2024-09-19T07:49:25.317551+0000","last_clean":"2024-09-19T07:49:25.317551+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:49:25.317551+0000","last_undegraded":"2024-09-19T07:49:25.317551+0000","last_fullsized":"2024-09-19T07:49:25.317551+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-19T07:49:25.317293+0000","last_deep_scrub":"65'8","last_deep_scrub_stamp":"2024-09-19T07:49:25.317293+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.317293+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:11:31.559475+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"64'3458","reported_seq":3565,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362082+0000","last_change":"2024-09-19T07:19:40.388972+0000","last_active":"2024-09-19T07:48:58.362082+0000","last_peered":"2024-09-19T07:48:58.362082+0000","last_clean":"2024-09-19T07:48:58.362082+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:48:58.362082+0000","last_undegraded":"2024-09-19T07:48:58.362082+0000","last_fullsized":"2024-09-19T07:48:58.362082+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3458,"log_dups_size":0,"ondisk_log_size":3458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18862,"num_write_kb":239540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"61'5647","reported_seq":5770,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.588566+0000","last_change":"2024-09-19T07:49:24.588680+0000","last_active":"2024-09-19T07:49:24.588566+0000","last_peered":"2024-09-19T07:49:24.588566+0000","last_clean":"2024-09-19T07:49:24.588566+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:49:24.588566+0000","last_undegraded":"2024-09-19T07:49:24.588566+0000","last_fullsized":"2024-09-19T07:49:24.588566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5647","last_scrub_stamp":"2024-09-19T07:49:24.588262+0000","last_deep_scrub":"61'5647","last_deep_scrub_stamp":"2024-09-19T07:49:24.588262+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.588262+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:37:02.235369+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29735,"num_write_kb":364939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"65'10","reported_seq":174,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.521410+0000","last_change":"2024-09-19T07:49:30.521476+0000","last_active":"2024-09-19T07:49:30.521410+0000","last_peered":"2024-09-19T07:49:30.521410+0000","last_clean":"2024-09-19T07:49:30.521410+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:49:30.521410+0000","last_undegraded":"2024-09-19T07:49:30.521410+0000","last_fullsized":"2024-09-19T07:49:30.521410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-19T07:49:30.521211+0000","last_deep_scrub":"65'10","last_deep_scrub_stamp":"2024-09-19T07:49:30.521211+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.521211+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:33:32.235334+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'45","reported_seq":222,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.324266+0000","last_change":"2024-09-19T07:49:22.324332+0000","last_active":"2024-09-19T07:49:22.324266+0000","last_peered":"2024-09-19T07:49:22.324266+0000","last_clean":"2024-09-19T07:49:22.324266+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:49:22.324266+0000","last_undegraded":"2024-09-19T07:49:22.324266+0000","last_fullsized":"2024-09-19T07:49:22.324266+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'45","last_scrub_stamp":"2024-09-19T07:49:22.324054+0000","last_deep_scrub":"65'45","last_deep_scrub_stamp":"2024-09-19T07:49:22.324054+0000","last_clean_scrub_stamp":"2024-09-19T07:49:22.324054+0000","objects_scrubbed":1,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:26.914491+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":108,"num_write":41,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"64'4007","reported_seq":4114,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363692+0000","last_change":"2024-09-19T07:19:40.390217+0000","last_active":"2024-09-19T07:48:58.363692+0000","last_peered":"2024-09-19T07:48:58.363692+0000","last_clean":"2024-09-19T07:48:58.363692+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:48:58.363692+0000","last_undegraded":"2024-09-19T07:48:58.363692+0000","last_fullsized":"2024-09-19T07:48:58.363692+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21606,"num_write_kb":266928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"61'4152","reported_seq":4266,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364963+0000","last_change":"2024-09-19T07:19:32.283027+0000","last_active":"2024-09-19T07:48:58.364963+0000","last_peered":"2024-09-19T07:48:58.364963+0000","last_clean":"2024-09-19T07:48:58.364963+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:48:58.364963+0000","last_undegraded":"2024-09-19T07:48:58.364963+0000","last_fullsized":"2024-09-19T07:48:58.364963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22349,"num_write_kb":269915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"65'475","reported_seq":619,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.204002+0000","last_change":"2024-09-19T07:49:34.204070+0000","last_active":"2024-09-19T07:49:34.204002+0000","last_peered":"2024-09-19T07:49:34.204002+0000","last_clean":"2024-09-19T07:49:34.204002+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:49:34.204002+0000","last_undegraded":"2024-09-19T07:49:34.204002+0000","last_fullsized":"2024-09-19T07:49:34.204002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'475","last_scrub_stamp":"2024-09-19T07:49:34.203776+0000","last_deep_scrub":"65'475","last_deep_scrub_stamp":"2024-09-19T07:49:34.203776+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.203776+0000","objects_scrubbed":1,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:50:41.627517+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":59,"num_write":1141,"num_write_kb":380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"64'10","reported_seq":127,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.434984+0000","last_change":"2024-09-19T07:49:30.435045+0000","last_active":"2024-09-19T07:49:30.434984+0000","last_peered":"2024-09-19T07:49:30.434984+0000","last_clean":"2024-09-19T07:49:30.434984+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:49:30.434984+0000","last_undegraded":"2024-09-19T07:49:30.434984+0000","last_fullsized":"2024-09-19T07:49:30.434984+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'10","last_scrub_stamp":"2024-09-19T07:49:30.434776+0000","last_deep_scrub":"64'10","last_deep_scrub_stamp":"2024-09-19T07:49:30.434776+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.434776+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:34:54.431736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"64'4199","reported_seq":4314,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.499655+0000","last_change":"2024-09-19T07:49:39.499782+0000","last_active":"2024-09-19T07:49:39.499655+0000","last_peered":"2024-09-19T07:49:39.499655+0000","last_clean":"2024-09-19T07:49:39.499655+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:49:39.499655+0000","last_undegraded":"2024-09-19T07:49:39.499655+0000","last_fullsized":"2024-09-19T07:49:39.499655+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4199","last_scrub_stamp":"2024-09-19T07:49:39.499410+0000","last_deep_scrub":"64'4199","last_deep_scrub_stamp":"2024-09-19T07:49:39.499410+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.499410+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:59.463907+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22884,"num_write_kb":280362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"61'5900","reported_seq":6023,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.263166+0000","last_change":"2024-09-19T07:49:28.263234+0000","last_active":"2024-09-19T07:49:28.263166+0000","last_peered":"2024-09-19T07:49:28.263166+0000","last_clean":"2024-09-19T07:49:28.263166+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:49:28.263166+0000","last_undegraded":"2024-09-19T07:49:28.263166+0000","last_fullsized":"2024-09-19T07:49:28.263166+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5900","last_scrub_stamp":"2024-09-19T07:49:28.262935+0000","last_deep_scrub":"61'5900","last_deep_scrub_stamp":"2024-09-19T07:49:28.262935+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.262935+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:09:31.336636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31464,"num_write_kb":364012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"61'2","reported_seq":127,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.351071+0000","last_change":"2024-09-19T07:49:40.351130+0000","last_active":"2024-09-19T07:49:40.351071+0000","last_peered":"2024-09-19T07:49:40.351071+0000","last_clean":"2024-09-19T07:49:40.351071+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:49:40.351071+0000","last_undegraded":"2024-09-19T07:49:40.351071+0000","last_fullsized":"2024-09-19T07:49:40.351071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-19T07:49:40.350876+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-19T07:49:40.350876+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.350876+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:01:39.933098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"64'3","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.540273+0000","last_change":"2024-09-19T07:49:35.540342+0000","last_active":"2024-09-19T07:49:35.540273+0000","last_peered":"2024-09-19T07:49:35.540273+0000","last_clean":"2024-09-19T07:49:35.540273+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:49:35.540273+0000","last_undegraded":"2024-09-19T07:49:35.540273+0000","last_fullsized":"2024-09-19T07:49:35.540273+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-09-19T07:49:35.540059+0000","last_deep_scrub":"64'3","last_deep_scrub_stamp":"2024-09-19T07:49:35.540059+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.540059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:11:27.485809+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"64'4404","reported_seq":4519,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.496603+0000","last_change":"2024-09-19T07:49:37.496663+0000","last_active":"2024-09-19T07:49:37.496603+0000","last_peered":"2024-09-19T07:49:37.496603+0000","last_clean":"2024-09-19T07:49:37.496603+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:49:37.496603+0000","last_undegraded":"2024-09-19T07:49:37.496603+0000","last_fullsized":"2024-09-19T07:49:37.496603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4404","last_scrub_stamp":"2024-09-19T07:49:37.496410+0000","last_deep_scrub":"64'4404","last_deep_scrub_stamp":"2024-09-19T07:49:37.496410+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.496410+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:42:33.605988+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24188,"num_write_kb":314198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"61'3777","reported_seq":3900,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.477401+0000","last_change":"2024-09-19T07:49:40.477460+0000","last_active":"2024-09-19T07:49:40.477401+0000","last_peered":"2024-09-19T07:49:40.477401+0000","last_clean":"2024-09-19T07:49:40.477401+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:49:40.477401+0000","last_undegraded":"2024-09-19T07:49:40.477401+0000","last_fullsized":"2024-09-19T07:49:40.477401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3777","last_scrub_stamp":"2024-09-19T07:49:40.477210+0000","last_deep_scrub":"61'3777","last_deep_scrub_stamp":"2024-09-19T07:49:40.477210+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.477210+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:25:24.847439+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20312,"num_write_kb":245406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"61'2","reported_seq":127,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.240822+0000","last_change":"2024-09-19T07:49:31.240961+0000","last_active":"2024-09-19T07:49:31.240822+0000","last_peered":"2024-09-19T07:49:31.240822+0000","last_clean":"2024-09-19T07:49:31.240822+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:49:31.240822+0000","last_undegraded":"2024-09-19T07:49:31.240822+0000","last_fullsized":"2024-09-19T07:49:31.240822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-19T07:49:31.240511+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-19T07:49:31.240511+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.240511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:19.769436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'4","reported_seq":121,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.158463+0000","last_change":"2024-09-19T07:49:23.158484+0000","last_active":"2024-09-19T07:49:23.158463+0000","last_peered":"2024-09-19T07:49:23.158463+0000","last_clean":"2024-09-19T07:49:23.158463+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:49:23.158463+0000","last_undegraded":"2024-09-19T07:49:23.158463+0000","last_fullsized":"2024-09-19T07:49:23.158463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-09-19T07:49:23.158382+0000","last_deep_scrub":"64'4","last_deep_scrub_stamp":"2024-09-19T07:49:23.158382+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.158382+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:28.877291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"64'4365","reported_seq":4480,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.457199+0000","last_change":"2024-09-19T07:49:40.457265+0000","last_active":"2024-09-19T07:49:40.457199+0000","last_peered":"2024-09-19T07:49:40.457199+0000","last_clean":"2024-09-19T07:49:40.457199+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:49:40.457199+0000","last_undegraded":"2024-09-19T07:49:40.457199+0000","last_fullsized":"2024-09-19T07:49:40.457199+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4365","last_scrub_stamp":"2024-09-19T07:49:40.456941+0000","last_deep_scrub":"64'4365","last_deep_scrub_stamp":"2024-09-19T07:49:40.456941+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.456941+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:56.706184+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24012,"num_write_kb":288918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'5396","reported_seq":5519,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.298054+0000","last_change":"2024-09-19T07:49:30.298222+0000","last_active":"2024-09-19T07:49:30.298054+0000","last_peered":"2024-09-19T07:49:30.298054+0000","last_clean":"2024-09-19T07:49:30.298054+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:49:30.298054+0000","last_undegraded":"2024-09-19T07:49:30.298054+0000","last_fullsized":"2024-09-19T07:49:30.298054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5396","last_scrub_stamp":"2024-09-19T07:49:30.297733+0000","last_deep_scrub":"61'5396","last_deep_scrub_stamp":"2024-09-19T07:49:30.297733+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.297733+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:06:29.592193+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28800,"num_write_kb":336838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'4","reported_seq":129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.460668+0000","last_change":"2024-09-19T07:49:34.460792+0000","last_active":"2024-09-19T07:49:34.460668+0000","last_peered":"2024-09-19T07:49:34.460668+0000","last_clean":"2024-09-19T07:49:34.460668+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:49:34.460668+0000","last_undegraded":"2024-09-19T07:49:34.460668+0000","last_fullsized":"2024-09-19T07:49:34.460668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4","last_scrub_stamp":"2024-09-19T07:49:34.460373+0000","last_deep_scrub":"61'4","last_deep_scrub_stamp":"2024-09-19T07:49:34.460373+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.460373+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:21:55.084729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"64'2","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.190194+0000","last_change":"2024-09-19T07:49:22.190249+0000","last_active":"2024-09-19T07:49:22.190194+0000","last_peered":"2024-09-19T07:49:22.190194+0000","last_clean":"2024-09-19T07:49:22.190194+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:49:22.190194+0000","last_undegraded":"2024-09-19T07:49:22.190194+0000","last_fullsized":"2024-09-19T07:49:22.190194+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-19T07:49:22.190085+0000","last_deep_scrub":"64'2","last_deep_scrub_stamp":"2024-09-19T07:49:22.190085+0000","last_clean_scrub_stamp":"2024-09-19T07:49:22.190085+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:11:28.432267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"64'4926","reported_seq":5041,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.633944+0000","last_change":"2024-09-19T07:49:28.634007+0000","last_active":"2024-09-19T07:49:28.633944+0000","last_peered":"2024-09-19T07:49:28.633944+0000","last_clean":"2024-09-19T07:49:28.633944+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:49:28.633944+0000","last_undegraded":"2024-09-19T07:49:28.633944+0000","last_fullsized":"2024-09-19T07:49:28.633944+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4926","last_scrub_stamp":"2024-09-19T07:49:28.633734+0000","last_deep_scrub":"64'4926","last_deep_scrub_stamp":"2024-09-19T07:49:28.633734+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.633734+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:48:59.019782+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26066,"num_write_kb":337871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'3534","reported_seq":3658,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.254070+0000","last_change":"2024-09-19T07:49:30.254070+0000","last_active":"2024-09-19T07:49:30.254070+0000","last_peered":"2024-09-19T07:49:30.254070+0000","last_clean":"2024-09-19T07:49:30.254070+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:49:30.254070+0000","last_undegraded":"2024-09-19T07:49:30.254070+0000","last_fullsized":"2024-09-19T07:49:30.254070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3534","last_scrub_stamp":"2024-09-19T07:49:30.253787+0000","last_deep_scrub":"61'3534","last_deep_scrub_stamp":"2024-09-19T07:49:30.253787+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.253787+0000","objects_scrubbed":0,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:15:25.385951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19019,"num_write_kb":227884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"65'46","reported_seq":230,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.704762+0000","last_change":"2024-09-19T07:49:23.704827+0000","last_active":"2024-09-19T07:49:23.704762+0000","last_peered":"2024-09-19T07:49:23.704762+0000","last_clean":"2024-09-19T07:49:23.704762+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:49:23.704762+0000","last_undegraded":"2024-09-19T07:49:23.704762+0000","last_fullsized":"2024-09-19T07:49:23.704762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'46","last_scrub_stamp":"2024-09-19T07:49:23.704580+0000","last_deep_scrub":"65'46","last_deep_scrub_stamp":"2024-09-19T07:49:23.704580+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.704580+0000","objects_scrubbed":1,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:07:33.376909+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":109,"num_write":40,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'10","reported_seq":166,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.245464+0000","last_change":"2024-09-19T07:49:25.245537+0000","last_active":"2024-09-19T07:49:25.245464+0000","last_peered":"2024-09-19T07:49:25.245464+0000","last_clean":"2024-09-19T07:49:25.245464+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:49:25.245464+0000","last_undegraded":"2024-09-19T07:49:25.245464+0000","last_fullsized":"2024-09-19T07:49:25.245464+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-19T07:49:25.245276+0000","last_deep_scrub":"65'10","last_deep_scrub_stamp":"2024-09-19T07:49:25.245276+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.245276+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:38:15.714808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"64'3425","reported_seq":3531,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.371780+0000","last_change":"2024-09-19T07:19:40.387573+0000","last_active":"2024-09-19T07:48:58.371780+0000","last_peered":"2024-09-19T07:48:58.371780+0000","last_clean":"2024-09-19T07:48:58.371780+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:48:58.371780+0000","last_undegraded":"2024-09-19T07:48:58.371780+0000","last_fullsized":"2024-09-19T07:48:58.371780+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19036,"num_write_kb":252575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"61'4467","reported_seq":4590,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:21.558948+0000","last_change":"2024-09-19T07:49:21.559009+0000","last_active":"2024-09-19T07:49:21.558948+0000","last_peered":"2024-09-19T07:49:21.558948+0000","last_clean":"2024-09-19T07:49:21.558948+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:49:21.558948+0000","last_undegraded":"2024-09-19T07:49:21.558948+0000","last_fullsized":"2024-09-19T07:49:21.558948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4467","last_scrub_stamp":"2024-09-19T07:49:21.558756+0000","last_deep_scrub":"61'4467","last_deep_scrub_stamp":"2024-09-19T07:49:21.558756+0000","last_clean_scrub_stamp":"2024-09-19T07:49:21.558756+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:10:47.453175+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23642,"num_write_kb":275072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'13","reported_seq":139,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.315255+0000","last_change":"2024-09-19T07:49:37.315255+0000","last_active":"2024-09-19T07:49:37.315255+0000","last_peered":"2024-09-19T07:49:37.315255+0000","last_clean":"2024-09-19T07:49:37.315255+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:49:37.315255+0000","last_undegraded":"2024-09-19T07:49:37.315255+0000","last_fullsized":"2024-09-19T07:49:37.315255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-19T07:49:37.314883+0000","last_deep_scrub":"65'13","last_deep_scrub_stamp":"2024-09-19T07:49:37.314883+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.314883+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:19:14.744779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'515","reported_seq":651,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.562330+0000","last_change":"2024-09-19T07:49:34.562394+0000","last_active":"2024-09-19T07:49:34.562330+0000","last_peered":"2024-09-19T07:49:34.562330+0000","last_clean":"2024-09-19T07:49:34.562330+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:49:34.562330+0000","last_undegraded":"2024-09-19T07:49:34.562330+0000","last_fullsized":"2024-09-19T07:49:34.562330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'515","last_scrub_stamp":"2024-09-19T07:49:34.562143+0000","last_deep_scrub":"65'515","last_deep_scrub_stamp":"2024-09-19T07:49:34.562143+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.562143+0000","objects_scrubbed":1,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:29.876600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":53,"num_write":1231,"num_write_kb":406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"64'4906","reported_seq":5013,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373062+0000","last_change":"2024-09-19T07:19:40.392973+0000","last_active":"2024-09-19T07:48:58.373062+0000","last_peered":"2024-09-19T07:48:58.373062+0000","last_clean":"2024-09-19T07:48:58.373062+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:48:58.373062+0000","last_undegraded":"2024-09-19T07:48:58.373062+0000","last_fullsized":"2024-09-19T07:48:58.373062+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27102,"num_write_kb":350472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"61'4338","reported_seq":4461,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.486534+0000","last_change":"2024-09-19T07:49:29.486599+0000","last_active":"2024-09-19T07:49:29.486534+0000","last_peered":"2024-09-19T07:49:29.486534+0000","last_clean":"2024-09-19T07:49:29.486534+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:49:29.486534+0000","last_undegraded":"2024-09-19T07:49:29.486534+0000","last_fullsized":"2024-09-19T07:49:29.486534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4338","last_scrub_stamp":"2024-09-19T07:49:29.486317+0000","last_deep_scrub":"61'4338","last_deep_scrub_stamp":"2024-09-19T07:49:29.486317+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.486317+0000","objects_scrubbed":0,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:22.201418+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22767,"num_write_kb":271598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"61'3","reported_seq":128,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.481888+0000","last_change":"2024-09-19T07:49:35.481953+0000","last_active":"2024-09-19T07:49:35.481888+0000","last_peered":"2024-09-19T07:49:35.481888+0000","last_clean":"2024-09-19T07:49:35.481888+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:49:35.481888+0000","last_undegraded":"2024-09-19T07:49:35.481888+0000","last_fullsized":"2024-09-19T07:49:35.481888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-19T07:49:35.481692+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-19T07:49:35.481692+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.481692+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:13:00.655487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"65'119","reported_seq":284,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.332555+0000","last_change":"2024-09-19T07:49:35.332555+0000","last_active":"2024-09-19T07:49:35.332555+0000","last_peered":"2024-09-19T07:49:35.332555+0000","last_clean":"2024-09-19T07:49:35.332555+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:49:35.332555+0000","last_undegraded":"2024-09-19T07:49:35.332555+0000","last_fullsized":"2024-09-19T07:49:35.332555+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'119","last_scrub_stamp":"2024-09-19T07:49:35.332138+0000","last_deep_scrub":"65'119","last_deep_scrub_stamp":"2024-09-19T07:49:35.332138+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.332138+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:01:01.847510+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'2","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.549773+0000","last_change":"2024-09-19T07:49:35.549832+0000","last_active":"2024-09-19T07:49:35.549773+0000","last_peered":"2024-09-19T07:49:35.549773+0000","last_clean":"2024-09-19T07:49:35.549773+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:49:35.549773+0000","last_undegraded":"2024-09-19T07:49:35.549773+0000","last_fullsized":"2024-09-19T07:49:35.549773+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-19T07:49:35.549573+0000","last_deep_scrub":"64'2","last_deep_scrub_stamp":"2024-09-19T07:49:35.549573+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.549573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:20:28.110800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"64'3599","reported_seq":3713,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.636328+0000","last_change":"2024-09-19T07:49:33.636392+0000","last_active":"2024-09-19T07:49:33.636328+0000","last_peered":"2024-09-19T07:49:33.636328+0000","last_clean":"2024-09-19T07:49:33.636328+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:49:33.636328+0000","last_undegraded":"2024-09-19T07:49:33.636328+0000","last_fullsized":"2024-09-19T07:49:33.636328+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3599","last_scrub_stamp":"2024-09-19T07:49:33.636132+0000","last_deep_scrub":"64'3599","last_deep_scrub_stamp":"2024-09-19T07:49:33.636132+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.636132+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:09:04.310677+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19570,"num_write_kb":246554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"61'5018","reported_seq":5132,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.379108+0000","last_change":"2024-09-19T07:19:32.268713+0000","last_active":"2024-09-19T07:48:58.379108+0000","last_peered":"2024-09-19T07:48:58.379108+0000","last_clean":"2024-09-19T07:48:58.379108+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:48:58.379108+0000","last_undegraded":"2024-09-19T07:48:58.379108+0000","last_fullsized":"2024-09-19T07:48:58.379108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26753,"num_write_kb":316367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"61'5","reported_seq":129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.694372+0000","last_change":"2024-09-19T07:49:24.694433+0000","last_active":"2024-09-19T07:49:24.694372+0000","last_peered":"2024-09-19T07:49:24.694372+0000","last_clean":"2024-09-19T07:49:24.694372+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:49:24.694372+0000","last_undegraded":"2024-09-19T07:49:24.694372+0000","last_fullsized":"2024-09-19T07:49:24.694372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5","last_scrub_stamp":"2024-09-19T07:49:24.694151+0000","last_deep_scrub":"61'5","last_deep_scrub_stamp":"2024-09-19T07:49:24.694151+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.694151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:06:52.476439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"64'3","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.447296+0000","last_change":"2024-09-19T07:49:25.447357+0000","last_active":"2024-09-19T07:49:25.447296+0000","last_peered":"2024-09-19T07:49:25.447296+0000","last_clean":"2024-09-19T07:49:25.447296+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:49:25.447296+0000","last_undegraded":"2024-09-19T07:49:25.447296+0000","last_fullsized":"2024-09-19T07:49:25.447296+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-09-19T07:49:25.447095+0000","last_deep_scrub":"64'3","last_deep_scrub_stamp":"2024-09-19T07:49:25.447095+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.447095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:56.882666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"64'4954","reported_seq":5068,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.594547+0000","last_change":"2024-09-19T07:49:35.594616+0000","last_active":"2024-09-19T07:49:35.594547+0000","last_peered":"2024-09-19T07:49:35.594547+0000","last_clean":"2024-09-19T07:49:35.594547+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:49:35.594547+0000","last_undegraded":"2024-09-19T07:49:35.594547+0000","last_fullsized":"2024-09-19T07:49:35.594547+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4954","last_scrub_stamp":"2024-09-19T07:49:35.594290+0000","last_deep_scrub":"64'4954","last_deep_scrub_stamp":"2024-09-19T07:49:35.594290+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.594290+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:56:14.519746+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27293,"num_write_kb":322594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"61'5220","reported_seq":5328,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.587513+0000","last_change":"2024-09-19T07:49:36.587574+0000","last_active":"2024-09-19T07:49:36.587513+0000","last_peered":"2024-09-19T07:49:36.587513+0000","last_clean":"2024-09-19T07:49:36.587513+0000","last_became_active":"2024-09-19T07:20:13.951449+0000","last_became_peered":"2024-09-19T07:20:13.951449+0000","last_unstale":"2024-09-19T07:49:36.587513+0000","last_undegraded":"2024-09-19T07:49:36.587513+0000","last_fullsized":"2024-09-19T07:49:36.587513+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5220","last_scrub_stamp":"2024-09-19T07:49:36.587313+0000","last_deep_scrub":"61'5220","last_deep_scrub_stamp":"2024-09-19T07:49:36.587313+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.587313+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:01:06.178431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27694,"num_write_kb":340542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"65'7","reported_seq":151,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.201648+0000","last_change":"2024-09-19T07:49:33.201712+0000","last_active":"2024-09-19T07:49:33.201648+0000","last_peered":"2024-09-19T07:49:33.201648+0000","last_clean":"2024-09-19T07:49:33.201648+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:49:33.201648+0000","last_undegraded":"2024-09-19T07:49:33.201648+0000","last_fullsized":"2024-09-19T07:49:33.201648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'7","last_scrub_stamp":"2024-09-19T07:49:33.201448+0000","last_deep_scrub":"65'7","last_deep_scrub_stamp":"2024-09-19T07:49:33.201448+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.201448+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:36:25.509689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'1405","reported_seq":1564,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.634210+0000","last_change":"2024-09-19T07:49:33.634354+0000","last_active":"2024-09-19T07:49:33.634210+0000","last_peered":"2024-09-19T07:49:33.634210+0000","last_clean":"2024-09-19T07:49:33.634210+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:49:33.634210+0000","last_undegraded":"2024-09-19T07:49:33.634210+0000","last_fullsized":"2024-09-19T07:49:33.634210+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1405","last_scrub_stamp":"2024-09-19T07:49:33.633973+0000","last_deep_scrub":"65'1405","last_deep_scrub_stamp":"2024-09-19T07:49:33.633973+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.633973+0000","objects_scrubbed":2,"log_size":1405,"log_dups_size":0,"ondisk_log_size":1405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:58:03.789148+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":63,"num_read_kb":30148,"num_write":1379,"num_write_kb":4466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"64'3869","reported_seq":3983,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.362788+0000","last_change":"2024-09-19T07:49:33.362844+0000","last_active":"2024-09-19T07:49:33.362788+0000","last_peered":"2024-09-19T07:49:33.362788+0000","last_clean":"2024-09-19T07:49:33.362788+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:49:33.362788+0000","last_undegraded":"2024-09-19T07:49:33.362788+0000","last_fullsized":"2024-09-19T07:49:33.362788+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3869","last_scrub_stamp":"2024-09-19T07:49:33.362596+0000","last_deep_scrub":"64'3869","last_deep_scrub_stamp":"2024-09-19T07:49:33.362596+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.362596+0000","objects_scrubbed":0,"log_size":3869,"log_dups_size":0,"ondisk_log_size":3869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:34:57.359555+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20920,"num_write_kb":264190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"61'6113","reported_seq":6213,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:20.241793+0000","last_change":"2024-09-19T07:20:14.209767+0000","last_active":"2024-09-19T07:49:20.241793+0000","last_peered":"2024-09-19T07:49:20.241793+0000","last_clean":"2024-09-19T07:49:20.241793+0000","last_became_active":"2024-09-19T07:20:14.209141+0000","last_became_peered":"2024-09-19T07:20:14.209141+0000","last_unstale":"2024-09-19T07:49:20.241793+0000","last_undegraded":"2024-09-19T07:49:20.241793+0000","last_fullsized":"2024-09-19T07:49:20.241793+0000","mapping_epoch":26,"log_start":"54'300","ondisk_log_start":"54'300","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32041,"num_write_kb":407182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"65'8","reported_seq":172,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.700404+0000","last_change":"2024-09-19T07:49:22.700453+0000","last_active":"2024-09-19T07:49:22.700404+0000","last_peered":"2024-09-19T07:49:22.700404+0000","last_clean":"2024-09-19T07:49:22.700404+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:49:22.700404+0000","last_undegraded":"2024-09-19T07:49:22.700404+0000","last_fullsized":"2024-09-19T07:49:22.700404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-19T07:49:22.700307+0000","last_deep_scrub":"65'8","last_deep_scrub_stamp":"2024-09-19T07:49:22.700307+0000","last_clean_scrub_stamp":"2024-09-19T07:49:22.700307+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:54:13.365948+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"64'5","reported_seq":121,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.516598+0000","last_change":"2024-09-19T07:49:26.516659+0000","last_active":"2024-09-19T07:49:26.516598+0000","last_peered":"2024-09-19T07:49:26.516598+0000","last_clean":"2024-09-19T07:49:26.516598+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:49:26.516598+0000","last_undegraded":"2024-09-19T07:49:26.516598+0000","last_fullsized":"2024-09-19T07:49:26.516598+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5","last_scrub_stamp":"2024-09-19T07:49:26.516376+0000","last_deep_scrub":"64'5","last_deep_scrub_stamp":"2024-09-19T07:49:26.516376+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.516376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:40:17.237269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"64'4634","reported_seq":4748,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.656723+0000","last_change":"2024-09-19T07:49:31.656792+0000","last_active":"2024-09-19T07:49:31.656723+0000","last_peered":"2024-09-19T07:49:31.656723+0000","last_clean":"2024-09-19T07:49:31.656723+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:49:31.656723+0000","last_undegraded":"2024-09-19T07:49:31.656723+0000","last_fullsized":"2024-09-19T07:49:31.656723+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4634","last_scrub_stamp":"2024-09-19T07:49:31.656515+0000","last_deep_scrub":"64'4634","last_deep_scrub_stamp":"2024-09-19T07:49:31.656515+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.656515+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:37.375425+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25462,"num_write_kb":298798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"61'4007","reported_seq":4129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.161629+0000","last_change":"2024-09-19T07:49:23.161741+0000","last_active":"2024-09-19T07:49:23.161629+0000","last_peered":"2024-09-19T07:49:23.161629+0000","last_clean":"2024-09-19T07:49:23.161629+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:49:23.161629+0000","last_undegraded":"2024-09-19T07:49:23.161629+0000","last_fullsized":"2024-09-19T07:49:23.161629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4007","last_scrub_stamp":"2024-09-19T07:49:23.161409+0000","last_deep_scrub":"61'4007","last_deep_scrub_stamp":"2024-09-19T07:49:23.161409+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.161409+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:38:54.800638+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21937,"num_write_kb":251948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"65'5","reported_seq":149,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.252104+0000","last_change":"2024-09-19T07:49:29.252167+0000","last_active":"2024-09-19T07:49:29.252104+0000","last_peered":"2024-09-19T07:49:29.252104+0000","last_clean":"2024-09-19T07:49:29.252104+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:49:29.252104+0000","last_undegraded":"2024-09-19T07:49:29.252104+0000","last_fullsized":"2024-09-19T07:49:29.252104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-19T07:49:29.251898+0000","last_deep_scrub":"65'5","last_deep_scrub_stamp":"2024-09-19T07:49:29.251898+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.251898+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:01:22.587094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"64'2","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.343672+0000","last_change":"2024-09-19T07:49:23.343732+0000","last_active":"2024-09-19T07:49:23.343672+0000","last_peered":"2024-09-19T07:49:23.343672+0000","last_clean":"2024-09-19T07:49:23.343672+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:49:23.343672+0000","last_undegraded":"2024-09-19T07:49:23.343672+0000","last_fullsized":"2024-09-19T07:49:23.343672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-19T07:49:23.343475+0000","last_deep_scrub":"64'2","last_deep_scrub_stamp":"2024-09-19T07:49:23.343475+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.343475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:48:49.929603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"64'3939","reported_seq":4044,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364222+0000","last_change":"2024-09-19T07:19:40.387159+0000","last_active":"2024-09-19T07:48:58.364222+0000","last_peered":"2024-09-19T07:48:58.364222+0000","last_clean":"2024-09-19T07:48:58.364222+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:48:58.364222+0000","last_undegraded":"2024-09-19T07:48:58.364222+0000","last_fullsized":"2024-09-19T07:48:58.364222+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20741,"num_write_kb":275861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'4660","reported_seq":4783,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.239203+0000","last_change":"2024-09-19T07:49:25.239268+0000","last_active":"2024-09-19T07:49:25.239203+0000","last_peered":"2024-09-19T07:49:25.239203+0000","last_clean":"2024-09-19T07:49:25.239203+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:49:25.239203+0000","last_undegraded":"2024-09-19T07:49:25.239203+0000","last_fullsized":"2024-09-19T07:49:25.239203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4660","last_scrub_stamp":"2024-09-19T07:49:25.238988+0000","last_deep_scrub":"61'4660","last_deep_scrub_stamp":"2024-09-19T07:49:25.238988+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.238988+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:00:54.830107+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25264,"num_write_kb":302007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"65'19","reported_seq":226,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:21.677882+0000","last_change":"2024-09-19T07:49:21.677922+0000","last_active":"2024-09-19T07:49:21.677882+0000","last_peered":"2024-09-19T07:49:21.677882+0000","last_clean":"2024-09-19T07:49:21.677882+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:49:21.677882+0000","last_undegraded":"2024-09-19T07:49:21.677882+0000","last_fullsized":"2024-09-19T07:49:21.677882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-19T07:49:21.677795+0000","last_deep_scrub":"65'19","last_deep_scrub_stamp":"2024-09-19T07:49:21.677795+0000","last_clean_scrub_stamp":"2024-09-19T07:49:21.677795+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:28:00.139065+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"64'3","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.555137+0000","last_change":"2024-09-19T07:49:23.555201+0000","last_active":"2024-09-19T07:49:23.555137+0000","last_peered":"2024-09-19T07:49:23.555137+0000","last_clean":"2024-09-19T07:49:23.555137+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:49:23.555137+0000","last_undegraded":"2024-09-19T07:49:23.555137+0000","last_fullsized":"2024-09-19T07:49:23.555137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-09-19T07:49:23.554923+0000","last_deep_scrub":"64'3","last_deep_scrub_stamp":"2024-09-19T07:49:23.554923+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.554923+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:14:31.786151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"64'4590","reported_seq":4703,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.466393+0000","last_change":"2024-09-19T07:49:32.466460+0000","last_active":"2024-09-19T07:49:32.466393+0000","last_peered":"2024-09-19T07:49:32.466393+0000","last_clean":"2024-09-19T07:49:32.466393+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:49:32.466393+0000","last_undegraded":"2024-09-19T07:49:32.466393+0000","last_fullsized":"2024-09-19T07:49:32.466393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4590","last_scrub_stamp":"2024-09-19T07:49:32.466159+0000","last_deep_scrub":"64'4590","last_deep_scrub_stamp":"2024-09-19T07:49:32.466159+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.466159+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:37.762557+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24790,"num_write_kb":304361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"61'3873","reported_seq":3994,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.549318+0000","last_change":"2024-09-19T07:49:34.549376+0000","last_active":"2024-09-19T07:49:34.549318+0000","last_peered":"2024-09-19T07:49:34.549318+0000","last_clean":"2024-09-19T07:49:34.549318+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:49:34.549318+0000","last_undegraded":"2024-09-19T07:49:34.549318+0000","last_fullsized":"2024-09-19T07:49:34.549318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3873","last_scrub_stamp":"2024-09-19T07:49:34.549125+0000","last_deep_scrub":"61'3873","last_deep_scrub_stamp":"2024-09-19T07:49:34.549125+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.549125+0000","objects_scrubbed":0,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:09:37.622826+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20508,"num_write_kb":250225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"61'5","reported_seq":128,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.469448+0000","last_change":"2024-09-19T07:49:27.469507+0000","last_active":"2024-09-19T07:49:27.469448+0000","last_peered":"2024-09-19T07:49:27.469448+0000","last_clean":"2024-09-19T07:49:27.469448+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:49:27.469448+0000","last_undegraded":"2024-09-19T07:49:27.469448+0000","last_fullsized":"2024-09-19T07:49:27.469448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5","last_scrub_stamp":"2024-09-19T07:49:27.469254+0000","last_deep_scrub":"61'5","last_deep_scrub_stamp":"2024-09-19T07:49:27.469254+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.469254+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:09:27.876094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"64'4","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.467925+0000","last_change":"2024-09-19T07:49:28.467989+0000","last_active":"2024-09-19T07:49:28.467925+0000","last_peered":"2024-09-19T07:49:28.467925+0000","last_clean":"2024-09-19T07:49:28.467925+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:49:28.467925+0000","last_undegraded":"2024-09-19T07:49:28.467925+0000","last_fullsized":"2024-09-19T07:49:28.467925+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-09-19T07:49:28.467324+0000","last_deep_scrub":"64'4","last_deep_scrub_stamp":"2024-09-19T07:49:28.467324+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.467324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:36:20.775564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"64'3485","reported_seq":3590,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366672+0000","last_change":"2024-09-19T07:19:40.373542+0000","last_active":"2024-09-19T07:48:58.366672+0000","last_peered":"2024-09-19T07:48:58.366672+0000","last_clean":"2024-09-19T07:48:58.366672+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:48:58.366672+0000","last_undegraded":"2024-09-19T07:48:58.366672+0000","last_fullsized":"2024-09-19T07:48:58.366672+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19669,"num_write_kb":242834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"61'5231","reported_seq":5352,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.282760+0000","last_change":"2024-09-19T07:49:27.282828+0000","last_active":"2024-09-19T07:49:27.282760+0000","last_peered":"2024-09-19T07:49:27.282760+0000","last_clean":"2024-09-19T07:49:27.282760+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:49:27.282760+0000","last_undegraded":"2024-09-19T07:49:27.282760+0000","last_fullsized":"2024-09-19T07:49:27.282760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5231","last_scrub_stamp":"2024-09-19T07:49:27.282524+0000","last_deep_scrub":"61'5231","last_deep_scrub_stamp":"2024-09-19T07:49:27.282524+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.282524+0000","objects_scrubbed":0,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:26:58.138512+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28071,"num_write_kb":333869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"61'3","reported_seq":126,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.439508+0000","last_change":"2024-09-19T07:49:36.439571+0000","last_active":"2024-09-19T07:49:36.439508+0000","last_peered":"2024-09-19T07:49:36.439508+0000","last_clean":"2024-09-19T07:49:36.439508+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:49:36.439508+0000","last_undegraded":"2024-09-19T07:49:36.439508+0000","last_fullsized":"2024-09-19T07:49:36.439508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-19T07:49:36.439312+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-19T07:49:36.439312+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.439312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:45:50.370417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'331","reported_seq":547,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.593122+0000","last_change":"2024-09-19T07:49:24.593194+0000","last_active":"2024-09-19T07:49:24.593122+0000","last_peered":"2024-09-19T07:49:24.593122+0000","last_clean":"2024-09-19T07:49:24.593122+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:49:24.593122+0000","last_undegraded":"2024-09-19T07:49:24.593122+0000","last_fullsized":"2024-09-19T07:49:24.593122+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'331","last_scrub_stamp":"2024-09-19T07:49:24.592917+0000","last_deep_scrub":"65'331","last_deep_scrub_stamp":"2024-09-19T07:49:24.592917+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.592917+0000","objects_scrubbed":4,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:26:55.448905+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":181,"num_read_kb":77990,"num_write":325,"num_write_kb":4282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"64'5112","reported_seq":5217,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363117+0000","last_change":"2024-09-19T07:19:40.364814+0000","last_active":"2024-09-19T07:48:58.363117+0000","last_peered":"2024-09-19T07:48:58.363117+0000","last_clean":"2024-09-19T07:48:58.363117+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:48:58.363117+0000","last_undegraded":"2024-09-19T07:48:58.363117+0000","last_fullsized":"2024-09-19T07:48:58.363117+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5112,"log_dups_size":0,"ondisk_log_size":5112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27577,"num_write_kb":360586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"61'4085","reported_seq":4206,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.581335+0000","last_change":"2024-09-19T07:49:32.581403+0000","last_active":"2024-09-19T07:49:32.581335+0000","last_peered":"2024-09-19T07:49:32.581335+0000","last_clean":"2024-09-19T07:49:32.581335+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:49:32.581335+0000","last_undegraded":"2024-09-19T07:49:32.581335+0000","last_fullsized":"2024-09-19T07:49:32.581335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4085","last_scrub_stamp":"2024-09-19T07:49:32.581108+0000","last_deep_scrub":"61'4085","last_deep_scrub_stamp":"2024-09-19T07:49:32.581108+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.581108+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:50:54.490092+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22122,"num_write_kb":273218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"65'12","reported_seq":175,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.596628+0000","last_change":"2024-09-19T07:49:25.596695+0000","last_active":"2024-09-19T07:49:25.596628+0000","last_peered":"2024-09-19T07:49:25.596628+0000","last_clean":"2024-09-19T07:49:25.596628+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:49:25.596628+0000","last_undegraded":"2024-09-19T07:49:25.596628+0000","last_fullsized":"2024-09-19T07:49:25.596628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-19T07:49:25.596435+0000","last_deep_scrub":"65'12","last_deep_scrub_stamp":"2024-09-19T07:49:25.596435+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.596435+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:16.739671+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'30","reported_seq":205,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.529991+0000","last_change":"2024-09-19T07:49:24.530059+0000","last_active":"2024-09-19T07:49:24.529991+0000","last_peered":"2024-09-19T07:49:24.529991+0000","last_clean":"2024-09-19T07:49:24.529991+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:49:24.529991+0000","last_undegraded":"2024-09-19T07:49:24.529991+0000","last_fullsized":"2024-09-19T07:49:24.529991+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-09-19T07:49:24.529796+0000","last_deep_scrub":"65'30","last_deep_scrub_stamp":"2024-09-19T07:49:24.529796+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.529796+0000","objects_scrubbed":2,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:50:09.224306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"64'4543","reported_seq":4648,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363471+0000","last_change":"2024-09-19T07:19:40.381443+0000","last_active":"2024-09-19T07:48:58.363471+0000","last_peered":"2024-09-19T07:48:58.363471+0000","last_clean":"2024-09-19T07:48:58.363471+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:48:58.363471+0000","last_undegraded":"2024-09-19T07:48:58.363471+0000","last_fullsized":"2024-09-19T07:48:58.363471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24993,"num_write_kb":317373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"61'3981","reported_seq":4102,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.312053+0000","last_change":"2024-09-19T07:49:28.312116+0000","last_active":"2024-09-19T07:49:28.312053+0000","last_peered":"2024-09-19T07:49:28.312053+0000","last_clean":"2024-09-19T07:49:28.312053+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:49:28.312053+0000","last_undegraded":"2024-09-19T07:49:28.312053+0000","last_fullsized":"2024-09-19T07:49:28.312053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3981","last_scrub_stamp":"2024-09-19T07:49:28.311812+0000","last_deep_scrub":"61'3981","last_deep_scrub_stamp":"2024-09-19T07:49:28.311812+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.311812+0000","objects_scrubbed":0,"log_size":3981,"log_dups_size":0,"ondisk_log_size":3981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:56.707334+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21890,"num_write_kb":263866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"65'355","reported_seq":579,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.492021+0000","last_change":"2024-09-19T07:49:29.492099+0000","last_active":"2024-09-19T07:49:29.492021+0000","last_peered":"2024-09-19T07:49:29.492021+0000","last_clean":"2024-09-19T07:49:29.492021+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:49:29.492021+0000","last_undegraded":"2024-09-19T07:49:29.492021+0000","last_fullsized":"2024-09-19T07:49:29.492021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'355","last_scrub_stamp":"2024-09-19T07:49:29.491736+0000","last_deep_scrub":"65'355","last_deep_scrub_stamp":"2024-09-19T07:49:29.491736+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.491736+0000","objects_scrubbed":4,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:46:11.249153+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":181,"num_read_kb":77990,"num_write":347,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"64'3","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.309488+0000","last_change":"2024-09-19T07:49:27.309551+0000","last_active":"2024-09-19T07:49:27.309488+0000","last_peered":"2024-09-19T07:49:27.309488+0000","last_clean":"2024-09-19T07:49:27.309488+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:49:27.309488+0000","last_undegraded":"2024-09-19T07:49:27.309488+0000","last_fullsized":"2024-09-19T07:49:27.309488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-09-19T07:49:27.309274+0000","last_deep_scrub":"64'3","last_deep_scrub_stamp":"2024-09-19T07:49:27.309274+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.309274+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:06:26.603734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"64'3743","reported_seq":3856,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.155804+0000","last_change":"2024-09-19T07:49:40.155886+0000","last_active":"2024-09-19T07:49:40.155804+0000","last_peered":"2024-09-19T07:49:40.155804+0000","last_clean":"2024-09-19T07:49:40.155804+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:49:40.155804+0000","last_undegraded":"2024-09-19T07:49:40.155804+0000","last_fullsized":"2024-09-19T07:49:40.155804+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3743","last_scrub_stamp":"2024-09-19T07:49:40.155596+0000","last_deep_scrub":"64'3743","last_deep_scrub_stamp":"2024-09-19T07:49:40.155596+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.155596+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:57:58.883136+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20626,"num_write_kb":263086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"61'6160","reported_seq":6281,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.351108+0000","last_change":"2024-09-19T07:49:31.351169+0000","last_active":"2024-09-19T07:49:31.351108+0000","last_peered":"2024-09-19T07:49:31.351108+0000","last_clean":"2024-09-19T07:49:31.351108+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:49:31.351108+0000","last_undegraded":"2024-09-19T07:49:31.351108+0000","last_fullsized":"2024-09-19T07:49:31.351108+0000","mapping_epoch":19,"log_start":"54'700","ondisk_log_start":"54'700","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'6160","last_scrub_stamp":"2024-09-19T07:49:31.350899+0000","last_deep_scrub":"61'6160","last_deep_scrub_stamp":"2024-09-19T07:49:31.350899+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.350899+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:24:40.317687+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32461,"num_write_kb":376158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"65'31","reported_seq":214,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.192661+0000","last_change":"2024-09-19T07:49:36.192783+0000","last_active":"2024-09-19T07:49:36.192661+0000","last_peered":"2024-09-19T07:49:36.192661+0000","last_clean":"2024-09-19T07:49:36.192661+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:49:36.192661+0000","last_undegraded":"2024-09-19T07:49:36.192661+0000","last_fullsized":"2024-09-19T07:49:36.192661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'31","last_scrub_stamp":"2024-09-19T07:49:36.192384+0000","last_deep_scrub":"65'31","last_deep_scrub_stamp":"2024-09-19T07:49:36.192384+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.192384+0000","objects_scrubbed":2,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:17:00.993654+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'10","reported_seq":165,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.312623+0000","last_change":"2024-09-19T07:49:27.312693+0000","last_active":"2024-09-19T07:49:27.312623+0000","last_peered":"2024-09-19T07:49:27.312623+0000","last_clean":"2024-09-19T07:49:27.312623+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:49:27.312623+0000","last_undegraded":"2024-09-19T07:49:27.312623+0000","last_fullsized":"2024-09-19T07:49:27.312623+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-19T07:49:27.312421+0000","last_deep_scrub":"65'10","last_deep_scrub_stamp":"2024-09-19T07:49:27.312421+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.312421+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:33:29.026544+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"64'4564","reported_seq":4679,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.546454+0000","last_change":"2024-09-19T07:49:40.546525+0000","last_active":"2024-09-19T07:49:40.546454+0000","last_peered":"2024-09-19T07:49:40.546454+0000","last_clean":"2024-09-19T07:49:40.546454+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:49:40.546454+0000","last_undegraded":"2024-09-19T07:49:40.546454+0000","last_fullsized":"2024-09-19T07:49:40.546454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4564","last_scrub_stamp":"2024-09-19T07:49:40.546195+0000","last_deep_scrub":"64'4564","last_deep_scrub_stamp":"2024-09-19T07:49:40.546195+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.546195+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:23.408969+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25533,"num_write_kb":303195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"61'5068","reported_seq":5189,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.605132+0000","last_change":"2024-09-19T07:49:28.605207+0000","last_active":"2024-09-19T07:49:28.605132+0000","last_peered":"2024-09-19T07:49:28.605132+0000","last_clean":"2024-09-19T07:49:28.605132+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:49:28.605132+0000","last_undegraded":"2024-09-19T07:49:28.605132+0000","last_fullsized":"2024-09-19T07:49:28.605132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5068","last_scrub_stamp":"2024-09-19T07:49:28.604922+0000","last_deep_scrub":"61'5068","last_deep_scrub_stamp":"2024-09-19T07:49:28.604922+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.604922+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:02:59.298244+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26828,"num_write_kb":316657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"61'3","reported_seq":126,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.401810+0000","last_change":"2024-09-19T07:49:38.401948+0000","last_active":"2024-09-19T07:49:38.401810+0000","last_peered":"2024-09-19T07:49:38.401810+0000","last_clean":"2024-09-19T07:49:38.401810+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:49:38.401810+0000","last_undegraded":"2024-09-19T07:49:38.401810+0000","last_fullsized":"2024-09-19T07:49:38.401810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-19T07:49:38.401480+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-19T07:49:38.401480+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.401480+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:45:57.064807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'19","reported_seq":218,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:21.625646+0000","last_change":"2024-09-19T07:49:21.625685+0000","last_active":"2024-09-19T07:49:21.625646+0000","last_peered":"2024-09-19T07:49:21.625646+0000","last_clean":"2024-09-19T07:49:21.625646+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:49:21.625646+0000","last_undegraded":"2024-09-19T07:49:21.625646+0000","last_fullsized":"2024-09-19T07:49:21.625646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-19T07:49:21.625564+0000","last_deep_scrub":"65'19","last_deep_scrub_stamp":"2024-09-19T07:49:21.625564+0000","last_clean_scrub_stamp":"2024-09-19T07:49:21.625564+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:17:31.726064+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"64'3771","reported_seq":3876,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373242+0000","last_change":"2024-09-19T07:19:40.389499+0000","last_active":"2024-09-19T07:48:58.373242+0000","last_peered":"2024-09-19T07:48:58.373242+0000","last_clean":"2024-09-19T07:48:58.373242+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:48:58.373242+0000","last_undegraded":"2024-09-19T07:48:58.373242+0000","last_fullsized":"2024-09-19T07:48:58.373242+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3771,"log_dups_size":0,"ondisk_log_size":3771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19869,"num_write_kb":255725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"61'4973","reported_seq":5094,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.455812+0000","last_change":"2024-09-19T07:49:39.455899+0000","last_active":"2024-09-19T07:49:39.455812+0000","last_peered":"2024-09-19T07:49:39.455812+0000","last_clean":"2024-09-19T07:49:39.455812+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:49:39.455812+0000","last_undegraded":"2024-09-19T07:49:39.455812+0000","last_fullsized":"2024-09-19T07:49:39.455812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4973","last_scrub_stamp":"2024-09-19T07:49:39.455603+0000","last_deep_scrub":"61'4973","last_deep_scrub_stamp":"2024-09-19T07:49:39.455603+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.455603+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:46:11.619885+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26723,"num_write_kb":320185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"65'6","reported_seq":129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.502511+0000","last_change":"2024-09-19T07:49:31.502568+0000","last_active":"2024-09-19T07:49:31.502511+0000","last_peered":"2024-09-19T07:49:31.502511+0000","last_clean":"2024-09-19T07:49:31.502511+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:49:31.502511+0000","last_undegraded":"2024-09-19T07:49:31.502511+0000","last_fullsized":"2024-09-19T07:49:31.502511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-19T07:49:31.502321+0000","last_deep_scrub":"65'6","last_deep_scrub_stamp":"2024-09-19T07:49:31.502321+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.502321+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:39:03.141550+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"64'5","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.520233+0000","last_change":"2024-09-19T07:49:26.520297+0000","last_active":"2024-09-19T07:49:26.520233+0000","last_peered":"2024-09-19T07:49:26.520233+0000","last_clean":"2024-09-19T07:49:26.520233+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:49:26.520233+0000","last_undegraded":"2024-09-19T07:49:26.520233+0000","last_fullsized":"2024-09-19T07:49:26.520233+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5","last_scrub_stamp":"2024-09-19T07:49:26.520032+0000","last_deep_scrub":"64'5","last_deep_scrub_stamp":"2024-09-19T07:49:26.520032+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.520032+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:15.048957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"64'5394","reported_seq":5501,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.382266+0000","last_change":"2024-09-19T07:19:40.394993+0000","last_active":"2024-09-19T07:48:58.382266+0000","last_peered":"2024-09-19T07:48:58.382266+0000","last_clean":"2024-09-19T07:48:58.382266+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:48:58.382266+0000","last_undegraded":"2024-09-19T07:48:58.382266+0000","last_fullsized":"2024-09-19T07:48:58.382266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5394,"log_dups_size":0,"ondisk_log_size":5394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30095,"num_write_kb":374705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"64'25","reported_seq":142,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.505869+0000","last_change":"2024-09-19T07:49:36.505929+0000","last_active":"2024-09-19T07:49:36.505869+0000","last_peered":"2024-09-19T07:49:36.505869+0000","last_clean":"2024-09-19T07:49:36.505869+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:49:36.505869+0000","last_undegraded":"2024-09-19T07:49:36.505869+0000","last_fullsized":"2024-09-19T07:49:36.505869+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'25","last_scrub_stamp":"2024-09-19T07:49:36.505667+0000","last_deep_scrub":"64'25","last_deep_scrub_stamp":"2024-09-19T07:49:36.505667+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.505667+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:17:24.384017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"61'4822","reported_seq":4943,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.186878+0000","last_change":"2024-09-19T07:49:37.186938+0000","last_active":"2024-09-19T07:49:37.186878+0000","last_peered":"2024-09-19T07:49:37.186878+0000","last_clean":"2024-09-19T07:49:37.186878+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:49:37.186878+0000","last_undegraded":"2024-09-19T07:49:37.186878+0000","last_fullsized":"2024-09-19T07:49:37.186878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4822","last_scrub_stamp":"2024-09-19T07:49:37.186673+0000","last_deep_scrub":"61'4822","last_deep_scrub_stamp":"2024-09-19T07:49:37.186673+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.186673+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:01:03.702046+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25208,"num_write_kb":326524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"61'2","reported_seq":125,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.409363+0000","last_change":"2024-09-19T07:49:37.409447+0000","last_active":"2024-09-19T07:49:37.409363+0000","last_peered":"2024-09-19T07:49:37.409363+0000","last_clean":"2024-09-19T07:49:37.409363+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:49:37.409363+0000","last_undegraded":"2024-09-19T07:49:37.409363+0000","last_fullsized":"2024-09-19T07:49:37.409363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-19T07:49:37.409161+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-19T07:49:37.409161+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.409161+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:35:07.137047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"64'4690","reported_seq":4805,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.516599+0000","last_change":"2024-09-19T07:49:38.516668+0000","last_active":"2024-09-19T07:49:38.516599+0000","last_peered":"2024-09-19T07:49:38.516599+0000","last_clean":"2024-09-19T07:49:38.516599+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:49:38.516599+0000","last_undegraded":"2024-09-19T07:49:38.516599+0000","last_fullsized":"2024-09-19T07:49:38.516599+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4690","last_scrub_stamp":"2024-09-19T07:49:38.516396+0000","last_deep_scrub":"64'4690","last_deep_scrub_stamp":"2024-09-19T07:49:38.516396+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.516396+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:16:22.493171+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25924,"num_write_kb":313037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"65'967","reported_seq":1129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.538996+0000","last_change":"2024-09-19T07:49:36.539068+0000","last_active":"2024-09-19T07:49:36.538996+0000","last_peered":"2024-09-19T07:49:36.538996+0000","last_clean":"2024-09-19T07:49:36.538996+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:49:36.538996+0000","last_undegraded":"2024-09-19T07:49:36.538996+0000","last_fullsized":"2024-09-19T07:49:36.538996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'967","last_scrub_stamp":"2024-09-19T07:49:36.538778+0000","last_deep_scrub":"65'967","last_deep_scrub_stamp":"2024-09-19T07:49:36.538778+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.538778+0000","objects_scrubbed":3,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:04.934300+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":8896,"num_write":940,"num_write_kb":4641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"61'5311","reported_seq":5424,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.362703+0000","last_change":"2024-09-19T07:19:32.277652+0000","last_active":"2024-09-19T07:48:58.362703+0000","last_peered":"2024-09-19T07:48:58.362703+0000","last_clean":"2024-09-19T07:48:58.362703+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:48:58.362703+0000","last_undegraded":"2024-09-19T07:48:58.362703+0000","last_fullsized":"2024-09-19T07:48:58.362703+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28108,"num_write_kb":333978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"61'3","reported_seq":126,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.451372+0000","last_change":"2024-09-19T07:49:38.451435+0000","last_active":"2024-09-19T07:49:38.451372+0000","last_peered":"2024-09-19T07:49:38.451372+0000","last_clean":"2024-09-19T07:49:38.451372+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:49:38.451372+0000","last_undegraded":"2024-09-19T07:49:38.451372+0000","last_fullsized":"2024-09-19T07:49:38.451372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-19T07:49:38.451165+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-19T07:49:38.451165+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.451165+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:04:06.428929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"64'5270","reported_seq":5383,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:41.125862+0000","last_change":"2024-09-19T07:49:41.125862+0000","last_active":"2024-09-19T07:49:41.125862+0000","last_peered":"2024-09-19T07:49:41.125862+0000","last_clean":"2024-09-19T07:49:41.125862+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:49:41.125862+0000","last_undegraded":"2024-09-19T07:49:41.125862+0000","last_fullsized":"2024-09-19T07:49:41.125862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5270","last_scrub_stamp":"2024-09-19T07:49:41.125538+0000","last_deep_scrub":"64'5270","last_deep_scrub_stamp":"2024-09-19T07:49:41.125538+0000","last_clean_scrub_stamp":"2024-09-19T07:49:41.125538+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:52.454340+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29270,"num_write_kb":363254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"65'416","reported_seq":545,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.518997+0000","last_change":"2024-09-19T07:49:28.518997+0000","last_active":"2024-09-19T07:49:28.518997+0000","last_peered":"2024-09-19T07:49:28.518997+0000","last_clean":"2024-09-19T07:49:28.518997+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:49:28.518997+0000","last_undegraded":"2024-09-19T07:49:28.518997+0000","last_fullsized":"2024-09-19T07:49:28.518997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'416","last_scrub_stamp":"2024-09-19T07:49:28.518729+0000","last_deep_scrub":"65'416","last_deep_scrub_stamp":"2024-09-19T07:49:28.518729+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.518729+0000","objects_scrubbed":1,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:27:59.800607+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2715831,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6193,"num_write":390,"num_write_kb":2885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"61'3569","reported_seq":3690,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:20.599569+0000","last_change":"2024-09-19T07:49:20.599569+0000","last_active":"2024-09-19T07:49:20.599569+0000","last_peered":"2024-09-19T07:49:20.599569+0000","last_clean":"2024-09-19T07:49:20.599569+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:49:20.599569+0000","last_undegraded":"2024-09-19T07:49:20.599569+0000","last_fullsized":"2024-09-19T07:49:20.599569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3569","last_scrub_stamp":"2024-09-19T07:49:20.599393+0000","last_deep_scrub":"61'3569","last_deep_scrub_stamp":"2024-09-19T07:49:20.599393+0000","last_clean_scrub_stamp":"2024-09-19T07:49:20.599393+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:56.986977+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18120,"num_write_kb":244659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'6","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.632351+0000","last_change":"2024-09-19T07:49:27.632351+0000","last_active":"2024-09-19T07:49:27.632351+0000","last_peered":"2024-09-19T07:49:27.632351+0000","last_clean":"2024-09-19T07:49:27.632351+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:49:27.632351+0000","last_undegraded":"2024-09-19T07:49:27.632351+0000","last_fullsized":"2024-09-19T07:49:27.632351+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-19T07:49:27.632080+0000","last_deep_scrub":"65'6","last_deep_scrub_stamp":"2024-09-19T07:49:27.632080+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.632080+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:32.222516+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"64'3415","reported_seq":3522,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:21.608355+0000","last_change":"2024-09-19T07:19:40.393950+0000","last_active":"2024-09-19T07:49:21.608355+0000","last_peered":"2024-09-19T07:49:21.608355+0000","last_clean":"2024-09-19T07:49:21.608355+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:49:21.608355+0000","last_undegraded":"2024-09-19T07:49:21.608355+0000","last_fullsized":"2024-09-19T07:49:21.608355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18072,"num_write_kb":240550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"65'713","reported_seq":859,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.490236+0000","last_change":"2024-09-19T07:49:37.490300+0000","last_active":"2024-09-19T07:49:37.490236+0000","last_peered":"2024-09-19T07:49:37.490236+0000","last_clean":"2024-09-19T07:49:37.490236+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:49:37.490236+0000","last_undegraded":"2024-09-19T07:49:37.490236+0000","last_fullsized":"2024-09-19T07:49:37.490236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'713","last_scrub_stamp":"2024-09-19T07:49:37.490039+0000","last_deep_scrub":"65'713","last_deep_scrub_stamp":"2024-09-19T07:49:37.490039+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.490039+0000","objects_scrubbed":1,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:01:37.072261+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":74743,"num_write":701,"num_write_kb":4298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"61'4395","reported_seq":4517,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.353982+0000","last_change":"2024-09-19T07:49:33.353982+0000","last_active":"2024-09-19T07:49:33.353982+0000","last_peered":"2024-09-19T07:49:33.353982+0000","last_clean":"2024-09-19T07:49:33.353982+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:49:33.353982+0000","last_undegraded":"2024-09-19T07:49:33.353982+0000","last_fullsized":"2024-09-19T07:49:33.353982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4395","last_scrub_stamp":"2024-09-19T07:49:33.353716+0000","last_deep_scrub":"61'4395","last_deep_scrub_stamp":"2024-09-19T07:49:33.353716+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.353716+0000","objects_scrubbed":0,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:15:28.485880+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22455,"num_write_kb":278564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'41","reported_seq":196,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.583937+0000","last_change":"2024-09-19T07:49:22.583937+0000","last_active":"2024-09-19T07:49:22.583937+0000","last_peered":"2024-09-19T07:49:22.583937+0000","last_clean":"2024-09-19T07:49:22.583937+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:49:22.583937+0000","last_undegraded":"2024-09-19T07:49:22.583937+0000","last_fullsized":"2024-09-19T07:49:22.583937+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'41","last_scrub_stamp":"2024-09-19T07:49:22.583818+0000","last_deep_scrub":"65'41","last_deep_scrub_stamp":"2024-09-19T07:49:22.583818+0000","last_clean_scrub_stamp":"2024-09-19T07:49:22.583818+0000","objects_scrubbed":1,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:00:52.174937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"64'4798","reported_seq":4904,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364338+0000","last_change":"2024-09-19T07:19:40.385581+0000","last_active":"2024-09-19T07:48:58.364338+0000","last_peered":"2024-09-19T07:48:58.364338+0000","last_clean":"2024-09-19T07:48:58.364338+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:48:58.364338+0000","last_undegraded":"2024-09-19T07:48:58.364338+0000","last_fullsized":"2024-09-19T07:48:58.364338+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27082,"num_write_kb":321689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"61'4997","reported_seq":5120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.488359+0000","last_change":"2024-09-19T07:49:26.488417+0000","last_active":"2024-09-19T07:49:26.488359+0000","last_peered":"2024-09-19T07:49:26.488359+0000","last_clean":"2024-09-19T07:49:26.488359+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:49:26.488359+0000","last_undegraded":"2024-09-19T07:49:26.488359+0000","last_fullsized":"2024-09-19T07:49:26.488359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4997","last_scrub_stamp":"2024-09-19T07:49:26.488156+0000","last_deep_scrub":"61'4997","last_deep_scrub_stamp":"2024-09-19T07:49:26.488156+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.488156+0000","objects_scrubbed":0,"log_size":4997,"log_dups_size":0,"ondisk_log_size":4997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:48:53.074284+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26206,"num_write_kb":309941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"64'5490","reported_seq":5605,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.150680+0000","last_change":"2024-09-19T07:49:39.150742+0000","last_active":"2024-09-19T07:49:39.150680+0000","last_peered":"2024-09-19T07:49:39.150680+0000","last_clean":"2024-09-19T07:49:39.150680+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:49:39.150680+0000","last_undegraded":"2024-09-19T07:49:39.150680+0000","last_fullsized":"2024-09-19T07:49:39.150680+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5490","last_scrub_stamp":"2024-09-19T07:49:39.149819+0000","last_deep_scrub":"64'5490","last_deep_scrub_stamp":"2024-09-19T07:49:39.149819+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.149819+0000","objects_scrubbed":0,"log_size":5490,"log_dups_size":0,"ondisk_log_size":5490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:19:16.579715+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29613,"num_write_kb":364653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"61'4890","reported_seq":5014,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.718723+0000","last_change":"2024-09-19T07:49:25.718783+0000","last_active":"2024-09-19T07:49:25.718723+0000","last_peered":"2024-09-19T07:49:25.718723+0000","last_clean":"2024-09-19T07:49:25.718723+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:49:25.718723+0000","last_undegraded":"2024-09-19T07:49:25.718723+0000","last_fullsized":"2024-09-19T07:49:25.718723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4890","last_scrub_stamp":"2024-09-19T07:49:25.718515+0000","last_deep_scrub":"61'4890","last_deep_scrub_stamp":"2024-09-19T07:49:25.718515+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.718515+0000","objects_scrubbed":0,"log_size":4890,"log_dups_size":0,"ondisk_log_size":4890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:21:01.552830+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25338,"num_write_kb":314559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"64'4569","reported_seq":4676,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366640+0000","last_change":"2024-09-19T07:19:40.381262+0000","last_active":"2024-09-19T07:48:58.366640+0000","last_peered":"2024-09-19T07:48:58.366640+0000","last_clean":"2024-09-19T07:48:58.366640+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:48:58.366640+0000","last_undegraded":"2024-09-19T07:48:58.366640+0000","last_fullsized":"2024-09-19T07:48:58.366640+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25275,"num_write_kb":314704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"61'5227","reported_seq":5344,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367575+0000","last_change":"2024-09-19T07:19:32.275719+0000","last_active":"2024-09-19T07:48:58.367575+0000","last_peered":"2024-09-19T07:48:58.367575+0000","last_clean":"2024-09-19T07:48:58.367575+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:48:58.367575+0000","last_undegraded":"2024-09-19T07:48:58.367575+0000","last_fullsized":"2024-09-19T07:48:58.367575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27444,"num_write_kb":331433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"64'4939","reported_seq":5045,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373610+0000","last_change":"2024-09-19T07:19:40.386432+0000","last_active":"2024-09-19T07:48:58.373610+0000","last_peered":"2024-09-19T07:48:58.373610+0000","last_clean":"2024-09-19T07:48:58.373610+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:48:58.373610+0000","last_undegraded":"2024-09-19T07:48:58.373610+0000","last_fullsized":"2024-09-19T07:48:58.373610+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27263,"num_write_kb":334786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"61'4385","reported_seq":4509,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.602889+0000","last_change":"2024-09-19T07:49:27.602949+0000","last_active":"2024-09-19T07:49:27.602889+0000","last_peered":"2024-09-19T07:49:27.602889+0000","last_clean":"2024-09-19T07:49:27.602889+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:49:27.602889+0000","last_undegraded":"2024-09-19T07:49:27.602889+0000","last_fullsized":"2024-09-19T07:49:27.602889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4385","last_scrub_stamp":"2024-09-19T07:49:27.602677+0000","last_deep_scrub":"61'4385","last_deep_scrub_stamp":"2024-09-19T07:49:27.602677+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.602677+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:12.330784+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22838,"num_write_kb":271812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"64'4859","reported_seq":4968,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372905+0000","last_change":"2024-09-19T07:19:40.378630+0000","last_active":"2024-09-19T07:48:58.372905+0000","last_peered":"2024-09-19T07:48:58.372905+0000","last_clean":"2024-09-19T07:48:58.372905+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:48:58.372905+0000","last_undegraded":"2024-09-19T07:48:58.372905+0000","last_fullsized":"2024-09-19T07:48:58.372905+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26164,"num_write_kb":338417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"61'4483","reported_seq":4606,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.536068+0000","last_change":"2024-09-19T07:49:25.536188+0000","last_active":"2024-09-19T07:49:25.536068+0000","last_peered":"2024-09-19T07:49:25.536068+0000","last_clean":"2024-09-19T07:49:25.536068+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:49:25.536068+0000","last_undegraded":"2024-09-19T07:49:25.536068+0000","last_fullsized":"2024-09-19T07:49:25.536068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4483","last_scrub_stamp":"2024-09-19T07:49:25.535753+0000","last_deep_scrub":"61'4483","last_deep_scrub_stamp":"2024-09-19T07:49:25.535753+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.535753+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:53.931275+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23870,"num_write_kb":292652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"64'4105","reported_seq":4221,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.471961+0000","last_change":"2024-09-19T07:49:38.472046+0000","last_active":"2024-09-19T07:49:38.471961+0000","last_peered":"2024-09-19T07:49:38.471961+0000","last_clean":"2024-09-19T07:49:38.471961+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:49:38.471961+0000","last_undegraded":"2024-09-19T07:49:38.471961+0000","last_fullsized":"2024-09-19T07:49:38.471961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4105","last_scrub_stamp":"2024-09-19T07:49:38.471730+0000","last_deep_scrub":"64'4105","last_deep_scrub_stamp":"2024-09-19T07:49:38.471730+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.471730+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:17:03.273001+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21923,"num_write_kb":284540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"61'5078","reported_seq":5200,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.144855+0000","last_change":"2024-09-19T07:49:38.144914+0000","last_active":"2024-09-19T07:49:38.144855+0000","last_peered":"2024-09-19T07:49:38.144855+0000","last_clean":"2024-09-19T07:49:38.144855+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:49:38.144855+0000","last_undegraded":"2024-09-19T07:49:38.144855+0000","last_fullsized":"2024-09-19T07:49:38.144855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5078","last_scrub_stamp":"2024-09-19T07:49:38.144648+0000","last_deep_scrub":"61'5078","last_deep_scrub_stamp":"2024-09-19T07:49:38.144648+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.144648+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:50:36.299270+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28059,"num_write_kb":313378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"64'3824","reported_seq":3931,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372543+0000","last_change":"2024-09-19T07:19:40.387116+0000","last_active":"2024-09-19T07:48:58.372543+0000","last_peered":"2024-09-19T07:48:58.372543+0000","last_clean":"2024-09-19T07:48:58.372543+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:48:58.372543+0000","last_undegraded":"2024-09-19T07:48:58.372543+0000","last_fullsized":"2024-09-19T07:48:58.372543+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20622,"num_write_kb":295649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"61'4725","reported_seq":4847,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.525724+0000","last_change":"2024-09-19T07:49:33.525791+0000","last_active":"2024-09-19T07:49:33.525724+0000","last_peered":"2024-09-19T07:49:33.525724+0000","last_clean":"2024-09-19T07:49:33.525724+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:49:33.525724+0000","last_undegraded":"2024-09-19T07:49:33.525724+0000","last_fullsized":"2024-09-19T07:49:33.525724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4725","last_scrub_stamp":"2024-09-19T07:49:33.525504+0000","last_deep_scrub":"61'4725","last_deep_scrub_stamp":"2024-09-19T07:49:33.525504+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.525504+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:06:32.819964+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25170,"num_write_kb":293477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"64'4764","reported_seq":4880,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.523504+0000","last_change":"2024-09-19T07:49:37.523566+0000","last_active":"2024-09-19T07:49:37.523504+0000","last_peered":"2024-09-19T07:49:37.523504+0000","last_clean":"2024-09-19T07:49:37.523504+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:49:37.523504+0000","last_undegraded":"2024-09-19T07:49:37.523504+0000","last_fullsized":"2024-09-19T07:49:37.523504+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4764","last_scrub_stamp":"2024-09-19T07:49:37.523305+0000","last_deep_scrub":"64'4764","last_deep_scrub_stamp":"2024-09-19T07:49:37.523305+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.523305+0000","objects_scrubbed":0,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:43:49.209515+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25208,"num_write_kb":309995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"61'5411","reported_seq":5534,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.498406+0000","last_change":"2024-09-19T07:49:31.498537+0000","last_active":"2024-09-19T07:49:31.498406+0000","last_peered":"2024-09-19T07:49:31.498406+0000","last_clean":"2024-09-19T07:49:31.498406+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:49:31.498406+0000","last_undegraded":"2024-09-19T07:49:31.498406+0000","last_fullsized":"2024-09-19T07:49:31.498406+0000","mapping_epoch":19,"log_start":"54'200","ondisk_log_start":"54'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5411","last_scrub_stamp":"2024-09-19T07:49:31.498168+0000","last_deep_scrub":"61'5411","last_deep_scrub_stamp":"2024-09-19T07:49:31.498168+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.498168+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:09:26.921508+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28760,"num_write_kb":348949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"64'4221","reported_seq":4327,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363552+0000","last_change":"2024-09-19T07:19:40.388597+0000","last_active":"2024-09-19T07:48:58.363552+0000","last_peered":"2024-09-19T07:48:58.363552+0000","last_clean":"2024-09-19T07:48:58.363552+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:48:58.363552+0000","last_undegraded":"2024-09-19T07:48:58.363552+0000","last_fullsized":"2024-09-19T07:48:58.363552+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21805,"num_write_kb":288249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"61'4974","reported_seq":5094,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.366637+0000","last_change":"2024-09-19T07:20:13.954029+0000","last_active":"2024-09-19T07:48:58.366637+0000","last_peered":"2024-09-19T07:48:58.366637+0000","last_clean":"2024-09-19T07:48:58.366637+0000","last_became_active":"2024-09-19T07:20:13.953118+0000","last_became_peered":"2024-09-19T07:20:13.953118+0000","last_unstale":"2024-09-19T07:48:58.366637+0000","last_undegraded":"2024-09-19T07:48:58.366637+0000","last_fullsized":"2024-09-19T07:48:58.366637+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25995,"num_write_kb":309048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"64'5715","reported_seq":5820,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375617+0000","last_change":"2024-09-19T07:19:40.383779+0000","last_active":"2024-09-19T07:48:58.375617+0000","last_peered":"2024-09-19T07:48:58.375617+0000","last_clean":"2024-09-19T07:48:58.375617+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:48:58.375617+0000","last_undegraded":"2024-09-19T07:48:58.375617+0000","last_fullsized":"2024-09-19T07:48:58.375617+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5715,"log_dups_size":0,"ondisk_log_size":5715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30021,"num_write_kb":374301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"61'4405","reported_seq":4527,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.494869+0000","last_change":"2024-09-19T07:49:27.495026+0000","last_active":"2024-09-19T07:49:27.494869+0000","last_peered":"2024-09-19T07:49:27.494869+0000","last_clean":"2024-09-19T07:49:27.494869+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:49:27.494869+0000","last_undegraded":"2024-09-19T07:49:27.494869+0000","last_fullsized":"2024-09-19T07:49:27.494869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4405","last_scrub_stamp":"2024-09-19T07:49:27.494549+0000","last_deep_scrub":"61'4405","last_deep_scrub_stamp":"2024-09-19T07:49:27.494549+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.494549+0000","objects_scrubbed":0,"log_size":4405,"log_dups_size":0,"ondisk_log_size":4405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:43.771195+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23742,"num_write_kb":275461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"64'3931","reported_seq":4045,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.673565+0000","last_change":"2024-09-19T07:49:29.673627+0000","last_active":"2024-09-19T07:49:29.673565+0000","last_peered":"2024-09-19T07:49:29.673565+0000","last_clean":"2024-09-19T07:49:29.673565+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:49:29.673565+0000","last_undegraded":"2024-09-19T07:49:29.673565+0000","last_fullsized":"2024-09-19T07:49:29.673565+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3931","last_scrub_stamp":"2024-09-19T07:49:29.673363+0000","last_deep_scrub":"64'3931","last_deep_scrub_stamp":"2024-09-19T07:49:29.673363+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.673363+0000","objects_scrubbed":0,"log_size":3931,"log_dups_size":0,"ondisk_log_size":3931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:24:17.132608+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22002,"num_write_kb":260466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"61'4975","reported_seq":5097,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.451782+0000","last_change":"2024-09-19T07:49:31.451897+0000","last_active":"2024-09-19T07:49:31.451782+0000","last_peered":"2024-09-19T07:49:31.451782+0000","last_clean":"2024-09-19T07:49:31.451782+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:49:31.451782+0000","last_undegraded":"2024-09-19T07:49:31.451782+0000","last_fullsized":"2024-09-19T07:49:31.451782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4975","last_scrub_stamp":"2024-09-19T07:49:31.451571+0000","last_deep_scrub":"61'4975","last_deep_scrub_stamp":"2024-09-19T07:49:31.451571+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.451571+0000","objects_scrubbed":0,"log_size":4975,"log_dups_size":0,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:16:31.212971+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26500,"num_write_kb":311194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"64'5003","reported_seq":5117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.543370+0000","last_change":"2024-09-19T07:49:34.543501+0000","last_active":"2024-09-19T07:49:34.543370+0000","last_peered":"2024-09-19T07:49:34.543370+0000","last_clean":"2024-09-19T07:49:34.543370+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:49:34.543370+0000","last_undegraded":"2024-09-19T07:49:34.543370+0000","last_fullsized":"2024-09-19T07:49:34.543370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5003","last_scrub_stamp":"2024-09-19T07:49:34.543108+0000","last_deep_scrub":"64'5003","last_deep_scrub_stamp":"2024-09-19T07:49:34.543108+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.543108+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:19:11.973004+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26890,"num_write_kb":325027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"61'5877","reported_seq":6001,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.556263+0000","last_change":"2024-09-19T07:49:22.556294+0000","last_active":"2024-09-19T07:49:22.556263+0000","last_peered":"2024-09-19T07:49:22.556263+0000","last_clean":"2024-09-19T07:49:22.556263+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:49:22.556263+0000","last_undegraded":"2024-09-19T07:49:22.556263+0000","last_fullsized":"2024-09-19T07:49:22.556263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5877","last_scrub_stamp":"2024-09-19T07:49:22.556138+0000","last_deep_scrub":"61'5877","last_deep_scrub_stamp":"2024-09-19T07:49:22.556138+0000","last_clean_scrub_stamp":"2024-09-19T07:49:22.556138+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:02:01.734204+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":30970,"num_write_kb":370397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"64'4684","reported_seq":4790,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364439+0000","last_change":"2024-09-19T07:19:40.390205+0000","last_active":"2024-09-19T07:48:58.364439+0000","last_peered":"2024-09-19T07:48:58.364439+0000","last_clean":"2024-09-19T07:48:58.364439+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:48:58.364439+0000","last_undegraded":"2024-09-19T07:48:58.364439+0000","last_fullsized":"2024-09-19T07:48:58.364439+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25566,"num_write_kb":331994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"61'4261","reported_seq":4383,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.429556+0000","last_change":"2024-09-19T07:49:23.429674+0000","last_active":"2024-09-19T07:49:23.429556+0000","last_peered":"2024-09-19T07:49:23.429556+0000","last_clean":"2024-09-19T07:49:23.429556+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:49:23.429556+0000","last_undegraded":"2024-09-19T07:49:23.429556+0000","last_fullsized":"2024-09-19T07:49:23.429556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4261","last_scrub_stamp":"2024-09-19T07:49:23.429282+0000","last_deep_scrub":"61'4261","last_deep_scrub_stamp":"2024-09-19T07:49:23.429282+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.429282+0000","objects_scrubbed":0,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:37:47.705458+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22573,"num_write_kb":274878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"64'5240","reported_seq":5348,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363810+0000","last_change":"2024-09-19T07:19:40.390559+0000","last_active":"2024-09-19T07:48:58.363810+0000","last_peered":"2024-09-19T07:48:58.363810+0000","last_clean":"2024-09-19T07:48:58.363810+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:48:58.363810+0000","last_undegraded":"2024-09-19T07:48:58.363810+0000","last_fullsized":"2024-09-19T07:48:58.363810+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":28154,"num_write_kb":354961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"61'5095","reported_seq":5217,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.330890+0000","last_change":"2024-09-19T07:49:34.330949+0000","last_active":"2024-09-19T07:49:34.330890+0000","last_peered":"2024-09-19T07:49:34.330890+0000","last_clean":"2024-09-19T07:49:34.330890+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:49:34.330890+0000","last_undegraded":"2024-09-19T07:49:34.330890+0000","last_fullsized":"2024-09-19T07:49:34.330890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5095","last_scrub_stamp":"2024-09-19T07:49:34.330690+0000","last_deep_scrub":"61'5095","last_deep_scrub_stamp":"2024-09-19T07:49:34.330690+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.330690+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:09:29.754030+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26748,"num_write_kb":324403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"64'4078","reported_seq":4193,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.560670+0000","last_change":"2024-09-19T07:49:28.560736+0000","last_active":"2024-09-19T07:49:28.560670+0000","last_peered":"2024-09-19T07:49:28.560670+0000","last_clean":"2024-09-19T07:49:28.560670+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:49:28.560670+0000","last_undegraded":"2024-09-19T07:49:28.560670+0000","last_fullsized":"2024-09-19T07:49:28.560670+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4078","last_scrub_stamp":"2024-09-19T07:49:28.560467+0000","last_deep_scrub":"64'4078","last_deep_scrub_stamp":"2024-09-19T07:49:28.560467+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.560467+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:39:49.769342+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22110,"num_write_kb":293452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"61'4908","reported_seq":5030,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.678274+0000","last_change":"2024-09-19T07:49:26.678335+0000","last_active":"2024-09-19T07:49:26.678274+0000","last_peered":"2024-09-19T07:49:26.678274+0000","last_clean":"2024-09-19T07:49:26.678274+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:49:26.678274+0000","last_undegraded":"2024-09-19T07:49:26.678274+0000","last_fullsized":"2024-09-19T07:49:26.678274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4908","last_scrub_stamp":"2024-09-19T07:49:26.678074+0000","last_deep_scrub":"61'4908","last_deep_scrub_stamp":"2024-09-19T07:49:26.678074+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.678074+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:51.615841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25574,"num_write_kb":315713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"64'3983","reported_seq":4097,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.498845+0000","last_change":"2024-09-19T07:49:39.498907+0000","last_active":"2024-09-19T07:49:39.498845+0000","last_peered":"2024-09-19T07:49:39.498845+0000","last_clean":"2024-09-19T07:49:39.498845+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:49:39.498845+0000","last_undegraded":"2024-09-19T07:49:39.498845+0000","last_fullsized":"2024-09-19T07:49:39.498845+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3983","last_scrub_stamp":"2024-09-19T07:49:39.498538+0000","last_deep_scrub":"64'3983","last_deep_scrub_stamp":"2024-09-19T07:49:39.498538+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.498538+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:29:38.865483+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21503,"num_write_kb":274721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"61'5975","reported_seq":6096,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.515361+0000","last_change":"2024-09-19T07:49:29.515423+0000","last_active":"2024-09-19T07:49:29.515361+0000","last_peered":"2024-09-19T07:49:29.515361+0000","last_clean":"2024-09-19T07:49:29.515361+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:49:29.515361+0000","last_undegraded":"2024-09-19T07:49:29.515361+0000","last_fullsized":"2024-09-19T07:49:29.515361+0000","mapping_epoch":19,"log_start":"53'200","ondisk_log_start":"53'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5975","last_scrub_stamp":"2024-09-19T07:49:29.515156+0000","last_deep_scrub":"61'5975","last_deep_scrub_stamp":"2024-09-19T07:49:29.515156+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.515156+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:24:00.638951+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31472,"num_write_kb":384213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"64'4930","reported_seq":5035,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.373529+0000","last_change":"2024-09-19T07:19:40.387341+0000","last_active":"2024-09-19T07:48:58.373529+0000","last_peered":"2024-09-19T07:48:58.373529+0000","last_clean":"2024-09-19T07:48:58.373529+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:48:58.373529+0000","last_undegraded":"2024-09-19T07:48:58.373529+0000","last_fullsized":"2024-09-19T07:48:58.373529+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26807,"num_write_kb":336825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"61'5543","reported_seq":5664,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.436075+0000","last_change":"2024-09-19T07:49:33.436157+0000","last_active":"2024-09-19T07:49:33.436075+0000","last_peered":"2024-09-19T07:49:33.436075+0000","last_clean":"2024-09-19T07:49:33.436075+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:49:33.436075+0000","last_undegraded":"2024-09-19T07:49:33.436075+0000","last_fullsized":"2024-09-19T07:49:33.436075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5543","last_scrub_stamp":"2024-09-19T07:49:33.435855+0000","last_deep_scrub":"61'5543","last_deep_scrub_stamp":"2024-09-19T07:49:33.435855+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.435855+0000","objects_scrubbed":0,"log_size":5543,"log_dups_size":0,"ondisk_log_size":5543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:49.685098+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29751,"num_write_kb":336461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"64'5766","reported_seq":5881,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.456958+0000","last_change":"2024-09-19T07:49:40.457080+0000","last_active":"2024-09-19T07:49:40.456958+0000","last_peered":"2024-09-19T07:49:40.456958+0000","last_clean":"2024-09-19T07:49:40.456958+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:49:40.456958+0000","last_undegraded":"2024-09-19T07:49:40.456958+0000","last_fullsized":"2024-09-19T07:49:40.456958+0000","mapping_epoch":23,"log_start":"58'300","ondisk_log_start":"58'300","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5766","last_scrub_stamp":"2024-09-19T07:49:40.456692+0000","last_deep_scrub":"64'5766","last_deep_scrub_stamp":"2024-09-19T07:49:40.456692+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.456692+0000","objects_scrubbed":0,"log_size":5466,"log_dups_size":0,"ondisk_log_size":5466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:29:06.554576+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32274,"num_write_kb":395900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"61'4235","reported_seq":4356,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.219232+0000","last_change":"2024-09-19T07:49:24.219358+0000","last_active":"2024-09-19T07:49:24.219232+0000","last_peered":"2024-09-19T07:49:24.219232+0000","last_clean":"2024-09-19T07:49:24.219232+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:49:24.219232+0000","last_undegraded":"2024-09-19T07:49:24.219232+0000","last_fullsized":"2024-09-19T07:49:24.219232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4235","last_scrub_stamp":"2024-09-19T07:49:24.218980+0000","last_deep_scrub":"61'4235","last_deep_scrub_stamp":"2024-09-19T07:49:24.218980+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.218980+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:38:08.256928+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22758,"num_write_kb":275574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"64'3417","reported_seq":3522,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.365981+0000","last_change":"2024-09-19T07:19:40.367578+0000","last_active":"2024-09-19T07:48:58.365981+0000","last_peered":"2024-09-19T07:48:58.365981+0000","last_clean":"2024-09-19T07:48:58.365981+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:48:58.365981+0000","last_undegraded":"2024-09-19T07:48:58.365981+0000","last_fullsized":"2024-09-19T07:48:58.365981+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18157,"num_write_kb":232641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"61'4616","reported_seq":4737,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.591381+0000","last_change":"2024-09-19T07:49:31.591446+0000","last_active":"2024-09-19T07:49:31.591381+0000","last_peered":"2024-09-19T07:49:31.591381+0000","last_clean":"2024-09-19T07:49:31.591381+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:49:31.591381+0000","last_undegraded":"2024-09-19T07:49:31.591381+0000","last_fullsized":"2024-09-19T07:49:31.591381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4616","last_scrub_stamp":"2024-09-19T07:49:31.591172+0000","last_deep_scrub":"61'4616","last_deep_scrub_stamp":"2024-09-19T07:49:31.591172+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.591172+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:50:16.285682+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24409,"num_write_kb":298784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"64'4320","reported_seq":4425,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.374968+0000","last_change":"2024-09-19T07:19:40.385641+0000","last_active":"2024-09-19T07:48:58.374968+0000","last_peered":"2024-09-19T07:48:58.374968+0000","last_clean":"2024-09-19T07:48:58.374968+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:48:58.374968+0000","last_undegraded":"2024-09-19T07:48:58.374968+0000","last_fullsized":"2024-09-19T07:48:58.374968+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22755,"num_write_kb":287920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"61'4140","reported_seq":4262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.488288+0000","last_change":"2024-09-19T07:49:35.488347+0000","last_active":"2024-09-19T07:49:35.488288+0000","last_peered":"2024-09-19T07:49:35.488288+0000","last_clean":"2024-09-19T07:49:35.488288+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:49:35.488288+0000","last_undegraded":"2024-09-19T07:49:35.488288+0000","last_fullsized":"2024-09-19T07:49:35.488288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4140","last_scrub_stamp":"2024-09-19T07:49:35.488085+0000","last_deep_scrub":"61'4140","last_deep_scrub_stamp":"2024-09-19T07:49:35.488085+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.488085+0000","objects_scrubbed":0,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:57:54.215625+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22395,"num_write_kb":253944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"64'3972","reported_seq":4077,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.364644+0000","last_change":"2024-09-19T07:19:40.367101+0000","last_active":"2024-09-19T07:48:58.364644+0000","last_peered":"2024-09-19T07:48:58.364644+0000","last_clean":"2024-09-19T07:48:58.364644+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:48:58.364644+0000","last_undegraded":"2024-09-19T07:48:58.364644+0000","last_fullsized":"2024-09-19T07:48:58.364644+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21876,"num_write_kb":260064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"61'3526","reported_seq":3647,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.350219+0000","last_change":"2024-09-19T07:49:34.350279+0000","last_active":"2024-09-19T07:49:34.350219+0000","last_peered":"2024-09-19T07:49:34.350219+0000","last_clean":"2024-09-19T07:49:34.350219+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:49:34.350219+0000","last_undegraded":"2024-09-19T07:49:34.350219+0000","last_fullsized":"2024-09-19T07:49:34.350219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3526","last_scrub_stamp":"2024-09-19T07:49:34.350012+0000","last_deep_scrub":"61'3526","last_deep_scrub_stamp":"2024-09-19T07:49:34.350012+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.350012+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:36:56.272224+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18762,"num_write_kb":235013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"64'3609","reported_seq":3722,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.622522+0000","last_change":"2024-09-19T07:49:34.622582+0000","last_active":"2024-09-19T07:49:34.622522+0000","last_peered":"2024-09-19T07:49:34.622522+0000","last_clean":"2024-09-19T07:49:34.622522+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:49:34.622522+0000","last_undegraded":"2024-09-19T07:49:34.622522+0000","last_fullsized":"2024-09-19T07:49:34.622522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3609","last_scrub_stamp":"2024-09-19T07:49:34.622304+0000","last_deep_scrub":"64'3609","last_deep_scrub_stamp":"2024-09-19T07:49:34.622304+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.622304+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:39:06.261533+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20332,"num_write_kb":258088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"61'4660","reported_seq":4778,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.381340+0000","last_change":"2024-09-19T07:20:13.952931+0000","last_active":"2024-09-19T07:48:58.381340+0000","last_peered":"2024-09-19T07:48:58.381340+0000","last_clean":"2024-09-19T07:48:58.381340+0000","last_became_active":"2024-09-19T07:20:13.952103+0000","last_became_peered":"2024-09-19T07:20:13.952103+0000","last_unstale":"2024-09-19T07:48:58.381340+0000","last_undegraded":"2024-09-19T07:48:58.381340+0000","last_fullsized":"2024-09-19T07:48:58.381340+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:31.227673+0000","last_clean_scrub_stamp":"2024-09-19T07:19:31.227673+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24375,"num_write_kb":290368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"64'3991","reported_seq":4096,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.375855+0000","last_change":"2024-09-19T07:19:40.387401+0000","last_active":"2024-09-19T07:48:58.375855+0000","last_peered":"2024-09-19T07:48:58.375855+0000","last_clean":"2024-09-19T07:48:58.375855+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:48:58.375855+0000","last_undegraded":"2024-09-19T07:48:58.375855+0000","last_fullsized":"2024-09-19T07:48:58.375855+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3991,"log_dups_size":0,"ondisk_log_size":3991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21657,"num_write_kb":287665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"61'5448","reported_seq":5569,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.499325+0000","last_change":"2024-09-19T07:49:32.499445+0000","last_active":"2024-09-19T07:49:32.499325+0000","last_peered":"2024-09-19T07:49:32.499325+0000","last_clean":"2024-09-19T07:49:32.499325+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:49:32.499325+0000","last_undegraded":"2024-09-19T07:49:32.499325+0000","last_fullsized":"2024-09-19T07:49:32.499325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5448","last_scrub_stamp":"2024-09-19T07:49:32.499060+0000","last_deep_scrub":"61'5448","last_deep_scrub_stamp":"2024-09-19T07:49:32.499060+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.499060+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:56:11.424517+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29399,"num_write_kb":347457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"64'4539","reported_seq":4652,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.500615+0000","last_change":"2024-09-19T07:49:30.500671+0000","last_active":"2024-09-19T07:49:30.500615+0000","last_peered":"2024-09-19T07:49:30.500615+0000","last_clean":"2024-09-19T07:49:30.500615+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:49:30.500615+0000","last_undegraded":"2024-09-19T07:49:30.500615+0000","last_fullsized":"2024-09-19T07:49:30.500615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4539","last_scrub_stamp":"2024-09-19T07:49:30.500418+0000","last_deep_scrub":"64'4539","last_deep_scrub_stamp":"2024-09-19T07:49:30.500418+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.500418+0000","objects_scrubbed":0,"log_size":4539,"log_dups_size":0,"ondisk_log_size":4539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:42:26.609996+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24944,"num_write_kb":313178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"61'6280","reported_seq":6403,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.317119+0000","last_change":"2024-09-19T07:49:30.317185+0000","last_active":"2024-09-19T07:49:30.317119+0000","last_peered":"2024-09-19T07:49:30.317119+0000","last_clean":"2024-09-19T07:49:30.317119+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:49:30.317119+0000","last_undegraded":"2024-09-19T07:49:30.317119+0000","last_fullsized":"2024-09-19T07:49:30.317119+0000","mapping_epoch":19,"log_start":"54'500","ondisk_log_start":"54'500","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'6280","last_scrub_stamp":"2024-09-19T07:49:30.316901+0000","last_deep_scrub":"61'6280","last_deep_scrub_stamp":"2024-09-19T07:49:30.316901+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.316901+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:46.593547+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":34141,"num_write_kb":395176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"64'5507","reported_seq":5614,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.363438+0000","last_change":"2024-09-19T07:19:40.378172+0000","last_active":"2024-09-19T07:48:58.363438+0000","last_peered":"2024-09-19T07:48:58.363438+0000","last_clean":"2024-09-19T07:48:58.363438+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:48:58.363438+0000","last_undegraded":"2024-09-19T07:48:58.363438+0000","last_fullsized":"2024-09-19T07:48:58.363438+0000","mapping_epoch":23,"log_start":"58'300","ondisk_log_start":"58'300","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30086,"num_write_kb":370527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"61'4537","reported_seq":4658,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.480933+0000","last_change":"2024-09-19T07:49:40.480998+0000","last_active":"2024-09-19T07:49:40.480933+0000","last_peered":"2024-09-19T07:49:40.480933+0000","last_clean":"2024-09-19T07:49:40.480933+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:49:40.480933+0000","last_undegraded":"2024-09-19T07:49:40.480933+0000","last_fullsized":"2024-09-19T07:49:40.480933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4537","last_scrub_stamp":"2024-09-19T07:49:40.480727+0000","last_deep_scrub":"61'4537","last_deep_scrub_stamp":"2024-09-19T07:49:40.480727+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.480727+0000","objects_scrubbed":0,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:22.886355+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24234,"num_write_kb":310307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"64'4532","reported_seq":4648,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.500610+0000","last_change":"2024-09-19T07:49:38.500678+0000","last_active":"2024-09-19T07:49:38.500610+0000","last_peered":"2024-09-19T07:49:38.500610+0000","last_clean":"2024-09-19T07:49:38.500610+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:49:38.500610+0000","last_undegraded":"2024-09-19T07:49:38.500610+0000","last_fullsized":"2024-09-19T07:49:38.500610+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4532","last_scrub_stamp":"2024-09-19T07:49:38.500398+0000","last_deep_scrub":"64'4532","last_deep_scrub_stamp":"2024-09-19T07:49:38.500398+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.500398+0000","objects_scrubbed":0,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:37:00.422610+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25325,"num_write_kb":306661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"61'4832","reported_seq":4953,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.209895+0000","last_change":"2024-09-19T07:49:24.209895+0000","last_active":"2024-09-19T07:49:24.209895+0000","last_peered":"2024-09-19T07:49:24.209895+0000","last_clean":"2024-09-19T07:49:24.209895+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:49:24.209895+0000","last_undegraded":"2024-09-19T07:49:24.209895+0000","last_fullsized":"2024-09-19T07:49:24.209895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4832","last_scrub_stamp":"2024-09-19T07:49:24.209617+0000","last_deep_scrub":"61'4832","last_deep_scrub_stamp":"2024-09-19T07:49:24.209617+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.209617+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:56:03.135074+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26222,"num_write_kb":306099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"64'3448","reported_seq":3556,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.372093+0000","last_change":"2024-09-19T07:19:40.396146+0000","last_active":"2024-09-19T07:48:58.372093+0000","last_peered":"2024-09-19T07:48:58.372093+0000","last_clean":"2024-09-19T07:48:58.372093+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:48:58.372093+0000","last_undegraded":"2024-09-19T07:48:58.372093+0000","last_fullsized":"2024-09-19T07:48:58.372093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3448,"log_dups_size":0,"ondisk_log_size":3448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18660,"num_write_kb":230569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"61'4317","reported_seq":4438,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.544477+0000","last_change":"2024-09-19T07:49:25.544477+0000","last_active":"2024-09-19T07:49:25.544477+0000","last_peered":"2024-09-19T07:49:25.544477+0000","last_clean":"2024-09-19T07:49:25.544477+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:49:25.544477+0000","last_undegraded":"2024-09-19T07:49:25.544477+0000","last_fullsized":"2024-09-19T07:49:25.544477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4317","last_scrub_stamp":"2024-09-19T07:49:25.543727+0000","last_deep_scrub":"61'4317","last_deep_scrub_stamp":"2024-09-19T07:49:25.543727+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.543727+0000","objects_scrubbed":0,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:50:47.452710+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22609,"num_write_kb":267014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"64'3912","reported_seq":4020,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:48:58.367756+0000","last_change":"2024-09-19T07:19:40.373552+0000","last_active":"2024-09-19T07:48:58.367756+0000","last_peered":"2024-09-19T07:48:58.367756+0000","last_clean":"2024-09-19T07:48:58.367756+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:48:58.367756+0000","last_undegraded":"2024-09-19T07:48:58.367756+0000","last_fullsized":"2024-09-19T07:48:58.367756+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T07:19:39.378602+0000","last_clean_scrub_stamp":"2024-09-19T07:19:39.378602+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21554,"num_write_kb":270790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"61'4720","reported_seq":4841,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.346538+0000","last_change":"2024-09-19T07:49:32.346538+0000","last_active":"2024-09-19T07:49:32.346538+0000","last_peered":"2024-09-19T07:49:32.346538+0000","last_clean":"2024-09-19T07:49:32.346538+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:49:32.346538+0000","last_undegraded":"2024-09-19T07:49:32.346538+0000","last_fullsized":"2024-09-19T07:49:32.346538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4720","last_scrub_stamp":"2024-09-19T07:49:32.346047+0000","last_deep_scrub":"61'4720","last_deep_scrub_stamp":"2024-09-19T07:49:32.346047+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.346047+0000","objects_scrubbed":0,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:25.061148+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24228,"num_write_kb":289919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"64'3227","reported_seq":3340,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.478502+0000","last_change":"2024-09-19T07:49:32.478502+0000","last_active":"2024-09-19T07:49:32.478502+0000","last_peered":"2024-09-19T07:49:32.478502+0000","last_clean":"2024-09-19T07:49:32.478502+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:49:32.478502+0000","last_undegraded":"2024-09-19T07:49:32.478502+0000","last_fullsized":"2024-09-19T07:49:32.478502+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3227","last_scrub_stamp":"2024-09-19T07:49:32.478236+0000","last_deep_scrub":"64'3227","last_deep_scrub_stamp":"2024-09-19T07:49:32.478236+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.478236+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:00:58.993608+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17253,"num_write_kb":237102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"61'4326","reported_seq":4450,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.550465+0000","last_change":"2024-09-19T07:49:27.550528+0000","last_active":"2024-09-19T07:49:27.550465+0000","last_peered":"2024-09-19T07:49:27.550465+0000","last_clean":"2024-09-19T07:49:27.550465+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:49:27.550465+0000","last_undegraded":"2024-09-19T07:49:27.550465+0000","last_fullsized":"2024-09-19T07:49:27.550465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4326","last_scrub_stamp":"2024-09-19T07:49:27.550257+0000","last_deep_scrub":"61'4326","last_deep_scrub_stamp":"2024-09-19T07:49:27.550257+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.550257+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:57.720889+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23399,"num_write_kb":274225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"64'3524","reported_seq":3637,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.675251+0000","last_change":"2024-09-19T07:49:30.675251+0000","last_active":"2024-09-19T07:49:30.675251+0000","last_peered":"2024-09-19T07:49:30.675251+0000","last_clean":"2024-09-19T07:49:30.675251+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:49:30.675251+0000","last_undegraded":"2024-09-19T07:49:30.675251+0000","last_fullsized":"2024-09-19T07:49:30.675251+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3524","last_scrub_stamp":"2024-09-19T07:49:30.674832+0000","last_deep_scrub":"64'3524","last_deep_scrub_stamp":"2024-09-19T07:49:30.674832+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.674832+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:08:06.894419+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18216,"num_write_kb":261619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"61'4656","reported_seq":4780,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.577483+0000","last_change":"2024-09-19T07:49:29.577545+0000","last_active":"2024-09-19T07:49:29.577483+0000","last_peered":"2024-09-19T07:49:29.577483+0000","last_clean":"2024-09-19T07:49:29.577483+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:49:29.577483+0000","last_undegraded":"2024-09-19T07:49:29.577483+0000","last_fullsized":"2024-09-19T07:49:29.577483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4656","last_scrub_stamp":"2024-09-19T07:49:29.577280+0000","last_deep_scrub":"61'4656","last_deep_scrub_stamp":"2024-09-19T07:49:29.577280+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.577280+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:47.082591+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24790,"num_write_kb":287946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"64'4503","reported_seq":4616,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.329668+0000","last_change":"2024-09-19T07:49:31.329668+0000","last_active":"2024-09-19T07:49:31.329668+0000","last_peered":"2024-09-19T07:49:31.329668+0000","last_clean":"2024-09-19T07:49:31.329668+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:49:31.329668+0000","last_undegraded":"2024-09-19T07:49:31.329668+0000","last_fullsized":"2024-09-19T07:49:31.329668+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4503","last_scrub_stamp":"2024-09-19T07:49:31.329217+0000","last_deep_scrub":"64'4503","last_deep_scrub_stamp":"2024-09-19T07:49:31.329217+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.329217+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:36:23.637457+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24213,"num_write_kb":310059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"61'5949","reported_seq":6072,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.286974+0000","last_change":"2024-09-19T07:49:26.287050+0000","last_active":"2024-09-19T07:49:26.286974+0000","last_peered":"2024-09-19T07:49:26.286974+0000","last_clean":"2024-09-19T07:49:26.286974+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:49:26.286974+0000","last_undegraded":"2024-09-19T07:49:26.286974+0000","last_fullsized":"2024-09-19T07:49:26.286974+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5949","last_scrub_stamp":"2024-09-19T07:49:26.286761+0000","last_deep_scrub":"61'5949","last_deep_scrub_stamp":"2024-09-19T07:49:26.286761+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.286761+0000","objects_scrubbed":0,"log_size":5849,"log_dups_size":0,"ondisk_log_size":5849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:01:19.621957+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31740,"num_write_kb":364971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"64'4360","reported_seq":4473,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.360009+0000","last_change":"2024-09-19T07:49:29.360009+0000","last_active":"2024-09-19T07:49:29.360009+0000","last_peered":"2024-09-19T07:49:29.360009+0000","last_clean":"2024-09-19T07:49:29.360009+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:49:29.360009+0000","last_undegraded":"2024-09-19T07:49:29.360009+0000","last_fullsized":"2024-09-19T07:49:29.360009+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4360","last_scrub_stamp":"2024-09-19T07:49:29.359658+0000","last_deep_scrub":"64'4360","last_deep_scrub_stamp":"2024-09-19T07:49:29.359658+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.359658+0000","objects_scrubbed":0,"log_size":4360,"log_dups_size":0,"ondisk_log_size":4360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:17.888583+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23533,"num_write_kb":311499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"61'4830","reported_seq":4954,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.228354+0000","last_change":"2024-09-19T07:49:26.228418+0000","last_active":"2024-09-19T07:49:26.228354+0000","last_peered":"2024-09-19T07:49:26.228354+0000","last_clean":"2024-09-19T07:49:26.228354+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:49:26.228354+0000","last_undegraded":"2024-09-19T07:49:26.228354+0000","last_fullsized":"2024-09-19T07:49:26.228354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4830","last_scrub_stamp":"2024-09-19T07:49:26.228143+0000","last_deep_scrub":"61'4830","last_deep_scrub_stamp":"2024-09-19T07:49:26.228143+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.228143+0000","objects_scrubbed":0,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:24:35.194932+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26589,"num_write_kb":315697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1521393,"num_write_kb":19214666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279640,"ondisk_log_size":279640,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19627747,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1449,"num_read_kb":190539,"num_write":5111,"num_write_kb":21283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39313408,"data_stored":39255494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4630,"ondisk_log_size":4630,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":0,"num_write":1651322,"num_write_kb":19736802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307839,"ondisk_log_size":307839,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":19579163,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1454,"num_read_kb":221531,"num_write":5068,"num_write_kb":21268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39223296,"data_stored":39158326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4647,"ondisk_log_size":4647,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542512,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1114512,"kb_used_data":13696,"kb_used_omap":1,"kb_used_meta":1100798,"kb_avail":93257328,"statfs":{"total":96636764160,"available":95495503872,"internally_reserved":0,"allocated":14024704,"data_stored":13620504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1127217610},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542513,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766708,"kb_used_data":11684,"kb_used_omap":1,"kb_used_meta":755006,"kb_avail":93605132,"statfs":{"total":96636764160,"available":95851655168,"internally_reserved":0,"allocated":11964416,"data_stored":11552279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":773126598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542511,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1069796,"kb_used_data":5012,"kb_used_omap":1,"kb_used_meta":1064766,"kb_avail":93302044,"statfs":{"total":96636764160,"available":95541293056,"internally_reserved":0,"allocated":5132288,"data_stored":4726264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1090320837},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542513,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1229016,"kb_used_data":14472,"kb_used_omap":1,"kb_used_meta":1214526,"kb_avail":93142824,"statfs":{"total":96636764160,"available":95378251776,"internally_reserved":0,"allocated":14819328,"data_stored":14400282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1243675077},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542511,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1080696,"kb_used_data":20008,"kb_used_omap":1,"kb_used_meta":1060670,"kb_avail":93291144,"statfs":{"total":96636764160,"available":95530131456,"internally_reserved":0,"allocated":20488192,"data_stored":20074195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1086126535},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542513,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688028,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":687102,"kb_avail":93683812,"statfs":{"total":96636764160,"available":95932223488,"internally_reserved":0,"allocated":929792,"data_stored":531268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":703592902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542512,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":891928,"kb_used_data":9032,"kb_used_omap":1,"kb_used_meta":882878,"kb_avail":93479912,"statfs":{"total":96636764160,"available":95723429888,"internally_reserved":0,"allocated":9248768,"data_stored":8836014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904067525},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542511,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":907476,"kb_used_data":9604,"kb_used_omap":1,"kb_used_meta":897854,"kb_avail":93464364,"statfs":{"total":96636764160,"available":95707508736,"internally_reserved":0,"allocated":9834496,"data_stored":9426166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919402949},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":4194942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":84646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8482816,"data_stored":8473926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6926336,"data_stored":6910261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6922240,"data_stored":6910807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11165696,"data_stored":11153717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7053312,"data_stored":7043469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4279622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:49:43.762 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.762 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.762 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-09-19T07:19:30.217638+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=30, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.763 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.763 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.763 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.763 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.763 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.763 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.764 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.764 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.765 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.765 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.765 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.765 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.766 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.766 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.766 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.766 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.767 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.767 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.767 INFO:tasks.ceph:pgid 5.26 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.767 INFO:tasks.ceph:pgid 5.24 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.768 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.768 INFO:tasks.ceph:pgid 5.25 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.768 INFO:tasks.ceph:pgid 5.22 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.768 INFO:tasks.ceph:pgid 5.20 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.768 INFO:tasks.ceph:pgid 5.2e last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.768 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.769 INFO:tasks.ceph:pgid 5.2f last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.769 INFO:tasks.ceph:pgid 5.2a last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.770 INFO:tasks.ceph:pgid 5.2b last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.770 INFO:tasks.ceph:pgid 5.36 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.770 INFO:tasks.ceph:pgid 5.34 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.770 INFO:tasks.ceph:pgid 5.35 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.770 INFO:tasks.ceph:pgid 5.32 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.770 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-09-19T07:19:31.227673+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=31, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.771 INFO:tasks.ceph:pgid 5.30 last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.771 INFO:tasks.ceph:pgid 5.3e last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.771 INFO:tasks.ceph:pgid 5.3c last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.771 INFO:tasks.ceph:pgid 5.3d last_scrub_stamp 2024-09-19T07:19:39.378602+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=19, tm_sec=39, tm_wday=3, tm_yday=263, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=19, tm_hour=7, tm_min=49, tm_sec=18, tm_wday=3, tm_yday=263, tm_isdst=0) 2024-09-19T07:49:43.771 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-19T07:49:45.918 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:49:46.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.048+0000 7fdf4f6c2640 1 Processor -- start 2024-09-19T07:49:46.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.048+0000 7fdf4f6c2640 1 -- start start 2024-09-19T07:49:46.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4f6c2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf480c28e0 0x7fdf481545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4f6c2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdf4806dea0 0x7fdf4806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4f6c2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf480a9e60 con 0x7fdf48071100 2024-09-19T07:49:46.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4f6c2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fdf48071b50 con 0x7fdf4806dea0 2024-09-19T07:49:46.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4f6c2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdf480c1ff0 con 0x7fdf480c28e0 2024-09-19T07:49:46.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4d45e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdf48071100 0x7fdf48150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34056/0 (socket says 172.21.15.83:34056) 2024-09-19T07:49:46.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4d45e640 1 -- 172.21.15.83:0/465713790 learned_addr learned my addr 172.21.15.83:0/465713790 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:46.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4dc5f640 1 --2- 172.21.15.83:0/465713790 >> v2:172.21.15.83:3300/0 conn(0x7fdf4806dea0 0x7fdf4806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4cc5d640 1 --2- 172.21.15.83:0/465713790 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf480c28e0 0x7fdf481545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf3ffff640 1 -- 172.21.15.83:0/465713790 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 973865877 0 0) 0x7fdf480a9e60 con 0x7fdf48071100 2024-09-19T07:49:46.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf3ffff640 1 -- 172.21.15.83:0/465713790 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf34003530 con 0x7fdf48071100 2024-09-19T07:49:46.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4dc5f640 1 -- 172.21.15.83:0/465713790 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf480c28e0 msgr2=0x7fdf481545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:46.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4dc5f640 1 --2- 172.21.15.83:0/465713790 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf480c28e0 0x7fdf481545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4dc5f640 1 -- 172.21.15.83:0/465713790 >> v1:172.21.15.83:6789/0 conn(0x7fdf48071100 legacy=0x7fdf48150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4dc5f640 1 -- 172.21.15.83:0/465713790 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf480aa140 con 0x7fdf4806dea0 2024-09-19T07:49:46.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.052+0000 7fdf4dc5f640 1 --2- 172.21.15.83:0/465713790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf4806dea0 0x7fdf4806e350 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009d90 tx=0x7fdf4001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156dc7dedf033b18 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:46.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf3ffff640 1 -- 172.21.15.83:0/465713790 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf400b2070 con 0x7fdf4806dea0 2024-09-19T07:49:46.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf3ffff640 1 -- 172.21.15.83:0/465713790 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf400a57a0 con 0x7fdf4806dea0 2024-09-19T07:49:46.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf3ffff640 1 -- 172.21.15.83:0/465713790 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf400a5b70 con 0x7fdf4806dea0 2024-09-19T07:49:46.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/465713790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf4806dea0 msgr2=0x7fdf4806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 --2- 172.21.15.83:0/465713790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf4806dea0 0x7fdf4806e350 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009d90 tx=0x7fdf4001c8e0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/465713790 shutdown_connections 2024-09-19T07:49:46.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 --2- 172.21.15.83:0/465713790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf4806dea0 0x7fdf4806e350 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 --2- 172.21.15.83:0/465713790 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf480c28e0 0x7fdf481545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/465713790 >> 172.21.15.83:0/465713790 conn(0x7fdf4806bc80 msgr2=0x7fdf4806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:46.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/465713790 shutdown_connections 2024-09-19T07:49:46.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/465713790 wait complete. 2024-09-19T07:49:46.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 Processor -- start 2024-09-19T07:49:46.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 -- start start 2024-09-19T07:49:46.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf4806dea0 0x7fdf481f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf48071100 0x7fdf4814c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf480c28e0 0x7fdf481ec120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48071660 con 0x7fdf480c28e0 2024-09-19T07:49:46.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdf480714e0 con 0x7fdf48071100 2024-09-19T07:49:46.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdf48160c60 con 0x7fdf4806dea0 2024-09-19T07:49:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4dc5f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf480c28e0 0x7fdf481ec120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4dc5f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf480c28e0 0x7fdf481ec120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50956/0 (socket says 172.21.15.83:50956) 2024-09-19T07:49:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4dc5f640 1 -- 172.21.15.83:0/1262752012 learned_addr learned my addr 172.21.15.83:0/1262752012 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4d45e640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf4806dea0 0x7fdf481f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4cc5d640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf48071100 0x7fdf4814c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4dc5f640 1 -- 172.21.15.83:0/1262752012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf4806dea0 msgr2=0x7fdf481f00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4dc5f640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf4806dea0 0x7fdf481f00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4dc5f640 1 -- 172.21.15.83:0/1262752012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf48071100 msgr2=0x7fdf4814c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4dc5f640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf48071100 0x7fdf4814c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4dc5f640 1 -- 172.21.15.83:0/1262752012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf48145310 con 0x7fdf480c28e0 2024-09-19T07:49:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4cc5d640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf48071100 0x7fdf4814c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4dc5f640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf480c28e0 0x7fdf481ec120 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fdf400b0040 tx=0x7fdf40019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf3dffb640 1 -- 172.21.15.83:0/1262752012 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf400b2050 con 0x7fdf480c28e0 2024-09-19T07:49:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf3dffb640 1 -- 172.21.15.83:0/1262752012 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf400be070 con 0x7fdf480c28e0 2024-09-19T07:49:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/1262752012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4814a1c0 con 0x7fdf480c28e0 2024-09-19T07:49:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf3dffb640 1 -- 172.21.15.83:0/1262752012 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf400ad890 con 0x7fdf480c28e0 2024-09-19T07:49:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.056+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/1262752012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf481ef8e0 con 0x7fdf480c28e0 2024-09-19T07:49:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.060+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/1262752012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf4806e350 con 0x7fdf480c28e0 2024-09-19T07:49:46.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.060+0000 7fdf3dffb640 1 -- 172.21.15.83:0/1262752012 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdf400ada30 con 0x7fdf480c28e0 2024-09-19T07:49:46.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.060+0000 7fdf3dffb640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf2c0678f0 0x7fdf2c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.060+0000 7fdf3dffb640 1 -- 172.21.15.83:0/1262752012 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fdf401217d0 con 0x7fdf480c28e0 2024-09-19T07:49:46.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.060+0000 7fdf4d45e640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf2c0678f0 0x7fdf2c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.060+0000 7fdf4d45e640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf2c0678f0 0x7fdf2c069db0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdf44005890 tx=0x7fdf44009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:46.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.064+0000 7fdf3dffb640 1 -- 172.21.15.83:0/1262752012 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdf400c7020 con 0x7fdf480c28e0 2024-09-19T07:49:46.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.256+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/1262752012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf480c4f30 con 0x7fdf480c28e0 2024-09-19T07:49:46.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.256+0000 7fdf3dffb640 1 -- 172.21.15.83:0/1262752012 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) ==== 95+0+6542 (secure 0 0 0) 0x7fdf48145310 con 0x7fdf480c28e0 2024-09-19T07:49:46.256 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:46.256 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":205,"btime":"2024-09-19T07:49:08:666589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6407,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:49:01.773306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8265,"qdb_cluster":[8265]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:49:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 205 2024-09-19T07:49:46.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.256+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/1262752012 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf2c0678f0 msgr2=0x7fdf2c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.256+0000 7fdf4f6c2640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf2c0678f0 0x7fdf2c069db0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdf44005890 tx=0x7fdf44009040 comp rx=0 tx=0).stop 2024-09-19T07:49:46.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.256+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/1262752012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf480c28e0 msgr2=0x7fdf481ec120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.256+0000 7fdf4f6c2640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf480c28e0 0x7fdf481ec120 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fdf400b0040 tx=0x7fdf40019680 comp rx=0 tx=0).stop 2024-09-19T07:49:46.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.256+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/1262752012 shutdown_connections 2024-09-19T07:49:46.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.256+0000 7fdf4f6c2640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdf2c0678f0 0x7fdf2c069db0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.256+0000 7fdf4f6c2640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf480c28e0 0x7fdf481ec120 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.256+0000 7fdf4f6c2640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf48071100 0x7fdf4814c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.256+0000 7fdf4f6c2640 1 --2- 172.21.15.83:0/1262752012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf4806dea0 0x7fdf481f00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.256+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/1262752012 >> 172.21.15.83:0/1262752012 conn(0x7fdf4806bc80 msgr2=0x7fdf4806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:46.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.260+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/1262752012 shutdown_connections 2024-09-19T07:49:46.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.260+0000 7fdf4f6c2640 1 -- 172.21.15.83:0/1262752012 wait complete. 2024-09-19T07:49:46.274 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-19T07:49:46.275 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-19T07:49:46.275 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:49:46.288 INFO:tasks.ceph.mds.c:Stopped 2024-09-19T07:49:46.289 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:49:46.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce61f91640 1 Processor -- start 2024-09-19T07:49:46.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce61f91640 1 -- start start 2024-09-19T07:49:46.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce61f91640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fce5c153480 0x7fce5c155e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce61f91640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce5c159dc0 con 0x7fce5c153860 2024-09-19T07:49:46.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce61f91640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce5c0c2bf0 con 0x7fce5c06be10 2024-09-19T07:49:46.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce61f91640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fce5c070040 con 0x7fce5c153480 2024-09-19T07:49:46.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fce5c153860 0x7fce5c156380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34066/0 (socket says 172.21.15.83:34066) 2024-09-19T07:49:46.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5affd640 1 -- 172.21.15.83:0/3190526011 learned_addr learned my addr 172.21.15.83:0/3190526011 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:46.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5b7fe640 1 --2- 172.21.15.83:0/3190526011 >> v2:172.21.15.184:3300/0 conn(0x7fce5c153480 0x7fce5c155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2390558028 0 0) 0x7fce5c159dc0 con 0x7fce5c153860 2024-09-19T07:49:46.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce44003530 con 0x7fce5c153860 2024-09-19T07:49:46.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1451600181 0 0) 0x7fce44003530 con 0x7fce5c153860 2024-09-19T07:49:46.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 >> v2:172.21.15.184:3300/0 conn(0x7fce5c153480 msgr2=0x7fce5c155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:46.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 --2- 172.21.15.83:0/3190526011 >> v2:172.21.15.184:3300/0 conn(0x7fce5c153480 0x7fce5c155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 >> v1:172.21.15.184:6789/0 conn(0x7fce5c06be10 legacy=0x7fce5c06c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c15a080 con 0x7fce5c153860 2024-09-19T07:49:46.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fce500016a0 con 0x7fce5c153860 2024-09-19T07:49:46.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 >> v1:172.21.15.83:6789/0 conn(0x7fce5c153860 legacy=0x7fce5c156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 --2- 172.21.15.83:0/3190526011 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce440039c0 0x7fce44003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 --2- 172.21.15.83:0/3190526011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce44004810 0x7fce44004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 --2- 172.21.15.83:0/3190526011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce440051d0 0x7fce4400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c070040 con 0x7fce44004810 2024-09-19T07:49:46.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fce4400c170 con 0x7fce440051d0 2024-09-19T07:49:46.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fce4400c370 con 0x7fce440039c0 2024-09-19T07:49:46.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5affd640 1 --2- 172.21.15.83:0/3190526011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce44004810 0x7fce44004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5bfff640 1 --2- 172.21.15.83:0/3190526011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce440051d0 0x7fce4400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5b7fe640 1 --2- 172.21.15.83:0/3190526011 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce440039c0 0x7fce44003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5affd640 1 -- 172.21.15.83:0/3190526011 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce440039c0 msgr2=0x7fce44003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:46.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5affd640 1 --2- 172.21.15.83:0/3190526011 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce440039c0 0x7fce44003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5affd640 1 -- 172.21.15.83:0/3190526011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce440051d0 msgr2=0x7fce4400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:46.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5affd640 1 --2- 172.21.15.83:0/3190526011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce440051d0 0x7fce4400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5bfff640 1 -- 172.21.15.83:0/3190526011 reap_dead start 2024-09-19T07:49:46.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5affd640 1 -- 172.21.15.83:0/3190526011 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce44003530 con 0x7fce44004810 2024-09-19T07:49:46.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.440+0000 7fce5b7fe640 1 --2- 172.21.15.83:0/3190526011 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce440039c0 0x7fce44003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:46.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5affd640 1 --2- 172.21.15.83:0/3190526011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce44004810 0x7fce44004c90 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fce5c06f210 tx=0x7fce5001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fbe1afc94f15c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:46.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50007d20 con 0x7fce44004810 2024-09-19T07:49:46.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce500a9020 con 0x7fce44004810 2024-09-19T07:49:46.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5a7fc640 1 -- 172.21.15.83:0/3190526011 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50005ae0 con 0x7fce44004810 2024-09-19T07:49:46.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 -- 172.21.15.83:0/3190526011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce44004810 msgr2=0x7fce44004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 --2- 172.21.15.83:0/3190526011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce44004810 0x7fce44004c90 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fce5c06f210 tx=0x7fce5001cc00 comp rx=0 tx=0).stop 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 -- 172.21.15.83:0/3190526011 shutdown_connections 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 --2- 172.21.15.83:0/3190526011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce44004810 0x7fce44004c90 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 -- 172.21.15.83:0/3190526011 >> 172.21.15.83:0/3190526011 conn(0x7fce5c0c8840 msgr2=0x7fce5c0c8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 -- 172.21.15.83:0/3190526011 shutdown_connections 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 -- 172.21.15.83:0/3190526011 wait complete. 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 Processor -- start 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 -- start start 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce44004810 0x7fce5c14feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c09d440 0x7fce5c09c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c1f1aa0 0x7fce5c1f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c156860 con 0x7fce5c09d440 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c0c1d80 con 0x7fce44004810 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce61f91640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fce5c1569e0 con 0x7fce5c1f1aa0 2024-09-19T07:49:46.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c09d440 0x7fce5c09c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c09d440 0x7fce5c09c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50960/0 (socket says 172.21.15.83:50960) 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5affd640 1 -- 172.21.15.83:0/2580482032 learned_addr learned my addr 172.21.15.83:0/2580482032 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5b7fe640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce44004810 0x7fce5c14feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5bfff640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c1f1aa0 0x7fce5c1f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5affd640 1 -- 172.21.15.83:0/2580482032 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c1f1aa0 msgr2=0x7fce5c1f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5affd640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c1f1aa0 0x7fce5c1f1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5affd640 1 -- 172.21.15.83:0/2580482032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce44004810 msgr2=0x7fce5c14feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5affd640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce44004810 0x7fce5c14feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5affd640 1 -- 172.21.15.83:0/2580482032 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c1f25e0 con 0x7fce5c09d440 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5b7fe640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce44004810 0x7fce5c14feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.444+0000 7fce5bfff640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c1f1aa0 0x7fce5c1f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.448+0000 7fce5affd640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c09d440 0x7fce5c09c260 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fce500019a0 tx=0x7fce44005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:46.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.448+0000 7fce43fff640 1 -- 172.21.15.83:0/2580482032 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50004c00 con 0x7fce5c09d440 2024-09-19T07:49:46.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.448+0000 7fce43fff640 1 -- 172.21.15.83:0/2580482032 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce50004da0 con 0x7fce5c09d440 2024-09-19T07:49:46.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.448+0000 7fce61f91640 1 -- 172.21.15.83:0/2580482032 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5c0ad020 con 0x7fce5c09d440 2024-09-19T07:49:46.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.448+0000 7fce43fff640 1 -- 172.21.15.83:0/2580482032 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50007310 con 0x7fce5c09d440 2024-09-19T07:49:46.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.448+0000 7fce61f91640 1 -- 172.21.15.83:0/2580482032 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce5c0aba40 con 0x7fce5c09d440 2024-09-19T07:49:46.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.448+0000 7fce61f91640 1 -- 172.21.15.83:0/2580482032 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce5c0c6260 con 0x7fce5c09d440 2024-09-19T07:49:46.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.448+0000 7fce43fff640 1 -- 172.21.15.83:0/2580482032 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fce500074b0 con 0x7fce5c09d440 2024-09-19T07:49:46.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.448+0000 7fce43fff640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce480678c0 0x7fce48069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:46.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.448+0000 7fce43fff640 1 -- 172.21.15.83:0/2580482032 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fce50120820 con 0x7fce5c09d440 2024-09-19T07:49:46.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.448+0000 7fce5b7fe640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce480678c0 0x7fce48069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:46.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.448+0000 7fce5b7fe640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce480678c0 0x7fce48069d80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fce5c051900 tx=0x7fce4c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:46.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.452+0000 7fce43fff640 1 -- 172.21.15.83:0/2580482032 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fce50016910 con 0x7fce5c09d440 2024-09-19T07:49:46.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.636+0000 7fce61f91640 1 -- 172.21.15.83:0/2580482032 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce5c159dc0 con 0x7fce5c09d440 2024-09-19T07:49:46.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce43fff640 1 -- 172.21.15.83:0/2580482032 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) ==== 95+0+6542 (secure 0 0 0) 0x7fce500f0290 con 0x7fce5c09d440 2024-09-19T07:49:46.638 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:46.638 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":205,"btime":"2024-09-19T07:49:08:666589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6407,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:49:01.773306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8265,"qdb_cluster":[8265]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:49:46.639 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 205 2024-09-19T07:49:46.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce61f91640 1 -- 172.21.15.83:0/2580482032 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce480678c0 msgr2=0x7fce48069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce61f91640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce480678c0 0x7fce48069d80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fce5c051900 tx=0x7fce4c008040 comp rx=0 tx=0).stop 2024-09-19T07:49:46.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce61f91640 1 -- 172.21.15.83:0/2580482032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c09d440 msgr2=0x7fce5c09c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:46.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce61f91640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c09d440 0x7fce5c09c260 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fce500019a0 tx=0x7fce44005840 comp rx=0 tx=0).stop 2024-09-19T07:49:46.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce61f91640 1 -- 172.21.15.83:0/2580482032 shutdown_connections 2024-09-19T07:49:46.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce61f91640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce480678c0 0x7fce48069d80 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce61f91640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce5c1f1aa0 0x7fce5c1f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce61f91640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce5c09d440 0x7fce5c09c260 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce61f91640 1 --2- 172.21.15.83:0/2580482032 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce44004810 0x7fce5c14feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:46.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce61f91640 1 -- 172.21.15.83:0/2580482032 >> 172.21.15.83:0/2580482032 conn(0x7fce5c0c8840 msgr2=0x7fce5c074f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:46.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce61f91640 1 -- 172.21.15.83:0/2580482032 shutdown_connections 2024-09-19T07:49:46.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:46.640+0000 7fce61f91640 1 -- 172.21.15.83:0/2580482032 wait complete. 2024-09-19T07:49:46.657 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:49:48.658 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:49:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9f4d66640 1 Processor -- start 2024-09-19T07:49:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9f4d66640 1 -- start start 2024-09-19T07:49:48.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9f4d66640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9f0071100 0x7fa9f01545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9f4d66640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9f006dea0 0x7fa9f006e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9f4d66640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9f00a9e60 con 0x7fa9f00c28e0 2024-09-19T07:49:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9f4d66640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa9f0071b50 con 0x7fa9f006dea0 2024-09-19T07:49:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9f4d66640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa9f00c1ff0 con 0x7fa9f0071100 2024-09-19T07:49:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9eed9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9f006dea0 0x7fa9f006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9eed9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9f006dea0 0x7fa9f006e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50964/0 (socket says 172.21.15.83:50964) 2024-09-19T07:49:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9eed9d640 1 -- 172.21.15.83:0/1411306792 learned_addr learned my addr 172.21.15.83:0/1411306792 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9ee59c640 1 --2- 172.21.15.83:0/1411306792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9f0071100 0x7fa9f01545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9eed9d640 1 -- 172.21.15.83:0/1411306792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9f0071100 msgr2=0x7fa9f01545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9eed9d640 1 --2- 172.21.15.83:0/1411306792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9f0071100 0x7fa9f01545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9eed9d640 1 -- 172.21.15.83:0/1411306792 >> v1:172.21.15.184:6789/0 conn(0x7fa9f00c28e0 legacy=0x7fa9f0150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.788+0000 7fa9eed9d640 1 -- 172.21.15.83:0/1411306792 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f00aa140 con 0x7fa9f006dea0 2024-09-19T07:49:48.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9eed9d640 1 --2- 172.21.15.83:0/1411306792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9f006dea0 0x7fa9f006e350 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8009e50 tx=0x7fa9e801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a41695b0fdad85 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:48.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9edd9b640 1 -- 172.21.15.83:0/1411306792 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e80a56d0 con 0x7fa9f006dea0 2024-09-19T07:49:48.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9edd9b640 1 -- 172.21.15.83:0/1411306792 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e80a5870 con 0x7fa9f006dea0 2024-09-19T07:49:48.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9edd9b640 1 -- 172.21.15.83:0/1411306792 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e80a5c40 con 0x7fa9f006dea0 2024-09-19T07:49:48.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 -- 172.21.15.83:0/1411306792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9f006dea0 msgr2=0x7fa9f006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:48.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 --2- 172.21.15.83:0/1411306792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9f006dea0 0x7fa9f006e350 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8009e50 tx=0x7fa9e801c960 comp rx=0 tx=0).stop 2024-09-19T07:49:48.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 -- 172.21.15.83:0/1411306792 shutdown_connections 2024-09-19T07:49:48.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 --2- 172.21.15.83:0/1411306792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9f006dea0 0x7fa9f006e350 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:48.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 --2- 172.21.15.83:0/1411306792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9f0071100 0x7fa9f01545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:48.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 -- 172.21.15.83:0/1411306792 >> 172.21.15.83:0/1411306792 conn(0x7fa9f006bc80 msgr2=0x7fa9f006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:48.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 -- 172.21.15.83:0/1411306792 shutdown_connections 2024-09-19T07:49:48.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 -- 172.21.15.83:0/1411306792 wait complete. 2024-09-19T07:49:48.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 Processor -- start 2024-09-19T07:49:48.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 -- start start 2024-09-19T07:49:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9f006dea0 0x7fa9f0153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9f0071100 0x7fa9f01519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9f00c28e0 0x7fa9f00c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f0071660 con 0x7fa9f0071100 2024-09-19T07:49:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f00714e0 con 0x7fa9f00c28e0 2024-09-19T07:49:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa9f0160c60 con 0x7fa9f006dea0 2024-09-19T07:49:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9e5d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9f0071100 0x7fa9f01519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9e5d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9f0071100 0x7fa9f01519d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50978/0 (socket says 172.21.15.83:50978) 2024-09-19T07:49:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9e5d9b640 1 -- 172.21.15.83:0/39476715 learned_addr learned my addr 172.21.15.83:0/39476715 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9eed9d640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9f00c28e0 0x7fa9f00c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9ee59c640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9f006dea0 0x7fa9f0153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9e5d9b640 1 -- 172.21.15.83:0/39476715 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9f006dea0 msgr2=0x7fa9f0153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9e5d9b640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9f006dea0 0x7fa9f0153720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9e5d9b640 1 -- 172.21.15.83:0/39476715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9f00c28e0 msgr2=0x7fa9f00c3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9e5d9b640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9f00c28e0 0x7fa9f00c3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9e5d9b640 1 -- 172.21.15.83:0/39476715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f01478c0 con 0x7fa9f0071100 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9eed9d640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9f00c28e0 0x7fa9f00c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9ee59c640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9f006dea0 0x7fa9f0153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9e5d9b640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9f0071100 0x7fa9f01519d0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800b3e0 tx=0x7fa9d800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9e77fe640 1 -- 172.21.15.83:0/39476715 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8015020 con 0x7fa9f0071100 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9e77fe640 1 -- 172.21.15.83:0/39476715 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d8011830 con 0x7fa9f0071100 2024-09-19T07:49:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9e77fe640 1 -- 172.21.15.83:0/39476715 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8011c60 con 0x7fa9f0071100 2024-09-19T07:49:48.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 -- 172.21.15.83:0/39476715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9f01f0fd0 con 0x7fa9f0071100 2024-09-19T07:49:48.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.792+0000 7fa9f4d66640 1 -- 172.21.15.83:0/39476715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9f01f0cf0 con 0x7fa9f0071100 2024-09-19T07:49:48.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.796+0000 7fa9f4d66640 1 -- 172.21.15.83:0/39476715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9f006e350 con 0x7fa9f0071100 2024-09-19T07:49:48.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.796+0000 7fa9e77fe640 1 -- 172.21.15.83:0/39476715 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa9d8018020 con 0x7fa9f0071100 2024-09-19T07:49:48.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.796+0000 7fa9e77fe640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9d00678c0 0x7fa9d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:48.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.796+0000 7fa9ee59c640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9d00678c0 0x7fa9d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:48.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.796+0000 7fa9e77fe640 1 -- 172.21.15.83:0/39476715 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fa9d8086a50 con 0x7fa9f0071100 2024-09-19T07:49:48.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.796+0000 7fa9ee59c640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9d00678c0 0x7fa9d0069d80 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009840 tx=0x7fa9e0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:48.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:48.800+0000 7fa9e77fe640 1 -- 172.21.15.83:0/39476715 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fa9d801f030 con 0x7fa9f0071100 2024-09-19T07:49:48.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.000+0000 7fa9f4d66640 1 -- 172.21.15.83:0/39476715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9f00742b0 con 0x7fa9f0071100 2024-09-19T07:49:49.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.000+0000 7fa9e77fe640 1 -- 172.21.15.83:0/39476715 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) ==== 95+0+6542 (secure 0 0 0) 0x7fa9f01478c0 con 0x7fa9f0071100 2024-09-19T07:49:49.000 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:49.001 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":205,"btime":"2024-09-19T07:49:08:666589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6407,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:49:01.773306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8265,"qdb_cluster":[8265]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:49:49.001 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 205 2024-09-19T07:49:49.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.004+0000 7fa9f4d66640 1 -- 172.21.15.83:0/39476715 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9d00678c0 msgr2=0x7fa9d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:49.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.004+0000 7fa9f4d66640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9d00678c0 0x7fa9d0069d80 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009840 tx=0x7fa9e0008040 comp rx=0 tx=0).stop 2024-09-19T07:49:49.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.004+0000 7fa9f4d66640 1 -- 172.21.15.83:0/39476715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9f0071100 msgr2=0x7fa9f01519d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:49.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.004+0000 7fa9f4d66640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9f0071100 0x7fa9f01519d0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800b3e0 tx=0x7fa9d800b8a0 comp rx=0 tx=0).stop 2024-09-19T07:49:49.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.004+0000 7fa9f4d66640 1 -- 172.21.15.83:0/39476715 shutdown_connections 2024-09-19T07:49:49.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.004+0000 7fa9f4d66640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fa9d00678c0 0x7fa9d0069d80 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:49.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.004+0000 7fa9f4d66640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa9f00c28e0 0x7fa9f00c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:49.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.004+0000 7fa9f4d66640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9f0071100 0x7fa9f01519d0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:49.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.004+0000 7fa9f4d66640 1 --2- 172.21.15.83:0/39476715 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9f006dea0 0x7fa9f0153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:49.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.004+0000 7fa9f4d66640 1 -- 172.21.15.83:0/39476715 >> 172.21.15.83:0/39476715 conn(0x7fa9f006bc80 msgr2=0x7fa9f0073000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:49.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.004+0000 7fa9f4d66640 1 -- 172.21.15.83:0/39476715 shutdown_connections 2024-09-19T07:49:49.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:49.004+0000 7fa9f4d66640 1 -- 172.21.15.83:0/39476715 wait complete. 2024-09-19T07:49:49.019 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:49:51.020 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:49:51.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb52d732640 1 Processor -- start 2024-09-19T07:49:51.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb52d732640 1 -- start start 2024-09-19T07:49:51.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb52d732640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb528071100 0x7fb5281545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:51.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb52d732640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5280c28e0 0x7fb528150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:51.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb52d732640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5280a9e60 con 0x7fb528151480 2024-09-19T07:49:51.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb52d732640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fb528071b50 con 0x7fb528071100 2024-09-19T07:49:51.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb52d732640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb5280c1ff0 con 0x7fb5280c28e0 2024-09-19T07:49:51.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb526ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb528071100 0x7fb5281545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:51.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb5277fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb528151480 0x7fb52806dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52664/0 (socket says 172.21.15.83:52664) 2024-09-19T07:49:51.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb5277fe640 1 -- 172.21.15.83:0/154907887 learned_addr learned my addr 172.21.15.83:0/154907887 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:51.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb5267fc640 1 --2- 172.21.15.83:0/154907887 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5280c28e0 0x7fb528150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:51.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb525ffb640 1 -- 172.21.15.83:0/154907887 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1611386843 0 0) 0x7fb5280a9e60 con 0x7fb528151480 2024-09-19T07:49:51.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb525ffb640 1 -- 172.21.15.83:0/154907887 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb510003530 con 0x7fb528151480 2024-09-19T07:49:51.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb526ffd640 1 -- 172.21.15.83:0/154907887 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5280c28e0 msgr2=0x7fb528150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:51.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb526ffd640 1 --2- 172.21.15.83:0/154907887 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5280c28e0 0x7fb528150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:51.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb526ffd640 1 -- 172.21.15.83:0/154907887 >> v1:172.21.15.83:6789/0 conn(0x7fb528151480 legacy=0x7fb52806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:51.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.151+0000 7fb526ffd640 1 -- 172.21.15.83:0/154907887 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5280aa140 con 0x7fb528071100 2024-09-19T07:49:51.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb526ffd640 1 --2- 172.21.15.83:0/154907887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb528071100 0x7fb5281545e0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fb5180097c0 tx=0x7fb51801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fae1b1380309959 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:51.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb525ffb640 1 -- 172.21.15.83:0/154907887 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5180b0070 con 0x7fb528071100 2024-09-19T07:49:51.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb525ffb640 1 -- 172.21.15.83:0/154907887 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5180ab600 con 0x7fb528071100 2024-09-19T07:49:51.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb525ffb640 1 -- 172.21.15.83:0/154907887 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5180ab9d0 con 0x7fb528071100 2024-09-19T07:49:51.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 -- 172.21.15.83:0/154907887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb528071100 msgr2=0x7fb5281545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:51.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 --2- 172.21.15.83:0/154907887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb528071100 0x7fb5281545e0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fb5180097c0 tx=0x7fb51801c410 comp rx=0 tx=0).stop 2024-09-19T07:49:51.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 -- 172.21.15.83:0/154907887 shutdown_connections 2024-09-19T07:49:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 --2- 172.21.15.83:0/154907887 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb5280c28e0 0x7fb528150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 --2- 172.21.15.83:0/154907887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb528071100 0x7fb5281545e0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 -- 172.21.15.83:0/154907887 >> 172.21.15.83:0/154907887 conn(0x7fb52806bc80 msgr2=0x7fb52806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 -- 172.21.15.83:0/154907887 shutdown_connections 2024-09-19T07:49:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 -- 172.21.15.83:0/154907887 wait complete. 2024-09-19T07:49:51.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 Processor -- start 2024-09-19T07:49:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 -- start start 2024-09-19T07:49:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5280c28e0 0x7fb528142790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb528151480 0x7fb528147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb528147950 0x7fb5281ef810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb528071660 con 0x7fb528147950 2024-09-19T07:49:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb5280714e0 con 0x7fb5280c28e0 2024-09-19T07:49:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.155+0000 7fb52d732640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb528160c60 con 0x7fb528151480 2024-09-19T07:49:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb5277fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb528147950 0x7fb5281ef810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb5277fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb528147950 0x7fb5281ef810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55254/0 (socket says 172.21.15.83:55254) 2024-09-19T07:49:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb5277fe640 1 -- 172.21.15.83:0/4176451990 learned_addr learned my addr 172.21.15.83:0/4176451990 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb5277fe640 1 -- 172.21.15.83:0/4176451990 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb528151480 msgr2=0x7fb528147410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb526ffd640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5280c28e0 0x7fb528142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb5267fc640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb528151480 0x7fb528147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb5277fe640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb528151480 0x7fb528147410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb5277fe640 1 -- 172.21.15.83:0/4176451990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5280c28e0 msgr2=0x7fb528142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb5277fe640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5280c28e0 0x7fb528142790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb5277fe640 1 -- 172.21.15.83:0/4176451990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb528148190 con 0x7fb528147950 2024-09-19T07:49:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb5277fe640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb528147950 0x7fb5281ef810 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fb52000cb40 tx=0x7fb52000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb50f7fe640 1 -- 172.21.15.83:0/4176451990 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520017070 con 0x7fb528147950 2024-09-19T07:49:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb50f7fe640 1 -- 172.21.15.83:0/4176451990 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb52000abe0 con 0x7fb528147950 2024-09-19T07:49:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb50f7fe640 1 -- 172.21.15.83:0/4176451990 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520012400 con 0x7fb528147950 2024-09-19T07:49:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb52d732640 1 -- 172.21.15.83:0/4176451990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5281ecdc0 con 0x7fb528147950 2024-09-19T07:49:51.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb52d732640 1 -- 172.21.15.83:0/4176451990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5280c6f30 con 0x7fb528147950 2024-09-19T07:49:51.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb52d732640 1 -- 172.21.15.83:0/4176451990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb528153780 con 0x7fb528147950 2024-09-19T07:49:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb50f7fe640 1 -- 172.21.15.83:0/4176451990 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb52001a070 con 0x7fb528147950 2024-09-19T07:49:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb50f7fe640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5140678f0 0x7fb514069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb50f7fe640 1 -- 172.21.15.83:0/4176451990 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fb520086b00 con 0x7fb528147950 2024-09-19T07:49:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb526ffd640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5140678f0 0x7fb514069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.159+0000 7fb526ffd640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5140678f0 0x7fb514069db0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb5180ad000 tx=0x7fb518008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.163+0000 7fb50f7fe640 1 -- 172.21.15.83:0/4176451990 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fb5200564e0 con 0x7fb528147950 2024-09-19T07:49:51.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.359+0000 7fb52d732640 1 -- 172.21.15.83:0/4176451990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5281545e0 con 0x7fb528147950 2024-09-19T07:49:51.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.359+0000 7fb50f7fe640 1 -- 172.21.15.83:0/4176451990 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) ==== 95+0+6542 (secure 0 0 0) 0x7fb520055de0 con 0x7fb528147950 2024-09-19T07:49:51.362 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:51.362 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":205,"btime":"2024-09-19T07:49:08:666589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6407,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:49:01.773306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8265,"qdb_cluster":[8265]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:49:51.362 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 205 2024-09-19T07:49:51.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.363+0000 7fb52d732640 1 -- 172.21.15.83:0/4176451990 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5140678f0 msgr2=0x7fb514069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:51.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.363+0000 7fb52d732640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5140678f0 0x7fb514069db0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb5180ad000 tx=0x7fb518008040 comp rx=0 tx=0).stop 2024-09-19T07:49:51.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.363+0000 7fb52d732640 1 -- 172.21.15.83:0/4176451990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb528147950 msgr2=0x7fb5281ef810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:51.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.363+0000 7fb52d732640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb528147950 0x7fb5281ef810 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fb52000cb40 tx=0x7fb52000a060 comp rx=0 tx=0).stop 2024-09-19T07:49:51.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.363+0000 7fb52d732640 1 -- 172.21.15.83:0/4176451990 shutdown_connections 2024-09-19T07:49:51.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.363+0000 7fb52d732640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fb5140678f0 0x7fb514069db0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:51.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.363+0000 7fb52d732640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb528147950 0x7fb5281ef810 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:51.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.363+0000 7fb52d732640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb528151480 0x7fb528147410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:51.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.363+0000 7fb52d732640 1 --2- 172.21.15.83:0/4176451990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb5280c28e0 0x7fb528142790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:51.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.363+0000 7fb52d732640 1 -- 172.21.15.83:0/4176451990 >> 172.21.15.83:0/4176451990 conn(0x7fb52806bc80 msgr2=0x7fb52806dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:51.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.363+0000 7fb52d732640 1 -- 172.21.15.83:0/4176451990 shutdown_connections 2024-09-19T07:49:51.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:51.363+0000 7fb52d732640 1 -- 172.21.15.83:0/4176451990 wait complete. 2024-09-19T07:49:51.384 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:49:53.385 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:49:53.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.515+0000 7f969faf5640 1 Processor -- start 2024-09-19T07:49:53.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.515+0000 7f969faf5640 1 -- start start 2024-09-19T07:49:53.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969faf5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9698071100 0x7f96981545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:53.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969faf5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f96980c28e0 0x7f9698150f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969faf5640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f9698151480 0x7f969806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969faf5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f9698071b50 con 0x7f96980c28e0 2024-09-19T07:49:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969faf5640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f96980c1ff0 con 0x7f9698151480 2024-09-19T07:49:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969faf5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f96980a9e60 con 0x7f9698071100 2024-09-19T07:49:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969d090640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f96980c28e0 0x7f9698150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969d090640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f96980c28e0 0x7f9698150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55258/0 (socket says 172.21.15.83:55258) 2024-09-19T07:49:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969d090640 1 -- 172.21.15.83:0/3618495368 learned_addr learned my addr 172.21.15.83:0/3618495368 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969e092640 1 --2- 172.21.15.83:0/3618495368 >> v2:172.21.15.184:3300/0 conn(0x7f9698151480 0x7f969806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969d891640 1 --2- 172.21.15.83:0/3618495368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9698071100 0x7f96981545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969d090640 1 -- 172.21.15.83:0/3618495368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9698071100 msgr2=0x7f96981545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969d090640 1 --2- 172.21.15.83:0/3618495368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9698071100 0x7f96981545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:53.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969d090640 1 -- 172.21.15.83:0/3618495368 >> v2:172.21.15.184:3300/0 conn(0x7f9698151480 msgr2=0x7f969806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:53.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969d090640 1 --2- 172.21.15.83:0/3618495368 >> v2:172.21.15.184:3300/0 conn(0x7f9698151480 0x7f969806dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:53.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969d090640 1 -- 172.21.15.83:0/3618495368 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96980aa140 con 0x7f96980c28e0 2024-09-19T07:49:53.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.519+0000 7f969d891640 1 --2- 172.21.15.83:0/3618495368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9698071100 0x7f96981545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:53.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969d090640 1 --2- 172.21.15.83:0/3618495368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96980c28e0 0x7f9698150f40 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f969806cd60 tx=0x7f969401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a32070c0c1386c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:53.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969c88f640 1 -- 172.21.15.83:0/3618495368 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96940b0070 con 0x7f96980c28e0 2024-09-19T07:49:53.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969c88f640 1 -- 172.21.15.83:0/3618495368 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96940ab600 con 0x7f96980c28e0 2024-09-19T07:49:53.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969c88f640 1 -- 172.21.15.83:0/3618495368 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96940ab9d0 con 0x7f96980c28e0 2024-09-19T07:49:53.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 -- 172.21.15.83:0/3618495368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96980c28e0 msgr2=0x7f9698150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:53.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 --2- 172.21.15.83:0/3618495368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96980c28e0 0x7f9698150f40 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f969806cd60 tx=0x7f969401c490 comp rx=0 tx=0).stop 2024-09-19T07:49:53.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 -- 172.21.15.83:0/3618495368 shutdown_connections 2024-09-19T07:49:53.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 --2- 172.21.15.83:0/3618495368 >> v2:172.21.15.184:3300/0 conn(0x7f9698151480 0x7f969806dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:53.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 --2- 172.21.15.83:0/3618495368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96980c28e0 0x7f9698150f40 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:53.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 --2- 172.21.15.83:0/3618495368 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9698071100 0x7f96981545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:53.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 -- 172.21.15.83:0/3618495368 >> 172.21.15.83:0/3618495368 conn(0x7f969806bc80 msgr2=0x7f969806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 -- 172.21.15.83:0/3618495368 shutdown_connections 2024-09-19T07:49:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 -- 172.21.15.83:0/3618495368 wait complete. 2024-09-19T07:49:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 Processor -- start 2024-09-19T07:49:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 -- start start 2024-09-19T07:49:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9698071100 0x7f9698155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96980c28e0 0x7f96980aa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9698151480 0x7f96980aace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f96981570d0 con 0x7f9698151480 2024-09-19T07:49:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9698071730 con 0x7f96980c28e0 2024-09-19T07:49:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969faf5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9698157250 con 0x7f9698071100 2024-09-19T07:49:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969e092640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9698151480 0x7f96980aace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969e092640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9698151480 0x7f96980aace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55274/0 (socket says 172.21.15.83:55274) 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969e092640 1 -- 172.21.15.83:0/760279542 learned_addr learned my addr 172.21.15.83:0/760279542 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969d090640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96980c28e0 0x7f96980aa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969d891640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9698071100 0x7f9698155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969e092640 1 -- 172.21.15.83:0/760279542 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9698071100 msgr2=0x7f9698155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969e092640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9698071100 0x7f9698155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969e092640 1 -- 172.21.15.83:0/760279542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96980c28e0 msgr2=0x7f96980aa7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969e092640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96980c28e0 0x7f96980aa7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969e092640 1 -- 172.21.15.83:0/760279542 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969815d7d0 con 0x7f9698151480 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f969e092640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9698151480 0x7f96980aace0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f9690007e60 tx=0x7f969000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.523+0000 7f968e7fc640 1 -- 172.21.15.83:0/760279542 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690014420 con 0x7f9698151480 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.527+0000 7f968e7fc640 1 -- 172.21.15.83:0/760279542 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9690019070 con 0x7f9698151480 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.527+0000 7f969faf5640 1 -- 172.21.15.83:0/760279542 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f969814c360 con 0x7f9698151480 2024-09-19T07:49:53.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.527+0000 7f968e7fc640 1 -- 172.21.15.83:0/760279542 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690014820 con 0x7f9698151480 2024-09-19T07:49:53.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.527+0000 7f969faf5640 1 -- 172.21.15.83:0/760279542 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f969814bb60 con 0x7f9698151480 2024-09-19T07:49:53.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.527+0000 7f969faf5640 1 -- 172.21.15.83:0/760279542 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9698150f40 con 0x7f9698151480 2024-09-19T07:49:53.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.527+0000 7f968e7fc640 1 -- 172.21.15.83:0/760279542 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f96900149c0 con 0x7f9698151480 2024-09-19T07:49:53.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.527+0000 7f968e7fc640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f967c0678c0 0x7f967c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:53.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.527+0000 7f968e7fc640 1 -- 172.21.15.83:0/760279542 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f9690087600 con 0x7f9698151480 2024-09-19T07:49:53.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.527+0000 7f969d891640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f967c0678c0 0x7f967c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:53.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.527+0000 7f969d891640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f967c0678c0 0x7f967c069d80 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f968800a9a0 tx=0x7f9688008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:53.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.531+0000 7f968e7fc640 1 -- 172.21.15.83:0/760279542 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f9690056ff0 con 0x7f9698151480 2024-09-19T07:49:53.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.727+0000 7f969faf5640 1 -- 172.21.15.83:0/760279542 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96980c4f30 con 0x7f9698151480 2024-09-19T07:49:53.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.727+0000 7f968e7fc640 1 -- 172.21.15.83:0/760279542 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) ==== 95+0+6542 (secure 0 0 0) 0x7f96900568f0 con 0x7f9698151480 2024-09-19T07:49:53.729 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:53.729 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":205,"btime":"2024-09-19T07:49:08:666589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6407,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:49:01.773306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8265,"qdb_cluster":[8265]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:49:53.730 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 205 2024-09-19T07:49:53.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.731+0000 7f969faf5640 1 -- 172.21.15.83:0/760279542 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f967c0678c0 msgr2=0x7f967c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:53.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.731+0000 7f969faf5640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f967c0678c0 0x7f967c069d80 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f968800a9a0 tx=0x7f9688008040 comp rx=0 tx=0).stop 2024-09-19T07:49:53.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.731+0000 7f969faf5640 1 -- 172.21.15.83:0/760279542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9698151480 msgr2=0x7f96980aace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:53.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.731+0000 7f969faf5640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9698151480 0x7f96980aace0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f9690007e60 tx=0x7f969000e4b0 comp rx=0 tx=0).stop 2024-09-19T07:49:53.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.731+0000 7f969faf5640 1 -- 172.21.15.83:0/760279542 shutdown_connections 2024-09-19T07:49:53.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.731+0000 7f969faf5640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f967c0678c0 0x7f967c069d80 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:53.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.731+0000 7f969faf5640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9698151480 0x7f96980aace0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:53.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.731+0000 7f969faf5640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96980c28e0 0x7f96980aa7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:53.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.731+0000 7f969faf5640 1 --2- 172.21.15.83:0/760279542 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9698071100 0x7f9698155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:53.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.731+0000 7f969faf5640 1 -- 172.21.15.83:0/760279542 >> 172.21.15.83:0/760279542 conn(0x7f969806bc80 msgr2=0x7f9698073ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:53.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.731+0000 7f969faf5640 1 -- 172.21.15.83:0/760279542 shutdown_connections 2024-09-19T07:49:53.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:53.731+0000 7f969faf5640 1 -- 172.21.15.83:0/760279542 wait complete. 2024-09-19T07:49:53.749 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:49:55.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:49:55.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef8303640 1 Processor -- start 2024-09-19T07:49:55.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef8303640 1 -- start start 2024-09-19T07:49:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef8303640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcef0153eb0 0x7fcef0154340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef8303640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef0150670 con 0x7fcef0150290 2024-09-19T07:49:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef8303640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef00c2880 con 0x7fcef0152b20 2024-09-19T07:49:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef8303640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fcef00716c0 con 0x7fcef0153eb0 2024-09-19T07:49:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef609f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcef0150290 0x7fcef00c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52666/0 (socket says 172.21.15.83:52666) 2024-09-19T07:49:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef609f640 1 -- 172.21.15.83:0/3722707144 learned_addr learned my addr 172.21.15.83:0/3722707144 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef68a0640 1 --2- 172.21.15.83:0/3722707144 >> v2:172.21.15.83:3300/0 conn(0x7fcef0153eb0 0x7fcef0154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef509d640 1 -- 172.21.15.83:0/3722707144 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3707065579 0 0) 0x7fcef0150670 con 0x7fcef0150290 2024-09-19T07:49:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef509d640 1 -- 172.21.15.83:0/3722707144 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcedc003530 con 0x7fcef0150290 2024-09-19T07:49:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef68a0640 1 -- 172.21.15.83:0/3722707144 >> v1:172.21.15.184:6789/0 conn(0x7fcef0152b20 legacy=0x7fcef00c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef68a0640 1 -- 172.21.15.83:0/3722707144 >> v1:172.21.15.83:6789/0 conn(0x7fcef0150290 legacy=0x7fcef00c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:55.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.887+0000 7fcef68a0640 1 -- 172.21.15.83:0/3722707144 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef00c2b60 con 0x7fcef0153eb0 2024-09-19T07:49:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef68a0640 1 --2- 172.21.15.83:0/3722707144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef0153eb0 0x7fcef0154340 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009d90 tx=0x7fcee801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8432d5c63a187061 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef509d640 1 -- 172.21.15.83:0/3722707144 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee80b2070 con 0x7fcef0153eb0 2024-09-19T07:49:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef509d640 1 -- 172.21.15.83:0/3722707144 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee80a5830 con 0x7fcef0153eb0 2024-09-19T07:49:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef509d640 1 -- 172.21.15.83:0/3722707144 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee80a5c00 con 0x7fcef0153eb0 2024-09-19T07:49:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 -- 172.21.15.83:0/3722707144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef0153eb0 msgr2=0x7fcef0154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 --2- 172.21.15.83:0/3722707144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef0153eb0 0x7fcef0154340 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009d90 tx=0x7fcee801c970 comp rx=0 tx=0).stop 2024-09-19T07:49:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 -- 172.21.15.83:0/3722707144 shutdown_connections 2024-09-19T07:49:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 --2- 172.21.15.83:0/3722707144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef0153eb0 0x7fcef0154340 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 -- 172.21.15.83:0/3722707144 >> 172.21.15.83:0/3722707144 conn(0x7fcef006bc80 msgr2=0x7fcef006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 -- 172.21.15.83:0/3722707144 shutdown_connections 2024-09-19T07:49:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 -- 172.21.15.83:0/3722707144 wait complete. 2024-09-19T07:49:55.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 Processor -- start 2024-09-19T07:49:55.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 -- start start 2024-09-19T07:49:55.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef0150290 0x7fcef00abc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:55.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcef0152b20 0x7fcef00c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:55.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef0153eb0 0x7fcef0141720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:55.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcef0154a00 con 0x7fcef0153eb0 2024-09-19T07:49:55.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcef0154880 con 0x7fcef0152b20 2024-09-19T07:49:55.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.891+0000 7fcef8303640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcef0160b70 con 0x7fcef0150290 2024-09-19T07:49:55.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef68a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef0153eb0 0x7fcef0141720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:55.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef68a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef0153eb0 0x7fcef0141720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55304/0 (socket says 172.21.15.83:55304) 2024-09-19T07:49:55.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef68a0640 1 -- 172.21.15.83:0/3534311685 learned_addr learned my addr 172.21.15.83:0/3534311685 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef589e640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcef0152b20 0x7fcef00c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef609f640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef0150290 0x7fcef00abc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef68a0640 1 -- 172.21.15.83:0/3534311685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef0150290 msgr2=0x7fcef00abc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef68a0640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef0150290 0x7fcef00abc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef68a0640 1 -- 172.21.15.83:0/3534311685 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcef0152b20 msgr2=0x7fcef00c59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef68a0640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcef0152b20 0x7fcef00c59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef68a0640 1 -- 172.21.15.83:0/3534311685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef00ab0d0 con 0x7fcef0153eb0 2024-09-19T07:49:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef68a0640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef0153eb0 0x7fcef0141720 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009d90 tx=0x7fcee8019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcee6ffd640 1 -- 172.21.15.83:0/3534311685 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee80b2050 con 0x7fcef0153eb0 2024-09-19T07:49:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcee6ffd640 1 -- 172.21.15.83:0/3534311685 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee80be070 con 0x7fcef0153eb0 2024-09-19T07:49:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef8303640 1 -- 172.21.15.83:0/3534311685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef00ab3b0 con 0x7fcef0153eb0 2024-09-19T07:49:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcee6ffd640 1 -- 172.21.15.83:0/3534311685 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee80ada90 con 0x7fcef0153eb0 2024-09-19T07:49:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef8303640 1 -- 172.21.15.83:0/3534311685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef00c6620 con 0x7fcef0153eb0 2024-09-19T07:49:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef8303640 1 -- 172.21.15.83:0/3534311685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef00c3e90 con 0x7fcef0153eb0 2024-09-19T07:49:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcee6ffd640 1 -- 172.21.15.83:0/3534311685 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcee80a56b0 con 0x7fcef0153eb0 2024-09-19T07:49:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcee6ffd640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fced40678f0 0x7fced4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef609f640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fced40678f0 0x7fced4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcee6ffd640 1 -- 172.21.15.83:0/3534311685 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fcee8121dd0 con 0x7fcef0153eb0 2024-09-19T07:49:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.895+0000 7fcef609f640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fced40678f0 0x7fced4069db0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fceec0011f0 tx=0x7fceec009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:55.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:55.899+0000 7fcee6ffd640 1 -- 172.21.15.83:0/3534311685 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fcee8016560 con 0x7fcef0153eb0 2024-09-19T07:49:56.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.095+0000 7fcef8303640 1 -- 172.21.15.83:0/3534311685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcef0154340 con 0x7fcef0153eb0 2024-09-19T07:49:56.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.095+0000 7fcee6ffd640 1 -- 172.21.15.83:0/3534311685 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) ==== 95+0+6542 (secure 0 0 0) 0x7fcee8123020 con 0x7fcef0153eb0 2024-09-19T07:49:56.094 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:56.094 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":205,"btime":"2024-09-19T07:49:08:666589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6407,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:49:01.773306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8265,"qdb_cluster":[8265]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:49:56.094 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 205 2024-09-19T07:49:56.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.099+0000 7fcef8303640 1 -- 172.21.15.83:0/3534311685 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fced40678f0 msgr2=0x7fced4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:56.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.099+0000 7fcef8303640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fced40678f0 0x7fced4069db0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fceec0011f0 tx=0x7fceec009040 comp rx=0 tx=0).stop 2024-09-19T07:49:56.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.099+0000 7fcef8303640 1 -- 172.21.15.83:0/3534311685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef0153eb0 msgr2=0x7fcef0141720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:56.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.099+0000 7fcef8303640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef0153eb0 0x7fcef0141720 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009d90 tx=0x7fcee8019710 comp rx=0 tx=0).stop 2024-09-19T07:49:56.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.099+0000 7fcef8303640 1 -- 172.21.15.83:0/3534311685 shutdown_connections 2024-09-19T07:49:56.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.099+0000 7fcef8303640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fced40678f0 0x7fced4069db0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:56.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.099+0000 7fcef8303640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcef0153eb0 0x7fcef0141720 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:56.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.099+0000 7fcef8303640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcef0152b20 0x7fcef00c59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:56.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.099+0000 7fcef8303640 1 --2- 172.21.15.83:0/3534311685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcef0150290 0x7fcef00abc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:56.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.099+0000 7fcef8303640 1 -- 172.21.15.83:0/3534311685 >> 172.21.15.83:0/3534311685 conn(0x7fcef006bc80 msgr2=0x7fcef0152f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:56.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.099+0000 7fcef8303640 1 -- 172.21.15.83:0/3534311685 shutdown_connections 2024-09-19T07:49:56.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:56.099+0000 7fcef8303640 1 -- 172.21.15.83:0/3534311685 wait complete. 2024-09-19T07:49:56.114 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:49:58.115 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:49:58.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d52349640 1 Processor -- start 2024-09-19T07:49:58.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d52349640 1 -- start start 2024-09-19T07:49:58.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d52349640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f2d4c0c28e0 0x7f2d4c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:58.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d52349640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d4c06dea0 0x7f2d4c06e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:58.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d52349640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d4c0a9e60 con 0x7f2d4c071100 2024-09-19T07:49:58.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d52349640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f2d4c071b50 con 0x7f2d4c0c28e0 2024-09-19T07:49:58.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d52349640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2d4c0c1ff0 con 0x7f2d4c06dea0 2024-09-19T07:49:58.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2d4c071100 0x7f2d4c150f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52678/0 (socket says 172.21.15.83:52678) 2024-09-19T07:49:58.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4bfff640 1 -- 172.21.15.83:0/3116830559 learned_addr learned my addr 172.21.15.83:0/3116830559 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:49:58.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2791373068 0 0) 0x7f2d4c0a9e60 con 0x7f2d4c071100 2024-09-19T07:49:58.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d43fff640 1 --2- 172.21.15.83:0/3116830559 >> v2:172.21.15.184:3300/0 conn(0x7f2d4c0c28e0 0x7f2d4c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:58.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d38003530 con 0x7f2d4c071100 2024-09-19T07:49:58.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d508e6640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d4c06dea0 0x7f2d4c06e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:58.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3872591038 0 0) 0x7f2d38003530 con 0x7f2d4c071100 2024-09-19T07:49:58.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d4c06dea0 msgr2=0x7f2d4c06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d4c06dea0 0x7f2d4c06e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 >> v2:172.21.15.184:3300/0 conn(0x7f2d4c0c28e0 msgr2=0x7f2d4c1545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 --2- 172.21.15.83:0/3116830559 >> v2:172.21.15.184:3300/0 conn(0x7f2d4c0c28e0 0x7f2d4c1545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d4c0aa140 con 0x7f2d4c071100 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f2d3c001680 con 0x7f2d4c071100 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 >> v1:172.21.15.83:6789/0 conn(0x7f2d4c071100 legacy=0x7f2d4c150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d380039c0 0x7f2d38003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d38004810 0x7f2d38004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d4c06dea0 0x7f2d380051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c071b50 con 0x7f2d38004810 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c0c1ff0 con 0x7f2d380039c0 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3800bb60 con 0x7f2d4c06dea0 2024-09-19T07:49:58.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d43fff640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d38004810 0x7f2d38004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:58.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d508e6640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d4c06dea0 0x7f2d380051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:58.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4bfff640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d380039c0 0x7f2d38003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:58.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d43fff640 1 -- 172.21.15.83:0/3116830559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d4c06dea0 msgr2=0x7f2d380051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:58.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d43fff640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d4c06dea0 0x7f2d380051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:58.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d43fff640 1 -- 172.21.15.83:0/3116830559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d380039c0 msgr2=0x7f2d38003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:49:58.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d43fff640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d380039c0 0x7f2d38003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:58.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d43fff640 1 -- 172.21.15.83:0/3116830559 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d38003530 con 0x7f2d38004810 2024-09-19T07:49:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.251+0000 7f2d4bfff640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d380039c0 0x7f2d38003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d43fff640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d38004810 0x7f2d38004c90 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400ba00 tx=0x7f2d3401c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649e5ced1f55d2f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3400b600 con 0x7f2d38004810 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d340b1070 con 0x7f2d38004810 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d4b7fe640 1 -- 172.21.15.83:0/3116830559 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d340ac840 con 0x7f2d38004810 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 -- 172.21.15.83:0/3116830559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d38004810 msgr2=0x7f2d38004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 --2- 172.21.15.83:0/3116830559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d38004810 0x7f2d38004c90 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400ba00 tx=0x7f2d3401c680 comp rx=0 tx=0).stop 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d508e6640 1 -- 172.21.15.83:0/3116830559 reap_dead start 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 -- 172.21.15.83:0/3116830559 shutdown_connections 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 -- 172.21.15.83:0/3116830559 >> 172.21.15.83:0/3116830559 conn(0x7f2d4c06bc80 msgr2=0x7f2d4c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 -- 172.21.15.83:0/3116830559 shutdown_connections 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 -- 172.21.15.83:0/3116830559 wait complete. 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 Processor -- start 2024-09-19T07:49:58.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 -- start start 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d38004810 0x7f2d4c15a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d4c14c770 0x7f2d4c0c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d4c14b5c0 0x7f2d4c14ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c071660 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c0714e0 con 0x7f2d4c14c770 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d52349640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2d4c15e440 con 0x7f2d38004810 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d508e6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d4c14b5c0 0x7f2d4c14ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d508e6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d4c14b5c0 0x7f2d4c14ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55316/0 (socket says 172.21.15.83:55316) 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d508e6640 1 -- 172.21.15.83:0/2021889759 learned_addr learned my addr 172.21.15.83:0/2021889759 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d4bfff640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d38004810 0x7f2d4c15a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.255+0000 7f2d43fff640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d4c14c770 0x7f2d4c0c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d508e6640 1 -- 172.21.15.83:0/2021889759 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d38004810 msgr2=0x7f2d4c15a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d508e6640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d38004810 0x7f2d4c15a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:58.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d508e6640 1 -- 172.21.15.83:0/2021889759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d4c14c770 msgr2=0x7f2d4c0c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:58.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d508e6640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d4c14c770 0x7f2d4c0c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:58.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d508e6640 1 -- 172.21.15.83:0/2021889759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d4c14c490 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d43fff640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d4c14c770 0x7f2d4c0c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:49:58.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d508e6640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d4c14b5c0 0x7f2d4c14ba50 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c1545b0 tx=0x7f2d4400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:58.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d497fa640 1 -- 172.21.15.83:0/2021889759 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4400fe00 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d497fa640 1 -- 172.21.15.83:0/2021889759 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d4401a070 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d52349640 1 -- 172.21.15.83:0/2021889759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d4c157690 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d497fa640 1 -- 172.21.15.83:0/2021889759 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44014630 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d52349640 1 -- 172.21.15.83:0/2021889759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d4c157ec0 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d52349640 1 -- 172.21.15.83:0/2021889759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d4c157480 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d497fa640 1 -- 172.21.15.83:0/2021889759 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2d440147d0 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d497fa640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d2c0678c0 0x7f2d2c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:49:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d497fa640 1 -- 172.21.15.83:0/2021889759 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7f2d440871b0 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d4bfff640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d2c0678c0 0x7f2d2c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:49:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.259+0000 7f2d4bfff640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d2c0678c0 0x7f2d2c069d80 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c001ec0 tx=0x7f2d3c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:49:58.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.263+0000 7f2d497fa640 1 -- 172.21.15.83:0/2021889759 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f2d44056b90 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.455+0000 7f2d52349640 1 -- 172.21.15.83:0/2021889759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d4c1545e0 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.455+0000 7f2d497fa640 1 -- 172.21.15.83:0/2021889759 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) ==== 95+0+6542 (secure 0 0 0) 0x7f2d44056490 con 0x7f2d4c14b5c0 2024-09-19T07:49:58.453 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:49:58.453 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":205,"btime":"2024-09-19T07:49:08:666589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6407,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:49:01.773306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8265,"qdb_cluster":[8265]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:49:58.453 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 205 2024-09-19T07:49:58.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.459+0000 7f2d52349640 1 -- 172.21.15.83:0/2021889759 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d2c0678c0 msgr2=0x7f2d2c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:58.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.459+0000 7f2d52349640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d2c0678c0 0x7f2d2c069d80 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c001ec0 tx=0x7f2d3c009040 comp rx=0 tx=0).stop 2024-09-19T07:49:58.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.459+0000 7f2d52349640 1 -- 172.21.15.83:0/2021889759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d4c14b5c0 msgr2=0x7f2d4c14ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:49:58.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.459+0000 7f2d52349640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d4c14b5c0 0x7f2d4c14ba50 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c1545b0 tx=0x7f2d4400f520 comp rx=0 tx=0).stop 2024-09-19T07:49:58.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.459+0000 7f2d52349640 1 -- 172.21.15.83:0/2021889759 shutdown_connections 2024-09-19T07:49:58.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.459+0000 7f2d52349640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f2d2c0678c0 0x7f2d2c069d80 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:58.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.459+0000 7f2d52349640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d4c14b5c0 0x7f2d4c14ba50 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:58.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.459+0000 7f2d52349640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d4c14c770 0x7f2d4c0c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:58.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.459+0000 7f2d52349640 1 --2- 172.21.15.83:0/2021889759 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d38004810 0x7f2d4c15a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:49:58.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.459+0000 7f2d52349640 1 -- 172.21.15.83:0/2021889759 >> 172.21.15.83:0/2021889759 conn(0x7f2d4c06bc80 msgr2=0x7f2d4c14ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:49:58.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.459+0000 7f2d52349640 1 -- 172.21.15.83:0/2021889759 shutdown_connections 2024-09-19T07:49:58.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:49:58.459+0000 7f2d52349640 1 -- 172.21.15.83:0/2021889759 wait complete. 2024-09-19T07:49:58.472 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:50:00.473 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:50:00.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.607+0000 7fe1387c0640 1 Processor -- start 2024-09-19T07:50:00.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.607+0000 7fe1387c0640 1 -- start start 2024-09-19T07:50:00.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe1300c6120 0x7fe1300c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:00.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe130155d70 0x7fe1301575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:00.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1301592f0 0x7fe130157b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:00.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe1300c9930 con 0x7fe1301592f0 2024-09-19T07:50:00.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe130156200 con 0x7fe1300c6120 2024-09-19T07:50:00.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe1301564e0 con 0x7fe130155d70 2024-09-19T07:50:00.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe136d5d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1301592f0 0x7fe130157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:00.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe136d5d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1301592f0 0x7fe130157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47286/0 (socket says 172.21.15.83:47286) 2024-09-19T07:50:00.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe136d5d640 1 -- 172.21.15.83:0/1044651687 learned_addr learned my addr 172.21.15.83:0/1044651687 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:00.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe13655c640 1 --2- 172.21.15.83:0/1044651687 >> v2:172.21.15.184:3300/0 conn(0x7fe1300c6120 0x7fe1300c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:00.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe135d5b640 1 --2- 172.21.15.83:0/1044651687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe130155d70 0x7fe1301575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:00.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe136d5d640 1 -- 172.21.15.83:0/1044651687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe130155d70 msgr2=0x7fe1301575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:00.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe136d5d640 1 --2- 172.21.15.83:0/1044651687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe130155d70 0x7fe1301575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:00.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe136d5d640 1 -- 172.21.15.83:0/1044651687 >> v2:172.21.15.184:3300/0 conn(0x7fe1300c6120 msgr2=0x7fe1300c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:00.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe136d5d640 1 --2- 172.21.15.83:0/1044651687 >> v2:172.21.15.184:3300/0 conn(0x7fe1300c6120 0x7fe1300c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:00.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe136d5d640 1 -- 172.21.15.83:0/1044651687 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1301567c0 con 0x7fe1301592f0 2024-09-19T07:50:00.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe13655c640 1 --2- 172.21.15.83:0/1044651687 >> v2:172.21.15.184:3300/0 conn(0x7fe1300c6120 0x7fe1300c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:00.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe135d5b640 1 --2- 172.21.15.83:0/1044651687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe130155d70 0x7fe1301575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:00.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe136d5d640 1 --2- 172.21.15.83:0/1044651687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1301592f0 0x7fe130157b10 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fe13008d540 tx=0x7fe12801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408078a0e2de48a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:00.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe13555a640 1 -- 172.21.15.83:0/1044651687 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1280b0070 con 0x7fe1301592f0 2024-09-19T07:50:00.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe13555a640 1 -- 172.21.15.83:0/1044651687 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1280ab600 con 0x7fe1301592f0 2024-09-19T07:50:00.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe13555a640 1 -- 172.21.15.83:0/1044651687 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1280ab9d0 con 0x7fe1301592f0 2024-09-19T07:50:00.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 -- 172.21.15.83:0/1044651687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1301592f0 msgr2=0x7fe130157b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:00.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 --2- 172.21.15.83:0/1044651687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1301592f0 0x7fe130157b10 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fe13008d540 tx=0x7fe12801c520 comp rx=0 tx=0).stop 2024-09-19T07:50:00.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 -- 172.21.15.83:0/1044651687 shutdown_connections 2024-09-19T07:50:00.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 --2- 172.21.15.83:0/1044651687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1301592f0 0x7fe130157b10 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:00.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 --2- 172.21.15.83:0/1044651687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe130155d70 0x7fe1301575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:00.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 --2- 172.21.15.83:0/1044651687 >> v2:172.21.15.184:3300/0 conn(0x7fe1300c6120 0x7fe1300c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:00.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 -- 172.21.15.83:0/1044651687 >> 172.21.15.83:0/1044651687 conn(0x7fe13006bc80 msgr2=0x7fe13006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:00.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 -- 172.21.15.83:0/1044651687 shutdown_connections 2024-09-19T07:50:00.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.611+0000 7fe1387c0640 1 -- 172.21.15.83:0/1044651687 wait complete. 2024-09-19T07:50:00.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe1387c0640 1 Processor -- start 2024-09-19T07:50:00.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe1387c0640 1 -- start start 2024-09-19T07:50:00.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe1387c0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1300c6120 0x7fe1300abd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:00.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe1387c0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1301592f0 0x7fe1301f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:00.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe1387c0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe1300cb550 0x7fe1300ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:00.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe1387c0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe1300c8b40 con 0x7fe1300c6120 2024-09-19T07:50:00.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe1387c0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe1300c89c0 con 0x7fe1300cb550 2024-09-19T07:50:00.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe1387c0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe1300c8cc0 con 0x7fe1301592f0 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe13655c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1300c6120 0x7fe1300abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe13655c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1300c6120 0x7fe1300abd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47302/0 (socket says 172.21.15.83:47302) 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe13655c640 1 -- 172.21.15.83:0/3825988402 learned_addr learned my addr 172.21.15.83:0/3825988402 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe135d5b640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1301592f0 0x7fe1301f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe136d5d640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe1300cb550 0x7fe1300ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe13655c640 1 -- 172.21.15.83:0/3825988402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1301592f0 msgr2=0x7fe1301f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe13655c640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1301592f0 0x7fe1301f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe13655c640 1 -- 172.21.15.83:0/3825988402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe1300cb550 msgr2=0x7fe1300ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe13655c640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe1300cb550 0x7fe1300ada30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe13655c640 1 -- 172.21.15.83:0/3825988402 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1300cb9e0 con 0x7fe1300c6120 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe13655c640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1300c6120 0x7fe1300abd50 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00cee0 tx=0x7fe12c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe126ffd640 1 -- 172.21.15.83:0/3825988402 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c01a070 con 0x7fe1300c6120 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe126ffd640 1 -- 172.21.15.83:0/3825988402 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe12c00d840 con 0x7fe1300c6120 2024-09-19T07:50:00.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe1387c0640 1 -- 172.21.15.83:0/3825988402 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1301f2890 con 0x7fe1300c6120 2024-09-19T07:50:00.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe126ffd640 1 -- 172.21.15.83:0/3825988402 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c00dc10 con 0x7fe1300c6120 2024-09-19T07:50:00.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe1387c0640 1 -- 172.21.15.83:0/3825988402 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1300cad00 con 0x7fe1300c6120 2024-09-19T07:50:00.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.615+0000 7fe1387c0640 1 -- 172.21.15.83:0/3825988402 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1300c8f60 con 0x7fe1300c6120 2024-09-19T07:50:00.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.619+0000 7fe126ffd640 1 -- 172.21.15.83:0/3825988402 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe12c01d070 con 0x7fe1300c6120 2024-09-19T07:50:00.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.619+0000 7fe126ffd640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe1140678c0 0x7fe114069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:00.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.619+0000 7fe135d5b640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe1140678c0 0x7fe114069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:00.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.619+0000 7fe126ffd640 1 -- 172.21.15.83:0/3825988402 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fe12c087480 con 0x7fe1300c6120 2024-09-19T07:50:00.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.619+0000 7fe135d5b640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe1140678c0 0x7fe114069d80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe1301546f0 tx=0x7fe120008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:00.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.623+0000 7fe126ffd640 1 -- 172.21.15.83:0/3825988402 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe12c0568a0 con 0x7fe1300c6120 2024-09-19T07:50:00.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.803+0000 7fe1387c0640 1 -- 172.21.15.83:0/3825988402 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1300abb30 con 0x7fe1300c6120 2024-09-19T07:50:00.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.803+0000 7fe126ffd640 1 -- 172.21.15.83:0/3825988402 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) ==== 95+0+6542 (secure 0 0 0) 0x7fe12c05aba0 con 0x7fe1300c6120 2024-09-19T07:50:00.802 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:50:00.802 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":205,"btime":"2024-09-19T07:49:08:666589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6407,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:49:01.773306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8265,"qdb_cluster":[8265]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:50:00.803 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 205 2024-09-19T07:50:00.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.807+0000 7fe1387c0640 1 -- 172.21.15.83:0/3825988402 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe1140678c0 msgr2=0x7fe114069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:00.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.807+0000 7fe1387c0640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe1140678c0 0x7fe114069d80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe1301546f0 tx=0x7fe120008040 comp rx=0 tx=0).stop 2024-09-19T07:50:00.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.807+0000 7fe1387c0640 1 -- 172.21.15.83:0/3825988402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1300c6120 msgr2=0x7fe1300abd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:00.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.807+0000 7fe1387c0640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1300c6120 0x7fe1300abd50 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00cee0 tx=0x7fe12c00a710 comp rx=0 tx=0).stop 2024-09-19T07:50:00.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.807+0000 7fe1387c0640 1 -- 172.21.15.83:0/3825988402 shutdown_connections 2024-09-19T07:50:00.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.807+0000 7fe1387c0640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe1140678c0 0x7fe114069d80 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:00.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.807+0000 7fe1387c0640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe1300cb550 0x7fe1300ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:00.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.807+0000 7fe1387c0640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1301592f0 0x7fe1301f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:00.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.807+0000 7fe1387c0640 1 --2- 172.21.15.83:0/3825988402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1300c6120 0x7fe1300abd50 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:00.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.807+0000 7fe1387c0640 1 -- 172.21.15.83:0/3825988402 >> 172.21.15.83:0/3825988402 conn(0x7fe13006bc80 msgr2=0x7fe1301575b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:00.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.807+0000 7fe1387c0640 1 -- 172.21.15.83:0/3825988402 shutdown_connections 2024-09-19T07:50:00.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:00.807+0000 7fe1387c0640 1 -- 172.21.15.83:0/3825988402 wait complete. 2024-09-19T07:50:00.821 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:50:02.821 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:50:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.955+0000 7fd7f3ab7640 1 Processor -- start 2024-09-19T07:50:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.955+0000 7fd7f3ab7640 1 -- start start 2024-09-19T07:50:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f3ab7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd7ec1545e0 0x7fd7ec06e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f3ab7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7ec154a70 con 0x7fd7ec0c28e0 2024-09-19T07:50:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f3ab7640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7ec0a9e60 con 0x7fd7ec071100 2024-09-19T07:50:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f3ab7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ec071b50 con 0x7fd7ec1545e0 2024-09-19T07:50:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f2054640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd7ec1545e0 0x7fd7ec06e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f1052640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd7ec0c28e0 0x7fd7ec06dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48846/0 (socket says 172.21.15.83:48846) 2024-09-19T07:50:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f1052640 1 -- 172.21.15.83:0/968798848 learned_addr learned my addr 172.21.15.83:0/968798848 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:50:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f2054640 1 -- 172.21.15.83:0/968798848 >> v1:172.21.15.184:6789/0 conn(0x7fd7ec071100 legacy=0x7fd7ec150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f2054640 1 -- 172.21.15.83:0/968798848 >> v1:172.21.15.83:6789/0 conn(0x7fd7ec0c28e0 legacy=0x7fd7ec06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f2054640 1 -- 172.21.15.83:0/968798848 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ec0aa140 con 0x7fd7ec1545e0 2024-09-19T07:50:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f0851640 1 -- 172.21.15.83:0/968798848 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 35764885 0 0) 0x7fd7ec154a70 con 0x7fd7ec0c28e0 2024-09-19T07:50:02.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f2054640 1 --2- 172.21.15.83:0/968798848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7ec1545e0 0x7fd7ec06e5b0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009d90 tx=0x7fd7e401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e455ca1fe73e47f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:02.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f0851640 1 -- 172.21.15.83:0/968798848 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40b2070 con 0x7fd7ec1545e0 2024-09-19T07:50:02.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f0851640 1 -- 172.21.15.83:0/968798848 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e40a5850 con 0x7fd7ec1545e0 2024-09-19T07:50:02.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f0851640 1 -- 172.21.15.83:0/968798848 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40a5c20 con 0x7fd7ec1545e0 2024-09-19T07:50:02.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/968798848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7ec1545e0 msgr2=0x7fd7ec06e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:02.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f3ab7640 1 --2- 172.21.15.83:0/968798848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7ec1545e0 0x7fd7ec06e5b0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009d90 tx=0x7fd7e401c8a0 comp rx=0 tx=0).stop 2024-09-19T07:50:02.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/968798848 shutdown_connections 2024-09-19T07:50:02.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f3ab7640 1 --2- 172.21.15.83:0/968798848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7ec1545e0 0x7fd7ec06e5b0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:02.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/968798848 >> 172.21.15.83:0/968798848 conn(0x7fd7ec06bc80 msgr2=0x7fd7ec06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:02.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/968798848 shutdown_connections 2024-09-19T07:50:02.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.959+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/968798848 wait complete. 2024-09-19T07:50:02.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f3ab7640 1 Processor -- start 2024-09-19T07:50:02.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f3ab7640 1 -- start start 2024-09-19T07:50:02.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f3ab7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7ec071100 0x7fd7ec145b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:02.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f3ab7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7ec0c28e0 0x7fd7ec1519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:02.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f3ab7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7ec1545e0 0x7fd7ec151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:02.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f3ab7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec1517d0 con 0x7fd7ec1545e0 2024-09-19T07:50:02.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f3ab7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec151650 con 0x7fd7ec0c28e0 2024-09-19T07:50:02.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f3ab7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ec0714e0 con 0x7fd7ec071100 2024-09-19T07:50:02.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f2054640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7ec1545e0 0x7fd7ec151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:02.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f2054640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7ec1545e0 0x7fd7ec151f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47320/0 (socket says 172.21.15.83:47320) 2024-09-19T07:50:02.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f2054640 1 -- 172.21.15.83:0/3847254686 learned_addr learned my addr 172.21.15.83:0/3847254686 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f1052640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7ec0c28e0 0x7fd7ec1519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f1853640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7ec071100 0x7fd7ec145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f2054640 1 -- 172.21.15.83:0/3847254686 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7ec071100 msgr2=0x7fd7ec145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f2054640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7ec071100 0x7fd7ec145b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f2054640 1 -- 172.21.15.83:0/3847254686 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7ec0c28e0 msgr2=0x7fd7ec1519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f2054640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7ec0c28e0 0x7fd7ec1519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f2054640 1 -- 172.21.15.83:0/3847254686 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ec0c6c70 con 0x7fd7ec1545e0 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f2054640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7ec1545e0 0x7fd7ec151f10 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009d90 tx=0x7fd7e4019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7e27fc640 1 -- 172.21.15.83:0/3847254686 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40bd020 con 0x7fd7ec1545e0 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7e27fc640 1 -- 172.21.15.83:0/3847254686 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e40ad6c0 con 0x7fd7ec1545e0 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7e27fc640 1 -- 172.21.15.83:0/3847254686 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40b2040 con 0x7fd7ec1545e0 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/3847254686 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ec0c6f50 con 0x7fd7ec1545e0 2024-09-19T07:50:02.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/3847254686 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ec14dda0 con 0x7fd7ec1545e0 2024-09-19T07:50:02.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.963+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/3847254686 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ec14e0b0 con 0x7fd7ec1545e0 2024-09-19T07:50:02.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.967+0000 7fd7e27fc640 1 -- 172.21.15.83:0/3847254686 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd7e40c0020 con 0x7fd7ec1545e0 2024-09-19T07:50:02.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.967+0000 7fd7e27fc640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7d00678f0 0x7fd7d0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:02.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.967+0000 7fd7e27fc640 1 -- 172.21.15.83:0/3847254686 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 10077+0+0 (secure 0 0 0) 0x7fd7e4121cc0 con 0x7fd7ec1545e0 2024-09-19T07:50:02.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.967+0000 7fd7f1853640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7d00678f0 0x7fd7d0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:02.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.967+0000 7fd7f1853640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7d00678f0 0x7fd7d0069db0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc00a750 tx=0x7fd7dc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:02.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:02.971+0000 7fd7e27fc640 1 -- 172.21.15.83:0/3847254686 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fd7e40c7070 con 0x7fd7ec1545e0 2024-09-19T07:50:03.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/3847254686 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7ec150f40 con 0x7fd7ec1545e0 2024-09-19T07:50:03.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7e27fc640 1 -- 172.21.15.83:0/3847254686 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) ==== 95+0+6542 (secure 0 0 0) 0x7fd7e4016520 con 0x7fd7ec1545e0 2024-09-19T07:50:03.163 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:50:03.163 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":205,"btime":"2024-09-19T07:49:08:666589+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6407,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:49:01.773306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8265":{"gid":8265,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":11,"addr":"172.21.15.83:6837/3426675376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3426675376},{"type":"v1","addr":"172.21.15.83:6837","nonce":3426675376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8265,"qdb_cluster":[8265]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:50:03.163 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 205 2024-09-19T07:50:03.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/3847254686 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7d00678f0 msgr2=0x7fd7d0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:03.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7d00678f0 0x7fd7d0069db0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc00a750 tx=0x7fd7dc008040 comp rx=0 tx=0).stop 2024-09-19T07:50:03.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/3847254686 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7ec1545e0 msgr2=0x7fd7ec151f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:03.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7ec1545e0 0x7fd7ec151f10 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009d90 tx=0x7fd7e4019640 comp rx=0 tx=0).stop 2024-09-19T07:50:03.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/3847254686 shutdown_connections 2024-09-19T07:50:03.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fd7d00678f0 0x7fd7d0069db0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:03.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7ec1545e0 0x7fd7ec151f10 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:03.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7ec0c28e0 0x7fd7ec1519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:03.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 --2- 172.21.15.83:0/3847254686 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7ec071100 0x7fd7ec145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:03.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/3847254686 >> 172.21.15.83:0/3847254686 conn(0x7fd7ec06bc80 msgr2=0x7fd7ec14e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:03.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/3847254686 shutdown_connections 2024-09-19T07:50:03.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.167+0000 7fd7f3ab7640 1 -- 172.21.15.83:0/3847254686 wait complete. 2024-09-19T07:50:03.182 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-19T07:50:03.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-19T07:50:03.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f1993437640 1 Processor -- start 2024-09-19T07:50:03.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f1993437640 1 -- start start 2024-09-19T07:50:03.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f1993437640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f198c151540 0x7f198c0c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:03.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f1993437640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198c0c8220 con 0x7f198c0c4270 2024-09-19T07:50:03.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f1993437640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198c0c8500 con 0x7f198c070be0 2024-09-19T07:50:03.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f1993437640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f198c0c8b50 con 0x7f198c151540 2024-09-19T07:50:03.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f19911d3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f198c0c4270 0x7f198c06de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48860/0 (socket says 172.21.15.83:48860) 2024-09-19T07:50:03.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f19911d3640 1 -- 172.21.15.83:0/2493181764 learned_addr learned my addr 172.21.15.83:0/2493181764 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:50:03.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f19919d4640 1 --2- 172.21.15.83:0/2493181764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f198c151540 0x7f198c0c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:03.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3094792826 0 0) 0x7f198c0c8220 con 0x7f198c0c4270 2024-09-19T07:50:03.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1974003530 con 0x7f198c0c4270 2024-09-19T07:50:03.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 29334803 0 0) 0x7f1974003530 con 0x7f198c0c4270 2024-09-19T07:50:03.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f198c151540 msgr2=0x7f198c0c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:03.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 --2- 172.21.15.83:0/2493181764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f198c151540 0x7f198c0c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:03.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 >> v1:172.21.15.184:6789/0 conn(0x7f198c070be0 legacy=0x7f198c150e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:03.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198c0c87e0 con 0x7f198c0c4270 2024-09-19T07:50:03.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f1984001680 con 0x7f198c0c4270 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 >> v1:172.21.15.83:6789/0 conn(0x7f198c0c4270 legacy=0x7f198c06de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 --2- 172.21.15.83:0/2493181764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740039f0 0x7f1974003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 --2- 172.21.15.83:0/2493181764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f198c151540 0x7f1974004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 --2- 172.21.15.83:0/2493181764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1974004d50 0x7f19740051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f198c0c8b50 con 0x7f19740039f0 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f197400bb00 con 0x7f1974004d50 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f197400bd00 con 0x7f198c151540 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f19911d3640 1 --2- 172.21.15.83:0/2493181764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740039f0 0x7f1974003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f19909d2640 1 --2- 172.21.15.83:0/2493181764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f198c151540 0x7f1974004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f19919d4640 1 --2- 172.21.15.83:0/2493181764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1974004d50 0x7f19740051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f19911d3640 1 -- 172.21.15.83:0/2493181764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f198c151540 msgr2=0x7f1974004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f19911d3640 1 --2- 172.21.15.83:0/2493181764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f198c151540 0x7f1974004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:03.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f19911d3640 1 -- 172.21.15.83:0/2493181764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1974004d50 msgr2=0x7f19740051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:03.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f19911d3640 1 --2- 172.21.15.83:0/2493181764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1974004d50 0x7f19740051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:03.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.907+0000 7f19911d3640 1 -- 172.21.15.83:0/2493181764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1974003530 con 0x7f19740039f0 2024-09-19T07:50:03.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f19911d3640 1 --2- 172.21.15.83:0/2493181764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740039f0 0x7f1974003e10 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f198400c210 tx=0x7f198401cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45caba4ce9f08f72 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:03.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1984007d40 con 0x7f19740039f0 2024-09-19T07:50:03.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19840a9070 con 0x7f19740039f0 2024-09-19T07:50:03.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f198bfff640 1 -- 172.21.15.83:0/2493181764 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1984005d40 con 0x7f19740039f0 2024-09-19T07:50:03.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f1993437640 1 -- 172.21.15.83:0/2493181764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740039f0 msgr2=0x7f1974003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:03.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f1993437640 1 --2- 172.21.15.83:0/2493181764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740039f0 0x7f1974003e10 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f198400c210 tx=0x7f198401cbe0 comp rx=0 tx=0).stop 2024-09-19T07:50:03.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f19919d4640 1 -- 172.21.15.83:0/2493181764 reap_dead start 2024-09-19T07:50:03.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f1993437640 1 -- 172.21.15.83:0/2493181764 shutdown_connections 2024-09-19T07:50:03.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f1993437640 1 -- 172.21.15.83:0/2493181764 >> 172.21.15.83:0/2493181764 conn(0x7f198c06ffa0 msgr2=0x7f198c0703b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:03.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f1993437640 1 -- 172.21.15.83:0/2493181764 shutdown_connections 2024-09-19T07:50:03.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f1993437640 1 -- 172.21.15.83:0/2493181764 wait complete. 2024-09-19T07:50:03.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f1993437640 1 Processor -- start 2024-09-19T07:50:03.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.911+0000 7f1993437640 1 -- start start 2024-09-19T07:50:03.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1993437640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f19740039f0 0x7f198c0af110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1993437640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f198c149f20 0x7f198c1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1993437640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f198c1f24b0 0x7f198c1f2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1993437640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f198c15fd20 con 0x7f198c1f24b0 2024-09-19T07:50:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1993437640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f198c154a10 con 0x7f198c149f20 2024-09-19T07:50:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1993437640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f198c15fea0 con 0x7f19740039f0 2024-09-19T07:50:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19919d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f198c1f24b0 0x7f198c1f2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19919d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f198c1f24b0 0x7f198c1f2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47338/0 (socket says 172.21.15.83:47338) 2024-09-19T07:50:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19919d4640 1 -- 172.21.15.83:0/3745042830 learned_addr learned my addr 172.21.15.83:0/3745042830 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19911d3640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f19740039f0 0x7f198c0af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19909d2640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f198c149f20 0x7f198c1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19919d4640 1 -- 172.21.15.83:0/3745042830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f19740039f0 msgr2=0x7f198c0af110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:03.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19919d4640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f19740039f0 0x7f198c0af110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19919d4640 1 -- 172.21.15.83:0/3745042830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f198c149f20 msgr2=0x7f198c1573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19919d4640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f198c149f20 0x7f198c1573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19919d4640 1 -- 172.21.15.83:0/3745042830 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198c15ca40 con 0x7f198c1f24b0 2024-09-19T07:50:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19919d4640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f198c1f24b0 0x7f198c1f2930 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f198c0af980 tx=0x7f197c011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1989ffb640 1 -- 172.21.15.83:0/3745042830 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c019020 con 0x7f198c1f24b0 2024-09-19T07:50:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1989ffb640 1 -- 172.21.15.83:0/3745042830 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f197c003620 con 0x7f198c1f24b0 2024-09-19T07:50:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1993437640 1 -- 172.21.15.83:0/3745042830 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f198c0af720 con 0x7f198c1f24b0 2024-09-19T07:50:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1989ffb640 1 -- 172.21.15.83:0/3745042830 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c003a10 con 0x7f198c1f24b0 2024-09-19T07:50:03.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1993437640 1 -- 172.21.15.83:0/3745042830 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f198c14a900 con 0x7f198c1f24b0 2024-09-19T07:50:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1993437640 1 -- 172.21.15.83:0/3745042830 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f198c150e30 con 0x7f198c1f24b0 2024-09-19T07:50:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1989ffb640 1 -- 172.21.15.83:0/3745042830 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f197c01c020 con 0x7f198c1f24b0 2024-09-19T07:50:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1989ffb640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f19780678f0 0x7f1978069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f1989ffb640 1 -- 172.21.15.83:0/3745042830 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10163+0+0 (secure 0 0 0) 0x7f197c087450 con 0x7f198c1f24b0 2024-09-19T07:50:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19911d3640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f19780678f0 0x7f1978069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:03.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.915+0000 7f19911d3640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f19780678f0 0x7f1978069db0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f19840ab000 tx=0x7f19840b1040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:03.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:03.919+0000 7f1989ffb640 1 -- 172.21.15.83:0/3745042830 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f197c023030 con 0x7f198c1f24b0 2024-09-19T07:50:04.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.079+0000 7f1993437640 1 -- 172.21.15.83:0/3745042830 --> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f198c0c3cf0 con 0x7f19780678f0 2024-09-19T07:50:04.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.095+0000 7f1989ffb640 1 -- 172.21.15.83:0/3745042830 <== mgr.4101 v2:172.21.15.83:6832/1237718513 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+495212 (secure 0 0 0) 0x7f1984135e80 con 0x7f19780678f0 2024-09-19T07:50:04.097 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:50:04.102 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-19T07:50:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.103+0000 7f1993437640 1 -- 172.21.15.83:0/3745042830 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f19780678f0 msgr2=0x7f1978069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.103+0000 7f1993437640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f19780678f0 0x7f1978069db0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f19840ab000 tx=0x7f19840b1040 comp rx=0 tx=0).stop 2024-09-19T07:50:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.103+0000 7f1993437640 1 -- 172.21.15.83:0/3745042830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f198c1f24b0 msgr2=0x7f198c1f2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.103+0000 7f1993437640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f198c1f24b0 0x7f198c1f2930 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f198c0af980 tx=0x7f197c011920 comp rx=0 tx=0).stop 2024-09-19T07:50:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.103+0000 7f1993437640 1 -- 172.21.15.83:0/3745042830 shutdown_connections 2024-09-19T07:50:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.103+0000 7f1993437640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f19780678f0 0x7f1978069db0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.103+0000 7f1993437640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f198c1f24b0 0x7f198c1f2930 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.103+0000 7f1993437640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f198c149f20 0x7f198c1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.103+0000 7f1993437640 1 --2- 172.21.15.83:0/3745042830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f19740039f0 0x7f198c0af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.103+0000 7f1993437640 1 -- 172.21.15.83:0/3745042830 >> 172.21.15.83:0/3745042830 conn(0x7f198c06ffa0 msgr2=0x7f198c1526e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:04.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.103+0000 7f1993437640 1 -- 172.21.15.83:0/3745042830 shutdown_connections 2024-09-19T07:50:04.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.103+0000 7f1993437640 1 -- 172.21.15.83:0/3745042830 wait complete. 2024-09-19T07:50:04.117 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":980,"stamp":"2024-09-19T07:50:03.634577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39797278,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3062,"num_read_kb":412170,"num_write":3183053,"num_write_kb":38996435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":596876,"ondisk_log_size":596876,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7752320,"kb_used_data":84416,"kb_used_omap":12,"kb_used_meta":7667763,"kb_avail":747222400,"statfs":{"total":773094113280,"available":765155737600,"internally_reserved":0,"allocated":86441984,"data_stored":83166972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":7851789873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004646"},"pg_stats":[{"pgid":"5.18","version":"64'4995","reported_seq":5108,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.421051+0000","last_change":"2024-09-19T07:49:37.421108+0000","last_active":"2024-09-19T07:49:37.421051+0000","last_peered":"2024-09-19T07:49:37.421051+0000","last_clean":"2024-09-19T07:49:37.421051+0000","last_became_active":"2024-09-19T07:19:40.357741+0000","last_became_peered":"2024-09-19T07:19:40.357741+0000","last_unstale":"2024-09-19T07:49:37.421051+0000","last_undegraded":"2024-09-19T07:49:37.421051+0000","last_fullsized":"2024-09-19T07:49:37.421051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4995","last_scrub_stamp":"2024-09-19T07:49:37.420845+0000","last_deep_scrub":"64'4995","last_deep_scrub_stamp":"2024-09-19T07:49:37.420845+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.420845+0000","objects_scrubbed":0,"log_size":4995,"log_dups_size":0,"ondisk_log_size":4995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:02:56.503206+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27070,"num_write_kb":342022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"65'956","reported_seq":1130,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.647691+0000","last_change":"2024-09-19T07:49:26.647769+0000","last_active":"2024-09-19T07:49:26.647691+0000","last_peered":"2024-09-19T07:49:26.647691+0000","last_clean":"2024-09-19T07:49:26.647691+0000","last_became_active":"2024-09-19T07:19:31.273615+0000","last_became_peered":"2024-09-19T07:19:31.273615+0000","last_unstale":"2024-09-19T07:49:26.647691+0000","last_undegraded":"2024-09-19T07:49:26.647691+0000","last_fullsized":"2024-09-19T07:49:26.647691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'956","last_scrub_stamp":"2024-09-19T07:49:26.647475+0000","last_deep_scrub":"65'956","last_deep_scrub_stamp":"2024-09-19T07:49:26.647475+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.647475+0000","objects_scrubbed":3,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:10:02.782343+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":18964,"num_write":931,"num_write_kb":4658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"61'4894","reported_seq":5018,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.462691+0000","last_change":"2024-09-19T07:49:33.462691+0000","last_active":"2024-09-19T07:49:33.462691+0000","last_peered":"2024-09-19T07:49:33.462691+0000","last_clean":"2024-09-19T07:49:33.462691+0000","last_became_active":"2024-09-19T07:19:32.286549+0000","last_became_peered":"2024-09-19T07:19:32.286549+0000","last_unstale":"2024-09-19T07:49:33.462691+0000","last_undegraded":"2024-09-19T07:49:33.462691+0000","last_fullsized":"2024-09-19T07:49:33.462691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4894","last_scrub_stamp":"2024-09-19T07:49:33.462422+0000","last_deep_scrub":"61'4894","last_deep_scrub_stamp":"2024-09-19T07:49:33.462422+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.462422+0000","objects_scrubbed":0,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:31.966862+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26245,"num_write_kb":306037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"64'3","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.613843+0000","last_change":"2024-09-19T07:49:23.613909+0000","last_active":"2024-09-19T07:49:23.613843+0000","last_peered":"2024-09-19T07:49:23.613843+0000","last_clean":"2024-09-19T07:49:23.613843+0000","last_became_active":"2024-09-19T07:19:39.401337+0000","last_became_peered":"2024-09-19T07:19:39.401337+0000","last_unstale":"2024-09-19T07:49:23.613843+0000","last_undegraded":"2024-09-19T07:49:23.613843+0000","last_fullsized":"2024-09-19T07:49:23.613843+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-09-19T07:49:23.613617+0000","last_deep_scrub":"64'3","last_deep_scrub_stamp":"2024-09-19T07:49:23.613617+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.613617+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:24:32.580406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"64'5242","reported_seq":5355,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.333595+0000","last_change":"2024-09-19T07:49:29.333657+0000","last_active":"2024-09-19T07:49:29.333595+0000","last_peered":"2024-09-19T07:49:29.333595+0000","last_clean":"2024-09-19T07:49:29.333595+0000","last_became_active":"2024-09-19T07:19:40.364623+0000","last_became_peered":"2024-09-19T07:19:40.364623+0000","last_unstale":"2024-09-19T07:49:29.333595+0000","last_undegraded":"2024-09-19T07:49:29.333595+0000","last_fullsized":"2024-09-19T07:49:29.333595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5242","last_scrub_stamp":"2024-09-19T07:49:29.333381+0000","last_deep_scrub":"64'5242","last_deep_scrub_stamp":"2024-09-19T07:49:29.333381+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.333381+0000","objects_scrubbed":0,"log_size":5242,"log_dups_size":0,"ondisk_log_size":5242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:16.081844+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27567,"num_write_kb":356452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"65'25","reported_seq":151,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.501749+0000","last_change":"2024-09-19T07:49:36.501749+0000","last_active":"2024-09-19T07:49:36.501749+0000","last_peered":"2024-09-19T07:49:36.501749+0000","last_clean":"2024-09-19T07:49:36.501749+0000","last_became_active":"2024-09-19T07:19:31.262917+0000","last_became_peered":"2024-09-19T07:19:31.262917+0000","last_unstale":"2024-09-19T07:49:36.501749+0000","last_undegraded":"2024-09-19T07:49:36.501749+0000","last_fullsized":"2024-09-19T07:49:36.501749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'25","last_scrub_stamp":"2024-09-19T07:49:36.501391+0000","last_deep_scrub":"65'25","last_deep_scrub_stamp":"2024-09-19T07:49:36.501391+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.501391+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:47.830192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"61'5997","reported_seq":6120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.631536+0000","last_change":"2024-09-19T07:49:30.631597+0000","last_active":"2024-09-19T07:49:30.631536+0000","last_peered":"2024-09-19T07:49:30.631536+0000","last_clean":"2024-09-19T07:49:30.631536+0000","last_became_active":"2024-09-19T07:19:32.270923+0000","last_became_peered":"2024-09-19T07:19:32.270923+0000","last_unstale":"2024-09-19T07:49:30.631536+0000","last_undegraded":"2024-09-19T07:49:30.631536+0000","last_fullsized":"2024-09-19T07:49:30.631536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5997","last_scrub_stamp":"2024-09-19T07:49:30.631337+0000","last_deep_scrub":"61'5997","last_deep_scrub_stamp":"2024-09-19T07:49:30.631337+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.631337+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:11:11.393948+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32297,"num_write_kb":399913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"64'2","reported_seq":117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.333780+0000","last_change":"2024-09-19T07:49:35.333920+0000","last_active":"2024-09-19T07:49:35.333780+0000","last_peered":"2024-09-19T07:49:35.333780+0000","last_clean":"2024-09-19T07:49:35.333780+0000","last_became_active":"2024-09-19T07:19:39.403058+0000","last_became_peered":"2024-09-19T07:19:39.403058+0000","last_unstale":"2024-09-19T07:49:35.333780+0000","last_undegraded":"2024-09-19T07:49:35.333780+0000","last_fullsized":"2024-09-19T07:49:35.333780+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-19T07:49:35.333511+0000","last_deep_scrub":"64'2","last_deep_scrub_stamp":"2024-09-19T07:49:35.333511+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.333511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:55.298008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"64'3801","reported_seq":3915,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:49.389310+0000","last_change":"2024-09-19T07:49:49.389375+0000","last_active":"2024-09-19T07:49:49.389310+0000","last_peered":"2024-09-19T07:49:49.389310+0000","last_clean":"2024-09-19T07:49:49.389310+0000","last_became_active":"2024-09-19T07:19:40.389842+0000","last_became_peered":"2024-09-19T07:19:40.389842+0000","last_unstale":"2024-09-19T07:49:49.389310+0000","last_undegraded":"2024-09-19T07:49:49.389310+0000","last_fullsized":"2024-09-19T07:49:49.389310+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3801","last_scrub_stamp":"2024-09-19T07:49:49.389085+0000","last_deep_scrub":"64'3801","last_deep_scrub_stamp":"2024-09-19T07:49:49.389085+0000","last_clean_scrub_stamp":"2024-09-19T07:49:49.389085+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:16:23.211052+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20272,"num_write_kb":245240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"65'45","reported_seq":208,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.341882+0000","last_change":"2024-09-19T07:49:39.342050+0000","last_active":"2024-09-19T07:49:39.341882+0000","last_peered":"2024-09-19T07:49:39.341882+0000","last_clean":"2024-09-19T07:49:39.341882+0000","last_became_active":"2024-09-19T07:19:31.280478+0000","last_became_peered":"2024-09-19T07:19:31.280478+0000","last_unstale":"2024-09-19T07:49:39.341882+0000","last_undegraded":"2024-09-19T07:49:39.341882+0000","last_fullsized":"2024-09-19T07:49:39.341882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'45","last_scrub_stamp":"2024-09-19T07:49:39.341627+0000","last_deep_scrub":"65'45","last_deep_scrub_stamp":"2024-09-19T07:49:39.341627+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.341627+0000","objects_scrubbed":1,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:50.670429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"61'4645","reported_seq":4768,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.558336+0000","last_change":"2024-09-19T07:49:22.558336+0000","last_active":"2024-09-19T07:49:22.558336+0000","last_peered":"2024-09-19T07:49:22.558336+0000","last_clean":"2024-09-19T07:49:22.558336+0000","last_became_active":"2024-09-19T07:19:32.293794+0000","last_became_peered":"2024-09-19T07:19:32.293794+0000","last_unstale":"2024-09-19T07:49:22.558336+0000","last_undegraded":"2024-09-19T07:49:22.558336+0000","last_fullsized":"2024-09-19T07:49:22.558336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4645","last_scrub_stamp":"2024-09-19T07:49:22.557969+0000","last_deep_scrub":"61'4645","last_deep_scrub_stamp":"2024-09-19T07:49:22.557969+0000","last_clean_scrub_stamp":"2024-09-19T07:49:22.557969+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:19:09.650471+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24199,"num_write_kb":281402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'765","reported_seq":906,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.307628+0000","last_change":"2024-09-19T07:49:28.307628+0000","last_active":"2024-09-19T07:49:28.307628+0000","last_peered":"2024-09-19T07:49:28.307628+0000","last_clean":"2024-09-19T07:49:28.307628+0000","last_became_active":"2024-09-19T07:19:39.407568+0000","last_became_peered":"2024-09-19T07:19:39.407568+0000","last_unstale":"2024-09-19T07:49:28.307628+0000","last_undegraded":"2024-09-19T07:49:28.307628+0000","last_fullsized":"2024-09-19T07:49:28.307628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'765","last_scrub_stamp":"2024-09-19T07:49:28.307365+0000","last_deep_scrub":"65'765","last_deep_scrub_stamp":"2024-09-19T07:49:28.307365+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.307365+0000","objects_scrubbed":1,"log_size":765,"log_dups_size":0,"ondisk_log_size":765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:38:18.776897+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":71635,"num_write":751,"num_write_kb":4331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"64'3560","reported_seq":3673,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:43.569805+0000","last_change":"2024-09-19T07:49:43.569930+0000","last_active":"2024-09-19T07:49:43.569805+0000","last_peered":"2024-09-19T07:49:43.569805+0000","last_clean":"2024-09-19T07:49:43.569805+0000","last_became_active":"2024-09-19T07:19:40.371672+0000","last_became_peered":"2024-09-19T07:19:40.371672+0000","last_unstale":"2024-09-19T07:49:43.569805+0000","last_undegraded":"2024-09-19T07:49:43.569805+0000","last_fullsized":"2024-09-19T07:49:43.569805+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3560","last_scrub_stamp":"2024-09-19T07:49:43.569483+0000","last_deep_scrub":"64'3560","last_deep_scrub_stamp":"2024-09-19T07:49:43.569483+0000","last_clean_scrub_stamp":"2024-09-19T07:49:43.569483+0000","objects_scrubbed":0,"log_size":3560,"log_dups_size":0,"ondisk_log_size":3560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:33:29.524182+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18641,"num_write_kb":242587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"65'6","reported_seq":169,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:41.348670+0000","last_change":"2024-09-19T07:49:41.348736+0000","last_active":"2024-09-19T07:49:41.348670+0000","last_peered":"2024-09-19T07:49:41.348670+0000","last_clean":"2024-09-19T07:49:41.348670+0000","last_became_active":"2024-09-19T07:19:31.291313+0000","last_became_peered":"2024-09-19T07:19:31.291313+0000","last_unstale":"2024-09-19T07:49:41.348670+0000","last_undegraded":"2024-09-19T07:49:41.348670+0000","last_fullsized":"2024-09-19T07:49:41.348670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-19T07:49:41.348467+0000","last_deep_scrub":"65'6","last_deep_scrub_stamp":"2024-09-19T07:49:41.348467+0000","last_clean_scrub_stamp":"2024-09-19T07:49:41.348467+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:42:18.665421+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"61'5989","reported_seq":6110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.572806+0000","last_change":"2024-09-19T07:49:23.572806+0000","last_active":"2024-09-19T07:49:23.572806+0000","last_peered":"2024-09-19T07:49:23.572806+0000","last_clean":"2024-09-19T07:49:23.572806+0000","last_became_active":"2024-09-19T07:19:32.291347+0000","last_became_peered":"2024-09-19T07:19:32.291347+0000","last_unstale":"2024-09-19T07:49:23.572806+0000","last_undegraded":"2024-09-19T07:49:23.572806+0000","last_fullsized":"2024-09-19T07:49:23.572806+0000","mapping_epoch":19,"log_start":"53'200","ondisk_log_start":"53'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5989","last_scrub_stamp":"2024-09-19T07:49:23.572533+0000","last_deep_scrub":"61'5989","last_deep_scrub_stamp":"2024-09-19T07:49:23.572533+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.572533+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:39.369315+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32376,"num_write_kb":375806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'385","reported_seq":502,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.371994+0000","last_change":"2024-09-19T07:49:36.371994+0000","last_active":"2024-09-19T07:49:36.371994+0000","last_peered":"2024-09-19T07:49:36.371994+0000","last_clean":"2024-09-19T07:49:36.371994+0000","last_became_active":"2024-09-19T07:19:39.406231+0000","last_became_peered":"2024-09-19T07:19:39.406231+0000","last_unstale":"2024-09-19T07:49:36.371994+0000","last_undegraded":"2024-09-19T07:49:36.371994+0000","last_fullsized":"2024-09-19T07:49:36.371994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'385","last_scrub_stamp":"2024-09-19T07:49:36.371687+0000","last_deep_scrub":"65'385","last_deep_scrub_stamp":"2024-09-19T07:49:36.371687+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.371687+0000","objects_scrubbed":1,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:50:43.795428+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2764415,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":637,"num_write":359,"num_write_kb":2908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"64'4768","reported_seq":4882,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:45.142903+0000","last_change":"2024-09-19T07:49:45.142964+0000","last_active":"2024-09-19T07:49:45.142903+0000","last_peered":"2024-09-19T07:49:45.142903+0000","last_clean":"2024-09-19T07:49:45.142903+0000","last_became_active":"2024-09-19T07:19:40.395731+0000","last_became_peered":"2024-09-19T07:19:40.395731+0000","last_unstale":"2024-09-19T07:49:45.142903+0000","last_undegraded":"2024-09-19T07:49:45.142903+0000","last_fullsized":"2024-09-19T07:49:45.142903+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4768","last_scrub_stamp":"2024-09-19T07:49:45.142682+0000","last_deep_scrub":"64'4768","last_deep_scrub_stamp":"2024-09-19T07:49:45.142682+0000","last_clean_scrub_stamp":"2024-09-19T07:49:45.142682+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:27.548310+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25484,"num_write_kb":302986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"61'5596","reported_seq":5719,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.478547+0000","last_change":"2024-09-19T07:49:24.478547+0000","last_active":"2024-09-19T07:49:24.478547+0000","last_peered":"2024-09-19T07:49:24.478547+0000","last_clean":"2024-09-19T07:49:24.478547+0000","last_became_active":"2024-09-19T07:19:32.315879+0000","last_became_peered":"2024-09-19T07:19:32.315879+0000","last_unstale":"2024-09-19T07:49:24.478547+0000","last_undegraded":"2024-09-19T07:49:24.478547+0000","last_fullsized":"2024-09-19T07:49:24.478547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5596","last_scrub_stamp":"2024-09-19T07:49:24.478287+0000","last_deep_scrub":"61'5596","last_deep_scrub_stamp":"2024-09-19T07:49:24.478287+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.478287+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:10:51.268354+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30081,"num_write_kb":342050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'1458","reported_seq":1626,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.320165+0000","last_change":"2024-09-19T07:49:36.320239+0000","last_active":"2024-09-19T07:49:36.320165+0000","last_peered":"2024-09-19T07:49:36.320165+0000","last_clean":"2024-09-19T07:49:36.320165+0000","last_became_active":"2024-09-19T07:19:31.277345+0000","last_became_peered":"2024-09-19T07:19:31.277345+0000","last_unstale":"2024-09-19T07:49:36.320165+0000","last_undegraded":"2024-09-19T07:49:36.320165+0000","last_fullsized":"2024-09-19T07:49:36.320165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1458","last_scrub_stamp":"2024-09-19T07:49:36.319951+0000","last_deep_scrub":"65'1458","last_deep_scrub_stamp":"2024-09-19T07:49:36.319951+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.319951+0000","objects_scrubbed":2,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:12:34.824390+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":42401,"num_write":1433,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'7","reported_seq":143,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.269934+0000","last_change":"2024-09-19T07:49:26.270002+0000","last_active":"2024-09-19T07:49:26.269934+0000","last_peered":"2024-09-19T07:49:26.269934+0000","last_clean":"2024-09-19T07:49:26.269934+0000","last_became_active":"2024-09-19T07:19:39.408743+0000","last_became_peered":"2024-09-19T07:19:39.408743+0000","last_unstale":"2024-09-19T07:49:26.269934+0000","last_undegraded":"2024-09-19T07:49:26.269934+0000","last_fullsized":"2024-09-19T07:49:26.269934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'7","last_scrub_stamp":"2024-09-19T07:49:26.269740+0000","last_deep_scrub":"65'7","last_deep_scrub_stamp":"2024-09-19T07:49:26.269740+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.269740+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:31.988649+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"64'4389","reported_seq":4503,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.653867+0000","last_change":"2024-09-19T07:49:32.653925+0000","last_active":"2024-09-19T07:49:32.653867+0000","last_peered":"2024-09-19T07:49:32.653867+0000","last_clean":"2024-09-19T07:49:32.653867+0000","last_became_active":"2024-09-19T07:19:40.380453+0000","last_became_peered":"2024-09-19T07:19:40.380453+0000","last_unstale":"2024-09-19T07:49:32.653867+0000","last_undegraded":"2024-09-19T07:49:32.653867+0000","last_fullsized":"2024-09-19T07:49:32.653867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4389","last_scrub_stamp":"2024-09-19T07:49:32.653662+0000","last_deep_scrub":"64'4389","last_deep_scrub_stamp":"2024-09-19T07:49:32.653662+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.653662+0000","objects_scrubbed":0,"log_size":4389,"log_dups_size":0,"ondisk_log_size":4389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:38:16.691610+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23791,"num_write_kb":304343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"61'3782","reported_seq":3904,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:42.484020+0000","last_change":"2024-09-19T07:49:42.484137+0000","last_active":"2024-09-19T07:49:42.484020+0000","last_peered":"2024-09-19T07:49:42.484020+0000","last_clean":"2024-09-19T07:49:42.484020+0000","last_became_active":"2024-09-19T07:19:32.278841+0000","last_became_peered":"2024-09-19T07:19:32.278841+0000","last_unstale":"2024-09-19T07:49:42.484020+0000","last_undegraded":"2024-09-19T07:49:42.484020+0000","last_fullsized":"2024-09-19T07:49:42.484020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3782","last_scrub_stamp":"2024-09-19T07:49:42.483666+0000","last_deep_scrub":"61'3782","last_deep_scrub_stamp":"2024-09-19T07:49:42.483666+0000","last_clean_scrub_stamp":"2024-09-19T07:49:42.483666+0000","objects_scrubbed":0,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:03:26.378592+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19997,"num_write_kb":252226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"61'3","reported_seq":128,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.213860+0000","last_change":"2024-09-19T07:49:32.213860+0000","last_active":"2024-09-19T07:49:32.213860+0000","last_peered":"2024-09-19T07:49:32.213860+0000","last_clean":"2024-09-19T07:49:32.213860+0000","last_became_active":"2024-09-19T07:19:31.262317+0000","last_became_peered":"2024-09-19T07:19:31.262317+0000","last_unstale":"2024-09-19T07:49:32.213860+0000","last_undegraded":"2024-09-19T07:49:32.213860+0000","last_fullsized":"2024-09-19T07:49:32.213860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-19T07:49:32.213558+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-19T07:49:32.213558+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.213558+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:48.490205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"64'5","reported_seq":121,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.302563+0000","last_change":"2024-09-19T07:49:24.302627+0000","last_active":"2024-09-19T07:49:24.302563+0000","last_peered":"2024-09-19T07:49:24.302563+0000","last_clean":"2024-09-19T07:49:24.302563+0000","last_became_active":"2024-09-19T07:19:39.410513+0000","last_became_peered":"2024-09-19T07:19:39.410513+0000","last_unstale":"2024-09-19T07:49:24.302563+0000","last_undegraded":"2024-09-19T07:49:24.302563+0000","last_fullsized":"2024-09-19T07:49:24.302563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5","last_scrub_stamp":"2024-09-19T07:49:24.302364+0000","last_deep_scrub":"64'5","last_deep_scrub_stamp":"2024-09-19T07:49:24.302364+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.302364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:09:24.709205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"64'3306","reported_seq":3420,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.479880+0000","last_change":"2024-09-19T07:49:39.479952+0000","last_active":"2024-09-19T07:49:39.479880+0000","last_peered":"2024-09-19T07:49:39.479880+0000","last_clean":"2024-09-19T07:49:39.479880+0000","last_became_active":"2024-09-19T07:19:40.382504+0000","last_became_peered":"2024-09-19T07:19:40.382504+0000","last_unstale":"2024-09-19T07:49:39.479880+0000","last_undegraded":"2024-09-19T07:49:39.479880+0000","last_fullsized":"2024-09-19T07:49:39.479880+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3306","last_scrub_stamp":"2024-09-19T07:49:39.479617+0000","last_deep_scrub":"64'3306","last_deep_scrub_stamp":"2024-09-19T07:49:39.479617+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.479617+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:23:47.514505+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17836,"num_write_kb":239496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"61'4850","reported_seq":4973,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.460644+0000","last_change":"2024-09-19T07:49:39.460644+0000","last_active":"2024-09-19T07:49:39.460644+0000","last_peered":"2024-09-19T07:49:39.460644+0000","last_clean":"2024-09-19T07:49:39.460644+0000","last_became_active":"2024-09-19T07:19:32.265528+0000","last_became_peered":"2024-09-19T07:19:32.265528+0000","last_unstale":"2024-09-19T07:49:39.460644+0000","last_undegraded":"2024-09-19T07:49:39.460644+0000","last_fullsized":"2024-09-19T07:49:39.460644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4850","last_scrub_stamp":"2024-09-19T07:49:39.460360+0000","last_deep_scrub":"61'4850","last_deep_scrub_stamp":"2024-09-19T07:49:39.460360+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.460360+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:35:55.967049+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26728,"num_write_kb":308134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"61'2","reported_seq":127,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.209522+0000","last_change":"2024-09-19T07:49:35.209522+0000","last_active":"2024-09-19T07:49:35.209522+0000","last_peered":"2024-09-19T07:49:35.209522+0000","last_clean":"2024-09-19T07:49:35.209522+0000","last_became_active":"2024-09-19T07:19:31.273726+0000","last_became_peered":"2024-09-19T07:19:31.273726+0000","last_unstale":"2024-09-19T07:49:35.209522+0000","last_undegraded":"2024-09-19T07:49:35.209522+0000","last_fullsized":"2024-09-19T07:49:35.209522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-19T07:49:35.209266+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-19T07:49:35.209266+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.209266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:12:48.875188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'5","reported_seq":141,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:21.158021+0000","last_change":"2024-09-19T07:49:21.158156+0000","last_active":"2024-09-19T07:49:21.158021+0000","last_peered":"2024-09-19T07:49:21.158021+0000","last_clean":"2024-09-19T07:49:21.158021+0000","last_became_active":"2024-09-19T07:19:39.407923+0000","last_became_peered":"2024-09-19T07:19:39.407923+0000","last_unstale":"2024-09-19T07:49:21.158021+0000","last_undegraded":"2024-09-19T07:49:21.158021+0000","last_fullsized":"2024-09-19T07:49:21.158021+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-19T07:49:21.157739+0000","last_deep_scrub":"65'5","last_deep_scrub_stamp":"2024-09-19T07:49:21.157739+0000","last_clean_scrub_stamp":"2024-09-19T07:49:21.157739+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:09:21.564579+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"64'5157","reported_seq":5271,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:48.527460+0000","last_change":"2024-09-19T07:49:48.527525+0000","last_active":"2024-09-19T07:49:48.527460+0000","last_peered":"2024-09-19T07:49:48.527460+0000","last_clean":"2024-09-19T07:49:48.527460+0000","last_became_active":"2024-09-19T07:19:40.394879+0000","last_became_peered":"2024-09-19T07:19:40.394879+0000","last_unstale":"2024-09-19T07:49:48.527460+0000","last_undegraded":"2024-09-19T07:49:48.527460+0000","last_fullsized":"2024-09-19T07:49:48.527460+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5157","last_scrub_stamp":"2024-09-19T07:49:48.527252+0000","last_deep_scrub":"64'5157","last_deep_scrub_stamp":"2024-09-19T07:49:48.527252+0000","last_clean_scrub_stamp":"2024-09-19T07:49:48.527252+0000","objects_scrubbed":0,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:36:32.724100+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27729,"num_write_kb":365262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"61'4508","reported_seq":4631,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.388460+0000","last_change":"2024-09-19T07:49:32.388460+0000","last_active":"2024-09-19T07:49:32.388460+0000","last_peered":"2024-09-19T07:49:32.388460+0000","last_clean":"2024-09-19T07:49:32.388460+0000","last_became_active":"2024-09-19T07:19:32.289279+0000","last_became_peered":"2024-09-19T07:19:32.289279+0000","last_unstale":"2024-09-19T07:49:32.388460+0000","last_undegraded":"2024-09-19T07:49:32.388460+0000","last_fullsized":"2024-09-19T07:49:32.388460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4508","last_scrub_stamp":"2024-09-19T07:49:32.388033+0000","last_deep_scrub":"61'4508","last_deep_scrub_stamp":"2024-09-19T07:49:32.388033+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.388033+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:55:19.136496+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24068,"num_write_kb":289158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"61'5","reported_seq":130,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.322517+0000","last_change":"2024-09-19T07:49:38.322517+0000","last_active":"2024-09-19T07:49:38.322517+0000","last_peered":"2024-09-19T07:49:38.322517+0000","last_clean":"2024-09-19T07:49:38.322517+0000","last_became_active":"2024-09-19T07:19:31.276953+0000","last_became_peered":"2024-09-19T07:19:31.276953+0000","last_unstale":"2024-09-19T07:49:38.322517+0000","last_undegraded":"2024-09-19T07:49:38.322517+0000","last_fullsized":"2024-09-19T07:49:38.322517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5","last_scrub_stamp":"2024-09-19T07:49:38.322115+0000","last_deep_scrub":"61'5","last_deep_scrub_stamp":"2024-09-19T07:49:38.322115+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.322115+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:57:57.049656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'8","reported_seq":164,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.317551+0000","last_change":"2024-09-19T07:49:25.317688+0000","last_active":"2024-09-19T07:49:25.317551+0000","last_peered":"2024-09-19T07:49:25.317551+0000","last_clean":"2024-09-19T07:49:25.317551+0000","last_became_active":"2024-09-19T07:19:39.423854+0000","last_became_peered":"2024-09-19T07:19:39.423854+0000","last_unstale":"2024-09-19T07:49:25.317551+0000","last_undegraded":"2024-09-19T07:49:25.317551+0000","last_fullsized":"2024-09-19T07:49:25.317551+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-19T07:49:25.317293+0000","last_deep_scrub":"65'8","last_deep_scrub_stamp":"2024-09-19T07:49:25.317293+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.317293+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:11:31.559475+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"64'3458","reported_seq":3574,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:51.555629+0000","last_change":"2024-09-19T07:49:51.555691+0000","last_active":"2024-09-19T07:49:51.555629+0000","last_peered":"2024-09-19T07:49:51.555629+0000","last_clean":"2024-09-19T07:49:51.555629+0000","last_became_active":"2024-09-19T07:19:40.387754+0000","last_became_peered":"2024-09-19T07:19:40.387754+0000","last_unstale":"2024-09-19T07:49:51.555629+0000","last_undegraded":"2024-09-19T07:49:51.555629+0000","last_fullsized":"2024-09-19T07:49:51.555629+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3458","last_scrub_stamp":"2024-09-19T07:49:51.555407+0000","last_deep_scrub":"64'3458","last_deep_scrub_stamp":"2024-09-19T07:49:51.555407+0000","last_clean_scrub_stamp":"2024-09-19T07:49:51.555407+0000","objects_scrubbed":0,"log_size":3458,"log_dups_size":0,"ondisk_log_size":3458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:01:27.146105+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18862,"num_write_kb":239540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"61'5647","reported_seq":5771,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.588680+0000","last_change":"2024-09-19T07:49:24.588680+0000","last_active":"2024-09-19T07:49:24.588680+0000","last_peered":"2024-09-19T07:49:24.588680+0000","last_clean":"2024-09-19T07:49:24.588680+0000","last_became_active":"2024-09-19T07:19:32.279350+0000","last_became_peered":"2024-09-19T07:19:32.279350+0000","last_unstale":"2024-09-19T07:49:24.588680+0000","last_undegraded":"2024-09-19T07:49:24.588680+0000","last_fullsized":"2024-09-19T07:49:24.588680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5647","last_scrub_stamp":"2024-09-19T07:49:24.588262+0000","last_deep_scrub":"61'5647","last_deep_scrub_stamp":"2024-09-19T07:49:24.588262+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.588262+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:37:02.235369+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29735,"num_write_kb":364939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"65'10","reported_seq":175,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.521476+0000","last_change":"2024-09-19T07:49:30.521476+0000","last_active":"2024-09-19T07:49:30.521476+0000","last_peered":"2024-09-19T07:49:30.521476+0000","last_clean":"2024-09-19T07:49:30.521476+0000","last_became_active":"2024-09-19T07:19:31.280913+0000","last_became_peered":"2024-09-19T07:19:31.280913+0000","last_unstale":"2024-09-19T07:49:30.521476+0000","last_undegraded":"2024-09-19T07:49:30.521476+0000","last_fullsized":"2024-09-19T07:49:30.521476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-19T07:49:30.521211+0000","last_deep_scrub":"65'10","last_deep_scrub_stamp":"2024-09-19T07:49:30.521211+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.521211+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:33:32.235334+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'45","reported_seq":222,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.324266+0000","last_change":"2024-09-19T07:49:22.324332+0000","last_active":"2024-09-19T07:49:22.324266+0000","last_peered":"2024-09-19T07:49:22.324266+0000","last_clean":"2024-09-19T07:49:22.324266+0000","last_became_active":"2024-09-19T07:19:39.411255+0000","last_became_peered":"2024-09-19T07:19:39.411255+0000","last_unstale":"2024-09-19T07:49:22.324266+0000","last_undegraded":"2024-09-19T07:49:22.324266+0000","last_fullsized":"2024-09-19T07:49:22.324266+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'45","last_scrub_stamp":"2024-09-19T07:49:22.324054+0000","last_deep_scrub":"65'45","last_deep_scrub_stamp":"2024-09-19T07:49:22.324054+0000","last_clean_scrub_stamp":"2024-09-19T07:49:22.324054+0000","objects_scrubbed":1,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:26.914491+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":108,"num_write":41,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"64'4007","reported_seq":4122,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:43.434656+0000","last_change":"2024-09-19T07:49:43.434714+0000","last_active":"2024-09-19T07:49:43.434656+0000","last_peered":"2024-09-19T07:49:43.434656+0000","last_clean":"2024-09-19T07:49:43.434656+0000","last_became_active":"2024-09-19T07:19:40.389650+0000","last_became_peered":"2024-09-19T07:19:40.389650+0000","last_unstale":"2024-09-19T07:49:43.434656+0000","last_undegraded":"2024-09-19T07:49:43.434656+0000","last_fullsized":"2024-09-19T07:49:43.434656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4007","last_scrub_stamp":"2024-09-19T07:49:43.434457+0000","last_deep_scrub":"64'4007","last_deep_scrub_stamp":"2024-09-19T07:49:43.434457+0000","last_clean_scrub_stamp":"2024-09-19T07:49:43.434457+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:54.763258+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21606,"num_write_kb":266928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"61'4152","reported_seq":4275,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:43.458886+0000","last_change":"2024-09-19T07:49:43.458886+0000","last_active":"2024-09-19T07:49:43.458886+0000","last_peered":"2024-09-19T07:49:43.458886+0000","last_clean":"2024-09-19T07:49:43.458886+0000","last_became_active":"2024-09-19T07:19:32.282299+0000","last_became_peered":"2024-09-19T07:19:32.282299+0000","last_unstale":"2024-09-19T07:49:43.458886+0000","last_undegraded":"2024-09-19T07:49:43.458886+0000","last_fullsized":"2024-09-19T07:49:43.458886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4152","last_scrub_stamp":"2024-09-19T07:49:43.458738+0000","last_deep_scrub":"61'4152","last_deep_scrub_stamp":"2024-09-19T07:49:43.458738+0000","last_clean_scrub_stamp":"2024-09-19T07:49:43.458738+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:26:41.019567+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22349,"num_write_kb":269915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"65'475","reported_seq":620,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.204070+0000","last_change":"2024-09-19T07:49:34.204070+0000","last_active":"2024-09-19T07:49:34.204070+0000","last_peered":"2024-09-19T07:49:34.204070+0000","last_clean":"2024-09-19T07:49:34.204070+0000","last_became_active":"2024-09-19T07:19:31.272096+0000","last_became_peered":"2024-09-19T07:19:31.272096+0000","last_unstale":"2024-09-19T07:49:34.204070+0000","last_undegraded":"2024-09-19T07:49:34.204070+0000","last_fullsized":"2024-09-19T07:49:34.204070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'475","last_scrub_stamp":"2024-09-19T07:49:34.203776+0000","last_deep_scrub":"65'475","last_deep_scrub_stamp":"2024-09-19T07:49:34.203776+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.203776+0000","objects_scrubbed":1,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:50:41.627517+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":59,"num_write":1141,"num_write_kb":380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"64'10","reported_seq":128,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.435045+0000","last_change":"2024-09-19T07:49:30.435045+0000","last_active":"2024-09-19T07:49:30.435045+0000","last_peered":"2024-09-19T07:49:30.435045+0000","last_clean":"2024-09-19T07:49:30.435045+0000","last_became_active":"2024-09-19T07:19:39.433921+0000","last_became_peered":"2024-09-19T07:19:39.433921+0000","last_unstale":"2024-09-19T07:49:30.435045+0000","last_undegraded":"2024-09-19T07:49:30.435045+0000","last_fullsized":"2024-09-19T07:49:30.435045+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'10","last_scrub_stamp":"2024-09-19T07:49:30.434776+0000","last_deep_scrub":"64'10","last_deep_scrub_stamp":"2024-09-19T07:49:30.434776+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.434776+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:34:54.431736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"64'4199","reported_seq":4314,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.499655+0000","last_change":"2024-09-19T07:49:39.499782+0000","last_active":"2024-09-19T07:49:39.499655+0000","last_peered":"2024-09-19T07:49:39.499655+0000","last_clean":"2024-09-19T07:49:39.499655+0000","last_became_active":"2024-09-19T07:19:40.379382+0000","last_became_peered":"2024-09-19T07:19:40.379382+0000","last_unstale":"2024-09-19T07:49:39.499655+0000","last_undegraded":"2024-09-19T07:49:39.499655+0000","last_fullsized":"2024-09-19T07:49:39.499655+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4199","last_scrub_stamp":"2024-09-19T07:49:39.499410+0000","last_deep_scrub":"64'4199","last_deep_scrub_stamp":"2024-09-19T07:49:39.499410+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.499410+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:15:59.463907+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22884,"num_write_kb":280362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"61'5900","reported_seq":6023,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.263166+0000","last_change":"2024-09-19T07:49:28.263234+0000","last_active":"2024-09-19T07:49:28.263166+0000","last_peered":"2024-09-19T07:49:28.263166+0000","last_clean":"2024-09-19T07:49:28.263166+0000","last_became_active":"2024-09-19T07:19:32.275614+0000","last_became_peered":"2024-09-19T07:19:32.275614+0000","last_unstale":"2024-09-19T07:49:28.263166+0000","last_undegraded":"2024-09-19T07:49:28.263166+0000","last_fullsized":"2024-09-19T07:49:28.263166+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5900","last_scrub_stamp":"2024-09-19T07:49:28.262935+0000","last_deep_scrub":"61'5900","last_deep_scrub_stamp":"2024-09-19T07:49:28.262935+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.262935+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:09:31.336636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31464,"num_write_kb":364012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"61'2","reported_seq":127,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.351071+0000","last_change":"2024-09-19T07:49:40.351130+0000","last_active":"2024-09-19T07:49:40.351071+0000","last_peered":"2024-09-19T07:49:40.351071+0000","last_clean":"2024-09-19T07:49:40.351071+0000","last_became_active":"2024-09-19T07:19:31.281409+0000","last_became_peered":"2024-09-19T07:19:31.281409+0000","last_unstale":"2024-09-19T07:49:40.351071+0000","last_undegraded":"2024-09-19T07:49:40.351071+0000","last_fullsized":"2024-09-19T07:49:40.351071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-19T07:49:40.350876+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-19T07:49:40.350876+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.350876+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:01:39.933098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"64'3","reported_seq":120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.540273+0000","last_change":"2024-09-19T07:49:35.540342+0000","last_active":"2024-09-19T07:49:35.540273+0000","last_peered":"2024-09-19T07:49:35.540273+0000","last_clean":"2024-09-19T07:49:35.540273+0000","last_became_active":"2024-09-19T07:19:39.411272+0000","last_became_peered":"2024-09-19T07:19:39.411272+0000","last_unstale":"2024-09-19T07:49:35.540273+0000","last_undegraded":"2024-09-19T07:49:35.540273+0000","last_fullsized":"2024-09-19T07:49:35.540273+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-09-19T07:49:35.540059+0000","last_deep_scrub":"64'3","last_deep_scrub_stamp":"2024-09-19T07:49:35.540059+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.540059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:11:27.485809+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"64'4404","reported_seq":4519,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.496603+0000","last_change":"2024-09-19T07:49:37.496663+0000","last_active":"2024-09-19T07:49:37.496603+0000","last_peered":"2024-09-19T07:49:37.496603+0000","last_clean":"2024-09-19T07:49:37.496603+0000","last_became_active":"2024-09-19T07:19:40.371012+0000","last_became_peered":"2024-09-19T07:19:40.371012+0000","last_unstale":"2024-09-19T07:49:37.496603+0000","last_undegraded":"2024-09-19T07:49:37.496603+0000","last_fullsized":"2024-09-19T07:49:37.496603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4404","last_scrub_stamp":"2024-09-19T07:49:37.496410+0000","last_deep_scrub":"64'4404","last_deep_scrub_stamp":"2024-09-19T07:49:37.496410+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.496410+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:42:33.605988+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24188,"num_write_kb":314198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"61'3777","reported_seq":3900,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.477401+0000","last_change":"2024-09-19T07:49:40.477460+0000","last_active":"2024-09-19T07:49:40.477401+0000","last_peered":"2024-09-19T07:49:40.477401+0000","last_clean":"2024-09-19T07:49:40.477401+0000","last_became_active":"2024-09-19T07:19:32.272593+0000","last_became_peered":"2024-09-19T07:19:32.272593+0000","last_unstale":"2024-09-19T07:49:40.477401+0000","last_undegraded":"2024-09-19T07:49:40.477401+0000","last_fullsized":"2024-09-19T07:49:40.477401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3777","last_scrub_stamp":"2024-09-19T07:49:40.477210+0000","last_deep_scrub":"61'3777","last_deep_scrub_stamp":"2024-09-19T07:49:40.477210+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.477210+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:25:24.847439+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20312,"num_write_kb":245406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"61'2","reported_seq":127,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.240822+0000","last_change":"2024-09-19T07:49:31.240961+0000","last_active":"2024-09-19T07:49:31.240822+0000","last_peered":"2024-09-19T07:49:31.240822+0000","last_clean":"2024-09-19T07:49:31.240822+0000","last_became_active":"2024-09-19T07:19:31.262727+0000","last_became_peered":"2024-09-19T07:19:31.262727+0000","last_unstale":"2024-09-19T07:49:31.240822+0000","last_undegraded":"2024-09-19T07:49:31.240822+0000","last_fullsized":"2024-09-19T07:49:31.240822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-19T07:49:31.240511+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-19T07:49:31.240511+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.240511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:19.769436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'4","reported_seq":121,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.158463+0000","last_change":"2024-09-19T07:49:23.158484+0000","last_active":"2024-09-19T07:49:23.158463+0000","last_peered":"2024-09-19T07:49:23.158463+0000","last_clean":"2024-09-19T07:49:23.158463+0000","last_became_active":"2024-09-19T07:19:39.420057+0000","last_became_peered":"2024-09-19T07:19:39.420057+0000","last_unstale":"2024-09-19T07:49:23.158463+0000","last_undegraded":"2024-09-19T07:49:23.158463+0000","last_fullsized":"2024-09-19T07:49:23.158463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-09-19T07:49:23.158382+0000","last_deep_scrub":"64'4","last_deep_scrub_stamp":"2024-09-19T07:49:23.158382+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.158382+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:28.877291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"64'4365","reported_seq":4480,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.457199+0000","last_change":"2024-09-19T07:49:40.457265+0000","last_active":"2024-09-19T07:49:40.457199+0000","last_peered":"2024-09-19T07:49:40.457199+0000","last_clean":"2024-09-19T07:49:40.457199+0000","last_became_active":"2024-09-19T07:19:40.382353+0000","last_became_peered":"2024-09-19T07:19:40.382353+0000","last_unstale":"2024-09-19T07:49:40.457199+0000","last_undegraded":"2024-09-19T07:49:40.457199+0000","last_fullsized":"2024-09-19T07:49:40.457199+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4365","last_scrub_stamp":"2024-09-19T07:49:40.456941+0000","last_deep_scrub":"64'4365","last_deep_scrub_stamp":"2024-09-19T07:49:40.456941+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.456941+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:56.706184+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24012,"num_write_kb":288918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'5396","reported_seq":5519,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.298054+0000","last_change":"2024-09-19T07:49:30.298222+0000","last_active":"2024-09-19T07:49:30.298054+0000","last_peered":"2024-09-19T07:49:30.298054+0000","last_clean":"2024-09-19T07:49:30.298054+0000","last_became_active":"2024-09-19T07:19:32.277108+0000","last_became_peered":"2024-09-19T07:19:32.277108+0000","last_unstale":"2024-09-19T07:49:30.298054+0000","last_undegraded":"2024-09-19T07:49:30.298054+0000","last_fullsized":"2024-09-19T07:49:30.298054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5396","last_scrub_stamp":"2024-09-19T07:49:30.297733+0000","last_deep_scrub":"61'5396","last_deep_scrub_stamp":"2024-09-19T07:49:30.297733+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.297733+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:06:29.592193+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28800,"num_write_kb":336838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'4","reported_seq":129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.460668+0000","last_change":"2024-09-19T07:49:34.460792+0000","last_active":"2024-09-19T07:49:34.460668+0000","last_peered":"2024-09-19T07:49:34.460668+0000","last_clean":"2024-09-19T07:49:34.460668+0000","last_became_active":"2024-09-19T07:19:31.280434+0000","last_became_peered":"2024-09-19T07:19:31.280434+0000","last_unstale":"2024-09-19T07:49:34.460668+0000","last_undegraded":"2024-09-19T07:49:34.460668+0000","last_fullsized":"2024-09-19T07:49:34.460668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4","last_scrub_stamp":"2024-09-19T07:49:34.460373+0000","last_deep_scrub":"61'4","last_deep_scrub_stamp":"2024-09-19T07:49:34.460373+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.460373+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:21:55.084729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"64'2","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.190194+0000","last_change":"2024-09-19T07:49:22.190249+0000","last_active":"2024-09-19T07:49:22.190194+0000","last_peered":"2024-09-19T07:49:22.190194+0000","last_clean":"2024-09-19T07:49:22.190194+0000","last_became_active":"2024-09-19T07:19:39.406854+0000","last_became_peered":"2024-09-19T07:19:39.406854+0000","last_unstale":"2024-09-19T07:49:22.190194+0000","last_undegraded":"2024-09-19T07:49:22.190194+0000","last_fullsized":"2024-09-19T07:49:22.190194+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-19T07:49:22.190085+0000","last_deep_scrub":"64'2","last_deep_scrub_stamp":"2024-09-19T07:49:22.190085+0000","last_clean_scrub_stamp":"2024-09-19T07:49:22.190085+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:11:28.432267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"64'4926","reported_seq":5042,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.634007+0000","last_change":"2024-09-19T07:49:28.634007+0000","last_active":"2024-09-19T07:49:28.634007+0000","last_peered":"2024-09-19T07:49:28.634007+0000","last_clean":"2024-09-19T07:49:28.634007+0000","last_became_active":"2024-09-19T07:19:40.368107+0000","last_became_peered":"2024-09-19T07:19:40.368107+0000","last_unstale":"2024-09-19T07:49:28.634007+0000","last_undegraded":"2024-09-19T07:49:28.634007+0000","last_fullsized":"2024-09-19T07:49:28.634007+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4926","last_scrub_stamp":"2024-09-19T07:49:28.633734+0000","last_deep_scrub":"64'4926","last_deep_scrub_stamp":"2024-09-19T07:49:28.633734+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.633734+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:48:59.019782+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26066,"num_write_kb":337871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'3534","reported_seq":3658,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.254070+0000","last_change":"2024-09-19T07:49:30.254070+0000","last_active":"2024-09-19T07:49:30.254070+0000","last_peered":"2024-09-19T07:49:30.254070+0000","last_clean":"2024-09-19T07:49:30.254070+0000","last_became_active":"2024-09-19T07:19:32.275921+0000","last_became_peered":"2024-09-19T07:19:32.275921+0000","last_unstale":"2024-09-19T07:49:30.254070+0000","last_undegraded":"2024-09-19T07:49:30.254070+0000","last_fullsized":"2024-09-19T07:49:30.254070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3534","last_scrub_stamp":"2024-09-19T07:49:30.253787+0000","last_deep_scrub":"61'3534","last_deep_scrub_stamp":"2024-09-19T07:49:30.253787+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.253787+0000","objects_scrubbed":0,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:15:25.385951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19019,"num_write_kb":227884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"65'46","reported_seq":230,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.704762+0000","last_change":"2024-09-19T07:49:23.704827+0000","last_active":"2024-09-19T07:49:23.704762+0000","last_peered":"2024-09-19T07:49:23.704762+0000","last_clean":"2024-09-19T07:49:23.704762+0000","last_became_active":"2024-09-19T07:19:31.260416+0000","last_became_peered":"2024-09-19T07:19:31.260416+0000","last_unstale":"2024-09-19T07:49:23.704762+0000","last_undegraded":"2024-09-19T07:49:23.704762+0000","last_fullsized":"2024-09-19T07:49:23.704762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'46","last_scrub_stamp":"2024-09-19T07:49:23.704580+0000","last_deep_scrub":"65'46","last_deep_scrub_stamp":"2024-09-19T07:49:23.704580+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.704580+0000","objects_scrubbed":1,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:07:33.376909+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":109,"num_write":40,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'10","reported_seq":167,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.245537+0000","last_change":"2024-09-19T07:49:25.245537+0000","last_active":"2024-09-19T07:49:25.245537+0000","last_peered":"2024-09-19T07:49:25.245537+0000","last_clean":"2024-09-19T07:49:25.245537+0000","last_became_active":"2024-09-19T07:19:39.420164+0000","last_became_peered":"2024-09-19T07:19:39.420164+0000","last_unstale":"2024-09-19T07:49:25.245537+0000","last_undegraded":"2024-09-19T07:49:25.245537+0000","last_fullsized":"2024-09-19T07:49:25.245537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-19T07:49:25.245276+0000","last_deep_scrub":"65'10","last_deep_scrub_stamp":"2024-09-19T07:49:25.245276+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.245276+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:38:15.714808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"64'3425","reported_seq":3540,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:47.508980+0000","last_change":"2024-09-19T07:49:47.508980+0000","last_active":"2024-09-19T07:49:47.508980+0000","last_peered":"2024-09-19T07:49:47.508980+0000","last_clean":"2024-09-19T07:49:47.508980+0000","last_became_active":"2024-09-19T07:19:40.386841+0000","last_became_peered":"2024-09-19T07:19:40.386841+0000","last_unstale":"2024-09-19T07:49:47.508980+0000","last_undegraded":"2024-09-19T07:49:47.508980+0000","last_fullsized":"2024-09-19T07:49:47.508980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3425","last_scrub_stamp":"2024-09-19T07:49:47.508720+0000","last_deep_scrub":"64'3425","last_deep_scrub_stamp":"2024-09-19T07:49:47.508720+0000","last_clean_scrub_stamp":"2024-09-19T07:49:47.508720+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:36:21.695178+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19036,"num_write_kb":252575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"61'4467","reported_seq":4590,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:21.558948+0000","last_change":"2024-09-19T07:49:21.559009+0000","last_active":"2024-09-19T07:49:21.558948+0000","last_peered":"2024-09-19T07:49:21.558948+0000","last_clean":"2024-09-19T07:49:21.558948+0000","last_became_active":"2024-09-19T07:19:32.312577+0000","last_became_peered":"2024-09-19T07:19:32.312577+0000","last_unstale":"2024-09-19T07:49:21.558948+0000","last_undegraded":"2024-09-19T07:49:21.558948+0000","last_fullsized":"2024-09-19T07:49:21.558948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4467","last_scrub_stamp":"2024-09-19T07:49:21.558756+0000","last_deep_scrub":"61'4467","last_deep_scrub_stamp":"2024-09-19T07:49:21.558756+0000","last_clean_scrub_stamp":"2024-09-19T07:49:21.558756+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:10:47.453175+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23642,"num_write_kb":275072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'13","reported_seq":139,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.315255+0000","last_change":"2024-09-19T07:49:37.315255+0000","last_active":"2024-09-19T07:49:37.315255+0000","last_peered":"2024-09-19T07:49:37.315255+0000","last_clean":"2024-09-19T07:49:37.315255+0000","last_became_active":"2024-09-19T07:19:31.277744+0000","last_became_peered":"2024-09-19T07:19:31.277744+0000","last_unstale":"2024-09-19T07:49:37.315255+0000","last_undegraded":"2024-09-19T07:49:37.315255+0000","last_fullsized":"2024-09-19T07:49:37.315255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-19T07:49:37.314883+0000","last_deep_scrub":"65'13","last_deep_scrub_stamp":"2024-09-19T07:49:37.314883+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.314883+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:19:14.744779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'515","reported_seq":652,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.562394+0000","last_change":"2024-09-19T07:49:34.562394+0000","last_active":"2024-09-19T07:49:34.562394+0000","last_peered":"2024-09-19T07:49:34.562394+0000","last_clean":"2024-09-19T07:49:34.562394+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:49:34.562394+0000","last_undegraded":"2024-09-19T07:49:34.562394+0000","last_fullsized":"2024-09-19T07:49:34.562394+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'515","last_scrub_stamp":"2024-09-19T07:49:34.562143+0000","last_deep_scrub":"65'515","last_deep_scrub_stamp":"2024-09-19T07:49:34.562143+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.562143+0000","objects_scrubbed":1,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:18:29.876600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":53,"num_write":1231,"num_write_kb":406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"64'4906","reported_seq":5021,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:50.415156+0000","last_change":"2024-09-19T07:49:50.415274+0000","last_active":"2024-09-19T07:49:50.415156+0000","last_peered":"2024-09-19T07:49:50.415156+0000","last_clean":"2024-09-19T07:49:50.415156+0000","last_became_active":"2024-09-19T07:19:40.392187+0000","last_became_peered":"2024-09-19T07:19:40.392187+0000","last_unstale":"2024-09-19T07:49:50.415156+0000","last_undegraded":"2024-09-19T07:49:50.415156+0000","last_fullsized":"2024-09-19T07:49:50.415156+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4906","last_scrub_stamp":"2024-09-19T07:49:50.414805+0000","last_deep_scrub":"64'4906","last_deep_scrub_stamp":"2024-09-19T07:49:50.414805+0000","last_clean_scrub_stamp":"2024-09-19T07:49:50.414805+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:45:32.491873+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27102,"num_write_kb":350472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"61'4338","reported_seq":4461,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.486534+0000","last_change":"2024-09-19T07:49:29.486599+0000","last_active":"2024-09-19T07:49:29.486534+0000","last_peered":"2024-09-19T07:49:29.486534+0000","last_clean":"2024-09-19T07:49:29.486534+0000","last_became_active":"2024-09-19T07:19:32.290780+0000","last_became_peered":"2024-09-19T07:19:32.290780+0000","last_unstale":"2024-09-19T07:49:29.486534+0000","last_undegraded":"2024-09-19T07:49:29.486534+0000","last_fullsized":"2024-09-19T07:49:29.486534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4338","last_scrub_stamp":"2024-09-19T07:49:29.486317+0000","last_deep_scrub":"61'4338","last_deep_scrub_stamp":"2024-09-19T07:49:29.486317+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.486317+0000","objects_scrubbed":0,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:22.201418+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22767,"num_write_kb":271598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"61'3","reported_seq":128,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.481888+0000","last_change":"2024-09-19T07:49:35.481953+0000","last_active":"2024-09-19T07:49:35.481888+0000","last_peered":"2024-09-19T07:49:35.481888+0000","last_clean":"2024-09-19T07:49:35.481888+0000","last_became_active":"2024-09-19T07:19:31.281134+0000","last_became_peered":"2024-09-19T07:19:31.281134+0000","last_unstale":"2024-09-19T07:49:35.481888+0000","last_undegraded":"2024-09-19T07:49:35.481888+0000","last_fullsized":"2024-09-19T07:49:35.481888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-19T07:49:35.481692+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-19T07:49:35.481692+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.481692+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:13:00.655487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"65'119","reported_seq":284,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.332555+0000","last_change":"2024-09-19T07:49:35.332555+0000","last_active":"2024-09-19T07:49:35.332555+0000","last_peered":"2024-09-19T07:49:35.332555+0000","last_clean":"2024-09-19T07:49:35.332555+0000","last_became_active":"2024-09-19T07:19:22.627810+0000","last_became_peered":"2024-09-19T07:19:22.627810+0000","last_unstale":"2024-09-19T07:49:35.332555+0000","last_undegraded":"2024-09-19T07:49:35.332555+0000","last_fullsized":"2024-09-19T07:49:35.332555+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'119","last_scrub_stamp":"2024-09-19T07:49:35.332138+0000","last_deep_scrub":"65'119","last_deep_scrub_stamp":"2024-09-19T07:49:35.332138+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.332138+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:01:01.847510+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'2","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.549773+0000","last_change":"2024-09-19T07:49:35.549832+0000","last_active":"2024-09-19T07:49:35.549773+0000","last_peered":"2024-09-19T07:49:35.549773+0000","last_clean":"2024-09-19T07:49:35.549773+0000","last_became_active":"2024-09-19T07:19:39.428373+0000","last_became_peered":"2024-09-19T07:19:39.428373+0000","last_unstale":"2024-09-19T07:49:35.549773+0000","last_undegraded":"2024-09-19T07:49:35.549773+0000","last_fullsized":"2024-09-19T07:49:35.549773+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-19T07:49:35.549573+0000","last_deep_scrub":"64'2","last_deep_scrub_stamp":"2024-09-19T07:49:35.549573+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.549573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:20:28.110800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"64'3599","reported_seq":3713,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.636328+0000","last_change":"2024-09-19T07:49:33.636392+0000","last_active":"2024-09-19T07:49:33.636328+0000","last_peered":"2024-09-19T07:49:33.636328+0000","last_clean":"2024-09-19T07:49:33.636328+0000","last_became_active":"2024-09-19T07:19:40.381531+0000","last_became_peered":"2024-09-19T07:19:40.381531+0000","last_unstale":"2024-09-19T07:49:33.636328+0000","last_undegraded":"2024-09-19T07:49:33.636328+0000","last_fullsized":"2024-09-19T07:49:33.636328+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3599","last_scrub_stamp":"2024-09-19T07:49:33.636132+0000","last_deep_scrub":"64'3599","last_deep_scrub_stamp":"2024-09-19T07:49:33.636132+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.636132+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:09:04.310677+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19570,"num_write_kb":246554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"61'5018","reported_seq":5140,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:42.334896+0000","last_change":"2024-09-19T07:49:42.334955+0000","last_active":"2024-09-19T07:49:42.334896+0000","last_peered":"2024-09-19T07:49:42.334896+0000","last_clean":"2024-09-19T07:49:42.334896+0000","last_became_active":"2024-09-19T07:19:32.267889+0000","last_became_peered":"2024-09-19T07:19:32.267889+0000","last_unstale":"2024-09-19T07:49:42.334896+0000","last_undegraded":"2024-09-19T07:49:42.334896+0000","last_fullsized":"2024-09-19T07:49:42.334896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5018","last_scrub_stamp":"2024-09-19T07:49:42.334675+0000","last_deep_scrub":"61'5018","last_deep_scrub_stamp":"2024-09-19T07:49:42.334675+0000","last_clean_scrub_stamp":"2024-09-19T07:49:42.334675+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:35:58.841364+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26753,"num_write_kb":316367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"61'5","reported_seq":129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.694372+0000","last_change":"2024-09-19T07:49:24.694433+0000","last_active":"2024-09-19T07:49:24.694372+0000","last_peered":"2024-09-19T07:49:24.694372+0000","last_clean":"2024-09-19T07:49:24.694372+0000","last_became_active":"2024-09-19T07:19:31.285794+0000","last_became_peered":"2024-09-19T07:19:31.285794+0000","last_unstale":"2024-09-19T07:49:24.694372+0000","last_undegraded":"2024-09-19T07:49:24.694372+0000","last_fullsized":"2024-09-19T07:49:24.694372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5","last_scrub_stamp":"2024-09-19T07:49:24.694151+0000","last_deep_scrub":"61'5","last_deep_scrub_stamp":"2024-09-19T07:49:24.694151+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.694151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:06:52.476439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"64'3","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.447296+0000","last_change":"2024-09-19T07:49:25.447357+0000","last_active":"2024-09-19T07:49:25.447296+0000","last_peered":"2024-09-19T07:49:25.447296+0000","last_clean":"2024-09-19T07:49:25.447296+0000","last_became_active":"2024-09-19T07:19:39.424804+0000","last_became_peered":"2024-09-19T07:19:39.424804+0000","last_unstale":"2024-09-19T07:49:25.447296+0000","last_undegraded":"2024-09-19T07:49:25.447296+0000","last_fullsized":"2024-09-19T07:49:25.447296+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-09-19T07:49:25.447095+0000","last_deep_scrub":"64'3","last_deep_scrub_stamp":"2024-09-19T07:49:25.447095+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.447095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:56.882666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"64'4954","reported_seq":5068,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.594547+0000","last_change":"2024-09-19T07:49:35.594616+0000","last_active":"2024-09-19T07:49:35.594547+0000","last_peered":"2024-09-19T07:49:35.594547+0000","last_clean":"2024-09-19T07:49:35.594547+0000","last_became_active":"2024-09-19T07:19:40.389596+0000","last_became_peered":"2024-09-19T07:19:40.389596+0000","last_unstale":"2024-09-19T07:49:35.594547+0000","last_undegraded":"2024-09-19T07:49:35.594547+0000","last_fullsized":"2024-09-19T07:49:35.594547+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4954","last_scrub_stamp":"2024-09-19T07:49:35.594290+0000","last_deep_scrub":"64'4954","last_deep_scrub_stamp":"2024-09-19T07:49:35.594290+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.594290+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:56:14.519746+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27293,"num_write_kb":322594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"61'5220","reported_seq":5328,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.587513+0000","last_change":"2024-09-19T07:49:36.587574+0000","last_active":"2024-09-19T07:49:36.587513+0000","last_peered":"2024-09-19T07:49:36.587513+0000","last_clean":"2024-09-19T07:49:36.587513+0000","last_became_active":"2024-09-19T07:20:13.951449+0000","last_became_peered":"2024-09-19T07:20:13.951449+0000","last_unstale":"2024-09-19T07:49:36.587513+0000","last_undegraded":"2024-09-19T07:49:36.587513+0000","last_fullsized":"2024-09-19T07:49:36.587513+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5220","last_scrub_stamp":"2024-09-19T07:49:36.587313+0000","last_deep_scrub":"61'5220","last_deep_scrub_stamp":"2024-09-19T07:49:36.587313+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.587313+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:01:06.178431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27694,"num_write_kb":340542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"65'7","reported_seq":151,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.201648+0000","last_change":"2024-09-19T07:49:33.201712+0000","last_active":"2024-09-19T07:49:33.201648+0000","last_peered":"2024-09-19T07:49:33.201648+0000","last_clean":"2024-09-19T07:49:33.201648+0000","last_became_active":"2024-09-19T07:19:31.267917+0000","last_became_peered":"2024-09-19T07:19:31.267917+0000","last_unstale":"2024-09-19T07:49:33.201648+0000","last_undegraded":"2024-09-19T07:49:33.201648+0000","last_fullsized":"2024-09-19T07:49:33.201648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'7","last_scrub_stamp":"2024-09-19T07:49:33.201448+0000","last_deep_scrub":"65'7","last_deep_scrub_stamp":"2024-09-19T07:49:33.201448+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.201448+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:36:25.509689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'1405","reported_seq":1564,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.634210+0000","last_change":"2024-09-19T07:49:33.634354+0000","last_active":"2024-09-19T07:49:33.634210+0000","last_peered":"2024-09-19T07:49:33.634210+0000","last_clean":"2024-09-19T07:49:33.634210+0000","last_became_active":"2024-09-19T07:19:39.410338+0000","last_became_peered":"2024-09-19T07:19:39.410338+0000","last_unstale":"2024-09-19T07:49:33.634210+0000","last_undegraded":"2024-09-19T07:49:33.634210+0000","last_fullsized":"2024-09-19T07:49:33.634210+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1405","last_scrub_stamp":"2024-09-19T07:49:33.633973+0000","last_deep_scrub":"65'1405","last_deep_scrub_stamp":"2024-09-19T07:49:33.633973+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.633973+0000","objects_scrubbed":2,"log_size":1405,"log_dups_size":0,"ondisk_log_size":1405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:58:03.789148+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":63,"num_read_kb":30148,"num_write":1379,"num_write_kb":4466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"64'3869","reported_seq":3983,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.362788+0000","last_change":"2024-09-19T07:49:33.362844+0000","last_active":"2024-09-19T07:49:33.362788+0000","last_peered":"2024-09-19T07:49:33.362788+0000","last_clean":"2024-09-19T07:49:33.362788+0000","last_became_active":"2024-09-19T07:19:40.375815+0000","last_became_peered":"2024-09-19T07:19:40.375815+0000","last_unstale":"2024-09-19T07:49:33.362788+0000","last_undegraded":"2024-09-19T07:49:33.362788+0000","last_fullsized":"2024-09-19T07:49:33.362788+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3869","last_scrub_stamp":"2024-09-19T07:49:33.362596+0000","last_deep_scrub":"64'3869","last_deep_scrub_stamp":"2024-09-19T07:49:33.362596+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.362596+0000","objects_scrubbed":0,"log_size":3869,"log_dups_size":0,"ondisk_log_size":3869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:34:57.359555+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20920,"num_write_kb":264190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"61'6113","reported_seq":6221,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:46.132629+0000","last_change":"2024-09-19T07:49:46.132693+0000","last_active":"2024-09-19T07:49:46.132629+0000","last_peered":"2024-09-19T07:49:46.132629+0000","last_clean":"2024-09-19T07:49:46.132629+0000","last_became_active":"2024-09-19T07:20:14.209141+0000","last_became_peered":"2024-09-19T07:20:14.209141+0000","last_unstale":"2024-09-19T07:49:46.132629+0000","last_undegraded":"2024-09-19T07:49:46.132629+0000","last_fullsized":"2024-09-19T07:49:46.132629+0000","mapping_epoch":26,"log_start":"54'300","ondisk_log_start":"54'300","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'6113","last_scrub_stamp":"2024-09-19T07:49:46.132422+0000","last_deep_scrub":"61'6113","last_deep_scrub_stamp":"2024-09-19T07:49:46.132422+0000","last_clean_scrub_stamp":"2024-09-19T07:49:46.132422+0000","objects_scrubbed":0,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:16:37.679828+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32041,"num_write_kb":407182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"65'8","reported_seq":172,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.700404+0000","last_change":"2024-09-19T07:49:22.700453+0000","last_active":"2024-09-19T07:49:22.700404+0000","last_peered":"2024-09-19T07:49:22.700404+0000","last_clean":"2024-09-19T07:49:22.700404+0000","last_became_active":"2024-09-19T07:19:31.257857+0000","last_became_peered":"2024-09-19T07:19:31.257857+0000","last_unstale":"2024-09-19T07:49:22.700404+0000","last_undegraded":"2024-09-19T07:49:22.700404+0000","last_fullsized":"2024-09-19T07:49:22.700404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-19T07:49:22.700307+0000","last_deep_scrub":"65'8","last_deep_scrub_stamp":"2024-09-19T07:49:22.700307+0000","last_clean_scrub_stamp":"2024-09-19T07:49:22.700307+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:54:13.365948+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"64'5","reported_seq":122,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.516659+0000","last_change":"2024-09-19T07:49:26.516659+0000","last_active":"2024-09-19T07:49:26.516659+0000","last_peered":"2024-09-19T07:49:26.516659+0000","last_clean":"2024-09-19T07:49:26.516659+0000","last_became_active":"2024-09-19T07:19:39.436997+0000","last_became_peered":"2024-09-19T07:19:39.436997+0000","last_unstale":"2024-09-19T07:49:26.516659+0000","last_undegraded":"2024-09-19T07:49:26.516659+0000","last_fullsized":"2024-09-19T07:49:26.516659+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5","last_scrub_stamp":"2024-09-19T07:49:26.516376+0000","last_deep_scrub":"64'5","last_deep_scrub_stamp":"2024-09-19T07:49:26.516376+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.516376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:40:17.237269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"64'4634","reported_seq":4749,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.656792+0000","last_change":"2024-09-19T07:49:31.656792+0000","last_active":"2024-09-19T07:49:31.656792+0000","last_peered":"2024-09-19T07:49:31.656792+0000","last_clean":"2024-09-19T07:49:31.656792+0000","last_became_active":"2024-09-19T07:19:40.377685+0000","last_became_peered":"2024-09-19T07:19:40.377685+0000","last_unstale":"2024-09-19T07:49:31.656792+0000","last_undegraded":"2024-09-19T07:49:31.656792+0000","last_fullsized":"2024-09-19T07:49:31.656792+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4634","last_scrub_stamp":"2024-09-19T07:49:31.656515+0000","last_deep_scrub":"64'4634","last_deep_scrub_stamp":"2024-09-19T07:49:31.656515+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.656515+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:03:37.375425+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25462,"num_write_kb":298798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"61'4007","reported_seq":4129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.161629+0000","last_change":"2024-09-19T07:49:23.161741+0000","last_active":"2024-09-19T07:49:23.161629+0000","last_peered":"2024-09-19T07:49:23.161629+0000","last_clean":"2024-09-19T07:49:23.161629+0000","last_became_active":"2024-09-19T07:19:32.289059+0000","last_became_peered":"2024-09-19T07:19:32.289059+0000","last_unstale":"2024-09-19T07:49:23.161629+0000","last_undegraded":"2024-09-19T07:49:23.161629+0000","last_fullsized":"2024-09-19T07:49:23.161629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4007","last_scrub_stamp":"2024-09-19T07:49:23.161409+0000","last_deep_scrub":"61'4007","last_deep_scrub_stamp":"2024-09-19T07:49:23.161409+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.161409+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:38:54.800638+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21937,"num_write_kb":251948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"65'5","reported_seq":149,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.252104+0000","last_change":"2024-09-19T07:49:29.252167+0000","last_active":"2024-09-19T07:49:29.252104+0000","last_peered":"2024-09-19T07:49:29.252104+0000","last_clean":"2024-09-19T07:49:29.252104+0000","last_became_active":"2024-09-19T07:19:31.252325+0000","last_became_peered":"2024-09-19T07:19:31.252325+0000","last_unstale":"2024-09-19T07:49:29.252104+0000","last_undegraded":"2024-09-19T07:49:29.252104+0000","last_fullsized":"2024-09-19T07:49:29.252104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-19T07:49:29.251898+0000","last_deep_scrub":"65'5","last_deep_scrub_stamp":"2024-09-19T07:49:29.251898+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.251898+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:01:22.587094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"64'2","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.343672+0000","last_change":"2024-09-19T07:49:23.343732+0000","last_active":"2024-09-19T07:49:23.343672+0000","last_peered":"2024-09-19T07:49:23.343672+0000","last_clean":"2024-09-19T07:49:23.343672+0000","last_became_active":"2024-09-19T07:19:39.412921+0000","last_became_peered":"2024-09-19T07:19:39.412921+0000","last_unstale":"2024-09-19T07:49:23.343672+0000","last_undegraded":"2024-09-19T07:49:23.343672+0000","last_fullsized":"2024-09-19T07:49:23.343672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-19T07:49:23.343475+0000","last_deep_scrub":"64'2","last_deep_scrub_stamp":"2024-09-19T07:49:23.343475+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.343475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:48:49.929603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"64'3939","reported_seq":4052,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:49.509231+0000","last_change":"2024-09-19T07:49:49.509295+0000","last_active":"2024-09-19T07:49:49.509231+0000","last_peered":"2024-09-19T07:49:49.509231+0000","last_clean":"2024-09-19T07:49:49.509231+0000","last_became_active":"2024-09-19T07:19:40.386650+0000","last_became_peered":"2024-09-19T07:19:40.386650+0000","last_unstale":"2024-09-19T07:49:49.509231+0000","last_undegraded":"2024-09-19T07:49:49.509231+0000","last_fullsized":"2024-09-19T07:49:49.509231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3939","last_scrub_stamp":"2024-09-19T07:49:49.509015+0000","last_deep_scrub":"64'3939","last_deep_scrub_stamp":"2024-09-19T07:49:49.509015+0000","last_clean_scrub_stamp":"2024-09-19T07:49:49.509015+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:16:48.758697+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20741,"num_write_kb":275861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'4660","reported_seq":4784,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.239268+0000","last_change":"2024-09-19T07:49:25.239268+0000","last_active":"2024-09-19T07:49:25.239268+0000","last_peered":"2024-09-19T07:49:25.239268+0000","last_clean":"2024-09-19T07:49:25.239268+0000","last_became_active":"2024-09-19T07:19:32.313339+0000","last_became_peered":"2024-09-19T07:19:32.313339+0000","last_unstale":"2024-09-19T07:49:25.239268+0000","last_undegraded":"2024-09-19T07:49:25.239268+0000","last_fullsized":"2024-09-19T07:49:25.239268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4660","last_scrub_stamp":"2024-09-19T07:49:25.238988+0000","last_deep_scrub":"61'4660","last_deep_scrub_stamp":"2024-09-19T07:49:25.238988+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.238988+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:00:54.830107+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25264,"num_write_kb":302007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"65'19","reported_seq":227,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:21.677922+0000","last_change":"2024-09-19T07:49:21.677922+0000","last_active":"2024-09-19T07:49:21.677922+0000","last_peered":"2024-09-19T07:49:21.677922+0000","last_clean":"2024-09-19T07:49:21.677922+0000","last_became_active":"2024-09-19T07:19:31.256853+0000","last_became_peered":"2024-09-19T07:19:31.256853+0000","last_unstale":"2024-09-19T07:49:21.677922+0000","last_undegraded":"2024-09-19T07:49:21.677922+0000","last_fullsized":"2024-09-19T07:49:21.677922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-19T07:49:21.677795+0000","last_deep_scrub":"65'19","last_deep_scrub_stamp":"2024-09-19T07:49:21.677795+0000","last_clean_scrub_stamp":"2024-09-19T07:49:21.677795+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:28:00.139065+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"64'3","reported_seq":118,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.555137+0000","last_change":"2024-09-19T07:49:23.555201+0000","last_active":"2024-09-19T07:49:23.555137+0000","last_peered":"2024-09-19T07:49:23.555137+0000","last_clean":"2024-09-19T07:49:23.555137+0000","last_became_active":"2024-09-19T07:19:39.405753+0000","last_became_peered":"2024-09-19T07:19:39.405753+0000","last_unstale":"2024-09-19T07:49:23.555137+0000","last_undegraded":"2024-09-19T07:49:23.555137+0000","last_fullsized":"2024-09-19T07:49:23.555137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-09-19T07:49:23.554923+0000","last_deep_scrub":"64'3","last_deep_scrub_stamp":"2024-09-19T07:49:23.554923+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.554923+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:14:31.786151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"64'4590","reported_seq":4703,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.466393+0000","last_change":"2024-09-19T07:49:32.466460+0000","last_active":"2024-09-19T07:49:32.466393+0000","last_peered":"2024-09-19T07:49:32.466393+0000","last_clean":"2024-09-19T07:49:32.466393+0000","last_became_active":"2024-09-19T07:19:40.365007+0000","last_became_peered":"2024-09-19T07:19:40.365007+0000","last_unstale":"2024-09-19T07:49:32.466393+0000","last_undegraded":"2024-09-19T07:49:32.466393+0000","last_fullsized":"2024-09-19T07:49:32.466393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4590","last_scrub_stamp":"2024-09-19T07:49:32.466159+0000","last_deep_scrub":"64'4590","last_deep_scrub_stamp":"2024-09-19T07:49:32.466159+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.466159+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:37.762557+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24790,"num_write_kb":304361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"61'3873","reported_seq":3995,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.549376+0000","last_change":"2024-09-19T07:49:34.549376+0000","last_active":"2024-09-19T07:49:34.549376+0000","last_peered":"2024-09-19T07:49:34.549376+0000","last_clean":"2024-09-19T07:49:34.549376+0000","last_became_active":"2024-09-19T07:19:32.305540+0000","last_became_peered":"2024-09-19T07:19:32.305540+0000","last_unstale":"2024-09-19T07:49:34.549376+0000","last_undegraded":"2024-09-19T07:49:34.549376+0000","last_fullsized":"2024-09-19T07:49:34.549376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3873","last_scrub_stamp":"2024-09-19T07:49:34.549125+0000","last_deep_scrub":"61'3873","last_deep_scrub_stamp":"2024-09-19T07:49:34.549125+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.549125+0000","objects_scrubbed":0,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:09:37.622826+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20508,"num_write_kb":250225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"61'5","reported_seq":129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.469507+0000","last_change":"2024-09-19T07:49:27.469507+0000","last_active":"2024-09-19T07:49:27.469507+0000","last_peered":"2024-09-19T07:49:27.469507+0000","last_clean":"2024-09-19T07:49:27.469507+0000","last_became_active":"2024-09-19T07:19:31.268364+0000","last_became_peered":"2024-09-19T07:19:31.268364+0000","last_unstale":"2024-09-19T07:49:27.469507+0000","last_undegraded":"2024-09-19T07:49:27.469507+0000","last_fullsized":"2024-09-19T07:49:27.469507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5","last_scrub_stamp":"2024-09-19T07:49:27.469254+0000","last_deep_scrub":"61'5","last_deep_scrub_stamp":"2024-09-19T07:49:27.469254+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.469254+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:09:27.876094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"64'4","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.467925+0000","last_change":"2024-09-19T07:49:28.467989+0000","last_active":"2024-09-19T07:49:28.467925+0000","last_peered":"2024-09-19T07:49:28.467925+0000","last_clean":"2024-09-19T07:49:28.467925+0000","last_became_active":"2024-09-19T07:19:39.429418+0000","last_became_peered":"2024-09-19T07:19:39.429418+0000","last_unstale":"2024-09-19T07:49:28.467925+0000","last_undegraded":"2024-09-19T07:49:28.467925+0000","last_fullsized":"2024-09-19T07:49:28.467925+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-09-19T07:49:28.467324+0000","last_deep_scrub":"64'4","last_deep_scrub_stamp":"2024-09-19T07:49:28.467324+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.467324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:36:20.775564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"64'3485","reported_seq":3598,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:45.485518+0000","last_change":"2024-09-19T07:49:45.485577+0000","last_active":"2024-09-19T07:49:45.485518+0000","last_peered":"2024-09-19T07:49:45.485518+0000","last_clean":"2024-09-19T07:49:45.485518+0000","last_became_active":"2024-09-19T07:19:40.372731+0000","last_became_peered":"2024-09-19T07:19:40.372731+0000","last_unstale":"2024-09-19T07:49:45.485518+0000","last_undegraded":"2024-09-19T07:49:45.485518+0000","last_fullsized":"2024-09-19T07:49:45.485518+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3485","last_scrub_stamp":"2024-09-19T07:49:45.485313+0000","last_deep_scrub":"64'3485","last_deep_scrub_stamp":"2024-09-19T07:49:45.485313+0000","last_clean_scrub_stamp":"2024-09-19T07:49:45.485313+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:21:29.905290+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19669,"num_write_kb":242834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"61'5231","reported_seq":5353,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.282828+0000","last_change":"2024-09-19T07:49:27.282828+0000","last_active":"2024-09-19T07:49:27.282828+0000","last_peered":"2024-09-19T07:49:27.282828+0000","last_clean":"2024-09-19T07:49:27.282828+0000","last_became_active":"2024-09-19T07:19:32.313480+0000","last_became_peered":"2024-09-19T07:19:32.313480+0000","last_unstale":"2024-09-19T07:49:27.282828+0000","last_undegraded":"2024-09-19T07:49:27.282828+0000","last_fullsized":"2024-09-19T07:49:27.282828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5231","last_scrub_stamp":"2024-09-19T07:49:27.282524+0000","last_deep_scrub":"61'5231","last_deep_scrub_stamp":"2024-09-19T07:49:27.282524+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.282524+0000","objects_scrubbed":0,"log_size":5231,"log_dups_size":0,"ondisk_log_size":5231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:26:58.138512+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28071,"num_write_kb":333869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"61'3","reported_seq":127,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.439571+0000","last_change":"2024-09-19T07:49:36.439571+0000","last_active":"2024-09-19T07:49:36.439571+0000","last_peered":"2024-09-19T07:49:36.439571+0000","last_clean":"2024-09-19T07:49:36.439571+0000","last_became_active":"2024-09-19T07:19:31.285430+0000","last_became_peered":"2024-09-19T07:19:31.285430+0000","last_unstale":"2024-09-19T07:49:36.439571+0000","last_undegraded":"2024-09-19T07:49:36.439571+0000","last_fullsized":"2024-09-19T07:49:36.439571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-19T07:49:36.439312+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-19T07:49:36.439312+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.439312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:45:50.370417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'331","reported_seq":547,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.593122+0000","last_change":"2024-09-19T07:49:24.593194+0000","last_active":"2024-09-19T07:49:24.593122+0000","last_peered":"2024-09-19T07:49:24.593122+0000","last_clean":"2024-09-19T07:49:24.593122+0000","last_became_active":"2024-09-19T07:19:39.410796+0000","last_became_peered":"2024-09-19T07:19:39.410796+0000","last_unstale":"2024-09-19T07:49:24.593122+0000","last_undegraded":"2024-09-19T07:49:24.593122+0000","last_fullsized":"2024-09-19T07:49:24.593122+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'331","last_scrub_stamp":"2024-09-19T07:49:24.592917+0000","last_deep_scrub":"65'331","last_deep_scrub_stamp":"2024-09-19T07:49:24.592917+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.592917+0000","objects_scrubbed":4,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:26:55.448905+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":181,"num_read_kb":77990,"num_write":325,"num_write_kb":4282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"64'5112","reported_seq":5225,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:44.483805+0000","last_change":"2024-09-19T07:49:44.483880+0000","last_active":"2024-09-19T07:49:44.483805+0000","last_peered":"2024-09-19T07:49:44.483805+0000","last_clean":"2024-09-19T07:49:44.483805+0000","last_became_active":"2024-09-19T07:19:40.364004+0000","last_became_peered":"2024-09-19T07:19:40.364004+0000","last_unstale":"2024-09-19T07:49:44.483805+0000","last_undegraded":"2024-09-19T07:49:44.483805+0000","last_fullsized":"2024-09-19T07:49:44.483805+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5112","last_scrub_stamp":"2024-09-19T07:49:44.483615+0000","last_deep_scrub":"64'5112","last_deep_scrub_stamp":"2024-09-19T07:49:44.483615+0000","last_clean_scrub_stamp":"2024-09-19T07:49:44.483615+0000","objects_scrubbed":0,"log_size":5112,"log_dups_size":0,"ondisk_log_size":5112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:03:53.635587+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27577,"num_write_kb":360586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"61'4085","reported_seq":4207,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.581403+0000","last_change":"2024-09-19T07:49:32.581403+0000","last_active":"2024-09-19T07:49:32.581403+0000","last_peered":"2024-09-19T07:49:32.581403+0000","last_clean":"2024-09-19T07:49:32.581403+0000","last_became_active":"2024-09-19T07:19:32.279922+0000","last_became_peered":"2024-09-19T07:19:32.279922+0000","last_unstale":"2024-09-19T07:49:32.581403+0000","last_undegraded":"2024-09-19T07:49:32.581403+0000","last_fullsized":"2024-09-19T07:49:32.581403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4085","last_scrub_stamp":"2024-09-19T07:49:32.581108+0000","last_deep_scrub":"61'4085","last_deep_scrub_stamp":"2024-09-19T07:49:32.581108+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.581108+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:50:54.490092+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22122,"num_write_kb":273218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"65'12","reported_seq":176,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.596695+0000","last_change":"2024-09-19T07:49:25.596695+0000","last_active":"2024-09-19T07:49:25.596695+0000","last_peered":"2024-09-19T07:49:25.596695+0000","last_clean":"2024-09-19T07:49:25.596695+0000","last_became_active":"2024-09-19T07:19:31.262656+0000","last_became_peered":"2024-09-19T07:19:31.262656+0000","last_unstale":"2024-09-19T07:49:25.596695+0000","last_undegraded":"2024-09-19T07:49:25.596695+0000","last_fullsized":"2024-09-19T07:49:25.596695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-19T07:49:25.596435+0000","last_deep_scrub":"65'12","last_deep_scrub_stamp":"2024-09-19T07:49:25.596435+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.596435+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:16.739671+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'30","reported_seq":205,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.529991+0000","last_change":"2024-09-19T07:49:24.530059+0000","last_active":"2024-09-19T07:49:24.529991+0000","last_peered":"2024-09-19T07:49:24.529991+0000","last_clean":"2024-09-19T07:49:24.529991+0000","last_became_active":"2024-09-19T07:19:39.430756+0000","last_became_peered":"2024-09-19T07:19:39.430756+0000","last_unstale":"2024-09-19T07:49:24.529991+0000","last_undegraded":"2024-09-19T07:49:24.529991+0000","last_fullsized":"2024-09-19T07:49:24.529991+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-09-19T07:49:24.529796+0000","last_deep_scrub":"65'30","last_deep_scrub_stamp":"2024-09-19T07:49:24.529796+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.529796+0000","objects_scrubbed":2,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:50:09.224306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"64'4543","reported_seq":4657,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:47.475189+0000","last_change":"2024-09-19T07:49:47.475189+0000","last_active":"2024-09-19T07:49:47.475189+0000","last_peered":"2024-09-19T07:49:47.475189+0000","last_clean":"2024-09-19T07:49:47.475189+0000","last_became_active":"2024-09-19T07:19:40.380643+0000","last_became_peered":"2024-09-19T07:19:40.380643+0000","last_unstale":"2024-09-19T07:49:47.475189+0000","last_undegraded":"2024-09-19T07:49:47.475189+0000","last_fullsized":"2024-09-19T07:49:47.475189+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4543","last_scrub_stamp":"2024-09-19T07:49:47.474937+0000","last_deep_scrub":"64'4543","last_deep_scrub_stamp":"2024-09-19T07:49:47.474937+0000","last_clean_scrub_stamp":"2024-09-19T07:49:47.474937+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:16:21.296904+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24993,"num_write_kb":317373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"61'3981","reported_seq":4102,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.312053+0000","last_change":"2024-09-19T07:49:28.312116+0000","last_active":"2024-09-19T07:49:28.312053+0000","last_peered":"2024-09-19T07:49:28.312053+0000","last_clean":"2024-09-19T07:49:28.312053+0000","last_became_active":"2024-09-19T07:19:32.283296+0000","last_became_peered":"2024-09-19T07:19:32.283296+0000","last_unstale":"2024-09-19T07:49:28.312053+0000","last_undegraded":"2024-09-19T07:49:28.312053+0000","last_fullsized":"2024-09-19T07:49:28.312053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3981","last_scrub_stamp":"2024-09-19T07:49:28.311812+0000","last_deep_scrub":"61'3981","last_deep_scrub_stamp":"2024-09-19T07:49:28.311812+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.311812+0000","objects_scrubbed":0,"log_size":3981,"log_dups_size":0,"ondisk_log_size":3981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:56.707334+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21890,"num_write_kb":263866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"65'355","reported_seq":580,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.492099+0000","last_change":"2024-09-19T07:49:29.492099+0000","last_active":"2024-09-19T07:49:29.492099+0000","last_peered":"2024-09-19T07:49:29.492099+0000","last_clean":"2024-09-19T07:49:29.492099+0000","last_became_active":"2024-09-19T07:19:31.271957+0000","last_became_peered":"2024-09-19T07:19:31.271957+0000","last_unstale":"2024-09-19T07:49:29.492099+0000","last_undegraded":"2024-09-19T07:49:29.492099+0000","last_fullsized":"2024-09-19T07:49:29.492099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'355","last_scrub_stamp":"2024-09-19T07:49:29.491736+0000","last_deep_scrub":"65'355","last_deep_scrub_stamp":"2024-09-19T07:49:29.491736+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.491736+0000","objects_scrubbed":4,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:46:11.249153+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":181,"num_read_kb":77990,"num_write":347,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"64'3","reported_seq":119,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.309551+0000","last_change":"2024-09-19T07:49:27.309551+0000","last_active":"2024-09-19T07:49:27.309551+0000","last_peered":"2024-09-19T07:49:27.309551+0000","last_clean":"2024-09-19T07:49:27.309551+0000","last_became_active":"2024-09-19T07:19:39.423749+0000","last_became_peered":"2024-09-19T07:19:39.423749+0000","last_unstale":"2024-09-19T07:49:27.309551+0000","last_undegraded":"2024-09-19T07:49:27.309551+0000","last_fullsized":"2024-09-19T07:49:27.309551+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-09-19T07:49:27.309274+0000","last_deep_scrub":"64'3","last_deep_scrub_stamp":"2024-09-19T07:49:27.309274+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.309274+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:06:26.603734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"64'3743","reported_seq":3857,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.155886+0000","last_change":"2024-09-19T07:49:40.155886+0000","last_active":"2024-09-19T07:49:40.155886+0000","last_peered":"2024-09-19T07:49:40.155886+0000","last_clean":"2024-09-19T07:49:40.155886+0000","last_became_active":"2024-09-19T07:19:40.381293+0000","last_became_peered":"2024-09-19T07:19:40.381293+0000","last_unstale":"2024-09-19T07:49:40.155886+0000","last_undegraded":"2024-09-19T07:49:40.155886+0000","last_fullsized":"2024-09-19T07:49:40.155886+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3743","last_scrub_stamp":"2024-09-19T07:49:40.155596+0000","last_deep_scrub":"64'3743","last_deep_scrub_stamp":"2024-09-19T07:49:40.155596+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.155596+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:57:58.883136+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20626,"num_write_kb":263086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"61'6160","reported_seq":6281,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.351108+0000","last_change":"2024-09-19T07:49:31.351169+0000","last_active":"2024-09-19T07:49:31.351108+0000","last_peered":"2024-09-19T07:49:31.351108+0000","last_clean":"2024-09-19T07:49:31.351108+0000","last_became_active":"2024-09-19T07:19:32.288226+0000","last_became_peered":"2024-09-19T07:19:32.288226+0000","last_unstale":"2024-09-19T07:49:31.351108+0000","last_undegraded":"2024-09-19T07:49:31.351108+0000","last_fullsized":"2024-09-19T07:49:31.351108+0000","mapping_epoch":19,"log_start":"54'700","ondisk_log_start":"54'700","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'6160","last_scrub_stamp":"2024-09-19T07:49:31.350899+0000","last_deep_scrub":"61'6160","last_deep_scrub_stamp":"2024-09-19T07:49:31.350899+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.350899+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:24:40.317687+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32461,"num_write_kb":376158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"65'31","reported_seq":214,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.192661+0000","last_change":"2024-09-19T07:49:36.192783+0000","last_active":"2024-09-19T07:49:36.192661+0000","last_peered":"2024-09-19T07:49:36.192661+0000","last_clean":"2024-09-19T07:49:36.192661+0000","last_became_active":"2024-09-19T07:19:31.290408+0000","last_became_peered":"2024-09-19T07:19:31.290408+0000","last_unstale":"2024-09-19T07:49:36.192661+0000","last_undegraded":"2024-09-19T07:49:36.192661+0000","last_fullsized":"2024-09-19T07:49:36.192661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'31","last_scrub_stamp":"2024-09-19T07:49:36.192384+0000","last_deep_scrub":"65'31","last_deep_scrub_stamp":"2024-09-19T07:49:36.192384+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.192384+0000","objects_scrubbed":2,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:17:00.993654+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":30,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'10","reported_seq":165,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.312623+0000","last_change":"2024-09-19T07:49:27.312693+0000","last_active":"2024-09-19T07:49:27.312623+0000","last_peered":"2024-09-19T07:49:27.312623+0000","last_clean":"2024-09-19T07:49:27.312623+0000","last_became_active":"2024-09-19T07:19:39.424173+0000","last_became_peered":"2024-09-19T07:19:39.424173+0000","last_unstale":"2024-09-19T07:49:27.312623+0000","last_undegraded":"2024-09-19T07:49:27.312623+0000","last_fullsized":"2024-09-19T07:49:27.312623+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-19T07:49:27.312421+0000","last_deep_scrub":"65'10","last_deep_scrub_stamp":"2024-09-19T07:49:27.312421+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.312421+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:33:29.026544+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"64'4564","reported_seq":4680,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.546525+0000","last_change":"2024-09-19T07:49:40.546525+0000","last_active":"2024-09-19T07:49:40.546525+0000","last_peered":"2024-09-19T07:49:40.546525+0000","last_clean":"2024-09-19T07:49:40.546525+0000","last_became_active":"2024-09-19T07:19:40.383478+0000","last_became_peered":"2024-09-19T07:19:40.383478+0000","last_unstale":"2024-09-19T07:49:40.546525+0000","last_undegraded":"2024-09-19T07:49:40.546525+0000","last_fullsized":"2024-09-19T07:49:40.546525+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4564","last_scrub_stamp":"2024-09-19T07:49:40.546195+0000","last_deep_scrub":"64'4564","last_deep_scrub_stamp":"2024-09-19T07:49:40.546195+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.546195+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:23.408969+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25533,"num_write_kb":303195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"61'5068","reported_seq":5189,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.605132+0000","last_change":"2024-09-19T07:49:28.605207+0000","last_active":"2024-09-19T07:49:28.605132+0000","last_peered":"2024-09-19T07:49:28.605132+0000","last_clean":"2024-09-19T07:49:28.605132+0000","last_became_active":"2024-09-19T07:19:32.270517+0000","last_became_peered":"2024-09-19T07:19:32.270517+0000","last_unstale":"2024-09-19T07:49:28.605132+0000","last_undegraded":"2024-09-19T07:49:28.605132+0000","last_fullsized":"2024-09-19T07:49:28.605132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5068","last_scrub_stamp":"2024-09-19T07:49:28.604922+0000","last_deep_scrub":"61'5068","last_deep_scrub_stamp":"2024-09-19T07:49:28.604922+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.604922+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:02:59.298244+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26828,"num_write_kb":316657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"61'3","reported_seq":126,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.401810+0000","last_change":"2024-09-19T07:49:38.401948+0000","last_active":"2024-09-19T07:49:38.401810+0000","last_peered":"2024-09-19T07:49:38.401810+0000","last_clean":"2024-09-19T07:49:38.401810+0000","last_became_active":"2024-09-19T07:19:31.285157+0000","last_became_peered":"2024-09-19T07:19:31.285157+0000","last_unstale":"2024-09-19T07:49:38.401810+0000","last_undegraded":"2024-09-19T07:49:38.401810+0000","last_fullsized":"2024-09-19T07:49:38.401810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-19T07:49:38.401480+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-19T07:49:38.401480+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.401480+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:45:57.064807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'19","reported_seq":219,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:21.625685+0000","last_change":"2024-09-19T07:49:21.625685+0000","last_active":"2024-09-19T07:49:21.625685+0000","last_peered":"2024-09-19T07:49:21.625685+0000","last_clean":"2024-09-19T07:49:21.625685+0000","last_became_active":"2024-09-19T07:19:39.410125+0000","last_became_peered":"2024-09-19T07:19:39.410125+0000","last_unstale":"2024-09-19T07:49:21.625685+0000","last_undegraded":"2024-09-19T07:49:21.625685+0000","last_fullsized":"2024-09-19T07:49:21.625685+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-19T07:49:21.625564+0000","last_deep_scrub":"65'19","last_deep_scrub_stamp":"2024-09-19T07:49:21.625564+0000","last_clean_scrub_stamp":"2024-09-19T07:49:21.625564+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:17:31.726064+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"64'3771","reported_seq":3885,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:48.375343+0000","last_change":"2024-09-19T07:49:48.375343+0000","last_active":"2024-09-19T07:49:48.375343+0000","last_peered":"2024-09-19T07:49:48.375343+0000","last_clean":"2024-09-19T07:49:48.375343+0000","last_became_active":"2024-09-19T07:19:40.386425+0000","last_became_peered":"2024-09-19T07:19:40.386425+0000","last_unstale":"2024-09-19T07:49:48.375343+0000","last_undegraded":"2024-09-19T07:49:48.375343+0000","last_fullsized":"2024-09-19T07:49:48.375343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3771","last_scrub_stamp":"2024-09-19T07:49:48.374947+0000","last_deep_scrub":"64'3771","last_deep_scrub_stamp":"2024-09-19T07:49:48.374947+0000","last_clean_scrub_stamp":"2024-09-19T07:49:48.374947+0000","objects_scrubbed":0,"log_size":3771,"log_dups_size":0,"ondisk_log_size":3771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:15:24.042312+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19869,"num_write_kb":255725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"61'4973","reported_seq":5094,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.455812+0000","last_change":"2024-09-19T07:49:39.455899+0000","last_active":"2024-09-19T07:49:39.455812+0000","last_peered":"2024-09-19T07:49:39.455812+0000","last_clean":"2024-09-19T07:49:39.455812+0000","last_became_active":"2024-09-19T07:19:32.271420+0000","last_became_peered":"2024-09-19T07:19:32.271420+0000","last_unstale":"2024-09-19T07:49:39.455812+0000","last_undegraded":"2024-09-19T07:49:39.455812+0000","last_fullsized":"2024-09-19T07:49:39.455812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4973","last_scrub_stamp":"2024-09-19T07:49:39.455603+0000","last_deep_scrub":"61'4973","last_deep_scrub_stamp":"2024-09-19T07:49:39.455603+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.455603+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:46:11.619885+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26723,"num_write_kb":320185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"65'6","reported_seq":129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.502511+0000","last_change":"2024-09-19T07:49:31.502568+0000","last_active":"2024-09-19T07:49:31.502511+0000","last_peered":"2024-09-19T07:49:31.502511+0000","last_clean":"2024-09-19T07:49:31.502511+0000","last_became_active":"2024-09-19T07:19:31.292974+0000","last_became_peered":"2024-09-19T07:19:31.292974+0000","last_unstale":"2024-09-19T07:49:31.502511+0000","last_undegraded":"2024-09-19T07:49:31.502511+0000","last_fullsized":"2024-09-19T07:49:31.502511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-19T07:49:31.502321+0000","last_deep_scrub":"65'6","last_deep_scrub_stamp":"2024-09-19T07:49:31.502321+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.502321+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:39:03.141550+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"64'5","reported_seq":121,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.520297+0000","last_change":"2024-09-19T07:49:26.520297+0000","last_active":"2024-09-19T07:49:26.520297+0000","last_peered":"2024-09-19T07:49:26.520297+0000","last_clean":"2024-09-19T07:49:26.520297+0000","last_became_active":"2024-09-19T07:19:39.416696+0000","last_became_peered":"2024-09-19T07:19:39.416696+0000","last_unstale":"2024-09-19T07:49:26.520297+0000","last_undegraded":"2024-09-19T07:49:26.520297+0000","last_fullsized":"2024-09-19T07:49:26.520297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5","last_scrub_stamp":"2024-09-19T07:49:26.520032+0000","last_deep_scrub":"64'5","last_deep_scrub_stamp":"2024-09-19T07:49:26.520032+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.520032+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:15.048957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"64'5394","reported_seq":5509,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:41.541715+0000","last_change":"2024-09-19T07:49:41.541862+0000","last_active":"2024-09-19T07:49:41.541715+0000","last_peered":"2024-09-19T07:49:41.541715+0000","last_clean":"2024-09-19T07:49:41.541715+0000","last_became_active":"2024-09-19T07:19:40.394547+0000","last_became_peered":"2024-09-19T07:19:40.394547+0000","last_unstale":"2024-09-19T07:49:41.541715+0000","last_undegraded":"2024-09-19T07:49:41.541715+0000","last_fullsized":"2024-09-19T07:49:41.541715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5394","last_scrub_stamp":"2024-09-19T07:49:41.541461+0000","last_deep_scrub":"64'5394","last_deep_scrub_stamp":"2024-09-19T07:49:41.541461+0000","last_clean_scrub_stamp":"2024-09-19T07:49:41.541461+0000","objects_scrubbed":0,"log_size":5394,"log_dups_size":0,"ondisk_log_size":5394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:34.256562+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30095,"num_write_kb":374705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"64'25","reported_seq":143,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.505929+0000","last_change":"2024-09-19T07:49:36.505929+0000","last_active":"2024-09-19T07:49:36.505929+0000","last_peered":"2024-09-19T07:49:36.505929+0000","last_clean":"2024-09-19T07:49:36.505929+0000","last_became_active":"2024-09-19T07:19:39.418559+0000","last_became_peered":"2024-09-19T07:19:39.418559+0000","last_unstale":"2024-09-19T07:49:36.505929+0000","last_undegraded":"2024-09-19T07:49:36.505929+0000","last_fullsized":"2024-09-19T07:49:36.505929+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'25","last_scrub_stamp":"2024-09-19T07:49:36.505667+0000","last_deep_scrub":"64'25","last_deep_scrub_stamp":"2024-09-19T07:49:36.505667+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.505667+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:17:24.384017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"61'4822","reported_seq":4943,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.186878+0000","last_change":"2024-09-19T07:49:37.186938+0000","last_active":"2024-09-19T07:49:37.186878+0000","last_peered":"2024-09-19T07:49:37.186878+0000","last_clean":"2024-09-19T07:49:37.186878+0000","last_became_active":"2024-09-19T07:19:32.274523+0000","last_became_peered":"2024-09-19T07:19:32.274523+0000","last_unstale":"2024-09-19T07:49:37.186878+0000","last_undegraded":"2024-09-19T07:49:37.186878+0000","last_fullsized":"2024-09-19T07:49:37.186878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4822","last_scrub_stamp":"2024-09-19T07:49:37.186673+0000","last_deep_scrub":"61'4822","last_deep_scrub_stamp":"2024-09-19T07:49:37.186673+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.186673+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:01:03.702046+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25208,"num_write_kb":326524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"61'2","reported_seq":125,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.409363+0000","last_change":"2024-09-19T07:49:37.409447+0000","last_active":"2024-09-19T07:49:37.409363+0000","last_peered":"2024-09-19T07:49:37.409363+0000","last_clean":"2024-09-19T07:49:37.409363+0000","last_became_active":"2024-09-19T07:19:31.285186+0000","last_became_peered":"2024-09-19T07:19:31.285186+0000","last_unstale":"2024-09-19T07:49:37.409363+0000","last_undegraded":"2024-09-19T07:49:37.409363+0000","last_fullsized":"2024-09-19T07:49:37.409363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-19T07:49:37.409161+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-19T07:49:37.409161+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.409161+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:35:07.137047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"64'4690","reported_seq":4806,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.516668+0000","last_change":"2024-09-19T07:49:38.516668+0000","last_active":"2024-09-19T07:49:38.516668+0000","last_peered":"2024-09-19T07:49:38.516668+0000","last_clean":"2024-09-19T07:49:38.516668+0000","last_became_active":"2024-09-19T07:19:40.373320+0000","last_became_peered":"2024-09-19T07:19:40.373320+0000","last_unstale":"2024-09-19T07:49:38.516668+0000","last_undegraded":"2024-09-19T07:49:38.516668+0000","last_fullsized":"2024-09-19T07:49:38.516668+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4690","last_scrub_stamp":"2024-09-19T07:49:38.516396+0000","last_deep_scrub":"64'4690","last_deep_scrub_stamp":"2024-09-19T07:49:38.516396+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.516396+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:16:22.493171+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25924,"num_write_kb":313037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"65'967","reported_seq":1129,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:36.538996+0000","last_change":"2024-09-19T07:49:36.539068+0000","last_active":"2024-09-19T07:49:36.538996+0000","last_peered":"2024-09-19T07:49:36.538996+0000","last_clean":"2024-09-19T07:49:36.538996+0000","last_became_active":"2024-09-19T07:19:39.431705+0000","last_became_peered":"2024-09-19T07:19:39.431705+0000","last_unstale":"2024-09-19T07:49:36.538996+0000","last_undegraded":"2024-09-19T07:49:36.538996+0000","last_fullsized":"2024-09-19T07:49:36.538996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'967","last_scrub_stamp":"2024-09-19T07:49:36.538778+0000","last_deep_scrub":"65'967","last_deep_scrub_stamp":"2024-09-19T07:49:36.538778+0000","last_clean_scrub_stamp":"2024-09-19T07:49:36.538778+0000","objects_scrubbed":3,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:04.934300+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":8896,"num_write":940,"num_write_kb":4641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"61'5311","reported_seq":5432,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:41.440297+0000","last_change":"2024-09-19T07:49:41.440362+0000","last_active":"2024-09-19T07:49:41.440297+0000","last_peered":"2024-09-19T07:49:41.440297+0000","last_clean":"2024-09-19T07:49:41.440297+0000","last_became_active":"2024-09-19T07:19:32.276404+0000","last_became_peered":"2024-09-19T07:19:32.276404+0000","last_unstale":"2024-09-19T07:49:41.440297+0000","last_undegraded":"2024-09-19T07:49:41.440297+0000","last_fullsized":"2024-09-19T07:49:41.440297+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5311","last_scrub_stamp":"2024-09-19T07:49:41.440086+0000","last_deep_scrub":"61'5311","last_deep_scrub_stamp":"2024-09-19T07:49:41.440086+0000","last_clean_scrub_stamp":"2024-09-19T07:49:41.440086+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:16:32.987491+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28108,"num_write_kb":333978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"61'3","reported_seq":126,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.451372+0000","last_change":"2024-09-19T07:49:38.451435+0000","last_active":"2024-09-19T07:49:38.451372+0000","last_peered":"2024-09-19T07:49:38.451372+0000","last_clean":"2024-09-19T07:49:38.451372+0000","last_became_active":"2024-09-19T07:19:31.271357+0000","last_became_peered":"2024-09-19T07:19:31.271357+0000","last_unstale":"2024-09-19T07:49:38.451372+0000","last_undegraded":"2024-09-19T07:49:38.451372+0000","last_fullsized":"2024-09-19T07:49:38.451372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-19T07:49:38.451165+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-19T07:49:38.451165+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.451165+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:04:06.428929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"64'5270","reported_seq":5383,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:41.125862+0000","last_change":"2024-09-19T07:49:41.125862+0000","last_active":"2024-09-19T07:49:41.125862+0000","last_peered":"2024-09-19T07:49:41.125862+0000","last_clean":"2024-09-19T07:49:41.125862+0000","last_became_active":"2024-09-19T07:19:40.384044+0000","last_became_peered":"2024-09-19T07:19:40.384044+0000","last_unstale":"2024-09-19T07:49:41.125862+0000","last_undegraded":"2024-09-19T07:49:41.125862+0000","last_fullsized":"2024-09-19T07:49:41.125862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5270","last_scrub_stamp":"2024-09-19T07:49:41.125538+0000","last_deep_scrub":"64'5270","last_deep_scrub_stamp":"2024-09-19T07:49:41.125538+0000","last_clean_scrub_stamp":"2024-09-19T07:49:41.125538+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:38:52.454340+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29270,"num_write_kb":363254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"65'416","reported_seq":545,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.518997+0000","last_change":"2024-09-19T07:49:28.518997+0000","last_active":"2024-09-19T07:49:28.518997+0000","last_peered":"2024-09-19T07:49:28.518997+0000","last_clean":"2024-09-19T07:49:28.518997+0000","last_became_active":"2024-09-19T07:19:31.268768+0000","last_became_peered":"2024-09-19T07:19:31.268768+0000","last_unstale":"2024-09-19T07:49:28.518997+0000","last_undegraded":"2024-09-19T07:49:28.518997+0000","last_fullsized":"2024-09-19T07:49:28.518997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'416","last_scrub_stamp":"2024-09-19T07:49:28.518729+0000","last_deep_scrub":"65'416","last_deep_scrub_stamp":"2024-09-19T07:49:28.518729+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.518729+0000","objects_scrubbed":1,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:27:59.800607+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2715831,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6193,"num_write":390,"num_write_kb":2885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"61'3569","reported_seq":3690,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:20.599569+0000","last_change":"2024-09-19T07:49:20.599569+0000","last_active":"2024-09-19T07:49:20.599569+0000","last_peered":"2024-09-19T07:49:20.599569+0000","last_clean":"2024-09-19T07:49:20.599569+0000","last_became_active":"2024-09-19T07:19:32.289051+0000","last_became_peered":"2024-09-19T07:19:32.289051+0000","last_unstale":"2024-09-19T07:49:20.599569+0000","last_undegraded":"2024-09-19T07:49:20.599569+0000","last_fullsized":"2024-09-19T07:49:20.599569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3569","last_scrub_stamp":"2024-09-19T07:49:20.599393+0000","last_deep_scrub":"61'3569","last_deep_scrub_stamp":"2024-09-19T07:49:20.599393+0000","last_clean_scrub_stamp":"2024-09-19T07:49:20.599393+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:07:56.986977+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18120,"num_write_kb":244659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'6","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.632351+0000","last_change":"2024-09-19T07:49:27.632351+0000","last_active":"2024-09-19T07:49:27.632351+0000","last_peered":"2024-09-19T07:49:27.632351+0000","last_clean":"2024-09-19T07:49:27.632351+0000","last_became_active":"2024-09-19T07:19:39.410296+0000","last_became_peered":"2024-09-19T07:19:39.410296+0000","last_unstale":"2024-09-19T07:49:27.632351+0000","last_undegraded":"2024-09-19T07:49:27.632351+0000","last_fullsized":"2024-09-19T07:49:27.632351+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-19T07:49:27.632080+0000","last_deep_scrub":"65'6","last_deep_scrub_stamp":"2024-09-19T07:49:27.632080+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.632080+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:50:32.222516+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"64'3415","reported_seq":3530,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:44.409991+0000","last_change":"2024-09-19T07:49:44.410052+0000","last_active":"2024-09-19T07:49:44.409991+0000","last_peered":"2024-09-19T07:49:44.409991+0000","last_clean":"2024-09-19T07:49:44.409991+0000","last_became_active":"2024-09-19T07:19:40.390711+0000","last_became_peered":"2024-09-19T07:19:40.390711+0000","last_unstale":"2024-09-19T07:49:44.409991+0000","last_undegraded":"2024-09-19T07:49:44.409991+0000","last_fullsized":"2024-09-19T07:49:44.409991+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3415","last_scrub_stamp":"2024-09-19T07:49:44.409790+0000","last_deep_scrub":"64'3415","last_deep_scrub_stamp":"2024-09-19T07:49:44.409790+0000","last_clean_scrub_stamp":"2024-09-19T07:49:44.409790+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:01:43.992011+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18072,"num_write_kb":240550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"65'713","reported_seq":859,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.490236+0000","last_change":"2024-09-19T07:49:37.490300+0000","last_active":"2024-09-19T07:49:37.490236+0000","last_peered":"2024-09-19T07:49:37.490236+0000","last_clean":"2024-09-19T07:49:37.490236+0000","last_became_active":"2024-09-19T07:19:31.266085+0000","last_became_peered":"2024-09-19T07:19:31.266085+0000","last_unstale":"2024-09-19T07:49:37.490236+0000","last_undegraded":"2024-09-19T07:49:37.490236+0000","last_fullsized":"2024-09-19T07:49:37.490236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'713","last_scrub_stamp":"2024-09-19T07:49:37.490039+0000","last_deep_scrub":"65'713","last_deep_scrub_stamp":"2024-09-19T07:49:37.490039+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.490039+0000","objects_scrubbed":1,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:01:37.072261+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":74743,"num_write":701,"num_write_kb":4298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"61'4395","reported_seq":4517,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.353982+0000","last_change":"2024-09-19T07:49:33.353982+0000","last_active":"2024-09-19T07:49:33.353982+0000","last_peered":"2024-09-19T07:49:33.353982+0000","last_clean":"2024-09-19T07:49:33.353982+0000","last_became_active":"2024-09-19T07:19:32.291295+0000","last_became_peered":"2024-09-19T07:19:32.291295+0000","last_unstale":"2024-09-19T07:49:33.353982+0000","last_undegraded":"2024-09-19T07:49:33.353982+0000","last_fullsized":"2024-09-19T07:49:33.353982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4395","last_scrub_stamp":"2024-09-19T07:49:33.353716+0000","last_deep_scrub":"61'4395","last_deep_scrub_stamp":"2024-09-19T07:49:33.353716+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.353716+0000","objects_scrubbed":0,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:15:28.485880+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22455,"num_write_kb":278564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'41","reported_seq":196,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.583937+0000","last_change":"2024-09-19T07:49:22.583937+0000","last_active":"2024-09-19T07:49:22.583937+0000","last_peered":"2024-09-19T07:49:22.583937+0000","last_clean":"2024-09-19T07:49:22.583937+0000","last_became_active":"2024-09-19T07:19:39.410370+0000","last_became_peered":"2024-09-19T07:19:39.410370+0000","last_unstale":"2024-09-19T07:49:22.583937+0000","last_undegraded":"2024-09-19T07:49:22.583937+0000","last_fullsized":"2024-09-19T07:49:22.583937+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'41","last_scrub_stamp":"2024-09-19T07:49:22.583818+0000","last_deep_scrub":"65'41","last_deep_scrub_stamp":"2024-09-19T07:49:22.583818+0000","last_clean_scrub_stamp":"2024-09-19T07:49:22.583818+0000","objects_scrubbed":1,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:00:52.174937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"64'4798","reported_seq":4913,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:48.509998+0000","last_change":"2024-09-19T07:49:48.509998+0000","last_active":"2024-09-19T07:49:48.509998+0000","last_peered":"2024-09-19T07:49:48.509998+0000","last_clean":"2024-09-19T07:49:48.509998+0000","last_became_active":"2024-09-19T07:19:40.384872+0000","last_became_peered":"2024-09-19T07:19:40.384872+0000","last_unstale":"2024-09-19T07:49:48.509998+0000","last_undegraded":"2024-09-19T07:49:48.509998+0000","last_fullsized":"2024-09-19T07:49:48.509998+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4798","last_scrub_stamp":"2024-09-19T07:49:48.509620+0000","last_deep_scrub":"64'4798","last_deep_scrub_stamp":"2024-09-19T07:49:48.509620+0000","last_clean_scrub_stamp":"2024-09-19T07:49:48.509620+0000","objects_scrubbed":0,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:35:40.738288+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27082,"num_write_kb":321689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"61'4997","reported_seq":5120,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.488359+0000","last_change":"2024-09-19T07:49:26.488417+0000","last_active":"2024-09-19T07:49:26.488359+0000","last_peered":"2024-09-19T07:49:26.488359+0000","last_clean":"2024-09-19T07:49:26.488359+0000","last_became_active":"2024-09-19T07:19:32.306432+0000","last_became_peered":"2024-09-19T07:19:32.306432+0000","last_unstale":"2024-09-19T07:49:26.488359+0000","last_undegraded":"2024-09-19T07:49:26.488359+0000","last_fullsized":"2024-09-19T07:49:26.488359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4997","last_scrub_stamp":"2024-09-19T07:49:26.488156+0000","last_deep_scrub":"61'4997","last_deep_scrub_stamp":"2024-09-19T07:49:26.488156+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.488156+0000","objects_scrubbed":0,"log_size":4997,"log_dups_size":0,"ondisk_log_size":4997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:48:53.074284+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26206,"num_write_kb":309941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"64'5490","reported_seq":5606,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.150742+0000","last_change":"2024-09-19T07:49:39.150742+0000","last_active":"2024-09-19T07:49:39.150742+0000","last_peered":"2024-09-19T07:49:39.150742+0000","last_clean":"2024-09-19T07:49:39.150742+0000","last_became_active":"2024-09-19T07:19:40.376890+0000","last_became_peered":"2024-09-19T07:19:40.376890+0000","last_unstale":"2024-09-19T07:49:39.150742+0000","last_undegraded":"2024-09-19T07:49:39.150742+0000","last_fullsized":"2024-09-19T07:49:39.150742+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5490","last_scrub_stamp":"2024-09-19T07:49:39.149819+0000","last_deep_scrub":"64'5490","last_deep_scrub_stamp":"2024-09-19T07:49:39.149819+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.149819+0000","objects_scrubbed":0,"log_size":5490,"log_dups_size":0,"ondisk_log_size":5490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:19:16.579715+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29613,"num_write_kb":364653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"61'4890","reported_seq":5014,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.718723+0000","last_change":"2024-09-19T07:49:25.718783+0000","last_active":"2024-09-19T07:49:25.718723+0000","last_peered":"2024-09-19T07:49:25.718723+0000","last_clean":"2024-09-19T07:49:25.718723+0000","last_became_active":"2024-09-19T07:19:32.296747+0000","last_became_peered":"2024-09-19T07:19:32.296747+0000","last_unstale":"2024-09-19T07:49:25.718723+0000","last_undegraded":"2024-09-19T07:49:25.718723+0000","last_fullsized":"2024-09-19T07:49:25.718723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4890","last_scrub_stamp":"2024-09-19T07:49:25.718515+0000","last_deep_scrub":"61'4890","last_deep_scrub_stamp":"2024-09-19T07:49:25.718515+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.718515+0000","objects_scrubbed":0,"log_size":4890,"log_dups_size":0,"ondisk_log_size":4890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:21:01.552830+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25338,"num_write_kb":314559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"64'4569","reported_seq":4685,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:46.504936+0000","last_change":"2024-09-19T07:49:46.504936+0000","last_active":"2024-09-19T07:49:46.504936+0000","last_peered":"2024-09-19T07:49:46.504936+0000","last_clean":"2024-09-19T07:49:46.504936+0000","last_became_active":"2024-09-19T07:19:40.377482+0000","last_became_peered":"2024-09-19T07:19:40.377482+0000","last_unstale":"2024-09-19T07:49:46.504936+0000","last_undegraded":"2024-09-19T07:49:46.504936+0000","last_fullsized":"2024-09-19T07:49:46.504936+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4569","last_scrub_stamp":"2024-09-19T07:49:46.504663+0000","last_deep_scrub":"64'4569","last_deep_scrub_stamp":"2024-09-19T07:49:46.504663+0000","last_clean_scrub_stamp":"2024-09-19T07:49:46.504663+0000","objects_scrubbed":0,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:20:40.050018+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25275,"num_write_kb":314704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"61'5227","reported_seq":5352,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:41.530609+0000","last_change":"2024-09-19T07:49:41.530669+0000","last_active":"2024-09-19T07:49:41.530609+0000","last_peered":"2024-09-19T07:49:41.530609+0000","last_clean":"2024-09-19T07:49:41.530609+0000","last_became_active":"2024-09-19T07:19:32.274113+0000","last_became_peered":"2024-09-19T07:19:32.274113+0000","last_unstale":"2024-09-19T07:49:41.530609+0000","last_undegraded":"2024-09-19T07:49:41.530609+0000","last_fullsized":"2024-09-19T07:49:41.530609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5227","last_scrub_stamp":"2024-09-19T07:49:41.530388+0000","last_deep_scrub":"61'5227","last_deep_scrub_stamp":"2024-09-19T07:49:41.530388+0000","last_clean_scrub_stamp":"2024-09-19T07:49:41.530388+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:14.435066+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27444,"num_write_kb":331433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"64'4939","reported_seq":5054,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:42.162248+0000","last_change":"2024-09-19T07:49:42.162248+0000","last_active":"2024-09-19T07:49:42.162248+0000","last_peered":"2024-09-19T07:49:42.162248+0000","last_clean":"2024-09-19T07:49:42.162248+0000","last_became_active":"2024-09-19T07:19:40.384909+0000","last_became_peered":"2024-09-19T07:19:40.384909+0000","last_unstale":"2024-09-19T07:49:42.162248+0000","last_undegraded":"2024-09-19T07:49:42.162248+0000","last_fullsized":"2024-09-19T07:49:42.162248+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4939","last_scrub_stamp":"2024-09-19T07:49:42.161968+0000","last_deep_scrub":"64'4939","last_deep_scrub_stamp":"2024-09-19T07:49:42.161968+0000","last_clean_scrub_stamp":"2024-09-19T07:49:42.161968+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:01:41.744189+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27263,"num_write_kb":334786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"61'4385","reported_seq":4509,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.602889+0000","last_change":"2024-09-19T07:49:27.602949+0000","last_active":"2024-09-19T07:49:27.602889+0000","last_peered":"2024-09-19T07:49:27.602889+0000","last_clean":"2024-09-19T07:49:27.602889+0000","last_became_active":"2024-09-19T07:19:32.267819+0000","last_became_peered":"2024-09-19T07:19:32.267819+0000","last_unstale":"2024-09-19T07:49:27.602889+0000","last_undegraded":"2024-09-19T07:49:27.602889+0000","last_fullsized":"2024-09-19T07:49:27.602889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4385","last_scrub_stamp":"2024-09-19T07:49:27.602677+0000","last_deep_scrub":"61'4385","last_deep_scrub_stamp":"2024-09-19T07:49:27.602677+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.602677+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:34:12.330784+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22838,"num_write_kb":271812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"64'4859","reported_seq":4976,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:43.327057+0000","last_change":"2024-09-19T07:49:43.327125+0000","last_active":"2024-09-19T07:49:43.327057+0000","last_peered":"2024-09-19T07:49:43.327057+0000","last_clean":"2024-09-19T07:49:43.327057+0000","last_became_active":"2024-09-19T07:19:40.376782+0000","last_became_peered":"2024-09-19T07:19:40.376782+0000","last_unstale":"2024-09-19T07:49:43.327057+0000","last_undegraded":"2024-09-19T07:49:43.327057+0000","last_fullsized":"2024-09-19T07:49:43.327057+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4859","last_scrub_stamp":"2024-09-19T07:49:43.326828+0000","last_deep_scrub":"64'4859","last_deep_scrub_stamp":"2024-09-19T07:49:43.326828+0000","last_clean_scrub_stamp":"2024-09-19T07:49:43.326828+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:25.732457+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26164,"num_write_kb":338417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"61'4483","reported_seq":4607,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.536188+0000","last_change":"2024-09-19T07:49:25.536188+0000","last_active":"2024-09-19T07:49:25.536188+0000","last_peered":"2024-09-19T07:49:25.536188+0000","last_clean":"2024-09-19T07:49:25.536188+0000","last_became_active":"2024-09-19T07:19:32.312431+0000","last_became_peered":"2024-09-19T07:19:32.312431+0000","last_unstale":"2024-09-19T07:49:25.536188+0000","last_undegraded":"2024-09-19T07:49:25.536188+0000","last_fullsized":"2024-09-19T07:49:25.536188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4483","last_scrub_stamp":"2024-09-19T07:49:25.535753+0000","last_deep_scrub":"61'4483","last_deep_scrub_stamp":"2024-09-19T07:49:25.535753+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.535753+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:51:53.931275+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23870,"num_write_kb":292652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"64'4105","reported_seq":4221,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.471961+0000","last_change":"2024-09-19T07:49:38.472046+0000","last_active":"2024-09-19T07:49:38.471961+0000","last_peered":"2024-09-19T07:49:38.471961+0000","last_clean":"2024-09-19T07:49:38.471961+0000","last_became_active":"2024-09-19T07:19:40.374082+0000","last_became_peered":"2024-09-19T07:19:40.374082+0000","last_unstale":"2024-09-19T07:49:38.471961+0000","last_undegraded":"2024-09-19T07:49:38.471961+0000","last_fullsized":"2024-09-19T07:49:38.471961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4105","last_scrub_stamp":"2024-09-19T07:49:38.471730+0000","last_deep_scrub":"64'4105","last_deep_scrub_stamp":"2024-09-19T07:49:38.471730+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.471730+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:17:03.273001+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21923,"num_write_kb":284540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"61'5078","reported_seq":5201,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.144914+0000","last_change":"2024-09-19T07:49:38.144914+0000","last_active":"2024-09-19T07:49:38.144914+0000","last_peered":"2024-09-19T07:49:38.144914+0000","last_clean":"2024-09-19T07:49:38.144914+0000","last_became_active":"2024-09-19T07:19:32.273504+0000","last_became_peered":"2024-09-19T07:19:32.273504+0000","last_unstale":"2024-09-19T07:49:38.144914+0000","last_undegraded":"2024-09-19T07:49:38.144914+0000","last_fullsized":"2024-09-19T07:49:38.144914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5078","last_scrub_stamp":"2024-09-19T07:49:38.144648+0000","last_deep_scrub":"61'5078","last_deep_scrub_stamp":"2024-09-19T07:49:38.144648+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.144648+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:50:36.299270+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28059,"num_write_kb":313378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"64'3824","reported_seq":3939,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:46.383998+0000","last_change":"2024-09-19T07:49:46.384123+0000","last_active":"2024-09-19T07:49:46.383998+0000","last_peered":"2024-09-19T07:49:46.383998+0000","last_clean":"2024-09-19T07:49:46.383998+0000","last_became_active":"2024-09-19T07:19:40.385665+0000","last_became_peered":"2024-09-19T07:19:40.385665+0000","last_unstale":"2024-09-19T07:49:46.383998+0000","last_undegraded":"2024-09-19T07:49:46.383998+0000","last_fullsized":"2024-09-19T07:49:46.383998+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3824","last_scrub_stamp":"2024-09-19T07:49:46.383694+0000","last_deep_scrub":"64'3824","last_deep_scrub_stamp":"2024-09-19T07:49:46.383694+0000","last_clean_scrub_stamp":"2024-09-19T07:49:46.383694+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:08:54.643159+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20622,"num_write_kb":295649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"61'4725","reported_seq":4848,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.525791+0000","last_change":"2024-09-19T07:49:33.525791+0000","last_active":"2024-09-19T07:49:33.525791+0000","last_peered":"2024-09-19T07:49:33.525791+0000","last_clean":"2024-09-19T07:49:33.525791+0000","last_became_active":"2024-09-19T07:19:32.302633+0000","last_became_peered":"2024-09-19T07:19:32.302633+0000","last_unstale":"2024-09-19T07:49:33.525791+0000","last_undegraded":"2024-09-19T07:49:33.525791+0000","last_fullsized":"2024-09-19T07:49:33.525791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4725","last_scrub_stamp":"2024-09-19T07:49:33.525504+0000","last_deep_scrub":"61'4725","last_deep_scrub_stamp":"2024-09-19T07:49:33.525504+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.525504+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:06:32.819964+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25170,"num_write_kb":293477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"64'4764","reported_seq":4880,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:37.523504+0000","last_change":"2024-09-19T07:49:37.523566+0000","last_active":"2024-09-19T07:49:37.523504+0000","last_peered":"2024-09-19T07:49:37.523504+0000","last_clean":"2024-09-19T07:49:37.523504+0000","last_became_active":"2024-09-19T07:19:40.356649+0000","last_became_peered":"2024-09-19T07:19:40.356649+0000","last_unstale":"2024-09-19T07:49:37.523504+0000","last_undegraded":"2024-09-19T07:49:37.523504+0000","last_fullsized":"2024-09-19T07:49:37.523504+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4764","last_scrub_stamp":"2024-09-19T07:49:37.523305+0000","last_deep_scrub":"64'4764","last_deep_scrub_stamp":"2024-09-19T07:49:37.523305+0000","last_clean_scrub_stamp":"2024-09-19T07:49:37.523305+0000","objects_scrubbed":0,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:43:49.209515+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25208,"num_write_kb":309995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"61'5411","reported_seq":5535,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.498537+0000","last_change":"2024-09-19T07:49:31.498537+0000","last_active":"2024-09-19T07:49:31.498537+0000","last_peered":"2024-09-19T07:49:31.498537+0000","last_clean":"2024-09-19T07:49:31.498537+0000","last_became_active":"2024-09-19T07:19:32.288599+0000","last_became_peered":"2024-09-19T07:19:32.288599+0000","last_unstale":"2024-09-19T07:49:31.498537+0000","last_undegraded":"2024-09-19T07:49:31.498537+0000","last_fullsized":"2024-09-19T07:49:31.498537+0000","mapping_epoch":19,"log_start":"54'200","ondisk_log_start":"54'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5411","last_scrub_stamp":"2024-09-19T07:49:31.498168+0000","last_deep_scrub":"61'5411","last_deep_scrub_stamp":"2024-09-19T07:49:31.498168+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.498168+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:09:26.921508+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28760,"num_write_kb":348949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"64'4221","reported_seq":4336,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:50.561265+0000","last_change":"2024-09-19T07:49:50.561265+0000","last_active":"2024-09-19T07:49:50.561265+0000","last_peered":"2024-09-19T07:49:50.561265+0000","last_clean":"2024-09-19T07:49:50.561265+0000","last_became_active":"2024-09-19T07:19:40.387329+0000","last_became_peered":"2024-09-19T07:19:40.387329+0000","last_unstale":"2024-09-19T07:49:50.561265+0000","last_undegraded":"2024-09-19T07:49:50.561265+0000","last_fullsized":"2024-09-19T07:49:50.561265+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4221","last_scrub_stamp":"2024-09-19T07:49:50.560852+0000","last_deep_scrub":"64'4221","last_deep_scrub_stamp":"2024-09-19T07:49:50.560852+0000","last_clean_scrub_stamp":"2024-09-19T07:49:50.560852+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:09:15.497815+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21805,"num_write_kb":288249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"61'4974","reported_seq":5102,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:49.521620+0000","last_change":"2024-09-19T07:49:49.521683+0000","last_active":"2024-09-19T07:49:49.521620+0000","last_peered":"2024-09-19T07:49:49.521620+0000","last_clean":"2024-09-19T07:49:49.521620+0000","last_became_active":"2024-09-19T07:20:13.953118+0000","last_became_peered":"2024-09-19T07:20:13.953118+0000","last_unstale":"2024-09-19T07:49:49.521620+0000","last_undegraded":"2024-09-19T07:49:49.521620+0000","last_fullsized":"2024-09-19T07:49:49.521620+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4974","last_scrub_stamp":"2024-09-19T07:49:49.521395+0000","last_deep_scrub":"61'4974","last_deep_scrub_stamp":"2024-09-19T07:49:49.521395+0000","last_clean_scrub_stamp":"2024-09-19T07:49:49.521395+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:59:58.892017+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25995,"num_write_kb":309048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"64'5715","reported_seq":5829,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:44.351231+0000","last_change":"2024-09-19T07:49:44.351231+0000","last_active":"2024-09-19T07:49:44.351231+0000","last_peered":"2024-09-19T07:49:44.351231+0000","last_clean":"2024-09-19T07:49:44.351231+0000","last_became_active":"2024-09-19T07:19:40.381283+0000","last_became_peered":"2024-09-19T07:19:40.381283+0000","last_unstale":"2024-09-19T07:49:44.351231+0000","last_undegraded":"2024-09-19T07:49:44.351231+0000","last_fullsized":"2024-09-19T07:49:44.351231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5715","last_scrub_stamp":"2024-09-19T07:49:44.350959+0000","last_deep_scrub":"64'5715","last_deep_scrub_stamp":"2024-09-19T07:49:44.350959+0000","last_clean_scrub_stamp":"2024-09-19T07:49:44.350959+0000","objects_scrubbed":0,"log_size":5715,"log_dups_size":0,"ondisk_log_size":5715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:09.619342+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30021,"num_write_kb":374301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"61'4405","reported_seq":4527,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.494869+0000","last_change":"2024-09-19T07:49:27.495026+0000","last_active":"2024-09-19T07:49:27.494869+0000","last_peered":"2024-09-19T07:49:27.494869+0000","last_clean":"2024-09-19T07:49:27.494869+0000","last_became_active":"2024-09-19T07:19:32.312361+0000","last_became_peered":"2024-09-19T07:19:32.312361+0000","last_unstale":"2024-09-19T07:49:27.494869+0000","last_undegraded":"2024-09-19T07:49:27.494869+0000","last_fullsized":"2024-09-19T07:49:27.494869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4405","last_scrub_stamp":"2024-09-19T07:49:27.494549+0000","last_deep_scrub":"61'4405","last_deep_scrub_stamp":"2024-09-19T07:49:27.494549+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.494549+0000","objects_scrubbed":0,"log_size":4405,"log_dups_size":0,"ondisk_log_size":4405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:43.771195+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23742,"num_write_kb":275461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"64'3931","reported_seq":4045,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.673565+0000","last_change":"2024-09-19T07:49:29.673627+0000","last_active":"2024-09-19T07:49:29.673565+0000","last_peered":"2024-09-19T07:49:29.673565+0000","last_clean":"2024-09-19T07:49:29.673565+0000","last_became_active":"2024-09-19T07:19:40.375526+0000","last_became_peered":"2024-09-19T07:19:40.375526+0000","last_unstale":"2024-09-19T07:49:29.673565+0000","last_undegraded":"2024-09-19T07:49:29.673565+0000","last_fullsized":"2024-09-19T07:49:29.673565+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3931","last_scrub_stamp":"2024-09-19T07:49:29.673363+0000","last_deep_scrub":"64'3931","last_deep_scrub_stamp":"2024-09-19T07:49:29.673363+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.673363+0000","objects_scrubbed":0,"log_size":3931,"log_dups_size":0,"ondisk_log_size":3931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:24:17.132608+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22002,"num_write_kb":260466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"61'4975","reported_seq":5097,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.451782+0000","last_change":"2024-09-19T07:49:31.451897+0000","last_active":"2024-09-19T07:49:31.451782+0000","last_peered":"2024-09-19T07:49:31.451782+0000","last_clean":"2024-09-19T07:49:31.451782+0000","last_became_active":"2024-09-19T07:19:32.276840+0000","last_became_peered":"2024-09-19T07:19:32.276840+0000","last_unstale":"2024-09-19T07:49:31.451782+0000","last_undegraded":"2024-09-19T07:49:31.451782+0000","last_fullsized":"2024-09-19T07:49:31.451782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4975","last_scrub_stamp":"2024-09-19T07:49:31.451571+0000","last_deep_scrub":"61'4975","last_deep_scrub_stamp":"2024-09-19T07:49:31.451571+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.451571+0000","objects_scrubbed":0,"log_size":4975,"log_dups_size":0,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:16:31.212971+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26500,"num_write_kb":311194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"64'5003","reported_seq":5117,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.543370+0000","last_change":"2024-09-19T07:49:34.543501+0000","last_active":"2024-09-19T07:49:34.543370+0000","last_peered":"2024-09-19T07:49:34.543370+0000","last_clean":"2024-09-19T07:49:34.543370+0000","last_became_active":"2024-09-19T07:19:40.360739+0000","last_became_peered":"2024-09-19T07:19:40.360739+0000","last_unstale":"2024-09-19T07:49:34.543370+0000","last_undegraded":"2024-09-19T07:49:34.543370+0000","last_fullsized":"2024-09-19T07:49:34.543370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5003","last_scrub_stamp":"2024-09-19T07:49:34.543108+0000","last_deep_scrub":"64'5003","last_deep_scrub_stamp":"2024-09-19T07:49:34.543108+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.543108+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:19:11.973004+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26890,"num_write_kb":325027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"61'5877","reported_seq":6001,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:22.556263+0000","last_change":"2024-09-19T07:49:22.556294+0000","last_active":"2024-09-19T07:49:22.556263+0000","last_peered":"2024-09-19T07:49:22.556263+0000","last_clean":"2024-09-19T07:49:22.556263+0000","last_became_active":"2024-09-19T07:19:32.285112+0000","last_became_peered":"2024-09-19T07:19:32.285112+0000","last_unstale":"2024-09-19T07:49:22.556263+0000","last_undegraded":"2024-09-19T07:49:22.556263+0000","last_fullsized":"2024-09-19T07:49:22.556263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5877","last_scrub_stamp":"2024-09-19T07:49:22.556138+0000","last_deep_scrub":"61'5877","last_deep_scrub_stamp":"2024-09-19T07:49:22.556138+0000","last_clean_scrub_stamp":"2024-09-19T07:49:22.556138+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:02:01.734204+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":30970,"num_write_kb":370397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"64'4684","reported_seq":4798,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:41.466178+0000","last_change":"2024-09-19T07:49:41.466262+0000","last_active":"2024-09-19T07:49:41.466178+0000","last_peered":"2024-09-19T07:49:41.466178+0000","last_clean":"2024-09-19T07:49:41.466178+0000","last_became_active":"2024-09-19T07:19:40.389669+0000","last_became_peered":"2024-09-19T07:19:40.389669+0000","last_unstale":"2024-09-19T07:49:41.466178+0000","last_undegraded":"2024-09-19T07:49:41.466178+0000","last_fullsized":"2024-09-19T07:49:41.466178+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4684","last_scrub_stamp":"2024-09-19T07:49:41.465979+0000","last_deep_scrub":"64'4684","last_deep_scrub_stamp":"2024-09-19T07:49:41.465979+0000","last_clean_scrub_stamp":"2024-09-19T07:49:41.465979+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:41:00.899040+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25566,"num_write_kb":331994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"61'4261","reported_seq":4383,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:23.429556+0000","last_change":"2024-09-19T07:49:23.429674+0000","last_active":"2024-09-19T07:49:23.429556+0000","last_peered":"2024-09-19T07:49:23.429556+0000","last_clean":"2024-09-19T07:49:23.429556+0000","last_became_active":"2024-09-19T07:19:32.316040+0000","last_became_peered":"2024-09-19T07:19:32.316040+0000","last_unstale":"2024-09-19T07:49:23.429556+0000","last_undegraded":"2024-09-19T07:49:23.429556+0000","last_fullsized":"2024-09-19T07:49:23.429556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4261","last_scrub_stamp":"2024-09-19T07:49:23.429282+0000","last_deep_scrub":"61'4261","last_deep_scrub_stamp":"2024-09-19T07:49:23.429282+0000","last_clean_scrub_stamp":"2024-09-19T07:49:23.429282+0000","objects_scrubbed":0,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:37:47.705458+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22573,"num_write_kb":274878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"64'5240","reported_seq":5356,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:42.421607+0000","last_change":"2024-09-19T07:49:42.421672+0000","last_active":"2024-09-19T07:49:42.421607+0000","last_peered":"2024-09-19T07:49:42.421607+0000","last_clean":"2024-09-19T07:49:42.421607+0000","last_became_active":"2024-09-19T07:19:40.389758+0000","last_became_peered":"2024-09-19T07:19:40.389758+0000","last_unstale":"2024-09-19T07:49:42.421607+0000","last_undegraded":"2024-09-19T07:49:42.421607+0000","last_fullsized":"2024-09-19T07:49:42.421607+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5240","last_scrub_stamp":"2024-09-19T07:49:42.421410+0000","last_deep_scrub":"64'5240","last_deep_scrub_stamp":"2024-09-19T07:49:42.421410+0000","last_clean_scrub_stamp":"2024-09-19T07:49:42.421410+0000","objects_scrubbed":0,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:18:50.676511+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":28154,"num_write_kb":354961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"61'5095","reported_seq":5217,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.330890+0000","last_change":"2024-09-19T07:49:34.330949+0000","last_active":"2024-09-19T07:49:34.330890+0000","last_peered":"2024-09-19T07:49:34.330890+0000","last_clean":"2024-09-19T07:49:34.330890+0000","last_became_active":"2024-09-19T07:19:32.274312+0000","last_became_peered":"2024-09-19T07:19:32.274312+0000","last_unstale":"2024-09-19T07:49:34.330890+0000","last_undegraded":"2024-09-19T07:49:34.330890+0000","last_fullsized":"2024-09-19T07:49:34.330890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5095","last_scrub_stamp":"2024-09-19T07:49:34.330690+0000","last_deep_scrub":"61'5095","last_deep_scrub_stamp":"2024-09-19T07:49:34.330690+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.330690+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:09:29.754030+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26748,"num_write_kb":324403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"64'4078","reported_seq":4193,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:28.560670+0000","last_change":"2024-09-19T07:49:28.560736+0000","last_active":"2024-09-19T07:49:28.560670+0000","last_peered":"2024-09-19T07:49:28.560670+0000","last_clean":"2024-09-19T07:49:28.560670+0000","last_became_active":"2024-09-19T07:19:40.357688+0000","last_became_peered":"2024-09-19T07:19:40.357688+0000","last_unstale":"2024-09-19T07:49:28.560670+0000","last_undegraded":"2024-09-19T07:49:28.560670+0000","last_fullsized":"2024-09-19T07:49:28.560670+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4078","last_scrub_stamp":"2024-09-19T07:49:28.560467+0000","last_deep_scrub":"64'4078","last_deep_scrub_stamp":"2024-09-19T07:49:28.560467+0000","last_clean_scrub_stamp":"2024-09-19T07:49:28.560467+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:39:49.769342+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22110,"num_write_kb":293452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"61'4908","reported_seq":5031,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.678335+0000","last_change":"2024-09-19T07:49:26.678335+0000","last_active":"2024-09-19T07:49:26.678335+0000","last_peered":"2024-09-19T07:49:26.678335+0000","last_clean":"2024-09-19T07:49:26.678335+0000","last_became_active":"2024-09-19T07:19:32.301966+0000","last_became_peered":"2024-09-19T07:19:32.301966+0000","last_unstale":"2024-09-19T07:49:26.678335+0000","last_undegraded":"2024-09-19T07:49:26.678335+0000","last_fullsized":"2024-09-19T07:49:26.678335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4908","last_scrub_stamp":"2024-09-19T07:49:26.678074+0000","last_deep_scrub":"61'4908","last_deep_scrub_stamp":"2024-09-19T07:49:26.678074+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.678074+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:51.615841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25574,"num_write_kb":315713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"64'3983","reported_seq":4097,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:39.498845+0000","last_change":"2024-09-19T07:49:39.498907+0000","last_active":"2024-09-19T07:49:39.498845+0000","last_peered":"2024-09-19T07:49:39.498845+0000","last_clean":"2024-09-19T07:49:39.498845+0000","last_became_active":"2024-09-19T07:19:40.373581+0000","last_became_peered":"2024-09-19T07:19:40.373581+0000","last_unstale":"2024-09-19T07:49:39.498845+0000","last_undegraded":"2024-09-19T07:49:39.498845+0000","last_fullsized":"2024-09-19T07:49:39.498845+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3983","last_scrub_stamp":"2024-09-19T07:49:39.498538+0000","last_deep_scrub":"64'3983","last_deep_scrub_stamp":"2024-09-19T07:49:39.498538+0000","last_clean_scrub_stamp":"2024-09-19T07:49:39.498538+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:29:38.865483+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21503,"num_write_kb":274721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"61'5975","reported_seq":6097,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.515423+0000","last_change":"2024-09-19T07:49:29.515423+0000","last_active":"2024-09-19T07:49:29.515423+0000","last_peered":"2024-09-19T07:49:29.515423+0000","last_clean":"2024-09-19T07:49:29.515423+0000","last_became_active":"2024-09-19T07:19:32.278152+0000","last_became_peered":"2024-09-19T07:19:32.278152+0000","last_unstale":"2024-09-19T07:49:29.515423+0000","last_undegraded":"2024-09-19T07:49:29.515423+0000","last_fullsized":"2024-09-19T07:49:29.515423+0000","mapping_epoch":19,"log_start":"53'200","ondisk_log_start":"53'200","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5975","last_scrub_stamp":"2024-09-19T07:49:29.515156+0000","last_deep_scrub":"61'5975","last_deep_scrub_stamp":"2024-09-19T07:49:29.515156+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.515156+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:24:00.638951+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31472,"num_write_kb":384213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"64'4930","reported_seq":5043,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:43.150289+0000","last_change":"2024-09-19T07:49:43.150349+0000","last_active":"2024-09-19T07:49:43.150289+0000","last_peered":"2024-09-19T07:49:43.150289+0000","last_clean":"2024-09-19T07:49:43.150289+0000","last_became_active":"2024-09-19T07:19:40.386281+0000","last_became_peered":"2024-09-19T07:19:40.386281+0000","last_unstale":"2024-09-19T07:49:43.150289+0000","last_undegraded":"2024-09-19T07:49:43.150289+0000","last_fullsized":"2024-09-19T07:49:43.150289+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4930","last_scrub_stamp":"2024-09-19T07:49:43.150070+0000","last_deep_scrub":"64'4930","last_deep_scrub_stamp":"2024-09-19T07:49:43.150070+0000","last_clean_scrub_stamp":"2024-09-19T07:49:43.150070+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:38:57.592863+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26807,"num_write_kb":336825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"61'5543","reported_seq":5665,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:33.436157+0000","last_change":"2024-09-19T07:49:33.436157+0000","last_active":"2024-09-19T07:49:33.436157+0000","last_peered":"2024-09-19T07:49:33.436157+0000","last_clean":"2024-09-19T07:49:33.436157+0000","last_became_active":"2024-09-19T07:19:32.272767+0000","last_became_peered":"2024-09-19T07:19:32.272767+0000","last_unstale":"2024-09-19T07:49:33.436157+0000","last_undegraded":"2024-09-19T07:49:33.436157+0000","last_fullsized":"2024-09-19T07:49:33.436157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5543","last_scrub_stamp":"2024-09-19T07:49:33.435855+0000","last_deep_scrub":"61'5543","last_deep_scrub_stamp":"2024-09-19T07:49:33.435855+0000","last_clean_scrub_stamp":"2024-09-19T07:49:33.435855+0000","objects_scrubbed":0,"log_size":5543,"log_dups_size":0,"ondisk_log_size":5543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:38:49.685098+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29751,"num_write_kb":336461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"64'5766","reported_seq":5881,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.456958+0000","last_change":"2024-09-19T07:49:40.457080+0000","last_active":"2024-09-19T07:49:40.456958+0000","last_peered":"2024-09-19T07:49:40.456958+0000","last_clean":"2024-09-19T07:49:40.456958+0000","last_became_active":"2024-09-19T07:19:40.384028+0000","last_became_peered":"2024-09-19T07:19:40.384028+0000","last_unstale":"2024-09-19T07:49:40.456958+0000","last_undegraded":"2024-09-19T07:49:40.456958+0000","last_fullsized":"2024-09-19T07:49:40.456958+0000","mapping_epoch":23,"log_start":"58'300","ondisk_log_start":"58'300","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5766","last_scrub_stamp":"2024-09-19T07:49:40.456692+0000","last_deep_scrub":"64'5766","last_deep_scrub_stamp":"2024-09-19T07:49:40.456692+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.456692+0000","objects_scrubbed":0,"log_size":5466,"log_dups_size":0,"ondisk_log_size":5466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:29:06.554576+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32274,"num_write_kb":395900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"61'4235","reported_seq":4357,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.219358+0000","last_change":"2024-09-19T07:49:24.219358+0000","last_active":"2024-09-19T07:49:24.219358+0000","last_peered":"2024-09-19T07:49:24.219358+0000","last_clean":"2024-09-19T07:49:24.219358+0000","last_became_active":"2024-09-19T07:19:32.290293+0000","last_became_peered":"2024-09-19T07:19:32.290293+0000","last_unstale":"2024-09-19T07:49:24.219358+0000","last_undegraded":"2024-09-19T07:49:24.219358+0000","last_fullsized":"2024-09-19T07:49:24.219358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4235","last_scrub_stamp":"2024-09-19T07:49:24.218980+0000","last_deep_scrub":"61'4235","last_deep_scrub_stamp":"2024-09-19T07:49:24.218980+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.218980+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:38:08.256928+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22758,"num_write_kb":275574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"64'3417","reported_seq":3530,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:42.560470+0000","last_change":"2024-09-19T07:49:42.560535+0000","last_active":"2024-09-19T07:49:42.560470+0000","last_peered":"2024-09-19T07:49:42.560470+0000","last_clean":"2024-09-19T07:49:42.560470+0000","last_became_active":"2024-09-19T07:19:40.366869+0000","last_became_peered":"2024-09-19T07:19:40.366869+0000","last_unstale":"2024-09-19T07:49:42.560470+0000","last_undegraded":"2024-09-19T07:49:42.560470+0000","last_fullsized":"2024-09-19T07:49:42.560470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3417","last_scrub_stamp":"2024-09-19T07:49:42.560267+0000","last_deep_scrub":"64'3417","last_deep_scrub_stamp":"2024-09-19T07:49:42.560267+0000","last_clean_scrub_stamp":"2024-09-19T07:49:42.560267+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:58:57.391172+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18157,"num_write_kb":232641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"61'4616","reported_seq":4738,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.591446+0000","last_change":"2024-09-19T07:49:31.591446+0000","last_active":"2024-09-19T07:49:31.591446+0000","last_peered":"2024-09-19T07:49:31.591446+0000","last_clean":"2024-09-19T07:49:31.591446+0000","last_became_active":"2024-09-19T07:19:32.272954+0000","last_became_peered":"2024-09-19T07:19:32.272954+0000","last_unstale":"2024-09-19T07:49:31.591446+0000","last_undegraded":"2024-09-19T07:49:31.591446+0000","last_fullsized":"2024-09-19T07:49:31.591446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4616","last_scrub_stamp":"2024-09-19T07:49:31.591172+0000","last_deep_scrub":"61'4616","last_deep_scrub_stamp":"2024-09-19T07:49:31.591172+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.591172+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:50:16.285682+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24409,"num_write_kb":298784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"64'4320","reported_seq":4433,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:45.339530+0000","last_change":"2024-09-19T07:49:45.339590+0000","last_active":"2024-09-19T07:49:45.339530+0000","last_peered":"2024-09-19T07:49:45.339530+0000","last_clean":"2024-09-19T07:49:45.339530+0000","last_became_active":"2024-09-19T07:19:40.384788+0000","last_became_peered":"2024-09-19T07:19:40.384788+0000","last_unstale":"2024-09-19T07:49:45.339530+0000","last_undegraded":"2024-09-19T07:49:45.339530+0000","last_fullsized":"2024-09-19T07:49:45.339530+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4320","last_scrub_stamp":"2024-09-19T07:49:45.339300+0000","last_deep_scrub":"64'4320","last_deep_scrub_stamp":"2024-09-19T07:49:45.339300+0000","last_clean_scrub_stamp":"2024-09-19T07:49:45.339300+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:23:29.601806+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22755,"num_write_kb":287920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"61'4140","reported_seq":4262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:35.488288+0000","last_change":"2024-09-19T07:49:35.488347+0000","last_active":"2024-09-19T07:49:35.488288+0000","last_peered":"2024-09-19T07:49:35.488288+0000","last_clean":"2024-09-19T07:49:35.488288+0000","last_became_active":"2024-09-19T07:19:32.284556+0000","last_became_peered":"2024-09-19T07:19:32.284556+0000","last_unstale":"2024-09-19T07:49:35.488288+0000","last_undegraded":"2024-09-19T07:49:35.488288+0000","last_fullsized":"2024-09-19T07:49:35.488288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4140","last_scrub_stamp":"2024-09-19T07:49:35.488085+0000","last_deep_scrub":"61'4140","last_deep_scrub_stamp":"2024-09-19T07:49:35.488085+0000","last_clean_scrub_stamp":"2024-09-19T07:49:35.488085+0000","objects_scrubbed":0,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:57:54.215625+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22395,"num_write_kb":253944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"64'3972","reported_seq":4085,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:45.449006+0000","last_change":"2024-09-19T07:49:45.449070+0000","last_active":"2024-09-19T07:49:45.449006+0000","last_peered":"2024-09-19T07:49:45.449006+0000","last_clean":"2024-09-19T07:49:45.449006+0000","last_became_active":"2024-09-19T07:19:40.366388+0000","last_became_peered":"2024-09-19T07:19:40.366388+0000","last_unstale":"2024-09-19T07:49:45.449006+0000","last_undegraded":"2024-09-19T07:49:45.449006+0000","last_fullsized":"2024-09-19T07:49:45.449006+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3972","last_scrub_stamp":"2024-09-19T07:49:45.448791+0000","last_deep_scrub":"64'3972","last_deep_scrub_stamp":"2024-09-19T07:49:45.448791+0000","last_clean_scrub_stamp":"2024-09-19T07:49:45.448791+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:43:01.743144+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21876,"num_write_kb":260064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"61'3526","reported_seq":3647,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.350219+0000","last_change":"2024-09-19T07:49:34.350279+0000","last_active":"2024-09-19T07:49:34.350219+0000","last_peered":"2024-09-19T07:49:34.350219+0000","last_clean":"2024-09-19T07:49:34.350219+0000","last_became_active":"2024-09-19T07:19:32.307961+0000","last_became_peered":"2024-09-19T07:19:32.307961+0000","last_unstale":"2024-09-19T07:49:34.350219+0000","last_undegraded":"2024-09-19T07:49:34.350219+0000","last_fullsized":"2024-09-19T07:49:34.350219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3526","last_scrub_stamp":"2024-09-19T07:49:34.350012+0000","last_deep_scrub":"61'3526","last_deep_scrub_stamp":"2024-09-19T07:49:34.350012+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.350012+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:36:56.272224+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18762,"num_write_kb":235013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"64'3609","reported_seq":3722,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:34.622522+0000","last_change":"2024-09-19T07:49:34.622582+0000","last_active":"2024-09-19T07:49:34.622522+0000","last_peered":"2024-09-19T07:49:34.622522+0000","last_clean":"2024-09-19T07:49:34.622522+0000","last_became_active":"2024-09-19T07:19:40.382773+0000","last_became_peered":"2024-09-19T07:19:40.382773+0000","last_unstale":"2024-09-19T07:49:34.622522+0000","last_undegraded":"2024-09-19T07:49:34.622522+0000","last_fullsized":"2024-09-19T07:49:34.622522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3609","last_scrub_stamp":"2024-09-19T07:49:34.622304+0000","last_deep_scrub":"64'3609","last_deep_scrub_stamp":"2024-09-19T07:49:34.622304+0000","last_clean_scrub_stamp":"2024-09-19T07:49:34.622304+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:39:06.261533+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20332,"num_write_kb":258088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"61'4660","reported_seq":4786,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:42.487724+0000","last_change":"2024-09-19T07:49:42.487869+0000","last_active":"2024-09-19T07:49:42.487724+0000","last_peered":"2024-09-19T07:49:42.487724+0000","last_clean":"2024-09-19T07:49:42.487724+0000","last_became_active":"2024-09-19T07:20:13.952103+0000","last_became_peered":"2024-09-19T07:20:13.952103+0000","last_unstale":"2024-09-19T07:49:42.487724+0000","last_undegraded":"2024-09-19T07:49:42.487724+0000","last_fullsized":"2024-09-19T07:49:42.487724+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4660","last_scrub_stamp":"2024-09-19T07:49:42.487481+0000","last_deep_scrub":"61'4660","last_deep_scrub_stamp":"2024-09-19T07:49:42.487481+0000","last_clean_scrub_stamp":"2024-09-19T07:49:42.487481+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:35:06.484441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24375,"num_write_kb":290368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"64'3991","reported_seq":4105,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:47.400967+0000","last_change":"2024-09-19T07:49:47.400967+0000","last_active":"2024-09-19T07:49:47.400967+0000","last_peered":"2024-09-19T07:49:47.400967+0000","last_clean":"2024-09-19T07:49:47.400967+0000","last_became_active":"2024-09-19T07:19:40.386356+0000","last_became_peered":"2024-09-19T07:19:40.386356+0000","last_unstale":"2024-09-19T07:49:47.400967+0000","last_undegraded":"2024-09-19T07:49:47.400967+0000","last_fullsized":"2024-09-19T07:49:47.400967+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3991","last_scrub_stamp":"2024-09-19T07:49:47.400684+0000","last_deep_scrub":"64'3991","last_deep_scrub_stamp":"2024-09-19T07:49:47.400684+0000","last_clean_scrub_stamp":"2024-09-19T07:49:47.400684+0000","objects_scrubbed":0,"log_size":3991,"log_dups_size":0,"ondisk_log_size":3991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:46:40.384852+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21657,"num_write_kb":287665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"61'5448","reported_seq":5569,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.499325+0000","last_change":"2024-09-19T07:49:32.499445+0000","last_active":"2024-09-19T07:49:32.499325+0000","last_peered":"2024-09-19T07:49:32.499325+0000","last_clean":"2024-09-19T07:49:32.499325+0000","last_became_active":"2024-09-19T07:19:32.299924+0000","last_became_peered":"2024-09-19T07:19:32.299924+0000","last_unstale":"2024-09-19T07:49:32.499325+0000","last_undegraded":"2024-09-19T07:49:32.499325+0000","last_fullsized":"2024-09-19T07:49:32.499325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5448","last_scrub_stamp":"2024-09-19T07:49:32.499060+0000","last_deep_scrub":"61'5448","last_deep_scrub_stamp":"2024-09-19T07:49:32.499060+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.499060+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:56:11.424517+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29399,"num_write_kb":347457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"64'4539","reported_seq":4652,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.500615+0000","last_change":"2024-09-19T07:49:30.500671+0000","last_active":"2024-09-19T07:49:30.500615+0000","last_peered":"2024-09-19T07:49:30.500615+0000","last_clean":"2024-09-19T07:49:30.500615+0000","last_became_active":"2024-09-19T07:19:40.360535+0000","last_became_peered":"2024-09-19T07:19:40.360535+0000","last_unstale":"2024-09-19T07:49:30.500615+0000","last_undegraded":"2024-09-19T07:49:30.500615+0000","last_fullsized":"2024-09-19T07:49:30.500615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4539","last_scrub_stamp":"2024-09-19T07:49:30.500418+0000","last_deep_scrub":"64'4539","last_deep_scrub_stamp":"2024-09-19T07:49:30.500418+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.500418+0000","objects_scrubbed":0,"log_size":4539,"log_dups_size":0,"ondisk_log_size":4539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:42:26.609996+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24944,"num_write_kb":313178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"61'6280","reported_seq":6403,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.317119+0000","last_change":"2024-09-19T07:49:30.317185+0000","last_active":"2024-09-19T07:49:30.317119+0000","last_peered":"2024-09-19T07:49:30.317119+0000","last_clean":"2024-09-19T07:49:30.317119+0000","last_became_active":"2024-09-19T07:19:32.278803+0000","last_became_peered":"2024-09-19T07:19:32.278803+0000","last_unstale":"2024-09-19T07:49:30.317119+0000","last_undegraded":"2024-09-19T07:49:30.317119+0000","last_fullsized":"2024-09-19T07:49:30.317119+0000","mapping_epoch":19,"log_start":"54'500","ondisk_log_start":"54'500","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'6280","last_scrub_stamp":"2024-09-19T07:49:30.316901+0000","last_deep_scrub":"61'6280","last_deep_scrub_stamp":"2024-09-19T07:49:30.316901+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.316901+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:44:46.593547+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":34141,"num_write_kb":395176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"64'5507","reported_seq":5622,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:46.460898+0000","last_change":"2024-09-19T07:49:46.460921+0000","last_active":"2024-09-19T07:49:46.460898+0000","last_peered":"2024-09-19T07:49:46.460898+0000","last_clean":"2024-09-19T07:49:46.460898+0000","last_became_active":"2024-09-19T07:19:40.372273+0000","last_became_peered":"2024-09-19T07:19:40.372273+0000","last_unstale":"2024-09-19T07:49:46.460898+0000","last_undegraded":"2024-09-19T07:49:46.460898+0000","last_fullsized":"2024-09-19T07:49:46.460898+0000","mapping_epoch":23,"log_start":"58'300","ondisk_log_start":"58'300","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'5507","last_scrub_stamp":"2024-09-19T07:49:46.460808+0000","last_deep_scrub":"64'5507","last_deep_scrub_stamp":"2024-09-19T07:49:46.460808+0000","last_clean_scrub_stamp":"2024-09-19T07:49:46.460808+0000","objects_scrubbed":0,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:26:05.542860+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30086,"num_write_kb":370527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"61'4537","reported_seq":4658,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:40.480933+0000","last_change":"2024-09-19T07:49:40.480998+0000","last_active":"2024-09-19T07:49:40.480933+0000","last_peered":"2024-09-19T07:49:40.480933+0000","last_clean":"2024-09-19T07:49:40.480933+0000","last_became_active":"2024-09-19T07:19:32.271531+0000","last_became_peered":"2024-09-19T07:19:32.271531+0000","last_unstale":"2024-09-19T07:49:40.480933+0000","last_undegraded":"2024-09-19T07:49:40.480933+0000","last_fullsized":"2024-09-19T07:49:40.480933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4537","last_scrub_stamp":"2024-09-19T07:49:40.480727+0000","last_deep_scrub":"61'4537","last_deep_scrub_stamp":"2024-09-19T07:49:40.480727+0000","last_clean_scrub_stamp":"2024-09-19T07:49:40.480727+0000","objects_scrubbed":0,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:28:22.886355+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24234,"num_write_kb":310307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"64'4532","reported_seq":4648,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:38.500610+0000","last_change":"2024-09-19T07:49:38.500678+0000","last_active":"2024-09-19T07:49:38.500610+0000","last_peered":"2024-09-19T07:49:38.500610+0000","last_clean":"2024-09-19T07:49:38.500610+0000","last_became_active":"2024-09-19T07:19:40.370024+0000","last_became_peered":"2024-09-19T07:19:40.370024+0000","last_unstale":"2024-09-19T07:49:38.500610+0000","last_undegraded":"2024-09-19T07:49:38.500610+0000","last_fullsized":"2024-09-19T07:49:38.500610+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4532","last_scrub_stamp":"2024-09-19T07:49:38.500398+0000","last_deep_scrub":"64'4532","last_deep_scrub_stamp":"2024-09-19T07:49:38.500398+0000","last_clean_scrub_stamp":"2024-09-19T07:49:38.500398+0000","objects_scrubbed":0,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:37:00.422610+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25325,"num_write_kb":306661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"61'4832","reported_seq":4953,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:24.209895+0000","last_change":"2024-09-19T07:49:24.209895+0000","last_active":"2024-09-19T07:49:24.209895+0000","last_peered":"2024-09-19T07:49:24.209895+0000","last_clean":"2024-09-19T07:49:24.209895+0000","last_became_active":"2024-09-19T07:19:32.288418+0000","last_became_peered":"2024-09-19T07:19:32.288418+0000","last_unstale":"2024-09-19T07:49:24.209895+0000","last_undegraded":"2024-09-19T07:49:24.209895+0000","last_fullsized":"2024-09-19T07:49:24.209895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4832","last_scrub_stamp":"2024-09-19T07:49:24.209617+0000","last_deep_scrub":"61'4832","last_deep_scrub_stamp":"2024-09-19T07:49:24.209617+0000","last_clean_scrub_stamp":"2024-09-19T07:49:24.209617+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:56:03.135074+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26222,"num_write_kb":306099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"64'3448","reported_seq":3565,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:44.172927+0000","last_change":"2024-09-19T07:49:44.172927+0000","last_active":"2024-09-19T07:49:44.172927+0000","last_peered":"2024-09-19T07:49:44.172927+0000","last_clean":"2024-09-19T07:49:44.172927+0000","last_became_active":"2024-09-19T07:19:40.395739+0000","last_became_peered":"2024-09-19T07:19:40.395739+0000","last_unstale":"2024-09-19T07:49:44.172927+0000","last_undegraded":"2024-09-19T07:49:44.172927+0000","last_fullsized":"2024-09-19T07:49:44.172927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3448","last_scrub_stamp":"2024-09-19T07:49:44.172643+0000","last_deep_scrub":"64'3448","last_deep_scrub_stamp":"2024-09-19T07:49:44.172643+0000","last_clean_scrub_stamp":"2024-09-19T07:49:44.172643+0000","objects_scrubbed":0,"log_size":3448,"log_dups_size":0,"ondisk_log_size":3448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:33:10.301625+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18660,"num_write_kb":230569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"61'4317","reported_seq":4438,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:25.544477+0000","last_change":"2024-09-19T07:49:25.544477+0000","last_active":"2024-09-19T07:49:25.544477+0000","last_peered":"2024-09-19T07:49:25.544477+0000","last_clean":"2024-09-19T07:49:25.544477+0000","last_became_active":"2024-09-19T07:19:32.284730+0000","last_became_peered":"2024-09-19T07:19:32.284730+0000","last_unstale":"2024-09-19T07:49:25.544477+0000","last_undegraded":"2024-09-19T07:49:25.544477+0000","last_fullsized":"2024-09-19T07:49:25.544477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4317","last_scrub_stamp":"2024-09-19T07:49:25.543727+0000","last_deep_scrub":"61'4317","last_deep_scrub_stamp":"2024-09-19T07:49:25.543727+0000","last_clean_scrub_stamp":"2024-09-19T07:49:25.543727+0000","objects_scrubbed":0,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:50:47.452710+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22609,"num_write_kb":267014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"64'3912","reported_seq":4028,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:44.536935+0000","last_change":"2024-09-19T07:49:44.536995+0000","last_active":"2024-09-19T07:49:44.536935+0000","last_peered":"2024-09-19T07:49:44.536935+0000","last_clean":"2024-09-19T07:49:44.536935+0000","last_became_active":"2024-09-19T07:19:40.372922+0000","last_became_peered":"2024-09-19T07:19:40.372922+0000","last_unstale":"2024-09-19T07:49:44.536935+0000","last_undegraded":"2024-09-19T07:49:44.536935+0000","last_fullsized":"2024-09-19T07:49:44.536935+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3912","last_scrub_stamp":"2024-09-19T07:49:44.536737+0000","last_deep_scrub":"64'3912","last_deep_scrub_stamp":"2024-09-19T07:49:44.536737+0000","last_clean_scrub_stamp":"2024-09-19T07:49:44.536737+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:21:43.065566+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21554,"num_write_kb":270790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"61'4720","reported_seq":4841,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.346538+0000","last_change":"2024-09-19T07:49:32.346538+0000","last_active":"2024-09-19T07:49:32.346538+0000","last_peered":"2024-09-19T07:49:32.346538+0000","last_clean":"2024-09-19T07:49:32.346538+0000","last_became_active":"2024-09-19T07:19:32.280082+0000","last_became_peered":"2024-09-19T07:19:32.280082+0000","last_unstale":"2024-09-19T07:49:32.346538+0000","last_undegraded":"2024-09-19T07:49:32.346538+0000","last_fullsized":"2024-09-19T07:49:32.346538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4720","last_scrub_stamp":"2024-09-19T07:49:32.346047+0000","last_deep_scrub":"61'4720","last_deep_scrub_stamp":"2024-09-19T07:49:32.346047+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.346047+0000","objects_scrubbed":0,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:56:25.061148+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24228,"num_write_kb":289919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"64'3227","reported_seq":3340,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:32.478502+0000","last_change":"2024-09-19T07:49:32.478502+0000","last_active":"2024-09-19T07:49:32.478502+0000","last_peered":"2024-09-19T07:49:32.478502+0000","last_clean":"2024-09-19T07:49:32.478502+0000","last_became_active":"2024-09-19T07:19:40.356273+0000","last_became_peered":"2024-09-19T07:19:40.356273+0000","last_unstale":"2024-09-19T07:49:32.478502+0000","last_undegraded":"2024-09-19T07:49:32.478502+0000","last_fullsized":"2024-09-19T07:49:32.478502+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3227","last_scrub_stamp":"2024-09-19T07:49:32.478236+0000","last_deep_scrub":"64'3227","last_deep_scrub_stamp":"2024-09-19T07:49:32.478236+0000","last_clean_scrub_stamp":"2024-09-19T07:49:32.478236+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:00:58.993608+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17253,"num_write_kb":237102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"61'4326","reported_seq":4451,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:27.550528+0000","last_change":"2024-09-19T07:49:27.550528+0000","last_active":"2024-09-19T07:49:27.550528+0000","last_peered":"2024-09-19T07:49:27.550528+0000","last_clean":"2024-09-19T07:49:27.550528+0000","last_became_active":"2024-09-19T07:19:32.284638+0000","last_became_peered":"2024-09-19T07:19:32.284638+0000","last_unstale":"2024-09-19T07:49:27.550528+0000","last_undegraded":"2024-09-19T07:49:27.550528+0000","last_fullsized":"2024-09-19T07:49:27.550528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4326","last_scrub_stamp":"2024-09-19T07:49:27.550257+0000","last_deep_scrub":"61'4326","last_deep_scrub_stamp":"2024-09-19T07:49:27.550257+0000","last_clean_scrub_stamp":"2024-09-19T07:49:27.550257+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:24:57.720889+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23399,"num_write_kb":274225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"64'3524","reported_seq":3637,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:30.675251+0000","last_change":"2024-09-19T07:49:30.675251+0000","last_active":"2024-09-19T07:49:30.675251+0000","last_peered":"2024-09-19T07:49:30.675251+0000","last_clean":"2024-09-19T07:49:30.675251+0000","last_became_active":"2024-09-19T07:19:40.375641+0000","last_became_peered":"2024-09-19T07:19:40.375641+0000","last_unstale":"2024-09-19T07:49:30.675251+0000","last_undegraded":"2024-09-19T07:49:30.675251+0000","last_fullsized":"2024-09-19T07:49:30.675251+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3524","last_scrub_stamp":"2024-09-19T07:49:30.674832+0000","last_deep_scrub":"64'3524","last_deep_scrub_stamp":"2024-09-19T07:49:30.674832+0000","last_clean_scrub_stamp":"2024-09-19T07:49:30.674832+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:08:06.894419+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18216,"num_write_kb":261619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"61'4656","reported_seq":4780,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.577483+0000","last_change":"2024-09-19T07:49:29.577545+0000","last_active":"2024-09-19T07:49:29.577483+0000","last_peered":"2024-09-19T07:49:29.577483+0000","last_clean":"2024-09-19T07:49:29.577483+0000","last_became_active":"2024-09-19T07:19:32.270395+0000","last_became_peered":"2024-09-19T07:19:32.270395+0000","last_unstale":"2024-09-19T07:49:29.577483+0000","last_undegraded":"2024-09-19T07:49:29.577483+0000","last_fullsized":"2024-09-19T07:49:29.577483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4656","last_scrub_stamp":"2024-09-19T07:49:29.577280+0000","last_deep_scrub":"61'4656","last_deep_scrub_stamp":"2024-09-19T07:49:29.577280+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.577280+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:43:47.082591+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24790,"num_write_kb":287946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"64'4503","reported_seq":4616,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:31.329668+0000","last_change":"2024-09-19T07:49:31.329668+0000","last_active":"2024-09-19T07:49:31.329668+0000","last_peered":"2024-09-19T07:49:31.329668+0000","last_clean":"2024-09-19T07:49:31.329668+0000","last_became_active":"2024-09-19T07:19:40.376606+0000","last_became_peered":"2024-09-19T07:19:40.376606+0000","last_unstale":"2024-09-19T07:49:31.329668+0000","last_undegraded":"2024-09-19T07:49:31.329668+0000","last_fullsized":"2024-09-19T07:49:31.329668+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4503","last_scrub_stamp":"2024-09-19T07:49:31.329217+0000","last_deep_scrub":"64'4503","last_deep_scrub_stamp":"2024-09-19T07:49:31.329217+0000","last_clean_scrub_stamp":"2024-09-19T07:49:31.329217+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:36:23.637457+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24213,"num_write_kb":310059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"61'5949","reported_seq":6072,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.286974+0000","last_change":"2024-09-19T07:49:26.287050+0000","last_active":"2024-09-19T07:49:26.286974+0000","last_peered":"2024-09-19T07:49:26.286974+0000","last_clean":"2024-09-19T07:49:26.286974+0000","last_became_active":"2024-09-19T07:19:32.285875+0000","last_became_peered":"2024-09-19T07:19:32.285875+0000","last_unstale":"2024-09-19T07:49:26.286974+0000","last_undegraded":"2024-09-19T07:49:26.286974+0000","last_fullsized":"2024-09-19T07:49:26.286974+0000","mapping_epoch":19,"log_start":"53'100","ondisk_log_start":"53'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'5949","last_scrub_stamp":"2024-09-19T07:49:26.286761+0000","last_deep_scrub":"61'5949","last_deep_scrub_stamp":"2024-09-19T07:49:26.286761+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.286761+0000","objects_scrubbed":0,"log_size":5849,"log_dups_size":0,"ondisk_log_size":5849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:01:19.621957+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31740,"num_write_kb":364971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"64'4360","reported_seq":4473,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:29.360009+0000","last_change":"2024-09-19T07:49:29.360009+0000","last_active":"2024-09-19T07:49:29.360009+0000","last_peered":"2024-09-19T07:49:29.360009+0000","last_clean":"2024-09-19T07:49:29.360009+0000","last_became_active":"2024-09-19T07:19:40.376825+0000","last_became_peered":"2024-09-19T07:19:40.376825+0000","last_unstale":"2024-09-19T07:49:29.360009+0000","last_undegraded":"2024-09-19T07:49:29.360009+0000","last_fullsized":"2024-09-19T07:49:29.360009+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4360","last_scrub_stamp":"2024-09-19T07:49:29.359658+0000","last_deep_scrub":"64'4360","last_deep_scrub_stamp":"2024-09-19T07:49:29.359658+0000","last_clean_scrub_stamp":"2024-09-19T07:49:29.359658+0000","objects_scrubbed":0,"log_size":4360,"log_dups_size":0,"ondisk_log_size":4360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:17.888583+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23533,"num_write_kb":311499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"61'4830","reported_seq":4954,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-19T07:49:26.228354+0000","last_change":"2024-09-19T07:49:26.228418+0000","last_active":"2024-09-19T07:49:26.228354+0000","last_peered":"2024-09-19T07:49:26.228354+0000","last_clean":"2024-09-19T07:49:26.228354+0000","last_became_active":"2024-09-19T07:19:32.285139+0000","last_became_peered":"2024-09-19T07:19:32.285139+0000","last_unstale":"2024-09-19T07:49:26.228354+0000","last_undegraded":"2024-09-19T07:49:26.228354+0000","last_fullsized":"2024-09-19T07:49:26.228354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'4830","last_scrub_stamp":"2024-09-19T07:49:26.228143+0000","last_deep_scrub":"61'4830","last_deep_scrub_stamp":"2024-09-19T07:49:26.228143+0000","last_clean_scrub_stamp":"2024-09-19T07:49:26.228143+0000","objects_scrubbed":0,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:24:35.194932+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26589,"num_write_kb":315697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1521393,"num_write_kb":19214666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279640,"ondisk_log_size":279640,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19627747,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1449,"num_read_kb":190539,"num_write":5111,"num_write_kb":21283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39313408,"data_stored":39255494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4630,"ondisk_log_size":4630,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":0,"num_write":1651322,"num_write_kb":19736802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307839,"ondisk_log_size":307839,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":19579163,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1454,"num_read_kb":221531,"num_write":5068,"num_write_kb":21268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39223296,"data_stored":39158326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4648,"ondisk_log_size":4648,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542516,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1114512,"kb_used_data":13696,"kb_used_omap":1,"kb_used_meta":1100798,"kb_avail":93257328,"statfs":{"total":96636764160,"available":95495503872,"internally_reserved":0,"allocated":14024704,"data_stored":13620504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1127217610},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542517,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":766708,"kb_used_data":11684,"kb_used_omap":1,"kb_used_meta":755006,"kb_avail":93605132,"statfs":{"total":96636764160,"available":95851655168,"internally_reserved":0,"allocated":11964416,"data_stored":11552279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":773126598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542515,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1069796,"kb_used_data":5012,"kb_used_omap":1,"kb_used_meta":1064766,"kb_avail":93302044,"statfs":{"total":96636764160,"available":95541293056,"internally_reserved":0,"allocated":5132288,"data_stored":4726264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1090320837},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542517,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1229016,"kb_used_data":14472,"kb_used_omap":1,"kb_used_meta":1214526,"kb_avail":93142824,"statfs":{"total":96636764160,"available":95378251776,"internally_reserved":0,"allocated":14819328,"data_stored":14400282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1243675077},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542515,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1080696,"kb_used_data":20008,"kb_used_omap":1,"kb_used_meta":1060670,"kb_avail":93291144,"statfs":{"total":96636764160,"available":95530131456,"internally_reserved":0,"allocated":20488192,"data_stored":20074195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1086126535},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542517,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688028,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":687102,"kb_avail":93683812,"statfs":{"total":96636764160,"available":95932223488,"internally_reserved":0,"allocated":929792,"data_stored":531268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":703592902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542516,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":891928,"kb_used_data":9032,"kb_used_omap":1,"kb_used_meta":882878,"kb_avail":93479912,"statfs":{"total":96636764160,"available":95723429888,"internally_reserved":0,"allocated":9248768,"data_stored":8836014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904067525},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542515,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":911636,"kb_used_data":9604,"kb_used_omap":1,"kb_used_meta":902014,"kb_avail":93460204,"statfs":{"total":96636764160,"available":95703248896,"internally_reserved":0,"allocated":9834496,"data_stored":9426166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923662789},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":4194942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":84646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8482816,"data_stored":8473926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6926336,"data_stored":6910261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6922240,"data_stored":6910807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11165696,"data_stored":11153717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7053312,"data_stored":7043469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4279622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T07:50:04.127 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-19T07:50:04.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdaea0ff640 1 Processor -- start 2024-09-19T07:50:04.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdaea0ff640 1 -- start start 2024-09-19T07:50:04.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdaea0ff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdae406dc40 0x7fdae406e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdaea0ff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fdae40c1f00 0x7fdae40c2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdaea0ff640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdae4073ee0 con 0x7fdae4073720 2024-09-19T07:50:04.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdaea0ff640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fdae4073c90 con 0x7fdae406dc40 2024-09-19T07:50:04.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdaea0ff640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fdae40c3700 con 0x7fdae40c1f00 2024-09-19T07:50:04.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdae37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdae406dc40 0x7fdae406e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdae37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdae406dc40 0x7fdae406e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47342/0 (socket says 172.21.15.83:47342) 2024-09-19T07:50:04.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdae37fe640 1 -- 172.21.15.83:0/1524031602 learned_addr learned my addr 172.21.15.83:0/1524031602 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:04.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdae3fff640 1 --2- 172.21.15.83:0/1524031602 >> v2:172.21.15.184:3300/0 conn(0x7fdae40c1f00 0x7fdae40c2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdae37fe640 1 -- 172.21.15.83:0/1524031602 >> v2:172.21.15.184:3300/0 conn(0x7fdae40c1f00 msgr2=0x7fdae40c2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:04.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdae37fe640 1 --2- 172.21.15.83:0/1524031602 >> v2:172.21.15.184:3300/0 conn(0x7fdae40c1f00 0x7fdae40c2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdae37fe640 1 -- 172.21.15.83:0/1524031602 >> v1:172.21.15.184:6789/0 conn(0x7fdae4073720 legacy=0x7fdae414f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:04.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdae37fe640 1 -- 172.21.15.83:0/1524031602 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae40c39e0 con 0x7fdae406dc40 2024-09-19T07:50:04.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.287+0000 7fdae3fff640 1 --2- 172.21.15.83:0/1524031602 >> v2:172.21.15.184:3300/0 conn(0x7fdae40c1f00 0x7fdae40c2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:04.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdae37fe640 1 --2- 172.21.15.83:0/1524031602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdae406dc40 0x7fdae406e020 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fdae4073bb0 tx=0x7fdad401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c074972d4b2793d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:04.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdae27fc640 1 -- 172.21.15.83:0/1524031602 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad40a64e0 con 0x7fdae406dc40 2024-09-19T07:50:04.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdae27fc640 1 -- 172.21.15.83:0/1524031602 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdad40a6680 con 0x7fdae406dc40 2024-09-19T07:50:04.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdae27fc640 1 -- 172.21.15.83:0/1524031602 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad40a6a50 con 0x7fdae406dc40 2024-09-19T07:50:04.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 -- 172.21.15.83:0/1524031602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdae406dc40 msgr2=0x7fdae406e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 --2- 172.21.15.83:0/1524031602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdae406dc40 0x7fdae406e020 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fdae4073bb0 tx=0x7fdad401c890 comp rx=0 tx=0).stop 2024-09-19T07:50:04.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 -- 172.21.15.83:0/1524031602 shutdown_connections 2024-09-19T07:50:04.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 --2- 172.21.15.83:0/1524031602 >> v2:172.21.15.184:3300/0 conn(0x7fdae40c1f00 0x7fdae40c2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 --2- 172.21.15.83:0/1524031602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdae406dc40 0x7fdae406e020 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 -- 172.21.15.83:0/1524031602 >> 172.21.15.83:0/1524031602 conn(0x7fdae406bc80 msgr2=0x7fdae406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:04.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 -- 172.21.15.83:0/1524031602 shutdown_connections 2024-09-19T07:50:04.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 -- 172.21.15.83:0/1524031602 wait complete. 2024-09-19T07:50:04.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 Processor -- start 2024-09-19T07:50:04.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 -- start start 2024-09-19T07:50:04.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdae406dc40 0x7fdae40c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdae4073720 0x7fdae41478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdae40c1f00 0x7fdae4150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdae415cc10 con 0x7fdae40c1f00 2024-09-19T07:50:04.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdae415ca90 con 0x7fdae4073720 2024-09-19T07:50:04.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdaea0ff640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdae415cd90 con 0x7fdae406dc40 2024-09-19T07:50:04.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdae3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdae40c1f00 0x7fdae4150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdae3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdae40c1f00 0x7fdae4150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47348/0 (socket says 172.21.15.83:47348) 2024-09-19T07:50:04.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.291+0000 7fdae3fff640 1 -- 172.21.15.83:0/2953617646 learned_addr learned my addr 172.21.15.83:0/2953617646 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:04.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdae37fe640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdae406dc40 0x7fdae40c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdae2ffd640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdae4073720 0x7fdae41478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdae3fff640 1 -- 172.21.15.83:0/2953617646 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdae406dc40 msgr2=0x7fdae40c4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdae3fff640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdae406dc40 0x7fdae40c4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdae3fff640 1 -- 172.21.15.83:0/2953617646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdae4073720 msgr2=0x7fdae41478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdae3fff640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdae4073720 0x7fdae41478f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdae3fff640 1 -- 172.21.15.83:0/2953617646 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae40c66b0 con 0x7fdae40c1f00 2024-09-19T07:50:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdae3fff640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdae40c1f00 0x7fdae4150060 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00c7f0 tx=0x7fdadc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdadbfff640 1 -- 172.21.15.83:0/2953617646 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc015020 con 0x7fdae40c1f00 2024-09-19T07:50:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdadbfff640 1 -- 172.21.15.83:0/2953617646 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdadc004250 con 0x7fdae40c1f00 2024-09-19T07:50:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdadbfff640 1 -- 172.21.15.83:0/2953617646 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc003620 con 0x7fdae40c1f00 2024-09-19T07:50:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdaea0ff640 1 -- 172.21.15.83:0/2953617646 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae4151f60 con 0x7fdae40c1f00 2024-09-19T07:50:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdaea0ff640 1 -- 172.21.15.83:0/2953617646 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae40c6fe0 con 0x7fdae40c1f00 2024-09-19T07:50:04.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdaea0ff640 1 -- 172.21.15.83:0/2953617646 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae40c71e0 con 0x7fdae40c1f00 2024-09-19T07:50:04.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdadbfff640 1 -- 172.21.15.83:0/2953617646 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdadc018020 con 0x7fdae40c1f00 2024-09-19T07:50:04.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdadbfff640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdac40678c0 0x7fdac4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdadbfff640 1 -- 172.21.15.83:0/2953617646 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10163+0+0 (secure 0 0 0) 0x7fdadc086d50 con 0x7fdae40c1f00 2024-09-19T07:50:04.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdae37fe640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdac40678c0 0x7fdac4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.295+0000 7fdae37fe640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdac40678c0 0x7fdac4069d80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdad40ae000 tx=0x7fdad40b1040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:04.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.299+0000 7fdadbfff640 1 -- 172.21.15.83:0/2953617646 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fdadc08a030 con 0x7fdae40c1f00 2024-09-19T07:50:04.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.471+0000 7fdaea0ff640 1 -- 172.21.15.83:0/2953617646 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fdae40c1bf0 con 0x7fdae40c1f00 2024-09-19T07:50:04.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.479+0000 7fdadbfff640 1 -- 172.21.15.83:0/2953617646 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7fdadc01f030 con 0x7fdae40c1f00 2024-09-19T07:50:04.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.479+0000 7fdadbfff640 1 -- 172.21.15.83:0/2953617646 <== mon.0 v2:172.21.15.83:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fdadc056760 con 0x7fdae40c1f00 2024-09-19T07:50:04.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.479+0000 7fdaea0ff640 1 -- 172.21.15.83:0/2953617646 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdac40678c0 msgr2=0x7fdac4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.479+0000 7fdaea0ff640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdac40678c0 0x7fdac4069d80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdad40ae000 tx=0x7fdad40b1040 comp rx=0 tx=0).stop 2024-09-19T07:50:04.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.479+0000 7fdaea0ff640 1 -- 172.21.15.83:0/2953617646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdae40c1f00 msgr2=0x7fdae4150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.479+0000 7fdaea0ff640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdae40c1f00 0x7fdae4150060 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00c7f0 tx=0x7fdadc00ccb0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.483+0000 7fdaea0ff640 1 -- 172.21.15.83:0/2953617646 shutdown_connections 2024-09-19T07:50:04.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.483+0000 7fdaea0ff640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fdac40678c0 0x7fdac4069d80 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.483+0000 7fdaea0ff640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdae40c1f00 0x7fdae4150060 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.483+0000 7fdaea0ff640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdae4073720 0x7fdae41478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.483+0000 7fdaea0ff640 1 --2- 172.21.15.83:0/2953617646 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdae406dc40 0x7fdae40c4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.483+0000 7fdaea0ff640 1 -- 172.21.15.83:0/2953617646 >> 172.21.15.83:0/2953617646 conn(0x7fdae406bc80 msgr2=0x7fdae4071f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:04.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.483+0000 7fdaea0ff640 1 -- 172.21.15.83:0/2953617646 shutdown_connections 2024-09-19T07:50:04.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.483+0000 7fdaea0ff640 1 -- 172.21.15.83:0/2953617646 wait complete. 2024-09-19T07:50:04.497 INFO:tasks.cephfs.filesystem:Destroying file system a and related pools 2024-09-19T07:50:04.498 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:50:04.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.615+0000 7f87d0d4b640 1 Processor -- start 2024-09-19T07:50:04.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.615+0000 7f87d0d4b640 1 -- start start 2024-09-19T07:50:04.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87d0d4b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87cc071100 0x7f87cc1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87d0d4b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87cc06dea0 0x7f87cc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87d0d4b640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87cc0a9e60 con 0x7f87cc0c28e0 2024-09-19T07:50:04.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87d0d4b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f87cc071b50 con 0x7f87cc06dea0 2024-09-19T07:50:04.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87d0d4b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f87cc0c1ff0 con 0x7f87cc071100 2024-09-19T07:50:04.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87cad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87cc06dea0 0x7f87cc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87cad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87cc06dea0 0x7f87cc06e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47358/0 (socket says 172.21.15.83:47358) 2024-09-19T07:50:04.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87cad9d640 1 -- 172.21.15.83:0/3375429129 learned_addr learned my addr 172.21.15.83:0/3375429129 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:04.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87ca59c640 1 --2- 172.21.15.83:0/3375429129 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87cc071100 0x7f87cc1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87cad9d640 1 -- 172.21.15.83:0/3375429129 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87cc071100 msgr2=0x7f87cc1545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:04.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87cad9d640 1 --2- 172.21.15.83:0/3375429129 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87cc071100 0x7f87cc1545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87cad9d640 1 -- 172.21.15.83:0/3375429129 >> v1:172.21.15.184:6789/0 conn(0x7f87cc0c28e0 legacy=0x7f87cc150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:04.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87cad9d640 1 -- 172.21.15.83:0/3375429129 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87cc0aa140 con 0x7f87cc06dea0 2024-09-19T07:50:04.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87cad9d640 1 --2- 172.21.15.83:0/3375429129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87cc06dea0 0x7f87cc06e350 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009e50 tx=0x7f87c401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158c7c729e4ae85a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:04.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87c959a640 1 -- 172.21.15.83:0/3375429129 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c40a56d0 con 0x7f87cc06dea0 2024-09-19T07:50:04.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87c959a640 1 -- 172.21.15.83:0/3375429129 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f87c40a5870 con 0x7f87cc06dea0 2024-09-19T07:50:04.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87c959a640 1 -- 172.21.15.83:0/3375429129 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c40a5c60 con 0x7f87cc06dea0 2024-09-19T07:50:04.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87d0d4b640 1 -- 172.21.15.83:0/3375429129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87cc06dea0 msgr2=0x7f87cc06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.619+0000 7f87d0d4b640 1 --2- 172.21.15.83:0/3375429129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87cc06dea0 0x7f87cc06e350 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009e50 tx=0x7f87c401c960 comp rx=0 tx=0).stop 2024-09-19T07:50:04.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 -- 172.21.15.83:0/3375429129 shutdown_connections 2024-09-19T07:50:04.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 --2- 172.21.15.83:0/3375429129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87cc06dea0 0x7f87cc06e350 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 --2- 172.21.15.83:0/3375429129 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87cc071100 0x7f87cc1545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 -- 172.21.15.83:0/3375429129 >> 172.21.15.83:0/3375429129 conn(0x7f87cc06bc80 msgr2=0x7f87cc06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:04.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 -- 172.21.15.83:0/3375429129 shutdown_connections 2024-09-19T07:50:04.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 -- 172.21.15.83:0/3375429129 wait complete. 2024-09-19T07:50:04.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 Processor -- start 2024-09-19T07:50:04.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 -- start start 2024-09-19T07:50:04.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87cc06dea0 0x7f87cc073260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87cc071100 0x7f87cc0c1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87cc0c28e0 0x7f87cc14c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc071660 con 0x7f87cc071100 2024-09-19T07:50:04.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc0714e0 con 0x7f87cc0c28e0 2024-09-19T07:50:04.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f87cc160c60 con 0x7f87cc06dea0 2024-09-19T07:50:04.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87c9d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87cc071100 0x7f87cc0c1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87c9d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87cc071100 0x7f87cc0c1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47360/0 (socket says 172.21.15.83:47360) 2024-09-19T07:50:04.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87c9d9b640 1 -- 172.21.15.83:0/2117571068 learned_addr learned my addr 172.21.15.83:0/2117571068 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:04.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87ca59c640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87cc06dea0 0x7f87cc073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87cad9d640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87cc0c28e0 0x7f87cc14c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87c9d9b640 1 -- 172.21.15.83:0/2117571068 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87cc06dea0 msgr2=0x7f87cc073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87c9d9b640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87cc06dea0 0x7f87cc073260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87c9d9b640 1 -- 172.21.15.83:0/2117571068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87cc0c28e0 msgr2=0x7f87cc14c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87c9d9b640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87cc0c28e0 0x7f87cc14c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87c9d9b640 1 -- 172.21.15.83:0/2117571068 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87cc0bcdf0 con 0x7f87cc071100 2024-09-19T07:50:04.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87ca59c640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87cc06dea0 0x7f87cc073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:04.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87c9d9b640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87cc071100 0x7f87cc0c1240 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f87c000b3e0 tx=0x7f87c000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:04.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87baffd640 1 -- 172.21.15.83:0/2117571068 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c0015020 con 0x7f87cc071100 2024-09-19T07:50:04.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87baffd640 1 -- 172.21.15.83:0/2117571068 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f87c0011830 con 0x7f87cc071100 2024-09-19T07:50:04.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 -- 172.21.15.83:0/2117571068 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87cc0c1b20 con 0x7f87cc071100 2024-09-19T07:50:04.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87baffd640 1 -- 172.21.15.83:0/2117571068 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c0011c80 con 0x7f87cc071100 2024-09-19T07:50:04.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 -- 172.21.15.83:0/2117571068 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87cc0bd370 con 0x7f87cc071100 2024-09-19T07:50:04.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.623+0000 7f87d0d4b640 1 -- 172.21.15.83:0/2117571068 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87cc06e350 con 0x7f87cc071100 2024-09-19T07:50:04.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.627+0000 7f87baffd640 1 -- 172.21.15.83:0/2117571068 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f87c0018020 con 0x7f87cc071100 2024-09-19T07:50:04.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.627+0000 7f87baffd640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87ac0678f0 0x7f87ac069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.627+0000 7f87baffd640 1 -- 172.21.15.83:0/2117571068 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10163+0+0 (secure 0 0 0) 0x7f87c00869c0 con 0x7f87cc071100 2024-09-19T07:50:04.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.627+0000 7f87ca59c640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87ac0678f0 0x7f87ac069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.627+0000 7f87ca59c640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87ac0678f0 0x7f87ac069db0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009840 tx=0x7f87bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:04.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.631+0000 7f87baffd640 1 -- 172.21.15.83:0/2117571068 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f87c0088e20 con 0x7f87cc071100 2024-09-19T07:50:04.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.815+0000 7f87d0d4b640 1 -- 172.21.15.83:0/2117571068 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87cc0c4f30 con 0x7f87cc071100 2024-09-19T07:50:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.815+0000 7f87baffd640 1 -- 172.21.15.83:0/2117571068 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) ==== 95+0+5712 (secure 0 0 0) 0x7f87c001f030 con 0x7f87cc071100 2024-09-19T07:50:04.814 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:50:04.814 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":207,"btime":"2024-09-19T07:50:04:707932+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:50:03.975964+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6407":{"gid":6407,"name":"d","rank":0,"incarnation":206,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:50:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 207 2024-09-19T07:50:04.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.819+0000 7f87d0d4b640 1 -- 172.21.15.83:0/2117571068 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87ac0678f0 msgr2=0x7f87ac069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.819+0000 7f87d0d4b640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87ac0678f0 0x7f87ac069db0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009840 tx=0x7f87bc008040 comp rx=0 tx=0).stop 2024-09-19T07:50:04.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.819+0000 7f87d0d4b640 1 -- 172.21.15.83:0/2117571068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87cc071100 msgr2=0x7f87cc0c1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.819+0000 7f87d0d4b640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87cc071100 0x7f87cc0c1240 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f87c000b3e0 tx=0x7f87c000b8a0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.819+0000 7f87d0d4b640 1 -- 172.21.15.83:0/2117571068 shutdown_connections 2024-09-19T07:50:04.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.819+0000 7f87d0d4b640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f87ac0678f0 0x7f87ac069db0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.819+0000 7f87d0d4b640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87cc0c28e0 0x7f87cc14c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.819+0000 7f87d0d4b640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87cc071100 0x7f87cc0c1240 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.819+0000 7f87d0d4b640 1 --2- 172.21.15.83:0/2117571068 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87cc06dea0 0x7f87cc073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.819+0000 7f87d0d4b640 1 -- 172.21.15.83:0/2117571068 >> 172.21.15.83:0/2117571068 conn(0x7f87cc06bc80 msgr2=0x7f87cc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:04.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.819+0000 7f87d0d4b640 1 -- 172.21.15.83:0/2117571068 shutdown_connections 2024-09-19T07:50:04.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.819+0000 7f87d0d4b640 1 -- 172.21.15.83:0/2117571068 wait complete. 2024-09-19T07:50:04.832 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:50:04.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec93002640 1 Processor -- start 2024-09-19T07:50:04.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec93002640 1 -- start start 2024-09-19T07:50:04.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec93002640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fec8c0bef00 0x7fec8c156840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec93002640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fec8c0cbb00 0x7fec8c0cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec93002640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec8c0ca240 con 0x7fec8c14ca60 2024-09-19T07:50:04.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec93002640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fec8c0c9c80 con 0x7fec8c0cbb00 2024-09-19T07:50:04.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec93002640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fec8c0c9f60 con 0x7fec8c0bef00 2024-09-19T07:50:04.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec9159f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fec8c0cbb00 0x7fec8c0cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec9159f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fec8c0cbb00 0x7fec8c0cbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47362/0 (socket says 172.21.15.83:47362) 2024-09-19T07:50:04.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec9159f640 1 -- 172.21.15.83:0/2645705939 learned_addr learned my addr 172.21.15.83:0/2645705939 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:04.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec8bfff640 1 --2- 172.21.15.83:0/2645705939 >> v2:172.21.15.184:3300/0 conn(0x7fec8c0bef00 0x7fec8c156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec9159f640 1 -- 172.21.15.83:0/2645705939 >> v2:172.21.15.184:3300/0 conn(0x7fec8c0bef00 msgr2=0x7fec8c156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:04.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec9159f640 1 --2- 172.21.15.83:0/2645705939 >> v2:172.21.15.184:3300/0 conn(0x7fec8c0bef00 0x7fec8c156840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec9159f640 1 -- 172.21.15.83:0/2645705939 >> v1:172.21.15.184:6789/0 conn(0x7fec8c14ca60 legacy=0x7fec8c159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:04.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.955+0000 7fec9159f640 1 -- 172.21.15.83:0/2645705939 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8c156040 con 0x7fec8c0cbb00 2024-09-19T07:50:04.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec9159f640 1 --2- 172.21.15.83:0/2645705939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec8c0cbb00 0x7fec8c0cbf00 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fec8c06bec0 tx=0x7fec8401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b92e34122d4940 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:04.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec8b7fe640 1 -- 172.21.15.83:0/2645705939 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec840b0070 con 0x7fec8c0cbb00 2024-09-19T07:50:04.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec8b7fe640 1 -- 172.21.15.83:0/2645705939 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fec840ab600 con 0x7fec8c0cbb00 2024-09-19T07:50:04.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec8b7fe640 1 -- 172.21.15.83:0/2645705939 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec840ab9f0 con 0x7fec8c0cbb00 2024-09-19T07:50:04.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec93002640 1 -- 172.21.15.83:0/2645705939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec8c0cbb00 msgr2=0x7fec8c0cbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec93002640 1 --2- 172.21.15.83:0/2645705939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec8c0cbb00 0x7fec8c0cbf00 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fec8c06bec0 tx=0x7fec8401c4e0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec93002640 1 -- 172.21.15.83:0/2645705939 shutdown_connections 2024-09-19T07:50:04.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec93002640 1 --2- 172.21.15.83:0/2645705939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec8c0cbb00 0x7fec8c0cbf00 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec93002640 1 --2- 172.21.15.83:0/2645705939 >> v2:172.21.15.184:3300/0 conn(0x7fec8c0bef00 0x7fec8c156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec93002640 1 -- 172.21.15.83:0/2645705939 >> 172.21.15.83:0/2645705939 conn(0x7fec8c072470 msgr2=0x7fec8c072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:04.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec93002640 1 -- 172.21.15.83:0/2645705939 shutdown_connections 2024-09-19T07:50:04.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec93002640 1 -- 172.21.15.83:0/2645705939 wait complete. 2024-09-19T07:50:04.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec93002640 1 Processor -- start 2024-09-19T07:50:04.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.959+0000 7fec93002640 1 -- start start 2024-09-19T07:50:04.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec93002640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec8c0bef00 0x7fec8c0bf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec93002640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec8c0cbb00 0x7fec8c0b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec93002640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec8c14ca60 0x7fec8c0c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec93002640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c0c98c0 con 0x7fec8c0bef00 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec93002640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c153a50 con 0x7fec8c14ca60 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec93002640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fec8c0c9a40 con 0x7fec8c0cbb00 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec90d9e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec8c0bef00 0x7fec8c0bf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec90d9e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec8c0bef00 0x7fec8c0bf2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47368/0 (socket says 172.21.15.83:47368) 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec90d9e640 1 -- 172.21.15.83:0/964191473 learned_addr learned my addr 172.21.15.83:0/964191473 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec90d9e640 1 -- 172.21.15.83:0/964191473 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec8c0cbb00 msgr2=0x7fec8c0b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec8bfff640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec8c0cbb00 0x7fec8c0b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec9159f640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec8c14ca60 0x7fec8c0c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec90d9e640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec8c0cbb00 0x7fec8c0b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec90d9e640 1 -- 172.21.15.83:0/964191473 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec8c14ca60 msgr2=0x7fec8c0c0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec90d9e640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec8c14ca60 0x7fec8c0c0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:04.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec90d9e640 1 -- 172.21.15.83:0/964191473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8c0bfbf0 con 0x7fec8c0bef00 2024-09-19T07:50:04.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec90d9e640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec8c0bef00 0x7fec8c0bf2e0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00ba50 tx=0x7fec7c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:04.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec897fa640 1 -- 172.21.15.83:0/964191473 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c019070 con 0x7fec8c0bef00 2024-09-19T07:50:04.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec897fa640 1 -- 172.21.15.83:0/964191473 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fec7c00cc20 con 0x7fec8c0bef00 2024-09-19T07:50:04.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec93002640 1 -- 172.21.15.83:0/964191473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8c0bfed0 con 0x7fec8c0bef00 2024-09-19T07:50:04.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec897fa640 1 -- 172.21.15.83:0/964191473 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c014440 con 0x7fec8c0bef00 2024-09-19T07:50:04.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec93002640 1 -- 172.21.15.83:0/964191473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec8c0b0db0 con 0x7fec8c0bef00 2024-09-19T07:50:04.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec93002640 1 -- 172.21.15.83:0/964191473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec8c156840 con 0x7fec8c0bef00 2024-09-19T07:50:04.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec897fa640 1 -- 172.21.15.83:0/964191473 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fec7c01c070 con 0x7fec8c0bef00 2024-09-19T07:50:04.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec897fa640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec780678c0 0x7fec78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:04.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec8bfff640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec780678c0 0x7fec78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:04.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec897fa640 1 -- 172.21.15.83:0/964191473 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10163+0+0 (secure 0 0 0) 0x7fec7c086af0 con 0x7fec8c0bef00 2024-09-19T07:50:04.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.963+0000 7fec8bfff640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec780678c0 0x7fec78069d80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fec8c15f8c0 tx=0x7fec80008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:04.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:04.967+0000 7fec897fa640 1 -- 172.21.15.83:0/964191473 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fec7c055e20 con 0x7fec8c0bef00 2024-09-19T07:50:05.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 -- 172.21.15.83:0/964191473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec8c06ca90 con 0x7fec8c0bef00 2024-09-19T07:50:05.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec897fa640 1 -- 172.21.15.83:0/964191473 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) ==== 95+0+5712 (secure 0 0 0) 0x7fec7c05a120 con 0x7fec8c0bef00 2024-09-19T07:50:05.163 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:50:05.163 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":207,"btime":"2024-09-19T07:50:04:707932+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:50:03.975964+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6407":{"gid":6407,"name":"d","rank":0,"incarnation":206,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:50:05.164 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 207 2024-09-19T07:50:05.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 -- 172.21.15.83:0/964191473 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec780678c0 msgr2=0x7fec78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec780678c0 0x7fec78069d80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fec8c15f8c0 tx=0x7fec80008040 comp rx=0 tx=0).stop 2024-09-19T07:50:05.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 -- 172.21.15.83:0/964191473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec8c0bef00 msgr2=0x7fec8c0bf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec8c0bef00 0x7fec8c0bf2e0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00ba50 tx=0x7fec7c00bf10 comp rx=0 tx=0).stop 2024-09-19T07:50:05.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 -- 172.21.15.83:0/964191473 shutdown_connections 2024-09-19T07:50:05.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fec780678c0 0x7fec78069d80 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec8c14ca60 0x7fec8c0c0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fec8c0cbb00 0x7fec8c0b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 --2- 172.21.15.83:0/964191473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fec8c0bef00 0x7fec8c0bf2e0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 -- 172.21.15.83:0/964191473 >> 172.21.15.83:0/964191473 conn(0x7fec8c072470 msgr2=0x7fec8c155450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:05.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 -- 172.21.15.83:0/964191473 shutdown_connections 2024-09-19T07:50:05.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.167+0000 7fec93002640 1 -- 172.21.15.83:0/964191473 wait complete. 2024-09-19T07:50:05.182 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:50:05.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:50:05.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.343+0000 7f99f2061640 1 Processor -- start 2024-09-19T07:50:05.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.343+0000 7f99f2061640 1 -- start start 2024-09-19T07:50:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99f2061640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f99ec14dbe0 0x7f99ec0c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99f2061640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f99ec0c44e0 0x7f99ec157720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99f2061640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99ec073d90 con 0x7f99ec0c3ab0 2024-09-19T07:50:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99f2061640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f99ec154700 con 0x7f99ec0c44e0 2024-09-19T07:50:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99f2061640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f99ec153b30 con 0x7f99ec14dbe0 2024-09-19T07:50:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99eaffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f99ec0c3ab0 0x7f99ec157010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48876/0 (socket says 172.21.15.83:48876) 2024-09-19T07:50:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99eaffd640 1 -- 172.21.15.83:0/982002084 learned_addr learned my addr 172.21.15.83:0/982002084 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:50:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99ebfff640 1 --2- 172.21.15.83:0/982002084 >> v2:172.21.15.83:3300/0 conn(0x7f99ec0c44e0 0x7f99ec157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99eb7fe640 1 --2- 172.21.15.83:0/982002084 >> v2:172.21.15.184:3300/0 conn(0x7f99ec14dbe0 0x7f99ec0c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99ea7fc640 1 -- 172.21.15.83:0/982002084 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3333009486 0 0) 0x7f99ec073d90 con 0x7f99ec0c3ab0 2024-09-19T07:50:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99ea7fc640 1 -- 172.21.15.83:0/982002084 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99d4003530 con 0x7f99ec0c3ab0 2024-09-19T07:50:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99ebfff640 1 -- 172.21.15.83:0/982002084 >> v2:172.21.15.184:3300/0 conn(0x7f99ec14dbe0 msgr2=0x7f99ec0c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99ebfff640 1 --2- 172.21.15.83:0/982002084 >> v2:172.21.15.184:3300/0 conn(0x7f99ec14dbe0 0x7f99ec0c3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99ebfff640 1 -- 172.21.15.83:0/982002084 >> v1:172.21.15.83:6789/0 conn(0x7f99ec0c3ab0 legacy=0x7f99ec157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99ea7fc640 1 -- 172.21.15.83:0/982002084 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3431313793 0 0) 0x7f99d4003530 con 0x7f99ec0c3ab0 2024-09-19T07:50:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99ebfff640 1 -- 172.21.15.83:0/982002084 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ec153dd0 con 0x7f99ec0c44e0 2024-09-19T07:50:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99ebfff640 1 --2- 172.21.15.83:0/982002084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ec0c44e0 0x7f99ec157720 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0ca220 tx=0x7f99e401c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4613720c3534d4a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99ea7fc640 1 -- 172.21.15.83:0/982002084 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e40b2070 con 0x7f99ec0c44e0 2024-09-19T07:50:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99ea7fc640 1 -- 172.21.15.83:0/982002084 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f99e40a5510 con 0x7f99ec0c44e0 2024-09-19T07:50:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.347+0000 7f99ea7fc640 1 -- 172.21.15.83:0/982002084 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0001680 con 0x7f99ec0c44e0 2024-09-19T07:50:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 -- 172.21.15.83:0/982002084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ec0c44e0 msgr2=0x7f99ec157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 --2- 172.21.15.83:0/982002084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ec0c44e0 0x7f99ec157720 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0ca220 tx=0x7f99e401c7c0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 -- 172.21.15.83:0/982002084 shutdown_connections 2024-09-19T07:50:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 --2- 172.21.15.83:0/982002084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ec0c44e0 0x7f99ec157720 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 --2- 172.21.15.83:0/982002084 >> v2:172.21.15.184:3300/0 conn(0x7f99ec14dbe0 0x7f99ec0c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 -- 172.21.15.83:0/982002084 >> 172.21.15.83:0/982002084 conn(0x7f99ec0c95a0 msgr2=0x7f99ec0c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 -- 172.21.15.83:0/982002084 shutdown_connections 2024-09-19T07:50:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 -- 172.21.15.83:0/982002084 wait complete. 2024-09-19T07:50:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 Processor -- start 2024-09-19T07:50:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 -- start start 2024-09-19T07:50:05.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16e498d640 1 Processor -- start 2024-09-19T07:50:05.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16e498d640 1 -- start start 2024-09-19T07:50:05.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ec0c44e0 0x7f99ec148280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99ec14dbe0 0x7f99ec1f0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99ec0ad490 0x7f99ec145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec160bc0 con 0x7f99ec0c44e0 2024-09-19T07:50:05.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec0741a0 con 0x7f99ec0ad490 2024-09-19T07:50:05.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f99ec160d40 con 0x7f99ec14dbe0 2024-09-19T07:50:05.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99eb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ec0c44e0 0x7f99ec148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99eb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ec0c44e0 0x7f99ec148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47374/0 (socket says 172.21.15.83:47374) 2024-09-19T07:50:05.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99eb7fe640 1 -- 172.21.15.83:0/1166904292 learned_addr learned my addr 172.21.15.83:0/1166904292 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16e498d640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f16e006f950 0x7f16e00716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16e498d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f16e0071e20 0x7f16e0150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16e498d640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e00c37c0 con 0x7f16e0151410 2024-09-19T07:50:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16e498d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f16e00c4670 con 0x7f16e0071e20 2024-09-19T07:50:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16e498d640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f16e00c3a60 con 0x7f16e006f950 2024-09-19T07:50:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16ddd9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f16e0071e20 0x7f16e0150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16ddd9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f16e0071e20 0x7f16e0150ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47390/0 (socket says 172.21.15.83:47390) 2024-09-19T07:50:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16ddd9b640 1 -- 172.21.15.83:0/46960369 learned_addr learned my addr 172.21.15.83:0/46960369 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99ebfff640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99ec0ad490 0x7f99ec145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99eaffd640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99ec14dbe0 0x7f99ec1f0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99eb7fe640 1 -- 172.21.15.83:0/1166904292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99ec14dbe0 msgr2=0x7f99ec1f0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99eb7fe640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99ec14dbe0 0x7f99ec1f0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99eb7fe640 1 -- 172.21.15.83:0/1166904292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99ec0ad490 msgr2=0x7f99ec145c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99eb7fe640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99ec0ad490 0x7f99ec145c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99eb7fe640 1 -- 172.21.15.83:0/1166904292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ec0af620 con 0x7f99ec0c44e0 2024-09-19T07:50:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99eaffd640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99ec14dbe0 0x7f99ec1f0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99ebfff640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99ec0ad490 0x7f99ec145c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16de59c640 1 --2- 172.21.15.83:0/46960369 >> v2:172.21.15.184:3300/0 conn(0x7f16e006f950 0x7f16e00716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16ddd9b640 1 -- 172.21.15.83:0/46960369 >> v2:172.21.15.184:3300/0 conn(0x7f16e006f950 msgr2=0x7f16e00716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16ddd9b640 1 --2- 172.21.15.83:0/46960369 >> v2:172.21.15.184:3300/0 conn(0x7f16e006f950 0x7f16e00716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16ddd9b640 1 -- 172.21.15.83:0/46960369 >> v1:172.21.15.184:6789/0 conn(0x7f16e0151410 legacy=0x7f16e01575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f16ddd9b640 1 -- 172.21.15.83:0/46960369 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e00c3d40 con 0x7f16e0071e20 2024-09-19T07:50:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99eb7fe640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ec0c44e0 0x7f99ec148280 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00e8a0 tx=0x7f99dc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99d3fff640 1 -- 172.21.15.83:0/1166904292 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99dc017070 con 0x7f99ec0c44e0 2024-09-19T07:50:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99d3fff640 1 -- 172.21.15.83:0/1166904292 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f99dc00c940 con 0x7f99ec0c44e0 2024-09-19T07:50:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 -- 172.21.15.83:0/1166904292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99ec0ac570 con 0x7f99ec0c44e0 2024-09-19T07:50:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99d3fff640 1 -- 172.21.15.83:0/1166904292 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99dc00cd10 con 0x7f99ec0c44e0 2024-09-19T07:50:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.351+0000 7f99f2061640 1 -- 172.21.15.83:0/1166904292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99ec14ab40 con 0x7f99ec0c44e0 2024-09-19T07:50:05.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.355+0000 7f99f2061640 1 -- 172.21.15.83:0/1166904292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99ec1f0b40 con 0x7f99ec0c44e0 2024-09-19T07:50:05.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.355+0000 7f99d3fff640 1 -- 172.21.15.83:0/1166904292 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f99dc01a070 con 0x7f99ec0c44e0 2024-09-19T07:50:05.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.355+0000 7f99d3fff640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f99d8067910 0x7f99d8069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.355+0000 7f99d3fff640 1 -- 172.21.15.83:0/1166904292 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10163+0+0 (secure 0 0 0) 0x7f99dc086cf0 con 0x7f99ec0c44e0 2024-09-19T07:50:05.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.355+0000 7f99eaffd640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f99d8067910 0x7f99d8069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.355+0000 7f16ddd9b640 1 --2- 172.21.15.83:0/46960369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16e0071e20 0x7f16e0150ed0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f16e0070100 tx=0x7f16d401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e8a9a63a13b254 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:05.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.355+0000 7f99eaffd640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f99d8067910 0x7f99d8069dd0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f99ec153950 tx=0x7f99e0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:05.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.359+0000 7f16dd59a640 1 -- 172.21.15.83:0/46960369 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d40ab620 con 0x7f16e0071e20 2024-09-19T07:50:05.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.359+0000 7f16dd59a640 1 -- 172.21.15.83:0/46960369 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f16d40b0070 con 0x7f16e0071e20 2024-09-19T07:50:05.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.359+0000 7f16dd59a640 1 -- 172.21.15.83:0/46960369 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d40aba00 con 0x7f16e0071e20 2024-09-19T07:50:05.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.359+0000 7f99d3fff640 1 -- 172.21.15.83:0/1166904292 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f99ec1f0b40 con 0x7f99ec0c44e0 2024-09-19T07:50:05.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.359+0000 7f16e498d640 1 -- 172.21.15.83:0/46960369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16e0071e20 msgr2=0x7f16e0150ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.359+0000 7f16e498d640 1 --2- 172.21.15.83:0/46960369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16e0071e20 0x7f16e0150ed0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f16e0070100 tx=0x7f16d401c420 comp rx=0 tx=0).stop 2024-09-19T07:50:05.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.359+0000 7f16e498d640 1 -- 172.21.15.83:0/46960369 shutdown_connections 2024-09-19T07:50:05.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.359+0000 7f16e498d640 1 --2- 172.21.15.83:0/46960369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16e0071e20 0x7f16e0150ed0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.359+0000 7f16e498d640 1 --2- 172.21.15.83:0/46960369 >> v2:172.21.15.184:3300/0 conn(0x7f16e006f950 0x7f16e00716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.359+0000 7f16e498d640 1 -- 172.21.15.83:0/46960369 >> 172.21.15.83:0/46960369 conn(0x7f16e00c48b0 msgr2=0x7f16e00c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:05.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.359+0000 7f16e498d640 1 -- 172.21.15.83:0/46960369 shutdown_connections 2024-09-19T07:50:05.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.359+0000 7f16e498d640 1 -- 172.21.15.83:0/46960369 wait complete. 2024-09-19T07:50:05.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16e498d640 1 Processor -- start 2024-09-19T07:50:05.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16e498d640 1 -- start start 2024-09-19T07:50:05.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16e498d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16e006f950 0x7f16e01efb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16e498d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16e0071e20 0x7f16e0156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16e498d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16e0151410 0x7f16e01ec360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16e498d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16e01519a0 con 0x7f16e0151410 2024-09-19T07:50:05.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16e498d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f16e0151820 con 0x7f16e006f950 2024-09-19T07:50:05.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16e498d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f16e006fd30 con 0x7f16e0071e20 2024-09-19T07:50:05.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ded9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16e0151410 0x7f16e01ec360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ded9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16e0151410 0x7f16e01ec360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47392/0 (socket says 172.21.15.83:47392) 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ded9d640 1 -- 172.21.15.83:0/2435272200 learned_addr learned my addr 172.21.15.83:0/2435272200 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ddd9b640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16e0071e20 0x7f16e0156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16de59c640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16e006f950 0x7f16e01efb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ded9d640 1 -- 172.21.15.83:0/2435272200 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16e0071e20 msgr2=0x7f16e0156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ded9d640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16e0071e20 0x7f16e0156b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ded9d640 1 -- 172.21.15.83:0/2435272200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16e006f950 msgr2=0x7f16e01efb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ded9d640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16e006f950 0x7f16e01efb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ded9d640 1 -- 172.21.15.83:0/2435272200 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e01f0380 con 0x7f16e0151410 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ded9d640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16e0151410 0x7f16e01ec360 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f16d800c780 tx=0x7f16d800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ceffd640 1 -- 172.21.15.83:0/2435272200 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d8004b70 con 0x7f16e0151410 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ceffd640 1 -- 172.21.15.83:0/2435272200 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f16d8004d10 con 0x7f16e0151410 2024-09-19T07:50:05.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ceffd640 1 -- 172.21.15.83:0/2435272200 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d800aab0 con 0x7f16e0151410 2024-09-19T07:50:05.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16e498d640 1 -- 172.21.15.83:0/2435272200 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16e01f0660 con 0x7f16e0151410 2024-09-19T07:50:05.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16e498d640 1 -- 172.21.15.83:0/2435272200 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16e01f0070 con 0x7f16e0151410 2024-09-19T07:50:05.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16e498d640 1 -- 172.21.15.83:0/2435272200 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16e00716a0 con 0x7f16e0151410 2024-09-19T07:50:05.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.363+0000 7f16ceffd640 1 -- 172.21.15.83:0/2435272200 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f16d800ac50 con 0x7f16e0151410 2024-09-19T07:50:05.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.367+0000 7f16ceffd640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16c00678c0 0x7f16c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.367+0000 7f16ceffd640 1 -- 172.21.15.83:0/2435272200 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10163+0+0 (secure 0 0 0) 0x7f16d8087170 con 0x7f16e0151410 2024-09-19T07:50:05.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.367+0000 7f16de59c640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16c00678c0 0x7f16c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.367+0000 7f16de59c640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16c00678c0 0x7f16c0069d80 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f16e00c1ef0 tx=0x7f16d0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:05.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.371+0000 7f16ceffd640 1 -- 172.21.15.83:0/2435272200 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f16d8020070 con 0x7f16e0151410 2024-09-19T07:50:05.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.519+0000 7f99f2061640 1 -- 172.21.15.83:0/1166904292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f99ec0c3fa0 con 0x7f99ec0c44e0 2024-09-19T07:50:05.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99d3fff640 1 -- 172.21.15.83:0/1166904292 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) ==== 74+0+26038 (secure 0 0 0) 0x7f99dc0566f0 con 0x7f99ec0c44e0 2024-09-19T07:50:05.519 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:50:05.519 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:50:03.370921+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-19T07:19:29.461167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-19T07:19:30.589046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.f","mappings":[{"from":7,"to":1}]},{"pgid":"3.28","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:6833/2502231963":"2024-09-20T07:48:58.353983+0000","172.21.15.83:6837/1602509240":"2024-09-20T07:48:23.346687+0000","172.21.15.83:6836/1602509240":"2024-09-20T07:48:23.346687+0000","172.21.15.83:6835/119695014":"2024-09-20T07:47:58.342001+0000","172.21.15.184:6833/1207928072":"2024-09-20T07:47:03.329199+0000","172.21.15.83:6836/3490578864":"2024-09-20T07:45:28.310975+0000","172.21.15.83:6835/1429520945":"2024-09-20T07:45:13.307981+0000","172.21.15.83:6834/1429520945":"2024-09-20T07:45:13.307981+0000","172.21.15.184:6833/3311721080":"2024-09-20T07:44:18.290353+0000","172.21.15.184:6832/3311721080":"2024-09-20T07:44:18.290353+0000","172.21.15.83:6837/1990119490":"2024-09-20T07:42:38.260871+0000","172.21.15.83:6836/1990119490":"2024-09-20T07:42:38.260871+0000","172.21.15.83:6834/2776269095":"2024-09-20T07:42:08.252858+0000","172.21.15.184:6833/1335198090":"2024-09-20T07:41:33.245545+0000","172.21.15.83:6837/3426675376":"2024-09-20T07:50:03.370836+0000","172.21.15.184:6832/1335198090":"2024-09-20T07:41:33.245545+0000","172.21.15.83:6835/365493751":"2024-09-20T07:41:08.239813+0000","172.21.15.83:6834/119695014":"2024-09-20T07:47:58.342001+0000","172.21.15.83:6835/104517500":"2024-09-20T07:28:38.045210+0000","172.21.15.83:6837/2326757290":"2024-09-20T07:40:23.230841+0000","172.21.15.184:6835/872719174":"2024-09-20T07:28:08.038207+0000","172.21.15.83:6834/3763485069":"2024-09-20T07:27:18.018593+0000","172.21.15.184:6834/872719174":"2024-09-20T07:28:08.038207+0000","172.21.15.184:6832/1207928072":"2024-09-20T07:47:03.329199+0000","172.21.15.83:6836/455754420":"2024-09-20T07:27:38.022965+0000","172.21.15.83:6835/3763485069":"2024-09-20T07:27:18.018593+0000","172.21.15.184:6833/1773885233":"2024-09-20T07:22:22.945705+0000","172.21.15.184:6835/144714632":"2024-09-20T07:25:36.204686+0000","172.21.15.83:6836/26497755":"2024-09-20T07:25:27.991655+0000","172.21.15.184:6833/2993984999":"2024-09-20T07:32:38.105001+0000","172.21.15.83:6836/4265456507":"2024-09-20T07:22:32.948957+0000","172.21.15.83:6834/1072903448":"2024-09-20T07:33:28.117348+0000","172.21.15.83:6835/3442755673":"2024-09-20T07:24:47.982274+0000","172.21.15.83:6837/3490578864":"2024-09-20T07:45:28.310975+0000","172.21.15.184:6835/149004692":"2024-09-20T07:20:57.928172+0000","172.21.15.184:6834/3841282254":"2024-09-20T07:30:48.076144+0000","172.21.15.184:6832/2502231963":"2024-09-20T07:48:58.353983+0000","172.21.15.83:6836/1229829432":"2024-09-20T07:29:48.061079+0000","172.21.15.83:6834/1819616410":"2024-09-20T07:21:07.930616+0000","172.21.15.83:6835/1819616410":"2024-09-20T07:21:07.930616+0000","172.21.15.184:6834/149004692":"2024-09-20T07:20:57.928172+0000","172.21.15.184:6832/1773885233":"2024-09-20T07:22:22.945705+0000","172.21.15.184:6835/2112478738":"2024-09-20T07:35:28.148108+0000","172.21.15.184:6832/2993984999":"2024-09-20T07:32:38.105001+0000","172.21.15.83:6837/26497755":"2024-09-20T07:25:27.991655+0000","172.21.15.184:6832/2898687304":"2024-09-20T07:34:18.130590+0000","172.21.15.83:6835/2776269095":"2024-09-20T07:42:08.252858+0000","172.21.15.83:6836/1049139637":"2024-09-20T07:23:37.966033+0000","172.21.15.83:6834/3442755673":"2024-09-20T07:24:47.982274+0000","172.21.15.83:6837/4265456507":"2024-09-20T07:22:32.948957+0000","172.21.15.184:6833/2898687304":"2024-09-20T07:34:18.130590+0000","172.21.15.184:6835/4115191178":"2024-09-20T07:23:52.164421+0000","172.21.15.184:6834/4115191178":"2024-09-20T07:23:52.164421+0000","172.21.15.83:6834/1774585869":"2024-09-20T07:23:07.959477+0000","172.21.15.184:6834/144714632":"2024-09-20T07:25:36.204686+0000","172.21.15.83:6834/104517500":"2024-09-20T07:28:38.045210+0000","172.21.15.83:6837/1049139637":"2024-09-20T07:23:37.966033+0000","172.21.15.83:6834/2004814399":"2024-09-20T07:30:48.076245+0000","172.21.15.83:6837/1229829432":"2024-09-20T07:29:48.061079+0000","172.21.15.83:6835/2004814399":"2024-09-20T07:30:48.076245+0000","172.21.15.184:6835/3841282254":"2024-09-20T07:30:48.076144+0000","172.21.15.83:6836/2581473241":"2024-09-20T07:31:18.082843+0000","172.21.15.83:6837/455754420":"2024-09-20T07:27:38.022965+0000","172.21.15.83:6834/1079027718":"2024-09-20T07:35:43.151965+0000","172.21.15.83:6837/2581473241":"2024-09-20T07:31:18.082843+0000","172.21.15.83:6836/2326757290":"2024-09-20T07:40:23.230841+0000","172.21.15.83:6835/1072903448":"2024-09-20T07:33:28.117348+0000","172.21.15.83:6835/1774585869":"2024-09-20T07:23:07.959477+0000","172.21.15.184:6835/1519298917":"2024-09-20T07:37:18.179638+0000","172.21.15.83:6836/1203406606":"2024-09-20T07:34:08.127781+0000","172.21.15.83:6834/365493751":"2024-09-20T07:41:08.239813+0000","172.21.15.83:6837/1203406606":"2024-09-20T07:34:08.127781+0000","172.21.15.83:6835/1079027718":"2024-09-20T07:35:43.151965+0000","172.21.15.184:6834/2112478738":"2024-09-20T07:35:28.148108+0000","172.21.15.184:6834/1519298917":"2024-09-20T07:37:18.179638+0000","172.21.15.83:6836/3707393259":"2024-09-20T07:37:18.179736+0000","172.21.15.83:6836/3426675376":"2024-09-20T07:50:03.370836+0000","172.21.15.83:6837/3707393259":"2024-09-20T07:37:18.179736+0000","172.21.15.83:6834/2168217861":"2024-09-20T07:38:08.197381+0000","172.21.15.83:6835/2168217861":"2024-09-20T07:38:08.197381+0000","172.21.15.184:6832/381468017":"2024-09-20T07:39:03.210538+0000","172.21.15.184:6833/381468017":"2024-09-20T07:39:03.210538+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:50:05.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99f2061640 1 -- 172.21.15.83:0/1166904292 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f99d8067910 msgr2=0x7f99d8069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99f2061640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f99d8067910 0x7f99d8069dd0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f99ec153950 tx=0x7f99e0009040 comp rx=0 tx=0).stop 2024-09-19T07:50:05.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99f2061640 1 -- 172.21.15.83:0/1166904292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ec0c44e0 msgr2=0x7f99ec148280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99f2061640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ec0c44e0 0x7f99ec148280 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00e8a0 tx=0x7f99dc00eca0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99f2061640 1 -- 172.21.15.83:0/1166904292 shutdown_connections 2024-09-19T07:50:05.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99f2061640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f99d8067910 0x7f99d8069dd0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99f2061640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99ec0ad490 0x7f99ec145c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99f2061640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99ec14dbe0 0x7f99ec1f0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99f2061640 1 --2- 172.21.15.83:0/1166904292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ec0c44e0 0x7f99ec148280 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99f2061640 1 -- 172.21.15.83:0/1166904292 >> 172.21.15.83:0/1166904292 conn(0x7f99ec0c95a0 msgr2=0x7f99ec151890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:05.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99f2061640 1 -- 172.21.15.83:0/1166904292 shutdown_connections 2024-09-19T07:50:05.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.523+0000 7f99f2061640 1 -- 172.21.15.83:0/1166904292 wait complete. 2024-09-19T07:50:05.543 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail a 2024-09-19T07:50:05.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.575+0000 7f16e498d640 1 -- 172.21.15.83:0/2435272200 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16e00c2490 con 0x7f16e0151410 2024-09-19T07:50:05.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.575+0000 7f16ceffd640 1 -- 172.21.15.83:0/2435272200 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) ==== 95+0+5712 (secure 0 0 0) 0x7f16d8056af0 con 0x7f16e0151410 2024-09-19T07:50:05.574 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:50:05.574 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":207,"btime":"2024-09-19T07:50:04:707932+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:50:03.975964+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6407":{"gid":6407,"name":"d","rank":0,"incarnation":206,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.184:6833/922771363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":922771363},{"type":"v1","addr":"172.21.15.184:6833","nonce":922771363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:50:05.574 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 207 2024-09-19T07:50:05.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.579+0000 7f16e498d640 1 -- 172.21.15.83:0/2435272200 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16c00678c0 msgr2=0x7f16c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.579+0000 7f16e498d640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16c00678c0 0x7f16c0069d80 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f16e00c1ef0 tx=0x7f16d0008040 comp rx=0 tx=0).stop 2024-09-19T07:50:05.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.579+0000 7f16e498d640 1 -- 172.21.15.83:0/2435272200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16e0151410 msgr2=0x7f16e01ec360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.579+0000 7f16e498d640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16e0151410 0x7f16e01ec360 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f16d800c780 tx=0x7f16d800cc40 comp rx=0 tx=0).stop 2024-09-19T07:50:05.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.579+0000 7f16e498d640 1 -- 172.21.15.83:0/2435272200 shutdown_connections 2024-09-19T07:50:05.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.579+0000 7f16e498d640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f16c00678c0 0x7f16c0069d80 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.579+0000 7f16e498d640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16e0151410 0x7f16e01ec360 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.579+0000 7f16e498d640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16e0071e20 0x7f16e0156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.579+0000 7f16e498d640 1 --2- 172.21.15.83:0/2435272200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16e006f950 0x7f16e01efb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.579+0000 7f16e498d640 1 -- 172.21.15.83:0/2435272200 >> 172.21.15.83:0/2435272200 conn(0x7f16e00c48b0 msgr2=0x7f16e00aa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:05.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.579+0000 7f16e498d640 1 -- 172.21.15.83:0/2435272200 shutdown_connections 2024-09-19T07:50:05.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.579+0000 7f16e498d640 1 -- 172.21.15.83:0/2435272200 wait complete. 2024-09-19T07:50:05.592 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-19T07:50:05.592 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-19T07:50:05.593 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:50:05.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc82a6ea640 1 Processor -- start 2024-09-19T07:50:05.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc82a6ea640 1 -- start start 2024-09-19T07:50:05.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc82a6ea640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8240c6230 0x7fc8240c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc82a6ea640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fc8240c7b00 0x7fc824150f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc82a6ea640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc824157260 con 0x7fc824151450 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc82a6ea640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fc824157540 con 0x7fc8240c7b00 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc82a6ea640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc824157820 con 0x7fc8240c6230 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc828c87640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc824151450 0x7fc824153b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48886/0 (socket says 172.21.15.83:48886) 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc828c87640 1 -- 172.21.15.83:0/1888025470 learned_addr learned my addr 172.21.15.83:0/1888025470 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc823fff640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8240c6230 0x7fc8240c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc81b7fe640 1 --2- 172.21.15.83:0/1888025470 >> v2:172.21.15.184:3300/0 conn(0x7fc8240c7b00 0x7fc824150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2570656071 0 0) 0x7fc824157260 con 0x7fc824151450 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.663+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc810003530 con 0x7fc824151450 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3429522545 0 0) 0x7fc810003530 con 0x7fc824151450 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8240c6230 msgr2=0x7fc8240c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8240c6230 0x7fc8240c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 >> v2:172.21.15.184:3300/0 conn(0x7fc8240c7b00 msgr2=0x7fc824150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:05.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 --2- 172.21.15.83:0/1888025470 >> v2:172.21.15.184:3300/0 conn(0x7fc8240c7b00 0x7fc824150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc824157b00 con 0x7fc824151450 2024-09-19T07:50:05.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fc81c001a00 con 0x7fc824151450 2024-09-19T07:50:05.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 >> v1:172.21.15.83:6789/0 conn(0x7fc824151450 legacy=0x7fc824153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8240c6230 0x7fc810003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc810004000 0x7fc810004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc810004d50 0x7fc8100051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc824157540 con 0x7fc810004000 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc824157820 con 0x7fc810004d50 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc81000ba50 con 0x7fc8240c6230 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc81b7fe640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc810004000 0x7fc810004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc828c87640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc810004d50 0x7fc8100051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc823fff640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8240c6230 0x7fc810003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc81b7fe640 1 -- 172.21.15.83:0/1888025470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8240c6230 msgr2=0x7fc810003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc81b7fe640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8240c6230 0x7fc810003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc81b7fe640 1 -- 172.21.15.83:0/1888025470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc810004d50 msgr2=0x7fc8100051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc81b7fe640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc810004d50 0x7fc8100051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc81b7fe640 1 -- 172.21.15.83:0/1888025470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc810003530 con 0x7fc810004000 2024-09-19T07:50:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc828c87640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc810004d50 0x7fc8100051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:50:05.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc81b7fe640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc810004000 0x7fc810004810 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009ab0 tx=0x7fc80c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2df3497cc727524 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:05.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c0b2070 con 0x7fc810004000 2024-09-19T07:50:05.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc80c0a5570 con 0x7fc810004000 2024-09-19T07:50:05.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc8237fe640 1 -- 172.21.15.83:0/1888025470 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c0096b0 con 0x7fc810004000 2024-09-19T07:50:05.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.667+0000 7fc82a6ea640 1 -- 172.21.15.83:0/1888025470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc810004000 msgr2=0x7fc810004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 --2- 172.21.15.83:0/1888025470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc810004000 0x7fc810004810 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009ab0 tx=0x7fc80c01ca90 comp rx=0 tx=0).stop 2024-09-19T07:50:05.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc828c87640 1 -- 172.21.15.83:0/1888025470 reap_dead start 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 -- 172.21.15.83:0/1888025470 shutdown_connections 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 -- 172.21.15.83:0/1888025470 >> 172.21.15.83:0/1888025470 conn(0x7fc8240c9570 msgr2=0x7fc8240c9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 -- 172.21.15.83:0/1888025470 shutdown_connections 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 -- 172.21.15.83:0/1888025470 wait complete. 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 Processor -- start 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 -- start start 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc810004000 0x7fc82414ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc82414b3a0 0x7fc8241567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc824156ce0 0x7fc824144280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc8240c6790 con 0x7fc810004000 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc8240c6610 con 0x7fc82414b3a0 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc8240c75c0 con 0x7fc824156ce0 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc823fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc810004000 0x7fc82414ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc823fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc810004000 0x7fc82414ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47398/0 (socket says 172.21.15.83:47398) 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc823fff640 1 -- 172.21.15.83:0/606753872 learned_addr learned my addr 172.21.15.83:0/606753872 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc828c87640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc824156ce0 0x7fc824144280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc81b7fe640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc82414b3a0 0x7fc8241567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc823fff640 1 -- 172.21.15.83:0/606753872 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc824156ce0 msgr2=0x7fc824144280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc823fff640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc824156ce0 0x7fc824144280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc823fff640 1 -- 172.21.15.83:0/606753872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc82414b3a0 msgr2=0x7fc8241567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc823fff640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc82414b3a0 0x7fc8241567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc823fff640 1 -- 172.21.15.83:0/606753872 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82414b900 con 0x7fc810004000 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc823fff640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc810004000 0x7fc82414ae60 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fc814011640 tx=0x7fc814011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc8217fa640 1 -- 172.21.15.83:0/606753872 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81400f8c0 con 0x7fc810004000 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc8217fa640 1 -- 172.21.15.83:0/606753872 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc81400fa60 con 0x7fc810004000 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 -- 172.21.15.83:0/606753872 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8240c3490 con 0x7fc810004000 2024-09-19T07:50:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc8217fa640 1 -- 172.21.15.83:0/606753872 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81400fe30 con 0x7fc810004000 2024-09-19T07:50:05.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 -- 172.21.15.83:0/606753872 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8241ebf40 con 0x7fc810004000 2024-09-19T07:50:05.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.671+0000 7fc82a6ea640 1 -- 172.21.15.83:0/606753872 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc824150f10 con 0x7fc810004000 2024-09-19T07:50:05.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.675+0000 7fc8217fa640 1 -- 172.21.15.83:0/606753872 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc814015460 con 0x7fc810004000 2024-09-19T07:50:05.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.675+0000 7fc8217fa640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8040678c0 0x7fc804069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.675+0000 7fc8217fa640 1 -- 172.21.15.83:0/606753872 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10163+0+0 (secure 0 0 0) 0x7fc814087790 con 0x7fc810004000 2024-09-19T07:50:05.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.675+0000 7fc81b7fe640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8040678c0 0x7fc804069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.675+0000 7fc81b7fe640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8040678c0 0x7fc804069d80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0af000 tx=0x7fc80c0046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:05.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.679+0000 7fc8217fa640 1 -- 172.21.15.83:0/606753872 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fc82414b900 con 0x7fc810004000 2024-09-19T07:50:05.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4ad4970640 1 Processor -- start 2024-09-19T07:50:05.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4ad4970640 1 -- start start 2024-09-19T07:50:05.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4ad4970640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f4ad0150f40 0x7f4ad00733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4ad4970640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad01539a0 0x7f4ad0153e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4ad4970640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ad00c3aa0 con 0x7f4ad00adae0 2024-09-19T07:50:05.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4ad4970640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f4ad00c3850 con 0x7f4ad0150f40 2024-09-19T07:50:05.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4ad4970640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4ad00c43d0 con 0x7f4ad01539a0 2024-09-19T07:50:05.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acdd9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4ad00adae0 0x7f4ad006f780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48892/0 (socket says 172.21.15.83:48892) 2024-09-19T07:50:05.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acdd9b640 1 -- 172.21.15.83:0/206620739 learned_addr learned my addr 172.21.15.83:0/206620739 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:50:05.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3065653671 0 0) 0x7f4ad00c3aa0 con 0x7f4ad00adae0 2024-09-19T07:50:05.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4ace59c640 1 --2- 172.21.15.83:0/206620739 >> v2:172.21.15.184:3300/0 conn(0x7f4ad0150f40 0x7f4ad00733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ab8003530 con 0x7f4ad00adae0 2024-09-19T07:50:05.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4aced9d640 1 --2- 172.21.15.83:0/206620739 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad01539a0 0x7f4ad0153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3436659013 0 0) 0x7f4ab8003530 con 0x7f4ad00adae0 2024-09-19T07:50:05.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad01539a0 msgr2=0x7f4ad0153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:05.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 --2- 172.21.15.83:0/206620739 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad01539a0 0x7f4ad0153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 >> v2:172.21.15.184:3300/0 conn(0x7f4ad0150f40 msgr2=0x7f4ad00733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:05.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 --2- 172.21.15.83:0/206620739 >> v2:172.21.15.184:3300/0 conn(0x7f4ad0150f40 0x7f4ad00733d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad00c3d80 con 0x7f4ad00adae0 2024-09-19T07:50:05.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f4ac4001680 con 0x7f4ad00adae0 2024-09-19T07:50:05.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 >> v1:172.21.15.83:6789/0 conn(0x7f4ad00adae0 legacy=0x7f4ad006f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 --2- 172.21.15.83:0/206620739 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad01539a0 0x7f4ab80039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 --2- 172.21.15.83:0/206620739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ab8003ff0 0x7f4ab8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 --2- 172.21.15.83:0/206620739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ab8004d50 0x7f4ab80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad00c3850 con 0x7f4ab8004d50 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad00c43d0 con 0x7f4ab8003ff0 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4ab800bad0 con 0x7f4ad01539a0 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.751+0000 7f4aced9d640 1 --2- 172.21.15.83:0/206620739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ab8004d50 0x7f4ab80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4ace59c640 1 --2- 172.21.15.83:0/206620739 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad01539a0 0x7f4ab80039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4acdd9b640 1 --2- 172.21.15.83:0/206620739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ab8003ff0 0x7f4ab8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4aced9d640 1 -- 172.21.15.83:0/206620739 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad01539a0 msgr2=0x7f4ab80039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4aced9d640 1 --2- 172.21.15.83:0/206620739 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad01539a0 0x7f4ab80039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4aced9d640 1 -- 172.21.15.83:0/206620739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ab8003ff0 msgr2=0x7f4ab8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4aced9d640 1 --2- 172.21.15.83:0/206620739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ab8003ff0 0x7f4ab8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4aced9d640 1 -- 172.21.15.83:0/206620739 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab8003530 con 0x7f4ab8004d50 2024-09-19T07:50:05.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4aced9d640 1 --2- 172.21.15.83:0/206620739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ab8004d50 0x7f4ab80051d0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000ba30 tx=0x7f4ac001c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd602afe33ea032 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac00ae070 con 0x7f4ab8004d50 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4ac00a9680 con 0x7f4ab8004d50 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4acd59a640 1 -- 172.21.15.83:0/206620739 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac000b600 con 0x7f4ab8004d50 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4ad4970640 1 -- 172.21.15.83:0/206620739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ab8004d50 msgr2=0x7f4ab80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4ad4970640 1 --2- 172.21.15.83:0/206620739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ab8004d50 0x7f4ab80051d0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000ba30 tx=0x7f4ac001c6a0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4aced9d640 1 -- 172.21.15.83:0/206620739 reap_dead start 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4ad4970640 1 -- 172.21.15.83:0/206620739 shutdown_connections 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4ad4970640 1 -- 172.21.15.83:0/206620739 >> 172.21.15.83:0/206620739 conn(0x7f4ad00c4940 msgr2=0x7f4ad00c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4ad4970640 1 -- 172.21.15.83:0/206620739 shutdown_connections 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.755+0000 7f4ad4970640 1 -- 172.21.15.83:0/206620739 wait complete. 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4ad4970640 1 Processor -- start 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4ad4970640 1 -- start start 2024-09-19T07:50:05.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4ad4970640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad00c14d0 0x7f4ad006d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4ad4970640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ad0146a30 0x7f4ad0145170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4ad4970640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad006f250 0x7f4ad013d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4ad4970640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad006fe90 con 0x7f4ad006f250 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4ad4970640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad0151320 con 0x7f4ad0146a30 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4ad4970640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4ad0070010 con 0x7f4ad00c14d0 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4aced9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad006f250 0x7f4ad013d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4aced9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad006f250 0x7f4ad013d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47420/0 (socket says 172.21.15.83:47420) 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4aced9d640 1 -- 172.21.15.83:0/1187687798 learned_addr learned my addr 172.21.15.83:0/1187687798 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4acdd9b640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ad0146a30 0x7f4ad0145170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4ace59c640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad00c14d0 0x7f4ad006d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4aced9d640 1 -- 172.21.15.83:0/1187687798 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad00c14d0 msgr2=0x7f4ad006d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4aced9d640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad00c14d0 0x7f4ad006d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4aced9d640 1 -- 172.21.15.83:0/1187687798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ad0146a30 msgr2=0x7f4ad0145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4aced9d640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ad0146a30 0x7f4ad0145170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4aced9d640 1 -- 172.21.15.83:0/1187687798 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad01ec0b0 con 0x7f4ad006f250 2024-09-19T07:50:05.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4aced9d640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad006f250 0x7f4ad013d4d0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f4ac001cb80 tx=0x7f4ab8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:05.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4abeffd640 1 -- 172.21.15.83:0/1187687798 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac00ae040 con 0x7f4ad006f250 2024-09-19T07:50:05.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4abeffd640 1 -- 172.21.15.83:0/1187687798 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4ac00a9c70 con 0x7f4ad006f250 2024-09-19T07:50:05.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4ad4970640 1 -- 172.21.15.83:0/1187687798 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ad0141c90 con 0x7f4ad006f250 2024-09-19T07:50:05.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4abeffd640 1 -- 172.21.15.83:0/1187687798 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0004020 con 0x7f4ad006f250 2024-09-19T07:50:05.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4ad4970640 1 -- 172.21.15.83:0/1187687798 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ad00c1db0 con 0x7f4ad006f250 2024-09-19T07:50:05.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4ad4970640 1 -- 172.21.15.83:0/1187687798 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ad0157da0 con 0x7f4ad006f250 2024-09-19T07:50:05.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.759+0000 7f4abeffd640 1 -- 172.21.15.83:0/1187687798 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4ac00b1070 con 0x7f4ad006f250 2024-09-19T07:50:05.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.763+0000 7f4abeffd640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ab00678f0 0x7f4ab0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:05.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.763+0000 7f4abeffd640 1 -- 172.21.15.83:0/1187687798 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10163+0+0 (secure 0 0 0) 0x7f4ac0121430 con 0x7f4ad006f250 2024-09-19T07:50:05.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.763+0000 7f4ace59c640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ab00678f0 0x7f4ab0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:05.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.763+0000 7f4ace59c640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ab00678f0 0x7f4ab0069db0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80042f0 tx=0x7f4ac800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:05.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.767+0000 7f4abeffd640 1 -- 172.21.15.83:0/1187687798 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f4ac00c1060 con 0x7f4ad006f250 2024-09-19T07:50:05.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.879+0000 7fc82a6ea640 1 -- 172.21.15.83:0/606753872 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "a"} v 0) -- 0x7fc8240c9b20 con 0x7fc810004000 2024-09-19T07:50:05.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.883+0000 7fc8217fa640 1 -- 172.21.15.83:0/606753872 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "a"}]=0 a marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v209) ==== 151+0+0 (secure 0 0 0) 0x7fc814019030 con 0x7fc810004000 2024-09-19T07:50:05.881 INFO:teuthology.orchestra.run.smithi083.stderr:a marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-19T07:50:05.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.887+0000 7fc82a6ea640 1 -- 172.21.15.83:0/606753872 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8040678c0 msgr2=0x7fc804069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.887+0000 7fc82a6ea640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8040678c0 0x7fc804069d80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0af000 tx=0x7fc80c0046d0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.887+0000 7fc82a6ea640 1 -- 172.21.15.83:0/606753872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc810004000 msgr2=0x7fc82414ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.887+0000 7fc82a6ea640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc810004000 0x7fc82414ae60 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fc814011640 tx=0x7fc814011b00 comp rx=0 tx=0).stop 2024-09-19T07:50:05.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.887+0000 7fc82a6ea640 1 -- 172.21.15.83:0/606753872 shutdown_connections 2024-09-19T07:50:05.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.887+0000 7fc82a6ea640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fc8040678c0 0x7fc804069d80 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.887+0000 7fc82a6ea640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc824156ce0 0x7fc824144280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.887+0000 7fc82a6ea640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc82414b3a0 0x7fc8241567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.887+0000 7fc82a6ea640 1 --2- 172.21.15.83:0/606753872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc810004000 0x7fc82414ae60 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.887+0000 7fc82a6ea640 1 -- 172.21.15.83:0/606753872 >> 172.21.15.83:0/606753872 conn(0x7fc8240c9570 msgr2=0x7fc824073670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:05.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.887+0000 7fc82a6ea640 1 -- 172.21.15.83:0/606753872 shutdown_connections 2024-09-19T07:50:05.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.887+0000 7fc82a6ea640 1 -- 172.21.15.83:0/606753872 wait complete. 2024-09-19T07:50:05.901 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm a --yes-i-really-mean-it 2024-09-19T07:50:05.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.955+0000 7f4ad4970640 1 -- 172.21.15.83:0/1187687798 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ad006d5a0 con 0x7f4ad006f250 2024-09-19T07:50:05.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4abeffd640 1 -- 172.21.15.83:0/1187687798 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+4874 (secure 0 0 0) 0x7f4ad006d5a0 con 0x7f4ad006f250 2024-09-19T07:50:05.955 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:50:05.955 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":209,"btime":"2024-09-19T07:50:05:884530+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":209,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:31.617640+0000","modified":"2024-09-19T07:50:05.884234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:50:05.955 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 209 2024-09-19T07:50:05.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4ad4970640 1 -- 172.21.15.83:0/1187687798 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ab00678f0 msgr2=0x7f4ab0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4ad4970640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ab00678f0 0x7f4ab0069db0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80042f0 tx=0x7f4ac800b040 comp rx=0 tx=0).stop 2024-09-19T07:50:05.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4ad4970640 1 -- 172.21.15.83:0/1187687798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad006f250 msgr2=0x7f4ad013d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:05.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4ad4970640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad006f250 0x7f4ad013d4d0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f4ac001cb80 tx=0x7f4ab8005840 comp rx=0 tx=0).stop 2024-09-19T07:50:05.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4ad4970640 1 -- 172.21.15.83:0/1187687798 shutdown_connections 2024-09-19T07:50:05.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4ad4970640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f4ab00678f0 0x7f4ab0069db0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4ad4970640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad006f250 0x7f4ad013d4d0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4ad4970640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ad0146a30 0x7f4ad0145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4ad4970640 1 --2- 172.21.15.83:0/1187687798 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4ad00c14d0 0x7f4ad006d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:05.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4ad4970640 1 -- 172.21.15.83:0/1187687798 >> 172.21.15.83:0/1187687798 conn(0x7f4ad00c4940 msgr2=0x7f4ad00c4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:05.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4ad4970640 1 -- 172.21.15.83:0/1187687798 shutdown_connections 2024-09-19T07:50:05.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:05.959+0000 7f4ad4970640 1 -- 172.21.15.83:0/1187687798 wait complete. 2024-09-19T07:50:05.959 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-19T07:50:05.974 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-19T07:50:05:884530+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 209, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:31.617640+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 209, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [ 0 ], "flags": 19, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": false, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": {}, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-19T07:50:05.884234+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": {} } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-19T07:19:39.753527+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 199, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8196": { "addr": "172.21.15.83:6835/3580797659", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 3580797659, "type": "v2" }, { "addr": "172.21.15.83:6835", "nonce": 3580797659, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8196, "incarnation": 196, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-19T07:48:26.723707+0000", "qdb_cluster": [ 8196 ], "qdb_leader": 8196, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8196 } } } ], "standbys": [ { "addr": "172.21.15.184:6835/3171991578", "addrs": { "addrvec": [ { "addr": "172.21.15.184:6834", "nonce": 3171991578, "type": "v2" }, { "addr": "172.21.15.184:6835", "nonce": 3171991578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 134, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-19T07:50:06.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.067+0000 7f71f7540640 1 Processor -- start 2024-09-19T07:50:06.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.067+0000 7f71f7540640 1 -- start start 2024-09-19T07:50:06.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.067+0000 7f71f7540640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f71f0151130 0x7f71f00c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:06.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.067+0000 7f71f7540640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f71f00c3590 0x7f71f0154120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:06.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.067+0000 7f71f7540640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f00c2330 con 0x7f71f00c2af0 2024-09-19T07:50:06.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.067+0000 7f71f7540640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f71f00715d0 con 0x7f71f00c3590 2024-09-19T07:50:06.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.067+0000 7f71f7540640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f71f00c1d70 con 0x7f71f0151130 2024-09-19T07:50:06.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.067+0000 7f71f5add640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f71f00c3590 0x7f71f0154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:06.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.067+0000 7f71f4adb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f71f00c2af0 0x7f71f0153a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48908/0 (socket says 172.21.15.83:48908) 2024-09-19T07:50:06.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.067+0000 7f71f4adb640 1 -- 172.21.15.83:0/3107041428 learned_addr learned my addr 172.21.15.83:0/3107041428 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:50:06.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.067+0000 7f71f52dc640 1 --2- 172.21.15.83:0/3107041428 >> v2:172.21.15.184:3300/0 conn(0x7f71f0151130 0x7f71f00c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:06.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f5add640 1 -- 172.21.15.83:0/3107041428 >> v2:172.21.15.184:3300/0 conn(0x7f71f0151130 msgr2=0x7f71f00c25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:06.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f5add640 1 --2- 172.21.15.83:0/3107041428 >> v2:172.21.15.184:3300/0 conn(0x7f71f0151130 0x7f71f00c25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:06.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f5add640 1 -- 172.21.15.83:0/3107041428 >> v1:172.21.15.83:6789/0 conn(0x7f71f00c2af0 legacy=0x7f71f0153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:06.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f5add640 1 -- 172.21.15.83:0/3107041428 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f00c2050 con 0x7f71f00c3590 2024-09-19T07:50:06.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71e7fff640 1 -- 172.21.15.83:0/3107041428 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 369551348 0 0) 0x7f71f00c2330 con 0x7f71f00c2af0 2024-09-19T07:50:06.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f52dc640 1 --2- 172.21.15.83:0/3107041428 >> v2:172.21.15.184:3300/0 conn(0x7f71f0151130 0x7f71f00c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:06.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f5add640 1 --2- 172.21.15.83:0/3107041428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71f00c3590 0x7f71f0154120 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f71f0071750 tx=0x7f71e801c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40354511d1bf3404 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:06.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71e7fff640 1 -- 172.21.15.83:0/3107041428 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e80b2070 con 0x7f71f00c3590 2024-09-19T07:50:06.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71e7fff640 1 -- 172.21.15.83:0/3107041428 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f71e80a57e0 con 0x7f71f00c3590 2024-09-19T07:50:06.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71e7fff640 1 -- 172.21.15.83:0/3107041428 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e80a5bd0 con 0x7f71f00c3590 2024-09-19T07:50:06.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f7540640 1 -- 172.21.15.83:0/3107041428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71f00c3590 msgr2=0x7f71f0154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:06.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f7540640 1 --2- 172.21.15.83:0/3107041428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71f00c3590 0x7f71f0154120 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f71f0071750 tx=0x7f71e801c810 comp rx=0 tx=0).stop 2024-09-19T07:50:06.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f7540640 1 -- 172.21.15.83:0/3107041428 shutdown_connections 2024-09-19T07:50:06.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f7540640 1 --2- 172.21.15.83:0/3107041428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71f00c3590 0x7f71f0154120 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:06.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f7540640 1 --2- 172.21.15.83:0/3107041428 >> v2:172.21.15.184:3300/0 conn(0x7f71f0151130 0x7f71f00c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:06.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f7540640 1 -- 172.21.15.83:0/3107041428 >> 172.21.15.83:0/3107041428 conn(0x7f71f006bc80 msgr2=0x7f71f006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:06.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f7540640 1 -- 172.21.15.83:0/3107041428 shutdown_connections 2024-09-19T07:50:06.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.071+0000 7f71f7540640 1 -- 172.21.15.83:0/3107041428 wait complete. 2024-09-19T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f7540640 1 Processor -- start 2024-09-19T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f7540640 1 -- start start 2024-09-19T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f7540640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71f00c2af0 0x7f71f00c6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f7540640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71f00c3590 0x7f71f014a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f7540640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71f0151130 0x7f71f014acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f7540640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f71f0151690 con 0x7f71f00c3590 2024-09-19T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f7540640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f71f0151510 con 0x7f71f00c2af0 2024-09-19T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f7540640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f71f0160b90 con 0x7f71f0151130 2024-09-19T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f4adb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71f00c3590 0x7f71f014a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f4adb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71f00c3590 0x7f71f014a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47444/0 (socket says 172.21.15.83:47444) 2024-09-19T07:50:06.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f4adb640 1 -- 172.21.15.83:0/3648105756 learned_addr learned my addr 172.21.15.83:0/3648105756 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:06.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f4adb640 1 -- 172.21.15.83:0/3648105756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71f0151130 msgr2=0x7f71f014acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:06.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f4adb640 1 --2- 172.21.15.83:0/3648105756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71f0151130 0x7f71f014acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:06.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f4adb640 1 -- 172.21.15.83:0/3648105756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71f00c2af0 msgr2=0x7f71f00c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:06.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f4adb640 1 --2- 172.21.15.83:0/3648105756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71f00c2af0 0x7f71f00c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:06.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f4adb640 1 -- 172.21.15.83:0/3648105756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f00bd390 con 0x7f71f00c3590 2024-09-19T07:50:06.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f4adb640 1 --2- 172.21.15.83:0/3648105756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71f00c3590 0x7f71f014a790 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f71e000c520 tx=0x7f71e000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:06.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71e5ffb640 1 -- 172.21.15.83:0/3648105756 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e0018070 con 0x7f71f00c3590 2024-09-19T07:50:06.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71e5ffb640 1 -- 172.21.15.83:0/3648105756 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f71e00074e0 con 0x7f71f00c3590 2024-09-19T07:50:06.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f7540640 1 -- 172.21.15.83:0/3648105756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71f01effd0 con 0x7f71f00c3590 2024-09-19T07:50:06.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71e5ffb640 1 -- 172.21.15.83:0/3648105756 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e0004a10 con 0x7f71f00c3590 2024-09-19T07:50:06.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f7540640 1 -- 172.21.15.83:0/3648105756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71f00c72c0 con 0x7f71f00c3590 2024-09-19T07:50:06.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f7540640 1 -- 172.21.15.83:0/3648105756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71f00c6dd0 con 0x7f71f00c3590 2024-09-19T07:50:06.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71e5ffb640 1 -- 172.21.15.83:0/3648105756 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f71e001b070 con 0x7f71f00c3590 2024-09-19T07:50:06.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71e5ffb640 1 --2- 172.21.15.83:0/3648105756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71d40678c0 0x7f71d4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:06.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71e5ffb640 1 -- 172.21.15.83:0/3648105756 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10249+0+0 (secure 0 0 0) 0x7f71e00862a0 con 0x7f71f00c3590 2024-09-19T07:50:06.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.075+0000 7f71f52dc640 1 --2- 172.21.15.83:0/3648105756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71d40678c0 0x7f71d4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:06.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.079+0000 7f71f52dc640 1 --2- 172.21.15.83:0/3648105756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71d40678c0 0x7f71d4069d80 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f71ec00a890 tx=0x7f71ec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:06.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.079+0000 7f71e5ffb640 1 -- 172.21.15.83:0/3648105756 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f71e0055b40 con 0x7f71f00c3590 2024-09-19T07:50:06.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.263+0000 7f71f7540640 1 -- 172.21.15.83:0/3648105756 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "a", "yes_i_really_mean_it": true} v 0) -- 0x7f71f00c6770 con 0x7f71f00c3590 2024-09-19T07:50:06.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.887+0000 7f71e5ffb640 1 -- 172.21.15.83:0/3648105756 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "a", "yes_i_really_mean_it": true}]=0 v210) ==== 99+0+0 (secure 0 0 0) 0x7f71e0055440 con 0x7f71f00c3590 2024-09-19T07:50:06.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.891+0000 7f71f7540640 1 -- 172.21.15.83:0/3648105756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71d40678c0 msgr2=0x7f71d4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:06.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.891+0000 7f71f7540640 1 --2- 172.21.15.83:0/3648105756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71d40678c0 0x7f71d4069d80 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f71ec00a890 tx=0x7f71ec008040 comp rx=0 tx=0).stop 2024-09-19T07:50:06.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.891+0000 7f71f7540640 1 -- 172.21.15.83:0/3648105756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71f00c3590 msgr2=0x7f71f014a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:06.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.891+0000 7f71f7540640 1 --2- 172.21.15.83:0/3648105756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71f00c3590 0x7f71f014a790 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f71e000c520 tx=0x7f71e000c9e0 comp rx=0 tx=0).stop 2024-09-19T07:50:06.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.891+0000 7f71f7540640 1 -- 172.21.15.83:0/3648105756 shutdown_connections 2024-09-19T07:50:06.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.891+0000 7f71f7540640 1 --2- 172.21.15.83:0/3648105756 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f71d40678c0 0x7f71d4069d80 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:06.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.891+0000 7f71f7540640 1 --2- 172.21.15.83:0/3648105756 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71f0151130 0x7f71f014acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:06.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.891+0000 7f71f7540640 1 --2- 172.21.15.83:0/3648105756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71f00c3590 0x7f71f014a790 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:06.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.891+0000 7f71f7540640 1 --2- 172.21.15.83:0/3648105756 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71f00c2af0 0x7f71f00c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:06.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.891+0000 7f71f7540640 1 -- 172.21.15.83:0/3648105756 >> 172.21.15.83:0/3648105756 conn(0x7f71f006bc80 msgr2=0x7f71f006d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:06.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.891+0000 7f71f7540640 1 -- 172.21.15.83:0/3648105756 shutdown_connections 2024-09-19T07:50:06.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:06.891+0000 7f71f7540640 1 -- 172.21.15.83:0/3648105756 wait complete. 2024-09-19T07:50:06.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm a_metadata a_metadata --yes-i-really-really-mean-it 2024-09-19T07:50:07.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff142991640 1 Processor -- start 2024-09-19T07:50:07.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff142991640 1 -- start start 2024-09-19T07:50:07.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff142991640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff13c151330 0x7ff13c070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff142991640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff13c153d00 con 0x7ff13c0713e0 2024-09-19T07:50:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff142991640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff13c153fe0 con 0x7ff13c0a9d30 2024-09-19T07:50:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff142991640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff13c154910 con 0x7ff13c151330 2024-09-19T07:50:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff13bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff13c151330 0x7ff13c070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff13b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff13c0713e0 0x7ff13c06ddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48920/0 (socket says 172.21.15.83:48920) 2024-09-19T07:50:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff13bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff13c151330 0x7ff13c070ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47456/0 (socket says 172.21.15.83:47456) 2024-09-19T07:50:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff13bfff640 1 -- 172.21.15.83:0/3037987114 learned_addr learned my addr 172.21.15.83:0/3037987114 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff13bfff640 1 -- 172.21.15.83:0/3037987114 >> v1:172.21.15.184:6789/0 conn(0x7ff13c0a9d30 legacy=0x7ff13c06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff13bfff640 1 -- 172.21.15.83:0/3037987114 >> v1:172.21.15.83:6789/0 conn(0x7ff13c0713e0 legacy=0x7ff13c06ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff13affd640 1 -- 172.21.15.83:0/3037987114 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1750878704 0 0) 0x7ff13c153d00 con 0x7ff13c0713e0 2024-09-19T07:50:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.035+0000 7ff13bfff640 1 -- 172.21.15.83:0/3037987114 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13c1542c0 con 0x7ff13c151330 2024-09-19T07:50:07.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff13bfff640 1 --2- 172.21.15.83:0/3037987114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff13c151330 0x7ff13c070ea0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7ff12c009e50 tx=0x7ff12c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5440ad61f36ca159 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:07.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff13affd640 1 -- 172.21.15.83:0/3037987114 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c0a5780 con 0x7ff13c151330 2024-09-19T07:50:07.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff13affd640 1 -- 172.21.15.83:0/3037987114 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ff12c0a5920 con 0x7ff13c151330 2024-09-19T07:50:07.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff13affd640 1 -- 172.21.15.83:0/3037987114 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c0a5d10 con 0x7ff13c151330 2024-09-19T07:50:07.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff142991640 1 -- 172.21.15.83:0/3037987114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff13c151330 msgr2=0x7ff13c070ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:07.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff142991640 1 --2- 172.21.15.83:0/3037987114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff13c151330 0x7ff13c070ea0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7ff12c009e50 tx=0x7ff12c01c8a0 comp rx=0 tx=0).stop 2024-09-19T07:50:07.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff142991640 1 -- 172.21.15.83:0/3037987114 shutdown_connections 2024-09-19T07:50:07.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff142991640 1 --2- 172.21.15.83:0/3037987114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff13c151330 0x7ff13c070ea0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:07.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff142991640 1 -- 172.21.15.83:0/3037987114 >> 172.21.15.83:0/3037987114 conn(0x7ff13c06bc80 msgr2=0x7ff13c06c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:07.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff142991640 1 -- 172.21.15.83:0/3037987114 shutdown_connections 2024-09-19T07:50:07.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff142991640 1 -- 172.21.15.83:0/3037987114 wait complete. 2024-09-19T07:50:07.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff142991640 1 Processor -- start 2024-09-19T07:50:07.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.039+0000 7ff142991640 1 -- start start 2024-09-19T07:50:07.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff142991640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff13c0713e0 0x7ff13c14e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:07.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff142991640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff13c0a9d30 0x7ff13c152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:07.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff142991640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff13c151330 0x7ff13c144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:07.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff142991640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c161e40 con 0x7ff13c151330 2024-09-19T07:50:07.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff142991640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c161cc0 con 0x7ff13c0a9d30 2024-09-19T07:50:07.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff142991640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff13c161fc0 con 0x7ff13c0713e0 2024-09-19T07:50:07.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff140f2e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff13c151330 0x7ff13c144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff140f2e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff13c151330 0x7ff13c144700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47470/0 (socket says 172.21.15.83:47470) 2024-09-19T07:50:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff140f2e640 1 -- 172.21.15.83:0/3312012313 learned_addr learned my addr 172.21.15.83:0/3312012313 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff13bfff640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff13c0713e0 0x7ff13c14e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff140f2e640 1 -- 172.21.15.83:0/3312012313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff13c0713e0 msgr2=0x7ff13c14e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff13b7fe640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff13c0a9d30 0x7ff13c152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff140f2e640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff13c0713e0 0x7ff13c14e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff140f2e640 1 -- 172.21.15.83:0/3312012313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff13c0a9d30 msgr2=0x7ff13c152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff140f2e640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff13c0a9d30 0x7ff13c152ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff140f2e640 1 -- 172.21.15.83:0/3312012313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13c14dd20 con 0x7ff13c151330 2024-09-19T07:50:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff13bfff640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff13c0713e0 0x7ff13c14e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff140f2e640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff13c151330 0x7ff13c144700 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7ff13400c3f0 tx=0x7ff13400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff138ff9640 1 -- 172.21.15.83:0/3312012313 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff134011790 con 0x7ff13c151330 2024-09-19T07:50:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff138ff9640 1 -- 172.21.15.83:0/3312012313 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ff134017070 con 0x7ff13c151330 2024-09-19T07:50:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff142991640 1 -- 172.21.15.83:0/3312012313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13c14c190 con 0x7ff13c151330 2024-09-19T07:50:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff138ff9640 1 -- 172.21.15.83:0/3312012313 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff134011b90 con 0x7ff13c151330 2024-09-19T07:50:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff142991640 1 -- 172.21.15.83:0/3312012313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff13c14c430 con 0x7ff13c151330 2024-09-19T07:50:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff142991640 1 -- 172.21.15.83:0/3312012313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff13c070ea0 con 0x7ff13c151330 2024-09-19T07:50:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff138ff9640 1 -- 172.21.15.83:0/3312012313 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff13401c460 con 0x7ff13c151330 2024-09-19T07:50:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff138ff9640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1280678c0 0x7ff128069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff138ff9640 1 -- 172.21.15.83:0/3312012313 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10249+0+0 (secure 0 0 0) 0x7ff134086d70 con 0x7ff13c151330 2024-09-19T07:50:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff13bfff640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1280678c0 0x7ff128069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.043+0000 7ff13bfff640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1280678c0 0x7ff128069d80 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7ff12c0ae000 tx=0x7ff12c0a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.047+0000 7ff138ff9640 1 -- 172.21.15.83:0/3312012313 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff134056610 con 0x7ff13c151330 2024-09-19T07:50:07.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.199+0000 7ff142991640 1 -- 172.21.15.83:0/3312012313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "a_metadata", "pool2": "a_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff13c15f1c0 con 0x7ff13c151330 2024-09-19T07:50:07.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.899+0000 7ff138ff9640 1 -- 172.21.15.83:0/3312012313 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "a_metadata", "pool2": "a_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'a_metadata' removed v68) ==== 166+0+0 (secure 0 0 0) 0x7ff134055f10 con 0x7ff13c151330 2024-09-19T07:50:07.895 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'a_metadata' removed 2024-09-19T07:50:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.899+0000 7ff142991640 1 -- 172.21.15.83:0/3312012313 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1280678c0 msgr2=0x7ff128069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.899+0000 7ff142991640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1280678c0 0x7ff128069d80 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7ff12c0ae000 tx=0x7ff12c0a7070 comp rx=0 tx=0).stop 2024-09-19T07:50:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.899+0000 7ff142991640 1 -- 172.21.15.83:0/3312012313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff13c151330 msgr2=0x7ff13c144700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.899+0000 7ff142991640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff13c151330 0x7ff13c144700 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7ff13400c3f0 tx=0x7ff13400c8b0 comp rx=0 tx=0).stop 2024-09-19T07:50:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.899+0000 7ff142991640 1 -- 172.21.15.83:0/3312012313 shutdown_connections 2024-09-19T07:50:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.899+0000 7ff142991640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff1280678c0 0x7ff128069d80 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.899+0000 7ff142991640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff13c151330 0x7ff13c144700 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.899+0000 7ff142991640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff13c0a9d30 0x7ff13c152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.899+0000 7ff142991640 1 --2- 172.21.15.83:0/3312012313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff13c0713e0 0x7ff13c14e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.899+0000 7ff142991640 1 -- 172.21.15.83:0/3312012313 >> 172.21.15.83:0/3312012313 conn(0x7ff13c06bc80 msgr2=0x7ff13c073100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.903+0000 7ff142991640 1 -- 172.21.15.83:0/3312012313 shutdown_connections 2024-09-19T07:50:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:07.903+0000 7ff142991640 1 -- 172.21.15.83:0/3312012313 wait complete. 2024-09-19T07:50:07.921 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm a_data a_data --yes-i-really-really-mean-it 2024-09-19T07:50:07.974 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:50:08.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.043+0000 7ff63225e640 1 Processor -- start 2024-09-19T07:50:08.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.043+0000 7ff63225e640 1 -- start start 2024-09-19T07:50:08.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.043+0000 7ff63225e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62c06cef0 0x7ff62c1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.043+0000 7ff63225e640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff62c0c3000 0x7ff62c150d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.043+0000 7ff63225e640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff62c070390 con 0x7ff62c06f560 2024-09-19T07:50:08.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.043+0000 7ff63225e640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff62c0708e0 con 0x7ff62c0c3000 2024-09-19T07:50:08.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.043+0000 7ff63225e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff62c153f10 con 0x7ff62c06cef0 2024-09-19T07:50:08.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.043+0000 7ff62b7fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62c06cef0 0x7ff62c1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.043+0000 7ff62bfff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff62c0c3000 0x7ff62c150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.043+0000 7ff62affd640 1 --1- >> v1:172.21.15.184:6789/0 conn(0x7ff62c06f560 0x7ff62c1543a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.184:6789/0 says I am v1:172.21.15.83:49666/0 (socket says 172.21.15.83:49666) 2024-09-19T07:50:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.043+0000 7ff62affd640 1 -- 172.21.15.83:0/3258760793 learned_addr learned my addr 172.21.15.83:0/3258760793 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:50:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff62b7fe640 1 -- 172.21.15.83:0/3258760793 >> v2:172.21.15.184:3300/0 conn(0x7ff62c0c3000 msgr2=0x7ff62c150d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff62b7fe640 1 --2- 172.21.15.83:0/3258760793 >> v2:172.21.15.184:3300/0 conn(0x7ff62c0c3000 0x7ff62c150d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff62b7fe640 1 -- 172.21.15.83:0/3258760793 >> v1:172.21.15.184:6789/0 conn(0x7ff62c06f560 legacy=0x7ff62c1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff62b7fe640 1 -- 172.21.15.83:0/3258760793 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62c070600 con 0x7ff62c06cef0 2024-09-19T07:50:08.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff62bfff640 1 --2- 172.21.15.83:0/3258760793 >> v2:172.21.15.184:3300/0 conn(0x7ff62c0c3000 0x7ff62c150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T07:50:08.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff62b7fe640 1 --2- 172.21.15.83:0/3258760793 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62c06cef0 0x7ff62c1579a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff624009d90 tx=0x7ff62401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc97ce4432728289 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:08.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff62a7fc640 1 -- 172.21.15.83:0/3258760793 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6240b2070 con 0x7ff62c06cef0 2024-09-19T07:50:08.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff62a7fc640 1 -- 172.21.15.83:0/3258760793 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ff6240a5830 con 0x7ff62c06cef0 2024-09-19T07:50:08.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff62a7fc640 1 -- 172.21.15.83:0/3258760793 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6240a5c20 con 0x7ff62c06cef0 2024-09-19T07:50:08.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff63225e640 1 -- 172.21.15.83:0/3258760793 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62c06cef0 msgr2=0x7ff62c1579a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:08.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff63225e640 1 --2- 172.21.15.83:0/3258760793 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62c06cef0 0x7ff62c1579a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff624009d90 tx=0x7ff62401c970 comp rx=0 tx=0).stop 2024-09-19T07:50:08.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff63225e640 1 -- 172.21.15.83:0/3258760793 shutdown_connections 2024-09-19T07:50:08.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff63225e640 1 --2- 172.21.15.83:0/3258760793 >> v2:172.21.15.184:3300/0 conn(0x7ff62c0c3000 0x7ff62c150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff63225e640 1 --2- 172.21.15.83:0/3258760793 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62c06cef0 0x7ff62c1579a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff63225e640 1 -- 172.21.15.83:0/3258760793 >> 172.21.15.83:0/3258760793 conn(0x7ff62c0c5100 msgr2=0x7ff62c0c5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:08.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff63225e640 1 -- 172.21.15.83:0/3258760793 shutdown_connections 2024-09-19T07:50:08.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.047+0000 7ff63225e640 1 -- 172.21.15.83:0/3258760793 wait complete. 2024-09-19T07:50:08.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff63225e640 1 Processor -- start 2024-09-19T07:50:08.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff63225e640 1 -- start start 2024-09-19T07:50:08.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff63225e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62c06cef0 0x7ff62c1567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff63225e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff62c06f560 0x7ff62c149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff63225e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62c0c3000 0x7ff62c1471e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff63225e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c06d450 con 0x7ff62c06cef0 2024-09-19T07:50:08.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff63225e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c06d2d0 con 0x7ff62c06f560 2024-09-19T07:50:08.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff63225e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff62c160b40 con 0x7ff62c0c3000 2024-09-19T07:50:08.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff62b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62c06cef0 0x7ff62c1567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff62b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62c06cef0 0x7ff62c1567b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47484/0 (socket says 172.21.15.83:47484) 2024-09-19T07:50:08.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff62b7fe640 1 -- 172.21.15.83:0/3521118206 learned_addr learned my addr 172.21.15.83:0/3521118206 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:08.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff62affd640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff62c06f560 0x7ff62c149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff62bfff640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62c0c3000 0x7ff62c1471e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff62b7fe640 1 -- 172.21.15.83:0/3521118206 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62c0c3000 msgr2=0x7ff62c1471e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:08.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff62b7fe640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62c0c3000 0x7ff62c1471e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff62b7fe640 1 -- 172.21.15.83:0/3521118206 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff62c06f560 msgr2=0x7ff62c149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:08.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff62b7fe640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff62c06f560 0x7ff62c149eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff62b7fe640 1 -- 172.21.15.83:0/3521118206 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62c157220 con 0x7ff62c06cef0 2024-09-19T07:50:08.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff62bfff640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62c0c3000 0x7ff62c1471e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:08.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff62b7fe640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62c06cef0 0x7ff62c1567b0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7ff62401a5a0 tx=0x7ff624019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:08.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff623fff640 1 -- 172.21.15.83:0/3521118206 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6240b2050 con 0x7ff62c06cef0 2024-09-19T07:50:08.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff623fff640 1 -- 172.21.15.83:0/3521118206 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ff6240be070 con 0x7ff62c06cef0 2024-09-19T07:50:08.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff63225e640 1 -- 172.21.15.83:0/3521118206 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff62c157500 con 0x7ff62c06cef0 2024-09-19T07:50:08.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff623fff640 1 -- 172.21.15.83:0/3521118206 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6240adab0 con 0x7ff62c06cef0 2024-09-19T07:50:08.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff63225e640 1 -- 172.21.15.83:0/3521118206 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff62c14afa0 con 0x7ff62c06cef0 2024-09-19T07:50:08.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.051+0000 7ff63225e640 1 -- 172.21.15.83:0/3521118206 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff62c150d10 con 0x7ff62c06cef0 2024-09-19T07:50:08.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.055+0000 7ff623fff640 1 -- 172.21.15.83:0/3521118206 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff6240adc50 con 0x7ff62c06cef0 2024-09-19T07:50:08.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.055+0000 7ff623fff640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff60c0678c0 0x7ff60c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.055+0000 7ff623fff640 1 -- 172.21.15.83:0/3521118206 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 9821+0+0 (secure 0 0 0) 0x7ff624121f80 con 0x7ff62c06cef0 2024-09-19T07:50:08.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.055+0000 7ff62affd640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff60c0678c0 0x7ff60c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.055+0000 7ff62affd640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff60c0678c0 0x7ff60c069d80 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff614000f70 tx=0x7ff614009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:08.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.059+0000 7ff623fff640 1 -- 172.21.15.83:0/3521118206 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff624016560 con 0x7ff62c06cef0 2024-09-19T07:50:08.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.103+0000 7ff536fa2640 1 Processor -- start 2024-09-19T07:50:08.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.103+0000 7ff536fa2640 1 -- start start 2024-09-19T07:50:08.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7ff53006cef0 0x7ff5300728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff5300c4990 0x7ff53014fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff530150140 0x7ff530156070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7ff5300c3e20 con 0x7ff5300c4990 2024-09-19T07:50:08.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7ff5300c2f90 con 0x7ff53006cef0 2024-09-19T07:50:08.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff5300c3210 con 0x7ff530150140 2024-09-19T07:50:08.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff52ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff5300c4990 0x7ff53014fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff52ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff5300c4990 0x7ff53014fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47500/0 (socket says 172.21.15.83:47500) 2024-09-19T07:50:08.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff52ffff640 1 -- 172.21.15.83:0/1680578954 learned_addr learned my addr 172.21.15.83:0/1680578954 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:08.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff534d3e640 1 --2- 172.21.15.83:0/1680578954 >> v2:172.21.15.184:3300/0 conn(0x7ff53006cef0 0x7ff5300728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff53553f640 1 --2- 172.21.15.83:0/1680578954 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff530150140 0x7ff530156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff52ffff640 1 -- 172.21.15.83:0/1680578954 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff530150140 msgr2=0x7ff530156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:08.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff52ffff640 1 --2- 172.21.15.83:0/1680578954 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff530150140 0x7ff530156070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff52ffff640 1 -- 172.21.15.83:0/1680578954 >> v2:172.21.15.184:3300/0 conn(0x7ff53006cef0 msgr2=0x7ff5300728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:08.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff52ffff640 1 --2- 172.21.15.83:0/1680578954 >> v2:172.21.15.184:3300/0 conn(0x7ff53006cef0 0x7ff5300728d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff52ffff640 1 -- 172.21.15.83:0/1680578954 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5300c34f0 con 0x7ff5300c4990 2024-09-19T07:50:08.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff534d3e640 1 --2- 172.21.15.83:0/1680578954 >> v2:172.21.15.184:3300/0 conn(0x7ff53006cef0 0x7ff5300728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:08.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff52ffff640 1 --2- 172.21.15.83:0/1680578954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5300c4990 0x7ff53014fc00 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7ff53006d800 tx=0x7ff52401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79349654f222b824 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:08.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff52f7fe640 1 -- 172.21.15.83:0/1680578954 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5240b2070 con 0x7ff5300c4990 2024-09-19T07:50:08.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff52f7fe640 1 -- 172.21.15.83:0/1680578954 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ff5240a9070 con 0x7ff5300c4990 2024-09-19T07:50:08.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff52f7fe640 1 -- 172.21.15.83:0/1680578954 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5240ad8a0 con 0x7ff5300c4990 2024-09-19T07:50:08.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 -- 172.21.15.83:0/1680578954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5300c4990 msgr2=0x7ff53014fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:08.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 --2- 172.21.15.83:0/1680578954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5300c4990 0x7ff53014fc00 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7ff53006d800 tx=0x7ff52401c490 comp rx=0 tx=0).stop 2024-09-19T07:50:08.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 -- 172.21.15.83:0/1680578954 shutdown_connections 2024-09-19T07:50:08.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 --2- 172.21.15.83:0/1680578954 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff530150140 0x7ff530156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 --2- 172.21.15.83:0/1680578954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5300c4990 0x7ff53014fc00 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 --2- 172.21.15.83:0/1680578954 >> v2:172.21.15.184:3300/0 conn(0x7ff53006cef0 0x7ff5300728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 -- 172.21.15.83:0/1680578954 >> 172.21.15.83:0/1680578954 conn(0x7ff5300c5090 msgr2=0x7ff5300c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:08.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 -- 172.21.15.83:0/1680578954 shutdown_connections 2024-09-19T07:50:08.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.107+0000 7ff536fa2640 1 -- 172.21.15.83:0/1680578954 wait complete. 2024-09-19T07:50:08.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff536fa2640 1 Processor -- start 2024-09-19T07:50:08.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff536fa2640 1 -- start start 2024-09-19T07:50:08.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff536fa2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff53006cef0 0x7ff5301496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff536fa2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5300c4990 0x7ff530142ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff536fa2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530150140 0x7ff530143220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff536fa2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff530152880 con 0x7ff530150140 2024-09-19T07:50:08.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff536fa2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff53006d520 con 0x7ff53006cef0 2024-09-19T07:50:08.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff536fa2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff530152a00 con 0x7ff5300c4990 2024-09-19T07:50:08.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff53553f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530150140 0x7ff530143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff53553f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530150140 0x7ff530143220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47514/0 (socket says 172.21.15.83:47514) 2024-09-19T07:50:08.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff53553f640 1 -- 172.21.15.83:0/1454452220 learned_addr learned my addr 172.21.15.83:0/1454452220 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:08.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff534d3e640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff53006cef0 0x7ff5301496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff52ffff640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5300c4990 0x7ff530142ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff53553f640 1 -- 172.21.15.83:0/1454452220 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5300c4990 msgr2=0x7ff530142ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:08.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff53553f640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5300c4990 0x7ff530142ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff53553f640 1 -- 172.21.15.83:0/1454452220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff53006cef0 msgr2=0x7ff5301496d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff53553f640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff53006cef0 0x7ff5301496d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff53553f640 1 -- 172.21.15.83:0/1454452220 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5301ec8a0 con 0x7ff530150140 2024-09-19T07:50:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff53553f640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530150140 0x7ff530143220 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7ff52800ccb0 tx=0x7ff52800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff52d7fa640 1 -- 172.21.15.83:0/1454452220 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52800ce60 con 0x7ff530150140 2024-09-19T07:50:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff52d7fa640 1 -- 172.21.15.83:0/1454452220 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ff5280039a0 con 0x7ff530150140 2024-09-19T07:50:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff536fa2640 1 -- 172.21.15.83:0/1454452220 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5301ecb40 con 0x7ff530150140 2024-09-19T07:50:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff52d7fa640 1 -- 172.21.15.83:0/1454452220 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528003d90 con 0x7ff530150140 2024-09-19T07:50:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff536fa2640 1 -- 172.21.15.83:0/1454452220 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5301ed100 con 0x7ff530150140 2024-09-19T07:50:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff536fa2640 1 -- 172.21.15.83:0/1454452220 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff53014fc00 con 0x7ff530150140 2024-09-19T07:50:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.111+0000 7ff52d7fa640 1 -- 172.21.15.83:0/1454452220 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff528003f50 con 0x7ff530150140 2024-09-19T07:50:08.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.115+0000 7ff52d7fa640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff51c0678c0 0x7ff51c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:08.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.115+0000 7ff52d7fa640 1 -- 172.21.15.83:0/1454452220 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 9821+0+0 (secure 0 0 0) 0x7ff528086fc0 con 0x7ff530150140 2024-09-19T07:50:08.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.115+0000 7ff534d3e640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff51c0678c0 0x7ff51c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:08.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.115+0000 7ff534d3e640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff51c0678c0 0x7ff51c069d80 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ff5301466e0 tx=0x7ff520008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:08.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.119+0000 7ff52d7fa640 1 -- 172.21.15.83:0/1454452220 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7ff528014030 con 0x7ff530150140 2024-09-19T07:50:08.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.215+0000 7ff63225e640 1 -- 172.21.15.83:0/3521118206 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "a_data", "pool2": "a_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff62c151e60 con 0x7ff62c06cef0 2024-09-19T07:50:08.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.323+0000 7ff536fa2640 1 -- 172.21.15.83:0/1454452220 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff530143780 con 0x7ff530150140 2024-09-19T07:50:08.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.323+0000 7ff52d7fa640 1 -- 172.21.15.83:0/1454452220 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) ==== 95+0+4404 (secure 0 0 0) 0x7ff528056b30 con 0x7ff530150140 2024-09-19T07:50:08.321 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:50:08.321 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":210,"btime":"2024-09-19T07:50:06:886590+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/368086462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":368086462},{"type":"v1","addr":"172.21.15.184:6833","nonce":368086462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:50:08.323 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 210 2024-09-19T07:50:08.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.323+0000 7ff536fa2640 1 -- 172.21.15.83:0/1454452220 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff51c0678c0 msgr2=0x7ff51c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:08.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.323+0000 7ff536fa2640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff51c0678c0 0x7ff51c069d80 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ff5301466e0 tx=0x7ff520008040 comp rx=0 tx=0).stop 2024-09-19T07:50:08.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.327+0000 7ff536fa2640 1 -- 172.21.15.83:0/1454452220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530150140 msgr2=0x7ff530143220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:08.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.327+0000 7ff536fa2640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530150140 0x7ff530143220 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7ff52800ccb0 tx=0x7ff52800a5c0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.327+0000 7ff536fa2640 1 -- 172.21.15.83:0/1454452220 shutdown_connections 2024-09-19T07:50:08.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.327+0000 7ff536fa2640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff51c0678c0 0x7ff51c069d80 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.327+0000 7ff536fa2640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530150140 0x7ff530143220 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.327+0000 7ff536fa2640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5300c4990 0x7ff530142ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.327+0000 7ff536fa2640 1 --2- 172.21.15.83:0/1454452220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff53006cef0 0x7ff5301496d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.327+0000 7ff536fa2640 1 -- 172.21.15.83:0/1454452220 >> 172.21.15.83:0/1454452220 conn(0x7ff5300c5090 msgr2=0x7ff53006fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:08.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.327+0000 7ff536fa2640 1 -- 172.21.15.83:0/1454452220 shutdown_connections 2024-09-19T07:50:08.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.327+0000 7ff536fa2640 1 -- 172.21.15.83:0/1454452220 wait complete. 2024-09-19T07:50:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.911+0000 7ff623fff640 1 -- 172.21.15.83:0/3521118206 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "a_data", "pool2": "a_data", "yes_i_really_really_mean_it": true}]=0 pool 'a_data' removed v69) ==== 154+0+0 (secure 0 0 0) 0x7ff6240ab340 con 0x7ff62c06cef0 2024-09-19T07:50:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'a_data' removed 2024-09-19T07:50:08.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.915+0000 7ff63225e640 1 -- 172.21.15.83:0/3521118206 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff60c0678c0 msgr2=0x7ff60c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:08.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.915+0000 7ff63225e640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff60c0678c0 0x7ff60c069d80 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff614000f70 tx=0x7ff614009040 comp rx=0 tx=0).stop 2024-09-19T07:50:08.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.915+0000 7ff63225e640 1 -- 172.21.15.83:0/3521118206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62c06cef0 msgr2=0x7ff62c1567b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:08.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.915+0000 7ff63225e640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62c06cef0 0x7ff62c1567b0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7ff62401a5a0 tx=0x7ff624019710 comp rx=0 tx=0).stop 2024-09-19T07:50:08.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.915+0000 7ff63225e640 1 -- 172.21.15.83:0/3521118206 shutdown_connections 2024-09-19T07:50:08.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.915+0000 7ff63225e640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7ff60c0678c0 0x7ff60c069d80 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.915+0000 7ff63225e640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff62c0c3000 0x7ff62c1471e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.915+0000 7ff63225e640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff62c06f560 0x7ff62c149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.915+0000 7ff63225e640 1 --2- 172.21.15.83:0/3521118206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff62c06cef0 0x7ff62c1567b0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:08.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.915+0000 7ff63225e640 1 -- 172.21.15.83:0/3521118206 >> 172.21.15.83:0/3521118206 conn(0x7ff62c0c5100 msgr2=0x7ff62c150160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:08.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.915+0000 7ff63225e640 1 -- 172.21.15.83:0/3521118206 shutdown_connections 2024-09-19T07:50:08.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:08.915+0000 7ff63225e640 1 -- 172.21.15.83:0/3521118206 wait complete. 2024-09-19T07:50:08.943 INFO:tasks.cephfs.filesystem:Destroying file system b and related pools 2024-09-19T07:50:08.944 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:50:09.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2c447640 1 Processor -- start 2024-09-19T07:50:09.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2c447640 1 -- start start 2024-09-19T07:50:09.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2c447640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea24151140 0x7fea2406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2c447640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fea24073e20 0x7fea24157260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2c447640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fea240c3010 0x7fea241577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2c447640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fea240c38e0 con 0x7fea240c3010 2024-09-19T07:50:09.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2c447640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fea24151520 con 0x7fea24073e20 2024-09-19T07:50:09.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2c447640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fea24153890 con 0x7fea24151140 2024-09-19T07:50:09.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2a9e4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fea240c3010 0x7fea241577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2a9e4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fea240c3010 0x7fea241577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47530/0 (socket says 172.21.15.83:47530) 2024-09-19T07:50:09.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2a9e4640 1 -- 172.21.15.83:0/2681842958 learned_addr learned my addr 172.21.15.83:0/2681842958 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:09.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea299e2640 1 --2- 172.21.15.83:0/2681842958 >> v2:172.21.15.184:3300/0 conn(0x7fea24073e20 0x7fea24157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2a9e4640 1 -- 172.21.15.83:0/2681842958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea24151140 msgr2=0x7fea2406ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:09.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2a9e4640 1 --2- 172.21.15.83:0/2681842958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea24151140 0x7fea2406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2a9e4640 1 -- 172.21.15.83:0/2681842958 >> v2:172.21.15.184:3300/0 conn(0x7fea24073e20 msgr2=0x7fea24157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:09.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2a9e4640 1 --2- 172.21.15.83:0/2681842958 >> v2:172.21.15.184:3300/0 conn(0x7fea24073e20 0x7fea24157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.135+0000 7fea2a9e4640 1 -- 172.21.15.83:0/2681842958 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea240c3600 con 0x7fea240c3010 2024-09-19T07:50:09.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.139+0000 7fea2a9e4640 1 --2- 172.21.15.83:0/2681842958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea240c3010 0x7fea241577a0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fea2406fb50 tx=0x7fea1c01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76f3b6d71940618 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:09.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.139+0000 7fea291e1640 1 -- 172.21.15.83:0/2681842958 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c0a5550 con 0x7fea240c3010 2024-09-19T07:50:09.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.139+0000 7fea291e1640 1 -- 172.21.15.83:0/2681842958 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fea1c0a56f0 con 0x7fea240c3010 2024-09-19T07:50:09.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.139+0000 7fea291e1640 1 -- 172.21.15.83:0/2681842958 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c0a5ae0 con 0x7fea240c3010 2024-09-19T07:50:09.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.143+0000 7fea2c447640 1 -- 172.21.15.83:0/2681842958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea240c3010 msgr2=0x7fea241577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.143+0000 7fea2c447640 1 --2- 172.21.15.83:0/2681842958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea240c3010 0x7fea241577a0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fea2406fb50 tx=0x7fea1c01c8d0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.143+0000 7fea2c447640 1 -- 172.21.15.83:0/2681842958 shutdown_connections 2024-09-19T07:50:09.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.143+0000 7fea2c447640 1 --2- 172.21.15.83:0/2681842958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea240c3010 0x7fea241577a0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.143+0000 7fea2c447640 1 --2- 172.21.15.83:0/2681842958 >> v2:172.21.15.184:3300/0 conn(0x7fea24073e20 0x7fea24157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.143+0000 7fea2c447640 1 --2- 172.21.15.83:0/2681842958 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea24151140 0x7fea2406ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.143+0000 7fea2c447640 1 -- 172.21.15.83:0/2681842958 >> 172.21.15.83:0/2681842958 conn(0x7fea240c87f0 msgr2=0x7fea240c8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:09.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.143+0000 7fea2c447640 1 -- 172.21.15.83:0/2681842958 shutdown_connections 2024-09-19T07:50:09.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.143+0000 7fea2c447640 1 -- 172.21.15.83:0/2681842958 wait complete. 2024-09-19T07:50:09.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.143+0000 7fea2c447640 1 Processor -- start 2024-09-19T07:50:09.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.143+0000 7fea2c447640 1 -- start start 2024-09-19T07:50:09.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2c447640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea24073e20 0x7fea24155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2c447640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea240c3010 0x7fea241f8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2c447640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea24151140 0x7fea241f8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2c447640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fea240c4610 con 0x7fea240c3010 2024-09-19T07:50:09.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2a9e4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea24151140 0x7fea241f8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2a9e4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea24151140 0x7fea241f8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.83:55930/0 (socket says 172.21.15.83:55930) 2024-09-19T07:50:09.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2a9e4640 1 -- 172.21.15.83:0/95374952 learned_addr learned my addr 172.21.15.83:0/95374952 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:09.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2a1e3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea24073e20 0x7fea24155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2c447640 1 -- 172.21.15.83:0/95374952 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fea240c4490 con 0x7fea24073e20 2024-09-19T07:50:09.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2c447640 1 -- 172.21.15.83:0/95374952 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fea240757a0 con 0x7fea24151140 2024-09-19T07:50:09.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea299e2640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea240c3010 0x7fea241f8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2a9e4640 1 -- 172.21.15.83:0/95374952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea24073e20 msgr2=0x7fea24155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2a9e4640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea24073e20 0x7fea24155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2a9e4640 1 -- 172.21.15.83:0/95374952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea240c3010 msgr2=0x7fea241f8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2a9e4640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea240c3010 0x7fea241f8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2a9e4640 1 -- 172.21.15.83:0/95374952 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2415ea10 con 0x7fea24151140 2024-09-19T07:50:09.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea299e2640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea240c3010 0x7fea241f8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:09.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2a9e4640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea24151140 0x7fea241f8960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0af040 tx=0x7fea1c019670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:09.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea1affd640 1 -- 172.21.15.83:0/95374952 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c0bc020 con 0x7fea24151140 2024-09-19T07:50:09.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea1affd640 1 -- 172.21.15.83:0/95374952 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fea1c0b8670 con 0x7fea24151140 2024-09-19T07:50:09.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea1affd640 1 -- 172.21.15.83:0/95374952 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c009510 con 0x7fea24151140 2024-09-19T07:50:09.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2c447640 1 -- 172.21.15.83:0/95374952 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2415e370 con 0x7fea24151140 2024-09-19T07:50:09.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.147+0000 7fea2c447640 1 -- 172.21.15.83:0/95374952 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea241fbc00 con 0x7fea24151140 2024-09-19T07:50:09.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.151+0000 7fea2c447640 1 -- 172.21.15.83:0/95374952 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea24075ba0 con 0x7fea24151140 2024-09-19T07:50:09.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.155+0000 7fea1affd640 1 -- 172.21.15.83:0/95374952 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fea1c0bf020 con 0x7fea24151140 2024-09-19T07:50:09.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.155+0000 7fea1affd640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fea08067910 0x7fea08069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.155+0000 7fea2a1e3640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fea08067910 0x7fea08069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.159+0000 7fea2a1e3640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fea08067910 0x7fea08069dd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fea200098f0 tx=0x7fea20006f90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:09.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.159+0000 7fea1affd640 1 -- 172.21.15.83:0/95374952 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(69..69 src has 1..69) ==== 9301+0+0 (secure 0 0 0) 0x7fea1c0ef6f0 con 0x7fea24151140 2024-09-19T07:50:09.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.175+0000 7fea1affd640 1 -- 172.21.15.83:0/95374952 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fea1c0c6030 con 0x7fea24151140 2024-09-19T07:50:09.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.371+0000 7fea2c447640 1 -- 172.21.15.83:0/95374952 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea2406c1f0 con 0x7fea24151140 2024-09-19T07:50:09.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.371+0000 7fea1affd640 1 -- 172.21.15.83:0/95374952 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) ==== 95+0+4404 (secure 0 0 0) 0x7fea1c0165e0 con 0x7fea24151140 2024-09-19T07:50:09.369 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:50:09.369 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":210,"btime":"2024-09-19T07:50:06:886590+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/368086462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":368086462},{"type":"v1","addr":"172.21.15.184:6833","nonce":368086462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:50:09.369 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 210 2024-09-19T07:50:09.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.375+0000 7fea2c447640 1 -- 172.21.15.83:0/95374952 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fea08067910 msgr2=0x7fea08069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.375+0000 7fea2c447640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fea08067910 0x7fea08069dd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fea200098f0 tx=0x7fea20006f90 comp rx=0 tx=0).stop 2024-09-19T07:50:09.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.375+0000 7fea2c447640 1 -- 172.21.15.83:0/95374952 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea24151140 msgr2=0x7fea241f8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.375+0000 7fea2c447640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea24151140 0x7fea241f8960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0af040 tx=0x7fea1c019670 comp rx=0 tx=0).stop 2024-09-19T07:50:09.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.375+0000 7fea2c447640 1 -- 172.21.15.83:0/95374952 shutdown_connections 2024-09-19T07:50:09.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.375+0000 7fea2c447640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fea08067910 0x7fea08069dd0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.375+0000 7fea2c447640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fea24151140 0x7fea241f8960 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.375+0000 7fea2c447640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea240c3010 0x7fea241f8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.375+0000 7fea2c447640 1 --2- 172.21.15.83:0/95374952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fea24073e20 0x7fea24155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.375+0000 7fea2c447640 1 -- 172.21.15.83:0/95374952 >> 172.21.15.83:0/95374952 conn(0x7fea240c87f0 msgr2=0x7fea241570b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:09.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.375+0000 7fea2c447640 1 -- 172.21.15.83:0/95374952 shutdown_connections 2024-09-19T07:50:09.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.375+0000 7fea2c447640 1 -- 172.21.15.83:0/95374952 wait complete. 2024-09-19T07:50:09.388 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-19T07:50:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce74df2640 1 Processor -- start 2024-09-19T07:50:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce74df2640 1 -- start start 2024-09-19T07:50:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce74df2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fce700c42e0 0x7fce7006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce74df2640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fce701566b0 0x7fce700abb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce74df2640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce701592c0 con 0x7fce70159a80 2024-09-19T07:50:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce74df2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fce700ac070 con 0x7fce700c42e0 2024-09-19T07:50:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce74df2640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fce701595f0 con 0x7fce701566b0 2024-09-19T07:50:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce6e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fce700c42e0 0x7fce7006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce6e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fce700c42e0 0x7fce7006e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47542/0 (socket says 172.21.15.83:47542) 2024-09-19T07:50:09.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce6e59c640 1 -- 172.21.15.83:0/2535362595 learned_addr learned my addr 172.21.15.83:0/2535362595 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:09.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce6dd9b640 1 --2- 172.21.15.83:0/2535362595 >> v2:172.21.15.184:3300/0 conn(0x7fce701566b0 0x7fce700abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce6e59c640 1 -- 172.21.15.83:0/2535362595 >> v2:172.21.15.184:3300/0 conn(0x7fce701566b0 msgr2=0x7fce700abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:09.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce6e59c640 1 --2- 172.21.15.83:0/2535362595 >> v2:172.21.15.184:3300/0 conn(0x7fce701566b0 0x7fce700abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce6e59c640 1 -- 172.21.15.83:0/2535362595 >> v1:172.21.15.184:6789/0 conn(0x7fce70159a80 legacy=0x7fce70153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:09.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.515+0000 7fce6e59c640 1 -- 172.21.15.83:0/2535362595 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7006e900 con 0x7fce700c42e0 2024-09-19T07:50:09.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6e59c640 1 --2- 172.21.15.83:0/2535362595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce700c42e0 0x7fce7006e270 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fce700c51b0 tx=0x7fce6401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657e5a5cfec5663c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:09.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6d59a640 1 -- 172.21.15.83:0/2535362595 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce640aa610 con 0x7fce700c42e0 2024-09-19T07:50:09.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6d59a640 1 -- 172.21.15.83:0/2535362595 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fce640af070 con 0x7fce700c42e0 2024-09-19T07:50:09.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6d59a640 1 -- 172.21.15.83:0/2535362595 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce640aa9f0 con 0x7fce700c42e0 2024-09-19T07:50:09.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 -- 172.21.15.83:0/2535362595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce700c42e0 msgr2=0x7fce7006e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 --2- 172.21.15.83:0/2535362595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce700c42e0 0x7fce7006e270 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fce700c51b0 tx=0x7fce6401c490 comp rx=0 tx=0).stop 2024-09-19T07:50:09.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 -- 172.21.15.83:0/2535362595 shutdown_connections 2024-09-19T07:50:09.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 --2- 172.21.15.83:0/2535362595 >> v2:172.21.15.184:3300/0 conn(0x7fce701566b0 0x7fce700abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 --2- 172.21.15.83:0/2535362595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce700c42e0 0x7fce7006e270 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 -- 172.21.15.83:0/2535362595 >> 172.21.15.83:0/2535362595 conn(0x7fce7006bc80 msgr2=0x7fce7006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:09.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 -- 172.21.15.83:0/2535362595 shutdown_connections 2024-09-19T07:50:09.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 -- 172.21.15.83:0/2535362595 wait complete. 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 Processor -- start 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 -- start start 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce700c42e0 0x7fce700c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce701566b0 0x7fce7014c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce70159a80 0x7fce7014cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fce701538f0 con 0x7fce701566b0 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fce70153770 con 0x7fce700c42e0 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fce700c46c0 con 0x7fce70159a80 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce701566b0 0x7fce7014c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce701566b0 0x7fce7014c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47558/0 (socket says 172.21.15.83:47558) 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6dd9b640 1 -- 172.21.15.83:0/3025689942 learned_addr learned my addr 172.21.15.83:0/3025689942 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6e59c640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce700c42e0 0x7fce700c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6ed9d640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce70159a80 0x7fce7014cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6dd9b640 1 -- 172.21.15.83:0/3025689942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce70159a80 msgr2=0x7fce7014cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6dd9b640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce70159a80 0x7fce7014cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6dd9b640 1 -- 172.21.15.83:0/3025689942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce700c42e0 msgr2=0x7fce700c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6dd9b640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce700c42e0 0x7fce700c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6dd9b640 1 -- 172.21.15.83:0/3025689942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7014d4c0 con 0x7fce701566b0 2024-09-19T07:50:09.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce6dd9b640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce701566b0 0x7fce7014c5d0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fce6000d800 tx=0x7fce6000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:09.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce5effd640 1 -- 172.21.15.83:0/3025689942 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6000b6c0 con 0x7fce701566b0 2024-09-19T07:50:09.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce5effd640 1 -- 172.21.15.83:0/3025689942 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fce60019070 con 0x7fce701566b0 2024-09-19T07:50:09.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 -- 172.21.15.83:0/3025689942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce70150d30 con 0x7fce701566b0 2024-09-19T07:50:09.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce5effd640 1 -- 172.21.15.83:0/3025689942 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6000bb00 con 0x7fce701566b0 2024-09-19T07:50:09.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.519+0000 7fce74df2640 1 -- 172.21.15.83:0/3025689942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce701512f0 con 0x7fce701566b0 2024-09-19T07:50:09.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.523+0000 7fce74df2640 1 -- 172.21.15.83:0/3025689942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce70159eb0 con 0x7fce701566b0 2024-09-19T07:50:09.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.523+0000 7fce5effd640 1 -- 172.21.15.83:0/3025689942 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fce6000bca0 con 0x7fce701566b0 2024-09-19T07:50:09.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.523+0000 7fce5effd640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce500678f0 0x7fce50069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.523+0000 7fce5effd640 1 -- 172.21.15.83:0/3025689942 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 9301+0+0 (secure 0 0 0) 0x7fce60086ff0 con 0x7fce701566b0 2024-09-19T07:50:09.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.523+0000 7fce6e59c640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce500678f0 0x7fce50069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.523+0000 7fce6e59c640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce500678f0 0x7fce50069db0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fce640ae000 tx=0x7fce640a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:09.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.527+0000 7fce5effd640 1 -- 172.21.15.83:0/3025689942 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fce60056d20 con 0x7fce701566b0 2024-09-19T07:50:09.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.723+0000 7fce74df2640 1 -- 172.21.15.83:0/3025689942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce7006e050 con 0x7fce701566b0 2024-09-19T07:50:09.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.723+0000 7fce5effd640 1 -- 172.21.15.83:0/3025689942 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) ==== 95+0+4404 (secure 0 0 0) 0x7fce60056620 con 0x7fce701566b0 2024-09-19T07:50:09.722 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:50:09.722 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":210,"btime":"2024-09-19T07:50:06:886590+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3171991578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3171991578},{"type":"v1","addr":"172.21.15.184:6835","nonce":3171991578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":6500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/368086462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":368086462},{"type":"v1","addr":"172.21.15.184:6833","nonce":368086462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T07:19:39.753527+0000","modified":"2024-09-19T07:48:26.723707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8196":{"gid":8196,"name":"a","rank":0,"incarnation":196,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6835/3580797659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3580797659},{"type":"v1","addr":"172.21.15.83:6835","nonce":3580797659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8196,"qdb_cluster":[8196]},"id":2}]} 2024-09-19T07:50:09.722 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 210 2024-09-19T07:50:09.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.727+0000 7fce74df2640 1 -- 172.21.15.83:0/3025689942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce500678f0 msgr2=0x7fce50069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.727+0000 7fce74df2640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce500678f0 0x7fce50069db0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fce640ae000 tx=0x7fce640a7000 comp rx=0 tx=0).stop 2024-09-19T07:50:09.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.727+0000 7fce74df2640 1 -- 172.21.15.83:0/3025689942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce701566b0 msgr2=0x7fce7014c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.727+0000 7fce74df2640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce701566b0 0x7fce7014c5d0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fce6000d800 tx=0x7fce6000dcc0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.727+0000 7fce74df2640 1 -- 172.21.15.83:0/3025689942 shutdown_connections 2024-09-19T07:50:09.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.727+0000 7fce74df2640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fce500678f0 0x7fce50069db0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.727+0000 7fce74df2640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce70159a80 0x7fce7014cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.727+0000 7fce74df2640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce701566b0 0x7fce7014c5d0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.727+0000 7fce74df2640 1 --2- 172.21.15.83:0/3025689942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce700c42e0 0x7fce700c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.727+0000 7fce74df2640 1 -- 172.21.15.83:0/3025689942 >> 172.21.15.83:0/3025689942 conn(0x7fce7006bc80 msgr2=0x7fce70155ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:09.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.727+0000 7fce74df2640 1 -- 172.21.15.83:0/3025689942 shutdown_connections 2024-09-19T07:50:09.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.727+0000 7fce74df2640 1 -- 172.21.15.83:0/3025689942 wait complete. 2024-09-19T07:50:09.742 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-19T07:50:09.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6c17ff640 1 Processor -- start 2024-09-19T07:50:09.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6c17ff640 1 -- start start 2024-09-19T07:50:09.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6c17ff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc150220 0x7fe6bc156220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6c17ff640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fe6bc06d3e0 0x7fe6bc153560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6c17ff640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6bc068290 con 0x7fe6bc150600 2024-09-19T07:50:09.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6c17ff640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fe6bc068570 con 0x7fe6bc06d3e0 2024-09-19T07:50:09.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6c17ff640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe6bc152d60 con 0x7fe6bc150220 2024-09-19T07:50:09.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6ba7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe6bc150600 0x7fe6bc0ad2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48922/0 (socket says 172.21.15.83:48922) 2024-09-19T07:50:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6ba7fc640 1 -- 172.21.15.83:0/2128248006 learned_addr learned my addr 172.21.15.83:0/2128248006 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:50:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6baffd640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc150220 0x7fe6bc156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6bb7fe640 1 --2- 172.21.15.83:0/2128248006 >> v2:172.21.15.184:3300/0 conn(0x7fe6bc06d3e0 0x7fe6bc153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 828567868 0 0) 0x7fe6bc068290 con 0x7fe6bc150600 2024-09-19T07:50:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6a4003530 con 0x7fe6bc150600 2024-09-19T07:50:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1636890092 0 0) 0x7fe6a4003530 con 0x7fe6bc150600 2024-09-19T07:50:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc150220 msgr2=0x7fe6bc156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc150220 0x7fe6bc156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 >> v2:172.21.15.184:3300/0 conn(0x7fe6bc06d3e0 msgr2=0x7fe6bc153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 --2- 172.21.15.83:0/2128248006 >> v2:172.21.15.184:3300/0 conn(0x7fe6bc06d3e0 0x7fe6bc153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6bc153040 con 0x7fe6bc150600 2024-09-19T07:50:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fe6b0001680 con 0x7fe6bc150600 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 >> v1:172.21.15.83:6789/0 conn(0x7fe6bc150600 legacy=0x7fe6bc0ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6a40039c0 0x7fe6a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6a4004810 0x7fe6a4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc150220 0x7fe6a40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc068570 con 0x7fe6a4004810 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc152d60 con 0x7fe6a40039c0 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe6a400dc60 con 0x7fe6bc150220 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6ba7fc640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6a4004810 0x7fe6a4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6bb7fe640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc150220 0x7fe6a40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6baffd640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6a40039c0 0x7fe6a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6ba7fc640 1 -- 172.21.15.83:0/2128248006 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc150220 msgr2=0x7fe6a40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6ba7fc640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc150220 0x7fe6a40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6ba7fc640 1 -- 172.21.15.83:0/2128248006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6a40039c0 msgr2=0x7fe6a4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:09.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6ba7fc640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6a40039c0 0x7fe6a4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6ba7fc640 1 -- 172.21.15.83:0/2128248006 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a4003530 con 0x7fe6a4004810 2024-09-19T07:50:09.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.867+0000 7fe6baffd640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6a40039c0 0x7fe6a4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6ba7fc640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6a4004810 0x7fe6a4004c90 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000c210 tx=0x7fe6b001cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51136f8661e7860 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0007d40 con 0x7fe6a4004810 2024-09-19T07:50:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe6b00a9020 con 0x7fe6a4004810 2024-09-19T07:50:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6b9ffb640 1 -- 172.21.15.83:0/2128248006 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0005b20 con 0x7fe6a4004810 2024-09-19T07:50:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2128248006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6a4004810 msgr2=0x7fe6a4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 --2- 172.21.15.83:0/2128248006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6a4004810 0x7fe6a4004c90 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000c210 tx=0x7fe6b001cbc0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6bb7fe640 1 -- 172.21.15.83:0/2128248006 reap_dead start 2024-09-19T07:50:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2128248006 shutdown_connections 2024-09-19T07:50:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2128248006 >> 172.21.15.83:0/2128248006 conn(0x7fe6bc0c48b0 msgr2=0x7fe6bc0c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2128248006 shutdown_connections 2024-09-19T07:50:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2128248006 wait complete. 2024-09-19T07:50:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 Processor -- start 2024-09-19T07:50:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 -- start start 2024-09-19T07:50:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6a4004810 0x7fe6bc1ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6bc155070 0x7fe6bc1554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc14b7b0 0x7fe6bc14bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc071a70 con 0x7fe6bc155070 2024-09-19T07:50:09.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc0718f0 con 0x7fe6a4004810 2024-09-19T07:50:09.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6c17ff640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe6bc071bf0 con 0x7fe6bc14b7b0 2024-09-19T07:50:09.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6ba7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6bc155070 0x7fe6bc1554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6ba7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6bc155070 0x7fe6bc1554f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47572/0 (socket says 172.21.15.83:47572) 2024-09-19T07:50:09.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6ba7fc640 1 -- 172.21.15.83:0/2137454375 learned_addr learned my addr 172.21.15.83:0/2137454375 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6baffd640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6a4004810 0x7fe6bc1ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6ba7fc640 1 -- 172.21.15.83:0/2137454375 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc14b7b0 msgr2=0x7fe6bc14bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6bb7fe640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc14b7b0 0x7fe6bc14bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6ba7fc640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc14b7b0 0x7fe6bc14bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6ba7fc640 1 -- 172.21.15.83:0/2137454375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6a4004810 msgr2=0x7fe6bc1ee1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6ba7fc640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6a4004810 0x7fe6bc1ee1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:09.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6ba7fc640 1 -- 172.21.15.83:0/2137454375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6bc1ee6f0 con 0x7fe6bc155070 2024-09-19T07:50:09.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6baffd640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6a4004810 0x7fe6bc1ee1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:09.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.871+0000 7fe6ba7fc640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6bc155070 0x7fe6bc1554f0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0007040 tx=0x7fe6a4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:09.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.875+0000 7fe6a37fe640 1 -- 172.21.15.83:0/2137454375 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b00049f0 con 0x7fe6bc155070 2024-09-19T07:50:09.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.875+0000 7fe6a37fe640 1 -- 172.21.15.83:0/2137454375 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe6b0004b90 con 0x7fe6bc155070 2024-09-19T07:50:09.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.875+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2137454375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6bc0c7b10 con 0x7fe6bc155070 2024-09-19T07:50:09.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.875+0000 7fe6a37fe640 1 -- 172.21.15.83:0/2137454375 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b00070d0 con 0x7fe6bc155070 2024-09-19T07:50:09.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.875+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2137454375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6bc0c7550 con 0x7fe6bc155070 2024-09-19T07:50:09.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.875+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2137454375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6bc153560 con 0x7fe6bc155070 2024-09-19T07:50:09.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.875+0000 7fe6a37fe640 1 -- 172.21.15.83:0/2137454375 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe6b0007270 con 0x7fe6bc155070 2024-09-19T07:50:09.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.875+0000 7fe6a37fe640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6a80678f0 0x7fe6a8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:09.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.875+0000 7fe6a37fe640 1 -- 172.21.15.83:0/2137454375 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 9301+0+0 (secure 0 0 0) 0x7fe6b0120570 con 0x7fe6bc155070 2024-09-19T07:50:09.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.875+0000 7fe6baffd640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6a80678f0 0x7fe6a8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:09.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.875+0000 7fe6baffd640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6a80678f0 0x7fe6a8069db0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc072ee0 tx=0x7fe6ac008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:09.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:09.879+0000 7fe6a37fe640 1 -- 172.21.15.83:0/2137454375 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fe6b00c7080 con 0x7fe6bc155070 2024-09-19T07:50:10.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.031+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2137454375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe6bc14f950 con 0x7fe6bc155070 2024-09-19T07:50:10.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6a37fe640 1 -- 172.21.15.83:0/2137454375 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+22178 (secure 0 0 0) 0x7fe6b0016990 con 0x7fe6bc155070 2024-09-19T07:50:10.031 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-19T07:50:10.031 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":69,"fsid":"86ed7344-2eac-4937-889b-a144de44eb64","created":"2024-09-19T07:19:08.198908+0000","modified":"2024-09-19T07:50:08.900455+0000","last_up_change":"2024-09-19T07:19:20.524221+0000","last_in_change":"2024-09-19T07:19:10.527409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T07:19:20.885610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-19T07:19:38.325531+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-19T07:19:38.737342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1394d19-a843-4204-a708-4c443d4e24ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6809","nonce":1414865007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6811","nonce":1414865007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6815","nonce":1414865007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1414865007},{"type":"v1","addr":"172.21.15.83:6813","nonce":1414865007}]},"public_addr":"172.21.15.83:6809/1414865007","cluster_addr":"172.21.15.83:6811/1414865007","heartbeat_back_addr":"172.21.15.83:6815/1414865007","heartbeat_front_addr":"172.21.15.83:6813/1414865007","state":["exists","up"]},{"osd":1,"uuid":"ece67475-bb01-4b2d-8e42-078cb90df26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6801","nonce":2685389983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6803","nonce":2685389983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6807","nonce":2685389983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2685389983},{"type":"v1","addr":"172.21.15.83:6805","nonce":2685389983}]},"public_addr":"172.21.15.83:6801/2685389983","cluster_addr":"172.21.15.83:6803/2685389983","heartbeat_back_addr":"172.21.15.83:6807/2685389983","heartbeat_front_addr":"172.21.15.83:6805/2685389983","state":["exists","up"]},{"osd":2,"uuid":"392df97a-5766-45b5-89c5-fe5940e93e02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6817","nonce":229839114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6819","nonce":229839114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6823","nonce":229839114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":229839114},{"type":"v1","addr":"172.21.15.83:6821","nonce":229839114}]},"public_addr":"172.21.15.83:6817/229839114","cluster_addr":"172.21.15.83:6819/229839114","heartbeat_back_addr":"172.21.15.83:6823/229839114","heartbeat_front_addr":"172.21.15.83:6821/229839114","state":["exists","up"]},{"osd":3,"uuid":"3934cb42-b02b-41c9-ba7d-0c1d23652e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6825","nonce":2952931730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6827","nonce":2952931730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6831","nonce":2952931730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2952931730},{"type":"v1","addr":"172.21.15.83:6829","nonce":2952931730}]},"public_addr":"172.21.15.83:6825/2952931730","cluster_addr":"172.21.15.83:6827/2952931730","heartbeat_back_addr":"172.21.15.83:6831/2952931730","heartbeat_front_addr":"172.21.15.83:6829/2952931730","state":["exists","up"]},{"osd":4,"uuid":"154ece4b-f44e-4b13-8370-c689d47887bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6809","nonce":3839042106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6811","nonce":3839042106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6815","nonce":3839042106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3839042106},{"type":"v1","addr":"172.21.15.184:6813","nonce":3839042106}]},"public_addr":"172.21.15.184:6809/3839042106","cluster_addr":"172.21.15.184:6811/3839042106","heartbeat_back_addr":"172.21.15.184:6815/3839042106","heartbeat_front_addr":"172.21.15.184:6813/3839042106","state":["exists","up"]},{"osd":5,"uuid":"efd87884-f2b4-4edc-b9ba-eb6b9f636f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6819","nonce":767241772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6821","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6823","nonce":767241772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6830","nonce":767241772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6825","nonce":767241772},{"type":"v1","addr":"172.21.15.184:6826","nonce":767241772}]},"public_addr":"172.21.15.184:6819/767241772","cluster_addr":"172.21.15.184:6823/767241772","heartbeat_back_addr":"172.21.15.184:6830/767241772","heartbeat_front_addr":"172.21.15.184:6826/767241772","state":["exists","up"]},{"osd":6,"uuid":"1422f4b2-c4aa-46f9-89d1-38e8a0032b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6801","nonce":1304502305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6803","nonce":1304502305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6807","nonce":1304502305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":1304502305},{"type":"v1","addr":"172.21.15.184:6805","nonce":1304502305}]},"public_addr":"172.21.15.184:6801/1304502305","cluster_addr":"172.21.15.184:6803/1304502305","heartbeat_back_addr":"172.21.15.184:6807/1304502305","heartbeat_front_addr":"172.21.15.184:6805/1304502305","state":["exists","up"]},{"osd":7,"uuid":"962fe708-81cb-4e58-be27-f0a8d0ba0df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6817","nonce":2447598614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6822","nonce":2447598614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6829","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6831","nonce":2447598614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2447598614},{"type":"v1","addr":"172.21.15.184:6827","nonce":2447598614}]},"public_addr":"172.21.15.184:6817/2447598614","cluster_addr":"172.21.15.184:6822/2447598614","heartbeat_back_addr":"172.21.15.184:6831/2447598614","heartbeat_front_addr":"172.21.15.184:6827/2447598614","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:18.065667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.788537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.757106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.695268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.819316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.969509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.689095+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T07:19:17.679483+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:6832/922771363":"2024-09-20T07:50:05.884222+0000","172.21.15.184:6833/2502231963":"2024-09-20T07:48:58.353983+0000","172.21.15.83:6837/1602509240":"2024-09-20T07:48:23.346687+0000","172.21.15.83:6836/1602509240":"2024-09-20T07:48:23.346687+0000","172.21.15.83:6835/119695014":"2024-09-20T07:47:58.342001+0000","172.21.15.184:6833/1207928072":"2024-09-20T07:47:03.329199+0000","172.21.15.83:6836/3490578864":"2024-09-20T07:45:28.310975+0000","172.21.15.83:6835/1429520945":"2024-09-20T07:45:13.307981+0000","172.21.15.83:6834/1429520945":"2024-09-20T07:45:13.307981+0000","172.21.15.184:6833/3311721080":"2024-09-20T07:44:18.290353+0000","172.21.15.184:6832/3311721080":"2024-09-20T07:44:18.290353+0000","172.21.15.83:6837/1990119490":"2024-09-20T07:42:38.260871+0000","172.21.15.83:6836/1990119490":"2024-09-20T07:42:38.260871+0000","172.21.15.83:6834/2776269095":"2024-09-20T07:42:08.252858+0000","172.21.15.184:6833/922771363":"2024-09-20T07:50:05.884222+0000","172.21.15.184:6833/1335198090":"2024-09-20T07:41:33.245545+0000","172.21.15.83:6837/3426675376":"2024-09-20T07:50:03.370836+0000","172.21.15.184:6832/1335198090":"2024-09-20T07:41:33.245545+0000","172.21.15.83:6835/365493751":"2024-09-20T07:41:08.239813+0000","172.21.15.83:6834/119695014":"2024-09-20T07:47:58.342001+0000","172.21.15.83:6835/104517500":"2024-09-20T07:28:38.045210+0000","172.21.15.83:6837/2326757290":"2024-09-20T07:40:23.230841+0000","172.21.15.184:6835/872719174":"2024-09-20T07:28:08.038207+0000","172.21.15.83:6834/3763485069":"2024-09-20T07:27:18.018593+0000","172.21.15.184:6834/872719174":"2024-09-20T07:28:08.038207+0000","172.21.15.184:6832/1207928072":"2024-09-20T07:47:03.329199+0000","172.21.15.83:6836/455754420":"2024-09-20T07:27:38.022965+0000","172.21.15.83:6835/3763485069":"2024-09-20T07:27:18.018593+0000","172.21.15.184:6833/1773885233":"2024-09-20T07:22:22.945705+0000","172.21.15.184:6835/144714632":"2024-09-20T07:25:36.204686+0000","172.21.15.83:6836/26497755":"2024-09-20T07:25:27.991655+0000","172.21.15.184:6833/2993984999":"2024-09-20T07:32:38.105001+0000","172.21.15.83:6836/4265456507":"2024-09-20T07:22:32.948957+0000","172.21.15.83:6834/1072903448":"2024-09-20T07:33:28.117348+0000","172.21.15.83:6835/3442755673":"2024-09-20T07:24:47.982274+0000","172.21.15.83:6837/3490578864":"2024-09-20T07:45:28.310975+0000","172.21.15.184:6835/149004692":"2024-09-20T07:20:57.928172+0000","172.21.15.184:6834/3841282254":"2024-09-20T07:30:48.076144+0000","172.21.15.184:6832/2502231963":"2024-09-20T07:48:58.353983+0000","172.21.15.83:6836/1229829432":"2024-09-20T07:29:48.061079+0000","172.21.15.83:6834/1819616410":"2024-09-20T07:21:07.930616+0000","172.21.15.83:6835/1819616410":"2024-09-20T07:21:07.930616+0000","172.21.15.184:6834/149004692":"2024-09-20T07:20:57.928172+0000","172.21.15.184:6832/1773885233":"2024-09-20T07:22:22.945705+0000","172.21.15.184:6835/2112478738":"2024-09-20T07:35:28.148108+0000","172.21.15.184:6832/2993984999":"2024-09-20T07:32:38.105001+0000","172.21.15.83:6837/26497755":"2024-09-20T07:25:27.991655+0000","172.21.15.184:6832/2898687304":"2024-09-20T07:34:18.130590+0000","172.21.15.83:6835/2776269095":"2024-09-20T07:42:08.252858+0000","172.21.15.83:6836/1049139637":"2024-09-20T07:23:37.966033+0000","172.21.15.83:6834/3442755673":"2024-09-20T07:24:47.982274+0000","172.21.15.83:6837/4265456507":"2024-09-20T07:22:32.948957+0000","172.21.15.184:6833/2898687304":"2024-09-20T07:34:18.130590+0000","172.21.15.184:6835/4115191178":"2024-09-20T07:23:52.164421+0000","172.21.15.184:6834/4115191178":"2024-09-20T07:23:52.164421+0000","172.21.15.83:6834/1774585869":"2024-09-20T07:23:07.959477+0000","172.21.15.184:6834/144714632":"2024-09-20T07:25:36.204686+0000","172.21.15.83:6834/104517500":"2024-09-20T07:28:38.045210+0000","172.21.15.83:6837/1049139637":"2024-09-20T07:23:37.966033+0000","172.21.15.83:6834/2004814399":"2024-09-20T07:30:48.076245+0000","172.21.15.83:6837/1229829432":"2024-09-20T07:29:48.061079+0000","172.21.15.83:6835/2004814399":"2024-09-20T07:30:48.076245+0000","172.21.15.184:6835/3841282254":"2024-09-20T07:30:48.076144+0000","172.21.15.83:6836/2581473241":"2024-09-20T07:31:18.082843+0000","172.21.15.83:6837/455754420":"2024-09-20T07:27:38.022965+0000","172.21.15.83:6834/1079027718":"2024-09-20T07:35:43.151965+0000","172.21.15.83:6837/2581473241":"2024-09-20T07:31:18.082843+0000","172.21.15.83:6836/2326757290":"2024-09-20T07:40:23.230841+0000","172.21.15.83:6835/1072903448":"2024-09-20T07:33:28.117348+0000","172.21.15.83:6835/1774585869":"2024-09-20T07:23:07.959477+0000","172.21.15.184:6835/1519298917":"2024-09-20T07:37:18.179638+0000","172.21.15.83:6836/1203406606":"2024-09-20T07:34:08.127781+0000","172.21.15.83:6834/365493751":"2024-09-20T07:41:08.239813+0000","172.21.15.83:6837/1203406606":"2024-09-20T07:34:08.127781+0000","172.21.15.83:6835/1079027718":"2024-09-20T07:35:43.151965+0000","172.21.15.184:6834/2112478738":"2024-09-20T07:35:28.148108+0000","172.21.15.184:6834/1519298917":"2024-09-20T07:37:18.179638+0000","172.21.15.83:6836/3707393259":"2024-09-20T07:37:18.179736+0000","172.21.15.83:6836/3426675376":"2024-09-20T07:50:03.370836+0000","172.21.15.83:6837/3707393259":"2024-09-20T07:37:18.179736+0000","172.21.15.83:6834/2168217861":"2024-09-20T07:38:08.197381+0000","172.21.15.83:6835/2168217861":"2024-09-20T07:38:08.197381+0000","172.21.15.184:6832/381468017":"2024-09-20T07:39:03.210538+0000","172.21.15.184:6833/381468017":"2024-09-20T07:39:03.210538+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T07:50:10.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2137454375 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6a80678f0 msgr2=0x7fe6a8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:10.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6c17ff640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6a80678f0 0x7fe6a8069db0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc072ee0 tx=0x7fe6ac008040 comp rx=0 tx=0).stop 2024-09-19T07:50:10.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2137454375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6bc155070 msgr2=0x7fe6bc1554f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:10.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6c17ff640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6bc155070 0x7fe6bc1554f0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0007040 tx=0x7fe6a4005840 comp rx=0 tx=0).stop 2024-09-19T07:50:10.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2137454375 shutdown_connections 2024-09-19T07:50:10.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6c17ff640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fe6a80678f0 0x7fe6a8069db0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6c17ff640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6bc14b7b0 0x7fe6bc14bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6c17ff640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6bc155070 0x7fe6bc1554f0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6c17ff640 1 --2- 172.21.15.83:0/2137454375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6a4004810 0x7fe6bc1ee1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2137454375 >> 172.21.15.83:0/2137454375 conn(0x7fe6bc0c48b0 msgr2=0x7fe6bc072310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:10.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2137454375 shutdown_connections 2024-09-19T07:50:10.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.035+0000 7fe6c17ff640 1 -- 172.21.15.83:0/2137454375 wait complete. 2024-09-19T07:50:10.050 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail b 2024-09-19T07:50:10.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2979612640 1 Processor -- start 2024-09-19T07:50:10.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2979612640 1 -- start start 2024-09-19T07:50:10.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2979612640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2974153570 0x7f29740c6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2979612640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f297406cf10 0x7f2974155d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2979612640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29740bf0a0 con 0x7f297415a000 2024-09-19T07:50:10.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2979612640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f29740c6800 con 0x7f2974153570 2024-09-19T07:50:10.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2979612640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f2974159520 con 0x7f297406cf10 2024-09-19T07:50:10.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2972ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2974153570 0x7f29740c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2972ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2974153570 0x7f29740c6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41046/0 (socket says 172.21.15.83:41046) 2024-09-19T07:50:10.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2972ffd640 1 -- 172.21.15.83:0/215652302 learned_addr learned my addr 172.21.15.83:0/215652302 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:10.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2972ffd640 1 -- 172.21.15.83:0/215652302 >> v2:172.21.15.184:3300/0 conn(0x7f297406cf10 msgr2=0x7f2974155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:10.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2972ffd640 1 --2- 172.21.15.83:0/215652302 >> v2:172.21.15.184:3300/0 conn(0x7f297406cf10 0x7f2974155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2972ffd640 1 -- 172.21.15.83:0/215652302 >> v1:172.21.15.184:6789/0 conn(0x7f297415a000 legacy=0x7f29740722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:10.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.171+0000 7f2972ffd640 1 -- 172.21.15.83:0/215652302 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2974159800 con 0x7f2974153570 2024-09-19T07:50:10.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2972ffd640 1 --2- 172.21.15.83:0/215652302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2974153570 0x7f29740c6100 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f29740688a0 tx=0x7f296401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a41deb9cc383cb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:10.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2971ffb640 1 -- 172.21.15.83:0/215652302 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29640a56f0 con 0x7f2974153570 2024-09-19T07:50:10.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2971ffb640 1 -- 172.21.15.83:0/215652302 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f29640a5890 con 0x7f2974153570 2024-09-19T07:50:10.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2971ffb640 1 -- 172.21.15.83:0/215652302 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29640a5c80 con 0x7f2974153570 2024-09-19T07:50:10.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2979612640 1 -- 172.21.15.83:0/215652302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2974153570 msgr2=0x7f29740c6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:10.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2979612640 1 --2- 172.21.15.83:0/215652302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2974153570 0x7f29740c6100 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f29740688a0 tx=0x7f296401c920 comp rx=0 tx=0).stop 2024-09-19T07:50:10.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2979612640 1 -- 172.21.15.83:0/215652302 shutdown_connections 2024-09-19T07:50:10.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2979612640 1 --2- 172.21.15.83:0/215652302 >> v2:172.21.15.184:3300/0 conn(0x7f297406cf10 0x7f2974155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2979612640 1 --2- 172.21.15.83:0/215652302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2974153570 0x7f29740c6100 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2979612640 1 -- 172.21.15.83:0/215652302 >> 172.21.15.83:0/215652302 conn(0x7f29740cb780 msgr2=0x7f29740cbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:10.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2979612640 1 -- 172.21.15.83:0/215652302 shutdown_connections 2024-09-19T07:50:10.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2979612640 1 -- 172.21.15.83:0/215652302 wait complete. 2024-09-19T07:50:10.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2979612640 1 Processor -- start 2024-09-19T07:50:10.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.175+0000 7f2979612640 1 -- start start 2024-09-19T07:50:10.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2979612640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f297406cf10 0x7f29740c4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2979612640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2974153570 0x7f297414caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2979612640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f297415a000 0x7f297414cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2979612640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29741630b0 con 0x7f297415a000 2024-09-19T07:50:10.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2979612640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2974162f30 con 0x7f297406cf10 2024-09-19T07:50:10.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2979612640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2974163230 con 0x7f2974153570 2024-09-19T07:50:10.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f29737fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f297415a000 0x7f297414cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f29737fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f297415a000 0x7f297414cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41048/0 (socket says 172.21.15.83:41048) 2024-09-19T07:50:10.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f29737fe640 1 -- 172.21.15.83:0/814337161 learned_addr learned my addr 172.21.15.83:0/814337161 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:10.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2972ffd640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f297406cf10 0x7f29740c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f29727fc640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2974153570 0x7f297414caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f29737fe640 1 -- 172.21.15.83:0/814337161 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2974153570 msgr2=0x7f297414caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:10.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f29737fe640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2974153570 0x7f297414caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f29737fe640 1 -- 172.21.15.83:0/814337161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f297406cf10 msgr2=0x7f29740c4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:10.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f29737fe640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f297406cf10 0x7f29740c4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f29737fe640 1 -- 172.21.15.83:0/814337161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29740c43b0 con 0x7f297415a000 2024-09-19T07:50:10.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2972ffd640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f297406cf10 0x7f29740c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:50:10.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f29737fe640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f297415a000 0x7f297414cfe0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f296c00b3c0 tx=0x7f296c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:10.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f295b7fe640 1 -- 172.21.15.83:0/814337161 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c011830 con 0x7f297415a000 2024-09-19T07:50:10.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f295b7fe640 1 -- 172.21.15.83:0/814337161 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f296c016070 con 0x7f297415a000 2024-09-19T07:50:10.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2979612640 1 -- 172.21.15.83:0/814337161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f297414d820 con 0x7f297415a000 2024-09-19T07:50:10.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f295b7fe640 1 -- 172.21.15.83:0/814337161 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c011c30 con 0x7f297415a000 2024-09-19T07:50:10.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2979612640 1 -- 172.21.15.83:0/814337161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f297414dde0 con 0x7f297415a000 2024-09-19T07:50:10.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2979612640 1 -- 172.21.15.83:0/814337161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2974155d90 con 0x7f297415a000 2024-09-19T07:50:10.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f295b7fe640 1 -- 172.21.15.83:0/814337161 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f296c01b460 con 0x7f297415a000 2024-09-19T07:50:10.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f295b7fe640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29600678f0 0x7f2960069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f295b7fe640 1 -- 172.21.15.83:0/814337161 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 9301+0+0 (secure 0 0 0) 0x7f296c086740 con 0x7f297415a000 2024-09-19T07:50:10.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2972ffd640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29600678f0 0x7f2960069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.179+0000 7f2972ffd640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29600678f0 0x7f2960069db0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f29640ae000 tx=0x7f29640a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:10.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.183+0000 7f295b7fe640 1 -- 172.21.15.83:0/814337161 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f296c056420 con 0x7f297415a000 2024-09-19T07:50:10.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.383+0000 7f2979612640 1 -- 172.21.15.83:0/814337161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "b"} v 0) -- 0x7f297414dfb0 con 0x7f297415a000 2024-09-19T07:50:10.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.383+0000 7f295b7fe640 1 -- 172.21.15.83:0/814337161 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "b"}]=0 b marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v211) ==== 151+0+0 (secure 0 0 0) 0x7f296c055d20 con 0x7f297415a000 2024-09-19T07:50:10.381 INFO:teuthology.orchestra.run.smithi083.stderr:b marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-19T07:50:10.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.387+0000 7f2979612640 1 -- 172.21.15.83:0/814337161 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29600678f0 msgr2=0x7f2960069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:10.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.387+0000 7f2979612640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29600678f0 0x7f2960069db0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f29640ae000 tx=0x7f29640a7070 comp rx=0 tx=0).stop 2024-09-19T07:50:10.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.387+0000 7f2979612640 1 -- 172.21.15.83:0/814337161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f297415a000 msgr2=0x7f297414cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:10.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.387+0000 7f2979612640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f297415a000 0x7f297414cfe0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f296c00b3c0 tx=0x7f296c00b880 comp rx=0 tx=0).stop 2024-09-19T07:50:10.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.387+0000 7f2979612640 1 -- 172.21.15.83:0/814337161 shutdown_connections 2024-09-19T07:50:10.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.387+0000 7f2979612640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f29600678f0 0x7f2960069db0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.387+0000 7f2979612640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f297415a000 0x7f297414cfe0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.387+0000 7f2979612640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2974153570 0x7f297414caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.387+0000 7f2979612640 1 --2- 172.21.15.83:0/814337161 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f297406cf10 0x7f29740c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.387+0000 7f2979612640 1 -- 172.21.15.83:0/814337161 >> 172.21.15.83:0/814337161 conn(0x7f29740cb780 msgr2=0x7f2974154990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:10.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.391+0000 7f2979612640 1 -- 172.21.15.83:0/814337161 shutdown_connections 2024-09-19T07:50:10.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.391+0000 7f2979612640 1 -- 172.21.15.83:0/814337161 wait complete. 2024-09-19T07:50:10.404 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm b --yes-i-really-mean-it 2024-09-19T07:50:10.436 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-09-19T07:50:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.527+0000 7fef90e81640 1 Processor -- start 2024-09-19T07:50:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.527+0000 7fef90e81640 1 -- start start 2024-09-19T07:50:10.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.527+0000 7fef90e81640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7fef8c14fbf0 0x7fef8c1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.527+0000 7fef90e81640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef8c0bedf0 0x7fef8c0c6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.527+0000 7fef90e81640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef8c156be0 con 0x7fef8c0c6590 2024-09-19T07:50:10.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.527+0000 7fef90e81640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7fef8c0733d0 con 0x7fef8c14fbf0 2024-09-19T07:50:10.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.527+0000 7fef90e81640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef8c073950 con 0x7fef8c0bedf0 2024-09-19T07:50:10.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.527+0000 7fef8ad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fef8c0c6590 0x7fef8c06c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57160/0 (socket says 172.21.15.83:57160) 2024-09-19T07:50:10.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.527+0000 7fef8ad9d640 1 -- 172.21.15.83:0/3079633404 learned_addr learned my addr 172.21.15.83:0/3079633404 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:50:10.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.527+0000 7fef89d9b640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef8c0bedf0 0x7fef8c0c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.527+0000 7fef8a59c640 1 --2- 172.21.15.83:0/3079633404 >> v2:172.21.15.184:3300/0 conn(0x7fef8c14fbf0 0x7fef8c1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.527+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3110004253 0 0) 0x7fef84001120 con 0x7fef8c0c6590 2024-09-19T07:50:10.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef78003530 con 0x7fef8c0c6590 2024-09-19T07:50:10.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3449592200 0 0) 0x7fef78003530 con 0x7fef8c0c6590 2024-09-19T07:50:10.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef8c0bedf0 msgr2=0x7fef8c0c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef8c0bedf0 0x7fef8c0c6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 >> v2:172.21.15.184:3300/0 conn(0x7fef8c14fbf0 msgr2=0x7fef8c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 --2- 172.21.15.83:0/3079633404 >> v2:172.21.15.184:3300/0 conn(0x7fef8c14fbf0 0x7fef8c1566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8c073670 con 0x7fef8c0c6590 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7fef84001850 con 0x7fef8c0c6590 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 >> v1:172.21.15.83:6789/0 conn(0x7fef8c0c6590 legacy=0x7fef8c06c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef780039c0 0x7fef78003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef8c0bedf0 0x7fef78004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef78004d50 0x7fef780051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c0733d0 con 0x7fef780039c0 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c073950 con 0x7fef78004d50 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef7800ba80 con 0x7fef8c0bedf0 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8a59c640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef780039c0 0x7fef78003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef89d9b640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef8c0bedf0 0x7fef78004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8ad9d640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef78004d50 0x7fef780051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8a59c640 1 -- 172.21.15.83:0/3079633404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef8c0bedf0 msgr2=0x7fef78004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:10.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8a59c640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef8c0bedf0 0x7fef78004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8a59c640 1 -- 172.21.15.83:0/3079633404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef78004d50 msgr2=0x7fef780051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:10.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8a59c640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef78004d50 0x7fef780051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8a59c640 1 -- 172.21.15.83:0/3079633404 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef78003530 con 0x7fef780039c0 2024-09-19T07:50:10.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.531+0000 7fef8ad9d640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef78004d50 0x7fef780051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:10.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.535+0000 7fef8a59c640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef780039c0 0x7fef78003de0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009ae0 tx=0x7fef7c01ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b904e167fac42b09 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:10.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.535+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0b2070 con 0x7fef780039c0 2024-09-19T07:50:10.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.535+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fef7c0a5560 con 0x7fef780039c0 2024-09-19T07:50:10.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.535+0000 7fef8959a640 1 -- 172.21.15.83:0/3079633404 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0096b0 con 0x7fef780039c0 2024-09-19T07:50:10.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.535+0000 7fef90e81640 1 -- 172.21.15.83:0/3079633404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef780039c0 msgr2=0x7fef78003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:10.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.535+0000 7fef90e81640 1 --2- 172.21.15.83:0/3079633404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef780039c0 0x7fef78003de0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009ae0 tx=0x7fef7c01ca80 comp rx=0 tx=0).stop 2024-09-19T07:50:10.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.535+0000 7fef8ad9d640 1 -- 172.21.15.83:0/3079633404 reap_dead start 2024-09-19T07:50:10.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.535+0000 7fef90e81640 1 -- 172.21.15.83:0/3079633404 shutdown_connections 2024-09-19T07:50:10.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.535+0000 7fef90e81640 1 -- 172.21.15.83:0/3079633404 >> 172.21.15.83:0/3079633404 conn(0x7fef8c0c8150 msgr2=0x7fef8c0c8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:10.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.535+0000 7fef90e81640 1 -- 172.21.15.83:0/3079633404 shutdown_connections 2024-09-19T07:50:10.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.535+0000 7fef90e81640 1 -- 172.21.15.83:0/3079633404 wait complete. 2024-09-19T07:50:10.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef90e81640 1 Processor -- start 2024-09-19T07:50:10.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef90e81640 1 -- start start 2024-09-19T07:50:10.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef90e81640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef780039c0 0x7fef8c0bf570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef90e81640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef8c148800 0x7fef8c0c0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef90e81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef8c1e5d20 0x7fef8c0c0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef90e81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c14ffd0 con 0x7fef8c1e5d20 2024-09-19T07:50:10.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef90e81640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c06d3b0 con 0x7fef8c148800 2024-09-19T07:50:10.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef90e81640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef8c1f2ab0 con 0x7fef780039c0 2024-09-19T07:50:10.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef8ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef8c1e5d20 0x7fef8c0c0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef8ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef8c1e5d20 0x7fef8c0c0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41080/0 (socket says 172.21.15.83:41080) 2024-09-19T07:50:10.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef8ad9d640 1 -- 172.21.15.83:0/1632615648 learned_addr learned my addr 172.21.15.83:0/1632615648 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:10.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef8a59c640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef780039c0 0x7fef8c0bf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef89d9b640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef8c148800 0x7fef8c0c0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef8ad9d640 1 -- 172.21.15.83:0/1632615648 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef780039c0 msgr2=0x7fef8c0bf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:10.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef8ad9d640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef780039c0 0x7fef8c0bf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef8ad9d640 1 -- 172.21.15.83:0/1632615648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef8c148800 msgr2=0x7fef8c0c0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:10.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef8ad9d640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef8c148800 0x7fef8c0c0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:10.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef8ad9d640 1 -- 172.21.15.83:0/1632615648 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8c0cb020 con 0x7fef8c1e5d20 2024-09-19T07:50:10.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef8ad9d640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef8c1e5d20 0x7fef8c0c0e40 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0c7d50 tx=0x7fef8400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:10.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef82ffd640 1 -- 172.21.15.83:0/1632615648 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef840075a0 con 0x7fef8c1e5d20 2024-09-19T07:50:10.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef82ffd640 1 -- 172.21.15.83:0/1632615648 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fef840047f0 con 0x7fef8c1e5d20 2024-09-19T07:50:10.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef90e81640 1 -- 172.21.15.83:0/1632615648 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef8c0cb300 con 0x7fef8c1e5d20 2024-09-19T07:50:10.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef82ffd640 1 -- 172.21.15.83:0/1632615648 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef84004c00 con 0x7fef8c1e5d20 2024-09-19T07:50:10.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef90e81640 1 -- 172.21.15.83:0/1632615648 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef8c1f2d90 con 0x7fef8c1e5d20 2024-09-19T07:50:10.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.539+0000 7fef90e81640 1 -- 172.21.15.83:0/1632615648 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef8c1566a0 con 0x7fef8c1e5d20 2024-09-19T07:50:10.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.543+0000 7fef82ffd640 1 -- 172.21.15.83:0/1632615648 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fef8401b020 con 0x7fef8c1e5d20 2024-09-19T07:50:10.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.543+0000 7fef82ffd640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef6c0678f0 0x7fef6c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:10.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.543+0000 7fef82ffd640 1 -- 172.21.15.83:0/1632615648 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 9387+0+0 (secure 0 0 0) 0x7fef840865b0 con 0x7fef8c1e5d20 2024-09-19T07:50:10.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.543+0000 7fef8a59c640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef6c0678f0 0x7fef6c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:10.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.543+0000 7fef8a59c640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef6c0678f0 0x7fef6c069db0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0af000 tx=0x7fef7c0046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:10.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.547+0000 7fef82ffd640 1 -- 172.21.15.83:0/1632615648 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7fef8c156be0 con 0x7fef8c1e5d20 2024-09-19T07:50:10.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:10.727+0000 7fef90e81640 1 -- 172.21.15.83:0/1632615648 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "b", "yes_i_really_mean_it": true} v 0) -- 0x7fef8c1492c0 con 0x7fef8c1e5d20 2024-09-19T07:50:11.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.387+0000 7fef82ffd640 1 -- 172.21.15.83:0/1632615648 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "b", "yes_i_really_mean_it": true}]=0 v212) ==== 99+0+0 (secure 0 0 0) 0x7fef84055b90 con 0x7fef8c1e5d20 2024-09-19T07:50:11.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.391+0000 7fef90e81640 1 -- 172.21.15.83:0/1632615648 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef6c0678f0 msgr2=0x7fef6c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:11.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.391+0000 7fef90e81640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef6c0678f0 0x7fef6c069db0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0af000 tx=0x7fef7c0046d0 comp rx=0 tx=0).stop 2024-09-19T07:50:11.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.391+0000 7fef90e81640 1 -- 172.21.15.83:0/1632615648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef8c1e5d20 msgr2=0x7fef8c0c0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:11.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.391+0000 7fef90e81640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef8c1e5d20 0x7fef8c0c0e40 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0c7d50 tx=0x7fef8400d9d0 comp rx=0 tx=0).stop 2024-09-19T07:50:11.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.391+0000 7fef90e81640 1 -- 172.21.15.83:0/1632615648 shutdown_connections 2024-09-19T07:50:11.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.391+0000 7fef90e81640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7fef6c0678f0 0x7fef6c069db0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:11.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.391+0000 7fef90e81640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef8c1e5d20 0x7fef8c0c0e40 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:11.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.391+0000 7fef90e81640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef8c148800 0x7fef8c0c0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:11.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.391+0000 7fef90e81640 1 --2- 172.21.15.83:0/1632615648 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef780039c0 0x7fef8c0bf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:11.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.391+0000 7fef90e81640 1 -- 172.21.15.83:0/1632615648 >> 172.21.15.83:0/1632615648 conn(0x7fef8c0c8150 msgr2=0x7fef8c155090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:11.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.391+0000 7fef90e81640 1 -- 172.21.15.83:0/1632615648 shutdown_connections 2024-09-19T07:50:11.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.391+0000 7fef90e81640 1 -- 172.21.15.83:0/1632615648 wait complete. 2024-09-19T07:50:11.408 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm b_metadata b_metadata --yes-i-really-really-mean-it 2024-09-19T07:50:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9c537640 1 Processor -- start 2024-09-19T07:50:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9c537640 1 -- start start 2024-09-19T07:50:11.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9c537640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f94151330 0x7f1f94070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:11.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9c537640 1 --2- >> v2:172.21.15.184:3300/0 conn(0x7f1f940a9d30 0x7f1f9406e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:11.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9c537640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f9406ea10 con 0x7f1f940713e0 2024-09-19T07:50:11.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9c537640 1 -- --> v2:172.21.15.184:3300/0 -- mon_getmap magic: 0 -- 0x7f1f94154910 con 0x7f1f940a9d30 2024-09-19T07:50:11.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9c537640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f94153d00 con 0x7f1f94151330 2024-09-19T07:50:11.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f99ad2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1f940713e0 0x7f1f9406ddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57166/0 (socket says 172.21.15.83:57166) 2024-09-19T07:50:11.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f99ad2640 1 -- 172.21.15.83:0/2313177164 learned_addr learned my addr 172.21.15.83:0/2313177164 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1126183674 0 0) 0x7f1f9406ea10 con 0x7f1f940713e0 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f80003530 con 0x7f1f940713e0 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9aad4640 1 --2- 172.21.15.83:0/2313177164 >> v2:172.21.15.184:3300/0 conn(0x7f1f940a9d30 0x7f1f9406e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9a2d3640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f94151330 0x7f1f94070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2467216774 0 0) 0x7f1f80003530 con 0x7f1f940713e0 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f94151330 msgr2=0x7f1f94070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f94151330 0x7f1f94070ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 >> v2:172.21.15.184:3300/0 conn(0x7f1f940a9d30 msgr2=0x7f1f9406e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 --2- 172.21.15.83:0/2313177164 >> v2:172.21.15.184:3300/0 conn(0x7f1f940a9d30 0x7f1f9406e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f94153fe0 con 0x7f1f940713e0 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f1f84001680 con 0x7f1f940713e0 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9aad4640 1 --2- 172.21.15.83:0/2313177164 >> v2:172.21.15.184:3300/0 conn(0x7f1f940a9d30 0x7f1f9406e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:50:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 >> v1:172.21.15.83:6789/0 conn(0x7f1f940713e0 legacy=0x7f1f9406ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:11.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f800039c0 0x7f1f80003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:11.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f94151330 0x7f1f80004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:11.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f80004d50 0x7f1f800051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:11.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1f94154910 con 0x7f1f800039c0 2024-09-19T07:50:11.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f94153d00 con 0x7f1f80004d50 2024-09-19T07:50:11.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f8000bbe0 con 0x7f1f94151330 2024-09-19T07:50:11.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9a2d3640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f800039c0 0x7f1f80003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:11.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9aad4640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f80004d50 0x7f1f800051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:11.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9a2d3640 1 -- 172.21.15.83:0/2313177164 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f94151330 msgr2=0x7f1f80004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:11.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f99ad2640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f94151330 0x7f1f80004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:11.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9a2d3640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f94151330 0x7f1f80004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:11.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9a2d3640 1 -- 172.21.15.83:0/2313177164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f80004d50 msgr2=0x7f1f800051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:11.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9a2d3640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f80004d50 0x7f1f800051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:11.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9a2d3640 1 -- 172.21.15.83:0/2313177164 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f80003530 con 0x7f1f800039c0 2024-09-19T07:50:11.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.579+0000 7f1f9aad4640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f80004d50 0x7f1f800051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:11.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9a2d3640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f800039c0 0x7f1f80003de0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f1f9000ba30 tx=0x7f1f9001c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a49c515d92d51391 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:11.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9000b600 con 0x7f1f800039c0 2024-09-19T07:50:11.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f1f900b0070 con 0x7f1f800039c0 2024-09-19T07:50:11.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f992d1640 1 -- 172.21.15.83:0/2313177164 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f900ab8e0 con 0x7f1f800039c0 2024-09-19T07:50:11.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 -- 172.21.15.83:0/2313177164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f800039c0 msgr2=0x7f1f80003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:11.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 --2- 172.21.15.83:0/2313177164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f800039c0 0x7f1f80003de0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f1f9000ba30 tx=0x7f1f9001c670 comp rx=0 tx=0).stop 2024-09-19T07:50:11.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9aad4640 1 -- 172.21.15.83:0/2313177164 reap_dead start 2024-09-19T07:50:11.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 -- 172.21.15.83:0/2313177164 shutdown_connections 2024-09-19T07:50:11.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 -- 172.21.15.83:0/2313177164 >> 172.21.15.83:0/2313177164 conn(0x7f1f9406bc80 msgr2=0x7f1f9406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:11.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 -- 172.21.15.83:0/2313177164 shutdown_connections 2024-09-19T07:50:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 -- 172.21.15.83:0/2313177164 wait complete. 2024-09-19T07:50:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 Processor -- start 2024-09-19T07:50:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 -- start start 2024-09-19T07:50:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f800039c0 0x7f1f94144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f94144990 0x7f1f94146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f941ed4d0 0x7f1f941ed960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1f94151890 con 0x7f1f800039c0 2024-09-19T07:50:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f94151710 con 0x7f1f94144990 2024-09-19T07:50:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.583+0000 7f1f9c537640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f9415eca0 con 0x7f1f941ed4d0 2024-09-19T07:50:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9a2d3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f800039c0 0x7f1f94144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9a2d3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f800039c0 0x7f1f94144e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41086/0 (socket says 172.21.15.83:41086) 2024-09-19T07:50:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9a2d3640 1 -- 172.21.15.83:0/4266523183 learned_addr learned my addr 172.21.15.83:0/4266523183 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f99ad2640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f94144990 0x7f1f94146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9aad4640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f941ed4d0 0x7f1f941ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9a2d3640 1 -- 172.21.15.83:0/4266523183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f941ed4d0 msgr2=0x7f1f941ed960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9a2d3640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f941ed4d0 0x7f1f941ed960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9a2d3640 1 -- 172.21.15.83:0/4266523183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f94144990 msgr2=0x7f1f94146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9a2d3640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f94144990 0x7f1f94146a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9a2d3640 1 -- 172.21.15.83:0/4266523183 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f94150160 con 0x7f1f800039c0 2024-09-19T07:50:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9aad4640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f941ed4d0 0x7f1f941ed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9a2d3640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f800039c0 0x7f1f94144e90 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f1f900ad040 tx=0x7f1f80005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f8affd640 1 -- 172.21.15.83:0/4266523183 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f900ba070 con 0x7f1f800039c0 2024-09-19T07:50:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f8affd640 1 -- 172.21.15.83:0/4266523183 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f1f90004220 con 0x7f1f800039c0 2024-09-19T07:50:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9c537640 1 -- 172.21.15.83:0/4266523183 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f94150440 con 0x7f1f800039c0 2024-09-19T07:50:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f8affd640 1 -- 172.21.15.83:0/4266523183 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f900ab640 con 0x7f1f800039c0 2024-09-19T07:50:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9c537640 1 -- 172.21.15.83:0/4266523183 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f94150a00 con 0x7f1f800039c0 2024-09-19T07:50:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f9c537640 1 -- 172.21.15.83:0/4266523183 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f9406ddc0 con 0x7f1f800039c0 2024-09-19T07:50:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f8affd640 1 -- 172.21.15.83:0/4266523183 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1f900ab7e0 con 0x7f1f800039c0 2024-09-19T07:50:11.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f8affd640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f780678c0 0x7f1f78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:11.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f8affd640 1 -- 172.21.15.83:0/4266523183 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 9387+0+0 (secure 0 0 0) 0x7f1f90121340 con 0x7f1f800039c0 2024-09-19T07:50:11.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f99ad2640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f780678c0 0x7f1f78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:11.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.587+0000 7f1f99ad2640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f780678c0 0x7f1f78069d80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f1f84001000 tx=0x7f1f84009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:11.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.591+0000 7f1f8affd640 1 -- 172.21.15.83:0/4266523183 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f1f900163b0 con 0x7f1f800039c0 2024-09-19T07:50:11.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:11.743+0000 7f1f9c537640 1 -- 172.21.15.83:0/4266523183 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "b_metadata", "pool2": "b_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1f94153670 con 0x7f1f800039c0 2024-09-19T07:50:12.282 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:50:12.284 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:50:12.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.395+0000 7f1f8affd640 1 -- 172.21.15.83:0/4266523183 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "b_metadata", "pool2": "b_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'b_metadata' removed v71) ==== 166+0+0 (secure 0 0 0) 0x7f1f900f10f0 con 0x7f1f800039c0 2024-09-19T07:50:12.392 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'b_metadata' removed 2024-09-19T07:50:12.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.399+0000 7f1f9c537640 1 -- 172.21.15.83:0/4266523183 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f780678c0 msgr2=0x7f1f78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:12.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.399+0000 7f1f9c537640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f780678c0 0x7f1f78069d80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f1f84001000 tx=0x7f1f84009040 comp rx=0 tx=0).stop 2024-09-19T07:50:12.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.399+0000 7f1f9c537640 1 -- 172.21.15.83:0/4266523183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f800039c0 msgr2=0x7f1f94144e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:12.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.399+0000 7f1f9c537640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f800039c0 0x7f1f94144e90 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f1f900ad040 tx=0x7f1f80005840 comp rx=0 tx=0).stop 2024-09-19T07:50:12.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.399+0000 7f1f9c537640 1 -- 172.21.15.83:0/4266523183 shutdown_connections 2024-09-19T07:50:12.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.399+0000 7f1f9c537640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f1f780678c0 0x7f1f78069d80 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:12.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.399+0000 7f1f9c537640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f941ed4d0 0x7f1f941ed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:12.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.399+0000 7f1f9c537640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f94144990 0x7f1f94146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:12.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.399+0000 7f1f9c537640 1 --2- 172.21.15.83:0/4266523183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f800039c0 0x7f1f94144e90 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:12.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.399+0000 7f1f9c537640 1 -- 172.21.15.83:0/4266523183 >> 172.21.15.83:0/4266523183 conn(0x7f1f9406bc80 msgr2=0x7f1f9406dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:12.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.399+0000 7f1f9c537640 1 -- 172.21.15.83:0/4266523183 shutdown_connections 2024-09-19T07:50:12.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.399+0000 7f1f9c537640 1 -- 172.21.15.83:0/4266523183 wait complete. 2024-09-19T07:50:12.421 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm b_data b_data --yes-i-really-really-mean-it 2024-09-19T07:50:12.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.547+0000 7f7b4be19640 1 Processor -- start 2024-09-19T07:50:12.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.547+0000 7f7b4be19640 1 -- start start 2024-09-19T07:50:12.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.547+0000 7f7b4be19640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b441533d0 0x7f7b440aa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:12.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.547+0000 7f7b4be19640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b44153970 con 0x7f7b440c2ee0 2024-09-19T07:50:12.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.547+0000 7f7b4be19640 1 -- --> v1:172.21.15.184:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b440c4970 con 0x7f7b440c3e80 2024-09-19T07:50:12.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.547+0000 7f7b4be19640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7b440aadb0 con 0x7f7b441533d0 2024-09-19T07:50:12.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.547+0000 7f7b4a3b6640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7b440c2ee0 0x7f7b4414f9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57174/0 (socket says 172.21.15.83:57174) 2024-09-19T07:50:12.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.547+0000 7f7b4a3b6640 1 -- 172.21.15.83:0/2774721122 learned_addr learned my addr 172.21.15.83:0/2774721122 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-19T07:50:12.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.547+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2128308482 0 0) 0x7f7b44153970 con 0x7f7b440c2ee0 2024-09-19T07:50:12.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.547+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b30003530 con 0x7f7b440c2ee0 2024-09-19T07:50:12.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.547+0000 7f7b493b4640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b441533d0 0x7f7b440aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:12.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1837911704 0 0) 0x7f7b30003530 con 0x7f7b440c2ee0 2024-09-19T07:50:12.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b441533d0 msgr2=0x7f7b440aa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:12.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b441533d0 0x7f7b440aa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 >> v1:172.21.15.184:6789/0 conn(0x7f7b440c3e80 legacy=0x7f7b440c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b440c2120 con 0x7f7b440c2ee0 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2546383505 0 0) 0x7f7b3c001a00 con 0x7f7b440c2ee0 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b493b4640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b441533d0 0x7f7b440aa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 >> v1:172.21.15.83:6789/0 conn(0x7f7b440c2ee0 legacy=0x7f7b4414f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b441533d0 0x7f7b300039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b30003fe0 0x7f7b30004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b30004d50 0x7f7b300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7b440aadb0 con 0x7f7b30004d50 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3000bbc0 con 0x7f7b30003fe0 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7b3000bdc0 con 0x7f7b441533d0 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4a3b6640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b30004d50 0x7f7b300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:12.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b49bb5640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b441533d0 0x7f7b300039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:12.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b493b4640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b30003fe0 0x7f7b30004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:12.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4a3b6640 1 -- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b441533d0 msgr2=0x7f7b300039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:12.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4a3b6640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b441533d0 0x7f7b300039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:12.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4a3b6640 1 -- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b30003fe0 msgr2=0x7f7b30004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T07:50:12.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4a3b6640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b30003fe0 0x7f7b30004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:12.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4a3b6640 1 -- 172.21.15.83:0/2774721122 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b30003530 con 0x7f7b30004d50 2024-09-19T07:50:12.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b49bb5640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b441533d0 0x7f7b300039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:12.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4a3b6640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b30004d50 0x7f7b300051d0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00a440 tx=0x7f7b3c021d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664728f57a89289f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:12.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c00a480 con 0x7f7b30004d50 2024-09-19T07:50:12.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7b3c0af070 con 0x7f7b30004d50 2024-09-19T07:50:12.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b48bb3640 1 -- 172.21.15.83:0/2774721122 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c0abbf0 con 0x7f7b30004d50 2024-09-19T07:50:12.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4be19640 1 -- 172.21.15.83:0/2774721122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b30004d50 msgr2=0x7f7b300051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:12.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4be19640 1 --2- 172.21.15.83:0/2774721122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b30004d50 0x7f7b300051d0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00a440 tx=0x7f7b3c021d10 comp rx=0 tx=0).stop 2024-09-19T07:50:12.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4a3b6640 1 -- 172.21.15.83:0/2774721122 reap_dead start 2024-09-19T07:50:12.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4be19640 1 -- 172.21.15.83:0/2774721122 shutdown_connections 2024-09-19T07:50:12.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4be19640 1 -- 172.21.15.83:0/2774721122 >> 172.21.15.83:0/2774721122 conn(0x7f7b4406bc80 msgr2=0x7f7b4406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:12.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4be19640 1 -- 172.21.15.83:0/2774721122 shutdown_connections 2024-09-19T07:50:12.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.551+0000 7f7b4be19640 1 -- 172.21.15.83:0/2774721122 wait complete. 2024-09-19T07:50:12.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4be19640 1 Processor -- start 2024-09-19T07:50:12.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4be19640 1 -- start start 2024-09-19T07:50:12.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4be19640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b30004d50 0x7f7b441de1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:12.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4be19640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b4414f030 0x7f7b441ec3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:12.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4be19640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b441de6f0 0x7f7b441ec920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:12.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4be19640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7b440c29c0 con 0x7f7b441de6f0 2024-09-19T07:50:12.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4be19640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b440c26e0 con 0x7f7b4414f030 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4be19640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7b441500d0 con 0x7f7b30004d50 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4a3b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b441de6f0 0x7f7b441ec920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4a3b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b441de6f0 0x7f7b441ec920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41104/0 (socket says 172.21.15.83:41104) 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4a3b6640 1 -- 172.21.15.83:0/120461532 learned_addr learned my addr 172.21.15.83:0/120461532 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b49bb5640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b30004d50 0x7f7b441de1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b493b4640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b4414f030 0x7f7b441ec3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4a3b6640 1 -- 172.21.15.83:0/120461532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b30004d50 msgr2=0x7f7b441de1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4a3b6640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b30004d50 0x7f7b441de1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4a3b6640 1 -- 172.21.15.83:0/120461532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b4414f030 msgr2=0x7f7b441ec3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4a3b6640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b4414f030 0x7f7b441ec3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4a3b6640 1 -- 172.21.15.83:0/120461532 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b44151940 con 0x7f7b441de6f0 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b493b4640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b4414f030 0x7f7b441ec3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T07:50:12.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4a3b6640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b441de6f0 0x7f7b441ec920 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00a920 tx=0x7f7b30005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:12.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b367fc640 1 -- 172.21.15.83:0/120461532 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c00a480 con 0x7f7b441de6f0 2024-09-19T07:50:12.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b367fc640 1 -- 172.21.15.83:0/120461532 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7b3c004b70 con 0x7f7b441de6f0 2024-09-19T07:50:12.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4be19640 1 -- 172.21.15.83:0/120461532 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b44073fc0 con 0x7f7b441de6f0 2024-09-19T07:50:12.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b367fc640 1 -- 172.21.15.83:0/120461532 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c001fe0 con 0x7f7b441de6f0 2024-09-19T07:50:12.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4be19640 1 -- 172.21.15.83:0/120461532 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b440c0a20 con 0x7f7b441de6f0 2024-09-19T07:50:12.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b4be19640 1 -- 172.21.15.83:0/120461532 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b44153830 con 0x7f7b441de6f0 2024-09-19T07:50:12.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.555+0000 7f7b367fc640 1 -- 172.21.15.83:0/120461532 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7b3c007070 con 0x7f7b441de6f0 2024-09-19T07:50:12.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.559+0000 7f7b367fc640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7b280678c0 0x7f7b28069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T07:50:12.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.559+0000 7f7b367fc640 1 -- 172.21.15.83:0/120461532 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8959+0+0 (secure 0 0 0) 0x7f7b3c122d60 con 0x7f7b441de6f0 2024-09-19T07:50:12.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.559+0000 7f7b49bb5640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7b280678c0 0x7f7b28069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T07:50:12.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.559+0000 7f7b49bb5640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7b280678c0 0x7f7b28069d80 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7b40001600 tx=0x7f7b40009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T07:50:12.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.563+0000 7f7b367fc640 1 -- 172.21.15.83:0/120461532 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180342 (secure 0 0 0) 0x7f7b3c01ba20 con 0x7f7b441de6f0 2024-09-19T07:50:12.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:12.739+0000 7f7b4be19640 1 -- 172.21.15.83:0/120461532 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "b_data", "pool2": "b_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7b44068290 con 0x7f7b441de6f0 2024-09-19T07:50:13.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.399+0000 7f7b367fc640 1 -- 172.21.15.83:0/120461532 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "b_data", "pool2": "b_data", "yes_i_really_really_mean_it": true}]=0 pool 'b_data' removed v72) ==== 154+0+0 (secure 0 0 0) 0x7f7b3c0f2b40 con 0x7f7b441de6f0 2024-09-19T07:50:13.395 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'b_data' removed 2024-09-19T07:50:13.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.403+0000 7f7b4be19640 1 -- 172.21.15.83:0/120461532 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7b280678c0 msgr2=0x7f7b28069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:13.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.403+0000 7f7b4be19640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7b280678c0 0x7f7b28069d80 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7b40001600 tx=0x7f7b40009040 comp rx=0 tx=0).stop 2024-09-19T07:50:13.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.403+0000 7f7b4be19640 1 -- 172.21.15.83:0/120461532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b441de6f0 msgr2=0x7f7b441ec920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T07:50:13.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.403+0000 7f7b4be19640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b441de6f0 0x7f7b441ec920 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00a920 tx=0x7f7b30005840 comp rx=0 tx=0).stop 2024-09-19T07:50:13.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.407+0000 7f7b4be19640 1 -- 172.21.15.83:0/120461532 shutdown_connections 2024-09-19T07:50:13.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.407+0000 7f7b4be19640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.83:6832/1237718513,v1:172.21.15.83:6833/1237718513] conn(0x7f7b280678c0 0x7f7b28069d80 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:13.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.407+0000 7f7b4be19640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b441de6f0 0x7f7b441ec920 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:13.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.407+0000 7f7b4be19640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b4414f030 0x7f7b441ec3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:13.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.407+0000 7f7b4be19640 1 --2- 172.21.15.83:0/120461532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b30004d50 0x7f7b441de1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T07:50:13.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.407+0000 7f7b4be19640 1 -- 172.21.15.83:0/120461532 >> 172.21.15.83:0/120461532 conn(0x7f7b4406bc80 msgr2=0x7f7b44071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T07:50:13.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.407+0000 7f7b4be19640 1 -- 172.21.15.83:0/120461532 shutdown_connections 2024-09-19T07:50:13.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-19T07:50:13.407+0000 7f7b4be19640 1 -- 172.21.15.83:0/120461532 wait complete. 2024-09-19T07:50:13.436 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-19T07:50:13.437 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-19T07:50:13.437 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:13.450 INFO:tasks.ceph.mds.a:Stopped 2024-09-19T07:50:13.450 ERROR:tasks.ceph.mds.c:tried to stop a non-running daemon 2024-09-19T07:50:13.450 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-19T07:50:13.450 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:13.504 INFO:tasks.ceph.mds.b:Stopped 2024-09-19T07:50:13.505 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-19T07:50:13.505 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:13.520 INFO:tasks.ceph.mds.d:Stopped 2024-09-19T07:50:13.520 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-19T07:50:13.520 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-19T07:50:13.520 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:18.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-19T07:50:19.622 INFO:tasks.ceph.osd.0:Stopped 2024-09-19T07:50:19.623 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-19T07:50:19.623 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:24.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-19T07:50:24.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-19T07:50:25.725 INFO:tasks.ceph.osd.1:Stopped 2024-09-19T07:50:25.726 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-19T07:50:25.726 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:25.796 INFO:tasks.ceph.osd.2:Stopped 2024-09-19T07:50:25.796 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-19T07:50:25.796 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:30.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-19T07:50:30.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-19T07:50:31.897 INFO:tasks.ceph.osd.3:Stopped 2024-09-19T07:50:31.898 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-19T07:50:31.898 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:35.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-19T07:50:35.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-19T07:50:38.000 INFO:tasks.ceph.osd.4:Stopped 2024-09-19T07:50:38.000 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-19T07:50:38.000 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:41.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-09-19T07:50:41.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-09-19T07:50:42.328 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:50:42.331 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:50:44.102 INFO:tasks.ceph.osd.5:Stopped 2024-09-19T07:50:44.102 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-19T07:50:44.102 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:47.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-09-19T07:50:47.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-09-19T07:50:50.204 INFO:tasks.ceph.osd.6:Stopped 2024-09-19T07:50:50.205 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-19T07:50:50.205 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:52.759 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-09-19T07:50:52.759 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-09-19T07:50:56.307 INFO:tasks.ceph.osd.7:Stopped 2024-09-19T07:50:56.307 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-19T07:50:56.308 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-19T07:50:56.308 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:56.338 INFO:tasks.ceph.mgr.y:Stopped 2024-09-19T07:50:56.339 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-19T07:50:56.339 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:56.367 INFO:tasks.ceph.mgr.x:Stopped 2024-09-19T07:50:56.368 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-19T07:50:56.368 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-19T07:50:56.368 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:56.403 INFO:tasks.ceph.mon.a:Stopped 2024-09-19T07:50:56.403 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-19T07:50:56.403 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:56.430 INFO:tasks.ceph.mon.b:Stopped 2024-09-19T07:50:56.431 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-19T07:50:56.431 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T07:50:56.499 INFO:tasks.ceph.mon.c:Stopped 2024-09-19T07:50:56.499 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-19T07:50:56.499 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | egrep -v 'Replacing daemon mds' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-19T07:50:56.634 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-19T07:50:56.634 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-19T07:50:56.903 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-19T07:50:56.903 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-19T07:50:57.171 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-19T07:50:57.172 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-19T07:50:57.409 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-19T07:50:57.410 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-19T07:50:57.665 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-19T07:50:57.666 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-19T07:50:57.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-09-19T07:50:58.099 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-19T07:50:58.099 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-19T07:50:58.329 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-19T07:50:58.329 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-19T07:50:58.545 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-19T07:50:58.546 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-19T07:50:58.763 INFO:tasks.ceph:Archiving mon data... 2024-09-19T07:50:58.765 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786/data/mon.a.tgz 2024-09-19T07:50:58.765 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-09-19T07:50:58.780 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.7o9iPaQv1l 2024-09-19T07:50:58.781 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7o9iPaQv1l 2024-09-19T07:50:58.949 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.7o9iPaQv1l 2024-09-19T07:50:59.039 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.7o9iPaQv1l is 495KB 2024-09-19T07:50:59.101 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.7o9iPaQv1l 2024-09-19T07:50:59.106 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786/data/mon.b.tgz 2024-09-19T07:50:59.106 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-09-19T07:50:59.124 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.bb1FehfZuL 2024-09-19T07:50:59.125 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.bb1FehfZuL 2024-09-19T07:50:59.290 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.bb1FehfZuL 2024-09-19T07:50:59.398 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.bb1FehfZuL is 496KB 2024-09-19T07:50:59.456 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.bb1FehfZuL 2024-09-19T07:50:59.461 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786/data/mon.c.tgz 2024-09-19T07:50:59.461 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-09-19T07:50:59.507 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.IRZGr8M26K 2024-09-19T07:50:59.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.IRZGr8M26K 2024-09-19T07:50:59.672 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.IRZGr8M26K 2024-09-19T07:50:59.734 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.IRZGr8M26K is 496KB 2024-09-19T07:50:59.833 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.IRZGr8M26K 2024-09-19T07:50:59.838 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-19T07:50:59.838 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-19T07:50:59.842 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-19T07:50:59.915 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-19T07:50:59.952 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-19T07:50:59.952 INFO:tasks.ceph:Archiving crash dumps... 2024-09-19T07:50:59.954 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786/remote/smithi083/crash 2024-09-19T07:50:59.955 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-19T07:50:59.972 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786/remote/smithi184/crash 2024-09-19T07:50:59.973 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-19T07:51:00.007 INFO:tasks.ceph:Shutting down logrotate 2024-09-19T07:51:00.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:51:00.016 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-19T07:51:00.108 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-19T07:51:00.121 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-19T07:51:00.132 INFO:tasks.ceph:Compressing logs... 2024-09-19T07:51:00.133 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-19T07:51:00.164 DEBUG:teuthology.orchestra.run.smithi184:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-19T07:51:00.181 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87397.log 2024-09-19T07:51:00.181 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43947.log 2024-09-19T07:51:00.182 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91232.log 2024-09-19T07:51:00.182 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87397.log.gz 2024-09-19T07:51:00.182 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36705.log 2024-09-19T07:51:00.182 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35681.log 2024-09-19T07:51:00.184 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36705.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91232.log.gz 2024-09-19T07:51:00.184 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35466.log 2024-09-19T07:51:00.184 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89623.log 2024-09-19T07:51:00.185 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62981.log 2024-09-19T07:51:00.185 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47111.log 2024-09-19T07:51:00.185 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62981.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.89623.log.gz 2024-09-19T07:51:00.185 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46041.log 2024-09-19T07:51:00.185 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47111.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62981.log.gz 2024-09-19T07:51:00.185 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88559.log 2024-09-19T07:51:00.190 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47111.log.gz 2024-09-19T07:51:00.190 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49808.log 2024-09-19T07:51:00.190 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36103.log 2024-09-19T07:51:00.190 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88559.log.gz 2024-09-19T07:51:00.190 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.49808.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47754.log 2024-09-19T07:51:00.190 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42923.log 2024-09-19T07:51:00.190 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49808.log.gz 2024-09-19T07:51:00.190 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91152.log 2024-09-19T07:51:00.190 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47754.log.gz 2024-09-19T07:51:00.190 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41828.log 2024-09-19T07:51:00.190 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35680.log 2024-09-19T07:51:00.191 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41828.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.91152.log.gz 2024-09-19T07:51:00.191 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34747.log 2024-09-19T07:51:00.191 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59810.log 2024-09-19T07:51:00.191 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34747.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-19T07:51:00.191 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41854.log 2024-09-19T07:51:00.191 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59810.log.gz 2024-09-19T07:51:00.191 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46473.log 2024-09-19T07:51:00.191 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40557.log 2024-09-19T07:51:00.191 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47579.log 2024-09-19T07:51:00.191 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34494.log 2024-09-19T07:51:00.201 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88782.log 2024-09-19T07:51:00.202 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47579.log.gz 2024-09-19T07:51:00.202 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41426.log 2024-09-19T07:51:00.202 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51224.log 2024-09-19T07:51:00.202 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88782.log.gz 2024-09-19T07:51:00.202 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34552.log 2024-09-19T07:51:00.202 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44002.log 2024-09-19T07:51:00.202 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51224.log.gz 2024-09-19T07:51:00.202 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34552.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41426.log.gz 2024-09-19T07:51:00.202 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35678.log 2024-09-19T07:51:00.202 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48376.log 2024-09-19T07:51:00.202 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41333.log 2024-09-19T07:51:00.202 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87119.log 2024-09-19T07:51:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48376.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43947.log.gz 2024-09-19T07:51:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36679.log 2024-09-19T07:51:00.203 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6%/var/log/ceph/ceph-client.admin.87119.log: -- replaced with /var/log/ceph/ceph-client.admin.48376.log.gz 2024-09-19T07:51:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34959.log 2024-09-19T07:51:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36679.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87119.log.gz 2024-09-19T07:51:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35193.log 2024-09-19T07:51:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84096.log 2024-09-19T07:51:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46099.log 2024-09-19T07:51:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45162.log 2024-09-19T07:51:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84096.log: /var/log/ceph/ceph-client.admin.46099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42451.log 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84096.log.gz 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36077.log 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42451.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40557.log.gz 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48491.log 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91414.log 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48899.log 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41333.log.gz 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89008.log 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48491.log.gz 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.91414.log.gz 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41174.log 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89008.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48899.log.gz 2024-09-19T07:51:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69275.log 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42451.log.gz 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.89008.log.gz 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89220.log 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69275.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69275.log.gz 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66763.log 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89220.log.gz 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88367.log 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41854.log.gz 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90633.log 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41174.log.gz 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88367.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66763.log.gz 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45108.log 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.88367.log.gz 2024-09-19T07:51:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44455.log 2024-09-19T07:51:00.206 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90633.log.gz 2024-09-19T07:51:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35470.log 2024-09-19T07:51:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44455.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34552.log.gz 2024-09-19T07:51:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34879.log 2024-09-19T07:51:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66873.log 2024-09-19T07:51:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48016.log 2024-09-19T07:51:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49635.log 2024-09-19T07:51:00.216 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29168.log 2024-09-19T07:51:00.217 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29096.log 2024-09-19T07:51:00.217 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.29168.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-19T07:51:00.217 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.29096.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-19T07:51:00.217 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34414.log 2024-09-19T07:51:00.217 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-19T07:51:00.217 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.1.34414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29144.log 2024-09-19T07:51:00.217 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose --/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.29048.log 2024-09-19T07:51:00.217 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.29144.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-19T07:51:00.217 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.29048.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-19T07:51:00.217 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29072.log 2024-09-19T07:51:00.217 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-19T07:51:00.217 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.29072.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29192.log 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.29192.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29024.log 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29120.log 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.29024.log: /var/log/ceph/ceph-client.admin.29120.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29168.log.gz 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29192.log.gz 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29048.log.gz 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.29072.log.gz 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.29120.log.gz 2024-09-19T07:51:00.218 INFO:teuthology.orchestra.run.smithi184.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29144.log.gz 2024-09-19T07:51:00.227 INFO:teuthology.orchestra.run.smithi184.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29096.log.gz 2024-09-19T07:51:00.234 INFO:teuthology.orchestra.run.smithi184.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29024.log.gz 2024-09-19T07:51:00.348 INFO:teuthology.orchestra.run.smithi184.stderr: 94.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-19T07:51:00.383 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66873.log.gz 2024-09-19T07:51:00.383 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88003.log 2024-09-19T07:51:00.383 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48016.log: /var/log/ceph/ceph-client.admin.49635.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46473.log.gz 2024-09-19T07:51:00.383 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48016.log.gzgzip 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47085.log 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49635.log.gz 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41200.log 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88003.log.gz 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57007.log 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47085.log.gz 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41200.log: /var/log/ceph/ceph-client.admin.41633.log 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50205.log 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41633.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57007.log.gz 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84281.log 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88286.log 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50205.log.gz 2024-09-19T07:51:00.384 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61593.log 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46041.log.gz 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84281.log.gz 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88286.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46099.log.gz 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88286.log.gz 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41633.log.gz 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40939.log 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61593.log.gz 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41226.log 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45292.log 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40725.log 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49057.log 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41739.log 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37576.log 2024-09-19T07:51:00.385 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44273.log 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49057.log.gz 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45690.log 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47501.log 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45690.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35193.log.gz 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37426.log 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46774.log 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37426.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47501.log.gz 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89985.log 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90870.log 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89985.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46774.log.gz 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48979.log 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35470.log.gz 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35466.log.gz 2024-09-19T07:51:00.386 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41501.log 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.89985.log.gz 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35680.log.gz 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89246.log 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90870.log.gz 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35681.log.gz 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35678.log.gz 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41501.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48979.log.gz 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49388.log 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58122.log 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41739.log.gz 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89246.log.gz 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66815.log 2024-09-19T07:51:00.387 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58122.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49388.log.gz 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45049.log 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58122.log.gz 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37192.log 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2%/var/log/ceph/ceph-client.admin.45049.log: -- replaced with /var/log/ceph/ceph-client.admin.66815.log.gz 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34825.log 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44922.log 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88440.log 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35479.log 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35219.log 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2%/var/log/ceph/ceph-client.admin.35479.log: -- replaced with /var/log/ceph/ceph-client.admin.88440.log.gz 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34468.log 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34587.log 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41906.log 2024-09-19T07:51:00.388 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49741.log 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63154.log 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41148.log 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49741.log.gz 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91000.log 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41148.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63154.log.gz 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42849.log 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90069.log 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91000.log.gz 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49083.log 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45742.log 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90069.log.gz 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71944.log 2024-09-19T07:51:00.389 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49083.log.gz 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49542.log 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37739.log 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3%/var/log/ceph/ceph-client.admin.49542.log: -- replaced with /var/log/ceph/ceph-client.admin.71944.log.gz 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43788.log 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49542.log.gz 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49954.log 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43119.log 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35141.log 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2%/var/log/ceph/ceph-client.admin.43119.log: -- replaced with /var/log/ceph/ceph-client.admin.49954.log.gz 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47964.log 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45716.log 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35671.log 2024-09-19T07:51:00.390 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45690.log.gz 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47964.log.gz -- replaced with /var/log/ceph/ceph-client.admin.41501.log.gz 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.45716.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48815.log 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37192.log.gz 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48681.log 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47629.log 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34773.log 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48815.log.gz 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35479.log.gz 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47629.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48681.log.gz 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35219.log.gz 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50832.log 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47629.log.gz 2024-09-19T07:51:00.391 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66789.log 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50832.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35141.log.gz 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34468.log.gz 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50832.log.gz 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46125.log 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66789.log.gz 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44217.log 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42923.log.gz 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51385.log 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90425.log 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42949.log 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51385.log.gz 2024-09-19T07:51:00.392 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45950.log 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90425.log.gz 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88953.log 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90818.log 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40939.log.gz 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37694.log 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41148.log.gz 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90818.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88953.log.gz 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67086.log 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37694.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.90818.log.gz 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34643.log 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43119.log.gz 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40751.log 2024-09-19T07:51:00.393 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67086.log.gz 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41962.log 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40751.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44002.log.gz 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45003.log 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89192.log 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44455.log.gz 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43921.log 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.89192.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36861.log 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45162.log.gz 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89192.log.gz 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37455.log 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43921.log: /var/log/ceph/ceph-client.admin.36861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47423.log 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43494.log 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34227.log 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43494.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47423.log.gz 2024-09-19T07:51:00.394 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46559.log 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89756.log 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.46559.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36442.log 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89357.log 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89756.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37576.log.gz 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89429.log 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.89756.log.gz 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37348.log 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89357.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40725.log.gz 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60485.log 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89357.log.gz 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37348.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89429.log.gz 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40995.log 2024-09-19T07:51:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90792.log 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42849.log.gz 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40995.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42949.log.gz 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60485.log.gz 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46893.log 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90792.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45266.log 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.90792.log.gz 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1%/var/log/ceph/ceph-mon.a.log: -- replaced with /var/log/ceph/ceph-client.admin.46893.log.gz 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88156.log 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35671.log.gz 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46748.log 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45266.log: /var/log/ceph/ceph-client.admin.88156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49687.log 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88156.log.gz 2024-09-19T07:51:00.396 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37322.log 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46748.log.gz 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35869.log 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49687.log.gz 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37244.log 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35869.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37739.log.gz 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36387.log 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37244.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41200.log.gz 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48441.log 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41226.log.gz 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41828.log.gz 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49135.log 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42581.log 2024-09-19T07:51:00.397 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46017.log 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48441.log.gz 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46125.log.gz 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89326.log 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49135.log.gz 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43921.log.gz 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37114.log 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46017.log: /var/log/ceph/ceph-client.admin.89326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47349.log 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.89326.log.gz 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90659.log 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47059.log 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47349.log: /var/log/ceph/ceph-client.admin.90659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71403.log 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47349.log.gz 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37036.log 88.4% 2024-09-19T07:51:00.398 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90659.log.gz 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47059.log.gz 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46828.log 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71403.log.gz 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88312.log 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44481.log 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46828.log.gz 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46384.log 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88312.log.gz 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44948.log 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50058.log 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71814.log 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47189.log 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50058.log.gz 2024-09-19T07:51:00.399 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90399.log 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71814.log.gz 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89166.log 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47189.log: /var/log/ceph/ceph-client.admin.90399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43656.log 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47189.log.gz 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69440.log 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90399.log.gz 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89166.log.gz 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63009.log 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43656.log: /var/log/ceph/ceph-client.admin.69440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51276.log 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69440.log.gz 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45610.log 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63009.log.gz 2024-09-19T07:51:00.400 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90040.log 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49980.log 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45610.log: 88.5%/var/log/ceph/ceph-client.admin.90040.log: -- replaced with /var/log/ceph/ceph-client.admin.51276.log.gz 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45638.log 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90040.log.gz 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49609.log 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45638.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49980.log.gz 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35037.log 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90123.log 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49609.log.gz 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43630.log 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89597.log 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36809.log 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90123.log.gz 2024-09-19T07:51:00.401 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43028.log 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48927.log 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.89597.log.gz 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90766.log 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43028.log: /var/log/ceph/ceph-client.admin.48927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48844.log 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90766.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48927.log.gz 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90607.log 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45950.log.gz 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90766.log.gz 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88808.log 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37694.log.gz 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90607.log: gzip 88.5% -5 89.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.40751.log.gz -- replaced with /var/log/ceph/ceph-client.admin.48844.log.gz -- 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.49583.log 2024-09-19T07:51:00.402 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90607.log.gz 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35477.log 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88808.log: /var/log/ceph/ceph-client.admin.49583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36991.log 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.88808.log.gz 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35477.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49583.log.gz 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43028.log.gz 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43894.log 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35262.log 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48629.log 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36913.log 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35262.log: /var/log/ceph/ceph-client.admin.48629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90097.log 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36913.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48629.log.gz 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45292.log.gz 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63100.log 2024-09-19T07:51:00.403 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45846.log 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41962.log.gz 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35481.log 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.90097.log.gz 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63100.log.gz 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49661.log 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48349.log 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50231.log 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44403.log 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49661.log.gz 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48349.log.gz 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49358.log 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50231.log: /var/log/ceph/ceph-client.admin.44403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66899.log 2024-09-19T07:51:00.404 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50231.log.gz 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66557.log 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49358.log.gz 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48170.log 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46854.log 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66899.log.gz 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48170.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66557.log.gz 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51331.log 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44922.log.gz/var/log/ceph/ceph-client.admin.46854.log: 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48170.log.gz 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49414.log 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46854.log.gz 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose 89.3% -- /var/log/ceph/ceph-client.admin.34343.log -- replaced with /var/log/ceph/ceph-client.admin.45003.log.gz 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:51:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51331.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45049.log.gz 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49838.log 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51331.log.gz 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49414.log.gz 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84003.log 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45266.log.gz 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34343.log: /var/log/ceph/ceph-client.admin.49838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37140.log 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84003.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49838.log.gz 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91050.log 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84003.log.gz 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62866.log 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51250.log 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91050.log.gz 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89782.log 2024-09-19T07:51:00.406 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62866.log.gzgzip 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: -5/var/log/ceph/ceph-client.admin.51250.log: --verbose -- /var/log/ceph/ceph-client.admin.46436.log 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41579.log 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51250.log.gz 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89782.log.gz 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90922.log 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49304.log 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37400.log 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43494.log.gz 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85102.log 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90922.log.gz 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49304.log.gz/var/log/ceph/ceph-client.admin.37400.log: 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90451.log 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35115.log 2024-09-19T07:51:00.407 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85102.log.gz 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28873.log 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35115.log: gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.44711.log 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41906.log.gz 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90451.log.gz 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.28873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47475.log 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91336.log 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91106.log 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47475.log.gz 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42425.log 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91336.log.gz 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84070.log 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1%/var/log/ceph/ceph-client.admin.91106.log: -- replaced with /var/log/ceph/ceph-client.admin.43788.log.gz 2024-09-19T07:51:00.408 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64970.log 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91106.log.gz 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45108.log.gz 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40911.log 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64970.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84070.log.gz 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37602.log 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34343.log.gz 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40911.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64970.log.gz 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89062.log 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47689.log 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83976.log 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5%/var/log/ceph/ceph-client.admin.47689.log: -- replaced with /var/log/ceph/ceph-client.admin.36077.log.gz 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48095.log 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89062.log.gz 2024-09-19T07:51:00.409 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36103.log.gz 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43125.log 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47689.log.gz 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48095.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83976.log.gz 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36207.log 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43125.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48095.log.gz 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89533.log 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90740.log 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36207.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37036.log.gz 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49031.log 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.36387.log.gz 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89533.log.gz 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90740.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37114.log.gz 2024-09-19T07:51:00.410 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88592.log 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36653.log 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49031.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90740.log.gz 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88592.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49031.log.gz 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57564.log 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35262.log.gz 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88592.log.gz 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45768.log 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.57564.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.90844.log 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44154.log 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57564.log.gz 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90844.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90844.log.gz 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41021.log 2024-09-19T07:51:00.411 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48144.log 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34907.log 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34227.log.gz 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35688.log 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34907.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48144.log.gz 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56877.log 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35477.log.gz 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35481.log.gz 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43000.log 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35688.log: /var/log/ceph/ceph-client.admin.56877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43816.log 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2%/var/log/ceph/ceph-client.admin.43000.log: -- replaced with /var/log/ceph/ceph-client.admin.56877.log.gz 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39383.log 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44685.log 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34587.log.gz 2024-09-19T07:51:00.412 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35269.log 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34643.log.gz 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39383.log: /var/log/ceph/ceph-client.admin.44685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35167.log 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42662.log 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42156.log 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90149.log 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42662.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37426.log.gz 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36309.log 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35037.log.gz 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36835.log 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36309.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90149.log.gz 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47527.log 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35063.log 2024-09-19T07:51:00.413 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89352.log 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47527.log.gz 88.2% 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35115.log.gz 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42741.log 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44507.log 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42741.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89352.log.gz 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36129.log 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40505.log 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44896.log 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35869.log.gz 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59257.log 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88533.log 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36991.log.gz 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45924.log 2024-09-19T07:51:00.414 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42581.log.gz 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59257.log.gz 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89140.log 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45924.log: gzip -5 --verbose 88.1% -- /var/log/ceph/ceph-client.admin.45820.log -- replaced with /var/log/ceph/ceph-client.admin.88533.log.gz 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48402.log 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43630.log.gz 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45820.log: gzip 87.4% 88.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.89140.log.gz -- replaced with /var/log/ceph/ceph-client.admin.43656.log.gz -- 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.45240.log 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43550.log 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48402.log: /var/log/ceph/ceph-client.admin.45240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36620.log 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48402.log.gz 2024-09-19T07:51:00.415 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43550.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44711.log.gz 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42290.log 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44948.log.gz 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88871.log 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48655.log 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44273.log.gz 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37218.log 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48655.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.88871.log.gz 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45610.log.gz 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37218.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48655.log.gz 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43052.log 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45638.log.gz 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48733.log 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42529.log 2024-09-19T07:51:00.416 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41988.log 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45846.log.gz 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48733.log.gz 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42529.log: gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.46384.log.gz --verbose 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph-client.admin.63126.log 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34376.log 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45333.log 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34376.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63126.log.gz 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46017.log.gz -5 --verbose 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph-client.admin.90685.log 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63061.log 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44100.log 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90685.log.gz 2024-09-19T07:51:00.417 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66406.log/var/log/ceph/ceph-client.admin.63061.log: 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91388.log 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46436.log.gz 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63061.log.gz 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36155.log 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40911.log.gz 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42425.log.gz 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66406.log.gz 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41281.log 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91388.log.gz 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90295.log 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42662.log.gz 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44658.log 2024-09-19T07:51:00.418 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46278.log 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.90295.log.gz 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34524.log 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90896.log 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88338.log 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90896.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.24005.log 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50961.log 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90896.log.gz 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.tmp-client.admin.24005.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43125.log.gz 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63560.log 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.24005.log.gz 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88338.log.gz 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50032.log 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46226.log 2024-09-19T07:51:00.419 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50961.log.gz 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63560.log.gz 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50032.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50032.log.gz 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40969.log 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43816.log.gz 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34721.log 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74742.log 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.34721.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41050.log 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.74742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47938.log 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.74742.log.gz 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40583.log 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41050.log: /var/log/ceph/ceph-client.admin.47938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91310.log 2024-09-19T07:51:00.420 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47938.log.gz 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35273.log 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43894.log.gz/var/log/ceph/ceph-client.admin.91310.log: 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43708.log 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.91310.log.gz 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42793.log/var/log/ceph/ceph-client.admin.35273.log: 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44403.log.gz 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89036.log 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40453.log 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44507.log.gz -- replaced with /var/log/ceph/ceph-client.admin.44481.log.gz 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47163.log 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40453.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89036.log.gz 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49715.log 2024-09-19T07:51:00.421 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48761.log 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43000.log.gz 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47163.log.gz 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35255.log 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41579.log.gz 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48761.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49715.log.gz -5 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49496.log 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41988.log.gz 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48761.log.gz 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35255.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42529.log.gz 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42715.log 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43550.log.gz 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42976.log 2024-09-19T07:51:00.422 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37348.log.gz 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44191.log 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3%/var/log/ceph/ceph-client.admin.42715.log: -- replaced with /var/log/ceph/ceph-client.admin.49496.log.gz 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37322.log.gz 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37400.log.gz 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35089.log 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37244.log.gz 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37218.log.gz 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44191.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41021.log.gz 88.1% 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40969.log.gz 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40995.log.gz 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40583.log.gz 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41281.log.gz 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41050.log.gz 2024-09-19T07:51:00.423 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36129.log.gz 89.4% 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36155.log.gz 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88260.log 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36207.log.gz 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91362.log 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41607.log 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88982.log 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88260.log.gz 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36233.log 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.91362.log.gz 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43576.log 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36233.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.88982.log.gz 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47320.log 2024-09-19T07:51:00.424 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36309.log.gz 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44974.log 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36783.log 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49867.log 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47320.log.gz 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34376.log.gz 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43425.log 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43260.log 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35167.log.gz 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43425.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.49867.log.gz --verbose 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph-client.admin.90503.log 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91024.log 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.90503.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41553.log 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44217.log.gz 2024-09-19T07:51:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44243.log 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90503.log.gz 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91024.log: /var/log/ceph/ceph-client.admin.41553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40609.log 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45716.log.gz 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91024.log.gz 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41096.log 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47242.log 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90347.log 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35255.log.gz 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41307.log 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35011.log 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47242.log.gz 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43399.log 2024-09-19T07:51:00.426 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.90347.log.gz 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35269.log.gz 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35011.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35273.log.gz 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90711.log 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44738.log 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35089.log.gz 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35063.log.gz 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90711.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44658.log.gz 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87684.log 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44738.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90711.log.gz 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35460.log 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35688.log.gz 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42156.log.gz 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47685.log 2024-09-19T07:51:00.427 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42793.log.gz 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42290.log.gz 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35460.log: 88.3% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42715.log.gz 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87684.log.gz 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.42741.log.gz -5 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52418.log 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47685.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43708.log.gz 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49239.log 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45333.log.gz 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47685.log.gz 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69986.log 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49239.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52418.log.gz 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44128.log 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49239.log.gz 2024-09-19T07:51:00.428 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41370.log 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69986.log.gz 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46410.log 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44128.log: /var/log/ceph/ceph-client.admin.41370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35263.log 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47635.log 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48271.log 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50084.log 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46605.log 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47635.log.gz 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51302.log 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48271.log.gz 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46605.log: gzip -5 --verbose -- 89.5% /var/log/ceph/ceph-client.admin.91076.log -- replaced with /var/log/ceph/ceph-client.admin.50084.log.gz 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:51:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43682.log 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51302.log: /var/log/ceph/ceph-client.admin.91076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41765.log 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51302.log.gz 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91076.log.gz 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49109.log 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43179.log 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49109.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88704.log 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49109.log.gz 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45464.log 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34959.log.gz 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89088.log 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44974.log.gz 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88704.log: /var/log/ceph/ceph-client.admin.45464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44047.log 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88704.log.gz 2024-09-19T07:51:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35903.log 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34985.log 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.89088.log.gz 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41253.log 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46332.log 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44429.log 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41396.log 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46154.log 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36335.log 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36965.log 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88897.log 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36335.log: gzip -5 --verbose -- 88.9% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45240.log.gz 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44685.log.gz 2024-09-19T07:51:00.431 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.49213.log 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44154.log.gz 2024-09-19T07:51:00.432 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88897.log: 2024-09-19T07:51:00.432 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-19T07:51:00.432 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.36965.log: 87.8% -5 --verbose -- /var/log/ceph/ceph-client.admin.43343.log 2024-09-19T07:51:00.432 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88897.log.gz 2024-09-19T07:51:00.432 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44870.log 2024-09-19T07:51:00.432 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49213.log.gz 2024-09-19T07:51:00.432 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89808.log 2024-09-19T07:51:00.432 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46358.log 2024-09-19T07:51:00.432 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44870.log: /var/log/ceph/ceph-client.admin.89808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84029.log 2024-09-19T07:51:00.432 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46951.log 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89808.log.gz 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48091.log 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46226.log.gz 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84029.log.gz 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46278.log.gz 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35873.log 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48091.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46951.log.gz 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42897.log 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48091.log.gz 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89927.log 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44764.log 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86814.log 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89927.log: /var/log/ceph/ceph-client.admin.44764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46499.log 2024-09-19T07:51:00.433 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89927.log.gz 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49278.log 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46499.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.86814.log.gz 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44348.log 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28989.log 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44348.log: gzip 89.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.49278.log.gz 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph-client.admin.47294.log 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.28989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46722.log 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44816.log 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48467.log 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47294.log.gz 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44816.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46722.log.gz 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41685.log 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50110.log 2024-09-19T07:51:00.434 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48467.log: /var/log/ceph/ceph-client.admin.41685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47397.log 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48467.log.gz 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43868.log 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88234.log 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50110.log.gz 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47397.log.gz 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89300.log 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88756.log 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88730.log 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.88234.log.gz 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88756.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89300.log.gz 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90581.log 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88756.log.gz 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40777.log 2024-09-19T07:51:00.435 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88730.log.gz 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36731.log 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90581.log.gz 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89888.log 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89653.log 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83583.log 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47553.log 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.89888.log.gz 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83583.log: gzip -5 --verbose -- 89.1% /var/log/ceph/ceph-client.admin.37062.log 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89653.log.gz 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47553.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83583.log.gz 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42040.log 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37062.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47553.log.gz 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88182.log 2024-09-19T07:51:00.436 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45664.log 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44535.log 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42371.log 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.88182.log.gz 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90948.log 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42767.log 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44561.log 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34747.log.gz 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.90948.log.gz 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34773.log.gz 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5/var/log/ceph/ceph-client.admin.42767.log: --verbose -- /var/log/ceph/ceph-client.admin.43317.log 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34825.log.gz 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90243.log 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44561.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34879.log.gz 2024-09-19T07:51:00.437 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48323.log 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34907.log.gz 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51357.log 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36442.log.gz 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90243.log.gz 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36620.log.gz 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41880.log 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36653.log.gz 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51357.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48323.log.gz 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49467.log 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36679.log.gz 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36705.log.gz 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51357.log.gz 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40661.log 2024-09-19T07:51:00.438 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36809.log.gz 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46252.log 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49467.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36835.log.gz 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42316.log 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49467.log.gz 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36861.log.gz 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37765.log 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36913.log.gz 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40531.log 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42316.log: /var/log/ceph/ceph-client.admin.37765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37642.log 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44074.log 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41096.log.gz 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42066.log 2024-09-19T07:51:00.439 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36361.log 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44600.log 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47003.log 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40687.log 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63035.log 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40687.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47003.log.gz 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45532.log 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37521.log 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63035.log: /var/log/ceph/ceph-client.admin.45532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35899.log 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63035.log.gz 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49005.log 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37521.log: /var/log/ceph/ceph-client.admin.35899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62380.log 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46919.log 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43425.log.gz 2024-09-19T07:51:00.440 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43399.log.gz 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62380.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49005.log.gz 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35917.log 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89434.log 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46919.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62380.log.gz 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88625.log 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46919.log.gz 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89481.log 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45820.log.gz 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88625.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89434.log.gz 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41122.log 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88625.log.gz 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47137.log 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89481.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45924.log.gz 2024-09-19T07:51:00.441 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48549.log 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89481.log.gz 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62090.log 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47137.log: /var/log/ceph/ceph-client.admin.48549.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47137.log.gz 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44047.log.gz 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48549.log.gz 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91262.log 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40479.log 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43973.log 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62090.log.gz 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48519.log 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.91262.log.gz 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40479.log: /var/log/ceph/ceph-client.admin.43973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89953.log 2024-09-19T07:51:00.442 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48519.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48519.log.gz 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90217.log 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45188.log 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44100.log.gz 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89953.log.gz 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90217.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40609.log.gz 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42897.log.gz 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.90217.log.gz 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89730.log 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42976.log.gz 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41553.log.gz 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88104.log 2024-09-19T07:51:00.443 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.45188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42345.log 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89730.log: /var/log/ceph/ceph-client.admin.88104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46643.log 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89730.log.gz 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44243.log.gz 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44738.log.gz 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42345.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44764.log.gz 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88104.log.gz 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41527.log 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45872.log 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40453.log.gz 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40505.log.gz 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46643.log.gz 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35259.log 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29100.log 2024-09-19T07:51:00.444 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35259.log: /var/log/ceph/ceph-client.admin.29100.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41765.log.gz 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37374.log 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47854.log 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70924.log 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44302.log 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43576.log.gz 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47854.log.gz 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44842.log 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34524.log.gz 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50147.log 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70924.log.gz 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44302.log: /var/log/ceph/ceph-client.admin.44842.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28989.log.gz 2024-09-19T07:51:00.445 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91105.log 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45082.log 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40531.log.gz 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50173.log 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91105.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50147.log.gz 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41455.log 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91105.log.gz 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37646.log 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41455.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50173.log.gz 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47801.log 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34494.log.gz 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42819.log 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37646.log: /var/log/ceph/ceph-client.admin.47801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89507.log 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42819.log: 87.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.47801.log.gz 2024-09-19T07:51:00.446 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45389.log 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37140.log.gz 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46800.log 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36181.log 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.89507.log.gz 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36757.log 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41307.log.gz 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46800.log.gz 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35901.log 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49187.log 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89834.log 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35901.log: /var/log/ceph/ceph-client.admin.49187.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.43179.log.gz -5 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.72453.log 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49187.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42262.log 2024-09-19T07:51:00.447 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43682.log.gz 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44128.log.gz 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72453.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36233.log.gz 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34613.log 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89834.log: /var/log/ceph/ceph-client.admin.42262.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72453.log.gz 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36335.log.gz 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47910.log 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36361.log.gz 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36783.log.gz 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89834.log.gz 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88076.log 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90269.log 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35263.log.gz 2024-09-19T07:51:00.448 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88076.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47910.log.gz 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44790.log 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35460.log.gz 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90269.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88076.log.gz 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66843.log 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35011.log.gz 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90269.log.gz 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.44790.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.90191.log 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43343.log.gz 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43260.log.gz 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43317.log.gz 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61026.log 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90191.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34669.log 2024-09-19T07:51:00.449 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66843.log.gz 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61026.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.90191.log.gz 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42014.log 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61026.log.gz 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45214.log 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34669.log: /var/log/ceph/ceph-client.admin.42014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44628.log 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42845.log 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35676.log 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60363.log 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45418.log 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90974.log 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45418.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44535.log.gz 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46306.log 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60363.log.gz 2024-09-19T07:51:00.450 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35480.log 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90974.log: /var/log/ceph/ceph-client.admin.46306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88673.log 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90974.log.gz 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89860.log 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88673.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44600.log.gz 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44561.log.gz 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88673.log.gz 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91487.log 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4%/var/log/ceph/ceph-client.admin.89860.log: -- replaced with /var/log/ceph/ceph-client.admin.39383.log.gz 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91204.log 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91487.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89860.log.gz 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45464.log.gz 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35478.log 2024-09-19T07:51:00.451 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.91487.log.gz 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43760.log 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91204.log: /var/log/ceph/ceph-client.admin.35478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34933.log 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91204.log.gz 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43369.log 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43760.log: gzip/var/log/ceph/ceph-client.admin.34933.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40885.log 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35873.log.gz 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36965.log.gz 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37296.log 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43369.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41370.log.gz 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.41396.log.gz 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45506.log 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41455.log.gz 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42206.log 2024-09-19T07:51:00.452 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45558.log 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45506.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46252.log.gz 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42767.log.gz 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46200.log 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42206.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43973.log.gz 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44870.log.gz 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45359.log 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45136.log 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45082.log.gz 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42238.log 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45359.log: /var/log/ceph/ceph-client.admin.45136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48789.log 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66737.log 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34851.log 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49330.log 2024-09-19T07:51:00.453 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48789.log.gz 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48297.log 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66737.log.gz 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49330.log: /var/log/ceph/ceph-client.admin.48297.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49330.log.gz 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48297.log.gz 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46410.log.gz 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45664.log.gz 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86668.log 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49441.log 89.5% 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46154.log.gz 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70429.log 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49161.log 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86668.log.gz/var/log/ceph/ceph-client.admin.70429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41791.log 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49441.log.gz 2024-09-19T07:51:00.454 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49161.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37455.log.gz 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90373.log 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49161.log.gz 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70429.log.gz 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67215.log 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42316.log.gz 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45794.log 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43868.log.gz 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67215.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90373.log.gz 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46069.log 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49928.log 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44429.log.gz 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67215.log.gz 2024-09-19T07:51:00.455 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35270.log 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88492.log 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49928.log.gz 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37088.log 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89682.log 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44842.log.gz 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44816.log.gz 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42607.log 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88130.log 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41122.log.gz 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37765.log.gz 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89682.log.gz 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42607.log: /var/log/ceph/ceph-client.admin.88492.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41880.log.gz 2024-09-19T07:51:00.456 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88130.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.88492.log.gz -5 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45898.log 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37646.log.gz 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88130.log.gz 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47886.log 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35689.log 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45898.log: /var/log/ceph/ceph-client.admin.47886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48023.log 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47886.log.gz 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58696.log 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42477.log 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40661.log.gz 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40777.log.gz 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40687.log.gz 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90321.log 2024-09-19T07:51:00.457 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48023.log.gz 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58696.log.gz 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62195.log 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42399.log 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62195.log: gzip 89.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.90321.log.gz -- 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.49782.log 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.42399.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39812.log 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62195.log.gz 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35902.log 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90011.log 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49782.log.gz 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35902.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42845.log.gz 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37270.log 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34799.log 2024-09-19T07:51:00.458 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90011.log.gz 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41527.log.gz 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.admin.46529.log 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44790.log.gz 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37270.log: /var/log/ceph/ceph-client.admin.34799.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42014.log.gzgzip 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48951.log 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40479.log.gz 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42066.log.gz 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47268.log 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46358.log.gz 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46332.log.gz 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46529.log: /var/log/ceph/ceph-client.admin.48951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34695.log 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42371.log.gz 2024-09-19T07:51:00.459 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47268.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48951.log.gz 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37062.log.gz 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42345.log.gz 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36181.log.gz 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47268.log.gz 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36757.log.gz 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90477.log 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45989.log 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90477.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35259.log.gz 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40807.log 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45989.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.90477.log.gz 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35270.log.gz 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88927.log 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88834.log 2024-09-19T07:51:00.460 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46669.log 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43369.log.gz 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88834.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88927.log.gz 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48575.log 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46669.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88834.log.gz 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40635.log 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48575.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46669.log.gz 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90555.log 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40635.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48575.log.gz 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47830.log 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88466.log 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44628.log.gz 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5%/var/log/ceph/ceph-client.admin.47830.log: -- replaced with /var/log/ceph/ceph-client.admin.90555.log.gz 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43842.log 2024-09-19T07:51:00.461 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47830.log.gz 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47605.log 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36939.log 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35689.log.gz 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37164.log 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88466.log.gz 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47605.log.gz 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42095.log 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37164.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45359.log.gz 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45389.log.gz 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35901.log.gz 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47216.log 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91258.log 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35902.log.gz 2024-09-19T07:51:00.462 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35661.log 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35917.log.gz 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose 88.7% -- /var/log/ceph/ceph-client.admin.40859.log -- replaced with /var/log/ceph/ceph-client.admin.47216.log.gz 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91258.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42238.log.gz 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46200.log.gz 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35661.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91258.log.gz 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42262.log.gz 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42607.log.gz 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42182.log 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43760.log.gz 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48245.log 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45898.log.gz 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42182.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46499.log.gz 2024-09-19T07:51:00.463 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36887.log 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89559.log 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48603.log 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48245.log.gz 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74001.log 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43451.log 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.89559.log.gz 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- 89.4% /var/log/ceph/ceph-client.admin.46695.log -- replaced with /var/log/ceph/ceph-client.admin.48603.log.gz 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46069.log.gz 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.74001.log: /var/log/ceph/ceph-client.admin.43451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89114.log 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74001.log.gz 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43095.log 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46695.log.gz 2024-09-19T07:51:00.464 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41936.log 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43095.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.89114.log.gz -5 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41713.log 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43183.log 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42633.log 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42503.log 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50006.log 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91459.log 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88208.log 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50006.log.gz 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88412.log 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46977.log 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.91459.log.gz 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.88412.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88208.log.gz 2024-09-19T07:51:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91158.log 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43604.log 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.88412.log.gz 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46977.log.gz 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.91158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43734.log 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35904.log 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91158.log.gz 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36259.log 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42555.log 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40833.log 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90529.log 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51187.log 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34442.log 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.90529.log: /var/log/ceph/ceph-client.admin.51187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35272.log 2024-09-19T07:51:00.466 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34442.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51187.log.gz 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47990.log 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37642.log.gz 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90529.log.gz 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43231.log 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47449.log 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43231.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48707.log 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47990.log.gz 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36570.log 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47449.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40885.log.gz 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43842.log.gz 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89274.log 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47449.log.gz 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48707.log.gz 2024-09-19T07:51:00.467 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.0.36570.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44074.log.gz 2024-09-19T07:51:00.468 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.89274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45584.log 2024-09-19T07:51:00.468 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89274.log.gz 2024-09-19T07:51:00.468 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41659.log 2024-09-19T07:51:00.468 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47749.log 2024-09-19T07:51:00.468 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41659.log: /var/log/ceph/ceph-client.admin.47749.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40807.log.gz 2024-09-19T07:51:00.468 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47749.log.gz 2024-09-19T07:51:00.468 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41791.log.gz 2024-09-19T07:51:00.468 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42477.log.gz 2024-09-19T07:51:00.474 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34695.log.gz 2024-09-19T07:51:00.474 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34721.log.gz 2024-09-19T07:51:00.474 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41685.log.gz 2024-09-19T07:51:00.474 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43451.log.gz 2024-09-19T07:51:00.474 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43183.log.gz 2024-09-19T07:51:00.474 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37088.log.gz 2024-09-19T07:51:00.474 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37602.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44191.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41607.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34799.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36887.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36939.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34442.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35478.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35480.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35661.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35676.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42095.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45989.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42206.log.gz 2024-09-19T07:51:00.475 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42182.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42819.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46529.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43604.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45872.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45584.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42399.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43095.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43231.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45136.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37521.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40635.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40833.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40859.log.gz 2024-09-19T07:51:00.476 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42503.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34613.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34985.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37164.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41253.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41659.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34851.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36731.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35272.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42040.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35904.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46306.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42633.log.gz 2024-09-19T07:51:00.477 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43734.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45742.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45768.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45794.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46559.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46605.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45558.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42555.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34669.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37296.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37270.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34933.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37374.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35899.log.gz 2024-09-19T07:51:00.478 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35903.log.gz 2024-09-19T07:51:00.479 INFO:teuthology.orchestra.run.smithi083.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.36259.log.gz 2024-09-19T07:51:00.479 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41936.log.gz 2024-09-19T07:51:00.479 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41713.log.gz 2024-09-19T07:51:00.479 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43052.log.gz 2024-09-19T07:51:00.479 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44302.log.gz 2024-09-19T07:51:00.479 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44348.log.gz 2024-09-19T07:51:00.480 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44896.log.gz 2024-09-19T07:51:00.494 INFO:teuthology.orchestra.run.smithi083.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.28873.log.gz 2024-09-19T07:51:00.497 INFO:teuthology.orchestra.run.smithi083.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-19T07:51:00.529 INFO:teuthology.orchestra.run.smithi184.stderr: 95.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-19T07:51:00.536 INFO:teuthology.orchestra.run.smithi083.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.29100.log.gz 2024-09-19T07:51:00.545 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45532.log.gz 2024-09-19T07:51:00.545 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45418.log.gz 2024-09-19T07:51:00.545 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45506.log.gz 2024-09-19T07:51:00.545 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45188.log.gz 2024-09-19T07:51:00.545 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45214.log.gz 2024-09-19T07:51:00.596 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39812.log.gz 2024-09-19T07:51:00.608 INFO:teuthology.orchestra.run.smithi184.stderr: 93.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-19T07:51:03.377 INFO:teuthology.orchestra.run.smithi083.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-19T07:51:04.154 INFO:teuthology.orchestra.run.smithi184.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-19T07:51:06.030 INFO:teuthology.orchestra.run.smithi184.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-19T07:51:09.089 INFO:teuthology.orchestra.run.smithi083.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-19T07:51:27.032 INFO:teuthology.orchestra.run.smithi184.stderr: 94.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-09-19T07:51:27.545 INFO:teuthology.orchestra.run.smithi083.stderr: 94.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-09-19T07:51:29.325 INFO:teuthology.orchestra.run.smithi083.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-19T07:51:33.540 INFO:teuthology.orchestra.run.smithi184.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-19T07:51:34.137 INFO:teuthology.orchestra.run.smithi083.stderr: 94.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-09-19T07:51:35.405 INFO:teuthology.orchestra.run.smithi184.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-09-19T07:51:36.023 INFO:teuthology.orchestra.run.smithi083.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-19T07:51:40.024 INFO:teuthology.orchestra.run.smithi083.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-19T07:51:46.412 INFO:teuthology.orchestra.run.smithi184.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-19T07:51:46.471 INFO:teuthology.orchestra.run.smithi083.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-19T07:51:47.808 INFO:teuthology.orchestra.run.smithi184.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-19T07:51:50.352 INFO:teuthology.orchestra.run.smithi184.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-19T07:54:03.568 INFO:teuthology.orchestra.run.smithi083.stderr: 96.0% -- replaced with /var/log/ceph/ceph-client.0.36570.log.gz 2024-09-19T07:54:03.569 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-19T07:54:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:real 3m3.402s 2024-09-19T07:54:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:user 4m25.405s 2024-09-19T07:54:03.569 INFO:teuthology.orchestra.run.smithi083.stderr:sys 0m12.231s 2024-09-19T07:54:06.032 INFO:teuthology.orchestra.run.smithi184.stderr: 96.1% -- replaced with /var/log/ceph/ceph-client.1.34414.log.gz 2024-09-19T07:54:06.034 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-09-19T07:54:06.034 INFO:teuthology.orchestra.run.smithi184.stderr:real 3m5.856s 2024-09-19T07:54:06.034 INFO:teuthology.orchestra.run.smithi184.stderr:user 4m29.917s 2024-09-19T07:54:06.034 INFO:teuthology.orchestra.run.smithi184.stderr:sys 0m12.821s 2024-09-19T07:54:06.035 INFO:tasks.ceph:Archiving logs... 2024-09-19T07:54:06.036 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786/remote/smithi083/log 2024-09-19T07:54:06.037 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-19T07:54:34.925 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786/remote/smithi184/log 2024-09-19T07:54:34.926 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-19T07:55:03.144 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-19T07:55:03.155 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-19T07:55:03.155 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-19T07:55:03.158 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-19T07:55:03.236 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-19T07:55:03.245 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-19T07:55:03.245 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T07:55:03.248 DEBUG:teuthology.orchestra.run.smithi184:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T07:55:03.297 INFO:teuthology.orchestra.run.smithi083.stdout: remote refid st t when poll reach delay offset jitter 2024-09-19T07:55:03.297 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================== 2024-09-19T07:55:03.297 INFO:teuthology.orchestra.run.smithi083.stdout:+hv01.front.sepi 67.205.162.81 3 u 53 64 377 0.066 -5.161 0.499 2024-09-19T07:55:03.297 INFO:teuthology.orchestra.run.smithi083.stdout:+hv02.front.sepi 63.231.80.2 3 u 2 64 377 0.079 -1.659 0.824 2024-09-19T07:55:03.297 INFO:teuthology.orchestra.run.smithi083.stdout:*hv03.front.sepi 74.6.168.72 3 u 50 64 377 0.079 -5.139 0.502 2024-09-19T07:55:03.297 INFO:teuthology.orchestra.run.smithi083.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-19T07:55:03.303 INFO:teuthology.orchestra.run.smithi184.stdout: remote refid st t when poll reach delay offset jitter 2024-09-19T07:55:03.303 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================== 2024-09-19T07:55:03.303 INFO:teuthology.orchestra.run.smithi184.stdout:*hv01.front.sepi 67.205.162.81 3 u 104 128 377 0.059 -6.444 0.762 2024-09-19T07:55:03.303 INFO:teuthology.orchestra.run.smithi184.stdout:+hv02.front.sepi 63.231.80.2 3 u 118 128 377 0.079 -3.886 1.470 2024-09-19T07:55:03.303 INFO:teuthology.orchestra.run.smithi184.stdout:+hv03.front.sepi 74.6.168.72 3 u 118 128 377 0.064 -7.415 1.270 2024-09-19T07:55:03.304 INFO:teuthology.orchestra.run.smithi184.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-19T07:55:03.304 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-19T07:55:03.314 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-19T07:55:03.314 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-19T07:55:03.325 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-19T07:55:03.335 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-19T07:55:03.346 INFO:teuthology.task.internal:Duration was 2809.277072 seconds 2024-09-19T07:55:03.346 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-19T07:55:03.355 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-19T07:55:03.356 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-19T07:55:03.358 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-19T07:55:03.523 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-19T07:55:03.523 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-09-19T07:55:03.524 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-19T07:55:03.535 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi184.front.sepia.ceph.com 2024-09-19T07:55:03.535 DEBUG:teuthology.orchestra.run.smithi184:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-19T07:55:03.552 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-19T07:55:03.552 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-19T07:55:03.581 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-19T07:55:03.629 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-19T07:55:03.630 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-19T07:55:03.657 DEBUG:teuthology.orchestra.run.smithi184:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-19T07:55:03.815 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-19T07:55:03.824 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-19T07:55:03.825 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-19T07:55:03.851 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-19T07:55:03.875 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-19T07:55:03.885 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-19T07:55:03.897 DEBUG:teuthology.orchestra.run.smithi184:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-19T07:55:03.917 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-09-19T07:55:03.941 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = core 2024-09-19T07:55:03.970 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-19T07:55:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:55:03.992 DEBUG:teuthology.orchestra.run.smithi184:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-19T07:55:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T07:55:04.014 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-19T07:55:04.024 INFO:teuthology.task.internal:Transferring archived files... 2024-09-19T07:55:04.026 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786/remote/smithi083 2024-09-19T07:55:04.026 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-19T07:55:04.057 DEBUG:teuthology.misc:Transferring archived files from smithi184:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911786/remote/smithi184 2024-09-19T07:55:04.058 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-19T07:55:04.078 INFO:teuthology.task.internal:Removing archive directory... 2024-09-19T07:55:04.078 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-19T07:55:04.097 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-19T07:55:04.123 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-19T07:55:04.134 INFO:teuthology.task.internal:Not uploading archives. 2024-09-19T07:55:04.134 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-19T07:55:04.145 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-19T07:55:04.145 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-19T07:55:04.147 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-19T07:55:04.151 INFO:teuthology.orchestra.run.smithi083.stdout: 658463 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 19 07:55 /home/ubuntu/cephtest 2024-09-19T07:55:04.171 INFO:teuthology.orchestra.run.smithi184.stdout: 658636 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 19 07:55 /home/ubuntu/cephtest 2024-09-19T07:55:04.172 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-19T07:55:04.182 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-19T07:55:04.210 INFO:teuthology.run:Summary data: description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} duration: 2809.2770717144012 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-19T07:55:04.210 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-19T07:55:04.296 INFO:teuthology.run:pass